Homelab Runbook — 2026-07-16

DNS migration to Cloudflare, first real workload (itg-prd-book) data restore, Kobo Sync, and MyAnonaMouse seedbox reconnect. Written for future-me. Everything runs on Fedora bootc image-mode infra, so the durable fixes live in the image (git), and machine-local secrets live on the box.

Key facts I keep needing: book VM is 10.10.10.226, colo public egress is 66.85.74.134, the worker service user is UID/GID 1001, and the Calibre library inside the containers is /books (which maps to /var/lib/book/library on the data disk).

---

1. Namecheap DNS → Cloudflare

Why this happened

The SWAG cert renewal died and took forge plus every subdomain with it. Root cause was not SWAG at all. It was Namecheap's API. To use their API you have to meet one of: 20+ domains, $50 account balance, or $50 spent in the last 2 years. I have 3 domains and my last qualifying spend rolled out of the 2-year window, so Namecheap silently kept letting the harmless read commands work (getList) while blocking the DNS-write commands (getHosts/setHosts) that certbot needs. That is why it broke "for no reason."

How I proved it: hit the Namecheap API by hand and watched getList return OK but getHosts return error 1011102 (API access not enabled). Same key, same IP, different command. That split is the fingerprint.

The migration (parity-first, zero downtime)

The golden rule: get every record into Cloudflare before flipping nameservers. Mail follows the MX record, so as long as MX is identical on both sides during propagation, no mail is ever lost, at any time of day. The only way to lose anything is a record missing on the new side at cutover.

1. Add the domain as a zone in Cloudflare (free plan). Let it scan, then compare its import against Namecheap's Advanced DNS record by record.
2. Copy A, CNAME, MX, and real TXT (SPF/DKIM/DMARC). Do not copy NS, SOA, the _acme-challenge graveyard (stale certbot litter), or old CA validation CNAMEs (the comodoca.com ones were dead PositiveSSL leftovers).
3. Grey-cloud (DNS only) everything for the homelab services, so Cloudflare answers with the real WAN IP and SWAG keeps serving its own cert, exactly like Namecheap did. Orange-cloud is a separate project.
4. Flip nameservers at the registrar (Namecheap → Domain tab → Nameservers → Custom DNS → the two Cloudflare nameservers, mine were clarissa/max). Keep the old Namecheap records intact as a rollback net.
5. DNSSEC stays OFF during a migration. A stale DS record pointing at the old provider's key = SERVFAIL and the domain vanishes. Re-enable it in Cloudflare later if wanted.

hendricks.life redirect gotcha

Namecheap's "URL Redirect Record" is proprietary. Cloudflare has no equivalent record type, so the imported A record just pointed at Namecheap's redirect server and would die at cutover. Rebuilt it natively: a proxied (orange) placeholder A record at 192.0.2.1 for the apex and www, plus a Single Redirect rule (Rules → Redirect Rules) matching Hostname eq hendricks.life OR www.hendricks.life, static 301 to https://itguyeric.com. Match on Hostname, not URI, so it catches http and https and any path.

---

2. SWAG cert reissue via Cloudflare DNS-01

In the SWAG podman-compose, changed DNSPLUGIN=namecheap to DNSPLUGIN=cloudflare, dropped a scoped Cloudflare API token (Zone.DNS Edit, single zone) into swag/dns-conf/cloudflare.ini as dns_cloudflare_api_token, chmod 600, and recreated the container.

Lesson: NXDOMAIN right after a zone goes live is timing, not a broken token. The first reissue failed with NXDOMAIN on every _acme-challenge. The certbot log proved the Cloudflare API calls all returned 200 and the records were created and cleaned up fine. The real problem: the zone had just gone authoritative and earlier failed runs had poisoned Let's Encrypt's resolvers with cached "does not exist" answers. certbot only waits 10s. Fix was simply to wait ~30 min for the negative caches to expire, then re-run. It sailed through. Confirm the zone is healthy first with dig NS <domain> @1.1.1.1 +short (should show Cloudflare) and dig <sub> @<cf-nameserver> +short.

---

3. itg-prd-book data restore

The book server never got rebuilt after the Proxmox move, so production appdata was empty and I restored from the linode/ backup on itg. Library (ebooks, has metadata.db) went to /var/lib/book/library; the calibre, calibre-web, and transmission config folders to their matching dirs under /var/lib/book.

rsync was not in the image. Added rsync to the itg-base Containerfile (one line, whole fleet gets it), but for the restore itself used tar-over-SSH since rsync has to exist on both ends:

ssh root@10.10.10.2 'tar -C /path/on/itg -cf - .' | sudo tar -C /var/lib/book/<dest> -xf -

Then sudo chown -R 1001:1001 /var/lib/book and start the apps. Stop the apps before restoring, always, or you can corrupt the live databases.

calibre-web "attempt to write a readonly database"

This one looked like SELinux and looked like ownership and was neither. The SELinux label matched, the file was owned by 1001, the app runs as 1001, no immutable flag, and the directory was writable (it was writing its own log there). The real cause: during the brief window after the tar extract, app.db was owned by nobody before the chown landed. The running calibre-web tried to write it, failed, and SQLAlchemy poisoned its whole session (PendingRollbackError cascades every request). My systemctl start afterward was a no-op because the container was already running, so it never reopened the file. Fix was sudo systemctl restart calibre-web to force a clean reopen. Lesson: a cascading PendingRollbackError after a file swap usually means a stale handle, restart before you go spelunking.

Old library path

Restored app configs remembered the Linode-era path (/media), but the new Quadlets mount the library at /books. Pointed Calibre desktop at /books (do NOT let it "repair"/create a new library) and calibre-web resolved it once it could write again.

calibre-web password reset

Restored app.db carries the old accounts. To reset (no vault copy of the password):

sudo podman exec calibre-web /lsiopy/bin/python3 -c "
from werkzeug.security import generate_password_hash
import sqlite3
con = sqlite3.connect('/config/app.db')
con.execute('UPDATE user SET password=? WHERE name=?', (generate_password_hash('NEWPASS'), 'itguyeric'))
con.commit(); print('rows changed:', con.total_changes)"

Then systemctl restart calibre-web. Use /lsiopy/bin/python3 so werkzeug is available.

Comics

Comics (CBZ) are loose files with no metadata.db, so they cannot be restored by copying into the library. Calibre only shows what is in metadata.db. Import them through Calibre's "Add books" (staged them into /var/lib/book/calibre/_import, which the container sees as /config/_import, then deleted the staging). Never hand-copy into the library folder.

---

4. Kobo Sync with calibre-web

The valid HTTPS cert from the Cloudflare fix is a prerequisite here, the Kobo demands it.

1. In calibre-web: Admin → Edit Basic Configuration → enable Kobo sync, set Server External Port to 443.
2. On the user: Create/View Kobo sync token. It gives an api_endpoint=https://calibre-web.itguyeric.com/kobo/<token> line.
3. On the device over USB, edit .kobo/Kobo/Kobo eReader.conf, [OneStoreServices] section, set api_endpoint to that URL (note the original storeapi.kobo.com value first in case I want to revert). Eject cleanly, then Sync.
4. Only EPUB/KEPUB sync to the device. Comics and PDFs stay in calibre-web only.

Verify from SWAG's access log, not calibre-web's (calibre-web does not log request paths by default): sudo tail -f swag/log/nginx/access.log | grep -i kobo, then Sync. Healthy sync shows /kobo/.../v1/initialization at 200 and /kobo/.../v1/library/sync.

Empty sync fix: the restored app.db carried the old Kobo sync state, so calibre-web thought the device already had everything and v1/library/sync returned 200 2 (empty []). The "Force Full Kobo Sync" button wipes that state, next sync returned 200 293859 and the whole library flowed.

---

5. MyAnonaMouse seedbox reconnect

MAM locks account activity to authorized IPs. Moving transmission to the colo meant a new, unrecognized egress (66.85.74.134), so I had to register it as a seedbox session.

The session

On MAM → Preferences → Security, create a session for the seedbox IP, and it must be promoted to a seedbox-type session (Manage Session → tick the dynamic-seedbox checkbox). A plain session returns Incorrect session type - not allowed this function from the update endpoint. Then View IP locked session cookie to get the mam_id.

Verify from the box (source IP matters, run it ON book):

curl -s -b 'mam_id=TOKEN' https://t.myanonamouse.net/json/dynamicSeedbox.php

Want {"Success":true,...}.

Keep it alive (baked into the book image)

Hourly systemd timer + oneshot, secret machine-local in /etc/mam/seedbox.env (chmod 600, never in git). Units live in the image at /usr/lib/systemd/system/mam-seedbox.{service,timer}, enabled via the Containerfile's systemctl enable ... mam-seedbox.timer. The service reads the secret via EnvironmentFile and curls dynamicSeedbox.php. Do not go faster than hourly, MAM rate-limits it. Trigger a manual check with sudo systemctl start mam-seedbox.service and read journalctl -u mam-seedbox.service.

Connectability (this is the one that bit)

MAM showed the client as "can not accept incoming connections - timeout." SWAG cannot help here: it is an HTTP reverse proxy and routes by hostname, but BitTorrent peer traffic has no hostname to route by, so it needs its own real port-forward. That is the one legitimate exception to "everything behind 443," and it is just one port.

• OPNsense: Firewall → NAT → Port Forward, WAN, TCP/UDP, WAN address, port 51413 → 10.10.10.226:51413, with the associated filter rule.
• In transmission, uncheck "Use port forwarding from my router" (UPnP). With a manual forward that box makes the port flap open-then-closed, because transmission reports the failed UPnP result instead of the real reachability. Unchecking it fixed the connectable check instantly.

---

Fleet win

Added rsync to the itg-base Containerfile. One line, one commit, and every host that builds from base has it forever. That is the whole point of image mode.

Pinned / next

• Readarr (+ Prowlarr) on the arr host to auto-grab from MAM and import into Calibre via hardlinks (hardlinks so seeding keeps working). This is the proper automated book pipeline.
• Calibre auto-import so finished downloads land in the library on their own (interim, book-local, if I want it before Readarr).
• Re-enable DNSSEC in Cloudflare for both domains once I feel like hardening.
• Still deferred from before: forge host cutover to the pipeline image (rehearse on a DR droplet first), DR forge on a cloud provider, secrets vault (Bitwarden/Hashi), Kubernetes.