Runbook 18: The Day After

The morning after the fleet went fully bootc, I woke up to a dead forge and spent the day learning that "it survived the migration" and "it survives a reboot" are two very different claims. That was the first papercut; it was not the last. This runbook collects the ones that only show up once the build is "done" and real time starts passing: a two-second database hiccup that became a seven-hour outage, a fleet quietly running on a third of the disk I specified, a root password that was never set, an ansible key that was never really a service account, and two storage leaks that only ratchet upward. Notes-to-self that double as a blog draft; verify snippets against the itg-bootc and infra repos.

Key facts I keep needing: forge is itg-prd-forge at 10.10.10.24, and it is three things at once, Forgejo, the container registry, and the CI hub, plus it serves the very image it upgrades itself from. Forgejo runs as a Quadlet that Requires=postgres.service and var-lib-forge.mount, both on the forge.network podman network. The fleet auto-updates nightly via bootc-fetch-apply-updates.timer and reboots into the new image.

---

Part 1: The reboot

The morning: a 502 that wasn't what it looked like

On the road, on public wifi over Tailscale, Proxmox and Transmission loaded fine but forge, Sonarr, and Radarr all threw 502s. The split is the clue: Proxmox is not behind SWAG (I hit it directly over the subnet route), Transmission goes through SWAG and SWAG could reach it, but the other three went through SWAG and SWAG could not reach their backends. A 502 means the proxy is up and the upstream is not.

The SWAG error log said which flavor of "not":

connect() failed (111: Connection refused) while connecting to upstream: "http://10.10.10.24:3000/...", host: "forge.itguyeric.com"

Connection refused, not "no route to host." The box at .24 was up and answering; nothing was listening on :3000. So this was not the DHCP-IP-churn I first guessed, it was Forgejo itself being down. Read the error number before theorizing: 111 (refused) means the service is dead, 113 (no route) or a timeout means the host or its address is wrong. Different word, different fix.

The single point of failure I built without noticing

sudo systemctl restart forgejo.service brought it straight back, and the whole fleet snapped back with it: the registry answered, the starved runners resumed fetching tasks, and the queued builds fired off on their own. That recovery is also the warning. Forge is a single point of failure with no self-recovery path, and this outage proved every edge of it:

• Down forge means no box can bootc upgrade or pull an AutoUpdate image, because forge is the registry.
• The runners can't fetch tasks, so nothing builds. My "how did the overnight rebuilds go" answer was that they didn't, the runners were starved all night.
• Forge can't even upgrade itself out of it: bootc upgrade on forge returned 502 Bad Gateway trying to pull its own image from the registry it wasn't serving. A snake that can't eat its own tail.

So when forge falls and can't get up, it takes the fleet's entire update and build capability with it, silently, until a human runs one restart.

Root cause: a two-second hiccup, a seven-hour outage

The journal told the whole story. The nightly upgrade rebooted forge at 03:20. Two things happened around that boot.

On the way down, podman tore down forgejo's container network and tried to (re)start aardvark-dns through a transient systemd scope, but systemd refused because a reboot was already queued:

aardvark-dns failed to start: ... Transaction ... is destructive (systemd-reboot.service has 'start' job queued...)

On the way back up, postgres and forgejo raced and lost:

03:22:34  postgres.service: start operation timed out. Terminating.   (first start hung ~57s)
03:22:46  postgres.service: Failed with result 'timeout'.
03:22:47  Dependency failed for forgejo.service.
03:22:47  postgres.service: Scheduled restart job (Restart=always)
03:22:49  postgres: database system is ready to accept connections

postgres's first start after the reboot hung for about 57 seconds, almost certainly stalled on the same podman network setup the shutdown left in a bad state, and got killed by its start timeout. forgejo Requires=postgres.service, so the instant postgres hit "failed," forgejo's start job was cancelled. Then postgres's own Restart=always restarted it and it came up clean two seconds later. But forgejo was already gone and never came back, because of the rule below. Seven hours of dead forge over a two-second database stumble that had already fixed itself.

The rule that bit: Restart does not cover dependency failures

Restart=always restarts a service whose own process dies. It does not retry a start that systemd cancelled because a Requires= dependency failed. A dependency-failed job is terminal; nothing retries it. So Restart=always on forgejo was worthless here, the failure was never forgejo's process, it was postgres briefly failing underneath it. This is the single most important thing to internalize about Quadlets on a box that reboots unattended: a hard Requires= on anything that can be slow or flaky at boot is a loaded gun, because one transient miss strands the dependent service permanently.

The fixes

Three changes, each targeting a different link in the chain.

forgejo: soften the service dependency, keep the storage dependency. Requires=postgres.service became Wants=postgres.service (with After= kept for ordering). Now if postgres is briefly down at boot, forgejo still starts, fails to reach the DB, exits, and its Restart=always retries until postgres, recovered by its own restart, is there. Self-healing instead of a permanent hole. The data mount stays Requires=var-lib-forge.mount, because a container must never start without its storage.

postgres: give the first start room. TimeoutStartSec=300, so a slow first start after a reboot completes on the first try instead of being killed and forced into the retry that cascaded forgejo. Defense in depth with the forgejo change: postgres is more likely to come up cleanly, and if it still doesn't, forgejo now recovers on its own.

forge: off unattended reboots entirely. This is the important one. systemctl mask bootc-fetch-apply-updates.timer, baked into forge's Containerfile so it survives rebuilds. The single point of failure with no self-recovery must not reboot itself while no one is watching. Forge now updates by hand, sudo bootc upgrade && sudo systemctl reboot, on my terms, watching it come back. Every other box keeps auto-updating; forge became deliberate.

What the fleet audit found

With the pattern understood, I swept every Quadlet for it.

The fatal shape, a container that Requires= another container, existed only on forge. Nothing else hard-depends on another container (Foundry calls ddb-proxy at runtime but does not Requires= it). Forge was unique because it was the one multi-container app with a real service-to-service startup dependency.

Everything else Requires= only its data mounts. I did not soften those, on purpose: a container that starts without its mount writes into the empty mountpoint instead of the real disk, which is worse than a restart loop. These are leaf containers with Restart=always, so they self-heal from the podman-network start hang, which is why arr's Sonarr and Radarr recovered on their own that morning where forge's dependency chain could not. The residual risk is a mount itself timing out at boot (same no-retry rule), but local xfs is fast and the NFS mount waits on network-online, so it is low, and the mitigation is mount reliability plus the reboot posture, not weakening the dependency.

The audit did turn up one latent bug: transmission.container on the book box mounted /var/mnt/downloads with no :z relabel flag, the same SELinux trap the arr downloads volume hit. It works today only because that directory got labeled correctly at some point; a clean rebuild would have failed it. Fixed by adding :z.

One deliberate tradeoff worth recording: I later put the arr containers on a shared podman network so they address SAB by name (sabnzbd:8080) instead of a DHCP IP. That turns on aardvark DNS for arr, the same component whose reboot race started all of this. I accepted it because arr's containers are leaf services with Restart=always and no inter-container Requires, so a boot hiccup self-heals rather than stranding them. The no-aardvark alternative was a DHCP reservation and addressing by a now-stable IP.

---

Part 2: The papercuts that took weeks to surface

The reboot bug announced itself with an outage. The next four did not. Each one had been wrong since the day the fleet was built, and each stayed invisible until something unrelated forced me to look.

Specified 32G, got 10G

The game box wedged. bootc update --apply refused to stage a 2.4 kB layer:

error: Upgrading: Insufficient free space ... (available: 0 bytes required: 2.4 kB)

"0 bytes available" with 212M free is ostree's min-free-space-percent reserve (3% by default) refusing to dip below its floor, not a literally full disk. df said /dev/sda4 was 8.5G at 98%. My first theory was container churn, and pruning did buy room, but the real question was why an 8.5G root existed at all on a VM I built with 32G.

lsblk answered it: sda was 10G total. Not 32G with unallocated space, 10G, with sda4 already filling it. growpart said NOCHANGE: partition 4 ... cannot be grown.

Root cause, in two independent halves, both in the IaC:

• vm_sizes in group_vars/all/sizes.yml defined only cores and memory. There was no disk field at all. The 32G I remembered specifying was never encoded anywhere.
• The provisioning role imports the bootc qcow2 with community.proxmox.proxmox_disk, import_from:, and no size parameter, so Proxmox creates scsi0 at exactly the qcow2's virtual size, about 10G. Data disks were always correct because those pass an explicit size:. Only the OS disk was orphaned.

So every VM in the fleet had been running on a ~8.5G root since day one, regardless of tier. It stayed invisible because nothing else had filled up yet.

The fix needs both halves, and neither works alone. sizes.yml now carries disk: 32 per tier (with an os_disk_size per-host override), and the role resizes scsi0 to it immediately after the import, guarded to fire only on a fresh import because Proxmox resize is grow-only. Then a grow-root unit baked into itg-base expands the partition and the filesystem into whatever the disk actually is. It runs on every boot, not just the first, so a later Proxmox resize is picked up by a plain reboot instead of hand surgery.

Two traps the unit encodes, both learned by hitting them:

• xfs_growfs /sysroot fails with "Read-only file system." bootc mounts the root filesystem read-only at /sysroot. /var is the writable mount of the same filesystem, so that is what you grow. This is the one that cost the most time, because the partition resize had already succeeded and it looked like the whole operation failed.
• growpart exits 2, not 0, when there is nothing to grow. That is the normal steady-state result on every boot after the first, so a naive ExecStart would mark the unit failed forever. Treat 2 as success and never fail the boot over any of it.

Remediating the existing fleet needed no rebuild and no downtime: qm resize <vmid> scsi0 32G on the host, then growpart /dev/sda 4 && xfs_growfs /var in the guest, both online. Six of eight grew. The other two, book and forge, came back shrinking disks is not supported and NOCHANGE, because they were already at 34296M. They had been sized correctly during their own migrations. That is exactly why the bug survived so long: the two boxes I touched most were the two that were fine.

The root password that was never set

While auditing console access I found root had no usable password. The build looked correct:

RUN --mount=type=secret,id=rootpw \
    echo "root:$(cat /run/secrets/rootpw)" | chpasswd -e

chpasswd -e takes an already-hashed password. The pipeline fed it ${{ secrets.ROOTPW_HASH }}. The secret in Forgejo was named ROOTPWD_HASH. One letter.

A missing secret in Actions resolves to an empty string, so the build became echo "root:" | chpasswd -e, which writes an empty shadow field and exits 0. Green pipeline, passwordless root, shipped to the entire fleet, for as long as that secret had existed.

Two fixes. The names now match, and the Containerfile validates the value before using it:

case "$hash" in
  '$6$'*|'$y$'*|'$2b$'*) : ;;
  '') echo "FATAL: rootpw secret is empty or missing" >&2; exit 1 ;;
  *)  echo "FATAL: rootpw secret is not a crypt hash" >&2; exit 1 ;;
esac

The guard matters more than the rename. A build that can silently produce a weaker system than intended must fail loudly, because nothing downstream will ever tell you. Generate the value with openssl passwd -6 so the plaintext never reaches shell history.

And the bootc-specific sting: fixing the image does not fix deployed hosts. /etc/shadow counts as locally modified on any running system, so the 3-way merge keeps the local copy. Every existing host needed a manual passwd root. The image fix only helps machines built from that point forward. Verify the image itself before believing it worked:

podman run --rm forge.itguyeric.com/itguyeric/itg-base:latest getent shadow root

Making ansible an actual service account

The ansible account had full passwordless root on every box via %wheel ALL=(ALL) NOPASSWD: ALL, and its key was an RSA keypair whose comment read itguyeric@friday. It was generated on my Mac, lived on my Mac, and was also handed to CI. A human key wearing a service account's badge.

Three structural problems, only one of which was obvious:

• The public key was deployed by a tmpfiles C line, which is copy-if-absent. Once ~/.ssh/authorized_keys existed on a host, no change to the key in the image ever reached it again. The fleet had been frozen on that file since first boot.
• ansible.cfg set private_key_file globally, so a playbook run from my Mac connected as my local user (itguyeric) while presenting the ansible key. A mismatched pair that only worked by accident of ssh fallback.
• The account inherited root through wheel, so the human path and the machine path could not be tuned separately.

The rebuild:

The key now lives in /usr, not in a home directory. The authorized_keys file is generated into /usr/share/ssh-keys/ansible.authorized_keys at build time and pointed at with an sshd Match block. It cannot be edited on a running host, nobody can append a backdoor key to ~/.ssh, and it always tracks the image, which kills the copy-if-absent staleness outright.

Match User ansible
    AuthorizedKeysFile /usr/share/ssh-keys/ansible.authorized_keys
Match all

That trailing Match all is not optional. A Match block stays in effect for the remainder of the parsed config, and sshd_config.d/*.conf is pulled in by an Include near the top of sshd_config. Without the reset, those settings leak onto every user parsed afterward.

The key line carries from="10.10.10.0/24",restrict,pty. restrict drops port, agent, and X11 forwarding; pty is added back because become sometimes wants one. I deliberately did not pin individual IPs: DHCP leases in this fleet have moved twice in two days, and a moved lease with an IP allowlist locks ansible out of every host simultaneously. More honestly, anyone holding the key is already inside the LAN, so subnet allowlisting is asking a burglar to use a particular door in a house they are already standing in. The real control is that the key exists nowhere except a CI secret.

ansible left wheel for its own /etc/sudoers.d/ansible. Same privilege, but now independently tunable. Note that systemd-sysusers only ever adds group membership, so dropping m ansible wheel from the image does not remove it from already-deployed hosts; that needs a gpasswd -d ansible wheel.

The key itself was rotated to a CI-only ed25519 pair, in three phases so nothing could strand: trust both keys and roll, swap the ANS_SSH_KEY secret and verify, then drop the old key and roll again. The workflows already wrote the secret to disk at job start and shredded it at the end, so the private half never persists anywhere.

The most valuable lesson came from the verification. After swapping the secret, the provision workflow came back green — and it had still authenticated with the old key. A repo-level ANS_SSH_KEY was shadowing the new user-level one, and because phase 1 trusted both keys, nothing broke and nothing complained. The pipeline could not tell me which credential it used. sshd could:

Accepted publickey for ansible from 10.10.10.176 ... ssh2: RSA SHA256:bjV...      <- old key
Accepted publickey for ansible from 10.10.10.176 ... ssh2: ED25519 SHA256:p6ZJ... <- after deleting the duplicate

"The pipeline is green" and "the pipeline used the credential I think it used" are different claims. The sshd accept log records algorithm and fingerprint on every success and is the only real evidence during a key migration. Phase 1's dual trust is what made discovering this safe rather than an outage.

Two storage leaks that only ratchet up

The disk problem exposed a second one underneath it: nothing in the fleet ever gave storage back.

Podman image churn. AutoUpdate=registry pulls a new :latest and leaves the previous one untagged. Nothing prunes it. Multiply by every container on every host across months and that is what actually filled the game box's root. Fixed with a podman-prune.timer in the base image, weekly, dangling images only. Deliberately not podman image prune -af, which collects any image without a container and would silently evict on-demand workloads like ARK (whose WantedBy is commented out) every week. The runners rebuild every image on every run and generate far more churn, so they get the aggressive version as an if: always() step in the pipeline instead: dangling, plus unreferenced images older than 72h, which spares whatever the current run just built.

LVM thin allocation. This one is sneakier, because the numbers disagree and the filesystem's number is the lie:

vm-206-disk-2   33.49g   96.08     <- book: pool says 96% allocated
/dev/sda4  32G  14G  19G  44% /var <- book: filesystem says 44% used
vm-200-disk-0   32.00g   99.97     <- coulson, at 60% real usage

A thin volume's Data% only ever climbs. Deleting a file frees it in the filesystem but never tells the pool, so the allocation ratchets upward until the pool is exhausted, which corrupts every VM on it at once rather than one at a time. Proxmox's own warning suggests thin_pool_autoextend_threshold, which does nothing here: autoextend needs free extents in the volume group to extend into, and there are none.

The fix is discard=on,ssd=1 on every disk plus fstrim. Three checks worth doing in order:

• lvs -o lv_name,discards vg-fast/fast-vmstore must read passdown. If it does not, trimming inside the guests reclaims nothing and everything after is wasted effort.
• Changing discard lands as a pending change. A reboot from inside the guest reuses the same QEMU process and will not pick it up; qm reboot from the host recreates it and does. (In practice Proxmox applied several live, so check rather than assume.)
• lsblk -D in the guest is the proof. DISC-MAX 0B means it did not take, and fstrim will silently do nothing.

Then fstrim.timer keeps it honest, scheduled after the weekly prune so freed blocks go back to the pool right behind the cleanup.

Gotchas, in the order they bit

• Read the errno on a 502. 111 refused means the backend is dead; 113/timeout means the host or address is wrong. It picks the fix.
• A hub that hosts its own registry can't upgrade itself when it's down. bootc upgrade on forge 502s pulling its own image. There is no bootstrap path out; only a human restart.
• Restart=always does not retry a dependency-failed start. It restarts dead processes, not cancelled start jobs. A hard Requires= on a boot-flaky dependency will strand the service permanently.
• The first container start after a bootc reboot can hang ~a minute on aardvark, then a retry is instant. Give slow-to-start containers a generous TimeoutStartSec so they aren't killed on that first try.
• Soften service dependencies to Wants=, never storage dependencies. Wants=postgres plus Restart=always self-heals; Wants= on a mount would let a container start without its disk and write to the wrong place.
• The single point of failure should not take unattended reboots. Mask its auto-apply timer in the image and upgrade it by hand.
• "Insufficient free space, available: 0 bytes" is ostree's 3% reserve, not a full disk. You hit it while df still shows a couple hundred MB free.
• import_from sizes a disk from the qcow2, and there is no size parameter on import. If the tier table has no disk field, nothing anywhere encodes the size you think you specified.
• Grow /var, not /sysroot. Same filesystem, but /sysroot is mounted read-only on bootc and xfs_growfs refuses it.
• growpart exits 2 when there is nothing to grow. That is success in steady state; treat it as such or the unit fails on every boot forever.
• chpasswd -e with an empty secret writes an empty password and exits 0. Validate build secrets and fail the build; a silently weaker system is the worst possible outcome of a green pipeline.
• /etc/shadow is locally modified, so image password changes never reach existing hosts. Same for anything else in /etc a running system touches.
• tmpfiles C is copy-if-absent. It will never update a file that already exists, which makes it the wrong tool for anything you might need to rotate.
• An sshd Match block leaks to everything parsed after it. End any dropped-in config with Match all.
• systemd-sysusers only adds group membership, never removes it. Dropping an m line from the image does nothing to deployed hosts.
• A green pipeline does not tell you which credential it used. The sshd accept log records algorithm and fingerprint; during a key rotation that is the only real evidence.
• Thin-pool Data% only ratchets up without discard, and thin_pool_autoextend_threshold cannot help when the VG has no free extents.
• discard=on is a pending change until the QEMU process is recreated. Verify with lsblk -D before trusting fstrim.

Part 3: The bastion, and the DNS that only half existed

Coulson's replacement is an image, not a pet

Coulson was the old build box and the jump host I lived on: a hand-built VM with tmux sessions I attached to over SSH. Its replacement, itg-prd-bast, is the same job done the fleet's way. The image bakes the jump-host tooling (tmux, mosh, git, ansible-core, python3-pip) plus the cockpit web console, since the bastion is the one box where a management GUI earns its keep. The rest of the fleet carries the cockpit plugins but never enables the socket; you do not want a console listening on every media box.

The tmux session is create-on-connect. friday runs one alias, ssh -t itg-prd-bast /usr/local/bin/bastion.sh, and the script builds the session on first connect and attaches. There is no boot service and no lingering. On a day I do not log in, bast holds no sessions and sits idle. The nightly reboot wipes whatever was open, and the next connect rebuilds it from the script. That is the point of moving off pets: the value is in the script, not in a session I am afraid to lose. A warm pane per prd host, the way Coulson used to hold them, was a pets-era instrument anyway, since the prd boxes now reboot nightly and reprovision from images. You do not keep standing connections to cattle.

The script is baked, not deployed, because /usr/local is read-only

My first instinct was to ship bastion.sh through Ansible at commission, next to the key it uses. That was wrong twice over. The commission run failed with "Destination /usr/local/bin not writable," because on a bootc host /usr/local lives inside the immutable /usr and you cannot write there at runtime, only at build. And the reasoning that put it in Ansible was itself off: the script is generic and not a secret. It only references ~/.ssh/id_bastion by path; the key is SCP'd in and never baked. bast is the only thing that runs the script. So it belongs in the image, copied into /usr/local/bin at build time when /usr is still writable, present read-only on every boot. The key stays machine-local, the script stays in the image, two different lifetimes and two different homes.

What is left is a clean two-key chain, each doing one job: friday reaches bast as itguyeric with id_rsa_itg (baked as the authorized key), and from bast the infra window reaches the pets as root with id_bastion (SCP'd, never baked).

The pets were never in DNS

The first time I ran the bast alias, the infra window could not resolve itg. The fleet had spoiled me. Every bootc box DHCPs with a static hostname and OPNsense's Unbound auto-registers that name under int.itguyeric.com, so bast, the runners, and the arr box resolve each other by short name with no configuration at all. itg (Proxmox) and itg-net-opn (OPNsense) are static pets. They never call into DHCP, so they were never registered. The auto-magic that covers the fleet does not cover the two boxes that predate it.

The fix is a manual Unbound host override, and the zone is the whole game. Three domains are in play. itguyeric.com holds the split-DNS billboards from Runbook 15: public names that also live in Cloudflare, overridden internally to point at SWAG. int.itguyeric.com is the internal host zone the fleet registers into, and it is the search domain every bootc box carries. So when bast types ssh root@itg it appends int.itguyeric.com and asks for itg.int.itguyeric.com. My override for itg sat under itguyeric.com, a zone bast never queries for a bare short name, so it missed every time. Moving itg's record into int.itguyeric.com fixed it, and that is its correct home anyway, alongside the rest of the fleet.

itg-net-opn needed no override at all, and that is the tell. OPNsense answers for its own name, so itg-net-opn.int.itguyeric.com already resolved through the box's own registration. The lesson: a static pet reached by short name needs a hand-written record in the internal zone, not the public one, and the only pets that need it are the ones that do not answer for themselves. The third domain, home.itguyeric.com, was a legacy remnant in itg's own resolver config. Fixing that (search domain to int.itguyeric.com, and itg's /etc/hosts pointing its own name at its LAN IP instead of the WAN egress) is what finally made the hypervisor consistent with everything built after it.

The through-line

Surviving a migration means it came up once, with me watching and fixing. Surviving a reboot means it comes up every time, at 3 a.m., with no one there. The gap between those two is entirely in the dependency graph: what Requires= what, whether a failed start can retry, and whether the one box that everything else depends on is allowed to fall over on a timer.

The rest of this runbook is the slower version of the same lesson. The disks, the root password, the ansible key, and the storage leaks were all wrong from day one, and all four produced a green build and a working fleet the entire time. What they have in common is that nothing was checking the thing I believed. The tier table never held a disk size, the pipeline never verified the secret it consumed, the key file could not be updated after first boot, and the pool never learned about deleted blocks. None of that shows up in a status check; it shows up weeks later when something unrelated forces you to look. The day after is where you find out which fleet you actually built, and it keeps being the day after for a while.