Runbook 14: SWAG and the Website

itg-prd-web migration to Fedora bootc, plus the pipeline overhaul it forced (build and bake consolidation, dynamic runner version, the commission workflow). Written for future-me. Durable fixes live in the image and the workflows (git); machine-local secrets live in vault or on the box.

Key facts I keep needing: web VM is itg-prd-web at 10.10.10.83, vmid 206, size S, no data disk. SWAG runs as a rootful podman quadlet, config at /var/lib/containers/swag bind-mounted to /config, PUID/PGID 1001 (the fleet worker user). Site is served from /config/www. The cert is a wildcard *.itguyeric.com via Cloudflare DNS-01. The website deploys over SSH from a Forgejo runner using a restricted rrsync key.

---

Part 1: itg-prd-web on bootc (SWAG)

The image

Web is deliberately diskless (no data disk). SWAG's /config is small and mostly reproducible: certs reissue via Cloudflare DNS-01, and the static site redeploys from the runner. It lives on machine-local /var, which survives bootc upgrade and is only lost on a full VM re-create. That fits the "not every VM needs a data disk" rule.

bootc/web/ is:

• quadlet/swag.container: lscr.io/linuxserver/swag:latest, AutoUpdate=registry, AddCapability=NET_ADMIN, publishes 80 and 443, Volume=/var/lib/containers/swag:/config:Z. Env: PUID/PGID 1001, URL=itguyeric.com, SUBDOMAINS=wildcard, VALIDATION=dns, DNSPLUGIN=cloudflare, plus TimeoutStartSec=900 so a first-boot cert issuance does not trip systemd's default start timeout.
• tmpfiles creates /var/lib/containers/swag (root), www (owned by ansible so the deploy key can write it), and dns-conf (owned by worker so certbot reads the token).
• rsync-rrsync installed (see gotchas).

The ansible web-config role (gated when: app == 'web') renders the Cloudflare token from vault into /config/dns-conf/cloudflare.ini (owner worker), plants the restricted deploy key in ansible's authorized_keys, and starts SWAG.

The migration play

migrate-web.yml, run by hand from the Mac. Phase A stages /config off the old box to the depot, Phase B stops the old VM, Phase C is import_playbook: site.yml to build the new VM and run web-config, Phase D restores /config on top (excluding dns-conf/*.ini so the vault-rendered token wins). Old-box coordinates live in a temporary host_vars/itg-prd-web-old.yml and a web_old inventory group, both deleted after cutover.

Gotchas, in the order they bit

• Stale DNS after an IP change, not Cloudflare. When forge was unreachable but every other subdomain worked, the cause was the Mac's resolver cache still pointing at the old IP. Fix: sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder, and clear Chrome's own cache at chrome://net-internals/#dns. dig bypasses the OS cache; dscacheutil -q host -a name <host> shows what the OS is handing apps. Lesson: right after an IP change, suspect the cache before the authoritative record.
• rsync missing on the old pet. Phase A failed because the old AlmaLinux box had no rsync. dnf install -y rsync on the old box. New bootc boxes get it from base.
• rrsync is a separate subpackage. Fedora split it out, so base rsync does not ship /usr/bin/rrsync. Add rsync-rrsync to the web image.
• The new rrsync rejects ./. The Python rrsync (rsync 3.2.4+) normalizes the client's ./ destination down to an empty string, then its own safety check fails because the restricted dir does not start with itself-plus-slash. Writing straight into the rrsync root is the thing that breaks. Fix: set the rrsync root to the parent (-wo /var/lib/containers/swag) and have the deploy write into the named subdir www/. So the authorized_keys line is command="/usr/bin/rrsync -wo /var/lib/containers/swag",restrict ... and the deploy rsyncs to ansible@itg-prd-web:www/.
• www ownership after the restore. Phase D restored the old box's /config with rsync -aH, which preserved the old numeric ownership on every subdirectory, so the deploy user (ansible) could not write into them. The tell was Permission denied on mkstemp only in subdirs, never at the top of www. One-time sudo chown -R ansible:ansible /var/lib/containers/swag/www. SWAG still serves fine because it reads as worker (1001) and Hugo's files are world-readable. A fresh provision never hits this (empty ansible-owned www).
• Deploy key scope tradeoff. Because the rrsync root moved up to swag/, the write-only deploy key can technically write anywhere under swag/, not just www/. Acceptable for a CI-only key, and --delete is bounded to the www/ target that gets sent. Tighten later by relocating the served directory if it ever matters.

---

Part 2: The pipeline overhaul

One workflow: pipeline.yml

Consolidated the old separate build.yml and bake.yml into a single pipeline.yml in the bootc repo. Jobs: matrix-prep (decides which images), base (always runs), and image (per-image matrix, fail-fast: false, builds then bakes in one leg).

Behavior by trigger:

• push: builds only the changed image dir(s), no bake, because running VMs just bootc upgrade. A change to a base file (root Containerfile, config.toml, images.json, files/) rebuilds everything.
• nightly (4 AM Central cron): builds and bakes every image.
• manual dispatch with image: <dir-or-name> or all: builds and bakes that selection.

Why build and bake are one job per image and not two jobs: per-image independence. If one image's build step fails, only its bake is skipped and every other image still builds and bakes, because fail-fast: false keeps the matrix legs apart. Two separate jobs would be all-or-nothing (a single failed build skips every bake), because Actions job dependencies cannot be scoped to a single matrix leg. The only hard gate is base: if base fails, nothing downstream runs, which is correct. The visual cost is one line per image in the UI instead of separate build and bake lines; that was an accepted tradeoff.

Dynamic runner version

Stopped hardcoding the Forgejo runner version. The run image resolves the latest release at build time from the Forgejo API (/releases/latest, parsed with python3) and downloads that binary. ARG RUNNER_VERSION=latest with an override to pin an exact version if a release ever misbehaves. Note /releases/latest is already latest-stable (it excludes drafts and prereleases); it is not a bleeding-edge tag. This is how the runners moved to 13.0.0 on their own.

commission.yml (day-0)

commission.yml in the infra repo is the one-stop for a brand-new node: one dispatch builds the image, bakes the qcow2, and provisions the VM. It checks out the bootc repo cross-repo for the build, then runs the ansible provision. Optional recreate: true deletes the VM first (via retire.yml) for a clean-slate rebuild during bring-up. Steady-state stays on pipeline.yml plus bootc upgrade; commission is only for birthing a host. Next customer is the Matrix server.

Secrets

No organization (personal account), so FORGE_TOKEN lives as a user-level Actions secret (user Settings, Actions, Secrets), inherited by every repo I own. It is a personal access token carrying write:package (registry push and pull) AND read:repository (commission's cross-repo checkout). A package-only token authenticates the registry but fails commission's checkout.

ansible.posix on the runners

web-config uses ansible.posix.authorized_key, so the run image bakes ansible.posix next to community.proxmox (ansible-galaxy collection install community.proxmox ansible.posix). Collections are baked into the image, not installed per job, so a new collection means rebuilding the run image and bootc upgrade on the runners (one at a time so a runner is always available).

Forgejo/act quirks worth remembering

• Matrix vars do not interpolate in step name: fields. They work in the job name: and in run:, but a step name like Build ${{ matrix.image }} renders literally in the UI. Use static step names.
• Skipped needs plus always() is fragile. A job whose needs: dependency was skipped gets skipped itself, even with if: always(), and needs.*.result handling is unreliable on act. That is why base always runs (it is a cache-cheap hit when unchanged) instead of being gated, and why every job-level if sticks to plain string checks like needs.matrix-prep.outputs.images != '[]'.

---

Status

tail, game, web, and plex are all on bootc. Arr is the last old VM. The GPU resource-mapping lessons from the plex migration (Mapping.Use needed on both the user and the token, and the mapping must carry path, id, iommugroup, and subsystem-id) belong in the Runbook 12 update, not here.