ITG Homelab Runbooks

How a single-node Fedora bootc homelab in a COLO became a full GitOps pipeline: I edit a file in VSCode, commit, and Forgejo Actions runners build and roll out new bootc images on a schedule. Written as notes-to-self that double as blog drafts.

Started as the bootc build. It now covers the whole estate, so it lives in the infra wiki rather than a repo-specific one.

The cast

• itg — the Proxmox host, 10.10.10.2, also my NFS server.
• OPNsense (itg-net-opn) — DNS (Unbound), DHCP, and firewall for 10.10.10.0/24; public egress 66.85.74.134.
• itg-prd-web — SWAG reverse proxy at 10.10.10.83, fronts everything on 443. Patches at 02:30.
• forge.itguyeric.com — Forgejo at itg-prd-forge (10.10.10.24): git, container registry, and Actions. The fleet SPOF; patches in a slot of its own at 02:00, ahead of everyone else.
• itg-prd-run01 / itg-prd-run02 — Forgejo Actions runners, themselves bootc images.
• itg-dev-bootc — the dev box, takes nightly updates with the herd.
• itg-prd-book — the book and media-reader host, VM 206, 10.10.10.226: Transmission, LazyLibrarian, Calibre-Web-Automated, AudioBookShelf, and Komga.
• itg-prd-plex — Plex Media Server, native RPM with an RTX 2070 passed through from Proxmox.
• itg-prd-arr — the arr stack on bootc (Radarr, Sonarr, SABnzbd, Bazarr, Prowlarr, Recyclarr, Mylar3, Pinchflat, MeTube), vmid 202; the last legacy pet retired.
• itg-prd-game — the game-server box (Foundry, ddb-proxy, Valheim, Palworld, Starbound, ARK).
• itg-prd-tail — the Tailscale subnet router.
• Coulson — 10.10.10.162, old build box and the retired HTTP registry, now the persistent bastion tmux host.
• itg-base — the golden base image everything else is built FROM.
• ansible@pve — the scoped Proxmox API service account that provisions VMs.

The build, in order

1. Image Mode and the itg-base Golden Image — image mode explained, and what goes in the base.
2. Storage on bootc: NFS and Data Disks — the /mnt symlink trap, systemd mount units, first-boot data disks, nosharecache.
3. Hostnames and DNS Registration on bootc — why static hostname beats a kernel arg, and stopping NetworkManager from clobbering it.
4. Container Registry on Forgejo — moving off the HTTP registry to the HTTPS Forgejo one, auth stores, and the bootc pull secret.
5. Forgejo Actions Runner as a bootc Image — the runner as an image, and the registration that actually works on modern Forgejo.
6. The CI/CD Pipeline — base build, children fan-out, lint gating, push.
7. Nightly Auto-Deploy with bootc — the built-in updater and the real cadence (build 1, forge 2, web 2:30, fleet 3), plus the bug that hid inside it: hosts pulled at 3 while the build did not start until 4, so the fleet sat a day behind with every timer green.
8. First Real Workload: itg-prd-book — Quadlets, data disk vs NFS, SELinux labels, and the Proxmox VM.
9. VSCode as the GitOps Cockpit — settings, extensions, and commit-triggered push.
10. Hugo Website Auto-Deploy — building the Hugo site in a throwaway runner container and rsyncing it to SWAG, plus the four mount/SELinux/uid gotchas that cost me a night.
11. Cloudflare Migration, Book Restore, Kobo and MAM — the day I migrated DNS to Cloudflare, restored the book data, and wired up Kobo and MAM.
12. Runner Creation and Ansible Provisioning — provisioning a runner from nothing: a scoped Proxmox token builds the VM, Forgejo's offline-registration CLI mints the runner, no clicks anywhere. Also the GPU resource-mapping lessons.
13. Tailscale Subnet Router on bootc — advertising 10.10.10.0/24 into the tailnet, with subnet-route SNAT on by default.
14. SWAG and the Website — the web/SWAG migration to bootc, and the pipeline overhaul into one pipeline.yml.
15. Split-DNS and LAN-Only Services — split-DNS so LAN-only services resolve to SWAG internally; the nginx allowlist, and the Unbound redirect-zone trap a wildcard override springs.
16. Plex Media Server — native Plex on bootc with NVIDIA GPU passthrough, the %pre systemd trap, and the /media canonical-path trap.
17. Media Library Support — the arr box, the last legacy VM: local disks for appdata and the SAB scratch, media stays on NFS.
18. The Day After — post-migration papercuts. Reboot resilience, then the four that were wrong from day one and stayed green for months.
19. Self-Hosted Media — the full ebook/audiobook/comics stack, plus MAM as the books workhorse.
20. The Hypervisor Joins the Fleet - bringing itg into the inventory without pretending it is an image: Proxmox ships without sudo, Debian's admin group is sudo and wants a password, the subscription nag and the assert that keeps the patch honest, and the two Ansible warnings I had been reading past for months (a pty from -tt turning the module's trailing newline into junk, and why pinning the interpreter globally breaks provisioning).
21. The Matrix Homeserver - a worker-mode Synapse built the bootc way: nine containers, the :z vs :Z SELinux trap that cost the most, postgres collation you can only set at initdb, an nginx router that caches dead upstreams until you give it a resolver, federation delegation plus the split-DNS override so LAN clients stop hairpinning, and the five gates a Forgejo build crosses before it lands in a Matrix room. Bridges to be appended.

Reference

• Infra Principles — the rules everything above is supposed to follow.
• Post-Migration Checklist
• DR Site and Backup Plan

The through-line

Three facts about bootc shape everything: /usr is immutable and comes from the image, /etc is a 3-way merge that preserves your local edits, and /var is machine-local and never touched by the image. Almost every design decision in these posts falls out of those three rules.