Part 7: Nightly Auto-Deploy with bootc

Once the pipeline builds images, something has to roll them out. bootc ships that something for free.

The built-in updater

bootc-fetch-apply-updates.timer is a systemd timer that comes with bootc. When it fires, it checks the host's image ref for a newer digest, and if there is one, pulls it, stages it, and reboots into it. That is the entire deploy mechanism, no Ansible push, no orchestrator. The host pulls its own future.

I set the schedule with a timer drop-in baked into the image so I control exactly when it runs:

[Timer]
OnCalendar=
OnCalendar=*-*-* 03:00:00
Persistent=true
RandomizedDelaySec=1h

The empty OnCalendar= first clears the vendor default before setting mine, otherwise you end up with both schedules active. RandomizedDelaySec spreads the fleet out so eight hosts do not all hit one registry VM in the same minute.

The cadence

The night is staggered so each stage feeds the next:

• midnight — backups (the slot is reserved; the DR work is still on the list).
• 1 AM — the pipeline rebuilds itg-base, then the child images fan out via needs: and push to the registry.
• 2 AM — itg-prd-forge, alone. It is Forgejo, the registry, and the CI hub, so it gets a window where nothing else in the fleet is mid-pull. No jitter: jitter exists to spread a fleet across one registry, and forge is the registry.
• 2:30 AM — itg-prd-web. It goes ahead of the fleet because every other host resolves forge.itguyeric.com through this box's SWAG (split DNS, Runbook 15). If web reboots inside the fleet's pull window, it kills everyone else's in-flight pull.
• 3 AM — the rest of the fleet pulls, spread across an hour by the jitter.

The ordering is the point: build before deploy, the hub before the proxy, the proxy before the things that depend on it, and everything after the backup so a bad night is always recoverable.

Note the dependency knot that shape untangles. forge pulls its own image through web's SWAG, and web pulls its image from forge. Neither can upgrade without the other, and each reboots. Thirty minutes of separation plus the retry drop-in below is what keeps that from being a deadlock: worst case one of them slips fifteen minutes, not a night.

The bug that hid inside a green cadence

For months this was written down as "build at 2, deploy at 3" and it was wrong in the one way that never shows up in a status page. The workflow's cron actually said:

- cron: 'TZ=America/Chicago 0 4 * * *'

Build at 4 AM. Hosts pulled at 3. So every host woke an hour before the build started, found no new digest, correctly did nothing, and exited zero. systemctl list-timers was green. bootc status showed a real image. The fleet was permanently one day behind and every single signal said it was fine.

The lesson is not "check your cron." It is that a puller and a pusher on separate schedules will silently agree to do nothing, and success is the default output of that disagreement. If the deploy step cannot tell you which build it deployed, it cannot tell you it is behind.

Timers do not retry

The second half of the same problem: a systemd timer does not retry a service that failed. It waits for the next OnCalendar. So one transient failure, the registry restarting, web still rebooting, a build job that had not pushed yet, left a host on the old image for a full 24 hours with nothing louder than a journal entry.

The fix is a service drop-in, not a timer one:

[Unit]
StartLimitIntervalSec=2h
StartLimitBurst=4

[Service]
Restart=on-failure
RestartSec=15min

Four attempts fifteen minutes apart rides out a reboot on the far end without hammering the registry. Note the split: StartLimit* goes in [Unit], Restart* in [Service]. Put them in the same section and systemd ignores half of it.

Bake is a separate workflow, not the second half of the build

The pipeline used to build and bake a qcow2 for every image on every scheduled run. That is what turned a 7-minute push run into 47 minutes to 1h48m, which is what let the build overrun the pull window in the first place.

The first fix was to keep one workflow and gate the bake step on which cron fired, using github.event.schedule. That worked, and it was still wrong. Coupling them meant every build carried the possibility of an hour of osbuild, and on any day where you are touching three parts of the fleet at once, that possibility is the thing that stops you iterating.

So bake.yml is its own workflow again. pipeline.yml builds, lints and pushes on a nightly cron and on every push. bake.yml runs weekly on Sunday night, or on demand with an image selector when you are about to provision something new.

Worth being honest that this partly reverses Runbook 14, where build.yml and bake.yml were consolidated into pipeline.yml. That consolidation was right at the time and its reason has expired. During the migration, new VMs were being cut constantly and image and template had to stay in lockstep. The fleet is built now. Templates are only read when provisioning a new VM; every existing host upgrades over the registry and never touches them. A decision can be correct and later stop being correct without either version having been a mistake.

Two details the split improved rather than just moved:

The bake job no longer checks out the repo, and pulls the image from the registry before baking. bootc-image-builder --local reads from local container storage, so previously it baked whatever the build job had just put there. Now it bakes exactly what the fleet is running, because that is what came out of the registry.

And mv -f only runs after a successful bake, so a failed run leaves the previous good template in place instead of a truncated one.

Enable it per host, not in the base

The auto-update timer is deliberately NOT enabled in itg-base, only the schedule drop-in is. Deploy policy is a per-host decision, so each host image enables the timer (or does not) rather than forcing the whole fleet onto one cadence. itg-dev-bootc takes updates with the herd, so if a build is broken something face-plants at 3 AM and I find out before I am using anything.

Unmasking forge: fix the collision, not the calendar

itg-prd-forge used to mask the timer outright, after a nightly reboot left it dead for seven hours (Runbook 18, part 1). Masking rather than disabling, because masking survives an image rebuild and a stray systemctl enable cannot undo it.

Taking it off the schedule was the right call at the time and the wrong diagnosis. The problem was never "forge reboots." It was "forge reboots on top of everything else," and the fix for that is a slot, not an exemption. What made it safe to lift, in order:

• forgejo now Wants= postgres instead of Requires=, so a slow database at boot no longer permanently cancels forgejo's start job.
• postgres has TimeoutStartSec=300, so the slow first start after a reboot completes instead of being killed.
• the retry drop-in above means a host that finds forge mid-reboot tries again rather than skipping a full day, which removes the blast radius that made forge's reboot scary in the first place.

The bootc trap on the way back out: systemctl mask writes a symlink into /etc, and /etc is a 3-way merge. Deleting the mask line from the Containerfile changes what the image ships; it is not a promise about what is already on disk. So the Containerfile unmasks explicitly rather than staying silent, and the host still gets checked by hand after the first upgrade:

RUN systemctl unmask bootc-fetch-apply-updates.timer && \
    systemctl enable bootc-fetch-apply-updates.timer

systemctl status bootc-fetch-apply-updates.timer   # not "masked"
systemctl list-timers bootc-fetch-apply-updates.timer

Overriding the base schedule in a child image

web needs 02:30 and forge needs 02:00 instead of the base's 03:00, which means a second drop-in in the same directory. systemd applies drop-ins in filename order, last one wins, and that is a trap here:

bootc-updates-timer-web.conf     <- sorts FIRST
bootc-updates-timer.conf

- is 0x2D and . is 0x2E, so the obvious name sorts before the base file and the base silently wins. The overrides are named zz-bootc-updates-timer-web.conf and zz-bootc-updates-timer-forge.conf for exactly that reason. It looks like superstition until you diff systemctl cat against what you thought you shipped.

Checking it

systemctl list-timers shows when the next run fires. systemctl cat bootc-fetch-apply-updates.timer shows the drop-ins as merged, which is the only way to catch the ordering trap above. bootc status (needs root) shows the currently booted image and whether a newer one is staged.

The check that actually matters is comparing the digest a host booted against the digest the pipeline pushed last night. Anything less than that will happily report green while the fleet drifts.

Lessons

• bootc's own timer is the deployer. You do not need to build one.
• Clear the vendor OnCalendar= before setting your own, or you get two schedules.
• A puller scheduled before its pusher exits zero forever. Verify by digest, not by timer state.
• Timers do not retry failed services. If you want retries, that is a Restart= drop-in on the service.
• Drop-ins merge in filename order. Name overrides so they sort last.
• When one box is too important to reboot with the others, give it a slot of its own rather than taking it off the schedule. Exempting it means it stops getting patched, which is its own outage on a longer fuse.
• Unmask explicitly. A mask symlink lives in /etc, and on bootc that means removing the line from the image is not the same as removing it from the host.
• Order the night as backup, build, hub, proxy, fleet.
• Bake templates on the schedule that provisioning needs, not the schedule that upgrades need, and in a workflow of their own. A slow step welded to a fast one makes the fast one slow.