Part 12: Runner Creation — Provisioning a Runner From Nothing

The goal was a Forgejo Actions runner that goes from "does not exist" to "green and idle" with no clicks anywhere, and it works. One command, run from a box on the LAN:

ansible-playbook site.yml -e target=itg-prd-run02

That builds the VM on Proxmox out of the nightly-built image, boots it, takes its hostname, registers it with Forgejo, writes its config, and starts the daemon. Deleted the VM and the runner, ran that one line, and it came back green with its labels. Every step was proven by hand before it got codified, and the gotchas below are the reason it took a marathon instead of an afternoon. (That one-liner is now also wrapped in a push-button provision workflow; see "Still to codify," which is mostly no longer to codify.)

The shape

Three roles, composed by two playbooks. Keeping the provider-specific part separate from the shared part is deliberate, so a DigitalOcean front-door can slot in later without touching the rest.

• provision-proxmox (phase 1): everything that talks to the Proxmox API, ending by pinning the VM's discovered IP. This is the only Proxmox-aware piece.
• hostname (phase 2): wait for SSH, set the static hostname, reboot if it changed. Provider-agnostic.
• runner-register (phase 3): derive the secret, register offline on forge, render the runner config, start it. Runners only.

provision.yml runs phase 1 then phase 2. site.yml imports provision.yml and adds phase 3, gated with when: "'runners' in group_names" so it's a no-op on non-runner boxes. site.yml has since grown sibling workload phases the same way (e.g. a game-config phase gated on when: "'game' in group_names" that renders the game hosts' machine-local secrets from vault), each a no-op off its group. The seam between provider-specific and shared is a single fact, run_ip, set at the end of phase 1.

The two credentials

The Proxmox side is a scoped API token. A dedicated ansible@pve user, in the PVE realm not PAM, so it has no shell and no OS presence, carrying a custom Provisioner role granted only on the paths it touches. Never root, never a password.

The Forgejo side needs no Forgejo credential at all, which surprised me. Offline registration runs as the git user inside the forge container, so there's no personal access token and no REST API. What it does need is a way in: the control node reaches forge over the same baked ansible SSH access every box has (key plus passwordless sudo, both from the golden image), and runs the CLI with podman exec. I burned an hour chasing an admin PAT and the registration-token endpoint before realizing the blessed path doesn't go through the API.

Scoping the Proxmox token

pveum user add ansible@pve --comment "Ansible provisioning service account"
pveum role add Provisioner --privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.GuestAgent.Audit Datastore.AllocateSpace Datastore.Audit Sys.Audit SDN.Use"
pveum acl modify /vms                    --user ansible@pve --role Provisioner
pveum acl modify /nodes/itg              --user ansible@pve --role Provisioner
pveum acl modify /storage/fast-vmstore   --user ansible@pve --role Provisioner
pveum acl modify /storage/bootc          --user ansible@pve --role Provisioner
pveum acl modify /sdn/zones/localnetwork --user ansible@pve --role Provisioner
pveum user token add ansible@pve provisioner --privsep 0

I did not front-load that privilege list. I started smaller and let the API's 403s tell me what was missing. SDN.Use only surfaced when a create failed naming it, because Proxmox 8+ moved bridge assignment under SDN, so attaching a NIC needs SDN.Use on the bridge's zone. VM.GuestAgent.Audit is needed to read the guest agent's network info, which is how phase 1 discovers the IP. --privsep 0 lets the token inherit the user's ACL, fine because this user does nothing but provision. Grab the secret when it prints, it shows once.

Two ACL facts worth keeping: the paths are Proxmox's own namespace (/vms, /storage/{id}, /sdn/...), never filesystem paths, and Proxmox does not validate the target exists, so a typo'd path silently creates a permission pointing at nothing.

Phase 1: build the VM

Auth for all three Proxmox modules goes in a module_defaults block at the top of the role, which is the role-friendly replacement for a play-level YAML anchor (anchors are file-local and don't reach into a role). Then, in order:

1. Create the shell (community.proxmox.proxmox_kvm, state: present) with scsihw: virtio-scsi-single, bios: ovmf, machine: q35, agent: enabled=1, an efidisk0 with format: raw, a NIC on the bridge, and onboot: true. The virtio-scsi-single is not optional: OVMF has no LSI driver, so LSI makes the VM unbootable. onboot: true matters more than it looks, learned the hard way when a power blip took the host down and a runner didn't come back on its own.
2. Read the current config, then import the bib qcow2 as scsi0 only if it's absent, so a rerun doesn't re-import. import_from: bootc:import/<image>.qcow2, storage: fast-vmstore, which converts the qcow2 into a raw lvmthin volume.
3. Create any local data disks the host asks for. provision-proxmox loops a data_disks list (slot plus size in GB) and creates each missing one, guarded so a rerun doesn't re-add. The list defaults to empty in the role, so a plain box gets nothing; group_vars/runners.yml sets it for the runner fleet as one scsi1 at 100G, and a workload host sets its own in host_vars (game's is a 100G scsi1 too). That blank disk is what the image formats and mounts on first boot as the box's local work disk. On a runner it becomes dataRun at /var/lib/dataRun, with /var/lib/containers bound off it, so podman's graph and the osbuild build scratch land on real local disk instead of the tiny /run tmpfs.
4. Set boot order to scsi0 (guarded so it's idempotent), start it.
5. Poll the guest agent's network-get-interfaces until a 10.10.10. address appears, pin it as run_ip, and then set ansible_host to it with a second set_fact.

That last split matters. Setting ansible_host as a play-level var poisons the phase-3 delegation: a play var isn't host-specific, so when the register task hops to forge, the play's ansible_host clobbers forge's real address and Ansible falls back to the bare inventory name, which a roaming control node can't resolve, and you get UNREACHABLE. Setting it as a per-host set_fact keeps it on the runner where it belongs, and the forge delegate uses its own inventory address.

Phase 2: own the hostname

wait_for_connection, then ansible.builtin.hostname, then reboot only when hostname_set is changed. The image bakes a default hostname service, so a fresh box comes up as the base name until phase 2 renames it and reboots so DHCP and DNS register the real one. Stripping that baked service so Ansible owns naming from first boot is still on the list, but the reboot-after-rename works today.

Phase 3: register, and the derived-secret model

This is the crown jewel and the biggest change from the by-hand version. There is no generate-secret step anymore. The secret is derived, so it never has to be stored per runner.

One master seed lives in the vault:

vault_runner_master_seed: "<openssl rand -hex 32, once, for the whole fleet>"

Each runner's secret is a pure function of its name and that seed:

runner_secret: "{{ (vm_name ~ vault_runner_master_seed) | hash('sha1') }}"

A SHA1 hex digest is exactly 40 characters, which is exactly what Forgejo wants for a runner secret. So itg-prd-run02 plus the seed always hashes to the same secret, forever, and I never generate or write one down. It's a key-derivation function: one master key in the safe, every door's key computed on demand from the master plus the door's name. The textbook-correct construction is HMAC, and the hash filter is a plain digest of the concatenation, but for a homelab runner secret whose security rests entirely on the seed staying in the vault, it's fine.

The register itself, delegated to forge:

podman exec --user git forgejo forgejo forgejo-cli actions register \
  --name <vm_name> --keep-labels --secret <runner_secret>

Three things I paid for in blood:

• --user git. Forgejo refuses to run as root, and podman exec without --user runs as root, so register bombs with a root-check error. Exec as git.
• --secret <value>, not --secret-stdin. The stdin flag has broken argument parsing: it swallows the next flag as its value, or errors when it's last. I chased a mis-named "runner" and a non-zero exit before giving up on it. Plain --secret with no_log: true on the task keeps it out of Ansible's logs; the only exposure is the argv on forge for the second it runs, visible to root, who already owns the box.
• --keep-labels. Re-registering an existing runner without it wipes the labels and exits non-zero. That non-zero was what aborted the whole role. With --keep-labels, re-registration is a clean idempotent update.

Register returns the uuid on stdout, which we capture with a regex. Interesting detail: the uuid is derived from the secret too (it's the first sixteen characters of the 40-char secret, encoded as ASCII bytes into UUID form), so it's as stable as the secret. We still read it back from the command rather than recompute it, so we're not depending on Forgejo's derivation staying the same.

Then render the config the daemon reads, which Ansible now owns outright:

server:
  connections:
    forge:
      url: "{{ forgejo_url }}"
      uuid: "{{ runner_uuid }}"
      token: "{{ runner_secret }}"
log:
  level: info
runner:
  capacity: 2
  labels:
    - native:host
    - podman:docker://node:20-bookworm
container:
  docker_host: "-"
  options: "--volume /etc/website-deploy:/mnt/deploy_ssh --volume /var/lib/hugo-cache:/mnt/hugo_cache"
  valid_volumes:
    - /etc/website-deploy
    - /var/lib/hugo-cache

Enable and start forgejo-runner, restart on config change via a handler, and it comes up green. Labels come from the runner declaring them off this config on connect, which is why register doesn't need --labels and why --keep-labels is safe.

(Note the /etc/website-deploy mount is now vestigial. The website deploy key moved out of a machine-local file and into a Forgejo CI secret written to a temp path at job time, so a rebuilt runner no longer needs the key pre-placed on disk. The mount is harmless and can come out of the config next time this is touched.)

The dead end I want to save future-me from: I once tried inventing a uuid with uuidgen and using a registration token as the connection token. The daemon crash-loops on unauthenticated: unregistered runner. The server.connections block is the output of registration, not a way to perform it.

The image side: config strip, and the build/bake split

Two pieces of the pipeline live in the bootc repo, not the Ansible repo.

The run image no longer bakes /etc/forgejo-runner/config.yaml. Ansible is the sole author of that file, because the secret has to live in it and a secret can't be baked. /etc is machine-local and three-way merged, so an existing runner keeps its rendered config across image upgrades; only a freshly provisioned box boots without one, and the register role writes it seconds later. Stripping the baked config is what makes Ansible the single author and kills the config drift I found (the baked file had pointed the hugo cache at the wrong path for weeks).

The image build and the qcow2 bake are now two separate workflows. build.yml builds and pushes the container image for every entry in images.json, on each push and nightly, and stops there, so iterating on an image is a two-minute turnaround instead of waiting on osbuild every time. A separate bake.yml builds the bootable qcow2 with bootc-image-builder (--type qcow2 --rootfs xfs --local), staging on the NFS-exported bootc folder and doing a final mv into import/ so a provision never catches a half-written file. It runs nightly and on manual dispatch with an optional single-image input, so before provisioning a box you bake just that one image and wait on one disk instead of the whole matrix.

Pulling the qcow2 off the push path is deliberate. Existing VMs never touch it; they roll forward by pulling the pushed image with bootc upgrade on the nightly timer. The qcow2 only matters at VM birth, when provisioning imports it, so paying the osbuild tax on every push bought nothing. And if you ever provision off a stale qcow2, one bootc upgrade --apply catches the box up to the registry anyway. Building a disk for forge itself is still real disaster insurance: if forge dies, you can re-provision it from a qcow2 on the Proxmox host without needing the registry forge hosts. One bake gotcha worth remembering: osbuild's build tree overflows the 3G /run tmpfs, so the bake binds a directory on the runner's local dataRun disk to /run/osbuild (which is the whole reason runners now get that data disk in phase 1).

Why rebuilds need no cleanup

Because the secret is derived and the uuid follows the secret, tearing down a runner and rebuilding it with the same name and seed re-asserts the exact same runner record. Destroy itg-prd-run02 and rebuild it ten times, you get one entry, always. No pruning, no duplicates. The only time you orphan a record is retiring a runner for good (drop it from inventory and never rebuild) or rotating the master seed (every secret changes at once, which is the breach-recovery lever, and the one time you'd want a bulk prune). A prune play for those two cases is still to build.

Inventory and vars

• Groups are flat top-level keys (forge, runners, dev, prd); all is implicit. No all: children: wrapper, that's only for nesting a group inside a group.
• ansible_user: ansible lives in group_vars/all/vars.yml, not per-host. Every box is reached as the baked service account, so it's a fleet fact. Leaving it out of a host is how a run silently tries to connect as your laptop's username and gets permission-denied.
• forge is in the inventory with an explicit ansible_host, so the phase-3 delegation can reach it. A delegate that isn't in inventory has no address and no user.
• data_disks is a role default of empty, overridden in group_vars/runners.yml for the runner fleet and in host_vars for one-off workload hosts. vm_sizes carries the T-shirt sizes; XL (12 cores / 32G) got added for the heavier workload hosts.

Run it from the LAN

Run this from a box on the LAN, the bastion or the CI runner, not a roaming laptop over Tailscale. The laptop can't resolve internal short names, which forces IP workarounds that then collide with the delegation, and every one of those was a separate hour. On the LAN, names resolve and delegation is boring. The CI runner is itself on the LAN, which is exactly why running this as a Forgejo job (now the provision workflow) makes the resolution problem disappear for free.

Gotchas, collected

• Module disk and net parameters are dicts, not the qm CLI string. Typed dict means structured keys.
• An efidisk on lvmthin needs format: raw. Block storage takes no file formats; same reason the boot disk is imported, not dropped in.
• Attaching a NIC needs SDN.Use on the bridge's zone on Proxmox 8+, and reading the guest IP needs VM.GuestAgent.Audit.
• A directory storage scans a fixed subdirectory per content type. Import images live in import/, not the storage root. A non-root token can import from a storage volume but not an arbitrary host path, which is the whole reason for the dedicated bootc storage.
• host_key_checking = False for a disposable fleet, or rotated host keys on rebuilt boxes block the automation.
• proxmoxer and requests must be installed in the same Python that runs ansible-playbook (phase 1 is connection: local). Fedora has no python3-proxmoxer package, so it's a dnf install python3-pip then pip install --break-system-packages proxmoxer, plus ansible-galaxy collection install community.proxmox. The run image bakes exactly this, since the runner is the control node when provisioning runs as a CI job.
• Set ansible_host as a per-host fact, never a play var, or it bleeds into delegate_to and breaks the forge hop.
• --user git on the exec, --secret not --secret-stdin, --keep-labels always. All three above.
• register and create-runner-file are deprecated; offline forgejo-cli actions register is the durable path.
• Forgejo workflow_dispatch inputs need an explicit type: string, or the UI rejects the dispatch with "Invalid input type."

Still to codify

The Forgejo-job wrapper is done. site.yml now runs from a workflow_dispatch job (provision.yml): pick the target host in the forge UI and go, no hand-typed ansible-playbook. The runner that executes it carries the vault password and SSH key as CI secrets and has proxmoxer plus the community.proxmox collection baked into its image. That was the same setup a fresh control node needs, so the runner image is the reproducible control node now.

One thing still left: a small prune play for the retire and seed-rotation cases. Known quantity, just not built.

---

Update (2026-08-03): pipeline consolidation, dynamic runner, commission, app-gating

Several things above are superseded or extended. The runner-creation flow itself (the three phases, the derived secret, the token scoping) is unchanged. What changed is the pipeline that builds the runner image and the way brand-new nodes get stood up. Full pipeline design lives in Runbook 14; the runner-relevant deltas:

build.yml + bake.yml are now one pipeline.yaml. The "build/bake split" section above described two separate workflows. They're merged into a single pipeline.yaml in the bootc repo. It resolves which images to build, always builds base, then a per-image matrix job builds AND bakes each image in one leg with fail-fast: false, so one image failing never blocks the others and base is the only hard gate. Push builds only the changed image and does not bake; the nightly (4 AM Central) builds and bakes everything; a manual dispatch does one image or all; and a plain push can force a bake for the changed image by putting [bake] in the commit message. The old separate-workflow behavior described above is history.

The runner version is no longer hardcoded. The run image used to pin RUNNER_VERSION=12.13.0. It now resolves the latest release from the Forgejo API at build time (/releases/latest, which is latest-stable, not a bleeding-edge tag), parsed with python3, with ARG RUNNER_VERSION still there to pin an exact version if a release misbehaves. This is how the fleet moved itself to 13.0.0.

ansible.posix is baked into the runner. The galaxy line is now ansible-galaxy collection install community.proxmox ansible.posix, because web-config uses ansible.posix.authorized_key. Collections are baked into the image, not installed per job, so adding one means rebuilding the run image and bootc upgradeing the runners one at a time (never both at once, or you lose all runners mid-build).

commission.yml is the day-0 one-stop. For a brand-new node, commission.yml in the infra repo does build, bake, and provision in a single dispatch: it cross-checks-out the bootc repo to build the image, bakes the qcow2, then runs site.yml. An optional recreate: true deletes the VM first via retire.yml for clean-slate bring-up. provision stays the tool for re-running against an existing host; commission is for birthing one. First real use is itg-prd-arr.

FORGE_TOKEN is a user-level secret now. With no organization on the personal account, FORGE_TOKEN lives as a user-level Actions secret, inherited by every repo. It needs write:package (registry push/pull) AND read:repository (commission's cross-repo checkout of bootc). A package-only token authenticates the registry but fails the checkout.

Workload phases gate on app, not group membership. site.yml's per-workload phases moved from when: "'game' in group_names" to when: app == 'game' (and app == 'web', etc.), and the single-host game/tail inventory groups were dropped since the app host_var already identifies them uniquely. runners stays a real group because it has group_vars and two members. Only genuinely multi-host things stay groups now.

The /etc/website-deploy mount is fully retired. The website deploy key is a Forgejo CI secret written at job time, and the web box installs rsync-rrsync for the restricted deploy target (the deploy writes into a named subdir because the newer python rrsync rejects a ./ destination). Nothing pre-placed on the runner.