Runbook 20: The Hypervisor Joins the Fleet

Every host in this estate is an image except one. itg is Debian, installed by the Proxmox ISO, and until today it was reachable only as root with a key that lived on coulson. The box that hosts every other box was the only box with no declared state anywhere.

That is not an aesthetic complaint. Runbook 17 has the bill: two separate NFS settings on itg, the export options and manage-gids in /etc/nfs.conf, quietly broke supplementary groups across the whole fleet. Plex could not delete a file and the reason lived on a machine that nothing described, so there was nothing to diff and nothing to read. The fix took a day of bisecting with setpriv because the config was invisible.

So itg goes into the inventory. Not all of it. Enough.

You cannot Ansible your way onto a box you have no account on

The first run has to come in as root, because neither account exists yet:

ansible-playbook site.yml -e target=itg \
  -e ansible_user=root \
  -e ansible_ssh_private_key_file=~/.ssh/id_bastion

Every run after that is ordinary. Phase 1 skips itself because itg has no vmid, which is the guard that keeps the provisioning playbook from trying to build a VM out of the hypervisor.

Three Fedora assumptions that are all false on Debian

The role was written with the images in mind, and every assumption it inherited from them turned out to be wrong here. Each failed differently, and only the first one was legible.

sudo is not installed. A stock Proxmox VE install is root-only. There is no sudo binary, no visudo, and no sudo group for a user to be a member of. This is the actual root cause, and it presents as three separate failures stacked on top of each other: the sudoers validate dies first, then creating the human account fails with "Group sudo does not exist", and even if you forced both through there would be nothing to run. Install the package first and the other two evaporate.

visudo lives in /usr/sbin, and Ansible runs modules under a non-login shell. PATH is the sh default, /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games, and /usr/sbin is not on it even as root. So this works on Fedora and fails with ENOENT on Debian:

    validate: "visudo -cf %s"        # wrong
    validate: "/usr/sbin/visudo -cf %s"   # right

I fixed this one before fixing the missing package, which is why I saw ENOENT twice in a row with two different paths. The second ENOENT was the useful one: a binary that is missing from PATH and from its canonical location is not missing, it is absent.

The admin group is sudo, not wheel, and Debian's stock rule wants a password. These accounts are created password_lock: true, so there is no password to give. The human account would have been SSH-able and completely unable to sudo. The images write %wheel ALL=(ALL) NOPASSWD: ALL; the Debian half has to be written explicitly:

- name: Give the sudo group passwordless sudo
  ansible.builtin.copy:
    dest: /etc/sudoers.d/sudo-nopasswd
    content: "%sudo ALL=(ALL) NOPASSWD: ALL\n"
    mode: "0440"
    validate: "/usr/sbin/visudo -cf %s"

That failure would not have surfaced during the run. It would have surfaced the first time I actually needed root on the hypervisor, which is the worst possible moment to discover it.

The proof the whole thing worked is one line, and it is worth running:

ssh itguyeric@10.10.10.2 sudo id
uid=0(root) gid=0(root) groups=0(root)

The subscription nag, and the assert that is the real content

PVE 9.2.10, proxmox-widget-toolkit 5.2.7. The 5.x series reformatted proxmoxlib.js into prettier-style wrapping, so the condition guarding the "No valid subscription" dialog is now spread across five lines:

                    if (
                        res === null ||
                        res === undefined ||
                        !res ||
                        res.data.status.toLowerCase() !== 'active'
                    ) {

Every sed one-liner on the internet targets the old single-line form. Read the file on your own host before you write the pattern. \s+ spans the newlines and indentation:

pve_nag_regexp: >-
  res === null \|\|\s+res === undefined \|\|\s+!res \|\|\s+res\.data\.status\.toLowerCase\(\) !== 'active'

Anchor on res === null. The same toLowerCase() !== 'active' test appears again around line 21500, where it sets subscriptionActive for the UI state. That one is load-bearing and matching it breaks the interface.

Now the part that matters more than the patch. ansible.builtin.replace reports ok when its regexp matches nothing. It does not warn, it does not fail, it just says ok. So the next time upstream rewraps that block, the dialog comes back and every run afterwards stays green while doing precisely nothing. That is the exact failure shape as all four bugs in Runbook 18. So the task re-reads the file and asserts:

- name: Confirm the dialog is actually patched out
  ansible.builtin.assert:
    that:
      - (pve_widget_lib_raw.content | b64decode) is search(pve_nag_patched_regexp)
    fail_msg: >-
      The replace matched nothing and the dialog is still live. Upstream changed
      the shape of the block. Do not widen the regex to make this pass.

No APT post-invoke hook, and no script planted on the host. A widget-toolkit update restores the dialog and the next site.yml run removes it again. The desired state lives in the repo. itg does not converge on its own, and the config should not pretend it does.

What I deliberately left out

The manage-gids=y fix in /etc/nfs.conf is still a hand edit made over SSH. It is tempting to fold it into the role now that the role exists, and I proposed exactly that. The answer was no, and the answer was right: do not IaC a host that was not built that way. itg is a pet until the hardware refresh, and a half- managed snowflake is worse than an honest one, because it invites you to trust a site.yml run that only covers the parts someone got around to writing.

The risk is real and should be written down rather than solved badly: nfs-kernel-server can rewrite /etc/nfs.conf on package upgrade. The single change that made Plex deletes work is one apt upgrade away from reverting, and when it does, the symptom will be a permission error on a host that has nothing to do with NFS. If that happens, start at Runbook 17.

Two warnings I had been reading past for months

Both of these had been printing on every fleet run since roughly forever. Both were treated as noise. Neither was.

"Module invocation had junk after the JSON data:"

Note the colon with nothing after it. That was the entire answer and it took me five wrong hypotheses to see it. I chased motd, the sshd banner, PAM, the Python 3.14 interpreter, and interpreter discovery. Every one of them was ruled out by a command run on the host, not by reasoning.

What actually cracked it was reading ansible/module_utils/json_utils.py instead of guessing what the message meant. _filter_non_json_lines silently discards anything before the first line starting with {, and warns only about lines after the last line ending in } -- and it prints them. An empty message therefore means the trailing junk contains no visible characters.

The chain, once you know it is a stray carriage return:

• ansible.cfg had no pipelining setting, so it defaulted to off.
• With no in_data, the ssh connection plugin adds -tt to any sudoable command. You can see it in -vvv: -tt appears on the module invocation line and on none of the others.
• -tt allocates a pty. A pty's line discipline rewrites the module's trailing \n as \r\n.
• The leftover \r splits as an extra line after the closing brace, and it prints as nothing.

One line fixes it:

[ssh_connection]
pipelining = True

Which also removes four SSH round trips per task, because the module is fed over stdin instead of being staged: no mkdir, no sftp upload, no chmod, no rm -rf. A fleet-wide site.yml got noticeably faster as a side effect of chasing a cosmetic warning. Pipelining needs sudoers without requiretty, which neither Fedora nor Debian has set for years, and which fails loudly rather than quietly if it is ever set.

"using the discovered Python interpreter at /usr/bin/python3.14"

interpreter_python = auto_silent in [defaults]. Discovery picking the newest python is correct here: the image decides which interpreters exist, and a version bump arrives through a rebuild like everything else.

The obvious-looking alternative is a trap. Putting ansible_python_interpreter in group_vars/all would also apply to the phase 1 localhost delegations, which would point the Proxmox API tasks at the Mac's system python, which has no proxmoxer, and break provisioning. That is the same shape as the ansible_user bug in Runbook 18: a global that reads as harmless right up until it reaches the one host it was never meant to describe.

Why there are no VM snapshots

Related decision, made the same day, because it turns on the same question of what is actually described somewhere.

A Proxmox snapshot of a bootc VM is mostly redundant. /usr comes from the image and the image is in git, so a bad deployment is fixed by bootc rollback, which is instant and needs nothing from Proxmox. Snapshotting the root disk to protect the OS is duplicating a job bootc already does better.

What a snapshot would protect is /var, the machine-local state the image never touches: Forgejo's postgres and repo data, the arr databases, Plex's library and watch history, AudioBookShelf and Komga progress. None of that is in git. But a snapshot is a poor way to protect it, because snapshots live on the same storage as the VM. If fast-vmstore dies, every snapshot on it dies at the same instant. A data disk is not a backup either; it protects against the root disk filling up and nothing else.

There is a specific reason not to automate them here. fast-vmstore is LVM-thin, and a snapshot pins every block it references. Runbook 18 already records that thin-pool Data% only ratchets up without discard. A forgotten snapshot would eat the pool faster than the leak that was already fixed, and the failure mode is the whole node going read-only, not just the one VM. Proxmox schedules backups but has no scheduler for snapshots, which means any automation here would be ours to write, and the deletion half is the half that gets skipped.

So: no snapshot automation, and no snapshot capability to build. Taking one by hand before something scary, a Forgejo major version bump or a postgres migration, is still the right move. That is a right-click, not a project.

The real gap this exposed is offsite backup, which is where the effort belongs when it belongs anywhere. Worth recording the shape of it now: the irreplaceable data here is small. Forgejo repos and database, the arr and Plex databases, the vault, the appdata. The media library is enormous and almost entirely re-acquirable. Those are two different problems with two different costs and they should not be solved as one.

Gotchas, in the order they bit

• Proxmox VE ships without sudo. Not misconfigured, not on a different path. Absent. Everything downstream of it fails first and louder.
• Ansible runs modules under a non-login shell, so /usr/sbin is not on PATH. Any validate: or command: that reaches for an sbin binary needs an absolute path, even when running as root.
• Debian's admin group is sudo and its stock rule prompts for a password. A password_lock: true account in that group cannot sudo at all.
• ansible.builtin.replace reports ok when it matches nothing. Any patch against a file you do not own needs an assert behind it or it will rot silently.
• A warning that prints nothing after its colon is telling you the thing it found has no visible characters. Read the source that emits the message before theorising about what the message means.
• -tt in the -vvv output means a pty, and a pty means CRLF. If exactly one of the EXEC lines has it, that is the one misbehaving.
• A Proxmox snapshot lives on the same storage as the VM it protects. It is an undo button, not a backup, and on LVM-thin a forgotten one takes the whole node down with the pool rather than just its own VM.
• Pipelining is off by default and costs four round trips per task. It is worth turning on for the speed alone; the pty fix is a bonus.

The through-line

Runbook 18 ended on the idea that the day after keeps being the day after, and that the bugs which survive are the ones nothing was checking. This is the same lesson from the other direction: itg was not broken, it was undescribed, and undescribed is where bugs go to live for months at a time.

The counterweight is knowing when to stop. Bringing the hypervisor into the inventory was right. Bringing all of it in would have been wrong, because itg was never built to be described and pretending otherwise buys a false green. Two accounts, sudo, and a cosmetic patch is the honest amount. The NFS config stays a hand edit with a note attached until the hardware refresh, when the whole box gets rebuilt as something that deserves the treatment.