Part 13: The Subnet Router — Tailscale on bootc

The goal was to retire the old Rocky pet that ran my Tailscale subnet router and replace it with a bootc host that provisions from nothing, advertises 10.10.10.0/24, and never needs me to babysit its auth. It works, and the part that fought me was not Tailscale itself but the auth model, specifically one default that would have quietly deleted the node on its next reboot.

Same pattern as the runner and game hosts: a bootc image FROM itg-base, a small ansible role for the machine-local bit that can't be baked (the auth), and the whole thing dispatched through the provision workflow. No data disk, tail is stateless enough that its only real state, /var/lib/tailscale, is tiny and rides the machine-local /var.

The shape

• Image tail/: tailscale from its own repo, tailscaled enabled at build, an IP-forwarding sysctl drop-in, the static-hostname oneshot. No podman-auto-update.timer, there are no containers here.
• Role tail-config: one task, tailscale up, gated in site.yml on when: "'tail' in group_names".
• host_vars: vmid 211, vm_size: S (a subnet router is a featherweight), no data_disks.

A subnet router has to forward packets, so the image bakes /usr/lib/sysctl.d/99-tailscale-forward.conf:

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Without that, the node joins the tailnet and advertises the route but no traffic actually crosses it, which is a confusing way to fail because everything looks connected.

The auth model, and why OAuth over a plain key

This is the whole story of this host. The naive move is a reusable auth key from the console. It works, but a Tailscale auth key expires in 90 days maximum, and that expiry is a disaster-recovery landmine: the running node is fine forever (its node key is a separate thing, and the tag disables its expiry), but if the box implodes and I re-provision on day 91, tailscale up with an expired key fails and the router comes up unauthenticated. The vault key gates DR.

The fix is a Tailscale OAuth client instead of a fixed key. The OAuth client secret does not expire, so the vault value stays good indefinitely and re-provisioning always works. That's the real win. Setup:

• Trust credentials, new credential, custom scopes, expand Keys and tick Auth Keys → Write only. Least privilege, that's the client's entire job.
• Ticking that scope forces you to attach a tag, because OAuth-minted keys must be tagged. Use tag:infra.
• The tskey-client-... secret it generates goes in the vault as vault_tailscale_authkey.

Two policy edits make the tag pull its weight, in Access controls, Policies (the HuJSON file):

"tagOwners": {
  "tag:infra": ["autogroup:admin"]
},
"autoApprovers": {
  "routes": {
    "10.10.10.0/24": ["tag:infra"]
  }
}

tagOwners lets the tag exist and be assigned. autoApprovers means a tag:infra node advertising 10.10.10.0/24 gets its route approved automatically, so I never click Approve. Tagging the node also disables its node-key expiry on its own, which is the "Expiry disabled" badge you want to see.

The gotcha that would have stranded me: ephemeral by default

Here is the one that cost real time. When you hand an OAuth client secret to tailscale up as the auth key, the resulting node is ephemeral by default. Ephemeral nodes are deleted from the tailnet the moment they go offline. That is exactly wrong for infrastructure: the first nightly bootc-update reboot would delete the router and drop the route, and once the old fallback was gone I'd have locked myself out of my own LAN remotely.

You cannot toggle ephemerality off after the fact, it's set at registration. The fix is a URL parameter appended to the key:

--authkey={{ vault_tailscale_authkey }}?ephemeral=false&preauthorized=true

ephemeral=false makes it persistent; preauthorized=true skips device approval. To un-ephemeral a node that already registered wrong, you delete it in the console and re-register, re-running up won't flip it.

So the whole role, tail-config/tasks/main.yml:

---
- name: Bring up Tailscale as a subnet router
  ansible.builtin.command: >
    tailscale up
    --authkey={{ vault_tailscale_authkey }}?ephemeral=false&preauthorized=true
    --advertise-routes=10.10.10.0/24
    --advertise-tags=tag:infra
    --accept-dns=false
    --hostname=itg-prd-tail
  changed_when: false
  no_log: true

--accept-dns=false keeps a subnet router from pulling MagicDNS and looping its own resolution. The & is safe unquoted because ansible's command module runs argv directly, no shell, so it's a literal part of one argument, not a background operator. tailscale up is idempotent and the identity persists in /var/lib/tailscale, so on every boot after the first it comes up already authenticated; the key is really only exercised at provision time.

Cutover

The new node comes up named itg-prd-tail-1, not itg-prd-tail, because the old box still holds that name in the tailnet and MagicDNS names must be unique. That's expected, you rename it at the end.

1. Provision the new box (provision workflow, target itg-prd-tail).
2. Verify in the admin console: the node shows tag:infra, Expiry disabled, Subnets, and crucially no Ephemeral badge. If Ephemeral is there, stop and fix it before going further, because the next step removes your fallback.
3. Confirm routing actually works through the new node before killing the old one. Both advertise the same subnet while both are up, so Tailscale could route through either and a test now can pass via the old one and fool you. Test from a tailnet client, and note that the client may need a Tailscale off/on cycle to re-pull the netmap when the routing node changes (my Mac did). Proxmox's web UI loading over the tunnel was my proof.
4. Shut down the old VM.
5. Delete the old itg-prd-tail node in the console (frees the name), then rename itg-prd-tail-1 to itg-prd-tail.

Leaving the old VM shut-down-but-not-deleted for a bit is cheap rollback insurance until you're sure.

Gotchas, collected

• OAuth-client-secret auth is ephemeral by default. Append ?ephemeral=false&preauthorized=true or the router deletes itself on reboot. This is the big one.
• OAuth client secrets don't expire (auth keys cap at 90 days), which is the only reason DR is truly hands-off. Use the client, not a key.
• A subnet router needs IP forwarding via sysctl, or traffic silently doesn't cross it even though the node looks healthy.
• --accept-dns=false on a subnet router, or you invite MagicDNS resolution loops.
• The node registers as <name>-1 when the old node still owns the name. Delete the old node, then rename.
• Clients cache the netmap; when the routing node changes, a Tailscale toggle on the client forces the re-pull. Not a server problem, don't chase it as one.
• tag:infra does double duty: disables node-key expiry, and with autoApprovers auto-approves the advertised route. Both need the policy entries above.
• OAuth client scope is Auth Keys write only, tagged tag:infra. Least privilege; it exists solely to mint tagged keys.

Reusable beyond tail

tag:infra and the OAuth pattern aren't tail-specific. Any future infra node that should join the tailnet non-interactively (an exit node, a second router at a different datacenter) reuses the same client, the same tag, and the same role shape with a different --advertise-routes. That was the point of tagging it infra rather than subnet-router.