DR Site and Backup Plan

Status: Planning. Target build a couple weeks out, after the current homelab work. This is the design we settled on, captured so we can execute against it and not re-derive it later. Think of it as the sequel to runbook 18, "the day after."

The problem

Two different safety nets, often confused, actually separate:

1. Backups. Somewhere durable and immutable that holds copies of the data I cannot recreate. Protects against deletion, corruption, and ransomware.
2. DR site. A small, always-available offsite control plane (a DR Forge plus an on-demand runner) that can rebuild the fleet from scratch, and that also bootstraps new servers when I stand up hardware in the apartment.

Backups answer "I lost the data." DR answers "I lost the thing that builds everything." I need both, and they are not the same purchase.

DR site

Shape: idle Forge, on-demand muscle (pilot light)

A tiny always-on box runs the DR Forge plus a minimal runner. That minimal runner exists to do one lightweight job: fire a workflow that hits the cloud provider's API, cloud-inits a runner-sized VM, registers it as a runner, runs the heavy bootc builds, then tears it down. Small box orchestrates, big ephemeral box builds, and I only pay for the muscle while it is actually flexing.

Leaning pilot light over warm standby because this is insurance and I am watching spend. The always-on cost is just a small box holding mirrors; the expensive part only exists on demand.

Mutual recovery

Colo rebuilds DR, DR rebuilds colo. Symmetric, so neither site is a single point of failure. Each needs the provisioning code, the secrets to run it, and a path to the target.

State vs compute (the key idea)

State, the git repos, is the crown jewel. Git is distributed, so every mirror is a full copy. Compute, the runner, is disposable and rebuilt on demand. Images are NOT synced: rebuild them from source at the DR end, since the Containerfiles are in the repo and the Fedora base images are public, so the whole registry reconstructs from scratch with no gigabytes of blobs to replicate.

Mirror direction

Point the DR Forge to pull from the neutral hub (GitHub/GitLab), not from the colo. The neutral hub is the thing that is always up, so pulling from it keeps DR current even when the colo is dark. Mirroring only off the colo would freeze DR's copy the moment the colo goes down, which is exactly when I need it fresh.

Secrets and bootstrap (the part that bites people)

The repos can be mirror-safe because the sensitive values live in Ansible Vault. But the vault password and the provider tokens are useless if they only live on a box in the colo. Encrypted-in-git is a locked safe; the combination has to live somewhere else.

Kept somewhere independent of both sites (password manager and/or an offline copy):

• The repo (already mirrored to the neutral hub).
• The Ansible Vault password.
• Provider credentials: cloud API token, Proxmox creds, Tailscale auth keys.
• This runbook, so the order of operations is written down before I need it.

Glue: Tailscale

DR Forge and runner join the tailnet. Every new server in the apartment joins the tailnet and pulls straight from the DR Forge. The same site that saves me from a colo failure is the bootstrap source for building out the home lab.

Boundary to stay honest about

DR can rebuild workloads and state onto a Proxmox host, but it cannot conjure the host itself. Bare-metal Proxmox at the colo still needs a machine to exist, which means remote hands or IPMI. The apartment side is easier because I will be standing right there. So "DR rebuilds the colo" assumes the hardware layer is handled separately.

Host

Hetzner Cloud is the front-runner: reliable, hourly billing, snapshots, one-click resize (which is what makes pilot light cheap), and US locations (Ashburn, Hillsboro) for decent latency from Kansas. Price it live from Hetzner's own calculator at build time. Note to self: do not trust secondhand pricing blogs, the last numbers I got fed were wrong.

Not SSD Nodes. For a control plane that has to be boringly reliable when everything else is on fire, an oversold budget VPS on a multi-year prepay is the wrong trade, especially since bootc image builds are I/O heavy and oversold I/O is where those hosts choke.

Backups

Separate from DR, and smaller than it feels once GitOps is doing its job.

What actually needs backing up

Most of the fleet is reproducible from the repo, so it does not need backing up, it needs rebuilding. The irreplaceable set is small:

• App databases: Radarr, Sonarr, Prowlarr, Bazarr (SQLite), Forgejo's DB.
• Ansible Vault and secrets.
• Personal or irreplaceable media (anything not re-acquirable).
• The Forgejo repos themselves (already mirrored, but a cold copy does not hurt).

What does NOT need offsite backup

• Reproducible configs and images (rebuild from git).
• Re-acquirable media, the bulk of the library. Back up selectively if at all; this is where per-TB cost would otherwise balloon.

Target and tool

Backblaze B2 (or similar object storage) as the primary target, driven by restic or kopia. Turn on Object Lock so backups are immutable, which is the piece that survives a compromised box with valid keys trying to wipe the offsite copy. If bulk media backup ever enters the picture, a Hetzner Storage Box is the better per-TB economics; B2 is the right home for the small, critical, immutable set.

Open decisions for build day

• Warm standby vs pilot light. Currently leaning pilot light.
• Neutral mirror hub: GitHub vs GitLab, or both.
• DR host region, and the build VM disk size (bootc builds want 80GB+, so size the ephemeral box, not the idle one).
• B2 vs Hetzner Storage Box if and when bulk media backup is in scope.
• Pull live Hetzner pricing and size the always-on box.

Relationship to the series

This is the planning doc. When it is built it likely becomes one or two posts after 18: the DR site stand-up, and the backup setup. Left the series index (00) alone for now since this is not a published post yet.