Runbook 17: Media Library Support

The arr box (Radarr, Sonarr, SABnzbd, and later Bazarr and Prowlarr) onto Fedora bootc, the last legacy pet retired. This one fought back harder than any other migration, and every fight was a small, specific bug worth remembering. Notes-to-self that double as a blog draft; verify the exact snippets against the itg-bootc and infra repos before publishing. The gotchas are the real value here, and there are a lot of them.

Key facts I keep needing: the host is itg-prd-arr, vmid 202 (the old AlmaLinux pet was 201, retired). Apps are Radarr (7878), Sonarr (8989), SABnzbd (8080), plus Bazarr (6767, subtitles) and Prowlarr (9696, indexer manager), all rootful podman Quadlets at PUID/PGID 1001 (the fleet worker), sharing a podman network named arr so they reach each other by container name. Two local data disks: scsi1 100G dataAppArr at /var/lib/arr (container configs + SQLite DBs), scsi2 1000G dataDownld at /var/mnt/downloads (SAB usenet scratch). Media stays on NFS, 10.10.10.2:/slow-media/media mounted read-write at /var/mnt/media, shared with Plex.

---

The design: local disks for state, NFS only for media

The old box kept everything on NFS, including the SQLite databases, which is a known way to corrupt them. So this rebuild deliberately splits storage by what the data actually is. The container configs and their SQLite DBs go on a local xfs disk (dataAppArr at /var/lib/arr), because SQLite hates network filesystems. The SAB download scratch goes on a second, larger local disk (dataDownld at /var/mnt/downloads), SSD-backed, because 4K usenet downloads are big and transient and there is no reason to push that churn over NFS. Only the finished media library stays on NFS, mounted read-write so the arr apps can rename imports into it. The tradeoff is that the local disks are now stateful and need a backup strategy instead of riding the NAS snapshots; that is an accepted cost for not running databases over NFS.

One consequence to know: because downloads (local xfs) and media (NFS) are different filesystems, Radarr/Sonarr cannot hardlink or atomic-move on import. "Use Hardlinks instead of Copy" is on, but it only works within one volume, so imports actually copy across the network to NFS. That is the price of the fast-local-scratch plus NFS-media split; the alternative (same filesystem) would give instant hardlinks but lose the local scratch.

PUID/PGID moved from the old box's 995 to the fleet-standard 1001. That meant re-chowning the entire media library 995 -> 1001 on itg (chown -R 1001:1001 /slow-media/media), a slow but non-destructive pass over 30TB. It does not affect Plex, which reads that tree read-only as a non-owner through the "other" bits.

The one thing that made this migration bearable: because the workloads are containers, the Quadlet maps host to container (/var/mnt/media:/media, /var/mnt/downloads:/downloads, /var/lib/arr/<app>:/config). The apps keep seeing /config, /media, and /downloads no matter where the host puts the bytes, so the SQLite databases never see a path change and there is no re-scan. This is the exact opposite of native Plex (Runbook 16), where real filesystem paths in the DB forced a re-path and rescan.

Why commission failed, and the pivot to a hand-run migrate

The plan was to use commission.yml recreate:true (build, bake, provision, destroying old 201 first). It reported success and did nothing useful: the box I logged into was still the old AlmaLinux pet, no bootc. The recap told the story. The retire step's "Delete the VM" came back changed=0, and every provision step after it ran idempotently against the still-running old VM, so "Import the bootc qcow2 as scsi0" skipped because scsi0 already existed on the old box, and it never installed the new image.

Root cause: retire.yml was a bare community.proxmox.proxmox_kvm ... state: absent, and that module will not remove a running VM. It no-ops. The play never checked, so a silent failure passed as success. The fix (now in the repo) probes for the VM, force-stops it, deletes it, and then verifies it is actually gone with a failed_when on a post-delete existence check, so a silent no-op can never again slip through.

But commission.yml runs on the Forgejo runner, which checks out the pushed repo, so the local retire fix would not reach it without a commit-and-push cycle at midnight. The faster, safer move was to hand-write a migrate-arr.yml and run it from the Mac, where it uses the local (fixed) files and never touches the runner path.

migrate-arr.yml, in the proven shape

Modeled on the archived migrate-web.yml and migrate-plex.yml, with one arr-specific simplification. The web and plex migrations had a Phase A that staged /config off the old box because it lived on local disk. Arr's config lives on NFS, so there is nothing to stage; we copy it straight from the NAS to the new local disk at the end. The phases:

• A: gracefully stop the old containers so the NFS SQLite DBs quiesce before anything else. Do not skip this and rely on the VM stop, which is not guaranteed graceful.
• B: stop, do not delete, the old VM, keeping it as a rollback. This is why the new box gets a different vmid (202) than the old (201): building alongside a stopped old box avoids the delete-before-build path that broke commission entirely.
• C: import_playbook: site.yml builds the fresh VM 202, imports the qcow2 as scsi0, attaches both data disks, sets the hostname. No workload role runs; arr's services are baked-in Quadlets.
• D: mount the old NFS appdata read-only, rsync -aH --delete each app's config into /var/lib/arr/<app>, chown 1001, start the services.

It needs a temporary host_vars/itg-prd-arr-old.yml (old vmid 201 + its ansible_host) and an arr_old inventory group, both deleted after old 201 is destroyed.

The bugs, in the order they bit

This is the part to actually remember.

XFS labels max out at 12 characters. dataDownloads is 13. mkfs.xfs rejected the -L value and then printed its usage ending in "<devicename> is required", which reads exactly like a missing-device error and sent me hunting the wrong thing. dataAppArr is 10, which is precisely why the appdata disk formatted and the downloads disk did not. This cascaded: no dataDownld label meant var-mnt-downloads.mount never found its device and dependency-failed, and all three apps Requires= that mount, so nothing started. Label is now dataDownld in both the init service and the mount unit.

The SELinux boolean was named wrong. arr-selinux-nfs.service ran setsebool container_use_nfs on, which errored with "Invalid boolean" because that name does not exist on this Fedora policy. getsebool -a | grep nfs showed the real one is virt_use_nfs, and it was already on. Fixed the unit to the correct name so it stops failing at boot.

Fresh xfs disks mount root-owned. After mkfs.xfs, the filesystem root is root:root, and the mount lands on top of the tmpfiles-created (worker-owned) mountpoint, masking that ownership. So /var/mnt/downloads came up root-owned and SAB (1001) could not create /downloads/complete. The migrate happened to dodge this for appdata because Phase D chowned /var/lib/arr after mounting, but a day-0 build would hit it on both disks. Fixed with arr-appdirs.service, a oneshot that runs after the mounts and before the apps, creating and chowning the app dirs to 1001.

The downloads volume had no SELinux relabel flag, and that is MAC not DAC. Even after the ownership was correct (worker:worker 0755), SAB still got PermissionError on /downloads/complete. The /config volumes carry :Z so podman relabels them to container_file_t; media is NFS and covered by virt_use_nfs; but the downloads volume had neither, so the local xfs kept a non-container label and the container was denied by SELinux despite correct ownership. Fix: add :z (lowercase, shared) to the downloads volume in the Quadlets. It must be :z and not :Z: SAB writes and Radarr/Sonarr read the same dir to import, so they need the shared label; :Z gives each container a private MCS category and would break the sharing. Live recovery was chcon -R -t container_file_t /var/mnt/downloads. The same trap was lurking on the book box's transmission.container (its downloads volume had no flag either); fixed there too.

The shared network, and the 403 it surfaced

Radarr and Sonarr originally addressed SAB by the box IP, and that broke every time the box's DHCP lease moved (it moved twice in two days). The fix is a Quadlet arr.network with Network=arr.network on every arr container, so they resolve each other by container name over podman's aardvark DNS. The download client in Radarr/Sonarr then points at sabnzbd:8080, IP-independent forever. The web UIs keep their PublishPort so SWAG (on the web box) still reaches them.

Changing the host to sabnzbd surfaced a second thing: the test came back 403 Forbidden, not a connection error. That is SABnzbd's hostname verification. SAB checks the incoming Host header against host_whitelist (Config, Special) and rejects anything it does not know. It had downloads.itguyeric.com, sabnzbd.itguyeric.com but not the bare sabnzbd the containers now use. Adding sabnzbd to host_whitelist fixed it. Lesson: name-based addressing means every name used to reach SAB has to be in its whitelist, and a 403 (not a refused connection) is the tell that you reached SAB but it rejected the hostname.

One tradeoff worth recording: putting arr on a user network turns on aardvark DNS for arr, the same component whose reboot race broke forge (Runbook 18). It is acceptable here because arr's containers are leaf services with Restart=always and no inter-container Requires, so a boot hiccup self-heals rather than stranding them. The no-DNS alternative was a DHCP reservation and addressing by a now-stable IP.

Completing the stack: Bazarr and Prowlarr

Once the core three were solid, Bazarr (subtitles) and Prowlarr (indexer manager) round out the stack. Both are leaf Quadlets, PUID/PGID 1001, Restart=always, on arr.network.

Bazarr needs its config on the appdata disk and the media NFS mount read-write (it writes subtitle files next to the media), but no downloads volume. Because it touches NFS media, it goes in arr-selinux-nfs.service's Before= list (so virt_use_nfs is set before it starts) and in arr-appdirs.service (so its config dir is owned 1001). Port 6767.

Prowlarr needs only its config dir, no media, no downloads. Its whole value is centralizing indexers in one place and syncing them down to Radarr and Sonarr, which retires the per-app indexer config the old box carried in duplicate. Port 9696.

The wiring, all by container name so it is IP-independent: in Prowlarr, add Radarr (http://radarr:7878) and Sonarr (http://sonarr:8989) with their API keys under Apps; in Bazarr, add Radarr and Sonarr the same way, then pick subtitle providers and languages. SWAG proxy-confs for bazarr and prowlarr are still hand-placed on the web box (the same not-in-git gap as the others); since both are admin tools, they are good candidates for the LAN-only allowlist from Runbook 15.

Verify, do not declare

The lesson that cost the most trust: a service showing active (running) proves the container is up, nothing more. On first boot the Quadlets auto-start against an empty disk and initialize fresh configs, so "it's running" can mean "it's running your real data" or "it built a blank one." What actually proved the migration:

• Populated databases: radarr.db at 68MB and sonarr.db at 128MB, appdata at 1.4G / 305M / 32M. A fresh install is under a couple MB with no library to scan.
• SAB carrying both news servers, and Radarr/Sonarr already having a download client to re-point (a fresh install would have neither).
• Media readable and writable by 1001 (sudo -u worker touch /var/mnt/media/.arrtest).
• Library scans coming back with files present, not a wall of missing.
• The real end-to-end: Radarr detected a missing file, searched, grabbed, and handed it to SAB to download. That single event proves library, indexers, download client, and the import path all at once.

The daylight follow-ups

The running box got to working through live hacks: a manual mkfs, an /etc override of the downloads mount, a chcon, and manual chowns. The repo fixes (dataDownld label, virt_use_nfs, arr-appdirs.service, :z on downloads, the arr.network) are what make a clean build reproduce all of that on its own. So the honest final test is a from-scratch commission of arr once everything is pushed and baked, to prove the day-0 path the migration never fully exercised. Remaining: remove the /etc downloads-mount override once the box is on the re-baked image so it stops shadowing the baked unit, wire up Bazarr and Prowlarr by container name, and get the SWAG proxy-confs (arr, bazarr, prowlarr, and the rest) into git.

The permission model on the shared media tree

Months later, going to delete a corrupt file from Plex surfaced the fact that nothing had ever decided how permissions on the media share were supposed to work.

/var/mnt/media was 0777. Every directory, world-writable, on an NFS export reachable by the whole 10.10.10.0/24 with no_root_squash. Nobody set that deliberately; it is the fossil of some earlier chmod -R 777 to make a permission problem go away, which is the classic move that trades a real fix for a permanent hole. Any host on the LAN could have deleted the library.

The model it should have had, and now does:

owner  worker (1001)   everything that writes media runs as worker
group  worker (1001)   Plex is a member, so it can read and unlink
dirs   2775            setgid
files  0664

Three parts have to be true together, and each is useless alone.

Plex has to be in the group. One m plex worker line in the plex image's sysusers file. Plex runs as its own uid because it is a native RPM, not a Quadlet, so it was never going to match 1001.

New files have to land group-writable. UMask=0002 on every Quadlet that writes media, plus Environment=UMASK=002 on the linuxserver images, because s6 sets its own umask from that variable and ignores the systemd directive. Without this the tree drifts straight back: the enforcement run that fixed 1,044 directories and 7,259 files found 56 fresh files two minutes later, all created at 0644 by containers still running the old umask.

Deleting needs write on the directory, not the file. This is the part that makes the symptom confusing. Plex could see a file it demonstrably could not remove, because unlink is a modification of the parent directory.

The setgid bit is honest belt-and-braces rather than load-bearing here. With every writer running 1001:1001, new entries already come out worker-owned; setgid only earns its keep the day something runs with a different primary group and worker only as a supplementary one.

Enforcement lives in the media-perms Ansible role rather than a script, and it audits before it acts, so a converge on an already-correct tree reports 0, 0, 0 and skips every fix task. That is the difference between a remediation and a policy: one is a fact about the day you ran it.

The one that hid all of it: --manage-gids

With the image right and the tree right, Plex still could not delete. Everything checked out: new digest booted, mount rw, /etc/group showing worker:x:1001:plex, the running process carrying Groups: 972 1001. Correct permissions, correct membership, EACCES anyway.

The measurement that broke it open was three setpriv calls against the same directory, varying only how the group arrived:

setpriv --reuid 1001 --regid 1001 --groups 1001 touch .p1   # owner            -> OK
setpriv --reuid 985  --regid 1001 --groups 1001 touch .p2   # worker PRIMARY   -> OK
setpriv --reuid 985  --regid 985  --groups 1001 touch .p3   # worker SUPPLEMENTARY -> denied

Same uid, same directory, same group. The only variable is primary versus supplementary, and only supplementary fails. That is rpc.mountd --manage-gids: the server throws away the group list in the RPC credential and looks the user's groups up in its own /etc/passwd and /etc/group instead. On the NFS server, uid 985 is nobody in particular and certainly not in gid 1001, so the request fell through to the other bits — r-x on a 2775 directory — and could not unlink.

Two things made this hard to find. It is on by default in Debian's nfs-kernel-server, and the NFS server here is the Proxmox host. And it does not appear in ps: modern nfs-utils reads /etc/nfs.conf, so ps -ef | grep rpc.mountd shows a bare process while manage-gids=y sits in a config file. Checking the command line and concluding it was off cost an hour.

nfsconf --get mountd manage-gids      # the check that actually answers
nfsconf --set mountd manage-gids n
systemctl restart nfs-kernel-server && exportfs -f

--manage-gids exists to dodge the 16-group ceiling in AUTH_SYS. Plex belongs to two groups.

Gotchas, in the order they bit

• chmod -R 777 is never the fix, and it outlives whoever typed it. It survived a full migration onto bootc and was still there months later on a share exported to the entire LAN.
• Deleting a file needs write on its directory. A user who can read a file and not remove it is not a contradiction.
• Environment=UMASK and systemd's UMask= are not interchangeable. linuxserver's s6 init sets its own umask from the env var; images with a plain entrypoint only see the unit directive. Set both.
• Do not hardcode a GID you do not control. The plex image asked for g plex 985, but Fedora's own basic.conf declares g empower - - - with an auto-allocated id, and it had already taken 985. The group creation lost, while u plex 985:985 pinned the user's primary group to that number anyway, leaving plex primary-grouped into an unrelated service's group. Reference the group by name and let it allocate.
• --manage-gids makes supplementary groups silently stop existing. The server's answer is only as good as the server's copy of /etc/group.
• Check config files, not just ps. A daemon reading /etc/nfs.conf shows no arguments at all.
• /etc/passwd and /etc/group are machine-local and 3-way merged. Fixing the image does not fix a deployed host, exactly like /etc/shadow and the root password in Runbook 18.
• NFS servers deserve config management too. Both halves of this bug lived on the one box Ansible reaches only through the Proxmox API, and both were invisible from every host that is managed.