Part 10: Auto-Deploying the Hugo Website From a Runner

Unlike the image pipeline in Part 6, this one does not build a bootc image. It builds a website. I push to the website repo, a Forgejo runner spins up a throwaway node:20-bookworm container, builds the Hugo site inside it, and rsyncs the output to the SWAG web root. The workflow lives at .forgejo/workflows/deploy.yml in the website repo and publishes itguyeric.com.

This one fought me harder than anything else in the series. The pipeline itself is simple. The four gotchas below are the whole reason this post exists, because run01 and every future runner will walk straight into them again.

The shape

The job runs in a container (runs-on: podman), so the runner has to hand that container two things from the host: the deploy SSH key and a persistent Hugo cache. Both come in as bind mounts declared in the runner's own config.yaml, not in the workflow:

container:
  docker_host: "-"
  valid_volumes:
    - /etc/website-deploy
    - /var/lib/hugo-cache
  options: "--volume /etc/website-deploy:/mnt/deploy_ssh --volume /var/lib/hugo-cache:/mnt/hugo_cache"

The workflow checks out with the Congo submodule, installs Hugo Extended, symlinks the cache into place, builds, then rsyncs:

      - name: Wire persistent Hugo resource cache
        run: |
          mkdir -p /mnt/hugo_cache
          ln -sfn /mnt/hugo_cache resources
      - name: Build site
        run: hugo --minify
      - name: Stage the deploy key
        run: install -m 600 /mnt/deploy_ssh/id_website /tmp/deploy_key
      - name: Deploy to web root
        run: |
          rsync -a --delete \
            -e "ssh -i /tmp/deploy_key -o StrictHostKeyChecking=accept-new" \
            public/ ansible@itg-prd-web:./

On itg-prd-web the incoming key is pinned to write-only, one directory, no shell, via a forced command in ansible's authorized_keys:

command="/usr/bin/rrsync -wo /var/lib/containers/swag/www",restrict ssh-ed25519 AAAA... website-deploy@itg-prd-run

The Hugo cache is content-addressed processed images (resources/_gen). The symlink points resources at the mounted cache so those variants persist on disk across builds. First build is cold, every build after is warm. A green run is about 70 seconds, and the build step itself drops to ~3s once the cache is hot.

The four gotchas that cost me a night

1. valid_volumes silently rejects any suffix, including :ro. Forgejo's runner will not let a job bind-mount an arbitrary host path unless it is in container.valid_volumes, and the match is against the volume spec. A plain source:dest bind matches a plain source-path entry. The moment you add :ro, it no longer matches and the mount is dropped with [path] is not a valid volume, will be ignored. Same story for a :z suffix. Fix: list plain source paths in valid_volumes, use plain source:dest binds, and handle read-only or SELinux another way (see gotcha 3). This is a known gap, forgejo/runner issue #79.

2. container.options is one space-delimited string, and it is unforgiving. When I stripped the :ro from a mount, I deleted the space with it and got .../mnt/deploy_ssh--volume /var/lib/hugo-cache:.... The runner parsed that as a single mount with the destination /mnt/deploy_ssh--volume, and the second --volume never registered as a flag at all. Two symptoms at once: the deploy key was "not found" at /mnt/deploy_ssh/id_website, and the cache mount silently did not exist so every build ran cold. One missing space. If a mount looks configured but the container cannot see it, print the live options line first, before touching anything else.

3. SELinux blocks the container writing the cache, even as root. The job container runs as root, and root bypasses Unix permissions but not SELinux. Reads of the mounted cache worked, but the first attempt to write a new processed image got permission denied. The clean answer is normally :z on the mount, which relabels it, but gotcha 1 means I cannot use the suffix. So I relabel the directory by hand instead: chcon -R -t container_file_t /var/lib/hugo-cache. New files inherit the label, so it is a one-time thing on a running box. chcon is machine-local and does not survive a full relabel, so the durable version is an fcontext rule baked into the image, or a context= mount option when the cache moves to the NFS depot.

4. The deploy user's uid must equal the web-content owner's uid. rsync came in as ansible (uid 1000) into a web root owned by SWAG's PUID (995), and every write, delete, and mkdir failed with permission denied, made louder by --delete trying to clear the old files. Chowning the web root to ansible does not stick, because SWAG re-chowns its whole /config to PUID on every container restart. The fix is to stop having two uids: set SWAG's PUID/PGID to the deploy user's (1000). Then SWAG owns the content as ansible, the deploy writes as ansible, and it stays that way across restarts. nginx keeps serving fine because it is only reading its own files.

Two more things that were not the problem

• The cold-build timeout was the box, not the cache. An early build hung for nearly three hours on a 2 CPU / 4 GB runner and died with context deadline exceeded. Image processing is memory-hungry, and 4 GB swap-thrashed on ~213 image variants. On an 8 GB runner the same cold build finishes in a couple of minutes. The cache is an optimization for speed, not the fix for the timeout. Runner capacity is 2, so size RAM for two concurrent cold builds or they will OOM each other.
• Jobs land on whichever runner matches the label. I burned time "fixing" a runner while my jobs were quietly landing on a second, stale runner (the old itg-prd-run) that still had the broken config and could not be fixed without a rebuild. The Set up job log names the runner that accepted the task by UUID. Check it. When in doubt, shut down the runners you are not actively fixing so routing is deterministic.

Lessons

• Forgejo's valid_volumes matches the whole spec, so no :ro, no :z, no suffixes. Plain source paths only.
• container.options is a single string. Mind the spaces between --volume args.
• Container-into-host bind mounts on Fedora need an SELinux relabel to be writable, container_file_t, done by hand since the :z shortcut is off the table.
• A push deploy needs the sender uid and the receiver's file-owner uid to be the same user. With SWAG, that means aligning its PUID, because it re-chowns on every restart.
• Size the runner for the cold path and for capacity, then let the cache make the happy path fast.
• Read the runner UUID in Set up job so you know which box actually ran the job.

Still parked

The runner still points at a local /var/lib/hugo-cache and carries a hand-run chcon. When run02 is rebuilt after run01, it comes up pointed at the shared depot cache (/var/mnt/depot/hugo-cache, already in the repo config), which needs the durable SELinux context and a hugo --minify --gc build to prune the migration-era seed before it seeds the shared cache. All of that rides along with the rebuild, not before it.