Runbook 21: The Matrix Homeserver

I wanted an avenue for alerts before there was anything to alert on, plus the bridges I used to run, plus my own federated identity back. So Matrix went in ahead of monitoring on the board, on purpose: build the place the notifications land first, wire the notifications to it second.

The identity is a returning one. @eric:itguyeric.com lived on a DigitalOcean Synapse years and several homelab iterations ago, under this same server_name. That box is long gone and its database with it, so this is a fresh homeserver reclaiming an old name, not a migration. Everything below is a bootc/mtrx/ image built from itg-base like the rest of the fleet, plus an mtrx-config Ansible role for the half that holds secrets.

This runbook covers the homeserver standing up and Forgejo build reporting landing in a room. The bridges (Discord, Signal, Slack, and hookshot doing more than webhooks) come in a later pass once federation has settled.

Workers, not a monolith

Synapse is Python, so the main process is one CPU core no matter how big the VM is. Workers are the only way to use more than one, and that is the entire reason they exist; everything else is a consequence. The split on itg-prd-mtrx:

• main does everything not delegated below, plus the replication listener.
• generic-1 takes /sync (the single heaviest thing a homeserver does) and the heavy client reads, plus inbound federation.
• fedsender-1 sends outbound federation. No HTTP listener at all: it works off the replication stream. This is the worker that matters most on a federating server, because sending events to remote homeservers is slow and blocking and must not compete with your own client requests.
• appservice handles bridge traffic. Isolated on purpose: a wedged bridge must not be able to stall client requests, and bridges are the whole point of this box.
• media-1 does uploads, downloads, thumbnails, and the job that expires remote media.
• redis is the replication bus between all of them. Persistence off; it holds nothing worth keeping across a restart.
• nginx routes requests to the right process.

There is no lock-in here. Monolith to workers is a config change, not a data migration, so the IaC argument actually favours starting split and correct rather than "simple now, migrate later."

The router's catch-all is the load-bearing line

The nginx routing table ends in location / to the main process. Every route above it is an optimisation. That means a Synapse release that adds endpoints degrades to "not offloaded yet" instead of 404ing quietly, which is the Runbook 18 failure mode. The offload list is a deliberately conservative subset of what upstream allows. Grow it from evidence (main is pegged, here is what it is serving), not from the docs.

The gotchas, roughly in the order they bit

This was a long day of self-inflicted wounds. Most of them share a shape: config that looks right, a green result, and the wrong thing happening underneath.

postgres collation has to be C, and Environment= splits on whitespace

Synapse refuses to start against a database that was not created with C collation, and it re-checks on every boot: Database has incorrect collation of 'en_US.utf8'. It is only settable at initdb time, so getting it wrong means dumping and recreating the cluster later.

The quadlet sets it, but the first attempt was:

Environment=POSTGRES_INITDB_ARGS=--encoding=UTF8 --lc-collate=C --lc-ctype=C

systemd splits Environment= on whitespace, so that set POSTGRES_INITDB_ARGS=--encoding=UTF8 and then tried to parse --lc-collate=C as a second assignment. initdb ran with the default locale. The value has to be quoted as a whole:

Environment="POSTGRES_INITDB_ARGS=--encoding=UTF8 --lc-collate=C --lc-ctype=C"

Verify it, do not trust it: psql -c "SELECT datcollate FROM pg_database" must show C, and note that in PostgreSQL 15+ lc_collate is per-database, not a server GUC, so SHOW lc_collate errors.

:z vs :Z, three times in one day

This is the one that cost the most, and I hit the same three-character mistake three separate ways.

:Z on a podman volume writes a PRIVATE SELinux MCS category pair, unique to one container, stamped at container start. :z writes a SHARED label. The rule is: any directory more than one thing touches must be :z.

• I mounted the Synapse /data dir with :Z into six things (main, four workers, and the Ansible generate step). Each start relabelled it and locked the other five out. It presented as a uid problem, because root inside a container holds CAP_DAC_OVERRIDE and punched through while uid 1001 got EACCES, which sent me chasing ownership and first-boot ordering that were fine all along. There was no AVC to find either, because by the time I looked a later :Z run had relabelled the directory back into agreement with itself. The tell was the same command being denied once and allowed minutes later with nothing but another container run in between.
• The SWAG config dir is :Z and always worked by accident of ordering: ship the confs, restart SWAG, and the restart relabels on the way up. My new nginx -t gate (below) validated BEFORE the restart, so it was the first thing to read that dir in its un-relabelled state, and nginx got Permission denied on a file that looked perfectly fine from the host. Fixed with chcon --reference in the role; the honest long-term fix is :z on that volume too.
• hookshot's /data gets its registration and passkey written by Ansible after the container may have started, so it is :z from the start.

Heuristic learned the hard way: if a directory is written by config management AND read by a container, or touched by more than one container, it is :z.

Log to journald, not to a rotating file under /data

Synapse's own generate writes a log config with a rotating FILE handler under /data. The main process crash-looped on Unable to configure handler 'file', 394 restarts before I looked. Even working it would be wrong: six processes sharing /data would be six of them rotating one file, and nothing else on this fleet logs anywhere but journald. Ship the same stdout log config the workers use and point all six at it.

The router caches the upstream IP forever

The homeserver was healthy and the router still returned 502. nginx resolves a hostname to an IP once, at config load, and caches it forever. Every Synapse container gets a new podman IP when it is recreated (a restart, a bootc upgrade, the morning's crash-loop), so a static upstream synapse:8008 block kept dialing an address that no longer existed. The fix is a resolver plus variables in proxy_pass, which forces per-request resolution via aardvark, and it is exactly the pattern the SWAG proxy-confs already use. mtrx.network pins the subnet so aardvark's address stays constant across rebuilds, and $request_uri is appended so a regex location passes the raw URI through unchanged (Matrix room and event IDs carry URL-encoded characters).

$HOME is unset in the runner

The deploy job runs with set -u and died on HOME: unbound variable building the SSH key path from $HOME/.ssh/.... Worth more than the one fix: the same $HOME expression in provision.yml and commission.yml had NO set -u, so those had been resolving the key path to /.ssh/... (a path that does not exist) and silently falling back to ssh defaults for who knows how long. set -u earned its keep. Keys now live next to .ans_pass under infra/ansible, referenced by $PWD.

The config role, and why the delegations fail open

The secret half lives in mtrx-config: homeserver.yaml, the DB password, and the signing key. The signing key is generated once with a creates: guard and never again; it is the homeserver's identity to the federation, and losing it means remote servers reject everything this server says with no fix from the database.

The important property: every worker delegation in homeserver.yaml FAILS OPEN. A typo in notify_appservices_from_worker or federation_sender_instances does not error; Synapse quietly keeps doing that job on the main process and every unit stays green. So the role ends by asking the router for /_matrix/client/versions and checking that /.well-known/matrix/server is actually served, and says so plainly when it is not. Verify, do not declare.

Two settings worth calling out. url_preview_enabled is off, and it is a security decision, not a taste one: with previews on a federated user can post a link to an internal address and use the homeserver as an SSRF proxy into 10.10.10.0/24, where Forgejo and Proxmox live. And enable_registration is off, because this server federates, so /register is reachable from the whole internet and open registration gets you a spam relay within days. Accounts are made by hand with the shared secret.

Federation delegation, and the split-DNS trap

Delegation is .well-known/matrix/server and client served on the apex by SWAG, pointing federation at matrix.itguyeric.com:443. No port 8448 anywhere, one TLS path. matrix.subdomain.conf is the ONE conf in that directory that deliberately does NOT include lan-only.conf: federation requires remote servers to reach it, and adding the allowlist breaks federation while the client API keeps working from inside the house, which is a maddening thing to debug. matrix.subfolder.conf serves the two .well-known paths as literal returns, not static files, because the Hugo deploy rsyncs over the web root and would wipe them.

The federation tester (federationtester.matrix.org) is the authoritative check for "can other servers reach me," and it came back green: resolves to matrix.itguyeric.com:443, connects to the COLO WAN IP, TLS and keys valid.

One trap surfaced immediately after, on the SWAG side: a shipped conf never reloaded because the role's restart handler only fires on a CHANGED file, and I had been moving confs around by hand. The confs were on disk but not in the running config. A clean site.yml run ships them as changed and reloads; the manual meddling was mine.

Forgejo build reporting: five gates, none of them the homeserver

Getting itguyeric pushed 1 commit to actually land in a room took clearing five separate real gates, and it is worth recording that not one of them was the homeserver or the webhook design:

1. Malformed room ID. The Matrix webhook wants the internal !id:server, not the #alias. I had pasted the form placeholder mashed with the alias.
2. Bot invited, not joined. @forgejo was Invited, and you cannot send to a room you have not joined. A bot account will not auto-accept; join it via the CS API with its token (joining by alias also returns the real room ID, two birds).
3. WAN hairpin timeout. Forgejo is on the LAN, matrix.itguyeric.com resolved to the public WAN IP, and the box tried to hairpin out and back, which this network does not do.
4. Split-DNS. The fix for 3: an Unbound host override so matrix and hookshot resolve to SWAG's LAN IP internally, like every other internal service. (OPNsense is a pet, so this is a manual override, not IaC.)
5. Forgejo's SSRF allowlist. Once split-DNS pointed at a private IP, Forgejo's webhook.ALLOWED_HOST_LIST refused it by default. Scoped it to *.itguyeric.com rather than the whole /24, so a webhook still cannot be aimed at an arbitrary internal box.

Chose the native Forgejo Matrix webhook over hookshot for this one, against my own recommendation. hookshot is the hub for things with no native Matrix support (Alertmanager, monitoring), and this keeps Forgejo the one snowflake that does not use it, plus a Matrix access token now lives hand-configured in Forgejo's database. Recorded as a deliberate call, not an accident.

The room as an announcement channel

The Infra room is public but read-only for anyone who is not me or a bot. Power-level model: events_default at Moderator so joiners (Default) can read but not send; my accounts are Admin; bots (hookshot, forgejo, and each generic-webhook ghost) get Moderator so they can post. The one non-obvious bit: state_default (Element calls it "Change settings") must be Moderator too, or hookshot cannot write its connection state event and reports "promote me to Admin/Moderator" despite already being a Moderator. The genuinely sensitive state events (room name, avatar, permissions, encryption) are individually pinned to Admin, so a Moderator bot can create connections but change nothing important.

The through-line

Almost every wound today was the same wound: something that looked correct, reported green, and did the wrong thing. :Z relabelling behind my back, nginx caching a dead IP, a delegation silently landing on the main process, a webhook firing into a room ID that did not exist, set -u off in two workflows so a missing key never errored. The homeserver itself was rarely the problem; the plumbing around it, and my own confident-but-wrong pronouncements about what was fixed, were. The lesson is Runbook 18's lesson again, sharper: build the check that fails loudly, verify the thing you are about to make live, and do not call it done until the actual result says so.

To be appended (bridges)

Coming in the next pass, once federation is confirmed settled overnight: mautrix-signal, mautrix-discord (bot-token login, the safe path), the Slack app integration, and hookshot doing generic webhooks for Alertmanager plus in-room commands. The appservice worker and the /etc/mtrx secret-rendering pattern are already in place for them; each bridge is another registration.yml from vault and another quadlet on the mtrx network.