Monitoring and alerting #2

New issue

Open

opened 2026-08-13 15:50:47 +00:00 by itguyeric · 1 comment

itguyeric commented

2026-08-13 15:50:47 +00:00

Owner

Highest-leverage item on the list. Three separate silent degradations were found by accident in a single day:

The whole fleet ran a day behind for weeks because the build cron fired an hour after the hosts pulled. Every timer reported success.
trusted.conf protected the arr web UIs while leaving /api wide open, because the include sat inside location /.
coulson's tailscaled was offline for hours and it was only noticed when it was needed.

Every one of those looked healthy from outside. The common shape is that nothing was checking the thing that was believed.

Minimum useful version: host up/down, systemd unit failures, disk and thin-pool usage, and a check that each host's booted bootc digest matches what the pipeline last pushed. Tailscale can webhook on node-offline for roughly ten minutes of setup and would have caught one of the three today.

Blocks nothing formally, but makes DR, backups and the Kubernetes experiment far less scary.

Highest-leverage item on the list. Three separate silent degradations were found by accident in a single day: - The whole fleet ran a day behind for weeks because the build cron fired an hour after the hosts pulled. Every timer reported success. - `trusted.conf` protected the arr web UIs while leaving `/api` wide open, because the include sat inside `location /`. - coulson's tailscaled was offline for hours and it was only noticed when it was needed. Every one of those looked healthy from outside. The common shape is that nothing was checking the thing that was believed. Minimum useful version: host up/down, systemd unit failures, disk and thin-pool usage, and a check that each host's booted bootc digest matches what the pipeline last pushed. Tailscale can webhook on node-offline for roughly ten minutes of setup and would have caught one of the three today. Blocks nothing formally, but makes DR, backups and the Kubernetes experiment far less scary.

itguyeric added the

type/project

area/observability

labels

2026-08-13 15:50:47 +00:00

itguyeric added this to the Homelab project

2026-08-13 15:59:41 +00:00

itguyeric self-assigned this

2026-08-13 16:01:26 +00:00

itguyeric referenced this issue

2026-08-14 17:49:49 +00:00

Add swap to all VMs #1

itguyeric commented

2026-08-19 02:26:42 +00:00

Author

Owner

Folding web analytics into this project: privacy-friendly, cookie-free analytics for the Hugo site with Umami or Plausible (self-hosted, behind SWAG, same Postgres pattern as everything else).

Also, given the Victoria Metrics contacts, worth evaluating VictoriaMetrics as the metrics backend/TSDB here instead of (or fronting) Prometheus, with Grafana on top. Companion pieces to slot in as we build this out:

Loki for logs
Uptime Kuma for internal service status (v2 supports MariaDB, so no more SQLite corruption)
an external check (Uptime Robot or similar) so we still get paged when the box itself is down
Scrutiny for disk SMART health on the NFS/storage
NUT/PeaNUT to Prometheus/VM if there is a UPS worth watching

Folding web analytics into this project: privacy-friendly, cookie-free analytics for the Hugo site with Umami or Plausible (self-hosted, behind SWAG, same Postgres pattern as everything else). Also, given the Victoria Metrics contacts, worth evaluating **VictoriaMetrics** as the metrics backend/TSDB here instead of (or fronting) Prometheus, with Grafana on top. Companion pieces to slot in as we build this out: - Loki for logs - Uptime Kuma for internal service status (v2 supports MariaDB, so no more SQLite corruption) - an external check (Uptime Robot or similar) so we still get paged when the box itself is down - Scrutiny for disk SMART health on the NFS/storage - NUT/PeaNUT to Prometheus/VM if there is a UPS worth watching