Add swap to all VMs #1

New issue

Open

opened 2026-08-13 15:50:47 +00:00 by itguyeric · 1 comment

itguyeric commented

2026-08-13 15:50:47 +00:00

Owner

Carried over from the post-migration list; the last open item on it.

No VM in the fleet has swap. Under memory pressure that means the OOM killer rather than degraded performance, and on the game box in particular that is a bad trade.

Do it in itg-base so it applies fleet-wide, most likely a systemd-swap-style unit or a swapfile on the local root created by a first-boot unit. Note /var is the writable mount and the root disk is now 32G (Runbook 18), so there is room.

Done when: every VM reports swap in free -h, and it survives a rebuild rather than being a one-off.

Carried over from the post-migration list; the last open item on it. No VM in the fleet has swap. Under memory pressure that means the OOM killer rather than degraded performance, and on the game box in particular that is a bad trade. Do it in `itg-base` so it applies fleet-wide, most likely a `systemd-swap`-style unit or a swapfile on the local root created by a first-boot unit. Note `/var` is the writable mount and the root disk is now 32G (Runbook 18), so there is room. Done when: every VM reports swap in `free -h`, and it survives a rebuild rather than being a one-off.

itguyeric added the

type/task

area/storage

labels

2026-08-13 15:50:47 +00:00

itguyeric added this to the Homelab project

2026-08-13 15:59:41 +00:00

itguyeric self-assigned this

2026-08-13 16:01:26 +00:00

itguyeric commented

2026-08-14 17:49:49 +00:00

Author

Owner

Blocked for #2, lets get metrics before we build a solution.

The thing that makes this interesting on your fleet is that these are VMs, so "add a swap partition" means the paging goes to the same Proxmox storage the VM's root disk already lives on. You are asking the hypervisor's disks to absorb memory pressure they are already absorbing as regular IO. It is the difference between a bigger desk and a bigger filing cabinet: swapping to disk gives you the filing cabinet, and every trip to it costs you.

That is why I would start with zram rather than disk. zram is a compressed block device in RAM that the kernel uses as swap. Cold pages get compressed at roughly two or three to one instead of being written out, so a host with 4G effectively gets a couple of gigabytes of headroom with no disk IO at all. Fedora has shipped it as the default swap on Workstation and Server for years, so this is not exotic. It costs CPU on the compress and decompress, which on your box is the cheap resource.

It also fits how you work. The config lives at /usr/lib/systemd/zram-generator.conf, which is image territory, so it goes in itg-base as a few lines in the Containerfile, ships to the whole fleet on the next nightly, and never touches Proxmox, sizes.yml, or a single VM config. No new disks to provision, nothing machine-local to drift.

The honest limit: zram buys headroom against spiky, compressible memory pressure. It does not help if a workload genuinely needs more resident memory than the VM has. Compressing pages you are actively touching just adds latency. So if something is OOMing because it actually needs the RAM, the answer is more RAM in the tier table, not swap of any kind.

There is a real gotcha if we do end up wanting disk-backed swap on a specific host. Your data disks are XFS, and swapon refuses a file with holes in it. fallocate creates unwritten extents, so the swapfile it produces gets rejected, and the fix is dd if=/dev/zero, which nobody remembers until it fails. Also, /var is machine-local on bootc, so a swapfile there cannot be baked into the image and needs a first-boot unit to create it if absent, exactly like the data-disk initialization in Runbook 2. A dedicated swap disk added by provision-proxmox with a systemd .swap unit is cleaner than a swapfile if we go that route.

One more piece worth checking either way: vm.swappiness. Fedora tunes it up sharply when zram is in play, because swapping to RAM is cheap. That is a sysctl drop-in in /usr/lib/sysctl.d/, so it rides along in the same image change.

So, which is this? Is #1 a general "these VMs have no swap and that makes me nervous" item, in which case I will put zram in itg-base and it covers the fleet in one commit, or did something actually run out of memory, in which case tell me which host and I will size real swap for that workload instead?

Blocked for #2, lets get metrics before we build a solution. The thing that makes this interesting on your fleet is that these are VMs, so "add a swap partition" means the paging goes to the same Proxmox storage the VM's root disk already lives on. You are asking the hypervisor's disks to absorb memory pressure they are already absorbing as regular IO. It is the difference between a bigger desk and a bigger filing cabinet: swapping to disk gives you the filing cabinet, and every trip to it costs you. That is why I would start with zram rather than disk. zram is a compressed block device in RAM that the kernel uses as swap. Cold pages get compressed at roughly two or three to one instead of being written out, so a host with 4G effectively gets a couple of gigabytes of headroom with no disk IO at all. Fedora has shipped it as the default swap on Workstation and Server for years, so this is not exotic. It costs CPU on the compress and decompress, which on your box is the cheap resource. It also fits how you work. The config lives at /usr/lib/systemd/zram-generator.conf, which is image territory, so it goes in itg-base as a few lines in the Containerfile, ships to the whole fleet on the next nightly, and never touches Proxmox, sizes.yml, or a single VM config. No new disks to provision, nothing machine-local to drift. The honest limit: zram buys headroom against spiky, compressible memory pressure. It does not help if a workload genuinely needs more resident memory than the VM has. Compressing pages you are actively touching just adds latency. So if something is OOMing because it actually needs the RAM, the answer is more RAM in the tier table, not swap of any kind. There is a real gotcha if we do end up wanting disk-backed swap on a specific host. Your data disks are XFS, and swapon refuses a file with holes in it. fallocate creates unwritten extents, so the swapfile it produces gets rejected, and the fix is dd if=/dev/zero, which nobody remembers until it fails. Also, /var is machine-local on bootc, so a swapfile there cannot be baked into the image and needs a first-boot unit to create it if absent, exactly like the data-disk initialization in Runbook 2. A dedicated swap disk added by provision-proxmox with a systemd .swap unit is cleaner than a swapfile if we go that route. One more piece worth checking either way: vm.swappiness. Fedora tunes it up sharply when zram is in play, because swapping to RAM is cheap. That is a sysctl drop-in in /usr/lib/sysctl.d/, so it rides along in the same image change. So, which is this? Is #1 a general "these VMs have no swap and that makes me nervous" item, in which case I will put zram in itg-base and it covers the fleet in one commit, or did something actually run out of memory, in which case tell me which host and I will size real swap for that workload instead?