Proxmox Deep Dive Part 3: ZFS, the Default Recommendation

Part 2 covered the storage landscape. ZFS gets its own part because it’s the storage backend the Proxmox installer itself offers as a first-class option, and the one most homelab builds should default to unless there’s a specific reason not to.

Pools, vdevs, and the RAID-level decision.

A ZFS pool is built from one or more vdevs (virtual devices), and the vdev topology is the actual RAID-level decision. A mirror vdev behaves like RAID1, full redundancy, and importantly, mirrors preserve close to a single drive’s IOPS per mirror pair regardless of how many mirror pairs get striped together. RAIDZ1/2/3 behave like RAID5/6/7 in redundancy terms (one/two/three drive failures tolerated), but for small-block random I/O, a RAIDZ vdev delivers roughly the IOPS of a single drive no matter how many drives make up the vdev, because every read or write touches every disk in the vdev. Proxmox VE 9 also added dRAID as an option, distributed RAID that rebuilds faster after a drive failure by spreading spare capacity across all drives instead of relying on one dedicated hot spare.

For VM workloads specifically, that IOPS behavior is the deciding factor more than raw capacity. Two mirrored pairs across four drives give roughly double the random IOPS of a four-drive RAIDZ2, at the cost of 50% less usable capacity for the same drive count. RAIDZ is the right call for bulk storage or streaming workloads where sequential throughput and capacity efficiency matter more than random IOPS; mirrors are the right call for VM disk storage, which is overwhelmingly random small-block I/O.

RAM: ZFS does not run on the storage layer alone.

ZFS is not a “storage feature” a system can quietly ignore the RAM cost of. The rule of thumb: at least 2 GiB base plus roughly 1 GiB per TiB of pool storage, so an 8 TiB pool wants on the order of 10 GiB dedicated to ZFS’s own use before a single VM’s own RAM allocation is considered. Since Proxmox VE 8.1, the ZFS ARC (Adaptive Replacement Cache, ZFS’s read cache) defaults to 10% of host RAM, capped at 16 GiB. The trap: Proxmox does not automatically shrink the ARC to make room when VMs need guaranteed memory, the ARC and the VMs compete for the same physical RAM pool. On a host running VMs that need their allocated RAM reliably available, capping zfs_arc_max manually in /etc/modprobe.d/zfs.conf is the fix, and for hosts running many memory-hungry VMs, that cap sometimes needs raising toward 25 to 30% of host RAM instead of leaving it at the conservative default, the correct number depends on the actual workload rather than a single rule.

Compression: on by default, and there’s little reason not to.

LZ4 compression adds minimal CPU overhead and typically reduces storage consumption by 20 to 30% on ordinary data. Enabling it is close to a strict win for a homelab, the CPU cost is small relative to the capacity and I/O savings, since compressed data means fewer physical blocks read and written for the same logical data. zstd is available as a higher-ratio, higher-CPU-cost alternative where the tradeoff makes sense.

Deduplication: the one ZFS feature that’s usually the wrong answer for a homelab.

Deduplication sounds appealing on paper, storing only one copy of any block that appears more than once across the pool, but the deduplication table has to live in RAM in its entirety, and it costs several GiB of RAM per TiB of deduplicated data. For most homelab-scale builds, that RAM cost, combined with a real performance penalty on every write while the dedup table gets checked, outweighs the storage saved. The practical guidance is to leave dedup=off unless there’s a specific, well-resourced reason to turn it on, LZ4 compression alone captures most of the easy win at a fraction of the cost.

Snapshots, clones, and scrubs.

ZFS snapshots are near-instant and cost nothing until the underlying data changes (copy-on-write means a snapshot only holds the blocks that have since been overwritten). Clones let a new dataset branch off a snapshot without copying the underlying data until it diverges, which is exactly how Proxmox implements fast VM template cloning on ZFS storage. Scrubs, a background integrity check that reads every block and verifies it against its checksum, should run on a schedule (Proxmox can schedule these directly); they’re the mechanism that actually catches silent data corruption before it’s discovered the hard way during a restore.

An L2ARC device, a fast SSD dedicated to holding ARC overflow, is worth adding when the working set is larger than RAM allows for and read-heavy, it’s a compounding optimization on top of RAM sizing rather than a substitute for getting the RAM sizing right in the first place.

Part 4 moves from storage to networking, the Linux bridges every VM’s virtual NIC attaches to, and the newer SDN stack that sits on top of them.