Proxmox Deep Dive Part 2: Storage Architecture

Part 1 covered the Debian base, KVM/LXC split, and pmxcfs. This part covers the decision that constrains nearly everything downstream in this series: storage.

Storage types.

Proxmox supports several storage backends, each suited to a different job. Directory storage is the simplest, a plain filesystem path, fine for ISOs and backups, workable but unremarkable for VM disks. LVM and LVM-thin sit on top of block devices, LVM-thin adding thin provisioning and fast snapshots that plain LVM lacks. ZFS gets its own full treatment in Part 3, since it’s the default recommendation for a homelab build from scratch. NFS and CIFS provide network-attached storage from an external server, useful for centralizing ISOs and backups across a cluster without needing shared block storage. iSCSI provides network block storage, a step up from NFS/CIFS in performance for VM disks, at the cost of more setup complexity. Ceph, covered in Part 9, is Proxmox’s own hyperconverged distributed storage, and sits in a different category entirely, it turns the cluster’s own nodes into the storage layer rather than pointing at an external one.

Content types, and why not every storage can hold everything.

Each storage definition in Proxmox declares which content types it’s allowed to hold: disk images (VM virtual disks), ISO images, container templates, backups (vzdump archives), and snippets (small config files like cloud-init snippets). A Directory storage can be configured to hold any combination of these; an LVM-thin storage can only hold disk images and containers, not ISOs or backups, because LVM-thin has no filesystem for Proxmox to place ordinary files into, it’s raw block storage carved into logical volumes. This is a common point of confusion for anyone new to Proxmox: creating an LVM-thin pool and then wondering why the ISO upload option doesn’t appear for it, that’s expected behavior, not a bug.

Thin versus thick provisioning.

Thick provisioning allocates the full declared size of a virtual disk immediately. Thin provisioning allocates space as data is actually written, so a 100 GiB virtual disk that currently holds 10 GiB of data only consumes 10 GiB on the underlying storage. Thin provisioning is the practical default for a homelab, more VMs fit on the same physical disk, at the cost of needing to actually monitor real usage, since the storage can be overcommitted past its physical capacity if every thin-provisioned VM’s data grows at once. LVM-thin and ZFS both support this natively; plain LVM does not, and Directory storage’s behavior depends on the underlying filesystem and the disk image format chosen (qcow2 supports thin provisioning, raw does not).

Local versus shared storage, and why it decides whether live migration works.

This is the distinction that actually matters most for anything covered later in this series. Local storage lives on one node only, LVM or ZFS on that node’s own physical disks. Shared storage is reachable from every node in a cluster, NFS, CIFS, iSCSI, or Ceph. Live migration, moving a running VM from one node to another with no downtime, requires the VM’s disk to already be reachable from the destination node. With shared storage, that’s automatic, the disk doesn’t move, only the VM’s running state does. With local storage, Proxmox can still migrate a running VM, but it has to copy the entire disk to the destination node during the migration, which takes proportionally longer and uses more bandwidth. HA, covered in Part 8, has an even harder requirement: it generally needs the VM’s disk on shared storage (or ZFS with replication, a middle ground covered in Part 7), because HA has to be able to start the VM on a different node after the original one has already failed, with no opportunity to copy a disk from a host that’s no longer reachable.

For a single-node homelab, this distinction is invisible, everything is local by definition. The moment a second node joins, it becomes the first architectural decision that actually has consequences, and it’s much easier to plan storage around it from the start than to migrate a running homelab onto shared storage after the fact.

Part 3 goes deep on ZFS specifically, RAID levels, RAM sizing, compression, and the pitfalls that show up more often in homelab-scale builds than in enterprise ones.