Proxmox Deep Dive Part 8: High Availability

Part 7 covered quorum, the voting mechanism a cluster uses to agree on its own membership. High Availability is built directly on top of that mechanism: it’s the feature that actually restarts a failed VM or container on a surviving node, automatically, using quorum to decide when that’s safe to do.

HA groups and resources.

An HA-managed guest is added as an HA resource, assigned to an HA group, a named subset of the cluster’s nodes the resource is allowed to run on, in a specified priority order. If the node currently running the resource fails, Proxmox’s HA manager starts it on the next available node in that group’s priority order. This is opt-in per guest, adding a VM or container to HA is a deliberate choice, not a cluster-wide default, which matters for a homelab where some services genuinely need to survive a node failure and plenty of others don’t.

Fencing: why HA can’t just assume a node is actually dead.

The hard problem HA has to solve isn’t restarting a VM, it’s being certain the VM isn’t still running somewhere else first. If a node stops responding because of a network blip rather than an actual failure, and HA starts the same VM on a different node without being sure the original copy has stopped, both copies could end up running against the same disk simultaneously, corrupting it. Fencing is the mechanism that prevents this: before HA starts a resource elsewhere, it needs a way to guarantee the original node either has genuinely stopped, or has been forced to stop.

Proxmox’s HA manager relies on a watchdog, a mechanism (either the software softdog kernel module or a hardware watchdog timer where the motherboard supports one) that must be reset periodically by the node’s own HA processes. If a node loses quorum, or its HA processes hang, the watchdog stops being reset and the hardware or kernel forces a reboot after a timeout, guaranteeing the node genuinely stops running anything before the rest of the cluster considers it safe to restart resources elsewhere. This is why HA has a real dependency on quorum from Part 7: a node that’s merely unreachable but still quorate won’t self-fence, and a node that’s lost quorum is, by the watchdog mechanism, guaranteed to actually stop.

Failure detection timing.

HA doesn’t react instantly, and that’s deliberate. A node has to be confirmed unreachable for a sustained period, not just miss a single heartbeat, before HA considers it failed and begins the fencing and restart sequence. That delay exists specifically to avoid triggering a full fencing-and-restart cycle over a transient network glitch, at the direct cost of a real outage taking longer to recover from automatically than a human noticing and intervening manually might.

Is HA actually worth it in a homelab?

Often, no, and it’s worth saying that plainly rather than treating HA as a feature every serious homelab should enable. HA requires shared or replicated storage (Part 2, Part 7) to make failover meaningful, a minimum of three nodes to have quorum survive a single node failure without a QDevice, and adds real operational complexity, fencing behavior, watchdog configuration, HA group priority, that has to be understood correctly or it becomes a source of unexpected reboots rather than protection. For a homelab running the kind of workloads this site’s own labs represent, a Sn1per attack-surface lab, a BloodHound graph lab, a FortiManager lab, none of them need to survive a node failure automatically; if the node hosting one of them goes down, restarting it manually once the node is back is entirely acceptable, and the complexity budget is better spent elsewhere. HA earns its keep for services where an automatic restart genuinely matters, a DNS resolver, a reverse proxy something else depends on, not for every VM by default.

Part 9 covers Ceph, the storage layer that makes HA’s shared-storage requirement possible without an external NAS, and an honest look at whether it’s worth the complexity at homelab scale either.