After the Patch: Six Hosts, One Silent apt Failure, and an Upgrade Playbook I'm Keeping

1 November 2026 Linux Security CVE Proxmox Ansible Automation Homelab Debian

In my CVE-2026-98432 post the “What I did” section was one line:

sudo apt update && sudo apt full-upgrade -y && sudo reboot

That’s true for a single machine. It isn’t how the work actually went. Across two Proxmox hosts, a physical NTP box and three guests, that one line turned into a morning of backups, a routing fault that apt hid behind “All packages are up to date”, and a playbook I’m keeping for every upgrade from now on. This post covers the whole process, including the one step I’d do differently, and the question I couldn’t answer from a version number: whether the Proxmox kernel was actually fixed.

The estate, and the order I chose

Six machines were in scope. I left out the lab networks on purpose, because those have their own posts and their own risk profile. I kept anything that also sits on the home LAN.

Host	What it is	How it got upgraded
relay	Beelink mini PC running Proxmox, with my Jellyfin container and consulting website VM	By hand
pve	The main Proxmox host, a Dell R640	By hand
herald	Physical Debian box, NTP and a Cloudflare tunnel	Ansible
jellyfin	Unprivileged LXC on relay	Ansible
website	Debian VM on relay	Ansible
ansible-ctrl	The control node from the Pi rebuild series	Ansible, last

The hypervisors got done by hand because they need a reboot and someone watching it. Ansible doesn’t add much there. The guests and herald are the repeatable part, and they’re what the playbook is for.

Back up before touching anything

On relay, the Jellyfin container’s root disk lives on a ZFS pool, and the media library is bind-mounted in from the same pool. I didn’t want a 280 GB media library inside a backup, and I didn’t need to exclude it by hand:

INFO: excluding bind mount point mp0 ('/media') from backup (not a volume)

vzdump never follows bind mounts. The container backup came out at 1.5 GB and the website VM at 1.8 GB, mostly because 80% of its 20 GB disk was zeroes.

The backups went onto a new ZFS dataset rather than the default local storage, because local sits on the same NVMe as the website VM:

sudo zfs create bulk/backup
sudo pvesm add dir bulk-backup --path /bulk/backup --content backup --is_mountpoint yes
sudo vzdump 100 101 --storage bulk-backup --mode snapshot --compress zstd

--is_mountpoint yes is the flag worth knowing. Without it, if the pool ever fails to import, Proxmox happily writes backups into an empty directory on the root disk and fills it.

One line in the log was worth acting on straight away:

INFO: skipping guest filesystem freeze - agent configured but not running?

The VM had the QEMU guest agent enabled in its config, but the agent wasn’t installed inside it. Installing it gives consistent snapshots and clean shutdowns. As it turned out, it also gave me a way into the VM later on, without SSH.

Last, a tarball of the host config that would hurt to rebuild: /etc/pve, /etc/network/interfaces, apt sources, modprobe settings and the kernel command line.

Mirrors, https, and what the signature actually protects

Both Proxmox hosts had Debian sources on plain http://. I switched them to https://, and I pointed the Proxmox repository at a mirror that was faster from here than the default host.

It’s worth being clear about what that does and doesn’t protect. apt checks every package against the repository’s signing key (Signed-By:), so neither plain http nor a third-party mirror can slip you a modified package. What https adds is privacy (nobody on the path sees which packages you fetch) and resistance to a stale-metadata attack, where someone on the path replays old indexes to hide new security fixes. Given that the whole exercise was about getting a security fix, the second point is enough reason on its own.

A third-party mirror has one more cost that the signature doesn’t cover: it can lag. The kernel fix reached my hosts through that mirror, so if it had been slow to sync, I’d have been waiting on someone else’s schedule without knowing it.

relay then upgraded cleanly. It took 29 packages and a new kernel, rebooted, and both guests came back by themselves because they’re set to onboot: 1. I checked the iGPU was still visible inside the Jellyfin container (/dev/dri/renderD128, gid 992 mapped to render), because a kernel update that breaks hardware transcoding is exactly the sort of failure you only notice a week later.

“All packages are up to date” that wasn’t

pve was more interesting. apt update printed a long run of errors for every Debian source, then this:

Err:1 https://deb.debian.org/debian trixie InRelease
  SSL connection failed: ... Connection reset by peer [IP: 151.101.2.132 443]
All packages are up to date.
Warning: Some index files failed to download. They have been ignored, or old ones used instead.

apt still exits successfully here. The Proxmox mirror worked, so apt compared the Proxmox packages against fresh indexes and the Debian packages against stale ones, then reported that nothing needed upgrading. If you only read the last line, or you run it from a script that checks the exit code, you’d conclude the box was patched. relay had picked up Debian security fixes the same morning, so pve almost certainly needed them too.

Working out the cause took four commands:

ip route get 151.101.2.132
ip route show
ping -c2 -M do -s 1472 151.101.2.132
curl -sv -o /dev/null https://deb.debian.org/debian/dists/trixie/InRelease

The default route was going out through a lab overlay network rather than the home LAN. I’d moved it to the LAN the night before with ip route replace, and the reboot onto the new kernel had quietly thrown that away. A command-line route is gone on the next boot.

The ping showed a path MTU of 1433, and it was tempting to blame that. But the gateway was sending back “Frag needed”, so path MTU discovery was working, and an MTU problem makes connections hang rather than reset. The mirror used the same route and worked. Something on the overlay path was resetting TLS to Fastly’s addresses specifically.

The fix was to make the LAN the default route permanently and send only the overlay’s own range back through the overlay. In /etc/network/interfaces I moved the gateway line from the overlay bridge to the LAN bridge, and added one line to the overlay bridge:

post-up ip route replace 10.x.0.0/16 via <overlay-gw> dev vmbr0

Proxmox keeps post-up lines when you edit networking in the GUI, so this survives. ifreload -a applied it without a reboot, and the iDRAC console was there in case it didn’t.

With Debian reachable again, pve was already current. That raises a point I’ll come back to: Debian’s kernel version tells you whether you’re patched, and a Proxmox kernel version doesn’t.

Getting Ansible to every host

The control node already existed. I gave this job its own repo instead of reusing the Pi project:

homelab-ansible/
├── ansible.cfg
├── inventory/hosts.yml
└── playbooks/upgrade.yml

The inventory splits hosts into debian (the hosts to upgrade) and proxmox (checks only). For root I used become with -K, so I’m prompted for the sudo password on each run. That’s the same reasoning as the Pi series: no passwordless sudo rule sitting on disk.

Two hosts needed more than an SSH key copied over, and both were good lessons.

The website VM only accepts keys. That meant ssh-copy-id couldn’t log in to install a key. Rather than open password login for a minute, I used the guest agent I’d installed an hour earlier. It lets the Proxmox host run commands inside the VM directly:

sudo qm guest exec 101 -- sh -c 'install -d -m 700 -o anon -g anon /home/anon/.ssh && echo "ssh-ed25519 AAAA... ansible-ctrl" >> /home/anon/.ssh/authorized_keys'

The Jellyfin container timed out on port 22. A timeout rather than “connection refused” pointed at a firewall, and the container’s Proxmox firewall did have policy_in: DROP with no rule for SSH. It also had no user account. I created the user and key, added one firewall rule for the LAN, and dropped in a keys-only SSH config. Then sshd -t failed:

Missing privilege separation directory: /run/sshd

The container’s SSH is started by systemd socket activation, which creates /run/sshd when the service starts. A manual config check running outside the service has no such directory. Creating it by hand is harmless, since /run is cleared on every boot. With the config confirmed, password login was refused and key login worked:

$ ssh -o PubkeyAuthentication=no anon@jellyfin
anon@jellyfin: Permission denied (publickey).

Then ansible all -K -m command -a 'id -un' returned root for all six hosts.

The playbook

The design rules came straight from the morning’s work:

Dry run by default. Nothing changes unless you pass -e do_upgrade=true.
One host at a time (serial: 1). If every host in a batch fails, Ansible stops the play, so a failure on one host can’t spread to the next.
Fail on any apt fetch error. This is pve’s lesson. APT::Update::Error-Mode=any turns “warning, used old indexes” into a hard failure.
Switch only Debian’s own sources to https, and report any other plain-http repos without changing them. Not every third-party repo supports https.
Keep existing config files when a package ships a new version (force-confdef,force-confold). That’s the equivalent of answering N at the prompt by hand.
Never reboot. Report whether a reboot is needed and leave the decision to a person.

---
# Debian upgrades, one host at a time.
#   Dry run (default):  ansible-playbook playbooks/upgrade.yml -K
#   Apply:              ansible-playbook playbooks/upgrade.yml -K -e do_upgrade=true
#   Single host:        add -l herald
- name: Debian package upgrades
  hosts: debian
  serial: 1
  become: true
  vars:
    do_upgrade: false
    apply: "{{ do_upgrade | bool }}"
    stamp: "{{ ansible_facts['date_time']['date'] | replace('-', '') }}"

  tasks:
    - name: Back up /etc/apt
      ansible.builtin.command: tar czf /root/apt-backup-{{ stamp }}.tgz /etc/apt
      args:
        creates: /root/apt-backup-{{ stamp }}.tgz
      when: apply

    - name: Find apt source files
      ansible.builtin.find:
        paths: /etc/apt
        patterns: ['sources.list', '*.list', '*.sources']
        recurse: true
      register: apt_sources

    - name: Switch Debian sources from http to https
      ansible.builtin.replace:
        path: "{{ item.path }}"
        regexp: 'http://((deb|security)\.debian\.org)'
        replace: 'https://\1'
      loop: "{{ apt_sources.files }}"
      loop_control:
        label: "{{ item.path }}"
      check_mode: "{{ not apply }}"
      diff: true

    - name: Report other http sources (left unchanged)
      ansible.builtin.shell: >
        grep -rnE '^[^#]*http://' /etc/apt
        --include='*.list' --include='*.sources' --include='sources.list'
        | grep -vE 'http://(deb|security)\.debian\.org'
      register: other_http
      changed_when: false
      failed_when: other_http.rc > 1

    - name: Show other http sources
      ansible.builtin.debug:
        var: other_http.stdout_lines
      when: other_http.stdout_lines | length > 0

    - name: apt update (fail on any fetch error)
      ansible.builtin.command: apt-get -q update -o APT::Update::Error-Mode=any
      environment:
        LC_ALL: C
      changed_when: false

    - name: List upgradable packages
      ansible.builtin.shell: apt list --upgradable 2>/dev/null | tail -n +2
      register: upgradable
      changed_when: false

    - name: Show upgradable packages
      ansible.builtin.debug:
        msg: "{{ upgradable.stdout_lines | length }} upgradable{{ ':' if upgradable.stdout_lines else '' }} {{ upgradable.stdout_lines }}"

    - name: Full upgrade (keep existing config files)
      ansible.builtin.apt:
        upgrade: full
        dpkg_options: force-confdef,force-confold
      environment:
        DEBIAN_FRONTEND: noninteractive
        NEEDRESTART_MODE: a
      when:
        - apply
        - upgradable.stdout_lines | length > 0

    - name: Check whether a reboot is needed
      ansible.builtin.shell: |
        running=$(uname -r)
        newest=$(ls /boot/vmlinuz-* 2>/dev/null | sed 's#.*/vmlinuz-##' | sort -V | tail -1)
        flag=no; [ -f /var/run/reboot-required ] && flag=yes
        echo "running=$running newest=${newest:-n/a-container} reboot-required=$flag"
      register: reboot_state
      changed_when: false

    - name: Reboot status
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }}: {{ reboot_state.stdout }}"

The http-to-https task runs in check mode during a dry run, with diff: true, so the dry run shows exactly which lines would change. The find is recursive, which matters on cloud-init images: the website VM’s sources go through /etc/apt/mirrors/debian.list, which a search of sources.list.d alone would miss.

The run

I dry-ran herald first. It was the only host with plain-http Debian sources, and the diff showed exactly six lines changing in /etc/apt/sources.list (main, security and updates, each with deb and deb-src) and nothing else. Its cloudflared, NodeSource and Caddy repos were already https. Two packages were pending, no reboot was needed, and nothing had been touched.

Applying to herald, then checking that the things it exists for still worked:

active                      # cloudflared
active                      # chrony
Reference ID    : ... Stratum 3

Then the rest of the debian group in one run:

PLAY RECAP *****************************************************************
ansible-ctrl               : ok=10   changed=1    unreachable=0    failed=0
herald                     : ok=10   changed=0    unreachable=0    failed=0
jellyfin                   : ok=11   changed=3    unreachable=0    failed=0
website                    : ok=11   changed=2    unreachable=0    failed=0

Herald’s changed=0 on the second run is the idempotency check from the Pi series again, for free. The playbook had nothing left to do there, so it did nothing. ansible-ctrl’s single change was its own apt backup.

The step I got wrong

The jellyfin line in that recap hides something. Its upgrade list included this:

jellyfin-server/unknown 12.4+deb13 amd64 [upgradable from: 10.11.11+deb13]

That’s a major version jump. Jellyfin migrates its database the first time a new major version starts, and there’s no simple way back to 10.11 afterwards. I’d dry-run herald because it was the host with source changes, then applied to everything else without dry-running it first. The dry run would have shown that line and given me the choice.

It worked out. The service came back on 12.4.0, and every migration in the journal logged “was successfully applied”. If it hadn’t, the vzdump from that morning was taken before the upgrade. The backup is what turned a mistake into a non-event, which is the whole argument for taking one before every upgrade, even a routine one.

The fix goes into the playbook rather than into me trying harder next time: hold the Jellyfin packages by default unless I pass -e allow_jellyfin=true, and make the dry run flag any package whose major version changes.

Is the CVE actually closed?

The Debian hosts are straightforward. Herald and the website VM both run 6.12.113+deb13-amd64, which is the fixed version from DSA-6568-1.

The Proxmox hosts are different. They run Proxmox’s own 7.0.14 kernel series, and 7.0 isn’t one of the upstream stable branches that got the fix. The fixed upstream releases are 5.10.272, 5.15.223, 6.1.190, 6.6.159, 6.12.113, 6.18.56, 7.2.10 and 7.3. A Proxmox kernel can still carry a backport, but the version number alone can’t tell you either way. Proxmox kernels are built from Ubuntu’s kernel sources, and Ubuntu’s tracker still listed every release as work in progress, so that was no help either.

The answer came from Proxmox itself. In a thread on the Proxmox forum, a staff member confirmed that kernels 7.0.14-25 and 7.0.14-26 include the upstream fix commit, and that -24 and earlier do not. Both of my hosts report the fixed build:

Linux pve 7.0.14-26-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-26 (2026-10-22T18:45Z) x86_64
Linux relay 7.0.14-26-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-26 (2026-10-22T18:45Z) x86_64

That covers the two containers as well. An LXC container runs on its host’s kernel, so jellyfin and ansible-ctrl are patched because relay and pve are.

Two caveats. I’m relying on a staff reply in a forum thread, not a security advisory, and the thread itself notes the CVE wasn’t posted in Proxmox’s Security Advisories forum. I also tried to confirm it from the kernel’s own changelog, and that didn’t help:

apt changelog proxmox-kernel-7.0.14-26-pve-signed 2>&1 | grep -iE 'inet_opt|ireq_opt|98432'

That returned nothing. The changelog downloaded fine, and the entries I read are Proxmox’s own fixes (ZFS races, a CXL revert for the NVIDIA modules) rather than a list of upstream stable patches. So a missing mention doesn’t show the fix is absent, and it doesn’t show it is present either. That leaves the staff statement as the evidence, and I’d rather say so plainly than pretend the changelog confirmed it.

A day later, a routine check on pve showed one new upgradable package: libnss3, a Debian security update unrelated to this CVE. relay already had it. That is a good argument for keeping the hypervisors as a manual look rather than something the playbook touches. The proxmox inventory group only reports, and a person decides what happens next.

Takeaways

“All packages are up to date” can be wrong. If any source failed to fetch, apt compared against stale indexes. Read the warnings, or make them fatal with APT::Update::Error-Mode=any.
A route added with ip route is gone after the next reboot. If a fix is meant to stay, it goes in /etc/network/interfaces.
Containers inherit the host’s kernel, so a kernel CVE on an LXC container is closed by patching the host, not the container.
A distribution’s kernel version tells you whether you’re patched; a hypervisor vendor’s kernel version may not. Ask the vendor, and don’t expect the changelog to name the CVE. Mine didn’t.
A third-party mirror is signed but can lag. The signature protects the package, not how quickly it arrives.
Dry-run every host, not just the one you’re worried about. The surprise came from the host I wasn’t watching.
Take the backup anyway. It’s what turned a major-version upgrade I didn’t plan for into a non-event.

The playbook is now how every Debian box here gets upgraded, CVE or not. The CVE was the reason I sat down to do it. What made it worth the morning is that next time it’s two commands, a dry run and a real one.