Hi guys,
I have a cozystack bare metal cluster (running on OVH) upgraded to 1.6.1 and then and I'm trying to run a Talos upgrade 1.13.0 -> 1.13.6 now but it seems I'm hitting a reboot wedge with the ZFS + DRBD/LINSTOR stack, and I'd like to know if it's a known issue? Here are some more details:
Full Setup: Talos v1.13.0 (ghcr.io/cozystack/cozystack/talos, upgrading to v1.13.6 via talm 0.34.0), kernel 6.18.24, extensions zfs 2.4.3 + drbd 9.3.2, LINSTOR storage on zvols.
Symptom: talm reboot (and the reboot phase of upgrades) never completes. The node was fully drained first; kubelet, etcd, trustd and cri all stop cleanly ("Service finished successfully"), then the sequence wedges on ext-zfs-service: stuck in "Stopping", machined sends SIGKILL and the task survives it (D-state / uninterruptible kernel I/O). MachineStatus stays stage: rebooting forever.
Kernel shows DRBD secondaries on the node hit local-IO errors as ZFS is torn down under them and detach one at a time — 12 of 15 volumes detached over ~20 min with an exponentially slowing tail, then total silence: zero kernel log lines for 25+ min, 3 volumes never detached. Looks like a teardown-ordering problem: the ZFS service is being stopped while DRBD still has the zvols open, and ext-zfs-service blocks in a zfs ioctl that can never complete.
The same cluster wedged the same way during my second attempt (that time on an undrained node it stuck at "stopping cri" first). Only exit both times: talm reboot -m force, which on our AMI boards then risks a POST hang from the dirty reset.
Is there a known-good teardown ordering (e.g. disconnecting DRBD before the zfs extension stops), or a fix planned? Happy to file a GitHub issue with full service states, kernel logs, and timelines — everything is captured.
I've been a pretty early adopted of Cozystack and this cluster is running for 245 days, having been upgraded several times up to 1.6.1 from as early as 0.40.x or so. Same goes for the respective Kubernetes and Talos versions - several upgrades each, and not going to lie - there were some issues in the past which I dealt with on my own but I don't remember getting to a fully hung state and a OVH power cycle being the only solution.