6 August 2026
Replymessage unavailable
kubectl patch gateway cozystack -n tenant-root --type merge -p '
spec:
listeners:
- name: http
allowedRoutes:
namespaces:
from: All
'
Artem Skubectl patch gateway cozystack -n tenant-root --type merge -p '
spec:
listeners:
- name: http
allowedRoutes:
namespaces:
from: All
'
thanks @Artem
let me digest first ya.
too much of information. 😂
Фотография
click to show
click to show
Catch you at the 70th Cozystack community meeting today!
When: Thursday at 18:00-19:00 (Central European Time, CET)
Where: Zoom
Link to join in:
https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5
This meeting is free and open to everyone!
Come and join us!
Minutes:
https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/edit?tab=t.0#heading=h.ygb3chatmgn2
📢 Agenda and Notes:
- Marketplace service in details
🎤 Open Floor:
-Test flakiness and platform stability: catching component conflicts earlier, isolating tests from external factors
- Discussion of versioning and release process
- Release frequency, semver vs calver and
- Backport strategy discussion
- Design proposal review: open proposals on GitHub https://github.com/cozystack/community/pulls
- Viewing backup process events in the web dashboard
🗓️Do not miss Cozystack community meetings:
https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2
https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack?view=month
💡Have a feature idea or design proposal for Cozystack?
Submit your Pull Request here: https://github.com/cozystack/community/pulls 🚀
298 · 7 August 2026
Фотография
click to show
click to show
🚀The Cozystack team has open-sourced Blockstor, a control plane for block storage in Kubernetes: LVM and ZFS as backends, replication over DRBD, and a LINSTOR-compatible REST API. The project lives in the cozystack organization and is developed as part of Cozystack, a platform accepted into the CNCF Sandbox. The license is Apache 2.0.
🔥The main thing that makes it worth a look: it is not a fork, and it is not a wrapper. Blockstor is written from scratch in Go, but it speaks the same REST API as LINSTOR — so all the client tooling you already run keeps working without a single change: the linstor CLI, linstor-csi, piraeus-operator, and the golinstor library.
Read the full article in Cozystack blog 👇
https://cozystack.io/blog/2026/08/blockstor-linstor-compatible-storage-for-kubernetes/
329 · 8 August 2026
hi guys, i noticed a weird thing.
i install cozystack v1.6.1 with following command:
helm upgrade --install cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer \
--version 1.6.1 \
--namespace cozy-system \
--create-namespace
but end up it installs 1.6.1-rc.1.
all pods are using 1.6.1-rc.1 images, including:
• cozystack-api
• dashboard
• backup-controller
• metallb-speaker
id 530991411hi guys, i noticed a weird thing.
i install cozystack v1.6.1 with following command:
helm upgrade --install cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer \
--version 1.6.1 \
--namespace cozy-system \
--create-namespace
but end up it installs 1.6.1-rc.1.
all pods are using 1.6.1
Hi, thanks! we have an issue about that. This is purely cosmetic, 1.6.1 and 1.6.1-rc.1 byte-identical, all our artifacts pinned by digests, and tags here are just human readable labels.
9 August 2026
Replymessage unavailable
hi @Aleksei , i testing the Gateway API on 1.6.1.
it seems like this issue HTTPRoute is not allowed to attach to this Gateway due to namespace selector restrictions still happened and blocking dashboard.
while another httproute bucket-cozy-backups-ui status report all good.
HTTPRoute successfully attached to the gateway.
BUT... when i try to access it, the connection completely closed and not allowed to access.
ok. issue confirmed.
it's because the api.<apex> HTTPRoute and TLSRoute conflicted in the gateway, causing cilium-envoy refuses to accept.
how i found:
i run kubectl get gateway -n tenant-root cozystack -o jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.hostname}{"\t"}{.protocol}{"\t"}{.port}{"\n"}{end}' | sort -k2
the result:
https-api-019a994c api.xxx.xxx HTTPS 443
tls-api api.xxx.xxx TLS 443
how i resolved this (temporary)
because i don't know how to force TenantGateway not to include HTTPRoute,
so I delete the kubernetes-api in default namespace.
force TenantGateway reconcile the non-conflict Gateway.
and it works!
now i can access my services via Gateway API.
here's the list of bugs i found:
1. api.xxx.xxx HTTPS and TLS conflicted in the tenant-root's TenantGateway cozystack,
causing cilium-envoy refuses to accept and listen on the port.
i notice this by looking into cilium-envoy logs.
and actually this causes many other side effects, like port 80 failed to open for acme challenge.
2. dashboard.xxx.xxx HTTPRoute .status.parents reports HTTPRoute is not allowed to attach to this Gateway due to namespace selector restrictions,
in fact, the namespace cozy-dashboard already labelled with the expected label: namespace.cozystack.io/gateway: tenant-root.
but actually it's accessible after the cilium-envoy issue fixed, so it's not the root cause of causing connection failed.
3. cert-manager is started without - --enable-gateway-api=true, causing the HTTPRoute failed to get its own cert.
i manually add the flag to the deployment, and it's not being reconcile away.
i checked the packages/system/cert-manager/values.yaml, seems like no flag to turn this on.
hope my findings can help.
let me know if anything else that i can help.
@lexfrei @tym83
id 530991411here's the list of bugs i found:
1. api.xxx.xxx HTTPS and TLS conflicted in the tenant-root's TenantGateway cozystack,
causing cilium-envoy refuses to accept and listen on the port.
i notice this by looking into cilium-envoy logs.
and actually this causes many other side effects, like port 80 fail
@lexfrei this is his right nickname, your @Aleksei is a random guy not from this group. Please, say it to your agent😁
Timur Tukaev@lexfrei this is his right nickname, your @Aleksei is a random guy not from this group. Please, say it to your agent😁
hahaha. i see.
alright. will tag the right person.
id 530991411here's the list of bugs i found:
1. api.xxx.xxx HTTPS and TLS conflicted in the tenant-root's TenantGateway cozystack,
causing cilium-envoy refuses to accept and listen on the port.
i notice this by looking into cilium-envoy logs.
and actually this causes many other side effects, like port 80 fail
Thanks for debugging, we'll look at that next week. Could you send an issue?
Timur TukaevThanks for debugging, we'll look at that next week. Could you send an issue?
sure thing. i will open the issues.
11 August 2026
Hi guys,
I have a cozystack bare metal cluster (running on OVH) upgraded to 1.6.1 and then and I'm trying to run a Talos upgrade 1.13.0 -> 1.13.6 now but it seems I'm hitting a reboot wedge with the ZFS + DRBD/LINSTOR stack, and I'd like to know if it's a known issue? Here are some more details:
Full Setup: Talos v1.13.0 (ghcr.io/cozystack/cozystack/talos, upgrading to v1.13.6 via talm 0.34.0), kernel 6.18.24, extensions zfs 2.4.3 + drbd 9.3.2, LINSTOR storage on zvols.
Symptom: talm reboot (and the reboot phase of upgrades) never completes. The node was fully drained first; kubelet, etcd, trustd and cri all stop cleanly ("Service finished successfully"), then the sequence wedges on ext-zfs-service: stuck in "Stopping", machined sends SIGKILL and the task survives it (D-state / uninterruptible kernel I/O). MachineStatus stays stage: rebooting forever.
Kernel shows DRBD secondaries on the node hit local-IO errors as ZFS is torn down under them and detach one at a time — 12 of 15 volumes detached over ~20 min with an exponentially slowing tail, then total silence: zero kernel log lines for 25+ min, 3 volumes never detached. Looks like a teardown-ordering problem: the ZFS service is being stopped while DRBD still has the zvols open, and ext-zfs-service blocks in a zfs ioctl that can never complete.
The same cluster wedged the same way during my second attempt (that time on an undrained node it stuck at "stopping cri" first). Only exit both times: talm reboot -m force, which on our AMI boards then risks a POST hang from the dirty reset.
Is there a known-good teardown ordering (e.g. disconnecting DRBD before the zfs extension stops), or a fix planned? Happy to file a GitHub issue with full service states, kernel logs, and timelines — everything is captured.
I've been a pretty early adopted of Cozystack and this cluster is running for 245 days, having been upgraded several times up to 1.6.1 from as early as 0.40.x or so. Same goes for the respective Kubernetes and Tal
fenris-Hi guys,
I have a cozystack bare metal cluster (running on OVH) upgraded to 1.6.1 and then and I'm trying to run a Talos upgrade 1.13.0 -> 1.13.6 now but it seems I'm hitting a reboot wedge with the ZFS + DRBD/LINSTOR stack, and I'd like to know if it's a known issue? Here are some more details:
Fu
Yes, known issue, it's upstream: siderolabs/extensions#1085 matches your v1.13.0 signature exactly. Talos v1.13.0 added zpool export -a at shutdown with no timeout; on pools with many zvols it overruns the fixed 10s grace and gets SIGKILLed mid-export. Fixed in siderolabs/extensions#1104, shipped with the zfs extension since v1.13.4, so the v1.13.6 image you're upgrading to already has it. The catch: the wedge fires on the reboot out of 1.13.0, which still runs the old code.
To get through the upgrade: after draining, exec into the LINSTOR satellite pod on that node, run drbdadm down all, then reboot right away. With DRBD not holding the zvols the export finishes inside the grace, and the satellite recreates the resources after reboot.
Two open follow-ups for after the upgrade: siderolabs/extensions#1174 (shutdown can still wedge when a zvol is mounted and in use) and siderolabs/extensions#1167 (boot-time zpool import -fal has no timeout and can hang on hyperconverged DRBD nodes — we hit that one ourselves, probably what Timur remembered).
And yes, please file the issue with your logs. The "3 volumes never detach, no self-recovery after 25+ min" timeline is a data point the upstream reports don't have.
Aleksei SviridkinYes, known issue, it's upstream: siderolabs/extensions#1085 matches your v1.13.0 signature exactly. Talos v1.13.0 added zpool export -a at shutdown with no timeout; on pools with many zvols it overruns the fixed 10s grace and gets SIGKILLed mid-export. Fixed in siderolabs/extensions#1104, shipped wi
amazing thank you for the quick reply, I'll try it and let you know how it goes, meanwhile submitted https://github.com/siderolabs/extensions/issues/1193