Upgrading Talos Linux to v1.14.5 (N150 Included)
Nine nodes, two CPU architectures, one command per node. Upgrading the homelab from Talos v1.13.4 to v1.14.5 was about as boring as an OS upgrade should be.
Back in June I moved the cluster from Talos v1.12.7 to v1.13.4. This weekend it was time for v1.14.5. Short version: it was easy. The longest part of the whole thing was waiting for Longhorn to say every volume was healthy again before moving on to the next node.
This was also the first Talos upgrade since the GEEKOM N150 joined the cluster, so for the first time the upgrade covered two CPU architectures.
What got upgraded
Nine nodes in total:
- 3 control planes: Turing RK1s (arm64, 8 cores / 16 GB each)
- 5 workers: also Turing RK1s
- 1 amd64 worker:
talosn150, the Intel N150 box (4 cores / 16 GB) that runs the Minecraft Bedrock servers natively instead of under emulation
Going from v1.13.4 to v1.14.5 brought:
- Linux kernel 6.18.58
- containerd 2.3.7
- etcd 3.7.3
Kubernetes stayed on 1.36.1 for now. 1.37 has only been out a short while, and I'd rather let it settle and a few patch releases land before I move the homelab to it.
Before touching anything: backups
I took an etcd snapshot and zipped up the Talos configs, talosconfig and kubeconfig, and copied the zip off the cluster. Talos makes the snapshot a single command:
talosctl -n talostrk1 etcd snapshot etcd-backup.dbThis one matters more than usual. The etcd 3.7 storage format means that once the control planes are on 1.14, they can't roll back to 1.13. If something went badly wrong, the fix would be a restore, not a downgrade.
The upgrade itself
I build my node images with the Talos Image Factory, and the schematic adds iscsi-tools and util-linux-tools for Longhorn. Upgrading a node is one command that points at the new installer image:
talosctl -e talostrk2 -n talostrk1 upgrade \
--image factory.talos.dev/metal-installer/<schematic-id>:v1.14.5One gotcha: the cluster's API VIP lives on whichever control plane currently holds it. If you upgrade that node while using the VIP as your endpoint, you cut off your own connection partway through. So I always pass -e with a different control plane as the endpoint for the node I'm upgrading.
The order:
- Control planes one at a time, with the etcd leader last
- The RK1 workers one at a time
- The N150 last
Between nodes I watched Longhorn until every volume was back to healthy with both replicas in place:
kubectl -n longhorn-system get volumes.longhorn.io \
-o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustnessDepending on how much data had to resync, that took anywhere from 2 to 9 minutes per node. talosctl upgrade cordons and drains the node on its own, reboots it into the new image and brings it back, so there was nothing to babysit apart from the storage.
The N150
I expected the amd64 node to need special handling, and it didn't. Image Factory installer images are multi-arch, so the N150 uses the same schematic and the same command as the RK1s and pulls the amd64 image itself. It came back up in a few minutes and the Minecraft servers picked up right where they left off.
With a mixed arm64/amd64 cluster on Talos, an OS upgrade is the same procedure on every node. Nice to see that confirmed.
After the upgrade
Once all nine nodes were on v1.14.5 I ran a few sanity checks:
- Cluster DNS resolves from pods on several nodes
- Services answer across nodes (pod on one node to a service backed by another)
- The Kubernetes API still refuses TLS 1.2 connections
- All Longhorn volumes healthy, all nodes
Ready
kubectl get nodes
# NAME STATUS ROLES VERSION OS-IMAGE
# talosn150 Ready <none> v1.36.1 Talos (v1.14.5)
# talostrk1 Ready control-plane v1.36.1 Talos (v1.14.5)
# talostrk2 Ready <none> v1.36.1 Talos (v1.14.5)
# ...
# talostrk8 Ready <none> v1.36.1 Talos (v1.14.5)Last step was bumping talos_version in the Ansible group vars to v1.14.5, so any node built from the repo later starts out on the right version.
Takeaways
- Take the etcd snapshot. With a one-way storage upgrade, it's your only way back.
- Don't use the VIP as your endpoint while upgrading the node that holds it.
- Let storage catch up between nodes. If you hurry, you end up with two nodes down and volumes running on a single replica.
- A second CPU architecture adds no extra work.