home-dc-kubernetes, Part 2: Talos as the Kubernetes OS

Kubernetes has a dirty secret: it runs on top of a general-purpose Linux that was never meant to host it. SSH into most cluster nodes and you’ll find a full distro — package managers, cron daemons, a thousand ways for the node to drift from the image you intended.

Talos takes the opposite bet: the OS is Kubernetes. No SSH. No package manager. No shell. Every node is a minimal, immutable Linux that speaks the Kubernetes API and exposes exactly one management surface — talosctl. If you can’t configure it via the API, you can’t configure it at all. That’s not a limitation; it’s the whole point.

Where the config lives #

The app cluster is generated with talhelper from two files in talos/app/:

Version pins are explicit and Renovate-managed:

# renovate: datasource=docker depName=ghcr.io/siderolabs/installer
talosVersion: v1.13.7
# renovate: datasource=docker depName=ghcr.io/siderolabs/kubelet
kubernetesVersion: v1.36.3

Talos and Kubernetes versions move independently, both bumped by the same Renovate bot that updates everything else in the repo. One dependency surface, one upgrade story.

Three nodes, one virtual IP #

The app cluster is three control-plane nodes (k8s-ctrl-01..03), each a Proxmox VM. Two details make this interesting:

• MAC-pinned interfaces: each NIC is selected by hardwareAddr, so the VM’s MAC address determines its IP — no DHCP surprises, no interface name drift.
• A VIP for the API: 10.0.40.101 floats across the three nodes. The endpoint is the VIP, not any single node — which means the API stays up even when a control-plane node dies.

Every node also carries a second NIC on a separate 10.0.70.0/24 network — the storage network, kept off the main LAN. Pod and service CIDRs are the default Talos ranges (10.42.0.0/16, 10.43.0.0/16), and the built-in CNI is disabled from day one:

# Disable built-in CNI to use Cilium
cniConfig:
  name: none

Global patches — the interesting part #

Rather than repeating per-node config, everything shared lives in talos/app/patches/global/. These read like a list of “things I learned running this thing”:

That kubelet nodeIP patch is the kind of thing you only write after a debugging session you never want again. Without it, kubelet can pick the wrong interface and the node goes NotReady with a mystery you’ll chase for hours.

Secrets: SOPS + age, generated at bootstrap #

Cluster secrets never sit in the repo in plaintext. The flow is:

talhelper gensecret | sops --encrypt /dev/stdin > talsecret.sops.yaml

One command generates the certificate authority, the bootstrap token, and the service account keys — then encrypts them with age before they touch disk. The age key itself is a recovery input (Stage 0 in the rebuild runbook), because losing it means regenerating cluster identity, not decrypting it.

Bootstrap — one task, full cluster #

The whole bring-up is a single Taskfile task:

task bootstrap:talos
  → gensecret (if no talsecret)
  → talhelper genconfig
  → apply config to all nodes (--insecure)
  → talosctl bootstrap
  → fetch kubeconfig

With retries built into each step. From bare VMs to a working cluster in minutes, no interactive SSH session anywhere in the path.

Day-2 ops are API calls, not logins #

Because there’s no SSH, every operation is explicit:

• task talos:apply-node IP=10.0.40.90 — push a new machine config
• task talos:upgrade-node IP=10.0.40.90 — upgrade Talos on one node
• task talos:upgrade-k8s — upgrade Kubernetes
• task talos:reset — back to maintenance mode (with a prompt, because it destroys)

Upgrades are rolling: one node at a time, verify, next. The immutable base means an upgrade is a swap, not a mutation — and if a node misbehaves, reset puts it back in maintenance mode where the whole bootstrap flow can re-apply it.

The takeaway #

Talos removes the class of problems that come from treating nodes as pets with login shells. The config is data, the API is the interface, and every change is either in git or it doesn’t exist. That’s the foundation the rest of this series builds on — because GitOps only works when the nodes underneath are honest about their state.

Up next: Part 3 — Argo CD app-of-apps and the cluster-aware manifest layout. How one Argo instance on the app cluster manages both clusters, and why the repo’s directory structure is the real design document.