The OS Layer Your Platform Ignores

Immutable nodes for Kubernetes

Brian "bex" Exelbierd Principal Product Manager, Microsoft Azure 🌐 www.bexelbie.com ✉️ [email protected] ✉️ [email protected] @[email protected]

--- # You have 100 Kubernetes nodes. # Can you prove their filesystems match? --- # Mutable systems drift - Nodes at different lifecycle stages get updates at different times - Package upgrade ordering varies across a fleet - Node A got update X before Y; Node B got Y before X → different state - Debug sessions leave artifacts: tcpdump, strace, hot-patched binaries - Debug artifacts and daemonset writes fill disks, blow quotas --- # The divergence is invisible Two nodes report the same kernel and OS version. Their userspace has diverged. **You can't tell from the outside.** --- # Kubernetes can't check this Kubernetes solves **orchestration**, not **OS integrity**. - It knows: is the node Ready? Is kubelet responding? - It doesn't know: what's on the filesystem. What changed since last boot. - Node replacement from the "same image" doesn't help — mutable images mutate. --- # "Shift Down" --- # You secured your container images. ## What about the node they run on? - SLSA, Sigstore, SBOMs → container supply chain ✓ - But the node runs a **mutable filesystem** - Detecting drift ≠ preventing drift - What if drift were **structurally impossible**? --- # Flatcar Container Linux A container-optimized, immutable Linux distribution - **CNCF Incubating** project (CoreOS → Kinvolk → Microsoft → CNCF) - Made for one thing: running containers reliably - The OS itself is treated like a container image - A **functionality contract**: you define what you need, Flatcar delivers and maintains it --- # Immutable by design | | Mutable OS | Flatcar | |:--|:--|:--| | `/usr` | Read-write, anyone can change it | **Read-only**, dm-verity protected | | Updates | Individual packages, any order | Entire OS atomically, as one unit | | Verification | Package-level audit | VERSION_ID + sysext list = full identity | | After reboot | Same node, new mysteries | Cryptographically verified identical userspace | --- # Demo: Provision & Prove Immutability ## What we're provisioning ```yaml # Butane config → transpiled to Ignition JSON variant: flatcar version: 1.0.0 storage: files: - path: /opt/extensions/kubernetes/kubernetes-v1.33.2-x86-64.raw contents: source: https://extensions.flatcar.org/... links: - path: /etc/extensions/kubernetes.raw target: /opt/extensions/kubernetes/kubernetes-v1.33.2-x86-64.raw systemd: units: - name: kubeadm.service enabled: true contents: | ... ExecStart=/usr/bin/kubeadm join :6443 --token ... ``` --- # Demo: Provision & Prove Immutability ## Deploying on Azure ```bash az vm create \ --resource-group kcd-flatcar-demo \ --name flatcar-worker-2 \ --image kinvolk:flatcar-container-linux-free:stable-gen2:4593.2.0 \ --size Standard_B2ms \ --admin-username core \ --custom-data worker.ign ``` Same command. Same image. Same config. Same result. --- # [Live demo on pre-staged node] --- # A/B Partition Updates ``` ┌──────────────┐ ┌──────────────┐ │ Partition A │ │ Partition B │ │ (running) │ │ (staging) │ └──────────────┘ └──────────────┘ ↕ reboot ↕ ``` - Update image written to inactive partition — running system untouched - Reboot activates the new OS - Rollback = reboot to old partition - **No intermediate states** — it works or it rolls back --- # Demo: OS Update with Kubernetes Lifecycle --- # systemd-sysext: Extend Without Breaking How do you add software to an immutable OS? - **System extensions** overlay files onto `/usr` at runtime - Immutable images — composable, versioned, independently updatable - Three tiers: - **Official (shipped)**: containerd, Docker, OEM tools — part of the base image - **Official (opt-in)**: NVIDIA drivers, Python, ZFS — enable in config - **Community (bakery)**: kubernetes, tailscale, nvidia-runtime — from extensions.flatcar.org --- # Component lifecycle with sysupdate Each sysext has its own update track — independent of the OS ```ini # /etc/sysupdate.kubernetes.d/kubernetes-v1.33.transfer [Source] Type=url-file Path=https://extensions.flatcar.org/extensions/kubernetes/ MatchPattern=kubernetes-v1.33.@v-%a.raw [Target] Path=/opt/extensions/kubernetes MatchPattern=kubernetes-v1.33.@v-%a.raw CurrentSymlink=/etc/extensions/kubernetes.raw ``` - `systemd-sysupdate` checks for new patch versions on a timer - Downloads the new sysext image, updates the symlink - Same atomic pattern as the OS — but per component --- # Demo: Kubelet Sysext Upgrade --- # What this gives your platform team - **Provable userspace**: VERSION_ID proves `/usr` (dm-verity). Sysext symlinks prove components. Two checks, full identity. - **Supply chain integrity**: Boot → OS → sysexts → workload. All verified. - **At scale today** (not future promises): - **Cluster API** provisions Flatcar node fleets — same Ignition mechanism, automated - **FLUO / Kured** coordinates rolling OS updates across the cluster - **systemd-sysupdate** keeps sysext versions current automatically - **Next**: `systemd-confext` brings the same overlay pattern to `/etc` — immutable config --- # Thank You **Visit Flatcar** → flatcar.org

Brian "bex" Exelbierd Principal Product Manager, Microsoft Azure 🌐 www.bexelbie.com ✉️ [email protected] ✉️ [email protected] @[email protected]

---