Netboot a set of blank "bare-metal" machines into a running k3s cluster - one command, fully automated, reproducible - and do it under UEFI Secure Boot with keys I mint myself. Blank disks on the wire → kubectl get nodes shows them Ready, with a boot chain the firmware cryptographically trusts end to end.
This is how real datacenters and bare-metal fleets provision servers at scale: a blank machine powers on, asks the network who it is and what to boot, and installs itself - no OS image, no USB stick, no human clicking through an installer. But the full modern chain - DHCP → UEFI HTTP Boot → bootloader → unattended install → cluster join - is rarely shown end to end, and the security layer on top of it (UEFI Secure Boot) is almost always left as "trust Microsoft's keys and move on."
I wanted to build the whole thing myself and own the root of trust: not just enable Secure Boot, but become the sole certificate authority for the entire boot chain - mint the keys, sign every link, and prove the firmware rejects anything I didn't sign.
To understand netbooting and Secure Boot from first principles by making them work, not reading about them - and to have a single command that goes from nothing to a cryptographically-trusted Kubernetes cluster. The organising idea is a trust chain I own end to end: the firmware trusts only my key, so every stage - shim, GRUB, kernel - must carry my signature, or it doesn't run.
It touches the core of SRE/infra work in one project: UEFI netbooting, UEFI Secure Boot internals, infrastructure-as-code (Terraform), configuration management (Ansible), and Kubernetes (k3s).
Real netboot needs physical machines on a LAN, and AWS deliberately won't let you PXE-boot EC2 instances - VPC networking blocks the broadcast DHCP netboot relies on, and Nitro owns the boot path. There's also no nested virtualisation on standard instances. So the whole lab runs as KVM virtual machines inside a single AWS bare-metal host (a1.metal, Arm/Graviton - the smallest, cheapest metal and UEFI-native). The VMs are "bare metal" from the boot's point of view: no disk, real UEFI firmware, they netboot - but the DHCP/boot traffic lives on an internal libvirt bridge the VPC never sees.
A node powers on with an empty disk and its virtual UEFI firmware (AAVMF) does UEFI HTTP Boot. dnsmasq answers DHCP with an IP and a boot URL; nginx serves the bootloader, Ubuntu ISO, kernel/initrd, and the autoinstall seed over HTTP. The firmware loads my signed shim, which loads my signed GRUB, which loads my signed kernel; Ubuntu autoinstall images the disk hands-off and powers off; the node flips to disk boot; and Ansible installs k3s (control-plane on cp, joins w1).
Why GRUB and not iPXE/pxelinux: pxelinux is x86-BIOS only (Arm has no legacy BIOS), and iPXE mangles the autoinstall datasource (it splits on the ; in ds=nocloud-net;s=…). GRUB is UEFI-native, quotes it correctly, and is the bootloader the Secure-Boot chain uses anyway.
Default Secure Boot trusts Microsoft's keys. Here the firmware's trust database holds only my certificate - no vendor keys at all (the "Purist" model) - so the entire chain must trace back to me:
PK (platform key), KEK, and db myself and enroll them into each VM's firmware varstore offline, with Secure Boot switched on - no firmware-menu clicking, fully reproducible.db; shim (which provides the shim_lock protocol GRUB needs) verifies my GRUB; GRUB verifies my kernel via shim_lock - all against the one cert I own.db as well (stock Ubuntu's GRUB + kernel chain off it via shim's built-in vendor cert).mokutil --sb-state → SecureBoot enabled, mokutil --db shows only my cert) and negatively (a scripted test serves an unsigned bootloader and the firmware refuses it with a Security Policy Violation).A SECURE_BOOT toggle builds the full signed chain by default, or a plain unsigned GRUB for teaching/comparison - so the exact difference between "trust everything" and "trust only me" is one flag and visible side by side.
Each VM's UEFI varstore (AAVMF) is a host file. Rather than click through a firmware menu, I edit it offline - strip the vendor keys out of an initialised template and write in my own - so "arm every node with my CA + Secure Boot on" is a scripted, deterministic step.
Ubuntu's GRUB won't boot a kernel under Secure Boot without the shim_lock protocol, so I keep shim - but re-signed with my key - purely to provide that protocol. My GRUB and kernel are then trusted via my db; stock Ubuntu's disk binaries ride behind the same shim via its built-in vendor cert.
Terraform stands up the a1.metal host and installs the whole toolchain via cloud-init (the single source of truth for packages); the on-host script only verifies the toolchain is present, so it also self-provisions on a bare homelab box. A runaway-cost guard auto-stops the instance after a set time.
| Decision | Gained | Gave up |
|---|---|---|
| Nested KVM VMs on one metal host | Real netboot + nested virt with zero owned gear; the VPC never sees the boot traffic | Not literally separate physical servers |
| Purist (db = my cert only) | Sole root of trust - no vendor keys anywhere | Must also re-sign the installed node's on-disk shim (vs. trusting a vendor CA) |
Arm a1.metal | Smallest, cheapest metal; UEFI-native | Arm-only artifacts; older Graviton capacity varies by region |
| k3s | Trivial multi-node join; auto-detects arm64 | Not a full upstream kubeadm cluster |
| Ephemeral lab | Spin up → prove → tear down; cheap | Not long-lived; nodes get new keys each rebuild |
The host is ~$0.41/hr on-demand and the design is "spin up → prove it → tear down," so a focused session is well under a dollar to a few dollars. A cloud-init timer auto-stops a forgotten box.
HTTPClient echoed in option 60 - otherwise it rejects the offer.AAVMF_VARS.fd ships headerless, so the enroll tool crashes on it; I start from an already-initialised template and strip its keys instead.vmlinuz is a gzip'd Image, not a PE - sbsign rejects it until it's decompressed to the raw EFI-stub Image.tmux on the host so an SSH drop or a laptop sleeping can't kill it.One-time setup, then one command on the host.
# from your laptop - rent the a1.metal host + copy the repo over $ make host-up && make wait $ make sync && make ssh # on the host (in tmux - it's a ~20-40 min unattended job) $ sudo make lab-up # netboot → signed chain → autoinstall → k3s $ export KUBECONFIG=ansible/kubeconfig && kubectl get nodes -o wide $ ssh cp 'mokutil --sb-state' # → SecureBoot enabled (your own CA) # prove enforcement, then tear it all down $ sudo bash secureboot/negative-test.sh $ make host-down
SECURE_BOOT=0 sudo make lab-up builds the same cluster with Secure Boot off, for comparison. Full write-up - concepts, the fetch-by-fetch runtime trace, and every hard-won gotcha - lives in the repo.
db trusts only my cert, each installed node's on-disk shim must be re-signed at install time; fine for an ephemeral lab, not for in-place kernel upgrades.Full detail, diagrams, and the Secure Boot concept guide live in the repo README and secureboot/ docs.