nabeem@sre:~$ running…

Bare Metal K8s

Netboot a set of blank "bare-metal" machines into a running k3s cluster - one command, fully automated, reproducible - and do it under UEFI Secure Boot with keys I mint myself. Blank disks on the wire → kubectl get nodes shows them Ready, with a boot chain the firmware cryptographically trusts end to end.

Arm64 · UEFI PXE / netboot UEFI HTTP Boot GRUB shim UEFI Secure Boot Own CA · PK/KEK/db AAVMF sbsign · SBAT libvirt / KVM dnsmasq nginx cloud-init · autoinstall Terraform Ansible k3s AWS a1.metal
Role
Author
Host
AWS a1.metal (Graviton)
Boot
UEFI HTTP → shim → GRUB
Secure Boot
my own CA (Purist)
Cost
~$0.41 / hr

Problem

This is how real datacenters and bare-metal fleets provision servers at scale: a blank machine powers on, asks the network who it is and what to boot, and installs itself - no OS image, no USB stick, no human clicking through an installer. But the full modern chain - DHCP → UEFI HTTP Boot → bootloader → unattended install → cluster join - is rarely shown end to end, and the security layer on top of it (UEFI Secure Boot) is almost always left as "trust Microsoft's keys and move on."

I wanted to build the whole thing myself and own the root of trust: not just enable Secure Boot, but become the sole certificate authority for the entire boot chain - mint the keys, sign every link, and prove the firmware rejects anything I didn't sign.

Why I built it

To understand netbooting and Secure Boot from first principles by making them work, not reading about them - and to have a single command that goes from nothing to a cryptographically-trusted Kubernetes cluster. The organising idea is a trust chain I own end to end: the firmware trusts only my key, so every stage - shim, GRUB, kernel - must carry my signature, or it doesn't run.

It touches the core of SRE/infra work in one project: UEFI netbooting, UEFI Secure Boot internals, infrastructure-as-code (Terraform), configuration management (Ansible), and Kubernetes (k3s).

Architecture

The twist - where it runs

Real netboot needs physical machines on a LAN, and AWS deliberately won't let you PXE-boot EC2 instances - VPC networking blocks the broadcast DHCP netboot relies on, and Nitro owns the boot path. There's also no nested virtualisation on standard instances. So the whole lab runs as KVM virtual machines inside a single AWS bare-metal host (a1.metal, Arm/Graviton - the smallest, cheapest metal and UEFI-native). The VMs are "bare metal" from the boot's point of view: no disk, real UEFI firmware, they netboot - but the DHCP/boot traffic lives on an internal libvirt bridge the VPC never sees.

The boot path

A node powers on with an empty disk and its virtual UEFI firmware (AAVMF) does UEFI HTTP Boot. dnsmasq answers DHCP with an IP and a boot URL; nginx serves the bootloader, Ubuntu ISO, kernel/initrd, and the autoinstall seed over HTTP. The firmware loads my signed shim, which loads my signed GRUB, which loads my signed kernel; Ubuntu autoinstall images the disk hands-off and powers off; the node flips to disk boot; and Ansible installs k3s (control-plane on cp, joins w1).

Why GRUB and not iPXE/pxelinux: pxelinux is x86-BIOS only (Arm has no legacy BIOS), and iPXE mangles the autoinstall datasource (it splits on the ; in ds=nocloud-net;s=…). GRUB is UEFI-native, quotes it correctly, and is the bootloader the Secure-Boot chain uses anyway.

Security model - become your own CA

Default Secure Boot trusts Microsoft's keys. Here the firmware's trust database holds only my certificate - no vendor keys at all (the "Purist" model) - so the entire chain must trace back to me:

  • Own key hierarchy - I mint the UEFI PK (platform key), KEK, and db myself and enroll them into each VM's firmware varstore offline, with Secure Boot switched on - no firmware-menu clicking, fully reproducible.
  • Every link signed by me - the firmware verifies my re-signed shim against db; shim (which provides the shim_lock protocol GRUB needs) verifies my GRUB; GRUB verifies my kernel via shim_lock - all against the one cert I own.
  • Self-contained GRUB - every GRUB module is baked into one signed image (Secure Boot forbids loading unsigned modules at runtime), plus a valid SBAT section or shim refuses it.
  • Disk boot too - after install, the node's on-disk shim is replaced with my re-signed one, so it boots from disk under my db as well (stock Ubuntu's GRUB + kernel chain off it via shim's built-in vendor cert).
  • Proven both ways - positively (mokutil --sb-state → SecureBoot enabled, mokutil --db shows only my cert) and negatively (a scripted test serves an unsigned bootloader and the firmware refuses it with a Security Policy Violation).
  • Blast radius - SSH is locked to a single IP; the netboot network is internal to the host and never touches the VPC.

Engineering decisions

Enforce locally, with one command

A SECURE_BOOT toggle builds the full signed chain by default, or a plain unsigned GRUB for teaching/comparison - so the exact difference between "trust everything" and "trust only me" is one flag and visible side by side.

Firmware is a file, so enroll keys offline

Each VM's UEFI varstore (AAVMF) is a host file. Rather than click through a firmware menu, I edit it offline - strip the vendor keys out of an initialised template and write in my own - so "arm every node with my CA + Secure Boot on" is a scripted, deterministic step.

shim stays in the chain

Ubuntu's GRUB won't boot a kernel under Secure Boot without the shim_lock protocol, so I keep shim - but re-signed with my key - purely to provide that protocol. My GRUB and kernel are then trusted via my db; stock Ubuntu's disk binaries ride behind the same shim via its built-in vendor cert.

Reproducible host provisioning

Terraform stands up the a1.metal host and installs the whole toolchain via cloud-init (the single source of truth for packages); the on-host script only verifies the toolchain is present, so it also self-provisions on a bare homelab box. A runaway-cost guard auto-stops the instance after a set time.

Trade-offs

DecisionGainedGave up
Nested KVM VMs on one metal hostReal netboot + nested virt with zero owned gear; the VPC never sees the boot trafficNot literally separate physical servers
Purist (db = my cert only)Sole root of trust - no vendor keys anywhereMust also re-sign the installed node's on-disk shim (vs. trusting a vendor CA)
Arm a1.metalSmallest, cheapest metal; UEFI-nativeArm-only artifacts; older Graviton capacity varies by region
k3sTrivial multi-node join; auto-detects arm64Not a full upstream kubeadm cluster
Ephemeral labSpin up → prove → tear down; cheapNot long-lived; nodes get new keys each rebuild

The host is ~$0.41/hr on-demand and the design is "spin up → prove it → tear down," so a focused session is well under a dollar to a few dollars. A cloud-init timer auto-stops a forgotten box.

Challenges

  • AAVMF uses UEFI HTTP Boot, not classic TFTP PXE. The firmware sends DHCP client-arch 19 and expects a full HTTP URL in option 67 plus HTTPClient echoed in option 60 - otherwise it rejects the offer.
  • The blank varstore can't be enrolled directly. Ubuntu's AAVMF_VARS.fd ships headerless, so the enroll tool crashes on it; I start from an already-initialised template and strip its keys instead.
  • SBAT generation. shim refused my GRUB with a Security Policy Violation until its SBAT generation met shim's required level - the signature was already trusted; SBAT is a separate gate.
  • The kernel is compressed. The arm64 vmlinuz is a gzip'd Image, not a PE - sbsign rejects it until it's decompressed to the raw EFI-stub Image.
  • Keeping a long run alive. The build is a 20–40 min unattended job; it runs in tmux on the host so an SSH drop or a laptop sleeping can't kill it.

Quick start

One-time setup, then one command on the host.

bash
# from your laptop - rent the a1.metal host + copy the repo over
$ make host-up && make wait
$ make sync && make ssh

# on the host (in tmux - it's a ~20-40 min unattended job)
$ sudo make lab-up          # netboot → signed chain → autoinstall → k3s
$ export KUBECONFIG=ansible/kubeconfig && kubectl get nodes -o wide
$ ssh cp 'mokutil --sb-state'   # → SecureBoot enabled (your own CA)

# prove enforcement, then tear it all down
$ sudo bash secureboot/negative-test.sh
$ make host-down

SECURE_BOOT=0 sudo make lab-up builds the same cluster with Secure Boot off, for comparison. Full write-up - concepts, the fetch-by-fetch runtime trace, and every hard-won gotcha - lives in the repo.

Current limitations

  • Nested, not literally separate servers - KVM VMs on one metal host stand in for physical machines (the netboot behaves identically, but it's one box).
  • Purist costs an extra signing step - because db trusts only my cert, each installed node's on-disk shim must be re-signed at install time; fine for an ephemeral lab, not for in-place kernel upgrades.
  • Static node roster - nodes are MAC-pinned and listed in the inventory (deliberate, for a reproducible demo) rather than auto-discovered.
  • One control-plane + one worker, single host/region - sized for a clear, affordable demo, not HA.

Future roadmap

  • Dynamic node discovery - drop the MAC pins to a DHCP range and have each node self-register (fetch the k3s token on first boot), the MAAS / Tinkerbell / Metal³ pattern.
  • TPM attestation - gate the join token on a vTPM measured-boot quote, so a node must prove it booted my signed chain before it's trusted with cluster membership (enforce locally + prove remotely).
  • More workers / ArgoCD bootstrap - scale the roster and bootstrap GitOps on first boot.

Full detail, diagrams, and the Secure Boot concept guide live in the repo README and secureboot/ docs.