nabeem@sre:~$ running…

Namespace Bootstrap Operator

Onboarding a tenant namespace is a checklist nobody remembers in full. This turns it into one resource - and then keeps the result correct, correcting drift on a watch event rather than at the next deploy.

controller-runtime crd golang kube-builder kubernetes kubernetes-operator namespace operator-sdk platform-engineering resourcequota
Role
Creator & maintainer
Stack
Go · Kubebuilder · controller-runtime
Generates
Namespace · Quota · Limits · NetworkPolicy · PSA
License
Apache 2.0

Problem

Giving a team its own namespace sounds like one command. It isn't. You create the namespace, cap its share of the cluster with a ResourceQuota, give containers sane defaults with a LimitRange so that cap doesn't reject every pod that forgot to declare requests, fence the network with NetworkPolicies, and set a Pod Security level. Miss one and the namespace is either unusable or unbounded.

The deeper problem is what happens afterwards. Every one of those objects is independently editable. Someone debugging a failing deploy raises a quota "just for today". A chart upgrade drops a NetworkPolicy. A well-meaning kubectl label relaxes a Pod Security level. None of it shows up anywhere, because the thing that created the namespace finished running weeks ago.

Why not a Helm chart and ArgoCD

This is the first question worth answering, because a chart templating those same five objects gets you to an identical initial state, and ArgoCD will even tell you when it drifts.

The difference is what "drift" means to each system. ArgoCD compares a whole Application against a Git revision on a polling interval, and auto-sync resolves it by reapplying the manifests. An operator watches the individual objects: a deleted ResourceQuota produces a watch event and is rebuilt in the next reconcile, typically inside a second, with no sync window and no commit. Deleting a managed object is just extreme drift and takes the same code path as editing one.

It also composes with ArgoCD rather than competing. Ship NamespaceProfile objects through ArgoCD and you get GitOps over intent plus continuous enforcement of outcome - so I wrote a Lua health check that makes ArgoCD report a profile's real state instead of "Synced" the moment the CR lands.

The resource

One cluster-scoped CR describes the whole namespace. Everything optional is genuinely optional; the operator only owns fields you set.

apiVersion: platform.nabeem.com/v1alpha1
kind: NamespaceProfile
metadata:
  name: payments-profile
spec:
  namespace: payments
  quota:
    requests:
      cpu: "8"
      memory: 16Gi
    limits:
      cpu: "16"
      memory: 32Gi
    pods: 50
    objectCounts:
      services: 10
  limitRange:
    default:
      cpu: 500m
      memory: 512Mi
    max:
      cpu: "4"
      memory: 8Gi
  network:
    defaultDenyIngress: true
    allowDNS: true
  security:
    podSecurityStandard: restricted

Validation lives in the API server, not the controller: 21 CEL rules, expanding to 36 in the generated schema, reject a bad profile at kubectl apply rather than letting it become a status condition nobody reads. Two of them had to be rewritten after exceeding CEL's cost budget - the namespace allow-lists use plain MaxLength and Pattern item constraints instead, which cost nothing to evaluate.

Architecture

The operator owns no new machinery. Everything it relies on - admission, quota enforcement, network policy, Pod Security - is already in Kubernetes; what the operator adds is a single place to declare the intent and a loop that keeps reality matching it.

A NamespaceProfile is applied to the kube-apiserver, passing schema defaulting and 36 CEL validation rules before reaching etcd. The operator pod watches four kinds through an informer cache; Reconcile combines that with desired state from pure builders in internal/resources, then writes the Namespace, ResourceQuota and LimitRange by server-side apply and the NetworkPolicies by create or update. The admission plugins read the quota, limit range and namespace labels to admit or reject pods; the CNI plugin reads the NetworkPolicies to allow or drop traffic.
Where the work actually happens. The operator writes five kinds of object and then gets out of the way - it is the API server's admission plugins and the CNI that enforce them, not the controller.

That split matters. The operator is never in the request path of a pod being created: if the operator is down, existing quotas, limit ranges and policies keep being enforced exactly as before. What stops while it is down is drift correction, not enforcement.

Inside the loop, the shape is deliberately boring - one flat Reconcile, idempotent and level-triggered, with the builders kept pure (spec in, desired objects out, no client and no context) so the interesting logic is testable without a cluster.

A watch event on a profile, namespace, quota, limit range or network policy maps to profiles via the spec.namespace field index, enters a workqueue that dedupes by key, and the profile is fetched. Not found drops metrics and returns; being deleted finalizes; a live profile has its finalizer ensured, then ownership is checked. If another profile owns the namespace it reports a conflict and touches nothing. Otherwise it ensures namespace, quota, limit range and network policies; if nothing differs there is no write at all, and if something differs it applies, counts drift and emits an event. Status is patched only when it changed.
One reconcile. Two properties the diagram is drawn to make obvious: nothing is written when nothing differs, and a profile that does not own the namespace does nothing at all.

The two dead ends are the load-bearing parts. No write when nothing differs is what prevents the classic operator failure mode - a status write producing a watch event, producing a reconcile, producing another status write - so a settled cluster is genuinely silent. And a profile that loses the ownership race does not retry: a conflict between two profiles naming the same namespace is a human's decision, not something exponential backoff can fix.

Drift: correct it, or refuse it

By default the operator corrects. Each managed object is watched; an edit or a deletion wakes the controller, which rebuilds the object from the profile and increments nsprofile_drift_corrections_total. A climbing count is the signal worth alerting on - it means something is repeatedly fighting the operator. The operator always wins, so nothing breaks; it simply never settles.

There is also an opt-in mode that prevents drift instead. Two ValidatingAdmissionPolicy objects make the generated resources and the namespace's labels refuse edits at admission, so the change is rejected rather than corrected. That is strictly better where you can afford it - nothing transiently wrong, no correction loop - and it carries a real lockout risk, so the policies exempt the operator's own service account and that exemption is covered by an end-to-end test which deliberately tries to break it.

Security model

An operator that configures security for other namespaces is itself a tempting target, so the interesting question is what it can do if someone gets a shell in its pod. The answer is deliberately dull.

ResourceVerbs it holds
namespaces, resourcequotas, limitrangescreate, delete, get, list, patch, watch
networkpoliciescreate, delete, get, list, update, watch
namespaceprofilesget, list, patch, watch
namespaceprofiles/status, /finalizersget, patch · update
eventscreate, patch

That is the whole ClusterRole. No pods, no workloads, no secrets, no configmaps, no RBAC objects - it cannot read a Secret in a namespace it just created, and it cannot grant anyone permissions. Enforcement is delegated entirely to the Kubernetes mechanisms a profile configures.

list and watch appear even for resources only ever read by name, because the manager's client is cache-backed and a single Get causes an informer to LIST and WATCH that type cluster-wide. patch is what server-side apply uses. update is granted only for NetworkPolicies, the one kind written with create/update rather than apply.

Notably it cannot create, update or delete a NamespaceProfile. A controller able to delete the objects describing its own intent could erase its instructions and then report that everything is fine.

The one dangerous permission

delete on namespaces cascades to Deployments, StatefulSets, PersistentVolumeClaims and their data - none of which this operator manages or could restore. It is reachable only through four independent gates, all of which must hold:

  1. lifecycle.deleteNamespaceOnRemoval is explicitly true.
  2. The annotation platform.nabeem.com/confirm-namespace-deletion contains the namespace name, typed out in full. A bare boolean is too easy to copy between profiles; requiring the name means a confirmation cannot be accidentally correct for the wrong namespace.
  3. The namespace carries this profile's ownership labels.
  4. The namespace was created by the operator, not adopted. The adoption marker is sticky - once adopted, permanently undeletable.

Any gate failing retains the namespace and records which one in an event. None of the checks list Pods: an "is it empty?" test would need cluster-wide Pod read access for a weaker signal than a human typing a name, since an empty namespace today may be a busy one tomorrow.

Anti-lockout

The optional admission policies refuse edits to the managed resources - including, if written naively, the operator's own. They exempt its service account, and because that exemption is the difference between "drift is prevented" and "nothing can ever be reconciled again", an end-to-end test deliberately tries to break it: it probes until a write from outside is actually refused, then confirms the operator can still reconcile.

Engineering decisions

Server-side apply everywhere except NetworkPolicies. SSA gives per-key ownership of maps, so labels written by other controllers survive a reconcile untouched. NetworkPolicies are the exception: spec.ingress is omitempty, so SSA cannot express "this list is empty" - a default-deny policy could never prune a rule somebody added by hand. Those use Create and Update with optimistic concurrency instead.

A finalizer, deliberately not owner references. Cleanup is a decision rather than a fixed object graph: whether to delete the namespace depends on lifecycle settings, a confirmation annotation, and whether the namespace was adopted rather than created. Owner references express none of that - and wiring a namespace into garbage collection would make deleting a profile capable of deleting live workloads.

Ownership is arbitrated deterministically. Two profiles naming the same namespace would otherwise take turns, each rewriting the other's work and both reporting success. An existing claim wins; otherwise the oldest profile wins with the name as tie-break, because creation timestamps have one-second granularity and ties are real. A profile being deleted is excluded, which is what lets a namespace pass from a retiring profile to its replacement with nobody intervening.

Deleting a namespace needs four independent gates - the lifecycle flag, an annotation containing the namespace name typed out in full, this profile's ownership labels, and the namespace not having been adopted. Any one failing retains the namespace and says which in an event. None of the checks read Pods: an "is it empty?" test would need cluster-wide Pod read access for a weaker signal than a human typing a name, since an empty namespace today may be a busy one tomorrow.

Trade-offs

Least privilege over convenience. The operator has no access to pods, workloads, secrets, configmaps, or any RBAC object. Enforcement is left to the Kubernetes mechanisms a profile configures. It cannot create, update or delete a NamespaceProfile either - a controller able to delete the objects describing its intent could erase its own instructions.

Correction is visible, not silent. Every correction emits an event and moves a metric, which is noisier than quietly winning - and that noise is the product. Drift you can't see is drift you'll rediscover during an incident.

Adoption is one-way. A pre-existing namespace can be adopted, but the marker is sticky: once adopted it can never be deleted by the operator, even if the lifecycle settings later say otherwise. It didn't create that namespace and cannot know what else depends on it.

Challenges

A silent infinite drift loop. The API server defaults Container entries in a LimitRange - filling in default and defaultRequest from max. The operator then read its own object back, saw fields it hadn't written, called it drift, and corrected it forever. Nothing failed; it just never stopped. The fix was to apply the same defaulting before comparing, and the bug was caught by an idempotency test rather than in review.

Optional fields with non-zero defaults must be pointers. A plain string field serialises its zero value explicitly, which bypasses the schema default and then fails the field's own pattern. Caught by a test that validates the shipped sample manifests against the real CRD.

Namespace handover was blocked by a stale ownership label: a deleted profile's label made the namespace refuse its intended successor, which had to be resolved in the controller rather than the builders, since by then ownership had already been arbitrated.

A chart-drift guard that couldn't fail. The check asserting the Helm chart matched the generated RBAC only verified that resource names appeared - so when I tightened the operator's own permissions, the chart kept the old over-granted verbs and the guard passed anyway. It now compares the rules rule by rule, and I verified it by re-introducing the exact drift it had missed.

Testing

Four layers, because they catch different things. Table-driven unit tests cover the pure builders - spec in, desired objects out, no I/O - at 98% statement coverage. envtest runs the controller against a real API server, which is what surfaces API-server defaulting and admission behaviour. A manager-driven integration suite exercises the actual watches and predicates. Finally 26 end-to-end specs run on a Kind cluster against the published Helm chart under the operator's real ClusterRole - the only layer that proves the RBAC is sufficient, since envtest runs as admin and never enforces it.

Release pipeline

One workflow. A pull request runs the full matrix and a multi-arch build that publishes nothing; merging to main publishes. The ordering is the point: the image is pushed by digest, untagged, scanned at that digest, cosign-signed keylessly, SBOM-attested and SLSA-provenanced, and only then tagged :version and :latest. A build that fails any step stays a dangling digest no tag advertises.

Published artifacts are immutable because the publishing IAM role has no s3:DeleteObject - the permission is absent rather than merely unused, so CI cannot remove a release even by mistake. Version, commit and build date are linked into the binary, so a running operator reports what it is in its first log line and in a build_info metric; the image tag is not a reliable answer once anything is deployed from :latest.

Current limitations

  • NetworkPolicy enforcement is your CNI's job. The operator creates correct policies; whether they are enforced depends on the cluster's network plugin, so that is deliberately not claimed.
  • One namespace per profile. No label-selector fan-out across many namespaces yet.
  • Quota keys are recorded, not inferred. Because objectCounts and extendedResources are open key spaces, the operator tracks which keys it wrote in an annotation so it can relinquish a key you removed - rather than guessing.
  • CEL has a cost budget, and this API is near it. Further validation may have to move to a webhook.

The repository is the source of truth - issues and ideas welcome.