Onboarding a tenant namespace is a checklist nobody remembers in full. This turns it into one resource - and then keeps the result correct, correcting drift on a watch event rather than at the next deploy.
Giving a team its own namespace sounds like one command. It isn't. You create the namespace, cap its share of the cluster with a ResourceQuota, give containers sane defaults with a LimitRange so that cap doesn't reject every pod that forgot to declare requests, fence the network with NetworkPolicies, and set a Pod Security level. Miss one and the namespace is either unusable or unbounded.
The deeper problem is what happens afterwards. Every one of those objects is independently editable. Someone debugging a failing deploy raises a quota "just for today". A chart upgrade drops a NetworkPolicy. A well-meaning kubectl label relaxes a Pod Security level. None of it shows up anywhere, because the thing that created the namespace finished running weeks ago.
This is the first question worth answering, because a chart templating those same five objects gets you to an identical initial state, and ArgoCD will even tell you when it drifts.
The difference is what "drift" means to each system. ArgoCD compares a whole Application against a Git revision on a polling interval, and auto-sync resolves it by reapplying the manifests. An operator watches the individual objects: a deleted ResourceQuota produces a watch event and is rebuilt in the next reconcile, typically inside a second, with no sync window and no commit. Deleting a managed object is just extreme drift and takes the same code path as editing one.
It also composes with ArgoCD rather than competing. Ship NamespaceProfile objects through ArgoCD and you get GitOps over intent plus continuous enforcement of outcome - so I wrote a Lua health check that makes ArgoCD report a profile's real state instead of "Synced" the moment the CR lands.
One cluster-scoped CR describes the whole namespace. Everything optional is genuinely optional; the operator only owns fields you set.
apiVersion: platform.nabeem.com/v1alpha1
kind: NamespaceProfile
metadata:
name: payments-profile
spec:
namespace: payments
quota:
requests:
cpu: "8"
memory: 16Gi
limits:
cpu: "16"
memory: 32Gi
pods: 50
objectCounts:
services: 10
limitRange:
default:
cpu: 500m
memory: 512Mi
max:
cpu: "4"
memory: 8Gi
network:
defaultDenyIngress: true
allowDNS: true
security:
podSecurityStandard: restricted
Validation lives in the API server, not the controller: 21 CEL rules, expanding to 36 in the generated schema, reject a bad profile at kubectl apply rather than letting it become a status condition nobody reads. Two of them had to be rewritten after exceeding CEL's cost budget - the namespace allow-lists use plain MaxLength and Pattern item constraints instead, which cost nothing to evaluate.
The operator owns no new machinery. Everything it relies on - admission, quota enforcement, network policy, Pod Security - is already in Kubernetes; what the operator adds is a single place to declare the intent and a loop that keeps reality matching it.
That split matters. The operator is never in the request path of a pod being created: if the operator is down, existing quotas, limit ranges and policies keep being enforced exactly as before. What stops while it is down is drift correction, not enforcement.
Inside the loop, the shape is deliberately boring - one flat Reconcile, idempotent and level-triggered, with the builders kept pure (spec in, desired objects out, no client and no context) so the interesting logic is testable without a cluster.
The two dead ends are the load-bearing parts. No write when nothing differs is what prevents the classic operator failure mode - a status write producing a watch event, producing a reconcile, producing another status write - so a settled cluster is genuinely silent. And a profile that loses the ownership race does not retry: a conflict between two profiles naming the same namespace is a human's decision, not something exponential backoff can fix.
By default the operator corrects. Each managed object is watched; an edit or a deletion wakes the controller, which rebuilds the object from the profile and increments nsprofile_drift_corrections_total. A climbing count is the signal worth alerting on - it means something is repeatedly fighting the operator. The operator always wins, so nothing breaks; it simply never settles.
There is also an opt-in mode that prevents drift instead. Two ValidatingAdmissionPolicy objects make the generated resources and the namespace's labels refuse edits at admission, so the change is rejected rather than corrected. That is strictly better where you can afford it - nothing transiently wrong, no correction loop - and it carries a real lockout risk, so the policies exempt the operator's own service account and that exemption is covered by an end-to-end test which deliberately tries to break it.
An operator that configures security for other namespaces is itself a tempting target, so the interesting question is what it can do if someone gets a shell in its pod. The answer is deliberately dull.
| Resource | Verbs it holds |
|---|---|
| namespaces, resourcequotas, limitranges | create, delete, get, list, patch, watch |
| networkpolicies | create, delete, get, list, update, watch |
| namespaceprofiles | get, list, patch, watch |
| namespaceprofiles/status, /finalizers | get, patch · update |
| events | create, patch |
That is the whole ClusterRole. No pods, no workloads, no secrets, no configmaps, no RBAC objects - it cannot read a Secret in a namespace it just created, and it cannot grant anyone permissions. Enforcement is delegated entirely to the Kubernetes mechanisms a profile configures.
list and watch appear even for resources only ever read by name, because the manager's client is cache-backed and a single Get causes an informer to LIST and WATCH that type cluster-wide. patch is what server-side apply uses. update is granted only for NetworkPolicies, the one kind written with create/update rather than apply.
Notably it cannot create, update or delete a NamespaceProfile. A controller able to delete the objects describing its own intent could erase its instructions and then report that everything is fine.
delete on namespaces cascades to Deployments, StatefulSets, PersistentVolumeClaims and their data - none of which this operator manages or could restore. It is reachable only through four independent gates, all of which must hold:
lifecycle.deleteNamespaceOnRemoval is explicitly true.platform.nabeem.com/confirm-namespace-deletion contains the namespace name, typed out in full. A bare boolean is too easy to copy between profiles; requiring the name means a confirmation cannot be accidentally correct for the wrong namespace.Any gate failing retains the namespace and records which one in an event. None of the checks list Pods: an "is it empty?" test would need cluster-wide Pod read access for a weaker signal than a human typing a name, since an empty namespace today may be a busy one tomorrow.
The optional admission policies refuse edits to the managed resources - including, if written naively, the operator's own. They exempt its service account, and because that exemption is the difference between "drift is prevented" and "nothing can ever be reconciled again", an end-to-end test deliberately tries to break it: it probes until a write from outside is actually refused, then confirms the operator can still reconcile.
Server-side apply everywhere except NetworkPolicies. SSA gives per-key ownership of maps, so labels written by other controllers survive a reconcile untouched. NetworkPolicies are the exception: spec.ingress is omitempty, so SSA cannot express "this list is empty" - a default-deny policy could never prune a rule somebody added by hand. Those use Create and Update with optimistic concurrency instead.
A finalizer, deliberately not owner references. Cleanup is a decision rather than a fixed object graph: whether to delete the namespace depends on lifecycle settings, a confirmation annotation, and whether the namespace was adopted rather than created. Owner references express none of that - and wiring a namespace into garbage collection would make deleting a profile capable of deleting live workloads.
Ownership is arbitrated deterministically. Two profiles naming the same namespace would otherwise take turns, each rewriting the other's work and both reporting success. An existing claim wins; otherwise the oldest profile wins with the name as tie-break, because creation timestamps have one-second granularity and ties are real. A profile being deleted is excluded, which is what lets a namespace pass from a retiring profile to its replacement with nobody intervening.
Deleting a namespace needs four independent gates - the lifecycle flag, an annotation containing the namespace name typed out in full, this profile's ownership labels, and the namespace not having been adopted. Any one failing retains the namespace and says which in an event. None of the checks read Pods: an "is it empty?" test would need cluster-wide Pod read access for a weaker signal than a human typing a name, since an empty namespace today may be a busy one tomorrow.
Least privilege over convenience. The operator has no access to pods, workloads, secrets, configmaps, or any RBAC object. Enforcement is left to the Kubernetes mechanisms a profile configures. It cannot create, update or delete a NamespaceProfile either - a controller able to delete the objects describing its intent could erase its own instructions.
Correction is visible, not silent. Every correction emits an event and moves a metric, which is noisier than quietly winning - and that noise is the product. Drift you can't see is drift you'll rediscover during an incident.
Adoption is one-way. A pre-existing namespace can be adopted, but the marker is sticky: once adopted it can never be deleted by the operator, even if the lifecycle settings later say otherwise. It didn't create that namespace and cannot know what else depends on it.
A silent infinite drift loop. The API server defaults Container entries in a LimitRange - filling in default and defaultRequest from max. The operator then read its own object back, saw fields it hadn't written, called it drift, and corrected it forever. Nothing failed; it just never stopped. The fix was to apply the same defaulting before comparing, and the bug was caught by an idempotency test rather than in review.
Optional fields with non-zero defaults must be pointers. A plain string field serialises its zero value explicitly, which bypasses the schema default and then fails the field's own pattern. Caught by a test that validates the shipped sample manifests against the real CRD.
Namespace handover was blocked by a stale ownership label: a deleted profile's label made the namespace refuse its intended successor, which had to be resolved in the controller rather than the builders, since by then ownership had already been arbitrated.
A chart-drift guard that couldn't fail. The check asserting the Helm chart matched the generated RBAC only verified that resource names appeared - so when I tightened the operator's own permissions, the chart kept the old over-granted verbs and the guard passed anyway. It now compares the rules rule by rule, and I verified it by re-introducing the exact drift it had missed.
Four layers, because they catch different things. Table-driven unit tests cover the pure builders - spec in, desired objects out, no I/O - at 98% statement coverage. envtest runs the controller against a real API server, which is what surfaces API-server defaulting and admission behaviour. A manager-driven integration suite exercises the actual watches and predicates. Finally 26 end-to-end specs run on a Kind cluster against the published Helm chart under the operator's real ClusterRole - the only layer that proves the RBAC is sufficient, since envtest runs as admin and never enforces it.
One workflow. A pull request runs the full matrix and a multi-arch build that publishes nothing; merging to main publishes. The ordering is the point: the image is pushed by digest, untagged, scanned at that digest, cosign-signed keylessly, SBOM-attested and SLSA-provenanced, and only then tagged :version and :latest. A build that fails any step stays a dangling digest no tag advertises.
Published artifacts are immutable because the publishing IAM role has no s3:DeleteObject - the permission is absent rather than merely unused, so CI cannot remove a release even by mistake. Version, commit and build date are linked into the binary, so a running operator reports what it is in its first log line and in a build_info metric; the image tag is not a reliable answer once anything is deployed from :latest.
objectCounts and extendedResources are open key spaces, the operator tracks which keys it wrote in an annotation so it can relinquish a key you removed - rather than guessing.The repository is the source of truth - issues and ideas welcome.