EKS GitOps Platform

Problem

Self-managed Kubernetes carries heavy operational toil - control-plane and etcd upgrades, config drift between environments, manual kubectl apply deploys, and a scattered story for observability, secrets, and policy. Standardising on managed EKS removes some of that, but a running cluster is only the first 10%.

Most references stop exactly there: "here's an EKS cluster." What's missing is everything that makes it a platform - how apps get deployed and reconciled, how you see metrics/logs/traces, how nodes scale, how you back up and restore, how policy is enforced, and what it all costs. Wiring those ~30 components together correctly is the actual work, and it's rarely shown end to end.

Why I built it

I wanted a concrete, opinionated, working example of how a real EKS + GitOps platform is assembled - closer to production complexity than a tutorial, but with the cost and shortcuts made explicit so you know exactly what to harden before taking it further.

The organising principle is GitOps as the single source of truth: Terraform provisions the AWS substrate, and from there everything - the app, monitoring, autoscaling, logging, tracing, backup, and policy - reconciles declaratively from this one repo via ArgoCD. Nothing is applied by hand; the cluster converges to what's in Git and self-heals when it drifts.

Architecture

Provisioning (Terraform)

A root module wires a VPC (2 public + 2 private subnets across 2 AZs) and an EKS 1.34 cluster. The managed node group is tainted app=core and runs only platform components; Karpenter provisions separate app=workload nodes on demand, so a crash-looping or noisy app can't destabilise the control-plane components that manage the cluster. Terraform also creates every IRSA role, the S3 buckets (Thanos / Loki / Velero / Tempo), and the Secrets Manager secrets - then hands off to GitOps through ArgoCD cluster-secret annotations, so the manifests carry no hardcoded account IDs, bucket names, or ARNs and the same repo works across clusters.

GitOps (ArgoCD, App-of-Apps)

ArgoCD is self-managed via Helm. A single root Application recursively syncs the repo and acts as a stable, self-healing anchor - you kubectl apply it to a fresh cluster and it takes over. (A plain ApplicationSet root would be circular: the generator must already be running before it can generate the apps that install it.) Each subsystem is then an ApplicationSet, ordered with sync-waves so CRDs land before the controllers that need them and workloads land last. The layers:

• Networking - AWS Load Balancer Controller (ALB Ingress) + ExternalDNS (Route53).
• Storage - EBS CSI Driver with gp3 as the default StorageClass.
• Secrets - External Secrets Operator + ClusterSecretStore syncing from Secrets Manager, with Reloader rolling deployments when a secret changes.
• Autoscaling - Cluster Autoscaler (managed group) + Karpenter (on-demand workload nodes) + metrics-server + HPA, with Prometheus Adapter for custom-metric HPA.
• Monitoring - kube-prometheus-stack, Thanos shipping to S3 for long-term retention, Grafana with a pre-built dashboard.
• Logging - Loki (S3-backed) + Fluent Bit, queryable in Grafana.
• Tracing - a two-tier OTel Collector (DaemonSet agent → Deployment gateway) into Grafana Tempo (S3-backed), with trace-to-log correlation through Loki.
• Alerting - PrometheusRules + Alertmanager routed to Slack.
• Backup - Velero backing up resources + EBS snapshots to S3 on a daily schedule.
• Security - Kyverno admission control + Policy Reporter UI.

The demo workload, SimpleTimeService (a tiny Python service returning timestamp + caller IP), ships as a Helm chart complete with Ingress, HPA, PodDisruptionBudget, NetworkPolicy, and ServiceMonitor - so it exercises the whole platform, not just "hello world."

Security model

• IRSA for every workload - each controller assumes its own least-privilege IAM role; no node-wide credentials.
• Bounded secret access - values live in AWS Secrets Manager, synced in by the External Secrets Operator via a ClusterSecretStore; the operator's IRSA policy grants GetSecretValue only on secrets tagged ExternalSecret=true, so a compromised ESO can't read arbitrary account secrets - just the opted-in set.
• Hardened pods - security contexts run non-root, read-only root filesystem, all Linux capabilities dropped.
• Admission policy - four Kyverno ClusterPolicy rules (resource limits, no privileged containers, no :latest, non-root) evaluate every pod and write PolicyReport objects (Audit mode by default, surfaced in Policy Reporter).
• Shift-left scanning - Trivy (images) and Checkov (Terraform) run in CI before anything ships.
• Least-privilege GitOps - six ArgoCD AppProjects (bootstrap / namespaces / platform / observability / security / workloads) whitelist the repos, destination namespaces, and cluster-scoped kinds each may touch - so workloads can't deploy into kube-system or mint a ClusterRoleBinding, and drift in one concern can't cascade into another.
• Data safety by default - the gp3 StorageClass is encrypted and uses reclaimPolicy: Retain, so an accidental prune or kubectl delete pvc leaves the EBS volume intact (a human must delete the PV) instead of silently destroying Prometheus/Thanos data.
• Network posture - private subnets; ALB Ingress is HTTPS with HTTP→HTTPS redirect; Prometheus and Alertmanager have no ingress at all (port-forward only), since Alertmanager's API would otherwise let an unauthenticated caller silence alerts.

Engineering decisions

App-of-Apps + ApplicationSets

One root app owns the platform; ApplicationSets template the ~30 components. Adding or changing a component is a Git commit, and ArgoCD reconciles and self-heals it - no imperative install steps to drift.

Sync-waves for a safe boot order

Components are interdependent, so waves sequence them: CRDs at wave 0, controllers next, workloads last. Prometheus Operator CRDs are a separate app so they can be upgraded safely without churning the whole stack.

Self-managed Prometheus + Thanos, not AMP

The full kube-prometheus-stack (Alertmanager, rules, dashboards) lives in Git and stays cloud-portable; Thanos ships blocks to S3 with tiered downsampling (raw 30d → 5m 90d → 1h 180d), which at this cardinality is materially cheaper than AMP's per-sample pricing - and AMP needs a self-managed Alertmanager anyway, so going half-managed buys little.

Karpenter + Cluster Autoscaler together

They operate at different levels: Cluster Autoscaler scales the managed node groups (minutes, ASG-coupled - a safety net for pinned/system workloads); Karpenter provisions right-sized nodes outside any group in seconds, bin-packs, and handles spot interruption natively. Karpenter does the fast, flexible app scaling; CA holds baseline capacity.

Everything durable goes to S3 - in separate buckets

Thanos, Loki, Tempo, and Velero each get their own S3 bucket (11 nines, pay-per-byte, survives cluster/AZ loss). Separate buckets give per-workload IRSA scoping, independent lifecycle/retention, and separate audit trails - a single shared bucket would need brittle prefix-based IAM.

Fluent Bit over Fluentd

The log agent is a DaemonSet on every node, so footprint compounds: Fluent Bit (C) idles at ~450 KB vs Fluentd's (Ruby) ~40 MB, and starts fast - which matters when Karpenter churns nodes and a slow agent means a window of dropped logs per launch.

Two-tier telemetry, not app→Tempo

A DaemonSet OTel agent takes each pod's spans node-locally (no cross-node hop; the app SDK stays fire-and-forget); a central gateway owns the durable buffering and the single Tempo connection - so a Tempo restart or backpressure is absorbed in one place, not propagated to every node.

Two entry paths

An app-bootstrap module gets a service up on a public NLB with one terraform apply (no ArgoCD) for a fast look; the full GitOps path provisions infra only and lets ArgoCD own the rest.

Trade-offs

Running continuously costs roughly $310–330/month at idle (on-demand, ap-south-1), $400–500 under load - documented up front, with a one-command teardown so at rest only S3/Route53/Secrets Manager accrue (< $10/mo).

Challenges

• Ordering ~30 interdependent components. Getting the sync-wave sequence right (CRDs → controllers → workloads), including isolating Prometheus CRDs for safe upgrades, was the crux of a clean first sync.
• Tearing a stateful platform down cleanly. The cleanup script runs six ordered steps - Kubernetes resources, leftover ALB/NLB/ELBs, EBS volumes + snapshots, draining versioned S3 buckets, Kyverno CRDs, then terraform destroy - because half-deleted LBs and un-emptied buckets otherwise block Terraform.
• ACM before ALB. The ALB controller discovers its certificate by hostname match, so a validated wildcard ACM cert has to exist before bootstrap - a sequencing gotcha worth documenting.
• Secrets that flow and roll. Wiring Secrets Manager → ESO → Kubernetes Secrets, with Reloader rolling the dependent deployments automatically on change, so rotation is a re-sync rather than a manual restart.
• Keeping the demo honest. Documenting cost and the deliberate shortcuts (Audit-mode policy, public Grafana, single cluster) so the boundary to production is never fuzzy.

Screenshots

Installation example

Two paths - a fast look, or the full platform.

GitOps path (the whole platform)

# 1. Slack webhook is required; admin passwords auto-generate if omitted
$ export TF_VAR_alertmanager_slack_webhook_url="https://hooks.slack.com/..."

# 2. Bootstrap - provisions infra, installs ArgoCD via Helm, applies projects + root app
$ bash terraform/scripts/bootstrap.sh

# 3. Watch ArgoCD reconcile the platform (first sync takes ~3 min)
$ kubectl get applications -n argocd -w

Once synced, services come up at argocd.platform.<domain>, grafana.platform.<domain>, and simple-time-service.platform.<domain> (Prometheus / Alertmanager / Policy Reporter via kubectl port-forward).

Bootstrap path (fastest)

# one apply provisions VPC + EKS + the Helm release, on a public NLB - no ArgoCD
$ cd terraform/app-bootstrap && terraform init && terraform apply
$ aws eks update-kubeconfig --region ap-south-1 --name simple-eks
$ curl $(terraform output -raw application_url)

A validated wildcard ACM certificate for *.platform.<domain> must exist before bootstrap.sh - the ALB controller discovers it by hostname match. Tear everything down with bash terraform/scripts/cleanup.sh.

Current limitations

• A reference, not a drop-in production platform - a working, opinionated example with the shortcuts documented, not something to point at prod untouched.
• Kyverno runs in Audit mode by default - policies report violations but don't block until you flip to Enforce.
• Single NAT gateway - saves ~$33/mo but is an AZ-level single point of failure for private-subnet egress (prod wants one per AZ).
• Public EKS API endpoint and Prometheus Operator webhooks/TLS disabled for bootstrap reliability - both are documented shortcuts to re-lock for production.
• NetworkPolicy off by default - the chart ships one, but east-west isolation needs the VPC CNI policy controller enabled first.
• Loki runs SingleBinary (replication 1) and single cluster / region - fine for the reference, not HA.
• Non-trivial cost if left running - tear it down when idle.

Future roadmap

The production-hardening directions (tracked in docs/security.md):

• Flip Kyverno policies from Audit to Enforce and expand the rule set.
• Tighten network policies and remove demo-only public exposure (private ArgoCD / Grafana behind SSO).
• Progressive delivery (Argo Rollouts - canary / blue-green) for the workloads.
• Cost controls - Karpenter consolidation and spot capacity, right-sized retention.
• Multi-cluster / multi-region and DR exercises building on the existing Velero backups.

Full detail lives in the repo's design-decisions and security docs.