Multi-Cluster GitOps with ArgoCD & ApplicationSets: Enterprise Fleet Management
Production guide to managing multi-cluster Kubernetes fleets using ArgoCD and ApplicationSets, covering Git repository topologies, dynamic generators, SyncWaves, automated drift remediation, and secret integration via External Secrets Operator.
GitOps establishes Git as the single source of truth (SSOT) for declarative infrastructure and applications. When organizations scale from a single Kubernetes cluster to dozens of clusters across multiple regions and clouds, managing individual ArgoCD applications becomes unmaintainable.
ArgoCD ApplicationSets solve this by automating the dynamic creation, updating, and deletion of ArgoCD Applications across an entire fleet of Kubernetes clusters.
1. Multi-Cluster Architecture: Hub & Spoke Model
Section titled “1. Multi-Cluster Architecture: Hub & Spoke Model”In an enterprise environment, ArgoCD is deployed using a centralized Hub-and-Spoke control plane topology:
- Management Hub Cluster: Runs the ArgoCD control plane (Argo Server, Application Controller, Repo Server, ApplicationSet Controller, Dex SSO).
- Spoke Target Clusters (
dev,staging,prod-us-east-1,prod-eu-west-1): Registered in ArgoCD via Kubernetes Service Account kubeconfigs stored as cluster Secrets in the Hub. - Network Isolation: The Hub only requires outbound network connectivity to the Spoke Kubernetes API servers (port 443). Spoke clusters do not require inbound internet access.
2. App-of-Apps Pattern vs ApplicationSets
Section titled “2. App-of-Apps Pattern vs ApplicationSets”Historically, teams chained root applications to child applications using the “App-of-Apps” pattern. ApplicationSets provide a major architectural upgrade:
| Capability | App-of-Apps Pattern | ArgoCD ApplicationSets |
|---|---|---|
| Creation Mechanism | Manually writing individual Application manifests |
Templated generation via dynamic generators |
| Multi-Cluster Scaling | Requires duplicating manifests per cluster | 1 ApplicationSet generates apps for 100+ clusters dynamically |
| Lifecycle Deletion | Deleting child app often leaves orphaned resources | Cascade deletion automatically cleans up removed clusters |
| Cluster Discovery | Hardcoded cluster URLs | Automated discovery via labels on cluster secrets |
3. Production ApplicationSet: The Cluster Generator Pattern
Section titled “3. Production ApplicationSet: The Cluster Generator Pattern”The Cluster Generator targets clusters registered in ArgoCD based on metadata labels, injecting values like {{name}}, {{server}}, and {{metadata.labels.env}} into the application template.
3.1 Fleet ApplicationSet Manifest
Section titled “3.1 Fleet ApplicationSet Manifest”apiVersion: argoproj.io/v1alpha1kind: ApplicationSetmetadata: name: platform-fleet-ingress namespace: argocdspec: generators: - clusters: selector: matchLabels: tier: production fleet: primary template: metadata: name: '{{name}}-ingress-nginx' spec: project: platform-engineering source: repoURL: 'https://github.com/enterprise-org/fleet-infrastructure.git' targetRevision: HEAD path: 'apps/ingress-nginx/overlays/{{metadata.labels.env}}' destination: server: '{{server}}' namespace: ingress-nginx syncPolicy: automated: prune: true # Automatically remove resources deleted from Git selfHeal: true # Automatically revert unauthorized manual kubectl edits syncOptions: - CreateNamespace=true - ApplyOutOfSyncOnly=true3.2 Matrix Generator: Combining Clusters & Git Directories
Section titled “3.2 Matrix Generator: Combining Clusters & Git Directories”For advanced scenarios where each cluster hosts multiple microservices mapped to Git subdirectories:
apiVersion: argoproj.io/v1alpha1kind: ApplicationSetmetadata: name: microservices-matrix namespace: argocdspec: generators: - matrix: generators: # Generator 1: Discover all clusters - clusters: selector: matchLabels: type: workload # Generator 2: Discover all service directories in Git - git: repoURL: 'https://github.com/enterprise-org/services.git' revision: HEAD directories: - path: 'services/*' template: metadata: name: '{{name}}-{{path.basename}}' spec: project: default source: repoURL: 'https://github.com/enterprise-org/services.git' targetRevision: HEAD path: '{{path}}' destination: server: '{{server}}' namespace: '{{path.basename}}' syncPolicy: automated: prune: true selfHeal: true4. Controlled Deployment Sequencing: ArgoCD SyncWaves
Section titled “4. Controlled Deployment Sequencing: ArgoCD SyncWaves”When provisioning infrastructure, dependencies must deploy in strict chronological order (e.g., Namespaces ➔ CRDs ➔ Secrets/ConfigMaps ➔ StatefulSets/Databases ➔ Backend Deployments ➔ Ingress).
ArgoCD uses the argocd.argoproj.io/sync-wave annotation. Waves are executed in ascending numerical order, waiting for the previous wave’s resources to achieve Healthy status before starting the next wave.
# Step 1: PreSync Database Schema Migration Hook (Wave -2)apiVersion: batch/v1kind: Jobmetadata: name: db-schema-migration annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/sync-wave: "-2" argocd.argoproj.io/hook-delete-policy: HookSucceededspec: template: spec: restartPolicy: Never containers: - name: migrate image: enterprise/db-migrator:v2.4---# Step 2: Core Microservice Deployment (Wave 0)apiVersion: apps/v1kind: Deploymentmetadata: name: payments-api annotations: argocd.argoproj.io/sync-wave: "0"spec: replicas: 3 template: spec: containers: - name: payments image: enterprise/payments:v2.4---# Step 3: Ingress Routing (Wave 2 - Deployed only after pods are healthy)apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: payments-ingress annotations: argocd.argoproj.io/sync-wave: "2"spec: rules: - host: payments.enterprise.com http: paths: - path: / pathType: Prefix backend: service: name: payments-api port: number: 805. Enterprise Secret Management: External Secrets Operator (ESO)
Section titled “5. Enterprise Secret Management: External Secrets Operator (ESO)”Plain-text Kubernetes Secrets should never be committed to Git. Instead, commit declarative ExternalSecret manifests that pull encrypted credentials dynamically from AWS Secrets Manager or HashiCorp Vault inside the target cluster.
apiVersion: external-secrets.io/v1beta1kind: ExternalSecretmetadata: name: payments-db-credentials namespace: payments annotations: argocd.argoproj.io/sync-wave: "-1"spec: refreshInterval: "1h" secretStoreRef: name: aws-secrets-manager-store kind: ClusterSecretStore target: name: payments-db-secret # Native Kubernetes Secret created in-cluster creationPolicy: Owner data: - secretKey: DB_PASSWORD remoteRef: key: "production/payments/rds" property: "password"6. SRE Best Practices for Fleet-Wide GitOps
Section titled “6. SRE Best Practices for Fleet-Wide GitOps”- Enforce
selfHeal: true&prune: true: Prevents configuration drift caused by engineers running ad-hockubectl editcommands on production clusters. - Implement Argo Rollouts for Progressive Delivery: Combine GitOps with Canary deployments, automated metrics analysis (Prometheus error rate queries), and instant rollbacks.
- Use Branch Protection & Signed Commits: Protect Git branches with required PR approvals, Cosign signature validation, and Policy-as-Code checks before merging.
- Segregate Secrets Management: Leverage External Secrets Operator or Sealed Secrets so that Git repositories remain 100% free of plaintext credentials.