Skip to content
About Me
----

Multi-Cluster GitOps with ArgoCD & ApplicationSets: Enterprise Fleet Management

Architect Key Takeaways

Production guide to managing multi-cluster Kubernetes fleets using ArgoCD and ApplicationSets, covering Git repository topologies, dynamic generators, SyncWaves, automated drift remediation, and secret integration via External Secrets Operator.

GitOps establishes Git as the single source of truth (SSOT) for declarative infrastructure and applications. When organizations scale from a single Kubernetes cluster to dozens of clusters across multiple regions and clouds, managing individual ArgoCD applications becomes unmaintainable.

ArgoCD ApplicationSets solve this by automating the dynamic creation, updating, and deletion of ArgoCD Applications across an entire fleet of Kubernetes clusters.

Multi-Cluster GitOps Architecture with ArgoCD & ApplicationSets


1. Multi-Cluster Architecture: Hub & Spoke Model

Section titled “1. Multi-Cluster Architecture: Hub & Spoke Model”

In an enterprise environment, ArgoCD is deployed using a centralized Hub-and-Spoke control plane topology:

  1. Management Hub Cluster: Runs the ArgoCD control plane (Argo Server, Application Controller, Repo Server, ApplicationSet Controller, Dex SSO).
  2. Spoke Target Clusters (dev, staging, prod-us-east-1, prod-eu-west-1): Registered in ArgoCD via Kubernetes Service Account kubeconfigs stored as cluster Secrets in the Hub.
  3. Network Isolation: The Hub only requires outbound network connectivity to the Spoke Kubernetes API servers (port 443). Spoke clusters do not require inbound internet access.

Historically, teams chained root applications to child applications using the “App-of-Apps” pattern. ApplicationSets provide a major architectural upgrade:

Capability App-of-Apps Pattern ArgoCD ApplicationSets
Creation Mechanism Manually writing individual Application manifests Templated generation via dynamic generators
Multi-Cluster Scaling Requires duplicating manifests per cluster 1 ApplicationSet generates apps for 100+ clusters dynamically
Lifecycle Deletion Deleting child app often leaves orphaned resources Cascade deletion automatically cleans up removed clusters
Cluster Discovery Hardcoded cluster URLs Automated discovery via labels on cluster secrets

3. Production ApplicationSet: The Cluster Generator Pattern

Section titled “3. Production ApplicationSet: The Cluster Generator Pattern”

The Cluster Generator targets clusters registered in ArgoCD based on metadata labels, injecting values like {{name}}, {{server}}, and {{metadata.labels.env}} into the application template.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: platform-fleet-ingress
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
tier: production
fleet: primary
template:
metadata:
name: '{{name}}-ingress-nginx'
spec:
project: platform-engineering
source:
repoURL: 'https://github.com/enterprise-org/fleet-infrastructure.git'
targetRevision: HEAD
path: 'apps/ingress-nginx/overlays/{{metadata.labels.env}}'
destination:
server: '{{server}}'
namespace: ingress-nginx
syncPolicy:
automated:
prune: true # Automatically remove resources deleted from Git
selfHeal: true # Automatically revert unauthorized manual kubectl edits
syncOptions:
- CreateNamespace=true
- ApplyOutOfSyncOnly=true

3.2 Matrix Generator: Combining Clusters & Git Directories

Section titled “3.2 Matrix Generator: Combining Clusters & Git Directories”

For advanced scenarios where each cluster hosts multiple microservices mapped to Git subdirectories:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: microservices-matrix
namespace: argocd
spec:
generators:
- matrix:
generators:
# Generator 1: Discover all clusters
- clusters:
selector:
matchLabels:
type: workload
# Generator 2: Discover all service directories in Git
- git:
repoURL: 'https://github.com/enterprise-org/services.git'
revision: HEAD
directories:
- path: 'services/*'
template:
metadata:
name: '{{name}}-{{path.basename}}'
spec:
project: default
source:
repoURL: 'https://github.com/enterprise-org/services.git'
targetRevision: HEAD
path: '{{path}}'
destination:
server: '{{server}}'
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true

4. Controlled Deployment Sequencing: ArgoCD SyncWaves

Section titled “4. Controlled Deployment Sequencing: ArgoCD SyncWaves”

When provisioning infrastructure, dependencies must deploy in strict chronological order (e.g., Namespaces ➔ CRDs ➔ Secrets/ConfigMaps ➔ StatefulSets/Databases ➔ Backend Deployments ➔ Ingress).

ArgoCD uses the argocd.argoproj.io/sync-wave annotation. Waves are executed in ascending numerical order, waiting for the previous wave’s resources to achieve Healthy status before starting the next wave.

# Step 1: PreSync Database Schema Migration Hook (Wave -2)
apiVersion: batch/v1
kind: Job
metadata:
name: db-schema-migration
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/sync-wave: "-2"
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: enterprise/db-migrator:v2.4
---
# Step 2: Core Microservice Deployment (Wave 0)
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
replicas: 3
template:
spec:
containers:
- name: payments
image: enterprise/payments:v2.4
---
# Step 3: Ingress Routing (Wave 2 - Deployed only after pods are healthy)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: payments-ingress
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
rules:
- host: payments.enterprise.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: payments-api
port:
number: 80

5. Enterprise Secret Management: External Secrets Operator (ESO)

Section titled “5. Enterprise Secret Management: External Secrets Operator (ESO)”

Plain-text Kubernetes Secrets should never be committed to Git. Instead, commit declarative ExternalSecret manifests that pull encrypted credentials dynamically from AWS Secrets Manager or HashiCorp Vault inside the target cluster.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: payments-db-credentials
namespace: payments
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
refreshInterval: "1h"
secretStoreRef:
name: aws-secrets-manager-store
kind: ClusterSecretStore
target:
name: payments-db-secret # Native Kubernetes Secret created in-cluster
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD
remoteRef:
key: "production/payments/rds"
property: "password"

6. SRE Best Practices for Fleet-Wide GitOps

Section titled “6. SRE Best Practices for Fleet-Wide GitOps”
  1. Enforce selfHeal: true & prune: true: Prevents configuration drift caused by engineers running ad-hoc kubectl edit commands on production clusters.
  2. Implement Argo Rollouts for Progressive Delivery: Combine GitOps with Canary deployments, automated metrics analysis (Prometheus error rate queries), and instant rollbacks.
  3. Use Branch Protection & Signed Commits: Protect Git branches with required PR approvals, Cosign signature validation, and Policy-as-Code checks before merging.
  4. Segregate Secrets Management: Leverage External Secrets Operator or Sealed Secrets so that Git repositories remain 100% free of plaintext credentials.