Skip to main content
< All Topics

EKS External Secrets Operator

Posted
Updated
Author -Rajkumar Aute
Views10

The External Secrets Operator (ESO) has rapidly become the de facto standard for Kubernetes secret management by elegantly solving the “secret zero” problem.

Core Architecture & Custom Resources (CRDs) ESO operates on a clear separation of concerns, dividing how to access secrets (the “Store”) from what to fetch (the “Secret”). It introduces three primary Custom Resource Definitions (CRDs):

  • SecretStore (Namespaced): Defines connection details and authentication mechanisms for external providers (e.g., AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault). Its namespaced scope ensures isolation, making it ideal for multi-tenant environments where different teams require secure, dedicated backends.+1
  • ClusterSecretStore (Global): Functions identically to a SecretStore but operates globally across the entire cluster. This allows central platform teams to manage a single external provider efficiently for the entire organization.
  • ExternalSecret (Namespaced): Acts as the blueprint. It references a specific store to dictate exactly which keys to fetch, the continuous refresh interval, and the precise formatting of the resulting native Kubernetes Secret.

Secure Authentication Methods in ESO When connecting ESO to external providers like AWS, operating with the principle of least privilege is paramount.

  • IAM Roles for Service Accounts (IRSA): This is the gold standard for AWS EKS. ESO runs using a Kubernetes Service Account linked directly to an AWS IAM Role via OIDC. ESO assumes the role dynamically requiring zero static credentials in your cluster.
  • Access Key ID & Secret Access Key: While supported, storing static credentials inside a Kubernetes Secret defeats the purpose of an external secret manager and is discouraged for production.
  • Cross-Account IAM Role Chaining: For complex environments, ESO can use IRSA to authenticate in your EKS account (Account A) and then assume a secondary role in your Secrets Manager account (Account B).

Advanced Features & Capabilities of ESO

The External Secrets Operator (ESO) extends far beyond a simple 1-to-1 mapping of external credentials to Kubernetes Secrets. It serves as a comprehensive credential orchestration engine, equipped with advanced features designed for modern DevSecOps workflows.

  • Dynamic Templating and Data Transformation: External secrets are frequently stored as flat strings or raw JSON blobs. However, legacy and specialized applications often require specific configuration file formats, such as config.yaml or .properties. ESO bridges this gap using a built-in Go-templating engine. It can fetch raw data from AWS, parse JSON on the fly, extract specific fields, and dynamically inject them into a formatted template before saving it as a Kubernetes Secret. Additional helper functions allow for inline Base64 decoding and extracting certificates from PKCS12 archives.
  • Continuous Synchronization & Auto-Rotation: By defining a refreshInterval (e.g., 15m or 1h) in the ExternalSecret manifest, the ESO controller will continuously poll the external provider. If a database password is rotated upstream in AWS Secrets Manager, ESO immediately detects the change and updates the native Kubernetes Secret. When combined with hot-reloading tools like Stakater Reloader, applications can pick up newly rotated passwords with zero downtime.
  • Generators for Ephemeral Secrets: Moving away from static, long-lived credentials is a core DevSecOps principle. ESO natively supports generating temporary, short-lived credentials on the fly. It can dynamically provision Vault tokens, AWS STS tokens, or highly secure random passwords, automatically destroying and regenerating them the moment they expire.
  • PushSecrets (Two-Way Synchronization): While ESO is primarily configured to pull secrets into a cluster, it also supports a PushSecret CRD. This facilitates reverse synchronization, allowing you to take natively generated Kubernetes Secrets such as dynamically minted TLS certificates from cert-manager and push them out to AWS Secrets Manager. This ensures vital cluster-generated material is securely backed up and accessible to infrastructure outside the Kubernetes environment.

The ESO Secret Lifecycle and Dynamic Templating

Understanding how the External Secrets Operator (ESO) manages the lifecycle of a secret is crucial for mastering Kubernetes security. ESO automates the entire process from cloud retrieval to pod consumption.

The 6-Step Secret Lifecycle Workflow

  1. Configuration: A cluster administrator creates a ClusterSecretStore pointing to a backend like AWS Secrets Manager, securely authenticating using IAM Roles for Service Accounts (IRSA).
  2. Definition: A developer deploys an ExternalSecret manifest targeting a specific secret path (e.g., /prod/db/password).
  3. Reconciliation: The ESO controller detects the new ExternalSecret, identifies the referenced ClusterSecretStore, and assumes the designated IAM role.
  4. Retrieval: ESO queries the external API (AWS Secrets Manager) to fetch the secret payload.
  5. Creation: ESO generates a native Kubernetes Secret object containing the retrieved data. It applies a creationPolicy: Owner reference, ensuring that if the ExternalSecret is deleted, the native Kubernetes Secret is automatically garbage-collected.
  6. Consumption: Application Pods mount the native Kubernetes Secret as an environment variable or volume, remaining completely unaware of ESO or the upstream AWS infrastructure.

Real-World Scenario: Dynamic Data Transformation Often, cloud infrastructure teams and application developers have conflicting requirements for secret formats. Imagine your database team stores production credentials in AWS Secrets Manager as a single JSON blob:

JSON

{
  "username": "admin",
  "password": "superSecretPassword123",
  "host": "rds-prod-01.aws.com",
  "port": "5432"
}

However, your legacy application cannot parse JSON. It expects a native Kubernetes Secret containing a single file named db-config.properties formatted exactly like this:

Properties

DB_USERNAME=admin
DB_PASSWORD=superSecretPassword123
CONNECTION_STRING=jdbc:postgresql://rds-prod-01.aws.com:5432/main_db

Instead of refactoring the application code, you can use ESO’s Go-templating engine to bridge this gap dynamically.

The ExternalSecret YAML Configuration Here is the manifest that performs this transformation:

YAML

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dynamic-db-secret
  namespace: my-app-namespace
spec:
  refreshInterval: 1h 
  secretStoreRef:
    name: aws-secret-store
    kind: ClusterSecretStore 
  target:
    name: app-db-config-secret 
    creationPolicy: Owner      
    template:
      type: Opaque
      data:
        db-config.properties: |
          DB_USERNAME={{ .db_user }}
          DB_PASSWORD={{ .db_pass }}
          CONNECTION_STRING=jdbc:postgresql://{{ .db_host }}:{{ .db_port }}/main_db
  data:
  - secretKey: db_user
    remoteRef:
      key: prod/db/credentials
      property: username 
  - secretKey: db_pass
    remoteRef:
      key: prod/db/credentials
      property: password
  - secretKey: db_host
    remoteRef:
      key: prod/db/credentials
      property: host
  - secretKey: db_port
    remoteRef:
      key: prod/db/credentials
      property: port

Breaking Down the Magic

  • The Fetch (data block): ESO queries the AWS Secret named prod/db/credentials. Because the upstream payload is JSON, the property field is used to extract specific JSON keys. These extracted values are temporarily stored in internal ESO variables (e.g., db_user, db_host).
  • The Transform (target.template block): ESO creates a data key named db-config.properties within the target Kubernetes Secret. Using Go-template syntax ({{ .variable_name }}), the internal variables are injected directly into the required string format, securely constructing the CONNECTION_STRING.
  • Continuous Synchronization: If the database password is ever rotated in AWS, ESO will detect the change during its next 1-hour refresh cycle, rebuild the .properties template with the new password, and instantly update the native Kubernetes Secret.

Application Workload Integration: Hot-Reloading Kubernetes Secrets

Updating a Kubernetes Secret via the External Secrets Operator (ESO) is only half the battle. The other half is ensuring your application actually consumes the new values without requiring manual intervention.

By default, Kubernetes handles updated Secrets differently depending on how they are injected into the Pod:

  • Environment Variables: These are static. If ESO updates a Secret injected as an environment variable, the Pod will not see the change until it is restarted.
  • Volume Mounts: Kubernetes will automatically update files in mounted volumes (usually within a minute or two via the kubelet sync loop). However, your application still needs internal logic to detect the file change and re-read it.

Here are the two industry-standard methods to handle hot-reloading when ESO updates a Secret.

Method 1: The “Native” Way (Volume Mounts + App Logic)

If your application is designed to watch its configuration files for changes (e.g., using libraries like Viper in Go, or Spring Boot’s automatic reload capabilities), you can simply mount the ESO-generated Secret as a volume.

Deployment Manifest Example:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: my-app-container
        image: my-company/my-app:1.0
        volumeMounts:
        - name: db-config-volume
          mountPath: /app/config  # The app reads /app/config/db-config.properties
          readOnly: true
      volumes:
      - name: db-config-volume
        secret:
          secretName: app-db-config-secret # The Secret generated by ESO

How it works: 1. ESO updates app-db-config-secret. 2. The Kubernetes kubelet updates the db-config.properties file inside the running Pod’s /app/config directory. 3. Your application natively detects the file change and reloads its database connection pool in the background, resulting in zero downtime.

Method 2: The “Operator” Way (Stakater Reloader)

What if your application is a legacy app that reads environment variables only on startup? The industry-standard solution is an open-source controller called Reloader (by Stakater). Reloader watches for changes in ConfigMaps and Secrets and automatically triggers a graceful rolling restart of the associated Pods.

You simply install the Reloader controller in your cluster and add a specific annotation to your Deployment.

Deployment Manifest Example (with Reloader):

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    # Tells Reloader to watch this specific Secret for changes
    secret.reloader.stakater.com/reload: "app-db-config-secret" 
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: my-app-container
        image: my-company/my-app:1.0
        envFrom:
        - secretRef:
            name: app-db-config-secret # Injected as Environment Variables

How it works:

  1. ESO detects a password change in AWS and updates the native app-db-config-secret.
  2. Reloader notices the Secret has changed and finds the Deployment with the matching annotation.
  3. Reloader modifies the Deployment slightly (by updating a hidden hash in the Pod template).
  4. Kubernetes triggers a standard, zero-downtime rolling update. Old Pods are gracefully terminated while new Pods spin up, pulling the fresh environment variables.

Which method should you choose?

  • Use Method 1 if your application natively supports hot-reloading files. It is highly efficient because it avoids container restarts.
  • Use Method 2 if your application relies on environment variables, or if you need a universal, foolproof way to ensure any application picks up secret changes without modifying the underlying application code.

Securing the External Secrets Operator: A DevSecOps Guide to RBAC and Scoping

When introducing a tool that acts as a bridge between your cloud provider’s secret manager and your Kubernetes cluster, security is paramount. From a DevSecOps perspective, a compromised External Secrets Operator (ESO) controller or a misconfigured permission set could expose every single credential in your infrastructure.

By default, Kubernetes operators often run with broad permissions. To enforce the principle of least privilege, we must divide Role-Based Access Control (RBAC) into three distinct security pillars: User Separation of Duties, Workload Restrictions, and Controller Scoping.

1. Separation of Duties (User & Developer RBAC)

The core security benefit of ESO’s architecture is the strict separation between how to authenticate (SecretStore) and what to fetch (ExternalSecret).

You must enforce a strict boundary:

  • Platform/Security Admins are the only personnel allowed to create, update, or delete ClusterSecretStores and SecretStores. They manage the AWS IAM roles and connection details.
  • Application Developers are only allowed to manage ExternalSecrets. They can request data from pre-approved stores but cannot create new connections to unauthorized cloud accounts.

Here is an example ClusterRole to bind to your developer groups. It grants full control over the blueprints but strictly read-only access to the connections:

YAML

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eso-developer-role
rules:
# 1. Full access to manage the blueprints (ExternalSecrets)
- apiGroups: ["external-secrets.io"]
  resources: ["externalsecrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# 2. Read-only access to see available stores, but NO permission to alter them
- apiGroups: ["external-secrets.io"]
  resources: ["secretstores", "clustersecretstores"]
  verbs: ["get", "list", "watch"]

2. Workload Restrictions (Pod & ServiceAccount RBAC)

Once ESO creates the native Kubernetes Secret, default cluster behavior dictates that any Pod in that namespace can mount it. This creates a massive blast radius; if a frontend Pod is compromised, it could theoretically read the database Secret meant exclusively for the backend Pod.

To mitigate this, every application must run under its own specific ServiceAccount with permissions to read only the exact secrets it requires.

Here is how you restrict a backend application to read only a specific generated secret:

YAML

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: backend-secret-reader
  namespace: my-app-namespace
rules:
- apiGroups: [""]
  resources: ["secrets"]
  # SECURITY: Explicitly hardcode the exact secret name. No wildcards.
  resourceNames: ["app-db-config-secret"] 
  verbs: ["get", "watch"] # 'list' is intentionally omitted so the app can't enumerate other secrets
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: backend-read-secret-binding
  namespace: my-app-namespace
subjects:
- kind: ServiceAccount
  name: backend-app-sa # The ServiceAccount assigned to your backend Deployment
roleRef:
  kind: Role
  name: backend-secret-reader
  apiGroup: rbac.authorization.k8s.io

3. Controller Scoping (Namespace-Scoped ESO)

By default, installing ESO via Helm deploys it globally. It watches all namespaces and utilizes a ClusterRole with permissions to read and write native Secrets across the entire cluster.

In highly sensitive, multi-tenant clusters where absolute isolation is required, a global controller presents an unacceptable risk. The solution is installing ESO in Scoped Mode:

  • The ESO controller is installed into a specific namespace (e.g., tenant-a-secrets).
  • Instead of a ClusterRole, the deployment provisions a standard Role and RoleBinding confined strictly to tenant-a-secrets.
  • The operator becomes physically incapable of seeing or creating secrets in tenant-b-secrets, neutralizing the threat of malicious ExternalSecret manifests from other tenants.

While this requires running multiple lightweight instances of the ESO controller (one per tenant namespace), it guarantees absolute isolation at the Kubernetes API level.

Configuration Drift Management in Kubernetes

Configuration drift where the actual state of the cluster diverges from the desired state is a massive security and compliance risk. If a developer manually edits a database password directly inside the cluster using kubectl edit secret, that change isn’t tracked in Git, isn’t audited in AWS, and creates a fragile “snowflake” configuration.

Here is how ESO handles drift, the common pitfalls to avoid, and how to lock down your front door.

1. The Self-Healing Reconciler

ESO is built on the standard Kubernetes operator pattern, meaning it runs a continuous reconciliation loop. It constantly compares the desired state (AWS Secrets Manager) against the actual state (the native Kubernetes Secret).

If someone manually alters the generated Kubernetes Secret, ESO aggressively overwrites their manual changes. The speed of this self-healing depends entirely on your refreshInterval:

  • refreshInterval: 1m: A manual, unauthorized change will exist for a maximum of 60 seconds before ESO wipes it out and restores the true value from AWS.
  • refreshInterval: 0: Automatic synchronization is disabled. ESO will not detect or correct drift until someone manually triggers a force-sync.

2. The GitOps Conflict (The “Infinite Drift” Trap)

The most common drift issue teams face with ESO isn’t malicious; it’s a conflict with GitOps tools like ArgoCD or Flux.

If you use ArgoCD, it constantly watches your cluster to ensure it matches your Git repository. If ArgoCD deploys a blank Secret placeholder, and then ESO injects the real password into that Secret, ArgoCD sees this as “drift.” ArgoCD will overwrite ESO’s password with the blank Git version, and then ESO will overwrite ArgoCD with the real password. They will fight in an infinite loop.

The Fix: You must explicitly tell your GitOps tool to ignore the data field of secrets managed by ESO. In ArgoCD, add an ignoreDifferences block to your Application manifest:

YAML

  ignoreDifferences:
  - group: ""
    kind: Secret
    managedFieldsManagers:
    - external-secrets # Tells ArgoCD to ignore drift caused by ESO

3. Drift Prevention (Locking the Front Door)

While ESO is great at self-healing, the best security posture prevents manual drift from happening in the first place. Do not rely solely on the refreshInterval to fix unauthorized changes; block them outright by treating Kubernetes Secrets as immutable infrastructure at the RBAC level.

  • Revoke Update Permissions: Ensure that developer Kubernetes roles do not have update, patch, or delete verbs for secrets.
  • Force Upstream Changes: If a developer needs to change a password, they must change it at the source of truth (AWS Secrets Manager).
  • Audit the Source: By forcing changes upstream, you can leverage AWS CloudTrail to get a perfect, immutable audit log of exactly who changed the secret and when, satisfying stringent compliance requirements (like SOC2 or HIPAA).

4. Lifecycle Management and Orphaned Secrets

Drift also occurs when infrastructure is deleted but the underlying secrets are left behind as unmanaged “orphans.”

In your ExternalSecret manifest, always ensure you set creationPolicy: Owner. When this is set, ESO adds an ownerReference to the generated Kubernetes Secret. If you delete the ExternalSecret (or if your GitOps tool removes it), Kubernetes’ native garbage collection instantly deletes the underlying native Secret. This guarantees your cluster stays clean and no stale credentials drift in the background.

Secrets Management & KMS

1. The Base64 Illusion When you create a standard Kubernetes Secret, you might think the weird text (cGFzc3dvcmQ=) is encrypted. It is not. It is simply Base64 encoded. Anyone who gains access to your cluster’s etcd database (the “Brain’s memory”) can instantly decode every single password, database string, and API key you own in plain text.

2. The Fix: KMS Envelope Encryption To solve the etcd vulnerability, AWS allows you to integrate AWS KMS (Key Management Service) directly into the Kubernetes API. When you enable this, Kubernetes uses Envelope Encryption:

  • Kubernetes generates a Data Encryption Key (DEK) to encrypt your secret.
  • AWS KMS provides a Key Encryption Key (KEK) to encrypt the DEK.
  • Only the encrypted DEK and the encrypted secret are stored in etcd. Even if a hacker steals the entire database backup, it is completely unreadable garbage without the master key inside AWS KMS.

3. The GitOps Problem: Where do secrets come from? Even if etcd is encrypted, how do you get the secret into Kubernetes in the first place? If you write a Secret.yaml file with your database password in it and commit it to GitHub, your security is immediately compromised. In a DevSecOps environment, you never store secrets in Git. You store them in a secure vault like AWS Secrets Manager.

4. Bridging the Gap: ESO vs. CSI Driver How do we automatically pull passwords from AWS Secrets Manager into our EKS cluster without human intervention? The industry uses two main tools:

  • The Secrets Store CSI Driver: This tool mounts the AWS secret directly into your Pod as a physical file on a virtual hard drive. (Best for legacy apps that need to read .conf files).
  • External Secrets Operator (ESO): The 2026 DevSecOps favorite. It runs in the background, securely fetches the secret from AWS, and automatically generates a native Kubernetes Secret object for you. (Best for modern apps that use Environment Variables and GitOps workflows like ArgoCD).
Contents
Scroll to Top