Skip to main content
< All Topics

Kubernetes Secrets Management at Scale

Posted
Updated
Author -Rajkumar Aute
Views7
Enterprise-Grade DevSecOps

But here is the shocking truth: In a real production environment, creating Kubernetes Secrets manually is considered a bad practice. If you go to a DevSecOps interview and say, “I manage passwords by creating Kubernetes Secret YAML files,” you will likely fail the interview. Why? Let’s find out.

Why Kubernetes Secrets Aren’t Safe Enough

(The Base64 Illusion)

Kubernetes Secret objects have three major flaws when used at an enterprise scale:

1. Base64 is NOT Encryption

When you write a Secret YAML, you encode the password in Base64. password123 becomes cGFzc3dvcmQxMjM=. This looks safe to the human eye, but anyone who can run kubectl get secret can instantly decode it using echo "cGFzc3dvcmQxMjM=" | base64 --decode. It provides zero cryptographic security.

2. The GitOps Problem

In modern DevOps, we store all our YAML files in Git (GitHub/GitLab) so we can deploy them automatically using tools like ArgoCD. You cannot commit a Kubernetes Secret YAML to GitHub. If you do, your database password is now exposed to every developer in the company (or the whole internet if the repo is public!).

3. The Secret Sprawl

If you have 50 microservices across 10 clusters that all need the same AWS API key, you have to manually create and update that Secret in 50 different places. When the key rotates every 90 days, it becomes a nightmare to manage.

The Solution: We must store our secrets outside of Kubernetes in a dedicated, highly secure vault, and let Kubernetes fetch them dynamically.

External Secrets Operator (ESO)

(The Bridge to the Vault)

The External Secrets Operator (ESO) is an open-source Kubernetes operator that solves the secret management problem. It integrates Kubernetes with external secret management systems like AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or Google Secret Manager.

How ESO Works:

  1. You store your highly sensitive password securely in AWS Secrets Manager or Vault.
  2. You install the ESO controller in your Kubernetes cluster.
  3. You create an ExternalSecret YAML file. This file does not contain the password. It only contains a “pointer” (e.g., “Go fetch the secret named prod-db-pass from AWS”).
  4. Because the ExternalSecret YAML contains no actual passwords, you can safely commit it to GitHub!
  5. ESO reads this file, securely connects to AWS, fetches the password, and automatically generates a native Kubernetes Secret in the cluster for your Pods to use.

If you rotate the password in AWS, ESO automatically detects the change and updates the Kubernetes Secret. Zero touch required.

Integrating with AWS Secrets Manager

(The Cloud-Native Way)

Since AWS is heavily used in the industry, let’s look at how to set this up using AWS Secrets Manager.

To make this work, we need two Custom Resource Definitions (CRDs) provided by ESO.

1. The SecretStore (The Connection)

This tells ESO how to connect to AWS. (Note: In AWS EKS, we use IRSA IAM Roles for Service Accounts so the ESO pod can securely authenticate to AWS without needing hardcoded AWS access keys).

YAML
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: my-app-namespace
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-service-account # This SA has an AWS IAM role attached

2. The ExternalSecret (The Fetcher)

Now, we tell ESO exactly what to fetch from AWS and what to name the resulting Kubernetes Secret.

YAML
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: fetch-db-password
  namespace: my-app-namespace
spec:
  refreshInterval: "1h"             # Check AWS every 1 hour for updates
  secretStoreRef:
    name: aws-secrets-manager       # Point to the SecretStore we created above
    kind: SecretStore
  target:
    name: app-db-secret             # The native K8s Secret it will create!
    creationPolicy: Owner
  data:
  - secretKey: db_password          # The key inside the K8s Secret
    remoteRef:
      key: prod/rds/mysql/credentials # The exact name of the secret in AWS
      property: password            # The specific JSON field in AWS to fetch

The Result:

When you apply these two files, ESO reaches out to AWS Secrets Manager, grabs the password, and silently creates a standard Kubernetes Secret named app-db-secret. Your Pod can now mount app-db-secret just like normal!

The DevSecOps Flow

To prove you understand this flow for interviews, memorize this architecture:

  1. The Admin creates the password in HashiCorp Vault or AWS Secrets Manager.
  2. The Developer writes an ExternalSecret YAML that points to that Vault.
  3. The CI/CD Pipeline pushes that YAML safely to GitHub.
  4. ArgoCD deploys the YAML to Kubernetes.
  5. ESO fetches the password and creates the native Secret.
  6. The Pod consumes the Secret.

Security Achieved: No passwords in code, no passwords in Git, central rotation, and perfect audit logs!

Contents
Scroll to Top