EKS Secrets Store CSI Driver

Secrets Store CSI Driver (with AWS Provider) in Kubernetes

The Secrets Store Container Storage Interface (CSI) Driver is a powerful, Kubernetes-native solution to handle sensitive data. Instead of saving your AWS parameters, passwords, or API keys directly in the Kubernetes cluster (where they might be exposed), this driver fetches them on the fly and mounts them as temporary files inside your application’s pod. It is a highly secure way to manage configurations without leaving traces behind!

Think of the CSI Driver like an exclusive, highly secure hotel safe. Normally, if you use standard Kubernetes Secrets (environment variables), it is like carrying your valuable jewelry in your pockets everywhere you go in the hotel risky, right? With the CSI Driver, you keep your jewelry in a high-tech safe (tmpfs memory volume) right inside your specific room (the Pod). The moment you check out of the room (the Pod is deleted), the safe instantly evaporates, leaving absolutely zero trace of your valuables behind.

Quick Reference

• Dynamically mounts external secrets (from AWS) as highly secure, ephemeral files directly inside Kubernetes pods.
• Mechanism: Uses the Container Storage Interface (CSI).
• Storage: Ephemeral memory (tmpfs), meaning it never touches the disk.
• Security: Bypasses Kubernetes etcd database entirely.
• Requirement: Apps must be able to read configurations from file paths, not just standard environment variables.

How It Works

By default, Kubernetes uses ConfigMaps and Secrets. However, native Kubernetes Secrets are just base64 encoded they are not truly encrypted by default, and they sit in the cluster’s brain, called etcd. If someone gets access to etcd, they get all your passwords.

To fix this, we use the Secrets Store CSI Driver.

When your pod starts, this driver talks to an external vault (like AWS Secrets Manager), fetches the secret, and creates a temporary file inside the pod. Your application simply reads this file to get the password. It is incredibly safe because the secret is stored in the pod’s RAM (memory), not on the hard drive. Once the pod dies, the secret vanishes completely.

Instead of creating a native ConfigMap or Secret object within the cluster, the AWS parameters are fetched dynamically at pod startup. The CSI driver then mounts these fetched values directly as files inside the pod’s filesystem.

Pros

• Highly Secure (Ephemeral Storage): The data is strictly tied to the pod’s lifecycle. It is stored entirely in memory (tmpfs) and never touches the disk or the Kubernetes etcd database, significantly reducing your attack surface.

Cons

• Application Compatibility: The variables exist as files on a mounted volume, not as standard environment variables. Applications must be explicitly written or adapted to read their configurations from these specific file paths.
• (Note: The CSI driver does have an optional Sync as Kubernetes Secret feature to bridge this environment variable gap, but mounting as ephemeral volumes is its primary and most secure design).

DevSecOps Architect Level

At an architectural level, the solution relies on two primary components: the Secrets Store CSI Driver (which runs as a DaemonSet on every node) and the AWS Provider (which acts as the translation layer to AWS APIs).

When a Pod is scheduled, the kubelet process on the worker node calls the CSI driver to mount the volume. The CSI driver then invokes the AWS Provider. The provider authenticates to AWS—typically using IAM Roles for Service Accounts (IRSA)—fetches the payloads from AWS Secrets Manager or Systems Manager Parameter Store, and hands them back to the CSI driver. The driver then creates a tmpfs mount (RAM disk) inside the Pod’s mount namespace and writes the secrets as files. Because it uses tmpfs, the data never persists to the Elastic Block Store (EBS) volumes or the worker node’s physical disk.

For a production-grade DevSecOps setup, this architecture strictly enforces the principle of least privilege. Instead of giving the node broad permissions, you map a Kubernetes ServiceAccount to an AWS IAM Role via OIDC (IRSA). Only the specific pod needing the secret can assume the role to fetch it.

1. Secret Rotation: By default, the mounted files are static. However, the CSI driver supports an Auto Rotation feature. If enabled, the driver periodically polls AWS and updates the mounted files if the secret changes, without needing to restart the pod!
2. The “Sync as Secret” Trade-off: The text mentions the optional sync to Kubernetes Secrets. It’s important to understand that using this feature completely defeats the primary security benefit of the CSI driver. Once synced, the secret is written back into etcd. It should only be used as a last resort for legacy applications that absolutely refuse to read from a file path.

Tools & Official Links:

• Kubernetes Secrets Store CSI Driver: Official GitHub/Docs
• AWS Provider for Secrets Store CSI Driver: AWS GitHub Repository

---

Architectural Paradigms: Operator vs. Storage Interface

When to Choose External Secrets Operator (ESO)

The External Secrets Operator (ESO) follows the classic Kubernetes Controller/Operator pattern. It continuously watches custom resources (like ExternalSecret and SecretStore) and actively reconciles their state by fetching data from external APIs (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) and materializing that data into standard, native Kubernetes Secret objects.

• Legacy Applications: Your app expects configuration via standard Environment Variables.
• Broad Compatibility: You need to use secrets for things like imagePullSecrets or Ingress TLS certificates, which must be native Kubernetes Secret objects.
• Ease of Management: You want a “set it and forget it” sync where the secret is always available in the cluster.

When to Choose Secrets Store CSI Driver

Conversely, the Secrets Store CSI Driver extends Kubernetes via the Container Storage Interface. It operates primarily as a DaemonSet deployed across worker nodes. Instead of creating cluster-wide objects, it intercepts volume mount requests during the Pod scheduling phase, fetches the external secret synchronously, and mounts it directly into the Pod’s localized memory as a tmpfs volume.

• High Compliance (SOC2/HIPAA): You want to minimize the footprint of sensitive data. Storing secrets in etcd is often a red flag for auditors.
• Large Secrets: You are mounting large configuration files or certificates that are too big for the 1MB limit of a Kubernetes Secret.
• Strong Identity: You want to leverage IAM Roles for Service Accounts (IRSA) to ensure only a specific Pod identity can “check out” a secret from AWS at runtime.

The “Hybrid” Reality

It is worth noting that many teams use both. They use ESO for infrastructure-level secrets (like TLS certs and registry credentials) and the CSI Driver for application-level secrets (like database passwords and API keys) to achieve the highest possible security for their code.

Feature	External Secrets Operator (ESO)	Secrets Store CSI Driver
Primary Mechanism	Controller syncs external data to a native K8s Secret object.	Driver mounts external data as a tmpfs volume directly into the Pod.
Storage in Cluster	Exists in etcd. Secrets are stored as native Kubernetes objects.	Does not exist in etcd. Secrets reside only in the Pod’s memory/filesystem.
Consumption	Native & Easy. Used as environment variables via secretKeyRef or as files.	File-based. Primarily accessed as files on a mount path.
Security Profile	Moderate. Relies on K8s RBAC and encryption-at-rest for etcd.	High/Zero-Trust. Secret data never touches the K8s API or disk.
Performance	Asynchronous. Secrets are synced regardless of whether a Pod is running.	Synchronous. Secrets are fetched at Pod startup; can slightly delay boot time.
Drift Handling	Corrects drift based on a refreshInterval.	Automatically updated when a Pod restarts (or via optional rotation feature).
GitOps Friendly	Requires ignoreDifferences in ArgoCD/Flux to avoid sync loops.	High. Since no native Secret is created, there is no state to conflict with.

---

AWS Service & Use-Case Decision Matrix

Use this table to map your specific AWS resource types and security requirements to the correct tool.

Requirement / AWS Service	External Secrets Operator (ESO)	Secrets Store CSI Driver
AWS Secrets Manager	Full Support (Syncs to K8s Secret).	Full Support (Mounts as Volume).
AWS Parameter Store (SSM)	Full Support.	Full Support.
IAM Roles for Service Accounts (IRSA)	Cluster-wide or Namespace-wide. One role often manages many secrets.	Pod-level granular. Each Pod can have a unique IAM role for specific secrets.
imagePullSecrets	Recommended. Required to be a native K8s Secret for the Kubelet.	Not supported natively (requires complex sync-back).
Ingress TLS Certificates	Recommended. Ingress controllers need native K8s Secrets.	Not recommended.
Environment Variables	Native. Uses standard envFrom or valueFrom.	Requires “Sync as Secret” feature enabled.
Application Config Files	Possible, but limited by 1MB Secret size.	Ideal. Mounts large configs/certs directly as files.
High Security/Compliance	Moderate (Data hits etcd).	Highest (Data stays in tmpfs memory).

---

Migration Guide: Moving from ESO to Secrets Store CSI Driver

If you are moving from ESO to the CSI Driver to improve your security posture (e.g., for SOC2 compliance), follow these steps:

Phase 1: Infrastructure Preparation

1. Update IAM Policies: Create specific IAM policies for your applications. Unlike ESO, which often uses a broad controller-level policy, the CSI Driver works best when each Pod has an IAM Role for Service Accounts (IRSA) that only grants access to its specific secrets.
2. Install the Driver: Install the secrets-store-csi-driver and the aws-secrets-manager-provider via Helm.

Phase 2: Define the SecretProviderClass

Instead of an ExternalSecret manifest, you will create a SecretProviderClass. This tells the CSI driver exactly which AWS secrets to fetch.

YAML

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-app-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/db/password"
        objectType: "secretsmanager"

Phase 3: Update the Application Deployment

You must modify your Deployment manifest to mount the secret as a volume.

• Add a Volume: Reference the csi driver and your SecretProviderClass.
• Add a VolumeMount: Choose the path where your app will read the file (e.g., /mnt/secrets-store).

Phase 4: Application Code Changes

Since the CSI Driver stores secrets as files, your application must be updated to read from the filesystem rather than looking for environment variables.

• Tip: If you cannot change the code, enable the secretObjects feature in the SecretProviderClass to mirror the file back into a K8s Secret (though this negates some of the “Zero-etcd” security benefits).

Phase 5: Cleanup

Once the Pod is successfully running and reading secrets from the volume:

1. Delete the ExternalSecret manifest.
2. ESO’s creationPolicy: Owner (if set) will automatically clean up the old native Kubernetes Secret.

---

1. The SecretProviderClass

This custom resource tells the CSI driver exactly which secret to fetch from AWS and how to format it.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: devsecopsguru-db-secrets
  namespace: default
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/devsecopsguru/db-password"
        objectType: "secretsmanager"
        objectAlias: "db-password.txt" # The filename it will be saved as

Key Details:

• objectName: The exact ARN or name of the secret in AWS Secrets Manager.
• objectAlias: This optional field dictates the exact filename that will appear inside your pod.

---

2. The Application Deployment

This manifest deploys your application and mounts the CSI volume. The application will find the secret waiting for it at /mnt/secrets/db-password.txt.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecopsguru-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devsecopsguru-app
  template:
    metadata:
      labels:
        app: devsecopsguru-app
    spec:
      # Crucial: The ServiceAccount must have an IAM Role attached (IRSA)
      # with permission to read "prod/devsecopsguru/db-password"
      serviceAccountName: devsecopsguru-app-sa
      containers:
      - name: main-app
        image: nginx:latest
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets"
          readOnly: true
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "devsecopsguru-db-secrets"

How it connects:

• volumes: The pod requests a CSI volume and explicitly references the devsecopsguru-db-secrets SecretProviderClass created in step 1.
• volumeMounts: The container mounts that CSI volume into its own filesystem at /mnt/secrets.
• serviceAccountName: The pod authenticates to AWS seamlessly using IAM Roles for Service Accounts (IRSA). You do not pass AWS access keys here.

1. Update the SecretProviderClass (Adding secretObjects)

You need to add the secretObjects block to your SecretProviderClass. This block tells the driver how to map the fetched AWS data into a native Kubernetes Secret.

YAML

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: devsecopsguru-db-secrets
  namespace: default
spec:
  provider: aws
  
  # --- NEW: Sync as K8s Secret Configuration ---
  secretObjects:
  - secretName: devsecopsguru-k8s-secret  # The name of the native Secret to create
    type: Opaque
    data:
    - objectName: "db-password.txt"       # Must match the objectAlias (or objectName) below
      key: DB_PASSWORD                    # The key that will appear inside the native Secret

  # --- Original AWS Parameters ---
  parameters:
    objects: |
      - objectName: "prod/devsecopsguru/db-password"
        objectType: "secretsmanager"
        objectAlias: "db-password.txt"

2. Update the Deployment (The Crucial “Catch”)

Here is the most important concept to understand about this feature: The native Kubernetes Secret is ONLY created when a Pod mounts the CSI volume. You cannot simply create the SecretProviderClass and expect the K8s Secret to appear. The pod’s startup process is the trigger. Therefore, your Deployment must do both: mount the volume to trigger the fetch, and then inject the resulting Secret as an environment variable.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecopsguru-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devsecopsguru-app
  template:
    metadata:
      labels:
        app: devsecopsguru-app
    spec:
      serviceAccountName: devsecopsguru-app-sa
      containers:
      - name: main-app
        image: nginx:latest
        
        # 2. Consume the synced native Secret as an Environment Variable
        env:
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: devsecopsguru-k8s-secret # Matches secretName from SecretProviderClass
              key: DB_PASSWORD               # Matches key from SecretProviderClass

        # 1. MUST STILL MOUNT THE VOLUME to trigger the driver to fetch from AWS
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets"
          readOnly: true
          
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "devsecopsguru-db-secrets"

Why use this over External Secrets Operator (ESO)?

If you end up syncing a native Kubernetes Secret anyway, why use the CSI Driver instead of ESO?

1. Identity (IRSA): The CSI driver still uses Pod-level IAM identity (IRSA). Only the devsecopsguru-app-sa ServiceAccount has the AWS permissions to fetch this specific secret. ESO typically relies on a cluster-wide or namespace-wide IAM role attached to the operator itself.
2. Automated Cleanup: If the Pod is deleted, and no other pods are using that SecretProviderClass, the CSI driver automatically deletes the mirrored devsecopsguru-k8s-secret.

By default, the Secrets Store CSI Driver only fetches your AWS parameters once: when the Pod initially starts up. If you change a password in AWS Secrets Manager after the Pod is running, the Pod will continue using the old password until it is restarted.

To solve this, you can enable Secret Rotation. This configures the CSI Driver to periodically poll AWS and automatically update the mounted files (and any synced Kubernetes Secrets) while the Pod is running.

---

Enabling Secret Rotation

Here is how to configure rotation, along with the crucial steps required to ensure your application actually uses the new credentials.

1. Enable Rotation on the CSI Driver

Secret rotation is not enabled in the SecretProviderClass or the Deployment; it is a global setting enabled when you install or upgrade the CSI Driver itself via Helm.

You must set enableSecretRotation: true and define a rotationPollInterval.

Bash

helm upgrade -i csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set enableSecretRotation=true \
  --set rotationPollInterval=2m # The driver will check AWS every 2 minutes

2. How the Driver Applies Updates

Once rotation is enabled, the driver checks AWS based on your polling interval. If it detects a change in the upstream secret:

• Mounted Files: The driver automatically overwrites the file inside the Pod’s volume mount (e.g., /mnt/secrets/db-password.txt) with the new value.
• Synced Secrets: If you are using the secretObjects feature to mirror the data into a native Kubernetes Secret, the driver will automatically patch that native Secret with the new value.

3. The Application Catch (Crucial)

This is the most common pitfall with secret rotation: Just because the file or the Kubernetes Secret has been updated does not mean your application is using the new password.

How your application handles the update depends entirely on how it consumes the secret:

• If consuming via Mounted Files: Your application code must be written to detect file changes. It can use a file watcher mechanism (like inotify in Linux) to re-read the configuration file into memory when the file is modified by the CSI driver. If your app only reads the file once on startup, rotation will not help.
• If consuming via Environment Variables (Synced Secrets): Environment variables are injected into a process at boot time and cannot be changed dynamically while the process is running. Even though the CSI driver updates the native K8s Secret, the running Pod will still have the old environment variable in memory.

4. Forcing Application Restarts (The Solution for Env Vars)

If your application relies on environment variables, the only way to apply the rotated secret is to restart the Pods.

Instead of doing this manually, the industry standard is to use a Kubernetes controller like Reloader (by Stakater). Reloader watches your Kubernetes Secrets and ConfigMaps. The moment the CSI Driver updates the synced Kubernetes Secret, Reloader detects the change and automatically triggers a rolling restart of your Deployment, ensuring the new Pods boot up with the fresh environment variables.

Here is how to bridge that final gap using Reloader.

When the Secrets Store CSI Driver successfully pulls a new, rotated password from AWS and updates the synced Kubernetes Secret, Reloader acts as the trigger mechanism to gracefully restart your application so it can ingest the new environment variables.

---

Automated Pod Restarts with Reloader

Reloader (by Stakater) is a lightweight Kubernetes controller that watches for changes in ConfigMap and Secret objects and triggers rolling upgrades on Pods associated with them.

1. Install Reloader

The easiest way to install Reloader into your cluster is via Helm. It runs as a single controller in your cluster and watches for specific annotations.

Bash

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install reloader stakater/reloader --namespace kube-system

2. Annotate the Deployment

You do not need to modify your application code or your SecretProviderClass. You only need to add a specific annotation to your Deployment metadata (not the Pod template metadata).

This annotation tells Reloader: “Watch the secret named devsecopsguru-k8s-secret. If it changes, restart my Pods.”

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecopsguru-app
  namespace: default
  annotations:
    # --- NEW: Tell Reloader to watch this specific Secret ---
    secret.reloader.stakater.com/reload: "devsecopsguru-k8s-secret" 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devsecopsguru-app
  template:
    metadata:
      labels:
        app: devsecopsguru-app
    spec:
      serviceAccountName: devsecopsguru-app-sa
      containers:
      - name: main-app
        image: nginx:latest
        env:
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: devsecopsguru-k8s-secret # The synced K8s Secret
              key: DB_PASSWORD
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets"
          readOnly: true
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "devsecopsguru-db-secrets"

3. The Complete Automated Lifecycle

With rotation enabled on the CSI driver and Reloader installed, here is the exact sequence of events when a database password is changed in AWS:

1. The Change: A DevSecOps engineer or an automated AWS Lambda function rotates the password in AWS Secrets Manager.
2. The Poll: Based on your rotationPollInterval (e.g., 2 minutes), the CSI Driver checks AWS and detects the new value.
3. The Sync: The CSI driver updates the file in the mounted volume and immediately updates the synced devsecopsguru-k8s-secret in the cluster.
4. The Trigger: Reloader instantly detects that devsecopsguru-k8s-secret has been modified.
5. The Restart: Reloader triggers a standard Kubernetes rolling update on the devsecopsguru-app Deployment.
6. The Fresh Boot: New Pods spin up, the CSI driver fetches the new secret to mount, the new Pods read the new environment variable, and the old Pods are smoothly terminated with zero downtime.