Back to blog External Secrets Operator with AWS Secrets Manager - Stop Mounting Secrets in ConfigMaps
· 10 min read · - views

External Secrets Operator with AWS Secrets Manager - Stop Mounting Secrets in ConfigMaps

K8sSecurity
#kubernetes#external-secrets#aws#secrets-manager#security#gitops

External Secrets Operator with AWS Secrets Manager - Stop Mounting Secrets in ConfigMaps

Your application needs database credentials. The traditional approach: store them in a Kubernetes Secret, reference it in your deployment. But now those credentials are in your Git repo (encrypted or not), detached from your central secret management, and a pain to rotate.

External Secrets Operator (ESO) solves this by syncing secrets from external providers (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault) into Kubernetes Secrets automatically. Change the secret in AWS, ESO updates the Kubernetes Secret. No Git commits, no manual kubectl apply.

This post covers ESO with AWS Secrets Manager - setup, authentication, patterns, and production gotchas.

TL;DR

  • ESO syncs external secrets to Kubernetes Secrets automatically
  • SecretStore defines how to connect to AWS Secrets Manager
  • ExternalSecret defines what to fetch and where to put it
  • Use IRSA (IAM Roles for Service Accounts) for authentication
  • Secrets refresh automatically based on refreshInterval
  • Works with GitOps - ExternalSecret manifests are safe to commit

Code Repository: All code from this post is available at github.com/moabukar/blog-code/external-secrets-operator

The Problem with Kubernetes Secrets

Traditional secret management in Kubernetes:

# This ends up in Git somehow...
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
data:
  username: YWRtaW4=        # admin (base64 ≠ encryption)
  password: c3VwZXJzZWNyZXQ= # supersecret

Problems:

  • Secrets in Git - Even with SOPS or Sealed Secrets, it’s friction
  • No central management - Secrets scattered across repos
  • Manual rotation - Change in AWS, update K8s, redeploy
  • No audit trail - Who changed what when?
  • Duplication - Same secret in multiple clusters

How External Secrets Operator Works

┌─────────────────────────────────────────────────────────────────┐
│                    External Secrets Flow                        │
└─────────────────────────────────────────────────────────────────┘

  SecretStore                ExternalSecret           Kubernetes
      │                           │                    Secret
      ▼                           ▼                      │
┌─────────────┐            ┌─────────────┐              ▼
│ AWS Secrets │ ◄───────── │     ESO     │ ──────► ┌─────────────┐
│  Manager    │   fetch    │ Controller  │  create │   Secret    │
│             │            │             │         │ (auto-sync) │
└─────────────┘            └─────────────┘         └─────────────┘
      │                           │
      └───── refresh (1h) ────────┘
  1. SecretStore - Defines connection to AWS Secrets Manager
  2. ExternalSecret - Declares what secrets to fetch
  3. ESO Controller - Fetches and creates Kubernetes Secrets
  4. Auto-sync - Periodically refreshes from source

Installation

Install ESO using Helm:

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

Or with Terraform:

resource "helm_release" "external_secrets" {
  name             = "external-secrets"
  repository       = "https://charts.external-secrets.io"
  chart            = "external-secrets"
  namespace        = "external-secrets"
  create_namespace = true
  version          = "0.9.11"

  set {
    name  = "installCRDs"
    value = "true"
  }

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.external_secrets.arn
  }
}

Authentication with IRSA

The recommended approach for EKS is IRSA (IAM Roles for Service Accounts). No static credentials needed.

Create IAM Role

# IAM Role for External Secrets
resource "aws_iam_role" "external_secrets" {
  name = "external-secrets-operator"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.eks.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub" = "system:serviceaccount:external-secrets:external-secrets"
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

# IAM Policy for Secrets Manager access
resource "aws_iam_role_policy" "external_secrets" {
  name = "secrets-manager-access"
  role = aws_iam_role.external_secrets.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret",
          "secretsmanager:ListSecrets"
        ]
        Resource = [
          "arn:aws:secretsmanager:${var.aws_region}:${var.account_id}:secret:app/*",
          "arn:aws:secretsmanager:${var.aws_region}:${var.account_id}:secret:shared/*"
        ]
      }
    ]
  })
}

Annotate Service Account

If using Helm, set the annotation during install. Otherwise:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/external-secrets-operator

SecretStore Configuration

SecretStore defines how to connect to AWS Secrets Manager. It’s namespace-scoped.

Basic SecretStore with IRSA

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: app
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

ClusterSecretStore (Cluster-wide)

For a central secret store accessible from all namespaces:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

SecretStore with Role Assumption

For cross-account access or fine-grained permissions:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: app
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      role: arn:aws:iam::123456789012:role/secrets-reader
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

ExternalSecret Examples

Basic Secret Fetch

Fetch a single secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: app/database
        property: username
    - secretKey: password
      remoteRef:
        key: app/database
        property: password

This creates a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: app
type: Opaque
data:
  username: <from AWS>
  password: <from AWS>

Fetch Entire Secret as JSON

If your AWS secret contains JSON:

{
  "username": "admin",
  "password": "supersecret",
  "host": "db.example.com",
  "port": "5432"
}

Fetch all properties:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-credentials
  dataFrom:
    - extract:
        key: app/database

Creates:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
data:
  username: YWRtaW4=
  password: c3VwZXJzZWNyZXQ=
  host: ZGIuZXhhbXBsZS5jb20=
  port: NTQzMg==

Template the Output

Create a connection string:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-url
    template:
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .database }}"
  data:
    - secretKey: username
      remoteRef:
        key: app/database
        property: username
    - secretKey: password
      remoteRef:
        key: app/database
        property: password
    - secretKey: host
      remoteRef:
        key: app/database
        property: host
    - secretKey: port
      remoteRef:
        key: app/database
        property: port
    - secretKey: database
      remoteRef:
        key: app/database
        property: database

Multiple Secrets from Different Sources

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-secrets
  data:
    # Database credentials
    - secretKey: DB_PASSWORD
      remoteRef:
        key: app/database
        property: password
    # API keys
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: app/stripe
        property: api_key
    # OAuth credentials
    - secretKey: OAUTH_CLIENT_SECRET
      remoteRef:
        key: shared/oauth
        property: client_secret

Production Patterns

Pattern 1: Namespace-Scoped SecretStores

Each team gets their own SecretStore with limited access:

# Team A - can only access team-a/* secrets
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: team-a-secrets
  namespace: team-a
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      role: arn:aws:iam::123456789012:role/team-a-secrets-reader
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

IAM policy for the role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret"
    ],
    "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:team-a/*"
  }]
}

Pattern 2: Environment-Specific Secrets

Use naming conventions:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: app-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/app/database  # Environment in path
        property: connection_string

Pattern 3: Shared Secrets Across Namespaces

Use ClusterSecretStore with ClusterExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: shared-tls-cert
spec:
  externalSecretSpec:
    refreshInterval: 24h
    secretStoreRef:
      name: aws-secrets-manager
      kind: ClusterSecretStore
    target:
      name: wildcard-tls
      creationPolicy: Owner
    data:
      - secretKey: tls.crt
        remoteRef:
          key: shared/wildcard-tls
          property: certificate
      - secretKey: tls.key
        remoteRef:
          key: shared/wildcard-tls
          property: private_key
  namespaceSelector:
    matchLabels:
      tls-enabled: "true"

Pattern 4: GitOps-Friendly Structure

Structure for ArgoCD/Flux:

├── base/
│   ├── secret-store.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   ├── external-secrets.yaml
    │   └── kustomization.yaml
    └── prod/
        ├── external-secrets.yaml
        └── kustomization.yaml

The ExternalSecret manifests are safe to commit - they only reference where secrets are, not the values.

Refresh and Rotation

Refresh Interval

ESO polls the external secret provider at the configured interval:

spec:
  refreshInterval: 1h  # Check every hour

For frequently rotated secrets:

spec:
  refreshInterval: 5m  # Check every 5 minutes

Cost consideration: Each refresh calls Secrets Manager APIs. With many ExternalSecrets and short intervals, costs add up.

Handling Rotation

When a secret rotates in AWS Secrets Manager, ESO updates the Kubernetes Secret on next refresh. But your pods won’t automatically restart.

Options:

  1. Reloader - Automatically restart pods when secrets change:
helm install reloader stakater/reloader -n kube-system

Annotate your deployment:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"
  1. Use secret hash in deployment - Forces rollout on change:
spec:
  template:
    metadata:
      annotations:
        checksum/secret: {{ include (print $.Template.BasePath "/external-secret.yaml") . | sha256sum }}

Troubleshooting

Secret Not Syncing

Check ExternalSecret status:

kubectl get externalsecret -n app
kubectl describe externalsecret database-credentials -n app

Look for conditions:

Conditions:
  Type     Status  Reason
  ----     ------  ------
  Ready    False   SecretSyncedError

SecretStore Connection Failed

Verify SecretStore:

kubectl get secretstore -n app
kubectl describe secretstore aws-secrets-manager -n app

Common issues:

  • IRSA not configured correctly
  • IAM role doesn’t have Secrets Manager permissions
  • Region mismatch

Check ESO Controller Logs

kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

Verify IAM Permissions

# Test from a pod with the service account
kubectl run test --rm -i --tty \
  --image=amazon/aws-cli \
  --serviceaccount=external-secrets \
  -n external-secrets \
  -- aws secretsmanager get-secret-value --secret-id app/database

Security Best Practices

1. Least Privilege IAM

Restrict to specific secret paths:

{
  "Resource": "arn:aws:secretsmanager:*:*:secret:app/production/*"
}

2. Use Namespace Isolation

Don’t use ClusterSecretStore unless necessary. Namespace-scoped SecretStores with role assumption provide better isolation.

3. Audit Access

Enable CloudTrail logging for Secrets Manager:

resource "aws_cloudtrail" "secrets_audit" {
  name           = "secrets-audit"
  s3_bucket_name = aws_s3_bucket.audit.id

  event_selector {
    read_write_type           = "All"
    include_management_events = true
  }
}

4. Rotate Secrets Regularly

Use AWS Secrets Manager rotation:

resource "aws_secretsmanager_secret_rotation" "database" {
  secret_id           = aws_secretsmanager_secret.database.id
  rotation_lambda_arn = aws_lambda_function.rotation.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

Terraform Module

Complete module for ESO with AWS:

# modules/external-secrets/main.tf

variable "cluster_name" {
  type = string
}

variable "cluster_oidc_provider_arn" {
  type = string
}

variable "cluster_oidc_provider_url" {
  type = string
}

variable "secrets_prefix" {
  type    = string
  default = "app/*"
}

# IAM Role
resource "aws_iam_role" "external_secrets" {
  name = "${var.cluster_name}-external-secrets"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = var.cluster_oidc_provider_arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.cluster_oidc_provider_url}:sub" = "system:serviceaccount:external-secrets:external-secrets"
          "${var.cluster_oidc_provider_url}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "external_secrets" {
  name = "secrets-manager-access"
  role = aws_iam_role.external_secrets.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecrets"
      ]
      Resource = "arn:aws:secretsmanager:*:*:secret:${var.secrets_prefix}"
    }]
  })
}

# Helm Release
resource "helm_release" "external_secrets" {
  name             = "external-secrets"
  repository       = "https://charts.external-secrets.io"
  chart            = "external-secrets"
  namespace        = "external-secrets"
  create_namespace = true
  version          = "0.9.11"

  set {
    name  = "installCRDs"
    value = "true"
  }

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.external_secrets.arn
  }
}

# ClusterSecretStore
resource "kubectl_manifest" "cluster_secret_store" {
  yaml_body = yamlencode({
    apiVersion = "external-secrets.io/v1beta1"
    kind       = "ClusterSecretStore"
    metadata = {
      name = "aws-secrets-manager"
    }
    spec = {
      provider = {
        aws = {
          service = "SecretsManager"
          region  = data.aws_region.current.name
          auth = {
            jwt = {
              serviceAccountRef = {
                name      = "external-secrets"
                namespace = "external-secrets"
              }
            }
          }
        }
      }
    }
  })

  depends_on = [helm_release.external_secrets]
}

output "role_arn" {
  value = aws_iam_role.external_secrets.arn
}

Conclusion

External Secrets Operator eliminates the need to manage secrets in Git. Your Kubernetes Secrets stay in sync with AWS Secrets Manager automatically. Combined with IRSA, you get secure, auditable secret management without static credentials.

Start simple: install ESO, create a ClusterSecretStore, and migrate one application. Once comfortable, expand to namespace isolation and automated rotation.

References

Found this helpful?

Related Posts

OpenTelemetry Changed How I Think About Observability

A practical, opinionated take on OpenTelemetry - why it matters, what it actually solves, and how to instrument across Kubernetes, Lambda, ECS, and EC2 without losing your mind.

AWS Control Tower Account Factory - The Gotchas Nobody Tells You

Real-world lessons from automating AWS account provisioning with Control Tower, Service Catalog, and Terraform. The silent failures, IAM traps, and StackSet timing issues that cost us days.

Building an Automated Multi-Account AWS Architecture with Control Tower and Terraform

A hands-on walkthrough of enabling AWS Control Tower, designing an OU structure, automating account provisioning via Service Catalog, and deploying security baselines - from zero to fully automated account vending in production.

Comments