Back to blog Database Backup to S3 with Kubernetes CronJobs
· 9 min read · - views

Database Backup to S3 with Kubernetes CronJobs

K8sDatabases
#kubernetes#postgresql#s3#backup#cronjob#localstack#devops#databases

Database Backup to S3 with Kubernetes CronJobs

Automated database backups are non-negotiable in production. This guide shows how to build a Kubernetes CronJob that streams PostgreSQL backups directly to S3, with a complete local testing environment using KIND and LocalStack.

+-------------+     +-------------+     +-------------+
|  PostgreSQL |---->|  K8s CronJob|---->|     S3      |
|   (source)  |     |  (pg_dump)  |     |  (storage)  |
+-------------+     +-------------+     +-------------+

TL;DR

Code Repository: All code from this post is available at github.com/moabukar/blog-code/database-backup-s3-kubernetes

  • Kubernetes CronJob runs pg_dump on schedule
  • Streams backup directly to S3 (no local disk needed)
  • LocalStack for local S3 testing
  • KIND cluster for Kubernetes testing
  • Full verify/restore workflow
  • One-command setup: make setup && make test-working && make verify

Architecture

                    +-----------------------+
                    |    Kubernetes Cluster |
                    |                       |
+----------+        |  +----------------+   |        +----------+
| Postgres |<-------+--|   CronJob Pod  |---+------->|    S3    |
| Primary  |   read |  | (backup runner)|   | upload | (bucket) |
+----------+        |  +----------------+   |        +----------+
     |              |         |             |             |
     v              |         v             |             v
+----------+        |  +----------------+   |        +----------+
| Postgres |        |  |    Secrets     |   |        | Lifecycle|
| Replica  |        |  | (DB + AWS creds)|   |        |  Policy  |
+----------+        |  +----------------+   |        +----------+
                    +-----------------------+

Components:

COMPONENT          FUNCTION
=========          ========
PostgreSQL         Source database (primary + replica)
CronJob            Scheduled backup execution
pg_dump            PostgreSQL native backup tool
aws-cli            S3 upload handler
LocalStack         Local S3 simulation
KIND               Local Kubernetes cluster

Project Structure

db-backup-s3/
├── Makefile                    # Build and test automation
├── docker-compose.yml          # LocalStack configuration
├── kind-config.yaml            # KIND cluster setup
├── postgres-deployment.yaml    # PostgreSQL deployments
├── backup-cron.yaml            # CronJob definition
├── local-backup-secret.yaml    # Secrets for testing
├── setup-localstack.sh         # S3 bucket setup
├── test-lab.sh                 # Full environment setup
├── verify-backup.sh            # Backup verification
└── README.md

Prerequisites

TOOL               INSTALLATION
====               ============
Docker             https://docs.docker.com/get-docker/
KIND               brew install kind
kubectl            brew install kubectl
awslocal           pip install awscli-local

Quick Start

# Clone the repository
git clone https://github.com/moabukar/db-backup-s3.git
cd db-backup-s3

# One-command setup and test
make quick

# Or step by step:
make setup          # Create KIND cluster + LocalStack + PostgreSQL
make test-working   # Run manual backup
make verify         # Verify backup integrity

Makefile Reference

RDS Backup Test Makefile
========================

Available targets:
  setup           - Setup KIND cluster, LocalStack, and PostgreSQL
  test            - Run manual backup test (cronjob-based)
  test-working    - Run working backup test (simple job)
  verify          - Verify backup integrity and restore
  status          - Show current environment status
  logs            - Show logs from most recent backup job
  logs-follow     - Follow logs of running backup job
  test-db         - Test database connectivity
  test-s3         - Test LocalStack S3 connectivity
  e2e             - Complete end-to-end test
  quick           - Quick test cycle (cleanup -> setup -> test -> verify)
  cleanup         - Remove all lab resources
  help            - Show this help

Quick start: make setup && make test-working && make verify

CronJob Configuration

The heart of the system - a Kubernetes CronJob that handles the backup:

# backup-cron.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: rds-backup-cronjob
  namespace: default
spec:
  schedule: "0 2 * * *"   # Daily at 2 AM UTC
  timeZone: "UTC"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 3
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: rds-backup
              image: amazon/aws-cli:latest
              command:
                - /bin/bash
                - -c
                - |
                  set -euo pipefail
                  
                  echo "Starting streaming backup..."
                  
                  # Install PostgreSQL client
                  yum update -y
                  yum install -y postgresql15
                  
                  # Set up environment
                  export PGPASSWORD=$(cat /secrets/db-password)
                  export AWS_ACCESS_KEY_ID=$(cat /aws-secrets/aws-access-key-id)
                  export AWS_SECRET_ACCESS_KEY=$(cat /aws-secrets/aws-secret-access-key)
                  export AWS_DEFAULT_REGION="us-east-1"
                  
                  DB_HOST="postgres-replica-service.default.svc.cluster.local"
                  TIMESTAMP=$(date +%F-%H-%M)
                  S3_PATH="s3://rds-db-backups/\$TIMESTAMP/backup.dump"
                  
                  echo "Target: \$DB_HOST"
                  echo "S3 destination: \$S3_PATH"
                  
                  # Verify database connection
                  pg_isready -h \$DB_HOST -p 5432 -U root
                  
                  # Create and upload backup
                  START_TIME=\$(date +%s)
                  pg_dump -h \$DB_HOST -U root -p 5432 \
                    --format=custom \
                    --blobs \
                    --no-password \
                    langfuse | aws s3 cp - \$S3_PATH
                  END_TIME=\$(date +%s)
                  
                  DURATION=\$((END_TIME - START_TIME))
                  echo "Backup completed in \${DURATION}s"
                  
                  # Verify upload
                  aws s3 ls \$S3_PATH
                  echo "BACKUP SUCCESSFUL"
                  
                  unset PGPASSWORD
              volumeMounts:
                - name: db-secrets
                  mountPath: /secrets
                  readOnly: true
                - name: aws-secrets
                  mountPath: /aws-secrets
                  readOnly: true
              resources:
                requests:
                  memory: "512Mi"
                  cpu: "500m"
                limits:
                  memory: "1Gi"
                  cpu: "1000m"
          volumes:
            - name: db-secrets
              secret:
                secretName: rds-db-root-secret
            - name: aws-secrets
              secret:
                secretName: aws-credentials

Key configuration points:

SETTING                VALUE                   NOTES
=======                =====                   =====
schedule               "0 2 * * *"             Daily at 2 AM UTC
concurrencyPolicy      Forbid                  Prevent overlapping jobs
failedJobsHistory      3                       Keep last 3 failed jobs
successfulJobsHistory  3                       Keep last 3 successful jobs
restartPolicy          OnFailure               Retry on transient errors

PostgreSQL Deployment

# postgres-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-replica
  labels:
    app: postgres-replica
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-replica
  template:
    metadata:
      labels:
        app: postgres-replica
    spec:
      containers:
      - name: postgres
        image: postgres:15
        env:
        - name: POSTGRES_DB
          value: langfuse
        - name: POSTGRES_USER
          value: root
        - name: POSTGRES_PASSWORD
          value: testpassword123
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: postgres-data
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-replica-service
spec:
  type: ClusterIP
  ports:
  - port: 5432
    targetPort: 5432
  selector:
    app: postgres-replica

Secrets Configuration

# local-backup-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: rds-db-root-secret
  namespace: default
type: Opaque
stringData:
  db-password: testpassword123
---
apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: default
type: Opaque
stringData:
  aws-access-key-id: test
  aws-secret-access-key: test

For production: Use external secrets management (AWS Secrets Manager, HashiCorp Vault, or Kubernetes External Secrets Operator).

LocalStack Setup

LocalStack simulates AWS S3 for local testing:

# docker-compose.yml
services:
  localstack:
    container_name: localstack
    image: localstack/localstack:3.0
    ports:
      - "4566:4566"
      - "4510-4559:4510-4559"
    environment:
      - DEBUG=1
      - SERVICES=s3,iam,sts
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "/tmp/localstack:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

S3 bucket setup script:

#!/bin/bash
# setup-localstack.sh

set -euo pipefail

echo "Setting up LocalStack S3..."

# Wait for LocalStack
while ! curl -s http://localhost:4566/health > /dev/null; do
  sleep 2
done

# Configure credentials (fake for LocalStack)
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# Create bucket
awslocal s3 mb s3://rds-db-backups-co-create

# Add lifecycle policy (move to Glacier after 30 days, delete after 365)
cat > lifecycle-policy.json << EOF
{
  "Rules": [
    {
      "ID": "move-to-glacier",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "Transitions": [
        { "Days": 30, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}
EOF

awslocal s3api put-bucket-lifecycle-configuration \
  --bucket rds-db-backups-co-create \
  --lifecycle-configuration file://lifecycle-policy.json

echo "LocalStack S3 setup complete!"

KIND Cluster Configuration

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: backup-test
nodes:
- role: control-plane
  image: kindest/node:v1.28.0
  extraPortMappings:
  - containerPort: 30080
    hostPort: 8080
    protocol: TCP
  - containerPort: 30432
    hostPort: 5432
    protocol: TCP

Backup Verification

Never trust a backup you haven’t tested. The verification script:

#!/bin/bash
# verify-backup.sh

set -euo pipefail

echo "Verifying backup integrity..."

export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# List available backups
echo "Available backups:"
awslocal s3 ls s3://rds-db-backups-co-create/ --recursive --human-readable

# Get latest backup
LATEST_BACKUP=$(awslocal s3 ls s3://rds-db-backups-co-create/ --recursive \
  | sort | tail -n 1 | awk '{print $4}')

if [ -z "$LATEST_BACKUP" ]; then
    echo "ERROR: No backups found!"
    exit 1
fi

# Download and restore
echo "Downloading: $LATEST_BACKUP"
awslocal s3 cp s3://rds-db-backups-co-create/$LATEST_BACKUP ./test-restore.dump

echo "Testing restore..."
kubectl exec deployment/postgres-replica -- dropdb -U root test_restore --if-exists
kubectl exec deployment/postgres-replica -- createdb -U root test_restore

kubectl exec -i deployment/postgres-replica -- pg_restore \
    -U root \
    -d test_restore \
    --verbose \
    --clean \
    --if-exists < ./test-restore.dump

# Verify data
echo "Verifying restored data..."
kubectl exec deployment/postgres-replica -- psql -U root -d test_restore -c "
SELECT 'Records restored:' as status, count(*) as count FROM test_backup;
"

echo "========================================"
echo "BACKUP VERIFICATION COMPLETE"
echo "========================================"
echo "  Streaming backup: SUCCESSFUL"
echo "  S3 upload:        SUCCESSFUL"
echo "  Data integrity:   VERIFIED"
echo "  Restore process:  WORKING"
echo "========================================"

Testing Commands

# LocalStack configuration (fake credentials)
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# List all backups with sizes
awslocal s3 ls s3://rds-db-backups-co-create/ --recursive --human-readable

# List backup folders only
awslocal s3 ls s3://rds-db-backups-co-create/

# Check specific backup (replace timestamp)
awslocal s3 ls s3://rds-db-backups-co-create/2025-09-02-09-10/ --human-readable

# Get file metadata
awslocal s3api head-object \
  --bucket rds-db-backups-co-create \
  --key "2025-09-02-09-10/langfuse_backup.dump"

# Test database connectivity
make test-db

# Test S3 connectivity
make test-s3

# View latest job logs
make logs

# Follow running job logs
make logs-follow

# Check environment status
make status

Production Considerations

1. Use Read Replicas

APPROACH           BENEFIT
========           =======
Backup from replica    No impact on primary performance
Dedicated backup user  Minimal privileges required
Connection pooling     Handle connection limits

2. Streaming to S3

For large databases, stream directly without local disk:

pg_dump -h $DB_HOST -U $USER -d $DATABASE \
  --format=custom \
  --blobs \
  | aws s3 cp - s3://bucket/path/backup.dump

3. Encryption

# Encrypt with GPG before upload
pg_dump ... | gpg --encrypt --recipient backup@company.com \
  | aws s3 cp - s3://bucket/backup.dump.gpg

# Or use S3 server-side encryption
aws s3 cp backup.dump s3://bucket/backup.dump \
  --sse aws:kms \
  --sse-kms-key-id alias/backup-key

4. Monitoring

# Add Prometheus metrics
- name: backup-exporter
  image: backup-exporter:latest
  ports:
  - containerPort: 9090

5. Alerting

# Alert on failed jobs
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: backup-alerts
spec:
  groups:
  - name: backup
    rules:
    - alert: BackupJobFailed
      expr: kube_job_status_failed{job_name=~"rds-backup.*"} > 0
      for: 5m

Cleanup

# Remove all lab resources
make cleanup

# This will:
# - Stop LocalStack container
# - Delete KIND cluster
# - Remove temporary files

Troubleshooting

CronJob not running:

kubectl get cronjobs
kubectl describe cronjob rds-backup-cronjob

Pod failing:

kubectl get pods
kubectl logs <pod-name>
kubectl describe pod <pod-name>

S3 upload failing:

# Test LocalStack connectivity
curl http://localhost:4566/health
awslocal s3 ls

Database connection failing:

kubectl exec deployment/postgres-replica -- \
  pg_isready -h localhost -p 5432 -U root

Repository

Full source code: https://github.com/moabukar/db-backup-s3

========================================
PostgreSQL --> K8s CronJob --> S3
========================================
Automated. Verified. Production-ready.
========================================
Found this helpful?

Related Posts

OpenTelemetry Changed How I Think About Observability

A practical, opinionated take on OpenTelemetry - why it matters, what it actually solves, and how to instrument across Kubernetes, Lambda, ECS, and EC2 without losing your mind.

Building an Automated Multi-Account AWS Architecture with Control Tower and Terraform

A hands-on walkthrough of enabling AWS Control Tower, designing an OU structure, automating account provisioning via Service Catalog, and deploying security baselines - from zero to fully automated account vending in production.

Spacelift from Scratch: Automating Terraform at Scale with Spaces, Stacks, OPA Policies, and a Private Module Registry

A complete guide to setting up Spacelift for multi-team Terraform automation - from zero to production with spaces, dynamic stacks, OPA security policies in Rego, private module registry, and GitOps-driven infrastructure.

Comments