Self-Hosted GitLab on Kubernetes - A Startup’s Journey

When we hit 50 engineers at the startup, GitLab.com’s pricing started to sting. The Premium tier at $29/user/month meant we were looking at $17,400/year just for source control and CI/CD. For a startup watching every pound, that’s significant.

We decided to self-host GitLab on our existing AKS clusters. This post documents the complete setup - the architecture decisions, Helm configuration, Azure SQL integration, and the lessons we learned along the way.

Why Self-Host?

The numbers:

• GitLab Premium (50 users): ~$17,400/year
• Self-hosted on existing K8s: ~$3,600/year (compute + storage)
• Savings: ~$13,800/year

Other benefits:

• Full control over data (compliance requirement for us)
• No rate limits on CI/CD
• Custom runners on our own infrastructure
• Integration with internal services

The trade-offs:

• Operational overhead (upgrades, backups, monitoring)
• Need K8s expertise
• You own the uptime

For a startup with a competent platform team, the trade-off made sense.

Code Repository: All code from this post is available at github.com/moabukar/blog-code/self-hosted-gitlab-kubernetes

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                         AKS Cluster                              │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    gitlab namespace                      │    │
│  │                                                          │    │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐              │    │
│  │  │ Webservice│  │ Sidekiq  │  │  Gitaly  │              │    │
│  │  │ (Rails)   │  │ (Jobs)   │  │ (Git RPC)│              │    │
│  │  └────┬─────┘  └────┬─────┘  └────┬─────┘              │    │
│  │       │              │              │                    │    │
│  │  ┌────┴──────────────┴──────────────┴────┐              │    │
│  │  │              Redis Cluster             │              │    │
│  │  │         (Azure Cache for Redis)        │              │    │
│  │  └───────────────────────────────────────┘              │    │
│  │                                                          │    │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐              │    │
│  │  │ Registry │  │   Shell  │  │  Toolbox │              │    │
│  │  │ (Images) │  │  (SSH)   │  │  (Rails) │              │    │
│  │  └──────────┘  └──────────┘  └──────────┘              │    │
│  │                                                          │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                  │
│  ┌──────────────────┐     ┌──────────────────────────────┐     │
│  │ Ingress (NGINX)  │     │   Azure Files (Persistent)   │     │
│  │ + Cert-Manager   │     │   - Git repos (Gitaly)       │     │
│  └──────────────────┘     │   - LFS objects              │     │
│                            │   - Uploads                  │     │
└────────────────────────────┴──────────────────────────────┴─────┘
              │                              │
              │                              │
┌─────────────┴───────────┐    ┌────────────┴─────────────┐
│      Azure SQL          │    │   Azure Blob Storage     │
│   (PostgreSQL Flexible) │    │   - Backups              │
│   - gitlab_production   │    │   - CI artifacts         │
│   - gitlab_geo (if DR)  │    │   - Terraform state      │
└─────────────────────────┘    └──────────────────────────┘

Key Decisions

1. External PostgreSQL (Azure SQL) - GitLab’s bundled PostgreSQL is fine for small installs, but for production we wanted managed backups, HA, and point-in-time recovery.

2. External Redis (Azure Cache) - Same reasoning. Plus, Redis is critical for GitLab - Sidekiq jobs, caching, sessions.

3. Azure Files for Gitaly - Git repositories need persistent storage. Azure Files Premium (NFS) gave us the IOPS we needed.

4. Azure Blob for artifacts - CI artifacts and LFS objects go to blob storage. Cheaper and scales infinitely.

---

Prerequisites

Before starting:

# AKS cluster running (we used 1.28)
# kubectl configured
# Helm 3.x installed
# Domain name ready (gitlab.yourcompany.com)
# SSL certificate (we used cert-manager with Let's Encrypt)

# Create namespace
kubectl create namespace gitlab

# Add GitLab Helm repo
helm repo add gitlab https://charts.gitlab.io/
helm repo update

---

Step 1: Set Up Azure SQL (PostgreSQL)

Create the Database Server

# Create resource group (if not exists)
az group create --name rg-gitlab-prod --location uksouth

# Create PostgreSQL Flexible Server
az postgres flexible-server create \
  --resource-group rg-gitlab-prod \
  --name gitlab-postgres-prod \
  --location uksouth \
  --admin-user gitlabadmin \
  --admin-password 'YourSecurePassword123!' \
  --sku-name Standard_D4s_v3 \
  --tier GeneralPurpose \
  --storage-size 256 \
  --version 14 \
  --high-availability ZoneRedundant \
  --public-access None  # We'll use private endpoint

Configure Private Endpoint

# Create private endpoint for PostgreSQL
az network private-endpoint create \
  --resource-group rg-gitlab-prod \
  --name gitlab-postgres-pe \
  --vnet-name aks-vnet \
  --subnet aks-subnet \
  --private-connection-resource-id $(az postgres flexible-server show \
    --resource-group rg-gitlab-prod \
    --name gitlab-postgres-prod \
    --query id -o tsv) \
  --group-id postgresqlServer \
  --connection-name gitlab-postgres-connection

Create the Database

# Connect to PostgreSQL
az postgres flexible-server connect \
  --name gitlab-postgres-prod \
  --resource-group rg-gitlab-prod \
  --admin-user gitlabadmin \
  --admin-password 'YourSecurePassword123!'

# Create database
CREATE DATABASE gitlab_production;

# Create GitLab user with limited privileges
CREATE USER gitlab WITH ENCRYPTED PASSWORD 'GitLabUserPassword123!';
GRANT ALL PRIVILEGES ON DATABASE gitlab_production TO gitlab;

# Required extensions
\c gitlab_production
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS btree_gist;
CREATE EXTENSION IF NOT EXISTS plpgsql;

PostgreSQL Configuration

GitLab needs specific PostgreSQL settings:

# Via Azure CLI
az postgres flexible-server parameter set \
  --resource-group rg-gitlab-prod \
  --server-name gitlab-postgres-prod \
  --name shared_preload_libraries \
  --value pg_stat_statements

az postgres flexible-server parameter set \
  --resource-group rg-gitlab-prod \
  --server-name gitlab-postgres-prod \
  --name max_connections \
  --value 200

---

Step 2: Set Up Azure Cache for Redis

# Create Redis Cache (Premium for clustering)
az redis create \
  --resource-group rg-gitlab-prod \
  --name gitlab-redis-prod \
  --location uksouth \
  --sku Premium \
  --vm-size P1 \
  --enable-non-ssl-port false \
  --minimum-tls-version 1.2

# Get connection details
az redis show \
  --resource-group rg-gitlab-prod \
  --name gitlab-redis-prod \
  --query "[hostName, sslPort, accessKeys.primaryKey]" -o tsv

---

Step 3: Set Up Azure Blob Storage

# Create storage account
az storage account create \
  --resource-group rg-gitlab-prod \
  --name gitlabstorageprod \
  --location uksouth \
  --sku Standard_ZRS \
  --kind StorageV2

# Create containers
az storage container create --name gitlab-artifacts --account-name gitlabstorageprod
az storage container create --name gitlab-uploads --account-name gitlabstorageprod
az storage container create --name gitlab-lfs --account-name gitlabstorageprod
az storage container create --name gitlab-packages --account-name gitlabstorageprod
az storage container create --name gitlab-backups --account-name gitlabstorageprod
az storage container create --name gitlab-registry --account-name gitlabstorageprod

# Get connection string
az storage account show-connection-string \
  --resource-group rg-gitlab-prod \
  --name gitlabstorageprod \
  --query connectionString -o tsv

---

Step 4: Create Kubernetes Secrets

# PostgreSQL password
kubectl create secret generic gitlab-postgresql-password \
  --namespace gitlab \
  --from-literal=postgresql-password='GitLabUserPassword123!'

# Redis password
kubectl create secret generic gitlab-redis-password \
  --namespace gitlab \
  --from-literal=redis-password='YourRedisPassword'

# Azure Storage credentials
kubectl create secret generic gitlab-azure-storage \
  --namespace gitlab \
  --from-literal=connection='DefaultEndpointsProtocol=https;AccountName=gitlabstorageprod;AccountKey=YOUR_KEY;EndpointSuffix=core.windows.net'

# GitLab Rails secret (generate a random one)
kubectl create secret generic gitlab-rails-secret \
  --namespace gitlab \
  --from-literal=secret='$(openssl rand -hex 64)'

# Initial root password
kubectl create secret generic gitlab-initial-root-password \
  --namespace gitlab \
  --from-literal=password='YourGitLabRootPassword123!'

---

Step 5: The Helm Values File

This is the critical part. Here’s our production values.yaml:

# values-production.yaml

global:
  # Domain configuration
  hosts:
    domain: yourcompany.com
    gitlab:
      name: gitlab.yourcompany.com
      https: true
    registry:
      name: registry.yourcompany.com
      https: true
    minio:
      enabled: false  # We use Azure Blob instead
    
  # Ingress configuration
  ingress:
    class: nginx
    annotations:
      kubernetes.io/tls-acme: "true"
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "900"
    configureCertmanager: false  # We manage cert-manager separately
    tls:
      enabled: true
      secretName: gitlab-tls

  # Time zone
  time_zone: Europe/London

  # Email configuration
  email:
    from: gitlab@yourcompany.com
    display_name: GitLab
    reply_to: noreply@yourcompany.com

  smtp:
    enabled: true
    address: smtp.sendgrid.net
    port: 587
    authentication: plain
    user_name: apikey
    password:
      secret: gitlab-smtp-password
      key: password
    starttls_auto: true

  # ============================================
  # EXTERNAL POSTGRESQL (Azure SQL)
  # ============================================
  psql:
    host: gitlab-postgres-prod.postgres.database.azure.com
    port: 5432
    database: gitlab_production
    username: gitlab
    password:
      secret: gitlab-postgresql-password
      key: postgresql-password
    ssl:
      enabled: true
      # Azure requires SSL
    
  # ============================================
  # EXTERNAL REDIS (Azure Cache)
  # ============================================
  redis:
    host: gitlab-redis-prod.redis.cache.windows.net
    port: 6380
    password:
      enabled: true
      secret: gitlab-redis-password
      key: redis-password
    scheme: rediss  # SSL

  # ============================================
  # GITALY (Git repository storage)
  # ============================================
  gitaly:
    enabled: true
    authToken:
      secret: gitlab-gitaly-secret
      key: token
    internal:
      names:
        - default
    external: []

  # ============================================
  # OBJECT STORAGE (Azure Blob)
  # ============================================
  minio:
    enabled: false  # Disable bundled MinIO

  appConfig:
    # LFS
    lfs:
      enabled: true
      proxy_download: true
      bucket: gitlab-lfs
      connection:
        secret: gitlab-rails-storage
        key: connection

    # Artifacts  
    artifacts:
      enabled: true
      proxy_download: true
      bucket: gitlab-artifacts
      connection:
        secret: gitlab-rails-storage
        key: connection

    # Uploads
    uploads:
      enabled: true
      proxy_download: true
      bucket: gitlab-uploads
      connection:
        secret: gitlab-rails-storage
        key: connection

    # Packages
    packages:
      enabled: true
      proxy_download: true
      bucket: gitlab-packages
      connection:
        secret: gitlab-rails-storage
        key: connection

    # Backups
    backups:
      bucket: gitlab-backups
      tmpBucket: gitlab-backups-tmp

    # Object storage connection template (Azure)
    object_store:
      enabled: true
      proxy_download: true
      storage_options: {}
      connection:
        secret: gitlab-rails-storage
        key: connection

  # ============================================
  # REGISTRY
  # ============================================
  registry:
    enabled: true
    bucket: gitlab-registry
    storage:
      secret: gitlab-registry-storage
      key: config

# ============================================
# DISABLE BUNDLED COMPONENTS
# ============================================
postgresql:
  install: false  # Using Azure SQL

redis:
  install: false  # Using Azure Cache

minio:
  install: false  # Using Azure Blob

# ============================================
# CERTMANAGER (we manage separately)
# ============================================
certmanager:
  install: false

# ============================================
# NGINX INGRESS (we manage separately)
# ============================================
nginx-ingress:
  enabled: false

# ============================================
# PROMETHEUS (optional - we use Azure Monitor)
# ============================================
prometheus:
  install: false

# ============================================
# GITLAB COMPONENTS
# ============================================

# Webservice (Rails application)
gitlab:
  webservice:
    replicaCount: 2
    minReplicas: 2
    maxReplicas: 10
    resources:
      requests:
        cpu: 900m
        memory: 2.5G
      limits:
        cpu: 2
        memory: 4G
    workerProcesses: 2
    workhorse:
      resources:
        requests:
          cpu: 100m
          memory: 100M
        limits:
          cpu: 500m
          memory: 500M
    hpa:
      targetAverageValue: 400m

  # Sidekiq (background jobs)
  sidekiq:
    replicas: 2
    minReplicas: 2
    maxReplicas: 10
    resources:
      requests:
        cpu: 500m
        memory: 2G
      limits:
        cpu: 2
        memory: 4G
    hpa:
      targetAverageValue: 350m
    pods:
      - name: all-in-1
        concurrency: 25
        queues: 

  # Gitaly (Git operations)
  gitaly:
    persistence:
      enabled: true
      size: 500Gi
      storageClass: azurefile-premium  # Azure Files Premium
    resources:
      requests:
        cpu: 300m
        memory: 1.5G
      limits:
        cpu: 2
        memory: 4G

  # GitLab Shell (SSH)
  gitlab-shell:
    replicaCount: 2
    minReplicas: 2
    maxReplicas: 4
    resources:
      requests:
        cpu: 50m
        memory: 32M
      limits:
        cpu: 500m
        memory: 128M

  # Toolbox (Rails console, backups)
  toolbox:
    enabled: true
    replicas: 1
    backups:
      cron:
        enabled: true
        schedule: "0 2 * * *"  # 2 AM daily
        persistence:
          enabled: true
          size: 100Gi
          storageClass: azurefile-premium
      objectStorage:
        config:
          secret: gitlab-rails-storage
          key: connection

  # Migrations (database migrations)
  migrations:
    enabled: true

  # GitLab Exporter (metrics)
  gitlab-exporter:
    enabled: true
    resources:
      requests:
        cpu: 50m
        memory: 100M
      limits:
        cpu: 200m
        memory: 256M

# Registry
registry:
  enabled: true
  replicas: 2
  hpa:
    minReplicas: 2
    maxReplicas: 5
  storage:
    secret: gitlab-registry-storage
    key: config
  resources:
    requests:
      cpu: 100m
      memory: 128M
    limits:
      cpu: 500m
      memory: 512M

# ============================================
# SHARED SETTINGS
# ============================================
shared-secrets:
  enabled: true
  rbac:
    create: true

---

Step 6: Azure Storage Connection Secret

Create the storage connection file:

# gitlab-rails-storage.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-rails-storage
  namespace: gitlab
type: Opaque
stringData:
  connection: |
    provider: AzureRM
    azure_storage_account_name: gitlabstorageprod
    azure_storage_access_key: YOUR_STORAGE_ACCOUNT_KEY
    azure_storage_domain: blob.core.windows.net

For the registry:

# gitlab-registry-storage.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-registry-storage
  namespace: gitlab
type: Opaque
stringData:
  config: |
    azure:
      accountname: gitlabstorageprod
      accountkey: YOUR_STORAGE_ACCOUNT_KEY
      container: gitlab-registry
      rootdirectory: /

Apply the secrets:

kubectl apply -f gitlab-rails-storage.yaml
kubectl apply -f gitlab-registry-storage.yaml

---

Step 7: Deploy GitLab

# Install GitLab
helm upgrade --install gitlab gitlab/gitlab \
  --namespace gitlab \
  --timeout 600s \
  --values values-production.yaml

# Watch the deployment
kubectl -n gitlab get pods -w

# Check for issues
kubectl -n gitlab get events --sort-by='.lastTimestamp'

First deployment takes 10-15 minutes. Watch for all pods to become Ready.

---

Step 8: Post-Installation

Get the Root Password

kubectl -n gitlab get secret gitlab-initial-root-password \
  -o jsonpath="{.data.password}" | base64 -d && echo

Access GitLab

1. Navigate to https://gitlab.yourcompany.com
2. Log in as root with the password above
3. Immediately change the root password
4. Disable sign-ups (Admin → Settings → General → Sign-up restrictions)

Create Your First User

# Via Rails console
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- gitlab-rails console

# In console:
user = User.new(username: 'admin', email: 'admin@yourcompany.com', name: 'Admin User', password: 'securepassword', password_confirmation: 'securepassword')
user.admin = true
user.skip_confirmation!
user.save!

---

Lessons Learned

1. Gitaly Storage is Critical

We initially used Azure Files Standard. Big mistake. Git operations were slow, and git clone on large repos took forever.

Fix: Use Azure Files Premium (NFS) with high IOPS. The cost difference is worth it.

# Storage class for Gitaly
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-premium
provisioner: file.csi.azure.com
parameters:
  skuName: Premium_LRS
  protocol: nfs
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true

2. Sidekiq Queues Matter

We started with a single Sidekiq pod handling all queues. CI jobs were slow because they competed with everything else.

Fix: Dedicate Sidekiq pods to specific queue groups:

gitlab:
  sidekiq:
    pods:
      - name: urgent
        concurrency: 10
        queues:
          - pipeline_processing
          - pipeline_default
          - pipeline_cache
      - name: default
        concurrency: 25
        queues:
      - name: slow
        concurrency: 5
        queues:
          - cronjob
          - repository_archive_cache

3. PostgreSQL Connection Limits

GitLab is connection-hungry. We hit Azure SQL’s connection limit during peak hours.

Fix:

• Set max_connections: 200 on PostgreSQL
• Use PgBouncer (GitLab Helm chart can deploy it):

global:
  psql:
    host: gitlab-pgbouncer  # PgBouncer service
    
gitlab:
  pgbouncer:
    enabled: true
    replicas: 2

4. Registry Garbage Collection

Container registry grows fast. Without cleanup, storage costs explode.

Fix: Enable registry garbage collection:

# Run GC manually
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- \
  gitlab-ctl registry-garbage-collect -m

# Or schedule it via cron job

5. Backup Testing

We set up backups but never tested restores. When we needed to restore a deleted project, we discovered our backup was incomplete.

Fix: Test restores monthly:

# Create backup
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- backup-utility

# Restore (to test environment)
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- backup-utility --restore

6. Resource Requests Were Wrong

Initial deployment used GitLab’s default resource requests. Pods were constantly OOMKilled.

Fix: Monitor actual usage for 2 weeks, then right-size:

# Get actual resource usage
kubectl -n gitlab top pods

# Check OOMKills
kubectl -n gitlab get events | grep -i oom

7. Upgrade Path Matters

GitLab doesn’t support skipping major versions. We tried to go from 15.x to 16.x directly and broke migrations.

Fix: Follow the upgrade path strictly:

• 15.11 → 16.0 → 16.3 → 16.7 → 16.11 → 17.x

Check GitLab’s upgrade path tool.

---

Monitoring

Key Metrics to Watch

# Prometheus rules (if using)
groups:
  - name: gitlab
    rules:
      - alert: GitLabSidekiqQueueBacklog
        expr: sidekiq_queue_size > 1000
        for: 10m
        
      - alert: GitLabGitalyHighLatency
        expr: gitaly_service_client_requests_seconds_bucket{le="1"} < 0.95
        for: 5m
        
      - alert: GitLabPostgreSQLConnections
        expr: pg_stat_activity_count > 180
        for: 5m

Useful Commands

# Check GitLab component health
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- gitlab-rake gitlab:check

# Check background jobs
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- gitlab-rake gitlab:sidekiq:check

# Database migrations status
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- gitlab-rake db:migrate:status

# Rails console (for debugging)
kubectl -n gitlab exec -it deploy/gitlab-toolbox -- gitlab-rails console

---

Cost Breakdown

Our monthly costs (50 users, moderate CI usage):

Component	SKU	Monthly Cost
Azure SQL (PostgreSQL)	D4s_v3, HA	~£280
Azure Cache (Redis)	P1	~£140
Azure Files Premium	500GB	~£85
Azure Blob Storage	~200GB	~£10
AKS Node Pool (dedicated)	2x D4s_v3	~£240
Total		~£755/month

vs GitLab Premium: ~£1,450/month

Savings: £700/month (£8,400/year)

---

Production Checklist

## Pre-Deployment
- [ ] Azure SQL created with HA enabled
- [ ] Redis Cache created (Premium)
- [ ] Blob Storage containers created
- [ ] Private endpoints configured
- [ ] SSL certificates ready
- [ ] DNS records configured

## Helm Configuration
- [ ] External PostgreSQL configured
- [ ] External Redis configured
- [ ] Object storage (Azure) configured
- [ ] Gitaly persistence configured
- [ ] Registry storage configured
- [ ] SMTP configured
- [ ] Resource requests/limits set
- [ ] HPA configured

## Post-Deployment
- [ ] Root password changed
- [ ] Sign-ups disabled
- [ ] First admin user created
- [ ] SSO/LDAP configured (if using)
- [ ] Backup cron job verified
- [ ] Backup restore tested
- [ ] Monitoring alerts configured
- [ ] Runner(s) registered

## Ongoing
- [ ] Monthly backup restore test
- [ ] Registry garbage collection scheduled
- [ ] Upgrade path documented
- [ ] Runbook for common issues

---

Key Takeaways

1. Use external PostgreSQL and Redis - The bundled ones are fine for testing, not production
2. Azure Files Premium for Gitaly - Don’t skimp on Git storage IOPS
3. Right-size after observing - GitLab’s defaults are conservative
4. Test your backups - Untested backups aren’t backups
5. Follow upgrade paths - GitLab migrations are version-sensitive
6. Monitor Sidekiq queues - They’re the first sign of trouble
7. Budget for the ops time - Self-hosting isn’t “set and forget”

Self-hosted GitLab on Kubernetes is absolutely viable for startups, but go in with eyes open. The cost savings are real, but so is the operational overhead.

---

Running GitLab on K8s? Hit any interesting issues? Find me on LinkedIn or GitHub.