CloudNativePG: Running Postgres in Kubernetes

Using CloudNativePG for database-per-service Postgres — shared vs dedicated clusters, barman WAL backups to MinIO, and the kured drain gotcha.

The conventional wisdom used to be "don't run databases in Kubernetes." That advice made sense when the tooling wasn't there — before CSI drivers made persistent storage reliable, before operators existed to handle failover and replication automatically. The implied alternative was always "run Postgres on a dedicated VM and connect to it from Kubernetes," which trades one set of operational problems for another.

CloudNativePG (CNPG) changes the calculation. It's a Kubernetes operator that manages PostgreSQL clusters as first-class resources: streaming replication, automatic failover, point-in-time recovery, and integrated backup to object storage. I've been running it in production in my on-prem cloud for over a year, and it's the most solid piece of infrastructure in the cluster.

This post covers how I structure the database layer, what the configuration looks like end-to-end, and the operational lessons — including a drain gotcha that will silently break your node patching if you don't know about it.

Shared vs Dedicated Clusters

The first decision is whether each service gets its own Postgres cluster, or whether they share one.

I use both, and the split is based on a simple rule: share when the service is boring about its database needs, dedicate when it isn't.

pg-main — shared cluster runs in the cnpg-clusters namespace and hosts databases for Vaultwarden, Authentik, Nextcloud, Infisical, and Apicurio. Each gets a CREATE DATABASE and a user; they otherwise share the same Postgres instance. Four replicas, barman backup to MinIO. These are infrastructure applications that have no unusual database requirements — isolating them would add operational overhead with no benefit.

auth-service-pg in the yana-stocks namespace is dedicated to the auth microservice. Auth credentials shouldn't share a cluster with unrelated services (blast radius if the cluster is compromised), and the service needs its own migration lifecycle — its golang-migrate migrations run at startup, independent of anything else in the cluster.

immich-postgres is dedicated to Immich and uses a custom image: ghcr.io/tensorchord/cloudnative-vectorchord:16-1.1.1. This is a Postgres build with the vectorchord extension for AI-powered image similarity search — Immich uses it for face recognition clustering. Standard Postgres can't run this workload at all. A custom image with a dedicated cluster is the only option.

The Cluster CRD

A minimal CNPG cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: auth-service-pg
  namespace: yana-stocks
spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:16

  storage:
    size: 5Gi
    storageClass: ceph-rbd

  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "256MB"

  bootstrap:
    initdb:
      database: auth_service
      owner: auth_service
      secret:
        name: auth-service-pg-app

  backup:
    barmanObjectStore:
      destinationPath: s3://cnpg-backups/auth-service-pg/
      s3Credentials:
        accessKeyId:
          name: cnpg-minio-credentials
          key: MINIO_ACCESS_KEY_ID
        secretAccessKey:
          name: cnpg-minio-credentials
          key: MINIO_SECRET_KEY
      endpointURL: http://minio.minio.svc.cluster.local:9000
      wal:
        compression: gzip
    retentionPolicy: 30d

CNPG creates and manages:

  • A primary pod and replica pod (one per instances count)
  • Three Services: auth-service-pg-rw (primary), auth-service-pg-ro (replicas), auth-service-pg-r (any)
  • A PodDisruptionBudget on the primary (more on this below)
  • A Secret auth-service-pg-app with connection string, username, password, and host

Application pods connect to auth-service-pg-rw.yana-stocks.svc.cluster.local:5432. If the primary fails, CNPG promotes the replica and updates the -rw Service selector — the application reconnects automatically.

The spec.bootstrap.initdb.secret reference (auth-service-pg-app) must exist before the cluster is created. It should contain at minimum a password key. CNPG uses it to set the initial user password and then enriches the same Secret with connection details.

Backup: Barman + MinIO + WAL Streaming

The backup.barmanObjectStore block wires up continuous WAL archiving. Every WAL segment (16 MB by default) is shipped to MinIO as it's completed. This is what gives you point-in-time recovery.

For base backups (which WAL replay needs as a starting point), add a ScheduledBackup:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: auth-service-pg-daily
  namespace: yana-stocks
spec:
  schedule: "0 1 * * *"
  backupOwnerReference: self
  cluster:
    name: auth-service-pg

This runs a full base backup at 01:00 every day. Combined with continuous WAL archiving, you can restore to any second within the retention window.

The MinIO credentials come from an ExternalSecret that pulls from Infisical:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: cnpg-minio-credentials
  namespace: yana-stocks
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: infisical
    kind: ClusterSecretStore
  target:
    name: cnpg-minio-credentials
    creationPolicy: Owner
  data:
    - secretKey: MINIO_ACCESS_KEY_ID
      remoteRef:
        key: /cnpg-clusters/MINIO_ACCESS_KEY_ID
    - secretKey: MINIO_SECRET_KEY
      remoteRef:
        key: /cnpg-clusters/MINIO_SECRET_KEY

One credentials pair covers all CNPG clusters — they each write to different prefixes within the same cnpg-backups bucket. auth-service-pg writes to s3://cnpg-backups/auth-service-pg/, pg-main writes to s3://cnpg-backups/pg-main/, and so on.

The kured Drain Gotcha

This is the failure mode that will silently break your node patching strategy if you don't know about it.

kured is a DaemonSet that reboots Kubernetes nodes when the OS signals it's required (kernel updates, etc.). It drains the node first, waits for pods to be rescheduled, then reboots. Drain respects PodDisruptionBudgets.

CNPG automatically creates a PDB for the primary pod. With instances: 2, the PDB allows one disruption — kured can evict the primary, the replica gets promoted, the node drains and reboots cleanly.

With instances: 1, CNPG sets the PDB to 0 allowed disruptions. There is no replica to promote to; disrupting the single instance would mean downtime. So the PDB blocks all disruptions.

The result: kured's drain request is permanently refused. The node never reboots. You don't see an error — kured just sits there indefinitely waiting for the drain to complete. Your cluster quietly accumulates unpatched nodes.

The rule is simple: always use instances: 2 or more, even for low-traffic databases where you don't need the replica for read scaling. The replica exists so the PDB allows disruptions, not for load distribution.

# Check PDB status if a drain appears stuck
kubectl get pdb -A

# Look for any ALLOWED DISRUPTIONS: 0
# Then check how many instances that cluster has:
kubectl get cluster -A

If you find a single-instance cluster, scaling it to 2 instances is a live operation — CNPG adds the replica without downtime.

Migrations at Startup

For the auth-service, database migrations run at pod startup using golang-migrate — not in an initContainer, not in a Job.

func runMigrations(db *sql.DB) error {
    driver, _ := postgres.WithInstance(db, &postgres.Config{})
    m, err := migrate.NewWithDatabaseInstance(
        "file://migrations",
        "postgres",
        driver,
    )
    if err != nil {
        return fmt.Errorf("creating migrator: %w", err)
    }
    if err := m.Up(); err != nil && err != migrate.ErrNoChange {
        return fmt.Errorf("running migrations: %w", err)
    }
    return nil
}

This works reliably because:

  1. Migrations are idempotent — golang-migrate tracks applied versions in schema_migrations and skips already-applied migrations.
  2. CNPG's readiness probe gates the -rw Service until Postgres is accepting connections, so the migration has a healthy database to connect to by the time the pod starts.
  3. If the migration fails, the pod crashes and Kubernetes restarts it. The restart loop makes failures visible immediately in pod events rather than being silently skipped.

The initContainer approach has one advantage: it runs before the main container, so migration failures stop the pod from starting. In practice, golang-migrate at startup achieves the same effect — a migration failure crashes the pod — with less YAML.

Point-in-Time Recovery

If you need to restore a cluster to a specific timestamp:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: auth-service-pg-restored
  namespace: yana-stocks
spec:
  instances: 2
  bootstrap:
    recovery:
      source: auth-service-pg-backup
      recoveryTarget:
        targetTime: "2026-07-01 03:00:00.000000+00"
  externalClusters:
    - name: auth-service-pg-backup
      barmanObjectStore:
        destinationPath: s3://cnpg-backups/auth-service-pg/
        s3Credentials:
          accessKeyId:
            name: cnpg-minio-credentials
            key: MINIO_ACCESS_KEY_ID
          secretAccessKey:
            name: cnpg-minio-credentials
            key: MINIO_SECRET_KEY
        endpointURL: http://minio.minio.svc.cluster.local:9000

CNPG finds the most recent base backup before the target time, restores it, then replays WAL segments until it hits the timestamp. No manual pg_restore, no shell scripts, no temporary VMs. It's the most satisfying part of the whole setup.

Checking Cluster Health

# Status of all CNPG clusters
kubectl get cluster -A

# Detailed status of a specific cluster
kubectl describe cluster auth-service-pg -n yana-stocks

# Check replication lag on replicas
kubectl exec -n yana-stocks auth-service-pg-1 -- \
  psql -U postgres -c "SELECT * FROM pg_stat_replication;"

# WAL archiving status
kubectl exec -n yana-stocks auth-service-pg-1 -- \
  psql -U postgres -c "SELECT * FROM pg_stat_archiver;"

The pg_stat_archiver query is particularly useful — it shows whether WAL archiving to MinIO is working, how many files have been archived, and when the last successful archive happened. If last_failed_time is recent and last_failed_wal is non-empty, your backups have stopped working.

Running Postgres in Kubernetes was the decision I was most sceptical about when I started building this cluster. After a year of operating CNPG — including a few actual failover events, one PITR restore, and daily node drains — I wouldn't go back to a dedicated VM. The operator handles the things that are genuinely hard about Postgres operations, and it does it in the same GitOps workflow as everything else in the cluster.

No ratings yet

Comments

0/2000

Connect

LinkedIn ↗akannkweiniGitHub ↗akann