Ceph CSI in Kubernetes: Bare-Metal Storage

When you run Kubernetes on bare metal, you lose the managed storage that cloud providers give you for free. No EBS volumes that just appear, no GCP Persistent Disks you can ignore until they're needed. You have to bring your own.

For my on-prem cloud, I chose Ceph: a distributed storage system that turns a pool of disks across multiple nodes into a single, redundant storage layer. Combined with the Ceph CSI driver, it gives Kubernetes dynamic PVC provisioning backed by real replicated block storage.

This post covers how I connected my Ceph cluster to Kubernetes, what the configuration looks like, and the one Cilium-specific gotcha that isn't documented anywhere and will block all volume mounts if you hit it.

---

Architecture

My Ceph cluster runs directly on the three Proxmox hypervisors, not inside Kubernetes. Each host contributes two disks to the OSD pool. The Ceph monitors — which coordinate cluster state — run at 192.168.22.11-13:6789.

The Ceph CSI driver runs inside Kubernetes as a DaemonSet (node plugin) and Deployment (controller), and it talks to the Ceph cluster over the network. This separation is intentional: if the Kubernetes cluster is unhealthy, the storage layer keeps running independently.

Proxmox (192.168.22.x)          Kubernetes (192.168.33.x)
──────────────────────          ──────────────────────────
Ceph monitors + OSDs     ←──→   Ceph CSI DaemonSet + Controller
pve1 .11  pve2 .12  pve3 .13          ↓
                                 StorageClass: ceph-rbd
                                 PVC → RBD image → pod volume

When a pod requests a PVC, the CSI controller creates an RBD image in Ceph. When the pod is scheduled, the node plugin mounts that image as a block device. When the pod is deleted, the image stays until the PVC is deleted.

---

Installing the CSI Driver

Install via Helm with your cluster's monitor addresses:

# ceph-csi-rbd/values.yaml
csiConfig:
  - clusterID: "<output of: ceph fsid>"
    monitors:
      - "192.168.22.11:6789"
      - "192.168.22.12:6789"
      - "192.168.22.13:6789"

The driver needs two Secrets — one for the controller (admin credentials to create and delete volumes) and one for nodes to authenticate when mounting:

apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi
stringData:
  userID: admin
  userKey: "<output of: ceph auth get-key client.admin>"

You can use a dedicated Ceph user with narrower permissions instead of admin — ceph auth get-or-create client.k8s with osd 'profile rbd pool=kubernetes' is the production-correct approach.

---

Preparing Ceph

Before the StorageClass can provision volumes, you need a pool on the Ceph side:

ceph osd pool create kubernetes 32
rbd pool init kubernetes

The 32 is the PG count — scale it up for larger clusters. For a 6-OSD on-prem cloud cluster, 32 is reasonable.

---

StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: "<your-cluster-id>"
  pool: kubernetes
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

imageFeatures: layering is important — without it, some Ceph versions fail to map the RBD image.

---

The Cilium Native Routing Gotcha

This is the part that isn't in any Ceph CSI tutorial, and it cost me a few hours.

When a Kubernetes node mounts an RBD volume, the CSI node plugin communicates with Ceph OSD daemons. OSDs don't use the monitor port (6789) — they have their own port range: 6802–6809.

In Cilium's native routing mode with NetworkPolicies applied, egress to external CIDRs is governed by policy. You might expect that a standard NetworkPolicy ipBlock rule covers it:

egress:
  - to:
      - ipBlock:
          cidr: 192.168.22.0/24
    ports:
      - port: 6789
        protocol: TCP

It doesn't — not reliably. The issue is that standard NetworkPolicy ipBlock operates at the IP layer, but Cilium in native routing mode uses eBPF maps that don't always apply these rules consistently for traffic that originates from the node plugin running as a privileged host-network pod.

The fix is a CiliumNetworkPolicy with toCIDR:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: ciliumnetpol-ceph-osd
  namespace: kube-system
spec:
  endpointSelector: {}
  egress:
    - toCIDR:
        - 192.168.22.0/24
      toPorts:
        - ports:
            - port: "6789"
              protocol: TCP
            - port: "6800"
              protocol: TCP
            - port: "6801"
              protocol: TCP
            - port: "6802"
              protocol: TCP
            - port: "6803"
              protocol: TCP
            - port: "6804"
              protocol: TCP
            - port: "6805"
              protocol: TCP
            - port: "6806"
              protocol: TCP
            - port: "6807"
              protocol: TCP
            - port: "6808"
              protocol: TCP
            - port: "6809"
              protocol: TCP

Apply this cluster-wide (empty endpointSelector: {} matches all pods). Without it, RBD volume mounts hang at ContainerCreating with no error message — the pod just waits indefinitely while the CSI plugin times out trying to map the image.

---

Verifying It Works

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-rbd-pvc
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi
EOF

kubectl get pvc test-rbd-pvc
# Expected: Bound

If it stays Pending, check the CSI controller logs:

kubectl logs -n ceph-csi \
  -l app=csi-rbdplugin-provisioner \
  -c csi-provisioner --tail=50

The most common failure messages and their causes:

Message	Cause
failed to connect to mon	Monitor unreachable — check firewall on Proxmox
unknown image feature 'fast-diff'	imageFeatures: layering missing
context deadline exceeded on mount	OSD ports blocked — apply the CiliumNetworkPolicy
invalid argument on pool	Run rbd pool init kubernetes

---

Once PVCs provision and bind reliably, Ceph RBD becomes the invisible foundation that everything else sits on — StatefulSets, databases, media libraries. The next piece is making sure the network policies that protect those workloads actually work the way you expect in Cilium native routing mode.