Building a 6-Node Kubernetes On-Prem Cloud

I spend a lot of time in Kubernetes professionally. But there's a difference between using Kubernetes at work — where someone else set it up, maintains the control plane, and has an SRE team on call — and building one from scratch at home, where you are the SRE, the on-call engineer, and the person who has to figure out why your Ceph monitors are dropping sessions at 2am.

This is the story of my on-prem cloud Kubernetes cluster, what I've learned from it, and why I think every serious infrastructure engineer should have one.

The Setup

All three Proxmox hosts are identical MINISFORUM MS-01 Mini Workstations — compact Intel 12th-gen mini-PCs with dual SFP+ 10GbE and dual 2.5GbE, purpose-built for clustering.

MINISFORUM MS-01 front and rear panel — MINISFORUM MS-01 — the same unit ×3

The physical rack housing the three Proxmox nodes and networking gear — The rack — three MS-01s, the TP-Link switch, and the router

The network is split across two VLANs, configured on both pfSense and the TP-Link SG2008 switch:

VLAN	Subnet	Purpose
22	192.168.22.0/24	Proxmox management — PVE hosts, Ceph monitors
33	192.168.33.0/24	Kubernetes VM network — all k8s nodes

Each Proxmox host connects to the switch via two 2.5GbE ports: enp87s0 on VLAN 22 for management, enp90s0 on VLAN 33 for Kubernetes traffic. The 10GbE SFP+ ports run direct DAC cables between nodes for Ceph replication and Corosync — they bypass the switch entirely. pfSense trunks both VLANs and is the gateway between them.

Six nodes running on these three Proxmox hypervisors, kubeadm-bootstrapped, with:

3 control plane nodes (k8s-cp-1/2/3) — each on a separate Proxmox host for HA
3 worker nodes (k8s-worker-1/2/3)
Cilium as the CNI in native routing mode (no VXLAN encapsulation)
Ceph RBD as the default StorageClass, managed by the Ceph CSI driver
ArgoCD for GitOps — everything in git, nothing applied manually
MetalLB for bare-metal LoadBalancer IPs
cert-manager for TLS via Let's Encrypt DNS-01 (Cloudflare)
External Secrets Operator pulling from Infisical

The cluster hosts real workloads — several personal applications, a NestJS/Go/Python microservices stack, Kafka, CNPG PostgreSQL clusters, and this website.

Proxmox cluster network topology — three MS-01 nodes connected via 10GbE SFP+ DAC and 2.5GbE switch — Network topology: 10GbE SFP+ direct links (DAC cables) for Ceph and Corosync, 2.5GbE switch for management and Kubernetes VM traffic

The Things You Don't Learn from Tutorials

1. Asymmetric routing will bite you in unusual ways

My Proxmox hosts are on 192.168.22.x and the Kubernetes VMs are on 192.168.33.x, both reachable via a pfSense router. Ceph monitors run on the Proxmox hosts.

The symptom was mysterious: Ceph OSD sessions would drop intermittently. The Kubernetes nodes could ping the Ceph monitors, but the TCP sessions weren't surviving.

The root cause: asymmetric routing. Traffic from 192.168.33.x (k8s nodes) would reach the Ceph monitors at 192.168.22.x via pfSense, but the return path — from the Proxmox hosts back to the k8s VMs — was going directly via the Proxmox bridge (vmbr1) rather than back through pfSense. The stateful firewall saw the return packets as unsolicited and dropped them.

Fix: Add static routes on all Proxmox hosts routing 192.168.33.0/24 via pfSense (192.168.22.1), not directly via vmbr1. Obvious in hindsight; took a while to diagnose.

2. Cilium native routing and webhooks don't always get along

Cilium in native routing mode is great — no overlay overhead, packets route directly between nodes at line speed. But it changes where the kube-apiserver makes connections to in-cluster services.

In native routing, the apiserver doesn't go through a NAT/proxy layer for in-cluster services; it routes directly to the pod IP. In my setup this means apiserver connections to admission webhooks traverse a real path that takes ~10 seconds when the webhook pod is on a specific node.

Symptom: Kong's ValidatingWebhookConfiguration has its timeoutSeconds set to 10. With a ~10s actual call latency, cert-manager's SSA PATCH on TLS secrets would hit context deadline exceeded. And Strimzi's fabric8 HTTP client would time out on Service webhook calls, causing crash loops.

Fix: Patch timeoutSeconds to 5 on the offending webhooks. They have failurePolicy: Ignore, so a timeout becomes a fast-fail instead of a hang. Persist the patch via ignoreDifferences in ArgoCD so it survives resyncs.

3. Static pod manifests on control planes should point at themselves

When I set up CP-2 and CP-3, the scheduler.conf and controller-manager.conf on those nodes were pointing to the management IP of CP-1 (192.168.22.x) rather than localhost. So for the lifetime of those nodes, all scheduling and controller work was being routed to CP-1.

Nothing visibly broke — CP-1's scheduler handled the load fine — but it was wasteful, created a single point of failure in the scheduler/controller layer, and would have caused problems if CP-1 ever went down.

Lesson: Always check the kubeconfig files in /etc/kubernetes/ on each control plane node. server: should be 127.0.0.1:6443 or the node's own IP, never another node's IP.

What I Run on It

Beyond infrastructure, the cluster hosts:

yana-stocks — a microservices trading signals app I'm building (Go auth service, NestJS portfolio/price services, Python ML predictor with Argo Rollouts canary)
Immich — self-hosted Google Photos replacement, with a dedicated CNPG Postgres cluster using the vectorchord extension
Nextcloud, Vaultwarden, Authentik, Grafana, Kafka, MinIO — the usual on-prem cloud stack
This website, deployed via ArgoCD from a private GitHub repo

The full breakdown of every service — what it does, how it's configured, and the non-obvious operational details — is on the Stack page.

Backups run via Velero to Backblaze B2 (PVC data) and barman to MinIO (Postgres WAL streaming + daily snapshots).

Why Bother?

The obvious answer is "learning." But that undersells it.

What you get from an on-prem cloud cluster that you cannot get from managed Kubernetes:

You see the full stack. Every problem is yours to diagnose. There's no "open a support ticket." You learn networking, storage, TLS, admission control, and all the ways they interact.
You feel the operational weight. When you're on-call for your own cluster — even an on-prem cloud — you start understanding why SRE teams care so much about runbooks, ignoreDifferences, and persistent change management.
You build real intuition. When I see a context deadline exceeded in cert-manager now, I immediately think "webhook latency." That instinct comes from having diagnosed it in my own cluster.
It's a playground for production patterns. ArgoCD GitOps, External Secrets Operator, CNPG, Argo Rollouts — I run all of these at home, so when I encounter them at work, I'm not reading the docs from scratch.

If you're a senior engineer who works with Kubernetes professionally and doesn't have an on-prem cloud: I'd genuinely recommend it. Start with a single node on an old PC or a couple of cheap VMs. The investment pays back quickly.

The rest of the series goes deeper on each layer — Ceph CSI, Cilium network policies, CloudNativePG, ArgoCD, and Authentik SSO.