From Flannel to Cilium: A Live CNI Migration

Every post in this series describes the cluster as it is today: Cilium in native routing mode, default-deny network policies everywhere, Hubble for observability. What none of them mention is that the cluster didn't start that way. It started on Flannel — and this post is about why that stopped being good enough, and what the migration actually looked like.

---

Flannel was the right first choice

Let me be fair to Flannel first. If you're standing up your first kubeadm cluster, Flannel is genuinely great: one manifest, no configuration decisions, and pods are talking to each other five minutes later. When the goal is "get a working cluster so I can learn everything else," that simplicity is exactly what you want. I don't regret starting there.

Here's what that setup looks like under the hood:

Flannel builds a VXLAN overlay: every packet leaving a pod goes through the cni0 bridge into the flannel.1 interface, which wraps it in an outer UDP packet (port 8472) addressed node-to-node. The receiving node unwraps it and delivers it to the destination pod. The underlying network never sees pod IPs — it just sees UDP between nodes.

That design is why Flannel needs zero configuration. It's also the root of everything that eventually made me leave.

---

The three breaking points

1. NetworkPolicy silently does nothing

This is the one that actually forced the decision. Flannel does not implement network policy — at all. You can apply a perfectly valid NetworkPolicy manifest and it is accepted, stored, and completely ignored. No error, no event, no warning. The API server has no idea the CNI isn't enforcing it.

While the cluster was running Vaultwarden and a photo server, I could live with "every pod can talk to every pod." But the workloads grew: a microservices trading app, a Kafka-mediated email pipeline, and eventually a RAG chatbot with a query endpoint that must only be reachable from exactly one other namespace — the network policy is the control that keeps it off the internet. That last requirement is simply not expressible on Flannel. Segmentation went from "nice to have" to "a feature I'm shipping depends on it."

2. Debugging was tcpdump and guesswork

With an overlay, every debugging session has an extra layer. Is the problem in the pod? The bridge? The VXLAN tunnel? The node network? tcpdump on eth0 shows you UDP blobs between node IPs; you have to capture on flannel.1 or inside the pod's netns to see the real traffic. There is no answer to "who is talking to whom right now?" short of instrumenting everything yourself.

3. Overhead I wasn't getting anything for

VXLAN adds roughly 50 bytes of headers to every single packet, and encap/decap work on every node. On top of that, kube-proxy translates every Service through iptables chains that grow with the number of Services. Neither was causing measurable pain at my scale — I want to be honest about that — but I was paying complexity for an overlay whose only job was to avoid making my network route pod CIDRs. And my network is three Proxmox hosts on a switch I control. I could just... route the pod CIDRs.

---

What Cilium changes

The contrast with the Flannel diagram is the whole argument:

• Native routing, no tunnel. Pod IPs are real, routable IPs on the network. The packet the pod sends is the packet on the wire. tcpdump anywhere on the path shows you the actual traffic.
• Policy enforced in the kernel. NetworkPolicies (and the more capable CiliumNetworkPolicy CRD) are compiled to eBPF and enforced at the pod's interface — L3 through L7. Applying a policy actually does something now.
• kube-proxy is gone. kubeProxyReplacement: true moves Service load-balancing into the same eBPF datapath. No more iptables chains.
• Hubble sees everything. Because every packet already passes through eBPF, Cilium emits a flow event for each one — and Hubble turns that into a live service map. When a policy drops traffic, Hubble shows the exact flow and which policy dropped it. Nearly every network bug I've debugged since the migration was diagnosed in Hubble, not by staring at YAML.

The whole thing is deployed as one ArgoCD Application. The values that matter:

helm:
  valuesObject:
    routingMode: native
    autoDirectNodeRoutes: true
    ipv4NativeRoutingCIDR: 10.244.0.0/16
    kubeProxyReplacement: true
    bpf:
      masquerade: true
    hubble:
      enabled: true
      relay:
        enabled: true
      ui:
        enabled: true

autoDirectNodeRoutes: true is what makes native routing zero-effort on a flat L2 network like mine — each Cilium agent installs routes for the other nodes' pod CIDRs directly, so I never had to touch the Proxmox hosts or the router.

---

The migration itself

There's no gentle way to swap a CNI. The CNI owns every pod's network interface, so this is a "the cluster will be down" operation, not a rolling one. My sequence:

1. Commit the Cilium ArgoCD Application to git first (GitOps discipline doesn't pause for migrations).
2. Delete the Flannel DaemonSet and its CNI config from every node (/etc/cni/net.d/).
3. Install Cilium, with kubeProxyReplacement: true — and remove kube-proxy, since Cilium takes over its job.
4. Restart every workload. Existing pods still hold veth interfaces wired to the old bridge; nothing gets Cilium-managed networking until it's rescheduled. kubectl rollout restart across all namespaces, then a node-by-node reboot to clear out cni0 and flannel.1 remnants.

For an on-prem cloud, a 20-minute outage is a non-event. If you're doing this on something with users, plan for it properly.

The migration was the easy part. The surprises came after — and they were substantial enough that they became their own post: native routing changes where policy is evaluated, standard NetworkPolicy can't express some things you'll need (Ceph OSD egress, some cross-namespace ClusterIP paths), and webhook traffic behaves differently. Cilium Network Policies in Native Routing Mode covers all of it.

Two more lessons that didn't fit there:

• The first policy to select a pod flips it to default-deny — for the whole traffic direction, not just what your new rule covers. I added one narrow egress rule to a previously-unrestricted app and silently broke its DNS and an unrelated API call. If a namespace has no policies yet, pair any new narrow rule with an explicit allow-all restoring the original posture.
• Don't pin the agent to one API server. With kubeProxyReplacement, Cilium can't bootstrap through the in-cluster Service VIP it itself implements, so you point it at the API server directly. I pointed it at control-plane 1. When that node went down, every Cilium agent that restarted anywhere in the cluster crash-looped. Cilium 1.18's k8s.apiServerURLs takes all three control planes and fails over client-side.

---

Was it worth it?

Yes, unambiguously. The cluster is more secure — every namespace is default-deny, and the things that must be internal-only actually are. It's easier to debug — Hubble answers in seconds what used to take a tcpdump session. And it's simpler on the wire — no tunnel, no kube-proxy, no bridge.

Flannel earned its place as the starting point; it let me learn Kubernetes without also learning networking internals on day one. But the moment network policy became a real requirement, the migration stopped being optional. If I were starting over, I'd install Cilium on day one — and write the first network policy the same week, while breaking things is still free.