Cilium Network Policy in Native Routing Mode

Cilium in native routing mode is one of the better architectural decisions I made for my on-prem cloud cluster. No VXLAN encapsulation, no overlay overhead — packets route directly between nodes using the kernel's own routing table. The result is a simpler network topology that's easier to reason about and debug with standard tools like tcpdump.

But native routing changes how traffic flows in ways that matter for network policy. There are specific scenarios where the standard Kubernetes NetworkPolicy API produces unexpected results, and you have to use Cilium's own CiliumNetworkPolicy CRD instead. This post covers exactly those scenarios.

---

How Native Routing Changes Traffic Flow

In overlay mode (VXLAN/Geneve), all pod traffic is encapsulated before it leaves the node. The kernel sees it as UDP between two node IPs. NetworkPolicy enforcement happens at the Cilium eBPF layer before encapsulation — policy always operates on the real pod IPs.

In native routing mode, pod IPs are real routable IPs on the underlying network. Packets go from pod directly to destination without wrapping. This means:

• Your underlying network must know how to route pod CIDRs. In my cluster, each node announces its pod subnet via static routes on the Proxmox bridge.
• The kube-apiserver connects to pods (for webhooks, metric scraping) via their real pod IPs — not via NAT through kube-proxy.
• Traffic that crosses DNAT (ClusterIP → pod IP) may be evaluated by Cilium at a different point than you expect.

That last point is the source of most of the surprises.

---

Where Standard NetworkPolicy Works Fine

For the majority of cases, standard NetworkPolicy works exactly as documented. Default deny-all:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: my-app
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Allowing traffic from a specific namespace:

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: ingress-nginx
    ports:
      - port: 3000

Both work. Cilium enforces them with eBPF rules and they behave as expected. The default-deny-all pattern is the right baseline — I apply it to every namespace.

---

Where It Breaks

1. Cross-namespace ClusterIP traffic

This is the most surprising failure mode.

When a pod in namespace A connects to a ClusterIP service in namespace B, kube-proxy (or Cilium's kube-proxy replacement) rewrites the destination from the ClusterIP to the actual pod IP via DNAT. In native routing mode, Cilium evaluates network policy after DNAT — so the policy sees the destination pod IP, not the ClusterIP.

The problem: a standard NetworkPolicy egress rule targeting a namespace may not match correctly because the DNAT evaluation order means Cilium sees a pod IP that belongs to namespace B, and the namespaceSelector match happens at a different point in the eBPF processing pipeline.

In practice this shows up as intermittent connectivity. The traffic works when the destination pod is on the same node (no DNAT required for local traffic), and fails when it's on a different node.

The fix is a CiliumNetworkPolicy using toEndpoints, which matches against the endpoint directly — DNAT-aware and pod-placement independent:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: ciliumnetpol-grafana-prometheus
  namespace: monitoring
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: grafana
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: prometheus
            io.kubernetes.pod.namespace: monitoring
      toPorts:
        - ports:
            - port: "9090"
              protocol: TCP

The io.kubernetes.pod.namespace label is automatically applied by Cilium to every endpoint — it's how you specify cross-namespace targets in a CiliumNetworkPolicy.

I needed this for:

• Grafana → Prometheus (same namespace, but DNAT still happens for ClusterIP)
• Grafana → CloudNativePG PostgreSQL in the yana-stocks namespace (cross-namespace ClusterIP to pod)

2. External CIDR egress (Ceph OSDs)

Covered in the previous post, but worth stating the pattern here for completeness.

Standard NetworkPolicy ipBlock is theoretically supposed to cover external CIDR egress. In practice, with Cilium native routing and a default-deny egress policy applied, it doesn't consistently allow traffic to external IP ranges when the traffic originates from host-network privileged pods (like the Ceph CSI node plugin).

CiliumNetworkPolicy with toCIDR resolves this:

egress:
  - toCIDR:
      - 192.168.22.0/24
    toPorts:
      - ports:
          - port: "6802"
            protocol: TCP
          # ... through 6809

The rule of thumb: if traffic goes to an IP outside the cluster CIDR and you're seeing intermittent failures, replace ipBlock with toCIDR in a CiliumNetworkPolicy.

3. kube-apiserver → admission webhook

This is the subtlest failure mode and the one that took me longest to fully understand.

Admission webhooks are called by the kube-apiserver when certain resources are created or modified. The call goes from the apiserver to an in-cluster Service, which gets DNAT'd to a pod.

In native routing mode, the apiserver makes this call from the control plane node's IP (e.g. 192.168.33.21), not from a pod IP. The call reaches the Service ClusterIP, DNAT rewrites to the webhook pod IP, and the webhook pod responds — but the response packet's return path goes back to the node IP, which may cross a routing boundary that Cilium's policy evaluation doesn't account for by default.

The result is context deadline exceeded in the apiserver logs. The webhook appears to receive the request but the response never arrives from the apiserver's perspective.

I hit this with the External Secrets Operator webhook. The fix I chose was to disable the webhook entirely (webhook.create: false in the ESO Helm values). For an on-prem cloud single-tenant cluster, the webhook's validation isn't providing meaningful safety guarantees — if I misconfigure an ExternalSecret, I'll find out when the secret doesn't populate.

If you need the webhook, the proper fix is an explicit CiliumNetworkPolicy that allows the control plane node's IP (192.168.33.21/32) to reach the webhook pod on port 443.

---

An Unexpected One: KEDA Intra-Namespace gRPC

One more I hit that doesn't fit neatly into the categories above.

KEDA's metrics-apiserver component communicates with the keda-operator via gRPC on port 9666. Both are in the keda namespace, so you'd expect a standard NetworkPolicy podSelector: {} ingress rule to cover it. It doesn't.

The issue: gRPC on port 9666 is not HTTP, and Cilium in native routing mode with L7 policy parsing enabled treats gRPC differently. The fix was adding an explicit ingress rule on port 9666 in the keda namespace policy:

- ports:
    - port: 9666
      protocol: TCP
  from:
    - podSelector: {}

The podSelector: {} (empty) matches all pods in the same namespace — it's the standard NetworkPolicy way to say "allow intra-namespace traffic on this port."

---

The Pattern That Works

After working through all of these, here's the decision table I now use:

Traffic type	Policy to use	Why
Intra-namespace pod→pod	Standard NetworkPolicy	Works reliably
Cross-namespace via ClusterIP	CiliumNetworkPolicy + toEndpoints	DNAT-aware matching
Pod→external IP range	CiliumNetworkPolicy + toCIDR	More reliable than ipBlock
kube-apiserver→webhook	Disable webhook OR CiliumNetworkPolicy with node CIDR	Node-IP routing special case
Default deny	Standard NetworkPolicy	Works fine
DNS egress	Standard NetworkPolicy port 53	Works fine

Start with standard NetworkPolicy everywhere. Reach for CiliumNetworkPolicy when you're seeing intermittent failures on cross-namespace or external traffic, or when traffic involves DNAT.

---

Debugging

When a connection is blocked and you don't know which policy is responsible:

# Watch policy drops in real time
kubectl exec -n kube-system -l k8s-app=cilium -- \
  cilium monitor --type drop

# List all endpoints and their policy status
kubectl exec -n kube-system -l k8s-app=cilium -- \
  cilium endpoint list

# Check policy for a specific endpoint
kubectl exec -n kube-system -l k8s-app=cilium -- \
  cilium endpoint get <endpoint-id>

The cilium monitor --type drop output shows exactly which policy rule caused the drop, which endpoint was the source, and which was the destination. Combined with endpoint IDs and the policy dump, you can trace any policy issue in a few minutes.

---

With the storage and networking layers sorted, the next piece is the database layer — and specifically, how to run Postgres in Kubernetes in a way you'd actually trust with real data.