ArgoCD: GitOps for an On-Prem Cloud That Takes Itself Seriously

GitOps is one of those ideas that sounds obvious once you've seen it — git is the source of truth, and something watches git and makes the cluster match it. But the gap between "concept" and "production GitOps" is larger than it first appears, and running ArgoCD on an on-prem cloud is one of the best ways to close it.

This post covers how I run ArgoCD on my cluster, the patterns that work, and the gotchas that took me an embarrassing amount of time to figure out.

---

The Core Principle

Nothing is deployed by kubectl apply without a subsequent commit to the k8s-apps repository. If it's not in git, it doesn't exist.

In practice this means:

• Every Helm chart, every manifest, every ConfigMap is in github.com/akann/k8s-apps
• ArgoCD's sync policy is automated: { prune: true, selfHeal: true } — it corrects drift automatically
• If I patch something live on the cluster (sometimes necessary for debugging), ArgoCD will revert it within seconds unless I also add ignoreDifferences in the Application manifest

This discipline is worth the friction. After a year of running it this way, I can rebuild the entire cluster from the git history. More practically: when something breaks, I can check git to know exactly what changed and when.

---

From Commit to Cluster

ArgoCD never builds images — that's CI's job. The two halves meet in git: CI writes the new image tag into a manifest, and ArgoCD notices the manifest changed. For this site, the full loop looks like this:

A push triggers GitHub Actions, which builds and pushes the image, patches the new tag into deployment.yaml, and commits that patch back to main with [skip ci] so it doesn't loop. ArgoCD sees the changed manifest and rolls the Deployment. The developer-facing rule falls straight out of this: commit to change the cluster, and never kubectl apply without a follow-up commit — selfHeal will revert it, and the next sync would clobber it anyway.

---

App-of-Apps

The repo uses the app-of-apps pattern, and it bottoms out in a single root Application called bootstrap. Its source is the repo root, and all it does is render the root kustomization.yaml — an ordered list of every other argocd-app-*.yaml in the repo. Sync that one app and it creates and manages all the others.

# bootstrap/argocd-app-bootstrap.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # runs before everything
spec:
  project: default
  source:
    repoURL: https://github.com/akann/k8s-apps
    path: .                              # the repo root kustomization
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true

Because bootstrap runs with selfHeal: true, the set of Applications is itself reconciled. Adding a service means committing its argocd-app-<name>.yaml and listing it in the root kustomization — ArgoCD picks it up on the next sync. The flip side bites too: kubectl apply an Application that isn't in git and bootstrap deletes it again within minutes.

The children split two ways. Infrastructure and in-cluster services (infrastructure/*, plus apps like Immich and Kafka) keep their manifests in k8s-apps itself. The application repos — this site included — keep their manifests next to the app in their own repo, and the Application here just points repoURL at that repo. Either way, ServerSideApply=true is set on everything: it avoids annotation bloat, handles merge conflicts better than client-side apply, and is required for Argo Rollouts (more on this below).

---

Sync Waves

Without ordering, ArgoCD tries to sync everything simultaneously. That breaks: you can't deploy cert-manager resources before cert-manager is running, and you can't deploy applications that need Postgres before CloudNativePG is installed.

Sync waves solve this with a simple annotation:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "3"

ArgoCD deploys wave 0 first, waits for all resources to be healthy, then proceeds to wave 1, and so on. My wave ordering:

Wave	What deploys
0	MetalLB, Ceph CSI
1	Cilium, cert-manager, ingress-nginx
2	MetalLB IPAddressPool, cert-manager ClusterIssuers
3	Reflector, Reloader, Kured, KEDA, Argo Rollouts, NetworkPolicies
4	Authentik, kube-prometheus-stack, Velero
5	Loki, Headlamp, Redis, MongoDB, MinIO, Kong
6	External Secrets Operator, Infisical
7	CNPG operator, CNPG clusters
8	Harbor, Actions Runner Controller
9+	Applications (Vaultwarden, Kafka, Immich, etc.)
10+	yana-stocks microservices

Wave 6 (ESO) must come after wave 5 (Infisical) because ESO talks to Infisical to fetch secrets. Wave 7 (CNPG clusters) must come after the operator at wave 7. The ordering requirement is obvious once you've watched ArgoCD fail trying to create a Cluster resource before the CNPG CRD exists.

---

ignoreDifferences

Some resources get mutated by other controllers after ArgoCD applies them. Without ignoreDifferences, ArgoCD sees the live cluster differs from git and reverts the mutation — breaking things.

The most painful example in my cluster: Kong's ValidatingWebhookConfiguration has three webhook entries with timeoutSeconds: 10. I patch them to 5 (for reasons covered in the Cilium post), but ArgoCD immediately reverts the patch unless I tell it to ignore those fields:

spec:
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: kong-controller-kong-validations
      jsonPointers:
        - /webhooks/0/timeoutSeconds
        - /webhooks/1/timeoutSeconds
        - /webhooks/2/timeoutSeconds

Another case: Infisical's bundled nginx chart injects a failurePolicy: Fail on its admission webhook. If Infisical is degraded, this blocks cluster-wide operations. I patch it to Ignore, and ignoreDifferences prevents ArgoCD from reverting it.

The general rule: when you need a live patch to stick, reach for ignoreDifferences rather than fighting ArgoCD.

---

The Helm Values Pitfall

When deploying a Helm chart, there are two ways to pass values in an ArgoCD Application:

# Wrong — YAML indentation issues
helm:
  values: |
    key: value
    nested:
      key: value

# Right
helm:
  valuesObject:
    key: value
    nested:
      key: value

values: | is a multiline string. If you indent your YAML incorrectly, the values are silently malformed and you get confusing Helm errors. valuesObject is parsed as structured YAML by ArgoCD, so indentation errors become parse errors — they fail loudly rather than silently.

I switched everything to valuesObject after spending an hour debugging a Prometheus deployment that was missing a key because of a two-space indentation difference.

---

The Argo Rollouts Diff Problem

Argo Rollouts CRDs use x-kubernetes-preserve-unknown-fields: true on .spec.template. ArgoCD's internal structured merge diff library doesn't know the schema for that field, so it throws ComparisonError: field not declared in schema — even though the Rollout is perfectly healthy.

The fix is to make the ArgoCD controller use server-side diff (dry-run SSA) instead of its internal library. There are two ways to do this, and only one works in ArgoCD v3:

# Per-Application — does NOT work in ArgoCD v3 without the global flag
syncOptions:
  - ServerSideDiff=true   # insufficient on its own

# Global flag in argocd-cmd-params-cm — this is what actually fixes it
configs:
  params:
    controller.diff.server.side: "true"

The per-Application flag is silently ignored unless the global flag is also set. I lost an afternoon to this before I found the ArgoCD v3 changelog entry.

---

Permanent OutOfSync Apps

Some apps show OutOfSync in the ArgoCD UI permanently, but are actually healthy and functioning correctly. These are cosmetic — they represent cases where something ArgoCD can't model causes a diff that doesn't correspond to real configuration drift.

The ones I have and why they exist:

App	Reason
actions-runner-controller	OCI registry limitation — ArgoCD can't compare OCI-sourced Helm charts
argo-rollouts	Cluster-scoped CRDs get tracked twice, creating a false diff
infisical	Bundled nginx chart mutates its own resources after sync
yana-stocks (ml-predictor)	SSA field-manager migration left a cosmetic diff; argocd app diff returns empty

The correct response to a permanent OutOfSync is to verify with argocd app diff <app> — if it returns empty, the diff is cosmetic. Do not spend time "fixing" it; the ArgoCD UI OutOfSync badge is not a reliable signal for these cases.

---

Source Path Discipline

One subtle ArgoCD behaviour: if your Application's spec.source.path is apps/my-service, ArgoCD only deploys files in that exact directory. Files in parent directories, sibling directories, or subdirectories are silently ignored.

This catches people out when an application has a manifests/ subdirectory. If the Application points to apps/my-service but the Deployments are in apps/my-service/manifests/, ArgoCD deploys nothing — silently.

The convention I follow: set spec.source.path to the directory that contains your manifests directly, not a parent directory. If you use a manifests/ subdirectory, point the Application at apps/my-service/manifests.

---

After a year of GitOps discipline, the operational feel of the cluster has changed noticeably. Debugging is faster — I know the cluster matches git, so the diff between expected and actual is always small. Changes are auditable. And rolling back is git revert and a push, not trying to reconstruct what kubectl apply commands you ran three weeks ago.

The patterns above — sync waves, ignoreDifferences, ServerSideApply, valuesObject — aren't obvious from the docs. They're things you discover by running ArgoCD on real workloads and hitting the edges.