shared-services: A Kafka Email Pipeline

Three of my apps sent email: a Go auth service (verification and password reset), a Next.js marketing site (contact form), and this personal site (also a contact form). Each one talked to SMTP2GO directly, with its own transport code and its own copy of the credentials. One of them — this site — had actually stopped sending real email months ago and nobody noticed, because the fallback path just logged to console instead of erroring.

This post is about replacing all three with one small service, the infrastructure problems that turned out to be the hard part, and an honest answer to the question I asked myself halfway through: is this over-engineered for what it does?

The Design

caller (yanatech / akan / auth-service)
  │  POST https://api-gateway.yanatech.co.uk/api/email/send
  │  header: apikey: <key>
  ▼
email-api  (NestJS HTTP)          — validates, assigns a messageId, publishes to Kafka, returns 202
  │
  ▼  notifications.email.send (Kafka)
  │
email-service  (NestJS consumer)  — sends via a swappable EmailProvider (SMTP2GO today)
  │  success → done
  │  failure → notifications.email.failed (dead-letter topic), single attempt, no retry

Callers never touch Kafka. They POST to an HTTP endpoint and get a 202 with a messageId back immediately — the actual send happens asynchronously. This means adding a new caller is a five-minute job: no SMTP credentials, no transport library, just a fetch() call.

The auth split — an HTTP ingest service in front of a Kafka consumer, rather than one service doing both — exists so the two failure modes stay separate. If SMTP2GO is having a bad day, email-api keeps accepting requests and queuing them; the backlog just grows in Kafka until email-service catches up. Callers never see that as an error.

email-api is reachable only through Kong, gated by Kong's key-auth plugin rather than an in-app check — the same pattern this cluster already uses for JWT auth on other services. A NetworkPolicy restricts ingress to the Service to the Kong namespace only, so the auth check can't be bypassed by hitting the ClusterIP directly from inside the cluster.

The Part That Actually Took Time

Writing email-api and email-service took about an hour. Getting the first one deployed took the rest of the day, because a brand-new service touches every piece of on-prem cloud infrastructure that already assumed it would only ever see infrastructure I'd already provisioned by hand.

Harbor needed a new project, and its robots API has a gotcha. I'd assumed I could reuse an existing Harbor credential from another project. I couldn't — Harbor robot accounts are scoped to one project each, so a copied credential just gets 401. Creating a new one meant learning that GET/POST /api/v2.0/projects/{id}/robots 404s on the Harbor version I'm running (2.15.1); the working route is the flat /api/v2.0/robots, with the project scope specified in the request body instead of the URL. Not documented anywhere I could find — I found it by trial and error against the live API.

A new private repo needs its own CI runner. harbor.yanatech.co.uk isn't publicly resolvable — it only exists on the on-prem cloud network — so GitHub-hosted runners can't reach it at all. Every repo that pushes images needs its own self-hosted ARC runner scale set, registered against that specific githubConfigUrl. I'd forgotten this was a per-repo thing, not a cluster-wide thing, until the build step failed with a DNS lookup error.

A new private repo also needs its own ArgoCD credential. ArgoCD couldn't clone the new repo at all — Repository not found, because credentials are registered per-repo as a Kubernetes Secret, and a brand-new repo obviously has no entry yet.

And then, once all of that was sorted, the app deployed and immediately failed its health checks — the container was listening on its local-dev default port instead of the one the Service and probes expected, because I'd forgotten to set PORT in the Deployment. A one-line fix, but a good reminder that "it builds and the manifest applies" isn't the same as "it works."

None of these are exotic problems. They're the ordinary tax of adding a new thing to an ecosystem that has quietly accumulated a lot of per-repo, per-project, hand-configured state. Nothing about it was hard — it was just five separate small pieces of infrastructure surface, each invisible until the new service needed it.

A Gotcha That Wasn't About the New Service At All

Partway through rolling the third caller (auth-service) onto the new pipeline, ArgoCD stopped picking up new commits. argocd.argoproj.io/refresh: hard did nothing — the app stayed Synced at a commit from twenty minutes earlier no matter how many times I patched the annotation.

Two things were going on. First, patching an annotation to the same value it already has doesn't fire a Kubernetes watch event — no-op, nothing refreshes. I'd been retriggering with the exact same string every time. Second, and more interesting: even after fixing that, the app-controller logs showed git_ms=7 for the repo fetch — far too fast to be a real network clone. The argocd-repo-server had a stale local git cache and no webhook was configured to bust it (these are on-prem cloud repos with no public ingress for GitHub to call back to), so it was quietly comparing against old content it had cached from before my pushes.

kubectl rollout restart deployment argocd-repo-server fixed it — forces a fresh clone on next reconcile. Filed away for next time, since this had nothing to do with the email service specifically and will bite me again on some future change.

What the Migration Found

Moving three existing contact-form-adjacent flows onto one code path surfaced two real bugs that had nothing to do with email infrastructure:

This site's contact form used Resend, but the API key was never actually wired into the Kubernetes Deployment. Every submission for months had been silently logged server-side instead of sent. The form "worked" — it returned success — it just never told me anyone had filled it in.
The marketing site's contact form 403'd for anyone visiting via www.yanatech.co.uk, because the origin allow-list only had the apex domain. The ingress happily serves both hostnames; the app-level check didn't agree.

Neither of these would have surfaced without touching the code again. Centralizing forced a second look at flows that had been "done" for a long time.

Was It Over-Engineered?

Genuinely, for what these three callers actually need — a handful of contact-form submissions a week, plus verification and reset emails at signup-scale volume — probably yes. Kafka, two services, a dedicated Kong route, a dedicated Harbor project and robot account, a dedicated CI runner: that's a lot of moving parts to replace what could have been a single shared function with SMTP2GO baked in.

What tips it back the other way for me: the actual payoff isn't "sending email," it's not having SMTP credentials duplicated across three codebases, and having a dead-letter queue in exactly one place instead of reimplemented (or, as it turned out, silently not implemented) three times. And the auth-service side — real signup emails blocking a real user-facing flow — is a much better fit for "queue it, don't let a slow SMTP provider become a slow registration endpoint" than a contact form ever was.

If I only had the two contact forms, I'd have written a shared npm package and called it done. The fact that all three callers now exist on the same infrastructure, for close to the same one-time cost, is the part that makes it worth having built.

Postscript: What Changed After Shipping This

A few things surfaced in the days right after this went live, worth recording rather than quietly editing away.

I removed the retry loop. The original design retried a failed send three times before dead-lettering it. Thinking it through properly: retrying only ever covers the email-service↔SMTP2GO hop — SMTP2GO's own best-effort delivery already handles the SMTP2GO↔recipient hop, and I have no visibility into that side at all (no bounce webhook, nothing). Worse, the retry didn't distinguish a permanent failure (bad address, auth) from a transient one, so a bad address got retried three times for no reason — and if a send ever times out waiting for SMTP2GO's ack after the message was actually already accepted, retrying risks sending the email twice. Correctly classifying SMTP error codes to retry only the genuinely transient ones is possible, but for the volume this handles, it wasn't worth the fragility. Single attempt, straight to the DLQ. The DLQ was always the part doing the real work here, not the retry.

KEDA and ArgoCD's self-heal will fight each other if you don't tell ArgoCD to stop caring. email-service scales 0→3 via a KEDA ScaledObject, but its Deployment manifest also declares replicas: 1. Every time ArgoCD synced, it reset replicas back to 1 (self-heal doing exactly its job), which KEDA then scaled back down moments later — visible as a new pod being created and torn down right after every routine sync. The fix is an ignoreDifferences entry for /spec/replicas on that one Deployment, telling ArgoCD to stop treating KEDA's changes as drift. yana-stocks already had this for all six of its KEDA-scaled services; I'd just forgotten to add the equivalent one when scaffolding this app.

No servers entry in an OpenAPI spec means Redoc defaults to the docs page's own origin. The generated spec had an empty servers: [], so the hosted Swagger/Redoc page's "try it" button was pointed at shared-api-docs.yanatech.co.uk — its own address — instead of the real API gateway. One line (.addServer(...)) fixes it, and I added a second localhost entry alongside production so the hosted docs can target a local dev instance too.

A cancelled CI run can silently drop a change from ever being built. I pushed a fix, then pushed a second, unrelated fix before the first run finished — the workflow's cancel-in-progress concurrency setting killed the first run to start the second. That's normal and desired. What's not obvious: the second run's change-detection only diffs against the commit immediately before it, so a file changed only in the cancelled run's commit doesn't show up as "changed" the second time either, and quietly never gets built. I only caught it because I checked which Docker jobs actually ran. The recovery was a manual workflow_dispatch with a build_all flag to force every service to rebuild regardless of detected changes — worth having that escape hatch wired up before you need it, not after.

None of these were expensive to fix once found. All of them were invisible until something forced a second look — which, four posts into "gotchas I found by running this for real," is starting to feel less like bad luck and more like the actual shape of operating infrastructure.

Next up in the series, if there is one: whatever breaks first. So far that's been the pattern.

shared-services: A Kafka-Mediated Email Pipeline (And Whether It Was Worth It)