A RAG Chatbot for the On-Prem Cloud’s Docs

k8s-apps/CLAUDE.md had quietly grown into a 600-line document — cluster topology, ArgoCD conventions, every gotcha from the last six posts, network policy pitfalls, secret paths. Genuinely useful, if you already know roughly where to look. The goal was something that could just be asked instead: "what StorageClass does this cluster use," "how do sync waves work here," and get a cited answer instead of a grep.

So: a small RAG chatbot, k8s-docs, first app in a new ml repo that's meant to grow into more than just this. This post is about what RAG actually is, what went into building one, and what building a small one reveals about the bigger idea.

Try it: akan.nkweini.org/k8s-docs — ask it about storage, ArgoCD, secrets, or Cilium networking in this cluster.

---

RAG, Briefly

Retrieval-Augmented Generation means: before asking an LLM to answer a question, go fetch the specific pieces of text most relevant to that question and hand them to the model as part of the prompt. The model then answers from what it was just given, instead of from whatever it happened to memorize during training.

It exists because of two limits every LLM has. First, an LLM only knows what was in its training data, frozen at whatever point that data was collected — it has no idea what's in k8s-apps/CLAUDE.md today, because that repo isn't public training data at all. Second, even models with huge context windows aren't a substitute for retrieval: dumping an entire 600-line doc (or an entire codebase) into every prompt is slow, expensive, and models still tend to pay less attention to the middle of a long context than the start or end. RAG sidesteps both — retrieve a handful of the right paragraphs, and even a small model can answer well.

(For a proper explanation with the underlying math, Wikipedia's article on retrieval-augmented generation is a solid starting point.)

The "retrieval" part almost always works via embeddings: a model turns text into a vector — a list of numbers positioned so that texts with similar meaning end up close together in that vector space. Turn the question into a vector the same way, find the stored chunks whose vectors are nearest to it, and those are your most relevant chunks. That's the whole trick. Everything else in this post is just the engineering around making that trick reliable.

---

How It Comes Together

Two pipelines, not one. Ingestion runs whenever k8s-apps' docs change: split the markdown by header into chunks small enough to be individually relevant, embed each chunk, store the vector alongside the source text and its file path in Postgres. Querying runs per chat message: embed the question with the same embedding model, ask pgvector for the nearest stored chunks, then hand those chunks plus the question to Claude and ask it to answer using only what it was given, citing which file each part of the answer came from.

Chunking by markdown header (rather than, say, fixed-size character windows) matters more than it looks: a chunk boundary that splits a sentence in half produces a vector that represents neither half well, and a chunk that's too large dilutes the one relevant sentence with four paragraphs of unrelated context. Header-based chunking keeps each chunk a coherent unit — one gotcha, one config example, one section — which is exactly what you want sitting next to its own embedding.

Storage is a dedicated CloudNativePG cluster — a Postgres+pgvector instance meant to be reusable for other ML side projects later, without unrelated apps being anywhere in its blast radius. It runs the same cloudnative-vectorchord image already proved out on this cluster, which happens to be a superset of plain pgvector — so the schema uses vanilla vector columns, with a faster index type available for free if it's ever needed.

---

Two Different Models, Doing Two Different Jobs

Embeddings and answering are deliberately different providers: OpenAI's text-embedding-3-small turns text into vectors for the similarity search, Claude turns the retrieved chunks into an actual answer with inline citations. Anthropic doesn't have a first-party embeddings API, so that split isn't optional here — but building it this way makes the underlying reason for the split obvious in a way reading about it never does.

Embedding and generation are genuinely different tasks. An embedding model's whole job is to place semantically similar text near each other in vector space — it never generates a word of output. A generative model's job is to produce fluent, grounded prose from whatever context it's handed. Nothing stops a single vendor from offering both (several do), but nothing about RAG requires one vendor to do both either — the retrieval half and the generation half don't need to know or care what produced the other's input, as long as the interface between them (a chunk of text plus a query) stays the same. That's a cleaner mental model of RAG than "an LLM with a database attached": it's two independent subsystems, one that finds things and one that reads them.

---

Keeping It Grounded, Not Just Chatty

The reason to bother with citations at all — "returns it with citations" in the diagram above — is that grounding is the entire point of RAG, not a nice-to-have on top of it. An LLM asked "what StorageClass does this cluster use" with no context will produce a plausible-sounding, generic Kubernetes answer, because that's what its training data looks like on average. The same question, answered from three retrieved chunks that literally contain this cluster's StorageClass name, produces a correct one — and the citation is what makes it possible to actually trust that it's correct without re-reading the source.

Scope stayed deliberately narrow: only k8s-apps is ingested, not the private repos in this workspace. Partly that's a security boundary — a chat page is a side channel, and private repos shouldn't be readable through it by proxy. But it's also a reminder of what retrieval actually is: the chatbot doesn't "know" anything that hasn't been explicitly indexed. It can't accidentally answer from a repo it was never given chunks from, the same way it can't answer from tomorrow's docs before they've been pushed. The knowledge boundary of a RAG system is exactly the ingestion boundary, and nothing else.

---

What Building This Reveals About RAG

Reading about RAG made it sound like a single clean idea: "retrieval, then generation." Building even a small version of it surfaced how much of the actual work is upstream of the LLM call entirely — how you split documents into chunks, which embedding model you pick, how many chunks you retrieve, whether the answering model is told to refuse when nothing relevant comes back. The generation step, the part that feels like "the AI," ended up being the smallest and least fiddly piece of the whole system. The retrieval half is where quality is actually won or lost.

It also reframes hallucination: it's not really a bug you patch in the model, it's a gap you close with better context. A generic answer isn't Claude being wrong — it's Claude being asked a question with nothing to ground it against. Once the right chunks are in the prompt, correctness mostly takes care of itself.

---

Try It

It's live at akan.nkweini.org/k8s-docs — ask it about storage, ArgoCD sync waves, secrets management, or Cilium networking in this cluster, and it'll answer from the actual k8s-apps docs with citations back to the source. It only knows what's in that one repo for now, on purpose.