The On-Prem Cloud Stack

How I use it

Central observability hub for the whole cluster. Every service exposes a /metrics endpoint; Prometheus scrapes them; Grafana visualises them. Loki handles structured log aggregation from Promtail; Tempo stores distributed traces.

Config note

Authentication via Authentik OIDC — no Grafana-native login. The role_attribute_path JMESPath expression maps Authentik group membership to Grafana Admin or Viewer roles automatically. Cross-namespace Prometheus and PostgreSQL datasources require CiliumNetworkPolicy (standard NetworkPolicy doesn't work for ClusterIP in Cilium native routing mode).

How I use it

All Alertmanager alerts route to Gotify via an alertmanager-gotify-bridge sidecar. This means every firing alert — node down, PVC full, pod crash-looping — lands as a push notification on my phone rather than in an email inbox I might not check.

Config note

Replaces email notifications entirely: AlertmanagerConfig routes all alerts to a single Gotify receiver. Critical-severity alerts get a separate receiver with higher priority. The gotify mobile app handles delivery.

How I use it

Invaluable for debugging NetworkPolicy issues. When a connection is being dropped and you're not sure which policy is responsible, Hubble shows you exactly which source endpoint tried to reach which destination, on which port, and whether Cilium allowed or dropped it.

Config note

Authentik forward-auth protected. No additional tooling required — Hubble is built into Cilium and just needs the UI enabled. The cilium monitor --type drop CLI command gives the same data in the terminal, but the Hubble UI makes the flow graph much easier to navigate.

How I use it

Monitors all public-facing services (stocks.yanatech.co.uk, yanatech.co.uk, argocd, grafana, etc.) with HTTP checks every 60 seconds. If a service goes down, Gotify gets the alert within a minute.

Config note

Runs as a single pod with a 5 Gi Ceph RBD PVC for its SQLite database. Authentik forward-auth protects the admin UI; the public status page is separate. Simpler than Prometheus alerting for basic HTTP endpoint monitoring.

How I use it

All custom images for yana-stocks and other projects are pushed here rather than to Docker Hub. The CI pipeline builds images on the self-hosted ARC runner and pushes to harbor.yanatech.co.uk/yana-stocks/<service>:<sha>. ArgoCD then picks up the new tag.

Config note

Trivy integration scans every pushed image for CVEs automatically. One recurring issue: harbor-jobservice and harbor-registry use ReadWriteOnce Ceph RBD PVCs — during rolling updates, if the new pod is scheduled on a different node, the PVC mount deadlocks. Fix is to delete the old pod manually to release the PVC, and set updateStrategy: Recreate in Helm values.

How I use it

Everything that needs object storage uses MinIO rather than a cloud provider: CNPG WAL archiving and base backups (s3://cnpg-backups/), Velero cluster backups, ml-predictor model artifacts (s3://yana-stocks-models), and the Turborepo remote cache bucket.

Config note

Single-node deployment with a 50 Gi Ceph RBD PVC. The S3 API endpoint is minio.yanatech.co.uk; the console is minio-console.yanatech.co.uk. One gotcha: the Turborepo remote cache image (ducktors/turborepo-remote-cache) ignores the S3_BUCKET env var and falls back to its hardcoded default 'turborepocache' — the bucket must be named exactly that.

How I use it

Kubernetes needs accurate resource requests and limits to schedule pods correctly and enforce QoS. Setting these by guesswork leads to either wasted capacity or OOM kills. Goldilocks runs the Vertical Pod Autoscaler in recommendation mode and presents the suggested requests/limits for every workload in a dashboard.

Config note

Authentik forward-auth protected. VPA runs in recommendation-only mode — it observes actual CPU and memory usage but never mutates pods. Goldilocks reads the VPA recommendations and presents them in a namespace-grouped UI. Useful to run after a period of normal traffic to baseline resource usage before setting production limits.

How I use it

Sometimes you want to click through pods, inspect logs, or read a ConfigMap without dropping into kubectl. Headlamp provides a clean, read-friendly view of the cluster. Useful for sharing cluster state with someone who doesn't have kubectl configured.

Config note

SSO via Authentik OIDC is broken upstream — there's an open issue in the Headlamp repo. Workaround is a long-lived ServiceAccount token (kubectl create token headlamp -n headlamp --duration=8760h) stored in Vaultwarden and rotated annually. Not elegant, but functional until upstream fixes it.

How I use it

The yana-stocks application is event-driven: price-ingestor publishes raw prices, price-processor consumes and enriches them, sentiment-analyzer publishes NLP signals, ml-predictor publishes predictions — all via Kafka topics. KEDA ScaledObjects scale the consumer services based on consumer group lag.

Config note

3 dual-role brokers (broker + controller in KRaft mode — no ZooKeeper). 6 topics, each with 3 partitions and RF3. Kafka UI (kafka-ui.yanatech.co.uk) provides a topic browser, consumer group lag viewer, and message inspector. One issue: Strimzi's bootstrap Service has a webhook timeout that conflicts with Kong's ValidatingWebhookConfiguration — adding Kong to ignoreDifferences was required.

How I use it

Profile data, OHLCV price history, portfolio holdings, sentiment articles, and ML predictions all live in MongoDB. The document model suits these workloads better than PostgreSQL — OHLCV history in particular benefits from MongoDB's flexible schema and time-series indexing.

Config note

2-replica replicaset (rs0) plus an arbiter for election quorum. Mongo Express at mongo.yanatech.co.uk provides a web UI (Authentik forward-auth protected). The replicaset connection string includes authSource=admin — omitting this causes MongoServerError: Unauthorized after a volume reset.

How I use it

auth-service stores refresh tokens in Redis (key: refresh:<token> → userId, 7-day TTL, single-use with rotation). portfolio-api caches aggregated responses (10s TTL). price-processor caches the latest price per symbol (5s TTL) so the frontend gets near-real-time quotes without hitting the database on every request.

Config note

Standalone deployment (not cluster mode) — sufficient for the workload. 5 Gi Ceph RBD PVC for AOF persistence. Redis Insight at redis.yanatech.co.uk provides a GUI for inspecting keys, running commands, and monitoring memory usage (Authentik forward-auth protected).

How I use it

Personal documents, notes, and files — nothing I want living on a third-party server. Nextcloud runs entirely on my own hardware, backed by a 100 Gi Ceph RBD PVC. The mobile and desktop sync clients work reliably for day-to-day file management.

Config note

Deployed via the official Nextcloud Helm chart, managed by ArgoCD. TLS via the wildcard cert reflected from cert-manager. Authentication is native Nextcloud — SSO via Authentik is possible but adds complexity I haven't needed. Velero backs up the PVC weekly to Backblaze B2.

How I use it

All personal photos backed up here instead of a cloud provider. The mobile app handles automatic upload; the web UI covers browsing, search, albums, and sharing. Face recognition and semantic search work surprisingly well for a self-hosted solution.

Config note

Runs its own dedicated CloudNativePG cluster (immich-postgres) using a custom image — ghcr.io/tensorchord/cloudnative-vectorchord:16-1.1.1 — which bundles the vectorchord extension for AI-powered image similarity search. Standard PostgreSQL can't run this workload. Library stored on a 200 Gi Ceph RBD PVC; Velero backs it up weekly to Backblaze B2.