Hindsight: Deployment Detail

Level 3 (Detail). Parent topic: Hindsight Architecture.

Scope

File-by-file breakdown of every manifest that deploys Hindsight in tazlab-k8s (branch master), the API environment contract, the Vault secret chain, and the four gotchas discovered during the 2026-08-28 deployment. No secret values are recorded here — only Vault paths and Kubernetes secret names.

Manifest Map — apps/base/hindsight/ (10 files)

Delivery chain: clusters/tazlab-k8s/apps-data-hindsight.yaml (Flux Kustomization apps-data-hindsight, path ./apps/cluster/hindsight, dependsOn: [infrastructure-configs], interval 1h, wait: false, prune: true) → apps/cluster/hindsight/kustomization.yaml (thin overlay) → apps/base/hindsight/. Namespace declared in infrastructure/operators/namespaces/hindsight.yaml.

API Environment Contract (deployment-api.yaml)

Worker additions: HINDSIGHT_API_PORT=8889 + HINDSIGHT_API_WORKER_HTTP_PORT=8889 (metrics), HINDSIGHT_API_WORKER_ID=hindsight-worker-1 (static — worker identity must survive pod recreation). Dashboard env: PORT=9999, HINDSIGHT_CP_DATAPLANE_API_URL=http://hindsight-api.hindsight.svc.cluster.local:8888, HINDSIGHT_CP_DATAPLANE_API_KEY/HINDSIGHT_CP_ACCESS_KEY from secret hindsight-cp-key.

Probes (API): liveness /health/live, readiness /health, both on 8888. Resources: API 512Mi–1Gi, worker/dashboard small. Both API and worker mount the db-client-hindsight-tls secret at /etc/certs and run the same ensure-db init container (postgres image, digest-pinned): wait for the DB, then hard-fail if the vector extension is missing.

The 4 Documented Gotchas

1. Kubelet env collision — HINDSIGHT_API_PORT. The kubelet injects HINDSIGHT_API_PORT into the pod from the Service named hindsight-api (service-links env mapping). Without an explicit HINDSIGHT_API_PORT=8888 in the container env, the API inherits the Service port (8006) and misbinds. The upstream chart sets the same override. (Worker pins it to 8889 for the same reason.)
2. Control plane ignores PORT. The hindsight-control-plane image binds its built-in default 9999 and ignores the PORT env var (verified live 2026-08-28). The PORT env is kept for documentation only; Services must target 9999.
3. pgvector 0.8.0 (Crunchy) is NOT trusted. vector.control lacks trusted = true, so CREATE EXTENSION requires superuser — the cert-only hindsight role cannot create it. Fix: user postgres is declared in the PostgresCluster spec.users, and a Flux-managed Job hindsight-db-extensions (infrastructure/instances/tazlab-db/hindsight-db-extensions-job.yaml, using tazlab-db-pguser-postgres credentials) runs CREATE EXTENSION IF NOT EXISTS vector; idempotently, waiting up to 10 min for the DB. The API/worker ensure-db init containers verify the extension and hard-fail if missing.
4. Non-root image uid/gid 1000. The Hindsight images run as uid 1000; mounted TLS secrets would be unreadable without group access. Fix: pod securityContext.fsGroup: 1000 (kubelet chgrps mounted files) + TLS secret defaultMode: 0640 (decimal 416).

Hostname for sslmode=verify-full: the certificate SAN matches the short in-cluster name — use tazlab-db-primary.tazlab-db.svc (not the full .cluster.local FQDN) or the TLS verification fails.

Database Chain (PGO tazlab-db)

• PostgresCluster tazlab-db: database hindsight, user hindsight with databases: [hindsight]; pg_hba lines hostssl hindsight hindsight 0.0.0.0/0 cert (+ IPv6) — cert-only mTLS, no password auth.
• Because PGO-owned databases keep public owned by postgres (non-writable for app roles on PG16), Hindsight uses the dedicated schema hindsight via HINDSIGHT_API_DATABASE_SCHEMA (Alembic-managed migrations); init-sql.yaml additionally aligns public ownership.
• Client certificate: VaultPKISecret db-client-hindsight-tls (below), renewed every 24h; both API and worker Deployments are rolloutRestartTargets so pods pick up renewals.

VSO Secret Chain (infrastructure/operators/vso/)

Static secrets (Vault KV → K8s Secret):

PKI recipe (reusable pattern for cert-only DB clients):

1. sa-vso-auth-sa-hindsight.yaml — VSO auth ServiceAccount in namespace hindsight.
2. vaultauth-pki-db-hindsight.yaml — VaultAuth vso-pki-db-hindsight, Kubernetes auth role vso-role-db-hindsight (referenced cross-namespace as vso-system/vso-pki-db-hindsight).
3. vaultpkisecret-client-hindsight.yaml — VaultPKISecret db-client-hindsight-tls: mount: pki_int, role: db-client-hindsight, commonName: hindsight, ttl: "24h", expiryOffset: "4h", destination kubernetes.io/tls with transformation.templates mapping tls.crt/tls.key/ca.crt (offline root CA pinned in the ca.crt template — the 2026-07-05 VSO template bug fix pattern), rolloutRestartTargets → hindsight-api, hindsight-worker.

Digest Pinning

All four images are digest-pinned (supply-chain doctrine, enforced by manifest-purity checks):

• ghcr.io/vectorize-io/hindsight-api:0.9.2-slim@sha256:7635a15739361dbdf221ba796ad25a813f876144fe113022eea8e26cb6ee75e7 (API + worker)
• ghcr.io/vectorize-io/hindsight-control-plane:0.9.2@sha256:d9bf57083f7cb0a53ee5011bdfde6aae2490dbf5c360d2d1bff10473b45eb112 (dashboard)
• postgres@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685 (init containers + extensions Job)

NetworkPolicy

Exposure: both LoadBalancer Services share MetalLB IP 192.168.1.240 (metallb.universe.tf/allow-shared-ip: tazlab-internal-dashboard) — 8006→8888 (API/MCP, also used by LAN agents), 8007→9999 (dashboard). No Traefik ingress; access is LAN/tailnet-direct.

Operations Quick Reference

• Reconcile: Flux picks up apps-data-hindsight automatically; force with flux reconcile kustomization apps-data-hindsight.
• Verify pods: all three Deployments ready in namespace hindsight; the ensure-db init container failing means the vector extension is missing (gotcha 3) or the DB/cert chain is broken.
• MCP token at runtime: fetch from the cluster without printing it (kubectl -n hindsight get secret hindsight-mcp-token -o jsonpath='{.data.value}' | base64 -d into a shell variable); never persist or echo it.
• After a cert renewal: VSO rotates the 24h client cert and rolloutRestartTargets restarts api/worker automatically; if a pod loops on TLS errors, check the secret’s ca.crt matches the current offline root CA.

See Also

• Parent topic: Hindsight Architecture
• Parent hub: hindsight
• Sibling details: tazlab-db Detail, External Secrets Detail, Mnemosyne Deployment Detail
• References: ExternalSecret variants