TazLab K8s: Flux Kustomizations Detail

Level 3 (Detail) — All 21 Flux Kustomizations with exact spec, dependencies, and health checks.

Concept

Flux watches the clusters/tazlab-k8s/ directory for Kustomization resources. Each Kustomization points to a path in the repository and defines its dependencies, interval, and post-build substitution. Flux applies them in dependency order.

DAG Overview

infrastructure-operators-namespaces ──┬── infrastructure-bridge ──── infrastructure-configs ──┬── infrastructure-instances ── infrastructure-auth
infrastructure-operators-core ────────┘         │                                              ├── apps-static (hugo-blog)
                                                │                                              ├── apps-static-wiki (hugo-wiki)
                                                │                                              ├── apps-data (mnemosyne-mcp)
                                                │                                              ├── apps-data-hindsight
                                                │                                              └── infrastructure-tls
infrastructure-operators-data ──────────────────────────────────────────────────────────────────┘
infrastructure-operators-eso ──────────────────────────────────────────────────────┐
infrastructure-tailscale ── infrastructure-operators-tailscale ── infrastructure-tailscale-dns ──┬── infrastructure-operators-vso ── infrastructure-vso-secrets
                                                                                                 └── infrastructure-operators-reflector
infrastructure-operators-reloader ── (root, wait: false)

Note: The Tailscale chain (infrastructure-tailscale → infrastructure-operators-tailscale → infrastructure-tailscale-dns) is an independent branch. infrastructure-operators-vso (Vault Secrets Operator) and infrastructure-operators-reflector hang off infrastructure-tailscale-dns because VSO authenticates to Vault through the tailnet (lushycorp-vault ExternalName service). infrastructure-vso-secrets is applied independently right after VSO is ready — no other Kustomization depends on it; secrets become available as soon as the VaultStaticSecrets reconcile.

Kustomization Inventory

All Kustomizations share:

• namespace: flux-system
• sourceRef: kind: GitRepository, name: flux-system
• postBuild.substituteFrom → ConfigMap cluster-vars (provisioned by ephemeral-castle) — except infrastructure-operators-eso, infrastructure-operators-tailscale, infrastructure-tailscale-dns, infrastructure-operators-reloader, infrastructure-operators-reflector
• interval: 1h
• prune: true

1. infrastructure-operators-namespaces

Declares the dex, hindsight, and monitoring namespaces. Other namespaces (cert-manager, traefik, auth, cloudflare-ddns, hugo-blog, hugo-wiki, tailscale, reflector, reloader, eso, vso) are declared inline in each operator’s folder.

2. infrastructure-operators-core

Aggregator that installs:

• cert-manager operator (HelmRelease v1.16.2)
• traefik operator (HelmRelease v34.0.0, LoadBalancer via MetalLB annotation loadBalancerIPs: ${traefik_lb_ip}, shared IP pool tazlab-internal-dashboard)
• dex namespace
• auth namespace
• cloudflare-ddns namespace
• hugo-blog namespace
• hugo-wiki namespace

3. infrastructure-operators-data

Installs:

• postgres-operator (Crunchy PGO v5.7.2)

4. infrastructure-operators-eso

Installs external-secrets (HelmRelease v0.10.3). Legacy operator: kept installed as a dependsOn guard for infrastructure-configs and infrastructure-auth, but the repository no longer contains any ExternalSecret resource — all secrets are provisioned through VSO (infrastructure-operators-vso / infrastructure-vso-secrets).

5. infrastructure-operators-vso

Installs HashiCorp Vault Secrets Operator (HelmRelease v1.4.0) plus its plumbing:

• VaultConnection, default VaultAuth (JWT-based, vso-system/vso-jwt-auth)
• per-namespace VaultAuths and ServiceAccounts (cert-manager, cloudflare-ddns, dex, auth, flux-system, hindsight, longhorn-system, monitoring, tazlab-db)
• VaultAuths for the Vault PKI roles: ingress, per-DB clients (grafana, hindsight, mnemosyne, pgadmin, tazlab-admin), database-server, local-mTLS, replication
• shared VaultPKISecrets (vaultpkisecret-pki-tls, vaultpkisecret-client-hindsight)

Layer: the operator must be ready before infrastructure-vso-secrets and any consumer of Vault secrets.

6. infrastructure-operators-reloader

Installs reloader (HelmRelease v1.2.1). Non-blocking: pods reconcile on their own schedule.

7. infrastructure-operators-reflector

Installs Reflector (chart from OCIRepository oci://ghcr.io/emberstack/helm-charts/reflector, tag 10.0.45). Used to auto-replicate the Vault PKI wildcard TLS secret (vault-pki-tls) into the auth and dex namespaces.

8. infrastructure-tailscale

Creates the tailscale namespace (pod-security enforce: privileged) and defines the HelmRepository for https://pkgs.tailscale.com/helmcharts. Layer 1 of the Tailscale Operator 3-layer DAG. The Tailscale OAuth secret is pre-created by the Terraform engine layer — this layer deploys no secrets.

9. infrastructure-operators-tailscale

Installs the Tailscale Operator HelmRelease (v1.96.5, tailscale/tailscale-operator). Layer 2 of the 3-layer DAG — only applies after the namespace and HelmRepository exist.

10. infrastructure-tailscale-dns

Tailnet DNS layer. Deploys:

• DNSConfig ts-dns (tailscale nameserver image tailscale/k8s-nameserver:unstable)
• static ClusterIP Service nameserver-static (10.96.0.101, ports 53 UDP/TCP → 1053)
• ProxyGroup lushycorp-apiserver-proxy (type kube-apiserver, mode noauth, 2 replicas, tag k8s)
• ExternalName Services for the tailnet endpoints lushycorp-vault and lushycorp-k8s (.magellanic-gondola.ts.net) — the Vault endpoint used by VSO authentication

Layer 3 — only applies after the Operator CRDs are available.

11. infrastructure-bridge

Aggregator for ./infrastructure/bridge. Configures:

• IngressClass traefik (marked default)
• ClusterIssuer tazlab-issuer (Let’s Encrypt prod, HTTP01 via traefik)

12. infrastructure-monitoring

Installs:

• kube-prometheus-stack (HelmRelease v61.3.1) — Dex OIDC auth-proxy auth, Postgres backend with mTLS client certs (db-client-grafana-tls VaultPKISecret + initContainer copy)
• metrics-server (v3.12.1)
• Grafana dashboards as ConfigMaps (cluster-health, nodes-pro)
• Grafana Ingress — tailnet-ingress.yaml, ingressClassName: tailscale (grafana.magellanic-gondola.ts.net via tailscale ingress)
• VaultDynamicSecret grafana-db-creds (mount database, path creds/grafana) → Secret grafana-dynamic-creds
• ServiceAccount grafana-sa

13. infrastructure-configs

Deploys static configuration only:

• cert-manager: ClusterIssuer letsencrypt-issuer + letsencrypt-staging (HTTP01 via traefik). The Cloudflare API token is delivered by VSO (vaultstaticsecret-cert-manager → Secret cloudflare-api-token in cert-manager)
• tazlab-db: empty placeholder kustomization (S3 backup creds arrive via VSO vaultstaticsecret-s3-backrest-creds)

All former ExternalSecrets (wildcard-tls, hugo-wiki, dex OIDC, ai-agents/OpenClaw tokens, github-external-secret) have been removed — the repository now contains zero ExternalSecrets. Secrets flow through VSO instead (infrastructure-vso-secrets), internal TLS uses Vault PKI (VaultPKISecret via VSO), and external TLS uses cert-manager + Let’s Encrypt.

14. infrastructure-vso-secrets

Aggregator for all Vault-sourced secrets:

• VaultConnection + default VaultAuth
• VaultStaticSecrets: mnemosyne-mcp-secrets, hindsight keys (gemini, mcp-token, cp, opencode), cloudflare-api-token, cert-manager Cloudflare token, github-api-token, aws-backup-secret, s3-backrest-creds, dex-google-oauth (dex-secrets-sync), oauth2-proxy secrets (oauth2-proxy-secrets-sync, oauth2-proxy-dex-secret-sync)
• VaultAuths for Vault PKI roles (ingress, per-DB clients, database-server, local-mTLS, replication)
• VaultPKISecrets: vault-pki-tls (wildcard *.tazlab.net from pki_int/internal-ingress, 168h TTL, auto-reflected by Reflector into auth and dex) and vaultpkisecret-client-hindsight (mTLS client cert)

15. infrastructure-instances

Aggregator for ./infrastructure/instances plus all image automation pipelines (hugo-blog, hugo-wiki, mnemosyne-mcp, teaching, ambrogio-site, halloween, homepage). Uses wait: false because some workloads (tazlab-db, longhorn, apps with init containers) can take variable time to become ready. Kubernetes handles Pending/Init states naturally.

Deploys:

• tazlab-db: PostgresCluster (Crunchy PGO 5.7.2, Postgres 17.2, 1 replica, 4Gi, S3 pgbackrest backup to tazlab-storage eu-central-1, databaseInitSQL, VaultPKISecret server TLS + client certs for mnemosyne/pgadmin/tazlab-admin, hindsight-db-extensions-job)
• traefik: ClusterIP traefik-dashboard Service (8003→8080) + tailnet ingress
• longhorn: Service (8002) + tailnet ingress
• dex: Deployment (dexidp v2.41.1) + Service + Ingress (vault-pki-tls) + ConfigMap + RBAC; credentials via VaultStaticSecrets (dex-secrets-sync, oauth2-proxy-dex-secret-sync)
• pgadmin: Deployment (dpage/pgadmin4:9.12) + PVC + Service + servers ConfigMap + tailnet ingress + RBAC
• homepage: gethomepage Dashboard UI (Namespace + SA + ClusterRole + ConfigMap + tailnet ingress)
• cloudflare-ddns: Deployment (image pinned favonia/cloudflare-ddns:1.17.0; token from VSO-provisioned cloudflare-api-token Secret)

16. infrastructure-auth

Deploys:

• OAuth2 Proxy Deployment + Service + Ingress
• ForwardAuth middleware
• allowed-emails ConfigMap
• RBAC for the oauth2-proxy ServiceAccount

17. infrastructure-tls

Empty kustomization (resources: []) — placeholder kept for future TLS work; wildcard *.tazlab.net TLS is provisioned by Vault PKI via VSO (vault-pki-tls) and external TLS by cert-manager + Let’s Encrypt.

18. apps-static (hugo-blog)

Deploys hugo-blog from apps/base/hugo-blog/:

• nginx serving static Hugo build
• Certificate CR blog-tazlab.net-tls via ClusterIssuer letsencrypt-issuer (blog.tazlab.net, tazlab.net, www.tazlab.net, lab.tazlab.net)
• Traefik Ingress + middlewares
• Redirect middleware (tazlab.net → blog.tazlab.net)

19. apps-static-wiki (hugo-wiki)

Deploys hugo-wiki from apps/base/hugo-wiki/:

• nginx serving static Hugo wiki build
• Ingress for wiki.tazlab.net
• Certificate CR wiki-tazlab.net-tls via ClusterIssuer letsencrypt-issuer (cert-manager/Let’s Encrypt — no more ExternalSecret-based wildcard TLS)

20. apps-data (mnemosyne-mcp)

Deploys mnemosyne-mcp from apps/base/mnemosyne-mcp/:

• Go MCP server (tazzo/mnemosyne-mcp)
• LoadBalancer Service port 8004 → 8080 (MetalLB shared IP pool tazlab-internal-dashboard, IP 192.168.1.240 — shared with the traefik LoadBalancer)
• No ExternalSecret: GEMINI_API_KEY comes from Secret mnemosyne-mcp-secrets, created by VSO VaultStaticSecret vaultstaticsecret-mnemosyne (Vault path tazlab-k8s/static/apps/mnemosyne-mcp/GEMINI_API_KEY) with rolloutRestartTargets on the Deployment
• wait-for-db initContainer patch (infrastructure/common/wait-for-db-patch.yaml)
• RBAC for secret reading
• Postgres mTLS client certs from VaultPKISecret (db-client-mnemosyne-tls, managed in infrastructure/instances/tazlab-db/vaultpkisecret-client-mnemosyne.yaml)
• DB credentials read from the PGO-generated tazlab-db-pguser-mnemosyne secret (DB_SSLMODE verify-full)

21. apps-data-hindsight

Deploys hindsight from apps/base/hindsight/:

• three Deployments: hindsight-api, hindsight-worker, hindsight-dashboard
• Services (api, dashboard, worker-metrics) + ServiceMonitor (hindsight-api, release kube-prometheus-stack)
• NetworkPolicy + RBAC (hindsight-sa)
• Secrets via VSO VaultStaticSecrets: hindsight-gemini-key (GEMINI_API_KEY), hindsight-mcp-token, hindsight-cp-key, hindsight-opencode-key — each with rolloutRestartTargets on the affected Deployments
• Postgres mTLS via VaultPKISecret vaultpkisecret-client-hindsight
• quota-gate initContainer (waits for the Gemini embedding quota window before starting) and ensure-db initContainer (extensions job against tazlab-db-primary)

Wait Policy Truth Table

DAG Integrity Rules

1. Namespaces must exist before their operators are installed
2. ClusterIssuer / IngressClass must exist before cert-manager Certificates and Ingresses reference them
3. VSO must be installed and reconciled before VaultStaticSecrets / VaultPKISecrets are applied (infrastructure-configs and infrastructure-vso-secrets both depend on infrastructure-operators-vso)
4. Operators must be installed before PostgresCluster CRs are applied — PGO before infrastructure-instances
5. Infrastructure instances must be applied before the auth layer depends on them
6. Vault PKI TLS secrets must exist (via VSO) before Ingresses reference them (dex uses vault-pki-tls, replicated by Reflector)
7. Never apply instances before their operator is ready — dependsOn enforces this

DAG Freeze: Root Cause and Hardening (2026-09-10)

• Symptom: ~16 hours after a node power-cycle, 13 of 23 Kustomizations sat in DependencyNotReady. A single root (infrastructure-tailscale-dns, whose dependency was already Ready) blocked 12 dependents. No reconciliation was failing — nothing was broken; the work simply was not being applied.
• Root cause: Flux does not re-queue a dependent when its dependency transitions to Ready if that event is missed (the dependency was already Ready when the dependent was evaluated, or the controller restarted meanwhile). The dependent then waits for its own interval. With chained dependencies and hourly intervals, a single missed event stalls the DAG for hours and can persist for days — the same failure class as the earlier “DAG frozen 32d” incident.
• Hardening in the repository:
  1. interval: 15m (plus retryInterval: 1m where defined) on every cluster Kustomization, so each level self-heals within minutes and drift correction is tighter.
  2. flux-dag-watchdog (manifests in infrastructure/watchdog/, deployed by the infrastructure-watchdog Kustomization, deliberately without dependsOn so it can run while the DAG is stuck): every 2 minutes, with 3 internal passes 30s apart, it detects Kustomizations that are not Ready while all their dependencies are Ready and forces reconcile.fluxcd.io/requestedAt on them. It never creates, deletes or modifies anything beyond that annotation.
  3. Watchdog RBAC is a dedicated ServiceAccount/Role/RoleBinding limited to get/list/watch/patch on kustomizations in flux-system; the container is non-root with a read-only root filesystem plus a writable /tmp emptyDir, all capabilities dropped and seccompProfile: RuntimeDefault. Image: alpine/k8s:1.31.0. (Manual proof that kubectl + jq are present: the deployment was validated by running a Job from the CronJob template.)
• Verification: with a genuinely frozen DAG (13/23 not Ready), one watchdog run logged pass 1 → pass 2 → pass 3, nudged 10 objects in waves, and the DAG converged to 23/23 Ready.
• Residual risk (TD-067): the mitigation depends on the watchdog itself being healthy, and nothing alerts when the DAG stalls — TD-057 (Alertmanager has no receivers) is precisely why this ran unnoticed for ~16h.

Historical Note: OpenClaw

OpenClaw was previously deployed through infrastructure-instances with a dedicated Kustomization section (MetalLB LoadBalancer 192.168.1.242, PVCs for config and workspace, ExternalSecrets for gateway token / Telegram / OpenAI / ElevenLabs). It is no longer part of the repository — the live infrastructure-instances kustomization deploys only tazlab-db, cloudflare-ddns, pgadmin, homepage, longhorn, traefik, and dex, and the repository contains zero ExternalSecrets. Its token management pattern has been superseded by VSO VaultStaticSecrets.

See Also

• Parent hub: tazlab-k8s
• Sibling topics: Flux DAG, Repository Mapping
• Sibling details: Image Automation Detail
• Reference: Kustomization Example