TazLab K8s: External Secrets Detail

Level 3 (Detail) — ESO configuration, ClusterSecretStore, and all ExternalSecret manifests.

Concept

External Secrets Operator (ESO) bridges external secret stores with native Kubernetes Secrets. All cluster secrets are served by HashiCorp Vault (tazlab-secrets-vault). Infisical (tazlab-secrets) is retained as a legacy store deployed by Terraform for engine-level backward compat only — zero ExternalSecrets reference it.

ClusterSecretStore

Two stores operating in parallel (two-store model):

StoreBackendStatusManaged by
tazlab-secrets-vaultHashiCorp Vault (Hetzner)✅ PrimaryFlux (GitOps, tazlab-k8s/)
tazlab-secretsInfisical EU (legacy)🔄 Empty creds, no consumersTerraform (engine layer)

ExternalSecret Inventory

All secrets stored at secret/data/tazlab-k8s/static/<domain>/<consumer>/<KEY> in Vault KV v2. remoteRef.property: value on all entries. 22 ExternalSecrets total.

cert-manager

File: infrastructure/configs/cert-manager/cloudflare-external-secret.yaml

Output SecretNamespaceKeyVault Path
cloudflare-api-tokencert-managerapi-tokentazlab-k8s/static/infra/cloudflare-ddns/CLOUDFLARE_API_TOKEN

Wildcard TLS

File: infrastructure/configs/wildcard-tls/external-secret.yaml

Output SecretNamespaceTemplate TypeVault Path
wildcard-tlshugo-blogkubernetes.io/tlstazlab-k8s/static/tls/wildcard/WILDCARD_*

The wildcard TLS is also replicated via ExternalSecret in namespaces:

  • dex — infrastructure/instances/dex/wildcard-tls-secret.yaml
  • auth — infrastructure/auth/oauth2-proxy/wildcard-tls-secret.yaml
  • longhorn-system — infrastructure/instances/longhorn/ingress.yaml

tazlab-db (S3)

File: infrastructure/configs/tazlab-db/s3-external-secret.yaml

Output SecretNamespaceTemplateVault Path
s3-backrest-credstazlab-dbGenerates s3.conftazlab-k8s/static/storage/tazlab-db/AWS_*

Uses a template to generate the PGBackrest configuration file format:

template:
  data:
    s3.conf: |
      [global]
      repo1-s3-key={{ .AWS_ACCESS_KEY_ID }}
      repo1-s3-key-secret={{ .AWS_SECRET_ACCESS_KEY }}

GitHub (Flux Automation)

File: infrastructure/configs/github-external-secret.yaml

Output SecretNamespaceKeyVault Path
github-api-tokenflux-systemgithub_tokentazlab-k8s/static/infra/github/GITHUB_TOKEN

Used by Flux ImageUpdateAutomation to commit image tag updates back to the tazlab-k8s repository.

OAuth2 Proxy

File: infrastructure/auth/oauth2-proxy/external-secret.yaml

Output SecretNamespaceKeysVault Path
oauth2-proxy-secretsauthOAUTH2_PROXY_CLIENT_SECRET, OAUTH2_PROXY_COOKIE_SECRETtazlab-k8s/static/auth/oauth2-proxy/OAUTH2_*

Dex

File: infrastructure/configs/dex/ or infrastructure/instances/dex/wildcard-tls-secret.yaml

Dex uses dex-rendered-config Secret combining ConfigMap template with OIDC credentials from Vault.

OpenClaw

File: infrastructure/configs/ai-agents/

Output SecretNamespaceKeysVault Path

Tailscale Operator

File: infrastructure/tailscale/externalsecret.yaml

Output SecretNamespaceKeysVault Path
tailscale-operator-oauthtailscaleTAILSCALE_OPERATOR_CLIENT_ID, TAILSCALE_OPERATOR_CLIENT_SECRETtazlab-k8s/static/infra/tailscale/TAILSCALE_OPERATOR_*

Note: This ExternalSecret was added after migration followup (12). Before that, Tailscale OAuth was seeded as a plain K8s Secret by Terraform.

Grafana Bootstrap

File: infrastructure/operators/monitoring/flux-secret-sync.yaml

Output SecretNamespaceKeyVault Path
grafana-bootstrap-secretflux-systemGRAFANA_DB_PASSWORDtazlab-k8s/static/monitoring/grafana/GRAFANA_DB_PASSWORD

Refresh Policy

All ExternalSecrets use refreshInterval: 1h except:

  • s3-backrest-creds: 24h (S3 credentials change rarely)
  • cluster-vars ConfigMap: static, provisioned once by ephemeral-castle

DAG Position

bridge (Level 1, creates ClusterIssuer/IngressClass)
→ configs (Level 2, creates all ExternalSecrets)
→ instances/apps (Level 3, consume the resulting Kubernetes Secrets)

Design Rules

  1. Never add inline secret values to manifests — always use ExternalSecret
  2. Each ExternalSecret should have creationPolicy: Owner unless explicitly sharing across namespaces
  3. Use template for format conversion (e.g., s3.conf generation)
  4. Wildcard TLS uses kubernetes.io/tls template type for proper secret format
  5. Run tests/verify_manifest_purity.sh after any manifest change

Future Direction

The Vault Agent Injector will progressively replace ESO for dynamic-capable workload secrets (Phase 2 of 15-tazlab-k8s-vault-dynamic-secrets-operator). ESO will remain for bootstrap/infrastructure-level static secrets.

See Also