TazLab Secret And Identity Flow

Scope

This page explains how operator identity, secret custody, and secret delivery move across the main TazLab layers. All cluster secrets are now served by HashiCorp Vault (tazlab-secrets-vault). Infisical is fully decommissioned as a secret delivery path.

Current Synthesis

TazLab uses a layered secret model rather than a single monolithic secret store:

  1. Level 1 — Operator secrets: plaintext files in ~/secrets/ on the operator host, managed by TazPod encrypted vault (AES-256-GCM, backed up to S3)
  2. Level 2 — Bootstrap secrets: Kubernetes Opaque Secrets created by Terraform (k8s-engine layer) by reading ~/secrets/ files — includes vault-ca-cert, vault-eso-token, tailscale-operator-oauth
  3. Level 3 — Cluster workload secrets: 22 ExternalSecrets pulling from Vault KV v2 via ESO tazlab-secrets-vault ClusterSecretStore

The concrete deployment points for these flows are documented in TazLab Flux DAG.

Main Flows

Operator Secret Custody

  • tazpod unlocks the working secret set into RAM (TazPod vault)
  • the durable operator recovery artifact is the encrypted vault stored in S3 (vault.tar.aes)
  • common operator credentials include GitHub, AWS, Proxmox, Tailscale OAuth, and Vault bootstrap material
  • bootstrap secrets are stored encrypted at rest in S3 and pulled on demand via tazpod pull vault

Infrastructure Bootstrap Consumption

  • ephemeral-castle consumes selected credentials from ~/secrets/ for bootstrap and destroy flows
  • examples include Proxmox API credentials, GitHub token, Tailscale OAuth credentials, and Vault CA cert + ESO token
  • these are read by Terraform’s get_env() and file() functions and used to create K8s Opaque Secrets in the engine layer

Cluster Secret Delivery

  • tazlab-k8s does not keep plaintext secrets in git
  • workloads consume secrets through ExternalSecret resources and the logical ClusterSecretStore tazlab-secrets-vault
  • the backend is HashiCorp Vault lushycorp-vault.magellanic-gondola.ts.net:8200 (KV v2), with 22 ExternalSecrets migrated
  • Infisical tazlab-secrets store is retained by Terraform for backward compat only — zero ExternalSecrets reference it
  • Future direction: Vault Agent Injector for dynamic secrets delivery (Phase 1: Grafana PostgreSQL, Phases 2-4: universal adoption, PKI, Transit)

Vault API Access Contract

  • the Hetzner Vault runtime exposes its API at the Tailscale MagicDNS FQDN lushycorp-vault.magellanic-gondola.ts.net:8200
  • reachable from the cluster via:
    • Talos node Tailscale bridge (OS-level system extension)
    • CoreDNS relay (forward ts.net → 10.96.0.101 for pod-to-Vault DNS resolution)
    • Tailscale Operator (service exposure for cluster admin surfaces)
  • the canonical operator token source lives in ~/secrets/lushycorp-vault/eso-reader-token.txt
  • ESO uses a static scoped token (eso-reader policy, 90-day TTL) via tazlab-secrets-vault ClusterSecretStore
  • helper scripts talk to Vault over the Tailscale mesh without a separate Tailscale credential for the Vault API

Architectural Tension

The system is transitioning from static KV-based secret delivery (ESO) to dynamic secret injection (Vault Agent Injector) for workload secrets. Bootstrap and infrastructure secrets remain on TazPod-managed ~/secrets/ files. The Vault Agent Injector will coexist with ESO during migration.

Relationships

Source Basis

  • AGENTS.ctx/memory/system-state.md
  • AGENTS.ctx/crisp/projects/hetzner-vault-platform/30-hetzner-vault-consumers/
  • tazlab-k8s/infrastructure/configs/vault/clustersecretstore.yaml
  • ephemeral-castle/clusters/tazlab-k8s/modules/k8s-engine/main.tf