TazLab Secret And Identity Flow

Scope

This page explains how operator identity, secret custody, and secret delivery move across the main TazLab layers. All cluster secrets are now served by HashiCorp Vault (tazlab-secrets-vault). Infisical is fully decommissioned as a secret delivery path.

Current Synthesis

  1. Level 1 — Operator secrets: plaintext files in ~/secrets/ on the operator host, managed by TazPod encrypted vault (AES-256-GCM, backed up to S3). Migrated to gopass (GPG-encrypted git store).
  2. Level 2 — Bootstrap secrets: Kubernetes Opaque Secrets created by Terraform (k8s-engine layer) by reading ~/secrets/ files (or via gopass) — includes vault-ca-cert, vault-eso-token, tailscale-operator-oauth
  3. Level 3 — Cluster workload secrets: delivered via VSO (Vault Secrets Operator) as primary mechanism, with legacy ESO (External Secrets Operator) dormant. Backend is Vault KV v2 + PKI engine. The concrete deployment points for these flows are documented in TazLab Flux DAG.

Main Flows

Operator Secret Custody

  • gopass store in /workspace/tazlab-secrets (GPG-encrypted git repo)
  • the durable operator recovery artifact is the gopass git repository on GitHub
  • common operator credentials include GitHub, AWS, Proxmox, Tailscale OAuth, and Vault bootstrap material
  • pulled via git pull su tazlab-secrets

Infrastructure Bootstrap Consumption

  • ephemeral-castle consumes selected credentials from gopass via gopass show for bootstrap and destroy flows
  • examples include Proxmox API credentials, GitHub token, Tailscale OAuth credentials, and Vault CA cert + ESO token
  • these are read by Terraform’s get_env() and file() functions and used to create K8s Opaque Secrets in the engine layer
  • Tutti i 13 progetti di migrazione (10, 22-27) sono stati completati

Cluster Secret Delivery

  • tazlab-k8s does not keep plaintext secrets in git
  • workloads consume secrets through ExternalSecret resources and the logical ClusterSecretStore tazlab-secrets-vault
  • the backend is HashiCorp Vault lushycorp-vault.magellanic-gondola.ts.net:8200 (KV v2), with 22 ExternalSecrets migrated
  • Infisical tazlab-secrets store is fully decommissioned — Terraform references removed, no ExternalSecrets reference it
  • Active: VSO (Vault Secrets Operator) is the primary secret delivery operator — handles both static secrets (KV v2 via VaultStaticSecret) and dynamic secrets (VaultPKISecret for PKI, database credentials, etc.). Vault Agent Injector was removed.

Vault API Access Contract

  • the Hetzner Vault runtime exposes its API at the Tailscale MagicDNS FQDN lushycorp-vault.magellanic-gondola.ts.net:8200
  • reachable from the cluster via:
    • Talos node Tailscale bridge (OS-level system extension)
    • CoreDNS relay (forward ts.net → 10.96.0.101 for pod-to-Vault DNS resolution)
    • Tailscale Operator (service exposure for cluster admin surfaces)
  • the canonical operator token source lives in ~/secrets/lushycorp-vault/eso-reader-token.txt
  • ESO uses a static scoped token (eso-reader policy, 90-day TTL) via tazlab-secrets-vault ClusterSecretStore
  • helper scripts talk to Vault over the Tailscale mesh without a separate Tailscale credential for the Vault API

Architectural Tension

ESO (External Secrets Operator) is dormant — no new ExternalSecrets are created; existing ones serve remaining static KV paths. VSO (Vault Secrets Operator) is the primary secret delivery operator for both static and dynamic workloads. Bootstrap and infrastructure secrets remain on TazPod-managed ~/secrets/ files.

Relationships

Source Basis

  • AGENTS.ctx/memory/system-state.md
  • AGENTS.ctx/crisp/projects/hetzner-vault-platform/30-hetzner-vault-consumers/
  • tazlab-k8s/infrastructure/configs/vault/clustersecretstore.yaml
  • ephemeral-castle/clusters/tazlab-k8s/modules/k8s-engine/main.tf