HashiCorp Vault
Scope
HashiCorp Vault is the secret-management platform for the entire TazLab cluster, replacing Infisical as the canonical secret backend. Vault is deployed on a Hetzner VM (lushycorp-vault.magellanic-gondola.ts.net:8200), reachable via Tailscale.
Current Synthesis
The Vault runtime is fully operational with local lifecycle and remote durability. Cluster consumption migrated: all 22 ExternalSecrets pull from Vault via tazlab-secrets-vault ClusterSecretStore. Infisical is retained as a legacy host only (empty credentials). Future direction: Vault Agent Injector for dynamic secret delivery, followed by PKI and Transit engine adoption.
Operational Status
| Layer | Status | Details |
|---|---|---|
| Vault runtime (Hetzner) | ✅ Operational | Local lifecycle + S3 backup/recovery validated |
| Tailscale connectivity | ✅ Converged | MagicDNS FQDN lushycorp-vault.magellanic-gondola.ts.net |
| Cluster DNS resolution | ✅ Working | Via CoreDNS relay → Tailscale nameserver 10.96.0.101 |
| ClusterSecretStore | ✅ Operational | tazlab-secrets-vault Valid, ReadWrite, Ready |
| Consumer migration | ✅ Completed | 22 ExternalSecrets migrated, all SecretSynced True |
| Tailscale Operator exposure | ✅ Completed | 5 admin surfaces + DB exposed on tailnet (2026-05-24) |
| Vault Agent Injector | ✅ Phase 1 Complete | Deployed (chart 0.32.0, injector-only). JWT auth via ProxyGroup. vault-k8s bug #660 workaround. Grafana migrated. |
| Vault Database Engine | ✅ Enabled | database/ engine at database/. Role grafana creates dynamic users inheriting grafana PG role. TTL 1h/max 24h. |
| VM Hetzner route acceptance | ✅ Configured | tailscale set --accept-routes=true — required for VIP Anycast reachability on Linux |
| API Server ProxyGroup | ✅ Deployed | lushycorp-apiserver-proxy (mode: noauth, 2 replicas). Replaces old LoadBalancer Service. |
| Infisical decommission | 📝 Draft | 20-infisical-decommission |
Key Characteristics
- barrier-protected storage
- integrated storage via Raft
- KV v2 secret engine at
secret/ - Database secret engine at
database/(PostgreSQL dynamic credentials for Grafana) - JWT auth backend at
auth/jwt(JWKS validation via ProxyGrouplushycorp-apiserver-proxy) - private CA TLS (
lushycorp-vault-private-ca) - TazPod operator recovery anchor (init.json, unseal-keys, root-token, admin-token)
- S3 raft snapshots at
tazlab-storage/vault/raft-snapshots/ - ESO static scoped token (
eso-readerrole, 90-day TTL) - Runs in Podman container (
hashicorp/vault:1.19,--network host) - SSH key:
~/secrets/ssh/lushycorp-vault/id_ed25519(user: admin) - Admin token:
/var/lib/lushycorp-vault/bootstrap/admin-token.txt - Vault Agent Injector deployed in cluster (chart 0.32.0, injector-only mode)
- Known bug: vault-k8s v1.7.2 issue #660 (
token_pathvspathfor JWT). Workaround:auth-config-path+auth-config-remove-jwt-after-readingannotations. - Multi-issuer
bound_issuer: K8s tokens concatenate both issuers with comma. Must match exactly.
CRISP Projects
09-vault-k8s-integration-prep— Completed (2026-05-22): PKI fetch-back, DNS convergence, ClusterSecretStore, ESO smoke test10-tazlab-k8s-vault-migration— Completed (2026-05-22): consumer-side cutover from Infisical to Vault, 20 secrets migrated12-tazlab-k8s-vault-migration-followup— Completed (2026-05-22): hardening, bootstrap secrets, Infisical provider removal15-tazlab-k8s-vault-dynamic-secrets-operator— Draft (Realigned): Vault Agent Injector adoption as universal delivery method (planned split: Phase 1 / Phases 2-4)20-infisical-decommission— Draft: final cleanup after Vault consumption is verified stable