HashiCorp Vault
Scope
HashiCorp Vault is the secret-management platform for the entire TazLab cluster, replacing the now-decommissioned Infisical as the canonical secret backend. Vault is deployed on a Hetzner VM (lushycorp-vault.magellanic-gondola.ts.net:8200), reachable via Tailscale.
Current Synthesis
The Vault runtime is fully operational with local lifecycle and remote durability. Vault Secrets Operator (VSO) is the primary pod-level secret delivery mechanism, replacing both External Secrets Operator (ESO, dormant) and the Vault Agent Injector (removed). PKI engine fully deployed with a 3-tier architecture (Offline Root GPG → Vault-imported Intermediate → Vault-generated leaf certificates) serving 8 PKI roles across 8 JWT-authenticated namespaces. All three cluster applications (Grafana, mnemosyne, pgAdmin) authenticate to PostgreSQL via client certificates (mTLS). The Vault Database Engine (database/) has been removed — no dynamic credentials are needed since all apps use mTLS. Cluster consumption includes VaultStaticSecret (static API tokens), VaultPKISecret (TLS/client certificates), and a small number of legacy ExternalSecrets (ESO, dormant).
Operational Status
| Layer | Status | Details |
|---|---|---|
| Vault runtime (Hetzner) | ✅ Operational | Local lifecycle + S3 backup/recovery validated |
| Tailscale connectivity | ✅ Converged | MagicDNS FQDN lushycorp-vault.magellanic-gondola.ts.net |
| Cluster DNS resolution | ✅ Working | Via CoreDNS relay → Tailscale nameserver 10.96.0.101 |
| ClusterSecretStore | ✅ Operational | tazlab-secrets-vault Valid, ReadWrite, Ready |
| Consumer migration | ✅ Completed | 22 ExternalSecrets migrated (dormant), all SecretSynced True |
| Tailscale Operator exposure | ✅ Completed | 5 admin surfaces + DB exposed on tailnet (2026-05-24) |
| Vault Secrets Operator (VSO) | ✅ Active | Pod-level secret delivery via VaultAuth/VaultPKISecret/VaultStaticSecret CRDs. Replaces Vault Agent Injector + ESO. |
PKI engine (pki_int) | ✅ Operational | 3-tier PKI (Offline Root → Intermediate → Leaf). 8 roles across 8 namespaces. 6 VaultPKISecret SYNCED+HEALTHY. |
| mTLS PostgreSQL | ✅ All apps | Grafana (3/3), mnemosyne (1/1), pgAdmin (1/1) — client certificate auth, sslmode=verify-full |
| Vault Database Engine | ❌ Removed | vault secrets disable database (2026-07-09). TD-044 closed. No consumers — all apps on mTLS. |
| VM Hetzner route acceptance | ✅ Configured | tailscale set --accept-routes=true — required for VIP Anycast reachability on Linux |
| API Server ProxyGroup | ✅ Deployed | lushycorp-apiserver-proxy (mode: noauth, 2 replicas). Replaces old LoadBalancer Service. |
| Infisical decommission | ✅ Completed | 20-infisical-decommission |
Key Characteristics
- barrier-protected storage
- integrated storage via Raft
- KV v2 secret engine at
secret/ - PKI engine at
pki_int/(3-tier: Offline Root GPG → Intermediate → Leaf) - JWT auth backend at
auth/jwt(JWKS validation via ProxyGrouplushycorp-apiserver-proxy) - private CA TLS (
lushycorp-vault-private-ca) - TazPod operator recovery anchor (init.json, unseal-keys, root-token, admin-token)
- S3 raft snapshots at
tazlab-storage/vault/raft-snapshots/ - ESO static scoped token (
eso-readerrole, 90-day TTL) — legacy, dormant - Runs in Podman container (
hashicorp/vault:1.19,--network host) - SSH key:
~/secrets/ssh/lushycorp-vault/id_ed25519(user: admin) - Admin token:
/var/lib/lushycorp-vault/bootstrap/admin-token.txt - Multi-issuer
bound_issuer: K8s tokens concatenate both issuers with comma. Must match exactly. - VSO active (VaultAuth, VaultPKISecret, VaultStaticSecret CRDs) via Flux DAG:
infrastructure-operators-vso→infrastructure-vso-secrets→infrastructure-tls - PKI 3-tier hierarchy: Offline Root GPG (Tier 0) → Vault-imported Intermediate (Tier 1) → Vault-generated leaf certificates (Tier 2)
- 8 PKI roles on
pki_int, 8 JWT policies for per-namespace authentication - 6 VaultPKISecret: server TLS (
vault-pki-tls), DB server cert (tazlab-db-server-tls), client certs for Grafana, mnemosyne, pgAdmin, tazlab-admin - Database engine
database/removed (2026-07-09) — no consumers, all apps on mTLS
CRISP Projects
09-vault-k8s-integration-prep— Completed (2026-05-22): PKI fetch-back, DNS convergence, ClusterSecretStore, ESO smoke test10-tazlab-k8s-vault-migration— Completed (2026-05-22): consumer-side cutover from Infisical (now decommissioned) to Vault, 20 secrets migrated12-tazlab-k8s-vault-migration-followup— Completed (2026-05-22): hardening, bootstrap secrets, decommissioned Infisical provider removal10-pki-certificate-authority— Completed (2026-06-17): 3-tier PKI design and implementation. 19 review rounds, 155 findings. VPKI engine, mTLS for Grafana, 5 destroy+create cycles, one-shot on cycle 5.12-mnemosyne-mtls-refactor— Completed (2026-07-08): Go refactor for sslmode=verify-full. DB_PASS and md5 rules removed.14-pgadmin-mtls-migration— Completed (2026-07-08): VaultPKISecret, pg_hba cert rules, servers.json ConfigMap.20-vault-db-engine-mtls— Completed (2026-07-09):vault secrets disable database. Lease revocation. TD-044 closed.20-infisical-decommission— Completed: final cleanup after Vault consumption is verified stable