HashiCorp Vault
Scope
HashiCorp Vault is the secret-management platform for the entire TazLab cluster, replacing the now-decommissioned Infisical as the canonical secret backend. Vault is deployed on a Hetzner VM (lushycorp-vault.magellanic-gondola.ts.net:8200), reachable via Tailscale.
Current Synthesis
The Vault runtime is fully operational with local lifecycle and remote durability. Vault Secrets Operator (VSO) is the primary pod-level secret delivery mechanism, replacing both External Secrets Operator (ESO, dormant) and the Vault Agent Injector (removed). PKI engine fully deployed with a 3-tier architecture (Offline Root GPG → Vault-imported Intermediate → Vault-generated leaf certificates) serving 9 PKI roles; 7 VaultPKISecret deliver the client/server certificates. Every remaining cluster application authenticates to PostgreSQL with a client certificate (mTLS): Grafana, mnemosyne, pgAdmin, tazlab-admin and hindsight (Paperclip and Multica were removed in 2026-10). The Vault Database Engine (database/) has been removed — no dynamic credentials are needed since all apps use mTLS. Cluster consumption includes VaultStaticSecret (static API tokens), VaultPKISecret (TLS/client certificates), and a small number of legacy ExternalSecrets (ESO, dormant).
Operational Status
| Layer | Status | Details |
|---|---|---|
| Vault runtime (Hetzner) | ✅ Operational | Local lifecycle + S3 backup/recovery validated |
| Tailscale connectivity | ✅ Converged | MagicDNS FQDN lushycorp-vault.magellanic-gondola.ts.net |
| Cluster DNS resolution | ✅ Working | Via CoreDNS relay → Tailscale nameserver 10.96.0.101 |
| ClusterSecretStore | ✅ Operational | tazlab-secrets-vault Valid, ReadWrite, Ready |
| Consumer migration | ✅ Completed | 22 ExternalSecrets migrated (dormant), all SecretSynced True |
| Tailscale Operator exposure | ✅ Completed | 5 admin surfaces + DB exposed on tailnet (2026-05-24) |
| Vault Secrets Operator (VSO) | ✅ Active | Pod-level secret delivery via VaultAuth/VaultPKISecret/VaultStaticSecret CRDs. Replaces Vault Agent Injector + ESO. |
PKI engine (pki_int) | ✅ Operational | 3-tier PKI (Offline Root → Intermediate → Leaf). 9 roles on pki_int; 7 VaultPKISecret SYNCED+HEALTHY (server TLS + vault-pki-tls + 5 client certs). |
| mTLS PostgreSQL | ✅ All apps | Client certificate auth, sslmode=verify-full: Grafana, mnemosyne, pgAdmin, tazlab-admin, hindsight. Paperclip (removed 2026-10-19) was the one deliberate password exception. |
| Vault Database Engine | ❌ Removed | vault secrets disable database (2026-07-09). TD-044 closed. No consumers — all apps on mTLS. |
| VM Hetzner route acceptance | ✅ Configured | tailscale set --accept-routes=true — required for VIP Anycast reachability on Linux |
| API Server ProxyGroup | ✅ Deployed | lushycorp-apiserver-proxy (mode: noauth, 2 replicas). Replaces old LoadBalancer Service. |
| Infisical decommission | ✅ Completed | 20-infisical-decommission |
Key Characteristics
- barrier-protected storage
- integrated storage via Raft
- KV v2 secret engine at
secret/ - PKI engine at
pki_int/(3-tier: Offline Root GPG → Intermediate → Leaf) - JWT auth backend at
auth/jwt(JWKS validation via ProxyGrouplushycorp-apiserver-proxy) - AppRole backend at
auth/approle-agent— one mount per trust domain; rolepaperclip-agentfor the CT108 agent container, CIDR-bound to192.168.1.208(see §Non-Kubernetes identities) - private CA TLS (
lushycorp-vault-private-ca) - TazPod operator recovery anchor (init.json, unseal-keys, root-token, admin-token)
- S3 raft snapshots at
tazlab-storage/vault/raft-snapshots/ - ESO static scoped token (
eso-readerrole, 90-day TTL) — legacy, dormant - Runs in Podman container (
hashicorp/vault:1.19,--network host) - SSH key:
~/secrets/ssh/lushycorp-vault/id_ed25519(user: admin) - Admin token:
/var/lib/lushycorp-vault/bootstrap/admin-token.txt - Multi-issuer
bound_issuer: K8s tokens concatenate both issuers with comma. Must match exactly. - VSO active (VaultAuth, VaultPKISecret, VaultStaticSecret CRDs) via Flux DAG:
infrastructure-operators-vso→infrastructure-vso-secrets→infrastructure-tls - PKI 3-tier hierarchy: Offline Root GPG (Tier 0) → Vault-imported Intermediate (Tier 1) → Vault-generated leaf certificates (Tier 2)
- 9 PKI roles on
pki_int, 7 VaultPKISecret: server TLS (vault-pki-tls), DB server cert (tazlab-db-server-tls), client certs for Grafana, mnemosyne, pgAdmin, tazlab-admin and hindsight — each in the namespace of its consumer - Database engine
database/removed (2026-07-09) — no consumers, all apps on mTLS
Non-Kubernetes identities
Everything above serves pods. The CT108 agent container is not a pod, has no ServiceAccount token for Vault, and therefore has the lab’s first AppRole trust domain — auth/approle-agent, “one mount per trust domain”, created by ephemeral-castle (modules/vault-approle-agent, live/persistent/vault-approle-agent, prevent_destroy = true so the identity outlives a cluster destroy/create).
Role paperclip-agent (verified live 2026-09-28 via vault read auth/approle-agent/role/paperclip-agent):
| Setting | Value | Why |
|---|---|---|
token_type | service | a batch token cannot be periodic or renewed |
token_period / token_explicit_max_ttl | 1h / 24h | renewing, but bounded |
token_num_uses | 0 | unlimited, which auto-auth requires |
token_policies | [agent-paperclip] — exactly one | for identical patterns Vault takes the union of capabilities, so a second, broader policy would silently beat the denies |
secret_id_num_uses / secret_id_ttl | 1 / 10m | single-use, short-lived |
secret_id_bound_cidrs / token_bound_cidrs | 192.168.1.208/32 | the container’s static address — which is why the address is static |
Policy agent-paperclip: the agent’s own KV v2 subtree (secret/data/agent/paperclip/*, secret/metadata/agent/paperclip/*), read on its own PKI role plus issue/sign through it and the CA chain, and self-management (auth/token/renew-self, lookup-self, sys/capabilities-self, lease renew/lookup/revoke). Confinement is omission plus explicit denies: sys/policies/*, sys/mounts/*, sys/audit/*, sys/raw/*, auth/token/create|accessors, pki_int/root*, pki_int/config/*, pki_int/issuer* and pki_int/revoke are denied, and the file states that the omission — not the deny list — is the real boundary. A login rate-limit quota approle-agent-login-rate-limit (rate 10 / interval 1, block_interval 30) mirrors the JWT layer’s.
Who mints the SecretID: the provisioning layer, never the agent and never Terraform state — ephemeral-castle/clusters/tazlab-k8s/proxmox/create.sh mints role_id and a response-wrapped SecretID into gopass (vault/approle-agent/role-id, vault/approle-agent/wrapped-secret-id); if the mint fails the run prints “could not mint a wrapped SecretID — the container’s egress layer will refuse to start”. The container-side RUNBOOK repeats the command for rotations:
vault write -wrap-ttl=120s -force auth/approle-agent/role/paperclip-agent/secret-id
Current consumption status: the agent container’s egress chokepoint is the intended consumer (the addon implements unwrap → login_path → in-memory token, injected as X-Vault-Token), but the shipped egress policy has no vault: rule, so the bootstrap is inert and the container today reaches Vault not at all. The identity exists, is verified, and is waiting for the security workstream — do not read this page as “the agent has a Vault token”.
CRISP Projects
09-vault-k8s-integration-prep— Completed (2026-05-22): PKI fetch-back, DNS convergence, ClusterSecretStore, ESO smoke test10-tazlab-k8s-vault-migration— Completed (2026-05-22): consumer-side cutover from Infisical (now decommissioned) to Vault, 20 secrets migrated12-tazlab-k8s-vault-migration-followup— Completed (2026-05-22): hardening, bootstrap secrets, decommissioned Infisical provider removal10-pki-certificate-authority— Completed (2026-06-17): 3-tier PKI design and implementation. 19 review rounds, 155 findings. VPKI engine, mTLS for Grafana, 5 destroy+create cycles, one-shot on cycle 5.12-mnemosyne-mtls-refactor— Completed (2026-07-08): Go refactor for sslmode=verify-full. DB_PASS and md5 rules removed.14-pgadmin-mtls-migration— Completed (2026-07-08): VaultPKISecret, pg_hba cert rules, servers.json ConfigMap.20-vault-db-engine-mtls— Completed (2026-07-09):vault secrets disable database. Lease revocation. TD-044 closed.20-infisical-decommission— Completed: final cleanup after Vault consumption is verified stable