TazLab K8s: Monitoring Detail

Level 3 (Detail) — Prometheus, Grafana, dashboards, metrics-server.

Concept

The monitoring stack uses kube-prometheus-stack (Prometheus + Grafana + Alertmanager) managed as HelmRelease. Grafana uses the shared PostgreSQL instance (tazlab-db) as its backend, making dashboards persistent across pod restarts. Dashboards are managed as ConfigMaps loaded by a sidecar.

HelmRelease

File: infrastructure/operators/monitoring/helmrelease.yaml

Manifests

File: infrastructure/operators/monitoring/

Grafana PostgreSQL Backend

Grafana is configured to use tazlab-db as its database backend (database grafana, user grafana). This ensures:

• Dashboard configurations survive pod restarts
• User sessions and preferences are persistent
• No PVC dependency for Grafana itself

The grafana-bootstrap-secret Secret (referenced by infrastructure-instances Kustomization’s substituteFrom) provides initial admin credentials.

Grafana PostgreSQL mTLS

File: infrastructure/instances/tazlab-db/cluster.yaml (pg_hba), infrastructure/operators/monitoring/helmrelease.yaml (Grafana config), infrastructure/operators/monitoring/vaultpkisecret-client-grafana.yaml (client cert management)

Server-Side: CN-mapped certificate authentication

The tazlab-db cluster is configured with customTLSSecret: tazlab-db-server-tls (from VaultPKISecret server-tls in infrastructure/instances/tazlab-db/vaultpkisecret-server-tls.yaml) and customReplicationTLSSecret pointing to the same secret. pg_hba includes hostssl cert rules for the grafana user, requiring a valid client certificate from the db-client-grafana Vault role:

Client-Side: VaultPKISecret db-client-grafana-tls

File: infrastructure/operators/monitoring/vaultpkisecret-client-grafana.yaml

The ca.crt template uses a range over ca_chain to write each CA certificate (the modern VaultPKISecret idiom — avoids the old single-CA assumption).

Known issue (2026-07-03): The ca_chain range template replaces an earlier {{ .Secrets.ca.crt | base64Decode }} reference that broke after a Vault PKI intermediate rotation because the template had been hardcoded to the old issuer. The range fix makes it tolerate chain changes automatically.

extraInitContainers: cert copy

The Grafana HelmRelease uses an extraInitContainers entry (copy-db-certs) to copy certificates from the mounted Secret to the expected paths:

1. extraSecretMounts mounts db-client-grafana-tls at /tmp/db-certs-in (readOnly, mode 0600).
2. extraEmptyDirMounts creates an emptyDir grafana-certs-empty at /etc/grafana/certs/.
3. The init container (image: alpine:3.21) copies certs with cp -rL, then chmod 600 /etc/grafana/certs/tls.key.
4. Grafana’s grafana.ini.database section sets:
   • ssl_mode: require (enforced via GODEBUG=tlskyber=0 to work around Kyber fragmentation on Proxmox VXLAN; upgrade to verify-full planned after Proxmox qemu-server >= 11.0+pve1)
   • ca_cert_path: /etc/grafana/certs/ca.crt
   • client_cert_path: /etc/grafana/certs/tls.crt
   • client_key_path: /etc/grafana/certs/tls.key

Note: The grafana-dynamic-creds valuesFrom reference (leftover from Vault Agent Injector era) was removed from the HelmRelease on 2026-07-11 as part of cluster cycle 2 cleanup.

GODEBUG=tlskyber=0

Grafana is started with GODEBUG=tlskyber=0 to work around Kyber TLS key exchange fragmentation issues on Proxmox VXLAN. The Go Kyber implementation generates large handshake messages that fragment across VXLAN’s reduced MTU, causing connection drops to tazlab-db. Setting tlskyber=0 disables the Kyber KEX, falling back to X25519.

Dashboards as Code

Dashboards are stored as ConfigMaps with label grafana_dashboard: "1". Grafana’s sidecar watches for this label and loads dashboards automatically.

Current dashboards:

• cluster-health.yaml — High-level node and pod metrics
• nodes-pro.yaml — Detailed hardware and kernel metrics

metrics-server

File: infrastructure/operators/monitoring/metrics-server.yaml

Provides resource metrics used by kubectl top and HorizontalPodAutoscaler. Installed alongside kube-prometheus-stack.

flux-secret-sync

Syncs Grafana admin credentials from ExternalSecret to the Grafana deployment. Ensures the initial admin password is available before Grafana starts.

DAG Position

operators-namespaces (Level 0, creates monitoring namespace)
→ monitoring (Level 1, installs kube-prometheus-stack and metrics-server)
→ configs (Level 2, creates S3 backup ExternalSecret for tazlab-db)
→ instances (Level 3, creates Grafana ingress + syncs grafana-bootstrap-secret)

Prometheus Status Note

As of the 2026-04-29 power loss recovery, Prometheus uses a manually salvaged Longhorn volume. The volume is healthy and the pod is 2/2 Running. Future consideration: if monitoring data becomes important, the Prometheus PVC should use 2 replicas.

See Also

• Parent topic: Monitoring & Dashboards
• Sibling details: tazlab-db Detail, External Secrets Detail