TazLab K8s Monitoring

Observability is managed as code to ensure dashboards and alerting rules are preserved during cluster rebirth.

The Monitoring Stack

  • Helm: prometheus-community/kube-prometheus-stack
  • Namespace: monitoring
  • Storage: Persisted in tazlab-db (PostgreSQL) for Grafana, connected via mTLS using a VaultPKISecret (db-client-grafana-tls).

Alerting Status (TD-057 — Open)

Alertmanager ships with kube-prometheus-stack but has no receivers configured — no alert routes anywhere reach a notification target. The live HelmRelease (infrastructure/operators/monitoring/helmrelease.yaml, chart 61.3.1) contains only values: grafana: keys; no alertmanagerSpec exists.

  • A Discord receiver fix was attempted and then reverted (commit b0b46bc, 2026-09) pending a real Discord webhook URL. The reverted approach used a VaultStaticSecret reading Vault path tazlab-k8s/static/infra/monitoring/ALERTMANAGER_DISCORD_WEBHOOK.
  • Correct Helm key for the eventual fix: alertmanager.alertmanagerSpec.configSecret (a pre-built Alertmanager config Secret). Note the chart silently ignores a bare alertmanager.configSecret — the value must live under alertmanagerSpec.
  • Tracked as TD-057 in TazLab K8s Known Issues. See Monitoring Detail.

Dashboard Management

Dashboards are stored in infrastructure/operators/monitoring/dashboards/ as Kubernetes ConfigMap resources.

  • Label: grafana_dashboard: "1".
  • Sidecar: Grafana uses a sidecar to watch these ConfigMaps and load them automatically.

Active Dashboards

  • Cluster Health: High-level node and pod metrics.
  • Nodes-Pro: Detailed hardware and kernel metrics.

See Also