TazLab K8s: Multica Detail

HISTORICAL — 2026-10-20. Multica was decommissioned and uninstalled on this date; this page is kept as a record of the experiment.

Level 3 (Detail) — Multica (agent orchestration control plane) on tazlab-k8s plus its daemon on CT106: namespace, chart values and the four rendered-output patches, LAN exposure, certificate database identity, Vault-delivered secrets, image lane, and the headless login that mints the daemon token.

Concept

Multica (multica.ai, github.com/multica-ai/multica) is a control plane for coding agents: work is filed as issues, assigned to agents like to a colleague, and executed on a runtime — a machine that runs the agent CLI and reports back. It is the lab’s answer to “the map and the visibility” problem: which session is doing what, and what it decided.

Two halves, one project:

CRISP record: project multica-agent-orchestration (RESEARCH.md, DESIGN.md, PLAN.md, tasks.md, REVIEW.md…REVIEW-5.md, retrospective.md).

Manifests

Cluster manifests (tazlab-k8s)

Registration points (the app is invisible to Flux without them): the namespace in infrastructure/operators/namespaces/kustomization.yaml, the ServiceAccount in infrastructure/operators/vso/kustomization.yaml, the two Vault objects in infrastructure/operators/vso-secrets/kustomization.yaml, the grants Job in the tazlab-db instance kustomization, and the automation in infrastructure/cluster-instances/kustomization.yaml.

Database identity — certificate, not password

Role multica on the shared cluster tazlab-db (PostgreSQL 17.2), declared in spec.users, with four pg_hba rows — two cert and two reject, above the generic rows, because pg_hba is first-match-wins and without the reject rows the role would still be reachable through md5:

hostssl multica multica 0.0.0.0/0 cert
hostssl multica multica ::/0      cert
host    all     multica 0.0.0.0/0 reject
host    all     multica ::/0      reject

The connection string carries no user password:

postgresql://multica@tazlab-db-primary.tazlab-db.svc:5432/multica?sslmode=verify-full&sslcert=/etc/certs/tls.crt&sslkey=/etc/certs/tls.key&sslrootcert=/etc/certs/ca.crt

The identity is the certificate CN=multica (Vault PKI role db-client-multica), so a destroy+create cycle reissues it instead of destroying a secret. Schema grants are applied by the Git-declared Job, not by the cluster’s databaseInitSQL (that runs only at initialisation, which already happened).

Secrets

multica-secrets is not templated by the chart (its note says so explicitly: “real values never need to land in git”), which is exactly what makes it compatible with VSO. One VaultStaticSecret reads one KV entry and turns its fields into the Secret’s keys — the lab’s “one path per key” convention would have produced a Secret with a single value key.

Source: secret/tazlab-k8s/static/apps/multica/merged (seven fields; the one declared imperative write of the project — no Git object can author a KV entry). db-client-multica-tls is a second, independent Vault delivery (PKI), mounted at /etc/certs by patch #3.

The four postRenderer patches

The chart hardcodes type: ClusterIP on both Services, renders no Service annotations, no extra volumes and no frontend probes. Rather than fork the chart, apps/base/multica/helmrelease.yaml patches the rendered output:

Patch targets are version-bound. They were checked against helm template output of chart 0.5.3; a chart bump must re-render, because an unmatched target is a silent no-op, not an error — the two ways a patch can vanish without a word are a renamed object (this deployment hit exactly that: targets were first written as multica-backend instead of multica-multica-backend) and a changed container name.

Exposure

Live: http://192.168.1.240:3000 → 200 from the VIP; /api/config answers with daemon_server_url/daemon_app_url = http://192.168.1.240:3000 and allow_signup: true. The root URL serves the upstream marketing site — the application itself is at /login, then /onboarding, then /<slug>/issues.

Authentication without an email provider

RESEND_API_KEY is empty, so EmailService starts in DEV mode and prints the login code to the backend’s stdout:

[DEV] Verification code for <email>: 6 digits

That is the path in use here, and it is not the same as MULTICA_DEV_VERIFICATION_CODE: that variable is read only when APP_ENV != production (isDevVerificationCode returns false immediately in production), and this deployment runs APP_ENV=production — as the lab should. So the fixed dev code is inert and the log is the source.

The mail path is configured but not armed. The HelmRelease sets resendFromEmail: onboarding@resend.dev — the chart default noreply@multica.ai belongs to a domain this lab does not own, so the moment RESEND_API_KEY is set Resend would answer 403 and the code would never arrive; onboarding@resend.dev is the sandbox sender of a free Resend account, and it delivers only to the address the account was created with — the operator’s mailbox, which is also the only address that can receive mail here (tazlab.net publishes no MX). With the key still empty the DEV-mode path above is what works, so a reader who sees the sender value should not conclude that email is live. allowSignup stays at the chart default: an allowlist would add no restriction the sandbox sender does not already apply, and the exposure is LAN-only.

The headless login (what multica login does in a browser, done with curl):

POST /auth/send-code      {"email":"…"}                      → 200 (code stored, printed to the log)
kubectl logs deploy/multica-multica-backend -n multica       → read the code
POST /auth/verify-code    {"email":"…","code":"…"}           → 200 { token: <session JWT> }
POST /api/tokens          {"name":"CLI (<host>)","expires_in_days":90}  (Bearer = JWT)
                                                             → 201 { token: "mul_…" }

The account is taz@tazlab.net (the lab’s git identity). The PAT is a 90-day credential; the daemon renews it in place (POST /api/tokens/current/renew) once it enters the 7-day window before expiry, so it does not expire under a running daemon. The token must be minted for the operator account (roberto.tazzoli@gmail.com): the runtime rows, the agents bound to them and the runtime MCP inventory are owned by the token’s user, so a token minted elsewhere registers runtimes nobody can route work to.

Image lane

Upstream releases every 2–3 days, which is why the version is pinned in Git and the daemon’s self-update is off: the layer owns the version + sha256, the automation owns the tag, and nothing moves on its own.

Rotation actually reaches the pods

envFrom is read at container start, so a rotated secret lands in the Secret and changes nothing until the Deployment restarts. Both Vault deliveries therefore carry rolloutRestartTargets, and both name the release-prefixed Deployments the chart renders:

The first revision named them multica-backend/multica-frontend, which do not exist: VSO logged “Rollout restart failed … not found” on every sync and a rotated key would have sat unused in the Secret. The same naming trap hit the postRenderers patch targets (§The four postRenderer patches) — every name in this deployment is release-prefixed, and a wrong one is silent.

The daemon on CT106

Ansible layer tazpod/runtimes/lxc → role tazpod, layer6-multica.yml (installed by create.sh phase 4, like every other layer). It is a pinned release binary (multica 0.5.3, checksum-verified from the release’s own checksums.txt, installed to /usr/local/bin/multica) plus a hand-owned systemd unit.

Two switches are off for a systemd reason, not a policy one: when its binary changes version the daemon spawns a successor and exits, which under Type=exec leaves systemd supervising a dead main process (“the unit reads as stopped, and the next start fails with daemon is already running”). The layer replaces the binary and its restart multica-daemon handler reloads it, so a version bump is a systemd event like everywhere else.

Operations

systemctl status multica-daemon                 # as root
multica daemon status --output json             # as tazpod: pid, agents, workspaces, runtimes
multica daemon logs -n 100                      # prints the resolved log path first
multica runtime list --workspace-id <uuid> --output json
multica workspace switch <slug>                 # the CLI's default workspace (see caveat)

Caveat — the CLI’s default workspace. The layer renders only server_url, app_url, token. The daemon does not need more (it fetches every workspace the account belongs to from the server), but interactive CLI commands need a workspace: pass --workspace-id / MULTICA_WORKSPACE_ID, or run multica workspace switch <slug> — noting that a later layer run rewrites config.json and drops that key again.

Rebuild survivability: binary and configuration come from Git, the token from gopass, the workspace cache from the pet volume. A rebuilt container comes back authenticated and re-registers its runtimes with no manual step.

The agent team (declared in Git)

The control plane’s database is the one place in this lab where an object can exist without a commit behind it — and an agent created in a dashboard comes back after a rebuild only if the backup was good and the timing lucky. The team is therefore declared in tazpod/runtimes/lxc/configs/multica-team/ and reconciled onto the control plane by Ansible layer layer7-multica-team.

Live (2026-09-28): eleven agents in the tazlab workspace, all public_to, all max_concurrent_tasks: 1, each bound to the shared skills; the daemon on CT106 reports three registered runtimes.

DAG Position

infrastructure-operators-namespaces (creates Namespace multica)
infrastructure-operators-vso        (VSO; allowedNamespaces must list multica)
infrastructure-operators-vso-secrets (Secret multica-secrets + db-client-multica-tls)
infrastructure-configs
→ apps-multica (Level 3: OCIRepository + HelmRelease → backend/frontend Deployments, Services, PVC)

apps-multica carries prune, dependsOn: infrastructure-configs and postBuild.substituteFrom: cluster-vars (that is where ${traefik_lb_ip} comes from). Image automation is not a Kustomization: it is registered in infrastructure/cluster-instances/kustomization.yaml.

See Also

• Parent hub: tazlab-k8s
• Database: tazlab-db Detail · Secrets Mapping
• Image lane: Image Automation Detail · Flux Kustomizations Detail
• Vault/VSO: Hashicorp Vault · Secrets Mapping
• Sibling app on the same host: Paperclip container detail (the other agent control plane — a dedicated CT108 guest since 2026-09-26; the CT106-era deployment is kept as history)
• Agent identity in the cluster: paperclip-observer Identity Detail
• Host: TazPod Proxmox Runtime Detail (CT106 LXC, pet volume, layer list)
• Source: github.com/multica-ai/multica · upstream CLI_AND_DAEMON.md
• Design record: CRISP multica-agent-orchestration (retrospective.md for what went wrong and what the design missed)