Hermes Agent: KVM VM Deployment Architecture

Level 2 (Topic)

SUPERSEDES hermes-lxc-deployment.md — the LXC container deployment (CT 105) was destroyed on 2026-08-12 and replaced by this dedicated KVM VM architecture.

Concept

Hermes Agent runs as a bare-metal installation inside a dedicated KVM VM (VM 501 hermes) on Proxmox node tazlab. The deployment follows the same enterprise pattern as the Hetzner Vault runtime: Terraform creates the VM, Ansible configures the internal state, and a top-level shell orchestrator (create.sh/destroy.sh) ties them together with structured logging and timing instrumentation.

The critical architectural challenge was achieving data persistence across destroy/create cycles on LVM-thin storage for VMs. Unlike LXC (where a “pet” container could own the volume), qm destroy deletes all volumes matching the vm-<vmid>-* naming convention — even unused ones, even with --destroy-unreferenced-disks 0. The solution is the lvrename orphan pattern: rename the data volume outside the naming convention before destroy, rename it back on the next create.

Architecture / Design

VM Model

VM 501 hermes: KVM, Ubuntu 26.04 LTS “resolute”, 4 vCPU, 8 GB RAM
Static IP 192.168.1.205/24, gw 192.168.1.1, machine q35, BIOS ovmf (UEFI)
Users: bootstrap (admin, sudo NOPASSWD), hermes (UID 10000, non-root, no sudo)
Hermes runs bare-metal (install.sh) — no Docker inside VM

Storage Architecture

Root disk: virtio0, 30 GB local-lvm — ephemeral, destroyed with the VM (proven: cowsay gone after cycle)
Data disk: virtio1, 20 GB local-lvm:vm-501-disk-2 — persistent pet, mounted at /home/hermes (whole home = state + code, option A)
No backup/restore needed: data disk survives destroy/create via the lvrename orphan pattern
Cloud image as reusable asset: lifecycle.ignore_changes on download (import doesn’t populate url → would force re-download + 403 delete)

Persistence Model (lvrename orphan pattern)

# create.sh (first run):
VOLID=$(pvesm alloc local-lvm 501 vm-501-disk-2 20G)   # canonical name vm-<VMID>-disk-<N>
terraform apply                                          # VM + root disk (data disk NOT in state)
qm set 501 --virtio1 "$VOLID"                            # attach with exact volid

# destroy.sh:
qm unlink 501 --idlist virtio1                           # creates unused[n]
lvrename local-lvm/vm-501-disk-2 local-lvm/vm-501-data-orphan   # OUT of naming convention
qm destroy 501                                            # deletes only vm-501-* (orphan safe)
terraform state rm ...                                   # clean state

# create.sh (subsequent):
lvrename local-lvm/vm-501-data-orphan local-lvm/vm-501-disk-2   # rename back
terraform apply                                          # idempotent re-run
qm set 501 --virtio1 local-lvm:vm-501-disk-2

Key guards:

delete_unreferenced_disks_on_destroy = false in Terraform (default true would delete the untracked data disk)
agent.wait_for_ip.disabled = true — enables virtio-serial channel but skips IP wait (static via cloud-init) → apply drops from ~15 min to <1 min
Existence check via pvesm list local-lvm --vmid 501 (not lvs | grep)

Lifecycle

create.sh:  Cleanup gate → Pre-flight → Terraform (<1 min) → Attach data →
            Wait SSH → Ansible Baseline → Agent → Configure → Verify
            TOTAL: ~2 min (fast path)

destroy.sh: qm unlink virtio1 → lvrename to orphan → qm destroy → terraform state rm
            (data disk vm-501-data-orphan preserved — verified with lvs)

Key Design Decisions

KVM instead of LXC: full kernel isolation, own kernel, full IP ownership (LXC shared the host kernel; KVM has separate kernel)
Bare-metal install, not Docker: same rationale as before (isolation provided by the VM itself)
Data disk OUTSIDE Terraform state: if declared in .tf, terraform destroy deletes it
lvrename instead of relying on flags: qm destroy deletes by naming convention — empirically verified with 4 destructive tests
Cloud-init: user account + SSH key injection + static IP (no snippets — token lacks Datastore.Allocate on local)
SSH host key: reused from gopass (infra/ssh-host-keys/hermes-agent/ed25519), injected via Ansible post-boot (never in Terraform state)

Dashboard Access

Dashboard binds 0.0.0.0:9119 (fix: was 127.0.0.1 — notebook couldn’t reach it)
Basic auth required for non-loopback bind: user hermes, password in gopass infra/hermes-vm/dashboard-password
Scrypt hash format (verified): scrypt$16384$8$1$<b64(salt)>$<b64(dk)> — STANDARD base64 with padding, maxmem=0. urlsafe/no-padding fails verification. Hash generated LOCALLY (where gopass exists), never inside the VM.
Login endpoint: POST JSON to /auth/password-login with body {provider, username, password, next} (not /auth/password). Verified: 200 → dashboard 200.
LLM backend configured by USER via dashboard at first access (30+ providers; onboarding UI)

Research History

skills/crisp/projects/hermes-agent-redesign/web-research/proxmox-vm-disk-persistence.md — VM disk persistence on Proxmox
skills/crisp/projects/hermes-agent-redesign/web-research/hermes-state-persistence.md — Hermes state persistence (option A: whole home)
Lesson: destructive tests are mandatory for Proxmox persistence patterns — research alone missed the qm destroy naming-convention behavior (4 blockers found by tests, 3 by independent review)

Iteration History

Iteration	Feature	Status
1	VM base + networking + SSH (cloud-init)	Done
2	Hermes bare-metal install (install.sh, –skip-setup)	Done
3	Configuration (config.yaml, systemd units, dashboard :9119)	Done
4	Persistence (lvrename orphan pattern, 4 destructive tests)	Done
5	Dashboard bind 0.0.0.0 + basic auth	Done
6	Idempotency (create.sh re-run = no-op)	Done

Known Issues / Technical Debt

TD-056: gopass GPG cache expires ~45 min — long builds fail with “gopass locked” (4+ times during build); gopass show -o truncates multi-line secrets (use gopass cat for SSH keys)
Root disk is ephemeral by design — any manual install outside /home/hermes is lost on cycle

Reference Blocks

gopass secret paths (create/destroy)

gopass show bootstrap/proxmox/token-id
gopass show bootstrap/proxmox/token-secret
gopass show infra/hermes-vm/dashboard-password
gopass cat infra/ssh-host-keys/hermes-agent/ed25519   # cat, NOT show -o (multi-line!)

Hermes Agent: KVM VM Deployment Architecture#

Concept#

Architecture / Design#

VM Model#

Storage Architecture#

Persistence Model (lvrename orphan pattern)#

Lifecycle#

Key Design Decisions#

Dashboard Access#

Research History#

Iteration History#

Known Issues / Technical Debt#

Reference Blocks#

gopass secret paths (create/destroy)#

See Also#