Hermes Agent: KVM VM Deployment Architecture

Level 2 (Topic)

SUPERSEDES hermes-lxc-deployment.md — the LXC container deployment (CT 105) was destroyed on 2026-08-12 and replaced by this dedicated KVM VM architecture.

⚠️ DECOMMISSIONED (2026-09-26) — VM 501 was destroyed. It had been unused. The page below is kept as the architectural record; the operational sections no longer describe a live surface. The data disk survives as local-lvm:vm-501-data-orphan, and ./create.sh can bring the VM back with zero re-onboarding (see Removed: VM 501 decommissioned).

Concept

Hermes Agent runs as a bare-metal installation inside a dedicated KVM VM (VM 501 hermes) on Proxmox node tazlab. The deployment follows the same enterprise pattern as the Hetzner Vault runtime: Terraform creates the VM, Ansible configures the internal state, and a top-level shell orchestrator (create.sh/destroy.sh) ties them together with structured logging and timing instrumentation.

The critical architectural challenge was achieving data persistence across destroy/create cycles on LVM-thin storage for VMs. Unlike LXC (where a “pet” container could own the volume), qm destroy deletes all volumes matching the vm-<vmid>-* naming convention — even unused ones, even with --destroy-unreferenced-disks 0. The solution is the lvrename orphan pattern: rename the data volume outside the naming convention before destroy, rename it back on the next create.

Architecture / Design

VM Model

• VM 501 hermes: KVM, Ubuntu 26.04 LTS “resolute”, 4 vCPU, 8 GB RAM
• Static IP 192.168.1.205/24, gw 192.168.1.1, machine q35, BIOS ovmf (UEFI)
• Users: bootstrap (admin, sudo NOPASSWD), hermes (UID 10000, non-root, no sudo)
• Hermes runs bare-metal (install.sh) — no Docker inside VM

Storage Architecture

• Root disk: virtio0, 30 GB local-lvm — ephemeral, destroyed with the VM (proven: cowsay gone after cycle)
• Data disk: virtio1, 20 GB local-lvm:vm-501-disk-2 — persistent pet, mounted at /home/hermes (whole home = state + code, option A)
• No backup/restore needed: data disk survives destroy/create via the lvrename orphan pattern
• Cloud image as reusable asset: lifecycle.ignore_changes on download (import doesn’t populate url → would force re-download + 403 delete)

Persistence Model (lvrename orphan pattern)

# create.sh (first run):
VOLID=$(pvesm alloc local-lvm 501 vm-501-disk-2 20G)   # canonical name vm-<VMID>-disk-<N>
terraform apply                                          # VM + root disk (data disk NOT in state)
qm set 501 --virtio1 "$VOLID"                            # attach with exact volid

# destroy.sh:
qm unlink 501 --idlist virtio1                           # creates unused[n]
lvrename local-lvm/vm-501-disk-2 local-lvm/vm-501-data-orphan   # OUT of naming convention
qm destroy 501                                            # deletes only vm-501-* (orphan safe)
terraform state rm ...                                   # clean state

# create.sh (subsequent):
lvrename local-lvm/vm-501-data-orphan local-lvm/vm-501-disk-2   # rename back
terraform apply                                          # idempotent re-run
qm set 501 --virtio1 local-lvm:vm-501-disk-2

Key guards:

• delete_unreferenced_disks_on_destroy = false in Terraform (default true would delete the untracked data disk)
• agent.wait_for_ip.disabled = true — enables virtio-serial channel but skips IP wait (static via cloud-init) → apply drops from ~15 min to <1 min
• Existence check via pvesm list local-lvm --vmid 501 (not lvs | grep)

Lifecycle

create.sh:  Cleanup gate → Pre-flight → Terraform (<1 min) → Attach data →
            Wait SSH → Ansible Baseline → Agent → Configure → Verify
            TOTAL: ~2 min (fast path)

destroy.sh: qm unlink virtio1 → lvrename to orphan → qm destroy → terraform state rm
            (data disk vm-501-data-orphan preserved — verified with lvs)

Key Design Decisions

• KVM instead of LXC: full kernel isolation, own kernel, full IP ownership (LXC shared the host kernel; KVM has separate kernel)
• Bare-metal install, not Docker: same rationale as before (isolation provided by the VM itself)
• Data disk OUTSIDE Terraform state: if declared in .tf, terraform destroy deletes it
• lvrename instead of relying on flags: qm destroy deletes by naming convention — empirically verified with 4 destructive tests
• Cloud-init: user account + SSH key injection + static IP (no snippets — token lacks Datastore.Allocate on local)
• SSH host key: reused from gopass (infra/ssh-host-keys/hermes-agent/ed25519), injected via Ansible post-boot (never in Terraform state)

Dashboard Access (historical — surface removed 2026-09-26)

• Dashboard binds 0.0.0.0:9119 (fix: was 127.0.0.1 — notebook couldn’t reach it)
• Basic auth required for non-loopback bind: user hermes, password in gopass infra/hermes-vm/dashboard-password
• Scrypt hash format (verified): scrypt$16384$8$1$<b64(salt)>$<b64(dk)> — STANDARD base64 with padding, maxmem=0. urlsafe/no-padding fails verification. Hash generated LOCALLY (where gopass exists), never inside the VM.
• Login endpoint: POST JSON to /auth/password-login with body {provider, username, password, next} (not /auth/password). Verified: 200 → dashboard 200.
• LLM backend configured by USER via dashboard at first access (30+ providers; onboarding UI)

Post-2026-08-12 Additions

Everything below was deployed AFTER the base build (2026-08-12) described by the preceding sections; the base architecture is unchanged.

Self-restart — hermes-restart.service (root oneshot, 2026-08-13)

• Root oneshot unit: Type=oneshot, ExecStart=/usr/bin/systemctl restart hermes-gateway.service hermes-dashboard.service, TimeoutStartSec=90. Runs as ROOT via systemd (not as hermes) so it survives the gateway shutdown — Hermes can restart itself without dying mid-command. Deployed via the agent role (agent/tasks/main.yml).
• Polkit scoping (baseline/templates/50-hermes-services.rules.j2): grants user hermes the start verb on hermes-restart.service (plus management of her own services) — no sudo anywhere.
• hermes gateway restart CLI is NOT bypassable: hardcoded root guard in the binary (SystemScopeRequiresRootError, see configure/files/50-hermes-gateway.rules header) — polkit does NOT bypass it. The working self-restart path is systemctl start hermes-restart.service.

User systemd units (2026-08-18)

• loginctl enable-linger hermes — the user manager survives logout and starts at boot (deployed + verified Linger=yes).
• Caddy research publisher — hermes-research.service: user-installed Caddy binary at ~/.local/bin/caddy (48 MB static), serving ~/research-site/public on :8080, VPN-only (no public domain); config at ~/research-site/Caddyfile. User units are managed by Hermes with systemctl --user — no polkit rule needed.
• hermes-chrome.service: durable single-instance headless Chrome (Playwright chromium, --headless=new --no-sandbox — accepted Playwright-on-Linux tradeoff), CDP on 127.0.0.1:9222, --user-data-dir=~/.browser-profile. Rule: Hermes connects via Playwright connect_over_cdp ONLY — it must NEVER spawn its own browser (single instance, no port conflict).

Samba/CIFS mount (2026-08-20)

• Source: //192.168.1.207/shared (Samba server on CT 107 lxc-storage) → /mnt/shared.
• Credentials: /etc/cifs-credentials-hermes (root, mode 0600, no_log task) written from playbook env SAMBA_USERNAME/SAMBA_PASSWORD (gopass infra/hermes-vm/samba-username + infra/hermes-vm/samba-password) — NEVER inline in fstab.
• fstab options: credentials=/etc/cifs-credentials-hermes,uid=10000,gid=10000,file_mode=0770,dir_mode=0770,_netdev,x-systemd.automount — automount avoids boot-time hangs when the share is unreachable.
• Server-side (NOT in IaC): the hermes/ subfolder on the share was chown’ed 1000:1000 with setgid set, so files created by Hermes are group-shared.

Refurb collector (VM-side only — NOT in the hermes-agent repo)

• Lives in /home/hermes on the VM: no Ansible role, no repo code (grep for refurb in the IaC repo returns zero hits). It is Hermes-managed workspace, deliberately outside the IaC lifecycle.
• Persistent JSON baseline + clickable annunci-live.html output.
• Sources v1 (Refurbed, Messoanuovo, SimpaticoTech) → v2 adds Amazon Ricondizionato, Ricondizionato.it, BuyTec; Back Market excluded (HTTP-level anti-bot).
• Filter refined to ONLY ThinkPad T14 / L14 / X13 Gen 2; cron every 2 days with an empty-cycle guard.

Removed: VM 501 decommissioned (2026-09-26)

The board destroyed the unused Hermes Agent VM on 2026-09-26 via destroy.sh (run log: /workspace/hermes-agent/logs/destroy_20260926_075300.log).

Evidence from the run:

[phase2] update VM 501: -delete virtio1          → unused0: local-lvm:vm-501-disk-2
[phase3] Renamed "vm-501-disk-2" to "vm-501-data-orphan"
         Logical volume "vm-501-cloudinit" removed / "vm-501-disk-1" removed / "vm-501-disk-0" removed
[phase3.5] Successfully removed 1 resource instance(s)   (terraform state)
[phase4] DATA DISK PRESERVED: local-lvm:vm-501-disk-2 (as vm-501-data-orphan)

Preserved on purpose: the 20 GB data disk (local-lvm:vm-501-disk-2), renamed to local-lvm:vm-501-data-orphan — the whole /home/hermes state. The lvrename orphan pattern (documented above) is exactly what made this safe: renaming the volume out of the vm-501-* convention before qm destroy leaves it untouched.

Recreate with zero re-onboarding: ./create.sh renames vm-501-data-orphan back to vm-501-disk-2, recreates the VM, reattaches the disk and re-runs Ansible against the same pet volume.

Live verification (2026-09-26, from CT106 tazpod-proxmox):

ssh root@192.168.1.200 'qm list | grep -E "^\s*501\s"'   # → no output: VM 501 NOT PRESENT
ssh root@192.168.1.200 'lvs | grep vm-501'
# → vm-501-data-orphan pve Vwi-a-tz-- 20.00g data 28.59

Historical context worth keeping: the lvrename orphan pattern was derived from four destructive tests and is the reason a decommission is reversible here. The VM root disk was always ephemeral; only the data volume is a pet. The design, the create/destroy phases and the Samba/Chrome/research additions below document the system as it was built (2026-08-12 to 2026-08-20).

Research History

• skills/crisp/projects/hermes-agent-redesign/web-research/proxmox-vm-disk-persistence.md — VM disk persistence on Proxmox
• skills/crisp/projects/hermes-agent-redesign/web-research/hermes-state-persistence.md — Hermes state persistence (option A: whole home)
• Lesson: destructive tests are mandatory for Proxmox persistence patterns — research alone missed the qm destroy naming-convention behavior (4 blockers found by tests, 3 by independent review)

Iteration History

Iterations 1–6 = base deployment (2026-08-12). Later additions appended below (not part of the base IaC feature set).

Known Issues / Technical Debt

• TD-056: gopass GPG cache expires ~45 min — long builds fail with “gopass locked” (4+ times during build); gopass show -o truncates multi-line secrets (use gopass cat for SSH keys)
• Root disk is ephemeral by design — any manual install outside /home/hermes is lost on cycle

gopass secret paths (create/destroy + samba)

gopass show bootstrap/proxmox/token-id
gopass show bootstrap/proxmox/token-secret
gopass show infra/hermes-vm/dashboard-password
gopass cat infra/ssh-host-keys/hermes-agent/ed25519   # cat, NOT show -o (multi-line!)
export SAMBA_USERNAME="$(gopass show -o infra/hermes-vm/samba-username)"   # before the configure run (samba)
export SAMBA_PASSWORD="$(gopass show -o infra/hermes-vm/samba-password)"

See Also

• Parent hub: Hermes Agent
• Superseded: LXC Deployment Architecture — historical CT 105 deployment (destroyed 2026-08-12)
• Infrastructure layer: ephemeral-castle — former home, hermes/ removed
• Operator environment: tazpod