Paperclip: the dedicated agent container (CT108 paperclip) Detail

HISTORICAL — 2026-10-19. Paperclip was decommissioned and uninstalled on this date; this page is kept as a record of the experiment.

Level 3 (Detail) — the LXC that runs the agent company: why it exists as its own guest, how it is provisioned from tazpod/runtimes/lxc-paperclip, the identity and secret store it owns, and the egress chokepoint that brokers its credentials.

Concept

Paperclip is the lab’s first agent control plane. It was built (2026-09-25) inside CT106, the operator container — and that was wrong in one specific way: the agents are meant to work, and the operator container is the operator’s own environment. As built on CT106 the service ran as tazpod, with /workspace writable in full (the Paperclip state, the master key and its own instructions included) and GITHUB_TOKEN in cleartext in the env file that every spawned agent inherited.

CT108 is the correction: a dedicated, unprivileged LXC built and provisioned on 2026-09-26 by project paperclip-dedicated-container, running the same company against the same database, with its own service account, its own gopass store, its own GPG key, a read-only cluster identity, and an egress chokepoint that holds the credentials instead of the container. The CT106 service was stopped and disabled when the company was carried, so only one scheduler runs.

The old CT106 deployment is kept as the historical record on Paperclip on the TazPod container; this page describes the container the company runs in now.

Repository layout

Everything is under tazpod/runtimes/lxc-paperclip/ (a sibling of the operator runtime tazpod/runtimes/lxc/):

runtimes/lxc-paperclip/
├── RUNBOOK.md                 # operator procedure: provision, secrets, carry, verify, rollback, routine, egress
├── create.sh / destroy.sh     # 6 timed phases; destroy detaches mp0 first so /workspace survives
├── configs/runtime.env        # CT id, address, pool, volume name/path, host, node
├── terraform/                 # the guest (bpg/proxmox); proxmox-access.tf is operator-gated (T4)
└── ansible/
    ├── paperclip-baseline.yml # play: roles `paperclip` then `paperclip-egress`; vars paperclip_operator_user: tazpod
    └── roles/
        ├── paperclip/         # layers 0, 0b, 1, 2, 3
        └── paperclip-egress/  # mitmproxy + nftables chokepoint

create.sh phases: 0 preflight, 1 Terraform create, 2 raw config apply, 3 wait SSH, 4 Ansible baseline, 5 verify. Both scripts refuse to run while the operator-gated Proxmox identity objects (T4) are in Terraform state — the guest’s lifecycle and the operator’s own PVE account are deliberately not the same apply.

The five layers (roles/paperclip)

The service unit

Rendered from templates/paperclip.service.j2:

ProtectSystem=strict plus ReadWritePaths=/workspace still means the whole data volume is writable by the agent — including its own master key and instructions. What changed with CT108 is whose volume it is: the operator environment, the repository checkouts and the operator’s gopass store are no longer inside it.

Secrets: the container’s own store

The container does not read the operator’s gopass. It has a separate store, agent, on the data volume at /workspace/gopass-agent, encrypted to a key generated inside the guest, addressed from the CLI as agent/<path>.

• Seeding is one-way and never touches a file, argv or a log: for each missing-or-empty entry the play reads gopass cat <operator path> on the orchestration host (as tazpod) and writes it with gopass insert --force over stdin (no_log: true). The map is paperclip_gopass_entries in the role defaults — the four Paperclip application secrets, the provider key, the Exa key, the GitHub token, and the read-only cluster token (paperclip/k8s-token, optional, not canary-guarded).
• core.readonly=true is a convenience guard, not enforcement. Rotation and addition flip it off, write, and flip it back (RUNBOOK §6).
• The canary is ExecStartPre, so a broken store is a failed start, not a misbehaving app. It checks in order: single recipient in .gpg-id (exit 2 missing, 3 not exactly one), keyring holds exactly one sec whose fingerprint matches (4), every canary entry decrypts cold and non-empty (5), an entry belonging to the operator store must not be readable (6, the scoping proof), and HOME/GNUPGHOME set (7). It length-tests values and never prints them.
• Brokered credentials are not in the store’s reach of the app: the GitHub token and the Proxmox token pair live in the chokepoint (§Egress chokepoint), which injects them per destination.

The carry (2026-09-26)

The company’s identity is four artefacts, and only those: secrets/master.key, config.json, data/storage and data/run-logs (~206 MB) under /workspace/.paperclip/instances/default/. rsynced to CT108 after stopping and disabling CT106’s unit — two schedulers against one database is the failure the design forbids.

• The layer never runs paperclipai onboard: it would mint a new instance id and a new master.key, orphaning the secrets the database stores. Until the carry artefacts exist the playbook says so and leaves the unit disabled; re-running it after the copy fixes rsync’s root-owned files, teaches the carried config.json the new address (paperclipai allowed-hostname) and starts the service.
• Rollback is one instance swap: stop CT108’s unit, pct exec 106 -- systemctl enable --now paperclip.service. CT108 can then be destroyed; /workspace survives (destroy.sh detaches mp0 before destroy).
• Proof of a valid carry: paperclipai doctor clean, the UI shows the same company/agents/issues, and one run binding a secret_ref succeeds — the failure to watch for is Secret decryption failed (master key fingerprint: …).

Identities

Tailnet

layer0b-tailnet installs Tailscale from the upstream repo and joins with a short-lived, ephemeral auth key minted on the controller from the Tailscale OAuth client (gopass show -o bootstrap/tailscale/oauth-client-…), handed over as TS_AUTHKEY=file:/run/paperclip-tailscale-authkey (0600, deleted after use — never argv), hostname paperclip-agent, --accept-dns=false --stateful-filtering=false.

• Userspace mode, because the device is absent: with no /dev/net/tun the role writes a tailscaled drop-in using --tun=userspace-networking (TD-093). This is not cosmetic: userspace mode has no direct sockets and no inbound, so the chokepoint cannot open a direct socket to Vault and the dashboard is reached on the LAN address.
• The layer extends the service’s allowed hostnames with the tailnet name, so the dashboard answers on both when the interface is up.
• Observed 2026-09-28 (from CT106, a tailnet member): tailscale status lists six peers and no paperclip-agent. An ephemeral node leaves the tailnet when it goes offline, so the guest’s tailnet presence is by design not durable — a reader should treat the tailnet name as reachable when the node is up, and 192.168.1.208 as the address that always answers. Whether the drop-in and tailscaled are currently healthy inside the guest was not verified from here: the provisioning key is pinned to the orchestration host, so CT108 is not directly reachable from CT106.

Egress chokepoint (T4b, roles/paperclip-egress)

The container’s outbound traffic goes through a mitmproxy forward proxy on 127.0.0.1:3128 that terminates TLS with its own CA, decides on the destination, and injects the credential the destination expects — so the token is not in the agent’s environment at all.

Fail-closed, four ways: the packet filter drops the agent’s direct egress, so proxy-down means “no egress” rather than “unproxied egress”; BindsTo= stops the agent with the chokepoint; the addon shuts the proxy down on an unreadable policy or credentials; and a Vault token that cannot be minted disables only the Vault rule. Provisioning proves it: a probe to 1.1.1.1:443 succeeds as paperclip-egress and fails as paperclip.

Two limits to know before trusting the brokering:

• The Proxmox rule is currently dead (TD-095): the API’s certificate is issued by the PVE Cluster Manager CA, which is neither sent in the handshake nor present in the guest, and the certificate names tazlab.net while the rule addresses 192.168.1.200. paperclip_egress_upstream_ca_file is empty, so an operator would have to install the node certificate first. The chokepoint’s GitHub injection is unaffected.
• No vault: block, so nothing is brokered to Vault (§Identities).

Operations

# from the orchestration host (tazlab) — the provisioning key is pinned to it
cd /root/tazpod/runtimes/lxc-paperclip && ./create.sh          # create/rebuild, all phases
ANSIBLE_CONFIG=ansible/ansible.cfg ansible-playbook -i ansible/inventory.ini ansible/paperclip-baseline.yml

# verification (RUNBOOK §4)
ssh root@192.168.1.208 systemctl is-active paperclip.service
curl -fsS http://192.168.1.208:3100/api/health
ssh root@192.168.1.208 runuser -u paperclip -- env HOME=/workspace/home \
  GNUPGHOME=/workspace/home/.gnupg /usr/local/bin/paperclip-secrets-canary     # exit 0
ssh root@192.168.1.208 runuser -u paperclip -- sudo -n -l                      # must fail
ssh root@192.168.1.208 runuser -u paperclip -- ssh -o BatchMode=yes root@192.168.1.200 true   # must fail

Routine: re-run the playbook to converge (--check --diff for a dry run); rotate a secret by flipping core.readonly (RUNBOOK §6); re-run the pi extension installs by removing /workspace/home/.pi/.layer3-extensions-installed; an upgrade of a pinned tool re-runs by deleting its binary/marker; alerts land in journalctl -t paperclip-alert, the unit log in journalctl -u paperclip.

Deliberately absent, each for a recorded reason: no paperclipai onboard (the carry), no Docker (nesting off; a reachable docker.sock is an escape), no operator keys or store, no store push remote (its credential cannot come from the store itself — add it when the repo exists: cd /workspace/gopass-agent && gopass git remote add origin <url>), and no egress rules in this layer.

Known limits and debt

• TD-093 — no /dev/net/tun, so the tailnet runs in userspace mode and the chokepoint cannot reach Vault directly; the fix is the two raw-config lines the operator runtime already writes for CT106, after which the layer picks kernel mode up by itself.
• TD-095 — the brokered Proxmox rule cannot verify its upstream TLS (§Egress chokepoint).
• TD-099 — Multica’s uploads PVC had no recurring backup; closed on 2026-10-20 when the PVC and its three volumes were removed with the Multica decommission.
• mitmproxy is installed from the Ubuntu package with no version pin; the addon’s measured behaviour is recorded against 8.1.1, including the crash that took the whole process down inside running().
• The container’s tailnet presence is ephemeral by design (see §Tailnet), so “the guest is on the tailnet” is a state, not a property.

See Also

• Parent hub: tazpod
• The previous host (historical): Paperclip on the TazPod container — the CT106 deployment, the read-the-code discoveries, and the CLI traps
• The agents it runs: TazLab Agent Company
• Operator runtime: Proxmox LXC Runtime Detail — CT106, the trusted orchestration host, and the pet volume model
• Database: tazlab-db Detail — role paperclip, md5 over TLS, the pinned NodePort
• Cluster identity: Multica Detail · TazLab K8s entity
• Vault: Hashicorp Vault — the approle-agent mount the chokepoint will consume
• CRISP: project paperclip-dedicated-container (RUNBOOK.md, DESIGN.md, retrospective in SKILLS/crisp/projects/)