TazLab K8s: paperclip-observer Identity Detail

HISTORICAL — 2026-10-19. Paperclip was decommissioned and uninstalled on this date; this page is kept as a record of the experiment.

Level 3 (Detail) — the read-only Kubernetes identity granted to the Paperclip agent container: what it may read, what it deliberately may not, how its token reaches the guest, and how to revoke it.

Concept

The agent container runs the lab’s agent company, and one of its recurring jobs is producing health reports about the cluster — “24/24 Kustomizations Ready at one revision, 12/12 Helm releases deployed, no unhealthy workload”. That requires cluster read access from outside the cluster.

The identity is shaped by one rule: an agent that can read Secrets or open a shell in a pod no longer needs the pull-request path. paperclip-observer is therefore a strictly read-only credential: get/list/watch on the built-in view surface, with no secrets resource in any API group and no pods/exec|attach|portforward. It cannot read the very Secret it authenticates with, and it cannot harvest another workload’s credential.

Owner: tazlab-k8s, Kustomization infrastructure-agents-paperclip (project paperclip-dedicated-container, task T4).

Resources

FileContent
infrastructure/agents/paperclip/kustomization.yamlresources: [rbac.yaml, token.yaml]
infrastructure/agents/paperclip/rbac.yamlServiceAccount + ClusterRole + ClusterRoleBinding paperclip-observer
infrastructure/agents/paperclip/token.yamlSecret paperclip-observer-token + the operator step that copies its token into gopass
clusters/tazlab-k8s/infrastructure-agents-paperclip.yamlFlux Kustomization: interval: 15m, path: ./infrastructure/agents/paperclip, prune: true, wait: true, retryInterval: 1m, timeout: 5m, no dependsOn — it is a leaf, and it must be able to apply while the rest of the DAG is stuck (TD-096 is exactly that failure)

Live (verified 2026-09-28): ClusterRole and ClusterRoleBinding present, created 2026-09-26T14:21:35Z, labelled kustomize.toolkit.fluxcd.io/name: infrastructure-agents-paperclip; ServiceAccount paperclip-observer in flux-system; Secret paperclip-observer-token bound via kubernetes.io/service-account.name.

The ServiceAccount lives in flux-system

Not in an application namespace, and it runs no pod: it is a credential container, deliberately placed beside flux-dag-watchdog, which is the other identity that reads the cluster from outside an application. The Secret carries no .secrets reference on the ServiceAccount — a reference would make the API server classify the token as auto-generated and attach a deprecation warning to every request (kubernetes.io/legacy-token-last-used still appears and changes on use; that is the platform, not drift).

What the ClusterRole grants

paperclip-observer is the literal union of the built-in view role and Flux’s flux-view-flux-system, transcribed rule by rule rather than composed, and every rule uses verbs: ["get", "list", "watch"] — no other verb appears in the file.

GroupRead surface
core ""pods (+/log, /status), services (+/status), endpoints, persistentvolumeclaims (+/status), namespaces (+/status), serviceaccounts, replicationcontrollers (+/scale, /status), limitranges, resourcequotas (+/status), bindings, events, nodes (+/status)
apps, batch, autoscaling, policy, networking.k8s.io, discovery.k8s.io, extensions, resource.k8s.iodeployments/daemonsets/replicasets/statefulsets (+scale, +status), jobs/cronjobs, horizontalpodautoscalers, poddisruptionbudgets, ingresses, networkpolicies, endpointslices, resourceclaims
Fluxhelm.toolkit.fluxcd.io, image.toolkit.fluxcd.io, kustomize.toolkit.fluxcd.io, notification.toolkit.fluxcd.io, source.toolkit.fluxcd.io, source.extensions.fluxcd.io (resources: ["*"])
add-onscert-manager.io certificates/certificaterequests/issuers, acme.cert-manager.io challenges/orders, external-secrets.io externalsecrets/secretstores/clustersecretstores/pushsecrets, generators.external-secrets.io, metrics.k8s.io nodes/pods

nodes was added on purpose (commit a7d5955, minutes after the first revision): a health report needs node conditions, versions and taints, and metrics.k8s.io does not provide them. The first revision of the file explicitly withheld it — the fix is recorded because the narrower alternative (a labelSelector) was considered and rejected as more machinery than the reading requires.

Explicitly absent: any secrets resource; any verb beyond get/list/watch; pods/exec|attach|portforward|proxy; read on Roles/RoleBindings.

How the token reaches the container

  1. Operator step (documented in token.yaml, cannot be automated from Git): the Secret’s token is copied into the operator store — kubectl -n flux-system get secret paperclip-observer-token -o jsonpath='{.data.token}' | base64 -d | gopass insert -f -m paperclip/k8s-token.
  2. The container’s layer (tazpod/runtimes/lxc-paperclip, layer3-agents + templates/kubeconfig.j2) reads agent/k8s/token as the service user and renders /workspace/home/.kube/config (0600, context paperclip-observer, namespace flux-system). Server address and certificate-authority-data come from the orchestration host’s kubeconfig — the cluster-admin credential in that file is never read; only the two connection fields are.
  3. Missing token ⇒ the kubeconfig is simply not rendered, with a debug message: the container still works, it just cannot see the cluster.

Revocation: label the Secret kubernetes.io/legacy-token-invalid-since, or delete it and re-run the gopass step; the API server validates a legacy token against the live Secret on every request, so revocation is immediate.

Verification

# the objects exist and are Flux-owned
kubectl get clusterrole,clusterrolebinding paperclip-observer -o yaml | head -40
kubectl get sa,secret -n flux-system paperclip-observer paperclip-observer-token

# from the container's kubeconfig, the read surface and its limits
kubectl --kubeconfig /workspace/home/.kube/config auth can-i list pods -A          # yes
kubectl --kubeconfig /workspace/home/.kube/config auth can-i get secrets -A        # no
kubectl --kubeconfig /workspace/home/.kube/config auth can-i create pods/exec -A   # no

Known limits

  • The kubeconfig is a long-lived credential by construction (a legacy service-account token, no expiry); the compensating control is its shape — read-only, no Secrets, no exec — plus the explicit revocation path above.
  • Read-only is the design, not a restriction to be lifted on demand. An agent that needs to change the cluster opens an issue; it does not get a write verb.
  • The identity is cluster-scoped because the reports are cluster-scoped. A namespaced variant was not chosen: the Flux objects it must read live in flux-system, the workloads it reports on do not.

See Also