Administrator's runbook

Prerequisites

Enrollment procedure

Worked commands for every step are in section 13.

1. On the new machine, generate the device key and produce a certificate signing request. Copy only the request to the control plane; the private key never moves.
2. On the control plane, admit the machine. Verify the reported Common Name matches the machine identifier you intend — this binding is what makes the identity meaningful.
3. Set the machine's slot count to the number of distinct executor identities you intend to seat there — one slot per account, not per concurrent run. Then set how many concurrent runs each slot may carry with --max-sessions (default 1). Do not inflate the slot count to buy concurrency: a slot is an identity's seat, and a machine with more slots than accounts advertises seats that nothing can fill.
4. Create one seat per executor identity you intend to run there, naming the agent backend. A machine with zero seats accepts no cast; a machine with two makes casts explicit. If you intend to run several accounts on one machine — separate subscriptions, a personal and a team account, one per client — give each its own seat. Authenticate each seat once, in its own profile home; do not authenticate an agent tool on a machine outside a seat, because that writes into the shared profile the seats exist to keep you out of. Also name the seat's billing class with --billing-class subscription, --billing-class metered, or --billing-class local-energy (for a seat whose model endpoint you run yourself, with no vendor account to meter), matching the vendor account you are about to authenticate into that profile home. This is a deliberate, separate decision from naming the backend: it is the fact that governs whether anything is later allowed to treat this seat as subscription-covered. If you omit it, the seat reads back as unknown — not as a problem, but as a fact you have not yet stated, and anything that only proceeds on an explicit subscription will treat it exactly as it would treat a metered seat. See 13.8 for the worked command.
5. Run capability probes. Until a capability is probe-confirmed, that machine will not receive the corresponding work — this is expected, not a fault. For a machine that will take engine-confined work over SSH, also run saphan machine probe-confinement — the measured confinement level is recorded on the machine's signed row, and it is the gate a remote confined dispatch reads (see 13.9).
6. Confirm the generated SSH configuration resolves as intended, and that the host key pin was established by admission rather than by a first-contact accept.

When the docker-oci probe says this machine cannot

Use this procedure on the runner intended to take container work. The commands below use the admitted machine id m2; substitute your runner's admitted machine id for m2 when it differs.

1. Confirm that the declared runtime is installed at /usr/bin/podman and that the pinned probe image is already in its local image store. The probe uses the local image only: it does not install the runtime, authenticate for you or pull an absent image.
2. Measure the outcome and let the probe write the record:

saphan machine probe --machine m2 --capability docker-oci \
--declared-path /usr/bin/podman --workhome /home/builder/work \
--probed-by m0 --workspace /Users/Sources/saphan/saphan-studio

3. Read the printed outcome and reason; do not reduce every failure to "containers do not work." A PATH or authentication reason says that the runtime CLI could not be resolved or authenticated, not that the machine is incapable. An absent local image or a runtime that starts no container reports that the container started nothing. If the container starts but the required denied/read markers are missing, repair the nested isolation outcome the probe names. Correct the runtime, local image, authentication or runtime policy named by that reason, then repeat step 2.
4. Do not hand-edit runtime, vm_cpus, vm_memory_bytes, the image digest or the probe lease. No verb, flag or configuration key accepts those facts by hand. Until the probe passes, leave this runner out of container routing or use another measured runner. A cast that reaches this state refuses as RS4A-NO-CAPABLE and prints its own complete probe command for the real machine and workspace.
5. After the probe passes, audit the run record before routing more container work:

saphan checks container-contradictions \
--workspace /Users/Sources/saphan/saphan-studio

checks container-contradictions: ok means every recorded container run had a fresh row for its exact machine and image digest at that run's start time. Any container-capability-contradiction is an audit failure; a later successful probe does not retroactively authorize the earlier run.

Settings with security meaning

Unknown configuration keys are refused against an explicit accepted vocabulary, which is listed in the error. There is no permissive fallback: a misspelled setting that silently does nothing is a defect class we decline to ship.

First mint: giving a workspace its signing key

Every owner-class decision — saphan gate, saphan merge — is signed by the workspace key, a 32-byte HMAC secret at <workspace>/.saphan/secret. A session that has none is refused (owner-key-absent) with nothing written. A new workspace has none, so the first act in it is to mint one:

1. Check what is there. saphan actor list --workspace <path> reports the key-id if a key exists, or workspace key — ABSENT if it does not. This verb is a read: it never mints. A read that minted would hand every keyless session a fresh self-authorising key as the price of looking, which is precisely the defect this separation exists to prevent.
2. Mint the key. saphan actor init --workspace <path>. It writes the secret at mode 0600, writes a .gitignore cover beside it so the value can never be committed, and prints the derived key-id (a wsk- fingerprint). The value itself is never printed, logged, or returned on any surface.
3. Enrol the key-id. Minting is not enrolment, and the mint verb says so in its own output. Until an actor row names this identity, the gate still refuses — under a different class, naming the roster rather than the key:

saphan identity enroll --kind actor --subject <name> --key-id <the wsk- id>

That is an owner-root ceremony: run it on the machine holding ~/.saphan/trust/root.key. 4. Confirm. saphan actor list now shows the key-id, and marks the roster row that matches it with *.

actor init refuses over an existing secret (workspace-key-exists) and there is no force flag, for the same reason root creation has none: a re-mint derives a new key-id, and every decision this workspace has already signed would verify against a key that no longer exists. The refusal fires on any existing secret file — including a mis-permissioned or a malformed one — because it asks whether the file is there, never whether it is good. Repairing a broken secret is therefore a deliberate act:

Reading the roster gap before a gate decision is refused

Actor status is enforced on every gate decision: the name recorded as having decided must be enrolled and active in the workspace's signed actor registry, or the write is refused. That check is unconditional and has no off switch — there is no warning tier and no grace period.

The consequence worth knowing in advance: a workspace's gate log may already carry decisions under names that are not in the registry — an operator who used to decide under a different name, a service identity retired long ago, a name that predates the registry entirely. Those recorded decisions are untouched and stay valid. What changes is that those names cannot write a new decision.

saphan actor list reports this as the roster gap: every actor name the gate log records, how many rows each has written, when each was last seen, and what the guard would say about it today. Run it before you rely on a name.

1. Read the gap. saphan actor list --workspace <path> (add --json for a machine-readable report under the gate_log_gap key).
2. Check the section's status word first. read means the gate log was measured and the list is authoritative. gate-log-absent means this workspace has never been written to. gate-log-unreadable means the log exists and could not be tallied — this is not an empty gap, and no conclusion about who would be refused may be drawn from it. Repair the store, then re-read.
3. Read the marked lines. Each name marked ! would be refused on its next gate write. The word in parentheses is the reason: not-enrolled, retired, non-human (engine and saphan are refused whoever is enrolled — enrolling them is not the remedy), registry-absent (this workspace has no registry at all, which is not consent), no-actor (an empty actor column on a historical row; the writers refuse it today), unencodable, registry-unwired (an engine wiring fault, not a workspace one — no composition root injected a store opener, so this binary cannot say who may sign; do not enrol anyone over it), registry-unreadable (the registry exists and its own chain refused — a missing owner root pin, a row that fails verification, or a sequence gap; not an empty registry), or unknown (the binary and the store disagree about schema version, so the question was never asked — treat it as unmeasured, never as a pass).
4. Decide per name, not in bulk. A name that has stopped deciding needs no registry row — leaving it in the gap is a legitimate outcome, and the recorded rows it already wrote keep their signatures either way. A name that must keep deciding is enrolled by an owner ceremony, on the machine holding the root key (section 4, Actors and delegation): saphan identity enroll --workspace <path> --kind actor --subject "<name>" --key-id <id>. The refusal an unenrolled name meets at the gate prints this same command with the name filled in.
5. Re-read to confirm. The same command; the name you enrolled should now read ok and lose its !.

actor list writes nothing, refuses nothing, and changes no gate's outcome. It reports; the decision is yours.

The actor column is a registry handle

The actor column of a gate row holds a resolved canonical registry name — the spelling the signed actor registry holds, byte for byte. The key-id lives in the signature, never in the column: a key-id does not survive rotation, and a column of hex would destroy the readability saphan gates exists to provide. Name for policy, key for crypto, bound to each other.

Two consequences you will meet at the terminal.

A near miss is refused, and the refusal tells you what to type. A name one fold from an enrolled active actor — differing only in case or whitespace, such as Marcin marzec, Marcin Marzec or Marcin Marzec — earns the actor-near-miss class, exit 2, and nothing is written. The refusal prints the canonical spelling in quotes (so a trailing space is visible) and the exact --actor argument to re-run with.

The engine will not correct it for you, and that is deliberate. The actor column is signed content. A silent normalisation would write a name you never typed into bytes your key then attests to — and it would be right almost every time, which is what makes it dangerous rather than what makes it safe. actor-near-miss is its own class precisely so you are not sent to the enrol ceremony: enrolling the misspelling would mint a second registry row for one person.

The fold is case and whitespace only; homoglyphs are out of scope and earn actor-not-enrolled. Mаrcin Marzec written with a Cyrillic а (U+0430), or Marcin Marzеc with a Cyrillic е (U+0435), is visually identical to the canonical spelling in every terminal — and it is not a near miss. It is an unknown name, and the refusal it earns invites you to run the enrol ceremony. Do not run it. Nothing was substituted and nothing was written, so no signature is at risk; what is wrong is only the advice, and following it would mint the second registry row the near-miss class exists to prevent. Check with saphan actor list and correct the name at the keyboard.

Why the fold is not widened to look-alike normalisation. The two mistakes are not equal in cost. A confusable fold that fires wrongly merges two identities inside signed content — silently, and irreversibly, because the distinct spellings were the only evidence the two were ever different people. The narrow fold's worst case is a misleading sentence in a refusal a human reads. The same boundary has the same effect for a retired actor: a name one fold from a retired row and nothing else is actor-not-enrolled too, so the remedy invites you to enrol someone this workspace retired on purpose. Both are remedy-text defects, not reasons to loosen the comparison.

Old spellings in the log are explained by signed alias records, never by rewriting history. Thousands of recorded rows bind their actor string inside the signed row, so no migration and no verb rewrites one. Instead an alias states that a historical spelling names the same person as a canonical actor, and saphan gates renders the stored string verbatim with a derived ≡ marker beside it.

Runbook — enrolling an alias (owner ceremony)

Run on the machine holding ~/.saphan/trust/root.key.

1. Confirm it is a HISTORICAL spelling, not a typo being made today. saphan actor list shows which names the gate log records and when each last wrote. A name still in active use should be corrected at the keyboard, not enrolled — see the near-miss refusal above.
2. Enrol it. saphan identity enroll --workspace <path> --kind alias --subject "<old spelling>" --canonical "<registry handle>" --reason "<why>" The subject is the spelling as it appears in the log; --canonical is the actor it resolves to. An alias that resolves to itself is refused by name.
3. Read it back. saphan gates --workspace <path> — rows at or below the gate-actor epoch now render by marcin (≡ Marcin Marzec). The line reporting what ≡ means rides every render.
4. Withdraw one if it was wrong. saphan identity revoke --workspace <path> --kind alias --subject "<old spelling>" — a new signed row, never a delete; the alias then reads as absent.

What an alias can and cannot do. It resolves names for aggregation and never a signature: the signature verdict never consults it, so enrolling or revoking one can never change whether a row verifies. And it is bounded below the gate-actor epoch — the mechanical boundary recorded once, at the migration that introduced this contract. At or below it an old spelling is history. Above it, an old spelling is a forgery, not an old spelling, and no alias applies. Because a row being written is always above the boundary, enrolling an alias can never make a name writable.

What to monitor

Verification you can perform yourself

• Verify the governance history against the root public key alone — admissions, delegations, seats, capability grants — without trusting the control plane that produced it.
• Walk a law chain: confirm a document's bytes hash to its ratified entry, in a manifest whose own signature validates against the root, at or above the serial you expect.
• Reproduce a confinement claim: a run record names the enforcing tool and the writable directories. Run the same tool with the same policy and attempt to write outside them.
• Re-resolve a binary's identity on a runner and compare with the run record; the record's value was resolved on that machine, not on the control plane.

Hardening we recommend, and do not perform for you

1. Restrict outbound network access per runner at your own network boundary. This is currently the highest-value control you can add and the one we do not yet provide. Deny by default; allow only the inference endpoint and the package sources you actually need; force DNS through a resolver you control with a name allowlist, since permitting general DNS permits exfiltration regardless of any address allowlist.
2. Restrict the SSH keys used to reach runners — pin them to the control plane's source address and to a single permitted command, and disable port, agent and terminal forwarding. Most SSH servers offer a single option that applies all current and future restrictions; prefer it, because it fails closed as the server gains capabilities.
3. Never enable SSH agent forwarding to a runner. A party able to reach the forwarded socket can authenticate as you elsewhere.
4. Keep nothing on a runner beyond that runner's own scope. Assume any secret readable on a runner is readable by every run on it. Personal machines used as runners deserve specific thought: what private keys and API tokens are in that home directory?
5. Keep delegation windows short, since expiry is the only revocation mechanism.
6. Switch instruction signature enforcement on once your signing practice is established. It defaults to off so adoption is measurable first — read the ledger, then flip.
7. Protect the control plane as your highest-value asset. It holds the means to reach every machine, and it is where instruction verification happens.
8. Store the root paper as you would a vault key. There is no root rotation: losing it means you can never sign a new delegation or admit a new machine under the existing chain, and re-founding the fleet's trust is the only recovery. Consider geographically separated copies with equal physical protection.

When saphan refuses to start: custody-binary-unsafe

saphan checks the custody of its own binary before it does anything (§8, The control-plane binary checks its own custody before it runs). If that check refuses, nothing ran: no verb, no workspace resolution, no write. The exit status is 2 and the refusal is printed on standard error.

The refusal is self-diagnosing — it prints the exact command for every clause it broke, and you do not have to derive anything. Run it in this order:

1. Read the whole refusal. It lists every broken clause, not the first one, so you fix the misconfiguration in one pass rather than re-running to discover the next problem. Each clause block carries a fix: line that is a literal, runnable command.
2. Check the two identity lines against each other — binary: … owner uid N group gid M and running as: euid K. If N is neither 0 nor K, the binary belongs to an account that is neither the system nor you: that is the misconfiguration this check exists to catch, and copying a chmod will not fix it. Decide which account should hold the control plane before you change anything.
3. Run the printed fix: command(s), in the order printed. They are chmod, chgrp and chown and nothing else. Whether a line carries sudo is computed for you, from who owns the thing being changed and who you are — not from the kind of command. A chmod on a root-owned binary you do not own carries sudo, because POSIX allows chmod only to the file's owner and to root; the same chmod on a binary you own carries none, and neither does any line printed to a caller who is already root, since a root shell in a container often has no sudo at all. Paste the line as printed. If a line still fails with Operation not permitted, that is worth reporting (§14) — the refusal is meant to name an act you can perform.
4. Re-run saphan version. A clean start is the confirmation; there is no separate verb to ask, because the check is the run condition.
5. If you meant to install for a single operator, the simplest correct shape is the one a per-user install produces: the binary owned by you at 0755 in a directory owned by you, with no write for group or others. The bootstrap sets that mode explicitly (chmod 0755, on the binary and on the install directory), so it does not depend on the umask you happened to run it under — an earlier version used chmod +x, which preserves the umask's write bits and produced a 0775 binary this check then refused. If you meant to install for a machine, the shape to use is the same 0755 owned by root in a root-owned directory — that is what every delivery route writes, and it is the shape the container arc runs. ⚠ 0550 is not "the installer's default" (nothing here writes it) and it is owner-only: a root-owned 0550 binary runs only as root, because a non-owner either reaches execute through the group — which clause 3 refuses — or has no execute bit at all. If you are holding a 0550 binary and it refuses, that is where you stand: run it as its owner, or set 0755. Keeping an account out of the binary's group does not make 0550 runnable for that account; it removes its last execute bit.
6. Do not "fix" it by making the binary or its directory more permissive. chmod 777 on either will silence one clause and trip another, which is the intended behaviour: the check is about who can replace the control plane, and widening permissions is the thing it refuses.

⚠ Read the limits of this check before you rely on it (§8). It stops misconfiguration and accidental use. It does not stop an attacker who can already write to the binary, and it does not see ACLs. If the refusal surprises you — if you did not change anything and it started refusing — treat it as a possible integrity event and go to Incident response below rather than reaching for chmod.

Incident response

• Suspected runner compromise. The machine holds no credential to the control plane, so containment is local. Revoke its reachability at your network boundary and rotate anything that was present on it. There is no fleet-level quarantine switch; removing the machine's reachability and letting its capability records go stale is the current path.
• Suspected control-plane compromise. Treat every dispatch since the suspected time as attacker-controlled, because runner-side verification does not exist. The governance log is still verifiable against the root, so what was signed remains provable even if what was dispatched does not.
• Suspected key exposure. For a delegate: retire the actor, which is checked independently of the grant and takes effect immediately. For the root: there is no rotation — contact us before acting, because re-founding the chain is an owner ceremony with consequences for every existing signature.