Runners: what happens during a run

Bounded runs per seat, enforced by the database

Capacity on a machine slot is a partial unique index over the machine, the slot and a session ordinal, for live, unfinished runs. Reservation enumerates the ordinals 0 .. max_sessions-1 and inserts at the first one that is free; each attempt is an insert whose conflict clause matches that index predicate, so a losing concurrent attempt affects zero rows. Capacity N is therefore N distinct index keys — there is no counter and no count is ever read. Exhausting the ordinals IS the refusal, and the refusal names both numbers: how many runs the slot carries and how many it may. A slot with no declared max_sessions has exactly one key and behaves precisely as it did before this existed. This replaced a read-then-act check with a real race, measured live: two dispatches could each read "clear" and each proceed.

Conflict handling is bounded and never waits indefinitely: read the blocking run, test whether its process is alive, finalise genuinely dead holders individually by run identifier, retry once, otherwise report the current holder.

Sandboxing: the question this section answers

An agent runs as a process on some machine. The useful question is never "was a sandbox option switched on" — an option is a claim, and we have measured agent-tool sandbox options that, as configured, did not confine. The question is: which mechanism, enforced by the operating system, refuses a write outside a named set of directories — and does the run record say what enforced, so you can reproduce the claim?

One sentence of scope, and it governs every table below: every boundary described here governs filesystem writes. None of them restricts what an agent reads (with one narrow, named exception for credential files, below). ⚠ And none of them restricts network egress for a run that does not declare an egress policy — which is every run in this fleet today, and is what the tables below describe. A run started with saphan run --egress-policy is a different shape: the local arms additionally compose a network cut around it and give it a per-run door, and a host that cannot be cut refuses the run rather than starting it open (Egress §8, and section 11 of this document). ⛔ That is an opt-in, never a default, so read every "network egress is not restricted" in the tables below as scoped to the silent run it describes.

Two rules carry through everything here:

• The record names what enforced, not what was requested. A run record does not quote a sandbox option; it names the tool that enforced and the directories left writable — a claim you can reproduce. This exists because we found the opposite in our own earlier work: a record quoting a vendor's sandbox flag as though it were a guarantee. We measured two such flags on two different agent tools and neither confined — one wrote to the system temp directory, the other to the home directory.
• When a boundary cannot be enforced, the run does not happen. A machine that cannot confine is a refusal with a named reason, never a warning followed by an unconfined run. The prevailing industry default is the opposite; we inverted it deliberately.

Container eligibility is a separate routing precondition, not a confinement claim. When a cast names an image digest, the machine is eligible only if the signed record carries a probe-confirmed docker-oci row for that exact digest and the row is fresh when casting reads it. A stale row counts as absent; a row for another digest does not qualify. The same successful probe is the only writer of the machine's runtime, vm_cpus and vm_memory_bytes facts. This gate proves that the machine ran the pinned image and produced the required outcome. It does not mean that the subsequent run executes inside a container; the current container boundary remains the documented, unenforced direction in section 11.

Who confines what

Rules that hold on every row:

• Fail closed. A machine that cannot enforce — no sandbox utility present, no Landlock, an unrecognised operating system — is a named refusal, never an unconfined run.
• The owner can hand confinement to the tool, on the record. A single installation setting (run.confinement, a closed two-value set: engine or vendor) may hand confinement to the agent tool instead of the engine; when set, the run record states in as many words that the engine makes no claim the agent is confined. Any other value is refused before any other branch, so a configuration typo surfaces on the next run of any shape.
• Bypass options cannot be requested. The engine's internal instruction format has no field that could ask an agent to disable its own safety mechanisms, and a second check refuses a closed list of known bypass tokens on the final command line, whichever tool they belong to. The same check refuses silent model substitution and silent session resumption.

The engine's own boundary (control-plane machines)

On a machine the control plane runs on directly, the engine composes the boundary itself, with one policy expressed two ways — so the two platforms cannot drift apart:

The write policy is deny-by-default with an explicit allow list. The permitted set:

Properties worth calling out, because each is unusual:

• Grants are derived, not configured. The engine reads the working directory to determine which version-control paths a run needs. Nothing is declared by an operator, so nothing can be mis-declared. Where a narrow grant cannot be expressed, the engine grants nothing and says so. Concretely: on Linux, whose bind-mount model cannot grant a not-yet-existing lock file without its parent directory, a run on a non-namespaced branch therefore cannot commit. That is an accepted inconvenience, chosen over the wider grant.
• Symbolic links are resolved before the rules are built. On macOS the system temp and variable directories are links into a private tree; a rule naming the link would grant nothing for the file the agent actually opens.
• Read denials, where they exist, are applied last. On Linux a later grant overrides an earlier denial at the same path, so a read denial has to be the last word about any path — measured, not assumed — or a subsequent grant silently defeats it.
• The engine does not nest inside a tool's own sandbox. Where an agent tool applies its own operating-system sandbox, wrapping it in the engine's boundary fails at the kernel — which is why one backend is confined by itself, not by us (below).

One residual is accepted and restated in section 11: the shared version-control object store is writable inside the boundary, because neither sandbox policy language can express "append only."

What the boundary refuses that a task may still need: measuring the base

The version-control grant above is per-worktree and by-name. A new worktree is therefore not creatable from inside it: git worktree add writes its bookkeeping into the main repository's metadata directory, which is a different path than the one the run was granted, so the boundary refuses it:

fatal: could not create directory of '<repo>/.git/worktrees/<name>': Operation not permitted

This is the boundary working, not a defect. But it makes one commonly-ordered measurement — "check out the base commit somewhere clean and prove this failure pre-exists your change" — impossible as usually spelled, and a step nobody can perform is a step whose result is whatever the reporter felt like writing. The same measurement has a reachable instrument.

Runbook: measure a named base commit from inside the boundary. Every path below is inside the working directory, which is writable; nothing touches the main repository.

1. Clone the repository into a scratch path within the working directory, sharing the object store rather than copying it, and without a checkout:

git clone --shared --no-checkout <repo> <workdir>/<scratch>

Pick a <scratch> the repository already ignores, or the clone shows up as the task's own uncommitted work at the return gate. Ignored is not invisible: a test that walks the repository tree — a source-hygiene guard, an import census — reads the scratch clone's files as if they were the working tree's own, and reports the base's contents as findings against your change. Measured: a tree-walking guard named three production call sites that existed only inside the scratch clone.

2. Detach the scratch clone at the exact commit you were told to measure:

git -C <workdir>/<scratch> checkout --detach <sha>

3. Verify the SHA, and report the verification, not the intention. The command above can resolve a branch, a tag or an abbreviation to something other than what the order named:

git -C <workdir>/<scratch> rev-parse HEAD

A measurement at an unverified commit is not a measurement of the base. If step 3 does not print the SHA the order named, the correct report is that the base was not measured — never a per-test verdict obtained somewhere else.

4. Run the measurement from inside the scratch clone, and report it per item — one named test, one named outcome. A total ("19 failures at base too") is not evidence about any individual failure.

5. Remove the scratch clone, then re-run the working tree's own suite. Steps 1-4 add a second copy of the repository underneath the first; leaving it there means every later tree-walking measurement is taken over two trees at once.

--shared is what keeps this cheap: the scratch clone borrows the existing object store instead of duplicating it, so the cost is a checkout and not a copy of the repository's history.

The engine's boundary on remote runners

For runs dispatched to a machine over SSH, the boundary is Landlock — a Linux kernel facility that needs no third-party software installed. It is established in three parts:

• The capability is measured, per machine. saphan machine probe-confinement connects to a machine over its own recorded transport and records the Landlock level it measured onto that machine's signed row. There is no option that accepts a hand-typed value, no override, and no force flag — by design. The value is written by the probe alone. Machines genuinely differ — some kernels offer a high level, some offer none (a container can offer none) — which is why the value is recorded per machine and read before any remote run is composed. A claimed level above the highest the engine knows is not believed.
• The boundary is the engine's own program, re-entered on the runner. The engine ships to the runner for everything else already; on the runner it applies a Landlock domain to itself and then starts the agent inside it. The policy travels as a single readable value — writable: the far working directory; readable: everything (an agent that cannot read the system libraries cannot run anything) — and that value also lands in the run record, so it can be audited later.
• The decision is a gate, and its only other outcome is a refusal. A remote run of an engine-confined tool is composed through that boundary only when the target machine's row asserts a Landlock level at or above the floor the engine will enforce. Anything else — a machine never probed, a machine measured incapable, a machine below the floor — is a named refusal carrying the row-computed remediation, and nothing runs unconfined on someone else's machine.

Why there is a floor. Below Landlock's third level the kernel does not cover file truncation, so a "confined" agent could still destroy the contents of any file it can open. A machine below the floor is refused by name — with its measured level in the message — never quietly confined at a weaker level. And the boundary re-checks the floor against the live kernel as it is applied, so a stale or hand-edited row becomes a failed run rather than a weaker boundary.

The agent's scratch space lives inside the jail. The agent tool's temporary directory is pointed at a scratch directory inside the far working directory and created after entering it — chosen over widening the policy toward the system temp directory. (The create is deliberately relative and performed after the change of directory, so a vanished working directory stays a loud failure instead of being silently recreated.)

What the remote boundary does not give, stated as plainly as what it does:

• Reads are not restricted. The policy grants read and execute everywhere, and a read grant cannot be carved out underneath. Anything readable by the runner's user is readable by the agent, including other accounts' configuration on that machine.
• Network egress is not restricted for a run that declares no egress policy — the same as every other boundary here (section 11). ⚠ On this lane a run that DOES declare one is answered per far host and never fleet-wide: it is governed where the host's measured rows read seatbelt and seatbelt-network-deny, and refused by name on a Landlock host, because this build emits no network lever there (Egress §9B.3). A far Linux runner therefore never runs a policy-carrying spawn ungoverned; it declines to run it.
• Metadata changes are outside Landlock at every level. Changing a file's permissions, owner, timestamps or extended attributes is not covered; the boundary means "cannot write file contents outside these directories," not "cannot modify anything."
• The record's ENFORCED claim is composed, not observed. The note is derived from the signed capability row and the policy at composition time; the far side also prints the kernel's own confirmation line into the captured output, but no verb currently asserts it. And the far copy of the engine that applies the boundary is resolved by name on the runner — it is not digest-pinned, unlike the agent binary it jails. Both facts are restated in section 11.
• The session file directory is granted here too, and it was not always. (Amended 2026-08-28. This bullet used to say the grant had no remote equivalent and that a remote run therefore could not be resumed. That was true, and it was the reason no far run in this fleet could ever be woken.) The obstacle was real: the directory belongs to the far seat's own configuration home, whose $HOME this engine cannot expand for another machine, and a path composed from a local resolution would name a directory that does not exist where the boundary is enforced. It is resolved by asking the machine that owns the filesystem: the same preflight dial that proves the far working directory exists now also expands that seat's configuration home in the far shell, derives this run's own project key from the working directory the far side resolved, creates that one directory there, and reports it back. The reported path is not taken on trust — the engine re-derives the key itself and refuses anything that is not this run's own leaf under a projects parent, so the configuration home and the directory above the leaf stay unwritable exactly as they do on a control-plane machine.

Two consequences worth stating plainly. The policy gained no new field, deliberately: a remote host running an older copy of the engine refuses by name any policy carrying a key it does not know, so the grant travels as one more directory in the writable list that every version already understands — no coordinated upgrade of remote machines is required, and this was verified against a copy of the engine built before the change. And a run whose dial could not resolve that directory is not refused — it does the whole job and simply cannot be resumed afterwards, which is a trade in the operator's favour and would otherwise be an invisible one, so the confinement note on that run says so in those words instead of describing a session that does not exist.

When the agent tool confines itself

One supported agent tool applies its own operating-system sandbox to every command it runs. Nesting the engine's boundary around it fails outright at the kernel, and the option that would disable the tool's own sandbox is one the engine forbids on principle. So for this tool the tool's own sandbox is the boundary, and the engine's job is to compose it as tightly as the tool's configuration allows and to record its limits honestly.

What the tool's sandbox is, and is not. It uses the platform's native facilities — Seatbelt on macOS, Landlock and a network filter on Linux, restricted tokens on Windows. Its default working mode reads files anywhere and writes only within the workspace, with the version-control directory and the tool's own configuration directory held read-only inside it. Two facts about its scope matter more than any option:

• It confines the commands the model runs — not everything. The tool's own process, any helper services it starts, and any configured hooks run outside this sandbox. A boundary on the commands the agent shells out to is not a boundary on the agent.
• It does not restrict reads, in any mode. Secrets outside the working directory are readable whenever the tool runs.

A measured escape that was a default, not a defect. We measured this tool writing to the system temp directory under its standard working mode and recorded it, at the time, as "not confined." On closer reading it was neither a bug nor an escape: the tool's working mode includes the system temp directory and the per-user temp directory in its writable set by default. The tool documents two settings that remove them, and the engine now sets both for every execute-profile run. Separately, the tool holds the working directory's own version-control directory read-only by default — which prevented commits until the engine named it a writable location, with the sandbox still on.

What the engine composes for this tool, on the command line and validated by the tool's own strict-configuration mode (an unrecognised setting is a hard error, so a mistyped setting can never quietly do nothing):

The last row is the reason all of these are set per run on the command line rather than left to a configuration file. The tool reads layered configuration, and any setting the engine does not pin takes whatever the machine's own files say — so a careless or altered configuration file could widen the writable set or re-enable the network, and the run record would never know. A command-line setting outranks every file, so the record's claim stays true regardless of what a runner's files accumulate.

None of this widens the boundary. What remains outside it is exactly what the scope paragraph above names: the tool's own process, its helper services and its hooks; all reads; and, on a kernel without Landlock, behaviour the tool does not document — which is one more reason the engine records a measured capability per machine rather than assuming one.

Permission modes are not boundaries

The other supported agent tool offers a permission mode — plan, or accept-edits. This is a policy the tool applies to itself before it acts; it is not an operating-system boundary, and we measured a run under accept-edits writing to the home directory. That is exactly why this tool is always confined by the engine: its permission mode composes cleanly inside our boundary because it is not itself an OS sandbox, and the engine's boundary is what enforces.

This tool also ships a native sandboxing feature of its own, including an outbound-network filter that restricts which domains a run may reach. Three of its defaults make it unsuited as the primary boundary for unattended fleet runs, and each is the inverse of a rule this engine holds: by default a missing sandbox dependency lets the run continue unconfined (the engine refuses); a denied command can be retried outside the sandbox unless that path is switched off (the engine forbids bypass); and its command-exclusion list has no administrator lock (the engine never lets a boundary be widened by anything the agent can write). So the engine's boundary remains the boundary here, and the tool's permission mode is policy.

Its network filter is nonetheless the one control that could close the egress gap on this tool's lanes. Delivering it as a locked administrator policy — filesystem isolation off (the engine already enforces that), a strict domain allow-list, no unconfined-command retry, and hard-fail when unavailable — would add an outbound allow-list on top of the engine's write boundary. It is a candidate, not a claim: it will be documented here as an enforced control only once it enforces on this fleet, which is the same standard every other control in this document is held to.

The engine's own agent loop is jailed too — and blinded to your credentials

The engine's built-in agent loop (the backend that drives local model lanes) does not ask for trust it does not extend to others: before doing any work it re-executes itself through the same confinement composer that jails agent tools — one composer, two callers, so the policies cannot drift apart. Its writable set is the working directory and its own return path. Beyond the write boundary, this is the one place a read denial is added:

• Every declared lane credential file — and the configuration file that names them — is denied for reading inside the jail. Not just the active lane's: all of them.
• The lane is therefore resolved in the outer, unconfined process, and the secret value crosses into the jailed child over a private file descriptor, never the environment — on Linux, /proc is mounted inside the jail and no PID namespace is created, so a same-user sibling could read a child's environment; a pipe it cannot.
• Every subprocess the loop's tool-calls spawn gets an environment scrubbed by name of the credential variable — deliberately catching the case where the operator's own shell already exported the key.
• These are pinned by live tests: a jailed child cannot read the credential by any measured route (direct read, search, an authored script, a hard link), and the credential never rides an environment.

One measured interaction between the jail and the vendor's credential store is worth a paragraph, because it produced a rule: on macOS, the system's security daemon refuses a keychain write from inside the sandbox. An agent tool that tries to refresh its own authentication token mid-run inside the jail silently loses the refresh — and the next spawn hours later fails authentication while a status read still looks healthy. The fix was not to widen the jail: the engine refreshes the seat's token before the confined child starts, from its own unconfined process, as a pre-flight (the same pre-flight that refuses a spawn onto a seat whose session is already dead — R23). The jail stayed shut.

Refusal rather than degradation

Each carries a named class. There is no exemption path, no override flag, and no "best-effort" mode.

Confinement: operator quick reference

What an agent is permitted to do

Permission classes are declarative JSON policy files with a deliberately narrow surface: allowed commands, write scope, network posture. Accepted schema versions are an explicit set; a file naming any other version fails to load rather than being interpreted charitably, and a file that claims one version while using another's features is rejected, because accepting it would change what that version means between releases. Command matching is exact-token on the base name — never prefix or substring. Where a credential is required, the policy names the variable and the engine mints the value at launch; values do not live in policy files. Where a path is required it must be absolute, mirroring the rule that PATH resolution is forbidden.

Bypass flags are unrepresentable. The engine's internal instruction format has no field that could ask an agent to disable its own safety mechanisms. As defence in depth, a second check inspects every assembled command immediately before launch and refuses nine known tokens, matching exact arguments and the --flag=value form only. The same check refuses silent model substitution and silent session resumption: nobody gets a model other than the one they named, and no run silently continues another session.

Software supply chain

Releases are a single static binary, built with path trimming for reproducibility and obfuscated on the release path, shipped in a minimal container image.

Delivery to a runner computes SHA-256 on the receiving machine using the platform's own digest utility, reading the file through redirection so a filename can never be confused with a digest, under a shell mode that aborts on any failing step. After installation the engine re-resolves the binary's identity — digest and version — on the machine that will execute it, because the point of the pin is to record what actually ran, and refuses on mismatch.

The control-plane binary checks its own custody before it runs

The workspace's key files are custody-checked — owner-only permissions, symlink-safe, exact length — before their contents are read. The same discipline now covers the binary itself: saphan measures the custody of the executable it is running and refuses to start if that binary is replaceable by an identity that is not its custodian. The refusal happens at the process entry, ahead of flag parsing, workspace resolution and every verb, and carries the refusal class custody-binary-unsafe.

The mode bits are not the custody — the owner and group are. A 0550 binary owned by the agent account is worse than a 0755 binary owned by root, so the run condition is four clauses about identity, not a comparison against a mode:

0550 is a reference value, not the run condition — and not what any installer here writes. Measured across every delivery route: the bootstrap script sets chmod 0755 on the binary and on the install directory, the macOS installer installs at mode 0755, and the deb and rpm declare 0755. None writes 0550. Earlier versions of this section called 0550 "the installer's default"; that was wrong, and the install-mode census in package repohygiene now asserts 0755 on the installed binary itself, so the sentence cannot drift back without a test turning red. A hard 0550 check would refuse every container, where the binary is normally root-owned at 0755 and the process runs as non-root. That shape passes, deliberately: o+rx is what a container executes through, and tightening it is the installer's job, not this check's refusal. So does the ordinary local install ($HOME/.saphan/bin/saphan, owned by you), an image that runs as an arbitrary uid with gid 0, and a root-owned 0755 binary carrying any non-zero group — root:staff, root:nogroup, COPY --chown=root:appgroup, or a Kubernetes fsGroup injected into the process's group set. No group id is special here, including 0. Clause 3 asks whether the group is what confers the reach, and at 0755 it is not: others already have execute, so the group grants nothing extra and there is nothing to refuse.

What this refuses, stated as the general rule rather than as one container anecdote. The honest one-line form is: a non-owner may execute the control plane only through others' execute bit. The check measures the binary the calling process is already running, so for a file with g+x and no o+x — 0550 and its neighbours — the only callers that can exist are the owner, which clause 3 admits, and a group member, which clause 3 refuses. A non-owner outside the group holds no execute bit and never reaches the check at all. ⇒ 0550 is owner-only. Root-owned, it is root-only. The OpenShift case is one instance of that rule: 0550 root:root executed by a non-root process holding group 0 — the arbitrary-uid convention taken as far as removing others' execute — is refused, because there the group really is the caller's only path to the control plane, which is the reach the invariant exists to deny.

⚠ An earlier version of this section ended with tightening advice that cannot be followed — "if you need the tighter mode, the accounts that run the binary must not hold its group." Do not follow it: an account that neither owns a 0550 binary nor holds its group has no execute bit, so it gets Permission denied from the kernel rather than a refusal from this check, and the printed cure produces a binary that does not run. The shape this project documents and mandates for containers is 0755, which passes. If you want 0550, run the binary as its owner — that is the whole set of identities the mode admits.

⚠ The class of defence this is, stated plainly. This is a barrier against misconfiguration and accidental use — someone installing a full saphan at 0755 somewhere an agent account can reach it. It is not a barrier against an attacker who can already write to the binary: that attacker edits the check out. The check runs inside the process it protects, which is exactly why it can only be the weaker of the two. We do not present it as more than that.

What it does not assert, printed with every refusal rather than left for you to discover: which identity is the agent (there is no agent-UID fact in the record yet — that lands with the UID-separation work); the ancestor chain above the immediate parent directory; ACLs and extended attributes, which can grant write beyond the mode bits a stat can see.

The remediation is not left to you to derive: see §12, When saphan refuses to start.