Updating a fleet that is already installed · Saphan Studio documentation

The same delivery, repeated — and the traps that make it look like it worked when it did not.

An update is the same delivery as the first install, repeated — there is no separate upgrade path and no in-place self-update. What changes is only that machines already have binaries, so every step below must be safe against a machine that is running work right now.

Step 1 — know which release you are delivering.

Name the source deliberately: the public line, a release channel (--channel stable), or a distribution tree you hold (--from <path or URL>). ⚠ A source that was not refreshed delivers the version it last held, with no error anywhere — so the first question of an update is which release the source is actually serving, and Verifying an install is how you answer it by content rather than by a version string.

Step 2 — deliver, one of two ways.

With the installer — the same line as the first install, run on each machine:

curl -fsSL https://saphan.ai/install.sh | sh -s -- --no-serve

It re-detects OS and architecture, downloads the whole machine payload — {saphan,saphan-agent,saphan-gateway,saphan-server} — re-verifies it against SHA256SUMS, and refuses on a mismatch with nothing installed. When the source publishes a version manifest, it also refuses to install an older version over a newer one unless you pass --allow-downgrade. Prefer this route: it is the only one that delivers every binary and verifies what it delivered.

⚠ --no-serve is what makes it an update. Run by a person at a terminal without it, the installer takes the first-install path: it refuses while a Saphan Studio process of the same installation is running, and otherwise sets the workspace up and starts the console again. With --no-serve it replaces the binaries and starts nothing.

Over ssh, without the installer, when you just want the new binaries on hosts you already reach, from a distribution tree laid out as <os>-<arch>/<binary>:

for b in saphan saphan-agent saphan-gateway saphan-server; do \
  scp "<dist>/<os>-<arch>/$b" "<host>:/tmp/.$b.new"; \
done
ssh <host> 'for b in saphan saphan-agent saphan-gateway saphan-server; do \
  chmod 0755 "/tmp/.$b.new" && mv -f "/tmp/.$b.new" "$HOME/.saphan/bin/$b"; \
done'

⚠ Ship the WHOLE payload the installer would have. A machine left with a new saphan and a stale saphan-agent reports the mismatch nowhere.

⚠ This route verifies nothing. Unlike the installer, scp checks no checksum; the content check in Step 3 is the only verification you get. ⚠ And "same version" is not "same artifact": two builds of one release can differ byte for byte, so compare against the tree you actually shipped from and never against a version string.

Two properties of that pair are load-bearing, not style:

Match the architecture per machine. uname -s/uname -m on each host; a fleet is routinely mixed (Linux runners plus a macOS machine).
mv -f is a rename, not an overwrite. A rename publishes a new inode, so work already executing the old binary keeps running against its own open file instead of being killed mid-run; an in-place cp over a live binary can also poison macOS's per-inode code-signature cache.

Then restart what runs. A running process keeps the binary it started with: until the console and the server are restarted (saphan-gateway stop / start, saphan-server stop / start, or your supervisor's own restart), they are the old release.

⚠ A far machine must run the same build as the machine that dispatches to it, or its governed runs are refused by name (legacy-far-wall-unavailable, at no cost, nothing spawned).

Step 3 — verify every machine, by content and by behaviour (Verifying an install). A fleet where some hosts took the update and others did not is worse than one where none did: the halves disagree, and the disagreement is silent. Verify the whole roster before calling the update done:

saphan machine list --workspace <workspace> # the roster to sweep