collin/anvil · 12a87324

Drop the Docker socket from the deployed container; document runners

Collin Richards · 2026-08-24 08:06 UTC · 12a873243540893c43dee47e9a55884c3f48dece · parent 9d6f3bcd · browse files

modifiedDEPLOY.md+66 −21

modifiedanvil.example.toml+11 −1

modifieddeploy/run.sh+20 −10

addeddeploy/worker/com.anvil.worker.plist+65 −0

modifieddocs/remote-runners.md+74 −33

⋯ 92 unchanged lines
93	93		docker run -d --name anvil --network hagrid --restart unless-stopped \
94	94		-p 2222:2222 \
95	95		-v anvil-data:/data \
96		-	-v /var/run/docker.sock:/var/run/docker.sock \
97		-	--group-add "$(stat -c '%g' /var/run/docker.sock)" \
98	96		anvil:latest
99	97		```
100	98
⋯ 2 unchanged lines
103	101		- `-v anvil-data:/data` — a named volume holding the SQLite DB, the bare repos,
104	102		and the persistent SSH host key. Use a named volume (not a host bind
105	103		mount) so it's owned by the in-container `anvil` user.
106		-	- `-v /var/run/docker.sock:/var/run/docker.sock` + `--group-add <sock gid>` —
107		-	lets the CI runner drive Docker on the host. The non-root `anvil` user
108		-	needs the socket's group to open it, hence `--group-add` with the socket's
109		-	gid (computed at run time by `run.sh`).
	104	+	- No Docker socket. anvil does not execute CI — runners dial in and run
	105	+	jobs on their own daemons (see [docs/remote-runners.md](docs/remote-runners.md)).
	106	+	The container has no reason to reach Docker, so the mount is gone, and with
	107	+	it the root-equivalent hold the internet-facing process used to have on the
	108	+	host.
110	109
111		-	> ⚠️ Security: mounting the Docker socket grants the container
112		-	> root-equivalent control of the host. This is acceptable here because anvil
113		-	> is single-tenant and owner-operated — CI only ever runs code you push.
114		-	> Do not open this instance to untrusted users while the socket is mounted.
115		-	> If you don't want CI, drop the `-v …docker.sock…` and `--group-add` flags;
116		-	> the forge runs fine without them (CI runs just error out).
	110	+	> Agent sessions are the exception. They still drive Docker locally, so
	111	+	> turning them on means putting the socket back
	112	+	> (`ANVIL_DOCKER_SOCK=/var/run/docker.sock ./deploy/run.sh`) and accepting that
	113	+	> the container again has root-equivalent control of the host. They are off
	114	+	> by default and absent from `deploy/anvil.toml`; read `docs/untrusted-mode.md`
	115	+	> §7 before changing that.
117	116
118	117		The baked config lives at `/etc/anvil/anvil.toml` (see `deploy/anvil.toml`).
119	118		Override it by bind-mounting your own file over that path.
⋯ 38 unchanged lines
158	157		also works: `git clone https://anvil.richardscollin.com/collin/anvil.git`
159	158		(pushes over HTTPS require your account password as the git password).
160	159
161		-	## 7. CI and the redeploy webhook (CD)
	160	+	## 7. CI: start a runner
	161	+
	162	+	anvil dispatches CI; it does not execute it. Nothing runs until a runner dials
	163	+	in, so this is a required step, not an optional one — see
	164	+	[docs/remote-runners.md](docs/remote-runners.md).
	165	+
	166	+	Set a shared secret in the config (`[ci] runner_token`), then on a machine with
	167	+	room to build — the Mac mini, not the droplet:
	168	+
	169	+	```sh
	170	+	cargo build --release --bin anvil-worker
	171	+	cp target/release/anvil-worker /usr/local/bin/
162	172
163		-	Any repo with a `.anvil/ci.yml` runs CI on push (see `[ci]` requires the Docker
164		-	socket mounted — section 4). A pipeline is just an image plus steps:
	173	+	anvil-worker --url https://anvil.richardscollin.com \
	174	+	--token "$ANVIL_RUNNER_TOKEN" --name macmini
	175	+	```
165	176
	177	+	`deploy/worker/com.anvil.worker.plist` runs it under launchd on macOS. Run it
	178	+	natively there rather than in a container: a containerized runner needs the
	179	+	Docker socket mounted into it, which rebuilds the hole section 4 just removed.
	180	+
	181	+	Isolation is not weaker for being on a Mac. Docker Desktop runs every container
	182	+	inside one Linux VM, so `--cap-drop=ALL`, `no-new-privileges` and the
	183	+	cgroup limits are enforced by the same kernel primitives as on Linux — with the
	184	+	VM as an extra boundary a bare-metal Linux host does not have.
	185	+
	186	+	Architecture. The Mac is arm64 and hagrid is x86_64, so a job running
	187	+	`cargo test` on the runner tests an architecture you do not ship. Images are
	188	+	unaffected (`deploy/build.sh` already cross-compiles and builds
	189	+	`--platform linux/amd64`), and per-pipeline platform selection is designed but
	190	+	not yet wired to a config key.
	191	+
	192	+	## 8. The redeploy webhook (CD)
	193	+
	194	+	Any repo with a `.anvil/ci.yml` runs CI on push. A pipeline is just an image
	195	+	plus steps:
	196	+
166	197		```yaml
167	198		image: rust:1.95-bookworm
168	199		steps:
⋯ 5 unchanged lines
174	205		Runs show up at `/{owner}/{repo}/ci`, with a per-commit status badge on the
175	206		commit list and a full log on each run's page.
176	207
177		-	Job sandbox. anvil is the only Docker client; the job container gets no
178		-	socket, no mounts (the checkout is uploaded as a tar), all capabilities
179		-	dropped, and `no-new-privileges`. Resource bounds are configurable under
180		-	`[ci]`: `memory_mb` (default 2048), `cpus` (2), `pids_limit` (512),
	208	+	Job sandbox. The runner is the only Docker client on its machine; the job
	209	+	container gets no socket, no mounts (the checkout is uploaded as a tar), all
	210	+	capabilities dropped, and `no-new-privileges`. Resource bounds come from the
	211	+	forge's `[ci]` config and travel with each job, so tightening one does not need
	212	+	runners redeployed: `memory_mb` (default 2048), `cpus` (2), `pids_limit` (512),
181	213		`timeout_secs` (1800, then the container is killed), `network` (true),
182	214		`run_as` (empty = image default), and `allowed_images` (empty = any; a tagless
183		-	entry like `"rust"` allows every tag). See `docs/untrusted-mode.md` for the
184		-	threat model and what this does/doesn't protect against.
	215	+	entry like `"rust"` allows every tag). The allowlist is applied on the forge
	216	+	when the job is built, so a runner cannot widen it. See
	217	+	`docs/untrusted-mode.md` for the threat model and what this does/doesn't
	218	+	protect against.
185	219
	220	+	Secrets now leave the host. A pipeline's secrets are sent to the runner
	221	+	with its job and sit in plaintext in a container on a machine anvil does not
	222	+	own. Scope repository secrets accordingly, and treat a runner host as being as
	223	+	trusted as the forge itself.
	224	+
186	225		Continuous deployment is deliberately scoped to one repository. On a
187	226		successful run of `deploy_branch` (default `main`) in the repo named by
188	227		`[ci] deploy_repo`, anvil POSTs JSON to `[ci] deploy_webhook`:
⋯ 10 unchanged lines
199	238		(section 3), "redeploy anvil" usually means: the receiver pulls the freshly
200	239		built image and re-runs `deploy/run.sh` — it does not build in-place.
201	240
	241	+	Building an image is not a CI job. Job containers get no Docker socket by
	242	+	design, so `docker build`/`docker push` cannot happen inside a pipeline. That
	243	+	work belongs to a separate deploy agent on the build host, triggered by this
	244	+	webhook, running outside the sandbox with Docker access — a different trust
	245	+	level from the runner, and deliberately a different process.
	246	+
202	247		> The deploy receiver runs with whatever privileges you give it — keep it
203	248		> minimal, secret-gated, and bound to localhost / the Docker host gateway only.
204	249
⋯ 16 unchanged lines

⋯ 49 unchanged lines
50	50		clone_user = "git"
51	51
52	52		[ci]
	53	+	# Shared secret a runner presents as X-Anvil-Runner-Token to claim and report
	54	+	# jobs (docs/remote-runners.md). anvil does NOT execute CI itself -- an empty
	55	+	# token means no runner can authenticate and queued runs simply sit there.
	56	+	#
	57	+	# Start a runner on a machine with room to build:
	58	+	# anvil-worker --url https://anvil.example.com --token "$ANVIL_RUNNER_TOKEN"
	59	+	runner_token = ""
	60	+
53	61		# Job sandbox. Containers always run with no Docker socket, no mounts, all
54	62		# capabilities dropped, and no-new-privileges; these knobs bound resources
55		-	# (0 = unlimited). See docs/untrusted-mode.md for the threat model.
	63	+	# (0 = unlimited). Enforced by the runner, from these values -- tightening a
	64	+	# limit here does not need runners redeployed. See docs/untrusted-mode.md for
	65	+	# the threat model.
56	66		# Images a pipeline may use: empty allows any; a tagless entry ("rust") allows
57	67		# every tag of that image; a tagged one ("alpine:3.20") exactly itself.
58	68		allowed_images = []
⋯ 61 unchanged lines

⋯ 12 unchanged lines
13	13		# (137.184.249.48) is reached via the anchor IP 10.15.0.6, where the host's
14	14		# own sshd listens — binding a specific IP here keeps the two off each other.
15	15		SSH_BIND_IP="${ANVIL_SSH_BIND_IP:-165.232.162.167}"
16		-	DOCKER_SOCK="${ANVIL_DOCKER_SOCK:-/var/run/docker.sock}"
	16	+	# NO DOCKER SOCKET. anvil does not execute CI any more -- runners dial in and
	17	+	# run jobs on their own daemons (docs/remote-runners.md), so the container has
	18	+	# no reason to reach Docker at all. Dropping the mount removes what used to be
	19	+	# a root-equivalent hold on the host from the internet-facing process.
	20	+	#
	21	+	# The one thing this gives up is agent sessions, which still drive Docker
	22	+	# locally (crates/anvil-agent). They are off in deploy/anvil.toml and off by
	23	+	# default, so nothing here regresses. Set ANVIL_DOCKER_SOCK=/var/run/docker.sock
	24	+	# to put the mount back if you turn them on -- and re-read docs/untrusted-mode.md
	25	+	# before you do.
	26	+	DOCKER_SOCK="${ANVIL_DOCKER_SOCK:-}"
17	27
18		-	# The CI runner drives Docker via the host socket. Mount it in, and add the
19		-	# socket's group to the non-root `anvil` user so it can actually open it.
20		-	# NOTE: socket access = root-equivalent on the host. We accept this because
21		-	# anvil is a single-tenant, owner-operated forge; CI only runs code the owner
22		-	# pushed. Do not expose this instance to untrusted users.
23		-	SOCK_GID="$(stat -c '%g' "$DOCKER_SOCK")"
	28	+	DOCKER_ARGS=()
	29	+	if [[ -n "$DOCKER_SOCK" ]]; then
	30	+	DOCKER_ARGS=(-v "${DOCKER_SOCK}:/var/run/docker.sock"
	31	+	--group-add "$(stat -c '%g' "$DOCKER_SOCK")")
	32	+	echo "==> WARNING: mounting ${DOCKER_SOCK} (root-equivalent on this host)"
	33	+	fi
24	34
25	35		# Single sign-on's client secret, if this instance uses one (docs/oidc.md).
26	36		#
⋯ 25 unchanged lines
52	62		--restart unless-stopped \
53	63		-p "${SSH_BIND_IP}:${SSH_PORT}:2222" \
54	64		-v anvil-data:/data \
55		-	-v "${DOCKER_SOCK}:/var/run/docker.sock" \
56		-	--group-add "$SOCK_GID" \
	65	+	"${DOCKER_ARGS[@]}" \
57	66		"${OIDC_ENV[@]}" \
58	67		"$IMAGE"
59	68
60		-	echo "==> anvil (re)started from $IMAGE (web: anvil:3000 via Caddy, ssh: ${SSH_BIND_IP}:${SSH_PORT}, docker.sock gid ${SOCK_GID})"
	69	+	echo "==> anvil (re)started from $IMAGE (web: anvil:3000 via Caddy, ssh: ${SSH_BIND_IP}:${SSH_PORT})"
	70	+	echo "==> CI needs a runner: anvil-worker --url https://anvil.richardscollin.com --token ..."

1	+	<?xml version="1.0" encoding="UTF-8"?>
2	+	<!--
3	+	launchd agent for anvil-worker on the build host (docs/remote-runners.md).
4	+
5	+	Install:
6	+	cp deploy/worker/com.anvil.worker.plist ~/Library/LaunchAgents/
7	+	# edit the paths, URL and token below first
8	+	chmod 600 ~/Library/LaunchAgents/com.anvil.worker.plist
9	+	launchctl load -w ~/Library/LaunchAgents/com.anvil.worker.plist
10	+
11	+	The token sits in this file in plaintext, so keep it 0600 and do not commit
12	+	a filled-in copy. It is the credential for every runner on the instance
13	+	(`[ci] runner_token`), not one scoped to this machine.
14	+
15	+	A LaunchAgent (per-user) rather than a LaunchDaemon (system): Docker Desktop
16	+	runs as the logged-in user, so its socket only exists in that user's session.
17	+	A root daemon would start before Docker and never find it.
18	+
19	+	Run natively like this, NOT in a container. A containerized runner needs the
20	+	Docker socket mounted into it, which rebuilds exactly the root-equivalent
21	+	hole that moving execution off the forge was meant to remove.
22	+	-->
23	+	<plist version="1.0">
24	+	<dict>
25	+	<key>Label</key>
26	+	<string>com.anvil.worker</string>
27	+
28	+	<key>ProgramArguments</key>
29	+	<array>
30	+	<string>/usr/local/bin/anvil-worker</string>
31	+	<string>--url</string>
32	+	<string>https://anvil.richardscollin.com</string>
33	+	<string>--name</string>
34	+	<string>macmini</string>
35	+	</array>
36	+
37	+	<key>EnvironmentVariables</key>
38	+	<dict>
39	+	<!-- Docker Desktop does not create /var/run/docker.sock unless "Allow
40	+	the default Docker socket to be used" is ticked, so point at the
41	+	real one. Colima/OrbStack put it elsewhere again. -->
42	+	<key>DOCKER_HOST</key>
43	+	<string>unix:///Users/collin/.docker/run/docker.sock</string>
44	+	<key>ANVIL_RUNNER_TOKEN</key>
45	+	<string>REPLACE_ME</string>
46	+	</dict>
47	+
48	+	<key>RunAtLoad</key>
49	+	<true/>
50	+
51	+	<!-- The claim loop is meant to run forever; if it exits, something is
52	+	wrong and it should come back. -->
53	+	<key>KeepAlive</key>
54	+	<true/>
55	+
56	+	<!-- Do not spin if it is crash-looping (a bad token, no daemon). -->
57	+	<key>ThrottleInterval</key>
58	+	<integer>30</integer>
59	+
60	+	<key>StandardOutPath</key>
61	+	<string>/tmp/anvil-worker.log</string>
62	+	<key>StandardErrorPath</key>
63	+	<string>/tmp/anvil-worker.log</string>
64	+	</dict>
65	+	</plist>

1	1		# Remote runners
2	2
3		-	Status: design (2026-08-24). Supersedes the in-process CI executor in
4		-	`crates/anvil-ci`.
	3	+	Status: M1 implemented (2026-08-24). Supersedes the in-process CI
	4	+	executor that used to live in `crates/anvil-ci`.
5	5
6		-	Today anvil is the runner: `run_worker` (`anvil-ci/src/lib.rs:73`) drains the
7		-	queue in-process and `execute` (`:387`) creates the job container directly on
8		-	the local Docker socket. The crate header states the model outright — "anvil is
9		-	the broker: it is the only Docker client."
10		-
11		-	That works, and it pins CI to whichever machine anvil runs on. anvil runs on
12		-	hagrid, a droplet small enough that a release build OOMs it (see
13		-	[DEPLOY.md](../DEPLOY.md) §3). So CI has to run somewhere else.
	6	+	anvil used to be the runner: `run_worker` drained the queue in-process and
	7	+	`execute` created the job container on the local Docker socket. That works, and
	8	+	it pins CI to whichever machine anvil runs on — which is hagrid, a droplet
	9	+	small enough that a release build OOMs it (see [DEPLOY.md](../DEPLOY.md) §3).
	10	+	So CI had to run somewhere else.
14	11
15	12		A runner is a separate binary that dials out to anvil, claims a job, runs it
16	13		in a sandboxed container on its own Docker daemon, and posts the result back.
⋯ 43 unchanged lines
60	57
61	58		## What moves
62	59
63		-	\| Stays in `anvild` \| Moves to `anvil-worker` \|
	60	+	\| Stays in `anvild` \| Moved to `anvil-worker` \|
64	61		\| ------------------------------------------ \| ---------------------------- \|
65	62		\| queue, `enqueue`/`requeue_interrupted` \| `execute` \|
66	63		\| repo + tree resolution, `build_tar` \| `collect_artifacts` \|
67		-	\| `parse_pipeline`, image allowlist check \| `store_artifact` \|
68		-	\| the secret vault (`app.vault.take`) \| `download_tar` \|
69		-	\| step-script assembly, `single_quote` \| `parse_meta_tar` \|
70		-	\| artifact storage, the swap, `gc_artifacts` \| `docker.rs` \|
	64	+	\| `parse_pipeline`, image allowlist check \| `download_tar` \|
	65	+	\| the secret vault (`app.vault.take`) \| `parse_meta_tar` \|
	66	+	\| step-script assembly, `single_quote` \| the `ArtifactSink` trait \|
	67	+	\| `store_artifact`, the swap, `gc_artifacts` \| \|
71	68		\| `mask_secrets` (applied on receipt) \| \|
72	69		\| the deploy webhook \| \|
73	70
	71	+	`store_artifact` deliberately stayed behind. The runner uploads the raw tar it
	72	+	pulled out of the container and anvil decides what to do with it, so on-disk
	73	+	layout — and `browse`, which turns a tarball into a servable directory tree —
	74	+	never becomes a runner's call. The upload response reports the stored size, so
	75	+	the per-run artifact budget is still charged what actually landed.
	76	+
74	77		`run_worker` becomes `run_dispatcher`: same queue drain, but instead of calling
75	78		`execute` it parks the job until a runner claims it.
76	79
⋯ 2 unchanged lines
79	82		sandbox caps, and a tar. That keeps the wire format stable as the pipeline
80	83		schema grows.
81	84
82		-	### The crate name
	85	+	### The crates
	86	+
	87	+	\| Crate \| Holds \|
	88	+	\| --------------- \| ------------------------------------------------------ \|
	89	+	\| `anvil-job` \| the wire format, and nothing else — serde only \|
	90	+	\| `anvil-docker` \| `connect`/`ensure_image`, shared with `anvil-agent` \|
	91	+	\| `anvil-worker` \| the runner binary: claim loop, client, executor \|
	92	+
	93	+	`anvil-job` exists so the runner does not link `anvil-core` — and therefore
	94	+	toasty, SQLite, gix and the rest of the forge — just to learn the shape of a
	95	+	job. `anvil-docker` exists so `anvil-agent` need not depend on the runner.
83	96
84	97		The binary is `anvil-worker`, not `anvil-runner`, because
85	98		`anvil-runner:latest` is already the image CI jobs and agent sessions run in
⋯ 38 unchanged lines
124	137
125	138		### Logs
126	139
127		-	The runner posts the whole log with `result`, which is exactly current
128		-	behaviour: `append_log` is called only twice in `process` — once on the
129		-	secrets-failure exit (`:161`) and once when the run ends (`:233`). The log
130		-	accumulates in memory and lands in one write. The run page is not live today
131		-	and does not become less live.
	140	+	The runner posts the whole log with `result`, which is exactly what the
	141	+	in-process runner did: `append_log` was only ever called when a run ended (and
	142	+	on the secrets-failure exit). The log accumulated in memory and landed in one
	143	+	write. The run page was not live before and is no less live now — a header
	144	+	naming the runner is now written at claim time, so a `running` run at least
	145	+	shows something.
132	146
133	147		Live logs are a genuine follow-up, and a remote runner makes them easier to
134	148		justify (there is now a producer that could stream). Out of scope here.
⋯ 1 unchanged line
136	150		### Auth
137	151
138	152		A shared secret in `[ci] runner_token`, sent as `X-Anvil-Runner-Token`,
139		-	constant-time compared.
	153	+	constant-time compared. The runner's self-asserted name rides alongside in
	154	+	`X-Anvil-Runner-Name` — it labels runs and keys leases, and is explicitly not
	155	+	a credential: everyone holding the token is one principal.
	156	+
	157	+	The four per-job endpoints additionally require the caller to hold that run's
	158	+	lease, so a valid token gets you a job rather than everyone else's. A lease
	159	+	mismatch answers 409, not 403: the caller is a legitimate runner whose claim
	160	+	simply expired.
140	161
141	162		This matches the existing `deploy_secret` pattern rather than inventing a
142	163		credential type. It is deliberate: API tokens are read-only and Bearer-only on
⋯ 4 unchanged lines
147	168
148	169		### Leases
149	170
150		-	Held in memory on the server: `run_id -> (runner_name, expires_at)`, bumped
151		-	by `heartbeat`, swept periodically. An expired lease returns the run to
	171	+	Held in memory on the server (`anvil_core::jobs::Dispatch`):
	172	+	`run_id -> (runner_name, expires_at, secrets)`, bumped by `heartbeat` every 30s
	173	+	against a 120s TTL, swept every 30s. An expired lease returns the run to
152	174		`queued`.
153	175
	176	+	A job's secrets are stashed on its lease rather than re-read from the vault
	177	+	when the result lands. `Vault::take` fails once the repository's unlock TTL
	178	+	lapses, and a job can easily outlive an unlock — re-reading would mean a long
	179	+	run silently skips log masking, which is exactly the run whose log is most
	180	+	likely to contain something. The values are already in this process's vault,
	181	+	so this is not new exposure.
	182	+
	183	+	The honest gap: an expired lease can double-run a job whose runner is alive but
	184	+	unreachable. The container keeps going and the requeued run may be claimed
	185	+	elsewhere. That is what a lease without fencing buys; CI steps are assumed
	186	+	idempotent.
	187	+
154	188		In-memory rather than columns on `CiRun` because Toasty migrations do not exist
155	189		yet — DEPLOY.md §Operations and TODO.md both flag that schema changes don't
156	190		auto-apply to the live database. Adding `claimed_by`/`lease_expires_at` to
⋯ 111 unchanged lines
268	302
269	303		## Milestones
270	304
271		-	M1 — the split. `anvil-docker` and `anvil-worker` crates, the five
272		-	endpoints, in-memory leases, `[ci] runner_token`, `connect_with_defaults`,
273		-	`run_worker` → `run_dispatcher`, socket mount dropped from `deploy/run.sh`.
274		-	Deploys keep using the existing webhook.
	305	+	M1 — the split. Done. `anvil-job`, `anvil-docker` and `anvil-worker`
	306	+	crates, the five endpoints, in-memory leases, `[ci] runner_token`,
	307	+	`connect_with_defaults`, `run_worker` → `run_dispatcher`, socket mount dropped
	308	+	from `deploy/run.sh`. Deploys keep using the existing webhook.
275	309
276		-	M2 — platform. `platform:` in `.anvil/ci.yml`, plumbed through both bollard
277		-	options. Runner advertises its native platform at claim time.
	310	+	M2 — platform. The plumbing is already live: `JobSpec.platform` reaches
	311	+	`CreateContainerOptions` and `CreateImageOptions`, and a runner advertises its
	312	+	native platform when it claims. What is missing is a source — a `platform:`
	313	+	key in `.anvil/ci.yml` (and probably a `[ci] platform` default), plus routing a
	314	+	job to a runner that has that architecture natively.
278	315
279	316		M3 — publish jobs. The deploy agent folds into the dial-out channel,
280	317		authorized by `is_deploy_target`. No inbound path to the build host remains.
⋯ 11 unchanged lines
292	329		plaintext in a container on a machine anvil does not own. This needs a
293	330		paragraph in `untrusted-mode.md` §1 — the exposure is no longer bounded by
294	331		hagrid.
295		-	- Concurrency. The dispatcher still hands out one job at a time, inherited
296		-	from `run_worker`. Multiple runners make that the binding constraint rather
297		-	than a sensible default.
	332	+	- Concurrency. Nothing bounds how many jobs are in flight beyond how many
	333	+	runners exist, and nothing stops one runner claiming repeatedly. The
	334	+	in-process runner's "one job at a time" was a property of the loop, and it is
	335	+	gone; a `max_concurrent` equivalent for CI does not exist.
	336	+	- `ensure_image`'s local fallback vs `platform`. A cached image of the
	337	+	wrong architecture satisfies the fallback, since it inspects presence and not
	338	+	arch. Only bites an offline runner asked to cross-build.