collin/anvil · 60a55cb5

Route CI jobs by platform (M2)

Collin Richards · 2026-08-24 09:18 UTC · 60a55cb5e27685a157866d9280eea906995f2083 · parent 6ba1be4c · browse files

modifiedCLAUDE.md+17 −12

modifiedanvil.example.toml+10 −0

modifiedcrates/anvil-ci/src/lib.rs+159 −9

modifiedcrates/anvil-core/src/ci.rs+60 −0

modifiedcrates/anvil-core/src/config.rs+40 −0

modifiedcrates/anvil-core/src/jobs.rs+89 −0

modifiedcrates/anvil-docker/src/lib.rs+43 −6

modifiedcrates/anvil-web/src/runner.rs+9 −2

modifieddocs/remote-runners.md+82 −19

⋯ 63 unchanged lines
64	64		`anvil-worker` (the runner binary), five endpoints under `/-/runner/`,
65	65		in-memory leases, and no Docker socket on the deployed container.
66	66
	67	+	M2 is done: `platform:` in `.anvil/ci.yml`, a `[ci] platform` default, and
	68	+	two-tier routing — a claiming runner is offered the runs that match its
	69	+	architecture (or name none), then the runs no connected runner is native to,
	70	+	so a lone arm64 Mac still runs `linux/amd64` pipelines under Rosetta instead of
	71	+	stranding them. `Dispatch` tracks who is connected from claims and heartbeats
	72	+	(`RUNNER_TTL`, 5 min); `anvil-docker::check_platform` fails a job whose image
	73	+	came back the wrong architecture.
	74	+
67	75		Invariants to not break. These are the reasons the split is shaped the way
68	76		it is, and each is easy to undo by accident:
69	77
⋯ 12 unchanged lines
82	90
83	91		Next, in order:
84	92
85		-	1. M2, platform. The plumbing is live end to end (`JobSpec.platform` reaches
86		-	both bollard option structs; runners advertise their native platform on
87		-	claim). What is missing is a source: a `platform:` key in `.anvil/ci.yml`,
88		-	probably a `[ci] platform` default, and routing a job to a runner that has
89		-	that architecture natively. This is what stops arm64 runners testing code
90		-	that ships as amd64.
91		-	2. M3, the deploy agent. Image builds cannot be CI jobs, because job
	93	+	1. M3, the deploy agent. Image builds cannot be CI jobs, because job
92	94		containers get no Docker socket by design. `docker build --platform
93	95		linux/amd64` plus `docker push` belongs to a separate process on the build
94	96		host at a different trust level, triggered by the existing `deploy_webhook`.
95	97		Folding it into the dial-out channel as a privileged "publish" job kind,
96	98		authorized by the `is_deploy_target` check that already scopes CD to one
97	99		repo, would remove the last inbound path to the build host.
98		-	3. CI concurrency. "One job at a time" was a property of the old in-process
	100	+	2. CI concurrency. "One job at a time" was a property of the old in-process
99	101		loop and is gone. Nothing bounds in-flight jobs now beyond how many runners
100		-	exist. Needs a `max_concurrent` equivalent.
101		-	4. Per-runner credentials. One shared `[ci] runner_token` means one
	102	+	exist. Needs a `max_concurrent` equivalent — more pressing now that platform
	103	+	routing makes a second runner worth having.
	104	+	3. Per-runner credentials. One shared `[ci] runner_token` means one
102	105		revocation for every runner, no `last_used_at`, and any holder can claim any
103	106		job and receive its secrets. Wants the API-token write scope first (see
104	107		TODO.md).
105		-	5. Live logs. The result POST is a single write, matching the old
	108	+	4. Live logs. The result POST is a single write, matching the old
106	109		behaviour. Streaming needs chunked append with offsets and a UI that
107	110		tolerates gaps. Newly worth doing now that a producer exists.
108	111
109	112		Operationally: nothing runs until a runner is started and `[ci]
110	113		runner_token` is set; queued runs just sit, with a warning logged at startup.
	114	+	Setting one up on the Mac mini (build, Rosetta, the launchd agent in
	115	+	`deploy/worker/`) is written out in `docs/remote-runners.md` § Isolation on
	116	+	macOS.
111	117		Agent sessions still drive Docker locally and are the one thing the dropped
112	118		socket mount gives up. They are off by default.
113	119
114	120		Current status, resume notes, the agreed next steps (a/b/c), and the roadmap live
115	121		in the TODO. Read it first:
116	122
117		-	@TODO.md

⋯ 80 unchanged lines
81	81		# anvil falls back to the local copy when the pull fails. Always allowed,
82	82		# whatever allowed_images says.
83	83		default_image = "anvil-runner:latest"
	84	+	# Platform (os/arch) for a pipeline that omits `platform:`. Empty runs every job
	85	+	# on whatever the claiming runner is native to.
	86	+	#
	87	+	# Set it to what you DEPLOY on when your runners are a different architecture:
	88	+	# an M-series Mac resolves `rust:1.95` to arm64 and will happily green-light
	89	+	# code you ship as amd64. anvil routes a job to a runner that is natively its
	90	+	# platform when one is connected, and falls back to an emulating runner (Rosetta
	91	+	# on macOS -- turn it on in Docker Desktop) when none is, rather than leaving the
	92	+	# run queued. See docs/remote-runners.md.
	93	+	platform = ""
84	94
85	95		[agent]
86	96		# Agent sessions: a container per session running tmux plus an agent CLI
⋯ 43 unchanged lines

⋯ 36 unchanged lines
37	37		/// rather than reading this field directly — it is empty when omitted.
38	38		#[serde(default)]
39	39		pub image: String,
	40	+	/// Docker platform to run on, `os/arch[/variant]` — e.g. `linux/amd64` on
	41	+	/// a repository that ships amd64 but has arm64 runners. Empty falls back to
	42	+	/// `[ci] platform`, and if that is empty too, to whatever the claiming
	43	+	/// runner's daemon is native to. Resolve it with
	44	+	/// [`CiConfig::resolve_platform`](crate::config::CiConfig::resolve_platform)
	45	+	/// rather than reading this field.
	46	+	///
	47	+	/// It is both an execution setting and a scheduling one: anvil hands the
	48	+	/// job to a runner that is natively this platform when it has one, and
	49	+	/// only falls back to an emulating runner when it does not.
	50	+	#[serde(default)]
	51	+	pub platform: String,
40	52		#[serde(default)]
41	53		pub steps: Vec<Step>,
42	54		#[serde(default)]
⋯ 45 unchanged lines
88	100		}
89	101		}
90	102
	103	+	/// Whether `platform` is a well-formed Docker platform: `os/arch`, optionally
	104	+	/// with a variant (`linux/arm/v7`), all lowercase `[a-z0-9._-]`.
	105	+	///
	106	+	/// Shape only, deliberately: the set of valid architectures is the daemon's to
	107	+	/// know, and a forge that hardcoded one would need a release to gain `riscv64`.
	108	+	/// What this does catch is the mistake worth catching — a bare `amd64` with no
	109	+	/// `os/`, which Docker would silently read as an operating system.
	110	+	pub fn valid_platform(platform: &str) -> bool {
	111	+	let parts: Vec<&str> = platform.split('/').collect();
	112	+	(2..=3).contains(&parts.len())
	113	+	&& parts.iter().all(\|part\| {
	114	+	!part.is_empty()
	115	+	&& part
	116	+	.chars()
	117	+	.all(\|c\| c.is_ascii_lowercase() \|\| c.is_ascii_digit() \|\| ".-_".contains(c))
	118	+	})
	119	+	}
	120	+
91	121		/// Parse a `.anvil/ci.yml` pipeline definition.
92	122		pub fn parse_pipeline(yaml: &str) -> Result<Pipeline> {
93	123		let mut pipeline: Pipeline = serde_yaml::from_str(yaml)
⋯ 3 unchanged lines
97	127		"{PIPELINE_PATH}: `image` is required"
98	128		)));
99	129		}
	130	+	pipeline.platform = pipeline.platform.trim().to_string();
	131	+	if !pipeline.platform.is_empty() && !valid_platform(&pipeline.platform) {
	132	+	return Err(Error::Invalid(format!(
	133	+	"{PIPELINE_PATH}: platform `{}` must be os/arch, e.g. linux/amd64",
	134	+	pipeline.platform
	135	+	)));
	136	+	}
100	137		let mut seen = std::collections::BTreeSet::new();
101	138		for a in &mut pipeline.artifacts {
102	139		let invalid =
⋯ 314 unchanged lines
417	454		assert!(parse_pipeline("steps: []\n").is_err());
418	455		}
419	456
	457	+	/// `platform:` is optional, and when present has to be `os/arch` — a bare
	458	+	/// `amd64` would otherwise reach Docker as an operating system named
	459	+	/// amd64, which fails much later and much less clearly.
	460	+	#[test]
	461	+	fn parses_and_validates_the_platform() {
	462	+	let p = parse_pipeline("image: rust:1.95\nplatform: linux/amd64\nsteps: []\n").unwrap();
	463	+	assert_eq!(p.platform, "linux/amd64");
	464	+	assert!(
	465	+	parse_pipeline("image: rust:1.95\nsteps: []\n")
	466	+	.unwrap()
	467	+	.platform
	468	+	.is_empty()
	469	+	);
	470	+
	471	+	assert!(valid_platform("linux/arm64"));
	472	+	assert!(valid_platform("linux/arm/v7"));
	473	+	assert!(!valid_platform("amd64"));
	474	+	assert!(!valid_platform("linux/"));
	475	+	assert!(!valid_platform("linux/amd64/v1/extra"));
	476	+	assert!(!valid_platform("Linux/AMD64"));
	477	+	assert!(parse_pipeline("image: rust:1.95\nplatform: amd64\nsteps: []\n").is_err());
	478	+	}
	479	+
420	480		#[test]
421	481		fn parses_and_validates_artifacts() {
422	482		let p = parse_pipeline(
⋯ 127 unchanged lines

⋯ 172 unchanged lines
173	173		/// pulled, which is why [`resolve_image`](CiConfig::resolve_image)'s caller
174	174		/// must tolerate a failed pull.
175	175		pub default_image: String,
	176	+	/// Platform (`os/arch`) for a pipeline that omits `platform:`. Empty runs
	177	+	/// every job on whatever the claiming runner is native to, which is the
	178	+	/// behaviour of an instance that never sets this.
	179	+	///
	180	+	/// Worth setting to the architecture you deploy on, on an instance whose
	181	+	/// runners are a different one: an M-series Mac resolves `rust:1.95` to
	182	+	/// arm64 and will happily test an architecture you never ship. See
	183	+	/// `docs/remote-runners.md`.
	184	+	pub platform: String,
176	185		/// Memory cap for a job container, in MiB (swap is capped to the same
177	186		/// value). `0` means unlimited. Defaults to 2048.
178	187		pub memory_mb: i64,
⋯ 171 unchanged lines
350	359		deploy_branch: "main".to_string(),
351	360		allowed_images: Vec::new(),
352	361		default_image: DEFAULT_RUNNER_IMAGE.to_string(),
	362	+	platform: String::new(),
353	363		memory_mb: 2048,
354	364		cpus: 2.0,
355	365		pids_limit: 512,
⋯ 26 unchanged lines
382	392		}
383	393		}
384	394
	395	+	/// The platform a pipeline runs on: what it asked for, then
	396	+	/// [`platform`](CiConfig::platform), then `None` — meaning the claiming
	397	+	/// runner's native architecture, and no constraint on which runner that is.
	398	+	pub fn resolve_platform<'a>(&'a self, requested: &'a str) -> Option<&'a str> {
	399	+	let platform = if requested.is_empty() {
	400	+	self.platform.as_str()
	401	+	} else {
	402	+	requested
	403	+	};
	404	+	(!platform.is_empty()).then_some(platform)
	405	+	}
	406	+
385	407		/// Whether `image` passes [`allowed_images`](CiConfig::allowed_images).
386	408		/// An empty allowlist permits any image; a tagless entry permits every tag
387	409		/// of that image; a tagged entry permits exactly itself. The default image
⋯ 222 unchanged lines
610	632		assert!(!ci.image_allowed("rust:1.95-bookworm"));
611	633		}
612	634
	635	+	/// Platform resolution has three levels, and the bottom one is "whatever
	636	+	/// the runner is", not a hardcoded architecture — an instance that never
	637	+	/// sets this behaves exactly as it did before platforms existed.
	638	+	#[test]
	639	+	fn a_platform_falls_back_from_pipeline_to_config_to_the_runner() {
	640	+	let ci = CiConfig::default();
	641	+	assert_eq!(ci.resolve_platform(""), None);
	642	+	assert_eq!(ci.resolve_platform("linux/arm64"), Some("linux/arm64"));
	643	+
	644	+	let ci = CiConfig {
	645	+	platform: "linux/amd64".to_string(),
	646	+	..CiConfig::default()
	647	+	};
	648	+	assert_eq!(ci.resolve_platform(""), Some("linux/amd64"));
	649	+	// A pipeline that names one still wins over the instance default.
	650	+	assert_eq!(ci.resolve_platform("linux/arm64"), Some("linux/arm64"));
	651	+	}
	652	+
613	653		/// Agent sessions are off unless the operator turns them on — they run a
614	654		/// model over repository content, which `docs/untrusted-mode.md` treats as
615	655		/// a different exposure from CI.
⋯ 81 unchanged lines

⋯ 48 unchanged lines
49	49		pub secrets: Vec<(String, String)>,
50	50		}
51	51
	52	+	/// How long a runner counts as present after its last claim or heartbeat.
	53	+	///
	54	+	/// Only platform routing reads this: a job that wants `linux/amd64` waits for
	55	+	/// a native runner while one is present, and is handed to an emulating runner
	56	+	/// once none is. Comfortably longer than both the claim poll (55s, so an idle
	57	+	/// runner refreshes itself) and [`HEARTBEAT_INTERVAL`] (30s, so a runner stays
	58	+	/// present through a long build), and short enough that a machine that went to
	59	+	/// sleep stops holding its architecture's jobs hostage for long.
	60	+	pub const RUNNER_TTL: Duration = Duration::from_secs(300);
	61	+
52	62		/// Shared dispatch state. Cheap to clone; all clones share one map.
53	63		#[derive(Clone, Default)]
54	64		pub struct Dispatch {
⋯ 3 unchanged lines
58	68		#[derive(Default)]
59	69		struct Inner {
60	70		leases: Mutex<HashMap<i64, Lease>>,
	71	+	/// Runner name → (advertised platform, last seen). What a runner says it
	72	+	/// is; not a credential and not checked — everyone holding the token is
	73	+	/// one principal, so a runner that lies about its architecture is only
	74	+	/// lying to itself about which jobs it gets.
	75	+	runners: Mutex<HashMap<String, (String, Instant)>>,
61	76		wake: tokio::sync::Notify,
62	77		/// Serializes claim attempts. Reading the queue and marking a run
63	78		/// `running` are separate awaits, so two runners polling at once could
⋯ 40 unchanged lines
104	119		);
105	120		}
106	121
	122	+	/// Note that `runner` is alive and what platform it says it is. Called on
	123	+	/// every claim and heartbeat, which is what keeps [`platforms`] honest
	124	+	/// about a busy runner as well as an idle one.
	125	+	///
	126	+	/// [`platforms`]: Dispatch::platforms
	127	+	pub fn seen(&self, runner: &str, platform: &str) {
	128	+	if platform.is_empty() {
	129	+	return;
	130	+	}
	131	+	self.inner
	132	+	.runners
	133	+	.lock()
	134	+	.unwrap()
	135	+	.insert(runner.to_string(), (platform.to_string(), Instant::now()));
	136	+	}
	137	+
	138	+	/// Platforms with a runner behind them right now (within [`RUNNER_TTL`]).
	139	+	///
	140	+	/// The dispatcher's answer to "is there anyone who could run this
	141	+	/// natively?" — if not, an emulating runner may take the job rather than
	142	+	/// leaving it queued forever.
	143	+	pub fn platforms(&self) -> Vec<String> {
	144	+	let now = Instant::now();
	145	+	let mut runners = self.inner.runners.lock().unwrap();
	146	+	runners.retain(\|_, (_, seen)\| now.duration_since(*seen) < RUNNER_TTL);
	147	+	let mut platforms: Vec<String> = runners.values().map(\|(p, _)\| p.clone()).collect();
	148	+	platforms.sort();
	149	+	platforms.dedup();
	150	+	platforms
	151	+	}
	152	+
	153	+	/// Every runner seen within [`RUNNER_TTL`], as (name, platform), for the
	154	+	/// admin view and for logging.
	155	+	pub fn runners(&self) -> Vec<(String, String)> {
	156	+	let now = Instant::now();
	157	+	let mut runners = self.inner.runners.lock().unwrap();
	158	+	runners.retain(\|_, (_, seen)\| now.duration_since(*seen) < RUNNER_TTL);
	159	+	let mut out: Vec<(String, String)> = runners
	160	+	.iter()
	161	+	.map(\|(name, (platform, _))\| (name.clone(), platform.clone()))
	162	+	.collect();
	163	+	out.sort();
	164	+	out
	165	+	}
	166	+
107	167		/// The secrets handed to `run_id`, for masking its log.
108	168		pub fn secrets_for(&self, run_id: i64) -> Vec<(String, String)> {
109	169		self.inner
⋯ 67 unchanged lines
177	237		.collect()
178	238		}
179	239		}
	240	+
	241	+	#[cfg(test)]
	242	+	mod tests {
	243	+	use super::*;
	244	+
	245	+	/// The dispatcher's view of who is out there: one entry per runner, one
	246	+	/// platform per architecture, and a re-registration under the same name
	247	+	/// (a runner restarted on a rebuilt machine) replaces rather than doubles.
	248	+	#[test]
	249	+	fn platforms_are_deduped_per_live_runner() {
	250	+	let jobs = Dispatch::new();
	251	+	assert!(jobs.platforms().is_empty());
	252	+
	253	+	jobs.seen("mac", "linux/arm64");
	254	+	jobs.seen("droplet", "linux/amd64");
	255	+	jobs.seen("laptop", "linux/arm64");
	256	+	assert_eq!(jobs.platforms(), vec!["linux/amd64", "linux/arm64"]);
	257	+	assert_eq!(jobs.runners().len(), 3);
	258	+
	259	+	jobs.seen("mac", "linux/amd64"); // same name, rebuilt as amd64
	260	+	assert_eq!(jobs.runners().len(), 3);
	261	+	assert_eq!(jobs.platforms(), vec!["linux/amd64", "linux/arm64"]);
	262	+
	263	+	// A runner that advertises nothing is not evidence of any platform,
	264	+	// and must not register as one — routing would then never fall back.
	265	+	jobs.seen("mystery", "");
	266	+	assert_eq!(jobs.runners().len(), 3);
	267	+	}
	268	+	}

⋯ 64 unchanged lines
65	65		"image {image} is not permitted by ci.allowed_images"
66	66		));
67	67		}
	68	+	// `platform:`, else `[ci] platform`, else the runner's native one. The
	69	+	// pipeline's own value was validated at parse time; this catches a
	70	+	// malformed `[ci] platform`, which nothing else would.
	71	+	let platform = cfg.resolve_platform(&pipeline.platform);
	72	+	if let Some(platform) = platform
	73	+	&& !ci::valid_platform(platform)
	74	+	{
	75	+	return Err(format!(
	76	+	"platform {platform} must be os/arch, e.g. linux/amd64 (from [ci] platform)"
	77	+	));
	78	+	}
68	79		Ok(JobSpec {
69	80		run_id,
70	81		image: image.to_string(),
71		-	// No source for this yet: the pipeline schema has no `platform:` key,
72		-	// so every job takes the runner daemon's native architecture, exactly
73		-	// as before. Honoured end to end the moment one is set.
74		-	platform: None,
	82	+	platform: platform.map(str::to_string),
75	83		script: build_script(pipeline),
76	84		env: env.to_vec(),
77	85		artifacts: pipeline
⋯ 113 unchanged lines
191	199		}
192	200		}
193	201
194		-	/// Hand the oldest queued run to `runner`, or `None` when there is nothing to
195		-	/// do.
	202	+	/// Hand a queued run to `runner`, or `None` when there is nothing for this
	203	+	/// runner to do.
	204	+	///
	205	+	/// `runner_platform` is what the runner advertised (`linux/arm64`, …) and
	206	+	/// decides which of the queued runs it is offered — see [`claim_order`].
196	207		///
197	208		/// Runs that cannot be dispatched at all — an unparseable pipeline, a sealed
198	209		/// vault, an image the allowlist forbids — are finished here rather than
199	210		/// handed out, and the next queued run is tried. That keeps a single bad
200	211		/// pipeline from wedging the queue.
201		-	pub async fn claim_next(app: &App, runner: &str) -> Option<JobSpec> {
	212	+	pub async fn claim_next(app: &App, runner: &str, runner_platform: &str) -> Option<JobSpec> {
202	213		// Held across the whole attempt: listing the queue and marking a run
203	214		// running are separate awaits, so without it two runners polling at the
204	215		// same moment could both be handed the same job.
⋯ 6 unchanged lines
211	222		return None;
212	223		}
213	224		};
	225	+
	226	+	// What each queued run wants to run on, oldest first. Reading the pipeline
	227	+	// is what tells us, so this is a git blob read per queued run — cheap, and
	228	+	// only on the claim path.
	229	+	let mut candidates = Vec::new();
214	230		for run_id in queued {
215	231		// Skip anything a concurrent claim already took: `queued_ids` reads
216	232		// committed state, and `prepare` marks running.
217	233		if app.jobs.holds_any(run_id) {
218	234		continue;
219	235		}
220		-	match prepare(app, run_id, runner).await {
	236	+	match target_platform(app, run_id).await {
	237	+	Ok(platform) => candidates.push((run_id, platform)),
	238	+	Err(e) => {
	239	+	tracing::error!("ci: preparing run {run_id} failed: {e}");
	240	+	fail_run(app, run_id, &format!("\n[dispatch error] {e}\n")).await;
	241	+	}
	242	+	}
	243	+	}
	244	+
	245	+	for run_id in claim_order(&candidates, runner_platform, &app.jobs.platforms()) {
	246	+	match prepare(app, run_id, runner, runner_platform).await {
221	247		Ok(Some(job)) => return Some(job),
222	248		Ok(None) => continue,
223	249		Err(e) => {
⋯ 5 unchanged lines
229	255		None
230	256		}
231	257
	258	+	/// Which of `candidates` this runner should be offered, in order.
	259	+	///
	260	+	/// Two tiers, each in queue order:
	261	+	///
	262	+	/// 1. Native — the run named this runner's platform, or named none at all.
	263	+	/// 2. Orphaned — the run named a platform that *no runner present right
	264	+	/// now* is native to. Someone has to run it, and an emulating runner beats
	265	+	/// a job that sits queued forever.
	266	+	///
	267	+	/// A run whose platform belongs to another live runner is left alone, which is
	268	+	/// the whole point: an arm64 Mac stops grabbing the amd64 jobs when an amd64
	269	+	/// runner exists to take them, and grabs them the moment it does not.
	270	+	///
	271	+	/// Note what this is not: a promise that the job runs natively. Nothing stops
	272	+	/// an operator from having exactly one arm64 runner and every pipeline asking
	273	+	/// for amd64 — that configuration emulates, which is correct, since testing
	274	+	/// the architecture you ship under Rosetta beats testing one you don't.
	275	+	fn claim_order(
	276	+	candidates: &[(i64, Option<String>)],
	277	+	runner_platform: &str,
	278	+	live_platforms: &[String],
	279	+	) -> Vec<i64> {
	280	+	let native = \|wanted: &Option<String>\| match wanted {
	281	+	None => true,
	282	+	Some(p) => p == runner_platform,
	283	+	};
	284	+	let orphaned = \|wanted: &Option<String>\| match wanted {
	285	+	None => false,
	286	+	Some(p) => !live_platforms.iter().any(\|live\| live == p),
	287	+	};
	288	+	candidates
	289	+	.iter()
	290	+	.filter(\|(_, wanted)\| native(wanted))
	291	+	.chain(
	292	+	candidates
	293	+	.iter()
	294	+	.filter(\|(_, wanted)\| !native(wanted) && orphaned(wanted)),
	295	+	)
	296	+	.map(\|(run_id, _)\| *run_id)
	297	+	.collect()
	298	+	}
	299	+
	300	+	/// The platform a queued run needs, for the routing decision — `None` when it
	301	+	/// names none and can run anywhere.
	302	+	///
	303	+	/// Reads the pipeline and nothing else: no vault, no lease, no `mark_running`.
	304	+	/// [`prepare`] does that for the run that is actually taken.
	305	+	async fn target_platform(app: &App, run_id: i64) -> Result<Option<String>, String> {
	306	+	let (_, _, repo_path, run) = resolve(app, run_id).await?;
	307	+	let yaml = browse::read_blob(&repo_path, &run.commit, ci::PIPELINE_PATH)
	308	+	.map_err(\|e\| e.to_string())?
	309	+	.ok_or_else(\|\| format!("{} missing at {}", ci::PIPELINE_PATH, run.commit))?;
	310	+	let pipeline =
	311	+	ci::parse_pipeline(&String::from_utf8_lossy(&yaml)).map_err(\|e\| e.to_string())?;
	312	+	Ok(app
	313	+	.config
	314	+	.ci
	315	+	.resolve_platform(&pipeline.platform)
	316	+	.map(str::to_string))
	317	+	}
	318	+
232	319		/// Everything anvil does before a job leaves the building: resolve the repo,
233	320		/// parse the pipeline, open the vault, build the spec, take the lease.
234	321		///
235	322		/// `Ok(None)` means the run was finished here and should not be dispatched.
236		-	async fn prepare(app: &App, run_id: i64, runner: &str) -> Result<Option<JobSpec>, String> {
	323	+	async fn prepare(
	324	+	app: &App,
	325	+	run_id: i64,
	326	+	runner: &str,
	327	+	runner_platform: &str,
	328	+	) -> Result<Option<JobSpec>, String> {
237	329		let run = ci::get(&app.db, run_id)
238	330		.await
239	331		.map_err(\|e\| e.to_string())?
⋯ 22 unchanged lines
262	354		run.ref_name,
263	355		app.config.ci.resolve_image(&pipeline.image)
264	356		);
	357	+	// Say so when the job is about to be emulated. Without this line a slow
	358	+	// amd64-on-arm64 build looks like a slow build.
	359	+	if let Some(platform) = app.config.ci.resolve_platform(&pipeline.platform) {
	360	+	log.push_str(&format!("platform: {platform}"));
	361	+	if !runner_platform.is_empty() && runner_platform != platform {
	362	+	log.push_str(&format!(" (emulated on {runner_platform})"));
	363	+	}
	364	+	log.push('\n');
	365	+	}
265	366
266	367		// Secrets the pipeline asked for, from the in-memory vault. anvil holds no
267	368		// key that opens the stored envelopes, so an unlock must have happened
⋯ 458 unchanged lines
726	827		b.into_inner().unwrap()
727	828		}
728	829
	830	+	/// Queued runs as (id, wanted platform), oldest first.
	831	+	fn queue(items: &[(i64, Option<&str>)]) -> Vec<(i64, Option<String>)> {
	832	+	items
	833	+	.iter()
	834	+	.map(\|(id, p)\| (*id, p.map(str::to_string)))
	835	+	.collect()
	836	+	}
	837	+
	838	+	/// A runner is offered its own platform's jobs and the unconstrained ones,
	839	+	/// in queue order, ahead of anything it would have to emulate.
	840	+	#[test]
	841	+	fn claim_order_prefers_native_work() {
	842	+	let queued = queue(&[
	843	+	(1, Some("linux/amd64")),
	844	+	(2, None),
	845	+	(3, Some("linux/arm64")),
	846	+	(4, Some("linux/amd64")),
	847	+	]);
	848	+	let live = ["linux/amd64".to_string(), "linux/arm64".to_string()];
	849	+
	850	+	// Both architectures are present, so neither runner touches the
	851	+	// other's jobs — not even when its own queue is empty.
	852	+	assert_eq!(claim_order(&queued, "linux/arm64", &live), vec![2, 3]);
	853	+	assert_eq!(claim_order(&queued, "linux/amd64", &live), vec![1, 2, 4]);
	854	+	}
	855	+
	856	+	/// With no runner of the named architecture present, the job is offered to
	857	+	/// whoever asks rather than sitting queued forever — after that runner's
	858	+	/// own work. This is the single-runner case: one arm64 Mac, pipelines that
	859	+	/// ask for amd64 because that is what they ship.
	860	+	#[test]
	861	+	fn claim_order_falls_back_to_emulation_when_nobody_is_native() {
	862	+	let queued = queue(&[(1, Some("linux/amd64")), (2, None)]);
	863	+	let live = ["linux/arm64".to_string()];
	864	+	assert_eq!(claim_order(&queued, "linux/arm64", &live), vec![2, 1]);
	865	+	}
	866	+
	867	+	/// An unknown platform (`linux/riscv64`, a typo) behaves like any other
	868	+	/// platform nobody is native to: it runs somewhere and fails visibly there,
	869	+	/// rather than disappearing from the queue.
	870	+	#[test]
	871	+	fn claim_order_never_strands_a_run() {
	872	+	let queued = queue(&[(1, Some("linux/riscv64"))]);
	873	+	assert_eq!(
	874	+	claim_order(&queued, "linux/arm64", &["linux/arm64".to_string()]),
	875	+	vec![1]
	876	+	);
	877	+	}
	878	+
729	879		fn spec(name: &str, path: &str, browse: bool) -> ArtifactSpec {
730	880		ArtifactSpec {
731	881		name: name.into(),
⋯ 55 unchanged lines

⋯ 62 unchanged lines
63	63		}
64	64
65	65		let Some(pull_error) = pull_error else {
66		-	return Ok(());
	66	+	return check_platform(docker, image, platform).await;
67	67		};
68	68
69	69		// The pull failed. That is fine if the image is already here — the local
70		-	// build case — and fatal otherwise. Note this does not check the local
71		-	// copy's architecture against `platform`: a cached image of the wrong arch
72		-	// satisfies it. Pulls are the normal path, so this only bites an offline
73		-	// runner that has been asked to cross-build.
	70	+	// build case — and fatal otherwise. The local copy still has to be the
	71	+	// right architecture: satisfying an offline cross-build from a cached
	72	+	// image of the wrong arch is how a job silently tests something other
	73	+	// than what it asked for.
74	74		match docker.inspect_image(image).await {
75	75		Ok(_) => {
76	76		tracing::debug!("pull of {image} failed ({pull_error}); using the local image");
77		-	Ok(())
	77	+	check_platform(docker, image, platform).await
78	78		}
79	79		Err(_) => Err(format!(
80	80		"image {image} is not available locally and could not be pulled: {pull_error}"
81	81		)),
82	82		}
83	83		}
	84	+
	85	+	/// Fail loudly when the local image is not the architecture the job asked for.
	86	+	///
	87	+	/// The daemon is not obliged to give you what `platform` asked for: a manifest
	88	+	/// with no such entry, or a pull that fell back to a cached image, both end
	89	+	/// with an image of the wrong arch and no error anywhere. That turns a job that
	90	+	/// exists precisely to test `linux/amd64` into one quietly testing arm64, and
	91	+	/// the only sign is a green run. Compared here rather than left to
	92	+	/// `create_container`, which happily starts an emulated container of the wrong
	93	+	/// architecture too.
	94	+	///
	95	+	/// The variant (`linux/arm/v7`) is not compared: it is rarely reported on the
	96	+	/// image and never the mistake worth catching.
	97	+	async fn check_platform(docker: &Docker, image: &str, platform: &str) -> Result<(), String> {
	98	+	if platform.is_empty() {
	99	+	return Ok(());
	100	+	}
	101	+	let mut wanted = platform.split('/');
	102	+	let (Some(want_os), Some(want_arch)) = (wanted.next(), wanted.next()) else {
	103	+	return Ok(()); // Not os/arch; the server validated it, so take it as given.
	104	+	};
	105	+
	106	+	let Ok(info) = docker.inspect_image(image).await else {
	107	+	return Ok(()); // Just pulled it; a failing inspect is not the job's problem.
	108	+	};
	109	+	let (Some(os), Some(arch)) = (info.os.as_deref(), info.architecture.as_deref()) else {
	110	+	return Ok(());
	111	+	};
	112	+	if os == want_os && arch == want_arch {
	113	+	return Ok(());
	114	+	}
	115	+	Err(format!(
	116	+	"image {image} is {os}/{arch}, not the requested {platform} — \
	117	+	the registry may have no {platform} manifest for it, or a local image \
	118	+	of the wrong architecture is shadowing the pull"
	119	+	))
	120	+	}

⋯ 118 unchanged lines
119	119		axum::Json(info): axum::Json<RunnerInfo>,
120	120		) -> Result<Response, Response> {
121	121		let name = authenticate(&app, &headers, &info)?;
	122	+	// Note the runner before claiming: what platforms are present decides
	123	+	// which jobs are routed where, and a runner asking for work is the freshest
	124	+	// evidence there is that its architecture has someone behind it.
	125	+	app.jobs.seen(&name, &info.platform);
122	126
123		-	if let Some(job) = anvil_ci::claim_next(&app, &name).await {
	127	+	if let Some(job) = anvil_ci::claim_next(&app, &name, &info.platform).await {
124	128		return Ok(axum::Json(job).into_response());
125	129		}
126	130		app.jobs.wait_for_work(CLAIM_POLL).await;
127		-	match anvil_ci::claim_next(&app, &name).await {
	131	+	match anvil_ci::claim_next(&app, &name, &info.platform).await {
128	132		Some(job) => Ok(axum::Json(job).into_response()),
129	133		None => Ok(StatusCode::NO_CONTENT.into_response()),
130	134		}
⋯ 25 unchanged lines
156	160		axum::Json(info): axum::Json<RunnerInfo>,
157	161		) -> Result<Response, Response> {
158	162		let name = authenticate(&app, &headers, &info)?;
	163	+	// A runner mid-build is not claiming, so its heartbeats are the only thing
	164	+	// keeping it visible to platform routing while a long job runs.
	165	+	app.jobs.seen(&name, &info.platform);
159	166		if app.jobs.heartbeat(run_id, &name) {
160	167		Ok(StatusCode::NO_CONTENT.into_response())
161	168		} else {
⋯ 66 unchanged lines

1	1		# Remote runners
2	2
3		-	Status: M1 implemented (2026-08-24). Supersedes the in-process CI
	3	+	Status: M1 + M2 implemented (2026-08-24). Supersedes the in-process CI
4	4		executor that used to live in `crates/anvil-ci`.
5	5
6	6		anvil used to be the runner: `run_worker` drained the queue in-process and
⋯ 200 unchanged lines
207	207		genuinely emulation-free. Nice-to-have.
208	208
209	209		What jobs run on. On an M2, `rust:1.95-bookworm` resolves to arm64, so
210		-	`cargo test` tests an architecture you don't ship. Fix: a `platform:` key in
211		-	`.anvil/ci.yml`, plumbed to `CreateContainerOptions.platform`
212		-	(bollard 0.18 `container.rs:107`) and `CreateImageOptions.platform`
213		-	(`image.rs:72`) — `ensure_image` currently leaves both at `Default::default()`.
214		-	Enable Docker Desktop's Rosetta option; it is far faster than QEMU for amd64
215		-	Linux binaries.
	210	+	`cargo test` tests an architecture you don't ship. That is what M2 fixes.
	211	+
	212	+	A job's platform comes from `platform:` in `.anvil/ci.yml`, falling back to
	213	+	`[ci] platform`, falling back to the claiming runner's native architecture —
	214	+	so an instance that sets neither behaves exactly as it did before. It reaches
	215	+	`CreateContainerOptions.platform` and `CreateImageOptions.platform`, and the
	216	+	value is validated as `os/arch[/variant]` at parse time (a bare `amd64` would
	217	+	otherwise reach Docker as an operating system named amd64).
216	218
217	219		Per-pipeline you then choose: native arm64 for lint and unit tests, amd64 under
218		-	Rosetta for anything arch-sensitive.
	220	+	Rosetta for anything arch-sensitive. Turn Rosetta on in Docker Desktop; it is
	221	+	far faster than QEMU for amd64 Linux binaries.
219	222
220		-	Once runners are plural, `platform` also becomes a scheduling hint — a job
221		-	declaring `linux/amd64` prefers a runner that is natively amd64. Not needed
222		-	with one runner; the field is forward-compatible with it.
	223	+	### Routing
223	224
	225	+	`platform` is also the scheduling dimension. Every claim and heartbeat records
	226	+	the runner's advertised architecture in `Dispatch` (`jobs.rs`), which expires
	227	+	after `RUNNER_TTL` — 5 minutes, longer than both the 55s claim poll and the 30s
	228	+	heartbeat, so an idle runner and a runner mid-build both stay visible. A
	229	+	claiming runner is then offered, in queue order:
	230	+
	231	+	1. runs that name its platform, or name none at all;
	232	+	2. then runs whose platform no currently connected runner is native to.
	233	+
	234	+	Tier 2 is what keeps one arm64 Mac usable as the only runner for pipelines that
	235	+	declare `linux/amd64`: nobody can run them natively, so it emulates them rather
	236	+	than leaving them queued forever. Add an amd64 runner and the Mac stops taking
	237	+	those jobs the moment the new runner's first claim registers it — no
	238	+	configuration, which is the property the dial-out model was chosen for.
	239	+
	240	+	The corollary worth stating plainly: `platform:` is not a promise of native
	241	+	execution. It is a promise about what the job runs, which the runner enforces
	242	+	by inspecting the image it ended up with (`anvil-docker::check_platform`) and
	243	+	failing the job if the architecture is not the one asked for. Where it runs is
	244	+	a scheduling preference. A run's log header says which it got:
	245	+
	246	+	```
	247	+	platform: linux/amd64 (emulated on linux/arm64)
	248	+	```
	249	+
224	250		## Isolation on macOS
225	251
226	252		Docker on macOS is a Linux VM (LinuxKit under Docker Desktop, Lima under
⋯ 21 unchanged lines
248	274		allocation. The dispatcher runs one job at a time, so this is slack rather than
249	275		a constraint.
250	276
	277	+	### Running one on a Mac mini
	278	+
	279	+	```sh
	280	+	# On the Mac, in a checkout of anvil:
	281	+	cargo build --release -p anvil-worker
	282	+	sudo cp target/release/anvil-worker /usr/local/bin/
	283	+
	284	+	# Docker Desktop → Settings → General:
	285	+	# ✓ Use Rosetta for x86_64/amd64 emulation on Apple Silicon
	286	+	# Without it, a linux/amd64 job runs under QEMU — correct, and much slower.
	287	+
	288	+	cp deploy/worker/com.anvil.worker.plist ~/Library/LaunchAgents/
	289	+	# Fill in --url, --name, ANVIL_RUNNER_TOKEN and DOCKER_HOST, then:
	290	+	chmod 600 ~/Library/LaunchAgents/com.anvil.worker.plist
	291	+	launchctl load -w ~/Library/LaunchAgents/com.anvil.worker.plist
	292	+	tail -f /tmp/anvil-worker.log # "anvil-worker macmini (linux/arm64) → …"
	293	+	```
	294	+
	295	+	The forge side needs `[ci] runner_token` set to the same secret; until it is,
	296	+	every claim gets a 503 saying so and queued runs sit.
	297	+
	298	+	The plist is a LaunchAgent, not a LaunchDaemon, because Docker Desktop's
	299	+	socket only exists inside the logged-in user's session — a root daemon starts
	300	+	before Docker and never finds it. The consequence to know about: the runner is
	301	+	only up while that user is logged in, and a Mac that sleeps stops claiming.
	302	+	That is the failure mode the dial-out design chose (a sleeping Mac reads as "no
	303	+	runner available"), and after `RUNNER_TTL` its architecture stops counting as
	304	+	present, so anything routed to it falls back to another runner.
	305	+
	306	+	`--name` matters: the default reads `$HOSTNAME`, which launchd does not set, so
	307	+	an unnamed runner is called `runner`.
	308	+
251	309		## Two processes on the build host
252	310
253	311		Image building cannot be a CI job. Job containers get no Docker socket by
⋯ 53 unchanged lines
307	365		`connect_with_defaults`, `run_worker` → `run_dispatcher`, and no socket mount in
308	366		the deployed container. Deploys keep using the existing webhook.
309	367
310		-	M2 — platform. The plumbing is already live: `JobSpec.platform` reaches
311		-	`CreateContainerOptions` and `CreateImageOptions`, and a runner advertises its
312		-	native platform when it claims. What is missing is a source — a `platform:`
313		-	key in `.anvil/ci.yml` (and probably a `[ci] platform` default), plus routing a
314		-	job to a runner that has that architecture natively.
	368	+	M2 — platform. Done. `platform:` in `.anvil/ci.yml`, a `[ci] platform`
	369	+	default, `CiConfig::resolve_platform`, the two-tier routing above (backed by a
	370	+	runner registry in `Dispatch`), the emulation note in the run header, and an
	371	+	architecture check on the image the runner actually got.
315	372
316	373		M3 — publish jobs. The deploy agent folds into the dial-out channel,
317	374		authorized by `is_deploy_target`. No inbound path to the build host remains.
⋯ 7 unchanged lines
325	382		runners, and no `last_used_at`. Wants the API-token write scope first.
326	383		- Runner labels. `platform` is the only scheduling dimension in M2. Tags
327	384		("has-postgres", "big-memory") are the obvious next axis and are not designed.
	385	+	- A runners page. `Dispatch::runners()` now knows every runner connected in
	386	+	the last five minutes and what it is. Nothing renders it, so "is my Mac
	387	+	actually claiming?" is still answered by reading logs.
	388	+	- Routing is per-claim, not per-queue. A runner that can take nothing sleeps
	389	+	until the next wake; it does not reserve the job it declined. With two runners
	390	+	and a job only one can run natively, the other simply keeps polling — correct,
	391	+	but it means a queue can look busy while a runner looks idle.
328	392		- Concurrency. Nothing bounds how many jobs are in flight beyond how many
329	393		runners exist, and nothing stops one runner claiming repeatedly. The
330	394		in-process runner's "one job at a time" was a property of the loop, and it is
331	395		gone; a `max_concurrent` equivalent for CI does not exist.
332		-	- `ensure_image`'s local fallback vs `platform`. A cached image of the
333		-	wrong architecture satisfies the fallback, since it inspects presence and not
334		-	arch. Only bites an offline runner asked to cross-build.
	396	+	- Concurrency, again. Platform routing makes a second runner useful, which
	397	+	makes the missing `max_concurrent` more pressing rather than less.