collin/anvil · 335e11b6

Show connected CI runners at /-/admin/runners

Collin Richards · 2026-08-24 10:15 UTC · 335e11b6b6b8fb35f6f5a1b43919bba4694b8cf5 · parent 47826aaf · browse files

modifiedCLAUDE.md+4 −1

modifiedcrates/anvil-core/src/jobs.rs+150 −37

modifiedcrates/anvil-core/src/secrets.rs+13 −0

modifiedcrates/anvil-job/src/lib.rs+17 −7

modifiedcrates/anvil-web/src/admin.rs+211 −1

modifiedcrates/anvil-web/src/runner.rs+3 −6

modifiedcrates/anvil-web/src/ui.rs+4 −1

modifieddocs/remote-runners.md+29 −0

⋯ 115 unchanged lines
116	116		macOS. Locally, `compose.override.yaml` runs two of them in containers
117	117		(`./deploy/build.sh --debug --worker`, then `docker compose up -d --build`) so
118	118		concurrency and routing are testable on one machine; that mount is a
119		-	development-only concession, never production's.
	119	+	development-only concession, never production's. Who is connected, and what
	120	+	each is running, is at `/-/admin/runners` — presence comes from ordinary claims
	121	+	and heartbeats rather than a ping of its own, so anything watching for a runner
	122	+	to go quiet has to allow a full claim poll first.
120	123		Agent sessions still drive Docker locally and are the one thing the dropped
121	124		socket mount gives up. They are off by default.
122	125
⋯ 3 unchanged lines

⋯ 851 unchanged lines
852	852
853	853		/// Fetch the named secrets for a CI run. Returns the names that are not
854	854		/// available as the error, so the runner can say exactly what is missing.
	855	+	///
	856	+	/// Asking for nothing always succeeds, sealed repository or not — the
	857	+	/// overwhelmingly common pipeline declares no `secrets:` at all, and
	858	+	/// failing it here would mean no repository could run CI until someone had
	859	+	/// unlocked it for secrets it does not use.
855	860		pub fn take(
856	861		&self,
857	862		repo_id: i64,
858	863		names: &[String],
859	864		) -> std::result::Result<Vec<(String, String)>, Vec<String>> {
	865	+	if names.is_empty() {
	866	+	return Ok(Vec::new());
	867	+	}
860	868		let mut map = self.inner.lock().expect("vault mutex");
861	869		let Some(entry) = map.get(&repo_id) else {
862	870		return Err(names.to_vec());
⋯ 168 unchanged lines
1031	1039		// A key with nothing unlocked reports every requested name missing.
1032	1040		let missing = vault.take(2, &["A".to_string()]).unwrap_err();
1033	1041		assert_eq!(missing, vec!["A".to_string()]);
	1042	+
	1043	+	// But a pipeline that declares no secrets is satisfiable by a sealed
	1044	+	// repository, which is the case nearly every pipeline is in: CI must
	1045	+	// not require an unlock for secrets it never asked for.
	1046	+	assert!(vault.take(2, &[]).unwrap().is_empty());
1034	1047		}
1035	1048
1036	1049		#[test]
⋯ 39 unchanged lines

⋯ 46 unchanged lines
47	47		/// runner so both halves of the pair stay in view of each other.
48	48		pub const HEARTBEAT_INTERVAL: Duration = Duration::from_secs(30);
49	49
	50	+	/// How long anvil parks a claim that finds nothing queued.
	51	+	///
	52	+	/// This is the other half of runner liveness: an idle runner is not
	53	+	/// heartbeating anything, so its claim poll returning and being reissued is
	54	+	/// what keeps it visible. Anything watching for a runner to go quiet has to
	55	+	/// allow for a full poll window plus the reconnect, which is why the number
	56	+	/// lives here rather than beside the endpoint that parks on it. The runner's
	57	+	/// own request timeout is set comfortably above it.
	58	+	pub const CLAIM_POLL: Duration = Duration::from_secs(55);
	59	+
50	60		/// A MiB cap from config as a byte count: `0` (unlimited) → [`u64::MAX`].
51	61		///
52	62		/// Here rather than on either side because both apply the same caps — the
⋯ 8 unchanged lines
61	71		}
62	72
63	73		/// What a runner says about itself when it asks for work.
64		-	#[derive(Clone, Debug, Serialize, Deserialize)]
	74	+	#[derive(Clone, Debug, Deserialize, Serialize)]
65	75		pub struct RunnerInfo {
66	76		/// Operator-chosen, for logs and (later) scheduling. Not a credential.
67	77		pub name: String,
⋯ 8 unchanged lines
76	86		/// The extractor commands themselves are not here — the server has already
77	87		/// baked them into the script. All the runner needs to know is whether to
78	88		/// expect output for this artifact under [`META_DIR`].
79		-	#[derive(Clone, Debug, Serialize, Deserialize)]
	89	+	#[derive(Clone, Debug, Deserialize, Serialize)]
80	90		pub struct ArtifactSpec {
81	91		pub name: String,
82	92		/// Relative to [`WORKDIR`].
⋯ 7 unchanged lines
90	100		/// The runner obeys these; it does not consult a config file of its own for
91	101		/// them. An operator tightening `ci.memory_mb` should not have to redeploy
92	102		/// every runner.
93		-	#[derive(Clone, Debug, Serialize, Deserialize)]
	103	+	#[derive(Clone, Debug, Deserialize, Serialize)]
94	104		pub struct Sandbox {
95	105		pub memory_mb: i64,
96	106		pub cpus: f64,
⋯ 19 unchanged lines
116	126		/// rather than carried here: base64 in a JSON body inflates a large tree by a
117	127		/// third for no benefit. Secrets are carried here, over TLS, so they never
118	128		/// sit behind a separately-fetchable URL.
119		-	#[derive(Clone, Debug, Serialize, Deserialize)]
	129	+	#[derive(Clone, Debug, Deserialize, Serialize)]
120	130		pub struct JobSpec {
121	131		pub run_id: i64,
122	132		/// Already resolved through `ci.default_image` and checked against
⋯ 13 unchanged lines
136	146		/// What storing one artifact produced. Returned by the upload endpoint, and by
137	147		/// the in-process sink, so the runner can charge the run budget the size that
138	148		/// actually landed on disk without knowing how it was laid out.
139		-	#[derive(Clone, Copy, Debug, Serialize, Deserialize)]
	149	+	#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
140	150		pub struct Stored {
141	151		pub size: i64,
142	152		pub is_dir: bool,
⋯ 1 unchanged line
144	154
145	155		/// One artifact the runner pulled out of the container and uploaded. The bytes
146	156		/// went ahead of this; here is the row to record for them.
147		-	#[derive(Clone, Debug, Serialize, Deserialize)]
	157	+	#[derive(Clone, Debug, Deserialize, Serialize)]
148	158		pub struct CollectedArtifact {
149	159		pub name: String,
150	160		pub size: i64,
⋯ 8 unchanged lines
159	169		/// The whole log arrives in one write, which is exactly what the in-process
160	170		/// runner did — `append_log` was only ever called when the run ended. Live
161	171		/// logs are a follow-up, not a regression.
162		-	#[derive(Clone, Debug, Serialize, Deserialize)]
	172	+	#[derive(Clone, Debug, Deserialize, Serialize)]
163	173		pub struct JobResult {
164	174		pub exit_code: i64,
165	175		pub log: String,
⋯ 6 unchanged lines

⋯ 121 unchanged lines
122	122		// Note the runner before claiming: what platforms are present decides
123	123		// which jobs are routed where, and a runner asking for work is the freshest
124	124		// evidence there is that its architecture has someone behind it.
125		-	app.jobs.seen(&name, &info.platform);
	125	+	app.jobs.seen(&name, &info.platform, &info.version);
126	126
127	127		if let Some(job) = anvil_ci::claim_next(&app, &name, &info.platform).await {
128	128		return Ok(axum::Json(job).into_response());
129	129		}
130		-	app.jobs.wait_for_work(CLAIM_POLL).await;
	130	+	app.jobs.wait_for_work(anvil_job::CLAIM_POLL).await;
131	131		match anvil_ci::claim_next(&app, &name, &info.platform).await {
132	132		Some(job) => Ok(axum::Json(job).into_response()),
133	133		None => Ok(StatusCode::NO_CONTENT.into_response()),
134	134		}
135	135		}
136	136
137		-	/// Matches the runner's own poll window; it gives up a shade later.
138		-	const CLAIM_POLL: std::time::Duration = std::time::Duration::from_secs(55);
139		-
140	137		/// The checkout for a claimed job, as an uncompressed tar.
141	138		async fn checkout(
142	139		State(app): State<App>,
⋯ 19 unchanged lines
162	159		let name = authenticate(&app, &headers, &info)?;
163	160		// A runner mid-build is not claiming, so its heartbeats are the only thing
164	161		// keeping it visible to platform routing while a long job runs.
165		-	app.jobs.seen(&name, &info.platform);
	162	+	app.jobs.seen(&name, &info.platform, &info.version);
166	163		if app.jobs.heartbeat(run_id, &name) {
167	164		Ok(StatusCode::NO_CONTENT.into_response())
168	165		} else {
⋯ 66 unchanged lines

⋯ 487 unchanged lines
488	488		summary { (u.username) }
489	489		div.nav-dropdown {
490	490		a href="/-/settings" { "Settings" }
491		-	@if u.is_admin { a href="/-/admin/usage" { "Disk usage" } }
	491	+	@if u.is_admin {
	492	+	a href="/-/admin/usage" { "Disk usage" }
	493	+	a href="/-/admin/runners" { "CI runners" }
	494	+	}
492	495		// Unboosted for the same reason as the
493	496		// SSO sign-in button: signing out of a
494	497		// provider-linked account redirects to
⋯ 2502 unchanged lines

⋯ 64 unchanged lines
65	65		inner: Arc<Inner>,
66	66		}
67	67
	68	+	/// What a runner has told us about itself, and when it last did.
	69	+	///
	70	+	/// The platform and version are self-asserted and not checked — everyone
	71	+	/// holding the token is one principal, so a runner that lies about its
	72	+	/// architecture is only lying to itself about which jobs it gets.
	73	+	struct Presence {
	74	+	platform: String,
	75	+	version: String,
	76	+	/// First contact since this process started. Not the runner's own uptime:
	77	+	/// an anvild restart resets it, and the page says as much.
	78	+	first_seen: Instant,
	79	+	last_seen: Instant,
	80	+	}
	81	+
	82	+	/// One runner as [`Dispatch::runners`] reports it: what it says it is, how
	83	+	/// stale that is, and what it is holding right now.
	84	+	#[derive(Clone, Debug)]
	85	+	pub struct RunnerStatus {
	86	+	pub name: String,
	87	+	/// Empty if the runner has only ever reached endpoints that identify it by
	88	+	/// header (no JSON body to carry a platform).
	89	+	pub platform: String,
	90	+	pub version: String,
	91	+	/// Since its last claim or heartbeat. An idle runner refreshes this once
	92	+	/// per claim poll (55s) and a busy one every [`HEARTBEAT_INTERVAL`], so
	93	+	/// anything much past a minute means it stopped talking.
	94	+	pub last_seen: Duration,
	95	+	/// Since anvild first heard from it, reset by an anvild restart.
	96	+	pub connected_for: Duration,
	97	+	/// Runs it currently holds a lease on.
	98	+	pub running: Vec<i64>,
	99	+	}
	100	+
68	101		#[derive(Default)]
69	102		struct Inner {
70	103		leases: Mutex<HashMap<i64, Lease>>,
71		-	/// Runner name → (advertised platform, last seen). What a runner says it
72		-	/// is; not a credential and not checked — everyone holding the token is
73		-	/// one principal, so a runner that lies about its architecture is only
74		-	/// lying to itself about which jobs it gets.
75		-	runners: Mutex<HashMap<String, (String, Instant)>>,
	104	+	/// Runner name → what it last told us. Presence is derived from ordinary
	105	+	/// traffic (claims and heartbeats) rather than a dedicated ping: a runner
	106	+	/// that is not polling for work or beating for a job is not a runner
	107	+	/// anything should be dispatched to, whatever a ping would claim.
	108	+	runners: Mutex<HashMap<String, Presence>>,
76	109		wake: tokio::sync::Notify,
77	110		/// Serializes claim attempts. Reading the queue and marking a run
78	111		/// `running` are separate awaits, so two runners polling at once could
⋯ 40 unchanged lines
119	152		);
120	153		}
121	154
122		-	/// Note that `runner` is alive and what platform it says it is. Called on
123		-	/// every claim and heartbeat, which is what keeps [`platforms`] honest
124		-	/// about a busy runner as well as an idle one.
	155	+	/// Note that `runner` is alive, and what it says it is. Called on every
	156	+	/// claim and heartbeat, which is what keeps [`platforms`] honest about a
	157	+	/// busy runner as well as an idle one.
	158	+	///
	159	+	/// An empty `platform` or `version` means "not stated on this request"
	160	+	/// rather than "unknown": the endpoints with no JSON body identify their
	161	+	/// caller by header alone, and must not blank out what a claim established.
125	162		///
126	163		/// [`platforms`]: Dispatch::platforms
127		-	pub fn seen(&self, runner: &str, platform: &str) {
128		-	if platform.is_empty() {
129		-	return;
	164	+	pub fn seen(&self, runner: &str, platform: &str, version: &str) {
	165	+	let now = Instant::now();
	166	+	let mut runners = self.inner.runners.lock().unwrap();
	167	+	let entry = runners
	168	+	.entry(runner.to_string())
	169	+	.or_insert_with(\|\| Presence {
	170	+	platform: String::new(),
	171	+	version: String::new(),
	172	+	first_seen: now,
	173	+	last_seen: now,
	174	+	});
	175	+	if !platform.is_empty() {
	176	+	entry.platform = platform.to_string();
	177	+	}
	178	+	if !version.is_empty() {
	179	+	entry.version = version.to_string();
130	180		}
131		-	self.inner
132		-	.runners
133		-	.lock()
134		-	.unwrap()
135		-	.insert(runner.to_string(), (platform.to_string(), Instant::now()));
	181	+	entry.last_seen = now;
136	182		}
137	183
	184	+	/// Drop everyone past [`RUNNER_TTL`] and hand back the rest. Pruned on read
	185	+	/// rather than on a timer: presence is only ever consulted here.
	186	+	fn live(&self) -> std::sync::MutexGuard<'_, HashMap<String, Presence>> {
	187	+	let now = Instant::now();
	188	+	let mut runners = self.inner.runners.lock().unwrap();
	189	+	runners.retain(\|_, p\| now.duration_since(p.last_seen) < RUNNER_TTL);
	190	+	runners
	191	+	}
	192	+
138	193		/// Platforms with a runner behind them right now (within [`RUNNER_TTL`]).
139	194		///
140	195		/// The dispatcher's answer to "is there anyone who could run this
141	196		/// natively?" — if not, an emulating runner may take the job rather than
142		-	/// leaving it queued forever.
	197	+	/// leaving it queued forever. A runner that has stated no platform is
	198	+	/// present but is not evidence of any architecture, so it is skipped here:
	199	+	/// counting it would strand every job behind a fallback that never fires.
143	200		pub fn platforms(&self) -> Vec<String> {
144		-	let now = Instant::now();
145		-	let mut runners = self.inner.runners.lock().unwrap();
146		-	runners.retain(\|_, (_, seen)\| now.duration_since(*seen) < RUNNER_TTL);
147		-	let mut platforms: Vec<String> = runners.values().map(\|(p, _)\| p.clone()).collect();
	201	+	let mut platforms: Vec<String> = self
	202	+	.live()
	203	+	.values()
	204	+	.filter(\|p\| !p.platform.is_empty())
	205	+	.map(\|p\| p.platform.clone())
	206	+	.collect();
148	207		platforms.sort();
149	208		platforms.dedup();
150	209		platforms
151	210		}
152	211
153		-	/// Every runner seen within [`RUNNER_TTL`], as (name, platform), for the
154		-	/// admin view and for logging.
155		-	pub fn runners(&self) -> Vec<(String, String)> {
	212	+	/// Every runner seen within [`RUNNER_TTL`], for the admin view and for
	213	+	/// logging. Sorted by name, so the page does not reshuffle between polls.
	214	+	pub fn runners(&self) -> Vec<RunnerStatus> {
	215	+	// Leases first and released before the presence lock is taken: the two
	216	+	// are never held together anywhere, which is what keeps them from
	217	+	// needing an ordering rule.
	218	+	let mut running: HashMap<String, Vec<i64>> = HashMap::new();
	219	+	for (run_id, runner) in self.in_flight() {
	220	+	running.entry(runner).or_default().push(run_id);
	221	+	}
	222	+
156	223		let now = Instant::now();
157		-	let mut runners = self.inner.runners.lock().unwrap();
158		-	runners.retain(\|_, (_, seen)\| now.duration_since(*seen) < RUNNER_TTL);
159		-	let mut out: Vec<(String, String)> = runners
	224	+	let mut out: Vec<RunnerStatus> = self
	225	+	.live()
160	226		.iter()
161		-	.map(\|(name, (platform, _))\| (name.clone(), platform.clone()))
	227	+	.map(\|(name, p)\| RunnerStatus {
	228	+	name: name.clone(),
	229	+	platform: p.platform.clone(),
	230	+	version: p.version.clone(),
	231	+	last_seen: now.duration_since(p.last_seen),
	232	+	connected_for: now.duration_since(p.first_seen),
	233	+	running: {
	234	+	let mut ids = running.remove(name).unwrap_or_default();
	235	+	ids.sort_unstable();
	236	+	ids
	237	+	},
	238	+	})
162	239		.collect();
163		-	out.sort();
	240	+	out.sort_by(\|a, b\| a.name.cmp(&b.name));
164	241		out
165	242		}
166	243
⋯ 83 unchanged lines
250	327		let jobs = Dispatch::new();
251	328		assert!(jobs.platforms().is_empty());
252	329
253		-	jobs.seen("mac", "linux/arm64");
254		-	jobs.seen("droplet", "linux/amd64");
255		-	jobs.seen("laptop", "linux/arm64");
	330	+	jobs.seen("mac", "linux/arm64", "0.1.0");
	331	+	jobs.seen("droplet", "linux/amd64", "0.1.0");
	332	+	jobs.seen("laptop", "linux/arm64", "0.1.0");
256	333		assert_eq!(jobs.platforms(), vec!["linux/amd64", "linux/arm64"]);
257	334		assert_eq!(jobs.runners().len(), 3);
258	335
259		-	jobs.seen("mac", "linux/amd64"); // same name, rebuilt as amd64
	336	+	jobs.seen("mac", "linux/amd64", "0.1.0"); // same name, rebuilt as amd64
260	337		assert_eq!(jobs.runners().len(), 3);
261	338		assert_eq!(jobs.platforms(), vec!["linux/amd64", "linux/arm64"]);
262	339
263		-	// A runner that advertises nothing is not evidence of any platform,
264		-	// and must not register as one — routing would then never fall back.
265		-	jobs.seen("mystery", "");
266		-	assert_eq!(jobs.runners().len(), 3);
	340	+	// A runner that advertises nothing is present — the status page should
	341	+	// show it — but is not evidence of any platform, and must not register
	342	+	// as one: routing would then never fall back to an emulating runner.
	343	+	jobs.seen("mystery", "", "");
	344	+	assert_eq!(jobs.runners().len(), 4);
	345	+	assert_eq!(jobs.platforms(), vec!["linux/amd64", "linux/arm64"]);
	346	+
	347	+	// A later request that states nothing (the checkout GET and the
	348	+	// artifact/result POSTs identify their caller by header alone) is a
	349	+	// liveness signal, not an erasure of what the claim established.
	350	+	jobs.seen("mac", "", "");
	351	+	let mac = jobs
	352	+	.runners()
	353	+	.into_iter()
	354	+	.find(\|r\| r.name == "mac")
	355	+	.expect("mac is present");
	356	+	assert_eq!(mac.platform, "linux/amd64");
	357	+	assert_eq!(mac.version, "0.1.0");
	358	+	}
	359	+
	360	+	/// The status page's other column: who is holding a run right now.
	361	+	#[test]
	362	+	fn runners_report_the_leases_they_hold() {
	363	+	let jobs = Dispatch::new();
	364	+	jobs.seen("mac", "linux/arm64", "0.1.0");
	365	+	jobs.seen("droplet", "linux/amd64", "0.1.0");
	366	+	jobs.claim(7, "mac", vec![]);
	367	+	jobs.claim(9, "mac", vec![]);
	368	+
	369	+	let by_name: HashMap<String, RunnerStatus> = jobs
	370	+	.runners()
	371	+	.into_iter()
	372	+	.map(\|r\| (r.name.clone(), r))
	373	+	.collect();
	374	+	assert_eq!(by_name["mac"].running, vec![7, 9]);
	375	+	assert!(by_name["droplet"].running.is_empty());
	376	+
	377	+	jobs.release(7);
	378	+	jobs.release(9);
	379	+	assert!(jobs.runners().iter().all(\|r\| r.running.is_empty()));
267	380		}
268	381		}

1	1		//! Admin-only dashboard pages (site-level), gated by `User.is_admin`.
2	2
	3	+	use std::time::Duration;
	4	+
3	5		use anvil_core::{
4	6		App,
	7	+	ci,
	8	+	jobs::{
	9	+	RUNNER_TTL,
	10	+	RunnerStatus,
	11	+	},
	12	+	repos,
5	13		usage,
	14	+	users,
6	15		};
7	16		use axum::{
8	17		Router,
⋯ 6 unchanged lines
15	24		};
16	25		use maud::{
17	26		Markup,
	27	+	PreEscaped,
18	28		html,
19	29		};
20	30
⋯ 8 unchanged lines
29	39		};
30	40
31	41		pub fn routes(router: Router<App>) -> Router<App> {
32		-	router.route("/-/admin/usage", get(usage_page))
	42	+	router
	43	+	.route("/-/admin/usage", get(usage_page))
	44	+	.route("/-/admin/runners", get(runners_page))
33	45		}
34	46
35	47		/// `GET /-/admin/usage` — site-wide disk usage by user and content type.
⋯ 63 unchanged lines
99	111		}
100	112		}
101	113		}
	114	+
	115	+	/// When a present runner starts reading as late.
	116	+	///
	117	+	/// Liveness is not a dedicated ping: it is the ordinary traffic a working
	118	+	/// runner already generates. An idle one refreshes itself once per parked
	119	+	/// claim ([`anvil_job::CLAIM_POLL`]), a busy one every
	120	+	/// [`HEARTBEAT_INTERVAL`](anvil_job::HEARTBEAT_INTERVAL). So a runner has to
	121	+	/// have missed a whole poll window and a heartbeat before silence means
	122	+	/// anything, and this is that sum. Between here and `RUNNER_TTL` a runner is
	123	+	/// still routed to — the page says "late", not "gone", because that is
	124	+	/// precisely what the dispatcher still believes.
	125	+	const STALE_AFTER: Duration =
	126	+	Duration::from_secs(anvil_job::CLAIM_POLL.as_secs() + anvil_job::HEARTBEAT_INTERVAL.as_secs());
	127	+
	128	+	/// How often the page reloads itself. Comfortably under `STALE_AFTER`, so a
	129	+	/// runner that goes quiet is visible as late without anyone pressing reload.
	130	+	const REFRESH_SECS: u64 = 15;
	131	+
	132	+	const REFRESH_JS: &str = "setTimeout(function(){ location.reload(); }, 15000);";
	133	+
	134	+	/// `GET /-/admin/runners` — the CI runners currently dialled in.
	135	+	///
	136	+	/// Non-admins (including anonymous) get a 404, same as the usage page: the
	137	+	/// list names the machines that build this instance's code and how idle they
	138	+	/// are, which is not something to leak by letting the route 403.
	139	+	async fn runners_page(State(app): State<App>, CurrentUser(user): CurrentUser) -> Response {
	140	+	if !user.as_ref().is_some_and(\|u\| u.is_admin) {
	141	+	return not_found("not found");
	142	+	}
	143	+
	144	+	let runners = app.jobs.runners();
	145	+	// Queued runs are the other half of the diagnosis: runners but no queue is
	146	+	// a healthy idle instance, a queue but no runners is a stuck one.
	147	+	let queued = ci::queued_ids(&app.db).await.unwrap_or_default();
	148	+
	149	+	// In-flight runs are keyed by id alone (the lease knows nothing of repos),
	150	+	// so resolve each to its page. A handful at most — one per busy runner.
	151	+	let mut links: Vec<(i64, String)> = Vec::new();
	152	+	for id in runners.iter().flat_map(\|r\| r.running.iter().copied()) {
	153	+	if let Some(href) = run_href(&app, id).await {
	154	+	links.push((id, href));
	155	+	}
	156	+	}
	157	+
	158	+	layout(
	159	+	"CI runners",
	160	+	user.as_ref(),
	161	+	render_runners(
	162	+	&runners,
	163	+	queued.len(),
	164	+	!app.config.ci.runner_token.is_empty(),
	165	+	&links,
	166	+	),
	167	+	)
	168	+	.into_response()
	169	+	}
	170	+
	171	+	/// The canonical page for a run id, or `None` if any part of the chain is
	172	+	/// missing (a run deleted mid-flight, most plausibly).
	173	+	async fn run_href(app: &App, run_id: i64) -> Option<String> {
	174	+	let run = ci::get(&app.db, run_id).await.ok()??;
	175	+	let repo = repos::find_by_id(&app.db, run.repo_id).await.ok()??;
	176	+	let owner = users::find_by_id(&app.db, repo.owner_id).await.ok()??;
	177	+	Some(format!("/{}/{}/ci/{run_id}", owner.username, repo.name))
	178	+	}
	179	+
	180	+	fn render_runners(
	181	+	runners: &[RunnerStatus],
	182	+	queued: usize,
	183	+	token_set: bool,
	184	+	links: &[(i64, String)],
	185	+	) -> Markup {
	186	+	html! {
	187	+	h1 { "CI runners" }
	188	+	p.muted {
	189	+	"anvil dispatches CI but does not execute it: runners dial in, claim jobs "
	190	+	"and run them on their own Docker daemon. This is who has been in touch "
	191	+	"within " (fmt_span(RUNNER_TTL)) " — the window the dispatcher itself "
	192	+	"treats as connected. Reloads every " (REFRESH_SECS) "s."
	193	+	}
	194	+
	195	+	@if !token_set {
	196	+	p.error-msg {
	197	+	"[ci] runner_token is empty, so every claim is refused with a 503 and "
	198	+	"no runner can connect at all. Set it in the config and restart."
	199	+	}
	200	+	}
	201	+
	202	+	@if runners.is_empty() {
	203	+	p.muted {
	204	+	"No runner connected. "
	205	+	@if queued > 0 {
	206	+	b { (queued) " run" @if queued != 1 { "s" } " queued" }
	207	+	" and nothing to run " @if queued == 1 { "it" } @else { "them" } "."
	208	+	} @else {
	209	+	"Nothing is queued, so nothing is stuck yet — but the next push has nowhere to go."
	210	+	}
	211	+	}
	212	+	} @else {
	213	+	table.usage {
	214	+	thead { tr {
	215	+	th { "Runner" }
	216	+	th { "Platform" }
	217	+	th { "Version" }
	218	+	th { "Last seen" }
	219	+	th { "Connected" }
	220	+	th { "Running" }
	221	+	} }
	222	+	tbody {
	223	+	@for r in runners {
	224	+	tr {
	225	+	td {
	226	+	(r.name)
	227	+	" "
	228	+	@if r.last_seen >= STALE_AFTER {
	229	+	span class="st failure" { "late" }
	230	+	} @else if r.running.is_empty() {
	231	+	span class="st success" { "idle" }
	232	+	} @else {
	233	+	span class="st running" { "busy" }
	234	+	}
	235	+	}
	236	+	td { @if r.platform.is_empty() { span.muted { "unstated" } } @else { code { (r.platform) } } }
	237	+	td { @if r.version.is_empty() { span.muted { "—" } } @else { (r.version) } }
	238	+	td.num { (fmt_span(r.last_seen)) " ago" }
	239	+	td.num { (fmt_span(r.connected_for)) }
	240	+	td {
	241	+	@if r.running.is_empty() {
	242	+	span.muted { "—" }
	243	+	} @else {
	244	+	@for id in &r.running {
	245	+	@match links.iter().find(\|(i, _)\| i == id) {
	246	+	Some((_, href)) => { a href=(href) { "#" (id) } " " }
	247	+	None => { span { "#" (id) } " " }
	248	+	}
	249	+	}
	250	+	}
	251	+	}
	252	+	}
	253	+	}
	254	+	}
	255	+	}
	256	+	p.muted {
	257	+	(queued) " run" @if queued != 1 { "s" } " queued."
	258	+	@if queued > 0 && runners.iter().all(\|r\| !r.running.is_empty()) {
	259	+	" Every runner is busy, so the queue is waiting on capacity rather than on a missing runner."
	260	+	}
	261	+	}
	262	+	}
	263	+
	264	+	p.muted {
	265	+	"\"Connected\" counts from this process's first contact, so an anvild "
	266	+	"restart resets it — it is not the runner's own uptime."
	267	+	}
	268	+	script { (PreEscaped(REFRESH_JS)) }
	269	+	}
	270	+	}
	271	+
	272	+	/// A duration as a short span (`4s`, `3m`, `2h 5m`). Deliberately finer than
	273	+	/// `fmt_relative`, whose floor is "just now": the whole point of the last-seen
	274	+	/// column is telling 4 seconds from 90.
	275	+	fn fmt_span(d: Duration) -> String {
	276	+	// A zero remainder is noise, not precision: the TTL should read "5m", not
	277	+	// "5m 0s".
	278	+	let pair = \|big: u64, bu: &str, small: u64, su: &str\| match small {
	279	+	0 => format!("{big}{bu}"),
	280	+	_ => format!("{big}{bu} {small}{su}"),
	281	+	};
	282	+	match d.as_secs() {
	283	+	s if s < 60 => format!("{s}s"),
	284	+	s if s < 3600 => pair(s / 60, "m", s % 60, "s"),
	285	+	s if s < 86_400 => pair(s / 3600, "h", (s % 3600) / 60, "m"),
	286	+	s => pair(s / 86_400, "d", (s % 86_400) / 3600, "h"),
	287	+	}
	288	+	}
	289	+
	290	+	#[cfg(test)]
	291	+	mod tests {
	292	+	use super::*;
	293	+
	294	+	#[test]
	295	+	fn spans_read_as_ages() {
	296	+	assert_eq!(fmt_span(Duration::from_secs(0)), "0s");
	297	+	assert_eq!(fmt_span(Duration::from_secs(31)), "31s");
	298	+	assert_eq!(fmt_span(Duration::from_secs(86)), "1m 26s");
	299	+	assert_eq!(fmt_span(RUNNER_TTL), "5m");
	300	+	assert_eq!(fmt_span(Duration::from_secs(7_500)), "2h 5m");
	301	+	assert_eq!(fmt_span(Duration::from_secs(90_000)), "1d 1h");
	302	+	}
	303	+
	304	+	/// The threshold has to sit above a full idle cycle, or every runner that
	305	+	/// is simply parked in a claim reads as late.
	306	+	#[test]
	307	+	fn stale_after_allows_a_whole_claim_poll() {
	308	+	assert!(STALE_AFTER > anvil_job::CLAIM_POLL);
	309	+	assert!(STALE_AFTER < RUNNER_TTL);
	310	+	}
	311	+	}

⋯ 246 unchanged lines
247	247		platform: linux/amd64 (emulated on linux/arm64)
248	248		```
249	249
	250	+	### Seeing who is connected
	251	+
	252	+	`/-/admin/runners` (admin-only, 404 for everyone else) lists every runner
	253	+	`Dispatch` still counts as present: name, advertised platform, worker version,
	254	+	how long since it last spoke, how long it has been connected, and the runs it
	255	+	holds right now, each linked to its CI page. Underneath it is the same map
	256	+	routing reads, so the page and the dispatcher can never disagree about who is
	257	+	out there. It also prints the queue depth, because the two together are the
	258	+	whole diagnosis: runners and no queue is a healthy idle instance, a queue and
	259	+	no runners is a stuck one, and a queue with every runner busy is neither — it
	260	+	is capacity.
	261	+
	262	+	There is no separate heartbeat for presence, on purpose. Liveness is the
	263	+	traffic a working runner already generates: an idle one re-registers itself
	264	+	every time its parked claim expires and it dials back in (`CLAIM_POLL`, 55s),
	265	+	and a busy one every `HEARTBEAT_INTERVAL` (30s) for as long as its job runs.
	266	+	Between them there is no state a runner can be in where it is useful and
	267	+	silent, so a dedicated ping would only add a way for a runner to look alive
	268	+	while claiming nothing.
	269	+
	270	+	What that costs is resolution, and the page is explicit about it rather than
	271	+	hiding it. A runner is shown "late" once it has been quiet for a whole claim
	272	+	poll plus a heartbeat (85s) — long enough that a runner merely parked in a
	273	+	long poll never reads as late. It stays listed, and keeps being routed to,
	274	+	until `RUNNER_TTL` (5 min), because that is exactly what the dispatcher still
	275	+	believes; the page's job is to show that belief, not to invent a second one.
	276	+	So a machine that loses power disappears from routing in up to five minutes and
	277	+	reads as late within ninety seconds. The page reloads itself every 15s.
	278	+
250	279		## Isolation on macOS
251	280
252	281		Docker on macOS is a Linux VM (LinuxKit under Docker Desktop, Lima under
⋯ 172 unchanged lines