collin/anvil · 47826aaf

Run two CI runners from the dev compose override

Collin Richards · 2026-08-24 09:52 UTC · 47826aafbbc9fb69e85c2509d3983e702529ce0c · parent 7ddd780b · browse files

modified.gitignore+2 −0

modifiedCLAUDE.md+4 −1

modifiedCargo.lock+1 −0

modifiedcompose.override.yaml+67 −3

modifiedcrates/anvil-worker/Cargo.toml+1 −0

modifiedcrates/anvil-worker/src/client.rs+12 −0

modifieddeploy/anvil.dev.toml+6 −0

modifieddeploy/build.sh+20 −9

addeddeploy/worker/Dockerfile+31 −0

modifieddocs/remote-runners.md+27 −0

⋯ 4 unchanged lines
5	5		*.db-shm
6	6		anvil.toml
7	7		/deploy/anvild
	8	+	# Staged by `./deploy/build.sh --worker` for the local runner containers.
	9	+	/deploy/anvil-worker
8	10		# CA bundle deploy/dev.sh builds so the dev container trusts portless (docs/oidc.md).
9	11		/deploy/dev-ca.crt
10	12		# Local-only API credentials for fetching attachments (never committed).
⋯ 1 unchanged line

⋯ 112 unchanged lines
113	113		runner_token` is set; queued runs just sit, with a warning logged at startup.
114	114		Setting one up on the Mac mini (build, Rosetta, the launchd agent in
115	115		`deploy/worker/`) is written out in `docs/remote-runners.md` § Isolation on
116		-	macOS.
	116	+	macOS. Locally, `compose.override.yaml` runs two of them in containers
	117	+	(`./deploy/build.sh --debug --worker`, then `docker compose up -d --build`) so
	118	+	concurrency and routing are testable on one machine; that mount is a
	119	+	development-only concession, never production's.
117	120		Agent sessions still drive Docker locally and are the one thing the dropped
118	121		socket mount gives up. They are off by default.
119	122
⋯ 3 unchanged lines

⋯ 294 unchanged lines
295	295		"clap",
296	296		"futures-util",
297	297		"reqwest",
	298	+	"rustls",
298	299		"serde_json",
299	300		"tar",
300	301		"tokio",
⋯ 5518 unchanged lines

⋯ 3 unchanged lines
4	4		# the SSO issuer is a public HTTPS URL with a normal CA, and there is
5	5		# deliberately no Docker socket.
6	6		#
7		-	# The image carries a prebuilt binary, so stage one first (--debug compiles in
8		-	# a fraction of the time a release build takes):
	7	+	# The images carry prebuilt binaries, so stage them first (--debug compiles in
	8	+	# a fraction of the time a release build takes; --worker adds anvil-worker for
	9	+	# the two runner containers below):
9	10		#
10		-	# ./deploy/build.sh --debug
	11	+	# ./deploy/build.sh --debug --worker
11	12		# docker compose up -d --build
12	13		#
13	14		# Then http://127.0.0.1:20640. `docker compose logs -f`, `docker compose down`.
14	15		#
15	16		# deploy/dev.sh remains the fuller path: it also generates deploy/dev-ca.crt
16	17		# (mounted below), waits for /-/healthz, and points portless at the container.
	18	+
	19	+	# Two CI runners, identical but for their names. anvil executes no jobs itself
	20	+	# (docs/remote-runners.md), so without these a queued run sits forever; two of
	21	+	# them rather than one is what makes concurrent pipelines and the "is any
	22	+	# runner connected" side of platform routing testable on one machine.
	23	+	#
	24	+	# LOCAL ONLY, and the reason is worth being explicit about: a containerized
	25	+	# runner needs the host's Docker socket, which is the root-equivalent hold that
	26	+	# moving CI off the forge removed. Real runners are native processes on their
	27	+	# own host (docs/remote-runners.md § Isolation on macOS). This file already
	28	+	# hands anvil the same socket for agent sessions, so the local trust boundary
	29	+	# is unchanged -- the deployed compose.yaml grants neither.
	30	+	x-runner: &runner
	31	+	image: anvil-worker-dev:latest
	32	+	build:
	33	+	context: .
	34	+	dockerfile: deploy/worker/Dockerfile
	35	+	platforms:
	36	+	- linux/amd64
	37	+	restart: unless-stopped
	38	+	depends_on:
	39	+	- anvil
	40	+	environment: &runner-env
	41	+	# Container-to-container over the hagrid network, by compose service name:
	42	+	# ANVIL_BASE_URL is what browsers use and does not resolve in here.
	43	+	ANVIL_URL: ${ANVIL_RUNNER_URL:-http://anvil:3000}
	44	+	# Must match `[ci] runner_token` baked in from deploy/anvil.dev.toml.
	45	+	ANVIL_RUNNER_TOKEN: ${ANVIL_RUNNER_TOKEN:-dev-runner-token}
	46	+	RUST_LOG: ${ANVIL_RUNNER_LOG:-anvil_worker=info}
	47	+	volumes:
	48	+	# The runner is a Docker client: it creates each job's sandbox as a sibling
	49	+	# container on this host's daemon. The JOB container still gets no socket
	50	+	# and no mounts -- the checkout is uploaded and artifacts downloaded through
	51	+	# the API (crates/anvil-worker/src/executor.rs).
	52	+	#
	53	+	# `label=disable` rather than a `:z` relabel, same as anvil above: :z would
	54	+	# rewrite the SELinux label on the host's socket, which every other
	55	+	# container on this machine also uses.
	56	+	- /var/run/docker.sock:/var/run/docker.sock
	57	+	group_add:
	58	+	- "${DOCKER_GID:-970}"
	59	+	security_opt:
	60	+	- label=disable
	61	+	networks:
	62	+	- hagrid
	63	+
17	64		services:
18	65		anvil:
19	66		# A distinct tag, so a local build carrying the DEV config can never be
⋯ 64 unchanged lines
84	131		# matches exactly.
85	132		ANVIL_BASE_URL: ${ANVIL_BASE_URL:-https://anvil.localhost}
86	133
	134	+	# A third is four lines: copy one of these and bump the name. The names are
	135	+	# what run headers and `[ci]` logs show, so keep them distinct -- an unnamed
	136	+	# runner falls back to its hostname, which in a container is a hex id.
	137	+	runner-1:
	138	+	<<: *runner
	139	+	container_name: anvil-runner-1
	140	+	environment:
	141	+	<<: *runner-env
	142	+	ANVIL_RUNNER_NAME: dev-1
	143	+
	144	+	runner-2:
	145	+	<<: *runner
	146	+	container_name: anvil-runner-2
	147	+	environment:
	148	+	<<: *runner-env
	149	+	ANVIL_RUNNER_NAME: dev-2
	150	+
87	151		volumes:
88	152		anvil-dev-data:
89	153		name: anvil-dev-data
⋯ 2 unchanged lines

⋯ 21 unchanged lines
22	22		tar.workspace = true
23	23		futures-util.workspace = true
24	24		reqwest.workspace = true
	25	+	rustls.workspace = true
25	26		serde_json.workspace = true
26	27		tokio.workspace = true
27	28		tracing.workspace = true
⋯ 1 unchanged line

⋯ 28 unchanged lines
29	29		}
30	30
31	31		impl Client {
	32	+	/// Build the client every request goes through.
	33	+	///
	34	+	/// The rustls ring provider is installed process-wide first: the
	35	+	/// workspace builds reqwest with `-no-provider` on purpose (aws-lc-rs
	36	+	/// needs cmake and breaks the zig musl cross-build), so a process that
	37	+	/// does not install one panics inside `build()` — before it has claimed
	38	+	/// anything, and identically on every host.
32	39		pub fn new(base: &str, token: &str, info: RunnerInfo) -> Result<Self, String> {
	40	+	static TLS_PROVIDER: std::sync::Once = std::sync::Once::new();
	41	+	TLS_PROVIDER.call_once(\|\| {
	42	+	let _ = rustls::crypto::ring::default_provider().install_default();
	43	+	});
	44	+
33	45		let http = reqwest::Client::builder()
34	46		.timeout(REQUEST_TIMEOUT)
35	47		.build()
⋯ 152 unchanged lines

⋯ 32 unchanged lines
33	33		clone_user = "git"
34	34
35	35		[ci]
	36	+	# The secret the local runners present (compose.override.yaml's `runner-1` and
	37	+	# `runner-2`). Committed like the rest of this file: it authenticates runners
	38	+	# to an instance that only listens on 127.0.0.1. Production's token lives in
	39	+	# the gitignored deploy/anvil.toml. Empty here would refuse every claim with a
	40	+	# 503 and leave queued runs sitting.
	41	+	runner_token = "dev-runner-token"
36	42		# The job container is a sibling on the host's Docker daemon, as in production.
37	43		memory_mb = 2048
38	44		cpus = 2.0
⋯ 18 unchanged lines

⋯ 8 unchanged lines
9	9		#
10	10		# ./deploy/build.sh release — what gets deployed
11	11		# ./deploy/build.sh --debug unoptimized, for the local compose.override.yaml
	12	+	# ./deploy/build.sh --worker also stage anvil-worker, for the local runners
	13	+	#
	14	+	# --worker is a local-development thing only: production runners are native
	15	+	# processes on the build host (docs/remote-runners.md), and nothing on hagrid
	16	+	# builds or runs the worker image.
12	17		#
13	18		# Prereqs (one-time):
14	19		# zig (dnf install zig, or brew install zig)
⋯ 4 unchanged lines
19	24		TARGET="x86_64-unknown-linux-musl"
20	25		PROFILE=release
21	26		CARGO_FLAGS=(--release)
	27	+	# Binary name → where the Dockerfile that COPYs it expects to find it.
	28	+	BINS=(anvild)
22	29
23		-	case "${1:-}" in
24		-	--debug) PROFILE=debug; CARGO_FLAGS=() ;;
25		-	"") ;;
26		-	*) echo "usage: $0 [--debug]" >&2; exit 64 ;;
27		-	esac
	30	+	for arg in "$@"; do
	31	+	case "$arg" in
	32	+	--debug) PROFILE=debug; CARGO_FLAGS=() ;;
	33	+	--worker) BINS+=(anvil-worker) ;;
	34	+	*) echo "usage: $0 [--debug] [--worker]" >&2; exit 64 ;;
	35	+	esac
	36	+	done
28	37
29	38		cd "$(dirname "$0")/.."
30	39
⋯ 4 unchanged lines
35	44
36	45		# Static musl even for --debug: the runtime image is Ubuntu, and a binary
37	46		# linked against Fedora's glibc would not run there.
38		-	echo "==> cross-compiling anvild for $TARGET ($PROFILE, native, via zig)"
39		-	cargo zigbuild --target "$TARGET" --bin anvild "${CARGO_FLAGS[@]}"
	47	+	for bin in "${BINS[@]}"; do
	48	+	echo "==> cross-compiling $bin for $TARGET ($PROFILE, native, via zig)"
	49	+	cargo zigbuild --target "$TARGET" --bin "$bin" "${CARGO_FLAGS[@]}"
40	50
41		-	echo "==> staging binary at deploy/anvild"
42		-	cp "target/$TARGET/$PROFILE/anvild" deploy/anvild
	51	+	echo "==> staging binary at deploy/$bin"
	52	+	cp "target/$TARGET/$PROFILE/$bin" "deploy/$bin"
	53	+	done
43	54
44	55		# Deliberately left in place: `docker compose build` runs after this and needs
45	56		# it in the context. It is gitignored, and the next build overwrites it.
⋯ 1 unchanged line

1	+	# anvil-worker image — LOCAL DEVELOPMENT ONLY.
2	+	#
3	+	# Production runners are native processes on their own host (a launchd agent on
4	+	# the Mac mini, see ../../docs/remote-runners.md § Isolation on macOS). This
5	+	# image exists so `compose.override.yaml` can bring up two runners next to the
6	+	# local forge and exercise concurrency and platform routing without a second
7	+	# machine. Do not deploy it: a containerized runner needs the host's Docker
8	+	# socket mounted in, which is the root-equivalent hold moving CI off the forge
9	+	# was meant to remove. That trade is already made locally — the dev compose
10	+	# file mounts the same socket into anvil for agent sessions.
11	+	#
12	+	# Same shape as ../../Dockerfile: no compilation here, just a COPY of the
13	+	# static x86_64-musl binary that `./deploy/build.sh --worker` stages.
14	+
15	+	FROM ubuntu:26.04
16	+
17	+	# ca-certificates so the claim loop can talk to an https:// forge. The local
18	+	# one is plain http over the compose network, but the image should not be the
19	+	# reason a runner cannot reach a real instance.
20	+	RUN apt-get update \
21	+	&& apt-get install -y --no-install-recommends ca-certificates \
22	+	&& rm -rf /var/lib/apt/lists/* \
23	+	&& useradd --system --user-group --home-dir /nonexistent worker
24	+
25	+	COPY deploy/anvil-worker /usr/local/bin/anvil-worker
26	+
27	+	# Unprivileged in the container; compose grants the docker group separately
28	+	# (`group_add`), which is the only host access the runner needs.
29	+	USER worker
30	+
31	+	ENTRYPOINT ["/usr/local/bin/anvil-worker"]

⋯ 305 unchanged lines
306	306		`--name` matters: the default reads `$HOSTNAME`, which launchd does not set, so
307	307		an unnamed runner is called `runner`.
308	308
	309	+	## Two runners locally
	310	+
	311	+	`compose.override.yaml` brings up `runner-1` and `runner-2` next to the local
	312	+	forge, so the parts of this design that only appear with more than one runner —
	313	+	concurrent pipelines, and the "no connected runner is native to this platform"
	314	+	half of routing — are testable without a second machine:
	315	+
	316	+	```sh
	317	+	./deploy/build.sh --debug --worker # stages deploy/anvild and deploy/anvil-worker
	318	+	docker compose up -d --build
	319	+	docker compose logs -f runner-1 runner-2
	320	+	# anvil-worker dev-1 (linux/amd64) → http://anvil:3000
	321	+	```
	322	+
	323	+	They reach the forge as `http://anvil:3000` over the compose network (the
	324	+	browser-facing `base_url` does not resolve inside a container) and authenticate
	325	+	with the `runner_token` committed in `deploy/anvil.dev.toml`. Both are amd64
	326	+	here, so to watch the fallback tier work, give a pipeline `platform:
	327	+	linux/arm64` and confirm it still gets claimed. Adding a third is a copy of the
	328	+	four-line service block with a new name.
	329	+
	330	+	These runners are containerized, and that is the one thing production must
	331	+	never copy: the socket mount is the root-equivalent hold that moving CI off the
	332	+	forge removed. It is acceptable locally only because the same file already
	333	+	mounts that socket into anvil for agent sessions, so the machine's trust
	334	+	boundary is unchanged. The deployed `compose.yaml` grants neither.
	335	+
309	336		## Two processes on the build host
310	337
311	338		Image building cannot be a CI job. Job containers get no Docker socket by
⋯ 86 unchanged lines