collin/anvil

main / DEPLOY.md

1	# Deploying anvil on hagrid
2
3	anvil runs as a single Docker container at `anvil.richardscollin.com`, fronted by
4	hagrid's Caddy reverse proxy.
5
6	## Deploying: `compose.yaml` + `hag`
7
8	`compose.yaml` at the repo root describes the deployment — the image, the
9	network, the volume, the published SSH port. Running it on hagrid is
10	[`hag`](https://anvil.richardscollin.com/collin/hagrid), the shared deploy tool
11	for every project on that host: it builds the image, pushes it to
12	`registry.vibe.richardscollin.com`, copies `compose.yaml` up, then pulls and
13	recreates the container there.
14
15	`compose.override.yaml` sits next to it for local development — loopback ports,
16	the dev config, the Docker socket agent sessions need. Compose merges it
17	automatically when you run `docker compose` in the repo root, and `hag` always
18	passes `-f compose.yaml` explicitly, so none of it ever reaches hagrid. See the
19	README for the local loop.
20
21	`hag deploy` is the whole deploy:
22
23	```sh
24	hag deploy # build, push, pull and recreate on hagrid
25	hag deploy -n # dry run: print every step, change nothing
26	```
27
28	`docker/Dockerfile` compiles anvild itself, from the build context, in a
29	builder stage that cross-links to a fully static `x86_64-unknown-linux-musl`
30	binary with cargo-zigbuild. The builder runs on your architecture
31	(`--platform=$BUILDPLATFORM`) and only the linker targets amd64, so an arm64
32	Mac produces hagrid's image at native speed rather than emulating rustc. The
33	runtime stage then `COPY --from=builder`s the binary onto `ubuntu:26.04`.
34
35	That the build context is the only input is the point: there is no staged
36	artifact in the working tree, so `hag deploy` cannot ship a binary older than
37	the commit it tags. It replaces the `deploy/build.sh` + `deploy/deploy.sh` pair
38	that used to sit in front of it.
39
40	Do not build on the VPS — a release build needs ~2–4 GB peak and OOMs a
41	cheap, swap-less droplet. The VPS never compiles anything; it only pulls.
42
43	One-time setup is now just the registry:
44
45	```sh
46	docker login registry.vibe.richardscollin.com # on this machine and hagrid
47	```
48
49	No host Rust toolchain, no `zig`, no `cargo-zigbuild` — the builder stage
50	carries all three, with zig pinned by version and sha256 and cargo-zigbuild
51	pinned by version. The Rust version still comes from `rust-toolchain.toml` and
52	nowhere else: the Dockerfile copies that file in ahead of the sources and lets
53	rustup materialize the pinned channel and the musl target, exactly as
54	`docker/runner/build.sh` reads it to bake the same toolchain into
55	`anvil-runner:rust`.
56
57	`Cargo.lock` is committed and the builder passes `--locked`, so the image is
58	built from the resolved dependency set the commit records — a build that would
59	need to change the lockfile fails instead of silently drifting.
60
61	Build caching. The builder mounts BuildKit caches over the cargo registry
62	and `target/`, so only what you changed recompiles. They are not part of any
63	layer, which is why the binaries are copied out to `/out` inside the same
64	`RUN`. `docker builder prune` throws these away and the next build is cold.
65
66	Every build is tagged twice: with the short git sha of the checkout (plus a
67	`-dirty` suffix when the working tree has uncommitted changes) and with
68	`latest`. The deploy pins the host to the exact sha it just pushed, so `hag
69	status` always names the running build. To roll back, run an older tag on the
70	host:
71
72	```sh
73	ssh hagrid 'cd compose/anvil && IMAGE_TAG=<sha> docker compose up -d --no-build'
74	```
75
76	Managing it afterwards needs no ssh and no remembering where it lives:
77
78	```sh
79	hag status # what the host is running
80	hag logs -f # its logs
81	hag restart # recreate its containers
82	hag down # stop it
83	hag config # what hag resolved for this project
84	hag ls # every compose project on the host
85	```
86
87	## 4. What `compose.yaml` runs
88
89	```yaml
90	image: registry.vibe.richardscollin.com/anvil:${IMAGE_TAG:-latest}
91	container_name: anvil
92	restart: unless-stopped
93	ports: ["165.232.162.167:22:2222"]
94	volumes: [anvil-data:/data]
95	networks: [hagrid]
96	```
97
98	- `networks: [hagrid]` — an external network, so Caddy can reach `anvil:3000`.
99	The web port is deliberately not published.
100	- `ports` — git-over-SSH, published on the droplet's default public IPv4 only.
101	The reserved IP (`137.184.249.48`) arrives on the anchor address `10.15.0.6`,
102	where the host's own sshd listens, so binding one specific IP keeps the two
103	off each other. `ANVIL_SSH_BIND_IP` / `ANVIL_SSH_PORT` in the host's `.env`
104	override it.
105	- `anvil-data:/data` — a named volume holding the SQLite DB, the bare repos and
106	the persistent SSH host key. Pinned with `name:` to that exact string, so
107	compose adopts the volume the pre-compose deploys created instead of deriving
108	a fresh empty `anvil_anvil-data` from the project name. A named volume (not a
109	host bind mount) keeps it owned by the in-container `anvil` user.
110	- No Docker socket. anvil does not execute CI — runners dial in and run
111	jobs on their own daemons (see [docs/remote-runners.md](docs/remote-runners.md)).
112	The container has no reason to reach Docker, so the mount is gone, and with
113	it the root-equivalent hold the internet-facing process used to have on the
114	host.
115
116	> Agent sessions are the exception. They still drive Docker locally, so
117	> turning them on means putting the socket back — a `volumes:` entry for
118	> `/var/run/docker.sock` and a matching `group_add:` — and accepting that the
119	> container again has root-equivalent control of the host. They are off by
120	> default and absent from `deploy/anvil.toml`; read `docs/untrusted-mode.md`
121	> §7 before changing that.
122
123	The baked config lives at `/etc/anvil/anvil.toml` (see `deploy/anvil.toml`).
124	Override it by bind-mounting your own file over that path.
125
126	Host-side config. `compose.yaml` loads `.env` from the project directory
127	on the host (`~/compose/anvil`) if it exists, and skips it if not — `hag` never
128	copies a local `.env` up, so the workstation and the host keep separate config.
129	Today that file carries the SSO client secret (§ [docs/oidc.md](docs/oidc.md)):
130
131	```sh
132	ssh hagrid 'printf "ANVIL_OIDC_CLIENT_SECRET=%s\n" \
133	"$(tr -d "[:space:]" < ~/.config/anvil/oidc-client-secret)" \
134	> ~/compose/anvil/.env && chmod 600 ~/compose/anvil/.env'
135	```
136
137	Leave the variable out entirely rather than setting it empty: empty would
138	override the baked config and turn a confidential OIDC client into a public one.
139
140	Migrating from the `docker run` deploys. Done on 2026-08-27, kept here
141	because the failure mode is not obvious. The old container was created by
142	`docker run`, so it carried no compose labels; compose read `container_name:
143	anvil` as taken by a foreign container and refused to adopt it. Clearing it
144	once (`ssh hagrid 'docker rm -f anvil'`) left the `anvil-data` volume untouched,
145	which is what the compose container picked up. `hag` also moved projects from
146	`~/<name>` to `~/compose/<name>` and refuses to deploy while state sits at the
147	old path — it prints the `mv` to run.
148
149	## 5. First run: create your account, key, and the repo
150
151	```sh
152	# admin user
153	docker exec anvil anvild -c /etc/anvil/anvil.toml \
154	user create collin --email you@example.com --password '<password>' --admin
155
156	# your SSH public key (so you can push over SSH)
157	docker exec -i anvil anvild -c /etc/anvil/anvil.toml \
158	user add-key collin --title laptop --key "$(cat ~/.ssh/id_ed25519.pub)"
159
160	# the anvil repo itself
161	docker exec anvil anvild -c /etc/anvil/anvil.toml repo create collin/anvil \
162	--description "a minimal git forge in Rust"
163	```
164
165	(You can also create the user, add keys, and create repos from the web UI once
166	signed in — the CLI is just convenient for the first admin.)
167
168	## 6. Self-host anvil on anvil
169
170	From your local anvil checkout:
171
172	```sh
173	git remote add origin ssh://git@anvil.richardscollin.com/collin/anvil.git
174	git push -u origin main
175	```
176
177	Then browse it at `https://anvil.richardscollin.com/collin/anvil`. HTTPS clone
178	also works: `git clone https://anvil.richardscollin.com/collin/anvil.git`
179	(pushes over HTTPS require your account password as the git password).
180
181	## 7. CI: start a runner
182
183	anvil dispatches CI; it does not execute it. Nothing runs until a runner dials
184	in, so this is a required step, not an optional one — see
185	[docs/remote-runners.md](docs/remote-runners.md).
186
187	Set a shared secret in the config (`[ci] runner_token`), then on a machine with
188	room to build — the Mac mini, not the droplet:
189
190	```sh
191	cargo build --release --bin anvil-worker
192	cp target/release/anvil-worker /usr/local/bin/
193
194	anvil-worker --url https://anvil.richardscollin.com \
195	--token "$ANVIL_RUNNER_TOKEN" --name macmini
196	```
197
198	`deploy/worker/com.anvil.worker.plist` runs it under launchd on macOS. Run it
199	natively there rather than in a container: a containerized runner needs the
200	Docker socket mounted into it, which rebuilds the hole section 4 just removed.
201
202	Isolation is not weaker for being on a Mac. Docker Desktop runs every container
203	inside one Linux VM, so `--cap-drop=ALL`, `no-new-privileges` and the
204	cgroup limits are enforced by the same kernel primitives as on Linux — with the
205	VM as an extra boundary a bare-metal Linux host does not have.
206
207	Architecture. The Mac is arm64 and hagrid is x86_64, so a job running
208	`cargo test` on the runner tests an architecture you do not ship. Images are
209	unaffected (the builder stage cross-compiles, and both it and `compose.yaml`
210	pin amd64), and per-pipeline platform selection is designed but not yet wired
211	to a config key.
212
213	## 8. The redeploy webhook (CD)
214
215	Any repo with a `.anvil/ci.toml` runs CI on push. A pipeline is just an image
216	plus steps:
217
218	```toml
219	image = "anvil-runner:rust"
220
221	[[steps]]
222	name = "test"
223	run = "cargo test --workspace"
224
225	[[steps]]
226	run = "cargo build --release"
227	```
228
229	Runs show up at `/{owner}/{repo}/ci`, with a per-commit status badge on the
230	commit list and a full log on each run's page.
231
232	Job sandbox. The runner is the only Docker client on its machine; the job
233	container gets no socket, no mounts (the checkout is uploaded as a tar), all
234	capabilities dropped, and `no-new-privileges`. Resource bounds come from the
235	forge's `[ci]` config and travel with each job, so tightening one does not need
236	runners redeployed: `memory_mb` (default 2048), `cpus` (2), `pids_limit` (512),
237	`timeout_secs` (1800, then the container is killed), `network` (true),
238	`run_as` (empty = image default), and `allowed_images` (empty = any; a tagless
239	entry like `"rust"` allows every tag). The allowlist is applied on the forge
240	when the job is built, so a runner cannot widen it. See
241	`docs/untrusted-mode.md` for the threat model and what this does/doesn't
242	protect against.
243
244	Secrets now leave the host. A pipeline's secrets are sent to the runner
245	with its job and sit in plaintext in a container on a machine anvil does not
246	own. Scope repository secrets accordingly, and treat a runner host as being as
247	trusted as the forge itself.
248
249	Continuous deployment is deliberately scoped to one repository. On a
250	successful run of `deploy_branch` (default `main`) in the repo named by
251	`[ci] deploy_repo`, anvil POSTs JSON to `[ci] deploy_webhook`:
252
253	```json
254	{ "repo": "collin/anvil", "ref": "main", "commit": "<oid>", "run_id": 42 }
255	```
256
257	No other repo can trigger this, even with passing CI. The webhook target is a
258	host-local plaintext receiver (HTTPS is unsupported, to keep the build
259	TLS-free) — typically a tiny script-runner on hagrid that, on a verified
260	request, runs the actual redeploy. Verify the `X-Anvil-Deploy-Secret` header
261	(set `[ci] deploy_secret`) before doing anything. Because anvil cross-compiles
262	(section 3), "redeploy anvil" usually means: the receiver pulls the freshly
263	built image and runs `docker compose up -d --no-build` in `~/anvil` — it does
264	not build in-place.
265
266	Building an image is not a CI job. Job containers get no Docker socket by
267	design, so `docker build`/`docker push` cannot happen inside a pipeline. That
268	work belongs to a separate deploy agent on the build host, triggered by this
269	webhook, running outside the sandbox with Docker access — a different trust
270	level from the runner, and deliberately a different process.
271
272	> The deploy receiver runs with whatever privileges you give it — keep it
273	> minimal, secret-gated, and bound to localhost / the Docker host gateway only.
274
275	## Operations
276
277	- Update: `hag deploy` (compile + push + restart in one command); or
278	`hag restart` to recreate the container from the image already on the host
279	(the `anvil-data` volume persists). NOTE: adding new DB tables in a
280	future version won't auto-apply to an existing database yet (Toasty migration
281	support is pending) — the git repos on disk are unaffected, but repo metadata
282	in SQLite may need recreating until migrations land.
283	- Backup: snapshot the `anvil-data` volume, e.g.
284	`docker run --rm -v anvil-data:/data -v "$PWD":/out debian:bookworm-slim \
285	tar czf /out/anvil-data.tgz -C /data .`
286	- Logs: `hag logs -f` (or `docker logs -f anvil` on the host).
287	- System routes live under `/-/` (e.g. sign in at
288	`https://anvil.richardscollin.com/-/login`); `/{username}` is the user/repo
289	namespace.