collin/anvil

ef6faf54bd5ff32b7b205125e8544b2994a00ec7 / DEPLOY.md

1	# Deploying anvil on hagrid
2
3	anvil runs as a single Docker container at `anvil.richardscollin.com`, fronted by
4	hagrid's Caddy reverse proxy.
5
6	## Deploying: `compose.yaml` + `hag`
7
8	`compose.yaml` at the repo root describes the deployment — the image, the
9	network, the volume, the published SSH port. Running it on hagrid is
10	[`hag`](https://anvil.richardscollin.com/collin/hagrid), the shared deploy tool
11	for every project on that host: it builds the image, pushes it to
12	`registry.vibe.richardscollin.com`, copies `compose.yaml` up, then pulls and
13	recreates the container there.
14
15	`compose.override.yaml` sits next to it for local development — loopback ports,
16	the dev config, the Docker socket agent sessions need. Compose merges it
17	automatically when you run `docker compose` in the repo root, and `hag` always
18	passes `-f compose.yaml` explicitly, so none of it ever reaches hagrid. See the
19	README for the local loop.
20
21	anvil needs one step in front of that, because the image carries a prebuilt
22	binary — `deploy/deploy.sh` is the whole deploy:
23
24	```sh
25	./deploy/deploy.sh # cross-compile, build, push, restart on hagrid
26	./deploy/deploy.sh -n # dry run: print every step, change nothing
27	```
28
29	which is:
30
31	1. `deploy/build.sh` — `cargo zigbuild --release --target
32	x86_64-unknown-linux-musl` cross-compiles a fully static binary natively
33	(~2 min, no emulation) and stages it at `docker/anvil/anvild`,
34	2. `hag deploy` — `docker compose build --push` builds the thin image that just
35	`COPY`s that binary in (the `Dockerfile` compiles nothing), pushes it, then
36	drives the host.
37
38	Do not build on the VPS — a release build needs ~2–4 GB peak and OOMs a
39	cheap, swap-less droplet. The VPS never compiles anything; it only pulls.
40
41	One-time toolchain setup:
42
43	```sh
44	dnf install zig # or: brew install zig
45	cargo install cargo-zigbuild
46	docker login registry.vibe.richardscollin.com # on this machine and hagrid
47	```
48
49	The Rust toolchain itself needs no setup step: `rust-toolchain.toml` pins the
50	version and the `x86_64-unknown-linux-musl` target, and rustup installs both
51	on the first `cargo` invocation in the repo. That file is the one place the
52	Rust version is set — `docker/runner/build.sh` reads it to bake the same
53	toolchain into `anvil-runner:rust`.
54
55	Every build is tagged twice: with the short git sha of the checkout (plus a
56	`-dirty` suffix when the working tree has uncommitted changes) and with
57	`latest`. The deploy pins the host to the exact sha it just pushed, so `hag
58	status` always names the running build. To roll back, run an older tag on the
59	host:
60
61	```sh
62	ssh hagrid 'cd anvil && IMAGE_TAG=<sha> docker compose up -d --no-build'
63	```
64
65	Managing it afterwards needs no ssh and no remembering where it lives:
66
67	```sh
68	hag status # what the host is running
69	hag logs -f # its logs
70	hag restart # recreate its containers
71	hag down # stop it
72	hag config # what hag resolved for this project
73	hag ls # every compose project on the host
74	```
75
76	## 4. What `compose.yaml` runs
77
78	```yaml
79	image: registry.vibe.richardscollin.com/anvil:${IMAGE_TAG:-latest}
80	container_name: anvil
81	restart: unless-stopped
82	ports: ["165.232.162.167:22:2222"]
83	volumes: [anvil-data:/data]
84	networks: [hagrid]
85	```
86
87	- `networks: [hagrid]` — an external network, so Caddy can reach `anvil:3000`.
88	The web port is deliberately not published.
89	- `ports` — git-over-SSH, published on the droplet's default public IPv4 only.
90	The reserved IP (`137.184.249.48`) arrives on the anchor address `10.15.0.6`,
91	where the host's own sshd listens, so binding one specific IP keeps the two
92	off each other. `ANVIL_SSH_BIND_IP` / `ANVIL_SSH_PORT` in the host's `.env`
93	override it.
94	- `anvil-data:/data` — a named volume holding the SQLite DB, the bare repos and
95	the persistent SSH host key. Pinned with `name:` to that exact string, so
96	compose adopts the volume the pre-compose deploys created instead of deriving
97	a fresh empty `anvil_anvil-data` from the project name. A named volume (not a
98	host bind mount) keeps it owned by the in-container `anvil` user.
99	- No Docker socket. anvil does not execute CI — runners dial in and run
100	jobs on their own daemons (see [docs/remote-runners.md](docs/remote-runners.md)).
101	The container has no reason to reach Docker, so the mount is gone, and with
102	it the root-equivalent hold the internet-facing process used to have on the
103	host.
104
105	> Agent sessions are the exception. They still drive Docker locally, so
106	> turning them on means putting the socket back — a `volumes:` entry for
107	> `/var/run/docker.sock` and a matching `group_add:` — and accepting that the
108	> container again has root-equivalent control of the host. They are off by
109	> default and absent from `deploy/anvil.toml`; read `docs/untrusted-mode.md`
110	> §7 before changing that.
111
112	The baked config lives at `/etc/anvil/anvil.toml` (see `deploy/anvil.toml`).
113	Override it by bind-mounting your own file over that path.
114
115	Host-side config. `compose.yaml` loads `~/anvil/.env` on the host if it
116	exists, and skips it if not — `hag` never copies a local `.env` up, so the
117	workstation and the host keep separate config. Today that file carries the SSO
118	client secret (§ [docs/oidc.md](docs/oidc.md)):
119
120	```sh
121	ssh hagrid 'printf "ANVIL_OIDC_CLIENT_SECRET=%s\n" \
122	"$(tr -d "[:space:]" < ~/.config/anvil/oidc-client-secret)" > ~/anvil/.env \
123	&& chmod 600 ~/anvil/.env'
124	```
125
126	Leave the variable out entirely rather than setting it empty: empty would
127	override the baked config and turn a confidential OIDC client into a public one.
128
129	Migrating from the `docker run` deploys. The old container was created by
130	`docker run`, so it carries no compose labels; compose reads `container_name:
131	anvil` as taken by a foreign container and refuses to adopt it. Clear it once,
132	before the first compose deploy:
133
134	```sh
135	ssh hagrid 'docker rm -f anvil'
136	```
137
138	The `anvil-data` volume is untouched by that and is what the new container
139	picks up.
140
141	## 5. First run: create your account, key, and the repo
142
143	```sh
144	# admin user
145	docker exec anvil anvild -c /etc/anvil/anvil.toml \
146	user create collin --email you@example.com --password '<password>' --admin
147
148	# your SSH public key (so you can push over SSH)
149	docker exec -i anvil anvild -c /etc/anvil/anvil.toml \
150	user add-key collin --title laptop --key "$(cat ~/.ssh/id_ed25519.pub)"
151
152	# the anvil repo itself
153	docker exec anvil anvild -c /etc/anvil/anvil.toml repo create collin/anvil \
154	--description "a minimal git forge in Rust"
155	```
156
157	(You can also create the user, add keys, and create repos from the web UI once
158	signed in — the CLI is just convenient for the first admin.)
159
160	## 6. Self-host anvil on anvil
161
162	From your local anvil checkout:
163
164	```sh
165	git remote add origin ssh://git@anvil.richardscollin.com/collin/anvil.git
166	git push -u origin main
167	```
168
169	Then browse it at `https://anvil.richardscollin.com/collin/anvil`. HTTPS clone
170	also works: `git clone https://anvil.richardscollin.com/collin/anvil.git`
171	(pushes over HTTPS require your account password as the git password).
172
173	## 7. CI: start a runner
174
175	anvil dispatches CI; it does not execute it. Nothing runs until a runner dials
176	in, so this is a required step, not an optional one — see
177	[docs/remote-runners.md](docs/remote-runners.md).
178
179	Set a shared secret in the config (`[ci] runner_token`), then on a machine with
180	room to build — the Mac mini, not the droplet:
181
182	```sh
183	cargo build --release --bin anvil-worker
184	cp target/release/anvil-worker /usr/local/bin/
185
186	anvil-worker --url https://anvil.richardscollin.com \
187	--token "$ANVIL_RUNNER_TOKEN" --name macmini
188	```
189
190	`deploy/worker/com.anvil.worker.plist` runs it under launchd on macOS. Run it
191	natively there rather than in a container: a containerized runner needs the
192	Docker socket mounted into it, which rebuilds the hole section 4 just removed.
193
194	Isolation is not weaker for being on a Mac. Docker Desktop runs every container
195	inside one Linux VM, so `--cap-drop=ALL`, `no-new-privileges` and the
196	cgroup limits are enforced by the same kernel primitives as on Linux — with the
197	VM as an extra boundary a bare-metal Linux host does not have.
198
199	Architecture. The Mac is arm64 and hagrid is x86_64, so a job running
200	`cargo test` on the runner tests an architecture you do not ship. Images are
201	unaffected (`deploy/build.sh` cross-compiles, and `compose.yaml` pins
202	`platforms: [linux/amd64]`), and per-pipeline platform selection is designed but
203	not yet wired to a config key.
204
205	## 8. The redeploy webhook (CD)
206
207	Any repo with a `.anvil/ci.yml` runs CI on push. A pipeline is just an image
208	plus steps:
209
210	```yaml
211	image: anvil-runner:rust
212	steps:
213	- name: test
214	run: cargo test --workspace
215	- run: cargo build --release
216	```
217
218	Runs show up at `/{owner}/{repo}/ci`, with a per-commit status badge on the
219	commit list and a full log on each run's page.
220
221	Job sandbox. The runner is the only Docker client on its machine; the job
222	container gets no socket, no mounts (the checkout is uploaded as a tar), all
223	capabilities dropped, and `no-new-privileges`. Resource bounds come from the
224	forge's `[ci]` config and travel with each job, so tightening one does not need
225	runners redeployed: `memory_mb` (default 2048), `cpus` (2), `pids_limit` (512),
226	`timeout_secs` (1800, then the container is killed), `network` (true),
227	`run_as` (empty = image default), and `allowed_images` (empty = any; a tagless
228	entry like `"rust"` allows every tag). The allowlist is applied on the forge
229	when the job is built, so a runner cannot widen it. See
230	`docs/untrusted-mode.md` for the threat model and what this does/doesn't
231	protect against.
232
233	Secrets now leave the host. A pipeline's secrets are sent to the runner
234	with its job and sit in plaintext in a container on a machine anvil does not
235	own. Scope repository secrets accordingly, and treat a runner host as being as
236	trusted as the forge itself.
237
238	Continuous deployment is deliberately scoped to one repository. On a
239	successful run of `deploy_branch` (default `main`) in the repo named by
240	`[ci] deploy_repo`, anvil POSTs JSON to `[ci] deploy_webhook`:
241
242	```json
243	{ "repo": "collin/anvil", "ref": "main", "commit": "<oid>", "run_id": 42 }
244	```
245
246	No other repo can trigger this, even with passing CI. The webhook target is a
247	host-local plaintext receiver (HTTPS is unsupported, to keep the build
248	TLS-free) — typically a tiny script-runner on hagrid that, on a verified
249	request, runs the actual redeploy. Verify the `X-Anvil-Deploy-Secret` header
250	(set `[ci] deploy_secret`) before doing anything. Because anvil cross-compiles
251	(section 3), "redeploy anvil" usually means: the receiver pulls the freshly
252	built image and runs `docker compose up -d --no-build` in `~/anvil` — it does
253	not build in-place.
254
255	Building an image is not a CI job. Job containers get no Docker socket by
256	design, so `docker build`/`docker push` cannot happen inside a pipeline. That
257	work belongs to a separate deploy agent on the build host, triggered by this
258	webhook, running outside the sandbox with Docker access — a different trust
259	level from the runner, and deliberately a different process.
260
261	> The deploy receiver runs with whatever privileges you give it — keep it
262	> minimal, secret-gated, and bound to localhost / the Docker host gateway only.
263
264	## Operations
265
266	- Update: `./deploy/deploy.sh` (cross-compile + push + restart in one
267	command); or `hag restart` to recreate the container from the image already
268	on the host (the `anvil-data` volume persists). NOTE: adding new DB tables in a
269	future version won't auto-apply to an existing database yet (Toasty migration
270	support is pending) — the git repos on disk are unaffected, but repo metadata
271	in SQLite may need recreating until migrations land.
272	- Backup: snapshot the `anvil-data` volume, e.g.
273	`docker run --rm -v anvil-data:/data -v "$PWD":/out debian:bookworm-slim \
274	tar czf /out/anvil-data.tgz -C /data .`
275	- Logs: `hag logs -f` (or `docker logs -f anvil` on the host).
276	- System routes live under `/-/` (e.g. sign in at
277	`https://anvil.richardscollin.com/-/login`); `/{username}` is the user/repo
278	namespace.