collin/anvil

2f35cebbbbd354d178741ac6c375a3f1dd547905 / docs / ci-artifacts.md

1	# CI artifacts — design
2
3	Status: implemented (2026-06-10). Companion to the CI runner in
4	`crates/anvil-ci` and the threat model in `docs/untrusted-mode.md`.
5	Deviations from the original sketch are noted inline.
6
7	## Goals
8
9	- A CI run can produce artifacts (binaries, reports, generated docs).
10	- Artifacts are keyed by commit and served from anvil: downloadable from the
11	run page and the commit page, with a "latest on branch" alias.
12	- A directory artifact can opt into being browsable — served as a static
13	site rather than downloaded. The canonical test case is rustdoc: a big
14	generated HTML subtree, viewable at a stable latest-on-branch URL.
15	- Jobs can declare how to extract metadata from artifacts (sizes, version
16	strings, test/coverage numbers) so the UI can surface it next to the
17	run/commit without downloading anything.
18	- The broker security model is unchanged: the job container still gets no
19	socket, no mounts, no volumes.
20
21	## How the broker gets files out of the container
22
23	The checkout already goes in via the Docker API (`upload_to_container`, a
24	tar). Artifacts come out the same way: after `wait_container` returns and
25	before the container is removed, the broker calls `download_from_container`
26	(`GET /containers/{id}/archive?path=...`) for each declared path. That works
27	on a stopped container, needs no shared filesystem, and keeps anvil the only
28	Docker client.
29
30	Rules:
31
32	- Declared paths are resolved under `/workspace`; absolute paths and `..` are
33	rejected at parse time.
34	- A path that is a directory arrives as a tar and is stored as
35	`<name>.tar.gz`; a single file is stored as-is.
36	- Collection happens on success and failure (test reports matter most on
37	red runs) but not after a timeout kill (the container is already gone).
38	Each artifact records which it was.
39	- Per-artifact and per-run size caps come from config (below). The download
40	stream is aborted, and the artifact skipped with a logged note, when a cap
41	is exceeded — never the run failed retroactively.
42
43	## Declaring artifacts in `.anvil/ci.yml`
44
45	```yaml
46	image: rust:1.95-bookworm
47	steps:
48	- name: build
49	run: cargo build --release
50	- name: test
51	run: cargo test --workspace
52
53	artifacts:
54	- name: anvild # unique per pipeline; [a-zA-Z0-9._-]+
55	path: target/release/anvild # file → download; dir → tar.gz download
56	meta:
57	version: ./target/release/anvild --version
58	size: stat -c %s target/release/anvild
59	- name: coverage
60	path: coverage/
61	meta:
62	line_pct: jq -r .line_pct coverage/summary.json
63	- name: doc
64	path: target/doc/ # rustdoc HTML subtree
65	browse: true # serve as a static site, don't download
66	```
67
68	`meta` is a map of key → shell command. The commands run **inside the job
69	container** (appended to the script after the steps, still `set -e`-free —
70	each is best-effort), because artifact content is untrusted and must never be
71	executed or parsed on the host. Each command's stdout is trimmed and capped
72	(1 KiB); the resulting key→value map is stored as JSON on the artifact row.
73	A failed extractor stores nothing for that key and appends a note to the log.
74
75	Implementation note: each extractor's stdout lands in its own file under
76	`/tmp/anvil-meta/<artifact>/<key>` (written by a generated script trailer that
77	runs even when a step fails — the steps execute in a subshell whose exit code
78	is preserved). The broker downloads that directory via the same archive
79	mechanism as artifacts; file-per-value avoids shell JSON-escaping entirely.
80
81	## Storage
82
83	```
84	data_dir/artifacts/{repo_id}/{commit}/{name} # file artifact
85	data_dir/artifacts/{repo_id}/{commit}/{name}.tar.gz # dir, download-only
86	data_dir/artifacts/{repo_id}/{commit}/{name}/... # dir, browse: true
87	```
88
89	- Keyed by `repo_id` (stable across renames) and full commit sha.
90	- Browsable directory artifacts are stored extracted so requests are plain
91	file reads (no per-request untar); download-only directories stay tar.gz.
92	Extraction rejects entries that escape the artifact root (`..`, absolute,
93	symlinks) — the tar comes from an untrusted container.
94	- A re-run of the same commit overwrites that commit's directory.
95	- New Toasty model `CiArtifact`: `id`, `run_id`, `repo_id`, `commit`, `name`,
96	`size`, `is_dir`, `browse`, `meta` (JSON string), `created_at`. (Toasty
97	migrations still don't exist — this lands as a new table, which
98	`db::connect` only creates on a fresh DB; same caveat as every schema
99	change so far.)
100
101	## Serving
102
103	- Run page: artifact list (name, size, metadata chips) with download links —
104	or a "browse" link for `browse: true` artifacts.
105	- `GET /{owner}/{repo}/ci/{run_id}/artifacts/{name}` — direct download.
106	- `GET /{owner}/{repo}/artifacts/{rev}/{name}` — alias: resolve `rev` (branch
107	or commit) to the latest run with that artifact on that commit, redirect to
108	the run-scoped URL. This gives "latest on branch" for free since CI runs on
109	every push tip.
110	- `GET /{owner}/{repo}/artifacts/{rev}/{name}/{*path}` — browsable artifacts
111	only: serve files from the extracted subtree exactly like `pages.rs` serves
112	a `pages` branch (extension→content-type map, `nosniff`, `index.html`
113	resolution with trailing-slash redirect so rustdoc's relative links work).
114	E.g. `/{owner}/{repo}/artifacts/main/doc/anvil_core/` is always the default
115	branch's latest rustdoc.
116	- Commit page and per-commit CI badges link through to the run's artifacts.
117	- Visibility follows the repo, like pages and CI logs.
118	- Download artifacts get `Content-Disposition: attachment` +
119	`application/octet-stream` + `nosniff`. Browsable artifacts serve inline by
120	design; that is the same stored-XSS-on-forge-origin exposure as pages
121	(threat model §2) — acceptable single-tenant, and both move to a separate
122	origin together before untrusted users.
123
124	## Retention / GC
125
126	Config, all under `[ci]`:
127
128	```toml
129	artifact_max_mb = 256 # per artifact
130	artifact_run_max_mb = 512 # per run, summed
131	artifact_quota_mb = 4096 # per repo, summed; 0 = unlimited
132	```
133
134	GC is deterministic and runs after each run's artifacts are stored: while the
135	repo is over `artifact_quota_mb`, delete the oldest commit-directory (and its
136	rows) — except directories that are the newest artifact-bearing commit of any
137	branch head, which are pinned. No background sweeper, no clocks to test; the
138	invariant holds whenever an artifact lands.
139
140	Orphan cleanup (repo deleted → remove `artifacts/{repo_id}`) will hook into
141	repo deletion when that feature exists — anvil currently has no way to
142	delete a repository at all, so there is nothing to hook yet.
143
144	## Out of scope (deliberately)
145
146	- Cross-run caching (e.g. cargo registry/target caching) — different problem,
147	different lifetime, mounts would pierce the sandbox.
148	- Artifact upload from outside CI (release uploads) — maybe later, different
149	authz.
150	- Dedup/content-addressing — at our scale, per-commit copies are fine.
151
152	## Implementation order
153
154	1. Schema + config: `CiArtifact` model, `[ci]` caps, parse `artifacts:` in
155	`ci.rs` (with path/name validation + tests).
156	2. Broker: collect declared paths via `download_from_container`, store to
157	disk, write rows; meta-extractor trailer + `.anvil-meta.json` pickup.
158	3. Web: run-page list + download route, then the `{rev}` alias route, then
159	the browse route (reusing the content-type/index helpers from `pages.rs`).
160	Test with rustdoc on this repo (`cargo doc` → `target/doc`, `browse: true`).
161	4. GC + repo-deletion hook.