collin/anvil
RenderedSource
| 1 | # CI artifacts — design |
| 2 | |
| 3 | Status: **implemented** (2026-06-10). Companion to the CI runner in |
| 4 | `crates/anvil-ci` and the threat model in `docs/untrusted-mode.md`. |
| 5 | Deviations from the original sketch are noted inline. |
| 6 | |
| 7 | ## Goals |
| 8 | |
| 9 | - A CI run can produce artifacts (binaries, reports, generated docs). |
| 10 | - Artifacts are keyed by commit and served from anvil: downloadable from the |
| 11 | run page and the commit page, with a "latest on branch" alias. |
| 12 | - A directory artifact can opt into being *browsable* — served as a static |
| 13 | site rather than downloaded. The canonical test case is rustdoc: a big |
| 14 | generated HTML subtree, viewable at a stable latest-on-branch URL. |
| 15 | - Jobs can declare how to extract metadata from artifacts (sizes, version |
| 16 | strings, test/coverage numbers) so the UI can surface it next to the |
| 17 | run/commit without downloading anything. |
| 18 | - The broker security model is unchanged: the job container still gets no |
| 19 | socket, no mounts, no volumes. |
| 20 | |
| 21 | ## How the broker gets files out of the container |
| 22 | |
| 23 | The checkout already goes *in* via the Docker API (`upload_to_container`, a |
| 24 | tar). Artifacts come *out* the same way: after `wait_container` returns and |
| 25 | before the container is removed, the broker calls `download_from_container` |
| 26 | (`GET /containers/{id}/archive?path=...`) for each declared path. That works |
| 27 | on a stopped container, needs no shared filesystem, and keeps anvil the only |
| 28 | Docker client. |
| 29 | |
| 30 | Rules: |
| 31 | |
| 32 | - Declared paths are resolved under `/workspace`; absolute paths and `..` are |
| 33 | rejected at parse time. |
| 34 | - A path that is a directory arrives as a tar and is stored as |
| 35 | `<name>.tar.gz`; a single file is stored as-is. |
| 36 | - Collection happens on success **and** failure (test reports matter most on |
| 37 | red runs) but not after a timeout kill (the container is already gone). |
| 38 | Each artifact records which it was. |
| 39 | - Per-artifact and per-run size caps come from config (below). The download |
| 40 | stream is aborted, and the artifact skipped with a logged note, when a cap |
| 41 | is exceeded — never the run failed retroactively. |
| 42 | |
| 43 | ## Declaring artifacts in `.anvil/ci.toml` |
| 44 | |
| 45 | ```toml |
| 46 | image = "anvil-runner:rust" |
| 47 | |
| 48 | [[steps]] |
| 49 | name = "build" |
| 50 | run = "cargo build --release" |
| 51 | |
| 52 | [[steps]] |
| 53 | name = "test" |
| 54 | run = "cargo test --workspace" |
| 55 | |
| 56 | [[artifacts]] |
| 57 | name = "anvild" # unique per pipeline; [a-zA-Z0-9._-]+ |
| 58 | path = "target/release/anvild" # file → download; dir → tar.gz download |
| 59 | meta.version = "./target/release/anvild --version" |
| 60 | meta.size = "stat -c %s target/release/anvild" |
| 61 | |
| 62 | [[artifacts]] |
| 63 | name = "coverage" |
| 64 | path = "coverage/" |
| 65 | meta.line_pct = "jq -r .line_pct coverage/summary.json" |
| 66 | |
| 67 | [[artifacts]] |
| 68 | name = "doc" |
| 69 | path = "target/doc/" # rustdoc HTML subtree |
| 70 | browse = true # serve as a static site, don't download |
| 71 | ``` |
| 72 | |
| 73 | Every bare key belongs to whichever `[[table]]` precedes it, so `image` (and |
| 74 | `platform`, and `secrets`) has to come before the first `[[steps]]` — put them |
| 75 | at the top and the rest reads in order. |
| 76 | |
| 77 | `meta` is a map of key → shell command. The commands run **inside the job |
| 78 | container** (appended to the script after the steps, still `set -e`-free — |
| 79 | each is best-effort), because artifact content is untrusted and must never be |
| 80 | executed or parsed on the host. Each command's stdout is trimmed and capped |
| 81 | (1 KiB); the resulting key→value map is stored as JSON on the artifact row. |
| 82 | A failed extractor stores nothing for that key and appends a note to the log. |
| 83 | |
| 84 | Implementation note: each extractor's stdout lands in its own file under |
| 85 | `/tmp/anvil-meta/<artifact>/<key>` (written by a generated script trailer that |
| 86 | runs even when a step fails — the steps execute in a subshell whose exit code |
| 87 | is preserved). The broker downloads that directory via the same archive |
| 88 | mechanism as artifacts; file-per-value avoids shell JSON-escaping entirely. |
| 89 | |
| 90 | ## Storage |
| 91 | |
| 92 | ``` |
| 93 | data_dir/artifacts/{repo_id}/{commit}/{name} # file artifact |
| 94 | data_dir/artifacts/{repo_id}/{commit}/{name}.tar.gz # dir, download-only |
| 95 | data_dir/artifacts/{repo_id}/{commit}/{name}/... # dir, browse: true |
| 96 | ``` |
| 97 | |
| 98 | - Keyed by `repo_id` (stable across renames) and full commit sha. |
| 99 | - Browsable directory artifacts are stored *extracted* so requests are plain |
| 100 | file reads (no per-request untar); download-only directories stay tar.gz. |
| 101 | Extraction rejects entries that escape the artifact root (`..`, absolute, |
| 102 | symlinks) — the tar comes from an untrusted container. |
| 103 | - A re-run of the same commit overwrites that commit's directory. |
| 104 | - New Toasty model `CiArtifact`: `id`, `run_id`, `repo_id`, `commit`, `name`, |
| 105 | `size`, `is_dir`, `browse`, `meta` (JSON string), `created_at`. (Toasty |
| 106 | migrations still don't exist — this lands as a new table, which |
| 107 | `db::connect` only creates on a fresh DB; same caveat as every schema |
| 108 | change so far.) |
| 109 | |
| 110 | ## Serving |
| 111 | |
| 112 | - Run page: artifact list (name, size, metadata chips) with download links — |
| 113 | or a "browse" link for `browse: true` artifacts. |
| 114 | - `GET /{owner}/{repo}/ci/{run_id}/artifacts/{name}` — direct download. |
| 115 | - `GET /{owner}/{repo}/artifacts/{rev}/{name}` — alias: resolve `rev` (branch |
| 116 | or commit) to the latest run with that artifact on that commit, redirect to |
| 117 | the run-scoped URL. This gives "latest on branch" for free since CI runs on |
| 118 | every push tip. |
| 119 | - `GET /{owner}/{repo}/artifacts/{rev}/{name}/{*path}` — browsable artifacts |
| 120 | only: serve files from the extracted subtree exactly like `pages.rs` serves |
| 121 | a `pages` branch (extension→content-type map, `nosniff`, `index.html` |
| 122 | resolution with trailing-slash redirect so rustdoc's relative links work). |
| 123 | E.g. `/{owner}/{repo}/artifacts/main/doc/anvil_core/` is always the default |
| 124 | branch's latest rustdoc. |
| 125 | - Commit page and per-commit CI badges link through to the run's artifacts. |
| 126 | - Visibility follows the repo, like pages and CI logs. |
| 127 | - Download artifacts get `Content-Disposition: attachment` + |
| 128 | `application/octet-stream` + `nosniff`. Browsable artifacts serve inline by |
| 129 | design; that is the same stored-XSS-on-forge-origin exposure as pages |
| 130 | (threat model §2) — acceptable single-tenant, and both move to a separate |
| 131 | origin together before untrusted users. |
| 132 | |
| 133 | ## Retention / GC |
| 134 | |
| 135 | Config, all under `[ci]`: |
| 136 | |
| 137 | ```toml |
| 138 | artifact_max_mb = 256 # per artifact |
| 139 | artifact_run_max_mb = 512 # per run, summed |
| 140 | artifact_quota_mb = 4096 # per repo, summed; 0 = unlimited |
| 141 | ``` |
| 142 | |
| 143 | GC is deterministic and runs after each run's artifacts are stored: while the |
| 144 | repo is over `artifact_quota_mb`, delete the oldest commit-directory (and its |
| 145 | rows) — except directories that are the newest artifact-bearing commit of any |
| 146 | branch head, which are pinned. No background sweeper, no clocks to test; the |
| 147 | invariant holds whenever an artifact lands. |
| 148 | |
| 149 | Orphan cleanup (repo deleted → remove `artifacts/{repo_id}`) will hook into |
| 150 | repo deletion **when that feature exists** — anvil currently has no way to |
| 151 | delete a repository at all, so there is nothing to hook yet. |
| 152 | |
| 153 | ## Out of scope (deliberately) |
| 154 | |
| 155 | - Cross-run caching (e.g. cargo registry/target caching) — different problem, |
| 156 | different lifetime, mounts would pierce the sandbox. |
| 157 | - Artifact upload from outside CI (release uploads) — maybe later, different |
| 158 | authz. |
| 159 | - Dedup/content-addressing — at our scale, per-commit copies are fine. |
| 160 | |
| 161 | ## Implementation order |
| 162 | |
| 163 | 1. Schema + config: `CiArtifact` model, `[ci]` caps, parse `artifacts` in |
| 164 | `ci.rs` (with path/name validation + tests). |
| 165 | 2. Broker: collect declared paths via `download_from_container`, store to |
| 166 | disk, write rows; meta-extractor trailer + `.anvil-meta.json` pickup. |
| 167 | 3. Web: run-page list + download route, then the `{rev}` alias route, then |
| 168 | the browse route (reusing the content-type/index helpers from `pages.rs`). |
| 169 | Test with rustdoc on this repo (`cargo doc` → `target/doc`, `browse: true`). |
| 170 | 4. GC + repo-deletion hook. |