anvilsign in

collin/anvil

cbf2877db1e86b00a92d823f25f5ca60541d7d61 / TODO.md

BoardRenderedSource

1# Misc TODO
2
3- [x] do we have a way to view the source in a repo by branch or at a given commit?
4 - yes: `/{owner}/{repo}/tree/{rev}` always accepted any branch/tag/commit;
5 what was missing was UI. Added: branch/tag switcher dropdown on the repo
6 and tree pages, "browse files" link on the commit page, percent-encoded
7 ref names so branches with `/` work, and an unborn-HEAD fallback so a
8 repo whose HEAD names a missing branch no longer renders as empty.
9- [x] implement the CI artifact system _(done per `docs/ci-artifacts.md`:
10 `artifacts:` in ci.yml with in-container meta extractors, broker
11 collection via `download_from_container`, run-page list, downloads +
12 `/{owner}/{repo}/artifacts/{rev}/{name}` alias + `browse: true` static
13 sites, quota GC with branch-tip pinning, and a schema shim so the table
14 appears on existing DBs. Verified end-to-end against real Docker incl.
15 the GC path. Repo-deletion cleanup deferred: repo deletion doesn't exist.)_
16 - every push triggers CI on the tip commit (already true); a run should be
17 able to produce artifacts
18 - artifacts are stored per-commit and served from anvil (download from the
19 run page / commit page; latest-on-branch alias would be nice)
20 - jobs can declare how to extract metadata from artifacts (e.g. sizes,
21 test/coverage numbers, version strings) so it can be surfaced in the UI
22 next to the run/commit
23 - sketch needed: where artifacts live on disk, retention/GC, how
24 `.anvil/ci.yml` declares artifact paths + metadata extractors, and how
25 the broker gets files out of the sandboxed container
26 - use rustdoc as an example when testing the feature: it generates a big
27 HTML subtree, so support rendering/serving a whole HTML artifact subtree
28 the way the pages feature does (rustdoc output as a served site)
29- [x] repo mirroring to GitHub (push mirror) _(per-repo "Mirror push URL" in
30 settings (`repositories.mirror_url`, column shim for existing DBs);
31 after every successful push over HTTP or SSH a background
32 `git push --mirror` fires — shells out to git (gix can't push yet;
33 runtime image now installs git). Credentials ride in the URL
34 (`https://x-access-token:<token>@github.com/...`), stored as-is and
35 redacted from logs. Verified e2e against a local bare "github".)_
36 - pull mirror (maybe, NOT done): a repo that virtually mirrors a GitHub
37 repo and just displays it here — periodically fetched, read-only on
38 the anvil side
39- [x] per-repository issue tracker _(`Issue`/`IssueComment` models with
40 schema shims; per-repo numbering (#1, #2, …); list page with
41 open/closed tabs, new-issue form, detail page with markdown bodies and
42 comments, close/reopen (issue author or repo writer). Any logged-in
43 reader can open issues and comment; visibility follows the repo.
44 "Issues" link in the repo nav. CSRF on all forms. Verified e2e through
45 the login + form flow. No labels/assignees/editing yet — deliberately
46 minimal.)_
47- [x] render a root `README.md` below the file tree on the repo page _(any
48 case of `readme.md` at the root; reuses `render_markdown`, links to the
49 blob view from the box header)_
50- [x] push-to-create: `git push` to a repo that doesn't exist yet creates it
51 _(it did NOT already work — both transports 404'd. Policy in
52 `repos::create_on_push`: pusher must own the namespace or be admin;
53 created repos are private. Over HTTP a missing repo now answers the
54 receive-pack advertisement with a Basic challenge so git prompts.
55 Verified e2e over both HTTP and SSH, incl. the cross-namespace denial.)_
56- [x] make SSH the default clone selection in the clone pill buttons, and put
57 it first (before HTTP) _(only when `[ssh] enabled`; HTTP stays the lone
58 pill otherwise)_
59- [x] rename the `anvil` crate to `anvil_cli` _(named it `anvil-cli` to match
60 the workspace's hyphenated crate names; the binary is still `anvild`, so
61 deploy scripts and docs needed no changes)_
62
63---
64
65# Error in git push _(FIXED 2026-06-10, uncommitted)_
66
67**Root cause:** every push after the first sends a *thin pack* — deltas whose
68base objects aren't in the pack (the server already has them), referenced by
69object id (`REF_DELTA`). `vendor/gitserver-core/src/receive_pack.rs::write_pack`
70passed `None` as `thin_pack_base_object_lookup` to
71`gix_pack::Bundle::write_to_directory`, so gix couldn't resolve the bases and
72aborted. First-push-to-empty-repo worked because that pack is self-contained.
73
74**Fix:** pass the already-open `gix::Repository` as the lookup (it implements
75`gix_object::Find`). Regression test
76`receive_thin_pack_with_ref_deltas` builds a real thin pack via
77`git pack-objects --thin`, asserts it contains ref-deltas, and pushes it
78through `receive_pack`. Verified the test fails without the fix.
79
80Original report:
81
82```
83collin@mini ~/C/anvil (main)> git push
84Enter passphrase for key '/Users/collin/.ssh/id_ed25519':
85Enumerating objects: 117, done.
86Counting objects: 100% (117/117), done.
87Delta compression using up to 8 threads
88Compressing objects: 100% (62/62), done.
89Writing objects: 100% (66/66), 27.57 KiB | 3.94 MiB/s, done.
90Total 66 (delta 39), reused 0 (delta 0), pack-reused 0 (from 0)
91send-pack: unexpected disconnect while reading sideband packet
92fatal: the remote end hung up unexpectedly
93collin@mini ~/C/anvil (main) [128]>
94```
95
96server logs:
97
98```
9926-06-09T21:53:46.466577Z INFO anvil_ssh: ssh auth: accepted key SHA256:Rg41caN7vw2WYYxiJN6lrIlX0DTXYF0rC2QzKZW1tB0 (user 1)
1002026-06-09T21:53:46.635946Z INFO anvil_ssh: ssh git-receive-pack on collin/anvil.git (user Some(1))
1012026-06-09T21:53:46.805146Z ERROR anvil_ssh: git ssh git-receive-pack: protocol error: failed to write incoming pack: Ref delta objects are not supported as there is no way to look them up. Resolve them beforehand.
1022026-06-09T21:54:28.571834Z INFO anvil_ssh: ssh auth: rejected unknown key SHA256:ZlWZyHqspqFeUQV84qaXtDQq4gcA33dR7y8dYbeg9u8
1032026-06-09T21:54:35.565597Z INFO anvil_ssh: ssh auth: accepted key SHA256:Rg41caN7vw2WYYxiJN6lrIlX0DTXYF0rC2QzKZW1tB0 (user 1)
1042026-06-09T21:54:35.676113Z INFO anvil_ssh: ssh git-receive-pack on collin/anvil.git (user Some(1))
1052026-06-09T21:54:36.268520Z ERROR anvil_ssh: git ssh git-receive-pack: protocol error: failed to write incoming pack: Ref delta objects are not supported as there is no way to look them up. Resolve them beforehand.
1062026-06-09T21:55:54.223033Z INFO anvil_ssh: ssh auth: rejected unknown key SHA256:ZlWZyHqspqFeUQV84qaXtDQq4gcA33dR7y8dYbeg9u8
1072026-06-09T21:56:00.758384Z INFO anvil_ssh: ssh auth: accepted key SHA256:Rg41caN7vw2WYYxiJN6lrIlX0DTXYF0rC2QzKZW1tB0 (user 1)
1082026-06-09T21:56:00.906553Z INFO anvil_ssh: ssh git-receive-pack on collin/anvil.git (user Some(1))
1092026-06-09T21:56:01.067903Z ERROR anvil_ssh: git ssh git-receive-pack: protocol error: failed to write incoming pack: Ref delta objects are not supported as there is no way to look them up. Resolve them beforehand.
110```
111
112---
113
114# Session notes / resume point
115
116_Last updated: 2026-06-10 (third session). Working state is clean: build,
117clippy, fmt, sort-derives, and `cargo test --workspace` all pass (the
118pre-commit hook runs all of these). Third-session work below is UNCOMMITTED
119(repo convention: commit only when asked). **All top-of-file TODO items are
120done** except the pull-mirror "maybe"._
121
122## Done this session (2026-06-10, third session)
123
124All six remaining TODO items — see the checked-off entries at the top of this
125file for the details. Headlines:
126
127- **CI artifacts** (`docs/ci-artifacts.md` is the canonical reference):
128 `artifacts:` in `.anvil/ci.yml`; broker pulls them out of the stopped
129 container via `download_from_container` (inverse of the checkout upload —
130 still no mounts); meta extractors run *inside* the sandbox, one file per
131 value under `/tmp/anvil-meta`; `browse: true` directories are served like
132 pages (rustdoc-ready); `/{owner}/{repo}/artifacts/{rev}/{name}` is the
133 latest-on-branch alias; per-repo quota GC pins branch tips. New table
134 `ci_artifacts` + `[ci] artifact_*_mb` caps.
135- **Schema shims for existing DBs** (`anvil-core/src/db.rs`): Toasty still
136 only pushes schema on a fresh file, so new tables/columns ship as
137 idempotent DDL applied on connect (`SCHEMA_SHIMS`/`COLUMN_SHIMS`), with
138 tests asserting fresh and migrated databases converge. New deps: rusqlite
139 (pinned to toasty's), flate2.
140- **Issues** (`anvil-core/src/issues.rs`, `anvil-web/src/issues.rs`):
141 minimal GitHub-shaped tracker; tables `issues` + `issue_comments`.
142- **Push mirroring** (`anvil-git/src/mirror.rs`): `repositories.mirror_url`
143 → background `git push --mirror` after each push; Dockerfile now installs
144 git (the one thing gix can't do yet is push).
145- **Push-to-create** (`repos::create_on_push` + both transports), **README
146 on repo page**, **SSH-first clone pills**, **rev-switcher/browse-at-rev UI**
147 (committed earlier this session, along with the `anvil-cli` rename).
148
149## Done earlier (2026-06-10, second session)
150
151- **UI quick wins** (`crates/anvil-web/src/ui.rs`) — latest-commit bar on the
152 repo page (sha + subject + author/time, attached above the file box, links
153 to the commit); profile page no longer shows the email; repo header reads
154 `owner / repo` without the leading `anvil /`.
155- **(c) sandboxed CI broker** — job containers now run with `cap_drop=ALL` +
156 `no-new-privileges` unconditionally, plus config-driven `pids_limit` (512),
157 `memory_mb`+swap (2048), `cpus` (2), wall-clock `timeout_secs` (1800,
158 force-removed on expiry), optional `network = false`, `run_as`, and an
159 `allowed_images` allowlist (empty = any; tagless entry allows all tags).
160 `CiConfig` in `crates/anvil-core/src/config.rs` (with `image_allowed` test);
161 `execute()` in `crates/anvil-ci/src/lib.rs`. Read-only rootfs deliberately
162 skipped (workspace lives in the container fs; no volumes ever attached).
163 Docs: `DEPLOY.md` §7, `anvil.example.toml` `[ci]`.
164- **(b) threat model** — `docs/untrusted-mode.md`: severity-ranked analysis
165 (CI containment, pages/stored-XSS origin, git resource exhaustion,
166 registration anti-abuse, authz granularity, webhook SSRF), the already-right
167 list, and the stance: single-tenant supported, untrusted gated on items 1–4.
168- **Pages hosting** — `crates/anvil-web/src/pages.rs`: serves a repo's `pages`
169 branch at `/{owner}/{repo}/pages/...`; top-level dirs are separate sites
170 (rustdoc, book, …); `index.html` resolution with trailing-slash redirect so
171 relative links work; extension→content-type map + `nosniff`; listing page
172 with publish hint; "Pages" button on the repo header. Visibility follows the
173 repo (private → 404). Publish with `git push origin <built-branch>:pages`.
174
175## Done earlier (same day, first session)
176
177- **CI UI** — runs list `/{owner}/{repo}/ci`, run-detail (status/timing/log),
178 per-commit status badges, "CI" nav link. (`crates/anvil-web/src/ui.rs`)
179- **CD redeploy webhook** — on a green run of `[ci] deploy_branch` in the single
180 `[ci] deploy_repo`, POST to `[ci] deploy_webhook` (`X-Anvil-Deploy-Secret`
181 header). Scoped to ONE repo. `CiConfig` in `crates/anvil-core/src/config.rs`;
182 `deploy()` in `crates/anvil-ci/src/lib.rs`. Docs: `DEPLOY.md` §7,
183 `deploy/anvil.toml`. `reqwest` added with NO TLS feature (keeps musl
184 cross-compile aws-lc-free).
185- **Docker socket on hagrid** — `deploy/run.sh` mounts it + `--group-add`s the
186 gid for the non-root user; caveat in `DEPLOY.md` §4.
187- **Toasty ORM cleanup** — `ci.rs` `list_by_repo`/`latest_for_commit`/
188 `queued_ids` now sort/limit/filter in SQL, not in memory. Verified by the new
189 `ordering_and_limit_run_in_the_database` test. (Sweep: these were the only
190 real instances; `repos::list_all_with_owner` sorts by a joined username and
191 needs all rows — intentionally left.)
192- **(a) CSRF + cookie hardening** —
193 - Cookie: `HttpOnly` + `SameSite=Lax` + `Secure` (auto via
194 `Config::secure_cookies()` when base_url is https).
195 - Synchronizer token `HMAC-SHA256(server_secret, session)`; secret persisted
196 at `data_dir/csrf_secret` (`App::csrf_token` in `crates/anvil-core/src/lib.rs`).
197 Deps `hmac`, `sha2`.
198 - `Csrf` extractor + constant-time `verify_csrf` (`crates/anvil-web/src/auth.rs`).
199 Hidden `csrf` field + verification on add/delete SSH key, new repo, repo
200 settings. Login exempt; logout relies on SameSite.
201 - htmx insurance: `auth::csrf_context` middleware → request-scoped task-local;
202 `layout` sends the token via `hx-headers` on every htmx request.
203
204## The a/b/c plan — ALL DONE
205
206(a) CSRF + cookie hardening, (b) `docs/untrusted-mode.md` threat model,
207(c) sandboxed CI broker. Stronger isolation tiers (gVisor/Kata/Firecracker,
208egress filtering, CI-minute quotas) are recorded in the threat model as the
209gate for untrusted tenants, not planned work.
210
211## Loose ends
212
213- **CSRF header consumption:** `hx-headers` sends the token as a `csrf` header,
214 but `verify_csrf` only reads the form field. When we add a tokenless htmx
215 action (raw `hx-post`/`hx-delete`, no `<form>`), also read the `csrf` header.
216- **Toasty migrations:** still no real migration system; the shim approach in
217 `db::connect` (`SCHEMA_SHIMS` for tables, `COLUMN_SHIMS` for columns, both
218 test-verified against a fresh `push_schema`) covers what we've needed so
219 far. New columns must be declared last in the model.
220- **Mirror URL secrecy:** `repositories.mirror_url` may embed a token and is
221 stored plaintext in SQLite (logged redacted). Fine single-tenant; revisit
222 if the DB ever leaves the box.
223- **Pages caveats (single-tenant-acceptable):** served from the forge origin —
224 move to a separate origin before untrusted users (threat model §2); whole
225 blobs load into memory per request (fine at our scale). The same applies to
226 `browse: true` CI artifacts.
227- **Issue tracker minimalism:** no labels, assignees, milestones, or
228 editing/deleting of posts. Per-repo numbering assumes a single server
229 process (matches deployment; noted in `models.rs`).
230
231## Remaining roadmap (plan milestones beyond a/b/c)
232
2339. Pull requests (gix merge) · 10. Webhooks (mind the SSRF item).
234(8. Issues shipped 2026-06-10.)