collin/anvil
d0d2c17b7e3a1d08ce52b9bf773a161b55af19e8 / TODO.md
| 1 | # Todo |
| 2 | |
| 3 | - [ ] pull mirror (maybe): a repo that virtually mirrors a GitHub repo |
| 4 | - just displays it here — periodically fetched, read-only on the anvil side |
| 5 | |
| 6 | # Done [x] |
| 7 | |
| 8 | - [x] render `TODO.md` per the todo-md spec (`~/Code/todo-md`) as a kanban board |
| 9 | - `anvil-web/src/todomd.rs`: spec parser (sections by heading, `[x]`-marked |
| 10 | done sections, checkbox tasks, indented details, fence-aware) + board |
| 11 | renderer; prose sections render as a collapsible notes area below the |
| 12 | board |
| 13 | - blob pages for any `TODO.md` get Board/Rendered/Source pills (board |
| 14 | default, falls back to markdown when the file has no tasks); the repo |
| 15 | page shows the board in a box below the README |
| 16 | - this file is itself spec-compliant now; structured updates come later |
| 17 | |
| 18 | - [x] browse source at any branch, tag, or commit |
| 19 | - `/{owner}/{repo}/tree/{rev}` always accepted any rev; what was missing |
| 20 | was UI. Added: branch/tag switcher dropdown on the repo and tree pages, |
| 21 | "browse files" link on the commit page, percent-encoded ref names so |
| 22 | branches with `/` work, and an unborn-HEAD fallback so a repo whose HEAD |
| 23 | names a missing branch no longer renders as empty. |
| 24 | - [x] CI artifact system |
| 25 | - done per `docs/ci-artifacts.md`: `artifacts:` in ci.yml with |
| 26 | in-container meta extractors, broker collection via |
| 27 | `download_from_container`, run-page list, downloads + |
| 28 | `/{owner}/{repo}/artifacts/{rev}/{name}` alias + `browse: true` static |
| 29 | sites, quota GC with branch-tip pinning, and a schema shim so the table |
| 30 | appears on existing DBs. |
| 31 | - verified end-to-end against real Docker incl. the GC path; rustdoc was |
| 32 | the test case (big HTML subtree served like pages) |
| 33 | - repo-deletion cleanup deferred: repo deletion doesn't exist |
| 34 | - [x] repo mirroring to GitHub (push mirror) |
| 35 | - per-repo "Mirror push URL" in settings (`repositories.mirror_url`, |
| 36 | column shim for existing DBs); after every successful push over HTTP or |
| 37 | SSH a background mirror push fires |
| 38 | - **pure gitoxide** (design rule in CLAUDE.md — no git binary): gix can't |
| 39 | push yet, so `anvil-git/src/push.rs` implements the send-pack client |
| 40 | itself — advertisement parse, mirror commands honoring `delete-refs`, |
| 41 | `gitserver_core::pack::build_raw_pack`, report-status — over smart |
| 42 | HTTP(S) (reqwest + rustls/*ring*; no aws-lc, musl zigbuild verified) or |
| 43 | a local path served by gitserver-core in-process |
| 44 | - credentials ride in the URL |
| 45 | (`https://x-access-token:<token>@github.com/...`), stored as-is and |
| 46 | redacted from logs; verified e2e: anvil→anvil over real HTTP incl. |
| 47 | branch-deletion propagation, cloned back with real git |
| 48 | - building this surfaced and fixed three gitserver-core server bugs: |
| 49 | (1) `git clone` of any repo containing an *annotated tag* died |
| 50 | mid-transfer on protocol v2 — tag-object wants weren't peeled into the |
| 51 | pack walk (regression test `tag_object_wants_are_peeled_and_packed`); |
| 52 | (2) the receive-pack advertisement peeled tag refs, so re-pushing an |
| 53 | identical annotated tag was wrongly rejected (v0 advertisement now |
| 54 | also emits proper `^{}` peeled lines); |
| 55 | (3) ref deletion was prohibited outright — now advertised and allowed |
| 56 | via `delete-refs`, with the HEAD (default) branch protected |
| 57 | - [x] per-repository issue tracker |
| 58 | - `Issue`/`IssueComment` models with schema shims; per-repo numbering |
| 59 | (#1, #2, ...); list page with open/closed tabs, new-issue form, detail |
| 60 | page with markdown bodies and comments, close/reopen (issue author or |
| 61 | repo writer) |
| 62 | - any logged-in reader can open issues and comment; visibility follows |
| 63 | the repo; "Issues" link in the repo nav; CSRF on all forms; verified |
| 64 | e2e through the login + form flow |
| 65 | - no labels/assignees/editing yet — deliberately minimal |
| 66 | - [x] render a root `README.md` below the file tree on the repo page |
| 67 | - any case of `readme.md` at the root; reuses `render_markdown`, links to |
| 68 | the blob view from the box header |
| 69 | - [x] push-to-create: `git push` to a repo that doesn't exist yet creates it |
| 70 | - it did NOT already work — both transports 404'd. Policy in |
| 71 | `repos::create_on_push`: pusher must own the namespace or be admin; |
| 72 | created repos are private. |
| 73 | - over HTTP a missing repo now answers the receive-pack advertisement |
| 74 | with a Basic challenge so git prompts; verified e2e over both HTTP and |
| 75 | SSH, incl. the cross-namespace denial |
| 76 | - [x] make SSH the default clone selection, first in the pill buttons |
| 77 | - only when `[ssh] enabled`; HTTP stays the lone pill otherwise |
| 78 | - [x] rename the `anvil` crate to `anvil_cli` |
| 79 | - named it `anvil-cli` to match the workspace's hyphenated crate names; |
| 80 | the binary is still `anvild`, so deploy scripts and docs needed no |
| 81 | changes |
| 82 | - [x] fix git push failing on thin packs (REF_DELTA) |
| 83 | - root cause: every push after the first sends a thin pack; gitserver-core |
| 84 | passed `None` as `thin_pack_base_object_lookup` to |
| 85 | `gix_pack::Bundle::write_to_directory`, so gix couldn't resolve the |
| 86 | delta bases and aborted ("Ref delta objects are not supported..."). |
| 87 | First-push-to-empty-repo worked because that pack is self-contained. |
| 88 | - fix: pass the already-open `gix::Repository` as the lookup (it |
| 89 | implements `gix_object::Find`). Regression test |
| 90 | `receive_thin_pack_with_ref_deltas` builds a real thin pack via |
| 91 | `git pack-objects --thin` and pushes it through `receive_pack`; |
| 92 | verified the test fails without the fix. |
| 93 | |
| 94 | # Session notes / resume point |
| 95 | |
| 96 | _Last updated: 2026-06-10 (third session). Working state is clean: build, |
| 97 | clippy, fmt, sort-derives, and `cargo test --workspace` all pass (the |
| 98 | pre-commit hook runs all of these)._ |
| 99 | |
| 100 | ## Done this session (2026-06-10, third session) |
| 101 | |
| 102 | All six remaining TODO items — see `# Done [x]` above for details. Headlines: |
| 103 | |
| 104 | - **CI artifacts** (`docs/ci-artifacts.md` is the canonical reference): |
| 105 | `artifacts:` in `.anvil/ci.yml`; broker pulls them out of the stopped |
| 106 | container via `download_from_container` (inverse of the checkout upload — |
| 107 | still no mounts); meta extractors run *inside* the sandbox, one file per |
| 108 | value under `/tmp/anvil-meta`; `browse: true` directories are served like |
| 109 | pages (rustdoc-ready); `/{owner}/{repo}/artifacts/{rev}/{name}` is the |
| 110 | latest-on-branch alias; per-repo quota GC pins branch tips. New table |
| 111 | `ci_artifacts` + `[ci] artifact_*_mb` caps. |
| 112 | - **Schema shims for existing DBs** (`anvil-core/src/db.rs`): Toasty still |
| 113 | only pushes schema on a fresh file, so new tables/columns ship as |
| 114 | idempotent DDL applied on connect (`SCHEMA_SHIMS`/`COLUMN_SHIMS`), with |
| 115 | tests asserting fresh and migrated databases converge. New deps: rusqlite |
| 116 | (pinned to toasty's), flate2. |
| 117 | - **Issues** (`anvil-core/src/issues.rs`, `anvil-web/src/issues.rs`): |
| 118 | minimal GitHub-shaped tracker; tables `issues` + `issue_comments`. |
| 119 | - **Push mirroring** (`anvil-git/src/mirror.rs`): `repositories.mirror_url` |
| 120 | → background `git push --mirror` after each push; Dockerfile now installs |
| 121 | git (the one thing gix can't do yet is push). |
| 122 | - **Push-to-create** (`repos::create_on_push` + both transports), **README |
| 123 | on repo page**, **SSH-first clone pills**, **rev-switcher/browse-at-rev UI** |
| 124 | (committed earlier this session, along with the `anvil-cli` rename). |
| 125 | |
| 126 | ## Done earlier (2026-06-10, second session) |
| 127 | |
| 128 | - **UI quick wins** (`crates/anvil-web/src/ui.rs`) — latest-commit bar on the |
| 129 | repo page (sha + subject + author/time, attached above the file box, links |
| 130 | to the commit); profile page no longer shows the email; repo header reads |
| 131 | `owner / repo` without the leading `anvil /`. |
| 132 | - **(c) sandboxed CI broker** — job containers now run with `cap_drop=ALL` + |
| 133 | `no-new-privileges` unconditionally, plus config-driven `pids_limit` (512), |
| 134 | `memory_mb`+swap (2048), `cpus` (2), wall-clock `timeout_secs` (1800, |
| 135 | force-removed on expiry), optional `network = false`, `run_as`, and an |
| 136 | `allowed_images` allowlist (empty = any; tagless entry allows all tags). |
| 137 | `CiConfig` in `crates/anvil-core/src/config.rs` (with `image_allowed` test); |
| 138 | `execute()` in `crates/anvil-ci/src/lib.rs`. Read-only rootfs deliberately |
| 139 | skipped (workspace lives in the container fs; no volumes ever attached). |
| 140 | Docs: `DEPLOY.md` §7, `anvil.example.toml` `[ci]`. |
| 141 | - **(b) threat model** — `docs/untrusted-mode.md`: severity-ranked analysis |
| 142 | (CI containment, pages/stored-XSS origin, git resource exhaustion, |
| 143 | registration anti-abuse, authz granularity, webhook SSRF), the already-right |
| 144 | list, and the stance: single-tenant supported, untrusted gated on items 1–4. |
| 145 | - **Pages hosting** — `crates/anvil-web/src/pages.rs`: serves a repo's `pages` |
| 146 | branch at `/{owner}/{repo}/pages/...`; top-level dirs are separate sites |
| 147 | (rustdoc, book, …); `index.html` resolution with trailing-slash redirect so |
| 148 | relative links work; extension→content-type map + `nosniff`; listing page |
| 149 | with publish hint; "Pages" button on the repo header. Visibility follows the |
| 150 | repo (private → 404). Publish with `git push origin <built-branch>:pages`. |
| 151 | |
| 152 | ## Done earlier (same day, first session) |
| 153 | |
| 154 | - **CI UI** — runs list `/{owner}/{repo}/ci`, run-detail (status/timing/log), |
| 155 | per-commit status badges, "CI" nav link. (`crates/anvil-web/src/ui.rs`) |
| 156 | - **CD redeploy webhook** — on a green run of `[ci] deploy_branch` in the single |
| 157 | `[ci] deploy_repo`, POST to `[ci] deploy_webhook` (`X-Anvil-Deploy-Secret` |
| 158 | header). Scoped to ONE repo. `CiConfig` in `crates/anvil-core/src/config.rs`; |
| 159 | `deploy()` in `crates/anvil-ci/src/lib.rs`. Docs: `DEPLOY.md` §7, |
| 160 | `deploy/anvil.toml`. `reqwest` added with NO TLS feature (keeps musl |
| 161 | cross-compile aws-lc-free). |
| 162 | - **Docker socket on hagrid** — `deploy/run.sh` mounts it + `--group-add`s the |
| 163 | gid for the non-root user; caveat in `DEPLOY.md` §4. |
| 164 | - **Toasty ORM cleanup** — `ci.rs` `list_by_repo`/`latest_for_commit`/ |
| 165 | `queued_ids` now sort/limit/filter in SQL, not in memory. Verified by the new |
| 166 | `ordering_and_limit_run_in_the_database` test. (Sweep: these were the only |
| 167 | real instances; `repos::list_all_with_owner` sorts by a joined username and |
| 168 | needs all rows — intentionally left.) |
| 169 | - **(a) CSRF + cookie hardening** — |
| 170 | - Cookie: `HttpOnly` + `SameSite=Lax` + `Secure` (auto via |
| 171 | `Config::secure_cookies()` when base_url is https). |
| 172 | - Synchronizer token `HMAC-SHA256(server_secret, session)`; secret persisted |
| 173 | at `data_dir/csrf_secret` (`App::csrf_token` in `crates/anvil-core/src/lib.rs`). |
| 174 | Deps `hmac`, `sha2`. |
| 175 | - `Csrf` extractor + constant-time `verify_csrf` (`crates/anvil-web/src/auth.rs`). |
| 176 | Hidden `csrf` field + verification on add/delete SSH key, new repo, repo |
| 177 | settings. Login exempt; logout relies on SameSite. |
| 178 | - htmx insurance: `auth::csrf_context` middleware → request-scoped task-local; |
| 179 | `layout` sends the token via `hx-headers` on every htmx request. |
| 180 | |
| 181 | ## The a/b/c plan — ALL DONE |
| 182 | |
| 183 | (a) CSRF + cookie hardening, (b) `docs/untrusted-mode.md` threat model, |
| 184 | (c) sandboxed CI broker. Stronger isolation tiers (gVisor/Kata/Firecracker, |
| 185 | egress filtering, CI-minute quotas) are recorded in the threat model as the |
| 186 | gate for untrusted tenants, not planned work. |
| 187 | |
| 188 | ## Loose ends |
| 189 | |
| 190 | - **CSRF header consumption:** `hx-headers` sends the token as a `csrf` header, |
| 191 | but `verify_csrf` only reads the form field. When we add a tokenless htmx |
| 192 | action (raw `hx-post`/`hx-delete`, no `<form>`), also read the `csrf` header. |
| 193 | - **Toasty migrations:** still no real migration system; the shim approach in |
| 194 | `db::connect` (`SCHEMA_SHIMS` for tables, `COLUMN_SHIMS` for columns, both |
| 195 | test-verified against a fresh `push_schema`) covers what we've needed so |
| 196 | far. New columns must be declared last in the model. |
| 197 | - **Mirror URL secrecy:** `repositories.mirror_url` may embed a token and is |
| 198 | stored plaintext in SQLite (logged redacted). Fine single-tenant; revisit |
| 199 | if the DB ever leaves the box. |
| 200 | - **Pages caveats (single-tenant-acceptable):** served from the forge origin — |
| 201 | move to a separate origin before untrusted users (threat model §2); whole |
| 202 | blobs load into memory per request (fine at our scale). The same applies to |
| 203 | `browse: true` CI artifacts. |
| 204 | - **Issue tracker minimalism:** no labels, assignees, milestones, or |
| 205 | editing/deleting of posts. Per-repo numbering assumes a single server |
| 206 | process (matches deployment; noted in `models.rs`). |
| 207 | |
| 208 | ## Remaining roadmap (plan milestones beyond a/b/c) |
| 209 | |
| 210 | 9. Pull requests (gix merge) · 10. Webhooks (mind the SSRF item). |
| 211 | (8. Issues shipped 2026-06-10.) |