anvilsign in

collin/anvil · d0d2c17b

docs: record the gitoxide mirroring rework + protocol fixes in TODO

Collin Richards · 2026-06-10 10:18 UTC · d0d2c17b7e3a1d08ce52b9bf773a161b55af19e8 · parent cbf2877d · browse files

modifiedTODO.md+88 −111
1-# Misc TODO
1+# Todo
2+
3+- [ ] pull mirror (maybe): a repo that virtually mirrors a GitHub repo
4+ - just displays it here — periodically fetched, read-only on the anvil side
5+
6+# Done [x]
27
3-- [x] do we have a way to view the source in a repo by branch or at a given commit?
4- - yes: `/{owner}/{repo}/tree/{rev}` always accepted any branch/tag/commit;
5- what was missing was UI. Added: branch/tag switcher dropdown on the repo
6- and tree pages, "browse files" link on the commit page, percent-encoded
7- ref names so branches with `/` work, and an unborn-HEAD fallback so a
8- repo whose HEAD names a missing branch no longer renders as empty.
9-- [x] implement the CI artifact system _(done per `docs/ci-artifacts.md`:
10- `artifacts:` in ci.yml with in-container meta extractors, broker
11- collection via `download_from_container`, run-page list, downloads +
8+- [x] render `TODO.md` per the todo-md spec (`~/Code/todo-md`) as a kanban board
9+ - `anvil-web/src/todomd.rs`: spec parser (sections by heading, `[x]`-marked
10+ done sections, checkbox tasks, indented details, fence-aware) + board
11+ renderer; prose sections render as a collapsible notes area below the
12+ board
13+ - blob pages for any `TODO.md` get Board/Rendered/Source pills (board
14+ default, falls back to markdown when the file has no tasks); the repo
15+ page shows the board in a box below the README
16+ - this file is itself spec-compliant now; structured updates come later
17+
18+- [x] browse source at any branch, tag, or commit
19+ - `/{owner}/{repo}/tree/{rev}` always accepted any rev; what was missing
20+ was UI. Added: branch/tag switcher dropdown on the repo and tree pages,
21+ "browse files" link on the commit page, percent-encoded ref names so
22+ branches with `/` work, and an unborn-HEAD fallback so a repo whose HEAD
23+ names a missing branch no longer renders as empty.
24+- [x] CI artifact system
25+ - done per `docs/ci-artifacts.md`: `artifacts:` in ci.yml with
26+ in-container meta extractors, broker collection via
27+ `download_from_container`, run-page list, downloads +
1228 `/{owner}/{repo}/artifacts/{rev}/{name}` alias + `browse: true` static
1329 sites, quota GC with branch-tip pinning, and a schema shim so the table
14- appears on existing DBs. Verified end-to-end against real Docker incl.
15- the GC path. Repo-deletion cleanup deferred: repo deletion doesn't exist.)_
16- - every push triggers CI on the tip commit (already true); a run should be
17- able to produce artifacts
18- - artifacts are stored per-commit and served from anvil (download from the
19- run page / commit page; latest-on-branch alias would be nice)
20- - jobs can declare how to extract metadata from artifacts (e.g. sizes,
21- test/coverage numbers, version strings) so it can be surfaced in the UI
22- next to the run/commit
23- - sketch needed: where artifacts live on disk, retention/GC, how
24- `.anvil/ci.yml` declares artifact paths + metadata extractors, and how
25- the broker gets files out of the sandboxed container
26- - use rustdoc as an example when testing the feature: it generates a big
27- HTML subtree, so support rendering/serving a whole HTML artifact subtree
28- the way the pages feature does (rustdoc output as a served site)
29-- [x] repo mirroring to GitHub (push mirror) _(per-repo "Mirror push URL" in
30- settings (`repositories.mirror_url`, column shim for existing DBs);
31- after every successful push over HTTP or SSH a background
32- `git push --mirror` fires — shells out to git (gix can't push yet;
33- runtime image now installs git). Credentials ride in the URL
30+ appears on existing DBs.
31+ - verified end-to-end against real Docker incl. the GC path; rustdoc was
32+ the test case (big HTML subtree served like pages)
33+ - repo-deletion cleanup deferred: repo deletion doesn't exist
34+- [x] repo mirroring to GitHub (push mirror)
35+ - per-repo "Mirror push URL" in settings (`repositories.mirror_url`,
36+ column shim for existing DBs); after every successful push over HTTP or
37+ SSH a background mirror push fires
38+ - **pure gitoxide** (design rule in CLAUDE.md — no git binary): gix can't
39+ push yet, so `anvil-git/src/push.rs` implements the send-pack client
40+ itself — advertisement parse, mirror commands honoring `delete-refs`,
41+ `gitserver_core::pack::build_raw_pack`, report-status — over smart
42+ HTTP(S) (reqwest + rustls/*ring*; no aws-lc, musl zigbuild verified) or
43+ a local path served by gitserver-core in-process
44+ - credentials ride in the URL
3445 (`https://x-access-token:<token>@github.com/...`), stored as-is and
35- redacted from logs. Verified e2e against a local bare "github".)_
36- - pull mirror (maybe, NOT done): a repo that virtually mirrors a GitHub
37- repo and just displays it here — periodically fetched, read-only on
38- the anvil side
39-- [x] per-repository issue tracker _(`Issue`/`IssueComment` models with
40- schema shims; per-repo numbering (#1, #2, …); list page with
41- open/closed tabs, new-issue form, detail page with markdown bodies and
42- comments, close/reopen (issue author or repo writer). Any logged-in
43- reader can open issues and comment; visibility follows the repo.
44- "Issues" link in the repo nav. CSRF on all forms. Verified e2e through
45- the login + form flow. No labels/assignees/editing yet — deliberately
46- minimal.)_
47-- [x] render a root `README.md` below the file tree on the repo page _(any
48- case of `readme.md` at the root; reuses `render_markdown`, links to the
49- blob view from the box header)_
46+ redacted from logs; verified e2e: anvil→anvil over real HTTP incl.
47+ branch-deletion propagation, cloned back with real git
48+ - building this surfaced and fixed three gitserver-core server bugs:
49+ (1) `git clone` of any repo containing an *annotated tag* died
50+ mid-transfer on protocol v2 — tag-object wants weren't peeled into the
51+ pack walk (regression test `tag_object_wants_are_peeled_and_packed`);
52+ (2) the receive-pack advertisement peeled tag refs, so re-pushing an
53+ identical annotated tag was wrongly rejected (v0 advertisement now
54+ also emits proper `^{}` peeled lines);
55+ (3) ref deletion was prohibited outright — now advertised and allowed
56+ via `delete-refs`, with the HEAD (default) branch protected
57+- [x] per-repository issue tracker
58+ - `Issue`/`IssueComment` models with schema shims; per-repo numbering
59+ (#1, #2, ...); list page with open/closed tabs, new-issue form, detail
60+ page with markdown bodies and comments, close/reopen (issue author or
61+ repo writer)
62+ - any logged-in reader can open issues and comment; visibility follows
63+ the repo; "Issues" link in the repo nav; CSRF on all forms; verified
64+ e2e through the login + form flow
65+ - no labels/assignees/editing yet — deliberately minimal
66+- [x] render a root `README.md` below the file tree on the repo page
67+ - any case of `readme.md` at the root; reuses `render_markdown`, links to
68+ the blob view from the box header
5069 - [x] push-to-create: `git push` to a repo that doesn't exist yet creates it
51- _(it did NOT already work — both transports 404'd. Policy in
70+ - it did NOT already work — both transports 404'd. Policy in
5271 `repos::create_on_push`: pusher must own the namespace or be admin;
53- created repos are private. Over HTTP a missing repo now answers the
54- receive-pack advertisement with a Basic challenge so git prompts.
55- Verified e2e over both HTTP and SSH, incl. the cross-namespace denial.)_
56-- [x] make SSH the default clone selection in the clone pill buttons, and put
57- it first (before HTTP) _(only when `[ssh] enabled`; HTTP stays the lone
58- pill otherwise)_
59-- [x] rename the `anvil` crate to `anvil_cli` _(named it `anvil-cli` to match
60- the workspace's hyphenated crate names; the binary is still `anvild`, so
61- deploy scripts and docs needed no changes)_
62-
63----
64-
65-# Error in git push _(FIXED 2026-06-10, uncommitted)_
66-
67-**Root cause:** every push after the first sends a *thin pack* — deltas whose
68-base objects aren't in the pack (the server already has them), referenced by
69-object id (`REF_DELTA`). `vendor/gitserver-core/src/receive_pack.rs::write_pack`
70-passed `None` as `thin_pack_base_object_lookup` to
71-`gix_pack::Bundle::write_to_directory`, so gix couldn't resolve the bases and
72-aborted. First-push-to-empty-repo worked because that pack is self-contained.
73-
74-**Fix:** pass the already-open `gix::Repository` as the lookup (it implements
75-`gix_object::Find`). Regression test
76-`receive_thin_pack_with_ref_deltas` builds a real thin pack via
77-`git pack-objects --thin`, asserts it contains ref-deltas, and pushes it
78-through `receive_pack`. Verified the test fails without the fix.
79-
80-Original report:
81-
82-```
83-collin@mini ~/C/anvil (main)> git push
84-Enter passphrase for key '/Users/collin/.ssh/id_ed25519':
85-Enumerating objects: 117, done.
86-Counting objects: 100% (117/117), done.
87-Delta compression using up to 8 threads
88-Compressing objects: 100% (62/62), done.
89-Writing objects: 100% (66/66), 27.57 KiB | 3.94 MiB/s, done.
90-Total 66 (delta 39), reused 0 (delta 0), pack-reused 0 (from 0)
91-send-pack: unexpected disconnect while reading sideband packet
92-fatal: the remote end hung up unexpectedly
93-collin@mini ~/C/anvil (main) [128]>
94-```
95-
96-server logs:
97-
98-```
99-26-06-09T21:53:46.466577Z INFO anvil_ssh: ssh auth: accepted key SHA256:Rg41caN7vw2WYYxiJN6lrIlX0DTXYF0rC2QzKZW1tB0 (user 1)
100-2026-06-09T21:53:46.635946Z INFO anvil_ssh: ssh git-receive-pack on collin/anvil.git (user Some(1))
101-2026-06-09T21:53:46.805146Z ERROR anvil_ssh: git ssh git-receive-pack: protocol error: failed to write incoming pack: Ref delta objects are not supported as there is no way to look them up. Resolve them beforehand.
102-2026-06-09T21:54:28.571834Z INFO anvil_ssh: ssh auth: rejected unknown key SHA256:ZlWZyHqspqFeUQV84qaXtDQq4gcA33dR7y8dYbeg9u8
103-2026-06-09T21:54:35.565597Z INFO anvil_ssh: ssh auth: accepted key SHA256:Rg41caN7vw2WYYxiJN6lrIlX0DTXYF0rC2QzKZW1tB0 (user 1)
104-2026-06-09T21:54:35.676113Z INFO anvil_ssh: ssh git-receive-pack on collin/anvil.git (user Some(1))
105-2026-06-09T21:54:36.268520Z ERROR anvil_ssh: git ssh git-receive-pack: protocol error: failed to write incoming pack: Ref delta objects are not supported as there is no way to look them up. Resolve them beforehand.
106-2026-06-09T21:55:54.223033Z INFO anvil_ssh: ssh auth: rejected unknown key SHA256:ZlWZyHqspqFeUQV84qaXtDQq4gcA33dR7y8dYbeg9u8
107-2026-06-09T21:56:00.758384Z INFO anvil_ssh: ssh auth: accepted key SHA256:Rg41caN7vw2WYYxiJN6lrIlX0DTXYF0rC2QzKZW1tB0 (user 1)
108-2026-06-09T21:56:00.906553Z INFO anvil_ssh: ssh git-receive-pack on collin/anvil.git (user Some(1))
109-2026-06-09T21:56:01.067903Z ERROR anvil_ssh: git ssh git-receive-pack: protocol error: failed to write incoming pack: Ref delta objects are not supported as there is no way to look them up. Resolve them beforehand.
110-```
111-
112----
72+ created repos are private.
73+ - over HTTP a missing repo now answers the receive-pack advertisement
74+ with a Basic challenge so git prompts; verified e2e over both HTTP and
75+ SSH, incl. the cross-namespace denial
76+- [x] make SSH the default clone selection, first in the pill buttons
77+ - only when `[ssh] enabled`; HTTP stays the lone pill otherwise
78+- [x] rename the `anvil` crate to `anvil_cli`
79+ - named it `anvil-cli` to match the workspace's hyphenated crate names;
80+ the binary is still `anvild`, so deploy scripts and docs needed no
81+ changes
82+- [x] fix git push failing on thin packs (REF_DELTA)
83+ - root cause: every push after the first sends a thin pack; gitserver-core
84+ passed `None` as `thin_pack_base_object_lookup` to
85+ `gix_pack::Bundle::write_to_directory`, so gix couldn't resolve the
86+ delta bases and aborted ("Ref delta objects are not supported...").
87+ First-push-to-empty-repo worked because that pack is self-contained.
88+ - fix: pass the already-open `gix::Repository` as the lookup (it
89+ implements `gix_object::Find`). Regression test
90+ `receive_thin_pack_with_ref_deltas` builds a real thin pack via
91+ `git pack-objects --thin` and pushes it through `receive_pack`;
92+ verified the test fails without the fix.
11393
11494 # Session notes / resume point
11595
11696 _Last updated: 2026-06-10 (third session). Working state is clean: build,
11797 clippy, fmt, sort-derives, and `cargo test --workspace` all pass (the
118-pre-commit hook runs all of these). Third-session work below is UNCOMMITTED
119-(repo convention: commit only when asked). **All top-of-file TODO items are
120-done** except the pull-mirror "maybe"._
98+pre-commit hook runs all of these)._
12199
122100 ## Done this session (2026-06-10, third session)
123101
124-All six remaining TODO items — see the checked-off entries at the top of this
125-file for the details. Headlines:
102+All six remaining TODO items — see `# Done [x]` above for details. Headlines:
126103
127104 - **CI artifacts** (`docs/ci-artifacts.md` is the canonical reference):
128105 `artifacts:` in `.anvil/ci.yml`; broker pulls them out of the stopped
⋯ 106 unchanged lines