collin/anvil
a5967525842c191da6222ccc680929935e41a68c / TODO.md
| 1 | # Todo |
| 2 | |
| 3 | ## ability to link to deployed / live site |
| 4 | |
| 5 | ## agent view, we have list of repos what about list of agents |
| 6 | |
| 7 | # Backlog |
| 8 | |
| 9 | - [ ] finish the CI pipeline move to TOML: rename `.anvil/ci.yml` to |
| 10 | `.anvil/ci.toml` (and convert it) in every repo on the instance, then delete |
| 11 | `ci::LEGACY_PIPELINE_PATH` and the legacy branch of `load_pipeline` / |
| 12 | `enqueue_ci_for_push`. Until then a stale `.anvil/ci.yml` still enqueues a |
| 13 | run, which fails telling you to rename the file — the point being that CI |
| 14 | going quiet is louder than CI going missing. |
| 15 | |
| 16 | - [ ] agent sessions, next milestones (docs/agent-sessions.md): |
| 17 | - a real checkout: the container clones from anvil's smart-HTTP endpoint and |
| 18 | pushes `agent/<id>` back. Needs a session-scoped push credential, which |
| 19 | does not exist (tokens are read-only, Bearer only on GET/HEAD) |
| 20 | - ref-scope that credential to `refs/heads/agent/*` — needs a ref filter in |
| 21 | receive-pack. Until it lands a session credential could write `main` |
| 22 | - trigger surfaces: a start button on a TODO item, an issue, a red CI run |
| 23 | - rate limiting, so automated pushes can't queue sessions endlessly once |
| 24 | triggers exist (`max_concurrent` bounds concurrency, not churn) |
| 25 | - a finished session's transcript rendered on its page (it is already on |
| 26 | disk under `sessions/<id>.log`; nothing reads it back yet) |
| 27 | |
| 28 | - [ ] pull requests (gix merge) |
| 29 | - [ ] pull mirror (maybe): a repo that virtually mirrors a GitHub repo |
| 30 | - just displays it here — periodically fetched, read-only on the anvil side |
| 31 | |
| 32 | - [ ] richer file editing: a real markdown editor with a live render preview |
| 33 | (reuse `render_markdown`) before committing |
| 34 | - [ ] webhooks (mind the SSRF item in `docs/untrusted-mode.md`) |
| 35 | - [ ] attachment reclaim: an orphan sweep (delete attachments no committed file |
| 36 | references) and/or a per-attachment delete action — the recourse once a repo |
| 37 | hits its quota. Deferred: deletion is destructive and "orphaned" is fuzzy |
| 38 | (tip-only vs any-ref), so it wants its own design pass |
| 39 | - [ ] admin usage: per-repo drill-down, and a cheap cached/periodic variant if |
| 40 | the on-demand disk walk gets slow on large instances |
| 41 | - [ ] periodic disk usage cache: run `usage::compute()` on a timer (e.g., hourly) |
| 42 | and store the result so the admin dashboard doesn't block on disk walks |
| 43 | - [ ] repository preview images: extract the first "real" image (>few hundred px) |
| 44 | from README.md on a periodic scan, cache the attachment hash, and display in |
| 45 | repo listings for visual browsing |
| 46 | - [ ] API tokens: a `write` scope (would need CSRF-exempt write paths) and |
| 47 | `last_used_at` tracking |
| 48 | - [ ] single sign-on follow-ups (docs/oidc.md): silent renewal |
| 49 | (`prompt=none` on a short local session, which is what makes revoking an SSO |
| 50 | session propagate here), an admin view of who is linked to which `sub`, and |
| 51 | unlinking an account from the settings page |
| 52 | - [ ] secrets follow-ups (docs/secrets.md): authenticate `anvild secret` with an |
| 53 | ssh signature instead of the account password; per-step rather than per- |
| 54 | pipeline scoping; `ssh-rsa` recipients (needs an RSA-OAEP branch in both the |
| 55 | Rust and the browser halves) |
| 56 | |
| 57 | ### Ideas from Origin https://cursor.com/blog/git-at-any-scale |
| 58 | |
| 59 | Cursor's writeup of Continuity, their Spokes replacement. Most of the post is |
| 60 | scale machinery anvil does not need (replicas, consensus, rendezvous hashing, |
| 61 | S3 as the source of truth) because that exists to serve a monorepo's CI from a |
| 62 | hundred read replicas. Three things do transfer, ranked by value per line. |
| 63 | |
| 64 | - [ ] **packfile compaction. anvil has none at all.** Every push writes a new |
| 65 | pack via `gix_pack::Bundle::write_to_directory` |
| 66 | (`vendor/gitserver-core/src/receive_pack.rs`, `write_pack`) and nothing ever |
| 67 | consolidates them. Object lookup is O(packs) because each index is only |
| 68 | efficient per-pack, so every push makes every later browse, clone and CI |
| 69 | checkout slower, permanently. The periodic runner already exists |
| 70 | (`crates/anvil-core/src/periodic.rs`), so this is a new `PeriodicJob`, not new |
| 71 | infrastructure. Two levels: |
| 72 | - multi-pack-index via `gix_pack::multi_index::File::write_from_index_paths` |
| 73 | (pure gix, no CLI). One binary-searchable lookup across all packs. Start |
| 74 | here: small, self-contained, fixes a problem already accumulating |
| 75 | - geometric repack via `gix_pack::data::output`, the same machinery |
| 76 | upload-pack uses to build packs. More work. The post's warning about |
| 77 | repacking being an availability hazard is a replica problem; with one |
| 78 | node there is nothing to fail over |
| 79 | |
| 80 | - [ ] **SQLite is not in WAL mode.** `db::connect` |
| 81 | (`crates/anvil-core/src/db.rs`) sets no pragmas, so it runs on the default |
| 82 | rollback journal with web, ssh, the CI dispatcher and four periodic jobs all |
| 83 | against one file in one process. Readers block the writer, and copying a live |
| 84 | `.db` under a rollback journal can yield a corrupt file, which makes the |
| 85 | documented backup (tar the running volume, DEPLOY.md § Operations) unsound. |
| 86 | `PRAGMA journal_mode=WAL` plus `busy_timeout`. |
| 87 | |
| 88 | - [ ] **a push log (the WAL idea, minus S3, consensus and replicas).** What |
| 89 | transfers is the observation that the pack bytes plus the ref transaction are |
| 90 | a complete description of a push, so recording them stops the disk from being |
| 91 | precious. Both are already in scope at one place: `apply_commands` |
| 92 | (`vendor/gitserver-core/src/receive_pack.rs`) writes the pack, builds `edits`, |
| 93 | then calls `edit_references`. The entry goes between those two steps. |
| 94 | - buys, in order of how much we would use it: force-push undo as a UI button |
| 95 | (every ref's prior value is recorded), a reflog that survives gc, |
| 96 | rebuildable repos, and eventually continuous off-box backup |
| 97 | - keep it simple by ordering it right: a local append-only file first. No S3 |
| 98 | client, no dependency, no network in the push path. That alone gets undo |
| 99 | and provenance. Shipping entries off-box is a separate additive step |
| 100 | - the rule that makes it worth anything, and the easy one to skip: do not |
| 101 | ack the push until the entry is durable. A log that might be missing the |
| 102 | entry you need is worse than no log, because you will trust it |
| 103 | - caveat: this covers git only. Issues, users, CI runs, secrets, attachments |
| 104 | and artifacts are not in it. Build this and keep tarring the volume for |
| 105 | the rest and we have added a system without retiring one, so either frame |
| 106 | it as provenance plus undo (a feature) rather than backup (an ops story), |
| 107 | or pair it with continuous SQLite replication so both halves match. |
| 108 | |
| 109 | Related, prompted by the post rather than in it: `App` |
| 110 | (`crates/anvil-core/src/lib.rs`) mixes durable state (`db`, disk) with |
| 111 | process-local state (`ci_tx`, `vault`, `user_vault`, `sessions`, `jobs`) with |
| 112 | nothing marking which is which. Each in-memory field is documented as "lost on |
| 113 | restart, which is safe because...", which is correct, but the invariant lives in |
| 114 | comments. The discipline underneath Continuity is knowing exactly what is truth |
| 115 | and what is cache. Costs nothing at one process; first thing to break at two. |