anvilsign in

collin/anvil · 0941637d

feat: mirror attachments into a git side ref for credential-free pulls

Collin Richards · 2026-06-10 15:17 UTC · 0941637d812deb5fe42de47f9c8950ea16b0ad3b · parent f99da5b4 · browse files

modified.gitignore+2 −0
⋯ 4 unchanged lines
55 *.db-shm
66 anvil.toml
77 /deploy/anvild
8+# Local-only API credentials for fetching attachments (never committed).
9+/.anvil-credentials
modifiedCLAUDE.md+20 −9
⋯ 19 unchanged lines
2020 ## Viewing attachments referenced in tasks
2121
2222 TODO items and tickets may embed an uploaded image as
23-`![...](/{owner}/{repo}/-/attachments/{hash})`. These bytes live outside git on
24-the anvil instance, so to actually *see* one, fetch it from the instance and
25-`Read` the file. With `ANVIL_BASE_URL` (the instance URL) and `ANVIL_TOKEN` (a
26-read-only PAT — mint one with `anvild user token create <user>`) set:
23+`![...](/{owner}/{repo}/-/attachments/{hash})`. These bytes live outside git
24+history, so to actually *see* one, get the bytes and `Read` the file. Two ways:
2725
26+**Preferred — over git, no credentials.** anvil mirrors every attachment into
27+`refs/anvil/attachments` (a flat tree of `<hash> → blob`, off the branch
28+namespace so a default pull never drags it down). From a clone, opt in once:
29+
2830 ```
31+git fetch origin '+refs/anvil/attachments:refs/anvil/attachments'
32+git cat-file -p "refs/anvil/attachments:<hash>" > /tmp/att && # then Read /tmp/att
33+```
34+
35+This uses the clone's existing git auth — no PAT — and works offline afterward.
36+
37+**Fallback — HTTP with a PAT** (no clone, or the ref isn't fetched). Credentials
38+live in the git-ignored `.anvil-credentials` at the repo root (`ANVIL_BASE_URL` +
39+a read-only `ANVIL_TOKEN`); `source` it, then:
40+
41+```
2942 curl -fsS -H "Authorization: Bearer $ANVIL_TOKEN" \
30- "$ANVIL_BASE_URL/{owner}/{repo}/-/attachments/{hash}" -o /tmp/att.png
43+ "$ANVIL_BASE_URL/{owner}/{repo}/-/attachments/{hash}" -o /tmp/att && # Read it
3144 ```
3245
33-Then `Read /tmp/att.png` to view it. The PAT is read-only (it authenticates
34-GET/HEAD only), so it's safe to hold; public-repo attachments need no token.
35-If those env vars aren't set, ask the user for the instance URL and a token
36-rather than guessing.
46+The PAT is read-only (GET/HEAD only), safe to hold. If neither the ref nor
47+`.anvil-credentials` is available, ask the user rather than guessing.
3748
3849 Current status, resume notes, the agreed next steps (a/b/c), and the roadmap live
3950 in the TODO. Read it first:
⋯ 2 unchanged lines
modifiedTODO.md+7 −0
⋯ 42 unchanged lines
4343 - [x] caps: per-repo attachment quota (`http.attachment_quota_mb`, 0 =
4444 unlimited) — a new upload over the cap is rejected; deduped re-uploads are
4545 always free. (Reject, not evict: evicting would break live Markdown links.)
46+- [x] carry attachments over git, credential-free: anvil mirrors each upload
47+ into `refs/anvil/attachments` (flat `hash → blob` tree, off the branch
48+ namespace). A default pull never fetches it; opt in with
49+ `git fetch origin '+refs/anvil/attachments:refs/anvil/attachments'` then
50+ `git cat-file -p refs/anvil/attachments:<hash>`. Disk+DB stay canonical;
51+ the ref is a downstream mirror (`anvil-git::attachments_ref`). All uploads
52+ remain web-only.
4653 - [ ] within-repo reclaim: an orphan sweep (delete attachments no committed file
4754 references) and/or a per-attachment delete action — the recourse once a repo
4855 hits its quota. Deferred: deletion is destructive and "orphaned" is fuzzy
⋯ 45 unchanged lines
addedcrates/anvil-git/src/attachments_ref.rs+171 −0
1+//! The attachments side ref: `refs/anvil/attachments`.
2+//!
3+//! Attachments are uploaded through the web UI and stored canonically on disk
4+//! (see `anvil-core`'s `attachments`), never in git history. To let a clone
5+//! pull them *without* a separate credential — git transport already carries
6+//! its own auth — anvil mirrors each stored blob into a single ref outside the
7+//! branch namespace: a commit whose flat tree maps `<sha256> → blob`.
8+//!
9+//! Because it isn't a branch or tag, a default `git fetch` never touches it; a
10+//! client opts in with an explicit refspec, then reads a blob by hash with
11+//! `git cat-file -p refs/anvil/attachments:<sha256>`.
12+
13+use std::path::Path;
14+
15+use gix::objs::tree;
16+
17+use crate::error::{
18+ Error,
19+ Result,
20+};
21+
22+/// The ref anvil mirrors attachments into. Off the branch/tag namespaces, so
23+/// it never rides a default pull.
24+pub const REF: &str = "refs/anvil/attachments";
25+
26+fn read(e: impl std::fmt::Display) -> Error {
27+ Error::Read(e.to_string())
28+}
29+
30+/// Mirror one attachment (`hash` → `content`) into [`REF`], creating or
31+/// advancing the ref. Idempotent: re-adding the same hash with the same bytes
32+/// is a no-op commit-wise only if nothing changed, but is always safe to call.
33+/// The ref is server-owned, so the update is an unconditional force (no CAS).
34+pub fn add(repo_path: &Path, hash: &str, content: &[u8]) -> Result<()> {
35+ let repo = gix::open(repo_path).map_err(read)?;
36+
37+ // Current ref tip (if any) and its tree, else start empty.
38+ let parent = crate::browse::resolve_commit(repo_path, REF)
39+ .ok()
40+ .and_then(|hex| gix::ObjectId::from_hex(hex.as_bytes()).ok());
41+ let mut tree: gix::objs::Tree = match parent {
42+ Some(commit_id) => {
43+ let tree_id = repo
44+ .find_object(commit_id)
45+ .map_err(read)?
46+ .peel_to_commit()
47+ .map_err(read)?
48+ .tree_id()
49+ .map_err(read)?
50+ .detach();
51+ let obj = repo.find_object(tree_id).map_err(read)?;
52+ gix::objs::TreeRef::from_bytes(&obj.data, gix::hash::Kind::Sha1)
53+ .map_err(read)?
54+ .into()
55+ }
56+ None => gix::objs::Tree {
57+ entries: Vec::new(),
58+ },
59+ };
60+
61+ let blob_id = repo.write_blob(content).map_err(read)?.detach();
62+ match tree.entries.iter_mut().find(|e| e.filename == hash) {
63+ Some(entry) => {
64+ if entry.oid == blob_id {
65+ return Ok(()); // already mirrored, identical bytes
66+ }
67+ entry.oid = blob_id;
68+ }
69+ None => tree.entries.push(tree::Entry {
70+ mode: tree::EntryKind::Blob.into(),
71+ filename: hash.into(),
72+ oid: blob_id,
73+ }),
74+ }
75+ // git requires tree entries sorted by name; the entries are all blobs
76+ // (flat tree), so a plain filename sort matches git's ordering.
77+ tree.entries.sort();
78+ let tree_id = repo.write_object(&tree).map_err(read)?.detach();
79+
80+ let sig = gix::actor::Signature {
81+ name: "anvil".into(),
82+ email: "anvil@localhost".into(),
83+ time: gix::date::Time::now_local_or_utc(),
84+ };
85+ let commit = gix::objs::Commit {
86+ tree: tree_id,
87+ parents: parent.into_iter().collect(),
88+ author: sig.clone(),
89+ committer: sig,
90+ encoding: None,
91+ message: format!("attachment {hash}").into(),
92+ extra_headers: Vec::new(),
93+ };
94+ let commit_id = repo.write_object(&commit).map_err(read)?.detach();
95+
96+ use gix::refs::{
97+ Target,
98+ transaction::{
99+ Change,
100+ LogChange,
101+ PreviousValue,
102+ RefEdit,
103+ RefLog,
104+ },
105+ };
106+ let name: gix::refs::FullName = REF
107+ .try_into()
108+ .map_err(|e: gix::validate::reference::name::Error| read(e))?;
109+ repo.edit_reference(RefEdit {
110+ change: Change::Update {
111+ log: LogChange {
112+ mode: RefLog::AndReference,
113+ force_create_reflog: false,
114+ message: "mirror attachment".into(),
115+ },
116+ expected: PreviousValue::Any,
117+ new: Target::Object(commit_id),
118+ },
119+ name,
120+ deref: false,
121+ })
122+ .map_err(read)?;
123+ Ok(())
124+}
125+
126+#[cfg(test)]
127+mod tests {
128+ use super::*;
129+
130+ fn git(dir: &Path, args: &[&str]) -> std::process::Output {
131+ std::process::Command::new("git")
132+ .args(args)
133+ .current_dir(dir)
134+ .env("GIT_AUTHOR_NAME", "t")
135+ .env("GIT_AUTHOR_EMAIL", "t@example.com")
136+ .env("GIT_COMMITTER_NAME", "t")
137+ .env("GIT_COMMITTER_EMAIL", "t@example.com")
138+ .output()
139+ .expect("run git")
140+ }
141+
142+ #[test]
143+ fn mirrors_blobs_addressable_by_hash() {
144+ let tmp = tempfile::tempdir().unwrap();
145+ let dir = tmp.path();
146+ assert!(git(dir, &["init", "-q", "-b", "main"]).status.success());
147+ std::fs::write(dir.join("f"), "seed").unwrap();
148+ git(dir, &["add", "."]);
149+ git(dir, &["commit", "-qm", "seed"]);
150+
151+ add(dir, "aaaa", b"first bytes").unwrap();
152+ add(dir, "bbbb", b"second bytes").unwrap();
153+ // Re-adding identical content is a no-op; new content updates in place.
154+ add(dir, "aaaa", b"first bytes").unwrap();
155+ add(dir, "aaaa", b"first bytes v2").unwrap();
156+
157+ // Each blob is retrievable by its hash via the side ref.
158+ let out = git(dir, &["cat-file", "-p", &format!("{REF}:bbbb")]);
159+ assert!(out.status.success());
160+ assert_eq!(out.stdout, b"second bytes");
161+ let out = git(dir, &["cat-file", "-p", &format!("{REF}:aaaa")]);
162+ assert_eq!(out.stdout, b"first bytes v2");
163+
164+ // The ref is off the branch namespace — main still has just its commit.
165+ let log = git(dir, &["log", "--oneline", "main"]);
166+ assert_eq!(String::from_utf8_lossy(&log.stdout).lines().count(), 1);
167+
168+ // fsck stays clean after our hand-built objects.
169+ assert!(git(dir, &["fsck", "--strict"]).status.success());
170+ }
171+}
modifiedcrates/anvil-git/src/lib.rs+1 −0
⋯ 12 unchanged lines
1313 //! protocol-v2 / upload-pack / receive-pack), behind this crate's interface so
1414 //! it can be replaced incrementally with an upstream-shaped implementation.
1515
16+pub mod attachments_ref;
1617 pub mod browse;
1718 pub mod edit;
1819 pub mod error;
⋯ 14 unchanged lines
modifiedcrates/anvil-web/src/attachments.rs+9 −1
⋯ 146 unchanged lines
147147 headers: HeaderMap,
148148 body: Bytes,
149149 ) -> Response {
150- let (_, meta) = match resolve_repo(&app, user.as_ref(), &owner, &repo).await {
150+ let (repo_path, meta) = match resolve_repo(&app, user.as_ref(), &owner, &repo).await {
151151 Ok(v) => v,
152152 Err(resp) => return resp,
153153 };
⋯ 58 unchanged lines
212212 return server_error(e);
213213 }
214214
215+ // Mirror into the attachments side ref so clones can fetch it over git
216+ // (credential-free, opt-in). Best-effort: the canonical store already has
217+ // it and serving works regardless, so a mirror failure must not fail the
218+ // upload — just log it.
219+ if let Err(e) = anvil_git::attachments_ref::add(&repo_path, &hash, &body) {
220+ tracing::warn!("attachment ref mirror failed for {hash}: {e}");
221+ }
222+
215223 let url = format!("/{owner}/{repo}/-/attachments/{hash}");
216224 let markdown = format!("![image]({url})");
217225 axum::Json(serde_json::json!({ "url": url, "markdown": markdown })).into_response()
⋯ 29 unchanged lines