Files
Marcos Paulo d7617f4351
ci / ci (macos-latest) (push) Has been cancelled
ci / ci (ubuntu-latest) (push) Has been cancelled
ci / ci (windows-latest) (push) Has been cancelled
feat: server revisions in the Versions tab, collaboration research
- engine: ServerClient file_history, file_revision and restore; commands server.history, server.revision, server.restore (bound vault only, 5 MiB cap, no token in any answer)
- ui-egui: the Versions tab lists the server's revisions under the local snapshots with Compare (diff against the note) and a two-click Restore
- specs: collaboration research (browsed, 866 lines, 211 sources) and its ranked next features added to the collaboration task list

Verified with cargo xtask ci (fmt, clippy -D warnings, workspace tests, layers, tree, i18n).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 12:07:43 -03:00

4.4 KiB

Attachments (chunking, resumable upload, dedup) and server version history

Source: research recommendations 5, 8, 9 (security-and-server.md, sync-landscape.md).

Problem

Files over a few MiB are sent whole in one request (25 MiB cap): a flaky mobile link restarts from zero, the same photo in two vaults is stored twice, and old versions of notes are lost on the server (only local snapshots exist). Obsidian Sync offers 12 months of note versions; sync clients like Syncthing and Dropbox chunk large files.

Goal

Large attachments sync reliably on bad networks; every note version is recoverable from the server for a retention window.

Non-Goals

Real-time delta sync of Markdown (files are small), cross-vault dedup in encrypted vaults (a deliberate leak we avoid), a general file manager.

Design

  1. Blob chunking. Files above 8 MiB upload as chunks of 4 MiB content-addressed by SHA-256: PUT /v1/vaults/{v}/chunks/{hash} (idempotent, size-capped), then the file record points to an ordered chunk list (PUT /files/{path} with X-Chunks: h1,h2,...). POST /batch/check already finds missing hashes; extended to chunk hashes. Resume = ask which chunks exist and send the rest. Server verifies every chunk's hash and the assembled size.
  2. Dedup. Identical chunks are stored once per vault (single owner) and, for plaintext vaults only, once per server across vaults of the same user (documented equality leak).
  3. Version history. The ledger keeps every applied record as a revision (file_revisions(vault, path, seq, hash, size, mtime, editor, deleted)); blobs of old revisions stay until retention expires. Retention: default 30 days and at least the last 20 revisions per file, per-vault setting up to 365 days; the GC (POST /gc) honours it. Endpoints: GET /files-history/{path} (list), GET /files/{path}?seq=N (bytes of a revision), POST /files-restore/{path} {seq} (writes the old bytes as a new revision, never rewrites history).
  4. Client. The sync driver chunks big files, retries failed chunks with backoff, shows per-file progress; the Versions tab lists server revisions (merged with local snapshots) and restores.
  5. Quota. Counts unique chunk bytes plus retained revisions; the owner sees the split.

Acceptance Criteria

  • Given a 60 MB file and a connection cut after 21 MB, when sync resumes, then at most one chunk (4 MiB) is re-sent and the final hash matches.
  • Given the same 30 MB attachment in two notes of one vault, then the server stores its chunks once.
  • Given a note edited 40 times over 3 days, then history lists the revisions with times and editors, and restoring revision 12 creates revision 41 holding revision 12's bytes.
  • Given revisions older than retention, then GC removes their unique blobs and bytes_used drops; newer ones stay.
  • Given a chunk with a wrong hash, then the server answers 400 and stores nothing.

Open Questions

  1. Retention default (30 days or 12 months like Obsidian, which counts toward quota)? Recommended: 30 days, owner can raise.
  2. Content-defined chunking (rolling hash) instead of fixed 4 MiB? Recommended: fixed first; revisit if edits to big files are common.

Tasks

  1. Server: chunks table and endpoints, assembled-file reads, quota accounting. Done 2026-10-20 by GLM 5.3 Flash via Pi.
  2. Server: file_revisions, history, restore, retention GC. Done 2026-10-20 by GLM 5.3 Flash via Pi (MiniMax was out of quota): src/history/, store/revisions.rs, endpoints files-history, files/{path}?seq=, files-restore, per-vault retention_days (default 30, newest 20 always kept), quota counts retained revisions; verified by the orchestrator (all server suites green, clippy clean). Engine and UI still open (task 4).
  3. Engine: chunked upload in the driver with resume and progress events (sync/chunks.rs, Remote::{missing_chunks, put_chunk, put_chunked}, SyncStatus.transfer). Done 2026-10-20 by MiniMax M3.1 Flash through the Claude Code harness; verified against the real server. Big-file download still goes through the existing get (capped by what fits in memory).
  4. UI: [x] Versions tab shows server revisions with Compare and Restore (done 2026-10-20: engine server.history, server.revision, server.restore; started by GLM 5.3 Flash, finished by the orchestrator after the agent was cut off; 15 tests). Open: transfer progress in the status pill and Settings > Sync.