Compare commits
68 Commits
v0.1.0-dev.2
...
develop
| Author | SHA1 | Date | |
|---|---|---|---|
| 18d02d3884 | |||
| aaaa58ff80 | |||
| 0437c4217a | |||
| 6fcbe3ed94 | |||
| 12c82f8b56 | |||
| cac6b0ed39 | |||
| c2b48bcba4 | |||
| c78fd488cc | |||
| e8c8c06d1e | |||
| 1cbc26429a | |||
| eb1245bd20 | |||
| b32650767c | |||
| 6ed829e5bd | |||
| 16a0786a2d | |||
| 179a5cdc47 | |||
| 3e3a9cf7eb | |||
| 86abc0c219 | |||
| 22b43c1254 | |||
| ca855c3ee4 | |||
| ca0d68f9ab | |||
| de3ec375ae | |||
| 4ea2e9fd5a | |||
| 6d3ba3096c | |||
| 77d538dd97 | |||
| b9d21de5b2 | |||
| 224ae2718a | |||
| 3694e73de8 | |||
| aeac86ddd2 | |||
| 00726d5498 | |||
| 5e9b5d3cfb | |||
| bc0b258e04 | |||
| 9993a66fee | |||
| 7f432138df | |||
| 08cdea58d7 | |||
| 4bc37caf04 | |||
| 68dd899844 | |||
| 2e23fd09a1 | |||
| 2be0e6586e | |||
| d396f409b9 | |||
| 7c8b21c2a5 | |||
| 1806455cd9 | |||
| 0af85cece0 | |||
| 6fe85202e1 | |||
| 1d06b71950 | |||
| 2ca3702db0 | |||
| c750fd2aa8 | |||
| cea7b45e23 | |||
| 72ad6bb611 | |||
| 3fdb7ea9e0 | |||
| 92ca84d293 | |||
| 24c2c33dc7 | |||
| 05afde4b7a | |||
| 9e6c2c7f66 | |||
| ab3f82f171 | |||
| 4cdc8b0cec | |||
| 79b61756d2 | |||
| 9d20cf3fd4 | |||
| 5d4a2c8fe1 | |||
| d9359ba93d | |||
| d7617f4351 | |||
| 9bd7dc931e | |||
| a71e294566 | |||
| 181aad91e6 | |||
| b562eeafde | |||
| 67bb5fe06f | |||
| 46d5c3ec96 | |||
| 6ecbeadcff | |||
| bb7f8ae79d |
@@ -0,0 +1,8 @@
|
||||
Disk is a shared, finite resource: a full system volume (`/private/tmp`) stops every agent and the harness itself (2026-10-21 incident, Bash returned ENOSPC). Rules for every agent and every build:
|
||||
- Build only in `target/agent-<name>` (absolute: `CARGO_TARGET_DIR=$PWD/target/agent-<name>`), never a relative path (it creates `crates/*/target`), never in `/tmp`, never in the user's `target/debug`.
|
||||
- Scratch copies: `rsync -a --delete --exclude 'target*/' --exclude '.git' --exclude '.notescraft' --exclude '.mm-agents' --exclude '/dist' --exclude '/.worktrees'` into ONE directory per agent under the scratchpad, never a second copy of a copy. Do not use the unanchored pattern `target*` (it drops `assets/icons/lucide/target.svg`): use `target*/` for directories.
|
||||
- When you finish (success or blocked), delete what you made: your scratch copy and your `target/agent-<name>`, then run `scripts/clean-agent-builds.sh --check` and put the free-space line in your report. Keep a build dir only when the next task of the same agent reuses it, and say so.
|
||||
- The lead deletes all `target/agent-*` and scratch copies after each commit with `scripts/clean-agent-builds.sh`. Never delete a dir that a running agent uses.
|
||||
- Before a long build run `df -h /System/Volumes/Data /Volumes/Storage`; under 20 GiB free on either: stop and clean first. Never write build output, copies or logs to the Data volume when `/Volumes/Storage` has room.
|
||||
- `scripts/` is gitignored (local agent tooling): the helper `scripts/clean-agent-builds.sh` lives there and in `~/.agents/scripts/`; if it is missing, do the same by hand (delete your `target/agent-<name>` and scratch copy).
|
||||
- Tests must delete the temp dirs they create (guard with `Drop`/`tempfile`-style cleanup; `std::env::temp_dir()` leaks otherwise: 21k `nc-*`/`notescraft-*` dirs ate 5 GB of the system volume). Never make a full copy of the repo or a baseline checkout in `$TMPDIR`/`/private/tmp`; copy to `/Volumes/Storage` (a baseline copy in `$TMPDIR/opencode/nc-fonts-before` took 4.7 GB). `scripts/clean-agent-builds.sh` purges `nc-*`/`notescraft-*` in `$TMPDIR`.
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
paths:
|
||||
- "crates/engine/**"
|
||||
- "crates/vault/**"
|
||||
---
|
||||
Behaviour is an engine command (namespaced id, typed params, `enabled`, `run`) registered like its neighbours; new write/delete commands go into the assistant gate lists. Disk writes only via `craft_atomic::atomic_write` or `Vault::write_private` (`.notescraft/`); never write `.obsidian/`; user deletes go to the vault trash. No egui in engine. Skill: `notescraft-engine-command`.
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
paths:
|
||||
- "crates/**/*.rs"
|
||||
- "apps/**/*.rs"
|
||||
---
|
||||
Non-test Rust here must never panic. No `unwrap`, `expect`, `panic!`, `todo!`, `unimplemented!`, `unreachable!`, `unsafe`. No `v[i]` or `s[a..b]` with an input-derived value, even guarded: use `get()`, `split_at_checked`, `checked_*`, `saturating_*`, `try_from`. Bound loops, recursion and allocations. Invalid input returns `Err`; never clamp or repair silently. Every pub item has rustdoc, every module `//!`. Files under about 400 lines, at most 12 loose `.rs` per directory. Clippy pedantic is `-D warnings`. Details and the lint-fix table: skill `notescraft-rust-safety`.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
paths:
|
||||
- "crates/sync/**"
|
||||
- "apps/notescraft-sync-server/**"
|
||||
- "crates/engine/src/sync/**"
|
||||
---
|
||||
Never lose data: doubt means a conflict copy; edit beats delete. Paths and hashes from the network pass `check_syncable` / `check_hash`, and downloaded bytes are hashed and compared. Tokens and pairing codes are stored only as SHA-256 hashes and never logged. Inject time as an argument in tests; never sleep minutes. Skill: `notescraft-sync-protocol`.
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
paths:
|
||||
- "**/tests/**/*.rs"
|
||||
---
|
||||
Test behaviour, not absence of panics. Write the failing test first. Run with `timeout 300 cargo test -p <crate>` and confirm the new tests are listed. Compute byte offsets with `str::find`. Unique temp dirs under `std::env::temp_dir()`. `unwrap`/`expect` are fine in tests; `assert!(x.is_empty())` is not (use `assert_eq!`). Skill: `notescraft-verify-loop`.
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
paths:
|
||||
- "crates/ui-egui/**"
|
||||
---
|
||||
No hex colours (theme tokens only), no literal user-facing text (i18n table: `crates/ui-egui/src/text/<panel>.rs` + `[ui.<panel>]` keys in `crates/i18n/locales/en.toml`; `cargo xtask i18n` must show 0). Named constants with unit docs. Stable widget ids. 44 pt touch targets in the compact layout (`design::style::compact`). Test by driving real events through frames and render offscreen to compare with `design/mobile/*.png`. Skill: `notescraft-egui-screen`.
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: notescraft-agent-workflow
|
||||
description: How to run a NotesCraft coding task end to end (read, plan, small steps, scope fences, no git, final report). Use at the start of every task, especially when working as a delegated agent in the shared tree.
|
||||
---
|
||||
|
||||
# Agent workflow for NotesCraft
|
||||
|
||||
## 1. Orient (before any edit)
|
||||
1. Read the task's OWNS, READ FIRST and DO NOT lists. OWNS is the only place you may write.
|
||||
2. Read `AGENTS.md`. Skim `ROADMAP.md` for the phase you touch and `docs/ARCHITECTURE.md` for the crate's public API.
|
||||
3. Read two or three neighbouring files in the crate and copy their style: naming, comment density, error type, test layout. Never invent a pattern that already exists.
|
||||
4. Find the real function you will call (`grep -rn "pub fn name" crates/`). Never guess a signature; open the file.
|
||||
|
||||
## 2. Plan in slices
|
||||
- One slice = one file or one function plus its test, at most about 150 lines.
|
||||
- Write the failing test first, see it fail for the right reason, then the code, then run the crate's tests.
|
||||
- One file per tool call, at most about 250 lines per response. A response that is too long is cut off and you stall.
|
||||
|
||||
## 3. Scope fences
|
||||
- Edit only OWNS. If you need a change outside it, stop and say so in the report; do not make it.
|
||||
- Never run `git add`, `commit`, `checkout`, `stash`, `reset`, `clean`, or delete files you did not create. Never touch `target/debug`.
|
||||
- Never edit root `Cargo.toml`, `xtask/`, `SPEC.md`, `ROADMAP.md`, `docs/`, or another agent's files unless OWNS lists them.
|
||||
- New third-party dependency: not allowed unless the task names it. Approved list is in `AGENTS.md`.
|
||||
- Never print or log secrets, tokens or pairing codes.
|
||||
|
||||
## 4. Stuck rules
|
||||
- Same error three times: stop, paste the exact error in the report, do not loop.
|
||||
- A test hangs or memory climbs: `pkill -9 -f <test binary>`, fix the loop (each iteration must consume input) before anything else.
|
||||
- Unsure whether something is allowed: treat it as not allowed and report.
|
||||
|
||||
## 5. Finish
|
||||
Run the `notescraft-verify-loop` commands, paste the last `cargo test` summary line, and report: what is implemented, public signatures, test counts, files changed, anything skipped. Max 15 lines. Do not claim success on a command you did not run.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: notescraft-egui-screen
|
||||
description: How to build or change a NotesCraft egui screen: theme tokens, i18n tables, touch targets, layout classes, driven tests, design comparison. Use for any ui-egui work.
|
||||
---
|
||||
|
||||
# egui screens in NotesCraft
|
||||
|
||||
Crate `crates/ui-egui` (layer L6, the only crate that may use egui). egui 0.36, eframe wgpu. No JavaScript, no webview.
|
||||
|
||||
## Colours and sizes
|
||||
- Never write a hex colour. Use the active theme tokens: `crate::theme::{Tokens, color}`; look at a neighbouring view (`views/trash_view/mod.rs`) for how `tokens` is passed and how `color(...)` helpers are used.
|
||||
- Sizes are named constants with a doc comment and a unit (`/// Height of one row, points.\nconst ROW_HEIGHT: f32 = 48.0;`). Spacing on a 4 pt scale (4, 8, 12, 16, 24, 32, 48).
|
||||
- Panels are rounded floating cards on the canvas colour separated by tone, not borders.
|
||||
- Touch: width class from `crate::design::style::compact` (`is_compact(ctx)`, `target(ctx, regular)`, `COMPACT_BELOW` = 600 pt, `MIN_TARGET_COMPACT` = 44 pt). Phone layouts need 44 pt hit areas.
|
||||
|
||||
## Strings (hard rule: no literal UI text in Rust)
|
||||
1. Add keys to `crates/i18n/locales/en.toml` under `[ui.<panel>]` (kebab-case keys, `{name}` placeholders).
|
||||
2. Add `crates/ui-egui/src/text/<panel>.rs` (or a sub-folder file) with the table macro:
|
||||
```rust
|
||||
//! Copy for the trash tab.
|
||||
|
||||
table! {
|
||||
title = "ui.trash-view.title";
|
||||
restore = "ui.trash-view.restore";
|
||||
}
|
||||
```
|
||||
3. Declare it in `crates/ui-egui/src/text.rs` and use it as `use crate::text::trash_view as strings;` then `strings::title()`.
|
||||
4. `cargo xtask i18n` must report 0 hardcoded literals. Only `en.toml` exists today.
|
||||
|
||||
## Structure
|
||||
- A view is a folder `views/<name>/mod.rs` returning a small action enum; the app shell handles the action. No engine calls from drawing code except through the data the shell hands in.
|
||||
- Keep drawing functions under 100 lines; split into `paint.rs`, `layout.rs`.
|
||||
- Give every interactive widget a stable id (`ui.id().with(("name", salt))`); duplicate ids trip a debug assert in tests.
|
||||
- Scroll areas: `ScrollArea::vertical().id_salt("notescraft-<name>-scroll").auto_shrink([false, false])`.
|
||||
- Animations must be skippable: `state.animations = false` in tests.
|
||||
|
||||
## Verify
|
||||
Driven test in `crates/ui-egui/tests/` (copy the `Driver` helper from `driven_features.rs`), plus an offscreen render compared with the design PNG. Designs: `design/mobile/*.png` and the PaperCraft project NotesCraft (page "Mobile and sync"); see `specs/mobile-and-sync/design.md`.
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
name: notescraft-engine-command
|
||||
description: How behaviour is added to the NotesCraft engine as commands, and the vault I/O rules (atomic writes, reserved folders). Use when touching crates/engine or crates/vault.
|
||||
---
|
||||
|
||||
# Engine commands and vault I/O
|
||||
|
||||
## Everything is a command
|
||||
User-visible behaviour is an engine command (id, label, typed params, `enabled`, `run`) so the UI, CLI, control channel, MCP and plugins share one surface. Ids are `[a-z0-9._-]`, 1..=128 chars, namespaced (`note.write`, `trash.restore`, `sync.now`). `register` validates ids; write commands are disabled on read-only sessions. Look at `crates/engine/src/registry.rs` and a small existing command group (for example `crates/engine/src/vault/trash.rs` and its registration in `crates/engine/src/builtin/`) and copy its shape exactly. The assistant gate classifies commands (`crates/engine/src/ai/assistant_gate.rs`): a new write or delete command must be added to the matching list, unknown commands default to write.
|
||||
|
||||
## Vault rules
|
||||
- Read and write notes through `notescraft_vault::Vault` (`read`, `write`, `write_private`, `delete_to_trash`, `rename`, `list`). They validate paths and refuse to leave the vault.
|
||||
- `.obsidian/` is never written. `.notescraft/` is written only through `Vault::write_private` (runtime data: history, workspaces, sync state). The `.notescraft/` folder is never committed.
|
||||
- Every other disk write goes through `craft_atomic::atomic_write(path, bytes)` (atomic rename, no half-written files).
|
||||
- Deleting a user note goes to the vault trash (`delete_to_trash`), never `remove_file`.
|
||||
- Paths are vault-relative with `/` separators; reject `..`, absolute, backslash and control characters before touching disk.
|
||||
|
||||
## Engine style
|
||||
- Library crates use `thiserror`, not `anyhow`. Commands return a `CommandError { code, message }` built from the crate error; messages are user-safe and never include secrets.
|
||||
- New code goes in a sub-folder with `mod.rs` when the directory already holds 12 loose files (`crates/engine/src/vault/` is full; `crates/engine/src/` has room for folders, not loose files).
|
||||
- Layering: engine is L4; it may depend on L0..L3 crates (`vault`, `markdown`, `index`, `sync`, ...) and never on egui. Check with `cargo xtask layers`.
|
||||
- Update `docs/ARCHITECTURE.md` (public API, numbers) and `ROADMAP.md` only if the task OWNS them.
|
||||
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: notescraft-rust-safety
|
||||
description: NotesCraft never-crash Rust idioms and the clippy pedantic traps that fail the build. Use whenever you write or edit Rust in this repo.
|
||||
---
|
||||
|
||||
# Never-crash Rust, NotesCraft style
|
||||
|
||||
The crates deny `unwrap_used`, `expect_used`, `panic`, `todo`, `unimplemented`, `unreachable` and forbid `unsafe`. Tests may use `unwrap`/`expect`. Clippy runs pedantic with `-D warnings`.
|
||||
|
||||
## Replace indexing and unwrap
|
||||
```rust
|
||||
// BAD: v[i], s[a..b], x.unwrap()
|
||||
// GOOD
|
||||
let first = items.get(index).ok_or(Error::Missing)?;
|
||||
let part = text.get(start..end).ok_or(Error::BadRange)?; // str slices are byte ranges and may split a char: get() returns None
|
||||
let (head, tail) = bytes.split_at_checked(n).ok_or(Error::Short)?;
|
||||
let total = a.checked_add(b).ok_or(Error::Overflow)?;
|
||||
let width = value.saturating_sub(pad);
|
||||
let n = u32::try_from(len).map_err(|_| Error::TooLarge)?; // never `len as u32`
|
||||
```
|
||||
Even a guarded `v[i]` is rejected in review when `i` comes from input (files, network, user).
|
||||
|
||||
## Hostile input
|
||||
- Cap every allocation sized by input (`Vec::with_capacity(n.min(CAP))`), bound recursion (depth constant), bound loops (each iteration consumes input or has a cap).
|
||||
- Invalid input returns `Err`. Never silently clamp, truncate or "repair" it.
|
||||
- Parse then validate, in one function at the boundary; trust values inside the crate afterwards.
|
||||
|
||||
## Errors
|
||||
One `thiserror` enum `Error` and `pub type Result<T> = std::result::Result<T, Error>;` per crate. No `anyhow` in libraries. Map foreign errors with `#[error("...: {0}")]` and `From` or `map_err`. Error text never contains secrets.
|
||||
|
||||
```rust
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum Error {
|
||||
#[error("invalid path: {0}")]
|
||||
InvalidPath(String),
|
||||
}
|
||||
pub type Result<T> = std::result::Result<T, Error>;
|
||||
```
|
||||
|
||||
## Style that passes review
|
||||
- Every public item has rustdoc; every module starts with `//!`.
|
||||
- Constants for numbers, with a doc comment: `/// Longest path accepted, in bytes.\npub const MAX_PATH_BYTES: usize = 1024;`.
|
||||
- Files at most about 400 lines; a directory holds at most 12 loose `.rs` files (`mod.rs`/`lib.rs` not counted): put new modules in a sub-folder with `mod.rs`.
|
||||
- Prefer let-chains over nested ifs: `if let Some(x) = a && x.ok() { ... }`.
|
||||
- Smallest design that works: no trait with one impl, no builder for two fields, no generic you do not need.
|
||||
|
||||
## Clippy pedantic traps seen here
|
||||
| Lint | Fix |
|
||||
|---|---|
|
||||
| `too_many_lines` (over 100) | split the function |
|
||||
| `many_single_char_names` | name loop vars `row`/`col`, `left`/`right` |
|
||||
| `assert!(x.is_empty())` in tests | `assert_eq!(x, Vec::<T>::new())` or `assert_eq!(x, [])` |
|
||||
| `explicit_iter_loop` | `for b in &digest` not `.iter()` |
|
||||
| `collapsible_if` / `collapsible_match` | `&&` let-chain |
|
||||
| `needless_pass_by_value` | take `&T` |
|
||||
| `format_push_string` | `write!(s, ...)` with `use std::fmt::Write` |
|
||||
| `cast_possible_truncation` and friends | `try_from`, `f64::from`, or `#[expect(clippy::..., reason = "...")]` with a real reason |
|
||||
| `struct_excessive_bools` | an enum or `#[expect]` with reason |
|
||||
| case-sensitive file extension compare | `Path::extension().is_some_and(\|e\| e.eq_ignore_ascii_case("md"))` |
|
||||
| `similar_names`, `items_after_statements`, `unused_self` | rename, move item up, make it an associated fn |
|
||||
| `dead_code` on cfg-gated code | `#[cfg_attr(..., allow(dead_code, reason = "..."))]` |
|
||||
No blanket `#![allow]`; a targeted `#[expect(lint, reason = "...")]` only when the lint is wrong here.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
name: notescraft-sync-protocol
|
||||
description: The NotesCraft vault sync protocol, its invariants, the pure notescraft-sync API, the server endpoints and how to test sync code. Use for any sync client, server or UI work.
|
||||
---
|
||||
|
||||
# Sync protocol (read `specs/mobile-and-sync/spec.md` for the full design)
|
||||
|
||||
## Invariants (never break)
|
||||
1. **Never lose data.** Unsure means keep both versions (conflict copy). Edit beats delete.
|
||||
2. The `.md` files are the truth; sync state lives in `.notescraft/sync/` (written with `Vault::write_private`).
|
||||
3. `.notescraft`, `.trash`, `.git`, `.obsidian` never sync. Every path from the network passes `notescraft_sync::check_syncable`; every hash passes `check_hash` and is verified against the downloaded bytes.
|
||||
4. The server never trusts the client: bodies capped (25 MiB), paths checked, tokens only as SHA-256 hashes, no secrets in logs or the audit table.
|
||||
|
||||
## Pure crate `notescraft-sync` (L1, no I/O; do not change it unless the task owns it)
|
||||
- `Ledger::{new, from_records, apply(Write) -> Outcome, changes_since(seq, limit), head, get}`; `Outcome::{Applied, Unchanged, Conflict}`; stale writer conflicts; stale edit over newer delete wins; stale delete over newer edit conflicts; identical hash spends no revision.
|
||||
- `plan(local_hashes, bases, remote_records) -> Vec<Action>`; `Action::{Download, DeleteLocal, Upload{path, base_seq}, DeleteRemote{path, base_seq}, Reconcile{remote, base_hash}, Adopt}`.
|
||||
- `merge3(base, ours, theirs) -> Merge::{Clean(String), Conflict}`; `conflict_path(path, device, stamp)`; `hash_hex`, `check_path`, `check_syncable`, `is_excluded`.
|
||||
- Types: `FileRecord{path, hash, size, mtime_ms, seq, deleted}`, `Changes{head, records, more}` with `cursor(since)`, `Base{hash, seq}`.
|
||||
|
||||
## HTTP API (server `apps/notescraft-sync-server/src/routes.rs`)
|
||||
Admin bearer: `POST /v1/vaults`, `POST /v1/vaults/{v}/pairing`. Device: `POST /v1/devices {code,name}`; per vault with device token: `GET /changes?since&limit&wait`, `GET|PUT|DELETE /files/{path}` (`X-Base-Seq`, `X-Mtime-Ms`, `ETag`, 201 applied / 200 unchanged / 409 `{current}`), `POST /batch/check {hashes}`. `GET /healthz`.
|
||||
|
||||
## Testing sync code
|
||||
- Pure logic: plain unit tests with explicit records.
|
||||
- Client driver: a `FakeRemote` implementing the `Remote` trait over `Ledger` + a blob map; two temp vault folders ("phone", "laptop"); assert both end identical, nothing lost, second run is a no-op.
|
||||
- Server: start it on `127.0.0.1:0` in a thread and speak real HTTP with `ureq` (see `apps/notescraft-sync-server/tests/api.rs`). Inject clocks as arguments (`now_ms`) instead of sleeping.
|
||||
- Add a randomised test when a rule has many cases (see `crates/sync/tests/merge_props.rs`, a tiny LCG, no extra dependency).
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
name: notescraft-verify-loop
|
||||
description: The exact build, test, lint and report loop for NotesCraft tasks, including offscreen UI checks. Use before saying a task is done.
|
||||
---
|
||||
|
||||
# Verify loop
|
||||
|
||||
Run from the repo root. `export PATH="$HOME/.cargo/bin:$PATH"`. Your launcher already set `CARGO_TARGET_DIR`; never use `target/debug`.
|
||||
|
||||
1. Test first. Write the failing test, run it, confirm it fails for the right reason.
|
||||
2. `timeout 300 cargo test -p notescraft-<crate>` (always with `timeout`; a looping parser once ate 40 GB).
|
||||
3. `cargo clippy -p notescraft-<crate> --all-targets -- -D warnings` after EVERY module, not at the end.
|
||||
4. `cargo fmt -p notescraft-<crate>`.
|
||||
5. Repo checks that apply: `cargo xtask layers` (dependency layers), `cargo xtask tree` (12 loose files per dir), `cargo xtask i18n` (no hardcoded UI strings). Package names are `notescraft-<dir>`; the sync server app is `notescraft-sync-server`.
|
||||
6. Confirm your new tests actually ran: the `cargo test` output must list them by name. A module not declared in `lib.rs`/`mod.rs` has tests that silently never run.
|
||||
|
||||
## Test conventions
|
||||
- Unit tests in `src/<file>.rs` under `#[cfg(test)] mod tests`; integration tests in `crates/<name>/tests/*.rs`.
|
||||
- Byte offsets in tests: compute with `str::find`, never by hand (`é` is 2 bytes).
|
||||
- Temp dirs: `std::env::temp_dir().join(format!("nc-{name}-{}", std::process::id()))`, remove before use.
|
||||
- Assert behaviour (values, files, events), not just "does not panic".
|
||||
- UI behaviour is tested by driving real events through whole frames (see `crates/ui-egui/tests/driven_features.rs`): build `NotesApp`, feed `RawInput` with pointer/key/text events, assert on painted text and state.
|
||||
|
||||
## Look at UI you changed
|
||||
`cargo run -p notescraft-ui-egui --example snapshot -- --vault <dir> --out /tmp/x.png --size 390x844 --open "a.md"` renders offscreen. Open the PNG with the Read tool and compare with the design PNG named in the task (`design/mobile/*.png`). List remaining differences in the report.
|
||||
|
||||
## Never
|
||||
- Weaken or delete a test to pass; add `#[allow]` to silence clippy; skip a failing command.
|
||||
- Claim success without pasting the final `test result:` line of your last run.
|
||||
@@ -10,3 +10,4 @@ snapshot.png
|
||||
/.worktrees/
|
||||
assistant-demo.png
|
||||
dist/
|
||||
/.mm-agents/
|
||||
|
||||
@@ -16,7 +16,7 @@ NotesCraft is a native, Obsidian-compatible notes app written in **Rust only** (
|
||||
- One crate per concern in `crates/<name>`, package `notescraft-<name>`, `[lints] workspace = true`, registered in the root `[workspace.dependencies]` and in `xtask/src/layers.rs`.
|
||||
- **Layering is enforced** by `cargo xtask layers`: a crate may depend only on strictly lower layers; nothing below L5 may use egui, eframe, winit or rfd. Layers: L0 `vault` `markdown`; L1 `index` `canvas` `diagrams` `whiteboard`; L2 `backlinks` `graph` `templates`; L3 `plugins` `themes`; L4 `engine`; L5 `editor` `mcp` `i18n`; L6 `ui-egui`.
|
||||
- Shared crates come from craft-libs (`craft-atomic`, `craft-settings`, `craft-ui`, `craft-quit`, `craft-print`), pinned by git rev in the root `Cargo.toml`. Do not change their API; ask first.
|
||||
- A new third-party dependency needs a written reason and owner approval. Already approved: `notify`, `pulldown-cmark`, `toml`, `wasmi`, `wat` (dev), `thiserror`, `serde`, `serde_json`.
|
||||
- A new third-party dependency needs a written reason and owner approval. Already approved: `notify`, `pulldown-cmark`, `toml`, `wasmi`, `wat` (dev), `thiserror`, `serde`, `serde_json`, and for sync (2026-10-19): `sha2`, `axum`, `tokio`, `ureq` (rustls), `rusqlite`, and for paste (2026-10-20): `arboard` (image-data), `image` (png feature only), and for the registry (2026-10-21): `ring` (ed25519 verify, already linked through rustls).
|
||||
|
||||
## Never crash (outranks features)
|
||||
|
||||
@@ -50,6 +50,11 @@ run `cargo xtask i18n` and do not add hardcoded user strings.
|
||||
|
||||
Check that every module is declared in `lib.rs` and that its tests actually run. After finishing, update `ROADMAP.md` and `docs/ARCHITECTURE.md` (status, public API, numbers). Commit only when the owner asks.
|
||||
|
||||
## Build hygiene (disk is shared and finite)
|
||||
|
||||
- Build only in an absolute `target/agent-<name>` dir, keep one scratch copy per agent with anchored excludes (`target*/` for directories), and delete both when you finish; the full rules are in `.claude/rules/build-hygiene.md`. `scripts/clean-agent-builds.sh --check` reports free space; without `--check` it deletes `target/agent-*`, stray `crates/*/target` and scratch copies (`--keep name` to spare a running agent).
|
||||
- Under 20 GiB free on the system volume or `/Volumes/Storage`: stop and clean before building. A full system volume stops every agent and the harness.
|
||||
|
||||
## Everything is a command
|
||||
|
||||
User-visible behaviour is an engine command (id, label, params, `enabled`, `run`) so the UI, CLI, control channel, MCP and plugins share one surface (SPEC 10). The `engine` crate does not exist yet; until it does, keep logic in pure library crates that a command can call.
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
|---|---|---|---|---|
|
||||
| `assets/fonts/Inter-Regular.ttf`, `Inter-Medium.ttf`, `Inter-SemiBold.ttf` | Inter 4.001 (UI font) | The Inter Project Authors (Rasmus Andersson) | <https://github.com/rsms/inter> | SIL OFL 1.1, `assets/fonts/OFL-Inter.txt` |
|
||||
| `assets/fonts/JetBrainsMono-Regular.ttf` | JetBrains Mono (monospace) | The JetBrains Mono Project Authors | <https://github.com/JetBrains/JetBrainsMono> | SIL OFL 1.1, `assets/fonts/OFL-JetBrainsMono.txt` |
|
||||
| `assets/fonts/JetBrainsMono-NoLigatures.ttf` | JetBrains Mono, modified copy used for code (the `calt` feature is emptied so `!=` and `->` stay as typed; renamed `JetBrains Mono NoLigatures`) | The JetBrains Mono Project Authors | <https://github.com/JetBrains/JetBrainsMono> | SIL OFL 1.1, `assets/fonts/OFL-JetBrainsMono.txt` |
|
||||
| `assets/fonts/STIXTwoMath-Regular.ttf` | STIX Two Math 2.13, subset to the math Unicode blocks (math type) | STIX Fonts Project | <https://github.com/stipub/stixfonts> (tag 2.13) | SIL OFL 1.1, `assets/fonts/OFL-STIXTwoMath.txt` |
|
||||
| `assets/icons/*.svg` | Lucide icons | Lucide Icons and Contributors | <https://github.com/lucide-icons/lucide> (`icons/<name>.svg`) | ISC, `assets/icons/LICENSE-lucide.txt` |
|
||||
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# NotesCraft
|
||||
|
||||
@AGENTS.md
|
||||
|
||||
Project skills live in `.claude/skills/` and path rules in `.claude/rules/`. Load the matching skill before you start: `notescraft-agent-workflow` (always), `notescraft-rust-safety` (any Rust), `notescraft-verify-loop` (before you claim done), `notescraft-egui-screen` (UI), `notescraft-engine-command` (engine or commands), `notescraft-sync-protocol` (sync).
|
||||
Generated
+911
-1
File diff suppressed because it is too large
Load Diff
+23
-1
@@ -35,8 +35,17 @@ notescraft-knowledge = { path = "crates/knowledge" }
|
||||
notescraft-lint = { path = "crates/lint" }
|
||||
notescraft-canvas = { path = "crates/canvas" }
|
||||
notescraft-graph = { path = "crates/graph" }
|
||||
notescraft-bases = { path = "crates/bases" }
|
||||
notescraft-templates = { path = "crates/templates" }
|
||||
notescraft-sync = { path = "crates/sync" }
|
||||
notescraft-collab = { path = "crates/collab" }
|
||||
notescraft-annotate = { path = "crates/annotate" }
|
||||
notescraft-calendar = { path = "crates/calendar" }
|
||||
notescraft-tasks = { path = "crates/tasks" }
|
||||
notescraft-textdiff = { path = "crates/textdiff" }
|
||||
notescraft-htmlmd = { path = "crates/htmlmd" }
|
||||
notescraft-srs = { path = "crates/srs" }
|
||||
notescraft-templater = { path = "crates/templater" }
|
||||
notescraft-plugins = { path = "crates/plugins" }
|
||||
notescraft-themes = { path = "crates/themes" }
|
||||
notescraft-engine = { path = "crates/engine" }
|
||||
@@ -57,6 +66,8 @@ muda = { version = "=0.21.2", default-features = false }
|
||||
egui = { version = "0.36", default-features = false }
|
||||
unicode-bidi = { version = "0.3.18", default-features = false, features = ["std", "hardcoded-data"] }
|
||||
egui_extras = { version = "0.36", default-features = false, features = ["svg"] }
|
||||
# Clipboard images and file lists for paste (owner-approved 2026-10-20; already in the lockfile through egui-winit). image-data only: no default features.
|
||||
arboard = { version = "3.6", default-features = false, features = ["image-data"] }
|
||||
eframe = { version = "0.36", default-features = false }
|
||||
egui_kittest = "0.36"
|
||||
log = "0.4"
|
||||
@@ -73,6 +84,14 @@ grep-searcher = "0.1"
|
||||
syntect = { version = "5", default-features = false, features = ["parsing", "default-syntaxes", "yaml-load", "regex-fancy"] }
|
||||
# SHA-256 of downloaded grammars (pure Rust, RustCrypto; owner-approved 2026-10-18).
|
||||
sha2 = { version = "0.11", default-features = false }
|
||||
# Sync server (apps/notescraft-sync-server only; owner-approved 2026-10-19).
|
||||
axum = { version = "0.8", default-features = false, features = ["http1", "tokio", "json", "query"] }
|
||||
tokio = { version = "1", default-features = false, features = ["rt-multi-thread", "macros", "net", "signal", "time", "sync"] }
|
||||
rusqlite = { version = "0.37", features = ["bundled"] }
|
||||
# Blocking HTTP client with rustls (sync server tests now, the engine sync client next).
|
||||
ureq = { version = "3", default-features = false, features = ["rustls", "json"] }
|
||||
# ed25519 verification of registry entries (owner-approved 2026-10-21, already linked through rustls).
|
||||
ring = { version = "0.17", default-features = false }
|
||||
# Pure-Rust WebAssembly interpreter, same configuration as photocraft-plugins.
|
||||
wasmi = { version = "2.0.0", default-features = false, features = ["std", "validate", "stable", "portable-dispatch"] }
|
||||
wat = "1.228"
|
||||
@@ -99,4 +118,7 @@ debug = "line-tables-only"
|
||||
|
||||
[profile.release]
|
||||
lto = "thin"
|
||||
codegen-units = 1
|
||||
codegen-units = 1
|
||||
# Size: owner-approved 2026-10-21 (specs/registry/measurements.md): -8 MB on macOS; crash backtraces lose symbol names.
|
||||
opt-level = "s"
|
||||
strip = "symbols"
|
||||
|
||||
+4
-2
@@ -26,6 +26,7 @@ One-page summary of where NotesCraft stands. The contract is [SPEC.md](SPEC.md);
|
||||
| 14 | Editor blocks and lint | `lint`, `blocks`, engine `lint.*` and `block.*`, slash menu, grip, Lint tab, Settings > Editor and lint | done: 12 lint rules (spelling slot only), one-click and safe fixes, learned style, dismissals, `fix_on_save`, slash menu with groups and preview in Source and Visual, block grip, drag, menu, Vim `]]` `[[` `gb`, Visual tables, note embeds, `%%` comments; not built: spell-check, mobile bottom toolbar, Fix popover in the editor, keyboard use of the block menu (specs/editor-blocks-and-lint) |
|
||||
| 15 | Fast open and large vaults | `vault` listing and `count_files`, engine `open_lazy`, `index_step`, `ContentSearch`, `index.*`, Explorer virtualization and navigation, large-folder warning | done: window in under a second on 10 000 notes, background indexing with a status card, search from disk while indexing, pause and resume, rename and move refused until the index is complete, virtualized Explorer (50 000 rows), note counts, filter, collapse all, reveal and follow, pinned folders, sticky folder path, warning above 5 000 files; no index cache (not needed); not built: honouring `.gitignore` in non-vault folders, jump-to-letter strip, key bindings for the Explorer commands (specs/fast-open-and-explorer) |
|
||||
| 16 | Editor previews | `editor::preview`, engine `editor.previews.*`, colour swatches, external-link arrows, tree card, Mermaid Source toggle | done: `detect_colors` (hex, `rgb`/`rgba`, `hsl`/`hsla` and the basic colour names) and `detect_tree`, both pure and bounded, swatches in body text, inline code and ten fenced languages, the right-click colour menu (Copy color, Copy as rgb, the two switches), external-link arrows, the TREE card, the Mermaid Diagram/Source toggle and the error/unsupported/too-big chips; not built: swatches and arrows in Source mode and in rows with inline math, swatches for a hex Obsidian reads as a tag (specs/editor-previews) |
|
||||
| 17 | Mobile, sync, collaboration, launcher | `sync` and `collab` (L1), `apps/notescraft-sync-server`, engine `sync/` and `prefs/catalog/`, `ui-egui` `launcher` and `shell/compact`, `apps/notescraft` `launcher_window` | done 2026-10-20: sync ledger, planner and merge (28 tests), server with accounts, many vaults, roles, invites, hardening, quota, GC (40 tests), engine client, worker and `sync.*` commands, vault catalog and launcher (host wired), OT core (22 tests), compact layout, 15 PaperCraft screens, research (3 reports) and specs (`mobile-and-sync`, `collaboration`, `vault-launcher`, `agents`); also done: sync UI (Settings > Sync, status pill, conflicts) and the live-editing endpoints (SSE rooms, flush); not built: live session client in the engine and the editor UI, Share dialog, Android and iOS shells, E2EE, deploy, attachments chunking, server version history (see the three tasks.md files) |
|
||||
| 11 | PDF export, packaging, i18n coverage, hardening | `pdf`, `packaging/`, release | PDF export done (crate, `export.pdf`, CLI, dialog; LaTeX prints as source, `craft-print` not wired); packaging done for macOS (built and smoke-tested), Windows (MSI scripts, compile-checked only) and Linux (deb/rpm/AppImage scripts, untested here); i18n migration done (252 keys in `en.toml`, `cargo xtask i18n --strict` is part of `ci`) |
|
||||
|
||||
Also built beyond the phase list: Vim mode (`vim` crate and app-wide navigation), resizable and collapsable cards with rails, Windows/Linux custom title bar.
|
||||
@@ -59,8 +60,9 @@ Done (crossed off):
|
||||
Pending, by area:
|
||||
- **Code blocks, grammars and formatting (done, 15 of 15 tasks, specs/code-blocks-and-formatting):** `toml` and `ts` fences now highlight through two original grammars embedded in `notescraft-syntax` (`crates/syntax/grammars/*.sublime-syntax`), and the batch review has a third check, "Add missing language labels": `blocks::detect_language` names the language of an unlabeled fence conservatively (shebang, PHP, HTML, XML, JSON, YAML, diff, shell prompts, Rust, Python, SQL, TOML, on the first 4 KiB and 40 non-blank lines) and each `ChangeKind::Label` is a zero-length insert right after the fence run that always needs review (`batch.start` takes `labels`, on by default). Not built — the language picker of the design (a mistyped label is fixed by the suggestion chip, and nothing offers to add a label while typing in source mode), the table-alignment checks of the design, and the "more info" card of the grammar dialog. A note with 50 code blocks draws its first frame in 15 ms median (release), 87 ms in a cold process.
|
||||
- **Editor previews (done, 11 of 11 tasks, specs/editor-previews):** not built or limited — no swatches or arrows in Source mode or in rows carrying inline math, and a hex that Obsidian reads as a tag (`#cba6f7` in prose) stays a tag, so only inline code and non-tag text get swatches; `Unsupported` is reachable only if the diagram parser returns an unrenderable kind (today it answers `NoHeader` for one it does not know, so those show the Error chip). Scanning 100 rows of 10 colours (1,000 hits) takes 0.05 ms median, 0.15 ms worst, in release.
|
||||
- **Obsidian basics (specs/obsidian-basics, audit in audit.md): all 18 tasks built.** Find and replace (with regex), back/forward, auto-pair, footnotes, explorer sort and drag-and-drop move, heading and list folding, search operators and `/regex/` search, vault-wide replace, quick switcher headings and blocks, random and unique notes, Open in default app, Copy Obsidian URL, pinned and reorderable tabs, link hover preview, outgoing links, split view (two panes with their own tabs; a click in a pane makes it the editing pane), spelling lint rule, settings (default view, font size, readable width, line numbers, new-note folder, link format and path, deleted files), PDF embeds drawn as an image of the page (`#page=N`; macOS PDFKit, elsewhere `pdftoppm` when installed, else a file card), audio embeds with Play/Stop through the system player, video embeds with a still, and in-note playback (ffmpeg decodes frames, ffplay the sound) when ffmpeg is installed, else Open in the system player, and a graph view with force sliders and colour groups that scales to 6.5k notes. Strict line breaks and show-hidden-files are settings too. Limits kept on purpose: video playback needs `ffmpeg` on the machine (no decoder is linked), no bundled spelling dictionary (the vault is the vocabulary). `crates/ui-egui/tests/driven_features.rs` drives tab drag, pinning, the link hover card, list folding, split view, hidden files, the regex find bar and Explorer drag-and-drop through real input events.
|
||||
- **Obsidian basics round 2 (specs/obsidian-basics/tasks.md 19-24): query embeds (19) and stacked tabs (21) built; editor follow-ups (23) built; pop-out windows (22) built but not seen in a native window; RTL (20) built for the reading view with `unicode-bidi` (owner approved; Arabic unjoined, Source mode logical order); round 3 (tasks 25-30, from the audit rows still missing or partial) started: note composer engine commands `note.extract` and `note.merge` built, UI open; unbuilt: hand check of the packaged app and a Windows playback run.
|
||||
- **Obsidian basics (specs/obsidian-basics, audit in audit.md): all 18 tasks built.** Find and replace (with regex), back/forward, auto-pair, footnotes, explorer sort and drag-and-drop move, heading and list folding, search operators and `/regex/` search, vault-wide replace, quick switcher headings and blocks, random and unique notes, Open in default app, Copy Obsidian URL, pinned and reorderable tabs, link hover preview, outgoing links, split view (two panes with their own tabs; a click in a pane makes it the editing pane), spelling lint rule, settings (default view, font size, readable width, line numbers, new-note folder, link format and path, deleted files), PDF embeds drawn as an image of the page (`#page=N`; macOS PDFKit, elsewhere `pdftoppm` when installed, else a file card), an audio tab and audio embeds that play in the app (ffplay sound, a waveform that seeks, a live spectrum from an own FFT), video embeds with a still, and in-note playback (ffmpeg decodes frames, ffplay the sound) when ffmpeg is installed, else Open in the system player, and a graph view with force sliders and colour groups that scales to 6.5k notes. Strict line breaks and show-hidden-files are settings too. Limits kept on purpose: video playback needs `ffmpeg` on the machine (no decoder is linked), no bundled spelling dictionary (the vault is the vocabulary). `crates/ui-egui/tests/driven_features.rs` drives tab drag, pinning, the link hover card, list folding, split view, hidden files, the regex find bar and Explorer drag-and-drop through real input events.
|
||||
- **Obsidian basics round 2 (specs/obsidian-basics/tasks.md 19-24): query embeds (19) and stacked tabs (21) built; editor follow-ups (23) built; pop-out windows (22) built but not seen in a native window; RTL (20) built for the reading view with `unicode-bidi` (owner approved; Arabic unjoined, Source mode logical order); round 3 (tasks 25-30, from the audit rows still missing or partial) started: note composer (`note.extract`, `note.merge`, the toolbar button and the Explorer "Merge into…") and trash `trash.list/restore/empty` and the Trash tab built, folder notes (27), reading mode and the indent settings (29), table helpers, user themes, the graph timelapse and Source-mode heading folding (30), workspaces (32), heading/block links (33) and note versions (31, local snapshots plus the git history) and Cmd/Ctrl+Enter on tasks in Source and Visual (36, 37) built; unbuilt: hand check of the packaged app and a Windows playback run.
|
||||
- **Linux and Windows feedback (2026-10-19):** built: the Explorer calendar folds (chevron, remembered, folded on a first run in a window under 720 pt), the Explorer toolbar is laid out by hand (filter box plus two pill buttons, fixed gaps, flush right), the empty Bookmarks section is gone, the menu bar words are flat text with a soft hover/open background, the Related tab scrolls, a new Outline pane (count, filter, folding, active heading, scroll), a performance overlay (`view.performance`: fps, frame ms, memory, CPU; `/proc`, `ps` or PowerShell, no new dependency), and a first-run UI scale taken from the desktop on Linux when the window was not told its scale (`QT_SCALE_FACTOR`, `GDK_SCALE`, `Xft.dpi`, GNOME text scaling). Not confirmed on those systems: the title bar blink (the bar is now drawn whole on every frame and sits above the cards) and Explorer icons drawn out of bounds on a fast scroll (not reproduced offscreen; rows are now clipped to the viewport). Windows paths (PowerShell sampler, menu bar) were not run.
|
||||
- **Editor (built, see phase 14):** spell-check needs an owner decision (new dependency or in-house checker); open: mobile bottom toolbar, inline fix popover, block menu by keyboard, embed with other text on one line, checkboxes inside embeds, embed refresh after editing another tab.
|
||||
- **Editor settings:** the page has only Vim mode; more settings need editor features first (line numbers, spellcheck, font size, ...), then sub-menus for them.
|
||||
- **Community plugins:** a browse page and sub-menu.
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
[package]
|
||||
name = "notescraft-sync-server"
|
||||
description = "NotesCraft vault sync service: pairing, change log and file blobs over HTTP"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
rust-version.workspace = true
|
||||
publish = false
|
||||
|
||||
[lib]
|
||||
path = "src/lib.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "notescraft-sync-server"
|
||||
path = "src/main.rs"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
[dependencies]
|
||||
axum = { workspace = true }
|
||||
craft-atomic = { workspace = true }
|
||||
futures-core = "0.3"
|
||||
notescraft-collab = { workspace = true }
|
||||
notescraft-markdown = { workspace = true }
|
||||
notescraft-sync = { workspace = true }
|
||||
pulldown-cmark = { workspace = true, features = ["html"] }
|
||||
rusqlite = { workspace = true }
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
sha2 = { workspace = true }
|
||||
tokio = { workspace = true }
|
||||
|
||||
[dev-dependencies]
|
||||
notescraft-engine = { workspace = true }
|
||||
notescraft-vault = { workspace = true }
|
||||
axum = { workspace = true }
|
||||
tokio = { workspace = true }
|
||||
ureq = { workspace = true }
|
||||
@@ -0,0 +1,56 @@
|
||||
//! The activity feed: what happened in a vault while a member was away.
|
||||
//!
|
||||
//! The feed is server side (D7 of `specs/comments-and-mentions/spec.md`): a table
|
||||
//! per vault, so it survives a restart of every client and reaches each member on
|
||||
//! each of their devices. The client posts the events it causes; the server never
|
||||
//! reads the sidecar, so it never sees a comment body and never logs one.
|
||||
//!
|
||||
//! One event is stored for the members it names. A `mention` reaches the members
|
||||
//! it mentions; every other kind reaches the members of the vault. The actor never
|
||||
//! sees their own event, and a member that was removed from the vault reads
|
||||
//! nothing at all.
|
||||
//!
|
||||
//! The read state is one row per (member, event), so a member that was away can
|
||||
//! ask for the events it missed and for how many of them it has not read yet.
|
||||
|
||||
pub(crate) mod routes;
|
||||
pub(crate) mod store;
|
||||
|
||||
/// A member was mentioned in a thread.
|
||||
pub const KIND_MENTION: &str = "mention";
|
||||
/// Someone commented on a note of the vault.
|
||||
pub const KIND_COMMENT: &str = "comment";
|
||||
/// Someone proposed a change to a note of the vault.
|
||||
pub const KIND_SUGGESTION: &str = "suggestion";
|
||||
/// Someone else changed a note of the vault.
|
||||
pub const KIND_NOTE_CHANGED: &str = "note_changed";
|
||||
/// An invite was accepted.
|
||||
pub const KIND_INVITE_ACCEPTED: &str = "invite_accepted";
|
||||
/// A member joined the vault.
|
||||
pub const KIND_MEMBER_JOINED: &str = "member_joined";
|
||||
/// The role of a member changed.
|
||||
pub const KIND_ROLE_CHANGED: &str = "role_changed";
|
||||
/// A thread was resolved.
|
||||
pub const KIND_THREAD_RESOLVED: &str = "thread_resolved";
|
||||
|
||||
/// Every kind the feed carries.
|
||||
pub const KINDS: [&str; 8] =
|
||||
[KIND_MENTION, KIND_COMMENT, KIND_SUGGESTION, KIND_NOTE_CHANGED, KIND_INVITE_ACCEPTED, KIND_MEMBER_JOINED, KIND_ROLE_CHANGED, KIND_THREAD_RESOLVED];
|
||||
|
||||
/// How many events a feed page holds when the client asks for none.
|
||||
pub const DEFAULT_PAGE: usize = 50;
|
||||
|
||||
/// Most events one feed page holds (D10).
|
||||
pub const MAX_PAGE: usize = 200;
|
||||
|
||||
/// Most members one event may mention (D10).
|
||||
pub const MAX_MENTIONS: usize = 20;
|
||||
|
||||
/// The audience of an event that reaches every member of the vault.
|
||||
pub(crate) const AUDIENCE_VAULT: &str = "vault";
|
||||
|
||||
/// The audience of an event that reaches the members it mentions.
|
||||
pub(crate) const AUDIENCE_MENTION: &str = "mention";
|
||||
|
||||
/// An event was posted to the feed of a vault.
|
||||
pub(crate) const ACTIVITY_POSTED: &str = "activity_posted";
|
||||
@@ -0,0 +1,254 @@
|
||||
//! The activity endpoints: post an event, list the feed of the caller, mark it read.
|
||||
//!
|
||||
//! The actor of an event is always the user of the token: a client cannot speak
|
||||
//! for somebody else, so the body may carry the actor only to be checked against
|
||||
//! the token. A viewer may read the feed and mark it read, but not post an event
|
||||
//! (D10): posting writes rows of the vault, and a viewer does not write.
|
||||
//!
|
||||
//! A member that was removed from the vault reads nothing and posts nothing: the
|
||||
//! feed belongs to the members of the vault (D7).
|
||||
|
||||
use axum::Json;
|
||||
use axum::body::Bytes;
|
||||
use axum::extract::{Path, Query, State};
|
||||
use axum::http::HeaderMap;
|
||||
use notescraft_sync::{check_syncable, hash_hex};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::activity::store::{Event, EventRow};
|
||||
use crate::activity::{ACTIVITY_POSTED, AUDIENCE_MENTION, AUDIENCE_VAULT, DEFAULT_PAGE, KIND_MENTION, KINDS, MAX_MENTIONS, MAX_PAGE};
|
||||
use crate::api::audit;
|
||||
use crate::error::ApiError;
|
||||
use crate::roles::{Member, Need, Role};
|
||||
use crate::state::{AppState, bearer, now_ms};
|
||||
|
||||
/// The body of `POST /v1/vaults/{vault}/activity`.
|
||||
#[derive(Deserialize)]
|
||||
pub struct NewEvent {
|
||||
/// The kind of the event; one of [`KINDS`].
|
||||
pub kind: String,
|
||||
/// The user the event is from; it must be the user of the token.
|
||||
#[serde(default)]
|
||||
pub actor: Option<String>,
|
||||
/// The note the event is about.
|
||||
#[serde(default)]
|
||||
pub note: Option<String>,
|
||||
/// The comment thread the event is about.
|
||||
#[serde(default)]
|
||||
pub thread_id: Option<String>,
|
||||
/// The line of the anchor of the event.
|
||||
#[serde(default)]
|
||||
pub line: Option<u64>,
|
||||
/// The members the event mentions.
|
||||
#[serde(default)]
|
||||
pub mentions: Vec<String>,
|
||||
}
|
||||
|
||||
/// The answer of `POST /v1/vaults/{vault}/activity`.
|
||||
#[derive(Serialize)]
|
||||
pub struct PostedEvent {
|
||||
/// The id of the new event.
|
||||
pub id: u64,
|
||||
/// The kind of the event.
|
||||
pub kind: String,
|
||||
/// The newest event id of the vault; the cursor of the next poll.
|
||||
pub head: u64,
|
||||
/// How many members the event was stored for.
|
||||
pub recipients: usize,
|
||||
}
|
||||
|
||||
/// The query of `GET /v1/vaults/{vault}/activity`.
|
||||
#[derive(Deserialize)]
|
||||
pub struct FeedQuery {
|
||||
/// The cursor: only the events after it are answered.
|
||||
#[serde(default)]
|
||||
pub since: u64,
|
||||
/// How many events to answer; capped at [`MAX_PAGE`].
|
||||
pub limit: Option<usize>,
|
||||
}
|
||||
|
||||
/// The answer of `GET /v1/vaults/{vault}/activity`.
|
||||
#[derive(Serialize)]
|
||||
pub struct Feed {
|
||||
/// The events of the member, oldest first.
|
||||
pub events: Vec<EventRow>,
|
||||
/// The newest event id of the vault; the cursor of the next poll.
|
||||
pub head: u64,
|
||||
/// True when more events are waiting behind the page.
|
||||
pub more: bool,
|
||||
/// How many of the events the member has not read.
|
||||
pub unread: u64,
|
||||
}
|
||||
|
||||
/// The body of `POST /v1/vaults/{vault}/activity/read`.
|
||||
#[derive(Deserialize)]
|
||||
pub struct ReadRequest {
|
||||
/// The ids to mark; every event the member sees when the list is absent.
|
||||
#[serde(default)]
|
||||
pub ids: Option<Vec<u64>>,
|
||||
}
|
||||
|
||||
/// The answer of `POST /v1/vaults/{vault}/activity/read`.
|
||||
#[derive(Serialize)]
|
||||
pub struct ReadReply {
|
||||
/// How many of the events the member has not read.
|
||||
pub unread: u64,
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/activity`: stores one event of the feed.
|
||||
///
|
||||
/// The event is stored for the members it names: a `mention` for the members it
|
||||
/// mentions, any other kind for the members of the vault. The actor is the user
|
||||
/// of the token, the mentions must be members and the note must exist.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a viewer or a member that was removed, 404 for an unknown vault
|
||||
/// or an unknown note, 400 for an unknown kind, a stranger in the mentions or a
|
||||
/// note outside the vault, 500 on database errors.
|
||||
pub(crate) async fn post(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(request): Json<NewEvent>,
|
||||
) -> Result<Json<PostedEvent>, ApiError> {
|
||||
let member = authorize_member(&state, &headers, &vault, Need::Write)?;
|
||||
let audience = check_event(&state, &vault, &member.user, &request)?;
|
||||
let event = Event {
|
||||
kind: &request.kind,
|
||||
actor: &member.user,
|
||||
audience,
|
||||
note: request.note.as_deref(),
|
||||
thread_id: request.thread_id.as_deref(),
|
||||
line: request.line,
|
||||
mentions: &request.mentions,
|
||||
};
|
||||
let core = state.core();
|
||||
let id = core.store.add_activity(&vault, &event, now_ms())?;
|
||||
let recipients = if audience == AUDIENCE_MENTION { request.mentions.len() } else { core.store.vault_members(&vault)?.len().saturating_sub(1) };
|
||||
audit::record(&core.store, ACTIVITY_POSTED, Some(&vault), Some(&member.device), Some(&request.kind))?;
|
||||
let head = core.store.activity_head(&vault)?;
|
||||
Ok(Json(PostedEvent { id, kind: request.kind, head, recipients }))
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/activity?since=&limit=`: the events of the member, the
|
||||
/// newest id of the vault and the number of events the member has not read.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a member that was removed, 404 for an unknown vault,
|
||||
/// 500 on database errors.
|
||||
pub(crate) async fn list(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Query(query): Query<FeedQuery>,
|
||||
) -> Result<Json<Feed>, ApiError> {
|
||||
let member = authorize_member(&state, &headers, &vault, Need::Read)?;
|
||||
let limit = query.limit.unwrap_or(DEFAULT_PAGE).clamp(1, MAX_PAGE);
|
||||
let core = state.core();
|
||||
let mut events = core.store.activity_events(&vault, &member.user, query.since, limit.saturating_add(1))?;
|
||||
let more = events.len() > limit;
|
||||
events.truncate(limit);
|
||||
let head = core.store.activity_head(&vault)?;
|
||||
let unread = core.store.activity_unread(&vault, &member.user)?;
|
||||
Ok(Json(Feed { events, head, more, unread }))
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/activity/read {ids}`: marks the events of the member as
|
||||
/// read; without ids every event the member sees is marked. Answers the number of
|
||||
/// events the member has not read.
|
||||
///
|
||||
/// The body is optional: an empty body marks every event the member sees.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a member that was removed, 404 for an unknown vault, 400 for a
|
||||
/// body that is not a list of ids or for too many ids, 500 on database errors.
|
||||
pub(crate) async fn read(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>, body: Bytes) -> Result<Json<ReadReply>, ApiError> {
|
||||
let member = authorize_member(&state, &headers, &vault, Need::Read)?;
|
||||
let ids = read_ids(&body)?;
|
||||
if ids.len() > MAX_PAGE {
|
||||
return Err(ApiError::bad(format!("at most {MAX_PAGE} ids at a time")));
|
||||
}
|
||||
let core = state.core();
|
||||
core.store.activity_mark_read(&vault, &member.user, &ids, now_ms())?;
|
||||
let unread = core.store.activity_unread(&vault, &member.user)?;
|
||||
Ok(Json(ReadReply { unread }))
|
||||
}
|
||||
|
||||
/// The ids of the body; an empty body marks every event the member sees.
|
||||
fn read_ids(body: &Bytes) -> Result<Vec<u64>, ApiError> {
|
||||
if body.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let request: ReadRequest = serde_json::from_slice(body).map_err(|_| ApiError::bad("the body must hold the ids of the events to mark"))?;
|
||||
Ok(request.ids.unwrap_or_default())
|
||||
}
|
||||
|
||||
/// The member behind the token, in `vault`.
|
||||
///
|
||||
/// A member that acts above its role is refused, and a user that is not a member
|
||||
/// of the vault is refused as well: the feed belongs to the members of a vault,
|
||||
/// and a removed member must read nothing of it.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without a token, 404 for an unknown vault, 403 for a user that is not a
|
||||
/// member or that acts above its role, 500 on database errors.
|
||||
fn authorize_member(state: &AppState, headers: &HeaderMap, vault: &str, need: Need) -> Result<Member, ApiError> {
|
||||
let token = bearer(headers).ok_or_else(ApiError::unauthorized)?;
|
||||
let token_hash = hash_hex(token.as_bytes());
|
||||
let core = state.core();
|
||||
let Some((device, user)) = core.store.device_for(&token_hash)? else {
|
||||
return Err(ApiError::unauthorized());
|
||||
};
|
||||
if !core.store.vault_live(vault)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
let Some(raw) = core.store.member_role(vault, &user)? else {
|
||||
return Err(ApiError::forbidden());
|
||||
};
|
||||
let Some(role) = Role::parse(&raw) else {
|
||||
return Err(ApiError::internal(&"the role of the member"));
|
||||
};
|
||||
if !role.satisfies(need) {
|
||||
return Err(ApiError::forbidden());
|
||||
}
|
||||
core.store.touch_device(&token_hash, now_ms())?;
|
||||
Ok(Member { device, user, role })
|
||||
}
|
||||
|
||||
/// Checks the event and answers its audience.
|
||||
///
|
||||
/// The kind must be one the feed carries, the actor must be the user of the
|
||||
/// token, the mentions must be members of the vault and the note must exist in
|
||||
/// it. A mention that names nobody is refused.
|
||||
fn check_event(state: &AppState, vault: &str, actor: &str, request: &NewEvent) -> Result<&'static str, ApiError> {
|
||||
if !KINDS.contains(&request.kind.as_str()) {
|
||||
return Err(ApiError::bad("the kind of the event is not one the feed carries"));
|
||||
}
|
||||
if let Some(claimed) = &request.actor
|
||||
&& claimed != actor
|
||||
{
|
||||
return Err(ApiError::forbidden());
|
||||
}
|
||||
if request.mentions.len() > MAX_MENTIONS {
|
||||
return Err(ApiError::bad(format!("the event names {} mentions, the cap is {MAX_MENTIONS}", request.mentions.len())));
|
||||
}
|
||||
let core = state.core();
|
||||
for mention in &request.mentions {
|
||||
if core.store.member_role(vault, mention)?.is_none() {
|
||||
return Err(ApiError::bad(format!("`{mention}` is not a member of the vault")));
|
||||
}
|
||||
}
|
||||
if let Some(note) = &request.note {
|
||||
check_syncable(note)?;
|
||||
if !core.store.file_live(vault, note)? {
|
||||
return Err(ApiError::missing("no such note"));
|
||||
}
|
||||
}
|
||||
if request.kind == KIND_MENTION {
|
||||
if request.mentions.is_empty() {
|
||||
return Err(ApiError::bad("a mention must name a member"));
|
||||
}
|
||||
return Ok(AUDIENCE_MENTION);
|
||||
}
|
||||
Ok(AUDIENCE_VAULT)
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
//! The rows of the activity feed: the events of a vault, the members they name
|
||||
//! and the read state of each member.
|
||||
//!
|
||||
//! Three tables hold the feed. `activity` holds the events, one row per event.
|
||||
//! `activity_mentions` names the members a mention event reaches. `activity_read`
|
||||
//! holds one row per (member, event) that was read. The id of an event is the
|
||||
//! cursor of the feed: a client polls with the newest id it has seen and gets
|
||||
//! the events after it.
|
||||
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use crate::store::{Store, int, uint};
|
||||
|
||||
/// One event of the feed, as the answer of the listing.
|
||||
#[derive(Clone, Debug, serde::Serialize)]
|
||||
pub struct EventRow {
|
||||
/// The id of the event; the cursor of the feed.
|
||||
pub id: u64,
|
||||
/// When the event was stored.
|
||||
pub ts_ms: u64,
|
||||
/// The kind of the event.
|
||||
pub kind: String,
|
||||
/// The user id of the actor.
|
||||
pub actor: String,
|
||||
/// The display name of the actor.
|
||||
pub actor_name: String,
|
||||
/// The note the event is about, when it is about one.
|
||||
pub note: Option<String>,
|
||||
/// The comment thread the event is about, when it is about one.
|
||||
pub thread_id: Option<String>,
|
||||
/// The line of the anchor, when the event carries one.
|
||||
pub line: Option<u64>,
|
||||
/// True when the member has read the event.
|
||||
pub read: bool,
|
||||
}
|
||||
|
||||
/// The event that is stored: what a client posts, with the actor and the audience
|
||||
/// the server decides.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub struct Event<'a> {
|
||||
/// The kind of the event.
|
||||
pub kind: &'a str,
|
||||
/// The user id of the actor.
|
||||
pub actor: &'a str,
|
||||
/// The audience of the event: the members of the vault or the ones it mentions.
|
||||
pub audience: &'a str,
|
||||
/// The note the event is about, when it is about one.
|
||||
pub note: Option<&'a str>,
|
||||
/// The comment thread the event is about, when it is about one.
|
||||
pub thread_id: Option<&'a str>,
|
||||
/// The line of the anchor, when the event carries one.
|
||||
pub line: Option<u64>,
|
||||
/// The members the event mentions.
|
||||
pub mentions: &'a [String],
|
||||
}
|
||||
|
||||
impl Store {
|
||||
/// Stores one event of the feed and the members it names; answers the id of the
|
||||
/// event.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn add_activity(&self, vault: &str, event: &Event<'_>, now_ms: u64) -> rusqlite::Result<u64> {
|
||||
let transaction = self.conn().unchecked_transaction()?;
|
||||
transaction.execute(
|
||||
"INSERT INTO activity (vault, ts_ms, kind, actor, audience, note, thread_id, line) \
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
|
||||
params![vault, int(now_ms)?, event.kind, event.actor, event.audience, event.note, event.thread_id, event.line.map(int).transpose()?],
|
||||
)?;
|
||||
let id = transaction.last_insert_rowid();
|
||||
for user in event.mentions {
|
||||
transaction.execute("INSERT OR IGNORE INTO activity_mentions (event_id, user) VALUES (?1, ?2)", params![id, user])?;
|
||||
}
|
||||
transaction.commit()?;
|
||||
u64::try_from(id).map_err(|_| rusqlite::Error::IntegralValueOutOfRange(0, i64::MAX))
|
||||
}
|
||||
|
||||
/// The events of a vault that `user` sees, oldest first, the ones after `since`.
|
||||
/// The actor of an event never sees it, and a `mention` reaches only the members
|
||||
/// it names.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn activity_events(&self, vault: &str, user: &str, since: u64, limit: usize) -> rusqlite::Result<Vec<EventRow>> {
|
||||
let mut statement = self.conn().prepare(
|
||||
"SELECT e.id, e.ts_ms, e.kind, e.actor, COALESCE(u.name, e.actor), e.note, e.thread_id, e.line, \
|
||||
EXISTS (SELECT 1 FROM activity_read r WHERE r.event_id = e.id AND r.user = ?2) \
|
||||
FROM activity e LEFT JOIN users u ON u.id = e.actor \
|
||||
WHERE e.vault = ?1 AND e.id > ?3 AND e.actor <> ?2 \
|
||||
AND (e.audience = 'vault' OR EXISTS (SELECT 1 FROM activity_mentions m WHERE m.event_id = e.id AND m.user = ?2)) \
|
||||
ORDER BY e.id LIMIT ?4",
|
||||
)?;
|
||||
let rows = statement.query_map(params![vault, user, int(since)?, limit], |row| {
|
||||
Ok(EventRow {
|
||||
id: uint(row.get(0)?)?,
|
||||
ts_ms: uint(row.get(1)?)?,
|
||||
kind: row.get(2)?,
|
||||
actor: row.get(3)?,
|
||||
actor_name: row.get(4)?,
|
||||
note: row.get(5)?,
|
||||
thread_id: row.get(6)?,
|
||||
line: row.get::<_, Option<i64>>(7)?.map(uint).transpose()?,
|
||||
read: row.get::<_, i64>(8)? != 0,
|
||||
})
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// How many of the events `user` sees are not read yet.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn activity_unread(&self, vault: &str, user: &str) -> rusqlite::Result<u64> {
|
||||
self.conn().query_row(
|
||||
"SELECT COUNT(*) FROM activity e \
|
||||
WHERE e.vault = ?1 AND e.actor <> ?2 \
|
||||
AND (e.audience = 'vault' OR EXISTS (SELECT 1 FROM activity_mentions m WHERE m.event_id = e.id AND m.user = ?2)) \
|
||||
AND NOT EXISTS (SELECT 1 FROM activity_read r WHERE r.event_id = e.id AND r.user = ?2)",
|
||||
params![vault, user],
|
||||
|row| uint(row.get(0)?),
|
||||
)
|
||||
}
|
||||
|
||||
/// Marks the events of `user` as read; an empty `ids` marks every event the
|
||||
/// member sees. Answers how many events were marked.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn activity_mark_read(&self, vault: &str, user: &str, ids: &[u64], now_ms: u64) -> rusqlite::Result<usize> {
|
||||
if ids.is_empty() {
|
||||
let marked = self.conn().execute(
|
||||
"INSERT OR IGNORE INTO activity_read (vault, user, event_id, read_ms) \
|
||||
SELECT ?1, ?2, e.id, ?3 FROM activity e \
|
||||
WHERE e.vault = ?1 AND e.actor <> ?2 \
|
||||
AND (e.audience = 'vault' OR EXISTS (SELECT 1 FROM activity_mentions m WHERE m.event_id = e.id AND m.user = ?2))",
|
||||
params![vault, user, int(now_ms)?],
|
||||
)?;
|
||||
return Ok(marked);
|
||||
}
|
||||
let mut targets = Vec::with_capacity(ids.len());
|
||||
for id in ids {
|
||||
if self.activity_visible(vault, user, *id)? {
|
||||
targets.push(*id);
|
||||
}
|
||||
}
|
||||
let transaction = self.conn().unchecked_transaction()?;
|
||||
for id in &targets {
|
||||
transaction.execute(
|
||||
"INSERT OR IGNORE INTO activity_read (vault, user, event_id, read_ms) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![vault, user, int(*id)?, int(now_ms)?],
|
||||
)?;
|
||||
}
|
||||
transaction.commit()?;
|
||||
Ok(targets.len())
|
||||
}
|
||||
|
||||
/// The newest event id of a vault; the cursor of the next poll.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn activity_head(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn().query_row("SELECT COALESCE(MAX(id), 0) FROM activity WHERE vault = ?1", params![vault], |row| uint(row.get(0)?))
|
||||
}
|
||||
|
||||
/// True when the event is one the member sees.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
fn activity_visible(&self, vault: &str, user: &str, id: u64) -> rusqlite::Result<bool> {
|
||||
Ok(self
|
||||
.conn()
|
||||
.query_row(
|
||||
"SELECT 1 FROM activity e WHERE e.vault = ?1 AND e.id = ?2 AND e.actor <> ?3 \
|
||||
AND (e.audience = 'vault' OR EXISTS (SELECT 1 FROM activity_mentions m WHERE m.event_id = e.id AND m.user = ?3))",
|
||||
params![vault, int(id)?, user],
|
||||
|_| Ok(()),
|
||||
)
|
||||
.optional()?
|
||||
.is_some())
|
||||
}
|
||||
|
||||
/// True when the vault holds a live file under `path`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn file_live(&self, vault: &str, path: &str) -> rusqlite::Result<bool> {
|
||||
Ok(self.conn().query_row("SELECT 1 FROM files WHERE vault = ?1 AND path = ?2 AND deleted = 0", params![vault, path], |_| Ok(())).optional()?.is_some())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A store with one vault.
|
||||
fn store() -> Store {
|
||||
let store = Store::in_memory().expect("db");
|
||||
store.create_vault("v1", 1).expect("vault");
|
||||
store
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_event_reaches_the_members_it_names() {
|
||||
let store = store();
|
||||
let mentions = ["u_bob".to_owned()];
|
||||
let event = Event { kind: "comment", actor: "u_ana", audience: "vault", note: None, thread_id: None, line: None, mentions: &[] };
|
||||
store.add_activity("v1", &event, 10).expect("event");
|
||||
let mention = Event { kind: "mention", actor: "u_ana", audience: "mention", note: None, thread_id: None, line: None, mentions: &mentions };
|
||||
store.add_activity("v1", &mention, 11).expect("event");
|
||||
|
||||
// The actor never sees their own events.
|
||||
assert_eq!(store.activity_events("v1", "u_ana", 0, 10).expect("events").len(), 0);
|
||||
assert_eq!(store.activity_unread("v1", "u_ana").expect("unread"), 0);
|
||||
|
||||
// The other member sees the event of the vault and the mention.
|
||||
let events = store.activity_events("v1", "u_bob", 0, 10).expect("events");
|
||||
assert_eq!(events.len(), 2);
|
||||
assert_eq!(events[0].kind, "comment");
|
||||
assert_eq!(events[1].kind, "mention");
|
||||
assert_eq!(store.activity_unread("v1", "u_bob").expect("unread"), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_read_marking_empties_the_unread_count() {
|
||||
let store = store();
|
||||
let event = Event { kind: "comment", actor: "u_ana", audience: "vault", note: None, thread_id: None, line: None, mentions: &[] };
|
||||
let id = store.add_activity("v1", &event, 10).expect("event");
|
||||
assert_eq!(store.activity_mark_read("v1", "u_bob", &[id], 20).expect("marked"), 1);
|
||||
assert_eq!(store.activity_unread("v1", "u_bob").expect("unread"), 0);
|
||||
let events = store.activity_events("v1", "u_bob", 0, 10).expect("events");
|
||||
assert!(events[0].read);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_purged_vault_leaves_no_event_behind() {
|
||||
let store = store();
|
||||
let event = Event { kind: "comment", actor: "u_ana", audience: "vault", note: Some("note.md"), thread_id: None, line: None, mentions: &[] };
|
||||
let id = store.add_activity("v1", &event, 10).expect("event");
|
||||
assert_eq!(store.activity_head("v1").expect("head"), id);
|
||||
store.purge_vault("v1").expect("purge");
|
||||
assert_eq!(store.activity_head("v1").expect("head"), 0);
|
||||
assert_eq!(store.activity_unread("v1", "u_bob").expect("unread"), 0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
//! Account bootstrap: admin-created users and admin-issued pairing codes.
|
||||
//!
|
||||
//! A pairing code is bound to a vault and optionally to a user (`?user=<id>`; absent
|
||||
//! means the redemption creates a new user, named after the device unless the request
|
||||
//! carries `user_name`). Issuing a code for a vault id that does not exist yet creates
|
||||
//! the vault: with account-based vault creation there is no other admin bootstrap.
|
||||
//! Redeeming a code makes its user the vault's `owner` unless they are already a member.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, Query, State};
|
||||
use axum::http::HeaderMap;
|
||||
use notescraft_sync::hash_hex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::api::replies::{MintedCode, grouped_code};
|
||||
use crate::error::ApiError;
|
||||
use crate::state::{AppState, PAIRING_TTL_MS, now_ms, random_hex, require_admin, valid_vault_id};
|
||||
|
||||
/// Longest a user or vault name kept, in characters.
|
||||
pub(crate) const MAX_NAME_CHARS: usize = 80;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct UserSpec {
|
||||
/// Display name.
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct UserCreated {
|
||||
/// The new user's id.
|
||||
pub user_id: String,
|
||||
/// The display name.
|
||||
pub name: String,
|
||||
/// The palette colour chosen for the name.
|
||||
pub color: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
|
||||
/// The query of a pairing-code issue: `?user=<id>` binds the code to an existing user.
|
||||
#[derive(Deserialize, Default)]
|
||||
pub struct PairingBinding {
|
||||
/// Optional user the code redeems under.
|
||||
pub user: Option<String>,
|
||||
}
|
||||
|
||||
/// A trimmed, length-checked display name; invalid input is refused, never clamped.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 for an empty or too long name.
|
||||
pub(crate) fn trim_name(raw: &str, what: &str, max_chars: usize) -> Result<String, ApiError> {
|
||||
let trimmed = raw.trim();
|
||||
let chars = trimmed.chars().count();
|
||||
if chars == 0 || chars > max_chars {
|
||||
return Err(ApiError::bad(format!("{what} must be 1..={max_chars} characters")));
|
||||
}
|
||||
Ok(trimmed.to_owned())
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
/// Accepts a pairing-issue target: creates the vault row when the id is well formed
|
||||
/// but unknown, refuses malformed ids and deleted vaults.
|
||||
///
|
||||
/// # Errors
|
||||
/// 404 for a malformed vault id or a deleted vault, 500 on database errors.
|
||||
pub(crate) fn pairing_vault(&self, vault: &str) -> Result<(), ApiError> {
|
||||
if !valid_vault_id(vault) {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
let core = self.core();
|
||||
if core.store.vault_exists(vault)? {
|
||||
if !core.store.vault_live(vault)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
} else {
|
||||
core.store.create_vault(vault, now_ms())?;
|
||||
audit::record(&core.store, audit::VAULT_CREATED, Some(vault), None, None)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST /v1/users`: creates a user (admin); the colour comes from the palette.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without the admin token, 400 for a bad name, 500 on database errors.
|
||||
pub async fn create_user(State(state): State<AppState>, headers: HeaderMap, Json(spec): Json<UserSpec>) -> Result<Json<UserCreated>, ApiError> {
|
||||
require_admin(&state, &headers)?;
|
||||
let name = trim_name(&spec.name, "name", MAX_NAME_CHARS)?;
|
||||
let user_id = format!("u-{}", random_hex(8)?);
|
||||
let core = state.core();
|
||||
let color = crate::store::palette_color(&name).to_owned();
|
||||
core.store.create_user(&user_id, &name, &color, now_ms())?;
|
||||
audit::record(&core.store, audit::USER_CREATED, None, None, Some(&user_id))?;
|
||||
Ok(Json(UserCreated { user_id, name, color }))
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/pairing`: issues a one-time pairing code (admin), optionally
|
||||
/// bound to an existing user via `?user=<id>`.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without the admin token, 404 for a malformed vault id, a deleted vault or an
|
||||
/// unknown user, 500 on database errors.
|
||||
pub async fn issue_pairing(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Query(binding): Query<PairingBinding>,
|
||||
) -> Result<Json<MintedCode>, ApiError> {
|
||||
require_admin(&state, &headers)?;
|
||||
state.pairing_vault(&vault)?;
|
||||
if let Some(user) = binding.user.as_deref()
|
||||
&& !state.core().store.user_exists(user)?
|
||||
{
|
||||
return Err(ApiError::missing("no such user"));
|
||||
}
|
||||
let raw = random_hex(6)?;
|
||||
let code = grouped_code(&raw);
|
||||
let expires_ms = now_ms().saturating_add(PAIRING_TTL_MS);
|
||||
let db = state.core();
|
||||
db.store.add_pairing(&hash_hex(raw.as_bytes()), &vault, binding.user.as_deref(), expires_ms)?;
|
||||
audit::record(&db.store, audit::PAIRING_CREATED, Some(&vault), None, binding.user.as_deref())?;
|
||||
Ok(Json(MintedCode { code, expires_in_s: PAIRING_TTL_MS / 1000 }))
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
//! Admin-only maintenance over one vault's blob folder.
|
||||
|
||||
use std::path::Path as FsPath;
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::HeaderMap;
|
||||
use serde::Serialize;
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::error::ApiError;
|
||||
use crate::history::retention::{MIN_KEEP_REVISIONS, retention_cutoff_ms};
|
||||
use crate::state::{AppState, now_ms, require_admin, valid_vault_id};
|
||||
use crate::store::Store;
|
||||
|
||||
/// Tombstones younger than this keep their blobs, and deleted vaults wait this long
|
||||
/// before a purge removes them.
|
||||
pub use crate::store::PURGE_AFTER_MS as TOMBSTONE_GRACE_MS;
|
||||
|
||||
/// What one garbage-collection pass removed.
|
||||
#[derive(Debug, Serialize, PartialEq)]
|
||||
pub struct GcReport {
|
||||
/// Blob files removed.
|
||||
pub removed: usize,
|
||||
/// Bytes removed.
|
||||
pub bytes: u64,
|
||||
}
|
||||
|
||||
fn audit_gc(store: &Store, vault: &str, report: &GcReport) -> Result<(), ApiError> {
|
||||
audit::record(store, audit::GC_RUN, Some(vault), None, Some(&format!("removed {} bytes {}", report.removed, report.bytes)))
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
/// Removes blob files of `vault` that no record or retained revision references any
|
||||
/// more: every hash is kept while any record is live, any tombstone is younger than
|
||||
/// `grace_ms`, or any revision still inside the vault's retention window. Expired
|
||||
/// revision rows beyond the newest [`MIN_KEEP_REVISIONS`] of their file are deleted
|
||||
/// with their unique blobs. `now` and `grace_ms` are arguments so tests can age
|
||||
/// tombstones and revisions without sleeping.
|
||||
///
|
||||
/// # Errors
|
||||
/// 404 for a vault id that is not a valid id, 500 on database or I/O errors.
|
||||
pub fn gc(&self, vault: &str, grace_ms: u64, now: u64) -> Result<GcReport, ApiError> {
|
||||
if !valid_vault_id(vault) {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
let protected = {
|
||||
let core = self.core();
|
||||
let cutoff = retention_cutoff_ms(now, core.store.retention_days(vault)?);
|
||||
core.store.gc_revisions(vault, cutoff, MIN_KEEP_REVISIONS)?;
|
||||
let mut protected = core.store.protected_hashes(vault, now, grace_ms)?;
|
||||
protected.extend(core.store.revision_hashes(vault)?);
|
||||
for hash in core.store.expired_tombstone_hashes(vault, now, grace_ms)? {
|
||||
protected.remove(&hash);
|
||||
}
|
||||
protected
|
||||
};
|
||||
let mut report = GcReport { removed: 0, bytes: 0 };
|
||||
Self::sweep_blobs(&self.blobs().join(vault), &protected, &mut report)?;
|
||||
crate::chunks::gc_chunks(self, vault, now, grace_ms, &mut report)?;
|
||||
audit_gc(&self.core().store, vault, &report)?;
|
||||
Ok(report)
|
||||
}
|
||||
|
||||
/// Deletes unreferenced files under one blob directory, never touching other vaults.
|
||||
fn sweep_blobs(dir: &FsPath, protected: &std::collections::HashSet<String>, report: &mut GcReport) -> Result<(), ApiError> {
|
||||
if !std::fs::exists(dir)? {
|
||||
return Ok(());
|
||||
}
|
||||
for entry in std::fs::read_dir(dir)? {
|
||||
let entry = entry?;
|
||||
let Ok(name) = entry.file_name().into_string() else { continue };
|
||||
if protected.contains(&name) || !entry.path().is_file() {
|
||||
continue;
|
||||
}
|
||||
let bytes = entry.metadata()?.len();
|
||||
std::fs::remove_file(entry.path())?;
|
||||
report.removed = report.removed.saturating_add(1);
|
||||
report.bytes = report.bytes.saturating_add(bytes);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/gc`: garbage-collections the vault's blobs.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without the admin token, 404 for an unknown vault, 500 on database or I/O errors.
|
||||
pub async fn run_gc(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<Json<GcReport>, ApiError> {
|
||||
require_admin(&state, &headers)?;
|
||||
if !state.core().store.vault_exists(&vault)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
Ok(Json(state.gc(&vault, TOMBSTONE_GRACE_MS, now_ms())?))
|
||||
}
|
||||
|
||||
/// What one purge pass removed.
|
||||
#[derive(Debug, Serialize, PartialEq)]
|
||||
pub struct PurgeReport {
|
||||
/// Vault ids deleted more than two weeks ago that are gone for good.
|
||||
pub removed: usize,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
/// Removes every vault marked deleted more than [`crate::store::PURGE_AFTER_MS`] ago:
|
||||
/// all rows (files, memberships, invites, pairing codes, the vault row) and the
|
||||
/// blob folder. `now` is injectable so tests can age the deletions. Called hourly
|
||||
/// by the server loop; restore stops working when this runs.
|
||||
///
|
||||
/// # Errors
|
||||
/// 500 on database or I/O errors.
|
||||
pub fn purge(&self, now: u64) -> Result<PurgeReport, ApiError> {
|
||||
let mut core = self.core();
|
||||
let targets = core.store.purgeable_vaults(now, TOMBSTONE_GRACE_MS)?;
|
||||
let mut report = PurgeReport { removed: 0 };
|
||||
for vault in targets {
|
||||
core.drop_ledger(&vault);
|
||||
core.store.purge_vault(&vault)?;
|
||||
if let Err(error) = std::fs::remove_dir_all(self.blobs().join(&vault))
|
||||
&& error.kind() != std::io::ErrorKind::NotFound
|
||||
{
|
||||
return Err(error.into());
|
||||
}
|
||||
audit::record(&core.store, audit::VAULT_PURGED, Some(&vault), None, None)?;
|
||||
report.removed = report.removed.saturating_add(1);
|
||||
}
|
||||
Ok(report)
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST /v1/admin/purge`: purges every vault deleted more than two weeks ago.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without the admin token, 500 on database or I/O errors.
|
||||
pub async fn purge_route(State(state): State<AppState>, headers: HeaderMap) -> Result<Json<PurgeReport>, ApiError> {
|
||||
require_admin(&state, &headers)?;
|
||||
Ok(Json(state.purge(now_ms())?))
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
//! Security-relevant events appended to the `audit` table, and the admin listing.
|
||||
//!
|
||||
//! Rows carry ids and numbers, never a token, a pairing code or a token hash.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Query, State};
|
||||
use axum::http::HeaderMap;
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::state::{AppState, now_ms, require_admin};
|
||||
use crate::store::Store;
|
||||
|
||||
/// A vault was created.
|
||||
pub const VAULT_CREATED: &str = "vault_created";
|
||||
/// A vault was renamed.
|
||||
pub const VAULT_RENAMED: &str = "vault_renamed";
|
||||
/// A vault was soft-deleted.
|
||||
pub const VAULT_DELETED: &str = "vault_deleted";
|
||||
/// A soft delete was undone within the window.
|
||||
pub const VAULT_RESTORED: &str = "vault_restored";
|
||||
/// A vault and all its rows were purged.
|
||||
pub const VAULT_PURGED: &str = "vault_purged";
|
||||
/// A user was created (admin, pairing migration or pairing redemption).
|
||||
pub const USER_CREATED: &str = "user_created";
|
||||
/// A user was added to a vault; the detail names the role.
|
||||
pub const MEMBER_ADDED: &str = "member_added";
|
||||
/// A member's role changed; the detail names the new role.
|
||||
pub const MEMBER_ROLE_CHANGED: &str = "member_role_changed";
|
||||
/// A membership was removed or left; the detail says which.
|
||||
pub const MEMBER_REMOVED: &str = "member_removed";
|
||||
/// An invite was minted; the detail names the role.
|
||||
pub const INVITE_CREATED: &str = "invite_created";
|
||||
/// An invite was accepted; the detail names the granted role.
|
||||
pub const INVITE_ACCEPTED: &str = "invite_accepted";
|
||||
/// An invite was withdrawn before use.
|
||||
pub const INVITE_DELETED: &str = "invite_deleted";
|
||||
/// An invite code was unknown, used or expired; the detail names the reason, never the code.
|
||||
pub const INVITE_FAILED: &str = "invite_failed";
|
||||
/// Detail stored with `INVITE_FAILED`.
|
||||
pub const INVITE_FAILED_DETAIL: &str = "unknown_or_expired";
|
||||
/// An invite redemption was refused because the failure window is full.
|
||||
pub const INVITE_LOCKED_OUT: &str = "invite_locked_out";
|
||||
/// A pairing code was issued.
|
||||
pub const PAIRING_CREATED: &str = "pairing_created";
|
||||
/// A device redeemed a pairing code.
|
||||
pub const DEVICE_PAIRED: &str = "device_paired";
|
||||
/// A code was unknown or expired; the detail names the reason, never the code.
|
||||
pub const PAIRING_FAILED: &str = "pairing_failed";
|
||||
/// Detail stored with [`PAIRING_FAILED`].
|
||||
pub const PAIRING_FAILED_DETAIL: &str = "unknown_or_expired";
|
||||
/// A redemption was refused because the failure window is full.
|
||||
pub const PAIRING_LOCKED_OUT: &str = "pairing_locked_out";
|
||||
/// A device token was rotated.
|
||||
pub const TOKEN_ROTATED: &str = "token_rotated";
|
||||
/// A device was revoked.
|
||||
pub const DEVICE_REVOKED: &str = "device_revoked";
|
||||
/// An upload was rejected for exceeding the vault quota.
|
||||
pub const QUOTA_REJECTED: &str = "quota_rejected";
|
||||
/// A garbage-collection pass over a vault finished.
|
||||
pub const GC_RUN: &str = "gc_run";
|
||||
/// A file revision was restored as a new revision; the detail names the path and both seqs.
|
||||
pub const FILE_RESTORED: &str = "file_restored";
|
||||
/// A vault's history retention window changed; the detail names the new value in days.
|
||||
pub const RETENTION_CHANGED: &str = "retention_changed";
|
||||
/// A note or folder was published; the detail names the target path.
|
||||
pub const PUBLICATION_CREATED: &str = "publication_created";
|
||||
/// A publication was removed; the detail names the target path.
|
||||
pub const PUBLICATION_REMOVED: &str = "publication_removed";
|
||||
/// A share link was minted; the detail names the note path, never the token.
|
||||
pub const SHARE_LINK_CREATED: &str = "share_link_created";
|
||||
/// A share link was revoked; the detail names the note path, never the token.
|
||||
pub const SHARE_LINK_REVOKED: &str = "share_link_revoked";
|
||||
/// A vault's publish slug changed; the detail names the new slug.
|
||||
pub const SLUG_CHANGED: &str = "slug_changed";
|
||||
|
||||
/// Appends one audit row with the current clock.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn record(store: &Store, kind: &str, vault: Option<&str>, device: Option<&str>, detail: Option<&str>) -> Result<(), ApiError> {
|
||||
store.audit(now_ms(), kind, vault, device, detail)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The query of the audit listing.
|
||||
#[derive(Deserialize)]
|
||||
pub struct Limit {
|
||||
/// How many rows to return; defaults to 100 and is capped by the store.
|
||||
#[serde(default)]
|
||||
pub limit: Option<usize>,
|
||||
}
|
||||
|
||||
/// `GET /v1/audit`: the audit log, newest first, capped at the store's maximum.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without the admin token, 500 on database errors.
|
||||
pub async fn list(State(state): State<AppState>, headers: HeaderMap, Query(limit): Query<Limit>) -> Result<Json<Vec<crate::store::AuditRow>>, ApiError> {
|
||||
require_admin(&state, &headers)?;
|
||||
Ok(Json(state.core().store.audit_rows(limit.limit.unwrap_or(100))?))
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
//! Device lifecycle: pairing redemption (with lockout, creating or reusing the account),
|
||||
//! listing, token rotation, revocation.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use notescraft_sync::hash_hex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::api::accounts::{MAX_NAME_CHARS, trim_name};
|
||||
use crate::api::audit;
|
||||
use crate::error::ApiError;
|
||||
use crate::limits::{PAIRING_MAX_FAILURES, PAIRING_WINDOW_MS};
|
||||
use crate::roles::{Need, Role};
|
||||
use crate::state::{AppState, authorize, now_ms, random_hex, require_device};
|
||||
|
||||
/// Longest a device name kept (silently shortened; it is a label, not an account).
|
||||
const MAX_DEVICE_NAME: usize = 64;
|
||||
|
||||
/// A pairing request from an unpaired device.
|
||||
#[derive(Deserialize)]
|
||||
pub struct Pair {
|
||||
/// The one-time code, as issued.
|
||||
pub code: String,
|
||||
/// Human label for the device.
|
||||
pub name: String,
|
||||
/// Display name for the account a code bound to "new user" creates; when absent,
|
||||
/// the device name is used.
|
||||
pub user_name: Option<String>,
|
||||
}
|
||||
|
||||
/// The answer to a successful pairing or rotation.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct Paired {
|
||||
/// The device's id.
|
||||
pub device_id: String,
|
||||
/// The vault the pairing code was bound to.
|
||||
pub vault_id: String,
|
||||
/// The user the device belongs to; one token works for every vault of the user.
|
||||
pub user_id: String,
|
||||
/// The new bearer token; the server only ever stores its hash.
|
||||
pub token: String,
|
||||
}
|
||||
|
||||
/// One row of the vault's device list.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct DeviceListing {
|
||||
/// The device's id.
|
||||
pub id: String,
|
||||
/// The device's name.
|
||||
pub name: String,
|
||||
/// When the device was paired.
|
||||
pub created_ms: u64,
|
||||
/// When the server saw the device last, if ever.
|
||||
pub last_seen_ms: Option<u64>,
|
||||
/// True for the device making the request.
|
||||
pub current: bool,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
/// Redeems a pairing code; `now` is injectable so tests can age the lockout window.
|
||||
/// A code with a user binding reuses that account, a code without one creates an
|
||||
/// account (named after `user_name` or the device); either way the account becomes
|
||||
/// the vault's `owner` unless it is already a member.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 for an empty name, 401 for an unknown or expired code, 429 while locked out.
|
||||
pub fn pair_device(&self, raw_code: &str, name: &str, user_name: Option<&str>, now: u64) -> Result<Paired, ApiError> {
|
||||
if let Some(retry_after_s) = self.lockout().retry_after(now, PAIRING_WINDOW_MS, PAIRING_MAX_FAILURES) {
|
||||
audit::record(&self.core().store, audit::PAIRING_LOCKED_OUT, None, None, None)?;
|
||||
return Err(ApiError::locked(retry_after_s));
|
||||
}
|
||||
let name: String = name.trim().chars().take(MAX_DEVICE_NAME).collect();
|
||||
if name.is_empty() {
|
||||
return Err(ApiError::bad("name is required"));
|
||||
}
|
||||
let user_name = match user_name {
|
||||
Some(raw) => Some(trim_name(raw, "user_name", MAX_NAME_CHARS)?),
|
||||
None => None,
|
||||
};
|
||||
let normalized: String = raw_code.chars().filter(char::is_ascii_alphanumeric).map(|c| c.to_ascii_lowercase()).collect();
|
||||
let (device_id, token) = (random_hex(8)?, random_hex(32)?);
|
||||
let core = self.core();
|
||||
let Some((vault, binding)) = core.store.take_pairing(&hash_hex(normalized.as_bytes()), now)? else {
|
||||
audit::record(&core.store, audit::PAIRING_FAILED, None, None, Some(audit::PAIRING_FAILED_DETAIL))?;
|
||||
self.lockout().record_failure(now);
|
||||
return Err(ApiError::unauthorized());
|
||||
};
|
||||
let user_id = if let Some(user) = binding {
|
||||
user
|
||||
} else {
|
||||
let new_name = user_name.as_deref().unwrap_or(&name);
|
||||
let user_id = format!("u-{device_id}");
|
||||
core.store.create_user(&user_id, new_name, crate::store::palette_color(new_name), now)?;
|
||||
audit::record(&core.store, audit::USER_CREATED, None, None, Some(&user_id))?;
|
||||
user_id
|
||||
};
|
||||
let (role, added) = core.store.ensure_member(&vault, &user_id, "owner", now)?;
|
||||
if added {
|
||||
audit::record(&core.store, audit::MEMBER_ADDED, Some(&vault), Some(&device_id), Some(&role))?;
|
||||
}
|
||||
core.store.add_device(&device_id, &vault, &user_id, &name, &hash_hex(token.as_bytes()), now)?;
|
||||
audit::record(&core.store, audit::DEVICE_PAIRED, Some(&vault), Some(&device_id), None)?;
|
||||
Ok(Paired { device_id, vault_id: vault, user_id, token })
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST /v1/devices`: redeem a one-time pairing code for a device token.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 for an empty name, 401 for a bad code, 429 while the lockout is active.
|
||||
pub async fn register_device(State(state): State<AppState>, Json(request): Json<Pair>) -> Result<Json<Paired>, ApiError> {
|
||||
let now = now_ms();
|
||||
Ok(Json(state.pair_device(&request.code, &request.name, request.user_name.as_deref(), now)?))
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/devices`: the vault's devices, no token or hash ever included.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without a valid device token of the vault, 500 on database errors.
|
||||
pub async fn list_devices(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<Json<Vec<DeviceListing>>, ApiError> {
|
||||
let device_id = require_device(&state, &headers, &vault)?;
|
||||
let rows = state.core().store.list_devices(&vault)?;
|
||||
let listing = rows.into_iter().map(|(id, name, created_ms, last_seen_ms)| DeviceListing { current: id == device_id, id, name, created_ms, last_seen_ms });
|
||||
Ok(Json(listing.collect()))
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/devices/me/rotate`: mints a new device token; the old one
|
||||
/// stops working at once.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without a valid device token of the vault, 500 on database errors.
|
||||
pub async fn rotate(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<Json<Paired>, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let token = random_hex(32)?;
|
||||
if !state.core().store.rotate_device(&member.device, &vault, &hash_hex(token.as_bytes()))? {
|
||||
return Err(ApiError::unauthorized());
|
||||
}
|
||||
audit::record(&state.core().store, audit::TOKEN_ROTATED, Some(&vault), Some(&member.device), None)?;
|
||||
Ok(Json(Paired { device_id: member.device, vault_id: vault, user_id: member.user, token }))
|
||||
}
|
||||
|
||||
/// `DELETE /v1/vaults/{vault}/devices/{id}`: revokes a device of the vault. An owner
|
||||
/// may revoke any device of the vault; a member only their own devices.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a device the vault does not know, 403 for a non-owner touching
|
||||
/// somebody else's device, 500 on database errors.
|
||||
pub async fn revoke(State(state): State<AppState>, headers: HeaderMap, Path((vault, id)): Path<(String, String)>) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let target_user = state.core().store.device_user(&vault, &id)?.ok_or_else(|| ApiError::missing("no such device"))?;
|
||||
if member.role != Role::Owner && target_user != member.user {
|
||||
return Err(ApiError::forbidden());
|
||||
}
|
||||
if !state.core().store.revoke_device(&id, &vault)? {
|
||||
return Err(ApiError::missing("no such device"));
|
||||
}
|
||||
audit::record(&state.core().store, audit::DEVICE_REVOKED, Some(&vault), Some(&id), None)?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
//! Invites: single-use codes an owner mints for a vault, redeemable by any device
|
||||
//! token. Codes look like pairings (12 hex characters, shown 4-4-4) and share the
|
||||
//! same lockout counter that throttles failed pairing redemptions: five bad codes in
|
||||
//! ten minutes lock both pairing and invites for ten minutes.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use notescraft_sync::hash_hex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::api::replies::{MintedCode, grouped_code};
|
||||
use crate::error::ApiError;
|
||||
use crate::limits::{PAIRING_MAX_FAILURES, PAIRING_WINDOW_MS};
|
||||
use crate::roles::{Need, Role};
|
||||
use crate::state::{AppState, authorize, now_ms, random_hex, require_user};
|
||||
|
||||
/// An invite code lives this long; `expires_in_s` of a fresh invite is this / 1000.
|
||||
pub const INVITE_TTL_MS: u64 = 172_800_000;
|
||||
|
||||
/// One row of `GET /v1/vaults/{vault}/invites`: never the code or its hash.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct InviteRow {
|
||||
/// The invite's id, for listing and deleting.
|
||||
pub id: String,
|
||||
/// The role an acceptance grants.
|
||||
pub role: String,
|
||||
/// When the invite was created.
|
||||
pub created_ms: u64,
|
||||
/// When it stops working.
|
||||
pub expires_ms: u64,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct InviteSpec {
|
||||
/// The role the code grants: `editor` or `viewer` (never `owner`).
|
||||
pub role: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct InviteAccept {
|
||||
/// The code, with or without 4-4-4 separators.
|
||||
pub code: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct InviteAccepted {
|
||||
/// The vault the code led to.
|
||||
pub vault_id: String,
|
||||
/// The vault's display name.
|
||||
pub name: String,
|
||||
/// The member's role after the accept (kept when it was already higher).
|
||||
pub role: String,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
/// Redeems an invite code for `user` (from device `device`); `now` is injectable so
|
||||
/// tests can age the code and the lockout window. A bad, used or expired code
|
||||
/// feeds the shared lockout and is refused at once.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 for a bad, used or expired code or an invite to a dead vault,
|
||||
/// 429 while locked out, 500 on database errors.
|
||||
pub fn accept_invite(&self, raw_code: &str, device: &str, user: &str, now: u64) -> Result<InviteAccepted, ApiError> {
|
||||
if let Some(retry_after_s) = self.lockout().retry_after(now, PAIRING_WINDOW_MS, PAIRING_MAX_FAILURES) {
|
||||
let core = self.core();
|
||||
audit::record(&core.store, audit::INVITE_LOCKED_OUT, None, None, None)?;
|
||||
return Err(ApiError::locked(retry_after_s));
|
||||
}
|
||||
let normalized: String = raw_code.chars().filter(char::is_ascii_alphanumeric).map(|c| c.to_ascii_lowercase()).collect();
|
||||
let core = self.core();
|
||||
let Some((vault, invite_role)) = core.store.take_invite(&hash_hex(normalized.as_bytes()), now)? else {
|
||||
audit::record(&core.store, audit::INVITE_FAILED, None, None, Some(audit::INVITE_FAILED_DETAIL))?;
|
||||
self.lockout().record_failure(now);
|
||||
return Err(ApiError::unauthorized());
|
||||
};
|
||||
let Some(invited) = Role::parse(&invite_role) else {
|
||||
return Err(ApiError::internal(&"invite role"));
|
||||
};
|
||||
if !core.store.vault_live(&vault)? {
|
||||
return Err(ApiError::unauthorized());
|
||||
}
|
||||
let name = core.store.vault_name(&vault)?.ok_or_else(|| ApiError::missing("no such vault"))?;
|
||||
let role = match core.store.member_role(&vault, user)? {
|
||||
None => {
|
||||
let (role, added) = core.store.ensure_member(&vault, user, invited.as_str(), now)?;
|
||||
let Some(granted) = Role::parse(&role) else { return Err(ApiError::internal(&"member role")) };
|
||||
if added {
|
||||
audit::record(&core.store, audit::MEMBER_ADDED, Some(&vault), Some(device), Some(&role))?;
|
||||
}
|
||||
granted
|
||||
}
|
||||
// Already a member: an invite never demotes, the higher role stays.
|
||||
Some(raw) => {
|
||||
let Some(existing) = Role::parse(&raw) else { return Err(ApiError::internal(&"member role")) };
|
||||
let granted = existing.max(invited);
|
||||
if granted != existing {
|
||||
core.store.set_member_role(&vault, user, granted.as_str())?;
|
||||
audit::record(&core.store, audit::MEMBER_ROLE_CHANGED, Some(&vault), Some(device), Some(granted.as_str()))?;
|
||||
}
|
||||
granted
|
||||
}
|
||||
};
|
||||
audit::record(&core.store, audit::INVITE_ACCEPTED, Some(&vault), Some(device), Some(role.as_str()))?;
|
||||
Ok(InviteAccepted { vault_id: vault, name, role: role.as_str().to_owned() })
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/invites {role}`: mints a single-use code (owner).
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign vault, 403 for a non-owner, 400 for a role that is not
|
||||
/// `editor` or `viewer`, 500 on database errors.
|
||||
pub async fn create(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(spec): Json<InviteSpec>,
|
||||
) -> Result<Json<MintedCode>, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
if !matches!(spec.role.as_str(), "editor" | "viewer") {
|
||||
return Err(ApiError::bad("role must be editor or viewer"));
|
||||
}
|
||||
let raw = random_hex(6)?;
|
||||
let (invite_id, expires_ms) = (random_hex(8)?, now_ms().saturating_add(INVITE_TTL_MS));
|
||||
let core = state.core();
|
||||
core.store.add_invite(&invite_id, &vault, &spec.role, &hash_hex(raw.as_bytes()), now_ms(), expires_ms)?;
|
||||
audit::record(&core.store, audit::INVITE_CREATED, Some(&vault), Some(&member.device), Some(&spec.role))?;
|
||||
Ok(Json(MintedCode { code: grouped_code(&raw), expires_in_s: INVITE_TTL_MS / 1000 }))
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/invites`: the vault's open invites (owner), never a code.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign vault, 403 for a non-owner, 500 on database errors.
|
||||
pub async fn list(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<Json<Vec<InviteRow>>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
let core = state.core();
|
||||
let rows = core.store.list_invites(&vault)?.into_iter().map(|(id, role, created_ms, expires_ms)| InviteRow { id, role, created_ms, expires_ms }).collect();
|
||||
Ok(Json(rows))
|
||||
}
|
||||
|
||||
/// `DELETE /v1/vaults/{vault}/invites/{id}`: withdraws an invite (owner).
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign vault or unknown invite, 403 for a non-owner,
|
||||
/// 500 on database errors.
|
||||
pub async fn revoke(State(state): State<AppState>, headers: HeaderMap, Path((vault, id)): Path<(String, String)>) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
let core = state.core();
|
||||
if !core.store.delete_invite(&vault, &id)? {
|
||||
return Err(ApiError::missing("no such invite"));
|
||||
}
|
||||
audit::record(&core.store, audit::INVITE_DELETED, Some(&vault), Some(&member.device), None)?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `POST /v1/invites/accept {code}`: any device token redeems an invite. An existing
|
||||
/// member keeps the higher of their role and the invite's; the code is consumed
|
||||
/// either way. Bad or expired codes feed the shared lockout.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 for a wrong token or a bad, used or expired code, 429 while locked out,
|
||||
/// 500 on database errors.
|
||||
pub async fn accept(State(state): State<AppState>, headers: HeaderMap, Json(request): Json<InviteAccept>) -> Result<Json<InviteAccepted>, ApiError> {
|
||||
let (device, user) = require_user(&state, &headers)?;
|
||||
let now = now_ms();
|
||||
Ok(Json(state.accept_invite(&request.code, &device, &user, now)?))
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
//! Members of a vault: list, change a role, remove a member (or leave).
|
||||
//!
|
||||
//! The last owner cannot be demoted, removed, or leave; anything else an owner may
|
||||
//! do to any member, and members may remove themselves.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::error::ApiError;
|
||||
use crate::roles::{Need, Role};
|
||||
use crate::state::{AppState, authorize};
|
||||
|
||||
/// One row of `GET /v1/vaults/{vault}/members`.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct MemberRow {
|
||||
/// The user's id.
|
||||
pub user_id: String,
|
||||
/// Display name.
|
||||
pub name: String,
|
||||
/// Palette colour.
|
||||
pub color: String,
|
||||
/// `owner`, `editor` or `viewer`.
|
||||
pub role: String,
|
||||
/// Newest `last_seen_ms` over the member's live devices, `null` never seen.
|
||||
pub last_seen_ms: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct RoleChange {
|
||||
/// The target role.
|
||||
pub role: String,
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/members`: the vault's members (any member).
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for viewers that pass (read level), 404 for a foreign vault,
|
||||
/// 500 on database errors.
|
||||
pub async fn list(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<Json<Vec<MemberRow>>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let core = state.core();
|
||||
let rows = core
|
||||
.store
|
||||
.vault_members(&vault)?
|
||||
.into_iter()
|
||||
.map(|(user_id, name, color, role, last_seen_ms)| MemberRow { user_id, name, color, role, last_seen_ms })
|
||||
.collect();
|
||||
Ok(Json(rows))
|
||||
}
|
||||
|
||||
/// `PATCH /v1/vaults/{vault}/members/{user} {role}`: change a role (owner). The last
|
||||
/// owner cannot be demoted.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403, 404 for a foreign vault or unknown member, 400 for a bad role name,
|
||||
/// 409 demoting the last owner, 500 on database errors.
|
||||
pub async fn set_role(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path((vault, user)): Path<(String, String)>,
|
||||
Json(change): Json<RoleChange>,
|
||||
) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
if Role::parse(&change.role).is_none() {
|
||||
return Err(ApiError::bad("role must be owner, editor or viewer"));
|
||||
}
|
||||
let core = state.core();
|
||||
let current = core.store.member_role(&vault, &user)?.ok_or_else(|| ApiError::missing("no such member"))?;
|
||||
if current == "owner" && change.role != "owner" && core.store.owner_count(&vault)? == 1 {
|
||||
return Err(ApiError::conflict("the last owner cannot be demoted"));
|
||||
}
|
||||
if !core.store.set_member_role(&vault, &user, &change.role)? {
|
||||
return Err(ApiError::missing("no such member"));
|
||||
}
|
||||
audit::record(&core.store, audit::MEMBER_ROLE_CHANGED, Some(&vault), Some(&member.device), Some(&change.role))?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `DELETE /v1/vaults/{vault}/members/{user}`: an owner removes anyone; a member
|
||||
/// removes themselves (leaves); the last owner neither leaves nor is removed.
|
||||
/// The removed member's next request to the vault is 404.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a member removing someone else, 404 for a foreign vault or unknown
|
||||
/// member, 409 removing the last owner, 500 on database errors.
|
||||
pub async fn remove(State(state): State<AppState>, headers: HeaderMap, Path((vault, user)): Path<(String, String)>) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let core = state.core();
|
||||
let current = core.store.member_role(&vault, &user)?.ok_or_else(|| ApiError::missing("no such member"))?;
|
||||
if user != member.user && member.role != crate::roles::Role::Owner {
|
||||
return Err(ApiError::forbidden());
|
||||
}
|
||||
if current == "owner" && core.store.owner_count(&vault)? == 1 {
|
||||
return Err(ApiError::conflict("the last owner cannot be removed"));
|
||||
}
|
||||
core.store.remove_member(&vault, &user)?;
|
||||
let detail = if user == member.user { "left" } else { "removed" };
|
||||
audit::record(&core.store, audit::MEMBER_REMOVED, Some(&vault), Some(&member.device), Some(detail))?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
//! The HTTP handlers, grouped by resource; `routes` assembles the router.
|
||||
//!
|
||||
//! Route style: extractors (`State`, `HeaderMap`, `Path`, `Json`), then one `authorize`
|
||||
//! or `require_admin` check, then the store work. Every handler returns `ApiError`.
|
||||
|
||||
pub mod accounts;
|
||||
pub mod admin;
|
||||
pub mod audit;
|
||||
pub mod devices;
|
||||
pub mod invites;
|
||||
pub mod members;
|
||||
pub mod replies;
|
||||
pub mod routes;
|
||||
pub mod vaults;
|
||||
|
||||
pub use replies::MetaReply;
|
||||
pub use routes::router;
|
||||
@@ -0,0 +1,28 @@
|
||||
//! Shared reply types and code formatting for the HTTP handlers.
|
||||
|
||||
/// The minted code of a pairing or an invite: only a hash of it is ever stored.
|
||||
#[derive(Debug, serde::Serialize)]
|
||||
pub struct MintedCode {
|
||||
/// The one-time code, 12 hex characters shown as 4-4-4.
|
||||
pub code: String,
|
||||
/// Seconds until the code expires.
|
||||
pub expires_in_s: u64,
|
||||
}
|
||||
|
||||
/// Authorship of one file: `GET /v1/vaults/{vault}/files-meta/{path}`.
|
||||
#[derive(Debug, serde::Serialize)]
|
||||
pub struct MetaReply {
|
||||
/// The user id of the last writer, `None` for a write older than accounts.
|
||||
pub editor_user: Option<String>,
|
||||
/// The display name of `editor_user`, when there was one.
|
||||
pub editor_name: Option<String>,
|
||||
/// Record sequence.
|
||||
pub seq: u64,
|
||||
/// Client-provided modification time.
|
||||
pub mtime_ms: u64,
|
||||
}
|
||||
|
||||
/// Shows a one-time code as 4-4-4 character groups; only a hash of `raw` is ever stored.
|
||||
pub fn grouped_code(raw: &str) -> String {
|
||||
raw.as_bytes().chunks(4).map(|chunk| String::from_utf8_lossy(chunk).into_owned()).collect::<Vec<_>>().join("-")
|
||||
}
|
||||
@@ -0,0 +1,321 @@
|
||||
//! The sync protocol routes: changes, files, batch check. Accounts and vaults live in
|
||||
//! the sibling modules; `router` here assembles them all.
|
||||
//!
|
||||
//! See `specs/mobile-and-sync/spec.md` for the table of endpoints and
|
||||
//! `specs/collaboration/spec.md` for the role rules.
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use axum::Json;
|
||||
use axum::Router;
|
||||
use axum::body::Bytes;
|
||||
use axum::extract::{DefaultBodyLimit, Path, Query, State};
|
||||
use axum::http::header::{ETAG, IF_NONE_MATCH};
|
||||
use axum::http::{HeaderMap, HeaderValue, StatusCode};
|
||||
use axum::middleware;
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use axum::routing::{any, get, post, put};
|
||||
use notescraft_sync::{Changes, FileRecord, Outcome, Write, check_hash, check_syncable, hash_hex};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::activity;
|
||||
use crate::api::{MetaReply, accounts, admin, audit, devices, invites, members, vaults};
|
||||
use crate::chunks;
|
||||
use crate::error::ApiError;
|
||||
use crate::history::routes as history;
|
||||
use crate::live;
|
||||
use crate::middleware::request_ids;
|
||||
use crate::publish;
|
||||
use crate::roles::Need;
|
||||
use crate::state::{AppState, authorize, now_ms, store_blob};
|
||||
|
||||
/// Largest file body accepted.
|
||||
const MAX_FILE_BYTES: usize = 25 * 1024 * 1024;
|
||||
/// Most records one `/changes` page may hold.
|
||||
const MAX_PAGE: usize = 5000;
|
||||
/// Longest a `/changes` request may wait for news.
|
||||
const MAX_WAIT_SECS: u64 = 25;
|
||||
/// Most hashes one `batch/check` or `chunks/check` may ask about.
|
||||
pub(crate) const MAX_BATCH: usize = 5000;
|
||||
/// Name of the temp file the readiness probe creates in the blob folder.
|
||||
const READY_PROBE: &str = ".readyz-probe";
|
||||
|
||||
/// Builds the router over a state.
|
||||
pub fn router(state: AppState) -> Router {
|
||||
state.live().ensure_sweeper(state.clone());
|
||||
Router::new()
|
||||
.route("/healthz", get(|| async { "ok" }))
|
||||
.route("/readyz", get(readyz))
|
||||
.route("/v1/audit", get(audit::list))
|
||||
.route("/v1/users", post(accounts::create_user))
|
||||
.route("/v1/vaults", get(vaults::list).post(vaults::create))
|
||||
.route("/v1/vaults/{vault}", axum::routing::patch(vaults::rename).delete(vaults::soft_delete))
|
||||
.route("/v1/vaults/{vault}/restore", post(vaults::restore))
|
||||
.route("/v1/vaults/{vault}/members", get(members::list))
|
||||
.route("/v1/vaults/{vault}/members/{user}", axum::routing::patch(members::set_role).delete(members::remove))
|
||||
.route("/v1/vaults/{vault}/invites", get(invites::list).post(invites::create))
|
||||
.route("/v1/vaults/{vault}/invites/{id}", axum::routing::delete(invites::revoke))
|
||||
.route("/v1/invites/accept", post(invites::accept))
|
||||
.route("/v1/devices", post(devices::register_device))
|
||||
.route("/v1/vaults/{vault}/pairing", post(accounts::issue_pairing))
|
||||
.route("/v1/vaults/{vault}/devices", get(devices::list_devices))
|
||||
.route("/v1/vaults/{vault}/devices/me/rotate", post(devices::rotate))
|
||||
.route("/v1/vaults/{vault}/devices/{id}", axum::routing::delete(devices::revoke))
|
||||
.route("/v1/admin/vaults", get(vaults::admin_list))
|
||||
.route("/v1/admin/purge", post(admin::purge_route))
|
||||
.route("/v1/vaults/{vault}/changes", get(changes))
|
||||
.route("/v1/vaults/{vault}/files/{*path}", get(get_file).put(put_file).delete(delete_file))
|
||||
.route("/v1/vaults/{vault}/files-history/{*path}", get(history::list))
|
||||
.route("/v1/vaults/{vault}/chunks/check", post(chunks::routes::check_chunks))
|
||||
.route("/v1/vaults/{vault}/chunks/{hash}", put(chunks::routes::put_chunk).layer(DefaultBodyLimit::max(state.chunk_max())))
|
||||
.route("/v1/vaults/{vault}/files-restore/{*path}", post(history::restore))
|
||||
.route("/v1/vaults/{vault}/files-meta/{*path}", get(file_meta))
|
||||
.route("/v1/vaults/{vault}/batch/check", post(batch_check))
|
||||
.route("/v1/vaults/{vault}/activity", get(activity::routes::list).post(activity::routes::post))
|
||||
.route("/v1/vaults/{vault}/activity/read", post(activity::routes::read))
|
||||
.route("/v1/vaults/{vault}/gc", post(admin::run_gc))
|
||||
.route("/v1/vaults/{vault}/publications", get(publish::routes::list_publications).post(publish::routes::create_publication))
|
||||
.route("/v1/vaults/{vault}/publications/{id}", axum::routing::delete(publish::routes::delete_publication))
|
||||
.route("/v1/vaults/{vault}/share-links", get(publish::routes::list_share_links).post(publish::routes::create_share_link))
|
||||
.route("/v1/vaults/{vault}/share-links/{id}", axum::routing::delete(publish::routes::revoke_share_link))
|
||||
.route("/p/{slug}", get(publish::routes::publish_root))
|
||||
.route("/p/{slug}/", get(publish::routes::publish_root))
|
||||
.route("/p/{slug}/{*path}", get(publish::routes::publish_page))
|
||||
.route("/s/{token}", get(publish::routes::share_page))
|
||||
.route("/v1/vaults/{vault}/live/{*rest}", any(live::dispatch).layer(DefaultBodyLimit::max(live::MAX_BODY_BYTES)))
|
||||
.layer(middleware::from_fn(request_ids))
|
||||
.layer(DefaultBodyLimit::max(MAX_FILE_BYTES))
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct Ready {
|
||||
ok: bool,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
failing: Option<&'static str>,
|
||||
}
|
||||
|
||||
/// `GET /readyz`: 200 while the database accepts writes and the blob folder is writable.
|
||||
///
|
||||
/// # Errors
|
||||
/// The failing check is answered as 503, not as an error.
|
||||
pub async fn readyz(State(state): State<AppState>) -> Response {
|
||||
let database_ok = state.core().store.health().is_ok();
|
||||
let blobs_ok = database_ok && check_writable(&state.blobs().join(READY_PROBE));
|
||||
let failing = match (database_ok, blobs_ok) {
|
||||
(false, _) => Some("database"),
|
||||
(true, false) => Some("blobs"),
|
||||
(true, true) => None,
|
||||
};
|
||||
let status = if failing.is_none() { StatusCode::OK } else { StatusCode::SERVICE_UNAVAILABLE };
|
||||
(status, Json(Ready { ok: failing.is_none(), failing })).into_response()
|
||||
}
|
||||
|
||||
/// True when a temp file can be created and removed in `probe`'s parent folder.
|
||||
fn check_writable(probe: &std::path::Path) -> bool {
|
||||
std::fs::write(probe, b"").is_ok() && std::fs::remove_file(probe).is_ok()
|
||||
}
|
||||
|
||||
pub(crate) fn header_u64(headers: &HeaderMap, name: &str) -> Result<Option<u64>, ApiError> {
|
||||
match headers.get(name) {
|
||||
None => Ok(None),
|
||||
Some(value) => {
|
||||
value.to_str().ok().and_then(|v| v.parse().ok()).map(Some).ok_or_else(|| ApiError::bad(format!("{name} must be a non-negative integer")))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct ChangesQuery {
|
||||
#[serde(default)]
|
||||
since: u64,
|
||||
limit: Option<usize>,
|
||||
#[serde(default)]
|
||||
wait: u64,
|
||||
}
|
||||
|
||||
async fn changes(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Query(query): Query<ChangesQuery>,
|
||||
) -> Result<Json<Changes>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let limit = query.limit.unwrap_or(1000).clamp(1, MAX_PAGE);
|
||||
let deadline = Instant::now() + Duration::from_secs(query.wait.min(MAX_WAIT_SECS));
|
||||
loop {
|
||||
// Subscribe before reading so a write between the read and the wait still wakes us.
|
||||
let notified = state.changed().notified();
|
||||
tokio::pin!(notified);
|
||||
notified.as_mut().enable();
|
||||
let page = state.core().ledger(&vault)?.changes_since(query.since, limit);
|
||||
let left = deadline.saturating_duration_since(Instant::now());
|
||||
if !page.records.is_empty() || left.is_zero() || tokio::time::timeout(left, notified).await.is_err() {
|
||||
return Ok(Json(page));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Header metadata of one file's reply: the `ETag` and the revision and mtime headers.
|
||||
pub(crate) fn record_headers(record: &FileRecord) -> HeaderMap {
|
||||
let mut headers = HeaderMap::new();
|
||||
for (name, value) in [("etag", record.hash.clone()), ("x-seq", record.seq.to_string()), ("x-mtime-ms", record.mtime_ms.to_string())] {
|
||||
if let Ok(value) = HeaderValue::from_str(&value) {
|
||||
headers.insert(name, value);
|
||||
}
|
||||
}
|
||||
headers
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct FileQuery {
|
||||
/// Read an old revision instead of the live record.
|
||||
#[serde(default)]
|
||||
seq: Option<u64>,
|
||||
}
|
||||
|
||||
async fn get_file(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path((vault, path)): Path<(String, String)>,
|
||||
Query(query): Query<FileQuery>,
|
||||
) -> Result<Response, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
check_syncable(&path)?;
|
||||
if let Some(seq) = query.seq {
|
||||
return history::read_revision(&state, &vault, &path, seq, &headers).await;
|
||||
}
|
||||
let record = state.core().ledger(&vault)?.get(&path).filter(|r| !r.deleted).cloned().ok_or_else(|| ApiError::missing("no such file"))?;
|
||||
let reply = record_headers(&record);
|
||||
if headers.get(IF_NONE_MATCH).and_then(|v| v.to_str().ok()) == Some(record.hash.as_str()) {
|
||||
return Ok((StatusCode::NOT_MODIFIED, reply).into_response());
|
||||
}
|
||||
let blob = state.blob_path(&vault, &record.hash);
|
||||
let bytes = tokio::task::spawn_blocking(move || std::fs::read(blob)).await.map_err(|e| ApiError::internal(&e))??;
|
||||
debug_assert!(reply.contains_key(ETAG));
|
||||
Ok((StatusCode::OK, reply, bytes).into_response())
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct Current<'a> {
|
||||
current: &'a FileRecord,
|
||||
}
|
||||
|
||||
/// Answers what the ledger decided: 201 for an applied write, 200 for an unchanged
|
||||
/// one, 409 with the current record for a conflict.
|
||||
pub(crate) fn answer(outcome: Outcome) -> Response {
|
||||
match outcome {
|
||||
Outcome::Applied(record) => (StatusCode::CREATED, Json(record)).into_response(),
|
||||
Outcome::Unchanged(record) => Json(record).into_response(),
|
||||
Outcome::Conflict(record) => (StatusCode::CONFLICT, Json(Current { current: &record })).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs a write through the ledger and persists the new record, stamping the editor's
|
||||
/// user id; announces the news.
|
||||
fn commit(state: &AppState, vault: &str, user: &str, write: Write) -> Result<Response, ApiError> {
|
||||
Ok(answer(apply_write(state, vault, user, write)?))
|
||||
}
|
||||
|
||||
/// Runs a write through the ledger and persists the new record, stamping the editor's
|
||||
/// user id; announces the news. Answers what the ledger decided.
|
||||
pub(crate) fn apply_write(state: &AppState, vault: &str, user: &str, write: Write) -> Result<Outcome, ApiError> {
|
||||
let outcome = {
|
||||
let mut core = state.core();
|
||||
let outcome = core.ledger(vault)?.apply(write)?;
|
||||
if let Outcome::Applied(record) = &outcome
|
||||
&& let Err(error) = core.store.record_write(vault, record, user, now_ms())
|
||||
{
|
||||
core.drop_ledger(vault);
|
||||
return Err(error.into());
|
||||
}
|
||||
outcome
|
||||
};
|
||||
if matches!(outcome, Outcome::Applied(_)) {
|
||||
state.announce();
|
||||
}
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
/// True when `new_size` on top of the vault's live bytes (minus a replaced file) fits the quota.
|
||||
/// The counted-bytes quota check: unique blob bytes (live records and retained
|
||||
/// revisions) minus a replaced file, plus the new size.
|
||||
pub(crate) fn over_quota(state: &AppState, vault: &str, path: &str, new_size: u64) -> Result<bool, ApiError> {
|
||||
let Some(quota) = state.quota_limit() else { return Ok(false) };
|
||||
let mut core = state.core();
|
||||
let replaced = core.ledger(vault)?.get(path).filter(|record| !record.deleted).map_or(0, |record| record.size);
|
||||
let used = core.store.used_bytes(vault)?;
|
||||
Ok(used.saturating_sub(replaced).saturating_add(new_size) > quota)
|
||||
}
|
||||
|
||||
async fn put_file(State(state): State<AppState>, headers: HeaderMap, Path((vault, path)): Path<(String, String)>, body: Bytes) -> Result<Response, ApiError> {
|
||||
// A chunked upload arrives with its content already stored as chunks; the body is empty.
|
||||
let chunk_list = headers.get("x-chunks").and_then(|value| value.to_str().ok()).map(str::to_owned);
|
||||
if let Some(chunk_list) = chunk_list {
|
||||
return chunks::routes::put_chunked_file(state, headers, vault, path, chunk_list, body).await;
|
||||
}
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
check_syncable(&path)?;
|
||||
let base_seq = header_u64(&headers, "x-base-seq")?.unwrap_or(0);
|
||||
let mtime_ms = header_u64(&headers, "x-mtime-ms")?.unwrap_or_else(now_ms);
|
||||
let hash = hash_hex(&body);
|
||||
let size = u64::try_from(body.len()).map_err(|e| ApiError::internal(&e))?;
|
||||
if over_quota(&state, &vault, &path, size)? {
|
||||
audit::record(&state.core().store, audit::QUOTA_REJECTED, Some(&vault), Some(&member.device), None)?;
|
||||
return Err(ApiError::quota());
|
||||
}
|
||||
let (blob, size) = (state.blob_path(&vault, &hash), size);
|
||||
tokio::task::spawn_blocking(move || store_blob(&blob, &body)).await.map_err(|e| ApiError::internal(&e))??;
|
||||
commit(&state, &vault, &member.user, Write { path, hash: Some(hash), size, mtime_ms, base_seq })
|
||||
}
|
||||
|
||||
async fn delete_file(State(state): State<AppState>, headers: HeaderMap, Path((vault, path)): Path<(String, String)>) -> Result<Response, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
let base_seq = header_u64(&headers, "x-base-seq")?.unwrap_or(0);
|
||||
commit(&state, &vault, &member.user, Write { path, hash: None, size: 0, mtime_ms: now_ms(), base_seq })
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub(crate) struct Hashes {
|
||||
pub(crate) hashes: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub(crate) struct Missing {
|
||||
pub(crate) missing: Vec<String>,
|
||||
}
|
||||
|
||||
async fn batch_check(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(request): Json<Hashes>,
|
||||
) -> Result<Json<Missing>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
if request.hashes.len() > MAX_BATCH {
|
||||
return Err(ApiError::bad(format!("at most {MAX_BATCH} hashes")));
|
||||
}
|
||||
request.hashes.iter().try_for_each(|h| check_hash(h))?;
|
||||
let missing = tokio::task::spawn_blocking(move || request.hashes.into_iter().filter(|h| !state.blob_path(&vault, h).exists()).collect())
|
||||
.await
|
||||
.map_err(|e| ApiError::internal(&e))?;
|
||||
Ok(Json(Missing { missing }))
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/files-meta/{path}`: who last wrote this file.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403, 404 like every vault route, 500 on database errors.
|
||||
async fn file_meta(State(state): State<AppState>, headers: HeaderMap, Path((vault, path)): Path<(String, String)>) -> Result<Json<MetaReply>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
check_syncable(&path)?;
|
||||
let core = state.core();
|
||||
let Some((editor_user, seq, mtime_ms)) = core.store.file_meta(&vault, &path)? else {
|
||||
return Err(ApiError::missing("no such file"));
|
||||
};
|
||||
let editor_name = match editor_user.as_deref() {
|
||||
Some(user) => core.store.user_name_color(user)?.map(|(name, _)| name),
|
||||
None => None,
|
||||
};
|
||||
Ok(Json(MetaReply { editor_user, editor_name, seq, mtime_ms }))
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
//! Vault management for accounts: list mine, create, rename, soft-delete, restore,
|
||||
//! plus the admin listing of everything. Roles come from `authorize`; a deleted vault
|
||||
//! answers 404 to every member request (restore is its own endpoint and checks the
|
||||
//! membership row directly).
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use notescraft_sync::hash_hex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::api::accounts::{MAX_NAME_CHARS, trim_name};
|
||||
use crate::api::admin::TOMBSTONE_GRACE_MS;
|
||||
use crate::api::audit;
|
||||
use crate::error::ApiError;
|
||||
use crate::history::valid_retention_days;
|
||||
use crate::publish::links::valid_slug;
|
||||
use crate::roles::{Need, Role};
|
||||
use crate::state::{AppState, authorize, bearer, now_ms, random_hex, require_admin, require_user};
|
||||
|
||||
/// Most vaults one user may own.
|
||||
const MAX_OWNED_VAULTS: u64 = 50;
|
||||
|
||||
/// One row of `GET /v1/vaults`, the caller's memberships by name.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct VaultListing {
|
||||
/// The vault id.
|
||||
pub id: String,
|
||||
/// The display name.
|
||||
pub name: String,
|
||||
/// The caller's role in this vault.
|
||||
pub role: String,
|
||||
/// How many members the vault has.
|
||||
pub members: u64,
|
||||
/// Counted bytes: unique contents of live records and retained revisions.
|
||||
pub bytes_used: u64,
|
||||
/// The server's per-vault live-bytes quota, `null` when unlimited.
|
||||
pub quota_bytes: Option<u64>,
|
||||
/// Highest applied sequence number.
|
||||
pub head_seq: u64,
|
||||
/// When the last write landed, or the creation when none did.
|
||||
pub last_change_ms: u64,
|
||||
}
|
||||
|
||||
/// One row of `GET /v1/admin/vaults`, every vault including deleted ones.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct AdminVaultRow {
|
||||
/// The vault id.
|
||||
pub id: String,
|
||||
/// The display name.
|
||||
pub name: String,
|
||||
/// When the vault was created.
|
||||
pub created_ms: u64,
|
||||
/// The deletion mark, when the vault is soft-deleted.
|
||||
pub deleted_ms: Option<u64>,
|
||||
/// How many members the vault has.
|
||||
pub members: u64,
|
||||
/// Live bytes over non-deleted files.
|
||||
pub bytes_used: u64,
|
||||
/// Highest applied sequence number.
|
||||
pub head_seq: u64,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct VaultName {
|
||||
/// The new display name.
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
/// The body of `PATCH /v1/vaults/{vault}`: a new display name, a new history retention
|
||||
/// window in days, a new publish slug, or any of them.
|
||||
#[derive(Deserialize)]
|
||||
pub struct VaultPatch {
|
||||
/// The new display name, when the caller changes it.
|
||||
#[serde(default)]
|
||||
pub name: Option<String>,
|
||||
/// The new history retention window in days (1..=365), when the caller changes it.
|
||||
#[serde(default)]
|
||||
pub retention_days: Option<u64>,
|
||||
/// The new publish slug (3..=40 characters of `a-z`, `0-9` and `-`), when the
|
||||
/// caller changes it.
|
||||
#[serde(default)]
|
||||
pub slug: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct VaultCreated {
|
||||
/// The server-generated vault id.
|
||||
pub vault_id: String,
|
||||
/// The display name.
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults`: the caller's memberships (one device token, every vault of the
|
||||
/// user), sorted by display name; other users' vaults are never listed.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 for a wrong token, 500 on database errors.
|
||||
pub async fn list(State(state): State<AppState>, headers: HeaderMap) -> Result<Json<Vec<VaultListing>>, ApiError> {
|
||||
let (_, user) = require_user(&state, &headers)?;
|
||||
let core = state.core();
|
||||
let quota_bytes = state.quota_limit();
|
||||
let mut listings = Vec::new();
|
||||
for (id, name, role) in core.store.user_vaults(&user)? {
|
||||
let members = core.store.member_count(&id)?;
|
||||
let bytes_used = core.store.used_bytes(&id)?;
|
||||
let head_seq = core.store.head_seq(&id)?;
|
||||
let last_change_ms = core.store.last_change_ms(&id)?;
|
||||
listings.push(VaultListing { id, name, role, members, bytes_used, quota_bytes, head_seq, last_change_ms });
|
||||
}
|
||||
Ok(Json(listings))
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults {name}`: creates a vault owned by the caller; names are
|
||||
/// 1..=80 characters after trimming, and one user owns at most 50 live vaults.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 for a wrong token, 400 for a bad name, 409 over the ownership cap,
|
||||
/// 500 on database errors.
|
||||
pub async fn create(State(state): State<AppState>, headers: HeaderMap, Json(request): Json<VaultName>) -> Result<Json<VaultCreated>, ApiError> {
|
||||
let (_, user) = require_user(&state, &headers)?;
|
||||
let name = trim_name(&request.name, "name", MAX_NAME_CHARS)?;
|
||||
let core = state.core();
|
||||
if core.store.owned_vaults(&user)? >= MAX_OWNED_VAULTS {
|
||||
return Err(ApiError::conflict(format!("at most {MAX_OWNED_VAULTS} owned vaults")));
|
||||
}
|
||||
let vault_id = format!("v-{}", random_hex(8)?);
|
||||
let stamp = now_ms();
|
||||
core.store.create_vault(&vault_id, stamp)?;
|
||||
core.store.ensure_member(&vault_id, &user, "owner", stamp)?;
|
||||
core.store.rename_vault(&vault_id, &name)?;
|
||||
audit::record(&core.store, audit::VAULT_CREATED, Some(&vault_id), None, Some(&name))?;
|
||||
Ok(Json(VaultCreated { vault_id, name }))
|
||||
}
|
||||
|
||||
/// `PATCH /v1/vaults/{vault}`: renames a vault, changes its history retention window or
|
||||
/// its publish slug (owner); names are 1..=80 characters after trimming, retention is
|
||||
/// 1..=365 days, a slug is 3..=40 characters of `a-z`, `0-9` and `-` and is unique
|
||||
/// across the server.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign or deleted vault, 403 for a non-owner, 400 for a bad name,
|
||||
/// retention or slug, 409 for a slug another vault already takes, 500 on database
|
||||
/// errors.
|
||||
pub async fn rename(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(request): Json<VaultPatch>,
|
||||
) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
let retention = request
|
||||
.retention_days
|
||||
.map(|days| if valid_retention_days(days) { Ok(days) } else { Err(ApiError::bad("retention_days must be 1..=365")) })
|
||||
.transpose()?;
|
||||
let name = request.name.map(|raw| trim_name(&raw, "name", MAX_NAME_CHARS)).transpose()?;
|
||||
let slug =
|
||||
request.slug.map(|raw| if valid_slug(&raw) { Ok(raw) } else { Err(ApiError::bad("slug must be 3..=40 characters of a-z, 0-9 and -")) }).transpose()?;
|
||||
if retention.is_none() && name.is_none() && slug.is_none() {
|
||||
return Err(ApiError::bad("name, retention_days or slug required"));
|
||||
}
|
||||
let core = state.core();
|
||||
if let Some(days) = retention {
|
||||
if !core.store.set_retention_days(&vault, days)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
audit::record(&core.store, audit::RETENTION_CHANGED, Some(&vault), Some(&member.device), Some(&days.to_string()))?;
|
||||
}
|
||||
if let Some(name) = name {
|
||||
if !core.store.rename_vault(&vault, &name)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
audit::record(&core.store, audit::VAULT_RENAMED, Some(&vault), Some(&member.device), Some(&name))?;
|
||||
}
|
||||
if let Some(slug) = slug {
|
||||
if !core.store.set_vault_slug(&vault, &slug)? {
|
||||
return Err(ApiError::conflict("slug already in use"));
|
||||
}
|
||||
audit::record(&core.store, audit::SLUG_CHANGED, Some(&vault), Some(&member.device), Some(&slug))?;
|
||||
}
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `DELETE /v1/vaults/{vault}`: soft-deletes the vault (owner). It disappears from
|
||||
/// lists, blocks every member request with 404, and is purgeable after two weeks.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign or already deleted vault, 403 for a non-owner,
|
||||
/// 500 on database errors.
|
||||
pub async fn soft_delete(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
let core = state.core();
|
||||
if !core.store.delete_vault(&vault, now_ms())? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
audit::record(&core.store, audit::VAULT_DELETED, Some(&vault), Some(&member.device), None)?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/restore`: undoes a soft delete (owner, within the purge
|
||||
/// window). The vault answers 404 everywhere while deleted, so membership, role and
|
||||
/// the deletions age are checked directly; every failure and a non-owner is `404`, so
|
||||
/// existence is never revealed.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 as above, 500 on database errors.
|
||||
pub async fn restore(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<StatusCode, ApiError> {
|
||||
// A deleted vault answers 404 to `authorize`, so authenticate by token alone and
|
||||
// look for the caller's membership in the still-existing vault row.
|
||||
let token = bearer(&headers).ok_or_else(ApiError::unauthorized)?;
|
||||
let hash = hash_hex(token.as_bytes());
|
||||
let core = state.core();
|
||||
if !core.store.vault_exists(&vault)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
let member = match core.store.device_for(&hash)? {
|
||||
None => None,
|
||||
Some((device, user)) => {
|
||||
let role = core.store.member_role(&vault, &user)?.and_then(|raw| Role::parse(&raw));
|
||||
role.map(|role| (device, role))
|
||||
}
|
||||
};
|
||||
let Some((device, Role::Owner)) = member else {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
};
|
||||
if !core.store.restore_vault(&vault, now_ms(), TOMBSTONE_GRACE_MS)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
audit::record(&core.store, audit::VAULT_RESTORED, Some(&vault), Some(&device), None)?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `GET /v1/admin/vaults`: every vault, live and deleted (admin).
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 without the admin token, 500 on database errors.
|
||||
pub async fn admin_list(State(state): State<AppState>, headers: HeaderMap) -> Result<Json<Vec<AdminVaultRow>>, ApiError> {
|
||||
require_admin(&state, &headers)?;
|
||||
let core = state.core();
|
||||
let mut rows = Vec::new();
|
||||
for (id, name, created_ms, deleted_ms) in core.store.all_vaults()? {
|
||||
let members = core.store.member_count(&id)?;
|
||||
let bytes_used = core.store.live_bytes(&id)?;
|
||||
let head_seq = core.store.head_seq(&id)?;
|
||||
rows.push(AdminVaultRow { id, name, created_ms, deleted_ms, members, bytes_used, head_seq });
|
||||
}
|
||||
Ok(Json(rows))
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
//! A small LRU cache: at most a cap of values per vault id, evicting the least
|
||||
//! recently stamped one when full. Values borrowed from it stay borrowed: eviction
|
||||
//! happens only on the next `get_or_insert`, so no value is dropped mid-use.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
use crate::error::ApiError;
|
||||
|
||||
/// Hand-rolled LRU to avoid a new dependency. Stamps order touches; a linear scan
|
||||
/// over the stamps is fine because caps stay tiny (the ledger cache cap is 64).
|
||||
pub(crate) struct Lru<V> {
|
||||
cap: usize,
|
||||
stamps: u64,
|
||||
entries: HashMap<String, (V, u64)>,
|
||||
}
|
||||
|
||||
impl<V> Lru<V> {
|
||||
/// An empty cache that keeps at most `cap` values.
|
||||
pub fn new(cap: usize) -> Self {
|
||||
Self { cap: cap.max(1), stamps: 0, entries: HashMap::new() }
|
||||
}
|
||||
|
||||
/// The mutable value of `key`: inserted from `fallback(key)` on a miss (an error
|
||||
/// passes through without caching), evicting to the cap after the insert.
|
||||
pub fn get_or_insert(&mut self, key: &str, fallback: impl Fn(&str) -> Result<V, ApiError>) -> Result<&mut V, ApiError> {
|
||||
if !self.entries.contains_key(key) {
|
||||
let value = fallback(key)?;
|
||||
self.stamps = self.stamps.wrapping_add(1);
|
||||
self.entries.insert(key.to_owned(), (value, self.stamps));
|
||||
self.evict();
|
||||
}
|
||||
let (_, stamp) = self.entries.get_mut(key).ok_or_else(|| ApiError::internal(&"lru entry vanished"))?;
|
||||
self.stamps = self.stamps.wrapping_add(1);
|
||||
*stamp = self.stamps;
|
||||
Ok(&mut self.entries.get_mut(key).ok_or_else(|| ApiError::internal(&"lru entry vanished"))?.0)
|
||||
}
|
||||
|
||||
/// Forgets a key so the next use reloads it.
|
||||
pub fn remove(&mut self, key: &str) {
|
||||
self.entries.remove(key);
|
||||
}
|
||||
|
||||
/// Drops the entry that was touched longest ago while over the cap.
|
||||
fn evict(&mut self) {
|
||||
while self.entries.len() > self.cap {
|
||||
let oldest = self.entries.iter().min_by_key(|(_, (_, stamp))| *stamp).map(|(key, _)| key.clone());
|
||||
let Some(oldest) = oldest else { break };
|
||||
self.entries.remove(&oldest);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::cell::Cell;
|
||||
|
||||
#[test]
|
||||
fn evicts_the_least_recently_used_over_a_cap_of_two() -> Result<(), ApiError> {
|
||||
let mut cache = Lru::new(2);
|
||||
let reloads = Cell::new(0_u32);
|
||||
// A fresh closure per call: no move aliases, and every miss counts.
|
||||
let miss = || {
|
||||
|_: &str| {
|
||||
reloads.set(reloads.get() + 1);
|
||||
Ok(1_u32)
|
||||
}
|
||||
};
|
||||
|
||||
let _ = *cache.get_or_insert("aa", miss())?;
|
||||
let _ = *cache.get_or_insert("bbb", miss())?;
|
||||
// Touch `aa` and mutate it: a hit reloads nothing.
|
||||
let held = cache.get_or_insert("aa", miss())?;
|
||||
*held = 10;
|
||||
assert_eq!(reloads.get(), 2);
|
||||
// The next miss evicts `bbb` (the least recent); only `aa` and `cccc` survive.
|
||||
let _ = *cache.get_or_insert("cccc", miss())?;
|
||||
assert_eq!(reloads.get(), 3);
|
||||
assert_eq!(*cache.get_or_insert("aa", miss())?, 10, "the touched survivor keeps its mutated value");
|
||||
assert_eq!(reloads.get(), 3);
|
||||
// The fresh `bb` misses and evicts `cccc` (the least recent); the touched `aa`
|
||||
// and its mutation survive untouched.
|
||||
let _ = *cache.get_or_insert("bb", miss())?;
|
||||
assert_eq!(reloads.get(), 4);
|
||||
*cache.get_or_insert("aa", miss())? = 10;
|
||||
assert_eq!(reloads.get(), 4, "aa was never evicted");
|
||||
// A later use of the evicted `cccc` reloads fresh.
|
||||
let _ = *cache.get_or_insert("cccc", miss())?;
|
||||
assert_eq!(reloads.get(), 5, "cccc was evicted between the two uses");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn errors_are_not_cached_and_remove_forces_a_reload() -> Result<(), ApiError> {
|
||||
let mut cache = Lru::new(2);
|
||||
let reloads = Cell::new(0_u32);
|
||||
let failing: fn(&str) -> Result<u32, ApiError> = |_| Err(ApiError::bad("miss"));
|
||||
assert!(cache.get_or_insert("aa", failing).is_err());
|
||||
assert_eq!(
|
||||
*cache.get_or_insert("aa", |_: &str| {
|
||||
reloads.set(reloads.get() + 1);
|
||||
Ok(1_u32)
|
||||
})?,
|
||||
1,
|
||||
"the failed insert left nothing cached"
|
||||
);
|
||||
cache.remove("aa");
|
||||
assert_eq!(
|
||||
*cache.get_or_insert("aa", |_: &str| {
|
||||
reloads.set(reloads.get() + 1);
|
||||
Ok(1_u32)
|
||||
})?,
|
||||
1,
|
||||
"removed keys reload"
|
||||
);
|
||||
assert_eq!(reloads.get(), 2);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
//! Assembling a chunked file: a streaming SHA-256 pass over the chunk files and a
|
||||
//! streaming atomic write into the vault's blob folder. No chunk's byte leaves disk
|
||||
//! in bulk: each file is read through a fixed buffer.
|
||||
|
||||
use std::fs::File;
|
||||
use std::io::{self, Read};
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
/// The largest single read at a time.
|
||||
const READ_BUFFER: usize = 64 * 1024;
|
||||
|
||||
/// What went wrong while assembling.
|
||||
#[derive(Debug)]
|
||||
pub(crate) enum AssembleError {
|
||||
/// The chunks' total exceeds the file cap.
|
||||
TooLarge(u64),
|
||||
/// Reading a chunk or writing the blob failed.
|
||||
Io(io::Error),
|
||||
}
|
||||
|
||||
impl From<io::Error> for AssembleError {
|
||||
fn from(error: io::Error) -> Self {
|
||||
Self::Io(error)
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads every chunk in order, answers its SHA-256 as lowercase hex and the total
|
||||
/// size, and stops beyond `file_cap` with [`AssembleError::TooLarge`]. The chunk
|
||||
/// paths must exist; the caller checks that first.
|
||||
///
|
||||
/// # Errors
|
||||
/// Reading a chunk failed, or the streaming total ran over the cap.
|
||||
pub(crate) fn chunk_stream_hash(chunk_paths: &[PathBuf], file_cap: u64) -> Result<(String, u64), AssembleError> {
|
||||
let mut hasher = Sha256::new();
|
||||
let mut total = 0u64;
|
||||
let mut buffer = vec![0u8; READ_BUFFER];
|
||||
for path in chunk_paths {
|
||||
let mut file = File::open(path)?;
|
||||
loop {
|
||||
let read = read_upto(&mut file, &mut buffer)?;
|
||||
if read == 0 {
|
||||
break;
|
||||
}
|
||||
// `read` is at most READ_BUFFER, a 64 KiB constant, so it always converts.
|
||||
total = total.checked_add(u64::try_from(read).unwrap_or(u64::MAX)).ok_or(AssembleError::TooLarge(u64::MAX))?;
|
||||
if total > file_cap {
|
||||
return Err(AssembleError::TooLarge(total));
|
||||
}
|
||||
hasher.update(&buffer[..read]);
|
||||
}
|
||||
}
|
||||
Ok((hex_digest(&hasher.finalize()), total))
|
||||
}
|
||||
|
||||
/// Writes the chunk files into `blob_path` in order through a streaming atomic
|
||||
/// rename when the blob is not there yet; skipped content-addressed, so an existing
|
||||
/// blob of the same name already holds exactly these bytes.
|
||||
///
|
||||
/// # Errors
|
||||
/// Opening, reading or writing failed.
|
||||
pub(crate) fn assemble_blob(chunk_paths: &[PathBuf], blob_path: &Path) -> io::Result<()> {
|
||||
if std::fs::exists(blob_path)? {
|
||||
return Ok(());
|
||||
}
|
||||
craft_atomic::atomic_write_with(blob_path, |target| {
|
||||
for path in chunk_paths {
|
||||
let mut source = File::open(path)?;
|
||||
io::copy(&mut source, target)?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
/// Reads into `buffer` until it is full or EOF; at most a full buffer.
|
||||
fn read_upto(file: &mut File, buffer: &mut [u8]) -> io::Result<usize> {
|
||||
let mut read = 0usize;
|
||||
while read < buffer.len() {
|
||||
match file.read(&mut buffer[read..]) {
|
||||
Ok(0) => break,
|
||||
Ok(n) => read = read.saturating_add(n),
|
||||
Err(error) if error.kind() == io::ErrorKind::Interrupted => {}
|
||||
Err(error) => return Err(error),
|
||||
}
|
||||
}
|
||||
Ok(read)
|
||||
}
|
||||
|
||||
/// The digest as 64 lowercase hex digits; the same shape `notescraft_sync::hash_hex`
|
||||
/// gives.
|
||||
fn hex_digest(digest: &[u8]) -> String {
|
||||
const HEX: &[u8; 16] = b"0123456789abcdef";
|
||||
let mut out = String::with_capacity(digest.len() * 2);
|
||||
for &byte in digest {
|
||||
let high = usize::from(byte >> 4);
|
||||
let low = usize::from(byte & 0x0f);
|
||||
out.push(char::from(HEX.get(high).copied().unwrap_or(b'0')));
|
||||
out.push(char::from(HEX.get(low).copied().unwrap_or(b'0')));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn temp_dir(name: &str) -> PathBuf {
|
||||
let dir = std::env::temp_dir().join(format!("nc-chunks-{name}-{}", std::process::id()));
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
std::fs::create_dir_all(&dir).expect("temp dir");
|
||||
dir
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_stream_hash_is_the_hash_of_the_joined_chunk_bytes() {
|
||||
let dir = temp_dir("hash");
|
||||
let (first, second) = ("hello ".as_bytes(), "world\n".as_bytes());
|
||||
std::fs::write(dir.join("a"), first).expect("chunk file");
|
||||
std::fs::write(dir.join("b"), second).expect("chunk file");
|
||||
let (hash, size) = chunk_stream_hash(&[dir.join("a"), dir.join("b")], 1000).expect("hash");
|
||||
let joined: Vec<u8> = first.iter().chain(second.iter()).copied().collect();
|
||||
assert_eq!((hash, size), (notescraft_sync::hash_hex(&joined), u64::try_from(joined.len()).expect("small")));
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_stream_over_the_cap_stops_early() {
|
||||
let dir = temp_dir("cap");
|
||||
std::fs::write(dir.join("c"), vec![7u8; 500]).expect("chunk file");
|
||||
let result = chunk_stream_hash(&[dir.join("c")], 100);
|
||||
let diagnosis = format!("{result:?}");
|
||||
assert!(matches!(result.map(|_| ()).err(), Some(AssembleError::TooLarge(_))), "{diagnosis}");
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assembly_streams_chunks_into_the_blob_and_skips_an_existing_one() {
|
||||
let dir = temp_dir("assemble");
|
||||
let blob = dir.join("blob");
|
||||
for (name, bytes) in [("a", "part1 ".as_bytes()), ("b", "part2".as_bytes())] {
|
||||
std::fs::write(dir.join(name), bytes).expect("chunk file");
|
||||
}
|
||||
assemble_blob(&[dir.join("a"), dir.join("b")], &blob).expect("assemble");
|
||||
assert_eq!(std::fs::read(&blob).expect("blob"), b"part1 part2");
|
||||
// A second assemble is a no-op: the blob already holds the right bytes.
|
||||
std::fs::write(dir.join("b"), b"CHANGED").expect("rewrite");
|
||||
assemble_blob(&[dir.join("a"), dir.join("b")], &blob).expect("skip");
|
||||
assert_eq!(std::fs::read(&blob).expect("blob"), b"part1 part2");
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
//! Chunked uploads over the sync HTTP API: large files arrive as 4 MiB
|
||||
//! content-addressed chunks and are assembled server side through the same ledger
|
||||
//! path as a plain `PUT`, so identical files dedup and clients read the assembled
|
||||
//! blob the way they read any file.
|
||||
//!
|
||||
//! Endpoints ([`routes`]): `PUT /v1/vaults/{vault}/chunks/{hash}` (one chunk,
|
||||
//! idempotent), `POST /v1/vaults/{vault}/chunks/check` (which chunks are missing,
|
||||
//! the resume look-ahead) and `PUT /files/{path}` with the `X-Chunks` header
|
||||
//! (assembly). [`assemble`] streams chunk bytes through a fixed buffer; no
|
||||
//! file-sized allocation is made.
|
||||
//!
|
||||
//! The module's quota choice, as the task asked: `bytes_used` counts the unique
|
||||
//! bytes of chunks **referenced by an assembled file** plus the unique blob bytes of
|
||||
//! files **not** assembled from chunks — a chunked file counts as its chunks, never
|
||||
//! both sides. An uploaded chunk is free until a file references it; re-`PUT`s are
|
||||
//! idempotent, and the GC (via `AppState::gc` → [`gc_chunks`]) frees never-assembled
|
||||
//! chunks once they are older than the tombstone grace.
|
||||
|
||||
mod assemble;
|
||||
pub(crate) mod routes;
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use crate::api::admin::GcReport;
|
||||
use crate::error::ApiError;
|
||||
use crate::state::AppState;
|
||||
|
||||
/// The largest single chunk, in bytes: 4 MiB (the chunk route's body cap).
|
||||
pub const MAX_CHUNK_BYTES: usize = 4 * 1024 * 1024;
|
||||
/// The largest file assembled from chunks, in bytes: the 25 MiB whole-`PUT` cap
|
||||
/// raised to 1 GiB for chunked uploads.
|
||||
pub const MAX_CHUNKED_BYTES: u64 = 1024 * 1024 * 1024;
|
||||
/// The most chunks one file may be assembled from.
|
||||
pub const MAX_FILE_CHUNKS: usize = 4096;
|
||||
/// The chunk folder's name inside a vault's blob folder.
|
||||
pub(crate) const CHUNK_DIR: &str = "chunks";
|
||||
/// How many chunk files one GC pass removes; the next pass takes the rest.
|
||||
const MAX_GC_FILES: usize = 5000;
|
||||
|
||||
/// Removes the vault's chunk debris: `file_chunks` rows of files no record or
|
||||
/// retained revision references any more, then the chunk rows and files nothing
|
||||
/// references (once older than `grace`), then orphan chunk files no row backs.
|
||||
/// `now` is injected so tests age without sleeping; counts and bytes join the
|
||||
/// report of the blob sweep.
|
||||
///
|
||||
/// # Errors
|
||||
/// 500 on database or I/O errors.
|
||||
pub(crate) fn gc_chunks(state: &AppState, vault: &str, now: u64, grace: u64, report: &mut GcReport) -> Result<(), ApiError> {
|
||||
let (collectible, mut keep) = {
|
||||
let core = state.core();
|
||||
core.store.gc_file_chunks(vault, now, grace)?;
|
||||
let keep = core.store.chunk_hashes(vault)?;
|
||||
(core.store.collectible_chunks(vault, now, grace)?, keep)
|
||||
};
|
||||
for (hash, size) in &collectible {
|
||||
remove_chunk_file(state.chunk_path(vault, hash).as_path(), *size, report)?;
|
||||
}
|
||||
let dead: Vec<String> = collectible.into_iter().map(|(hash, _)| hash).collect();
|
||||
{
|
||||
let core = state.core();
|
||||
core.store.drop_chunks(vault, &dead)?;
|
||||
}
|
||||
for hash in &dead {
|
||||
keep.remove(hash);
|
||||
}
|
||||
sweep_orphans(state, vault, &keep, now, grace, report)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Removes one chunk file, joining it to the report; true when it was there.
|
||||
fn remove_chunk_file(path: &std::path::Path, size: u64, report: &mut GcReport) -> Result<bool, ApiError> {
|
||||
match std::fs::remove_file(path) {
|
||||
Ok(()) => {
|
||||
report.removed = report.removed.saturating_add(1);
|
||||
report.bytes = report.bytes.saturating_add(size);
|
||||
Ok(true)
|
||||
}
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false),
|
||||
Err(error) => Err(error.into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes chunk files no chunk row backs — crash leftovers of an upload whose row
|
||||
/// never landed — once they are older than the grace, at most [`MAX_GC_FILES`] a pass.
|
||||
fn sweep_orphans(state: &AppState, vault: &str, keep: &HashSet<String>, now: u64, grace: u64, report: &mut GcReport) -> Result<(), ApiError> {
|
||||
let dir = state.chunk_dir(vault);
|
||||
if !std::fs::exists(&dir)? {
|
||||
return Ok(());
|
||||
}
|
||||
let mut swept = 0usize;
|
||||
for entry in std::fs::read_dir(&dir)? {
|
||||
if swept >= MAX_GC_FILES {
|
||||
return Ok(());
|
||||
}
|
||||
let entry = entry?;
|
||||
let Ok(name) = entry.file_name().into_string() else { continue };
|
||||
if keep.contains(&name) || !entry.path().is_file() {
|
||||
continue;
|
||||
}
|
||||
let metadata = entry.metadata()?;
|
||||
let Some(mtime_ms) = metadata.modified().ok().and_then(epoch_ms) else { continue };
|
||||
if now.saturating_sub(mtime_ms) < grace {
|
||||
continue; // young: its row may still be landing
|
||||
}
|
||||
if remove_chunk_file(&entry.path(), metadata.len(), report)? {
|
||||
swept = swept.saturating_add(1);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A time as milliseconds since the epoch; `None` before it or on a broken clock.
|
||||
fn epoch_ms(time: std::time::SystemTime) -> Option<u64> {
|
||||
time.duration_since(std::time::UNIX_EPOCH).ok().and_then(|span| u64::try_from(span.as_millis()).ok())
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
//! The chunk endpoints: one chunk up (`PUT /chunks/{hash}`), the resume check
|
||||
//! (`POST /chunks/check`) and the chunked file assembly (`PUT /files/{path}` with
|
||||
//! `X-Chunks`). Assembly writes the blob and records the file's chunk list, then
|
||||
//! applies the write through the same ledger path a plain `PUT` takes; reads come
|
||||
//! from the assembled blob over the normal `GET`, so clients and history need no
|
||||
//! change. Chunk bytes count toward the quota the way `PUT` counts them.
|
||||
|
||||
use axum::Json;
|
||||
use axum::body::Bytes;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use notescraft_sync::{Write, check_hash, check_syncable, hash_hex};
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::api::routes::{Hashes, MAX_BATCH, Missing, answer, apply_write, header_u64, over_quota};
|
||||
use crate::chunks::assemble::{AssembleError, assemble_blob, chunk_stream_hash};
|
||||
use crate::error::ApiError;
|
||||
use crate::roles::Need;
|
||||
use crate::state::{AppState, authorize, now_ms, store_blob};
|
||||
|
||||
/// `{"missing": [...]}` to a chunked assembly whose chunks are incomplete.
|
||||
fn conflict_missing(missing: Vec<String>) -> Response {
|
||||
(StatusCode::CONFLICT, Json(Missing { missing })).into_response()
|
||||
}
|
||||
|
||||
/// `PUT /v1/vaults/{vault}/chunks/{hash}`: stores one 1..=`chunk_max`-byte chunk
|
||||
/// whose SHA-256 answers the path hash. 201 when stored, 200 when already there,
|
||||
/// 400 on a mismatching body, 413 beyond the chunk cap or the vault quota.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403, 404 as every vault route answers; 400 for a bad hash or an empty body,
|
||||
/// 413 for a too large body or quota, 500 on database or I/O errors.
|
||||
pub(crate) async fn put_chunk(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path((vault, hash)): Path<(String, String)>,
|
||||
body: Bytes,
|
||||
) -> Result<Response, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
check_hash(&hash)?;
|
||||
let chunk_max = state.chunk_max();
|
||||
if body.is_empty() {
|
||||
return Err(ApiError::bad("a chunk must carry at least one byte"));
|
||||
}
|
||||
if body.len() > chunk_max {
|
||||
return Err(ApiError::too_large("one chunk is at most the chunk limit"));
|
||||
}
|
||||
let device = member.device.clone();
|
||||
let stored = tokio::task::spawn_blocking(move || put_chunk_blocking(&state, &vault, &hash, &body, &device)).await;
|
||||
let stored = stored.map_err(|joined| ApiError::internal(&joined))??;
|
||||
Ok((stored, ()).into_response())
|
||||
}
|
||||
|
||||
/// The disk-and-database half of a chunk upload, off the async runtime.
|
||||
fn put_chunk_blocking(state: &AppState, vault: &str, hash: &str, body: &Bytes, device: &str) -> Result<StatusCode, ApiError> {
|
||||
let size = content_size(body.as_ref())?;
|
||||
if hash_hex(body.as_ref()) != hash {
|
||||
return Err(ApiError::bad("the body's hash does not match the path hash"));
|
||||
}
|
||||
let path = state.chunk_path(vault, hash);
|
||||
let had_file = std::fs::exists(&path)?;
|
||||
if !had_file {
|
||||
if over_chunk_quota(state, vault, size)? {
|
||||
let core = state.core();
|
||||
audit::record(&core.store, audit::QUOTA_REJECTED, Some(vault), Some(device), None)?;
|
||||
return Err(ApiError::quota());
|
||||
}
|
||||
// Content addressed: a file that appeared in between is already right.
|
||||
store_blob(&path, body.as_ref())?;
|
||||
}
|
||||
{
|
||||
let core = state.core();
|
||||
core.store.insert_chunk(vault, hash, size, now_ms())?;
|
||||
}
|
||||
Ok(if had_file { StatusCode::OK } else { StatusCode::CREATED })
|
||||
}
|
||||
|
||||
/// `bytes.len()` as a `u64`; a body can never reach the conversion's ceiling.
|
||||
fn content_size(bytes: &[u8]) -> Result<u64, ApiError> {
|
||||
u64::try_from(bytes.len()).map_err(|error| ApiError::internal(&error))
|
||||
}
|
||||
|
||||
/// True when a new chunk of `size` would exceed the vault's quota.
|
||||
fn over_chunk_quota(state: &AppState, vault: &str, size: u64) -> Result<bool, ApiError> {
|
||||
let Some(quota) = state.quota_limit() else { return Ok(false) };
|
||||
let used = state.core().store.used_bytes(vault)?;
|
||||
Ok(used.saturating_add(size) > quota)
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/chunks/check` `{hashes}`: which of the named chunks the
|
||||
/// vault has not stored yet, in request order, so a resuming client sends only those.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403, 404 as above; 400 beyond the batch cap or for a malformed hash.
|
||||
pub(crate) async fn check_chunks(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(request): Json<Hashes>,
|
||||
) -> Result<Json<Missing>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
if request.hashes.len() > MAX_BATCH {
|
||||
return Err(ApiError::bad(format!("at most {MAX_BATCH} hashes")));
|
||||
}
|
||||
request.hashes.iter().try_for_each(|hash| check_hash(hash))?;
|
||||
let missing = tokio::task::spawn_blocking(move || request.hashes.into_iter().filter(|hash| !state.chunk_path(&vault, hash).exists()).collect())
|
||||
.await
|
||||
.map_err(|joined| ApiError::internal(&joined))?;
|
||||
Ok(Json(Missing { missing }))
|
||||
}
|
||||
|
||||
/// The chunked half of `PUT /files/{path}`: verifies every chunk of the `X-Chunks`
|
||||
/// list is stored, streams their bytes once to compute the file's hash and total,
|
||||
/// checks the optional `X-Content-Hash` and the quota, writes the assembled blob
|
||||
/// when it is missing, records the chunk list, then applies the write as a plain
|
||||
/// `PUT` would. The body must be empty.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 for a non-empty body, a list beyond `MAX_FILE_CHUNKS` or a mismatching
|
||||
/// content hash; 409 with the missing hashes; 413 beyond the file cap or the quota;
|
||||
/// 500 on database or I/O errors.
|
||||
pub(crate) async fn put_chunked_file(
|
||||
state: AppState,
|
||||
headers: HeaderMap,
|
||||
vault: String,
|
||||
path: String,
|
||||
chunk_list: String,
|
||||
body: Bytes,
|
||||
) -> Result<Response, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
check_syncable(&path)?;
|
||||
if !body.is_empty() {
|
||||
return Err(ApiError::bad("a chunked upload carries no body; its content is the chunks"));
|
||||
}
|
||||
let hashes = parse_hash_list(&chunk_list, crate::chunks::MAX_FILE_CHUNKS)?;
|
||||
let expected = content_hash(&headers)?;
|
||||
let base_seq = header_u64(&headers, "x-base-seq")?.unwrap_or(0);
|
||||
let mtime_ms = header_u64(&headers, "x-mtime-ms")?.unwrap_or_else(now_ms);
|
||||
|
||||
let missing = tokio::task::spawn_blocking({
|
||||
let state = state.clone();
|
||||
let vault = vault.clone();
|
||||
let hashes = hashes.clone();
|
||||
move || hashes.iter().filter(|hash| !state.chunk_path(&vault, hash).exists()).cloned().collect::<Vec<String>>()
|
||||
})
|
||||
.await
|
||||
.map_err(|joined| ApiError::internal(&joined))?;
|
||||
if !missing.is_empty() {
|
||||
return Ok(conflict_missing(missing));
|
||||
}
|
||||
|
||||
let cap = state.chunk_file_cap();
|
||||
let streamed = tokio::task::spawn_blocking({
|
||||
let state = state.clone();
|
||||
let vault = vault.clone();
|
||||
let hashes = hashes.clone();
|
||||
move || chunk_stream_hash(&hash_paths(&state, &vault, &hashes), cap)
|
||||
})
|
||||
.await;
|
||||
let (hash, size) = match streamed {
|
||||
Ok(Ok(found)) => found,
|
||||
Ok(Err(error)) => return Err(assemble_error(error)),
|
||||
Err(joined) => return Err(ApiError::internal(&joined)),
|
||||
};
|
||||
if expected.as_deref().is_some_and(|expected| expected != hash) {
|
||||
return Err(ApiError::bad("the content hash header does not match the assembled bytes"));
|
||||
}
|
||||
|
||||
if over_quota(&state, &vault, &path, size)? {
|
||||
let core = state.core();
|
||||
audit::record(&core.store, audit::QUOTA_REJECTED, Some(&vault), Some(&member.device), None)?;
|
||||
return Err(ApiError::quota());
|
||||
}
|
||||
let blob = state.blob_path(&vault, &hash);
|
||||
tokio::task::spawn_blocking({
|
||||
let state = state.clone();
|
||||
let vault = vault.clone();
|
||||
let hashes = hashes.clone();
|
||||
move || assemble_blob(&hash_paths(&state, &vault, &hashes), &blob)
|
||||
})
|
||||
.await
|
||||
.map_err(|joined| ApiError::internal(&joined))??;
|
||||
state.core().store.record_file_chunks(&vault, &hash, &hashes)?;
|
||||
Ok(answer(apply_write(&state, &vault, &member.user, Write { path, hash: Some(hash), size, mtime_ms, base_seq })?))
|
||||
}
|
||||
|
||||
/// The chunk paths of a hash list, in list order.
|
||||
fn hash_paths(state: &AppState, vault: &str, hashes: &[String]) -> Vec<std::path::PathBuf> {
|
||||
hashes.iter().map(|hash| state.chunk_path(vault, hash)).collect()
|
||||
}
|
||||
|
||||
/// The `X-Chunks` hash list: one `check_hash`ed value per comma, at most `max`.
|
||||
fn parse_hash_list(raw: &str, max: usize) -> Result<Vec<String>, ApiError> {
|
||||
if raw.is_empty() {
|
||||
return Err(ApiError::bad("the chunk list names no chunks"));
|
||||
}
|
||||
let mut hashes = Vec::new();
|
||||
for part in raw.split(',') {
|
||||
check_hash(part)?;
|
||||
if hashes.len() >= max {
|
||||
return Err(ApiError::bad(format!("at most {max} chunks per file")));
|
||||
}
|
||||
hashes.push(part.to_owned());
|
||||
}
|
||||
Ok(hashes)
|
||||
}
|
||||
|
||||
/// The optional `X-Content-Hash` header, shape-checked when present.
|
||||
fn content_hash(headers: &HeaderMap) -> Result<Option<String>, ApiError> {
|
||||
let Some(value) = headers.get("x-content-hash").and_then(|value| value.to_str().ok()) else { return Ok(None) };
|
||||
check_hash(value)?;
|
||||
Ok(Some(value.to_owned()))
|
||||
}
|
||||
|
||||
/// Maps assembly failures to their answers: a streaming total beyond the cap is 413,
|
||||
/// the rest is 500.
|
||||
fn assemble_error(error: AssembleError) -> ApiError {
|
||||
match error {
|
||||
AssembleError::TooLarge(streamed) => ApiError::too_large(format!("the chunks assemble to at least {streamed} bytes, beyond the server's cap")),
|
||||
AssembleError::Io(error) => error.into(),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
//! Settings from the environment: no config file, no secrets on the command line.
|
||||
|
||||
use std::net::SocketAddr;
|
||||
use std::path::PathBuf;
|
||||
|
||||
/// Shortest admin token accepted.
|
||||
const MIN_ADMIN_TOKEN: usize = 24;
|
||||
/// Default per-vault live-bytes quota.
|
||||
const DEFAULT_QUOTA_MB: u64 = 5120;
|
||||
/// Bytes in one MB of quota.
|
||||
const QUOTA_MIB: u64 = 1024 * 1024;
|
||||
|
||||
/// Everything the service needs to start.
|
||||
#[derive(Debug)]
|
||||
pub struct Config {
|
||||
/// Where `sync.db` and `blobs/` live (`NOTESCRAFT_SYNC_DATA`, default `./data`).
|
||||
pub data_dir: PathBuf,
|
||||
/// Listen address (`NOTESCRAFT_SYNC_BIND`, default `127.0.0.1:8480`).
|
||||
pub bind: SocketAddr,
|
||||
/// Secret for the vault and pairing endpoints (`NOTESCRAFT_SYNC_ADMIN_TOKEN`, 24+ characters).
|
||||
pub admin_token: String,
|
||||
/// Live-bytes limit per vault (`NOTESCRAFT_SYNC_QUOTA_MB`, default 5120, `0` unlimited).
|
||||
pub quota_bytes: Option<u64>,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
/// Reads and validates the environment.
|
||||
///
|
||||
/// # Errors
|
||||
/// A message naming the variable that is missing or wrong.
|
||||
pub fn from_env() -> Result<Self, String> {
|
||||
let admin_token = std::env::var("NOTESCRAFT_SYNC_ADMIN_TOKEN").map_err(|_| "set NOTESCRAFT_SYNC_ADMIN_TOKEN (24+ characters)".to_owned())?;
|
||||
if admin_token.len() < MIN_ADMIN_TOKEN {
|
||||
return Err(format!("NOTESCRAFT_SYNC_ADMIN_TOKEN must be at least {MIN_ADMIN_TOKEN} characters"));
|
||||
}
|
||||
let bind =
|
||||
std::env::var("NOTESCRAFT_SYNC_BIND").unwrap_or_else(|_| "127.0.0.1:8480".into()).parse().map_err(|e| format!("NOTESCRAFT_SYNC_BIND: {e}"))?;
|
||||
let data_dir = std::env::var_os("NOTESCRAFT_SYNC_DATA").map_or_else(|| PathBuf::from("data"), PathBuf::from);
|
||||
let raw = std::env::var("NOTESCRAFT_SYNC_QUOTA_MB").unwrap_or_else(|_| DEFAULT_QUOTA_MB.to_string());
|
||||
let quota_mb: u64 = raw.trim().parse().map_err(|e| format!("NOTESCRAFT_SYNC_QUOTA_MB: {e}"))?;
|
||||
if quota_mb == 0 {
|
||||
return Ok(Self { data_dir, bind, admin_token, quota_bytes: None });
|
||||
}
|
||||
let quota_bytes = Some(quota_mb.checked_mul(QUOTA_MIB).ok_or_else(|| format!("NOTESCRAFT_SYNC_QUOTA_MB: {quota_mb} MB is too large"))?);
|
||||
Ok(Self { data_dir, bind, admin_token, quota_bytes })
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
//! API errors: a status, a short message and an optional delay; internals are logged, never sent.
|
||||
|
||||
use axum::Json;
|
||||
use axum::http::{HeaderName, HeaderValue, StatusCode};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use serde::Serialize;
|
||||
|
||||
/// The header carrying an answer's `Retry-After`, in seconds.
|
||||
const RETRY_AFTER: HeaderName = HeaderName::from_static("retry-after");
|
||||
|
||||
/// An error answered as `{"error": "..."}`, plus an optional `Retry-After`.
|
||||
#[derive(Debug)]
|
||||
pub struct ApiError {
|
||||
/// The HTTP status.
|
||||
pub status: StatusCode,
|
||||
/// The public message.
|
||||
pub message: String,
|
||||
/// Seconds until the client may retry, carried as a `Retry-After` header.
|
||||
pub retry_after: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct Body<'a> {
|
||||
error: &'a str,
|
||||
}
|
||||
|
||||
impl ApiError {
|
||||
fn plain(status: StatusCode, message: impl Into<String>) -> Self {
|
||||
Self { status, message: message.into(), retry_after: None }
|
||||
}
|
||||
|
||||
/// 400.
|
||||
pub fn bad(message: impl Into<String>) -> Self {
|
||||
Self::plain(StatusCode::BAD_REQUEST, message)
|
||||
}
|
||||
|
||||
/// 401.
|
||||
#[must_use]
|
||||
pub fn unauthorized() -> Self {
|
||||
Self::plain(StatusCode::UNAUTHORIZED, "missing or wrong token")
|
||||
}
|
||||
|
||||
/// 403.
|
||||
#[must_use]
|
||||
pub fn forbidden() -> Self {
|
||||
Self::plain(StatusCode::FORBIDDEN, "token is not valid for this vault")
|
||||
}
|
||||
|
||||
/// 404.
|
||||
pub fn missing(message: impl Into<String>) -> Self {
|
||||
Self::plain(StatusCode::NOT_FOUND, message)
|
||||
}
|
||||
|
||||
/// 413, the per-vault live-bytes quota is full.
|
||||
#[must_use]
|
||||
pub fn quota() -> Self {
|
||||
Self::plain(StatusCode::PAYLOAD_TOO_LARGE, "vault storage quota exceeded")
|
||||
}
|
||||
|
||||
/// 410, the content behind the request was already collected.
|
||||
#[must_use]
|
||||
pub fn gone() -> Self {
|
||||
Self::plain(StatusCode::GONE, "that revision's blob was already collected")
|
||||
}
|
||||
|
||||
/// 409, a precondition on server state fails (ownership cap, last owner).
|
||||
#[must_use]
|
||||
pub fn conflict(message: impl Into<String>) -> Self {
|
||||
Self::plain(StatusCode::CONFLICT, message)
|
||||
}
|
||||
|
||||
/// 413, the payload is larger than the server accepts.
|
||||
#[must_use]
|
||||
pub fn too_large(message: impl Into<String>) -> Self {
|
||||
Self::plain(StatusCode::PAYLOAD_TOO_LARGE, message)
|
||||
}
|
||||
|
||||
/// 415, the payload is not of a kind the server can read.
|
||||
#[must_use]
|
||||
pub fn unsupported_media_type(message: impl Into<String>) -> Self {
|
||||
Self::plain(StatusCode::UNSUPPORTED_MEDIA_TYPE, message)
|
||||
}
|
||||
|
||||
/// 405, the path exists but not for this method.
|
||||
#[must_use]
|
||||
pub fn method_not_allowed() -> Self {
|
||||
Self::plain(StatusCode::METHOD_NOT_ALLOWED, "wrong method for this endpoint")
|
||||
}
|
||||
|
||||
/// 503, the server cannot take the request right now.
|
||||
#[must_use]
|
||||
pub fn unavailable(message: impl Into<String>) -> Self {
|
||||
Self::plain(StatusCode::SERVICE_UNAVAILABLE, message)
|
||||
}
|
||||
|
||||
/// 429 with a `Retry-After`; the redemption lockout is active.
|
||||
#[must_use]
|
||||
pub fn locked(retry_after_s: u64) -> Self {
|
||||
Self { status: StatusCode::TOO_MANY_REQUESTS, message: "too many failed codes; try later".into(), retry_after: Some(retry_after_s) }
|
||||
}
|
||||
|
||||
/// 500; the cause goes to stderr only.
|
||||
pub fn internal(cause: &dyn std::fmt::Display) -> Self {
|
||||
eprintln!("sync-server: internal error: {cause}");
|
||||
Self::plain(StatusCode::INTERNAL_SERVER_ERROR, "internal error")
|
||||
}
|
||||
}
|
||||
|
||||
impl IntoResponse for ApiError {
|
||||
fn into_response(self) -> Response {
|
||||
let mut response = (self.status, Json(Body { error: &self.message })).into_response();
|
||||
if let Some(seconds) = self.retry_after
|
||||
&& let Ok(value) = HeaderValue::from_str(&seconds.to_string())
|
||||
{
|
||||
response.headers_mut().insert(RETRY_AFTER, value);
|
||||
}
|
||||
response
|
||||
}
|
||||
}
|
||||
|
||||
impl From<notescraft_sync::Error> for ApiError {
|
||||
fn from(error: notescraft_sync::Error) -> Self {
|
||||
use notescraft_sync::Error;
|
||||
match error {
|
||||
Error::InvalidPath(_) | Error::Excluded(_) | Error::InvalidHash(_) => Self::bad(error.to_string()),
|
||||
Error::NotFound(_) => Self::missing(error.to_string()),
|
||||
Error::SeqOverflow => Self::internal(&error),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<rusqlite::Error> for ApiError {
|
||||
fn from(error: rusqlite::Error) -> Self {
|
||||
Self::internal(&error)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<std::io::Error> for ApiError {
|
||||
fn from(error: std::io::Error) -> Self {
|
||||
Self::internal(&error)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
//! Server-side file history: every applied write is recorded as a revision and stays
|
||||
//! restorable inside the vault's retention window.
|
||||
//!
|
||||
//! [`crate::store::revisions`] holds the rows; [`routes`] adds the endpoints; [`retention`]
|
||||
//! carries the policy the GC applies.
|
||||
|
||||
pub mod retention;
|
||||
pub(crate) mod routes;
|
||||
|
||||
pub use retention::{DEFAULT_RETENTION_DAYS, MAX_RETENTION_DAYS, MIN_KEEP_REVISIONS, MIN_RETENTION_DAYS, valid_retention_days};
|
||||
@@ -0,0 +1,37 @@
|
||||
//! The retention policy: how long a revision stays restorable and what the GC reads.
|
||||
|
||||
/// How long revisions are kept for, in days, when a vault sets nothing else.
|
||||
pub const DEFAULT_RETENTION_DAYS: u64 = 30;
|
||||
/// Every revision among the newest this many of its file is kept, however old it is.
|
||||
pub const MIN_KEEP_REVISIONS: u64 = 20;
|
||||
/// Milliseconds one retention day spans.
|
||||
pub const MS_PER_DAY: u64 = 24 * 60 * 60 * 1000;
|
||||
/// Lowest retention a vault may request, in days.
|
||||
pub const MIN_RETENTION_DAYS: u64 = 1;
|
||||
/// Highest retention a vault may request, in days.
|
||||
pub const MAX_RETENTION_DAYS: u64 = 365;
|
||||
|
||||
/// The `recorded_ms` below which a revision has outlived the retention window.
|
||||
#[must_use]
|
||||
pub fn retention_cutoff_ms(now_ms: u64, retention_days: u64) -> u64 {
|
||||
now_ms.saturating_sub(retention_days.saturating_mul(MS_PER_DAY))
|
||||
}
|
||||
|
||||
/// True for a retention a vault may request: 1..=365 days.
|
||||
#[must_use]
|
||||
pub fn valid_retention_days(days: u64) -> bool {
|
||||
(MIN_RETENTION_DAYS..=MAX_RETENTION_DAYS).contains(&days)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn cutoff_and_bounds() {
|
||||
assert_eq!(retention_cutoff_ms(2 * MS_PER_DAY, DEFAULT_RETENTION_DAYS), 0);
|
||||
assert_eq!(retention_cutoff_ms(1, 0), 1, "a wrong retention never pushes the cutoff up");
|
||||
assert!(valid_retention_days(1));
|
||||
assert!(!valid_retention_days(0) && !valid_retention_days(366));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
//! The history endpoints: list a path's revisions, read one, restore one as a new
|
||||
//! write. A restore never rewrites history: it applies the old content as a new
|
||||
//! revision, as any `PUT` would.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, Query, State};
|
||||
use axum::http::HeaderMap;
|
||||
use axum::http::StatusCode;
|
||||
use axum::http::header::IF_NONE_MATCH;
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use notescraft_sync::{FileRecord, Outcome, Write, check_syncable};
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::api::routes::{answer, apply_write, record_headers};
|
||||
use crate::error::ApiError;
|
||||
use crate::roles::Need;
|
||||
use crate::state::{AppState, authorize, now_ms};
|
||||
use crate::store::RevisionRow;
|
||||
|
||||
/// Most revisions one history listing returns.
|
||||
const MAX_HISTORY: u64 = 200;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub(crate) struct HistoryQuery {
|
||||
/// How many revisions to return; capped at [`MAX_HISTORY`].
|
||||
#[serde(default)]
|
||||
pub limit: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub(crate) struct RestoreRequest {
|
||||
/// The revision whose content the restore brings back.
|
||||
pub seq: u64,
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/files-history/{path}`: the path's revisions, newest first.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403, 404 like every vault route, 400 for a wrong path, 500 on database errors.
|
||||
pub(crate) async fn list(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path((vault, path)): Path<(String, String)>,
|
||||
Query(query): Query<HistoryQuery>,
|
||||
) -> Result<Json<Vec<RevisionRow>>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
check_syncable(&path)?;
|
||||
let limit = query.limit.unwrap_or(MAX_HISTORY).clamp(1, MAX_HISTORY);
|
||||
Ok(Json(state.core().store.revisions(&vault, &path, limit)?))
|
||||
}
|
||||
|
||||
/// The bytes of one revision, for `GET /v1/vaults/{vault}/files/{path}?seq=N`: the
|
||||
/// body answers 404 when the revision or its blob is gone, with that revision's
|
||||
/// revision metadata and its hash as `ETag`.
|
||||
///
|
||||
/// # Errors
|
||||
/// As above; 404 also for a collected blob.
|
||||
pub(crate) async fn read_revision(state: &AppState, vault: &str, path: &str, seq: u64, headers: &HeaderMap) -> Result<Response, ApiError> {
|
||||
let revision = state.core().store.find_revision(vault, path, seq)?.ok_or_else(|| ApiError::missing("no such revision"))?;
|
||||
let record = FileRecord {
|
||||
path: path.to_owned(),
|
||||
hash: revision.hash.clone(),
|
||||
size: revision.size,
|
||||
mtime_ms: revision.mtime_ms,
|
||||
seq: revision.seq,
|
||||
deleted: revision.deleted,
|
||||
};
|
||||
let reply = record_headers(&record);
|
||||
if headers.get(IF_NONE_MATCH).and_then(|value| value.to_str().ok()) == Some(record.hash.as_str()) {
|
||||
return Ok((StatusCode::NOT_MODIFIED, reply).into_response());
|
||||
}
|
||||
let blob = state.blob_path(vault, &record.hash);
|
||||
let bytes = tokio::task::spawn_blocking(move || std::fs::read(blob)).await.map_err(|joined| ApiError::internal(&joined))?.map_err(|error| {
|
||||
if error.kind() == std::io::ErrorKind::NotFound { ApiError::missing("the blob of that revision is gone") } else { ApiError::from(error) }
|
||||
})?;
|
||||
Ok((StatusCode::OK, reply, bytes).into_response())
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/files-restore/{path} {seq}`: writes the revision's bytes
|
||||
/// as a new revision through the ledger (201 like a `PUT`) so history keeps both.
|
||||
///
|
||||
/// # Errors
|
||||
/// 404 for an unknown revision or a non-member, 403 for a viewer, 410 when the blob
|
||||
/// was already collected, 500 on database or I/O errors.
|
||||
pub(crate) async fn restore(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path((vault, path)): Path<(String, String)>,
|
||||
Json(request): Json<RestoreRequest>,
|
||||
) -> Result<Response, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
check_syncable(&path)?;
|
||||
let revision = state.core().store.find_revision(&vault, &path, request.seq)?.ok_or_else(|| ApiError::missing("no such revision"))?;
|
||||
let blob = state.blob_path(&vault, &revision.hash);
|
||||
let present = tokio::task::spawn_blocking(move || std::fs::exists(blob)).await.map_err(|joined| ApiError::internal(&joined))??;
|
||||
if !present {
|
||||
return Err(ApiError::gone());
|
||||
}
|
||||
let base_seq = state.core().ledger(&vault)?.get(&path).map_or(0, |record| record.seq);
|
||||
let outcome = apply_write(&state, &vault, &member.user, Write { path, hash: Some(revision.hash), size: revision.size, mtime_ms: now_ms(), base_seq })?;
|
||||
if let Outcome::Applied(record) = &outcome {
|
||||
let detail = format!("{}@{} restored as seq {}", record.path, request.seq, record.seq);
|
||||
audit::record(&state.core().store, audit::FILE_RESTORED, Some(&vault), Some(&member.device), Some(&detail))?;
|
||||
}
|
||||
Ok(answer(outcome))
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
//! `NotesCraft` sync service: accounts and vaults, pairing, a per-vault change log and
|
||||
//! file blobs over HTTP.
|
||||
//!
|
||||
//! Decisions about who wins a write live in `notescraft-sync`'s `Ledger`; this crate adds
|
||||
//! accounts (a user owns devices and many vaults, roles enforced per vault), authentication,
|
||||
//! SQLite persistence, content-addressed blob files and the routes. It also keeps the
|
||||
//! device lifecycle (pairing lockout, rotation, revocation), single-use invites, an audit
|
||||
//! log, per-vault quotas, soft delete with purge and blob garbage collection.
|
||||
//!
|
||||
//! Environment:
|
||||
//! - `NOTESCRAFT_SYNC_ADMIN_TOKEN` (required, 24+ characters)
|
||||
//! - `NOTESCRAFT_SYNC_DATA` (default `./data`)
|
||||
//! - `NOTESCRAFT_SYNC_BIND` (default `127.0.0.1:8480`)
|
||||
//! - `NOTESCRAFT_SYNC_QUOTA_MB` (default `5120`, `0` means unlimited)
|
||||
//!
|
||||
//! The pairing and invite lockout shares one failure counter in memory and clears on restart.
|
||||
|
||||
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
|
||||
|
||||
mod activity;
|
||||
mod api;
|
||||
mod cache;
|
||||
mod chunks;
|
||||
mod config;
|
||||
pub mod error;
|
||||
mod history;
|
||||
mod limits;
|
||||
pub mod live;
|
||||
mod middleware;
|
||||
pub mod publish;
|
||||
mod roles;
|
||||
mod state;
|
||||
mod store;
|
||||
|
||||
pub use api::admin::{GcReport, PurgeReport, TOMBSTONE_GRACE_MS};
|
||||
pub use api::devices::{DeviceListing, Paired};
|
||||
pub use api::invites::INVITE_TTL_MS;
|
||||
pub use api::router;
|
||||
pub use chunks::{MAX_CHUNK_BYTES, MAX_CHUNKED_BYTES, MAX_FILE_CHUNKS};
|
||||
pub use config::Config;
|
||||
pub use history::{DEFAULT_RETENTION_DAYS, MAX_RETENTION_DAYS, MIN_KEEP_REVISIONS, MIN_RETENTION_DAYS};
|
||||
pub use limits::{Lockout, PAIRING_MAX_FAILURES, PAIRING_WINDOW_MS};
|
||||
pub use live::{FlushReport, Timings, flush};
|
||||
pub use state::{AppState, now_ms, random_hex, valid_vault_id};
|
||||
pub use store::{AuditRow, Store};
|
||||
@@ -0,0 +1,65 @@
|
||||
//! Pairing lockout: too many failed code redemptions block new redemptions for a while.
|
||||
//!
|
||||
//! Failure timestamps live in memory only, shared by all callers: a restart clears the
|
||||
//! window and a multi-process deployment would need the database instead. Attempts are
|
||||
//! not counted per IP because the real client IP is not visible behind a tunnel.
|
||||
|
||||
use std::sync::{Mutex, PoisonError};
|
||||
|
||||
/// Failed redemptions allowed in the window before redemptions lock.
|
||||
pub const PAIRING_MAX_FAILURES: usize = 5;
|
||||
/// Sliding window over which failed redemptions are counted.
|
||||
pub const PAIRING_WINDOW_MS: u64 = 10 * 60 * 1000;
|
||||
|
||||
/// The shared pairing lockout state.
|
||||
#[derive(Default)]
|
||||
pub struct Lockout {
|
||||
/// Clock-free design: raw timestamps of recent failures, older ones dropped at each look.
|
||||
failures: Mutex<Vec<u64>>,
|
||||
}
|
||||
|
||||
impl Lockout {
|
||||
/// Seconds until redemption may resume, or `None` while the window holds fewer than
|
||||
/// `max_failures` failures. Also drops timestamps older than the window.
|
||||
pub fn retry_after(&self, now: u64, window_ms: u64, max_failures: usize) -> Option<u64> {
|
||||
let mut failures = self.failures.lock().unwrap_or_else(PoisonError::into_inner);
|
||||
failures.retain(|ms| now.saturating_sub(*ms) < window_ms);
|
||||
if failures.len() < max_failures {
|
||||
return None;
|
||||
}
|
||||
let oldest = failures.iter().min().copied()?;
|
||||
Some(window_ms.saturating_sub(now.saturating_sub(oldest)).div_ceil(1000))
|
||||
}
|
||||
|
||||
/// Records one failed redemption; only called while the lockout is not active.
|
||||
pub fn record_failure(&self, now: u64) {
|
||||
self.failures.lock().unwrap_or_else(PoisonError::into_inner).push(now);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn locks_after_max_failures_and_clears_beyond_the_window() {
|
||||
let lockout = Lockout::default();
|
||||
let window = 10_000;
|
||||
for t in [9_000_u64, 9_500, 9_900, 10_500, 11_000] {
|
||||
lockout.record_failure(t);
|
||||
}
|
||||
assert_eq!(lockout.retry_after(11_000, window, 5), Some(8)); // oldest failure is 9_000
|
||||
assert_eq!(lockout.retry_after(11_000, window, 6), None);
|
||||
// Everything older than the window, relative to `now`, is gone.
|
||||
assert_eq!(lockout.retry_after(24_000, window, 5), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retry_after_counts_from_the_oldest_failure() {
|
||||
let lockout = Lockout::default();
|
||||
for t in [0_u64, 1, 2, 3, 4] {
|
||||
lockout.record_failure(t);
|
||||
}
|
||||
assert_eq!(lockout.retry_after(9, 10_000, 5), Some(10));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
//! The events a room sends to its peers: their JSON shape and the Server-Sent
|
||||
//! Events stream one peer reads.
|
||||
//!
|
||||
//! The room hands every peer a bounded channel; the stream in this module is the
|
||||
//! reader of that channel. It ends when the peer leaves (the room drops the
|
||||
//! sender) or when the peer stays idle for too long.
|
||||
|
||||
use std::convert::Infallible;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::task::{Context, Poll};
|
||||
use std::time::Duration;
|
||||
|
||||
use axum::response::sse::Event;
|
||||
use notescraft_collab::{Peer, TextOp};
|
||||
use serde::Serialize;
|
||||
use tokio::sync::mpsc::Receiver;
|
||||
use tokio::time::Sleep;
|
||||
|
||||
/// One event of a room's event stream.
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum LiveEvent {
|
||||
/// An operation of another peer, already transformed by the room.
|
||||
Op {
|
||||
/// The revision the operation produced.
|
||||
rev: u64,
|
||||
/// The client id of the author.
|
||||
client: String,
|
||||
/// The operation, as the room applied it.
|
||||
op: TextOp,
|
||||
},
|
||||
/// The acknowledgement of the caller's own operation.
|
||||
Ack {
|
||||
/// The revision the operation produced.
|
||||
rev: u64,
|
||||
},
|
||||
/// The peers in the room.
|
||||
Presence {
|
||||
/// The peers, ordered by client id.
|
||||
peers: Vec<Peer>,
|
||||
},
|
||||
/// A peer left the room.
|
||||
Leave {
|
||||
/// The client id that left.
|
||||
client: String,
|
||||
},
|
||||
}
|
||||
|
||||
/// The payload of an `op` event.
|
||||
#[derive(Serialize)]
|
||||
struct OpPayload<'a> {
|
||||
rev: u64,
|
||||
client: &'a str,
|
||||
op: &'a TextOp,
|
||||
}
|
||||
|
||||
/// The payload of an `ack` event.
|
||||
#[derive(Serialize)]
|
||||
struct AckPayload {
|
||||
rev: u64,
|
||||
}
|
||||
|
||||
/// The payload of a `presence` event.
|
||||
#[derive(Serialize)]
|
||||
struct PresencePayload<'a> {
|
||||
peers: &'a [Peer],
|
||||
}
|
||||
|
||||
/// The payload of a `leave` event.
|
||||
#[derive(Serialize)]
|
||||
struct LeavePayload<'a> {
|
||||
client: &'a str,
|
||||
}
|
||||
|
||||
impl LiveEvent {
|
||||
/// The name of the Server-Sent Events event.
|
||||
#[must_use]
|
||||
pub fn name(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Op { .. } => "op",
|
||||
Self::Ack { .. } => "ack",
|
||||
Self::Presence { .. } => "presence",
|
||||
Self::Leave { .. } => "leave",
|
||||
}
|
||||
}
|
||||
|
||||
/// The JSON payload of the event.
|
||||
#[must_use]
|
||||
pub fn payload(&self) -> String {
|
||||
match self {
|
||||
Self::Op { rev, client, op } => json(&OpPayload { rev: *rev, client, op }),
|
||||
Self::Ack { rev } => json(&AckPayload { rev: *rev }),
|
||||
Self::Presence { peers } => json(&PresencePayload { peers }),
|
||||
Self::Leave { client } => json(&LeavePayload { client }),
|
||||
}
|
||||
}
|
||||
|
||||
/// The Server-Sent Events event of this payload.
|
||||
pub fn to_sse(&self) -> Event {
|
||||
Event::default().event(self.name()).data(self.payload())
|
||||
}
|
||||
}
|
||||
|
||||
/// Serialises a payload. The payloads hold plain data, so this cannot fail; a
|
||||
/// broken serialiser must not take the server down.
|
||||
fn json<T: Serialize>(value: &T) -> String {
|
||||
serde_json::to_string(value).unwrap_or_else(|_| String::from("{}"))
|
||||
}
|
||||
|
||||
/// The event stream of one peer: the events the room sends it, until the peer
|
||||
/// leaves or stays idle for too long.
|
||||
pub struct EventStream {
|
||||
/// The room's channel for this peer.
|
||||
receiver: Receiver<LiveEvent>,
|
||||
/// How long the stream may stay without an event.
|
||||
idle: Duration,
|
||||
/// The timer that ends the stream when it fires.
|
||||
timer: Pin<Box<Sleep>>,
|
||||
/// Set once the stream has ended.
|
||||
ended: bool,
|
||||
}
|
||||
|
||||
impl EventStream {
|
||||
/// A stream over `receiver` that ends after `idle` without an event.
|
||||
#[must_use]
|
||||
pub fn new(receiver: Receiver<LiveEvent>, idle: Duration) -> Self {
|
||||
Self { receiver, idle, timer: Box::pin(tokio::time::sleep(idle)), ended: false }
|
||||
}
|
||||
}
|
||||
|
||||
impl futures_core::Stream for EventStream {
|
||||
type Item = Result<Event, Infallible>;
|
||||
|
||||
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
|
||||
let this = self.get_mut();
|
||||
if this.ended {
|
||||
return Poll::Ready(None);
|
||||
}
|
||||
match this.receiver.poll_recv(cx) {
|
||||
Poll::Ready(Some(event)) => {
|
||||
this.timer.as_mut().reset(tokio::time::Instant::now() + this.idle);
|
||||
Poll::Ready(Some(Ok(event.to_sse())))
|
||||
}
|
||||
Poll::Ready(None) => {
|
||||
this.ended = true;
|
||||
Poll::Ready(None)
|
||||
}
|
||||
Poll::Pending => match this.timer.as_mut().poll(cx) {
|
||||
Poll::Ready(()) => {
|
||||
this.ended = true;
|
||||
Poll::Ready(None)
|
||||
}
|
||||
Poll::Pending => Poll::Pending,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
//! Live co-editing on the sync server: rooms in memory, operations over HTTP,
|
||||
//! events over Server-Sent Events and the flush of a room's text into the vault
|
||||
//! ledger.
|
||||
//!
|
||||
//! The server keeps at most [`Timings::max_rooms`] rooms in memory. A room holds
|
||||
//! the document (`notescraft_collab::Room`), its peers and their presence; a peer
|
||||
//! is a bounded channel the room sends its events on. The text of a room reaches
|
||||
//! the vault through [`flush`], the same path a `PUT` takes: the blob store and
|
||||
//! the ledger, with the room's last known sequence as the base.
|
||||
//!
|
||||
//! See `specs/collaboration/spec.md` for the design and the acceptance criteria.
|
||||
|
||||
mod events;
|
||||
mod registry;
|
||||
mod room;
|
||||
mod routes;
|
||||
|
||||
use notescraft_collab::{Peer, Selection};
|
||||
use notescraft_sync::{Outcome, Write, check_syncable, conflict_path, hash_hex};
|
||||
use serde::Serialize;
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::api::routes::{apply_write, over_quota};
|
||||
use crate::error::ApiError;
|
||||
use crate::live::room::{Edit, RoomKey};
|
||||
use crate::state::{AppState, store_blob};
|
||||
|
||||
pub use events::LiveEvent;
|
||||
pub(crate) use registry::Registry;
|
||||
pub(crate) use routes::dispatch;
|
||||
|
||||
/// A peer joined a room.
|
||||
pub const LIVE_JOINED: &str = "live_joined";
|
||||
/// A peer left a room.
|
||||
pub const LIVE_LEFT: &str = "live_left";
|
||||
/// A room's text was written to the vault.
|
||||
pub const LIVE_FLUSHED: &str = "live_flushed";
|
||||
/// A room's text went to a conflict copy.
|
||||
pub const LIVE_CONFLICT: &str = "live_conflict";
|
||||
|
||||
/// The device label a conflict copy carries.
|
||||
const CONFLICT_DEVICE: &str = "live";
|
||||
/// The largest document a room may hold, in bytes.
|
||||
pub use notescraft_collab::MAX_DOCUMENT_BYTES;
|
||||
/// The largest body a live endpoint accepts, in bytes.
|
||||
pub const MAX_BODY_BYTES: usize = 1024 * 1024;
|
||||
|
||||
/// The windows the live endpoints work with.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct Timings {
|
||||
/// How often the event stream sends a keepalive comment, in millis.
|
||||
pub keepalive_ms: u64,
|
||||
/// How long a stream may go without an event before it ends, in millis.
|
||||
pub idle_ms: u64,
|
||||
/// How long a peer may go without a heartbeat before it is pruned, in millis.
|
||||
pub presence_ttl_ms: u64,
|
||||
/// How long a room waits after its last operation before it is flushed, in millis.
|
||||
pub flush_debounce_ms: u64,
|
||||
/// How often the background sweep looks for rooms to flush, in millis.
|
||||
pub sweep_ms: u64,
|
||||
/// How many rooms the server holds at once.
|
||||
pub max_rooms: usize,
|
||||
/// How many peers one room holds at once.
|
||||
pub max_peers: usize,
|
||||
}
|
||||
|
||||
impl Default for Timings {
|
||||
fn default() -> Self {
|
||||
Self { keepalive_ms: 15_000, idle_ms: 60_000, presence_ttl_ms: 30_000, flush_debounce_ms: 2_000, sweep_ms: 250, max_rooms: 256, max_peers: 64 }
|
||||
}
|
||||
}
|
||||
|
||||
/// The answer to a join: the document, its revision and the peers in the room.
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct JoinReply {
|
||||
/// The text of the document.
|
||||
pub text: String,
|
||||
/// The revision of the document.
|
||||
pub rev: u64,
|
||||
/// The peers in the room.
|
||||
pub peers: Vec<Peer>,
|
||||
}
|
||||
|
||||
/// What a flush did.
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct FlushReport {
|
||||
/// Whether the room's text was written.
|
||||
pub flushed: bool,
|
||||
/// The path it was written to; a conflict copy when there was a conflict.
|
||||
pub path: String,
|
||||
/// Whether the ledger held a newer version and a conflict copy was written.
|
||||
pub conflict: bool,
|
||||
}
|
||||
|
||||
/// Joins a client to the room of a note, creating the room when it is new. The
|
||||
/// room starts with the text the vault holds for the note.
|
||||
pub(crate) fn join(state: &AppState, vault: &str, path: &str, client: &str, name: &str, device: &str, now_ms: u64) -> Result<JoinReply, ApiError> {
|
||||
check_syncable(path)?;
|
||||
let timings = state.live().timings();
|
||||
let room = if let Some(room) = state.live().room(vault, path) {
|
||||
room
|
||||
} else {
|
||||
let (text, base_seq) = current_text(state, vault, path)?.unwrap_or_default();
|
||||
state.live().create_room(RoomKey { vault: vault.to_owned(), path: path.to_owned() }, text, base_seq, now_ms)?
|
||||
};
|
||||
let (text, rev, peers) = room.join(client, name, now_ms, timings)?;
|
||||
audit::record(&state.core().store, LIVE_JOINED, Some(vault), Some(device), Some(path))?;
|
||||
Ok(JoinReply { text, rev, peers })
|
||||
}
|
||||
|
||||
/// Applies an operation of a peer and answers the revision it produced.
|
||||
pub(crate) fn submit(state: &AppState, vault: &str, path: &str, edit: Edit<'_>) -> Result<u64, ApiError> {
|
||||
let Some(room) = state.live().room(vault, path) else {
|
||||
return Err(ApiError::missing("join the room first"));
|
||||
};
|
||||
room.submit(edit, state.live().timings())
|
||||
}
|
||||
|
||||
/// Records the heartbeat of a peer and answers who is in the room.
|
||||
pub(crate) fn presence(state: &AppState, vault: &str, path: &str, client: &str, selection: Option<Selection>, now_ms: u64) -> Result<Vec<Peer>, ApiError> {
|
||||
let Some(room) = state.live().room(vault, path) else {
|
||||
return Err(ApiError::missing("no such room"));
|
||||
};
|
||||
room.presence(client, selection, now_ms, state.live().timings())
|
||||
}
|
||||
|
||||
/// Removes a peer from a room; the room is written and forgotten when it was the
|
||||
/// last one.
|
||||
pub(crate) fn leave(state: &AppState, vault: &str, path: &str, client: &str, now_ms: u64) -> Result<(), ApiError> {
|
||||
let Some(room) = state.live().room(vault, path) else {
|
||||
return Ok(());
|
||||
};
|
||||
let empty = room.leave(client);
|
||||
audit::record(&state.core().store, LIVE_LEFT, Some(vault), None, Some(path))?;
|
||||
if empty && flush(state, vault, path, now_ms).is_ok() {
|
||||
state.live().remove_room(vault, path);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Writes the text of a room through the same path a `PUT` takes: the blob store
|
||||
/// and the vault ledger, with the room's last known sequence as the base. When
|
||||
/// the ledger answers Conflict (someone wrote the file outside the room), the text
|
||||
/// goes to a conflict copy and the file itself is left alone.
|
||||
///
|
||||
/// # Errors
|
||||
/// 413 when the vault is over its quota, 500 on database or blob errors.
|
||||
pub fn flush(state: &AppState, vault: &str, path: &str, now_ms: u64) -> Result<FlushReport, ApiError> {
|
||||
let Some(room) = state.live().room(vault, path) else {
|
||||
return Ok(FlushReport { flushed: false, path: path.to_owned(), conflict: false });
|
||||
};
|
||||
let Some((text, base_seq, editor)) = room.flush_data() else {
|
||||
return Ok(FlushReport { flushed: false, path: path.to_owned(), conflict: false });
|
||||
};
|
||||
let bytes = text.as_bytes();
|
||||
let size = u64::try_from(bytes.len()).map_err(|e| ApiError::internal(&e))?;
|
||||
if over_quota(state, vault, path, size)? {
|
||||
audit::record(&state.core().store, audit::QUOTA_REJECTED, Some(vault), Some(&editor), None)?;
|
||||
return Err(ApiError::quota());
|
||||
}
|
||||
let hash = hash_hex(bytes);
|
||||
let blob = state.blob_path(vault, &hash);
|
||||
store_blob(&blob, bytes)?;
|
||||
let write = Write { path: path.to_owned(), hash: Some(hash.clone()), size, mtime_ms: now_ms, base_seq };
|
||||
match apply_write(state, vault, &editor, write)? {
|
||||
Outcome::Applied(record) => {
|
||||
room.mark_clean(Some(record.seq));
|
||||
audit::record(&state.core().store, LIVE_FLUSHED, Some(vault), Some(&editor), Some(path))?;
|
||||
Ok(FlushReport { flushed: true, path: path.to_owned(), conflict: false })
|
||||
}
|
||||
Outcome::Unchanged(record) => {
|
||||
room.mark_clean(Some(record.seq));
|
||||
Ok(FlushReport { flushed: true, path: path.to_owned(), conflict: false })
|
||||
}
|
||||
Outcome::Conflict(_) => {
|
||||
let copy = conflict_path(path, CONFLICT_DEVICE, &date_stamp(now_ms));
|
||||
let base = {
|
||||
let mut core = state.core();
|
||||
core.ledger(vault)?.get(©).map_or(0, |record| record.seq)
|
||||
};
|
||||
let copy_write = Write { path: copy.clone(), hash: Some(hash), size, mtime_ms: now_ms, base_seq: base };
|
||||
apply_write(state, vault, &editor, copy_write)?;
|
||||
room.mark_clean(None);
|
||||
audit::record(&state.core().store, LIVE_CONFLICT, Some(vault), Some(&editor), Some(©))?;
|
||||
Ok(FlushReport { flushed: true, path: copy, conflict: true })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Flushes the rooms whose edits have settled and forgets the rooms without peers.
|
||||
pub(crate) fn sweep(state: &AppState, now_ms: u64) {
|
||||
let timings = state.live().timings();
|
||||
for room in state.live().rooms() {
|
||||
let key = room.key().clone();
|
||||
if room.is_empty() {
|
||||
if flush(state, &key.vault, &key.path, now_ms).is_ok() {
|
||||
state.live().remove_room(&key.vault, &key.path);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if room.is_due(now_ms, timings.flush_debounce_ms) {
|
||||
let _ = flush(state, &key.vault, &key.path, now_ms);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The text the vault holds for a note, with the sequence it has.
|
||||
fn current_text(state: &AppState, vault: &str, path: &str) -> Result<Option<(String, u64)>, ApiError> {
|
||||
let record = {
|
||||
let mut core = state.core();
|
||||
core.ledger(vault)?.get(path).filter(|record| !record.deleted).cloned()
|
||||
};
|
||||
let Some(record) = record else {
|
||||
return Ok(None);
|
||||
};
|
||||
if record.size > u64::try_from(MAX_DOCUMENT_BYTES).unwrap_or(u64::MAX) {
|
||||
return Err(ApiError::too_large("the file is too large for live editing"));
|
||||
}
|
||||
let blob = state.blob_path(vault, &record.hash);
|
||||
let bytes = std::fs::read(&blob)?;
|
||||
let text = String::from_utf8(bytes).map_err(|_| ApiError::unsupported_media_type("the file is not UTF-8"))?;
|
||||
Ok(Some((text, record.seq)))
|
||||
}
|
||||
|
||||
/// The UTC date of a millisecond timestamp, as `YYYY-MM-DD`.
|
||||
fn date_stamp(now_ms: u64) -> String {
|
||||
let days = i64::try_from(now_ms / 86_400_000).unwrap_or(i64::MAX);
|
||||
let shifted = days.saturating_add(719_468);
|
||||
let era = shifted.div_euclid(146_097);
|
||||
let day_of_era = shifted.rem_euclid(146_097);
|
||||
let year_of_era = (day_of_era - day_of_era / 1460 + day_of_era / 36_524 - day_of_era / 146_096) / 365;
|
||||
let year = year_of_era.saturating_add(era.saturating_mul(400));
|
||||
let day_of_year = day_of_era - (365 * year_of_era + year_of_era / 4 - year_of_era / 100);
|
||||
let month_part = (5 * day_of_year + 2) / 153;
|
||||
let day = day_of_year - (153 * month_part + 2) / 5 + 1;
|
||||
let month = if month_part < 10 { month_part + 3 } else { month_part - 9 };
|
||||
let year = if month <= 2 { year.saturating_add(1) } else { year };
|
||||
format!("{year:04}-{month:02}-{day:02}")
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
//! The rooms of one server: the map of live documents the server holds.
|
||||
//!
|
||||
//! The registry keeps at most as many rooms as its windows allow and hands them
|
||||
//! out. A room itself lives in the sibling module; the text of a room reaches the
|
||||
//! vault through the flush.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
|
||||
use std::time::Duration;
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::live::Timings;
|
||||
use crate::live::room::{Room, RoomKey};
|
||||
|
||||
/// The shortest interval the background sweep runs at.
|
||||
pub const MIN_SWEEP_MS: u64 = 50;
|
||||
|
||||
/// The rooms of one server.
|
||||
pub struct Registry {
|
||||
inner: Mutex<Inner>,
|
||||
sweeper: AtomicBool,
|
||||
}
|
||||
|
||||
struct Inner {
|
||||
rooms: HashMap<RoomKey, Arc<Room>>,
|
||||
timings: Timings,
|
||||
}
|
||||
|
||||
impl Registry {
|
||||
/// A registry with the given windows.
|
||||
#[must_use]
|
||||
pub fn new(timings: Timings) -> Self {
|
||||
Self { inner: Mutex::new(Inner { rooms: HashMap::new(), timings }), sweeper: AtomicBool::new(false) }
|
||||
}
|
||||
|
||||
fn lock(&self) -> MutexGuard<'_, Inner> {
|
||||
self.inner.lock().unwrap_or_else(PoisonError::into_inner)
|
||||
}
|
||||
|
||||
/// The windows in force.
|
||||
#[must_use]
|
||||
pub fn timings(&self) -> Timings {
|
||||
self.lock().timings
|
||||
}
|
||||
|
||||
/// Replaces the windows in force.
|
||||
pub fn set_timings(&self, timings: Timings) {
|
||||
self.lock().timings = timings;
|
||||
}
|
||||
|
||||
/// The room of a note, when the server holds one.
|
||||
#[must_use]
|
||||
pub fn room(&self, vault: &str, path: &str) -> Option<Arc<Room>> {
|
||||
let key = RoomKey { vault: vault.to_owned(), path: path.to_owned() };
|
||||
self.lock().rooms.get(&key).cloned()
|
||||
}
|
||||
|
||||
/// Every room, for the background sweep.
|
||||
#[must_use]
|
||||
pub fn rooms(&self) -> Vec<Arc<Room>> {
|
||||
self.lock().rooms.values().cloned().collect()
|
||||
}
|
||||
|
||||
/// Drops the peers of a room that have not been heard from; answers their ids.
|
||||
pub(crate) fn prune_presence(&self, vault: &str, path: &str, now_ms: u64) -> Vec<String> {
|
||||
let Some(room) = self.room(vault, path) else {
|
||||
return Vec::new();
|
||||
};
|
||||
room.prune(now_ms, self.timings().presence_ttl_ms)
|
||||
}
|
||||
|
||||
/// Forgets a room.
|
||||
pub fn remove_room(&self, vault: &str, path: &str) {
|
||||
let key = RoomKey { vault: vault.to_owned(), path: path.to_owned() };
|
||||
self.lock().rooms.remove(&key);
|
||||
}
|
||||
|
||||
/// Creates a room, unless the server already holds as many as it may.
|
||||
pub(crate) fn create_room(&self, key: RoomKey, text: String, base_seq: u64, now_ms: u64) -> Result<Arc<Room>, ApiError> {
|
||||
let mut inner = self.lock();
|
||||
if let Some(room) = inner.rooms.get(&key) {
|
||||
return Ok(room.clone());
|
||||
}
|
||||
if inner.rooms.len() >= inner.timings.max_rooms {
|
||||
return Err(ApiError::unavailable("too many live rooms"));
|
||||
}
|
||||
let room = Arc::new(Room::new(key.clone(), text, base_seq, now_ms));
|
||||
inner.rooms.insert(key, room.clone());
|
||||
Ok(room)
|
||||
}
|
||||
|
||||
/// Starts the background sweep that flushes rooms whose edits have settled,
|
||||
/// once per server and only inside a runtime.
|
||||
pub fn ensure_sweeper(&self, state: crate::state::AppState) {
|
||||
if self.sweeper.swap(true, Ordering::SeqCst) {
|
||||
return;
|
||||
}
|
||||
let Ok(handle) = tokio::runtime::Handle::try_current() else {
|
||||
self.sweeper.store(false, Ordering::SeqCst);
|
||||
return;
|
||||
};
|
||||
handle.spawn(async move {
|
||||
loop {
|
||||
let timings = state.live().timings();
|
||||
tokio::time::sleep(Duration::from_millis(timings.sweep_ms.max(MIN_SWEEP_MS))).await;
|
||||
let worker = state.clone();
|
||||
let _ = tokio::task::spawn_blocking(move || crate::live::sweep(&worker, crate::state::now_ms())).await;
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
//! One live document in memory: its text, its peers and their presence.
|
||||
//!
|
||||
//! The document is a `notescraft_collab::Room`, the presence is the collab crate's
|
||||
//! `Presence`, and every peer has a bounded channel the room sends its events on.
|
||||
//! The room keeps the ledger sequence its text is based on, so the flush can tell
|
||||
//! whether somebody wrote the file meanwhile.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Mutex, MutexGuard, PoisonError};
|
||||
|
||||
use notescraft_collab::{Error as CollabError, Peer, Presence, Room as Document, Selection, TextOp};
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::live::Timings;
|
||||
use crate::live::events::LiveEvent;
|
||||
|
||||
/// How many events a peer's channel holds before the peer is dropped.
|
||||
pub const PEER_CHANNEL: usize = 64;
|
||||
|
||||
/// The key of a room: a vault and a path in it.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
pub struct RoomKey {
|
||||
/// The vault the note lives in.
|
||||
pub vault: String,
|
||||
/// The path of the note in the vault.
|
||||
pub path: String,
|
||||
}
|
||||
|
||||
/// One peer of a room: the channel the room sends its events on.
|
||||
struct PeerSlot {
|
||||
/// The sender half; the room keeps it.
|
||||
tx: mpsc::Sender<LiveEvent>,
|
||||
/// The receiver half; the event stream takes it.
|
||||
rx: Option<mpsc::Receiver<LiveEvent>>,
|
||||
}
|
||||
|
||||
/// The mutable state of one room.
|
||||
struct RoomState {
|
||||
/// The document: text, revision and the bounded history.
|
||||
document: Document,
|
||||
/// Who is in the room.
|
||||
presence: Presence,
|
||||
/// The peers' channels, by client id.
|
||||
peers: HashMap<String, PeerSlot>,
|
||||
/// The ledger sequence the room's text is based on.
|
||||
base_seq: u64,
|
||||
/// The user id of the last editor.
|
||||
last_editor: Option<String>,
|
||||
/// Whether the text changed since the last flush.
|
||||
dirty: bool,
|
||||
/// When the last operation arrived.
|
||||
last_op_ms: u64,
|
||||
}
|
||||
|
||||
impl RoomState {
|
||||
/// The peers in the room, ordered by client id.
|
||||
fn peers(&self) -> Vec<Peer> {
|
||||
self.presence.peers().into_iter().cloned().collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// One edit a peer sends: an operation and who sends it.
|
||||
pub(crate) struct Edit<'a> {
|
||||
/// The client id of the author.
|
||||
pub client: &'a str,
|
||||
/// The revision the operation was built on.
|
||||
pub base_rev: u64,
|
||||
/// The operation.
|
||||
pub op: TextOp,
|
||||
/// Where the author's caret is.
|
||||
pub selection: Option<Selection>,
|
||||
/// The user id of the author; it signs the flush.
|
||||
pub user: &'a str,
|
||||
/// When the edit arrived.
|
||||
pub now_ms: u64,
|
||||
}
|
||||
|
||||
/// One live document in memory.
|
||||
pub struct Room {
|
||||
key: RoomKey,
|
||||
state: Mutex<RoomState>,
|
||||
}
|
||||
|
||||
impl Room {
|
||||
/// A room over `text` at revision 0, based on the ledger sequence `base_seq`.
|
||||
pub(crate) fn new(key: RoomKey, text: String, base_seq: u64, now_ms: u64) -> Self {
|
||||
let state = RoomState {
|
||||
document: Document::new(text),
|
||||
presence: Presence::new(),
|
||||
peers: HashMap::new(),
|
||||
base_seq,
|
||||
last_editor: None,
|
||||
dirty: false,
|
||||
last_op_ms: now_ms,
|
||||
};
|
||||
Self { key, state: Mutex::new(state) }
|
||||
}
|
||||
|
||||
/// The key of the room.
|
||||
#[must_use]
|
||||
pub fn key(&self) -> &RoomKey {
|
||||
&self.key
|
||||
}
|
||||
|
||||
fn lock(&self) -> MutexGuard<'_, RoomState> {
|
||||
self.state.lock().unwrap_or_else(PoisonError::into_inner)
|
||||
}
|
||||
|
||||
/// True when nobody is in the room.
|
||||
pub(crate) fn is_empty(&self) -> bool {
|
||||
self.lock().peers.is_empty()
|
||||
}
|
||||
|
||||
/// True when the text changed and the debounce has passed.
|
||||
pub(crate) fn is_due(&self, now_ms: u64, debounce_ms: u64) -> bool {
|
||||
let state = self.lock();
|
||||
state.dirty && now_ms.saturating_sub(state.last_op_ms) >= debounce_ms
|
||||
}
|
||||
|
||||
/// The text, the sequence it is based on and its last editor, when the room
|
||||
/// has changes to write.
|
||||
pub(crate) fn flush_data(&self) -> Option<(String, u64, String)> {
|
||||
let state = self.lock();
|
||||
if !state.dirty {
|
||||
return None;
|
||||
}
|
||||
let editor = state.last_editor.clone()?;
|
||||
Some((state.document.text().to_owned(), state.base_seq, editor))
|
||||
}
|
||||
|
||||
/// Marks the room written; the sequence moves on only when the file itself
|
||||
/// was written.
|
||||
pub(crate) fn mark_clean(&self, seq: Option<u64>) {
|
||||
let mut state = self.lock();
|
||||
state.dirty = false;
|
||||
if let Some(seq) = seq {
|
||||
state.base_seq = seq;
|
||||
}
|
||||
}
|
||||
|
||||
/// Adds a peer and answers the snapshot it adopts.
|
||||
pub(crate) fn join(&self, client: &str, name: &str, now_ms: u64, timings: Timings) -> Result<(String, u64, Vec<Peer>), ApiError> {
|
||||
let mut state = self.lock();
|
||||
prune_locked(&mut state, now_ms, timings.presence_ttl_ms);
|
||||
if !state.peers.contains_key(client) && state.peers.len() >= timings.max_peers {
|
||||
return Err(ApiError::conflict("too many peers in the room"));
|
||||
}
|
||||
let (tx, rx) = mpsc::channel(PEER_CHANNEL);
|
||||
let peer = Peer { client: client.to_owned(), name: name.to_owned(), color: palette_index(name), selection: None, last_seen_ms: now_ms };
|
||||
state.presence.update(peer, now_ms).map_err(|error| collab_error(&error))?;
|
||||
state.peers.insert(client.to_owned(), PeerSlot { tx, rx: Some(rx) });
|
||||
let (text, rev) = state.document.snapshot();
|
||||
let peers = state.peers();
|
||||
Ok((text, rev, peers))
|
||||
}
|
||||
|
||||
/// Applies an operation and tells the room about it.
|
||||
pub(crate) fn submit(&self, edit: Edit<'_>, timings: Timings) -> Result<u64, ApiError> {
|
||||
let mut state = self.lock();
|
||||
prune_locked(&mut state, edit.now_ms, timings.presence_ttl_ms);
|
||||
if !state.peers.contains_key(edit.client) {
|
||||
return Err(ApiError::missing("join the room first"));
|
||||
}
|
||||
let accepted = state.document.submit(edit.base_rev, edit.op).map_err(|error| collab_error(&error))?;
|
||||
state.presence.transform_all(&accepted.op);
|
||||
if let Some(selection) = edit.selection {
|
||||
set_selection(&mut state, edit.client, selection, edit.now_ms);
|
||||
}
|
||||
state.dirty = true;
|
||||
state.last_op_ms = edit.now_ms;
|
||||
state.last_editor = Some(edit.user.to_owned());
|
||||
if let Some(slot) = state.peers.get(edit.client) {
|
||||
let _ = slot.tx.try_send(LiveEvent::Ack { rev: accepted.rev });
|
||||
}
|
||||
let event = LiveEvent::Op { rev: accepted.rev, client: edit.client.to_owned(), op: accepted.op };
|
||||
broadcast_locked(&mut state, &event, Some(edit.client));
|
||||
Ok(accepted.rev)
|
||||
}
|
||||
|
||||
/// Records a heartbeat and tells the room who is there.
|
||||
pub(crate) fn presence(&self, client: &str, selection: Option<Selection>, now_ms: u64, timings: Timings) -> Result<Vec<Peer>, ApiError> {
|
||||
let mut state = self.lock();
|
||||
prune_locked(&mut state, now_ms, timings.presence_ttl_ms);
|
||||
let Some(peer) = state.presence.peers().into_iter().find(|peer| peer.client == client).cloned() else {
|
||||
return Err(ApiError::missing("no such peer"));
|
||||
};
|
||||
let mut peer = peer;
|
||||
if let Some(selection) = selection {
|
||||
peer.selection = Some(selection);
|
||||
}
|
||||
state.presence.update(peer, now_ms).map_err(|error| collab_error(&error))?;
|
||||
let peers = state.peers();
|
||||
let event = LiveEvent::Presence { peers: peers.clone() };
|
||||
broadcast_locked(&mut state, &event, None);
|
||||
Ok(peers)
|
||||
}
|
||||
|
||||
/// Removes a peer; answers whether the room is empty now.
|
||||
pub(crate) fn leave(&self, client: &str) -> bool {
|
||||
let mut state = self.lock();
|
||||
state.peers.remove(client);
|
||||
state.presence.remove(client);
|
||||
let event = LiveEvent::Leave { client: client.to_owned() };
|
||||
broadcast_locked(&mut state, &event, None);
|
||||
state.peers.is_empty()
|
||||
}
|
||||
|
||||
/// Drops the peers that have not been heard from; answers their ids.
|
||||
pub(crate) fn prune(&self, now_ms: u64, ttl_ms: u64) -> Vec<String> {
|
||||
let mut state = self.lock();
|
||||
prune_locked(&mut state, now_ms, ttl_ms)
|
||||
}
|
||||
|
||||
/// Hands the event channel of a peer to its reader.
|
||||
pub(crate) fn take_stream(&self, client: &str) -> Result<mpsc::Receiver<LiveEvent>, ApiError> {
|
||||
let mut state = self.lock();
|
||||
let Some(slot) = state.peers.get_mut(client) else {
|
||||
return Err(ApiError::missing("no such peer"));
|
||||
};
|
||||
slot.rx.take().ok_or_else(|| ApiError::conflict("the event stream is already open"))
|
||||
}
|
||||
}
|
||||
|
||||
/// Sends an event to every peer but `except`; a peer whose channel is full or
|
||||
/// closed is dropped and has to rejoin.
|
||||
fn broadcast_locked(state: &mut RoomState, event: &LiveEvent, except: Option<&str>) {
|
||||
let mut dropped: Vec<String> = Vec::new();
|
||||
for (client, slot) in &state.peers {
|
||||
if Some(client.as_str()) == except {
|
||||
continue;
|
||||
}
|
||||
if slot.tx.try_send(event.clone()).is_err() {
|
||||
dropped.push(client.clone());
|
||||
}
|
||||
}
|
||||
for client in &dropped {
|
||||
state.peers.remove(client);
|
||||
state.presence.remove(client);
|
||||
}
|
||||
for client in dropped {
|
||||
let leave = LiveEvent::Leave { client };
|
||||
for slot in state.peers.values() {
|
||||
let _ = slot.tx.try_send(leave.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Drops the peers that have not been heard from and tells the others.
|
||||
fn prune_locked(state: &mut RoomState, now_ms: u64, ttl_ms: u64) -> Vec<String> {
|
||||
let removed = state.presence.prune(now_ms, ttl_ms);
|
||||
for client in &removed {
|
||||
state.peers.remove(client);
|
||||
}
|
||||
for client in &removed {
|
||||
let leave = LiveEvent::Leave { client: client.clone() };
|
||||
for slot in state.peers.values() {
|
||||
let _ = slot.tx.try_send(leave.clone());
|
||||
}
|
||||
}
|
||||
removed
|
||||
}
|
||||
|
||||
/// Moves the selection of a peer; a peer that is gone is left alone.
|
||||
fn set_selection(state: &mut RoomState, client: &str, selection: Selection, now_ms: u64) {
|
||||
let Some(peer) = state.presence.peers().into_iter().find(|peer| peer.client == client).cloned() else {
|
||||
return;
|
||||
};
|
||||
let mut peer = peer;
|
||||
peer.selection = Some(selection);
|
||||
let _ = state.presence.update(peer, now_ms);
|
||||
}
|
||||
|
||||
/// The palette index of a display name, `0..=8`.
|
||||
fn palette_index(name: &str) -> u8 {
|
||||
let sum: usize = name.bytes().map(usize::from).fold(0, |acc, byte| acc.wrapping_add(byte).wrapping_mul(31));
|
||||
u8::try_from(sum % 9).unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Maps a collab error to the HTTP answer.
|
||||
fn collab_error(error: &CollabError) -> ApiError {
|
||||
match error {
|
||||
CollabError::StaleRevision { .. } | CollabError::RevisionTooNew { .. } => ApiError::conflict("stale"),
|
||||
CollabError::TooManyPeers { .. } => ApiError::conflict("too many peers in the room"),
|
||||
CollabError::DocumentTooLarge { .. } => ApiError::too_large("the document is too large for live editing"),
|
||||
CollabError::LengthMismatch { .. } | CollabError::Malformed(_) | CollabError::InvalidClient(_) | CollabError::InvalidColor { .. } => {
|
||||
ApiError::bad(error.to_string())
|
||||
}
|
||||
CollabError::RevisionOverflow | CollabError::NothingOutstanding => ApiError::internal(error),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
//! The HTTP endpoints of live editing: join, ops, events, presence and leave.
|
||||
//!
|
||||
//! Every endpoint lives under `/v1/vaults/{vault}/live/{path}` and names its
|
||||
//! action in the last part of the path (`note.md/join`, `note.md/ops`, ...).
|
||||
//! The role of the caller decides what it may do: a viewer may join and watch,
|
||||
//! but its operations are refused.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use axum::Json;
|
||||
use axum::body::Bytes;
|
||||
use axum::extract::{Path, Query, State};
|
||||
use axum::http::{HeaderMap, Method, StatusCode};
|
||||
use axum::response::sse::{KeepAlive, KeepAliveStream, Sse};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use notescraft_collab::{Peer, Selection, TextOp};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::live;
|
||||
use crate::live::events::EventStream;
|
||||
use crate::live::room::Edit;
|
||||
use crate::roles::Need;
|
||||
use crate::state::{AppState, authorize, now_ms};
|
||||
|
||||
/// What a live path asks for.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum Action {
|
||||
/// Join a room and take its snapshot.
|
||||
Join,
|
||||
/// Send an operation.
|
||||
Ops,
|
||||
/// Read the event stream.
|
||||
Events,
|
||||
/// Send a heartbeat.
|
||||
Presence,
|
||||
/// Leave the room.
|
||||
Leave,
|
||||
}
|
||||
|
||||
/// The query of the event stream.
|
||||
#[derive(Deserialize)]
|
||||
pub struct EventsQuery {
|
||||
/// The client id of the peer that reads the stream.
|
||||
pub client: Option<String>,
|
||||
}
|
||||
|
||||
/// The body of a join.
|
||||
#[derive(Deserialize)]
|
||||
struct JoinRequest {
|
||||
/// The client id of the peer.
|
||||
client: String,
|
||||
/// The display name of the peer.
|
||||
#[serde(default)]
|
||||
name: String,
|
||||
}
|
||||
|
||||
/// The body of an operation.
|
||||
#[derive(Deserialize)]
|
||||
struct OpsRequest {
|
||||
/// The client id of the author.
|
||||
client: String,
|
||||
/// The revision the operation was built on.
|
||||
base_rev: u64,
|
||||
/// The operation, in the ot.js wire format.
|
||||
op: TextOp,
|
||||
/// Where the author's caret is.
|
||||
#[serde(default)]
|
||||
selection: Option<Selection>,
|
||||
}
|
||||
|
||||
/// The body of a heartbeat.
|
||||
#[derive(Deserialize)]
|
||||
struct PresenceRequest {
|
||||
/// The client id of the peer.
|
||||
client: String,
|
||||
/// Where the peer's caret is.
|
||||
#[serde(default)]
|
||||
selection: Option<Selection>,
|
||||
}
|
||||
|
||||
/// The body of a leave.
|
||||
#[derive(Deserialize)]
|
||||
struct LeaveRequest {
|
||||
/// The client id of the peer.
|
||||
client: String,
|
||||
}
|
||||
|
||||
/// The answer to an operation.
|
||||
#[derive(Serialize)]
|
||||
struct AckReply {
|
||||
/// The revision the operation produced.
|
||||
rev: u64,
|
||||
}
|
||||
|
||||
/// The answer to a heartbeat.
|
||||
#[derive(Serialize)]
|
||||
struct PeersReply {
|
||||
/// The peers in the room.
|
||||
peers: Vec<Peer>,
|
||||
}
|
||||
|
||||
/// Answers every live endpoint; the action is the last part of the path.
|
||||
pub async fn dispatch(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
method: Method,
|
||||
Path((vault, rest)): Path<(String, String)>,
|
||||
Query(query): Query<EventsQuery>,
|
||||
body: Bytes,
|
||||
) -> Result<Response, ApiError> {
|
||||
let Some((action, path)) = split(&rest) else {
|
||||
return Err(ApiError::missing("no such live endpoint"));
|
||||
};
|
||||
match action {
|
||||
Action::Join if method == Method::POST => join(state, headers, vault, path, body).await,
|
||||
Action::Ops if method == Method::POST => ops(&state, &headers, &vault, &path, &body),
|
||||
Action::Events if method == Method::GET => events(&state, &headers, &vault, &path, query).map(IntoResponse::into_response),
|
||||
Action::Presence if method == Method::POST => presence(&state, &headers, &vault, &path, &body),
|
||||
Action::Leave if method == Method::POST => leave(state, headers, vault, path, body).await,
|
||||
_ => Err(ApiError::method_not_allowed()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Splits a live path into the action and the note it acts on.
|
||||
fn split(rest: &str) -> Option<(Action, String)> {
|
||||
const ACTIONS: [(&str, Action); 5] =
|
||||
[("/join", Action::Join), ("/ops", Action::Ops), ("/events", Action::Events), ("/presence", Action::Presence), ("/leave", Action::Leave)];
|
||||
for (suffix, action) in ACTIONS {
|
||||
if let Some(path) = rest.strip_suffix(suffix) {
|
||||
if path.is_empty() {
|
||||
return None;
|
||||
}
|
||||
return Some((action, path.to_owned()));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Parses a JSON body; a body that does not fit the shape is a bad request.
|
||||
fn parse<T: serde::de::DeserializeOwned>(body: &Bytes) -> Result<T, ApiError> {
|
||||
if body.len() > live::MAX_BODY_BYTES {
|
||||
return Err(ApiError::too_large("the body is too large"));
|
||||
}
|
||||
serde_json::from_slice(body).map_err(|error| ApiError::bad(format!("malformed body: {error}")))
|
||||
}
|
||||
|
||||
/// A client id is 1 to 64 letters, digits or dashes.
|
||||
fn check_client(client: &str) -> Result<(), ApiError> {
|
||||
let allowed = !client.is_empty() && client.len() <= 64 && client.bytes().all(|byte| byte.is_ascii_alphanumeric() || byte == b'-');
|
||||
if allowed { Ok(()) } else { Err(ApiError::bad("the client id must be 1 to 64 letters, digits or dashes")) }
|
||||
}
|
||||
|
||||
/// `POST .../join`: joins a peer to the room and answers the snapshot it adopts.
|
||||
async fn join(state: AppState, headers: HeaderMap, vault: String, path: String, body: Bytes) -> Result<Response, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let request: JoinRequest = parse(&body)?;
|
||||
check_client(&request.client)?;
|
||||
let worker = state.clone();
|
||||
let reply = tokio::task::spawn_blocking(move || live::join(&worker, &vault, &path, &request.client, &request.name, &member.device, now_ms()))
|
||||
.await
|
||||
.map_err(|e| ApiError::internal(&e))??;
|
||||
Ok(Json(reply).into_response())
|
||||
}
|
||||
|
||||
/// `POST .../ops`: applies an operation and answers the revision it produced.
|
||||
fn ops(state: &AppState, headers: &HeaderMap, vault: &str, path: &str, body: &Bytes) -> Result<Response, ApiError> {
|
||||
let member = authorize(state, headers, vault, Need::Write)?;
|
||||
let request: OpsRequest = parse(body)?;
|
||||
let edit = Edit { client: &request.client, base_rev: request.base_rev, op: request.op, selection: request.selection, user: &member.user, now_ms: now_ms() };
|
||||
let rev = live::submit(state, vault, path, edit)?;
|
||||
Ok(Json(AckReply { rev }).into_response())
|
||||
}
|
||||
|
||||
/// `POST .../presence`: records a heartbeat and answers who is in the room.
|
||||
fn presence(state: &AppState, headers: &HeaderMap, vault: &str, path: &str, body: &Bytes) -> Result<Response, ApiError> {
|
||||
let _member = authorize(state, headers, vault, Need::Read)?;
|
||||
let request: PresenceRequest = parse(body)?;
|
||||
let peers = live::presence(state, vault, path, &request.client, request.selection, now_ms())?;
|
||||
Ok(Json(PeersReply { peers }).into_response())
|
||||
}
|
||||
|
||||
/// `POST .../leave`: removes a peer from the room.
|
||||
async fn leave(state: AppState, headers: HeaderMap, vault: String, path: String, body: Bytes) -> Result<Response, ApiError> {
|
||||
let _member = authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let request: LeaveRequest = parse(&body)?;
|
||||
let worker = state.clone();
|
||||
tokio::task::spawn_blocking(move || live::leave(&worker, &vault, &path, &request.client, now_ms())).await.map_err(|e| ApiError::internal(&e))??;
|
||||
Ok(StatusCode::NO_CONTENT.into_response())
|
||||
}
|
||||
|
||||
/// `GET .../events`: the Server-Sent Events stream of one peer.
|
||||
fn events(state: &AppState, headers: &HeaderMap, vault: &str, path: &str, query: EventsQuery) -> Result<Sse<KeepAliveStream<EventStream>>, ApiError> {
|
||||
let _member = authorize(state, headers, vault, Need::Read)?;
|
||||
let Some(client) = query.client else {
|
||||
return Err(ApiError::bad("the client id is required"));
|
||||
};
|
||||
let Some(room) = state.live().room(vault, path) else {
|
||||
return Err(ApiError::missing("no such room"));
|
||||
};
|
||||
let receiver = room.take_stream(&client)?;
|
||||
let timings = state.live().timings();
|
||||
let stream = EventStream::new(receiver, Duration::from_millis(timings.idle_ms));
|
||||
let keep_alive = KeepAlive::new().interval(Duration::from_millis(timings.keepalive_ms)).text("keepalive");
|
||||
Ok(Sse::new(stream).keep_alive(keep_alive))
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
//! The sync service binary: reads the environment, opens the data folder, serves HTTP.
|
||||
|
||||
use std::process::ExitCode;
|
||||
|
||||
use notescraft_sync_server::{AppState, Config, Store, router};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> ExitCode {
|
||||
match run().await {
|
||||
Ok(()) => ExitCode::SUCCESS,
|
||||
Err(message) => {
|
||||
eprintln!("notescraft-sync-server: {message}");
|
||||
ExitCode::from(1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn run() -> Result<(), String> {
|
||||
let config = Config::from_env()?;
|
||||
std::fs::create_dir_all(config.data_dir.join("blobs")).map_err(|e| format!("data folder: {e}"))?;
|
||||
let store = Store::open(&config.data_dir.join("sync.db")).map_err(|e| format!("database: {e}"))?;
|
||||
let state = AppState::with_quota(store, config.data_dir.join("blobs"), &config.admin_token, config.quota_bytes);
|
||||
purge_hourly(state.clone());
|
||||
let listener = tokio::net::TcpListener::bind(config.bind).await.map_err(|e| format!("bind {}: {e}", config.bind))?;
|
||||
eprintln!("notescraft-sync-server: listening on {}", config.bind);
|
||||
axum::serve(listener, router(state))
|
||||
.with_graceful_shutdown(async {
|
||||
tokio::signal::ctrl_c().await.ok();
|
||||
})
|
||||
.await
|
||||
.map_err(|e| format!("serve: {e}"))
|
||||
}
|
||||
|
||||
/// How long between purge checks: about an hour; `now` makes it injectable in tests.
|
||||
const PURGE_CHECK_INTERVAL_S: u64 = 60 * 60;
|
||||
|
||||
/// Removes vaults deleted more than two weeks ago, about once an hour.
|
||||
fn purge_hourly(state: AppState) {
|
||||
tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(tokio::time::Duration::from_secs(PURGE_CHECK_INTERVAL_S));
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
|
||||
interval.tick().await; // the interval fires immediately once
|
||||
loop {
|
||||
interval.tick().await;
|
||||
if let Err(error) = state.purge(notescraft_sync_server::now_ms()) {
|
||||
eprintln!("notescraft-sync-server: purge failed: {error:?}");
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
//! Per-request ids: echoed from the client or generated, plus one stderr line per request.
|
||||
//!
|
||||
//! The log line carries `request_id method path status elapsed_ms` and never the
|
||||
//! `Authorization` header or the query string.
|
||||
|
||||
use std::time::Instant;
|
||||
|
||||
use axum::extract::Request;
|
||||
use axum::http::HeaderValue;
|
||||
use axum::middleware::Next;
|
||||
use axum::response::Response;
|
||||
|
||||
use crate::state::{random_hex, valid_vault_id};
|
||||
|
||||
/// The header carrying the id.
|
||||
const HEADER: &str = "x-request-id";
|
||||
/// Length of a generated id in characters.
|
||||
const GENERATED_LEN: usize = 8;
|
||||
/// Never logged ids come out as this when even `/dev/urandom` failed (16 hex chars).
|
||||
const FALLBACK: &str = "0000000000000000";
|
||||
|
||||
/// Reads a valid id from the request, or `None`.
|
||||
fn incoming(request: &Request) -> Option<String> {
|
||||
request.headers().get(HEADER).and_then(|value| value.to_str().ok()).filter(|id| valid_vault_id(id)).map(str::to_owned)
|
||||
}
|
||||
|
||||
/// The middleware: gives every request an id, answers with it, and logs one line.
|
||||
pub async fn request_ids(request: Request, next: Next) -> Response {
|
||||
let id = incoming(&request).unwrap_or_else(|| random_hex(GENERATED_LEN).ok().filter(|id| id.len() == GENERATED_LEN * 2).unwrap_or_else(|| FALLBACK.into()));
|
||||
let (method, path, started) = (request.method().as_str().to_owned(), request.uri().path().to_owned(), Instant::now());
|
||||
let mut response = next.run(request).await;
|
||||
if let Ok(value) = HeaderValue::from_str(&id) {
|
||||
response.headers_mut().insert(HEADER, value);
|
||||
}
|
||||
let elapsed_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
eprintln!("{id} {method} {path} {} {elapsed_ms}", response.status().as_u16());
|
||||
response
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
//! Token buckets for the anonymous publish routes: one per (route, slug or token
|
||||
//! hash) and one global, so scanning unknown slugs is limited too.
|
||||
//!
|
||||
//! The clock is injected (`now_ms`), so a test can drive the buckets without sleeping.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Mutex, PoisonError};
|
||||
|
||||
/// Requests one slug or token may make per minute.
|
||||
pub const PER_KEY_PER_MINUTE: u64 = 60;
|
||||
/// Requests all anonymous routes together may make per minute.
|
||||
pub const GLOBAL_PER_MINUTE: u64 = 600;
|
||||
/// Most per-key buckets kept at once; beyond it only the global bucket applies.
|
||||
const MAX_KEYS: usize = 4096;
|
||||
/// One request costs this many milli-tokens.
|
||||
const COST: u64 = 1000;
|
||||
/// Milli-tokens a full per-key bucket holds.
|
||||
const KEY_CAPACITY: u64 = PER_KEY_PER_MINUTE * COST;
|
||||
/// Milli-tokens the full global bucket holds.
|
||||
const GLOBAL_CAPACITY: u64 = GLOBAL_PER_MINUTE * COST;
|
||||
|
||||
/// One bucket: tokens left and the clock they were last refilled at.
|
||||
struct Bucket {
|
||||
tokens: u64,
|
||||
last_ms: u64,
|
||||
}
|
||||
|
||||
impl Bucket {
|
||||
/// A full bucket: the first requests are free, the budget only runs out later.
|
||||
fn new(now_ms: u64, capacity: u64) -> Self {
|
||||
Self { tokens: capacity, last_ms: now_ms }
|
||||
}
|
||||
|
||||
/// Adds one token per elapsed millisecond, up to `capacity`.
|
||||
fn refill(&mut self, now_ms: u64, capacity: u64) {
|
||||
let elapsed = now_ms.saturating_sub(self.last_ms);
|
||||
self.tokens = self.tokens.saturating_add(elapsed).min(capacity);
|
||||
self.last_ms = now_ms;
|
||||
}
|
||||
|
||||
/// Takes one token, or says how long to wait for one.
|
||||
fn take(&mut self, now_ms: u64, capacity: u64) -> Result<(), u64> {
|
||||
self.refill(now_ms, capacity);
|
||||
if self.tokens >= COST {
|
||||
self.tokens -= COST;
|
||||
return Ok(());
|
||||
}
|
||||
let missing = COST.saturating_sub(self.tokens);
|
||||
Err(missing.div_ceil(COST).max(1))
|
||||
}
|
||||
}
|
||||
|
||||
/// The rate limiter of the anonymous routes.
|
||||
pub struct Budget {
|
||||
keys: Mutex<HashMap<String, Bucket>>,
|
||||
global: Mutex<Bucket>,
|
||||
}
|
||||
|
||||
impl Budget {
|
||||
/// An empty budget; the clock starts at `now_ms`.
|
||||
#[must_use]
|
||||
pub fn new(now_ms: u64) -> Self {
|
||||
Self { keys: Mutex::new(HashMap::new()), global: Mutex::new(Bucket::new(now_ms, GLOBAL_CAPACITY)) }
|
||||
}
|
||||
|
||||
/// Takes one token from the global bucket and from the bucket of `key`.
|
||||
///
|
||||
/// # Errors
|
||||
/// The number of seconds to wait, when a bucket is empty.
|
||||
pub fn take(&self, key: &str, now_ms: u64) -> Result<(), u64> {
|
||||
{
|
||||
let mut global = self.global.lock().unwrap_or_else(PoisonError::into_inner);
|
||||
global.take(now_ms, GLOBAL_CAPACITY)?;
|
||||
}
|
||||
let mut keys = self.keys.lock().unwrap_or_else(PoisonError::into_inner);
|
||||
if !keys.contains_key(key) {
|
||||
if keys.len() >= MAX_KEYS {
|
||||
// Too many keys to remember: the global bucket alone bounds the traffic.
|
||||
return Ok(());
|
||||
}
|
||||
keys.insert(key.to_owned(), Bucket::new(now_ms, KEY_CAPACITY));
|
||||
}
|
||||
let bucket = keys.get_mut(key).ok_or(1u64)?;
|
||||
bucket.take(now_ms, KEY_CAPACITY)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_key_gets_sixty_requests_a_minute() {
|
||||
let budget = Budget::new(0);
|
||||
for _ in 0..PER_KEY_PER_MINUTE {
|
||||
assert!(budget.take("a", 0).is_ok());
|
||||
}
|
||||
let wait = budget.take("a", 0).expect_err("over budget");
|
||||
assert!(wait >= 1, "waits at least a second");
|
||||
// Another key is untouched by the first key's spending.
|
||||
assert!(budget.take("b", 0).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_bucket_refills_with_the_clock() {
|
||||
let budget = Budget::new(0);
|
||||
for _ in 0..PER_KEY_PER_MINUTE {
|
||||
assert!(budget.take("a", 0).is_ok());
|
||||
}
|
||||
assert!(budget.take("a", 0).is_err());
|
||||
assert!(budget.take("a", 60_000).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_global_bucket_bounds_unknown_keys() {
|
||||
let budget = Budget::new(0);
|
||||
for round in 0..GLOBAL_PER_MINUTE {
|
||||
assert!(budget.take(&format!("slug-{round}"), 0).is_ok(), "round {round}");
|
||||
}
|
||||
assert!(budget.take("another", 0).is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
//! Slugs, vault paths and share-link tokens: the rules that decide what a published
|
||||
//! page may show and how a link is addressed.
|
||||
|
||||
use notescraft_sync::{check_path, check_syncable, hash_hex};
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::state::random_hex;
|
||||
use crate::store::{ShareLinkRow, Store};
|
||||
|
||||
/// Folders that are never published, at any depth.
|
||||
const HIDDEN_FOLDERS: [&str; 4] = [".obsidian", ".notescraft", ".trash", ".git"];
|
||||
/// How many slugs are drawn before a vault gives up on one.
|
||||
const SLUG_TRIES: usize = 8;
|
||||
/// The kind of a publication: one note or a folder of notes.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum PublicationKind {
|
||||
/// One note.
|
||||
Note,
|
||||
/// A folder and everything under it.
|
||||
Folder,
|
||||
}
|
||||
|
||||
impl PublicationKind {
|
||||
/// Parses the `kind` of the request; anything else is invalid input.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 when the kind is neither `note` nor `folder`.
|
||||
pub fn parse(raw: &str) -> Result<Self, ApiError> {
|
||||
match raw {
|
||||
"note" => Ok(Self::Note),
|
||||
"folder" => Ok(Self::Folder),
|
||||
other => Err(ApiError::bad(format!("kind must be note or folder, not {other}"))),
|
||||
}
|
||||
}
|
||||
|
||||
/// The name as the database and the API spell it.
|
||||
#[must_use]
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Note => "note",
|
||||
Self::Folder => "folder",
|
||||
}
|
||||
}
|
||||
|
||||
/// The canonical vault path of a target: notes end in `.md`, folders do not.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 when the path is not a clean vault path.
|
||||
pub fn target(self, raw: &str) -> Result<String, ApiError> {
|
||||
match self {
|
||||
Self::Note => note_target(raw),
|
||||
Self::Folder => folder_target(raw),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// True when a slug is 3..=40 characters of `a-z`, `0-9` and `-`.
|
||||
#[must_use]
|
||||
pub fn valid_slug(slug: &str) -> bool {
|
||||
(3..=40).contains(&slug.len()) && slug.bytes().all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-')
|
||||
}
|
||||
|
||||
/// The slug a vault gets by default: its name, slugified, plus a short salt.
|
||||
#[must_use]
|
||||
pub fn default_slug(name: &str, salt: &str) -> String {
|
||||
let mut base = String::new();
|
||||
for character in name.chars() {
|
||||
if character.is_ascii_alphanumeric() {
|
||||
base.push(character.to_ascii_lowercase());
|
||||
} else if !base.ends_with('-') {
|
||||
base.push('-');
|
||||
}
|
||||
}
|
||||
let trimmed: String = base.trim_matches('-').chars().take(30).collect();
|
||||
let base = trimmed.trim_matches('-');
|
||||
let base = if base.is_empty() { "notes" } else { base };
|
||||
format!("{base}-{salt}")
|
||||
}
|
||||
|
||||
/// The canonical path of a published note: `notes/a.md`.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 when the path is not a clean, syncable vault path.
|
||||
pub fn note_target(raw: &str) -> Result<String, ApiError> {
|
||||
let trimmed = raw.trim().trim_start_matches('/');
|
||||
if trimmed.is_empty() {
|
||||
return Err(ApiError::bad("target must name a note"));
|
||||
}
|
||||
check_syncable(trimmed).map_err(|error| ApiError::bad(error.to_string()))?;
|
||||
Ok(if is_markdown(trimmed) { trimmed.to_owned() } else { format!("{trimmed}.md") })
|
||||
}
|
||||
|
||||
/// The canonical path of a published folder: `notes`, or the empty path for the root.
|
||||
///
|
||||
/// # Errors
|
||||
/// 400 when the path is not a clean, syncable vault path.
|
||||
pub fn folder_target(raw: &str) -> Result<String, ApiError> {
|
||||
let trimmed = raw.trim().trim_start_matches('/').trim_end_matches('/');
|
||||
if trimmed.is_empty() {
|
||||
return Ok(String::new());
|
||||
}
|
||||
check_syncable(trimmed).map_err(|error| ApiError::bad(error.to_string()))?;
|
||||
Ok(trimmed.to_owned())
|
||||
}
|
||||
|
||||
/// The path a request asks for, without the leading and trailing slashes and
|
||||
/// without the `.md` suffix; `None` when the path may not be served at all.
|
||||
#[must_use]
|
||||
pub fn request_path(raw: &str) -> Option<String> {
|
||||
let trimmed = raw.trim().trim_start_matches('/').trim_end_matches('/');
|
||||
if trimmed.is_empty() {
|
||||
return Some(String::new());
|
||||
}
|
||||
check_path(trimmed).ok()?;
|
||||
Some(trimmed.to_owned())
|
||||
}
|
||||
|
||||
/// True when a path sits under a folder that is never published.
|
||||
#[must_use]
|
||||
pub fn is_hidden(path: &str) -> bool {
|
||||
path.split('/').any(|segment| HIDDEN_FOLDERS.contains(&segment))
|
||||
}
|
||||
|
||||
/// True when a path names a Markdown file.
|
||||
#[must_use]
|
||||
pub fn is_markdown(path: &str) -> bool {
|
||||
path.len() > 3 && path.get(path.len() - 3..).is_some_and(|suffix| suffix.eq_ignore_ascii_case(".md"))
|
||||
}
|
||||
|
||||
/// The note path a request asks for: the request path plus `.md` when needed.
|
||||
#[must_use]
|
||||
pub fn note_candidate(path: &str) -> Option<String> {
|
||||
if path.is_empty() {
|
||||
return None;
|
||||
}
|
||||
Some(if is_markdown(path) { path.to_owned() } else { format!("{path}.md") })
|
||||
}
|
||||
|
||||
/// True when `path` sits inside the published folder `folder`.
|
||||
#[must_use]
|
||||
pub fn inside_folder(folder: &str, path: &str) -> bool {
|
||||
folder.is_empty() || path.starts_with(&format!("{folder}/"))
|
||||
}
|
||||
|
||||
/// The public path of a published target.
|
||||
#[must_use]
|
||||
pub fn url_path(slug: &str, target: &str, kind: PublicationKind) -> String {
|
||||
match kind {
|
||||
PublicationKind::Note => {
|
||||
let stem = target.strip_suffix(".md").unwrap_or(target);
|
||||
format!("/p/{slug}/{stem}")
|
||||
}
|
||||
PublicationKind::Folder => format!("/p/{slug}/{target}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The slug a vault publishes under, taking the default one when it has none.
|
||||
///
|
||||
/// # Errors
|
||||
/// 409 when no slug could be taken.
|
||||
pub fn ensure_slug(store: &Store, vault: &str) -> Result<String, ApiError> {
|
||||
if let Some(slug) = store.vault_slug(vault)? {
|
||||
return Ok(slug);
|
||||
}
|
||||
let name = store.vault_name(vault)?.unwrap_or_else(|| vault.to_owned());
|
||||
for _ in 0..SLUG_TRIES {
|
||||
let salt = random_hex(2)?;
|
||||
let slug = default_slug(&name, &salt);
|
||||
if !valid_slug(&slug) {
|
||||
continue;
|
||||
}
|
||||
if store.set_vault_slug(vault, &slug)? {
|
||||
return Ok(slug);
|
||||
}
|
||||
}
|
||||
Err(ApiError::conflict("could not take a publish slug"))
|
||||
}
|
||||
|
||||
/// The share link a token names, when it is neither revoked nor expired at `now_ms`.
|
||||
///
|
||||
/// The clock is injected, so a test can expire a link without sleeping.
|
||||
///
|
||||
/// # Errors
|
||||
/// 500 on database errors.
|
||||
pub fn lookup_share_link(store: &Store, token: &str, now_ms: u64) -> Result<Option<ShareLinkRow>, ApiError> {
|
||||
Ok(store.share_link_by_token(&hash_hex(token.as_bytes()), now_ms)?)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn slugs_are_checked() {
|
||||
assert!(valid_slug("abc"));
|
||||
assert!(valid_slug("my-vault-2"));
|
||||
assert!(!valid_slug("ab"));
|
||||
assert!(!valid_slug(&"a".repeat(41)));
|
||||
assert!(!valid_slug("My Vault"));
|
||||
assert!(!valid_slug("a/b"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_default_slug_comes_from_the_name() {
|
||||
assert_eq!(default_slug("My Vault", "ab12"), "my-vault-ab12");
|
||||
assert_eq!(default_slug("***", "ab12"), "notes-ab12");
|
||||
assert!(valid_slug(&default_slug(&"x".repeat(200), "ab12")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn note_targets_end_in_md() {
|
||||
assert_eq!(note_target("notes/a").expect("target"), "notes/a.md");
|
||||
assert_eq!(note_target("/notes/a.md").expect("target"), "notes/a.md");
|
||||
assert!(note_target("..").is_err());
|
||||
assert!(note_target(".obsidian/x").is_err());
|
||||
assert!(note_target("").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn folder_targets_lose_the_trailing_slash() {
|
||||
assert_eq!(folder_target("notes/").expect("target"), "notes");
|
||||
assert_eq!(folder_target("/").expect("target"), "");
|
||||
assert!(folder_target("../x").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_request_paths_are_refused() {
|
||||
assert_eq!(request_path("/notes/a/").expect("path"), "notes/a");
|
||||
assert!(request_path("../secret").is_none());
|
||||
assert!(request_path("a/../../b").is_none());
|
||||
assert!(request_path(&"a".repeat(5000)).is_none());
|
||||
assert!(request_path("a\0b").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hidden_folders_are_hidden_at_any_depth() {
|
||||
assert!(is_hidden(".obsidian/x.md"));
|
||||
assert!(is_hidden("notes/.trash/x.md"));
|
||||
assert!(!is_hidden("notes/x.md"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
//! Publishing notes and folders as web pages, and expiring share links.
|
||||
//!
|
||||
//! A publication makes a note or a folder of the vault reachable under the vault's
|
||||
//! slug (`/p/<slug>/<path>`); a share link opens one note under a secret path
|
||||
//! (`/s/<token>`) until it expires or is revoked. Both are rendered by the same
|
||||
//! static, safe HTML renderer, and both are bounded by a rate limit.
|
||||
|
||||
pub mod budget;
|
||||
pub mod links;
|
||||
pub mod pages;
|
||||
pub mod render;
|
||||
pub mod routes;
|
||||
pub mod wikilinks;
|
||||
|
||||
pub use links::{PublicationKind, default_slug, lookup_share_link, valid_slug};
|
||||
@@ -0,0 +1,255 @@
|
||||
//! The anonymous pages: what a published path serves, and how a note is read and
|
||||
//! rendered for the public.
|
||||
//!
|
||||
//! A path that is not published, hidden, excluded or missing answers 404 with an
|
||||
//! empty body, so nothing about the vault leaks.
|
||||
|
||||
use axum::body::Body;
|
||||
use axum::http::header::{CACHE_CONTROL, CONTENT_TYPE};
|
||||
use axum::http::{HeaderMap, HeaderValue, StatusCode};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::publish::links;
|
||||
use crate::publish::render;
|
||||
use crate::state::{AppState, Core};
|
||||
use crate::store::PublicationRow;
|
||||
|
||||
/// Most notes one index page lists.
|
||||
const MAX_INDEX_ENTRIES: usize = 200;
|
||||
/// Bytes of a note the index reads to see its frontmatter and its title.
|
||||
const INDEX_HEAD_BYTES: usize = 16 * 1024;
|
||||
/// The content security policy of every published page.
|
||||
const CSP: &str = "default-src 'none'; style-src 'unsafe-inline'; img-src https: http:";
|
||||
|
||||
/// The page a published path serves.
|
||||
pub async fn published_page(state: &AppState, slug: &str, path: &str) -> Response {
|
||||
match page_of(state, slug, path).await {
|
||||
Ok(response) => response,
|
||||
Err(error) if error.status == StatusCode::NOT_FOUND => not_found(false),
|
||||
Err(error) => error.into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The page a share-link token opens.
|
||||
pub async fn shared_page(state: &AppState, token: &str, now_ms: u64) -> Response {
|
||||
if !valid_token(token) {
|
||||
return not_found(true);
|
||||
}
|
||||
let found = state.core().store.share_link_by_token(¬escraft_sync::hash_hex(token.as_bytes()), now_ms);
|
||||
let Ok(Some(link)) = found else {
|
||||
return not_found(true);
|
||||
};
|
||||
match note_page(state, &link.vault, &link.path, true).await {
|
||||
Ok(Some(response)) => response,
|
||||
Ok(None) => not_found(true),
|
||||
Err(error) => error.into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The page a published path serves, or an error.
|
||||
async fn page_of(state: &AppState, slug: &str, path: &str) -> Result<Response, ApiError> {
|
||||
let Some(request) = links::request_path(path) else {
|
||||
return Err(ApiError::missing("no such page"));
|
||||
};
|
||||
let (vault, publications) = {
|
||||
let core = state.core();
|
||||
let Some(vault) = core.store.vault_by_slug(slug)? else {
|
||||
return Err(ApiError::missing("no such page"));
|
||||
};
|
||||
let publications = core.store.publications(&vault)?;
|
||||
(vault, publications)
|
||||
};
|
||||
if publications.is_empty() {
|
||||
return Err(ApiError::missing("no such page"));
|
||||
}
|
||||
match resolve(&publications, &request) {
|
||||
Some(Target::Note(path)) => match note_page(state, &vault, &path, false).await? {
|
||||
Some(response) => Ok(response),
|
||||
None => Err(ApiError::missing("no such page")),
|
||||
},
|
||||
Some(Target::Index(prefix)) => index_page(state, &vault, slug, &publications, &prefix).await,
|
||||
None => Err(ApiError::missing("no such page")),
|
||||
}
|
||||
}
|
||||
|
||||
/// What a published path serves: a note or an index of notes.
|
||||
enum Target {
|
||||
/// One note, by its vault path.
|
||||
Note(String),
|
||||
/// The index of a folder, by its vault path.
|
||||
Index(String),
|
||||
}
|
||||
|
||||
/// What a request for a published path gets.
|
||||
fn resolve(publications: &[PublicationRow], request: &str) -> Option<Target> {
|
||||
if let Some(candidate) = links::note_candidate(request)
|
||||
&& publications.iter().any(|row| row.kind == "note" && row.target == candidate)
|
||||
{
|
||||
return Some(Target::Note(candidate));
|
||||
}
|
||||
if publications.iter().any(|row| row.kind == "folder" && row.target == request) {
|
||||
return Some(Target::Index(request.to_owned()));
|
||||
}
|
||||
if let Some(candidate) = links::note_candidate(request)
|
||||
&& publications.iter().any(|row| row.kind == "folder" && links::inside_folder(&row.target, request))
|
||||
{
|
||||
return Some(Target::Note(candidate));
|
||||
}
|
||||
if request.is_empty() {
|
||||
return Some(Target::Index(String::new()));
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// True when a publication covers a note path.
|
||||
fn covered(publications: &[PublicationRow], path: &str) -> bool {
|
||||
publications.iter().any(|row| if row.kind == "note" { row.target == path } else { links::inside_folder(&row.target, path) })
|
||||
}
|
||||
|
||||
/// The page of one note, or `None` when the note may not be shown.
|
||||
async fn note_page(state: &AppState, vault: &str, path: &str, share: bool) -> Result<Option<Response>, ApiError> {
|
||||
if links::is_hidden(path) {
|
||||
return Ok(None);
|
||||
}
|
||||
let Some(source) = read_note(state, vault, path).await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
if render::excluded(&source) {
|
||||
return Ok(None);
|
||||
}
|
||||
let body = render::body_of(&source);
|
||||
let title = render::title_of(path, body);
|
||||
let html = render::page(&title, &render::markdown_to_html(body));
|
||||
Ok(Some(html_response(StatusCode::OK, html, share)))
|
||||
}
|
||||
|
||||
/// The index of a published folder: one link per published note.
|
||||
async fn index_page(state: &AppState, vault: &str, slug: &str, publications: &[PublicationRow], prefix: &str) -> Result<Response, ApiError> {
|
||||
let paths = published_paths(state, vault, publications, prefix)?;
|
||||
let mut entries = Vec::new();
|
||||
for path in &paths {
|
||||
let Some(head) = read_head(state, vault, path).await? else {
|
||||
continue;
|
||||
};
|
||||
if render::excluded(&head) {
|
||||
continue;
|
||||
}
|
||||
let label = render::title_of(path, &head);
|
||||
let stem = path.strip_suffix(".md").unwrap_or(path);
|
||||
entries.push((label, format!("/p/{slug}/{stem}")));
|
||||
}
|
||||
let title = if prefix.is_empty() { "Published notes".to_owned() } else { render::file_stem(prefix) };
|
||||
Ok(html_response(StatusCode::OK, render::index_page(&title, &entries), false))
|
||||
}
|
||||
|
||||
/// The note paths an index lists.
|
||||
fn published_paths(state: &AppState, vault: &str, publications: &[PublicationRow], prefix: &str) -> Result<Vec<String>, ApiError> {
|
||||
let core = state.core();
|
||||
let records = core.store.load_records(vault)?;
|
||||
let mut paths: Vec<String> = records
|
||||
.iter()
|
||||
.filter(|record| !record.deleted)
|
||||
.map(|record| record.path.clone())
|
||||
.filter(|path| links::is_markdown(path) && !links::is_hidden(path))
|
||||
.filter(|path| prefix.is_empty() || path.starts_with(&format!("{prefix}/")))
|
||||
.filter(|path| covered(publications, path))
|
||||
.collect();
|
||||
paths.sort();
|
||||
paths.dedup();
|
||||
paths.truncate(MAX_INDEX_ENTRIES);
|
||||
Ok(paths)
|
||||
}
|
||||
|
||||
/// The hash of a live note, when the vault has one.
|
||||
pub(crate) fn live_hash(core: &mut Core, vault: &str, path: &str) -> Result<Option<String>, ApiError> {
|
||||
let ledger = core.ledger(vault)?;
|
||||
let Some(record) = ledger.get(path).filter(|record| !record.deleted) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let limit = u64::try_from(render::MAX_SOURCE_BYTES).unwrap_or(u64::MAX);
|
||||
if record.size > limit {
|
||||
return Err(ApiError::too_large("note is too large to publish"));
|
||||
}
|
||||
Ok(Some(record.hash.clone()))
|
||||
}
|
||||
|
||||
/// The text of a live note, when the vault has one.
|
||||
async fn read_note(state: &AppState, vault: &str, path: &str) -> Result<Option<String>, ApiError> {
|
||||
let hash = {
|
||||
let mut core = state.core();
|
||||
match live_hash(&mut core, vault, path)? {
|
||||
Some(hash) => hash,
|
||||
None => return Ok(None),
|
||||
}
|
||||
};
|
||||
let blob = state.blob_path(vault, &hash);
|
||||
let bytes = tokio::task::spawn_blocking(move || std::fs::read(blob)).await.map_err(|error| ApiError::internal(&error))??;
|
||||
Ok(String::from_utf8(bytes).ok())
|
||||
}
|
||||
|
||||
/// The beginning of a live note: enough to read its frontmatter and its title.
|
||||
async fn read_head(state: &AppState, vault: &str, path: &str) -> Result<Option<String>, ApiError> {
|
||||
let hash = {
|
||||
let mut core = state.core();
|
||||
let ledger = core.ledger(vault)?;
|
||||
let Some(record) = ledger.get(path).filter(|record| !record.deleted) else {
|
||||
return Ok(None);
|
||||
};
|
||||
record.hash.clone()
|
||||
};
|
||||
let blob = state.blob_path(vault, &hash);
|
||||
let limit = u64::try_from(INDEX_HEAD_BYTES).unwrap_or(u64::MAX);
|
||||
let bytes = tokio::task::spawn_blocking(move || {
|
||||
use std::io::Read;
|
||||
let file = std::fs::File::open(&blob)?;
|
||||
let mut buffer = Vec::new();
|
||||
file.take(limit).read_to_end(&mut buffer)?;
|
||||
Ok::<_, std::io::Error>(buffer)
|
||||
})
|
||||
.await
|
||||
.map_err(|error| ApiError::internal(&error))??;
|
||||
Ok(Some(String::from_utf8_lossy(&bytes).into_owned()))
|
||||
}
|
||||
|
||||
/// True when a token has the shape of a share-link token.
|
||||
fn valid_token(token: &str) -> bool {
|
||||
token.len() == 32 && token.bytes().all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
|
||||
}
|
||||
|
||||
/// The headers a published page carries.
|
||||
fn page_headers(share: bool) -> HeaderMap {
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(CONTENT_TYPE, HeaderValue::from_static("text/html; charset=utf-8"));
|
||||
if let Ok(value) = HeaderValue::from_str(CSP) {
|
||||
headers.insert("content-security-policy", value);
|
||||
}
|
||||
headers.insert("x-content-type-options", HeaderValue::from_static("nosniff"));
|
||||
headers.insert("referrer-policy", HeaderValue::from_static("no-referrer"));
|
||||
if share {
|
||||
headers.insert("x-robots-tag", HeaderValue::from_static("noindex"));
|
||||
headers.insert(CACHE_CONTROL, HeaderValue::from_static("no-store"));
|
||||
} else {
|
||||
headers.insert(CACHE_CONTROL, HeaderValue::from_static("public, max-age=60"));
|
||||
}
|
||||
headers
|
||||
}
|
||||
|
||||
/// A page with its headers.
|
||||
fn html_response(status: StatusCode, body: String, share: bool) -> Response {
|
||||
let mut response = Response::new(Body::from(body));
|
||||
*response.status_mut() = status;
|
||||
*response.headers_mut() = page_headers(share);
|
||||
response
|
||||
}
|
||||
|
||||
/// The answer of a path that may not be shown: 404 with an empty body.
|
||||
fn not_found(share: bool) -> Response {
|
||||
html_response(StatusCode::NOT_FOUND, String::new(), share)
|
||||
}
|
||||
|
||||
/// The answer of a request over budget: 429 with a `Retry-After`.
|
||||
#[must_use]
|
||||
pub fn rate_limited(retry_after_s: u64) -> Response {
|
||||
ApiError { status: StatusCode::TOO_MANY_REQUESTS, message: "too many requests".to_owned(), retry_after: Some(retry_after_s) }.into_response()
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
//! Renders Markdown to safe, static HTML.
|
||||
//!
|
||||
//! Nothing from the source reaches the page as markup: raw HTML becomes escaped
|
||||
//! text, links with a scheme other than `http`, `https` or `mailto` become `#`, and
|
||||
//! images that are not `http(s)` are dropped (their text stays). Wikilinks render as
|
||||
//! plain text; embeds show the file name.
|
||||
|
||||
use notescraft_markdown::yaml::{Value, block_range, parse};
|
||||
use pulldown_cmark::{CowStr, Event, Options, Parser, Tag, TagEnd};
|
||||
|
||||
use crate::publish::wikilinks::strip_wikilinks;
|
||||
use std::fmt::Write as _;
|
||||
|
||||
/// The largest source the renderer accepts, in bytes.
|
||||
pub const MAX_SOURCE_BYTES: usize = 1024 * 1024;
|
||||
|
||||
/// The one inline stylesheet every published page carries.
|
||||
const STYLE: &str = "\
|
||||
:root { color-scheme: dark; }
|
||||
body { background: #1e1e2e; color: #cdd6f4; font-family: ui-sans-serif, system-ui, sans-serif; line-height: 1.6; margin: 0; }
|
||||
main { max-width: 720px; margin: 0 auto; padding: 2.5rem 1.25rem 4rem; }
|
||||
h1, h2, h3, h4 { line-height: 1.25; }
|
||||
a { color: #89b4fa; }
|
||||
code { background: #313244; border-radius: 4px; padding: 0.1rem 0.3rem; font-size: 0.9em; }
|
||||
pre { background: #313244; border-radius: 8px; padding: 1rem; overflow-x: auto; }
|
||||
pre code { background: none; padding: 0; }
|
||||
blockquote { border-left: 3px solid #45475a; color: #a6adc8; margin-left: 0; padding-left: 1rem; }
|
||||
table { border-collapse: collapse; width: 100%; }
|
||||
th, td { border: 1px solid #45475a; padding: 0.4rem 0.6rem; text-align: left; }
|
||||
img { max-width: 100%; height: auto; }
|
||||
hr { border: none; border-top: 1px solid #45475a; margin: 2rem 0; }
|
||||
ul.contains-task-list { list-style: none; padding-left: 1.2rem; }
|
||||
nav a { margin-right: 0.8rem; }
|
||||
";
|
||||
|
||||
/// A whole page: the head, the inline stylesheet and the body.
|
||||
#[must_use]
|
||||
pub fn page(title: &str, body: &str) -> String {
|
||||
format!(
|
||||
"<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n\
|
||||
<title>{}</title>\n<style>\n{STYLE}</style>\n</head>\n<body>\n<main>\n{body}\n</main>\n</body>\n</html>\n",
|
||||
escape_html(title)
|
||||
)
|
||||
}
|
||||
|
||||
/// The index of a published folder: one link per published note.
|
||||
#[must_use]
|
||||
pub fn index_page(title: &str, entries: &[(String, String)]) -> String {
|
||||
let mut body = format!("<h1>{}</h1>\n", escape_html(title));
|
||||
if entries.is_empty() {
|
||||
body.push_str("<p>No published notes yet.</p>\n");
|
||||
} else {
|
||||
body.push_str("<ul>\n");
|
||||
for (label, url) in entries {
|
||||
let _ = writeln!(body, "<li><a href=\"{}\">{}</a></li>", escape_html(url), escape_html(label));
|
||||
}
|
||||
body.push_str("</ul>\n");
|
||||
}
|
||||
page(title, &body)
|
||||
}
|
||||
|
||||
/// True when the note's frontmatter says `publish: false`, which excludes it even
|
||||
/// inside a published folder.
|
||||
#[must_use]
|
||||
pub fn excluded(source: &str) -> bool {
|
||||
let Some(frontmatter) = parse(source) else {
|
||||
return false;
|
||||
};
|
||||
match frontmatter.get("publish") {
|
||||
Some(Value::Bool(false)) => true,
|
||||
Some(Value::String(text)) => text.trim().eq_ignore_ascii_case("false"),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// The text of a note without its frontmatter block.
|
||||
#[must_use]
|
||||
pub fn body_of(source: &str) -> &str {
|
||||
match block_range(source) {
|
||||
Some(range) => source.get(range.end..).unwrap_or(source),
|
||||
None => source,
|
||||
}
|
||||
}
|
||||
|
||||
/// The text of a title, escaped for HTML.
|
||||
#[must_use]
|
||||
pub fn escape_html(text: &str) -> String {
|
||||
let mut out = String::with_capacity(text.len());
|
||||
for character in text.chars() {
|
||||
match character {
|
||||
'&' => out.push_str("&"),
|
||||
'<' => out.push_str("<"),
|
||||
'>' => out.push_str(">"),
|
||||
'"' => out.push_str("""),
|
||||
'\'' => out.push_str("'"),
|
||||
other => out.push(other),
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The title of a page: the first `#` heading of the source, or the file name.
|
||||
#[must_use]
|
||||
pub fn title_of(path: &str, source: &str) -> String {
|
||||
first_heading(source).unwrap_or_else(|| file_stem(path))
|
||||
}
|
||||
|
||||
/// The name of a file without its folder and without the `.md` suffix.
|
||||
#[must_use]
|
||||
pub fn file_stem(path: &str) -> String {
|
||||
let name = path.rsplit('/').next().unwrap_or(path);
|
||||
name.strip_suffix(".md").unwrap_or(name).to_owned()
|
||||
}
|
||||
|
||||
/// The first ATX heading of a source, outside code blocks.
|
||||
#[must_use]
|
||||
pub fn first_heading(source: &str) -> Option<String> {
|
||||
let mut in_fence = false;
|
||||
for line in source.lines() {
|
||||
let trimmed = line.trim_start();
|
||||
if trimmed.starts_with("```") || trimmed.starts_with("~~~") {
|
||||
in_fence = !in_fence;
|
||||
continue;
|
||||
}
|
||||
if in_fence {
|
||||
continue;
|
||||
}
|
||||
if let Some(rest) = trimmed.strip_prefix('#') {
|
||||
let rest = rest.trim_start();
|
||||
if rest.starts_with('#') {
|
||||
continue;
|
||||
}
|
||||
let text = rest.trim();
|
||||
if !text.is_empty() {
|
||||
return Some(text.to_owned());
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Renders Markdown to HTML with every unsafe part of the source neutralised.
|
||||
#[must_use]
|
||||
pub fn markdown_to_html(source: &str) -> String {
|
||||
let options = Options::ENABLE_TABLES | Options::ENABLE_STRIKETHROUGH | Options::ENABLE_TASKLISTS | Options::ENABLE_FOOTNOTES;
|
||||
let text = strip_wikilinks(source);
|
||||
let parser = Parser::new_ext(&text, options);
|
||||
let mut dropping_image = false;
|
||||
let events = parser.map(move |event| match event {
|
||||
Event::Html(html) | Event::InlineHtml(html) => Event::Text(html),
|
||||
Event::Start(Tag::Link { dest_url, title, id, link_type }) => {
|
||||
let href = if is_safe_href(&dest_url) { dest_url.clone() } else { CowStr::Borrowed("#") };
|
||||
Event::Start(Tag::Link { dest_url: href, title, id, link_type })
|
||||
}
|
||||
Event::Start(Tag::Image { dest_url, title, id, link_type }) => {
|
||||
if is_http(&dest_url) {
|
||||
Event::Start(Tag::Image { dest_url, title, id, link_type })
|
||||
} else {
|
||||
dropping_image = true;
|
||||
Event::Text(String::new().into())
|
||||
}
|
||||
}
|
||||
Event::End(TagEnd::Image) => {
|
||||
if dropping_image {
|
||||
dropping_image = false;
|
||||
Event::Text(String::new().into())
|
||||
} else {
|
||||
Event::End(TagEnd::Image)
|
||||
}
|
||||
}
|
||||
other => other,
|
||||
});
|
||||
let mut out = String::new();
|
||||
pulldown_cmark::html::push_html(&mut out, events);
|
||||
out
|
||||
}
|
||||
|
||||
/// True when a link may keep its destination: `http`, `https`, `mailto` and in-page
|
||||
/// anchors; every other scheme becomes `#`.
|
||||
fn is_safe_href(dest: &str) -> bool {
|
||||
let lower = dest.trim().to_ascii_lowercase();
|
||||
lower.starts_with("http://") || lower.starts_with("https://") || lower.starts_with("mailto:") || lower.starts_with('#')
|
||||
}
|
||||
|
||||
/// True when a URL is an `http(s)` one, the only images that are kept.
|
||||
fn is_http(url: &str) -> bool {
|
||||
let lower = url.trim().to_ascii_lowercase();
|
||||
lower.starts_with("http://") || lower.starts_with("https://")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn raw_html_is_escaped() {
|
||||
let html = markdown_to_html("<script>alert(1)</script>\n\n<img src=x onerror=alert(1)>\n");
|
||||
assert!(!html.contains("<script"), "{html}");
|
||||
assert!(!html.contains("<img"), "{html}");
|
||||
assert!(html.contains("<script>"), "{html}");
|
||||
assert!(html.contains("<img src=x onerror=alert(1)>"), "{html}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_safe_link_schemes_survive() {
|
||||
let html = markdown_to_html("[a](javascript:alert(1)) [b](https://example.com) [c](mailto:x@y.z) [d](#part)");
|
||||
assert!(html.contains("href=\"#\""), "{html}");
|
||||
assert!(html.contains("href=\"https://example.com\""), "{html}");
|
||||
assert!(html.contains("href=\"mailto:x@y.z\""), "{html}");
|
||||
assert!(html.contains("href=\"#part\""), "{html}");
|
||||
assert!(!html.contains("javascript:"), "{html}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn local_images_are_dropped_but_their_text_stays() {
|
||||
let html = markdown_to_html("\n");
|
||||
assert!(!html.contains("<img"), "{html}");
|
||||
assert!(html.contains("photo.png"), "{html}");
|
||||
let remote = markdown_to_html("\n");
|
||||
assert!(remote.contains("<img"), "{remote}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_title_comes_from_the_first_heading() {
|
||||
assert_eq!(first_heading("# Title\n\nbody"), Some("Title".to_owned()));
|
||||
assert_eq!(first_heading("body only"), None);
|
||||
assert_eq!(first_heading("```\n# Code\n```\n# Real"), Some("Real".to_owned()));
|
||||
assert_eq!(title_of("notes/a.md", "no heading"), "a");
|
||||
assert_eq!(file_stem("notes/a.md"), "a");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_page_carries_no_script() {
|
||||
let html = page("T", "<p>body</p>");
|
||||
assert!(html.contains("<title>T</title>"), "{html}");
|
||||
assert!(!html.contains("<script"), "{html}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,279 @@
|
||||
//! The publish endpoints: the authenticated management of publications and share
|
||||
//! links, and the anonymous pages that serve them.
|
||||
//!
|
||||
//! Published pages are static, safe HTML: no script, no raw HTML from the source, and
|
||||
//! a strict content security policy. A path that is not published, hidden, excluded or
|
||||
//! missing answers 404 with an empty body, so nothing is revealed.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use axum::response::Response;
|
||||
use notescraft_sync::hash_hex;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::api::audit;
|
||||
use crate::error::ApiError;
|
||||
use crate::publish::links::{self, PublicationKind};
|
||||
use crate::publish::pages::{self, live_hash};
|
||||
use crate::roles::Need;
|
||||
use crate::state::{AppState, authorize, now_ms, random_hex};
|
||||
use crate::store::NewShareLink;
|
||||
|
||||
/// Most publications one vault may have.
|
||||
pub const MAX_PUBLICATIONS: u64 = 200;
|
||||
/// Most share links one vault may have.
|
||||
pub const MAX_SHARE_LINKS: u64 = 500;
|
||||
/// Shortest share link lifetime, in seconds.
|
||||
pub const MIN_LINK_SECONDS: u64 = 60;
|
||||
/// Longest share link lifetime, in seconds.
|
||||
pub const MAX_LINK_SECONDS: u64 = 2_592_000;
|
||||
/// Default share link lifetime, in seconds.
|
||||
pub const DEFAULT_LINK_SECONDS: u64 = 604_800;
|
||||
/// The body of `POST /v1/vaults/{vault}/publications`.
|
||||
#[derive(Deserialize)]
|
||||
pub struct PublicationRequest {
|
||||
/// The vault path to publish.
|
||||
pub target: String,
|
||||
/// `note` or `folder`.
|
||||
pub kind: String,
|
||||
}
|
||||
|
||||
/// The answer of `POST /v1/vaults/{vault}/publications`.
|
||||
#[derive(Serialize)]
|
||||
pub struct PublicationReply {
|
||||
/// The publication id.
|
||||
pub id: String,
|
||||
/// The vault's publish slug.
|
||||
pub slug: String,
|
||||
/// Where the page lives.
|
||||
pub url_path: String,
|
||||
}
|
||||
|
||||
/// One row of the publication listing.
|
||||
#[derive(Serialize)]
|
||||
pub struct PublicationListing {
|
||||
/// The publication id.
|
||||
pub id: String,
|
||||
/// `note` or `folder`.
|
||||
pub kind: String,
|
||||
/// The vault path it publishes.
|
||||
pub target: String,
|
||||
/// Where the page lives.
|
||||
pub url_path: String,
|
||||
/// When it was published.
|
||||
pub created_ms: u64,
|
||||
}
|
||||
|
||||
/// The body of `POST /v1/vaults/{vault}/share-links`.
|
||||
#[derive(Deserialize)]
|
||||
pub struct ShareLinkRequest {
|
||||
/// The note the link opens.
|
||||
pub path: String,
|
||||
/// How long the link lives, in seconds.
|
||||
pub expires_in_s: Option<u64>,
|
||||
}
|
||||
|
||||
/// The answer of `POST /v1/vaults/{vault}/share-links`: the token is in the path and is
|
||||
/// shown only here.
|
||||
#[derive(Serialize)]
|
||||
pub struct ShareLinkReply {
|
||||
/// The link id.
|
||||
pub id: String,
|
||||
/// Where the page lives; the token is the last part of it.
|
||||
pub url_path: String,
|
||||
/// When the link stops working, in milliseconds since the epoch.
|
||||
pub expires_ms: u64,
|
||||
}
|
||||
|
||||
/// One row of the share-link listing; never the token.
|
||||
#[derive(Serialize)]
|
||||
pub struct ShareLinkListing {
|
||||
/// The link id.
|
||||
pub id: String,
|
||||
/// The note the link opens.
|
||||
pub path: String,
|
||||
/// When it was minted.
|
||||
pub created_ms: u64,
|
||||
/// When it stops working.
|
||||
pub expires_ms: u64,
|
||||
/// True once it was revoked.
|
||||
pub revoked: bool,
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/publications {target, kind}`: publishes one note or one
|
||||
/// folder (owner). The vault's slug is taken on the first publication.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a non-owner, 404 for a missing note, 400 for a bad target or kind,
|
||||
/// 409 over the publication cap or for an already published target, 500 on database
|
||||
/// errors.
|
||||
pub async fn create_publication(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(request): Json<PublicationRequest>,
|
||||
) -> Result<Json<PublicationReply>, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
let kind = PublicationKind::parse(&request.kind)?;
|
||||
let target = kind.target(&request.target)?;
|
||||
let mut core = state.core();
|
||||
if kind == PublicationKind::Note && live_hash(&mut core, &vault, &target)?.is_none() {
|
||||
return Err(ApiError::missing("no such note"));
|
||||
}
|
||||
if core.store.publication_count(&vault)? >= MAX_PUBLICATIONS {
|
||||
return Err(ApiError::conflict(format!("at most {MAX_PUBLICATIONS} publications")));
|
||||
}
|
||||
if core.store.publication_by_target(&vault, &target)?.is_some() {
|
||||
return Err(ApiError::conflict("already published"));
|
||||
}
|
||||
let slug = links::ensure_slug(&core.store, &vault)?;
|
||||
let id = format!("pub-{}", random_hex(8)?);
|
||||
core.store.create_publication(&id, &vault, &target, kind.as_str(), now_ms(), Some(&member.device))?;
|
||||
audit::record(&core.store, audit::PUBLICATION_CREATED, Some(&vault), Some(&member.device), Some(&target))?;
|
||||
Ok(Json(PublicationReply { id, slug: slug.clone(), url_path: links::url_path(&slug, &target, kind) }))
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/publications`: the vault's publications.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign vault, 500 on database errors.
|
||||
pub async fn list_publications(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
) -> Result<Json<Vec<PublicationListing>>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let core = state.core();
|
||||
let slug = core.store.vault_slug(&vault)?.unwrap_or_default();
|
||||
let mut rows = Vec::new();
|
||||
for publication in core.store.publications(&vault)? {
|
||||
let kind = if publication.kind == "folder" { PublicationKind::Folder } else { PublicationKind::Note };
|
||||
rows.push(PublicationListing {
|
||||
id: publication.id,
|
||||
kind: publication.kind.clone(),
|
||||
target: publication.target.clone(),
|
||||
url_path: links::url_path(&slug, &publication.target, kind),
|
||||
created_ms: publication.created_ms,
|
||||
});
|
||||
}
|
||||
Ok(Json(rows))
|
||||
}
|
||||
|
||||
/// `DELETE /v1/vaults/{vault}/publications/{id}`: unpublishes at once (owner); the
|
||||
/// page answers 404 from then on.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a non-owner, 404 for an unknown publication, 500 on database errors.
|
||||
pub async fn delete_publication(State(state): State<AppState>, headers: HeaderMap, Path((vault, id)): Path<(String, String)>) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Admin)?;
|
||||
let core = state.core();
|
||||
let Some(publication) = core.store.publication(&vault, &id)? else {
|
||||
return Err(ApiError::missing("no such publication"));
|
||||
};
|
||||
core.store.delete_publication(&vault, &id)?;
|
||||
audit::record(&core.store, audit::PUBLICATION_REMOVED, Some(&vault), Some(&member.device), Some(&publication.target))?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `POST /v1/vaults/{vault}/share-links {path, expires_in_s}`: mints a read-only
|
||||
/// link to one note (editor or owner). The token is in the answer and is stored only
|
||||
/// as a hash.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a viewer, 404 for a missing note, 400 for a bad path or lifetime,
|
||||
/// 409 over the link cap, 500 on database errors.
|
||||
pub async fn create_share_link(
|
||||
State(state): State<AppState>,
|
||||
headers: HeaderMap,
|
||||
Path(vault): Path<String>,
|
||||
Json(request): Json<ShareLinkRequest>,
|
||||
) -> Result<Json<ShareLinkReply>, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
let seconds = request.expires_in_s.unwrap_or(DEFAULT_LINK_SECONDS);
|
||||
if !(MIN_LINK_SECONDS..=MAX_LINK_SECONDS).contains(&seconds) {
|
||||
return Err(ApiError::bad(format!("expires_in_s must be {MIN_LINK_SECONDS}..={MAX_LINK_SECONDS}")));
|
||||
}
|
||||
let path = links::note_target(&request.path)?;
|
||||
let mut core = state.core();
|
||||
if live_hash(&mut core, &vault, &path)?.is_none() {
|
||||
return Err(ApiError::missing("no such note"));
|
||||
}
|
||||
if core.store.share_link_count(&vault)? >= MAX_SHARE_LINKS {
|
||||
return Err(ApiError::conflict(format!("at most {MAX_SHARE_LINKS} share links")));
|
||||
}
|
||||
let token = random_hex(16)?;
|
||||
let id = format!("sl-{}", random_hex(8)?);
|
||||
let created = now_ms();
|
||||
let expires = created.saturating_add(seconds.saturating_mul(1000));
|
||||
core.store.create_share_link(&NewShareLink {
|
||||
id: &id,
|
||||
vault: &vault,
|
||||
path: &path,
|
||||
token_hash: &hash_hex(token.as_bytes()),
|
||||
created_ms: created,
|
||||
expires_ms: expires,
|
||||
device: Some(&member.device),
|
||||
})?;
|
||||
audit::record(&core.store, audit::SHARE_LINK_CREATED, Some(&vault), Some(&member.device), Some(&path))?;
|
||||
Ok(Json(ShareLinkReply { id, url_path: format!("/s/{token}"), expires_ms: expires }))
|
||||
}
|
||||
|
||||
/// `GET /v1/vaults/{vault}/share-links`: the vault's share links, never their tokens.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 404 for a foreign vault, 500 on database errors.
|
||||
pub async fn list_share_links(State(state): State<AppState>, headers: HeaderMap, Path(vault): Path<String>) -> Result<Json<Vec<ShareLinkListing>>, ApiError> {
|
||||
authorize(&state, &headers, &vault, Need::Read)?;
|
||||
let core = state.core();
|
||||
let mut rows = Vec::new();
|
||||
for link in core.store.share_links(&vault)? {
|
||||
rows.push(ShareLinkListing { id: link.id, path: link.path, created_ms: link.created_ms, expires_ms: link.expires_ms, revoked: link.revoked });
|
||||
}
|
||||
Ok(Json(rows))
|
||||
}
|
||||
|
||||
/// `DELETE /v1/vaults/{vault}/share-links/{id}`: revokes a share link at once
|
||||
/// (editor or owner); the page answers 404 from then on.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403 for a viewer, 404 for an unknown link, 500 on database errors.
|
||||
pub async fn revoke_share_link(State(state): State<AppState>, headers: HeaderMap, Path((vault, id)): Path<(String, String)>) -> Result<StatusCode, ApiError> {
|
||||
let member = authorize(&state, &headers, &vault, Need::Write)?;
|
||||
let core = state.core();
|
||||
let Some(link) = core.store.share_link(&vault, &id)? else {
|
||||
return Err(ApiError::missing("no such share link"));
|
||||
};
|
||||
core.store.revoke_share_link(&vault, &id)?;
|
||||
audit::record(&core.store, audit::SHARE_LINK_REVOKED, Some(&vault), Some(&member.device), Some(&link.path))?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
/// `GET /p/{slug}`: the index of everything the vault publishes.
|
||||
pub async fn publish_root(State(state): State<AppState>, Path(slug): Path<String>) -> Response {
|
||||
serve_published(&state, &slug, "").await
|
||||
}
|
||||
|
||||
/// `GET /p/{slug}/{*path}`: one published page.
|
||||
pub async fn publish_page(State(state): State<AppState>, Path((slug, path)): Path<(String, String)>) -> Response {
|
||||
serve_published(&state, &slug, &path).await
|
||||
}
|
||||
|
||||
/// `GET /s/{token}`: the note a share link opens.
|
||||
pub async fn share_page(State(state): State<AppState>, Path(token): Path<String>) -> Response {
|
||||
let now = now_ms();
|
||||
let key = format!("s:{}", hash_hex(token.as_bytes()));
|
||||
if let Err(wait) = state.publish_budget().take(&key, now) {
|
||||
return pages::rate_limited(wait);
|
||||
}
|
||||
pages::shared_page(&state, &token, now).await
|
||||
}
|
||||
|
||||
/// The page a published path serves, within the anonymous budget.
|
||||
async fn serve_published(state: &AppState, slug: &str, path: &str) -> Response {
|
||||
let now = now_ms();
|
||||
if let Err(wait) = state.publish_budget().take(&format!("p:{slug}"), now) {
|
||||
return pages::rate_limited(wait);
|
||||
}
|
||||
pages::published_page(state, slug, path).await
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
//! Wikilinks in a note: `[[note]]` and the embed `![[photo.png]]`.
|
||||
//!
|
||||
//! A published page shows plain text, never a link into the vault, so the wikilinks
|
||||
//! of a source are replaced by their text: `[[note]]` shows `note` and an embed shows
|
||||
//! the file name. Code blocks and code spans keep their text as they are.
|
||||
|
||||
/// Replaces the wikilinks of a source by their text: `[[note]]` shows `note` and an
|
||||
/// embed `![[photo.png]]` shows the file name. Code blocks and code spans keep their
|
||||
/// text as they are.
|
||||
#[must_use]
|
||||
pub fn strip_wikilinks(source: &str) -> String {
|
||||
let mut out = String::with_capacity(source.len());
|
||||
let mut in_fence = false;
|
||||
for line in source.split_inclusive('\n') {
|
||||
let trimmed = line.trim_start();
|
||||
if trimmed.starts_with("```") || trimmed.starts_with("~~~") {
|
||||
in_fence = !in_fence;
|
||||
out.push_str(line);
|
||||
continue;
|
||||
}
|
||||
if in_fence {
|
||||
out.push_str(line);
|
||||
continue;
|
||||
}
|
||||
out.push_str(&replace_wikilinks(line));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// One line without its wikilinks.
|
||||
fn replace_wikilinks(line: &str) -> String {
|
||||
let characters: Vec<char> = line.chars().collect();
|
||||
let mut out = String::with_capacity(line.len());
|
||||
let mut index = 0;
|
||||
while index < characters.len() {
|
||||
let character = characters[index];
|
||||
if character == '`' {
|
||||
let end = code_span_end(&characters, index);
|
||||
for part in characters.get(index..end).unwrap_or_default() {
|
||||
out.push(*part);
|
||||
}
|
||||
index = end;
|
||||
continue;
|
||||
}
|
||||
let embed = character == '!' && characters.get(index + 1) == Some(&'[');
|
||||
if (embed || character == '[')
|
||||
&& let Some((text, end)) = read_wikilink(&characters, index + usize::from(embed))
|
||||
{
|
||||
out.push_str(&text);
|
||||
index = end;
|
||||
continue;
|
||||
}
|
||||
out.push(character);
|
||||
index += 1;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The end of the code span that starts at `start`, the line's end when it never closes.
|
||||
fn code_span_end(characters: &[char], start: usize) -> usize {
|
||||
let mut ticks = 0;
|
||||
while characters.get(start + ticks) == Some(&'`') {
|
||||
ticks += 1;
|
||||
}
|
||||
let mut index = start + ticks;
|
||||
while index < characters.len() {
|
||||
if characters.get(index) != Some(&'`') {
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
let mut run = 0;
|
||||
while characters.get(index + run) == Some(&'`') {
|
||||
run += 1;
|
||||
}
|
||||
if run == ticks {
|
||||
return index + run;
|
||||
}
|
||||
index += run;
|
||||
}
|
||||
characters.len()
|
||||
}
|
||||
|
||||
/// The text of the wikilink that starts at `start` and the index past it. A wikilink
|
||||
/// is `[[target]]` or `[[target|alias]]`.
|
||||
fn read_wikilink(characters: &[char], start: usize) -> Option<(String, usize)> {
|
||||
if characters.get(start) != Some(&'[') || characters.get(start + 1) != Some(&'[') {
|
||||
return None;
|
||||
}
|
||||
let mut index = start + 2;
|
||||
while index + 1 < characters.len() {
|
||||
if characters.get(index) == Some(&']') && characters.get(index + 1) == Some(&']') {
|
||||
let inner: String = characters.get(start + 2..index).unwrap_or_default().iter().collect();
|
||||
let label = inner.split('|').next_back().unwrap_or("").trim().to_owned();
|
||||
let target = inner.split('|').next().unwrap_or("").trim();
|
||||
let name = target.rsplit('/').next().unwrap_or(target);
|
||||
let text = if label.is_empty() { name.to_owned() } else { label };
|
||||
return Some((text, index + 2));
|
||||
}
|
||||
index += 1;
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn wikilinks_become_plain_text() {
|
||||
assert_eq!(strip_wikilinks("see [[note]] now"), "see note now");
|
||||
assert_eq!(strip_wikilinks("![[photo.png]]"), "photo.png");
|
||||
assert_eq!(strip_wikilinks("[[folder/note|Alias]]"), "Alias");
|
||||
assert_eq!(strip_wikilinks("`[[code]]`"), "`[[code]]`");
|
||||
assert_eq!(strip_wikilinks("```\n[[code]]\n```\n"), "```\n[[code]]\n```\n");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unclosed_wikilink_stays_as_it_is() {
|
||||
assert_eq!(strip_wikilinks("a [[broken"), "a [[broken");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
//! Membership roles and the access levels `authorize` hands out.
|
||||
|
||||
/// A member's power over one vault, from the `members.role` column.
|
||||
///
|
||||
/// Ordered least powerful first, so `max` keeps the stronger membership.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub(crate) enum Role {
|
||||
/// Reads only.
|
||||
Viewer,
|
||||
/// Reads and writes files.
|
||||
Editor,
|
||||
/// Reads, writes, manages members, invites and the vault itself.
|
||||
Owner,
|
||||
}
|
||||
|
||||
impl Role {
|
||||
/// Parses the column value; anything else is a programming or database error.
|
||||
#[must_use]
|
||||
pub fn parse(raw: &str) -> Option<Self> {
|
||||
match raw {
|
||||
"owner" => Some(Self::Owner),
|
||||
"editor" => Some(Self::Editor),
|
||||
"viewer" => Some(Self::Viewer),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The name as the database and the HTTP API spell it.
|
||||
#[must_use]
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Owner => "owner",
|
||||
Self::Editor => "editor",
|
||||
Self::Viewer => "viewer",
|
||||
}
|
||||
}
|
||||
|
||||
/// The higher of two roles (an owner stays an owner).
|
||||
#[must_use]
|
||||
pub fn max(self, other: Self) -> Self {
|
||||
if self < other { other } else { self }
|
||||
}
|
||||
|
||||
/// True when this role may act at `need`.
|
||||
#[must_use]
|
||||
pub fn satisfies(self, need: Need) -> bool {
|
||||
match need {
|
||||
Need::Read => true,
|
||||
Need::Write => self >= Self::Editor, // owner and editor, not viewer
|
||||
Need::Admin => self == Self::Owner,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What a handler wants from a vault: read a file, change one, or manage members and invites.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub(crate) enum Need {
|
||||
/// `GET /changes`, `GET` files, `batch/check`, device listings, `files-meta`.
|
||||
Read,
|
||||
/// `PUT` and `DELETE` files.
|
||||
Write,
|
||||
/// Members, invites, renames, deletes, pairing-issue side effects.
|
||||
Admin,
|
||||
}
|
||||
|
||||
/// Who a request authenticates as, plus the role it holds in one vault.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct Member {
|
||||
/// The device id the token belongs to.
|
||||
pub device: String,
|
||||
/// The user id the device belongs to.
|
||||
pub user: String,
|
||||
/// The role in the vault of the request.
|
||||
pub role: Role,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn roles_parse_and_rank() {
|
||||
for (raw, role) in [("owner", Role::Owner), ("editor", Role::Editor), ("viewer", Role::Viewer)] {
|
||||
assert_eq!(Role::parse(raw), Some(role));
|
||||
assert_eq!(role.as_str(), raw);
|
||||
}
|
||||
assert_eq!(Role::parse("god"), None);
|
||||
|
||||
assert!(Role::Viewer.satisfies(Need::Read) && !Role::Viewer.satisfies(Need::Write) && !Role::Viewer.satisfies(Need::Admin));
|
||||
assert!(Role::Editor.satisfies(Need::Read) && Role::Editor.satisfies(Need::Write) && !Role::Editor.satisfies(Need::Admin));
|
||||
assert!(Role::Owner.satisfies(Need::Read) && Role::Owner.satisfies(Need::Write) && Role::Owner.satisfies(Need::Admin));
|
||||
|
||||
assert_eq!(Role::Viewer.max(Role::Editor), Role::Editor);
|
||||
assert_eq!(Role::Owner.max(Role::Viewer), Role::Owner);
|
||||
assert!(Role::Viewer < Role::Editor && Role::Editor < Role::Owner);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
//! Shared server state: the database, the per-vault ledger cache, the blob folder, the
|
||||
//! admin key, the pairing lockout and the request authentication helpers.
|
||||
//!
|
||||
//! Authentication is account based: a device belongs to a user, and what the request
|
||||
//! may do in a vault is the user's role there ([`crate::api::roles::Need`] levels).
|
||||
//! Non-members and deleted vaults answer 404, viewers writing answer 403.
|
||||
|
||||
use std::io::Read;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use axum::http::HeaderMap;
|
||||
use notescraft_sync::{Ledger, hash_hex};
|
||||
use tokio::sync::Notify;
|
||||
|
||||
use crate::cache::Lru;
|
||||
use crate::chunks::{CHUNK_DIR, MAX_CHUNK_BYTES, MAX_CHUNKED_BYTES};
|
||||
use crate::error::ApiError;
|
||||
use crate::limits::Lockout;
|
||||
use crate::live;
|
||||
use crate::publish;
|
||||
use crate::roles::{Member, Need, Role};
|
||||
use crate::store::Store;
|
||||
|
||||
/// Ledgers the server keeps in memory at once: listing or touching hundreds of vaults
|
||||
/// reloads older ones from SQLite instead of growing without bound.
|
||||
pub const LEDGER_CACHE_CAP: usize = 64;
|
||||
|
||||
/// Pairing codes live this long.
|
||||
pub const PAIRING_TTL_MS: u64 = 10 * 60 * 1000;
|
||||
/// Default per-vault live-bytes quota in MB; `0` in the environment means unlimited.
|
||||
const DEFAULT_QUOTA_MB: u64 = 5120;
|
||||
|
||||
pub(crate) struct Core {
|
||||
pub store: Store,
|
||||
ledgers: Lru<Ledger>,
|
||||
}
|
||||
|
||||
impl Core {
|
||||
/// The vault's ledger, loaded from the database on a cache miss and evicted when
|
||||
/// the cache runs over its cap. A ledger in use is guarded by the lock, so it
|
||||
/// cannot be evicted or replaced while a request holds a borrow of it.
|
||||
///
|
||||
/// # Errors
|
||||
/// 500 on database errors.
|
||||
pub fn ledger(&mut self, vault: &str) -> Result<&mut Ledger, ApiError> {
|
||||
let store = &self.store;
|
||||
self.ledgers.get_or_insert(vault, |vault| Ok(Ledger::from_records(store.load_records(vault)?)))
|
||||
}
|
||||
|
||||
/// Forgets a cached ledger so the next use reloads it from the database.
|
||||
pub fn drop_ledger(&mut self, vault: &str) {
|
||||
self.ledgers.remove(vault);
|
||||
}
|
||||
}
|
||||
|
||||
struct Inner {
|
||||
core: Mutex<Core>,
|
||||
blobs: PathBuf,
|
||||
admin_hash: String,
|
||||
changed: Notify,
|
||||
lockout: Lockout,
|
||||
quota_bytes: Option<u64>,
|
||||
live: live::Registry,
|
||||
publish_budget: publish::budget::Budget,
|
||||
chunk_max: AtomicUsize,
|
||||
chunk_file_cap: AtomicU64,
|
||||
}
|
||||
|
||||
/// Cheap to clone; handed to every handler.
|
||||
#[derive(Clone)]
|
||||
pub struct AppState(Arc<Inner>);
|
||||
|
||||
impl AppState {
|
||||
/// Builds the state over an open store and a blob folder, with the default quota.
|
||||
pub fn new(store: Store, blobs: PathBuf, admin_token: &str) -> Self {
|
||||
Self::with_quota(store, blobs, admin_token, Some(DEFAULT_QUOTA_MB * 1024 * 1024))
|
||||
}
|
||||
|
||||
/// Builds the state with a per-vault live-bytes quota; `None` means unlimited.
|
||||
/// The pairing lockout starts empty: failure timestamps live in memory only.
|
||||
pub fn with_quota(store: Store, blobs: PathBuf, admin_token: &str, quota_bytes: Option<u64>) -> Self {
|
||||
Self(Arc::new(Inner {
|
||||
core: Mutex::new(Core { store, ledgers: Lru::new(LEDGER_CACHE_CAP) }),
|
||||
blobs,
|
||||
admin_hash: hash_hex(admin_token.as_bytes()),
|
||||
changed: Notify::new(),
|
||||
lockout: Lockout::default(),
|
||||
quota_bytes,
|
||||
live: live::Registry::new(live::Timings::default()),
|
||||
publish_budget: publish::budget::Budget::new(now_ms()),
|
||||
chunk_max: AtomicUsize::new(MAX_CHUNK_BYTES),
|
||||
chunk_file_cap: AtomicU64::new(MAX_CHUNKED_BYTES),
|
||||
}))
|
||||
}
|
||||
|
||||
pub(crate) fn core(&self) -> MutexGuard<'_, Core> {
|
||||
self.0.core.lock().unwrap_or_else(PoisonError::into_inner)
|
||||
}
|
||||
|
||||
/// Wakes every long-polling `/changes` request.
|
||||
pub(crate) fn announce(&self) {
|
||||
self.0.changed.notify_waiters();
|
||||
}
|
||||
|
||||
pub(crate) fn changed(&self) -> &Notify {
|
||||
&self.0.changed
|
||||
}
|
||||
|
||||
pub(crate) fn blob_path(&self, vault: &str, hash: &str) -> PathBuf {
|
||||
self.0.blobs.join(vault).join(hash)
|
||||
}
|
||||
|
||||
/// The blob folder root.
|
||||
pub(crate) fn blobs(&self) -> &Path {
|
||||
&self.0.blobs
|
||||
}
|
||||
|
||||
/// The chunk folder of a vault, beside its blob files.
|
||||
pub(crate) fn chunk_dir(&self, vault: &str) -> PathBuf {
|
||||
self.0.blobs.join(vault).join(CHUNK_DIR)
|
||||
}
|
||||
|
||||
/// The content-addressed chunk path of one hash.
|
||||
pub(crate) fn chunk_path(&self, vault: &str, hash: &str) -> PathBuf {
|
||||
self.chunk_dir(vault).join(hash)
|
||||
}
|
||||
|
||||
/// The per-chunk body cap, in bytes.
|
||||
pub(crate) fn chunk_max(&self) -> usize {
|
||||
self.0.chunk_max.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// The file cap for chunked assemblies, in bytes.
|
||||
pub(crate) fn chunk_file_cap(&self) -> u64 {
|
||||
self.0.chunk_file_cap.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// Replaces the chunk limits (the 4 MiB chunk cap and the 1 GiB file cap by
|
||||
/// default) with the ones the tests need; read before the router is built.
|
||||
pub fn set_chunk_limits(&self, chunk_max: usize, chunk_file_cap: u64) {
|
||||
self.0.chunk_max.store(chunk_max, Ordering::Relaxed);
|
||||
self.0.chunk_file_cap.store(chunk_file_cap, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// The per-vault live-bytes quota, `None` when unlimited.
|
||||
pub(crate) fn quota_limit(&self) -> Option<u64> {
|
||||
self.0.quota_bytes
|
||||
}
|
||||
|
||||
/// The pairing lockout, shared by pairing and invite redemption.
|
||||
pub(crate) fn lockout(&self) -> &Lockout {
|
||||
&self.0.lockout
|
||||
}
|
||||
|
||||
/// The live rooms of this server.
|
||||
pub(crate) fn live(&self) -> &live::Registry {
|
||||
&self.0.live
|
||||
}
|
||||
|
||||
/// Replaces the windows the live endpoints work with; the tests shorten them.
|
||||
pub fn set_live_timings(&self, timings: live::Timings) {
|
||||
self.0.live.set_timings(timings);
|
||||
}
|
||||
|
||||
/// Drops the peers of a live room that have not been heard from since
|
||||
/// `now_ms`; answers their client ids.
|
||||
#[must_use]
|
||||
pub fn prune_live_presence(&self, vault: &str, path: &str, now_ms: u64) -> Vec<String> {
|
||||
self.0.live.prune_presence(vault, path, now_ms)
|
||||
}
|
||||
|
||||
pub(crate) fn is_admin(&self, token: &str) -> bool {
|
||||
ct_eq(&hash_hex(token.as_bytes()), &self.0.admin_hash)
|
||||
}
|
||||
|
||||
/// The rate limiter of the anonymous publish routes.
|
||||
pub(crate) fn publish_budget(&self) -> &publish::budget::Budget {
|
||||
&self.0.publish_budget
|
||||
}
|
||||
|
||||
/// Runs `f` with the store, under the state lock; for read-only helpers and
|
||||
/// tests that need the store behind a server.
|
||||
pub fn with_store<T>(&self, f: impl FnOnce(&Store) -> T) -> T {
|
||||
let core = self.core();
|
||||
f(&core.store)
|
||||
}
|
||||
}
|
||||
|
||||
/// Constant-time equality of two equally long hex strings.
|
||||
fn ct_eq(a: &str, b: &str) -> bool {
|
||||
a.len() == b.len() && a.bytes().zip(b.bytes()).fold(0u8, |acc, (x, y)| acc | (x ^ y)) == 0
|
||||
}
|
||||
|
||||
/// Milliseconds since the epoch.
|
||||
#[must_use]
|
||||
pub fn now_ms() -> u64 {
|
||||
SystemTime::now().duration_since(UNIX_EPOCH).ok().and_then(|d| u64::try_from(d.as_millis()).ok()).unwrap_or(0)
|
||||
}
|
||||
|
||||
/// `bytes` random bytes from the OS as lowercase hex.
|
||||
///
|
||||
/// # Errors
|
||||
/// When `/dev/urandom` cannot be read.
|
||||
pub fn random_hex(bytes: usize) -> Result<String, ApiError> {
|
||||
let mut buffer = vec![0u8; bytes];
|
||||
std::fs::File::open("/dev/urandom").and_then(|mut f| f.read_exact(&mut buffer))?;
|
||||
Ok(hash_hex(&buffer).chars().take(bytes * 2).collect())
|
||||
}
|
||||
|
||||
/// True for a vault id of 1..=64 ASCII letters, digits or `-`.
|
||||
#[must_use]
|
||||
pub fn valid_vault_id(id: &str) -> bool {
|
||||
!id.is_empty() && id.len() <= 64 && id.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-')
|
||||
}
|
||||
|
||||
/// Writes a blob once (content-addressed, so an existing file is already right).
|
||||
///
|
||||
/// # Errors
|
||||
/// I/O errors.
|
||||
pub fn store_blob(path: &Path, bytes: &[u8]) -> std::io::Result<()> {
|
||||
if path.exists() {
|
||||
return Ok(());
|
||||
}
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
craft_atomic::atomic_write(path, bytes).map_err(|e| std::io::Error::other(e.to_string()))
|
||||
}
|
||||
|
||||
/// The bearer token of an `Authorization` header, when well formed.
|
||||
pub(crate) fn bearer(headers: &HeaderMap) -> Option<&str> {
|
||||
headers.get("authorization")?.to_str().ok()?.strip_prefix("Bearer ")
|
||||
}
|
||||
|
||||
/// 401 unless the caller carries the admin token.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 when the token is wrong, 500 on database errors.
|
||||
pub(crate) fn require_admin(state: &AppState, headers: &HeaderMap) -> Result<(), ApiError> {
|
||||
if bearer(headers).is_some_and(|token| state.is_admin(token)) { Ok(()) } else { Err(ApiError::unauthorized()) }
|
||||
}
|
||||
|
||||
/// The `(device id, user id)` behind the caller's device token; updates the device's
|
||||
/// `last_seen_ms` (at most once a minute).
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 for a missing or wrong token, 500 on database errors.
|
||||
pub(crate) fn require_user(state: &AppState, headers: &HeaderMap) -> Result<(String, String), ApiError> {
|
||||
let token = bearer(headers).ok_or_else(ApiError::unauthorized)?;
|
||||
let token_hash = hash_hex(token.as_bytes());
|
||||
let core = state.core();
|
||||
let Some(found) = core.store.device_for(&token_hash)? else {
|
||||
return Err(ApiError::unauthorized());
|
||||
};
|
||||
core.store.touch_device(&token_hash, now_ms())?;
|
||||
Ok(found)
|
||||
}
|
||||
|
||||
/// Checks the token, the user's membership in `vault` and the vault itself:
|
||||
/// a viewer may only `Read`, a non-member or a deleted vault is `404` (existence is
|
||||
/// never revealed), and a member acting above their role is `403`.
|
||||
///
|
||||
/// # Errors
|
||||
/// 401, 403, 404 as above, 500 on database errors.
|
||||
pub(crate) fn authorize(state: &AppState, headers: &HeaderMap, vault: &str, need: Need) -> Result<Member, ApiError> {
|
||||
let token = bearer(headers).ok_or_else(ApiError::unauthorized)?;
|
||||
let token_hash = hash_hex(token.as_bytes());
|
||||
let core = state.core();
|
||||
let Some((device, user)) = core.store.device_for(&token_hash)? else {
|
||||
return Err(ApiError::unauthorized());
|
||||
};
|
||||
let role = core.store.member_role(vault, &user)?.and_then(|raw| Role::parse(&raw));
|
||||
if !core.store.vault_live(vault)? {
|
||||
return Err(ApiError::missing("no such vault"));
|
||||
}
|
||||
let Some(role) = role else { return Err(ApiError::missing("no such vault")) };
|
||||
if !role.satisfies(need) {
|
||||
return Err(ApiError::forbidden());
|
||||
}
|
||||
core.store.touch_device(&token_hash, now_ms())?;
|
||||
Ok(Member { device, user, role })
|
||||
}
|
||||
|
||||
/// The device id and membership of the caller in `vault`, at read level; the old
|
||||
/// per-vault device check, now a thin wrapper around [`authorize`].
|
||||
///
|
||||
/// # Errors
|
||||
/// 401 for a wrong token, 404 for a foreign token, a deleted vault or a non-member,
|
||||
/// 403 for a viewer (they may read, so readers pass), 500 on database errors.
|
||||
pub(crate) fn require_device(state: &AppState, headers: &HeaderMap, vault: &str) -> Result<String, ApiError> {
|
||||
Ok(authorize(state, headers, vault, Need::Read)?.device)
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
//! Appending and reading the `audit` table, newest first in listings.
|
||||
|
||||
use rusqlite::params;
|
||||
|
||||
use super::{Store, int, uint};
|
||||
|
||||
/// Most audit rows one query may return.
|
||||
pub const MAX_AUDIT_ROWS: usize = 500;
|
||||
|
||||
/// One row of the audit log, newest first in listings.
|
||||
#[derive(Debug, serde::Serialize)]
|
||||
pub struct AuditRow {
|
||||
/// Monotonic row id, in insertion order.
|
||||
pub id: i64,
|
||||
/// When the event happened.
|
||||
pub ts_ms: u64,
|
||||
/// What happened, one of the kinds in [`crate::api::audit`].
|
||||
pub kind: String,
|
||||
/// Vault id the event touched, when there was one.
|
||||
pub vault: Option<String>,
|
||||
/// Device id the event touched, when there was one.
|
||||
pub device: Option<String>,
|
||||
/// Free-form context; never a token, code or hash.
|
||||
pub detail: Option<String>,
|
||||
}
|
||||
|
||||
impl Store {
|
||||
/// Appends one audit row; `vault`, `device` and `detail` are optional context.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn audit(&self, ts_ms: u64, kind: &str, vault: Option<&str>, device: Option<&str>, detail: Option<&str>) -> rusqlite::Result<()> {
|
||||
self.conn
|
||||
.execute("INSERT INTO audit (ts_ms, kind, vault, device, detail) VALUES (?1, ?2, ?3, ?4, ?5)", params![int(ts_ms)?, kind, vault, device, detail])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Audit rows, newest first, capped at [`MAX_AUDIT_ROWS`].
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn audit_rows(&self, limit: usize) -> rusqlite::Result<Vec<AuditRow>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, ts_ms, kind, vault, device, detail FROM audit ORDER BY id DESC LIMIT ?1")?;
|
||||
let rows = statement.query_map(params![limit.clamp(1, MAX_AUDIT_ROWS)], |row| {
|
||||
Ok(AuditRow { id: row.get(0)?, ts_ms: uint(row.get(1)?)?, kind: row.get(2)?, vault: row.get(3)?, device: row.get(4)?, detail: row.get(5)? })
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
//! Chunk rows: the uploaded chunk records and the `file_chunks` reference map that
|
||||
//! ties an assembled file to its chunks, for dedup, quota and GC.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
#[cfg(test)]
|
||||
use rusqlite::Connection;
|
||||
use rusqlite::params;
|
||||
|
||||
use super::{Store, int, uint};
|
||||
|
||||
/// Most chunk rows one GC pass removes; the next pass takes the rest (per-request cap).
|
||||
const MAX_GC_ROWS: u64 = 5000;
|
||||
|
||||
impl Store {
|
||||
/// Records a stored chunk of `vault`; true when the row is new (201 was stored,
|
||||
/// false means it already existed, 200).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn insert_chunk(&self, vault: &str, hash: &str, size: u64, created_ms: u64) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute(
|
||||
"INSERT OR IGNORE INTO chunks (vault, hash, size, created_ms) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![vault, hash, int(size)?, int(created_ms)?],
|
||||
)?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Records the ordered chunk list of an assembled file, replacing any previous
|
||||
/// mapping of the same file hash; every chunk becomes referenced by it.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn record_file_chunks(&self, vault: &str, file_hash: &str, chunk_hashes: &[String]) -> rusqlite::Result<()> {
|
||||
let transaction = self.conn.unchecked_transaction()?;
|
||||
transaction.execute("DELETE FROM file_chunks WHERE vault = ?1 AND file_hash = ?2", params![vault, file_hash])?;
|
||||
{
|
||||
let mut insert = transaction.prepare_cached("INSERT INTO file_chunks (vault, file_hash, idx, chunk_hash) VALUES (?1, ?2, ?3, ?4)")?;
|
||||
for (idx, chunk_hash) in chunk_hashes.iter().enumerate() {
|
||||
let idx = u64::try_from(idx).unwrap_or(u64::MAX);
|
||||
insert.execute(params![vault, file_hash, int(idx)?, chunk_hash])?;
|
||||
}
|
||||
}
|
||||
transaction.commit()
|
||||
}
|
||||
|
||||
/// Deletes the `file_chunks` rows of `vault` whose assembled file is referenced by
|
||||
/// neither a record (a live file, or a tombstone still inside `grace_ms`) nor a
|
||||
/// retained revision; those chunks become collectible in the next pass.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn gc_file_chunks(&self, vault: &str, now_ms: u64, grace_ms: u64) -> rusqlite::Result<usize> {
|
||||
self.conn.execute(
|
||||
"DELETE FROM file_chunks WHERE vault = ?1 AND file_hash NOT IN (
|
||||
SELECT hash FROM files WHERE vault = ?1 AND (deleted = 0 OR updated_ms > ?2)
|
||||
UNION SELECT hash FROM file_revisions WHERE vault = ?1)",
|
||||
params![vault, int(now_ref(now_ms, grace_ms))?],
|
||||
)
|
||||
}
|
||||
|
||||
/// The chunk rows GC may remove: referenced by no `file_chunks` row any more and
|
||||
/// older than `grace_ms`, at most [`MAX_GC_ROWS`] of them.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub(crate) fn collectible_chunks(&self, vault: &str, now_ms: u64, grace_ms: u64) -> rusqlite::Result<Vec<(String, u64)>> {
|
||||
let mut statement = self.conn.prepare(
|
||||
"SELECT c.hash, c.size FROM chunks c
|
||||
WHERE c.vault = ?1 AND c.created_ms <= ?2
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM file_chunks fc
|
||||
WHERE fc.vault = c.vault AND fc.chunk_hash = c.hash
|
||||
AND fc.file_hash IN (
|
||||
SELECT hash FROM files WHERE vault = ?1 AND (deleted = 0 OR updated_ms > ?2)
|
||||
UNION SELECT hash FROM file_revisions WHERE vault = ?1))
|
||||
ORDER BY c.created_ms
|
||||
LIMIT ?3",
|
||||
)?;
|
||||
let rows = statement
|
||||
.query_map(params![vault, int(now_ref(now_ms, grace_ms))?, int(MAX_GC_ROWS)?], |row| Ok((row.get::<_, String>(0)?, uint(row.get::<_, i64>(1)?)?)))?
|
||||
.collect::<rusqlite::Result<Vec<(String, u64)>>>()?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// Every chunk hash recorded for a vault.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn chunk_hashes(&self, vault: &str) -> rusqlite::Result<HashSet<String>> {
|
||||
let mut statement = self.conn.prepare("SELECT hash FROM chunks WHERE vault = ?1")?;
|
||||
statement.query_map(params![vault], |row| row.get::<_, String>(0))?.collect()
|
||||
}
|
||||
|
||||
/// Removes the named chunk rows of `vault` (their files are gone or going).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn drop_chunks(&self, vault: &str, hashes: &[String]) -> rusqlite::Result<()> {
|
||||
let transaction = self.conn.unchecked_transaction()?;
|
||||
{
|
||||
let mut drop_row = transaction.prepare_cached("DELETE FROM chunks WHERE vault = ?1 AND hash = ?2")?;
|
||||
for hash in hashes {
|
||||
drop_row.execute(params![vault, hash])?;
|
||||
}
|
||||
}
|
||||
transaction.commit()
|
||||
}
|
||||
}
|
||||
|
||||
/// A chunk's mtime cutoff: `now - grace`, never overflowing backwards.
|
||||
/// A chunk counts as past its grace when `now_ms - created_ms >= grace_ms`, which is
|
||||
/// the same as `created_ms <= now_ms - grace_ms` over the saturating subtraction.
|
||||
#[must_use]
|
||||
fn now_ref(now_ms: u64, grace_ms: u64) -> u64 {
|
||||
now_ms.saturating_sub(grace_ms)
|
||||
}
|
||||
|
||||
/// Whether a `file_chunks` row references `chunk_hash` in `conn`, for tests.
|
||||
#[cfg(test)]
|
||||
fn referenced(conn: &Connection, vault: &str, chunk_hash: &str) -> bool {
|
||||
let mut statement = conn.prepare("SELECT 1 FROM file_chunks WHERE vault = ?1 AND chunk_hash = ?2").expect("select");
|
||||
statement.exists(params![vault, chunk_hash]).unwrap_or_default()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use notescraft_sync::hash_hex;
|
||||
|
||||
#[test]
|
||||
fn chunk_rows_record_once_and_collect_once() {
|
||||
let store = Store::in_memory().expect("db");
|
||||
store.create_vault("v", 1).expect("vault");
|
||||
let (hash, other) = (hash_hex(b"one"), hash_hex(b"two"));
|
||||
assert!(store.insert_chunk("v", &hash, 3, 100).expect("insert"), "the first insert stores");
|
||||
assert!(!store.insert_chunk("v", &hash, 3, 100).expect("again"), "a second insert is idempotent");
|
||||
store.record_file_chunks("v", &hash_hex(b"file"), &[other.clone(), hash.clone()]).expect("map");
|
||||
store.record_file_chunks("v", &hash_hex(b"file"), std::slice::from_ref(&hash)).expect("re-map replaces");
|
||||
assert!(referenced(&store.conn, "v", &hash), "one remains referenced");
|
||||
assert!(!referenced(&store.conn, "v", &other), "the replaced list dropped the other");
|
||||
store
|
||||
.conn
|
||||
.execute(
|
||||
"INSERT INTO files (vault, path, hash, size, mtime_ms, seq, deleted, updated_ms) VALUES ('v', 'a', ?1, 10, 1, 1, 0, 0)",
|
||||
params![hash_hex(b"file")],
|
||||
)
|
||||
.expect("the live file row the map points at");
|
||||
|
||||
// A referenced chunk is never collectible, however old the clock says it is; a
|
||||
// fresh chunk stays until it ages past the grace.
|
||||
assert_eq!(store.collectible_chunks("v", 1012, 0).expect("collect"), Vec::<(String, u64)>::new(), "still referenced");
|
||||
assert_eq!(store.collectible_chunks("v", 110, 1_000_000).expect("collect"), Vec::<(String, u64)>::new());
|
||||
|
||||
// Three stale uploads age out; the mapped one is kept by its live file.
|
||||
let stale = [hash_hex(b"gone one"), hash_hex(b"gone two"), hash_hex(b"gone three")];
|
||||
for (idx, stale_hash) in stale.iter().enumerate() {
|
||||
store.insert_chunk("v", stale_hash, 1, u64::try_from(idx).expect("small")).expect("insert");
|
||||
}
|
||||
store.gc_file_chunks("v", 110, 1_000_000).expect("gc keeps the mapped file");
|
||||
let expected: Vec<(String, u64)> = stale.iter().map(|hash| (hash.clone(), 1)).collect();
|
||||
assert_eq!(store.collectible_chunks("v", 4000, 1).expect("collect"), expected, "unreferenced and aged, oldest first");
|
||||
let dropped: Vec<String> = expected.iter().map(|(hash, _)| hash.clone()).collect();
|
||||
store.drop_chunks("v", &dropped).expect("row drop");
|
||||
assert_eq!(store.chunk_hashes("v").expect("rows"), HashSet::from([hash.clone()]), "the referenced row remains");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunks_of_dead_files_become_collectible_through_their_records_and_revisions() {
|
||||
use rusqlite::params;
|
||||
|
||||
let store = Store::in_memory().expect("db");
|
||||
store.create_vault("v", 1).expect("vault");
|
||||
let chunk = hash_hex(b"c");
|
||||
let file = hash_hex(b"file");
|
||||
store.insert_chunk("v", &chunk, 1, 0).expect("insert");
|
||||
store.record_file_chunks("v", &file, std::slice::from_ref(&chunk)).expect("map");
|
||||
store
|
||||
.conn
|
||||
.execute("INSERT INTO files (vault, path, hash, size, mtime_ms, seq, deleted, updated_ms) VALUES ('v', 'a', ?1, 1, 1, 1, 1, 100)", params![file])
|
||||
.expect("tombstone row");
|
||||
// A tombstone inside its grace keeps the chunks.
|
||||
store.gc_file_chunks("v", 1_000, 5_000).expect("gc");
|
||||
assert!(referenced(&store.conn, "v", &chunk));
|
||||
// Past the grace they are collectible.
|
||||
store.gc_file_chunks("v", 1_000, 50).expect("gc");
|
||||
assert_eq!(store.collectible_chunks("v", 1_000, 50).expect("collect").len(), 1, "the chunk and its row go");
|
||||
// A retained revision holds its chunk alive.
|
||||
store.record_file_chunks("v", &file, std::slice::from_ref(&chunk)).expect("map");
|
||||
store.conn.execute("INSERT INTO file_revisions (vault, path, seq, hash, size, mtime_ms, editor_user, deleted, recorded_ms) VALUES ('v', 'a', 1, ?1, 1, 1, NULL, 1, 100)", params![file]).expect("revision row");
|
||||
store.gc_file_chunks("v", 10_000, 0).expect("gc");
|
||||
assert!(referenced(&store.conn, "v", &chunk), "revisions keep their chunks");
|
||||
assert_eq!(store.collectible_chunks("v", 10_000, 0).expect("collect"), Vec::<(String, u64)>::new());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
//! Device rows: pairing, lookup, listing, token rotation and revocation.
|
||||
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::{Store, int, uint};
|
||||
|
||||
/// `devices.last_seen_ms` moves at most this many milliseconds apart.
|
||||
const LAST_SEEN_UPDATE_MS: u64 = 60 * 1000;
|
||||
|
||||
/// One row of a device listing: `(id, name, created_ms, last_seen_ms)`;
|
||||
/// `last_seen_ms` is `None` for a device the server never saw.
|
||||
pub type DeviceRow = (String, String, u64, Option<u64>);
|
||||
|
||||
impl Store {
|
||||
/// Adds a device with its token hash and its user.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn add_device(&self, id: &str, vault: &str, user_id: &str, name: &str, token_hash: &str, now_ms: u64) -> rusqlite::Result<()> {
|
||||
self.conn.execute(
|
||||
"INSERT INTO devices (id, vault, user_id, name, token_hash, created_ms) VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
|
||||
params![id, vault, user_id, name, token_hash, int(now_ms)?],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The `(device id, user id)` a token hash belongs to; revoked tokens match nothing.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn device_for(&self, token_hash: &str) -> rusqlite::Result<Option<(String, String)>> {
|
||||
self.conn
|
||||
.query_row("SELECT id, user_id FROM devices WHERE token_hash = ?1 AND revoked = 0", params![token_hash], |r| Ok((r.get(0)?, r.get(1)?)))
|
||||
.optional()
|
||||
}
|
||||
|
||||
/// Which user owns the live device `id` of `vault`, if any.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn device_user(&self, vault: &str, id: &str) -> rusqlite::Result<Option<String>> {
|
||||
self.conn.query_row("SELECT user_id FROM devices WHERE id = ?1 AND vault = ?2 AND revoked = 0", params![id, vault], |r| r.get(0)).optional()
|
||||
}
|
||||
|
||||
/// Updates a device's `last_seen_ms`, at most once a minute per device.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn touch_device(&self, token_hash: &str, now_ms: u64) -> rusqlite::Result<()> {
|
||||
self.conn.execute(
|
||||
"UPDATE devices SET last_seen_ms = ?2 WHERE token_hash = ?1 AND (last_seen_ms IS NULL OR (last_seen_ms + ?3) <= ?2)",
|
||||
params![token_hash, int(now_ms)?, int(LAST_SEEN_UPDATE_MS)?],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A vault's devices as `(id, name, created_ms, last_seen_ms)`; revoked ones are hidden.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn list_devices(&self, vault: &str) -> rusqlite::Result<Vec<DeviceRow>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, name, created_ms, last_seen_ms FROM devices WHERE vault = ?1 AND revoked = 0")?;
|
||||
let rows = statement.query_map(params![vault], |row| {
|
||||
let last_seen: Option<i64> = row.get(3)?;
|
||||
Ok((row.get(0)?, row.get(1)?, uint(row.get(2)?)?, last_seen.map(uint).transpose()?))
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// Replaces a device's token hash; the old token stops authenticating at once.
|
||||
/// Returns false when the device is not a live device of the vault.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn rotate_device(&self, id: &str, vault: &str, token_hash: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE devices SET token_hash = ?3 WHERE id = ?1 AND vault = ?2 AND revoked = 0", params![id, vault, token_hash])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Marks a device revoked. Returns false when it is not a live device of the vault.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn revoke_device(&self, id: &str, vault: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE devices SET revoked = 1 WHERE id = ?1 AND vault = ?2 AND revoked = 0", params![id, vault])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
//! File records: persistence reads, live bytes, GC protection and the head sequence.
|
||||
//! Writes go through `revisions::record_write`, which records the revision with them.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use notescraft_sync::FileRecord;
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::{Store, uint};
|
||||
|
||||
impl Store {
|
||||
/// Every stored record of a vault.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn load_records(&self, vault: &str) -> rusqlite::Result<Vec<FileRecord>> {
|
||||
let mut statement = self.conn.prepare("SELECT path, hash, size, mtime_ms, seq, deleted FROM files WHERE vault = ?1")?;
|
||||
let rows = statement.query_map(params![vault], |r| {
|
||||
Ok(FileRecord {
|
||||
path: r.get(0)?,
|
||||
hash: r.get(1)?,
|
||||
size: uint(r.get(2)?)?,
|
||||
mtime_ms: uint(r.get(3)?)?,
|
||||
seq: uint(r.get(4)?)?,
|
||||
deleted: r.get::<_, i64>(5)? != 0,
|
||||
})
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// The vault's live bytes: the sum of `size` over non-deleted records.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn live_bytes(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row("SELECT COALESCE(SUM(size), 0) FROM files WHERE vault = ?1 AND deleted = 0", params![vault], |row| uint(row.get(0)?))
|
||||
}
|
||||
|
||||
/// The head sequence of a vault: the highest `seq` any record reached (a read-only
|
||||
/// SQL maximum, so listing many vaults never loads their ledgers).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn head_seq(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row("SELECT COALESCE(MAX(seq), 0) FROM files WHERE vault = ?1", params![vault], |row| uint(row.get(0)?))
|
||||
}
|
||||
|
||||
/// The last write of a vault: the newest `updated_ms`, or the vault's creation.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn last_change_ms(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row(
|
||||
"SELECT COALESCE(MAX(updated_ms), (SELECT created_ms FROM vaults WHERE id = ?1)) FROM files WHERE vault = ?1",
|
||||
params![vault],
|
||||
|row| uint(row.get(0)?),
|
||||
)
|
||||
}
|
||||
|
||||
/// Authorship of a live file: `(editor user id, seq, mtime_ms)`; `editor_user` is
|
||||
/// `None` for rows written before accounts existed.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn file_meta(&self, vault: &str, path: &str) -> rusqlite::Result<Option<(Option<String>, u64, u64)>> {
|
||||
self.conn
|
||||
.query_row("SELECT editor_user, seq, mtime_ms FROM files WHERE vault = ?1 AND path = ?2 AND deleted = 0", params![vault, path], |row| {
|
||||
Ok((row.get(0)?, uint(row.get(1)?)?, uint(row.get(2)?)?))
|
||||
})
|
||||
.optional()
|
||||
}
|
||||
|
||||
/// Hashes GC must leave alone: live records and tombstones younger than `grace_ms`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn protected_hashes(&self, vault: &str, now_ms: u64, grace_ms: u64) -> rusqlite::Result<HashSet<String>> {
|
||||
let mut statement = self.conn.prepare("SELECT hash, deleted, updated_ms FROM files WHERE vault = ?1")?;
|
||||
let rows = statement.query_map(params![vault], |row| Ok((row.get::<_, String>(0)?, row.get::<_, i64>(1)?, row.get::<_, i64>(2)?)))?;
|
||||
let mut keep = HashSet::new();
|
||||
for row in rows {
|
||||
let (hash, deleted, updated) = row?;
|
||||
let in_grace = now_ms.saturating_sub(uint(updated)?) < grace_ms;
|
||||
if deleted == 0 || in_grace {
|
||||
keep.insert(hash);
|
||||
}
|
||||
}
|
||||
Ok(keep)
|
||||
}
|
||||
|
||||
/// Hashes of tombstones past their grace: the one part of a vault's content the
|
||||
/// retention window must not keep alive, so the GC un-protects it again.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn expired_tombstone_hashes(&self, vault: &str, now_ms: u64, grace_ms: u64) -> rusqlite::Result<HashSet<String>> {
|
||||
let mut statement = self.conn.prepare("SELECT DISTINCT hash, updated_ms FROM files WHERE vault = ?1 AND deleted = 1")?;
|
||||
let rows = statement.query_map(params![vault], |row| Ok((row.get::<_, String>(0)?, row.get::<_, i64>(1)?)))?;
|
||||
let mut expired = HashSet::new();
|
||||
for row in rows {
|
||||
let (hash, updated) = row?;
|
||||
if now_ms.saturating_sub(uint(updated)?) >= grace_ms {
|
||||
expired.insert(hash);
|
||||
}
|
||||
}
|
||||
Ok(expired)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
//! Invites: single-use codes an owner mint for a vault, hashed like pairing codes.
|
||||
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::{Store, int, uint};
|
||||
|
||||
impl Store {
|
||||
/// Stores an issued invite: an id to list and delete it by, and the code hash.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn add_invite(&self, id: &str, vault: &str, role: &str, code_hash: &str, created_ms: u64, expires_ms: u64) -> rusqlite::Result<()> {
|
||||
self.conn.execute(
|
||||
"INSERT INTO invites (id, vault, role, code_hash, created_ms, expires_ms) VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
|
||||
params![id, vault, role, code_hash, int(created_ms)?, int(expires_ms)?],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Consumes an invite code: returns its `(vault, role)` when it exists and has not
|
||||
/// expired, and deletes it either way (a code works once, even twice is refused).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn take_invite(&self, code_hash: &str, now_ms: u64) -> rusqlite::Result<Option<(String, String)>> {
|
||||
let row: Option<(String, String, i64)> = self
|
||||
.conn
|
||||
.query_row("SELECT vault, role, expires_ms FROM invites WHERE code_hash = ?1", params![code_hash], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))
|
||||
.optional()?;
|
||||
self.conn.execute("DELETE FROM invites WHERE code_hash = ?1 OR expires_ms < ?2", params![code_hash, int(now_ms)?])?;
|
||||
Ok(row.filter(|(_, _, expires)| u64::try_from(*expires).is_ok_and(|e| e >= now_ms)).map(|(vault, role, _)| (vault, role)))
|
||||
}
|
||||
|
||||
/// A vault's invites as `(id, role, created_ms, expires_ms)`; never the code or hash.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn list_invites(&self, vault: &str) -> rusqlite::Result<Vec<(String, String, u64, u64)>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, role, created_ms, expires_ms FROM invites WHERE vault = ?1 ORDER BY created_ms, id")?;
|
||||
let rows = statement.query_map(params![vault], |row| Ok((row.get(0)?, row.get(1)?, uint(row.get(2)?)?, uint(row.get(3)?)?)))?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// Deletes one invite of a vault; false when the id is unknown to the vault.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn delete_invite(&self, vault: &str, id: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("DELETE FROM invites WHERE vault = ?1 AND id = ?2", params![vault, id])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Every invite of a vault's rows are gone; used when a vault is purged.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn drop_vault_invites(&self, vault: &str) -> rusqlite::Result<()> {
|
||||
self.conn.execute("DELETE FROM invites WHERE vault = ?1", params![vault])?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,235 @@
|
||||
//! SQLite persistence: users, vaults, memberships, pairing codes, invites, devices,
|
||||
//! file records and their revisions. Only hashes of tokens, pairing and invite codes are
|
||||
//! stored.
|
||||
|
||||
pub(crate) mod users;
|
||||
|
||||
mod audit;
|
||||
mod chunks;
|
||||
mod devices;
|
||||
mod files;
|
||||
mod invites;
|
||||
mod pairings;
|
||||
mod publish;
|
||||
mod revisions;
|
||||
mod vaults;
|
||||
|
||||
pub use audit::AuditRow;
|
||||
pub use publish::{NewShareLink, PublicationRow, ShareLinkRow};
|
||||
pub use revisions::RevisionRow;
|
||||
pub use vaults::PURGE_AFTER_MS;
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use rusqlite::{Connection, params};
|
||||
|
||||
const SCHEMA: &str = "
|
||||
CREATE TABLE IF NOT EXISTS users (id TEXT PRIMARY KEY, name TEXT NOT NULL, color TEXT NOT NULL, created_ms INTEGER NOT NULL);
|
||||
CREATE TABLE IF NOT EXISTS vaults (id TEXT PRIMARY KEY, created_ms INTEGER NOT NULL, name TEXT, deleted_ms INTEGER, retention_days INTEGER NOT NULL DEFAULT 30);
|
||||
CREATE TABLE IF NOT EXISTS file_revisions (
|
||||
vault TEXT NOT NULL, path TEXT NOT NULL, seq INTEGER NOT NULL, hash TEXT NOT NULL,
|
||||
size INTEGER NOT NULL, mtime_ms INTEGER NOT NULL, editor_user TEXT, deleted INTEGER NOT NULL,
|
||||
recorded_ms INTEGER NOT NULL, PRIMARY KEY (vault, path, seq)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS members (
|
||||
vault TEXT NOT NULL, user TEXT NOT NULL,
|
||||
role TEXT NOT NULL CHECK (role IN ('owner','editor','viewer')),
|
||||
added_ms INTEGER NOT NULL, PRIMARY KEY (vault, user)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS invites (
|
||||
id TEXT PRIMARY KEY, vault TEXT NOT NULL, role TEXT NOT NULL CHECK (role IN ('editor','viewer')),
|
||||
code_hash TEXT NOT NULL, created_ms INTEGER NOT NULL, expires_ms INTEGER NOT NULL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS pairings (code_hash TEXT PRIMARY KEY, vault TEXT NOT NULL, user TEXT, expires_ms INTEGER NOT NULL);
|
||||
CREATE TABLE IF NOT EXISTS devices (
|
||||
id TEXT PRIMARY KEY, vault TEXT NOT NULL, name TEXT NOT NULL,
|
||||
token_hash TEXT NOT NULL UNIQUE, user_id TEXT, created_ms INTEGER NOT NULL,
|
||||
last_seen_ms INTEGER,
|
||||
revoked INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
vault TEXT NOT NULL, path TEXT NOT NULL, hash TEXT NOT NULL, size INTEGER NOT NULL,
|
||||
mtime_ms INTEGER NOT NULL, seq INTEGER NOT NULL, deleted INTEGER NOT NULL,
|
||||
updated_ms INTEGER NOT NULL DEFAULT 0, editor_user TEXT,
|
||||
PRIMARY KEY (vault, path)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS audit (
|
||||
id INTEGER PRIMARY KEY, ts_ms INTEGER NOT NULL, kind TEXT NOT NULL,
|
||||
vault TEXT, device TEXT, detail TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS chunks (vault TEXT NOT NULL, hash TEXT NOT NULL, size INTEGER NOT NULL, created_ms INTEGER NOT NULL, PRIMARY KEY (vault, hash));
|
||||
CREATE TABLE IF NOT EXISTS file_chunks (vault TEXT NOT NULL, file_hash TEXT NOT NULL, idx INTEGER NOT NULL, chunk_hash TEXT NOT NULL, PRIMARY KEY (vault, file_hash, idx));
|
||||
CREATE TABLE IF NOT EXISTS health (id INTEGER PRIMARY KEY);
|
||||
CREATE TABLE IF NOT EXISTS publications (
|
||||
id TEXT PRIMARY KEY, vault TEXT NOT NULL, target TEXT NOT NULL,
|
||||
kind TEXT NOT NULL CHECK (kind IN ('note','folder')),
|
||||
created_ms INTEGER NOT NULL, device TEXT, UNIQUE (vault, target)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS share_links (
|
||||
id TEXT PRIMARY KEY, vault TEXT NOT NULL, path TEXT NOT NULL,
|
||||
token_hash TEXT NOT NULL UNIQUE, created_ms INTEGER NOT NULL,
|
||||
expires_ms INTEGER NOT NULL, revoked INTEGER NOT NULL DEFAULT 0, device TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS activity (
|
||||
id INTEGER PRIMARY KEY, vault TEXT NOT NULL, ts_ms INTEGER NOT NULL,
|
||||
kind TEXT NOT NULL, actor TEXT NOT NULL, audience TEXT NOT NULL,
|
||||
note TEXT, thread_id TEXT, line INTEGER
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS activity_mentions (event_id INTEGER NOT NULL, user TEXT NOT NULL, PRIMARY KEY (event_id, user));
|
||||
CREATE TABLE IF NOT EXISTS activity_read (vault TEXT NOT NULL, user TEXT NOT NULL, event_id INTEGER NOT NULL, read_ms INTEGER NOT NULL, PRIMARY KEY (vault, user, event_id));
|
||||
CREATE INDEX IF NOT EXISTS activity_vault_id ON activity (vault, id);
|
||||
CREATE TRIGGER IF NOT EXISTS activity_vault_purge AFTER DELETE ON vaults BEGIN
|
||||
DELETE FROM activity WHERE vault = OLD.id;
|
||||
DELETE FROM activity_read WHERE vault = OLD.id;
|
||||
END;
|
||||
CREATE TRIGGER IF NOT EXISTS activity_event_purge AFTER DELETE ON activity BEGIN
|
||||
DELETE FROM activity_mentions WHERE event_id = OLD.id;
|
||||
END;
|
||||
";
|
||||
|
||||
/// The database handle; one per server, used under the state lock.
|
||||
pub struct Store {
|
||||
conn: Connection,
|
||||
}
|
||||
|
||||
pub(crate) fn int(value: u64) -> rusqlite::Result<i64> {
|
||||
i64::try_from(value).map_err(|_| rusqlite::Error::IntegralValueOutOfRange(0, i64::MAX))
|
||||
}
|
||||
|
||||
pub(crate) fn uint(value: i64) -> rusqlite::Result<u64> {
|
||||
u64::try_from(value).map_err(|_| rusqlite::Error::IntegralValueOutOfRange(0, value))
|
||||
}
|
||||
|
||||
impl Store {
|
||||
/// Opens (creating if needed) the database file.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn open(path: &Path) -> rusqlite::Result<Self> {
|
||||
Self::init(Connection::open(path)?)
|
||||
}
|
||||
|
||||
/// A throwaway in-memory database.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn in_memory() -> rusqlite::Result<Self> {
|
||||
Self::init(Connection::open_in_memory()?)
|
||||
}
|
||||
|
||||
fn init(conn: Connection) -> rusqlite::Result<Self> {
|
||||
conn.execute_batch("PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;")?;
|
||||
conn.execute_batch(SCHEMA)?;
|
||||
Self::migrate_columns(&conn)?;
|
||||
Self::migrate_users(&conn)?;
|
||||
publish::migrate(&conn)?;
|
||||
Ok(Self { conn })
|
||||
}
|
||||
|
||||
/// The database handle, for the modules that keep their own rows.
|
||||
pub(crate) fn conn(&self) -> &Connection {
|
||||
&self.conn
|
||||
}
|
||||
|
||||
/// Adds columns that databases created before them miss; idempotent, so it may run
|
||||
/// on every start.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn migrate_columns(conn: &Connection) -> rusqlite::Result<()> {
|
||||
for (table, column, decl) in [
|
||||
("vaults", "name", "TEXT"),
|
||||
("vaults", "deleted_ms", "INTEGER"),
|
||||
("devices", "last_seen_ms", "INTEGER"),
|
||||
("devices", "revoked", "INTEGER NOT NULL DEFAULT 0"),
|
||||
("devices", "user_id", "TEXT"),
|
||||
("pairings", "user", "TEXT"),
|
||||
("files", "updated_ms", "INTEGER NOT NULL DEFAULT 0"),
|
||||
("files", "editor_user", "TEXT"),
|
||||
("vaults", "retention_days", "INTEGER NOT NULL DEFAULT 30"),
|
||||
("vaults", "publish_slug", "TEXT"),
|
||||
] {
|
||||
if !Self::has_column(conn, table, column)? {
|
||||
// Table and column are program constants, never input.
|
||||
conn.execute(&format!("ALTER TABLE {table} ADD COLUMN {column} {decl}"), [])?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn has_column(conn: &Connection, table: &str, column: &str) -> rusqlite::Result<bool> {
|
||||
let mut statement = conn.prepare(&format!("PRAGMA table_info({table})"))?;
|
||||
let names = statement.query_map([], |row| row.get::<_, String>(1))?.collect::<rusqlite::Result<Vec<String>>>()?;
|
||||
Ok(names.iter().any(|name| name == column))
|
||||
}
|
||||
|
||||
/// Devices paired before accounts existed get one user each, named after the
|
||||
/// device, which becomes the `owner` of the device's pairing vault; every token
|
||||
/// keeps working. Idempotent: migrated devices have `user_id` set, so a restart
|
||||
/// skips them. A user created but not yet linked is reused, never duplicated.
|
||||
/// The user id is derived from the device id, so a re-run cannot invent a second one.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
fn migrate_users(conn: &Connection) -> rusqlite::Result<()> {
|
||||
let mut statement = conn.prepare("SELECT id, vault, name FROM devices WHERE user_id IS NULL AND revoked = 0")?;
|
||||
let pending = statement
|
||||
.query_map([], |row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?, row.get::<_, String>(2)?)))?
|
||||
.collect::<rusqlite::Result<Vec<(String, String, String)>>>()?;
|
||||
let created = migration_now_ms();
|
||||
for (device, vault, name) in pending {
|
||||
let user = format!("u-{device}");
|
||||
conn.execute(
|
||||
"INSERT OR IGNORE INTO users (id, name, color, created_ms) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![user, name, palette_color(&name), int(created)?],
|
||||
)?;
|
||||
conn.execute("INSERT OR IGNORE INTO members (vault, user, role, added_ms) VALUES (?1, ?2, 'owner', ?3)", params![vault, user, int(created)?])?;
|
||||
conn.execute("UPDATE devices SET user_id = ?2 WHERE id = ?1", params![device, user])?;
|
||||
for (kind, vault) in [("user_created", None), ("member_added", Some(vault.as_str()))] {
|
||||
conn.execute("INSERT INTO audit (ts_ms, kind, vault, detail) VALUES (?1, ?2, ?3, ?4)", params![int(created)?, kind, vault, user])?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Checks the database accepts writes: insert plus delete inside a transaction that
|
||||
/// is rolled back after the returns.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn health(&self) -> rusqlite::Result<()> {
|
||||
let transaction = self.conn.unchecked_transaction()?;
|
||||
transaction.execute("INSERT INTO health DEFAULT VALUES", [])?;
|
||||
transaction.execute("DELETE FROM health", [])?;
|
||||
Ok(())
|
||||
// `transaction` drops uncommitted, which rolls it back.
|
||||
}
|
||||
}
|
||||
|
||||
/// Milliseconds since the epoch, for stamping rows that the migration adds.
|
||||
fn migration_now_ms() -> u64 {
|
||||
std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).ok().and_then(|d| u64::try_from(d.as_millis()).ok()).unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Picks the colour palette entry a display name hashes to, for users created without
|
||||
/// an explicit colour. Catppuccin Mocha accents.
|
||||
#[must_use]
|
||||
pub(crate) fn palette_color(name: &str) -> &'static str {
|
||||
const PALETTE: [&str; 14] = [
|
||||
"#f5e0dc", "#f2cdcd", "#f5c2e7", "#cba6f7", "#f38ba8", "#eba0ac", "#fab387", "#f9e2af", "#a6e3a1", "#94e2d5", "#89dceb", "#74c7ec", "#89b4fa",
|
||||
"#b4befe",
|
||||
];
|
||||
let sum: usize = name.bytes().map(usize::from).fold(0, |acc, byte| acc.wrapping_add(byte).wrapping_mul(31));
|
||||
PALETTE[sum % PALETTE.len()]
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn schema_creates_every_table_twice() {
|
||||
Store::in_memory().expect("db");
|
||||
Store::in_memory().expect("second run is idempotent");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
//! Pairing codes: admin-issued, bound to a vault and optionally to a user. Only the
|
||||
//! code's hash is stored; a code works once.
|
||||
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::{Store, int};
|
||||
|
||||
impl Store {
|
||||
/// Stores a pairing code hash for a vault, optionally bound to a user. A `user`
|
||||
/// binding of `None` creates a new user when the code is redeemed.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn add_pairing(&self, code_hash: &str, vault: &str, user: Option<&str>, expires_ms: u64) -> rusqlite::Result<()> {
|
||||
self.conn
|
||||
.execute("INSERT INTO pairings (code_hash, vault, user, expires_ms) VALUES (?1, ?2, ?3, ?4)", params![code_hash, vault, user, int(expires_ms)?])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Consumes a pairing code: returns its `(vault, user binding)` when it exists and
|
||||
/// has not expired, and deletes it either way so a code works once.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn take_pairing(&self, code_hash: &str, now_ms: u64) -> rusqlite::Result<Option<(String, Option<String>)>> {
|
||||
let row: Option<(String, Option<String>, i64)> = self
|
||||
.conn
|
||||
.query_row("SELECT vault, user, expires_ms FROM pairings WHERE code_hash = ?1", params![code_hash], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))
|
||||
.optional()?;
|
||||
self.conn.execute("DELETE FROM pairings WHERE code_hash = ?1 OR expires_ms < ?2", params![code_hash, int(now_ms)?])?;
|
||||
Ok(row.filter(|(_, _, expires)| u64::try_from(*expires).is_ok_and(|e| e >= now_ms)).map(|(vault, user, _)| (vault, user)))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,277 @@
|
||||
//! Published pages and share links: the rows that decide what the anonymous routes
|
||||
//! serve, and the vault slug a publication lives under.
|
||||
//!
|
||||
//! A share link stores the SHA-256 of its token, never the token itself.
|
||||
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
use super::{Store, int, uint};
|
||||
|
||||
/// One published note or folder.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PublicationRow {
|
||||
/// The row id.
|
||||
pub id: String,
|
||||
/// The vault it belongs to.
|
||||
pub vault: String,
|
||||
/// The vault path it publishes, `notes/a.md` or a folder path.
|
||||
pub target: String,
|
||||
/// `note` or `folder`.
|
||||
pub kind: String,
|
||||
/// When it was published.
|
||||
pub created_ms: u64,
|
||||
}
|
||||
|
||||
/// A share link that is about to be recorded.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct NewShareLink<'a> {
|
||||
/// The row id.
|
||||
pub id: &'a str,
|
||||
/// The vault it belongs to.
|
||||
pub vault: &'a str,
|
||||
/// The note path it opens.
|
||||
pub path: &'a str,
|
||||
/// The SHA-256 of the token; the token itself is never stored.
|
||||
pub token_hash: &'a str,
|
||||
/// When it was minted.
|
||||
pub created_ms: u64,
|
||||
/// When it stops working.
|
||||
pub expires_ms: u64,
|
||||
/// The device that minted it.
|
||||
pub device: Option<&'a str>,
|
||||
}
|
||||
|
||||
/// One share link.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ShareLinkRow {
|
||||
/// The row id.
|
||||
pub id: String,
|
||||
/// The vault it belongs to.
|
||||
pub vault: String,
|
||||
/// The note path it opens.
|
||||
pub path: String,
|
||||
/// When it was minted.
|
||||
pub created_ms: u64,
|
||||
/// When it stops working.
|
||||
pub expires_ms: u64,
|
||||
/// True once it was revoked.
|
||||
pub revoked: bool,
|
||||
}
|
||||
|
||||
/// Adds the publish index; idempotent, so it may run on every start.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn migrate(conn: &Connection) -> rusqlite::Result<()> {
|
||||
conn.execute("CREATE UNIQUE INDEX IF NOT EXISTS vaults_publish_slug ON vaults (publish_slug)", [])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl Store {
|
||||
/// The slug a vault publishes under, when it has one.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn vault_slug(&self, vault: &str) -> rusqlite::Result<Option<String>> {
|
||||
match self.conn.query_row("SELECT publish_slug FROM vaults WHERE id = ?1", params![vault], |row| row.get::<_, Option<String>>(0)) {
|
||||
Ok(slug) => Ok(slug),
|
||||
Err(rusqlite::Error::QueryReturnedNoRows) => Ok(None),
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
}
|
||||
|
||||
/// The live vault a slug names, when there is one.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn vault_by_slug(&self, slug: &str) -> rusqlite::Result<Option<String>> {
|
||||
self.conn.query_row("SELECT id FROM vaults WHERE publish_slug = ?1 AND deleted_ms IS NULL", params![slug], |row| row.get(0)).optional()
|
||||
}
|
||||
|
||||
/// Gives a vault its publish slug; false when the slug is taken by another vault
|
||||
/// or the vault is unknown. Setting the slug a vault already has succeeds.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn set_vault_slug(&self, vault: &str, slug: &str) -> rusqlite::Result<bool> {
|
||||
if self.vault_slug(vault)?.as_deref() == Some(slug) {
|
||||
return Ok(true);
|
||||
}
|
||||
let changed = self.conn.execute(
|
||||
"UPDATE vaults SET publish_slug = ?2 WHERE id = ?1 \
|
||||
AND NOT EXISTS (SELECT 1 FROM vaults other WHERE other.publish_slug = ?2 AND other.id <> ?1)",
|
||||
params![vault, slug],
|
||||
)?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Records one publication.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error; a duplicate target in the same vault is a constraint error.
|
||||
pub fn create_publication(&self, id: &str, vault: &str, target: &str, kind: &str, created_ms: u64, device: Option<&str>) -> rusqlite::Result<()> {
|
||||
self.conn.execute(
|
||||
"INSERT INTO publications (id, vault, target, kind, created_ms, device) VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
|
||||
params![id, vault, target, kind, int(created_ms)?, device],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every publication of a vault, oldest first.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn publications(&self, vault: &str) -> rusqlite::Result<Vec<PublicationRow>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, vault, target, kind, created_ms FROM publications WHERE vault = ?1 ORDER BY created_ms, id")?;
|
||||
let rows = statement.query_map(params![vault], |row| {
|
||||
Ok(PublicationRow { id: row.get(0)?, vault: row.get(1)?, target: row.get(2)?, kind: row.get(3)?, created_ms: uint(row.get(4)?)? })
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// One publication of a vault, when it exists.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn publication(&self, vault: &str, id: &str) -> rusqlite::Result<Option<PublicationRow>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, vault, target, kind, created_ms FROM publications WHERE vault = ?1 AND id = ?2")?;
|
||||
let mut rows = statement.query_map(params![vault, id], |row| {
|
||||
Ok(PublicationRow { id: row.get(0)?, vault: row.get(1)?, target: row.get(2)?, kind: row.get(3)?, created_ms: uint(row.get(4)?)? })
|
||||
})?;
|
||||
match rows.next() {
|
||||
Some(row) => Ok(Some(row?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// The publication of a vault that publishes a target, when there is one.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn publication_by_target(&self, vault: &str, target: &str) -> rusqlite::Result<Option<PublicationRow>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, vault, target, kind, created_ms FROM publications WHERE vault = ?1 AND target = ?2")?;
|
||||
let mut rows = statement.query_map(params![vault, target], |row| {
|
||||
Ok(PublicationRow { id: row.get(0)?, vault: row.get(1)?, target: row.get(2)?, kind: row.get(3)?, created_ms: uint(row.get(4)?)? })
|
||||
})?;
|
||||
match rows.next() {
|
||||
Some(row) => Ok(Some(row?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes one publication; false when the vault has no such publication.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn delete_publication(&self, vault: &str, id: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("DELETE FROM publications WHERE vault = ?1 AND id = ?2", params![vault, id])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// How many publications a vault has.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn publication_count(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
let count: i64 = self.conn.query_row("SELECT COUNT(*) FROM publications WHERE vault = ?1", params![vault], |row| row.get(0))?;
|
||||
Ok(u64::try_from(count).unwrap_or(0))
|
||||
}
|
||||
|
||||
/// Records one share link; only the hash of its token is stored.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn create_share_link(&self, link: &NewShareLink<'_>) -> rusqlite::Result<()> {
|
||||
self.conn.execute(
|
||||
"INSERT INTO share_links (id, vault, path, token_hash, created_ms, expires_ms, revoked, device) VALUES (?1, ?2, ?3, ?4, ?5, ?6, 0, ?7)",
|
||||
params![link.id, link.vault, link.path, link.token_hash, int(link.created_ms)?, int(link.expires_ms)?, link.device],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every share link of a vault, oldest first.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn share_links(&self, vault: &str) -> rusqlite::Result<Vec<ShareLinkRow>> {
|
||||
let mut statement =
|
||||
self.conn.prepare("SELECT id, vault, path, created_ms, expires_ms, revoked FROM share_links WHERE vault = ?1 ORDER BY created_ms, id")?;
|
||||
let rows = statement.query_map(params![vault], |row| {
|
||||
Ok(ShareLinkRow {
|
||||
id: row.get(0)?,
|
||||
vault: row.get(1)?,
|
||||
path: row.get(2)?,
|
||||
created_ms: uint(row.get(3)?)?,
|
||||
expires_ms: uint(row.get(4)?)?,
|
||||
revoked: row.get::<_, i64>(5)? != 0,
|
||||
})
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// One share link of a vault, when it exists.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn share_link(&self, vault: &str, id: &str) -> rusqlite::Result<Option<ShareLinkRow>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, vault, path, created_ms, expires_ms, revoked FROM share_links WHERE vault = ?1 AND id = ?2")?;
|
||||
let mut rows = statement.query_map(params![vault, id], |row| {
|
||||
Ok(ShareLinkRow {
|
||||
id: row.get(0)?,
|
||||
vault: row.get(1)?,
|
||||
path: row.get(2)?,
|
||||
created_ms: uint(row.get(3)?)?,
|
||||
expires_ms: uint(row.get(4)?)?,
|
||||
revoked: row.get::<_, i64>(5)? != 0,
|
||||
})
|
||||
})?;
|
||||
match rows.next() {
|
||||
Some(row) => Ok(Some(row?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// The live share link a token hash names: not revoked and not expired at `now_ms`.
|
||||
/// The clock is a parameter, so a test can expire a link without sleeping.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn share_link_by_token(&self, token_hash: &str, now_ms: u64) -> rusqlite::Result<Option<ShareLinkRow>> {
|
||||
let mut statement = self.conn.prepare(
|
||||
"SELECT id, vault, path, created_ms, expires_ms, revoked FROM share_links \
|
||||
WHERE token_hash = ?1 AND revoked = 0 AND expires_ms > ?2",
|
||||
)?;
|
||||
let mut rows = statement.query_map(params![token_hash, int(now_ms)?], |row| {
|
||||
Ok(ShareLinkRow {
|
||||
id: row.get(0)?,
|
||||
vault: row.get(1)?,
|
||||
path: row.get(2)?,
|
||||
created_ms: uint(row.get(3)?)?,
|
||||
expires_ms: uint(row.get(4)?)?,
|
||||
revoked: row.get::<_, i64>(5)? != 0,
|
||||
})
|
||||
})?;
|
||||
match rows.next() {
|
||||
Some(row) => Ok(Some(row?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Revokes one share link; false when the vault has no such link.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn revoke_share_link(&self, vault: &str, id: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE share_links SET revoked = 1 WHERE vault = ?1 AND id = ?2", params![vault, id])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// How many share links a vault has.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn share_link_count(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
let count: i64 = self.conn.query_row("SELECT COUNT(*) FROM share_links WHERE vault = ?1", params![vault], |row| row.get(0))?;
|
||||
Ok(u64::try_from(count).unwrap_or(0))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
//! File revisions: persistence of every applied write, history listings, the per-vault
|
||||
//! retention GC and the bytes quota reads.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use notescraft_sync::FileRecord;
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
use serde::Serialize;
|
||||
|
||||
use super::{Store, int, uint};
|
||||
use crate::history::retention::DEFAULT_RETENTION_DAYS;
|
||||
|
||||
/// One stored revision of one path: what a write did and when the server recorded it.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize)]
|
||||
pub struct RevisionRow {
|
||||
/// The revision number, unique per (vault, path).
|
||||
pub seq: u64,
|
||||
/// The content size in bytes (0 for a tombstone).
|
||||
pub size: u64,
|
||||
/// The writer's modification time.
|
||||
pub mtime_ms: u64,
|
||||
/// The user id of the writer, `None` for rows written before accounts existed.
|
||||
pub editor_user: Option<String>,
|
||||
/// The display name of `editor_user`, when there was one.
|
||||
pub editor_name: Option<String>,
|
||||
/// True for a tombstone; its `hash` still names the content a restore would bring back.
|
||||
pub deleted: bool,
|
||||
/// When the server recorded the revision.
|
||||
pub recorded_ms: u64,
|
||||
/// The content hash; not sent in listings.
|
||||
#[serde(skip)]
|
||||
pub(crate) hash: String,
|
||||
}
|
||||
|
||||
/// The history listing: revision rows plus, for records written before revisions were
|
||||
/// kept, the live `files` row standing in for its own sequence number. Column order
|
||||
/// matches [`row`]: `NULL` fills the hash the listing never needs.
|
||||
const LIST_SQL: &str = "
|
||||
SELECT seq, size, mtime_ms, editor_user, editor_name, deleted, recorded_ms, NULL FROM (
|
||||
SELECT r.seq AS seq, r.size AS size, r.mtime_ms AS mtime_ms, r.editor_user AS editor_user,
|
||||
u.name AS editor_name, r.deleted AS deleted, r.recorded_ms AS recorded_ms, NULL AS hash
|
||||
FROM file_revisions r LEFT JOIN users u ON u.id = r.editor_user
|
||||
WHERE r.vault = ?1 AND r.path = ?2
|
||||
UNION ALL
|
||||
SELECT f.seq, f.size, f.mtime_ms, f.editor_user, u.name, f.deleted, f.updated_ms, NULL
|
||||
FROM files f LEFT JOIN users u ON u.id = f.editor_user
|
||||
WHERE f.vault = ?1 AND f.path = ?2
|
||||
AND NOT EXISTS (SELECT 1 FROM file_revisions r2 WHERE r2.vault = f.vault AND r2.path = f.path AND r2.seq = f.seq)
|
||||
) ORDER BY seq DESC LIMIT ?3";
|
||||
|
||||
/// Expired revisions are those beyond the retention window that are neither the current
|
||||
/// one nor among the newest [`MIN_KEEP_REVISIONS`] of their file.
|
||||
const GC_SQL: &str = "
|
||||
DELETE FROM file_revisions AS r
|
||||
WHERE r.vault = ?1 AND r.recorded_ms < ?2
|
||||
AND r.seq IS NOT COALESCE((SELECT f.seq FROM files f WHERE f.vault = r.vault AND f.path = r.path), -1)
|
||||
AND (SELECT COUNT(*) FROM file_revisions n WHERE n.vault = r.vault AND n.path = r.path AND n.seq > r.seq) >= ?3";
|
||||
|
||||
/// The vault's counted bytes: union of each live record's and each retained revision's
|
||||
/// size, with a content hash counted once.
|
||||
/// The vault's counted bytes: unique content of live records plus retained revisions.
|
||||
/// Chunked files count as their referenced chunks' bytes, never as the assembled blob
|
||||
/// too — the module's choice, documented in [`crate::chunks`].
|
||||
const USED_SQL: &str = "
|
||||
SELECT COALESCE(SUM(size), 0) FROM (
|
||||
SELECT DISTINCT c.hash AS hash, c.size AS size FROM chunks c
|
||||
WHERE c.vault = ?1 AND EXISTS (
|
||||
SELECT 1 FROM file_chunks fc
|
||||
WHERE fc.vault = c.vault AND fc.chunk_hash = c.hash AND fc.file_hash IN (
|
||||
SELECT hash FROM files WHERE vault = ?1
|
||||
UNION SELECT hash FROM file_revisions WHERE vault = ?1))
|
||||
UNION
|
||||
SELECT DISTINCT hash, size FROM files
|
||||
WHERE vault = ?1 AND deleted = 0 AND NOT EXISTS (
|
||||
SELECT 1 FROM file_chunks x WHERE x.vault = files.vault AND x.file_hash = files.hash)
|
||||
UNION
|
||||
SELECT DISTINCT hash, size FROM file_revisions
|
||||
WHERE vault = ?1 AND NOT EXISTS (
|
||||
SELECT 1 FROM file_chunks x WHERE x.vault = file_revisions.vault AND x.file_hash = file_revisions.hash)
|
||||
)";
|
||||
|
||||
/// Maps one revision query row; the last column is the `NULL` placeholder where a hash
|
||||
/// was not selected.
|
||||
fn revision_row(r: &rusqlite::Row<'_>) -> rusqlite::Result<RevisionRow> {
|
||||
Ok(RevisionRow {
|
||||
seq: uint(r.get(0)?)?,
|
||||
size: uint(r.get(1)?)?,
|
||||
mtime_ms: uint(r.get(2)?)?,
|
||||
editor_user: r.get(3)?,
|
||||
editor_name: r.get(4)?,
|
||||
deleted: r.get::<_, i64>(5)? != 0,
|
||||
recorded_ms: uint(r.get(6)?)?,
|
||||
hash: r.get::<_, Option<String>>(7)?.unwrap_or_default(),
|
||||
})
|
||||
}
|
||||
|
||||
const FIND_REVISION_SQL: &str = "
|
||||
SELECT r.seq, r.size, r.mtime_ms, r.editor_user, u.name, r.deleted, r.recorded_ms, r.hash
|
||||
FROM file_revisions r LEFT JOIN users u ON u.id = r.editor_user
|
||||
WHERE r.vault = ?1 AND r.path = ?2 AND r.seq = ?3";
|
||||
|
||||
/// The files row of a path with no revision row stands in for its own sequence number.
|
||||
const FIND_STAND_IN_SQL: &str = "
|
||||
SELECT f.seq, f.size, f.mtime_ms, f.editor_user, u.name, f.deleted, f.updated_ms, f.hash
|
||||
FROM files f LEFT JOIN users u ON u.id = f.editor_user
|
||||
WHERE f.vault = ?1 AND f.path = ?2 AND f.seq = ?3";
|
||||
|
||||
impl Store {
|
||||
/// Persists an applied ledger write in one transaction: the `files` upsert with the
|
||||
/// editor stamped, and the `file_revisions` row recording the same record. On any
|
||||
/// error both stay unwritten, and the caller drops the vault's cached ledger.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn record_write(&self, vault: &str, record: &FileRecord, editor_user: &str, updated_ms: u64) -> rusqlite::Result<()> {
|
||||
let transaction = self.conn.unchecked_transaction()?;
|
||||
transaction.execute(
|
||||
"INSERT OR REPLACE INTO files (vault, path, hash, size, mtime_ms, seq, deleted, updated_ms, editor_user) \
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
|
||||
params![
|
||||
vault,
|
||||
record.path,
|
||||
record.hash,
|
||||
int(record.size)?,
|
||||
int(record.mtime_ms)?,
|
||||
int(record.seq)?,
|
||||
i64::from(record.deleted),
|
||||
int(updated_ms)?,
|
||||
editor_user
|
||||
],
|
||||
)?;
|
||||
transaction.execute(
|
||||
"INSERT OR REPLACE INTO file_revisions (vault, path, seq, hash, size, mtime_ms, editor_user, deleted, recorded_ms) \
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
|
||||
params![
|
||||
vault,
|
||||
record.path,
|
||||
int(record.seq)?,
|
||||
record.hash,
|
||||
int(record.size)?,
|
||||
int(record.mtime_ms)?,
|
||||
editor_user,
|
||||
i64::from(record.deleted),
|
||||
int(updated_ms)?
|
||||
],
|
||||
)?;
|
||||
transaction.commit()
|
||||
}
|
||||
|
||||
/// The revisions of one path, newest first, at most `limit` of them; the live `files`
|
||||
/// row of a record with no revision row (written before revisions were kept) stands
|
||||
/// in for itself.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn revisions(&self, vault: &str, path: &str, limit: u64) -> rusqlite::Result<Vec<RevisionRow>> {
|
||||
let mut statement = self.conn.prepare(LIST_SQL)?;
|
||||
statement.query_map(params![vault, path, int(limit)?], revision_row)?.collect()
|
||||
}
|
||||
|
||||
/// The stored revision of one (vault, path, seq): a `file_revisions` row, or the
|
||||
/// live `files` row for sequence numbers of records written before revisions were
|
||||
/// kept. `None` when the revision is unknown.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn find_revision(&self, vault: &str, path: &str, seq: u64) -> rusqlite::Result<Option<RevisionRow>> {
|
||||
let stored = self.conn.query_row(FIND_REVISION_SQL, params![vault, path, int(seq)?], revision_row).optional()?;
|
||||
if stored.is_some() {
|
||||
return Ok(stored);
|
||||
}
|
||||
self.conn.query_row(FIND_STAND_IN_SQL, params![vault, path, int(seq)?], revision_row).optional()
|
||||
}
|
||||
|
||||
/// Deletes the vault's revisions past [`MIN_KEEP_REVISIONS`] and its retention window;
|
||||
/// a revision that is also the file's current record always stays.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn gc_revisions(&self, vault: &str, cutoff_ms: u64, min_keep: u64) -> rusqlite::Result<()> {
|
||||
self.conn.execute(GC_SQL, params![vault, int(cutoff_ms)?, int(min_keep)?])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every retained revision's content hash of a vault; the GC reads it as the blobs
|
||||
/// it must keep.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn revision_hashes(&self, vault: &str) -> rusqlite::Result<HashSet<String>> {
|
||||
let mut statement = self.conn.prepare("SELECT DISTINCT hash FROM file_revisions WHERE vault = ?1")?;
|
||||
statement.query_map(params![vault], |r| r.get::<_, String>(0))?.collect()
|
||||
}
|
||||
|
||||
/// The vault's counted bytes: unique content of live records plus retained revisions.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn used_bytes(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row(USED_SQL, params![vault], |r| uint(r.get(0)?))
|
||||
}
|
||||
|
||||
/// The vault's retention window in days; the default when it set nothing else.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn retention_days(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row("SELECT COALESCE(retention_days, ?2) FROM vaults WHERE id = ?1", params![vault, int(DEFAULT_RETENTION_DAYS)?], |r| uint(r.get(0)?))
|
||||
}
|
||||
|
||||
/// Sets the vault's retention window in days; false when the vault is unknown.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn set_retention_days(&self, vault: &str, days: u64) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE vaults SET retention_days = ?2 WHERE id = ?1", params![vault, int(days)?])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use notescraft_sync::hash_hex;
|
||||
|
||||
/// A record at revision `seq` with the content `body`.
|
||||
fn record(path: &str, body: &str, seq: u64) -> FileRecord {
|
||||
FileRecord {
|
||||
path: path.to_owned(),
|
||||
hash: hash_hex(body.as_bytes()),
|
||||
size: u64::try_from(body.len()).unwrap_or(u64::MAX),
|
||||
mtime_ms: seq,
|
||||
seq,
|
||||
deleted: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// A user row, so the listing can join an editor name.
|
||||
fn user(store: &Store, id: &str, name: &str) {
|
||||
store.conn.execute("INSERT INTO users (id, name, color, created_ms) VALUES (?1, ?2, '#fff', 1)", params![id, name]).expect("user row");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_applied_write_records_a_revision_and_lists_it_newest_first() {
|
||||
let store = Store::in_memory().expect("db");
|
||||
user(&store, "u-1", "One");
|
||||
user(&store, "u-2", "Two");
|
||||
store.record_write("v", &record("a.md", "one", 1), "u-1", 100).expect("first write");
|
||||
store.record_write("v", &record("a.md", "two", 2), "u-2", 200).expect("second write");
|
||||
|
||||
let history = store.revisions("v", "a.md", 10).expect("listing");
|
||||
assert_eq!(history.len(), 2, "newest first");
|
||||
assert_eq!((history[0].seq, history[0].editor_name.as_deref()), (2, Some("Two")));
|
||||
assert_eq!((history[1].seq, history[1].editor_name.as_deref(), history[1].deleted), (1, Some("One"), false));
|
||||
assert_eq!(store.find_revision("v", "a.md", 1).expect("find").map(|r| r.hash), Some(record("a.md", "one", 1).hash));
|
||||
assert_eq!(store.find_revision("v", "a.md", 9).expect("find"), None, "unknown revision");
|
||||
assert_eq!(store.used_bytes("v").expect("bytes"), 6, "both distinct contents count");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_row_without_a_revision_stands_in_for_its_own_seq_and_gc_keeps_the_rest() {
|
||||
let store = Store::in_memory().expect("db");
|
||||
let first = record("a.md", "one", 1);
|
||||
let first_hash = first.hash.clone();
|
||||
// A record written before revisions were kept: only its `files` row exists.
|
||||
store
|
||||
.conn
|
||||
.execute(
|
||||
"INSERT INTO files (vault, path, hash, size, mtime_ms, seq, deleted, updated_ms) VALUES ('v', 'a.md', ?1, 3, 1, 1, 0, 100)",
|
||||
params![first.hash],
|
||||
)
|
||||
.expect("legacy row");
|
||||
assert_eq!(store.revisions("v", "a.md", 10).expect("listing").len(), 1, "the live row is listed");
|
||||
assert_eq!(store.find_revision("v", "a.md", 1).expect("find").map(|r| r.hash), Some(first_hash), "the files row answers for seq 1");
|
||||
assert_eq!(store.used_bytes("v").expect("bytes"), 3, "the only content counts");
|
||||
|
||||
// The same content again: its unique bytes must not count twice.
|
||||
store.record_write("v", &first, "u-1", 200).expect("write");
|
||||
assert_eq!(store.used_bytes("v").expect("bytes"), 3, "a hash counts once across live rows and revisions");
|
||||
store.record_write("v", &record("a.md", "three", 3), "u-1", 300).expect("write");
|
||||
|
||||
// Everything is past its (injected) window: the current one and the newest 20 stay.
|
||||
store.gc_revisions("v", 1_000, 20).expect("gc");
|
||||
assert_eq!(store.revisions("v", "a.md", 10).expect("listing").len(), 2, "min keep protects both");
|
||||
store.gc_revisions("v", 1_000, 1).expect("gc");
|
||||
assert_eq!(store.revisions("v", "a.md", 10).expect("listing").len(), 1, "seq 2 was beyond the newest 1");
|
||||
assert_eq!(store.used_bytes("v").expect("bytes"), 5, "the freed bytes belonged to the dropped hash");
|
||||
assert_eq!(store.revision_hashes("v").expect("hashes"), HashSet::from([record("a.md", "three", 3).hash]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retention_days_default_and_set() {
|
||||
let store = Store::in_memory().expect("db");
|
||||
store.create_vault("v", 1).expect("vault");
|
||||
assert_eq!(store.retention_days("v").expect("days"), DEFAULT_RETENTION_DAYS, "the default");
|
||||
assert!(store.set_retention_days("v", 365).expect("set"));
|
||||
assert_eq!(store.retention_days("v").expect("days"), 365);
|
||||
assert!(!store.set_retention_days("missing", 7).expect("set"), "unknown vault");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
//! Users and memberships: creation, role queries and last-owner protection.
|
||||
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::{Store, int, uint};
|
||||
|
||||
/// `(id, name, created_ms, deleted_ms)` of one vault.
|
||||
pub type VaultSummary = (String, String, u64, Option<u64>);
|
||||
/// `(user, name, color, role, last_seen_ms)` of one member.
|
||||
pub type MemberRow = (String, String, String, String, Option<u64>);
|
||||
|
||||
impl Store {
|
||||
/// Adds a user; every user id is new, so callers generate ids.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn create_user(&self, id: &str, name: &str, color: &str, now_ms: u64) -> rusqlite::Result<()> {
|
||||
self.conn.execute("INSERT INTO users (id, name, color, created_ms) VALUES (?1, ?2, ?3, ?4)", params![id, name, color, int(now_ms)?])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// True when a user row exists.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn user_exists(&self, id: &str) -> rusqlite::Result<bool> {
|
||||
Ok(self.conn.query_row("SELECT 1 FROM users WHERE id = ?1", params![id], |_| Ok(())).optional()?.is_some())
|
||||
}
|
||||
|
||||
/// A user's `(name, color)`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn user_name_color(&self, id: &str) -> rusqlite::Result<Option<(String, String)>> {
|
||||
self.conn.query_row("SELECT name, color FROM users WHERE id = ?1", params![id], |row| Ok((row.get(0)?, row.get(1)?))).optional()
|
||||
}
|
||||
|
||||
/// Adds `user` to `vault` with `role` unless a membership already exists (which is
|
||||
/// left as is). Returns `(effective role, was added)`.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn ensure_member(&self, vault: &str, user: &str, role: &str, now_ms: u64) -> rusqlite::Result<(String, bool)> {
|
||||
let changed = self
|
||||
.conn
|
||||
.execute("INSERT OR IGNORE INTO members (vault, user, role, added_ms) VALUES (?1, ?2, ?3, ?4)", params![vault, user, role, int(now_ms)?])?;
|
||||
let effective = self.member_role(vault, user)?.ok_or_else(|| rusqlite::Error::QueryReturnedNoRows)?;
|
||||
Ok((effective, changed == 1))
|
||||
}
|
||||
|
||||
/// The membership role of `user` in `vault`, when there is one.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn member_role(&self, vault: &str, user: &str) -> rusqlite::Result<Option<String>> {
|
||||
self.conn.query_row("SELECT role FROM members WHERE vault = ?1 AND user = ?2", params![vault, user], |r| r.get(0)).optional()
|
||||
}
|
||||
|
||||
/// Changes a member's role; false when the membership is missing.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn set_member_role(&self, vault: &str, user: &str, role: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE members SET role = ?3 WHERE vault = ?1 AND user = ?2", params![vault, user, role])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Removes a membership; false when there was none.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn remove_member(&self, vault: &str, user: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("DELETE FROM members WHERE vault = ?1 AND user = ?2", params![vault, user])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// How many owners a vault has left.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn owner_count(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row("SELECT COUNT(*) FROM members WHERE vault = ?1 AND role = 'owner'", params![vault], |row| uint(row.get(0)?))
|
||||
}
|
||||
|
||||
/// How many members a vault has.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn member_count(&self, vault: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row("SELECT COUNT(*) FROM members WHERE vault = ?1", params![vault], |row| uint(row.get(0)?))
|
||||
}
|
||||
|
||||
/// How many live vaults `user` owns.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn owned_vaults(&self, user: &str) -> rusqlite::Result<u64> {
|
||||
self.conn.query_row(
|
||||
"SELECT COUNT(*) FROM members m JOIN vaults v ON m.vault = v.id \
|
||||
WHERE m.user = ?1 AND m.role = 'owner' AND v.deleted_ms IS NULL",
|
||||
params![user],
|
||||
|row| uint(row.get(0)?),
|
||||
)
|
||||
}
|
||||
|
||||
/// The live vaults `user` is a member of, as `(id, name or id, role)`, by name.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn user_vaults(&self, user: &str) -> rusqlite::Result<Vec<(String, String, String)>> {
|
||||
let mut statement = self.conn.prepare(
|
||||
"SELECT v.id, COALESCE(v.name, v.id), m.role FROM members m \
|
||||
JOIN vaults v ON m.vault = v.id WHERE m.user = ?1 AND v.deleted_ms IS NULL ORDER BY v.name, v.id",
|
||||
)?;
|
||||
let rows = statement.query_map(params![user], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)))?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// Every vault, live and deleted, as `(id, name or id, created_ms, deleted_ms)`, by name.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn all_vaults(&self) -> rusqlite::Result<Vec<VaultSummary>> {
|
||||
let mut statement = self.conn.prepare("SELECT id, COALESCE(name, id), created_ms, deleted_ms FROM vaults ORDER BY COALESCE(name, id), id")?;
|
||||
let rows = statement.query_map([], |row| Ok((row.get(0)?, row.get(1)?, uint(row.get(2)?)?, row.get::<_, Option<i64>>(3)?.map(uint).transpose()?)))?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// A vault's members as `(user id, name, color, role, last seen)`, newest seen first;
|
||||
/// the last seen is the newest over the member's live devices.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn vault_members(&self, vault: &str) -> rusqlite::Result<Vec<MemberRow>> {
|
||||
let mut statement = self.conn.prepare(
|
||||
"SELECT m.user, u.name, u.color, m.role, \
|
||||
(SELECT MAX(last_seen_ms) FROM devices d WHERE d.user_id = m.user AND d.revoked = 0) \
|
||||
FROM members m JOIN users u ON u.id = m.user WHERE m.vault = ?1 ORDER BY m.role, u.name, m.user",
|
||||
)?;
|
||||
let rows = statement.query_map(params![vault], |row| {
|
||||
let last_seen: Option<i64> = row.get(4)?;
|
||||
Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, last_seen.map(uint).transpose()?))
|
||||
})?;
|
||||
rows.collect()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
//! Vault rows: creation, existence, names, soft delete, restore, purge bookkeeping
|
||||
//! and the per-vault listing queries.
|
||||
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::{Store, int};
|
||||
|
||||
/// A deleted vault and everything under it goes this long before a purge removes it.
|
||||
pub const PURGE_AFTER_MS: u64 = 14 * 24 * 60 * 60 * 1000;
|
||||
|
||||
impl Store {
|
||||
/// Registers a vault id (idempotent).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn create_vault(&self, id: &str, now_ms: u64) -> rusqlite::Result<()> {
|
||||
self.conn.execute("INSERT OR IGNORE INTO vaults (id, created_ms) VALUES (?1, ?2)", params![id, int(now_ms)?])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// True when the vault row exists (live or deleted).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn vault_exists(&self, id: &str) -> rusqlite::Result<bool> {
|
||||
Ok(self.conn.query_row("SELECT 1 FROM vaults WHERE id = ?1", params![id], |_| Ok(())).optional()?.is_some())
|
||||
}
|
||||
|
||||
/// True when the vault row exists and is not deleted (`None` for an unknown vault,
|
||||
/// `Some(false)` for a deleted one).
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn vault_live(&self, id: &str) -> rusqlite::Result<bool> {
|
||||
Ok(self.conn.query_row("SELECT 1 FROM vaults WHERE id = ?1 AND deleted_ms IS NULL", params![id], |_| Ok(())).optional()?.is_some())
|
||||
}
|
||||
|
||||
/// The display name of a vault, or the id when it is anonymous.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn vault_name(&self, id: &str) -> rusqlite::Result<Option<String>> {
|
||||
self.conn.query_row("SELECT COALESCE(name, id) FROM vaults WHERE id = ?1", params![id], |row| row.get(0)).optional()
|
||||
}
|
||||
|
||||
/// Sets the vault's display name; false when the vault is unknown.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn rename_vault(&self, id: &str, name: &str) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE vaults SET name = ?2 WHERE id = ?1", params![id, name])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Marks the vault deleted at `deleted_ms`; false when it is unknown or already deleted.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn delete_vault(&self, id: &str, deleted_ms: u64) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute("UPDATE vaults SET deleted_ms = ?2 WHERE id = ?1 AND deleted_ms IS NULL", params![id, int(deleted_ms)?])?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Clears the vault's deletion mark when it was deleted less than `within_ms` ago;
|
||||
/// false when the vault is unknown, live, or deleted too long ago. A clock behind
|
||||
/// the deletion is still inside the window.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn restore_vault(&self, id: &str, now_ms: u64, within_ms: u64) -> rusqlite::Result<bool> {
|
||||
let changed = self.conn.execute(
|
||||
"UPDATE vaults SET deleted_ms = NULL WHERE id = ?1 AND deleted_ms IS NOT NULL AND (?2 - deleted_ms) < ?3",
|
||||
params![id, int(now_ms)?, int(within_ms)?],
|
||||
)?;
|
||||
Ok(changed == 1)
|
||||
}
|
||||
|
||||
/// Live vaults marked deleted more than `after_ms` ago, oldest deletion first.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn purgeable_vaults(&self, now_ms: u64, after_ms: u64) -> rusqlite::Result<Vec<String>> {
|
||||
let cutoff = now_ms.saturating_sub(after_ms);
|
||||
let mut statement = self.conn.prepare("SELECT id FROM vaults WHERE deleted_ms IS NOT NULL AND deleted_ms < ?1 ORDER BY deleted_ms, id")?;
|
||||
let rows = statement.query_map(params![int(cutoff)?], |row| row.get::<_, String>(0))?;
|
||||
rows.collect()
|
||||
}
|
||||
|
||||
/// Removes a vault and everything under it: file records, memberships, invites,
|
||||
/// pairing codes and the vault row itself. The blob folder is the caller's job.
|
||||
///
|
||||
/// # Errors
|
||||
/// Any SQLite error.
|
||||
pub fn purge_vault(&self, id: &str) -> rusqlite::Result<()> {
|
||||
let transaction = self.conn.unchecked_transaction()?;
|
||||
for table in ["files", "file_revisions", "members", "invites", "pairings", "chunks", "file_chunks", "publications", "share_links", "vaults"] {
|
||||
let statement = Self::delete_by_vault(table);
|
||||
transaction.execute(&statement, params![id])?;
|
||||
}
|
||||
transaction.commit()
|
||||
}
|
||||
|
||||
/// A `DELETE FROM <table> WHERE vault = ?1` for tables that have the column; the
|
||||
/// vault row itself is keyed by `id`.
|
||||
fn delete_by_vault(table: &str) -> String {
|
||||
let column = if table == "vaults" { "id" } else { "vault" };
|
||||
format!("DELETE FROM {table} WHERE {column} = ?1")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
//! Accounts over real HTTP: legacy migration, pairing creates or reuses an account,
|
||||
//! `GET /v1/vaults` lists only the caller's memberships, and the ledger cache cap
|
||||
//! keeps every vault answering after eviction.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use notescraft_sync::hash_hex;
|
||||
use serde_json::json;
|
||||
|
||||
/// The legacy schema, before accounts existed (devices were bound to vaults directly).
|
||||
const LEGACY: &str = "
|
||||
CREATE TABLE vaults (id TEXT PRIMARY KEY, created_ms INTEGER NOT NULL);
|
||||
CREATE TABLE pairings (code_hash TEXT PRIMARY KEY, vault TEXT NOT NULL, expires_ms INTEGER NOT NULL);
|
||||
CREATE TABLE devices (id TEXT PRIMARY KEY, vault TEXT NOT NULL, name TEXT NOT NULL, token_hash TEXT NOT NULL UNIQUE, created_ms INTEGER NOT NULL);
|
||||
CREATE TABLE files (vault TEXT NOT NULL, path TEXT NOT NULL, hash TEXT NOT NULL, size INTEGER NOT NULL,
|
||||
mtime_ms INTEGER NOT NULL, seq INTEGER NOT NULL, deleted INTEGER NOT NULL, PRIMARY KEY (vault, path));
|
||||
";
|
||||
|
||||
#[test]
|
||||
fn migration_keeps_an_old_token_working_and_creates_an_owner() {
|
||||
let dir = temp_dir("accounts-legacy");
|
||||
let path = dir.join("sync.db");
|
||||
{
|
||||
let conn = rusqlite::Connection::open(&path).expect("db");
|
||||
conn.execute_batch(LEGACY).expect("legacy schema");
|
||||
let token = "legacy-device-token";
|
||||
conn.execute("INSERT INTO vaults (id, created_ms) VALUES ('v-old', 1)", []).expect("vault");
|
||||
conn.execute("INSERT INTO devices (id, vault, name, token_hash, created_ms) VALUES ('d-old', 'v-old', 'keeper', ?1, 1)", [hash_hex(token.as_bytes())])
|
||||
.expect("device");
|
||||
}
|
||||
let state = notescraft_sync_server::AppState::new(notescraft_sync_server::Store::open(&path).expect("open"), dir.join("blobs"), ADMIN);
|
||||
let server = common::start_with_state(state);
|
||||
|
||||
// The old token now belongs to a fresh account that owns the old vault.
|
||||
let (status, rows) = server.json("GET", "/v1/vaults", Some("legacy-device-token"), &[], None);
|
||||
assert_eq!(status, 200, "the legacy token must keep working");
|
||||
let rows = rows.as_array().expect("rows");
|
||||
assert_eq!(rows.len(), 1);
|
||||
assert_eq!(rows[0]["id"], "v-old");
|
||||
assert_eq!(rows[0]["role"], "owner");
|
||||
assert_eq!(rows[0]["name"], "v-old");
|
||||
|
||||
// The migrated user is named after the device and is the vault's owner.
|
||||
let (status, members) = server.json("GET", "/v1/vaults/v-old/members", Some("legacy-device-token"), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let members = members.as_array().expect("members");
|
||||
assert_eq!(members.len(), 1);
|
||||
assert_eq!(members[0]["name"], "keeper");
|
||||
assert_eq!(members[0]["role"], "owner");
|
||||
assert!(members[0]["user_id"].as_str().expect("user id").starts_with("u-"));
|
||||
|
||||
// Reading and writing through the legacy token is unchanged.
|
||||
assert_eq!(server.put("v-old", "legacy-device-token", "a.md", "x", 0).0, 201);
|
||||
assert!(server.audit().iter().any(|row| row["kind"] == "member_added"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pair_twice_with_the_same_user_leads_to_one_account() {
|
||||
let server = Server::start_temp("accounts-two-devices");
|
||||
let user = server.create_user("marc");
|
||||
let vault = Server::fresh_vault_id("two-devices");
|
||||
|
||||
let (status, phone) = server.pair_user(&user, &vault, "phone");
|
||||
assert_eq!(status, 200);
|
||||
let (status, laptop) = server.pair_user(&user, &vault, "laptop");
|
||||
assert_eq!(status, 200);
|
||||
assert_ne!(phone, laptop);
|
||||
|
||||
// Both devices belong to the same account and see exactly that account's vaults.
|
||||
let (_, mine_phone) = server.json("GET", "/v1/vaults", Some(&phone), &[], None);
|
||||
let (_, mine_laptop) = server.json("GET", "/v1/vaults", Some(&laptop), &[], None);
|
||||
assert_eq!(mine_phone, mine_laptop);
|
||||
assert_eq!(mine_phone.as_array().expect("rows").len(), 1);
|
||||
assert_eq!(mine_phone[0]["id"], vault);
|
||||
|
||||
let (_, devices) = server.json("GET", &format!("/v1/vaults/{vault}/devices"), Some(&phone), &[], None);
|
||||
assert_eq!(devices.as_array().expect("rows").len(), 2);
|
||||
|
||||
// A second pairing did not demote the existing membership: still one member row.
|
||||
let (_, members) = server.json("GET", &format!("/v1/vaults/{vault}/members"), Some(&phone), &[], None);
|
||||
assert_eq!(members.as_array().expect("members").len(), 1);
|
||||
|
||||
// `?user=` for a user that does not exist is refused.
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{}/pairing?user=u-missing", Server::fresh_vault_id("none")), Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn vault_listing_shows_only_my_memberships_sorted_by_name() {
|
||||
let server = Server::start_temp("accounts-twelve");
|
||||
let (bootstrap, token, _) = server.owned_vault("home");
|
||||
assert_eq!(server.json("PATCH", &format!("/v1/vaults/{bootstrap}"), Some(&token), &[], Some(&json!({"name": "00 home"}))).0, 204);
|
||||
|
||||
let wanted =
|
||||
["20 work", "10 personal", "30 course", "40 game jam", "50 cooking", "60 music", "70 travel", "80 reading", "90 garden", "95 blog", "99 archive"];
|
||||
let mut ids = vec![bootstrap.clone()];
|
||||
for name in wanted {
|
||||
let (status, vault) = server.make_vault(&token, name);
|
||||
assert_eq!(status, 200, "{name}");
|
||||
ids.push(vault);
|
||||
}
|
||||
assert_eq!(ids.len(), 12);
|
||||
|
||||
// Another account's vault never shows up.
|
||||
let (intruder_vault, _) = server.paired("intruder");
|
||||
let _ = intruder_vault;
|
||||
|
||||
let (_, mine) = server.json("GET", "/v1/vaults", Some(&token), &[], None);
|
||||
let rows = mine.as_array().expect("rows");
|
||||
assert_eq!(rows.len(), 12);
|
||||
let names: Vec<&str> = rows.iter().map(|row| row["name"].as_str().expect("name")).collect();
|
||||
let mut sorted = names.clone();
|
||||
sorted.sort_unstable();
|
||||
assert_eq!(names, sorted, "sorted by display name");
|
||||
assert!(!names.contains(&"intruder"), "no other user's vault");
|
||||
assert!(!names.contains(&"v-intruder"));
|
||||
for row in rows {
|
||||
assert_eq!(row["role"], "owner");
|
||||
assert_eq!(row["members"], 1);
|
||||
let _ = (row["bytes_used"].as_u64(), row["quota_bytes"].as_u64().unwrap_or(u64::MAX) > 0, row["head_seq"].as_u64(), row["last_change_ms"].as_u64());
|
||||
assert!(ids.contains(&row["id"].as_str().expect("id").to_owned()), "row {} of mine", row["id"]);
|
||||
}
|
||||
assert_eq!(rows.iter().find(|row| row["id"] == bootstrap.as_str()).map(|row| row["name"].as_str()), Some(Some("00 home")));
|
||||
|
||||
// An unknown token lists nothing: 401.
|
||||
assert_eq!(server.json("GET", "/v1/vaults", Some("nonsense"), &[], None).0, 401);
|
||||
assert_eq!(server.json("GET", "/v1/vaults", None, &[], None).0, 401);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn vault_names_are_trimmed_validated_and_capped_per_user() {
|
||||
let server = Server::start_temp("accounts-cap");
|
||||
let (bootstrap, token, _) = server.owned_vault("cap");
|
||||
assert_eq!(server.make_vault(&token, " ").0, 400, "empty names are refused");
|
||||
assert_eq!(server.make_vault(&token, "x".repeat(81).as_str()).0, 400);
|
||||
let (status, vault) = server.make_vault(&token, " padded ");
|
||||
assert_eq!(status, 200);
|
||||
let (_, mine) = server.json("GET", "/v1/vaults", Some(&token), &[], None);
|
||||
let row = mine.as_array().expect("rows").iter().find(|row| row["id"] == vault.as_str()).expect("row");
|
||||
assert_eq!(row["name"], "padded", "names are trimmed");
|
||||
|
||||
// The bootstrap vault plus 49 created reach the ownership cap; 51 is refused.
|
||||
for index in 0..48 {
|
||||
let (status, _) = server.make_vault(&token, &format!("vault {index:02}"));
|
||||
assert_eq!(status, 200, "vault {index}");
|
||||
}
|
||||
assert_eq!(server.make_vault(&token, "one too many").0, 409);
|
||||
let _ = bootstrap;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn seventy_vaults_stay_answerable_across_lru_evictions() {
|
||||
let server = Server::start_temp("accounts-lru");
|
||||
let (owner_vault, owner, _) = server.owned_vault("lru-owner");
|
||||
let mut vaults = vec![owner_vault.clone()];
|
||||
for index in 0..49 {
|
||||
let (status, vault) = server.make_vault(&owner, &format!("owned {index:02}"));
|
||||
assert_eq!(status, 200);
|
||||
vaults.push(vault);
|
||||
}
|
||||
// A second account adds 20 more, so the cap of 64 is passed together.
|
||||
let (other_vault, other) = server.paired("lru-other");
|
||||
vaults.push(other_vault);
|
||||
for index in 0..19 {
|
||||
let (status, vault) = server.make_vault(&other, &format!("theirs {index:02}"));
|
||||
assert_eq!(status, 200);
|
||||
vaults.push(vault);
|
||||
}
|
||||
assert_eq!(vaults.len(), 70);
|
||||
|
||||
// Touch all 70: loading walks past the cache cap, so older ledgers are evicted.
|
||||
for (index, vault) in vaults.iter().enumerate() {
|
||||
let token = if index < 50 { owner.as_str() } else { other.as_str() };
|
||||
let name = format!("note{index}.md");
|
||||
assert_eq!(server.put(vault, token, &name, "written", 0).0, 201, "vault {index}");
|
||||
let (status, page) = server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(token), &[], None);
|
||||
assert_eq!((status, page["records"].as_array().map(Vec::len)), (200, Some(1)), "vault {index}");
|
||||
}
|
||||
|
||||
// Every vault, including the earliest (long-evicted), still answers correctly.
|
||||
for (index, vault) in vaults.iter().enumerate() {
|
||||
let token = if index < 50 { owner.as_str() } else { other.as_str() };
|
||||
let (status, page) = server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(token), &[], None);
|
||||
assert_eq!((status, page["records"][0]["path"].as_str()), (200, Some(format!("note{index}.md").as_str())), "vault {index}");
|
||||
}
|
||||
let (_, mine) = server.json("GET", "/v1/vaults", Some(&owner), &[], None);
|
||||
assert_eq!(mine.as_array().expect("rows").len(), 50);
|
||||
assert_eq!(mine[0]["role"], "owner");
|
||||
let _ = owner_vault;
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
//! The activity feed over real HTTP: what is stored, who sees it, who may post
|
||||
//! and what the server never writes down.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::Server;
|
||||
use serde_json::json;
|
||||
|
||||
/// A vault with three members: the owner, an editor and a viewer.
|
||||
struct Team {
|
||||
vault: String,
|
||||
owner: String,
|
||||
owner_id: String,
|
||||
editor: String,
|
||||
editor_id: String,
|
||||
viewer: String,
|
||||
viewer_id: String,
|
||||
}
|
||||
|
||||
/// Builds a vault with an owner, an editor and a viewer.
|
||||
fn team(server: &Server, name: &str) -> Team {
|
||||
let (vault, owner, owner_id) = server.owned_vault(name);
|
||||
let (_, editor, editor_id) = server.owned_vault(name);
|
||||
let (_, viewer, viewer_id) = server.owned_vault(name);
|
||||
for (token, role) in [(&editor, "editor"), (&viewer, "viewer")] {
|
||||
let (status, code) = server.invite(&vault, &owner, role);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(token, &code).0, 200);
|
||||
}
|
||||
Team { vault, owner, owner_id, editor, editor_id, viewer, viewer_id }
|
||||
}
|
||||
|
||||
/// The feed of `token`, as `(status, body)`.
|
||||
fn feed(server: &Server, vault: &str, token: &str) -> (u16, serde_json::Value) {
|
||||
server.json("GET", &format!("/v1/vaults/{vault}/activity"), Some(token), &[], None)
|
||||
}
|
||||
|
||||
/// Posts one event of `kind` and answers the body of the server.
|
||||
fn post(server: &Server, vault: &str, token: &str, kind: &str, extra: &serde_json::Value) -> (u16, serde_json::Value) {
|
||||
let mut body = json!({ "kind": kind });
|
||||
if let (Some(target), Some(source)) = (body.as_object_mut(), extra.as_object()) {
|
||||
for (key, value) in source {
|
||||
target.insert(key.clone(), value.clone());
|
||||
}
|
||||
}
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/activity"), Some(token), &[], Some(&body))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_event_reaches_the_other_members_and_not_the_actor() {
|
||||
let server = Server::start_temp("activity-members");
|
||||
let team = team(&server, "members");
|
||||
let (status, _) = server.put(&team.vault, &team.owner, "note.md", "# Note\n", 0);
|
||||
assert_eq!(status, 201);
|
||||
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({ "note": "note.md", "thread_id": "c-1a2b" }));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
assert_eq!(reply["kind"], "comment");
|
||||
let id = reply["id"].as_u64().expect("id");
|
||||
assert_eq!(reply["head"], id);
|
||||
|
||||
// The actor does not see their own event.
|
||||
let (status, own) = feed(&server, &team.vault, &team.owner);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(own["events"].as_array().expect("events").len(), 0);
|
||||
assert_eq!(own["unread"], 0);
|
||||
|
||||
// The other members do.
|
||||
for token in [&team.editor, &team.viewer] {
|
||||
let (status, reply) = feed(&server, &team.vault, token.as_str());
|
||||
assert_eq!(status, 200);
|
||||
let events = reply["events"].as_array().expect("events");
|
||||
assert_eq!(events.len(), 1, "{reply}");
|
||||
assert_eq!(events[0]["kind"], "comment");
|
||||
assert_eq!(events[0]["note"], "note.md");
|
||||
assert_eq!(events[0]["thread_id"], "c-1a2b");
|
||||
assert_eq!(events[0]["actor"], team.owner_id);
|
||||
assert_eq!(events[0]["read"], false);
|
||||
assert_eq!(reply["unread"], 1);
|
||||
assert_eq!(reply["head"], id);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_mention_reaches_only_the_members_it_names() {
|
||||
let server = Server::start_temp("activity-mention");
|
||||
let team = team(&server, "mention");
|
||||
let (status, _) = server.put(&team.vault, &team.owner, "note.md", "# Note\n", 0);
|
||||
assert_eq!(status, 201);
|
||||
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "mention", &json!({ "note": "note.md", "mentions": [team.editor_id] }));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
assert_eq!(reply["recipients"], 1);
|
||||
|
||||
let (status, reply) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 200);
|
||||
let events = reply["events"].as_array().expect("events");
|
||||
assert_eq!(events.len(), 1, "{reply}");
|
||||
assert_eq!(events[0]["kind"], "mention");
|
||||
assert_eq!(reply["unread"], 1);
|
||||
|
||||
// The viewer was not mentioned: nothing reaches them.
|
||||
let (status, reply) = feed(&server, &team.vault, &team.viewer);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["events"].as_array().expect("events").len(), 0);
|
||||
assert_eq!(reply["unread"], 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_unread_count_follows_the_read_marking() {
|
||||
let server = Server::start_temp("activity-read");
|
||||
let team = team(&server, "read");
|
||||
let (status, first) = post(&server, &team.vault, &team.owner, "comment", &json!({}));
|
||||
assert_eq!(status, 200, "{first}");
|
||||
let (status, second) = post(&server, &team.vault, &team.owner, "comment", &json!({}));
|
||||
assert_eq!(status, 200, "{second}");
|
||||
let first_id = first["id"].as_u64().expect("id");
|
||||
|
||||
let (status, reply) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["unread"], 2);
|
||||
|
||||
// Marking one event read leaves the other one unread.
|
||||
let (status, marked) =
|
||||
server.json("POST", &format!("/v1/vaults/{}/activity/read", team.vault), Some(&team.editor), &[], Some(&json!({ "ids": [first_id] })));
|
||||
assert_eq!(status, 200, "{marked}");
|
||||
assert_eq!(marked["unread"], 1);
|
||||
|
||||
// Marking them all read empties the badge; the body may be empty.
|
||||
let (status, body) = server.send("POST", &format!("/v1/vaults/{}/activity/read", team.vault), Some(&team.editor), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let marked: serde_json::Value = serde_json::from_slice(&body).expect("json");
|
||||
assert_eq!(marked["unread"], 0);
|
||||
|
||||
let (status, reply) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["unread"], 0);
|
||||
for event in reply["events"].as_array().expect("events") {
|
||||
assert_eq!(event["read"], true, "{event}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_cursor_walks_the_feed() {
|
||||
let server = Server::start_temp("activity-cursor");
|
||||
let team = team(&server, "cursor");
|
||||
let mut ids = Vec::new();
|
||||
for _ in 0..3 {
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({}));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
ids.push(reply["id"].as_u64().expect("id"));
|
||||
}
|
||||
|
||||
let (status, reply) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["events"].as_array().expect("events").len(), 3);
|
||||
assert_eq!(reply["head"], ids[2]);
|
||||
assert_eq!(reply["more"], false);
|
||||
|
||||
// The cursor skips the events the client already has.
|
||||
let (status, reply) = server.json("GET", &format!("/v1/vaults/{}/activity?since={}", team.vault, ids[0]), Some(&team.editor), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["events"].as_array().expect("events").len(), 2);
|
||||
|
||||
// A page holds a few events and says that more are waiting.
|
||||
let (status, reply) = server.json("GET", &format!("/v1/vaults/{}/activity?limit=1", team.vault), Some(&team.editor), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["events"].as_array().expect("events").len(), 1);
|
||||
assert_eq!(reply["more"], true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_viewer_and_a_removed_member_are_refused() {
|
||||
let server = Server::start_temp("activity-roles");
|
||||
let team = team(&server, "roles");
|
||||
|
||||
// A viewer reads the feed but posts no event.
|
||||
let (status, _) = feed(&server, &team.vault, &team.viewer);
|
||||
assert_eq!(status, 200);
|
||||
let (status, reply) = post(&server, &team.vault, &team.viewer, "comment", &json!({}));
|
||||
assert_eq!(status, 403, "{reply}");
|
||||
|
||||
// The owner removes the editor.
|
||||
let (status, _) = server.send("DELETE", &format!("/v1/vaults/{}/members/{}", team.vault, team.editor_id), Some(&team.owner), &[], None);
|
||||
assert_eq!(status, 204);
|
||||
|
||||
// The removed member reads nothing and posts nothing.
|
||||
let (status, _) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 403);
|
||||
let (status, _) = post(&server, &team.vault, &team.editor, "comment", &json!({}));
|
||||
assert_eq!(status, 403);
|
||||
|
||||
// The feed of the other members did not change.
|
||||
let (status, reply) = feed(&server, &team.vault, &team.viewer);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["events"].as_array().expect("events").len(), 0);
|
||||
assert_eq!(reply["unread"], 0);
|
||||
|
||||
// Only the editor left the vault.
|
||||
let (status, members) = server.json("GET", &format!("/v1/vaults/{}/members", team.vault), Some(&team.owner), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let ids: Vec<String> = members.as_array().expect("members").iter().map(|row| row["user_id"].as_str().expect("user id").to_owned()).collect();
|
||||
assert!(ids.contains(&team.viewer_id), "{ids:?}");
|
||||
assert!(!ids.contains(&team.editor_id), "{ids:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_actor_is_the_user_of_the_token() {
|
||||
let server = Server::start_temp("activity-actor");
|
||||
let team = team(&server, "actor");
|
||||
|
||||
// A client cannot speak for somebody else.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({ "actor": team.editor_id }));
|
||||
assert_eq!(status, 403, "{reply}");
|
||||
|
||||
// The actor of the token is the actor of the event.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({ "actor": team.owner_id }));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let (status, reply) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 200);
|
||||
let events = reply["events"].as_array().expect("events");
|
||||
assert_eq!(events.len(), 1);
|
||||
assert_eq!(events[0]["actor"], team.owner_id);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_bad_event_is_refused() {
|
||||
let server = Server::start_temp("activity-bad");
|
||||
let team = team(&server, "bad");
|
||||
|
||||
// A kind the feed does not carry.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "gossip", &json!({}));
|
||||
assert_eq!(status, 400, "{reply}");
|
||||
|
||||
// A mention of somebody who is not a member.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "mention", &json!({ "mentions": ["u_stranger"] }));
|
||||
assert_eq!(status, 400, "{reply}");
|
||||
|
||||
// A mention that names nobody.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "mention", &json!({ "mentions": [] }));
|
||||
assert_eq!(status, 400, "{reply}");
|
||||
|
||||
// A note that is not in the vault.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({ "note": "missing.md" }));
|
||||
assert_eq!(status, 404, "{reply}");
|
||||
|
||||
// A path that leaves the vault.
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({ "note": "../escape.md" }));
|
||||
assert_eq!(status, 400, "{reply}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_server_never_writes_a_comment_body_to_the_log() {
|
||||
let server = Server::start_temp("activity-body");
|
||||
let team = team(&server, "body");
|
||||
let (status, _) = server.put(&team.vault, &team.owner, "note.md", "# Note\n", 0);
|
||||
assert_eq!(status, 201);
|
||||
let body = "the secret body of the comment";
|
||||
|
||||
let (status, reply) = post(&server, &team.vault, &team.owner, "comment", &json!({ "note": "note.md", "body": body }));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
|
||||
let audit = serde_json::to_string(&server.audit()).expect("audit");
|
||||
assert!(!audit.contains(body), "{audit}");
|
||||
assert!(audit.contains("activity_posted"), "{audit}");
|
||||
|
||||
let (status, reply) = feed(&server, &team.vault, &team.editor);
|
||||
assert_eq!(status, 200);
|
||||
let listing = reply.to_string();
|
||||
assert!(!listing.contains(body), "{listing}");
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
//! End-to-end over real HTTP: a server on an ephemeral port and the `ureq` client.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn two_devices_converge_and_conflicts_are_reported() {
|
||||
let server = Server::start_temp("flow");
|
||||
let (vault, phone) = server.paired("phone");
|
||||
let laptop = server.pair_to(&vault, "laptop");
|
||||
|
||||
let (status, record) = server.put(&vault, &phone, "notes/a.md", "# one\n", 0);
|
||||
assert_eq!((status, record["seq"].as_u64()), (201, Some(1)));
|
||||
|
||||
let (_, page) = server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(&laptop), &[], None);
|
||||
assert_eq!(page["head"], 1);
|
||||
assert_eq!(page["records"][0]["path"], "notes/a.md");
|
||||
let (status, bytes) = server.send("GET", &format!("/v1/vaults/{vault}/files/notes/a.md"), Some(&laptop), &[], None);
|
||||
assert_eq!((status, bytes.as_slice()), (200, b"# one\n".as_slice()));
|
||||
|
||||
// The laptop edits on top of seq 1; the phone then writes from the same stale base.
|
||||
assert_eq!(server.put(&vault, &laptop, "notes/a.md", "# laptop\n", 1).0, 201);
|
||||
let (status, conflict) = server.put(&vault, &phone, "notes/a.md", "# phone\n", 1);
|
||||
assert_eq!(status, 409);
|
||||
assert_eq!(conflict["current"]["seq"], 2);
|
||||
|
||||
// An identical write is a no-op.
|
||||
assert_eq!(server.put(&vault, &laptop, "notes/a.md", "# laptop\n", 2).0, 200);
|
||||
|
||||
// Delete, then a stale edit over the delete wins.
|
||||
let (status, _) = server.json("DELETE", &format!("/v1/vaults/{vault}/files/notes/a.md"), Some(&laptop), &[("X-Base-Seq", "2")], None);
|
||||
assert_eq!(status, 201);
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/files/notes/a.md"), Some(&phone), &[], None).0, 404);
|
||||
assert_eq!(server.put(&vault, &phone, "notes/a.md", "# phone again\n", 2).0, 201);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn etag_and_batch_check() {
|
||||
let server = Server::start_temp("etag");
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, record) = server.put(&vault, &token, "a.md", "x", 0);
|
||||
let hash = record["hash"].as_str().expect("hash").to_owned();
|
||||
let path = format!("/v1/vaults/{vault}/files/a.md");
|
||||
assert_eq!(server.send("GET", &path, Some(&token), &[("If-None-Match", &hash)], None).0, 304);
|
||||
let unknown = "0".repeat(64);
|
||||
let (status, missing) = server.json("POST", &format!("/v1/vaults/{vault}/batch/check"), Some(&token), &[], Some(&json!({"hashes": [hash, unknown]})));
|
||||
assert_eq!((status, missing["missing"].as_array().map(Vec::len)), (200, Some(1)));
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{vault}/batch/check"), Some(&token), &[], Some(&json!({"hashes": ["nope"]})));
|
||||
assert_eq!(status, 400);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn auth_and_input_are_enforced() {
|
||||
let server = Server::start_temp("auth");
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, other_token) = server.paired("intruder");
|
||||
let changes = format!("/v1/vaults/{vault}/changes");
|
||||
|
||||
assert_eq!(server.send("GET", &changes, None, &[], None).0, 401);
|
||||
assert_eq!(server.send("GET", &changes, Some("wrong"), &[], None).0, 401);
|
||||
// A token of another account is a non-member: 404, so the vault stays invisible.
|
||||
assert_eq!(server.send("GET", &changes, Some(&other_token), &[], None).0, 404);
|
||||
// Vault creation is a device call now: the admin token is no special guest, a
|
||||
// missing body is a malformed request, a real token with a name creates.
|
||||
assert_eq!(server.json("POST", "/v1/vaults", Some("wrong-admin"), &[], Some(&json!({"name": "mine"}))).0, 401);
|
||||
assert_eq!(server.json("POST", "/v1/vaults", Some(ADMIN), &[], Some(&json!({"name": "mine"}))).0, 401);
|
||||
assert_eq!(server.send("POST", "/v1/vaults", Some(&token), &[], None).0, 415);
|
||||
|
||||
for bad in ["../x.md", ".obsidian/app.json", ".notescraft/x", "a/../b.md"] {
|
||||
let (status, _) = server.put(&vault, &token, bad, "x", 0);
|
||||
assert!(status == 400 || status == 404, "{bad} -> {status}");
|
||||
}
|
||||
assert_eq!(server.put(&vault, &token, ".obsidian/app.json", "x", 0).0, 400);
|
||||
let (status, _) = server.send("PUT", &format!("/v1/vaults/{vault}/files/a.md"), Some(&token), &[("X-Base-Seq", "-1")], Some(b"x"));
|
||||
assert_eq!(status, 400);
|
||||
assert_eq!(server.send("DELETE", &format!("/v1/vaults/{vault}/files/missing.md"), Some(&token), &[], None).0, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pairing_codes_work_once_and_names_are_required() {
|
||||
let server = Server::start_temp("pair");
|
||||
let (vault, _) = server.paired("first");
|
||||
let (_, code) = server.json("POST", &format!("/v1/vaults/{vault}/pairing"), Some(common::ADMIN), &[], None);
|
||||
let body = json!({"code": code["code"], "name": "second"});
|
||||
assert_eq!(server.json("POST", "/v1/devices", None, &[], Some(&body)).0, 200);
|
||||
assert_eq!(server.json("POST", "/v1/devices", None, &[], Some(&body)).0, 401);
|
||||
let (_, code) = server.json("POST", &format!("/v1/vaults/{vault}/pairing"), Some(common::ADMIN), &[], None);
|
||||
assert_eq!(server.json("POST", "/v1/devices", None, &[], Some(&json!({"code": code["code"], "name": " "}))).0, 400);
|
||||
// Admin bootstrap: a well-formed new vault id is created by its first pairing code; a malformed id is 404.
|
||||
assert_eq!(server.json("POST", "/v1/vaults/bad%20id/pairing", Some(common::ADMIN), &[], None).0, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn long_poll_wakes_on_a_write() {
|
||||
let server = Server::start_temp("poll");
|
||||
let (vault, token) = server.paired("pc");
|
||||
let writer = {
|
||||
let (base, token, vault) = (server.base.clone(), token.clone(), vault.clone());
|
||||
std::thread::spawn(move || {
|
||||
std::thread::sleep(Duration::from_millis(400));
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
agent
|
||||
.put(format!("{base}/v1/vaults/{vault}/files/late.md"))
|
||||
.header("Authorization", format!("Bearer {token}"))
|
||||
.send(b"late".as_slice())
|
||||
.expect("put");
|
||||
})
|
||||
};
|
||||
let started = Instant::now();
|
||||
let (_, page) = server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0&wait=10"), Some(&token), &[], None);
|
||||
assert_eq!(page["records"][0]["path"], "late.md");
|
||||
assert!(started.elapsed() < Duration::from_secs(5), "returned after {:?}", started.elapsed());
|
||||
writer.join().expect("writer");
|
||||
// Nothing new: the wait ends empty at the deadline.
|
||||
let started = Instant::now();
|
||||
let (_, page) = server.json("GET", &format!("/v1/vaults/{vault}/changes?since=1&wait=1"), Some(&token), &[], None);
|
||||
assert_eq!(page["records"].as_array().map(Vec::len), Some(0));
|
||||
assert!(started.elapsed() >= Duration::from_millis(900));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn state_survives_a_restart() {
|
||||
let data = temp_dir("restart");
|
||||
let (vault, token, base_seq) = {
|
||||
let server = common::start(&data);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, record) = server.put(&vault, &token, "keep.md", "kept", 0);
|
||||
(vault, token, record["seq"].as_u64().expect("seq"))
|
||||
};
|
||||
// A second server over the same folder (the first keeps running; SQLite WAL allows it).
|
||||
let server = common::start(&data);
|
||||
let (_, page) = server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(&token), &[], None);
|
||||
assert_eq!(page["records"][0]["seq"], base_seq);
|
||||
let (status, bytes) = server.send("GET", &format!("/v1/vaults/{vault}/files/keep.md"), Some(&token), &[], None);
|
||||
assert_eq!((status, bytes.as_slice()), (200, b"kept".as_slice()));
|
||||
assert_eq!(server.put(&vault, &token, "keep.md", "kept 2", base_seq).1["seq"], base_seq + 1);
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
//! The real engine sync client against the real server: a big file travels in
|
||||
//! chunks, a cut connection resumes with the chunks that never arrived.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Mutex;
|
||||
|
||||
use notescraft_engine::{Changes, HttpRemote, Limits, Progress, PutReply, Remote, SyncError, SyncOptions, SyncReport, SyncResult, SyncState, sync_once_with};
|
||||
use notescraft_sync_server::{AppState, Store, router};
|
||||
use notescraft_vault::Vault;
|
||||
|
||||
const ADMIN: &str = "test-admin-token-0123456789abcdef";
|
||||
/// The size of one chunk the client sends.
|
||||
const CHUNK: usize = 32 * 1024;
|
||||
/// The size above which the client sends a file in chunks.
|
||||
const THRESHOLD: u64 = 16 * 1024;
|
||||
/// The biggest chunk the server takes.
|
||||
const SERVER_CHUNK_MAX: usize = 64 * 1024;
|
||||
/// The biggest file the server assembles.
|
||||
const SERVER_FILE_CAP: u64 = 1024 * 1024;
|
||||
/// The size of the file that travels.
|
||||
const FILE_BYTES: usize = 300 * 1024;
|
||||
|
||||
/// A temp folder that is empty when the test starts.
|
||||
fn temp(name: &str) -> PathBuf {
|
||||
let dir = std::env::temp_dir().join(format!("nc-chunk-e2e-{name}-{}", std::process::id()));
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
std::fs::create_dir_all(&dir).expect("dir");
|
||||
dir
|
||||
}
|
||||
|
||||
/// Starts the server on an ephemeral port and returns its base url.
|
||||
fn start(data: &Path) -> String {
|
||||
std::fs::create_dir_all(data.join("blobs")).expect("blobs");
|
||||
let state = AppState::new(Store::open(&data.join("sync.db")).expect("db"), data.join("blobs"), ADMIN);
|
||||
// The chunk limits are set before the router is built: the route takes its
|
||||
// body cap from the state.
|
||||
state.set_chunk_limits(SERVER_CHUNK_MAX, SERVER_FILE_CAP);
|
||||
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind");
|
||||
listener.set_nonblocking(true).expect("nonblocking");
|
||||
let port = listener.local_addr().expect("addr").port();
|
||||
std::thread::spawn(move || {
|
||||
let runtime = tokio::runtime::Builder::new_multi_thread().enable_all().build().expect("runtime");
|
||||
runtime.block_on(async {
|
||||
let listener = tokio::net::TcpListener::from_std(listener).expect("listener");
|
||||
axum::serve(listener, router(state)).await.ok();
|
||||
});
|
||||
});
|
||||
format!("http://127.0.0.1:{port}")
|
||||
}
|
||||
|
||||
fn admin(base: &str, path: &str) -> serde_json::Value {
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
let mut response = agent.post(format!("{base}{path}")).header("Authorization", format!("Bearer {ADMIN}")).send_empty().expect("admin call");
|
||||
response.body_mut().read_json().expect("json")
|
||||
}
|
||||
|
||||
/// Issues a pairing code for the vault, bound to `user` when given.
|
||||
fn issue(base: &str, vault_id: &str, user: Option<&str>) -> String {
|
||||
let path = user.map_or_else(|| format!("/v1/vaults/{vault_id}/pairing"), |user| format!("/v1/vaults/{vault_id}/pairing?user={user}"));
|
||||
admin(base, &path)["code"].as_str().expect("code").to_owned()
|
||||
}
|
||||
|
||||
/// Redeems a code with raw HTTP to also learn the account the device joined.
|
||||
fn pair_raw(base: &str, code: &str, name: &str) -> (String, String) {
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
let mut response = agent.post(format!("{base}/v1/devices")).send_json(serde_json::json!({"code": code, "name": name})).expect("pair");
|
||||
let device: serde_json::Value = response.body_mut().read_json().expect("json");
|
||||
(device["token"].as_str().expect("token").to_owned(), device["user_id"].as_str().expect("user id").to_owned())
|
||||
}
|
||||
|
||||
/// A remote that can cut the connection in the middle of a chunked upload.
|
||||
struct CutRemote {
|
||||
inner: HttpRemote,
|
||||
/// How many chunks the client sent.
|
||||
sent: Mutex<usize>,
|
||||
/// From this send on, the connection is cut.
|
||||
cut_from: Mutex<Option<usize>>,
|
||||
}
|
||||
|
||||
impl CutRemote {
|
||||
fn new(inner: HttpRemote) -> Self {
|
||||
Self { inner, sent: Mutex::new(0), cut_from: Mutex::new(None) }
|
||||
}
|
||||
|
||||
/// Cuts the connection from the given chunk send on.
|
||||
fn cut_from(&self, call: usize) {
|
||||
*self.cut_from.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Some(call);
|
||||
}
|
||||
|
||||
/// Lets the connection through again.
|
||||
fn mend(&self) {
|
||||
*self.cut_from.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = None;
|
||||
}
|
||||
|
||||
/// How many chunks were sent since the last count.
|
||||
fn take_sent(&self) -> usize {
|
||||
let mut sent = self.sent.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
std::mem::take(&mut *sent)
|
||||
}
|
||||
}
|
||||
|
||||
impl Remote for CutRemote {
|
||||
fn changes(&self, since: u64, limit: usize, wait: u64) -> SyncResult<Changes> {
|
||||
self.inner.changes(since, limit, wait)
|
||||
}
|
||||
|
||||
fn get(&self, path: &str) -> SyncResult<Vec<u8>> {
|
||||
self.inner.get(path)
|
||||
}
|
||||
|
||||
fn put(&self, path: &str, bytes: &[u8], base_seq: u64, mtime_ms: u64) -> SyncResult<PutReply> {
|
||||
self.inner.put(path, bytes, base_seq, mtime_ms)
|
||||
}
|
||||
|
||||
fn delete(&self, path: &str, base_seq: u64) -> SyncResult<PutReply> {
|
||||
self.inner.delete(path, base_seq)
|
||||
}
|
||||
|
||||
fn missing_chunks(&self, hashes: &[String]) -> SyncResult<Vec<String>> {
|
||||
self.inner.missing_chunks(hashes)
|
||||
}
|
||||
|
||||
fn put_chunk(&self, hash: &str, bytes: &[u8]) -> SyncResult<()> {
|
||||
let mut sent = self.sent.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
*sent = sent.saturating_add(1);
|
||||
let cut = *self.cut_from.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if cut.is_some_and(|from| *sent >= from) {
|
||||
return Err(SyncError::Http("the connection was cut".to_owned()));
|
||||
}
|
||||
drop(sent);
|
||||
self.inner.put_chunk(hash, bytes)
|
||||
}
|
||||
|
||||
fn put_chunked(&self, path: &str, chunk_hashes: &[String], content_hash: &str, base_seq: u64, mtime_ms: u64) -> SyncResult<PutReply> {
|
||||
self.inner.put_chunked(path, chunk_hashes, content_hash, base_seq, mtime_ms)
|
||||
}
|
||||
}
|
||||
|
||||
/// A device with its own vault, its own state and a remote that can cut.
|
||||
struct Device {
|
||||
vault: Vault,
|
||||
remote: CutRemote,
|
||||
state: SyncState,
|
||||
name: &'static str,
|
||||
}
|
||||
|
||||
impl Device {
|
||||
fn new(base: &str, vault_id: &str, token: &str, name: &'static str) -> Self {
|
||||
let root = temp(name);
|
||||
let vault = Vault::open(&root).expect("vault");
|
||||
Self { vault, remote: CutRemote::new(HttpRemote::new(base, vault_id, token)), state: SyncState::default(), name }
|
||||
}
|
||||
|
||||
fn write(&self, path: &str, bytes: &[u8]) {
|
||||
self.vault.write(path, bytes).expect("write");
|
||||
}
|
||||
|
||||
fn read(&self, path: &str) -> Vec<u8> {
|
||||
self.vault.read(path).expect("read")
|
||||
}
|
||||
|
||||
/// Syncs once and returns the report.
|
||||
fn sync(&mut self) -> SyncReport {
|
||||
let options = SyncOptions { limits: Limits { threshold: THRESHOLD, chunk: CHUNK }, progress: Box::new(|_: &Progress| {}) };
|
||||
sync_once_with(&self.vault, &self.remote, &mut self.state, self.name, "2026-10-20", options).expect("sync")
|
||||
}
|
||||
}
|
||||
|
||||
/// Bytes that look random but come from a small generator, so the test is stable.
|
||||
fn random_bytes(len: usize, seed: u64) -> Vec<u8> {
|
||||
let mut state = seed;
|
||||
(0..len)
|
||||
.map(|_| {
|
||||
state = state.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1_442_695_040_888_963_407);
|
||||
u8::try_from((state >> 33) & 0xff).expect("byte")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// How many chunks the server holds for the vault.
|
||||
fn stored_chunks(blobs: &Path, vault_id: &str) -> usize {
|
||||
let dir = blobs.join(vault_id).join("chunks");
|
||||
let Ok(entries) = std::fs::read_dir(dir) else { return 0 };
|
||||
entries.filter_map(std::result::Result::ok).filter(|entry| entry.path().is_file()).count()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_big_file_travels_in_chunks_and_a_cut_resumes_with_the_rest() {
|
||||
let data = temp("server-data");
|
||||
let base = start(&data);
|
||||
let blobs = data.join("blobs");
|
||||
let vault_id = format!("v-chunk-{}", std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.subsec_nanos()));
|
||||
let (phone_token, user_id) = pair_raw(&base, &issue(&base, &vault_id, None), "phone");
|
||||
let laptop = pair_raw(&base, &issue(&base, &vault_id, Some(&user_id)), "laptop");
|
||||
let mut phone = Device::new(&base, &vault_id, &phone_token, "phone");
|
||||
let mut laptop = Device::new(&base, &vault_id, &laptop.0, "laptop");
|
||||
|
||||
// A file that is too big for one request travels in chunks.
|
||||
let bytes = random_bytes(FILE_BYTES, 42);
|
||||
phone.write("big.bin", &bytes);
|
||||
let report = phone.sync();
|
||||
assert_eq!(report.uploaded, vec!["big.bin".to_owned()]);
|
||||
let chunks = FILE_BYTES.div_ceil(CHUNK);
|
||||
assert_eq!(stored_chunks(&blobs, &vault_id), chunks, "the server holds every chunk");
|
||||
|
||||
// The other device gets the very same bytes.
|
||||
let report = laptop.sync();
|
||||
assert_eq!(report.downloaded, vec!["big.bin".to_owned()]);
|
||||
assert_eq!(laptop.read("big.bin"), bytes);
|
||||
|
||||
// The connection dies on the fourth chunk of the next file.
|
||||
let other = random_bytes(FILE_BYTES, 7);
|
||||
phone.write("other.bin", &other);
|
||||
phone.remote.take_sent();
|
||||
phone.remote.cut_from(4);
|
||||
let error = sync_once_with(
|
||||
&phone.vault,
|
||||
&phone.remote,
|
||||
&mut phone.state,
|
||||
"phone",
|
||||
"2026-10-20",
|
||||
SyncOptions { limits: Limits { threshold: THRESHOLD, chunk: CHUNK }, progress: Box::new(|_: &Progress| {}) },
|
||||
)
|
||||
.expect_err("cut");
|
||||
assert!(matches!(error, SyncError::Http(_)), "{error:?}");
|
||||
phone.remote.take_sent();
|
||||
assert_eq!(stored_chunks(&blobs, &vault_id), chunks + 3, "the three chunks that arrived are kept");
|
||||
|
||||
// The next pass sends the chunks that never arrived and nothing else.
|
||||
phone.remote.mend();
|
||||
phone.sync();
|
||||
assert_eq!(phone.remote.take_sent(), chunks - 3, "only the missing chunks travel again");
|
||||
assert_eq!(stored_chunks(&blobs, &vault_id), chunks * 2, "the server holds every chunk of both files");
|
||||
|
||||
// The other device converges on the second file too.
|
||||
laptop.sync();
|
||||
assert_eq!(laptop.read("other.bin"), other);
|
||||
assert_eq!(laptop.read("big.bin"), bytes);
|
||||
}
|
||||
@@ -0,0 +1,348 @@
|
||||
//! Chunked, resumable, deduplicated uploads: `PUT /chunks/{hash}`, `POST /chunks/check`
|
||||
//! and `PUT /files/{path}` with `X-Chunks`. Real HTTP with shrunken limits.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use notescraft_sync::hash_hex;
|
||||
use notescraft_sync_server::{AppState, Store, TOMBSTONE_GRACE_MS, now_ms};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
/// Tests upload 1 KiB chunks instead of the 4 MiB default.
|
||||
const CHUNK_MAX: usize = 1024;
|
||||
/// Files may reach 8 KiB in tests instead of the 1 GiB default.
|
||||
const FILE_CAP: u64 = 8 * 1024;
|
||||
/// A 64 hex digit shape for hostile or placeholder hashes.
|
||||
fn hex_zeros(n: usize) -> String {
|
||||
"0".repeat(n)
|
||||
}
|
||||
|
||||
fn chunks_server(name: &str, quota: Option<u64>) -> (Server, PathBuf) {
|
||||
let data = temp_dir(name);
|
||||
let state = AppState::with_quota(Store::open(&data.join("sync.db")).expect("db"), data.join("blobs"), ADMIN, quota);
|
||||
state.set_chunk_limits(CHUNK_MAX, FILE_CAP);
|
||||
(common::start_with_state(state), data)
|
||||
}
|
||||
|
||||
/// `len` bytes of one value, so chunks of different seeds differ.
|
||||
fn chunk_data(seed: u8, len: usize) -> Vec<u8> {
|
||||
vec![seed; len]
|
||||
}
|
||||
|
||||
/// The chunk files of a vault, by name.
|
||||
fn chunk_names(data: &Path, vault: &str) -> Vec<String> {
|
||||
let dir = data.join("blobs").join(vault).join("chunks");
|
||||
let mut names: Vec<String> = std::fs::read_dir(&dir)
|
||||
.map_or_else(|_| Vec::new(), |entries| entries.filter_map(|entry| entry.ok().and_then(|e| e.file_name().into_string().ok())).collect());
|
||||
names.sort();
|
||||
names
|
||||
}
|
||||
|
||||
/// `PUT /chunks/{hash}`: the status.
|
||||
fn put_chunk(server: &Server, vault: &str, token: &str, hash: &str, body: &[u8]) -> u16 {
|
||||
server.send("PUT", &format!("/v1/vaults/{vault}/chunks/{hash}"), Some(token), &[], Some(body)).0
|
||||
}
|
||||
|
||||
/// Uploads the seeds as [`CHUNK_MAX`]-sized chunks; answers their joined bytes and hashes.
|
||||
fn upload_chunks(server: &Server, vault: &str, token: &str, seeds: &[u8]) -> (Vec<u8>, Vec<String>) {
|
||||
let mut content = Vec::new();
|
||||
let mut hashes = Vec::new();
|
||||
for seed in seeds {
|
||||
let bytes = chunk_data(*seed, CHUNK_MAX);
|
||||
assert_eq!(put_chunk(server, vault, token, &hash_hex(&bytes), &bytes), 201);
|
||||
hashes.push(hash_hex(&bytes));
|
||||
content.extend_from_slice(&bytes);
|
||||
}
|
||||
(content, hashes)
|
||||
}
|
||||
|
||||
/// `POST /chunks/check`: `(status, missing)`.
|
||||
fn check_chunks(server: &Server, vault: &str, token: &str, hashes: &[String]) -> (u16, Vec<String>) {
|
||||
let (status, reply) = server.json("POST", &format!("/v1/vaults/{vault}/chunks/check"), Some(token), &[], Some(&json!({"hashes": hashes})));
|
||||
let missing = reply["missing"].as_array().map_or_else(Vec::new, |rows| rows.iter().filter_map(|v| v.as_str().map(str::to_owned)).collect());
|
||||
(status, missing)
|
||||
}
|
||||
|
||||
/// `PUT /files/{path}` with `X-Chunks`: `(status, reply body)`.
|
||||
fn assemble(server: &Server, vault: &str, token: &str, path: &str, hashes: &[String], base_seq: u64, content_hash: Option<&str>) -> (u16, Value) {
|
||||
let joined = hashes.join(",");
|
||||
let base = base_seq.to_string();
|
||||
let mut named: Vec<(&str, &str)> = vec![("X-Chunks", joined.as_str()), ("X-Base-Seq", base.as_str())];
|
||||
named.extend(content_hash.map(|hash| ("X-Content-Hash", hash)));
|
||||
let (status, reply) = server.send("PUT", &format!("/v1/vaults/{vault}/files/{path}"), Some(token), &named, None);
|
||||
(status, serde_json::from_slice(&reply).unwrap_or(Value::Null))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunks_upload_idempotently_assemble_and_read_back_exactly() {
|
||||
let (server, data) = chunks_server("chunks", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (content, hashes) = upload_chunks(&server, &vault, &token, &[1, 2, 3, 4, 5]);
|
||||
assert_eq!(chunk_names(&data, &vault), sorted(hashes.as_slice()), "five chunks stored once each");
|
||||
|
||||
// Everything is there; the resume endpoint answers complete.
|
||||
let (status, missing) = check_chunks(&server, &vault, &token, &hashes);
|
||||
assert_eq!((status, missing), (200, Vec::<String>::new()));
|
||||
|
||||
let expected_hash = hash_hex(&content);
|
||||
let (status, record) = assemble(&server, &vault, &token, "big.bin", &hashes, 0, None);
|
||||
assert_eq!(status, 201);
|
||||
assert_eq!(record["hash"].as_str(), Some(expected_hash.as_str()));
|
||||
assert_eq!(record["size"].as_u64(), Some(u64::try_from(content.len()).expect("small")));
|
||||
let (status, bytes) = server.send("GET", &format!("/v1/vaults/{vault}/files/big.bin"), Some(&token), &[], None);
|
||||
assert_eq!((status, bytes.as_slice()), (200, content.as_slice()));
|
||||
|
||||
// A second assembly over a fresh path reuses the chunk files.
|
||||
let (status, _) = assemble(&server, &vault, &token, "same.bin", &hashes, 0, None);
|
||||
assert_eq!(status, 201, "a fresh path is a new file");
|
||||
assert_eq!(chunk_names(&data, &vault), sorted(hashes.as_slice()), "chunks untouched by assembly");
|
||||
}
|
||||
|
||||
/// The hashes a folder holds, ascending, for comparisons with `chunk_names`.
|
||||
fn sorted(hashes: &[String]) -> Vec<String> {
|
||||
let mut sorted = hashes.to_vec();
|
||||
sorted.sort();
|
||||
sorted
|
||||
}
|
||||
|
||||
/// `PUT /files/{path}` with an `X-Chunks` header through a raw HTTP connection, closed
|
||||
/// after the answer; `(status, body)`.
|
||||
fn raw_put(server: &Server, token: &str, vault: &str, path: &str, chunks: &str) -> (u16, String) {
|
||||
use std::io::{Read, Write};
|
||||
let port: u16 = server.base.strip_prefix("http://127.0.0.1:").and_then(|raw| raw.parse().ok()).expect("port");
|
||||
let request = format!(
|
||||
"PUT /v1/vaults/{vault}/files{path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\nAuthorization: Bearer {token}\r\nX-Chunks: {chunks}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
);
|
||||
let mut stream = std::net::TcpStream::connect(("127.0.0.1", port)).expect("connect");
|
||||
stream.write_all(request.as_bytes()).expect("write");
|
||||
let mut reply = Vec::new();
|
||||
stream.read_to_end(&mut reply).expect("read");
|
||||
let text = String::from_utf8_lossy(&reply).into_owned();
|
||||
let code = text.split_whitespace().nth(1).and_then(|word| word.parse().ok()).unwrap_or(0);
|
||||
(code, text)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resume_after_a_cut_connection_only_sends_missing_chunks() {
|
||||
let (server, _) = chunks_server("resume", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let seeds = [1u8, 2, 3, 4, 5];
|
||||
let hashes: Vec<String> = seeds.iter().map(|seed| hash_hex(&chunk_data(*seed, CHUNK_MAX))).collect();
|
||||
let content: Vec<u8> = seeds.iter().flat_map(|seed| chunk_data(*seed, CHUNK_MAX)).collect();
|
||||
|
||||
// Three of five arrived before the cut.
|
||||
for (index, seed) in seeds.iter().enumerate().take(3) {
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hashes[index], &chunk_data(*seed, CHUNK_MAX)), 201);
|
||||
}
|
||||
let (status, missing) = check_chunks(&server, &vault, &token, &hashes);
|
||||
assert_eq!((status, missing), (200, hashes[3..5].to_vec()));
|
||||
|
||||
// The resumed connection sends only what is missing, then assembles.
|
||||
for (index, seed) in seeds.iter().enumerate().skip(3) {
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hashes[index], &chunk_data(*seed, CHUNK_MAX)), 201);
|
||||
}
|
||||
assert_eq!(assemble(&server, &vault, &token, "big.bin", &hashes, 0, None).0, 201);
|
||||
let (status, bytes) = server.send("GET", &format!("/v1/vaults/{vault}/files/big.bin"), Some(&token), &[], None);
|
||||
assert_eq!((status, bytes.as_slice()), (200, content.as_slice()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_chunk_whose_hash_does_not_match_is_refused_and_stores_nothing() {
|
||||
let (server, data) = chunks_server("badhash", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let body = chunk_data(1, 100);
|
||||
for wrong in ["a".repeat(64), "A".repeat(64), "0".repeat(63), String::from("x")] {
|
||||
let (status, _) = server.send("PUT", &format!("/v1/vaults/{vault}/chunks/{wrong}"), Some(&token), &[], Some(body.as_slice()));
|
||||
assert_eq!(status, 400, "hash {wrong}");
|
||||
}
|
||||
// A well formed hash that does not match the body.
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hash_hex(b"other"), &body), 400);
|
||||
// The empty body is refused as well.
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hash_hex(&body), b""), 400);
|
||||
assert_eq!(chunk_names(&data, &vault), Vec::<String>::new(), "nothing was stored");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_oversized_chunk_is_refused() {
|
||||
let (server, data) = chunks_server("bigchunk", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let body = chunk_data(1, CHUNK_MAX + 1);
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hex_zeros(64), &body), 413);
|
||||
assert_eq!(chunk_names(&data, &vault), Vec::<String>::new());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assembling_with_a_missing_chunk_answers_409_and_makes_no_file() {
|
||||
let (server, _) = chunks_server("missing", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let bytes = chunk_data(1, CHUNK_MAX);
|
||||
let hash = hash_hex(&bytes);
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hash, &bytes), 201);
|
||||
let unknown = "f".repeat(64);
|
||||
let (status, reply) = assemble(&server, &vault, &token, "big.bin", &[hash, unknown.clone()], 0, None);
|
||||
assert_eq!(status, 409);
|
||||
assert_eq!(reply["missing"], json!([unknown]), "the absent hash is named");
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/files/big.bin"), Some(&token), &[], None).0, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assembling_with_a_wrong_content_hash_is_400() {
|
||||
let (server, _) = chunks_server("wrongcontent", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &token, &[1, 2]);
|
||||
let (status, _) = assemble(&server, &vault, &token, "big.bin", &hashes, 0, Some(&"f".repeat(64)));
|
||||
assert_eq!(status, 400, "the header must match the computed hash");
|
||||
let content: Vec<u8> = chunk_data(1, CHUNK_MAX).into_iter().chain(chunk_data(2, CHUNK_MAX)).collect();
|
||||
let (status, record) = assemble(&server, &vault, &token, "big.bin", &hashes, 0, Some(&hash_hex(&content)));
|
||||
assert_eq!((status, record["size"].as_u64()), (201, Some(u64::try_from(content.len()).expect("small"))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_assembly_over_the_file_cap_is_413() {
|
||||
let (server, _) = chunks_server("filecap", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let seeds: Vec<u8> = (1..=10).collect();
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &token, &seeds);
|
||||
let (status, reply) = assemble(&server, &vault, &token, "huge.bin", &hashes, 0, None);
|
||||
assert_eq!(status, 413, "10 KiB of chunks exceed the 8 KiB file cap");
|
||||
assert!(reply["error"].is_string());
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/files/huge.bin"), Some(&token), &[], None).0, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_second_file_sharing_chunks_stores_them_once() {
|
||||
let (server, data) = chunks_server("dedup", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &token, &[1, 2, 3]);
|
||||
assert_eq!(chunk_names(&data, &vault), sorted(hashes.as_slice()));
|
||||
|
||||
// A second file with the same chunks in a different order: a different file hash,
|
||||
// and the chunk folder does not grow.
|
||||
let reversed: Vec<String> = hashes.iter().rev().cloned().collect();
|
||||
assert_eq!(assemble(&server, &vault, &token, "a.bin", &hashes, 0, None).0, 201);
|
||||
assert_eq!(assemble(&server, &vault, &token, "b.bin", &reversed, 0, None).0, 201);
|
||||
assert_eq!(chunk_names(&data, &vault), sorted(hashes.as_slice()), "three chunks, two files");
|
||||
|
||||
// Re-uploading a stored chunk is idempotent.
|
||||
let bytes = chunk_data(1, CHUNK_MAX);
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hash_hex(&bytes), &bytes), 200, "already there");
|
||||
assert_eq!(chunk_names(&data, &vault), sorted(hashes.as_slice()));
|
||||
}
|
||||
#[test]
|
||||
fn a_viewer_cannot_upload_chunks_or_assemble() {
|
||||
let (server, _) = chunks_server("viewer", None);
|
||||
let (vault, owner) = server.paired("pc");
|
||||
let (_, viewer) = server.paired("vw");
|
||||
let (status, code) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(&viewer, &code).0, 200);
|
||||
|
||||
let bytes = chunk_data(1, 100);
|
||||
assert_eq!(put_chunk(&server, &vault, &viewer, &hash_hex(&bytes), &bytes), 403);
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &owner, &[1, 2]);
|
||||
assert_eq!(assemble(&server, &vault, &viewer, "big.bin", &hashes, 0, None).0, 403, "viewer assembly refused");
|
||||
assert_eq!(assemble(&server, &vault, &owner, "big.bin", &hashes, 0, None).0, 201);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_quota_refuses_a_chunk_and_an_assembly() {
|
||||
// A chunk larger than what the quota still allows: refused at upload.
|
||||
let (server, _) = chunks_server("quota-chunk", Some(512));
|
||||
let (vault, token) = server.paired("pc");
|
||||
let bytes = chunk_data(1, CHUNK_MAX);
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &hash_hex(&bytes), &bytes), 413);
|
||||
assert!(server.audit().iter().any(|row| row["kind"] == "quota_rejected"));
|
||||
|
||||
// Two chunks fit (their upload counts them), the assembly does not.
|
||||
let (server, _) = chunks_server("quota-assemble", Some(1536));
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &token, &[1, 2]);
|
||||
let (status, reply) = assemble(&server, &vault, &token, "big.bin", &hashes, 0, None);
|
||||
assert_eq!(status, 413);
|
||||
assert_eq!(reply["error"], json!("vault storage quota exceeded"));
|
||||
assert!(server.audit().iter().any(|row| row["kind"] == "quota_rejected"));
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/files/big.bin"), Some(&token), &[], None).0, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_chunked_file_has_revisions_and_their_bytes_stay_restorable() {
|
||||
let (server, _) = chunks_server("history", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (v1, hashes1) = upload_chunks(&server, &vault, &token, &[1, 2]);
|
||||
let (status, _) = assemble(&server, &vault, &token, "doc.bin", &hashes1, 0, None);
|
||||
assert_eq!(status, 201);
|
||||
|
||||
// The edit arrives as its own chunk set on a fresh base.
|
||||
let mut hashes2 = hashes1.clone();
|
||||
let (extra_bytes, extra_hash) = (chunk_data(3, CHUNK_MAX), hash_hex(&chunk_data(3, CHUNK_MAX)));
|
||||
assert_eq!(put_chunk(&server, &vault, &token, &extra_hash, &extra_bytes), 201);
|
||||
hashes2[0] = extra_hash;
|
||||
let (status, record) = assemble(&server, &vault, &token, "doc.bin", &hashes2, 1, None);
|
||||
assert_eq!(status, 201, "the edited version wins with a fresh base");
|
||||
let seq2 = record["seq"].as_u64().expect("seq");
|
||||
assert_eq!(seq2, 2);
|
||||
|
||||
let (status, history) = server.json("GET", &format!("/v1/vaults/{vault}/files-history/doc.bin"), Some(&token), &[], None);
|
||||
assert_eq!((status, history.as_array().map(Vec::len)), (200, Some(2)));
|
||||
let (status, old) = server.send("GET", &format!("/v1/vaults/{vault}/files/doc.bin?seq=1"), Some(&token), &[], None);
|
||||
assert_eq!((status, old.as_slice()), (200, v1.as_slice()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gc_removes_unreferenced_chunks_and_keeps_assembled_ones() {
|
||||
let (server, data) = chunks_server("gc", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (_, referenced) = upload_chunks(&server, &vault, &token, &[1, 2]);
|
||||
let _spare = upload_chunks(&server, &vault, &token, &[3, 4]);
|
||||
assert_eq!(assemble(&server, &vault, &token, "big.bin", &referenced, 0, None).0, 201);
|
||||
let content: Vec<u8> = chunk_data(1, CHUNK_MAX).into_iter().chain(chunk_data(2, CHUNK_MAX)).collect();
|
||||
|
||||
// Unreferenced chunks of a dead upload and an orphan file: collected after the grace.
|
||||
let orphan = data.join("blobs").join(&vault).join("chunks").join("garbage");
|
||||
std::fs::write(&orphan, b"waste").expect("orphan file");
|
||||
let aged = now_ms() + TOMBSTONE_GRACE_MS + 1;
|
||||
let report = server.state.gc(&vault, TOMBSTONE_GRACE_MS, aged).expect("gc");
|
||||
assert_eq!(report.removed, 3, "two spare chunks plus the orphan file");
|
||||
assert_eq!(chunk_names(&data, &vault), sorted(referenced.as_slice()), "the assembled file's chunks stay");
|
||||
let (status, bytes) = server.send("GET", &format!("/v1/vaults/{vault}/files/big.bin"), Some(&token), &[], None);
|
||||
assert_eq!((status, bytes.as_slice()), (200, content.as_slice()), "the assembled file survives GC");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_input_is_refused_without_panic() {
|
||||
let (server, data) = chunks_server("hostile", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
|
||||
// A chunk check of 5001 hashes is refused; 5000 are the cap.
|
||||
let many: Vec<String> = (0..5001).map(|i| format!("{i:064x}")).collect();
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{vault}/chunks/check"), Some(&token), &[], Some(&json!({"hashes": many})));
|
||||
assert_eq!(status, 400);
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{vault}/chunks/check"), Some(&token), &[], Some(&json!({"hashes": []})));
|
||||
assert_eq!(status, 200, "an empty check answers complete");
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &token, &[1, 2, 3]);
|
||||
let (status, missing) = check_chunks(&server, &vault, &token, &hashes);
|
||||
assert_eq!((status, missing), (200, Vec::<String>::new()));
|
||||
|
||||
// An X-Chunks of 5000 entries is refused: beyond the 4096 list limit. The header
|
||||
// fits hyper's 408 KiB parse buffer, so it reaches the handler — ureq cannot send
|
||||
// headers this large, so the request goes through a raw HTTP connection.
|
||||
let beyond: Vec<String> = (0..5000).map(|i| format!("{i:064x}")).collect();
|
||||
let (status, _) = raw_put(&server, &token, &vault, "/x.bin", &beyond.join(","));
|
||||
assert_eq!(status, 400);
|
||||
|
||||
// Malformed check entries, a hostile path and a non-empty body.
|
||||
|
||||
// Malformed check entries, a hostile path and a non-empty body.
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{vault}/chunks/check"), Some(&token), &[], Some(&json!({"hashes": ["zz"]})));
|
||||
assert_eq!(status, 400);
|
||||
let (_, hashes) = upload_chunks(&server, &vault, &token, &[8]);
|
||||
assert_eq!(assemble(&server, &vault, &token, "a/../x.bin", &hashes, 0, None).0, 400);
|
||||
let (status, _) =
|
||||
server.send("PUT", &format!("/v1/vaults/{vault}/files/x.bin"), Some(&token), &[("X-Chunks", &hash_hex(&chunk_data(8, CHUNK_MAX)))], Some(b"not empty"));
|
||||
assert_eq!(status, 400, "a chunked upload carries no body");
|
||||
let (status, _) = server.send("PUT", &format!("/v1/vaults/{vault}/files/x.bin"), Some(&token), &[("X-Chunks", "")], None);
|
||||
assert_eq!(status, 400, "an empty X-Chunks names no chunks");
|
||||
assert!(chunk_names(&data, &vault).len() >= 2, "the honest chunks still exist");
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
//! The real engine sync client against the real server over HTTP: two vault folders converge.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use notescraft_engine::{HttpRemote, SyncState, pair, sync_once};
|
||||
use notescraft_sync_server::{AppState, Store, router};
|
||||
use notescraft_vault::Vault;
|
||||
|
||||
const ADMIN: &str = "test-admin-token-0123456789abcdef";
|
||||
|
||||
fn temp(name: &str) -> PathBuf {
|
||||
let dir = std::env::temp_dir().join(format!("nc-e2e-{name}-{}", std::process::id()));
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
std::fs::create_dir_all(&dir).expect("dir");
|
||||
dir
|
||||
}
|
||||
|
||||
/// Starts the server on an ephemeral port and returns its base url.
|
||||
fn start(data: &Path) -> String {
|
||||
std::fs::create_dir_all(data.join("blobs")).expect("blobs");
|
||||
let state = AppState::new(Store::open(&data.join("sync.db")).expect("db"), data.join("blobs"), ADMIN);
|
||||
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind");
|
||||
listener.set_nonblocking(true).expect("nonblocking");
|
||||
let port = listener.local_addr().expect("addr").port();
|
||||
std::thread::spawn(move || {
|
||||
let runtime = tokio::runtime::Builder::new_multi_thread().enable_all().build().expect("runtime");
|
||||
runtime.block_on(async {
|
||||
let listener = tokio::net::TcpListener::from_std(listener).expect("listener");
|
||||
axum::serve(listener, router(state)).await.ok();
|
||||
});
|
||||
});
|
||||
format!("http://127.0.0.1:{port}")
|
||||
}
|
||||
|
||||
fn admin(base: &str, path: &str) -> serde_json::Value {
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
let mut response = agent.post(format!("{base}{path}")).header("Authorization", format!("Bearer {ADMIN}")).send_empty().expect("admin call");
|
||||
response.body_mut().read_json().expect("json")
|
||||
}
|
||||
|
||||
struct Device {
|
||||
vault: Vault,
|
||||
remote: HttpRemote,
|
||||
state: SyncState,
|
||||
name: &'static str,
|
||||
}
|
||||
|
||||
impl Device {
|
||||
/// A local vault over an already-paired device token. `name` keeps temp folders apart.
|
||||
fn new(base: &str, vault_id: &str, token: &str, name: &'static str) -> Self {
|
||||
let root = temp(name);
|
||||
let vault = Vault::open(&root).expect("vault");
|
||||
Self { vault, remote: HttpRemote::new(base, vault_id, token), state: SyncState::default(), name }
|
||||
}
|
||||
|
||||
fn write(&self, path: &str, text: &str) {
|
||||
self.vault.write(path, text.as_bytes()).expect("write");
|
||||
}
|
||||
|
||||
fn read(&self, path: &str) -> String {
|
||||
String::from_utf8(self.vault.read(path).expect("read")).expect("utf8")
|
||||
}
|
||||
|
||||
fn sync(&mut self) -> notescraft_engine::SyncReport {
|
||||
let report = sync_once(&self.vault, &self.remote, &mut self.state, self.name, "2026-10-20").expect("sync");
|
||||
self.state.save(&self.vault).expect("save state");
|
||||
report
|
||||
}
|
||||
}
|
||||
|
||||
/// Issues a pairing code (admin) for the vault, bound to `user` when given; issuing
|
||||
/// for an unknown vault id creates the vault (accounts have no other bootstrap).
|
||||
fn issue(base: &str, vault_id: &str, user: Option<&str>) -> String {
|
||||
let path = user.map_or_else(|| format!("/v1/vaults/{vault_id}/pairing"), |user| format!("/v1/vaults/{vault_id}/pairing?user={user}"));
|
||||
admin(base, &path)["code"].as_str().expect("code").to_owned()
|
||||
}
|
||||
|
||||
/// Redeems a code with raw HTTP to also learn the account the device joined.
|
||||
fn pair_raw(base: &str, code: &str, name: &str) -> (String, String) {
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
let mut response = agent.post(format!("{base}/v1/devices")).send_json(serde_json::json!({"code": code, "name": name})).expect("pair");
|
||||
let device: serde_json::Value = response.body_mut().read_json().expect("json");
|
||||
(device["token"].as_str().expect("token").to_owned(), device["user_id"].as_str().expect("user id").to_owned())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_devices_converge_through_the_real_server() {
|
||||
let base = start(&temp("server-data"));
|
||||
// Accounts: the first pairing creates the vault and its owner account; the second
|
||||
// device joins the same account through a `?user=`-bound pairing code for the
|
||||
// same vault, so one token works for the account, not just one vault.
|
||||
let vault_id = format!("v-e2e-{}", std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.subsec_nanos()));
|
||||
let (phone_token, user_id) = pair_raw(&base, &issue(&base, &vault_id, None), "phone");
|
||||
let laptop_pairing = pair(&base, &issue(&base, &vault_id, Some(&user_id)), "laptop").expect("pair");
|
||||
assert_eq!(laptop_pairing.vault_id, vault_id);
|
||||
let mut phone = Device::new(&base, &vault_id, &phone_token, "phone");
|
||||
let mut laptop = Device::new(&base, &laptop_pairing.vault_id, &laptop_pairing.token, "laptop");
|
||||
|
||||
phone.write("notes/a.md", "one\ntwo\nthree\nfour\nfive\n");
|
||||
phone.write("b.md", "bee\n");
|
||||
assert_eq!(phone.sync().uploaded.len(), 2);
|
||||
assert_eq!(laptop.sync().downloaded.len(), 2);
|
||||
assert_eq!(laptop.read("notes/a.md"), "one\ntwo\nthree\nfour\nfive\n");
|
||||
|
||||
// Offline edits on different lines merge without a conflict copy.
|
||||
phone.write("notes/a.md", "ONE\ntwo\nthree\nfour\nfive\n");
|
||||
laptop.write("notes/a.md", "one\ntwo\nthree\nfour\nFIVE\n");
|
||||
assert_eq!(phone.sync().uploaded, vec!["notes/a.md".to_owned()]);
|
||||
let report = laptop.sync();
|
||||
assert_eq!(report.merged, vec!["notes/a.md".to_owned()]);
|
||||
assert_eq!(report.conflicts, Vec::<String>::new());
|
||||
phone.sync();
|
||||
assert_eq!(phone.read("notes/a.md"), "ONE\ntwo\nthree\nfour\nFIVE\n");
|
||||
assert_eq!(laptop.read("notes/a.md"), "ONE\ntwo\nthree\nfour\nFIVE\n");
|
||||
|
||||
// The same line edited twice keeps both versions.
|
||||
phone.write("b.md", "phone\n");
|
||||
laptop.write("b.md", "laptop\n");
|
||||
phone.sync();
|
||||
let report = laptop.sync();
|
||||
assert_eq!(report.conflicts.len(), 1);
|
||||
phone.sync();
|
||||
assert_eq!(phone.read("b.md"), "phone\n");
|
||||
assert_eq!(phone.read(&report.conflicts[0]), "laptop\n");
|
||||
|
||||
// Delete on one device reaches the other; a second run is quiet.
|
||||
phone.vault.delete_to_trash("b.md").expect("delete");
|
||||
phone.sync();
|
||||
laptop.sync();
|
||||
assert!(laptop.vault.read("b.md").is_err());
|
||||
assert_eq!(laptop.sync(), notescraft_engine::SyncReport::default());
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
//! Shared helpers for the end-to-end tests: a server on an ephemeral port and a ureq client.
|
||||
//!
|
||||
//! Accounts: `paired` bootstraps a user-owned vault from a fresh admin pairing code
|
||||
//! (a code issued for a vault id the server does not know creates the vault),
|
||||
//! `create_user` and `pair_user` drive the admin user endpoints, `make_vault` creates
|
||||
//! a vault owned by a device's user, and `invite` helpers share vaults.
|
||||
#![allow(dead_code)] // every helper is used by one of the test binaries
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Mutex;
|
||||
|
||||
use notescraft_sync_server::{AppState, Store, router};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
pub const ADMIN: &str = "test-admin-token-0123456789abcdef";
|
||||
|
||||
/// A running server, a ureq client, and the secrets it must never leak.
|
||||
pub struct Server {
|
||||
pub base: String,
|
||||
pub agent: ureq::Agent,
|
||||
pub state: AppState,
|
||||
secrets: std::sync::Mutex<Vec<String>>,
|
||||
}
|
||||
|
||||
pub fn temp_dir(name: &str) -> PathBuf {
|
||||
let dir = std::env::temp_dir().join(format!("nc-sync-{name}-{}", std::process::id()));
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
std::fs::create_dir_all(dir.join("blobs")).expect("temp dir");
|
||||
dir
|
||||
}
|
||||
|
||||
/// Starts a server with the default quota.
|
||||
pub fn start(data: &Path) -> Server {
|
||||
start_with_state(AppState::new(Store::open(&data.join("sync.db")).expect("db"), data.join("blobs"), ADMIN))
|
||||
}
|
||||
|
||||
/// Runs the state the test has built; the temp folder comes from its store and blobs.
|
||||
pub fn start_with_state(state: AppState) -> Server {
|
||||
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind");
|
||||
listener.set_nonblocking(true).expect("nonblocking");
|
||||
let port = listener.local_addr().expect("addr").port();
|
||||
std::thread::spawn({
|
||||
let state = state.clone();
|
||||
move || {
|
||||
let runtime = tokio::runtime::Builder::new_multi_thread().enable_all().build().expect("runtime");
|
||||
runtime.block_on(async {
|
||||
let listener = tokio::net::TcpListener::from_std(listener).expect("listener");
|
||||
axum::serve(listener, router(state)).await.ok();
|
||||
});
|
||||
}
|
||||
});
|
||||
let agent: ureq::Agent = ureq::Agent::config_builder().http_status_as_error(false).build().into();
|
||||
Server { state, base: format!("http://127.0.0.1:{port}"), agent, secrets: Mutex::new(Vec::new()) }
|
||||
}
|
||||
|
||||
impl Server {
|
||||
/// Creates the default-state server over a fresh temp folder.
|
||||
pub fn start_temp(name: &str) -> Server {
|
||||
start(&temp_dir(name))
|
||||
}
|
||||
|
||||
/// Registers a secret so later checks can assert it never shows up in responses.
|
||||
pub fn leak(&self, secret: &str) {
|
||||
self.secrets.lock().expect("secrets").push(secret.to_owned());
|
||||
}
|
||||
|
||||
/// Asserts none of the registered secrets appear in `text`.
|
||||
pub fn assert_no_secrets(&self, text: &str) {
|
||||
for secret in self.secrets.lock().expect("secrets").iter() {
|
||||
assert!(!text.contains(secret.as_str()), "secret leaked: {secret}");
|
||||
}
|
||||
}
|
||||
|
||||
/// `(status, headers as (name, value) pairs, body)`.
|
||||
pub fn send_full(
|
||||
&self,
|
||||
method: &str,
|
||||
path: &str,
|
||||
token: Option<&str>,
|
||||
headers: &[(&str, &str)],
|
||||
body: Option<&[u8]>,
|
||||
) -> (u16, Vec<(String, String)>, Vec<u8>) {
|
||||
let url = format!("{}{path}", self.base);
|
||||
let mut builder = ureq::http::Request::builder().method(method).uri(url);
|
||||
if let Some(token) = token {
|
||||
builder = builder.header("Authorization", format!("Bearer {token}"));
|
||||
}
|
||||
for (name, value) in headers {
|
||||
builder = builder.header(*name, *value);
|
||||
}
|
||||
let request = builder.body(body.unwrap_or_default().to_vec()).expect("request");
|
||||
let mut response = self.agent.run(request).expect("transport");
|
||||
let heads = response.headers().iter().map(|(name, value)| (name.as_str().to_owned(), String::from_utf8_lossy(value.as_bytes()).into_owned())).collect();
|
||||
(response.status().as_u16(), heads, response.body_mut().read_to_vec().expect("body"))
|
||||
}
|
||||
|
||||
pub fn send(&self, method: &str, path: &str, token: Option<&str>, headers: &[(&str, &str)], body: Option<&[u8]>) -> (u16, Vec<u8>) {
|
||||
let (status, _, body) = self.send_full(method, path, token, headers, body);
|
||||
(status, body)
|
||||
}
|
||||
|
||||
pub fn json(&self, method: &str, path: &str, token: Option<&str>, headers: &[(&str, &str)], body: Option<&Value>) -> (u16, Value) {
|
||||
let bytes = body.map(|b| serde_json::to_vec(b).expect("json"));
|
||||
let mut all = vec![("Content-Type", "application/json")];
|
||||
all.extend_from_slice(headers);
|
||||
let (status, reply) = self.send(method, path, token, &all, bytes.as_deref());
|
||||
(status, serde_json::from_slice(&reply).unwrap_or(Value::Null))
|
||||
}
|
||||
|
||||
/// Creates a fresh vault id for admin pairing bootstraps (unique within a test run).
|
||||
pub fn fresh_vault_id(name: &str) -> String {
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
static NEXT: AtomicU32 = AtomicU32::new(0);
|
||||
let serial = NEXT.fetch_add(1, Ordering::Relaxed);
|
||||
format!("v-{name}-{serial}-{}", std::process::id())
|
||||
}
|
||||
|
||||
/// Creates a vault owned by a new user in one bootstrap: a fresh vault id, an admin
|
||||
/// pairing code for it, one redemption. `(vault_id, token, user_id)`.
|
||||
pub fn owned_vault(&self, name: &str) -> (String, String, String) {
|
||||
let vault = Self::fresh_vault_id(name);
|
||||
let (status, code) = self.json("POST", &format!("/v1/vaults/{vault}/pairing"), Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let (status, device, token) = self.redeem(code["code"].as_str().expect("code"), name);
|
||||
assert_eq!(status, 200, "{device}");
|
||||
(vault, token, device["user_id"].as_str().expect("user id").to_owned())
|
||||
}
|
||||
|
||||
/// An owner-only alias of [`Self::owned_vault`] for the earlier two-field callers.
|
||||
pub fn paired(&self, name: &str) -> (String, String) {
|
||||
let (vault, token, _) = self.owned_vault(name);
|
||||
(vault, token)
|
||||
}
|
||||
|
||||
/// Creates a user (admin) and returns its id.
|
||||
pub fn create_user(&self, name: &str) -> String {
|
||||
let (status, user) = self.json("POST", "/v1/users", Some(ADMIN), &[], Some(&json!({"name": name})));
|
||||
assert_eq!(status, 200);
|
||||
user["user_id"].as_str().expect("user id").to_owned()
|
||||
}
|
||||
|
||||
/// Pairs a device under an existing user via a `?user=` bound pairing code.
|
||||
pub fn pair_user(&self, user: &str, vault: &str, name: &str) -> (u16, String) {
|
||||
let (status, code) = self.json("POST", &format!("/v1/vaults/{vault}/pairing?user={user}"), Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let (status, reply, token) = self.redeem(code["code"].as_str().expect("code"), name);
|
||||
let _ = reply;
|
||||
(status, token)
|
||||
}
|
||||
|
||||
/// Creates a vault owned by `token`'s user: `(status, vault_id)`.
|
||||
pub fn make_vault(&self, token: &str, name: &str) -> (u16, String) {
|
||||
let (status, reply) = self.json("POST", "/v1/vaults", Some(token), &[], Some(&json!({"name": name})));
|
||||
(status, reply["vault_id"].as_str().unwrap_or_default().to_owned())
|
||||
}
|
||||
|
||||
/// Mints an invite from `vault` with `role` (owner `token`); registers the code as
|
||||
/// secret. `(status, code)`.
|
||||
pub fn invite(&self, vault: &str, token: &str, role: &str) -> (u16, String) {
|
||||
let (status, minted) = self.json("POST", &format!("/v1/vaults/{vault}/invites"), Some(token), &[], Some(&json!({"role": role})));
|
||||
let code = minted["code"].as_str().unwrap_or_default().to_owned();
|
||||
self.leak(&code);
|
||||
(status, code)
|
||||
}
|
||||
|
||||
/// The invite listing of a vault (owner `token`).
|
||||
pub fn invites(&self, vault: &str, token: &str) -> Vec<Value> {
|
||||
let (status, rows) = self.json("GET", &format!("/v1/vaults/{vault}/invites"), Some(token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
rows.as_array().expect("invite rows").clone()
|
||||
}
|
||||
|
||||
/// Accepts `code` with `token`; `(status, body)`.
|
||||
pub fn accept(&self, token: &str, code: &str) -> (u16, Value) {
|
||||
self.json("POST", "/v1/invites/accept", Some(token), &[], Some(&json!({"code": code})))
|
||||
}
|
||||
|
||||
/// Issues a pairing code for the vault and redeems it; registers both as secrets.
|
||||
pub fn pair_to(&self, vault_id: &str, name: &str) -> String {
|
||||
let (status, code) = self.json("POST", &format!("/v1/vaults/{vault_id}/pairing"), Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let (status, device, token) = self.redeem(code["code"].as_str().expect("code"), name);
|
||||
assert_eq!(status, 200, "{device}");
|
||||
self.leak(code["code"].as_str().expect("code"));
|
||||
token
|
||||
}
|
||||
|
||||
/// Redeems a code without registering secrets; returns `(status, body, token)`.
|
||||
pub fn redeem(&self, code: &str, name: &str) -> (u16, Value, String) {
|
||||
let (status, device) = self.json("POST", "/v1/devices", None, &[], Some(&json!({"code": code, "name": name})));
|
||||
let token = device["token"].as_str().unwrap_or_default().to_owned();
|
||||
(status, device, token)
|
||||
}
|
||||
|
||||
pub fn put(&self, vault: &str, token: &str, path: &str, body: &str, base_seq: u64) -> (u16, Value) {
|
||||
let (status, reply) =
|
||||
self.send("PUT", &format!("/v1/vaults/{vault}/files/{path}"), Some(token), &[("X-Base-Seq", &base_seq.to_string())], Some(body.as_bytes()));
|
||||
(status, serde_json::from_slice(&reply).unwrap_or(Value::Null))
|
||||
}
|
||||
|
||||
/// The audit rows (`/v1/audit`, admin), newest first.
|
||||
pub fn audit(&self) -> Vec<Value> {
|
||||
let (status, rows) = self.json("GET", "/v1/audit?limit=100", Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
rows.as_array().expect("audit rows").clone()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
//! Hardening: pairing lockout, device lifecycle, audit log, request ids and readiness.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use notescraft_sync_server::now_ms;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
const LOCKOUT_KIND: &str = "pairing_locked_out";
|
||||
const FAILED_KIND: &str = "pairing_failed";
|
||||
|
||||
fn kinds_of(rows: &[Value]) -> Vec<String> {
|
||||
rows.iter().map(|row| row["kind"].as_str().expect("kind").to_owned()).collect()
|
||||
}
|
||||
|
||||
fn body_text(body: &[u8]) -> String {
|
||||
String::from_utf8_lossy(body).into_owned()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lockout_after_five_bad_codes_answers_429_even_with_a_good_code() {
|
||||
let server = Server::start_temp("lockout");
|
||||
let (vault, _) = server.paired("keeper");
|
||||
for _ in 0..5 {
|
||||
let (status, _) = server.json("POST", "/v1/devices", None, &[], Some(&json!({"code": "bad", "name": "ghost"})));
|
||||
assert_eq!(status, 401);
|
||||
}
|
||||
|
||||
// A correct code is refused, too, with Retry-After.
|
||||
let (_, code) = server.json("POST", &format!("/v1/vaults/{vault}/pairing"), Some(ADMIN), &[], None);
|
||||
server.leak(code["code"].as_str().expect("code"));
|
||||
let request = serde_json::to_vec(&json!({"code": code["code"], "name": "late"})).expect("body");
|
||||
let (status, heads, body) = server.send_full("POST", "/v1/devices", None, &[("Content-Type", "application/json")], Some(&request));
|
||||
assert_eq!(status, 429, "{}", body_text(&body));
|
||||
let retry_after = heads.iter().find(|(name, _)| name == "retry-after").map(|(_, value)| value.as_str()).expect("retry-after header");
|
||||
assert!(retry_after.parse::<u64>().is_ok_and(|seconds| (1..=600).contains(&seconds)), "retry-after: {retry_after}");
|
||||
|
||||
// The lockout is audited; failed redemptions are, and none leaks the code.
|
||||
let kinds = kinds_of(&server.audit());
|
||||
assert_eq!(kinds.iter().filter(|k| *k == FAILED_KIND).count(), 5);
|
||||
assert_eq!(kinds.iter().filter(|k| k.as_str() == LOCKOUT_KIND).count(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pairing_works_again_once_the_lockout_window_has_passed() {
|
||||
let server = Server::start_temp("lockout-window");
|
||||
let state = server.state.clone();
|
||||
// Vault and code come first, then failure stamps a second before the code existed.
|
||||
let vault = Server::fresh_vault_id("lockout-window");
|
||||
let (status, code) = server.json("POST", &format!("/v1/vaults/{vault}/pairing"), Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let good = code["code"].as_str().expect("code").to_owned();
|
||||
let base = now_ms().saturating_sub(1000);
|
||||
for _ in 0..notescraft_sync_server::PAIRING_MAX_FAILURES {
|
||||
let error = match state.pair_device("garbage", "ghost", None, base) {
|
||||
Err(error) => error,
|
||||
Ok(paired) => panic!("redeemed without a code: {paired:?}"),
|
||||
};
|
||||
assert_eq!(error.status.as_u16(), 401);
|
||||
}
|
||||
|
||||
let error = match state.pair_device(&good, "late", None, base) {
|
||||
Err(error) => error,
|
||||
Ok(paired) => panic!("redeemed while locked: {paired:?}"),
|
||||
};
|
||||
assert_eq!(error.status.as_u16(), 429);
|
||||
assert!(error.retry_after.is_some());
|
||||
|
||||
// Nothing was consumed; with `now` past the window the same code redeems.
|
||||
let paired = state.pair_device(&good, "late", None, base + notescraft_sync_server::PAIRING_WINDOW_MS + 1).expect("redeemed");
|
||||
assert_eq!(paired.vault_id, vault);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rotation_gives_a_new_token_and_the_old_one_stops_working() {
|
||||
let server = Server::start_temp("rotate");
|
||||
let (vault, token) = server.paired("phone");
|
||||
let changes = format!("/v1/vaults/{vault}/changes?since=0");
|
||||
assert_eq!(server.send("GET", &changes, Some(&token), &[], None).0, 200);
|
||||
|
||||
let (status, rotated) = server.json("POST", &format!("/v1/vaults/{vault}/devices/me/rotate"), Some(&token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let fresh = rotated["token"].as_str().expect("token").to_owned();
|
||||
server.leak(&fresh);
|
||||
assert_ne!(fresh, token);
|
||||
assert_ne!(rotated["device_id"].as_str().expect("device id"), "");
|
||||
|
||||
assert_eq!(server.send("GET", &changes, Some(&token), &[], None).0, 401);
|
||||
assert_eq!(server.send("GET", &changes, Some(&fresh), &[], None).0, 200);
|
||||
// The rotation is audited with the device id, never the token.
|
||||
let kinds = kinds_of(&server.audit());
|
||||
assert!(kinds.iter().any(|k| k == "token_rotated"));
|
||||
server.assert_no_secrets(&serde_json::to_string(&server.audit()).expect("rows"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn revocation_kills_the_device_token_and_the_list_hides_secrets() {
|
||||
let server = Server::start_temp("revoke");
|
||||
let (vault, phone) = server.paired("phone");
|
||||
let laptop = server.pair_to(&vault, "laptop");
|
||||
server.leak(&laptop);
|
||||
let list_url = format!("/v1/vaults/{vault}/devices");
|
||||
|
||||
let (_, devices) = server.json("GET", &list_url, Some(&laptop), &[], None);
|
||||
let rows = devices.as_array().expect("device rows");
|
||||
assert_eq!(rows.len(), 2);
|
||||
let laptop_row = rows.iter().find(|row| row["name"] == "laptop").expect("laptop row");
|
||||
let phone_row = rows.iter().find(|row| row["name"] == "phone").expect("phone row");
|
||||
assert_eq!(laptop_row["current"], true);
|
||||
assert_eq!(phone_row["current"], false);
|
||||
server.assert_no_secrets(&serde_json::to_string(rows).expect("rows text"));
|
||||
for wanted in ["id", "name", "created_ms", "last_seen_ms", "current"] {
|
||||
assert!(laptop_row.get(wanted).is_some(), "missing field {wanted}");
|
||||
}
|
||||
|
||||
let laptop_id = laptop_row["id"].as_str().expect("id").to_owned();
|
||||
let (status, _) = server.send("DELETE", &format!("{list_url}/{laptop_id}"), Some(&phone), &[], None);
|
||||
assert_eq!(status, 204);
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(&laptop), &[], None).0, 401);
|
||||
assert_eq!(server.send("DELETE", &format!("{list_url}/{laptop_id}"), Some(&phone), &[], None).0, 404);
|
||||
|
||||
// A device may revoke itself.
|
||||
let phone_id = phone_row["id"].as_str().expect("id").to_owned();
|
||||
assert_eq!(server.json("GET", &list_url, Some(&phone), &[], None).1.as_array().expect("rows").len(), 1);
|
||||
let (status, _) = server.send("DELETE", &format!("{list_url}/{phone_id}"), Some(&phone), &[], None);
|
||||
assert_eq!(status, 204);
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(&phone), &[], None).0, 401);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn audit_records_events_in_order_and_without_secrets() {
|
||||
let server = Server::start_temp("audit");
|
||||
let (vault, token) = server.paired("phone");
|
||||
server.leak(&token);
|
||||
let (status, rotated) = server.json("POST", &format!("/v1/vaults/{vault}/devices/me/rotate"), Some(&token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
server.leak(rotated["token"].as_str().expect("token"));
|
||||
|
||||
// One failed redemption, so `pairing_failed` is in the log too.
|
||||
let (status, _) = server.json("POST", "/v1/devices", None, &[], Some(&json!({"code": "nope", "name": "ghost"})));
|
||||
assert_eq!(status, 401);
|
||||
assert_eq!(server.send("GET", "/v1/audit?limit=100", Some("not-admin"), &[], None).0, 401);
|
||||
|
||||
let rows = server.audit();
|
||||
let kinds = kinds_of(&rows);
|
||||
for wanted in ["vault_created", "pairing_created", "device_paired", FAILED_KIND, "token_rotated"] {
|
||||
assert!(kinds.iter().any(|k| k == wanted), "no {wanted} row; kinds: {kinds:?}");
|
||||
}
|
||||
// Newest first.
|
||||
let ids: Vec<i64> = rows.iter().map(|row| row["id"].as_i64().expect("id")).collect();
|
||||
let sorted: Vec<i64> = ids.clone();
|
||||
assert_eq!(ids, sorted, "ids in insertion order");
|
||||
let mut descending = ids.clone();
|
||||
descending.sort_unstable();
|
||||
descending.reverse();
|
||||
assert_eq!(ids, descending);
|
||||
// The failed row names the reason, never the code.
|
||||
let failed = rows.iter().find(|row| row["kind"] == FAILED_KIND).expect("failed row");
|
||||
assert_eq!(failed["detail"], "unknown_or_expired");
|
||||
server.assert_no_secrets(&serde_json::to_string(&rows).expect("rows text"));
|
||||
}
|
||||
#[test]
|
||||
fn every_answer_carries_an_echoed_or_generated_request_id() {
|
||||
let server = Server::start_temp("request-id");
|
||||
let id_of =
|
||||
|heads: &[(String, String)]| heads.iter().find(|(name, _)| name == "x-request-id").map(|(_, value)| value.clone()).expect("x-request-id header");
|
||||
let hex16 = |id: String| id.len() == 16 && id.chars().all(|c| c.is_ascii_hexdigit());
|
||||
|
||||
let (status, heads, _) = server.send_full("GET", "/healthz", None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert!(hex16(id_of(&heads)), "generated id is 16 hex chars: {}", id_of(&heads));
|
||||
|
||||
// A valid incoming id (at most 64 chars of [A-Za-z0-9-]) is echoed back.
|
||||
let valid = "Echo-Id-01234567890123456789012345678901234567890123456789";
|
||||
assert_eq!(valid.len(), 58);
|
||||
let (status, heads, _) = server.send_full("GET", "/healthz", None, &[("X-Request-Id", valid)], None);
|
||||
assert_eq!((status, id_of(&heads)), (200, valid.to_owned()));
|
||||
// Boundary: exactly 64 characters is still valid.
|
||||
let boundary = "a".repeat(64);
|
||||
let (status, heads, _) = server.send_full("GET", "/healthz", None, &[("X-Request-Id", boundary.as_str())], None);
|
||||
assert_eq!((status, id_of(&heads)), (200, boundary));
|
||||
|
||||
// Invalid ids are replaced by a generated one.
|
||||
for bad in ["", "no/slash", "no space", "a".repeat(65).as_str()] {
|
||||
let (status, heads, _) = server.send_full("GET", "/healthz", None, &[("X-Request-Id", bad)], None);
|
||||
assert_eq!(status, 200);
|
||||
let id = id_of(&heads);
|
||||
assert_ne!(id, bad);
|
||||
assert!(hex16(id.clone()), "generated replacement: {id}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn readyz_checks_database_and_blob_folder() {
|
||||
let data = temp_dir("readyz");
|
||||
let server = common::start(&data);
|
||||
let (status, reply) = server.json("GET", "/readyz", None, &[], None);
|
||||
assert_eq!((status, reply["ok"].clone()), (200, Value::Bool(true)));
|
||||
assert_eq!(reply.get("failing"), None);
|
||||
|
||||
// A read-only blob folder fails the readiness probe (unix only).
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let blobs = data.join("blobs");
|
||||
std::fs::set_permissions(&blobs, std::fs::Permissions::from_mode(0o555)).expect("chmod blobs");
|
||||
let (status, reply) = server.json("GET", "/readyz", None, &[], None);
|
||||
assert_eq!(status, 503);
|
||||
assert_eq!(reply["ok"], Value::Bool(false));
|
||||
assert_eq!(reply["failing"], "blobs");
|
||||
std::fs::set_permissions(&blobs, std::fs::Permissions::from_mode(0o755)).expect("chmod back");
|
||||
let (status, _) = server.json("GET", "/readyz", None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
std::fs::remove_dir_all(&data).expect("cleanup");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
//! File history over real HTTP: every applied write is a revision, listing and reading
|
||||
//! old revisions, restore, and the retention rules the GC applies. Non-members are 404,
|
||||
//! viewers read but never restore.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use notescraft_sync_server::{AppState, GcReport, Store, TOMBSTONE_GRACE_MS, now_ms};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
/// A server over a fresh temp folder; the folder comes back for blob assertions.
|
||||
fn server(name: &str) -> (Server, PathBuf) {
|
||||
let data = temp_dir(name);
|
||||
let state = AppState::new(Store::open(&data.join("sync.db")).expect("db"), data.join("blobs"), ADMIN);
|
||||
(common::start_with_state(state), data)
|
||||
}
|
||||
|
||||
/// A fresh vault with an owner token.
|
||||
fn vault(server: &Server, name: &str) -> (String, String) {
|
||||
server.paired(name)
|
||||
}
|
||||
|
||||
/// Writes `count` successive versions of one file, the old sequence as the base each
|
||||
/// time; `(seq, body, hash)` in write order.
|
||||
fn edits(server: &Server, vault: &str, token: &str, count: u64) -> Vec<(u64, String, String)> {
|
||||
let mut base_seq = 0;
|
||||
(1..=count)
|
||||
.map(|index| {
|
||||
let body = format!("body-{index}");
|
||||
let (status, record) = server.put(vault, token, "a.md", &body, base_seq);
|
||||
assert_eq!(status, 201, "write {index}");
|
||||
let seq = record["seq"].as_u64().expect("seq");
|
||||
let hash = record["hash"].as_str().expect("hash").to_owned();
|
||||
base_seq = seq;
|
||||
(seq, body, hash)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The vault's own row of `GET /v1/vaults`.
|
||||
fn listing(server: &Server, token: &str, vault: &str) -> Value {
|
||||
let (status, rows) = server.json("GET", "/v1/vaults", Some(token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
rows.as_array().expect("vaults").iter().find(|row| row["id"] == *vault).expect("vault row").clone()
|
||||
}
|
||||
|
||||
/// The bytes and headers of a `GET` of one file with optional `?seq=`.
|
||||
fn read(server: &Server, vault: &str, token: &str, query: &str) -> (u16, String, Vec<u8>) {
|
||||
let (status, headers, body) = server.send_full("GET", &format!("/v1/vaults/{vault}/files/a.md{query}"), Some(token), &[], None);
|
||||
let etag = headers.iter().find(|(name, _)| name == "etag").map(|(_, value)| value.clone()).unwrap_or_default();
|
||||
(status, etag, body)
|
||||
}
|
||||
|
||||
fn history(server: &Server, vault: &str, token: &str, query: &str) -> Vec<Value> {
|
||||
let (status, rows) = server.json("GET", &format!("/v1/vaults/{vault}/files-history/a.md{query}"), Some(token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
rows.as_array().expect("history rows").clone()
|
||||
}
|
||||
|
||||
fn restore(server: &Server, vault: &str, token: &str, seq: u64) -> (u16, Value) {
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/files-restore/a.md"), Some(token), &[], Some(&json!({"seq": seq})))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn forty_edits_list_forty_revisions_newest_first_with_a_limit() {
|
||||
let (server, _) = server("history-list");
|
||||
let (vault, token) = vault(&server, "hist");
|
||||
let writes = edits(&server, &vault, &token, 40);
|
||||
assert_eq!(writes.iter().map(|(seq, _, _)| *seq).max(), Some(40));
|
||||
|
||||
let rows = history(&server, &vault, &token, "");
|
||||
assert_eq!(rows.len(), 40);
|
||||
assert_eq!(rows[0]["seq"], 40, "newest first");
|
||||
assert_eq!(rows[39]["seq"], 1);
|
||||
assert_eq!(rows[0]["deleted"], false);
|
||||
assert!(rows[0]["size"].as_u64().is_some_and(|n| n > 0), "sizes come along");
|
||||
assert!(rows[0]["mtime_ms"].as_u64().is_some_and(|n| n > 0));
|
||||
assert!(rows[0]["recorded_ms"].as_u64().is_some_and(|n| n > 0));
|
||||
assert!(rows[0]["editor_user"].as_str().is_some(), "the editor is stamped");
|
||||
assert!(rows[0]["editor_name"].as_str().is_some(), "the editor's name resolves");
|
||||
|
||||
let limited = history(&server, &vault, &token, "?limit=10");
|
||||
assert_eq!(limited.len(), 10);
|
||||
assert_eq!(limited.iter().map(|row| row["seq"].as_u64().expect("seq")).collect::<Vec<_>>(), (31..=40).rev().collect::<Vec<_>>());
|
||||
assert_eq!(history(&server, &vault, &token, "?limit=0").len(), 1, "0 clamps up to one entry");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn get_by_seq_returns_that_revisions_bytes() {
|
||||
let (server, _) = server("history-read");
|
||||
let (vault, token) = vault(&server, "hist");
|
||||
let writes = edits(&server, &vault, &token, 12);
|
||||
let (_, body12, hash12) = writes.iter().find(|(seq, _, _)| *seq == 12).expect("revision 12").clone();
|
||||
|
||||
let (status, etag, bytes) = read(&server, &vault, &token, "?seq=12");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(etag, hash12);
|
||||
assert_eq!(bytes, body12.as_bytes());
|
||||
assert_eq!(read(&server, &vault, &token, "").1, hash12, "without a seq it reads the live record");
|
||||
assert_eq!(read(&server, &vault, &token, "?seq=999").0, 404, "unknown revision");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restore_of_revision_12_creates_revision_41_with_those_bytes() {
|
||||
let (server, _) = server("history-restore");
|
||||
let (vault, token) = vault(&server, "hist");
|
||||
let writes = edits(&server, &vault, &token, 40);
|
||||
let (_, body12, hash12) = writes.iter().find(|(seq, _, _)| *seq == 12).expect("revision 12").clone();
|
||||
|
||||
let (status, record) = restore(&server, &vault, &token, 12);
|
||||
assert_eq!(status, 201);
|
||||
assert_eq!(record["seq"].as_u64(), Some(41));
|
||||
assert_eq!(record["hash"].as_str(), Some(hash12.as_str()));
|
||||
assert_eq!(record["size"].as_u64(), Some(u64::try_from(body12.len()).expect("small")));
|
||||
|
||||
let rows = history(&server, &vault, &token, "");
|
||||
assert_eq!(rows.len(), 41, "history is never rewritten: 41 entries");
|
||||
assert_eq!(rows[0]["seq"], 41);
|
||||
|
||||
let (status, etag, bytes) = read(&server, &vault, &token, "");
|
||||
assert_eq!((status, etag, bytes), (200, hash12.clone(), body12.into_bytes()), "the file reads back as revision 12");
|
||||
|
||||
let audit = server.audit();
|
||||
assert!(audit.iter().any(|row| row["kind"] == "file_restored"), "the restore is audited");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restore_of_a_deleted_file_recreates_it() {
|
||||
let (server, _) = server("history-recreate");
|
||||
let (vault, token) = vault(&server, "hist");
|
||||
assert_eq!(server.put(&vault, &token, "a.md", "body-1", 0).0, 201);
|
||||
let (status, _) = server.json("DELETE", &format!("/v1/vaults/{vault}/files/a.md"), Some(&token), &[("X-Base-Seq", "1")], None);
|
||||
assert_eq!(status, 201);
|
||||
assert_eq!(read(&server, &vault, &token, "").0, 404, "the file is gone");
|
||||
|
||||
let (status, record) = restore(&server, &vault, &token, 1);
|
||||
assert_eq!(status, 201, "the tombstone yields to an edit");
|
||||
assert_eq!(record["seq"], 3, "one for the write, one for the delete");
|
||||
assert_eq!(record["deleted"], false);
|
||||
assert_eq!((read(&server, &vault, &token, "").0, read(&server, &vault, &token, "").2), (200, b"body-1".to_vec()));
|
||||
assert_eq!(history(&server, &vault, &token, "").len(), 3, "both the tombstone and the recreate stay listed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_seq_is_404_viewer_reads_but_cannot_restore_and_a_non_member_is_404() {
|
||||
let (server, _) = server("history-roles");
|
||||
let (vault, owner) = vault(&server, "hist");
|
||||
edits(&server, &vault, &owner, 3);
|
||||
|
||||
let (_, viewer) = server.paired("hist-viewer");
|
||||
let (status, code) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(&viewer, &code).0, 200);
|
||||
let (_, stranger) = server.paired("hist-stranger");
|
||||
|
||||
assert_eq!(restore(&server, &vault, &viewer, 1).0, 403, "a viewer cannot restore");
|
||||
assert_eq!(restore(&server, &vault, &viewer, 999).0, 403, "role first, never existence");
|
||||
assert_eq!(restore(&server, &vault, &owner, 999).0, 404, "unknown revision");
|
||||
assert_eq!(read(&server, &vault, &owner, "?seq=999").0, 404, "unknown revision read");
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/files-history/a.md"), Some(&stranger), &[], None).0, 404, "a non-member");
|
||||
assert_eq!(restore_status(&server, &vault, &stranger, 1), 404);
|
||||
}
|
||||
|
||||
/// The raw restore status, for assertions where the body does not parse as JSON.
|
||||
fn restore_status(server: &Server, vault: &str, token: &str, seq: u64) -> u16 {
|
||||
server
|
||||
.send(
|
||||
"POST",
|
||||
&format!("/v1/vaults/{vault}/files-restore/a.md"),
|
||||
Some(token),
|
||||
&[("Content-Type", "application/json")],
|
||||
Some(serde_json::to_vec(&json!({"seq": seq})).expect("body").as_slice()),
|
||||
)
|
||||
.0
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retention_days_0_and_366_are_refused() {
|
||||
let (server, _) = server("history-bounds");
|
||||
let (vault, token) = vault(&server, "hist");
|
||||
for days in [0, 366] {
|
||||
let (status, _) = server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(&token), &[], Some(&json!({"retention_days": days})));
|
||||
assert_eq!(status, 400, "retention_days = {days}");
|
||||
}
|
||||
assert_eq!(server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(&token), &[], Some(&json!({"retention_days": 365}))).0, 204, "the range top is fine");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retention_gc_removes_expired_unique_blobs_and_keeps_the_newest_20() {
|
||||
let (server, data) = server("history-retention");
|
||||
let (vault, token) = vault(&server, "hist");
|
||||
// Revision 2 shares its content with revision 30: its blob must survive.
|
||||
let bodies: Vec<String> = (1..=40).map(|index| if index == 2 { "body-30".to_owned() } else { format!("body-{index}") }).collect();
|
||||
let mut writes = Vec::new();
|
||||
let mut base_seq = 0;
|
||||
for body in &bodies {
|
||||
let (status, record) = server.put(&vault, &token, "a.md", body, base_seq);
|
||||
assert_eq!(status, 201);
|
||||
base_seq = record["seq"].as_u64().expect("seq");
|
||||
writes.push((base_seq, body.clone(), record["hash"].as_str().expect("hash").to_owned()));
|
||||
}
|
||||
|
||||
let dir = data.join("blobs").join(&vault);
|
||||
let (hash2, hash30) = (writes[1].2.clone(), writes[29].2.clone());
|
||||
assert_eq!(hash2, hash30, "the shared content is one blob");
|
||||
assert_eq!(listing(&server, &token, &vault)["bytes_used"].as_u64(), Some(265), "39 distinct contents");
|
||||
|
||||
let (status, _) = server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(&token), &[], Some(&json!({"retention_days": 1})));
|
||||
assert_eq!(status, 204);
|
||||
let aged = now_ms() + 2 * 24 * 60 * 60 * 1000;
|
||||
let report = server.state.gc(&vault, TOMBSTONE_GRACE_MS, aged).expect("gc");
|
||||
assert_eq!(report, GcReport { removed: 19, bytes: 125 }, "the expired unique blobs went, the shared one stayed");
|
||||
|
||||
let rows = history(&server, &vault, &token, "");
|
||||
assert_eq!(rows.len(), 20, "the current one and the newest 20 stay");
|
||||
assert_eq!(rows.first().and_then(|row| row["seq"].as_u64()), Some(40));
|
||||
assert_eq!(rows.last().and_then(|row| row["seq"].as_u64()), Some(21));
|
||||
assert!(dir.join(&hash2).exists(), "a blob shared by a kept revision survives");
|
||||
assert!(dir.join(&writes[39].2).exists(), "the current content stays");
|
||||
assert!(!dir.join(&writes[0].2).exists(), "an expired revision's unique blob is gone");
|
||||
assert_eq!(listing(&server, &token, &vault)["bytes_used"].as_u64(), Some(140), "20 distinct contents of the kept revisions");
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
//! Vault lifecycle over real HTTP: soft delete hides and blocks, restore undoes it
|
||||
//! inside the window, an injected clock drives the purge (rows and blob folder gone),
|
||||
//! file authorship is readable, and the audit log holds every lifecycle event without
|
||||
//! a single secret.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use notescraft_sync_server::{PurgeReport, TOMBSTONE_GRACE_MS};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
#[test]
|
||||
fn a_soft_deleted_vault_serves_404_and_restores() {
|
||||
let server = Server::start_temp("lifecycle-delete");
|
||||
let (vault, owner, _) = server.owned_vault("delete");
|
||||
assert_eq!(server.put(&vault, &owner, "a.md", "x", 0).0, 201);
|
||||
|
||||
// Views and writes are 404 while deleted (owner included; restore is the exception).
|
||||
assert_eq!(server.send("DELETE", &format!("/v1/vaults/{vault}"), Some(&owner), &[], None).0, 204);
|
||||
assert_eq!(server.send("DELETE", &format!("/v1/vaults/{vault}"), Some(&owner), &[], None).0, 404, "never twice");
|
||||
assert_eq!(server.json("GET", "/v1/vaults", Some(&owner), &[], None).1.as_array().expect("rows").len(), 0, "hidden from lists");
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(&owner), &[], None).0, 404);
|
||||
assert_eq!(server.put(&vault, &owner, "a.md", "y", 1).0, 404);
|
||||
assert_eq!(server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(&owner), &[], Some(&json!({"name": "again"}))).0, 404);
|
||||
|
||||
// A stranger still cannot see it; the admin listing shows the deletion mark.
|
||||
let (intruder_vault, intruder) = server.paired("intruder");
|
||||
let _ = intruder_vault;
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(&intruder), &[], None).0, 404);
|
||||
let (status, admin_rows) = server.json("GET", "/v1/admin/vaults", Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let rows = admin_rows.as_array().expect("rows");
|
||||
let row = rows.iter().find(|row| row["id"] == vault.as_str()).expect("row");
|
||||
assert!(row["deleted_ms"].as_u64().is_some(), "marked deleted: {row}");
|
||||
|
||||
// The owner restores inside the window; writes work again.
|
||||
assert_eq!(server.json("POST", &format!("/v1/vaults/{vault}/restore"), Some(&owner), &[], None).0, 204);
|
||||
let (_, mine) = server.json("GET", "/v1/vaults", Some(&owner), &[], None);
|
||||
assert_eq!(mine.as_array().expect("rows").len(), 1);
|
||||
assert_eq!(server.put(&vault, &owner, "a.md", "y", 1).0, 201);
|
||||
let (status, admin_rows) = server.json("GET", "/v1/admin/vaults", Some(ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let row = admin_rows.as_array().expect("rows").iter().find(|row| row["id"] == vault.as_str()).expect("row");
|
||||
assert_eq!(row["deleted_ms"], Value::Null, "restored: {row}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restore_window_closes_when_the_purge_removes_the_vault() {
|
||||
let dir = temp_dir("lifecycle-purge");
|
||||
let server = common::start(&dir);
|
||||
let (vault, owner, _) = server.owned_vault("purge");
|
||||
assert_eq!(server.put(&vault, &owner, "gone.md", "blob content", 0).0, 201);
|
||||
let blobs = dir.join("blobs").join(&vault);
|
||||
assert_eq!(std::fs::read_dir(&blobs).expect("blob dir").count(), 1, "the blob exists");
|
||||
|
||||
assert_eq!(server.send("DELETE", &format!("/v1/vaults/{vault}"), Some(&owner), &[], None).0, 204);
|
||||
// The admin route at the real clock: the vault is minutes old, nothing to purge.
|
||||
let (status, report) = server.json("POST", "/v1/admin/purge", Some(ADMIN), &[], None);
|
||||
assert_eq!((status, report["removed"].as_u64()), (200, Some(0)));
|
||||
|
||||
// With the injected clock past the window the vault, its rows and folder are gone.
|
||||
let aged = notescraft_sync_server::now_ms() + TOMBSTONE_GRACE_MS + 1;
|
||||
let report = server.state.purge(aged).expect("purge");
|
||||
assert_eq!(report, PurgeReport { removed: 1 });
|
||||
assert!(!blobs.exists(), "the blob folder is gone");
|
||||
assert_eq!(server.state.purge(aged + 1).expect("purge"), PurgeReport { removed: 0 }, "purging twice is quiet");
|
||||
assert_eq!(server.json("POST", &format!("/v1/vaults/{vault}/restore"), Some(&owner), &[], None).0, 404, "restore is too late");
|
||||
// The purged vault vanished from the admin listing as well.
|
||||
let (_, rows) = server.json("GET", "/v1/admin/vaults", Some(ADMIN), &[], None);
|
||||
assert!(rows.as_array().expect("rows").iter().all(|row| row["id"] != vault.as_str()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn files_meta_records_the_last_editor() {
|
||||
let server = Server::start_temp("lifecycle-meta");
|
||||
let (vault, owner, owner_user) = server.owned_vault("meta");
|
||||
assert_eq!(server.put(&vault, &owner, "a.md", "first", 0).0, 201);
|
||||
let (_, meta) = server.json("GET", &format!("/v1/vaults/{vault}/files-meta/a.md"), Some(&owner), &[], None);
|
||||
assert_eq!(meta["editor_user"], owner_user);
|
||||
assert_eq!(meta["seq"], 1);
|
||||
assert_eq!(meta["editor_name"].as_str().expect("name"), "meta"); // the bootstrapped user's device name
|
||||
|
||||
// An invited editor's write carries their user id and name.
|
||||
let (stranger_vault, stranger) = server.paired("member");
|
||||
let _ = stranger_vault;
|
||||
let (status, code) = server.invite(&vault, &owner, "editor");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(&stranger, &code).0, 200);
|
||||
assert_eq!(server.put(&vault, &stranger, "b.md", "theirs", 0).0, 201);
|
||||
let (_, meta) = server.json("GET", &format!("/v1/vaults/{vault}/files-meta/b.md"), Some(&owner), &[], None);
|
||||
assert_ne!(meta["editor_user"], owner_user);
|
||||
assert_eq!(meta["editor_name"], "member");
|
||||
assert_eq!(meta["seq"], 2);
|
||||
|
||||
// Reads: any member; strangers 404; missing and deleted files 404.
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/files-meta/b.md"), Some(&stranger), &[], None).0, 200);
|
||||
let (ghost_vault, ghost) = server.paired("ghost");
|
||||
let _ = ghost_vault;
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/files-meta/b.md"), Some(&ghost), &[], None).0, 404, "a stranger");
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/files-meta/missing.md"), Some(&owner), &[], None).0, 404);
|
||||
let (status, record) = server.json("DELETE", &format!("/v1/vaults/{vault}/files/b.md"), Some(&stranger), &[("X-Base-Seq", "2")], None);
|
||||
assert_eq!(status, 201, "{record}");
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/files-meta/b.md"), Some(&owner), &[], None).0, 404, "a tombstone has no meta");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_audit_log_has_every_lifecycle_event_and_no_secrets() {
|
||||
let server = Server::start_temp("lifecycle-audit");
|
||||
// A user id is not a secret (it names rows); tokens, codes and hashes are.
|
||||
let (vault, owner, owner_user) = server.owned_vault("audited");
|
||||
server.leak(&owner);
|
||||
let _ = owner_user;
|
||||
|
||||
// Rename the owner's own vault; mint, accept and withdraw an invite on it.
|
||||
assert_eq!(server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(&owner), &[], Some(&json!({"name": "renamed"}))).0, 204);
|
||||
let (status, code) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
let (guest_vault, guest) = server.paired("guest");
|
||||
let _ = guest_vault;
|
||||
assert_eq!(server.accept(&guest, &code).0, 200);
|
||||
let (status, withdraw) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
assert!(withdraw.len() > 3);
|
||||
let id = server.invites(&vault, &owner)[0]["id"].as_str().expect("id").to_owned();
|
||||
assert_eq!(server.send("DELETE", &format!("/v1/vaults/{vault}/invites/{id}"), Some(&owner), &[], None).0, 204);
|
||||
assert_eq!(server.send("DELETE", &format!("/v1/vaults/{vault}"), Some(&owner), &[], None).0, 204);
|
||||
assert_eq!(server.json("POST", &format!("/v1/vaults/{vault}/restore"), Some(&owner), &[], None).0, 204);
|
||||
server.leak(&code);
|
||||
server.leak(&withdraw);
|
||||
|
||||
// A role change and a removal are logged too; the guest's user id from the rows.
|
||||
let (_, members) = server.json("GET", &format!("/v1/vaults/{vault}/members"), Some(&owner), &[], None);
|
||||
let guest_user =
|
||||
members.as_array().expect("members").iter().find(|row| row["role"] == "viewer").expect("guest row")["user_id"].as_str().expect("id").to_owned();
|
||||
assert_eq!(server.json("PATCH", &format!("/v1/vaults/{vault}/members/{guest_user}"), Some(&owner), &[], Some(&json!({"role": "editor"}))).0, 204);
|
||||
assert_eq!(server.json("DELETE", &format!("/v1/vaults/{vault}/members/{guest_user}"), Some(&owner), &[], None).0, 204);
|
||||
|
||||
let kinds: Vec<String> = server.audit().iter().map(|row| row["kind"].as_str().expect("kind").to_owned()).collect();
|
||||
for wanted in [
|
||||
"user_created",
|
||||
"member_added",
|
||||
"vault_created",
|
||||
"vault_renamed",
|
||||
"invite_created",
|
||||
"invite_accepted",
|
||||
"invite_deleted",
|
||||
"vault_deleted",
|
||||
"vault_restored",
|
||||
] {
|
||||
assert!(kinds.iter().any(|k| k == wanted), "no {wanted} in {kinds:?}");
|
||||
}
|
||||
// Invite lockout and purge events come from a dedicated flow on a fresh server.
|
||||
let purged = Server::start_temp("lifecycle-audit-purge");
|
||||
let (purge_vault, purge_owner, _) = purged.owned_vault("doomed");
|
||||
assert_eq!(purged.send("DELETE", &format!("/v1/vaults/{purge_vault}"), Some(&purge_owner), &[], None).0, 204);
|
||||
assert_eq!(purged.state.purge(notescraft_sync_server::now_ms() + TOMBSTONE_GRACE_MS + 1).expect("purge").removed, 1);
|
||||
assert!(purged.audit().iter().any(|row| row["kind"] == "vault_purged"));
|
||||
purged.assert_no_secrets(&serde_json::to_string(&purged.audit()).expect("rows text"));
|
||||
server.assert_no_secrets(&serde_json::to_string(&server.audit()).expect("rows text"));
|
||||
}
|
||||
@@ -0,0 +1,425 @@
|
||||
//! The live endpoints over real HTTP: rooms, operations, the event stream, presence
|
||||
//! and the flush of a room's text into the vault.
|
||||
//!
|
||||
//! Time is injected, never waited for: the flush is called directly and the presence
|
||||
//! is pruned with an explicit clock. The event stream is read by a thread that takes
|
||||
//! the Server-Sent Events line by line.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::io::BufRead;
|
||||
use std::sync::mpsc::Receiver;
|
||||
use std::time::Duration;
|
||||
|
||||
use common::Server;
|
||||
use notescraft_collab::{Client, HISTORY_CAP, TextOp};
|
||||
use notescraft_sync_server::{AppState, Store, Timings, flush, now_ms};
|
||||
use serde_json::{Value, json};
|
||||
|
||||
/// The live windows the tests use: a short keepalive, no background sweep.
|
||||
fn windows() -> Timings {
|
||||
Timings { keepalive_ms: 200, idle_ms: 5_000, presence_ttl_ms: 30_000, flush_debounce_ms: 60_000, sweep_ms: 60_000, max_rooms: 8, max_peers: 64 }
|
||||
}
|
||||
|
||||
/// A server over a fresh folder with the given live windows.
|
||||
fn server(name: &str, windows: Timings) -> Server {
|
||||
let dir = common::temp_dir(name);
|
||||
let state = AppState::new(Store::open(&dir.join("sync.db")).expect("db"), dir.join("blobs"), common::ADMIN);
|
||||
state.set_live_timings(windows);
|
||||
common::start_with_state(state)
|
||||
}
|
||||
|
||||
/// `POST .../join`: the snapshot a peer adopts.
|
||||
fn join(server: &Server, vault: &str, token: &str, path: &str, client: &str, name: &str) -> (u16, Value) {
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/live/{path}/join"), Some(token), &[], Some(&json!({"client": client, "name": name})))
|
||||
}
|
||||
|
||||
/// `POST .../ops`: one operation of a peer.
|
||||
fn ops(server: &Server, vault: &str, token: &str, path: &str, client: &str, base_rev: u64, op: &TextOp) -> (u16, Value) {
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/live/{path}/ops"), Some(token), &[], Some(&json!({"client": client, "base_rev": base_rev, "op": op})))
|
||||
}
|
||||
|
||||
/// The event stream of a peer, read line by line in a thread.
|
||||
struct Stream {
|
||||
lines: Receiver<String>,
|
||||
}
|
||||
|
||||
impl Stream {
|
||||
/// Opens the stream of a peer that already joined the room.
|
||||
fn open(server: &Server, vault: &str, token: &str, path: &str, client: &str) -> Stream {
|
||||
let url = format!("{}/v1/vaults/{vault}/live/{path}/events?client={client}", server.base);
|
||||
let token = token.to_owned();
|
||||
let agent = server.agent.clone();
|
||||
let (tx, lines) = std::sync::mpsc::channel();
|
||||
std::thread::spawn(move || {
|
||||
let Ok(response) = agent.get(&url).header("Authorization", format!("Bearer {token}")).call() else { return };
|
||||
let reader = std::io::BufReader::new(response.into_body().into_reader());
|
||||
for line in reader.lines() {
|
||||
let Ok(line) = line else { break };
|
||||
if tx.send(line).is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
Stream { lines }
|
||||
}
|
||||
|
||||
/// The next raw line of the stream.
|
||||
fn line(&self) -> Option<String> {
|
||||
self.lines.recv_timeout(Duration::from_secs(10)).ok()
|
||||
}
|
||||
|
||||
/// The next event `(name, data)`; comments are skipped.
|
||||
fn next(&self) -> Option<(String, String)> {
|
||||
let mut name = String::new();
|
||||
let mut data = String::new();
|
||||
loop {
|
||||
let line = self.line()?;
|
||||
if line.is_empty() {
|
||||
if !name.is_empty() {
|
||||
return Some((name, data));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if let Some(rest) = line.strip_prefix("event: ") {
|
||||
rest.clone_into(&mut name);
|
||||
} else if let Some(rest) = line.strip_prefix("data: ") {
|
||||
rest.clone_into(&mut data);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The next event with the given name.
|
||||
fn next_named(&self, wanted: &str) -> Option<(String, String)> {
|
||||
loop {
|
||||
let (name, data) = self.next()?;
|
||||
if name == wanted {
|
||||
return Some((name, data));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The operation an event carries.
|
||||
fn event_op(data: &str) -> TextOp {
|
||||
let value: Value = serde_json::from_str(data).expect("event data");
|
||||
serde_json::from_value(value["op"].clone()).expect("operation")
|
||||
}
|
||||
|
||||
/// The body the file endpoint serves.
|
||||
fn file_body(server: &Server, vault: &str, token: &str, path: &str) -> (u16, String) {
|
||||
let (status, body) = server.send("GET", &format!("/v1/vaults/{vault}/files/{path}"), Some(token), &[], None);
|
||||
(status, String::from_utf8_lossy(&body).into_owned())
|
||||
}
|
||||
|
||||
/// A path with the spaces and brackets a conflict copy carries, in URL form.
|
||||
fn url_path(path: &str) -> String {
|
||||
path.replace(' ', "%20").replace('(', "%28").replace(')', "%29")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn joining_answers_the_snapshot_and_the_revision() {
|
||||
let server = server("live-join", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-join");
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "hello", 0).0, 201);
|
||||
|
||||
let (status, reply) = join(&server, &vault, &token, "note.md", "client-a", "Ana");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["text"], json!("hello"), "the room starts with the file's text");
|
||||
assert_eq!(reply["rev"], json!(0), "a new room is at revision zero");
|
||||
let peers = reply["peers"].as_array().expect("peers");
|
||||
assert_eq!(peers.len(), 1);
|
||||
assert_eq!(peers[0]["client"], json!("client-a"));
|
||||
assert_eq!(peers[0]["name"], json!("Ana"));
|
||||
|
||||
// A second peer sees the same room.
|
||||
let (status, reply) = join(&server, &vault, &token, "note.md", "client-b", "Bo");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["rev"], json!(0));
|
||||
assert_eq!(reply["peers"].as_array().map(Vec::len), Some(2));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_clients_converge_on_the_room_text() {
|
||||
let server = server("live-converge", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-converge");
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "hi", 0).0, 201);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "b", "Bo").0, 200);
|
||||
|
||||
let stream_a = Stream::open(&server, &vault, &token, "note.md", "a");
|
||||
let stream_b = Stream::open(&server, &vault, &token, "note.md", "b");
|
||||
|
||||
// Both edit the same base revision.
|
||||
let mut client_a = Client::new(0);
|
||||
let mut client_b = Client::new(0);
|
||||
let edit_a = TextOp::new().retain(2).insert("A");
|
||||
let edit_b = TextOp::new().retain(2).insert("B");
|
||||
let submit_a = client_a.local_edit(edit_a.clone()).expect("edit").expect("submit");
|
||||
let submit_b = client_b.local_edit(edit_b.clone()).expect("edit").expect("submit");
|
||||
let mut text_a = edit_a.apply("hi").expect("apply");
|
||||
let mut text_b = edit_b.apply("hi").expect("apply");
|
||||
|
||||
// The first operation is accepted as it is.
|
||||
let (status, reply) = ops(&server, &vault, &token, "note.md", "a", submit_a.base_rev, &submit_a.op);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["rev"], json!(1));
|
||||
// The second one is transformed against it.
|
||||
let (status, reply) = ops(&server, &vault, &token, "note.md", "b", submit_b.base_rev, &submit_b.op);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["rev"], json!(2));
|
||||
|
||||
// A hears its own acknowledgement first, then the other operation.
|
||||
let (name, data) = stream_a.next_named("ack").expect("ack");
|
||||
assert_eq!(name, "ack");
|
||||
assert_eq!(data, json!({"rev": 1}).to_string());
|
||||
client_a.server_ack().expect("ack");
|
||||
let (_name, data) = stream_a.next_named("op").expect("op");
|
||||
let applied = client_a.remote_op(event_op(&data)).expect("remote");
|
||||
text_a = applied.apply(&text_a).expect("apply");
|
||||
|
||||
// B hears the other operation first, then its own acknowledgement.
|
||||
let (_name, data) = stream_b.next_named("op").expect("op");
|
||||
let applied = client_b.remote_op(event_op(&data)).expect("remote");
|
||||
text_b = applied.apply(&text_b).expect("apply");
|
||||
client_b.server_ack().expect("ack");
|
||||
|
||||
// Both hold the text the room holds.
|
||||
let (status, joined) = join(&server, &vault, &token, "note.md", "c", "Cy");
|
||||
assert_eq!(status, 200);
|
||||
let room = joined["text"].as_str().expect("text");
|
||||
assert_eq!(text_a, room, "the first client converged");
|
||||
assert_eq!(text_b, room, "the second client converged");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_viewer_may_join_but_not_edit() {
|
||||
let server = server("live-viewer", windows());
|
||||
let (vault, owner, _) = server.owned_vault("live-viewer");
|
||||
let (_, viewer) = server.paired("live-viewer");
|
||||
let (status, code) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(&viewer, &code).0, 200);
|
||||
|
||||
// The viewer joins the room and sees it.
|
||||
let (status, reply) = join(&server, &vault, &viewer, "note.md", "v", "Vic");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["text"], json!(""));
|
||||
|
||||
// Its operations are refused.
|
||||
let op = TextOp::new().insert("x");
|
||||
let (status, _reply) = ops(&server, &vault, &viewer, "note.md", "v", 0, &op);
|
||||
assert_eq!(status, 403);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_base_revision_older_than_the_history_is_refused() {
|
||||
let server = server("live-stale", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-stale");
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
|
||||
// The room keeps a bounded history; past it a client has to rejoin.
|
||||
let mut len = 0usize;
|
||||
for rev in 0..=HISTORY_CAP as u64 {
|
||||
let op = TextOp::new().retain(len).insert("x");
|
||||
let (status, _reply) = ops(&server, &vault, &token, "note.md", "a", rev, &op);
|
||||
assert_eq!(status, 200, "revision {rev}");
|
||||
len += 1;
|
||||
}
|
||||
|
||||
// The first revision is out of the kept history now.
|
||||
let op = TextOp::new().retain(len).insert("x");
|
||||
let (status, reply) = ops(&server, &vault, &token, "note.md", "a", 0, &op);
|
||||
assert_eq!(status, 409);
|
||||
assert_eq!(reply["error"], json!("stale"), "the client must rejoin");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_malformed_operation_is_a_bad_request() {
|
||||
let server = server("live-malformed", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-malformed");
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
|
||||
// Not an operation at all.
|
||||
let (status, _reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/live/note.md/ops"), Some(&token), &[], Some(&json!({"client": "a", "base_rev": 0, "op": "nonsense"})));
|
||||
assert_eq!(status, 400);
|
||||
|
||||
// Not JSON at all.
|
||||
let (status, _body) = server.send("POST", &format!("/v1/vaults/{vault}/live/note.md/ops"), Some(&token), &[], Some(b"{not json"));
|
||||
assert_eq!(status, 400);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_body_over_the_cap_is_refused() {
|
||||
let server = server("live-oversize", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-oversize");
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
|
||||
let big = "x".repeat(1024 * 1024 + 64);
|
||||
let body = json!({"client": "a", "base_rev": 0, "op": TextOp::new().insert(&big)}).to_string();
|
||||
let (status, _reply) = server.send("POST", &format!("/v1/vaults/{vault}/live/note.md/ops"), Some(&token), &[], Some(body.as_bytes()));
|
||||
assert_eq!(status, 413);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_file_that_is_too_large_or_not_text_cannot_be_joined() {
|
||||
let server = server("live-files", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-files");
|
||||
|
||||
// A file over the document cap.
|
||||
let big = "x".repeat(2 * 1024 * 1024 + 1);
|
||||
let (status, _body) = server.send("PUT", &format!("/v1/vaults/{vault}/files/big.md"), Some(&token), &[("X-Base-Seq", "0")], Some(big.as_bytes()));
|
||||
assert_eq!(status, 201);
|
||||
let (status, _reply) = join(&server, &vault, &token, "big.md", "a", "Ana");
|
||||
assert_eq!(status, 413);
|
||||
|
||||
// A file that is not UTF-8.
|
||||
let (status, _body) = server.send("PUT", &format!("/v1/vaults/{vault}/files/raw.md"), Some(&token), &[("X-Base-Seq", "0")], Some(&[0xff, 0xfe, 0x00]));
|
||||
assert_eq!(status, 201);
|
||||
let (status, _reply) = join(&server, &vault, &token, "raw.md", "a", "Ana");
|
||||
assert_eq!(status, 415);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn presence_lists_the_peers_and_prunes_the_silent_ones() {
|
||||
let server = server("live-presence", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-presence");
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "b", "Bo").0, 200);
|
||||
|
||||
// A heartbeat with a caret.
|
||||
let (status, reply) = server.json(
|
||||
"POST",
|
||||
&format!("/v1/vaults/{vault}/live/note.md/presence"),
|
||||
Some(&token),
|
||||
&[],
|
||||
Some(&json!({"client": "a", "selection": {"anchor": 1, "head": 1}})),
|
||||
);
|
||||
assert_eq!(status, 200);
|
||||
let peers = reply["peers"].as_array().expect("peers");
|
||||
assert_eq!(peers.len(), 2);
|
||||
let first = peers.iter().find(|peer| peer["client"] == json!("a")).expect("peer a");
|
||||
assert_eq!(first["selection"], json!({"anchor": 1, "head": 1}));
|
||||
|
||||
// Time passes: the silent peers are dropped.
|
||||
let pruned = server.state.prune_live_presence(&vault, "note.md", now_ms() + 60_000);
|
||||
assert_eq!(pruned.len(), 2, "both peers fell silent: {pruned:?}");
|
||||
|
||||
// A pruned peer is gone.
|
||||
let (status, _reply) = server.json("POST", &format!("/v1/vaults/{vault}/live/note.md/presence"), Some(&token), &[], Some(&json!({"client": "a"})));
|
||||
assert_eq!(status, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flushing_a_room_writes_its_text_to_the_vault() {
|
||||
let server = server("live-flush", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-flush");
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "hi", 0).0, 201);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
let edit = TextOp::new().retain(2).insert("!");
|
||||
let (status, reply) = ops(&server, &vault, &token, "note.md", "a", 0, &edit);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(reply["rev"], json!(1));
|
||||
|
||||
// The flush is a plain call: no waiting.
|
||||
let report = flush(&server.state, &vault, "note.md", now_ms()).expect("flush");
|
||||
assert!(report.flushed);
|
||||
assert!(!report.conflict);
|
||||
assert_eq!(report.path, "note.md");
|
||||
|
||||
// The file the normal endpoint serves is the room's text.
|
||||
let (status, body) = file_body(&server, &vault, &token, "note.md");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(body, "hi!");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flushing_after_an_external_write_makes_a_conflict_copy() {
|
||||
let server = server("live-conflict", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-conflict");
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "one", 0).0, 201);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
let edit = TextOp::new().retain(3).insert("!");
|
||||
assert_eq!(ops(&server, &vault, &token, "note.md", "a", 0, &edit).0, 200);
|
||||
|
||||
// Someone else writes the file while the room is open.
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "two", 1).0, 201);
|
||||
|
||||
let report = flush(&server.state, &vault, "note.md", now_ms()).expect("flush");
|
||||
assert!(report.flushed);
|
||||
assert!(report.conflict, "the ledger held a newer version");
|
||||
assert_ne!(report.path, "note.md");
|
||||
|
||||
// The file itself is untouched.
|
||||
let (status, body) = file_body(&server, &vault, &token, "note.md");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(body, "two", "the external write stands");
|
||||
|
||||
// The room's text went to the conflict copy.
|
||||
let (status, body) = file_body(&server, &vault, &token, &url_path(&report.path));
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(body, "one!");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_last_peer_leaving_writes_the_room_text() {
|
||||
let server = server("live-leave", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-leave");
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "hi", 0).0, 201);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
let edit = TextOp::new().retain(2).insert("!");
|
||||
assert_eq!(ops(&server, &vault, &token, "note.md", "a", 0, &edit).0, 200);
|
||||
|
||||
let (status, _reply) = server.json("POST", &format!("/v1/vaults/{vault}/live/note.md/leave"), Some(&token), &[], Some(&json!({"client": "a"})));
|
||||
assert_eq!(status, 204);
|
||||
|
||||
let (status, body) = file_body(&server, &vault, &token, "note.md");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(body, "hi!", "the room's text reached the vault");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_room_and_peer_caps_are_enforced() {
|
||||
let mut windows = windows();
|
||||
windows.max_rooms = 2;
|
||||
windows.max_peers = 2;
|
||||
let server = server("live-caps", windows);
|
||||
let (vault, token, _) = server.owned_vault("live-caps");
|
||||
|
||||
// Two rooms fit.
|
||||
assert_eq!(join(&server, &vault, &token, "one.md", "a", "Ana").0, 200);
|
||||
assert_eq!(join(&server, &vault, &token, "two.md", "b", "Bo").0, 200);
|
||||
// The third room has no room left.
|
||||
let (status, _reply) = join(&server, &vault, &token, "three.md", "c", "Cy");
|
||||
assert_eq!(status, 503);
|
||||
|
||||
// The peer cap holds per room.
|
||||
assert_eq!(join(&server, &vault, &token, "one.md", "d", "Di").0, 200);
|
||||
let (status, _reply) = join(&server, &vault, &token, "one.md", "e", "Ed");
|
||||
assert_eq!(status, 409);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_event_stream_keeps_alive() {
|
||||
let server = server("live-keepalive", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-keepalive");
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
|
||||
let stream = Stream::open(&server, &vault, &token, "note.md", "a");
|
||||
let line = stream.line().expect("a line");
|
||||
assert!(line.contains("keepalive"), "the stream comments itself alive: {line}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_audit_log_names_the_live_events_without_the_text() {
|
||||
let server = server("live-audit", windows());
|
||||
let (vault, token, _) = server.owned_vault("live-audit");
|
||||
assert_eq!(server.put(&vault, &token, "note.md", "secret text", 0).0, 201);
|
||||
assert_eq!(join(&server, &vault, &token, "note.md", "a", "Ana").0, 200);
|
||||
|
||||
let rows = server.audit();
|
||||
let joined = rows.iter().find(|row| row["kind"] == json!("live_joined")).expect("a join row");
|
||||
assert_eq!(joined["detail"], json!("note.md"));
|
||||
assert!(!joined.to_string().contains("secret text"), "the note text is not logged");
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
//! The engine's live session against the real sync server: two peers type in the
|
||||
//! same room, their texts meet and the room reaches the vault through the flush.
|
||||
//!
|
||||
//! The time is never waited for: the windows of the room are long, the flush is
|
||||
//! called by hand and the clients are polled until they are quiet.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use common::Server;
|
||||
use notescraft_engine::{LiveConfig, LiveSession};
|
||||
use notescraft_sync_server::{AppState, Store, Timings, flush, now_ms};
|
||||
use serde_json::json;
|
||||
|
||||
/// How many rounds both peers type in.
|
||||
const ROUNDS: usize = 5;
|
||||
/// How long a test waits for the clients to settle.
|
||||
const DEADLINE: Duration = Duration::from_secs(10);
|
||||
/// How many polls without an event mean that the clients are quiet.
|
||||
const QUIET_POLLS: usize = 40;
|
||||
/// The note both peers type in.
|
||||
const NOTE: &str = "notes/note.md";
|
||||
|
||||
/// The windows of the room: a short keepalive and long debounces, so the only
|
||||
/// write of the test is the flush at the end.
|
||||
fn windows() -> Timings {
|
||||
Timings { keepalive_ms: 200, idle_ms: 30_000, presence_ttl_ms: 60_000, flush_debounce_ms: 60_000, sweep_ms: 60_000, max_rooms: 8, max_peers: 64 }
|
||||
}
|
||||
|
||||
/// A server over a fresh folder with the windows of the test.
|
||||
fn server(name: &str) -> Server {
|
||||
let dir = common::temp_dir(name);
|
||||
let state = AppState::new(Store::open(&dir.join("sync.db")).expect("db"), dir.join("blobs"), common::ADMIN);
|
||||
state.set_live_timings(windows());
|
||||
common::start_with_state(state)
|
||||
}
|
||||
|
||||
/// The connection details of one peer of the room.
|
||||
fn config(server: &Server, vault: &str, token: &str, client: &str, name: &str) -> LiveConfig {
|
||||
LiveConfig { server: server.base.clone(), vault_id: vault.to_owned(), token: token.to_owned(), client_id: client.to_owned(), name: name.to_owned() }
|
||||
}
|
||||
|
||||
/// A generator of positions that repeats itself, so a failing run repeats too.
|
||||
struct Lcg(u64);
|
||||
|
||||
impl Lcg {
|
||||
/// A generator with the given seed.
|
||||
fn new(seed: u64) -> Self {
|
||||
Self(seed)
|
||||
}
|
||||
|
||||
/// The next number of the sequence.
|
||||
fn next(&mut self) -> u64 {
|
||||
self.0 = self.0.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1_442_695_040_888_963_407);
|
||||
self.0 >> 33
|
||||
}
|
||||
|
||||
/// A position inside a text of `len` characters.
|
||||
fn position(&mut self, len: usize) -> usize {
|
||||
let span = u64::try_from(len).unwrap_or(u64::MAX).max(1);
|
||||
usize::try_from(self.next() % span).unwrap_or(0)
|
||||
}
|
||||
}
|
||||
|
||||
/// The places a marker may go: the start of a line. A marker put there keeps
|
||||
/// its shape, whatever the other peers type later.
|
||||
fn line_starts(text: &str) -> Vec<usize> {
|
||||
let mut starts = vec![0];
|
||||
for (index, character) in text.chars().enumerate() {
|
||||
if character == '\n' {
|
||||
starts.push(index + 1);
|
||||
}
|
||||
}
|
||||
starts
|
||||
}
|
||||
|
||||
/// The text with `marker` put in at the given position.
|
||||
fn insert_at(text: &str, at: usize, marker: &str) -> String {
|
||||
let mut out = String::with_capacity(text.len() + marker.len());
|
||||
for (index, character) in text.chars().enumerate() {
|
||||
if index == at {
|
||||
out.push_str(marker);
|
||||
}
|
||||
out.push(character);
|
||||
}
|
||||
if at >= text.chars().count() {
|
||||
out.push_str(marker);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Polls both clients until they are quiet and hold the same text.
|
||||
fn settle(a: &mut LiveSession, b: &mut LiveSession) -> bool {
|
||||
let deadline = Instant::now() + DEADLINE;
|
||||
let mut quiet = 0;
|
||||
while Instant::now() < deadline {
|
||||
let events = a.poll().len() + b.poll().len();
|
||||
if events == 0 {
|
||||
quiet += 1;
|
||||
} else {
|
||||
quiet = 0;
|
||||
}
|
||||
if quiet >= QUIET_POLLS && a.text() == b.text() && a.rev() == b.rev() {
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Polls the client until it sees the peer with that id.
|
||||
fn sees_peer(session: &mut LiveSession, client: &str) -> bool {
|
||||
let deadline = Instant::now() + DEADLINE;
|
||||
while Instant::now() < deadline {
|
||||
session.poll();
|
||||
if session.peers().iter().any(|peer| peer.client == client) {
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// The text and the revision of the room, read by joining as a third peer.
|
||||
fn room_snapshot(server: &Server, vault: &str, token: &str, path: &str) -> (String, u64) {
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/live/{path}/join"), Some(token), &[], Some(&json!({"client": "observer", "name": "Observer"})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let text = reply["text"].as_str().expect("the text of the room").to_owned();
|
||||
let rev = reply["rev"].as_u64().expect("the revision of the room");
|
||||
(text, rev)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_peers_type_in_the_same_room() {
|
||||
let server = server("live-e2e");
|
||||
let (vault, token, _user) = server.owned_vault("live-e2e");
|
||||
let (status, _) = server.put(&vault, &token, NOTE, "one\ntwo\nthree\n", 0);
|
||||
assert_eq!(status, 201, "the note is in the vault");
|
||||
|
||||
let mut lcg = Lcg::new(0x5EED_1234);
|
||||
let (mut a, text_a) = LiveSession::join(config(&server, &vault, &token, "client-a", "Ana"), NOTE).expect("join a");
|
||||
assert_eq!(text_a, "one\ntwo\nthree\n", "the room hands its text to the first client");
|
||||
let (mut b, text_b) = LiveSession::join(config(&server, &vault, &token, "client-b", "Ben"), NOTE).expect("join b");
|
||||
assert_eq!(text_b, "one\ntwo\nthree\n", "the room hands its text to the second client");
|
||||
|
||||
for round in 0..ROUNDS {
|
||||
let marker_a = format!("A{round} ");
|
||||
let marker_b = format!("B{round} ");
|
||||
let starts_a = line_starts(a.text());
|
||||
let starts_b = line_starts(b.text());
|
||||
let at_a = starts_a.get(lcg.position(starts_a.len())).copied().unwrap_or(0);
|
||||
let at_b = starts_b.get(lcg.position(starts_b.len())).copied().unwrap_or(0);
|
||||
let edited_a = insert_at(a.text(), at_a, &marker_a);
|
||||
let edited_b = insert_at(b.text(), at_b, &marker_b);
|
||||
a.local_edit(&edited_a).expect("the edit of the first client");
|
||||
b.local_edit(&edited_b).expect("the edit of the second client");
|
||||
assert!(settle(&mut a, &mut b), "the clients did not settle in round {round}");
|
||||
}
|
||||
|
||||
assert!(sees_peer(&mut a, "client-b"), "the first client sees the second one");
|
||||
assert!(sees_peer(&mut b, "client-a"), "the second client sees the first one");
|
||||
|
||||
let (room_text, room_rev) = room_snapshot(&server, &vault, &token, NOTE);
|
||||
assert_eq!(a.text(), room_text, "the first client holds the text of the room");
|
||||
assert_eq!(b.text(), room_text, "the second client holds the text of the room");
|
||||
assert_eq!(a.rev(), room_rev, "the first client is at the revision of the room");
|
||||
assert_eq!(b.rev(), room_rev, "the second client is at the revision of the room");
|
||||
for round in 0..ROUNDS {
|
||||
let marker_a = format!("A{round} ");
|
||||
let marker_b = format!("B{round} ");
|
||||
assert!(room_text.contains(&marker_a), "the marker {marker_a} was lost");
|
||||
assert!(room_text.contains(&marker_b), "the marker {marker_b} was lost");
|
||||
}
|
||||
|
||||
a.leave();
|
||||
b.leave();
|
||||
|
||||
let report = flush(&server.state, &vault, NOTE, now_ms()).expect("the room is written");
|
||||
assert!(report.flushed, "the room was written to the vault");
|
||||
assert!(!report.conflict, "the room did not conflict with the vault");
|
||||
|
||||
let (status, body) = server.send("GET", &format!("/v1/vaults/{vault}/files/{NOTE}"), Some(&token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(String::from_utf8_lossy(&body), room_text, "the file holds the text of the room");
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
//! The `live.*` commands against the real sync server: two sessions type in the
|
||||
//! same room at the same time, their texts meet and the attribution names the
|
||||
//! editor of the last write.
|
||||
//!
|
||||
//! The time is never waited for: the windows of the room are long, the flush is
|
||||
//! called by hand and the sessions are polled until they are quiet.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use common::Server;
|
||||
use notescraft_engine::{Registry, Session, bind_folder, register_core};
|
||||
use notescraft_sync_server::{AppState, Store, Timings, flush, now_ms};
|
||||
use serde_json::json;
|
||||
|
||||
/// How many polls without an event mean that the sessions are quiet.
|
||||
const QUIET_POLLS: usize = 40;
|
||||
/// How long a test waits for the sessions to settle.
|
||||
const DEADLINE: Duration = Duration::from_secs(10);
|
||||
/// The note both sessions type in.
|
||||
const NOTE: &str = "notes/note.md";
|
||||
/// The text the note holds before the two sessions type.
|
||||
const START: &str = "one\ntwo\n";
|
||||
|
||||
/// The windows of the room: a short keepalive and long debounces, so the only
|
||||
/// write of the test is the flush at the end.
|
||||
fn windows() -> Timings {
|
||||
Timings { keepalive_ms: 200, idle_ms: 30_000, presence_ttl_ms: 60_000, flush_debounce_ms: 60_000, sweep_ms: 60_000, max_rooms: 8, max_peers: 64 }
|
||||
}
|
||||
|
||||
/// A server over a fresh folder with the windows of the test.
|
||||
fn server(name: &str) -> Server {
|
||||
let dir = common::temp_dir(name);
|
||||
let state = AppState::new(Store::open(&dir.join("sync.db")).expect("db"), dir.join("blobs"), common::ADMIN);
|
||||
state.set_live_timings(windows());
|
||||
common::start_with_state(state)
|
||||
}
|
||||
|
||||
/// The folder of one engine session, bound to the server.
|
||||
fn folder(name: &str, server: &Server, vault: &str, token: &str) -> PathBuf {
|
||||
let dir = std::env::temp_dir().join(format!("nc-live-cmd-{name}-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).expect("the folder of the session");
|
||||
bind_folder(&dir, &server.base, vault, token, "device-1", None).expect("bind the folder");
|
||||
dir
|
||||
}
|
||||
|
||||
/// The text of the room, read by joining as a third peer.
|
||||
fn room_text(server: &Server, vault: &str, token: &str) -> String {
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/live/{NOTE}/join"), Some(token), &[], Some(&json!({"client": "observer", "name": "Observer"})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
reply["text"].as_str().expect("the text of the room").to_owned()
|
||||
}
|
||||
|
||||
/// Polls both rooms until they are quiet and hold the same text.
|
||||
fn settle(reg: &Registry<Session>, a: &mut Session, b: &mut Session) -> (String, String) {
|
||||
let deadline = Instant::now() + DEADLINE;
|
||||
let mut quiet = 0;
|
||||
let mut texts = (String::new(), String::new());
|
||||
while Instant::now() < deadline {
|
||||
let (first, events_a) = poll(reg, a);
|
||||
let (second, events_b) = poll(reg, b);
|
||||
texts = (first.clone(), second.clone());
|
||||
if events_a + events_b == 0 {
|
||||
quiet += 1;
|
||||
} else {
|
||||
quiet = 0;
|
||||
}
|
||||
if quiet >= QUIET_POLLS && first == second {
|
||||
return texts;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
texts
|
||||
}
|
||||
|
||||
/// The text of the room of one session and how many events it heard.
|
||||
fn poll(reg: &Registry<Session>, session: &mut Session) -> (String, usize) {
|
||||
let answer = reg.run("live.poll", session, &json!({ "path": NOTE })).expect("poll");
|
||||
let text = answer["text"].as_str().expect("the text of the room").to_owned();
|
||||
let events = answer["events"].as_array().map_or(0, Vec::len);
|
||||
(text, events)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_sessions_type_in_the_same_room() {
|
||||
let server = server("live-cmd-e2e");
|
||||
let (vault, token_a, _user_a) = server.owned_vault("Ana");
|
||||
let user_b = server.create_user("Ben");
|
||||
let (status, token_b) = server.pair_user(&user_b, &vault, "Ben's phone");
|
||||
assert_eq!(status, 200, "the second user is paired with the vault");
|
||||
let (status, _) = server.put(&vault, &token_a, NOTE, START, 0);
|
||||
assert_eq!(status, 201, "the note is in the vault");
|
||||
|
||||
let mut reg = Registry::new();
|
||||
register_core(&mut reg).expect("register the core commands");
|
||||
let mut a = Session::open(&folder("a", &server, &vault, &token_a)).expect("open the first session");
|
||||
let mut b = Session::open(&folder("b", &server, &vault, &token_b)).expect("open the second session");
|
||||
|
||||
let answer = reg.run("live.join", &mut a, &json!({ "path": NOTE })).expect("join the first session");
|
||||
assert_eq!(answer["text"], json!(START), "the room hands its text to the first session");
|
||||
reg.run("live.join", &mut b, &json!({ "path": NOTE })).expect("join the second session");
|
||||
|
||||
// Both sessions type at the same time: neither has seen the other yet.
|
||||
reg.run("live.edit", &mut a, &json!({ "path": NOTE, "text": "one\ntwo\nA" })).expect("the edit of the first session");
|
||||
reg.run("live.edit", &mut b, &json!({ "path": NOTE, "text": "one\ntwo\nB" })).expect("the edit of the second session");
|
||||
|
||||
let (text_a, text_b) = settle(®, &mut a, &mut b);
|
||||
let room = room_text(&server, &vault, &token_a);
|
||||
assert_eq!(text_a, room, "the first session holds the text of the room");
|
||||
assert_eq!(text_b, room, "the second session holds the text of the room");
|
||||
assert!(room.contains('A'), "the marker of the first session was lost: {room}");
|
||||
assert!(room.contains('B'), "the marker of the second session was lost: {room}");
|
||||
|
||||
// The room is written and the attribution names the editor of the write.
|
||||
let report = flush(&server.state, &vault, NOTE, now_ms()).expect("the room is written");
|
||||
assert!(report.flushed, "the room was written to the vault");
|
||||
assert!(!report.conflict, "the room did not conflict with the vault");
|
||||
let answer = reg.run("server.file_meta", &mut a, &json!({ "path": NOTE })).expect("the attribution");
|
||||
let name = answer["editor_name"].as_str().expect("the name of the last editor");
|
||||
assert!(name == "Ana" || name == "Ben", "the last editor is one of the two: {name}");
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
//! Publish pages and share links over real HTTP: what is served, what stays hidden,
|
||||
//! who may publish, and how the anonymous routes are bounded.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::Server;
|
||||
use notescraft_sync_server::publish;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
/// The header value of a response, or an empty string.
|
||||
fn header(headers: &[(String, String)], name: &str) -> String {
|
||||
headers.iter().find(|(key, _)| key == name).map(|(_, value)| value.clone()).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// The body of a response as text.
|
||||
fn text(body: &[u8]) -> String {
|
||||
String::from_utf8_lossy(body).into_owned()
|
||||
}
|
||||
|
||||
/// Writes one note and asserts it landed.
|
||||
fn note(server: &Server, vault: &str, token: &str, path: &str, body: &str) {
|
||||
let (status, _) = server.put(vault, token, path, body, 0);
|
||||
assert_eq!(status, 201, "put {path}");
|
||||
}
|
||||
|
||||
/// Publishes one note and returns the slug and the public path.
|
||||
fn publish_note(server: &Server, vault: &str, token: &str, target: &str) -> (String, String) {
|
||||
let (status, reply) = server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(token), &[], Some(&json!({"target": target, "kind": "note"})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let slug = reply["slug"].as_str().expect("slug").to_owned();
|
||||
let url = reply["url_path"].as_str().expect("url").to_owned();
|
||||
(slug, url)
|
||||
}
|
||||
|
||||
/// The member ids of a vault, by role.
|
||||
fn members_by_role(server: &Server, vault: &str, token: &str) -> Vec<(String, String)> {
|
||||
let (_, members) = server.json("GET", &format!("/v1/vaults/{vault}/members"), Some(token), &[], None);
|
||||
members
|
||||
.as_array()
|
||||
.expect("members")
|
||||
.iter()
|
||||
.map(|row| (row["role"].as_str().expect("role").to_owned(), row["user_id"].as_str().expect("user id").to_owned()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_published_note_is_a_safe_static_page() {
|
||||
let server = Server::start_temp("publish-note");
|
||||
let (vault, token, _) = server.owned_vault("note");
|
||||
note(&server, &vault, &token, "hello.md", "# Hello\n\n<script>alert(1)</script>\n\n<img src=x onerror=alert(1)>\n\n[x](javascript:alert(1))\n");
|
||||
let (slug, url) = publish_note(&server, &vault, &token, "hello.md");
|
||||
assert_eq!(url, format!("/p/{slug}/hello"));
|
||||
|
||||
let (status, headers, body) = server.send_full("GET", &url, None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let page = text(&body);
|
||||
assert!(page.contains("Hello"), "{page}");
|
||||
assert!(!page.contains("<script"), "{page}");
|
||||
assert!(!page.contains("<img src=x"), "{page}");
|
||||
assert!(page.contains("<script>"), "{page}");
|
||||
assert!(page.contains("<img src=x onerror=alert(1)>"), "{page}");
|
||||
assert!(page.contains("href=\"#\""), "{page}");
|
||||
assert_eq!(header(&headers, "content-type"), "text/html; charset=utf-8");
|
||||
assert_eq!(header(&headers, "content-security-policy"), "default-src 'none'; style-src 'unsafe-inline'; img-src https: http:");
|
||||
assert_eq!(header(&headers, "x-content-type-options"), "nosniff");
|
||||
assert_eq!(header(&headers, "referrer-policy"), "no-referrer");
|
||||
assert_eq!(header(&headers, "cache-control"), "public, max-age=60");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_published_folder_serves_only_its_markdown_notes() {
|
||||
let server = Server::start_temp("publish-folder");
|
||||
let (vault, token, _) = server.owned_vault("folder");
|
||||
note(&server, &vault, &token, "docs/a.md", "# Note A\n");
|
||||
note(&server, &vault, &token, "docs/nested/b.md", "# Note B\n");
|
||||
note(&server, &vault, &token, "docs/hidden.md", "---\npublish: false\n---\n# Hidden note\n");
|
||||
note(&server, &vault, &token, "docs/picture.png", "not markdown");
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], Some(&json!({"target": "docs", "kind": "folder"})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let slug = reply["slug"].as_str().expect("slug").to_owned();
|
||||
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/docs/a"), None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert!(text(&body).contains("Note A"));
|
||||
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/docs/nested/b"), None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert!(text(&body).contains("Note B"));
|
||||
|
||||
for hidden in ["docs/hidden", "docs/picture.png", ".obsidian/x.md", "docs/hidden.md"] {
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/{hidden}"), None, &[], None);
|
||||
assert_eq!(status, 404, "{hidden} must not be served");
|
||||
assert!(body.is_empty(), "{hidden} must answer with an empty body");
|
||||
}
|
||||
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/docs"), None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let index = text(&body);
|
||||
assert!(index.contains("Note A"), "{index}");
|
||||
assert!(index.contains("Note B"), "{index}");
|
||||
assert!(!index.contains("Hidden note"), "{index}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_comment_sidecar_is_never_published() {
|
||||
let server = Server::start_temp("publish-sidecar");
|
||||
let (vault, token, _) = server.owned_vault("sidecar");
|
||||
note(&server, &vault, &token, "docs/a.md", "# Note A\n");
|
||||
note(&server, &vault, &token, "docs/a.md.comments.json", "{\"version\":1,\"note\":\"docs/a.md\",\"comments\":[]}\n");
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], Some(&json!({"target": "docs", "kind": "folder"})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let slug = reply["slug"].as_str().expect("slug").to_owned();
|
||||
|
||||
// The index of the folder lists the note, never the sidecar.
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/docs"), None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let index = text(&body);
|
||||
assert!(index.contains("Note A"), "{index}");
|
||||
assert!(!index.contains("comments.json"), "{index}");
|
||||
|
||||
// And the sidecar itself answers 404 with an empty body.
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/docs/a.md.comments.json"), None, &[], None);
|
||||
assert_eq!(status, 404);
|
||||
assert!(body.is_empty(), "the sidecar must answer with an empty body");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unpublishing_answers_404_at_once() {
|
||||
let server = Server::start_temp("publish-unpublish");
|
||||
let (vault, token, _) = server.owned_vault("unpublish");
|
||||
note(&server, &vault, &token, "gone.md", "# Gone\n");
|
||||
let (slug, url) = publish_note(&server, &vault, &token, "gone.md");
|
||||
assert_eq!(server.send_full("GET", &url, None, &[], None).0, 200);
|
||||
|
||||
let (status, reply) = server.json("GET", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let id = reply[0]["id"].as_str().expect("id").to_owned();
|
||||
let (status, _) = server.send("DELETE", &format!("/v1/vaults/{vault}/publications/{id}"), Some(&token), &[], None);
|
||||
assert_eq!(status, 204);
|
||||
let (status, _, body) = server.send_full("GET", &url, None, &[], None);
|
||||
assert_eq!(status, 404);
|
||||
assert_eq!(body, b"".as_slice());
|
||||
let (status, _) = server.send("GET", &format!("/p/{slug}/"), None, &[], None);
|
||||
assert_eq!(status, 404);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_the_owner_publishes_and_only_an_editor_shares() {
|
||||
let server = Server::start_temp("publish-roles");
|
||||
let (vault, owner, _) = server.owned_vault("roles");
|
||||
let (_, editor) = server.paired("roles");
|
||||
let (_, viewer) = server.paired("roles");
|
||||
let (status, code) = server.invite(&vault, &owner, "editor");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(&editor, &code).0, 200);
|
||||
let (status, code) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(server.accept(&viewer, &code).0, 200);
|
||||
note(&server, &vault, &owner, "shared.md", "# Shared\n");
|
||||
|
||||
for role in ["editor", "viewer"] {
|
||||
let token = if role == "editor" { editor.as_str() } else { viewer.as_str() };
|
||||
let (status, _) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(token), &[], Some(&json!({"target": "shared.md", "kind": "note"})));
|
||||
assert_eq!(status, 403, "a {role} may not publish");
|
||||
}
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{vault}/share-links"), Some(&editor), &[], Some(&json!({"path": "shared.md"})));
|
||||
assert_eq!(status, 200, "an editor may share");
|
||||
let (status, _) = server.json("POST", &format!("/v1/vaults/{vault}/share-links"), Some(&viewer), &[], Some(&json!({"path": "shared.md"})));
|
||||
assert_eq!(status, 403, "a viewer may not share");
|
||||
assert!(members_by_role(&server, &vault, &owner).iter().any(|(role, _)| role == "owner"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_share_link_serves_one_note_until_it_is_revoked_or_expires() {
|
||||
let server = Server::start_temp("publish-share");
|
||||
let (vault, token, _) = server.owned_vault("share");
|
||||
note(&server, &vault, &token, "shared.md", "# Shared note\n");
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/share-links"), Some(&token), &[], Some(&json!({"path": "shared.md", "expires_in_s": 3600})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let url = reply["url_path"].as_str().expect("url").to_owned();
|
||||
let id = reply["id"].as_str().expect("id").to_owned();
|
||||
let token_part = url.strip_prefix("/s/").expect("token path").to_owned();
|
||||
assert_eq!(token_part.len(), 32);
|
||||
assert!(token_part.bytes().all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)));
|
||||
|
||||
let (status, headers, body) = server.send_full("GET", &url, None, &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert!(text(&body).contains("Shared note"));
|
||||
assert_eq!(header(&headers, "x-robots-tag"), "noindex");
|
||||
assert_eq!(header(&headers, "cache-control"), "no-store");
|
||||
|
||||
// The token never shows up in the listing nor in the audit.
|
||||
let (status, rows) = server.json("GET", &format!("/v1/vaults/{vault}/share-links"), Some(&token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let listing = rows.to_string();
|
||||
assert!(!listing.contains(&token_part), "{listing}");
|
||||
assert!(listing.contains("shared.md"), "{listing}");
|
||||
let audit = serde_json::to_string(&server.audit()).expect("audit json");
|
||||
assert!(!audit.contains(&token_part), "{audit}");
|
||||
assert!(audit.contains("share_link_created"), "{audit}");
|
||||
|
||||
// The clock is injected: the link is live now and expired later.
|
||||
let live = server.state.with_store(|store| publish::lookup_share_link(store, &token_part, 0));
|
||||
assert!(live.expect("lookup").is_some());
|
||||
let later = u64::try_from(i64::MAX).expect("max clock");
|
||||
let expired = server.state.with_store(|store| publish::lookup_share_link(store, &token_part, later));
|
||||
assert!(expired.expect("lookup").is_none());
|
||||
|
||||
// Revoking it answers 404 at once.
|
||||
let (status, _) = server.send("DELETE", &format!("/v1/vaults/{vault}/share-links/{id}"), Some(&token), &[], None);
|
||||
assert_eq!(status, 204);
|
||||
let (status, _, body) = server.send_full("GET", &url, None, &[], None);
|
||||
assert_eq!(status, 404);
|
||||
assert_eq!(body, b"".as_slice());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_slug_is_unique_across_the_server() {
|
||||
let server = Server::start_temp("publish-slug");
|
||||
let (first, first_token, _) = server.owned_vault("slug");
|
||||
let (second, second_token, _) = server.owned_vault("slug");
|
||||
let (status, _) = server.json("PATCH", &format!("/v1/vaults/{first}"), Some(&first_token), &[], Some(&json!({"slug": "my-notes"})));
|
||||
assert_eq!(status, 204);
|
||||
let (status, reply) = server.json("PATCH", &format!("/v1/vaults/{second}"), Some(&second_token), &[], Some(&json!({"slug": "my-notes"})));
|
||||
assert_eq!(status, 409, "{reply}");
|
||||
let (status, _) = server.json("PATCH", &format!("/v1/vaults/{second}"), Some(&second_token), &[], Some(&json!({"slug": "other-notes"})));
|
||||
assert_eq!(status, 204);
|
||||
let (status, _) = server.json("PATCH", &format!("/v1/vaults/{second}"), Some(&second_token), &[], Some(&json!({"slug": "not a slug"})));
|
||||
assert_eq!(status, 400);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_publication_follows_the_slug_of_its_vault() {
|
||||
let server = Server::start_temp("publish-slug-page");
|
||||
let (vault, token, _) = server.owned_vault("slugpage");
|
||||
note(&server, &vault, &token, "a.md", "# A\n");
|
||||
let (status, _) = server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(&token), &[], Some(&json!({"slug": "my-vault"})));
|
||||
assert_eq!(status, 204);
|
||||
let (slug, url) = publish_note(&server, &vault, &token, "a.md");
|
||||
assert_eq!(slug, "my-vault");
|
||||
assert_eq!(url, "/p/my-vault/a");
|
||||
assert_eq!(server.send_full("GET", &url, None, &[], None).0, 200);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_paths_never_escape_the_vault() {
|
||||
let server = Server::start_temp("publish-hostile");
|
||||
let (vault, token, _) = server.owned_vault("hostile");
|
||||
note(&server, &vault, &token, "a.md", "# A\n");
|
||||
let (slug, _) = publish_note(&server, &vault, &token, "a.md");
|
||||
let long = "a".repeat(5000);
|
||||
for path in ["%2e%2e/secret", "docs/%2e%2e/%2e%2e/etc/passwd", "a%00b", long.as_str()] {
|
||||
let (status, _, body) = server.send_full("GET", &format!("/p/{slug}/{path}"), None, &[], None);
|
||||
assert!(status == 404 || status == 400, "{path} answered {status}");
|
||||
assert!(!text(&body).contains("root:"), "{path} leaked a file");
|
||||
}
|
||||
let (status, _) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], Some(&json!({"target": "../escape.md", "kind": "note"})));
|
||||
assert_eq!(status, 400);
|
||||
let (status, _) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], Some(&json!({"target": ".obsidian/x.md", "kind": "note"})));
|
||||
assert_eq!(status, 400);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_anonymous_routes_are_rate_limited() {
|
||||
let server = Server::start_temp("publish-limit");
|
||||
let (vault, token, _) = server.owned_vault("limit");
|
||||
note(&server, &vault, &token, "a.md", "# A\n");
|
||||
let (_slug, url) = publish_note(&server, &vault, &token, "a.md");
|
||||
|
||||
// One slug may be read sixty times a minute; the next answer is 429.
|
||||
let mut answer = 200;
|
||||
for _ in 0..80 {
|
||||
let (status, headers, _) = server.send_full("GET", &url, None, &[], None);
|
||||
if status == 429 {
|
||||
assert!(!header(&headers, "retry-after").is_empty(), "a 429 carries Retry-After");
|
||||
answer = status;
|
||||
break;
|
||||
}
|
||||
answer = status;
|
||||
}
|
||||
assert_eq!(answer, 429);
|
||||
|
||||
// A scan of unknown slugs is bounded by the global bucket.
|
||||
let mut scan = 200;
|
||||
for round in 0..700 {
|
||||
let (status, headers, _) = server.send_full("GET", &format!("/p/unknown-{round}"), None, &[], None);
|
||||
if status == 429 {
|
||||
assert_ne!(header(&headers, "retry-after"), "");
|
||||
scan = status;
|
||||
break;
|
||||
}
|
||||
scan = status;
|
||||
}
|
||||
assert_eq!(scan, 429);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_publication_listing_names_every_publication() {
|
||||
let server = Server::start_temp("publish-list");
|
||||
let (vault, token, _) = server.owned_vault("list");
|
||||
note(&server, &vault, &token, "a.md", "# A\n");
|
||||
let (slug, _) = publish_note(&server, &vault, &token, "a.md");
|
||||
let (status, reply) = server.json("GET", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let rows: &Vec<Value> = reply.as_array().expect("rows");
|
||||
assert_eq!(rows.len(), 1);
|
||||
assert_eq!(rows[0]["kind"], "note");
|
||||
assert_eq!(rows[0]["target"], "a.md");
|
||||
assert_eq!(rows[0]["url_path"], format!("/p/{slug}/a"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_root_of_a_publication_lists_its_notes() {
|
||||
let server = Server::start_temp("publish-root");
|
||||
let (vault, token, _) = server.owned_vault("root");
|
||||
note(&server, &vault, &token, "a.md", "# Root A\n");
|
||||
let (slug, _) = publish_note(&server, &vault, &token, "a.md");
|
||||
for path in [format!("/p/{slug}"), format!("/p/{slug}/")] {
|
||||
let (status, _, body) = server.send_full("GET", &path, None, &[], None);
|
||||
assert_eq!(status, 200, "{path}");
|
||||
assert!(text(&body).contains("Root A"), "{path}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_note_over_the_size_cap_is_not_published() {
|
||||
let server = Server::start_temp("publish-large");
|
||||
let (vault, token, _) = server.owned_vault("large");
|
||||
let big = format!("# Big\n\n{}", "x".repeat(1024 * 1024 + 16));
|
||||
note(&server, &vault, &token, "big.md", &big);
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/publications"), Some(&token), &[], Some(&json!({"target": "big.md", "kind": "note"})));
|
||||
assert_eq!(status, 413, "{reply}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_share_link_lifetime_stays_inside_its_bounds() {
|
||||
let server = Server::start_temp("publish-lifetime");
|
||||
let (vault, token, _) = server.owned_vault("lifetime");
|
||||
note(&server, &vault, &token, "a.md", "# A\n");
|
||||
for seconds in [0, 59, 2_592_001] {
|
||||
let (status, _) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/share-links"), Some(&token), &[], Some(&json!({"path": "a.md", "expires_in_s": seconds})));
|
||||
assert_eq!(status, 400, "{seconds} seconds must be refused");
|
||||
}
|
||||
let (status, reply) =
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/share-links"), Some(&token), &[], Some(&json!({"path": "a.md", "expires_in_s": 60})));
|
||||
assert_eq!(status, 200, "{reply}");
|
||||
let expires = reply["expires_ms"].as_u64().expect("expires");
|
||||
assert!(expires > 0, "{reply}");
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
//! Quota and garbage collection, over real HTTP with injectable clocks and limits.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use common::{ADMIN, Server, temp_dir};
|
||||
use notescraft_sync_server::{GcReport, Store, TOMBSTONE_GRACE_MS};
|
||||
|
||||
/// One megabyte of quota, small enough that a test can spend it by hand.
|
||||
const QUOTA_BYTES: u64 = 1024 * 1024;
|
||||
|
||||
fn server_with_quota(name: &str, quota_bytes: Option<u64>) -> (Server, PathBuf) {
|
||||
let data = temp_dir(name);
|
||||
let state = notescraft_sync_server::AppState::with_quota(Store::open(&data.join("sync.db")).expect("db"), data.join("blobs"), ADMIN, quota_bytes);
|
||||
(common::start_with_state(state), data)
|
||||
}
|
||||
|
||||
fn blob_files(data: &std::path::Path, vault: &str) -> Vec<PathBuf> {
|
||||
let mut files: Vec<PathBuf> = std::fs::read_dir(data.join("blobs").join(vault))
|
||||
.expect("blob dir")
|
||||
.filter_map(|entry| entry.expect("entry").file_name().into_string().ok())
|
||||
.map(|name| data.join("blobs").join(vault).join(name))
|
||||
.collect();
|
||||
files.sort();
|
||||
files
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quota_rejects_uploads_over_the_live_bytes_limit_before_storing() {
|
||||
let (server, data) = server_with_quota("quota", Some(QUOTA_BYTES));
|
||||
let (vault, token) = server.paired("pc");
|
||||
let near = "x".repeat(usize::try_from(QUOTA_BYTES - 50).expect("small quota"));
|
||||
let (status, record) = server.put(&vault, &token, "a.md", &near, 0);
|
||||
assert_eq!((status, record["seq"].as_u64()), (201, Some(1)));
|
||||
|
||||
// A sibling file would exceed the quota: 413, audited, and no blob stored.
|
||||
let before = blob_files(&data, &vault);
|
||||
let (status, reply) = server.put(&vault, &token, "b.md", &"y".repeat(60), 0);
|
||||
assert_eq!(status, 413);
|
||||
assert_eq!(reply["error"], "vault storage quota exceeded");
|
||||
assert_eq!(blob_files(&data, &vault), before);
|
||||
assert!(server.audit().iter().any(|row| row["kind"] == "quota_rejected"));
|
||||
server.assert_no_secrets(&serde_json::to_string(&server.audit()).expect("rows"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quota_replacement_counts_only_the_size_difference() {
|
||||
let (server, _) = server_with_quota("quota-replace", Some(QUOTA_BYTES));
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (status, record) = server.put(&vault, &token, "a.md", &"x".repeat(usize::try_from(QUOTA_BYTES - 50).expect("small quota")), 0);
|
||||
assert_eq!(status, 201);
|
||||
let seq = record["seq"].as_u64().expect("seq");
|
||||
|
||||
// As a new file, +60 bytes overflows; as a replacement, +10 bytes fits the slack.
|
||||
assert_eq!(server.put(&vault, &token, "b.md", &"y".repeat(60), 0).0, 413);
|
||||
let (status, replaced) = server.put(&vault, &token, "a.md", &"z".repeat(usize::try_from(QUOTA_BYTES - 40).expect("small quota")), seq);
|
||||
assert_eq!(replaced["seq"].as_u64(), Some(seq + 1));
|
||||
assert_eq!(status, 201);
|
||||
// The next new file is still over.
|
||||
assert_eq!(server.put(&vault, &token, "b.md", &"y".repeat(59), 0).0, 413);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gc_removes_unreferenced_blobs_and_keeps_referenced_and_in_grace_ones() {
|
||||
let (server, data) = server_with_quota("gc", None);
|
||||
let (vault, token) = server.paired("pc");
|
||||
let (other, other_token) = server.paired("other");
|
||||
let (_, record_a) = server.put(&vault, &token, "a.md", "gone", 0);
|
||||
let (_, record_b) = server.put(&vault, &token, "b.md", "stays", 0);
|
||||
let (_, record_c) = server.put(&other, &other_token, "c.md", "untouched", 0);
|
||||
let (hash_a, hash_b, hash_c): (&str, &str, &str) =
|
||||
(record_a["hash"].as_str().expect("hash"), record_b["hash"].as_str().expect("hash"), record_c["hash"].as_str().expect("hash"));
|
||||
|
||||
// Tombstone a.md; its blob is unreferenced by live records but inside the grace.
|
||||
let (status, _) = server.json("DELETE", &format!("/v1/vaults/{vault}/files/a.md"), Some(&token), &[("X-Base-Seq", "1")], None);
|
||||
assert_eq!(status, 201);
|
||||
let dir = data.join("blobs").join(&vault);
|
||||
assert!(dir.join(hash_a).exists() && dir.join(hash_b).exists());
|
||||
|
||||
// Default 14-day grace keeps the fresh tombstone's blob.
|
||||
let report = server.state.gc(&vault, TOMBSTONE_GRACE_MS, notescraft_sync_server::now_ms()).expect("gc");
|
||||
assert_eq!(report, GcReport { removed: 0, bytes: 0 });
|
||||
|
||||
// Aged past the grace (injected clock), the tombstone's blob goes, the live one stays.
|
||||
let aged = notescraft_sync_server::now_ms() + TOMBSTONE_GRACE_MS + 1;
|
||||
let report = server.state.gc(&vault, TOMBSTONE_GRACE_MS, aged).expect("gc");
|
||||
assert_eq!(report, GcReport { removed: 1, bytes: u64::try_from("gone".len()).expect("small") });
|
||||
assert!(!dir.join(hash_a).exists());
|
||||
assert!(dir.join(hash_b).exists());
|
||||
// A zero grace takes even a live tombstone only through its own vault's files.
|
||||
assert_eq!(server.state.gc(&vault, 0, aged).expect("gc").removed, 0);
|
||||
assert!(data.join("blobs").join(&other).join(hash_c).exists());
|
||||
|
||||
// The run itself is audited.
|
||||
assert!(server.audit().iter().any(|row| row["kind"] == "gc_run"));
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
//! Role enforcement over real HTTP: a viewer/editor/owner matrix on every vault
|
||||
//! endpoint, membership management rules, and the invite lifecycle. Non-members are
|
||||
//! always 404, viewers above their level are 403.
|
||||
|
||||
mod common;
|
||||
|
||||
use common::Server;
|
||||
use serde_json::json;
|
||||
|
||||
/// A vault with an owner, an invited editor, an invited viewer and a stranger that
|
||||
/// only belongs to a different vault.
|
||||
struct Fixture {
|
||||
vault: String,
|
||||
owner: String,
|
||||
editor: String,
|
||||
viewer: String,
|
||||
stranger: String,
|
||||
editor_user: String,
|
||||
viewer_user: String,
|
||||
}
|
||||
|
||||
fn fixture(server: &Server, name: &str) -> Fixture {
|
||||
let (vault, owner, _) = server.owned_vault(name);
|
||||
// Tokens that are not members yet come from accounts of their own bootstrap vaults:
|
||||
// each pairing without `user` makes a new account; only the invite adds membership.
|
||||
let (_, editor) = server.paired(name);
|
||||
let (_, viewer) = server.paired(name);
|
||||
let (_, stranger) = server.paired(name);
|
||||
|
||||
let (status, code) = server.invite(&vault, &owner, "editor");
|
||||
assert_eq!(status, 200);
|
||||
let (status, _) = server.accept(&editor, &code);
|
||||
assert_eq!(status, 200);
|
||||
let (status, code) = server.invite(&vault, &owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
let (status, _) = server.accept(&viewer, &code);
|
||||
assert_eq!(status, 200);
|
||||
|
||||
// The owner writes one file every role can read.
|
||||
assert_eq!(server.put(&vault, &owner, "a.md", "shared", 0).0, 201);
|
||||
|
||||
// Member ids for the management routes come from the member listing.
|
||||
let (_, members) = server.json("GET", &format!("/v1/vaults/{vault}/members"), Some(&owner), &[], None);
|
||||
let member_of = |role: &str| {
|
||||
members.as_array().expect("members").iter().find(|row| row["role"] == role).unwrap_or_else(|| panic!("no {role} member"))["user_id"]
|
||||
.as_str()
|
||||
.expect("user id")
|
||||
.to_owned()
|
||||
};
|
||||
Fixture { vault, owner, editor, viewer, stranger, editor_user: member_of("editor"), viewer_user: member_of("viewer") }
|
||||
}
|
||||
|
||||
/// Expected status of a read for `role`: members 200, non-members 404.
|
||||
fn read_expect(role: &str) -> u16 {
|
||||
if role == "stranger" { 404 } else { 200 }
|
||||
}
|
||||
|
||||
/// Expected status of a file write for `role`.
|
||||
fn write_expect(role: &str) -> u16 {
|
||||
match role {
|
||||
"stranger" => 404,
|
||||
"editor" | "owner" => 201,
|
||||
_ => 403, // viewers
|
||||
}
|
||||
}
|
||||
|
||||
/// Expected status of an admin operation for `role` when the owner has it working.
|
||||
fn admin_expect(role: &str, owner_status: u16) -> u16 {
|
||||
match role {
|
||||
"stranger" => 404,
|
||||
"owner" => owner_status,
|
||||
_ => 403,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn roles_match_the_matrix_on_every_vault_endpoint() {
|
||||
let server = Server::start_temp("roles-matrix");
|
||||
let fix = fixture(&server, "matrix");
|
||||
let vault = fix.vault.as_str();
|
||||
let tokens: [(&str, &String); 4] = [("owner", &fix.owner), ("editor", &fix.editor), ("viewer", &fix.viewer), ("stranger", &fix.stranger)];
|
||||
// (No pre-created files: each role deletes the file it wrote, on top of its
|
||||
// own sequence, so the delete case exercises application, not the missing-file 404.)
|
||||
for (role, token) in tokens {
|
||||
// Reads: every member (viewer included) 200, a stranger 404.
|
||||
assert_eq!(server.send("GET", &format!("/v1/vaults/{vault}/changes?since=0"), Some(token), &[], None).0, read_expect(role), "changes {role}");
|
||||
// Writes, on role-named paths so the roles do not interfere; whoever wrote
|
||||
// (201) deletes the same file on top of the sequence the write got.
|
||||
let (status, record) = server.put(vault, token, &format!("write-{role}.md"), "x", 0);
|
||||
assert_eq!(status, write_expect(role), "file put {role}");
|
||||
if let Some(seq) = record["seq"].as_u64() {
|
||||
let (status, _) =
|
||||
server.send("DELETE", &format!("/v1/vaults/{vault}/files/write-{role}.md"), Some(token), &[("X-Base-Seq", &seq.to_string())], None);
|
||||
assert_eq!(status, 201, "file delete {role}");
|
||||
}
|
||||
// Admin: rename, invite listing, minting.
|
||||
assert_eq!(
|
||||
server.json("PATCH", &format!("/v1/vaults/{vault}"), Some(token), &[], Some(&json!({"name": "matrix"}))).0,
|
||||
admin_expect(role, 204),
|
||||
"rename {role}"
|
||||
);
|
||||
assert_eq!(server.json("GET", &format!("/v1/vaults/{vault}/invites"), Some(token), &[], None).0, admin_expect(role, 200), "invite list {role}");
|
||||
assert_eq!(
|
||||
server.json("POST", &format!("/v1/vaults/{vault}/invites"), Some(token), &[], Some(&json!({"role": "viewer"}))).0,
|
||||
admin_expect(role, 200),
|
||||
"invite mint {role}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn members_are_listed_changed_removed_and_left() {
|
||||
let server = Server::start_temp("roles-members");
|
||||
let fix = fixture(&server, "members");
|
||||
let base = format!("/v1/vaults/{}", fix.vault);
|
||||
|
||||
// The listing names every member with the fields the task lists.
|
||||
let (_, members) = server.json("GET", &format!("{base}/members"), Some(&fix.viewer), &[], None);
|
||||
let rows = members.as_array().expect("members");
|
||||
assert_eq!(rows.len(), 3);
|
||||
for row in rows {
|
||||
assert!(row.get("user_id").is_some() && row.get("name").is_some() && row.get("color").is_some() && row.get("role").is_some());
|
||||
}
|
||||
assert_eq!(rows.iter().filter(|row| row["role"] == "owner").count(), 1);
|
||||
|
||||
// Role changes are owner-only; the last owner can never be demoted.
|
||||
let owner_user = rows.iter().find(|row| row["role"] == "owner").expect("owner")["user_id"].as_str().expect("id").to_owned();
|
||||
let change =
|
||||
|token: &str, user: &str, role: &str| server.json("PATCH", &format!("{base}/members/{user}"), Some(token), &[], Some(&json!({"role": role}))).0;
|
||||
assert_eq!(change(&fix.editor, &fix.viewer_user, "owner"), 403, "only the owner changes roles");
|
||||
assert_eq!(change(&fix.owner, &owner_user, "viewer"), 409, "the last owner cannot be demoted");
|
||||
assert_eq!(change(&fix.owner, &fix.editor_user, "owner"), 204);
|
||||
assert_eq!(change(&fix.owner, &fix.editor_user, "editor"), 204, "a role change back works with two owners");
|
||||
assert_eq!(change(&fix.owner, "u-unknown", "viewer"), 404, "an unknown member");
|
||||
|
||||
// Removal: owner removes anyone; a member leaves alone; the last owner stays.
|
||||
let drop = |token: &str, user: &str| server.send("DELETE", &format!("{base}/members/{user}"), Some(token), &[], None).0;
|
||||
assert_eq!(drop(&fix.viewer, &fix.editor_user), 403, "a non-owner removes nobody but themselves");
|
||||
assert_eq!(drop(&fix.owner, &fix.editor_user), 204, "the owner removes anyone");
|
||||
assert_eq!(server.json("GET", &format!("{base}/changes?since=0"), Some(&fix.editor), &[], None).0, 404, "removed at once");
|
||||
assert_eq!(drop(&fix.owner, "u-unknown"), 404);
|
||||
assert_eq!(drop(&fix.owner, &owner_user), 409, "the last owner cannot leave");
|
||||
assert_eq!(drop(&fix.viewer, &fix.viewer_user), 204, "a member leaves themselves");
|
||||
assert_eq!(drop(&fix.owner, &fix.viewer_user), 404, "already left");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invites_are_single_use_role_keeping_and_owner_managed() {
|
||||
let server = Server::start_temp("roles-invites");
|
||||
let fix = fixture(&server, "single");
|
||||
let base = format!("/v1/vaults/{}", fix.vault);
|
||||
|
||||
// Minting and listing are owner-only; rows never contain a code or its hash.
|
||||
let (status, viewer_code) = server.invite(&fix.vault, &fix.owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
let rows = server.invites(&fix.vault, &fix.owner);
|
||||
assert_eq!(rows.len(), 1, "only the fresh viewer invite");
|
||||
for row in &rows {
|
||||
assert!(row.get("code").is_none(), "no code in listings");
|
||||
assert!(row.get("id").is_some() && row.get("role").is_some() && row.get("created_ms").is_some() && row.get("expires_ms").is_some());
|
||||
}
|
||||
assert_eq!(server.invite(&fix.vault, &fix.editor, "editor").0, 403, "only the owner mints");
|
||||
assert_eq!(server.json("GET", &format!("{base}/invites"), Some(&fix.viewer), &[], None).0, 403);
|
||||
|
||||
// An invite keeps the higher role of an existing member and is consumed either way.
|
||||
let (status, editor_reply) = server.accept(&fix.viewer, &server.invite(&fix.vault, &fix.owner, "editor").1);
|
||||
assert_eq!(status, 200, "{editor_reply}");
|
||||
assert_eq!(editor_reply["role"], "editor", "a viewer accepting an editor invite is upgraded");
|
||||
let (status, kept) = server.accept(&fix.viewer, &viewer_code);
|
||||
assert_eq!(status, 200, "{kept}");
|
||||
assert_eq!(kept["role"], "editor", "an editor accepting a viewer invite stays editor");
|
||||
assert_eq!(server.invites(&fix.vault, &fix.owner).len(), 0, "every accept consumed its invite");
|
||||
let (status, owner_reply) = server.accept(&fix.owner, &server.invite(&fix.vault, &fix.owner, "viewer").1);
|
||||
assert_eq!(status, 200, "{owner_reply}");
|
||||
assert_eq!(owner_reply["role"], "owner", "an owner accepting stays owner, never demoted");
|
||||
assert_eq!(server.accept(&fix.editor, &viewer_code).0, 401, "an invite is single use");
|
||||
|
||||
// Deleting an invite withdraws it before use.
|
||||
let (status, one_time) = server.invite(&fix.vault, &fix.owner, "editor");
|
||||
assert_eq!(status, 200);
|
||||
let id = server.invites(&fix.vault, &fix.owner)[0]["id"].as_str().expect("id").to_owned();
|
||||
assert_eq!(server.send("DELETE", &format!("{base}/invites/{id}"), Some(&fix.owner), &[], None).0, 204);
|
||||
assert_eq!(server.send("DELETE", &format!("{base}/invites/{id}"), Some(&fix.owner), &[], None).0, 404);
|
||||
assert_eq!(server.accept(&fix.stranger, &one_time).0, 401, "a withdrawn code is refused");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invite_lockout_shares_the_pairing_counter_and_expires() {
|
||||
let server = Server::start_temp("roles-lockout");
|
||||
let fix = fixture(&server, "lockout");
|
||||
|
||||
// Five bad accepts poison the shared counter; the next good code is refused at 429.
|
||||
for _ in 0..5 {
|
||||
let (status, _) = server.accept(&fix.stranger, "no-such-code");
|
||||
assert_eq!(status, 401);
|
||||
}
|
||||
let (status, code) = server.invite(&fix.vault, &fix.owner, "editor");
|
||||
assert_eq!(status, 200);
|
||||
let (status, reply) = server.accept(&fix.stranger, &code);
|
||||
assert_eq!(status, 429, "locked out: {reply}");
|
||||
|
||||
// The state-level window uses the injectable clock, so the lockout clears.
|
||||
let state = server.state.clone();
|
||||
let granted = state.accept_invite(&code, "d-test", "u-test", notescraft_sync_server::now_ms() + notescraft_sync_server::PAIRING_WINDOW_MS + 1);
|
||||
assert!(granted.as_ref().is_ok_and(|reply| reply.role == "editor"), "the lockout cleared: {granted:?}");
|
||||
assert_eq!(granted.expect("granted").role, "editor");
|
||||
|
||||
// An expired code is dead: the same code again, now beyond its 48 hours.
|
||||
let (status, fresh) = server.invite(&fix.vault, &fix.owner, "viewer");
|
||||
assert_eq!(status, 200);
|
||||
let late = notescraft_sync_server::now_ms() + notescraft_sync_server::INVITE_TTL_MS + 1;
|
||||
assert!(state.accept_invite(&fresh, "d-test", "u-late", late).is_err(), "expired");
|
||||
let _ = fresh;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_removed_member_gets_404_at_once_and_the_last_owner_cannot_leave() {
|
||||
let server = Server::start_temp("roles-last-owner");
|
||||
let fix = fixture(&server, "last-owner");
|
||||
let base = format!("/v1/vaults/{}", fix.vault);
|
||||
|
||||
// A second owner only arrives through an account-bound pairing code.
|
||||
let second_owner_user = server.create_user("co-owner");
|
||||
let (status, second_owner) = server.pair_user(&second_owner_user, &fix.vault, "co-phone");
|
||||
assert_eq!(status, 200);
|
||||
|
||||
// An owner removes anyone; the removed member's very next request is 404.
|
||||
let (status, _) = server.send("DELETE", &format!("{base}/members/{second_owner_user}"), Some(&fix.owner), &[], None);
|
||||
assert_eq!(status, 204);
|
||||
assert_eq!(server.json("GET", &format!("{base}/changes?since=0"), Some(&second_owner), &[], None).0, 404);
|
||||
|
||||
// With one owner left, neither leaving nor demotion works for the account owner.
|
||||
let (_, members) = server.json("GET", &format!("{base}/members"), Some(&fix.owner), &[], None);
|
||||
let owner_user = members.as_array().expect("members").iter().find(|row| row["role"] == "owner").expect("owner")["user_id"].as_str().expect("id").to_owned();
|
||||
assert_eq!(server.send("DELETE", &format!("{base}/members/{owner_user}"), Some(&fix.owner), &[], None).0, 409, "the last owner cannot leave");
|
||||
assert_eq!(server.json("PATCH", &format!("{base}/members/{owner_user}"), Some(&fix.owner), &[], Some(&json!({"role": "editor"}))).0, 409);
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
//! The share commands against the real sync server: two accounts, one shared
|
||||
//! vault and the roles that guard it.
|
||||
//!
|
||||
//! The owner pairs, makes two more vaults and invites an editor; the editor
|
||||
//! is a second account, accepts the code, binds a folder and syncs a file.
|
||||
//! Then the roles are pulled: a viewer cannot write, a removed member sees
|
||||
//! nothing, and a used code is refused.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
|
||||
use common::Server;
|
||||
use notescraft_engine::{HttpRemote, Registry, Session, SyncState, bind_folder, register_core, sync_once};
|
||||
use notescraft_vault::Vault;
|
||||
use serde_json::{Value, json};
|
||||
|
||||
/// Counter making two folders of the same test different.
|
||||
static COUNTER: AtomicU32 = AtomicU32::new(0);
|
||||
|
||||
/// The stamp the sync pass carries.
|
||||
const STAMP: &str = "2026-10-20T00:00:00Z";
|
||||
|
||||
/// A fresh folder for a vault.
|
||||
fn folder(name: &str) -> PathBuf {
|
||||
let serial = COUNTER.fetch_add(1, Ordering::Relaxed);
|
||||
let path = std::env::temp_dir().join(format!("nc-share-{name}-{serial}-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&path);
|
||||
std::fs::create_dir_all(&path).expect("create the folder");
|
||||
path
|
||||
}
|
||||
|
||||
/// The ids of the vaults the answer lists.
|
||||
fn ids(answer: &Value) -> Vec<String> {
|
||||
let mut ids: Vec<String> =
|
||||
answer["vaults"].as_array().expect("the vaults").iter().map(|vault| vault["id"].as_str().unwrap_or_default().to_owned()).collect();
|
||||
ids.sort();
|
||||
ids
|
||||
}
|
||||
|
||||
/// Two accounts share a vault, and the roles guard it.
|
||||
#[test]
|
||||
fn two_accounts_share_a_vault() {
|
||||
let server = Server::start_temp("share-e2e");
|
||||
let (vault, owner_token, owner_user) = server.owned_vault("share-e2e");
|
||||
|
||||
// The owner opens a vault bound to the server.
|
||||
let owner_folder = folder("owner");
|
||||
bind_folder(&owner_folder, &server.base, &vault, &owner_token, "device-owner", Some(&owner_user)).expect("bind the owner");
|
||||
let mut reg = Registry::new();
|
||||
register_core(&mut reg).expect("register the core commands");
|
||||
let mut owner = Session::open(&owner_folder).expect("open the owner vault");
|
||||
|
||||
// The account owns two more vaults.
|
||||
let first = reg.run("server.create_vault", &mut owner, &json!({ "name": "Work" })).expect("server.create_vault");
|
||||
let second = reg.run("server.create_vault", &mut owner, &json!({ "name": "Personal" })).expect("server.create_vault");
|
||||
let first_id = first["vault"]["id"].as_str().expect("the id of the first vault").to_owned();
|
||||
let second_id = second["vault"]["id"].as_str().expect("the id of the second vault").to_owned();
|
||||
|
||||
// The list holds exactly the three vaults of the account, and no other.
|
||||
let listed = reg.run("server.vaults", &mut owner, &Value::Null).expect("server.vaults");
|
||||
let mut expected = vec![vault.clone(), first_id.clone(), second_id.clone()];
|
||||
expected.sort();
|
||||
assert_eq!(ids(&listed), expected);
|
||||
assert_eq!(listed["vaults"][0]["role"], json!("owner"), "the creator owns its vault");
|
||||
|
||||
// The owner invites an editor; the code is in the answer and nowhere else.
|
||||
let invite = reg.run("share.invite", &mut owner, &json!({ "role": "editor" })).expect("share.invite");
|
||||
let code = invite["code"].as_str().expect("the invite code").to_owned();
|
||||
let open = reg.run("share.invites", &mut owner, &Value::Null).expect("share.invites");
|
||||
assert!(!open.to_string().contains(&code), "the listing never carries the code");
|
||||
|
||||
// A second account: a new user, paired under its own vault.
|
||||
let editor_vault = Server::fresh_vault_id("editor");
|
||||
let editor_token = server.pair_to(&editor_vault, "Editor");
|
||||
let editor_folder = folder("editor");
|
||||
bind_folder(&editor_folder, &server.base, &editor_vault, &editor_token, "device-editor", None).expect("bind the editor");
|
||||
let mut editor = Session::open(&editor_folder).expect("open the editor vault");
|
||||
|
||||
// The editor redeems the code and binds a folder to the shared vault.
|
||||
let shared_folder = folder("shared");
|
||||
let accepted = reg.run("share.accept", &mut editor, &json!({ "code": code, "folder": shared_folder.display().to_string() })).expect("share.accept");
|
||||
assert_eq!(accepted["vault_id"], json!(vault));
|
||||
assert_eq!(accepted["role"], json!("editor"));
|
||||
let binding = std::fs::read_to_string(shared_folder.join(".notescraft/sync/binding.json")).expect("the binding");
|
||||
assert!(binding.contains(&vault), "{binding}");
|
||||
assert!(!binding.contains(&editor_token), "the token never reaches the binding");
|
||||
|
||||
// The owner sees the editor among the members.
|
||||
let members = reg.run("share.members", &mut owner, &Value::Null).expect("share.members");
|
||||
let rows = members["members"].as_array().expect("the members").clone();
|
||||
assert_eq!(rows.len(), 2, "{rows:?}");
|
||||
let editor_row = rows.iter().find(|row| row["user_id"] != json!(owner_user)).expect("the editor");
|
||||
assert_eq!(editor_row["role"], json!("editor"));
|
||||
let editor_user = editor_row["user_id"].as_str().expect("the user of the editor").to_owned();
|
||||
|
||||
// The editor syncs a file of the shared vault.
|
||||
std::fs::write(shared_folder.join("note.md"), "# Shared\n").expect("write the note");
|
||||
let shared = Vault::open(&shared_folder).expect("open the shared folder");
|
||||
let remote = HttpRemote::new(&server.base, &vault, &editor_token);
|
||||
let mut state = SyncState::default();
|
||||
let report = sync_once(&shared, &remote, &mut state, "device-editor", STAMP).expect("sync");
|
||||
assert!(report.uploaded.contains(&"note.md".to_owned()), "{report:?}");
|
||||
let (status, body) = server.send("GET", &format!("/v1/vaults/{vault}/files/note.md"), Some(&editor_token), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
assert_eq!(String::from_utf8_lossy(&body), "# Shared\n", "the note reached the server");
|
||||
|
||||
// A viewer cannot write any more.
|
||||
reg.run("share.set_role", &mut owner, &json!({ "user": editor_user, "role": "viewer" })).expect("share.set_role");
|
||||
let (status, _) = server.put(&vault, &editor_token, "note.md", "changed\n", 0);
|
||||
assert_eq!(status, 403, "a viewer cannot write");
|
||||
|
||||
// A removed member sees nothing of the vault any more.
|
||||
reg.run("share.remove", &mut owner, &json!({ "user": editor_user })).expect("share.remove");
|
||||
let (status, _) = server.put(&vault, &editor_token, "note.md", "again\n", 0);
|
||||
assert_eq!(status, 404, "a removed member is a stranger to the vault");
|
||||
|
||||
// The code is used: it fails the second time, with a message.
|
||||
let again = reg.run("share.accept", &mut editor, &json!({ "code": code, "folder": shared_folder.display().to_string() }));
|
||||
let Err(notescraft_engine::Error::Command { source, .. }) = again else {
|
||||
panic!("a used code must be refused");
|
||||
};
|
||||
assert!(source.message.contains("401"), "{}", source.message);
|
||||
}
|
||||
|
||||
/// A one-time code connects a folder to a server, from an unbound vault.
|
||||
#[test]
|
||||
fn a_code_connects_a_folder_to_a_server() {
|
||||
let server = Server::start_temp("share-connect");
|
||||
let vault = Server::fresh_vault_id("connect");
|
||||
let (status, minted) = server.json("POST", &format!("/v1/vaults/{vault}/pairing"), Some(common::ADMIN), &[], None);
|
||||
assert_eq!(status, 200);
|
||||
let code = minted["code"].as_str().expect("the pairing code").to_owned();
|
||||
|
||||
let mut reg = Registry::new();
|
||||
register_core(&mut reg).expect("register the core commands");
|
||||
let unbound = folder("unbound");
|
||||
let mut session = Session::open(&unbound).expect("open the vault");
|
||||
|
||||
let target = folder("connected");
|
||||
let answer = reg
|
||||
.run("server.connect", &mut session, &json!({ "server": server.base, "code": code, "folder": target.display().to_string() }))
|
||||
.expect("server.connect");
|
||||
assert_eq!(answer["vault_id"], json!(vault));
|
||||
|
||||
let binding = std::fs::read_to_string(target.join(".notescraft/sync/binding.json")).expect("the binding");
|
||||
assert!(binding.contains(&vault), "{binding}");
|
||||
assert!(binding.contains(&server.base), "{binding}");
|
||||
let credentials = target.join(".notescraft/sync/credentials.json");
|
||||
assert!(std::fs::read_to_string(&credentials).expect("the credentials").contains("token"));
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let mode = std::fs::metadata(&credentials).expect("stat").permissions().mode() & 0o777;
|
||||
assert_eq!(mode, 0o600, "the token file is private");
|
||||
}
|
||||
|
||||
// The bound folder sees the vaults of its account.
|
||||
let mut connected = Session::open(&target).expect("open the connected folder");
|
||||
let listed = reg.run("server.vaults", &mut connected, &Value::Null).expect("server.vaults");
|
||||
assert_eq!(ids(&listed), vec![vault]);
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
//! Store level: idempotent migration of old databases, device lifecycle, audit row order.
|
||||
|
||||
use notescraft_sync_server::Store;
|
||||
|
||||
/// The legacy schema, before `last_seen_ms`, `revoked` and `updated_ms` existed.
|
||||
const LEGACY: &str = "
|
||||
CREATE TABLE vaults (id TEXT PRIMARY KEY, created_ms INTEGER NOT NULL);
|
||||
CREATE TABLE pairings (code_hash TEXT PRIMARY KEY, vault TEXT NOT NULL, expires_ms INTEGER NOT NULL);
|
||||
CREATE TABLE devices (id TEXT PRIMARY KEY, vault TEXT NOT NULL, name TEXT NOT NULL, token_hash TEXT NOT NULL UNIQUE, created_ms INTEGER NOT NULL);
|
||||
CREATE TABLE files (vault TEXT NOT NULL, path TEXT NOT NULL, hash TEXT NOT NULL, size INTEGER NOT NULL,
|
||||
mtime_ms INTEGER NOT NULL, seq INTEGER NOT NULL, deleted INTEGER NOT NULL, PRIMARY KEY (vault, path));
|
||||
";
|
||||
|
||||
fn columns_of(path: &std::path::Path, table: &str) -> Vec<String> {
|
||||
let conn = rusqlite::Connection::open(path).expect("open");
|
||||
let mut statement = conn.prepare(&format!("PRAGMA table_info({table})")).expect("pragma");
|
||||
statement.query_map([], |row| row.get::<_, String>(1)).expect("names").collect::<rusqlite::Result<Vec<String>>>().expect("rows")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn migration_adds_missing_columns_and_runs_more_than_once() {
|
||||
let dir = std::env::temp_dir().join(format!("nc-sync-store-{}", std::process::id()));
|
||||
std::fs::remove_dir_all(&dir).ok();
|
||||
std::fs::create_dir_all(&dir).expect("temp");
|
||||
let path = dir.join("old.db");
|
||||
rusqlite::Connection::open(&path).expect("open").execute_batch(LEGACY).expect("legacy schema");
|
||||
|
||||
Store::open(&path).expect("first open migrates");
|
||||
Store::open(&path).expect("second open is idempotent");
|
||||
for (table, column) in [
|
||||
("devices", "last_seen_ms"),
|
||||
("devices", "revoked"),
|
||||
("devices", "user_id"),
|
||||
("files", "updated_ms"),
|
||||
("files", "editor_user"),
|
||||
("pairings", "user"),
|
||||
("vaults", "name"),
|
||||
("vaults", "deleted_ms"),
|
||||
] {
|
||||
assert!(columns_of(&path, table).iter().any(|name| name == column), "{table}.{column} missing");
|
||||
}
|
||||
let conn = rusqlite::Connection::open(&path).expect("open");
|
||||
for table in ["users", "members", "invites"] {
|
||||
// `sqlite_master` row proves the table exists after the migration.
|
||||
let count: u32 = conn.query_row("SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?1", [table], |row| row.get(0)).expect("query");
|
||||
assert_eq!(count, 1, "{table} missing");
|
||||
}
|
||||
std::fs::remove_dir_all(&dir).expect("cleanup");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rotation_and_revocation_take_the_old_token_out_of_lookup() {
|
||||
let store = Store::in_memory().expect("db");
|
||||
store.create_user("u-1", "phone user", "#89b4fa", 0).expect("user");
|
||||
store.add_device("d", "v", "u-1", "phone", "old-hash", 1).expect("add");
|
||||
assert_eq!(store.device_for("old-hash").expect("query"), Some(("d".to_owned(), "u-1".to_owned())));
|
||||
|
||||
assert!(store.rotate_device("d", "v", "new-hash").expect("rotate"));
|
||||
assert_eq!(store.device_for("old-hash").expect("query"), None);
|
||||
assert_eq!(store.device_for("new-hash").expect("query"), Some(("d".to_owned(), "u-1".to_owned())));
|
||||
|
||||
// `last_seen_ms` moves at most once a minute.
|
||||
store.touch_device("new-hash", 10_000).expect("first touch");
|
||||
store.touch_device("new-hash", 40_000).expect("too soon, ignored");
|
||||
let (_, _, _, last_seen): (String, String, u64, Option<u64>) = store.list_devices("v").expect("rows").remove(0);
|
||||
assert_eq!(last_seen, Some(10_000));
|
||||
store.touch_device("new-hash", 70_000).expect("a minute later, applied");
|
||||
let (_, _, _, last_seen): (String, String, u64, Option<u64>) = store.list_devices("v").expect("rows").remove(0);
|
||||
assert_eq!(last_seen, Some(70_000));
|
||||
|
||||
assert!(store.revoke_device("d", "v").expect("revoke"));
|
||||
assert_eq!(store.device_for("new-hash").expect("query"), None);
|
||||
assert_eq!(store.list_devices("v").expect("rows").len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn audit_rows_come_back_newest_first_and_capped() {
|
||||
let store = Store::in_memory().expect("db");
|
||||
for (ts, kind) in [(1_u64, "a"), (2, "b"), (3, "c")] {
|
||||
store.audit(ts, kind, Some("v"), Some("d"), None).expect("audit");
|
||||
}
|
||||
let rows = store.audit_rows(2).expect("rows");
|
||||
assert_eq!(rows.iter().map(|row| row.kind.as_str()).collect::<Vec<_>>(), vec!["c", "b"]);
|
||||
assert_eq!(rows[0].ts_ms, 3);
|
||||
assert_eq!(rows[0].vault.as_deref(), Some("v"));
|
||||
}
|
||||
@@ -16,13 +16,14 @@ workspace = true
|
||||
|
||||
[dependencies]
|
||||
notescraft-ui-egui = { workspace = true }
|
||||
egui_extras = { workspace = true, features = ["image"] }
|
||||
notescraft-engine = { workspace = true }
|
||||
notescraft-themes = { workspace = true }
|
||||
notescraft-calendar = { workspace = true }
|
||||
notescraft-templates = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
egui = { workspace = true }
|
||||
eframe = { workspace = true, features = ["wgpu", "default_fonts", "accesskit", "wayland", "x11"] }
|
||||
eframe = { workspace = true, features = ["wgpu", "accesskit", "wayland", "x11"] }
|
||||
log = { workspace = true }
|
||||
cpal = { workspace = true }
|
||||
craft-quit = { workspace = true, features = ["egui"] }
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
//! The launcher window: the main menu shown when no vault is named and none can be reopened, or
|
||||
//! with `--launcher`. It lists the remembered vaults ([`notescraft_ui_egui::launcher::Launcher`]) and
|
||||
//! starts the chosen one as a separate process, because a vault is bound to one `Session` for the
|
||||
//! life of its process.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::ExitCode;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
use eframe::egui;
|
||||
use notescraft_engine::{AppSettings, LAUNCH_BUDGET, MAX_NAME_CHARS, list_rows};
|
||||
use notescraft_ui_egui::launcher::{Launcher, LauncherAction};
|
||||
use notescraft_ui_egui::theme::Tokens;
|
||||
|
||||
/// Window size of the launcher, points.
|
||||
const WINDOW_SIZE: [f32; 2] = [1100.0, 720.0];
|
||||
/// Smallest launcher window, points (phone-width layouts start below 600).
|
||||
const MIN_SIZE: [f32; 2] = [360.0, 480.0];
|
||||
|
||||
/// The eframe app around the launcher screen.
|
||||
struct LauncherWindow {
|
||||
launcher: Launcher,
|
||||
tokens: Tokens,
|
||||
}
|
||||
|
||||
/// Milliseconds since the epoch.
|
||||
pub fn now_ms() -> u64 {
|
||||
SystemTime::now().duration_since(UNIX_EPOCH).ok().and_then(|since| u64::try_from(since.as_millis()).ok()).unwrap_or(0)
|
||||
}
|
||||
|
||||
/// The catalog rows for the launcher, with the launch time budget for the folder checks.
|
||||
fn rows() -> Vec<notescraft_engine::VaultRow> {
|
||||
let (settings, _notes) = notescraft_engine::load();
|
||||
list_rows(&settings, LAUNCH_BUDGET.max(Duration::from_millis(1)))
|
||||
}
|
||||
|
||||
/// Starts `path` as a vault in its own process; `false` when it could not be started.
|
||||
fn open_vault(path: &Path) -> bool {
|
||||
std::env::current_exe().and_then(|exe| std::process::Command::new(exe).arg(path).spawn()).is_ok()
|
||||
}
|
||||
|
||||
/// Loads the settings, applies `change` and saves them; errors are logged, never fatal.
|
||||
fn edit_settings(change: impl FnOnce(&mut AppSettings)) {
|
||||
let (mut settings, _notes) = notescraft_engine::load();
|
||||
change(&mut settings);
|
||||
let _ = settings.validate();
|
||||
if let Err(error) = notescraft_engine::save(&settings) {
|
||||
log::warn!("could not save the vault list: {error}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Shows `path` in the platform file manager.
|
||||
fn reveal(path: &Path) {
|
||||
let target = path.to_string_lossy().into_owned();
|
||||
let result = if cfg!(target_os = "macos") {
|
||||
std::process::Command::new("open").args(["-R", &target]).spawn()
|
||||
} else if cfg!(target_os = "windows") {
|
||||
std::process::Command::new("explorer").arg(format!("/select,{target}")).spawn()
|
||||
} else {
|
||||
std::process::Command::new("xdg-open").arg(path.parent().unwrap_or(path)).spawn()
|
||||
};
|
||||
if let Err(error) = result {
|
||||
log::warn!("could not show the folder: {error}");
|
||||
}
|
||||
}
|
||||
|
||||
impl LauncherWindow {
|
||||
/// Performs what the user asked; true when the launcher should close (a vault window took over).
|
||||
fn perform(&mut self, ctx: &egui::Context, action: LauncherAction) -> bool {
|
||||
match action {
|
||||
LauncherAction::Open { path, new_window } => open_vault(Path::new(&path)) && !new_window,
|
||||
LauncherAction::OpenFolder | LauncherAction::CreateVault => {
|
||||
let picked = rfd::FileDialog::new().set_title("Open or create a vault folder").pick_folder();
|
||||
picked.is_some_and(|folder| open_vault(&folder))
|
||||
}
|
||||
LauncherAction::Pin { path, pinned } => {
|
||||
edit_settings(|settings| notescraft_engine::set_pinned(settings, &path, pinned));
|
||||
self.launcher.set_rows(rows());
|
||||
false
|
||||
}
|
||||
LauncherAction::Rename { path, name } => {
|
||||
let name = name.trim().to_owned();
|
||||
if name.chars().count() <= MAX_NAME_CHARS {
|
||||
edit_settings(|settings| notescraft_engine::set_name(settings, &path, (!name.is_empty()).then_some(name)));
|
||||
}
|
||||
self.launcher.set_rows(rows());
|
||||
false
|
||||
}
|
||||
LauncherAction::Remove { path } => {
|
||||
edit_settings(|settings| {
|
||||
notescraft_engine::forget(settings, &path);
|
||||
});
|
||||
self.launcher.set_rows(rows());
|
||||
false
|
||||
}
|
||||
LauncherAction::Reveal { path } => {
|
||||
reveal(Path::new(&path));
|
||||
false
|
||||
}
|
||||
LauncherAction::CopyPath { path } => {
|
||||
ctx.copy_text(path);
|
||||
false
|
||||
}
|
||||
// Server vaults, Settings and Help come with the sync UI; nothing to start yet.
|
||||
// Connect and Join are dialogs on the launcher itself; a finished one reports the bound folder.
|
||||
LauncherAction::ConnectToServer | LauncherAction::JoinWithCode | LauncherAction::OpenSettings | LauncherAction::OpenHelp => false,
|
||||
LauncherAction::ConnectedFolder { path } => open_vault(Path::new(&path)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl eframe::App for LauncherWindow {
|
||||
fn ui(&mut self, ui: &mut egui::Ui, _frame: &mut eframe::Frame) {
|
||||
let ctx = ui.ctx().clone();
|
||||
if let Some(action) = self.launcher.show(&ctx, &self.tokens)
|
||||
&& self.perform(&ctx, action)
|
||||
{
|
||||
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
|
||||
}
|
||||
}
|
||||
|
||||
fn clear_color(&self, _visuals: &egui::Visuals) -> [f32; 4] {
|
||||
notescraft_ui_egui::theme::color(self.tokens.theme.colors.canvas).to_normalized_gamma_f32()
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs the launcher until the window closes.
|
||||
pub fn run() -> ExitCode {
|
||||
let Some(theme) = notescraft_themes::builtin("catppuccin-mocha") else {
|
||||
eprintln!("built-in theme catppuccin-mocha missing (broken build)");
|
||||
return ExitCode::from(1);
|
||||
};
|
||||
let viewport = egui::ViewportBuilder::default().with_title("NotesCraft").with_inner_size(WINDOW_SIZE).with_min_inner_size(MIN_SIZE);
|
||||
let options = eframe::NativeOptions { viewport, ..Default::default() };
|
||||
let result = eframe::run_native(
|
||||
"NotesCraft",
|
||||
options,
|
||||
Box::new(move |creation| {
|
||||
egui_extras::install_image_loaders(&creation.egui_ctx);
|
||||
let tokens = Tokens::new(theme);
|
||||
notescraft_ui_egui::fonts::install(&creation.egui_ctx);
|
||||
notescraft_ui_egui::style::apply(&creation.egui_ctx, &tokens);
|
||||
Ok(Box::new(LauncherWindow { launcher: Launcher::new(rows(), now_ms()), tokens }) as Box<dyn eframe::App>)
|
||||
}),
|
||||
);
|
||||
match result {
|
||||
Ok(()) => ExitCode::SUCCESS,
|
||||
Err(error) => {
|
||||
eprintln!("window error: {error}");
|
||||
ExitCode::from(1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The folder a plain launch should reopen: the newest recent vault that still exists.
|
||||
pub fn last_vault() -> Option<PathBuf> {
|
||||
let (settings, _notes) = notescraft_engine::load();
|
||||
settings.recent_vaults.iter().map(PathBuf::from).find(|path| path.is_dir())
|
||||
}
|
||||
+17
-14
@@ -13,6 +13,7 @@ mod audio;
|
||||
mod desktop;
|
||||
mod google;
|
||||
mod keychain;
|
||||
mod launcher_window;
|
||||
#[cfg(target_os = "macos")]
|
||||
mod menu;
|
||||
mod mixer;
|
||||
@@ -78,6 +79,13 @@ impl eframe::App for NotesCraftWindow {
|
||||
VaultRequest::Open(path) => (Some(path), false),
|
||||
VaultRequest::OpenLarge(path) => (Some(path), true),
|
||||
VaultRequest::Pick => (rfd::FileDialog::new().set_title("Open or create a vault folder").pick_folder(), false),
|
||||
VaultRequest::Launcher => {
|
||||
// A separate window; this one keeps its vault open.
|
||||
if let Err(error) = std::env::current_exe().and_then(|exe| std::process::Command::new(exe).arg("--launcher").spawn()) {
|
||||
log::warn!("could not start the launcher: {error}");
|
||||
}
|
||||
return;
|
||||
}
|
||||
};
|
||||
let relaunch = |target: std::path::PathBuf| {
|
||||
let mut command = std::process::Command::new(std::env::current_exe()?);
|
||||
@@ -125,8 +133,10 @@ enum Launch {
|
||||
Print(String),
|
||||
/// Open this vault.
|
||||
Vault(std::path::PathBuf),
|
||||
/// No vault named: use the last one or ask.
|
||||
/// No vault named: use the last one or show the launcher.
|
||||
Choose,
|
||||
/// `--launcher`: show the vault launcher even when a recent vault exists.
|
||||
Launcher,
|
||||
}
|
||||
|
||||
/// Reads the command line: `--version`, `--help`, one vault path, or nothing. Arguments the OS adds
|
||||
@@ -135,8 +145,9 @@ fn parse_launch(args: &[std::ffi::OsString]) -> Result<Launch, StartError> {
|
||||
let args: Vec<&std::ffi::OsString> = args.iter().filter(|arg| !arg.to_string_lossy().starts_with("-psn_")).collect();
|
||||
match args.first().map(|arg| arg.to_string_lossy()) {
|
||||
Some(flag) if matches!(flag.as_ref(), "--version" | "-V") => Ok(Launch::Print(version_line())),
|
||||
Some(flag) if flag == "--launcher" => Ok(Launch::Launcher),
|
||||
Some(flag) if matches!(flag.as_ref(), "--help" | "-h") => Ok(Launch::Print(format!(
|
||||
"{}\n\nUsage: notescraft [vault-folder]\n\nWith no folder it reopens the last vault, or asks for one.\nNOTESCRAFT_VAULT names a vault too.",
|
||||
"{}\n\nUsage: notescraft [vault-folder | --launcher]\n\nWith no folder it reopens the last vault, or shows the vault launcher.\n--launcher always shows it. NOTESCRAFT_VAULT names a vault too.",
|
||||
version_line()
|
||||
))),
|
||||
Some(_) if args.len() == 1 => Ok(args.first().map_or(Launch::Choose, |arg| Launch::Vault(std::path::PathBuf::from(arg)))),
|
||||
@@ -145,16 +156,6 @@ fn parse_launch(args: &[std::ffi::OsString]) -> Result<Launch, StartError> {
|
||||
}
|
||||
}
|
||||
|
||||
/// The vault to open when none was named: the newest recent one that still exists, else a folder
|
||||
/// dialog. `None` when the user cancels.
|
||||
fn choose_vault() -> Option<std::path::PathBuf> {
|
||||
let (settings, _notes) = notescraft_engine::load();
|
||||
if let Some(recent) = settings.recent_vaults.iter().map(std::path::PathBuf::from).find(|path| path.is_dir()) {
|
||||
return Some(recent);
|
||||
}
|
||||
rfd::FileDialog::new().set_title("Open a vault folder").pick_folder()
|
||||
}
|
||||
|
||||
/// Recent vaults that still exist, newest first.
|
||||
fn recent_vaults() -> Vec<std::path::PathBuf> {
|
||||
let (settings, _notes) = notescraft_engine::load();
|
||||
@@ -165,7 +166,7 @@ fn recent_vaults() -> Vec<std::path::PathBuf> {
|
||||
fn remember_vault(path: &std::path::Path) {
|
||||
let (mut settings, _notes) = notescraft_engine::load();
|
||||
let text = path.canonicalize().unwrap_or_else(|_| path.to_path_buf()).to_string_lossy().into_owned();
|
||||
settings.recent_vaults.insert(0, text);
|
||||
notescraft_engine::touch(&mut settings, &text, launcher_window::now_ms());
|
||||
let _ = settings.validate();
|
||||
if let Err(error) = notescraft_engine::save(&settings) {
|
||||
log::warn!("could not remember the vault: {error}");
|
||||
@@ -193,9 +194,11 @@ fn main() -> ExitCode {
|
||||
}
|
||||
Ok(Launch::Vault(path)) => path,
|
||||
Ok(Launch::Choose) => {
|
||||
let Some(path) = choose_vault() else { return ExitCode::SUCCESS };
|
||||
// The newest recent vault reopens at once; with none left the launcher is the main menu.
|
||||
let Some(path) = launcher_window::last_vault() else { return launcher_window::run() };
|
||||
path
|
||||
}
|
||||
Ok(Launch::Launcher) => return launcher_window::run(),
|
||||
Err(error) => {
|
||||
eprintln!("{error}");
|
||||
return ExitCode::from(2);
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,92 @@
|
||||
This Font Software is licensed under the SIL Open Font License,
|
||||
Version 1.1.
|
||||
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
http://scripts.sil.org/OFL
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font
|
||||
creation efforts of academic and linguistic communities, and to
|
||||
provide a free and open framework in which fonts may be shared and
|
||||
improved in partnership with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply to
|
||||
any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software
|
||||
components as distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to,
|
||||
deleting, or substituting -- in part or in whole -- any of the
|
||||
components of the Original Version, by changing formats or by porting
|
||||
the Font Software to a new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed,
|
||||
modify, redistribute, and sell modified and unmodified copies of the
|
||||
Font Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components, in
|
||||
Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the
|
||||
corresponding Copyright Holder. This restriction only applies to the
|
||||
primary font name as presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created using
|
||||
the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.75" stroke-linecap="round" stroke-linejoin="round"><path d="M16 7h.01"/><path d="M3.4 18H12a8 8 0 0 0 8-8V7a4 4 0 0 0-7.28-2.3L2 20"/><path d="m20 7 2 .5-2 .5"/><path d="M10 18v3"/><path d="M14 17.75V21"/></svg>
|
||||
|
After Width: | Height: | Size: 345 B |
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m21 16-4 4-4-4"/><path d="M17 20V4"/><path d="m3 8 4-4 4 4"/><path d="M7 4v16"/></svg>
|
||||
|
After Width: | Height: | Size: 277 B |
@@ -0,0 +1,14 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
width="24"
|
||||
height="24"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
>
|
||||
<path d="M10.268 21a2 2 0 0 0 3.464 0" />
|
||||
<path d="M3.262 15.326A1 1 0 0 0 4 17h16a1 1 0 0 0 .74-1.673C19.41 13.956 18 12.499 18 8A6 6 0 0 0 6 8c0 4.499-1.411 5.956-2.738 7.326" />
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 393 B |
@@ -0,0 +1,15 @@
|
||||
<svg
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
width="24"
|
||||
height="24"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
>
|
||||
<circle cx="12" cy="12" r="10" />
|
||||
<path d="m15 9-6 6" />
|
||||
<path d="m9 9 6 6" />
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 293 B |
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m18 16 4-4-4-4"/><path d="m6 8-4 4 4 4"/><path d="m14.5 4-5 16"/></svg>
|
||||
|
After Width: | Height: | Size: 262 B |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user