notescraft/specs/vault-knowledge/spec.md

Vault knowledge (retrieval for the assistant and related notes)

Problem

The assistant answers about the vault by calling note-level search and reading whole notes. That costs tokens, misses the right paragraph inside a long note, and misses notes that use other words for the same idea. The user also cannot see which notes the answer came from, and has no way to find notes related to the one they are reading beyond backlinks. Evidence: owner request in the 2026-10-17 session; the assistant already burned 60-70k tokens per prompt before the lean-config work; search returns whole-note hits (crates/index/src/search/rank.rs).

Goal

Every assistant question and every open note can be answered with the few most relevant passages of the vault, shown with their sources, found on the device without an embedding model.

Decision: no vector index in the first release

• Retrieval is passage-level BM25 plus three signals the vault already has: the link graph, tags, and a vocabulary learned from the user's own notes (words that appear together). This covers the vocabulary gap that vectors are normally bought for, with no model download, no new dependency and no data leaving the device.
• The scorer is behind a small trait so a dense scorer can join later. It is added only if the evaluation set (below) shows recall@6 under 80% for the hybrid scorer. Adding it needs owner approval for a model and a crate.

Non-Goals

• Embeddings, a vector database or any network call.
• Generating answers inside NotesCraft. The configured agent writes the answer; NotesCraft supplies the passages.
• Indexing attachments, PDFs or images.
• A native mobile build. The new screens must lay out at phone width (390 pt), but the app shell is unchanged.
• Cross-vault search.

User Stories

• As a note taker, I want the assistant to start from the best passages of my notes, so that its answers are about my vault and cheap.
• As a reader, I want to see which notes and passages an answer used, so that I can trust and open them.
• As a writer, I want a Related tab for the open note, so that I find connected notes without exact words.
• As a privacy-minded user, I want to exclude folders and know nothing leaves the device, so that private notes stay private.
• As a phone user, I want the same views in one column with a bottom sheet for sources, so that they are usable at 390 pt.
• As a scripter, I want the same retrieval from the CLI and MCP with JSON output.

Acceptance Criteria

Retrieval

• Given a vault, when knowledge.search runs with a question, then it returns at most limit passages (default 6), each with note path, heading path, byte range, text, score and the signals that raised it (text, link, tag, vocab).
• Given a note longer than 1,200 characters, when it is indexed, then it is split at headings and then paragraphs into passages of at most 1,200 characters, and every passage keeps its heading path.
• Given a question using a word the vault never contains but a learned co-occurring word does (for example reminder and notify), when searching, then notes about the co-occurring word can be returned and the vocab signal names the added words.
• Given a hit note, when follow links is on, then notes it links to or that link to it rank above equal-text notes, and the link signal is listed.
• Given an excluded folder, when indexing or searching, then no passage from it is ever stored or returned.
• Given a note edited, created, renamed or deleted (by the app or by an external change), when the change is applied, then the passage index reflects it within 500 ms and never returns stale text.
• Given the evaluation set crates/knowledge/tests/eval/, when it runs, then recall@6 is at least 80% and the numbers are printed.
• Given a vault of 10,000 notes, when searching, then p99 latency is under 50 ms and a full build takes under 3 s.

Commands, CLI, assistant

• Given the capabilities knowledge.search, knowledge.context, knowledge.related, knowledge.status, knowledge.rebuild, when invoked as an engine command, from the CLI with --json, and as an MCP tool, then all three return the same result.
• Given "add context automatically" is on, when the user sends a prompt, then the panel adds the top passages (within the token budget) to the prompt and lists them under the message as sources.
• Given the assistant calls vault_context, when the call is a read, then it needs no review, and passages from excluded folders never appear.
• Given the per-plugin tool toggle knowledge is off, when the assistant lists tools, then no knowledge tool appears and no context is added.

Screens (design: PaperCraft project NotesCraft, page Knowledge, artboards "Related notes", "Assistant sources", "Settings / Knowledge", "Mobile / Sources sheet")

• Given the right card, when it renders, then tabs Backlinks, Related, Lint (added by the editor spec, hidden until that ships), Outline, Assistant show; the active tab shows its icon and label, the others only the icon.
• Given the Related tab, when a note is open, then each row shows name, folder, a 48-pt score bar, a one-line snippet and signal chips; selecting a row expands "Why this result" with link, tag and shared-word lines; the list updates when the note changes.
• Given an answer, when it has sources, then a Context card lists numbered passages (name, folder and heading, score, two-line snippet) with Use all and Edit sources, and the answer carries [n] markers and source chips that open the note at the passage.
• Given Settings > AI assistant, when opened, then a Knowledge section shows index status (notes, passages, last build, search time, Rebuild) and the toggles, sliders and excluded-folder chips from the design.
• Given a window narrower than 600 pt, when these screens render, then rows wrap without clipping, sources open as a bottom sheet, and every control is at least 44 pt tall.
• Given every new user-visible string, when cargo xtask i18n runs, then it passes.

Robustness

• Given hostile input (huge notes, binary text, 1 MB questions, deep headings, invalid ranges), when indexed or searched, then nothing panics and caps return an error or truncate with a flag.

Constraints

• Stack: Rust only, egui, no new third-party crate. New pure crate notescraft-knowledge at L2 (depends on notescraft-index); layering per cargo xtask layers.
• Everything is an engine command (SPEC 10). Private state only under .notescraft/ through the atomic writer; the passage index is memory-only and rebuilt on open; only the learned vocabulary is cached (.notescraft/knowledge-vocab.json).
• Performance: see Acceptance Criteria; the index updates incrementally, never on the UI thread for a full build.
• Security: nothing leaves the device; excluded folders and .notescraft/, .obsidian/ are never indexed; passages sent to the agent are bounded by the token budget.
• Never crash: no unwrap/panics; hostile numbers use checked arithmetic.
• Design: follow the PaperCraft artboards above. Sizes: 4/8 pt spacing, rows at least 40 pt (44 pt in compact), radii 12/16, tokens from the active theme.

Open Questions

1. ASSUMED default: passage size 1,200 characters, default 6 passages, default budget 1,200 tokens (4 characters per token).
2. ASSUMED default: auto context is on for new users.
3. ASSUMED default: English stop words and a light suffix stripper only; other languages use plain tokens.
4. ASSUMED default: "Prefer recent notes" is off by default.