Editor previews (colour swatches, arrows, embeds), the 18 obsidian-basics tasks (find and replace with regex, back/forward, auto-pair, footnotes, folding, search operators, split view, PDF/audio/ video embeds, spelling, drag and drop, pinned tabs, settings) and round 2: ```query blocks, stacked tabs, pop-out windows, tasks ticked inside embeds, text around an embed, embeds that follow edits, block menu shortcut. New crates: notescraft-regex (linear-time engine) and notescraft-syntax. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
12 KiB
Plan: Fast open and large-vault explorer
Approach
Open in three stages: list names (instant), show the window, then read notes in small time-boxed steps driven by the UI frame loop. A cooperative step loop keeps the Session single-owner (no threads, locks or shared index) and reuses knowledge_begin_scan / knowledge_end_scan, which already bracket a scan. Content search before the index is ready is a streaming grep over disk on its own thread (ripgrep library crates). The index cache is deferred until a release-build profile shows which stage dominates, so we cache the real cost and nothing else. The explorer becomes a virtualized list over a flat row model.
Measured today (release, 10,000 notes, cold): info takes 2.6 s. The window cannot appear during that time because Session::open runs before eframe::run_native (apps/notescraft/src/main.rs:180). The debug build and a folder with many non-note files (for example a target/ folder) are far worse.
Architecture
- Data flow, open:
main→Vault::open+Vault::list(names, sizes, mtimes) →Session::open_lazy(empty indexes, scan queue) → window → each frameNotesApp::uicallsSession::index_step(budget)→ note read →Index::update_note,SearchIndex::update, knowledge and lint feeds (same callsindex_notemakes today) → at queue endknowledge_end_scan. - Data flow, tree: built from the listing (file names), not from the index, so it exists before any note is parsed.
- Data flow, search while indexing:
SearchPanel::refresh→Session::search→ if not done,ContentSearchthread (walk + grep-searcher over the vault) → hits merged into the panel on later frames; a new query cancels the old thread through a shared flag. - New modules:
- indexing step loop and progress state in the engine (queue, counters, pause flag,
IndexProgress) - content search over disk in the engine (thread, cancel flag, hit type reusing the existing search
Hit) - large-folder pre-count (bounded walk that stops past the threshold) in the vault crate
- index status card and indexing chip in the UI
- large-folder dialog in the UI
- indexing step loop and progress state in the engine (queue, counters, pause flag,
- Modified modules:
Vault::listalso returns size and modified time;Sessiongetsopen_lazy,index_step,index_progress;NotesApp::newtakes the tree from the listing;tree.rsvirtualizes rows and carries folder note counts; sidebar gets filter box, toolbar actions, pinned section, sticky breadcrumb; main shows the dialog before opening. - Commands (all UI commands, no default key binding):
explorer.collapse_all,explorer.reveal_active,explorer.focus_filter,index.pause,index.resume.Session::open(eager) stays for the CLI, MCP and tests.
Files
Paths checked; line numbers from the code as read.
- Create
crates/engine/src/vault/indexing.rs— scan queue,index_step, progress, pause; newest and focused notes first.crates/engine/src/vault/content_search.rs— threaded grep over disk with cancel; hostile-input bounds (line length, file size, hit cap).crates/ui-egui/src/panels/index_status.rs— status card and header chip.crates/ui-egui/src/widgets/large_folder_dialog.rs— warning dialog.
- Modify
crates/vault/src/lib.rs:72,332—VaultEntrygainssizeandmodified; add a boundedcount_files(limit)for the pre-count; hidden entries and symlinks stay skipped aslistdoes.crates/engine/src/vault/session.rs:73-100,320-355— addopen_lazy;with_vaultsplit so eager open = lazy open + run all steps;scanbecomes the queue seed.is_markdownunchanged.crates/engine/src/vault/mod.rs— declare the two new modules, registerindex.*commands.crates/engine/src/vault/search.rs:38-41—searchstays; add a thinsearch_progressthat says whether results are partial.crates/engine/src/knowledge/sync.rs:96-119— no behavior change;begin/endcalled from the lazy path (end runs when the queue empties).apps/notescraft/src/main.rs:163-184— count files first and show the dialog; useopen_lazy.crates/ui-egui/src/shell/app.rs:147-153— tree from the listing; callindex_stepinuiwith a frame budget andrequest_repaintwhile not done.crates/ui-egui/src/shell/app/window/commands.rs:308-318—refresh_treereads the listing, not the index; tree rebuild no longer needsnode_ids(already O(n) per refresh, so call it only on structure change events).crates/ui-egui/src/panels/tree.rs:41-50,228-272—TreeNode.notescount;visibleproduces a flat row list cached per (model version, collapse set);show_with_bookmarksusesScrollArea::show_rowswith overscan; scroll-to-row API for reveal;MAX_NODEShit becomes a visible "truncated" row, not a silent cut.crates/ui-egui/src/panels/sidebar.rs:46-160— toolbar (filter, collapse all, reveal toggle, pin), pinned section, sticky breadcrumb, status card slot.crates/ui-egui/src/panels/search_panel.rs:48-56— show partial results and a "still reading" line; stream content-search hits.xtask/src/layers.rsand rootCargo.toml— register new dependencies (ignore,grep-searcher,grep-regex,grep-matcher) for the engine crate only.- i18n string tables — new keys;
cargo xtask i18nmust stay green. ROADMAP.md,docs/ARCHITECTURE.md— status and API (per AGENTS.md).
- Delete: none.
Data
- No schema change in phase 1.
VaultEntry { path, is_dir, size, modified }(additive).IndexProgress { total, done, paused, finished }.- Measured (2026-10-18, owner's Mac, release build): 5,000 notes: list 8.6 ms,
open_lazy8.8 ms, read all 63 ms, link index 22 ms, search index 59 ms, knowledge passages 38 ms, vocabulary 90 ms (already cached), whole index 435 ms. 10,000-note vault/tmp/big: the window process answers on its control channel after about 0.5 s and the index is complete about 2 s after launch (eager open was 2.6 s before the window could appear). Explorer with 50,500 rows: median frame 46 us, 161 shapes. No stage is slow enough to justify a cache, so tasks 10 and 11 close without one. - Cache (phase 3, only if profiling justifies it; it did not): one file in the vault state folder, JSON like the existing vocabulary cache, keyed per note by path, size and modified time; version field; discarded on any mismatch or parse failure; written with
write_private; read size capped like the vocabulary cache. - Pinned folders and the reveal toggle: stored in the existing UI state persisted by
persist_ui.
Dependencies
ignore,grep-searcher,grep-regex,grep-matcher(BurntSushi, pure Rust, owner-approved). Chosen over shelling out torg(not installed on user machines, not on Windows) and over a hand-written scanner (re-implements line, encoding and binary-file handling). Versions pinned at implementation time from crates.io.- No other new crates. No breaking upgrades.
Risks
- Per-frame step starves the UI → a fixed time budget per frame (about 8 ms, measured with
Instant), at least one note per frame, and a repaint request only while work remains. - Partial index shown as truth (backlinks, graph, unresolved links count too low or too high) → every consumer shows an "indexing" state; the unresolved count and
doctorwait for completion. - Features that read the index at startup (tree, tabs restore, calendar, knowledge status) assume it is full → grep every
session.index()caller incrates/ui-egui/src/shelland gate or defer it; this is the largest unknown and gets a checklist task. - Rename or move rewrites links from
backlinks()(crates/engine/src/vault/notes.rs:181,folders.rs:194,205): an unindexed referrer keeps its old link and breaks silently on disk → refuse rename, move and folder move with a clear "still indexing" error until the index is complete (menu items disabled while!finished); this is a hard gate, not a UI hint. - Per-frame resolvers show false "broken" state while partial (
crates/ui-egui/src/shell/app/window/cards.rs:65,views/note_view/blocks/embed_data.rs:66, graph ghost nodes atviews/graph_view.rs:100-109) → while indexing, treat "unresolved" as "unknown": render links neutral, embeds as loading, and build the graph only after completion (or label it partial). - File-change stamp is
index().len()(shell/app/window/frame.rs:160): it changes every step and rebuilds sidebar data and graph cache each frame → switch to a coarse stamp (progress bucket per ~5%) while indexing. - Startup reads of the index (
shell/app.rs:152,window/commands.rs:309, tags panel,@mentions, palette note list) → tree from the listing; the others read partial data and refresh once at completion; add an explicit "index finished" event that clearssidebar_stamp,right_validand the graph cache once. - Learned style votes (lint) and the vocabulary cache take partial input (
engine/src/lint/sync.rs,knowledge/sync.rs:97-105) → lint style verdicts wait for completion; the vocabulary cache is written only inknowledge_end_scanat queue end, never mid-scan. - Stale open buffers vs. later index step for the same note →
index_stepskips paths the watcher or editor already updated (a "seen" set). - Content-search thread leaks or races on fast typing → cancel flag checked per file, one thread at a time, results tagged with a query id and dropped when stale.
- Hostile files (huge, binary, non-UTF-8, giant lines) → skip with a diagnostic; per-file and per-line caps; no
unwrap/index panics (clippy deny stays on). - Tree node cap silently truncating a very large folder → show a truncated row and a count; the cap (64,000) is above the spec's 50,000 test.
- Virtual list scroll jump when collapse state changes → anchor on the first visible row path, restore after rebuild.
- Cache correctness (phase 3) → trust only on exact path+size+mtime match; any doubt re-reads; never an error.
- Performance targets are measured on the owner's Mac in release against the 10,000-note vault; debug numbers are not a gate.
Out of Scope (Defer)
- Honoring
.gitignoreor skippingtarget/node_modulesin non-vault folders (the warning covers the accident; revisit if folders keep being opened by mistake). - Persisting the search index itself, and incremental re-index of edited notes beyond today's watcher path.
- Multi-threaded parsing.
- Fuzzy filter ranking in the tree filter (plain substring first).
- Jump-to-letter strip.
- Keyboard bindings for the explorer commands.
Test Strategy
- Unit:
index_stepbudget and ordering (focused note first, pause stops, resume continues, finishes exactly once);open_lazy+ all steps equals eageropen(same index contents); content search cancel, caps and hostile files;count_filesstops at the limit; tree flat-row cache andshow_rowswindow math; folder note counts; breadcrumb path for a row index; filter keeps parents and restores collapse state on clear. - Integration: the existing vault-open tests run against both open paths; CLI and MCP still use eager open; a generated 5,000-note vault test checks the first-frame time budget with the lazy path (
#[ignore]d perf test inperf/). - Manual: open
/tmp/bigin a release build and time to first window; scroll with the wheel fast and drag the scrollbar; type a search while the status card is counting; pause and resume; open the project folder and confirm the warning appears and Cancel writes nothing; compare screenshots with the artboards via thesnapshotexample at 1440x900 and scale 2, and runget_design_specverification for the new UI modules. - Gate before done:
cargo test -pfor each touched crate, clippy with-D warnings,cargo fmt,cargo xtask ci,cargo xtask i18n.