Files
Marcos Paulo 5e8020aa38
ci / ci (macos-latest) (push) Has been cancelled
ci / ci (ubuntu-latest) (push) Has been cancelled
ci / ci (windows-latest) (push) Has been cancelled
feat: Obsidian basics, editor previews, graph rework, query blocks, stacked tabs, pop-out windows
Editor previews (colour swatches, arrows, embeds), the 18 obsidian-basics tasks (find and replace
with regex, back/forward, auto-pair, footnotes, folding, search operators, split view, PDF/audio/
video embeds, spelling, drag and drop, pinned tabs, settings) and round 2: ```query blocks, stacked
tabs, pop-out windows, tasks ticked inside embeds, text around an embed, embeds that follow edits,
block menu shortcut. New crates: notescraft-regex (linear-time engine) and notescraft-syntax.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-19 11:21:19 -03:00

12 KiB

Plan: Fast open and large-vault explorer

Approach

Open in three stages: list names (instant), show the window, then read notes in small time-boxed steps driven by the UI frame loop. A cooperative step loop keeps the Session single-owner (no threads, locks or shared index) and reuses knowledge_begin_scan / knowledge_end_scan, which already bracket a scan. Content search before the index is ready is a streaming grep over disk on its own thread (ripgrep library crates). The index cache is deferred until a release-build profile shows which stage dominates, so we cache the real cost and nothing else. The explorer becomes a virtualized list over a flat row model.

Measured today (release, 10,000 notes, cold): info takes 2.6 s. The window cannot appear during that time because Session::open runs before eframe::run_native (apps/notescraft/src/main.rs:180). The debug build and a folder with many non-note files (for example a target/ folder) are far worse.

Architecture

  • Data flow, open: main → Vault::open + Vault::list (names, sizes, mtimes) → Session::open_lazy (empty indexes, scan queue) → window → each frame NotesApp::ui calls Session::index_step(budget) → note read → Index::update_note, SearchIndex::update, knowledge and lint feeds (same calls index_note makes today) → at queue end knowledge_end_scan.
  • Data flow, tree: built from the listing (file names), not from the index, so it exists before any note is parsed.
  • Data flow, search while indexing: SearchPanel::refresh → Session::search → if not done, ContentSearch thread (walk + grep-searcher over the vault) → hits merged into the panel on later frames; a new query cancels the old thread through a shared flag.
  • New modules:
    • indexing step loop and progress state in the engine (queue, counters, pause flag, IndexProgress)
    • content search over disk in the engine (thread, cancel flag, hit type reusing the existing search Hit)
    • large-folder pre-count (bounded walk that stops past the threshold) in the vault crate
    • index status card and indexing chip in the UI
    • large-folder dialog in the UI
  • Modified modules: Vault::list also returns size and modified time; Session gets open_lazy, index_step, index_progress; NotesApp::new takes the tree from the listing; tree.rs virtualizes rows and carries folder note counts; sidebar gets filter box, toolbar actions, pinned section, sticky breadcrumb; main shows the dialog before opening.
  • Commands (all UI commands, no default key binding): explorer.collapse_all, explorer.reveal_active, explorer.focus_filter, index.pause, index.resume. Session::open (eager) stays for the CLI, MCP and tests.

Files

Paths checked; line numbers from the code as read.

  • Create
    • crates/engine/src/vault/indexing.rs — scan queue, index_step, progress, pause; newest and focused notes first.
    • crates/engine/src/vault/content_search.rs — threaded grep over disk with cancel; hostile-input bounds (line length, file size, hit cap).
    • crates/ui-egui/src/panels/index_status.rs — status card and header chip.
    • crates/ui-egui/src/widgets/large_folder_dialog.rs — warning dialog.
  • Modify
    • crates/vault/src/lib.rs:72,332 — VaultEntry gains size and modified; add a bounded count_files(limit) for the pre-count; hidden entries and symlinks stay skipped as list does.
    • crates/engine/src/vault/session.rs:73-100,320-355 — add open_lazy; with_vault split so eager open = lazy open + run all steps; scan becomes the queue seed. is_markdown unchanged.
    • crates/engine/src/vault/mod.rs — declare the two new modules, register index.* commands.
    • crates/engine/src/vault/search.rs:38-41 — search stays; add a thin search_progress that says whether results are partial.
    • crates/engine/src/knowledge/sync.rs:96-119 — no behavior change; begin/end called from the lazy path (end runs when the queue empties).
    • apps/notescraft/src/main.rs:163-184 — count files first and show the dialog; use open_lazy.
    • crates/ui-egui/src/shell/app.rs:147-153 — tree from the listing; call index_step in ui with a frame budget and request_repaint while not done.
    • crates/ui-egui/src/shell/app/window/commands.rs:308-318 — refresh_tree reads the listing, not the index; tree rebuild no longer needs node_ids (already O(n) per refresh, so call it only on structure change events).
    • crates/ui-egui/src/panels/tree.rs:41-50,228-272 — TreeNode.notes count; visible produces a flat row list cached per (model version, collapse set); show_with_bookmarks uses ScrollArea::show_rows with overscan; scroll-to-row API for reveal; MAX_NODES hit becomes a visible "truncated" row, not a silent cut.
    • crates/ui-egui/src/panels/sidebar.rs:46-160 — toolbar (filter, collapse all, reveal toggle, pin), pinned section, sticky breadcrumb, status card slot.
    • crates/ui-egui/src/panels/search_panel.rs:48-56 — show partial results and a "still reading" line; stream content-search hits.
    • xtask/src/layers.rs and root Cargo.toml — register new dependencies (ignore, grep-searcher, grep-regex, grep-matcher) for the engine crate only.
    • i18n string tables — new keys; cargo xtask i18n must stay green.
    • ROADMAP.md, docs/ARCHITECTURE.md — status and API (per AGENTS.md).
  • Delete: none.

Data

  • No schema change in phase 1.
  • VaultEntry { path, is_dir, size, modified } (additive).
  • IndexProgress { total, done, paused, finished }.
  • Measured (2026-10-18, owner's Mac, release build): 5,000 notes: list 8.6 ms, open_lazy 8.8 ms, read all 63 ms, link index 22 ms, search index 59 ms, knowledge passages 38 ms, vocabulary 90 ms (already cached), whole index 435 ms. 10,000-note vault /tmp/big: the window process answers on its control channel after about 0.5 s and the index is complete about 2 s after launch (eager open was 2.6 s before the window could appear). Explorer with 50,500 rows: median frame 46 us, 161 shapes. No stage is slow enough to justify a cache, so tasks 10 and 11 close without one.
  • Cache (phase 3, only if profiling justifies it; it did not): one file in the vault state folder, JSON like the existing vocabulary cache, keyed per note by path, size and modified time; version field; discarded on any mismatch or parse failure; written with write_private; read size capped like the vocabulary cache.
  • Pinned folders and the reveal toggle: stored in the existing UI state persisted by persist_ui.

Dependencies

  • ignore, grep-searcher, grep-regex, grep-matcher (BurntSushi, pure Rust, owner-approved). Chosen over shelling out to rg (not installed on user machines, not on Windows) and over a hand-written scanner (re-implements line, encoding and binary-file handling). Versions pinned at implementation time from crates.io.
  • No other new crates. No breaking upgrades.

Risks

  • Per-frame step starves the UI → a fixed time budget per frame (about 8 ms, measured with Instant), at least one note per frame, and a repaint request only while work remains.
  • Partial index shown as truth (backlinks, graph, unresolved links count too low or too high) → every consumer shows an "indexing" state; the unresolved count and doctor wait for completion.
  • Features that read the index at startup (tree, tabs restore, calendar, knowledge status) assume it is full → grep every session.index() caller in crates/ui-egui/src/shell and gate or defer it; this is the largest unknown and gets a checklist task.
  • Rename or move rewrites links from backlinks() (crates/engine/src/vault/notes.rs:181, folders.rs:194,205): an unindexed referrer keeps its old link and breaks silently on disk → refuse rename, move and folder move with a clear "still indexing" error until the index is complete (menu items disabled while !finished); this is a hard gate, not a UI hint.
  • Per-frame resolvers show false "broken" state while partial (crates/ui-egui/src/shell/app/window/cards.rs:65, views/note_view/blocks/embed_data.rs:66, graph ghost nodes at views/graph_view.rs:100-109) → while indexing, treat "unresolved" as "unknown": render links neutral, embeds as loading, and build the graph only after completion (or label it partial).
  • File-change stamp is index().len() (shell/app/window/frame.rs:160): it changes every step and rebuilds sidebar data and graph cache each frame → switch to a coarse stamp (progress bucket per ~5%) while indexing.
  • Startup reads of the index (shell/app.rs:152, window/commands.rs:309, tags panel, @ mentions, palette note list) → tree from the listing; the others read partial data and refresh once at completion; add an explicit "index finished" event that clears sidebar_stamp, right_valid and the graph cache once.
  • Learned style votes (lint) and the vocabulary cache take partial input (engine/src/lint/sync.rs, knowledge/sync.rs:97-105) → lint style verdicts wait for completion; the vocabulary cache is written only in knowledge_end_scan at queue end, never mid-scan.
  • Stale open buffers vs. later index step for the same note → index_step skips paths the watcher or editor already updated (a "seen" set).
  • Content-search thread leaks or races on fast typing → cancel flag checked per file, one thread at a time, results tagged with a query id and dropped when stale.
  • Hostile files (huge, binary, non-UTF-8, giant lines) → skip with a diagnostic; per-file and per-line caps; no unwrap/index panics (clippy deny stays on).
  • Tree node cap silently truncating a very large folder → show a truncated row and a count; the cap (64,000) is above the spec's 50,000 test.
  • Virtual list scroll jump when collapse state changes → anchor on the first visible row path, restore after rebuild.
  • Cache correctness (phase 3) → trust only on exact path+size+mtime match; any doubt re-reads; never an error.
  • Performance targets are measured on the owner's Mac in release against the 10,000-note vault; debug numbers are not a gate.

Out of Scope (Defer)

  • Honoring .gitignore or skipping target/node_modules in non-vault folders (the warning covers the accident; revisit if folders keep being opened by mistake).
  • Persisting the search index itself, and incremental re-index of edited notes beyond today's watcher path.
  • Multi-threaded parsing.
  • Fuzzy filter ranking in the tree filter (plain substring first).
  • Jump-to-letter strip.
  • Keyboard bindings for the explorer commands.

Test Strategy

  • Unit: index_step budget and ordering (focused note first, pause stops, resume continues, finishes exactly once); open_lazy + all steps equals eager open (same index contents); content search cancel, caps and hostile files; count_files stops at the limit; tree flat-row cache and show_rows window math; folder note counts; breadcrumb path for a row index; filter keeps parents and restores collapse state on clear.
  • Integration: the existing vault-open tests run against both open paths; CLI and MCP still use eager open; a generated 5,000-note vault test checks the first-frame time budget with the lazy path (#[ignore]d perf test in perf/).
  • Manual: open /tmp/big in a release build and time to first window; scroll with the wheel fast and drag the scrollbar; type a search while the status card is counting; pause and resume; open the project folder and confirm the warning appears and Cancel writes nothing; compare screenshots with the artboards via the snapshot example at 1440x900 and scale 2, and run get_design_spec verification for the new UI modules.
  • Gate before done: cargo test -p for each touched crate, clippy with -D warnings, cargo fmt, cargo xtask ci, cargo xtask i18n.