notescraft/specs/code-blocks-and-formatting/plan.md

Plan: Code blocks, language grammars and batch formatting

Approach

Fenced code becomes a card (uppercase language label, Copy, Wrap) whose text is colored by a new pure crate notescraft-syntax (L0, wraps the owner-approved syntect with the fancy-regex backend; also owns language suggestions and grammar fetch/validate). The same crate serves both text paths: rendered mode (code_card) and source mode (editor layout job). Large blocks highlight on one worker thread behind a content-hash cache, so the UI thread never waits. JSON detect/validate/format/minify is a pure helper in the blocks crate (next to splice) and commits through the existing Session::lint_commit, so every rewrite is one undo step. Batch format and lint is an engine state machine: a stepped dry run collects per-file change proposals without writing, the UI reviews them as diffs, Apply recomputes each accepted file from fresh text and commits it via lint_commit. Nothing here may panic on hostile input.

Architecture

• Render, rendered mode: show_contents fence branch (note_view.rs:~305, code_block at note_view/blocks.rs:29) -> new code_card (header row with label + Copy/Wrap, then body rows) -> SyntaxCache::get(lang, body) returns Option<Arc<Highlighted>> (token ranges per line) without blocking, spawns the worker on a miss -> rows painted with a new Span::CodeTok(text, Color32) (monospace + token color; the existing Span::Colored uses the proportional font, so it cannot be reused). Today code_card (blocks.rs:56) paints the fence lines too; the new card hides them (label replaces the opening fence) while click-to-edit still maps to the same lines.
• Render, source mode: editor::highlight::layout_job (highlight.rs:88) treats a whole fence as one priority-3 Sty::Code assignment (inline::code_fences, inline.rs:18). New editor/src/fence_syntax.rs splits each fence body into sub-assignments Sty::Syntax(TokenKind) (same priority, same code_background) from the same SyntaxCache; a miss paints today's plain Sty::Code and the worker's completion triggers a repaint. Fences over 2,000 lines or 256 KiB stay plain in source mode (documented cap; source mode has no visible-window knowledge).
• Large blocks: the worker parses sequentially with a ParseState checkpoint every 100 lines; rendered mode asks for the visible window plus margin, so a block over 1,000 lines highlights only what is on screen and scroll reuses cached chunks. Blocks up to 200 lines highlight in one job. Cache key = hash(language, body); an edit changes the key, so no stale colors and no invalidation code. Cache is bounded (entries and bytes); eviction by age.
• Inline code chip: egui::TextFormat.background cannot round corners. text_row_rect (painting.rs:~159) paints a rounded rect (radius 6, radius_small()) behind each Span::Mono run using the existing hit_rect (painting.rs:~131), then the galley on top; fmt_span stops setting a background for Mono. Rows with inline math (flow.rs) keep the old flat background (documented gap). Source mode keeps its flat Sty::Code background. A theme test asserts text/code_background contrast >= 4.5:1 for every built-in theme.
• Grammar flow: picker -> grammar.info / grammar.install engine commands -> syntax::fetch -> gates (https only, size cap, parses as grammar, sample highlights without error) -> craft_atomic::atomic_write into the app config dir -> AppSettings.grammars entry. Failure at any gate saves nothing and changes no state; offline gives a data error. Removal deletes the file and the entry; blocks fall back to plain monospace.
• JSON: chip validates on draw (cached by body hash, 1 MiB cap, serde_json depth limit stays on). json.format_block / json.minify_block (path, block index) reuse the block_edit::edit pattern (engine/src/block_edit/mod.rs:79): pure fn over parse_blocks text_range -> Edited -> lint_commit. Invalid JSON returns Err, text unchanged. Formatting is a small re-indenter over the validated text (not serde_json::to_string_pretty): serde_json without preserve_order/arbitrary_precision sorts keys and rewrites numbers such as 1e5, which would change data.
• Batch: batch.start(scope, checks) queues note paths (scope note/folder/vault from vault.list); UI calls batch.step(budget) per frame (same cooperative pattern as the fast-open plan, no threads; progress + cancel; nothing written). Per note it records Proposal { path, content_hash, status, changes[] }. Changes are byte-range replacements against the text read at dry-run time: JSON block format (labeled json, valid = proposed; unlabeled = needs_review) and lint fixes (notescraft_lint::fix, safe ones = proposed, the rest = needs_review). Unreadable, invalid JSON, read-only = Not edited with reason. batch.decide stores Accept/Skip per change; batch.apply re-reads each file, compares the content hash from the dry run (mismatch -> skipped and reported, never overwritten), applies only accepted non-overlapping changes in descending offset order, then one lint_commit per file (atomic via vault.write -> craft_atomic, one undo step per file when the note is open). A write error is reported and the file stays intact.

Files

• Create
  • crates/syntax/ (Cargo.toml, src/lib.rs, load.rs, highlight.rs, cache.rs, scopes.rs, suggest.rs, fetch.rs, validate.rs) — L0, no egui. scopes.rs maps scopes to 8 TokenKinds (keyword, string, number, comment, type, function, property, operator) by prefix rules, unknown -> none.
  • crates/themes/src/syntax.rs — SyntaxColors (8 tokens), per-built-in dark/light values, optional [syntax] table in user themes with variant defaults.
  • crates/editor/src/fence_syntax.rs — source-mode glue above.
  • crates/blocks/src/json.rs — check_json, format_json_block, minify_json_block, looks_like_json, caps.
  • crates/engine/src/grammars.rs, crates/engine/src/batch/{mod.rs,commands.rs,propose.rs}, JSON commands registered from block_edit/mod.rs (111 lines today).
  • crates/ui-egui/src/views/note_view/blocks/code_card.rs (note_view/ is at the 12-file cap, so new note-view code goes in the blocks/ subfolder, 4 files today); crates/ui-egui/src/widgets/lang_picker.rs; crates/ui-egui/src/views/batch_view/{mod.rs,review.rs} (views/ has 11 loose files).
• Modify
  • Cargo.toml — workspace deps syntect (default-features off; parsing, default-syntaxes, dump-load, regex-fancy; exact feature set fixed by the prototype), notescraft-syntax. xtask/src/layers.rs — ("syntax", Class::Layer(0)) in the L0 block of TABLE.
  • crates/editor/Cargo.toml, src/lib.rs, src/colors.rs:~11 (EditorColors.syntax + fallback), src/highlight.rs:28-50,88 (Sty::Syntax); crates/blocks/Cargo.toml (+serde_json, already approved); crates/engine/Cargo.toml.
  • crates/themes/src/{model.rs,builtin.rs,load.rs,lib.rs} — Theme.syntax; crates/ui-egui/src/design/theme.rs and the shell mapping that builds EditorColors.
  • crates/engine/src/prefs/app_settings.rs:~35 — grammars: BTreeMap<String, GrammarInfo> (serde default); crates/engine/src/commands.rs:~25 — chain grammar_specs, batch::specs; crates/engine/src/vault/session.rs — batch state field.
  • crates/ui-egui/src/views/note_view.rs (fence branch ~305; already 581 lines, so only the call site changes), note_view/blocks.rs (code_block/code_card delegate), note_view/inlines.rs:24 (Span::CodeTok), note_view/painting.rs (fmt_span, chip paint), views/mod.rs, src/text/views/* string tables (new code_card.rs table file; folder has 11).
  • ROADMAP.md, docs/ARCHITECTURE.md.
• Delete: none.

Data

• Registry: https://git.marcospaulo.dev.br/masi/notescraft-registry/raw/branch/<default>/index.json (Gitea raw path; branch name to confirm): { version, grammars: [{ id, name, format: "sublime-syntax", file, size, sha256, license, sample }] }. Only this host is accepted for registry entries; redirects are refused.
• Installed: <config dir>/grammars/<id>.sublime-syntax; AppSettings.grammars[id] = { file, size, license, installed_at }.
• Dismissed JSON offers: list "<path>#<body-hash>" in the opaque ui blob (cap 500, inside MAX_UI_BYTES).
• Per-block UI state (wrap, copy confirm): egui temp memory keyed by body hash, never persisted.
• Batch: Proposal { path, content_hash, status: proposed|needs_review|not_edited|clean, reason?, changes: [{ id, kind: json|lint, range, before, after, rule?, decision }] }; summary counts for the four statuses.

Dependencies

• syntect (fancy-regex backend, no onig): owner-approved.
• Downloads: no HTTP client or hash crate exists in the workspace (Cargo.lock has none). Recommended: a bounded curl subprocess, the pattern already in the tree at crates/acp/src/openai/transport.rs:67 (and crates/git/src/process.rs for caps and deadlines): --disable --proto =https --proto-redir =https --max-redirs 0 --max-filesize <cap> --connect-timeout --max-time -sS --fail, stdout read with a hard byte cap in Rust, host allow-list checked in Rust before spawn. User links: https only, no credentials, no IP literals or localhost, size cap; no checksum exists for them, the other gates apply. No new crate for transport.
• sha256 of registry files: needs owner approval for sha2 (pure Rust, RustCrypto, default features off). Until approved, the Grammar-source checksum requirement is not met; install still enforces size + parse + sample.
• Owner decisions (2026-10-18): sha2 approved (pure Rust, default features off); bounded curl subprocess approved for downloads; missing TOML or TypeScript grammars are embedded as licensed .sublime-syntax files once the prototype confirms the gap; source-mode highlighting cap of 2,000 lines or 256 KiB per fence accepted.
• Nothing else new; diffs reuse notescraft-git/diff_view, writes reuse craft-atomic.

Risks

• Spec criterion: built-in TOML and TypeScript. Prototype (2026-10-18, syntect 5, fancy-regex, no onig) confirmed: the bundled pack lacks TOML and TypeScript and has the other 15 checked languages; if confirmed, two licensed .sublime-syntax files are embedded with include_str! so they work offline. Owner decision on which grammars/licenses.
• Grammar format: registry serves .sublime-syntax; syntect reads it natively (no converter). .tmLanguage is rejected with a clear error.
• First-use cost of loading the syntax dump: built lazily on the worker thread after window show; rows paint plain until ready; process-wide cache.
• Hostile input: per-line (10 KiB) and per-block caps in highlight.rs, ParseState errors become plain text, JSON depth limit stays on, all offsets from get/split_at_checked; fuzz-style test with random bytes and 1 MiB lines.
• Worker thread leaks or floods: single thread, bounded queue (latest request wins), result delivered through the cache plus ctx.request_repaint.
• Batch Apply overwrites newer disk state: content-hash re-check immediately before writing; open dirty buffers are edited through the buffer path so the user's unsaved text is never discarded.
• Overlapping accepted changes (JSON block inside a lint fix range): later overlapping change is skipped and listed, never merged silently.
• Reserved folders: batch walks vault.list (hidden entries skipped); .obsidian/ and .notescraft/ are never written.
• Source-mode caps mean a 5,000-line fence is plain in source mode but colored in rendered mode; accepted.

Out of Scope (Defer)

• YAML/TOML formatters, autocomplete, LSP, running code, grammar editor, per-language indent setting (2 spaces fixed; answers the open question).
• Backup before Apply (per-file undo is the guarantee); folder context-menu entry for batch (command palette and Tools menu only).
• Lint rule set beyond apply_all_safe; table alignment fixes.
• Swatches and the link arrow (editor previews plan).

Test Strategy

• Unit: blocks::json (valid/invalid/minify; key order and 1e5 preserved byte-for-byte; depth and size caps; hostile bytes); suggest_language typos/ties; scopes (each of the 8 kinds reachable, unknown ignored); highlight caps and checkpoints (window result equals full-run result); validate gates (http, oversize, non-grammar, bad sample); fence label rewrite is a char-boundary-safe edit; theme contrast >= 4.5:1 for chip and 4.5:1 for each syntax token on code_background in all built-ins.
• Engine integration (headless registry, like the existing block_edit tests): json.* is one undo step; invalid JSON leaves text identical; batch dry run on a temp vault writes nothing (fs compared), counts correct, unreadable file listed under Not edited; Apply skips a file changed after the dry run; cancel writes nothing; fetch against an injectable transport (no network in tests), offline error.
• UI (kittest/snapshot): card shows uppercase label, Copy puts body without fences on the clipboard, Wrap reflows; unknown language renders plain; colored tokens for the 12 built-in languages; source-mode layout job colors a fence; 1,000+ line block draws only the window.
• Manual: snapshot example with JSON/YAML/Rust/diff blocks at 1440x900 and --scale 2 in Mocha and Latte; real registry download, broken link, offline; batch on the owner's vault; release-build scroll of a 5,000-line block; note with 50 blocks opens within the 50 ms budget; cargo xtask ci and cargo xtask i18n.