notescraft/specs/code-blocks-and-formatting/plan.md

# Plan: Code blocks, language grammars and batch formatting

## Approach
Fenced code becomes a card (uppercase language label, Copy, Wrap) whose text is colored by a new pure crate `notescraft-syntax` (L0, wraps the owner-approved `syntect` with the `fancy-regex` backend; also owns language suggestions and grammar fetch/validate). The same crate serves both text paths: rendered mode (`code_card`) and source mode (`editor` layout job). Large blocks highlight on one worker thread behind a content-hash cache, so the UI thread never waits. JSON detect/validate/format/minify is a pure helper in the `blocks` crate (next to `splice`) and commits through the existing `Session::lint_commit`, so every rewrite is one undo step. Batch format and lint is an engine state machine: a stepped dry run collects per-file change proposals without writing, the UI reviews them as diffs, Apply recomputes each accepted file from fresh text and commits it via `lint_commit`. Nothing here may panic on hostile input.

## Architecture
- Render, rendered mode: `show_contents` fence branch (`note_view.rs:~305`, `code_block` at `note_view/blocks.rs:29`) -> new `code_card` (header row with label + Copy/Wrap, then body rows) -> `SyntaxCache::get(lang, body)` returns `Option<Arc<Highlighted>>` (token ranges per line) without blocking, spawns the worker on a miss -> rows painted with a new `Span::CodeTok(text, Color32)` (monospace + token color; the existing `Span::Colored` uses the proportional font, so it cannot be reused). Today `code_card` (`blocks.rs:56`) paints the fence lines too; the new card hides them (label replaces the opening fence) while click-to-edit still maps to the same lines.
- Render, source mode: `editor::highlight::layout_job` (`highlight.rs:88`) treats a whole fence as one priority-3 `Sty::Code` assignment (`inline::code_fences`, `inline.rs:18`). New `editor/src/fence_syntax.rs` splits each fence body into sub-assignments `Sty::Syntax(TokenKind)` (same priority, same `code_background`) from the same `SyntaxCache`; a miss paints today's plain `Sty::Code` and the worker's completion triggers a repaint. Fences over 2,000 lines or 256 KiB stay plain in source mode (documented cap; source mode has no visible-window knowledge).
- Large blocks: the worker parses sequentially with a `ParseState` checkpoint every 100 lines; rendered mode asks for the visible window plus margin, so a block over 1,000 lines highlights only what is on screen and scroll reuses cached chunks. Blocks up to 200 lines highlight in one job. Cache key = hash(language, body); an edit changes the key, so no stale colors and no invalidation code. Cache is bounded (entries and bytes); eviction by age.
- Inline code chip: `egui::TextFormat.background` cannot round corners. `text_row_rect` (`painting.rs:~159`) paints a rounded rect (radius 6, `radius_small()`) behind each `Span::Mono` run using the existing `hit_rect` (`painting.rs:~131`), then the galley on top; `fmt_span` stops setting a background for `Mono`. Rows with inline math (`flow.rs`) keep the old flat background (documented gap). Source mode keeps its flat `Sty::Code` background. A theme test asserts text/`code_background` contrast >= 4.5:1 for every built-in theme.
- Grammar flow: picker -> `grammar.info` / `grammar.install` engine commands -> `syntax::fetch` -> gates (https only, size cap, parses as grammar, sample highlights without error) -> `craft_atomic::atomic_write` into the app config dir -> `AppSettings.grammars` entry. Failure at any gate saves nothing and changes no state; offline gives a data error. Removal deletes the file and the entry; blocks fall back to plain monospace.
- JSON: chip validates on draw (cached by body hash, 1 MiB cap, serde_json depth limit stays on). `json.format_block` / `json.minify_block` (path, block index) reuse the `block_edit::edit` pattern (`engine/src/block_edit/mod.rs:79`): pure fn over `parse_blocks` `text_range` -> `Edited` -> `lint_commit`. Invalid JSON returns `Err`, text unchanged. Formatting is a small re-indenter over the validated text (not `serde_json::to_string_pretty`): serde_json without `preserve_order`/`arbitrary_precision` sorts keys and rewrites numbers such as `1e5`, which would change data.
- Batch: `batch.start(scope, checks)` queues note paths (scope note/folder/vault from `vault.list`); UI calls `batch.step(budget)` per frame (same cooperative pattern as the fast-open plan, no threads; progress + cancel; nothing written). Per note it records `Proposal { path, content_hash, status, changes[] }`. Changes are byte-range replacements against the text read at dry-run time: JSON block format (labeled json, valid = proposed; unlabeled = needs_review) and lint fixes (`notescraft_lint::fix`, safe ones = proposed, the rest = needs_review). Unreadable, invalid JSON, read-only = Not edited with reason. `batch.decide` stores Accept/Skip per change; `batch.apply` re-reads each file, compares the content hash from the dry run (mismatch -> skipped and reported, never overwritten), applies only accepted non-overlapping changes in descending offset order, then one `lint_commit` per file (atomic via `vault.write` -> `craft_atomic`, one undo step per file when the note is open). A write error is reported and the file stays intact.

## Files
- **Create**
  - `crates/syntax/` (`Cargo.toml`, `src/lib.rs`, `load.rs`, `highlight.rs`, `cache.rs`, `scopes.rs`, `suggest.rs`, `fetch.rs`, `validate.rs`) — L0, no egui. `scopes.rs` maps scopes to 8 `TokenKind`s (keyword, string, number, comment, type, function, property, operator) by prefix rules, unknown -> none.
  - `crates/themes/src/syntax.rs` — `SyntaxColors` (8 tokens), per-built-in dark/light values, optional `[syntax]` table in user themes with variant defaults.
  - `crates/editor/src/fence_syntax.rs` — source-mode glue above.
  - `crates/blocks/src/json.rs` — `check_json`, `format_json_block`, `minify_json_block`, `looks_like_json`, caps.
  - `crates/engine/src/grammars.rs`, `crates/engine/src/batch/{mod.rs,commands.rs,propose.rs}`, JSON commands registered from `block_edit/mod.rs` (111 lines today).
  - `crates/ui-egui/src/views/note_view/blocks/code_card.rs` (note_view/ is at the 12-file cap, so new note-view code goes in the `blocks/` subfolder, 4 files today); `crates/ui-egui/src/widgets/lang_picker.rs`; `crates/ui-egui/src/views/batch_view/{mod.rs,review.rs}` (views/ has 11 loose files).
- **Modify**
  - `Cargo.toml` — workspace deps `syntect` (default-features off; parsing, default-syntaxes, dump-load, regex-fancy; exact feature set fixed by the prototype), `notescraft-syntax`. `xtask/src/layers.rs` — `("syntax", Class::Layer(0))` in the L0 block of `TABLE`.
  - `crates/editor/Cargo.toml`, `src/lib.rs`, `src/colors.rs:~11` (`EditorColors.syntax` + fallback), `src/highlight.rs:28-50,88` (`Sty::Syntax`); `crates/blocks/Cargo.toml` (+`serde_json`, already approved); `crates/engine/Cargo.toml`.
  - `crates/themes/src/{model.rs,builtin.rs,load.rs,lib.rs}` — `Theme.syntax`; `crates/ui-egui/src/design/theme.rs` and the shell mapping that builds `EditorColors`.
  - `crates/engine/src/prefs/app_settings.rs:~35` — `grammars: BTreeMap<String, GrammarInfo>` (serde default); `crates/engine/src/commands.rs:~25` — chain `grammar_specs`, `batch::specs`; `crates/engine/src/vault/session.rs` — batch state field.
  - `crates/ui-egui/src/views/note_view.rs` (fence branch ~305; already 581 lines, so only the call site changes), `note_view/blocks.rs` (`code_block`/`code_card` delegate), `note_view/inlines.rs:24` (`Span::CodeTok`), `note_view/painting.rs` (`fmt_span`, chip paint), `views/mod.rs`, `src/text/views/*` string tables (new `code_card.rs` table file; folder has 11).
  - `ROADMAP.md`, `docs/ARCHITECTURE.md`.
- **Delete**: none.

## Data
- Registry: `https://git.marcospaulo.dev.br/masi/notescraft-registry/raw/branch/<default>/index.json` (Gitea raw path; branch name to confirm): `{ version, grammars: [{ id, name, format: "sublime-syntax", file, size, sha256, license, sample }] }`. Only this host is accepted for registry entries; redirects are refused.
- Installed: `<config dir>/grammars/<id>.sublime-syntax`; `AppSettings.grammars[id] = { file, size, license, installed_at }`.
- Dismissed JSON offers: list `"<path>#<body-hash>"` in the opaque `ui` blob (cap 500, inside `MAX_UI_BYTES`).
- Per-block UI state (wrap, copy confirm): egui temp memory keyed by body hash, never persisted.
- Batch: `Proposal { path, content_hash, status: proposed|needs_review|not_edited|clean, reason?, changes: [{ id, kind: json|lint, range, before, after, rule?, decision }] }`; summary counts for the four statuses.

## Dependencies
- `syntect` (fancy-regex backend, no onig): owner-approved.
- Downloads: no HTTP client or hash crate exists in the workspace (`Cargo.lock` has none). Recommended: a bounded `curl` subprocess, the pattern already in the tree at `crates/acp/src/openai/transport.rs:67` (and `crates/git/src/process.rs` for caps and deadlines): `--disable --proto =https --proto-redir =https --max-redirs 0 --max-filesize <cap> --connect-timeout --max-time -sS --fail`, stdout read with a hard byte cap in Rust, host allow-list checked in Rust before spawn. User links: https only, no credentials, no IP literals or localhost, size cap; no checksum exists for them, the other gates apply. No new crate for transport.
- sha256 of registry files: needs owner approval for `sha2` (pure Rust, RustCrypto, default features off). Until approved, the Grammar-source checksum requirement is not met; install still enforces size + parse + sample.
- Owner decisions (2026-10-18): `sha2` approved (pure Rust, default features off); bounded `curl` subprocess approved for downloads; missing TOML or TypeScript grammars are embedded as licensed `.sublime-syntax` files once the prototype confirms the gap; source-mode highlighting cap of 2,000 lines or 256 KiB per fence accepted.
- Nothing else new; diffs reuse `notescraft-git`/`diff_view`, writes reuse `craft-atomic`.

## Risks
- Spec criterion: built-in TOML and TypeScript. Prototype (2026-10-18, syntect 5, fancy-regex, no onig) confirmed: the bundled pack lacks TOML and TypeScript and has the other 15 checked languages; if confirmed, two licensed `.sublime-syntax` files are embedded with `include_str!` so they work offline. Owner decision on which grammars/licenses.
- Grammar format: registry serves `.sublime-syntax`; syntect reads it natively (no converter). `.tmLanguage` is rejected with a clear error.
- First-use cost of loading the syntax dump: built lazily on the worker thread after window show; rows paint plain until ready; process-wide cache.
- Hostile input: per-line (10 KiB) and per-block caps in `highlight.rs`, `ParseState` errors become plain text, JSON depth limit stays on, all offsets from `get`/`split_at_checked`; fuzz-style test with random bytes and 1 MiB lines.
- Worker thread leaks or floods: single thread, bounded queue (latest request wins), result delivered through the cache plus `ctx.request_repaint`.
- Batch Apply overwrites newer disk state: content-hash re-check immediately before writing; open dirty buffers are edited through the buffer path so the user's unsaved text is never discarded.
- Overlapping accepted changes (JSON block inside a lint fix range): later overlapping change is skipped and listed, never merged silently.
- Reserved folders: batch walks `vault.list` (hidden entries skipped); `.obsidian/` and `.notescraft/` are never written.
- Source-mode caps mean a 5,000-line fence is plain in source mode but colored in rendered mode; accepted.

## Out of Scope (Defer)
- YAML/TOML formatters, autocomplete, LSP, running code, grammar editor, per-language indent setting (2 spaces fixed; answers the open question).
- Backup before Apply (per-file undo is the guarantee); folder context-menu entry for batch (command palette and Tools menu only).
- Lint rule set beyond `apply_all_safe`; table alignment fixes.
- Swatches and the link arrow (editor previews plan).

## Test Strategy
- Unit: `blocks::json` (valid/invalid/minify; key order and `1e5` preserved byte-for-byte; depth and size caps; hostile bytes); `suggest_language` typos/ties; `scopes` (each of the 8 kinds reachable, unknown ignored); `highlight` caps and checkpoints (window result equals full-run result); `validate` gates (http, oversize, non-grammar, bad sample); fence label rewrite is a char-boundary-safe edit; theme contrast >= 4.5:1 for chip and 4.5:1 for each syntax token on `code_background` in all built-ins.
- Engine integration (headless registry, like the existing block_edit tests): `json.*` is one undo step; invalid JSON leaves text identical; batch dry run on a temp vault writes nothing (fs compared), counts correct, unreadable file listed under Not edited; Apply skips a file changed after the dry run; cancel writes nothing; `fetch` against an injectable transport (no network in tests), offline error.
- UI (kittest/snapshot): card shows uppercase label, Copy puts body without fences on the clipboard, Wrap reflows; unknown language renders plain; colored tokens for the 12 built-in languages; source-mode layout job colors a fence; 1,000+ line block draws only the window.
- Manual: `snapshot` example with JSON/YAML/Rust/diff blocks at 1440x900 and `--scale 2` in Mocha and Latte; real registry download, broken link, offline; batch on the owner's vault; release-build scroll of a 5,000-line block; note with 50 blocks opens within the 50 ms budget; `cargo xtask ci` and `cargo xtask i18n`.