notescraft/specs/file-viewers/spec.md

File viewers: every file in the Explorer, an image lightbox with annotations, media, code and PDF

Raised by the owner on 2026-10-20: the Explorer shows only .md, canvas and draw files. They want to see images, maybe videos and other files, even code; and for images something to draw on them, saved as metadata, with a lightbox that can zoom, pan and more.

Problem

shell/app/vault/indexing.rs builds the Explorer tree with TabKind::of(path) != TabKind::Text, so any file that is not a note, canvas, whiteboard or diagram is invisible, although the vault holds them (attachments, screenshots, PDFs, scripts). Images only appear inside notes, with no way to open one on its own, zoom into it, or mark it up. Obsidian shows all supported files and has a setting to detect every extension.

Goal

Every file in the vault shows in the Explorer with the right icon and opens in the right viewer: images in a lightbox (zoom, pan, fit, rotate, neighbours) with non-destructive annotations, video and audio with a player, code and text with syntax highlighting, PDFs as pages, and anything else as an info card with "Open with the system".

Non-Goals

• Editing image pixels (crop, filters): annotations are vector overlays only.
• A full video editor, a hex editor, Office document rendering.
• Changing the vault format: the original file is never modified by annotating.

Design

1. All files listed. The tree includes every non-hidden file. A setting files.show_all_types (default on) and an exclude list by extension (files.hidden_extensions, default empty) decide; folders stay as they are; hidden entries (.obsidian, .git, .notescraft) stay hidden. Large folders keep the existing virtualization and the 12-loose-file guard is unaffected (that is a source rule, not a vault rule).
2. Tab kinds. TabKind gains Image (png, jpg, jpeg, gif, webp, bmp, svg), Video (mp4, webm, mov, mkv), Audio (mp3, wav, ogg, m4a, flac), Pdf, Code (known source extensions: the syntax crate's list) and Other; the old Text stays for plain text files (txt, log, csv, json, toml, yaml, ...). Each kind has an icon (the Explorer and the tab strip) and a view.
3. Image lightbox (full tab, and a modal when an embed is clicked in a note): wheel and pinch zoom around the cursor (10% to 3200%), drag to pan, double click toggles fit and 100%, keys + - 0 1 F and arrows, rotate left and right, fit width and fit height, a checkerboard under transparency, info line (dimensions, size, format), Copy image, Reveal, and Left/Right to step through the images of the same folder ([/] too). SVG renders through the existing SVG loader. Decoding is capped (longest side 8192 px, 64 MiB decoded) and happens off the UI thread.
4. Annotations (metadata, non-destructive). A "Draw" mode over the image: pen, highlighter, line, arrow, rectangle, ellipse, text label, eraser (whole stroke), 8 theme colours, 3 widths, undo and redo, clear. Annotations live in a sidecar <file>.annotations.json next to the image (photo.png -> photo.png.annotations.json), format { "version": 1, "image": {"hash","width","height"}, "items": [ ... ] } with coordinates in image pixels so they survive resizing; items are strokes (points), shapes and text. The sidecar is an ordinary vault file: it syncs, shows in history, and is hidden from the Explorer by default when its image exists (a toggle shows it). The note view draws annotations over an embedded image when a sidecar exists. "Export flattened copy" writes a new PNG (never replaces the original).
5. Video and audio: a tab with play/pause, seek bar, volume, time, speed (0.5x to 2x) using the existing ffmpeg pipeline (shell/app/video.rs, player.rs); without ffmpeg the card says so and offers Open with the system. Audio shows a waveform-free bar.
6. Code and text: opens in the Source editor (existing) with syntax highlighting by extension through the syntax crate, line numbers, wrap toggle, read-only for files over 2 MiB; images of code are not needed. Other (binary) shows name, size, type, modified time, a hash on demand, and Open with the system and Reveal.
7. PDF: pages rendered through the existing PDF preview path (engine/src/vault/pdf_preview.rs), vertical scroll, page counter, zoom; without the renderer, the info card.
8. Phone: pinch zoom, two-finger pan, double tap fit, bottom tool bar for the annotation tools (44 pt), swipe left/right between images.
9. Safety: file reads capped per kind, paths validated as everywhere, annotation JSON validated on load (item count, point count, coordinates finite and within 4x the image size, text length), unknown keys kept on save, a corrupt sidecar is renamed to .bad and never silently overwritten.

Acceptance Criteria

• Given a vault with a.md, shot.png, clip.mp4, tool.rs, doc.pdf and data.bin, then the Explorer lists all six with distinct icons and each opens in its own viewer.
• Given an image of 4000 x 3000, when the user wheel-zooms at a point, then that image point stays under the cursor, zoom is clamped to 10%..3200%, and decode never blocks a frame beyond 16 ms.
• Given an image, when the user draws two strokes and one arrow and closes the tab, then shot.png.annotations.json exists, shot.png is byte-identical, and reopening shows all three items.
• Given an annotated image embedded in a note, then the note view shows the annotations over it.
• Given two devices that annotate the same image, then the sidecar syncs and a clash becomes a conflict copy like any file.
• Given a corrupt sidecar, then the viewer opens the image, shows a notice, keeps the bad file as .bad and starts a new sidecar only when the user draws.
• Given files.hidden_extensions = ["tmp"], then .tmp files disappear from the Explorer.
• Given a 390 pt window, then the lightbox supports pinch, pan, double tap and the 44 pt tool bar.

Constraints

• Rust, egui; no new dependency without approval: images decode through the egui image loaders already in use; no image crate use beyond what egui_extras provides; PNG export for flattening may need an encoder: use egui's ColorImage plus the png writing already pulled in by egui_extras' image feature if exposed, otherwise defer the export (Open Question 1).
• Performance: tree with 50,000 files stays smooth (existing virtualization); thumbnails are not generated in v1.

Open Questions

1. Flattened export needs a PNG encoder: approve the png crate (small, pure Rust), or defer the export? Recommended: approve png.
2. Show the sidecar in the Explorer by default? Recommended: hidden when its image exists, with a toggle.
3. Video decode relies on an installed ffmpeg: acceptable on desktop; on phones a native player is needed (later).

Tasks

1. Pure crate crates/annotate (L1): the annotation model, JSON load and save with validation, hit-testing for the eraser, bounds, undo and redo stack, coordinate transforms, tests and property tests. Done 2026-10-20 by GLM 5.3 Flash via Pi (19 tests, clippy clean, verified). Brief scripts/mm/annotate-core.md.
2. Explorer shows all files; TabKind kinds, icons, sidecar hiding, a placeholder card for kinds without a viewer, Settings > Files toggles show_all_types and show_annotation_files. Engine settings by GLM 5.3 Flash via Pi (brief scripts/mm/explorer-all-files.md); UI half by Claude 2026-10-20. Open: a Settings row for hidden_extensions (list editor). 2b. [x] Engine commands file.info, file.hash, file.neighbours, image.annotations.load/save and header-only image sizing (crates/engine/src/viewers/). Done 2026-10-20 by MiniMax M3.1 Flash via the harness; verified. Brief scripts/mm/viewer-commands.md.
3. Image lightbox view (views/image_view/: zoom, pan, fit, rotate, neighbours, info; 5 integration tests). Done 2026-10-20 by MiniMax M3.1 Flash via OpenCode Zen (free route); verified from a clean checkout. Open: copy and reveal buttons, phone pinch gestures check on a device.
4. (done 2026-10-20, MiniMax via OpenCode Zen free route; views/image_view/annotate/; deviation: the view reads and writes the sidecar through Vault on a worker thread, not through the engine commands, which the view cannot reach; Clear is a click-twice button) Annotation tools over the lightbox using the core crate; sidecar read/write through engine commands image.annotations.load/save; note-embed overlay.
5. [~] Info card for audio/video/pdf/other and code_view highlighter done (GLM); Video tab and PDF tab done (MiniMax, views/media_view/; no elapsed time, no page total). Open: audio playback (needs a crate decision), wiring code_view as a read-only tab, thumbnails.
6. Flattened export (after the png decision), annotation sync conflict handling, thumbnails (later).

Designs

PaperCraft project NotesCraft, page "File viewers"; exports (dark and -light) in design/files/, rated in design/files/RATING.md (overall 8.2 of 10, with the fixes to apply during the build): 01 Explorer all files, 02 Image lightbox, 03 Image annotate, 04 Mobile lightbox, 05 Mobile annotate, 06 Video viewer, 07 Code viewer, 08 PDF viewer, 09 Other file card, 10 Settings Files.

Build log

• 2026-10-20: briefs written: image-lightbox (MiniMax harness), info-and-code-viewers and hidden-extensions-row (GLM via Pi) launched in parallel; image-annotate-ui queued behind the lightbox; media (video/audio) and PDF viewers need a spec pass on decoding first (ffmpeg on desktop, crates/pdf for PDF pages).
• Findings for the media and PDF viewers: note embeds already play video through ffmpeg when installed (crates/ui-egui/src/shell/app/video.rs, frames to raw RGBA) and render PDF pages (pdf_page in shell/attachment_drop/images.rs). The tab viewers reuse those paths instead of adding decoders: Video tab = that player at full size with a seek bar; Audio tab = ffmpeg audio decode is out of scope, so show the info card plus "Open with the system" until a playback crate is approved; PDF tab = page raster from the same code with next/previous page and zoom. ffmpeg and ffprobe exist on this machine (/opt/homebrew/bin).
• 2026-10-20 (evening): image-annotate (draw tools + sidecar) and publish-commands (engine) running on the free MiniMax route; agy still out of quota (resets in about 96 h).