Merge branch 'main' into add_nix_build

This commit is contained in:
𝐀𝖓𝖆𝖗𝖎𝖔𝐍
2026-10-07 19:32:10 +02:00
committed by GitHub
114 changed files with 14021 additions and 381 deletions
+3
View File
@@ -11,3 +11,6 @@ plan/
# nix build / nix develop
/result
/result-*
# model weights are never committed (SAM 3: docs/ai-masks.md)
*.safetensors
*.safetensors.part
Generated
+1453 -8
View File
File diff suppressed because it is too large Load Diff
+5
View File
@@ -29,6 +29,7 @@ lightcraft-merge = { path = "crates/merge" }
lightcraft-engine = { path = "crates/engine" }
lightcraft-ui-egui = { path = "crates/ui-egui" }
lightcraft-mcp = { path = "crates/mcp" }
lightcraft-segment = { path = "crates/segment" }
# foundation
thiserror = "2"
@@ -62,6 +63,10 @@ jxl-oxide = { version = "0.12", default-features = false }
ravif = { version = "0.13", default-features = false, features = ["threading"] } # no "asm": it needs nasm (non-Rust)
# gpu (crates/gpu; native backends only, no GL: everything stays pure Rust)
# SAM 3 inference (lightcraft-segment). 0.9.2: later releases pull `tokenizers` with the
# Oniguruma C library into candle-core, and the product is pure Rust.
candle-core = { version = "=0.9.2", default-features = false }
candle-nn = { version = "=0.9.2", default-features = false }
wgpu = { version = "30", default-features = false, features = ["std", "parking_lot", "metal", "vulkan", "dx12", "wgsl"] }
pollster = "1"
+12
View File
@@ -15,6 +15,18 @@ list (author, source, license) is in assets/ATTRIBUTION.md.
Inter (assets/fonts/Inter-*.ttf) — SIL Open Font License 1.1. See assets/fonts/OFL-Inter.txt.
Showcase photographs by Ansel Adams, Dorothea Lange and NASA (docs/images/) — public domain. See assets/ATTRIBUTION.md.
crates/segment (lightcraft-segment) ports the SAM 3 model code of Hugging Face Transformers
(https://github.com/huggingface/transformers, src/transformers/models/sam3 and sam3_tracker),
Copyright The HuggingFace Team and Meta Platforms, Inc., licensed under the Apache License 2.0.
The port was modified by the LightCraft contributors (2026): translated from Python/PyTorch to
Rust on candle and restructured; each ported file says so. Because of this, crates/segment is
licensed under the Apache License 2.0 only (LICENSE-APACHE), not under the MIT option.
The SAM 3 weights are not part of LightCraft and are never distributed with it. A user downloads
the facebook/sam3 checkpoint only when they choose to (in the app, or by hand), under Meta's SAM
License (https://github.com/facebookresearch/sam3/blob/main/LICENSE), a separate, non-OSI
licence. See docs/ai-masks.md.
Builds made with craft-fonts (the optional CRAFT_FONTS_DIR build input; all official releases)
also embed the fonts listed in https://github.com/storytold/craft-fonts/blob/main/ATTRIBUTION.md
(the Japanese fonts BIZ UDPGothic, BIZ UDMincho and Shippori Mincho; the web build embeds only BIZ
+6 -3
View File
@@ -334,16 +334,19 @@ cargo run --release -p lightcraft-cli -- render photo.jpg -o out.jpg --set light
cargo xtask ci # fmt, clippy, tests, layering, wasm checks
```
**Japanese text** needs the shared font repo, an optional build input (official releases always include it):
**Chinese and Japanese text** need the shared font repo, an optional build input (official releases always include it):
```sh
git clone https://github.com/storytold/craft-fonts ../craft-fonts
CRAFT_FONTS_DIR=../craft-fonts cargo run --release -p lightcraft
```
Without it LightCraft builds and runs the same, but Japanese text has no glyphs. Fonts are never committed to this
Without it LightCraft builds and runs the same, but Chinese and Japanese text have no glyphs. Fonts are never committed to this
repo; see [craftrules `standards/fonts.md`](https://github.com/storytold/craftrules/blob/main/standards/fonts.md).
**Interface language:** **Edit → Language** (English, 简体中文, 繁體中文(台灣), 日本語) or **Settings → General →
Language**; the choice applies immediately and persists. See [docs/localization.md](docs/localization.md).
The web build needs the `wasm32-unknown-unknown` target and the matching `wasm-bindgen` CLI
(`cargo xtask web` prints the exact install command); see [docs/web.md](docs/web.md).
@@ -457,7 +460,7 @@ Showcase photographs are public-domain works, used via Wikimedia Commons: Ansel
(1942, U.S. National Archives); Dorothea Lange, *Migrant Mother* (1936, Library of Congress); Bill Anders / NASA,
*Earthrise* (1968); NASA, *The Blue Marble* (1972). The demo library is procedurally generated by LightCraft. UI font:
Inter (SIL OFL 1.1). Builds made with [craft-fonts](https://github.com/storytold/craft-fonts) (all official releases)
also embed its Japanese fonts (BIZ UDPGothic, BIZ UDMincho, Shippori Mincho; SIL OFL 1.1), listed in its
also embed its Chinese and Japanese fonts (Noto Sans CJK SC, BIZ UDPGothic, BIZ UDMincho, Shippori Mincho; SIL OFL 1.1), listed in its
[ATTRIBUTION.md](https://github.com/storytold/craft-fonts/blob/main/ATTRIBUTION.md). All icons are original.
The ArtCraft name, wordmark and logos in [`docs/brand/`](docs/brand/) are trademarks of the
+13 -9
View File
@@ -29,7 +29,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
| **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |
| **Platform & robustness** | ~70% | macOS native; Windows/Linux builds; web via WASM; no-panic lints workspace-wide, `unsafe` confined to `crates/sysmem`; failed saves are reported; GPU errors fall back to CPU | Windows installer UI unverified on Windows (PR #79); GPU path proven only on Apple + user reports; Japanese/English UI (see docs/localization-ja.md); remaining technical errors and other languages; accessibility partial; headless UI tests time out under machine load |
| **Platform & robustness** | ~70% | macOS native; Windows/Linux builds; web via WASM; no-panic lints workspace-wide, `unsafe` confined to `crates/sysmem`; failed saves are reported; GPU errors fall back to CPU | Windows installer UI unverified on Windows (PR #79); GPU path proven only on Apple + user reports; English, Simplified/Traditional Chinese and Japanese UI (see docs/localization.md); remaining technical errors; Traditional Chinese uses the Simplified Chinese font until craft-fonts has a TC face; accessibility partial; headless UI tests time out under machine load |
### By kind of user
@@ -39,7 +39,7 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW and NEF preview estimates are only a starting point) |
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only (CR3: the full-size JPEG with full metadata) |
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
| Relies on AI (masks, denoise) | ~25% | No segmentation or AI denoise models |
| Relies on AI (masks, denoise) | ~25% | Object / Describe masks via optional SAM 3 (download not yet hosted); no AI denoise |
## Where we're going
@@ -57,7 +57,9 @@ Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/par
(references stay in the local `plan/`), then tune against the numbers.
5. **Lens profiles of our own** (LR-EDIT-OPTICS-PROFILE, P1).
6. **AI model strategy** (maintainer decision): which permissively licensed models (or our own training) for
segmentation masks and denoise; then pure-Rust inference. Unblocks M12 and Enhance.
segmentation masks and denoise; then pure-Rust inference. Unblocks M12 and Enhance. Object / Describe masks now
run Meta's SAM 3 in pure Rust (optional, user-consented download under the non-OSI SAM License; CDN mirrors still
to be set up, docs/ai-masks.md); Subject / Sky / People and Enhance are still open.
7. **Then:** HDR (Q), the Classic output modules (Print first, then Map view, Book, Slideshow), video (R), localisation
and accessibility.
@@ -87,7 +89,7 @@ hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
| M15 | Classic modules | Map, Book, Slideshow, Print, Web; smart collections, stacks, virtual copies, publish services, tethering | 25–40 | 🚧 (smart albums, stacks, virtual copies, compare/survey ✅; Map/Book/Slideshow/Print/Web ⬜) |
| M16 | 1.0 polish | preferences, shortcut editor, accessibility, localization, packaging (dmg/msi/AppImage/web), hardening | 10–20 | 🚧 (settings, keyboard shortcuts sheet, packaging basics ✅; Japanese/English localisation 🟡; accessibility and other locales ⬜) |
| M16 | 1.0 polish | preferences, shortcut editor, accessibility, localization, packaging (dmg/msi/AppImage/web), hardening | 10–20 | 🚧 (settings, keyboard shortcuts sheet, packaging basics ✅; English/Chinese (Simplified, Traditional)/Japanese localisation 🟡; accessibility and further locales ⬜) |
## Parity estimate (feature count updated 2026-10-05; effort estimate from 2026-10-02)
@@ -192,12 +194,14 @@ own chart-based calibration (M11.4). Adobe matrices are never used.
export hardening, clippy 1.99. Added the honest *Where we stand* assessment and *Where we're going* priorities; added
tracker rows for camera colour, camera coverage and render fidelity.
## Japanese interface and text watermarks
## Chinese and Japanese interfaces, and text watermarks
English/Japanese interface language is persisted in UI state. Core menus have Japanese
translations; untranslated panels and dialogs retain English. Japanese glyphs (UI: BIZ UDPGothic;
watermarks: BIZ UDMincho) come from storytold/craft-fonts, embedded by builds made with the
optional `CRAFT_FONTS_DIR` input (all releases), so no system fonts are needed. Text watermarks now accept
English, Simplified Chinese, Traditional Chinese (Taiwan) and Japanese interface languages are persisted in UI state, and the
language table (`crates/ui-egui/src/i18n.rs`) drives the menus, settings and fonts, so another
language is a table entry plus two catalogs (docs/localization.md). Core menus and panels are
translated; untranslated text retains English. CJK glyphs (Chinese: Noto Sans CJK SC; Japanese UI:
BIZ UDPGothic; watermarks: BIZ UDMincho) come from storytold/craft-fonts, embedded by builds made
with the optional `CRAFT_FONTS_DIR` input (all releases), so no system fonts are needed. Text watermarks now accept
`vertical: true` in export JSON/presets and expose an orientation selector. Japanese
characters stay upright in top-to-bottom columns, with newlines starting columns to the
left. This is basic lettering, without tate-chu-yoko, ruby, kinsoku, or general vertical
+116
View File
@@ -9,6 +9,7 @@
//! lightcraft-cli synth-merge hdr|panorama -o DIR
//! lightcraft-cli commands [--json]
//! lightcraft-cli controls [--json]
//! lightcraft-cli calibrate [--max N] [--out DIR] FOLDERS/FILES…
//! ```
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -84,6 +85,12 @@ USAGE:
Write synthetic merge inputs (procedural scene; bracketed DNGs or overlapping PNG views).
lightcraft-cli commands [--json] list every command id with its parameters
lightcraft-cli controls [--json] list every develop control id with its range
lightcraft-cli calibrate [--max N] [--out DIR] FOLDERS/FILES…
Fit a colour profile per camera model from raw files and their embedded camera JPEGs
(Sony ARW, Nikon NEF): up to N files spread over the folders (default 300; 0 = all), pooled per
model, written as <model>.json to DIR (default: the profiles folder LightCraft reads,
<config>/camera-profiles, or $LIGHTCRAFT_CAMERA_PROFILES). Raws of a profiled model then
take their colour from the profile and only their tone from their own JPEG.
lightcraft-cli --version | --help
";
@@ -125,6 +132,7 @@ fn main() -> ExitCode {
Some("merge") => merge(&args[1..]),
Some("synth-merge") => synth_merge(&args[1..]),
Some("controls") => controls(&args[1..]),
Some("calibrate") => calibrate(&args[1..]),
Some("--version" | "-V" | "version") => {
println!("lightcraft-cli {}", env!("CARGO_PKG_VERSION"));
Ok(())
@@ -144,6 +152,114 @@ fn main() -> ExitCode {
}
}
/// Raw files below `path` (or `path` itself), skipping hidden and NAS metadata folders.
fn raw_files(path: &Path, out: &mut Vec<std::path::PathBuf>, depth: usize) {
let is_raw = |p: &Path| p.extension().and_then(|e| e.to_str()).is_some_and(|e| ["arw", "nef", "nrw"].iter().any(|x| e.eq_ignore_ascii_case(x)));
if path.is_file() {
if is_raw(path) {
out.push(path.to_path_buf());
}
return;
}
let Ok(entries) = std::fs::read_dir(path) else { return };
if depth > 32 {
return;
}
for entry in entries.flatten() {
let name = entry.file_name();
if name.to_string_lossy().starts_with(['.', '@']) {
continue;
}
raw_files(&entry.path(), out, depth + 1);
}
}
fn calibrate(args: &[String]) -> Result<(), String> {
let mut max = 300usize;
let mut out: Option<std::path::PathBuf> = None;
let mut inputs = Vec::new();
let mut i = 0;
while i < args.len() {
match args[i].as_str() {
"--max" => max = take_value(args, &mut i, "--max")?.parse().map_err(|_| "--max needs a number".to_string())?,
"--out" => out = Some(take_value(args, &mut i, "--out")?.into()),
a if a.starts_with("--") => return Err(format!("unknown option `{a}`")),
f => inputs.push(f.to_string()),
}
i += 1;
}
if inputs.is_empty() {
return Err("calibrate needs folders or raw files".into());
}
let mut files = Vec::new();
for input in &inputs {
raw_files(Path::new(input), &mut files, 0);
}
files.sort();
files.dedup();
let found = files.len();
if max > 0 && files.len() > max {
// spread over all folders (dates, scenes) rather than the first N
files = (0..max).filter_map(|k| files.get(k * found / max).cloned()).collect();
}
eprintln!("calibrate: {} of {found} raw files", files.len());
let dir = out.or_else(lightcraft_engine::camera_profiles::dir).ok_or("no profiles folder: pass --out DIR")?;
let workers = std::thread::available_parallelism().map_or(4, |n| n.get()).clamp(1, 6);
let next = std::sync::atomic::AtomicUsize::new(0);
let done = std::sync::atomic::AtomicUsize::new(0);
let pools: Vec<lightcraft_engine::camera_profiles::Pool> = std::thread::scope(|scope| {
let handles: Vec<_> = (0..workers)
.map(|_| {
scope.spawn(|| {
let mut pool = lightcraft_engine::camera_profiles::Pool::default();
while let Some(path) = files.get(next.fetch_add(1, std::sync::atomic::Ordering::Relaxed)) {
let result = std::fs::read(path).map_err(|e| e.to_string()).and_then(|bytes| pool.add(&bytes));
let n = done.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + 1;
match result {
Ok(Some(model)) => eprintln!("[{n}/{}] {} ({model})", files.len(), path.display()),
Ok(None) => eprintln!("[{n}/{}] {}: skipped (no usable camera JPEG or colour)", files.len(), path.display()),
Err(e) => eprintln!("[{n}/{}] {}: {e}", files.len(), path.display()),
}
}
pool
})
})
.collect();
handles.into_iter().filter_map(|h| h.join().ok()).collect()
});
let mut pool = lightcraft_engine::camera_profiles::Pool::default();
for p in pools {
pool.merge(p);
}
let mut written = 0;
for result in pool.fit(5) {
match result {
Ok(profile) => {
let path = lightcraft_engine::camera_profiles::save(&profile, &dir)?;
println!(
"{}: {} photos, {} colour pairs, hue/saturation table {} → {}",
profile.model,
profile.files,
profile.samples,
profile.hue_sat.is_some(),
path.display()
);
written += 1;
}
Err(e) => eprintln!("calibrate: {e}"),
}
}
for (model, n) in pool.files() {
if n < 5 {
eprintln!("calibrate: {model}: only {n} usable photo(s), at least 5 needed");
}
}
if written == 0 {
return Err("no profile written".into());
}
Ok(())
}
fn mcp(args: &[String]) -> Result<(), String> {
let mut connect: Option<String> = None;
let mut demo = false;
+1 -1
View File
@@ -10,7 +10,7 @@ default-run = "lightcraft"
workspace = true
[dependencies]
lightcraft-engine = { workspace = true }
lightcraft-engine = { workspace = true, features = ["sam"] }
lightcraft-sysmem = { workspace = true }
lightcraft-ui-egui = { workspace = true }
lightcraft-codecs = { workspace = true }
+40 -11
View File
@@ -109,16 +109,7 @@ fn gpu_crash_notice(what: &str) -> String {
}
fn config_dir() -> Option<std::path::PathBuf> {
if cfg!(target_os = "macos") {
std::env::var_os("HOME").map(|h| std::path::PathBuf::from(h).join("Library/Application Support/LightCraft"))
} else if cfg!(windows) {
std::env::var_os("APPDATA").map(|a| std::path::PathBuf::from(a).join("LightCraft"))
} else {
std::env::var_os("XDG_CONFIG_HOME")
.map(std::path::PathBuf::from)
.or_else(|| std::env::var_os("HOME").map(|h| std::path::PathBuf::from(h).join(".config")))
.map(|c| c.join("lightcraft"))
}
lightcraft_engine::camera_profiles::config_dir()
}
/// The saved UI state and app settings (`<config>/ui.json`), if any, and a warning for the user
@@ -415,10 +406,42 @@ OPTIONS:
ENVIRONMENT:
LIGHTCRAFT_GPU_BACKEND=dx12|vulkan|metal|auto|off graphics backend (default: DX12 on Windows, Metal on macOS,
Vulkan on Linux; off = render on the CPU); else WGPU_BACKEND. LIGHTCRAFT_GPU=0: CPU rendering.
LIGHTCRAFT_SAM3_DIR=DIR the SAM 3 model for Object / Describe masks (default: <settings folder>/models/sam3;
optional: LightCraft offers to download it when first needed)
LIGHTCRAFT_SAM3_MIRRORS=URL,… where to download the SAM 3 model from (base URLs, tried in order)
";
/// Warnings and errors (failed commands, AI mask analysis) on stderr; `LIGHTCRAFT_LOG=info`
/// (or `debug`) shows more.
struct StderrLog(log::LevelFilter);
impl log::Log for StderrLog {
fn enabled(&self, m: &log::Metadata) -> bool {
m.level() <= self.0 && (m.level() <= log::Level::Warn || m.target().starts_with("lightcraft"))
}
fn log(&self, r: &log::Record) {
if self.enabled(r.metadata()) {
eprintln!("lightcraft: {} {}: {}", r.level(), r.target(), r.args());
}
}
fn flush(&self) {}
}
fn install_log() {
let level = match std::env::var("LIGHTCRAFT_LOG").unwrap_or_default().as_str() {
"debug" => log::LevelFilter::Debug,
"info" => log::LevelFilter::Info,
_ => log::LevelFilter::Warn,
};
static LOGGER: std::sync::OnceLock<StderrLog> = std::sync::OnceLock::new();
if log::set_logger(LOGGER.get_or_init(|| StderrLog(level))).is_ok() {
log::set_max_level(level);
}
}
fn main() -> eframe::Result {
lightcraft_engine::guard::install_hook(std::env::temp_dir().join("lightcraft-panics.log"));
install_log();
alloc_release::install();
let mut control_port: Option<u16> = std::env::var("LIGHTCRAFT_CONTROL_PORT").ok().and_then(|p| p.parse().ok());
let mut files = Vec::new();
@@ -481,7 +504,13 @@ fn main() -> eframe::Result {
"LightCraft",
options,
Box::new(move |cc| {
let (session, problem) = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let (mut session, problem) = open_session(in_memory, library_dir, seed_demo && files.is_empty());
// AI masks: the SAM 3 checkpoint (facebook/sam3) in <config>/models/sam3, or LIGHTCRAFT_SAM3_DIR
// (never required: without it, AI masks offer to download it; see docs/ai-masks.md)
session.segmenter.dir =
std::env::var_os("LIGHTCRAFT_SAM3_DIR").map(std::path::PathBuf::from).or_else(|| config_dir().map(|d| d.join("models").join("sam3")));
// the user's own download locations, one base URL per line (LIGHTCRAFT_SAM3_MIRRORS too)
session.segmenter.mirrors_file = config_dir().map(|d| d.join("models").join("sam3-mirrors.txt"));
let mut app = LightcraftApp::new(session, services());
if let Some(ui) = prefs {
app.ui = ui;
+22 -1
View File
@@ -171,7 +171,7 @@ fn structure_of(bar: &[(String, Vec<MenuNode>)]) -> String {
out.push(';');
}
}
let mut s = format!("{};", lightcraft_ui_egui::i18n::is_japanese());
let mut s = format!("{};", lightcraft_ui_egui::i18n::language().code());
for (t, items) in bar {
s.push_str(t);
walk(items, &mut s);
@@ -419,6 +419,27 @@ impl NativeMenu {
mod tests {
use super::*;
/// The menu model's titles stay English, so only the language can tell the native menu to
/// rebuild its titles (and the macOS app menu): every language switch — also between two CJK
/// languages — must change the structure key.
#[test]
fn switching_language_rebuilds_native_menu_structure() {
use lightcraft_ui_egui::i18n::{Locale, set_language};
let app = LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
set_language(Locale::En);
let bar = menu_bar(&app);
let mut seen = std::collections::BTreeSet::new();
for language in Locale::ALL {
set_language(*language);
assert!(seen.insert(structure_of(&bar)), "{language:?} shares another language's menu structure key");
}
set_language(Locale::ZhHant);
assert_eq!(label_text("File"), "檔案");
assert_eq!(label_text("Settings…"), "設定…");
assert_eq!(label_text("Quit LightCraft"), "結束 LightCraft");
set_language(Locale::En);
}
#[test]
fn shortcuts_map_to_accelerators() {
assert_eq!(accelerator("Cmd+Shift+Z"), Some(Accelerator::new(Modifiers::META | Modifiers::SHIFT, Code::KeyZ)));
+1
View File
@@ -10,6 +10,7 @@ covers a whole directory), or if a licence file referenced in the Licence column
|---|---|---|---|---|---|---|
| `assets/fonts/Inter-*.ttf` (Regular, Medium, SemiBold) | Inter | Rasmus Andersson | https://github.com/rsms/inter | SIL Open Font License 1.1 (`assets/fonts/OFL-Inter.txt`) | 2026-09-30 | none |
| `assets/app-icon/` (`lightcraft.svg` and the PNG/ICNS/ICO/SVG files derived from it) | LightCraft app icon (lynx) | LightCraft project owner (drawn in ArtCraft) | original work (vectorised from the owner's drawing; see `assets/app-icon/README.md`) | MIT OR Apache-2.0 (`assets/app-icon/LICENSE.txt`) | 2026-10-02 | vectorised to SVG; rendered to PNG/ICNS/ICO by `packaging/icons.sh` |
| `assets/camera-profiles/*.json` | Camera colour profiles (matrix + hue/saturation/value table), built into LightCraft | ILCE-7M4: Harald Hoyer | original work: fitted by `lightcraft-cli calibrate` to the contributor's own raw photos and their in-camera JPEGs (597 ILCE-7M4 photos, 2026); aggregate colour statistics only, no image content; no Adobe or third-party profile data | MIT OR Apache-2.0 | 2026-10-07 | none |
| `crates/scenes/` (generated images) | Procedural demo photographs | LightCraft contributors | original work (generated at runtime, no source imagery) | MIT OR Apache-2.0 | 2026-09-30 | n/a |
| `crates/ui-egui/src/icons.rs` | UI icons drawn as vector paths in code | LightCraft contributors | original work | MIT OR Apache-2.0 | 2026-09-30 | n/a |
| `docs/images/*-tetons.jpg`, `docs/images/grid-pd.jpg` (screenshots containing it) | "The Tetons and the Snake River" | Ansel Adams (1942), U.S. National Archives | https://commons.wikimedia.org/wiki/File:Adams_The_Tetons_and_the_Snake_River.jpg | Public domain (U.S. federal government work) | 2026-09-30 | developed in LightCraft; shown inside app screenshots |
File diff suppressed because one or more lines are too long
+1
View File
@@ -10,6 +10,7 @@ workspace = true
[dependencies]
lightcraft-develop = { workspace = true }
lightcraft-meta = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
+1 -1
View File
@@ -35,7 +35,7 @@ use lightcraft_develop::DevelopSettings;
pub use local::{DEFAULT_FORGET_DAYS, ForgetPlan, folder_of};
pub use lock::{LibraryLock, LockError, LockOwner};
pub use model::*;
pub use query::{DateGroup, Filter, RatingOp, Sort, SortKey};
pub use query::{DateGroup, Filter, Person, RatingOp, Sort, SortKey};
pub use rules::{Match, Rule, RuleSet};
use serde::{Deserialize, Serialize};
pub use store::{FsStore, MemStore, Store};
+5
View File
@@ -183,6 +183,11 @@ pub struct Meta {
pub copyright_url: String,
pub creator: String,
pub keywords: Vec<String>,
/// Face/pet/focus regions read from XMP (MWG-RS), on the upright (EXIF-oriented) photo.
/// Removing or resizing one edits the catalog only; LightCraft never writes regions to XMP.
/// Left out of the catalog JSON when empty (most photos), so older catalogs read unchanged.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub regions: Vec<lightcraft_meta::Region>,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
+70 -3
View File
@@ -39,6 +39,8 @@ pub struct Filter {
pub date: Option<String>,
/// A keyword; hierarchical keywords match their children too (`travel` finds `travel|italy`).
pub keyword: Option<String>,
/// A person: photos with a named face region of this name (case-insensitive), as read from XMP.
pub person: Option<String>,
pub camera: Option<String>,
/// Lens (case-insensitive substring).
pub lens: Option<String>,
@@ -113,6 +115,7 @@ fn token_matches(p: &Photo, tok: &str) -> bool {
"camera" => p.meta.camera.to_lowercase().contains(val),
"lens" => p.meta.lens.to_lowercase().contains(val),
"keyword" | "kw" => p.meta.keywords.iter().any(|k| crate::keywords::is_under(k, val)),
"person" | "who" => has_person(p, val),
"type" | "kind" => format!("{:?}", p.kind).eq_ignore_ascii_case(val),
"edited" => (val == "true" || val == "yes") == p.is_edited(),
"date" => p.date().starts_with(val),
@@ -125,6 +128,12 @@ fn token_matches(p: &Photo, tok: &str) -> bool {
hay.iter().any(|h| h.to_lowercase().contains(&t)) || p.meta.keywords.iter().any(|k| k.to_lowercase().contains(&t))
}
/// Whether `p` has a named face region called `name` (case-insensitive).
fn has_person(p: &Photo, name: &str) -> bool {
let name = name.trim().to_lowercase();
p.meta.regions.iter().any(|r| r.kind == lightcraft_meta::RegionKind::Face && r.name.as_deref().is_some_and(|n| n.to_lowercase() == name))
}
impl Filter {
/// Human-readable summary of the active rules (smart album tooltips, `album.list`).
pub fn describe(&self) -> String {
@@ -152,9 +161,14 @@ impl Filter {
if let Some(e) = self.edited {
v.push(if e { "edited".into() } else { "unedited".into() });
}
for (name, val) in
[("keyword", &self.keyword), ("camera", &self.camera), ("lens", &self.lens), ("date", &self.date), ("imported", &self.imported)]
{
for (name, val) in [
("keyword", &self.keyword),
("person", &self.person),
("camera", &self.camera),
("lens", &self.lens),
("date", &self.date),
("imported", &self.imported),
] {
if let Some(x) = val {
v.push(format!("{name} {x}"));
}
@@ -267,6 +281,11 @@ impl Filter {
{
return false;
}
if let Some(n) = &self.person
&& !has_person(p, n)
{
return false;
}
if let Some(c) = &self.camera
&& !p.meta.camera.eq_ignore_ascii_case(c)
{
@@ -330,6 +349,54 @@ impl Catalog {
}
m.into_iter().collect()
}
/// The people named on faces in the library (MWG regions read from XMP): how many photos each
/// appears in and the photo showing their largest face (for a card's picture). Most photos first,
/// then by name. Names that differ only in case are one person, shown as first seen; a person
/// twice in one photo counts once.
pub fn people(&self) -> Vec<Person> {
self.people_in(&Filter::default())
}
/// [`Self::people`] among the photos `filter` lets through (its `person` is ignored): what the
/// People view offers while other filters (a date, a rating, an album…) are active, so picking
/// a person never ends in an empty grid.
pub fn people_in(&self, filter: &Filter) -> Vec<Person> {
let filter = Filter { person: None, ..filter.clone() };
let mut m: std::collections::HashMap<String, (Person, f64)> = Default::default();
for p in self.photos().filter(|p| filter.matches(p, self)) {
let mut seen: Vec<String> = Vec::new();
for r in p.meta.regions.iter().filter(|r| r.kind == lightcraft_meta::RegionKind::Face) {
let Some(name) = r.name.as_deref().map(str::trim).filter(|n| !n.is_empty()) else { continue };
let key = name.to_lowercase();
// the face's size in pixels of the photo
let area = (r.rect.x1 - r.rect.x0) * (r.rect.y1 - r.rect.y0) * p.width as f64 * p.height as f64;
let (person, best) =
m.entry(key.clone()).or_insert_with(|| (Person { name: name.to_string(), count: 0, photo: p.id, face: r.rect }, area));
if !seen.contains(&key) {
person.count += 1;
seen.push(key);
}
if area > *best || (area == *best && p.id < person.photo) {
(person.photo, person.face, *best) = (p.id, r.rect, area);
}
}
}
let mut v: Vec<Person> = m.into_values().map(|(p, _)| p).collect();
v.sort_by(|a, b| b.count.cmp(&a.count).then_with(|| a.name.to_lowercase().cmp(&b.name.to_lowercase())));
v
}
}
/// A person named on faces in the library ([`Catalog::people`]).
#[derive(Clone, Debug, PartialEq)]
pub struct Person {
pub name: String,
/// Photos they appear in.
pub count: usize,
/// The photo with their largest face, and that face (normalized, in the photo's upright frame).
pub photo: PhotoId,
pub face: lightcraft_meta::Rect,
}
#[derive(Clone, Debug, PartialEq, Serialize)]
+83
View File
@@ -297,3 +297,86 @@ fn folder_identity_ignores_spelling() {
assert!(folder_within("/a/b", "/"));
assert!(folder_within("C:\\x", "c:\\"));
}
/// People come from named *face* regions: counted once per photo, case-insensitive, most photos
/// first; pets and unnamed faces are not people; the `person` filter and `person:` token match.
#[test]
fn people_from_named_face_regions() {
use lightcraft_meta::{Rect, Region, RegionKind};
let region = |name: Option<&str>, kind: RegionKind| Region {
rect: Rect { x0: 0.4, y0: 0.4, x1: 0.6, y1: 0.6 },
kind,
name: name.map(str::to_string),
description: None,
};
let mut c = Catalog::new();
let mut add = |name: &str, regions: Vec<Region>| {
let id = c.alloc_photo_id();
let mut p = Photo::new(id, Source::Demo { scene: 1 }, name, "JPEG", 6000, 4000, "2026-09-30T10:00:00");
p.meta.regions = regions;
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
id
};
let a = add(
"a.jpg",
vec![region(Some("Jane Doe"), RegionKind::Face), region(Some("jane doe"), RegionKind::Face), region(Some("Rex"), RegionKind::Pet)],
);
let b =
add("b.jpg", vec![region(Some("JANE DOE"), RegionKind::Face), region(Some("John Roe"), RegionKind::Face), region(None, RegionKind::Face)]);
let big = Rect { x0: 0.2, y0: 0.2, x1: 0.8, y1: 0.8 };
let d = add("d.jpg", vec![Region { rect: big, ..region(Some("John Roe"), RegionKind::Face) }]);
let e = add("e.jpg", vec![region(Some("Sam"), RegionKind::Face)]);
add("f.jpg", vec![]);
let people = c.people();
let summary: Vec<(&str, usize)> = people.iter().map(|p| (p.name.as_str(), p.count)).collect();
assert_eq!(summary, vec![("Jane Doe", 2), ("John Roe", 2), ("Sam", 1)], "once per photo, pets and unnamed faces left out");
// the picture is the person's largest face; ties go to the lower photo id
assert_eq!((people[0].photo, people[0].face), (a, Rect { x0: 0.4, y0: 0.4, x1: 0.6, y1: 0.6 }));
assert_eq!((people[1].photo, people[1].face), (d, big), "the larger face wins over an earlier, smaller one");
let q = |f: Filter| c.query(&f, &Sort::default());
assert_eq!(q(Filter { person: Some("jane doe".into()), ..Default::default() }).len(), 2);
let mut got = q(Filter { person: Some("John Roe".into()), ..Default::default() });
got.sort();
assert_eq!(got, vec![b, d]);
assert!(q(Filter { person: Some("Rex".into()), ..Default::default() }).is_empty(), "a pet is not a person");
assert_eq!(q(Filter { text: "person:SAM".into(), ..Default::default() }), vec![e], "the search token finds a person, any case");
assert!(q(Filter { person: Some("Jane Doe".into()), ..Default::default() }).contains(&a));
// other filters narrow who is offered (so picking a person never ends in an empty grid); the
// `person` filter itself does not
c.apply(Op::SetRating { id: b, rating: 3 }).unwrap();
let names = |f: Filter| c.people_in(&f).into_iter().map(|p| (p.name, p.count)).collect::<Vec<_>>();
let rated = Filter { rating: 1, ..Default::default() };
assert_eq!(
names(rated.clone()),
vec![("JANE DOE".to_string(), 1), ("John Roe".to_string(), 1)],
"only the rated photo's people, counted within it, spelled as first seen there"
);
assert_eq!(
names(Filter { person: Some("Sam".into()), ..rated }),
names(Filter { rating: 1, ..Default::default() }),
"the person filter is ignored"
);
assert_eq!(names(Filter { rating: 5, ..Default::default() }), vec![], "nobody in the filtered photos");
}
/// Most photos have no regions: the field is left out of the catalog JSON then, and a catalog written
/// before regions existed (no `regions` key) reads with none.
#[test]
fn empty_regions_are_not_serialized_and_default_when_missing() {
let m = Meta { keywords: vec!["k".into()], ..Default::default() };
let v = serde_json::to_value(&m).unwrap();
assert!(v.get("regions").is_none(), "{v}");
let back: Meta = serde_json::from_value(v).unwrap();
assert!(back.regions.is_empty());
let mut with = m.clone();
with.regions.push(lightcraft_meta::Region {
rect: lightcraft_meta::Rect { x0: 0.1, y0: 0.1, x1: 0.2, y1: 0.2 },
kind: lightcraft_meta::RegionKind::Face,
name: Some("A".into()),
description: None,
});
let back: Meta = serde_json::from_str(&serde_json::to_string(&with).unwrap()).unwrap();
assert_eq!(back, with);
}
+1
View File
@@ -13,6 +13,7 @@ lightcraft-geom = { workspace = true }
lightcraft-color = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
miniz_oxide = { workspace = true }
[dev-dependencies]
proptest = { workspace = true }
+2
View File
@@ -8,10 +8,12 @@
pub mod controls;
pub mod presets;
pub mod segmask;
pub mod settings;
pub use controls::{CONTROLS, ControlSpec, Section, Track};
pub use presets::{Preset, SettingsGroup, apply_partial, extract_groups};
pub use segmask::SegMask;
pub use settings::*;
use serde_json::Value;
+183
View File
@@ -0,0 +1,183 @@
//! Model segmentations stored with a mask ([`SegMask`]), so rendering never needs the model.
use serde::{Deserialize, Serialize};
/// Logits are stored in steps of 1/`SCALE`, clamped to ±127/`SCALE` (≈ ±15.9: a sigmoid of
/// 1 − 1.2e-7, i.e. fully in or out).
const SCALE: f32 = 8.0;
/// Largest grid a stored segmentation may claim (hostile input caps the allocation).
pub const MAX_SIDE: usize = 1024;
/// A segmentation computed by a model (SAM 3 in `lightcraft-segment`): `side × side` logits
/// over the uncropped, oriented image stretched to a square (row `y`, column `x` covers
/// normalized `((x + 0.5) / side, (y + 0.5) / side)`), quantized to `i8`, deflated and base64
/// encoded. Coverage at a point is the sigmoid of the bilinearly sampled logit, so edges stay
/// smooth at any output size.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct SegMask {
pub side: u32,
pub data: String,
/// The part of the image the grid covers, normalized `[x0, y0, x1, y1]` (`None` = all of
/// it). A zoomed-in pass over one object covers just its surroundings, at a higher
/// resolution than a pass over the whole photo.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub rect: Option<[f64; 4]>,
}
impl SegMask {
/// From `side × side` logits over the whole image.
pub fn from_logits(side: usize, logits: &[f32]) -> SegMask {
let q: Vec<u8> = logits.iter().take(side * side).map(|l| ((l * SCALE).round().clamp(-127.0, 127.0) as i8) as u8).collect();
let z = miniz_oxide::deflate::compress_to_vec(&q, 8);
SegMask { side: side as u32, data: base64_encode(&z), rect: None }
}
/// From `side × side` logits over the part `rect` of the image.
pub fn from_logits_in(side: usize, logits: &[f32], rect: [f64; 4]) -> SegMask {
SegMask { rect: Some(rect), ..SegMask::from_logits(side, logits) }
}
/// The covered part, normalized (the whole image without a `rect`); `None` when damaged.
pub fn bounds(&self) -> Option<[f64; 4]> {
let r = self.rect.unwrap_or([0.0, 0.0, 1.0, 1.0]);
(r.iter().all(|v| v.is_finite()) && r[2] > r[0] && r[3] > r[1]).then_some(r)
}
/// The logits, or `None` when the data is damaged or claims an absurd size.
pub fn logits(&self) -> Option<Vec<f32>> {
let side = self.side as usize;
if side == 0 || side > MAX_SIDE {
return None;
}
let z = base64_decode(&self.data)?;
let q = miniz_oxide::inflate::decompress_to_vec_with_limit(&z, side * side).ok()?;
(q.len() == side * side).then(|| q.iter().map(|b| f32::from(*b as i8) / SCALE).collect())
}
}
/// Most zoomed-in detail patches one AI mask component keeps (the app makes one; each costs a
/// decode per render, so a hostile document can't make rendering explode).
pub const MAX_DETAIL: usize = 4;
/// Deserialize a component's detail patches, keeping at most [`MAX_DETAIL`] (the rest are
/// skipped without being decoded).
pub(crate) fn de_detail<'de, D: serde::Deserializer<'de>>(d: D) -> Result<Vec<SegMask>, D::Error> {
struct Capped;
impl<'de> serde::de::Visitor<'de> for Capped {
type Value = Vec<SegMask>;
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("a list of segmentations")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(self, mut seq: A) -> Result<Vec<SegMask>, A::Error> {
let mut out = Vec::new();
while out.len() < MAX_DETAIL {
match seq.next_element::<SegMask>()? {
Some(m) => out.push(m),
None => return Ok(out),
}
}
while seq.next_element::<serde::de::IgnoredAny>()?.is_some() {}
Ok(out)
}
}
d.deserialize_seq(Capped)
}
const B64: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
fn base64_encode(bytes: &[u8]) -> String {
let mut out = String::with_capacity(bytes.len().div_ceil(3) * 4);
for chunk in bytes.chunks(3) {
let b = [chunk.first().copied().unwrap_or(0), chunk.get(1).copied().unwrap_or(0), chunk.get(2).copied().unwrap_or(0)];
let n = (u32::from(b[0]) << 16) | (u32::from(b[1]) << 8) | u32::from(b[2]);
for i in 0..4 {
if i <= chunk.len() {
out.push(char::from(B64[((n >> (18 - 6 * i)) & 63) as usize]));
} else {
out.push('=');
}
}
}
out
}
fn base64_decode(s: &str) -> Option<Vec<u8>> {
let mut out = Vec::with_capacity(s.len() / 4 * 3);
let (mut acc, mut bits) = (0u32, 0u32);
for c in s.bytes().filter(|c| !c.is_ascii_whitespace()) {
if c == b'=' {
break;
}
let v = B64.iter().position(|b| *b == c)? as u32;
acc = (acc << 6) | v;
bits += 6;
if bits >= 8 {
bits -= 8;
out.push((acc >> bits) as u8);
acc &= (1 << bits) - 1;
}
}
Some(out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn base64_round_trips() {
for n in 0..10 {
let v: Vec<u8> = (0..n).map(|i| (i * 37 + 11) as u8).collect();
assert_eq!(base64_decode(&base64_encode(&v)).unwrap(), v);
}
assert_eq!(base64_encode(b"Man"), "TWFu");
assert_eq!(base64_encode(b"Ma"), "TWE=");
assert!(base64_decode("TW!u").is_none());
}
#[test]
fn logits_round_trip_quantized() {
let side = 16;
let l: Vec<f32> = (0..side * side).map(|i| (i as f32 - 128.0) / 4.0).collect();
let m = SegMask::from_logits(side, &l);
let back = m.logits().unwrap();
for (a, b) in l.iter().zip(&back) {
assert!((a.clamp(-127.0 / SCALE, 127.0 / SCALE) - b).abs() <= 0.5 / SCALE + 1e-6);
}
}
#[test]
fn a_hostile_document_keeps_at_most_max_detail_patches() {
let patch = serde_json::to_value(SegMask::from_logits(4, &[1.0; 16])).unwrap();
let shape = serde_json::json!({"kind": "object", "hint": [{"x": 0.5, "y": 0.5}], "detail": vec![patch; 10_000]});
let s: crate::MaskShape = serde_json::from_value(shape).unwrap();
let crate::MaskShape::Object { detail, .. } = s else { panic!("an Object") };
assert_eq!(detail.len(), MAX_DETAIL);
// and a whole develop document with them still loads
let mut d = crate::DevelopSettings::default();
d.masks.push(crate::Mask {
id: 1,
components: vec![crate::MaskComponent {
name: None,
op: crate::MaskOp::Add,
invert: false,
shape: crate::MaskShape::Prompt { text: "x".into(), seg: None, detail: vec![SegMask::from_logits(4, &[1.0; 16]); 50], edge: 0.0 },
}],
..Default::default()
});
let back: crate::DevelopSettings = serde_json::from_str(&serde_json::to_string(&d).unwrap()).unwrap();
let crate::MaskShape::Prompt { detail, .. } = &back.masks[0].components[0].shape else { panic!("a Prompt") };
assert_eq!(detail.len(), MAX_DETAIL);
}
#[test]
fn damaged_data_is_none() {
let mut m = SegMask::from_logits(8, &[1.0; 64]);
m.side = 9;
assert!(m.logits().is_none(), "size mismatch");
m.side = 100_000;
assert!(m.logits().is_none(), "absurd size");
assert!(SegMask { side: 8, data: "not base64 !".into(), rect: None }.logits().is_none());
assert!(SegMask { rect: Some([0.5, 0.0, 0.2, 1.0]), ..SegMask::from_logits(2, &[0.0; 4]) }.bounds().is_none());
}
}
+25
View File
@@ -772,8 +772,33 @@ pub enum MaskShape {
Subject,
Sky,
Background,
/// One object picked by clicks (SAM 3 point prompts): `hint` holds the clicks that include,
/// `exclude` the ones that exclude; `seg` the segmentation computed from them.
Object {
hint: Vec<Point>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
exclude: Vec<Point>,
#[serde(default, skip_serializing_if = "Option::is_none")]
seg: Option<crate::SegMask>,
/// Zoomed-in passes over parts of the image (higher resolution than `seg`), used
/// inside their rectangles.
#[serde(default, skip_serializing_if = "Vec::is_empty", deserialize_with = "crate::segmask::de_detail")]
detail: Vec<crate::SegMask>,
/// Edge −100..100: below 0 harder (a steeper transition), above 0 softer (feathered).
#[serde(default, skip_serializing_if = "is_zero")]
edge: f64,
},
/// Everything a description names ("sky", "the red car": SAM 3 concept prompts); `seg` is
/// the segmentation computed for it.
Prompt {
text: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
seg: Option<crate::SegMask>,
#[serde(default, skip_serializing_if = "Vec::is_empty", deserialize_with = "crate::segmask::de_detail")]
detail: Vec<crate::SegMask>,
/// Edge −100..100, as for `Object`.
#[serde(default, skip_serializing_if = "is_zero")]
edge: f64,
},
People {
person: u32,
+5
View File
@@ -30,6 +30,11 @@ serde_json = { workspace = true }
thiserror = { workspace = true }
log = { workspace = true }
ab_glyph = { workspace = true }
lightcraft-segment = { workspace = true, optional = true }
[features]
# AI masks with SAM 3 (Object / Describe). Off for the web and CLI builds.
sam = ["dep:lightcraft-segment"]
[dev-dependencies]
libc = { workspace = true }
+510 -41
View File
@@ -1,20 +1,31 @@
//! Estimate the starting look of a raw without a camera colour matrix (Sony ARW, Nikon NEF) from its
//! own JPEG. Colour and luminance are fitted separately; the JPEG supplies correspondences only,
//! never output pixels or a replacement for RAW editing.
use lightcraft_color::{Mat3, luminance_2020};
//! A global matrix can't follow the camera's hue-dependent rendering (the best matrix rendered a
//! lime shirt olive that the camera kept lime): a hue/saturation table fitted to the residuals
//! (applied like a DNG `ProfileHueSatMap`) corrects that when it also improves the held-out pixels.
use lightcraft_color::{D50, D65, Mat3, PROPHOTO, REC2020, bradford, luminance_2020};
use lightcraft_pipeline::tone::{CameraTone, ToneMap};
use lightcraft_raster::{
Rgb32f,
resample::{Filter, fit},
};
use lightcraft_raw::{RawFormat, RawImage, color::CameraTransform};
use lightcraft_raw::{RawFormat, RawImage, color::CameraTransform, profile::HsvTable};
#[derive(Clone, Copy, Debug)]
#[derive(Clone, Debug)]
pub(crate) struct CameraLook {
pub matrix: Mat3,
pub tone: CameraTone,
/// Hue/saturation correction after `matrix`, in linear ProPhoto RGB (DNG `ProfileHueSatMap`).
pub hue_sat: Option<HsvTable>,
}
/// Long edge of the sensor/JPEG proxy a single photo's look is fitted on (the acceptance gates
/// below were set at this size).
const PROXY: usize = 96;
/// Long edge of the proxies pooled for a camera profile: small objects (a shirt) get 4× the samples.
pub(crate) const PROFILE_PROXY: usize = 192;
/// Raw formats whose decoder supplies vendor white-balance multipliers but no camera colour matrix:
/// their starting look is fitted to the file's own JPEG, and white balance is relative to the
/// as-shot look (`docs/camera-preview-colour.md`). The catalog's `Photo::relative_wb` matches the
@@ -27,8 +38,31 @@ pub(crate) fn fit_preview(raw: &RawImage, bytes: &[u8], transform: &CameraTransf
if !transform.matrix_is_fallback || !file_local_look(raw.format) {
return None;
}
let (sensor, reference) = proxies(raw, bytes, transform, PROXY)?;
// A camera profile pooled from many photos knows colours this photo shows too little of;
// only the tone and chroma curves are fitted per photo (DRO and picture styles vary).
let profile = raw.metadata.model.as_deref().and_then(crate::camera_profiles::get);
let colour = profile.as_ref().and_then(|p| Some((p.matrix().mul(&transform.matrix.inverse()?), p.hue_sat.clone())));
let look = fit_pairs_with(&sensor, &reference, colour)?;
if lightcraft_pipeline::profiling() {
eprintln!(
"[profile] {:?} camera look: {:?}, {:?}, hue/sat table {}, camera profile {}",
raw.format,
look.matrix.0,
look.tone,
look.hue_sat.is_some(),
profile.is_some()
);
}
Some(look)
}
/// Same-size proxies of the sensor (white-balanced, baseline exposure, through `transform`'s
/// matrix: the generic camera ≈ sRGB model) and of the file's embedded camera JPEG (linear Rec.2020).
fn proxies(raw: &RawImage, bytes: &[u8], transform: &CameraTransform, size: usize) -> Option<(Rgb32f, Rgb32f)> {
let jpeg = lightcraft_raw::embedded_preview(bytes)?;
let decoded = lightcraft_codecs::decode(&jpeg, lightcraft_codecs::DecodeOptions { max_size: Some((384, 384)), max_pixels: 64_000_000 }).ok()?;
let edge = (2 * size).max(384) as u32;
let decoded = lightcraft_codecs::decode(&jpeg, lightcraft_codecs::DecodeOptions { max_size: Some((edge, edge)), max_pixels: 64_000_000 }).ok()?;
let reference = decoded.to_working();
let crop = raw.crop.clipped(raw.active_area.width, raw.active_area.height);
if crop.width == 0 || crop.height == 0 || reference.width == 0 || reference.height == 0 {
@@ -39,17 +73,34 @@ pub(crate) fn fit_preview(raw: &RawImage, bytes: &[u8], transform: &CameraTransf
return None;
}
// Fixed, bounded proxy: the selected look cannot depend on thumbnail/export resolution.
let k = (crop.width.max(crop.height).div_ceil(384).max(2)).div_ceil(2) * 2;
let k = (crop.width.max(crop.height).div_ceil(edge as usize).max(2)).div_ceil(2) * 2;
let sensor = raw.develop_binned(k, 0.99).ok()??;
let mut sensor = fit(&sensor, 96, 96, Filter::Box);
let mut sensor = fit(&sensor, size, size, Filter::Box);
let reference = fit(&reference, sensor.width, sensor.height, Filter::Box);
let gain = 2f32.powf(transform.baseline_exposure as f32);
sensor.map_in_place(|p| transform.matrix.apply_f32(std::array::from_fn(|i| p[i] * transform.wb[i] * gain)));
let look = fit_pairs(&sensor, &reference)?;
if lightcraft_pipeline::profiling() {
eprintln!("[profile] {:?} camera look: {:?}, {:?}", raw.format, look.matrix.0, look.tone);
Some((sensor, reference))
}
/// Colour training pairs of one raw for a camera profile: white-balanced camera RGB (with the
/// baseline exposure) → its camera JPEG (linear Rec.2020). `None` for formats with their own
/// colour matrices, other files or unusable previews.
pub(crate) fn profile_pairs(raw: &RawImage, bytes: &[u8]) -> Option<Vec<([f64; 3], [f64; 3])>> {
if !file_local_look(raw.format) || lightcraft_raw::color::has_matrix(&raw.color) {
return None;
}
Some(look)
let transform = lightcraft_raw::color::camera_transform(raw, lightcraft_raw::color::as_shot_white_xy(raw));
let (sensor, reference) = proxies(raw, bytes, &transform, PROFILE_PROXY)?;
let to_camera = transform.matrix.inverse()?;
let (pairs, _) = collect_pairs(&sensor, &reference)?;
Some(pairs.into_iter().map(|(x, y)| (to_camera.apply(x), y)).collect())
}
/// A camera profile's colour model (matrix from white-balanced camera RGB, hue/saturation table)
/// fitted to pairs pooled from many photos.
pub(crate) fn fit_profile(pairs: &[([f64; 3], [f64; 3])]) -> Option<(Mat3, Option<HsvTable>)> {
let matrix = fit_matrix(pairs)?;
Some((matrix, fit_hue_sat(pairs, &matrix)))
}
/// A fit must cut the held-out squared error to below this share of the fallback's.
@@ -64,27 +115,43 @@ fn luma(p: [f64; 3]) -> f64 {
p[0] * 0.2627 + p[1] * 0.6780 + p[2] * 0.0593
}
/// The finish stage's tone map and chroma curve (`lightcraft_pipeline::finish`).
fn displayed(scene: [f64; 3], tone: &ToneMap) -> [f64; 3] {
let scene = scene.map(|v| v.max(0.0));
let y = luma(scene);
if y <= 0.0 {
return [0.0; 3];
}
scene.map(|v| v * f64::from(tone.apply(y as f32)) / y)
let o = f64::from(tone.apply(y as f32));
let k = f64::from(tone.chroma_scale(o as f32));
scene.map(|v| o + (v * o / y - o) * k)
}
#[cfg(test)]
fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
fit_pairs_with(sensor, reference, None)
}
/// Training pairs (sensor → JPEG, unclipped midtones) and the wider set including highlights
/// (for the chroma curve); `None` when too few, or the photo has too little colour.
type Pairs = Vec<([f64; 3], [f64; 3])>;
fn collect_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<(Pairs, Pairs)> {
if (sensor.width, sensor.height) != (reference.width, reference.height) || sensor.data.len() != reference.data.len() {
return None;
}
let mut pairs = Vec::new();
// Highlights too (camera JPEGs bleach colours toward white there), for the chroma curve only.
let mut bright = Vec::new();
let mut colour = 0;
for (input, output) in sensor.data.iter().zip(&reference.data) {
let y = luminance_2020(*output);
if !input.iter().all(|v| v.is_finite() && *v > 0.001 && *v < 1.5)
|| !output.iter().all(|v| v.is_finite() && *v > 0.004 && *v < 0.98)
|| !(0.015..0.85).contains(&y)
{
if !input.iter().all(|v| v.is_finite() && *v > 0.001 && *v < 1.5) || !output.iter().all(|v| v.is_finite() && *v >= 0.0) {
continue;
}
if (0.015..=1.0).contains(&y) {
bright.push((input.map(f64::from), output.map(f64::from)));
}
if !output.iter().all(|v| *v > 0.004 && *v < 0.98) || !(0.015..0.85).contains(&y) {
continue;
}
let min = output.iter().copied().fold(f32::INFINITY, f32::min);
@@ -92,9 +159,11 @@ fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
colour += usize::from(max - min > 0.05);
pairs.push((input.map(f64::from), output.map(f64::from)));
}
if pairs.len() < 256 || colour < pairs.len() / 20 {
return None;
}
(pairs.len() >= 256 && colour >= pairs.len() / 20).then_some((pairs, bright))
}
/// Ridge-regularised 3×3 chromaticity matrix (luminance-normalised RGB) on the training pairs.
fn fit_matrix(pairs: &[([f64; 3], [f64; 3])]) -> Option<Mat3> {
let mut gram = [[0.0; 3]; 3];
let mut cross = [[0.0; 3]; 3];
for (i, (x, y)) in pairs.iter().enumerate() {
@@ -102,6 +171,9 @@ fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
continue;
}
let (lx, ly) = (luma(*x), luma(*y));
if lx <= 0.0 || ly <= 0.0 {
continue;
}
for row in 0..3 {
for col in 0..3 {
// Normalising by luminance prevents a camera S-curve from corrupting colour.
@@ -122,29 +194,62 @@ fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
cross[i][i] += regularization;
}
let matrix = Mat3(cross).mul(&Mat3(gram).inverse()?);
if !matrix.0.iter().flatten().all(|v| v.is_finite() && v.abs() < 8.0) {
return None;
matrix.0.iter().flatten().all(|v| v.is_finite() && v.abs() < 8.0).then_some(matrix)
}
/// The photo's look: its own matrix (with and without a hue/saturation table), or the given
/// colour model (a camera profile's, in the sensor proxy's space), each completed with a tone
/// and chroma curve fitted to this photo.
fn fit_pairs_with(sensor: &Rgb32f, reference: &Rgb32f, colour: Option<(Mat3, Option<HsvTable>)>) -> Option<CameraLook> {
let (pairs, bright) = collect_pairs(sensor, reference)?;
let candidates = match colour {
Some(given) => vec![given],
None => {
let matrix = fit_matrix(&pairs)?;
vec![(matrix, fit_hue_sat(&pairs, &matrix)), (matrix, None)]
}
};
// Matrix + table when the table helps the held-out pixels, else the matrix alone. Chosen by
// error in gamma-encoded display values (closer to what is seen: in linear values a slightly
// missed bright rock outweighs a clearly wrong dark shirt); the acceptance gates below stay linear.
let mut best: Option<(f64, f64, usize, CameraLook)> = None;
for (matrix, hue_sat) in candidates {
let correction = hue_sat.as_ref().and_then(HueSat::new);
let colour = |x: [f64; 3]| {
let p = matrix.apply(x);
correction.as_ref().map_or(p, |c| c.apply(p.map(|v| v as f32)).map(f64::from))
};
let tone_pairs: Vec<_> =
pairs.iter().enumerate().filter(|(i, _)| i % 3 != 0).map(|(_, (x, y))| (luma(colour(*x).map(|v| v.max(0.0))), luma(*y))).collect();
let Some(curve) = fit_tone(tone_pairs) else { continue };
let mut look = CameraLook { matrix, tone: curve, hue_sat: hue_sat.clone() };
if let Some(tone) = fit_chroma(&bright, &look) {
look.tone = tone;
}
let tone = ToneMap::camera(&look.tone, 0.0, 0.0, 0.0);
let (mut linear, mut perceptual, mut samples) = (0.0, 0.0, 0);
for (x, target) in pairs.iter().step_by(3) {
let corrected = displayed(colour(*x), &tone);
for c in 0..3 {
linear += (corrected[c] - target[c]).powi(2);
perceptual += (corrected[c].max(0.0).powf(1.0 / 2.2) - target[c].max(0.0).powf(1.0 / 2.2)).powi(2);
samples += 1;
}
}
if linear.is_finite() && perceptual.is_finite() && best.as_ref().is_none_or(|b| perceptual < b.0) {
best = Some((perceptual, linear, samples, look));
}
}
let tone_pairs: Vec<_> =
pairs.iter().enumerate().filter(|(i, _)| i % 3 != 0).map(|(_, (x, y))| (luma(matrix.apply(*x).map(|v| v.max(0.0))), luma(*y))).collect();
let curve = fit_tone(tone_pairs)?;
let tone = ToneMap::camera(&curve, 0.0, 0.0, 0.0);
let (_, after, samples, look) = best?;
let original_tone = ToneMap::new(0.0, 0.0, 0.0);
let mut before = 0.0;
let mut after = 0.0;
let mut samples = 0;
for (i, (x, target)) in pairs.iter().enumerate() {
if i % 3 != 0 {
continue;
}
let corrected = displayed(matrix.apply(*x), &tone);
let original = displayed(*x, &original_tone);
for c in 0..3 {
before += (original[c] - target[c]).powi(2);
after += (corrected[c] - target[c]).powi(2);
samples += 1;
}
}
let before: f64 = pairs
.iter()
.step_by(3)
.map(|(x, target)| {
let original = displayed(*x, &original_tone);
(0..3).map(|c| (original[c] - target[c]).powi(2)).sum::<f64>()
})
.sum();
if lightcraft_pipeline::profiling() {
eprintln!(
"[profile] camera look holdout RMS {:.5} -> {:.5} ({samples} channels)",
@@ -152,10 +257,211 @@ fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
(after / samples as f64).sqrt()
);
}
if !after.is_finite() || samples == 0 || after >= before * MIN_IMPROVEMENT || after / samples as f64 > MAX_HOLDOUT_RMS.powi(2) {
if samples == 0 || after >= before * MIN_IMPROVEMENT || after / samples as f64 > MAX_HOLDOUT_RMS.powi(2) {
return None;
}
Some(CameraLook { matrix, tone: curve })
Some(look)
}
/// Linear Rec.2020 D65 → linear ProPhoto RGB D50, the space DNG hue/saturation tables work in.
fn to_prophoto() -> Mat3 {
PROPHOTO.from_xyz().mul(&bradford(D65, D50)).mul(&REC2020.to_xyz())
}
/// A fitted [`CameraLook::hue_sat`] table, ready to apply to linear Rec.2020 pixels. It changes
/// hue and saturation only: luminance is restored, the camera tone curve owns it.
pub(crate) struct HueSat<'a> {
table: &'a HsvTable,
to: [[f32; 3]; 3],
from: [[f32; 3]; 3],
}
impl<'a> HueSat<'a> {
pub fn new(table: &'a HsvTable) -> Option<HueSat<'a>> {
let to = to_prophoto();
Some(HueSat { table, to: to.to_f32(), from: to.inverse()?.to_f32() })
}
#[inline]
pub fn apply(&self, rgb: [f32; 3]) -> [f32; 3] {
let mul = |m: &[[f32; 3]; 3], v: [f32; 3]| -> [f32; 3] { std::array::from_fn(|i| m[i][0] * v[0] + m[i][1] * v[1] + m[i][2] * v[2]) };
let out = mul(&self.from, self.table.apply(mul(&self.to, rgb)));
let (before, after) = (luminance_2020(rgb), luminance_2020(out));
if before > 0.0 && after > 0.0 && out.iter().all(|v| v.is_finite()) { out.map(|v| v * before / after) } else { rgb }
}
}
/// HSV hue (degrees) and saturation; `None` for black or non-finite colours.
fn hue_saturation(p: [f64; 3]) -> Option<(f64, f64)> {
let max = p[0].max(p[1]).max(p[2]);
let min = p[0].min(p[1]).min(p[2]);
if !(max.is_finite() && min.is_finite()) || max <= 0.0 {
return None;
}
let d = max - min;
if d <= 0.0 {
return Some((0.0, 0.0));
}
let h = if max == p[0] {
((p[1] - p[2]) / d).rem_euclid(6.0)
} else if max == p[1] {
(p[2] - p[0]) / d + 2.0
} else {
(p[0] - p[1]) / d + 4.0
};
Some((h * 60.0, d / max))
}
/// Table resolution: 5° hue steps (a coarser grid blurred the lime shirt into neighbouring browns
/// that want the opposite shift), saturation 0, 0.25 … 1. Value is not an axis: tone is fitted separately.
const TABLE_HUES: usize = 72;
const TABLE_SATS: usize = 5;
/// Value levels (sRGB-encoded, as DNG tables with encoding 1): cameras turn dark yellows toward
/// orange but bright yellow-greens toward green, which one shift per hue averages away.
const TABLE_VALS: usize = 5;
/// Kernel widths around each table node (hue in degrees, saturation, encoded value).
const KERNEL_HUE: f64 = 2.5;
const KERNEL_SAT: f64 = 0.15;
const KERNEL_VAL: f64 = 0.12;
/// Kernel weight at which a node keeps half of its estimate; sparse nodes shrink to identity.
const SHRINK_WEIGHT: f64 = 5.0;
/// Fit hue shifts and saturation scales of the training pairs left after `matrix`, per (hue,
/// saturation) node, kernel-weighted and shrunk toward identity where the photo has few samples.
/// Saturation 0 stays identity, so neutrals are never tinted.
fn fit_hue_sat(pairs: &[([f64; 3], [f64; 3])], matrix: &Mat3) -> Option<HsvTable> {
let to = to_prophoto();
let samples: Vec<(f64, f64, f64, f64, f64)> = pairs
.iter()
.enumerate()
.filter(|(i, _)| i % 3 != 0)
.filter_map(|(_, (x, y))| {
let p = to.apply(matrix.apply(*x));
let (hp, sp) = hue_saturation(p)?;
let value = f64::from(lightcraft_color::transfer::linear_to_srgb(p[0].max(p[1]).max(p[2]).min(1.0) as f32));
let (ht, st) = hue_saturation(to.apply(*y))?;
// hue is meaningless near neutral
if sp < 0.08 || st < 0.02 {
return None;
}
let shift = ((ht - hp + 180.0).rem_euclid(360.0) - 180.0).clamp(-30.0, 30.0);
let log_scale = (st / sp).ln().clamp(-0.7, 0.7);
Some((hp, sp.min(1.0), value, shift, log_scale))
})
.collect();
if samples.len() < 64 {
return None;
}
// Samples by table hue step: a node only looks at hues within its kernel's reach.
let step = 360.0 / TABLE_HUES as f64;
let mut by_hue: Vec<Vec<(f64, f64, f64, f64, f64)>> = vec![Vec::new(); TABLE_HUES];
for sample in samples {
if let Some(bin) = by_hue.get_mut(((sample.0.rem_euclid(360.0) / step) as usize).min(TABLE_HUES - 1)) {
bin.push(sample);
}
}
let reach = (4.0 * KERNEL_HUE / step).ceil() as usize + 1;
use rayon::prelude::*;
let data: Vec<[f32; 3]> = (0..TABLE_VALS * TABLE_HUES * TABLE_SATS)
.into_par_iter()
.map(|index| {
let (v, h, s) = (index / (TABLE_HUES * TABLE_SATS), index / TABLE_SATS % TABLE_HUES, index % TABLE_SATS);
if s == 0 {
return [0.0, 1.0, 1.0];
}
let (val, hue, sat) = (v as f64 / (TABLE_VALS - 1) as f64, h as f64 * step, s as f64 / (TABLE_SATS - 1) as f64);
let (mut weight, mut shift, mut log_scale) = (0.0, 0.0, 0.0);
for offset in 0..=2 * reach {
let Some(bin) = by_hue.get((h + TABLE_HUES * 2 + offset - reach) % TABLE_HUES) else { continue };
for &(hp, sp, vp, dh, ls) in bin {
let dhue = (hp - hue + 180.0).rem_euclid(360.0) - 180.0;
if dhue.abs() > 4.0 * KERNEL_HUE {
continue;
}
let d2 = (dhue / KERNEL_HUE).powi(2) + ((sp - sat) / KERNEL_SAT).powi(2) + ((vp - val) / KERNEL_VAL).powi(2);
let k = (-0.5 * d2).exp() * sp;
weight += k;
shift += k * dh;
log_scale += k * ls;
}
}
if weight <= 0.0 {
return [0.0, 1.0, 1.0];
}
let shrink = 1.0 / (weight + SHRINK_WEIGHT);
[(shift * shrink) as f32, (log_scale * shrink).exp() as f32, 1.0]
})
.collect();
data.iter().all(|e| e.iter().all(|v| v.is_finite())).then_some(HsvTable {
hue_divisions: TABLE_HUES,
sat_divisions: TABLE_SATS,
val_divisions: TABLE_VALS,
data,
srgb_value: true,
})
}
/// Kernel width of the chroma curve's nodes (display luminance) and the weight at which a node
/// keeps half of its estimate.
const CHROMA_KERNEL: f64 = 0.08;
const CHROMA_SHRINK: f64 = 2.0;
/// The look's tone curve with a chroma-by-display-luminance curve fitted to `pairs` (two thirds
/// train, one third held out), when that lowers the held-out error. Colourfulness is compared as
/// the distance from neutral of luminance-normalised RGB.
fn fit_chroma(pairs: &[([f64; 3], [f64; 3])], look: &CameraLook) -> Option<CameraTone> {
let correction = look.hue_sat.as_ref().and_then(HueSat::new);
let scene = |x: &[f64; 3]| {
let p = look.matrix.apply(*x);
correction.as_ref().map_or(p, |c| c.apply(p.map(|v| v as f32)).map(f64::from))
};
let tone = ToneMap::camera(&look.tone, 0.0, 0.0, 0.0);
let predict = |x: &[f64; 3]| displayed(scene(x), &tone);
let chroma = |p: [f64; 3]| {
let y = luma(p);
(y > 0.0).then(|| p.iter().map(|v| (v / y - 1.0).powi(2)).sum::<f64>().sqrt())
};
let samples: Vec<(f64, f64, f64)> = pairs
.iter()
.enumerate()
.filter(|(i, _)| i % 3 != 0)
.filter_map(|(_, (x, y))| {
let p = predict(x);
let (cp, cy) = (chroma(p)?, chroma(*y)?);
(cp > 0.05).then(|| (luma(p), cp, (cy.max(1e-3) / cp).ln().clamp(0.05f64.ln(), 2.5f64.ln())))
})
.collect();
if samples.len() < 64 {
return None;
}
let mut curve = [1.0f32; lightcraft_pipeline::tone::CHROMA_N];
for (j, node) in curve.iter_mut().enumerate() {
let at = j as f64 / (lightcraft_pipeline::tone::CHROMA_N - 1) as f64;
let (mut weight, mut sum) = (0.0, 0.0);
for &(o, cp, log_ratio) in &samples {
let k = (-0.5 * ((o - at) / CHROMA_KERNEL).powi(2)).exp() * cp;
weight += k;
sum += k * log_ratio;
}
*node = (sum / (weight + CHROMA_SHRINK)).exp() as f32;
}
let fitted = look.tone.with_chroma(curve)?;
let with = ToneMap::camera(&fitted, 0.0, 0.0, 0.0);
let error = |map: &ToneMap| -> f64 {
pairs
.iter()
.step_by(3)
.map(|(x, y)| {
let p = displayed(scene(x), map);
(0..3).map(|c| (p[c] - y[c]).powi(2)).sum::<f64>()
})
.sum()
};
let (before, after) = (error(&tone), error(&with));
if lightcraft_pipeline::profiling() {
eprintln!("[profile] ARW chroma curve {curve:?}: held-out error {before:.4} -> {after:.4}");
}
(after.is_finite() && after < before).then_some(fitted)
}
fn median(values: &mut [f64]) -> Option<f64> {
@@ -261,6 +567,169 @@ mod tests {
assert!(fit_pairs(&sensor, &reference).is_some());
}
/// A camera that rotates saturated yellow-greens toward green (like Sony's "Standard" on a lime
/// shirt) can't be followed by a matrix alone: the fitted hue/saturation table must close most
/// of the gap, keep neutrals neutral and keep luminance.
#[test]
fn hue_table_follows_a_hue_dependent_camera_rendering() {
let to = to_prophoto();
let from = to.inverse().unwrap();
let mut sensor = Rgb32f::new(96, 64);
let mut reference = sensor.clone();
let mut lime = Vec::new();
for (i, (src, dst)) in sensor.data.iter_mut().zip(&mut reference.data).enumerate() {
let ev = 0.05 + (i % 29) as f32 * 0.02;
// mostly greys and mild colours, plus a patch of saturated yellow-green
*src = if i % 7 == 0 {
[ev * 0.75, ev, ev * 0.2]
} else {
[ev * (0.85 + (i % 11) as f32 * 0.03), ev, ev * (0.85 + (i % 13) as f32 * 0.025)]
};
let (h, s) = hue_saturation(to.apply(src.map(f64::from))).unwrap();
// the camera turns hues in 50°..110° (ProPhoto) by up to +15° and boosts their saturation
let k = (1.0 - ((h - 80.0) / 30.0).powi(2)).max(0.0) * s.min(1.0);
let mut p = to.apply(src.map(f64::from)).map(|v| v as f32);
let table = HsvTable {
hue_divisions: 1,
sat_divisions: 2,
val_divisions: 1,
data: vec![[0.0, 1.0, 1.0], [15.0 * k as f32, 1.0 + 0.3 * k as f32, 1.0]],
srgb_value: false,
};
p = table.apply(p);
let p = from.apply(p.map(f64::from)).map(|v| v as f32);
let scale = luminance_2020(*src) / luminance_2020(p);
let p = p.map(|v| v * scale);
let y = luminance_2020(p);
*dst = p.map(|v| v * (1.0 - (-2.5 * y).exp()) / y);
if i % 7 == 0 {
lime.push(i);
}
}
let fit = fit_pairs(&sensor, &reference).unwrap();
let table = fit.hue_sat.as_ref().expect("a hue/saturation table is fitted");
let hue_sat = HueSat::new(table).unwrap();
let hue_error = |with_table: bool| {
lime.iter()
.map(|&i| {
let p = fit.matrix.apply(sensor.data[i].map(f64::from)).map(|v| v as f32);
let p = if with_table { hue_sat.apply(p) } else { p };
let (h, _) = hue_saturation(to.apply(p.map(f64::from))).unwrap();
let (t, _) = hue_saturation(to.apply(reference.data[i].map(f64::from))).unwrap();
((h - t + 180.0).rem_euclid(360.0) - 180.0).abs()
})
.sum::<f64>()
/ lime.len() as f64
};
let (before, after) = (hue_error(false), hue_error(true));
assert!(after < before * 0.5, "lime hue error {before:.2}° -> {after:.2}°");
// neutrals pass through and luminance is kept
let grey = [0.3, 0.3, 0.3];
assert!(hue_sat.apply(grey).iter().all(|v| (v - 0.3).abs() < 1e-4), "{:?}", hue_sat.apply(grey));
let green = [0.2, 0.4, 0.05];
assert!((luminance_2020(hue_sat.apply(green)) - luminance_2020(green)).abs() < 1e-5);
}
/// A camera that saturates shadows and bleaches highlights toward white (a per-channel curve),
/// which a luminance tone curve can't: the fitted chroma curve must follow it, so bright
/// warm-tinted rock renders white instead of cream.
#[test]
fn chroma_curve_follows_highlight_bleaching() {
let known = Mat3([[1.8, -0.4, -0.1], [-0.2, 1.5, -0.1], [-0.05, -0.3, 1.7]]);
let camera_chroma = |o: f32| 1.3 - 1.1 * o;
let mut sensor = Rgb32f::new(96, 64);
let mut reference = sensor.clone();
for (i, (src, dst)) in sensor.data.iter_mut().zip(&mut reference.data).enumerate() {
let ev = 0.02 + (i % 37) as f32 * 0.03;
*src = [ev * (0.75 + (i % 11) as f32 * 0.05), ev, ev * (0.7 + (i % 13) as f32 * 0.05)];
let p = known.apply_f32(*src);
let y = luminance_2020(p);
let o = 1.0 - (-2.5 * y).exp();
let k = camera_chroma(o);
*dst = p.map(|v| (o + (v * o / y - o) * k).clamp(0.0, 0.999));
}
let fit = fit_pairs(&sensor, &reference).unwrap();
// relative to the matrix, which already carries the average colourfulness
let chroma = fit.tone.chroma();
assert!(chroma[6] < 0.5 * chroma[1], "highlights bleach relative to shadows: {chroma:?}");
// and the rendered highlights land on the camera's, far closer than without the curve
let with = ToneMap::camera(&fit.tone, 0.0, 0.0, 0.0);
let without = ToneMap::camera(&fit.tone.with_chroma([1.0; lightcraft_pipeline::tone::CHROMA_N]).unwrap(), 0.0, 0.0, 0.0);
let highlight_error = |map: &ToneMap| -> f64 {
sensor
.data
.iter()
.zip(&reference.data)
.filter(|(_, y)| luminance_2020(**y) > 0.7)
.map(|(x, y)| {
let p = displayed(fit.matrix.apply(x.map(f64::from)), map);
(0..3).map(|c| (p[c] - f64::from(y[c])).powi(2)).sum::<f64>()
})
.sum()
};
let (a, b) = (highlight_error(&without), highlight_error(&with));
assert!(b < 0.5 * a, "highlight error {a:.4} -> {b:.4}");
}
/// Pooled pairs of several "photos" recover the camera's matrix, and a photo given that
/// colour model (a camera profile) keeps it, fitting only its own tone.
#[test]
fn profile_colour_is_pooled_and_then_used_as_given() {
let known = Mat3([[1.8, -0.4, -0.1], [-0.2, 1.5, -0.1], [-0.05, -0.3, 1.7]]);
let photo = |seed: usize, strength: f32| {
let mut sensor = Rgb32f::new(64, 48);
let mut reference = sensor.clone();
for (i, (src, dst)) in sensor.data.iter_mut().zip(&mut reference.data).enumerate() {
let i = i + seed * 7;
let ev = 0.04 + (i % 23) as f32 * 0.02;
*src = [ev * (0.7 + (i % 11) as f32 * 0.05), ev, ev * (0.7 + (i % 13) as f32 * 0.05)];
let p = known.apply_f32(*src);
let y = luminance_2020(p);
// each photo has its own tone (DRO, picture style)
*dst = p.map(|v| v * (1.0 - (-strength * y).exp()) / y);
}
(sensor, reference)
};
let mut pool = Vec::new();
for seed in 0..4 {
let (sensor, reference) = photo(seed, 2.0 + seed as f32 * 0.5);
pool.extend(collect_pairs(&sensor, &reference).unwrap().0);
}
let (matrix, _) = fit_profile(&pool).unwrap();
// the camera's colours (luminance-normalised: the tone curve sets brightness)
for x in [[0.8, 1.0, 0.75], [1.1, 1.0, 0.8], [0.75, 1.0, 1.2], [1.0; 3]] {
let (a, b) = (matrix.apply(x), known.apply(x));
let (la, lb) = (luma(a), luma(b));
assert!((0..3).all(|c| (a[c] / la - b[c] / lb).abs() < 0.01), "{x:?}: {a:?} vs {b:?}");
}
let (sensor, reference) = photo(9, 3.5);
let look = fit_pairs_with(&sensor, &reference, Some((matrix, None))).unwrap();
assert_eq!(look.matrix, matrix);
assert!(look.hue_sat.is_none());
let tone = ToneMap::camera(&look.tone, 0.0, 0.0, 0.0);
let error = sensor
.data
.iter()
.zip(&reference.data)
.map(|(x, y)| {
let p = displayed(look.matrix.apply(x.map(f64::from)), &tone);
(0..3).map(|c| (p[c] - f64::from(y[c])).powi(2)).sum::<f64>() / 3.0
})
.sum::<f64>()
/ sensor.data.len() as f64;
assert!(error.sqrt() < 0.02, "this photo's own tone is followed: RMS {}", error.sqrt());
}
#[test]
fn hue_sat_passes_black_and_non_finite_pixels_through() {
let table = HsvTable { hue_divisions: 4, sat_divisions: 2, val_divisions: 1, data: vec![[10.0, 1.5, 1.0]; 8], srgb_value: false };
let hue_sat = HueSat::new(&table).unwrap();
assert_eq!(hue_sat.apply([0.0; 3]), [0.0; 3]);
let nan = hue_sat.apply([f32::NAN, 0.2, 0.1]);
assert!(nan[0].is_nan() && nan[1] == 0.2 && nan[2] == 0.1);
assert!(fit_hue_sat(&[], &Mat3::IDENTITY).is_none(), "too few samples");
}
#[test]
fn sony_and_nikon_raws_get_a_file_local_look() {
assert!([RawFormat::Arw, RawFormat::Nef, RawFormat::Nrw].into_iter().all(file_local_look));
+310
View File
@@ -0,0 +1,310 @@
//! Camera colour profiles of our own, pooled from many photos of one camera model.
//!
//! A raw file without colour matrices (Sony ARW, Nikon NEF) gets its look fitted to its own embedded camera
//! JPEG (`camera_preview`), but one photo shows too little of some colours: a lime shirt covering a
//! few dozen proxy pixels next to a hillside of foliage at the same hue. `lightcraft-cli calibrate`
//! pools the colour pairs of many photos per model and fits one matrix and hue/saturation/value
//! table; photos of that model then only fit their tone and chroma curves.
//!
//! Profiles are JSON files (`<model>.json`) in [`dir`]: `LIGHTCRAFT_CAMERA_PROFILES`, else
//! `<config>/camera-profiles`; a local profile replaces the one built in ([`BUNDLED`], from
//! `assets/camera-profiles/`). They are read once per process; a damaged or hostile file is
//! ignored with a warning. They hold aggregate colour statistics only, never image content.
use lightcraft_color::Mat3;
use lightcraft_raw::profile::HsvTable;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, OnceLock, PoisonError};
/// Format version of the profile files.
const VERSION: u32 = 1;
/// Largest profile file read (a 5 × 72 × 5 table is ~60 KB of JSON).
const MAX_FILE: u64 = 4 << 20;
/// Profiles built into LightCraft (`assets/camera-profiles/`, see `assets/ATTRIBUTION.md`):
/// `(model, JSON)`.
pub const BUNDLED: &[(&str, &str)] = &[("ILCE-7M4", include_str!("../../../assets/camera-profiles/ILCE-7M4.json"))];
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct CameraProfile {
pub version: u32,
/// The camera model as the files report it (Exif `Model`), e.g. `ILCE-7M4`.
pub model: String,
/// Photos and colour pairs the profile was fitted on.
pub files: usize,
pub samples: usize,
/// White-balanced camera RGB (with the baseline exposure) → linear Rec.2020, rows.
matrix: [[f64; 3]; 3],
/// Hue/saturation/value correction after `matrix` (linear ProPhoto RGB).
pub hue_sat: Option<HsvTable>,
}
impl CameraProfile {
pub fn new(model: &str, files: usize, samples: usize, matrix: Mat3, hue_sat: Option<HsvTable>) -> CameraProfile {
CameraProfile { version: VERSION, model: model.to_owned(), files, samples, matrix: matrix.0, hue_sat }
}
pub fn matrix(&self) -> Mat3 {
Mat3(self.matrix)
}
/// Whether the data is usable (bounded matrix, table shape matching its data).
fn valid(&self) -> bool {
let matrix = self.matrix.iter().flatten().all(|v| v.is_finite() && v.abs() < 8.0) && Mat3(self.matrix).inverse().is_some();
let table = self.hue_sat.as_ref().is_none_or(|t| {
let dims = (1..=4096).contains(&t.hue_divisions) && (2..=4096).contains(&t.sat_divisions) && (1..=4096).contains(&t.val_divisions);
let len = t.hue_divisions.checked_mul(t.sat_divisions).and_then(|n| n.checked_mul(t.val_divisions));
dims && len == Some(t.data.len()) && t.data.iter().flatten().all(|v| v.is_finite()) && t.data.iter().all(|e| e[1] >= 0.0 && e[2] >= 0.0)
});
self.version == VERSION && !self.model.is_empty() && matrix && table
}
}
/// LightCraft's configuration folder (settings, GPU marker, camera profiles).
pub fn config_dir() -> Option<PathBuf> {
if cfg!(target_os = "macos") {
std::env::var_os("HOME").map(|h| PathBuf::from(h).join("Library/Application Support/LightCraft"))
} else if cfg!(windows) {
std::env::var_os("APPDATA").map(|a| PathBuf::from(a).join("LightCraft"))
} else {
std::env::var_os("XDG_CONFIG_HOME")
.map(PathBuf::from)
.or_else(|| std::env::var_os("HOME").map(|h| PathBuf::from(h).join(".config")))
.map(|c| c.join("lightcraft"))
}
}
/// Where camera profiles are read from and written to.
pub fn dir() -> Option<PathBuf> {
std::env::var_os("LIGHTCRAFT_CAMERA_PROFILES").map(PathBuf::from).or_else(|| config_dir().map(|d| d.join("camera-profiles")))
}
/// File name of `model`'s profile: letters, digits, `-` and `_` kept, anything else `_`.
pub fn file_name(model: &str) -> String {
let name: String = model.trim().chars().map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '_' { c } else { '_' }).collect();
format!("{name}.json")
}
/// Read and validate a profile file.
pub fn load(path: &Path) -> Result<CameraProfile, String> {
let size = std::fs::metadata(path).map_err(|e| format!("{}: {e}", path.display()))?.len();
if size > MAX_FILE {
return Err(format!("{}: {size} bytes is too large for a camera profile", path.display()));
}
let bytes = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?;
parse(&bytes, &path.display().to_string())
}
/// Parse and validate a profile's JSON (`origin` names it in errors).
fn parse(json: &[u8], origin: &str) -> Result<CameraProfile, String> {
let profile: CameraProfile = serde_json::from_slice(json).map_err(|e| format!("{origin}: {e}"))?;
if !profile.valid() {
return Err(format!("{origin}: not a usable camera profile (version {}, model {:?})", profile.version, profile.model));
}
Ok(profile)
}
/// The built-in profile for `model`, if any.
fn bundled(model: &str) -> Option<CameraProfile> {
let (_, json) = BUNDLED.iter().find(|(m, _)| *m == model)?;
parse(json.as_bytes(), &format!("built-in profile {model}")).inspect_err(|e| eprintln!("lightcraft: ignoring camera profile {e}")).ok()
}
/// Write `profile` to `dir` (created if needed) as `<model>.json`; returns the path.
pub fn save(profile: &CameraProfile, dir: &Path) -> Result<PathBuf, String> {
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let path = dir.join(file_name(&profile.model));
let json = serde_json::to_vec_pretty(profile).map_err(|e| e.to_string())?;
let tmp = path.with_extension("json.tmp");
std::fs::write(&tmp, json).map_err(|e| format!("{}: {e}", tmp.display()))?;
std::fs::rename(&tmp, &path).map_err(|e| format!("{}: {e}", path.display()))?;
Ok(path)
}
fn cache() -> &'static Mutex<HashMap<String, Option<Arc<CameraProfile>>>> {
static CACHE: OnceLock<Mutex<HashMap<String, Option<Arc<CameraProfile>>>>> = OnceLock::new();
CACHE.get_or_init(Default::default)
}
/// The profile for camera `model`: from [`dir`], else built in; read once per process.
pub fn get(model: &str) -> Option<Arc<CameraProfile>> {
let model = model.trim();
if model.is_empty() {
return None;
}
let mut cache = cache().lock().unwrap_or_else(PoisonError::into_inner);
if let Some(hit) = cache.get(model) {
return hit.clone();
}
let path = dir().map(|d| d.join(file_name(model)));
let profile = match path {
Some(path) if path.is_file() => match load(&path) {
Ok(p) if p.model.trim() == model => Some(Arc::new(p)),
Ok(p) => {
eprintln!("lightcraft: ignoring camera profile {}: it is for {:?}, not {model:?}", path.display(), p.model);
None
}
Err(e) => {
eprintln!("lightcraft: ignoring camera profile {e}");
None
}
},
_ => None,
};
let profile = profile.or_else(|| bundled(model).map(Arc::new));
cache.insert(model.to_owned(), profile.clone());
profile
}
/// Changes when the profiles folder's contents change (per process): part of the render cache
/// keys, so thumbnails rendered before a profile existed are not reused after.
pub fn cache_key() -> u64 {
static KEY: OnceLock<u64> = OnceLock::new();
*KEY.get_or_init(|| {
let entries = dir().and_then(|d| std::fs::read_dir(d).ok());
let mut files: Vec<(String, u64, u64)> = entries
.into_iter()
.flatten()
.flatten()
.filter(|e| e.path().extension().is_some_and(|x| x == "json"))
.filter_map(|e| {
let meta = e.metadata().ok()?;
let modified = meta.modified().ok()?.duration_since(std::time::UNIX_EPOCH).ok()?.as_secs();
Some((e.file_name().to_string_lossy().into_owned(), meta.len(), modified))
})
.collect();
files.sort();
let mut h = lightcraft_preview::Hasher128::new();
for (name, len, modified) in &files {
h.str(name).u64(*len).u64(*modified);
}
for (model, json) in BUNDLED {
h.str(model).str(json);
}
h.finish().0 as u64
})
}
/// Per camera model: the photos read and their pooled colour pairs.
#[derive(Default)]
pub struct Pool {
models: HashMap<String, (usize, Vec<([f64; 3], [f64; 3])>)>,
}
/// Most colour pairs kept per photo, so a few busy photos can't dominate a profile.
const PAIRS_PER_FILE: usize = 4000;
impl Pool {
/// Add one raw file's colour pairs. `Ok(None)` when the file can't contribute (not an ARW or
/// NEF without colour matrices, no usable camera JPEG, too little colour).
pub fn add(&mut self, bytes: &[u8]) -> Result<Option<String>, String> {
let raw = lightcraft_raw::decode(bytes).map_err(|e| e.to_string())?;
let Some(model) = raw.metadata.model.as_deref().map(str::trim).filter(|m| !m.is_empty()) else { return Ok(None) };
let Some(pairs) = crate::camera_preview::profile_pairs(&raw, bytes) else { return Ok(None) };
let step = pairs.len().div_ceil(PAIRS_PER_FILE).max(1);
let entry = self.models.entry(model.to_owned()).or_default();
entry.0 += 1;
entry.1.extend(pairs.into_iter().step_by(step));
Ok(Some(model.to_owned()))
}
/// Fit a profile per model with at least `min_files` photos.
pub fn fit(&self, min_files: usize) -> Vec<Result<CameraProfile, String>> {
let mut models: Vec<_> = self.models.iter().collect();
models.sort_by(|a, b| a.0.cmp(b.0));
models
.into_iter()
.filter(|(_, (files, _))| *files >= min_files)
.map(|(model, (files, pairs))| {
let (matrix, hue_sat) = crate::camera_preview::fit_profile(pairs).ok_or_else(|| format!("{model}: no usable colour fit"))?;
Ok(CameraProfile::new(model, *files, pairs.len(), matrix, hue_sat))
})
.collect()
}
/// Add another pool's photos and pairs (pools filled on separate threads).
pub fn merge(&mut self, other: Pool) {
for (model, (files, pairs)) in other.models {
let entry = self.models.entry(model).or_default();
entry.0 += files;
entry.1.extend(pairs);
}
}
/// Photos read per model.
pub fn files(&self) -> Vec<(String, usize)> {
let mut v: Vec<_> = self.models.iter().map(|(m, (n, _))| (m.clone(), *n)).collect();
v.sort();
v
}
}
#[cfg(test)]
mod tests {
use super::*;
fn profile() -> CameraProfile {
let table = HsvTable { hue_divisions: 4, sat_divisions: 2, val_divisions: 1, data: vec![[5.0, 1.2, 1.0]; 8], srgb_value: false };
CameraProfile::new("ILCE-7M4", 12, 3456, Mat3([[1.6, -0.5, -0.1], [-0.2, 1.3, -0.1], [-0.1, -0.2, 1.3]]), Some(table))
}
#[test]
fn profiles_round_trip_and_reject_bad_files() {
let dir = std::env::temp_dir().join(format!("lc-camera-profiles-{}", std::process::id()));
let path = save(&profile(), &dir).unwrap();
assert_eq!(path.file_name().unwrap(), "ILCE-7M4.json");
assert_eq!(load(&path).unwrap(), profile());
let bad = |p: CameraProfile| {
let path = dir.join("bad.json");
std::fs::write(&path, serde_json::to_vec(&p).unwrap()).unwrap();
load(&path).is_err()
};
let mut p = profile();
p.matrix[0][0] = f64::NAN;
assert!(bad(p), "non-finite matrix");
let mut p = profile();
p.matrix = [[0.0; 3]; 3];
assert!(bad(p), "singular matrix");
let mut p = profile();
if let Some(t) = p.hue_sat.as_mut() {
t.data.truncate(3);
}
assert!(bad(p), "table data doesn't match its shape");
let mut p = profile();
p.version = 99;
assert!(bad(p), "unknown version");
std::fs::write(dir.join("junk.json"), b"{not json").unwrap();
assert!(load(&dir.join("junk.json")).is_err());
assert!(load(&dir.join("missing.json")).is_err());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn bundled_profiles_are_valid_and_named_after_their_model() {
assert!(!BUNDLED.is_empty());
for (model, json) in BUNDLED {
let p = parse(json.as_bytes(), model).unwrap();
assert_eq!(p.model, *model);
assert!(p.files >= 5 && p.hue_sat.is_some(), "{model}: {} photos", p.files);
assert_eq!(bundled(model), Some(p));
}
assert!(bundled("No Such Camera").is_none());
}
#[test]
fn file_names_are_safe() {
assert_eq!(file_name("ILCE-7M4"), "ILCE-7M4.json");
assert_eq!(file_name(" DSC-RX100M3 "), "DSC-RX100M3.json");
assert_eq!(file_name("../../etc/passwd"), "______etc_passwd.json");
assert_eq!(file_name("Ω 1/2"), "__1_2.json");
}
#[test]
fn pool_skips_files_that_cannot_contribute() {
let mut pool = Pool::default();
assert!(pool.add(b"not a raw file").is_err());
assert!(pool.fit(1).is_empty());
assert!(pool.files().is_empty());
}
}
+61 -1
View File
@@ -263,7 +263,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Filter",
[],
None,
"partial Filter: {text?, rating?, ratingOp?: atLeast|exactly|atMost, flag?: pick|reject|none|null, label?, kind?, merged?: hdr|panorama|hdrPanorama|any, edited?, date?, keyword?, camera?}",
"partial Filter: {text?, rating?, ratingOp?: atLeast|exactly|atMost, flag?: pick|reject|none|null, label?, kind?, merged?: hdr|panorama|hdrPanorama|any, edited?, date?, keyword?, person?, camera?}",
always,
|s, p| {
let mut v = serde_json::to_value(&s.filter).unwrap_or_default();
@@ -446,6 +446,66 @@ pub fn specs() -> Vec<CommandSpec> {
cmd!("photo.rotateRight", "Rotate Right", ["Photo"], Some("Cmd+]"), "{ids?}", has_selection, |s, p| rotate(s, p, true)),
cmd!("photo.flipHorizontal", "Flip Horizontal", ["Photo"], None, "{ids?}", has_selection, |s, p| flip(s, p, true)),
cmd!("photo.flipVertical", "Flip Vertical", ["Photo"], None, "{ids?}", has_selection, |s, p| flip(s, p, false)),
// ---- face / pet regions
cmd!(
"photo.removeRegion",
"Remove Face Box",
[],
None,
"{id?, index} — remove one face / pet region (by its position in the photo's regions) from the photo in the catalog; undoable. The XMP sidecar is never rewritten for this, even with auto-write on, so reading the metadata from the file brings the region back",
always,
|s, p| {
let id =
p.get("id").and_then(Value::as_u64).map(PhotoId).or_else(|| s.active()).ok_or_else(|| bad("photo.removeRegion", "no photo"))?;
let index = p
.get("index")
.and_then(Value::as_u64)
.and_then(|i| usize::try_from(i).ok())
.ok_or_else(|| bad("photo.removeRegion", "missing or invalid `index`"))?;
let mut meta = s.catalog.photo(id).ok_or_else(|| bad("photo.removeRegion", "no such photo"))?.meta.clone();
if index >= meta.regions.len() {
return Err(bad("photo.removeRegion", "no such region"));
}
let gone = meta.regions.remove(index);
s.commit("Remove Face Box", Op::SetMeta { id, meta: Box::new(meta) })?;
s.skip_auto_write = true;
Ok(json!({"removed": gone.name}))
}
),
cmd!(
"photo.setRegion",
"Resize Face Box",
[],
None,
"{id?, index, rect: {x0, y0, x1, y1}} — set one face / pet region's box (normalized, in the photo's upright frame; clamped to the photo, at least 0.5 % each way) in the catalog; undoable. Like photo.removeRegion it never rewrites the XMP sidecar",
always,
|s, p| {
const C: &str = "photo.setRegion";
// a drag is one undo step however many frames it took: the caller commits once, on release
let id = p.get("id").and_then(Value::as_u64).map(PhotoId).or_else(|| s.active()).ok_or_else(|| bad(C, "no photo"))?;
let index = p
.get("index")
.and_then(Value::as_u64)
.and_then(|i| usize::try_from(i).ok())
.ok_or_else(|| bad(C, "missing or invalid `index`"))?;
let r = p.get("rect").ok_or_else(|| bad(C, "missing `rect`"))?;
let num = |k: &str| {
r.get(k).and_then(Value::as_f64).filter(|v| v.is_finite()).ok_or_else(|| bad(C, format!("`rect.{k}` must be a finite number")))
};
let (a, b, c, d) = (num("x0")?, num("y0")?, num("x1")?, num("y1")?);
let (x0, x1) = (a.min(c).clamp(0.0, 1.0), a.max(c).clamp(0.0, 1.0));
let (y0, y1) = (b.min(d).clamp(0.0, 1.0), b.max(d).clamp(0.0, 1.0));
if x1 - x0 < 0.005 || y1 - y0 < 0.005 {
return Err(bad(C, "the box would be too small"));
}
let mut meta = s.catalog.photo(id).ok_or_else(|| bad(C, "no such photo"))?.meta.clone();
let region = meta.regions.get_mut(index).ok_or_else(|| bad(C, "no such region"))?;
region.rect = lightcraft_geom::Rect { x0, y0, x1, y1 };
s.commit("Resize Face Box", Op::SetMeta { id, meta: Box::new(meta) })?;
s.skip_auto_write = true;
Ok(json!({"rect": {"x0": x0, "y0": y0, "x1": x1, "y1": y1}}))
}
),
// ---- delete / restore
cmd!("photo.delete", "Delete Photo", ["Photo"], Some("Delete"), "{ids?} — moves to Recently Deleted", has_selection, |s, p| {
let v = for_targets(s, p, "Delete", |id| Some(Op::SetDeleted { id, deleted: true }))?;
+271 -8
View File
@@ -5,6 +5,7 @@ use lightcraft_geom::Point;
use serde_json::{Value, json};
use super::{CommandSpec, bad, bool_or, cmd, f64_or, has_active, ok, point, str_param};
use crate::segment::{Click, MAX_CLICKS};
use crate::{Result, Session};
fn shape_from(kind: &str, p: &Value, c: &str) -> Result<MaskShape> {
@@ -38,10 +39,100 @@ fn shape_from(kind: &str, p: &Value, c: &str) -> Result<MaskShape> {
.unwrap_or_default();
MaskShape::ColorRange { samples, refine: f64_or(p, "refine", 50.0) }
}
other => return Err(bad(c, format!("unknown mask kind `{other}` (brush|linear|radial|sky|subject|background|luminanceRange|colorRange)"))),
"object" => MaskShape::Object { hint: points(p, "points"), exclude: points(p, "exclude"), seg: seg_param(p), detail: vec![], edge: 0.0 },
"prompt" => {
MaskShape::Prompt { text: str_param(p, "text").unwrap_or_default().trim().to_string(), seg: seg_param(p), detail: vec![], edge: 0.0 }
}
other => {
return Err(bad(
c,
format!("unknown mask kind `{other}` (brush|linear|radial|sky|subject|background|luminanceRange|colorRange|object|prompt)"),
));
}
})
}
/// `[[x, y], …]` normalized points under `key`.
fn points(p: &Value, key: &str) -> Vec<Point> {
p.get(key)
.and_then(Value::as_array)
.map(|a| a.iter().filter_map(|q| Some(Point::new(q.get(0)?.as_f64()?, q.get(1)?.as_f64()?))).collect())
.unwrap_or_default()
}
/// A segmentation passed in (`seg`: a stored or replayed AI mask; the model isn't needed).
fn seg_param(p: &Value) -> Option<lightcraft_develop::SegMask> {
p.get("seg").and_then(|v| serde_json::from_value(v.clone()).ok())
}
/// What a new AI shape still needs once it is added.
enum Later {
Nothing,
/// Background mode: these Object clicks are segmented on the worker.
Clicks(Vec<Point>, Vec<Point>),
/// Background mode: the Describe selection is computed first; the mask is added when found.
Text(String),
}
/// Compute the segmentation of a new AI shape (Object with clicks, Prompt) — or, in background
/// mode, say what to queue. An Object without clicks stays empty until the photo is clicked
/// (the photo is analyzed meanwhile); a Prompt that matches nothing is an error. Shapes given
/// with their `seg` need no model.
fn resolve_ai(s: &mut Session, shape: &mut MaskShape, c: &str) -> Result<Later> {
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
let background = s.segmenter.background;
match shape {
MaskShape::Object { seg: Some(_), .. } | MaskShape::Prompt { seg: Some(_), .. } => {}
MaskShape::Object { hint, exclude, .. } if !hint.is_empty() && background => {
s.segmenter.model_dir().map_err(ai_err)?;
return Ok(Later::Clicks(std::mem::take(hint), std::mem::take(exclude)));
}
MaskShape::Object { hint, exclude, seg, .. } if !hint.is_empty() => {
*seg = Some(s.segment_clicks(id, &clicks_of(hint, exclude)).map_err(ai_err)?);
}
MaskShape::Object { .. } => {
// the clicks come later: refuse an Object mask that could never be computed (no
// model) rather than leave an empty one; in the app, analyze the photo meanwhile
s.segmenter.model_dir().map_err(ai_err)?;
if background {
s.segment_prepare(id).map_err(ai_err)?;
}
}
MaskShape::Prompt { text, .. } if background => {
s.segmenter.model_dir().map_err(ai_err)?;
if text.trim().is_empty() {
return Err(ai_err("describe what to select"));
}
return Ok(Later::Text(text.clone()));
}
MaskShape::Prompt { text, seg, .. } => {
let found = s.segment_text(id, text).map_err(ai_err)?;
*seg = Some(found.ok_or_else(|| ai_err(format!("nothing matching “{text}” was found in this photo")))?);
}
_ => {}
}
Ok(Later::Nothing)
}
/// Queue what a new AI component (component `comp` of mask `mask`) still needs.
fn queue_later(s: &mut Session, later: Later, mask: u32, comp: usize, mut out: Value) -> Result<Value> {
if let Later::Clicks(hint, exclude) = later {
let id = s.active().ok_or_else(|| ai_err("no active photo"))?;
s.segment_clicks_later(id, mask, comp, hint, exclude).map_err(ai_err)?;
out["pending"] = json!(true);
}
Ok(out)
}
/// AI mask failures are shown as they are (not as "invalid parameters").
fn ai_err(msg: impl Into<String>) -> crate::EngineError {
crate::EngineError::Other(msg.into())
}
pub(crate) fn clicks_of(include: &[Point], exclude: &[Point]) -> Vec<Click> {
include.iter().map(|p| Click { at: *p, include: true }).chain(exclude.iter().map(|p| Click { at: *p, include: false })).collect()
}
fn masks_edit(s: &mut Session, c: &str, label: &str, f: impl FnOnce(&mut Vec<Mask>, &mut Option<u32>) -> Result<()>) -> Result<Value> {
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
let mut d = (*s.develop_of(id).unwrap_or_default()).clone();
@@ -67,14 +158,20 @@ pub fn specs() -> Vec<CommandSpec> {
"Create New Mask",
[],
None,
"{kind: brush|linear|radial|sky|subject|background|luminanceRange|colorRange, ...shape params (start/end, center/rx/ry/angle/feather, lo/hi…), name?}",
"{kind: brush|linear|radial|sky|subject|background|luminanceRange|colorRange|object|prompt, ...shape params (start/end, center/rx/ry/angle/feather, lo/hi…; object: points/exclude [[x,y],…]; prompt: text; object/prompt: seg? a stored segmentation, no model needed), name?} — AI kinds need the SAM 3 model; in the app a prompt returns {pending} and the mask appears when found",
has_active,
|s, p| {
let kind = str_param(p, "kind").unwrap_or("radial").to_string();
let shape = shape_from(&kind, p, "mask.add")?;
let mut shape = shape_from(&kind, p, "mask.add")?;
let later = resolve_ai(s, &mut shape, "mask.add")?;
let name = str_param(p, "name").map(str::to_string);
if let Later::Text(text) = &later {
let id = s.active().ok_or_else(|| bad("mask.add", "no active photo"))?;
s.segment_text_later(id, None, "add", name, text).map_err(ai_err)?;
return Ok(json!({"pending": true}));
}
let next = s.active().and_then(|id| s.develop_of(id)).map(|d| d.next_mask_id()).unwrap_or(1);
masks_edit(s, "mask.add", "Add Mask", |masks, active| {
let out = masks_edit(s, "mask.add", "Add Mask", |masks, active| {
masks.push(Mask {
id: next,
name: name.unwrap_or_else(|| format!("Mask {next}")),
@@ -83,7 +180,8 @@ pub fn specs() -> Vec<CommandSpec> {
});
*active = Some(next);
Ok(())
})
})?;
queue_later(s, later, next, 0, out)
}
),
cmd!(
@@ -95,16 +193,29 @@ pub fn specs() -> Vec<CommandSpec> {
has_active,
|s, p| {
let kind = str_param(p, "kind").unwrap_or("brush").to_string();
let shape = shape_from(&kind, p, "mask.addComponent")?;
let op: MaskOp =
serde_json::from_value(p.get("op").cloned().unwrap_or(json!("add"))).map_err(|e| bad("mask.addComponent", e.to_string()))?;
let active = s.active_mask;
let mid = mask_id(p, active, "mask.addComponent")?;
masks_edit(s, "mask.addComponent", "Edit Mask", |masks, _| {
let id = s.active().ok_or_else(|| bad("mask.addComponent", "no active photo"))?;
if !s.develop_of(id).is_some_and(|d| d.masks.iter().any(|m| m.id == mid)) {
return Err(bad("mask.addComponent", format!("no mask {mid}")));
}
let mut shape = shape_from(&kind, p, "mask.addComponent")?;
let later = resolve_ai(s, &mut shape, "mask.addComponent")?;
if let Later::Text(text) = &later {
let op = p.get("op").and_then(Value::as_str).unwrap_or("add");
s.segment_text_later(id, Some(mid), op, None, text).map_err(ai_err)?;
return Ok(json!({"pending": true}));
}
let mut comp = 0;
let out = masks_edit(s, "mask.addComponent", "Edit Mask", |masks, _| {
let i = find(masks, mid, "mask.addComponent")?;
masks[i].components.push(MaskComponent { name: None, op, invert: bool_or(p, "invert", false), shape });
comp = masks[i].components.len() - 1;
Ok(())
})
})?;
queue_later(s, later, mid, comp, out)
}
),
cmd!(
@@ -213,6 +324,158 @@ pub fn specs() -> Vec<CommandSpec> {
Ok(json!({"samples": out}))
}
),
cmd!(
"mask.objectPoint",
"Add Object Click",
[],
None,
"{x, y: normalized image point, exclude?: bool (⌥-click: leave this part out), id?: maskId} — a click on the selected mask's Object selection (its last Object component); SAM 3 re-segments the object → {include, exclude}",
has_active,
|s, p| {
let c = "mask.objectPoint";
let at = Point::new(super::f64_req(p, "x", c)?, super::f64_req(p, "y", c)?);
if !(0.0..=1.0).contains(&at.x) || !(0.0..=1.0).contains(&at.y) {
return Err(bad(c, "the point is outside the photo"));
}
let exclude = bool_or(p, "exclude", false);
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
let mid = mask_id(p, s.active_mask, c)?;
let d = s.develop_of(id).unwrap_or_default();
let m = d.masks.iter().find(|m| m.id == mid).ok_or_else(|| bad(c, format!("no mask {mid}")))?;
let k = m
.components
.iter()
.rposition(|x| matches!(x.shape, MaskShape::Object { .. }))
.ok_or_else(|| bad(c, "the mask has no Object selection (add one first)"))?;
let Some(MaskShape::Object { hint, exclude: ex, edge, .. }) = m.components.get(k).map(|x| &x.shape) else {
return Err(bad(c, "not an Object component"));
};
let (mut hint, mut ex, edge) = (hint.clone(), ex.clone(), *edge);
// clicks still on their way to the model count too
if let Some(pending) = s.segmenter.pending_clicks().filter(|q| (q.photo, q.mask, q.comp) == (id, mid, k)) {
(hint, ex) = (pending.hint.clone(), pending.exclude.clone());
}
if hint.len() + ex.len() >= MAX_CLICKS {
return Err(ai_err(format!("an Object selection takes at most {MAX_CLICKS} clicks; start a new one")));
}
if exclude {
ex.push(at)
} else {
hint.push(at)
}
if s.segmenter.background && !hint.is_empty() {
let out = json!({"include": hint.len(), "exclude": ex.len(), "pending": true});
s.segment_clicks_later(id, mid, k, hint, ex).map_err(ai_err)?;
return Ok(out);
}
let seg = if hint.is_empty() { None } else { Some(s.segment_clicks(id, &clicks_of(&hint, &ex)).map_err(ai_err)?) };
let out = json!({"include": hint.len(), "exclude": ex.len()});
masks_edit(s, c, "Object Mask", |masks, _| {
let i = find(masks, mid, c)?;
if let Some(comp) = masks[i].components.get_mut(k) {
comp.shape = MaskShape::Object { hint, exclude: ex, seg, detail: vec![], edge };
}
Ok(())
})?;
Ok(out)
}
),
cmd!(
"mask.refineDetail",
"Refine AI Mask Detail",
[],
None,
"{id?: maskId, component?} — a zoomed-in SAM 3 pass over an Object/Describe selection (default: the mask's last one): the photo around it is analyzed again at a higher resolution, in the background; the mask updates when it's done → {started}",
has_active,
|s, p| {
let c = "mask.refineDetail";
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
let mid = mask_id(p, s.active_mask, c)?;
let d = s.develop_of(id).unwrap_or_default();
let m = d.masks.iter().find(|m| m.id == mid).ok_or_else(|| bad(c, format!("no mask {mid}")))?;
let k = match p.get("component").and_then(Value::as_u64) {
Some(k) => k as usize,
None => m
.components
.iter()
.rposition(|x| matches!(x.shape, MaskShape::Object { .. } | MaskShape::Prompt { .. }))
.ok_or_else(|| bad(c, "the mask has no Object or Describe selection"))?,
};
let started = s.segment_detail(id, mid, k).map_err(ai_err)?;
Ok(json!({"started": started}))
}
),
cmd!(
query "segment.prepare",
"Prepare AI Masks",
[],
None,
"{} — load SAM 3 and analyze the active photo in the background, so Object clicks are instant → {busy}",
has_active,
|s, _| {
let c = "segment.prepare";
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
s.segment_prepare(id).map_err(ai_err)?;
Ok(json!({"busy": s.segmenter.busy()}))
}
),
cmd!(
query "segment.model.status",
"AI Mask Model Status",
[],
None,
"{} — whether this build has AI masks, whether the SAM 3 model is installed (and where), loaded, busy, and the download's progress → {available, installed, dir, loaded, busy, analyzing, sizeBytes, license, licenseUrl, mirrors, download: {running, done, total, file, error, finished}}",
super::always,
|s, _| {
let g = &s.segmenter;
Ok(json!({
"available": crate::segment::Segmenter::AVAILABLE,
"installed": g.installed(),
"dir": g.dir.as_ref().map(|d| d.display().to_string()),
"loaded": g.loaded(),
"busy": g.busy(),
"analyzing": g.analyzing(),
"sizeBytes": crate::segment::MODEL_BYTES,
"license": crate::segment::LICENSE_NAME,
"licenseUrl": crate::segment::LICENSE_URL,
"mirrors": g.mirrors().len(),
"download": g.download_status(),
}))
}
),
cmd!(
query "segment.model.download",
"Download AI Mask Model",
[],
None,
"{acknowledged: true} — download the SAM 3 model (about 3.4 GB, Meta's SAM License, not LightCraft's) in the background, from the configured mirrors; only after the user agreed to it. Watch segment.model.status; segment.model.cancel stops it (it resumes later) → {started, installed, downloading}",
super::always,
|s, p| {
let c = "segment.model.download";
if p.get("acknowledged").and_then(Value::as_bool) != Some(true) {
return Err(bad(
c,
format!(
"the SAM 3 model is a {:.1} GB download under Meta's {} ({}): pass `acknowledged: true` once the user has agreed to download it",
crate::segment::MODEL_BYTES as f64 / 1e9,
crate::segment::LICENSE_NAME,
crate::segment::LICENSE_URL
),
));
}
let started = s.segmenter.start_download().map_err(ai_err)?;
Ok(json!({"started": started, "installed": s.segmenter.installed(), "downloading": s.segmenter.download_status().running}))
}
),
cmd!(
query "segment.model.cancel",
"Cancel AI Mask Model Download",
[],
None,
"{} — stop the SAM 3 download (what has arrived is kept, and a new download resumes from it) → {cancelled}",
super::always,
|s, _| Ok(json!({"cancelled": s.segmenter.cancel_download()}))
),
cmd!(
"mask.refine",
"Refine Mask Edges",
+7 -6
View File
@@ -245,14 +245,15 @@ impl Default for Watermark {
/// Inter SemiBold (OFL, see assets/ATTRIBUTION.md).
static WATERMARK_FONT: &[u8] = include_bytes!("../../../assets/fonts/Inter-SemiBold.ttf");
/// The watermark faces: Inter first, then the craft-fonts Japanese faces (Mincho first, the
/// watermark's serif look; then any other Japanese face). Without craft-fonts that is Inter alone,
/// and Japanese characters draw as Inter's missing-glyph box.
/// The watermark faces: Inter first, then the craft-fonts CJK faces (Mincho first, the watermark's
/// serif look; then any other CJK face — a watermark has no UI language, so every CJK script is
/// offered). Without craft-fonts that is Inter alone, and CJK characters draw as Inter's
/// missing-glyph box.
fn watermark_fonts(craft: &'static [crate::fonts::CraftFont]) -> Vec<ab_glyph::FontRef<'static>> {
let mut japanese: Vec<_> = crate::fonts::japanese(craft).collect();
japanese.sort_by_key(|f| !f.is_mincho());
let mut cjk: Vec<_> = crate::fonts::cjk(craft).collect();
cjk.sort_by_key(|f| !f.is_mincho());
std::iter::once(WATERMARK_FONT)
.chain(japanese.into_iter().map(|f| f.bytes))
.chain(cjk.into_iter().map(|f| f.bytes))
.filter_map(|bytes| ab_glyph::FontRef::try_from_slice(bytes).ok())
.collect()
}
+10 -6
View File
@@ -38,6 +38,7 @@ fn meta_of(m: &lightcraft_meta::Metadata) -> (Meta, Option<String>) {
copyright_url: m.copyright_url.clone().unwrap_or_default(),
creator: m.artist.clone().unwrap_or_default(),
keywords: m.keywords.clone(),
regions: m.regions.clone(),
};
(meta, m.capture_time.as_ref().map(|d| d.to_iso()))
}
@@ -250,7 +251,8 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
stages.push(("transform", t0.elapsed()));
lightcraft_raw::highlight::reconstruct(&mut img, t.wb, HIGHLIGHT_CLIP);
stages.push(("highlights", t0.elapsed()));
let m = camera_look.map(|p| p.matrix.mul(&t.matrix)).unwrap_or(t.matrix).to_f32();
let m = camera_look.as_ref().map(|p| p.matrix.mul(&t.matrix)).unwrap_or(t.matrix).to_f32();
let hue_sat = camera_look.as_ref().and_then(|p| p.hue_sat.as_ref()).and_then(crate::camera_preview::HueSat::new);
let gain = 2f32.powf(t.baseline_exposure as f32);
let wb = t.wb;
// A DNG's own profile look (hue/saturation map, look table), DNG spec chapter 6.
@@ -262,10 +264,12 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
m[1][0] * c[0] + m[1][1] * c[1] + m[1][2] * c[2],
m[2][0] * c[0] + m[2][1] * c[1] + m[2][2] * c[2],
];
match &tables {
Some(tables) => tables.apply(rgb, gain).map(|v| v.max(0.0)),
None => rgb.map(|v| (v * gain).max(0.0)),
}
let rgb = match &tables {
Some(tables) => tables.apply(rgb, gain),
None => rgb.map(|v| v * gain),
};
// after the baseline exposure, as when it was fitted
hue_sat.as_ref().map_or(rgb, |h| h.apply(rgb)).map(|v| v.max(0.0))
});
stages.push(("colour", t0.elapsed()));
let img = fit(&img, max_edge, max_edge, Filter::Box);
@@ -286,7 +290,7 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
}
let (temp, tint) = xy_to_temp_tint(xy);
let relative = crate::camera_preview::file_local_look(raw.format) && t.matrix_is_fallback;
let camera_tone = camera_look.map(|p| p.tone).or_else(|| raw.color.profile.tone_curve.as_ref().and_then(dng_tone_curve));
let camera_tone = camera_look.as_ref().map(|p| p.tone).or_else(|| raw.color.profile.tone_curve.as_ref().and_then(dng_tone_curve));
let (temp, tint) = if relative { (6500.0, 0.0) } else { (temp.round(), tint.round()) };
return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone }));
}
+66 -3
View File
@@ -1,9 +1,9 @@
//! Fonts from the optional craft-fonts build input (storytold/craft-fonts).
//!
//! Built with `CRAFT_FONTS_DIR=<craft-fonts checkout>`, [`CRAFT_FONTS`] holds every font in its
//! manifest (today: the Japanese UI and document fonts); otherwise it is empty and LightCraft uses
//! only its own bundled fonts (Inter). Both the egui UI and the export watermark renderer read it
//! from here. See craftrules `standards/fonts.md`.
//! manifest (today: the Japanese and Simplified-Chinese UI and document fonts); otherwise it is
//! empty and LightCraft uses only its own bundled fonts (Inter). Both the egui UI and the export
//! watermark renderer read it from here. See craftrules `standards/fonts.md`.
/// A font from the optional craft-fonts build input (empty unless built with `CRAFT_FONTS_DIR`).
pub struct CraftFont {
@@ -26,6 +26,13 @@ impl CraftFont {
pub fn is_mincho(&self) -> bool {
self.family.contains("Mincho")
}
/// Whether the font can stand in for `script`: it is meant for it, or — until craft-fonts has a
/// Traditional Chinese face — it is a Simplified Chinese face, which carries the Traditional
/// characters too (with mainland glyph forms; still closer than the Japanese faces).
pub fn serves(&self, script: &str) -> bool {
self.covers(script) || (script == "Hant" && self.covers("Hans"))
}
}
/// The craft-fonts faces for Japanese, in `fonts`' order (empty without craft-fonts).
@@ -33,6 +40,43 @@ pub fn japanese(fonts: &'static [CraftFont]) -> impl Iterator<Item = &'static Cr
fonts.iter().filter(|f| f.covers("Jpan"))
}
/// The craft-fonts faces for any CJK script: what any CJK text needs, regardless of language.
pub fn cjk(fonts: &'static [CraftFont]) -> impl Iterator<Item = &'static CraftFont> {
fonts.iter().filter(|f| f.scripts.iter().any(|script| is_cjk_script(script)))
}
/// Whether an ISO 15924 script is written with Han, kana or hangul characters (Unicode's CJK
/// scripts), which Inter has no glyphs for.
pub fn is_cjk_script(script: &str) -> bool {
matches!(script, "Hans" | "Hant" | "Jpan" | "Kore" | "Hang" | "Hani" | "Bopo" | "Yiii" | "Nshu" | "Tang")
}
/// The craft-fonts faces to fall back on for `script`, in preference order: the faces of that script
/// first (for `style` when asked), then every other CJK face. One CJK face list serves every
/// language, but Han characters are shared between them: a Chinese reader must get the Chinese
/// forms and a Japanese reader the Japanese ones, so the active language's own script comes first.
/// Empty without craft-fonts.
pub fn cjk_fallback<'a>(fonts: &'a [CraftFont], script: &str, style: &str) -> Vec<&'a CraftFont> {
let mut faces: Vec<&CraftFont> = fonts.iter().filter(|font| covers_cjk(font)).collect();
faces.sort_by_key(|font| {
(
// The language's own script first. A face dedicated to the script is preferred over a
// multi-script face (Noto Sans CJK SC claims `Hans,Latn`; a Japanese face claims `Jpan`).
!font.serves(script),
font.scripts.len() != 1,
font.family != "BIZ UDPGothic",
font.style != style,
font.is_mincho(),
)
});
faces
}
/// Whether the font is for any CJK script.
fn covers_cjk(font: &CraftFont) -> bool {
font.scripts.iter().any(|script| is_cjk_script(script))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -49,4 +93,23 @@ mod tests {
}
assert!(japanese(CRAFT_FONTS).next().is_some(), "craft-fonts carries Japanese faces");
}
/// Each language's own faces come first, so shared Han keeps the language's forms, and Latin
/// text still reaches Inter before any CJK fallback.
#[test]
fn a_language_gets_its_own_script_first() {
static FACES: &[CraftFont] = &[
CraftFont { family: "BIZ UDPGothic", style: "Regular", scripts: &["Jpan", "Latn"], bytes: &[] },
CraftFont { family: "Noto Sans CJK SC", style: "Regular", scripts: &["Hans", "Latn"], bytes: &[] },
CraftFont { family: "BIZ UDMincho", style: "Regular", scripts: &["Jpan", "Latn"], bytes: &[] },
];
let first = |script: &str| cjk_fallback(FACES, script, "Regular").first().map(|f| f.family);
assert_eq!(first("Hans"), Some("Noto Sans CJK SC"));
assert_eq!(first("Jpan"), Some("BIZ UDPGothic"));
// No Traditional Chinese face yet: the Chinese face stands in before the Japanese ones.
assert_eq!(first("Hant"), Some("Noto Sans CJK SC"));
// Every CJK face is still offered to every language.
assert_eq!(cjk_fallback(FACES, "Hans", "Regular").len(), 3);
assert_eq!(cjk_fallback(&[], "Hans", "Regular").len(), 0);
}
}
+16 -2
View File
@@ -12,6 +12,7 @@
pub mod availability;
mod camera_preview;
pub mod camera_profiles;
pub mod cmd;
pub mod crs;
pub mod crs_masks;
@@ -32,6 +33,7 @@ pub mod preset_import;
pub mod preset_luminar;
pub mod presets;
pub mod rename;
pub mod segment;
pub mod sidecar;
pub mod smart;
mod view;
@@ -125,6 +127,9 @@ pub struct Session {
pub undo: Vec<UndoEntry>,
pub redo: Vec<UndoEntry>,
pub interaction: Option<Interaction>,
/// Set by a command whose change must not rewrite the photo's XMP sidecar even with auto-write on
/// (a catalog-only edit of data the sidecar writer does not emit); consumed when the command ends.
pub(crate) skip_auto_write: bool,
/// Copied develop settings (partial JSON) for Paste.
pub clipboard: Option<Value>,
/// The folder on disk the [`LibrarySource::Folder`] view browses.
@@ -150,6 +155,8 @@ pub struct Session {
depth: u32,
/// Selected mask (Masking panel), by mask id.
pub active_mask: Option<u32>,
/// AI masks (SAM 3): the model and the last photo prepared for it.
pub segmenter: segment::Segmenter,
/// Selected spot (Remove panel), by index into the active photo's spots.
pub active_spot: Option<usize>,
/// The persistent library this session writes to (`None` = in-memory only).
@@ -223,6 +230,7 @@ impl Session {
undo: Vec::new(),
redo: Vec::new(),
interaction: None,
skip_auto_write: false,
clipboard: None,
meta_clipboard: None,
browse: None,
@@ -237,6 +245,7 @@ impl Session {
clock: Box::new(|| "2026-09-30T12:00:00".to_string()),
depth: 0,
active_mask: None,
segmenter: segment::Segmenter::default(),
active_spot: None,
library: None,
xmp: sidecar::XmpPrefs::default(),
@@ -308,8 +317,11 @@ impl Session {
self.journal.drain(..1000);
}
}
if r.is_ok() && self.depth == 0 && self.xmp.auto_write && self.interaction.is_none() && self.pending_log.len() > log_start {
self.auto_write_sidecars(&self.pending_log[log_start..]);
if self.depth == 0 {
let skip = std::mem::take(&mut self.skip_auto_write);
if r.is_ok() && !skip && self.xmp.auto_write && self.interaction.is_none() && self.pending_log.len() > log_start {
self.auto_write_sidecars(&self.pending_log[log_start..]);
}
}
if self.depth == 0 && self.library.is_some() {
// Make the command durable before reporting success. When the command's own records
@@ -715,6 +727,8 @@ mod tests_persist;
#[cfg(test)]
mod tests_prefs;
#[cfg(test)]
mod tests_segment;
#[cfg(test)]
mod tests_settings_files;
#[cfg(test)]
mod tests_spots;
+5 -11
View File
@@ -4,7 +4,7 @@
//! ```text
//! LightCraft Library/
//! catalog.snap catalog.log (lightcraft-catalog journal)
//! presets.json view.json (user presets + favourites; last source/filter/sort/selection)
//! presets.json view.json (user presets + favourites; last source/sort/selection)
//! prefs.json (library preferences: XMP sidecars, import defaults, cache size, last export)
//! thumbs/ (rendered thumbnail cache, safe to delete)
//! Originals/ (photos imported with "copy into library")
@@ -102,7 +102,8 @@ struct PresetsFile {
struct ViewFile {
source: LibrarySource,
browse: Option<crate::Browse>,
filter: lightcraft_catalog::Filter,
// No filter: a library opens unfiltered. A date, keyword or person left over from the last session
// would silently hide photos, with only a small badge to say so.
sort: lightcraft_catalog::Sort,
selection: Selection,
}
@@ -346,7 +347,6 @@ impl Session {
if let Some(v) = settings.read::<ViewFile>(files.as_mut(), "view.json") {
self.source = v.source;
self.browse = v.browse;
self.filter = v.filter;
self.sort = v.sort;
self.selection = v.selection;
self.selection.ids.retain(|id| self.catalog.photo(*id).is_some());
@@ -494,17 +494,11 @@ impl Session {
}
fn view_json(&self) -> Vec<u8> {
let view = ViewFile {
source: self.source,
browse: self.browse.clone(),
filter: self.filter.clone(),
sort: self.sort,
selection: self.selection.clone(),
};
let view = ViewFile { source: self.source, browse: self.browse.clone(), sort: self.sort, selection: self.selection.clone() };
serde_json::to_vec_pretty(&view).unwrap_or_default()
}
/// Save the view state (source, filter, sort, selection) if it changed since it was last
/// Save the view state (source, sort, selection) if it changed since it was last
/// written. Native hosts get this from [`Session::close_library`]; the browser host calls it
/// periodically, since a tab can be closed without notice.
pub fn save_view(&mut self) {
+66 -4
View File
@@ -26,7 +26,7 @@ use lightcraft_raster::{Histogram, Rgb32f, Rgba8};
use serde::{Deserialize, Serialize};
/// Bump when the pipeline's output changes, to invalidate cached thumbnails.
pub const RENDER_CACHE_VERSION: u64 = 9;
pub const RENDER_CACHE_VERSION: u64 = 11;
/// Thumbnails render at one of these long edges (so window/cell size changes reuse the cache).
pub const THUMB_SIZES: [usize; 4] = [128, 256, 384, 512];
@@ -647,7 +647,13 @@ impl crate::Session {
^ id.0.wrapping_mul(0x9e37_79b9_7f4a_7c15)
^ content.rotate_left(17);
let cache = thumb_bucket.map(|b| {
let k = Hasher128::new().str(&content_key(&p)).u64(settings.hash64()).u64(b as u64).u64(RENDER_CACHE_VERSION).finish();
let k = Hasher128::new()
.str(&content_key(&p))
.u64(settings.hash64())
.u64(b as u64)
.u64(RENDER_CACHE_VERSION)
.u64(crate::camera_profiles::cache_key())
.finish();
(self.media.rendered.clone(), k)
});
Some(RenderJob {
@@ -676,7 +682,14 @@ impl crate::Session {
/// Cache key of a variant thumbnail ([`Self::variant_job`]).
pub fn variant_key(p: &Photo, settings: &DevelopSettings, edge: usize) -> Hash128 {
Hasher128::new().str(&content_key(p)).str("variant").u64(settings.hash64()).u64(edge as u64).u64(RENDER_CACHE_VERSION).finish()
Hasher128::new()
.str(&content_key(p))
.str("variant")
.u64(settings.hash64())
.u64(edge as u64)
.u64(RENDER_CACHE_VERSION)
.u64(crate::camera_profiles::cache_key())
.finish()
}
/// A thumbnail of `id` rendered with `settings` instead of its own (profile and preset
@@ -705,9 +718,58 @@ impl crate::Session {
})
}
/// A square close-up of `face` (normalized, in the photo's upright frame) for a People card: the
/// photo's own look with its crop replaced by a square around the face (room for hair and chin),
/// `edge` pixels across (≤ 512). The source level is the smallest that keeps the face sharp, up
/// to the preview. Cached like [`Self::variant_job`], and `key` likewise follows the content.
pub fn face_job(&mut self, id: PhotoId, face: lightcraft_geom::Rect, edge: usize) -> Option<RenderJob> {
let p = self.catalog.photo(id)?.clone();
// `face` is on the upright (EXIF-oriented) photo; the crop below is in the frame after the
// user's Rotate Left/Right, so carry the box (and the photo's size) over to it.
let orient = p.develop.orientation;
let face = orient.map_norm_rect(face);
let (w, h) = (f64::from(p.width.max(1)), f64::from(p.height.max(1)));
let (w, h) = if orient.swaps_axes() { (h, w) } else { (w, h) };
let side = ((face.x1 - face.x0) * w).max((face.y1 - face.y0) * h) * 1.7;
// finite, within the photo, and never wider than its short side (so the clamps below are valid)
let side = if side.is_finite() { side.clamp(1.0, w.min(h)) } else { w.min(h) };
let (cx, cy) = ((face.x0 + face.x1) / 2.0 * w, (face.y0 + face.y1) / 2.0 * h);
let (cx, cy) = (if cx.is_finite() { cx } else { w / 2.0 }, if cy.is_finite() { cy } else { h / 2.0 });
let (x0, y0) = (cx.clamp(side / 2.0, w - side / 2.0) - side / 2.0, cy.clamp(side / 2.0, h - side / 2.0) - side / 2.0);
let rect = lightcraft_geom::Rect { x0: x0 / w, y0: y0 / h, x1: (x0 + side) / w, y1: (y0 + side) / h };
let mut settings = (*p.develop).clone();
settings.crop = lightcraft_develop::Crop { geometry: lightcraft_geom::CropGeometry { rect, angle: 0.0 }, ..Default::default() };
let edge = edge.clamp(16, SourceLevel::Thumb.max_edge());
// pixels along the long edge that leave `edge` across the crop
let needed = (edge as f64 * w.max(h) / side).ceil().min(SourceLevel::Preview.max_edge() as f64);
let level = SourceLevel::for_size(needed as usize);
let source = self.media.source_ref(&p, level);
let ck = Self::variant_key(&p, &settings, edge);
Some(RenderJob {
photo: id,
level,
source,
origin: p.source.clone(),
info: source_info(&p),
settings: Arc::new(settings),
request: RenderRequest { apply_crop: true, ..RenderRequest::fit(edge, edge) },
key: (ck.0 as u64) ^ ((ck.0 >> 64) as u64),
cache: Some((self.media.rendered.clone(), ck)),
stages: None,
view_cache: None,
})
}
/// Size-independent cache key of a photo's view render (loupe) for its current settings.
fn view_key(p: &Photo, apply_crop: bool) -> Hash128 {
Hasher128::new().str(&content_key(p)).str("view").u64(p.develop.hash64()).u64(apply_crop as u64).u64(RENDER_CACHE_VERSION).finish()
Hasher128::new()
.str(&content_key(p))
.str("view")
.u64(p.develop.hash64())
.u64(apply_crop as u64)
.u64(RENDER_CACHE_VERSION)
.u64(crate::camera_profiles::cache_key())
.finish()
}
/// The loupe's render job: like [`Self::render_job`], and a full-quality result is kept as the
+146
View File
@@ -0,0 +1,146 @@
//! The model download as a background task the session starts, watches and cancels without
//! ever blocking: progress lives in atomics, messages behind a mutex only ever `try_lock`ed
//! by the session (the download thread holds it for a few instructions at a time).
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use lightcraft_segment::fetch::{self, Options, Progress};
/// What the session sees of a download.
#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize)]
pub struct Status {
pub running: bool,
pub done: u64,
pub total: u64,
/// The file being fetched.
pub file: String,
/// The last download's error (cleared when a new one starts).
pub error: Option<String>,
/// The last download finished and the files are in place.
pub finished: bool,
}
#[derive(Default)]
struct Text {
file: String,
error: Option<String>,
}
#[derive(Default)]
struct Shared {
running: AtomicBool,
cancel: AtomicBool,
finished: AtomicBool,
done: AtomicU64,
total: AtomicU64,
text: Mutex<Text>,
}
/// Clears `running` when the download thread ends, however it ends.
struct Running(Arc<Shared>);
impl Drop for Running {
fn drop(&mut self) {
self.0.running.store(false, Ordering::SeqCst);
}
}
#[derive(Default)]
pub struct Downloader {
shared: Arc<Shared>,
/// The last text seen (when the download thread held the lock at that moment).
last: Mutex<Text>,
}
fn lock(m: &Mutex<Text>) -> std::sync::MutexGuard<'_, Text> {
m.lock().unwrap_or_else(std::sync::PoisonError::into_inner)
}
impl Downloader {
pub fn status(&self) -> Status {
let s = &self.shared;
if let Ok(t) = s.text.try_lock()
&& let Ok(mut last) = self.last.try_lock()
{
last.file.clone_from(&t.file);
last.error.clone_from(&t.error);
}
// (only the session's thread locks `last`)
let (file, error) = self.last.try_lock().map(|l| (l.file.clone(), l.error.clone())).unwrap_or_default();
Status {
running: s.running.load(Ordering::SeqCst),
done: s.done.load(Ordering::SeqCst),
total: s.total.load(Ordering::SeqCst),
file,
error,
finished: s.finished.load(Ordering::SeqCst),
}
}
pub fn running(&self) -> bool {
self.shared.running.load(Ordering::SeqCst)
}
/// Ask a running download to stop (it keeps its partial files to resume later).
pub fn cancel(&self) -> bool {
let running = self.running();
if running {
self.shared.cancel.store(true, Ordering::SeqCst);
}
running
}
/// Start downloading `files` from `mirrors` into `dir` on a background thread. False when
/// one is already running.
pub fn start(&self, files: &'static [fetch::FileSpec], mirrors: Vec<String>, dir: PathBuf, opts: Options) -> Result<bool, String> {
let s = self.shared.clone();
if s.running.swap(true, Ordering::SeqCst) {
return Ok(false);
}
s.cancel.store(false, Ordering::SeqCst);
s.finished.store(false, Ordering::SeqCst);
s.done.store(0, Ordering::SeqCst);
s.total.store(files.iter().filter_map(|f| f.size).sum(), Ordering::SeqCst);
// nothing else runs while `running` was false
*lock(&s.text) = Text::default();
*lock(&self.last) = Text::default();
let guard = Running(s.clone());
let spawned = std::thread::Builder::new().name("sam3-download".into()).spawn(move || {
let s = guard.0.clone();
let mut last_file = String::new();
let mut on_progress = |p: &Progress| {
s.done.store(p.done, Ordering::SeqCst);
s.total.store(p.total, Ordering::SeqCst);
if p.file != last_file {
last_file.clone_from(&p.file);
lock(&s.text).file.clone_from(&p.file);
}
};
let r =
std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| fetch::download(files, &mirrors, &dir, &opts, &s.cancel, &mut on_progress)));
let error = match r {
Ok(Ok(())) => {
s.finished.store(true, Ordering::SeqCst);
log::info!("SAM 3 model downloaded to {}", dir.display());
None
}
Ok(Err(e)) => Some(e.to_string()),
Err(_) => Some("the download failed unexpectedly".to_string()),
};
if let Some(e) = &error {
log::warn!("SAM 3 download: {e}");
}
lock(&s.text).error = error;
drop(guard);
});
match spawned {
Ok(_) => Ok(true),
Err(e) => {
// the closure (and its guard) was dropped: `running` is false again
Err(format!("could not start the download: {e}"))
}
}
}
}
+715
View File
@@ -0,0 +1,715 @@
//! AI masks: Object (clicks) and Describe (text) selections computed with SAM 3
//! (`lightcraft-segment`, cargo feature `sam`; the desktop app enables it).
//!
//! **Nothing requires the model.** It is not part of LightCraft (SAM License); the user
//! downloads it when they first use an AI mask and agree to (`segment.model.download`), or
//! puts the files in the model folder themselves. Without it, AI mask requests fail with a
//! clear "not installed" error; renders, exports and every other feature never touch it
//! (stored segmentations are plain data in the develop settings).
//!
//! The model sees the photo as developed, uncropped and without masks, at 1008 px — the frame
//! mask coordinates live in — so a stored segmentation lines up with the photo at any size.
//! Everything that touches the model (loading it, rendering and encoding the photo, clicks,
//! descriptions, detail passes) runs on one worker thread ([`worker`]); the session talks to it
//! over channels and never waits on a lock. With [`Segmenter::background`] (the desktop app)
//! commands return at once and [`Session::segment_poll`] applies the results; without it (CLI,
//! MCP, tests) they wait for the worker.
#[cfg(feature = "sam")]
mod download;
#[cfg(feature = "sam")]
mod worker;
use std::path::PathBuf;
use lightcraft_catalog::PhotoId;
use lightcraft_develop::{MaskShape, SegMask};
use lightcraft_geom::Point;
use crate::Session;
/// Long edge of the image the model sees.
pub const INPUT_EDGE: usize = 1008;
/// Download size of the model (`model.safetensors`; the tokenizer files add ~2 MB).
pub const MODEL_BYTES: u64 = 3_439_938_512;
/// The model's licence (not LightCraft's): shown before downloading.
pub const LICENSE_NAME: &str = "SAM License (Meta)";
pub const LICENSE_URL: &str = "https://github.com/facebookresearch/sam3/blob/main/LICENSE";
/// How errors about a missing model start (the UI offers the download on it).
pub const NOT_INSTALLED: &str = "The SAM 3 model is not installed";
/// Longest a waiting command (CLI, MCP) waits for the model (loading on a CPU can take minutes).
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
const WAIT: std::time::Duration = std::time::Duration::from_secs(15 * 60);
/// Most clicks one Object selection takes (the model's prompt grows with each).
pub const MAX_CLICKS: usize = 64;
/// One click of an Object selection (normalized image coordinates).
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Click {
pub at: Point,
pub include: bool,
}
/// What a request's result is for (how [`Session::segment_poll`] applies it).
#[derive(Clone, Debug, PartialEq)]
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
pub(crate) enum Tag {
/// A command waiting for the result itself.
Wait,
/// Load the model and encode the photo ahead of the first click.
Prepare,
/// Object clicks on component `comp` of mask `mask`.
Clicks { photo: PhotoId, mask: u32, comp: usize, hint: Vec<Point>, exclude: Vec<Point> },
/// A Describe selection: a new mask (`mask: None`) or a component combined by `op`.
NewPrompt { photo: PhotoId, mask: Option<u32>, op: String, name: Option<String>, text: String },
/// A zoomed-in pass for a component, computed from `shape` (without its detail).
Detail { photo: PhotoId, mask: u32, comp: usize, shape: MaskShape },
}
/// The model input: 8-bit RGB.
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
pub(crate) struct Input {
rgb: Vec<u8>,
w: usize,
h: usize,
}
/// What a detail pass re-runs.
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
pub(crate) enum Prompt {
Clicks(Vec<Click>),
Text(String),
}
/// Object clicks sent to the worker and not applied yet (background mode): further clicks add
/// to these, and the loupe shows them.
#[derive(Clone, Debug, PartialEq)]
pub struct PendingClicks {
pub photo: PhotoId,
pub mask: u32,
pub comp: usize,
pub hint: Vec<Point>,
pub exclude: Vec<Point>,
}
/// What [`Session::segment_poll`] did.
#[derive(Debug, Default)]
pub struct Polled {
/// A mask changed.
pub changed: bool,
/// Errors and notices from background work, for the user.
pub messages: Vec<String>,
/// A mask that got a new selection (worth a detail pass).
pub refine: Option<u32>,
}
/// The model download (`segment.model.*`).
#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize)]
pub struct DownloadStatus {
pub running: bool,
pub done: u64,
pub total: u64,
pub file: String,
pub error: Option<String>,
pub finished: bool,
}
/// The model (on its worker thread), the download, and AI mask requests in flight.
pub struct Segmenter {
/// Where the `facebook/sam3` files live (set by the app; `None`: no AI masks).
pub dir: Option<PathBuf>,
/// The user's list of download mirrors (one base URL per line), if any.
pub mirrors_file: Option<PathBuf>,
/// Run requests in the background and apply them in [`Session::segment_poll`] (the
/// desktop app); otherwise commands wait for their result (CLI, MCP, tests).
pub background: bool,
pending: Option<PendingClicks>,
messages: Vec<String>,
#[cfg(feature = "sam")]
worker: worker::Worker,
#[cfg(feature = "sam")]
download: download::Downloader,
#[cfg(feature = "sam")]
results: (std::sync::mpsc::Sender<worker::Outcome>, std::sync::mpsc::Receiver<worker::Outcome>),
}
// (derivable only without the `sam` feature: the results channel has no Default)
#[allow(clippy::derivable_impls)]
impl Default for Segmenter {
fn default() -> Self {
Segmenter {
dir: None,
mirrors_file: None,
background: false,
pending: None,
messages: Vec::new(),
#[cfg(feature = "sam")]
worker: worker::Worker::default(),
#[cfg(feature = "sam")]
download: download::Downloader::default(),
#[cfg(feature = "sam")]
results: std::sync::mpsc::channel(),
}
}
}
impl Segmenter {
/// Whether this build can compute AI masks at all.
pub const AVAILABLE: bool = cfg!(feature = "sam");
/// Whether the model's files are in place.
pub fn installed(&self) -> bool {
#[cfg(feature = "sam")]
if let Some(dir) = &self.dir {
return lightcraft_segment::is_model_dir(dir);
}
false
}
/// Whether requests are queued or running (loading, analyzing the photo, a click…).
pub fn busy(&self) -> bool {
#[cfg(feature = "sam")]
return self.worker.pending() > 0;
#[cfg(not(feature = "sam"))]
false
}
/// Whether the slow part runs now: loading the model or analyzing a photo.
pub fn analyzing(&self) -> bool {
#[cfg(feature = "sam")]
return self.worker.encoding();
#[cfg(not(feature = "sam"))]
false
}
/// Whether a zoomed-in detail pass is queued or running.
pub fn detail_busy(&self) -> bool {
#[cfg(feature = "sam")]
return self.worker.detail() > 0;
#[cfg(not(feature = "sam"))]
false
}
/// Whether the model is in memory (it is unloaded after a while without use).
pub fn loaded(&self) -> bool {
#[cfg(feature = "sam")]
return self.worker.loaded();
#[cfg(not(feature = "sam"))]
false
}
/// Clicks waiting for their segmentation.
pub fn pending_clicks(&self) -> Option<&PendingClicks> {
self.pending.as_ref()
}
/// The model download's state.
pub fn download_status(&self) -> DownloadStatus {
#[cfg(feature = "sam")]
{
let s = self.download.status();
DownloadStatus { running: s.running, done: s.done, total: s.total, file: s.file, error: s.error, finished: s.finished }
}
#[cfg(not(feature = "sam"))]
DownloadStatus::default()
}
/// Where the model is downloaded from, in order (the user's mirrors first).
pub fn mirrors(&self) -> Vec<String> {
#[cfg(feature = "sam")]
{
let env = std::env::var(lightcraft_segment::fetch::MIRRORS_ENV).ok();
lightcraft_segment::fetch::mirrors(env.as_deref(), self.mirrors_file.as_deref())
}
#[cfg(not(feature = "sam"))]
Vec::new()
}
/// The model folder, if the model is there; otherwise an error that says what to do.
pub fn model_dir(&self) -> Result<PathBuf, String> {
if !Self::AVAILABLE {
return Err("AI masks are not available in this build".into());
}
let dir = self.dir.clone().ok_or("no folder is set for the SAM 3 model")?;
if self.installed() {
return Ok(dir);
}
let d = self.download_status();
if d.running {
let pct = d.done.saturating_mul(100).checked_div(d.total).unwrap_or(0);
return Err(format!("{NOT_INSTALLED} yet: it is downloading ({pct} %)."));
}
Err(format!(
"{NOT_INSTALLED}. Download it (about {:.1} GB, {LICENSE_NAME}) when LightCraft offers it, with `segment.model.download {{\"acknowledged\": true}}`, or put model.safetensors, vocab.json and merges.txt in {}.",
MODEL_BYTES as f64 / 1e9,
dir.display()
))
}
/// Start downloading the model (on a background thread; see `segment.model.download`).
/// `Ok(false)` when it is installed or downloading already.
pub fn start_download(&self) -> Result<bool, String> {
if !Self::AVAILABLE {
return Err("AI masks are not available in this build".into());
}
let dir = self.dir.clone().ok_or("no folder is set for the SAM 3 model")?;
if self.installed() {
return Ok(false);
}
#[cfg(feature = "sam")]
{
let mirrors = self.mirrors();
if mirrors.is_empty() {
return Err(lightcraft_segment::fetch::DownloadError::NoMirrors.to_string());
}
self.download.start(lightcraft_segment::fetch::SAM3_FILES, mirrors, dir, lightcraft_segment::fetch::Options::default())
}
#[cfg(not(feature = "sam"))]
{
let _ = dir;
Ok(false)
}
}
/// Stop a running download (its partial files stay, to resume). False when none runs.
pub fn cancel_download(&self) -> bool {
#[cfg(feature = "sam")]
return self.download.cancel();
#[cfg(not(feature = "sam"))]
false
}
}
/// Mix `v` into a 64-bit hash.
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
fn mix(h: u64, v: u64) -> u64 {
(h ^ v).wrapping_mul(0x0100_0000_01b3).rotate_left(29)
}
/// Objects covering more than this much of the photo gain little from a zoomed pass.
const DETAIL_MAX_AREA: f64 = 0.4;
/// The part of the photo to zoom into for `seg`'s selection: its bounding box with a margin of
/// a quarter of its size, at least 12 % of the long edge, at most 2:1, inside the photo
/// (`aspect` = width / height). `None` when nothing is selected or it is most of the photo.
pub fn detail_region(seg: &SegMask, aspect: f64) -> Option<[f64; 4]> {
let logits = seg.logits()?;
let r = seg.bounds()?;
let side = seg.side as usize;
let (mut x0, mut y0, mut x1, mut y1) = (usize::MAX, usize::MAX, 0, 0);
for (i, l) in logits.iter().enumerate() {
if *l > 0.0 {
let (x, y) = (i % side, i / side);
(x0, y0, x1, y1) = (x0.min(x), y0.min(y), x1.max(x + 1), y1.max(y + 1));
}
}
if x0 == usize::MAX {
return None;
}
let (rw, rh) = (r[2] - r[0], r[3] - r[1]);
let nx = |c: usize| r[0] + c as f64 / side as f64 * rw;
let ny = |c: usize| r[1] + c as f64 / side as f64 * rh;
let (bx0, by0, bx1, by1) = (nx(x0), ny(y0), nx(x1), ny(y1));
if (bx1 - bx0) * (by1 - by0) > DETAIL_MAX_AREA {
return None;
}
// in units of the photo's height: x × aspect
let a = if aspect.is_finite() { aspect.clamp(0.1, 10.0) } else { 1.0 };
let (cx, cy) = ((bx0 + bx1) / 2.0 * a, (by0 + by1) / 2.0);
let size = ((bx1 - bx0) * a).max(by1 - by0);
let m = 0.25 * size;
let mut w = ((bx1 - bx0) * a + 2.0 * m).max(0.12 * a.max(1.0));
let mut h = ((by1 - by0) + 2.0 * m).max(0.12 * a.max(1.0));
w = w.max(h / 2.0).min(a);
h = h.max(w / 2.0).min(1.0);
let x = (cx - w / 2.0).clamp(0.0, (a - w).max(0.0));
let y = (cy - h / 2.0).clamp(0.0, (1.0 - h).max(0.0));
let out = [x / a, y, (x + w) / a, y + h];
// a region nearly the whole photo isn't worth a second pass
((out[2] - out[0]) * (out[3] - out[1]) < 0.8).then_some(out)
}
/// `shape` without its detail patches (what a detail pass was computed from)
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
fn strip_detail(mut shape: MaskShape) -> MaskShape {
// (nor its edge setting: moving the slider meanwhile keeps the pass)
if let MaskShape::Object { detail, edge, .. } | MaskShape::Prompt { detail, edge, .. } = &mut shape {
detail.clear();
*edge = 0.0;
}
shape
}
fn clicks_of(include: &[Point], exclude: &[Point]) -> Vec<Click> {
include.iter().map(|p| Click { at: *p, include: true }).chain(exclude.iter().map(|p| Click { at: *p, include: false })).collect()
}
impl Session {
/// The key and settings of photo `id` as the model sees it: its look without masks, uncropped.
#[cfg_attr(not(feature = "sam"), allow(dead_code))]
fn segment_key(&self, id: PhotoId) -> Option<(u64, lightcraft_develop::DevelopSettings)> {
let p = self.catalog.photo(id)?;
let mut d = (*p.develop).clone();
d.masks.clear();
let content = crate::media::content_key(p);
let mut h = mix(0xcbf2_9ce4_8422_2325, d.hash64());
h = mix(h, id.0);
for b in content.bytes() {
h = mix(h, u64::from(b));
}
Some((h, d))
}
/// Clicks (in normalized photo coordinates) for the model.
#[cfg(feature = "sam")]
fn model_clicks(clicks: &[Click]) -> Vec<lightcraft_segment::Click> {
clicks.iter().map(|c| lightcraft_segment::Click { x: c.at.x as f32, y: c.at.y as f32, positive: c.include }).collect()
}
/// A worker request about photo `id` (the render job for its model input goes along; the
/// worker runs it only when its cache doesn't hold the photo). `region` renders that part
/// of the photo, from a source sharp enough for it (detail passes).
#[cfg(feature = "sam")]
fn segment_job(
&mut self,
id: PhotoId,
kind: worker::Kind,
tag: Tag,
reply: std::sync::mpsc::Sender<worker::Outcome>,
region: Option<[f64; 4]>,
) -> Result<worker::Job, String> {
let dir = self.segmenter.model_dir()?;
let (key, mut d) = self.segment_key(id).ok_or("no such photo")?;
let p = self.catalog.photo(id).ok_or("no such photo")?.clone();
let missing = match &p.source {
lightcraft_catalog::Source::File { path } if !std::path::Path::new(path).exists() => Some(path.clone()),
_ => None,
};
let render = match region {
None => self.preview_job(id, INPUT_EDGE, INPUT_EDGE, false, &d).ok_or("no such photo")?,
Some(region) => {
d.crop.geometry =
lightcraft_geom::CropGeometry { rect: lightcraft_geom::Rect::new(region[0], region[1], region[2], region[3]), angle: 0.0 };
d.crop.flip_h = false;
d.crop.flip_v = false;
d.geometry.constrain_crop = false;
let mut job = self.preview_job(id, INPUT_EDGE, INPUT_EDGE, true, &d).ok_or("no such photo")?;
let frac = (region[2] - region[0]).max(region[3] - region[1]).max(0.01);
job.level = crate::media::SourceLevel::for_size((INPUT_EDGE as f64 / frac) as usize);
job.source = self.media.source_ref(&p, job.level);
job
}
};
Ok(worker::Job { dir, key: if region.is_some() { 0 } else { key }, render: Some(render), missing, kind, tag, reply })
}
/// Send a request and wait for its result (CLI, MCP, tests; or when not in background mode).
#[cfg(feature = "sam")]
fn segment_wait(&mut self, id: PhotoId, kind: worker::Kind, region: Option<[f64; 4]>) -> Result<Option<SegMask>, String> {
let (tx, rx) = std::sync::mpsc::channel();
let job = self.segment_job(id, kind, Tag::Wait, tx, region)?;
self.segmenter.worker.submit(job)?;
match rx.recv_timeout(WAIT) {
Ok(o) => o.result,
Err(_) => Err("the AI model took too long; try again".into()),
}
}
/// Send a request whose result [`Session::segment_poll`] applies.
#[cfg(feature = "sam")]
fn segment_queue(&mut self, id: PhotoId, kind: worker::Kind, tag: Tag, region: Option<[f64; 4]>) -> Result<(), String> {
let reply = self.segmenter.results.0.clone();
let job = self.segment_job(id, kind, tag, reply, region)?;
self.segmenter.worker.submit(job)
}
/// Start preparing photo `id` for AI masks (load the model, encode the image), so the first
/// click is fast. In background mode it returns at once.
pub fn segment_prepare(&mut self, id: PhotoId) -> Result<(), String> {
self.segmenter.model_dir()?;
#[cfg(feature = "sam")]
{
if self.segmenter.background {
return self.segment_queue(id, worker::Kind::Prepare, Tag::Prepare, None);
}
self.segment_wait(id, worker::Kind::Prepare, None)?;
}
#[cfg(not(feature = "sam"))]
let _ = id;
Ok(())
}
/// Segment the object `clicks` pick on photo `id`, waiting for the model.
pub fn segment_clicks(&mut self, id: PhotoId, clicks: &[Click]) -> Result<SegMask, String> {
self.segmenter.model_dir()?;
if !clicks.iter().any(|c| c.include) {
return Err("click the object to include it first".into());
}
#[cfg(feature = "sam")]
{
self.segment_wait(id, worker::Kind::Clicks(Self::model_clicks(clicks)), None)?.ok_or_else(|| "nothing was selected".to_string())
}
#[cfg(not(feature = "sam"))]
{
let _ = id;
Err("AI masks are not available in this build".into())
}
}
/// Segment everything `text` describes on photo `id`, waiting for the model; `Ok(None)`
/// when nothing matches.
pub fn segment_text(&mut self, id: PhotoId, text: &str) -> Result<Option<SegMask>, String> {
self.segmenter.model_dir()?;
let text = text.trim();
if text.is_empty() {
return Err("describe what to select".into());
}
#[cfg(feature = "sam")]
{
self.segment_wait(id, worker::Kind::Text(text.to_string()), None)
}
#[cfg(not(feature = "sam"))]
{
let _ = id;
Err("AI masks are not available in this build".into())
}
}
/// Background mode: queue Object clicks for component `comp` of mask `mask`; the result
/// (clicks and segmentation, one undo step) is applied by [`Session::segment_poll`].
pub(crate) fn segment_clicks_later(&mut self, id: PhotoId, mask: u32, comp: usize, hint: Vec<Point>, exclude: Vec<Point>) -> Result<(), String> {
self.segmenter.model_dir()?;
if hint.is_empty() {
return Err("click the object to include it first".into());
}
#[cfg(feature = "sam")]
{
let clicks = Self::model_clicks(&clicks_of(&hint, &exclude));
let tag = Tag::Clicks { photo: id, mask, comp, hint: hint.clone(), exclude: exclude.clone() };
self.segment_queue(id, worker::Kind::Clicks(clicks), tag, None)?;
self.segmenter.pending = Some(PendingClicks { photo: id, mask, comp, hint, exclude });
Ok(())
}
#[cfg(not(feature = "sam"))]
{
let _ = (id, mask, comp, exclude);
Err("AI masks are not available in this build".into())
}
}
/// Background mode: queue a Describe selection; the mask (or component, combined by `op`
/// into mask `mask`) is created by [`Session::segment_poll`] when something matches.
pub(crate) fn segment_text_later(&mut self, id: PhotoId, mask: Option<u32>, op: &str, name: Option<String>, text: &str) -> Result<(), String> {
self.segmenter.model_dir()?;
let text = text.trim();
if text.is_empty() {
return Err("describe what to select".into());
}
#[cfg(feature = "sam")]
{
let tag = Tag::NewPrompt { photo: id, mask, op: op.to_string(), name, text: text.to_string() };
self.segment_queue(id, worker::Kind::Text(text.to_string()), tag, None)
}
#[cfg(not(feature = "sam"))]
{
let _ = (id, mask, op, name);
Err("AI masks are not available in this build".into())
}
}
/// Start a zoomed-in pass for component `comp` of mask `mask` on photo `id` (an Object or
/// Describe selection): the photo around the selection is rendered at a higher resolution
/// and segmented again. In background mode [`Session::segment_poll`] applies the result;
/// otherwise it is applied before this returns. `Ok(false)` when there is nothing to
/// refine (no selection yet, or it is most of the photo) or a pass is running already.
pub fn segment_detail(&mut self, id: PhotoId, mask: u32, comp: usize) -> Result<bool, String> {
self.segmenter.model_dir()?;
let d = self.develop_of(id).ok_or("no such photo")?;
let shape = d.masks.iter().find(|m| m.id == mask).and_then(|m| m.components.get(comp)).map(|c| c.shape.clone()).ok_or("no such mask")?;
let (seg, prompt) = match &shape {
// (a document with absurdly many clicks isn't run through the model)
MaskShape::Object { hint, exclude, seg: Some(seg), .. } if !hint.is_empty() && hint.len() + exclude.len() <= MAX_CLICKS => {
(seg.clone(), Prompt::Clicks(clicks_of(hint, exclude)))
}
MaskShape::Prompt { text, seg: Some(seg), .. } => (seg.clone(), Prompt::Text(text.clone())),
_ => return Ok(false),
};
let p = self.catalog.photo(id).ok_or("no such photo")?;
let (pw, ph) = (f64::from(p.width.max(1)), f64::from(p.height.max(1)));
let aspect = if d.orientation.swaps_axes() { ph / pw } else { pw / ph };
let Some(region) = detail_region(&seg, aspect) else { return Ok(false) };
#[cfg(feature = "sam")]
{
let kind = worker::Kind::Detail { prompt, region };
let tag = Tag::Detail { photo: id, mask, comp, shape: strip_detail(shape) };
if self.segmenter.background {
if self.segmenter.detail_busy() {
return Ok(false);
}
self.segment_queue(id, kind, tag, Some(region))?;
} else {
let patch = self.segment_wait(id, kind, Some(region))?;
let outcome = worker::Outcome { tag, result: Ok(patch), superseded: false };
let mut polled = Polled::default();
self.apply_outcome(outcome, &mut polled);
}
Ok(true)
}
#[cfg(not(feature = "sam"))]
{
let _ = (prompt, region, mask);
Ok(false)
}
}
/// Apply finished background requests (to masks that haven't changed since) and collect
/// their messages. Cheap when nothing is pending: call it every frame.
pub fn segment_poll(&mut self) -> Polled {
#[cfg_attr(not(feature = "sam"), allow(unused_mut))]
let mut polled = Polled { messages: std::mem::take(&mut self.segmenter.messages), ..Default::default() };
#[cfg(feature = "sam")]
{
// (before draining: a worker that is idle now has sent everything)
let idle = !self.segmenter.busy();
let done: Vec<worker::Outcome> = self.segmenter.results.1.try_iter().collect();
for o in done {
self.apply_outcome(o, &mut polled);
}
// clicks whose request was lost (the worker restarted) aren't pending forever
if idle {
self.segmenter.pending = None;
}
}
polled
}
#[cfg(feature = "sam")]
fn apply_outcome(&mut self, o: worker::Outcome, polled: &mut Polled) {
if o.superseded {
return;
}
match o.tag {
Tag::Wait => {}
Tag::Prepare => {
if let Err(e) = o.result {
polled.messages.push(e);
}
}
Tag::Clicks { photo, mask, comp, hint, exclude } => {
let latest = PendingClicks { photo, mask, comp, hint, exclude };
if self.segmenter.pending.as_ref() != Some(&latest) {
return; // newer clicks are on their way
}
self.segmenter.pending = None;
let seg = match o.result {
Ok(Some(seg)) => seg,
Ok(None) => return,
Err(e) => return polled.messages.push(e),
};
let Some(d) = self.develop_of(photo) else { return };
let mut d = (*d).clone();
let Some(c) = d.masks.iter_mut().find(|m| m.id == mask).and_then(|m| m.components.get_mut(comp)) else { return };
let MaskShape::Object { edge, .. } = c.shape else { return };
c.shape = MaskShape::Object { hint: latest.hint, exclude: latest.exclude, seg: Some(seg), detail: vec![], edge };
match self.set_develop(photo, d, "Object Mask") {
Ok(()) => {
polled.changed = true;
polled.refine = Some(mask);
}
Err(e) => polled.messages.push(e.to_string()),
}
}
Tag::NewPrompt { photo, mask, op, name, text } => {
let seg = match o.result {
Ok(Some(seg)) => seg,
Ok(None) => return polled.messages.push(format!("Nothing matching “{text}” was found in this photo.")),
Err(e) => return polled.messages.push(e),
};
if self.active() != Some(photo) {
return polled.messages.push(format!("The photo changed before “{text}” was found; try again."));
}
let Ok(seg) = serde_json::to_value(seg) else { return };
let mut p = serde_json::json!({"kind": "prompt", "text": text, "seg": seg});
if let Some(n) = name {
p["name"] = serde_json::json!(n);
}
let r = match mask {
None => self.execute("mask.add", &p),
Some(m) => {
if !self.develop_of(photo).is_some_and(|d| d.masks.iter().any(|x| x.id == m)) {
return polled.messages.push("The mask was removed before the selection was found.".into());
}
self.active_mask = Some(m);
p["op"] = serde_json::json!(op);
self.execute("mask.addComponent", &p)
}
};
match r {
Ok(_) => {
polled.changed = true;
polled.refine = self.active_mask;
}
Err(e) => polled.messages.push(e.to_string()),
}
}
Tag::Detail { photo, mask, comp, shape } => {
let patch = match o.result {
Ok(Some(p)) => p,
Ok(None) => return,
Err(e) => return log::warn!("SAM 3 detail pass: {e}"),
};
let Some(d) = self.develop_of(photo) else { return };
let mut d = (*d).clone();
let Some(c) = d.masks.iter_mut().find(|m| m.id == mask).and_then(|m| m.components.get_mut(comp)) else { return };
if strip_detail(c.shape.clone()) != shape {
return; // clicked again meanwhile: a newer pass will follow
}
match &mut c.shape {
MaskShape::Object { detail, .. } | MaskShape::Prompt { detail, .. } => *detail = vec![patch],
_ => return,
}
match self.set_develop(photo, d, "AI Mask Detail") {
Ok(()) => polled.changed = true,
Err(e) => log::warn!("AI mask detail: {e}"),
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn seg_box(side: usize, x0: usize, y0: usize, x1: usize, y1: usize) -> SegMask {
let l: Vec<f32> =
(0..side * side).map(|i| if (x0..x1).contains(&(i % side)) && (y0..y1).contains(&(i / side)) { 8.0 } else { -8.0 }).collect();
SegMask::from_logits(side, &l)
}
#[test]
fn detail_region_frames_the_selection_with_a_margin() {
// a small object in the middle of a square photo
let r = detail_region(&seg_box(100, 40, 45, 50, 55), 1.0).unwrap();
assert!(r[0] < 0.4 && r[2] > 0.5 && r[1] < 0.45 && r[3] > 0.55, "contains it: {r:?}");
assert!(r[2] - r[0] < 0.25 && r[3] - r[1] < 0.25, "zoomed in: {r:?}");
assert!(r.iter().all(|v| (0.0..=1.0).contains(v)));
// at the corner: shifted inside the photo, not cut
let r = detail_region(&seg_box(100, 0, 0, 5, 5), 1.5).unwrap();
assert!(r[0] >= 0.0 && r[1] >= 0.0 && r[0] < 1e-9 && r[1] < 1e-9);
// a thin object stays at most 2:1 (in photo pixels, 1.5:1 photo)
let r = detail_region(&seg_box(100, 10, 50, 90, 52), 1.5).unwrap();
let (w, h) = ((r[2] - r[0]) * 1.5, r[3] - r[1]);
assert!(w / h <= 2.0 + 1e-9, "{w} × {h}");
// a hostile aspect ratio stays sane
assert!(detail_region(&seg_box(100, 40, 45, 50, 55), f64::NAN).is_some_and(|r| r.iter().all(|v| v.is_finite())));
}
#[test]
fn no_region_for_nothing_or_most_of_the_photo() {
assert!(detail_region(&seg_box(50, 0, 0, 0, 0), 1.0).is_none());
assert!(detail_region(&seg_box(50, 2, 2, 48, 48), 1.0).is_none());
let damaged = SegMask { side: 50, data: "?".into(), rect: None };
assert!(detail_region(&damaged, 1.0).is_none());
}
}
+426
View File
@@ -0,0 +1,426 @@
//! The SAM 3 worker thread: it owns the model and the last encoded photo, and runs every
//! request (load, render the model input, encode, clicks, descriptions, detail passes) in
//! turn. The session only sends jobs and receives outcomes over channels, so no thread ever
//! waits on a lock the model holds, and the UI thread never runs the model.
//!
//! Every job runs under `catch_unwind`: a panic inside candle becomes an error outcome, drops
//! the (possibly inconsistent) model and leaves the worker ready for the next job. The busy
//! counters are decremented by drop guards, so they can't stay stuck. The model is unloaded
//! after [`IDLE_UNLOAD`] without requests (it holds several GB).
use std::collections::VecDeque;
use std::panic::AssertUnwindSafe;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::mpsc::{self, RecvTimeoutError};
use std::time::Duration;
use lightcraft_develop::SegMask;
use lightcraft_segment::{Encoded, Sam3};
use super::{Input, Prompt, Tag};
use crate::media::RenderJob;
/// Unload the model after this long without a request.
pub const IDLE_UNLOAD: Duration = Duration::from_secs(10 * 60);
/// What to compute.
pub enum Kind {
/// Load the model and encode the photo (so the first click is fast).
Prepare,
Clicks(Vec<lightcraft_segment::Click>),
Text(String),
/// A zoomed-in pass over `region` (normalized) of the photo, rendered by the job's `render`.
Detail {
prompt: Prompt,
region: [f64; 4],
},
/// Panics (tests of the worker's recovery).
#[cfg(test)]
Panic,
}
/// One request.
pub struct Job {
/// The model folder.
pub dir: PathBuf,
/// Which photo and look the request is about (the encode cache's key).
pub key: u64,
/// Renders the model input when the cache doesn't hold `key` (always for a detail pass).
pub render: Option<RenderJob>,
/// The photo's file, when it is missing on disk (for a clearer error if rendering fails).
pub missing: Option<String>,
pub kind: Kind,
pub tag: Tag,
pub reply: mpsc::Sender<Outcome>,
}
/// A finished request: the segmentation (`None`: nothing selected / nothing to report).
pub struct Outcome {
pub tag: Tag,
pub result: Result<Option<SegMask>, String>,
/// A newer request for the same selection came in; this one was skipped.
pub superseded: bool,
}
/// Counters the session reads without locking.
#[derive(Default)]
pub struct Shared {
/// Requests other than detail passes, queued or running.
pub pending: AtomicUsize,
/// Detail passes queued or running.
pub detail: AtomicUsize,
/// The model is in memory.
pub loaded: AtomicBool,
/// Loading the model or encoding a photo right now (the slow parts).
pub encoding: AtomicBool,
}
/// Decrements a counter when dropped (also when unwinding).
struct Done<'a>(&'a AtomicUsize);
impl Drop for Done<'_> {
fn drop(&mut self) {
let _ = self.0.try_update(Ordering::SeqCst, Ordering::SeqCst, |n| Some(n.saturating_sub(1)));
}
}
/// Sets a flag while alive.
struct Flag<'a>(&'a AtomicBool);
impl<'a> Flag<'a> {
fn raise(f: &'a AtomicBool) -> Self {
f.store(true, Ordering::SeqCst);
Flag(f)
}
}
impl Drop for Flag<'_> {
fn drop(&mut self) {
self.0.store(false, Ordering::SeqCst);
}
}
/// The session's handle on the worker thread (started on the first request, restarted if it
/// ever ends).
#[derive(Default)]
pub struct Worker {
tx: Option<mpsc::Sender<Job>>,
thread: Option<std::thread::JoinHandle<()>>,
pub shared: Arc<Shared>,
}
impl Worker {
fn counter(&self, job: &Job) -> &AtomicUsize {
if matches!(job.kind, Kind::Detail { .. }) { &self.shared.detail } else { &self.shared.pending }
}
/// Whether the thread has ended (it never should); its counters are then meaningless.
fn dead(&self) -> bool {
self.thread.as_ref().is_some_and(|t| t.is_finished())
}
pub fn pending(&self) -> usize {
if self.dead() { 0 } else { self.shared.pending.load(Ordering::SeqCst) }
}
pub fn detail(&self) -> usize {
if self.dead() { 0 } else { self.shared.detail.load(Ordering::SeqCst) }
}
pub fn encoding(&self) -> bool {
!self.dead() && self.shared.encoding.load(Ordering::SeqCst)
}
pub fn loaded(&self) -> bool {
!self.dead() && self.shared.loaded.load(Ordering::SeqCst)
}
/// Queue `job`; returns at once.
pub fn submit(&mut self, job: Job) -> Result<(), String> {
if self.dead() {
// a thread that ended took its counters' meaning with it
self.tx = None;
self.thread = None;
self.shared = Arc::new(Shared::default());
}
self.counter(&job).fetch_add(1, Ordering::SeqCst);
let job = match &self.tx {
Some(tx) => match tx.send(job) {
Ok(()) => return Ok(()),
Err(mpsc::SendError(job)) => job,
},
None => job,
};
// (re)start the thread
let (tx, rx) = mpsc::channel();
let shared = self.shared.clone();
match std::thread::Builder::new().name("sam3".into()).spawn(move || run(&rx, &shared, IDLE_UNLOAD)) {
Ok(t) => {
self.thread = Some(t);
self.tx = Some(tx.clone());
tx.send(job).map_err(|_| "the AI mask worker stopped".to_string())
}
Err(e) => {
let _ = self.counter(&job).try_update(Ordering::SeqCst, Ordering::SeqCst, |n| Some(n.saturating_sub(1)));
Err(format!("could not start the AI mask worker: {e}"))
}
}
}
}
/// The model and the last encoded photo.
#[derive(Default)]
struct State {
model: Option<(PathBuf, Sam3)>,
cache: Option<(u64, Encoded)>,
}
/// Whether `later` makes `job` pointless (it asks for the same selection again).
fn supersedes(later: &Job, job: &Job) -> bool {
match (&later.tag, &job.tag) {
(Tag::Prepare, Tag::Prepare) => later.key == job.key,
(Tag::Clicks { photo: a, mask: am, comp: ac, .. }, Tag::Clicks { photo: b, mask: bm, comp: bc, .. }) => (a, am, ac) == (b, bm, bc),
(Tag::Detail { photo: a, mask: am, comp: ac, .. }, Tag::Detail { photo: b, mask: bm, comp: bc, .. }) => (a, am, ac) == (b, bm, bc),
_ => false,
}
}
fn run(rx: &mpsc::Receiver<Job>, shared: &Shared, idle: Duration) {
let mut state = State::default();
let mut queue: VecDeque<Job> = VecDeque::new();
loop {
if queue.is_empty() {
match rx.recv_timeout(idle) {
Ok(j) => queue.push_back(j),
Err(RecvTimeoutError::Timeout) => {
if state.model.is_some() {
state = State::default();
shared.loaded.store(false, Ordering::SeqCst);
log::info!("SAM 3 unloaded after {} minutes without use", idle.as_secs() / 60);
}
continue;
}
Err(RecvTimeoutError::Disconnected) => return,
}
}
while let Ok(j) = rx.try_recv() {
queue.push_back(j);
}
let Some(mut job) = queue.pop_front() else { continue };
let is_detail = matches!(job.kind, Kind::Detail { .. });
let _done = Done(if is_detail { &shared.detail } else { &shared.pending });
if queue.iter().any(|later| supersedes(later, &job)) {
let _ = job.reply.send(Outcome { tag: job.tag, result: Ok(None), superseded: true });
continue;
}
let result = match std::panic::catch_unwind(AssertUnwindSafe(|| run_job(&mut state, shared, &mut job))) {
Ok(r) => r,
Err(p) => {
let why = p.downcast_ref::<&str>().map(|s| s.to_string()).or_else(|| p.downcast_ref::<String>().cloned()).unwrap_or_default();
log::error!("SAM 3 panicked: {why}");
// don't trust what the model kept: load it again next time
state = State::default();
Err("the AI model failed unexpectedly (it will be reloaded on the next try)".to_string())
}
};
if result.is_err() && !is_detail {
state.cache = None;
}
shared.loaded.store(state.model.is_some(), Ordering::SeqCst);
let _ = job.reply.send(Outcome { tag: job.tag, result, superseded: false });
}
}
/// The model for `dir`, loading it when needed.
fn model<'a>(state: &'a mut State, shared: &Shared, dir: &std::path::Path) -> Result<&'a mut Sam3, String> {
if state.model.as_ref().is_none_or(|(d, _)| d != dir) {
state.model = None;
state.cache = None;
let _busy = Flag::raise(&shared.encoding);
let t = web_time::Instant::now();
let m = Sam3::load(dir).map_err(|e| e.to_string())?;
log::info!("SAM 3 loaded in {:?}", t.elapsed());
state.model = Some((dir.to_path_buf(), m));
shared.loaded.store(true, Ordering::SeqCst);
}
state.model.as_mut().map(|(_, m)| m).ok_or_else(|| "the model did not load".to_string())
}
/// Render the model input.
fn render(job: &mut Job) -> Result<Input, String> {
let r = job.render.take().ok_or("the photo is not available")?.run();
let img = match r.rendered {
Ok(r) => r.image,
Err(e) => {
return Err(match &job.missing {
Some(path) => format!(
"This photo's file is missing (moved or deleted), so it can't be analyzed for AI masks: {path}. Library ▸ Find Missing Photos can relink it."
),
None => format!("couldn't render the photo for AI masks: {e}"),
});
}
};
let rgb = img.data.iter().flat_map(|p| [p[0], p[1], p[2]]).collect();
Ok(Input { rgb, w: img.width, h: img.height })
}
/// The model and the encoded photo `job.key` (rendering and encoding it when needed).
fn encoded<'a>(state: &'a mut State, shared: &Shared, job: &mut Job) -> Result<(&'a mut Sam3, &'a mut Encoded), String> {
model(state, shared, &job.dir)?;
if state.cache.as_ref().is_none_or(|(k, _)| *k != job.key) {
state.cache = None;
let input = render(job)?;
let _busy = Flag::raise(&shared.encoding);
let m = state.model.as_mut().map(|(_, m)| m).ok_or("the model did not load")?;
let t = web_time::Instant::now();
let mut enc = m.encode(&input.rgb, input.w, input.h).map_err(|e| e.to_string())?;
m.prepare_clicks(&mut enc).map_err(|e| e.to_string())?;
log::info!("SAM 3 encoded {}×{} in {:?}", input.w, input.h, t.elapsed());
state.cache = Some((job.key, enc));
}
let State { model, cache } = state;
match (model.as_mut(), cache.as_mut()) {
(Some((_, m)), Some((_, e))) => Ok((m, e)),
_ => Err("the model is not ready".into()),
}
}
/// Every phrase of `text` ("car, road"), merged (the per-pixel maximum); `None` when nothing
/// matches.
fn text_logits(model: &mut Sam3, enc: &mut Encoded, text: &str) -> Result<Option<Vec<f32>>, String> {
let mut merged: Option<Vec<f32>> = None;
for phrase in text.split([',', ';']).map(str::trim).filter(|p| !p.is_empty()) {
if let Some(p) = model.segment_text(enc, phrase, 0.5).map_err(|e| e.to_string())? {
merged = Some(match merged {
None => p.logits,
Some(m) => m.iter().zip(&p.logits).map(|(a, b)| a.max(*b)).collect(),
});
}
}
Ok(merged)
}
fn run_job(state: &mut State, shared: &Shared, job: &mut Job) -> Result<Option<SegMask>, String> {
let side = lightcraft_segment::MASK_SIDE;
match std::mem::replace(&mut job.kind, Kind::Prepare) {
Kind::Prepare => {
encoded(state, shared, job)?;
Ok(None)
}
Kind::Clicks(clicks) => {
let (m, enc) = encoded(state, shared, job)?;
let p = m.segment_clicks(enc, &clicks).map_err(|e| e.to_string())?;
Ok(Some(SegMask::from_logits(side, &p.logits)))
}
Kind::Text(text) => {
let (m, enc) = encoded(state, shared, job)?;
Ok(text_logits(m, enc, &text)?.map(|l| SegMask::from_logits(side, &l)))
}
#[cfg(test)]
#[allow(clippy::panic)]
Kind::Panic => panic!("a test panic"),
Kind::Detail { prompt, region } => {
let input = render(job)?;
let m = model(state, shared, &job.dir)?;
let t = web_time::Instant::now();
let mut enc = m.encode(&input.rgb, input.w, input.h).map_err(|e| e.to_string())?;
let (rw, rh) = (region[2] - region[0], region[3] - region[1]);
if !(rw > 0.0 && rh > 0.0) {
return Ok(None);
}
let logits = match prompt {
Prompt::Clicks(clicks) => {
let inside: Vec<lightcraft_segment::Click> = clicks
.iter()
.map(|c| ((c.at.x - region[0]) / rw, (c.at.y - region[1]) / rh, c.include))
.filter(|(x, y, _)| (0.0..=1.0).contains(x) && (0.0..=1.0).contains(y))
.map(|(x, y, positive)| lightcraft_segment::Click { x: x as f32, y: y as f32, positive })
.collect();
if !inside.iter().any(|c| c.positive) {
return Ok(None);
}
Some(m.segment_clicks(&mut enc, &inside).map_err(|e| e.to_string())?.logits)
}
Prompt::Text(text) => text_logits(m, &mut enc, &text)?,
};
log::info!("SAM 3 detail pass ({}×{}) in {:?}", input.w, input.h, t.elapsed());
Ok(logits.map(|l| SegMask::from_logits_in(side, &l, region)))
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn job(tag: Tag, reply: &mpsc::Sender<Outcome>) -> Job {
// an empty folder: loading fails with an error (no model files)
let dir = std::env::temp_dir().join(format!("lc-sam3-worker-none-{}", std::process::id()));
Job { dir, key: 1, render: None, missing: None, kind: Kind::Prepare, tag, reply: reply.clone() }
}
#[test]
fn errors_come_back_and_the_counters_return_to_zero() {
let mut w = Worker::default();
let (tx, rx) = mpsc::channel();
for _ in 0..3 {
w.submit(job(Tag::Wait, &tx)).unwrap();
}
for _ in 0..3 {
let o = rx.recv_timeout(Duration::from_secs(30)).unwrap();
assert!(o.result.unwrap_err().contains("not found"));
}
let t = web_time::Instant::now();
while w.pending() > 0 && t.elapsed() < Duration::from_secs(5) {
std::thread::yield_now();
}
assert_eq!(w.pending(), 0);
assert!(!w.encoding() && !w.loaded());
}
#[test]
fn a_panic_in_a_job_is_an_error_and_the_worker_carries_on() {
let shared = Shared::default();
let (tx, rx) = mpsc::channel();
let (jtx, jrx) = mpsc::channel();
let mut j = job(Tag::Wait, &tx);
j.kind = Kind::Panic;
jtx.send(j).unwrap();
jtx.send(job(Tag::Wait, &tx)).unwrap();
shared.pending.store(2, Ordering::SeqCst);
drop(jtx);
run(&jrx, &shared, Duration::from_secs(60));
let first = rx.recv().unwrap();
assert!(first.result.unwrap_err().contains("failed unexpectedly"));
let second = rx.recv().unwrap();
assert!(second.result.is_err(), "the next job still runs");
assert_eq!(shared.pending.load(Ordering::SeqCst), 0, "counters reset");
assert!(!shared.encoding.load(Ordering::SeqCst));
}
#[test]
fn a_newer_request_for_the_same_selection_supersedes_a_queued_one() {
let (tx, rx) = mpsc::channel();
let (jtx, jrx) = mpsc::channel();
let clicks = |n: usize| Tag::Clicks {
photo: lightcraft_catalog::PhotoId(1),
mask: 1,
comp: 0,
hint: vec![lightcraft_geom::Point::new(0.5, 0.5); n],
exclude: vec![],
};
for n in 1..=3 {
jtx.send(job(clicks(n), &tx)).unwrap();
}
drop(jtx);
let shared = Shared::default();
shared.pending.store(3, Ordering::SeqCst);
run(&jrx, &shared, Duration::from_secs(60));
let out: Vec<Outcome> = rx.try_iter().collect();
assert_eq!(out.len(), 3);
assert!(out[0].superseded && out[1].superseded && !out[2].superseded);
assert_eq!(shared.pending.load(Ordering::SeqCst), 0);
}
}
+10
View File
@@ -99,6 +99,7 @@ pub struct SidecarData {
pub alt_text: Option<String>,
pub extended_description: Option<String>,
pub keywords: Option<Vec<String>>,
pub regions: Option<Vec<lightcraft_meta::Region>>,
/// Capture time (ISO 8601) from `exif:DateTimeOriginal`, `photoshop:DateCreated` or
/// `xmp:CreateDate` (first found). Used only when the file itself has no capture time.
pub captured: Option<String>,
@@ -133,6 +134,12 @@ pub fn parse_sidecar(xmp: &str, raw: bool) -> std::result::Result<SidecarData, S
alt_text: m.alt_text.clone(),
extended_description: m.extended_description.clone(),
keywords: (!m.keywords.is_empty()).then(|| m.keywords.clone()),
// A sidecar that has `mwg-rs:Regions` at all (even an empty list) was written by an app that
// knows about regions, so it's authoritative: its list, empty or not, replaces the catalog's.
// One without it (most writers, LightCraft's own included, which keeps another app's
// `mwg-rs:Regions` byte for byte but never writes one) says nothing about regions, and the
// photo's are kept.
regions: d.values.contains_key("mwg-rs:Regions").then(|| m.regions.clone()),
captured: m.capture_time.map(|d| d.to_iso()),
..Default::default()
};
@@ -202,6 +209,9 @@ pub fn merge_into(p: &mut Photo, sc: &SidecarData, now: &str) -> bool {
if let Some(k) = &sc.keywords {
m.keywords = k.clone();
}
if let Some(r) = &sc.regions {
m.regions = r.clone();
}
let develop = match &sc.develop {
Some(DevelopPatch::Full(s)) => (**s).clone(),
Some(DevelopPatch::Partial(v)) => {
+90
View File
@@ -727,3 +727,93 @@ fn curve_reset_by_channel_and_whole() {
assert_eq!(active_dev(&s).curve, lightcraft_develop::ToneCurve::default());
assert!(s.execute("curve.reset", &json!({"channel": "alpha"})).is_err());
}
#[test]
fn ai_masks_without_the_model_explain_themselves() {
let mut s = demo();
let id = s.active().unwrap();
// an Object mask that could never be computed is refused, not left empty
let e = s.execute("mask.add", &json!({"kind": "object"})).unwrap_err().to_string();
assert!(e.contains("SAM 3") || e.contains("not available"), "{e}");
assert!(!e.contains("invalid parameters"), "shown as is: {e}");
assert!(s.develop_of(id).unwrap().masks.is_empty());
// an empty Object component (as an older document may hold) still takes clicks
let shape = lightcraft_develop::MaskShape::Object { hint: vec![], exclude: vec![], seg: None, detail: vec![], edge: 0.0 };
let mut d = (*s.develop_of(id).unwrap()).clone();
d.masks.push(lightcraft_develop::Mask {
id: 1,
components: vec![lightcraft_develop::MaskComponent { name: None, op: lightcraft_develop::MaskOp::Add, invert: false, shape }],
..Default::default()
});
s.set_develop(id, d, "test").unwrap();
s.active_mask = Some(1);
// clicks and descriptions need the model: an error that says what's missing, nothing changed
let e = s.execute("mask.objectPoint", &json!({"x": 0.5, "y": 0.5})).unwrap_err().to_string();
assert!(e.contains("SAM 3") || e.contains("not available"), "{e}");
let e = s.execute("mask.add", &json!({"kind": "prompt", "text": "sky"})).unwrap_err().to_string();
assert!(e.contains("SAM 3") || e.contains("not available"), "{e}");
let empty = std::env::temp_dir().join(format!("lc-no-sam3-{}", std::process::id()));
s.segmenter.dir = Some(empty.clone());
let e = s.execute("mask.objectPoint", &json!({"x": 0.5, "y": 0.5})).unwrap_err().to_string();
assert!(e.contains(&empty.display().to_string()) || e.contains("not available"), "names the folder: {e}");
assert_eq!(s.develop_of(id).unwrap().masks.len(), 1);
// bad input is an error, not a crash
assert!(s.execute("mask.objectPoint", &json!({"x": 1.5, "y": 0.5})).is_err());
assert!(s.execute("mask.add", &json!({"kind": "prompt", "text": " "})).is_err());
}
/// A People card's close-up is a square (in pixels) inside the photo, around the face, whatever the
/// face rectangle says: corners, oversized and degenerate rectangles never leave the frame or panic.
#[test]
fn face_job_crops_a_square_inside_the_photo() {
use lightcraft_geom::Rect;
let mut s = demo();
let id = s.active().unwrap();
let (w, h) = {
let p = s.catalog.photo(id).unwrap();
(f64::from(p.width), f64::from(p.height))
};
let faces = [
Rect { x0: 0.40, y0: 0.30, x1: 0.50, y1: 0.45 },
Rect { x0: 0.0, y0: 0.0, x1: 0.05, y1: 0.05 },
Rect { x0: 0.95, y0: 0.95, x1: 1.0, y1: 1.0 },
Rect { x0: -5.0, y0: -5.0, x1: 5.0, y1: 5.0 },
Rect { x0: 0.5, y0: 0.5, x1: 0.5, y1: 0.5 },
Rect { x0: f64::NAN, y0: 0.5, x1: 0.6, y1: f64::INFINITY },
];
for face in faces {
let job = s.face_job(id, face, 256).expect("a job for a photo in the library");
let r = job.settings.crop.geometry.rect;
assert!((0.0..=1.0).contains(&r.x0) && (0.0..=1.0).contains(&r.y0) && r.x1 <= 1.0 + 1e-9 && r.y1 <= 1.0 + 1e-9, "{face:?} → {r:?}");
assert!(r.x1 > r.x0 && r.y1 > r.y0, "{face:?} → {r:?}");
let (pw, ph) = ((r.x1 - r.x0) * w, (r.y1 - r.y0) * h);
assert!((pw - ph).abs() < 1e-6 * w.max(h), "square in pixels: {face:?} → {pw} × {ph}");
assert_eq!(job.settings.crop.geometry.angle, 0.0);
}
// a face around (0.45, 0.375) is centred in its crop (room to spare on every side)
let r = s.face_job(id, faces[0], 256).unwrap().settings.crop.geometry.rect;
assert!((((r.x0 + r.x1) / 2.0) - 0.45).abs() < 1e-9 && (((r.y0 + r.y1) / 2.0) - 0.375).abs() < 1e-9, "{r:?}");
assert!(s.face_job(lightcraft_catalog::PhotoId(u64::MAX), faces[0], 256).is_none());
}
/// Regions are on the upright photo; after Rotate Right the close-up still frames the face: the box
/// is carried into the rotated frame the crop lives in, and the crop stays square in rotated pixels.
#[test]
fn face_job_follows_rotate_right() {
use lightcraft_geom::Rect;
let mut s = demo();
let id = s.active().unwrap();
let (w, h) = {
let p = s.catalog.photo(id).unwrap();
(f64::from(p.width), f64::from(p.height))
};
// a small face left of centre on the upright photo
let face = Rect { x0: 0.30, y0: 0.45, x1: 0.35, y1: 0.50 };
s.execute("photo.rotateRight", &json!({})).unwrap();
let r = s.face_job(id, face, 256).unwrap().settings.crop.geometry.rect;
// a quarter turn clockwise puts the left of the photo at the top: centre (0.325, 0.475) → (0.525, 0.325)
assert!((((r.x0 + r.x1) / 2.0) - 0.525).abs() < 1e-9 && (((r.y0 + r.y1) / 2.0) - 0.325).abs() < 1e-9, "{r:?}");
// the rotated photo is h × w pixels
let (pw, ph) = ((r.x1 - r.x0) * h, (r.y1 - r.y0) * w);
assert!((pw - ph).abs() < 1e-6 * w.max(h), "square in rotated pixels: {pw} × {ph}");
}
+19
View File
@@ -201,6 +201,25 @@ fn album_source_reports_in_library_state_and_persists_the_view() {
let _ = std::fs::remove_dir_all(&dir);
}
/// A library opens unfiltered: the source and sort come back, a filter from the last session does not
/// (it would hide photos behind a small badge).
#[test]
fn filters_are_not_restored_when_a_library_reopens() {
let dir = temp_dir("filter-not-restored");
let mut s = open(&dir);
let all = s.visible_cloned().len();
s.execute("library.sort", &json!({"key": "fileName"})).unwrap();
s.execute("library.filter", &json!({"rating": 5, "date": "2026-01-16", "person": "Jane Doe"})).unwrap();
assert!(s.visible_cloned().len() < all);
s.close_library().unwrap();
drop(s);
let mut s2 = open(&dir);
assert_eq!(s2.filter, lightcraft_catalog::Filter::default());
assert_eq!(s2.visible_cloned().len(), all);
assert_eq!(serde_json::to_value(s2.sort).unwrap()["key"], "fileName", "the rest of the view is restored");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn filter_chips_clear_one_filter_and_total_ignores_the_filter() {
let dir = temp_dir("chips");
+232
View File
@@ -0,0 +1,232 @@
//! AI masks without the SAM 3 model (it is never required): every request is a clear error that
//! comes back at once, nothing waits on the network or the model, a damaged model is an error
//! (not a crash or a stuck "busy"), and the download command needs consent and fails cleanly.
//! No test touches the internet: downloads go to a local server.
use std::time::{Duration, Instant};
use serde_json::json;
use crate::Session;
use crate::segment::{NOT_INSTALLED, Segmenter};
fn tmp(name: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-sam3-engine-{name}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
d
}
/// A folder that looks installed but holds no usable model.
fn damaged_model(name: &str) -> std::path::PathBuf {
let d = tmp(name);
std::fs::create_dir_all(&d).unwrap();
std::fs::write(d.join("model.safetensors"), b"\x10\x00\x00\x00\x00\x00\x00\x00{not json at all}").unwrap();
std::fs::write(d.join("vocab.json"), b"{}").unwrap();
std::fs::write(d.join("merges.txt"), b"#version: 0.2\n").unwrap();
d
}
/// Poll until the worker is idle (or `limit` passes), collecting messages.
fn settle(s: &mut Session, limit: Duration) -> Vec<String> {
let t = Instant::now();
let mut messages = Vec::new();
loop {
let p = s.segment_poll();
messages.extend(p.messages);
if !s.segmenter.busy() && !s.segmenter.detail_busy() {
messages.extend(s.segment_poll().messages);
return messages;
}
assert!(t.elapsed() < limit, "the worker never finished");
std::thread::sleep(Duration::from_millis(10));
}
}
#[test]
fn status_and_model_commands_without_the_model() {
let mut s = Session::with_demo();
let st = s.execute("segment.model.status", &json!({})).unwrap();
assert_eq!(st["available"], Segmenter::AVAILABLE);
assert_eq!(st["installed"], false);
assert_eq!(st["download"]["running"], false);
assert!(st["license"].as_str().unwrap().contains("SAM License"));
// downloading needs the user's consent, and names the size and the licence
let e = s.execute("segment.model.download", &json!({})).unwrap_err().to_string();
assert!(e.contains("acknowledged") && e.contains("SAM License") && e.contains("GB"), "{e}");
let e = s.execute("segment.model.download", &json!({"acknowledged": "yes"})).unwrap_err().to_string();
assert!(e.contains("acknowledged"), "{e}");
assert_eq!(s.execute("segment.model.cancel", &json!({})).unwrap()["cancelled"], false);
if !Segmenter::AVAILABLE {
assert!(s.execute("segment.model.download", &json!({"acknowledged": true})).is_err());
return;
}
// no folder, then no download location: clear errors, nothing started
let e = s.execute("segment.model.download", &json!({"acknowledged": true})).unwrap_err().to_string();
assert!(e.contains("no folder"), "{e}");
let dir = tmp("nomirrors");
s.segmenter.dir = Some(dir.clone());
s.segmenter.mirrors_file = Some(dir.join("none.txt"));
if s.segmenter.mirrors().is_empty() {
let e = s.execute("segment.model.download", &json!({"acknowledged": true})).unwrap_err().to_string();
assert!(e.contains("LIGHTCRAFT_SAM3_MIRRORS"), "{e}");
}
assert!(!s.segmenter.download_status().running);
}
#[cfg(feature = "sam")]
#[test]
fn a_failing_download_ends_with_an_error_and_never_blocks() {
use std::io::{BufRead, BufReader, Write};
// a local mirror that has nothing
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let base = format!("http://{}/sam3", l.local_addr().unwrap());
std::thread::spawn(move || {
for mut c in l.incoming().flatten() {
let mut r = BufReader::new(c.try_clone().unwrap());
let mut line = String::new();
while r.read_line(&mut line).unwrap_or(0) > 2 {
line.clear();
}
let _ = c.write_all(b"HTTP/1.1 404 Not Found\r\nContent-Length: 13\r\nConnection: close\r\n\r\n<h1>404</h1>\n");
}
});
let dir = tmp("download404");
let mirrors = dir.with_extension("mirrors.txt");
std::fs::write(&mirrors, format!("# test mirror\n{base}\n")).unwrap();
let mut s = Session::with_demo();
s.segmenter.dir = Some(dir.clone());
s.segmenter.mirrors_file = Some(mirrors.clone());
let t = Instant::now();
let r = s.execute("segment.model.download", &json!({"acknowledged": true})).unwrap();
assert_eq!(r["started"], true);
assert!(t.elapsed() < Duration::from_secs(1), "starting a download returns at once");
// a second request while it runs starts nothing
let again = s.execute("segment.model.download", &json!({"acknowledged": true})).unwrap();
assert!(again["started"] == false || again["downloading"] == false);
let t = Instant::now();
while s.segmenter.download_status().running {
assert!(t.elapsed() < Duration::from_secs(30), "the download never ended");
// AI mask requests meanwhile: an error at once, never a wait
let q = Instant::now();
let e = s.execute("mask.add", &json!({"kind": "prompt", "text": "sky"})).unwrap_err().to_string();
assert!(e.contains(NOT_INSTALLED), "{e}");
assert!(q.elapsed() < Duration::from_millis(500));
std::thread::sleep(Duration::from_millis(20));
}
let st = s.segmenter.download_status();
assert!(!st.finished);
let e = st.error.unwrap();
assert!(e.contains("not found"), "{e}");
assert!(!s.segmenter.installed());
// nothing was left behind under a real name, and no error page was saved
for f in ["model.safetensors", "vocab.json", "merges.txt", "vocab.json.part"] {
assert!(!dir.join(f).exists(), "{f}");
}
let _ = std::fs::remove_dir_all(&dir);
let _ = std::fs::remove_file(&mirrors);
}
/// The app (background mode) never waits: with the model missing every request fails at once;
/// with a damaged model the requests are queued, fail on the worker, and come back as messages.
#[test]
fn background_requests_never_wait_and_failures_come_back_as_messages() {
let mut s = Session::with_demo();
s.segmenter.background = true;
let id = s.active().unwrap();
s.segmenter.dir = Some(tmp("missing"));
for (cmd, p) in [("mask.add", json!({"kind": "object"})), ("mask.add", json!({"kind": "prompt", "text": "sky"})), ("segment.prepare", json!({}))]
{
let t = Instant::now();
let e = s.execute(cmd, &p).unwrap_err().to_string();
assert!(t.elapsed() < Duration::from_millis(500), "{cmd}");
if Segmenter::AVAILABLE {
assert!(e.starts_with(NOT_INSTALLED), "{cmd}: {e}");
} else {
assert!(e.contains("not available"), "{cmd}: {e}");
}
}
assert!(s.develop_of(id).unwrap().masks.is_empty());
if !Segmenter::AVAILABLE {
return;
}
// a model folder whose weights are damaged
let dir = damaged_model("damaged");
s.segmenter.dir = Some(dir.clone());
assert!(s.segmenter.installed());
let t = Instant::now();
s.execute("mask.add", &json!({"kind": "object"})).unwrap();
let r = s.execute("mask.objectPoint", &json!({"x": 0.5, "y": 0.5})).unwrap();
assert_eq!(r["pending"], true);
assert!(s.segmenter.pending_clicks().is_some_and(|p| p.hint.len() == 1));
// a second click while the first is on its way builds on it
let r = s.execute("mask.objectPoint", &json!({"x": 0.6, "y": 0.5, "exclude": true})).unwrap();
assert_eq!((r["include"].as_u64(), r["exclude"].as_u64()), (Some(1), Some(1)));
let r = s.execute("mask.add", &json!({"kind": "prompt", "text": "the sky"})).unwrap();
assert_eq!(r["pending"], true);
assert!(t.elapsed() < Duration::from_secs(1), "nothing waited for the model: {:?}", t.elapsed());
let messages = settle(&mut s, Duration::from_secs(60));
assert!(!messages.is_empty() && messages.iter().all(|m| !m.is_empty()), "{messages:?}");
assert!(messages.iter().any(|m| m.contains("model.safetensors") || m.contains("SAM 3")), "{messages:?}");
assert!(s.segmenter.pending_clicks().is_none());
// the Object mask is there, without a selection; no Describe mask was made from nothing
let d = s.develop_of(id).unwrap();
assert_eq!(d.masks.len(), 1);
assert!(matches!(&d.masks[0].components[0].shape, lightcraft_develop::MaskShape::Object { seg: None, .. }));
assert!(!s.segmenter.busy() && !s.segmenter.loaded());
let _ = std::fs::remove_dir_all(&dir);
}
/// CLI / MCP (waiting mode): a damaged model is an error from the command, quickly.
#[test]
fn waiting_requests_with_a_damaged_model_are_errors() {
if !Segmenter::AVAILABLE {
return;
}
let mut s = Session::with_demo();
let id = s.active().unwrap();
let dir = damaged_model("damaged-wait");
s.segmenter.dir = Some(dir.clone());
let e = s.execute("mask.add", &json!({"kind": "prompt", "text": "sky"})).unwrap_err().to_string();
assert!(e.contains("model.safetensors") || e.contains("SAM 3"), "{e}");
let e = s.execute("mask.add", &json!({"kind": "object", "points": [[0.5, 0.5]]})).unwrap_err().to_string();
assert!(!e.is_empty());
assert!(s.develop_of(id).unwrap().masks.is_empty());
assert!(!s.segmenter.busy());
// a stored segmentation needs no model at all (replayed or pasted masks)
let seg = serde_json::to_value(lightcraft_develop::SegMask::from_logits(4, &[5.0; 16])).unwrap();
s.segmenter.dir = None;
s.execute("mask.add", &json!({"kind": "prompt", "text": "sky", "seg": seg})).unwrap();
assert_eq!(s.develop_of(id).unwrap().masks.len(), 1);
let _ = std::fs::remove_dir_all(&dir);
}
/// Too many clicks on one Object selection are refused, not sent to the model.
#[test]
fn object_clicks_are_capped() {
if !Segmenter::AVAILABLE {
return;
}
let mut s = Session::with_demo();
s.segmenter.background = true;
let dir = damaged_model("clicks");
s.segmenter.dir = Some(dir.clone());
let id = s.active().unwrap();
let mut d = (*s.develop_of(id).unwrap()).clone();
let many = vec![lightcraft_geom::Point::new(0.5, 0.5); crate::segment::MAX_CLICKS];
d.masks.push(lightcraft_develop::Mask {
id: 1,
components: vec![lightcraft_develop::MaskComponent {
name: None,
op: lightcraft_develop::MaskOp::Add,
invert: false,
shape: lightcraft_develop::MaskShape::Object { hint: many, exclude: vec![], seg: None, detail: vec![], edge: 0.0 },
}],
..Default::default()
});
s.set_develop(id, d, "test").unwrap();
s.active_mask = Some(1);
let e = s.execute("mask.objectPoint", &json!({"x": 0.5, "y": 0.5})).unwrap_err().to_string();
assert!(e.contains("at most"), "{e}");
assert!(!s.segmenter.busy());
let _ = std::fs::remove_dir_all(&dir);
}
+156
View File
@@ -173,6 +173,162 @@ fn foreign_crs_sidecar_on_import_and_read_from_file() {
let _ = std::fs::remove_dir_all(&src);
}
const MWG_REGIONS: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#"
xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
<mwg-rs:Regions>
<rdf:Description>
<mwg-rs:AppliedToDimensions stDim:w="100" stDim:h="100" stDim:unit="pixel"/>
<mwg-rs:RegionList>
<rdf:Bag>
<rdf:li>
<rdf:Description mwg-rs:Name="Jane Doe" mwg-rs:Type="Face">
<mwg-rs:Area stArea:x="0.5" stArea:y="0.5" stArea:w="0.3" stArea:h="0.4" stArea:unit="normalized"/>
</rdf:Description>
</rdf:li>
</rdf:Bag>
</mwg-rs:RegionList>
</rdf:Description>
</mwg-rs:Regions>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>"#;
/// A face region written by Lightroom (MWG-RS) into a sidecar reaches `Photo.meta.regions` on
/// import — the same path real catalogs use, exercised end to end rather than just at the parser.
#[test]
fn regions_from_sidecar_are_read_on_import() {
let src = temp_dir("regions");
write_png(&src.join("portrait.png"), 9);
std::fs::write(src.join("portrait.xmp"), MWG_REGIONS).unwrap();
let mut s = Session::new().with_fs();
let r = s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
assert_eq!(r["sidecars"], 1);
let p = s.catalog.photos().next().unwrap().clone();
assert_eq!(p.meta.regions.len(), 1);
let region = &p.meta.regions[0];
assert_eq!(region.name.as_deref(), Some("Jane Doe"));
assert_eq!(region.kind, lightcraft_meta::RegionKind::Face);
assert!((region.rect.width() - 0.3).abs() < 1e-9, "{:?}", region.rect);
let _ = std::fs::remove_dir_all(&src);
}
/// Removing a face box is a catalog-only, undoable edit: it never rewrites the sidecar (Lightroom's
/// regions and everything else in it stay byte for byte), even with auto-write on.
#[test]
fn removing_a_region_is_undoable_and_leaves_the_sidecar_alone() {
let src = temp_dir("region-remove");
write_png(&src.join("portrait.png"), 9);
std::fs::write(src.join("portrait.xmp"), MWG_REGIONS).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
s.execute("library.xmpPreferences", &json!({"autoWrite": true})).unwrap();
let before = std::fs::read(src.join("portrait.xmp")).unwrap();
let n = s.catalog.photo(id).unwrap().meta.regions.len();
assert!(n >= 1);
assert!(s.execute("photo.removeRegion", &json!({"index": n})).is_err(), "out of range");
assert!(s.execute("photo.removeRegion", &json!({})).is_err(), "no index");
let r = s.execute("photo.removeRegion", &json!({"index": 0})).unwrap();
assert_eq!(r["removed"], "Jane Doe");
assert_eq!(s.catalog.photo(id).unwrap().meta.regions.len(), n - 1);
assert_eq!(std::fs::read(src.join("portrait.xmp")).unwrap(), before, "the sidecar is untouched");
// an ordinary edit still auto-writes (the guard is per command)
s.execute("photo.rate", &json!({"rating": 4})).unwrap();
assert_ne!(std::fs::read(src.join("portrait.xmp")).unwrap(), before, "auto-write still works for other edits");
// undo brings the region back
s.execute("edit.undo", &json!({})).unwrap();
s.execute("edit.undo", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().meta.regions.len(), n);
let _ = std::fs::remove_dir_all(&src);
}
/// Resizing a face box is a catalog-only, undoable edit with strict input handling; it leaves the
/// sidecar alone like removal does.
#[test]
fn resizing_a_region_validates_clamps_undoes_and_leaves_the_sidecar_alone() {
let src = temp_dir("region-resize");
write_png(&src.join("portrait.png"), 9);
std::fs::write(src.join("portrait.xmp"), MWG_REGIONS).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
s.execute("library.xmpPreferences", &json!({"autoWrite": true})).unwrap();
let before = std::fs::read(src.join("portrait.xmp")).unwrap();
let original = s.catalog.photo(id).unwrap().meta.regions[0].rect;
let set = |s: &mut Session, rect: serde_json::Value| s.execute("photo.setRegion", &json!({"index": 0, "rect": rect}));
// hostile input is an error and changes nothing
for bad in [
json!({"x0": 0.1, "y0": 0.1, "x1": 0.1, "y1": 0.5}),
json!({"x0": 0.1, "y0": 0.1, "x1": 0.1001, "y1": 0.5}),
json!({"x0": 0.1, "y0": 0.1, "x1": 0.5}),
json!({"x0": "a", "y0": 0.1, "x1": 0.5, "y1": 0.5}),
json!({"x0": null, "y0": 0.1, "x1": 0.5, "y1": 0.5}),
] {
assert!(set(&mut s, bad.clone()).is_err(), "{bad}");
}
assert!(s.execute("photo.setRegion", &json!({"index": 9, "rect": {"x0": 0.1, "y0": 0.1, "x1": 0.5, "y1": 0.5}})).is_err(), "no such region");
assert!(s.execute("photo.setRegion", &json!({"index": 0})).is_err(), "no rect");
assert_eq!(s.catalog.photo(id).unwrap().meta.regions[0].rect, original);
// corners given in any order and outside the photo are clamped into it
let r = set(&mut s, json!({"x0": 0.6, "y0": 1.5, "x1": -0.2, "y1": 0.2})).unwrap();
assert_eq!(r["rect"], json!({"x0": 0.0, "y0": 0.2, "x1": 0.6, "y1": 1.0}));
let now = s.catalog.photo(id).unwrap().meta.regions[0].rect;
assert_eq!((now.x0, now.y0, now.x1, now.y1), (0.0, 0.2, 0.6, 1.0));
assert_eq!(std::fs::read(src.join("portrait.xmp")).unwrap(), before, "the sidecar is untouched");
s.execute("edit.undo", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().meta.regions[0].rect, original);
let _ = std::fs::remove_dir_all(&src);
}
/// Re-reading a sidecar: one that has `mwg-rs:Regions` is authoritative, so an emptied region list
/// clears the stale regions; one without `mwg-rs:Regions` (an app that doesn't do regions) leaves the
/// photo's regions alone.
#[test]
fn rereading_a_sidecar_clears_regions_only_when_it_states_them() {
let src = temp_dir("region-reread");
write_png(&src.join("portrait.png"), 9);
std::fs::write(src.join("portrait.xmp"), MWG_REGIONS).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
assert_eq!(s.catalog.photo(id).unwrap().meta.regions.len(), 1);
// another app rewrote the sidecar without any notion of regions: they stay
std::fs::write(
src.join("portrait.xmp"),
r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/" xmp:Rating="3"/>
</rdf:RDF></x:xmpmeta>"#,
)
.unwrap();
s.execute("photo.readMetadataFromFile", &json!({})).unwrap();
let p = s.catalog.photo(id).unwrap();
assert_eq!(p.rating, 3, "the sidecar was read");
assert_eq!(p.meta.regions.len(), 1, "a sidecar without mwg-rs:Regions keeps the photo's regions");
// a region-aware app removed every region: the sidecar says so, and the catalog follows
let emptied = MWG_REGIONS.replace(
r#"<rdf:li>
<rdf:Description mwg-rs:Name="Jane Doe" mwg-rs:Type="Face">
<mwg-rs:Area stArea:x="0.5" stArea:y="0.5" stArea:w="0.3" stArea:h="0.4" stArea:unit="normalized"/>
</rdf:Description>
</rdf:li>"#,
"",
);
assert_ne!(emptied, MWG_REGIONS);
std::fs::write(src.join("portrait.xmp"), emptied).unwrap();
s.execute("photo.readMetadataFromFile", &json!({})).unwrap();
assert!(s.catalog.photo(id).unwrap().meta.regions.is_empty(), "an emptied region list clears stale regions");
// undo brings them back
s.execute("edit.undo", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().meta.regions.len(), 1);
let _ = std::fs::remove_dir_all(&src);
}
fn synthetic_dng(xmp: &str) -> Vec<u8> {
synthetic_dng_with(Some(xmp), lightcraft_meta::Metadata::default())
}
+3 -1
View File
@@ -163,7 +163,9 @@ pub fn filter_chips(f: &Filter, cat: &Catalog) -> Vec<FilterChip> {
if let Some(e) = f.edited {
add(if e { "Edited".into() } else { "Unedited".into() }, json!({"edited": Null}));
}
for (name, key, val) in [("Camera", "camera", &f.camera), ("Lens", "lens", &f.lens), ("Keyword", "keyword", &f.keyword)] {
for (name, key, val) in
[("Camera", "camera", &f.camera), ("Lens", "lens", &f.lens), ("Keyword", "keyword", &f.keyword), ("Person", "person", &f.person)]
{
if let Some(x) = val {
add(format!("{name}: {x}"), json!({key: Null}));
}
+30
View File
@@ -332,6 +332,23 @@ impl Orientation {
// flipH ∘ rot(t) = rot(-t) ∘ flipH
Orientation::from_parts(!f, (4 - t) % 4)
}
/// The orientation that undoes this one (`o.inverse().map(o.map(p)) == p`). Flips and half turns
/// are their own inverses; quarter turns swap.
pub fn inverse(self) -> Orientation {
match self {
Orientation::Rotate90 => Orientation::Rotate270,
Orientation::Rotate270 => Orientation::Rotate90,
o => o,
}
}
/// Map a rectangle in normalized (0..1) coordinates of the stored image to normalized coordinates
/// of the oriented one (e.g. a face region given on the upright photo → the frame after the user's
/// Rotate Left/Right).
pub fn map_norm_rect(self, r: Rect) -> Rect {
let (ax, ay) = self.map(r.x0, r.y0, 1.0, 1.0);
let (bx, by) = self.map(r.x1, r.y1, 1.0, 1.0);
Rect::from_points(Point::new(ax, ay), Point::new(bx, by))
}
/// Map a point in stored pixel space (w×h stored dims) to oriented space.
pub fn map(self, x: f64, y: f64, w: f64, h: f64) -> (f64, f64) {
let (flip, t) = self.to_parts();
@@ -417,6 +434,19 @@ mod tests {
assert_eq!(Orientation::Rotate90.flipped_h().flipped_h(), Orientation::Rotate90);
}
#[test]
fn orientation_inverse_and_norm_rect() {
let r = Rect::new(0.1, 0.2, 0.3, 0.6);
for v in 1..=8 {
let o = Orientation::from_exif(v);
let back = o.inverse().map_norm_rect(o.map_norm_rect(r));
assert!([back.x0 - r.x0, back.y0 - r.y0, back.x1 - r.x1, back.y1 - r.y1].iter().all(|d| d.abs() < 1e-12), "{o:?}: {back:?}");
}
// a quarter turn clockwise: the left of the upright photo becomes its top
let q = Orientation::Rotate90.map_norm_rect(r);
assert!([q.x0 - 0.4, q.y0 - 0.1, q.x1 - 0.8, q.y1 - 0.3].iter().all(|d| d.abs() < 1e-12), "{q:?}");
}
proptest::proptest! {
#[test]
fn affine_inverse_prop(a in -3.0f64..3.0, sx in 0.1f64..5.0, sy in 0.1f64..5.0, tx in -50.0f64..50.0, px in -20.0f64..20.0, py in -20.0f64..20.0) {
+4 -2
View File
@@ -233,14 +233,16 @@ pub struct Gpu {
/// WGSL constants shared by every module (generated from the CPU pipeline's values).
fn constants() -> String {
use lightcraft_pipeline::finish::{GRAIN_HASH, MASK_SUMS, MASK_TERMS, SRGB_LUT_N};
use lightcraft_pipeline::tone::{LUT_MAX_EV, LUT_MIN_EV, LUT_N};
use lightcraft_pipeline::tone::{CHROMA_N, LUT_MAX_EV, LUT_MIN_EV, LUT_N};
let mut s = String::new();
let [to_lms, from_lms, to_lab, from_lab] = lightcraft_color::perceptual::oklab_matrices();
for (name, m) in [("OK_TO_LMS", to_lms), ("OK_FROM_LMS", from_lms), ("OK_TO_LAB", to_lab), ("OK_FROM_LAB", from_lab)] {
let rows: Vec<String> = m.iter().map(|r| format!("vec3<f32>({:?}, {:?}, {:?})", r[0], r[1], r[2])).collect();
s += &format!("const {name} = array<vec3<f32>, 3>({});\n", rows.join(", "));
}
s += &format!("const TONE_MIN_EV: f32 = {LUT_MIN_EV:?};\nconst TONE_MAX_EV: f32 = {LUT_MAX_EV:?};\nconst TONE_N: u32 = {LUT_N}u;\n");
s += &format!(
"const TONE_MIN_EV: f32 = {LUT_MIN_EV:?};\nconst TONE_MAX_EV: f32 = {LUT_MAX_EV:?};\nconst TONE_N: u32 = {LUT_N}u;\nconst CHROMA_N: u32 = {CHROMA_N}u;\n"
);
s += &format!("const TONE_MIN_GAIN: f32 = {:?};\n", 2f32.powf(LUT_MIN_EV));
let b = lightcraft_pipeline::geometry::BLANK;
s += &format!("const BLANK_R: f32 = {:?};\nconst BLANK_G: f32 = {:?};\nconst BLANK_B: f32 = {:?};\n", b[0], b[1], b[2]);
+2 -1
View File
@@ -136,10 +136,11 @@ pub struct Present {
pub chroma: bool,
}
/// The `finish` kernel's parameter block and auxiliary table (tone LUT | sRGB LUT | curve LUTs |
/// The `finish` kernel's parameter block and auxiliary table (tone LUT | chroma curve | sRGB LUT | curve LUTs |
/// mask terms) for `fp` with `masks` (their local adjustments' terms).
pub fn finish_block(fp: &FinishParams, masks: &[[f32; MASK_TERMS]], present: &Present) -> (Vec<u32>, Vec<f32>) {
let mut aux: Vec<f32> = fp.tone.lut().to_vec();
aux.extend_from_slice(fp.tone.chroma_lut());
let srgb_off = aux.len();
aux.extend_from_slice(&srgb_lut()[..]);
let curve_off = aux.len();
+18 -1
View File
@@ -1,7 +1,7 @@
// The per-pixel stage: a straight port of `lightcraft_pipeline::finish` (keep in step with it).
// Bindings: img (rgb, pre-exposure), log_l, base, clar, tex, dark, masks (NMASK planes, then the
// blurred chromaticity when HAS_CHROMA), aux
// (tone LUT | sRGB LUT | curve LUTs | mask terms), out (packed RGBA8).
// (tone LUT | chroma curve | sRGB LUT | curve LUTs | mask terms), out (packed RGBA8).
fn tone_apply(y: f32) -> f32 {
if (y <= 0.0) {
@@ -18,6 +18,19 @@ fn tone_apply(y: f32) -> f32 {
return v;
}
// The camera chroma curve follows the tone LUT in `aux` (`ToneMap::chroma_scale`).
fn chroma_scale(o: f32) -> f32 {
let f = clamp(o, 0.0, 1.0) * f32(CHROMA_N - 1u);
let i = min(u32(f), CHROMA_N - 2u);
let t = f - f32(i);
let a = aux[TONE_N + i];
let b = aux[TONE_N + i + 1u];
if (a == b) {
return a;
}
return a + (b - a) * t;
}
fn encode_srgb(v: f32) -> f32 {
let o = pu(F_SRGB_OFF);
let f = clamp(v, 0.0, 1.0) * f32(SRGB_N);
@@ -400,6 +413,10 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
if (yl > 1e-9) {
d = c * o / yl;
}
let k = chroma_scale(o);
if (k != 1.0) {
d = vec3<f32>(o) + (d - vec3<f32>(o)) * k;
}
let mx = max(d.x, max(d.y, d.z));
if (mx > 1.0) {
let t = clamp((mx - 1.0) / max(mx - o, 1e-6), 0.0, 1.0);
+5
View File
@@ -41,6 +41,11 @@ fn camera_tone_and_relative_wb() {
s.wb.mode = WbMode::Custom;
s.wb.temp = 8000.0;
check("camera tone edited", &src, &info, &s, &RenderRequest::fit(320, 240));
// a camera chroma curve: richer shadows, highlights bleached toward white
let curve = curve.with_chroma([1.4, 1.3, 1.1, 1.0, 0.7, 0.4, 0.25, 0.2]).unwrap();
let info = SourceInfo { camera_tone: Some(curve), ..info };
check("camera chroma curve", &src, &info, &DevelopSettings::default(), &RenderRequest::fit(320, 240));
check("camera chroma curve edited", &src, &info, &s, &RenderRequest::fit(320, 240));
}
fn scene(i: usize, w: usize, h: usize) -> Arc<Rgb32f> {
+34 -1
View File
@@ -26,7 +26,7 @@ pub use datetime::DateTime;
pub use exif::{from_tiff, read_exif, strip_exif_header, try_read_exif, write_exif};
pub use gpx::{GpxError, Match, TrackPoint, Tracklog, parse_gpx};
pub use iptc::parse_iptc;
pub use lightcraft_geom::Orientation;
pub use lightcraft_geom::{Orientation, Rect};
pub use tags::{TagRow, file_tag_rows, tag_rows};
pub use xmp::{CRS_NS, LC_NS, XmpData, XmpError, XmpValue, parse_xmp, write_xmp, write_xmp_lc};
pub use xmp_merge::{MergeRules, merge_xmp};
@@ -50,6 +50,31 @@ pub struct Flash {
pub raw: u16,
}
/// A named region of interest on a photo (MWG Region Guidelines, `mwg-rs:Regions`): most often a face,
/// drawn by Lightroom, digiKam, Picasa or similar tools. Read-only for now — LightCraft does not write
/// regions yet (see `docs/xmp-interop.md`).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct Region {
/// Normalized to the image's full, oriented frame (`Metadata::width`/`height`), y-down, 0..1 on
/// both axes — MWG's own convention, and what `AppliedToDimensions` areas are converted to.
pub rect: Rect,
pub kind: RegionKind,
/// `mwg-rs:Name` (e.g. the person's name).
pub name: Option<String>,
pub description: Option<String>,
}
/// `mwg-rs:Type`. Unrecognised values are kept verbatim rather than dropped.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum RegionKind {
Face,
Pet,
Focus,
BarCode,
/// `mwg-rs:Type` was missing, empty, or a value we don't recognise.
Other(String),
}
/// Everything LightCraft shows or searches about a photo. All fields are optional; unknown = `None`/empty.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct Metadata {
@@ -115,6 +140,8 @@ pub struct Metadata {
pub rating: Option<i8>,
/// Colour label name (e.g. "Red").
pub label: Option<String>,
/// Face/pet/focus/barcode regions (`mwg-rs:Regions`), read from XMP only.
pub regions: Vec<Region>,
}
macro_rules! fill {
@@ -175,6 +202,9 @@ impl Metadata {
if self.hierarchical_keywords.is_empty() {
self.hierarchical_keywords = other.hierarchical_keywords.clone();
}
if self.regions.is_empty() {
self.regions = other.regions.clone();
}
}
/// Replace the user-editable fields (rating, label, title, caption, artist, copyright, keywords, GPS) with
@@ -187,6 +217,9 @@ impl Metadata {
if !other.hierarchical_keywords.is_empty() {
self.hierarchical_keywords = other.hierarchical_keywords.clone();
}
if !other.regions.is_empty() {
self.regions = other.regions.clone();
}
}
/// Exposure time formatted like cameras show it: `1/250`, `0.5`, `2″`.
+277 -1
View File
@@ -8,7 +8,8 @@
//! Writing emits the interchange subset in standard namespaces (dc, xmp, photoshop, exif, exifEX, tiff, lr) and
//! LightCraft's full develop settings as an opaque JSON string in `lc:settings`.
use crate::{DateTime, Flash, Gps, Metadata, Orientation, parse_number};
use crate::{DateTime, Flash, Gps, Metadata, Orientation, Region, RegionKind, parse_number};
use lightcraft_geom::{Point, Rect};
use quick_xml::escape::{escape, partial_escape};
use quick_xml::events::Event;
use std::collections::BTreeMap;
@@ -32,6 +33,11 @@ const NAMESPACES: &[(&str, &str)] = &[
("lc", LC_NS),
// Read-only: develop settings written by other raw developers (interchange; see docs/xmp-interop.md).
("crs", CRS_NS),
// Read-only: face/pet/focus regions written by Lightroom, digiKam, Picasa and others (MWG Region
// Guidelines v2.0). Canonicalising these three lets a packet use any prefix for them.
("mwg-rs", "http://www.metadataworkinggroup.com/schemas/regions/"),
("stArea", "http://ns.adobe.com/xmp/sType/Area#"),
("stDim", "http://ns.adobe.com/xap/1.0/sType/Dimensions#"),
];
/// The camera-raw-settings namespace URI (prefix `crs`), read for interchange only.
@@ -374,6 +380,72 @@ fn parse_gps_coord(s: &str) -> Option<f64> {
v.is_finite().then_some(v)
}
/// `mwg-rs:Regions` → every region in its `RegionList`, converted to the full-image normalized frame.
/// Malformed or out-of-range entries are skipped rather than rejecting the whole list: one bad region
/// (from Lightroom, digiKam, Picasa, or a hand-edited file) shouldn't hide the rest.
fn parse_regions(regions: &XmpValue) -> Vec<Region> {
// `AppliedToDimensions` documents the pixel size regions were authored against. It's only consulted
// for the rare region whose own Area is in pixel units instead of MWG's normalized default below.
let px_dims = regions.field("mwg-rs:AppliedToDimensions").and_then(|d| {
let w = d.field("stDim:w").and_then(XmpValue::text).and_then(parse_number)?;
let h = d.field("stDim:h").and_then(XmpValue::text).and_then(parse_number)?;
(w > 0.0 && h > 0.0).then_some((w, h))
});
let Some(list) = regions.field("mwg-rs:RegionList") else { return Vec::new() };
list.items().iter().filter_map(|item| parse_one_region(item, px_dims)).collect()
}
fn parse_one_region(item: &XmpValue, px_dims: Option<(f64, f64)>) -> Option<Region> {
let area = item.field("mwg-rs:Area")?;
let num_attr = |k: &str| area.field(k).and_then(XmpValue::text).and_then(parse_number);
// stArea:x/y are the region's CENTER, not its top-left corner (MWG Region Guidelines §Area).
let (cx, cy, w, h) = (num_attr("stArea:x")?, num_attr("stArea:y")?, num_attr("stArea:w")?, num_attr("stArea:h")?);
let finite_and_sane = [cx, cy, w, h].iter().all(|v| v.is_finite() && v.abs() <= 1.0e6);
if !finite_and_sane || w <= 0.0 || h <= 0.0 {
return None;
}
// `stArea:unit` is "normalized" (a fraction of AppliedToDimensions) in every writer we know of, and
// that needs no cross-referencing at all. Only a region that explicitly says "pixel" gets divided
// down, and only if AppliedToDimensions is there to divide by; otherwise it's dropped rather than
// stored as a nonsense fraction.
let is_pixels = area.field("stArea:unit").and_then(XmpValue::text).is_some_and(|u| u.eq_ignore_ascii_case("pixel"));
let (cx, cy, w, h) = match (is_pixels, px_dims) {
(false, _) => (cx, cy, w, h),
(true, Some((dw, dh))) => (cx / dw, cy / dh, w / dw, h / dh),
(true, None) => return None,
};
// Lightroom Classic stores the box of a photo with Exif orientation 3 / 6 / 8 in the sensor's own
// frame and writes the orientation as the region's `mwg-rs:Rotation` (π for 3, −π/2 for 6, +π/2 for
// 8; 0 for 1) while `AppliedToDimensions` names the upright size. Turn it into the upright frame.
// MWG has no way to say "mirrored", so orientations 2/4/5/7 can't be told apart from their unmirrored
// twins here; boxes on mirrored photos are taken as written (see LR-LIB-PEOPLE in docs/parity.md).
let rotation = item.field("mwg-rs:Rotation").and_then(XmpValue::text).and_then(parse_number).filter(|r| r.is_finite());
let (quarter, half) = (std::f64::consts::FRAC_PI_2, std::f64::consts::PI);
let (cx, cy, w, h) = match rotation {
Some(r) if (r + quarter).abs() < 0.05 => (1.0 - cy, cx, h, w),
Some(r) if (r - quarter).abs() < 0.05 => (cy, 1.0 - cx, h, w),
Some(r) if (r.abs() - half).abs() < 0.05 => (1.0 - cx, 1.0 - cy, w, h),
_ => (cx, cy, w, h),
};
// Keep only the part of the box inside the photo: a region hanging off an edge (common for faces
// cut by the frame) is clipped, and one entirely outside it (hostile or stale) is dropped, so every
// stored rect is within 0..1 and non-empty.
let rect = Rect::from_center(Point::new(cx, cy), w, h).intersect(&Rect::UNIT);
if rect.is_empty() {
return None;
}
let name = item.field("mwg-rs:Name").and_then(XmpValue::text).map(str::to_string).filter(|s| !s.is_empty());
let description = item.field("mwg-rs:Description").and_then(XmpValue::text).map(str::to_string).filter(|s| !s.is_empty());
let kind = match item.field("mwg-rs:Type").and_then(XmpValue::text).unwrap_or("") {
t if t.eq_ignore_ascii_case("face") => RegionKind::Face,
t if t.eq_ignore_ascii_case("pet") => RegionKind::Pet,
t if t.eq_ignore_ascii_case("focus") => RegionKind::Focus,
t if t.eq_ignore_ascii_case("barcode") => RegionKind::BarCode,
t => RegionKind::Other(t.to_string()),
};
Some(Region { rect, kind, name, description })
}
fn fmt_gps_coord(v: f64, pos: char, neg: char) -> String {
let a = v.abs();
let d = a.trunc();
@@ -470,6 +542,7 @@ pub fn parse_xmp(s: &str) -> Result<XmpData, XmpError> {
let altitude = num("exif:GPSAltitude").map(|a| if first("exif:GPSAltitudeRef").as_deref() == Some("1") { -a } else { a });
m.gps = Some(Gps { latitude: lat, longitude: lon, altitude });
}
m.regions = values.get("mwg-rs:Regions").map(parse_regions).unwrap_or_default();
let lc_settings = props.get("lc:settings").and_then(|v| v.first()).cloned();
Ok(XmpData { metadata: m, lc_settings, properties: props, values })
}
@@ -610,6 +683,9 @@ mod tests {
fn full() -> Metadata {
Metadata {
// write_xmp doesn't emit regions yet (read-only interchange; see docs/xmp-interop.md), so the
// roundtrip fixture must leave this empty or `roundtrip_all_fields` can't round-trip it.
regions: Vec::new(),
make: Some("Maker & Sons".into()),
model: Some("X <1>".into()),
serial_number: Some("SN1".into()),
@@ -764,4 +840,204 @@ mod tests {
assert_eq!(parse_gps_coord("é"), None);
assert_eq!(fmt_gps_coord(-45.51, 'N', 'S'), "45,30.600000S");
}
/// MWG Region Guidelines v2.0, element form (`rdf:parseType="Resource"`, self-closing `Area`) — how
/// Lightroom typically writes it. One named face, one untyped/unnamed region (still usable), one
/// region with a description. `AppliedToDimensions` is present but irrelevant: normalized areas
/// (the default, and the only unit real files use) don't need it.
#[test]
fn mwg_regions_element_form() {
let x = r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#"
xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
<mwg-rs:Regions rdf:parseType="Resource">
<mwg-rs:AppliedToDimensions rdf:parseType="Resource" stDim:w="4000" stDim:h="3000" stDim:unit="pixel"/>
<mwg-rs:RegionList>
<rdf:Bag>
<rdf:li rdf:parseType="Resource">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.5" stArea:y="0.4" stArea:w="0.2" stArea:h="0.3" stArea:unit="normalized"/>
<mwg-rs:Name>Jane Doe</mwg-rs:Name>
<mwg-rs:Type>Face</mwg-rs:Type>
</rdf:li>
<rdf:li rdf:parseType="Resource">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.1" stArea:y="0.1" stArea:w="0.05" stArea:h="0.05" stArea:unit="normalized"/>
</rdf:li>
<rdf:li rdf:parseType="Resource">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.8" stArea:y="0.2" stArea:w="0.1" stArea:h="0.1" stArea:unit="normalized"/>
<mwg-rs:Type>Pet</mwg-rs:Type>
<mwg-rs:Name>Rex</mwg-rs:Name>
<mwg-rs:Description>Good boy</mwg-rs:Description>
</rdf:li>
</rdf:Bag>
</mwg-rs:RegionList>
</mwg-rs:Regions>
</rdf:Description></rdf:RDF></x:xmpmeta>"#;
let regions = parse_xmp(x).unwrap().metadata.regions;
assert_eq!(regions.len(), 3, "{regions:?}");
assert_eq!(regions[0].kind, RegionKind::Face);
assert_eq!(regions[0].name.as_deref(), Some("Jane Doe"));
assert_eq!(regions[0].description, None);
let r = regions[0].rect;
assert!((r.x0 - 0.4).abs() < 1e-9 && (r.x1 - 0.6).abs() < 1e-9, "{r:?}");
assert!((r.y0 - 0.25).abs() < 1e-9 && (r.y1 - 0.55).abs() < 1e-9, "{r:?}");
// No mwg-rs:Type or mwg-rs:Name at all: still a usable region, kind falls back to `Other("")`.
assert_eq!(regions[1].kind, RegionKind::Other(String::new()));
assert_eq!(regions[1].name, None);
assert_eq!(regions[2].kind, RegionKind::Pet);
assert_eq!(regions[2].name.as_deref(), Some("Rex"));
assert_eq!(regions[2].description.as_deref(), Some("Good boy"));
}
/// Lightroom Classic writes a rotated photo's (Exif orientation 3 / 6 / 8) box in the sensor's frame
/// and records the orientation as `mwg-rs:Rotation` (±π / −π/2 / +π/2); the reader returns the
/// upright frame.
#[test]
fn mwg_regions_quarter_turn_rotation_is_mapped_to_the_upright_frame() {
let x = r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#">
<mwg-rs:Regions rdf:parseType="Resource">
<mwg-rs:RegionList>
<rdf:Bag>
<rdf:li rdf:parseType="Resource" mwg-rs:Rotation="-1.57080" mwg-rs:Type="Face">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.3" stArea:y="0.4" stArea:w="0.2" stArea:h="0.3"/>
</rdf:li>
<rdf:li rdf:parseType="Resource" mwg-rs:Rotation="1.57080" mwg-rs:Type="Face">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.3" stArea:y="0.4" stArea:w="0.2" stArea:h="0.3"/>
</rdf:li>
<rdf:li rdf:parseType="Resource" mwg-rs:Rotation="0.00000" mwg-rs:Type="Face">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.3" stArea:y="0.4" stArea:w="0.2" stArea:h="0.3"/>
</rdf:li>
<rdf:li rdf:parseType="Resource" mwg-rs:Rotation="3.14159" mwg-rs:Type="Face">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.3" stArea:y="0.4" stArea:w="0.2" stArea:h="0.3"/>
</rdf:li>
<rdf:li rdf:parseType="Resource" mwg-rs:Rotation="-3.14159" mwg-rs:Type="Face">
<mwg-rs:Area rdf:parseType="Resource" stArea:x="0.3" stArea:y="0.4" stArea:w="0.2" stArea:h="0.3"/>
</rdf:li>
</rdf:Bag>
</mwg-rs:RegionList>
</mwg-rs:Regions>
</rdf:Description></rdf:RDF></x:xmpmeta>"#;
let regions = parse_xmp(x).unwrap().metadata.regions;
assert_eq!(regions.len(), 5, "{regions:?}");
let near = |r: Rect, b: [f64; 4]| [r.x0 - b[0], r.y0 - b[1], r.x1 - b[2], r.y1 - b[3]].iter().all(|d| d.abs() < 1e-9);
assert!(near(regions[0].rect, [0.45, 0.2, 0.75, 0.4]), "−π/2 (orientation 6): {:?}", regions[0].rect);
assert!(near(regions[1].rect, [0.25, 0.6, 0.55, 0.8]), "+π/2 (orientation 8): {:?}", regions[1].rect);
assert!(near(regions[2].rect, [0.2, 0.25, 0.4, 0.55]), "0 is untouched: {:?}", regions[2].rect);
assert!(near(regions[3].rect, [0.6, 0.45, 0.8, 0.75]), "π (orientation 3): {:?}", regions[3].rect);
assert!(near(regions[4].rect, [0.6, 0.45, 0.8, 0.75]), "−π (orientation 3): {:?}", regions[4].rect);
}
/// A second, differently-shaped encoding: each region is its own `rdf:Description` (rather than an
/// `rdf:li rdf:parseType="Resource"`), `mwg-rs:Type` as an attribute instead of an element, and an
/// unrecognised `mwg-rs:Type` value kept verbatim.
#[test]
fn mwg_regions_attribute_form_and_unknown_type() {
let x = r#"<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#">
<mwg-rs:Regions rdf:parseType="Resource">
<mwg-rs:RegionList>
<rdf:Bag>
<rdf:li>
<rdf:Description mwg-rs:Type="BarCode">
<mwg-rs:Area stArea:x="0.3" stArea:y="0.3" stArea:w="0.1" stArea:h="0.1"/>
</rdf:Description>
</rdf:li>
<rdf:li>
<rdf:Description mwg-rs:Type="Sticker">
<mwg-rs:Area stArea:x="0.6" stArea:y="0.6" stArea:w="0.1" stArea:h="0.1"/>
</rdf:Description>
</rdf:li>
</rdf:Bag>
</mwg-rs:RegionList>
</mwg-rs:Regions>
</rdf:Description></rdf:RDF>"#;
let regions = parse_xmp(x).unwrap().metadata.regions;
assert_eq!(regions.len(), 2, "{regions:?}");
assert_eq!(regions[0].kind, RegionKind::BarCode);
assert_eq!(regions[1].kind, RegionKind::Other("Sticker".to_string()));
}
/// Hostile/malformed regions are dropped individually, never panic, and never poison the rest of
/// the list: missing Area, non-finite or absurd numbers, zero/negative size, and a pixel-unit area
/// with no `AppliedToDimensions` to convert it (so it can't be stored as a nonsense fraction).
#[test]
fn mwg_regions_hostile_inputs_are_dropped_not_panicking() {
let x = r#"<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#">
<mwg-rs:Regions rdf:parseType="Resource">
<mwg-rs:RegionList>
<rdf:Bag>
<rdf:li rdf:parseType="Resource"><mwg-rs:Name>No Area</mwg-rs:Name></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="NaN" stArea:y="0.1" stArea:w="0.1" stArea:h="0.1"/></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="1e300" stArea:y="0.1" stArea:w="0.1" stArea:h="0.1"/></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="0.5" stArea:y="0.5" stArea:w="0" stArea:h="0.1"/></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="0.5" stArea:y="0.5" stArea:w="-0.1" stArea:h="0.1"/></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="100" stArea:y="100" stArea:w="50" stArea:h="50" stArea:unit="pixel"/></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="0.5" stArea:y="0.5" stArea:w="0.2" stArea:h="0.2"/><mwg-rs:Type>Face</mwg-rs:Type></rdf:li>
</rdf:Bag>
</mwg-rs:RegionList>
</mwg-rs:Regions>
</rdf:Description></rdf:RDF>"#;
let regions = parse_xmp(x).unwrap().metadata.regions;
assert_eq!(regions.len(), 1, "only the last, well-formed region should survive: {regions:?}");
assert_eq!(regions[0].kind, RegionKind::Face);
}
/// Regions are clipped to the photo frame, and ones that miss the photo entirely are dropped: a
/// hostile sidecar can't park a box at x=1e5 (or a 1e5-wide one) that the loupe or the People view
/// would then draw or crop far outside the image.
#[test]
fn mwg_regions_are_clipped_to_the_frame_and_off_photo_ones_dropped() {
let x = r#"<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#">
<mwg-rs:Regions rdf:parseType="Resource">
<mwg-rs:RegionList>
<rdf:Bag>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="1e5" stArea:y="0.5" stArea:w="0.1" stArea:h="0.1"/><mwg-rs:Name>Far right</mwg-rs:Name></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="-3" stArea:y="-3" stArea:w="0.5" stArea:h="0.5"/><mwg-rs:Name>Far left</mwg-rs:Name></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="1.2" stArea:y="0.5" stArea:w="0.2" stArea:h="0.2"/><mwg-rs:Name>Touching edge</mwg-rs:Name></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="0.5" stArea:y="0.5" stArea:w="1e5" stArea:h="1e5"/><mwg-rs:Name>Huge</mwg-rs:Name></rdf:li>
<rdf:li rdf:parseType="Resource"><mwg-rs:Area rdf:parseType="Resource" stArea:x="0.95" stArea:y="0.05" stArea:w="0.2" stArea:h="0.2"/><mwg-rs:Name>Corner</mwg-rs:Name></rdf:li>
</rdf:Bag>
</mwg-rs:RegionList>
</mwg-rs:Regions>
</rdf:Description></rdf:RDF>"#;
let regions = parse_xmp(x).unwrap().metadata.regions;
let names: Vec<_> = regions.iter().map(|r| r.name.as_deref().unwrap_or("")).collect();
assert_eq!(names, ["Huge", "Corner"], "{regions:?}");
for r in &regions {
let q = r.rect;
assert!(q.x0 >= 0.0 && q.y0 >= 0.0 && q.x1 <= 1.0 && q.y1 <= 1.0 && !q.is_empty(), "{q:?}");
}
assert_eq!(regions[0].rect, Rect::UNIT);
let c = regions[1].rect;
assert!((c.x0 - 0.85).abs() < 1e-9 && c.x1 == 1.0 && c.y0 == 0.0 && (c.y1 - 0.15).abs() < 1e-9, "{c:?}");
}
/// `extract()`'s precedence (EXIF/IPTC never carry regions; XMP is read and filled in like keywords).
#[test]
fn regions_merge_like_keywords() {
let with_region = Region { rect: Rect::from_center(Point::new(0.5, 0.5), 0.2, 0.2), kind: RegionKind::Face, name: None, description: None };
let mut a = Metadata::default();
let b = Metadata { regions: vec![with_region.clone()], ..Default::default() };
a.fill_missing(&b);
assert_eq!(a.regions, vec![with_region.clone()]);
let mut a = Metadata::default();
a.overlay_user_fields(&b);
assert_eq!(a.regions, vec![with_region]);
}
}
+4
View File
@@ -483,6 +483,10 @@ pub(crate) fn finish_with<T: Copy + Default + Send>(
let yl = luminance_2020(c);
let o = tone.apply(yl);
let mut d = if yl > 1e-9 { c.map(|v| v * o / yl) } else { [0.0; 3] };
let k = tone.chroma_scale(o);
if k != 1.0 {
d = d.map(|v| o + (v - o) * k);
}
let mx = d[0].max(d[1]).max(d[2]);
if mx > 1.0 {
let t = ((mx - 1.0) / (mx - o).max(1e-6)).clamp(0.0, 1.0);
+156 -1
View File
@@ -3,7 +3,7 @@
//! Shapes are defined in normalized oriented-image coordinates; sizes are in "long-edge units".
//! Components combine with Add (max), Subtract (a·(1−c)) and Intersect (a·c).
use lightcraft_develop::{BrushStroke, LocalAdjustments, Mask, MaskOp, MaskShape};
use lightcraft_develop::{BrushStroke, LocalAdjustments, Mask, MaskOp, MaskShape, SegMask};
use lightcraft_geom::Point;
use lightcraft_raster::{Plane, Rgb32f};
@@ -153,6 +153,19 @@ pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &R
}
smooth_plane(&mut out, 0.01 * frame_px(frame, w));
}
MaskShape::Object { seg, detail, edge, .. } | MaskShape::Prompt { seg, detail, edge, .. } if seg.is_some() || !detail.is_empty() => {
// Edge: below 0 a steeper transition (up to 8× the logits), above 0 feathered (up to
// 2 % of the long edge, so previews and exports match)
let e = (*edge / 100.0).clamp(-1.0, 1.0) as f32;
let gain = if e < 0.0 { 1.0 - 7.0 * e } else { 1.0 };
sample_seg(seg.as_ref(), detail, gain, frame, w, h, &mut out);
if e > 0.0 {
smooth_plane(&mut out, e * 0.02 * frame_px(frame, w));
}
}
// not computed yet (no clicks, or a build without the model): nothing selected
MaskShape::Object { hint, .. } if hint.is_empty() => {}
MaskShape::Prompt { .. } => {}
MaskShape::Subject | MaskShape::Object { .. } | MaskShape::People { .. } => {
// Saliency heuristic: centre-weighted local contrast (replaced by the segmenter in M12).
let m = frame.out_to_norm(w, h);
@@ -176,6 +189,61 @@ pub fn shape_alpha(shape: &MaskShape, frame: &Frame, w: usize, h: usize, img: &R
out
}
/// A decoded [`SegMask`]: its logits and the normalized rectangle they cover.
struct SegGrid {
logits: Vec<f32>,
side: usize,
r: [f64; 4],
}
impl SegGrid {
fn new(seg: &SegMask) -> Option<SegGrid> {
Some(SegGrid { logits: seg.logits()?, side: seg.side as usize, r: seg.bounds()? })
}
fn contains(&self, n: Point) -> bool {
n.x >= self.r[0] && n.x <= self.r[2] && n.y >= self.r[1] && n.y <= self.r[3]
}
/// The bilinearly sampled logit at normalized image point `n` (cell centres at
/// `(i + 0.5) / side` of the rectangle).
fn logit(&self, n: Point) -> f32 {
let side = self.side;
let at = |x: usize, y: usize| self.logits.get(y.min(side - 1) * side + x.min(side - 1)).copied().unwrap_or(-16.0);
let u = (n.x - self.r[0]) / (self.r[2] - self.r[0]);
let v = (n.y - self.r[1]) / (self.r[3] - self.r[1]);
let (fx, fy) = ((u * side as f64 - 0.5) as f32, (v * side as f64 - 0.5) as f32);
let (fx, fy) = (fx.clamp(0.0, (side - 1) as f32), fy.clamp(0.0, (side - 1) as f32));
let (x0, y0) = (fx.floor() as usize, fy.floor() as usize);
let (tx, ty) = (fx - x0 as f32, fy - y0 as f32);
let top = at(x0, y0) * (1.0 - tx) + at(x0 + 1, y0) * tx;
let bot = at(x0, y0 + 1) * (1.0 - tx) + at(x0 + 1, y0 + 1) * tx;
top * (1.0 - ty) + bot * ty
}
}
/// Stored model segmentations at each output pixel: the zoomed-in `detail` passes inside
/// their rectangles (the highest of them where they overlap), the whole-photo `seg`
/// elsewhere; the logit (times `gain`: > 1 is a harder edge) through a sigmoid. Damaged
/// data selects nothing.
fn sample_seg(seg: Option<&SegMask>, detail: &[SegMask], gain: f32, frame: &Frame, w: usize, h: usize, out: &mut Plane) {
let coarse = seg.and_then(SegGrid::new);
// (capped: a hostile document can hold many)
let patches: Vec<SegGrid> = detail.iter().take(lightcraft_develop::segmask::MAX_DETAIL).filter_map(SegGrid::new).collect();
if coarse.is_none() && patches.is_empty() {
return;
}
let m = frame.out_to_norm(w, h);
for_rows(&mut out.data, w, |y, row| {
for (x, v) in row.iter_mut().enumerate() {
let n = m.apply(Point::new(x as f64 + 0.5, y as f64 + 0.5));
let fine = patches.iter().filter(|p| p.contains(n)).map(|p| p.logit(n)).reduce(f32::max);
let l = fine.or_else(|| coarse.as_ref().map(|c| c.logit(n))).unwrap_or(-16.0);
*v = 1.0 / (1.0 + (-l * gain).exp());
}
});
}
fn frame_px(frame: &Frame, w: usize) -> f32 {
frame.px_per_long(w) as f32
}
@@ -356,6 +424,93 @@ mod tests {
assert!((a.get(50, 25) - 0.5).abs() < 0.05);
}
#[test]
fn stored_segmentations_render_at_any_size_and_follow_the_crop() {
// the left half selected, on a 16² logit grid
let side = 16;
let logits: Vec<f32> = (0..side * side).map(|i| if i % side < side / 2 { 12.0 } else { -12.0 }).collect();
let seg = SegMask::from_logits(side, &logits);
for shape in [
MaskShape::Object { hint: vec![Point::new(0.25, 0.5)], exclude: vec![], seg: Some(seg.clone()), detail: vec![], edge: 0.0 },
MaskShape::Prompt { text: "left".into(), seg: Some(seg.clone()), detail: vec![], edge: 0.0 },
] {
for (w, h) in [(40, 20), (400, 200)] {
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
assert!(a.get(1, h / 2) > 0.99 && a.get(w - 2, h / 2) < 0.01, "{w}×{h}");
if w >= 400 {
let soft = (0..w).filter(|x| (0.05..0.95).contains(&a.get(*x, h / 2))).count();
assert!((2..30).contains(&soft), "an anti-aliased edge, not a ramp: {soft} px");
}
}
}
// cropped to the right half: nothing left selected
let mut d = DevelopSettings::default();
d.crop.geometry.rect = lightcraft_geom::Rect::from_center(Point::new(0.75, 0.5), 0.5, 1.0);
let f = Frame::new(40, 20, &d, true);
let shape = MaskShape::Prompt { text: "left".into(), seg: Some(seg), detail: vec![], edge: 0.0 };
let a = shape_alpha(&shape, &f, 20, 20, &Rgb32f::new(20, 20), &Plane::new(20, 20), 0.0);
assert!(a.data.iter().all(|v| *v < 0.05));
}
#[test]
fn detail_patches_replace_the_coarse_mask_inside_their_rectangle() {
// coarse: nothing; a patch over the right half selects its own left half (x 0.5..0.75)
let coarse = SegMask::from_logits(4, &[-12.0; 16]);
let side = 16;
let l: Vec<f32> = (0..side * side).map(|i| if i % side < side / 2 { 12.0 } else { -12.0 }).collect();
let patch = SegMask::from_logits_in(side, &l, [0.5, 0.0, 1.0, 1.0]);
let shape = MaskShape::Object { hint: vec![Point::new(0.6, 0.5)], exclude: vec![], seg: Some(coarse), detail: vec![patch], edge: 0.0 };
let (w, h) = (200, 100);
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
assert!(a.get(20, 50) < 0.01, "outside the patch: the coarse mask");
assert!(a.get(120, 50) > 0.99, "inside the patch, its selected half");
assert!(a.get(190, 50) < 0.01, "inside the patch, its unselected half");
}
#[test]
fn a_hostile_number_of_detail_patches_is_capped() {
// built in memory (a deserialized document is capped already): only the first
// MAX_DETAIL are decoded and sampled, so this needs 4 × 4 MB, not 20 000 × 4 MB
let side = 1024;
let l: Vec<f32> = vec![12.0; side * side];
let patch = SegMask::from_logits_in(side, &l, [0.0, 0.0, 0.5, 1.0]);
let shape = MaskShape::Object { hint: vec![Point::new(0.2, 0.5)], exclude: vec![], seg: None, detail: vec![patch; 20_000], edge: 0.0 };
let (w, h) = (64, 32);
let t = std::time::Instant::now();
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
assert!(a.get(5, 16) > 0.99 && a.get(60, 16) < 0.01);
assert!(t.elapsed() < std::time::Duration::from_secs(20), "{:?}", t.elapsed());
}
#[test]
fn the_edge_setting_hardens_or_feathers_the_transition() {
let side = 16;
let l: Vec<f32> = (0..side * side).map(|i| ((i % side) as f32 - 7.5) * -0.6).collect(); // a gentle ramp
let seg = SegMask::from_logits(side, &l);
let (w, h) = (400, 40);
let soft_px = |edge: f64| {
let shape = MaskShape::Prompt { text: "x".into(), seg: Some(seg.clone()), detail: vec![], edge };
let a = shape_alpha(&shape, &frame(w, h), w, h, &Rgb32f::new(w, h), &Plane::new(w, h), 0.0);
assert!(a.get(2, h / 2) > 0.9 && a.get(w - 3, h / 2) < 0.1, "still the same selection at {edge}");
(0..w).filter(|x| (0.1..0.9).contains(&a.get(*x, h / 2))).count()
};
let (hard, plain, soft) = (soft_px(-100.0), soft_px(0.0), soft_px(100.0));
assert!(hard < plain && plain < soft, "transition widths {hard} < {plain} < {soft}");
}
#[test]
fn ai_shapes_without_a_segmentation_select_nothing() {
let f = frame(30, 30);
let (img, l) = (Rgb32f::new(30, 30), Plane::new(30, 30));
for shape in [
MaskShape::Object { hint: vec![], exclude: vec![], seg: None, detail: vec![], edge: 0.0 },
MaskShape::Prompt { text: "sky".into(), seg: None, detail: vec![], edge: 0.0 },
MaskShape::Prompt { text: "sky".into(), seg: Some(SegMask { side: 4, data: "damaged!".into(), rect: None }), detail: vec![], edge: 0.0 },
] {
assert!(shape_alpha(&shape, &f, 30, 30, &img, &l, 0.0).data.iter().all(|v| *v == 0.0));
}
}
#[test]
fn radial_inside_and_invert() {
let f = frame(100, 100);
+76 -5
View File
@@ -13,13 +13,19 @@ pub const GREY: f32 = 0.18;
pub const LUT_MIN_EV: f32 = -14.0;
pub const LUT_MAX_EV: f32 = 10.0;
pub const LUT_N: usize = 4096;
/// Nodes of a camera chroma curve, evenly spaced over display luminance 0..=1.
pub const CHROMA_N: usize = 8;
const NO_CHROMA: [f32; CHROMA_N] = [1.0; CHROMA_N];
/// A file-local camera look, fitted independently of the scene-linear colour transform.
/// Knots are scene/display-linear luminance pairs. Keeping this in the finish stage preserves
/// RAW exposure and highlight headroom; it is never baked into the decoded sensor pixels.
/// `chroma` scales colourfulness by display luminance after the curve (a camera's per-channel
/// curve saturates shadows and bleaches highlights toward white, which a luminance curve can't).
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize)]
pub struct CameraTone {
knots: [[f32; 2]; 32],
chroma: [f32; CHROMA_N],
}
impl<'de> serde::Deserialize<'de> for CameraTone {
@@ -27,9 +33,16 @@ impl<'de> serde::Deserialize<'de> for CameraTone {
#[derive(serde::Deserialize)]
struct Wire {
knots: [[f32; 2]; 32],
// absent in smart previews written before the chroma curve existed
#[serde(default)]
chroma: Option<[f32; CHROMA_N]>,
}
let w = Wire::deserialize(d)?;
Self::new(w.knots).ok_or_else(|| serde::de::Error::custom("invalid camera tone curve"))
let tone = Self::new(w.knots).ok_or_else(|| serde::de::Error::custom("invalid camera tone curve"))?;
match w.chroma {
Some(c) => tone.with_chroma(c).ok_or_else(|| serde::de::Error::custom("invalid camera chroma curve")),
None => Ok(tone),
}
}
}
@@ -42,7 +55,16 @@ impl CameraTone {
}
previous = p;
}
Some(Self { knots })
Some(Self { knots, chroma: NO_CHROMA })
}
/// The curve with chroma scales at display luminance 0, 1/7 … 1 (each finite, 0..=4).
pub fn with_chroma(self, chroma: [f32; CHROMA_N]) -> Option<Self> {
chroma.iter().all(|k| k.is_finite() && (0.0..=4.0).contains(k)).then_some(Self { chroma, ..self })
}
pub fn chroma(&self) -> &[f32; CHROMA_N] {
&self.chroma
}
pub fn apply(&self, y: f32) -> f32 {
@@ -68,6 +90,7 @@ impl CameraTone {
#[derive(Clone, Debug)]
pub struct ToneMap {
lut: Vec<f32>,
chroma: [f32; CHROMA_N],
}
impl ToneMap {
@@ -81,7 +104,7 @@ impl ToneMap {
if neutral { y } else { adjustment.apply(y) }
})
.collect();
ToneMap { lut }
ToneMap { lut, chroma: curve.chroma }
}
/// `contrast`, `whites`, `blacks` in −100..100 (Lightroom slider units).
pub fn new(contrast: f64, whites: f64, blacks: f64) -> ToneMap {
@@ -113,7 +136,7 @@ impl ToneMap {
o.clamp(0.0, 1.0)
})
.collect();
ToneMap { lut }
ToneMap { lut, chroma: NO_CHROMA }
}
/// Tone map for display-referred sources: identity at neutral settings.
@@ -147,7 +170,7 @@ impl ToneMap {
o.clamp(0.0, 1.0)
})
.collect();
ToneMap { lut }
ToneMap { lut, chroma: NO_CHROMA }
}
/// The table (`LUT_N` entries, see [`ToneMap::apply`]).
@@ -155,6 +178,24 @@ impl ToneMap {
&self.lut
}
/// The chroma curve (`CHROMA_N` entries, see [`ToneMap::chroma_scale`]).
pub fn chroma_lut(&self) -> &[f32] {
&self.chroma
}
/// Chroma scale at display luminance `o` (exactly 1 everywhere unless a camera look sets it).
#[inline]
pub fn chroma_scale(&self, o: f32) -> f32 {
if !o.is_finite() {
return 1.0;
}
let f = o.clamp(0.0, 1.0) * (CHROMA_N - 1) as f32;
let i = (f as usize).min(CHROMA_N - 2);
let t = f - i as f32;
let (a, b) = (self.chroma.get(i).copied().unwrap_or(1.0), self.chroma.get(i + 1).copied().unwrap_or(1.0));
if a == b { a } else { a + (b - a) * t }
}
/// Scene luminance → display-linear luminance.
#[inline]
pub fn apply(&self, y: f32) -> f32 {
@@ -179,6 +220,36 @@ fn smooth(e0: f32, e1: f32, x: f32) -> f32 {
mod tests {
use super::*;
fn knots() -> [[f32; 2]; 32] {
std::array::from_fn(|i| {
let x = 0.004 * 1.18f32.powi(i as i32);
[x, 1.0 - (-2.0 * x).exp()]
})
}
#[test]
fn chroma_curve_is_identity_unless_set_and_survives_serde() {
let plain = CameraTone::new(knots()).unwrap();
let map = ToneMap::camera(&plain, 0.0, 0.0, 0.0);
assert!([0.0, 0.3, 0.77, 1.0, 2.0].iter().all(|o| map.chroma_scale(*o) == 1.0));
assert!([0.0, 0.5, 1.0].iter().all(|o| ToneMap::new(0.0, 0.0, 0.0).chroma_scale(*o) == 1.0));
// smart previews written before the chroma curve existed still load (identity)
let old = serde_json::json!({ "knots": knots() });
assert_eq!(serde_json::from_value::<CameraTone>(old).unwrap(), plain);
let tone = plain.with_chroma([1.4, 1.3, 1.1, 1.0, 0.7, 0.4, 0.25, 0.2]).unwrap();
let back: CameraTone = serde_json::from_value(serde_json::to_value(tone).unwrap()).unwrap();
assert_eq!(back, tone);
let map = ToneMap::camera(&tone, 0.0, 0.0, 0.0);
assert!((map.chroma_scale(0.0) - 1.4).abs() < 1e-6 && (map.chroma_scale(1.0) - 0.2).abs() < 1e-6);
assert!((map.chroma_scale(0.5 / 7.0) - 1.35).abs() < 1e-5, "interpolates between nodes");
assert_eq!(map.chroma_scale(f32::NAN), 1.0);
// hostile values are rejected, also when deserialized
assert!(plain.with_chroma([f32::NAN; CHROMA_N]).is_none());
assert!(plain.with_chroma([-1.0; CHROMA_N]).is_none());
let bad = serde_json::json!({ "knots": knots(), "chroma": [9.0, 1, 1, 1, 1, 1, 1, 1] });
assert!(serde_json::from_value::<CameraTone>(bad).is_err());
}
#[test]
fn camera_curve_preserves_black_and_extends_headroom() {
let knots = std::array::from_fn(|i| {
+28
View File
@@ -0,0 +1,28 @@
[package]
name = "lightcraft-segment"
version.workspace = true
edition.workspace = true
license = "Apache-2.0"
description = "LightCraft segmentation: SAM 3 (Segment Anything with Concepts) inference in pure Rust (L3)"
[lints]
workspace = true
[dependencies]
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
log = { workspace = true }
# the model runs natively only (the web build has no AI masks)
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
candle-core = { workspace = true }
candle-nn = { workspace = true }
rustls = { version = "0.23", default-features = false, features = ["std", "tls12"] }
rustls-rustcrypto = "0.0.2-alpha"
webpki-roots = "1"
sha2 = { version = "0.10", default-features = false, features = ["std"] }
[target.'cfg(target_os = "macos")'.dependencies]
candle-core = { workspace = true, features = ["metal"] }
candle-nn = { workspace = true, features = ["metal"] }
+25
View File
@@ -0,0 +1,25 @@
//! Times SAM 3 on this machine (a synthetic gradient image; the first run includes compiling
//! the GPU kernels): `cargo run -p lightcraft-segment --release --example sam3_bench -- <model dir>`.
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut args = std::env::args().skip(1);
let dir = std::path::PathBuf::from(args.next().ok_or("usage: sam3_bench <model dir>")?);
let t = std::time::Instant::now();
let mut model = lightcraft_segment::Sam3::load(&dir)?;
eprintln!("load: {:?} on {:?}", t.elapsed(), model.device());
let (w, h) = (1200, 800);
let rgb: Vec<u8> = (0..w * h * 3).map(|i| ((i / 3 % w) * 255 / w) as u8).collect();
for run in 0..3 {
let t = std::time::Instant::now();
let mut enc = model.encode(&rgb, w, h)?;
lightcraft_segment::sync(&model)?;
let e = t.elapsed();
let t = std::time::Instant::now();
let p = model.segment_clicks(&mut enc, &[lightcraft_segment::Click { x: 0.5, y: 0.5, positive: true }])?;
let c = t.elapsed();
let t = std::time::Instant::now();
let d = model.segment_text(&mut enc, "sky", 0.5)?;
eprintln!("run {run}: encode {e:?}, click {c:?} (score {:.2}), text {:?} ({})", p.score, t.elapsed(), d.is_some());
}
Ok(())
}
+81
View File
@@ -0,0 +1,81 @@
//! The CLIP text encoder (24 layers, 1024 wide, causal) that turns a prompt into the token
//! features SAM 3's detector attends to.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3/modeling_sam3.py`, CLIP text model), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use candle_core::{Device, Module, Tensor};
use candle_nn::{Embedding, LayerNorm, VarBuilder};
use crate::Result;
use crate::nn::{Attention, Mlp, ln};
use crate::tokenizer::CONTEXT;
const HIDDEN: usize = 1024;
const LAYERS: usize = 24;
const HEADS: usize = 16;
const EPS: f64 = 1e-5;
struct Layer {
ln1: LayerNorm,
attn: Attention,
ln2: LayerNorm,
mlp: Mlp,
}
pub struct TextEncoder {
tokens: Embedding,
positions: Tensor,
layers: Vec<Layer>,
final_ln: LayerNorm,
device: Device,
}
impl TextEncoder {
/// `vb` at `detector_model.text_encoder.text_model`.
pub fn new(vb: VarBuilder) -> Result<Self> {
let e = vb.pp("embeddings");
let mut layers = Vec::with_capacity(LAYERS);
for i in 0..LAYERS {
let l = vb.pp(format!("encoder.layers.{i}"));
layers.push(Layer {
ln1: ln(HIDDEN, EPS, l.pp("layer_norm1"))?,
attn: Attention::named(HIDDEN, HIDDEN, HEADS, "out_proj", l.pp("self_attn"))?,
ln2: ln(HIDDEN, EPS, l.pp("layer_norm2"))?,
mlp: Mlp::new(HIDDEN, 4 * HIDDEN, true, l.pp("mlp"))?,
});
}
Ok(Self {
tokens: candle_nn::embedding(49408, HIDDEN, e.pp("token_embedding"))?,
positions: e.get((CONTEXT, HIDDEN), "position_embedding.weight")?.unsqueeze(0)?,
layers,
final_ln: ln(HIDDEN, EPS, vb.pp("final_layer_norm"))?,
device: vb.device().clone(),
})
}
/// `[1, 32, 1024]` features for token `ids` (`valid` marks real tokens).
pub fn forward(&self, ids: &[u32], valid: &[bool]) -> Result<Tensor> {
let ids = Tensor::from_slice(ids, (1, ids.len()), &self.device)?;
let mut x = self.tokens.forward(&ids)?.broadcast_add(&self.positions)?;
// causal, and padding keys masked
let n = valid.len();
let mut bias = vec![0f32; n * n];
for (q, row) in bias.chunks_mut(n).enumerate() {
for (k, v) in row.iter_mut().enumerate() {
if k > q || !valid.get(k).copied().unwrap_or(false) {
*v = f32::MIN;
}
}
}
let bias = Tensor::from_vec(bias, (1, 1, n, n), &self.device)?;
for l in &self.layers {
let h = l.ln1.forward(&x)?;
x = (&x + l.attn.forward(&h, &h, &h, Some(&bias))?)?;
let h = l.ln2.forward(&x)?;
x = (&x + l.mlp.forward(&h)?)?;
}
Ok(self.final_ln.forward(&x)?)
}
}
+308
View File
@@ -0,0 +1,308 @@
//! Text prompts (SAM 3's concept detector): the prompt's CLIP features fused into the image
//! features by a DETR encoder, 200 object queries refined by a DETR decoder (with box
//! relative-position bias and a presence token), per-query masks from a pixel decoder.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3/modeling_sam3.py`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use candle_core::{D, Device, IndexOp, Module, Tensor};
use candle_nn::{Conv2d, Conv2dConfig, GroupNorm, LayerNorm, Linear, VarBuilder};
use crate::clip::TextEncoder;
use crate::neck::{FPN, Neck};
use crate::nn::{Attention, DecoderMlp, Mlp, TORCH_LN_EPS, cxcywh_to_xyxy, inverse_sigmoid, linear, ln, padding_bias, sine_grid, to_vec};
use crate::{Error, Prediction, Result};
const HIDDEN: usize = FPN;
const HEADS: usize = 8;
const QUERIES: usize = 200;
const LAYERS: usize = 6;
/// Detector image features, computed once per image.
pub struct Features {
/// Levels 0..=2 (288, 144, 72 per side).
levels: Vec<Tensor>,
/// Level 2 flattened `[1, 5184, 256]` and its sine position encoding.
flat: Tensor,
pos: Tensor,
side: usize,
}
/// Everything the detector predicts for one prompt.
pub struct Output {
/// Per-query mask logits `[200, 288, 288]` flattened.
pub masks: Tensor,
/// Per-query logits (before the presence score).
pub logits: Vec<f32>,
/// Whether the concept is in the image at all (logit).
pub presence: f32,
/// Boxes (x0, y0, x1, y1), normalized.
pub boxes: Vec<[f32; 4]>,
/// The text features `[1, 32, 256]` (tests).
pub text: Tensor,
}
struct EncoderLayer {
ln1: LayerNorm,
self_attn: Attention,
ln2: LayerNorm,
cross_attn: Attention,
ln3: LayerNorm,
mlp: Mlp,
}
struct DecoderLayer {
self_attn: Attention,
self_ln: LayerNorm,
text_attn: Attention,
text_ln: LayerNorm,
vision_attn: Attention,
vision_ln: LayerNorm,
mlp: Mlp,
mlp_ln: LayerNorm,
}
pub struct Detector {
device: Device,
neck: Neck,
text: TextEncoder,
text_proj: Linear,
encoder: Vec<EncoderLayer>,
decoder: Vec<DecoderLayer>,
out_ln: LayerNorm,
box_head: DecoderMlp,
query_embed: Tensor,
reference_points: Tensor,
presence_token: Tensor,
presence_head: DecoderMlp,
presence_ln: LayerNorm,
ref_point_head: DecoderMlp,
rpb_x: DecoderMlp,
rpb_y: DecoderMlp,
// dot-product scoring
score_text_mlp: DecoderMlp,
score_text_ln: LayerNorm,
score_text_proj: Linear,
score_query_proj: Linear,
// mask decoder
prompt_attn: Attention,
prompt_ln: LayerNorm,
pixel_convs: Vec<Conv2d>,
pixel_norms: Vec<GroupNorm>,
instance_proj: Conv2d,
mask_embedder: Vec<Linear>,
}
impl Detector {
/// `vb` at `detector_model`.
pub fn new(vb: VarBuilder) -> Result<Self> {
let e = vb.pp("detr_encoder");
let mut encoder = Vec::with_capacity(LAYERS);
for i in 0..LAYERS {
let l = e.pp(format!("layers.{i}"));
encoder.push(EncoderLayer {
ln1: ln(HIDDEN, TORCH_LN_EPS, l.pp("layer_norm1"))?,
self_attn: Attention::new(HIDDEN, HIDDEN, HEADS, l.pp("self_attn"))?,
ln2: ln(HIDDEN, TORCH_LN_EPS, l.pp("layer_norm2"))?,
cross_attn: Attention::new(HIDDEN, HIDDEN, HEADS, l.pp("cross_attn"))?,
ln3: ln(HIDDEN, TORCH_LN_EPS, l.pp("layer_norm3"))?,
mlp: Mlp::new(HIDDEN, 2048, false, l.pp("mlp"))?,
});
}
let d = vb.pp("detr_decoder");
let mut decoder = Vec::with_capacity(LAYERS);
for i in 0..LAYERS {
let l = d.pp(format!("layers.{i}"));
decoder.push(DecoderLayer {
self_attn: Attention::new(HIDDEN, HIDDEN, HEADS, l.pp("self_attn"))?,
self_ln: ln(HIDDEN, TORCH_LN_EPS, l.pp("self_attn_layer_norm"))?,
text_attn: Attention::new(HIDDEN, HIDDEN, HEADS, l.pp("text_cross_attn"))?,
text_ln: ln(HIDDEN, TORCH_LN_EPS, l.pp("text_cross_attn_layer_norm"))?,
vision_attn: Attention::new(HIDDEN, HIDDEN, HEADS, l.pp("vision_cross_attn"))?,
vision_ln: ln(HIDDEN, TORCH_LN_EPS, l.pp("vision_cross_attn_layer_norm"))?,
mlp: Mlp::new(HIDDEN, 2048, false, l.pp("mlp"))?,
mlp_ln: ln(HIDDEN, TORCH_LN_EPS, l.pp("mlp_layer_norm"))?,
});
}
let s = vb.pp("dot_product_scoring");
let m = vb.pp("mask_decoder");
let p = m.pp("pixel_decoder");
let conv3 = Conv2dConfig { padding: 1, ..Default::default() };
Ok(Self {
device: vb.device().clone(),
neck: Neck::new(vb.pp("vision_encoder.neck"))?,
text: TextEncoder::new(vb.pp("text_encoder.text_model"))?,
text_proj: linear(1024, HIDDEN, vb.pp("text_projection"))?,
encoder,
decoder,
out_ln: ln(HIDDEN, TORCH_LN_EPS, d.pp("output_layer_norm"))?,
box_head: DecoderMlp::new(HIDDEN, HIDDEN, 4, 3, d.pp("box_head"))?,
query_embed: d.get((QUERIES, HIDDEN), "query_embed.weight")?.unsqueeze(0)?,
reference_points: d.get((QUERIES, 4), "reference_points.weight")?.unsqueeze(0)?,
presence_token: d.get((1, HIDDEN), "presence_token.weight")?.unsqueeze(0)?,
presence_head: DecoderMlp::new(HIDDEN, HIDDEN, 1, 3, d.pp("presence_head"))?,
presence_ln: ln(HIDDEN, TORCH_LN_EPS, d.pp("presence_layer_norm"))?,
ref_point_head: DecoderMlp::new(2 * HIDDEN, HIDDEN, HIDDEN, 2, d.pp("ref_point_head"))?,
rpb_x: DecoderMlp::new(2, HIDDEN, HEADS, 2, d.pp("box_rpb_embed_x"))?,
rpb_y: DecoderMlp::new(2, HIDDEN, HEADS, 2, d.pp("box_rpb_embed_y"))?,
score_text_mlp: DecoderMlp::new(HIDDEN, 2048, HIDDEN, 2, s.pp("text_mlp"))?,
score_text_ln: ln(HIDDEN, TORCH_LN_EPS, s.pp("text_mlp_out_norm"))?,
score_text_proj: linear(HIDDEN, HIDDEN, s.pp("text_proj"))?,
score_query_proj: linear(HIDDEN, HIDDEN, s.pp("query_proj"))?,
prompt_attn: Attention::new(HIDDEN, HIDDEN, HEADS, m.pp("prompt_cross_attn"))?,
prompt_ln: ln(HIDDEN, TORCH_LN_EPS, m.pp("prompt_cross_attn_norm"))?,
pixel_convs: (0..3).map(|i| Ok(candle_nn::conv2d(HIDDEN, HIDDEN, 3, conv3, p.pp(format!("conv_layers.{i}")))?)).collect::<Result<_>>()?,
pixel_norms: (0..3).map(|i| Ok(candle_nn::group_norm(8, HIDDEN, 1e-5, p.pp(format!("norms.{i}")))?)).collect::<Result<_>>()?,
instance_proj: candle_nn::conv2d(HIDDEN, HIDDEN, 1, Conv2dConfig::default(), m.pp("instance_projection"))?,
mask_embedder: (0..3).map(|i| linear(HIDDEN, HIDDEN, m.pp(format!("mask_embedder.layers.{i}")))).collect::<Result<_>>()?,
})
}
/// The detector's feature pyramid for the backbone output `[1, 1024, 72, 72]`.
pub fn features(&self, backbone: &Tensor) -> Result<Features> {
let levels = self.neck.forward(backbone, 3)?;
let last = levels.last().ok_or_else(|| Error::Model("detector neck".into()))?;
let (_, _, side, _) = last.dims4()?;
let flat = last.flatten_from(2)?.transpose(1, 2)?.contiguous()?;
let pos = sine_grid(side, side, HIDDEN / 2, &self.device)?.flatten_from(2)?.transpose(1, 2)?.contiguous()?;
Ok(Features { levels, flat, pos, side })
}
/// Box relative-position bias `[1, heads, 1 + 200, side²]` for reference boxes (cx, cy, w, h).
fn rpb(&self, boxes: &Tensor, side: usize) -> Result<Tensor> {
let xyxy = cxcywh_to_xyxy(boxes)?.squeeze(0)?; // [200, 4]
let coords = Tensor::arange(0u32, side as u32, &self.device)?.to_dtype(candle_core::DType::F32)?.affine(1.0 / side as f64, 0.0)?;
let coords = coords.reshape((1, side, 1))?;
let log_scale = |d: Tensor| -> Result<Tensor> {
let d = (d * 8.0)?;
Ok(d.sign()?.mul(&((d.abs()? + 1.0)?.log()? / (8f64.ln()))?)?)
};
let pick = |a: usize, b: usize| -> Result<Tensor> { Ok(Tensor::cat(&[xyxy.narrow(1, a, 1)?, xyxy.narrow(1, b, 1)?], 1)?.unsqueeze(1)?) };
let dy = log_scale(coords.broadcast_sub(&pick(1, 3)?)?)?; // [200, side, 2]
let dx = log_scale(coords.broadcast_sub(&pick(0, 2)?)?)?;
let ey = self.rpb_y.forward(&dy)?; // [200, side, heads]
let ex = self.rpb_x.forward(&dx)?;
let m = ey.unsqueeze(2)?.broadcast_add(&ex.unsqueeze(1)?)?; // [200, side, side, heads]
let m = m.reshape((QUERIES, side * side, HEADS))?.permute((2, 0, 1))?; // [heads, 200, side²]
let m = m.pad_with_zeros(1, 1, 0)?; // the presence token attends everywhere
Ok(m.unsqueeze(0)?.contiguous()?)
}
/// Sine embedding of boxes (cx, cy, w, h) `[1, 200, 512]` (y, x, w, h features).
fn box_sine(&self, boxes: &Tensor) -> Result<Tensor> {
let feats = HIDDEN / 2;
let dim_t: Vec<f32> = (0..feats).map(|i| 10000f64.powf(2.0 * (i / 2) as f64 / feats as f64) as f32).collect();
let dim_t = Tensor::from_vec(dim_t, (1, 1, feats), &self.device)?;
let even: Vec<f32> = (0..feats).map(|i| if i % 2 == 0 { 1.0 } else { 0.0 }).collect();
let even = Tensor::from_vec(even, (1, 1, feats), &self.device)?;
let one = |i: usize| -> Result<Tensor> {
let p = (boxes.narrow(D::Minus1, i, 1)? * (2.0 * std::f64::consts::PI))?.broadcast_div(&dim_t)?;
Ok((p.sin()?.broadcast_mul(&even)? + p.cos()?.broadcast_mul(&(1.0 - &even)?)?)?)
};
Ok(Tensor::cat(&[one(1)?, one(0)?, one(2)?, one(3)?], D::Minus1)?)
}
/// Run the detector for one tokenized prompt.
pub fn predict(&self, f: &Features, ids: &[u32], valid: &[bool]) -> Result<Output> {
let text = self.text_proj.forward(&self.text.forward(ids, valid)?)?; // [1, 32, 256]
let text_bias = padding_bias(valid, &self.device)?;
// DETR encoder: image tokens attend to themselves and to the prompt
let mut x = f.flat.clone();
for l in &self.encoder {
let h = l.ln1.forward(&x)?;
let hp = (&h + &f.pos)?;
x = (&x + l.self_attn.forward(&hp, &hp, &h, None)?)?;
let h = l.ln2.forward(&x)?;
x = (&x + l.cross_attn.forward(&h, &text, &text, Some(&text_bias))?)?;
let h = l.ln3.forward(&x)?;
x = (&x + l.mlp.forward(&h)?)?;
}
let memory = x;
// DETR decoder: presence token + 200 queries with iterative box refinement
let mut boxes = candle_nn::ops::sigmoid(&self.reference_points)?;
let mut h = Tensor::cat(&[self.presence_token.clone(), self.query_embed.clone()], 1)?;
let key_pos = (&memory + &f.pos)?;
let mut queries = None;
let mut presence = None;
for l in &self.decoder {
let qpos = self.ref_point_head.forward(&self.box_sine(&boxes)?)?.pad_with_zeros(1, 1, 0)?;
let bias = self.rpb(&boxes, f.side)?;
let q = (&h + &qpos)?;
h = l.self_ln.forward(&(&h + l.self_attn.forward(&q, &q, &h, None)?)?)?;
let q = (&h + &qpos)?;
h = l.text_ln.forward(&(&h + l.text_attn.forward(&q, &text, &text, Some(&text_bias))?)?)?;
let q = (&h + &qpos)?;
h = l.vision_ln.forward(&(&h + l.vision_attn.forward(&q, &key_pos, &memory, Some(&bias))?)?)?;
h = l.mlp_ln.forward(&(&h + l.mlp.forward(&h)?)?)?;
let qh = self.out_ln.forward(&h.narrow(1, 1, QUERIES)?)?;
boxes = candle_nn::ops::sigmoid(&(self.box_head.forward(&qh)? + inverse_sigmoid(&boxes)?)?)?;
let p = self.presence_head.forward(&self.presence_ln.forward(&h.narrow(1, 0, 1)?)?)?;
presence = Some(p.clamp(-10f32, 10f32)?);
queries = Some(qh);
}
let (Some(queries), Some(presence)) = (queries, presence) else { return Err(Error::Model("decoder has no layers".into())) };
// dot-product scoring against the mean of the (refined) prompt tokens
let t = self.score_text_ln.forward(&(self.score_text_mlp.forward(&text)? + &text)?)?;
let w: Vec<f32> = valid.iter().map(|v| if *v { 1.0 } else { 0.0 }).collect();
let count = w.iter().sum::<f32>().max(1.0);
let w = Tensor::from_vec(w, (1, valid.len(), 1), &self.device)?;
let pooled = (t.broadcast_mul(&w)?.sum(1)? / f64::from(count))?; // [1, 256]
let pt = self.score_text_proj.forward(&pooled)?; // [1, 256]
let pq = self.score_query_proj.forward(&queries)?; // [1, 200, 256]
let logits = (pq.matmul(&pt.unsqueeze(2)?)? / (HIDDEN as f64).sqrt())?.clamp(-12f32, 12f32)?;
// masks: the encoder memory attends to the prompt, then a pixel decoder up to 288²
let mem = (&memory + self.prompt_attn.forward(&self.prompt_ln.forward(&memory)?, &text, &text, Some(&text_bias))?)?;
let [l0, l1, _] = &f.levels[..] else { return Err(Error::Model("detector levels".into())) };
let mut prev = mem.transpose(1, 2)?.reshape((1, HIDDEN, f.side, f.side))?;
for (i, skip) in [l1, l0].into_iter().enumerate() {
let (_, _, sh, sw) = skip.dims4()?;
prev = (prev.upsample_nearest2d(sh, sw)? + skip)?;
let (Some(conv), Some(norm)) = (self.pixel_convs.get(i), self.pixel_norms.get(i)) else {
return Err(Error::Model("pixel decoder".into()));
};
prev = norm.forward(&conv.forward(&prev)?)?.relu()?;
}
let inst = self.instance_proj.forward(&prev)?; // [1, 256, 288, 288]
let (_, c, mh, mw) = inst.dims4()?;
let mut e = queries.clone();
for (i, l) in self.mask_embedder.iter().enumerate() {
e = l.forward(&e)?;
if i + 1 < self.mask_embedder.len() {
e = e.relu()?;
}
}
let masks = e.matmul(&inst.reshape((1, c, mh * mw))?)?.reshape((QUERIES, mh, mw))?;
let boxes = cxcywh_to_xyxy(&boxes)?;
let bv = to_vec(&boxes)?;
Ok(Output {
masks,
logits: to_vec(&logits)?,
presence: to_vec(&presence)?.first().copied().unwrap_or(-10.0),
boxes: bv.as_chunks::<4>().0.to_vec(),
text,
})
}
}
/// The instances of `out` scoring above `threshold` (sigmoid(logit) × sigmoid(presence)),
/// merged into one mask (the per-pixel maximum of their probabilities, as logits).
pub fn merge(out: &Output, threshold: f32) -> Option<Prediction> {
let s = |x: f32| 1.0 / (1.0 + (-x).exp());
let presence = s(out.presence);
let keep: Vec<(usize, f32)> = out.logits.iter().enumerate().map(|(i, l)| (i, s(*l) * presence)).filter(|(_, p)| *p > threshold).collect();
let best = keep.iter().map(|k| k.1).fold(0f32, f32::max);
let mut merged: Option<Vec<f32>> = None;
for (i, _) in &keep {
let m = out.masks.i(*i).ok().and_then(|t| to_vec(&t).ok())?;
merged = Some(match merged {
None => m,
Some(acc) => acc.iter().zip(&m).map(|(a, b)| a.max(*b)).collect(),
});
}
merged.map(|logits| Prediction { logits, score: best })
}
+541
View File
@@ -0,0 +1,541 @@
//! A small HTTP/1.1 GET client for model downloads, in pure Rust: `std::net` sockets and, for
//! `https://`, rustls with the RustCrypto provider (`rustls-rustcrypto`; no `ring` or
//! `aws-lc-rs`, so nothing is compiled from C or assembly) and the Mozilla root certificates
//! (`webpki-roots`).
//!
//! Only what a large-file download needs: one request per connection (`Connection: close`),
//! `Range` requests for resuming, redirects (followed by the caller), `Content-Length`,
//! chunked and read-to-close bodies. Every read waits at most `stall` for data and checks the
//! cancel flag at least twice a second, so a dead server or a cancelled download never hangs
//! the downloading thread. No proxies (see docs/ai-masks.md).
use std::io::{ErrorKind, Read, Write};
use std::net::{TcpStream, ToSocketAddrs};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
/// Why a request failed.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum HttpError {
/// The URL can't be used (scheme, host, characters).
BadUrl(String),
/// Name resolution or connecting failed (or took longer than the connect timeout).
Connect(String),
/// No data arrived for the stall timeout.
Stalled,
/// The cancel flag was raised.
Cancelled,
/// TLS failed (certificate, handshake).
Tls(String),
/// The server's response could not be understood.
Protocol(String),
/// Any other I/O error.
Io(String),
}
impl std::fmt::Display for HttpError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
HttpError::BadUrl(e) => write!(f, "unusable URL: {e}"),
HttpError::Connect(e) => write!(f, "could not connect: {e}"),
HttpError::Stalled => write!(f, "the server stopped sending data"),
HttpError::Cancelled => write!(f, "cancelled"),
HttpError::Tls(e) => write!(f, "secure connection failed: {e}"),
HttpError::Protocol(e) => write!(f, "unexpected response: {e}"),
HttpError::Io(e) => write!(f, "{e}"),
}
}
}
/// A parsed `http://` or `https://` URL.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Url {
pub tls: bool,
pub host: String,
pub port: u16,
/// Path and query, starting with `/`.
pub path: String,
}
impl Url {
pub fn parse(s: &str) -> Result<Url, HttpError> {
let bad = |why: &str| HttpError::BadUrl(format!("{s}: {why}"));
let s = s.trim();
if s.chars().any(|c| c.is_control() || c == ' ') {
return Err(bad("contains spaces or control characters"));
}
let (tls, rest) = if let Some(r) = s.strip_prefix("https://") {
(true, r)
} else if let Some(r) = s.strip_prefix("http://") {
(false, r)
} else {
return Err(bad("only http:// and https:// are supported"));
};
let rest = rest.split('#').next().unwrap_or_default();
let (authority, path) = match rest.find(['/', '?']) {
Some(i) => (rest.get(..i).unwrap_or_default(), rest.get(i..).unwrap_or_default()),
None => (rest, ""),
};
let path = if path.starts_with('/') { path.to_string() } else { format!("/{path}") };
if authority.contains('@') {
return Err(bad("user names and passwords in URLs are not supported"));
}
let (host, port) = if let Some(v6) = authority.strip_prefix('[') {
let end = v6.find(']').ok_or_else(|| bad("unterminated IPv6 address"))?;
let host = v6.get(..end).unwrap_or_default();
let after = v6.get(end + 1..).unwrap_or_default();
(host.to_string(), after.strip_prefix(':'))
} else {
match authority.rsplit_once(':') {
Some((h, p)) => (h.to_string(), Some(p)),
None => (authority.to_string(), None),
}
};
let port = match port {
Some(p) => p.parse::<u16>().ok().filter(|p| *p != 0).ok_or_else(|| bad("bad port"))?,
None if tls => 443,
None => 80,
};
if host.is_empty() || !host.chars().all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '.' | ':' | '_')) {
return Err(bad("bad host name"));
}
Ok(Url { tls, host: host.to_ascii_lowercase(), port, path })
}
/// `Location` of a redirect, resolved against this URL.
pub fn join(&self, location: &str) -> Result<Url, HttpError> {
let l = location.trim();
if l.starts_with("http://") || l.starts_with("https://") {
return Url::parse(l);
}
// another scheme (`ftp:`, `file:`, `data:`…)
if let Some((scheme, _)) = l.split_once(':')
&& !scheme.is_empty()
&& scheme.chars().all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.'))
{
return Err(HttpError::BadUrl(format!("redirect to an unsupported location `{scheme}:`")));
}
let scheme = if self.tls { "https" } else { "http" };
let host = if self.host.contains(':') { format!("[{}]", self.host) } else { self.host.clone() };
let base = format!("{scheme}://{host}:{}", self.port);
if let Some(rest) = l.strip_prefix("//") {
return Url::parse(&format!("{scheme}://{rest}"));
}
if l.starts_with('/') {
return Url::parse(&format!("{base}{l}"));
}
// relative to the current directory
let dir = self.path.split('?').next().unwrap_or("/");
let dir = dir.rsplit_once('/').map_or("", |(d, _)| d);
Url::parse(&format!("{base}{dir}/{l}"))
}
fn host_header(&self) -> String {
let host = if self.host.contains(':') { format!("[{}]", self.host) } else { self.host.clone() };
let default = if self.tls { 443 } else { 80 };
if self.port == default { host } else { format!("{host}:{}", self.port) }
}
/// The URL without its query (for messages: a query may hold a signed token).
pub fn display(&self) -> String {
let scheme = if self.tls { "https" } else { "http" };
format!("{scheme}://{}{}", self.host_header(), self.path.split('?').next().unwrap_or("/"))
}
}
/// Timeouts and the cancel flag for one request.
pub struct Limits<'a> {
pub connect: Duration,
pub stall: Duration,
pub cancel: &'a AtomicBool,
}
/// How often a blocked read wakes up to check the cancel flag.
const TICK: Duration = Duration::from_millis(250);
/// Longest status line + headers accepted.
const MAX_HEAD: usize = 64 * 1024;
enum Stream {
Plain(TcpStream),
Tls(Box<rustls::ClientConnection>, TcpStream),
}
fn is_timeout(e: &std::io::Error) -> bool {
matches!(e.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut | ErrorKind::Interrupted)
}
fn tls_config() -> Result<Arc<rustls::ClientConfig>, HttpError> {
let roots = rustls::RootCertStore { roots: webpki_roots::TLS_SERVER_ROOTS.to_vec() };
let config = rustls::ClientConfig::builder_with_provider(Arc::new(rustls_rustcrypto::provider()))
.with_safe_default_protocol_versions()
.map_err(|e| HttpError::Tls(e.to_string()))?
.with_root_certificates(roots)
.with_no_client_auth();
Ok(Arc::new(config))
}
impl Stream {
fn connect(url: &Url, limits: &Limits) -> Result<Stream, HttpError> {
let addrs: Vec<_> = (url.host.as_str(), url.port).to_socket_addrs().map_err(|e| HttpError::Connect(format!("{}: {e}", url.host)))?.collect();
if addrs.is_empty() {
return Err(HttpError::Connect(format!("{}: no address", url.host)));
}
let mut last = String::new();
let mut sock = None;
for a in addrs {
if limits.cancel.load(Ordering::Relaxed) {
return Err(HttpError::Cancelled);
}
match TcpStream::connect_timeout(&a, limits.connect) {
Ok(s) => {
sock = Some(s);
break;
}
Err(e) => last = format!("{}: {e}", url.host),
}
}
let sock = sock.ok_or(HttpError::Connect(last))?;
sock.set_read_timeout(Some(TICK)).map_err(|e| HttpError::Io(e.to_string()))?;
sock.set_write_timeout(Some(TICK)).map_err(|e| HttpError::Io(e.to_string()))?;
let _ = sock.set_nodelay(true);
if !url.tls {
return Ok(Stream::Plain(sock));
}
let name = rustls::pki_types::ServerName::try_from(url.host.clone()).map_err(|e| HttpError::Tls(format!("{}: {e}", url.host)))?;
let conn = rustls::ClientConnection::new(tls_config()?, name).map_err(|e| HttpError::Tls(e.to_string()))?;
let mut s = Stream::Tls(Box::new(conn), sock);
s.handshake(limits)?;
Ok(s)
}
/// Wait for `f` to make progress, retrying timeouts until `stall` passes without any.
fn retry<T>(limits: &Limits, mut f: impl FnMut() -> std::io::Result<T>) -> Result<T, HttpError> {
let start = Instant::now();
loop {
if limits.cancel.load(Ordering::Relaxed) {
return Err(HttpError::Cancelled);
}
match f() {
Ok(v) => return Ok(v),
Err(e) if is_timeout(&e) => {
if start.elapsed() >= limits.stall {
return Err(HttpError::Stalled);
}
}
Err(e) if e.kind() == ErrorKind::InvalidData => return Err(HttpError::Tls(e.to_string())),
Err(e) => return Err(HttpError::Io(e.to_string())),
}
}
}
fn handshake(&mut self, limits: &Limits) -> Result<(), HttpError> {
let Stream::Tls(conn, sock) = self else { return Ok(()) };
while conn.is_handshaking() {
Self::retry(limits, || conn.complete_io(sock))?;
}
Ok(())
}
fn write_all(&mut self, data: &[u8], limits: &Limits) -> Result<(), HttpError> {
match self {
Stream::Plain(sock) => {
let mut rest = data;
while !rest.is_empty() {
let n = Self::retry(limits, || sock.write(rest))?;
if n == 0 {
return Err(HttpError::Io("connection closed while sending the request".into()));
}
rest = rest.get(n..).unwrap_or_default();
}
Ok(())
}
Stream::Tls(conn, sock) => {
// buffered by rustls, then sent as TLS records
conn.writer().write_all(data).map_err(|e| HttpError::Tls(e.to_string()))?;
while conn.wants_write() {
Self::retry(limits, || conn.write_tls(sock))?;
}
Ok(())
}
}
}
/// Read some bytes (0 at the end of the stream), waiting at most `stall` for them.
fn read(&mut self, buf: &mut [u8], limits: &Limits) -> Result<usize, HttpError> {
match self {
Stream::Plain(sock) => Self::retry(limits, || sock.read(buf)),
Stream::Tls(conn, sock) => {
let start = Instant::now();
loop {
if limits.cancel.load(Ordering::Relaxed) {
return Err(HttpError::Cancelled);
}
match conn.reader().read(buf) {
Ok(n) => return Ok(n),
// closed without close_notify: the caller checks the length it got
Err(e) if e.kind() == ErrorKind::UnexpectedEof => return Ok(0),
Err(e) if e.kind() == ErrorKind::WouldBlock => {}
Err(e) => return Err(HttpError::Tls(e.to_string())),
}
// no plaintext buffered: more TLS records from the socket
match conn.read_tls(sock) {
Ok(0) => return Ok(0),
Ok(_) => {
conn.process_new_packets().map_err(|e| HttpError::Tls(e.to_string()))?;
while conn.wants_write() {
Self::retry(limits, || conn.write_tls(sock))?;
}
}
Err(e) if is_timeout(&e) => {
if start.elapsed() >= limits.stall {
return Err(HttpError::Stalled);
}
}
Err(e) => return Err(HttpError::Io(e.to_string())),
}
}
}
}
}
}
/// The connection with a read buffer (headers and chunk sizes are read line by line).
struct Conn {
stream: Stream,
buf: Vec<u8>,
pos: usize,
}
impl Conn {
fn fill(&mut self, limits: &Limits) -> Result<usize, HttpError> {
if self.pos >= self.buf.len() {
self.buf.clear();
self.pos = 0;
}
let mut tmp = [0u8; 64 * 1024];
let n = self.stream.read(&mut tmp, limits)?;
self.buf.extend_from_slice(tmp.get(..n).unwrap_or_default());
Ok(n)
}
/// One line without its CRLF; at most `max` bytes.
fn line(&mut self, max: usize, limits: &Limits) -> Result<String, HttpError> {
loop {
let avail = self.buf.get(self.pos..).unwrap_or_default();
if let Some(i) = avail.iter().position(|b| *b == b'\n') {
let line = avail.get(..i).unwrap_or_default();
let line = line.strip_suffix(b"\r").unwrap_or(line);
let s = String::from_utf8_lossy(line).into_owned();
self.pos = self.pos.saturating_add(i + 1);
return Ok(s);
}
if avail.len() > max {
return Err(HttpError::Protocol("header line too long".into()));
}
// keep the partial line, read more after it
if self.pos > 0 {
self.buf.drain(..self.pos.min(self.buf.len()));
self.pos = 0;
}
let mut tmp = [0u8; 16 * 1024];
let n = self.stream.read(&mut tmp, limits)?;
if n == 0 {
return Err(HttpError::Protocol("connection closed in the middle of the response".into()));
}
self.buf.extend_from_slice(tmp.get(..n).unwrap_or_default());
}
}
/// Up to `out.len()` body bytes (0 at the end of the stream).
fn read(&mut self, out: &mut [u8], limits: &Limits) -> Result<usize, HttpError> {
if self.pos >= self.buf.len() && self.fill(limits)? == 0 {
return Ok(0);
}
let avail = self.buf.get(self.pos..).unwrap_or_default();
let n = avail.len().min(out.len());
if let (Some(dst), Some(src)) = (out.get_mut(..n), avail.get(..n)) {
dst.copy_from_slice(src);
}
self.pos = self.pos.saturating_add(n);
Ok(n)
}
}
/// A response's status, headers and body.
pub struct Response {
pub status: u16,
headers: Vec<(String, String)>,
conn: Conn,
body: BodyMode,
}
enum BodyMode {
Length(u64),
Chunked { left: u64, done: bool },
Close,
Done,
}
impl Response {
/// The first header named `name` (case-insensitive).
pub fn header(&self, name: &str) -> Option<&str> {
self.headers.iter().find(|(k, _)| k.eq_ignore_ascii_case(name)).map(|(_, v)| v.as_str())
}
/// The body's length when the server announced it.
pub fn content_length(&self) -> Option<u64> {
match self.body {
BodyMode::Length(n) => Some(n),
_ => None,
}
}
/// Read body bytes into `out` (0 at the end of the body).
pub fn read(&mut self, out: &mut [u8], limits: &Limits) -> Result<usize, HttpError> {
match &mut self.body {
BodyMode::Done => Ok(0),
BodyMode::Length(left) => {
if *left == 0 {
self.body = BodyMode::Done;
return Ok(0);
}
let want = usize::try_from(*left).unwrap_or(usize::MAX).min(out.len());
let n = self.conn.read(out.get_mut(..want).unwrap_or_default(), limits)?;
if n == 0 {
return Err(HttpError::Protocol("the connection closed before the end of the file".into()));
}
*left = left.saturating_sub(n as u64);
Ok(n)
}
BodyMode::Close => {
let n = self.conn.read(out, limits)?;
if n == 0 {
self.body = BodyMode::Done;
}
Ok(n)
}
BodyMode::Chunked { left, done } => {
if *done {
return Ok(0);
}
if *left == 0 {
let line = self.conn.line(1024, limits)?;
let hex = line.split(';').next().unwrap_or_default().trim();
let size = u64::from_str_radix(hex, 16).map_err(|_| HttpError::Protocol(format!("bad chunk size `{hex}`")))?;
if size == 0 {
// trailers, up to the empty line
let mut n = 0;
while !self.conn.line(MAX_HEAD, limits)?.is_empty() {
n += 1;
if n > 100 {
return Err(HttpError::Protocol("too many trailers".into()));
}
}
*done = true;
return Ok(0);
}
*left = size;
}
let want = usize::try_from(*left).unwrap_or(usize::MAX).min(out.len());
let n = self.conn.read(out.get_mut(..want).unwrap_or_default(), limits)?;
if n == 0 {
return Err(HttpError::Protocol("the connection closed in the middle of a chunk".into()));
}
*left = left.saturating_sub(n as u64);
if *left == 0 && !self.conn.line(16, limits)?.is_empty() {
return Err(HttpError::Protocol("missing CRLF after a chunk".into()));
}
Ok(n)
}
}
}
}
/// Send `GET url` with extra `headers` and read the status line and headers.
pub fn get(url: &Url, headers: &[(&str, String)], limits: &Limits) -> Result<Response, HttpError> {
let mut req = format!(
"GET {} HTTP/1.1\r\nHost: {}\r\nUser-Agent: LightCraft/{}\r\nAccept: */*\r\nAccept-Encoding: identity\r\nConnection: close\r\n",
url.path,
url.host_header(),
env!("CARGO_PKG_VERSION")
);
for (k, v) in headers {
if v.chars().any(|c| c == '\r' || c == '\n') || k.chars().any(|c| c == '\r' || c == '\n' || c == ':') {
return Err(HttpError::BadUrl("header with a line break".into()));
}
req.push_str(&format!("{k}: {v}\r\n"));
}
req.push_str("\r\n");
let mut stream = Stream::connect(url, limits)?;
stream.write_all(req.as_bytes(), limits)?;
let mut conn = Conn { stream, buf: Vec::new(), pos: 0 };
let status_line = conn.line(MAX_HEAD, limits)?;
let mut parts = status_line.split_whitespace();
let version = parts.next().unwrap_or_default();
if !version.starts_with("HTTP/1.") {
return Err(HttpError::Protocol(format!("not an HTTP/1.x response: {}", status_line.chars().take(60).collect::<String>())));
}
let status = parts.next().and_then(|s| s.parse::<u16>().ok()).ok_or_else(|| HttpError::Protocol("bad status line".into()))?;
let mut headers = Vec::new();
let mut total = status_line.len();
loop {
let line = conn.line(MAX_HEAD, limits)?;
if line.is_empty() {
break;
}
total = total.saturating_add(line.len());
if total > MAX_HEAD || headers.len() > 200 {
return Err(HttpError::Protocol("response headers too large".into()));
}
if let Some((k, v)) = line.split_once(':') {
headers.push((k.trim().to_string(), v.trim().to_string()));
}
}
let find = |name: &str| headers.iter().find(|(k, _)| k.eq_ignore_ascii_case(name)).map(|(_, v)| v.as_str());
let body = if status == 204 || status == 304 || (100..200).contains(&status) {
BodyMode::Done
} else if find("transfer-encoding").is_some_and(|v| v.to_ascii_lowercase().contains("chunked")) {
BodyMode::Chunked { left: 0, done: false }
} else if let Some(n) = find("content-length") {
BodyMode::Length(n.trim().parse::<u64>().map_err(|_| HttpError::Protocol(format!("bad Content-Length `{n}`")))?)
} else {
BodyMode::Close
};
Ok(Response { status, headers, conn, body })
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parses_urls() {
let u = Url::parse("https://cdn.example.com/models/sam3/model.safetensors?sig=1#x").unwrap();
assert_eq!((u.tls, u.host.as_str(), u.port, u.path.as_str()), (true, "cdn.example.com", 443, "/models/sam3/model.safetensors?sig=1"));
assert_eq!(u.display(), "https://cdn.example.com/models/sam3/model.safetensors");
let u = Url::parse("http://127.0.0.1:8080").unwrap();
assert_eq!((u.tls, u.port, u.path.as_str()), (false, 8080, "/"));
let u = Url::parse("http://[::1]:9/a").unwrap();
assert_eq!((u.host.as_str(), u.port), ("::1", 9));
for bad in ["ftp://x/y", "https://", "https://a b/", "https://u:p@h/", "http://h:0/", "http://h:99999/", "https://h/\r\nX: y", "file:///etc"]
{
assert!(Url::parse(bad).is_err(), "{bad}");
}
}
#[test]
fn resolves_redirects() {
let u = Url::parse("https://a.example/x/y/file?q").unwrap();
assert_eq!(u.join("https://b.example/z").unwrap().host, "b.example");
assert_eq!(u.join("/root").unwrap().path, "/root");
assert_eq!(u.join("other").unwrap().path, "/x/y/other");
assert_eq!(u.join("//c.example/p").unwrap().host, "c.example");
assert!(u.join("ftp://nope").is_err());
}
#[test]
fn the_tls_config_builds_with_the_pure_rust_provider() {
assert!(tls_config().is_ok());
}
}
+424
View File
@@ -0,0 +1,424 @@
//! Downloading the SAM 3 model, only when the user asked for it: from an ordered list of mirrors
//! (the next one when a mirror fails), resuming partial files, with connect and stall timeouts,
//! a size cap, progress and cancellation, and checked (exact size, SHA-256 where pinned) before
//! each file is moved into the model folder. Runs on whatever thread calls [`download`] (the
//! engine uses a background thread; the UI thread never waits for the network).
//!
//! Files are written to `<name>.part` next to their final place and renamed when verified, so a
//! model folder never holds a half-written file under a real name, and an interrupted download
//! resumes from where it stopped. A file that fails its check is deleted, never kept.
//!
//! The weights are never part of LightCraft (SAM License, see docs/ai-masks.md); where they are
//! downloaded from is configured by [`DEFAULT_MIRRORS`] and the user's own list.
pub mod http;
use std::fs::{File, OpenOptions};
use std::io::{Read, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use sha2::{Digest, Sha256};
use http::{HttpError, Limits, Url};
/// One file of the model.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FileSpec {
pub name: &'static str,
/// Exact size in bytes, when pinned (then the download resumes and is checked against it).
pub size: Option<u64>,
/// SHA-256 (lower-case hex), when pinned.
pub sha256: Option<&'static str>,
/// Largest size accepted when `size` is not pinned.
pub max: u64,
}
/// The `facebook/sam3` checkpoint files [`crate::Sam3::load`] needs.
///
/// `model.safetensors` is pinned to the official checkpoint (Hugging Face LFS SHA-256). The two
/// tokenizer files are small text files without a pinned hash yet: they are only parsed by the
/// tokenizer (never executed) and capped in size.
// TODO(maintainer): pin `vocab.json` and `merges.txt` (size + SHA-256) when the files are
// uploaded to LightCraft's CDN.
pub const SAM3_FILES: &[FileSpec] = &[
FileSpec { name: "vocab.json", size: None, sha256: None, max: 16 << 20 },
FileSpec { name: "merges.txt", size: None, sha256: None, max: 16 << 20 },
FileSpec {
name: crate::WEIGHTS_FILE,
size: Some(SAM3_WEIGHTS_SIZE),
sha256: Some("6d06f0a5f84e435071fe6603e61d0b4cc7b40e0d39d487cfd4d67d8cc11cc14a"),
max: SAM3_WEIGHTS_SIZE,
},
];
/// Size of the official `model.safetensors` (3.44 GB).
pub const SAM3_WEIGHTS_SIZE: u64 = 3_439_938_512;
/// Where LightCraft downloads the model from, in order: each is a base URL, and file `f` is at
/// `<base>/<f>`. **Empty for now**: LightCraft has no download location of its own yet, so the
/// in-app download needs the user's list (`LIGHTCRAFT_SAM3_MIRRORS` or the mirrors file, see
/// [`mirrors`] and docs/ai-masks.md) until this is filled in.
///
/// Hugging Face's `facebook/sam3` can't be a default: it is gated (each user must accept the
/// SAM License there and download with their own token).
// TODO(maintainer): add LightCraft's CDN locations (https, primary first), e.g.
// "https://<cdn-host>/models/sam3/<revision>" — the files there must match `SAM3_FILES`.
pub const DEFAULT_MIRRORS: &[&str] = &[];
/// Environment variable with extra mirrors (base URLs separated by commas, spaces or newlines),
/// tried before the defaults.
pub const MIRRORS_ENV: &str = "LIGHTCRAFT_SAM3_MIRRORS";
/// Most mirrors used (a hostile list can't make a download loop forever).
const MAX_MIRRORS: usize = 16;
/// Redirects followed per request.
const MAX_REDIRECTS: usize = 8;
/// The mirrors to try, in order: the environment variable's, then the mirrors file's (one base
/// URL per line, `#` comments), then [`DEFAULT_MIRRORS`]; duplicates and unusable URLs dropped.
pub fn mirrors(env: Option<&str>, file: Option<&Path>) -> Vec<String> {
let mut out: Vec<String> = Vec::new();
let mut add = |s: &str| {
let s = s.trim().trim_end_matches('/');
if !s.is_empty() && Url::parse(s).is_ok() && !out.iter().any(|m| m == s) && out.len() < MAX_MIRRORS {
out.push(s.to_string());
}
};
if let Some(env) = env {
env.split([',', ' ', '\n', '\t']).for_each(&mut add);
}
// a small text file; anything larger isn't a mirror list
if let Some(text) = file.and_then(|f| std::fs::metadata(f).ok().filter(|m| m.len() < 64 * 1024).and_then(|_| std::fs::read_to_string(f).ok())) {
text.lines().map(|l| l.split('#').next().unwrap_or_default()).for_each(&mut add);
}
DEFAULT_MIRRORS.iter().for_each(|m| add(m));
out
}
/// Download settings.
#[derive(Clone, Debug)]
pub struct Options {
/// Longest wait to connect to a mirror.
pub connect_timeout: Duration,
/// Longest wait for the next bytes once connected.
pub stall_timeout: Duration,
/// Tries per mirror and file (each resumes where the last one stopped).
pub attempts: u32,
}
impl Default for Options {
fn default() -> Self {
Options { connect_timeout: Duration::from_secs(15), stall_timeout: Duration::from_secs(30), attempts: 3 }
}
}
/// Where a download is.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct Progress {
pub file: String,
/// Bytes of all files on disk so far, and the total expected (pinned sizes + announced ones).
pub done: u64,
pub total: u64,
/// The mirror in use (without query).
pub mirror: String,
}
/// Why a download failed.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum DownloadError {
/// No mirror is configured.
NoMirrors,
Cancelled,
/// A local file problem (folder not writable, disk full…).
Disk(String),
/// Every mirror failed for `file`; the reason for each.
AllMirrorsFailed {
file: String,
errors: Vec<String>,
},
}
impl std::fmt::Display for DownloadError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
DownloadError::NoMirrors => write!(
f,
"no download location is configured for the SAM 3 model in this build (set {MIRRORS_ENV}, or put the files in the model folder yourself; see docs/ai-masks.md)"
),
DownloadError::Cancelled => write!(f, "the download was cancelled"),
DownloadError::Disk(e) => write!(f, "{e}"),
DownloadError::AllMirrorsFailed { file, errors } => write!(f, "could not download {file}: {}", errors.join("; ")),
}
}
}
/// One mirror's failure for one file; `Retry` resumes on the same mirror.
enum Fail {
Retry(String),
NextMirror(String),
Cancelled,
Disk(String),
}
/// Download `files` into `dir` from `mirrors` (in order; see [`mirrors`]). Files already there
/// and verified are kept. `progress` is called as bytes arrive (often: throttle in it).
pub fn download(
files: &[FileSpec],
mirrors: &[String],
dir: &Path,
opts: &Options,
cancel: &AtomicBool,
progress: &mut dyn FnMut(&Progress),
) -> Result<(), DownloadError> {
std::fs::create_dir_all(dir).map_err(|e| DownloadError::Disk(format!("can't create {}: {e}", dir.display())))?;
// what's left to fetch
let mut todo = Vec::new();
for f in files {
if !verified(f, &dir.join(f.name))? {
todo.push(f);
}
}
if todo.is_empty() {
return Ok(());
}
if mirrors.is_empty() {
return Err(DownloadError::NoMirrors);
}
let total: u64 = files.iter().map(|f| f.size.unwrap_or(0)).sum();
let mut before: u64 = files.iter().filter(|f| !todo.contains(f)).filter_map(|f| f.size).sum();
for f in todo {
let mut errors = Vec::new();
let mut ok = false;
'mirrors: for m in mirrors {
let url = format!("{m}/{}", f.name);
for _ in 0..opts.attempts.max(1) {
if cancel.load(Ordering::Relaxed) {
return Err(DownloadError::Cancelled);
}
let mut report = |n: u64, announced: u64| {
let mirror = Url::parse(m).map(|u| u.display()).unwrap_or_default();
progress(&Progress {
file: f.name.to_string(),
done: before.saturating_add(n),
total: total.max(before.saturating_add(announced)),
mirror,
})
};
match fetch_file(f, &url, dir, opts, cancel, &mut report) {
Ok(()) => {
ok = true;
break 'mirrors;
}
Err(Fail::Cancelled) => return Err(DownloadError::Cancelled),
Err(Fail::Disk(e)) => return Err(DownloadError::Disk(e)),
Err(Fail::Retry(e)) => {
log::warn!("SAM 3 download of {url}: {e} (retrying)");
errors.push(format!("{}: {e}", short(m)));
}
Err(Fail::NextMirror(e)) => {
log::warn!("SAM 3 download of {url}: {e}");
errors.push(format!("{}: {e}", short(m)));
continue 'mirrors;
}
}
}
}
if !ok {
return Err(DownloadError::AllMirrorsFailed { file: f.name.to_string(), errors });
}
before = before.saturating_add(std::fs::metadata(dir.join(f.name)).map(|m| m.len()).unwrap_or(0));
}
Ok(())
}
/// A mirror for messages (no query string: it may hold a token).
fn short(m: &str) -> String {
Url::parse(m).map(|u| u.display()).unwrap_or_else(|_| "mirror".into())
}
/// Whether `path` is already the right file (exact size and hash when pinned).
fn verified(f: &FileSpec, path: &Path) -> Result<bool, DownloadError> {
let Ok(meta) = std::fs::metadata(path) else { return Ok(false) };
if !meta.is_file() || meta.len() == 0 {
return Ok(false);
}
if f.size.is_some_and(|s| s != meta.len()) || meta.len() > f.max {
return Ok(false);
}
match f.sha256 {
Some(want) => Ok(sha256_file(path).map_err(DownloadError::Disk)? == want),
None => Ok(true),
}
}
fn sha256_file(path: &Path) -> Result<String, String> {
let mut file = File::open(path).map_err(|e| format!("{}: {e}", path.display()))?;
let mut h = Sha256::new();
let mut buf = vec![0u8; 1 << 20];
loop {
let n = file.read(&mut buf).map_err(|e| format!("{}: {e}", path.display()))?;
if n == 0 {
break;
}
h.update(buf.get(..n).unwrap_or_default());
}
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
}
fn part_path(dir: &Path, name: &str) -> PathBuf {
dir.join(format!("{name}.part"))
}
/// Fetch one file from one mirror into `<name>.part` (resuming it when its size is pinned),
/// check it and move it into place.
fn fetch_file(f: &FileSpec, url: &str, dir: &Path, opts: &Options, cancel: &AtomicBool, report: &mut dyn FnMut(u64, u64)) -> Result<(), Fail> {
let part = part_path(dir, f.name);
let limit = f.size.unwrap_or(f.max);
let disk = |e: std::io::Error| Fail::Disk(format!("{}: {e}", part.display()));
// resume only a pinned file (an unpinned one can't be checked after resuming)
let mut have = match std::fs::metadata(&part) {
Ok(m) if f.size.is_some() && m.len() <= limit => m.len(),
Ok(_) => {
std::fs::remove_file(&part).map_err(disk)?;
0
}
Err(_) => 0,
};
let fetched = f.size != Some(have) || have == 0;
if fetched {
let limits = Limits { connect: opts.connect_timeout, stall: opts.stall_timeout, cancel };
let mut url = Url::parse(url).map_err(|e| Fail::NextMirror(e.to_string()))?;
let origin = url.host.clone();
let mut redirects = 0;
let mut resp = loop {
let mut headers = vec![];
if have > 0 {
headers.push(("Range", format!("bytes={have}-")));
}
// a token for the configured host only (never sent on to a redirect's host)
if let Ok(t) = std::env::var("LIGHTCRAFT_SAM3_TOKEN")
&& !t.trim().is_empty()
&& url.tls
&& url.host == origin
{
headers.push(("Authorization", format!("Bearer {}", t.trim())));
}
let r = http::get(&url, &headers, &limits).map_err(http_fail)?;
match r.status {
301 | 302 | 303 | 307 | 308 => {
redirects += 1;
if redirects > MAX_REDIRECTS {
return Err(Fail::NextMirror("too many redirects".into()));
}
let loc = r.header("location").ok_or_else(|| Fail::NextMirror("redirect without a location".into()))?;
url = url.join(loc).map_err(|e| Fail::NextMirror(e.to_string()))?;
}
_ => break r,
}
};
let mut file = match resp.status {
200 => {
// the whole file (from the start, also when a range was asked for)
if resp.content_length().is_some_and(|n| n > limit || f.size.is_some_and(|s| s != n)) {
return Err(Fail::NextMirror(format!("the server's file has the wrong size ({:?} bytes)", resp.content_length())));
}
have = 0;
File::create(&part).map_err(disk)?
}
206 if have > 0 => {
let range = resp.header("content-range").unwrap_or_default();
let (start, total) = parse_content_range(range).ok_or_else(|| Fail::NextMirror(format!("bad Content-Range `{range}`")))?;
if start != have || total.is_some_and(|t| Some(t) != f.size) {
return Err(Fail::NextMirror(format!("the server resumed at the wrong place ({range})")));
}
OpenOptions::new().append(true).open(&part).map_err(disk)?
}
416 => {
// the partial file doesn't fit what the server has: start over
std::fs::remove_file(&part).map_err(disk)?;
return Err(Fail::Retry("the partial download didn't match; starting over".into()));
}
401 | 403 => return Err(Fail::NextMirror(format!("access denied (HTTP {})", resp.status))),
404 | 410 => return Err(Fail::NextMirror(format!("not found (HTTP {})", resp.status))),
s if s >= 500 => return Err(Fail::Retry(format!("server error (HTTP {s})"))),
s => return Err(Fail::NextMirror(format!("HTTP {s}"))),
};
let announced = f.size.unwrap_or_else(|| resp.content_length().unwrap_or(0));
report(have, announced);
let mut buf = vec![0u8; 256 * 1024];
loop {
let n = match resp.read(&mut buf, &limits) {
Ok(n) => n,
Err(e) => {
let _ = file.flush();
if f.size.is_none() {
drop(file);
let _ = std::fs::remove_file(&part);
}
return Err(http_fail(e));
}
};
if n == 0 {
break;
}
if have.saturating_add(n as u64) > limit {
drop(file);
let _ = std::fs::remove_file(&part);
return Err(Fail::NextMirror(format!("the server sent more than the expected {limit} bytes")));
}
file.write_all(buf.get(..n).unwrap_or_default()).map_err(disk)?;
have = have.saturating_add(n as u64);
report(have, announced);
}
file.sync_all().map_err(disk)?;
}
// checks: exact size, hash
let len = std::fs::metadata(&part).map_err(disk)?.len();
if f.size.is_some_and(|s| s != len) {
return Err(Fail::Retry(format!("incomplete ({len} of {} bytes)", f.size.unwrap_or(0))));
}
if len == 0 {
let _ = std::fs::remove_file(&part);
return Err(Fail::NextMirror("the server sent an empty file".into()));
}
if let Some(want) = f.sha256 {
let got = sha256_file(&part).map_err(Fail::Disk)?;
if got != want {
std::fs::remove_file(&part).map_err(disk)?;
let why = format!("the file is damaged or not the official one (SHA-256 {got})");
// a damaged partial file from an earlier run isn't this mirror's fault
return Err(if fetched { Fail::NextMirror(why) } else { Fail::Retry(why) });
}
}
std::fs::rename(&part, dir.join(f.name)).map_err(disk)?;
Ok(())
}
fn http_fail(e: HttpError) -> Fail {
match e {
HttpError::Cancelled => Fail::Cancelled,
HttpError::Stalled | HttpError::Io(_) | HttpError::Protocol(_) => Fail::Retry(e.to_string()),
HttpError::BadUrl(_) | HttpError::Connect(_) | HttpError::Tls(_) => Fail::NextMirror(e.to_string()),
}
}
/// `bytes a-b/total` → (a, total) (`*` total → None).
fn parse_content_range(v: &str) -> Option<(u64, Option<u64>)> {
let v = v.trim().strip_prefix("bytes")?.trim();
let (range, total) = v.split_once('/')?;
let (a, b) = range.split_once('-')?;
let (a, b) = (a.trim().parse::<u64>().ok()?, b.trim().parse::<u64>().ok()?);
if b < a {
return None;
}
let total = match total.trim() {
"*" => None,
t => Some(t.parse::<u64>().ok()?),
};
Some((a, total))
}
#[cfg(test)]
mod tests;
+314
View File
@@ -0,0 +1,314 @@
//! The downloader against a local HTTP server (std `TcpListener`; never the internet): mirror
//! fallback, redirects, resuming, chunked bodies, stalls, hash mismatches, size caps, cancel.
use std::io::{BufRead, BufReader, Write};
use std::net::{TcpListener, TcpStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use super::*;
/// The served file: 300 kB of a deterministic pattern.
fn payload() -> Vec<u8> {
(0..300_000u32).map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect()
}
fn sha_hex(b: &[u8]) -> String {
Sha256::digest(b).iter().map(|b| format!("{b:02x}")).collect()
}
fn leak(s: String) -> &'static str {
Box::leak(s.into_boxed_str())
}
fn pinned() -> FileSpec {
let p = payload();
FileSpec { name: "model.bin", size: Some(p.len() as u64), sha256: Some(leak(sha_hex(&p))), max: p.len() as u64 }
}
struct Server {
base: String,
hits: Arc<AtomicUsize>,
}
/// Serve on 127.0.0.1; the first path segment picks the behaviour.
fn server() -> Server {
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let base = format!("http://{}", l.local_addr().unwrap());
let hits = Arc::new(AtomicUsize::new(0));
let h = hits.clone();
std::thread::spawn(move || {
for s in l.incoming().flatten() {
let h = h.clone();
std::thread::spawn(move || handle(s, &h));
}
});
Server { base, hits }
}
fn handle(mut s: TcpStream, hits: &AtomicUsize) {
let n = hits.fetch_add(1, Ordering::SeqCst);
let mut r = BufReader::new(s.try_clone().unwrap());
let mut first = String::new();
if r.read_line(&mut first).is_err() {
return;
}
let mut range: Option<u64> = None;
loop {
let mut l = String::new();
if r.read_line(&mut l).unwrap_or(0) == 0 || l.trim().is_empty() {
break;
}
if let Some(v) = l.to_ascii_lowercase().strip_prefix("range: bytes=") {
range = v.trim().trim_end_matches('-').parse().ok();
}
}
let path = first.split_whitespace().nth(1).unwrap_or("/").to_string();
let mode = path.trim_start_matches('/').split('/').next().unwrap_or("").to_string();
let body = payload();
let send = |s: &mut TcpStream, status: &str, headers: &str, data: &[u8]| {
let _ = s.write_all(format!("HTTP/1.1 {status}\r\n{headers}Connection: close\r\n\r\n").as_bytes());
let _ = s.write_all(data);
};
match mode.as_str() {
"missing" => send(&mut s, "404 Not Found", "Content-Length: 9\r\n", b"not found"),
"error" => send(&mut s, "500 Oops", "Content-Length: 25\r\n", b"<html>server error</html>"),
"redirect" => send(&mut s, "302 Found", &format!("Location: {}\r\nContent-Length: 0\r\n", path.replacen("/redirect", "/good", 1)), b""),
"wrong" => {
let mut b = body.clone();
b[1000] ^= 0xff;
send(&mut s, "200 OK", &format!("Content-Length: {}\r\n", b.len()), &b)
}
"toolong" => {
// no length announced, more bytes than the file can have
let mut b = body.clone();
b.extend_from_slice(&[7u8; 5000]);
send(&mut s, "200 OK", "", &b)
}
"chunked" => {
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n");
for c in body.chunks(70_001) {
let _ = s.write_all(format!("{:x};ext=1\r\n", c.len()).as_bytes());
let _ = s.write_all(c);
let _ = s.write_all(b"\r\n");
}
let _ = s.write_all(b"0\r\nX-Trailer: 1\r\n\r\n");
}
"stall" => {
// half the file, then nothing
let _ = s.write_all(format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n", body.len()).as_bytes());
let _ = s.write_all(&body[..body.len() / 2]);
std::thread::sleep(Duration::from_secs(5));
}
"slow" => {
let _ = s.write_all(format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n", body.len()).as_bytes());
for c in body.chunks(1000) {
if s.write_all(c).is_err() {
return;
}
std::thread::sleep(Duration::from_millis(20));
}
}
"flaky" if n.is_multiple_of(2) && range.is_none() => {
// the first try breaks off after a third
let _ = s.write_all(format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n", body.len()).as_bytes());
let _ = s.write_all(&body[..body.len() / 3]);
}
"range416" if range.is_some() => send(&mut s, "416 Range Not Satisfiable", "Content-Length: 0\r\n", b""),
"norange" | "range416" => send(&mut s, "200 OK", &format!("Content-Length: {}\r\n", body.len()), &body),
_ => match range {
Some(from) if from as usize >= body.len() => send(&mut s, "416 Range Not Satisfiable", "Content-Length: 0\r\n", b""),
Some(from) => send(
&mut s,
"206 Partial Content",
&format!("Content-Range: bytes {from}-{}/{}\r\nContent-Length: {}\r\n", body.len() - 1, body.len(), body.len() - from as usize),
&body[from as usize..],
),
None => send(&mut s, "200 OK", &format!("Content-Length: {}\r\n", body.len()), &body),
},
}
}
fn tmp(name: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-sam3-fetch-{name}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
d
}
fn quick() -> Options {
Options { connect_timeout: Duration::from_secs(2), stall_timeout: Duration::from_millis(600), attempts: 2 }
}
fn run(files: &[FileSpec], mirrors: &[String], dir: &Path, cancel: &AtomicBool) -> (Result<(), DownloadError>, Vec<Progress>) {
let mut seen = Vec::new();
let r = download(files, mirrors, dir, &quick(), cancel, &mut |p| seen.push(p.clone()));
(r, seen)
}
#[test]
fn falls_back_to_the_next_mirror_and_verifies() {
let srv = server();
let dir = tmp("fallback");
let mirrors = vec![
format!("{}/missing", srv.base),
format!("{}/error", srv.base),
"http://127.0.0.1:1/unreachable".to_string(),
format!("{}/redirect", srv.base),
];
let (r, seen) = run(&[pinned()], &mirrors, &dir, &AtomicBool::new(false));
r.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
assert!(!part_path(&dir, "model.bin").exists());
let last = seen.last().unwrap();
assert_eq!((last.done, last.total), (300_000, 300_000));
// an error page never ends up in the file: nothing left over from the failed mirrors
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 1);
// installed: nothing is fetched again
let hits = srv.hits.load(Ordering::SeqCst);
run(&[pinned()], &mirrors, &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(srv.hits.load(Ordering::SeqCst), hits);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_hash_mismatch_is_deleted_and_the_next_mirror_used() {
let srv = server();
let dir = tmp("hash");
let (r, _) = run(&[pinned()], &[format!("{}/wrong", srv.base)], &dir, &AtomicBool::new(false));
let e = r.unwrap_err().to_string();
assert!(e.contains("SHA-256"), "{e}");
assert!(!dir.join("model.bin").exists() && !part_path(&dir, "model.bin").exists(), "a damaged file is never kept");
run(&[pinned()], &[format!("{}/wrong", srv.base), format!("{}/good", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn resumes_partial_files_and_restarts_bad_ones() {
let srv = server();
let dir = tmp("resume");
std::fs::create_dir_all(&dir).unwrap();
// a third already there
std::fs::write(part_path(&dir, "model.bin"), &payload()[..100_000]).unwrap();
run(&[pinned()], &[format!("{}/good", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
// a partial file with the wrong bytes: the hash fails, it's deleted, the next try starts over
std::fs::remove_file(dir.join("model.bin")).unwrap();
std::fs::write(part_path(&dir, "model.bin"), vec![0u8; 100_000]).unwrap();
run(&[pinned()], &[format!("{}/good", srv.base), format!("{}/good", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
// a server ignoring Range: the whole file again, not appended
std::fs::remove_file(dir.join("model.bin")).unwrap();
std::fs::write(part_path(&dir, "model.bin"), &payload()[..5]).unwrap();
run(&[pinned()], &[format!("{}/norange", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
// a connection breaking off is resumed on the same mirror
std::fs::remove_file(dir.join("model.bin")).unwrap();
run(&[pinned()], &[format!("{}/flaky", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_416_or_a_bad_complete_part_starts_over() {
let srv = server();
let dir = tmp("416");
std::fs::create_dir_all(&dir).unwrap();
// the server refuses the range: the part is dropped and the file fetched whole (no loop)
std::fs::write(part_path(&dir, "model.bin"), &payload()[..100]).unwrap();
run(&[pinned()], &[format!("{}/range416", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
// as long as the file but wrong: hash fails → deleted → downloaded again from the same mirror
std::fs::remove_file(dir.join("model.bin")).unwrap();
std::fs::write(part_path(&dir, "model.bin"), vec![1u8; 300_000]).unwrap();
run(&[pinned()], &[format!("{}/good", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn chunked_bodies_and_unpinned_files() {
let srv = server();
let dir = tmp("chunked");
let f = FileSpec { name: "vocab.json", size: None, sha256: None, max: 1 << 20 };
run(std::slice::from_ref(&f), &[format!("{}/chunked", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("vocab.json")).unwrap(), payload());
// more than the cap: refused, nothing kept
let small = FileSpec { max: 1000, ..f };
let dir2 = tmp("cap");
let e = run(&[small], &[format!("{}/good", srv.base)], &dir2, &AtomicBool::new(false)).0.unwrap_err().to_string();
assert!(e.contains("wrong size") || e.contains("more than"), "{e}");
assert!(!dir2.join("vocab.json").exists() && !part_path(&dir2, "vocab.json").exists());
let _ = std::fs::remove_dir_all(&dir);
let _ = std::fs::remove_dir_all(&dir2);
}
#[test]
fn a_server_sending_too_much_is_cut_off() {
let srv = server();
let dir = tmp("toolong");
let e = run(&[pinned()], &[format!("{}/toolong", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap_err().to_string();
assert!(e.contains("more than"), "{e}");
assert!(!dir.join("model.bin").exists() && !part_path(&dir, "model.bin").exists());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_stalled_server_times_out_and_the_next_mirror_finishes() {
let srv = server();
let dir = tmp("stall");
let t = Instant::now();
let (r, _) = run(&[pinned()], &[format!("{}/stall", srv.base), format!("{}/good", srv.base)], &dir, &AtomicBool::new(false));
r.unwrap();
assert!(t.elapsed() < Duration::from_secs(4), "{:?}", t.elapsed());
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn cancel_stops_quickly_and_keeps_the_part_for_later() {
let srv = server();
let dir = tmp("cancel");
let cancel = Arc::new(AtomicBool::new(false));
let c = cancel.clone();
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(400));
c.store(true, Ordering::SeqCst);
});
let t = Instant::now();
let (r, _) = run(&[pinned()], &[format!("{}/slow", srv.base)], &dir, &cancel);
assert_eq!(r, Err(DownloadError::Cancelled));
assert!(t.elapsed() < Duration::from_secs(3), "{:?}", t.elapsed());
assert!(!dir.join("model.bin").exists());
let kept = std::fs::metadata(part_path(&dir, "model.bin")).map(|m| m.len()).unwrap_or(0);
assert!(kept > 0 && kept < 300_000, "{kept}");
// and the next run resumes it
run(&[pinned()], &[format!("{}/good", srv.base)], &dir, &AtomicBool::new(false)).0.unwrap();
assert_eq!(std::fs::read(dir.join("model.bin")).unwrap(), payload());
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn no_mirrors_is_a_clear_error_and_mirror_lists_are_parsed() {
let dir = tmp("nomirror");
let e = run(&[pinned()], &[], &dir, &AtomicBool::new(false)).0.unwrap_err();
assert_eq!(e, DownloadError::NoMirrors);
assert!(e.to_string().contains(MIRRORS_ENV));
let file = std::env::temp_dir().join(format!("lc-sam3-mirrors-{}.txt", std::process::id()));
std::fs::write(&file, "# mine\nhttps://b.example/m/ # second\n\nnot a url\nhttps://a.example/x\n").unwrap();
let m = mirrors(Some("https://a.example/x/, ftp://no"), Some(&file));
assert_eq!(m[..2], ["https://a.example/x".to_string(), "https://b.example/m".to_string()]);
assert_eq!(m.len(), 2 + DEFAULT_MIRRORS.len());
assert_eq!(mirrors(None, Some(Path::new("/nonexistent/mirrors.txt"))).len(), DEFAULT_MIRRORS.len());
let _ = std::fs::remove_file(&file);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn content_ranges() {
assert_eq!(parse_content_range("bytes 100-299/300"), Some((100, Some(300))));
assert_eq!(parse_content_range("bytes 0-0/*"), Some((0, None)));
assert_eq!(parse_content_range("bytes 9-1/300"), None);
assert_eq!(parse_content_range("items 1-2/3"), None);
}
+263
View File
@@ -0,0 +1,263 @@
//! SAM 3 (Segment Anything with Concepts, Meta, 2025) inference in pure Rust on candle.
//!
//! - **Point prompts** ([`Sam3::segment_clicks`]): positive and negative clicks select one
//! object, via SAM 3's interactive (SAM 2 style) prompt encoder and mask decoder.
//! - **Text prompts** ([`Sam3::segment_text`]): a noun phrase ("the red car", "sky") selects
//! every instance of that concept, via the CLIP text encoder and the DETR detector.
//!
//! The expensive part, the 32-layer ViT over the 1008 × 1008 input, runs once per image
//! ([`Sam3::encode`]); each click or prompt then reuses the [`Encoded`] features.
//!
//! On macOS the model runs on the GPU through Metal; elsewhere (for now) on the CPU. The
//! weights are not part of LightCraft: they are the `facebook/sam3` checkpoint (SAM License),
//! read from a directory the user downloads them to ([`Sam3::load`]).
//!
//! Ported from the Hugging Face `transformers` implementation (Apache-2.0); see `NOTICE`.
//! This crate is therefore licensed under the Apache License 2.0 only (not the MIT option of
//! the rest of LightCraft). No UI dependencies (L3).
//!
//! [`fetch`] downloads the model, only when the user asks for it (from configurable mirrors,
//! verified before use). Nothing in LightCraft requires the model: without it, AI masks report
//! that it isn't installed and everything else works.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`src/transformers/models/sam3` and `sam3_tracker`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
#[cfg(not(target_arch = "wasm32"))]
mod clip;
#[cfg(not(target_arch = "wasm32"))]
mod detector;
#[cfg(not(target_arch = "wasm32"))]
pub mod fetch;
pub mod mask;
#[cfg(not(target_arch = "wasm32"))]
mod neck;
#[cfg(not(target_arch = "wasm32"))]
mod nn;
pub mod tokenizer;
#[cfg(not(target_arch = "wasm32"))]
mod tracker;
#[cfg(not(target_arch = "wasm32"))]
mod vit;
#[cfg(not(target_arch = "wasm32"))]
mod weights;
use std::path::{Path, PathBuf};
#[cfg(not(target_arch = "wasm32"))]
use std::sync::Arc;
#[cfg(not(target_arch = "wasm32"))]
use candle_core::{DType, Device, Tensor};
#[cfg(not(target_arch = "wasm32"))]
use candle_nn::VarBuilder;
#[cfg(not(target_arch = "wasm32"))]
pub use tracker::Click;
#[derive(Debug, thiserror::Error)]
pub enum Error {
#[error("SAM 3 model: {0}")]
Model(String),
#[error("SAM 3 model files not found in {0} (expected model.safetensors, vocab.json, merges.txt)")]
Missing(PathBuf),
#[cfg(not(target_arch = "wasm32"))]
#[error(transparent)]
Candle(#[from] candle_core::Error),
}
pub type Result<T> = std::result::Result<T, Error>;
/// Side of the square model input.
pub const INPUT: usize = 1008;
/// Side of the low-resolution masks the model predicts.
pub const MASK_SIDE: usize = 288;
#[cfg(not(target_arch = "wasm32"))]
const _: () = assert!(MASK_SIDE == tracker::LOW_RES && INPUT == vit::IMAGE);
/// The files of a model directory.
pub const WEIGHTS_FILE: &str = "model.safetensors";
/// Whether `dir` holds a usable checkpoint.
pub fn is_model_dir(dir: &Path) -> bool {
dir.join(WEIGHTS_FILE).is_file() && dir.join("vocab.json").is_file() && dir.join("merges.txt").is_file()
}
#[cfg(not(target_arch = "wasm32"))]
/// The best device here: Metal on macOS (when a GPU is available), else the CPU.
pub fn best_device() -> Device {
#[cfg(target_os = "macos")]
if let Ok(d) = Device::new_metal(0) {
return d;
}
Device::Cpu
}
#[cfg(not(target_arch = "wasm32"))]
/// A loaded model. The vision backbone loads with it; the point and text heads load on first use.
pub struct Sam3 {
dir: PathBuf,
device: Device,
weights: Arc<weights::Weights>,
backbone: vit::Vit,
tracker_neck: Option<neck::Neck>,
tracker: Option<tracker::Tracker>,
detector: Option<detector::Detector>,
tokenizer: Option<tokenizer::Tokenizer>,
}
#[cfg(not(target_arch = "wasm32"))]
/// An image run through the backbone, with the head-specific features computed on demand.
pub struct Encoded {
backbone: Tensor,
tracker: Option<tracker::Features>,
detector: Option<detector::Features>,
}
/// A predicted mask: `MASK_SIDE²` logits over the model's square input (which the image was
/// stretched to), and a confidence 0..1.
#[derive(Clone, Debug)]
pub struct Prediction {
pub logits: Vec<f32>,
pub score: f32,
}
#[cfg(not(target_arch = "wasm32"))]
impl Sam3 {
/// Load from a directory holding `model.safetensors` (the `facebook/sam3` checkpoint in
/// Hugging Face format), `vocab.json` and `merges.txt`.
pub fn load(dir: &Path) -> Result<Self> {
Self::load_on(dir, best_device())
}
pub fn load_on(dir: &Path, device: Device) -> Result<Self> {
if !is_model_dir(dir) {
return Err(Error::Missing(dir.to_path_buf()));
}
let weights = weights::Weights::open(&dir.join(WEIGHTS_FILE))?;
let backbone = vit::Vit::new(Self::vb(&weights, &device).pp("detector_model.vision_encoder.backbone"))?;
Ok(Self { dir: dir.to_path_buf(), device, weights, backbone, tracker_neck: None, tracker: None, detector: None, tokenizer: None })
}
fn vb<'a>(w: &Arc<weights::Weights>, device: &Device) -> VarBuilder<'a> {
VarBuilder::from_backend(Box::new(weights::Backend(w.clone())), DType::F32, device.clone())
}
pub fn device(&self) -> &Device {
&self.device
}
/// Run the backbone on an RGB image (8-bit, row-major, `w × h`), stretched to the square
/// input like the reference preprocessing.
pub fn encode(&self, rgb: &[u8], w: usize, h: usize) -> Result<Encoded> {
let pixels = mask::preprocess(rgb, w, h, INPUT)?;
let pixels = Tensor::from_vec(pixels, (1, 3, INPUT, INPUT), &self.device)?;
self.encode_pixels(&pixels)
}
/// Run the backbone on preprocessed pixels `[1, 3, 1008, 1008]` (−1..1).
pub fn encode_pixels(&self, pixels: &Tensor) -> Result<Encoded> {
Ok(Encoded { backbone: self.backbone.forward(&pixels.to_device(&self.device)?)?, tracker: None, detector: None })
}
fn tracker_parts(&mut self) -> Result<(&neck::Neck, &tracker::Tracker)> {
if self.tracker_neck.is_none() {
self.tracker_neck = Some(neck::Neck::new(Self::vb(&self.weights, &self.device).pp("tracker_neck"))?);
}
if self.tracker.is_none() {
self.tracker = Some(tracker::Tracker::new(Self::vb(&self.weights, &self.device).pp("tracker_model"))?);
}
match (&self.tracker_neck, &self.tracker) {
(Some(n), Some(t)) => Ok((n, t)),
_ => Err(Error::Model("tracker failed to load".into())),
}
}
/// Compute the click decoder's features for `enc` now (so the first click is fast).
pub fn prepare_clicks(&mut self, enc: &mut Encoded) -> Result<()> {
let (neck, tracker) = self.tracker_parts()?;
if enc.tracker.is_none() {
let levels = neck.forward(&enc.backbone, 3)?;
enc.tracker = Some(tracker.features(&levels)?);
}
Ok(())
}
/// Segment one object from clicks (positions normalized to the image, 0..1).
pub fn segment_clicks(&mut self, enc: &mut Encoded, clicks: &[Click]) -> Result<Prediction> {
if clicks.is_empty() {
return Err(Error::Model("no clicks".into()));
}
let (neck, tracker) = self.tracker_parts()?;
if enc.tracker.is_none() {
let levels = neck.forward(&enc.backbone, 3)?;
enc.tracker = Some(tracker.features(&levels)?);
}
let f = enc.tracker.as_ref().ok_or_else(|| Error::Model("tracker features".into()))?;
// one click: SAM's three alternatives, best by predicted IoU; more: the single mask
let p = tracker.predict(f, clicks, clicks.len() == 1)?;
Ok(Prediction { logits: p.logits, score: p.iou * sigmoid(p.object_score) })
}
fn detector_parts(&mut self) -> Result<(&detector::Detector, &tokenizer::Tokenizer)> {
if self.detector.is_none() {
self.detector = Some(detector::Detector::new(Self::vb(&self.weights, &self.device).pp("detector_model"))?);
}
if self.tokenizer.is_none() {
self.tokenizer = Some(tokenizer::Tokenizer::load(&self.dir)?);
}
match (&self.detector, &self.tokenizer) {
(Some(d), Some(t)) => Ok((d, t)),
_ => Err(Error::Model("detector failed to load".into())),
}
}
/// Segment every instance of the concept `text` names. Instances scoring above
/// `threshold` (0..1, 0.5 is the reference default) are merged into one mask; `None` when
/// nothing matches.
pub fn segment_text(&mut self, enc: &mut Encoded, text: &str, threshold: f32) -> Result<Option<Prediction>> {
let (det, tok) = self.detector_parts()?;
if enc.detector.is_none() {
enc.detector = Some(det.features(&enc.backbone)?);
}
let f = enc.detector.as_ref().ok_or_else(|| Error::Model("detector features".into()))?;
let (ids, valid) = tok.encode(text);
let out = det.predict(f, &ids, &valid)?;
Ok(detector::merge(&out, threshold))
}
/// Raw detector outputs for `text` (tests and tools).
pub fn detect(&mut self, enc: &mut Encoded, text: &str) -> Result<detector::Output> {
let (det, tok) = self.detector_parts()?;
if enc.detector.is_none() {
enc.detector = Some(det.features(&enc.backbone)?);
}
let f = enc.detector.as_ref().ok_or_else(|| Error::Model("detector features".into()))?;
let (ids, valid) = tok.encode(text);
det.predict(f, &ids, &valid)
}
}
#[cfg(not(target_arch = "wasm32"))]
impl Encoded {
/// The backbone output `[1, 1024, 72, 72]` (tests).
pub fn backbone(&self) -> &Tensor {
&self.backbone
}
}
#[cfg(not(target_arch = "wasm32"))]
fn sigmoid(x: f32) -> f32 {
1.0 / (1.0 + (-x).exp())
}
#[cfg(not(target_arch = "wasm32"))]
pub use detector::Output as DetectorOutput;
#[cfg(not(target_arch = "wasm32"))]
/// Wait for queued GPU work to finish (benchmarks).
pub fn sync(model: &Sam3) -> Result<()> {
Ok(model.device.synchronize()?)
}
+130
View File
@@ -0,0 +1,130 @@
//! Image preprocessing and mask post-processing (CPU, no model needed).
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3/image_processing_sam3*.py`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use crate::{Error, Result};
/// Separable resampling weights for `src → dst` samples with an antialiased triangle filter
/// (Pillow's BILINEAR: the support widens with the downscale factor).
fn weights(src: usize, dst: usize) -> Vec<(usize, Vec<f32>)> {
let scale = src as f64 / dst as f64;
let ss = scale.max(1.0);
let support = ss;
(0..dst)
.map(|o| {
let center = (o as f64 + 0.5) * scale;
let lo = ((center - support + 0.5).floor().max(0.0)) as usize;
let hi = ((center + support + 0.5).floor() as usize).min(src);
let mut w: Vec<f32> = (lo..hi).map(|i| (1.0 - ((i as f64 - center + 0.5) / ss).abs()).max(0.0) as f32).collect();
let sum: f32 = w.iter().sum();
if sum > 0.0 {
w.iter_mut().for_each(|v| *v /= sum);
}
(lo, w)
})
.collect()
}
/// An 8-bit RGB image (`w × h`, row-major) stretched to `side × side`, scaled to −1..1 and laid
/// out channel-first (`[3, side, side]`), as SAM 3's processor does.
pub fn preprocess(rgb: &[u8], w: usize, h: usize, side: usize) -> Result<Vec<f32>> {
let n = w.checked_mul(h).and_then(|n| n.checked_mul(3)).ok_or_else(|| Error::Model("image too large".into()))?;
if w == 0 || h == 0 || rgb.len() < n {
return Err(Error::Model(format!("image buffer of {} bytes for {w} × {h}", rgb.len())));
}
let (wx, wy) = (weights(w, side), weights(h, side));
// horizontal pass: h rows × side columns × 3
let mut tmp = vec![0f32; h * side * 3];
for y in 0..h {
for (ox, (lo, ws)) in wx.iter().enumerate() {
let mut acc = [0f32; 3];
for (k, wt) in ws.iter().enumerate() {
let i = (y * w + lo + k) * 3;
for (c, a) in acc.iter_mut().enumerate() {
*a += wt * f32::from(rgb.get(i + c).copied().unwrap_or(0));
}
}
if let Some(dst) = tmp.get_mut((y * side + ox) * 3..(y * side + ox) * 3 + 3) {
dst.copy_from_slice(&acc);
}
}
}
let mut out = vec![0f32; 3 * side * side];
for (oy, (lo, ws)) in wy.iter().enumerate() {
for ox in 0..side {
let mut acc = [0f32; 3];
for (k, wt) in ws.iter().enumerate() {
let i = ((lo + k) * side + ox) * 3;
for (c, a) in acc.iter_mut().enumerate() {
*a += wt * tmp.get(i + c).copied().unwrap_or(0.0);
}
}
for (c, a) in acc.iter().enumerate() {
// Pillow rounds to 8 bits; then /255 and (x − 0.5) / 0.5
let v = a.round().clamp(0.0, 255.0) / 255.0;
if let Some(d) = out.get_mut((c * side + oy) * side + ox) {
*d = (v - 0.5) / 0.5;
}
}
}
}
Ok(out)
}
/// Mask `logits` (`side × side`) resampled bilinearly (half-pixel centres) to `w × h` and
/// mapped to 0..1 coverage (sigmoid).
pub fn alpha(logits: &[f32], side: usize, w: usize, h: usize) -> Vec<f32> {
let mut out = vec![0f32; w * h];
if side == 0 || logits.len() < side * side {
return out;
}
let sample = |x: usize, y: usize| logits.get(y.min(side - 1) * side + x.min(side - 1)).copied().unwrap_or(-20.0);
let (sx, sy) = (side as f32 / w.max(1) as f32, side as f32 / h.max(1) as f32);
for y in 0..h {
let fy = ((y as f32 + 0.5) * sy - 0.5).max(0.0);
let (y0, ty) = (fy.floor() as usize, fy.fract());
for x in 0..w {
let fx = ((x as f32 + 0.5) * sx - 0.5).max(0.0);
let (x0, tx) = (fx.floor() as usize, fx.fract());
let top = sample(x0, y0) * (1.0 - tx) + sample(x0 + 1, y0) * tx;
let bot = sample(x0, y0 + 1) * (1.0 - tx) + sample(x0 + 1, y0 + 1) * tx;
let l = top * (1.0 - ty) + bot * ty;
if let Some(o) = out.get_mut(y * w + x) {
*o = 1.0 / (1.0 + (-l).exp());
}
}
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn preprocess_is_bounded_and_channel_first() {
let rgb: Vec<u8> = (0..20 * 10 * 3).map(|i| (i % 256) as u8).collect();
let p = preprocess(&rgb, 20, 10, 8).unwrap();
assert_eq!(p.len(), 3 * 64);
assert!(p.iter().all(|v| (-1.0..=1.0).contains(v)));
// a flat image stays flat
let flat = vec![255u8; 30 * 30 * 3];
assert!(preprocess(&flat, 30, 30, 7).unwrap().iter().all(|v| (*v - 1.0).abs() < 1e-6));
}
#[test]
fn short_buffers_are_errors() {
assert!(preprocess(&[0; 5], 2, 2, 4).is_err());
assert!(preprocess(&[], 0, 0, 4).is_err());
}
#[test]
fn alpha_follows_the_logits() {
let logits: Vec<f32> = (0..16).map(|i| if i % 4 < 2 { 10.0 } else { -10.0 }).collect();
let a = alpha(&logits, 4, 8, 2);
assert!(a[0] > 0.99 && a[7] < 0.01);
assert!(alpha(&[], 4, 3, 3).iter().all(|v| *v == 0.0));
}
}
+77
View File
@@ -0,0 +1,77 @@
//! The feature pyramid (`Sam3VisionNeck`): the backbone's 72 × 72 map at four scales
//! (288, 144, 72 and 36 per side), each projected to 256 channels.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3/modeling_sam3.py`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use candle_core::{Module, Tensor};
use candle_nn::{Conv2d, Conv2dConfig, ConvTranspose2d, ConvTranspose2dConfig, VarBuilder};
use crate::Result;
use crate::vit::HIDDEN;
pub const FPN: usize = 256;
enum Scale {
/// Two 2× transposed convolutions with GELU between (4×).
Up4(ConvTranspose2d, ConvTranspose2d),
Up2(ConvTranspose2d),
Same,
/// 2 × 2 max pooling (½×).
Down2,
}
struct Level {
scale: Scale,
proj1: Conv2d,
proj2: Conv2d,
}
pub struct Neck {
levels: Vec<Level>,
}
impl Neck {
/// `vb` at `…neck` (detector) or `tracker_neck`.
pub fn new(vb: VarBuilder) -> Result<Self> {
let up = ConvTranspose2dConfig { stride: 2, ..Default::default() };
let mut levels = Vec::new();
for (i, factor) in [4.0f32, 2.0, 1.0, 0.5].into_iter().enumerate() {
let l = vb.pp(format!("fpn_layers.{i}"));
let s = l.pp("scale_layers");
let (scale, inter) = if factor == 4.0 {
let a = candle_nn::conv_transpose2d(HIDDEN, HIDDEN / 2, 2, up, s.pp("0"))?;
let b = candle_nn::conv_transpose2d(HIDDEN / 2, HIDDEN / 4, 2, up, s.pp("2"))?;
(Scale::Up4(a, b), HIDDEN / 4)
} else if factor == 2.0 {
(Scale::Up2(candle_nn::conv_transpose2d(HIDDEN, HIDDEN / 2, 2, up, s.pp("0"))?), HIDDEN / 2)
} else if factor == 1.0 {
(Scale::Same, HIDDEN)
} else {
(Scale::Down2, HIDDEN)
};
levels.push(Level {
scale,
proj1: candle_nn::conv2d(inter, FPN, 1, Conv2dConfig::default(), l.pp("proj1"))?,
proj2: candle_nn::conv2d(FPN, FPN, 3, Conv2dConfig { padding: 1, ..Default::default() }, l.pp("proj2"))?,
});
}
Ok(Self { levels })
}
/// `x`: the backbone's `[1, 1024, 72, 72]`; `n`: how many levels (from the finest) to compute.
pub fn forward(&self, x: &Tensor, n: usize) -> Result<Vec<Tensor>> {
let mut out = Vec::with_capacity(n);
for l in self.levels.iter().take(n) {
let y = match &l.scale {
Scale::Up4(a, b) => b.forward(&a.forward(x)?.gelu_erf()?)?,
Scale::Up2(a) => a.forward(x)?,
Scale::Same => x.clone(),
Scale::Down2 => x.max_pool2d(2)?,
};
out.push(l.proj2.forward(&l.proj1.forward(&y)?)?);
}
Ok(out)
}
}
+242
View File
@@ -0,0 +1,242 @@
//! Shared building blocks: attention, MLPs, layer norms, sine position encodings.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3/modeling_sam3.py`, `models/sam3_tracker`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use candle_core::{D, DType, Device, Module, Tensor};
use candle_nn::{LayerNorm, Linear, VarBuilder};
use crate::Result;
/// `nn.LayerNorm(d)` (PyTorch's default eps).
pub const TORCH_LN_EPS: f64 = 1e-5;
pub fn ln(d: usize, eps: f64, vb: VarBuilder) -> Result<LayerNorm> {
Ok(candle_nn::layer_norm(d, eps, vb)?)
}
pub fn linear(i: usize, o: usize, vb: VarBuilder) -> Result<Linear> {
Ok(candle_nn::linear(i, o, vb)?)
}
/// Largest attention-score block computed at once (elements): the 5184-token global layers
/// would otherwise need ~1.7 GB for all 16 heads.
const SCORE_BLOCK: usize = 64 << 20;
/// Scaled dot-product attention on `[B, H, L, d]` tensors, with an optional additive bias
/// broadcastable to `[B, H, Lq, Lk]`. Heads are processed in groups to bound memory.
pub fn sdpa(q: &Tensor, k: &Tensor, v: &Tensor, bias: Option<&Tensor>, scale: f64) -> Result<Tensor> {
let (b, h, lq, _) = q.dims4()?;
let lk = k.dim(2)?;
let per_head = b.saturating_mul(lq).saturating_mul(lk).max(1);
let group = (SCORE_BLOCK / per_head).clamp(1, h);
let bias_heads = match bias {
Some(t) => t.dim(1)?,
None => 1,
};
let mut outs = Vec::with_capacity(h.div_ceil(group));
let mut start = 0;
while start < h {
let n = group.min(h - start);
let qs = q.narrow(1, start, n)?;
let ks = k.narrow(1, start, n)?;
let vs = v.narrow(1, start, n)?;
let mut att = (qs.matmul(&ks.t()?)? * scale)?;
if let Some(bias) = bias {
let bs = if bias_heads > 1 { bias.narrow(1, start, n)? } else { bias.clone() };
att = att.broadcast_add(&bs)?;
}
let att = candle_nn::ops::softmax_last_dim(&att)?;
outs.push(att.matmul(&vs)?);
start += n;
}
Ok(Tensor::cat(&outs, 1)?)
}
/// Multi-head attention with q/k/v/o projections (`Sam3Attention`, `Sam3TrackerAttention`
/// with `internal` = hidden / downsample rate).
pub struct Attention {
q: Linear,
k: Linear,
v: Linear,
o: Linear,
heads: usize,
head_dim: usize,
}
impl Attention {
pub fn new(hidden: usize, internal: usize, heads: usize, vb: VarBuilder) -> Result<Self> {
Self::named(hidden, internal, heads, "o_proj", vb)
}
/// With the output projection called `out` (CLIP's `out_proj`).
pub fn named(hidden: usize, internal: usize, heads: usize, out: &str, vb: VarBuilder) -> Result<Self> {
Ok(Self {
q: linear(hidden, internal, vb.pp("q_proj"))?,
k: linear(hidden, internal, vb.pp("k_proj"))?,
v: linear(hidden, internal, vb.pp("v_proj"))?,
o: linear(internal, hidden, vb.pp(out))?,
heads,
head_dim: internal / heads,
})
}
fn split(&self, x: &Tensor) -> Result<Tensor> {
let (b, l, _) = x.dims3()?;
Ok(x.reshape((b, l, self.heads, self.head_dim))?.transpose(1, 2)?.contiguous()?)
}
/// The q, k and v projections of `x` (for attention variants that transform them).
pub fn qkv(&self, x: &Tensor) -> Result<(Tensor, Tensor, Tensor)> {
Ok((self.q.forward(x)?, self.k.forward(x)?, self.v.forward(x)?))
}
/// The output projection.
pub fn out(&self, x: &Tensor) -> Result<Tensor> {
Ok(self.o.forward(x)?)
}
/// `q`, `k`, `v`: `[B, L, hidden]`; `bias` broadcastable to `[B, heads, Lq, Lk]`.
pub fn forward(&self, q: &Tensor, k: &Tensor, v: &Tensor, bias: Option<&Tensor>) -> Result<Tensor> {
let (b, lq, _) = q.dims3()?;
let q = self.split(&self.q.forward(q)?)?;
let k = self.split(&self.k.forward(k)?)?;
let v = self.split(&self.v.forward(v)?)?;
let o = sdpa(&q, &k, &v, bias, (self.head_dim as f64).powf(-0.5))?;
let o = o.transpose(1, 2)?.reshape((b, lq, self.heads * self.head_dim))?;
Ok(self.o.forward(&o)?)
}
}
/// An additive key-padding bias `[1, 1, 1, L]` (0 for valid tokens, the f32 minimum for padding).
pub fn padding_bias(valid: &[bool], device: &Device) -> Result<Tensor> {
let v: Vec<f32> = valid.iter().map(|&ok| if ok { 0.0 } else { f32::MIN }).collect();
Ok(Tensor::from_vec(v, (1, 1, 1, valid.len()), device)?)
}
/// `Sam3MLP`: fc1 → activation → fc2.
pub struct Mlp {
fc1: Linear,
fc2: Linear,
gelu: bool,
}
impl Mlp {
pub fn new(hidden: usize, inter: usize, gelu: bool, vb: VarBuilder) -> Result<Self> {
Ok(Self { fc1: linear(hidden, inter, vb.pp("fc1"))?, fc2: linear(inter, hidden, vb.pp("fc2"))?, gelu })
}
pub fn forward(&self, x: &Tensor) -> Result<Tensor> {
let x = self.fc1.forward(x)?;
let x = if self.gelu { x.gelu_erf()? } else { x.relu()? };
Ok(self.fc2.forward(&x)?)
}
}
/// `Sam3DecoderMLP`: `layer1 … layerN` with ReLU between (2 or 3 layers).
pub struct DecoderMlp {
layers: Vec<Linear>,
}
impl DecoderMlp {
pub fn new(input: usize, hidden: usize, output: usize, n: usize, vb: VarBuilder) -> Result<Self> {
let mut layers = Vec::with_capacity(n);
for i in 0..n {
let (a, b) = (if i == 0 { input } else { hidden }, if i + 1 == n { output } else { hidden });
layers.push(linear(a, b, vb.pp(format!("layer{}", i + 1)))?);
}
Ok(Self { layers })
}
pub fn forward(&self, x: &Tensor) -> Result<Tensor> {
let mut x = x.clone();
for (i, l) in self.layers.iter().enumerate() {
x = l.forward(&x)?;
if i + 1 < self.layers.len() {
x = x.relu()?;
}
}
Ok(x)
}
}
/// `Sam3TrackerFeedForward`: proj_in → act → (layers.i → act)* → proj_out (→ sigmoid).
pub struct FeedForward {
proj_in: Linear,
layers: Vec<Linear>,
proj_out: Linear,
sigmoid: bool,
}
impl FeedForward {
pub fn new(input: usize, hidden: usize, output: usize, n: usize, sigmoid: bool, vb: VarBuilder) -> Result<Self> {
let mut layers = Vec::new();
for i in 0..n.saturating_sub(2) {
layers.push(linear(hidden, hidden, vb.pp(format!("layers.{i}")))?);
}
Ok(Self { proj_in: linear(input, hidden, vb.pp("proj_in"))?, layers, proj_out: linear(hidden, output, vb.pp("proj_out"))?, sigmoid })
}
pub fn forward(&self, x: &Tensor) -> Result<Tensor> {
let mut x = self.proj_in.forward(x)?.relu()?;
for l in &self.layers {
x = l.forward(&x)?.relu()?;
}
let x = self.proj_out.forward(&x)?;
Ok(if self.sigmoid { candle_nn::ops::sigmoid(&x)? } else { x })
}
}
/// Layer norm over the channel axis of an NCHW tensor (`Sam3TrackerLayerNorm`, channels_first).
pub fn ln_channels(x: &Tensor, norm: &LayerNorm) -> Result<Tensor> {
let x = x.permute((0, 2, 3, 1))?;
Ok(norm.forward(&x)?.permute((0, 3, 1, 2))?.contiguous()?)
}
/// The DETR sine embedding of an `h × w` grid, normalized (`Sam3SinePositionEmbedding` with
/// `normalize=True`, scale 2π): `[1, 2·feats, h, w]`, y features first.
pub fn sine_grid(h: usize, w: usize, feats: usize, device: &Device) -> Result<Tensor> {
let scale = 2.0 * std::f64::consts::PI;
let dim_t: Vec<f64> = (0..feats).map(|i| 10000f64.powf(2.0 * (i / 2) as f64 / feats as f64)).collect();
let mut out = vec![0f32; 2 * feats * h * w];
for y in 0..h {
let ye = (y + 1) as f64 / (h as f64 + 1e-6) * scale;
for x in 0..w {
let xe = (x + 1) as f64 / (w as f64 + 1e-6) * scale;
for (i, d) in dim_t.iter().enumerate() {
let (py, px) = (ye / d, xe / d);
let (vy, vx) = if i % 2 == 0 { (py.sin(), px.sin()) } else { (py.cos(), px.cos()) };
if let Some(v) = out.get_mut((i * h + y) * w + x) {
*v = vy as f32;
}
if let Some(v) = out.get_mut(((feats + i) * h + y) * w + x) {
*v = vx as f32;
}
}
}
}
Ok(Tensor::from_vec(out, (1, 2 * feats, h, w), device)?)
}
/// `inverse_sigmoid` with the reference's clamping (eps 1e-3).
pub fn inverse_sigmoid(x: &Tensor) -> Result<Tensor> {
let x = x.clamp(0f32, 1f32)?;
let x1 = x.clamp(1e-3f32, f32::MAX)?;
let x2 = (1.0 - &x)?.clamp(1e-3f32, f32::MAX)?;
Ok((x1 / x2)?.log()?)
}
/// `[..., 4]` boxes (cx, cy, w, h) → (x0, y0, x1, y1).
pub fn cxcywh_to_xyxy(b: &Tensor) -> Result<Tensor> {
let cx = b.narrow(D::Minus1, 0, 1)?;
let cy = b.narrow(D::Minus1, 1, 1)?;
let w = (b.narrow(D::Minus1, 2, 1)? * 0.5)?;
let h = (b.narrow(D::Minus1, 3, 1)? * 0.5)?;
Ok(Tensor::cat(&[(&cx - &w)?, (&cy - &h)?, (&cx + &w)?, (&cy + &h)?], D::Minus1)?)
}
/// A tensor's values as `f32` on the CPU.
pub fn to_vec(t: &Tensor) -> Result<Vec<f32>> {
Ok(t.to_dtype(DType::F32)?.flatten_all()?.to_vec1::<f32>()?)
}
+177
View File
@@ -0,0 +1,177 @@
//! The CLIP byte-pair-encoding tokenizer (lower-cased, `</w>` word ends), padded to the
//! 32-token context SAM 3's text encoder takes.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (the CLIP tokenizer used by `models/sam3`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use std::collections::HashMap;
use std::path::Path;
use crate::{Error, Result};
pub const CONTEXT: usize = 32;
const BOS: u32 = 49406;
const EOS: u32 = 49407;
pub struct Tokenizer {
vocab: HashMap<String, u32>,
ranks: HashMap<(String, String), usize>,
byte_char: [char; 256],
}
/// GPT-2's reversible byte → printable character map.
fn bytes_to_unicode() -> [char; 256] {
let mut map = ['\0'; 256];
let mut n = 0u32;
for b in 0..=255u8 {
let printable = (b'!'..=b'~').contains(&b) || (0xA1..=0xAC).contains(&b) || (0xAE..=0xFF).contains(&b);
let c = if printable {
u32::from(b)
} else {
n += 1;
255 + n
};
if let Some(slot) = map.get_mut(usize::from(b)) {
*slot = char::from_u32(c).unwrap_or('?');
}
}
map
}
impl Tokenizer {
/// From `vocab.json` and `merges.txt` in `dir`.
pub fn load(dir: &Path) -> Result<Self> {
let read = |name: &str| std::fs::read_to_string(dir.join(name)).map_err(|e| Error::Model(format!("{name}: {e}")));
let vocab: HashMap<String, u32> = serde_json::from_str(&read("vocab.json")?).map_err(|e| Error::Model(format!("vocab.json: {e}")))?;
let merges = read("merges.txt")?;
Ok(Self::new(vocab, &merges))
}
pub fn new(vocab: HashMap<String, u32>, merges: &str) -> Self {
let ranks = merges
.lines()
.filter(|l| !l.starts_with("#version") && !l.trim().is_empty())
.filter_map(|l| l.split_once(' ').map(|(a, b)| (a.to_string(), b.to_string())))
.enumerate()
.map(|(i, p)| (p, i))
.collect();
Self { vocab, ranks, byte_char: bytes_to_unicode() }
}
/// Token ids (start, words, end, padding) and which positions are real tokens.
pub fn encode(&self, text: &str) -> (Vec<u32>, Vec<bool>) {
let mut ids = vec![BOS];
for word in pre_tokenize(&text.split_whitespace().collect::<Vec<_>>().join(" ").to_lowercase()) {
let mapped: String = word.bytes().map(|b| self.byte_char.get(usize::from(b)).copied().unwrap_or('?')).collect();
for piece in self.bpe(&mapped) {
ids.push(self.vocab.get(&piece).copied().unwrap_or(EOS));
}
}
ids.truncate(CONTEXT - 1);
ids.push(EOS);
let n = ids.len();
ids.resize(CONTEXT, EOS);
let valid = (0..CONTEXT).map(|i| i < n).collect();
(ids, valid)
}
fn bpe(&self, word: &str) -> Vec<String> {
let mut syms: Vec<String> = word.chars().map(String::from).collect();
if let Some(last) = syms.last_mut() {
last.push_str("</w>");
}
loop {
let best = syms
.windows(2)
.enumerate()
.filter_map(|(i, w)| match w {
[a, b] => self.ranks.get(&(a.clone(), b.clone())).map(|r| (*r, i)),
_ => None,
})
.min();
let Some((_, i)) = best else { break };
let (Some(a), Some(b)) = (syms.get(i).cloned(), syms.get(i + 1).cloned()) else { break };
let merged = a.clone() + &b;
// merge every occurrence of the pair, left to right
let mut out = Vec::with_capacity(syms.len());
let mut j = 0;
while j < syms.len() {
if j + 1 < syms.len() && syms.get(j) == Some(&a) && syms.get(j + 1) == Some(&b) {
out.push(merged.clone());
j += 2;
} else {
out.extend(syms.get(j).cloned());
j += 1;
}
}
syms = out;
if syms.len() == 1 {
break;
}
}
syms
}
}
/// CLIP's pre-tokenizer: contractions, letter runs, single digits, and runs of other
/// non-space characters.
fn pre_tokenize(text: &str) -> Vec<String> {
let chars: Vec<char> = text.chars().collect();
let mut out = Vec::new();
let mut i = 0;
while let Some(&c) = chars.get(i) {
if c.is_whitespace() {
i += 1;
continue;
}
if c == '\'' {
let rest: String = chars.get(i + 1..(i + 3).min(chars.len())).map(|s| s.iter().collect()).unwrap_or_default();
if let Some(k) = ["re", "ve", "ll"]
.iter()
.find(|k| rest.starts_with(**k))
.map(|k| k.len())
.or_else(|| rest.chars().next().filter(|c| matches!(c, 's' | 't' | 'm' | 'd')).map(|_| 1))
{
out.push(chars.get(i..i + 1 + k).map(|s| s.iter().collect()).unwrap_or_default());
i += 1 + k;
continue;
}
}
let start = i;
if c.is_alphabetic() {
while chars.get(i).is_some_and(|c| c.is_alphabetic()) {
i += 1;
}
} else if c.is_numeric() {
i += 1;
} else {
while chars.get(i).is_some_and(|c| !c.is_whitespace() && !c.is_alphabetic() && !c.is_numeric()) {
i += 1;
}
}
out.push(chars.get(start..i).map(|s| s.iter().collect()).unwrap_or_default());
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn splits_like_clip() {
assert_eq!(pre_tokenize("a dog's toy, 25 cats!"), ["a", "dog", "'s", "toy", ",", "2", "5", "cats", "!"]);
}
#[test]
fn merges_by_rank() {
let vocab: HashMap<String, u32> =
[("c", 1), ("a", 2), ("t</w>", 3), ("ca", 4), ("cat</w>", 5)].into_iter().map(|(k, v)| (k.to_string(), v)).collect();
let t = Tokenizer::new(vocab, "#version: 0.2\nc a\nca t</w>\n");
let (ids, valid) = t.encode(" CAT ");
assert_eq!(&ids[..3], &[BOS, 5, EOS]);
assert_eq!(ids.len(), CONTEXT);
assert_eq!(valid.iter().filter(|v| **v).count(), 3);
}
}
+253
View File
@@ -0,0 +1,253 @@
//! Point prompts (SAM 3's interactive "tracker" head, SAM 2 style): a prompt encoder for
//! positive/negative clicks and a two-way-transformer mask decoder over the image features.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3_tracker/modeling_sam3_tracker.py`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use candle_core::{DType, Device, IndexOp, Module, Tensor};
use candle_nn::{Conv2d, Conv2dConfig, ConvTranspose2d, ConvTranspose2dConfig, LayerNorm, VarBuilder};
use crate::nn::{Attention, FeedForward, TORCH_LN_EPS, ln, ln_channels};
use crate::vit::{GRID, IMAGE};
use crate::{Result, nn::to_vec};
const HIDDEN: usize = 256;
const MASK_TOKENS: usize = 4;
/// Low-resolution mask side (4 × the 72 grid).
pub const LOW_RES: usize = 288;
/// The image features the decoder needs, computed once per image.
pub struct Features {
/// Level 2 (72 × 72, 256 channels).
pub image: Tensor,
/// `conv_s0`(level 0): `[1, 32, 288, 288]`.
pub s0: Tensor,
/// `conv_s1`(level 1): `[1, 64, 144, 144]`.
pub s1: Tensor,
}
/// One click: position normalized to the image (0..1), and whether it includes or excludes.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Click {
pub x: f32,
pub y: f32,
pub positive: bool,
}
/// A predicted mask: 288 × 288 logits over the (squared) model input, and its quality.
pub struct Prediction {
pub logits: Vec<f32>,
pub iou: f32,
pub object_score: f32,
}
/// Random-Fourier positional encoding of points normalized to 0..1 (`Sam3TrackerPositionalEmbedding`).
struct Fourier {
pe: Vec<f32>, // [2, 128]
}
impl Fourier {
fn encode(&self, x: f32, y: f32) -> Vec<f32> {
let half = self.pe.len() / 2;
let (cx, cy) = (2.0 * x - 1.0, 2.0 * y - 1.0);
let proj: Vec<f32> = (0..half)
.map(|i| {
let a = cx * self.pe.get(i).copied().unwrap_or(0.0) + cy * self.pe.get(half + i).copied().unwrap_or(0.0);
2.0 * std::f32::consts::PI * a
})
.collect();
proj.iter().map(|a| a.sin()).chain(proj.iter().map(|a| a.cos())).collect()
}
}
struct TwoWayBlock {
self_attn: Attention,
ln1: LayerNorm,
t2i: Attention,
ln2: LayerNorm,
mlp: FeedForward,
ln3: LayerNorm,
ln4: LayerNorm,
i2t: Attention,
skip_first_pe: bool,
}
/// `Sam3TrackerAttention` on `[1, L, C]` (the point batch of one is folded away).
fn attn(a: &Attention, q: &Tensor, k: &Tensor, v: &Tensor) -> Result<Tensor> {
a.forward(q, k, v, None)
}
impl TwoWayBlock {
fn forward(&self, queries: &Tensor, keys: &Tensor, qpe: &Tensor, kpe: &Tensor) -> Result<(Tensor, Tensor)> {
let queries = if self.skip_first_pe {
attn(&self.self_attn, queries, queries, queries)?
} else {
let q = (queries + qpe)?;
(queries + attn(&self.self_attn, &q, &q, queries)?)?
};
let queries = self.ln1.forward(&queries)?;
let (q, k) = ((&queries + qpe)?, (keys + kpe)?);
let queries = self.ln2.forward(&(&queries + attn(&self.t2i, &q, &k, keys)?)?)?;
let queries = self.ln3.forward(&(&queries + self.mlp.forward(&queries)?)?)?;
let (q, k) = ((&queries + qpe)?, (keys + kpe)?);
let keys = self.ln4.forward(&(keys + attn(&self.i2t, &k, &q, &queries)?)?)?;
Ok((queries, keys))
}
}
pub struct Tracker {
device: Device,
point_pe: Fourier,
image_pe: Tensor,
point_embed: Tensor,
not_a_point: Tensor,
no_mask: Tensor,
conv_s0: Conv2d,
conv_s1: Conv2d,
output_tokens: Tensor,
blocks: Vec<TwoWayBlock>,
final_t2i: Attention,
ln_final: LayerNorm,
up1: ConvTranspose2d,
up_ln: LayerNorm,
up2: ConvTranspose2d,
hyper: Vec<FeedForward>,
iou_head: FeedForward,
obj_head: FeedForward,
}
fn read_vec(vb: &VarBuilder, shape: (usize, usize), name: &str) -> Result<Vec<f32>> {
to_vec(&vb.get(shape, name)?)
}
impl Tracker {
/// `vb` at `tracker_model`.
pub fn new(vb: VarBuilder) -> Result<Self> {
let device = vb.device().clone();
let pe = vb.pp("prompt_encoder");
let md = vb.pp("mask_decoder");
let image_fourier = Fourier { pe: read_vec(&vb, (2, HIDDEN / 2), "shared_image_embedding.positional_embedding")? };
// image-wide positional embedding: pixel centres of the 72 × 72 grid
let mut grid = Vec::with_capacity(GRID * GRID * HIDDEN);
for r in 0..GRID {
for c in 0..GRID {
grid.extend(image_fourier.encode((c as f32 + 0.5) / GRID as f32, (r as f32 + 0.5) / GRID as f32));
}
}
let image_pe = Tensor::from_vec(grid, (1, GRID * GRID, HIDDEN), &device)?;
let t = md.pp("transformer");
let mut blocks = Vec::new();
for i in 0..2 {
let b = t.pp(format!("layers.{i}"));
blocks.push(TwoWayBlock {
self_attn: Attention::new(HIDDEN, HIDDEN, 8, b.pp("self_attn"))?,
ln1: ln(HIDDEN, TORCH_LN_EPS, b.pp("layer_norm1"))?,
t2i: Attention::new(HIDDEN, HIDDEN / 2, 8, b.pp("cross_attn_token_to_image"))?,
ln2: ln(HIDDEN, TORCH_LN_EPS, b.pp("layer_norm2"))?,
mlp: FeedForward::new(HIDDEN, 2048, HIDDEN, 2, false, b.pp("mlp"))?,
ln3: ln(HIDDEN, TORCH_LN_EPS, b.pp("layer_norm3"))?,
ln4: ln(HIDDEN, TORCH_LN_EPS, b.pp("layer_norm4"))?,
i2t: Attention::new(HIDDEN, HIDDEN / 2, 8, b.pp("cross_attn_image_to_token"))?,
skip_first_pe: i == 0,
});
}
let output_tokens = Tensor::cat(
&[
md.get((1, HIDDEN), "obj_score_token.weight")?,
md.get((1, HIDDEN), "iou_token.weight")?,
md.get((MASK_TOKENS, HIDDEN), "mask_tokens.weight")?,
],
0,
)?;
let up = ConvTranspose2dConfig { stride: 2, ..Default::default() };
let one = Conv2dConfig::default();
Ok(Self {
point_pe: Fourier { pe: read_vec(&pe, (2, HIDDEN / 2), "shared_embedding.positional_embedding")? },
image_pe,
point_embed: pe.get((4, HIDDEN), "point_embed.weight")?.to_device(&Device::Cpu)?,
not_a_point: pe.get((1, HIDDEN), "not_a_point_embed.weight")?.to_device(&Device::Cpu)?,
no_mask: pe.get((1, HIDDEN), "no_mask_embed.weight")?.reshape((1, HIDDEN, 1, 1))?,
conv_s0: candle_nn::conv2d(HIDDEN, HIDDEN / 8, 1, one, md.pp("conv_s0"))?,
conv_s1: candle_nn::conv2d(HIDDEN, HIDDEN / 4, 1, one, md.pp("conv_s1"))?,
output_tokens,
blocks,
final_t2i: Attention::new(HIDDEN, HIDDEN / 2, 8, t.pp("final_attn_token_to_image"))?,
ln_final: ln(HIDDEN, TORCH_LN_EPS, t.pp("layer_norm_final_attn"))?,
up1: candle_nn::conv_transpose2d(HIDDEN, HIDDEN / 4, 2, up, md.pp("upscale_conv1"))?,
up_ln: ln(HIDDEN / 4, 1e-6, md.pp("upscale_layer_norm"))?,
up2: candle_nn::conv_transpose2d(HIDDEN / 4, HIDDEN / 8, 2, up, md.pp("upscale_conv2"))?,
hyper: (0..MASK_TOKENS)
.map(|i| FeedForward::new(HIDDEN, HIDDEN, HIDDEN / 8, 3, false, md.pp(format!("output_hypernetworks_mlps.{i}"))))
.collect::<Result<_>>()?,
iou_head: FeedForward::new(HIDDEN, HIDDEN, MASK_TOKENS, 3, true, md.pp("iou_prediction_head"))?,
obj_head: FeedForward::new(HIDDEN, HIDDEN, 1, 3, false, md.pp("pred_obj_score_head"))?,
device,
})
}
/// The decoder's image features from the tracker neck's levels (288, 144, 72 per side).
pub fn features(&self, levels: &[Tensor]) -> Result<Features> {
let [l0, l1, l2, ..] = levels else { return Err(crate::Error::Model("tracker needs three feature levels".into())) };
Ok(Features { image: l2.clone(), s0: self.conv_s0.forward(l0)?, s1: self.conv_s1.forward(l1)? })
}
/// Sparse prompt embeddings `[1, n + 1, 256]` (a padding point closes the list).
fn embed_clicks(&self, clicks: &[Click]) -> Result<Tensor> {
let point_embed = to_vec(&self.point_embed)?;
let not_a_point = to_vec(&self.not_a_point)?;
let mut rows = Vec::with_capacity((clicks.len() + 1) * HIDDEN);
for c in clicks {
// the reference shifts pixel coordinates (in the 1008² input) by half a pixel
let (x, y) = ((c.x * IMAGE as f32 + 0.5) / IMAGE as f32, (c.y * IMAGE as f32 + 0.5) / IMAGE as f32);
let e = self.point_pe.encode(x, y);
let label = usize::from(c.positive);
rows.extend(e.iter().enumerate().map(|(i, v)| v + point_embed.get(label * HIDDEN + i).copied().unwrap_or(0.0)));
}
rows.extend_from_slice(&not_a_point);
Ok(Tensor::from_vec(rows, (1, clicks.len() + 1, HIDDEN), &self.device)?)
}
/// Predict masks for `clicks`. `multimask`: the three alternatives (best by predicted IoU is
/// returned), else the single-mask output with SAM 2's stability fallback.
pub fn predict(&self, f: &Features, clicks: &[Click], multimask: bool) -> Result<Prediction> {
let sparse = self.embed_clicks(clicks)?;
let tokens = Tensor::cat(&[self.output_tokens.unsqueeze(0)?, sparse], 1)?; // [1, T, 256]
let image = f.image.broadcast_add(&self.no_mask)?; // dense prompt: "no mask"
let (b, c, h, w) = image.dims4()?;
let keys = image.flatten_from(2)?.transpose(1, 2)?.contiguous()?; // [1, HW, 256]
let (mut q, mut k) = (tokens.clone(), keys);
for blk in &self.blocks {
(q, k) = blk.forward(&q, &k, &tokens, &self.image_pe)?;
}
let qf = (&q + &tokens)?;
let kf = (&k + &self.image_pe)?;
let q = self.ln_final.forward(&(&q + attn(&self.final_t2i, &qf, &kf, &k)?)?)?;
let iou_token = q.i((.., 1, ..))?;
let image = k.transpose(1, 2)?.reshape((b, c, h, w))?;
let up = (self.up1.forward(&image)? + &f.s1)?;
let up = ln_channels(&up, &self.up_ln)?.gelu_erf()?;
let up = (self.up2.forward(&up)? + &f.s0)?.gelu_erf()?; // [1, 32, 288, 288]
let (_, uc, uh, uw) = up.dims4()?;
let hyper = (0..MASK_TOKENS)
.map(|i| self.hyper.get(i).map_or(Err(crate::Error::Model("hypernetwork".into())), |m| m.forward(&q.i((.., 2 + i, ..))?)))
.collect::<Result<Vec<_>>>()?;
let hyper = Tensor::stack(&hyper, 1)?; // [1, 4, 32]
let masks = hyper.matmul(&up.reshape((b, uc, uh * uw))?)?.reshape((b, MASK_TOKENS, uh, uw))?;
let iou = to_vec(&self.iou_head.forward(&iou_token)?)?;
let object_score = to_vec(&self.obj_head.forward(&q.i((.., 0, ..))?)?)?.first().copied().unwrap_or(0.0);
let pick = |i: usize| -> Result<Prediction> {
Ok(Prediction { logits: to_vec(&masks.i((0, i))?)?, iou: iou.get(i).copied().unwrap_or(0.0), object_score })
};
let best_multi = || (1..MASK_TOKENS).max_by(|a, b| iou.get(*a).partial_cmp(&iou.get(*b)).unwrap_or(std::cmp::Ordering::Equal)).unwrap_or(1);
if multimask {
return pick(best_multi());
}
// single-mask output unless it is unstable (area changes > 2 % when the threshold moves ±0.05)
let single = masks.i((0, 0))?;
let inter = single.gt(0.05)?.to_dtype(DType::F32)?.sum_all()?.to_scalar::<f32>()?;
let union = single.gt(-0.05)?.to_dtype(DType::F32)?.sum_all()?.to_scalar::<f32>()?;
let stability = if union > 0.0 { inter / union } else { 1.0 };
if stability >= 0.98 { pick(0) } else { pick(best_multi()) }
}
}
+170
View File
@@ -0,0 +1,170 @@
//! The SAM 3 vision backbone: a 32-layer ViT (Perception Encoder) with 2-D axial rotary
//! position encoding, windowed attention (24 × 24) in most layers and global attention in four.
//!
//! Modified work (Apache License 2.0, §4(b)): ported by the LightCraft contributors in 2026 from
//! the Python/PyTorch SAM 3 code of Hugging Face Transformers (`models/sam3/modeling_sam3.py`), Copyright The HuggingFace
//! Team and Meta Platforms, Inc.; translated to Rust on candle and restructured. See NOTICE.
use candle_core::{D, Device, Module, Tensor};
use candle_nn::{Conv2d, Conv2dConfig, LayerNorm, VarBuilder};
use crate::nn::{Attention, Mlp, ln, sdpa};
use crate::{Error, Result};
pub const IMAGE: usize = 1008;
pub const PATCH: usize = 14;
/// Patches per side (72).
pub const GRID: usize = IMAGE / PATCH;
pub const HIDDEN: usize = 1024;
const HEADS: usize = 16;
const HEAD_DIM: usize = HIDDEN / HEADS;
const LAYERS: usize = 32;
const WINDOW: usize = 24;
const GLOBAL: [usize; 4] = [7, 15, 23, 31];
const PRETRAIN_GRID: usize = 336 / PATCH;
const EPS: f64 = 1e-6;
/// Cosine and sine tables `[N, head_dim]` for a `side × side` grid whose coordinates are scaled
/// by `scale` (`Sam3ViTRotaryEmbedding`: x frequencies then y, each duplicated pairwise).
fn rope_tables(side: usize, scale: f64, device: &Device) -> Result<(Tensor, Tensor)> {
let spatial = HEAD_DIM / 2;
let inv: Vec<f64> = (0..spatial).step_by(2).map(|i| 1.0 / 10000f64.powf(i as f64 / spatial as f64)).collect();
let n = side * side;
let (mut cos, mut sin) = (Vec::with_capacity(n * HEAD_DIM), Vec::with_capacity(n * HEAD_DIM));
for r in 0..side {
for c in 0..side {
let (x, y) = (c as f64 * scale, r as f64 * scale);
for pos in [x, y] {
for f in &inv {
let a = (pos * f) as f32 as f64;
let (s, co) = (a.sin() as f32, a.cos() as f32);
cos.extend([co, co]);
sin.extend([s, s]);
}
}
}
}
Ok((Tensor::from_vec(cos, (n, HEAD_DIM), device)?, Tensor::from_vec(sin, (n, HEAD_DIM), device)?))
}
/// `x·cos + rotate_pairwise(x)·sin`, where rotate_pairwise maps (a, b) pairs to (−b, a).
fn apply_rope(x: &Tensor, cos: &Tensor, sin: &Tensor) -> Result<Tensor> {
let dims = x.dims().to_vec();
let last = *dims.last().ok_or_else(|| Error::Model("rope on a scalar".into()))?;
let mut paired = dims.clone();
paired.pop();
paired.extend([last / 2, 2]);
let p = x.reshape(paired.as_slice())?;
let a = p.narrow(D::Minus1, 0, 1)?;
let b = p.narrow(D::Minus1, 1, 1)?;
let rot = Tensor::cat(&[b.neg()?, a], D::Minus1)?.reshape(dims.as_slice())?;
Ok((x.broadcast_mul(cos)? + rot.broadcast_mul(sin)?)?)
}
struct Layer {
ln1: LayerNorm,
attn: Attention,
ln2: LayerNorm,
mlp: Mlp,
window: usize,
}
impl Layer {
/// `x`: `[B, H, W, C]`.
fn forward(&self, x: &Tensor, rope: &(Tensor, Tensor)) -> Result<Tensor> {
let (b, h, w, c) = x.dims4()?;
let y = self.ln1.forward(x)?;
let y = match (h.checked_div(self.window), w.checked_div(self.window)) {
(Some(nh), Some(nw)) => {
// non-overlapping windows (72 is a multiple of 24: no padding)
let ws = self.window;
let y = y.reshape((b, nh, ws, nw, ws, c))?.permute((0, 1, 3, 2, 4, 5))?.reshape((b * nh * nw, ws, ws, c))?;
let y = self.attend(&y, rope)?;
y.reshape((b, nh, nw, ws, ws, c))?.permute((0, 1, 3, 2, 4, 5))?.reshape((b, h, w, c))?
}
// global attention
_ => self.attend(&y, rope)?,
};
let x = (x + y)?;
let y = self.mlp.forward(&self.ln2.forward(&x)?)?;
Ok((x + y)?)
}
/// RoPE self-attention over `[B, H, W, C]`.
fn attend(&self, x: &Tensor, rope: &(Tensor, Tensor)) -> Result<Tensor> {
let (b, h, w, c) = x.dims4()?;
let seq = x.reshape((b, h * w, c))?;
Ok(self.attn_rope(&seq, rope)?.reshape((b, h, w, c))?)
}
fn attn_rope(&self, x: &Tensor, rope: &(Tensor, Tensor)) -> Result<Tensor> {
let (b, n, _) = x.dims3()?;
let split = |t: Tensor| -> Result<Tensor> { Ok(t.reshape((b, n, HEADS, HEAD_DIM))?.transpose(1, 2)?.contiguous()?) };
let (q, k, v) = self.attn.qkv(x)?;
let q = apply_rope(&split(q)?, &rope.0, &rope.1)?;
let k = apply_rope(&split(k)?, &rope.0, &rope.1)?;
let v = split(v)?;
let o = sdpa(&q, &k, &v, None, (HEAD_DIM as f64).powf(-0.5))?;
let o = o.transpose(1, 2)?.reshape((b, n, HIDDEN))?;
self.attn.out(&o)
}
}
pub struct Vit {
patch: Conv2d,
pos: Tensor,
ln_pre: LayerNorm,
layers: Vec<Layer>,
rope_window: (Tensor, Tensor),
rope_global: (Tensor, Tensor),
}
impl Vit {
/// `vb` at `detector_model.vision_encoder.backbone`.
pub fn new(vb: VarBuilder) -> Result<Self> {
let device = vb.device().clone();
let cfg = Conv2dConfig { stride: PATCH, ..Default::default() };
let patch = candle_nn::conv2d_no_bias(3, HIDDEN, PATCH, cfg, vb.pp("embeddings.patch_embeddings.projection"))?;
let pos = vb.get((1, PRETRAIN_GRID * PRETRAIN_GRID, HIDDEN), "embeddings.position_embeddings")?;
// tile (not interpolate) the 24 × 24 pretraining table over the 72 × 72 grid
let reps = GRID.div_ceil(PRETRAIN_GRID);
let pos = pos
.reshape((1, PRETRAIN_GRID, PRETRAIN_GRID, HIDDEN))?
.repeat((1, reps, reps, 1))?
.narrow(1, 0, GRID)?
.narrow(2, 0, GRID)?
.reshape((1, GRID * GRID, HIDDEN))?;
let mut layers = Vec::with_capacity(LAYERS);
for i in 0..LAYERS {
let l = vb.pp(format!("layers.{i}"));
layers.push(Layer {
ln1: ln(HIDDEN, EPS, l.pp("layer_norm1"))?,
attn: Attention::new(HIDDEN, HIDDEN, HEADS, l.pp("attention"))?,
ln2: ln(HIDDEN, EPS, l.pp("layer_norm2"))?,
mlp: Mlp::new(HIDDEN, 4736, true, l.pp("mlp"))?,
window: if GLOBAL.contains(&i) { 0 } else { WINDOW },
});
}
Ok(Self {
patch,
pos,
ln_pre: ln(HIDDEN, EPS, vb.pp("layer_norm"))?,
layers,
rope_window: rope_tables(WINDOW, 1.0, &device)?,
rope_global: rope_tables(GRID, WINDOW as f64 / GRID as f64, &device)?,
})
}
/// `pixels`: `[1, 3, 1008, 1008]` normalized to −1..1. Returns `[1, 1024, 72, 72]`.
pub fn forward(&self, pixels: &Tensor) -> Result<Tensor> {
let x = self.patch.forward(pixels)?; // [1, C, 72, 72]
let b = x.dim(0)?;
let x = x.flatten_from(2)?.transpose(1, 2)?.broadcast_add(&self.pos)?;
let mut x = self.ln_pre.forward(&x.reshape((b, GRID, GRID, HIDDEN))?)?;
for l in &self.layers {
let rope = if l.window > 0 { &self.rope_window } else { &self.rope_global };
x = l.forward(&x, rope)?;
}
Ok(x.permute((0, 3, 1, 2))?.contiguous()?)
}
}
+151
View File
@@ -0,0 +1,151 @@
//! Safetensors weights read on demand.
//!
//! Tensors are read from the file when a module asks for them (positional reads, no memory
//! map: `unsafe` stays in `lightcraft-sysmem`), so building only the point-prompt model never
//! touches the text encoder's bytes.
use std::collections::HashMap;
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
use std::path::Path;
use std::sync::{Arc, Mutex};
use candle_core::{DType, Device, Shape, Tensor};
use candle_nn::var_builder::SimpleBackend;
use crate::{Error, Result};
struct Entry {
shape: Vec<usize>,
start: u64,
end: u64,
}
/// An open safetensors file: its header, and the file to read tensor bytes from.
pub struct Weights {
file: Mutex<File>,
entries: HashMap<String, Entry>,
data_start: u64,
}
impl Weights {
pub fn open(path: &Path) -> Result<Arc<Self>> {
let mut file = File::open(path).map_err(|e| Error::Model(format!("{}: {e}", path.display())))?;
let file_len = file.metadata().map_err(|e| Error::Model(format!("{}: {e}", path.display())))?.len();
let mut len = [0u8; 8];
file.read_exact(&mut len).map_err(|e| Error::Model(format!("{}: {e}", path.display())))?;
let n = u64::from_le_bytes(len);
if n > 100 << 20 || n.saturating_add(8) > file_len {
return Err(Error::Model(format!("{}: not a safetensors file (header of {n} bytes)", path.display())));
}
let mut header = vec![0u8; n as usize];
file.read_exact(&mut header).map_err(|e| Error::Model(format!("{}: {e}", path.display())))?;
let header: serde_json::Map<String, serde_json::Value> =
serde_json::from_slice(&header).map_err(|e| Error::Model(format!("{}: bad header: {e}", path.display())))?;
let mut entries = HashMap::new();
for (name, v) in header {
if name == "__metadata__" {
continue;
}
let dtype = match v.get("dtype").and_then(|d| d.as_str()) {
Some("F32") => DType::F32,
other => return Err(Error::Model(format!("{name}: unsupported dtype {other:?} (the SAM 3 checkpoint is F32)"))),
};
let shape: Vec<usize> = v
.get("shape")
.and_then(|s| s.as_array())
.map(|a| a.iter().filter_map(|x| x.as_u64().map(|x| x as usize)).collect())
.unwrap_or_default();
let offsets: Vec<u64> =
v.get("data_offsets").and_then(|s| s.as_array()).map(|a| a.iter().filter_map(|x| x.as_u64()).collect()).unwrap_or_default();
let [start, end] = offsets[..] else { return Err(Error::Model(format!("{name}: bad data_offsets"))) };
let want = shape.iter().try_fold(dtype.size_in_bytes(), |a, d| a.checked_mul(*d));
if end.checked_sub(start) != want.map(|w| w as u64) {
return Err(Error::Model(format!("{name}: size does not match its shape")));
}
// the bytes must be in the file (a damaged or truncated download fails here, not later)
if (8 + n).checked_add(end).is_none_or(|e| e > file_len) {
return Err(Error::Model(format!("{}: tensor {name} lies outside the file (incomplete or damaged download?)", path.display())));
}
entries.insert(name, Entry { shape, start, end });
}
Ok(Arc::new(Self { file: Mutex::new(file), entries, data_start: 8 + n }))
}
pub fn contains(&self, name: &str) -> bool {
self.entries.contains_key(name)
}
/// Tensor `name` as stored, on the CPU.
pub fn read(&self, name: &str) -> Result<Tensor> {
let e = self.entries.get(name).ok_or_else(|| Error::Model(format!("missing tensor `{name}`")))?;
// checked against the file's length in `open`
let len = usize::try_from(e.end.saturating_sub(e.start)).map_err(|_| Error::Model(format!("{name}: too large")))?;
let mut bytes = vec![0u8; len];
{
let mut f = self.file.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
f.seek(SeekFrom::Start(self.data_start + e.start)).map_err(|err| Error::Model(format!("{name}: {err}")))?;
f.read_exact(&mut bytes).map_err(|err| Error::Model(format!("{name}: {err}")))?;
}
let v: Vec<f32> = bytes.as_chunks::<4>().0.iter().map(|c| f32::from_le_bytes(*c)).collect();
Ok(Tensor::from_vec(v, e.shape.as_slice(), &Device::Cpu)?)
}
}
/// [`Weights`] as a candle `VarBuilder` backend.
pub struct Backend(pub Arc<Weights>);
impl SimpleBackend for Backend {
fn get(&self, s: Shape, name: &str, _h: candle_nn::Init, dtype: DType, dev: &Device) -> candle_core::Result<Tensor> {
let t = self.get_unchecked(name, dtype, dev)?;
if t.shape() != &s {
candle_core::bail!("shape mismatch for {name}: expected {s:?}, got {:?}", t.shape())
}
Ok(t)
}
fn get_unchecked(&self, name: &str, dtype: DType, dev: &Device) -> candle_core::Result<Tensor> {
let t = self.0.read(name).map_err(|e| candle_core::Error::Msg(e.to_string()))?;
t.to_dtype(dtype)?.to_device(dev)
}
fn contains_tensor(&self, name: &str) -> bool {
self.0.contains(name)
}
}
#[cfg(test)]
mod tests {
use super::*;
fn write(name: &str, header: &str, data_len: usize) -> std::path::PathBuf {
let p = std::env::temp_dir().join(format!("lc-weights-{name}-{}.safetensors", std::process::id()));
let mut b = (header.len() as u64).to_le_bytes().to_vec();
b.extend_from_slice(header.as_bytes());
b.extend(std::iter::repeat_n(0u8, data_len));
std::fs::write(&p, b).unwrap();
p
}
#[test]
fn tensors_must_lie_inside_the_file() {
let ok = write("ok", r#"{"a":{"dtype":"F32","shape":[2,2],"data_offsets":[0,16]}}"#, 16);
let w = Weights::open(&ok).unwrap();
assert_eq!(w.read("a").unwrap().dims(), &[2, 2]);
// a truncated download: the header promises bytes that aren't there
let short = write("short", r#"{"a":{"dtype":"F32","shape":[2,2],"data_offsets":[0,16]}}"#, 8);
assert!(Weights::open(&short).err().unwrap().to_string().contains("outside the file"));
// hostile offsets: huge, or beyond u64
let huge = write("huge", r#"{"a":{"dtype":"F32","shape":[1073741824,1073741824],"data_offsets":[0,4611686018427387904]}}"#, 16);
assert!(Weights::open(&huge).is_err());
let wrap = write("wrap", r#"{"a":{"dtype":"F32","shape":[1],"data_offsets":[18446744073709551612,18446744073709551616]}}"#, 16);
assert!(Weights::open(&wrap).is_err());
// a header longer than the file
let p = std::env::temp_dir().join(format!("lc-weights-hdr-{}.safetensors", std::process::id()));
std::fs::write(&p, 1000u64.to_le_bytes()).unwrap();
assert!(Weights::open(&p).is_err());
for f in [ok, short, huge, wrap, p] {
let _ = std::fs::remove_file(f);
}
}
}
+97
View File
@@ -0,0 +1,97 @@
//! The port against the reference implementation (Hugging Face `transformers`, fp32).
//!
//! Needs the checkpoint and a reference file made by `tools/sam3_reference.py`:
//! `LIGHTCRAFT_SAM3_DIR=<model dir> LIGHTCRAFT_SAM3_REF=<ref.safetensors> cargo test -p lightcraft-segment --release -- --nocapture`.
//! Skipped (passes) when either is missing.
use std::collections::HashMap;
use std::path::PathBuf;
use candle_core::{Device, IndexOp, Tensor};
use lightcraft_segment::{Click, Sam3, tokenizer::Tokenizer};
fn setup() -> Option<(PathBuf, HashMap<String, Tensor>)> {
let dir = PathBuf::from(std::env::var_os("LIGHTCRAFT_SAM3_DIR")?);
let reference = PathBuf::from(std::env::var_os("LIGHTCRAFT_SAM3_REF")?);
let r = candle_core::safetensors::load(&reference, &Device::Cpu).ok()?;
Some((dir, r))
}
fn vec(t: &Tensor) -> Vec<f32> {
t.to_device(&Device::Cpu).unwrap().flatten_all().unwrap().to_vec1::<f32>().unwrap()
}
/// Max |a − b| relative to the reference's max |b|.
fn rel_err(name: &str, a: &[f32], b: &[f32]) -> f32 {
assert_eq!(a.len(), b.len(), "{name}: length");
let scale = b.iter().fold(0f32, |m, v| m.max(v.abs())).max(1e-6);
let err = a.iter().zip(b).fold(0f32, |m, (x, y)| m.max((x - y).abs()));
eprintln!("{name:>18}: max abs err {err:.2e}, rel {:.2e} (scale {scale:.2e})", err / scale);
err / scale
}
/// Fraction of pixels where the two logit maps disagree on the mask (> 0).
fn mask_disagreement(a: &[f32], b: &[f32]) -> f32 {
a.iter().zip(b).filter(|(x, y)| (**x > 0.0) != (**y > 0.0)).count() as f32 / a.len() as f32
}
#[test]
fn matches_reference() {
let Some((dir, r)) = setup() else {
eprintln!("skipped: set LIGHTCRAFT_SAM3_DIR and LIGHTCRAFT_SAM3_REF");
return;
};
let mut model = Sam3::load(&dir).unwrap();
eprintln!("device: {:?}", model.device());
let t = std::time::Instant::now();
let mut enc = model.encode_pixels(&r["pixel_values"]).unwrap();
let backbone = enc.backbone().permute((0, 2, 3, 1)).unwrap();
assert!(rel_err("backbone", &vec(&backbone), &vec(&r["backbone"])) < 1e-3);
eprintln!("backbone: {:?}", t.elapsed());
// points: two clicks (single-mask output), one click (best of three)
let clicks = [Click { x: 0.5, y: 0.55, positive: true }, Click { x: 0.2, y: 0.2, positive: false }];
let t = std::time::Instant::now();
let p = model.segment_clicks(&mut enc, &clicks).unwrap();
eprintln!("clicks (incl. tracker load + features): {:?}", t.elapsed());
let want = vec(&r["trk_masks"]);
assert!(rel_err("2-click logits", &p.logits, &want) < 1e-2);
assert!(mask_disagreement(&p.logits, &want) < 1e-3);
let t = std::time::Instant::now();
let p = model.segment_clicks(&mut enc, &clicks[..1]).unwrap();
eprintln!("one more click: {:?}", t.elapsed());
let iou = vec(&r["trk_multi_iou"]);
let best = (0..3).max_by(|a, b| iou[*a].partial_cmp(&iou[*b]).unwrap()).unwrap();
let want = vec(&r["trk_multi_masks"].i((0, 0, best)).unwrap());
assert!(rel_err("1-click logits", &p.logits, &want) < 1e-2);
assert!(mask_disagreement(&p.logits, &want) < 1e-3);
// tokenizer
let tok = Tokenizer::load(&dir).unwrap();
let cases = ["person", "Red Car", "a dog's toy", "sky, clouds!", "café 2 people", "tree-line"];
for (i, c) in cases.iter().enumerate() {
let (ids, _) = tok.encode(c);
let want: Vec<i64> = r[&format!("tok{i}")].flatten_all().unwrap().to_vec1().unwrap();
assert_eq!(ids.iter().map(|v| *v as i64).collect::<Vec<_>>(), want, "tokens of {c:?}");
}
// text prompts
for (i, text) in ["moon", "sky", "bird"].iter().enumerate() {
let t = std::time::Instant::now();
let out = model.detect(&mut enc, text).unwrap();
eprintln!("detect {text:?}: {:?}", t.elapsed());
assert!(rel_err("text features", &vec(&out.text), &vec(&r[&format!("det{i}_text")])) < 1e-3);
assert!(rel_err("query logits", &out.logits, &vec(&r[&format!("det{i}_logits")])) < 1e-2);
let presence = vec(&r[&format!("det{i}_presence")])[0];
assert!((out.presence - presence).abs() < 1e-2, "presence {} vs {presence}", out.presence);
let boxes: Vec<f32> = out.boxes.iter().flatten().copied().collect();
assert!(rel_err("boxes", &boxes, &vec(&r[&format!("det{i}_boxes")])) < 1e-2);
// the best instance's mask
let logits = vec(&r[&format!("det{i}_logits")]);
let q = (0..logits.len()).max_by(|a, b| logits[*a].partial_cmp(&logits[*b]).unwrap()).unwrap();
let ours = vec(&out.masks.i(q).unwrap());
let want = vec(&r[&format!("det{i}_masks")].i((0, q)).unwrap());
assert!(rel_err("best mask", &ours, &want) < 2e-2);
assert!(mask_disagreement(&ours, &want) < 2e-3);
}
}
+1
View File
@@ -10,6 +10,7 @@ workspace = true
[dependencies]
lightcraft-geom = { workspace = true }
lightcraft-meta = { workspace = true }
lightcraft-color = { workspace = true }
lightcraft-raster = { workspace = true }
lightcraft-develop = { workspace = true }
+96 -14
View File
@@ -1,18 +1,100 @@
//! Compile translated format strings so Rust checks both languages' placeholders.
use std::{collections::BTreeMap, env, fs, path::PathBuf};
//! Compile the translated format strings into the `tr_format!` macro.
//!
//! `locales/<code>-formats.json` maps an English format string to its translation for one language.
//! Every English message becomes one `tr_format!` arm that picks the active language and formats
//! that language's translation with `format!` — so Rust checks every catalog's placeholders against
//! the call site's arguments (a wrong or forgotten placeholder is a compile error, and `{:.1}`,
//! `{:+}`… keep their meaning). The English source is the fallback arm. Adding a language means
//! adding its file (and its `language_table!` entry); this script picks the file up by name.
//!
//! Every catalog must carry the same messages: a `tr_format!` call whose message is in no catalog
//! does not compile, so a message missing from one language would otherwise go unnoticed until it
//! shows English.
use std::{
collections::{BTreeMap, BTreeSet},
env, fs,
path::PathBuf,
};
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("cargo:rerun-if-changed=locales/ja-formats.json");
let messages: BTreeMap<String, String> = serde_json::from_str(&fs::read_to_string("locales/ja-formats.json")?)?;
let mut source = String::from("macro_rules! tr_format {\n");
for (english, japanese) in messages {
let en = serde_json::to_string(&english)?;
let ja = serde_json::to_string(&japanese)?;
source.push_str(&format!(
"({en} $(, $($args:tt)*)?) => {{ if $crate::i18n::is_japanese() {{ format!({ja} $(, $($args)*)?) }} else {{ format!({en} $(, $($args)*)?) }} }};\n"
));
}
source.push_str("}\npub(crate) use tr_format;\n");
fs::write(PathBuf::from(env::var("OUT_DIR")?).join("ja-formats.rs"), source)?;
println!("cargo:rerun-if-changed=build.rs");
// The directory itself, so a catalog added or removed reruns this script.
println!("cargo:rerun-if-changed=locales");
let out = PathBuf::from(env::var("OUT_DIR")?);
fs::write(out.join("tr-formats.rs"), generate()?)?;
Ok(())
}
/// `ja` → `Ja`, `zh-hans` → `ZhHans`: the `Locale` variant the `language_table!` entry declares.
/// The code must be a lowercase BCP-47-like tag (`^[a-z]+(-[a-z]+)*$`), so the variant is always a
/// valid Rust identifier.
fn variant_for(code: &str) -> Result<String, String> {
let valid = !code.is_empty() && code.split('-').all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_lowercase()));
if !valid {
return Err(format!("locales/{code}-formats.json: the language code must match ^[a-z]+(-[a-z]+)*$"));
}
Ok(code.split('-').map(capitalize).collect())
}
fn capitalize(part: &str) -> String {
let mut chars = part.chars();
match chars.next() {
Some(first) => first.to_uppercase().collect::<String>() + chars.as_str(),
None => String::new(),
}
}
fn generate() -> Result<String, Box<dyn std::error::Error>> {
let mut catalogs: Vec<(String, String, BTreeMap<String, String>)> = Vec::new();
for entry in fs::read_dir("locales")? {
let path = entry?.path();
let name = path.file_name().and_then(|name| name.to_str()).unwrap_or_default().to_string();
// `en` is the source language: it has no catalog, its formats are the source themselves.
let Some(code) = name.strip_suffix("-formats.json").filter(|code| *code != "en") else { continue };
println!("cargo:rerun-if-changed=locales/{name}");
let variant = variant_for(code)?;
let messages: BTreeMap<String, String> =
serde_json::from_str(&fs::read_to_string(&path)?).map_err(|error| format!("locales/{name}: {error}"))?;
catalogs.push((code.to_string(), variant, messages));
}
catalogs.sort_by(|a, b| a.0.cmp(&b.0));
// Every catalog translates the same messages.
let all: BTreeSet<&String> = catalogs.iter().flat_map(|(_, _, messages)| messages.keys()).collect();
let mut problems = Vec::new();
for (code, _, messages) in &catalogs {
for key in &all {
if !messages.contains_key(*key) {
problems.push(format!("locales/{code}-formats.json is missing {key:?}"));
}
}
}
if !problems.is_empty() {
return Err(problems.join("\n").into());
}
let mut source = String::from(concat!(
"/// Translate a format string, checked by `format!` in every language.\n",
"/// `build.rs` writes one arm per message from `locales/*-formats.json`.\n",
"macro_rules! tr_format {\n",
));
for english in all {
let en = serde_json::to_string(english)?;
source.push_str(&format!(" ({en} $(, $($args:tt)*)?) => {{\n match $crate::i18n::language() {{\n"));
for (_, variant, messages) in &catalogs {
if let Some(translated) = messages.get(english) {
let translated = serde_json::to_string(translated)?;
source.push_str(&format!(" $crate::i18n::Locale::{variant} => format!({translated} $(, $($args)*)?),\n"));
}
}
source.push_str(&format!(" _ => format!({en} $(, $($args)*)?),\n }}\n }};\n"));
}
source.push_str(concat!(
"}\n",
"\n",
"// `macro_rules!` is textual: the module re-export is what lets call sites say\n",
"// `crate::i18n::tr_format!`.\n",
"pub(crate) use tr_format;\n",
));
Ok(source)
}
+16 -1
View File
@@ -103,5 +103,20 @@
"Adding photos… {} of {}": "写真を追加中:{1}枚中{0}枚",
" ({} in Recently Deleted)": "(うち{}枚は最近削除した写真)",
"Already in the library: {} photo{} selected in All Photos": "ライブラリに登録済み:すべての写真で{}枚を選択しました{:.0}",
"Already in the library, in Recently Deleted: {} photo{} selected · right-click ▸ Restore, or Delete Permanently to import afresh": "ライブラリの「最近削除した写真」にあります:{}枚を選択しました{:.0}・右クリック ▸ 復元、または完全に削除してから読み込み直し"
"Already in the library, in Recently Deleted: {} photo{} selected · right-click ▸ Restore, or Delete Permanently to import afresh": "ライブラリの「最近削除した写真」にあります:{}枚を選択しました{:.0}・右クリック ▸ 復元、または完全に削除してから読み込み直し",
"Auto ({})": "自動 ({})",
"Build 1:1 Previews ({scope})": "1:1プレビューを作成({scope})",
"Build Smart Previews ({scope})": "スマートプレビューを作成({scope})",
"Build Standard-Sized Previews ({scope})": "標準プレビューを作成({scope})",
"Discard Smart Previews ({scope})": "スマートプレビューを破棄({scope})",
"Label Set: {name}": "ラベルセット:{name}",
"Quick Develop: {name}": "クイック現像: {name}",
"Reset {name}": "{name}をリセット",
"Updated “{name}”": "「{name}」を更新しました",
"{month}": "{month}月",
"{month} {year}": "{year}年{month}月",
"{verb} {n} Photo{}": "{n}枚を{verb}{:.0}",
"{weekday}, {day}": "{day}日({weekday})",
"{weekday}, {day} {month} {year}": "{year}年{month}月{day}日({weekday})",
"{year}": "{year}年"
}
+9 -1
View File
@@ -1019,5 +1019,13 @@
"Back Up Library…": "ライブラリをバックアップ…",
"Restore Library from Backup…": "バックアップからライブラリを復元…",
"In Recently Deleted": "最近削除した写真内",
"Reload from Disk": "ディスクから再読み込み"
"Reload from Disk": "ディスクから再読み込み",
"Monday": "月曜日",
"Tuesday": "火曜日",
"Wednesday": "水曜日",
"Thursday": "木曜日",
"Friday": "金曜日",
"Saturday": "土曜日",
"Sunday": "日曜日",
"Unknown Date": "日付不明"
}
+122
View File
@@ -0,0 +1,122 @@
{
" · {skipped} skipped (file exists)": " · 已跳过 {skipped} 个(文件已存在)",
" · {} duplicate{} skipped": " · 已跳过 {0} 个重复项{1:.0}",
" · {} failed: {}": " · {0} 个失败:{1}",
" · {} moved": " · 已移动 {0} 个",
" · {} not readable": " · {0} 个无法读取",
", {failed} file{} not readable": ",{failed} 个文件无法读取{0:.0}",
"Add {local_n} to My Photos": "将 {local_n} 张照片添加到我的照片",
"Add {} Photo{}": "添加 {0} 张照片{1:.0}",
"Add “{}” to Favorites": "将“{0}”添加到收藏夹",
"Added {n} photo{} to My Photos": "已将 {n} 张照片添加到我的照片{0:.0}",
"Added {n} photo{} to “{}”": "已将 {n} 张照片添加到“{1}”{0:.0}",
"All {n} selected photos move by the same amount.": "所选 {n} 张照片将按相同幅度移动。",
"Auto Sync: edits apply to {n} photos": "自动同步:编辑将应用到 {n} 张照片",
"Automatic ({} MB)": "自动({0} MB)",
"Click {}:1": "单击 {0}:1",
"Create {n} Virtual Copies": "创建 {n} 个虚拟副本",
"Decoder: {reason}": "解码器:{reason}",
"Delete \"{label}\"": "删除 \"{label}\"",
"Delete \"{label}\" (and the mask)": "删除 \"{label}\"(及蒙版)",
"Delete {n} Photos": "删除 {n} 张照片",
"Delete “{current}”": "删除“{current}”",
"Duplicate \"{label}\"": "复制 \"{label}\"",
"Example: {example}": "示例:{example}",
"Exported {n} photo{}": "已导出 {n} 张照片{0:.0}",
"Exported {ok} of {total} photo{}": "已导出 {total} 张中的 {ok} 张照片{0:.0}",
"Exported {} point curve preset{}": "已导出 {0} 个点曲线预设{1:.0}",
"Exported {} preset{}": "已导出 {0} 个预设{1:.0}",
"Exporting {} of {total}": "正在导出 {total} 张中的第 {0} 张",
"Filter bar — {active} active filter{}": "筛选栏 — 已启用 {active} 个筛选{0:.0}",
"Healed {n} dust spot{}": "已修复 {n} 处尘点{0:.0}",
"Imported {n} point curve preset{}": "已导入 {n} 个点曲线预设{0:.0}",
"Imported {n} preset{}": "已导入 {n} 个预设{0:.0}",
"Imported {n} preset{}, {f} file{} not readable": "已导入 {n} 个预设,{f} 个文件无法读取{0:.0}{1:.0}",
"Imported {} photo{}": "已导入 {0} 张照片{1:.0}",
"Matches: {}": "匹配项:{0}",
"Merging… {stage} {:.0}%": "合成中… {stage} {0:.0}%",
"Move {what} to Recently Deleted?": "将 {what} 移到最近删除?",
"Named ({named_n})": "已命名({named_n})",
"Paste into {n} photo{} — only settings that were copied are pasted": "粘贴到 {n} 张照片 — 仅粘贴已复制的设置{0:.0}",
"Preset: {name}": "预设:{name}",
"Preview only — editing the camera's embedded JPEG ({})": "仅预览 — 正在编辑相机内嵌的 JPEG({0})",
"Preview only — {}": "仅预览 — {0}",
"Profile: {}": "配置文件:{0}",
"Proof Copy ({})": "校样副本({0})",
"Proof Preview · {}": "校样预览 · {0}",
"Quick Develop · {n} photos": "快速编辑 · {n} 张照片",
"Rated {}": "已评级 {0}",
"Reading photos… {done} of {total}": "正在读取照片… {total} 张中的 {done} 张",
"Redo {}": "重做:{0}",
"Remove “{}” from Favorites": "从收藏夹中移除“{0}”",
"Rename {n} Photos…": "重命名 {n} 张照片…",
"Rename “{name}”": "重命名“{name}”",
"Rendering on the CPU: {why}": "正在使用 CPU 渲染:{why}",
"Reset {} Channel": "重置 {0} 通道",
"Saved export preset “{}”": "已保存导出预设“{0}”",
"Show {hidden} hidden location{}": "显示 {hidden} 个隐藏位置{0:.0}",
"Showing {n} of {}": "显示 {0} 张中的 {n} 张",
"Smart album: {}": "智能相册:{0}",
"Undo {}": "撤销 {0}",
"Version {} — a clean-room, pure-Rust photo library and raw developer.": "版本 {0} — 净室实现的纯 Rust 照片图库与 RAW 编辑工具。",
"Version: {}": "版本:{0}",
"none (CPU rendering): {why}": "无(CPU 渲染):{why}",
"the {n} selected photos": "选中的 {n} 张照片",
"the {} photos in view": "视图中的 {0} 张照片",
"these {count} photos": "这 {count} 张照片",
"{count} smart preview{} · {:.1} MB": "{count} 个智能预览{0:.0} · {1:.1} MB",
"{f} folders (and their subfolders)": "{f} 个文件夹(及其子文件夹)",
"{label} Grading": "{label} 颜色分级",
"{label} {d:+} on every selected photo": "每张选中照片的 {label} {d:+}",
"{label}, {n} photos": "{label},{n} 张照片",
"{matching} of {t} photos": "{t} 张照片中的 {matching} 张",
"{matching} photos": "{matching} 张照片",
"{mode} spot {} of {}": "{mode}:第 {0} 个污点,共 {1} 个",
"{msg} ({failed} failed)": "{msg}({failed} 项失败)",
"{name} Label": "{name} 标签",
"{name} Off": "已关闭 {name}",
"{name} On": "已开启 {name}",
"{name} — double-click to reset it": "{name} — 双击可重置",
"{n} change{} saved in memory but not written to disk: {e}\nLightCraft retries automatically; quitting now would lose {}.": "{n} 项更改已保存在内存中,但尚未写入磁盘:{e}\nLightCraft 会自动重试;现在退出将丢失这些更改。{0:.0}{1:.0}",
"{n} found · {sel} selected": "找到 {n} 张 · 已选 {sel} 张",
"{n} photos": "{n} 张照片",
"{n} photos in {} are hidden by the filters above": "{0} 中的 {n} 张照片被上方的过滤器隐藏",
"{n} photo{}": "{n} 张照片{0:.0}",
"{n} photo{} match · updates automatically as photos change": "{n} 张照片符合 · 随照片变化自动更新{0:.0}",
"{n} photo{} now · updates automatically as photos change": "当前 {n} 张照片 · 随照片变化自动更新{0:.0}",
"{sel_n} selected · {counted}": "已选择 {sel_n} · {counted}",
"{stage}… {:.0}%": "{stage}… {0:.0}%",
"{what} merge added": "已添加 {what} 合并",
"{what} merge failed: {e}": "{what} 合并失败:{e}",
"{} and higher": "{0} 及以上",
"{} files": "{0} 个文件",
"{} files and {f} folder{}": "{0} 个文件和 {f} 个文件夹{1:.0}",
"{} of 4 guides — drag along lines that should be vertical or horizontal.": "已画 {0} 条参考线(共 4 条)— 沿应为垂直或水平的线拖动。",
"{} photos": "{0} 张照片",
"{} photo{} not readable": "{0} 张照片无法读取{1:.0}",
"{} spot(s) on this photo": "此照片上有 {0} 处污点",
"· {dups} already in the library": "· {dups} 张已在图库中",
"· {n} photo{}": "· {n} 张照片{0:.0}",
"… and {more} more": "…以及另外 {more} 项",
"Import cancelled · {} photo{} added": "导入已取消 · 已添加 {0} 张照片{1:.0}",
"Reading photos… {} of {}": "正在读取照片… 第 {0} 张,共 {1} 张",
"Adding photos… {} of {}": "正在添加照片… 第 {0} 张,共 {1} 张",
" ({} in Recently Deleted)": "(其中 {0} 张在“最近删除”中)",
"Already in the library: {} photo{} selected in All Photos": "已在图库中:已在“所有照片”中选择 {0} 张照片{1:.0}",
"Already in the library, in Recently Deleted: {} photo{} selected · right-click ▸ Restore, or Delete Permanently to import afresh": "已在图库的“最近删除”中:已选择 {0} 张照片{1:.0} · 右键单击 ▸ 恢复,或永久删除后重新导入",
"Auto ({})": "自动({})",
"Build 1:1 Previews ({scope})": "生成 1:1 预览({scope})",
"Build Smart Previews ({scope})": "生成智能预览({scope})",
"Build Standard-Sized Previews ({scope})": "生成标准尺寸预览({scope})",
"Discard Smart Previews ({scope})": "丢弃智能预览({scope})",
"Label Set: {name}": "标签集:{name}",
"Quick Develop: {name}": "快速编辑:{name}",
"Reset {name}": "重置{name}",
"Updated “{name}”": "已更新“{name}”",
"{verb} {n} Photo{}": "{verb} {n} 张照片{:.0}",
"{year}": "{year}年",
"{month}": "{month}月",
"{month} {year}": "{year}年{month}月",
"{weekday}, {day}": "{day}日 {weekday}",
"{weekday}, {day} {month} {year}": "{year}年{month}月{day}日 {weekday}"
}
File diff suppressed because it is too large Load Diff
+122
View File
@@ -0,0 +1,122 @@
{
" · {skipped} skipped (file exists)": " · 已略過 {skipped} 個(檔案已存在)",
" · {} duplicate{} skipped": " · 已略過 {} 張重複照片{:.0}",
" · {} failed: {}": " · {} 個失敗:{}",
" · {} moved": " · 已移動 {} 個",
" · {} not readable": " · {} 個無法讀取",
", {failed} file{} not readable": ",{failed} 個檔案無法讀取{:.0}",
"Add {local_n} to My Photos": "將 {local_n} 張加入我的照片",
"Add {} Photo{}": "新增 {} 張照片{:.0}",
"Add “{}” to Favorites": "將「{}」加入喜好項目",
"Added {n} photo{} to My Photos": "已將 {n} 張照片加入我的照片{:.0}",
"Added {n} photo{} to “{}”": "已將 {n} 張照片加入「{1}」{0:.0}",
"Adding photos… {} of {}": "正在新增照片… {}/{}",
"All {n} selected photos move by the same amount.": "所有 {n} 張選取的照片會調整相同的時間。",
"Auto ({})": "自動({})",
"Auto Sync: edits apply to {n} photos": "自動同步:編輯會套用至 {n} 張照片",
"Automatic ({} MB)": "自動({} MB)",
"Build 1:1 Previews ({scope})": "建立 1:1 預覽({scope})",
"Build Smart Previews ({scope})": "建立智慧型預覽({scope})",
"Build Standard-Sized Previews ({scope})": "建立標準尺寸預覽({scope})",
"Click {}:1": "點擊時 {}:1",
"Create {n} Virtual Copies": "建立 {n} 張虛擬副本",
"Decoder: {reason}": "解碼器:{reason}",
"Delete \"{label}\"": "刪除「{label}」",
"Delete \"{label}\" (and the mask)": "刪除「{label}」及遮色片",
"Delete {n} Photos": "刪除 {n} 張照片",
"Delete “{current}”": "刪除「{current}」",
"Discard Smart Previews ({scope})": "捨棄智慧型預覽({scope})",
"Duplicate \"{label}\"": "複製「{label}」",
"Example: {example}": "範例:{example}",
"Exported {n} photo{}": "已匯出 {n} 張照片{:.0}",
"Exported {ok} of {total} photo{}": "已匯出 {ok}/{total} 張照片{:.0}",
"Exported {} point curve preset{}": "已匯出 {} 個點曲線預設集{:.0}",
"Exported {} preset{}": "已匯出 {} 個預設集{:.0}",
"Exporting {} of {total}": "正在匯出第 {}/{total} 張",
"Filter bar — {active} active filter{}": "篩選列 — 已套用 {active} 個篩選條件{:.0}",
"Healed {n} dust spot{}": "已修復 {n} 個灰塵斑點{:.0}",
"Import cancelled · {} photo{} added": "已取消匯入 · 已加入 {} 張照片{:.0}",
"Imported {n} point curve preset{}": "已匯入 {n} 個點曲線預設集{:.0}",
"Imported {n} preset{}": "已匯入 {n} 個預設集{:.0}",
"Imported {n} preset{}, {f} file{} not readable": "已匯入 {n} 個預設集{:.0},{f} 個檔案無法讀取{:.0}",
"Imported {} photo{}": "已匯入 {} 張照片{:.0}",
"Label Set: {name}": "標籤集:{name}",
"Matches: {}": "符合條件:{}",
"Merging… {stage} {:.0}%": "正在合併… {stage} {:.0}%",
"Move {what} to Recently Deleted?": "要將{what}移至「最近刪除」嗎?",
"Named ({named_n})": "已命名({named_n})",
"Paste into {n} photo{} — only settings that were copied are pasted": "貼上至 {n} 張照片{:.0} — 只貼上已複製的設定",
"Preset: {name}": "預設集:{name}",
"Preview only — editing the camera's embedded JPEG ({})": "僅供預覽 — 正在編輯相機內嵌的 JPEG({})",
"Preview only — {}": "僅供預覽 — {}",
"Profile: {}": "描述檔:{}",
"Proof Copy ({})": "校樣副本({})",
"Proof Preview · {}": "校樣預覽 · {}",
"Quick Develop · {n} photos": "快速顯像 · {n} 張照片",
"Quick Develop: {name}": "快速顯像:{name}",
"Rated {}": "評分:{}",
"Reading photos… {done} of {total}": "正在讀取照片… {done}/{total}",
"Reading photos… {} of {}": "正在讀取照片… {}/{}",
"Redo {}": "重做 {}",
"Remove “{}” from Favorites": "將「{}」從喜好項目移除",
"Rename {n} Photos…": "重新命名 {n} 張照片…",
"Rename “{name}”": "重新命名「{name}」",
"Rendering on the CPU: {why}": "使用 CPU 算繪:{why}",
"Reset {name}": "重設{name}",
"Reset {} Channel": "重設 {} 通道",
"Saved export preset “{}”": "已儲存匯出預設集「{}」",
"Show {hidden} hidden location{}": "顯示 {hidden} 個隱藏的位置{:.0}",
"Showing {n} of {}": "顯示 {n}/{} 張",
"Smart album: {}": "智慧型相簿:{}",
"Undo {}": "復原 {}",
"Updated “{name}”": "已更新「{name}」",
"Version {} — a clean-room, pure-Rust photo library and raw developer.": "版本 {} — 以純 Rust 獨立實作的照片管理與 RAW 顯像程式。",
"Version: {}": "版本:{}",
"none (CPU rendering): {why}": "無(使用 CPU 算繪):{why}",
"the {n} selected photos": "選取的 {n} 張照片",
"the {} photos in view": "目前顯示的 {} 張照片",
"these {count} photos": "這 {count} 張照片",
"{count} smart preview{} · {:.1} MB": "{count} 張智慧型預覽{:.0} · {:.1} MB",
"{f} folders (and their subfolders)": "{f} 個資料夾(及其子資料夾)",
"{label} Grading": "{label}色彩分級",
"{label} {d:+} on every selected photo": "每張選取照片的{label}調整 {d:+}",
"{label}, {n} photos": "{label},{n} 張照片",
"{matching} of {t} photos": "{matching}/{t} 張照片",
"{matching} photos": "{matching} 張照片",
"{mode} spot {} of {}": "{mode}斑點:第 {}/{} 個",
"{month}": "{month}月",
"{month} {year}": "{year}年{month}月",
"{msg} ({failed} failed)": "{msg}({failed} 個失敗)",
"{name} Label": "{name}標籤",
"{name} Off": "關閉{name}",
"{name} On": "開啟{name}",
"{name} — double-click to reset it": "{name} — 按兩下可重設",
"{n} change{} saved in memory but not written to disk: {e}\nLightCraft retries automatically; quitting now would lose {}.": "{n} 項變更{:.0}已保留在記憶體,但尚未寫入磁碟:{e}\nLightCraft 會自動重試;現在結束程式會遺失{}。",
"{n} found · {sel} selected": "找到 {n} 張 · 已選取 {sel} 張",
"{n} photos": "{n} 張照片",
"{n} photos in {} are hidden by the filters above": "{}中的 {n} 張照片被上方篩選條件隱藏",
"{n} photo{}": "{n} 張照片{:.0}",
"{n} photo{} match · updates automatically as photos change": "{n} 張照片符合{:.0} · 隨照片變更自動更新",
"{n} photo{} now · updates automatically as photos change": "目前 {n} 張照片{:.0} · 隨照片變更自動更新",
"{sel_n} selected · {counted}": "已選取 {sel_n} 張 · {counted}",
"{stage}… {:.0}%": "{stage}… {:.0}%",
"{verb} {n} Photo{}": "{verb} {n} 張照片{:.0}",
"{weekday}, {day}": "{day}日 {weekday}",
"{weekday}, {day} {month} {year}": "{year}年{month}月{day}日 {weekday}",
"{what} merge added": "已新增{what}合併照片",
"{what} merge failed: {e}": "{what}合併失敗:{e}",
"{year}": "{year}年",
"{} and higher": "{} 以上",
"{} files": "{} 個檔案",
"{} files and {f} folder{}": "{} 個檔案與 {f} 個資料夾{:.0}",
"{} of 4 guides — drag along lines that should be vertical or horizontal.": "已繪製 {}/4 條參考線 — 沿應為垂直或水平的線條拖曳。",
"{} photos": "{} 張照片",
"{} photo{} not readable": "{} 張照片無法讀取{:.0}",
"{} spot(s) on this photo": "此照片有 {} 個斑點",
"· {dups} already in the library": "· {dups} 張已在照片庫中",
"· {n} photo{}": "· {n} 張照片{:.0}",
"… and {more} more": "…以及另外 {more} 個",
" ({} in Recently Deleted)": "(其中 {} 張在「最近刪除」中)",
"Already in the library: {} photo{} selected in All Photos": "已在照片庫中:已在「所有照片」中選取 {} 張照片{:.0}",
"Already in the library, in Recently Deleted: {} photo{} selected · right-click ▸ Restore, or Delete Permanently to import afresh": "已在照片庫的「最近刪除」中:已選取 {} 張照片{:.0} · 按右鍵 ▸ 還原,或永久刪除後重新匯入"
}
File diff suppressed because it is too large Load Diff
+5 -2
View File
@@ -55,7 +55,7 @@ fn wrap(r: Result<Value, String>) -> Outcome {
pub fn all_commands(app: &LightcraftApp) -> Value {
let mut v: Vec<Value> = app.session.commands().into_iter().map(|c| serde_json::to_value(c).unwrap_or_default()).collect();
for (id, label, sc, menu) in crate::menus::UI_COMMANDS {
for (id, label, sc, menu) in crate::menus::ui_commands() {
v.push(json!({"id": id, "label": label, "shortcut": sc, "menu": [menu], "enabled": crate::menus::ui_enabled(app, id), "ui": true}));
}
Value::Array(v)
@@ -267,6 +267,7 @@ pub fn handle(app: &mut LightcraftApp, ctx: &egui::Context, req: &ControlRequest
u.dialog = app.ui.dialog.clone();
u.status = app.ui.status.clone();
app.ui = u;
crate::i18n::set_language(app.ui.language);
ctx.request_repaint();
ok(serde_json::to_value(&app.ui).unwrap_or_default())
}
@@ -277,7 +278,9 @@ pub fn handle(app: &mut LightcraftApp, ctx: &egui::Context, req: &ControlRequest
Some(d) => {
let r = crate::panels::dialogs::confirm_dialog(app, &d);
// the import review stays open on an error, as with its button
if r.is_err() && matches!(d, crate::state::Dialog::Import { .. }) {
if (r.is_err() && matches!(d, crate::state::Dialog::Import { .. } | crate::state::Dialog::SamModel { .. }))
|| (r.is_ok() && crate::panels::dialogs::keeps_open(app, &d))
{
app.ui.dialog = Some(d);
}
wrap(r)
+631 -125
View File
@@ -1,69 +1,274 @@
//! Presentation-only localization. Command ids, user names and catalog data remain unchanged.
use std::{cell::Cell, collections::BTreeMap, sync::OnceLock};
//!
//! A language is one entry in [`language_table!`] below: its BCP-47 code, the endonym shown in the
//! Language menus, the ISO 15924 script its text needs (which picks the CJK font fallback), and its
//! message catalog (`locales/<code>.json`, embedded at build time). Adding a language is that entry
//! plus its catalogs — every menu and settings surface is built from the table, so nothing else in
//! the UI changes.
use serde::{Deserialize, Serialize};
use std::{cell::RefCell, collections::BTreeMap, sync::OnceLock};
include!(concat!(env!("OUT_DIR"), "/ja-formats.rs"));
use serde::{Deserialize, Deserializer, Serialize, Serializer};
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub enum Language {
#[default]
#[serde(rename = "en")]
En,
#[serde(rename = "ja")]
Ja,
/// The language table.
///
/// The source language comes first and alone: it is the text the UI is authored in, which every
/// other language translates. `name` is written in the language itself (an endonym, so a user who
/// cannot read the current UI language can still find their own).
macro_rules! language_table {
($source:ident, $source_code:literal; $( $variant:ident, $code:literal, $name:literal, $script:literal, $catalog:expr );* $(;)?) => {
/// A UI language.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord)]
pub enum Locale {
/// The language the UI is written in.
#[default]
$source,
$($variant,)*
}
impl Locale {
/// Every language, in menu order (the source language first).
pub const ALL: &'static [Self] = &[Self::$source, $(Self::$variant,)*];
pub const fn code(self) -> &'static str {
match self { Self::$source => $source_code, $(Self::$variant => $code,)* }
}
/// The language's own name, as shown in the Language menus.
pub const fn name(self) -> &'static str {
match self { Self::$source => "English", $(Self::$variant => $name,)* }
}
/// The ISO 15924 script this language is written in (`"Latn"`, `"Jpan"`, `"Hans"`…):
/// the craft-fonts faces for it are the ones this language needs.
pub const fn script(self) -> &'static str {
match self { Self::$source => "Latn", $(Self::$variant => $script,)* }
}
/// The language's embedded catalog, parsed once. The source language has none: its
/// lookups return the source text itself.
fn catalog(self) -> &'static BTreeMap<String, String> {
static CATALOGS: OnceLock<BTreeMap<Locale, BTreeMap<String, String>>> = OnceLock::new();
static EMPTY: OnceLock<BTreeMap<String, String>> = OnceLock::new();
// A malformed catalog logs and comes back empty: a broken translation degrades to
// English text, it never takes the app down.
let parse = |catalog: Option<&'static str>| match catalog {
Some(json) => match serde_json::from_str(json) {
Ok(messages) => messages,
Err(error) => {
log::error!("Invalid message catalog for {}: {error}", self.code());
BTreeMap::new()
}
},
None => BTreeMap::new(),
};
CATALOGS
.get_or_init(|| {
Locale::ALL
.iter()
.map(|language| {
let catalog = match *language { $(Self::$variant => Some($catalog),)* _ => None };
(*language, parse(catalog))
})
.collect()
})
.get(&self)
.unwrap_or_else(|| EMPTY.get_or_init(BTreeMap::new))
}
pub fn parse(code: &str) -> Option<Self> {
match code { $source_code => Some(Self::$source), $($code => Some(Self::$variant),)* _ => None }
}
/// Lenient lookup: `zh`, `zh-CN`, `zh_Hans`, `en-US`, `ja_JP.UTF-8`… all reach a
/// shipped language. A region or script the table does not ship falls back to the
/// language itself, so a user is never left with no language at all.
pub fn parse_tag(tag: &str) -> Option<Self> {
let tag = tag.split(['.', '@']).next().unwrap_or(tag).replace('_', "-");
let lower = tag.to_ascii_lowercase();
if let Some(exact) = Self::ALL.iter().find(|language| language.code().eq_ignore_ascii_case(&lower)) {
return Some(*exact);
}
let mut parts = lower.split('-');
let language = parts.next()?;
// `zh-Hans-CN`, `zh-CN` and `zh` all name the same base language.
let script = parts.next().and_then(|part| match part {
"hans" | "cn" | "sg" => Some("Hans"),
"hant" | "tw" | "hk" | "mo" => Some("Hant"),
_ => None,
});
Self::ALL
.iter()
.find(|candidate| {
let base = candidate.code().split('-').next().unwrap_or_default();
base == language && script.is_none_or(|script| candidate.script() == script)
})
.copied()
}
pub fn tr(self, source: &str) -> &str {
tr_in(self, source)
}
}
};
}
impl Language {
pub const ALL: [Self; 2] = [Self::En, Self::Ja];
pub fn name(self) -> &'static str {
match self {
Self::En => "English",
Self::Ja => "日本語",
}
// The source language, then one entry per translation: code, endonym, ISO 15924 script, embedded
// catalog (`locales/<code>.json`).
language_table! {
En, "en";
ZhHans, "zh-hans", "简体中文", "Hans", include_str!("../locales/zh-hans.json");
ZhHant, "zh-hant", "繁體中文(台灣)", "Hant", include_str!("../locales/zh-hant.json");
Ja, "ja", "日本語", "Jpan", include_str!("../locales/ja.json");
}
// The settings file stores the BCP-47 code (`"zh-hans"`), never the Rust variant name, so a
// language can be renamed in code without invalidating anyone's settings.
impl Serialize for Locale {
fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
serializer.serialize_str(self.code())
}
pub fn parse(code: &str) -> Option<Self> {
match code {
"en" => Some(Self::En),
"ja" => Some(Self::Ja),
_ => None,
}
}
pub fn tr(self, source: &str) -> &str {
if self == Self::Ja { japanese().get(source).map(String::as_str).unwrap_or(source) } else { source }
}
impl<'de> Deserialize<'de> for Locale {
fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
let code = String::deserialize(deserializer)?;
Locale::parse(&code).or_else(|| Locale::parse_tag(&code)).ok_or_else(|| serde::de::Error::custom(format!("unknown language {code:?}")))
}
}
thread_local! {
static LANGUAGE: Cell<Language> = const { Cell::new(Language::En) };
static LOCALE: std::cell::Cell<Locale> = const { std::cell::Cell::new(Locale::En) };
}
pub fn default_language() -> Language {
if std::env::var("LIGHTCRAFT_LANGUAGE").as_deref() == Ok("ja") { Language::Ja } else { Language::En }
/// The UI language from the environment (`LIGHTCRAFT_LANGUAGE=zh-hans`), for headless runs.
pub fn default_language() -> Locale {
std::env::var("LIGHTCRAFT_LANGUAGE").ok().and_then(|value| Locale::parse_tag(&value)).unwrap_or(Locale::En)
}
pub fn set_language(language: Language) {
LANGUAGE.with(|value| value.set(language));
pub fn set_language(language: Locale) {
LOCALE.with(|value| value.set(language));
}
pub fn is_japanese() -> bool {
LANGUAGE.with(|value| value.get() == Language::Ja)
}
fn japanese() -> &'static BTreeMap<String, String> {
static MESSAGES: OnceLock<BTreeMap<String, String>> = OnceLock::new();
MESSAGES.get_or_init(|| {
serde_json::from_str(include_str!("../locales/ja.json")).unwrap_or_else(|error| {
log::error!("Invalid Japanese message catalog: {error}");
BTreeMap::new()
})
})
pub fn language() -> Locale {
LOCALE.with(std::cell::Cell::get)
}
/// Translate a built-in display label, preserving unknown labels verbatim.
/// Never call this on editable user text, filenames or command identifiers.
pub fn tr(source: &str) -> &str {
if is_japanese() { japanese().get(source).map(String::as_str).unwrap_or(source) } else { source }
tr_in(language(), source)
}
/// Catalog values are `'static` (they live in the embedded file), so a translation can be handed
/// out for as long as the `'static` key it was looked up by.
fn tr_in(language: Locale, source: &str) -> &str {
if language.catalog().is_empty() {
return source;
}
thread_local! {
static VERBATIM: RefCell<BTreeMap<Locale, BTreeMap<String, &'static str>>> = const { RefCell::new(BTreeMap::new()) };
}
VERBATIM.with_borrow_mut(|cache| {
let verbatim = cache.entry(language).or_insert_with(|| catalog_verbatim(language));
verbatim.get(source).copied().unwrap_or(source)
})
}
/// One language's catalog keyed for lookup, with values that outlive the catalog's own borrow.
fn catalog_verbatim(language: Locale) -> BTreeMap<String, &'static str> {
language.catalog().iter().map(|(key, value)| (key.clone(), value.as_str())).collect()
}
// `tr_format!` (one arm per message, one `format!` per language) is generated by `build.rs` from
// `locales/*-formats.json`.
include!(concat!(env!("OUT_DIR"), "/tr-formats.rs"));
/// A stock preset's or profile's name in the UI language; an imported or user-edited name stays
/// exactly as its owner wrote it.
pub fn builtin_label(source: &str, builtin: bool) -> &str {
if builtin { tr(source) } else { source }
}
/// An Activity (history) step for display. The step's stored label stays English; generated steps
/// are translated around the names they carry, and a user preset's name is kept verbatim.
pub fn history_label(source: &str, presets: &[lightcraft_develop::Preset]) -> String {
if let Some(name) = source.strip_prefix("Preset: ") {
// A user preset may share a stock preset's name: then the step could be either, so it
// keeps the name as written.
let builtin = presets.iter().any(|p| p.builtin && p.name == name) && !presets.iter().any(|p| !p.builtin && p.name == name);
return tr_format!("Preset: {name}", name = builtin_label(name, builtin));
}
if let Some(name) = source.strip_prefix("Reset ") {
return tr_format!("Reset {name}", name = tr(name));
}
if let Some(name) = source.strip_prefix("Quick Develop: ") {
return tr_format!("Quick Develop: {name}", name = tr(name));
}
tr(source).to_string()
}
/// A library source's heading (All Photos, Recently Deleted…) in the UI language; an album's
/// name is the user's and stays verbatim.
pub fn source_label(source: lightcraft_engine::LibrarySource, catalog: &lightcraft_catalog::Catalog) -> String {
let label = source.label(catalog);
if matches!(source, lightcraft_engine::LibrarySource::Album(id) if catalog.album(id).is_some()) { label } else { tr(&label).to_string() }
}
/// A date group heading (`2026-09-20`, `2026-09`, `2026`) in the UI language: the grid's full
/// form, or the date sidebar's `short` one ("Sunday, 20" / "September" / "2026"). English keeps the
/// catalog's own wording; every other language formats it with its `*-formats.json` date
/// patterns and weekday names. The grouping keys themselves never change.
pub fn date_group_label(key: &str, short: bool) -> String {
let label = lightcraft_catalog::dates::group_label(key);
if language() == Locale::En {
return if short {
match key.len() {
// "September 2026" → "September"
7 => label.split(' ').next().unwrap_or(&label).to_string(),
// "Sunday, 20 September 2026" → "Sunday, 20"
10 => label.rsplitn(3, ' ').nth(2).unwrap_or(&label).to_string(),
_ => label,
}
} else {
label
};
}
let Some(year) = key.get(..4).and_then(|y| y.parse::<u32>().ok()) else { return tr(&label).to_string() };
if key.len() == 4 {
return tr_format!("{year}", year = year);
}
let Some(month) = key.get(5..7).and_then(|m| m.parse::<u32>().ok()).filter(|m| (1..=12).contains(m)) else {
return tr(&label).to_string();
};
if key.len() == 7 {
return if short { tr_format!("{month}", month = month) } else { tr_format!("{month} {year}", month = month, year = year) };
}
if key.len() == 10
&& let Some(weekday) = lightcraft_catalog::dates::weekday(key)
&& let Some(day) = key.get(8..10).and_then(|d| d.parse::<u32>().ok())
{
let weekday = tr(weekday);
return if short {
tr_format!("{weekday}, {day}", weekday = weekday, day = day)
} else {
tr_format!("{weekday}, {day} {month} {year}", weekday = weekday, day = day, month = month, year = year)
};
}
tr(&label).to_string()
}
/// A capture time for display in the UI language (English: "March 30, 2022 at 10:11:11 PM");
/// the metadata itself is never rewritten, and a value that doesn't parse is shown as it is.
pub fn display_time(iso: &str) -> String {
if language() == Locale::En || lightcraft_catalog::dates::normalize_iso(iso).is_none() {
return lightcraft_catalog::dates::display_time(iso);
}
let date = date_group_label(iso.get(..10).unwrap_or(iso), false);
match iso.get(11..).filter(|time| !time.is_empty()) {
Some(time) => format!("{date} {time}"),
None => date,
}
}
#[cfg(test)]
@@ -71,53 +276,239 @@ mod tests {
use super::*;
#[test]
fn catalog_is_valid_and_contains_core_workflows() {
let messages: BTreeMap<String, String> = serde_json::from_str(include_str!("../locales/ja.json")).unwrap();
for key in ["Import Photos…", "Export…", "Exposure", "White Balance", "Settings", "Language"] {
assert!(messages.get(key).is_some_and(|value| !value.is_empty() && value != key), "{key}");
fn every_language_has_a_catalog_covering_the_source() {
assert!(crate::i18n::Locale::En.catalog().is_empty(), "the source language needs no catalog");
for language in Locale::ALL {
assert!(!language.name().is_empty(), "{language:?} has a name");
assert!(!language.script().is_empty(), "{language:?} has a script");
assert_eq!(Locale::parse(language.code()), Some(*language), "{language:?} parses its own code");
if *language == Locale::En {
continue;
}
let messages = language.catalog();
assert!(!messages.is_empty(), "{language:?}: {} has no messages", language.code());
for (key, value) in messages {
assert!(!value.is_empty(), "{language:?}: empty translation for {key:?}");
}
}
}
/// Where two catalogs translate the same text they agree on its placeholders. Catalogs may cover
/// different sets of messages (a message a language lacks shows in English): the gaps are
/// reported, not failed, so a translation can grow at its own pace.
#[test]
fn language_switches_and_unknown_text_survives() {
set_language(Language::Ja);
assert_eq!(tr("Exposure"), "露出");
assert_eq!(tr("my-photo.jpg"), "my-photo.jpg");
assert_eq!(tr("develop.set"), "develop.set");
assert_eq!(crate::menubar::display_item_label("album.addPhotos", &serde_json::json!({"id": 1}), "Color"), "Color");
assert_eq!(crate::menubar::display_item_label("app.export", &serde_json::json!({"preset": "Color"}), "Color"), "Color");
assert_eq!(crate::menubar::display_item_label("view.photoGrid", &serde_json::Value::Null, "Color"), "カラー");
set_language(Language::En);
assert_eq!(tr("Exposure"), "Exposure");
fn catalogs_agree_on_placeholders_and_report_gaps() {
let fields = |text: &str| {
let mut out: Vec<String> = Vec::new();
let mut rest = text;
while let Some(start) = rest.find('{') {
let Some(end) = rest[start..].find('}') else { break };
out.push(rest[start + 1..start + end].split(':').next().unwrap_or("").to_string());
rest = &rest[start + end + 1..];
}
out.sort();
out
};
let translated: Vec<Locale> = Locale::ALL.iter().copied().filter(|language| *language != Locale::En).collect();
let all: std::collections::BTreeSet<&String> = translated.iter().flat_map(|language| language.catalog().keys()).collect();
for language in &translated {
let catalog = language.catalog();
for (key, value) in catalog {
assert_eq!(fields(key), fields(value), "{language:?}: {key:?} -> {value:?}");
}
let missing: Vec<&&String> = all.iter().filter(|key| !catalog.contains_key(key.as_str())).collect();
if !missing.is_empty() {
eprintln!("{} lacks {} message(s) another language translates (shown in English): {missing:?}", language.code(), missing.len());
}
}
}
/// Locale tags are matched leniently, so a system locale reaches a shipped language.
#[test]
fn locale_tags_normalize() {
assert_eq!(Locale::parse_tag("en"), Some(Locale::En));
assert_eq!(Locale::parse_tag("en-US"), Some(Locale::En));
assert_eq!(Locale::parse_tag("zh-hans"), Some(Locale::ZhHans));
assert_eq!(Locale::parse_tag("zh"), Some(Locale::ZhHans));
assert_eq!(Locale::parse_tag("zh_CN"), Some(Locale::ZhHans));
assert_eq!(Locale::parse_tag("zh-hant"), Some(Locale::ZhHant));
assert_eq!(Locale::parse_tag("zh-Hant"), Some(Locale::ZhHant));
assert_eq!(Locale::parse_tag("zh-TW"), Some(Locale::ZhHant));
assert_eq!(Locale::parse_tag("zh_TW.UTF-8"), Some(Locale::ZhHant));
assert_eq!(Locale::parse_tag("zh-HK"), Some(Locale::ZhHant));
assert_eq!(Locale::parse_tag("zh-Hant-TW"), Some(Locale::ZhHant));
assert_eq!(Locale::parse_tag("zh-CN"), Some(Locale::ZhHans));
assert_eq!(Locale::parse_tag("ja_JP.UTF-8"), Some(Locale::Ja));
assert_eq!(Locale::parse_tag("de"), None);
}
#[test]
fn translated_formats_preserve_counts_and_remove_english_plural_suffixes() {
set_language(Language::Ja);
assert_eq!(tr_format!("{n} photo{}", "s", n = 12), "12枚");
fn switching_language_translates_and_restores() {
set_language(Locale::ZhHans);
assert_eq!(tr("Exposure"), "曝光");
assert_eq!(tr("Settings"), "设置");
// Untranslated and data-like text passes through untouched.
assert_eq!(tr("my-photo.jpg"), "my-photo.jpg");
assert_eq!(tr("develop.set"), "develop.set");
assert_eq!(tr("A string no catalog has"), "A string no catalog has");
set_language(Locale::Ja);
assert_eq!(tr("Exposure"), "露出");
set_language(Locale::En);
assert_eq!(tr("Exposure"), "Exposure");
}
/// Format strings carry their values in every language, and a language without a translation
/// falls back to the English format rather than dropping the values.
#[test]
fn formats_render_in_every_language() {
// Every language's format string is checked by `format!` against the call site's values.
set_language(Locale::En);
assert_eq!(tr_format!("Imported {} photo{}", 12, "s"), "Imported 12 photos");
assert_eq!(tr_format!("Exported {ok} of {total} photo{}", "s", ok = 4, total = 12), "Exported 4 of 12 photos");
set_language(Locale::ZhHans);
assert_eq!(tr_format!("Imported {} photo{}", 12, "s"), "已导入 12 张照片");
assert_eq!(tr_format!("Exported {ok} of {total} photo{}", "s", ok = 4, total = 12), "已导出 12 张中的 4 张照片");
set_language(Locale::Ja);
assert_eq!(tr_format!("Imported {} photo{}", 12, "s"), "12枚を読み込みました");
assert_eq!(tr_format!("Exported {ok} of {total} photo{}", "s", ok = 4, total = 12), "12枚中4枚を書き出しました");
set_language(Language::En);
assert_eq!(tr_format!("{n} photo{}", "s", n = 12), "12 photos");
// Format specs survive translation: precision, sign, and values a language reorders.
for language in Locale::ALL {
set_language(*language);
let text = tr_format!("{count} smart preview{} · {:.1} MB", "s", 12.345, count = 3);
assert!(text.contains("12.3") && !text.contains("12.34"), "{language:?}: {text}");
let merging = tr_format!("Merging… {stage} {:.0}%", 42.4, stage = "Aligning");
assert!(merging.contains("42%"), "{language:?}: {merging}");
let delta = tr_format!("{label} {d:+} on every selected photo", label = "Exposure", d = 5);
assert!(delta.contains("+5"), "{language:?}: {delta}");
let reading = tr_format!("Reading photos… {} of {}", 3, 10);
let (three, ten) = (reading.find('3'), reading.find("10"));
assert!(three.is_some() && ten.is_some(), "{language:?}: {reading}");
}
set_language(Locale::En);
assert_eq!(tr_format!("Reading photos… {} of {}", 3, 10), "Reading photos… 3 of 10");
}
/// Every character a catalog paints has a glyph in the fonts the UI installs, so a translation
/// never shows as boxes. Without craft-fonts there are no CJK faces at all, and the CJK
/// catalogs are skipped (the same degradation the Japanese UI has always had).
#[test]
fn catalog_characters_are_paintable() {
let ctx = egui::Context::default();
ctx.set_fonts(crate::theme::font_definitions(lightcraft_engine::CRAFT_FONTS));
let mut out = ctx.run_ui(egui::RawInput::default(), |_| {});
out.textures_delta.clear();
for language in Locale::ALL {
let messages = language.catalog();
if messages.is_empty() {
continue;
}
// The faces this language's script needs; a language the craft-fonts input does not
// cover (a translation added ahead of its font) is reported, not failed.
let mut faces: Vec<&str> =
lightcraft_engine::CRAFT_FONTS.iter().filter(|font| font.serves(language.script())).map(|font| font.family).collect();
faces.dedup();
if faces.is_empty() {
eprintln!("skipped {}: built without a craft-fonts face for {}", language.code(), language.script());
continue;
}
ctx.fonts_mut(|fonts| {
for family in [egui::FontFamily::Proportional, egui::FontFamily::Name(crate::theme::FONT_SEMIBOLD.into())] {
let font = egui::FontId::new(13.0, family.clone());
let mut missing: Vec<char> = messages
.values()
.flat_map(|message| message.chars())
.filter(|ch| !ch.is_whitespace() && !fonts.has_glyph(&font, *ch))
.collect();
missing.sort_unstable();
missing.dedup();
assert!(
missing.is_empty(),
"{} in {family:?} (faces: {faces:?}) lack glyphs: {}",
language.code(),
missing.iter().collect::<String>()
);
}
});
}
}
/// The interface's own symbols paint in every language: the CJK faces stop at Han and kana, so a
/// translation must not carry a symbol no installed font has (egui's defaults cover the rest).
#[test]
fn interface_symbols_are_paintable() {
let ctx = egui::Context::default();
ctx.set_fonts(crate::theme::font_definitions(lightcraft_engine::CRAFT_FONTS));
let mut out = ctx.run_ui(egui::RawInput::default(), |_| {});
out.textures_delta.clear();
// ⌥ and ⌫ come from egui's bundled fonts; ▸ and ▾ are in neither (measured 2026-10-07).
ctx.fonts_mut(|fonts| {
for symbol in ['⌥', '⌫', '⇧', '⌘', '→', '…', '—', '“', '”', '·', '»', '⌄', '★'] {
let font = egui::FontId::new(13.0, egui::FontFamily::Proportional);
assert!(fonts.has_glyph(&font, symbol), "{symbol:?} has no glyph in any installed font");
}
});
}
/// The language menu covers every language, and a language's own command selects it.
#[test]
fn language_commands_cover_every_language() {
let commands = [
("app.language.traditionalChinese", Locale::ZhHant),
("app.language.english", Locale::En),
("app.language.simplifiedChinese", Locale::ZhHans),
("app.language.japanese", Locale::Ja),
];
// One command per language, and every command reachable from the menu table.
assert_eq!(commands.len(), Locale::ALL.len());
for (id, language) in commands {
assert_eq!(crate::menus::language_from_command(id), Some(language), "{id}");
assert!(crate::menus::ui_commands().any(|command| command.0 == id), "{id} is not in the menu");
}
assert_eq!(crate::menus::language_from_command("app.language.klingon"), None);
assert_eq!(crate::menus::language_from_command("view.detail"), None);
}
#[test]
fn preferences_round_trip_and_old_settings_remain_readable() {
let old: crate::state::UiState = serde_json::from_str("{}").unwrap();
assert_eq!(old.language, Language::En);
let settings = crate::state::UiState { language: Language::Ja, ..old };
assert_eq!(old.language, Locale::En);
// Settings written before the language list grew still load.
let legacy: crate::state::UiState = serde_json::from_str(r#"{"language":"ja"}"#).unwrap();
assert_eq!(legacy.language, Locale::Ja);
let settings = crate::state::UiState { language: Locale::ZhHans, ..old };
let saved = serde_json::to_string(&settings).unwrap();
let restored: crate::state::UiState = serde_json::from_str(&saved).unwrap();
assert_eq!(restored.language, Language::Ja);
assert_eq!(restored.language, Locale::ZhHans);
}
#[test]
fn japanese_is_painted_and_both_font_weights_cover_the_catalog() {
let ctx = egui::Context::default();
crate::theme::install_fonts(&ctx);
let mut app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
app.ui.language = Language::Ja;
app.ui.left_panel = true;
let mut text = String::new();
fn catalogs_contain_core_workflows() {
for language in Locale::ALL.iter().filter(|language| **language != Locale::En) {
for key in ["Import Photos…", "Export…", "Exposure", "White Balance", "Settings", "Language"] {
let value = language.tr(key);
assert!(!value.is_empty() && value != key, "{language:?}: {key}");
}
}
}
/// User-named menu items (presets, albums, label sets) are never translated; built-in labels are.
#[test]
fn menu_labels_translate_but_user_names_survive() {
for language in Locale::ALL {
set_language(*language);
assert_eq!(crate::menubar::display_item_label("album.addPhotos", &serde_json::json!({"id": 1}), "Color"), "Color");
assert_eq!(crate::menubar::display_item_label("app.export", &serde_json::json!({"preset": "Color"}), "Color"), "Color");
assert_eq!(crate::menubar::display_item_label("view.photoGrid", &serde_json::Value::Null, "Color"), language.tr("Color"));
}
set_language(Locale::ZhHans);
assert_eq!(crate::menubar::display_item_label("view.photoGrid", &serde_json::Value::Null, "Color"), "颜色");
set_language(Locale::Ja);
assert_eq!(crate::menubar::display_item_label("view.photoGrid", &serde_json::Value::Null, "Color"), "カラー");
set_language(Locale::En);
}
/// The text the whole window paints over a few frames in `language`.
fn painted_text(ctx: &egui::Context, app: &mut crate::LightcraftApp, language: Locale) -> String {
fn collect(shape: &egui::epaint::Shape, text: &mut String) {
match shape {
egui::epaint::Shape::Text(shape) => {
@@ -128,41 +519,49 @@ mod tests {
_ => {}
}
}
app.ui.language = language;
let mut text = String::new();
for frame in 0..4 {
let input = crate::headless::HeadlessView::raw_input(egui::vec2(1600.0, 1000.0), 1.0, frame as f64 / 60.0, vec![]);
let mut out = ctx.run_ui(input, |ui| {
app.logic(ui.ctx());
app.ui(ui);
});
// This assertion inspects shapes without a renderer; discard texture uploads explicitly.
// This inspects shapes without a renderer; discard texture uploads explicitly.
out.textures_delta.clear();
text.clear();
for shape in out.shapes {
collect(&shape.shape, &mut text);
}
}
assert!(text.contains("マイフォト"), "{text}");
assert!(text.contains("すべての写真"), "{text}");
if lightcraft_engine::fonts::japanese(lightcraft_engine::CRAFT_FONTS).next().is_none() {
eprintln!("skipped glyph coverage: built without CRAFT_FONTS_DIR, so there is no Japanese UI font");
} else {
ctx.fonts_mut(|fonts| {
for family in [egui::FontFamily::Proportional, egui::FontFamily::Name(crate::theme::FONT_SEMIBOLD.into())] {
let font = egui::FontId::new(13.0, family);
for message in japanese().values() {
for ch in message.chars().filter(|ch| !ch.is_whitespace()) {
assert!(fonts.has_glyph(&font, ch), "Missing glyph {ch} in {message}");
}
}
}
});
}
// Locale affects presentation only: command ids remain the same.
let ids = |app: &crate::LightcraftApp| crate::menus::menu_entries(app).into_iter().map(|entry| entry.id).collect::<Vec<_>>();
let japanese_ids = ids(&app);
set_language(Language::En);
assert_eq!(japanese_ids, ids(&app));
text
}
/// Every language is painted by the real window, switching at runtime reinstalls the fonts
/// (their CJK fallback order follows the language), and command ids never change with it.
#[test]
fn every_language_is_painted_and_menu_ids_stay_the_same() {
let ctx = egui::Context::default();
crate::theme::install_fonts(&ctx);
let mut app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
app.ui.left_panel = true;
let ids = |app: &crate::LightcraftApp| crate::menus::menu_entries(app).into_iter().map(|entry| entry.id).collect::<Vec<_>>();
let english_ids = ids(&app);
for language in Locale::ALL {
let text = painted_text(&ctx, &mut app, *language);
for label in ["My Photos", "All Photos"] {
assert!(text.contains(language.tr(label)), "{language:?}: {label} -> {:?} not in\n{text}", language.tr(label));
}
assert_eq!(app.font_language, *language, "the fonts follow the language");
assert_eq!(ids(&app), english_ids, "{language:?}: command ids are presentation-independent");
}
let text = painted_text(&ctx, &mut app, Locale::Ja);
assert!(text.contains("マイフォト") && text.contains("すべての写真"), "{text}");
let text = painted_text(&ctx, &mut app, Locale::ZhHans);
assert!(text.contains("我的照片") && text.contains("所有照片"), "{text}");
set_language(Locale::En);
}
/// The UI families, after one frame so the font definitions are loaded.
fn fonts_ctx(craft: &'static [lightcraft_engine::CraftFont]) -> egui::Context {
let ctx = egui::Context::default();
@@ -172,31 +571,36 @@ mod tests {
ctx
}
fn ui_families() -> [egui::FontFamily; 3] {
[egui::FontFamily::Proportional, egui::FontFamily::Name(crate::theme::FONT_SEMIBOLD.into()), egui::FontFamily::Monospace]
}
/// Built with craft-fonts, Japanese renders with real glyphs (no tofu) in every UI family.
/// Built with craft-fonts, every CJK language renders with real glyphs (no tofu) in every UI
/// family, Monospace included.
#[test]
fn craft_fonts_render_japanese_in_the_ui() {
if lightcraft_engine::fonts::japanese(lightcraft_engine::CRAFT_FONTS).next().is_none() {
eprintln!("skipped: built without CRAFT_FONTS_DIR, so there is no Japanese UI font");
return;
}
let ctx = fonts_ctx(lightcraft_engine::CRAFT_FONTS);
ctx.fonts_mut(|fonts| {
for family in ui_families() {
let font = egui::FontId::new(13.0, family);
for ch in "日本語の文字".chars() {
assert!(fonts.has_glyph(&font, ch), "{ch} in {font:?}");
}
let galley = fonts.layout_no_wrap("日本語の文字".into(), font.clone(), egui::Color32::WHITE);
assert!(galley.size().x > 13.0 * 5.0, "{font:?}: six full-width glyphs, {:?}", galley.size());
fn craft_fonts_render_cjk_in_the_ui() {
let samples = [(Locale::Ja, "日本語の文字"), (Locale::ZhHans, "简体中文字"), (Locale::ZhHant, "繁體中文字")];
for (language, sample) in samples {
if !lightcraft_engine::CRAFT_FONTS.iter().any(|font| font.serves(language.script())) {
eprintln!("skipped {}: built without a craft-fonts face for {}", language.code(), language.script());
continue;
}
});
set_language(language);
let ctx = fonts_ctx(lightcraft_engine::CRAFT_FONTS);
ctx.fonts_mut(|fonts| {
for family in
[egui::FontFamily::Proportional, egui::FontFamily::Name(crate::theme::FONT_SEMIBOLD.into()), egui::FontFamily::Monospace]
{
let font = egui::FontId::new(13.0, family);
for ch in sample.chars() {
assert!(fonts.has_glyph(&font, ch), "{language:?}: {ch} in {font:?}");
}
let galley = fonts.layout_no_wrap(sample.into(), font.clone(), egui::Color32::WHITE);
let wide = 13.0 * (sample.chars().count() as f32 - 1.0);
assert!(galley.size().x > wide, "{language:?} {font:?}: full-width glyphs, {:?}", galley.size());
}
});
}
set_language(Locale::En);
}
/// Built without craft-fonts, the UI (in Japanese, too) still installs its fonts and runs;
/// Built without craft-fonts, the UI (in every language) still installs its fonts and runs;
/// Latin text keeps Inter.
#[test]
fn the_ui_works_without_craft_fonts() {
@@ -210,15 +614,117 @@ mod tests {
}
});
let mut app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
app.ui.language = Language::Ja;
for frame in 0..3 {
let input = crate::headless::HeadlessView::raw_input(egui::vec2(1200.0, 800.0), 1.0, frame as f64 / 60.0, vec![]);
let mut out = ctx.run_ui(input, |ui| {
app.logic(ui.ctx());
app.ui(ui);
});
out.textures_delta.clear();
for language in Locale::ALL {
app.ui.language = *language;
for frame in 0..3 {
let input = crate::headless::HeadlessView::raw_input(egui::vec2(1200.0, 800.0), 1.0, frame as f64 / 60.0, vec![]);
let mut out = ctx.run_ui(input, |ui| {
app.logic(ui.ctx());
app.ui(ui);
});
out.textures_delta.clear();
}
}
set_language(Language::En);
set_language(Locale::En);
}
/// Traditional Chinese through the menu command and the control channel, persisted, with
/// localised headings and dates; ids and user text are untouched.
#[test]
fn traditional_chinese_switches_through_menu_and_control_and_persists() {
use crate::control::{ControlRequest, Outcome};
let mut app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
app.run("app.language.traditionalChinese", serde_json::json!({})).unwrap();
assert_eq!(app.ui.language, Locale::ZhHant);
// Applied at once, not on the next frame.
assert_eq!(tr("File"), "檔案");
assert_eq!(source_label(lightcraft_engine::LibrarySource::All, &app.session.catalog), "所有照片");
assert_eq!(date_group_label("2026-09-20", false), "2026年9月20日 星期日");
assert_eq!(date_group_label("2026-09-20", true), "20日 星期日");
assert_eq!(date_group_label("2026-09", false), "2026年9月");
assert_eq!(date_group_label("2026-09", true), "9月");
assert_eq!(date_group_label("2026", false), "2026年");
assert_eq!(date_group_label("", false), "未知日期");
assert_eq!(display_time("2026-09-20T16:04:05"), "2026年9月20日 星期日 16:04:05");
assert_eq!(display_time("a-user-value"), "a-user-value");
assert_eq!(crate::menubar::checked(&app, "app.language.traditionalChinese"), Some(true));
assert_eq!(crate::menubar::checked(&app, "app.language.english"), Some(false));
assert_eq!(tr("my-photo.jpg"), "my-photo.jpg");
for id in ["album.addPhotos", "metadata.applyPreset", "label.applySet"] {
assert_eq!(crate::menubar::display_item_label(id, &serde_json::json!({"id": 1}), "Color"), "Color");
}
assert_eq!(crate::menubar::display_item_label("view.photoGrid", &serde_json::Value::Null, "Color"), "色彩");
assert_eq!(tr_format!("{n} photo{}", "s", n = 12), "12 張照片");
assert_eq!(tr_format!("Exported {ok} of {total} photo{}", "s", ok = 4, total = 12), "已匯出 4/12 張照片");
let saved = serde_json::to_string(&app.ui).unwrap();
assert!(saved.contains(r#""language":"zh-hant""#), "{saved}");
assert_eq!(serde_json::from_str::<crate::state::UiState>(&saved).unwrap().language, Locale::ZhHant);
let ctx = egui::Context::default();
app.ui.language = Locale::En;
for code in ["zh-hant", "zh-Hant", "zh-TW", "zh_TW", "zh-HK"] {
let (req, _) = ControlRequest::new("ui.set", serde_json::json!({"language": code}));
let Outcome::Done(reply) = crate::control::handle(&mut app, &ctx, &req) else { panic!("expected a reply") };
assert_eq!(reply["ok"], true, "{code}");
assert_eq!(app.ui.language, Locale::ZhHant, "{code}");
assert_eq!(language(), Locale::ZhHant, "{code} applies at once");
}
let (req, _) = ControlRequest::new("ui.set", serde_json::json!({"language": "xx"}));
let Outcome::Done(reply) = crate::control::handle(&mut app, &ctx, &req) else { panic!("expected a reply") };
assert_eq!(reply["ok"], false);
assert_eq!(app.ui.language, Locale::ZhHant);
let ids = |app: &crate::LightcraftApp| crate::menus::menu_entries(app).into_iter().map(|entry| entry.id).collect::<Vec<_>>();
let chinese_ids = ids(&app);
app.run("app.language.english", serde_json::json!({})).unwrap();
assert_eq!(tr("File"), "File");
assert_eq!(chinese_ids, ids(&app));
}
/// Dates follow every language's own patterns; English keeps the catalog's wording.
#[test]
fn date_headings_in_every_language() {
set_language(Locale::En);
assert_eq!(date_group_label("2026-09-20", false), "Sunday, 20 September 2026");
assert_eq!(date_group_label("2026-09-20", true), "Sunday, 20");
assert_eq!(date_group_label("2026-09", true), "September");
assert_eq!(display_time("2026-09-20T16:04:05"), lightcraft_catalog::dates::display_time("2026-09-20T16:04:05"));
set_language(Locale::Ja);
assert_eq!(date_group_label("2026-09-20", false), "2026年9月20日(日曜日)");
assert_eq!(date_group_label("2026-09", true), "9月");
set_language(Locale::ZhHans);
assert_eq!(date_group_label("2026-09-20", false), "2026年9月20日 星期日");
for language in Locale::ALL {
set_language(*language);
// A key that is not a date never panics and never shows a raw pattern.
for key in ["", "abcd", "2026-xx", "2026-13", "2026-09-xx", "2026-09"] {
assert!(!date_group_label(key, false).contains('{'), "{language:?}: {key:?}");
assert!(!date_group_label(key, true).contains('{'), "{language:?}: {key:?}");
}
}
set_language(Locale::En);
}
/// Stock presets and panel headers are translated; a user preset keeps its name, also when it
/// shares a stock preset's name.
#[test]
fn traditional_chinese_presets_and_panel_headers_are_painted() {
let ctx = egui::Context::default();
crate::theme::install_fonts(&ctx);
let mut app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
app.ui.presets = true;
let builtin = app.session.presets.iter().find(|p| p.name == "Warm Glow").unwrap().clone();
app.session.presets.push(lightcraft_develop::Preset { id: "user.test".into(), builtin: false, ..builtin });
for (panel, title) in [(crate::state::RightPanel::Activity, "歷史紀錄"), (crate::state::RightPanel::Versions, "版本")] {
app.ui.right = panel;
let text = painted_text(&ctx, &mut app, Locale::ZhHant);
assert!(text.contains(title), "{text}");
assert!(!text.contains("History") && !text.contains("Versions"), "{text}");
assert!(text.contains("暖光"), "stock preset: {text}");
assert!(text.contains("Warm Glow"), "user preset: {text}");
assert!(text.contains("色彩"), "stock group: {text}");
}
assert_eq!(history_label("Exposure", &app.session.presets), "曝光");
assert_eq!(history_label("Preset: Warm Film", &app.session.presets), "預設集:暖調底片");
assert_eq!(history_label("Preset: Warm Glow", &app.session.presets), "預設集:Warm Glow");
set_language(Locale::En);
}
}
+10
View File
@@ -78,6 +78,8 @@ pub enum Icon {
/// Community chat (opens the ArtCraft Discord): a speech bubble with three dots. Our own
/// generic drawing, not any service's logo.
Chat,
/// Face boxes on/off: four corner brackets round a small face.
FaceBox,
}
struct Pen<'a> {
@@ -404,6 +406,14 @@ pub fn paint(p: &Painter, rect: Rect, icon: Icon, color: Color32) {
pen.arc(10.0, 18.0, 6.5, 200.0, 340.0);
pen.rect(2.5, 2.5, 17.5, 17.5, 2.0);
}
FaceBox => {
pen.line(&[(2.5, 7.0), (2.5, 2.5), (7.0, 2.5)]);
pen.line(&[(13.0, 2.5), (17.5, 2.5), (17.5, 7.0)]);
pen.line(&[(2.5, 13.0), (2.5, 17.5), (7.0, 17.5)]);
pen.line(&[(13.0, 17.5), (17.5, 17.5), (17.5, 13.0)]);
pen.circle(10.0, 8.5, 2.4);
pen.arc(10.0, 16.0, 4.5, 215.0, 325.0);
}
Picker => {
pen.line(&[(4.0, 16.0), (11.5, 8.5)]);
pen.closed(&[(11.0, 5.0), (15.0, 9.0), (17.0, 7.0), (13.0, 3.0)]);
+113 -2
View File
@@ -1,7 +1,7 @@
//! LightCraft's egui frontend: a Lightroom-style UI over `lightcraft-engine`.
//!
//! The UI is thin: every action goes through [`LightcraftApp::run`], which handles UI commands
//! (views, panels, zoom — see [`menus::UI_COMMANDS`]) and forwards everything else to the engine.
//! (views, panels, zoom — see [`menus::ui_commands`]) and forwards everything else to the engine.
//! The same entry point serves menus, shortcuts, buttons and the control channel ([`control`]).
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -157,6 +157,9 @@ pub struct LightcraftApp {
/// Clear `synthetic_mods` on the next frame.
synthetic_mods_release: bool,
styled: bool,
/// The language the installed fonts were built for: the CJK fallback order follows the UI
/// language's script, so switching language reinstalls them.
font_language: i18n::Locale,
fonts_ready: bool,
last_time: f64,
/// Rect of the photo canvas and the displayed image (screen points) from the last frame.
@@ -200,7 +203,9 @@ pub struct LightcraftApp {
}
impl LightcraftApp {
pub fn new(session: Session, services: Services) -> Self {
pub fn new(mut session: Session, services: Services) -> Self {
// AI mask requests run on the model's worker; frames apply their results (never wait)
session.segmenter.background = true;
Self {
session,
ui: UiState::default(),
@@ -223,6 +228,7 @@ impl LightcraftApp {
synthetic_mods: egui::Modifiers::NONE,
synthetic_mods_release: false,
styled: false,
font_language: i18n::Locale::En,
fonts_ready: false,
last_time: 0.0,
canvas_rect: None,
@@ -260,6 +266,7 @@ impl LightcraftApp {
}
let r = self.session.execute(id, &params).map_err(|e| e.to_string());
if let Err(e) = &r {
log::warn!("{id}: {e}");
self.ui.status = e.clone();
}
r
@@ -353,6 +360,84 @@ impl LightcraftApp {
self.ui.toast = Some((text.into(), t + secs));
}
/// AI masks: apply finished background requests (clicks, descriptions, detail passes) and
/// show their errors; start a zoomed-in detail pass once the clicking stops
/// (`ui.detail_due`); watch the model download. Never waits for the model.
fn ai_mask_detail(&mut self, ctx: &egui::Context) {
let polled = self.session.segment_poll();
let now = ctx.input(|i| i.time);
if polled.changed {
ctx.request_repaint();
}
if let Some(mask) = polled.refine {
self.ui.detail_due = Some((now + 0.3, mask));
}
if let Some(e) = polled.messages.into_iter().last() {
self.ai_error(ctx, e, None);
}
let seg = &self.session.segmenter;
if seg.busy() || seg.pending_clicks().is_some() {
ctx.request_repaint_after(std::time::Duration::from_millis(50));
}
let download = seg.download_status();
if download.running {
ctx.request_repaint_after(std::time::Duration::from_millis(250));
}
// a download finishing while its dialog is closed: say so once
if self.ui.sam_downloading && !download.running {
self.ui.sam_downloading = false;
let open = matches!(self.ui.dialog, Some(state::Dialog::SamModel { .. }));
match (&download.error, open) {
(_, true) => {}
(Some(e), false) if e.contains("cancelled") => {
self.toast(ctx, crate::i18n::tr("SAM 3 download stopped: it resumes where it left off next time."))
}
(Some(e), false) => self.toast_error(ctx, format!("The SAM 3 download failed: {e}")),
(None, false) if download.finished => {
self.toast_error(ctx, crate::i18n::tr("The SAM 3 model is installed: Object and Describe masks are ready."))
}
(None, false) => {}
}
}
if let Some((due, mask)) = self.ui.detail_due {
if now >= due && !self.session.segmenter.busy() && !self.session.segmenter.detail_busy() {
self.ui.detail_due = None;
if let Err(e) = self.run("mask.refineDetail", serde_json::json!({"id": mask})) {
log::warn!("detail pass: {e}");
}
} else {
ctx.request_repaint_after(std::time::Duration::from_millis(150));
}
}
if self.session.segmenter.detail_busy() {
ctx.request_repaint_after(std::time::Duration::from_millis(200));
}
}
/// An AI mask error: when the model isn't installed, the dialog that offers to download it
/// (`then`: the AI mask to start afterwards); otherwise a toast.
pub fn ai_error(&mut self, ctx: &egui::Context, e: impl Into<String>, then: Option<(&str, &str)>) {
let e = e.into();
if lightcraft_engine::segment::Segmenter::AVAILABLE && e.starts_with(lightcraft_engine::segment::NOT_INSTALLED) {
self.offer_sam_download(then);
} else {
self.toast_error(ctx, e);
}
}
/// Open the dialog offering the SAM 3 download (never downloads by itself).
pub fn offer_sam_download(&mut self, then: Option<(&str, &str)>) {
if self.ui.dialog.is_none() || matches!(self.ui.dialog, Some(state::Dialog::SamModel { .. })) {
self.ui.dialog = Some(state::Dialog::SamModel { then: then.map(|(k, o)| (k.to_string(), o.to_string())), error: None });
}
}
/// A toast for an error the user has to read and act on (stays 6 s).
pub fn toast_error(&mut self, ctx: &egui::Context, text: impl Into<String>) {
let t = ctx.input(|i| i.time);
self.ui.toast = Some((text.into(), t + 6.0));
}
fn drain_control(&mut self, ctx: &egui::Context) {
let Some(rx) = self.control_rx.take() else { return };
while let Ok(req) = rx.try_recv() {
@@ -502,6 +587,12 @@ impl LightcraftApp {
let repaint = ctx.clone();
self.session.media.availability.run_in_background(std::sync::Arc::new(move || repaint.request_repaint()));
self.styled = true;
self.font_language = self.ui.language;
} else if self.font_language != self.ui.language {
// Shared Han characters take the active language's forms (Japanese faces for 日本語,
// the Simplified Chinese face for 简体中文): rebuild the fallback order.
theme::install_fonts(ctx);
self.font_language = self.ui.language;
} else {
self.fonts_ready = true;
}
@@ -678,6 +769,7 @@ impl LightcraftApp {
}
// panels set it again this frame while the pointer rests on a preset or profile
self.hover_preview = None;
self.ai_mask_detail(&ctx);
if self.ui.fullscreen {
// full-screen preview: the photo alone on black
egui::CentralPanel::default().frame(egui::Frame::NONE.fill(egui::Color32::BLACK)).show(ui, |ui| panels::detail::show(self, ui));
@@ -718,6 +810,7 @@ impl LightcraftApp {
state::ViewMode::Compare => panels::compare::show_compare(self, ui),
state::ViewMode::Survey => panels::compare::show_survey(self, ui),
state::ViewMode::Reference => panels::compare::show_reference(self, ui),
state::ViewMode::People => panels::people::show(self, ui),
});
panels::second::show(self, &ctx);
panels::notices::show(self, &ctx);
@@ -794,6 +887,7 @@ mod drop_tests {
#[derive(Default)]
pub struct Caches {
keyword_tree: Option<(u64, std::sync::Arc<Vec<lightcraft_catalog::KeywordNode>>)>,
people: Option<(u64, lightcraft_catalog::Filter, std::sync::Arc<Vec<lightcraft_catalog::Person>>)>,
suggestions: Option<(u64, std::sync::Arc<Vec<String>>)>,
counts: Option<(u64, LibraryCounts)>,
date_groups: Option<(u64, std::sync::Arc<Vec<lightcraft_catalog::DateGroup>>)>,
@@ -840,6 +934,23 @@ impl Caches {
}
}
}
/// The people named on faces among the photos the filter lets through (its own `person` aside),
/// with photo counts.
pub fn people(
&mut self,
cat: &lightcraft_catalog::Catalog,
filter: &lightcraft_catalog::Filter,
) -> std::sync::Arc<Vec<lightcraft_catalog::Person>> {
let key = lightcraft_catalog::Filter { person: None, ..filter.clone() };
match &self.people {
Some((r, f, t)) if *r == cat.revision && *f == key => t.clone(),
_ => {
let t = std::sync::Arc::new(cat.people_in(&key));
self.people = Some((cat.revision, key, t.clone()));
t
}
}
}
/// All Photos / Picks / Recently Deleted counts.
pub fn counts(&mut self, cat: &lightcraft_catalog::Catalog) -> LibraryCounts {
if let Some((r, c)) = self.counts
+7 -3
View File
@@ -1,5 +1,5 @@
//! The menu bar model: File, Edit, View, Photo, Window, Help, generated from the command registry
//! (engine commands with menu paths + [`crate::menus::UI_COMMANDS`]) with live labels, shortcuts,
//! (engine commands with menu paths + [`crate::menus::ui_commands`]) with live labels, shortcuts,
//! enabled and checked state.
//!
//! One model drives every menu surface: the native macOS menu bar (built by the desktop host), the
@@ -119,9 +119,11 @@ const LAYOUT: &[(&str, &[&str])] = &[
"view.detail",
"view.compare",
"view.survey",
"view.people",
"---",
"view.leftPanel",
"view.photoCounts",
"view.faceBoxes",
"view.filmstrip",
"view.histogram",
"view.navigator",
@@ -282,8 +284,8 @@ pub fn checked(app: &LightcraftApp, id: &str) -> Option<bool> {
let u = &app.ui;
let panel = |p: RightPanel| Some(u.right == p);
match id {
"app.language.english" => Some(u.language == crate::i18n::Language::En),
"app.language.japanese" => Some(u.language == crate::i18n::Language::Ja),
// Every language's command is checked when it is the active one.
_ if crate::menus::language_from_command(id).is_some() => Some(crate::menus::language_from_command(id) == Some(u.language)),
"develop.autoSync" => Some(app.session.auto_sync),
"view.photoCounts" => Some(u.show_counts),
"view.secondWindow" => Some(u.second_window),
@@ -292,8 +294,10 @@ pub fn checked(app: &LightcraftApp, id: &str) -> Option<bool> {
"view.detail" => Some(u.view == ViewMode::Detail),
"view.compare" => Some(u.view == ViewMode::Compare),
"view.survey" => Some(u.view == ViewMode::Survey),
"view.people" => Some(u.view == ViewMode::People),
"view.reference" => Some(u.view == ViewMode::Reference),
"view.leftPanel" => Some(u.left_panel),
"view.faceBoxes" => Some(u.face_boxes),
"view.filmstrip" => Some(u.filmstrip),
"view.histogram" => Some(u.histogram),
"view.clipping" => Some(u.show_clipping),
+43 -6
View File
@@ -10,9 +10,34 @@ use crate::state::{BeforeAfter, Dialog, RightPanel, ViewMode, Zoom};
/// (id, label, shortcut, menu path)
pub type UiCommand = (&'static str, &'static str, Option<&'static str>, &'static str);
/// The "Edit → Language" entries, one per language in [`crate::i18n::Locale::ALL`], so a language
/// added to the table shows up in the menu (and in the control channel's command list) by itself.
pub const LANGUAGE_COMMANDS: &[UiCommand] = &[
("app.language.english", crate::i18n::Locale::En.name(), None, "Edit>Language"),
("app.language.simplifiedChinese", crate::i18n::Locale::ZhHans.name(), None, "Edit>Language"),
("app.language.traditionalChinese", crate::i18n::Locale::ZhHant.name(), None, "Edit>Language"),
("app.language.japanese", crate::i18n::Locale::Ja.name(), None, "Edit>Language"),
];
/// Every UI command: the languages, then everything else. `xtask parity` reads both tables from
/// this file, so an id listed in `docs/parity.md` is checked wherever it is declared.
pub fn ui_commands() -> impl Iterator<Item = &'static UiCommand> {
LANGUAGE_COMMANDS.iter().chain(UI_COMMANDS)
}
/// The language a Language-menu command selects, if the id is one. The engine and the UI both go
/// through here, so the menu, the settings row and the control channel agree on the mapping.
pub fn language_from_command(id: &str) -> Option<crate::i18n::Locale> {
match id {
"app.language.english" => Some(crate::i18n::Locale::En),
"app.language.simplifiedChinese" => Some(crate::i18n::Locale::ZhHans),
"app.language.traditionalChinese" => Some(crate::i18n::Locale::ZhHant),
"app.language.japanese" => Some(crate::i18n::Locale::Ja),
_ => None,
}
}
pub const UI_COMMANDS: &[UiCommand] = &[
("app.language.english", "English", None, "Edit>Language"),
("app.language.japanese", "日本語", None, "Edit>Language"),
("view.photoGrid", "Photo Grid", None, "View"),
("view.squareGrid", "Square Grid", None, "View"),
// G: Photo Grid ↔ Square Grid (from other views: the photo grid)
@@ -21,6 +46,8 @@ pub const UI_COMMANDS: &[UiCommand] = &[
("view.detail", "Detail", Some("D"), "View"),
("view.compare", "Compare", Some("Shift+C"), "View"),
("view.survey", "Survey", Some("N"), "View"),
("view.people", "People", None, "View"),
("view.faceBoxes", "Face Boxes", None, "View"),
("view.reference", "Reference View", Some("Shift+R"), "View"),
("photo.setReference", "Set as Reference Photo", None, ""),
("compare.swap", "Swap Compare Photos", None, "View"),
@@ -206,8 +233,11 @@ pub fn parse_rgb(v: &Value) -> Option<[u8; 3]> {
/// Handle UI commands; `None` means "not a UI command — send it to the engine".
pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Result<Value, String>> {
if matches!(id, "app.language.english" | "app.language.japanese") {
app.ui.language = if id == "app.language.japanese" { crate::i18n::Language::Ja } else { crate::i18n::Language::En };
if let Some(language) = language_from_command(id) {
app.ui.language = language;
// Immediately, not on the next frame: the reply and anything else run this frame
// (menus rebuilt from it, toasts) are already in the new language.
crate::i18n::set_language(app.ui.language);
return Some(Ok(json!(app.ui.language)));
}
let ctx = egui::Context::default();
@@ -263,6 +293,14 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
Ok(Value::Null)
}
"view.compare" => crate::panels::compare::enter_compare(app),
"view.faceBoxes" => {
app.ui.face_boxes = p.get("show").and_then(Value::as_bool).unwrap_or(!app.ui.face_boxes);
Ok(json!({"show": app.ui.face_boxes}))
}
"view.people" => {
app.ui.view = ViewMode::People;
Ok(json!({"people": app.session.catalog.people().len()}))
}
"view.survey" => {
app.ui.view = ViewMode::Survey;
Ok(json!({"photos": crate::panels::compare::survey_photos(app).len()}))
@@ -1251,8 +1289,7 @@ pub struct MenuEntry {
/// The flattened menu model (UI commands + engine commands with menu paths).
pub fn menu_entries(app: &LightcraftApp) -> Vec<MenuEntry> {
let mut v: Vec<MenuEntry> = UI_COMMANDS
.iter()
let mut v: Vec<MenuEntry> = ui_commands()
.filter(|c| !c.3.is_empty())
.map(|(id, label, sc, m)| MenuEntry {
id: id.to_string(),
+6
View File
@@ -26,6 +26,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
("detail", Icon::Single, ViewMode::Detail, "Detail (D)"),
("compare", Icon::Compare, ViewMode::Compare, "Compare (Shift+C)"),
("survey", Icon::Survey, ViewMode::Survey, "Survey (N)"),
("people", Icon::Subject, ViewMode::People, "People"),
] {
if icon_button(ui, id, icon, vec2(32.0, 32.0), app.ui.view == mode, true, tip).clicked() {
let _ = app.run(&format!("view.{id}"), json!({}));
@@ -165,6 +166,11 @@ fn right_side(app: &mut LightcraftApp, ui: &mut egui::Ui, full: Rect) -> f32 {
{
let _ = app.run("view.beforeAfter", json!({}));
}
if app.ui.view == ViewMode::Detail
&& icon_button(&mut child, "faceBoxes", Icon::FaceBox, vec2(30.0, 30.0), app.ui.face_boxes, true, "Face boxes").clicked()
{
let _ = app.run("view.faceBoxes", json!({}));
}
if icon_button(&mut child, "filmstrip", Icon::Filmstrip, vec2(30.0, 30.0), app.ui.filmstrip, true, "Filmstrip (/)").clicked() {
let _ = app.run("view.filmstrip", json!({}));
}
+212
View File
@@ -334,6 +334,20 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
let map = CanvasMap::new(&frame, img_rect);
let resp = ui.interact(canvas, egui::Id::new("loupe"), Sense::click_and_drag());
info_overlay(app, &p, canvas, &photo);
if !fullscreen {
filter_pill(app, ui, canvas);
}
if app.ui.face_boxes {
match region_overlay(ui, &p, &map, &photo, d.orientation) {
Some(RegionEdit::Remove(index)) => {
let _ = app.run("photo.removeRegion", json!({"id": id.0, "index": index}));
}
Some(RegionEdit::Resize(index, r)) => {
let _ = app.run("photo.setRegion", json!({"id": id.0, "index": index, "rect": {"x0": r.x0, "y0": r.y0, "x1": r.x1, "y1": r.y1}}));
}
None => {}
}
}
// a fine grid while a transform (geometry) slider is dragged, to judge verticals
if app.ui.dragging_control.as_deref().is_some_and(|c| c.starts_with("geometry.")) {
let n = 12;
@@ -421,6 +435,163 @@ fn info_overlay(app: &LightcraftApp, p: &egui::Painter, canvas: Rect, photo: &li
register(p.ctx(), "canvas:infoOverlay", Rect::from_min_max(canvas.min, pos2(canvas.left() + 320.0, y)));
}
/// What the user did to a face box this frame.
enum RegionEdit {
/// The × was clicked.
Remove(usize),
/// A handle drag ended: the region's new box (normalized, upright frame).
Resize(usize, lightcraft_geom::Rect),
}
/// Handles of a box: (x, y) as fractions of its width and height, and the cursor they show.
const REGION_HANDLES: [(f32, f32, egui::CursorIcon); 8] = [
(0.0, 0.0, egui::CursorIcon::ResizeNwSe),
(0.5, 0.0, egui::CursorIcon::ResizeVertical),
(1.0, 0.0, egui::CursorIcon::ResizeNeSw),
(1.0, 0.5, egui::CursorIcon::ResizeHorizontal),
(1.0, 1.0, egui::CursorIcon::ResizeNwSe),
(0.5, 1.0, egui::CursorIcon::ResizeVertical),
(0.0, 1.0, egui::CursorIcon::ResizeNeSw),
(0.0, 0.5, egui::CursorIcon::ResizeHorizontal),
];
/// Face/pet/focus regions read from XMP (MWG-RS), drawn as boxes over the photo. Hovering a box shows
/// a × in its corner and eight resize handles. A drag previews the new box live and is reported once,
/// on release (one undo step); the × reports the region to remove. Both are catalog-only edits:
/// LightCraft doesn't write regions to XMP. Regions are stored on the upright (EXIF-oriented) photo;
/// `orient` is the user's Rotate / Flip on top of it, which the loupe's normalized frame includes.
fn region_overlay(
ui: &egui::Ui,
p: &egui::Painter,
map: &CanvasMap,
photo: &lightcraft_catalog::Photo,
orient: lightcraft_geom::Orientation,
) -> Option<RegionEdit> {
let t = Tokens::get(p.ctx());
let clip = p.clip_rect();
let pointer = ui.input(|i| i.pointer.hover_pos());
// the box being dragged: (region index, its box so far)
let drag_key = egui::Id::new("region-drag");
let live: Option<(usize, lightcraft_geom::Rect)> = ui.data(|d| d.get_temp(drag_key));
let mut edit = None;
for (index, r) in photo.meta.regions.iter().enumerate() {
let dragging = live.filter(|(i, _)| *i == index);
let norm = orient.map_norm_rect(dragging.map_or(r.rect, |(_, n)| n));
let rect = Rect::from_two_pos(map.screen(Point::new(norm.x0, norm.y0)), map.screen(Point::new(norm.x1, norm.y1)));
// white with a black keyline just outside it, so the box shows on any background
p.rect_stroke(rect.expand(1.0), 0.0, Stroke::new(1.0, Color32::from_black_alpha(190)), StrokeKind::Outside);
p.rect_stroke(rect, 0.0, Stroke::new(1.0, Color32::WHITE), StrokeKind::Outside);
if dragging.is_some() || (live.is_none() && pointer.is_some_and(|h| rect.expand(8.0).contains(h))) {
for (h, (fx, fy, cursor)) in REGION_HANDLES.iter().enumerate() {
let c = pos2(rect.left() + fx * rect.width(), rect.top() + fy * rect.height());
let resp = ui.interact(Rect::from_center_size(c, vec2(14.0, 14.0)), egui::Id::new(("region-handle", index, h)), Sense::drag());
if resp.hovered() || resp.dragged() {
ui.ctx().set_cursor_icon(*cursor);
}
let sq = Rect::from_center_size(c, vec2(7.0, 7.0));
p.rect_filled(sq, 0.0, if resp.hovered() || resp.dragged() { Color32::from_rgb(120, 190, 255) } else { Color32::WHITE });
p.rect_stroke(sq, 0.0, Stroke::new(1.0, Color32::from_black_alpha(220)), StrokeKind::Outside);
if resp.dragged()
&& let Some(pp) = resp.interact_pointer_pos()
{
// move the dragged edge(s) to the pointer, never closer than 12 px to the opposite one
let mut n = rect;
if *fx == 0.0 {
n.min.x = pp.x.min(n.max.x - 12.0);
} else if *fx == 1.0 {
n.max.x = pp.x.max(n.min.x + 12.0);
}
if *fy == 0.0 {
n.min.y = pp.y.min(n.max.y - 12.0);
} else if *fy == 1.0 {
n.max.y = pp.y.max(n.min.y + 12.0);
}
let (a, b) = (map.norm(n.min), map.norm(n.max));
let shown = lightcraft_geom::Rect {
x0: a.x.min(b.x).clamp(0.0, 1.0),
y0: a.y.min(b.y).clamp(0.0, 1.0),
x1: a.x.max(b.x).clamp(0.0, 1.0),
y1: a.y.max(b.y).clamp(0.0, 1.0),
};
// back to the upright frame the region is stored in
let new = orient.inverse().map_norm_rect(shown);
ui.data_mut(|d| d.insert_temp(drag_key, (index, new)));
ui.ctx().request_repaint();
}
if resp.drag_stopped() {
if let Some((i, new)) = ui.data(|d| d.get_temp::<(usize, lightcraft_geom::Rect)>(drag_key)) {
edit = Some(RegionEdit::Resize(i, new));
}
ui.data_mut(|d| d.remove_temp::<(usize, lightcraft_geom::Rect)>(drag_key));
}
}
}
if dragging.is_none() && live.is_none() && pointer.is_some_and(|h| rect.expand(3.0).contains(h)) {
let xr = Rect::from_center_size(pos2(rect.right() - 13.0, rect.top() + 13.0), vec2(16.0, 16.0));
let resp = ui.interact(xr, egui::Id::new(("region-x", index)), Sense::click());
register(ui.ctx(), format!("regionRemove:{index}"), xr);
let fill = Color32::from_black_alpha(if resp.hovered() { 235 } else { 190 });
p.rect_filled(xr, 2.0, fill);
p.rect_stroke(xr, 2.0, Stroke::new(1.0, Color32::from_white_alpha(120)), StrokeKind::Inside);
let (c, m) = (xr.center(), 3.5);
let cross = Stroke::new(1.4, if resp.hovered() { Color32::WHITE } else { Color32::from_gray(210) });
p.line_segment([c - vec2(m, m), c + vec2(m, m)], cross);
p.line_segment([c - vec2(m, -m), c + vec2(m, -m)], cross);
if resp.on_hover_text("Remove this face box (undo with Edit ▸ Undo)").clicked() {
edit = Some(RegionEdit::Remove(index));
}
}
let Some(name) = &r.name else { continue };
// the name in a dark label with a caret, centred above the box (below it when there is no room)
let g = p.layout_no_wrap(name.clone(), t.font(13.0), Color32::from_gray(225));
let (pad, caret) = (vec2(14.0, 7.0), 5.0);
let size = g.size() + pad * 2.0;
let above = rect.top() - caret - size.y >= clip.top();
let (top, tip, base) = if above {
(rect.top() - caret - size.y, rect.top() - 1.0, rect.top() - caret - 1.0)
} else {
(rect.bottom() + caret, rect.bottom() + 1.0, rect.bottom() + caret + 1.0)
};
let left = (rect.center().x - size.x / 2.0).clamp(clip.left(), (clip.right() - size.x).max(clip.left()));
let label = Rect::from_min_size(pos2(left, top), size);
let fill = Color32::from_rgba_unmultiplied(56, 56, 56, 235);
p.rect_filled(label, 3.0, fill);
p.rect_stroke(label, 3.0, Stroke::new(1.0, Color32::from_black_alpha(160)), StrokeKind::Inside);
let cx = rect.center().x.clamp(label.left() + caret + 4.0, label.right() - caret - 4.0);
p.add(egui::Shape::convex_polygon(vec![pos2(cx - caret, base), pos2(cx + caret, base), pos2(cx, tip)], fill, Stroke::NONE));
p.galley(label.min + pad, g, Color32::from_gray(225));
}
edit
}
/// A small pill at the loupe's top left naming the active filters: the filmstrip and Next / Previous
/// follow them, so a filter must never be invisible here. A click clears them all. It sits in the
/// canvas margin, so the photo does not move.
fn filter_pill(app: &mut LightcraftApp, ui: &mut egui::Ui, canvas: Rect) {
let chips = lightcraft_engine::filter_chips(&app.session.filter, &app.session.catalog);
if chips.is_empty() {
return;
}
let t = Tokens::get(ui.ctx());
let p = ui.painter_at(canvas);
let label = chips.iter().map(|c| c.label.as_str()).collect::<Vec<_>>().join(" · ");
let g = p.layout_no_wrap(format!("Filtered: {label}"), t.font(12.0), Color32::from_gray(225));
let w = (g.size().x + 40.0).min((canvas.width() - 24.0).max(60.0));
let r = Rect::from_min_size(canvas.min + vec2(12.0, 3.0), vec2(w, 20.0));
let resp = ui.interact(r, egui::Id::new("loupe-filter-pill"), Sense::click());
register(ui.ctx(), "loupe:filters", r);
p.rect_filled(r, 10.0, Color32::from_black_alpha(if resp.hovered() { 225 } else { 185 }));
let text_clip = Rect::from_min_max(r.min, pos2(r.right() - 24.0, r.max.y));
p.with_clip_rect(text_clip).galley(pos2(r.left() + 10.0, r.center().y - g.size().y / 2.0), g, Color32::from_gray(225));
let (c, m) = (pos2(r.right() - 13.0, r.center().y), 3.0);
let cross = Stroke::new(1.3, if resp.hovered() { Color32::WHITE } else { Color32::from_gray(190) });
p.line_segment([c - vec2(m, m), c + vec2(m, m)], cross);
p.line_segment([c - vec2(m, -m), c + vec2(m, -m)], cross);
if resp.on_hover_text("Click to clear these filters").clicked() {
let _ = app.run("library.clearFilter", json!({}));
}
}
/// A raw shown from its embedded JPEG (issue #10): a pill at the canvas' top centre saying so;
/// hovering it explains (registered as `notice:previewOnly:loupe`).
fn preview_only_pill(ui: &mut egui::Ui, canvas: Rect, reason: &str) {
@@ -1001,6 +1172,47 @@ fn mask_overlay(app: &mut LightcraftApp, ui: &mut egui::Ui, resp: &egui::Respons
}
}
}
// Object selections (SAM 3) of the selected mask: each click, green to include, red to exclude
// (clicks still on their way to the model included)
let pending = app.session.segmenter.pending_clicks().cloned();
for m in d.masks.iter().filter(|m| Some(m.id) == active) {
for (k, c) in m.components.iter().enumerate() {
if let MaskShape::Object { hint, exclude, .. } = &c.shape {
let (hint, exclude) = match pending.as_ref().filter(|q| (q.mask, q.comp) == (m.id, k)) {
Some(q) => (&q.hint, &q.exclude),
None => (hint, exclude),
};
for (pts, col) in [(hint, Color32::from_rgb(40, 200, 90)), (exclude, Color32::from_rgb(230, 60, 60))] {
for q in pts {
let q = map.screen(*q);
p.circle_filled(q, 5.0, col);
p.circle_stroke(q, 5.0, Stroke::new(1.5, Color32::WHITE));
}
}
}
}
}
// Object tool: a click includes what's under it, ⌥-click leaves it out
if app.ui.tool == "object" {
if resp.hovered() {
ui.ctx().set_cursor_icon(egui::CursorIcon::Crosshair);
}
if resp.clicked()
&& let Some(q) = resp.interact_pointer_pos()
{
let n = map.norm(q);
let exclude = ui.input(|i| i.modifiers.alt);
match app.run("mask.objectPoint", json!({"x": n.x, "y": n.y, "exclude": exclude})) {
Ok(_) => {
if let Some(mid) = app.session.active_mask {
app.ui.detail_due = Some((ui.input(|i| i.time) + 1.0, mid));
}
}
Err(e) => app.ai_error(ui.ctx(), e, Some(("object", "new"))),
}
}
return;
}
// brush tool
if app.ui.tool == "brush" {
let r = (app.ui.brush_size as f64 * long) as f32;
+122 -11
View File
@@ -94,6 +94,7 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
Dialog::Merge { opts } => opts.title(),
Dialog::Settings { .. } => "Settings",
Dialog::ConfirmDelete { .. } => "Delete Photos",
Dialog::SamModel { .. } => "Download the SAM 3 Model?",
Dialog::About => "About LightCraft",
Dialog::Shortcuts => "Keyboard Shortcuts",
}
@@ -466,7 +467,7 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
ui.separator();
}
builtin = b;
if ui.selectable_label(false, &p.name).clicked() {
if ui.selectable_label(false, crate::i18n::builtin_label(&p.name, b)).clicked() {
chosen = Some(p.name);
}
}
@@ -700,8 +701,9 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
Dialog::Merge { opts } => crate::merge::body(app, ui, opts),
Dialog::Import { opts } => crate::import::body(app, ui, opts),
Dialog::Settings { tab } => crate::panels::settings::body(app, ui, tab),
Dialog::SamModel { error, .. } => sam_model_body(app, ui, error.as_deref()),
Dialog::ConfirmDelete { count } => {
let what = if *count == 1 { "this photo".to_string() } else { crate::i18n::tr_format!("these {count} photos", count = count) };
let what = if *count == 1 { crate::i18n::tr("this photo").to_string() } else { crate::i18n::tr_format!("these {count} photos", count = count) };
ui.label(crate::i18n::tr_format!("Move {what} to Recently Deleted?", what = what));
ui.label(egui::RichText::new(crate::i18n::tr("They can be restored from Recently Deleted until it is emptied.")).color(t.text_dim));
}
@@ -710,7 +712,7 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
ui.label(crate::i18n::tr_format!("Version {} — a clean-room, pure-Rust photo library and raw developer.", env!("CARGO_PKG_VERSION")));
ui.label(format!("MIT OR Apache-2.0. Fonts: {} (OFL). Icons: original.", crate::theme::font_credits()));
ui.add_space(10.0);
let discord = egui::Button::new(egui::RichText::new("Join the ArtCraft Discord").font(t.semibold(15.0)).color(egui::Color32::WHITE))
let discord = egui::Button::new(egui::RichText::new(crate::i18n::tr("Join the ArtCraft Discord")).font(t.semibold(15.0)).color(egui::Color32::WHITE))
.fill(t.accent)
.min_size(egui::vec2(260.0, 34.0));
let r = ui.add(discord).on_hover_text(crate::links::DISCORD);
@@ -724,7 +726,7 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
("Source code on GitHub", crate::links::GITHUB),
("ArtCraft — more creative apps", crate::links::WEBSITE),
] {
let r = ui.link(label).on_hover_text(url);
let r = ui.link(crate::i18n::tr(label)).on_hover_text(url);
if r.clicked() {
let _ = crate::links::open(app, url);
}
@@ -733,7 +735,7 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
Dialog::Shortcuts => {
egui::ScrollArea::vertical().max_height(420.0).show(ui, |ui| {
egui::Grid::new("shortcuts").striped(true).show(ui, |ui| {
for (id, label, sc, _) in crate::menus::UI_COMMANDS {
for (id, label, sc, _) in crate::menus::ui_commands() {
if let Some(sc) = sc {
ui.label(crate::i18n::tr(label));
ui.label(*sc);
@@ -750,8 +752,7 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
}
}
for (sc, id, _) in crate::shortcuts::ALIASES {
let label = crate::menus::UI_COMMANDS
.iter()
let label = crate::menus::ui_commands()
.find(|c| c.0 == *id)
.map(|c| c.1)
.or_else(|| lightcraft_engine::find_command(id).map(|c| c.label))
@@ -768,23 +769,44 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
ui.add_space(4.0);
ui.horizontal(|ui| {
let informational = matches!(dlg, Dialog::About | Dialog::Shortcuts | Dialog::Settings { .. });
if !informational && ui.button(crate::i18n::tr("Cancel")).clicked() {
close = true;
let sam = &app.session.segmenter;
let (sam_installed, sam_running, sam_failed) = (sam.installed(), sam.download_status().running, sam.download_status().error.is_some());
// no download location in this build: nothing to offer but the manual install
let sam_nowhere = !sam_installed && !sam_running && sam.mirrors().is_empty();
let cancel = match &dlg {
Dialog::SamModel { .. } if sam_running || sam_installed || sam_nowhere => "Close",
Dialog::SamModel { .. } => "Not Now",
_ => "Cancel",
};
if !informational {
let r = ui.button(crate::i18n::tr(cancel));
crate::widgets::register(ui.ctx(), "button:dialogCancel", r.rect);
if r.clicked() {
close = true;
}
}
let add_label;
let ok = match &dlg {
Dialog::Import { opts } => {
let n = opts.selected_paths().len();
let verb = if opts.copy && opts.move_files { "Move" } else { "Import" };
add_label = format!("{verb} {n} Photo{}", if n == 1 { "" } else { "s" });
add_label = crate::i18n::tr_format!("{verb} {n} Photo{}", if n == 1 { "" } else { "s" }, verb = crate::i18n::tr(verb), n = n);
add_label.as_str()
}
Dialog::Merge { .. } => "Merge",
Dialog::ConfirmDelete { .. } => "Delete",
Dialog::SamModel { then: Some(_), .. } if sam_installed => "Continue",
Dialog::SamModel { .. } if sam_installed || sam_running || sam_nowhere => "",
Dialog::SamModel { error, .. } if error.is_some() || sam_failed => "Try Again",
Dialog::SamModel { .. } => "Download",
_ if informational => "Close",
_ => "OK",
};
if ui.button(crate::i18n::tr(ok)).clicked() {
let r = (!ok.is_empty()).then(|| ui.button(crate::i18n::tr(ok)));
if let Some(r) = &r {
crate::widgets::register(ui.ctx(), "button:dialogOk", r.rect);
}
if r.is_some_and(|r| r.clicked()) {
if informational {
close = true;
} else {
@@ -804,6 +826,17 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
match confirm_dialog(app, &dlg) {
// the import review stays open on an error (e.g. an unusable folder template)
Err(e) if matches!(dlg, Dialog::Import { .. }) => app.toast(ctx, e),
// the SAM 3 dialog stays open to show the download (or why it can't start)
Err(e) if matches!(dlg, Dialog::SamModel { .. }) => {
if let Dialog::SamModel { error, .. } = &mut dlg {
*error = Some(e);
}
}
Ok(_) if keeps_open(app, &dlg) => {
if let Dialog::SamModel { error, .. } = &mut dlg {
*error = None;
}
}
_ => close = true,
}
}
@@ -824,8 +857,86 @@ pub fn fmt_gap(v: f64) -> String {
}
}
/// Whether a dialog stays open after its action succeeded (the SAM 3 dialog while the model
/// downloads).
pub fn keeps_open(app: &LightcraftApp, dlg: &Dialog) -> bool {
matches!(dlg, Dialog::SamModel { .. }) && !app.session.segmenter.installed()
}
/// The SAM 3 dialog: what the model is, its size and licence, and the download's progress.
fn sam_model_body(app: &mut LightcraftApp, ui: &mut egui::Ui, error: Option<&str>) {
use lightcraft_engine::segment::{LICENSE_NAME, LICENSE_URL, MODEL_BYTES};
let t = Tokens::get(ui.ctx());
let seg = &app.session.segmenter;
let d = seg.download_status();
if seg.installed() {
ui.label(crate::i18n::tr("The SAM 3 model is installed: Object and Describe masks are ready."));
return;
}
let gb = |b: u64| b as f64 / 1e9;
ui.label(crate::i18n::tr(
"Object and Describe masks use SAM 3, Meta's segmentation model. It isn't part of LightCraft, and everything else works without it.",
));
let dir = seg.dir.as_ref().map(|d| d.display().to_string()).unwrap_or_default();
ui.label(format!("{} {:.1} GB, {} {dir}", crate::i18n::tr("A one-time download of about"), gb(MODEL_BYTES), crate::i18n::tr("saved in")));
ui.label(
egui::RichText::new(format!(
"{} {LICENSE_NAME} — {}",
crate::i18n::tr("Licence:"),
crate::i18n::tr("Meta's terms, not LightCraft's. Downloading it means accepting them.")
))
.color(t.text_label),
);
let r = ui.link(crate::i18n::tr("Read the SAM License")).on_hover_text(LICENSE_URL);
crate::widgets::register(ui.ctx(), "link:samLicense", r.rect);
if r.clicked() {
let _ = crate::links::open(app, LICENSE_URL);
}
let seg = &app.session.segmenter;
if d.running {
ui.add_space(4.0);
let frac = if d.total > 0 { d.done as f64 / d.total as f64 } else { 0.0 };
let text = format!("{:.2} / {:.2} GB · {}", gb(d.done), gb(d.total), d.file);
let r = ui.add(egui::ProgressBar::new(frac as f32).text(text));
crate::widgets::register(ui.ctx(), "progress:samDownload", r.rect);
let r = ui.button(crate::i18n::tr("Cancel Download"));
crate::widgets::register(ui.ctx(), "button:samCancel", r.rect);
if r.clicked() {
app.session.segmenter.cancel_download();
}
ui.label(egui::RichText::new(crate::i18n::tr("You can close this: the download continues, and resumes if interrupted.")).color(t.text_dim));
ui.ctx().request_repaint_after(std::time::Duration::from_millis(250));
return;
}
if seg.mirrors().is_empty() {
ui.label(
egui::RichText::new(crate::i18n::tr(
"This build has no download location for the model yet: put the files in the folder above yourself (see docs/ai-masks.md).",
))
.color(t.text_dim),
);
}
if let Some(e) = error.map(str::to_string).or(d.error) {
ui.label(egui::RichText::new(format!("{} {e}", crate::i18n::tr("The download didn't work:"))).color(egui::Color32::from_rgb(230, 90, 80)));
}
}
pub fn confirm_dialog(app: &mut LightcraftApp, dlg: &Dialog) -> Result<serde_json::Value, String> {
match dlg {
Dialog::SamModel { then, .. } => {
if app.session.segmenter.installed() {
// installed: start what the user was doing
if let Some((kind, op)) = then {
crate::panels::masking::begin_ai(app, kind, op)?;
}
return Ok(serde_json::Value::Null);
}
let r = app.run("segment.model.download", json!({"acknowledged": true}));
if r.is_ok() {
app.ui.sam_downloading = true;
}
r
}
Dialog::NewAlbum { name, folder } => app.run("album.create", json!({"name": name, "folder": folder, "addSelected": !folder})),
Dialog::RenameAlbum { id, name } => app.run("album.rename", json!({"id": id, "name": name})),
Dialog::TextPrompt { value, command, params, key, .. } => {
+11 -10
View File
@@ -140,8 +140,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
ui.horizontal(|ui| {
ui.label(egui::RichText::new(crate::i18n::tr("Profile")).font(t.font(13.0)).color(t.text_dim));
let name = lightcraft_engine::presets::profile(&d.profile.id).map(|p| p.name).unwrap_or("Color");
let r =
crate::widgets::dropdown(ui, "profile", if name == "Color" { crate::i18n::tr("Color") } else { name }, t.font(15.0), t.text_label);
let r = crate::widgets::dropdown(ui, "profile", crate::i18n::tr(name), t.font(15.0), t.text_label);
egui::Popup::menu(&r).show(|ui| profile_menu(app, ui, &d));
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if crate::widgets::icon_button(ui, "profileBrowser", Icon::ProfileGrid, vec2(28.0, 28.0), false, true, "Browse Profiles").clicked() {
@@ -356,7 +355,7 @@ fn profile_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
key: &str,
p: &'static lightcraft_engine::presets::ProfileInfo,
pick: &mut (Option<&'static str>, Option<&'static str>)| {
let r = ui.selectable_label(p.id == cur, p.name);
let r = ui.selectable_label(p.id == cur, crate::i18n::tr(p.name));
register(ui.ctx(), format!("profileMenu:{key}:{}", p.id), r.rect);
if r.clicked() {
pick.0 = Some(p.id);
@@ -366,7 +365,7 @@ fn profile_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
}
};
let heading = |ui: &mut egui::Ui, s: &str| {
ui.label(egui::RichText::new(s).font(t.semibold(11.5)).color(t.text_dim));
ui.label(egui::RichText::new(crate::i18n::tr(s)).font(t.semibold(11.5)).color(t.text_dim));
};
for (title, key, ids) in [("Favorites", "fav", app.session.profile_favorites.clone()), ("Recent", "recent", app.session.profile_recent.clone())] {
let list: Vec<_> = ids.iter().filter_map(|id| profile(id)).collect();
@@ -380,7 +379,7 @@ fn profile_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
ui.separator();
}
for g in profile_groups() {
let r = ui.menu_button(g, |ui| {
let r = ui.menu_button(crate::i18n::tr(g), |ui| {
ui.set_min_width(170.0);
for p in PROFILES.iter().filter(|p| p.group == g) {
item(ui, "group", p, &mut pick);
@@ -392,9 +391,9 @@ fn profile_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
if let Some(p) = profile(cur) {
let fav = app.session.profile_favorites.iter().any(|f| f == p.id);
let label = if fav {
crate::i18n::tr_format!("Remove “{}” from Favorites", p.name)
crate::i18n::tr_format!("Remove “{}” from Favorites", crate::i18n::tr(p.name))
} else {
crate::i18n::tr_format!("Add “{}” to Favorites", p.name)
crate::i18n::tr_format!("Add “{}” to Favorites", crate::i18n::tr(p.name))
};
let r = ui.button(label);
register(ui.ctx(), "profileMenu:toggleFavorite", r.rect);
@@ -414,7 +413,7 @@ fn profile_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
let mut s = d.clone();
s.profile.id = p.id.to_string();
s.profile.amount = 100.0;
app.hover_preview = Some(crate::HoverPreview { label: crate::i18n::tr_format!("Profile: {}", p.name), settings: s });
app.hover_preview = Some(crate::HoverPreview { label: crate::i18n::tr_format!("Profile: {}", crate::i18n::tr(p.name)), settings: s });
}
if let Some(id) = pick.0 {
let _ = app.run("develop.profile", json!({"id": id}));
@@ -811,7 +810,9 @@ fn curve_footer(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
ui.horizontal(|ui| {
ui.label(RichText::new(crate::i18n::tr("Point Curve")).font(t.font(12.0)).color(t.text_label));
let current = matching_preset(&app.session, &d.curve);
let r = crate::widgets::dropdown(ui, "curvePreset", current.as_deref().unwrap_or("Custom"), t.font(12.0), t.text);
let builtin = current.as_ref().is_none_or(|name| all_presets(&app.session).iter().any(|p| p.builtin && &p.name == name));
let label = crate::i18n::builtin_label(current.as_deref().unwrap_or("Custom"), builtin);
let r = crate::widgets::dropdown(ui, "curvePreset", label, t.font(12.0), t.text);
egui::Popup::menu(&r).show(|ui| {
ui.set_min_width(180.0);
let presets = all_presets(&app.session);
@@ -821,7 +822,7 @@ fn curve_footer(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings)
user_seen = true;
ui.separator();
}
let r = ui.selectable_label(current.as_deref() == Some(p.name.as_str()), &p.name);
let r = ui.selectable_label(current.as_deref() == Some(p.name.as_str()), crate::i18n::builtin_label(&p.name, p.builtin));
register(ui.ctx(), format!("curvePreset:{}", p.name), r.rect);
if r.clicked() {
let _ = app.run("curve.applyPreset", json!({"name": p.name}));
+9 -5
View File
@@ -150,7 +150,7 @@ fn show_inner(app: &mut LightcraftApp, ui: &mut egui::Ui) {
folder_header(app, ui, hr, &b, &ids, local, &cnt)
}
None => {
let title = app.session.source.label(&app.session.catalog);
let title = crate::i18n::source_label(app.session.source, &app.session.catalog);
ui.painter().text(pos2(hr.left() + 20.0, hr.center().y), Align2::LEFT_CENTER, &title, t.semibold(17.0), t.text);
ui.painter().text(pos2(hr.right() - 20.0, hr.center().y), Align2::RIGHT_CENTER, cnt, t.font(12.5), t.text_dim);
}
@@ -164,7 +164,11 @@ fn show_inner(app: &mut LightcraftApp, ui: &mut egui::Ui) {
if !chips.is_empty() {
let body = match total {
Some(n) if n > 0 => {
crate::i18n::tr_format!("{n} photos in {} are hidden by the filters above", app.session.source.label(&app.session.catalog), n = n)
crate::i18n::tr_format!(
"{n} photos in {} are hidden by the filters above",
crate::i18n::source_label(app.session.source, &app.session.catalog),
n = n
)
}
_ => "Remove a filter above, or choose Clear all".to_string(),
};
@@ -409,7 +413,7 @@ fn group_header(app: &mut LightcraftApp, ui: &mut egui::Ui, run: &DateRun, ids:
p.rect_filled(r, 0.0, t.canvas);
p.hline(r.x_range(), r.bottom(), Stroke::new(1.0, t.divider));
}
let g = p.layout_no_wrap(run.label.clone(), t.semibold(15.0), if resp.hovered() { t.text } else { t.text_label });
let g = p.layout_no_wrap(crate::i18n::date_group_label(&run.key, false), t.semibold(15.0), if resp.hovered() { t.text } else { t.text_label });
let x = r.left() + 8.0;
let gw = g.size().x;
p.galley(pos2(x, r.center().y - g.size().y / 2.0), g, t.text);
@@ -520,7 +524,7 @@ fn cell(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId, r: Rect, square
.collect();
if parts.is_empty() { "—".to_string() } else { parts.join(" · ") }
}
"date" => photo.captured.as_deref().map(lightcraft_catalog::dates::display_time).unwrap_or_else(|| "No date".into()),
"date" => photo.captured.as_deref().map(crate::i18n::display_time).unwrap_or_else(|| crate::i18n::tr("No date").into()),
_ => photo.file_name.rsplit_once('.').map(|(n, _)| n.to_string()).unwrap_or(photo.file_name.clone()),
};
p.text(pos2(r.left() + 8.0, r.top() + 12.0), Align2::LEFT_CENTER, name, t.font(10.5), t.text_dim);
@@ -786,7 +790,7 @@ pub fn context_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
ui.separator();
ui.menu_button(crate::i18n::tr("Set Rating"), |ui| {
for r in 0..=5 {
if ui.button(if r == 0 { "No Stars".to_string() } else { "★".repeat(r) }).clicked() {
if ui.button(if r == 0 { crate::i18n::tr("No Stars").to_string() } else { "★".repeat(r) }).clicked() {
let _ = app.run("photo.rate", json!({"rating": r}));
}
}
+3 -5
View File
@@ -120,14 +120,12 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
ui.painter().text(pos2(dr.left() + 18.0, dr.center().y), Align2::LEFT_CENTER, crate::i18n::tr("By Date"), t.semibold(13.5), t.text_label);
for g in app.caches.date_groups(&app.session.catalog).iter() {
// year → month → day; a click filters by that prefix, the triangle opens a level
if date_row(app, ui, &g.year, &g.year, g.count, 0.0) {
if date_row(app, ui, &g.year, &crate::i18n::date_group_label(&g.year, true), g.count, 0.0) {
for (m, n) in &g.months {
let label = lightcraft_catalog::dates::group_label(m).split(' ').next().unwrap_or(m).to_string();
let label = crate::i18n::date_group_label(m, true);
if date_row(app, ui, m, &label, *n, 16.0) {
for (d, n) in g.days.iter().filter(|(d, _)| d.starts_with(m.as_str())) {
let label = lightcraft_catalog::dates::group_label(d);
// "Sunday, 20 September 2026" → "Sunday, 20"
let label = label.rsplitn(3, ' ').nth(2).unwrap_or(&label).to_string();
let label = crate::i18n::date_group_label(d, true);
date_row(app, ui, d, &label, *n, 32.0);
}
}
+191 -2
View File
@@ -53,6 +53,7 @@ fn kind_label(s: &MaskShape) -> (&'static str, Icon) {
MaskShape::Sky => ("Sky", Icon::Sky),
MaskShape::Background => ("Background", Icon::Subject),
MaskShape::Object { .. } => ("Object", Icon::Subject),
MaskShape::Prompt { .. } => ("Describe", Icon::Subject),
MaskShape::People { .. } => ("People", Icon::Subject),
MaskShape::Landscape { .. } => ("Landscape", Icon::Sky),
}
@@ -75,7 +76,9 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
egui::Frame::NONE.inner_margin(egui::Margin { left: 24, right: 22, top: 0, bottom: 10 }).show(ui, |ui| {
ui.label(egui::RichText::new(crate::i18n::tr("Create New Mask")).color(t.text_dim));
ui.add_space(6.0);
let tiles: [(&str, &str, Icon); 8] = [
let tiles: [(&str, &str, Icon); 10] = [
("object", "Object", Icon::Subject),
("prompt", "Describe", Icon::Subject),
("subject", "Subject", Icon::Subject),
("sky", "Sky", Icon::Sky),
("background", "Background", Icon::Subject),
@@ -102,6 +105,8 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
app.ui.tool = "colorRange".into();
app.toast(ui.ctx(), "Click the photo to pick a colour · ⇧-click adds more");
}
"object" => start_object(app, ui.ctx(), "new"),
"prompt" => start_describe(app, "new"),
"brush" | "linear" | "radial" => {
app.ui.tool = kind.to_string();
if *kind != "brush" {
@@ -118,6 +123,37 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
}
}
});
describe_field(app, ui, true);
let seg = &app.session.segmenter;
let status = if seg.analyzing() {
Some("Analyzing the photo for AI masks…")
} else if seg.busy() {
Some("Selecting…")
} else if seg.detail_busy() || app.ui.detail_due.is_some() {
Some("Refining the mask's detail…")
} else {
None
};
if let Some(status) = status {
ui.add_space(4.0);
ui.label(egui::RichText::new(crate::i18n::tr(status)).color(t.text_dim));
ui.ctx().request_repaint_after(std::time::Duration::from_millis(250));
}
// the model download, while its dialog is closed
let download = seg.download_status();
if download.running && app.ui.dialog.is_none() {
ui.add_space(4.0);
ui.horizontal(|ui| {
let pct = if download.total > 0 { download.done as f64 / download.total as f64 } else { 0.0 };
let r = ui.add(
egui::ProgressBar::new(pct as f32).desired_width(ui.available_width() - 70.0).text(format!("SAM 3: {} %", (pct * 100.0).floor())),
);
register(ui.ctx(), "maskSamProgress", r.rect);
if text_button(ui, "maskSamDetails", crate::i18n::tr("Details"), false).clicked() {
app.offer_sam_download(None);
}
});
}
});
divider(ui);
// mask list
@@ -201,6 +237,18 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
app.ui.renaming_mask = Some((m.id, m.name.clone()));
}
resp.context_menu(|ui| mask_menu(app, ui, m.id, &m.name, m.visible, index, count));
if sel {
// right under the selected mask: + adds to it, − takes away from it (any mask
// type; Describe… opens its field here)
ui.horizontal(|ui| {
ui.add_space(30.0);
let plus = text_button(ui, &format!("maskPlus:{}", m.id), "+", false).on_hover_text(crate::i18n::tr("Add to this mask"));
egui::Popup::menu(&plus).show(|ui| component_menu(app, ui, "add"));
let minus = text_button(ui, &format!("maskMinus:{}", m.id), "−", false).on_hover_text(crate::i18n::tr("Subtract from this mask"));
egui::Popup::menu(&minus).show(|ui| component_menu(app, ui, "subtract"));
});
describe_field(app, ui, false);
}
}
if !d.masks.is_empty() {
ui.add_space(6.0);
@@ -213,7 +261,10 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
egui::Frame::NONE.inner_margin(egui::Margin { left: 24, right: 22, top: 8, bottom: 8 }).show(ui, |ui| {
for (i, c) in m.components.iter().enumerate() {
let (kind, icon) = kind_label(&c.shape);
let label = c.name.clone().unwrap_or_else(|| kind.to_string());
let label = c.name.clone().unwrap_or_else(|| match &c.shape {
MaskShape::Prompt { text, .. } => format!("“{text}”"),
_ => kind.to_string(),
});
ui.horizontal(|ui| {
let (r, _) = ui.allocate_exact_size(vec2(16.0, 16.0), Sense::hover());
paint(ui.painter(), r, icon, t.text_label);
@@ -486,6 +537,30 @@ fn range_controls(app: &mut LightcraftApp, ui: &mut egui::Ui, comp: usize, shape
}
}
}
MaskShape::Object { edge, .. } | MaskShape::Prompt { edge, .. } => {
// Edge: how crisp the selection's border is (−100 hard … 0 as computed … 100 soft)
let spec = ControlSpec {
id: "edge",
label: "Edge",
section: Section::Light,
min: -100.0,
max: 100.0,
default: 0.0,
step: 1.0,
decimals: 0,
track: Track::Centered,
};
let out = slider(ui, &spec, *edge, true, None);
let base = shape.clone();
apply_slider_out(app, &spec, out, |app, v| {
let mut s = base.clone();
if let MaskShape::Object { edge, .. } | MaskShape::Prompt { edge, .. } = &mut s {
*edge = v.clamp(-100.0, 100.0);
}
app.run("mask.update", json!({"component": comp, "shape": s}))
});
ui.label(egui::RichText::new(crate::i18n::tr("− harder border · + softer border")).color(t.text_dim).size(11.0));
}
MaskShape::ColorRange { samples, refine } => {
ui.horizontal(|ui| {
ui.label(
@@ -560,7 +635,121 @@ fn component_row_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, mask: u32, k:
}
}
/// Whether the SAM 3 model is missing in a build that could use it: then the download is
/// offered (with `then` to start afterwards) instead of starting an AI mask.
fn needs_model(app: &mut LightcraftApp, kind: &str, op: &str) -> bool {
let seg = &app.session.segmenter;
let missing = lightcraft_engine::segment::Segmenter::AVAILABLE && seg.dir.is_some() && !seg.installed();
if missing {
app.offer_sam_download(Some((kind, op)));
}
missing
}
/// Start an Object selection (SAM 3 clicks): a new mask, or a component of the selected one
/// combined by `op`; the photo is analyzed meanwhile (in the background).
pub(crate) fn start_object(app: &mut LightcraftApp, ctx: &egui::Context, op: &str) {
if needs_model(app, "object", op) {
return;
}
let r =
if op == "new" { app.run("mask.add", json!({"kind": "object"})) } else { app.run("mask.addComponent", json!({"op": op, "kind": "object"})) };
match r {
Ok(_) => {
app.ui.tool = "object".into();
app.toast(ctx, "Click the object to select it · ⌥-click leaves a part out");
}
Err(e) => app.ai_error(ctx, e, Some(("object", op))),
}
}
/// Start an AI mask of `kind` (object|prompt) combined by `op` (new|add|subtract|intersect),
/// after the model was installed.
pub(crate) fn begin_ai(app: &mut LightcraftApp, kind: &str, op: &str) -> Result<serde_json::Value, String> {
match kind {
"object" => {
let r = if op == "new" {
app.run("mask.add", json!({"kind": "object"}))
} else {
app.run("mask.addComponent", json!({"op": op, "kind": "object"}))
}?;
app.ui.tool = "object".into();
Ok(r)
}
_ => {
app.ui.describe = Some((op.to_string(), String::new()));
Ok(serde_json::Value::Null)
}
}
}
/// Open the Describe field (a new mask, or a component combined by `op`).
pub(crate) fn start_describe(app: &mut LightcraftApp, op: &str) {
if needs_model(app, "prompt", op) {
return;
}
app.ui.describe = Some((op.to_string(), String::new()));
}
/// The Describe field: type what to select ("sky", "the red car") and press Return.
fn describe_field(app: &mut LightcraftApp, ui: &mut egui::Ui, new_mask: bool) {
let Some((op, mut text)) = app.ui.describe.clone() else { return };
// a new mask's field sits under the tiles; one that combines, under the selected mask
if (op == "new") != new_mask {
return;
}
ui.add_space(8.0);
let prompt = match op.as_str() {
"subtract" => "Describe what to leave out",
"intersect" => "Describe what to keep",
_ => "Describe what to select",
};
ui.label(crate::i18n::tr(prompt));
let mut submit = false;
ui.horizontal(|ui| {
let r =
ui.add(egui::TextEdit::singleline(&mut text).hint_text("e.g. sky · the red car · car, road").desired_width(ui.available_width() - 64.0));
register(ui.ctx(), "maskDescribe", r.rect);
if !r.has_focus() && !r.lost_focus() && text.is_empty() {
r.request_focus();
}
if r.lost_focus() && ui.input(|i| i.key_pressed(egui::Key::Enter)) {
submit = true;
}
if text_button(ui, "maskDescribeGo", crate::i18n::tr("Select"), false).clicked() {
submit = true;
}
});
if ui.input(|i| i.key_pressed(egui::Key::Escape)) {
app.ui.describe = None;
return;
}
app.ui.describe = Some((op.clone(), text.clone()));
if submit && !text.trim().is_empty() {
let r = if op == "new" {
app.run("mask.add", json!({"kind": "prompt", "text": text}))
} else {
app.run("mask.addComponent", json!({"op": op, "kind": "prompt", "text": text}))
};
match r {
// the mask appears when the model has found it (a detail pass follows)
Ok(_) => app.ui.describe = None,
Err(e) => app.ai_error(ui.ctx(), e, Some(("prompt", op.as_str()))),
}
}
}
fn component_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, op: &str) {
let b = ui.button(crate::i18n::tr("Object"));
register(ui.ctx(), format!("maskComp:{op}:object"), b.rect);
if b.clicked() {
start_object(app, ui.ctx(), op);
}
let b = ui.button(crate::i18n::tr("Describe…"));
register(ui.ctx(), format!("maskComp:{op}:prompt"), b.rect);
if b.clicked() {
start_describe(app, op);
}
for (kind, label) in [
("brush", "Brush"),
("linear", "Linear Gradient"),
+1
View File
@@ -13,6 +13,7 @@ pub mod left;
pub mod library_problem;
pub mod masking;
pub mod notices;
pub mod people;
pub mod presets;
pub mod profiles;
pub mod right;
+87
View File
@@ -0,0 +1,87 @@
//! People: a card per person named on faces (read from XMP): a close-up of their largest face, the
//! name and how many photos they are in. A click shows that person's photos in the grid.
//!
//! Only the rows on screen ask for a face render (the engine caches them, memory and disk).
use egui::{Align2, Color32, Rect, Sense, Stroke, StrokeKind, pos2, vec2};
use lightcraft_catalog::Person;
use serde_json::json;
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
/// A card's face picture (points); the name and count sit below it.
const CARD: f32 = 150.0;
const LABEL_H: f32 = 44.0;
const GAP: f32 = 16.0;
const PAD: f32 = 20.0;
const HEADER_H: f32 = 44.0;
pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
let t = Tokens::get(ui.ctx());
let people = app.caches.people(&app.session.catalog, &app.session.filter);
let (head, _) = ui.allocate_exact_size(vec2(ui.available_width(), HEADER_H), Sense::hover());
ui.painter().text(pos2(head.left() + PAD, head.center().y), Align2::LEFT_CENTER, "Named People", t.semibold(15.0), t.text);
ui.painter().text(pos2(head.right() - PAD, head.center().y), Align2::RIGHT_CENTER, people.len().to_string(), t.font(13.0), t.text_dim);
// the filters narrowing the list (a date, a keyword…), removable here
let chips = lightcraft_engine::filter_chips(&app.session.filter, &app.session.catalog);
super::chips::show(app, ui, &chips);
if people.is_empty() {
let (title, body) = if chips.is_empty() {
("No named people yet", "Face names written to XMP by Lightroom and other apps show up here.")
} else {
("No named people in these photos", "Remove a filter above, or choose Clear all")
};
super::empty_message(ui, ui.available_rect_before_wrap(), title, body);
return;
}
let ppp = ui.ctx().pixels_per_point();
let active = app.session.filter.person.clone();
egui::ScrollArea::vertical().auto_shrink(false).show_viewport(ui, |ui, viewport| {
let width = ui.available_width();
let cols = (((width - PAD * 2.0 + GAP) / (CARD + GAP)).floor() as usize).max(1);
let row_h = CARD + LABEL_H + GAP;
let rows = people.len().div_ceil(cols);
let (area, _) = ui.allocate_exact_size(vec2(width, PAD * 2.0 + rows as f32 * row_h), Sense::hover());
let first = ((viewport.top() - PAD) / row_h).floor().max(0.0) as usize;
let last = (((viewport.bottom() - PAD) / row_h).ceil().max(0.0) as usize).min(rows);
for row in first..last {
for col in 0..cols {
let Some(person) = people.get(row * cols + col) else { break };
let min = area.min + vec2(PAD + col as f32 * (CARD + GAP), PAD + row as f32 * row_h);
let selected = active.as_deref().is_some_and(|a| a.eq_ignore_ascii_case(&person.name));
card(app, ui, person, Rect::from_min_size(min, vec2(CARD, CARD + LABEL_H)), ppp, selected);
}
}
});
}
fn card(app: &mut LightcraftApp, ui: &mut egui::Ui, person: &Person, r: Rect, ppp: f32, selected: bool) {
let t = Tokens::get(ui.ctx());
let face = Rect::from_min_size(r.min, vec2(CARD, CARD));
let resp = ui.interact(r, egui::Id::new(("person-card", &person.name)), Sense::click());
register(ui.ctx(), format!("person:{}", person.name), r);
let p = ui.painter();
p.rect_filled(face, 3.0, t.canvas);
if let Some(job) = app.session.face_job(person.photo, person.face, (CARD * ppp).ceil() as usize)
&& let Some(tex) = app.renderer.variant(job)
{
p.image(tex.tex.id(), face, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE);
}
if selected {
p.rect_stroke(face, 3.0, Stroke::new(2.0, Color32::WHITE), StrokeKind::Outside);
} else if resp.hovered() {
p.rect_stroke(face, 3.0, Stroke::new(1.0, t.text_dim), StrokeKind::Outside);
}
// a long name must not run past the card
let name =
if person.name.chars().count() > 19 { format!("{}…", person.name.chars().take(18).collect::<String>()) } else { person.name.clone() };
p.text(pos2(r.left() + 2.0, face.bottom() + 14.0), Align2::LEFT_CENTER, name, t.semibold(13.0), t.text);
let photos = if person.count == 1 { "1 photo".to_string() } else { format!("{} photos", person.count) };
p.text(pos2(r.left() + 2.0, face.bottom() + 32.0), Align2::LEFT_CENTER, photos, t.font(12.0), t.text_dim);
if resp.on_hover_text(format!("{} — show their photos", person.name)).clicked() {
let _ = app.run("library.filter", json!({"person": person.name}));
let _ = app.run("view.photoGrid", json!({}));
}
}
+13 -4
View File
@@ -85,6 +85,9 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
groups.entry(p.group.clone()).or_default().push(p.clone());
}
for (g, items) in groups {
// A group holding stock presets is a stock group (shown in the UI language), even
// when user presets share its name; a group of user presets keeps its name.
let builtin = items.iter().any(|p| p.builtin);
let open_id = egui::Id::new(("preset-group", &g));
let open: bool = ui.data(|d| d.get_temp(open_id)).unwrap_or(true);
let (r, resp) = ui.allocate_exact_size(vec2(ui.available_width(), 32.0), Sense::click());
@@ -95,7 +98,13 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
if open { Icon::ChevronDown } else { Icon::ChevronRight },
t.text_label,
);
ui.painter().text(pos2(r.left() + 36.0, r.center().y), Align2::LEFT_CENTER, &g, t.semibold(13.0), t.text_label);
ui.painter().text(
pos2(r.left() + 36.0, r.center().y),
Align2::LEFT_CENTER,
crate::i18n::builtin_label(&g, builtin),
t.semibold(13.0),
t.text_label,
);
if resp.clicked() {
ui.data_mut(|d| d.insert_temp(open_id, !open));
}
@@ -108,7 +117,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
continue;
}
for pr in items {
let (pid, name, fav) = (pr.id.clone(), pr.name.clone(), pr.favorite);
let (pid, name, fav) = (pr.id.clone(), crate::i18n::builtin_label(&pr.name, pr.builtin).to_string(), pr.favorite);
let (r, resp) = ui.allocate_exact_size(vec2(ui.available_width(), row_h), Sense::click());
register(ui.ctx(), format!("preset:{pid}"), r);
let is_last = last.as_ref().is_some_and(|(l, _)| *l == pid);
@@ -156,7 +165,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
}
let (builtin, group) = (pr.builtin, pr.group.clone());
resp.context_menu(|ui| {
if ui.button(if fav { "Remove from Favorites" } else { "Add to Favorites" }).clicked() {
if ui.button(crate::i18n::tr(if fav { "Remove from Favorites" } else { "Add to Favorites" })).clicked() {
let _ = app.run("preset.favorite", json!({"id": pid}));
}
if builtin {
@@ -168,7 +177,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
.clicked()
{
match app.run("preset.update", json!({"id": pid})) {
Ok(_) => app.toast(ui.ctx(), format!("Updated “{name}”")),
Ok(_) => app.toast(ui.ctx(), crate::i18n::tr_format!("Updated “{name}”", name = name)),
Err(e) => app.toast(ui.ctx(), e),
}
}
+6 -6
View File
@@ -35,7 +35,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
let (cr, _) = ui.allocate_exact_size(vec2(ui.available_width(), 34.0), Sense::hover());
let name = profile(&d.profile.id).map(|p| p.name).unwrap_or("Color");
ui.painter().text(pos2(cr.left() + 24.0, cr.center().y), Align2::LEFT_CENTER, crate::i18n::tr("Profile"), t.font(13.0), t.text_dim);
ui.painter().text(pos2(cr.left() + 84.0, cr.center().y), Align2::LEFT_CENTER, name, t.font(14.0), t.text_label);
ui.painter().text(pos2(cr.left() + 84.0, cr.center().y), Align2::LEFT_CENTER, crate::i18n::tr(name), t.font(14.0), t.text_label);
if d.profile.id != "lc.color" {
super::edit::control(app, ui, &d, "profile.amount", true);
ui.add_space(6.0);
@@ -60,7 +60,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
if open { Icon::ChevronDown } else { Icon::ChevronRight },
t.text_label,
);
ui.painter().text(pos2(r.left() + 36.0, r.center().y), Align2::LEFT_CENTER, title, t.semibold(13.0), t.text_label);
ui.painter().text(pos2(r.left() + 36.0, r.center().y), Align2::LEFT_CENTER, crate::i18n::tr(title), t.semibold(13.0), t.text_label);
ui.painter().text(pos2(r.right() - 22.0, r.center().y), Align2::RIGHT_CENTER, list.len().to_string(), t.font(12.0), t.text_dim);
if resp.clicked() {
ui.data_mut(|m| m.insert_temp(open_id, !open));
@@ -90,7 +90,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
&& let Some(p) = profile(&saved_profile_id)
{
let s = with_profile(&d, p);
app.hover_preview = Some(HoverPreview { label: crate::i18n::tr_format!("Profile: {}", p.name), settings: s });
app.hover_preview = Some(HoverPreview { label: crate::i18n::tr_format!("Profile: {}", crate::i18n::tr(p.name)), settings: s });
}
} else {
ui.data_mut(|m| m.remove::<(PhotoId, String)>(last_hover_id));
@@ -129,7 +129,7 @@ fn cell_ui(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings, id:
painter.rect_stroke(thumb, 3.0, Stroke::new(1.0, t.text_dim), egui::StrokeKind::Outside);
}
let name_color = if applied { t.text } else { t.text_label };
painter.text(pos2(cell.left() + 2.0, thumb.bottom() + 12.0), Align2::LEFT_CENTER, p.name, t.font(12.0), name_color);
painter.text(pos2(cell.left() + 2.0, thumb.bottom() + 12.0), Align2::LEFT_CENTER, crate::i18n::tr(p.name), t.font(12.0), name_color);
// favourite star (always shown when set, on hover otherwise)
let fav = app.session.profile_favorites.iter().any(|f| f == p.id);
let star = Rect::from_center_size(pos2(thumb.right() - 13.0, thumb.top() + 13.0), vec2(20.0, 20.0));
@@ -140,10 +140,10 @@ fn cell_ui(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &DevelopSettings, id:
paint(painter, star.shrink(4.0), if fav { Icon::StarFilled } else { Icon::Star }, if fav || over_star { t.star } else { Color32::WHITE });
}
if resp.hovered() && !applied && !over_star {
app.hover_preview = Some(HoverPreview { label: crate::i18n::tr_format!("Profile: {}", p.name), settings: s.clone() });
app.hover_preview = Some(HoverPreview { label: crate::i18n::tr_format!("Profile: {}", crate::i18n::tr(p.name)), settings: s.clone() });
ui.data_mut(|m| m.insert_temp(egui::Id::new("last-profile-hover"), (id, p.id.to_string())));
}
let resp = resp.on_hover_text(format!("{} ({})", p.name, p.group));
let resp = resp.on_hover_text(format!("{} ({})", crate::i18n::tr(p.name), crate::i18n::tr(p.group)));
if resp.clicked() {
if resp.interact_pointer_pos().is_some_and(|q| star.contains(q)) {
let _ = app.run("profile.favorite", json!({"id": p.id}));
+19 -8
View File
@@ -48,7 +48,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
pub fn header(ui: &mut egui::Ui, title: &str) {
let t = Tokens::get(ui.ctx());
let (r, _) = ui.allocate_exact_size(vec2(ui.available_width(), 46.0), Sense::hover());
ui.painter().text(pos2(r.left() + 24.0, r.center().y + 2.0), Align2::LEFT_CENTER, title, t.semibold(15.0), t.text);
ui.painter().text(pos2(r.left() + 24.0, r.center().y + 2.0), Align2::LEFT_CENTER, crate::i18n::tr(title), t.semibold(15.0), t.text);
}
pub fn label_row(ui: &mut egui::Ui, label: &str, value: &str) {
@@ -69,7 +69,7 @@ fn crop(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
ui.horizontal(|ui| {
ui.label(crate::i18n::tr("Aspect Ratio"));
let cur = d.crop.aspect.map(|(w, h)| format!("{} × {}", w as f64 / 100.0, h as f64 / 100.0)).unwrap_or_else(|| "Free".into());
let r = ui.add(egui::Button::new(cur).frame(false));
let r = ui.add(egui::Button::new(crate::i18n::tr(&cur)).frame(false));
register(ui.ctx(), "button:cropAspect", r.rect);
egui::Popup::menu(&r).show(|ui| {
for (label, a) in [
@@ -84,7 +84,7 @@ fn crop(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
("16 × 9", "16x9"),
("16 × 10", "16x10"),
] {
if ui.button(label).clicked() {
if ui.button(crate::i18n::tr(label)).clicked() {
let _ = app.run("crop.aspect", json!({"aspect": a}));
}
}
@@ -494,7 +494,7 @@ fn info(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
if crate::widgets::icon_button(ui, "editCaptureTime", Icon::Pencil, vec2(20.0, 20.0), false, true, "Edit Capture Time…").clicked() {
let _ = app.run("dialog.captureTime", json!({}));
}
let cap = p.captured.as_deref().map(lightcraft_catalog::dates::display_time).unwrap_or_else(|| "—".into());
let cap = p.captured.as_deref().map(crate::i18n::display_time).unwrap_or_else(|| "—".into());
ui.with_layout(egui::Layout::left_to_right(egui::Align::Center), |ui| {
ui.add(egui::Label::new(egui::RichText::new(cap).size(12.5).color(t.text)).truncate());
});
@@ -651,7 +651,7 @@ fn copyright_status(app: &mut LightcraftApp, ui: &mut egui::Ui, current: lightcr
fn meta_field(app: &mut LightcraftApp, ui: &mut egui::Ui, label: &str, key: &str, value: &str, lines: usize) {
let t = Tokens::get(ui.ctx());
ui.label(egui::RichText::new(label).size(11.5).color(t.text_dim));
ui.label(egui::RichText::new(crate::i18n::tr(label)).size(11.5).color(t.text_dim));
let id = egui::Id::new(("info-field", key));
let mut text: String = ui.data(|d| d.get_temp(id)).unwrap_or_else(|| value.to_string());
let edit = if lines > 1 { egui::TextEdit::multiline(&mut text).desired_rows(lines) } else { egui::TextEdit::singleline(&mut text) };
@@ -830,7 +830,8 @@ fn versions(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
let mut auto: bool = ui.data(|d| d.get_temp(tab_id)).unwrap_or(false);
let named_n = p.versions.iter().filter(|v| !v.auto).count();
let items = [("Named", "named"), ("Auto", "auto")];
let labels = [crate::i18n::tr_format!("Named ({named_n})", named_n = named_n), format!("Auto ({})", p.versions.len() - named_n)];
let labels =
[crate::i18n::tr_format!("Named ({named_n})", named_n = named_n), crate::i18n::tr_format!("Auto ({})", p.versions.len() - named_n)];
let items: Vec<(&str, &str)> = items.iter().zip(&labels).map(|((_, k), l)| (l.as_str(), *k)).collect();
if let Some(i) = crate::widgets::segmented(ui, "versionsTab", &items, Some(auto as usize), 2) {
auto = i == 1;
@@ -840,7 +841,7 @@ fn versions(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
let shown: Vec<(usize, &lightcraft_catalog::Version)> = p.versions.iter().enumerate().filter(|(_, v)| v.auto == auto).collect();
if shown.is_empty() {
let msg = if auto { "No automatic versions." } else { "No versions yet. Create one to keep this look." };
ui.label(egui::RichText::new(msg).color(t.text_dim));
ui.label(egui::RichText::new(crate::i18n::tr(msg)).color(t.text_dim));
}
// newest first
for (i, v) in shown.into_iter().rev() {
@@ -907,6 +908,10 @@ fn versions(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
/// "Sep 30, 2026, 12:00 PM" from an ISO time.
fn short_time(iso: &str) -> String {
// Other languages use their own date patterns (`i18n::display_time`).
if crate::i18n::language() != crate::i18n::Locale::En {
return crate::i18n::display_time(iso);
}
let long = lightcraft_catalog::dates::display_time(iso);
// "September 30, 2026 at 12:00:00 PM" → month abbreviated, seconds dropped
let (date, time) = long.split_once(" at ").map_or((long.as_str(), None), |(d, t)| (d, Some(t)));
@@ -935,7 +940,13 @@ fn activity(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
ui.painter().rect_filled(r, 3.0, t.hover);
}
paint(ui.painter(), Rect::from_min_size(r.min + vec2(0.0, 4.0), vec2(16.0, 16.0)), Icon::Clock, t.icon);
ui.painter().text(pos2(r.left() + 24.0, r.center().y), Align2::LEFT_CENTER, &h.label, t.font(12.5), t.text_label);
ui.painter().text(
pos2(r.left() + 24.0, r.center().y),
Align2::LEFT_CENTER,
crate::i18n::history_label(&h.label, &app.session.presets),
t.font(12.5),
t.text_label,
);
if resp.clicked() {
let _ = app.run("history.restore", json!({"index": i}));
}
+2 -1
View File
@@ -91,7 +91,8 @@ fn choices<V: PartialEq + Copy>(ui: &mut egui::Ui, id: &str, options: &[(V, &str
fn general_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {
row(ui, t, crate::i18n::tr("Language"), |ui| {
choices(ui, "settingsLanguage", &[(crate::i18n::Language::Ja, "日本語"), (crate::i18n::Language::En, "English")], &mut app.ui.language);
let languages: Vec<_> = crate::i18n::Locale::ALL.iter().map(|language| (*language, language.name())).collect();
choices(ui, "settingsLanguage", &languages, &mut app.ui.language);
crate::i18n::set_language(app.ui.language);
});
heading(ui, t, crate::i18n::tr("Library"));
+4 -4
View File
@@ -89,7 +89,7 @@ pub fn handle(app: &mut LightcraftApp, ctx: &egui::Context) {
let mut aliased: Vec<(&str, serde_json::Value)> = Vec::new();
ctx.input(|i| {
let mut ui_keys = Vec::new();
for (id, _, sc, _) in crate::menus::UI_COMMANDS {
for (id, _, sc, _) in crate::menus::ui_commands() {
if let Some((m, k)) = sc.and_then(parse) {
ui_keys.push((m, k));
if !native(sc.unwrap_or_default()) && matches(i, m, k) {
@@ -286,7 +286,7 @@ mod tests {
assert_eq!(parse("Right").unwrap().1, Key::ArrowRight);
assert_eq!(parse("\\").unwrap().1, Key::Backslash);
// every declared shortcut parses
for (id, _, sc, _) in crate::menus::UI_COMMANDS {
for (id, _, sc, _) in crate::menus::ui_commands() {
if let Some(sc) = sc {
assert!(parse(sc).is_some(), "{id}: {sc}");
}
@@ -300,7 +300,7 @@ mod tests {
#[test]
fn aliases_parse_and_target_existing_commands() {
let ui: Vec<&str> = crate::menus::UI_COMMANDS.iter().map(|c| c.0).collect();
let ui: Vec<&str> = crate::menus::ui_commands().map(|c| c.0).collect();
for (sc, id, params) in ALIASES {
assert!(parse(sc).is_some(), "{id}: {sc}");
assert!(ui.contains(id) || lightcraft_engine::find_command(id).is_some(), "alias {sc} → unknown command {id}");
@@ -315,7 +315,7 @@ mod tests {
#[test]
fn no_conflicting_bindings() {
let mut ui: Vec<((Modifiers, Key), String)> = Vec::new();
for (id, _, sc, _) in crate::menus::UI_COMMANDS {
for (id, _, sc, _) in crate::menus::ui_commands() {
if let Some(k) = sc.and_then(parse) {
ui.push((k, id.to_string()));
}
+29 -1
View File
@@ -16,6 +16,8 @@ pub enum ViewMode {
Survey,
/// A reference photo (left, fixed) beside the active photo (right, being edited).
Reference,
/// A card per person named on faces (close-up, name, photo count).
People,
}
/// The right-hand tool/panel shown next to the tool strip.
@@ -201,7 +203,7 @@ pub const MIN_PHOTO_WIDTH: f32 = 360.0;
#[serde(default, rename_all = "camelCase")]
pub struct UiState {
#[serde(default = "crate::i18n::default_language")]
pub language: crate::i18n::Language,
pub language: crate::i18n::Locale,
/// The Build Previews run last announced (its identity, finished?).
#[serde(skip)]
pub preview_build_seen: Option<(usize, bool)>,
@@ -262,6 +264,8 @@ pub struct UiState {
pub grid_info: String,
/// Photo counts next to sources and albums in the left panel.
pub show_counts: bool,
/// Face / pet boxes (read from XMP) over the photo in the loupe.
pub face_boxes: bool,
/// Local sidebar locations hidden with “Remove from Local” (folders on disk are untouched).
pub hidden_locations: Vec<String>,
/// Copies opened in an external editor this session (reloaded when the window is focused
@@ -283,6 +287,17 @@ pub struct UiState {
/// A mask being renamed in the Masks list: its id and the edited name.
#[serde(skip)]
pub renaming_mask: Option<(u32, String)>,
/// The Describe field (AI mask from a text prompt) while open: how the selection combines
/// (`new` mask, or `add`/`subtract`/`intersect` on the selected one) and the text typed.
#[serde(skip)]
pub describe: Option<(String, String)>,
/// A SAM 3 download was started from the app (to report its end once).
#[serde(skip)]
pub sam_downloading: bool,
/// When to start the zoomed-in detail pass of an AI mask (app time) and which mask: set by
/// each click or description, so the pass runs once the clicking stops.
#[serde(skip)]
pub detail_due: Option<(f64, u32)>,
/// A mask component being renamed inline: (mask id, component index, name).
pub renaming_component: Option<(u32, usize, String)>,
/// Close the window on the next frame (File → Quit).
@@ -502,6 +517,15 @@ pub enum Dialog {
Settings {
tab: String,
},
/// Object and Describe masks need the SAM 3 model, which isn't installed: offer to download
/// it (size, licence, progress). `then`: the AI mask to start once it is there (`kind`
/// object|prompt, `op` new|add|subtract|intersect).
SamModel {
then: Option<(String, String)>,
/// Why the download couldn't start (shown in the dialog).
#[serde(default)]
error: Option<String>,
},
/// Confirm moving photos to Recently Deleted.
ConfirmDelete {
count: usize,
@@ -548,6 +572,7 @@ impl Default for UiState {
show_filenames: true,
grid_info: "filename".into(),
show_counts: true,
face_boxes: true,
hidden_locations: Vec::new(),
dragging_control: None,
external_edits: Vec::new(),
@@ -556,6 +581,9 @@ impl Default for UiState {
search: String::new(),
focus_search: false,
renaming_mask: None,
describe: None,
detail_due: None,
sam_downloading: false,
renaming_component: None,
quit: false,
dragging_photos: None,
+55
View File
@@ -620,3 +620,58 @@ fn soft_proofing_flags_out_of_gamut_colours_and_makes_proof_copies() {
assert!(!h.app.ui.soft_proof);
assert_ne!(h.app.session.catalog.stack_of(copy.id).unwrap().collapsed, stack.collapsed, "S toggled the stack");
}
/// Object / Describe without the SAM 3 model: the download is offered (never started without a
/// yes), the dialog says why it can't start when no mirror is configured, nothing freezes and no
/// empty mask is left behind.
#[test]
fn ai_masks_without_the_model_offer_the_download() {
use crate::state::Dialog;
use lightcraft_engine::segment::Segmenter;
let mut h = detail("panel.masking");
let dir = std::env::temp_dir().join(format!("lc-ui-no-sam3-{}", std::process::id()));
h.app.session.segmenter.dir = Some(dir.clone());
h.app.session.segmenter.mirrors_file = Some(dir.join("none.txt"));
let t = std::time::Instant::now();
let r = h.request("ui.clickWidget", json!({"id": "maskNew:object"}), T);
assert_eq!(r["ok"], true, "{r}");
assert!(develop(&h).masks.is_empty());
if !Segmenter::AVAILABLE {
// a build without AI masks says so (a toast), no dialog
assert_eq!(h.app.ui.dialog, None);
return;
}
assert_eq!(h.app.ui.dialog, Some(Dialog::SamModel { then: Some(("object".into(), "new".into())), error: None }));
assert!(!h.app.session.segmenter.download_status().running, "nothing downloads without a yes");
if h.app.session.segmenter.mirrors().is_empty() {
// no location configured in this build: no Download button (only Close), and an agent
// confirming anyway gets the reason; the dialog stays
h.step();
let r = h.request("ui.clickWidget", json!({"id": "button:dialogOk"}), T);
assert_eq!(r["ok"], false, "{r}");
let r = h.request("ui.dialog.confirm", json!({}), T);
assert_eq!(r["ok"], false, "{r}");
assert!(r["error"].as_str().unwrap_or_default().contains("LIGHTCRAFT_SAM3_MIRRORS"), "{r}");
assert!(matches!(h.app.ui.dialog, Some(Dialog::SamModel { .. })), "stays open");
} else {
// with a mirror: Download starts it in the background (here it fails: nothing listens)
let r = h.request("ui.clickWidget", json!({"id": "button:dialogOk"}), T);
assert_eq!(r["ok"], true, "{r}");
}
// (a frame with the message laid out, so the buttons are where they are drawn)
h.step();
h.step();
let r = h.request("ui.clickWidget", json!({"id": "button:dialogCancel"}), T);
assert_eq!(r["ok"], true, "{r}");
assert_eq!(h.app.ui.dialog, None);
// Describe asks too
let r = h.request("ui.clickWidget", json!({"id": "maskNew:prompt"}), T);
assert_eq!(r["ok"], true, "{r}");
assert_eq!(h.app.ui.dialog, Some(Dialog::SamModel { then: Some(("prompt".into(), "new".into())), error: None }));
assert_eq!(h.app.ui.describe, None);
// a click command from an agent gets the not-installed error at once, and the app offers the model
let e = h.app.run("mask.add", json!({"kind": "prompt", "text": "sky"})).unwrap_err();
assert!(e.starts_with(lightcraft_engine::segment::NOT_INSTALLED), "{e}");
assert!(develop(&h).masks.is_empty());
assert!(t.elapsed() < SETTLE, "{:?}", t.elapsed());
}
+16 -12
View File
@@ -110,31 +110,35 @@ pub fn install_fonts(ctx: &egui::Context) {
ctx.set_fonts(font_definitions(lightcraft_engine::CRAFT_FONTS));
}
/// Inter (bundled) for Latin text, egui's default fonts, then the craft-fonts Japanese faces as
/// the last fallback of every family (BIZ UDPGothic first; Bold first for the semibold family).
/// Without craft-fonts (`craft` empty) Japanese text has no glyphs and shows as boxes.
/// Inter (bundled) for Latin text, egui's default fonts, then the craft-fonts CJK faces as the last
/// fallback of every family — the active language's own script first, so shared Han characters keep
/// that language's forms. Without craft-fonts (`craft` empty) CJK text has no glyphs and shows as
/// boxes.
pub fn font_definitions(craft: &'static [lightcraft_engine::CraftFont]) -> FontDefinitions {
let mut fonts = FontDefinitions::default();
fonts.font_data.insert("Inter".into(), Arc::new(FontData::from_static(include_bytes!("../../../assets/fonts/Inter-Regular.ttf"))));
fonts.font_data.insert("Inter-SemiBold".into(), Arc::new(FontData::from_static(include_bytes!("../../../assets/fonts/Inter-SemiBold.ttf"))));
let japanese: Vec<_> = lightcraft_engine::fonts::japanese(craft).collect();
for f in &japanese {
fonts.font_data.insert(craft_font_name(f), Arc::new(FontData::from_static(f.bytes)));
}
// Craft-fonts faces in preference order for a family drawn in `style`.
let fallback = |style: &str| {
let mut faces = japanese.clone();
faces.sort_by_key(|f| (f.family != "BIZ UDPGothic", f.style != style, f.family.contains("Mincho")));
faces.into_iter().map(craft_font_name).collect::<Vec<_>>()
lightcraft_engine::fonts::cjk_fallback(craft, crate::i18n::language().script(), style)
.into_iter()
.map(craft_font_name)
.collect::<Vec<String>>()
};
let (regular, bold) = (fallback("Regular"), fallback("Bold"));
for name in regular.iter().chain(bold.iter()) {
if let Some(font) = craft.iter().find(|font| &craft_font_name(font) == name) {
fonts.font_data.insert(name.clone(), Arc::new(FontData::from_static(font.bytes)));
}
}
let defaults: Vec<String> = fonts.families.get(&FontFamily::Proportional).cloned().unwrap_or_default();
let mut prop = vec!["Inter".to_string()];
prop.extend(defaults.iter().cloned());
prop.extend(fallback("Regular"));
prop.extend(regular);
fonts.families.insert(FontFamily::Proportional, prop);
let mut semi = vec!["Inter-SemiBold".to_string()];
semi.extend(defaults);
semi.extend(fallback("Bold"));
semi.extend(bold);
fonts.families.insert(FontFamily::Name(FONT_SEMIBOLD.into()), semi);
fonts.families.entry(FontFamily::Monospace).or_default().extend(fallback("Regular"));
fonts
+198
View File
@@ -0,0 +1,198 @@
# AI masks (Object and Describe)
LightCraft selects objects with **SAM 3** (Segment Anything with Concepts, Meta 2025), run in
pure Rust by `crates/segment` on [candle](https://github.com/huggingface/candle): on the GPU
through Metal on macOS, on the CPU elsewhere (for now).
**The model is optional.** It is not part of LightCraft, and nothing requires it: without it,
Object and Describe offer to download it, every other feature (and every existing mask, see
below) works, and no render, export or command ever waits for it.
## Using them
In the Masking panel (M):
- **Object**: click the thing you want; every click refines the selection. ⌥-click (Alt) a part
to leave it out. Clicks show as green (include) and red (leave out) dots.
- **Describe**: type what to select — `sky`, `trees`, `the red car` — and press Return. Every
instance the model finds (score above 0.5) joins the mask; if nothing matches, nothing is
created and you are told so.
- Several things at once: `car, road` (comma-separated) selects each and merges them.
- **+ / −** under the selected mask (and the Add / Subtract / Intersect menus) combine
selections: Describe `car`, click the mask, + ▸ Describe… `road`; − ▸ Describe… `people` takes
the people out of a sky mask; − ▸ Object then clicks remove one object.
- **Edge** (per selection, −100…100): below 0 a crisper border, above 0 a feathered one (up to 2 %
of the long edge, so previews and exports match).
- **Detail:** about a second after the last click (or right after a description), the photo
around the selection is analyzed again, zoomed in, from a sharper source (~5 s on an M4 Pro,
in the background); the result is kept as a high-resolution patch (at most 4 per selection),
so small objects get 5–10× finer edges than one pass over the whole photo gives.
The first Object or Describe on a photo analyzes it (the image encoder runs once per photo and
look: ~3 s on an Apple M4 Pro, plus a one-time ~6 s for compiling GPU kernels and loading the
model after launch). That starts in the background as soon as you pick Object; after it, each
click takes ~35 ms and each description ~0.6 s. Everything the model does runs on its own worker
thread: the window stays responsive, a click shows its dot at once and the selection follows
when it's computed ("Selecting…" / "Analyzing the photo…" under the mask tiles). The model is
unloaded after 10 minutes without use (it takes several GB of memory) and loaded again when
needed.
The result is stored with the mask: the model's 288 × 288 mask logits over the uncropped photo
(quantized, compressed, ~10–30 KB). Renders and exports sample them at any resolution and never
need the model, so masks survive moving the library to a machine without it, and the web build
renders them. Editing the photo's look later does not recompute a mask (click again to update).
## Getting the model
### In the app (recommended)
The first time you pick Object or Describe without the model, LightCraft asks:
> **Download the SAM 3 model?** Object and Describe masks use SAM 3, Meta's segmentation model.
> It isn't part of LightCraft, and everything else works without it. Download it once (about
> 3.4 GB) · *folder*. Licence: SAM License (Meta) — Meta's terms, not LightCraft's. Downloading
> it means accepting them. [Read the SAM License] — **Not Now** / **Download**
Nothing is downloaded until you choose **Download**. The download runs in the background (close
the dialog and keep working; the Masking panel shows its progress), can be cancelled, resumes
where it stopped if interrupted (also after quitting), and tries the next download location when
one fails or stalls. Each file is written as `<name>.part` and moved into the model folder only
when complete and verified — `model.safetensors` must have the exact size and SHA-256 of Meta's
official checkpoint; a file that doesn't match is deleted, never used. If no location works, the
dialog says why and LightCraft carries on without the model.
Agents and scripts use the same commands (control channel, `lightcraft-cli mcp --connect`):
`segment.model.status`, `segment.model.download {"acknowledged": true}` (refused without the
explicit `acknowledged`: pass it only after the user agreed to the download and the licence),
`segment.model.cancel`.
### Download locations
The model is fetched from an ordered list of mirrors: base URLs where `<base>/model.safetensors`,
`<base>/vocab.json` and `<base>/merges.txt` live. The list is, in order:
1. `LIGHTCRAFT_SAM3_MIRRORS` (environment variable; URLs separated by commas or spaces);
2. the file `models/sam3-mirrors.txt` in LightCraft's settings folder (one URL per line, `#`
comments) — `~/Library/Application Support/LightCraft/` (macOS), `%APPDATA%\LightCraft\`
(Windows), `~/.config/lightcraft/` (Linux);
3. the built-in list, `DEFAULT_MIRRORS` in `crates/segment/src/fetch/mod.rs`.
> **Maintainers:** the built-in list is **empty** until LightCraft's own CDN locations exist
> (see the `TODO(maintainer)` there): add them in order of preference, host the three files
> unchanged, and pin `vocab.json` / `merges.txt` (size + SHA-256) in `SAM3_FILES` at the same
> time. Until then the in-app download needs a user-configured mirror, and the dialog says so.
Hugging Face's `facebook/sam3` can't be a default: it is **gated** (each person must accept the
licence there, wait for approval and download with their own token). Someone with access can
use it as their mirror:
```sh
LIGHTCRAFT_SAM3_MIRRORS=https://huggingface.co/facebook/sam3/resolve/main \
LIGHTCRAFT_SAM3_TOKEN=hf_... lightcraft
```
(`LIGHTCRAFT_SAM3_TOKEN` is sent as a bearer token only over https, and only to the mirror's own
host — not to the storage host it redirects to.)
The download is plain HTTP/1.1 over TLS in pure Rust (rustls with the RustCrypto provider and
the Mozilla root certificates; no OpenSSL, `ring` or `aws-lc`, nothing compiled from C), with a
15 s connect timeout and a 30 s stall timeout; it follows redirects but doesn't use HTTP proxies.
### By hand
Put `model.safetensors`, `vocab.json` and `merges.txt` from `facebook/sam3` in the model folder:
`models/sam3/` in the settings folder above, or the folder `LIGHTCRAFT_SAM3_DIR` names. Developers
can use the installer scripts, which download from Hugging Face with your token and verify the
weights:
```sh
HF_TOKEN=hf_... tools/install-sam3.sh # macOS, Linux
tools/install-sam3.sh --check # verify an installation
```
```powershell
$env:HF_TOKEN = "hf_..."; .\tools\install-sam3.ps1 # Windows
```
## Licence of the model
LightCraft's code is MIT OR Apache-2.0; `crates/segment` (a port of Apache-2.0 code) is
Apache-2.0. **The SAM 3 weights are neither**: they are Meta's, released under the **SAM
License** (<https://github.com/facebookresearch/sam3/blob/main/LICENSE>), a custom licence that
is not an OSI open-source licence. It grants a non-exclusive, worldwide, non-transferable,
royalty-free licence to use, reproduce, distribute, copy, modify and create derivative works of
the "SAM Materials", subject to its terms, including: anyone redistributing them must provide a
copy of the licence with them, and derivative works are distributed under its terms; uses
subject to ITAR, military or warfare purposes, nuclear industries, espionage, guns or illegal
weapons, and reverse engineering are prohibited, and trade controls (sanctions, export rules)
apply; the materials come "as is", without warranty; Meta may terminate the licence on a breach
(then the materials must be deleted); California law governs. Read the licence itself — this
summary is not legal advice.
Therefore:
- **The weights are never part of this repository, its releases or installers**, and must not be
committed.
- They reach a user's computer only when that user asks for them, after the dialog names the
licence (or when they install them by hand).
- Whoever hosts a mirror redistributes the SAM Materials and must follow the licence (ship it
next to the files).
## Commands (control channel, MCP)
| Command | Params | |
|---|---|---|
| `mask.add` | `{kind: "object", points?: [[x,y],…], exclude?: [[x,y],…], seg?}` | an Object mask (empty until clicked) |
| `mask.add` | `{kind: "prompt", text, seg?}` | a Describe mask |
| `mask.addComponent` | `{op: add\|subtract\|intersect, kind: "object"\|"prompt", …}` | the same as a component |
| `mask.objectPoint` | `{x, y, exclude?: bool, id?}` | one click on the selected mask's Object selection (≤ 64 clicks) |
| `mask.refineDetail` | `{id?, component?}` | the zoomed-in detail pass → `{started}` |
| `segment.prepare` | `{}` | load the model and analyze the active photo → `{busy}` |
| `segment.model.status` | `{}` | `{available, installed, dir, loaded, busy, analyzing, sizeBytes, license, licenseUrl, mirrors, download}` |
| `segment.model.download` | `{acknowledged: true}` | start the download in the background → `{started}` |
| `segment.model.cancel` | `{}` | stop it (it resumes next time) → `{cancelled}` |
Coordinates are normalized to the uncropped, oriented photo, like every mask shape. `seg` passes
a stored segmentation (as `mask.list` / the develop settings hold it): the mask is made without
the model. Without the model, the AI kinds fail with an error starting "The SAM 3 model is not
installed" that says how to get it.
In the desktop app the commands don't wait for the model: `mask.objectPoint` returns
`{pending: true}` and the selection is applied when it's computed (one undo step per settled
selection); `mask.add {kind: "prompt"}` returns `{pending: true}` and the mask appears when
something is found (or a message says nothing was). `lightcraft-cli` and the headless MCP server
wait and return the result (when built with the `sam` feature; the default CLI has no AI masks).
## Implementation notes
- `crates/segment` ports the Hugging Face `transformers` implementation (Apache-2.0; see
`NOTICE` — the ported files say so and that they were modified): the 32-layer ViT backbone
with 2-D RoPE and windowed attention, the feature pyramids, the SAM 2-style prompt encoder and
two-way mask decoder for clicks, and the CLIP text encoder, DETR encoder/decoder (box
relative-position bias, presence token) and pixel decoder for text. The CLIP tokenizer is a
small BPE in `tokenizer.rs`. `fetch/` is the downloader (original code).
- candle is pinned at 0.9.2: later releases make `candle-core` depend on `tokenizers` with the
Oniguruma C library, and the product is pure Rust. Weights are read with positional reads (no
memory map: `unsafe` stays in `lightcraft-sysmem`), only the tensors a path needs, and every
tensor's byte range is checked against the file's length when it is opened (a truncated or
damaged file is an error, never a crash).
- The engine (`crates/engine/src/segment/`) owns one worker thread with the model and the last
encoded photo; the session sends it jobs and receives results over channels, never holding a
lock the model holds. Each job runs under `catch_unwind`: a panic inside candle becomes an
error and the model is reloaded on the next request; busy counters are reset by drop guards.
- Hostile documents: a selection keeps at most 4 detail patches (when read and when rendered),
each at most 1024² logits; more than 64 clicks are never sent to the model.
- Accuracy: `crates/segment/tests/reference.rs` compares every stage against the Python
reference (`tools/sam3_reference.py`); on Metal the backbone matches to 7e-5 relative error
and masks to ~5e-5. f16 was tried and rejected (overflow, ~7 % faster).
- The engine feature `sam` (on in the desktop app, off for web and CLI) gates the model; without
it the commands report that the build has no AI masks.
**Developers:** the accuracy test against the Python reference needs `torch` and
`transformers` (`tools/requirements-sam3-reference.txt`):
```sh
python3 -m venv .venv-sam3 && .venv-sam3/bin/pip install -r tools/requirements-sam3-reference.txt
.venv-sam3/bin/python tools/sam3_reference.py photo.jpg ref.safetensors
LIGHTCRAFT_SAM3_DIR=<model dir> LIGHTCRAFT_SAM3_REF=ref.safetensors cargo test -p lightcraft-segment --release -- --nocapture
```
+13 -2
View File
@@ -6,7 +6,11 @@ ARW and NEF decoding supply a Bayer mosaic and camera white-balance multipliers,
A fixed sensor proxy of at most 96×96 is developed with the file's WB, black/white levels and active crop. The embedded JPEG is decoded with DCT scaling and a 64-million-pixel input limit, converted through the codec's colour management, and reduced to the same size. Aspect ratios must agree within 2%. Near-black, clipped and nonfinite samples are excluded. At least 256 paired pixels, 5% coloured samples and a nonsingular input covariance with bounded condition are required.
Two thirds of the usable pixels train a ridge-regularised 3×3 **chromaticity** correction (luminance-normalised RGB). This avoids trying to encode the camera's nonlinear tone curve in a matrix, which left skin grey in the earlier matrix-only estimate. Coefficients are bounded by 8. Luminance is fitted separately: 32 equal-population bins with robust median knots and pooled-adjacent-violators isotonic regression. The curve preserves black, cannot reverse, and extends beyond observed highlights with a bounded shoulder.
Two thirds of the usable pixels train a ridge-regularised 3×3 **chromaticity** correction (luminance-normalised RGB). This avoids trying to encode the camera's nonlinear tone curve in a matrix, which left skin grey in the earlier matrix-only estimate. Coefficients are bounded by 8. A matrix can't follow the camera's hue-dependent rendering, though: on a photo of a lime shirt among grey rocks the best matrix rendered the shirt olive (16° hue error), because the few saturated pixels have little weight against the many dull ones, and a matrix fitted to the shirt alone breaks everything else. So a **hue/saturation table** (5 values × 72 hues × 5 saturations, the DNG `ProfileHueSatMap` layout with an sRGB-encoded value axis, applied in linear ProPhoto RGB after the baseline exposure) is fitted to the residual hue shifts and saturation ratios after the matrix: Gaussian kernels around each node (2.5° hue, 0.15 saturation, 0.12 encoded value), weighted by saturation, shrunk toward identity where the photo has few samples, saturation 0 left as identity so neutrals stay neutral. The value axis matters: on a sunny waterfall scene dark yellows had to turn ~12° toward orange while brighter yellow-greens turned ~9° toward green, and one shift per hue made both worse. The table restores each pixel's luminance (the tone curve owns luminance; without that, saturation boosts darkened colours and the table lost on held-out pixels).
The JPEG's colourfulness also depends on brightness, as a camera's per-channel tone curve makes it: relative to a luminance-only curve, shadows were up to 1.4× more colourful and highlights above 0.85 luminance bleached to 0.1–0.3×, so warm sunlit rock rendered cream instead of white (+4 b* in the brightest fifth of the picture). A **chroma curve** (8 nodes over display luminance, `CameraTone::chroma`) is fitted to that ratio, including the bright pixels the colour and tone fits leave out, and applied in the finish stage right after the tone map (CPU and GPU), scaling chroma toward luminance so luminance is kept. It is kept only when it lowers the held-out error; smart previews carry it in their header, and older headers load without one.
Matrix-only and matrix + table looks are each completed (tone curve, then chroma curve) and compared on the held-out pixels in gamma-encoded display values: in linear values a slightly missed bright rock outweighed a clearly wrong dark shirt. The acceptance gates below still use linear values. Luminance is fitted separately: 32 equal-population bins with robust median knots and pooled-adjacent-violators isotonic regression. The curve preserves black, cannot reverse, and extends beyond observed highlights with a bounded shoulder.
The matrix is applied to scene-linear sensor pixels. The curve is carried in `SourceInfo` and applied in the pipeline finish stage, after exposure/WB/local edits. It is never baked into decoded luminance or clipped to JPEG precision. CPU and GPU use the same tone LUT. Contrast, whites and blacks adjust the starting look through the display-domain controls. JPEG neutral rendering and the existing DNG colour model are unchanged.
@@ -26,7 +30,7 @@ Synthetic tests cover colour with a nonlinear camera tone, preserved scene-linea
Five local Sony ILCE-6700 ARWs were accepted. Held-out per-channel linear-display RMS values were 0.04244, 0.03349, 0.04552, 0.02871 and 0.03100. All five unedited supplied JPEG exports stayed pixel-identical to the installed pre-change renderer at equal size/quality. Four supplied JPEGs use Display P3 and two have a narrower crop than the RAW, so they are visual references rather than aligned pixel ground truth.
Public CC0 samples (raw.pixls.us; 10 files, ILCE-6000, -6400 ×2, -6700, -7M3, -7M4 ×2, -7RM4, -7SM3, DSC-RX100M3), rendered at 1200 px and compared with each file's embedded JPEG (mean CIE76 ΔE at 200 px): 7 fits accepted, ΔE 17–26 → 3–10 with matching mean lightness. (The RX100M3 was rejected at the time because the decoder read neither its white balance nor its black level and its render was green; since issue #148 its fit is accepted, held-out RMS 0.153 → 0.087.) Rejected and unchanged: an ILCE-7RM4 dusk scene whose fit was worse than the fallback (ΔE 7.6), and a reduced-size lossless ARW (no fit attempted). Saturated hues can still differ (an orange-red hull renders yellow).
Public CC0 samples (raw.pixls.us; 10 files, ILCE-6000, -6400 ×2, -6700, -7M3, -7M4 ×2, -7RM4, -7SM3, DSC-RX100M3), rendered at 1200 px and compared with each file's embedded JPEG (mean CIE76 ΔE at 200 px): 7 fits accepted, ΔE 17–26 → 3–10 with matching mean lightness. (The RX100M3 was rejected at the time because the decoder read neither its white balance nor its black level and its render was green; since issue #148 its fit is accepted, held-out RMS 0.153 → 0.087.) Rejected and unchanged: an ILCE-7RM4 dusk scene whose fit was worse than the fallback (ΔE 7.6), and a reduced-size lossless ARW (no fit attempted). Saturated hues can still differ (an orange-red hull renders yellow). With the hue/saturation table, 28 local ILCE-7M4 ARWs (one month of photos, one in 24 picked) went from mean ΔE 4.74 to 4.27 against their camera JPEGs (200 px, CIE76); 27 accepted a table, none got worse overall by more than 0.1, and a few saturated regions moved away (12.4 → 15.2 on one). The lime shirt's hue error dropped from 16° to 6–8°. With the value axis, the chroma curve and gamma-domain selection, the same set plus the shirt photo (29 files) went from mean ΔE 4.23 to 3.54 (4.74 before the table), every file closer than before, saturated areas 7.65 → 6.41; all 29 now use a table. A small object stays hard: in a second shot of the lime shirt only 7 of the 6144 proxy pixels show it, so the table can't learn its colour and the shirt still renders yellower than the JPEG.
### Nikon NEF
@@ -37,3 +41,10 @@ Measuring this found a decoder bug: 12-bit NEFs store maker note `0x003d` BlackL
On one sample, Exposure +1 EV, Saturation +100, Contrast +50, warm/cool WB, tint, highlights and shadows all changed the output. The same matrix and curve were selected for 400-pixel and full-resolution renders. Headless UI checks wait until the loupe reports `source: render` and no jobs remain; the camera JPEG stand-in does not count as verification.
This is a per-file camera-look estimate, **not measured spectral calibration or Lightroom parity**. Camera local tone, picture styles, noise reduction and lens warps cannot all be reproduced by a global matrix/curve; same-aspect crops may evade the aspect check. Skin, mixed-light scenes and saturated highlights can still differ from a supplied JPEG. Broader Sony and Nikon model/lighting coverage, chart-based calibration, absolute WB and other manufacturers remain unverified. Rejected fits and other RAW formats retain their existing behaviour.
## Camera profiles (pooled per model)
One photo can show too little of a colour for its own fit to learn it: in a second shot of the lime shirt only a few dozen proxy pixels show it, next to a hillside of foliage at the same hue that the camera renders differently, and the shirt stayed yellow (hue 68° against the JPEG's 82°; a 192-pixel proxy only reached 74°). `lightcraft-cli calibrate [--max N] [--out DIR] FOLDERS…` therefore pools the colour pairs of many ARWs per camera model — each against its own embedded JPEG, at a 192-pixel proxy, at most 4000 pairs per photo, files spread evenly over the folders — and fits one matrix (from white-balanced camera RGB, so no single photo's white point is baked in) and one hue/saturation/value table. Models need at least 5 usable photos.
Profiles are JSON files `<model>.json` in `$LIGHTCRAFT_CAMERA_PROFILES`, else `<config>/camera-profiles` (macOS `~/Library/Application Support/LightCraft/camera-profiles`); a local profile replaces a built-in one. Built-in profiles live in `assets/camera-profiles/` (listed in `assets/ATTRIBUTION.md`) and are compiled in, so the app, CLI and web build share them: ILCE-7M4, fitted to 597 photos (2.2 million colour pairs) shot in 2026, mostly with the Standard creative style and DRO Auto. They hold aggregate colour statistics only. A photo of a profiled model takes its colour from the profile and fits only its own tone and chroma curves (DRO and picture styles vary per shot); the acceptance gates still apply, and a rejected fit falls back as before. Files are read once per process and validated (version, bounded invertible matrix, table shape and finite data); a damaged file is ignored with a warning. The profiles folder's contents are part of the render cache keys, so thumbnails rendered before a profile existed are redone; smart previews built before keep their colour until rebuilt.

Some files were not shown because too many files have changed in this diff Show More