lightcraft/docs/denoise.md

AI Denoise

Edit ▸ Detail ▸ AI Denoise is a per-photo switch above Amount (0–100). Turning it on starts at 50, or preserves an already positive Amount. Turning it off preserves Amount and mixes none of the cleaned picture. It affects only the requesting photo, independently of Auto Sync, and is undoable. Manual luminance and colour noise reduction remain separate controls.

The built-in denoiser

AI Denoise needs no set-up. Unless you choose a model of your own (below), it runs the built-in denoiser (crates/denoise-core/src/builtin.rs, id builtin): it measures the photo's noise level once, from blocks of all four colour planes spread over the whole mosaic (variance = a · signal + b; see How the noise is measured), turns the noise into unit variance with a variance-stabilising transform, and cleans the four Bayer planes together with block-matching collaborative filtering (crates/denoise-core/src/bm3d.rs, after the published BM3D / CBM3D method). The four planes are rotated into a brightness plane and three colour-difference planes; 4 × 4-cell blocks of the brightness plane that look alike are stacked into groups (the colour planes follow the same matches, so colour noise is cleaned with the brightness). Each stack is transformed (a 2-D DCT of every block, a Haar transform across the stack), small coefficients are removed, and the blocks are put back with weights; a second step matches blocks again on that cleaner estimate and shrinks by a Wiener filter built from it. How hard it thresholds follows the photo's noise: gently at low noise, to keep fine detail, and harder as the noise grows. It then demosaics with edge-directed green and colour-difference red and blue. It is written for LightCraft, pure Rust, MIT OR Apache-2.0, has no weights and no download, and uses the same tiling, cache, queue, preview and export path as a model. Its tile is 256 cells with a 16-cell overlap. The filter (the grouping, the transforms and the putting back, which are nearly all of its work) runs on the graphics card when there is one that agrees with the processor and is faster (crates/gpu/src/bm3d.rs, WGSL kernels in crates/gpu/src/wgsl/bm3d_*.wgsl); measuring the noise, the variance-stabilising transform and the demosaic stay on the processor. The card is set up for the first photo that may use it, like a model's: its kernels are built, a check tile is run on both devices and the card's filter must match the processor's to within a third of a noise deviation at the worst place and a thousandth on average (they differ by the rounding of the arithmetic: a few blocks whose match or threshold falls either side of a tie change by about a tenth), and both are timed. Settings ▸ AI Denoise ▸ Run on chooses Automatic (the card where it is faster), Graphics card or Processor, and says where it runs; a set-up that crashed or ran out of time is not tried again until you choose (Try the graphics card again). A tile the card gets wrong is run on the processor, and three stop the card for the session. On an RTX 4080 SUPER a tile takes 22 ms on the card against 408 ms on one processor thread, and the 1536 × 1024 benchmark crop 0.13 to 0.24 s against 1.0 to 2.1 s on four threads, with the very same scores. A change to the algorithm bumps builtin::VERSION, so cached pictures are made again. The id builtin is reserved: a model with that id is refused. It is a classical filter, not a trained network, so a good model can still do better on very noisy files. The earlier non-local-means filter (crates/denoise-core/src/nlm.rs) is still there as a fallback (Tuning::bm3d = None). Why block matching and not something else: see Why block matching.

To measure a change to it, run the opt-in benchmark on a public CC0 raw (see cargo xtask corpus): LC_DENOISE_RAW="$(git rev-parse --show-toplevel)/corpus/raw/arw-sony-a7m3-compressed.arw" LC_DENOISE_OUT=/tmp/dn cargo test --release -p lightcraft-engine --test denoise_builtin_quality -- --ignored --nocapture (the path must be absolute: tests run from the crate's folder). It adds noise of four known levels to the raw, runs the denoiser and prints the PSNR against the original, plus a texture figure (how much of the true picture's fine structure the result keeps; noise can't inflate it); LC_DENOISE_OUT also writes truth | noisy | denoised PNGs. Use public files only. LC_B_N, LC_B_STEP, LC_B_S, LC_B_K, LC_B_T1, LC_B_T2, LC_B_LOFF (added to every point of the threshold curve), LC_B_SIG and LC_B_WIENER override the matching bm3d::Params field for a sweep; LC_NLM=1 runs the non-local-means filter instead (then LC_S1, LC_S2, LC_HL, LC_HH, LC_H2, LC_LIGHT, LC_HEAVY, LC_SC, LC_HC, LC_P1, LC_P2, LC_LOOSE and LC_FEW override its nlm::Tuning fields). PSNR and texture trade off (smoothing more raises one and lowers the other), so judge a change on both and on the crops. On the Sony A7 III crop (PSNR gain over the noisy picture in dB / texture, light, medium, heavy, extreme noise): non-local means +3.70 / 0.62, +7.22 / 0.42, +9.69 / 0.22, +10.37 / 0.09; block matching +4.28 / 0.58, +7.60 / 0.44, +10.04 / 0.22, +10.62 / 0.12, in about the same time (1 to 2 s for the 1536 × 1024 crop on 4 cores). LC_GPU=1 (with --features denoise) runs the block-matching filter on the graphics card instead: the same scores in 0.13 to 0.24 s on an RTX 4080 SUPER. The tests lightcraft-gpu bm3d::tests (the card against bm3d::filter on synthetic tiles, shared between threads) and the engine's gpu_tests (set-up check, fallback after wrong tiles, the processor chosen, a marker left behind) need a graphics card and record a skip without one.

Why block matching

The built-in filter was chosen by measuring, on the benchmark above (the public Sony A7 III crop with noise of four known levels; PSNR gain over the noisy picture in dB / texture), not by taste. Candidates, all written from the published papers (no GPL code), in the order tried:

Filter	light	medium	heavy	extreme	Time per crop (4 CPU threads)
Non-local means (the first built-in)	+3.70 / 0.62	+7.22 / 0.42	+9.69 / 0.22	+10.37 / 0.09	1.1–1.7 s
Block matching, hard threshold then Wiener (BM3D / CBM3D)	+4.28 / 0.58	+7.60 / 0.44	+10.04 / 0.22	+10.62 / 0.12	1.1–2.1 s
Weighted nuclear norm minimisation (WNNM), best setting per level	+4.68 / 0.56	+8.14 / 0.39	+10.04 / 0.22	+10.35 / 0.15	15–20 s

• Block matching beats non-local means on PSNR at every level and on texture at medium, heavy and extreme noise (only light noise keeps slightly less: 0.58 against 0.62, for 0.6 dB more). The crops show more needle and branch detail and no colour blotches or isolated specks. It costs about the same on the processor.
• WNNM, the strongest classical method we know of (singular values of each group of blocks shrunk with weights), gains about 0.4 dB at light noise and 0.03 of texture at medium and extreme, and nothing at heavy noise, for about ten times the processing time (a 24 MP photo would take about five minutes on the processor, against about a quarter of a minute). Its singular values came from Jacobi rotations of each group's Gram matrix; more iterations, a smaller feedback and 6 × 6 blocks (115 s a crop) did not help. Not kept. Untried: patches over all four planes at once (about fourteen times the work per group).
• What the tuning showed (for whoever changes it): the hard threshold has to rise with the noise (Params::lambda is a curve of the noise's deviation at mid-grey against the threshold: 1.2 at 0.0028, 1.9 at 0.0062, 2.3 at 0.0094, 2.65 at 0.0152, 3.2 at 0.038, 3.5 at 0.14; a fixed value wins at one level and loses at another, see Fitted on real cameras); a higher threshold on the colour planes than on the brightness plane hurts both PSNR and texture, because the colour differences carry real signal; 4 × 4 blocks beat 3, 5, 6 and 8 on quality (3 × 3 is a little faster, 6 × 6 and 8 × 8 much slower); the match tolerances and the group size (16 against 8 or 32) barely matter; a search of 8 cells and a step of 3 cost 0.1 dB against 12 and 2 and keep the processor as fast as non-local means. The Wiener step adds a little over the hard threshold alone.
• The first benchmark's limit: its noise is synthetic and white, and its reference (a public raw's own mosaic) has noise of its own, which hides what happens at the low noise levels of real cameras. Real high-ISO raws also have correlated, low-frequency colour noise and banding, which none of this measures; judge a change to the real thing by eye on a high-ISO raw too.

How the noise is measured

builtin::estimate_noise reads (a, b) of a photo (variance = a · signal + b, signal in 0..1 after black and white levels) from the picture itself, so no camera table is needed. Each of the four colour planes is read separately (they sit at different levels in a raw), cut into 12 × 12-cell blocks, and a second-difference operator ([1 -2 1; -2 4 -2; 1 -2 1], which cancels flat and smoothly sloping signal; white noise has 36 σ² of variance through it) gives each block's variance; texture and edges only add to it. At each brightness (about 300 blocks of each) the quietest 5 % of blocks are the noise, divided by the bias of that quantile of pure noise (0.67, measured by denoise_noise_estimate::a_quantile_of_pure_noise_blocks_is_a_known_fraction_of_the_true_variance), and a line is fitted through the points. The first version mixed the colour planes in each block and took the lower quarter of a plain neighbour difference: it read the picture's texture as noise, never below about 0.003 at mid-grey whatever the true noise (estimate_noise_v1, kept to compare). On 30 different public pictures with known noise added, the ratio of estimated to true deviation at mid-grey, at the ISO 6400 level of a Canon R-series body (0.0094), had a 10th to 90th percentile of 0.97 to 1.29 against 0.94 to 1.96 for the first version, and at the ISO 1600 level (0.0062) 0.99 to 1.45 against 0.97 to 2.77 (LC_CORPUS=<corpus/raw> cargo test --release -p lightcraft-engine --test denoise_noise_estimate -- --ignored --nocapture). A picture with no flat area at all still reads somewhat high, never low.

Fitted on real cameras

A Canon EOS R, EOS R6 Mark III, M50 Mark II and some EF-era bodies were measured on a private library of about 8,400 raws (only aggregates were kept; no photo is in the repository). The deviation of the noise at mid-grey (the measure the filter judges a photo by) was about 0.0024 to 0.0028 at ISO 400 and below, 0.005 to 0.006 at ISO 500 to 1600, 0.009 to 0.010 at ISO 2000 to 6400, 0.015 to 0.018 at ISO 8000 to 25600 and 0.038 above that: far lower than the first benchmark's levels (0.024 to 0.14), so a filter tuned on those was too gentle exactly where these photos live. The curve above was fitted on clean low-ISO photos of those cameras with noise of their measured levels added (LC_REAL=1; LC_FULL=1 measures the noise on the whole frame, as the app does). Against the first estimator with a constant threshold of 1.9, averaged over 10 EOS R and 5 R6 Mark III photos, the output reached (PSNR in dB, EOS R / R6 Mark III): ISO 400 level 43.76 / 44.93 (from 43.27 / 44.66), 1600 41.03 / 41.86 (41.05 / 41.82), 6400 39.93 / 40.69 (39.30 / 39.74), 25600 38.36 / 38.86 (36.18 / 36.21) and 51200 level 33.83 / 34.45 (28.42 / 28.78), with the noise left in the result ("detail" above 1) falling from 1.3 to 3.1 down to 0.73 to 0.99. The threshold at each level is the lowest that stays within about a quarter of a decibel of the best, so texture is kept (on the EOS R at the 25600 level the texture figure is 0.28, against 0.25 at the highest-PSNR threshold). The first estimator also cost PSNR by itself: told the true noise instead of its estimate, the same filter did the right thing at every level; the new estimator is as good as that from ISO 6400 up.

Models and terms (optional)

No model weights are bundled. Choosing a model replaces the built-in denoiser; Stop using (or removing the model) goes back to it. Models live in the per-user denoise-models folder (override with LIGHTCRAFT_DENOISE_MODELS). Settings provides Install from file… for a user-supplied ONNX file with a denoise-model.json beside it. The default build offers no model downloads. The separate rawnind-model Cargo feature enables the pinned RawNIND offer only when a release's model policy permits it. Terms must be acknowledged before installation or download. Downloads use the shared pure-Rust lightcraft-fetch client, with resume, progress, cancellation, size limits and SHA-256 verification.

RawNIND's optional ONNX weights are GPL-3.0, as recorded in the pinned model catalog. They are downloaded separately after consent and read by our own Rust executor. Whether releases should offer this download remains a maintainer policy decision; permissively licensed or independently trained weights remain a gap. This is not a claim that optional downloading settles licensing questions. No GPL decoder or neural runtime code is included. Model metadata is in crates/denoise-core/src/known.rs.

Installation checks the manifest, model hash and size, then runs a synthetic self-test: finite output of the promised shape, repeatability, the declared scale and reduced noise. Unsupported operators, malformed files and a failed self-test return errors. The UI performs copying, extraction, hashing, parsing and CPU/GPU self-tests on one background installation worker. denoise.models.downloads and denoise.pump report installing, installed or failed; an existing model is preserved on a failed replacement. Command clients can request the same asynchronous path with background: true. The synchronous command remains available for scripts that require the result immediately.

The current contract is bayerToRgb: normalized, black-subtracted, non-white-balanced RGGB planes [1, 4, tile, tile] produce camera RGB [1, 3, 2*tile, 2*tile]. RawNIND uses a 512-cell tile, 64-cell overlap and a mean-matched nominal gain of 1e6. See DenoiserManifest for the validated JSON schema.

Photos, previews and exports

Original RAW files stay untouched. A cleaned picture is cached under <library>/denoise/<key>.lcdn, keyed by source content, model and algorithm version. The compressed half-float cache has checked headers, bounded windows and checksums. It can be cleared or recreated; it is not a new catalog photo or a duplicate DNG.

Plain and cleaned camera RGB go through the same highlight reconstruction, white balance, colour matrix, DNG profile tables, camera hue/saturation look, crop and orientation. Amount blends the two before the remaining develop pipeline, so edits and exports use the same data. At zero Amount or with Detail disabled, the ordinary render path remains in use.

The queue prioritizes the open and selected photos, pauses after input, and adjusts its worker count while the user works. Progress, cancellation and retry are available in Settings and through commands. The default cache limit is 20 GB; unused products are evicted oldest first. A failed file is not continuously retried.

An export with a nonzero effective Amount creates a missing product synchronously. It returns a clear error if it cannot produce the denoised result; it never silently exports the plain image instead. Unsupported sensor layouts retain manual noise reduction and report their limitation.

Pure-Rust CPU and GPU execution

The CPU executor reads a bounded subset of ONNX into checked network data. Convolutions use bounded im2col blocks and faer-core 0.17.1 with its pure-Rust GEMM backend, without BLAS, C dependencies or an inner thread pool. That backend includes architecture-specific assembly in its dependencies. Tiles share the outer worker pool. A scalar implementation checks the optimized operators on synthetic networks. Tract is not a workspace dependency or a product feature. CPU inference and the GPU model module are feature-gated; ordinary engine/web builds do not compile the faer/GEMM backend. The engine depends on the inference crate only through denoise = ["dep:lightcraft-denoise", …]; lightweight lightcraft-denoise-core metadata and cache formats remain available without inference. Pictures are capped at 110 megapixels; at most four tile calls run together, chosen within the process-wide working-memory budget. One idle CPU workspace is retained, and cache costs reserve room for the blended picture.

The GPU executor uses our WGSL kernels through wgpu. It checks a real model's output against the CPU and falls back on errors, unsupported networks or a slower device. Setup has a time limit and a crash marker. Automatic selection estimates throughput from a check tile; this estimate can choose imperfectly on unmeasured hardware. Settings offers Automatic, Graphics card and Processor. GPU tests cover supported synthetic networks against the scalar executor; a real-model test compares CPU/GPU.

The convolution kernel stores its shared input tile as scalar elements, each with one writer, and gathers vectors after a workgroup barrier. Writing separate lanes of the same shared vector from different invocations is not safe under WGSL (component references); this caused incorrect Metal results (issue #479). Synthetic equivalence tests include repeated partial tiles and concurrent calls, and reject non-finite outputs.

Historical performance, measured 2026-10-07

These are the contributor's measurements before the current upstream integration, on a Ryzen 9 7945HX and RTX 4090 Laptop GPU (DX12), release build, model SHA-256 da27509dab6a2915da67e988acd86cf71f9d5bbc8d1aa0ed32933578a887b901. The 24.34 MP Sony a7 III photo is 35 tiles. Numbers exclude model loading, source decode, GPU setup, cache encoding and export, but include packing and blending.

Work	Pure-Rust CPU	Prior Tract reference	GPU
One warm 512-cell tile, one caller	1148 ms	1463 ms	15.1 ms
Whole photo, 16 CPU / 4 GPU feeding workers	5.185 s	9.534 s	0.582 s

The largest relative differences were 1.68e-6 CPU/reference and 1.57e-6 CPU/GPU for the check tile, below the required 1e-3 tolerance. Reference comparisons were made on the older contributor branch; that reference runtime is intentionally excluded from the current workspace. These numbers are not a fresh benchmark of this rebase.

Fresh checks can be run using an already installed model and CC0 raw samples:

LC_DENOISE_MODEL=<model.onnx> cargo test --release -p lightcraft-gpu --lib real_model_on_the_gpu_matches_cpu -- --ignored --nocapture
LC_DENOISE_MODEL=<model.onnx> LC_DENOISE_RAW=<corpus/raw> cargo test --release -p lightcraft-engine --features rawnind-model --test it denoise_cpu:: -- --ignored --nocapture
LC_DENOISE_MODEL=<model.onnx> LC_DENOISE_RAW=<corpus/raw> cargo test --release -p lightcraft-engine --features rawnind-model --test it denoise_real:: -- --ignored --nocapture

Another computer's graphics card

A laptop can hand tiles to a desktop with a bigger graphics card over a LAN or a VPN. The server runs the installed, selected model the way the app does (the card when it can run it, checked against the CPU first; else the CPU):

export LIGHTCRAFT_DENOISE_TOKEN='a long random string, at least 16 characters'
lightcraft-cli denoise-serve --listen 100.101.102.103:7990          # e.g. the machine's Tailscale address

On the laptop, before LightCraft starts:

export LIGHTCRAFT_DENOISE_URL=100.101.102.103:7990
export LIGHTCRAFT_DENOISE_TOKEN='…the same token…'

When the laptop first uses the model it greets the server and times a check tile there and back. If that is faster than a tile on the laptop (its graphics card, or its processor with the tiles it runs at once), the server's slots (four tiles in flight) are added to the laptop's own: a tile goes to the server when one of its slots is free and runs locally otherwise. If it is slower, nothing is sent, so a photo never waits for the slower machine: denoise.status → device.remote.state is slower, with both times (remoteMs, localMs). Measured with RawNIND: an Apple M3 laptop does a tile in 116 ms, an RTX 3060 Ti reached over Tailscale 345 ms there and back, so that laptop keeps its tiles; one without a fast card hands them over. A tile the server fails runs locally; after three failures the server is no longer used for that model. Both ends must run the same model file: the client compares the manifest's SHA-256 (or the file's) when it first greets the server, and the server checks it again for every tile. denoise.status → device.remote reports the server's address, its device and whether it is used, or why not.

The protocol is not encrypted. Listen on a VPN address (Tailscale, WireGuard) or a LAN, or bind every interface behind a firewall rule that admits only the VPN; never expose the port to the internet. Requests carry the shared token (compared in constant time); the server refuses other models, other tile sizes, oversized headers and payloads (also ones that inflate past their tile) and caps concurrent connections (--max-connections, default 8).

Protocol version 1 (crates/denoise/src/remote.rs): one JSON line per message, then a payload of bytes bytes — the tile's samples as little-endian half floats, byte-shuffled (low bytes, then high bytes) and deflated:

→ {"v":1,"token":"…","op":"hello"}
← {"ok":true,"v":1,"device":"GPU: …","model":{"id":"…","version":"…","sha256":"…"},"tile":T}
→ {"v":1,"token":"…","op":"run","model":"<sha256>","tile":T,"bytes":N}   + N bytes (4 planes of T × T cells)
← {"ok":true,"tile":T,"bytes":N}                                        + N bytes (3 planes of 2T × 2T pixels)
← {"ok":false,"error":"…"}

Commands and limits

UI, CLI, control channel and MCP dispatch denoise.toggle, denoise.settings, denoise.queue, .cancel, .status, .pump, .clear, and denoise.models.* (list, install, download, downloads, downloadCancel, test, remove, select). The develop control is enhance.denoise.

Native apps enable inference through their denoise feature. A build without it and the web build report inference unavailable. Manifest/cache primitives still compile for WASM. Settings and models remain opt-in; nothing is downloaded on launch.

This implementation supports Bayer RAW only. X-Trans, Foveon, demosaiced DNG and non-RAW images remain unsupported by this model contract. No measured quality parity with Lightroom is claimed. Apple/Metal and wider hardware verification, model policy, a linear-RGB contract and a side-by-side fidelity suite remain future work.