From 897f9f37ada9e363b1b660191ff28f1612819b99 Mon Sep 17 00:00:00 2001 From: JohnnYoru Date: Sat, 10 Oct 2026 06:19:11 -0300 Subject: [PATCH] Add the Remove tool with non-local patch completion (#2193) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add the Remove tool with non-local patch completion Tool::Remove (J flyout) paints a hole that paint.remove fills from its surroundings as a background job, in one undo step. A stroke around an object also removes what it encloses; the selection limits the hole. The fill (photocraft_algo::nonlocal) follows Newson, Almansa, Gousseau and Pérez (IPOL 2017): texture features in the patch distance, onion-peel initialisation, match fields upsampled between levels and a best-patch final copy, plus gradient-domain hiding of edge and interior seams. Sparse holes are filled in tiles. inpaint::complete (Content-Aware Fill, Spot Healing, Content-Aware Move) is unchanged. On 20 holes in public-domain photos (remove_quality) the fill keeps 0.89 of the texture it replaces, against 0.54 with the Wexler/PatchMatch completion. 24 MP (remove_bench): 100 px ring 120 ms, 1000 px ring 2.5 s. * Translate the Remove tool's label and tighten comments Adds "Remove Tool" to every UI catalog, following each one's pattern for tool names and its existing word for Remove. Corrects comments that no longer matched the code: the benchmark's description, the tile and margin minimums, and the reasons given for two parameters. * Put the Remove tool first in the J group The J flyout's slot now shows Remove until another J tool is picked, and J from another group selects it; repeated J goes on to Spot Healing Brush, Healing Brush, Patch, Content-Aware Move and Red Eye. --------- Co-authored-by: Brandon Thomas --- ATTRIBUTION.md | 3 +- README.md | 4 +- assets/icons/bandage-sparkle.svg | 21 + crates/algo/src/inpaint.rs | 8 +- crates/algo/src/lib.rs | 2 + crates/algo/src/nonlocal.rs | 1015 +++++++++++++++++ crates/algo/src/remove.rs | 310 +++++ crates/engine/examples/remove_bench.rs | 35 + crates/engine/examples/remove_quality.rs | 217 ++++ crates/engine/src/channel_cmds.rs | 1 + crates/engine/src/jobs_tests.rs | 21 + crates/engine/src/retouch_cmds.rs | 15 +- .../src/retouch_cmds/content_aware_move.rs | 12 +- crates/engine/src/retouch_cmds/remove.rs | 153 +++ crates/engine/src/retouch_cmds/tests.rs | 100 +- crates/ui-egui/src/canvas.rs | 8 +- crates/ui-egui/src/i18n/cs.tsv | 1 + crates/ui-egui/src/i18n/de.tsv | 1 + crates/ui-egui/src/i18n/el.tsv | 1 + crates/ui-egui/src/i18n/es.tsv | 1 + crates/ui-egui/src/i18n/fr.tsv | 1 + crates/ui-egui/src/i18n/id.tsv | 1 + crates/ui-egui/src/i18n/it.tsv | 1 + crates/ui-egui/src/i18n/ja.tsv | 1 + crates/ui-egui/src/i18n/ko.tsv | 1 + crates/ui-egui/src/i18n/nl.tsv | 1 + crates/ui-egui/src/i18n/pl.tsv | 1 + crates/ui-egui/src/i18n/pt-br.tsv | 1 + crates/ui-egui/src/i18n/ru.tsv | 1 + crates/ui-egui/src/i18n/uk.tsv | 1 + crates/ui-egui/src/i18n/zh-hans.tsv | 1 + crates/ui-egui/src/i18n/zh-hant.tsv | 1 + crates/ui-egui/src/icon_data.rs | 1 + crates/ui-egui/src/icons.rs | 1 + crates/ui-egui/src/panels.rs | 4 +- crates/ui-egui/src/retouch_ui.rs | 32 +- crates/ui-egui/src/state.rs | 13 +- docs/development.md | 21 + docs/roadmap.md | 10 + scorecard/tools.toml | 7 + 40 files changed, 2005 insertions(+), 25 deletions(-) create mode 100644 assets/icons/bandage-sparkle.svg create mode 100644 crates/algo/src/nonlocal.rs create mode 100644 crates/algo/src/remove.rs create mode 100644 crates/engine/examples/remove_bench.rs create mode 100644 crates/engine/examples/remove_quality.rs create mode 100644 crates/engine/src/retouch_cmds/remove.rs diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index 9e80593c91..dfc31372f2 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -20,12 +20,13 @@ next to them. | `crates/ui-egui/src/i18n/uk.tsv` | Ukrainian UI translations | PhotoCraft contributors | Original translations of PhotoCraft's English labels | MIT OR Apache-2.0, [`LICENSE-translations.txt`](crates/ui-egui/src/i18n/LICENSE-translations.txt) | | `assets/fonts/Inter-Regular.ttf`, `Inter-Medium.ttf`, `Inter-SemiBold.ttf` | Inter 4.001 (UI font) | The Inter Project Authors (Rasmus Andersson) | | SIL OFL 1.1, [`assets/fonts/OFL-Inter.txt`](assets/fonts/OFL-Inter.txt) | | `assets/fonts/JetBrainsMono-Regular.ttf` | JetBrains Mono 2.305 (numeric font) | The JetBrains Mono Project Authors | | SIL OFL 1.1, [`assets/fonts/OFL-JetBrainsMono.txt`](assets/fonts/OFL-JetBrainsMono.txt) | -| `assets/icons/*.svg` except `slice-knife.svg`, `clip-below.svg`, `clip-release.svg`, `layer-mask.svg`, `adjustment-layer.svg`, `eraser-background.svg`, `eraser-magic.svg`, `lasso-magnetic.svg`, `direct-select.svg`, `painting-symmetry.svg` and `symmetry-*.svg` (110 files) | Lucide icons | Lucide Icons and Contributors | (`icons/.svg`; `align-*` are now named `text-align-*` upstream) | ISC, [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | +| `assets/icons/*.svg` except `slice-knife.svg`, `clip-below.svg`, `clip-release.svg`, `layer-mask.svg`, `adjustment-layer.svg`, `eraser-background.svg`, `eraser-magic.svg`, `lasso-magnetic.svg`, `direct-select.svg`, `bandage-sparkle.svg`, `painting-symmetry.svg` and `symmetry-*.svg` (110 files) | Lucide icons | Lucide Icons and Contributors | (`icons/.svg`; `align-*` are now named `text-align-*` upstream) | ISC, [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | | `assets/icons/{check,chevron-down,chevron-right,chevron-up,chevrons-left,chevrons-right,circle,clock,compass,info,link,link-2,lock,minus,monitor,moon,move,navigation,plus,search,square,trash,triangle,type,x,zoom-in}.svg` | Lucide icons derived from Feather (subset of the row above) | Cole Bemis (Feather), Lucide Contributors | via Lucide | MIT (Feather) and ISC (Lucide), [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | | `assets/icons/slice-knife.svg` | Slice tool glyph | PhotoCraft contributors | Original work | MIT OR Apache-2.0 | | `assets/icons/clip-below.svg`, `assets/icons/clip-release.svg` | Clipping-mask cursors (⌥ over the line between two layers): a hooked arrow onto the layer below; the same with a small "no" badge to release | PhotoCraft contributors | Original work | MIT OR Apache-2.0 | | `assets/icons/layer-mask.svg`, `assets/icons/adjustment-layer.svg` | Layers panel footer glyphs: Add a mask (a frame around a solid disc) and New Fill or Adjustment Layer (a circle, one half solid), drawn on Lucide's 24 px grid and stroke | PhotoCraft contributors | Original work | MIT OR Apache-2.0 | | `assets/icons/eraser-background.svg`, `assets/icons/eraser-magic.svg` | Background Eraser and Magic Eraser tool glyphs: Lucide `eraser` plus added marks | Lucide Icons and Contributors; marks by PhotoCraft contributors | Derived from `icons/eraser.svg` | ISC, [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | +| `assets/icons/bandage-sparkle.svg` | Remove tool glyph: Lucide `bandage` moved down, plus a sparkle mark | Lucide Icons and Contributors; mark by PhotoCraft contributors | Derived from `icons/bandage.svg` | ISC, [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | | `assets/icons/direct-select.svg` | Direct Selection tool glyph: Lucide `mouse-pointer-2` plus an anchor square | Lucide Icons and Contributors; square by PhotoCraft contributors | Derived from `icons/mouse-pointer-2.svg` | ISC, [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | | `assets/icons/painting-symmetry.svg` | Painting symmetry glyph: Lucide `heart` with a central axis | Lucide Icons and Contributors; adapted by @shorbank | Derived from `icons/heart.svg` | ISC, [`assets/icons/LICENSE-lucide.txt`](assets/icons/LICENSE-lucide.txt) | | `assets/icons/symmetry-{vertical,horizontal,dual,diagonal}.svg` | Painting symmetry axis glyphs | @shorbank | Original work; supplied SVG line geometry centred on a common square canvas, stroke adjusted to match the painting-symmetry glyph | MIT OR Apache-2.0, [`assets/icons/LICENSE-symmetry-axes.txt`](assets/icons/LICENSE-symmetry-axes.txt) | diff --git a/README.md b/README.md index eba47a9164..c378d6f922 100644 --- a/README.md +++ b/README.md @@ -178,8 +178,8 @@ Every screenshot here is the real app at work on public-domain art, rendered off
-

🧰 52 tools

- Move · Rectangular and Elliptical Marquee · Lasso · Polygonal Lasso · Magnetic Lasso · Object Selection · Quick Selection · Magic Wand · Crop · Slice · Slice Select · Eyedropper · Ruler · Note · Count · Spot Healing Brush · Healing Brush · Patch · Content-Aware Move · Red Eye · Brush · Pencil · Mixer Brush · Clone Stamp · Pattern Stamp · History Brush · Eraser · Background Eraser · Magic Eraser · Gradient · Paint Bucket · Blur · Sharpen · Smudge · Dodge · Burn · Sponge · Pen · Horizontal and Vertical Type · Path Selection · Direct Selection · Rectangle · Ellipse · Triangle · Polygon · Line · Custom Shape · Hand · Rotate View · Zoom +

🧰 53 tools

+ Move · Rectangular and Elliptical Marquee · Lasso · Polygonal Lasso · Magnetic Lasso · Object Selection · Quick Selection · Magic Wand · Crop · Slice · Slice Select · Eyedropper · Ruler · Note · Count · Remove · Spot Healing Brush · Healing Brush · Patch · Content-Aware Move · Red Eye · Brush · Pencil · Mixer Brush · Clone Stamp · Pattern Stamp · History Brush · Eraser · Background Eraser · Magic Eraser · Gradient · Paint Bucket · Blur · Sharpen · Smudge · Dodge · Burn · Sponge · Pen · Horizontal and Vertical Type · Path Selection · Direct Selection · Rectangle · Ellipse · Triangle · Polygon · Line · Custom Shape · Hand · Rotate View · Zoom

🖌️ A real brush engine

diff --git a/assets/icons/bandage-sparkle.svg b/assets/icons/bandage-sparkle.svg new file mode 100644 index 0000000000..9c7bf22cb9 --- /dev/null +++ b/assets/icons/bandage-sparkle.svg @@ -0,0 +1,21 @@ + + + + + + + + + + + diff --git a/crates/algo/src/inpaint.rs b/crates/algo/src/inpaint.rs index b777bf4d77..6708c94293 100644 --- a/crates/algo/src/inpaint.rs +++ b/crates/algo/src/inpaint.rs @@ -29,12 +29,12 @@ const PAR_MIN: usize = 128 * 128; /// SplitMix64: tiny deterministic RNG. #[derive(Clone)] -struct Rng(u64); +pub(crate) struct Rng(u64); impl Rng { - fn new(seed: u64) -> Self { + pub(crate) fn new(seed: u64) -> Self { Rng(seed ^ 0x9E37_79B9_7F4A_7C15) } - fn next(&mut self) -> u64 { + pub(crate) fn next(&mut self) -> u64 { self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); let mut z = self.0; z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); @@ -42,7 +42,7 @@ impl Rng { z ^ (z >> 31) } /// Uniform integer in `lo..=hi`. - fn range(&mut self, lo: i32, hi: i32) -> i32 { + pub(crate) fn range(&mut self, lo: i32, hi: i32) -> i32 { if hi <= lo { return lo; } diff --git a/crates/algo/src/lib.rs b/crates/algo/src/lib.rs index bfa7eb754b..96ef9d4df7 100644 --- a/crates/algo/src/lib.rs +++ b/crates/algo/src/lib.rs @@ -42,6 +42,7 @@ pub mod liquify; pub mod magnetic; pub mod matting; mod noise; +pub mod nonlocal; mod oil; mod other; mod other2; @@ -56,6 +57,7 @@ pub mod pyramid; pub mod quantize; pub mod redeye; mod relight; +pub mod remove; mod render; pub mod render2; pub mod resample; diff --git a/crates/algo/src/nonlocal.rs b/crates/algo/src/nonlocal.rs new file mode 100644 index 0000000000..148a1bd32d --- /dev/null +++ b/crates/algo/src/nonlocal.rs @@ -0,0 +1,1015 @@ +//! Non-local patch-based completion, the Remove Tool's fill: A. Newson, A. Almansa, Y. Gousseau, +//! P. Pérez, *Non-Local Patch-Based Image Inpainting*, Image Processing On Line 7 (2017), 373–385. +//! Like [`crate::inpaint::complete`] it alternates, coarse to fine, a PatchMatch search for each +//! patch's nearest fully-known patch (C. Barnes et al., SIGGRAPH 2009) with a similarity-weighted +//! vote of the matches (Y. Wexler, E. Shechtman, M. Irani, TPAMI 2007). What it changes: +//! +//! * **Texture features** join the patch distance: the local mean of `|∂x|` and `|∂y|` (after +//! Liu and Caselles), weighted by [`Params::lambda`]. Colour alone can't tell grass from a blurred +//! background of the same mean colour, and a smooth guess in the hole then attracts smooth +//! sources until the fill is a blur. The features are filled along with the colour. +//! * **Onion-peel initialisation** at the coarsest level: the hole is filled one boundary layer at +//! a time from partially known patches, which carries structures in where a harmonic first guess +//! would blur them. +//! * **The match field is upsampled** between levels, not the image: each finer level is rebuilt +//! from its own full-resolution sources instead of from an enlarged blur. +//! * **Each final pixel comes from the best patch covering it** instead of the weighted vote, so +//! the fill keeps the grain of its sources. +//! * **Seams are hidden in the gradient domain** (not in the paper): a membrane carries the colour +//! mismatch along the hole's edge into the fill (P. Pérez et al., *Poisson Image Editing*, 2003), +//! and steps between areas copied from different places, where texture doesn't hide them, are +//! spread out by a smooth correction (after S. Darabi et al., *Image Melding*, 2012). +//! * **The number of levels follows the hole's thickness** (`log2(2·No/N)`, `No` the erosions that +//! empty the hole, `N` the patch side), not its extent, so long thin holes stay at fine scales. +//! +//! Buffers are interleaved `w × h × ch` normalised floats (any colour model and depth) with a +//! `w × h` hole mask. Deterministic for a seed, whatever the thread count. + +use crate::inpaint::Rng; +use photocraft_raster::{Cancelled, Interrupt}; + +/// Below this many grid cells a level runs single-threaded (thread hand-off costs more than it saves). +#[cfg(not(target_arch = "wasm32"))] +const PAR_MIN: usize = 128 * 128; + +/// Parameters for [`complete_with`]. The defaults are the paper's, except for the PatchMatch +/// passes per iteration (two instead of ten, for speed). +#[derive(Clone, Debug, PartialEq)] +pub struct Params { + /// Patch radius (patches are `2r+1` square). + pub patch_radius: usize, + /// Weight of the texture features against the colour in the patch distance. + pub lambda: f32, + /// Most EM iterations (search, then vote) per level. + pub max_iters: usize, + /// PatchMatch passes per EM iteration. + pub pm_passes: usize, + /// A level is done once an iteration changes the hole by less than this on average (0..1 units). + pub tolerance: f32, + pub seed: u64, +} + +impl Default for Params { + fn default() -> Self { + Self { patch_radius: 3, lambda: 50.0, max_iters: 10, pm_passes: 2, tolerance: 0.1 / 255.0, seed: 1 } + } +} + +/// One pyramid level: `cha` channels per cell (the colour, then the two texture features). +struct Level { + w: usize, + h: usize, + img: Vec, + hole: Vec, +} + +/// Fully-known patch centres (the whole patch on the grid and outside the hole), as a mask and a list. +fn valid_sources(w: usize, h: usize, hole: &[bool], r: usize) -> (Vec, Vec<(i32, i32)>) { + let mut valid = vec![false; w * h]; + let mut list = Vec::new(); + if w < 2 * r + 1 || h < 2 * r + 1 { + return (valid, list); + } + // Summed-area table of the hole. + let w1 = w + 1; + let mut sat = vec![0u32; w1 * (h + 1)]; + for y in 0..h { + let mut row = 0u32; + for x in 0..w { + row += u32::from(hole[y * w + x]); + sat[(y + 1) * w1 + x + 1] = sat[y * w1 + x + 1] + row; + } + } + for y in r..h - r { + for x in r..w - r { + let (x0, y0, x1, y1) = (x - r, y - r, x + r + 1, y + r + 1); + if sat[y1 * w1 + x1] + sat[y0 * w1 + x0] == sat[y0 * w1 + x1] + sat[y1 * w1 + x0] { + valid[y * w + x] = true; + list.push((x as i32, y as i32)); + } + } + } + (valid, list) +} + +/// The texture features at every known cell: the mean of `|∂x|` and of `|∂y|` of the channel mean +/// over a `side`-wide window, counting only derivatives between two known cells. Hole cells get 0 +/// (the completion fills them in). +fn texture_features(w: usize, h: usize, ch: usize, img: &[f32], hole: &[bool], side: usize) -> (Vec, Vec) { + let g: Vec = img.chunks_exact(ch).map(|p| p.iter().sum::() / ch as f32).collect(); + let w1 = w + 1; + // Summed-area tables of each derivative and of where it is defined. + let table = |dx: usize, dy: usize| { + let mut sum = vec![0.0f64; w1 * (h + 1)]; + let mut cnt = vec![0u32; w1 * (h + 1)]; + for y in 0..h { + let (mut rs, mut rc) = (0.0f64, 0u32); + for x in 0..w { + let i = y * w + x; + let (nx, ny) = (x + dx, y + dy); + if nx < w && ny < h && !hole[i] && !hole[ny * w + nx] { + rs += f64::from((g[ny * w + nx] - g[i]).abs()); + rc += 1; + } + sum[(y + 1) * w1 + x + 1] = sum[y * w1 + x + 1] + rs; + cnt[(y + 1) * w1 + x + 1] = cnt[y * w1 + x + 1] + rc; + } + } + let half = side / 2; + (0..w * h) + .map(|i| { + if hole[i] { + return 0.0; + } + let (x, y) = (i % w, i / w); + let (x0, y0, x1, y1) = (x.saturating_sub(half), y.saturating_sub(half), (x + half + 1).min(w), (y + half + 1).min(h)); + let s = sum[y1 * w1 + x1] + sum[y0 * w1 + x0] - sum[y0 * w1 + x1] - sum[y1 * w1 + x0]; + let n = cnt[y1 * w1 + x1] + cnt[y0 * w1 + x0] - cnt[y0 * w1 + x1] - cnt[y1 * w1 + x0]; + if n == 0 { 0.0 } else { (s / f64::from(n)) as f32 } + }) + .collect::>() + }; + (table(1, 0), table(0, 1)) +} + +/// The next coarser level: colour from a 3×3 Gaussian (σ = 1.5, as in the paper) over the known +/// cells, texture features by nearest-neighbour subsampling (blurring them would erase them), and a +/// cell is in the hole when any of the four it covers is. +fn downsample(l: &Level, cha: usize, ch: usize) -> Level { + const K: [f32; 3] = [0.6412, 0.8007, 0.6412]; + let (w, h) = (l.w.div_ceil(2), l.h.div_ceil(2)); + let mut img = vec![0.0f32; w * h * cha]; + let mut hole = vec![false; w * h]; + for y in 0..h { + for x in 0..w { + let (fx, fy) = (x * 2, y * 2); + let o = (y * w + x) * cha; + hole[y * w + x] = (0..2).any(|dy| (0..2).any(|dx| fx + dx < l.w && fy + dy < l.h && l.hole[(fy + dy) * l.w + fx + dx])); + let mut wsum = 0.0f32; + for (ky, oy) in (-1i32..=1).enumerate() { + for (kx, ox) in (-1i32..=1).enumerate() { + let (sx, sy) = (fx as i32 + ox, fy as i32 + oy); + if sx < 0 || sy < 0 || sx as usize >= l.w || sy as usize >= l.h { + continue; + } + let si = sy as usize * l.w + sx as usize; + if l.hole[si] { + continue; + } + let k = K[kx] * K[ky]; + for c in 0..ch { + img[o + c] += l.img[si * cha + c] * k; + } + wsum += k; + } + } + if wsum > 0.0 { + img[o..o + ch].iter_mut().for_each(|v| *v /= wsum); + } + let si = (fy * l.w + fx) * cha; + img[o + ch..o + cha].copy_from_slice(&l.img[si + ch..si + cha]); + } + } + Level { w, h, img, hole } +} + +/// How many levels: `⌈log2(2·No/N)⌉` (at least 1), `No` the hole's thickness in erosions and `N` +/// the patch side, while the coarsest level stays at least three patches across. +fn level_count(w: usize, h: usize, hole: &[bool], r: usize) -> usize { + let n = 2 * r + 1; + let no = crate::remove::hole_radius(w, h, hole).min(1 << 20); + let want = (2.0 * f64::from(no) / n as f64).log2().ceil().max(1.0) as usize; + let (mut lw, mut lh, mut levels) = (w, h, 1); + while levels < want && lw / 2 >= 3 * n && lh / 2 >= 3 * n { + (lw, lh, levels) = (lw.div_ceil(2), lh.div_ceil(2), levels + 1); + } + levels +} + +/// Per-level search and reconstruction state. +struct Solver<'a> { + w: usize, + h: usize, + cha: usize, + /// Colour channels (the first `ch` of `cha`): the convergence test looks at these only. + ch: usize, + r: i32, + img: Vec, + hole: &'a [bool], + valid: Vec, + valid_list: Vec<(i32, i32)>, + /// Patch centres whose patch overlaps the hole: the patches the energy is summed over. + targets: Vec, + ctl: &'a Interrupt<'a>, +} + +impl Solver<'_> { + #[inline] + fn is_valid(&self, x: i32, y: i32) -> bool { + x >= 0 && y >= 0 && (x as usize) < self.w && (y as usize) < self.h && self.valid[y as usize * self.w + x as usize] + } + + fn random_source(&self, rng: &mut Rng) -> (i32, i32) { + let n = self.valid_list.len().max(1) as u64; + self.valid_list.get((rng.next() % n) as usize).copied().unwrap_or((0, 0)) + } + + /// Mean squared difference between the target patch at `t` (clipped to the grid; with `known`, + /// only its known cells) and the source patch at `s`. Gives up (∞) once it can't beat `cutoff`. + fn dist(&self, t: (i32, i32), s: (i32, i32), cutoff: f32, known: Option<&[bool]>) -> f32 { + let (w, h, cha, r) = (self.w as i32, self.h as i32, self.cha, self.r); + let full = ((2 * r + 1) * (2 * r + 1)) as f32 * cha as f32; + let (mut sum, mut n) = (0.0f32, 0usize); + for dy in -r..=r { + let ty = t.1 + dy; + if ty < 0 || ty >= h { + continue; + } + for dx in -r..=r { + let tx = t.0 + dx; + if tx < 0 || tx >= w { + continue; + } + let tc = ty as usize * self.w + tx as usize; + if known.is_some_and(|k| !k[tc]) { + continue; + } + let (ti, si) = (tc * cha, ((s.1 + dy) as usize * self.w + (s.0 + dx) as usize) * cha); + for c in 0..cha { + let d = self.img[ti + c] - self.img[si + c]; + sum += d * d; + } + n += cha; + } + // The mean can only end up at or above sum / (full patch size). + if known.is_none() && sum > cutoff * full { + return f32::INFINITY; + } + } + if n == 0 { f32::INFINITY } else { sum / n as f32 } + } + + /// One PatchMatch pass over the targets, in parallel row bands (propagation stays in a band). + fn patchmatch(&self, nnf: &mut [(i32, i32)], cost: &mut [f32], pass: usize, seed: u64, radius: i32) { + const BAND: usize = 16; + let w = self.w; + let reverse = pass % 2 == 1; + let band = |bi: usize, nn: &mut [(i32, i32)], co: &mut [f32]| { + if self.ctl.cancelled() { + return; + } + let mut rng = Rng::new(seed ^ (bi as u64).wrapping_mul(0x2545_F491_4F6C_DD1D) ^ ((pass as u64) << 40)); + let rows = nn.len() / w; + let step: i32 = if reverse { -1 } else { 1 }; + for ry in 0..rows { + let ly = if reverse { rows - 1 - ry } else { ry }; + let y = bi * BAND + ly; + for rx in 0..w { + let x = if reverse { w - 1 - rx } else { rx }; + if !self.targets[y * w + x] { + continue; + } + let li = ly * w + x; + let t = (x as i32, y as i32); + let (mut best, mut bc) = (nn[li], co[li]); + let try_cand = |cand: (i32, i32), best: &mut (i32, i32), bc: &mut f32| { + if cand != *best && self.is_valid(cand.0, cand.1) { + let d = self.dist(t, cand, *bc, None); + if d < *bc { + (*best, *bc) = (cand, d); + } + } + }; + // Propagation from the neighbours visited before, within this band. + let nx = x as i32 - step; + if nx >= 0 && (nx as usize) < w { + let c = nn[ly * w + nx as usize]; + try_cand((c.0 + step, c.1), &mut best, &mut bc); + } + let nly = ly as i32 - step; + if nly >= 0 && (nly as usize) < rows { + let c = nn[nly as usize * w + x]; + try_cand((c.0, c.1 + step), &mut best, &mut bc); + } + // Random search in exponentially shrinking windows. + let mut rad = radius; + while rad >= 1 { + let cand = (best.0 + rng.range(-rad, rad), best.1 + rng.range(-rad, rad)); + try_cand(cand, &mut best, &mut bc); + rad /= 2; + } + nn[li] = best; + co[li] = bc; + } + } + }; + #[cfg(not(target_arch = "wasm32"))] + if w * self.h >= PAR_MIN { + use rayon::prelude::*; + nnf.par_chunks_mut(BAND * w).zip(cost.par_chunks_mut(BAND * w)).enumerate().for_each(|(bi, (nn, co))| band(bi, nn, co)); + return; + } + for (bi, (nn, co)) in nnf.chunks_mut(BAND * w).zip(cost.chunks_mut(BAND * w)).enumerate() { + band(bi, nn, co); + } + } + + /// Refresh every target's distance to its current match (hole cells change between iterations). + fn refresh_costs(&self, nnf: &[(i32, i32)], cost: &mut [f32]) { + let one = |(i, c): (usize, &mut f32)| { + *c = if self.targets[i] { self.dist(((i % self.w) as i32, (i / self.w) as i32), nnf[i], f32::INFINITY, None) } else { f32::INFINITY }; + }; + #[cfg(not(target_arch = "wasm32"))] + if self.w * self.h >= PAR_MIN { + use rayon::prelude::*; + cost.par_iter_mut().enumerate().for_each(one); + return; + } + cost.iter_mut().enumerate().for_each(one); + } + + /// `pick(x, y)` for every hole cell, in parallel rows: the cells it returns something for. + fn per_hole_cell(&self, pick: impl Fn(usize, usize) -> Option + Sync) -> Vec<(usize, T)> { + let (w, h) = (self.w, self.h); + let row = |y: usize| -> Vec<(usize, T)> { + if self.ctl.cancelled() { + return Vec::new(); + } + (0..w).filter(|&x| self.hole[y * w + x]).filter_map(|x| pick(x, y).map(|v| (y * w + x, v))).collect() + }; + #[cfg(not(target_arch = "wasm32"))] + let rows: Vec> = if w * h >= PAR_MIN { + use rayon::prelude::*; + (0..h).into_par_iter().map(row).collect() + } else { + (0..h).map(row).collect() + }; + #[cfg(target_arch = "wasm32")] + let rows: Vec> = (0..h).map(row).collect(); + rows.into_iter().flatten().collect() + } + + /// The cells of the patches covering hole cell `(x, y)`: each match's source cell for it, with + /// that patch's cost. + fn sources(&self, x: usize, y: usize, nnf: &[(i32, i32)], cost: &[f32], mut f: impl FnMut(usize, f32)) { + let (w, h, r) = (self.w as i32, self.h as i32, self.r); + for dy in -r..=r { + let ty = y as i32 + dy; + if ty < 0 || ty >= h { + continue; + } + for dx in -r..=r { + let tx = x as i32 + dx; + if tx < 0 || tx >= w { + continue; + } + let ti = ty as usize * self.w + tx as usize; + if !self.targets[ti] || !cost[ti].is_finite() { + continue; + } + let s = nnf[ti]; + let (sx, sy) = (s.0 - dx, s.1 - dy); + if sx >= 0 && sy >= 0 && sx < w && sy < h { + f(sy as usize * self.w + sx as usize, cost[ti]); + } + } + } + } + + /// Re-estimate the hole by the similarity-weighted vote of the overlapping matches (`uniform`: + /// all weights equal, for a field fresh from the coarser level whose costs mean nothing yet). + /// Returns the mean absolute change of the colour channels. + fn vote(&mut self, nnf: &[(i32, i32)], cost: &[f32], uniform: bool) -> f32 { + let mut cs: Vec = cost.iter().zip(&self.targets).filter(|(c, t)| **t && c.is_finite()).map(|(c, _)| *c).collect(); + // σ² from the 75th percentile of the match costs, as Wexler et al. and the paper. + let sigma2 = if cs.is_empty() { + 1.0 + } else { + let k = (cs.len() * 3 / 4).min(cs.len() - 1); + let (_, v, _) = cs.select_nth_unstable_by(k, f32::total_cmp); + v.max(1e-6) + }; + let cha = self.cha; + let new = self.per_hole_cell(|x, y| { + let mut v = vec![0.0f32; cha]; + let mut wsum = 0.0f32; + self.sources(x, y, nnf, cost, |si, c| { + let wt = if uniform { 1.0 } else { (-c / (2.0 * sigma2)).exp().max(1e-8) }; + for (o, s) in v.iter_mut().zip(&self.img[si * cha..(si + 1) * cha]) { + *o += s * wt; + } + wsum += wt; + }); + (wsum > 0.0).then(|| { + v.iter_mut().for_each(|o| *o /= wsum); + v + }) + }); + let (mut change, mut n) = (0.0f64, 0usize); + for (i, v) in new { + for (c, nv) in v.iter().enumerate().take(self.ch) { + change += f64::from((nv - self.img[i * cha + c]).abs()); + } + n += self.ch; + self.img[i * cha..(i + 1) * cha].copy_from_slice(&v); + } + if n == 0 { 0.0 } else { (change / n as f64) as f32 } + } + + /// The final reconstruction (the paper's eq. 5): each hole cell copies the source cell of the + /// lowest-cost patch covering it. Returns `(hole cell, source cell, that patch's cost)`. + fn copy_best(&mut self, nnf: &[(i32, i32)], cost: &[f32]) -> Vec<(usize, usize, f32)> { + let cha = self.cha; + let picked = self.per_hole_cell(|x, y| { + let mut best: Option<(usize, f32)> = None; + self.sources(x, y, nnf, cost, |si, c| { + if best.is_none_or(|(_, bc)| c < bc) { + best = Some((si, c)); + } + }); + best + }); + for &(i, (si, _)) in &picked { + self.img.copy_within(si * cha..(si + 1) * cha, i * cha); + } + picked.into_iter().map(|(i, (si, c))| (i, si, c)).collect() + } + + /// Hide the colour seam along the hole's edge (gradient-domain blending, P. Pérez, M. Gangnet, + /// A. Blake, *Poisson Image Editing*, SIGGRAPH 2003): at each known cell next to the hole, the + /// difference between its value and the one the copied source predicts there (the source cell's + /// own neighbour), as a median along the edge, is spread over the hole as a harmonic membrane + /// and added to the fill. Only slow changes of brightness and colour move; the copied texture + /// stays. + fn blend_seams(&mut self, copied: &[(usize, usize, f32)], texture_scale: f32) { + /// Radius of the median taken along the edge. + const EDGE_MEDIAN: usize = 8; + /// Largest shift the edge passes on, beyond twice its local texture. + const MAX_SHIFT: f32 = 0.05; + let (w, h, cha, ch) = (self.w, self.h, self.cha, self.ch); + if copied.is_empty() { + return; + } + let unscale = if texture_scale > 0.0 { 0.5 / texture_scale } else { 0.0 }; + // Work on the hole's bounding box grown by one cell, which holds the ring of known cells. + let (x0, y0, x1, y1) = + copied.iter().fold((w, h, 0, 0), |(x0, y0, x1, y1), (p, _, _)| (x0.min(p % w), y0.min(p / w), x1.max(p % w + 1), y1.max(p / w + 1))); + let (x0, y0, x1, y1) = (x0.saturating_sub(1), y0.saturating_sub(1), (x1 + 1).min(w), (y1 + 1).min(h)); + let (bw, bh) = (x1.saturating_sub(x0), y1.saturating_sub(y0)); + let local = |i: usize| (i / w - y0) * bw + (i % w - x0); + let img = &self.img; + let hole = self.hole; + let channel = |c: usize| -> Vec { + let mut v = vec![0.0f32; bw * bh]; + let mut n = vec![0u8; bw * bh]; + for &(p, s, _) in copied { + let (px, py, sx, sy) = ((p % w) as i32, (p / w) as i32, (s % w) as i32, (s / w) as i32); + for (dx, dy) in [(-1, 0), (1, 0), (0, -1), (0, 1)] { + let (qx, qy, tx, ty) = (px + dx, py + dy, sx + dx, sy + dy); + let inside = |x: i32, y: i32| x >= 0 && y >= 0 && (x as usize) < w && (y as usize) < h; + if !inside(qx, qy) || !inside(tx, ty) { + continue; + } + let (q, t) = (qy as usize * w + qx as usize, ty as usize * w + tx as usize); + if hole[q] { + continue; + } + let lq = local(q); + v[lq] += img[q * cha + c] - img[t * cha + c]; + n[lq] = n[lq].saturating_add(1); + } + } + for (v, n) in v.iter_mut().zip(&n) { + if *n > 0 { + *v /= f32::from(*n); + } + } + // Each edge cell takes the median difference of the edge cells around it, capped at + // `MAX_SHIFT` plus twice the local texture: a steady, moderate shift (a sky gradient, a + // change of light) survives; a lone or large one (a structure crossing the edge, foam + // meeting a dark source) would only glow into the fill. + let raw = v.clone(); + let mut near = Vec::new(); + for lq in (0..bw * bh).filter(|&lq| n[lq] > 0) { + let (qx, qy) = (lq % bw, lq / bw); + near.clear(); + for y in qy.saturating_sub(EDGE_MEDIAN)..(qy + EDGE_MEDIAN + 1).min(bh) { + for x in qx.saturating_sub(EDGE_MEDIAN)..(qx + EDGE_MEDIAN + 1).min(bw) { + if n[y * bw + x] > 0 { + near.push(raw[y * bw + x]); + } + } + } + let mid = near.len() / 2; + if !near.is_empty() { + let (_, m, _) = near.select_nth_unstable_by(mid, f32::total_cmp); + let q = (y0 + qy) * w + x0 + qx; + let texture = (img[q * cha + ch] + img[q * cha + ch + 1]) * unscale; + let cap = MAX_SHIFT + 2.0 * texture; + v[lq] = m.clamp(-cap, cap); + } + } + // Known cells off the ring take the value of the nearest ring cell (breadth first), so + // the solver's coarse levels, which average known cells, see the ring's values rather + // than zeros; then the hole is solved. + let mut queue: std::collections::VecDeque = (0..bw * bh).filter(|&lq| n[lq] > 0).collect(); + let mut seen: Vec = n.iter().map(|n| *n > 0).collect(); + let in_hole = |lq: usize| hole[(y0 + lq / bw) * w + x0 + lq % bw]; + while let Some(lq) = queue.pop_front() { + let (x, y) = (lq % bw, lq / bw); + let around = [(x > 0).then(|| lq - 1), (x + 1 < bw).then(|| lq + 1), (y > 0).then(|| lq - bw), (y + 1 < bh).then(|| lq + bw)]; + for nb in around.into_iter().flatten() { + if !seen[nb] && !in_hole(nb) { + seen[nb] = true; + v[nb] = v[lq]; + queue.push_back(nb); + } + } + } + let unknown: Vec = (0..bw * bh).map(in_hole).collect(); + crate::poisson::solve_membrane(bw, bh, &unknown, &mut v); + v + }; + #[cfg(not(target_arch = "wasm32"))] + let shifts: Vec> = { + use rayon::prelude::*; + (0..ch).into_par_iter().map(channel).collect() + }; + #[cfg(target_arch = "wasm32")] + let shifts: Vec> = (0..ch).map(channel).collect(); + for &(p, _, _) in copied { + let lp = local(p); + for (c, shift) in shifts.iter().enumerate() { + self.img[p * cha + c] += shift[lp]; + } + } + } + + /// Remove the visible seams between areas copied from different places (gradient-domain + /// fusion, after S. Darabi et al., *Image Melding*, SIGGRAPH 2012). Across a seam, two + /// neighbouring cells should differ as their sources do: as the better-matched side's source + /// says. Where the actual step departs from that by more than `SEAM_CONTRAST` times the local + /// texture `(Tx + Ty) / 2`, the seam shows (within a texture, where both steps are random and + /// about that size, the departure is already about √2 times it). A correction `δ` is solved for those steps only (`Δδ = div e`, `δ = 0` at known + /// cells, red-black SOR) and added to the fill: it is smooth away from the seams, so the copied + /// texture keeps its grain while each step is spread out. Steps hidden by texture, and the edge + /// of the hole (matched by `blend_seams`), add nothing. + fn fuse_seams(&mut self, copied: &[(usize, usize, f32)], texture_scale: f32, iters: usize) { + const SEAM_CONTRAST: f32 = 3.0; + const OMEGA: f32 = 1.8; + let (w, h, cha, ch) = (self.w, self.h, self.cha, self.ch); + let mut src = vec![(usize::MAX, f32::INFINITY); w * h]; + for &(p, s, c) in copied { + src[p] = (s, c); + } + let cell = |x: i32, y: i32| (x >= 0 && y >= 0 && (x as usize) < w && (y as usize) < h).then(|| y as usize * w + x as usize); + let unscale = if texture_scale > 0.0 { 0.5 / texture_scale } else { 0.0 }; + // `div e` per copied cell and channel: the sum over its visible-seam edges of how far the + // step to the neighbour is off (`e(p → q)`, and `−e` seen from q). + let mut div = vec![0.0f32; w * h * ch]; + let mut any = false; + for &(p, sp, cp) in copied { + let (px, py) = ((p % w) as i32, (p / w) as i32); + let texture = (self.img[p * cha + ch] + self.img[p * cha + ch + 1]) * unscale; + for (dx, dy) in [(1, 0), (0, 1)] { + let Some(q) = cell(px + dx, py + dy) else { continue }; + let (sq, cq) = src[q]; + if sq == usize::MAX { + continue; + } + // The step each side's source predicts (p's source one cell on, q's one cell back). + let from_p = cell((sp % w) as i32 + dx, (sp / w) as i32 + dy); + if from_p == Some(sq) { + continue; + } + let from_q = cell((sq % w) as i32 - dx, (sq / w) as i32 - dy); + let (a, b) = match (from_p, from_q) { + (Some(n), Some(m)) => { + if cp <= cq { + (sp, n) + } else { + (m, sq) + } + } + (Some(n), None) => (sp, n), + (None, Some(m)) => (m, sq), + (None, None) => continue, + }; + let e: Vec = (0..ch).map(|c| (self.img[b * cha + c] - self.img[a * cha + c]) - (self.img[q * cha + c] - self.img[p * cha + c])).collect(); + if e.iter().map(|v| v.abs()).sum::() / ch as f32 <= SEAM_CONTRAST * texture { + continue; + } + for (c, v) in e.iter().enumerate() { + div[p * ch + c] += v; + div[q * ch + c] -= v; + } + any = true; + } + } + if !any { + return; + } + // δ_p ← (Σ δ_q − div_p) / deg over the in-grid neighbours, δ = 0 at known cells. δ is + // harmonic away from the seams and fades within a few dozen cells, so only cells within + // `FUSE_BAND` of a seam are solved for. + const FUSE_BAND: usize = 16; + let seams: Vec = (0..w * h).map(|i| div[i * ch..(i + 1) * ch].iter().any(|v| *v != 0.0)).collect(); + let band = crate::remove::dilate(w, h, &seams, FUSE_BAND); + let halves: [Vec; 2] = [0, 1].map(|colour| copied.iter().map(|(p, _, _)| *p).filter(|&p| band[p] && (p % w + p / w) % 2 == colour).collect()); + let mut delta = vec![0.0f32; w * h * ch]; + let mut next: Vec = Vec::new(); + for _ in 0..iters { + if self.ctl.cancelled() { + return; + } + for cells in &halves { + next.clear(); + next.resize(cells.len() * ch, 0.0); + let d = δ + let relax = |(&p, out): (&usize, &mut [f32])| { + let (x, y) = ((p % w) as i32, (p / w) as i32); + let mut deg = 0.0f32; + out.fill(0.0); + for (dx, dy) in [(-1, 0), (1, 0), (0, -1), (0, 1)] { + let Some(q) = cell(x + dx, y + dy) else { continue }; + deg += 1.0; + out.iter_mut().zip(&d[q * ch..(q + 1) * ch]).for_each(|(s, v)| *s += v); + } + for (c, o) in out.iter_mut().enumerate() { + let cur = d[p * ch + c]; + *o = cur + OMEGA * ((*o - div[p * ch + c]) / deg.max(1.0) - cur); + } + }; + #[cfg(not(target_arch = "wasm32"))] + { + use rayon::prelude::*; + cells.par_iter().zip(next.par_chunks_mut(ch)).for_each(relax); + } + #[cfg(target_arch = "wasm32")] + cells.iter().zip(next.chunks_mut(ch)).for_each(relax); + for (p, v) in cells.iter().zip(next.chunks(ch)) { + delta[p * ch..(p + 1) * ch].copy_from_slice(v); + } + } + } + for &(p, _, _) in copied { + for c in 0..ch { + self.img[p * cha + c] += delta[p * ch + c]; + } + } + } + + /// Onion-peel initialisation (the paper's §3.4): fill the hole one boundary layer at a time. + /// Each known patch centre next to the layer gets a match by its known cells; each layer cell + /// is then the weighted vote of those matches. Leaves those matches in `nnf` (`has` marks them). + fn onion_peel(&mut self, nnf: &mut [(i32, i32)], has: &mut [bool], seed: u64) -> Result<(), Cancelled> { + let (w, h, r, cha) = (self.w, self.h, self.r, self.cha); + let mut known: Vec = self.hole.iter().map(|m| !*m).collect(); + // The cells still unknown, and per layer the known centres found (lists, so a layer costs + // its own size rather than a scan of the grid). + let mut unknown: Vec = (0..w * h).filter(|&i| self.hole[i]).collect(); + let mut centre = vec![false; w * h]; + let mut layer_cost = vec![f32::INFINITY; w * h]; + let mut layer_no = 0u64; + loop { + self.ctl.check()?; + let neighbour_known = |i: usize, known: &[bool]| { + let (x, y) = (i % w, i / w); + (x > 0 && known[i - 1]) || (x + 1 < w && known[i + 1]) || (y > 0 && known[i - w]) || (y + 1 < h && known[i + w]) + }; + let layer: Vec = unknown.iter().copied().filter(|&i| neighbour_known(i, &known)).collect(); + if layer.is_empty() { + return Ok(()); + } + // Known centres whose patch reaches into the layer. + let mut centres = Vec::new(); + for &i in &layer { + let (x, y) = ((i % w) as i32, (i / w) as i32); + for cy in (y - r).max(0)..=(y + r).min(h as i32 - 1) { + for cx in (x - r).max(0)..=(x + r).min(w as i32 - 1) { + let ci = cy as usize * w + cx as usize; + if known[ci] && !centre[ci] { + centre[ci] = true; + centres.push(ci); + } + } + } + } + for &ci in ¢res { + centre[ci] = false; + } + let this = &*self; + let known_ref = &known; + let (nnf_ref, has_ref) = (&*nnf, &*has); + let search = |&ci: &usize| -> (usize, (i32, i32), f32) { + let t = ((ci % w) as i32, (ci / w) as i32); + let mut rng = Rng::new(seed ^ (ci as u64).wrapping_mul(0x9E37_79B9) ^ (layer_no << 48)); + let (mut best, mut bc) = ((0, 0), f32::INFINITY); + let try_cand = |cand: (i32, i32), best: &mut (i32, i32), bc: &mut f32| { + if this.is_valid(cand.0, cand.1) { + let d = this.dist(t, cand, *bc, Some(known_ref)); + if d < *bc { + (*best, *bc) = (cand, d); + } + } + }; + if has_ref[ci] { + try_cand(nnf_ref[ci], &mut best, &mut bc); + } + // Neighbours' matches, shifted (propagation), and a few random sources. + for (ox, oy) in [(-1, 0), (1, 0), (0, -1), (0, 1)] { + let (nx, ny) = (t.0 + ox, t.1 + oy); + if nx >= 0 && ny >= 0 && (nx as usize) < w && (ny as usize) < h && has_ref[ny as usize * w + nx as usize] { + let s = nnf_ref[ny as usize * w + nx as usize]; + try_cand((s.0 - ox, s.1 - oy), &mut best, &mut bc); + } + } + for _ in 0..8 { + try_cand(this.random_source(&mut rng), &mut best, &mut bc); + } + for _ in 0..2 { + let mut rad = w.max(h) as i32; + while rad >= 1 { + try_cand((best.0 + rng.range(-rad, rad), best.1 + rng.range(-rad, rad)), &mut best, &mut bc); + rad /= 2; + } + } + (ci, best, bc) + }; + #[cfg(not(target_arch = "wasm32"))] + let found: Vec<(usize, (i32, i32), f32)> = { + use rayon::prelude::*; + centres.par_iter().map(search).collect() + }; + #[cfg(target_arch = "wasm32")] + let found: Vec<(usize, (i32, i32), f32)> = centres.iter().map(search).collect(); + let mut cs = Vec::with_capacity(found.len()); + for &(ci, s, c) in &found { + if c.is_finite() { + nnf[ci] = s; + has[ci] = true; + layer_cost[ci] = c; + cs.push(c); + } + } + let sigma2 = if cs.is_empty() { + 1.0 + } else { + let k = (cs.len() * 3 / 4).min(cs.len() - 1); + let (_, v, _) = cs.select_nth_unstable_by(k, f32::total_cmp); + v.max(1e-6) + }; + let mut values = Vec::with_capacity(layer.len()); + for &i in &layer { + let (x, y) = ((i % w) as i32, (i / w) as i32); + let mut v = vec![0.0f32; cha]; + let mut wsum = 0.0f32; + for cy in (y - r).max(0)..=(y + r).min(h as i32 - 1) { + for cx in (x - r).max(0)..=(x + r).min(w as i32 - 1) { + let ci = cy as usize * w + cx as usize; + if !layer_cost[ci].is_finite() { + continue; + } + let s = nnf[ci]; + let (sx, sy) = (s.0 + x - cx, s.1 + y - cy); + if sx < 0 || sy < 0 || sx as usize >= w || sy as usize >= h { + continue; + } + let si = sy as usize * w + sx as usize; + let wt = (-layer_cost[ci] / (2.0 * sigma2)).exp().max(1e-8); + for (o, s) in v.iter_mut().zip(&self.img[si * cha..(si + 1) * cha]) { + *o += s * wt; + } + wsum += wt; + } + } + if wsum > 0.0 { + v.iter_mut().for_each(|o| *o /= wsum); + values.push((i, v)); + } + } + for &(ci, _, _) in &found { + layer_cost[ci] = f32::INFINITY; + } + if values.is_empty() { + // No known patch reaches the layer: leave the rest to the EM iterations. + return Ok(()); + } + for (i, v) in values { + self.img[i * cha..(i + 1) * cha].copy_from_slice(&v); + known[i] = true; + } + unknown.retain(|&i| !known[i]); + layer_no += 1; + } + } +} + +/// Complete `hole` in `img` (see the module docs). `Ok(None)` when no fully-known patch exists to +/// copy from; `Err(Cancelled)` when `ctl` was cancelled. A buffer or mask of the wrong size, or no +/// hole, comes back unchanged. +pub fn complete_with(w: usize, h: usize, ch: usize, img: &[f32], hole: &[bool], p: &Params, ctl: &Interrupt) -> Result>, Cancelled> { + let n = w.saturating_mul(h); + if ch == 0 || hole.len() != n || img.len() != n.saturating_mul(ch) || !hole.iter().any(|m| *m) { + return Ok(Some(img.to_vec())); + } + let r = p.patch_radius.max(1); + let levels = level_count(w, h, hole, r); + // Texture features at full resolution, averaged over about a coarsest-level cell (`2^levels`). + let (tx, ty) = texture_features(w, h, ch, img, hole, 1 << levels.min(8)); + let cha = ch + 2; + let k = p.lambda.max(0.0).sqrt(); + let mut data = Vec::with_capacity(n * cha); + for (i, px) in img.chunks_exact(ch).enumerate() { + data.extend_from_slice(px); + data.extend_from_slice(&[tx[i] * k, ty[i] * k]); + } + let mut pyramid = vec![Level { w, h, img: data, hole: hole.to_vec() }]; + while pyramid.len() < levels { + ctl.check()?; + let Some(next) = pyramid.last().map(|l| downsample(l, cha, ch)) else { break }; + // A level with nothing to copy from can't seed the finer ones: stop above it. + if valid_sources(next.w, next.h, &next.hole, r).1.is_empty() { + break; + } + pyramid.push(next); + } + let total_px: usize = pyramid.iter().map(|l| l.w * l.h).sum(); + let mut done_px = 0usize; + let mut prev: Option<(usize, Vec<(i32, i32)>)> = None; + let mut result = None; + for (li, level) in pyramid.iter().enumerate().rev() { + ctl.check()?; + let (lw, lh) = (level.w, level.h); + let (valid, valid_list) = valid_sources(lw, lh, &level.hole, r); + if valid_list.is_empty() { + return Ok(None); + } + let ri = r as i32; + let mut targets = vec![false; lw * lh]; + for (i, _) in level.hole.iter().enumerate().filter(|(_, m)| **m) { + let (x, y) = ((i % lw) as i32, (i / lw) as i32); + for ty in (y - ri).max(0)..=(y + ri).min(lh as i32 - 1) { + for tx in (x - ri).max(0)..=(x + ri).min(lw as i32 - 1) { + targets[ty as usize * lw + tx as usize] = true; + } + } + } + let mut s = Solver { w: lw, h: lh, cha, ch, r: ri, img: level.img.clone(), hole: &level.hole, valid, valid_list, targets, ctl }; + let mut nnf = vec![(0i32, 0i32); lw * lh]; + let mut cost = vec![f32::INFINITY; lw * lh]; + let mut rng = Rng::new(p.seed ^ ((li as u64) << 20)); + match prev.take() { + None => { + // Coarsest level: onion peel, then random matches for the targets it didn't reach. + let mut has = vec![false; lw * lh]; + s.onion_peel(&mut nnf, &mut has, p.seed ^ 0x0c10_4e00)?; + for i in 0..lw * lh { + if s.targets[i] && !has[i] { + nnf[i] = s.random_source(&mut rng); + } + } + } + Some((pw, pnnf)) => { + // Upsample the coarser match field, then rebuild the hole from it. + for y in 0..lh { + for x in 0..lw { + let i = y * lw + x; + if !s.targets[i] { + continue; + } + let c = pnnf.get((y / 2) * pw + x / 2).copied().unwrap_or((-1, -1)); + let cand = (c.0 * 2 + (x % 2) as i32, c.1 * 2 + (y % 2) as i32); + nnf[i] = if s.is_valid(cand.0, cand.1) { + cand + } else if s.is_valid(c.0 * 2, c.1 * 2) { + (c.0 * 2, c.1 * 2) + } else { + s.random_source(&mut rng) + }; + } + } + // Uniform weights: the costs only need to be finite. + for (c, t) in cost.iter_mut().zip(&s.targets) { + if *t { + *c = 0.0; + } + } + s.vote(&nnf, &cost, true); + } + } + // Wide random search at the coarsest level, where the structure is decided; finer levels + // refine the upsampled field locally. + let coarsest = li + 1 == pyramid.len(); + let full = lw.max(lh) as i32; + for it in 0..p.max_iters.max(1) { + ctl.check()?; + s.refresh_costs(&nnf, &mut cost); + let radius = if coarsest || it == 0 { full } else { (8 * (2 * ri + 1)).min(full) }; + for pass in 0..p.pm_passes.max(1) { + ctl.check()?; + s.patchmatch(&mut nnf, &mut cost, pass + it * p.pm_passes, p.seed.wrapping_add((li * 1000 + it) as u64), radius); + } + ctl.check()?; + let change = s.vote(&nnf, &cost, false); + ctl.check()?; + if change < p.tolerance { + break; + } + } + if li == 0 { + s.refresh_costs(&nnf, &mut cost); + let copied = s.copy_best(&nnf, &cost); + ctl.check()?; + s.blend_seams(&copied, k); + ctl.check()?; + s.fuse_seams(&copied, k, 100); + ctl.check()?; + result = Some(s.img); + } + done_px += lw * lh; + ctl.progress(done_px as f32 / total_px.max(1) as f32); + prev = Some((lw, nnf)); + } + let Some(data) = result else { return Ok(None) }; + // The colour channels; known cells exactly as given. + let mut out = img.to_vec(); + for (i, m) in hole.iter().enumerate() { + if *m { + out[i * ch..(i + 1) * ch].copy_from_slice(&data[i * cha..i * cha + ch]); + } + } + Ok(Some(out)) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn noise(i: usize) -> f32 { + ((i as u32).wrapping_mul(2_654_435_761) >> 8) as f32 / (1u32 << 24) as f32 + } + + /// Mean |∂x| + |∂y| over the masked cells (one channel). + fn grain(w: usize, img: &[f32], m: &[bool]) -> f32 { + let (mut s, mut n) = (0.0, 0); + for (i, _) in m.iter().enumerate().filter(|(_, m)| **m) { + if i % w + 1 < w && i + w < img.len() { + s += (img[i + 1] - img[i]).abs() + (img[i + w] - img[i]).abs(); + n += 1; + } + } + s / n.max(1) as f32 + } + + /// The failure that motivates the texture features: a hole in noise next to a flat area of the + /// same mean must be filled with noise, not with the flat area. + #[test] + fn a_hole_in_texture_next_to_a_flat_area_of_the_same_mean_gets_texture() { + let (w, h) = (96, 64); + let img: Vec = (0..w * h).map(|i| if i % w < 48 { 0.5 } else { 0.25 + 0.5 * noise(i) }).collect(); + let hole: Vec = (0..w * h).map(|i| (60..84).contains(&(i % w)) && (20..44).contains(&(i / w))).collect(); + let out = complete_with(w, h, 1, &img, &hole, &Params::default(), &Interrupt::NONE).unwrap().unwrap(); + let (filled, truth) = (grain(w, &out, &hole), grain(w, &img, &hole)); + assert!(filled > truth * 0.7, "the fill is smoother than the texture around it: {filled} vs {truth}"); + assert!(out.iter().zip(&img).zip(&hole).all(|((o, i), m)| *m || o == i), "known cells are kept"); + } + + #[test] + fn a_hole_in_a_gradient_follows_the_gradient() { + let (w, h, ch) = (80, 60, 2); + let img: Vec = (0..w * h).flat_map(|i| [(i % w) as f32 / w as f32, 1.0]).collect(); + let hole: Vec = (0..w * h).map(|i| (30..50).contains(&(i % w)) && (20..40).contains(&(i / w))).collect(); + let out = complete_with(w, h, ch, &img, &hole, &Params::default(), &Interrupt::NONE).unwrap().unwrap(); + for (i, _) in hole.iter().enumerate().filter(|(_, m)| **m) { + assert!((out[i * ch] - img[i * ch]).abs() < 0.1, "cell {i}: {} vs {}", out[i * ch], img[i * ch]); + } + } + + #[test] + fn levels_follow_the_hole_thickness() { + let (w, h) = (400, 400); + let line: Vec = (0..w * h).map(|i| i / w == 200 && (20..380).contains(&(i % w))).collect(); + assert_eq!(level_count(w, h, &line, 3), 1, "a thin line stays at full resolution"); + let disc: Vec = (0..w * h).map(|i| ((i % w) as f32 - 200.0).hypot((i / w) as f32 - 200.0) < 90.0).collect(); + assert_eq!(level_count(w, h, &disc, 3), 5, "⌈log2(2·90/7)⌉"); + } + + #[test] + fn nothing_to_copy_from_and_bad_sizes() { + let img = vec![0.5f32; 12 * 12]; + assert_eq!(complete_with(12, 12, 1, &img, &[true; 144], &Params::default(), &Interrupt::NONE).unwrap(), None); + assert_eq!(complete_with(12, 12, 1, &img, &[true; 3], &Params::default(), &Interrupt::NONE).unwrap(), Some(img.clone())); + assert_eq!(complete_with(12, 12, 0, &img, &[true; 144], &Params::default(), &Interrupt::NONE).unwrap(), Some(img.clone())); + } + + #[test] + fn cancelled_is_an_error() { + let (w, h) = (64, 64); + let img: Vec = (0..w * h).map(noise).collect(); + let hole: Vec = (0..w * h).map(|i| (20..40).contains(&(i % w)) && (20..40).contains(&(i / w))).collect(); + let cancel = || true; + let progress = |_: f32| {}; + assert_eq!(complete_with(w, h, 1, &img, &hole, &Params::default(), &Interrupt::new(&cancel, &progress)), Err(Cancelled)); + } +} diff --git a/crates/algo/src/remove.rs b/crates/algo/src/remove.rs new file mode 100644 index 0000000000..550bda5809 --- /dev/null +++ b/crates/algo/src/remove.rs @@ -0,0 +1,310 @@ +//! The Remove Tool: fill what a brush stroke painted over from its surroundings. +//! +//! * [`close_loops`]: a stroke drawn around an object also removes what it encloses (every pixel +//! that can't reach the grid's border without crossing the stroke). +//! * [`dilate`]: the hole grows by a few pixels so an object's soft edge and halo go with it. +//! * [`remove_with`]: the hole is completed by [`crate::nonlocal::complete_with`] (patch-based, +//! with texture features so the fill keeps the grain of its surroundings, thin wires included). +//! * [`hole_radius`]: how thick a hole is, which sets how many scales the completion uses. +//! +//! Buffers are interleaved `w × h × ch` normalised floats (any model/depth) with `w × h` masks; +//! deterministic. + +use crate::nonlocal::{Params, complete_with}; +use crate::poisson::membrane_fill_with; +use photocraft_raster::{Cancelled, Interrupt}; + +/// `mask` plus everything it encloses: the cells from which no path of unmasked cells (4-connected) +/// reaches the grid's border. A mask of the wrong length is returned unchanged. +pub fn close_loops(w: usize, h: usize, mask: &[bool]) -> Vec { + if w == 0 || h == 0 || mask.len() != w.saturating_mul(h) { + return mask.to_vec(); + } + let mut outside = vec![false; mask.len()]; + let mut stack: Vec = Vec::new(); + let visit = |i: usize, outside: &mut Vec, stack: &mut Vec| { + if !mask[i] && !outside[i] { + outside[i] = true; + stack.push(i); + } + }; + for x in 0..w { + visit(x, &mut outside, &mut stack); + visit((h - 1) * w + x, &mut outside, &mut stack); + } + for y in 0..h { + visit(y * w, &mut outside, &mut stack); + visit(y * w + w - 1, &mut outside, &mut stack); + } + while let Some(i) = stack.pop() { + let (x, y) = (i % w, i / w); + if x > 0 { + visit(i - 1, &mut outside, &mut stack); + } + if x + 1 < w { + visit(i + 1, &mut outside, &mut stack); + } + if y > 0 { + visit(i - w, &mut outside, &mut stack); + } + if y + 1 < h { + visit(i + w, &mut outside, &mut stack); + } + } + outside.iter().map(|o| !*o).collect() +} + +/// `mask` grown by `r` cells (a square structuring element), in two separable passes. +pub fn dilate(w: usize, h: usize, mask: &[bool], r: usize) -> Vec { + if r == 0 || mask.len() != w.saturating_mul(h) { + return mask.to_vec(); + } + // A pass sets each cell when any cell within `r` along one axis is set (a running count). + let pass = |src: &[bool], horizontal: bool| -> Vec { + let (lines, len) = if horizontal { (h, w) } else { (w, h) }; + let at = |line: usize, k: usize| if horizontal { line * w + k } else { k * w + line }; + let mut out = vec![false; src.len()]; + for line in 0..lines { + let mut count = 0usize; + for k in 0..len.min(r) { + count += usize::from(src[at(line, k)]); + } + for k in 0..len { + if k + r < len { + count += usize::from(src[at(line, k + r)]); + } + if k > r { + count -= usize::from(src[at(line, k - r - 1)]); + } + out[at(line, k)] = count > 0; + } + } + out + }; + pass(&pass(mask, true), false) +} + +/// The largest chessboard distance from a hole cell to the nearest cell outside the hole (cells +/// beyond the grid don't count as outside: a hole at the canvas edge is not thinner for it). +/// `u32::MAX` when no cell lies outside the hole, 0 for a mask of the wrong size. +pub fn hole_radius(w: usize, h: usize, hole: &[bool]) -> u32 { + if hole.len() != w.saturating_mul(h) { + return 0; + } + let mut d: Vec = hole.iter().map(|m| if *m { u32::MAX } else { 0 }).collect(); + let relax = |d: &mut Vec, i: usize, j: usize| { + let v = d[j].saturating_add(1); + if v < d[i] { + d[i] = v; + } + }; + for y in 0..h { + for x in 0..w { + let i = y * w + x; + if x > 0 { + relax(&mut d, i, i - 1); + } + if y > 0 { + relax(&mut d, i, i - w); + if x > 0 { + relax(&mut d, i, i - w - 1); + } + if x + 1 < w { + relax(&mut d, i, i - w + 1); + } + } + } + } + for y in (0..h).rev() { + for x in (0..w).rev() { + let i = y * w + x; + if x + 1 < w { + relax(&mut d, i, i + 1); + } + if y + 1 < h { + relax(&mut d, i, i + w); + if x + 1 < w { + relax(&mut d, i, i + w + 1); + } + if x > 0 { + relax(&mut d, i, i + w - 1); + } + } + } + } + d.iter().zip(hole).filter(|(_, m)| **m).map(|(v, _)| *v).max().unwrap_or(0) +} + +/// Holes covering less than this share of their bounding box (a long stroke, scattered marks) are +/// completed tile by tile, so the work follows the hole rather than its bounding box. +const SPARSE: f64 = 0.25; +/// Smallest tile side, in pixels. +const MIN_TILE: usize = 256; + +/// Fill `hole` in `img` from the pixels around it. Returns the full buffer (unchanged outside the +/// hole); a buffer or mask of the wrong size comes back unchanged. When nothing around the hole is +/// fully known (a hole about as big as `img`), it is diffused from its edges instead. +/// +/// A sparse hole is filled in tiles eight times its thickness (at least [`MIN_TILE`] pixels), row +/// by row, each with a margin of six times its thickness (at least 48 pixels): each tile sees the +/// ones before it as filled, so the fill runs on across tile edges. +pub fn remove_with(w: usize, h: usize, ch: usize, img: &[f32], hole: &[bool], seed: u64, ctl: &Interrupt) -> Result, Cancelled> { + let n = w.saturating_mul(h); + if ch == 0 || hole.len() != n || img.len() != n.saturating_mul(ch) { + return Ok(img.to_vec()); + } + let (mut bx0, mut by0, mut bx1, mut by1, mut count) = (w, h, 0, 0, 0usize); + for (i, _) in hole.iter().enumerate().filter(|(_, m)| **m) { + let (x, y) = (i % w, i / w); + (bx0, by0, bx1, by1, count) = (bx0.min(x), by0.min(y), bx1.max(x + 1), by1.max(y + 1), count + 1); + } + if count == 0 { + return Ok(img.to_vec()); + } + let thickness = hole_radius(w, h, hole).min(1 << 20) as usize; + let tile = (8 * thickness).max(MIN_TILE); + let area = (bx1 - bx0) * (by1 - by0); + if count as f64 >= SPARSE * area as f64 || (bx1 - bx0).max(by1 - by0) <= 2 * tile { + return fill(w, h, ch, img, hole, seed, ctl); + } + let margin = (6 * thickness).max(48); + let mut out = img.to_vec(); + let mut left = hole.to_vec(); + let origins: Vec<(usize, usize)> = (by0..by1).step_by(tile).flat_map(|y| (bx0..bx1).step_by(tile).map(move |x| (x, y))).collect(); + for (k, &(tx0, ty0)) in origins.iter().enumerate() { + ctl.check()?; + let (tx1, ty1) = ((tx0 + tile).min(bx1), (ty0 + tile).min(by1)); + if !(ty0..ty1).any(|y| left[y * w + tx0..y * w + tx1].iter().any(|m| *m)) { + continue; + } + let (x0, y0, x1, y1) = (tx0.saturating_sub(margin), ty0.saturating_sub(margin), (tx1 + margin).min(w), (ty1 + margin).min(h)); + let ww = x1 - x0; + let sub_img: Vec = (y0..y1).flat_map(|y| out[(y * w + x0) * ch..(y * w + x1) * ch].iter().copied()).collect(); + let sub_hole: Vec = (y0..y1).flat_map(|y| left[y * w + x0..y * w + x1].iter().copied()).collect(); + let cancel = || ctl.cancelled(); + let progress = |f: f32| ctl.progress((k as f32 + f) / origins.len() as f32); + let filled = fill(ww, y1 - y0, ch, &sub_img, &sub_hole, seed.wrapping_add(k as u64), &Interrupt::new(&cancel, &progress))?; + for y in ty0..ty1 { + for x in tx0..tx1 { + let i = y * w + x; + if left[i] { + let j = (y - y0) * ww + (x - x0); + out[i * ch..(i + 1) * ch].copy_from_slice(&filled[j * ch..(j + 1) * ch]); + left[i] = false; + } + } + } + } + Ok(out) +} + +/// Complete one hole, or diffuse it when there is no fully-known patch to copy from. +fn fill(w: usize, h: usize, ch: usize, img: &[f32], hole: &[bool], seed: u64, ctl: &Interrupt) -> Result, Cancelled> { + match complete_with(w, h, ch, img, hole, &Params { seed, ..Params::default() }, ctl)? { + Some(out) => Ok(out), + None => membrane_fill_with(w, h, ch, img, hole, ctl), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn grid(w: usize, h: usize, f: impl Fn(usize, usize) -> bool) -> Vec { + (0..w * h).map(|i| f(i % w, i / w)).collect() + } + + #[test] + fn a_ring_closes_into_a_disc_and_an_open_line_does_not() { + let (w, h) = (40, 40); + let ring = grid(w, h, |x, y| { + let d = (x as f32 - 20.0).hypot(y as f32 - 20.0); + (10.0..12.0).contains(&d) + }); + let closed = close_loops(w, h, &ring); + assert!(closed[20 * w + 20], "the centre is enclosed"); + assert!(!closed[2 * w + 2], "the outside stays outside"); + let line = grid(w, h, |x, y| y == 20 && (5..35).contains(&x)); + assert_eq!(close_loops(w, h, &line), line, "an open stroke encloses nothing"); + } + + #[test] + fn a_loop_open_at_one_pixel_encloses_nothing() { + let (w, h) = (20, 20); + let mut sq = grid(w, h, |x, y| (x == 5 || x == 14 || y == 5 || y == 14) && (5..=14).contains(&x) && (5..=14).contains(&y)); + sq[5 * w + 9] = false; + assert!(!close_loops(w, h, &sq)[10 * w + 10]); + } + + #[test] + fn dilate_grows_by_the_radius() { + let (w, h) = (11, 11); + let dot = grid(w, h, |x, y| (x, y) == (5, 5)); + let d = dilate(w, h, &dot, 2); + assert_eq!(d.iter().filter(|m| **m).count(), 25); + assert!(d[3 * w + 3] && d[7 * w + 7] && !d[2 * w + 5]); + assert_eq!(dilate(w, h, &dot, 0), dot); + } + + #[test] + fn radius_measures_the_thickest_part() { + let (w, h) = (30, 30); + let line = grid(w, h, |x, y| y == 10 && (5..25).contains(&x)); + assert_eq!(hole_radius(w, h, &line), 1); + let block = grid(w, h, |x, y| (5..16).contains(&x) && (5..16).contains(&y)); + assert_eq!(hole_radius(w, h, &block), 6); + assert_eq!(hole_radius(w, h, &vec![true; w * h]), u32::MAX); + assert_eq!(hole_radius(w, h, &[true]), 0, "a mask of the wrong size"); + } + + /// A hole in a horizontal gradient is filled close to the gradient, both thin and wide. + #[test] + fn removes_thin_and_wide_holes_from_a_gradient() { + let (w, h, ch) = (64, 48, 2); + let truth: Vec = (0..w * h).flat_map(|i| [(i % w) as f32 / w as f32, 1.0]).collect(); + for hole in [grid(w, h, |x, y| y == 24 && (10..54).contains(&x)), grid(w, h, |x, y| (24..40).contains(&x) && (16..32).contains(&y))] { + let mut img = truth.clone(); + for (i, m) in hole.iter().enumerate() { + if *m { + img[i * ch] = 1.0; + } + } + let out = remove_with(w, h, ch, &img, &hole, 7, &Interrupt::NONE).unwrap(); + for (i, m) in hole.iter().enumerate() { + if *m { + assert!((out[i * ch] - truth[i * ch]).abs() < 0.12, "pixel {i}: {} vs {}", out[i * ch], truth[i * ch]); + } else { + assert_eq!(out[i * ch..(i + 1) * ch], img[i * ch..(i + 1) * ch], "outside the hole is untouched"); + } + } + } + } + + /// A long, thin diagonal hole goes tile by tile; every cell is filled, the rest is untouched. + #[test] + fn a_long_thin_hole_is_filled_in_tiles() { + let (w, h, ch) = (900, 700, 1); + let img: Vec = (0..w * h).map(|i| 0.3 + 0.4 * (((i % w) / 7 + (i / w) / 5) % 2) as f32).collect(); + let hole = grid(w, h, |x, y| (x as f32 - y as f32 * 1.2 - 20.0).abs() < 3.0 && (40..680).contains(&y)); + let mut holed = img.clone(); + hole.iter().enumerate().filter(|(_, m)| **m).for_each(|(i, _)| holed[i] = 1.0); + let out = remove_with(w, h, ch, &holed, &hole, 3, &Interrupt::NONE).unwrap(); + for (i, m) in hole.iter().enumerate() { + if *m { + assert!((0.29..=0.71).contains(&out[i]), "cell {i} not filled from the stripes: {}", out[i]); + } else { + assert_eq!(out[i], holed[i]); + } + } + } + + #[test] + fn hostile_sizes_return_the_input() { + let img = vec![0.5f32; 12]; + assert_eq!(remove_with(2, 2, 3, &img, &[true], 1, &Interrupt::NONE).unwrap(), img); + assert_eq!(remove_with(2, 2, 0, &img, &[true; 4], 1, &Interrupt::NONE).unwrap(), img); + assert_eq!(remove_with(3, 2, 2, &img, &[false; 6], 1, &Interrupt::NONE).unwrap(), img); + assert!(close_loops(0, 0, &[]).is_empty()); + assert_eq!(dilate(3, 3, &[true], 1), vec![true]); + } +} diff --git a/crates/engine/examples/remove_bench.rs b/crates/engine/examples/remove_bench.rs new file mode 100644 index 0000000000..581f89848d --- /dev/null +++ b/crates/engine/examples/remove_bench.rs @@ -0,0 +1,35 @@ +//! Timing: the Remove Tool (`paint.remove`) on a 6000×4000 (24 MP) document, through the full +//! command path (`Session::execute`: window read, hole, fill, write, undo snapshot). +//! `cargo run --release -p photocraft-engine --example remove_bench [depth]` +//! +//! Rings around objects of 100, 300 and 1000 px (one completion each) and a 3000 px thin line +//! (completed tile by tile), each undone before the next. +use std::time::Instant; + +use photocraft_engine::Session; +use serde_json::json; + +fn ring(cx: f64, cy: f64, r: f64) -> Vec<[f64; 2]> { + (0..=96).map(|i| f64::from(i) / 96.0 * std::f64::consts::TAU).map(|t| [cx + r * t.cos(), cy + r * t.sin()]).collect() +} + +fn main() { + let depth: u64 = std::env::args().nth(1).and_then(|v| v.parse().ok()).unwrap_or(8); + let mut s = Session::new(); + s.execute("file.new", json!({"width": 6000, "height": 4000, "depth": depth})).unwrap(); + s.execute("filter.render.clouds", json!({"seed": 3})).unwrap(); + s.execute("filter.noise.addNoise", json!({"amount": 8, "seed": 1})).unwrap(); + println!("document 6000x4000, {depth}-bit RGB"); + let cases: Vec<(String, serde_json::Value)> = vec![ + ("ring 100 px".into(), json!({"points": ring(3000.0, 2000.0, 60.0), "size": 16})), + ("ring 300 px".into(), json!({"points": ring(3000.0, 2000.0, 160.0), "size": 24})), + ("ring 1000 px".into(), json!({"points": ring(3000.0, 2000.0, 510.0), "size": 32})), + ("line 3000 px, 4 px wide".into(), json!({"points": [[1500, 1000], [4500, 3000]], "size": 2})), + ]; + for (name, p) in cases { + let t = Instant::now(); + s.execute("paint.remove", p).unwrap(); + println!("{name:<26} {:>8.1?}", t.elapsed()); + s.execute("edit.undo", json!({})).unwrap(); + } +} diff --git a/crates/engine/examples/remove_quality.rs b/crates/engine/examples/remove_quality.rs new file mode 100644 index 0000000000..9edcc8c4c3 --- /dev/null +++ b/crates/engine/examples/remove_quality.rs @@ -0,0 +1,217 @@ +//! Quality of the Remove Tool's fill (`photocraft_algo::remove::remove_with`) on real photos. +//! `cargo run --release -p photocraft-engine --example remove_quality -- [out dir]` +//! +//! The images are `01.jpg`, `02.jpg`, `03.jpg`, `04.jpg` and `06.jpg`: public-domain photos from +//! Wikimedia Commons, not committed (the list is in `docs/development.md` › Remove Tool quality). +//! Each gets holes over background (discs and thin lines, placed in fractions of the image size) +//! that are filled through a window like the command's. Measured in the hole against the original: +//! +//! * `psnr`: fidelity in dB (a plausible fill need not match, so read it as a trend); +//! * `texture`: mean gradient magnitude of the fill over the original's (1 is ideal; below 1 the +//! fill is smoother than what it replaces); +//! * `seam`: the same on the band along the hole's edge (above 1 the edge shows). +//! +//! With an out dir, each case is also written as `-.png`: original | hole | fill. +use std::time::Instant; + +use photocraft_codecs::{ChannelLayout, EncodeOptions, Format, Image, SampleType}; +use photocraft_raster::Interrupt; + +/// A hole: a disc (centre in fractions of the width and height, radius in fractions of the width) +/// or a line (ends in fractions, width in pixels). +#[derive(Clone, Copy)] +enum Shape { + Disc { c: (f64, f64), r: f64 }, + Line { a: (f64, f64), b: (f64, f64), width: f64 }, +} + +fn cases(image: &str) -> Vec<(&'static str, Shape)> { + use Shape::{Disc, Line}; + match image { + // The Tetons and the Snake River: forest, river bend, ridge, clouds. + "01" => vec![ + ("forest", Disc { c: (0.75, 0.62), r: 0.03 }), + ("river-bend", Disc { c: (0.45, 0.72), r: 0.045 }), + ("ridge", Disc { c: (0.5, 0.28), r: 0.035 }), + ("clouds", Disc { c: (0.25, 0.12), r: 0.06 }), + ("wire", Line { a: (0.3, 0.55), b: (0.7, 0.85), width: 5.0 }), + ], + // A dam in a canyon: rock, water, the dam's wall. + "02" => vec![ + ("rock", Disc { c: (0.85, 0.45), r: 0.04 }), + ("water", Disc { c: (0.6, 0.7), r: 0.035 }), + ("dam-wall", Disc { c: (0.35, 0.25), r: 0.025 }), + ("wire", Line { a: (0.05, 0.6), b: (0.95, 0.5), width: 4.0 }), + ], + // Oak trees by a gravel path: grass, gravel, a trunk, branches. + "03" => vec![ + ("grass", Disc { c: (0.75, 0.85), r: 0.045 }), + ("gravel", Disc { c: (0.25, 0.85), r: 0.035 }), + ("trunk", Disc { c: (0.48, 0.45), r: 0.02 }), + ("branches", Disc { c: (0.8, 0.2), r: 0.04 }), + ], + // A meadow anemone in grass, shallow depth of field. + "04" => vec![ + ("bokeh", Disc { c: (0.2, 0.35), r: 0.05 }), + ("stems", Disc { c: (0.85, 0.6), r: 0.04 }), + ("wire", Line { a: (0.1, 0.1), b: (0.9, 0.3), width: 6.0 }), + ], + // A coast from above: surf, sand, mountains. + "06" => vec![ + ("surf", Disc { c: (0.75, 0.85), r: 0.045 }), + ("sand", Disc { c: (0.3, 0.75), r: 0.04 }), + ("mountains", Disc { c: (0.5, 0.12), r: 0.03 }), + ("wire", Line { a: (0.0, 0.55), b: (1.0, 0.65), width: 5.0 }), + ], + _ => Vec::new(), + } +} + +/// The hole of `shape` on a `w × h` image. +fn mask(shape: Shape, w: usize, h: usize) -> Vec { + let (wf, hf) = (w as f64, h as f64); + (0..w * h) + .map(|i| { + let (x, y) = ((i % w) as f64 + 0.5, (i / w) as f64 + 0.5); + match shape { + Shape::Disc { c, r } => (x - c.0 * wf).hypot(y - c.1 * hf) < r * wf, + Shape::Line { a, b, width } => { + let (ax, ay, bx, by) = (a.0 * wf, a.1 * hf, b.0 * wf, b.1 * hf); + let (dx, dy) = (bx - ax, by - ay); + let t = (((x - ax) * dx + (y - ay) * dy) / (dx * dx + dy * dy)).clamp(0.0, 1.0); + (x - ax - t * dx).hypot(y - ay - t * dy) < width / 2.0 + } + } + }) + .collect() +} + +/// Bounding box `(x0, y0, x1, y1)` of the set cells. +fn bbox(w: usize, m: &[bool]) -> Option<(usize, usize, usize, usize)> { + let mut b: Option<(usize, usize, usize, usize)> = None; + for (i, _) in m.iter().enumerate().filter(|(_, v)| **v) { + let (x, y) = (i % w, i / w); + b = Some(b.map_or((x, y, x + 1, y + 1), |(x0, y0, x1, y1)| (x0.min(x), y0.min(y), x1.max(x + 1), y1.max(y + 1)))); + } + b +} + +/// A `(x0, y0, x1, y1)` crop of an interleaved buffer. +fn crop(src: &[f32], w: usize, ch: usize, r: (usize, usize, usize, usize)) -> Vec { + (r.1..r.3).flat_map(|y| src[(y * w + r.0) * ch..(y * w + r.2) * ch].iter().copied()).collect() +} + +/// Gradient magnitude (forward differences, summed over channels) at every cell. +fn gradient(img: &[f32], w: usize, h: usize, ch: usize) -> Vec { + (0..w * h) + .map(|i| { + let (x, y) = (i % w, i / w); + let (r, d) = ((y * w + (x + 1).min(w - 1)) * ch, ((y + 1).min(h - 1) * w + x) * ch); + (0..ch).map(|c| (img[r + c] - img[i * ch + c]).abs() + (img[d + c] - img[i * ch + c]).abs()).sum() + }) + .collect() +} + +fn mean_over(v: &[f32], m: &[bool]) -> f64 { + let (s, n) = v.iter().zip(m).filter(|(_, m)| **m).fold((0.0f64, 0usize), |(s, n), (v, _)| (s + f64::from(*v), n + 1)); + if n == 0 { 0.0 } else { s / n as f64 } +} + +struct Score { + psnr: f64, + texture: f64, + seam: f64, +} + +fn score(orig: &[f32], out: &[f32], hole: &[bool], w: usize, h: usize, ch: usize) -> Score { + let (mut se, mut n) = (0.0f64, 0usize); + for (i, _) in hole.iter().enumerate().filter(|(_, m)| **m) { + for c in 0..ch { + se += f64::from(out[i * ch + c] - orig[i * ch + c]).powi(2); + n += 1; + } + } + let psnr = if se == 0.0 { 99.0 } else { 10.0 * (n as f64 / se).log10() }; + let (go, gf) = (gradient(orig, w, h, ch), gradient(out, w, h, ch)); + // The edge band: cells within 2 px of the hole's boundary, on both sides. + let grown = photocraft_algo::remove::dilate(w, h, hole, 2); + let inner: Vec = { + let outside: Vec = hole.iter().map(|m| !*m).collect(); + photocraft_algo::remove::dilate(w, h, &outside, 2).iter().map(|o| !*o).collect() + }; + let band: Vec = grown.iter().zip(&inner).map(|(g, i)| *g && !*i).collect(); + let ratio = |m: &[bool]| { + let o = mean_over(&go, m); + if o > 0.0 { mean_over(&gf, m) / o } else { 1.0 } + }; + Score { psnr, texture: ratio(hole), seam: ratio(&band) } +} + +fn write_png(path: &std::path::Path, data: &[f32], w: usize, h: usize, ch: usize) { + let layout = if ch == 1 { ChannelLayout::Gray } else { ChannelLayout::Rgb }; + let img = Image::from_normalized(w as u32, h as u32, layout, SampleType::U8, data).expect("image"); + std::fs::write(path, photocraft_codecs::encode(&img, Format::Png, &EncodeOptions::default()).expect("encode")).expect("write"); +} + +fn main() { + let args: Vec = std::env::args().collect(); + let dir = std::path::PathBuf::from(args.get(1).expect("usage: remove_quality [out dir]")); + let out_dir = args.get(2).map(std::path::PathBuf::from); + if let Some(d) = &out_dir { + std::fs::create_dir_all(d).expect("out dir"); + } + println!("{:<4} {:<11} {:>7} {:>8} {:>6} {:>9}", "img", "case", "psnr", "texture", "seam", "ms"); + let (mut sum_psnr, mut sum_tex, mut sum_seam, mut sum_ms, mut count) = (0.0, 0.0, 0.0, 0.0, 0); + for name in ["01", "02", "03", "04", "06"] { + let Ok(bytes) = std::fs::read(dir.join(format!("{name}.jpg"))) else { + println!("{name}: missing, skipped"); + continue; + }; + let img = photocraft_codecs::decode(&bytes).expect("decode"); + let (w, h) = (img.width() as usize, img.height() as usize); + let ch = img.layout().channels(); + let px = img.to_normalized(); + for (case, shape) in cases(name) { + let full = mask(shape, w, h); + let Some(b) = bbox(w, &full) else { continue }; + // The command's window: the hole plus three quarters of its extent (32..512 px). + let ext = (b.2 - b.0).max(b.3 - b.1); + let m = (ext * 3 / 4).clamp(32, 512); + let win = (b.0.saturating_sub(m), b.1.saturating_sub(m), (b.2 + m).min(w), (b.3 + m).min(h)); + let (ww, wh) = (win.2 - win.0, win.3 - win.1); + let orig = crop(&px, w, ch, win); + let hole: Vec = (win.1..win.3).flat_map(|y| (win.0..win.2).map(move |x| (x, y))).map(|(x, y)| full[y * w + x]).collect(); + let mut holed = orig.clone(); + for (i, _) in hole.iter().enumerate().filter(|(_, m)| **m) { + holed[i * ch..(i + 1) * ch].fill(1.0); + } + let t = Instant::now(); + let out = photocraft_algo::remove::remove_with(ww, wh, ch, &holed, &hole, 0x0052_e30e, &Interrupt::NONE).expect("never cancelled"); + let ms = t.elapsed().as_secs_f64() * 1000.0; + let s = score(&orig, &out, &hole, ww, wh, ch); + println!("{name:<4} {case:<11} {:>7.2} {:>8.3} {:>6.3} {ms:>9.1}", s.psnr, s.texture, s.seam); + (sum_psnr, sum_tex, sum_seam, sum_ms, count) = (sum_psnr + s.psnr, sum_tex + s.texture, sum_seam + s.seam, sum_ms + ms, count + 1); + if let Some(d) = &out_dir { + // Original | hole | fill, cropped to the hole plus half its extent. + let pad = ext / 2 + 8; + let lb = b.0 - win.0; + let tb = b.1 - win.1; + let r = (lb.saturating_sub(pad), tb.saturating_sub(pad), (lb + (b.2 - b.0) + pad).min(ww), (tb + (b.3 - b.1) + pad).min(wh)); + let (cw, chh) = (r.2 - r.0, r.3 - r.1); + let parts = [crop(&orig, ww, ch, r), crop(&holed, ww, ch, r), crop(&out, ww, ch, r)]; + let mut sheet = vec![0.0f32; cw * 3 * chh * ch]; + for (k, p) in parts.iter().enumerate() { + for y in 0..chh { + let dst = (y * cw * 3 + k * cw) * ch; + sheet[dst..dst + cw * ch].copy_from_slice(&p[y * cw * ch..(y + 1) * cw * ch]); + } + } + write_png(&d.join(format!("{name}-{case}.png")), &sheet, cw * 3, chh, ch); + } + } + } + if count > 0 { + let n = f64::from(count); + println!("{:<16} {:>7.2} {:>8.3} {:>6.3} {:>9.1}", "mean", sum_psnr / n, sum_tex / n, sum_seam / n, sum_ms / n); + } +} diff --git a/crates/engine/src/channel_cmds.rs b/crates/engine/src/channel_cmds.rs index 7e056d1071..5ced9899fa 100644 --- a/crates/engine/src/channel_cmds.rs +++ b/crates/engine/src/channel_cmds.rs @@ -579,6 +579,7 @@ fn routed(id: &str) -> bool { | "paint.smudge" | "paint.historyBrush" | "paint.redEye" + | "paint.remove" ) } diff --git a/crates/engine/src/jobs_tests.rs b/crates/engine/src/jobs_tests.rs index 8da27e59f0..3dba16d0e6 100644 --- a/crates/engine/src/jobs_tests.rs +++ b/crates/engine/src/jobs_tests.rs @@ -364,6 +364,27 @@ fn content_aware_move_runs_as_a_job() { assert_eq!(pixels(&s), before.layer(s.active().unwrap().active_layer.unwrap()).unwrap().surface().unwrap().read_region(before.bounds())); } +#[test] +fn remove_runs_as_a_job() { + let mut inline = session(160, 120); + let mut s = session(160, 120); + let p = json!({"points": [[60, 50], [100, 70]], "size": 24}); + inline.execute("paint.remove", p.clone()).unwrap(); + let before = pixels(&s); + let id = job(s.start("paint.remove", p.clone()).unwrap()); + let e = wait_event(&mut s, id); + let JobOutcome::Done(v) = &e.outcome else { panic!("{e:?}") }; + assert!(v["damage"].is_array(), "{v}"); + assert_eq!(pixels(&s), pixels(&inline), "same result as the synchronous command"); + // Cancelled at once: the document stays as it was. + s.undo(); + let id = job(s.start("paint.remove", p).unwrap()); + assert!(s.cancel_job(id)); + s.join_cancelled_jobs(); + assert_eq!(wait_event(&mut s, id).outcome, JobOutcome::Cancelled); + assert_eq!(pixels(&s), before); +} + #[test] fn open_runs_as_a_job_and_cancel_adds_nothing() { let mut src = session(40, 30); diff --git a/crates/engine/src/retouch_cmds.rs b/crates/engine/src/retouch_cmds.rs index 394746e79b..8ef1988eb4 100644 --- a/crates/engine/src/retouch_cmds.rs +++ b/crates/engine/src/retouch_cmds.rs @@ -1,5 +1,5 @@ //! Retouching tools as commands: Clone Stamp, Pattern Stamp, Healing Brush, Spot Healing Brush, -//! Patch, Content-Aware Move, Dodge, Burn, Sponge, Blur, Sharpen, Smudge and History Brush. +//! Patch, Content-Aware Move, Remove, Dodge, Burn, Sponge, Blur, Sharpen, Smudge and History Brush. //! //! Every command takes a Photoshop-style brush (`points`, `size`, `hardness`, `opacity`, `flow`, //! `spacing`, `layer`), respects the active selection as a mask and the layer's transparency lock, and @@ -26,6 +26,7 @@ use crate::{EngineError, Result, Session}; mod content_aware_move; mod patch; mod pattern_stamp; +mod remove; pub use patch::preview as patch_preview; fn bad(cmd: &str, msg: impl Into) -> EngineError { @@ -1099,6 +1100,18 @@ pub fn specs() -> Vec { run: content_aware_move::content_aware_move, journal: true, }, + CommandSpec { + id: "paint.remove", + label: "Remove", + menu: &[], + shortcut: None, + params: brush_params!( + r#","closeLoops":bool=true (a stroke around an object removes what it encloses),"sampleAllLayers":bool=false → {"damage"} (a background job)"# + ), + enabled: has_pixel_layer, + run: remove::remove, + journal: true, + }, CommandSpec { id: "paint.dodge", label: "Dodge", diff --git a/crates/engine/src/retouch_cmds/content_aware_move.rs b/crates/engine/src/retouch_cmds/content_aware_move.rs index 5b31bd1bff..0f1e403622 100644 --- a/crates/engine/src/retouch_cmds/content_aware_move.rs +++ b/crates/engine/src/retouch_cmds/content_aware_move.rs @@ -166,16 +166,16 @@ fn read(doc: &mut Document, plan: &Plan, p: &Value, rect: Rect) -> Result Result { - let (surf, lock) = crate::channel_cmds::target_surface(doc, plan.id, p)?; +/// they are. Returns the written rectangle. Shared with the Remove Tool. +pub(super) fn replace(doc: &mut Document, id: Option, p: &Value, out: &Region, mask: &[bool]) -> Result { + let (surf, lock) = crate::channel_cmds::target_surface(doc, id, p)?; let fmt = surf.format(); let a = alpha_index(&fmt); let rect = out.rect; let mut px = surf.read_region(rect); let n = out.ch; if px.len() != out.data.len() || mask.len() * n != px.len() { - return Err(EngineError::Other("internal error: content-aware move buffers disagree in size".into())); + return Err(EngineError::Other("internal error: fill buffers disagree in size".into())); } for ((dst, src), m) in px.chunks_exact_mut(n).zip(out.data.chunks_exact(n)).zip(mask) { if !*m { @@ -256,7 +256,7 @@ fn run_move(doc: &mut Document, sel: &Surface, plan: &Plan, p: &Value, ctx: &cra let allowed: Vec = old_place.iter().map(|o| !*o).collect(); buf.data = ctx.stage(0.0, split, label, |ctl| fill_with(w, h, n, &buf.data, &band, &allowed, &opts, ctl)).map_err(cancelled)?; } - let mut damage = replace(doc, plan, p, &buf, &moved)?; + let mut damage = replace(doc, plan.id, p, &buf, &moved)?; // 2. Move: fill the place the content left (where the content didn't land on it). if plan.mode == Mode::Move { @@ -269,7 +269,7 @@ fn run_move(doc: &mut Document, sel: &Surface, plan: &Plan, p: &Value, ctx: &cra if hole.iter().any(|h| *h) { let allowed: Vec = landed.iter().map(|l| !*l).collect(); let data = ctx.stage(split, 1.0, label, |ctl| fill_with(w, h, img.ch, &img.data, &hole, &allowed, &opts, ctl)).map_err(cancelled)?; - damage = damage.union(&replace(doc, plan, p, &Region { rect: win, ch: img.ch, data }, &hole)?); + damage = damage.union(&replace(doc, plan.id, p, &Region { rect: win, ch: img.ch, data }, &hole)?); } } let surf = crate::channel_cmds::target_surface(doc, plan.id, p)?.0.clone(); diff --git a/crates/engine/src/retouch_cmds/remove.rs b/crates/engine/src/retouch_cmds/remove.rs new file mode 100644 index 0000000000..57847e7fc3 --- /dev/null +++ b/crates/engine/src/retouch_cmds/remove.rs @@ -0,0 +1,153 @@ +//! The Remove Tool as a command: paint over something and it is replaced by its surroundings. +//! +//! The stroke's footprint is the hole. A stroke that closes around an object also removes what it +//! encloses (`closeLoops`), and the hole grows by [`GROW`] pixels so the object's soft edge and +//! halo go with it. The hole is completed from a window around the stroke by patch-based +//! completion with texture features (`photocraft_algo::remove`, `photocraft_algo::nonlocal`). The +//! selection, when there is one, limits what changes. Large holes take seconds, so the command +//! runs as a background job (#210): with progress, cancellable, and the document unchanged until +//! it applies, in one undo step. + +use super::content_aware_move::replace; +use super::patch::coverage; +use super::*; + +const CMD: &str = "paint.remove"; + +/// Largest stroke, in pixels of its bounding box (as the Patch Tool's and Content-Aware Move's). +/// Checked before the stroke's coverage is rendered, which allocates its whole bounding box, also +/// where it leaves the canvas. +const MAX_PIXELS: u64 = 16 << 20; + +/// Cap on the sampling margin around the stroke (the window grows with the stroke). +const MAX_MARGIN: i32 = 512; + +/// How far the hole grows beyond the stroke, in pixels. +const GROW: usize = 2; + +/// Seed of the completion, so a removal is reproducible. +const SEED: u64 = 0x0052_e30e; + +/// What a removal does, checked before any work starts. +struct Plan { + id: Option, + stroke: Stroke, + /// Where the fill samples from: the stroke plus a margin, on the canvas. + window: Rect, + close_loops: bool, + all_layers: bool, +} + +/// The stroke's bounding box (points ± the brush radius), before any coverage is rendered. +fn stroke_bounds(stroke: &Stroke) -> Rect { + let r = f64::from(stroke.brush.size) / 2.0 + 2.0; + let (mut x0, mut y0, mut x1, mut y1) = (f64::MAX, f64::MAX, f64::MIN, f64::MIN); + for q in &stroke.points { + (x0, y0, x1, y1) = (x0.min(q.x), y0.min(q.y), x1.max(q.x), y1.max(q.y)); + } + // `check_coords` keeps points within ±MAX_COORD and sizes are capped, so these fit an i32. + Rect::new((x0 - r).floor() as i32, (y0 - r).floor() as i32, (x1 + r).ceil() as i32, (y1 + r).ceil() as i32) +} + +fn plan(s: &Session, p: &Value) -> Result { + let (stroke, id) = parse_brush(s, p, CMD)?; + let d = s.active().ok_or(EngineError::NoDocument)?; + let canvas = d.doc.bounds(); + let bounds = stroke_bounds(&stroke); + let pixels = u64::from(bounds.width()) * u64::from(bounds.height()); + if pixels > MAX_PIXELS { + return Err(bad(CMD, format!("the stroke is too large to remove ({pixels} pixels, at most {MAX_PIXELS})"))); + } + let area = bounds.intersect(&canvas); + if area.is_empty() { + return Err(bad(CMD, "the stroke is outside the canvas")); + } + let window = area.inflate(crate::fill_cmds::sampling_margin(area).min(MAX_MARGIN)).intersect(&canvas); + let close_loops = flag(p, "closeLoops", true); + let all_layers = flag(p, "sampleAllLayers", false) && targets_pixels(p); + Ok(Plan { id, stroke, window, close_loops, all_layers }) +} + +/// The hole over the plan's window: the stroke's footprint, its enclosed areas and a margin, +/// limited to the selection. Empty when nothing would change. +fn hole(plan: &Plan, sel: Option<&Surface>) -> Vec { + let win = plan.window; + let (w, h) = (win.width() as usize, win.height() as usize); + let (bounds, cov) = stroke_coverage(&plan.stroke); + let mut m = vec![false; w * h]; + let bw = bounds.width() as usize; + for (i, c) in cov.iter().enumerate() { + let (x, y) = (bounds.x0 + (i % bw.max(1)) as i32, bounds.y0 + (i / bw.max(1)) as i32); + if *c > 0.0 && win.contains(x, y) { + let j = (y - win.y0) as usize * w + (x - win.x0) as usize; + if let Some(v) = m.get_mut(j) { + *v = true; + } + } + } + if plan.close_loops { + m = photocraft_algo::remove::close_loops(w, h, &m); + } + m = photocraft_algo::remove::dilate(w, h, &m, GROW); + if let Some(sel) = sel { + for (v, c) in m.iter_mut().zip(coverage(sel, win)) { + *v &= c > 0.0; + } + } + m +} + +/// The filled window and the hole it fills, computed on `doc` (a snapshot). +fn run_remove(doc: &mut Document, plan: &Plan, p: &Value, ctx: &crate::jobs::JobCtx, label: &str) -> Result<(Region, Vec)> { + let hole = hole(plan, doc.selection.as_ref()); + if !hole.iter().any(|m| *m) { + return Err(bad(CMD, "the stroke doesn't cover anything that can change (check the selection)")); + } + if hole.iter().all(|m| *m) { + return Err(bad(CMD, "nothing is left around the stroke to fill it from")); + } + ctx.check()?; + let win = plan.window; + let (img, fmt) = { + let (surf, _) = crate::channel_cmds::target_surface(doc, plan.id, p)?; + let fmt = surf.format(); + let img = if plan.all_layers { composite_region(doc, None, SampleLayers::All, win, fmt) } else { Region::read(surf, win) }; + (img, fmt) + }; + let (w, h) = (win.width() as usize, win.height() as usize); + let mut data = ctx + .stage(0.0, 1.0, label, |ctl| photocraft_algo::remove::remove_with(w, h, img.ch, &img.data, &hole, SEED, ctl)) + .map_err(|_| EngineError::Cancelled)?; + // The fill's seam blending can step slightly past the storable range. + clamp_samples(&fmt, &mut data); + Ok((Region { rect: win, ch: img.ch, data }, hole)) +} + +pub(super) fn remove(s: &mut Session, p: &Value) -> Result { + let plan = plan(s, p)?; + let label = "Remove"; + let d = s.active().ok_or(EngineError::NoDocument)?; + let doc = d.doc.clone(); + let p = p.clone(); + let id = plan.id; + crate::jobs::run( + s, + label, + true, + { + let p = p.clone(); + move |ctx| { + ctx.progress(0.0, label); + let mut work = (*doc).clone(); + run_remove(&mut work, &plan, &p, ctx, label) + } + }, + move |s, (out, hole)| { + let damage = s.edit(label, |doc, _| replace(doc, id, &p, &out, &hole))?; + if let Some(st) = s.active_mut() { + st.last_damage = Some(damage); + } + Ok(json!({ "damage": damage_json(damage) })) + }, + ) +} diff --git a/crates/engine/src/retouch_cmds/tests.rs b/crates/engine/src/retouch_cmds/tests.rs index 6e4c118cef..a41f5445b8 100644 --- a/crates/engine/src/retouch_cmds/tests.rs +++ b/crates/engine/src/retouch_cmds/tests.rs @@ -400,6 +400,7 @@ fn retouch_commands_are_registered_and_need_a_pixel_layer() { "paint.sharpen", "paint.smudge", "paint.historyBrush", + "paint.remove", ]; for id in ids { let spec = crate::commands::find(id).unwrap_or_else(|| panic!("{id} missing")); @@ -416,7 +417,7 @@ fn retouch_commands_are_registered_and_need_a_pixel_layer() { // Paint target (#207): mask, alpha channel, Quick Mask // --------------------------------------------------------------------------------------------- -const RETOUCH_IDS: [&str; 11] = [ +const RETOUCH_IDS: [&str; 12] = [ "paint.cloneStamp", "paint.patternStamp", "paint.healingBrush", @@ -428,6 +429,7 @@ const RETOUCH_IDS: [&str; 11] = [ "paint.sharpen", "paint.smudge", "paint.historyBrush", + "paint.remove", ]; /// Grey noise in 0.3..0.7 (no period along the clone offset used below). @@ -484,10 +486,11 @@ fn retouch_params(id: &str, history_state: usize, target: Value) -> Value { p } -/// Tools whose stroke must visibly change grey noise (Sponge has no colour to change on grey, and -/// healing a noise texture with itself may land on the same values). +/// Tools whose stroke must visibly change grey noise (Sponge has no colour to change on grey, +/// healing a noise texture with itself may land on the same values, and Remove completes this +/// periodic noise exactly). fn changes_gray(id: &str) -> bool { - !matches!(id, "paint.sponge" | "paint.healingBrush" | "paint.spotHealing") + !matches!(id, "paint.sponge" | "paint.healingBrush" | "paint.spotHealing" | "paint.remove") } #[test] @@ -1301,6 +1304,7 @@ fn retouch_strokes_refuse_absurd_point_coordinates() { "paint.spotHealing", "paint.historyBrush", "paint.patternStamp", + "paint.remove", ]; // A single far point first: cheap even without the bound, so a regression fails fast. let far = [json!([[1_000_001, 0]]), json!([[0, 0], [0, -2_000_000]]), json!([[0, 0], [1e300, 0]]), json!([[1e300, 1e300]])]; @@ -1321,3 +1325,91 @@ fn retouch_strokes_refuse_absurd_point_coordinates() { assert_ne!(rgba(&s, 16, 16), before, "{depth}"); } } + +// --------------------------------------------------------------------------------------------- +// Remove Tool +// --------------------------------------------------------------------------------------------- + +/// A flat grey layer with a dark square at 40..52 × 24..36. +fn square_session(depth: u64, mode: &str) -> Session { + let mut s = session(96, 64, depth, mode); + paint_layer(&mut s, |x, y| if (40..52).contains(&x) && (24..36).contains(&y) { [0.05, 0.05, 0.05, 1.0] } else { [0.5, 0.5, 0.5, 1.0] }); + s +} + +/// A ring of points around the square. +fn ring() -> Vec<[f64; 2]> { + (0..=64).map(|i| f64::from(i) / 64.0 * std::f64::consts::TAU).map(|t| [46.0 + 13.0 * t.cos(), 30.0 + 13.0 * t.sin()]).collect() +} + +fn near(a: [f32; 4], b: [f32; 4], tol: f32) -> bool { + a.iter().zip(b).all(|(x, y)| (x - y).abs() <= tol) +} + +#[test] +fn remove_a_ring_around_an_object_removes_it_in_one_undo_step() { + for (depth, mode) in [(8, "rgb"), (16, "gray"), (32, "rgb"), (8, "cmyk"), (16, "lab")] { + let mut s = square_session(depth, mode); + let bg = rgba(&s, 10, 10); + assert!(!near(rgba(&s, 46, 30), bg, 0.05), "{mode}/{depth}: the square is there"); + let past = s.active().unwrap().history.past_len(); + let r = s.execute("paint.remove", json!({"points": ring(), "size": 4})).unwrap(); + assert!(r["damage"].is_array(), "{r}"); + for (x, y) in [(46, 30), (40, 24), (51, 35)] { + assert!(near(rgba(&s, x, y), bg, 0.03), "{mode}/{depth} ({x}, {y}): {:?} vs {bg:?}", rgba(&s, x, y)); + } + assert_eq!(s.active().unwrap().history.past_len(), past + 1, "{mode}/{depth}: one undo step"); + s.execute("edit.undo", json!({})).unwrap(); + assert!(!near(rgba(&s, 46, 30), bg, 0.05), "{mode}/{depth}: undo brings the square back"); + } +} + +#[test] +fn remove_open_strokes_and_the_selection_limit_what_changes() { + let mut s = square_session(8, "rgb"); + let bg = rgba(&s, 10, 10); + // Without closeLoops a ring leaves its inside alone. + s.execute("paint.remove", json!({"points": ring(), "size": 4, "closeLoops": false})).unwrap(); + assert!(!near(rgba(&s, 46, 30), bg, 0.05)); + // A stroke straight over the square removes it. + s.execute("paint.remove", json!({"points": [[38, 30], [54, 30]], "size": 16})).unwrap(); + assert!(near(rgba(&s, 46, 30), bg, 0.03), "{:?}", rgba(&s, 46, 30)); + // Selected elsewhere, a stroke over the square changes nothing. + let mut s = square_session(8, "rgb"); + s.execute("select.rect", json!({"x": 0, "y": 0, "width": 20, "height": 20})).unwrap(); + let err = s.execute("paint.remove", json!({"points": [[46, 30]], "size": 20})).unwrap_err().to_string(); + assert!(err.contains("check the selection"), "{err}"); + assert!(!near(rgba(&s, 46, 30), bg, 0.05)); +} + +#[test] +fn remove_thin_scratches_take_their_surroundings() { + let mut s = session(80, 40, 16, "rgb"); + paint_layer(&mut s, |x, y| if y == 20 { [1.0, 1.0, 1.0, 1.0] } else { [x as f32 / 80.0, 0.4, 0.6, 1.0] }); + s.execute("paint.remove", json!({"points": [[5, 20], [75, 20]], "size": 2})).unwrap(); + for x in [10, 40, 70] { + assert!(near(rgba(&s, x, 20), [x as f32 / 80.0, 0.4, 0.6, 1.0], 0.02), "x {x}: {:?}", rgba(&s, x, 20)); + } +} + +#[test] +fn remove_bad_params_are_errors() { + assert!(Session::new().execute("paint.remove", json!({"points": [[1, 1]]})).is_err(), "no document"); + let mut s = square_session(8, "rgb"); + for p in [ + json!({}), + json!({"points": []}), + json!({"points": "x"}), + json!({"points": [[1, 1]], "size": "big", "closeLoops": "yes", "sampleAllLayers": 3}), + json!({"points": [[1, 1]], "layer": 999_999}), + ] { + let _ = s.execute("paint.remove", p); + } + let err = s.execute("paint.remove", json!({"points": [[-500, -500]], "size": 10})).unwrap_err().to_string(); + assert!(err.contains("outside the canvas"), "{err}"); + // A stroke far off the canvas is refused before its coverage (its whole bounding box) is rendered. + let err = s.execute("paint.remove", json!({"points": [[0, 0], [99_999, 99_999]], "size": 5000})).unwrap_err().to_string(); + assert!(err.contains("too large"), "{err}"); + let err = s.execute("paint.remove", json!({"points": [[48, 32]], "size": 3000})).unwrap_err().to_string(); + assert!(err.contains("nothing is left"), "{err}"); +} diff --git a/crates/ui-egui/src/canvas.rs b/crates/ui-egui/src/canvas.rs index d436b21895..8af9dd6462 100644 --- a/crates/ui-egui/src/canvas.rs +++ b/crates/ui-egui/src/canvas.rs @@ -533,6 +533,7 @@ pub(crate) fn freehand_tool(tool: Tool) -> bool { | Tool::Eraser | Tool::BackgroundEraser | Tool::HistoryBrush + | Tool::Remove | Tool::SpotHealing | Tool::Healing | Tool::CloneStamp @@ -3444,7 +3445,12 @@ fn draw_drag_preview(app: &mut PhotocraftApp, painter: &egui::Painter, xf: &View t if t.is_brushlike() || t == Tool::QuickSelection => { // Retouching strokes preview as a translucent trail of the brush footprint: a mask, // not a brush-wide egui polyline (which zoomed in tessellates into wedges, #189). - let col = Color32::from_white_alpha(if t == Tool::QuickSelection { 40 } else { 60 }); + let col = match t { + Tool::QuickSelection => Color32::from_white_alpha(40), + // The Remove Tool tints what it will remove, like Photoshop's. + Tool::Remove => crate::theme::Tokens::get(painter.ctx()).danger.gamma_multiply(0.45), + _ => Color32::from_white_alpha(60), + }; let Some(st) = app.session.active() else { return }; let size = [st.doc.size.width, st.doc.size.height]; let doc_rect = xf.doc_rect(st.doc.bounds()); diff --git a/crates/ui-egui/src/i18n/cs.tsv b/crates/ui-egui/src/i18n/cs.tsv index 28b22ce72b..d91d3a715d 100644 --- a/crates/ui-egui/src/i18n/cs.tsv +++ b/crates/ui-egui/src/i18n/cs.tsv @@ -1323,6 +1323,7 @@ Remove All Pins Odstranit všechny špendlíky Remove Background Odstranit pozadí Remove Planes Odstranit roviny + Remove Tool Nástroj odstranění Remove stop Odstranit zarážku Remove variable Odstranit proměnnou Rename Přejmenovat diff --git a/crates/ui-egui/src/i18n/de.tsv b/crates/ui-egui/src/i18n/de.tsv index 354231229b..ffca857f42 100644 --- a/crates/ui-egui/src/i18n/de.tsv +++ b/crates/ui-egui/src/i18n/de.tsv @@ -1307,6 +1307,7 @@ Remove All Pins Alle Pins entfernen Remove Background Hintergrund entfernen Remove Planes Flächen entfernen + Remove Tool Entfernen-Werkzeug Remove stop Farbstopp entfernen Remove variable Variable entfernen Rename Umbenennen diff --git a/crates/ui-egui/src/i18n/el.tsv b/crates/ui-egui/src/i18n/el.tsv index 2033883653..5ff515f27d 100644 --- a/crates/ui-egui/src/i18n/el.tsv +++ b/crates/ui-egui/src/i18n/el.tsv @@ -2796,6 +2796,7 @@ Remove All Pins Αφαίρεση όλων των καρφιτσών Remove Background Αφαίρεση φόντου Remove Planes Αφαίρεση επιπέδων προοπτικής + Remove Tool Αφαίρεση (Remove Tool) Remove Warp Split Αφαίρεση διαχωρισμού παραμόρφωσης Remove stop Αφαίρεση σημείου διαβάθμισης Remove variable Αφαίρεση μεταβλητής diff --git a/crates/ui-egui/src/i18n/es.tsv b/crates/ui-egui/src/i18n/es.tsv index aa84c2b473..76dde17e1b 100644 --- a/crates/ui-egui/src/i18n/es.tsv +++ b/crates/ui-egui/src/i18n/es.tsv @@ -1308,6 +1308,7 @@ Remove All Pins Quitar todos los pines Remove Background Quitar el fondo Remove Planes Quitar los planos + Remove Tool Herramienta Quitar Remove stop Quitar punto Remove variable Quitar variable Rename Cambiar nombre diff --git a/crates/ui-egui/src/i18n/fr.tsv b/crates/ui-egui/src/i18n/fr.tsv index 305920f82d..f94d4a982c 100644 --- a/crates/ui-egui/src/i18n/fr.tsv +++ b/crates/ui-egui/src/i18n/fr.tsv @@ -1292,6 +1292,7 @@ Remove All Pins Retirer toutes les épingles Remove Background Supprimer l'arrière-plan Remove Planes Supprimer les plans + Remove Tool Outil Retirer Remove stop Retirer le point Remove variable Retirer la variable Rename Renommer diff --git a/crates/ui-egui/src/i18n/id.tsv b/crates/ui-egui/src/i18n/id.tsv index 8cd30fd3d6..f9974abd95 100644 --- a/crates/ui-egui/src/i18n/id.tsv +++ b/crates/ui-egui/src/i18n/id.tsv @@ -1286,6 +1286,7 @@ Remove All Pins Hapus Semua Pin Remove Background Hapus Background Remove Planes Hapus Bidang + Remove Tool Alat Hapus Remove stop Hapus stop Remove variable Hapus variabel Rename Ganti Nama diff --git a/crates/ui-egui/src/i18n/it.tsv b/crates/ui-egui/src/i18n/it.tsv index 6c357abdf9..ab43fdb86a 100644 --- a/crates/ui-egui/src/i18n/it.tsv +++ b/crates/ui-egui/src/i18n/it.tsv @@ -1307,6 +1307,7 @@ Remove All Pins Rimuovi tutti i pin Remove Background Rimuovi sfondo Remove Planes Rimuovi piani + Remove Tool Strumento Rimuovi Remove stop Rimuovi punto Remove variable Rimuovi variabile Rename Rinomina diff --git a/crates/ui-egui/src/i18n/ja.tsv b/crates/ui-egui/src/i18n/ja.tsv index 407e27f8e0..36cab85125 100644 --- a/crates/ui-egui/src/i18n/ja.tsv +++ b/crates/ui-egui/src/i18n/ja.tsv @@ -2548,6 +2548,7 @@ fontWeight Italic 斜体 Reduce Noise ノイズを軽減 Relief レリーフ Remove Jpeg Artifact JPEG アーティファクトを除去 + Remove Tool 削除ツール Reorder Layer レイヤーを並べ替え Repeat 繰り返し Reticulation 粒状化 diff --git a/crates/ui-egui/src/i18n/ko.tsv b/crates/ui-egui/src/i18n/ko.tsv index 5d047df996..616f6f72b1 100644 --- a/crates/ui-egui/src/i18n/ko.tsv +++ b/crates/ui-egui/src/i18n/ko.tsv @@ -2199,6 +2199,7 @@ cameraRaw Color 색상 Reduce Color Noise 색상 노이즈 감소 Sharpen Details 세부 묘사 선명하게 Remove Jpeg Artifact JPEG 가공물 제거 + Remove Tool 제거 도구 Blade Curvature 조리개 날 곡률 Depth Map 깊이 맵 Focal Distance 초점 거리 diff --git a/crates/ui-egui/src/i18n/nl.tsv b/crates/ui-egui/src/i18n/nl.tsv index 011e7c1e72..9252ecf72c 100644 --- a/crates/ui-egui/src/i18n/nl.tsv +++ b/crates/ui-egui/src/i18n/nl.tsv @@ -2828,6 +2828,7 @@ cameraRaw Color Kleur Reduce Color Noise Kleurruis verminderen Sharpen Details Details verscherpen Remove Jpeg Artifact JPEG-artefacten verwijderen + Remove Tool Verwijdergereedschap Blade Curvature Kromming lamellen Depth Map Dieptekaart Focal Distance Scherpstelafstand diff --git a/crates/ui-egui/src/i18n/pl.tsv b/crates/ui-egui/src/i18n/pl.tsv index f030df93d0..901f1bfe98 100644 --- a/crates/ui-egui/src/i18n/pl.tsv +++ b/crates/ui-egui/src/i18n/pl.tsv @@ -2690,6 +2690,7 @@ cameraRaw Roughness Chropowatość Reduce Color Noise Redukuj szum kolorów Sharpen Details Wyostrz szczegóły Remove Jpeg Artifact Usuń artefakty JPEG + Remove Tool Narzędzie Usuń Blade Curvature Krzywizna listków Depth Map Mapa głębi Focal Distance Odległość ogniskowania diff --git a/crates/ui-egui/src/i18n/pt-br.tsv b/crates/ui-egui/src/i18n/pt-br.tsv index f8b53367ff..9036f2c16c 100644 --- a/crates/ui-egui/src/i18n/pt-br.tsv +++ b/crates/ui-egui/src/i18n/pt-br.tsv @@ -1291,6 +1291,7 @@ Remove All Pins Remover todos os alfinetes Remove Background Remover plano de fundo Remove Planes Remover planos + Remove Tool Ferramenta Remover Remove stop Remover parada Remove variable Remover variável Rename Renomear diff --git a/crates/ui-egui/src/i18n/ru.tsv b/crates/ui-egui/src/i18n/ru.tsv index c67a85a7f4..0ba9ff0d11 100644 --- a/crates/ui-egui/src/i18n/ru.tsv +++ b/crates/ui-egui/src/i18n/ru.tsv @@ -1660,6 +1660,7 @@ Add Pins Добавить булавки Delete Pins Удалить булавки Remove All Pins Удалить все булавки + Remove Tool Инструмент «Удаление» Pin Depth: Глубина булавки: Bring the selected pin forward Переместить выбранную булавку вперёд Send the selected pin backward Переместить выбранную булавку назад diff --git a/crates/ui-egui/src/i18n/uk.tsv b/crates/ui-egui/src/i18n/uk.tsv index d20554d38a..86b88a73dd 100644 --- a/crates/ui-egui/src/i18n/uk.tsv +++ b/crates/ui-egui/src/i18n/uk.tsv @@ -2853,6 +2853,7 @@ cameraRaw Masking Маскування Reduce Color Noise Зменшити колірний шум Sharpen Details Підвищити різкість деталей Remove Jpeg Artifact Прибрати артефакти JPEG + Remove Tool Інструмент «Вилучення» Blade Curvature Кривина пелюсток діафрагми Depth Map Карта глибини Focal Distance Фокусна відстань diff --git a/crates/ui-egui/src/i18n/zh-hans.tsv b/crates/ui-egui/src/i18n/zh-hans.tsv index 8b742eb8e0..f8a381a8c9 100644 --- a/crates/ui-egui/src/i18n/zh-hans.tsv +++ b/crates/ui-egui/src/i18n/zh-hans.tsv @@ -1317,6 +1317,7 @@ Remove All Pins 移除所有控制点 Remove Background 移除背景 Remove Planes 移除平面 + Remove Tool 移除工具 Remove stop 移除色标 Remove variable 移除变量 Rename 重命名 diff --git a/crates/ui-egui/src/i18n/zh-hant.tsv b/crates/ui-egui/src/i18n/zh-hant.tsv index 0e8be787f1..89cd8ce41c 100644 --- a/crates/ui-egui/src/i18n/zh-hant.tsv +++ b/crates/ui-egui/src/i18n/zh-hant.tsv @@ -1314,6 +1314,7 @@ Remove All Pins 移除所有圖釘 Remove Background 移除背景 Remove Planes 移除平面 + Remove Tool 移除工具 Remove stop 移除色標 Remove variable 移除變數 Rename 重新命名 diff --git a/crates/ui-egui/src/icon_data.rs b/crates/ui-egui/src/icon_data.rs index a08bcd5a15..d845ce38eb 100644 --- a/crates/ui-egui/src/icon_data.rs +++ b/crates/ui-egui/src/icon_data.rs @@ -10,6 +10,7 @@ pub static ICONS: &[(&str, &[u8])] = &[ ("arrow-left-right", include_bytes!("../../../assets/icons/arrow-left-right.svg")), ("ban", include_bytes!("../../../assets/icons/ban.svg")), ("bandage", include_bytes!("../../../assets/icons/bandage.svg")), + ("bandage-sparkle", include_bytes!("../../../assets/icons/bandage-sparkle.svg")), ("blend", include_bytes!("../../../assets/icons/blend.svg")), ("brush", include_bytes!("../../../assets/icons/brush.svg")), ("brush-cleaning", include_bytes!("../../../assets/icons/brush-cleaning.svg")), diff --git a/crates/ui-egui/src/icons.rs b/crates/ui-egui/src/icons.rs index fbdc700ad8..09ad7ab1b8 100644 --- a/crates/ui-egui/src/icons.rs +++ b/crates/ui-egui/src/icons.rs @@ -119,6 +119,7 @@ pub fn tool_icon(t: Tool) -> &'static str { Tool::RotateView => "compass", Tool::Zoom => "zoom-in", Tool::SpotHealing | Tool::Healing => "bandage", + Tool::Remove => "bandage-sparkle", Tool::Patch => "lasso-select", Tool::ContentAwareMove => "arrow-left-right", Tool::RedEye => "eye", diff --git a/crates/ui-egui/src/panels.rs b/crates/ui-egui/src/panels.rs index 09859478bf..3a1d364566 100644 --- a/crates/ui-egui/src/panels.rs +++ b/crates/ui-egui/src/panels.rs @@ -24,7 +24,7 @@ const TOOL_SECTIONS: &[&[&[Tool]]] = &[ &[Tool::Eyedropper, Tool::Ruler, Tool::Note, Tool::Count], ], &[ - &[Tool::SpotHealing, Tool::Healing, Tool::Patch, Tool::ContentAwareMove, Tool::RedEye], + &[Tool::Remove, Tool::SpotHealing, Tool::Healing, Tool::Patch, Tool::ContentAwareMove, Tool::RedEye], &[Tool::Brush, Tool::Pencil, Tool::MixerBrush], &[Tool::CloneStamp, Tool::PatternStamp], &[Tool::HistoryBrush], @@ -3710,6 +3710,8 @@ mod type_flyout_tests { let j = TOOL_SECTIONS.iter().flat_map(|section| section.iter()).find(|slot| slot.contains(&Tool::SpotHealing)).expect("J group"); assert!(j.contains(&Tool::RedEye), "{j:?}"); assert_eq!(j.last(), Some(&Tool::RedEye)); + // Remove leads the flyout, so the slot shows it until another J tool is picked. + assert_eq!(j.first(), Some(&Tool::Remove)); } #[test] diff --git a/crates/ui-egui/src/retouch_ui.rs b/crates/ui-egui/src/retouch_ui.rs index b809f2ff4b..b17fc654a4 100644 --- a/crates/ui-egui/src/retouch_ui.rs +++ b/crates/ui-egui/src/retouch_ui.rs @@ -66,6 +66,7 @@ pub fn finish_stroke(app: &mut PhotocraftApp, tool: Tool, points: &[[f64; 3]], m let pts = json!(points); let (cmd, mut p): (&str, Value) = match tool { Tool::SpotHealing => ("paint.spotHealing", json!({"type": o.spot_type, "sampleAllLayers": o.sample_all_layers})), + Tool::Remove => ("paint.remove", json!({"sampleAllLayers": o.sample_all_layers})), Tool::MixerBrush => ("paint.mixerBrush", json!({})), Tool::PatternStamp => { let mut p = json!({ @@ -308,6 +309,9 @@ pub fn options_bar(app: &mut PhotocraftApp, ui: &mut egui::Ui, tool: Tool) -> bo crate::widgets::vline(ui, 22.0); crate::widgets::checkbox(ui, &mut o.sample_all_layers, tl!("Sample All Layers")); } + Tool::Remove => { + crate::widgets::checkbox(ui, &mut o.sample_all_layers, tl!("Sample All Layers")); + } Tool::Patch => { opt(ui, tl!("Patch:")); for (k, l) in [("source", tl!("Source")), ("destination", tl!("Destination"))] { @@ -541,7 +545,7 @@ mod tests { #[test] fn sample_all_layers_reaches_the_command() { // #207, #731: the options-bar checkbox reaches the command. - for tool in [Tool::SpotHealing, Tool::Blur, Tool::Sharpen, Tool::Smudge] { + for tool in [Tool::SpotHealing, Tool::Remove, Tool::Blur, Tool::Sharpen, Tool::Smudge] { for all in [false, true] { let mut app = app(); stripes(&mut app, 4, "pixels"); @@ -761,13 +765,37 @@ mod tests { assert!(app.session.active().is_some_and(|st| st.doc.selection.is_some()), "the selection stays"); } + #[test] + fn remove_tool_stroke_removes_what_it_covers() { + let mut app = app(); + app.run("paint.pencil", json!({"points": [[0, 30], [100, 30]], "size": 200, "color": "#808080"})).unwrap(); + app.run("paint.pencil", json!({"points": [[30, 20], [30, 40]], "size": 4, "color": "#000000"})).unwrap(); + let bg = active(&app).surface().unwrap().rgba(70, 30); + app.ui.tool = Tool::Remove; + let m = egui::Modifiers::NONE; + tool_event(&mut app, ToolEvent::Down { x: 30.0, y: 18.0, pressure: 1.0 }, m); + tool_event(&mut app, ToolEvent::Move { x: 30.0, y: 42.0, pressure: 1.0 }, m); + tool_event(&mut app, ToolEvent::Up { x: 30.0, y: 42.0 }, m); + assert!(!app.ui.status_error, "{}", app.ui.status); + assert_eq!(app.session.journal.last().map(|(id, _)| id.as_str()), Some("paint.remove")); + let px = active(&app).surface().unwrap().rgba(30, 30); + assert!(px.iter().zip(bg).all(|(a, b)| (a - b).abs() < 0.02), "{px:?} vs {bg:?}"); + } + #[test] fn red_eye_is_in_the_j_flyout() { - let j = [Tool::SpotHealing, Tool::Healing, Tool::Patch, Tool::ContentAwareMove, Tool::RedEye]; + let j = [Tool::Remove, Tool::SpotHealing, Tool::Healing, Tool::Patch, Tool::ContentAwareMove, Tool::RedEye]; assert!(j.contains(&Tool::RedEye)); assert!(j.iter().all(|t| t.key() == 'J')); } + #[test] + fn j_cycles_from_remove() { + // J from another group picks the first J tool, and repeated J goes on in this order. + let group: Vec = Tool::ALL.iter().copied().filter(|t| t.key() == 'J').collect(); + assert_eq!(group, [Tool::Remove, Tool::SpotHealing, Tool::Healing, Tool::Patch, Tool::ContentAwareMove, Tool::RedEye]); + } + #[test] fn s_group_is_clone_stamp_then_pattern_stamp() { assert_eq!(Tool::CloneStamp.key(), 'S'); diff --git a/crates/ui-egui/src/state.rs b/crates/ui-egui/src/state.rs index fc9d6ddb68..4e75923b83 100644 --- a/crates/ui-egui/src/state.rs +++ b/crates/ui-egui/src/state.rs @@ -87,6 +87,7 @@ pub enum Tool { Hand, RotateView, Zoom, + Remove, SpotHealing, Healing, Patch, @@ -117,7 +118,7 @@ pub enum Tool { } impl Tool { - pub const ALL: [Tool; 52] = [ + pub const ALL: [Tool; 53] = [ Tool::Move, Tool::RectMarquee, Tool::EllipseMarquee, @@ -143,6 +144,7 @@ impl Tool { Tool::Hand, Tool::RotateView, Tool::Zoom, + Tool::Remove, Tool::SpotHealing, Tool::Healing, Tool::Patch, @@ -201,6 +203,7 @@ impl Tool { Tool::Hand => "Hand Tool", Tool::RotateView => "Rotate View Tool", Tool::Zoom => "Zoom Tool", + Tool::Remove => "Remove Tool", Tool::SpotHealing => "Spot Healing Brush Tool", Tool::Healing => "Healing Brush Tool", Tool::Patch => "Patch Tool", @@ -241,6 +244,7 @@ impl Tool { | Tool::MixerBrush | Tool::Eraser | Tool::BackgroundEraser + | Tool::Remove | Tool::SpotHealing | Tool::Healing | Tool::CloneStamp @@ -270,7 +274,7 @@ impl Tool { Tool::Hand => 'H', Tool::RotateView => 'R', Tool::Zoom => 'Z', - Tool::SpotHealing | Tool::Healing | Tool::Patch | Tool::ContentAwareMove | Tool::RedEye => 'J', + Tool::Remove | Tool::SpotHealing | Tool::Healing | Tool::Patch | Tool::ContentAwareMove | Tool::RedEye => 'J', Tool::CloneStamp | Tool::PatternStamp => 'S', Tool::HistoryBrush => 'Y', Tool::Blur | Tool::Sharpen | Tool::Smudge => '\0', @@ -1106,7 +1110,10 @@ mod tests { assert!(!Tool::RedEye.is_brushlike()); assert_eq!(Tool::from_name("patternStamp"), Some(Tool::PatternStamp)); assert_eq!(Tool::from_name("Pattern Stamp Tool"), Some(Tool::PatternStamp)); - assert_eq!(Tool::ALL.len(), 52); + assert_eq!(Tool::ALL.len(), 53); + assert_eq!(Tool::from_name("Remove Tool"), Some(Tool::Remove)); + assert_eq!(Tool::Remove.key(), 'J'); + assert!(Tool::Remove.is_brushlike()); assert_eq!(Tool::from_name("RotateView"), Some(Tool::RotateView)); assert_eq!(Tool::from_name("Rotate View Tool"), Some(Tool::RotateView)); assert_eq!(Tool::RotateView.key(), 'R'); diff --git a/docs/development.md b/docs/development.md index 2afc528541..4b0f091143 100644 --- a/docs/development.md +++ b/docs/development.md @@ -395,6 +395,27 @@ Input calls (`ui.key`, `ui.type`, `ui.click`) now reply only after the app has p so a following `ui.inspect` observes their effect. +## Remove Tool quality + +`cargo run --release -p photocraft-engine --example remove_quality -- [out dir]` fills +holes (discs and thin lines over background) in five photos with `photocraft_algo::remove` and +prints, per hole, the PSNR against the original, `texture` (the fill's mean gradient over the +original's: below 1 it is smoother than what it replaces) and `seam` (the same on the band along the +hole's edge: above 1 the edge shows). With an out dir it writes original | hole | fill crops. The +photos are public domain, not committed; download them at 1920 px wide into `` as: + +| File | Photo | +|---|---| +| `01.jpg` | [The Tetons and the Snake River](https://commons.wikimedia.org/wiki/File:Adams_The_Tetons_and_the_Snake_River.jpg), Ansel Adams, 1942 (U.S. National Archives) | +| `02.jpg` | [Ansel Adams, National Archives 79-AAB-01](https://commons.wikimedia.org/wiki/File:Ansel_Adams_-_National_Archives_79-AAB-01.jpg) (U.S. National Archives) | +| `03.jpg` | [Hiking at Crane Meadows National Wildlife Refuge](https://commons.wikimedia.org/wiki/File:Hiking_at_Crane_Meadows_National_Wildlife_Refuge_(49018614166).jpg) (USFWS) | +| `04.jpg` | [Meadow Anemone, Lake Andes National Wildlife Refuge](https://commons.wikimedia.org/wiki/File:Meadow_Anemone_Lake_Andes_National_Wildlife_Refuge_South_Dakota_(53923253226).jpg) (USFWS) | +| `06.jpg` | [Line5070, NOAA Photo Library](https://commons.wikimedia.org/wiki/File:Line5070_-_Flickr_-_NOAA_Photo_Library.jpg) (NOAA) | + +`https://commons.wikimedia.org/wiki/Special:FilePath/?width=1600` serves each at the +size the cases were placed for (Commons rounds it to 1920). Timing on a 24 MP document through the +command: `cargo run --release -p photocraft-engine --example remove_bench`. + ## Test corpora Real files are our best oracles, but they are large binaries, so they never go into this diff --git a/docs/roadmap.md b/docs/roadmap.md index de8d08027e..182c521f09 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -132,6 +132,16 @@ Bounding the mask table to the hole and pruning losing SSD candidates reduced ta by about 90–93 percent there. Full 24–36 MP kernel inputs improved 4.60–6.24x, but normal engine strokes already use a cropped region. See [method and limits](proximity-match-performance.md). +2026-10-09: the Remove tool is in the J flyout (`paint.remove`, a background job). A stroke +around an object also removes what it encloses. The fill is non-local patch completion with +texture features (A. Newson et al., IPOL 2017: `crates/algo/src/nonlocal.rs`), finished by a +best-patch copy and gradient-domain seam hiding. On 20 holes in public-domain photos +(`remove_quality`, `docs/development.md` › Remove Tool quality) the fill keeps 0.89 of the +original's texture (Wexler/PatchMatch completion as Content-Aware Fill uses it: 0.54). On a 24 MP +document a 100 px ring takes 120 ms and a 1000 px ring 2.5 s (`remove_bench`). It is not +generative: large objects over complex structure fill less convincingly than Photoshop's AI mode +(#41). + 2026-10-08: the tools update above is checked against the current toolbar groups in [`panels.rs`](../crates/ui-egui/src/panels.rs), the Pencil and tool-cycle tests in [`pencil_tests.rs`](../crates/ui-egui/src/pencil_tests.rs), Patch's live-preview test in diff --git a/scorecard/tools.toml b/scorecard/tools.toml index 07153ac67a..fdda94529b 100644 --- a/scorecard/tools.toml +++ b/scorecard/tools.toml @@ -277,3 +277,10 @@ target = "Crop gear menu: Show Cropped Area (H), Enable Crop Shield with Match C status = "done" issue = 1919 note = "crop_shield.rs: fill() resolves the shield (canvas.rs paints it around upright and turned frames), gear popup in the options bar, H ahead of the Hand key in shortcuts.rs, Custom… opens the Color Picker; ToolOptions crop_shield, ui.set cropShield. No Classic Mode / Auto Center Preview (PhotoCraft is always Classic Mode)" + +[[item]] +id = "TOOL-213-19" +target = "Remove tool: paint over an object (or circle it) and it is filled from its surroundings" +status = "partial" +issue = 213 +note = "paint.remove (retouch_cmds/remove.rs, background job) + Tool::Remove in the J flyout; loops close over what they enclose; fill by non-local patch completion with texture features, onion-peel start, best-patch copy and gradient-domain seam hiding (algo/nonlocal.rs, Newson et al. IPOL 2017); sparse holes in tiles (algo/remove.rs). remove_quality on 20 holes in public-domain photos: the fill keeps 0.89 of the texture it replaces (0.54 with Content-Aware Fill's completion), edge gradient 1.03 of the original's; remove_bench 24 MP: 100 px ring 120 ms, 1000 px ring 2.5 s, 3000 px line 1.7 s. Missing: generative AI mode, Remove after each stroke off, live preview of the result"