mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-18 20:22:39 +00:00
bff4be14c3
## What this changes Recognize text now finds small print on large, dense pages (newspapers, small print, blurred photos). The ocrs detection model reads a fixed 600 × 800 pixel input, so ocrs shrinks every page to that size before detecting words. A full page at 300 dpi (2480 × 3508) is shrunk about four times, and small letters disappear. With this change: - **When (size):** an axis is cut in two when the page is at least twice the detector's input along it (width ≥ 1200 px, height ≥ 1600 px for the ocrs models). Any full page at 300 dpi becomes 2 × 2 tiles; a landscape page may become 2 × 1. The input size is read from the detection model when it loads. Pages below that size (72 to 150 dpi renders, screenshots) are read exactly as before and cost nothing extra. - **When (text size):** a page that passes the size rule is tiled only if its text is small once the page is shrunk to the detector's input, decided by a cheap pixel signal, the **line pitch** (`crates/ocr/src/pitch.rs`). The page is box-averaged to a grey copy of at most 1600 px a side and cut into a 6 × 6 grid of blocks. In each block, the ink is summed per pixel row, the slow trend (shading, pictures) is removed with a moving average, and the autocorrelation is taken. The block autocorrelations are averaged, and the first strong peak is the pitch. Lines in columns that don't line up still agree on the period, and blank blocks are skipped. The same runs on pixel columns for sideways pages, and the axis with the stronger peak wins. The pitch is converted to detector pixels (pitch × 800 / page height). The rule: - pitch **under 9 detector pixels**: tile; - ink but **no clear line spacing** (no autocorrelation peak above 0.05): tile, because the measurement showed these pages (blurred, creased, oblique or moiré photos) gain on average; - large print, and blank pages: detect whole, as before. It costs 6.6 ms per page on average (median 5.1 ms, max 45 ms) and reads every pixel once. - **How:** tiles overlap by 4 % of the page on each side of a cut. Words are **detected** tile by tile. A word is kept only by the tile whose core cell (its share of the page without the overlap) holds its centre, so a word in an overlap counts once. The words are moved back to page pixels, and then lines are formed and **recognised on the whole page** as before. Lines that cross a cut stay single lines, and recognition reads the same pixels it read before. - The edge cells reach to infinity, so every finite word centre belongs to exactly one tile, including boxes that stick out past the page. The change is small and lives in `crates/ocr/src/tiles.rs` plus a `detect_words` step in `Ocr::recognize`. While I was there I also made the existing RGBA length check use checked arithmetic (`width * height * 4` was a `u32` product). ### Measured MDPBench de/es/fr (480 pages), official evaluation with CDM off, `en` recogniser. I ran the same harness build with and without the change (the baseline reproduces the earlier one exactly): | | Digital | Photo | All | Time per page | |---|---|---|---|---| | before | 78.9 | 41.5 | 50.8 | 0.39 s (190 s total) | | tiling every large page (this PR's first commit) | 80.4 | 42.8 | 52.2 | 0.58 s (281 s total) | | **tiling only small or unclear print (this PR)** | **81.2** | **42.9** | **52.5** | 0.54 s (259 s total) | | earlier prototype (full OCR per tile, outside the app) | 81.4 | 42.4 | 52.1 | n/a | The three rows of this PR come from one harness binary with the rule switched by an environment variable. Each was timed twice (190/191, 281/281, 259/259 s). Per language, the line-pitch rule against tiling every large page: DE 62.9 vs 62.7, ES 47.9 vs 47.8, FR 46.8 vs 46.1. It tiles **214 of the 457** pages that pass the size rule. **How the rule was chosen.** I measured first, in Python on the per-page scores of the official evaluation (`overall` per page, base vs. tiled). Per-page score gain from tiling, by measured pitch: | pitch (detector px) | pages | mean gain | pages gaining / losing > 0.02 | |---|---|---|---| | < 4 | 40 | +0.018 | 10 / 4 | | 4–6 | 40 | **+0.114** | 24 / 2 | | 6–10 | 82 | +0.004 | 10 / 6 | | 10–14 | 80 | **−0.013** | 5 / 26 | | 14–35 | 144 | −0.003 | 22 / 25 | | no clear line spacing | 82 | +0.029 | – | The gain sits at small pitch, and pitches of 10–14 detector pixels (ordinary body text on a full page) **lose** a little when tiled. Picking tiled or base per page by this rule, and running the official evaluation on that mix, predicted exactly what the Rust build then scored (52.5 / 81.2 / 42.9). Thresholds between 8 and 10 give 52.4–52.5. Treating "no clear spacing" as "don't tile" instead gives 52.0 with only 132 pages tiled. The choice is tuned on these 480 pages, so take the last few tenths with that in mind. **Time.** The saving is smaller than the page count suggests. The pages that stay tiled are the dense ones, where tiling finds many more words, and recognising those words costs more than the three extra detector runs. Tiling every large page cost +91 s. The pitch rule costs +69 s, so it removes about a quarter of the overhead and still scores higher. Word recall in the benchmark's own per-kind table, before → after: de digital 61.9 → 78.5 %, fr digital 17.8 → 24.5 %, de/es/fr photos 38.1/37.7/10.1 → 42.1/43.2/12.6 %. Sideways photos are unchanged; those need rotation, which is a separate PR. Detection runs four times instead of once on a tiled page (machine: Ryzen AI Max+ 395, CPU). ### How it composes with the open OCR PRs - #5157 (reading order) ends `recognize` in `layout::reading_order`. This PR changes only the detection step before `find_text_lines`, so tiles are merged before lines are formed and before reading order runs, which is the order we want. I measured both together: the numbers above come from a harness tree that has #5157's reading order applied. The two diffs touch neighbouring lines in `recognize`, so whichever merges second needs a trivial rebase. - #5171 (layout regions) and #5058 (line padding) work on lines and regions after detection, so they compose the same way. ## How you verified it - [x] `cargo fmt --check` - [x] `cargo clippy -p pdfcraft-ocr --all-targets -- -D warnings` - [x] `cargo test -p pdfcraft-ocr`: 26 passed, **with the OCR models installed** (CI has none, so CI skips the model-backed tests). With the models, I also ran the OCR tests of `pdfcraft-engine` (`ocr`, `recogni` filters) and `pdfcraft-automation` (`ocr`, `scan`) locally, and all pass. - [x] `cargo xtask layers`, `cargo xtask assets`, `cargo xtask parity` **Tests.** - Unit tests, no models (`crates/ocr/src/tiles.rs`): the tile geometry for small pages and for A4 at 300 dpi; every word centre, on cuts, edges and outside the page, belongs to exactly one tile, and NaN or infinity belongs to none; cropping, including out-of-range and overflowing crops, returns `None` without panicking. - Unit tests, no models (`crates/ocr/src/pitch.rs`), on synthetic A4 pages at 300 dpi: a blank page is not tiled; large print (lines every 100 px, 22.8 detector px) is not; small print (every 30 px, 6.8 detector px) is, in grey, RGB and RGBA alike; sideways small print (lines running down the page) is; noise (ink without lines) is; the first strong autocorrelation peak is the pitch, not its multiple; empty, 1-pixel, short or long buffers, 0 or 5 channels, overflowing sizes, a zero-sized detector and slivers give an answer without panicking. - Model-backed test `finds_small_print_on_a_large_page`: an A4 page at 300 dpi with four lines of small print. One sits inside a tile, one crosses the vertical cut, one is half in the overlap, and one crosses the horizontal cut. Each line must come back as exactly two words at the right place in page pixels. - **Seen failing by reverting the fix:** - Detection on the whole page instead of tiles: 5 words instead of 8, unreadable ("FIOWOR", "In3"). - Keeping every tile's words without the centre-in-core rule: 14 instead of 8, because the lines in the overlaps doubled. - Finite edge cells: `every_word_centre_belongs_to_exactly_one_tile` fails. - Never cutting: the geometry tests fail. - Line-pitch rule, each mutation reverted afterwards: tiling every page (the first commit's behaviour) fails `a_blank_page_is_not_tiled` and `large_print_is_not_tiled`. Never tiling on a measured pitch fails `small_print_is_tiled` and `sideways_small_print_is_tiled`. Rows only, without the column profile, fails `sideways_small_print_is_tiled`. Taking the strongest peak instead of the first strong one fails `the_first_strong_peak_is_the_pitch`. Not tiling unclear pages fails `ink_without_lines_is_tiled`. No detrending fails four tests. With the threshold lowered to 5 detector pixels, the model-backed `finds_small_print_on_a_large_page` finds 5 words instead of 8, so it exercises the rule too (its page measures 5.8). ## Never-crash - [x] No new `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!` outside tests - [x] No indexing or slicing with input-derived positions: crops use checked arithmetic and `get`; tile bounds are clamped to the page; float-to-int casts are of values clamped to `0..=len`. The pitch measure checks that the buffer matches width × height × channels with checked arithmetic, reads pixels through iterators and `get`, and works on at most 1600 × 1600 values whatever the page size. It divides only by non-zero counts, so it returns an answer for any input. - [x] No `unsafe` - [x] n/a, this fixes no crash ## Provenance - [x] No asset from an Adobe product - [x] No font files - [x] No GPL/AGPL code - [x] Behaviour comes from our own measurements on MDPBench. The approach (overlapping tiles, keep by centre; line pitch from the autocorrelation of a projection profile) is a generic technique, not read from another implementation. - [x] No new assets. One new dependency edge, `rten-imageproc` 0.26 (MIT/Apache-2.0), which was already in the tree through `ocrs`. ## Security - [x] No path handling changed - [x] No MCP or control channel changes - [x] No network access ## Parity and reach - [x] `ocr.recognize-text` notes say when pages are tiled (size and line pitch), and that the model-backed test runs only where the models are installed. The tiling and pitch unit tests are evidence. - [x] Reachable headlessly: this is inside `Ocr::recognize`, so `ocr_recognize` and `ocr_recognize_files` get it unchanged. - [x] No documented behaviour changed ## Anything a reviewer should look at first - **The tiling rule:** an axis is cut only if each half still fills the detector's input, so a tile is never padded. A lower threshold would also tile mid-size pages. I kept it conservative so that small pages cost nothing. The prototype tiled every page; under this rule, 468 of the 481 benchmark images are cut along at least one axis and 337 along both. - **The line-pitch rule (`pitch.rs`).** Its constants (1600 px working size, 6 × 6 blocks, peak floor 0.05, first peak ≥ 60 % of the strongest, 9 detector px) were chosen on MDPBench as described above. "No clear line spacing → tile" is deliberate. My first idea was the opposite, but the measurement favoured tiling. - **Detect per tile, recognise on the whole page**, rather than full OCR per tile like the prototype. This is simpler: there are no lines to stitch back together and no rows to re-sort. It scores about the same as the prototype (all 52.2 vs 52.1, photo 42.8 vs 42.4, digital 80.4 vs 81.4).