From fe05b73d08bf2095cf9d769042df1f255665ebe4 Mon Sep 17 00:00:00 2001 From: Grzegorz Martens Date: Mon, 26 Oct 2026 12:09:03 +0100 Subject: [PATCH] M11: JBIG2 symbol coding: tests for region numbering; the symbol cap applies to each plan Review of the previous commit: - Nothing checked the segment numbers the region split assigns: a mutant numbering every region 2 decoded. The many-marks test now reads each segment's number, and has a bar too tall to be a symbol, so the generic region is numbered after the last text region. - The decoder's 32,767-symbol limit was checked against all distinct marks, before either plan, so an image with more distinct marks lost the second plan, which codes only the repeated ones. The limit now applies to each plan (MAX_CODED_SYMBOLS); finding marks keeps its memory bound of 65,536 (MAX_SYMBOLS), as before the previous commit. A test has 32,768 distinct marks above a page of repeated text, which encode_jbig2 still symbol codes through the second plan. - The dense-page optimizer test now checks that it takes several text regions, and a compile-time assert keeps segment numbers within a byte's reach. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG --- crates/filters/README.md | 4 +- crates/filters/src/jbig2/symbols.rs | 114 +++++++++++++++++++--------- crates/optimize/tests/optimize.rs | 11 +++ 3 files changed, 93 insertions(+), 36 deletions(-) diff --git a/crates/filters/README.md b/crates/filters/README.md index 0b06a485b6..333ee1e59b 100644 --- a/crates/filters/README.md +++ b/crates/filters/README.md @@ -44,7 +44,7 @@ encode_jbig2_generic(data, columns, rows) -> Result, FilterError> // t | ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` | | RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 | | CCITTFaxDecode | not here (`Unsupported`; renderers decode it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt | -| JBIG2Decode | not here (`Unsupported`; the renderer and image export decode it with `hayro-jbig2`) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 32,767 symbols (that decoder refuses more) or 16 MiB of symbol pixels, or more than 2^24 rows, and the generic coding is used | +| JBIG2Decode | not here (`Unsupported`; the renderer and image export decode it with `hayro-jbig2`) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, or more than 2^24 rows, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made | ## Tests @@ -54,7 +54,7 @@ encode_jbig2_generic(data, columns, rows) -> Result, FilterError> // t - `ccitt.rs`: Group 4 output decoded by `hayro-ccitt` (dev-dependency, an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics. - `mq.rs`: the arithmetic coder reproduces Annex H.2's 30 bytes from its 256 decisions. - `jbig2.rs`: output decoded by `hayro-jbig2` (dev-dependency, an independent decoder) matches the input bits: a text-like page (smaller than Group 4), edges, padding, all-white and all-black regions (typical rows), and property tests over bitmaps of noise and of runs; any input returns rather than panics. -- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 32,767 symbols or 2^24 symbol pixels the image is coded generically, at each cap it is still looked at, and at the two symbol caps it decodes; more than 10,000 marks are placed by several text regions, the dictionary retained for each but the last; any input returns rather than panics. +- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics. - `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`). ## Status diff --git a/crates/filters/src/jbig2/symbols.rs b/crates/filters/src/jbig2/symbols.rs index a34653e748..e9fed24441 100644 --- a/crates/filters/src/jbig2/symbols.rs +++ b/crates/filters/src/jbig2/symbols.rs @@ -21,18 +21,25 @@ const MAX_SYMBOL_SIDE: u32 = 256; /// which suits it better anyway; this bounds the memory the marks take. const MAX_RUNS: usize = 1 << 20; -/// The most distinct symbols, and the most pixels their bitmaps may hold together. The decoder -/// PdfCraft renders JBIG2 with (`hayro-jbig2`) refuses 32,768 symbols or more, and others may -/// too: past it the image is coded generically. -const MAX_SYMBOLS: usize = (1 << 15) - 1; +/// The most distinct marks sorted into symbols, and the most pixels their bitmaps may hold +/// together: past either the image is left to generic coding. These bound the memory they take. +const MAX_SYMBOLS: usize = 1 << 16; const MAX_SYMBOL_PIXELS: usize = 1 << 24; +/// The most symbols one plan codes. The decoder PdfCraft renders JBIG2 with (`hayro-jbig2`) +/// refuses 32,768 symbols or more, and others may too, so a plan with more isn't made. +const MAX_CODED_SYMBOLS: usize = (1 << 15) - 1; + /// The most symbol instances one text region places. The decoder PdfCraft renders JBIG2 with /// refuses a text region of more than 10,000 (a page it then draws blank), so a page with more is -/// placed by several regions, each over a band of rows, all referring to the one dictionary. At -/// most [`MAX_RUNS`] marks keeps their segment numbers below 256. +/// placed by several regions, each over a band of rows, all referring to the one dictionary. const MAX_REGION_INSTANCES: usize = 10_000; +// Segment numbers stay at most 256, so a referred-to one takes a byte (7.2.5): the page, the +// dictionary, a text region for each MAX_REGION_INSTANCES of at most MAX_RUNS marks, and the +// generic region. +const _: () = assert!(MAX_RUNS.div_ceil(MAX_REGION_INSTANCES) + 3 <= 256); + /// The most rows symbol coded. With rows at most 2^20 samples wide, every coordinate and /// difference coded then fits the 32-bit integers Annex A codes. const MAX_ROWS: u32 = 1 << 24; @@ -94,7 +101,8 @@ struct Instance { /// `rows` rows of `columns` 1-bit PDF samples (black 0), symbol coded: the smaller of two plans, /// every mark of at most [`MAX_SYMBOL_SIDE`] a side as a symbol, or only those with a copy. -/// `None` when neither plan has two symbols, or the image is too large or too busy to look at. +/// `None` when neither plan has from two to [`MAX_CODED_SYMBOLS`] symbols, or the image is too +/// large or too busy to look at. pub(super) fn encode(samples: &[u8], columns: u32, rows: u32) -> Option> { let page = page(samples, columns, rows)?; [1, 2].into_iter().filter_map(|copies| plan(&page, columns, rows, copies, None)).min_by_key(Vec::len) @@ -294,13 +302,14 @@ fn symbols(page: &mut Page) -> Option<()> { /// One plan's stream: the symbols with at least `copies` uses in a symbol dictionary, text /// regions placing them (in strips of 2^`log_strips` rows, or whichever size codes smallest), and -/// the other marks in a generic region. `None` with fewer than two such symbols: their IDs would -/// take no bits at all (SBSYMCODELEN 0), a corner of the standard left alone here. +/// the other marks in a generic region. `None` with fewer than two such symbols (their IDs would +/// take no bits at all, SBSYMCODELEN 0, a corner of the standard left alone here) or more than +/// [`MAX_CODED_SYMBOLS`]. fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option) -> Option> { let symbol = |s: u32| page.symbols.get(s as usize); // The dictionary's order (6.5.5): height classes from the shortest, each narrowest first. let mut kept: Vec = (0..page.symbols.len() as u32).filter(|s| symbol(*s).is_some_and(|s| s.uses >= copies)).collect(); - if kept.len() < 2 { + if kept.len() < 2 || kept.len() > MAX_CODED_SYMBOLS { return None; } kept.sort_by_key(|s| symbol(*s).map(|s| (s.height, s.width))); @@ -628,9 +637,9 @@ mod tests { page } - /// A segment's type, whether it is retained, and the segments it refers to, each with its - /// retain bit. - type Segment = (u8, bool, Vec<(u8, bool)>); + /// A segment's number and type, whether it is retained, and the segments it refers to, each + /// with its retain bit. + type Segment = (u32, u8, bool, Vec<(u8, bool)>); /// The segments of a stream in the embedded organisation, from their headers (7.2). fn segments(data: &[u8]) -> Vec { @@ -641,14 +650,15 @@ mod tests { let length_at = at + 6 + refers + 1; let length = u32::from_be_bytes(data[length_at..length_at + 4].try_into().unwrap()) as usize; let referred = data[at + 6..at + 6 + refers].iter().enumerate().map(|(k, n)| (*n, flags >> (k + 1) & 1 == 1)).collect(); - segments.push((data[at + 4] & 0x3F, flags & 1 == 1, referred)); + let number = u32::from_be_bytes(data[at..at + 4].try_into().unwrap()); + segments.push((number, data[at + 4] & 0x3F, flags & 1 == 1, referred)); at = length_at + 4 + length; } segments } fn segment_types(data: &[u8]) -> Vec { - segments(data).into_iter().map(|(kind, ..)| kind).collect() + segments(data).into_iter().map(|(_, kind, ..)| kind).collect() } /// Every plan, at every strip size, decodes to the picture; returns how many plans there were. @@ -796,32 +806,60 @@ mod tests { assert!(page(&tall, 1, MAX_ROWS).is_some()); } - /// Past 32,767 distinct symbols, or 2^24 pixels of them, the image is left to generic coding; - /// at either cap it is symbol coded, and decodes. + /// A plan codes at most 32,767 symbols, which an independent decoder reads back; with more, + /// the second plan can still code the repeated ones. Past 65,536 distinct marks, or 2^24 + /// pixels of them, the image is left to generic coding; at either cap it is still looked at. #[test] fn too_many_symbols_are_left_to_generic_coding() { - // `n` distinct marks of 17 × 3: full top and bottom rows joined by a code of 1 to n, in - // binary, on the row between. - let codes = |n: usize| { - let (columns, rows) = (256 * 18, n.div_ceil(256) * 4); + // `n` distinct marks of 17 × 3 (full top and bottom rows joined by a code of 1 to n, in + // binary, on the row between), then `again` more with the codes 1 and 2 in turn. + let codes = |n: usize, again: usize| { + let (columns, rows) = (256 * 18, (n + again).div_ceil(256) * 4); let mut picture = vec![vec![0u8; columns]; rows]; - for k in 0..n { + for k in 0..n + again { let (x, y) = (k % 256 * 18, k / 256 * 4); + let code = if k < n { k + 1 } else { (k - n) % 2 + 1 }; picture[y][x..x + 17].fill(1); picture[y + 2][x..x + 17].fill(1); - for b in (0..17).filter(|b| (k + 1) >> b & 1 == 1) { + for b in (0..17).filter(|b| code >> b & 1 == 1) { picture[y + 1][x + b] = 1; } } (samples(&picture), columns as u32, rows as u32) }; - // At the cap the symbols are coded, and an independent decoder reads them back. - let (data, columns, rows) = codes(MAX_SYMBOLS); + let (data, columns, rows) = codes(MAX_CODED_SYMBOLS, 0); let at_cap = page(&data, columns, rows).unwrap(); - assert_eq!(at_cap.symbols.len(), MAX_SYMBOLS); - let coded = plan(&at_cap, columns, rows, 1, None).unwrap(); + assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows)); + // One more distinct mark, each once: neither plan is made. + let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 0); + let past = page(&data, columns, rows).unwrap(); + assert!(plan(&past, columns, rows, 1, None).is_none() && plan(&past, columns, rows, 2, None).is_none()); + assert_eq!(encode_jbig2(&data, columns, rows).unwrap(), encode_jbig2_generic(&data, columns, rows).unwrap()); + // With two of them repeated, the second plan codes those two. + let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 200); + let past = page(&data, columns, rows).unwrap(); + assert!(plan(&past, columns, rows, 1, None).is_none()); + assert_eq!(decode(&plan(&past, columns, rows, 2, None).unwrap()), unpack(&data, columns, rows)); + // Above a page of text whose letters repeat, that second plan is the one `encode_jbig2` + // keeps: the first, of every mark, would code too many symbols. + let mut picture: Bitmap = vec![vec![0u8; 256 * 18]; 512]; + for k in 0..=MAX_CODED_SYMBOLS { + let (x, y) = (k % 256 * 18, k / 256 * 4); + picture[y][x..x + 17].fill(1); + picture[y + 2][x..x + 17].fill(1); + for b in (0..17).filter(|b| (k + 1) >> b & 1 == 1) { + picture[y + 1][x + b] = 1; + } + } + picture.extend(text(256 * 18, 600, &glyphs(40, 7), 11)); + let data = samples(&picture); + let (columns, rows) = (picture[0].len() as u32, picture.len() as u32); + let coded = encode_jbig2(&data, columns, rows).unwrap(); + assert_eq!(segment_types(&coded)[1], 0, "symbol coded"); assert_eq!(decode(&coded), unpack(&data, columns, rows)); - let (data, columns, rows) = codes(MAX_SYMBOLS + 1); + let (data, columns, rows) = codes(MAX_SYMBOLS, 0); + assert_eq!(page(&data, columns, rows).map(|p| p.symbols.len()), Some(MAX_SYMBOLS)); + let (data, columns, rows) = codes(MAX_SYMBOLS + 1, 0); assert!(page(&data, columns, rows).is_none()); // `n` black squares of 256 × 256, each with a white pixel in a different place: 2^16 // pixels a symbol, so 256 of them are exactly the cap. @@ -847,22 +885,30 @@ mod tests { /// More than 10,000 marks are placed by several text regions, each of at most 10,000 (more /// than the decoder PdfCraft renders with accepts in one), over bands of rows, all referring to - /// the dictionary, which each region but the last keeps retained. + /// the dictionary, which each region but the last keeps retained; the generic region comes + /// after them, every segment numbered one more than the one before. #[test] fn more_than_10000_marks_are_placed_by_several_regions() { - // 25,000 copies of two 17 × 3 shapes, 256 a line, the lines 5 rows apart. - let mut picture = vec![vec![0u8; 256 * 18]; 25_000usize.div_ceil(256) * 5]; + // 25,000 copies of two 17 × 3 shapes, 256 a line, the lines 5 rows apart, and a bar too + // tall to be a symbol beside them. + let mut picture = vec![vec![0u8; 256 * 18 + 100]; 25_000usize.div_ceil(256) * 5]; for k in 0..25_000 { let (x, y) = (k % 256 * 18, k / 256 * 5); picture[y][x..x + 17].fill(1); picture[y + 2][x..x + 17].fill(1); picture[y + 1][x + k % 2] = 1; } + for row in &mut picture[0..300] { + row[256 * 18 + 50] = 1; + } let data = samples(&picture); let (columns, rows) = (picture[0].len() as u32, picture.len() as u32); let coded = encode_jbig2(&data, columns, rows).unwrap(); - let region = |kept| (7, false, vec![(1, kept)]); - assert_eq!(segments(&coded), [(48, false, vec![]), (0, true, vec![]), region(true), region(true), region(false)]); + let region = |number, kept| (number, 7, false, vec![(1, kept)]); + assert_eq!( + segments(&coded), + [(0, 48, false, vec![]), (1, 0, true, vec![]), region(2, true), region(3, true), region(4, false), (5, 39, false, vec![])] + ); assert_eq!(decode(&coded), unpack(&data, columns, rows)); assert_eq!(every_plan_round_trips(&picture), 8); } @@ -876,7 +922,7 @@ mod tests { let coded = encode_jbig2(&data, 1700, 400).unwrap(); let generic = encode_jbig2_generic(&data, 1700, 400).unwrap(); // The dictionary is retained for the text region, which refers to it (7.2.4, 7.2.5). - assert_eq!(segments(&coded), [(48, false, vec![]), (0, true, vec![]), (7, false, vec![(1, false)])]); + assert_eq!(segments(&coded), [(0, 48, false, vec![]), (1, 0, true, vec![]), (2, 7, false, vec![(1, false)])]); assert_eq!(decode(&coded), unpack(&data, 1700, 400)); assert!(coded.len() * 2 < generic.len(), "symbols {} bytes, generic {}", coded.len(), generic.len()); assert_eq!(every_plan_round_trips(&picture), 8); diff --git a/crates/optimize/tests/optimize.rs b/crates/optimize/tests/optimize.rs index a349d8cb1b..f8e28506a1 100644 --- a/crates/optimize/tests/optimize.rs +++ b/crates/optimize/tests/optimize.rs @@ -749,6 +749,17 @@ fn a_dense_page_of_rendered_text_looks_the_same_as_jbig2_symbols() { let s = stream(&doc, im); assert_eq!(s.dict.name(b"Filter"), Some(&b"JBIG2Decode"[..])); assert!(s.raw.len() * 2 < pdfcraft_filters::encode_jbig2_generic(&text, w, h).unwrap().len(), "symbol coded"); + // The copies take several text regions (segment type 7, ISO/IEC 14492 7.3), read from the + // segment headers (7.2): number, flags, referred-to count and numbers, page, data length. + let mut regions = 0; + let mut at = 0; + while let Some(header) = s.raw.get(at..at + 6) { + let refers = usize::from(header[5] >> 5); + let length = u32::from_be_bytes(s.raw[at + 7 + refers..at + 11 + refers].try_into().unwrap()) as usize; + regions += usize::from(header[4] & 0x3F == 7); + at += 11 + refers + length; + } + assert!(regions > 1, "{regions} text regions"); assert!(render(&doc) == before, "the page looks the same"); }