M11: JBIG2 symbol coding: at most 10,000 copies a text region and 32,767 symbols

A review found that a dense page came out as a stream PdfCraft's own
renderer refuses, so the page drew blank after optimizing: hayro-jbig2
decodes a text region of at most 10,000 symbol instances, and refuses
32,768 symbols or more. The stream was valid (Poppler decoded it), but
every mark went into one text region and up to 65,536 symbols were kept.
A 300 dpi letter page of the optimizer test's own text crossed it.

A page with more than 10,000 marks is now placed by several text
regions, each over a band of rows and all referring to the one symbol
dictionary, which each region but the last keeps retained (7.2.4). Past
32,767 distinct symbols the image is coded generically.

New tests: 25,000 copies placed by three regions, which decode; the
symbol caps decoded, not just reached; and in the optimizer, a whole
letter page of rendered text that looks the same as JBIG2 symbols (it
drew blank before). The READMEs say "still looked at" for the caps, and
that the scans figure is the six pages' total (20% on one page).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG

crates/filters/README.md

@@ -44,7 +44,7 @@ encode_jbig2_generic(data, columns, rows) -> Result<Vec<u8>, FilterError>   // t
 | ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` |
 | RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 |
 | CCITTFaxDecode | not here (`Unsupported`; renderers decode it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt |
-| JBIG2Decode | not here (`Unsupported`; the renderer and image export decode it with `hayro-jbig2`) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by an immediate lossless text region referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 symbols or 16 MiB of symbol pixels, or more than 2^24 rows, and the generic coding is used |
+| JBIG2Decode | not here (`Unsupported`; the renderer and image export decode it with `hayro-jbig2`) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 32,767 symbols (that decoder refuses more) or 16 MiB of symbol pixels, or more than 2^24 rows, and the generic coding is used |
 
 ## Tests
 
@@ -54,7 +54,7 @@ encode_jbig2_generic(data, columns, rows) -> Result<Vec<u8>, FilterError>   // t
 - `ccitt.rs`: Group 4 output decoded by `hayro-ccitt` (dev-dependency, an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics.
 - `mq.rs`: the arithmetic coder reproduces Annex H.2's 30 bytes from its 256 decisions.
 - `jbig2.rs`: output decoded by `hayro-jbig2` (dev-dependency, an independent decoder) matches the input bits: a text-like page (smaller than Group 4), edges, padding, all-white and all-black regions (typical rows), and property tests over bitmaps of noise and of runs; any input returns rather than panics.
-- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 65,536 symbols or 2^24 symbol pixels the image is coded generically, and at each cap it is not; any input returns rather than panics.
+- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 32,767 symbols or 2^24 symbol pixels the image is coded generically, at each cap it is still looked at, and at the two symbol caps it decodes; more than 10,000 marks are placed by several text regions, the dictionary retained for each but the last; any input returns rather than panics.
 - `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`).
 
 ## Status

crates/filters/src/jbig2.rs

@@ -139,16 +139,18 @@ fn at_pixels(out: &mut Vec<u8>) {
 }
 
 /// A segment (7.2): its header (number, type, the segments it refers to, page 1, data length),
-/// then its data. `retained`: a later segment refers to this one. Segment numbers here are below
-/// 256, so each referred-to one takes a byte (7.2.5); a segment refers to at most four, so their
-/// count and retain bits take one byte too (7.2.4), and the retain bits of the referred-to ones
-/// are 0: nothing after this segment refers to them.
-fn segment(out: &mut Vec<u8>, number: u32, kind: u8, refers_to: &[u8], retained: bool, data: &[u8]) {
+/// then its data. `retained`: a later segment refers to this one. `refers_to` holds each
+/// referred-to segment's number and whether a segment after this one refers to it too (its
+/// retain bit, 7.2.4). Segment numbers here stay below 256, so each referred-to one takes a byte
+/// (7.2.5); a segment refers to at most four, so their count and retain bits take one byte too.
+fn segment(out: &mut Vec<u8>, number: u32, kind: u8, refers_to: &[(u8, bool)], retained: bool, data: &[u8]) {
     out.extend_from_slice(&number.to_be_bytes());
     out.push(kind);
-    let count = refers_to.len().min(4);
-    out.push((count as u8) << 5 | u8::from(retained));
-    out.extend_from_slice(refers_to.get(..count).unwrap_or_default());
+    // At most four, so the count fits its three bits.
+    let refers_to = refers_to.get(..refers_to.len().min(4)).unwrap_or_default();
+    let kept = refers_to.iter().enumerate().fold(0u8, |bits, (k, (_, kept))| bits | u8::from(*kept) << (k + 1));
+    out.push((refers_to.len() as u8) << 5 | kept | u8::from(retained));
+    out.extend(refers_to.iter().map(|(n, _)| *n));
     out.push(1);
     // A segment's data never comes near 4 GiB here: a row is at most 1 MiB wide.
     out.extend_from_slice(&u32::try_from(data.len()).unwrap_or(u32::MAX).to_be_bytes());

crates/filters/src/jbig2/symbols.rs

@@ -1,8 +1,8 @@
 //! JBIG2 symbol coding, lossless: each distinct mark of an image (a group of black pixels that
-//! touch, side by side or at a corner) is stored once, as a symbol in a symbol dictionary, and a
-//! text region places a copy of it wherever it occurs. Marks too large to be symbols (rules,
-//! frames, pictures) are left to a generic region beside it, and so, in [`encode`]'s second plan,
-//! are marks that occur only once. Only exact copies share a symbol, so the image decodes to the
+//! touch, side by side or at a corner) is stored once, as a symbol in a symbol dictionary, and
+//! text regions place a copy of it wherever it occurs. Marks too large to be symbols (rules,
+//! frames, pictures) are left to a generic region beside them, and so, in [`encode`]'s second
+//! plan, are marks that occur only once. Only exact copies share a symbol, so the image decodes to the
 //! same bits. Written from the committee draft's 6.4 (the text region decoding procedure), 6.5
 //! (the symbol dictionary decoding procedure), 7.4.2 and 7.4.3 (their segments) and Annex A (the
 //! arithmetic integer coders).
@@ -21,10 +21,18 @@ const MAX_SYMBOL_SIDE: u32 = 256;
 /// which suits it better anyway; this bounds the memory the marks take.
 const MAX_RUNS: usize = 1 << 20;
 
-/// The most distinct symbols, and the most pixels their bitmaps may hold together.
-const MAX_SYMBOLS: usize = 1 << 16;
+/// The most distinct symbols, and the most pixels their bitmaps may hold together. The decoder
+/// PdfCraft renders JBIG2 with (`hayro-jbig2`) refuses 32,768 symbols or more, and others may
+/// too: past it the image is coded generically.
+const MAX_SYMBOLS: usize = (1 << 15) - 1;
 const MAX_SYMBOL_PIXELS: usize = 1 << 24;
 
+/// The most symbol instances one text region places. The decoder PdfCraft renders JBIG2 with
+/// refuses a text region of more than 10,000 (a page it then draws blank), so a page with more is
+/// placed by several regions, each over a band of rows, all referring to the one dictionary. At
+/// most [`MAX_RUNS`] marks keeps their segment numbers below 256.
+const MAX_REGION_INSTANCES: usize = 10_000;
+
 /// The most rows symbol coded. With rows at most 2^20 samples wide, every coordinate and
 /// difference coded then fits the 32-bit integers Annex A codes.
 const MAX_ROWS: u32 = 1 << 24;
@@ -73,14 +81,15 @@ struct Page {
     large: Vec<u32>,
 }
 
-/// A symbol instance (6.4.5): the symbol, and where its bottom-left pixel goes, as S (the column)
-/// and T (the row). `width` is the symbol's.
+/// A symbol instance (6.4.5): the symbol, and where its bottom-left pixel goes on the page, as S
+/// (the column) and T (the row). `width` and `height` are the symbol's.
 #[derive(Clone, Copy, Debug)]
 struct Instance {
     id: u32,
     s: u32,
     t: u32,
     width: u32,
+    height: u32,
 }
 
 /// `rows` rows of `columns` 1-bit PDF samples (black 0), symbol coded: the smaller of two plans,
@@ -283,8 +292,8 @@ fn symbols(page: &mut Page) -> Option<()> {
     Some(())
 }
 
-/// One plan's stream: the symbols with at least `copies` uses in a symbol dictionary, a text
-/// region placing them (in strips of 2^`log_strips` rows, or whichever size codes smallest), and
+/// One plan's stream: the symbols with at least `copies` uses in a symbol dictionary, text
+/// regions placing them (in strips of 2^`log_strips` rows, or whichever size codes smallest), and
 /// the other marks in a generic region. `None` with fewer than two such symbols: their IDs would
 /// take no bits at all (SBSYMCODELEN 0), a corner of the standard left alone here.
 fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option<u32>) -> Option<Vec<u8>> {
@@ -305,23 +314,37 @@ fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option<u3
     let mut left = page.large.clone();
     for (s, m) in &page.copies {
         match (id.get(*s as usize).copied().flatten(), page.marks.get(*m as usize), symbol(*s)) {
-            (Some(id), Some(mark), Some(symbol)) => instances.push(Instance { id, s: mark.x0, t: mark.y1.saturating_sub(1), width: symbol.width }),
+            (Some(id), Some(mark), Some(symbol)) => {
+                instances.push(Instance { id, s: mark.x0, t: mark.y1.saturating_sub(1), width: symbol.width, height: symbol.height })
+            }
             _ => left.push(*m),
         }
     }
     let kept: Vec<&Symbol> = kept.iter().filter_map(|s| symbol(*s)).collect();
     let dictionary = symbol_dictionary(&kept)?;
     let count = u32::try_from(kept.len()).ok()?;
-    let text = match log_strips {
-        Some(l) => text_region(&mut instances, count, columns, rows, l)?,
-        None => (0..4).filter_map(|l| text_region(&mut instances, count, columns, rows, l)).min_by_key(Vec::len)?,
-    };
+    // Bands of at most MAX_REGION_INSTANCES, from the top.
+    instances.sort_by_key(|i| (i.t, i.s));
+    let mut regions = Vec::new();
+    for band in instances.chunks_mut(MAX_REGION_INSTANCES) {
+        let top = band.iter().map(|i| i.t.saturating_add(1).saturating_sub(i.height)).min()?;
+        let height = band.iter().map(|i| i.t).max()?.checked_sub(top)?.checked_add(1)?;
+        regions.push(match log_strips {
+            Some(l) => text_region(band, count, columns, top, height, l)?,
+            None => (0..4).filter_map(|l| text_region(band, count, columns, top, height, l)).min_by_key(Vec::len)?,
+        });
+    }
     let mut out = Vec::new();
     segment(&mut out, 0, 48, &[], false, &page_information(columns, rows));
     segment(&mut out, 1, 0, &[], true, &dictionary);
-    segment(&mut out, 2, 7, &[1], false, &text);
+    let mut number = 2;
+    for (k, text) in regions.iter().enumerate() {
+        // The dictionary stays retained for the next region, until the last (7.2.4).
+        segment(&mut out, number, 7, &[(1, k + 1 < regions.len())], false, text);
+        number += 1;
+    }
     if !left.is_empty() {
-        segment(&mut out, 3, 39, &[], false, &leftover_region(&left, page)?);
+        segment(&mut out, number, 39, &[], false, &leftover_region(&left, page)?);
     }
     Some(out)
 }
@@ -363,20 +386,22 @@ fn symbol_dictionary(symbols: &[&Symbol]) -> Option<Vec<u8>> {
     Some(data)
 }
 
-/// An immediate lossless text region segment's data (7.4.3) placing `instances` over the whole
-/// page: arithmetic coded, no refinement, strips of 2^`log_strips` rows, each symbol's
-/// bottom-left pixel at (S, T) (REFCORNER BOTTOMLEFT, so a line's letters share their T),
-/// not transposed, OR, default pixel 0, SBDSOFFSET 0. Sorts `instances` into strips, each
-/// left to right.
-fn text_region(instances: &mut [Instance], symbols: u32, columns: u32, rows: u32, log_strips: u32) -> Option<Vec<u8>> {
+/// An immediate lossless text region segment's data (7.4.3) placing `instances` in the band of
+/// `height` rows from row `top` (which holds them): arithmetic coded, no refinement, strips of
+/// 2^`log_strips` rows, each symbol's bottom-left pixel at (S, T) (REFCORNER BOTTOMLEFT, so a
+/// line's letters share their T), not transposed, OR, default pixel 0, SBDSOFFSET 0. Sorts
+/// `instances` into strips, each left to right.
+fn text_region(instances: &mut [Instance], symbols: u32, columns: u32, top: u32, height: u32, log_strips: u32) -> Option<Vec<u8>> {
     // LOGSBSTRIPS has two bits: strips of 1, 2, 4 or 8 rows.
     if log_strips > 3 {
         return None;
     }
     let strips = 1i64 << log_strips;
-    instances.sort_by_key(|i| (i.t >> log_strips, i.s));
+    // T counts from the band's top row.
+    let t = |i: &Instance| i.t.saturating_sub(top);
+    instances.sort_by_key(|i| (t(i) >> log_strips, i.s));
     let mut data = Vec::new();
-    region_information(&mut data, columns, rows, 0, 0);
+    region_information(&mut data, columns, height, 0, top);
     data.extend_from_slice(&((log_strips as u16) << 2).to_be_bytes());
     data.extend_from_slice(&u32::try_from(instances.len()).ok()?.to_be_bytes());
     // SBSYMCODELEN (Table 31): ⌈log2 SBNUMSYMS⌉.
@@ -388,10 +413,10 @@ fn text_region(instances: &mut [Instance], symbols: u32, columns: u32, rows: u32
     // strip size, from the strip before.
     iadt.encode(&mut mq, Some(0));
     let (mut strip_t, mut first_s) = (0i64, 0i64);
-    for strip in instances.chunk_by(|a, b| a.t >> log_strips == b.t >> log_strips) {
-        let t = i64::from(strip.first()?.t >> log_strips << log_strips);
-        iadt.encode(&mut mq, Some(int((t - strip_t) / strips)?));
-        strip_t = t;
+    for strip in instances.chunk_by(|a, b| t(a) >> log_strips == t(b) >> log_strips) {
+        let first = i64::from(t(strip.first()?) >> log_strips << log_strips);
+        iadt.encode(&mut mq, Some(int((first - strip_t) / strips)?));
+        strip_t = first;
         let mut cur_s = 0;
         for (k, instance) in strip.iter().enumerate() {
             let s = i64::from(instance.s);
@@ -402,7 +427,7 @@ fn text_region(instances: &mut [Instance], symbols: u32, columns: u32, rows: u32
                 iads.encode(&mut mq, Some(int(s - cur_s)?));
             }
             if strips > 1 {
-                iait.encode(&mut mq, Some(int(i64::from(instance.t) - strip_t)?));
+                iait.encode(&mut mq, Some(int(i64::from(t(instance)) - strip_t)?));
             }
             encode_id(&mut mq, &mut ids, length, instance.id);
             // The next S counts from this symbol's last column (6.4.5 step 3c x).
@@ -603,15 +628,20 @@ mod tests {
         page
     }
 
-    /// The segments of a stream in the embedded organisation, from their headers (7.2): each one's
-    /// type, whether it is retained, and the segments it refers to.
-    fn segments(data: &[u8]) -> Vec<(u8, bool, Vec<u8>)> {
+    /// A segment's type, whether it is retained, and the segments it refers to, each with its
+    /// retain bit.
+    type Segment = (u8, bool, Vec<(u8, bool)>);
+
+    /// The segments of a stream in the embedded organisation, from their headers (7.2).
+    fn segments(data: &[u8]) -> Vec<Segment> {
         let (mut segments, mut at) = (Vec::new(), 0);
         while at < data.len() {
-            let refers = usize::from(data[at + 5] >> 5);
+            let flags = data[at + 5];
+            let refers = usize::from(flags >> 5);
             let length_at = at + 6 + refers + 1;
             let length = u32::from_be_bytes(data[length_at..length_at + 4].try_into().unwrap()) as usize;
-            segments.push((data[at + 4] & 0x3F, data[at + 5] & 1 == 1, data[at + 6..at + 6 + refers].to_vec()));
+            let referred = data[at + 6..at + 6 + refers].iter().enumerate().map(|(k, n)| (*n, flags >> (k + 1) & 1 == 1)).collect();
+            segments.push((data[at + 4] & 0x3F, flags & 1 == 1, referred));
             at = length_at + 4 + length;
         }
         segments
@@ -752,7 +782,7 @@ mod tests {
         assert_eq!(every_plan_round_trips(&picture), 8);
     }
 
-    /// Past 2^20 black runs or 2^24 rows the image is left to generic coding; at 2^20 runs it is
+    /// Past 2^20 black runs or 2^24 rows the image is left to generic coding; at either cap it is
     /// still looked at.
     #[test]
     fn busy_or_tall_images_are_left_to_generic_coding() {
@@ -766,8 +796,8 @@ mod tests {
         assert!(page(&tall, 1, MAX_ROWS).is_some());
     }
 
-    /// Past 65,536 distinct symbols, or 2^24 pixels of them, the image is left to generic coding;
-    /// at either cap it is still looked at.
+    /// Past 32,767 distinct symbols, or 2^24 pixels of them, the image is left to generic coding;
+    /// at either cap it is symbol coded, and decodes.
     #[test]
     fn too_many_symbols_are_left_to_generic_coding() {
         // `n` distinct marks of 17 × 3: full top and bottom rows joined by a code of 1 to n, in
@@ -785,8 +815,12 @@ mod tests {
             }
             (samples(&picture), columns as u32, rows as u32)
         };
+        // At the cap the symbols are coded, and an independent decoder reads them back.
         let (data, columns, rows) = codes(MAX_SYMBOLS);
-        assert_eq!(page(&data, columns, rows).map(|p| p.symbols.len()), Some(MAX_SYMBOLS));
+        let at_cap = page(&data, columns, rows).unwrap();
+        assert_eq!(at_cap.symbols.len(), MAX_SYMBOLS);
+        let coded = plan(&at_cap, columns, rows, 1, None).unwrap();
+        assert_eq!(decode(&coded), unpack(&data, columns, rows));
         let (data, columns, rows) = codes(MAX_SYMBOLS + 1);
         assert!(page(&data, columns, rows).is_none());
         // `n` black squares of 256 × 256, each with a white pixel in a different place: 2^16
@@ -804,11 +838,35 @@ mod tests {
             (samples(&picture), columns as u32, rows as u32)
         };
         let (data, columns, rows) = squares(256);
-        assert_eq!(page(&data, columns, rows).map(|p| p.symbols.len()), Some(256));
+        let at_cap = page(&data, columns, rows).unwrap();
+        assert_eq!(at_cap.symbols.len(), 256);
+        assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows));
         let (data, columns, rows) = squares(257);
         assert!(page(&data, columns, rows).is_none());
     }
 
+    /// More than 10,000 marks are placed by several text regions, each of at most 10,000 (more
+    /// than the decoder PdfCraft renders with accepts in one), over bands of rows, all referring to
+    /// the dictionary, which each region but the last keeps retained.
+    #[test]
+    fn more_than_10000_marks_are_placed_by_several_regions() {
+        // 25,000 copies of two 17 × 3 shapes, 256 a line, the lines 5 rows apart.
+        let mut picture = vec![vec![0u8; 256 * 18]; 25_000usize.div_ceil(256) * 5];
+        for k in 0..25_000 {
+            let (x, y) = (k % 256 * 18, k / 256 * 5);
+            picture[y][x..x + 17].fill(1);
+            picture[y + 2][x..x + 17].fill(1);
+            picture[y + 1][x + k % 2] = 1;
+        }
+        let data = samples(&picture);
+        let (columns, rows) = (picture[0].len() as u32, picture.len() as u32);
+        let coded = encode_jbig2(&data, columns, rows).unwrap();
+        let region = |kept| (7, false, vec![(1, kept)]);
+        assert_eq!(segments(&coded), [(48, false, vec![]), (0, true, vec![]), region(true), region(true), region(false)]);
+        assert_eq!(decode(&coded), unpack(&data, columns, rows));
+        assert_eq!(every_plan_round_trips(&picture), 8);
+    }
+
     /// A page of text whose letters repeat exactly, as rendered text does, keeps each letter once:
     /// a symbol dictionary and a text region, much smaller than the generic coding.
     #[test]
@@ -818,7 +876,7 @@ mod tests {
         let coded = encode_jbig2(&data, 1700, 400).unwrap();
         let generic = encode_jbig2_generic(&data, 1700, 400).unwrap();
         // The dictionary is retained for the text region, which refers to it (7.2.4, 7.2.5).
-        assert_eq!(segments(&coded), [(48, false, vec![]), (0, true, vec![]), (7, false, vec![1])]);
+        assert_eq!(segments(&coded), [(48, false, vec![]), (0, true, vec![]), (7, false, vec![(1, false)])]);
         assert_eq!(decode(&coded), unpack(&data, 1700, 400));
         assert!(coded.len() * 2 < generic.len(), "symbols {} bytes, generic {}", coded.len(), generic.len());
         assert_eq!(every_plan_round_trips(&picture), 8);

crates/optimize/README.md

@@ -42,8 +42,8 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
   symbol, when the image's marks repeat exactly, as rendered text's do: on six pages of
   300 dpi text that came to 28% of a generic region's size and 17% of Group 4's. On scans,
   whose copies of a letter are seldom identical, symbols gain little: on the same pages with 2%
-  or more of the letters' edge pixels flipped at random, at most 8% below a generic region,
-  which was 27–33% below Group 4. It is lossless: the
+  or more of the letters' edge pixels flipped at random, at most 8% below a generic region in
+  total (20% on one page), and the generic region was 27–33% below Group 4. It is lossless: the
   samples decode to the same bits, so the colour space, decode array and masks stay. Group 4
   wins on scans, whose ragged edges defeat Flate; on clean computer-drawn 1-bit art Flate can
   be smaller, and the image then stays. Not yet:

crates/optimize/tests/optimize.rs

@@ -734,6 +734,24 @@ fn rendered_text_becomes_jbig2_symbols() {
     assert!(render(&doc) == before, "the page looks the same");
 }
 
+/// A whole 300 dpi letter page of rendered text, more marks than one JBIG2 text region of the
+/// decoder PdfCraft renders with takes, still looks the same as JBIG2 symbols.
+#[test]
+fn a_dense_page_of_rendered_text_looks_the_same_as_jbig2_symbols() {
+    let mut doc = Document::new_empty();
+    let (w, h) = (2550, 3300);
+    let text = rendered_text(w, h);
+    let im = mono_image(&mut doc, w, h, &text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
+    page(&mut doc, &[("Im", im)], "q 612 0 0 792 0 0 cm /Im Do Q");
+    let before = render(&doc);
+    let r = optimize(&mut doc, &Settings { mono: MonoCompression::Jbig2, ..Settings::default() }).unwrap();
+    assert_eq!(r.images_recompressed, 1);
+    let s = stream(&doc, im);
+    assert_eq!(s.dict.name(b"Filter"), Some(&b"JBIG2Decode"[..]));
+    assert!(s.raw.len() * 2 < pdfcraft_filters::encode_jbig2_generic(&text, w, h).unwrap().len(), "symbol coded");
+    assert!(render(&doc) == before, "the page looks the same");
+}
+
 #[test]
 fn settings_choose_what_happens() {
     let mut doc = Document::new_empty();

parity/acrobat-features.toml

@@ -7385,7 +7385,7 @@ status = "partial"
 commands = ["optimize.advanced", "optimize.reduce"]
 tools = ["doc_optimize", "doc_reduce"]
 evidence = ["crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy"]
-notes = "CCITT Group 4 (the default, so Reduce File Size does it too) or JBIG2, both lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. JBIG2 stores each distinct mark once in a symbol dictionary and places its exact copies with a text region, larger and lone marks in a generic region (template 0, typical prediction), or the whole image as a generic region, whichever is smaller: on rendered 300 dpi text 28% of the generic region's size and 17% of Group 4's; on scans, whose copies of a letter are seldom identical, little: on the same pages with 2% or more of the letters' edge pixels flipped at random, at most 8% below the generic region, which was 27–33% below Group 4. A document below PDF 1.4 gets /Version 1.4. Not yet: JBIG2 refinement coding of near-copies (where lossless gains on scans would come from), a symbol dictionary shared between images (/JBIG2Globals), JBIG2's lossy mode, monochrome downsampling, inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples (JBIG2 takes rows of up to 1,048,576). Redaction doesn't decode JBIG2 or Group 4 yet: a mark over part of such an image removes the whole image (nothing leaks). Export reads JBIG2, but nothing that exports an image reads Group 4 yet: Export ▸ Images and Save Image As refuse it, Copy Image fails, and Word, PowerPoint and HTML export leave the picture out."
+notes = "CCITT Group 4 (the default, so Reduce File Size does it too) or JBIG2, both lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. JBIG2 stores each distinct mark once in a symbol dictionary and places its exact copies with text regions (at most 10,000 copies each), larger and lone marks in a generic region (template 0, typical prediction), or the whole image as a generic region, whichever is smaller: on rendered 300 dpi text 28% of the generic region's size and 17% of Group 4's; on scans, whose copies of a letter are seldom identical, little: on the same pages with 2% or more of the letters' edge pixels flipped at random, at most 8% below the generic region in total (20% on one page), and the generic region was 27–33% below Group 4. A document below PDF 1.4 gets /Version 1.4. Not yet: JBIG2 refinement coding of near-copies (where lossless gains on scans would come from), a symbol dictionary shared between images (/JBIG2Globals), JBIG2's lossy mode, monochrome downsampling, inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples (JBIG2 takes rows of up to 1,048,576). Redaction doesn't decode JBIG2 or Group 4 yet: a mark over part of such an image removes the whole image (nothing leaks). Export reads JBIG2, but nothing that exports an image reads Group 4 yet: Export ▸ Images and Save Image As refuse it, Copy Image fails, and Word, PowerPoint and HTML export leave the picture out."
 
 [[feature]]
 id = "optimize.only-if-smaller"