diff --git a/crates/filters/README.md b/crates/filters/README.md index 90e1485177..cdaef6ff95 100644 --- a/crates/filters/README.md +++ b/crates/filters/README.md @@ -27,10 +27,10 @@ encode(filter, params, data) -> Result, FilterError> encode_flate(data) -> Vec // zlib, level 6 encode_g4(data, columns, rows) -> Result, FilterError> // 1-bit rows as CCITT Group 4; columns <= MAX_G4_COLUMNS pub struct CcittParams { k, columns, rows, end_of_line, encoded_byte_align, end_of_block, black_is_1 } // Table 11; Default = its defaults -decode_ccitt(data, ¶ms, max_rows) -> Result, FilterError> // CCITT fax as CCITTFaxDecode gives it; columns <= MAX_CCITT_COLUMNS -decode_jbig2(data, globals, columns, rows, max) -> Result, FilterError> // JBIG2 as JBIG2Decode gives it; a page of columns × rows, within max and MAX_JBIG2_PIXELS -encode_jbig2_generic(data, columns, rows) -> Result, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode; columns <= MAX_JBIG2_COLUMNS -encode_jbig2(data, columns, rows) -> Result, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode, symbol coded when that is smaller; columns <= MAX_JBIG2_COLUMNS +decode_ccitt_within(data, ¶ms, max_rows, work: &mut u64) -> Result, FilterError> // CCITT fax as CCITTFaxDecode gives it; columns <= MAX_CCITT_COLUMNS; its rows paid for from work before they are decoded +decode_jbig2_within(data, globals, columns, rows, max, work: &mut u64) -> Result, FilterError> // JBIG2 as JBIG2Decode gives it; a page of columns × rows, within max and MAX_JBIG2_PIXELS; reading and decoding it spend from work, in the decoder's steps +encode_jbig2_generic(data, columns, rows) -> Result, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode; columns and rows <= MAX_JBIG2_COLUMNS (65,535) and MAX_JBIG2_PIXELS (2^28) samples, what the decoder takes +encode_jbig2(data, columns, rows) -> Result, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode, symbol coded when that is smaller; the same limits ``` - `chain` is applied first to last, as in a `/Filter` array. @@ -50,8 +50,8 @@ encode_jbig2(data, columns, rows) -> Result, FilterError> // 1-bit row | ASCIIHexDecode | white space ignored, odd digit padded with 0, `>` EOD (optional), other characters → Corrupt | upper case, newline every 64 digits, `>` | | ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` | | RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 | -| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::Document::decoded_image` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt | -| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, about the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::Document::decoded_image` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, or more than 2^24 rows, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made | +| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt_within` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::ImageBudget::decode` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt | +| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2_within` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::ImageBudget::decode` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 65,535 samples on a side and 2^28 in all, where `hayro-jbig2` stops; no rows or columns, or too large → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made | ## Tests @@ -61,8 +61,8 @@ encode_jbig2(data, columns, rows) -> Result, FilterError> // 1-bit row - `ccitt.rs`: `decode_ccitt` gives back the rows `encode_g4` was given (with `/BlackIs1`, `/Rows` and a row limit), refuses bad widths, and bounds its output for any input; byte-aligned Group 3 in each layout, over 32 seeded pictures, decodes to its picture, and a byte short to fewer rows. Group 4 output decoded by `hayro-ccitt` (an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics. - `mq.rs`: the arithmetic coder reproduces Annex H.2's 30 bytes from its 256 decisions. - `jbig2.rs`: output decoded by `hayro-jbig2` (dev-dependency, an independent decoder) matches the input bits: a text-like page (smaller than Group 4), edges, padding, all-white and all-black regions (typical rows), and property tests over bitmaps of noise and of runs; any input returns rather than panics. -- `jbig2/decode.rs`: `decode_jbig2` gives back the rows of both codings, padding bits 0; refuses a page of another width or height, past `max` or past `MAX_JBIG2_PIXELS` before decoding it (2^28 pixels take about 34 MB, a bit each), globals that aren't JBIG2 and damaged data, and returns rather than panics on every prefix of a stream. -- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics. +- `jbig2/decode.rs`: `decode_jbig2_within` gives back the rows of both codings, padding bits 0; refuses a page of another width or height, past `max` or past `MAX_JBIG2_PIXELS` before decoding it (2^28 pixels take about 34 MB, a bit each), globals that aren't JBIG2 and damaged data, and returns rather than panics on every prefix of a stream. +- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics. - `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`). ## Status diff --git a/crates/filters/src/jbig2.rs b/crates/filters/src/jbig2.rs index 1b705b0c72..ffcfe45e77 100644 --- a/crates/filters/src/jbig2.rs +++ b/crates/filters/src/jbig2.rs @@ -12,9 +12,11 @@ pub use decode::{MAX_JBIG2_PIXELS, decode_jbig2_within}; use crate::FilterError; use crate::mq::{Context, MqEncoder}; -/// The widest row [`encode_jbig2_generic`] encodes. JBIG2 allows 32-bit widths; this keeps a -/// row's buffers small and its positions in range. -pub const MAX_JBIG2_COLUMNS: u32 = 1 << 20; +/// The widest row, and the most rows, [`encode_jbig2_generic`] encodes: what `hayro-jbig2`, the +/// decoder behind `JBIG2Decode` here, takes on a side. JBIG2 allows 32-bit sizes, but past this +/// nothing here could read the stream, and the optimizer, which doesn't read back what it stores, +/// replaced such an image with one that draws nothing. +pub const MAX_JBIG2_COLUMNS: u32 = 65_535; /// Template 0's pixels (Figure 3), as (x, y) offsets from the pixel being coded, each a bit of the /// context in this order: the four on the current row, the five fixed ones on the row above, the @@ -72,10 +74,12 @@ pub fn encode_jbig2(data: &[u8], columns: u32, rows: u32) -> Result, Fil /// samples have it. Bytes after the last row are ignored, as are the bits that pad a row to a whole /// byte. /// -/// Errors: no rows or columns, or rows wider than [`MAX_JBIG2_COLUMNS`] -/// ([`FilterError::Unsupported`]); fewer bytes than the rows need ([`FilterError::Corrupt`]). +/// Errors: no rows or columns, more of either than [`MAX_JBIG2_COLUMNS`], or more samples than +/// [`MAX_JBIG2_PIXELS`], what the decoder takes ([`FilterError::Unsupported`]); fewer bytes than the +/// rows need ([`FilterError::Corrupt`]). pub fn encode_jbig2_generic(data: &[u8], columns: u32, rows: u32) -> Result, FilterError> { - if columns == 0 || rows == 0 || columns > MAX_JBIG2_COLUMNS { + let too_large = columns > MAX_JBIG2_COLUMNS || rows > MAX_JBIG2_COLUMNS || u64::from(columns) * u64::from(rows) > MAX_JBIG2_PIXELS; + if columns == 0 || rows == 0 || too_large { return Err(FilterError::Unsupported(format!("JBIG2 encoding of {columns} × {rows} samples"))); } let width = columns as usize; @@ -155,7 +159,7 @@ fn segment(out: &mut Vec, number: u32, kind: u8, refers_to: &[(u8, bool)], r out.push((refers_to.len() as u8) << 5 | kept | u8::from(retained)); out.extend(refers_to.iter().map(|(n, _)| *n)); out.push(1); - // A segment's data never comes near 4 GiB here: a row is at most 1 MiB wide. + // A segment's data never comes near 4 GiB here: an image is at most 2^28 samples. out.extend_from_slice(&u32::try_from(data.len()).unwrap_or(u32::MAX).to_be_bytes()); out.extend_from_slice(data); } @@ -343,9 +347,44 @@ mod tests { fn what_it_cannot_encode_is_an_error() { assert!(matches!(encode_jbig2_generic(&[0; 8], 0, 1), Err(FilterError::Unsupported(_)))); assert!(matches!(encode_jbig2_generic(&[0; 8], 8, 0), Err(FilterError::Unsupported(_)))); - assert!(matches!(encode_jbig2_generic(&[0; 8], MAX_JBIG2_COLUMNS + 1, 1), Err(FilterError::Unsupported(_)))); assert!(matches!(encode_jbig2_generic(&[0; 3], 16, 2), Err(FilterError::Corrupt { .. })), "two rows of 16 need 4 bytes"); - assert!(matches!(encode_jbig2_generic(&[], 1, u32::MAX), Err(FilterError::Corrupt { .. }))); + assert!(matches!(encode_jbig2_generic(&[], 1, 2), Err(FilterError::Corrupt { .. }))); + } + + /// The encoder stops where the decoder does, on each side and in total: past that it would + /// write a stream nothing here can read. 16,384 squared is exactly the bound on samples. + #[test] + fn it_encodes_only_what_the_decoder_takes() { + let unsupported = |columns, rows| matches!(encode_jbig2_generic(&[], columns, rows), Err(FilterError::Unsupported(_))); + assert!(unsupported(MAX_JBIG2_COLUMNS + 1, 1), "too wide"); + assert!(unsupported(1, MAX_JBIG2_COLUMNS + 1), "too tall"); + assert!(unsupported(16_385, 16_384), "too many samples, each side within the bound"); + assert!(unsupported(u32::MAX, u32::MAX), "the product doesn't overflow"); + // At the bounds what it writes reads back, through the decoder that has to read it. + let wide = vec![0x0F; (MAX_JBIG2_COLUMNS as usize).div_ceil(8) * 4]; + assert_eq!(decode(&encode_jbig2_generic(&wide, MAX_JBIG2_COLUMNS, 4).unwrap()), unpack(&wide, MAX_JBIG2_COLUMNS, 4)); + let tall = vec![0x7F; MAX_JBIG2_COLUMNS as usize]; + assert_eq!(decode(&encode_jbig2_generic(&tall, 1, MAX_JBIG2_COLUMNS).unwrap()), unpack(&tall, 1, MAX_JBIG2_COLUMNS)); + struct Rows(u32, u64); + impl hayro_jbig2::Decoder for Rows { + fn push_pixel(&mut self, black: bool) { + self.1 += u64::from(black); + } + fn push_pixel_chunk(&mut self, black: bool, chunks: u32) { + self.1 += u64::from(black) * u64::from(chunks) * 8; + } + fn next_line(&mut self) { + self.0 += 1; + } + } + // One black pixel, so a page that came back blank wouldn't pass. + let mut largest = vec![0xFF; 2_048 * 16_384]; + largest[0] = 0x7F; + let largest = encode_jbig2_generic(&largest, 16_384, 16_384).unwrap(); + let mut rows = Rows(0, 0); + hayro_jbig2::Image::new_embedded(&largest, None).unwrap().decode(&mut rows).unwrap(); + assert_eq!(rows.0, 16_384); + assert_eq!(rows.1, 1); } proptest::proptest! { diff --git a/crates/filters/src/jbig2/decode.rs b/crates/filters/src/jbig2/decode.rs index a018e75937..7832b96aaa 100644 --- a/crates/filters/src/jbig2/decode.rs +++ b/crates/filters/src/jbig2/decode.rs @@ -4,7 +4,8 @@ use crate::FilterError; use crate::ccitt::Rows; /// The largest JBIG2 page [`decode_jbig2_within`] decodes, in pixels. The decoder decodes a page whole, -/// and takes pages of up to about this many (20,000 × 10,000 decodes). +/// and takes pages of up to this many (20,000 × 10,000 decodes). It is also the most +/// [`encode_jbig2_generic`](crate::encode_jbig2_generic) encodes, so what it stores reads back. pub const MAX_JBIG2_PIXELS: u64 = 1 << 28; /// A `JBIG2Decode` stream's `data`, with what its `/JBIG2Globals` stream decodes to, decoded as the diff --git a/crates/filters/src/jbig2/symbols.rs b/crates/filters/src/jbig2/symbols.rs index e9fed24441..82244750b7 100644 --- a/crates/filters/src/jbig2/symbols.rs +++ b/crates/filters/src/jbig2/symbols.rs @@ -27,7 +27,9 @@ const MAX_SYMBOLS: usize = 1 << 16; const MAX_SYMBOL_PIXELS: usize = 1 << 24; /// The most symbols one plan codes. The decoder PdfCraft renders JBIG2 with (`hayro-jbig2`) -/// refuses 32,768 symbols or more, and others may too, so a plan with more isn't made. +/// refuses a dictionary whose new and exported symbols pass 65,535 together, and a plan exports +/// every symbol it codes, so it refuses 32,768 or more; others may too, so a plan with more isn't +/// made. const MAX_CODED_SYMBOLS: usize = (1 << 15) - 1; /// The most symbol instances one text region places. The decoder PdfCraft renders JBIG2 with @@ -40,10 +42,6 @@ const MAX_REGION_INSTANCES: usize = 10_000; // generic region. const _: () = assert!(MAX_RUNS.div_ceil(MAX_REGION_INSTANCES) + 3 <= 256); -/// The most rows symbol coded. With rows at most 2^20 samples wide, every coordinate and -/// difference coded then fits the 32-bit integers Annex A codes. -const MAX_ROWS: u32 = 1 << 24; - /// The integer coders' ranges (Table A.1): how many 1 bits pick the range, how many bits give the /// offset into it, and its first value. The last range has no 0 after its prefix. const RANGES: [(u32, u32, u32); 6] = [(0, 2, 0), (1, 4, 4), (2, 6, 20), (3, 8, 84), (4, 12, 340), (5, 32, 4436)]; @@ -104,15 +102,12 @@ struct Instance { /// `None` when neither plan has from two to [`MAX_CODED_SYMBOLS`] symbols, or the image is too /// large or too busy to look at. pub(super) fn encode(samples: &[u8], columns: u32, rows: u32) -> Option> { - let page = page(samples, columns, rows)?; + let page = page(samples, columns)?; [1, 2].into_iter().filter_map(|copies| plan(&page, columns, rows, copies, None)).min_by_key(Vec::len) } /// The image's marks and symbols; `None` past the limits above. -fn page(samples: &[u8], columns: u32, rows: u32) -> Option { - if rows > MAX_ROWS { - return None; - } +fn page(samples: &[u8], columns: u32) -> Option { let width = usize::try_from(columns).ok()?; let stride = width.div_ceil(8); if stride == 0 { @@ -665,7 +660,7 @@ mod tests { fn every_plan_round_trips(picture: &Bitmap) -> usize { let (columns, rows) = (picture[0].len() as u32, picture.len() as u32); let data = samples(picture); - let page = page(&data, columns, rows).unwrap(); + let page = page(&data, columns).unwrap(); let mut plans = 0; for copies in [1, 2] { for log_strips in 0..4 { @@ -735,7 +730,7 @@ mod tests { row[stride - 1] &= 0xFF << (8 * stride - columns); } let (c, r) = (columns as u32, 120); - let page = page(&data, c, r).unwrap(); + let page = page(&data, c).unwrap(); assert!(page.marks.iter().all(|m| m.x1 as usize <= columns), "{columns} columns"); for copies in [1, 2] { if let Some(coded) = plan(&page, c, r, copies, None) { @@ -771,7 +766,7 @@ mod tests { row[290] = 1; } let data = samples(&picture); - let page = page(&data, 300, 300).unwrap(); + let page = page(&data, 300).unwrap(); let boxes: Vec<(u32, u32, u32, u32)> = page.marks.iter().map(|m| (m.x0, m.y0, m.x1 - m.x0, m.y1 - m.y0)).collect(); assert_eq!( boxes, @@ -792,18 +787,14 @@ mod tests { assert_eq!(every_plan_round_trips(&picture), 8); } - /// Past 2^20 black runs or 2^24 rows the image is left to generic coding; at either cap it is - /// still looked at. + /// Past 2^20 black runs the image is left to generic coding; at the cap it is still looked at. #[test] - fn busy_or_tall_images_are_left_to_generic_coding() { + fn busy_images_are_left_to_generic_coding() { // 0x55: four black pixels a byte, each its own run, under the same ones on the next row. let columns = 1 << 20; let row = vec![0x55u8; columns / 8]; - assert!(page(&row.repeat(2), columns as u32, 2).is_some(), "2^20 runs"); - assert!(page(&row.repeat(3), columns as u32, 3).is_none(), "1.5 × 2^20 runs"); - let tall = vec![0xFFu8; MAX_ROWS as usize + 1]; - assert!(page(&tall, 1, MAX_ROWS + 1).is_none()); - assert!(page(&tall, 1, MAX_ROWS).is_some()); + assert!(page(&row.repeat(2), columns as u32).is_some(), "2^20 runs"); + assert!(page(&row.repeat(3), columns as u32).is_none(), "1.5 × 2^20 runs"); } /// A plan codes at most 32,767 symbols, which an independent decoder reads back; with more, @@ -828,16 +819,16 @@ mod tests { (samples(&picture), columns as u32, rows as u32) }; let (data, columns, rows) = codes(MAX_CODED_SYMBOLS, 0); - let at_cap = page(&data, columns, rows).unwrap(); + let at_cap = page(&data, columns).unwrap(); assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows)); // One more distinct mark, each once: neither plan is made. let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 0); - let past = page(&data, columns, rows).unwrap(); + let past = page(&data, columns).unwrap(); assert!(plan(&past, columns, rows, 1, None).is_none() && plan(&past, columns, rows, 2, None).is_none()); assert_eq!(encode_jbig2(&data, columns, rows).unwrap(), encode_jbig2_generic(&data, columns, rows).unwrap()); // With two of them repeated, the second plan codes those two. let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 200); - let past = page(&data, columns, rows).unwrap(); + let past = page(&data, columns).unwrap(); assert!(plan(&past, columns, rows, 1, None).is_none()); assert_eq!(decode(&plan(&past, columns, rows, 2, None).unwrap()), unpack(&data, columns, rows)); // Above a page of text whose letters repeat, that second plan is the one `encode_jbig2` @@ -857,10 +848,10 @@ mod tests { let coded = encode_jbig2(&data, columns, rows).unwrap(); assert_eq!(segment_types(&coded)[1], 0, "symbol coded"); assert_eq!(decode(&coded), unpack(&data, columns, rows)); - let (data, columns, rows) = codes(MAX_SYMBOLS, 0); - assert_eq!(page(&data, columns, rows).map(|p| p.symbols.len()), Some(MAX_SYMBOLS)); - let (data, columns, rows) = codes(MAX_SYMBOLS + 1, 0); - assert!(page(&data, columns, rows).is_none()); + let (data, columns, _) = codes(MAX_SYMBOLS, 0); + assert_eq!(page(&data, columns).map(|p| p.symbols.len()), Some(MAX_SYMBOLS)); + let (data, columns, _) = codes(MAX_SYMBOLS + 1, 0); + assert!(page(&data, columns).is_none()); // `n` black squares of 256 × 256, each with a white pixel in a different place: 2^16 // pixels a symbol, so 256 of them are exactly the cap. let squares = |n: usize| { @@ -876,11 +867,11 @@ mod tests { (samples(&picture), columns as u32, rows as u32) }; let (data, columns, rows) = squares(256); - let at_cap = page(&data, columns, rows).unwrap(); + let at_cap = page(&data, columns).unwrap(); assert_eq!(at_cap.symbols.len(), 256); assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows)); - let (data, columns, rows) = squares(257); - assert!(page(&data, columns, rows).is_none()); + let (data, columns, _) = squares(257); + assert!(page(&data, columns).is_none()); } /// More than 10,000 marks are placed by several text regions, each of at most 10,000 (more diff --git a/crates/optimize/README.md b/crates/optimize/README.md index 2a6c664e6a..6616e1b558 100644 --- a/crates/optimize/README.md +++ b/crates/optimize/README.md @@ -30,7 +30,9 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going threshold are resampled with a bicubic (Catmull-Rom) filter to a target resolution and stored as JPEG (Acrobat's quality steps) or ZIP; soft masks are resampled with their image. A stencil `/Mask` has its own resolution, so the picture changes under it and it stays as it is. - An image is replaced only if the result is smaller. Left alone by these colour and grey + An image is replaced only if the result is smaller. Images over 80 million pixels are left + alone by the image steps, here and below, whatever their settings (memory); Clean Up may still + Flate an unfiltered or LZW one. Left alone by these colour and grey settings: CMYK and other colour spaces, image masks, colour-key masks, decode arrays, more than 8 bits, JPEG 2000, JBIG2 and CCITT (black-and-white images, CCITT and JBIG2 among them, are below). A JPEG is read with its colour transform (an Adobe marker's, else `/ColorTransform`, ISO 32000-2 §7.4.8), so one @@ -96,8 +98,9 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going space, decode array and `/Mask` stay, but for packed RGB, which becomes grey with one decode pair. Group 4 wins on scans, whose ragged edges defeat Flate; on clean computer-drawn 1-bit art Flate can be smaller. Not yet: inline images, an image's own `/SMask` or `/Mask` - image (only images drawn on a page are visited), and Group 4 for rows wider than 32,767 - samples. + image (only images drawn on a page are visited), Group 4 for rows wider than 32,767 + samples, and JBIG2 past 65,535 samples on a side, where the decoder stops (its 2^28 in all + is past the 80 million pixels above). - Lossless JBIG2 uses bounded generic-region encoding or exact-copy symbol dictionaries when smaller. No lossy symbol substitution is used. Export, redaction and this step read JBIG2 diff --git a/crates/optimize/tests/optimize.rs b/crates/optimize/tests/optimize.rs index ad710f3097..fe607bcd8b 100644 --- a/crates/optimize/tests/optimize.rs +++ b/crates/optimize/tests/optimize.rs @@ -3621,6 +3621,129 @@ fn indexed_thin_rules_survive_averaging_for_both_palette_orders_and_decodes() { } } +/// An image past what the JBIG2 decoder takes on a side stays as it is: JBIG2 can state the size, +/// but PdfCraft's decoder refuses such a stream, so stored as JBIG2 it drew nothing here. At 65,535 +/// on a side, the same bars do become JBIG2, so the fixture does reach that choice, and the coding +/// stored (symbols, here) reads back through that decoder. +#[test] +fn an_image_too_large_for_the_jbig2_decoder_stays_as_it_is() { + for (w, h, stored) in + [(65_535u32, 16u32, &b"JBIG2Decode"[..]), (16, 65_535, b"JBIG2Decode"), (65_536, 16, b"FlateDecode"), (16, 65_536, b"FlateDecode")] + { + let mut doc = Document::new_empty(); + let image = mono_image(&mut doc, w, h, &bars(w, h), None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]); + page(&mut doc, &[("Im", image)], &format!("q {w} 0 0 {h} 0 0 cm /Im Do Q")); + let before = stream(&doc, image).raw.clone(); + let settings = + Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, downsample: false, ..MonoSettings::REDUCE }, ..Settings::default() }; + optimize(&mut doc, &settings).unwrap(); + let s = stream(&doc, image); + assert_eq!(s.dict.name(b"Filter"), Some(stored), "{w} × {h}"); + if stored == b"FlateDecode" { + assert_eq!(s.raw, before, "{w} × {h}"); + } else { + // The decoder gives padding bits as 0; the bars have them 1. + let mut bits = bars(w, h); + if w % 8 != 0 { + for row in bits.chunks_mut((w as usize).div_ceil(8)) { + if let Some(last) = row.last_mut() { + *last &= 0xFFu8 << (8 - w % 8); + } + } + } + assert!(decoded_image(&doc, &s).unwrap() == bits, "{w} × {h} reads back"); + } + } +} + +/// The colour and grey steps leave an image over 80 million pixels as it is, whatever their settings +/// (memory): 10,000 × 8,000 grey, the limit itself, drawn at 600 ppi, is resampled, and a row more +/// stays as it is. +#[test] +fn grey_images_are_resampled_up_to_the_pixel_limit() { + for (h, resampled) in [(8_000u32, true), (8_001, false)] { + let mut doc = Document::new_empty(); + let grey = image(&mut doc, 10_000, h, 1, false, None); + let before = stream(&doc, grey).raw.clone(); + let (pw, ph) = (10_000.0 * 72.0 / 600.0, f64::from(h) * 72.0 / 600.0); + page(&mut doc, &[("G", grey)], &format!("q {pw} 0 0 {ph} 0 0 cm /G Do Q")); + let report = optimize(&mut doc, &Settings::default()).unwrap(); + assert_eq!(report.images_resampled, usize::from(resampled), "10,000 × {h}: {report:?}"); + let s = stream(&doc, grey); + if resampled { + assert_eq!(s.dict.int(b"Width"), Some(2_500), "10,000 × {h}"); + } else { + assert!(s.dict.int(b"Width") == Some(10_000) && s.raw == before, "10,000 × {h} stays as it is"); + } + } +} + +/// Dark bars of irregular lengths across the long side of a `w` × `h` 1-bit image, which Flate +/// shrinks less than JBIG2. +fn bars(w: u32, h: u32) -> Vec { + let stride = (w as usize).div_ceil(8); + let mut data = vec![0xFF; stride * h as usize]; + let mut state = 0x1234_5678u32; + let mut next = move || { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + state as usize + }; + let long = w.max(h) as usize; + let mut at = 10; + while at + 60 < long { + let len = 5 + next() % 50; + for along in at..at + len { + for across in 0..12 { + let (x, y) = if w > h { (along, across) } else { (across, along) }; + data[y * stride + x / 8] &= !(0x80 >> (x % 8)); + } + } + at += len + 5 + next() % 50; + } + data +} + +/// Past what the JBIG2 decoder takes on either side, an image packed from more bits, or one +/// averaged to a size still past it, isn't stored as JBIG2 either: it becomes 1-bit Flate. Within +/// the bound the same pictures do become JBIG2, so the fixtures reach that choice. +#[test] +fn what_is_too_large_for_jbig2_is_stored_another_way() { + let settings = |downsample| Settings { + mono: MonoSettings { compression: MonoCompression::Jbig2, downsample, ..MonoSettings::REDUCE }, + ..Settings::default() + }; + let (jbig2, flate) = (&b"JBIG2Decode"[..], &b"FlateDecode"[..]); + // Eight-bit grey of only black and white, packed to one bit first, wide and tall. + for (w, h, stored) in [(65_535u32, 16u32, jbig2), (65_536, 16, flate), (16, 65_535, jbig2), (16, 65_536, flate)] { + let (bits, stride) = (bars(w, h), (w as usize).div_ceil(8)); + let grey: Vec = (0..h as usize) + .flat_map(|y| (0..w as usize).map(move |x| (y, x))) + .map(|(y, x)| if bits[y * stride + x / 8] & (0x80 >> (x % 8)) == 0 { 0 } else { 255 }) + .collect(); + let mut doc = Document::new_empty(); + let image = mono_image(&mut doc, w, h, &grey, None, &[("BitsPerComponent", 8.into()), ("ColorSpace", Object::name("DeviceGray"))]); + page(&mut doc, &[("Im", image)], &format!("q {w} 0 0 {h} 0 0 cm /Im Do Q")); + optimize(&mut doc, &settings(false)).unwrap(); + let s = stream(&doc, image); + assert_eq!(s.dict.name(b"Filter"), Some(stored), "{w} × {h}, packed"); + assert_eq!(s.dict.int(b"BitsPerComponent"), Some(1), "{w} × {h}, packed"); + } + // One bit, drawn at 600 ppi and averaged to 300: half the size, so 140,000 is still past it. + for (w, h, stored) in [(70_000u32, 16u32, jbig2), (140_000, 16, flate), (16, 70_000, jbig2), (16, 140_000, flate)] { + let mut doc = Document::new_empty(); + let image = mono_image(&mut doc, w, h, &bars(w, h), None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]); + let (pw, ph) = (f64::from(w) * 72.0 / 600.0, f64::from(h) * 72.0 / 600.0); + page(&mut doc, &[("Im", image)], &format!("q {pw} 0 0 {ph} 0 0 cm /Im Do Q")); + optimize(&mut doc, &settings(true)).unwrap(); + let s = stream(&doc, image); + assert_eq!(s.dict.name(b"Filter"), Some(stored), "{w} × {h}, averaged"); + assert_eq!(s.dict.int(b"BitsPerComponent"), Some(1), "{w} × {h}, averaged"); + assert_eq!((s.dict.int(b"Width"), s.dict.int(b"Height")), (Some(i64::from(w / 2)), Some(i64::from(h / 2))), "{w} × {h}, averaged"); + } +} + /// An indexed image also drawn where the optimizer doesn't read (an annotation's appearance, where /// other default colour spaces may be in force and change which palette entry is darker) keeps its /// resolution, as black-and-white RGB keeps its colour space; drawn only on the page, it is diff --git a/parity/acrobat-features.toml b/parity/acrobat-features.toml index 8b7ac909a0..e60a1ad2fb 100644 --- a/parity/acrobat-features.toml +++ b/parity/acrobat-features.toml @@ -7439,8 +7439,8 @@ milestone = "M11" status = "partial" commands = ["optimize.advanced", "optimize.reduce"] tools = ["doc_optimize", "doc_reduce"] -evidence = ["crates/optimize/src/images.rs::flate_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::jbig2_images_past_the_runs_allowance_stay_as_they_are"] -notes = "All of a run's images share one allowance of decoding, filters, decoders and pixels alike (about 1,000 scanned pages' worth, more for a longer file); those past it stay as they are, those the image pass read being counted in the report, which doc_optimize and doc_reduce return and the panel doesn't show yet. CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples." +evidence = ["crates/optimize/src/images.rs::flate_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::jbig2_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::an_image_too_large_for_the_jbig2_decoder_stays_as_it_is", "crates/optimize/tests/optimize.rs::what_is_too_large_for_jbig2_is_stored_another_way", "crates/optimize/tests/optimize.rs::group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit"] +notes = "All of a run's images share one allowance of decoding, filters, decoders and pixels alike (about 1,000 scanned pages' worth, more for a longer file); those past it stay as they are, those the image pass read being counted in the report, which doc_optimize and doc_reduce return and the panel doesn't show yet. CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: images over 80 million pixels (left alone by the image steps, whatever their settings; Clean Up may still Flate an unfiltered or LZW one), inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples, JBIG2 past 65,535 samples on a side, where the decoder stops: such an image isn't stored as JBIG2 (the image step leaves a 1-bit one as it is unless it is averaged, and may store one packed from more bits as 1-bit Flate)." [[feature]] id = "optimize.only-if-smaller"