mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-06 02:21:45 +00:00
M11: JBIG2 encodes only what its decoder takes: Optimize no longer stores large images PdfCraft can't draw (#5511)
* M11: JBIG2 encodes only what its decoder takes: Optimize no longer loses large images encode_jbig2 took rows of up to 2^20 samples and any number of them, while hayro-jbig2, which draws and reads these streams, takes 65,535 on a side and 2^28 samples. The optimizer doesn't read back what it stores, so with Monochrome Images on JBIG2, any 1-bit image with a side of 65,536 or more was replaced by a stream that draws nothing: 65,536 × 16 went from 37,875 dark pixels to none, and was reported as recompressed. The encoder now refuses what the decoder can't read, so such an image stays as it is. Symbol coding's limit of 2^24 rows can't be reached any more, so it goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: the JBIG2 boundary test checks a pixel It counted rows, which a page decoded blank would pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: the encoder's bound on samples is MAX_JBIG2_PIXELS, the decoder's main brought decode_jbig2 and its public MAX_JBIG2_PIXELS into this crate, 2^28 like this branch's private MAX_JBIG2_SAMPLES and for the same reason. One constant now bounds both, so the encoder can't drift past what the decoder reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: an image too large for JBIG2 isn't stored as JBIG2, as the notes say The parity note said such an image stays as it is. A 1-bit one does unless it is averaged, but one packed from more bits may become 1-bit Flate. The note now says it isn't stored as JBIG2 and cites an_image_too_large_for_the_jbig2_decoder_stays_as_it_is, and the optimizer README lists the bound under Not yet. MAX_JBIG2_PIXELS is the most the decoder takes, not about that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: the optimizer's own size limit, and tests for what JBIG2 can't take The optimizer leaves images over 80 million pixels alone whatever the settings, and nothing said so; the README and parity note say it now, and that JBIG2's 2^28 in all is past it, so from Optimize the side bound is the one that applies. The note says what the image step does: it leaves a 1-bit image past the bound as it is unless it is averaged, and may store one packed from more bits as 1-bit Flate. what_is_too_large_for_jbig2_is_stored_another_way tests those two cases: eight-bit grey of black and white at 65,536 wide becomes 1-bit Flate, and a 1-bit image at 600 ppi averaged to 70,000 wide does too. At 65,535 and 35,000 both become JBIG2. The symbol cap's comment says why hayro-jbig2 refuses 32,768 symbols: it caps new and exported symbols together at 65,535, and a plan exports every symbol it codes. Measured: 32,767 decode, 32,768, 40,000 and 65,535 are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: the 80-million-pixel limit is the image steps', tested on both sides The README and parity note said images over 80 million pixels are left alone whatever the settings, but Clean Up still Flates an unfiltered or LZW one. They now say the image steps leave them alone, and the note cites group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit for it. (The last commit said the note also gives 2^28 as past that limit; only the README does.) what_is_too_large_for_jbig2_is_stored_another_way covered the width bound only: it now runs tall images too, packed and averaged, and checks the averaged ones are one bit and half the size both ways. The older test uses the same bars() fixture, and its comment says such a stream drew nothing in PdfCraft, not that the picture was lost: Poppler and Ghostscript draw it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: the JBIG2 stored at the decoder's bound reads back, and grey keeps the pixel limit - The optimize test at 65,535 on a side decoded nothing it stored: it checked the filter only, and the coding stored there is symbols, not the generic coding the filters test reads back. It now decodes the stored stream and compares it with the bars. - The README's 80-million-pixel sentence covers the colour and grey steps too, and only the 1-bit step had a test. A grey image of 10,000 × 8,000 at 600 ppi is resampled; a row more stays as it is. - The filters README names decode_jbig2_within, with its work, as main renamed decode_jbig2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: the filters README names #5569's other renames too The README named decode_jbig2_within but still decode_ccitt, which #5569 made private behind decode_ccitt_within, and gave Document::decoded_image as both decoders' caller, which is test-only now: ImageBudget::decode calls them, as lib.rs's doc says. The ccitt.rs test line keeps decode_ccitt, which those tests call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3683150ee0
commit
bc7c60bb41
7 files changed
+211
-54
No files matched your search
@@ -27,10 +27,10 @@ encode(filter, params, data) -> Result<Vec<u8>, FilterError>
|
||||
encode_flate(data) -> Vec<u8> // zlib, level 6
|
||||
encode_g4(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as CCITT Group 4; columns <= MAX_G4_COLUMNS
|
||||
pub struct CcittParams { k, columns, rows, end_of_line, encoded_byte_align, end_of_block, black_is_1 } // Table 11; Default = its defaults
|
||||
decode_ccitt(data, ¶ms, max_rows) -> Result<Vec<u8>, FilterError> // CCITT fax as CCITTFaxDecode gives it; columns <= MAX_CCITT_COLUMNS
|
||||
decode_jbig2(data, globals, columns, rows, max) -> Result<Vec<u8>, FilterError> // JBIG2 as JBIG2Decode gives it; a page of columns × rows, within max and MAX_JBIG2_PIXELS
|
||||
encode_jbig2_generic(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode; columns <= MAX_JBIG2_COLUMNS
|
||||
encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode, symbol coded when that is smaller; columns <= MAX_JBIG2_COLUMNS
|
||||
decode_ccitt_within(data, ¶ms, max_rows, work: &mut u64) -> Result<Vec<u8>, FilterError> // CCITT fax as CCITTFaxDecode gives it; columns <= MAX_CCITT_COLUMNS; its rows paid for from work before they are decoded
|
||||
decode_jbig2_within(data, globals, columns, rows, max, work: &mut u64) -> Result<Vec<u8>, FilterError> // JBIG2 as JBIG2Decode gives it; a page of columns × rows, within max and MAX_JBIG2_PIXELS; reading and decoding it spend from work, in the decoder's steps
|
||||
encode_jbig2_generic(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode; columns and rows <= MAX_JBIG2_COLUMNS (65,535) and MAX_JBIG2_PIXELS (2^28) samples, what the decoder takes
|
||||
encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode, symbol coded when that is smaller; the same limits
|
||||
```
|
||||
|
||||
- `chain` is applied first to last, as in a `/Filter` array.
|
||||
@@ -50,8 +50,8 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
| ASCIIHexDecode | white space ignored, odd digit padded with 0, `>` EOD (optional), other characters → Corrupt | upper case, newline every 64 digits, `>` |
|
||||
| ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` |
|
||||
| RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 |
|
||||
| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::Document::decoded_image` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt |
|
||||
| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, about the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::Document::decoded_image` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 1,048,576 samples a row; no rows or columns, or too wide → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, or more than 2^24 rows, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made |
|
||||
| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt_within` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::ImageBudget::decode` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt |
|
||||
| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2_within` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::ImageBudget::decode` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 65,535 samples on a side and 2^28 in all, where `hayro-jbig2` stops; no rows or columns, or too large → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made |
|
||||
|
||||
## Tests
|
||||
|
||||
@@ -61,8 +61,8 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
- `ccitt.rs`: `decode_ccitt` gives back the rows `encode_g4` was given (with `/BlackIs1`, `/Rows` and a row limit), refuses bad widths, and bounds its output for any input; byte-aligned Group 3 in each layout, over 32 seeded pictures, decodes to its picture, and a byte short to fewer rows. Group 4 output decoded by `hayro-ccitt` (an independent decoder) matches the input bits: a text-like page, padding bits, the widest row, and property tests over bitmaps of noise and of runs; any input returns rather than panics.
|
||||
- `mq.rs`: the arithmetic coder reproduces Annex H.2's 30 bytes from its 256 decisions.
|
||||
- `jbig2.rs`: output decoded by `hayro-jbig2` (dev-dependency, an independent decoder) matches the input bits: a text-like page (smaller than Group 4), edges, padding, all-white and all-black regions (typical rows), and property tests over bitmaps of noise and of runs; any input returns rather than panics.
|
||||
- `jbig2/decode.rs`: `decode_jbig2` gives back the rows of both codings, padding bits 0; refuses a page of another width or height, past `max` or past `MAX_JBIG2_PIXELS` before decoding it (2^28 pixels take about 34 MB, a bit each), globals that aren't JBIG2 and damaged data, and returns rather than panics on every prefix of a stream.
|
||||
- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 2^24 rows, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics.
|
||||
- `jbig2/decode.rs`: `decode_jbig2_within` gives back the rows of both codings, padding bits 0; refuses a page of another width or height, past `max` or past `MAX_JBIG2_PIXELS` before decoding it (2^28 pixels take about 34 MB, a bit each), globals that aren't JBIG2 and damaged data, and returns rather than panics on every prefix of a stream.
|
||||
- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics.
|
||||
- `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`).
|
||||
|
||||
## Status
|
||||
|
||||
@@ -12,9 +12,11 @@ pub use decode::{MAX_JBIG2_PIXELS, decode_jbig2_within};
|
||||
use crate::FilterError;
|
||||
use crate::mq::{Context, MqEncoder};
|
||||
|
||||
/// The widest row [`encode_jbig2_generic`] encodes. JBIG2 allows 32-bit widths; this keeps a
|
||||
/// row's buffers small and its positions in range.
|
||||
pub const MAX_JBIG2_COLUMNS: u32 = 1 << 20;
|
||||
/// The widest row, and the most rows, [`encode_jbig2_generic`] encodes: what `hayro-jbig2`, the
|
||||
/// decoder behind `JBIG2Decode` here, takes on a side. JBIG2 allows 32-bit sizes, but past this
|
||||
/// nothing here could read the stream, and the optimizer, which doesn't read back what it stores,
|
||||
/// replaced such an image with one that draws nothing.
|
||||
pub const MAX_JBIG2_COLUMNS: u32 = 65_535;
|
||||
|
||||
/// Template 0's pixels (Figure 3), as (x, y) offsets from the pixel being coded, each a bit of the
|
||||
/// context in this order: the four on the current row, the five fixed ones on the row above, the
|
||||
@@ -72,10 +74,12 @@ pub fn encode_jbig2(data: &[u8], columns: u32, rows: u32) -> Result<Vec<u8>, Fil
|
||||
/// samples have it. Bytes after the last row are ignored, as are the bits that pad a row to a whole
|
||||
/// byte.
|
||||
///
|
||||
/// Errors: no rows or columns, or rows wider than [`MAX_JBIG2_COLUMNS`]
|
||||
/// ([`FilterError::Unsupported`]); fewer bytes than the rows need ([`FilterError::Corrupt`]).
|
||||
/// Errors: no rows or columns, more of either than [`MAX_JBIG2_COLUMNS`], or more samples than
|
||||
/// [`MAX_JBIG2_PIXELS`], what the decoder takes ([`FilterError::Unsupported`]); fewer bytes than the
|
||||
/// rows need ([`FilterError::Corrupt`]).
|
||||
pub fn encode_jbig2_generic(data: &[u8], columns: u32, rows: u32) -> Result<Vec<u8>, FilterError> {
|
||||
if columns == 0 || rows == 0 || columns > MAX_JBIG2_COLUMNS {
|
||||
let too_large = columns > MAX_JBIG2_COLUMNS || rows > MAX_JBIG2_COLUMNS || u64::from(columns) * u64::from(rows) > MAX_JBIG2_PIXELS;
|
||||
if columns == 0 || rows == 0 || too_large {
|
||||
return Err(FilterError::Unsupported(format!("JBIG2 encoding of {columns} × {rows} samples")));
|
||||
}
|
||||
let width = columns as usize;
|
||||
@@ -155,7 +159,7 @@ fn segment(out: &mut Vec<u8>, number: u32, kind: u8, refers_to: &[(u8, bool)], r
|
||||
out.push((refers_to.len() as u8) << 5 | kept | u8::from(retained));
|
||||
out.extend(refers_to.iter().map(|(n, _)| *n));
|
||||
out.push(1);
|
||||
// A segment's data never comes near 4 GiB here: a row is at most 1 MiB wide.
|
||||
// A segment's data never comes near 4 GiB here: an image is at most 2^28 samples.
|
||||
out.extend_from_slice(&u32::try_from(data.len()).unwrap_or(u32::MAX).to_be_bytes());
|
||||
out.extend_from_slice(data);
|
||||
}
|
||||
@@ -343,9 +347,44 @@ mod tests {
|
||||
fn what_it_cannot_encode_is_an_error() {
|
||||
assert!(matches!(encode_jbig2_generic(&[0; 8], 0, 1), Err(FilterError::Unsupported(_))));
|
||||
assert!(matches!(encode_jbig2_generic(&[0; 8], 8, 0), Err(FilterError::Unsupported(_))));
|
||||
assert!(matches!(encode_jbig2_generic(&[0; 8], MAX_JBIG2_COLUMNS + 1, 1), Err(FilterError::Unsupported(_))));
|
||||
assert!(matches!(encode_jbig2_generic(&[0; 3], 16, 2), Err(FilterError::Corrupt { .. })), "two rows of 16 need 4 bytes");
|
||||
assert!(matches!(encode_jbig2_generic(&[], 1, u32::MAX), Err(FilterError::Corrupt { .. })));
|
||||
assert!(matches!(encode_jbig2_generic(&[], 1, 2), Err(FilterError::Corrupt { .. })));
|
||||
}
|
||||
|
||||
/// The encoder stops where the decoder does, on each side and in total: past that it would
|
||||
/// write a stream nothing here can read. 16,384 squared is exactly the bound on samples.
|
||||
#[test]
|
||||
fn it_encodes_only_what_the_decoder_takes() {
|
||||
let unsupported = |columns, rows| matches!(encode_jbig2_generic(&[], columns, rows), Err(FilterError::Unsupported(_)));
|
||||
assert!(unsupported(MAX_JBIG2_COLUMNS + 1, 1), "too wide");
|
||||
assert!(unsupported(1, MAX_JBIG2_COLUMNS + 1), "too tall");
|
||||
assert!(unsupported(16_385, 16_384), "too many samples, each side within the bound");
|
||||
assert!(unsupported(u32::MAX, u32::MAX), "the product doesn't overflow");
|
||||
// At the bounds what it writes reads back, through the decoder that has to read it.
|
||||
let wide = vec![0x0F; (MAX_JBIG2_COLUMNS as usize).div_ceil(8) * 4];
|
||||
assert_eq!(decode(&encode_jbig2_generic(&wide, MAX_JBIG2_COLUMNS, 4).unwrap()), unpack(&wide, MAX_JBIG2_COLUMNS, 4));
|
||||
let tall = vec![0x7F; MAX_JBIG2_COLUMNS as usize];
|
||||
assert_eq!(decode(&encode_jbig2_generic(&tall, 1, MAX_JBIG2_COLUMNS).unwrap()), unpack(&tall, 1, MAX_JBIG2_COLUMNS));
|
||||
struct Rows(u32, u64);
|
||||
impl hayro_jbig2::Decoder for Rows {
|
||||
fn push_pixel(&mut self, black: bool) {
|
||||
self.1 += u64::from(black);
|
||||
}
|
||||
fn push_pixel_chunk(&mut self, black: bool, chunks: u32) {
|
||||
self.1 += u64::from(black) * u64::from(chunks) * 8;
|
||||
}
|
||||
fn next_line(&mut self) {
|
||||
self.0 += 1;
|
||||
}
|
||||
}
|
||||
// One black pixel, so a page that came back blank wouldn't pass.
|
||||
let mut largest = vec![0xFF; 2_048 * 16_384];
|
||||
largest[0] = 0x7F;
|
||||
let largest = encode_jbig2_generic(&largest, 16_384, 16_384).unwrap();
|
||||
let mut rows = Rows(0, 0);
|
||||
hayro_jbig2::Image::new_embedded(&largest, None).unwrap().decode(&mut rows).unwrap();
|
||||
assert_eq!(rows.0, 16_384);
|
||||
assert_eq!(rows.1, 1);
|
||||
}
|
||||
|
||||
proptest::proptest! {
|
||||
|
||||
@@ -4,7 +4,8 @@ use crate::FilterError;
|
||||
use crate::ccitt::Rows;
|
||||
|
||||
/// The largest JBIG2 page [`decode_jbig2_within`] decodes, in pixels. The decoder decodes a page whole,
|
||||
/// and takes pages of up to about this many (20,000 × 10,000 decodes).
|
||||
/// and takes pages of up to this many (20,000 × 10,000 decodes). It is also the most
|
||||
/// [`encode_jbig2_generic`](crate::encode_jbig2_generic) encodes, so what it stores reads back.
|
||||
pub const MAX_JBIG2_PIXELS: u64 = 1 << 28;
|
||||
|
||||
/// A `JBIG2Decode` stream's `data`, with what its `/JBIG2Globals` stream decodes to, decoded as the
|
||||
|
||||
@@ -27,7 +27,9 @@ const MAX_SYMBOLS: usize = 1 << 16;
|
||||
const MAX_SYMBOL_PIXELS: usize = 1 << 24;
|
||||
|
||||
/// The most symbols one plan codes. The decoder PdfCraft renders JBIG2 with (`hayro-jbig2`)
|
||||
/// refuses 32,768 symbols or more, and others may too, so a plan with more isn't made.
|
||||
/// refuses a dictionary whose new and exported symbols pass 65,535 together, and a plan exports
|
||||
/// every symbol it codes, so it refuses 32,768 or more; others may too, so a plan with more isn't
|
||||
/// made.
|
||||
const MAX_CODED_SYMBOLS: usize = (1 << 15) - 1;
|
||||
|
||||
/// The most symbol instances one text region places. The decoder PdfCraft renders JBIG2 with
|
||||
@@ -40,10 +42,6 @@ const MAX_REGION_INSTANCES: usize = 10_000;
|
||||
// generic region.
|
||||
const _: () = assert!(MAX_RUNS.div_ceil(MAX_REGION_INSTANCES) + 3 <= 256);
|
||||
|
||||
/// The most rows symbol coded. With rows at most 2^20 samples wide, every coordinate and
|
||||
/// difference coded then fits the 32-bit integers Annex A codes.
|
||||
const MAX_ROWS: u32 = 1 << 24;
|
||||
|
||||
/// The integer coders' ranges (Table A.1): how many 1 bits pick the range, how many bits give the
|
||||
/// offset into it, and its first value. The last range has no 0 after its prefix.
|
||||
const RANGES: [(u32, u32, u32); 6] = [(0, 2, 0), (1, 4, 4), (2, 6, 20), (3, 8, 84), (4, 12, 340), (5, 32, 4436)];
|
||||
@@ -104,15 +102,12 @@ struct Instance {
|
||||
/// `None` when neither plan has from two to [`MAX_CODED_SYMBOLS`] symbols, or the image is too
|
||||
/// large or too busy to look at.
|
||||
pub(super) fn encode(samples: &[u8], columns: u32, rows: u32) -> Option<Vec<u8>> {
|
||||
let page = page(samples, columns, rows)?;
|
||||
let page = page(samples, columns)?;
|
||||
[1, 2].into_iter().filter_map(|copies| plan(&page, columns, rows, copies, None)).min_by_key(Vec::len)
|
||||
}
|
||||
|
||||
/// The image's marks and symbols; `None` past the limits above.
|
||||
fn page(samples: &[u8], columns: u32, rows: u32) -> Option<Page> {
|
||||
if rows > MAX_ROWS {
|
||||
return None;
|
||||
}
|
||||
fn page(samples: &[u8], columns: u32) -> Option<Page> {
|
||||
let width = usize::try_from(columns).ok()?;
|
||||
let stride = width.div_ceil(8);
|
||||
if stride == 0 {
|
||||
@@ -665,7 +660,7 @@ mod tests {
|
||||
fn every_plan_round_trips(picture: &Bitmap) -> usize {
|
||||
let (columns, rows) = (picture[0].len() as u32, picture.len() as u32);
|
||||
let data = samples(picture);
|
||||
let page = page(&data, columns, rows).unwrap();
|
||||
let page = page(&data, columns).unwrap();
|
||||
let mut plans = 0;
|
||||
for copies in [1, 2] {
|
||||
for log_strips in 0..4 {
|
||||
@@ -735,7 +730,7 @@ mod tests {
|
||||
row[stride - 1] &= 0xFF << (8 * stride - columns);
|
||||
}
|
||||
let (c, r) = (columns as u32, 120);
|
||||
let page = page(&data, c, r).unwrap();
|
||||
let page = page(&data, c).unwrap();
|
||||
assert!(page.marks.iter().all(|m| m.x1 as usize <= columns), "{columns} columns");
|
||||
for copies in [1, 2] {
|
||||
if let Some(coded) = plan(&page, c, r, copies, None) {
|
||||
@@ -771,7 +766,7 @@ mod tests {
|
||||
row[290] = 1;
|
||||
}
|
||||
let data = samples(&picture);
|
||||
let page = page(&data, 300, 300).unwrap();
|
||||
let page = page(&data, 300).unwrap();
|
||||
let boxes: Vec<(u32, u32, u32, u32)> = page.marks.iter().map(|m| (m.x0, m.y0, m.x1 - m.x0, m.y1 - m.y0)).collect();
|
||||
assert_eq!(
|
||||
boxes,
|
||||
@@ -792,18 +787,14 @@ mod tests {
|
||||
assert_eq!(every_plan_round_trips(&picture), 8);
|
||||
}
|
||||
|
||||
/// Past 2^20 black runs or 2^24 rows the image is left to generic coding; at either cap it is
|
||||
/// still looked at.
|
||||
/// Past 2^20 black runs the image is left to generic coding; at the cap it is still looked at.
|
||||
#[test]
|
||||
fn busy_or_tall_images_are_left_to_generic_coding() {
|
||||
fn busy_images_are_left_to_generic_coding() {
|
||||
// 0x55: four black pixels a byte, each its own run, under the same ones on the next row.
|
||||
let columns = 1 << 20;
|
||||
let row = vec![0x55u8; columns / 8];
|
||||
assert!(page(&row.repeat(2), columns as u32, 2).is_some(), "2^20 runs");
|
||||
assert!(page(&row.repeat(3), columns as u32, 3).is_none(), "1.5 × 2^20 runs");
|
||||
let tall = vec![0xFFu8; MAX_ROWS as usize + 1];
|
||||
assert!(page(&tall, 1, MAX_ROWS + 1).is_none());
|
||||
assert!(page(&tall, 1, MAX_ROWS).is_some());
|
||||
assert!(page(&row.repeat(2), columns as u32).is_some(), "2^20 runs");
|
||||
assert!(page(&row.repeat(3), columns as u32).is_none(), "1.5 × 2^20 runs");
|
||||
}
|
||||
|
||||
/// A plan codes at most 32,767 symbols, which an independent decoder reads back; with more,
|
||||
@@ -828,16 +819,16 @@ mod tests {
|
||||
(samples(&picture), columns as u32, rows as u32)
|
||||
};
|
||||
let (data, columns, rows) = codes(MAX_CODED_SYMBOLS, 0);
|
||||
let at_cap = page(&data, columns, rows).unwrap();
|
||||
let at_cap = page(&data, columns).unwrap();
|
||||
assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows));
|
||||
// One more distinct mark, each once: neither plan is made.
|
||||
let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 0);
|
||||
let past = page(&data, columns, rows).unwrap();
|
||||
let past = page(&data, columns).unwrap();
|
||||
assert!(plan(&past, columns, rows, 1, None).is_none() && plan(&past, columns, rows, 2, None).is_none());
|
||||
assert_eq!(encode_jbig2(&data, columns, rows).unwrap(), encode_jbig2_generic(&data, columns, rows).unwrap());
|
||||
// With two of them repeated, the second plan codes those two.
|
||||
let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 200);
|
||||
let past = page(&data, columns, rows).unwrap();
|
||||
let past = page(&data, columns).unwrap();
|
||||
assert!(plan(&past, columns, rows, 1, None).is_none());
|
||||
assert_eq!(decode(&plan(&past, columns, rows, 2, None).unwrap()), unpack(&data, columns, rows));
|
||||
// Above a page of text whose letters repeat, that second plan is the one `encode_jbig2`
|
||||
@@ -857,10 +848,10 @@ mod tests {
|
||||
let coded = encode_jbig2(&data, columns, rows).unwrap();
|
||||
assert_eq!(segment_types(&coded)[1], 0, "symbol coded");
|
||||
assert_eq!(decode(&coded), unpack(&data, columns, rows));
|
||||
let (data, columns, rows) = codes(MAX_SYMBOLS, 0);
|
||||
assert_eq!(page(&data, columns, rows).map(|p| p.symbols.len()), Some(MAX_SYMBOLS));
|
||||
let (data, columns, rows) = codes(MAX_SYMBOLS + 1, 0);
|
||||
assert!(page(&data, columns, rows).is_none());
|
||||
let (data, columns, _) = codes(MAX_SYMBOLS, 0);
|
||||
assert_eq!(page(&data, columns).map(|p| p.symbols.len()), Some(MAX_SYMBOLS));
|
||||
let (data, columns, _) = codes(MAX_SYMBOLS + 1, 0);
|
||||
assert!(page(&data, columns).is_none());
|
||||
// `n` black squares of 256 × 256, each with a white pixel in a different place: 2^16
|
||||
// pixels a symbol, so 256 of them are exactly the cap.
|
||||
let squares = |n: usize| {
|
||||
@@ -876,11 +867,11 @@ mod tests {
|
||||
(samples(&picture), columns as u32, rows as u32)
|
||||
};
|
||||
let (data, columns, rows) = squares(256);
|
||||
let at_cap = page(&data, columns, rows).unwrap();
|
||||
let at_cap = page(&data, columns).unwrap();
|
||||
assert_eq!(at_cap.symbols.len(), 256);
|
||||
assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows));
|
||||
let (data, columns, rows) = squares(257);
|
||||
assert!(page(&data, columns, rows).is_none());
|
||||
let (data, columns, _) = squares(257);
|
||||
assert!(page(&data, columns).is_none());
|
||||
}
|
||||
|
||||
/// More than 10,000 marks are placed by several text regions, each of at most 10,000 (more
|
||||
|
||||
@@ -30,7 +30,9 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
threshold are resampled with a bicubic (Catmull-Rom) filter to a target resolution and
|
||||
stored as JPEG (Acrobat's quality steps) or ZIP; soft masks are resampled with their image. A
|
||||
stencil `/Mask` has its own resolution, so the picture changes under it and it stays as it is.
|
||||
An image is replaced only if the result is smaller. Left alone by these colour and grey
|
||||
An image is replaced only if the result is smaller. Images over 80 million pixels are left
|
||||
alone by the image steps, here and below, whatever their settings (memory); Clean Up may still
|
||||
Flate an unfiltered or LZW one. Left alone by these colour and grey
|
||||
settings: CMYK and other colour spaces, image masks, colour-key masks, decode arrays, more than 8 bits, JPEG 2000,
|
||||
JBIG2 and CCITT (black-and-white images, CCITT and JBIG2 among them, are below). A JPEG is read with
|
||||
its colour transform (an Adobe marker's, else `/ColorTransform`, ISO 32000-2 §7.4.8), so one
|
||||
@@ -96,8 +98,9 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
space, decode array and `/Mask` stay, but for packed RGB, which becomes grey with one decode
|
||||
pair. Group 4 wins on scans, whose ragged edges defeat Flate; on clean computer-drawn 1-bit
|
||||
art Flate can be smaller. Not yet: inline images, an image's own `/SMask` or `/Mask`
|
||||
image (only images drawn on a page are visited), and Group 4 for rows wider than 32,767
|
||||
samples.
|
||||
image (only images drawn on a page are visited), Group 4 for rows wider than 32,767
|
||||
samples, and JBIG2 past 65,535 samples on a side, where the decoder stops (its 2^28 in all
|
||||
is past the 80 million pixels above).
|
||||
|
||||
- Lossless JBIG2 uses bounded generic-region encoding or exact-copy symbol dictionaries when
|
||||
smaller. No lossy symbol substitution is used. Export, redaction and this step read JBIG2
|
||||
|
||||
@@ -3621,6 +3621,129 @@ fn indexed_thin_rules_survive_averaging_for_both_palette_orders_and_decodes() {
|
||||
}
|
||||
}
|
||||
|
||||
/// An image past what the JBIG2 decoder takes on a side stays as it is: JBIG2 can state the size,
|
||||
/// but PdfCraft's decoder refuses such a stream, so stored as JBIG2 it drew nothing here. At 65,535
|
||||
/// on a side, the same bars do become JBIG2, so the fixture does reach that choice, and the coding
|
||||
/// stored (symbols, here) reads back through that decoder.
|
||||
#[test]
|
||||
fn an_image_too_large_for_the_jbig2_decoder_stays_as_it_is() {
|
||||
for (w, h, stored) in
|
||||
[(65_535u32, 16u32, &b"JBIG2Decode"[..]), (16, 65_535, b"JBIG2Decode"), (65_536, 16, b"FlateDecode"), (16, 65_536, b"FlateDecode")]
|
||||
{
|
||||
let mut doc = Document::new_empty();
|
||||
let image = mono_image(&mut doc, w, h, &bars(w, h), None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", image)], &format!("q {w} 0 0 {h} 0 0 cm /Im Do Q"));
|
||||
let before = stream(&doc, image).raw.clone();
|
||||
let settings =
|
||||
Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, downsample: false, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
optimize(&mut doc, &settings).unwrap();
|
||||
let s = stream(&doc, image);
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(stored), "{w} × {h}");
|
||||
if stored == b"FlateDecode" {
|
||||
assert_eq!(s.raw, before, "{w} × {h}");
|
||||
} else {
|
||||
// The decoder gives padding bits as 0; the bars have them 1.
|
||||
let mut bits = bars(w, h);
|
||||
if w % 8 != 0 {
|
||||
for row in bits.chunks_mut((w as usize).div_ceil(8)) {
|
||||
if let Some(last) = row.last_mut() {
|
||||
*last &= 0xFFu8 << (8 - w % 8);
|
||||
}
|
||||
}
|
||||
}
|
||||
assert!(decoded_image(&doc, &s).unwrap() == bits, "{w} × {h} reads back");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The colour and grey steps leave an image over 80 million pixels as it is, whatever their settings
|
||||
/// (memory): 10,000 × 8,000 grey, the limit itself, drawn at 600 ppi, is resampled, and a row more
|
||||
/// stays as it is.
|
||||
#[test]
|
||||
fn grey_images_are_resampled_up_to_the_pixel_limit() {
|
||||
for (h, resampled) in [(8_000u32, true), (8_001, false)] {
|
||||
let mut doc = Document::new_empty();
|
||||
let grey = image(&mut doc, 10_000, h, 1, false, None);
|
||||
let before = stream(&doc, grey).raw.clone();
|
||||
let (pw, ph) = (10_000.0 * 72.0 / 600.0, f64::from(h) * 72.0 / 600.0);
|
||||
page(&mut doc, &[("G", grey)], &format!("q {pw} 0 0 {ph} 0 0 cm /G Do Q"));
|
||||
let report = optimize(&mut doc, &Settings::default()).unwrap();
|
||||
assert_eq!(report.images_resampled, usize::from(resampled), "10,000 × {h}: {report:?}");
|
||||
let s = stream(&doc, grey);
|
||||
if resampled {
|
||||
assert_eq!(s.dict.int(b"Width"), Some(2_500), "10,000 × {h}");
|
||||
} else {
|
||||
assert!(s.dict.int(b"Width") == Some(10_000) && s.raw == before, "10,000 × {h} stays as it is");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Dark bars of irregular lengths across the long side of a `w` × `h` 1-bit image, which Flate
|
||||
/// shrinks less than JBIG2.
|
||||
fn bars(w: u32, h: u32) -> Vec<u8> {
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut data = vec![0xFF; stride * h as usize];
|
||||
let mut state = 0x1234_5678u32;
|
||||
let mut next = move || {
|
||||
state ^= state << 13;
|
||||
state ^= state >> 17;
|
||||
state ^= state << 5;
|
||||
state as usize
|
||||
};
|
||||
let long = w.max(h) as usize;
|
||||
let mut at = 10;
|
||||
while at + 60 < long {
|
||||
let len = 5 + next() % 50;
|
||||
for along in at..at + len {
|
||||
for across in 0..12 {
|
||||
let (x, y) = if w > h { (along, across) } else { (across, along) };
|
||||
data[y * stride + x / 8] &= !(0x80 >> (x % 8));
|
||||
}
|
||||
}
|
||||
at += len + 5 + next() % 50;
|
||||
}
|
||||
data
|
||||
}
|
||||
|
||||
/// Past what the JBIG2 decoder takes on either side, an image packed from more bits, or one
|
||||
/// averaged to a size still past it, isn't stored as JBIG2 either: it becomes 1-bit Flate. Within
|
||||
/// the bound the same pictures do become JBIG2, so the fixtures reach that choice.
|
||||
#[test]
|
||||
fn what_is_too_large_for_jbig2_is_stored_another_way() {
|
||||
let settings = |downsample| Settings {
|
||||
mono: MonoSettings { compression: MonoCompression::Jbig2, downsample, ..MonoSettings::REDUCE },
|
||||
..Settings::default()
|
||||
};
|
||||
let (jbig2, flate) = (&b"JBIG2Decode"[..], &b"FlateDecode"[..]);
|
||||
// Eight-bit grey of only black and white, packed to one bit first, wide and tall.
|
||||
for (w, h, stored) in [(65_535u32, 16u32, jbig2), (65_536, 16, flate), (16, 65_535, jbig2), (16, 65_536, flate)] {
|
||||
let (bits, stride) = (bars(w, h), (w as usize).div_ceil(8));
|
||||
let grey: Vec<u8> = (0..h as usize)
|
||||
.flat_map(|y| (0..w as usize).map(move |x| (y, x)))
|
||||
.map(|(y, x)| if bits[y * stride + x / 8] & (0x80 >> (x % 8)) == 0 { 0 } else { 255 })
|
||||
.collect();
|
||||
let mut doc = Document::new_empty();
|
||||
let image = mono_image(&mut doc, w, h, &grey, None, &[("BitsPerComponent", 8.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", image)], &format!("q {w} 0 0 {h} 0 0 cm /Im Do Q"));
|
||||
optimize(&mut doc, &settings(false)).unwrap();
|
||||
let s = stream(&doc, image);
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(stored), "{w} × {h}, packed");
|
||||
assert_eq!(s.dict.int(b"BitsPerComponent"), Some(1), "{w} × {h}, packed");
|
||||
}
|
||||
// One bit, drawn at 600 ppi and averaged to 300: half the size, so 140,000 is still past it.
|
||||
for (w, h, stored) in [(70_000u32, 16u32, jbig2), (140_000, 16, flate), (16, 70_000, jbig2), (16, 140_000, flate)] {
|
||||
let mut doc = Document::new_empty();
|
||||
let image = mono_image(&mut doc, w, h, &bars(w, h), None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
let (pw, ph) = (f64::from(w) * 72.0 / 600.0, f64::from(h) * 72.0 / 600.0);
|
||||
page(&mut doc, &[("Im", image)], &format!("q {pw} 0 0 {ph} 0 0 cm /Im Do Q"));
|
||||
optimize(&mut doc, &settings(true)).unwrap();
|
||||
let s = stream(&doc, image);
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(stored), "{w} × {h}, averaged");
|
||||
assert_eq!(s.dict.int(b"BitsPerComponent"), Some(1), "{w} × {h}, averaged");
|
||||
assert_eq!((s.dict.int(b"Width"), s.dict.int(b"Height")), (Some(i64::from(w / 2)), Some(i64::from(h / 2))), "{w} × {h}, averaged");
|
||||
}
|
||||
}
|
||||
|
||||
/// An indexed image also drawn where the optimizer doesn't read (an annotation's appearance, where
|
||||
/// other default colour spaces may be in force and change which palette entry is darker) keeps its
|
||||
/// resolution, as black-and-white RGB keeps its colour space; drawn only on the page, it is
|
||||
|
||||
@@ -7439,8 +7439,8 @@ milestone = "M11"
|
||||
status = "partial"
|
||||
commands = ["optimize.advanced", "optimize.reduce"]
|
||||
tools = ["doc_optimize", "doc_reduce"]
|
||||
evidence = ["crates/optimize/src/images.rs::flate_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::jbig2_images_past_the_runs_allowance_stay_as_they_are"]
|
||||
notes = "All of a run's images share one allowance of decoding, filters, decoders and pixels alike (about 1,000 scanned pages' worth, more for a longer file); those past it stay as they are, those the image pass read being counted in the report, which doc_optimize and doc_reduce return and the panel doesn't show yet. CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples."
|
||||
evidence = ["crates/optimize/src/images.rs::flate_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::jbig2_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::an_image_too_large_for_the_jbig2_decoder_stays_as_it_is", "crates/optimize/tests/optimize.rs::what_is_too_large_for_jbig2_is_stored_another_way", "crates/optimize/tests/optimize.rs::group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit"]
|
||||
notes = "All of a run's images share one allowance of decoding, filters, decoders and pixels alike (about 1,000 scanned pages' worth, more for a longer file); those past it stay as they are, those the image pass read being counted in the report, which doc_optimize and doc_reduce return and the panel doesn't show yet. CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: images over 80 million pixels (left alone by the image steps, whatever their settings; Clean Up may still Flate an unfiltered or LZW one), inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples, JBIG2 past 65,535 samples on a side, where the decoder stops: such an image isn't stored as JBIG2 (the image step leaves a 1-bit one as it is unless it is averaged, and may store one packed from more bits as 1-bit Flate)."
|
||||
|
||||
[[feature]]
|
||||
id = "optimize.only-if-smaller"
|
||||
|
||||
Reference in new issue
Block a user