mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-05 18:19:50 +00:00
M11: JBIG2 images share a symbol dictionary, up to 16 at a time (optimize.mono-compression) (#5910)
* M11: JBIG2 images share a symbol dictionary, up to 16 at a time With Monochrome Images set to JBIG2, Optimize stored each image's letters in a symbol dictionary of its own, so a document of N pages carried its font's shapes N times. Now up to 16 images the run stores as JBIG2 share one dictionary in a /JBIG2Globals stream, holding the marks found in more than one of them; each image's text regions refer to it first, then to a dictionary of its own for the marks only it repeats. An image is stored that way when it comes out smaller, and a group only when what its images save is more than the globals take. - filters: encode_jbig2_shared codes a group: the globals are a symbol dictionary segment numbered 0 and associated with no page (7.2.6); each page numbers its segments from 1, as a segment refers only to lower numbers (7.2.5), and the shared symbols' IDs come first, as the dictionaries are referred to in that order (Table 31, SBSYMS). At most 16,383 shared marks, those in the most images, so a page keeps room for its own under the decoder's 32,767. - optimize: the run holds the 1-bit samples of the images it stores as JBIG2 (16 at most, 64 MiB at most) and stores them again sharing, weighing each with the longest reference the globals could have. Eight pages of 10 pt text rendered at 300 dpi take 41% less in Helvetica and 42% in Times than coded alone, 10% in Courier, and 46% in Helvetica at 600 dpi; groups of 2, 4, 16 and 32 save 19%, 33%, 45% and 47%. Redacting one image stores the others of its group again without the dictionary, so 16 keeps that to 15 images. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: test what the mutants found untested in shared JBIG2 dictionaries - filters: the group goes through encode_jbig2_shared, which refuses it for an image the encoder refuses; two marks in common are enough to share; past the 65,536 distinct marks a group holds, a mark first seen later isn't shared while those held count every image. - optimize: optimizing again keeps both the shared pages and the pages coded alone as they are; under Retain, averaged JBIG2 pages share nothing; a last group of two shares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: only images redaction can store again share a JBIG2 dictionary Redacting one image of a group stores the others again without the dictionary, which can hold the shapes of marks under the area, and refuses the redaction for one it can't: it takes image masks and images of one component as its own reading counts them (no DeviceN, the family named directly), and at most 64 Mi pixels. Optimize took DeviceN of one colourant and up to 80 million pixels, so such an image in a group made any redaction of the others fail, where before it was coded alone and the same redaction worked (the Consumers review). Only an image mask or a DeviceGray image named directly, of at most 2^26 pixels, now joins a group; others are coded alone. An engine test optimizes three pages, two DeviceGray and one DeviceN, and redacts the first: the DeviceN page stays alone, the second is stored again, and no image names the dictionary after a save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: optimize and redaction share one limit on the images redaction stores again The second Consumers review: Optimize's limit on images that share a JBIG2 dictionary copied redaction's private 64 Mi, with nothing tying the two, so a lower one in redaction would bring back refused redactions with no test going red. Both now read pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS. The engine test adds an image mask, the other kind that shares, so a redaction that stopped storing either kind again fails it. doc_optimize's description, the README, the parity note and ROADMAP say which images share (named directly). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: a JBIG2 group is coded before its samples would pass the limit CodeRabbit on #5910: hold added an image to the group and then checked SHARED_BYTES, so a group could pass the 64 MiB it is documented to stay within by its last image, which can itself be up to 64 MiB (an image that shares has at most 2^26 pixels, and as its rows are whole bytes, at most that many bytes; a 1-pixel-wide one takes them all). hold_in codes the group first when an image would take it past the limit, and after when it has 16 images; with room for 10 bytes, images of 4 go two to a group, where the old order made one of 12. CodeRabbit's follow-up too: the 18-image test decodes every image and compares it with its samples, across the boundary between the groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * optimize: count the globals' object framing; filters: test the shared coder's size limit - A JBIG2 group shares a dictionary only when it saves more than the globals' dictionary and data plus NEW_OBJECT (96 bytes: the object's framing, its xref entry, and an encrypted file's IV and padding), so a group that saves less than that no longer makes the file larger. - encode_jbig2_shared is Unsupported for an image taller than the decoder takes and codes one at the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filters: drop the unused rows binding in encode_shared page() no longer takes the rows since #5511. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filters: the shared coder's at-bound test says what it returns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
This commit is contained in:
15 files changed
+936
-75
No files matched your search
@@ -223,6 +223,8 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
|
||||
|
||||
## Progress log
|
||||
|
||||
- **2026-11-03 (M11, shared JBIG2 dictionary):** With Monochrome Images set to JBIG2, Optimize has up to 16 JBIG2 images share one symbol dictionary (`/JBIG2Globals`), the letters found on more than one page stored once, when that is smaller (`pdfcraft_filters::encode_jbig2_shared`). Eight pages of 10 pt text rendered at 300 dpi take 41% less in Helvetica and 42% in Times than coded alone, 10% in Courier; groups of 16 save 45%, of 32 only 47%. Only images redaction can store again without the dictionary share one (image masks and DeviceGray named directly, at most 64 Mi pixels), and redacting one stores the rest of its group so, which the group bounds to 15.
|
||||
|
||||
- **2026-11-03 (M5, #5850 review):** Resolve indirect IT when recognizing typewriter contents and Fill & Sign marks. Comment listing and property lookup retain their existing direct-Subtype behavior. A real double-click Edit composer preserves the typewriter’s text as written; ordinary comment edits keep trimming. Regressions cover direct Subtype with indirect intent, save/reopen, raw posted contents and same-index comments on different pages. The existing live-movement test now commits its setup with an outside click rather than Enter. Estimates unchanged.
|
||||
|
||||
- **2026-11-03 (M5, #5850):** Fill & Sign Enter inserts a line break in a growing editor; clicking outside commits and Escape cancels. Nonempty text keeps leading, internal and trailing line breaks. The headless fill_sign_add path sizes its typewriter box for the same wrapped lines so saved appearances show every row. Production-order keyboard and engine save/reopen/render regressions exercise the UI and headless callers. Estimates unchanged.
|
||||
|
||||
@@ -1459,7 +1459,7 @@ pub fn tools() -> Vec<ToolDef> {
|
||||
.with(schema(json!({ "doc": doc() }), &["doc"])),
|
||||
t("doc_open_revision", "Open a revision", "Open saved revision `revision` (1 = the oldest) of a document as a new, unsaved document, to see the file as it was then.")
|
||||
.with(schema(json!({ "doc": doc(), "revision": { "type": "integer", "minimum": 1 } }), &["doc", "revision"])),
|
||||
t("doc_optimize", "Optimize PDF", "Write an optimized copy to `path` (Acrobat's PDF Optimizer). preset: standard (default; Reduce File Size's choices) or mobile (images to 96 ppi above 144, JPEG high quality); the options below change it. compatibility: retain (default; keeps the document's version, or 1.5 when object streams are written), 1.4, 1.5, 1.6, 1.7 or 2.0. merge_fragments (default true) first puts a page picture drawn as many stacked image strips (as some PDF printers write it) back together into one image, so it can be resampled (merged_images, fragments_merged). color / gray: { downsample, ppi, above_ppi, compression: jpeg|zip|retain, quality 1–100 } (defaults: downsample to 150 ppi above 225, JPEG 60). mono: { downsample, ppi, above_ppi, compression: ccitt_g4|jbig2|retain } for black-and-white images and image masks (default: averaged down to 300 ppi above 450 ppi, counted in images_resampled; stored losslessly as CCITT Group 4 or JBIG2 (or 1-bit Flate, once resampled or packed from more bits) when that is smaller, Group 3 fax images, JBIG2 images, black-and-white RGB and indexed images using only palette entries 0 and 1 among them, counted in images_recompressed; retain resamples only, a CCITT image staying CCITT, a JBIG2 image JBIG2 and any other becoming Flate). Images past what one run may decode stay as they are, counted in images_refused; streams it only stores with Flate past that stay LZW, uncounted. Images are measured where pages draw them and replaced only when smaller. discard_*: thumbnails (default true), alternate_images (true), tags, print_settings; flate_unencoded (true); object_streams (default true; always off for 1.4); lzw_to_flate (true) decodes LZW streams and stores them with Flate instead, leaving any it cannot decode unchanged (lzw_streams counts the streams converted); remove_invalid_links and remove_unreferenced_dests (true); images and forms the pages list but never draw are always dropped (unused_xobjects). discard: Remove Hidden Information categories (metadata, attachments, comments, form-fields, hidden-text, hidden-layers, bookmarks, links-actions-scripts, private-data). Signed documents are refused. The open document is unchanged.")
|
||||
t("doc_optimize", "Optimize PDF", "Write an optimized copy to `path` (Acrobat's PDF Optimizer). preset: standard (default; Reduce File Size's choices) or mobile (images to 96 ppi above 144, JPEG high quality); the options below change it. compatibility: retain (default; keeps the document's version, or 1.5 when object streams are written), 1.4, 1.5, 1.6, 1.7 or 2.0. merge_fragments (default true) first puts a page picture drawn as many stacked image strips (as some PDF printers write it) back together into one image, so it can be resampled (merged_images, fragments_merged). color / gray: { downsample, ppi, above_ppi, compression: jpeg|zip|retain, quality 1–100 } (defaults: downsample to 150 ppi above 225, JPEG 60). mono: { downsample, ppi, above_ppi, compression: ccitt_g4|jbig2|retain } for black-and-white images and image masks (default: averaged down to 300 ppi above 450 ppi, counted in images_resampled; stored losslessly as CCITT Group 4 or JBIG2 (or 1-bit Flate, once resampled or packed from more bits) when that is smaller, up to 16 JBIG2 images sharing one symbol dictionary (/JBIG2Globals) where that is smaller still (image masks and DeviceGray images named directly, of at most 64 Mi pixels, which redaction can store again without it), Group 3 fax images, JBIG2 images, black-and-white RGB and indexed images using only palette entries 0 and 1 among them, counted in images_recompressed; retain resamples only, a CCITT image staying CCITT, a JBIG2 image JBIG2 and any other becoming Flate). Images past what one run may decode stay as they are, counted in images_refused; streams it only stores with Flate past that stay LZW, uncounted. Images are measured where pages draw them and replaced only when smaller. discard_*: thumbnails (default true), alternate_images (true), tags, print_settings; flate_unencoded (true); object_streams (default true; always off for 1.4); lzw_to_flate (true) decodes LZW streams and stores them with Flate instead, leaving any it cannot decode unchanged (lzw_streams counts the streams converted); remove_invalid_links and remove_unreferenced_dests (true); images and forms the pages list but never draw are always dropped (unused_xobjects). discard: Remove Hidden Information categories (metadata, attachments, comments, form-fields, hidden-text, hidden-layers, bookmarks, links-actions-scripts, private-data). Signed documents are refused. The open document is unchanged.")
|
||||
.cmd("optimize.advanced")
|
||||
.with(schema(
|
||||
json!({
|
||||
|
||||
@@ -25,6 +25,12 @@ pub const MAX_PAGE_DECODED: usize = 256 << 20;
|
||||
/// Form XObjects one page's content walk may enter, each entry counted again.
|
||||
pub const MAX_PAGE_FORMS: usize = 50_000;
|
||||
|
||||
/// The most pixels an image redaction decodes, to clear the ones under a mark or to store again
|
||||
/// without the `/JBIG2Globals` it shares with a redacted image (`pdfcraft_redact`). Optimize has
|
||||
/// JBIG2 images share a dictionary only within it (`pdfcraft_optimize`), so that redaction can
|
||||
/// store each of them again: one it couldn't would refuse the redaction.
|
||||
pub const MAX_REDACTED_IMAGE_PIXELS: u64 = 64 << 20;
|
||||
|
||||
/// The steps a sample of a JPEG costs: a few bytes can hold a page of one grey, and decoding,
|
||||
/// resampling and storing it took about 20 to 45 ns a sample.
|
||||
const JPEG_WORK: u64 = 2;
|
||||
|
||||
@@ -22,7 +22,7 @@ mod parser;
|
||||
mod security;
|
||||
mod writer;
|
||||
|
||||
pub use budget::{DecodeBudget, ImageBudget, MAX_PAGE_DECODED, MAX_PAGE_FORMS};
|
||||
pub use budget::{DecodeBudget, ImageBudget, MAX_PAGE_DECODED, MAX_PAGE_FORMS, MAX_REDACTED_IMAGE_PIXELS};
|
||||
pub use bytes::Bytes;
|
||||
pub use document::{Document, ReadState, Revision, SourceIdentity, XrefEntry};
|
||||
pub use object::{Dict, MAX_DECODED, MAX_JBIG2_GLOBALS, Name, ObjRef, Object, PdfString, Stream};
|
||||
|
||||
@@ -10252,6 +10252,107 @@ fn image_transform_engine_refuses_singular_matrices_without_mutation() {
|
||||
}
|
||||
}
|
||||
|
||||
/// Optimize has JBIG2 images share a symbol dictionary only where redaction can store each again
|
||||
/// without it. Of four pages of the same scan, the two in DeviceGray and the image mask share one;
|
||||
/// the one in a DeviceN of one colourant, which redaction doesn't store again, is coded alone.
|
||||
/// Redacting the first page then stores the second and the mask again without the dictionary,
|
||||
/// which leaves the file.
|
||||
#[test]
|
||||
fn redacting_an_optimized_jbig2_page_stores_the_rest_of_its_group_again() {
|
||||
use pdfcraft_cos::{Dict, Document, ObjRef, Object, SaveOptions, Stream, write_full};
|
||||
let (w, h) = (600usize, 480usize);
|
||||
// Lines of 26 letters, each 32 rows of a run from a stem down its left side, of 2 to 24 pixels,
|
||||
// the same to the pixel wherever it is, picked and spaced at random.
|
||||
let mut seed = 0x2545_F491_4F6C_DD1Du64;
|
||||
let mut next = move || {
|
||||
seed ^= seed << 13;
|
||||
seed ^= seed >> 7;
|
||||
seed ^= seed << 17;
|
||||
seed as usize
|
||||
};
|
||||
let letters: Vec<Vec<usize>> = (0..26).map(|_| (0..16).map(|_| 1 + next() % 12).collect()).collect();
|
||||
let stride = w.div_ceil(8);
|
||||
let mut bits = vec![0xFFu8; stride * h];
|
||||
for top in (8..h - 40).step_by(44) {
|
||||
let mut x = 8 + next() % 8;
|
||||
while x + 32 < w {
|
||||
let letter = &letters[next() % 26];
|
||||
for (row, run) in letter.iter().enumerate() {
|
||||
for (px, py) in (x..x + 2 * run).flat_map(|px| [(px, top + 2 * row), (px, top + 2 * row + 1)]) {
|
||||
bits[py * stride + px / 8] &= !(0x80 >> (px % 8));
|
||||
}
|
||||
}
|
||||
x += 2 * letter.iter().max().unwrap_or(&1) + 4 + next() % 5;
|
||||
}
|
||||
}
|
||||
let mut doc = Document::new_empty();
|
||||
let pages = doc.root().and_then(|r| doc.get(r).as_dict().and_then(|d| d.reference(b"Pages"))).unwrap();
|
||||
let device_n = Object::Array(vec![Object::name("DeviceN"), Object::Array(vec![Object::name("Black")]), Object::name("DeviceGray"), {
|
||||
let mut tint = Dict::new();
|
||||
tint.set(b"FunctionType".to_vec(), Object::Int(2));
|
||||
tint.set(b"Domain".to_vec(), Object::Array(vec![Object::Int(0), Object::Int(1)]));
|
||||
tint.set(b"N".to_vec(), Object::Int(1));
|
||||
Object::Dict(tint)
|
||||
}]);
|
||||
let mut images: Vec<ObjRef> = Vec::new();
|
||||
let grey = || ("ColorSpace", Object::name("DeviceGray"));
|
||||
for (k, (key, value)) in [grey(), grey(), ("ColorSpace", device_n), ("ImageMask", Object::Bool(true))].into_iter().enumerate() {
|
||||
let mut d = Dict::new();
|
||||
d.set(b"Type".to_vec(), Object::name("XObject"));
|
||||
d.set(b"Subtype".to_vec(), Object::name("Image"));
|
||||
d.set(b"Width".to_vec(), Object::Int(w as i64));
|
||||
d.set(b"Height".to_vec(), Object::Int(h as i64));
|
||||
d.set(b"BitsPerComponent".to_vec(), Object::Int(1));
|
||||
d.set(key.as_bytes().to_vec(), value);
|
||||
let image = doc.add(Object::Stream(Stream::flate(d, &bits)));
|
||||
images.push(image);
|
||||
let content = doc.add(Object::Stream(Stream::from_raw(Dict::new(), b"q 144 0 0 115.2 36 600 cm /Im Do Q".to_vec())));
|
||||
let mut xobjects = Dict::new();
|
||||
xobjects.set(b"Im".to_vec(), Object::Ref(image));
|
||||
let mut resources = Dict::new();
|
||||
resources.set(b"XObject".to_vec(), Object::Dict(xobjects));
|
||||
let mut page = Dict::new();
|
||||
page.set(b"Type".to_vec(), Object::name("Page"));
|
||||
page.set(b"Parent".to_vec(), Object::Ref(pages));
|
||||
page.set(b"MediaBox".to_vec(), Object::Array(vec![0.into(), 0.into(), 612.into(), 792.into()]));
|
||||
page.set(b"Resources".to_vec(), Object::Dict(resources));
|
||||
page.set(b"Contents".to_vec(), Object::Ref(content));
|
||||
if k == 0 {
|
||||
let mut mark = Dict::new();
|
||||
mark.set(b"Type".to_vec(), Object::name("Annot"));
|
||||
mark.set(b"Subtype".to_vec(), Object::name("Redact"));
|
||||
mark.set(b"Rect".to_vec(), Object::Array(vec![40.into(), 610.into(), 100.into(), 650.into()]));
|
||||
let mark = doc.add(Object::Dict(mark));
|
||||
page.set(b"Annots".to_vec(), Object::Array(vec![Object::Ref(mark)]));
|
||||
}
|
||||
let page = doc.add(Object::Dict(page));
|
||||
doc.update_dict(pages, |d| {
|
||||
let mut kids = d.get(b"Kids").and_then(|k| k.as_array().cloned()).unwrap_or_default();
|
||||
kids.push(Object::Ref(page));
|
||||
d.set(b"Count".to_vec(), Object::Int(kids.len() as i64));
|
||||
d.set(b"Kids".to_vec(), Object::Array(kids));
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
let mono = pdfcraft_optimize::MonoSettings {
|
||||
compression: pdfcraft_optimize::MonoCompression::Jbig2,
|
||||
downsample: false,
|
||||
..pdfcraft_optimize::MonoSettings::REDUCE
|
||||
};
|
||||
pdfcraft_optimize::optimize(&mut doc, &pdfcraft_optimize::Settings { mono, ..Default::default() }).unwrap();
|
||||
let globals = |doc: &Document, image: ObjRef| match &*doc.get(image) {
|
||||
Object::Stream(s) => s.dict.get(b"DecodeParms").and_then(Object::as_dict).and_then(|d| d.reference(b"JBIG2Globals")),
|
||||
_ => None,
|
||||
};
|
||||
let shared = globals(&doc, images[0]);
|
||||
assert!(shared.is_some() && globals(&doc, images[1]) == shared && globals(&doc, images[3]) == shared, "DeviceGray and the mask share");
|
||||
assert_eq!(globals(&doc, images[2]), None, "the DeviceN page is coded alone");
|
||||
pdfcraft_redact::apply(&mut doc, None).unwrap();
|
||||
let saved = Document::open(Arc::new(write_full(&doc, &SaveOptions::default()).unwrap())).unwrap();
|
||||
let named = saved.scan_objects().filter(|(_, o)| matches!(&**o, Object::Stream(s) if saved.jbig2_globals_ref(s).is_some())).count();
|
||||
assert_eq!(named, 0, "no image names the dictionary, which left with them");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fill_sign_multiline_text_survives_engine_save_reopen_and_render() {
|
||||
let (mut session, id) = rotated_blank(0);
|
||||
|
||||
@@ -31,6 +31,7 @@ decode_ccitt_within(data, ¶ms, max_rows, work: &mut u64) -> Result<Vec<u8>,
|
||||
decode_jbig2_within(data, globals, columns, rows, max, work: &mut u64) -> Result<Vec<u8>, FilterError> // JBIG2 as JBIG2Decode gives it; a page of columns × rows, within max and MAX_JBIG2_PIXELS; reading and decoding it spend from work, in the decoder's steps
|
||||
encode_jbig2_generic(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode; columns and rows <= MAX_JBIG2_COLUMNS (65,535) and MAX_JBIG2_PIXELS (2^28) samples, what the decoder takes
|
||||
encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit rows as lossless JBIG2 for JBIG2Decode, symbol coded when that is smaller; the same limits
|
||||
encode_jbig2_shared(&[(data, columns, rows)]) -> Result<Option<SharedJbig2>, FilterError> // images symbol coded sharing one dictionary: the /JBIG2Globals data, and each image's stream or None; the same limits
|
||||
```
|
||||
|
||||
- `chain` is applied first to last, as in a `/Filter` array.
|
||||
@@ -51,7 +52,7 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
| ASCII85Decode | `<~` prefix, `z`, white space, `~>` (or bare `~`, or none) EOD, final partial group; `z` inside a group, a 1-char final group, overflow or bad characters → Corrupt | `z` for zero groups, newline every 75 characters, `~>` |
|
||||
| RunLengthDecode | 0–127 literal, 129–255 repeat, 128 EOD (optional); truncated run → Corrupt | runs of ≥2 repeat, literals up to 128, 128 |
|
||||
| CCITTFaxDecode | not in a chain (`Unsupported`); `decode_ccitt_within` decodes Group 3 and Group 4 on its own, as rows of 1-bit samples each on a new byte, black 0 unless `black_is_1`, at most `max_rows` rows (fewer with `/Rows`), the rows before any damage, up to 1,048,576 samples a row; Group 3 rows on bytes are read by their end-of-line codes when the data starts with one and on bytes otherwise, whatever `end_of_line` says, so data with codes but none before the first row (not as T.4 sends them) is read on bytes, as the renderer draws it, and at `k` 0 may come out as other rows (`pdfcraft_cos::ImageBudget::decode` calls it for a fax image, after the filters before it) | `encode_g4`: Group 4 (`/K -1`) with an end-of-block mark, 0 bits coded black so `/BlackIs1 false` decodes the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 32,767 samples a row (the encoder counts positions in 16 bits); no rows or columns, or too wide → Unsupported, too few bytes → Corrupt |
|
||||
| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2_within` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::ImageBudget::decode` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 65,535 samples on a side and 2^28 in all, where `hayro-jbig2` stops; no rows or columns, or too large → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made |
|
||||
| JBIG2Decode | not in a chain (`Unsupported`); `decode_jbig2_within` decodes it on its own with `hayro-jbig2`, given what its `/JBIG2Globals` decode to, as rows of 1-bit samples each on a new byte, black 0; the page has to be `columns` × `rows` (PdfCraft's renderer draws the whole page into the image), within `max` bytes as such rows and 2^28 pixels (`MAX_JBIG2_PIXELS`, the most the decoder takes), checked before it is decoded; regions inside it are bounded by the decoder alone (a region may claim more pixels than its page); damaged data, or globals that don't decode, are `Corrupt` (`pdfcraft_cos::ImageBudget::decode` calls it for a JBIG2 image, after the filters before it) | `encode_jbig2_generic`: the embedded organisation (D.3) with a page information segment and an immediate lossless generic region (7.4.6), no `/JBIG2Globals`; template 0 with its nominal AT pixels and typical prediction (6.2.5), coded with the MQ arithmetic coder (Annex E, Table E.1) and ended by its 0xFF 0xAC marker. A 0 sample is coded as a JBIG2 1 (black), since `JBIG2Decode` gives black as 0, so the stream decodes to the same bits; rows packed MSB first, each on a new byte, padding bits and trailing bytes ignored; up to 65,535 samples on a side and 2^28 in all, where `hayro-jbig2` stops; no rows or columns, or too large → Unsupported, too few bytes → Corrupt. `encode_jbig2`: the smaller of that and symbol coding (`jbig2/symbols.rs`): each distinct mark (8-connected black pixels, at most 256 a side) stored once in a symbol dictionary (7.4.2; arithmetic coded, each bitmap a template 0 generic region without typical prediction, the bitmaps sharing their contexts) and placed by immediate lossless text regions referring to it (7.4.3; bottom-left reference corner, strips of 1, 2, 4 or 8 rows, whichever is smallest), each of at most 10,000 instances over a band of rows, as the decoder PdfCraft renders with (`hayro-jbig2`) takes no more in one region, with larger marks, and in a second plan marks that occur once, in a generic region over their box. Only exact copies share a symbol (no refinement), so the stream decodes to the same bits; fewer than two symbols, more than 2^20 black runs, 65,536 distinct marks or 16 MiB of their pixels, and the generic coding is used; a plan of more than 32,767 symbols (that decoder refuses more) isn't made. `encode_jbig2_shared`: the marks found in more than one image of a group (at most 16,383, those in the most images first, within the same limits on distinct marks and their pixels as one image's) go once in a symbol dictionary segment associated with no page, number 0, which is the `/JBIG2Globals` data (7.2.6, D.3); each image's stream numbers its segments from 1 (a segment refers only to lower numbers, 7.2.5) and its text regions refer to that dictionary first, then to one of its own for the marks only it repeats, so the shared symbols' IDs come first (Table 31's SBSYMS); the rest is coded as `encode_jbig2`'s symbol coding codes it, the smaller of its two plans. An image that places none of the shared marks gets no stream, to be stored alone; whether the rest save more than the globals take is the caller's to weigh |
|
||||
|
||||
## Tests
|
||||
|
||||
@@ -62,7 +63,7 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
- `mq.rs`: the arithmetic coder reproduces Annex H.2's 30 bytes from its 256 decisions.
|
||||
- `jbig2.rs`: output decoded by `hayro-jbig2` (dev-dependency, an independent decoder) matches the input bits: a text-like page (smaller than Group 4), edges, padding, all-white and all-black regions (typical rows), and property tests over bitmaps of noise and of runs; any input returns rather than panics.
|
||||
- `jbig2/decode.rs`: `decode_jbig2_within` gives back the rows of both codings, padding bits 0; refuses a page of another width or height, past `max` or past `MAX_JBIG2_PIXELS` before decoding it (2^28 pixels take about 34 MB, a bit each), globals that aren't JBIG2 and damaged data, and returns rather than panics on every prefix of a stream.
|
||||
- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; any input returns rather than panics.
|
||||
- `jbig2/symbols.rs`: the integer coders against Annex A's example and Table A.1; symbol-coded output decoded by `hayro-jbig2` matches the input bits for every plan and strip size: text whose letters repeat (under half the generic coding's size, the dictionary retained and referred to), large and lone marks left to a generic region, marks on the edges, touching at corners and inside each other's boxes, and property tests over glyphs scattered over each other and the edges with stray pixels (never larger than the generic coding); one shape, a blank page and noise are coded generically; marks joined at a corner, either way, are one mark, and 256 pixels a side is a symbol while 257 wide or tall is left out; black padding bits are ignored; coordinates past 4,435 use Annex A's 32-bit range and PREV stays below 512; past 2^20 runs, 65,536 distinct marks or 2^24 symbol pixels the image is coded generically, and at each cap it is still looked at; a plan codes at most 32,767 symbols, which decode, and past that the second plan still codes the repeated ones; more than 10,000 marks are placed by several text regions, numbered in turn, the dictionary retained for each but the last; pages in one font share one dictionary, segment 0 of no page, retained, which each page's text regions refer to first, taking less together than alone, a page's own repeated marks in a dictionary referred to after it, a page sharing nothing left alone, images with fewer than two marks in common sharing nothing, and past 16,383 shared marks those in the most images kept, with a property test over groups of scattered pages; any input returns rather than panics.
|
||||
- `fuzz/`: a detached cargo-fuzz crate (`cargo +nightly fuzz run decode`, run from `crates/filters/`).
|
||||
|
||||
## Status
|
||||
|
||||
+75
-18
@@ -1,8 +1,9 @@
|
||||
//! JBIG2 encoding (ITU-T T.88 | ISO/IEC 14492), for `JBIG2Decode` (ISO 32000-2 §7.4.7), lossless:
|
||||
//! a generic region, template 0 with its nominal adaptive pixels, typical prediction on, MQ
|
||||
//! arithmetic coding ([`crate::mq`]); or the image's marks as symbols ([`symbols`]). Written from
|
||||
//! the standard's 6.2.5 (coding a generic region), 7.2 (segment headers), 7.4.1, 7.4.6 and 7.4.8
|
||||
//! (the segments) and D.3 (the embedded organisation PDF uses). Decoding is in [`decode`].
|
||||
//! arithmetic coding ([`crate::mq`]); or the image's marks as symbols ([`symbols`]), which a group
|
||||
//! of images may share through `/JBIG2Globals`. Written from the standard's 6.2.5 (coding a generic
|
||||
//! region), 7.2 (segment headers), 7.4.1, 7.4.6 and 7.4.8 (the segments) and D.3 (the embedded
|
||||
//! organisation PDF uses). Decoding is in [`decode`].
|
||||
|
||||
mod decode;
|
||||
mod symbols;
|
||||
@@ -56,9 +57,7 @@ const PAD: usize = 4;
|
||||
/// Errors: as [`encode_jbig2_generic`].
|
||||
pub fn encode_jbig2(data: &[u8], columns: u32, rows: u32) -> Result<Vec<u8>, FilterError> {
|
||||
let generic = encode_jbig2_generic(data, columns, rows)?;
|
||||
// `encode_jbig2_generic` checked that the rows are there.
|
||||
let stride = (columns as usize).div_ceil(8);
|
||||
let samples = usize::try_from(rows).ok().and_then(|r| r.checked_mul(stride)).and_then(|n| data.get(..n));
|
||||
let samples = rows_to_encode(data, columns, rows).ok();
|
||||
Ok(match samples.and_then(|s| symbols::encode(s, columns, rows)) {
|
||||
Some(coded) if coded.len() < generic.len() => coded,
|
||||
_ => generic,
|
||||
@@ -78,16 +77,9 @@ pub fn encode_jbig2(data: &[u8], columns: u32, rows: u32) -> Result<Vec<u8>, Fil
|
||||
/// [`MAX_JBIG2_PIXELS`], what the decoder takes ([`FilterError::Unsupported`]); fewer bytes than the
|
||||
/// rows need ([`FilterError::Corrupt`]).
|
||||
pub fn encode_jbig2_generic(data: &[u8], columns: u32, rows: u32) -> Result<Vec<u8>, FilterError> {
|
||||
let too_large = columns > MAX_JBIG2_COLUMNS || rows > MAX_JBIG2_COLUMNS || u64::from(columns) * u64::from(rows) > MAX_JBIG2_PIXELS;
|
||||
if columns == 0 || rows == 0 || too_large {
|
||||
return Err(FilterError::Unsupported(format!("JBIG2 encoding of {columns} × {rows} samples")));
|
||||
}
|
||||
let samples = rows_to_encode(data, columns, rows)?;
|
||||
let width = columns as usize;
|
||||
let stride = width.div_ceil(8);
|
||||
let samples = usize::try_from(rows).ok().and_then(|r| r.checked_mul(stride)).and_then(|n| data.get(..n)).ok_or_else(|| FilterError::Corrupt {
|
||||
filter: "JBIG2Decode",
|
||||
detail: format!("{} bytes for {rows} rows of {columns} samples", data.len()),
|
||||
})?;
|
||||
let mut contexts: Vec<Context> = vec![0; 1 << 16];
|
||||
let mut mq = MqEncoder::new();
|
||||
let height = samples.len() / stride;
|
||||
@@ -105,6 +97,48 @@ pub fn encode_jbig2_generic(data: &[u8], columns: u32, rows: u32) -> Result<Vec<
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// The samples of `rows` rows of `columns` 1-bit samples in `data`, when the encoders here take
|
||||
/// that size; the errors are [`encode_jbig2_generic`]'s.
|
||||
fn rows_to_encode(data: &[u8], columns: u32, rows: u32) -> Result<&[u8], FilterError> {
|
||||
let too_large = columns > MAX_JBIG2_COLUMNS || rows > MAX_JBIG2_COLUMNS || u64::from(columns) * u64::from(rows) > MAX_JBIG2_PIXELS;
|
||||
if columns == 0 || rows == 0 || too_large {
|
||||
return Err(FilterError::Unsupported(format!("JBIG2 encoding of {columns} × {rows} samples")));
|
||||
}
|
||||
let stride = (columns as usize).div_ceil(8);
|
||||
usize::try_from(rows)
|
||||
.ok()
|
||||
.and_then(|r| r.checked_mul(stride))
|
||||
.and_then(|n| data.get(..n))
|
||||
.ok_or_else(|| FilterError::Corrupt { filter: "JBIG2Decode", detail: format!("{} bytes for {rows} rows of {columns} samples", data.len()) })
|
||||
}
|
||||
|
||||
/// A group of images coded by [`encode_jbig2_shared`]: the data of the `/JBIG2Globals` stream
|
||||
/// they share, and each image's own stream, in the order given, or `None` for one that shares
|
||||
/// nothing with the others.
|
||||
#[derive(Debug)]
|
||||
pub struct SharedJbig2 {
|
||||
pub globals: Vec<u8>,
|
||||
pub images: Vec<Option<Vec<u8>>>,
|
||||
}
|
||||
|
||||
/// Images, each `rows` rows of `columns` 1-bit PDF samples, as lossless JBIG2 sharing one symbol
|
||||
/// dictionary: the marks found in more than one of them are stored once, in a symbol dictionary
|
||||
/// segment not associated with a page, which is the `/JBIG2Globals` stream's data, and each image's
|
||||
/// stream, in the embedded organisation, places them from there, with its other marks coded as
|
||||
/// [`encode_jbig2`]'s symbol coding codes them (see [`symbols`]). With the globals, each decodes to
|
||||
/// its own bits. An image that holds none of those marks, or that symbol coding doesn't take, has
|
||||
/// no stream here; it is stored alone. `None` when the images share fewer than two marks.
|
||||
///
|
||||
/// What each image's stream saves is left for the caller to weigh against what the globals take,
|
||||
/// which the images that use them share.
|
||||
///
|
||||
/// Errors: as [`encode_jbig2_generic`], for the first image it would refuse.
|
||||
pub fn encode_jbig2_shared(images: &[(&[u8], u32, u32)]) -> Result<Option<SharedJbig2>, FilterError> {
|
||||
let images =
|
||||
images.iter().map(|(data, columns, rows)| Ok((rows_to_encode(data, *columns, *rows)?, *columns, *rows))).collect::<Result<Vec<_>, _>>()?;
|
||||
Ok(symbols::encode_shared(&images).map(|(globals, images)| SharedJbig2 { globals, images }))
|
||||
}
|
||||
|
||||
/// A page information segment's data (7.4.8): the page's size, resolution unknown, eventually
|
||||
/// lossless, default pixel 0 (white), OR combination, not striped.
|
||||
fn page_information(columns: u32, rows: u32) -> Vec<u8> {
|
||||
@@ -145,12 +179,18 @@ fn at_pixels(out: &mut Vec<u8>) {
|
||||
}
|
||||
}
|
||||
|
||||
/// A segment (7.2): its header (number, type, the segments it refers to, page 1, data length),
|
||||
/// then its data. `retained`: a later segment refers to this one. `refers_to` holds each
|
||||
/// A segment (7.2) of page 1: its header (number, type, the segments it refers to, the page, data
|
||||
/// length), then its data. `retained`: a later segment refers to this one. `refers_to` holds each
|
||||
/// referred-to segment's number and whether a segment after this one refers to it too (its
|
||||
/// retain bit, 7.2.4). Segment numbers here stay below 256, so each referred-to one takes a byte
|
||||
/// (7.2.5); a segment refers to at most four, so their count and retain bits take one byte too.
|
||||
fn segment(out: &mut Vec<u8>, number: u32, kind: u8, refers_to: &[(u8, bool)], retained: bool, data: &[u8]) {
|
||||
segment_of(out, 1, number, kind, refers_to, retained, data);
|
||||
}
|
||||
|
||||
/// [`segment`], of page `page`: 0 for a segment associated with no page (7.2.6), such as the
|
||||
/// global ones `/JBIG2Globals` holds, which a page's segments may refer to.
|
||||
fn segment_of(out: &mut Vec<u8>, page: u8, number: u32, kind: u8, refers_to: &[(u8, bool)], retained: bool, data: &[u8]) {
|
||||
out.extend_from_slice(&number.to_be_bytes());
|
||||
out.push(kind);
|
||||
// At most four, so the count fits its three bits.
|
||||
@@ -158,7 +198,7 @@ fn segment(out: &mut Vec<u8>, number: u32, kind: u8, refers_to: &[(u8, bool)], r
|
||||
let kept = refers_to.iter().enumerate().fold(0u8, |bits, (k, (_, kept))| bits | u8::from(*kept) << (k + 1));
|
||||
out.push((refers_to.len() as u8) << 5 | kept | u8::from(retained));
|
||||
out.extend(refers_to.iter().map(|(n, _)| *n));
|
||||
out.push(1);
|
||||
out.push(page);
|
||||
// A segment's data never comes near 4 GiB here: an image is at most 2^28 samples.
|
||||
out.extend_from_slice(&u32::try_from(data.len()).unwrap_or(u32::MAX).to_be_bytes());
|
||||
out.extend_from_slice(data);
|
||||
@@ -216,6 +256,11 @@ mod tests {
|
||||
/// The rows an independent decoder (`hayro-jbig2`) reads from `data`, as PDF samples: true
|
||||
/// for white (a JBIG2 0).
|
||||
pub(super) fn decode(data: &[u8]) -> Vec<Vec<bool>> {
|
||||
decode_with(data, None)
|
||||
}
|
||||
|
||||
/// [`decode`], with `/JBIG2Globals`.
|
||||
pub(super) fn decode_with(data: &[u8], globals: Option<&[u8]>) -> Vec<Vec<bool>> {
|
||||
struct Rows(Vec<Vec<bool>>, Vec<bool>, usize);
|
||||
impl hayro_jbig2::Decoder for Rows {
|
||||
fn push_pixel(&mut self, black: bool) {
|
||||
@@ -230,7 +275,7 @@ mod tests {
|
||||
self.0.push(row);
|
||||
}
|
||||
}
|
||||
let image = hayro_jbig2::Image::new_embedded(data, None).unwrap();
|
||||
let image = hayro_jbig2::Image::new_embedded(data, globals).unwrap();
|
||||
let mut rows = Rows(Vec::new(), Vec::new(), image.width() as usize);
|
||||
image.decode(&mut rows).unwrap();
|
||||
rows.0
|
||||
@@ -353,6 +398,18 @@ mod tests {
|
||||
|
||||
/// The encoder stops where the decoder does, on each side and in total: past that it would
|
||||
/// write a stream nothing here can read. 16,384 squared is exactly the bound on samples.
|
||||
/// `encode_jbig2_shared` refuses what `encode_jbig2_generic` refuses: an image taller than
|
||||
/// the decoder takes is `Unsupported`, one at the bound isn't refused. (Its one tall mark is
|
||||
/// too large to be a symbol, so nothing is shared: `Ok(None)`.)
|
||||
#[test]
|
||||
fn shared_coding_takes_only_what_the_decoder_takes() {
|
||||
let past = vec![0x7F; MAX_JBIG2_COLUMNS as usize + 1];
|
||||
let tall = (&past[..], 1, MAX_JBIG2_COLUMNS + 1);
|
||||
assert!(matches!(encode_jbig2_shared(&[tall, tall]), Err(FilterError::Unsupported(_))), "too tall");
|
||||
let at = (&past[..MAX_JBIG2_COLUMNS as usize], 1, MAX_JBIG2_COLUMNS);
|
||||
assert!(matches!(encode_jbig2_shared(&[at, at]), Ok(None)), "at the bound");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn it_encodes_only_what_the_decoder_takes() {
|
||||
let unsupported = |columns, rows| matches!(encode_jbig2_generic(&[], columns, rows), Err(FilterError::Unsupported(_)));
|
||||
|
||||
@@ -11,7 +11,7 @@ use std::collections::HashMap;
|
||||
use std::hash::{DefaultHasher, Hash, Hasher};
|
||||
use std::ops::Range;
|
||||
|
||||
use super::{at_pixels, code_generic, generic_region, page_information, region_information, segment};
|
||||
use super::{at_pixels, code_generic, generic_region, page_information, region_information, segment, segment_of};
|
||||
use crate::mq::{Context, MqEncoder};
|
||||
|
||||
/// The longest side of a mark stored as a symbol. A larger one is left to the generic region.
|
||||
@@ -37,10 +37,14 @@ const MAX_CODED_SYMBOLS: usize = (1 << 15) - 1;
|
||||
/// placed by several regions, each over a band of rows, all referring to the one dictionary.
|
||||
const MAX_REGION_INSTANCES: usize = 10_000;
|
||||
|
||||
// Segment numbers stay at most 256, so a referred-to one takes a byte (7.2.5): the page, the
|
||||
// dictionary, a text region for each MAX_REGION_INSTANCES of at most MAX_RUNS marks, and the
|
||||
// generic region.
|
||||
const _: () = assert!(MAX_RUNS.div_ceil(MAX_REGION_INSTANCES) + 3 <= 256);
|
||||
/// The most symbols a group's shared dictionary holds ([`encode_shared`]): half of what a page may
|
||||
/// place, so each page keeps room for its own.
|
||||
const MAX_SHARED_SYMBOLS: usize = MAX_CODED_SYMBOLS / 2;
|
||||
|
||||
// Segment numbers stay at most 256, so a referred-to one takes a byte (7.2.5): the shared
|
||||
// dictionary, the page, its own dictionary, a text region for each MAX_REGION_INSTANCES of at most
|
||||
// MAX_RUNS marks, and the generic region.
|
||||
const _: () = assert!(MAX_RUNS.div_ceil(MAX_REGION_INSTANCES) + 4 <= 256);
|
||||
|
||||
/// The integer coders' ranges (Table A.1): how many 1 bits pick the range, how many bits give the
|
||||
/// offset into it, and its first value. The last range has no 0 after its prefix.
|
||||
@@ -97,13 +101,107 @@ struct Instance {
|
||||
height: u32,
|
||||
}
|
||||
|
||||
/// A group of images coded by [`encode_shared`]: the globals' data, and each image's stream.
|
||||
type SharedCoding = (Vec<u8>, Vec<Option<Vec<u8>>>);
|
||||
|
||||
/// The symbols of a group's shared dictionary, as one image's [`plan`] places them: the ID each of
|
||||
/// the image's symbols has there, if it is there, and how many symbols that dictionary exports.
|
||||
struct Shared {
|
||||
ids: Vec<Option<u32>>,
|
||||
count: u32,
|
||||
}
|
||||
|
||||
/// `rows` rows of `columns` 1-bit PDF samples (black 0), symbol coded: the smaller of two plans,
|
||||
/// every mark of at most [`MAX_SYMBOL_SIDE`] a side as a symbol, or only those with a copy.
|
||||
/// `None` when neither plan has from two to [`MAX_CODED_SYMBOLS`] symbols, or the image is too
|
||||
/// large or too busy to look at.
|
||||
pub(super) fn encode(samples: &[u8], columns: u32, rows: u32) -> Option<Vec<u8>> {
|
||||
let page = page(samples, columns)?;
|
||||
[1, 2].into_iter().filter_map(|copies| plan(&page, columns, rows, copies, None)).min_by_key(Vec::len)
|
||||
[1, 2].into_iter().filter_map(|copies| plan(&page, columns, rows, copies, None, None)).min_by_key(Vec::len)
|
||||
}
|
||||
|
||||
/// Images, each as [`encode`] takes one, symbol coded sharing one dictionary: the globals' data, a
|
||||
/// symbol dictionary segment associated with no page (number 0) holding the distinct marks found
|
||||
/// in more than one image, and each image's stream placing them from there, the smaller of
|
||||
/// [`encode`]'s two plans for its other marks. The group's marks are held within the limits one
|
||||
/// image's are ([`MAX_SYMBOLS`], [`MAX_SYMBOL_PIXELS`]): past them, later marks aren't shared. Past
|
||||
/// [`MAX_SHARED_SYMBOLS`] shared marks, those in the most images are kept.
|
||||
///
|
||||
/// An image too large or too busy to look at, or that places none of the shared marks, gets `None`;
|
||||
/// `None` overall when fewer than two marks are shared.
|
||||
pub(super) fn encode_shared(images: &[(&[u8], u32, u32)]) -> Option<SharedCoding> {
|
||||
// The group's distinct marks, each with how many images hold it (as its uses), looked at an
|
||||
// image at a time.
|
||||
let mut table: Vec<Symbol> = Vec::new();
|
||||
let mut by_hash: HashMap<u64, Vec<u32>> = HashMap::new();
|
||||
let mut held = 0usize;
|
||||
for (samples, columns, _) in images {
|
||||
let Some(page) = page(samples, *columns) else { continue };
|
||||
// An image's symbols are distinct, so each counts an image once.
|
||||
for symbol in page.symbols {
|
||||
match find(&table, &by_hash, &symbol) {
|
||||
Some(t) => {
|
||||
if let Some(s) = table.get_mut(t as usize) {
|
||||
s.uses = s.uses.saturating_add(1);
|
||||
}
|
||||
}
|
||||
None => {
|
||||
let Some(more) = held.checked_add(symbol.pixels.len()).filter(|h| *h <= MAX_SYMBOL_PIXELS && table.len() < MAX_SYMBOLS) else {
|
||||
continue;
|
||||
};
|
||||
held = more;
|
||||
let t = u32::try_from(table.len()).ok()?;
|
||||
by_hash.entry(fingerprint(&symbol)).or_default().push(t);
|
||||
table.push(Symbol { uses: 1, ..symbol });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let uses = |t: &u32| table.get(*t as usize).map_or(0, |s| s.uses);
|
||||
let mut shared: Vec<u32> = (0..u32::try_from(table.len()).ok()?).filter(|t| uses(t) >= 2).collect();
|
||||
if shared.len() < 2 {
|
||||
return None;
|
||||
}
|
||||
shared.sort_by_key(|t| std::cmp::Reverse(uses(t)));
|
||||
shared.truncate(MAX_SHARED_SYMBOLS);
|
||||
// The dictionary's order, as `plan` keeps it.
|
||||
shared.sort_by_key(|t| table.get(*t as usize).map(|s| (s.height, s.width)));
|
||||
let mut global = vec![None; table.len()];
|
||||
for (k, t) in shared.iter().enumerate() {
|
||||
if let Some(slot) = global.get_mut(*t as usize) {
|
||||
*slot = u32::try_from(k).ok();
|
||||
}
|
||||
}
|
||||
let symbols: Vec<&Symbol> = shared.iter().filter_map(|t| table.get(*t as usize)).collect();
|
||||
let mut globals = Vec::new();
|
||||
segment_of(&mut globals, 0, 0, 0, &[], true, &symbol_dictionary(&symbols)?);
|
||||
let count = u32::try_from(symbols.len()).ok()?;
|
||||
let streams = images
|
||||
.iter()
|
||||
.map(|(samples, columns, rows)| {
|
||||
let page = page(samples, *columns)?;
|
||||
let ids = page.symbols.iter().map(|s| find(&table, &by_hash, s).and_then(|t| global.get(t as usize).copied().flatten())).collect();
|
||||
let shared = Shared { ids, count };
|
||||
[1, 2].into_iter().filter_map(|copies| plan(&page, *columns, *rows, copies, None, Some(&shared))).min_by_key(Vec::len)
|
||||
})
|
||||
.collect();
|
||||
Some((globals, streams))
|
||||
}
|
||||
|
||||
/// What sorts a symbol's bitmap ([`symbols`] and [`encode_shared`] look exact copies up by it).
|
||||
fn fingerprint(s: &Symbol) -> u64 {
|
||||
let mut hasher = DefaultHasher::new();
|
||||
(s.width, s.height, &s.pixels).hash(&mut hasher);
|
||||
hasher.finish()
|
||||
}
|
||||
|
||||
/// The index in `table` of the symbol with `s`'s bitmap, looked up by its [`fingerprint`].
|
||||
fn find(table: &[Symbol], by_hash: &HashMap<u64, Vec<u32>>, s: &Symbol) -> Option<u32> {
|
||||
by_hash
|
||||
.get(&fingerprint(s))?
|
||||
.iter()
|
||||
.copied()
|
||||
.find(|t| table.get(*t as usize).is_some_and(|t| (t.width, t.height) == (s.width, s.height) && t.pixels == s.pixels))
|
||||
}
|
||||
|
||||
/// The image's marks and symbols; `None` past the limits above.
|
||||
@@ -267,23 +365,18 @@ fn symbols(page: &mut Page) -> Option<()> {
|
||||
page.large.push(m);
|
||||
continue;
|
||||
}
|
||||
let pixels = pixels(mark, &page.runs, &page.order);
|
||||
let mut hasher = DefaultHasher::new();
|
||||
(width, height, &pixels).hash(&mut hasher);
|
||||
let same = by_hash.entry(hasher.finish()).or_default();
|
||||
let symbol = Symbol { width, height, pixels: pixels(mark, &page.runs, &page.order), uses: 0 };
|
||||
let symbols = &mut page.symbols;
|
||||
let known =
|
||||
same.iter().copied().find(|s| symbols.get(*s as usize).is_some_and(|s| (s.width, s.height) == (width, height) && s.pixels == pixels));
|
||||
let id = match known {
|
||||
let id = match find(symbols, &by_hash, &symbol) {
|
||||
Some(id) => id,
|
||||
None => {
|
||||
kept = kept.checked_add(pixels.len())?;
|
||||
kept = kept.checked_add(symbol.pixels.len())?;
|
||||
if kept > MAX_SYMBOL_PIXELS || symbols.len() == MAX_SYMBOLS {
|
||||
return None;
|
||||
}
|
||||
let id = u32::try_from(symbols.len()).ok()?;
|
||||
symbols.push(Symbol { width, height, pixels, uses: 0 });
|
||||
same.push(id);
|
||||
by_hash.entry(fingerprint(&symbol)).or_default().push(id);
|
||||
symbols.push(symbol);
|
||||
id
|
||||
}
|
||||
};
|
||||
@@ -300,18 +393,28 @@ fn symbols(page: &mut Page) -> Option<()> {
|
||||
/// the other marks in a generic region. `None` with fewer than two such symbols (their IDs would
|
||||
/// take no bits at all, SBSYMCODELEN 0, a corner of the standard left alone here) or more than
|
||||
/// [`MAX_CODED_SYMBOLS`].
|
||||
fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option<u32>) -> Option<Vec<u8>> {
|
||||
///
|
||||
/// With `shared`, the symbols a group's dictionary holds are placed from there, whatever their uses
|
||||
/// here, and the page's own dictionary holds only the others (none, when it has none), as
|
||||
/// [`encode_shared`] writes them; `None` too for a page that places none of the shared ones.
|
||||
fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option<u32>, shared: Option<&Shared>) -> Option<Vec<u8>> {
|
||||
let symbol = |s: u32| page.symbols.get(s as usize);
|
||||
let global = |s: u32| shared.and_then(|g| g.ids.get(s as usize).copied().flatten());
|
||||
let globals = shared.map_or(0, |g| g.count);
|
||||
// The dictionary's order (6.5.5): height classes from the shortest, each narrowest first.
|
||||
let mut kept: Vec<u32> = (0..page.symbols.len() as u32).filter(|s| symbol(*s).is_some_and(|s| s.uses >= copies)).collect();
|
||||
if kept.len() < 2 || kept.len() > MAX_CODED_SYMBOLS {
|
||||
let mut kept: Vec<u32> =
|
||||
(0..page.symbols.len() as u32).filter(|s| global(*s).is_none() && symbol(*s).is_some_and(|s| s.uses >= copies)).collect();
|
||||
let count = usize::try_from(globals).ok()?.checked_add(kept.len())?;
|
||||
if !(2..=MAX_CODED_SYMBOLS).contains(&count) {
|
||||
return None;
|
||||
}
|
||||
kept.sort_by_key(|s| symbol(*s).map(|s| (s.height, s.width)));
|
||||
let mut id = vec![None; page.symbols.len()];
|
||||
// The shared symbols' IDs come first, as their dictionary is referred to first (Table 31's
|
||||
// SBSYMS).
|
||||
let mut id: Vec<Option<u32>> = (0..page.symbols.len() as u32).map(global).collect();
|
||||
for (k, s) in kept.iter().enumerate() {
|
||||
if let Some(slot) = id.get_mut(*s as usize) {
|
||||
*slot = u32::try_from(k).ok();
|
||||
*slot = u32::try_from(k).ok().and_then(|k| k.checked_add(globals));
|
||||
}
|
||||
}
|
||||
let mut instances = Vec::new();
|
||||
@@ -324,9 +427,12 @@ fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option<u3
|
||||
_ => left.push(*m),
|
||||
}
|
||||
}
|
||||
if shared.is_some() && !instances.iter().any(|i| i.id < globals) {
|
||||
return None;
|
||||
}
|
||||
let kept: Vec<&Symbol> = kept.iter().filter_map(|s| symbol(*s)).collect();
|
||||
let dictionary = symbol_dictionary(&kept)?;
|
||||
let count = u32::try_from(kept.len()).ok()?;
|
||||
let dictionary = if kept.is_empty() { None } else { Some(symbol_dictionary(&kept)?) };
|
||||
let count = u32::try_from(count).ok()?;
|
||||
// Bands of at most MAX_REGION_INSTANCES, from the top.
|
||||
instances.sort_by_key(|i| (i.t, i.s));
|
||||
let mut regions = Vec::new();
|
||||
@@ -339,12 +445,21 @@ fn plan(page: &Page, columns: u32, rows: u32, copies: u32, log_strips: Option<u3
|
||||
});
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
segment(&mut out, 0, 48, &[], false, &page_information(columns, rows));
|
||||
segment(&mut out, 1, 0, &[], true, &dictionary);
|
||||
let mut number = 2;
|
||||
// The shared dictionary is segment 0, in the globals, and a segment refers only to lower
|
||||
// numbers (7.2.5): the page's own follow it.
|
||||
let mut number = u32::from(shared.is_some());
|
||||
segment(&mut out, number, 48, &[], false, &page_information(columns, rows));
|
||||
number += 1;
|
||||
let mut dictionaries: Vec<u8> = shared.map(|_| 0).into_iter().collect();
|
||||
if let Some(dictionary) = &dictionary {
|
||||
segment(&mut out, number, 0, &[], true, dictionary);
|
||||
dictionaries.push(u8::try_from(number).ok()?);
|
||||
number += 1;
|
||||
}
|
||||
for (k, text) in regions.iter().enumerate() {
|
||||
// The dictionary stays retained for the next region, until the last (7.2.4).
|
||||
segment(&mut out, number, 7, &[(1, k + 1 < regions.len())], false, text);
|
||||
// The dictionaries stay retained for the next region, until the last (7.2.4).
|
||||
let refers_to: Vec<(u8, bool)> = dictionaries.iter().map(|n| (*n, k + 1 < regions.len())).collect();
|
||||
segment(&mut out, number, 7, &refers_to, false, text);
|
||||
number += 1;
|
||||
}
|
||||
if !left.is_empty() {
|
||||
@@ -538,8 +653,9 @@ fn encode_id(mq: &mut MqEncoder, contexts: &mut [Context], length: u32, id: u32)
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::jbig2::tests::{decode, samples, unpack};
|
||||
use crate::jbig2::{encode_jbig2, encode_jbig2_generic};
|
||||
use crate::FilterError;
|
||||
use crate::jbig2::tests::{decode, decode_with, samples, unpack};
|
||||
use crate::jbig2::{encode_jbig2, encode_jbig2_generic, encode_jbig2_shared};
|
||||
|
||||
type Bitmap = Vec<Vec<u8>>;
|
||||
|
||||
@@ -664,7 +780,7 @@ mod tests {
|
||||
let mut plans = 0;
|
||||
for copies in [1, 2] {
|
||||
for log_strips in 0..4 {
|
||||
if let Some(coded) = plan(&page, columns, rows, copies, Some(log_strips)) {
|
||||
if let Some(coded) = plan(&page, columns, rows, copies, Some(log_strips), None) {
|
||||
assert_eq!(decode(&coded), unpack(&data, columns, rows), "{copies} copies, strips of {}", 1 << log_strips);
|
||||
plans += 1;
|
||||
}
|
||||
@@ -733,7 +849,7 @@ mod tests {
|
||||
let page = page(&data, c).unwrap();
|
||||
assert!(page.marks.iter().all(|m| m.x1 as usize <= columns), "{columns} columns");
|
||||
for copies in [1, 2] {
|
||||
if let Some(coded) = plan(&page, c, r, copies, None) {
|
||||
if let Some(coded) = plan(&page, c, r, copies, None, None) {
|
||||
assert_eq!(decode(&coded), unpack(&data, c, r), "{columns} columns");
|
||||
}
|
||||
}
|
||||
@@ -820,17 +936,17 @@ mod tests {
|
||||
};
|
||||
let (data, columns, rows) = codes(MAX_CODED_SYMBOLS, 0);
|
||||
let at_cap = page(&data, columns).unwrap();
|
||||
assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows));
|
||||
assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None, None).unwrap()), unpack(&data, columns, rows));
|
||||
// One more distinct mark, each once: neither plan is made.
|
||||
let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 0);
|
||||
let past = page(&data, columns).unwrap();
|
||||
assert!(plan(&past, columns, rows, 1, None).is_none() && plan(&past, columns, rows, 2, None).is_none());
|
||||
assert!(plan(&past, columns, rows, 1, None, None).is_none() && plan(&past, columns, rows, 2, None, None).is_none());
|
||||
assert_eq!(encode_jbig2(&data, columns, rows).unwrap(), encode_jbig2_generic(&data, columns, rows).unwrap());
|
||||
// With two of them repeated, the second plan codes those two.
|
||||
let (data, columns, rows) = codes(MAX_CODED_SYMBOLS + 1, 200);
|
||||
let past = page(&data, columns).unwrap();
|
||||
assert!(plan(&past, columns, rows, 1, None).is_none());
|
||||
assert_eq!(decode(&plan(&past, columns, rows, 2, None).unwrap()), unpack(&data, columns, rows));
|
||||
assert!(plan(&past, columns, rows, 1, None, None).is_none());
|
||||
assert_eq!(decode(&plan(&past, columns, rows, 2, None, None).unwrap()), unpack(&data, columns, rows));
|
||||
// Above a page of text whose letters repeat, that second plan is the one `encode_jbig2`
|
||||
// keeps: the first, of every mark, would code too many symbols.
|
||||
let mut picture: Bitmap = vec![vec![0u8; 256 * 18]; 512];
|
||||
@@ -869,7 +985,7 @@ mod tests {
|
||||
let (data, columns, rows) = squares(256);
|
||||
let at_cap = page(&data, columns).unwrap();
|
||||
assert_eq!(at_cap.symbols.len(), 256);
|
||||
assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None).unwrap()), unpack(&data, columns, rows));
|
||||
assert_eq!(decode(&plan(&at_cap, columns, rows, 1, None, None).unwrap()), unpack(&data, columns, rows));
|
||||
let (data, columns, _) = squares(257);
|
||||
assert!(page(&data, columns).is_none());
|
||||
}
|
||||
@@ -984,7 +1100,199 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Each segment's page association (7.2.6): the byte after the numbers it refers to.
|
||||
fn associations(data: &[u8]) -> Vec<u8> {
|
||||
let (mut pages, mut at) = (Vec::new(), 0);
|
||||
while at < data.len() {
|
||||
let refers = usize::from(data[at + 5] >> 5);
|
||||
pages.push(data[at + 6 + refers]);
|
||||
let length_at = at + 6 + refers + 1;
|
||||
at = length_at + 4 + u32::from_be_bytes(data[length_at..length_at + 4].try_into().unwrap()) as usize;
|
||||
}
|
||||
pages
|
||||
}
|
||||
|
||||
/// `pictures`, `columns` × `rows` each, coded sharing a dictionary; each image's stream, which
|
||||
/// has to be there, decodes with the globals to its picture.
|
||||
fn shared_round_trip(pictures: &[Bitmap]) -> (Vec<u8>, Vec<Vec<u8>>) {
|
||||
let (columns, rows) = (pictures[0][0].len() as u32, pictures[0].len() as u32);
|
||||
let data: Vec<Vec<u8>> = pictures.iter().map(|p| samples(p)).collect();
|
||||
let images: Vec<(&[u8], u32, u32)> = data.iter().map(|d| (&d[..], columns, rows)).collect();
|
||||
let shared = encode_jbig2_shared(&images).unwrap().unwrap();
|
||||
let (globals, coded) = (shared.globals, shared.images);
|
||||
let coded: Vec<Vec<u8>> = coded.into_iter().map(Option::unwrap).collect();
|
||||
for (d, c) in data.iter().zip(&coded) {
|
||||
assert_eq!(decode_with(c, Some(&globals)), unpack(d, columns, rows));
|
||||
}
|
||||
(globals, coded)
|
||||
}
|
||||
|
||||
/// `font` with a stem 20 pixels tall down each glyph's left side: a font sharing no glyph with
|
||||
/// any [`glyphs`] gives, whose glyphs are at most 18 pixels tall.
|
||||
fn stemmed(font: &[Bitmap]) -> Vec<Bitmap> {
|
||||
font.iter()
|
||||
.map(|g| {
|
||||
(0..g.len().max(20)).map(|y| std::iter::once(1).chain(g.get(y).cloned().unwrap_or_else(|| vec![0; g[0].len()])).collect()).collect()
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Mark `k` of a set of distinct ones: a stroke 33 pixels long with a pixel above every other
|
||||
/// column for each 1 bit of `k`, below 2^17.
|
||||
fn comb(k: usize) -> Bitmap {
|
||||
let mut g = vec![vec![0u8; 33]; 2];
|
||||
g[1].fill(1);
|
||||
for bit in 0..17 {
|
||||
g[0][2 * bit] = u8::from(k >> bit & 1 == 1);
|
||||
}
|
||||
g
|
||||
}
|
||||
|
||||
/// The combs `ks`, 120 a row, the rows four pixels apart.
|
||||
fn combs(ks: impl ExactSizeIterator<Item = usize>) -> Bitmap {
|
||||
let mut page = vec![vec![0u8; 120 * 36]; ks.len().div_ceil(120) * 4];
|
||||
for (i, k) in ks.enumerate() {
|
||||
draw(&mut page, &comb(k), (i % 120 * 36) as i64, (i / 120 * 4) as i64);
|
||||
}
|
||||
page
|
||||
}
|
||||
|
||||
/// The symbols a symbol dictionary segment exports (SDNUMEXSYMS, 7.4.2.1.4), in a stream
|
||||
/// that is that segment alone: after its header, flags and adaptive pixels.
|
||||
fn exported(globals: &[u8]) -> usize {
|
||||
u32::from_be_bytes(globals[21..25].try_into().unwrap()) as usize
|
||||
}
|
||||
|
||||
/// Pages in one font share its letters: one symbol dictionary, segment 0, associated with no
|
||||
/// page and retained (7.2.4, 7.2.6), which each page's text region refers to first, its own
|
||||
/// segments numbered after it (7.2.5). Together they take less than they take alone.
|
||||
#[test]
|
||||
fn pages_in_one_font_share_one_dictionary() {
|
||||
let font = glyphs(40, 7);
|
||||
let pictures: Vec<Bitmap> = (0..4).map(|seed| text(1200, 300, &font, 20 + seed)).collect();
|
||||
let (globals, coded) = shared_round_trip(&pictures);
|
||||
assert_eq!(segments(&globals), [(0, 0, true, vec![])]);
|
||||
assert_eq!(associations(&globals), [0]);
|
||||
let (mut shared, mut alone) = (globals.len(), 0);
|
||||
for (picture, c) in pictures.iter().zip(&coded) {
|
||||
let segments = segments(c);
|
||||
assert_eq!(segments.first().map(|s| s.0), Some(1), "the page's own segments follow");
|
||||
assert!(segments.iter().filter(|s| s.1 == 7).all(|s| s.3.first() == Some(&(0, false))), "{segments:?}");
|
||||
assert!(associations(c).iter().all(|page| *page == 1));
|
||||
shared += c.len();
|
||||
alone += encode_jbig2(&samples(picture), 1200, 300).unwrap().len();
|
||||
}
|
||||
assert!(shared < alone, "shared {shared} bytes, alone {alone}");
|
||||
}
|
||||
|
||||
/// Two pages alike place every mark from the shared dictionary and have none of their own; a
|
||||
/// page with letters of its own as well keeps them in its own dictionary, referred to after the
|
||||
/// shared one (its symbols' IDs follow), and the page with nothing shared has no stream.
|
||||
#[test]
|
||||
fn what_a_page_has_alone_stays_its_own() {
|
||||
let (font, other) = (glyphs(30, 7), stemmed(&glyphs(20, 99)));
|
||||
let same = text(900, 240, &font, 3);
|
||||
let mut more = text(900, 240, &font, 4);
|
||||
more.extend(text(900, 120, &other, 5));
|
||||
let (globals, coded) = shared_round_trip(&[same.clone(), same.clone()]);
|
||||
assert!(!globals.is_empty());
|
||||
assert!(coded.iter().all(|c| segment_types(c) == [48, 7]), "{:?}", coded.iter().map(|c| segment_types(c)).collect::<Vec<_>>());
|
||||
let with_more = |p: &Bitmap| p.iter().cloned().chain(std::iter::repeat_n(vec![0u8; 900], 120)).collect::<Bitmap>();
|
||||
let (_, coded) = shared_round_trip(&[with_more(&same), more]);
|
||||
assert_eq!(segment_types(&coded[1])[..3], [48, 0, 7]);
|
||||
assert_eq!(segments(&coded[1])[2].3, [(0, false), (2, false)]);
|
||||
let alone = text(900, 360, &other, 6);
|
||||
let data: Vec<Vec<u8>> = [&same, &same, &alone].iter().map(|p| samples(p)).collect();
|
||||
let images: Vec<(&[u8], u32, u32)> = vec![(&data[0], 900, 240), (&data[1], 900, 240), (&data[2], 900, 360)];
|
||||
let (_, coded) = encode_shared(&images).unwrap();
|
||||
assert!(coded[0].is_some() && coded[1].is_some() && coded[2].is_none());
|
||||
}
|
||||
|
||||
/// Images that share fewer than two marks share nothing; two are enough.
|
||||
#[test]
|
||||
fn images_with_nothing_in_common_share_nothing() {
|
||||
let font = glyphs(30, 7);
|
||||
let (a, b) = (samples(&text(600, 200, &font, 1)), samples(&text(600, 200, &stemmed(&font), 1)));
|
||||
assert!(encode_shared(&[(&a, 600, 200), (&b, 600, 200)]).is_none());
|
||||
assert!(encode_shared(&[(&a, 600, 200)]).is_none(), "one image");
|
||||
assert!(encode_shared(&[]).is_none());
|
||||
// The two commonest letters of the first page, below the last line of the second.
|
||||
let mut two = text(600, 200, &stemmed(&font), 1);
|
||||
draw(&mut two, &font[0], 10, 180);
|
||||
draw(&mut two, &font[1], 100, 180);
|
||||
let two = samples(&two);
|
||||
let (globals, coded) = encode_shared(&[(&a, 600, 200), (&two, 600, 200)]).unwrap();
|
||||
assert_eq!(exported(&globals), 2);
|
||||
assert_eq!(decode_with(coded[1].as_ref().unwrap(), Some(&globals)), unpack(&two, 600, 200));
|
||||
}
|
||||
|
||||
/// The images of a group are checked as `encode_jbig2_generic` checks one, and an image it
|
||||
/// would refuse refuses the group.
|
||||
#[test]
|
||||
fn a_group_with_an_image_the_encoder_refuses_is_refused() {
|
||||
let page = samples(&text(600, 200, &glyphs(30, 7), 1));
|
||||
assert!(matches!(encode_jbig2_shared(&[(&page, 600, 200), (&page[..10], 600, 200)]), Err(FilterError::Corrupt { .. })));
|
||||
assert!(matches!(encode_jbig2_shared(&[(&page, 600, 200), (&page, 0, 200)]), Err(FilterError::Unsupported(_))));
|
||||
assert!(encode_jbig2_shared(&[(&page, 600, 200), (&page, 600, 200)]).unwrap().is_some());
|
||||
}
|
||||
|
||||
/// A group holds its distinct marks within what one image's are held to: past 65,536 a mark
|
||||
/// first seen later isn't held, so isn't shared, while those held count every image they're
|
||||
/// in. Ten marks of the first image are in two images after it, with 464 first seen there.
|
||||
#[test]
|
||||
fn past_the_marks_a_group_holds_later_ones_arent_shared() {
|
||||
let (a, b) = (samples(&combs(0..40_000)), samples(&combs(40_000..65_536)));
|
||||
let late = combs((0..10).chain(65_536..66_000).collect::<Vec<_>>().into_iter());
|
||||
let (rows, late) = (late.len() as u32, samples(&late));
|
||||
let (columns, a_rows, b_rows) = (120 * 36, 40_000u32.div_ceil(120) * 4, 25_536u32.div_ceil(120) * 4);
|
||||
let images = [(&a[..], columns, a_rows), (&b[..], columns, b_rows), (&late[..], columns, rows), (&late[..], columns, rows)];
|
||||
let (globals, coded) = encode_shared(&images).unwrap();
|
||||
assert_eq!(exported(&globals), 10);
|
||||
for c in &coded[2..] {
|
||||
assert_eq!(decode_with(c.as_ref().unwrap(), Some(&globals)), unpack(&late, columns, rows));
|
||||
}
|
||||
}
|
||||
|
||||
/// Past what one dictionary shares, it keeps the marks the most images hold: 20,000 shared,
|
||||
/// the first 10,000 in all three images and the rest in two. The third image then places all
|
||||
/// its marks from it, and the other two code the rest themselves (in a dictionary of their own
|
||||
/// or a generic region, whichever is smaller).
|
||||
#[test]
|
||||
fn past_the_shared_limit_the_most_widely_held_marks_are_shared() {
|
||||
let (both, first) = (combs(0..20_000), combs(0..10_000));
|
||||
let rows = both.len();
|
||||
let first: Bitmap = first.iter().cloned().chain(std::iter::repeat_n(vec![0u8; 120 * 36], rows - first.len())).collect();
|
||||
let (globals, coded) = shared_round_trip(&[both.clone(), both, first]);
|
||||
assert_eq!(exported(&globals), MAX_SHARED_SYMBOLS);
|
||||
assert_eq!(segment_types(&coded[2]), [48, 7]);
|
||||
assert!(segment_types(&coded[0]).iter().any(|t| *t == 0 || *t == 39), "{:?}", segment_types(&coded[0]));
|
||||
}
|
||||
|
||||
proptest::proptest! {
|
||||
/// Glyphs scattered over pages of one font: whatever is shared, every page with a stream
|
||||
/// decodes with the globals to its picture.
|
||||
#[test]
|
||||
fn shared_groups_round_trip(
|
||||
columns in 1usize..200,
|
||||
rows in 1usize..100,
|
||||
shapes in 2usize..8,
|
||||
n in 0usize..60,
|
||||
pages in 2usize..5,
|
||||
seed in proptest::num::u64::ANY,
|
||||
) {
|
||||
let font = glyphs(shapes, seed);
|
||||
let data: Vec<Vec<u8>> = (0..pages).map(|p| samples(&scatter(columns, rows, &font, seed ^ p as u64, n))).collect();
|
||||
let (c, r) = (columns as u32, rows as u32);
|
||||
let images: Vec<(&[u8], u32, u32)> = data.iter().map(|d| (&d[..], c, r)).collect();
|
||||
if let Some((globals, coded)) = encode_shared(&images) {
|
||||
for (d, coded) in data.iter().zip(coded) {
|
||||
if let Some(coded) = coded {
|
||||
proptest::prop_assert_eq!(decode_with(&coded, Some(&globals)), unpack(d, c, r));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Glyphs anywhere, over the edges and each other, with stray pixels: every plan decodes to
|
||||
/// the picture, and `encode_jbig2` is never larger than the generic coding.
|
||||
#[test]
|
||||
@@ -1015,6 +1323,7 @@ mod tests {
|
||||
fn any_input_returns(data in proptest::collection::vec(0u8..=255, 0..600), columns in 0u32..5000, rows in 0u32..80) {
|
||||
let _ = encode_jbig2(&data, columns, rows);
|
||||
let _ = encode(&data, columns, rows);
|
||||
let _ = encode_shared(&[(&data, columns, rows), (&data, columns, rows.saturating_sub(1))]);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -27,7 +27,8 @@
|
||||
//! ```
|
||||
//!
|
||||
//! 1-bit samples can be encoded as CCITT Group 4 ([`encode_g4`]) or lossless JBIG2
|
||||
//! ([`encode_jbig2`]).
|
||||
//! ([`encode_jbig2`]), a group of images sharing their marks through `/JBIG2Globals`
|
||||
//! ([`encode_jbig2_shared`]).
|
||||
//!
|
||||
//! Every decoder is bounded by a caller-supplied `max_output` (decompression-bomb
|
||||
//! defence) and never panics on malformed input.
|
||||
@@ -47,7 +48,7 @@ mod runlength;
|
||||
use std::borrow::Cow;
|
||||
|
||||
pub use ccitt::{CcittParams, MAX_CCITT_COLUMNS, MAX_G4_COLUMNS, decode_ccitt_within, encode_g4};
|
||||
pub use jbig2::{MAX_JBIG2_COLUMNS, MAX_JBIG2_PIXELS, decode_jbig2_within, encode_jbig2, encode_jbig2_generic};
|
||||
pub use jbig2::{MAX_JBIG2_COLUMNS, MAX_JBIG2_PIXELS, SharedJbig2, decode_jbig2_within, encode_jbig2, encode_jbig2_generic, encode_jbig2_shared};
|
||||
|
||||
/// A PDF stream filter, identified by its `/Filter` name.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
|
||||
|
||||
@@ -89,7 +89,15 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
alone when JBIG2 is asked for, is stored again as the compression asks when that is smaller (as JBIG2 symbols where
|
||||
its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact
|
||||
repeats and often comes out larger, and then stays as it is. A JBIG2 image stored again refers to no
|
||||
`/JBIG2Globals`, and one whose page isn't its size (PdfCraft's
|
||||
`/JBIG2Globals` of its own. With JBIG2 asked for, the images stored as JBIG2, up to 16 at a
|
||||
time, then share one symbol dictionary in a new `/JBIG2Globals` stream, holding the marks found
|
||||
in more than one of them: those that come out smaller that way, when together they save more
|
||||
than it takes. Only images that redaction can store again without it share one: image masks
|
||||
and DeviceGray images named directly, of at most 64 Mi pixels (redaction's limit,
|
||||
`pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS`); others are coded alone.
|
||||
Redacting one of them stores the others of its group again without it, so the group keeps that
|
||||
to 15 images, and each image's decode reads the dictionary of 16 at most. A
|
||||
JBIG2 image whose page isn't its size (PdfCraft's
|
||||
renderer draws the whole page into the image) is left alone. With Retain, a resampled image
|
||||
keeps its kind of compression (CCITT, JBIG2, or Flate) and nothing else changes; an image of
|
||||
more bits is then left to the grey or colour settings. Every change is kept only when it is
|
||||
@@ -103,7 +111,7 @@ let report = optimize_with_progress(&mut doc, &settings, &mut |stage| keep_going
|
||||
is past the 80 million pixels above).
|
||||
|
||||
- Lossless JBIG2 uses bounded generic-region encoding or exact-copy symbol dictionaries when
|
||||
smaller. No lossy symbol substitution is used. Export, redaction and this step read JBIG2
|
||||
smaller, a dictionary shared by up to 16 images when that is smaller still. No lossy symbol substitution is used. Export, redaction and this step read JBIG2
|
||||
images, with bounded shared globals. All of a run's images share one allowance of decoding
|
||||
(`pdfcraft_cos::ImageBudget`), filters, decoders and pixels alike, JPEG samples and merged
|
||||
pictures too: about
|
||||
|
||||
@@ -1117,12 +1117,23 @@ fn downsample_bits_checked(data: &[u8], w: u32, h: u32, nw: u32, nh: u32, dark_i
|
||||
/// to be resampled, and so is JBIG2 alone when Group 4 is asked for; Group 3, Group 4 inside
|
||||
/// another filter, and JBIG2 otherwise are stored again like any other. With `Retain`, a resampled image keeps its kind of compression (CCITT, JBIG2, or
|
||||
/// Flate), and nothing else changes: an image of more bits stays as it is, for the grey or
|
||||
/// colour settings. A JBIG2 image stored again refers to no `/JBIG2Globals`.
|
||||
/// colour settings. A JBIG2 image stored again refers to no `/JBIG2Globals`; with `Jbig2`, [`share`]
|
||||
/// may then have a group of them share one.
|
||||
///
|
||||
/// The samples keep their meaning, so the colour space, decode array and `/Mask` stay, but for
|
||||
/// black-and-white RGB, which becomes grey with one decode pair, and an indexed image's default
|
||||
/// decode array, which goes as the default at one bit.
|
||||
fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSettings, budget: &mut ImageBudget) -> Option<(Stream, bool)> {
|
||||
///
|
||||
/// Stored as JBIG2 with `Jbig2`, it comes with the 1-bit samples it was coded from, for [`share`]
|
||||
/// to code again with the run's other JBIG2 images.
|
||||
fn mono(
|
||||
doc: &Document,
|
||||
s: &Stream,
|
||||
ppi: f64,
|
||||
defaults: bool,
|
||||
settings: &MonoSettings,
|
||||
budget: &mut ImageBudget,
|
||||
) -> Option<(Stream, bool, Option<Vec<u8>>)> {
|
||||
let d = &s.dict;
|
||||
let downsample = settings.downsample && ppi.is_finite() && ppi > settings.above_ppi && settings.target_ppi > 0.0 && !d.contains(b"SMask");
|
||||
let retain = settings.compression == MonoCompression::Retain;
|
||||
@@ -1206,7 +1217,106 @@ fn mono(doc: &Document, s: &Stream, ppi: f64, defaults: bool, settings: &MonoSet
|
||||
let flate = ((resampled || packed_from_more) && !(retain && (ccitt || jbig2))).then(|| Stream::flate(dict, &data));
|
||||
// Smaller as stored, counting the dictionary: a tiny image can lose by Group 4's parameters.
|
||||
let best = [g4, jbig, flate].into_iter().flatten().filter_map(|c| Some((crate::stored_len(&c)?, c))).min_by_key(|(len, _)| *len)?;
|
||||
(best.0 < crate::stored_len(s)?).then_some((best.1, resampled))
|
||||
let shareable = settings.compression == MonoCompression::Jbig2 && best.1.dict.name(b"Filter") == Some(b"JBIG2Decode");
|
||||
(best.0 < crate::stored_len(s)?).then(|| (best.1, resampled, shareable.then_some(data)))
|
||||
}
|
||||
|
||||
/// The most images that share one symbol dictionary ([`share`]). Redaction stores again every
|
||||
/// image that shares a redacted one's dictionary, and each image's decode reads all of it, so a
|
||||
/// group keeps both to that many; most of what sharing saves comes with the first few images.
|
||||
const SHARED_IMAGES: usize = 16;
|
||||
|
||||
/// The most bytes of 1-bit samples a group holds: a group is coded before it would pass them
|
||||
/// ([`hold_in`]). An image that shares has at most [`SHARED_PIXELS`], and as its rows are whole
|
||||
/// bytes, at most that many bytes, which is this: one always fits a group with nothing in it.
|
||||
const SHARED_BYTES: usize = 64 << 20;
|
||||
|
||||
/// What the globals cost as an object of their own besides their dictionary and data: `N 0 obj`,
|
||||
/// `stream`, `endstream` and `endobj`, a cross-reference entry, and an encrypted file's IV and
|
||||
/// padding. A group is shared only when it saves more than that as well.
|
||||
const NEW_OBJECT: usize = 96;
|
||||
|
||||
/// Adds `image` to `group`, coding the group (`share`, which empties it) first when the image's
|
||||
/// samples would take it past `cap` bytes, and after when it has [`SHARED_IMAGES`].
|
||||
fn hold_in(group: &mut Vec<Held>, image: Held, cap: usize, mut share: impl FnMut(&mut Vec<Held>)) {
|
||||
if group.iter().map(|h| h.bits.len()).fold(image.bits.len(), usize::saturating_add) > cap {
|
||||
share(group);
|
||||
}
|
||||
group.push(image);
|
||||
if group.len() == SHARED_IMAGES {
|
||||
share(group);
|
||||
}
|
||||
}
|
||||
|
||||
/// The most pixels an image sharing a dictionary has: redaction's own limit on the images it
|
||||
/// stores again (see [`shares`]).
|
||||
const SHARED_PIXELS: u64 = pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS;
|
||||
|
||||
/// Whether an image stored as JBIG2 (its dictionary `d`) may share a dictionary: only one that
|
||||
/// redaction can store again without it, an image mask or a DeviceGray image, named directly, of
|
||||
/// at most [`SHARED_PIXELS`]. Redacting one image of a group stores the others so, as the
|
||||
/// dictionary can hold the shapes of marks under the area, and refuses the redaction for one it
|
||||
/// can't; an image of another colour space, though redaction may take it, is coded alone.
|
||||
fn shares(d: &Dict) -> bool {
|
||||
let side = |key: &[u8]| d.int(key).and_then(|v| u64::try_from(v).ok());
|
||||
let within = side(b"Width").zip(side(b"Height")).is_some_and(|(w, h)| w.saturating_mul(h) <= SHARED_PIXELS);
|
||||
within && (matches!(d.get(b"ImageMask"), Some(Object::Bool(true))) || d.name(b"ColorSpace") == Some(b"DeviceGray"))
|
||||
}
|
||||
|
||||
/// An image the run stored as JBIG2, held to be stored again sharing a dictionary ([`share`]): its
|
||||
/// 1-bit samples, `width` × `height`.
|
||||
struct Held {
|
||||
image: ObjRef,
|
||||
bits: Vec<u8>,
|
||||
width: u32,
|
||||
height: u32,
|
||||
}
|
||||
|
||||
/// Stores the images of `group`, each JBIG2 already, again sharing one symbol dictionary in a
|
||||
/// `/JBIG2Globals` stream ([`pdfcraft_filters::encode_jbig2_shared`]): those that come out smaller
|
||||
/// that way, if together they save more than the globals take. The others stay as they are, and
|
||||
/// so do all of them when that saves nothing. `report` counts the bytes as they end up, the globals
|
||||
/// included.
|
||||
fn share(doc: &mut Document, group: &mut Vec<Held>, report: &mut Report) {
|
||||
let held = std::mem::take(group);
|
||||
if held.len() < 2 {
|
||||
return;
|
||||
}
|
||||
let images: Vec<(&[u8], u32, u32)> = held.iter().map(|h| (&h.bits[..], h.width, h.height)).collect();
|
||||
let Ok(Some(shared)) = pdfcraft_filters::encode_jbig2_shared(&images) else { return };
|
||||
let with_globals = |s: &Stream, coded: Vec<u8>, globals: ObjRef| {
|
||||
let mut dict = s.dict.clone();
|
||||
let mut parms = Dict::new();
|
||||
parms.set(b"JBIG2Globals".to_vec(), Object::Ref(globals));
|
||||
dict.set(b"DecodeParms".to_vec(), Object::Dict(parms));
|
||||
Stream::from_raw(dict, coded)
|
||||
};
|
||||
// Weighed naming the globals with the longest reference there is: the real one is no longer.
|
||||
let longest = ObjRef { num: u32::MAX, generation: u16::MAX };
|
||||
let (mut chosen, mut saved) = (Vec::new(), 0usize);
|
||||
for (h, coded) in held.iter().zip(shared.images) {
|
||||
let Some(coded) = coded else { continue };
|
||||
let object = doc.get(h.image);
|
||||
let Object::Stream(alone) = &*object else { continue };
|
||||
let shorter = crate::stored_len(alone).zip(crate::stored_len(&with_globals(alone, coded.clone(), longest))).filter(|(a, b)| b < a);
|
||||
if let Some((before, after)) = shorter {
|
||||
saved = saved.saturating_add(before - after);
|
||||
chosen.push((h.image, coded));
|
||||
}
|
||||
}
|
||||
let globals = Stream::from_raw(Dict::new(), shared.globals);
|
||||
if chosen.is_empty() || crate::stored_len(&globals).is_none_or(|takes| saved <= takes.saturating_add(NEW_OBJECT)) {
|
||||
return;
|
||||
}
|
||||
report.image_bytes_after = report.image_bytes_after.saturating_add(globals.raw.len());
|
||||
let globals = doc.add(Object::Stream(globals));
|
||||
for (image, coded) in chosen {
|
||||
let object = doc.get(image);
|
||||
let Object::Stream(alone) = &*object else { continue };
|
||||
report.image_bytes_after = report.image_bytes_after.saturating_sub(alone.raw.len()).saturating_add(coded.len());
|
||||
let new = with_globals(alone, coded, globals);
|
||||
doc.set(image, Object::Stream(new));
|
||||
}
|
||||
}
|
||||
|
||||
/// One pass over the document's images, within `budget`. An image past it stays as it is, and is
|
||||
@@ -1239,6 +1349,13 @@ pub(crate) fn run(
|
||||
refs.sort_by_key(|(r, _)| r.num);
|
||||
let total = refs.len();
|
||||
let mut jbig2 = false;
|
||||
let mut group: Vec<Held> = Vec::new();
|
||||
// Holds a JBIG2 image for `share`, which codes the group once it is full.
|
||||
let mut hold = |doc: &mut Document, image: ObjRef, size: (Option<i64>, Option<i64>), bits: Option<Vec<u8>>, report: &mut Report| {
|
||||
let side = |v: Option<i64>| v.and_then(|v| u32::try_from(v).ok());
|
||||
let (Some(bits), Some(width), Some(height)) = (bits, side(size.0), side(size.1)) else { return };
|
||||
hold_in(&mut group, Held { image, bits, width, height }, SHARED_BYTES, |group| share(doc, group, report));
|
||||
};
|
||||
for (done, (r, ppi)) in refs.into_iter().enumerate() {
|
||||
if !progress(Stage::Images { done, total }) {
|
||||
return Err(OptimizeError::Cancelled);
|
||||
@@ -1254,7 +1371,7 @@ pub(crate) fn run(
|
||||
Some(m) if packed_from_more => (Some(m), None),
|
||||
m => (None, m),
|
||||
};
|
||||
if let Some((new, resampled)) = mono {
|
||||
if let Some((new, resampled, bits)) = mono {
|
||||
report.image_bytes_before += s.raw.len();
|
||||
report.image_bytes_after += new.raw.len();
|
||||
if resampled {
|
||||
@@ -1263,7 +1380,10 @@ pub(crate) fn run(
|
||||
report.images_recompressed += 1;
|
||||
}
|
||||
jbig2 |= new.dict.name(b"Filter") == Some(b"JBIG2Decode");
|
||||
let size = (new.dict.int(b"Width"), new.dict.int(b"Height"));
|
||||
let bits = bits.filter(|_| shares(&new.dict));
|
||||
doc.set(r, Object::Stream(new));
|
||||
hold(doc, r, size, bits, report);
|
||||
continue;
|
||||
}
|
||||
let n = s.dict.get(b"ColorSpace").and_then(|cs| components(doc, cs));
|
||||
@@ -1278,10 +1398,10 @@ pub(crate) fn run(
|
||||
}),
|
||||
};
|
||||
let total = |new: &Stream, smask: Option<&(ObjRef, Stream)>| Some(crate::stored_len(new)?.saturating_add(smask_len(smask).unwrap_or(0)));
|
||||
let packed_wins = packed.as_ref().is_some_and(|(p, _)| {
|
||||
let packed_wins = packed.as_ref().is_some_and(|(p, _, _)| {
|
||||
processed.as_ref().is_none_or(|(new, smask)| total(p, None).zip(total(new, smask.as_ref())).is_some_and(|(a, b)| a <= b))
|
||||
});
|
||||
if let Some((new, resampled)) = packed.filter(|_| packed_wins) {
|
||||
if let Some((new, resampled, bits)) = packed.filter(|_| packed_wins) {
|
||||
jbig2 |= new.dict.name(b"Filter") == Some(b"JBIG2Decode");
|
||||
report.image_bytes_before += s.raw.len();
|
||||
report.image_bytes_after += new.raw.len();
|
||||
@@ -1290,7 +1410,10 @@ pub(crate) fn run(
|
||||
} else {
|
||||
report.images_recompressed += 1;
|
||||
}
|
||||
let size = (new.dict.int(b"Width"), new.dict.int(b"Height"));
|
||||
let bits = bits.filter(|_| shares(&new.dict));
|
||||
doc.set(r, Object::Stream(new));
|
||||
hold(doc, r, size, bits, report);
|
||||
continue;
|
||||
}
|
||||
let Some((new, smask)) = processed else {
|
||||
@@ -1311,6 +1434,7 @@ pub(crate) fn run(
|
||||
doc.set(mr, Object::Stream(m));
|
||||
}
|
||||
}
|
||||
share(doc, &mut group, report);
|
||||
if !progress(Stage::Images { done: total, total }) {
|
||||
return Err(OptimizeError::Cancelled);
|
||||
}
|
||||
@@ -1338,6 +1462,58 @@ fn at_least_pdf_1_4(doc: &mut Document) -> Result<(), OptimizeError> {
|
||||
mod dimension_tests {
|
||||
use super::*;
|
||||
|
||||
/// A group is coded before an image's samples would take it past its limit, and once it has
|
||||
/// [`SHARED_IMAGES`]: with room for 10 bytes, images of 4 go two to a group, never three, and
|
||||
/// with room for 8, two of 4 fill one exactly; one of exactly the limit joins a group with
|
||||
/// nothing in it.
|
||||
#[test]
|
||||
fn a_group_is_coded_before_its_samples_would_pass_the_limit() {
|
||||
let held = |bytes: usize| Held { image: ObjRef { num: 1, generation: 0 }, bits: vec![0; bytes], width: 8, height: 8 };
|
||||
let mut coded: Vec<Vec<usize>> = Vec::new();
|
||||
let mut group = Vec::new();
|
||||
let mut code = |g: &mut Vec<Held>| coded.push(std::mem::take(g).iter().map(|h| h.bits.len()).collect());
|
||||
for _ in 0..5 {
|
||||
hold_in(&mut group, held(4), 10, &mut code);
|
||||
}
|
||||
hold_in(&mut group, held(10), 10, &mut code);
|
||||
assert_eq!(coded, [vec![4, 4], vec![4, 4], vec![4]]);
|
||||
assert_eq!(group.iter().map(|h| h.bits.len()).collect::<Vec<_>>(), [10]);
|
||||
// Exactly the limit is within it: with room for 8, two images of 4 stay together.
|
||||
let (mut count, mut group) = (0, Vec::new());
|
||||
let mut code = |g: &mut Vec<Held>| count += std::mem::take(g).len();
|
||||
hold_in(&mut group, held(4), 8, &mut code);
|
||||
hold_in(&mut group, held(4), 8, &mut code);
|
||||
assert_eq!((count, group.len()), (0, 2));
|
||||
let (mut count, mut group) = (0, Vec::new());
|
||||
for _ in 0..SHARED_IMAGES {
|
||||
hold_in(&mut group, held(1), usize::MAX, |g| count += std::mem::take(g).len());
|
||||
}
|
||||
assert_eq!((count, group.len()), (SHARED_IMAGES, 0), "coded on the 16th");
|
||||
}
|
||||
|
||||
/// Only what redaction can store again without the dictionary shares one: an image mask, or
|
||||
/// DeviceGray named directly, of at most 2^26 pixels.
|
||||
#[test]
|
||||
fn only_what_redaction_can_store_again_shares_a_dictionary() {
|
||||
let dict = |entries: Vec<(&str, Object)>| {
|
||||
let mut d = Dict::new();
|
||||
for (k, v) in entries {
|
||||
d.set(k.as_bytes().to_vec(), v);
|
||||
}
|
||||
d
|
||||
};
|
||||
let sized = |w: i64, h: i64, cs: Object| dict(vec![("Width", Object::Int(w)), ("Height", Object::Int(h)), ("ColorSpace", cs)]);
|
||||
assert!(shares(&sized(8192, 8192, Object::name("DeviceGray"))));
|
||||
assert!(!shares(&sized(8192, 8193, Object::name("DeviceGray"))), "a row past 2^26 pixels");
|
||||
assert!(shares(&dict(vec![("Width", Object::Int(8)), ("Height", Object::Int(8)), ("ImageMask", Object::Bool(true))])));
|
||||
let device_n =
|
||||
Object::Array(vec![Object::name("DeviceN"), Object::Array(vec![Object::name("Black")]), Object::name("DeviceGray"), Object::Null]);
|
||||
assert!(!shares(&sized(8, 8, device_n)));
|
||||
assert!(!shares(&sized(8, 8, Object::name("CalGray"))));
|
||||
assert!(!shares(&sized(8, 8, Object::Ref(ObjRef { num: 3, generation: 0 }))), "a colour space by reference");
|
||||
assert!(!shares(&dict(vec![("ColorSpace", Object::name("DeviceGray"))])), "no size");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pixel_reader_rejects_hostile_dimensions_before_decoding() {
|
||||
for (width, height, channels) in [(-1, 1, 1), (0, 1, 1), (i64::from(u32::MAX) + 101, 1, 1), (80_000_001, 1, 1), (2, 2, usize::MAX)] {
|
||||
|
||||
@@ -1951,8 +1951,17 @@ fn black_and_white_images_become_jbig2() {
|
||||
assert!(r.image_bytes_after < as_g4.image_bytes_after, "JBIG2 {} bytes, Group 4 {}", r.image_bytes_after, as_g4.image_bytes_after);
|
||||
let images: Vec<Stream> = image_refs(&jbig2).into_iter().map(|i| stream(&jbig2, i)).collect();
|
||||
assert_eq!(images.len(), 3);
|
||||
// The same text three times: the grey image and the mask share one symbol dictionary, and the
|
||||
// indexed image, which only those share, is coded alone.
|
||||
let globals_of = |s: &Stream| s.dict.get(b"DecodeParms").and_then(Object::as_dict).map(|d| (d.len(), d.reference(b"JBIG2Globals")));
|
||||
let is_indexed = |s: &Stream| s.dict.get(b"ColorSpace").and_then(Object::as_array).is_some();
|
||||
let shared: Vec<_> = images.iter().filter(|s| !is_indexed(s)).map(globals_of).collect();
|
||||
assert!(shared.len() == 2 && shared[0].is_some_and(|(n, g)| n == 1 && g.is_some()) && shared[0] == shared[1], "{shared:?}");
|
||||
for s in &images {
|
||||
assert_eq!((s.dict.name(b"Filter"), s.dict.contains(b"DecodeParms")), (Some(&b"JBIG2Decode"[..]), false));
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(&b"JBIG2Decode"[..]));
|
||||
if is_indexed(s) {
|
||||
assert!(!s.dict.contains(b"DecodeParms"));
|
||||
}
|
||||
}
|
||||
assert!(images.iter().any(|s| s.dict.get(b"Decode") == Some(&decode)), "the decode array stays");
|
||||
let indexed_kept =
|
||||
@@ -1999,6 +2008,11 @@ fn a_catalog_version_given_by_reference_is_read() {
|
||||
/// `#` a 3 × 3 block, three columns apart with a wider gap between words, every copy of a letter
|
||||
/// the same to the pixel.
|
||||
fn rendered_text(w: u32, h: u32) -> Vec<u8> {
|
||||
rendered_text_from(w, h, 0x2545_F491_4F6C_DD1D)
|
||||
}
|
||||
|
||||
/// [`rendered_text`], its letters picked from `seed`, which isn't 0.
|
||||
fn rendered_text_from(w: u32, h: u32, mut seed: u64) -> Vec<u8> {
|
||||
const LETTERS: [&[&str]; 5] = [
|
||||
&["##", "##", "##", "##", "##", "##", "##", "##", "##", "##"],
|
||||
&["......", ".####.", "##..##", "##..##", "##..##", "##..##", ".####."],
|
||||
@@ -2008,7 +2022,6 @@ fn rendered_text(w: u32, h: u32) -> Vec<u8> {
|
||||
];
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut data = vec![0xFF; stride * h as usize];
|
||||
let mut seed = 0x2545_F491_4F6C_DD1Du64;
|
||||
let mut next = move || {
|
||||
seed ^= seed << 13;
|
||||
seed ^= seed >> 7;
|
||||
@@ -2053,9 +2066,194 @@ fn rendered_text_becomes_jbig2_symbols() {
|
||||
assert_eq!(s.dict.name(b"Filter"), Some(&b"JBIG2Decode"[..]));
|
||||
let generic = pdfcraft_filters::encode_jbig2_generic(&text, w, h).unwrap();
|
||||
assert!(s.raw.len() * 2 < generic.len(), "symbols {} bytes, generic {}", s.raw.len(), generic.len());
|
||||
assert!(!s.dict.contains(b"DecodeParms"), "alone, it shares no dictionary");
|
||||
assert!(render(&doc) == before, "the page looks the same");
|
||||
}
|
||||
|
||||
/// Page `index` of `doc`, saved and opened again, rendered at scale 1.
|
||||
fn render_page(doc: &Document, index: usize) -> Vec<u8> {
|
||||
let bytes = write_full(doc, &SaveOptions::default()).unwrap();
|
||||
let mut r = pdfcraft_render::PageRenderer::new(Arc::new(bytes), pdfcraft_render::RenderConfig::default());
|
||||
let out = r.render(pdfcraft_render::RenderRequest { page: index, scale: 1.0, ..Default::default() });
|
||||
assert!(out.error.is_none(), "{:?}", out.error);
|
||||
out.rgba.to_vec()
|
||||
}
|
||||
|
||||
/// Text in a font of 26 letters, about 10 pt at 300 dpi: each letter 32 rows of a run from a stem
|
||||
/// down its left side, of 2 to 24 pixels, every copy the same to the pixel; lines 44 rows apart,
|
||||
/// the letters picked from `seed`, which isn't 0.
|
||||
fn font_text(w: u32, h: u32, mut seed: u64) -> Vec<u8> {
|
||||
let mut font_seed = 0x2545_F491_4F6C_DD1Du64;
|
||||
let next = |seed: &mut u64| {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
*seed
|
||||
};
|
||||
let letters: Vec<Vec<usize>> = (0..26).map(|_| (0..16).map(|_| 1 + (next(&mut font_seed) % 12) as usize).collect()).collect();
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let mut data = vec![0xFF; stride * h as usize];
|
||||
for top in (8..(h as usize).saturating_sub(40)).step_by(44) {
|
||||
let mut x = 8;
|
||||
while x + 32 < w as usize {
|
||||
let letter = &letters[(next(&mut seed) % 26) as usize];
|
||||
for (row, run) in letter.iter().enumerate() {
|
||||
for y in [top + 2 * row, top + 2 * row + 1] {
|
||||
for px in x..x + 2 * run {
|
||||
data[y * stride + px / 8] &= !(0x80 >> (px % 8));
|
||||
}
|
||||
}
|
||||
}
|
||||
x += 2 * letter.iter().max().unwrap() + 4 + if next(&mut seed) % 6 == 0 { 14 } else { 0 };
|
||||
}
|
||||
}
|
||||
data
|
||||
}
|
||||
|
||||
/// The `/JBIG2Globals` an image names in its `/DecodeParms`.
|
||||
fn globals_of(doc: &Document, image: ObjRef) -> Option<ObjRef> {
|
||||
stream(doc, image).dict.get(b"DecodeParms")?.as_dict()?.reference(b"JBIG2Globals")
|
||||
}
|
||||
|
||||
/// Pages of text in one font, each letter the same to the pixel on every page, stored as JBIG2
|
||||
/// share one symbol dictionary (`/JBIG2Globals`): together they take less than each page coded
|
||||
/// alone, and every page looks the same, the last one, whose crosses no other page has, placing
|
||||
/// them from a dictionary of its own beside the shared one. The report counts the bytes they
|
||||
/// take, the globals once.
|
||||
#[test]
|
||||
fn jbig2_pages_share_one_symbol_dictionary() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (600u32, 480u32);
|
||||
let stride = (w as usize).div_ceil(8);
|
||||
let pages: Vec<Vec<u8>> = (1..=4u64)
|
||||
.map(|k| {
|
||||
let mut text = font_text(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15));
|
||||
if k == 4 {
|
||||
// Crosses down the right margin, three pixels a stroke.
|
||||
for top in (40..440).step_by(40) {
|
||||
for (dx, dy) in (0..9).map(|i| (i, 4)).chain((0..9).map(|i| (4, i))) {
|
||||
let (x, y) = (580 + dx, top + dy);
|
||||
text[y * stride + x / 8] &= !(0x80 >> (x % 8));
|
||||
}
|
||||
}
|
||||
}
|
||||
text
|
||||
})
|
||||
.collect();
|
||||
let images: Vec<ObjRef> = pages
|
||||
.iter()
|
||||
.map(|text| {
|
||||
let im = mono_image(&mut doc, w, h, text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", im)], "q 600 0 0 480 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let before: Vec<Vec<u8>> = (0..pages.len()).map(|p| render_page(&doc, p)).collect();
|
||||
let alone: usize = pages.iter().map(|text| pdfcraft_filters::encode_jbig2(text, w, h).unwrap().len()).sum();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
let report = optimize(&mut doc, &settings).unwrap();
|
||||
let globals = globals_of(&doc, images[0]).expect("the first page names the globals");
|
||||
assert!(images.iter().all(|im| globals_of(&doc, *im) == Some(globals)), "every page names the same globals");
|
||||
let shared = stream(&doc, globals).raw.len() + images.iter().map(|im| stream(&doc, *im).raw.len()).sum::<usize>();
|
||||
assert!(shared * 4 < alone * 3, "shared {shared} bytes, alone {alone}");
|
||||
assert_eq!(report.image_bytes_after, shared);
|
||||
for (p, before) in before.iter().enumerate() {
|
||||
assert!(render_page(&doc, p) == *before, "page {p} looks the same");
|
||||
let s = stream(&doc, images[p]);
|
||||
assert_eq!(decoded_image(&doc, &s).unwrap(), pages[p], "page {p} decodes to its samples");
|
||||
}
|
||||
// Optimized again, each page would come out larger alone than it is beside the globals: the
|
||||
// sharing stays.
|
||||
let streams: Vec<Stream> = images.iter().map(|im| stream(&doc, *im)).collect();
|
||||
let again = optimize(&mut doc, &settings).unwrap();
|
||||
assert_eq!((again.images_recompressed, again.images_resampled), (0, 0));
|
||||
assert!(images.iter().zip(&streams).all(|(im, s)| stream(&doc, *im) == *s));
|
||||
}
|
||||
|
||||
/// Images share a dictionary only where that saves more than it takes: two pages of five small
|
||||
/// letters save less than their globals would take, and stay as they are coded alone.
|
||||
#[test]
|
||||
fn a_dictionary_that_saves_less_than_it_takes_isnt_shared() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (600u32, 480u32);
|
||||
let images: Vec<ObjRef> = (1..=2u64)
|
||||
.map(|k| {
|
||||
let text = rendered_text_from(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15));
|
||||
let im = mono_image(&mut doc, w, h, &text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", im)], "q 600 0 0 480 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
let report = optimize(&mut doc, &settings).unwrap();
|
||||
assert_eq!(report.images_recompressed, 2);
|
||||
for im in &images {
|
||||
let s = stream(&doc, *im);
|
||||
assert_eq!((s.dict.name(b"Filter"), s.dict.contains(b"DecodeParms")), (Some(&b"JBIG2Decode"[..]), false));
|
||||
}
|
||||
// Optimized again, they code to what they are, no smaller: they stay.
|
||||
let streams: Vec<Stream> = images.iter().map(|im| stream(&doc, *im)).collect();
|
||||
assert_eq!(optimize(&mut doc, &settings).unwrap().images_recompressed, 0);
|
||||
assert!(images.iter().zip(&streams).all(|(im, s)| stream(&doc, *im) == *s));
|
||||
}
|
||||
|
||||
/// Only JBIG2 asked for shares: under Retain, JBIG2 pages drawn at 600 ppi are averaged and stored
|
||||
/// as JBIG2 again, each alone, as Retain changes nothing but their resolution.
|
||||
#[test]
|
||||
fn retained_jbig2_shares_nothing() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (1200u32, 960u32);
|
||||
let images: Vec<ObjRef> = (1..=4u64)
|
||||
.map(|k| {
|
||||
let text = font_text(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15));
|
||||
let mut d = Dict::new();
|
||||
for (key, value) in [("Type", Object::name("XObject")), ("Subtype", Object::name("Image")), ("ColorSpace", Object::name("DeviceGray"))] {
|
||||
d.set(key.as_bytes().to_vec(), value);
|
||||
}
|
||||
for (key, value) in [("Width", i64::from(w)), ("Height", i64::from(h)), ("BitsPerComponent", 1)] {
|
||||
d.set(key.as_bytes().to_vec(), Object::Int(value));
|
||||
}
|
||||
d.set(b"Filter".to_vec(), Object::name("JBIG2Decode"));
|
||||
let im = doc.add(Object::Stream(Stream::from_raw(d, pdfcraft_filters::encode_jbig2(&text, w, h).unwrap())));
|
||||
// 1200 samples over 144 pt: 600 ppi.
|
||||
page(&mut doc, &[("Im", im)], "q 144 0 0 115.2 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Retain, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
assert_eq!(optimize(&mut doc, &settings).unwrap().images_resampled, 4);
|
||||
for im in images {
|
||||
let s = stream(&doc, im);
|
||||
assert_eq!((s.dict.name(b"Filter"), s.dict.contains(b"DecodeParms")), (Some(&b"JBIG2Decode"[..]), false));
|
||||
}
|
||||
}
|
||||
|
||||
/// Sixteen images at most share one dictionary, and two are enough: of eighteen, the first
|
||||
/// sixteen share one and the other two another.
|
||||
#[test]
|
||||
fn sixteen_images_at_most_share_a_dictionary() {
|
||||
let mut doc = Document::new_empty();
|
||||
let (w, h) = (600u32, 480u32);
|
||||
let texts: Vec<Vec<u8>> = (1..=18u64).map(|k| font_text(w, h, k.wrapping_mul(0x9E37_79B9_7F4A_7C15))).collect();
|
||||
let images: Vec<ObjRef> = texts
|
||||
.iter()
|
||||
.map(|text| {
|
||||
let im = mono_image(&mut doc, w, h, text, None, &[("BitsPerComponent", 1.into()), ("ColorSpace", Object::name("DeviceGray"))]);
|
||||
page(&mut doc, &[("Im", im)], "q 600 0 0 480 6 300 cm /Im Do Q");
|
||||
im
|
||||
})
|
||||
.collect();
|
||||
let settings = Settings { mono: MonoSettings { compression: MonoCompression::Jbig2, ..MonoSettings::REDUCE }, ..Settings::default() };
|
||||
optimize(&mut doc, &settings).unwrap();
|
||||
let globals: Vec<Option<ObjRef>> = images.iter().map(|im| globals_of(&doc, *im)).collect();
|
||||
assert!(globals.iter().all(Option::is_some), "{globals:?}");
|
||||
assert!(globals[..16].iter().all(|g| *g == globals[0]) && globals[16..].iter().all(|g| *g == globals[16]), "{globals:?}");
|
||||
assert_ne!(globals[0], globals[16]);
|
||||
for (k, (im, text)) in images.iter().zip(&texts).enumerate() {
|
||||
assert!(decoded_image(&doc, &stream(&doc, *im)).unwrap() == *text, "image {k} decodes to its samples");
|
||||
}
|
||||
}
|
||||
|
||||
/// A JBIG2 scan goes through Monochrome Images like any black-and-white image. Above 450 ppi it
|
||||
/// is averaged as its Flate copy is, to the same samples, and stored as each compression asks:
|
||||
/// Group 4 or JBIG2, or Flate when that is smaller, or under Retain JBIG2 again; alone or behind another
|
||||
|
||||
@@ -10,8 +10,9 @@
|
||||
use pdfcraft_content::{Matrix, overlaps};
|
||||
use pdfcraft_cos::{Dict, Document, ImageBudget, Object, Stream};
|
||||
|
||||
/// Most pixels an image may have to be cleared in place (larger ones are removed).
|
||||
const MAX_PIXELS: u64 = 64 << 20;
|
||||
/// Most pixels an image may have to be cleared in place (larger ones are removed), or to be stored
|
||||
/// again without shared `/JBIG2Globals` (see [`without_globals`]).
|
||||
const MAX_PIXELS: u64 = pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS;
|
||||
|
||||
fn components(doc: &Document, s: &Stream) -> Option<usize> {
|
||||
let cs = doc.resolve(s.dict.get(b"ColorSpace")?);
|
||||
|
||||
@@ -304,6 +304,7 @@ Estimates unchanged.
|
||||
|
||||
| Date | Change | Summary |
|
||||
|---|---|---|
|
||||
| 2026-11-03 | minor | M11: Optimize's JBIG2 images share a symbol dictionary (`/JBIG2Globals`), up to 16 at a time and when smaller (`pdfcraft_filters::encode_jbig2_shared`): 41–46% smaller on rendered text in proportional fonts, 10% in Courier |
|
||||
| 2026-11-03 | patch | M5 #5850 review: indirect typewriter intent and composer edits preserve outer whitespace; movement regression commits by outside click; estimates unchanged. |
|
||||
| 2026-11-03 | patch | M5 #5850: Enter adds Fill & Sign line breaks; outside clicks commit, Escape cancels, and saved multiline appearances retain their text and height; estimates unchanged. |
|
||||
| 2026-11-02 | patch | M10 #5743: macOS print jobs carry the job name and owner some vendors' filters need; not yet checked on a physical printer |
|
||||
|
||||
@@ -7439,8 +7439,8 @@ milestone = "M11"
|
||||
status = "partial"
|
||||
commands = ["optimize.advanced", "optimize.reduce"]
|
||||
tools = ["doc_optimize", "doc_reduce"]
|
||||
evidence = ["crates/optimize/src/images.rs::flate_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::jbig2_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::an_image_too_large_for_the_jbig2_decoder_stays_as_it_is", "crates/optimize/tests/optimize.rs::what_is_too_large_for_jbig2_is_stored_another_way", "crates/optimize/tests/optimize.rs::group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit"]
|
||||
notes = "All of a run's images share one allowance of decoding, filters, decoders and pixels alike (about 1,000 scanned pages' worth, more for a longer file); those past it stay as they are, those the image pass read being counted in the report, which doc_optimize and doc_reduce return and the panel doesn't show yet. CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export; lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: images over 80 million pixels (left alone by the image steps, whatever their settings; Clean Up may still Flate an unfiltered or LZW one), inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples, JBIG2 past 65,535 samples on a side, where the decoder stops: such an image isn't stored as JBIG2 (the image step leaves a 1-bit one as it is unless it is averaged, and may store one packed from more bits as 1-bit Flate)."
|
||||
evidence = ["crates/optimize/src/images.rs::flate_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::eight_bit_black_and_white_images_become_one_bit", "crates/filters/src/ccitt.rs::bitmaps_round_trip", "crates/filters/src/ccitt.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/filters/src/ccitt.rs::what_it_cannot_encode_is_an_error", "crates/optimize/tests/optimize.rs::black_and_white_images_become_ccitt_group_4", "crates/optimize/tests/optimize.rs::group_3_fax_images_become_group_4", "crates/optimize/tests/optimize.rs::rgb_and_two_colour_indexed_black_and_white_images_become_one_bit", "crates/optimize/tests/optimize.rs::black_and_white_rgb_used_beyond_the_page_content_read_stays_rgb", "crates/optimize/tests/optimize.rs::black_and_white_rgb_drawn_only_where_the_optimizer_reads_becomes_grey", "crates/optimize/tests/optimize.rs::black_and_white_rgb_in_resources_its_page_tree_and_a_form_share_becomes_grey", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_ccitt_group_4", "crates/ui-egui/tests/it/create.rs::the_pdf_optimizer_dialog_saves_an_optimized_copy", "crates/filters/src/mq.rs::the_annex_h_test_sequence_encodes_to_its_bytes", "crates/filters/src/jbig2.rs::bitmaps_round_trip", "crates/filters/src/jbig2.rs::a_page_of_text_like_marks_round_trips_and_shrinks", "crates/optimize/tests/optimize.rs::black_and_white_images_become_jbig2", "crates/automation/tests/it/automation.rs::doc_optimize_stores_black_and_white_images_as_jbig2", "crates/filters/src/jbig2/symbols.rs::repeated_marks_are_stored_once", "crates/filters/src/jbig2/symbols.rs::scattered_marks_round_trip", "crates/optimize/tests/optimize.rs::rendered_text_becomes_jbig2_symbols", "crates/optimize/tests/optimize.rs::jbig2_scans_are_averaged_and_stored_again", "crates/optimize/src/images.rs::jbig2_images_past_the_runs_allowance_stay_as_they_are", "crates/optimize/tests/optimize.rs::an_image_too_large_for_the_jbig2_decoder_stays_as_it_is", "crates/optimize/tests/optimize.rs::what_is_too_large_for_jbig2_is_stored_another_way", "crates/optimize/tests/optimize.rs::group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit", "crates/filters/src/jbig2/symbols.rs::pages_in_one_font_share_one_dictionary", "crates/filters/src/jbig2/symbols.rs::what_a_page_has_alone_stays_its_own", "crates/filters/src/jbig2/symbols.rs::past_the_shared_limit_the_most_widely_held_marks_are_shared", "crates/optimize/tests/optimize.rs::jbig2_pages_share_one_symbol_dictionary", "crates/optimize/tests/optimize.rs::a_dictionary_that_saves_less_than_it_takes_isnt_shared", "crates/optimize/tests/optimize.rs::sixteen_images_at_most_share_a_dictionary", "crates/engine/src/tests.rs::redacting_an_optimized_jbig2_page_stores_the_rest_of_its_group_again", "crates/optimize/src/images.rs::only_what_redaction_can_store_again_shares_a_dictionary"]
|
||||
notes = "All of a run's images share one allowance of decoding, filters, decoders and pixels alike (about 1,000 scanned pages' worth, more for a longer file); those past it stay as they are, those the image pass read being counted in the report, which doc_optimize and doc_reduce return and the panel doesn't show yet. CCITT Group 4, lossless: PDF Optimizer ▸ Images ▸ Monochrome Images (on by default, so Reduce File Size does it too; doc_optimize mono) stores a 1-bit image of one colour component (grey, indexed, a separation) or an image mask, kept with no filter or general-purpose ones, as Group 4 when that is smaller as stored; the samples decode to the same bits, so colour space, decode array and masks stay. Packed to one bit first, then stored the same way (colour-key masks aside): eight-bit grey of only black and white, eight-bit DeviceRGB of only black and white (as DeviceGray; calibrated RGB, RGB with a profile, RGB drawn or listed where /DefaultGray or /DefaultRGB is in force or referred to from anywhere but the page and form resources read (an annotation's appearance or a pattern drawing it, directly or through resources or a form shared with a page; another image's /Alternates), and RGB whose soft mask has a /Matte aside), and an indexed image of 2, 4 or 8 bits that uses only palette entries 0 and 1 (keeping its palette), unless the grey or colour settings make it smaller still, as they can a dithered scan downsampled to JPEG, its soft mask counted. Group 4 wins on scans; on clean computer-drawn 1-bit art Flate can be smaller, and the image then stays. Group 3 fax images are stored again as Group 4 when that is smaller (damaged Group 3 with end-of-line codes as the decoder recovers it, as it reports no damage there; byte-aligned Group 3 with end-of-line codes but none before the first row, not as T.4 sends them, as PdfCraft draws it, which at /K 0 may be other rows than were coded). CCITT fax images are decoded to be downsampled (optimize.downsample-images), and a resampled image is stored as Group 4 or 1-bit Flate, whichever is smaller. Lossless JBIG2 with bounded exact-copy symbol coding is available too, with image export, up to 16 images sharing one symbol dictionary (/JBIG2Globals) where that is smaller still (image masks and DeviceGray images named directly, of at most 64 Mi pixels, which redaction can store again without it); lossy JBIG2 remains unsupported. JBIG2 images are read too: averaged like any (under Retain stored as JBIG2 again), and, not resampled, stored again as the compression asks when that is smaller (as JBIG2 symbols where its own coding is a generic region, say); a symbol-coded scan, averaged, loses its exact repeats and is often larger, and then stays as it is; one whose JBIG2 page isn't its size is left alone. Not yet: images over 80 million pixels (left alone by the image steps, whatever their settings; Clean Up may still Flate an unfiltered or LZW one), inline images, soft masks and /Mask images, Group 4 for rows wider than 32,767 samples, JBIG2 past 65,535 samples on a side, where the decoder stops: such an image isn't stored as JBIG2 (the image step leaves a 1-bit one as it is unless it is averaged, and may store one packed from more bits as 1-bit Flate)."
|
||||
|
||||
[[feature]]
|
||||
id = "optimize.only-if-smaller"
|
||||
|
||||
Reference in new issue
Block a user