* M11: JBIG2 images share a symbol dictionary, up to 16 at a time
With Monochrome Images set to JBIG2, Optimize stored each image's
letters in a symbol dictionary of its own, so a document of N pages
carried its font's shapes N times. Now up to 16 images the run stores
as JBIG2 share one dictionary in a /JBIG2Globals stream, holding the
marks found in more than one of them; each image's text regions refer
to it first, then to a dictionary of its own for the marks only it
repeats. An image is stored that way when it comes out smaller, and a
group only when what its images save is more than the globals take.
- filters: encode_jbig2_shared codes a group: the globals are a symbol
dictionary segment numbered 0 and associated with no page (7.2.6);
each page numbers its segments from 1, as a segment refers only to
lower numbers (7.2.5), and the shared symbols' IDs come first, as the
dictionaries are referred to in that order (Table 31, SBSYMS). At
most 16,383 shared marks, those in the most images, so a page keeps
room for its own under the decoder's 32,767.
- optimize: the run holds the 1-bit samples of the images it stores as
JBIG2 (16 at most, 64 MiB at most) and stores them again sharing,
weighing each with the longest reference the globals could have.
Eight pages of 10 pt text rendered at 300 dpi take 41% less in
Helvetica and 42% in Times than coded alone, 10% in Courier, and 46%
in Helvetica at 600 dpi; groups of 2, 4, 16 and 32 save 19%, 33%, 45%
and 47%. Redacting one image stores the others of its group again
without the dictionary, so 16 keeps that to 15 images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: test what the mutants found untested in shared JBIG2 dictionaries
- filters: the group goes through encode_jbig2_shared, which refuses it
for an image the encoder refuses; two marks in common are enough to
share; past the 65,536 distinct marks a group holds, a mark first seen
later isn't shared while those held count every image.
- optimize: optimizing again keeps both the shared pages and the pages
coded alone as they are; under Retain, averaged JBIG2 pages share
nothing; a last group of two shares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: only images redaction can store again share a JBIG2 dictionary
Redacting one image of a group stores the others again without the
dictionary, which can hold the shapes of marks under the area, and
refuses the redaction for one it can't: it takes image masks and
images of one component as its own reading counts them (no DeviceN,
the family named directly), and at most 64 Mi pixels. Optimize took
DeviceN of one colourant and up to 80 million pixels, so such an image
in a group made any redaction of the others fail, where before it was
coded alone and the same redaction worked (the Consumers review).
Only an image mask or a DeviceGray image named directly, of at most
2^26 pixels, now joins a group; others are coded alone. An engine test
optimizes three pages, two DeviceGray and one DeviceN, and redacts the
first: the DeviceN page stays alone, the second is stored again, and
no image names the dictionary after a save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: optimize and redaction share one limit on the images redaction stores again
The second Consumers review: Optimize's limit on images that share a
JBIG2 dictionary copied redaction's private 64 Mi, with nothing tying
the two, so a lower one in redaction would bring back refused
redactions with no test going red. Both now read
pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS. The engine test adds an image
mask, the other kind that shares, so a redaction that stopped storing
either kind again fails it. doc_optimize's description, the README,
the parity note and ROADMAP say which images share (named directly).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: a JBIG2 group is coded before its samples would pass the limit
CodeRabbit on #5910: hold added an image to the group and then checked
SHARED_BYTES, so a group could pass the 64 MiB it is documented to stay
within by its last image, which can itself be up to 64 MiB (an image
that shares has at most 2^26 pixels, and as its rows are whole bytes,
at most that many bytes; a 1-pixel-wide one takes them all). hold_in
codes the group first when an image would take it past the limit, and
after when it has 16 images; with room for 10 bytes, images of 4 go two
to a group, where the old order made one of 12. CodeRabbit's follow-up
too: the 18-image test decodes every image and compares it with its
samples, across the boundary between the groups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* optimize: count the globals' object framing; filters: test the shared coder's size limit
- A JBIG2 group shares a dictionary only when it saves more than the
globals' dictionary and data plus NEW_OBJECT (96 bytes: the object's
framing, its xref entry, and an encrypted file's IV and padding), so a
group that saves less than that no longer makes the file larger.
- encode_jbig2_shared is Unsupported for an image taller than the
decoder takes and codes one at the bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filters: drop the unused rows binding in encode_shared
page() no longer takes the rows since #5511.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filters: the shared coder's at-bound test says what it returns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
* M11: JBIG2 encodes only what its decoder takes: Optimize no longer loses large images
encode_jbig2 took rows of up to 2^20 samples and any number of them, while hayro-jbig2, which
draws and reads these streams, takes 65,535 on a side and 2^28 samples. The optimizer doesn't
read back what it stores, so with Monochrome Images on JBIG2, any 1-bit image with a side of
65,536 or more was replaced by a stream that draws nothing: 65,536 × 16 went from 37,875 dark
pixels to none, and was reported as recompressed.
The encoder now refuses what the decoder can't read, so such an image stays as it is. Symbol
coding's limit of 2^24 rows can't be reached any more, so it goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the JBIG2 boundary test checks a pixel
It counted rows, which a page decoded blank would pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the encoder's bound on samples is MAX_JBIG2_PIXELS, the decoder's
main brought decode_jbig2 and its public MAX_JBIG2_PIXELS into this crate,
2^28 like this branch's private MAX_JBIG2_SAMPLES and for the same reason.
One constant now bounds both, so the encoder can't drift past what the
decoder reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: an image too large for JBIG2 isn't stored as JBIG2, as the notes say
The parity note said such an image stays as it is. A 1-bit one does unless
it is averaged, but one packed from more bits may become 1-bit Flate. The
note now says it isn't stored as JBIG2 and cites
an_image_too_large_for_the_jbig2_decoder_stays_as_it_is, and the optimizer
README lists the bound under Not yet. MAX_JBIG2_PIXELS is the most the
decoder takes, not about that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the optimizer's own size limit, and tests for what JBIG2 can't take
The optimizer leaves images over 80 million pixels alone whatever the
settings, and nothing said so; the README and parity note say it now, and
that JBIG2's 2^28 in all is past it, so from Optimize the side bound is the
one that applies. The note says what the image step does: it leaves a
1-bit image past the bound as it is unless it is averaged, and may store
one packed from more bits as 1-bit Flate.
what_is_too_large_for_jbig2_is_stored_another_way tests those two cases:
eight-bit grey of black and white at 65,536 wide becomes 1-bit Flate, and
a 1-bit image at 600 ppi averaged to 70,000 wide does too. At 65,535 and
35,000 both become JBIG2.
The symbol cap's comment says why hayro-jbig2 refuses 32,768 symbols: it
caps new and exported symbols together at 65,535, and a plan exports every
symbol it codes. Measured: 32,767 decode, 32,768, 40,000 and 65,535 are
refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the 80-million-pixel limit is the image steps', tested on both sides
The README and parity note said images over 80 million pixels are left
alone whatever the settings, but Clean Up still Flates an unfiltered or
LZW one. They now say the image steps leave them alone, and the note cites
group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit for it.
(The last commit said the note also gives 2^28 as past that limit; only
the README does.)
what_is_too_large_for_jbig2_is_stored_another_way covered the width
bound only: it now runs tall images too, packed and averaged, and checks
the averaged ones are one bit and half the size both ways. The older test
uses the same bars() fixture, and its comment says such a stream drew
nothing in PdfCraft, not that the picture was lost: Poppler and
Ghostscript draw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the JBIG2 stored at the decoder's bound reads back, and grey keeps the pixel limit
- The optimize test at 65,535 on a side decoded nothing it stored: it
checked the filter only, and the coding stored there is symbols, not
the generic coding the filters test reads back. It now decodes the
stored stream and compares it with the bars.
- The README's 80-million-pixel sentence covers the colour and grey
steps too, and only the 1-bit step had a test. A grey image of
10,000 × 8,000 at 600 ppi is resampled; a row more stays as it is.
- The filters README names decode_jbig2_within, with its work, as main
renamed decode_jbig2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the filters README names #5569's other renames too
The README named decode_jbig2_within but still decode_ccitt, which
#5569 made private behind decode_ccitt_within, and gave
Document::decoded_image as both decoders' caller, which is test-only
now: ImageBudget::decode calls them, as lib.rs's doc says. The ccitt.rs
test line keeps decode_ccitt, which those tests call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(optimize): make pictures under a stencil mask smaller
An image with any /Mask was skipped by the colour and grey settings. That is
right for a colour-key mask, which names sample values that resampling and JPEG
would change. A stencil mask is an image of its own, with its own resolution
(ISO 32000-2 8.9.6.3), so the picture can change under it and the mask stays as
it is. Some exporters put one on every picture, even where it hides nothing, so
whole drawing sets of pictures were left at full size.
A mask that can't be read as a stencil, such as a reference to nothing, still
leaves its picture alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(optimize): only a mask that says /ImageMask true counts as a stencil
A stream /Mask without /ImageMask true let its picture be replaced, although a
mask that isn't a stencil is meant to leave its picture alone. It now does.
The stencil test also checks that the hole still shows what's behind it. The first
pixel all round is left out, because the coarser picture blends into it a little
(at most 26 of 255 in this test).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(optimize): draw a coloured page behind the stencil test's picture
On a white page, a hole painted white looks the same as an open hole, so the check
on the hole couldn't tell them apart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: quincy-jc <184324679+quincy-jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Reading a JPEG's header was free, and it can be most of the file: a mask of
1 MiB of tables that disagreed with its dictionary, shared by 4,000 images,
took 101 s with none refused, and that time grows as the square of the file.
A JPEG now pays 4 steps a byte before its tables are parsed, on every path, and
a mask's header is read once. That is a bound and not a speed-up: the same
file of 4,000 images fits the allowance and still takes 97 s, and 12,000 stop
after 10,233 in 250 s.
A chain that can't run pays nothing for its pixels, and the docs say what past
the allowance isn't counted: LZW streams the image pass didn't read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Four ways round the allowance, each measured by a review of hostile input:
A Flate stream read as zlib that fails at once, and then as raw deflate that
decodes everything before failing, gave nothing back and so was charged
nothing, here and in DecodeBudget: 110 s for a 387 KB file, all decoded. What
the retry decoded now counts, though the data is still dropped as it was.
Fax and 1-bit Flate images paid by the byte for work done by the pixel: 19
bytes can be a blank page of 100 million pixels, about a second each. Every
image decoded here now pays for its pixels first, as JBIG2 already did.
A JPEG paid nothing, and what one asks for doesn't grow with its length: 156
bytes declared 80 megapixels, 72 s for an 8 KB file. A JPEG pays for its
samples before it is decoded, in Optimize and in an export that has to decode
one for its mask.
Merging fragments paid for reading strips and not for the picture it built:
393 s for 1,154 bytes. It pays for the picture before allocating it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Storing LZW streams with Flate decoded image streams outside the allowance,
with a limit for each stream and no total, and nothing asked whether to go
on: a 593 KB file of 100 images took 113 s with none refused, and images
refused by the image pass were still rewritten. The pass now pays for what
it decodes and for storing and checking it, at 8 steps a byte since Flate
takes up to 78 ns a byte, and streams past that stay LZW.
An image is counted as refused only when it stayed as it was: one packed
before the allowance ran out on its second read was rewritten. doc_reduce
reports images_refused as doc_optimize does, and the app's action says so
in its log. A refused image no longer reads as an unsupported filter:
FilterError::Spent says what happened.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Filters were charged only for what they handed a JBIG2 decoder, once they
had handed it over: a chain that failed, or went past its image's limit,
cost its time and nothing else, and images that weren't JBIG2 weren't
charged at all. A 734 KB file of 200 images of Flate inside Flate took an
image export 111 s, and that grew with the file. It now takes 12 s.
Each byte read and each byte a stage decodes is now a step of the same
allowance, paid as it happens, so no stage runs past what is left. That
holds for every image read through ImageBudget, and a fax image pays for
its rows and its code. JBIG2 globals pay when they have to be read, and
kept ones don't. Optimize's two reads that went round the budget go
through it, and merging fragments shares it.
Optimize reads an image twice, to pack it and to resample it, so a refused
image is now counted once a run rather than once a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From a review of who reads a refusal.
The rule is to leave it out and say so, and it held on some paths only. An image whose JBIG2 mask is
refused no longer comes out opaque. The first image past the allowance reads as that and not as
corrupt. A page reports what it left out. The Office warning stopped blaming forms. Redaction refuses
the run, leaving the document as it was: removing a readable image whole would take what no mark
covers with it, unreported.
The 2 GiB an export holds is twice the document if that is more, so a large honest file loses nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images
made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes
them again.
Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its
page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before
anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes
it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for
each byte of the file, so a long scan is read whole. An image past it spends all that's left, and
later ones aren't read, globals included.
Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it
out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the
picture out and says the page was truncated. Redaction removes it whole, as any image it can't read.
Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts
afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73).
It's a thread-local, unbounded until a page sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
An ordinary scan stored as JBIG2 was decoded for Group 4 by default, at 0.26 s a page at 300 ppi,
and came out unchanged: Group 4 of a symbol-coded page was 201,132 bytes against 17,291. Over a
500-page scan that is minutes for nothing. Dropping the guard changed no output, as a cost guard
wouldn't, so its test counts the pixels decoded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From a review of the spellings an indexed image can take:
- The palette was read for every 1-bit indexed image, resampled or not, and twice each. Images
share a table, so 1,000 images sharing one with 16 MiB of raw data took 16 to 60 s, none of
them resampled. It is now read only for an image about to be resampled, and a table over
4 KiB is refused before it is copied or decoded, since no palette needs that.
- A colour key can hide the dark sample a tie goes to, so a thin rule of the other was lost: a
white rule on a hidden black went from 255 to the backdrop's 128. An image with a colour key
now keeps its resolution. Grey had this since #5426 and is covered too.
- `/Indexed` by reference, and a base of `[/DeviceRGB]`, are the same colour space and are read
so. `/I`, an inline abbreviation, and a real hival aren't, and such an image is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From CodeRabbit's review: default colour spaces apply to an indexed colour space's
device base, so one in force where an annotation's appearance (or a pattern, or
another image's /Alternates) draws the image may change which palette entry is darker,
and the tie a resampled image keeps. Only black-and-white RGB went through
used_elsewhere; indexed images now do too when downsampling is on, and one used beyond
the page and form resources the walk read keeps its resolution, as such RGB keeps its
colour space.
A grey image stays out of this check, since sample 0 stays dark in a grey space, and #5426's rule
for grey is unchanged: the test now has a grey image drawn on the page and in an annotation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
#5461 made Document::decoded_image read JBIG2, page exactly the image's size, and
kept the optimizer's monochrome step off it until it could weigh JBIG2 again. Now
one_bit reads JBIG2 last after any filters, within the raster's allowance, and mono
treats it as it does any black-and-white image:
- above the threshold it is averaged, and stored as the compression asks, the
smallest of what that may store; under Retain it is stored as JBIG2 again, its kind
of compression kept;
- not resampled, it is stored again as the compression asks when that is smaller: a
generic region becomes symbols with JBIG2 asked for, and JBIG2 behind ASCIIHex is
stored again alone (as Group 4 where that is smaller);
- a JBIG2 image stored again refers to no /JBIG2Globals; one whose page isn't its
size isn't read, and stays.
jbig2_images_are_left_as_they_are becomes jbig2_scans_are_averaged_and_stored_again:
a JBIG2 scan at 600 ppi, alone, behind ASCIIHex and with globals, averages to the
same samples as its Flate copy under each compression (Retain keeping JBIG2); at 300
ppi a generic region of rendered text stays under Group 4 (larger) and Retain and
becomes symbols under JBIG2 (under half the size); behind ASCIIHex it is stored again
alone, as Group 4 where the hex is spaced out; a taller page stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5426 left indexed images at their resolution rather than guess which sample is
dark. The palette says: dark_is_one now maps samples 0 and 1 through the decode array
to palette entries (rounded and clipped to hival, ISO 32000-2 §8.9.5.2) and compares
how light those entries are (`lightness`: the grey, or Rec. 601 luma over DeviceRGB).
Only DeviceGray and DeviceRGB bases count, the table a string or a stream (within
4 KiB), directly or by reference: there the palette's values are the colour, and
under /DefaultGray or /DefaultRGB no polarity is asked for. Other bases (CMYK,
calibrated, ICC-based, a resource name), two entries alike, an entry the table
doesn't hold or a hival past 255 still say nothing; separations and DeviceN still
aren't averaged, their tint transforms unread. #5426's tone guard applies as to any
image.
#5426's indexed thin-rule test now checks what it was for: with the palette either
way round and the decode array either way, the averaged image holds the averaged
DeviceGray one's samples (flipped where its samples are) and renders the same. A unit
test covers each base, both orders, an inverting decode array, the weights of luma,
the table by reference and in a stream, and every refusal; the older polarity test's
doc says what is known now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decode_jbig2 sits next to decode_ccitt, through hayro-jbig2 as Export ▸ Images
used it, and shares CCITT's row sink. The JBIG2 page has to be the image's size:
PdfCraft's renderer draws the whole page into the image (black box: a 4-row page
behind /Height 2 fills the image box with all four rows), so the rows of a taller
page aren't the image's. The decoder decodes a page whole, so the page is checked
before it is decoded: that size, within the caller's bound as rows of one bit, and
at most 2^28 pixels (MAX_JBIG2_PIXELS; the decoder keeps a bit a pixel, about 34 MB
there).
Document::decoded_image_within decodes a JBIG2 image as it does a fax image: the
filters before it undone first, then the decoder with the /JBIG2Globals of the
image's own /DecodeParms entry (/Crypt counts in the position; fax_params and the
globals now share last_filter_parms). Globals have a limit of their own,
MAX_JBIG2_GLOBALS (32 MiB, moved from Export), and globals that can't be read are an
error rather than ignored.
Export ▸ Images reads JBIG2 through it, so its own decoder, row sink and DecodeParms
reader go, and hayro-jbig2 leaves pdfcraft-create. JBIG2 behind another filter now
exports, and so copies (Copy Image uses the same function); a page taller than the
image, which Export cut to the image's rows, now skips the image, as one of another
width did. doc_export_all_images' description no longer lists JBIG2 inside other
filters as skipped.
The optimizer's monochrome step read images through decoded_image_within and
relied on JBIG2 failing there: it now leaves JBIG2 images alone explicitly, as
before (a resampled one would come out Flate under Retain), and the README and
parity note list them as not yet handled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The integration in #5426 made downsampling conservative: dark_is_one answers only for
DeviceGray and image masks (a decode array of two different numbers or none), and not
under /DefaultGray or /DefaultRGB, and a square of four new pixels that each cover both
samples keeps the image at its resolution. The README, the crate docs, mono()'s comment
and the optimize.downsample-images parity note still said separations and indexed
images were averaged, a tie going to full ink or entry 0.
Measured on main: a 600 ppi page of the ragged-text fixture is no longer averaged
(it is with the square guard switched off), so the docs say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Includes #4325#1952#4180#2529#4320#4805, preserving their original histories.
Keep paragraph alignment and hard/soft wrap metadata together; retain CCITT/downsampling/RGB ownership guards with bounded JBIG2 export and exact-copy symbol encoding.
Combined interaction regressions added. Validation pending.
Packing makes an RGB image grey for every use at once, but /DefaultGray and
/DefaultRGB were checked only where the walk reads: page content and forms.
An annotation's appearance, a pattern, a Type3 glyph or a soft mask's group
drawing the same image can have other default colour spaces, so its colours
could change there.
The walk now records, by object number, every reference in the resources it
reads (each /XObject entry, drawn or not, an /XObject dictionary under its
resources, resources under a page, page-tree node or form), reading the
resources of every page and of every node above one, each page and node a
root. An image the walk reached stays covered unless an object reachable
from the trailer refers to it, or to an object on its way from a page, other
than through those references, or it is held by one that isn't covered; an
eight-bit DeviceRGB image that isn't covered stays RGB. So resources, an
/XObject dictionary or a form a page shares with an appearance keep it RGB,
as does a reference with another generation, while LibreOffice's layout (one
resources object for a page, its /Pages node, and a form inside it that
names it again) still packs. The scan runs only when packing is on and there
is such an image.
The images drawn or listed where the defaults are in force are kept by
object number too, so one named with another generation there stays RGB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A review found that a dense page came out as a stream PdfCraft's own
renderer refuses, so the page drew blank after optimizing: hayro-jbig2
decodes a text region of at most 10,000 symbol instances, and refuses
32,768 symbols or more. The stream was valid (Poppler decoded it), but
every mark went into one text region and up to 65,536 symbols were kept.
A 300 dpi letter page of the optimizer test's own text crossed it.
A page with more than 10,000 marks is now placed by several text
regions, each over a band of rows and all referring to the one symbol
dictionary, which each region but the last keeps retained (7.2.4). Past
32,767 distinct symbols the image is coded generically.
New tests: 25,000 copies placed by three regions, which decode; the
symbol caps decoded, not just reached; and in the optimizer, a whole
letter page of rendered text that looks the same as JBIG2 symbols (it
drew blank before). The READMEs say "still looked at" for the caps, and
that the scans figure is the six pages' total (20% on one page).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
A second review found two rules tested one way only: a mark 257 tall
(not just wide) is left to the generic region, which also bounds the
bitmap allocated for each mark, and runs joined at the lower-left corner
are one mark. The test now has a 1 x 257 column and an anti-diagonal.
The caps on distinct symbols (65,536) and their pixels (2^24) are tested
at and past them, as the run and row caps are.
The scans figure now names its noise: with 2% or more of the letters'
edge pixels flipped, symbols save at most 8% over the generic region.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Review found behaviour the tests let break unnoticed (each was a
surviving mutant, and each new test fails on it):
- black padding bits: a run reaching past the row would stop symbol
coding on many scans;
- Annex A's 32-bit range and PREV at exactly 256, where a decision would
go uncoded: coordinates past 4,435 on a 12,000-pixel-wide and a
10,050-row page, and the contexts of 4436's coding;
- marks joined at a corner are one mark; 256 pixels a side is a symbol,
257 is left to the generic region;
- the run and row caps, at and past them.
The roadmap's status line for this change now comes first, and the
"about a third smaller than Group 4 on scans" figure is replaced by what
was measured: 27-33% on the same pages with scan-like noise added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
pdfcraft_filters::encode_jbig2 stores each distinct mark of a 1-bit
image (8-connected black pixels, at most 256 a side) once, as a symbol
in a symbol dictionary, and places its exact copies with an immediate
lossless text region that refers to it. Larger marks, and in a second
plan marks that occur once, go to a generic region over their box. It
keeps whichever of that and the generic coding (encode_jbig2_generic,
unchanged) is smaller. Only exact copies share a symbol, so the stream
decodes to the same bits.
Written from the JBIG committee's ISO/IEC 14492 Final Committee Draft:
6.4 and 6.5 (text region and symbol dictionary decoding), 7.2, 7.4.2
and 7.4.3 (segment headers and syntax) and Annex A (the integer coders,
checked against its worked example and Table A.1).
On six pages of rendered 300 dpi text the output is 28% of the generic
coding's size and 17% of Group 4's, at about 100 ms a page; hayro-jbig2
and Poppler (pdfimages) both decode it to the original bits. Scans,
whose copies of a letter are seldom identical, gain little: that needs
refinement coding. A dictionary shared between pages (/JBIG2Globals)
is a further step.
The optimizer's Monochrome Images > JBIG2 (lossless) and doc_optimize
mono jbig2 use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
With end-of-line codes, the CCITT decoder picks a bad row up again at
the next code and reports no damage (an invalid code, a row of too many
pixels: probed as a black box), so such an image is stored again as
Group 4 the way it decodes. Without them, damage stops the decoding,
too few rows come out, and the image is left alone. Telling damage
apart would take a Group 3 parser of our own; the optimize README and
the mono-compression parity note say so instead.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- Packed to one bit, black-and-white DeviceRGB became DeviceGray. Where
/DefaultGray or /DefaultRGB is in force (ISO 32000-2 §8.6.5.6), that
drew it otherwise: through DefaultGray as grey, or no longer through
DefaultRGB. The page walk now notes the images drawn, on any use, where
a page's or form's resources (an enclosing one's included) name
either, and those stay RGB. When a use couldn't be followed, every
image counts as such.
- One whose soft mask has a /Matte stays RGB too: the matte has one
number for each of the parent's components, three, and grey has one.
- The optimize README, the mono-compression parity note and doc_reduce's
description said Group 3 to Group 4 wasn't done yet, that only images
packed from eight bits could become Flate, and that the colour space
and decode array always stay. They now match the code.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Monochrome Images only took eight-bit grey of only black and white
besides 1-bit images, so the same scan written as RGB, or as an indexed
image of two palette entries, stayed at eight bits (and RGB went to the
colour settings, as JPEG). Now these are packed to one bit first:
- eight-bit DeviceRGB whose every pixel is black or white becomes
DeviceGray; a decode array the same for each component becomes that
pair, another stops it. Calibrated RGB and RGB with a profile stay.
- an indexed image of 2, 4 or 8 bits that uses only palette entries 0
and 1 keeps its palette, whatever the two colours; a decode array
other than the default stops it.
Colour-key masks still stop all of them, and Retain leaves them to the
grey and colour settings. They are decoded through /DecodeParms given
by reference too (Document::with_direct_parms), and when downsampled a
tie goes to the dark sample of the image as packed: packed RGB is grey,
with grey's decode pair.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
A CCITT fax image that wasn't resampled was left as it was, so Group 3
images (the 1-D and 2-D codings fax software writes) stayed Group 3.
Group 4 codes each row against the one above and is usually much
smaller. Now only Group 4 alone skips decoding; Group 3 is stored again
as Group 4 when that is smaller as stored, and Retain still keeps it.
It reads an image's CCITT parameters (its /DecodeParms entry with Table
11's defaults) through pdfcraft_cos::Document::fax_params, from #4384.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
decode_ccitt reads byte-aligned Group 3 whose data has end-of-line codes
but none before the first row (not as T.4 sends them) on bytes. At /K 0
with the fill ending each code on a byte, about one small picture in
five is then read as other rows; at fax width almost none are, as such
pages come out short and are refused. Export already said so. The
optimizer, when it resamples such an image or stores it again as Group 4
from behind another filter, and redaction, when it clears pixels in one,
write those rows back, and nothing said that.
Black-box, PdfCraft's renderer draws such an image exactly as
decode_ccitt reads it (all 42 misread of 200 seeded pictures, behind
ASCII85Decode too), so the written image is what PdfCraft showed before.
Refusing such data instead (a code after nearly every row but none
first) would also refuse some ordinary Modified Huffman images, whose
rows can meet 11 zeros by chance (0.5 to 5% of small ones in a
black-box probe), so the limit is stated rather than guarded: in
decode_ccitt's doc, the filters, optimize and redact READMEs, and the
core.image-filters and mono-compression notes. The optimize README's and
the mono-compression note's "not yet: Group 3 to Group 4 without
resampling" now say a lone Group 3 image: one behind another filter is
stored again as Group 4.
Also from the review: docs/roadmap.md keeps #4335's status line under
this PR's, rather than replacing it; the filters README's tests and the
core.image-filters evidence name the Group 3 layout test; the filters
crate's description names Group 3 and 4 decoding.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- The optimize README, the crate doc and both parity notes say a
halftone or dither, and an image wider than a fax row, isn't averaged
down.
- The README's "Left alone" list is the colour and grey settings'; image
masks, decode arrays and CCITT go to Monochrome Images now. Its
monochrome "Not yet" names an image's own /SMask or /Mask image again
(only images drawn on a page are visited), as the parity note does;
the parity note's row limit is Group 4's.
- docs/architecture.md says CCITT decodes through pdfcraft-filters for
export, redaction and the optimizer, not only the renderer.
- docs/target-app-parity.md no longer lists monochrome downsampling as
missing.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
doc_reduce, the engine's reduced_bytes doc, the optimize README's
presets and the optimize.reduce parity note still said every image
above 225 ppi goes to 150 ppi; black-and-white images go to 300 ppi
above 450 ppi, as CCITT Group 4 when smaller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Create PDF and the optimizer now store black-and-white scans as Group 4,
and redaction couldn't decode CCITT, so redacting one word on such a
scan removed the whole page image. A CCITT image (the last filter,
after any others) now decodes through Document::decoded_image and is
cleared in a Flate copy like any other image; one that can't be decoded
in full is still removed whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README said an indexed image's palette decides which entry is the
dark one on a tie, but dark_is_one never reads the palette: entry 0
wins whatever its colour. The README, the function's comment and the
parity note now say so (from FredApps' review).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 2 × 2 tie always gave sample 0, black in grey and the painted part of
an image mask with the default decode array. With an inverting one,
[1 0], 0 is white or unpainted, and in a separation 0 is no ink, so a
line half a new pixel wide could vanish. dark_is_one says which sample
is dark from the colour space and decode array: 0 in grey and of a
mask, 1 in a separation or a DeviceN of one colourant, each the other
way round when the decode array inverts; an indexed image keeps entry
0, as its palette, not the decode array, says which is darker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
The PDF Optimizer and Reduce File Size downsampled colour and grey images
but not black-and-white ones. Settings::mono is now MonoSettings {
downsample, target_ppi, above_ppi, compression }, 300 ppi above 450 ppi
and CCITT Group 4 by default (Monochrome Images ▸ Downsample to;
doc_optimize mono: { downsample, ppi, above_ppi, compression }).
A 1-bit image or image mask (no filter, general-purpose filters, or CCITT
fax after any of them) and eight-bit grey of only black and white are averaged by
area: each new pixel takes the sample covering more of it, a tie 0, so a
dark line half a new pixel wide stays. They are stored as Group 4, or as
1-bit Flate when that is smaller; with Retain a resampled image keeps its
kind of compression. Images with a soft mask keep their size.
CCITT decoding moves from Export ▸ Images into pdfcraft-filters
(decode_ccitt, CcittParams, through hayro-ccitt), and
pdfcraft_cos::Document::decoded_image reads a fax image's /DecodeParms
for it (undoing filters before CCITT, such as ASCII85Decode, with any
/DecodeParms given by reference made direct first), so export
and the optimizer share one reader, and Group 4 scans can be
downsampled. A 600 ppi text page went from 79 KB to 40 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
- A noisy 640 × 800 soft mask under 1200 × 1500 text: resampled to the
image's new size it shrinks only a little, by less than the grey
settings' JPEG adds over the packed image, so packing wins; not
counting the new mask passed every test before.
- The hex-text half of the bomb test also stores eight-bit
black-and-white, which must pack, as its comment says.
- The optimize README, the mono-compression parity note and both
progress logs say the soft mask counts.
- The Copy Image parity note no longer lists CCITT as missing: it
copies through Export's decoder, which reads CCITT now.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
An eight-bit grey image of only black and white was packed to one bit
and stored, and the grey settings never saw it. For a dithered scan that
is bigger than what Reduce File Size made of it before, downsampled to
JPEG: 1-bit Flate can't shrink the dots (a 1200 × 1500 dither at 300 ppi
was 157 KB packed, 67 KB as before). Both are made now, and the smaller
as stored wins, counting a soft mask resampled with the image. Clean
text still packs to one bit. The packing also decodes no further than
its samples can need, as the other image steps do now.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Group 4 images (now written by Create PDF and the optimizer) would
otherwise stop exporting: they were reported as "can't be exported
yet". A CCITT fax image (its only filter) is decoded through
hayro-ccitt with its /DecodeParms (K, Columns, Rows, EndOfBlock,
EndOfLine, EncodedByteAlign, BlackIs1), masks included; /Columns has to
be the image's width, so a hostile value can't size the buffers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
Export ▸ Images skips an image whose JBIG2 page has another width or
more pixels than an export takes; a JBIG2 stencil mask refused the same
way is left out and the image exported opaque, as JPX and CCITT masks
are, which the export test now checks. The README and parity note said
only "is skipped". And redaction doesn't decode JBIG2 or Group 4 yet,
so a mark over part of an image the optimizer stored either way removes
the whole image; the optimizer README and parity note now say so (#4384
adds Group 4). Found in pre-push review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The optimizer README's and optimize.mono-compression's "not yet" lists
said rows wider than 32,767 samples aren't stored. That is Group 4's
limit (MAX_G4_COLUMNS); JBIG2 takes rows of up to 1,048,576
(MAX_JBIG2_COLUMNS). Found by CodeRabbit on #4805.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
pdfcraft_filters::encode_jbig2_generic writes 1-bit rows as a JBIG2
stream for JBIG2Decode (embedded organisation: a page information
segment and an immediate lossless generic region, no globals): template
0 with its nominal AT pixels, typical prediction, and the MQ arithmetic
coder (mq.rs), written from the JBIG committee's ISO/IEC 14492 Final
Committee Draft. The coder reproduces its Annex H.2 test sequence byte
for byte; output round-trips through hayro-jbig2, and Poppler renders
it the same as the original.
The optimizer's Monochrome Images gets JBIG2 (lossless) beside CCITT
Group 4 (still the default), doc_optimize mono jbig2; a document below
PDF 1.4 gets /Version /1.4 in its catalog. On 300 dpi text JBIG2 is
about a third smaller than Group 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
- Not only Export ▸ Images refuses a CCITT image until #4335: so does
everything that exports an image through it. Save Image As refuses
it, Copy Image fails, and Word, PowerPoint and HTML export leave the
picture out, with the page not reported. The optimize README and the
mono-compression parity note name them.
- optimize.mono-compression is partial now, so docs/target-app-parity.md
counts it on the table's own measurement: K Optimize and standards 15
shipped and 11 partial of 58 (35.3%), breadth 57.0% (423 + 75 of 808),
mainstream 63.3% (292 + 45 of 497), with a status line and a revision
row. ROADMAP.md's breadth rows and its K row say the same.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- Redaction doesn't decode CCITT, so a mark over part of an image the
optimizer stored as Group 4 removes the whole image (nothing leaks),
and Export ▸ Images refuses it. The optimize README and the
mono-compression parity note say so. Later PRs of this stack add both
(#4335 export, #4384 redaction).
- doc_reduce's description, Engine::reduced_bytes and docs/cli-batch.md
now list the black-and-white step; docs/target-app-parity.md's
optimize row gives the measured counts and no longer lists CCITT as
missing; the M11 milestone row lists it as done.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Document::with_direct_parms resolved an indirect /DecodeParms and its
array entries but not the numbers inside (/Predictor 9 0 R), which
Dict::int doesn't follow. A 1-bit image with such a predictor was
decoded without it, its rows shifted by their tag bytes, and Group 4 of
that stored in its place; resampling read images the same way. It now
keeps, of each parameters dictionary, the numbers and flags the decoders
read, resolved, and /JBIG2Globals as a reference. Only those small
values are copied (an entry that is no dictionary is null), so a file
naming one big object from every key and filter can't have it copied a
hundred times.
The change is #4384's, moved down so that every PR stacked on this one
reads such parameters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Stream::decoded reads only direct parameter dictionaries, so an image
whose /DecodeParms is a reference (whole, or as array entries) was
decoded without its predictor. For a 1-bit image with a PNG predictor
the rows came out shifted by their tag bytes, Group 4 of that was still
smaller, and the picture was replaced by a skewed one; resampling and
merging image strips read the same way.
pdfcraft_cos::Document::with_direct_parms gives the stream with the
parameters its filters use made direct: only the entries at the
filters' places, and none for a chain longer than any decoder takes,
so at most MAX_CHAIN objects are copied. The optimizer's pixels() and
the Group 4 step decode through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_g4 encodes packed 1-bit rows as Group 4 (ITU-T
T.6, /K -1) through the fax crate (MIT, already in the tree via tiff),
with rows capped at 32,767 samples so its 16-bit positions can't
overflow; round trips are checked against hayro-ccitt.
The optimizer (Settings::mono, CCITT G4 by default, so Reduce File Size
too) re-encodes a 1-bit image of one colour component, or an image
mask, stored with no filter or general-purpose ones, when that is
smaller as stored. The samples decode to the same bits, so colour
space, decode array and masks stay. PDF Optimizer gets a Monochrome
Images row and doc_optimize a mono option. optimize.mono-compression
goes from planned to partial (JBIG2 and monochrome downsampling not
yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
ROADMAP.md: kept both progress lines.
crates/optimize/README.md: PR's decoder sentence plus the target-version section from main.
Cargo.lock: auto-merged.
Conflicts: ATTRIBUTION.toml: took main's side (hashes recomputed after the merges). crates/optimize/src/lib.rs, crates/automation/src/lib.rs, tools.rs: Settings and tool schema keep object_streams (#3227) and lzw_to_flate (#4181); doc_optimize description names both options. crates/optimize/tests/optimize.rs: both sides' imports and tests kept.
Conflicts: docs/roadmap.md: all status lines kept, M10 row from main, M11 row with #3227's two sentences. parity/acrobat-features.toml: save-with-object-streams entry from #3227 with its test paths under tests/it/.
