A refinement whose reference is offset right of it (GRREFERENCEDX above
0, which a refined text instance has from a wider bitmap, ISO/IEC 14492
Table 12, and a refined symbol from an RDX above 0, Table 18) read the
reference's first 32 columns as 0. hayro-jbig2 emptied its buffers of
the reference left of it and, at its first column, didn't load them
again, so the refinement decoded to other pixels than were coded, or
failed and lost the page.
Vendored patch 15 (13 is #5999, 14 #6016): left of the reference the
buffers keep the words of its first column each row starts with. With
an offset of 0 or less nothing changes.
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
* M11: JBIG2 images share a symbol dictionary, up to 16 at a time
With Monochrome Images set to JBIG2, Optimize stored each image's
letters in a symbol dictionary of its own, so a document of N pages
carried its font's shapes N times. Now up to 16 images the run stores
as JBIG2 share one dictionary in a /JBIG2Globals stream, holding the
marks found in more than one of them; each image's text regions refer
to it first, then to a dictionary of its own for the marks only it
repeats. An image is stored that way when it comes out smaller, and a
group only when what its images save is more than the globals take.
- filters: encode_jbig2_shared codes a group: the globals are a symbol
dictionary segment numbered 0 and associated with no page (7.2.6);
each page numbers its segments from 1, as a segment refers only to
lower numbers (7.2.5), and the shared symbols' IDs come first, as the
dictionaries are referred to in that order (Table 31, SBSYMS). At
most 16,383 shared marks, those in the most images, so a page keeps
room for its own under the decoder's 32,767.
- optimize: the run holds the 1-bit samples of the images it stores as
JBIG2 (16 at most, 64 MiB at most) and stores them again sharing,
weighing each with the longest reference the globals could have.
Eight pages of 10 pt text rendered at 300 dpi take 41% less in
Helvetica and 42% in Times than coded alone, 10% in Courier, and 46%
in Helvetica at 600 dpi; groups of 2, 4, 16 and 32 save 19%, 33%, 45%
and 47%. Redacting one image stores the others of its group again
without the dictionary, so 16 keeps that to 15 images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: test what the mutants found untested in shared JBIG2 dictionaries
- filters: the group goes through encode_jbig2_shared, which refuses it
for an image the encoder refuses; two marks in common are enough to
share; past the 65,536 distinct marks a group holds, a mark first seen
later isn't shared while those held count every image.
- optimize: optimizing again keeps both the shared pages and the pages
coded alone as they are; under Retain, averaged JBIG2 pages share
nothing; a last group of two shares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: only images redaction can store again share a JBIG2 dictionary
Redacting one image of a group stores the others again without the
dictionary, which can hold the shapes of marks under the area, and
refuses the redaction for one it can't: it takes image masks and
images of one component as its own reading counts them (no DeviceN,
the family named directly), and at most 64 Mi pixels. Optimize took
DeviceN of one colourant and up to 80 million pixels, so such an image
in a group made any redaction of the others fail, where before it was
coded alone and the same redaction worked (the Consumers review).
Only an image mask or a DeviceGray image named directly, of at most
2^26 pixels, now joins a group; others are coded alone. An engine test
optimizes three pages, two DeviceGray and one DeviceN, and redacts the
first: the DeviceN page stays alone, the second is stored again, and
no image names the dictionary after a save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: optimize and redaction share one limit on the images redaction stores again
The second Consumers review: Optimize's limit on images that share a
JBIG2 dictionary copied redaction's private 64 Mi, with nothing tying
the two, so a lower one in redaction would bring back refused
redactions with no test going red. Both now read
pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS. The engine test adds an image
mask, the other kind that shares, so a redaction that stopped storing
either kind again fails it. doc_optimize's description, the README,
the parity note and ROADMAP say which images share (named directly).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: a JBIG2 group is coded before its samples would pass the limit
CodeRabbit on #5910: hold added an image to the group and then checked
SHARED_BYTES, so a group could pass the 64 MiB it is documented to stay
within by its last image, which can itself be up to 64 MiB (an image
that shares has at most 2^26 pixels, and as its rows are whole bytes,
at most that many bytes; a 1-pixel-wide one takes them all). hold_in
codes the group first when an image would take it past the limit, and
after when it has 16 images; with room for 10 bytes, images of 4 go two
to a group, where the old order made one of 12. CodeRabbit's follow-up
too: the 18-image test decodes every image and compares it with its
samples, across the boundary between the groups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* optimize: count the globals' object framing; filters: test the shared coder's size limit
- A JBIG2 group shares a dictionary only when it saves more than the
globals' dictionary and data plus NEW_OBJECT (96 bytes: the object's
framing, its xref entry, and an encrypted file's IV and padding), so a
group that saves less than that no longer makes the file larger.
- encode_jbig2_shared is Unsupported for an image taller than the
decoder takes and codes one at the bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filters: drop the unused rows binding in encode_shared
page() no longer takes the rows since #5511.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filters: the shared coder's at-bound test says what it returns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
vendor/README.md: patch (13), from #5999, goes into this branch's
hayro-jbig2 row ahead of (14) and (16).
The page-area test now fills the page's words with 1s, padding included:
since patch (13) new_with clears the padding, so the test no longer
exercised the page-width half of the mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* M2: a JBIG2 refinement reads 0 past its reference's width
hayro-jbig2 keeps each row of a bitmap in 32-bit words and filled a new
bitmap whole, so with a default pixel of 1 the bits past its width in
each row's last word were 1s too. A refinement reads its reference a word
at a time, those bits included, where ISO/IEC 14492 6.3.5.2 says a pixel
outside the reference is 0. A refinement of the page, or of an
intermediate text or halftone region, whose default pixel is 1 and whose
width isn't a multiple of 32 decoded to other pixels than were coded.
Vendored patch 13: Bitmap::clear_padding clears those bits when
new_with or reinitialize fills a bitmap with 1s, as cutting symbols and
patterns out of a collective bitmap already did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M2: the doc on a JBIG2 bitmap's words says their padding is 0
Three places in hayro-jbig2 now clear the bits past a row's width, and
the field's doc said only that rows are padded. It now says the padding
is 0 and why: generic and refinement decoding read the rows above their
own pixels a word at a time, as a refinement reads its reference, so
whatever next writes whole words keeps it 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* M11: JBIG2 encodes only what its decoder takes: Optimize no longer loses large images
encode_jbig2 took rows of up to 2^20 samples and any number of them, while hayro-jbig2, which
draws and reads these streams, takes 65,535 on a side and 2^28 samples. The optimizer doesn't
read back what it stores, so with Monochrome Images on JBIG2, any 1-bit image with a side of
65,536 or more was replaced by a stream that draws nothing: 65,536 × 16 went from 37,875 dark
pixels to none, and was reported as recompressed.
The encoder now refuses what the decoder can't read, so such an image stays as it is. Symbol
coding's limit of 2^24 rows can't be reached any more, so it goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the JBIG2 boundary test checks a pixel
It counted rows, which a page decoded blank would pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the encoder's bound on samples is MAX_JBIG2_PIXELS, the decoder's
main brought decode_jbig2 and its public MAX_JBIG2_PIXELS into this crate,
2^28 like this branch's private MAX_JBIG2_SAMPLES and for the same reason.
One constant now bounds both, so the encoder can't drift past what the
decoder reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: an image too large for JBIG2 isn't stored as JBIG2, as the notes say
The parity note said such an image stays as it is. A 1-bit one does unless
it is averaged, but one packed from more bits may become 1-bit Flate. The
note now says it isn't stored as JBIG2 and cites
an_image_too_large_for_the_jbig2_decoder_stays_as_it_is, and the optimizer
README lists the bound under Not yet. MAX_JBIG2_PIXELS is the most the
decoder takes, not about that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the optimizer's own size limit, and tests for what JBIG2 can't take
The optimizer leaves images over 80 million pixels alone whatever the
settings, and nothing said so; the README and parity note say it now, and
that JBIG2's 2^28 in all is past it, so from Optimize the side bound is the
one that applies. The note says what the image step does: it leaves a
1-bit image past the bound as it is unless it is averaged, and may store
one packed from more bits as 1-bit Flate.
what_is_too_large_for_jbig2_is_stored_another_way tests those two cases:
eight-bit grey of black and white at 65,536 wide becomes 1-bit Flate, and
a 1-bit image at 600 ppi averaged to 70,000 wide does too. At 65,535 and
35,000 both become JBIG2.
The symbol cap's comment says why hayro-jbig2 refuses 32,768 symbols: it
caps new and exported symbols together at 65,535, and a plan exports every
symbol it codes. Measured: 32,767 decode, 32,768, 40,000 and 65,535 are
refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the 80-million-pixel limit is the image steps', tested on both sides
The README and parity note said images over 80 million pixels are left
alone whatever the settings, but Clean Up still Flates an unfiltered or
LZW one. They now say the image steps leave them alone, and the note cites
group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit for it.
(The last commit said the note also gives 2^28 as past that limit; only
the README does.)
what_is_too_large_for_jbig2_is_stored_another_way covered the width
bound only: it now runs tall images too, packed and averaged, and checks
the averaged ones are one bit and half the size both ways. The older test
uses the same bars() fixture, and its comment says such a stream drew
nothing in PdfCraft, not that the picture was lost: Poppler and
Ghostscript draw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the JBIG2 stored at the decoder's bound reads back, and grey keeps the pixel limit
- The optimize test at 65,535 on a side decoded nothing it stored: it
checked the filter only, and the coding stored there is symbols, not
the generic coding the filters test reads back. It now decodes the
stored stream and compares it with the bars.
- The README's 80-million-pixel sentence covers the colour and grey
steps too, and only the 1-bit step had a test. A grey image of
10,000 × 8,000 at 600 ppi is resampled; a row more stays as it is.
- The filters README names decode_jbig2_within, with its work, as main
renamed decode_jbig2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the filters README names #5569's other renames too
The README named decode_jbig2_within but still decode_ccitt, which
#5569 made private behind decode_ccitt_within, and gave
Document::decoded_image as both decoders' caller, which is test-only
now: ImageBudget::decode calls them, as lib.rs's doc says. The ccitt.rs
test line keeps decode_ccitt, which those tests call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A refinement of the page refines "the current contents of the page
buffer, restricted to the area of the page buffer specified by this
segment's region segment information field" (ISO/IEC 14492 7.4.7.4), at
an offset of 0 (Table 35), and reads 0 outside that area (6.3.5.2).
hayro-jbig2 read the whole page at the refinement's location, the
pixels around the area too, so a refinement of the page decoded to
other pixels than were coded, and one wider or taller than the page was
refused.
Vendored patch 16: Bitmap::area copies the page's area, 0 off the page
and past its width whatever its rows' padding holds, and both handlers
refine that. Every refinement region now reads its reference at offset
0, so the per-page offset #6016 kept goes.
And RefinementContextGatherer::start_row loads the reference's words a
column left of where a row starts in it, as the reload would, not from
column 0. With the reference offset 32 or more left of a refinement (a
refined text instance from a narrower bitmap or an RDX below -31, a
refined symbol from such an RDX), each row's first context read 0 for
the reference column it starts at, and from 33 left for the one before
it too. That half was left out of #6048: alone
it read the page column left of a page refinement's area, which the
area now keeps at 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
"GRREFERENCEDX 0, GRREFERENCEDY 0" (ISO/IEC 14492 Table 35): a refinement
region segment that refers to another region refines that region where
it is. hayro-jbig2 offset the reference by the difference of the two
locations, which 7.4.7.5 says must be equal; where a stream has them
differ, the refinement decoded against a shifted reference, or, the
region past i32::MAX, against none. It now decodes as coded, and only a
refinement of the page takes an offset, from its own location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
A region's location is unsigned (ISO/IEC 14492 5.4.4), and hayro-jbig2
took it for a signed one. A region at 0xFFFFFFFC was drawn back onto the
page four pixels left of it, and a refinement anywhere past i32::MAX, of
the page or of a region at the same place (7.4.7.5), was refused as an
overflow, and the page with it.
Vendored patch 14 (13 is #5999): Bitmap::combine_region keeps such a
location past the page, for the four kinds of region drawn onto it, and
a refinement's offset to its reference is taken in 64 bits and held to
an i32, which leaves every reference pixel it reads outside the
reference, as the exact offset does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
`ImageBudget::decode` is now the one public way to decode an image's
CCITT fax or JBIG2 data: `Document::decoded_image` and
`decoded_image_within`, which decoded with no limit on the work, are
test-only in pdfcraft-cos, and pdfcraft-filters exports only
`decode_ccitt_within` and `decode_jbig2_within`, which spend from a
caller's allowance (`decode_jbig2` is gone, `decode_ccitt` private).
So an operation over many images can't decode them without sharing an
allowance; the compiler holds new callers to it. Doctests that must not
compile check both crates; with the old entry points public they fail.
Tests outside the crates decode through an unlimited ImageBudget.
Streams in general (`Stream::decoded`) are left for a follow-up.
Four ways round the allowance, each measured by a review of hostile input:
A Flate stream read as zlib that fails at once, and then as raw deflate that
decodes everything before failing, gave nothing back and so was charged
nothing, here and in DecodeBudget: 110 s for a 387 KB file, all decoded. What
the retry decoded now counts, though the data is still dropped as it was.
Fax and 1-bit Flate images paid by the byte for work done by the pixel: 19
bytes can be a blank page of 100 million pixels, about a second each. Every
image decoded here now pays for its pixels first, as JBIG2 already did.
A JPEG paid nothing, and what one asks for doesn't grow with its length: 156
bytes declared 80 megapixels, 72 s for an 8 KB file. A JPEG pays for its
samples before it is decoded, in Optimize and in an export that has to decode
one for its mask.
Merging fragments paid for reading strips and not for the picture it built:
393 s for 1,154 bytes. It pays for the picture before allocating it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Storing LZW streams with Flate decoded image streams outside the allowance,
with a limit for each stream and no total, and nothing asked whether to go
on: a 593 KB file of 100 images took 113 s with none refused, and images
refused by the image pass were still rewritten. The pass now pays for what
it decodes and for storing and checking it, at 8 steps a byte since Flate
takes up to 78 ns a byte, and streams past that stay LZW.
An image is counted as refused only when it stayed as it was: one packed
before the allowance ran out on its second read was rewritten. doc_reduce
reports images_refused as doc_optimize does, and the app's action says so
in its log. A refused image no longer reads as an unsupported filter:
FilterError::Spent says what happened.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Filters were charged only for what they handed a JBIG2 decoder, once they
had handed it over: a chain that failed, or went past its image's limit,
cost its time and nothing else, and images that weren't JBIG2 weren't
charged at all. A 734 KB file of 200 images of Flate inside Flate took an
image export 111 s, and that grew with the file. It now takes 12 s.
Each byte read and each byte a stage decodes is now a step of the same
allowance, paid as it happens, so no stage runs past what is left. That
holds for every image read through ImageBudget, and a fax image pays for
its rows and its code. JBIG2 globals pay when they have to be read, and
kept ones don't. Optimize's two reads that went round the budget go
through it, and merging fragments shares it.
Optimize reads an image twice, to pack it and to resample it, so a refused
image is now counted once a run rather than once a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From a review of who reads a refusal.
The rule is to leave it out and say so, and it held on some paths only. An image whose JBIG2 mask is
refused no longer comes out opaque. The first image past the allowance reads as that and not as
corrupt. A page reports what it left out. The Office warning stopped blaming forms. Redaction refuses
the run, leaving the document as it was: removing a readable image whole would take what no mark
covers with it, unreported.
The 2 GiB an export holds is twice the document if that is more, so a large honest file loses nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images
made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes
them again.
Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its
page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before
anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes
it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for
each byte of the file, so a long scan is read whole. An image past it spends all that's left, and
later ones aren't read, globals included.
Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it
out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the
picture out and says the page was truncated. Redaction removes it whole, as any image it can't read.
Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts
afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73).
It's a thread-local, unbounded until a page sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
The decoder has the same 2^28 cap as decode_jbig2, and can refuse a page first, so the doc says either
error can come. The page at the cap decodes, which the test now asserts, rather than failing later.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
The test said such a page is refused here, as Unsupported. #5491 makes hayro-jbig2 refuse it when
the stream is opened, as Corrupt, so whichever of the two PRs landed second would fail this test.
A page past the cap is refused either way, and that's what the test now says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
decode_jbig2 sits next to decode_ccitt, through hayro-jbig2 as Export ▸ Images
used it, and shares CCITT's row sink. The JBIG2 page has to be the image's size:
PdfCraft's renderer draws the whole page into the image (black box: a 4-row page
behind /Height 2 fills the image box with all four rows), so the rows of a taller
page aren't the image's. The decoder decodes a page whole, so the page is checked
before it is decoded: that size, within the caller's bound as rows of one bit, and
at most 2^28 pixels (MAX_JBIG2_PIXELS; the decoder keeps a bit a pixel, about 34 MB
there).
Document::decoded_image_within decodes a JBIG2 image as it does a fax image: the
filters before it undone first, then the decoder with the /JBIG2Globals of the
image's own /DecodeParms entry (/Crypt counts in the position; fax_params and the
globals now share last_filter_parms). Globals have a limit of their own,
MAX_JBIG2_GLOBALS (32 MiB, moved from Export), and globals that can't be read are an
error rather than ignored.
Export ▸ Images reads JBIG2 through it, so its own decoder, row sink and DecodeParms
reader go, and hayro-jbig2 leaves pdfcraft-create. JBIG2 behind another filter now
exports, and so copies (Copy Image uses the same function); a page taller than the
image, which Export cut to the image's rows, now skips the image, as one of another
width did. doc_export_all_images' description no longer lists JBIG2 inside other
filters as skipped.
The optimizer's monochrome step read images through decoded_image_within and
relied on JBIG2 failing there: it now leaves JBIG2 images alone explicitly, as
before (a resampled one would come out Flate under Retain), and the README and
parity note list them as not yet handled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Includes #4325#1952#4180#2529#4320#4805, preserving their original histories.
Keep paragraph alignment and hard/soft wrap metadata together; retain CCITT/downsampling/RGB ownership guards with bounded JBIG2 export and exact-copy symbol encoding.
Combined interaction regressions added. Validation pending.
Review of the previous commit:
- Nothing checked the segment numbers the region split assigns: a
mutant numbering every region 2 decoded. The many-marks test now
reads each segment's number, and has a bar too tall to be a symbol,
so the generic region is numbered after the last text region.
- The decoder's 32,767-symbol limit was checked against all distinct
marks, before either plan, so an image with more distinct marks lost
the second plan, which codes only the repeated ones. The limit now
applies to each plan (MAX_CODED_SYMBOLS); finding marks keeps its
memory bound of 65,536 (MAX_SYMBOLS), as before the previous commit.
A test has 32,768 distinct marks above a page of repeated text, which
encode_jbig2 still symbol codes through the second plan.
- The dense-page optimizer test now checks that it takes several text
regions, and a compile-time assert keeps segment numbers within a
byte's reach.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
A review found that a dense page came out as a stream PdfCraft's own
renderer refuses, so the page drew blank after optimizing: hayro-jbig2
decodes a text region of at most 10,000 symbol instances, and refuses
32,768 symbols or more. The stream was valid (Poppler decoded it), but
every mark went into one text region and up to 65,536 symbols were kept.
A 300 dpi letter page of the optimizer test's own text crossed it.
A page with more than 10,000 marks is now placed by several text
regions, each over a band of rows and all referring to the one symbol
dictionary, which each region but the last keeps retained (7.2.4). Past
32,767 distinct symbols the image is coded generically.
New tests: 25,000 copies placed by three regions, which decode; the
symbol caps decoded, not just reached; and in the optimizer, a whole
letter page of rendered text that looks the same as JBIG2 symbols (it
drew blank before). The READMEs say "still looked at" for the caps, and
that the scans figure is the six pages' total (20% on one page).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
A second review found two rules tested one way only: a mark 257 tall
(not just wide) is left to the generic region, which also bounds the
bitmap allocated for each mark, and runs joined at the lower-left corner
are one mark. The test now has a 1 x 257 column and an anti-diagonal.
The caps on distinct symbols (65,536) and their pixels (2^24) are tested
at and past them, as the run and row caps are.
The scans figure now names its noise: with 2% or more of the letters'
edge pixels flipped, symbols save at most 8% over the generic region.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Review found behaviour the tests let break unnoticed (each was a
surviving mutant, and each new test fails on it):
- black padding bits: a run reaching past the row would stop symbol
coding on many scans;
- Annex A's 32-bit range and PREV at exactly 256, where a decision would
go uncoded: coordinates past 4,435 on a 12,000-pixel-wide and a
10,050-row page, and the contexts of 4436's coding;
- marks joined at a corner are one mark; 256 pixels a side is a symbol,
257 is left to the generic region;
- the run and row caps, at and past them.
The roadmap's status line for this change now comes first, and the
"about a third smaller than Group 4 on scans" figure is replaced by what
was measured: 27-33% on the same pages with scan-like noise added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
pdfcraft_filters::encode_jbig2 stores each distinct mark of a 1-bit
image (8-connected black pixels, at most 256 a side) once, as a symbol
in a symbol dictionary, and places its exact copies with an immediate
lossless text region that refers to it. Larger marks, and in a second
plan marks that occur once, go to a generic region over their box. It
keeps whichever of that and the generic coding (encode_jbig2_generic,
unchanged) is smaller. Only exact copies share a symbol, so the stream
decodes to the same bits.
Written from the JBIG committee's ISO/IEC 14492 Final Committee Draft:
6.4 and 6.5 (text region and symbol dictionary decoding), 7.2, 7.4.2
and 7.4.3 (segment headers and syntax) and Annex A (the integer coders,
checked against its worked example and Table A.1).
On six pages of rendered 300 dpi text the output is 28% of the generic
coding's size and 17% of Group 4's, at about 100 ms a page; hayro-jbig2
and Poppler (pdfimages) both decode it to the original bits. Scans,
whose copies of a letter are seldom identical, gain little: that needs
refinement coding. A dictionary shared between pages (/JBIG2Globals)
is a further step.
The optimizer's Monochrome Images > JBIG2 (lossless) and doc_optimize
mono jbig2 use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
decode_ccitt reads byte-aligned Group 3 whose data has end-of-line codes
but none before the first row (not as T.4 sends them) on bytes. At /K 0
with the fill ending each code on a byte, about one small picture in
five is then read as other rows; at fax width almost none are, as such
pages come out short and are refused. Export already said so. The
optimizer, when it resamples such an image or stores it again as Group 4
from behind another filter, and redaction, when it clears pixels in one,
write those rows back, and nothing said that.
Black-box, PdfCraft's renderer draws such an image exactly as
decode_ccitt reads it (all 42 misread of 200 seeded pictures, behind
ASCII85Decode too), so the written image is what PdfCraft showed before.
Refusing such data instead (a code after nearly every row but none
first) would also refuse some ordinary Modified Huffman images, whose
rows can meet 11 zeros by chance (0.5 to 5% of small ones in a
black-box probe), so the limit is stated rather than guarded: in
decode_ccitt's doc, the filters, optimize and redact READMEs, and the
core.image-filters and mono-compression notes. The optimize README's and
the mono-compression note's "not yet: Group 3 to Group 4 without
resampling" now say a lone Group 3 image: one behind another filter is
stored again as Group 4.
Also from the review: docs/roadmap.md keeps #4335's status line under
this PR's, rather than replacing it; the filters README's tests and the
core.image-filters evidence name the Group 3 layout test; the filters
crate's description names Group 3 and 4 decoding.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The rule came into decode_ccitt with the decoding it moved from Export ▸
Images (#4335's "Export reads byte-aligned Group 3 by the codes in its
data"), where the create test checks it through the export. Here it gets
tests of its own:
- 32 seeded text-like pictures, 8 to 207 samples wide, at k 0 and 1, in
each layout (no codes, with end_of_line and without; codes, each
starting on a byte or ending on one, with and without). Each decodes to
its picture and, a byte short, to fewer rows, not to other rows read
another way.
- Group 4 rows on bytes keep their aligned read, end_of_line or not.
- The any-input property varies end_of_line and encoded_byte_align too.
With the rule this replaced (rows on bytes read unaligned under
end_of_line, aligned otherwise) the test fails; cargo-mutants leaves no
mutant of the rule's lines alive. The filters README says how rows on
bytes are read.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The PDF Optimizer and Reduce File Size downsampled colour and grey images
but not black-and-white ones. Settings::mono is now MonoSettings {
downsample, target_ppi, above_ppi, compression }, 300 ppi above 450 ppi
and CCITT Group 4 by default (Monochrome Images ▸ Downsample to;
doc_optimize mono: { downsample, ppi, above_ppi, compression }).
A 1-bit image or image mask (no filter, general-purpose filters, or CCITT
fax after any of them) and eight-bit grey of only black and white are averaged by
area: each new pixel takes the sample covering more of it, a tie 0, so a
dark line half a new pixel wide stays. They are stored as Group 4, or as
1-bit Flate when that is smaller; with Retain a resampled image keeps its
kind of compression. Images with a soft mask keep their size.
CCITT decoding moves from Export ▸ Images into pdfcraft-filters
(decode_ccitt, CcittParams, through hayro-ccitt), and
pdfcraft_cos::Document::decoded_image reads a fax image's /DecodeParms
for it (undoing filters before CCITT, such as ASCII85Decode, with any
/DecodeParms given by reference made direct first), so export
and the optimizer share one reader, and Group 4 scans can be
downsampled. A 600 ppi text page went from 79 KB to 40 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
JBIG2 images, and JBIG2 stencil masks, were reported as "can't be exported
yet", so an image the Optimizer stores as JBIG2 would stop exporting. They
are now decoded with hayro-jbig2 (already in the tree through hayro), with
their /JBIG2Globals from a DecodeParms dictionary or array, as JBIG2Decode
gives them: black as 0 (ISO 32000-2 §7.4.7). Unreadable globals, a JBIG2
page of another width, or fewer rows than the image skip the image; rows
past its height are dropped, so the output is bounded by the checked size.
docs/pdf-spec-parity.md §7.4 no longer says there is no JBIG2 encoding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_jbig2_generic writes 1-bit rows as a JBIG2
stream for JBIG2Decode (embedded organisation: a page information
segment and an immediate lossless generic region, no globals): template
0 with its nominal AT pixels, typical prediction, and the MQ arithmetic
coder (mq.rs), written from the JBIG committee's ISO/IEC 14492 Final
Committee Draft. The coder reproduces its Annex H.2 test sequence byte
for byte; output round-trips through hayro-jbig2, and Poppler renders
it the same as the original.
The optimizer's Monochrome Images gets JBIG2 (lossless) beside CCITT
Group 4 (still the default), doc_optimize mono jbig2; a document below
PDF 1.4 gets /Version /1.4 in its catalog. On 300 dpi text JBIG2 is
about a third smaller than Group 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_g4 encodes packed 1-bit rows as Group 4 (ITU-T
T.6, /K -1) through the fax crate (MIT, already in the tree via tiff),
with rows capped at 32,767 samples so its 16-bit positions can't
overflow; round trips are checked against hayro-ccitt.
The optimizer (Settings::mono, CCITT G4 by default, so Reduce File Size
too) re-encodes a 1-bit image of one colour component, or an image
mask, stored with no filter or general-purpose ones, when that is
smaller as stored. The samples decode to the same bits, so colour
space, decode array and masks stay. PDF Optimizer gets a Monochrome
Images row and doc_optimize a mono option. optimize.mono-compression
goes from planned to partial (JBIG2 and monochrome downsampling not
yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
Every stage of a stream's filter chain could produce up to the output
limit and nothing bounded the number of stages, so a long /Filter array
over data that decodes to itself (FF FF under RunLengthDecode) repeated
the full work once per array entry. Chains of more than eight filters
are unsupported, like an image codec, and stay encoded.
The app, crates, binaries, bundle id, packaging, icons, screenshots and docs are now
PdfCraft / pdfcraft (repo storytold/pdfcraft). On first launch the desktop app moves the
PrintCraft settings and crash-recovery folders to the new name, and both apps read settings
saved under the old storage key, so upgrades keep recent files, preferences and unsaved work.
The signing test fixtures keep their PrintCraft subject names (they are binary).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- js: `host()` returns a script error instead of panicking when the host state is missing.
- optimize, a11y: the graphics-state stack is read with `last`/`last_mut` instead of `expect`.
- fonts: the standard-14 ASCII map is built from `u8`, which can't fail.
- filters: zlib encoding into a `Vec` keeps the standard's narrow, documented `expect` exception.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
