* M1: a crypt filter selected by reference is used
A stream can name its own crypt filter, and every part of that can be indirect. cos read only the
direct forms, and took anything else for no name, which is /Identity. So a stream selecting a filter
by reference was saved in the clear in a file meant to be encrypted, and in a file from elsewhere
its ciphertext was read as data. The /Encrypt dictionary's /StmF, /StrF, /EFF, /CFM, /Filter,
/EncryptMetadata and a filter's /Length had the same fault, and /StmF by reference left every
stream undecrypted. A stream's /Type and a signature dictionary's are read through references too.
transform now reads these from the document, so a full save encrypts before it renumbers: the
references in what it reads have to be the document's. Dropping a crypt filter that new protection
doesn't define comes before the walk, so what it copies out of a reference is encrypted too.
Loading can re-enter here, and LoadGuard ends a cycle, which reads as absent. A chain longer than
any decoder takes names no filter: its entries would be followed for every stream sharing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M2: a stream with a crypt filter of its own is drawn
The renderer decrypted every stream with the document's filter and then failed on /Crypt as a filter
it couldn't decode, so a stream naming its own filter read as nothing, however it was spelled, and a
page whose content was such a stream stayed blank. It now uses the filter the stream names, /Identity
without one, and /Crypt is nothing further to decode (hayro-syntax patch 30).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: tests for a crypt filter's place among the others
A lone parameter dictionary is the first filter's, so /Crypt behind another filter has no name of
its own, and an array too short for /Crypt's entry stays as it is when the filter is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: encryption reads every legal form, and a full save keeps them
From a review of the crypt filter fix.
The renderer read a stream's own filter from the form /DecodeParms had to match the form of /Filter,
so [/Crypt] with a lone dictionary, an everyday spelling, read as /Identity and the page stayed
blank. Parameters are now taken in either form, which also gives [/FlateDecode] its predictor.
It also refused a V4 file without /StmF or /StrF, which are optional, and used a /CF entry named
/Identity, which is never one. A chain past what cos decodes names no filter there, so it names none
here either (hayro-syntax patch 30).
A full save lost what a direct /Encrypt refers to, and kept its references under the old numbers, so
nothing in the file read afterwards. /O or /U given by reference were decrypted and encrypted again
under a new number, so the file it wrote no longer opened. What /Encrypt and /ID refer to now stays
as stored. /ID by reference, which has to be direct, is read, and written direct.
A signature dictionary is known by a /ByteRange given by reference, and a null or dangling /Type or
/Subtype is an absent one. References to references are followed in Document::resolve, so /Name by
way of two no longer saves a stream in the clear. New protection drops every /Crypt in a chain.
An object stream's /DecodeParms can be given by reference.
Reading what decides how an object is treated can load more objects, each doing the same, which a
file could make a tree that doubles with every level once the cache lets go of it. What is loaded
for one of these no longer follows its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: what a read follows is bounded by a budget, and nothing shared is copied per stream
From a review of the fixes before this, all in them.
A flag that kept what is loaded for a selector from following its own stopped the fan-out, but a
legal file lost object stream members to it: read first through a selector, an object stream with
parameters by reference was kept undecoded, and a full save wrote nulls without saying so. A budget
replaces it: one read or write of an object, with all it loads on the way, follows up to 32
references, which is more than any object needs when all of it is by reference. Past that they
read as absent, so a file's ladder of selectors still stops, and nothing a real file holds is lost.
Finding out whether a dictionary is a signature's loaded objects for every dictionary of an object,
and a load that fails takes a scan of the file, so one read could hang. Only a dictionary with a
direct string /Contents is asked now, and the budget bounds those.
/ID written direct copied every entry, so a 1 MB file naming a large string thousands of times ran
out of memory: it is at most two strings now. Dropping a crypt filter copied a shared /DecodeParms or
/Filter into every stream, 1 GB from a 1 MB file: of what's shared only filter names and the
parameters decoders read are copied now, through Document::with_direct_parms.
The renderer skips /Crypt in a chain instead of copying the data for each entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: only an object itself follows references, and a failed scan is remembered
From a review of the budget.
Following a reference while walking deep inside an object added that depth again for each object
loaded, which overflowed a 2 MiB stack on a file under 10 KB and took the process with it, past any
catch_unwind. References are now followed for the object itself and never for what is nested in it.
A stream can't be nested, and a signature dictionary nested in another object reads as it says
directly, as it did before this PR. walk is back to following nothing.
A reference that fails to load scans the file, and nothing remembered that, so a 1 MB file took
minutes to save. What a scan didn't find is kept for the document: the file doesn't change.
Filter names copied out of a shared chain are bounded, since a name can be any length. /ID keeps each
entry in its place, or the key made from the first entry changes. direct_id's comment was on another
function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: tests with no deadline a busy machine can reach
Wide margins on the two that time what they guard. A test of dictionaries nested in an object goes:
nothing nested follows a reference now, which the small-stack test covers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: a crypt filter past what a read may follow is the document's own
Past the budget a reference reads as absent, and a crypt filter with no /Name is /Identity, so a stream
that selected one by reference had its ciphertext read as data. What can't be told takes the document's
filter, as a /Filter that can't be read does.
Found by CodeRabbit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: a read refused, not a budget full, is what leaves a crypt filter unknown
My last commit took a read that used up all it may follow for one that was refused, so an /Identity
stream read at exactly 32 follows was decrypted to garbage. Refused reads are counted now, and only
one on the way to this stream's /Name says anything about it.
Found by CodeRabbit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M1: a save keeps the handler, the key and the permissions
Under every algorithm and by either kind of save, `/Encrypt`, `/ID`, the file key and the permissions
come back as they were. No test held that. `/ID` by reference is held through an incremental save
too, which leaves it as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* cos: encrypt an edited or added object whatever /Encrypt names
stored_as_is now also requires the object to be unedited, so an edit of
an object /Encrypt or /ID refers to, or a new object given a number they
name, is encrypted on both saves. encryption_members skips numbers with
no object behind them and streams, which those dictionaries never hold.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
* A rebuilt xref finds an object whose header names another generation (#5580)
`poppler-85140-0.pdf` is 340 bytes with no cross-reference table, and the
header of its one page reads `3 18446744073709551616 obj` while the page tree
points at `3 0 R`. Before #427 the digits wrapped and 2^64 wrapped to exactly
0, so the header read as `3 0 obj` and the file opened by accident. Once an
overflowing integer became a real, the object was unreachable and the file
stopped opening at all, which `cargo xtask check` reports as a regression.
The generation is not really the point. `3 99999 obj` with the same page tree
is refused in exactly the same way on main -- the control for the new test
fails on that row first, with "page 1 does not exist" -- so this is a
mismatched generation in a rebuilt table, not an integer overflow. A table
rebuilt by scanning the file has no revision history to tell two generations of
an object apart, and a file that needs rebuilding references everything as
`N 0 R` whatever its headers say.
So, in the reconstruction path only:
- a generation number that cannot be read is taken as 0, the value such a
file's references use (`read_object_header`). A generation is at most
65,535 (ISO 32000-2 §7.5.4), so there is nothing to lose by it;
- every scanned object is also filed under generation 0, with `or_insert` so
a real `N 0 obj` is never displaced by the alias;
- the object is read back behind the same lenient header, since reading it
parses the header again and the strict identifier check rejected it
(`read_repaired_body`, gated on `repaired`).
Reading numbers anywhere else is unchanged, and a table read from the file's
own xref keeps the exact identifier match. The scan already files objects found
in object streams under generation 0, and cos's table parser already takes an
unreadable generation as 0, so this follows what the code does elsewhere.
The test covers generations 0, 99999 and 2^64 with no table and no `startxref`,
and asserts the page is found at its own size. Its fixture is built in the test
rather than taken from the corpus, so it holds without the pinned checkout.
* The generation-0 stand-in is one key, not a second object
Two consequences of filing a scanned object under generation 0 as well,
both from review.
`XRef::objects` walks the table's keys, so an object with a stand-in came out
once per key: a document whose page tree cannot be read is recovered by looking
for objects that are pages, and it reported the page twice. Enumeration now
yields one object per location, after the sort it already does. Both keys stay
for lookups; only this walk is deduplicated.
And `or_insert` never replaces, so a later `3 1 obj` moved the exact `3 1` key
to the later body while the `3 0` stand-in stayed on the earlier one -- `3 0 R`
and `3 1 R` would then read different objects. The scan now tracks which
generation-0 keys are stand-ins: a real `N 0 obj` takes the key and keeps it,
and a later header for the same number replaces a stand-in.
The tests check both through what comes out: page 2 must not exist for a
single-page file recovered without a page tree, and with two `3 1 obj` headers
the page comes back at the later body's 300x150 rather than the earlier
200x100.
* M2: the repair scan keeps its place on a non-header, and an object stream's first member is enumerated (#5580)
- read_object_header's lenient path works on a clone, so `42 3 0 obj` no longer costs the
header after the stray number (CodeRabbit, minor).
- XRef::objects deduplicates by location and object number: an object stream and its first
member share (offset, 0) and are two objects (CodeRabbit, minor).
- vendor/README.md: hayro-syntax patch (34) and its tests.
Both new tests fail with their fix reverted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
They use std::thread and std::sync::mpsc, so they take the same gate, and
so does the helper only they use. The crate's test build doesn't compile
without std on main either, for reasons of its own; these add nothing to
that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
`IndirectObject<f32>` read an object holding `6 0 R` as the number 6, so `/Decode [11 0 R 0]`, with
11 naming 6 and 6 holding 1, read as 6, and `/W` gave glyphs such widths. As anything but a number,
such an object was lost.
`XRef::get_with` is how every reference is read, to an object stored on its own or in an object
stream, so it now follows such objects in a loop, up to 32 references as `Document::resolve` does in
#5548. A loop takes no stack however long the chain, and a cycle runs out of references and reads as
a missing object, as a reference to nothing does: the image with such a `/Decode` isn't drawn, as
with `/Decode [99 0 R 0]`.
An object stream isn't reached that way: it has to be a stream stored as that object, and following
a reference to one stored inside another let a 1 KB file run out of stack. `XRef::objects` leaves
such objects out, or a page came out once more for each one holding a reference to it, and a comment
inside a reference counts as white space, as it does everywhere else.
Reading an object stream's index now counts against the 64 nested loads, and reads its dictionary
alone. That is older than this branch, which only made it slightly worse: a chain of object streams,
each with a member of the next for its `/N`, `/First` or `/Length`, recursed with no limit, holding a
copy of the chain at each, and aborted on 750 KB with an 8 MiB stack, or 250 KB with 2 MiB, in a test
build. Parsing the stream read `/Length` as well as `/N`, two ways to the next stream's index, and a
failed index isn't remembered, so the work doubled at each stream: 20 streams in a ring took 20 s on
`main`. ISO 32000-2 7.5.7 keeps an object stream's `/Length` out of object streams, so no legal file
has a `/Length` chain. Nothing forbids `/N` or `/First` by reference, though no writer I know of makes
such a chain. Past the limit it reads as missing unless a stream within 64 of it has been read.
`/W` and `/W2` with their list by reference are tested too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From a review of who reads a refusal.
The rule is to leave it out and say so, and it held on some paths only. An image whose JBIG2 mask is
refused no longer comes out opaque. The first image past the allowance reads as that and not as
corrupt. A page reports what it left out. The Office warning stopped blaming forms. Redaction refuses
the run, leaving the document as it was: removing a readable image whole would take what no mark
covers with it, unreported.
The 2 GiB an export holds is twice the document if that is more, so a large honest file loses nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images
made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes
them again.
Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its
page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before
anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes
it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for
each byte of the file, so a long scan is read whole. An image past it spends all that's left, and
later ones aren't read, globals included.
Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it
out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the
picture out and says the page was truncated. Redaction removes it whole, as any image it can't read.
Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts
afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73).
It's a thread-local, unbounded until a page sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Two more ways a reference was lost in the same readers. A Pattern space read its family name twice,
which took the underlying space when that was a name or a reference, so an uncoloured pattern took
its components as RGB: [/Pattern /DeviceCMYK] drew a cyan as red, direct names included (patch 72).
Tuples of array items, which /Decode, /W and /W2 are read as, took a reference at their start for the
whole tuple. A tuple can't be given by reference, so it was lost and the array ended there: an image
with /Decode [7 0 R 0] wasn't drawn, and CID widths fell back to /DW. A tuple now reads its items
directly, and each follows its own reference (hayro-syntax patch 32).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Each JBIG2 bitmap was bounded on its own, but a few kilobytes could still exhaust memory:
nothing bounded what a page's segments hold for later ones, or how far one segment could
multiply them. Measured on black-box streams, peak memory before and after:
- intermediate regions, 16 in 774 bytes: 514 MiB, now 98 MiB;
- an 8 MiB symbol exported 1,200 times, a 2,989-byte PDF: killed at 8 GiB, now 79 MB;
- dictionaries of zero-width symbols: 781 MB, now 72 MB;
- dictionaries retaining their coding contexts, 1.1 MB: 2.89 GB, now 71 MB;
- 65,535 references to one dictionary, 78 KB: about 17 GB, now 6 MB;
- a halftone grid of 16,384 cells a side, 98 bytes: 1.1 GB, now 2.4 MB;
- a pattern dictionary of 65,535 patterns, 48 bytes: 539 MB, now 15 MB, and it still draws;
- a custom Huffman table behind nested Flate, a 1,297-byte PDF: killed at 8 GiB, now 278 MB,
which is mostly the decoded stream;
- 11-byte segments behind nested Flate in a stream and its globals, a 4,765-byte PDF: 5 GB, now
602 MB, which is mostly the two decoded streams;
- an Indexed image whose page header claims 2^20 a side, a 743-byte PDF: aborted asking for
137 GB, now 23 MB;
- a chain of 4,000 /JBIG2Globals streams that are JBIG2 themselves, on a 2 MiB stack: overflowed
it and aborted, now 24 MB.
So PageState counts what it stores: regions, pattern and symbol dictionaries and Huffman
tables, with bitmap headers, retained contexts and a charge per item, up to 64 MiB, two square
bitmaps of 16,384 a side. A dictionary's new and exported symbols count against what is left as they are
made, not once it is stored. A segment may refer to at most 2^20 symbols, checked before any are
collected. A halftone grid has at most 2^25 cells. A pattern dictionary keeps at most 16 MiB of
shifted copies, which only speed up drawing, and none that wouldn't fit what the page has left. A custom Huffman table has at most 2^16 lines, and a
stream with its globals at most 2^20 segments and 2^22 references, checked as they are read. A
page past the bounds on a bitmap is refused when the stream is opened, since callers size their
output from its header before the page bitmap is made. And in hayro-syntax, a /JBIG2Globals
stream that is itself JBIG2 isn't decoded: globals are raw segments, so it reads as globals that
can't be decoded.
An image past a bound isn't drawn, as with the other caps, and the rest of the page is. This
bounds memory, not time: decoding work has no total yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
The vendored hayro-syntax ran PNG predictors through the bit-level
BitChunks/BitWriter machinery TIFF predictors need, although PNG filters are
defined on bytes (ISO 32000-2 §7.4.4.4, PNG specification §9). A 993 x 936
RGB image took about 63 ms to unfilter whatever its row filters; undone on
byte slices it takes about 3 ms (Up) to 16 ms (Paeth) in a release build.
It also fixes a row that ends part-way through a pixel (three 4-bit samples
at an odd width): the bit-level path dropped its last byte, each later row
started a byte early and the picture came out sheared.
TIFF predictor 2 keeps the old path. Vendored hayro-syntax patch 29.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9
Review is right that `Stack::push` always returned `Some(())`, so `overflowed`
never became true and the whole-instruction discard was dead code.
Making it live breaks the page. `an_operator_with_hundreds_of_operands_does_
not_end_the_content` (#4260) puts 300 stray operands before `0 1 0 rg` and
asserts the `rg` still reads its own three: with the discard alive the fill
stays the previous red, measured as `left: [255, 0, 0]` against
`right: [0, 255, 0]`.
The ring buffer already gives the right answer. It keeps the newest operands,
which are the ones nearest the operator -- its own -- and no operator takes
anywhere near 256 of them: `scn` takes one per colourant and ISO 32000 allows a
DeviceN space 32. So the strays fall off the back and the operator is applied to
its own operands, which is what Acrobat and Chromium do. Dropping the
instruction would lose content that is there to be read.
The dead branch and the flag are gone, `push` no longer returns a result it
cannot produce, and a test pins what the buffer does: the strays are `1` and the
operator's own operand is `0`, so the two answers differ. Keeping the oldest
operands instead fails it at 256 strays.
A malformed token ended the content stream where it stood, so the renderer,
text extraction, Find and search-based redaction all lost the rest of the
page -- while the edit reader, MuPDF and pdftotext read it (#3565, #3568). A
redaction that cannot see the text below the bad token cannot remove it,
which is the part that matters.
Each malformed shape is stepped over instead: an unterminated string, a
number that is only a sign or a dot, a name with a bad hex escape, an
unbalanced dictionary or array. Content after an inline image whose data
holds `EI` is still read, under a bounded lookahead.
Rebased onto main, 1,898 commits on: the tokenizer is still
vendor/hayro-syntax/src/content/mod.rs and applied as it was, and render's
raster.rs is split, so the two tests went to render/raster/tests.rs.
Main took hayro-syntax patches 26 and 27 (xref streams), so the patch
that keeps up to 256 operands per instruction moves to 28 in
vendor/README.md and in every comment that cites it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175FWj4xFhMhSH3UpMU8DSC
ISO 32000-2 §8.6.6.5: "DeviceN colour spaces (PDF 1.3) may contain an
arbitrary number of colour components", and scn takes one operand per
component. The content parser (vendored hayro-syntax) held ten operands
per instruction; pushing the eleventh failed, and the failure ended the
whole content stream. A twelve-ink background fill therefore drew
nothing, and nothing after it was drawn either: a blank page, no error.
hayro-syntax patch (26): an instruction keeps up to 256 operands (enough
for the largest DeviceN space the interpreter reads, 255 inks, plus a
pattern name). Past that the oldest are dropped, in order, rather than
the stream ending: operators already read the operands nearest to them,
so this is how stray operands are treated below the cap too.
hayro-interpret patch (55): a Separation or DeviceN tint transform that
can't be read no longer makes the colour space unreadable (its paints
were drawn in DeviceGray from the first tint, white for a full tint);
it keeps the space, and like a transform that fails when evaluated it
paints that colour in the alternate space's initial colour. Those
fallbacks, and a cs/CS colour space that can't be read, were silent:
they are reported once per content stream as
InterpreterWarning::ColorFallback, which pdfcraft-render records as
RenderWarning::ColourFallback and pdfcraft-cli render prints.
Tests, each seen failing with its fix reverted: the report's twelve-ink
page paints black and draws what follows; a four-ink control; 300 stray
operands before rg still draw; a calculator tint transform that pops
too much, and an unknown function type, fall back with one warning and
the rest of the page drawn; hayro-syntax keeps all twelve scn operands
and, past the cap, the latest 256 in order (fails without the reorder).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175FWj4xFhMhSH3UpMU8DSC
* fix(render): bound xref streams, JBIG2 referred-to counts, saved graphics states and /W2 runs
Input bounds and crash fixes in the vendored hayro crates, ported from
the snapshot craft-libs carries (storytold/craft-libs#176, #239, #241):
- hayro-syntax: an xref stream subsection must be backed by its decoded
data. All-zero /W widths and a count longer than the data are refused
before any entry is inserted; a tiny stream with a large /Size or
/Index count inserted that many entries.
- hayro-syntax: a subsection running past object number u32::MAX is
refused; start + i overflowed.
- hayro-syntax: the xref stream type field is read big-endian at its
full /W[0] width, and /W[0] over 8 bytes is refused like /W[1].
- hayro-jbig2: the long-form referred-to segment count is capped at
u16::MAX before its retention bytes are read; the 29-bit count went
straight to Vec::with_capacity (about 2 GiB) and aborted the process.
- hayro-interpret: at most 4096 saved graphics states holding at most
64 MiB together. A q past either limit is counted but not copied, and
its Q still removes the clips added since.
- hayro-interpret: a /W2 run whose first CID is past MAX_CID stops there,
as /W does; a first CID of 4294967295 discarded the whole /W2 table.
Each patch is marked in the source, has a regression test in its crate,
and is listed in vendor/README.md.
* fix(render): a binary usecmap counts against the CMap nesting limit; /W2 checks the CID first
- hayro-cmap: a binary CMap's usecmap passed its own depth to the nested
parse, so a self-referencing binary CMap recursed until the stack
overflowed. It now counts like a text usecmap (ledger hayro-cmap 2).
- hayro-interpret: the /W2 first-CID check runs before the run's metrics
are read, so an incomplete run past MAX_CID no longer discards the rest
of the table (ledger 52).
* fix(render): refuse xref-stream object numbers past i32::MAX
ObjectIdentifier stores an i32; a number between i32::MAX and u32::MAX
was cast to a negative object number (ledger hayro-syntax 26).
crates/cos/src/document.rs: kept #2220's shifted_subsection_end and object_headers and this PR's prev_offset.
vendor/README.md: hayro-syntax row already has (22) and (23); this PR's patch and tests are numbered (24).
Fall back only when missing sample or terminator bytes cannot fit the
content-window budget, including retained header and separator overhead.
Preserve available deterministic boundaries and ordinary large streams.
Add oversized, exact-budget, overflow and large-stream regressions.
* fix(render): read each object stream's index once (vendored hayro-syntax patch 22)
`XRef::get_with` parsed an object stream's whole index of `objnum offset`
pairs again for every object it fetched from that stream, so reading all of a
stream of N objects took N² steps. Firefox keeps every object of a page in one
stream: the pdf.js corpus file bug1978317.pdf (65,542 objects in one stream)
took 37 s to draw and 40 s to extract its text, and was a known timeout in the
robustness baseline.
The parsed index is now kept per stream, beside the decoded stream data it
indexes (`ObjectStreamIndexes`), and never cleared, as that data isn't. A
malformed index is remembered as malformed; `/N` or `/First` that can't be
read (a reference that can't be followed while another stream loads) is not
remembered. An offset past `usize` is malformed instead of wrapping. A poisoned
lock is still used: entries are inserted whole.
Release build, Linux: bug1978317.pdf draws in 0.5 s (was 37 s) and extracts
its text in 1.9 s (was 40 s), and leaves the baseline's known timeouts;
40,000 pages in one stream open in 0.2 s (was 54 s).
`raster::tests::objects_in_one_large_object_stream_are_read_in_linear_time`
times out at 20 s with the fix reverted and passes in 0.2 s with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoEx9Nj8KC4WKDcf5YaokQ
* test(render): an object stream offset past usize is malformed, not a panic
Review follow-up on #2427. Mutation testing by mtlcl410 found that replacing
patch (22)'s `checked_add` in `read_object_stream_index` with a plain `+`
survived every test, although in a debug build that `+` panics on the very
input it guards against (and wraps in a release build). Their test pins it:
it fails with "attempt to add with overflow" under that mutant and passes on
this branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`read_inner` now reads `-+280` and `++280` as well as `--280`, but
`Number::skip` consumed one sign and then required a digit, a dot or a
further `-`. So a skip failed where a read succeeded, and `ReaderExt::skip`
puts the reader back where it started -- `Object::skip` would leave the
number in place rather than step over it. `--280` was already covered,
by the `b'-'` arm.
One more sign is consumed, and the six sign forms are asserted skippable
to the end of the token. Without this the test says `-+280: not skipped`.
A third sign is still read and skipped differently (`---280` reads as 0
and skips whole), which was already so before this branch and is left
alone.
Found in review of #3630.
iText's indirectPrev.pdf writes /Prev 6 0 R, and object 6, listed in the
same update, holds 470, the offset of the first section. Both readers took
the object number as the offset, so cos found no section at 6, rebuilt the
file from object headers and lost its revision list, and hayro's chain
stopped there.
When /Prev is a reference, the sections read so far (cos) or the section's
own entries (hayro, which now reads them before following /Prev) are asked
for that object first. If it holds a non-negative integer, that is the
offset; otherwise the object number still is, as the tiff2pdf files need.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NH3itd85jPuM6wNjgcGfEH
The cos reader learnt this in the previous commit, but the viewer and
the save's reopen check count pages with hayro, which still stopped at
the reference-valued /Prev: a four-page file showed one page, and the
correctly written full save was refused as a page-count change.
A number written with a doubled sign was read by the renderer's lexer as
two numbers. `--280` set the sign, found no digit before the second
`-`, and returned 0, leaving `-280` to be read next: a `cm` carrying it
got seven operands, kept the last six, and the matrix `0 0 1 0 -280 10`
hid everything drawn under it. The page rendered blank while text
extraction still reported the text, because the glyphs were there and
only the matrix was wrong.
Measured on the report's own file, at 72 dpi, counting pixels darker
than mid-grey:
double_minus.pdf before: 0 dark px (blank) after: 581, leftmost at x=121
control.pdf (-280) 581, leftmost at x=121 -- unchanged
so the two files now render identically, which is what the report asks
for and what MuPDF does.
One extra sign is absorbed and the value is negative if either sign was
a minus, matching the COS lexer since #3299 (`--280` and `+-280` are
both -280). `Number::skip` already consumed the whole token, so read and
skip had disagreed about where the number ended.
Fixes#3565. #3299 fixed the same input in the COS lexer only.
The end-of-data look-ahead rejected a valid EI candidate when any byte
above 0x7F lay between it and the next EI. Those bytes are the content
stream that follows the image, and content streams carry such bytes
inside strings, so no candidate qualified and the image data ran to the
end of the stream. The operator check that follows now decides alone,
bounded as before by MAX_EI_CANDIDATES.
Fixes#2456
Review on #3184: with no /BitsPerComponent the decoded image took the
codestream's precision, which for a palette image is that of the indices.
The resolved colours are decoded to 8 bits, so a 1-bit index rescaled them to
0 or 255 per channel ((64, 128, 192) became (0, 255, 255)). When the palette
is applied the samples are now reported as 8-bit; an image without a palette,
or one whose raw indices are kept for an /Indexed space, keeps its precision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
Review follow-up: num_components is unset both when the dictionary has no
/ColorSpace and when it has one that does not resolve. In the second case
the JP2 header's colours are the only ones available, so applying the
palette is intended; the comment and vendor/README.md now say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
hayro-syntax decoded every JPX image with resolve_palette_indices off. That
is right when the image dictionary gives a /ColorSpace (an /Indexed space
needs the raw indices), but with no /ColorSpace the colours come from the
JP2 header (ISO 32000-2 §7.4.9) and nothing applied its palette: the
indices were drawn as a gray image. The palette is now resolved exactly
when the dictionary has no /ColorSpace.
Two unit tests decode a 4x2 palette JP2 written for the test: without a
colour space it is red/blue RGB, with /Indexed the raw indices stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RUNQSPVXx3dDBabRZHAnjz
* M2: a JP2 image in a colour space the decoder refuses is drawn by its component count (#3306)
The veraPDF fixture's JPX header names CIEJab (enumerated colour space 19),
which hayro-jpeg2000 refuses, so the image failed and the page drew blank.
When the decoder refuses the image and its colr boxes name such a space, it is
decoded again from a copy naming greyscale, sRGB or CMYK by component count,
as MuPDF and Poppler draw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
* M2: copy the JP2 stream only when a colour enum needs replacing (#3306)
Review follow-up on #3363: a stream the decoder refused for another reason was
copied for nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv2WwdgjmBz33GWkwgNn3Q
* Renumber the JP2 colour-space patch (21): #3378 takes hayro-syntax (20)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
* fix(parse): accept spaces between stream and its line break
Microsoft Reporting Services writes `stream \r\n`, a space before the line break.
hayro-syntax and pdfcraft-cos took the stream data to start at the space, so the
content stream failed to inflate: the page rendered blank (white, or black with
inverted colours) and an edited save corrupted the stream. Spaces and tabs are now
skipped, but only when a line break follows.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Number the stream end-of-line patch (20), as vendor/README does
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Apply the minimum after page-unit scaling without inflating small raw crops. Retain the existing unit policy and nearly-zero fallback, with regression coverage for origins, rotations and reciprocal rounding.
Co-authored-by: sanyamsingh001 <314051937+sanyamsingh001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
The vendored hayro-syntax passed each filter an output limit, but
ASCII85 ignored it: every `z` expands one byte to four, so a Flate
stage at its limit could be grown another 4x past every budget.
ASCII85 now writes only what fits under the limit, stops with a
warning once it is reached (like RunLength, LZW and Flate), and caps
its starting capacity at the limit.
Test: decompression_bombs_stop_at_the_stream_limit gets a
[/FlateDecode /ASCII85Decode] bomb; it decoded 301,989,892 bytes
before and stops at MAX_DECODED_STREAM now.
Co-authored-by: skylar taylor-barrick <hi@skylar.cc>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
ISO 32000-2 Table 11 defines /Rows 0 (the default) as "height not
predetermined", but the vendored hayro-syntax decoder took it as zero
rows, so scanned pages written that way rendered blank. Treat 0 like an
absent /Rows and fall back to the image height.
Adds a synthetic regression test: an 8x4 all-black Group 4 stream,
hand-encoded from ITU-T T.6, rendered with /Rows absent, 4 and 0. It
fails without the patch.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
KenM76andClaude Opus 5.5
Hamza
mtlcl410andClaude Opus 5.5
aiongg
aiongg
jonathanwxh-cell
NetterSilverandClaude Haiku 5.5
Ahmad Mujahid
wstockandClaude Opus 5.5
Luigi Pinna
cyfung1031
Eugene HindererandClaude Opus 5.5