mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-17 00:04:18 +00:00
b429031b7e15ddea6bec7cabb5856ea46afdf37a
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
203c4fd8fa |
Merge pull request #6927 from Piohrun/fix/redact-marked-content-last-operands
M8: redaction reads an operator's operands from the end, as the renderer does |
||
|
|
042371bda9 |
M8: a test of its own for each operand read the fix changes
Review of the whole change found reads it fixes that no test failed without: each was put back to main's read alone, and the tests passed. Each now has a case that fails without it: - the proof's `Tf` size (text starting outside the region reaches into it only at its size) and font (an undecodable font's text leaves the region unverifiable), and the strings a `TJ` carries; - the proof's `l`, `c`, `v` and `y` points after a surplus, from a point outside the region, and the walk's `m`, `l`, `c`, `v` and `y` points beside a mark, each alone; - the property list a second page sharing the resources still names; - a Type 3 font an ExtGState's `/Font` sets, kept by the page and its glyph by the font, and one without resources of its own, which keeps the page's pattern its glyph may paint with; - a Type 3 font only a soft mask's group selects, the mask's ExtGState named after a surplus; - the named property lists read once, with each `BDC` naming another entry, so that it is the lists, not each name's answer, that are kept. And the walk's `cs` says why it tells a Pattern space by the name `/Pattern` alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY |
||
|
|
5e2552dcb0 |
M8: named property lists and their MCIDs are read once for a stream
The tags cleanup resolved the page's `/Properties` at every named `BDC`, on main too: 20,000 of them with 5,000 entries took 28 s, as measured in review (optimized), against 0.07 s with one entry. And the second commit's `MCID` by reference was read again at every `BDC`: the document keeps an object larger than its cache only until it reads another, so reading the `MCID` evicted a property list that large, which was parsed again, all of it, at the next `BDC` (50 `BDC` took 12.3 s against 0.44 s with the `MCID` given directly). Now the named property lists are read once for the stream, in place when they are direct, and each name's `MCID` once. Tests (`crates/redact/tests/operator_work.rs`, a binary of its own whose tests run one at a time), each the least of three runs against one entry or operator, allowing ten times as long plus a second: `named_property_lists_are_read_once_for_a_page` (5,000 `BDC`, 2,000 entries) and `an_oversized_property_list_is_read_once_for_a_stream` (100 `BDC` naming a list too large to keep). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY |
||
|
|
cf26980471 |
M8: a long form /Matrix is read cheaply, an unreadable one leaves layer removal alone
Review of the walk's read of a form's `/Matrix`. Removing hidden layers alone never asks where content lands, but a form whose `/Matrix` isn't six numbers was counted in its scan and refused, in a document without layers too. Such a form is now walked as it stands there; Apply Redactions, and Remove Hidden Information with hidden text, which tells hidden text by where it lands, still refuse it. `matrix_of` resolved and copied every entry before checking there were six, and the walk reads it at every `Do`. Measured in review (debug build), 20,000 draws of a form whose `/Matrix` is 20,000 entries long took 9 s by reference (16 ms before the walk read entries) and 12 s written directly (2.6 s before, the array being copied by `resolve`). It now reads an array written directly in place, and checks the length before resolving any entry. Tests: `a_form_whose_matrix_is_not_six_numbers_leaves_removing_hidden_layers_alone`, and `tests/matrix_work.rs` (a binary of its own), whose `a_long_form_matrix_costs_no_more_than_a_short_one` times 4,000 draws of a form with a 4,000-entry `/Matrix` against a 7-entry one, written directly and by reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY |
||
|
|
20e138fbf5 |
M8: tests pin the Type 3 cleanup's refusal of bytes it can't read
The Type 3 cleanup refuses a redaction when its scan of a page or form,
or its read of a kept glyph's program, meets bytes the parser can't
read. Nothing tested either refusal: cargo-mutants found both checks
could be dropped with every test passing, once
|
||
|
|
8bf63c8792 |
M8: the Type 3 cleanup reads content and glyph programs one operator at a time
The Type 3 cleanup at the end of a redaction (unlink_type3) scans every page and every form it reaches for the glyphs they show, and reads the programs of the glyphs it keeps for the resources they use. Both parsed each stream whole before checking for bytes they couldn't read, outside the page's allowance and up to 1 GiB a stream. So a form with resources of its own outside the mark, which no other reader follows, cost about 79 bytes of memory per byte of content: the peak grew by 632 MiB for 8 MiB of `n` operators. On main too, wherever a Type 3 font survives in a page's or a form's resources. Both now read with pdfcraft_content::ops, one operator at a time, and refuse for unreadable bytes once they have read the whole stream, as before. The scan only reads, and either error refuses the whole operation, so acting on operators before that check changes nothing. The memory test gains a case for each: the_forms_followed_through_resources_are_read_one_operator_at_a_time fails with either read reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY |
||
|
|
2976caeb32 |
M8: the forms redaction follows through resources are read one operator at a time
resource_ops parsed the content and each form it follows whole before checking its 2,000,000-operator limit, and content_draws parsed whole streams with no limit at all. A parsed operator takes about 120 bytes, and a form of `n` operators outside the mark cost 78 to 138 bytes of memory per byte of content. Measured as the growth of the process's peak: 629 MiB at 8 MiB, 1,253 MiB at 16 MiB and 4,415 MiB at 32 MiB, so tens of GiB at the 256 MiB the page's allowance admits, from a file of a few hundred KiB. The page's check of what it still draws (prune), added in this branch, reads such forms on pages with no fonts or properties too. On main the font and property cleanups read them, and the unlinking check (content_draws) on any page that shares a dictionary with a removed image, so the content_draws half fixes what main does too. Both now read with pdfcraft_content::ops, one operator at a time. resource_ops holds at most one past its limit, and content_draws holds no operators, as it only looks for Do. Measured as the peak resident memory of the whole process (dev profile): the page's check over a form of 8, 16 and 64 MiB of `n` takes 311, 327 and 422 MiB, and the unlinking check over one of 16 MiB takes 69 MiB, against 1,269 before (2,527 MiB at 32 MiB). This bounds the operators held, not their operands: the parser still holds the operands that no operator ends, so content of operands alone costs about 30 bytes per byte (974 MiB for 32 MiB of `0`). Bounding those is pdfcraft-content's, for every reader of content. It also drops resource_ops' check that it followed no more than 2,000,000 forms. Each form it follows is a Do among the operators it holds, which the operator limit already bounds, so that check could never fail; cargo-mutants found it untested. Test the_forms_followed_through_resources_are_read_one_operator_at_a_time (tests/operator_memory.rs, a binary of its own) measures how much the process's peak grows over 32 MiB of `n` operators through each check, on Linux, and fails with either read reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY |
||
|
|
97cef90e01 |
Redaction: a pattern nothing paints with any more leaves the page (#3680)
* Redaction: a pattern nothing paints with any more leaves the page
A tiling pattern's cell is a content stream of its own and can draw
anything, text included. Redacting a shape filled with such a pattern
removed the fill from the page's content -- the page looked right, a
black box -- but left the pattern in /Resources /Pattern, and its cell,
with the text that was under the mark, was written into the saved file
unchanged.
The pass does not enter pattern cells, so it cannot redact inside one.
What it can do is notice that nothing paints with the pattern any more.
The graphics state now carries the fill and stroke pattern a `scn` or
`SCN` selected (and `cs`, `g`, `rg`, `k` and their stroke forms clear
it), every painting operator that survives the rewrite records the
pattern it drew with, and a pattern the page selected but no surviving
paint used leaves the page's resources.
Measured on the report's file:
before applied: 1, (SECRET) in the saved bytes
after applied: 1, no /Pattern on the page, (SECRET) gone
A pattern some surviving paint still uses is kept, and so is its cell:
with two boxes filled from one pattern and only the first marked,
patterns_removed is 0 and the resources keep the entry -- the second box
still shows the word, so it is not a leak.
Report gains patterns_removed.
Fixes #3647.
* Keep a pattern every operator that can paint with it still uses
Two defects CodeRabbit found in the first version of this change, both
of which could remove a pattern the page still draws with.
Pattern use was recorded for path painting only. Text shows with the
fill pattern too -- and with the stroke pattern in the stroking render
modes -- as do image masks and anything a form draws, and a form's
content is interpreted with a graphics state of its own, so a pattern it
inherits is never seen inside it. Content like
/Pattern cs /P0 scn 20 140 100 30 re f BT (KEEP) Tj ET
with the box redacted therefore lost the fill of text still on the page.
Every operator that can paint with the current pattern now records it:
text (except mode 3, which paints nothing, and mode 7, which only
clips), inline images, and image or form XObjects. Only path painting
knows exactly what it drew and whether the paint survived, so the others
are recorded as they are met. That can keep a pattern whose paint was in
fact all redacted, which costs an unused resource entry, while dropping
one something still draws with takes paint off the page.
/Resources /Pattern may be an indirect object, like any value. Reading
the reference itself as a dictionary found none and pruned nothing, so
on such a file the cell -- with the text that was under the mark --
stayed in the saved bytes, which is the leak this change is for. It is
resolved now, and what is written back is a dictionary of this page's
own, so pruning never reaches through a /Pattern dictionary another page
shares.
The controls in the test: without the first fix the surviving text's
pattern is removed (left: 1, right: 0), and without the second the
indirect file prunes nothing (left: 0, right: 1).
* M8: refuse retained unsafe tiling patterns
* M8: guard shared appearances and narrow pattern scans
* M8: update TLS dependencies for new advisories
* Redaction: the comment on prune_patterns said more than the code does
`prune_patterns` drops the `/Pattern` resource entry; the `scn` that selected
it stays in the content, so the page names a pattern its resources no longer
hold. The comment claimed the selector went too (noted in review).
ISO 32000-1 8.6.6.2 makes that operand a key of the `/Pattern` subdictionary,
so the open question is whether a reader stops reading the stream at the stale
name. `a_page_whose_pruned_pattern_is_still_selected_renders` answers it with
the renderer rather than with prose: the redacted page is saved, reopened and
rasterized, and
* it renders with no error and the same 300x300 page box,
* the text written *after* the stale selector is still painted -- which is
exactly what a reader would drop if it gave up at the bad operand,
* the redaction overlay is black over the box the pattern filled.
Control: with the drawing after the selector removed, the pixel assertion fails
("the text after the stale selector is still painted"), so it is reading that
text and not a stray pixel.
The selector is left in place rather than cut out: removing operators is a
rewrite of the content this pass does not otherwise do, and nothing surviving
paints with that colour. `pdfcraft-render` is a dev dependency of this crate
only -- layer 4 on layer 3, which the layering allows.
* WIP: pattern guard depth bound sized from the interpreter's
* Pattern tests: keep the guard's cases reaching the guard now a hit path is removed whole
* Pattern automation tests: keep the pattern in use where they test the guard, since a hit path is now removed whole
* Redaction: a form without resources keeps the page pattern it still paints
A form with no /Resources of its own paints with its caller's patterns, but
its surviving pattern paint was recorded only in the form's own output and
never reached the page. With the page's own fill under the mark removed
whole, prune_patterns then dropped /P0 from the page while the form outside
the mark still painted with it, so that fill lost its paint in the saved file.
Pass the painted names up from a form that inherits its caller's resources.
Only the painted side goes up, so this can keep a pattern, never prune one.
* Redaction: a pruned pattern does not stay named by the page tree
prune_patterns writes only the page's own /Resources, and the guard walked
only pages' effective resources. A /Pages node that still named the pruned
pattern kept its cell reachable from Root, so a full save wrote the cell,
text included, while apply returned Ok. Three shapes did it: a page with no
/Parent inheriting from the root, a /Parent that disagrees with the /Kids
tree, and a root /Resources shadowed by a nearer holder that was retired.
Patterns pruned from a page are now unlinked from page-tree holders no page
inherits from, as unlink_doomed does for XObjects, when every /Parent agrees
with the tree. Whenever a pattern was pruned (or a selected page had any),
the guard also inspects the /Resources of every /Pages node Root reaches
down /Kids and up each page's /Parent chain, so whatever stays reachable is
proven safe or the redaction is refused, leaving the document unchanged.
* Redaction: a Type 3 glyph without resources keeps the page pattern it paints
A Type 3 font with no /Resources looks the names in its glyph descriptions
up in the resources of the page or form showing it, so a glyph outside the
mark can paint with the page's pattern. Text recorded only the fill and
stroke pattern of the graphics state, so with the page's own fill under the
mark removed whole, prune_patterns dropped the pattern and the glyph lost its
paint. Showing such a font now keeps every pattern of the scope, as a form
without /Resources keeps what it paints.
* Redaction: say what text-recorded pattern use keeps
Text, inline images and XObjects record their pattern before the hit test,
so a pattern painted only by redacted text keeps its cell. The guard refuses
a cell with text or images; a vector-only cell stays in the file. The comment
said this cost only an unused resource entry. Pin the behaviour in a test.
* Redaction: the page tree is followed up every /Parent link
A full save writes every node a /Parent link reaches, but the guard followed
/Parent only from pages and stopped at the first node it had already seen,
and unlinking never followed it. A /Pages node reached only through the
/Parent of a /Kids node -- a root /Pages with a /Parent, or an intermediate
node whose /Parent leads out of the tree -- kept naming the pruned pattern,
and its cell, text included, stayed in the saved file while apply returned
Ok.
The guard now follows /Parent from every node it visits, page or /Pages, with
the same seen-set against cycles. Unlinking treats a root with a /Parent as an
inconsistent tree, like a disagreeing link, and leaves the decision to the
guard, which inspects every node those links reach and refuses an unsafe
cell.
* Redaction: a long page tree does not exhaust the pattern guard's budget
The page-tree walk counted every node, pages included, against the guard's
16,384-object budget, so redacting a pattern page of a 17,000-page document
was refused with a limit error where it had applied before. Only the holder
nodes the guard inspects count against that budget now; the walk itself is
bounded at a million nodes, as unlink_doomed bounds the same tree.
* Redaction: a Type 3 font set by an ExtGState or shown in a form keeps the page patterns
The resourceless Type 3 check covered only a font set by Tf in the scope that
paints. A Type 3 font without /Resources set through an ExtGState /Font entry
was not flagged, and a glyph shown inside a form with /Resources of its own
recorded its patterns in the form's scope, which the page never saw. In both
cases the page's own fill under the mark was removed whole, prune_patterns
dropped the pattern, and the glyph outside the mark lost its paint.
An ExtGState font is now checked the same way, and a glyph shown in any
nested form marks the page as keeping every pattern it selected. This can
only keep patterns; whatever is kept is checked by the guard.
* Redaction: a pruned pattern a full save would still write is proven safe or refused
The walks that check retained pattern cells follow the references they know
-- resource categories, the page tree, appearances -- but a full save writes
whatever the trailer reaches by any reference. A direct /Parent dictionary, a
/Kids array on a page, or a stray resource entry (/Shading naming the pattern
stream) kept a pruned cell, text included, in the saved file while apply
returned Ok.
pdfcraft_cos gains reachable_objects, which runs the same breadth-first walk
write_full uses (factored out as full_save_order, so the two cannot drift).
Before apply succeeds, every pruned pattern that walk still reaches is
inspected like a retained cell: vector-only passes, anything else refuses
with the existing pattern error. It runs only when a pattern was pruned and
costs one walk of the reachable objects.
* Redaction: a direct /Parent dictionary is part of the page tree too
A /Parent given as a dictionary rather than a reference is written inline by
a full save, but the page-tree walk followed /Parent only as a reference, and
unlinking took such a tree for consistent. A root /Pages whose direct /Parent
held /Resources naming the pruned pattern kept its cell in the saved file.
The guard now walks a direct /Parent like any node (its /Resources, /Parent
and /Kids), and unlinking counts any /Parent that is not the reference /Kids
came from as an inconsistent tree, leaving the decision to the guard.
* Redaction: page-tree walks queue each node once and borrow /Kids
On a hostile tree whose /Pages nodes share one long /Kids array, the guard's
page-tree walk and pattern unlinking pushed every /Kids entry of every node
before checking whether it had been seen, and copied the array on each
visit. Both now queue only nodes not seen yet and read /Kids in place;
unlinking also stops at the first node reached twice, since its /Parent can
agree with at most one of the nodes above it, and leaves that tree to the
guard.
* Redaction: a Type 3 font with /Resources null is resourceless too
/Resources null is the same as no /Resources, but the check counted any
entry, so such a font's glyph outside the mark lost the page pattern it
painted with. A Type 3 glyph shown by a kept annotation appearance is a known
limit, pinned as an ignored test: appearances are not interpreted for pattern
use, and under ISO 32000 the glyph resolves in the appearance's own resources.
* Redaction: a pruned pattern is recorded along its whole reference chain
A pattern name can lead to a reference-valued object whose value is the cell
(/P0 7 0 R, object 7 being 6 0 R). Pruning recorded only object 7, so the
reachability backstop checked an object nothing reached while a stray
reference kept the cell, object 6, in the saved file. Every object along the
chain is recorded now, bounded like the other walks.
* Pattern tests: pin the file identifier where two saves are compared
main now gives a redacted rewrite a new identifier from fresh entropy, so an
incremental and a full save after Apply differ in /ID alone. The test that
checks they are the same save passes fixed entropy to both.
* Redaction: replies and pop-ups removed on other pages feed the shared-appearance check
Apply checks every page for retained unsafe pattern cells when a removed
annotation's appearance is shared with one that stays on an unselected page.
Only annotations removed under a mark fed that check. Replies and pop-ups
that go with them in step 2b (removed_everywhere), often listed on another
page, did not, so a text-bearing cell in an appearance a kept annotation
shares with them escaped the all-pages check and Apply returned Ok.
Their appearance references now go into removed_appearances too, with the
same limit. Raised by FredApps' review of the rebase.
---------
Co-authored-by: skylar taylor-barrick <hi@skylar.cc>
|