One conflict, in redact lib.rs: the module list, where main added `elsewhere` and
`identification` (#6714) and this branch `globals`. All three stay, in order.
Main's #2448 work also added a sixth step to `apply`, after the verification and the pattern
guard: `metadata::scrub` removes the `/Info` entries and the catalog's and pages' XMP that repeat
the removed text. An image that only such a packet refers to leaves the file then, after
`unshare` had looked for what left, so its shared dictionary stayed. `unshare` now runs after
the scrub, last of what can take something out. Storing images again changes no page's content
and no pattern cell, so the verification and the pattern guard before it check the same pages
they did. The crate's doc says the dictionary goes last. Remove Hidden Information's order is
unchanged: main's changes to `sanitize.rs` only read (`owned_values`).
Test: `an_image_only_the_scrubbed_metadata_refers_to_takes_a_shared_jbig2_dictionary_too`. An
image under the mark is also named by the catalog's XMP, which repeats the text removed beside
it. After the redaction the packet is gone, the image on the other page is stored again, and
the dictionary is gone too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
A third merge of main, at ba28458ad. Main's #2448 (in #6714) collects
the text a redaction removes from the walk's scope, which this branch
also reads after the walk for what it left of the page's allowance.
- `interp.rs`: `Scope` keeps this branch's `into_budget` and gains
main's `with_removed_room`, `flush` and `removed`.
- `lib.rs`: the page's walk starts with main's room for removed text,
and after it the scope's removed text is collected as on main, then
what it left of the page's allowance is taken for the cleanups, last,
as taking it consumes the scope.
Main's new code reads no content stream or form in the walk or the
cleanups, so nothing of this branch's allowance or one-operator reads
needed carrying into it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One conflict, in redact sanitize.rs. Main's #6021 (930b44131) split `remove_hidden` in two:
- it now counts only what the chosen categories need (`scan_only`);
- it hands the removal to `remove_counted`, which tests also call with `every_page`.
This branch had added one line to the old body: `kept_before`, what a full save keeps before
anything is removed. Main's side is taken whole, and that line is now the first thing
`remove_counted` does after finding the catalog, before any change to the document. So every
caller of `remove_counted` has it. `unshare_unmarked` stays last, after `unlink_undrawn`, and
merged cleanly there. The only early return after a change is `NothingToApply` when nothing
was removed and nothing unlinked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Conflicts, and how they were resolved:
- cos writer.rs and lib.rs: main now numbers a full save in `full_save_order` and exports
`reachable_objects`, the object numbers a full save writes, for the pattern guard. This
branch had added the same walk as `walk` and `reachable`. Main's is kept, with its alias
handling, and this branch's is dropped: `globals::kept` is now `reachable_objects`, so the
JBIG2 cleanup and the pattern guard read one walk. `kept` returns the walk's error (a save
that can't number its objects), which redaction and Remove Hidden Information pass on. The
cos test of what is reachable, and two redaction tests, read `reachable_objects` instead.
The exports keep this branch's `DECODE_PARMS_KEYS` and main's `TextChars`,
`civil_from_days` and `new_file_id`.
- redact lib.rs: main unlinks pruned patterns from `/Pages` nodes after `unlink_doomed`; this
branch replaced `restore_shared_jbig2` with `globals::unshare`. Both stay, the unlink first:
an image a pruned pattern's cell drew leaves the file with it, and is a user of a dictionary
that left too.
- redact sanitize.rs: main's `strip_outline_actions` and this branch's `unshare_unmarked` both
stay. At the end of `remove_hidden`, `unshare_unmarked` runs after main's `unlink_undrawn`,
so an image unlinked for being drawn nowhere takes a shared dictionary with it, and is
followed by main's `require_full_save_with_new_id`.
- redact image.rs: `MAX_PIXELS` is `pub(crate)` on both sides; the comment names
`globals::without_dictionary`, where this branch moved `without_globals`.
Tests: `an_undrawn_image_unlinked_takes_a_shared_jbig2_dictionary_too` (Remove Hidden
Information, through `unlink_undrawn`) and
`an_image_a_pruned_patterns_cell_drew_takes_a_shared_jbig2_dictionary_too` (a redaction,
through `unlink_pruned_patterns`) pin the order of the two new routes before `unshare`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Main reworked the redaction cleanups this branch changes: #5969 reads
each inherited form once for all names (`names_content_draws`), #5924
prunes the forms a surviving Type 3 glyph draws and passes the page to
`prune_type3_from_resources`, and #6474 reads a page's annotation
appearances toward one operator total and one byte allowance.
- `interp/prune.rs`: `names_content_draws` keeps main's one read per
form and takes this branch's three changes to `content_draws`: it
reads operators one at a time, holding no stream's whole and decoding
each form only in its turn; it reads forms with `/DecodeParms` given
by reference; and a `/Resources` that isn't a dictionary is none of
the form's own. It charges no allowance, as `content_draws` didn't:
it runs after the pages' walks, over every holder, where no page's
allowance is left to spend.
- `type3.rs`: `prune_type3_from_resources` takes both the page's
allowance and main's `page`. The appearances main reads are charged
to that allowance, which the content and its forms spend too, in
place of a fresh one of their own, so one page has one allowance.
Main's `record_used` serves this branch's one-operator-at-a-time read
of glyph programs, and `scan_stream_object`, which took over
`scan_xobject`'s read, reads with `/DecodeParms` by reference.
`prune_font_forms` gives each form's walk the allowance of a page, as
`prune_retired_properties` does for each content.
- `interp.rs`, `lib.rs`, `sanitize.rs`: both arguments at each call.
- `edit/src/rules.rs`: the refusal of incomplete content stays, with
main's `text::joined`.
- Tests: both sides' kept, in `edit/src/tests.rs` and
`redact/src/tests/surviving_references.rs`. A new one,
`a_page_s_appearances_spend_what_its_content_left_of_the_allowance`,
pins the shared allowance: 1 MiB of content and an appearance within
half a MiB of the allowance are refused together, though the
appearance alone fits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Main's Replace Text (#5689) makes the caret the group's primary, as this
branch does. Kept: this branch's member-by-member deletion and release of
orphaned subordinates, with main's seed() (the strikeout also seeds its
caret) and its pair restyle; the writer returns the caret (the group's
comment) and leaves the strikeout without /Contents. Main's tests read the
strikeout from the caret's group instead of its replies, and the automation
test reads the strikeout's own colour from an XFDF export, since
comment_list gives a group its primary's attributes only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCHD1R3rywQBPCHXZA
Conflict: crates/automation/tests/it/automation.rs (kept the group test and main's ungroup tests).
Also carries the one-line fix for main's crates/edit/src/rules.rs (content_streams() now returns
ContentStream): page_rules joins its streams with crate::text::joined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCHD1R3rywQBPCHXZA
Resolve the conflicts in crates/engine/src/lib.rs (keep both the
redaction-identification and user-regex re-exports) and Cargo.lock
(main's, plus the redact dependencies). Drop the branch's copy of the
test helper pdf_of, which main now defines identically. Fix main's
pdfcraft-edit build: rules.rs joins its content streams through
crate::text::joined now that content_streams returns ContentStream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCHD1R3rywQBPCHXZA
Once a redaction removes an image or form under a mark, it takes the name
out of the page's resources unless the page's content still draws it
elsewhere. It read the page's streams again for that with `decoded`, which
reads `/DecodeParms` only when direct, so a stream behind a predictor whose
parameters are an indirect object seemed not to draw it, and the page lost
the image there.
It now looks in what it read for the redaction, with the parameters by
reference, and decodes again only the streams it has just written, which
have none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
`DecodeBudget::decode_strict` read a stream's `/DecodeParms` only when
direct, so content behind a predictor whose parameters are an indirect
object decoded with the predictor's tag bytes left in. Redaction then found
no glyph under a mark, left the text in the file and reported the mark
applied. ISO 32000-2 lets those parameters be indirect.
`decode_strict` takes the document and reads through `decoding_view`, as
`decode_with_damage` does. Its other readers read the parameters too:
Remove Hidden Information's hidden text and layers, the audit's covered-text
check, and edit's backdrop scan.
The accessibility_mark_tag_artifact description says that content which is,
or may be, in a stream damaged partway is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
A quad counts only when all 8 entries are present, numeric and finite.
Anything else contributes nothing — never folded, shifted or zero-filled
into a box the document does not describe. With no usable quad the
existing /Rect fallback runs, and with neither the mark stays unreadable.
The rule lives in one helper, geom::quad_points::usable_quads, and redact,
render-inspect, annot summaries and annot appearance drawing all read
through it, so the outline shown is the area apply removes. Redact also
follows indirect numbers in /QuadPoints now, which it used to drop.
On a page, the check of which names the rewritten content still draws
(`prune`) ran before the rewritten XObjects were added to the page's
/XObject dictionary, so it couldn't follow a rewritten form under its
new name. A rewritten form without resources of its own gets a copy of
the page's, pruned to what it draws, and draws through that. It keeps
none of its own only when the copy loses nothing, which takes a form
that draws itself, here outside the mark. Such a form draws through the
page's resources, and the page lost the names it draws there: the
original form, and an image under the mark that it draws again outside
it through an unchanged form.
The page now installs them first, as a rewritten form does with the
forms it rewrites. Remove Hidden Information's page does the same. There
every form is entered, so a form that draws itself is followed to the
depth limit and every copy above it keeps resources of its own (measured
with such a form); its order changes so that the two read alike.
Test an_image_a_rewritten_form_draws_again_through_the_pages_resources_is_kept
fails with Apply Redactions' old order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
The walk of the forms that content draws through its resources
(`resource_ops`) decoded each one outside the page's decode allowance:
up to 1 GiB a form, read again by each cleanup that walks them, on every
page. The interpreter skips a form outside the marks, so for such a form
this walk is the only read.
`resource_ops` now charges the bytes it decodes to the `DecodeBudget` it
is given. The content walk hands back what it left of the page's
allowance (`Scope::into_budget`), and the cleanups after it spend that:
the check of what the page still draws (`prune`), the properties and the
Type 3 fonts, in Apply Redactions and in Remove Hidden Information. A
rewritten form's cleanups spend the walk's allowance as they go. The
unlinking pass's check of the properties, which walks every page that
shares a resource dictionary, gives each page's content an allowance of
its own.
Only bytes are charged, not form entries: each cleanup reads the same
forms again, so a page drawing many small forms would run out of entries
having decoded little.
Tests, each failing with the charge reverted, and the last three with
their own part of it reverted (the hand-back on either page, the
rewritten form's cleanups):
- what_content_draws_through_its_resources_is_read_within_its_allowance
(a 2,000-byte form read with 2,000 bytes left, refused with 1,999);
- the_cleanups_after_the_content_walk_spend_what_it_left_of_the_pages_allowance;
- the_cleanups_of_a_rewritten_form_spend_the_pages_allowance;
- removing_hidden_text_spends_what_the_content_walk_left_of_the_pages_allowance.
In each of the last three a form without resources of its own is read
three times: refused at 100 MiB, more than the page's 256 MiB, and not
at 50 MiB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
When a redaction removes or replaces an image or form under a mark, it
takes the name out of the page's resources unless the page still draws
it. It looked only at the page's own `Do` operators. A form with no
/Resources of its own draws through the page's, so when such a form
outside the mark drew the same image, the page lost the name and the
form drew nothing there.
The page now keeps a name that a form drawing through its resources
still draws, found as the cleanup of the fonts and properties these
forms use already finds them. As that cleanup does, it refuses the page
when such a form can't be read. On a page with fonts or properties to
clean up that was already the case; on a page with neither, the name
used to go.
A rewritten form had the same fault one level down. It dropped the names
it replaced before `prune_form_resources` ran. That function already
keeps what a form drawing through these resources still draws, so the
rewritten form now leaves the decision to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* Redaction: a pattern nothing paints with any more leaves the page
A tiling pattern's cell is a content stream of its own and can draw
anything, text included. Redacting a shape filled with such a pattern
removed the fill from the page's content -- the page looked right, a
black box -- but left the pattern in /Resources /Pattern, and its cell,
with the text that was under the mark, was written into the saved file
unchanged.
The pass does not enter pattern cells, so it cannot redact inside one.
What it can do is notice that nothing paints with the pattern any more.
The graphics state now carries the fill and stroke pattern a `scn` or
`SCN` selected (and `cs`, `g`, `rg`, `k` and their stroke forms clear
it), every painting operator that survives the rewrite records the
pattern it drew with, and a pattern the page selected but no surviving
paint used leaves the page's resources.
Measured on the report's file:
before applied: 1, (SECRET) in the saved bytes
after applied: 1, no /Pattern on the page, (SECRET) gone
A pattern some surviving paint still uses is kept, and so is its cell:
with two boxes filled from one pattern and only the first marked,
patterns_removed is 0 and the resources keep the entry -- the second box
still shows the word, so it is not a leak.
Report gains patterns_removed.
Fixes#3647.
* Keep a pattern every operator that can paint with it still uses
Two defects CodeRabbit found in the first version of this change, both
of which could remove a pattern the page still draws with.
Pattern use was recorded for path painting only. Text shows with the
fill pattern too -- and with the stroke pattern in the stroking render
modes -- as do image masks and anything a form draws, and a form's
content is interpreted with a graphics state of its own, so a pattern it
inherits is never seen inside it. Content like
/Pattern cs /P0 scn 20 140 100 30 re f BT (KEEP) Tj ET
with the box redacted therefore lost the fill of text still on the page.
Every operator that can paint with the current pattern now records it:
text (except mode 3, which paints nothing, and mode 7, which only
clips), inline images, and image or form XObjects. Only path painting
knows exactly what it drew and whether the paint survived, so the others
are recorded as they are met. That can keep a pattern whose paint was in
fact all redacted, which costs an unused resource entry, while dropping
one something still draws with takes paint off the page.
/Resources /Pattern may be an indirect object, like any value. Reading
the reference itself as a dictionary found none and pruned nothing, so
on such a file the cell -- with the text that was under the mark --
stayed in the saved bytes, which is the leak this change is for. It is
resolved now, and what is written back is a dictionary of this page's
own, so pruning never reaches through a /Pattern dictionary another page
shares.
The controls in the test: without the first fix the surviving text's
pattern is removed (left: 1, right: 0), and without the second the
indirect file prunes nothing (left: 0, right: 1).
* M8: refuse retained unsafe tiling patterns
* M8: guard shared appearances and narrow pattern scans
* M8: update TLS dependencies for new advisories
* Redaction: the comment on prune_patterns said more than the code does
`prune_patterns` drops the `/Pattern` resource entry; the `scn` that selected
it stays in the content, so the page names a pattern its resources no longer
hold. The comment claimed the selector went too (noted in review).
ISO 32000-1 8.6.6.2 makes that operand a key of the `/Pattern` subdictionary,
so the open question is whether a reader stops reading the stream at the stale
name. `a_page_whose_pruned_pattern_is_still_selected_renders` answers it with
the renderer rather than with prose: the redacted page is saved, reopened and
rasterized, and
* it renders with no error and the same 300x300 page box,
* the text written *after* the stale selector is still painted -- which is
exactly what a reader would drop if it gave up at the bad operand,
* the redaction overlay is black over the box the pattern filled.
Control: with the drawing after the selector removed, the pixel assertion fails
("the text after the stale selector is still painted"), so it is reading that
text and not a stray pixel.
The selector is left in place rather than cut out: removing operators is a
rewrite of the content this pass does not otherwise do, and nothing surviving
paints with that colour. `pdfcraft-render` is a dev dependency of this crate
only -- layer 4 on layer 3, which the layering allows.
* WIP: pattern guard depth bound sized from the interpreter's
* Pattern tests: keep the guard's cases reaching the guard now a hit path is removed whole
* Pattern automation tests: keep the pattern in use where they test the guard, since a hit path is now removed whole
* Redaction: a form without resources keeps the page pattern it still paints
A form with no /Resources of its own paints with its caller's patterns, but
its surviving pattern paint was recorded only in the form's own output and
never reached the page. With the page's own fill under the mark removed
whole, prune_patterns then dropped /P0 from the page while the form outside
the mark still painted with it, so that fill lost its paint in the saved file.
Pass the painted names up from a form that inherits its caller's resources.
Only the painted side goes up, so this can keep a pattern, never prune one.
* Redaction: a pruned pattern does not stay named by the page tree
prune_patterns writes only the page's own /Resources, and the guard walked
only pages' effective resources. A /Pages node that still named the pruned
pattern kept its cell reachable from Root, so a full save wrote the cell,
text included, while apply returned Ok. Three shapes did it: a page with no
/Parent inheriting from the root, a /Parent that disagrees with the /Kids
tree, and a root /Resources shadowed by a nearer holder that was retired.
Patterns pruned from a page are now unlinked from page-tree holders no page
inherits from, as unlink_doomed does for XObjects, when every /Parent agrees
with the tree. Whenever a pattern was pruned (or a selected page had any),
the guard also inspects the /Resources of every /Pages node Root reaches
down /Kids and up each page's /Parent chain, so whatever stays reachable is
proven safe or the redaction is refused, leaving the document unchanged.
* Redaction: a Type 3 glyph without resources keeps the page pattern it paints
A Type 3 font with no /Resources looks the names in its glyph descriptions
up in the resources of the page or form showing it, so a glyph outside the
mark can paint with the page's pattern. Text recorded only the fill and
stroke pattern of the graphics state, so with the page's own fill under the
mark removed whole, prune_patterns dropped the pattern and the glyph lost its
paint. Showing such a font now keeps every pattern of the scope, as a form
without /Resources keeps what it paints.
* Redaction: say what text-recorded pattern use keeps
Text, inline images and XObjects record their pattern before the hit test,
so a pattern painted only by redacted text keeps its cell. The guard refuses
a cell with text or images; a vector-only cell stays in the file. The comment
said this cost only an unused resource entry. Pin the behaviour in a test.
* Redaction: the page tree is followed up every /Parent link
A full save writes every node a /Parent link reaches, but the guard followed
/Parent only from pages and stopped at the first node it had already seen,
and unlinking never followed it. A /Pages node reached only through the
/Parent of a /Kids node -- a root /Pages with a /Parent, or an intermediate
node whose /Parent leads out of the tree -- kept naming the pruned pattern,
and its cell, text included, stayed in the saved file while apply returned
Ok.
The guard now follows /Parent from every node it visits, page or /Pages, with
the same seen-set against cycles. Unlinking treats a root with a /Parent as an
inconsistent tree, like a disagreeing link, and leaves the decision to the
guard, which inspects every node those links reach and refuses an unsafe
cell.
* Redaction: a long page tree does not exhaust the pattern guard's budget
The page-tree walk counted every node, pages included, against the guard's
16,384-object budget, so redacting a pattern page of a 17,000-page document
was refused with a limit error where it had applied before. Only the holder
nodes the guard inspects count against that budget now; the walk itself is
bounded at a million nodes, as unlink_doomed bounds the same tree.
* Redaction: a Type 3 font set by an ExtGState or shown in a form keeps the page patterns
The resourceless Type 3 check covered only a font set by Tf in the scope that
paints. A Type 3 font without /Resources set through an ExtGState /Font entry
was not flagged, and a glyph shown inside a form with /Resources of its own
recorded its patterns in the form's scope, which the page never saw. In both
cases the page's own fill under the mark was removed whole, prune_patterns
dropped the pattern, and the glyph outside the mark lost its paint.
An ExtGState font is now checked the same way, and a glyph shown in any
nested form marks the page as keeping every pattern it selected. This can
only keep patterns; whatever is kept is checked by the guard.
* Redaction: a pruned pattern a full save would still write is proven safe or refused
The walks that check retained pattern cells follow the references they know
-- resource categories, the page tree, appearances -- but a full save writes
whatever the trailer reaches by any reference. A direct /Parent dictionary, a
/Kids array on a page, or a stray resource entry (/Shading naming the pattern
stream) kept a pruned cell, text included, in the saved file while apply
returned Ok.
pdfcraft_cos gains reachable_objects, which runs the same breadth-first walk
write_full uses (factored out as full_save_order, so the two cannot drift).
Before apply succeeds, every pruned pattern that walk still reaches is
inspected like a retained cell: vector-only passes, anything else refuses
with the existing pattern error. It runs only when a pattern was pruned and
costs one walk of the reachable objects.
* Redaction: a direct /Parent dictionary is part of the page tree too
A /Parent given as a dictionary rather than a reference is written inline by
a full save, but the page-tree walk followed /Parent only as a reference, and
unlinking took such a tree for consistent. A root /Pages whose direct /Parent
held /Resources naming the pruned pattern kept its cell in the saved file.
The guard now walks a direct /Parent like any node (its /Resources, /Parent
and /Kids), and unlinking counts any /Parent that is not the reference /Kids
came from as an inconsistent tree, leaving the decision to the guard.
* Redaction: page-tree walks queue each node once and borrow /Kids
On a hostile tree whose /Pages nodes share one long /Kids array, the guard's
page-tree walk and pattern unlinking pushed every /Kids entry of every node
before checking whether it had been seen, and copied the array on each
visit. Both now queue only nodes not seen yet and read /Kids in place;
unlinking also stops at the first node reached twice, since its /Parent can
agree with at most one of the nodes above it, and leaves that tree to the
guard.
* Redaction: a Type 3 font with /Resources null is resourceless too
/Resources null is the same as no /Resources, but the check counted any
entry, so such a font's glyph outside the mark lost the page pattern it
painted with. A Type 3 glyph shown by a kept annotation appearance is a known
limit, pinned as an ignored test: appearances are not interpreted for pattern
use, and under ISO 32000 the glyph resolves in the appearance's own resources.
* Redaction: a pruned pattern is recorded along its whole reference chain
A pattern name can lead to a reference-valued object whose value is the cell
(/P0 7 0 R, object 7 being 6 0 R). Pruning recorded only object 7, so the
reachability backstop checked an object nothing reached while a stray
reference kept the cell, object 6, in the saved file. Every object along the
chain is recorded now, bounded like the other walks.
* Pattern tests: pin the file identifier where two saves are compared
main now gives a redacted rewrite a new identifier from fresh entropy, so an
incremental and a full save after Apply differ in /ID alone. The test that
checks they are the same save passes fixed entropy to both.
* Redaction: replies and pop-ups removed on other pages feed the shared-appearance check
Apply checks every page for retained unsafe pattern cells when a removed
annotation's appearance is shared with one that stays on an unselected page.
Only annotations removed under a mark fed that check. Replies and pop-ups
that go with them in step 2b (removed_everywhere), often listed on another
page, did not, so a text-bearing cell in an appearance a kept annotation
shares with them escaped the all-pages check and Apply returned Ok.
Their appearance references now go into removed_appearances too, with the
same limit. Raised by FredApps' review of the rebase.
---------
Co-authored-by: skylar taylor-barrick <hi@skylar.cc>
* redact: add the independent redaction proof machinery
* redact: harden the proof's paint check and count removed codes
The manifest counts character codes, not raw bytes: a composite font's
codes are several bytes, so the byte length overstated what was cut.
Annotation appearances are checked whatever their storage (a direct
dictionary has no reference of its own to dedupe on), and a stream
shared by several annotations is checked at each annotation's own
placement. The graphics-state stack and the clip holes of one walk are
bounded; deeper or more is unverifiable, never unbounded memory.
Even-odd clips cut holes only where the rule can be told cheaply - all
subpaths axis-aligned, disjoint, few - and anything else keeps its
paint reported, failing closed. The /W widths of a descendant font
cost bounded work, like a CMap.
* redact: settle the survivors a page sweep counted before its work ran out
A page that the search budget stops on left the counts made on earlier
pages unsettled: the survivors were seen but never reported, and the
regions that owned them were never marked. The sweep now stops and
still settles, so a located survivor is reported where it sits (the
page reads unswept either way, and the proof still fails).
* redact: prove the redacted output of every supported encryption
The authenticated-reopen regression now covers all four crypt
algorithms the writer supports, not only AES-128: the caller's
credentials prove the saved output, and empty or wrong credentials
still fail closed.
* redact: answer the review round on the proof
A glyph lying under several regions is now accounted for by every one
of them, so overlapping marks each carry their own share in a shared
proof instead of the first one absorbing the rest and the others
reading as if no text was there.
Each finding is recorded once per region as it is found, so a page
that paints over a region a thousand times neither grows the report
nor the memory with repeats.
The byte search walks the haystack once (the pattern's prefix function
is built once), so a needle of one letter over a haystack full of it
can no longer spend comparisons the search budget never charged for.
A numeric XML character reference may carry any number of leading
zeros; the padded form now decodes like the short one, so a survivor
spelled that way in the XMP is still found.
The /W widths of a descendant font are paid for from the shared work
budget: a form invoked again rebuilds its fonts, and a per-font
counter alone let the same array be paid for many times over.
* redact: sweep the proof's survivors out of every place a reader could look
Plant the removed text where the review listed it and expect the sweep to
report it: a compressed object stream, a page thumbnail, a form field
value, an XFA packet, and an object nothing refers to. A file whose page
content is bigger than the proof may decode for one stream is now shown
to come out unverified on its surfaces and its region, never clean.
* redact: spare the names a redacted word legitimately sits in, and charge failed decodes
A redacted word that is also a structural name (a /Subtype, a /Filter, a
/BaseFont) is structure, not a leftover copy: the name surface is now
swept against the matches the input itself had, and only a surplus of
matches is a survivor. The input's names are counted when the snapshot is
taken, for the region's needles and for strings added later alike.
A decode that fails after doing part of its work paid nothing: the
proof's cumulative decode allowance now charges every attempt, failures
included, so the same bomb cannot be decoded (and fail) again and again
for free.
* redact: count the input's names once each, and say so when the count runs out
The name allowance counted one entry per occurrence, so a document whose
pages repeat the same structural name paid for the same scan again and
again; names are now kept once, with their count. A name search that runs
out of work read as no matches, which would have swept names with none
allowed: that is reported on the text surface now, and the proof fails
rather than pass on an uncounted allowance.
* Bound redaction proof searches across many needles
Assisted-by: OpenAI Codex
* Redaction proof: drop find_all, which nothing calls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Cargo.lock: put aho-corasick in its sorted place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Redaction proof: review notes
A word split across a TJ array is looked for whole. Overlapping marks count a
removed copy once. A spent decode budget is a problem, not a raw-only stream.
Short needles stay out of the attachment text search, and an indirect /Decode
counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com>
Co-authored-by: FredApps <FredApps@users.noreply.github.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* cos: give a redacted or sanitized full save a new file identifier
A full save that the document asked for (Document::require_full_save_with_new_id)
now writes a fresh /ID pair, so a redacted or sanitized copy does not claim to be the
file it came from (ISO 32000-2 section 14.4). Ordinary saves are unchanged.
An encrypted file keeps the first string of /ID, because the file key is derived from
it, and gets a new second string. An encrypted file whose /ID is not a pair of strings
is left as it was, as before. SaveOptions::id_entropy lets a caller or a test supply
entropy; with it the new identifier depends only on that entropy and the written bytes.
Without it the identifier mixes the standard library's random hash keys and a
process-wide counter, the same facility the engine uses for new encryption keys.
Checked: the new tests in crates/cos/tests/file_identifier.rs fail on the clean base
(compile errors for the missing API), fail on assertions when only the API is in place,
and fail again when each pinned line is reverted (the regeneration call, the first-string
keep, the flag, the entropy input, and the ordinary-save guard). cargo test -p
pdfcraft-cos passes, and clippy with warnings denied is clean. A probe that wrote 16
ordinary full and incremental saves gave byte-identical files with and without this change.
* redact: give a redacted or sanitized copy a new file identifier
Applying redactions, sanitizing and removing hidden information now ask the cos
layer for a new file identifier (Document::require_full_save_with_new_id), so the
saved copy no longer carries the /ID of its source. Before this change the
previous revision was dropped but the identifier still linked the copy to the
original.
The request sits in remove_hidden itself, because its per-category callers (the
remove-hidden edit and the optimizer's discard option) call it directly, not
through sanitize(). sanitize() asks only on its nothing-hidden path, where
remove_hidden returned before asking: the rewrite is a new file either way.
Checked: the new tests a_redacted_or_sanitized_copy_gets_a_new_identifier and
a_remove_hidden_copy_gets_a_new_identifier fail before this change (the request
is missing after apply, and after remove_hidden) and fail again when the apply
call, the nothing-hidden branch or the remove_hidden request is reverted. cargo
test -p pdfcraft-redact passes with the tests included (152 passed, 1 ignored);
cargo clippy with warnings denied and cargo fmt --check are clean.
* docs(cos): say when a full save regenerates the file identifier
The writing section of the crate README described full saves without saying what
happens to /ID. It now says that an ordinary save keeps the identifier, that a
document which asks for a new one (redaction and sanitizing) gets both strings
regenerated, that SaveOptions::id_entropy supplies entropy for reproducible output,
and that an encrypted file keeps its first string because the key depends on it.
Checked: the text matches the code added in e1f761cc (regenerate_id, new_id_bytes,
require_full_save_with_new_id) and the tests in crates/cos/tests/file_identifier.rs.
No code changed in this commit.
* cos: hash the whole rewritten file into the new identifier
The identifier of a requested rewrite hashed only the first 64 KiB of the written file.
With caller-supplied id_entropy the identifier is deterministic, so two rewrites of
documents with the same length that differ only after byte 65,536 received the same
/ID, and a later edit of a large document could keep its previous /ID.
Every written byte is now hashed. The output is read once: one loop over 4 KiB chunks
feeds both halves of the 16-byte identifier, so the extra cost is one linear pass over
the output. Measured with a probe in the workspace dev profile (opt-level 1): hashing
64 MiB took 0.024 s, about 24 ms per 64 MiB of output. A release build was not measured.
Ordinary saves never reach this code, and their output is unchanged.
Checked: crates/cos/tests/file_identifier.rs gains
the_identifier_depends_on_the_whole_rewritten_file. It fails on the branch head before
this change ("content past 64 KiB must change the identifier", both identifiers equal)
and passes after it; reverting only the chunk loop to the 64 KiB prefix makes it fail
again. The six existing tests in that file pass, and so do the full cos and redact test
suites.
* cos: keep the key valid when an encrypted rewrite has no usable /ID
A requested rewrite of an encrypted file whose /ID was absent, had one entry, or had a
first entry that is not a string wrote no identifier at all, although the document had
asked for a new one.
The reader takes the empty string as the key input for these shapes (Document::first_id),
so the rewrite now writes the empty string (<>) as the first entry and a new second entry.
The key stays valid. A file whose first entry is a string keeps that string, as before.
Ordinary saves and unencrypted rewrites are unchanged.
Checked: an_encrypted_rewrite_without_a_usable_identifier_still_gets_a_new_one in
crates/cos/tests/file_identifier.rs covers the three shapes for RC4-128, AES-128 and
AES-256. After the rewrite each file opens with its password, the first string is empty
and the second is new. On the branch head the test fails with "the rewrite writes a
pair" (left 0, right 2). Reverting only the guard to the head's early return makes it
fail again. Isolated, the one-entry shape fails with left 1, right 2, and the non-string
shape with left Some(0), right Some(16). The other seven tests in the file pass.
* cos: derive the identifier from caller entropy with a fixed hash
With SaveOptions::id_entropy the identifier was built with std's DefaultHasher, whose
algorithm the standard library does not specify, and with write_usize (4 bytes on wasm32,
8 on 64-bit targets) and native-endian integers. The same entropy and the same file could
therefore give different identifiers on different platforms or toolchains, although the
documentation promised the bytes depend only on the entropy and the written file.
The identifier is now FNV-1a (64 bit, the published constants) over fixed-width big-endian
fields and the whole written file: the salt, the lane, id_seed and the length, then the
bytes. With caller entropy the salt is the FNV-1a hash of that entropy. Without entropy it
is still the standard library's random hash of a process counter, so only the salt differs
between the two paths. Every written byte is hashed, in one pass that feeds both lanes.
Cost: this pass is slower than the SipHash it replaces. A temporary probe in the workspace
dev profile (opt-level 1) hashed 64 MiB through both lanes in 0.129 s (about 500 MiB/s),
against 0.024 s before. A release build was not measured. The cost is paid once per
requested rewrite; ordinary saves do not reach this code.
Checked: the golden test fixed_entropy_rewrite in crates/cos/tests/file_identifier.rs pins
the 16 bytes for a fixed entropy and the standard fixture. On the previous hash it fails
(left [64, 145, ...], right [50, 125, ...]). An independent re-implementation of the same
formula in Python, run over the bytes the writer hashed (everything before startxref),
gives the same 16 bytes. The unit test fnv1a_is_the_published_64_bit_algorithm checks the
constants against the published vectors. FNV is not collision-resistant against chosen
content; that matters only if someone could choose a file's identifier, and the identifier
only has to be unique.
* Repair review branch integration
* cos: take fresh identifier entropy from the browser on wasm
Without SaveOptions::id_entropy the salt of a rewritten file's identifier was
RandomState and a process-wide counter. On wasm32-unknown-unknown those hash
keys are fixed at compile time, so two fresh web apps that followed the same
allocation and call sequence reproduced the same salt and the same identifier
for identical output, and a redacted copy stayed linkable to its source.
The default salt now comes from the browser first: crypto.getRandomValues
(Web Crypto, through js_sys, sixteen bytes folded with the same FNV-1a the
identifier uses), or Math.random where that API is missing, and only when
neither is reachable the previous RandomState salt. js-sys and wasm-bindgen
join pdfcraft-cos as wasm32-only dependencies; both are already in the
dependency tree of the web shell, and native builds compile neither. Caller
entropy is untouched: with id_entropy the identifier is still a pure function
of that entropy, the seed and the written bytes.
Checked: separate_fresh_processes_get_different_identifiers re-runs this test
binary as two fresh child processes over the same fixture and fails when the
default salt is made constant (so does the in-process divergence test);
stand_in_for_a_fresh_process is its no-op child half. Both pass with the real
salt. On WebAssembly no browser test harness exists, so the browser path is
covered by the wasm32 cargo check and by construction, not by a test. cargo
check --target wasm32-unknown-unknown -p pdfcraft-cos passes, cargo test -p
pdfcraft-cos and -p pdfcraft-redact pass, and clippy with warnings denied is
clean on both.
* Extract full-save trailer assembly
Assisted-by: OpenAI Codex
---------
Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com>
Co-authored-by: FredApps <FredApps@users.noreply.github.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
Since b1f221611 the dictionary is read within the allowance too, so the test
that gave unshare no allowance was refused reading the dictionary, before the
image was decoded: turning the image's refusal into "can't be read" left every
test green (Claims review of 5bef9b0e7). The test now also gives exactly what
reading the dictionary costs, so the image is what is refused.
The same review: the README lists the other images that can't be stored
again (more than one component, no size) and names ImageBudget::decode, not
Document::decoded_image_within, which #5569 made private;
SharedJbig2Budget's doc and remove_hidden's comment name the dictionaries read
to know copies too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Main (#5569) decodes every image through cos's ImageBudget, which counts the
JBIG2 decoder's own work and the globals it reads, through main's own
restore_shared_jbig2 and image::without_globals for the images that share a
redacted image's /JBIG2Globals too. This branch replaces both with
globals::unshare, so the merge keeps unshare and drops them.
without_dictionary now decodes through ImageBudget::decode instead of
Document::decoded_image_within, which main made private. That closes the gap
the PR noted: the decoder's own work (up to hayro-jbig2's 2^30 steps an image)
was charged one unit of the run budget. apply passes its run's allowance, what
clearing left of it, as main's restore did. Past it apply is refused with
ImagesPastAllowance, as main's restore was. remove_hidden gives unshare an
allowance of its own (sanitize::unshare_unmarked), and past it is refused with
SharedJbig2Budget: "redact fewer pages at a time" means nothing there, and
Optimize's discard runs it too. SharedJbig2Budget's text now says finding and
storing again.
Tests: main's allowance test runs on unshare, with ImageBudget::with_work(0)
refused as ImagesPastAllowance, through unshare_unmarked as SharedJbig2Budget,
and u64::MAX storing one image. Test helpers decode through ImageBudget, since
Document::decoded_image is no longer public.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Conflicts:
- crates/redact/src/lib.rs: main added the run's ImageBudget to Applying
and Scope::new; kept both sides (the removed-text fields and the image
budget), with main's refused-images check before the destructuring.
- crates/redact/src/tests.rs (auto-merged): this PR's two tests that build
a Scope or an Applying now pass main's ImageBudget.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyG234xeYkJfr41E6iP
The run and byte limits were per page, and a glyph's /ToUnicode text can
be 256 UTF-16 units, so a short content stream could keep gigabytes of
removed text across pages. The document now keeps at most 16 MiB; past
that the rest is not kept and the metadata check fails closed, as for any
other text it couldn't keep (CodeRabbit's retained concern on #2539).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyG234xeYkJfr41E6iP
Reviews kept finding ways a small file could make this run long or large:
readable() fetched the same objects again for each image that names a
dictionary (~3.9 s each when two large objects push each other out of
cos's caches), and the memo of names grew with distinct arrays times
their entries (~3.7 GB from 2 MiB). Nothing bounded the run as a whole.
Everything unshare reads now goes through one Run: what a reference
leads to is fetched at most once a run for each way it is read (a size, a
filter, parameters, an entry, a flag, /F, a colour space, an ICC
profile), looked up before it is fetched and kept only as what is read
of it. Objects fetched, names remembered, and what is decoded (a unit for
each 4 KiB) count against RUN_BUDGET (2^22), and past it the run is
refused with SharedJbig2Budget. A scan reads a few objects a page, so a
book of thousands of pages sharing one dictionary stays well within it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Conflicts:
- crates/engine/src/comments.rs (comment_kind): main added the
Freehand Highlight arm, this branch the Replace Text arm; both kept.
- crates/automation/tests/it/automation.rs and
crates/ui-egui/tests/it/comments.rs: both sides appended tests at the
end of the file; both kept, each test closed on its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
A kept image that named the dictionary added every dictionary it names to
what had to leave, in one pass. An image using only one of those, earlier
in the file, had been passed over, so it kept that dictionary and the
redaction refused, saying something else referred to the data. In the
other order it was stored again. Now only the names that matched have to
leave, so another dictionary stays with what uses it, in either order.
The variant's doc says what is counted: something that names the data,
JBIG2 or not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
When nothing that names the dictionary is stuck and something else refers
to it (a private key, say), the refusal said 1 item(s) couldn't be stored
without it. It now counts 0 and says something else refers to the data.
remove_hidden's refusal gets a test: the comment has gone by then, and the
text doesn't say nothing was changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
apply works on a copy, so nothing changes when it refuses, but remove_hidden
isn't transactional: a direct caller's document is part changed by then. The
message said nothing was changed for both. The app's callers work on copies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Redaction starts from the images a run replaces or removes and the forms
and soft masks under them, so a dictionary used by something that left
another way stays, with apply returning Ok: a stamp or widget appearance, a
Type 3 glyph, an /Alternates entry, a /Mask, or an image with /Filter by
reference or /DecodeParms through a reference to a reference. A
byte-identical copy stays too, which is what pdfseparate then pdfunite
leaves, and Remove Hidden Information never takes a dictionary out.
Compare what a full save would keep before and after the run instead, the
same walk write_full makes, now cos::reachable. What left names the
dictionaries, in any entry of its parameters; what stays that names one, or
a copy, is stored again without it; and the walk is taken once more, so a
dictionary still kept refuses the run, whatever keeps it. An image with
/Width, /Height or /Filter by reference is read, and is one bit in the copy,
as it was when drawn. /Alternates stays with what is stored again, and a
stream that isn't JBIG2 refuses rather than being stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
BFlat
Grzegorz MartensandClaude Opus 5.5
dotCooCoo
mtlcl410
FredApps
FredApps
oracola91
vangel76andClaude Sonnet 5.5
Josias Eli Rodriguez
Hamza Naveedandskylar taylor-barrick
mtlcl410andClaude Opus 5.5