Keeps main's recovery renewal after applying redactions alongside the
combined notification, and reads XMP and rich-text streams with their
/DecodeParms given by reference made direct, as main's stream readers do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCHD1R3rywQBPCHXZA
prune_type3_from_resources decoded every appearance without resources of
its own into one list before counting any operators, each up to the
single-stream limit, so their sum had no bound. They are now decoded one
at a time against one page's DecodeBudget, as the page content is, and
passing it refuses the redaction.
A tiling pattern must have /Resources of its own, so its cell never
draws through the page's. Since #3680, Apply Redactions refuses a page
whose tiling pattern shows text or has no /Resources (pattern_guard), so
prune_type3_from_resources reads only soft mask groups now. Remove Hidden
Information runs no pattern guard: there, a page font that only such a
malformed pattern uses goes, as it did before this PR.
unlink_type3 still follows a pattern's cell, for patterns on pages the
redaction does not touch. Its test now puts the pattern on a second page.
When a page lists a Type 3 font, prune_type3_from_resources (#5197) adds the
operators of the page's annotation appearances without resources of their
own. Each appearance was checked against the 2,000,000 cap on its own, so
several under it could add up to far more, and the prune went on with a
total no other path allows.
The appearances now share the running total, and passing it refuses the
redaction with the same kind of error as the other caps.
unlink_doomed and unlink_from_forms asked, name by name, whether the
remaining content still draws an XObject, and each question parsed the
content and decoded every form without /Resources of its own again. A
holder with many aliases of one removed image, drawn by a large
inheriting form, multiplied that decoding by the number of aliases.
names_drawn now works out once per content which names it draws,
directly or through such forms (each read once), and whether any of it
could not be read; every alias is answered from that set. A name not in
the set while something could not be read still fails closed.
Redaction and hidden-content removal cleared /Alt, /ActualText and /E from
the structure element owning removed marked content, but kept its /T title,
which producers often fill with the element's text. Ancestor cleanup and
in-place description detection shared a constant that also omitted /T, while
annotation removal had its own list that included it.
One DESCRIPTIONS policy (Alt, ActualText, T, E) now serves the owning
element, its ancestors, in-place detection and annotation removal.
#2539 review: the scrub removes the catalog's XMP packet whole when it holds
the removed text, and with it the file's PDF/A and PDF/UA claims, which
Report didn't say. The identification (pdfaid part, conformance, amd, rev;
pdfuaid part, rev) is now read by namespace (roxmltree, already in the build
through pdfcraft-xfa), each value kept only when it is the short code it must
be and doesn't hold the removed text (one-character codes always), and
written back in a packet holding nothing else, with a fixed pdfuaid extension
schema under PDF/A-1 to -3 (left out, with the PDF/UA claim, if the redacted
words are among its words). Report::identification says, per standard,
None / Kept / Lost; the engine passes it on as
EditNotice::RedactionIdentification, the Apply notification says it after the
count, and redact_apply returns identification { pdfa, pdfua }.
Tests (redact): the_identification_of_a_removed_packet_is_written_back,
a_packet_without_identification_is_only_removed,
an_identification_that_cannot_be_kept_is_reported_lost,
unsearched_text_keeps_only_one_character_codes,
the_extension_schema_is_left_out_when_it_uses_a_redacted_word; engine
applied_redactions_report_the_pdfa_identification_written_back; automation
redact_apply_reports_the_identification_written_back; ui-egui
applying_says_the_pdfa_identification_was_written_back. With the write-back
replaced by "nothing kept", the redact tests fail (Kept expected, None
found); with the notice not pushed, the engine, automation and UI tests fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
#2539 review: ISO 32000-2 §7.9.2.2.1 defines text strings in PDFDocEncoding,
UTF-16BE (FE FF) and UTF-8 (EF BB BF) only, so an FF FE string was read
through PDFDocEncoding and Cyrillic or CJK text in it could never match.
It is now read as UTF-16LE for the search (fail-closed), in the /Info scrub
and in the bookmark, attachment and comment counts.
Test: a_utf16le_string_is_searched_as_utf16le (failed before: 0 /Info
entries removed; with only the count reverted, bookmarks 0 instead of 1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
#2539 review: a decomposed "Müller" in a UTF-16BE /Title, a U+FB01 ligature
from a ToUnicode map and full-width letters never matched their plain forms.
Both sides are now normalized to NFKC and lower case (unicode-normalization,
already in the build through pdfcraft-crypt's stringprep, so no new crate).
Normalized text more than four times its input's length (U+FDFA and the like)
isn't searched: it counts as holding the text.
Tests: unicode_forms_of_the_removed_text_match (failed before: Title, Subject
and Keywords kept), a_value_that_normalizes_to_far_more_text_counts_as_holding_it
(failed with the growth limit raised to 100: 0 removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
#2539 review: a heading tracked with TJ numbers ([(B) -300 (L) -300 ...] TJ)
was collected as "B L U E F I N", which neither held "Bluefin" nor had a word
of three letters, and a value spelled "Blue fin" or "BLUE-FIN" didn't hold the
term "bluefin". A TJ gap can't be told from a word space, so the glyph ends stay
as they are; instead every term with three letters or digits is also looked
for by its letters and digits alone, and a value is part of a run when its
letters and digits are, starting and ending at the run's word edges ("True"
still isn't part of "untrue"). It over-matches only towards removal.
Tests: a_heading_tracked_with_tj_numbers_matches_the_metadata,
a_value_spaced_or_hyphenated_otherwise_matches (both failed before: /Title
kept, metadata 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
The unread-form branch was tested for a form nested past the depth limit only; a form under a
mark whose filter can't be decoded is removed unread too, and now has an end-to-end test, seen
failing with that branch's flag removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
Report::still_holding (Elsewhere) counts the places apply leaves as they are that still hold the
removed text: bookmark titles, attachment names, file names and descriptions, the text and authors
of the comments that remain, and XMP on objects other than the catalog and the pages. Nothing is
changed; Remove hidden information removes them. The text is matched as the metadata check
matches it (one Search type now serves both), with its own work limit. When the removed text
couldn't be searched for, every place holding any text counts and `unsearched` says so. The walks
read each outline item and name-tree node once, within caps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
End-to-end tests for a removed run over 64 KiB, more than 20,000 terms, a run over 4,096
characters, and XMP that is corrupt, cut short, behind two filters, over 16 MiB or in another
encoding. Each was seen failing with its guard removed or loosened.
The corrupt and cut-short cases showed that the tolerant decoder returned the part of the packet
before the damage, which was then searched as if it were the whole packet. XMP is now decoded
strictly, and a packet behind more than one filter goes unread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
The page font prune counted the page content, forms without /Resources
and annotation appearances, but not tiling patterns or soft mask groups
without /Resources. Those draw through the page's resources too, so a
Type 3 font only they selected left the page and their text lost its
font. They are now read as well, only when the page lists a Type 3
font.
Two new tests fail without the change.
An annotation's /AP can hold entries that no reader draws. Only the
normal, rollover and down appearances (N, R, D) now keep a Type 3 glyph,
so an extra entry such as /X can no longer keep a redacted glyph program
in the saved file.
Rewriting a page also dropped a Type 3 font that only an annotation
appearance without /Resources selects, although that appearance draws
through the page's resources. The page prune now reads those appearances
too. It reads them only when the page lists a Type 3 font, so other files
are not refused over an appearance it cannot parse.
Two tests, both failing without this change.
unlink_type3 (#4579) unlinks the Type 3 glyph programs that nothing
shows any more. It found what is still shown by scanning page content
and the forms reached through /XObject only. Annotation appearances,
tiling pattern cells and soft mask groups draw too. A glyph only they
showed counted as unused, so its program left /CharProcs, or the whole
font lost /CharProcs and left the page's /Font. This ran on every Apply
Redactions, even when the mark covered text in another font.
The scan now also walks the appearance streams of each page's
annotations, the tiling pattern an scn or SCN selects and the group of
the soft mask a gs sets. Each is scanned once per stream, font and
holder, through its own /Resources or its caller's, with no current
font. Forms take the same path through scan_stream_object.
Three tests, one per drawer. All three fail without the fix.
content_draws tells whether some content draws an XObject, directly or
through a form that has no /Resources of its own and so draws through
its caller's. Since #4356 the recursion passed the caller's content
again instead of the nested form's decoded data, so a name drawn only
inside such a form was never found.
Both callers then dropped the entry: unlink_doomed for page-tree
resources (Apply Redactions and the content pass of Remove Hidden
Information) and prune_doomed_xobjects for surviving forms. An image
removed under a mark but still shown elsewhere through such a form went
missing from there as well.
The recursion reads the nested form's data again, as it did before
#4356. Two tests cover the page-tree case and the surviving-form case.
Both fail without the fix.
Filtered inline image data has no length, so its end is an EI chosen
among candidates, and the content parser chose by a different rule
than the renderer's tokenizer (hayro-syntax): the renderer also takes an
EI with no whitespace before it, and checks a candidate by reading what
follows as operators. With image data such as "7FEI 1 0 0 1 200 0 cm",
the renderer ran a cm that redaction read as image data, so text was
drawn 200 pt from where removal and its verify pass looked.
The parser now takes each inline image's end from the renderer's
tokenizer (already in the tree through pdfcraft-cos), so both read the
same operators after the image; a test runs both on the same fixtures.
Only where that tokenizer reads no image (no EI it accepts, or
parameters it can't read) does the parser fall back to its own rules,
counting the image in Parsed::ambiguous_images, and applying redactions
refuses such a page as unreadable.
Structure cleanup cleared alternate text, actual text and expansions
only on the element that directly owned changed marked content, and an
element that lost an annotation reference kept its descriptions unless
it lost every kid. An ancestor's /ActualText covering a redacted span,
or a link element's /Alt repeating the address of a removed link, stayed
in the saved file. Elements that lose an annotation now always lose
their descriptions, and every ancestor of an affected element (found
through both /P and the tree's /K, which may disagree) loses its own.
An element written in place above one with a description to clear
refuses the operation, as other in-place cases already do.
Finding which marked content a redaction changed copied every operator
into the buffer of each MCID enclosing it, so content nested n deep was
held n times: a modest stream of nested BDCs could exhaust memory.
Each operator inside marked content is now serialized once, and each
MCID keeps the ranges its sequences span. The bytes compared are capped;
MCIDs past the cap count as changed, so their descriptions are cleared
rather than kept.
Clearing the pixels of a partly covered image decoded its stream with
the global 1 GiB allowance and only then cut it to the raster length its
Width, Height and bit depth declare. A small Flate stream could make
every such image allocate and inflate far more than it can use. Decoding
now stops at twice the raster plus a byte per row (filter-chain and
predictor overhead) and a little slack; an image past that is removed
whole, as an undecodable one already is.
After applying redactions, removed images and forms were unlinked from
page-tree resource dictionaries but not from the resources of forms the
page still draws. A form sharing the page's original resource dictionary
(a common producer layout) kept the removed image reachable, so a full
save wrote it. The walk over surviving forms now drops XObject entries
for removed objects that the form's own content does not draw, and fails
closed when that cannot be determined.
The redaction interpreter started every form XObject with an empty text
state (no font, size 0, no spacing) although a form inherits its caller's
graphics state, and it ignored the /Font entry of an ExtGState set by gs.
Text shown that way was placed at the wrong positions, so removal and
the verify pass could both miss glyphs under a mark. Forms now start
from the caller's state, and gs sets the font and size as Tf does.
The engine's renderer cross-check test used the gs case as its example
of interpreter/renderer disagreement; it now runs the check directly on
an unredacted page, which also covers an input stream labelled as a
redaction overlay.
Deleting a field took it off its pages and out of the field tree but left
its entry in the form's /CO array. That reference kept the field
dictionary, its value and its widgets reachable, so a full save after
applying redactions over the field still wrote the redacted value.
mtlcl410
Dylan M. Taylor
ljkx
bflatastic
oracola91
mtlcl410andClaude Opus 5.5
aiongg
