pdfcraft/crates/redact

pdfcraft-redact

Layer L4 (may use edit): redaction, architecture §11.2.

// Marks are Redact annotations (pdfcraft-annot: Shape::Redact { quads, overlay }).
let marks: Vec<Mark> = marks(&doc);                 // page, areas (user space), fill, overlay text
let report = apply(&mut doc, None)?;                 // everything, or Some(&[pages])
clear_marks(&mut doc, None)?;                        // remove marks without applying

apply removes, under every marked area:

• text: glyphs are cut out of Tj/TJ/'/" and replaced by a TJ displacement of the same advance, so the rest of the line keeps its exact position. Widths come from /Widths, /W//DW (composite fonts, embedded CMaps with codespace and CID ranges), Type 3 font matrices, and approximations for the standard 14. A glyph goes when it overlaps an area by more than a fifth of its size (or a point) both ways; zero-size text goes when its origin is inside;
• images: fully covered → removed; partly covered → a copy with the covered pixels cleared (8/16-bit and 1/2/4-bit images, image masks; Flate/LZW/RL/A85/AHx, and CCITT fax or JBIG2 as the last filter, decoded through ImageBudget::decode, a JBIG2 image with its /JBIG2Globals; /DecodeParms given by reference, numbers inside included, count). The copy is Flate, or for a one-bit image of one component (a scan as a rule) CCITT Group 4, or JBIG2 when it was JBIG2, whichever is smallest as stored, coded from the cleared samples; it refers to no globals. Globals hold the shapes of every image that shares them (a scan's pages, often), those of marks under the area included, so whatever left the document, by any route, takes its globals with it on the full rewrite: anything else the save would keep that names them, or a copy of them (the same bytes once decoded), is stored again without them, as a copy is (Flate, Group 4 or JBIG2, whichever is smallest; Report::images_restored). When one can't be (it isn't JBIG2, its colour space is known to have more than one component, it has no size or one past what redaction clears, its data is in another file, or it can't be decoded), or something else refers to them, apply refuses and changes nothing (RedactError::SharedJbig2Globals). remove_hidden takes them out the same way, but it has changed the document by the time it refuses: a caller passes a copy it can drop. Another dictionary holding the same shapes in other bytes isn't found, nor is a copy nothing names, and an image that hasn't left keeps its globals with its pixels: one another page still draws, or one something that doesn't draw it still refers to. What an image names as globals is read by object number, whatever generation a reference gives, as a reader that doesn't compare generations draws it (cos reads such a reference as null); what one is stored again with, its size, filters and parameters, is read as cos reads it. Finding what to store again reads each object at most once for each way it is read (a /JBIG2Globals chain named part way, a ring of references or a chain of more than 33, or a value read through a reference of another generation, may be read again), and what it fetches, remembers and decodes (a decode that fails counting as much as it was allowed, and an image the dictionaries it names) counts against one budget for the run; past it the redaction is refused (RedactError::SharedJbig2Budget). The images stored again, and the dictionaries read to know copies of them, are decoded within the run's image allowance, every filter stage and the JBIG2 decoder's own work included, after what clearing spent: past it apply is refused as one clearing too many images is (RedactError::ImagesPastAllowance), and remove_hidden, which chose no pages, as one past the run's budget. Codecs redaction doesn't decode (DCT, JPX) are removed whole (fail-closed), and so is a JBIG2 image whose page isn't its size (the renderer draws the whole page into the image, so the rows cleared wouldn't be the ones shown) or of more than one bit or component, and Group 3 on bytes (/K 0 or more, /EncodedByteAlign): PdfCraft's renderer draws some of its layouts (the fill ending each end-of-line code on a byte) as other rows than the decoder reads, so a mark over what is shown needn't cover what is cleared; inline images under an area are removed. Images with separate /SMask or /Mask data, and images whose data decodes to more than three times their declared raster and a byte per row, plus 4 KiB (ASCIIHex text broken into lines under Flate is a little over twice; for a fax image's code, about a byte per pixel), are removed whole; cleared copies drop /Alternates so alternate pixels cannot survive; A run that would decode more of its images than one redaction may (pdfcraft_cos::ImageBudget, filters, decoders and pixels alike) is refused and changes nothing: such an image can't be cleared, and removing it whole would take what no mark covers with it.
• vectors: covered paths are removed; partly covered paths and shadings are clipped so nothing paints inside the areas;
• form XObjects: rewritten recursively into new objects (pages sharing the original keep it);
• annotations whose rectangle overlaps (with their pop-ups and their replies, /IRT), and form fields with a widget under a mark;
• the marks themselves, replaced by boxes in their fill colour (and overlay text) drawn into the page as a /PCMark /Redaction stream;
• metadata that repeats the removed text (#2448): /Info entries containing it or wholly part of it as whole words (three letters or digits at least), and the catalog's and pages' XMP streams whose text (element content, attribute values other than xmlns declarations, comments and processing instructions, entities decoded) contains it or has a value wholly part of it, removed whole. When the catalog's packet goes, its PDF/A (pdfaid) and PDF/UA (pdfuaid) identification, read by namespace, is written back in a packet holding nothing else: the part number, the PDF/A conformance letter, amendment and revision, and PDF/UA's part and revision, each kept only if it is the short code it must be and doesn't itself hold the removed text (a one-character code can't, so it is kept even when the text couldn't be searched for), plus a fixed extension schema for pdfuaid under PDF/A-1 to -3, left out with the PDF/UA claim if the redacted words are among its words. Report::identification says, for each, whether the claim was kept or lost (a value that isn't a valid code or holds the text, or a packet that can't be read); the notification and redact_apply say it too. /Info entries that remain aren't mirrored into the new packet, so a PDF/A validator may still report them. The removed glyphs' text is collected in runs by position (character and word spacing included, so letter-spaced text stays one word), and each run (two or more letters or digits) and each of its words of three or more are matched without regard to case or spacing, and also by their letters and digits alone when they have three or more: "Blue fin" and "BLUE-FIN" hold "Bluefin", and so does a heading whose letters TJ numbers spread apart (collected as "B L U E F I N", since a TJ gap can't be told from a word space). A value is wholly part of a run when its letters and digits are, starting and ending where the run's words do, so "True" isn't part of "untrue". This over-matches only towards removal: "is a" also holds "Lisa". Both sides are compared in Unicode compatibility form (NFKC) and lower case, so a decomposed "Müller" (u and U+0308), full-width letters and a "fi" ligature from a ToUnicode map match their plain forms. A string behind an FF FE mark, which ISO 32000-2 §7.9.2.2.1 doesn't define, is read as UTF-16LE, here and in the places reported below, rather than through PDFDocEncoding, in which it could never match. A value that shares only words of one or two letters with the removed text, and isn't wholly part of it, stays: Sanitize's metadata category removes the rest. Report::metadata counts what went.
  • Fail-closed. When the check can't know what the removed text was, every /Info string and every catalog and page XMP stream goes: more than 100,000 runs on a page, a run over 64 KiB, or more than 16 MiB in the whole document removed (the collection limits); a removed glyph whose text isn't known (no /ToUnicode or encoding gives it); a form XObject removed without being read, because it is nested past the depth limit (MAX_DEPTH, 50) or its content can't be read; or more than 20,000 terms, or a run longer than 4,096 characters, to search for. Once the search's work limit (2^28 steps) runs out, every entry and packet not yet checked goes too. An XMP packet that isn't valid UTF-8 or UTF-16, can't be decoded strictly (damaged or cut-short filter data, or more than one filter), or decodes to more than 16 MiB goes without being read, and a value whose normalized form would be more than four times as long (U+FDFA alone is 18 letters) counts as holding the text.
  • Not scrubbed, but reported (Remove hidden information's categories remove them): bookmark titles, attachments (names in the document's list of embedded files, file names and descriptions, and the files of file attachment comments), the comments that remain (their text, rich text, subject and author; not form fields or pop-ups), and XMP streams on anything but the catalog and the pages (images, fonts, form XObjects, annotations and other objects). apply changes none of them, but counts those that still hold the removed text, matched as above, in Report::still_holding (Elsewhere), so the interface and the redact_apply tool can say where it is; this search has a work limit of its own, past which every place counts. When the removed text couldn't be searched for (the cases above in which every /Info string goes), every one of them that holds any text is counted and unsearched is set; an XMP stream or rich text stream that can't be read counts as holding it. The walks are bounded: at most 100,000 bookmarks and 100,000 name-tree nodes (32 deep) are read, each once, and a walk cut short counts one more place.

A verification pass re-reads each redacted page and fails the operation if any glyph or inline image is still under an area. It checks every content stream the page had, whatever its dictionary calls it; only the overlay streams this apply added (Report::overlays) are skipped. Text is placed with the font and size set by Tf or by an ExtGState's /Font through gs, and a form XObject starts from the text state of the content that draws it. Content that can't be decoded makes the operation fail rather than leave it unredacted, and so does an inline image the renderer's tokenizer reads no image for (RedactError::Unreadable): where the page's operators resume after it would be a guess. A Redact annotation with neither a usable /QuadPoints array nor a usable /Rect makes it fail with RedactError::UnreadableMark before anything changes, instead of skipping that mark. The count covers the marks on the pages being applied.

Tiling-pattern cells are separate streams. Applying a mark removes a pattern resource when no surviving paint uses it. If a cell remains reachable after rewriting the selected pages and may contain text, images, replacement descriptions, or unreadable content, Apply refuses with a RedactError::Pattern and restores the document and pending marks. This includes patterns owned by Forms and patterns inherited by Forms. Vector-only retained cells remain supported. The bounded check is conservative: an unrelated retained pattern on a selected page can also cause refusal when its content cannot be proven safe. If a selected page had a pattern resource, the check covers all pages after rewriting so another page cannot keep the same cell in the saved PDF.

Before changing any page, apply prepares the redaction appearances with a shared 8 MiB budget for their uncompressed PDF content across the selected pages. It bounds repeated-text counts and working line buffers before allocating them, and measures and encodes each repeated line once. Invalid or excessive appearances return RedactError::OverlayLimit without changing the document. Reduce the marked area, disable repeated text, or apply fewer pages at a time. This limits appearance generation; it is not a limit on the input PDF or the whole process's memory.

Structure cleanup resolves named MCIDs and indirect content references. Form content references are updated to the cleaned copy; unchanged sibling descriptions remain. The structure tree's object references (/OBJR) to the annotations and form fields apply removes are dropped, as Sanitize drops them, so a full rewrite of a tagged PDF can't keep a removed comment or field value. A tree where that can't be done (an element written in place, or one too large to walk) makes apply fail with RedactError::Tags rather than leave the annotation reachable.

Replacement descriptions (/Alt, /ActualText, /E) are removed from marked-content spans changed by cleanup, from every structure element above them (found through both /P and /K), and from elements that lose an annotation reference, which also lose /T. An element written in place above one whose descriptions go makes apply fail with RedactError::Tags. Unchanged spans retain their descriptions; unused property entries are retired during redaction and hidden-content cleanup, including aliases in shared/inherited resources. Descriptions still used by surviving pages or forms are kept.

A redacted page that inherits /Resources from the page tree gets them pinned (by reference; a direct inherited dictionary is made one shared object first) before it is rewritten. The parent's /Resources is then retired unless another page still inherits it, so it cannot keep a removed XObject reachable. Surviving form XObjects lose the resource entries for removed XObjects that neither their own content nor a form without resources of its own drawn through them still draws, for the same reason, and a redacted page keeps a name under that rule too. Pages that aren't redacted are left untouched.

Remove hidden information / Sanitize

let found = sanitize::scan(&doc);                          // [(Hidden, count)] per category
sanitize::remove_hidden(&mut doc, &[Hidden::Comments])?;   // chosen categories
sanitize::sanitize(&mut doc)?;                             // all of them

Categories: metadata (/Info, XMP), file attachments, comments, form fields (flattened so the values stay visible), hidden text (render modes 3 and 7, or wholly off the page), hidden layers (content of off optional-content groups, then the groups), bookmarks, links/actions/JavaScript, and /PieceInfo private data. XMP metadata, private data, associated files (/AF) and additional actions (/AA) are removed from every object reachable from the catalog that owns them (images, fonts, Forms, annotations and fields as well as the catalog and pages), never from name maps such as resource dictionaries. Content is read strictly: a page whose content can't be read in full is refused, not cleaned in part. Both make the next save a full rewrite.

patterns finds phone numbers, e-mail addresses, credit card numbers (Luhn-checked), US Social Security numbers and dates for Search & Redact. A phone label such as Tel. or Fax: also enables Italian international geographic numbers with a slash, such as Tel. +39 0426/123456 or Fax: +390426/123456 (#3205 follow-up). The country code and national number are matched together, with the significant initial 0 and 6 to 11 national digits described in the AGCOM numbering plan published by ITU. This narrow case accepts at most one space after +39 and one slash immediately after the area code; international slash numbers without a phone label, or with country codes other than +39, remain unsupported. Patterns are heuristic: dates, protocol identifiers and longer number runs are retained, and users still need to review the marks before applying them.

codes holds the redaction code sets (U.S. FOIA and U.S. Privacy Act exemptions): CodeSet::from_id("foia")?.overlay(&["(b)(6)"]) gives the overlay text for the picked codes. Custom code sets aren't supported yet.

Not yet: pattern locales, struct-tree/alt-text cleanup, re-encoding of DCT images (they are removed instead).