145 Commits

Include renames

Author	SHA1	Message	Date
Hamza Naveedandskylar taylor-barrick	97cef90e01	Redaction: a pattern nothing paints with any more leaves the page (#3680) ... * Redaction: a pattern nothing paints with any more leaves the page A tiling pattern's cell is a content stream of its own and can draw anything, text included. Redacting a shape filled with such a pattern removed the fill from the page's content -- the page looked right, a black box -- but left the pattern in /Resources /Pattern, and its cell, with the text that was under the mark, was written into the saved file unchanged. The pass does not enter pattern cells, so it cannot redact inside one. What it can do is notice that nothing paints with the pattern any more. The graphics state now carries the fill and stroke pattern a `scn` or `SCN` selected (and `cs`, `g`, `rg`, `k` and their stroke forms clear it), every painting operator that survives the rewrite records the pattern it drew with, and a pattern the page selected but no surviving paint used leaves the page's resources. Measured on the report's file: before applied: 1, (SECRET) in the saved bytes after applied: 1, no /Pattern on the page, (SECRET) gone A pattern some surviving paint still uses is kept, and so is its cell: with two boxes filled from one pattern and only the first marked, patterns_removed is 0 and the resources keep the entry -- the second box still shows the word, so it is not a leak. Report gains patterns_removed. Fixes #3647. * Keep a pattern every operator that can paint with it still uses Two defects CodeRabbit found in the first version of this change, both of which could remove a pattern the page still draws with. Pattern use was recorded for path painting only. Text shows with the fill pattern too -- and with the stroke pattern in the stroking render modes -- as do image masks and anything a form draws, and a form's content is interpreted with a graphics state of its own, so a pattern it inherits is never seen inside it. Content like /Pattern cs /P0 scn 20 140 100 30 re f BT (KEEP) Tj ET with the box redacted therefore lost the fill of text still on the page. Every operator that can paint with the current pattern now records it: text (except mode 3, which paints nothing, and mode 7, which only clips), inline images, and image or form XObjects. Only path painting knows exactly what it drew and whether the paint survived, so the others are recorded as they are met. That can keep a pattern whose paint was in fact all redacted, which costs an unused resource entry, while dropping one something still draws with takes paint off the page. /Resources /Pattern may be an indirect object, like any value. Reading the reference itself as a dictionary found none and pruned nothing, so on such a file the cell -- with the text that was under the mark -- stayed in the saved bytes, which is the leak this change is for. It is resolved now, and what is written back is a dictionary of this page's own, so pruning never reaches through a /Pattern dictionary another page shares. The controls in the test: without the first fix the surviving text's pattern is removed (left: 1, right: 0), and without the second the indirect file prunes nothing (left: 0, right: 1). * M8: refuse retained unsafe tiling patterns * M8: guard shared appearances and narrow pattern scans * M8: update TLS dependencies for new advisories * Redaction: the comment on prune_patterns said more than the code does `prune_patterns` drops the `/Pattern` resource entry; the `scn` that selected it stays in the content, so the page names a pattern its resources no longer hold. The comment claimed the selector went too (noted in review). ISO 32000-1 8.6.6.2 makes that operand a key of the `/Pattern` subdictionary, so the open question is whether a reader stops reading the stream at the stale name. `a_page_whose_pruned_pattern_is_still_selected_renders` answers it with the renderer rather than with prose: the redacted page is saved, reopened and rasterized, and * it renders with no error and the same 300x300 page box, * the text written *after* the stale selector is still painted -- which is exactly what a reader would drop if it gave up at the bad operand, * the redaction overlay is black over the box the pattern filled. Control: with the drawing after the selector removed, the pixel assertion fails ("the text after the stale selector is still painted"), so it is reading that text and not a stray pixel. The selector is left in place rather than cut out: removing operators is a rewrite of the content this pass does not otherwise do, and nothing surviving paints with that colour. `pdfcraft-render` is a dev dependency of this crate only -- layer 4 on layer 3, which the layering allows. * WIP: pattern guard depth bound sized from the interpreter's * Pattern tests: keep the guard's cases reaching the guard now a hit path is removed whole * Pattern automation tests: keep the pattern in use where they test the guard, since a hit path is now removed whole * Redaction: a form without resources keeps the page pattern it still paints A form with no /Resources of its own paints with its caller's patterns, but its surviving pattern paint was recorded only in the form's own output and never reached the page. With the page's own fill under the mark removed whole, prune_patterns then dropped /P0 from the page while the form outside the mark still painted with it, so that fill lost its paint in the saved file. Pass the painted names up from a form that inherits its caller's resources. Only the painted side goes up, so this can keep a pattern, never prune one. * Redaction: a pruned pattern does not stay named by the page tree prune_patterns writes only the page's own /Resources, and the guard walked only pages' effective resources. A /Pages node that still named the pruned pattern kept its cell reachable from Root, so a full save wrote the cell, text included, while apply returned Ok. Three shapes did it: a page with no /Parent inheriting from the root, a /Parent that disagrees with the /Kids tree, and a root /Resources shadowed by a nearer holder that was retired. Patterns pruned from a page are now unlinked from page-tree holders no page inherits from, as unlink_doomed does for XObjects, when every /Parent agrees with the tree. Whenever a pattern was pruned (or a selected page had any), the guard also inspects the /Resources of every /Pages node Root reaches down /Kids and up each page's /Parent chain, so whatever stays reachable is proven safe or the redaction is refused, leaving the document unchanged. * Redaction: a Type 3 glyph without resources keeps the page pattern it paints A Type 3 font with no /Resources looks the names in its glyph descriptions up in the resources of the page or form showing it, so a glyph outside the mark can paint with the page's pattern. Text recorded only the fill and stroke pattern of the graphics state, so with the page's own fill under the mark removed whole, prune_patterns dropped the pattern and the glyph lost its paint. Showing such a font now keeps every pattern of the scope, as a form without /Resources keeps what it paints. * Redaction: say what text-recorded pattern use keeps Text, inline images and XObjects record their pattern before the hit test, so a pattern painted only by redacted text keeps its cell. The guard refuses a cell with text or images; a vector-only cell stays in the file. The comment said this cost only an unused resource entry. Pin the behaviour in a test. * Redaction: the page tree is followed up every /Parent link A full save writes every node a /Parent link reaches, but the guard followed /Parent only from pages and stopped at the first node it had already seen, and unlinking never followed it. A /Pages node reached only through the /Parent of a /Kids node -- a root /Pages with a /Parent, or an intermediate node whose /Parent leads out of the tree -- kept naming the pruned pattern, and its cell, text included, stayed in the saved file while apply returned Ok. The guard now follows /Parent from every node it visits, page or /Pages, with the same seen-set against cycles. Unlinking treats a root with a /Parent as an inconsistent tree, like a disagreeing link, and leaves the decision to the guard, which inspects every node those links reach and refuses an unsafe cell. * Redaction: a long page tree does not exhaust the pattern guard's budget The page-tree walk counted every node, pages included, against the guard's 16,384-object budget, so redacting a pattern page of a 17,000-page document was refused with a limit error where it had applied before. Only the holder nodes the guard inspects count against that budget now; the walk itself is bounded at a million nodes, as unlink_doomed bounds the same tree. * Redaction: a Type 3 font set by an ExtGState or shown in a form keeps the page patterns The resourceless Type 3 check covered only a font set by Tf in the scope that paints. A Type 3 font without /Resources set through an ExtGState /Font entry was not flagged, and a glyph shown inside a form with /Resources of its own recorded its patterns in the form's scope, which the page never saw. In both cases the page's own fill under the mark was removed whole, prune_patterns dropped the pattern, and the glyph outside the mark lost its paint. An ExtGState font is now checked the same way, and a glyph shown in any nested form marks the page as keeping every pattern it selected. This can only keep patterns; whatever is kept is checked by the guard. * Redaction: a pruned pattern a full save would still write is proven safe or refused The walks that check retained pattern cells follow the references they know -- resource categories, the page tree, appearances -- but a full save writes whatever the trailer reaches by any reference. A direct /Parent dictionary, a /Kids array on a page, or a stray resource entry (/Shading naming the pattern stream) kept a pruned cell, text included, in the saved file while apply returned Ok. pdfcraft_cos gains reachable_objects, which runs the same breadth-first walk write_full uses (factored out as full_save_order, so the two cannot drift). Before apply succeeds, every pruned pattern that walk still reaches is inspected like a retained cell: vector-only passes, anything else refuses with the existing pattern error. It runs only when a pattern was pruned and costs one walk of the reachable objects. * Redaction: a direct /Parent dictionary is part of the page tree too A /Parent given as a dictionary rather than a reference is written inline by a full save, but the page-tree walk followed /Parent only as a reference, and unlinking took such a tree for consistent. A root /Pages whose direct /Parent held /Resources naming the pruned pattern kept its cell in the saved file. The guard now walks a direct /Parent like any node (its /Resources, /Parent and /Kids), and unlinking counts any /Parent that is not the reference /Kids came from as an inconsistent tree, leaving the decision to the guard. * Redaction: page-tree walks queue each node once and borrow /Kids On a hostile tree whose /Pages nodes share one long /Kids array, the guard's page-tree walk and pattern unlinking pushed every /Kids entry of every node before checking whether it had been seen, and copied the array on each visit. Both now queue only nodes not seen yet and read /Kids in place; unlinking also stops at the first node reached twice, since its /Parent can agree with at most one of the nodes above it, and leaves that tree to the guard. * Redaction: a Type 3 font with /Resources null is resourceless too /Resources null is the same as no /Resources, but the check counted any entry, so such a font's glyph outside the mark lost the page pattern it painted with. A Type 3 glyph shown by a kept annotation appearance is a known limit, pinned as an ignored test: appearances are not interpreted for pattern use, and under ISO 32000 the glyph resolves in the appearance's own resources. * Redaction: a pruned pattern is recorded along its whole reference chain A pattern name can lead to a reference-valued object whose value is the cell (/P0 7 0 R, object 7 being 6 0 R). Pruning recorded only object 7, so the reachability backstop checked an object nothing reached while a stray reference kept the cell, object 6, in the saved file. Every object along the chain is recorded now, bounded like the other walks. * Pattern tests: pin the file identifier where two saves are compared main now gives a redacted rewrite a new identifier from fresh entropy, so an incremental and a full save after Apply differ in /ID alone. The test that checks they are the same save passes fixed entropy to both. * Redaction: replies and pop-ups removed on other pages feed the shared-appearance check Apply checks every page for retained unsafe pattern cells when a removed annotation's appearance is shared with one that stays on an unselected page. Only annotations removed under a mark fed that check. Replies and pop-ups that go with them in step 2b (removed_everywhere), often listed on another page, did not, so a text-bearing cell in an appearance a kept annotation shares with them escaped the all-pages check and Apply returned Ok. Their appearance references now go into removed_appearances too, with the same limit. Raised by FredApps' review of the rebase. --------- Co-authored-by: skylar taylor-barrick <hi@skylar.cc>	2026-11-06 10:01:42 +01:00
5 people	d7ce875dd6	redact: independent redaction proof machinery (snapshot, sweep, budget) (#5557) ... * redact: add the independent redaction proof machinery * redact: harden the proof's paint check and count removed codes The manifest counts character codes, not raw bytes: a composite font's codes are several bytes, so the byte length overstated what was cut. Annotation appearances are checked whatever their storage (a direct dictionary has no reference of its own to dedupe on), and a stream shared by several annotations is checked at each annotation's own placement. The graphics-state stack and the clip holes of one walk are bounded; deeper or more is unverifiable, never unbounded memory. Even-odd clips cut holes only where the rule can be told cheaply - all subpaths axis-aligned, disjoint, few - and anything else keeps its paint reported, failing closed. The /W widths of a descendant font cost bounded work, like a CMap. * redact: settle the survivors a page sweep counted before its work ran out A page that the search budget stops on left the counts made on earlier pages unsettled: the survivors were seen but never reported, and the regions that owned them were never marked. The sweep now stops and still settles, so a located survivor is reported where it sits (the page reads unswept either way, and the proof still fails). * redact: prove the redacted output of every supported encryption The authenticated-reopen regression now covers all four crypt algorithms the writer supports, not only AES-128: the caller's credentials prove the saved output, and empty or wrong credentials still fail closed. * redact: answer the review round on the proof A glyph lying under several regions is now accounted for by every one of them, so overlapping marks each carry their own share in a shared proof instead of the first one absorbing the rest and the others reading as if no text was there. Each finding is recorded once per region as it is found, so a page that paints over a region a thousand times neither grows the report nor the memory with repeats. The byte search walks the haystack once (the pattern's prefix function is built once), so a needle of one letter over a haystack full of it can no longer spend comparisons the search budget never charged for. A numeric XML character reference may carry any number of leading zeros; the padded form now decodes like the short one, so a survivor spelled that way in the XMP is still found. The /W widths of a descendant font are paid for from the shared work budget: a form invoked again rebuilds its fonts, and a per-font counter alone let the same array be paid for many times over. * redact: sweep the proof's survivors out of every place a reader could look Plant the removed text where the review listed it and expect the sweep to report it: a compressed object stream, a page thumbnail, a form field value, an XFA packet, and an object nothing refers to. A file whose page content is bigger than the proof may decode for one stream is now shown to come out unverified on its surfaces and its region, never clean. * redact: spare the names a redacted word legitimately sits in, and charge failed decodes A redacted word that is also a structural name (a /Subtype, a /Filter, a /BaseFont) is structure, not a leftover copy: the name surface is now swept against the matches the input itself had, and only a surplus of matches is a survivor. The input's names are counted when the snapshot is taken, for the region's needles and for strings added later alike. A decode that fails after doing part of its work paid nothing: the proof's cumulative decode allowance now charges every attempt, failures included, so the same bomb cannot be decoded (and fail) again and again for free. * redact: count the input's names once each, and say so when the count runs out The name allowance counted one entry per occurrence, so a document whose pages repeat the same structural name paid for the same scan again and again; names are now kept once, with their count. A name search that runs out of work read as no matches, which would have swept names with none allowed: that is reported on the text surface now, and the proof fails rather than pass on an uncounted allowance. * Bound redaction proof searches across many needles Assisted-by: OpenAI Codex * Redaction proof: drop find_all, which nothing calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cargo.lock: put aho-corasick in its sorted place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Redaction proof: review notes A word split across a TJ array is looked for whole. Overlapping marks count a removed copy once. A spent decode budget is a problem, not a raw-only stream. Short needles stay out of the attachment text search, and an indirect /Decode counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 09:15:07 +01:00
5 people	9f04acccef	M10: OCR engines, confidence, preprocessing and a review workflow (#977) ... * M10.20: plan Unicode text for OCR output in Type3 faces (fonts, edit) OCR output used to flatten every character beyond WinAnsi to a silent `?`. This adds the two pieces that let the OCR layers write the recognized words as they are; the OCR code that uses them follows in M10.21. pdfcraft-fonts - `TextPlan::for_words` maps each word to the standard Helvetica when WinAnsi covers it, else to Type3 fonts drawn from the craft-fonts faces (the Japanese document face first, then the rest of the build input). Characters no face has a glyph for are counted (`missing`), never dropped silently. At most 240 characters per font and 2400 per plan, so one hostile word cannot balloon. - Which face has which character is asked once per (face, character) for a plan, not once per word. - `face_has`, `craft_glyph`, `win_ansi_char`, `fits_win_ansi` are public for it. pdfcraft-edit - `add_type3_fonts` / `stamp_with_fonts` register a plan's fonts on a page under a free `PCOcr`, `PCOcr1`, ... name, build each font once and share the object with later pages. A shared face is registered under a free name on a later page that already has a font of the name it was given first, instead of replacing that page's own font. - The fonts carry real advances and a ToUnicode CMap, so extraction reads the text back. pdfcraft-redact - A test shows a hand-built Type3 text layer (the shape the OCR output writes) is redacted like any other text and the verifier passes. Tests: pdfcraft-fonts 62 passed, pdfcraft-edit 129 passed, pdfcraft-redact 33 passed; clippy -D warnings clean on the three crates. The name-reuse and the redaction tests build their fonts by hand, so they run without the craft-fonts build input; the tests that plan real faces skip without it and were not run in this commit's own check. * M12: give the hostile-XFA open test more room on a loaded machine The test opens a form whose calculate script sets a value 50 000 times, against the one-second budget of an unasked script and a five-second wall-clock bound on the open. Both sit close to what a machine running several suites at once can take, so the test can fail without anything being wrong. It now runs 20 000 iterations (the total is 19999) and allows 30 seconds, which still fails for a script that hangs or is not capped. Test-only; unrelated to the OCR work in this series, so it is a commit of its own. Tests: both hostile_xfa tests in pdfcraft-engine pass. * M10.21: pluggable OCR engines, preprocessing, review sessions and the OCR tools (ocr, engine, create, automation) Scan & OCR gets a second engine, optional raster clean-up, a review step before anything is applied, editable output and a batch mode, headlessly first: the interface for them follows in M10.22. pdfcraft-ocr - A `Recognizer` trait with two engines: the built-in ocrs (moved out of lib.rs) and the installed tesseract program, run as a fenced external process: fixed argv, allow-listed language codes, a private temp directory (mode 0700 on Unix), 120 s timeout, 64 MiB output cap, 50 000-word reading cap. A location from PDFCRAFT_TESSERACT is validated (absolute, existing, executable) and an invalid one fails closed instead of falling back to the PATH. `--list-langs` and `--version` are read from stdout or, for older versions, stderr. - Strategies: PrimaryOnly, ConfidenceWeighted (the second engine joins when the first reading's mean is under 70 or unknown) and a ROVER merge (higher confidence wins per word; against an ocrs word of unknown confidence a tesseract word wins only at 70 or more; capped at 100 M comparisons; a failing partner degrades to the primary with a note). - Preprocessing: opt-in auto-rotate, deskew, denoise and binarize on the RGBA raster, each with an exact inverse so word boxes map back to the scan. The 300 dpi normalization the steps are calibrated for runs only when a step is on: with every step off the render is read as rendered, as before. (Enlarging a 72 dpi render 4x on its own made the models read "due: €250" as "ue: ?250".) `OcrImage` is validated by construction (side and 64 MP area caps, checked before any buffer is allocated, including by the resize). - Confidence bands (High >= 90, Medium 70-89, Low < 70) drive review. pdfcraft-engine, pdfcraft-create - `OcrSettings` gains engine, strategy, the four steps, output mode, region, the skip flags and the low-confidence threshold; Automatic reads confidence weighted unless a strategy is chosen. - A recognition is a session: the document guard refuses a stale result with its reason, pages can be applied one at a time (the guard is taken again after each applied page), and applying is one undoable step. A per-language dictionary (a plain-text file: words added from any document stay on disk after it closes). - Output is searchable (an invisible layer) or a NEW editable document of visible text at the recognized boxes (`create::from_contents`); words beyond WinAnsi use the Type3 faces of M10.20 in both, with the shared plan across pages. - Engine discovery never blocks the interface: results are cached, a stale answer is shown while a background probe retries, and the models search and `--version` are cached. Batch: pages or files that already have text are skipped unless forced, and inputs sharing a file name are found up front. pdfcraft-automation - ocr_recognize (engine, strategy, steps, region, output mode), ocr_words (with a retained review handle), ocr_review_apply, ocr_dictionary, ocr_status and ocr_recognize_files. The editable `out` must be a new file inside --root and is refused for a searchable run (it used to be ignored), two batch inputs of one file name are refused before anything is read, and the batch folder is checked for existing outputs. pdfcraft-ui-egui - Only what the new API needs: the Recognize Text dialog reads with `recognizers(&settings)`. `ocr.correct` is registered with its label in every catalog; the screen itself is M10.22. Tests: pdfcraft-ocr 78 passed; pdfcraft-create passed; pdfcraft-engine and pdfcraft-automation 450 passed and 1 failed (the automation lib 17, automation.rs 157, hostile_args 2 among them); pdfcraft-ui-egui lib + ocr + control tests 391 passed; clippy -D warnings clean on the five crates. The one failure is in a test this commit does not change: recognize_text_makes_a_scanned_page_searchable fails where the models are installed (they read "quick" as "auick" at that render size); the roadmap on main already records it. The real tesseract binary ran in tesseract::tests::recognize_runs_the_installed_binary (tesseract 5.4.0, PATH). * M10.22: OCR Verify screen, the Recognize Text options and the interface catalogs (ui-egui) Scan & OCR ▸ Correct recognized text: a full-window screen where a recognition is reviewed before anything is applied, plus the dialog and batch options for the engines of M10.21. OCR Verify screen - Four panes: the page list, the source page with confidence-coloured word boxes (click to select, drag to draw a region), the recognized text, and a zoom plus word inspector. Corrections commit on Enter (an empty commit removes the word), Remove/Restore, "+ Dictionary" (a per-language file in the preferences folder, so a word added from a document stays on disk after it closes), per-word and per-page Verified, F8 / Shift+F8 walk the low-confidence words, Ctrl+T marks the page reviewed, Ctrl+Tab moves the pane focus, and review edits undo within the panel. - Re-OCR of the whole page or a drawn region. A region re-read moves the review marks of the words it keeps to their new positions and selects the first new word; a whole-page re-read clears that page's marks (and the undo brings them back). The region outline is scaled onto the pane, not mapped through user space again, so it is right on rotated and cropped pages. - Accept / Reject per page or whole. Pages accepted one at a time each apply once (the document guard is taken again after each applied page; an edit made elsewhere in between is still refused with its reason), and Accept in searchable mode is one undoable step. Editable mode writes a NEW `<name>_ocr.pdf` through a save dialog and never the source. A real Cancel keeps the pages already read. - The engine and strategy menus disable a choice that cannot run and say why; the check never waits on a process (a background probe fills a cache), and choosing Automatic moves "Primary only" to Confidence weighted. Recognize Text dialog and batch - Engine, strategy, the four clean-up steps and the skip flags are remembered with the other choices (a saved Automatic without a strategy reads confidence weighted). The batch refuses two inputs of one file name and existing outputs before reading anything, notes low-confidence pages at the settings' threshold, and its toast has a Cancel (the file in flight finishes). Catalogs and attribution - The 90 new interface strings are in all 21 catalogs (the Lithuanian ones were written without native review, and the Bulgarian ones avoid U+045D, which the bundled monospace face lacks). ATTRIBUTION.toml carries the new hashes, a note on the Lithuanian entries and the README screenshot; ATTRIBUTION.md is regenerated. - README: a screenshot of the Verify screen, taken with `cargo xtask screenshots` from a synthetic scan (`shot --ocr-verify on`) built from allowed assets; the page reads it with the real engines. Tests: pdfcraft-ui-egui lib 373 passed (1 ignored: the Windows hinting measurement probe), tests ocr 6, control 46, fonts 9; clippy -D warnings clean on the engine, ui-egui, pdfcraft and xtask packages; `cargo xtask assets` ok. The review state machine has unit tests for every fix above; each was seen failing with its fix reverted. * M10.23: parity entries, roadmap and localization notes for the OCR workflow Records what M10.20-M10.22 ship, and what they do not. parity/acrobat-features.toml - Status changes against main: ocr.rotation-detection and ocr.deskew planned -> shipped (the opt-in steps of M10.21, with their tests); ocr.correct-recognized- text and ocr.ocr-engines planned -> partial. Other Area I entries get their evidence and notes brought up to date. - ocr.correct-recognized-text: the evidence includes the one-page-at-a-time accept and the region re-read tests; the notes say what the headless tools cover (ocr_words with a retained review, ocr_review_apply through the same stale- document guard, ocr_dictionary) and what is still UI-only (suggestions, per-word Verified marks, region re-reads). - ocr.ocr-engines stays partial rather than shipped: part of its evidence is conditional (the real-binary test runs only where tesseract is installed and skips elsewhere, and the process-level tests are Unix-only). It cites the tests that always run (location validation and the fail-closed rule, --list-langs parsing from stdout or stderr, the merge rules, the default strategy) and says which part is not proven everywhere. - `cargo xtask parity`: area I 32.7% -> 44.2% weighted (5 -> 7 shipped, 7 -> 9 partial); overall 52.8% -> 53.0% shipped and 58.2% -> 58.5% weighted. ROADMAP.md - One progress-log entry (2026-10-13, newest first) and the banner line. It says the catalogs are 21 with the Lithuanian additions not natively reviewed, that the end-to-end redaction test of the Type3 layer needs the craft-fonts build input, and that recognize_text_makes_a_scanned_page_searchable still fails where the models are installed, as main's log already records. docs/localization.md, docs/localization-parity.md - The Lithuanian entry count (2,578 -> 2,668) and the counts of the other catalogs that carry entry counts are re-measured with the 90 new entries; the Lithuanian OCR entries are marked as not natively reviewed. Checked: `cargo xtask parity` (numbers above, evidence tests exist) and `cargo xtask assets` pass on this tree. * M10.24: ocr: apply a late result only to the document it was read from The Recognize Text dialog closes on Go, so the document can be edited, or have text redacted, while a long run is still reading. The result was applied without the stale check the tools and the Verify screen use, so words read before a redaction could be written back as invisible text. The run now keeps its guard and applies through the checked path. Accept with the editable output in the Verify screen had the same gap (Exit keeps the review open): it now refuses when the document changed since the review started. Both new tests fail with their guard switched off. * M10.25: ocr: ocr_dictionary takes dictionary file names only Updating sorts, lower-cases and rewrites the file it is given, so pointed at .env or a README it would corrupt it. The tool now accepts only ocr-dictionary-<language>.txt style names, for reading as well as writing. The size-limit test used a file name the rule now refuses first, so it renamed its fixture to keep testing the size limit. The new test fails with the rule switched off. * M10.26: pt-BR: keep (s) on the word counts and say mecanismo for engine Two review fixes from a native speaker: the two word(s) lines had lost their (s), so a count of 1 read as 1 palavras, and engine was translated both as motor and mecanismo. Mecanismo is the Brazilian word for a software engine and two lines already used it, so the other six now match. * M10: fit the rebased OCR workflow to current main's structure * Fix OCR workflow issues from the review Honor the Recognize dialog's editable-output choice: its result opens in the review screen (whose Accept writes the editable document) instead of an invisible layer. Make page acceptance idempotent: a second Accept entire page never appends the OCR layer twice and keeps later corrections. Move page and region re-OCR onto a worker thread with a status line and a cancel that discards the reading. Skip the Tesseract probe for jobs that name their engines. Bound Tesseract's stderr while the child runs, not only after it exits. Surface the editable output's substitution notes in the tools' and the run's results. Also repair the OCR batch tool against the current structure: inputs are held to the read root again, and a batch whose inputs share an output name is refused before the output folder is created, compared the way the default filesystem compares names; and the editable writer test's word stays inside the fixture's pages. * Apply pre-review fixes to the OCR workflow Hold the automation OCR tools to the document's copying permission, as the other text tools are: the recognized text is left out of the results when copying is forbidden, and the paths that carry the words out of the document (the editable output, the review word list) are refused. Mark the ocr_dictionary tool as overwriting its path. Close the searchable layer's graphics state around its invisible text so the rendering mode cannot leak into later content. Read the OCR user dictionary folder from the portable data folder in portable mode, like the recovery store does. Resolve the attribution manifest's leftover conflict markers and regenerate the generated attribution. Describe the review flow, the searchable layer's Type3 fonts and the parity evidence references accurately, and drop an undeclared copy of the recovery module. * OCR: reject readings that exceed the word limit Assisted-by: OpenAI Codex * OCR: new documents go through the page tree builder main added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OCR: review notes and lints The editable output refuses a file created during recognition. ocr_words takes skip_text_pages and force_ocr. Automatic OCR checks the engine it will use. The stderr test stays below the kill cap, and a failed --ocr-verify shot exits with an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OCR: restore the Verify screenshot attribution, follow the moved constants The merge dropped the screenshot row with the translation rows. The model filename constants live in ocrs.rs now, so the license check reads them there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribution: the Verify screenshot entry goes after the header comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribution: the screenshot row in the table follows the manifest order Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 23:39:55 +01:00
4 people	60f60fc0a2	Give a redacted or sanitized copy a new file identifier (#3734) ... * cos: give a redacted or sanitized full save a new file identifier A full save that the document asked for (Document::require_full_save_with_new_id) now writes a fresh /ID pair, so a redacted or sanitized copy does not claim to be the file it came from (ISO 32000-2 section 14.4). Ordinary saves are unchanged. An encrypted file keeps the first string of /ID, because the file key is derived from it, and gets a new second string. An encrypted file whose /ID is not a pair of strings is left as it was, as before. SaveOptions::id_entropy lets a caller or a test supply entropy; with it the new identifier depends only on that entropy and the written bytes. Without it the identifier mixes the standard library's random hash keys and a process-wide counter, the same facility the engine uses for new encryption keys. Checked: the new tests in crates/cos/tests/file_identifier.rs fail on the clean base (compile errors for the missing API), fail on assertions when only the API is in place, and fail again when each pinned line is reverted (the regeneration call, the first-string keep, the flag, the entropy input, and the ordinary-save guard). cargo test -p pdfcraft-cos passes, and clippy with warnings denied is clean. A probe that wrote 16 ordinary full and incremental saves gave byte-identical files with and without this change. * redact: give a redacted or sanitized copy a new file identifier Applying redactions, sanitizing and removing hidden information now ask the cos layer for a new file identifier (Document::require_full_save_with_new_id), so the saved copy no longer carries the /ID of its source. Before this change the previous revision was dropped but the identifier still linked the copy to the original. The request sits in remove_hidden itself, because its per-category callers (the remove-hidden edit and the optimizer's discard option) call it directly, not through sanitize(). sanitize() asks only on its nothing-hidden path, where remove_hidden returned before asking: the rewrite is a new file either way. Checked: the new tests a_redacted_or_sanitized_copy_gets_a_new_identifier and a_remove_hidden_copy_gets_a_new_identifier fail before this change (the request is missing after apply, and after remove_hidden) and fail again when the apply call, the nothing-hidden branch or the remove_hidden request is reverted. cargo test -p pdfcraft-redact passes with the tests included (152 passed, 1 ignored); cargo clippy with warnings denied and cargo fmt --check are clean. * docs(cos): say when a full save regenerates the file identifier The writing section of the crate README described full saves without saying what happens to /ID. It now says that an ordinary save keeps the identifier, that a document which asks for a new one (redaction and sanitizing) gets both strings regenerated, that SaveOptions::id_entropy supplies entropy for reproducible output, and that an encrypted file keeps its first string because the key depends on it. Checked: the text matches the code added in e1f761cc (regenerate_id, new_id_bytes, require_full_save_with_new_id) and the tests in crates/cos/tests/file_identifier.rs. No code changed in this commit. * cos: hash the whole rewritten file into the new identifier The identifier of a requested rewrite hashed only the first 64 KiB of the written file. With caller-supplied id_entropy the identifier is deterministic, so two rewrites of documents with the same length that differ only after byte 65,536 received the same /ID, and a later edit of a large document could keep its previous /ID. Every written byte is now hashed. The output is read once: one loop over 4 KiB chunks feeds both halves of the 16-byte identifier, so the extra cost is one linear pass over the output. Measured with a probe in the workspace dev profile (opt-level 1): hashing 64 MiB took 0.024 s, about 24 ms per 64 MiB of output. A release build was not measured. Ordinary saves never reach this code, and their output is unchanged. Checked: crates/cos/tests/file_identifier.rs gains the_identifier_depends_on_the_whole_rewritten_file. It fails on the branch head before this change ("content past 64 KiB must change the identifier", both identifiers equal) and passes after it; reverting only the chunk loop to the 64 KiB prefix makes it fail again. The six existing tests in that file pass, and so do the full cos and redact test suites. * cos: keep the key valid when an encrypted rewrite has no usable /ID A requested rewrite of an encrypted file whose /ID was absent, had one entry, or had a first entry that is not a string wrote no identifier at all, although the document had asked for a new one. The reader takes the empty string as the key input for these shapes (Document::first_id), so the rewrite now writes the empty string (<>) as the first entry and a new second entry. The key stays valid. A file whose first entry is a string keeps that string, as before. Ordinary saves and unencrypted rewrites are unchanged. Checked: an_encrypted_rewrite_without_a_usable_identifier_still_gets_a_new_one in crates/cos/tests/file_identifier.rs covers the three shapes for RC4-128, AES-128 and AES-256. After the rewrite each file opens with its password, the first string is empty and the second is new. On the branch head the test fails with "the rewrite writes a pair" (left 0, right 2). Reverting only the guard to the head's early return makes it fail again. Isolated, the one-entry shape fails with left 1, right 2, and the non-string shape with left Some(0), right Some(16). The other seven tests in the file pass. * cos: derive the identifier from caller entropy with a fixed hash With SaveOptions::id_entropy the identifier was built with std's DefaultHasher, whose algorithm the standard library does not specify, and with write_usize (4 bytes on wasm32, 8 on 64-bit targets) and native-endian integers. The same entropy and the same file could therefore give different identifiers on different platforms or toolchains, although the documentation promised the bytes depend only on the entropy and the written file. The identifier is now FNV-1a (64 bit, the published constants) over fixed-width big-endian fields and the whole written file: the salt, the lane, id_seed and the length, then the bytes. With caller entropy the salt is the FNV-1a hash of that entropy. Without entropy it is still the standard library's random hash of a process counter, so only the salt differs between the two paths. Every written byte is hashed, in one pass that feeds both lanes. Cost: this pass is slower than the SipHash it replaces. A temporary probe in the workspace dev profile (opt-level 1) hashed 64 MiB through both lanes in 0.129 s (about 500 MiB/s), against 0.024 s before. A release build was not measured. The cost is paid once per requested rewrite; ordinary saves do not reach this code. Checked: the golden test fixed_entropy_rewrite in crates/cos/tests/file_identifier.rs pins the 16 bytes for a fixed entropy and the standard fixture. On the previous hash it fails (left [64, 145, ...], right [50, 125, ...]). An independent re-implementation of the same formula in Python, run over the bytes the writer hashed (everything before startxref), gives the same 16 bytes. The unit test fnv1a_is_the_published_64_bit_algorithm checks the constants against the published vectors. FNV is not collision-resistant against chosen content; that matters only if someone could choose a file's identifier, and the identifier only has to be unique. * Repair review branch integration * cos: take fresh identifier entropy from the browser on wasm Without SaveOptions::id_entropy the salt of a rewritten file's identifier was RandomState and a process-wide counter. On wasm32-unknown-unknown those hash keys are fixed at compile time, so two fresh web apps that followed the same allocation and call sequence reproduced the same salt and the same identifier for identical output, and a redacted copy stayed linkable to its source. The default salt now comes from the browser first: crypto.getRandomValues (Web Crypto, through js_sys, sixteen bytes folded with the same FNV-1a the identifier uses), or Math.random where that API is missing, and only when neither is reachable the previous RandomState salt. js-sys and wasm-bindgen join pdfcraft-cos as wasm32-only dependencies; both are already in the dependency tree of the web shell, and native builds compile neither. Caller entropy is untouched: with id_entropy the identifier is still a pure function of that entropy, the seed and the written bytes. Checked: separate_fresh_processes_get_different_identifiers re-runs this test binary as two fresh child processes over the same fixture and fails when the default salt is made constant (so does the in-process divergence test); stand_in_for_a_fresh_process is its no-op child half. Both pass with the real salt. On WebAssembly no browser test harness exists, so the browser path is covered by the wasm32 cargo check and by construction, not by a test. cargo check --target wasm32-unknown-unknown -p pdfcraft-cos passes, cargo test -p pdfcraft-cos and -p pdfcraft-redact pass, and clippy with warnings denied is clean on both. * Extract full-save trailer assembly Assisted-by: OpenAI Codex --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>	2026-11-05 21:25:30 +01:00
3 people	f9c150c17c	Add IBAN matching to Search and Redact patterns (#4332) ... * Add IBAN matching to Search and Redact patterns The Pattern/PATTERNS/from_id interface is consumed generically by redact_mark and the UI pattern combobox through the engine re-exports, so one new variant and matcher exposes IBAN in Search and Redact everywhere. A unit test shows valid compact and spaced IBANs are marked while wrong check digits, short runs, unknown countries, letter check digits and token-glued IBANs are not. Add an Iban variant to Pattern, list it in PATTERNS, handle it in label, id and find, and add an iban matcher with a country-code length table, compact or group-of-four-spaced acceptance, alphanumeric start and end boundaries and ISO 7064 mod-97 check-digit validation. Test: iban_matches_compact_spaced_and_rejects_near_matches failed before the change and passes after. * Address review for pull request 4332 * Address review for pull request 4332 * review: address CodeRabbit findings - patterns.rs: fix SWIFT registry IBAN lengths EG 27->29, LY 27->25, SD 24->18 and add NI 28, BI 27, DJ 27, OM 23 (crates/redact/src/patterns.rs:545,549,551,558-561) - patterns.rs: state table source as SWIFT IBAN registry without claiming full ISO 13616 coverage (crates/redact/src/patterns.rs:475) - patterns.rs: regression test with valid registry examples for EG, LY, SD, NI, BI, DJ, OM and a wrong-length EG rejection (crates/redact/src/patterns.rs:892) * Complete the IBAN country table and accept spaces only between groups The country table now lists all 89 countries of the SWIFT IBAN registry (release 99) as one sorted constant. Honduras, Russia, Yemen, Falkland Islands, Mongolia and Somalia were missing. A space is accepted only after a full group of four characters, so a code followed by ordinary words ("BA37 leaves at noon") ends the run instead of being tried against the check digits. Tests: the registry example of every country matches compact and printed, changes to one digit are rejected, and the table and the examples agree. * Advertise the IBAN pattern in the redact_mark tool schema The pattern enum in the advertised schema was kept by hand and still listed phone, email, credit-card, ssn and date, so clients that obey the schema could not select iban, which the runtime already accepted. The enum is now built from the engine's pattern registry, so it cannot drift from it again, and the description names iban. Tests: tools/list advertises exactly the registry's pattern ids and names each in the description, and an iban mark made through the tool covers the spaced German sample and nothing else. --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>	2026-11-05 20:32:24 +01:00
eliets2andeliets2	c15c909745	Fuzz target for redaction apply (#6305) ... Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com>	2026-11-05 20:11:21 +01:00
3 people	d579f26de3	M11: JBIG2 images share a symbol dictionary, up to 16 at a time (optimize.mono-compression) (#5910) ... * M11: JBIG2 images share a symbol dictionary, up to 16 at a time With Monochrome Images set to JBIG2, Optimize stored each image's letters in a symbol dictionary of its own, so a document of N pages carried its font's shapes N times. Now up to 16 images the run stores as JBIG2 share one dictionary in a /JBIG2Globals stream, holding the marks found in more than one of them; each image's text regions refer to it first, then to a dictionary of its own for the marks only it repeats. An image is stored that way when it comes out smaller, and a group only when what its images save is more than the globals take. - filters: encode_jbig2_shared codes a group: the globals are a symbol dictionary segment numbered 0 and associated with no page (7.2.6); each page numbers its segments from 1, as a segment refers only to lower numbers (7.2.5), and the shared symbols' IDs come first, as the dictionaries are referred to in that order (Table 31, SBSYMS). At most 16,383 shared marks, those in the most images, so a page keeps room for its own under the decoder's 32,767. - optimize: the run holds the 1-bit samples of the images it stores as JBIG2 (16 at most, 64 MiB at most) and stores them again sharing, weighing each with the longest reference the globals could have. Eight pages of 10 pt text rendered at 300 dpi take 41% less in Helvetica and 42% in Times than coded alone, 10% in Courier, and 46% in Helvetica at 600 dpi; groups of 2, 4, 16 and 32 save 19%, 33%, 45% and 47%. Redacting one image stores the others of its group again without the dictionary, so 16 keeps that to 15 images. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: test what the mutants found untested in shared JBIG2 dictionaries - filters: the group goes through encode_jbig2_shared, which refuses it for an image the encoder refuses; two marks in common are enough to share; past the 65,536 distinct marks a group holds, a mark first seen later isn't shared while those held count every image. - optimize: optimizing again keeps both the shared pages and the pages coded alone as they are; under Retain, averaged JBIG2 pages share nothing; a last group of two shares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: only images redaction can store again share a JBIG2 dictionary Redacting one image of a group stores the others again without the dictionary, which can hold the shapes of marks under the area, and refuses the redaction for one it can't: it takes image masks and images of one component as its own reading counts them (no DeviceN, the family named directly), and at most 64 Mi pixels. Optimize took DeviceN of one colourant and up to 80 million pixels, so such an image in a group made any redaction of the others fail, where before it was coded alone and the same redaction worked (the Consumers review). Only an image mask or a DeviceGray image named directly, of at most 2^26 pixels, now joins a group; others are coded alone. An engine test optimizes three pages, two DeviceGray and one DeviceN, and redacts the first: the DeviceN page stays alone, the second is stored again, and no image names the dictionary after a save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: optimize and redaction share one limit on the images redaction stores again The second Consumers review: Optimize's limit on images that share a JBIG2 dictionary copied redaction's private 64 Mi, with nothing tying the two, so a lower one in redaction would bring back refused redactions with no test going red. Both now read pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS. The engine test adds an image mask, the other kind that shares, so a redaction that stopped storing either kind again fails it. doc_optimize's description, the README, the parity note and ROADMAP say which images share (named directly). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * M11: a JBIG2 group is coded before its samples would pass the limit CodeRabbit on #5910: hold added an image to the group and then checked SHARED_BYTES, so a group could pass the 64 MiB it is documented to stay within by its last image, which can itself be up to 64 MiB (an image that shares has at most 2^26 pixels, and as its rows are whole bytes, at most that many bytes; a 1-pixel-wide one takes them all). hold_in codes the group first when an image would take it past the limit, and after when it has 16 images; with room for 10 bytes, images of 4 go two to a group, where the old order made one of 12. CodeRabbit's follow-up too: the 18-image test decodes every image and compares it with its samples, across the boundary between the groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY * optimize: count the globals' object framing; filters: test the shared coder's size limit - A JBIG2 group shares a dictionary only when it saves more than the globals' dictionary and data plus NEW_OBJECT (96 bytes: the object's framing, its xref entry, and an encrypted file's IV and padding), so a group that saves less than that no longer makes the file larger. - encode_jbig2_shared is Unsupported for an image taller than the decoder takes and codes one at the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filters: drop the unused rows binding in encode_shared page() no longer takes the rows since #5511. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filters: the shared coder's at-bound test says what it returns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>	2026-11-04 13:34:08 +01:00
Dylan M. Taylor	359ed56ffa	fix(redact): structure titles go with the descriptions of removed content (#5866) ... Redaction and hidden-content removal cleared /Alt, /ActualText and /E from the structure element owning removed marked content, but kept its /T title, which producers often fill with the element's text. Ancestor cleanup and in-place description detection shared a constant that also omitted /T, while annotation removal had its own list that included it. One DESCRIPTIONS policy (Alt, ActualText, T, E) now serves the owning element, its ancestors, in-place detection and annotation removal.	2026-11-02 23:34:18 -05:00
Dylan M. Taylor	69ddcd89bf	fix(redact): rewritten form XObjects drop /PCAdded (#5868) ... Page content streams rewritten by redaction or hidden-content removal already dropped /PCAdded, the source text of an added item, but a form XObject whose content was rewritten kept its original dictionary, /PCAdded included, so the removed text stayed readable in the saved file.	2026-11-02 23:31:39 -05:00
FredApps	ec25e7d327	M2.6: images decode outside the renderer only through an ImageBudget ... `ImageBudget::decode` is now the one public way to decode an image's CCITT fax or JBIG2 data: `Document::decoded_image` and `decoded_image_within`, which decoded with no limit on the work, are test-only in pdfcraft-cos, and pdfcraft-filters exports only `decode_ccitt_within` and `decode_jbig2_within`, which spend from a caller's allowance (`decode_jbig2` is gone, `decode_ccitt` private). So an operation over many images can't decode them without sharing an allowance; the compiler holds new callers to it. Doctests that must not compile check both crates; with the old entry points public they fail. Tests outside the crates decode through an unlimited ImageBudget. Streams in general (`Stream::decoded`) are left for a follow-up.	2026-11-02 14:24:21 +01:00
FredApps	74d51ebf4a	M2.6: a JBIG2 image stored again without its globals decodes within the run's allowance	2026-11-01 23:50:39 +01:00
FredApps	bc4acf7bfc	Merge origin/main into feat/jbig2-run-allowance	2026-11-01 23:50:04 +01:00
FredApps	616ddca371	M8: images stored again without shared JBIG2 globals are stored as a cleared copy is ... An image that shared a redacted image's /JBIG2Globals is now stored again as the smallest of Flate, Group 4 or JBIG2 coded alone (smallest), like a cleared copy, rather than always Flate, so the other pages of a scan don't balloon either. The shared-globals test's second page is a ragged scan and checks it is no longer Flate and is smaller.	2026-11-01 22:48:29 +01:00
FredApps	54d0ea7c59	Merge branch 'main' into feat/redact-bilevel-codec	2026-11-01 22:44:58 +01:00
FredApps	a87c4fb6b7	M8: redaction stores again without them the images that share a redacted JBIG2 image's globals, so the globals leave the file (protect.redact-image-pixels) ... A JBIG2 image's /JBIG2Globals hold the symbol dictionary of every image that shares them (jbig2enc -s and most scanners give one to all of a scan's pages), so the bitmaps of marks drawn only under a redaction stayed in the file while another page's image used them. Every image using the globals of a removed or replaced image (or of one in a removed or replaced form) is now decoded and stored again in place as Flate without them; a full save then drops them. One that can't be decoded for certain refuses the redaction (RedactError::SharedJbig2Globals). Report::images_restored counts them.	2026-11-01 22:14:52 +01:00
Grzegorz MartensandClaude Opus 5.5	8a6373525e	M8: the JBIG2 redaction test compares the whole saved picture ... One eight-byte search would miss a clearing that took only the first byte of it: the window is gone and seven covered bytes stay. The saved picture now has to be the one with every pixel under the mark cleared and no other changed, on both coded forms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-11-01 17:17:59 +01:00
Grzegorz MartensandClaude Opus 5.5	c14968edf1	M8: no stream of a saved file holds the pixels under a mark on a JBIG2 scan ... The tests of this checked that the JBIG2 code and its globals had gone, and that the pixels came out cleared. This reads every stream of the saved file as it would be shown, and searches them for pixels found only under the mark, and for pixels beside it, which have to stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-11-01 13:46:01 +01:00
Grzegorz MartensandClaude Opus 5.5	5660075295	M8: the test of rows too wide for Group 4 says Group 4 refused them ... Flate coming out didn't show that: Group 4 could have been coded and lost on size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-11-01 12:15:58 +01:00
Grzegorz MartensandClaude Opus 5.5	ade8ebbdb4	M2.6: a run pays for what a stream costs, not what it holds ... Four ways round the allowance, each measured by a review of hostile input: A Flate stream read as zlib that fails at once, and then as raw deflate that decodes everything before failing, gave nothing back and so was charged nothing, here and in DecodeBudget: 110 s for a 387 KB file, all decoded. What the retry decoded now counts, though the data is still dropped as it was. Fax and 1-bit Flate images paid by the byte for work done by the pixel: 19 bytes can be a blank page of 100 million pixels, about a second each. Every image decoded here now pays for its pixels first, as JBIG2 already did. A JPEG paid nothing, and what one asks for doesn't grow with its length: 156 bytes declared 80 megapixels, 72 s for an 8 KB file. A JPEG pays for its samples before it is decoded, in Optimize and in an export that has to decode one for its mask. Merging fragments paid for reading strips and not for the picture it built: 393 s for 1,154 bytes. It pays for the picture before allocating it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-11-01 00:20:43 +01:00
Grzegorz MartensandClaude Opus 5.5	083191f970	M2.6: every filter of a run's images pays, whether or not it works ... Filters were charged only for what they handed a JBIG2 decoder, once they had handed it over: a chain that failed, or went past its image's limit, cost its time and nothing else, and images that weren't JBIG2 weren't charged at all. A 734 KB file of 200 images of Flate inside Flate took an image export 111 s, and that grew with the file. It now takes 12 s. Each byte read and each byte a stage decodes is now a step of the same allowance, paid as it happens, so no stage runs past what is left. That holds for every image read through ImageBudget, and a fax image pays for its rows and its code. JBIG2 globals pay when they have to be read, and kept ones don't. Optimize's two reads that went round the budget go through it, and merging fragments shares it. Optimize reads an image twice, to pack it and to resample it, so a refused image is now counted once a run rather than once a read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 22:23:06 +01:00
Grzegorz MartensandClaude Opus 5.5	199354246b	M2.6: redaction reads past /Crypt, and what can't be averaged spends nothing ... Redaction checked the raw /Filter array, so /Crypt in a chain removed an image it could have cleared, though decoding leaves /Crypt out. It leaves it out too. Decoded only to be resampled, an image that can't be averaged came to None after the decode, having spent the run's allowance and perhaps been counted as refused. Each such case now stops before it. Found by CodeRabbit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 17:36:44 +01:00
Grzegorz MartensandClaude Opus 5.5	0f325d44b6	M2.6: a picture whose JBIG2 mask is past the allowance isn't drawn either ... Without its mask a picture shows what the mask hides, so the page leaves it out, as export does. Any other unreadable mask still leaves the picture, which is the renderer's tolerance for bad masks. What an export may hold is a function of its own, tested down to the step and at the top of the range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 14:24:53 +01:00
Grzegorz MartensandClaude Opus 5.5	4aa732ea91	M2.6: what is left out past the JBIG2 allowance is said, and redaction stops ... From a review of who reads a refusal. The rule is to leave it out and say so, and it held on some paths only. An image whose JBIG2 mask is refused no longer comes out opaque. The first image past the allowance reads as that and not as corrupt. A page reports what it left out. The Office warning stopped blaming forms. Redaction refuses the run, leaving the document as it was: removing a readable image whole would take what no mark covers with it, unreported. The 2 GiB an export holds is twice the document if that is more, so a large honest file loses nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 14:13:48 +01:00
Grzegorz MartensandClaude Opus 5.5	0ed339901e	M2.6: the JBIG2 images of an operation or a page share one allowance of decoding ... One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes them again. Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for each byte of the file, so a long scan is read whole. An image past it spends all that's left, and later ones aren't read, globals included. Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the picture out and says the page was truncated. Redaction removes it whole, as any image it can't read. Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73). It's a thread-local, unbounded until a page sets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 13:21:45 +01:00
Grzegorz MartensandClaude Opus 5.5	53874375cb	M8: test: a cleared copy stored as Group 4 or JBIG2 can be redacted again; the README's not-yet list ... The README still listed metadata and hidden-information sanitising as not done, though sanitize does it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 02:53:16 +01:00
Grzegorz MartensandClaude Opus 5.5	689a1e34cf	M8: test: a cleared image of more than one bit stays Flate ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY	2026-10-31 00:41:23 +01:00
Grzegorz MartensandClaude Opus 5.5	7f9554f29b	M8: a cleared black-and-white scan is stored as Group 4, or JBIG2 again, when smaller ... Redaction wrote every cleared copy as Flate, so a redacted Group 4 or JBIG2 scan page grew several times over until the optimizer ran. A cleared one-bit image of one component is now stored as the smallest of Flate, CCITT Group 4 and, when it was JBIG2, JBIG2 (PDF 1.4, which such a file already is), as stored, dictionary included: a tiny image can lose by Group 4's parameters. Both are coded from the cleared samples, so nothing of the original's code is carried over; rows Group 4 or JBIG2 can't code (too wide) leave Flate. Images of more bits stay Flate. Tests: a cleared one-bit Flate scan of ragged word blocks (where Group 4 is smaller than Flate) comes out Group 4 and an eight-bit image Flate, each decoding to the cleared samples (the whole buffer); a tiny image whose Group 4 code is shorter but whose stream isn't, and rows wider than Group 4 codes, stay Flate and clear; the redacted JBIG2 scan of #5461's test comes out JBIG2, smaller than Flate, with no globals and no stream holding the original code. The tests' image helpers decode through Document::decoded_image, as image codecs need. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 19:08:32 +01:00
Grzegorz MartensandClaude Opus 5.5	78fc1e5568	M8: redaction clears the covered pixels of JBIG2 images instead of removing them ... Part of #283. A scan stored as JBIG2 (as the optimizer writes them) is decoded through Document::decoded_image_within, its /JBIG2Globals included, and its covered pixels cleared in a Flate copy that refers to no globals, as CCITT fax images are. Before, any JBIG2 image under a mark was removed whole, blanking the page. Still removed whole (fail-closed): a JBIG2 page of another size than the image, which decoded_image refuses (the renderer draws the whole page into the image, so the rows cleared wouldn't be the ones shown); globals that can't be read; damaged data; an image claiming more than one bit or one component, even where its rows take as many bytes; a page far past the image's raster, refused before it is decoded. Globals no other image uses leave the file with the original on the full rewrite (tested); globals another image still uses stay, and the README and parity note say they may hold the shapes of marks from under the area, not where they were. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-30 18:03:06 +01:00
Grzegorz MartensandClaude Opus 5.5	1ad951e25b	M11: redaction removes Group 3 on bytes whole, which the renderer may draw as other rows ... PdfCraft's bootstrap renderer draws byte-aligned Group 3 whose fill ends each end-of-line code on a byte (as TIFF writes it) as other rows than decode_ccitt reads (black box, at every /K, with /EndOfLine or not: mostly the first row, then blank). Redaction cleared pixels in the picture decode_ccitt reads, so a mark over what PdfCraft showed needn't cover what was cleared, and content shown blank stayed in the copy. Such an image (/K 0 or more, /EncodedByteAlign) is now removed whole, as every CCITT image was before this PR; Group 3 off bytes and Group 4 are still cleared. The image's parameters come from Document::fax_params, the reading of a fax image's /DecodeParms entry that decoded_image already did, now on its own (it was to come with #4404, which uses it too). The redaction README, the module doc, the redaction and core.image-filters notes and the spec parity table's status line and revision row say so, and the core.image-filters note now records the renderer's misdrawing. group_3_on_bytes_is_removed_whole_and_other_fax_images_cleared: a white 16 × 2 picture, checked to decode as such, is removed whole on bytes (no codes; codes at /K 0; at /K 1 with /EncodedByteAlign by reference) and cleared off bytes and as Group 4 on bytes. It fails without the guard; cargo-mutants catches every viable mutant of it. From reviews with the pdfcraft-review skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	aaa3d815c1	M11: say where Group 3 with no code before the first row is written back as read ... decode_ccitt reads byte-aligned Group 3 whose data has end-of-line codes but none before the first row (not as T.4 sends them) on bytes. At /K 0 with the fill ending each code on a byte, about one small picture in five is then read as other rows; at fax width almost none are, as such pages come out short and are refused. Export already said so. The optimizer, when it resamples such an image or stores it again as Group 4 from behind another filter, and redaction, when it clears pixels in one, write those rows back, and nothing said that. Black-box, PdfCraft's renderer draws such an image exactly as decode_ccitt reads it (all 42 misread of 200 seeded pictures, behind ASCII85Decode too), so the written image is what PdfCraft showed before. Refusing such data instead (a code after nearly every row but none first) would also refuse some ordinary Modified Huffman images, whose rows can meet 11 zeros by chance (0.5 to 5% of small ones in a black-box probe), so the limit is stated rather than guarded: in decode_ccitt's doc, the filters, optimize and redact READMEs, and the core.image-filters and mono-compression notes. The optimize README's and the mono-compression note's "not yet: Group 3 to Group 4 without resampling" now say a lone Group 3 image: one behind another filter is stored again as Group 4. Also from the review: docs/roadmap.md keeps #4335's status line under this PR's, rather than replacing it; the filters README's tests and the core.image-filters evidence name the Group 3 layout test; the filters crate's description names Group 3 and 4 decoding. From reviews with the pdfcraft-review skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	5f3fde33d6	M11: tests for what Monochrome Images keeps, and for the fax bounds; Retain's wording ... Mutation testing found behaviour no test held, now covered: - a Group 4 halftone above 450 ppi isn't resampled and stays byte for byte, alone by default and behind ASCIIHexDecode with Retain; - under Retain a resampled Flate scan stays Flate, though Group 4 would be smaller (ragged bands); - sixteen-bit grey of only 0x0000 and 0xFFFF keeps its bytes; - a busy Group 4 image behind ASCIIHexDecode, whose code is over three times its raster, decodes within the fax bound and is stored again as Group 4 alone. Redaction's busy-fax test is that busy again: since the bound for other images is three times the raster, the old checkerboard fitted it, and dropping the fax bound passed. The redaction README says three times, and doc_optimize's description says what Retain keeps: CCITT stays CCITT, anything else becomes Flate. From reviews with the pdfcraft-review skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	68c6d4b052	M11: redaction's decode allowance takes ASCIIHex text broken into lines; tests for the bounds ... - Redaction decodes an image no further than twice its raster plus a byte a row, meant for ASCIIHex text under Flate; hex text broken into lines is a little more, so such an image was removed whole instead of cleared (already so on main). It is three bytes for each now, as the optimizer allows. - Tests: grey stored as hex text in lines under Flate, and RGB behind a filter array (which a fax image's bound would refuse), are cleared; decoded_image_within gives a fax image no more rows than its bound holds. From reviews with the pdfcraft-review skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	bd56936795	M11: redaction reads an image mask's /Decode given by reference ... Redaction clears an image mask's covered samples to the value that paints nothing, which an inverting /Decode [1 0] flips. It read the array only when direct, so given by reference (or with its first number so) the cleared samples painted instead: the covered area came out inked. That was already so on main, and this PR sends CCITT masks through it. The array and its first number are resolved now, as export and the optimizer read them; the fax test covers all three forms. From reviews with the pdfcraft-review skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	1f801bf4c9	M11: redaction bounds a fax image's decoding; a test for numbers inside /DecodeParms given by reference ... - Redacting a CCITT image went through decoded_image, which undoes the filters before the fax decoder up to MAX_DECODED, so a Flate bomb in front of a fax image cost a gigabyte while every other image stops at twice its raster. Document::decoded_image_within takes a bound, and redaction gives a fax image's code about a byte per pixel. A fax image cut short, or behind a bomb, is removed whole; a busy one whose code is bigger than twice its raster is still cleared. - Every image is now decoded through decoded_image_within, which reads /DecodeParms through Document::with_direct_parms: given by reference, the numbers inside (/Predictor 7 0 R) included. That reading was first written here and has moved down to #4304. The predictor test now covers the dictionary, an array's entry and the number itself given by reference; read without them, the cleared bytes drift off the covered pixel and the covered sample stays in the copy. From reviews with the pdfcraft-review skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	e3b9e175b5	M11: redaction clears pixels in CCITT fax images instead of removing them whole ... Create PDF and the optimizer now store black-and-white scans as Group 4, and redaction couldn't decode CCITT, so redacting one word on such a scan removed the whole page image. A CCITT image (the last filter, after any others) now decodes through Document::decoded_image and is cleared in a Flate copy like any other image; one that can't be decoded in full is still removed whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 20:48:03 +01:00
Grzegorz MartensandClaude Opus 5.5	6a99d26c7d	M11: redaction reads /DecodeParms given by reference before clearing pixels ... Stream::decoded_within reads /DecodeParms only when it is direct, so an image whose PNG predictor was given by reference decoded with its row tags still in the data. The cleared bytes then drifted one byte further from the covered pixels each row, and the covered samples survived in the redacted copy, recoverable by applying the predictor. Parameters are now made direct first (Document::with_direct_parms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 20:48:03 +01:00
KenM76andClaude Opus 5.5	47909b04dd	Merge origin/main into fix/text-matrix-survives-q ... Upstream moved the interpreter state into a Walk struct; the saved-state stack now carries (Gs, tm, tlm) there. Re-proved: the redact test fails with upstream's interp.rs and passes with this one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9	2026-10-27 21:35:24 -04:00
Dylan M. Taylor	33e23da8a9	Refactor redact for strict Clippy checks	2026-10-27 23:12:51 +00:00
KenM76andClaude Opus 5.5	0457aa88f9	fix(redact): Q inside a text object restores the text matrices; revert the render change ... Review on #4831 pointed out the PDF Association's erratum to ISO 32000-2 (pdf-association/pdf-issues #368): inside a text object, q/Q also save and restore Tm and Tlm (§9.4.1, referenced from §8.4.2). Acrobat 26.2 agrees on a synthetic PDF 2.0 page. The renderer on main already did this, so the previous commit's hayro-interpret patch (67) was wrong: it is reverted here, with its vendor/README row and the render tests. The real disagreement was in the redactor's own interpreter, which kept Tm and Tlm across Q. Find took its boxes from the renderer's text, the redactor measured the glyphs elsewhere, and redact_apply refused on residue. Its state stack now holds (Gs, Tm, Tlm) and Q restores all three. Test: it::redact_text_after_q, end to end on %PDF-2.0 fixtures, three cases (font change inside q/Q; q 100 0 Td Q; the same followed by T*). With the redactor change reverted, each case fails on its own at redact_apply. With only Tm restored, the T* case fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9	2026-10-26 19:51:58 -04:00
aiongg	cc9b5057bf	refactor(redact): split interp.rs into interp/ ... New modules: interp/layers.rs (hidden-layer handling), interp/prune.rs (resource pruning and unlinking).	2026-10-26 22:18:50 +00:00
Dylan M. Taylor	32a9af4d08	proposal(redact): unlink unused Type 3 glyph programs and fonts on redaction (F-011)	2026-10-26 16:01:29 -04:00
Dylan M. Taylor	833cd63c8a	proposal(redact): prevent retention of confidential vector paths behind clipping (F-008) (#4577) ... Partial vector redaction previously hid covered areas with clipping (`clip_out`, `W/W*`) while retaining the complete original drawing instructions and painting operators in the content stream. This allowed recipients to recover confidential outlined text or other vector drawings from supposedly redacted documents. This change removes vector paths that intersect redaction marks completely and refuses partial clip retention, ensuring no confidential geometry is preserved behind clipping. Report counts are updated to record removed paths.	2026-10-25 16:14:06 -04:00
skylar taylor-barrick	e90f7882f1	Migrate PdfCraft to egui 0.37 (#4591) ... * Record successful PdfCraft web application check * E21: bound startup requests and validate font and license inputs * E21: separate pending control completion and gesture regression phases * M-egui037: enforce control admission and per-family font notices * M-egui037: qualify capture fixes and document migration behavior * egui-037: identify current and historical qualification results	2026-10-25 14:30:31 -04:00
Dylan M. Taylor	aa9a9105fc	fix(redact): remove AcroForm on Hidden::FormFields even when field count is zero (#4571) ... * fix(redact): remove AcroForm on Hidden::FormFields even when field count is zero * fix(redact): report AcroForm presence in scan() and format	2026-10-24 21:07:16 -04:00
aiongg	f94b34adc9	Integration: fix(redact,cos): cleared redaction marks leave nothing behind (#4226)	2026-10-24 22:48:11 +00:00
oracola91	bc4f8e2602	Redaction keeps images still drawn through a form without /Resources (#4413) ... content_draws tells whether some content draws an XObject, directly or through a form that has no /Resources of its own and so draws through its caller's. Since #4356 the recursion passed the caller's content again instead of the nested form's decoded data, so a name drawn only inside such a form was never found. Both callers then dropped the entry: unlink_doomed for page-tree resources (Apply Redactions and the content pass of Remove Hidden Information) and prune_doomed_xobjects for surviving forms. An image removed under a mark but still shown elsewhere through such a form went missing from there as well. The recursion reads the nested form's data again, as it did before #4356. Two tests cover the page-tree case and the surviving-form case. Both fail without the fix.	2026-10-24 10:29:28 -04:00
aiongg	7733b619cf	Integration: #3670's fan-out test allows for a loaded machine	2026-10-24 07:11:16 +00:00
aiongg	a9fe3d4e7f	Integration: drop the doc comment of the removed visit cap (#3670 × main)	2026-10-24 07:11:16 +00:00
aiongg	2a91f1d717	Integration: the page's decode budget owns the form count in redaction (#3670 × main) ... #3670 raised the form depth limit to 50 and added its own 50,000-visit cap. Main's per-page DecodeBudget (e2a677164) has the same 50,000-form limit, and the visit cap fired first, so Apply removed the forms instead of refusing the page as main does. The visit cap goes; the fan-out test expects the refusal, and the depth-limit sanitization test builds its chain from MAX_DEPTH.	2026-10-24 07:11:16 +00:00
aiongg	8a64b65e4f	Merge PR #4330: M8: redact labelled Italian international slash numbers ... ROADMAP.md: kept both progress-log lines, newest first by position (HEAD's #4280 line, then this one).	2026-10-24 06:23:03 +00:00