mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-09 04:22:20 +00:00
44d7fb996afe74fb7420528be91c5adad481e913
4478
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bmh129
|
44d7fb996a |
M10: name the web OCR pool's run router runs (ocr.ocr-web)
page_modules_read_the_pointer_only_through_page_pointer treats any file that mentions `.router.` as a page module, and failed on workflows/ocr_web.rs: its `self.router` is the OCR pool's ocr_route::Router, which hands each page's text to the run that sent it, not the pages' PageRouter. Adding the file to PAGE_MODULES would put the workers under the page-input checks, so the field is renamed `runs` instead. Tests: the page_pointer tests and all 808 pdfcraft-ui-egui lib tests pass, and cargo check for wasm32-unknown-unknown builds. |
||
|
bmh129
|
4bcc63e059 |
M10: the Verify screen downloads the web OCR models itself (ocr.ocr-web)
In the web build, Verify's Run OCR and its page/region re-reads never
started the model download. On a fresh page they failed at once with the
desktop's "set PDFCRAFT_MODELS" message, and worked only after Recognize
Text had downloaded the models.
Both now start the download (web_models_missing), say "Downloading the
text recognition models…" and remember what was asked; poll_models runs
it once the models land, or shows "Couldn't download the text
recognition models: {e}" when the download fails. Nothing runs if the
screen was closed meanwhile, and an engine the browser can never run
says why instead of waiting.
Tests: models_landed starts a waiting run and a waiting re-read, reports
a failed download without touching the review, and runs nothing for a
closed screen. The first three fail without the fix. Checked by hand in
Chrome: a fresh page, Verify > Run OCR downloads the models and reads
the page (47 words on a real scan); with the models missing it shows
the HTTP 404 and a second click after restoring them works.
|
||
|
bmh129
|
ce25eb2aef |
M10: web OCR docs, parity and roadmap after the port onto #977 (ocr.ocr-web)
Hosting notes for the models and workers, the platform parity text, the ocr.ocr-web parity entry (now citing the ported tests, with the known limits: the page runs the cleaning steps, the Verify screen re-reads on the page, ocrs only), and the roadmap logs. |
||
|
bmh129
|
a10906e991 |
M10: Recognize Text in the browser, read in Web Workers (ocr.ocr-web)
The web build downloads the OCR models the first time Recognize Text (or Edit on a scanned page) needs them, installs the engine, and reads pages in up to two Web Workers (apps/pdfcraft-ocr-worker) built with WebAssembly SIMD, so the page keeps responding. The page prepares each page and sends its image to an idle worker, which runs ocrs and sends the lines back. Where no worker can start, the page reads with the installed engine. Answers go back to the run that sent the page (ocr_route), so a single run and a batch can overlap. Recognize Text in Multiple Files on the web keeps main's skipped-files bucket, low-confidence notes and cancel between files. Files chosen before the models are in wait for them. A scan Edit finds while no engine can read it waits for the models instead of counting as no scan. Trunk builds the worker beside the app; `cargo xtask web-worker-link` hashes its files and names it in index.html. The web zip ships the models in models/. |
||
|
bmh129
|
0bfb50a853 |
M10: read OCR jobs a page at a time; install a downloaded engine (ocr.ocr-web)
`OcrJob::steps` splits `run` where the engine reads: `OcrSteps::next_page` renders the page, crops it to the region and preprocesses it, an engine reads `OcrShot::image`, and `OcrShot::place` maps the words back through the preprocessing and the crop. `run` is the same loop with `OcrShot::read`, so desktop results don't change. With only one engine, every strategy is that engine's reading, which is what the web build's workers return. `FileJob` splits `recognize_file` the same way (open, read, finish), keeping the skip-files probe on opening. `ocr::install` puts a recogniser loaded from downloaded bytes in the engine cache, so `recognizers`, `available`, `engine_unavailable_reason` and `engines_status` see it. |
||
|
bmh129
|
9121f65564 |
M10: OCR models load from bytes; recognised lines travel as JSON (ocr.ocr-web)
The web build has no model files: it downloads them. `Ocr::from_bytes` loads the pair only when each file's SHA-256 matches ATTRIBUTION.toml (rten's ONNX parser can spin forever on bytes that aren't a model), and `OcrsRecognizer::from_ocr` wraps the result as an engine that stays available without files. The web build's workers send what they read back as JSON: `lines_to_json` / `lines_from_json` carry each word's box and, when it has one, its confidence; the reader credits the words to the engine the worker ran, and refuses anything else (a missing or non-finite number, too many words) with an error. |
||
 Chiranjeevi U
|
adbf1dc01f |
M9: signed appearances show their real validation state (#79) (#2808)
* M9: redraw named-layer signature status layers - /N drawing one Form that holds n1 and n3: n1, n3 and n4 redrawn for the state - copies made once per original and state; originals never edited - Lucide icons; captions in Helvetica, embedded Inter, or craft-fonts outlines Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: draw signature states in the display copy, exports and print - marks follow the bytes shown; a changed verdict redraws after a metadata edit - background opens pick up the trust and captions current at commit Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: status captions in the interface language - the three captions in all 22 catalogues - Signatures panel colours from sign::status_rgb Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: signature badge in the widget's corner - a quarter of the shorter side, 48pt cap; outline only under 32pt - dropped for a drawn valid state; bottom-right where the appearance draws one - sign_list reports verdict_shown; parity and roadmap Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: keep Arabic signature captions in logical order Pass raw RTL catalogue text to the signature appearance shaper so UAX #9 is applied exactly once. This keeps the rendered word order and exported /ActualText consistent. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M2: read /ActualText over painted paths and pictures - A replacement that marks no glyph reads where it paints, cut to what is visible and along the way it is drawn. - A span the page content leaves open ends before the annotations. - OCR and scan field detection still count such a page as having no text of its own. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M8: redaction strips /ActualText around paint it clips - A path or shading written again under a clip counts as changed, so a partly redacted word drawn as outlines leaves no name in the file. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: recognise only layer forms that draw n1 and n3 - Settle on the one form holding both first, then read its content once per call. - Skip an /N that is its own layer form; copies keep the layer's /OC. - A verdict counts as drawn when any visible widget shows it. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: keep the signature badge and redraw wherever a state changes - The badge stays for every state; its mark sits bottom-right as the widget lies on the page. - An edit that changes a verdict redraws every page; captions apply before startup files open. - Print, the CLI's render and the repair log follow the display copy. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: restore the attribution usage the rebase dropped - Usage lines for the three icons, Inter and the craft-fonts input; ATTRIBUTION.md rebuilt from them. - Roadmap and localization notes for this round. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: pin signature field names on a nested form Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: bound the signature field walk - kids share their parent's name and type instead of each copying them - a widget stays listed under every field that claims it, as one shared copy - a field written in place in a shared /Kids array is walked once - caps of 20,000 fields, 100,000 entries and 4096-byte names; no states drawn past them Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: keep a document signed when the field walk stops early - list_cached adds a signed row of unknown validity for the fields the walk didn't read - Save As and actions then append, and Optimize, Sanitize and full rewrites are refused, as for any signed file Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M8: removing hidden content no longer strips the names of shadings - With no regions to clip, a shading is left alone instead of wrapped and marked clipped - clean_replacements skips scopes already marked, so deep nesting stays linear Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M2: read a picture's /ActualText over the whole picture - draw_image's transform maps pixels, not a unit square, so the text sat on one pixel in the corner; the test now draws a 2x2 picture - Painted grows in place instead of being copied through an Option Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: cap the bytes of names the signature field walk keeps - A kid's name repeats its parent's, so one long name over 20,000 kids came to 80 MB; the walk now stops at 4 MiB of names and counts the rest as unread - Cover the widget written into a shared /Kids array and the field cap among kids, which no test caught - sig_fields is private again, and the cap docs say what happens past each cap Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: skip status-layer work that changes nothing; fix tests that could not fail - show_status_layers and has_status_layers return before reading the form when nothing is signed, so an unsigned form no longer walks every field on each edit - revalidate and draw_captions leave documents with no drawn state alone, and no longer copy the signature list to update it - Scan form detection reads the page's text once; comment and badge tips share one hover rule - Tests: drop a table row that repeated another, stop taking an expectation from the code under test, stop comparing an Arc with itself, give bare asserts messages Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: test the signature field name cap at the engine boundary - A 4,096-byte name is read and blocks a full rewrite; at 4,097 bytes the list holds the unread-fields row and a full rewrite is still refused Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: test signature states beside fields the viewer draws - One display copy holds both a field's drawn appearance and the signature's state. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: draw Thai and Devanagari status captions in their own script - script_of names Thai and Devanagari, so a craft-fonts face meant for either draws the caption as outlines; without one it stays English. - Localization, parity and the craft-fonts usage line list the new scripts. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M2: paint left out by an exclusion mixes its /ActualText replacement - A replacement placed by outlines or pictures, some of them excluded, fails filtered extraction as one with excluded glyphs does. - Glyphs still place a replacement that has them. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: a long-named field that can't hold a signature leaves the walk whole - A leaf of another type passed over for its name is no signature field and has none below it, so an unsigned form stays unsigned. - A signature field, or any field with kids, still counts as unread. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: status captions use document faces only - Each script takes the face PdfCraft writes it into documents with, so Chinese stays English although the pinned craft-fonts' interface-only Chinese face is meant for Hani. - The craft-fonts usage line names Devanagari: the pinned revision has a face for it, and Hindi captions are drawn in it. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: refuse a new signature field the field walk can't read back - Signing adds a field only when the walk reads the whole form with it: past a cap the signature would be listed as unread fields, and its name could repeat an unread field's. - A field the walk already reads is signed as before. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: refuse a new signature field with a blank name - The walk lists no field without a name, so signing now also requires the walk to list the new field: one named only with spaces or NULs would be a signature nothing shows. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: an unnamed signed field or an unreadable form counts as unread - A signed field with no usable /T, or an /AcroForm or /Fields of the wrong type, cuts the field walk short: the document keeps its unread-fields row and stays signed. - Signing refuses a blank new field name before it adds the field, so the refusal still says why. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: no display copy when no signature state is drawn - display_bytes_for makes a copy only when it added an appearance or drew a state, as the app does. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: the verdict mark stays on the page - A widget past the crop box takes its mark in the part on the page, or none when too little of it is there. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: typed values stay drawn while a signature redraws every page - A form edit that changes a signature's state puts its values back in unbaked after the pages are invalidated. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: the action, Save As and shading tests check their effects - The signed action and Save As tests reopen the output for the title and the comment; the shading test checks its clip. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: values typed earlier stay drawn while a signature redraws every page - A form edit that changes a signature's state keeps every value still waiting on its page, not only its own. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: comment and content edits keep typed values drawn too - An edit to one page's comments or content that changes a signature's state moves no page, so the values still waiting on their pages stay pending, as after a form edit. - The pending-value test runs a field deletion and a comment after a fill, under a certification that allows the fill; its messages say what it checks. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: the language-change test skips without catalogs - Like the other language tests, it returns early when the build has no craft-i18n catalogs. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> * M9: signed fields behind kids without a name count - A kid with no /T but kids of its own is walked as a field node, so a signed field under it is listed by its full name. - An unnamed kid claiming a signature its field doesn't have cuts the walk short, so the document stays signed and keeps its unread-fields row. Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> --------- Signed-off-by: Chiranjeevi U <244287281+chiranjeevi-max@users.noreply.github.com> |
||
 Luis
|
f841666d5f |
Release: refresh contributor credits for each release (#6487) (#6488)
About > Contributors still showed an old snapshot, because nothing regenerated contributors/contributors.json before a release. Opt in to the shared refresh from craft-common-actions: a read-only credits job regenerates the snapshot for the release commit, and the shared release workflow installs it before every source build. The snapshot is not committed back. |
||
  eliets2andeliets2
|
5bb8bbf20b |
Add dry-run preview to doc_batch without writing outputs (#6486)
* Add dry-run preview to doc_batch without writing outputs Users cannot preview a batch before it writes files, so they cannot check the planned steps and output names for a folder first. Add a dry_run option to doc_batch that reuses discovery, returns each file's ordered steps and planned output name, and skips processing and publishing. Test: doc_batch_dry_run_lists_outputs_without_writing failed before the change and passes after. * Address review for pull request 6486 --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> |
||
 FredApps
|
5644d56c64 |
Merge pull request #6243 from alehel/refactor/page-press-edit-text
refactor(ui): tell Edit text, the text layer and links whether a press is theirs (router PR 4b) |
||
|
FredApps
|
9dbede8709 |
Merge pull request #5816 from KenM76/feat/unreadable-object-report
feat(cos): list the objects that could not be read in doc_info |
||
 FredApps
|
33e3945c04 | Merge main and preserve unreadable diagnostics with cached failures | ||
|
FredApps
|
784ce9212b |
Merge pull request #6497 from helenkwok/web/page-api
pdfcraft-web: window.pdfcraft, the page command API from craft-web-api, opt-in with ?control |
||
|
FredApps
|
a94e90da65 |
Merge pull request #6471 from ZIDAN44/fix/6440-add-text-cmyk-gray
Add text: gray or CMYK colour, painted with g or k (#6440) |
||
|
FredApps
|
09a63d6dd6 |
Merge pull request #6197 from KenM76/feat/rich-copy
M3: copy keeps formatting as HTML (Copy with formatting) |
||
|
FredApps
|
21747a4e58 |
Merge pull request #5802 from KenM76/perf/failed-object-cache
perf(cos): an object that can't be read is searched for once, not on every read |
||
 Helen Kwok
|
a9ccca5b71 | pdfcraft-web: say ui.open answers with an error on the web | ||
|
Helen Kwok
|
6ec23789c1 | pdfcraft-web: say a timed-out call is not guaranteed to have been skipped | ||
 Zinadin Zidan
|
9db177c163 |
Add text: the ink tests reach the non-number and content_update paths (review)
The hand-edited ink test's "CMYK entry not a number" case had five entries, so the length check refused it before any entry was read; it now has four. content_update gets its own refusals: two inks at once, an ink out of range beside another style change, and gray or cmyk on an image, with the item list unchanged after each. |
||
|
Helen Kwok
|
a9e98e0db6 |
pdfcraft-web: window.pdfcraft, the page command API from craft-web-api, opt-in with ?control
?control attaches the UI control channel (the one --control serves) and installs window.pdfcraft: request, execute (ui.command), commands (ui.commands), docInspect (ui.state), uiInspect (ui.inspect) and screenshot. The channel keeps the accessibility tree of every frame, so it stays off unless the page asks. Opening, saving, the file name and the unsaved flag stay with the host messages. |
||
|
Zinadin Zidan
|
2153da0644 |
Add text: a stored ink is read without copying a long array (review)
Reading an item's /CMYK resolved the value, which copied a direct array, then collected every entry before checking there were four. A file with a huge numeric /CMYK paid for that on each content listing and item lookup. Now a direct value is read where it is, a reference is followed without a copy, and the array's length is checked before any entry is read. /Gray is read the same way. A long /CMYK, direct or by reference, and a /Gray that is a long array still read as no ink. |
||
|
Zinadin Zidan
|
c44896df95 |
Add text: an ink is read only from numbers and kept as painted (review)
A stored /CMYK with a non-numeric entry was read from its numbers alone, so [0 /Bad 0 0 1] became 100 K; a /Gray that isn't a number with a /CMYK beside it fell through to the CMYK. Both now give no ink, and the text paints in its /Color. g and k are written to the thousandth, as rg is, so an ink set or read with finer digits painted other than what the item, its stored /Gray or /CMYK and content_list held. The ink is now kept as painted, with its look, when set and when read back. RGB text is unchanged. |
||
|
Zinadin Zidan
|
7e8551e4bc |
Add text: gray or CMYK colour, painted with g or k (#6440)
page_add_text and content_update took only RGB, so text added to a CMYK print file painted 0 0 0 rg, rich black on press. They now also take gray (0 to 1) or cmyk ([c, m, y, k]), painted with g or k and kept in /PCAdded as /Gray or /CMYK beside /Color, which holds the matching look. An ink applies only while the colour still shows its look: picking another colour, also in the Format panel, drops it for good. RGB text is written byte for byte as before, and a stored ink that doesn't match its colour is ignored. |
||
 alehel
|
513b0fae60 |
Merge remote-tracking branch 'origin/main' into refactor/page-press-edit-text
# Conflicts: # CLAUDE.md |
||
|
FredApps
|
f42fda7342 |
Merge pull request #6416 from storytold/house-rules
Copy the shared house rules into AGENTS.md |
||
|
dc5cb122db |
trust certificate key type when CNG algorithm group is unreadable (#6391)
* trust certificate key type when CNG algorithm group is unreadable When the algorithm group can't be read from the key store due to a provider bug, the certificate should not be rejected: its SubjectPublicKeyInfo already states the key type, and a real mismatch will surface at signing time instead of at enumeration. Change matches_key to fall back to Ok(()) when key.algorithm_group() returns an error, trusting the certificate's key type; extract the group comparison into key_group_matches(group: Option<&AlgorithmGroup>, want: StoreKey) so the fallback and mismatch cases can be unit tested without a live CNG key. Test: key_group_matches_trusts_when_group_unreadable failed before the change and passes after. * Sign: back to LF line endings, shorter test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1d71d6039f |
Add PCX image support to Create PDF conversion (#6396)
* Add PCX image support to Create PDF conversion PCX is listed as an accepted Create source elsewhere but PdfCraft rejects it, so users cannot convert PCX images to PDF. Detect PCX by its header in is_image and decode it in embed, adding pcx to CONVERTIBLE so existing from_images and source_kind paths handle it. Test: pcx_images_become_pages failed before the change and passes after. * Create: PCX scanlines hold every plane's row, 8-bit indexed images, older versions, standard test files Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 KenM76andClaude Opus 5.5
|
cf3ee49fbc |
Merge origin/main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9 |
||
|
FredApps
|
465783fc47 | docs: preserve local guidance in house-rules rollout | ||
|
KenM76andClaude Opus 5.5
|
1107834209 |
Merge origin/main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9 |
||
|
KenM76andClaude Opus 5.5
|
80ab49d09a |
Merge origin/main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9 |
||
|
FredApps
|
81b0833b24 |
Copy the shared house rules into AGENTS.md.
The blocks come from craft-contrib at dde09ea31e2e457f5b4653a6e76f7cdf55944bdf. CLAUDE.md imports AGENTS.md. The house-rules workflow checks the copy. |
||
|
|
a0811650de |
Add Pages range field to Compare Files dialog (#6336)
* Add Pages range field to Compare Files dialog Users cannot limit a comparison to selected pages, so the dialog needs a validated Pages field that blocks Compare on invalid ranges. Add a Pages text field under New file, keep its text in pages_text, parse comma-separated numbers and ranges to 0-based pages, show an error and disable Compare for zero, reversed, or out-of-range pages, and leave the comparison itself unchanged. Test: compare_dialog_pages_field_rejects_out_of_range failed before the change and passes after. * Compare Files: the Pages field limits the comparison, and says what is wrong with a bad range Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Compare: a change across a page break counts on both pages; a bad range is refused, not read as all pages Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
af76b5ca89 |
Delete key removes selected content object in edit mode (#6384)
* Delete key removes selected content object in edit mode The reporter says Delete does nothing for a selected textbox or image in edit mode; the only Delete/Backspace handlers in the UI are for organize pages and comments, so a selected content object has no delete path. Add Delete/Backspace handling to the object selection input in object_ui.rs that deletes the currently selected object(s), and cover it with a unit test that selects an object and presses Delete. Test: delete_key_removes_selected_object failed before the change and passes after. * Tests: the Delete key test takes PageToolContext and imports PagePointer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
  Hamza Naveedandskylar taylor-barrick
|
97cef90e01 |
Redaction: a pattern nothing paints with any more leaves the page (#3680)
* Redaction: a pattern nothing paints with any more leaves the page
A tiling pattern's cell is a content stream of its own and can draw
anything, text included. Redacting a shape filled with such a pattern
removed the fill from the page's content -- the page looked right, a
black box -- but left the pattern in /Resources /Pattern, and its cell,
with the text that was under the mark, was written into the saved file
unchanged.
The pass does not enter pattern cells, so it cannot redact inside one.
What it can do is notice that nothing paints with the pattern any more.
The graphics state now carries the fill and stroke pattern a `scn` or
`SCN` selected (and `cs`, `g`, `rg`, `k` and their stroke forms clear
it), every painting operator that survives the rewrite records the
pattern it drew with, and a pattern the page selected but no surviving
paint used leaves the page's resources.
Measured on the report's file:
before applied: 1, (SECRET) in the saved bytes
after applied: 1, no /Pattern on the page, (SECRET) gone
A pattern some surviving paint still uses is kept, and so is its cell:
with two boxes filled from one pattern and only the first marked,
patterns_removed is 0 and the resources keep the entry -- the second box
still shows the word, so it is not a leak.
Report gains patterns_removed.
Fixes #3647.
* Keep a pattern every operator that can paint with it still uses
Two defects CodeRabbit found in the first version of this change, both
of which could remove a pattern the page still draws with.
Pattern use was recorded for path painting only. Text shows with the
fill pattern too -- and with the stroke pattern in the stroking render
modes -- as do image masks and anything a form draws, and a form's
content is interpreted with a graphics state of its own, so a pattern it
inherits is never seen inside it. Content like
/Pattern cs /P0 scn 20 140 100 30 re f BT (KEEP) Tj ET
with the box redacted therefore lost the fill of text still on the page.
Every operator that can paint with the current pattern now records it:
text (except mode 3, which paints nothing, and mode 7, which only
clips), inline images, and image or form XObjects. Only path painting
knows exactly what it drew and whether the paint survived, so the others
are recorded as they are met. That can keep a pattern whose paint was in
fact all redacted, which costs an unused resource entry, while dropping
one something still draws with takes paint off the page.
/Resources /Pattern may be an indirect object, like any value. Reading
the reference itself as a dictionary found none and pruned nothing, so
on such a file the cell -- with the text that was under the mark --
stayed in the saved bytes, which is the leak this change is for. It is
resolved now, and what is written back is a dictionary of this page's
own, so pruning never reaches through a /Pattern dictionary another page
shares.
The controls in the test: without the first fix the surviving text's
pattern is removed (left: 1, right: 0), and without the second the
indirect file prunes nothing (left: 0, right: 1).
* M8: refuse retained unsafe tiling patterns
* M8: guard shared appearances and narrow pattern scans
* M8: update TLS dependencies for new advisories
* Redaction: the comment on prune_patterns said more than the code does
`prune_patterns` drops the `/Pattern` resource entry; the `scn` that selected
it stays in the content, so the page names a pattern its resources no longer
hold. The comment claimed the selector went too (noted in review).
ISO 32000-1 8.6.6.2 makes that operand a key of the `/Pattern` subdictionary,
so the open question is whether a reader stops reading the stream at the stale
name. `a_page_whose_pruned_pattern_is_still_selected_renders` answers it with
the renderer rather than with prose: the redacted page is saved, reopened and
rasterized, and
* it renders with no error and the same 300x300 page box,
* the text written *after* the stale selector is still painted -- which is
exactly what a reader would drop if it gave up at the bad operand,
* the redaction overlay is black over the box the pattern filled.
Control: with the drawing after the selector removed, the pixel assertion fails
("the text after the stale selector is still painted"), so it is reading that
text and not a stray pixel.
The selector is left in place rather than cut out: removing operators is a
rewrite of the content this pass does not otherwise do, and nothing surviving
paints with that colour. `pdfcraft-render` is a dev dependency of this crate
only -- layer 4 on layer 3, which the layering allows.
* WIP: pattern guard depth bound sized from the interpreter's
* Pattern tests: keep the guard's cases reaching the guard now a hit path is removed whole
* Pattern automation tests: keep the pattern in use where they test the guard, since a hit path is now removed whole
* Redaction: a form without resources keeps the page pattern it still paints
A form with no /Resources of its own paints with its caller's patterns, but
its surviving pattern paint was recorded only in the form's own output and
never reached the page. With the page's own fill under the mark removed
whole, prune_patterns then dropped /P0 from the page while the form outside
the mark still painted with it, so that fill lost its paint in the saved file.
Pass the painted names up from a form that inherits its caller's resources.
Only the painted side goes up, so this can keep a pattern, never prune one.
* Redaction: a pruned pattern does not stay named by the page tree
prune_patterns writes only the page's own /Resources, and the guard walked
only pages' effective resources. A /Pages node that still named the pruned
pattern kept its cell reachable from Root, so a full save wrote the cell,
text included, while apply returned Ok. Three shapes did it: a page with no
/Parent inheriting from the root, a /Parent that disagrees with the /Kids
tree, and a root /Resources shadowed by a nearer holder that was retired.
Patterns pruned from a page are now unlinked from page-tree holders no page
inherits from, as unlink_doomed does for XObjects, when every /Parent agrees
with the tree. Whenever a pattern was pruned (or a selected page had any),
the guard also inspects the /Resources of every /Pages node Root reaches
down /Kids and up each page's /Parent chain, so whatever stays reachable is
proven safe or the redaction is refused, leaving the document unchanged.
* Redaction: a Type 3 glyph without resources keeps the page pattern it paints
A Type 3 font with no /Resources looks the names in its glyph descriptions
up in the resources of the page or form showing it, so a glyph outside the
mark can paint with the page's pattern. Text recorded only the fill and
stroke pattern of the graphics state, so with the page's own fill under the
mark removed whole, prune_patterns dropped the pattern and the glyph lost its
paint. Showing such a font now keeps every pattern of the scope, as a form
without /Resources keeps what it paints.
* Redaction: say what text-recorded pattern use keeps
Text, inline images and XObjects record their pattern before the hit test,
so a pattern painted only by redacted text keeps its cell. The guard refuses
a cell with text or images; a vector-only cell stays in the file. The comment
said this cost only an unused resource entry. Pin the behaviour in a test.
* Redaction: the page tree is followed up every /Parent link
A full save writes every node a /Parent link reaches, but the guard followed
/Parent only from pages and stopped at the first node it had already seen,
and unlinking never followed it. A /Pages node reached only through the
/Parent of a /Kids node -- a root /Pages with a /Parent, or an intermediate
node whose /Parent leads out of the tree -- kept naming the pruned pattern,
and its cell, text included, stayed in the saved file while apply returned
Ok.
The guard now follows /Parent from every node it visits, page or /Pages, with
the same seen-set against cycles. Unlinking treats a root with a /Parent as an
inconsistent tree, like a disagreeing link, and leaves the decision to the
guard, which inspects every node those links reach and refuses an unsafe
cell.
* Redaction: a long page tree does not exhaust the pattern guard's budget
The page-tree walk counted every node, pages included, against the guard's
16,384-object budget, so redacting a pattern page of a 17,000-page document
was refused with a limit error where it had applied before. Only the holder
nodes the guard inspects count against that budget now; the walk itself is
bounded at a million nodes, as unlink_doomed bounds the same tree.
* Redaction: a Type 3 font set by an ExtGState or shown in a form keeps the page patterns
The resourceless Type 3 check covered only a font set by Tf in the scope that
paints. A Type 3 font without /Resources set through an ExtGState /Font entry
was not flagged, and a glyph shown inside a form with /Resources of its own
recorded its patterns in the form's scope, which the page never saw. In both
cases the page's own fill under the mark was removed whole, prune_patterns
dropped the pattern, and the glyph outside the mark lost its paint.
An ExtGState font is now checked the same way, and a glyph shown in any
nested form marks the page as keeping every pattern it selected. This can
only keep patterns; whatever is kept is checked by the guard.
* Redaction: a pruned pattern a full save would still write is proven safe or refused
The walks that check retained pattern cells follow the references they know
-- resource categories, the page tree, appearances -- but a full save writes
whatever the trailer reaches by any reference. A direct /Parent dictionary, a
/Kids array on a page, or a stray resource entry (/Shading naming the pattern
stream) kept a pruned cell, text included, in the saved file while apply
returned Ok.
pdfcraft_cos gains reachable_objects, which runs the same breadth-first walk
write_full uses (factored out as full_save_order, so the two cannot drift).
Before apply succeeds, every pruned pattern that walk still reaches is
inspected like a retained cell: vector-only passes, anything else refuses
with the existing pattern error. It runs only when a pattern was pruned and
costs one walk of the reachable objects.
* Redaction: a direct /Parent dictionary is part of the page tree too
A /Parent given as a dictionary rather than a reference is written inline by
a full save, but the page-tree walk followed /Parent only as a reference, and
unlinking took such a tree for consistent. A root /Pages whose direct /Parent
held /Resources naming the pruned pattern kept its cell in the saved file.
The guard now walks a direct /Parent like any node (its /Resources, /Parent
and /Kids), and unlinking counts any /Parent that is not the reference /Kids
came from as an inconsistent tree, leaving the decision to the guard.
* Redaction: page-tree walks queue each node once and borrow /Kids
On a hostile tree whose /Pages nodes share one long /Kids array, the guard's
page-tree walk and pattern unlinking pushed every /Kids entry of every node
before checking whether it had been seen, and copied the array on each
visit. Both now queue only nodes not seen yet and read /Kids in place;
unlinking also stops at the first node reached twice, since its /Parent can
agree with at most one of the nodes above it, and leaves that tree to the
guard.
* Redaction: a Type 3 font with /Resources null is resourceless too
/Resources null is the same as no /Resources, but the check counted any
entry, so such a font's glyph outside the mark lost the page pattern it
painted with. A Type 3 glyph shown by a kept annotation appearance is a known
limit, pinned as an ignored test: appearances are not interpreted for pattern
use, and under ISO 32000 the glyph resolves in the appearance's own resources.
* Redaction: a pruned pattern is recorded along its whole reference chain
A pattern name can lead to a reference-valued object whose value is the cell
(/P0 7 0 R, object 7 being 6 0 R). Pruning recorded only object 7, so the
reachability backstop checked an object nothing reached while a stray
reference kept the cell, object 6, in the saved file. Every object along the
chain is recorded now, bounded like the other walks.
* Pattern tests: pin the file identifier where two saves are compared
main now gives a redacted rewrite a new identifier from fresh entropy, so an
incremental and a full save after Apply differ in /ID alone. The test that
checks they are the same save passes fixed entropy to both.
* Redaction: replies and pop-ups removed on other pages feed the shared-appearance check
Apply checks every page for retained unsafe pattern cells when a removed
annotation's appearance is shared with one that stays on an unselected page.
Only annotations removed under a mark fed that check. Replies and pop-ups
that go with them in step 2b (removed_everywhere), often listed on another
page, did not, so a text-bearing cell in an appearance a kept annotation
shares with them escaped the all-pages check and Apply returned Ok.
Their appearance references now go into removed_appearances too, with the
same limit. Raised by FredApps' review of the rebase.
---------
Co-authored-by: skylar taylor-barrick <hi@skylar.cc>
|
||
|
|
d7ce875dd6 |
redact: independent redaction proof machinery (snapshot, sweep, budget) (#5557)
* redact: add the independent redaction proof machinery * redact: harden the proof's paint check and count removed codes The manifest counts character codes, not raw bytes: a composite font's codes are several bytes, so the byte length overstated what was cut. Annotation appearances are checked whatever their storage (a direct dictionary has no reference of its own to dedupe on), and a stream shared by several annotations is checked at each annotation's own placement. The graphics-state stack and the clip holes of one walk are bounded; deeper or more is unverifiable, never unbounded memory. Even-odd clips cut holes only where the rule can be told cheaply - all subpaths axis-aligned, disjoint, few - and anything else keeps its paint reported, failing closed. The /W widths of a descendant font cost bounded work, like a CMap. * redact: settle the survivors a page sweep counted before its work ran out A page that the search budget stops on left the counts made on earlier pages unsettled: the survivors were seen but never reported, and the regions that owned them were never marked. The sweep now stops and still settles, so a located survivor is reported where it sits (the page reads unswept either way, and the proof still fails). * redact: prove the redacted output of every supported encryption The authenticated-reopen regression now covers all four crypt algorithms the writer supports, not only AES-128: the caller's credentials prove the saved output, and empty or wrong credentials still fail closed. * redact: answer the review round on the proof A glyph lying under several regions is now accounted for by every one of them, so overlapping marks each carry their own share in a shared proof instead of the first one absorbing the rest and the others reading as if no text was there. Each finding is recorded once per region as it is found, so a page that paints over a region a thousand times neither grows the report nor the memory with repeats. The byte search walks the haystack once (the pattern's prefix function is built once), so a needle of one letter over a haystack full of it can no longer spend comparisons the search budget never charged for. A numeric XML character reference may carry any number of leading zeros; the padded form now decodes like the short one, so a survivor spelled that way in the XMP is still found. The /W widths of a descendant font are paid for from the shared work budget: a form invoked again rebuilds its fonts, and a per-font counter alone let the same array be paid for many times over. * redact: sweep the proof's survivors out of every place a reader could look Plant the removed text where the review listed it and expect the sweep to report it: a compressed object stream, a page thumbnail, a form field value, an XFA packet, and an object nothing refers to. A file whose page content is bigger than the proof may decode for one stream is now shown to come out unverified on its surfaces and its region, never clean. * redact: spare the names a redacted word legitimately sits in, and charge failed decodes A redacted word that is also a structural name (a /Subtype, a /Filter, a /BaseFont) is structure, not a leftover copy: the name surface is now swept against the matches the input itself had, and only a surplus of matches is a survivor. The input's names are counted when the snapshot is taken, for the region's needles and for strings added later alike. A decode that fails after doing part of its work paid nothing: the proof's cumulative decode allowance now charges every attempt, failures included, so the same bomb cannot be decoded (and fail) again and again for free. * redact: count the input's names once each, and say so when the count runs out The name allowance counted one entry per occurrence, so a document whose pages repeat the same structural name paid for the same scan again and again; names are now kept once, with their count. A name search that runs out of work read as no matches, which would have swept names with none allowed: that is reported on the text surface now, and the proof fails rather than pass on an uncounted allowance. * Bound redaction proof searches across many needles Assisted-by: OpenAI Codex * Redaction proof: drop find_all, which nothing calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cargo.lock: put aho-corasick in its sorted place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Redaction proof: review notes A word split across a TJ array is looked for whole. Overlapping marks count a removed copy once. A spent decode budget is a problem, not a raw-only stream. Short needles stay out of the attachment text search, and an indirect /Decode counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 Dylan M. Taylor
|
5aaa8186fd |
Merge pull request #6375 from dylanmtaylor/land-sign-remaining
sign: land the still-valid parts of #6358, #5187 and #5173 |
||
 LetoCoder26
|
fbe079c953 |
fix(sign): compare the catalog's /Perms by content under P=1
Rebased #5173 onto
|
||
|
LetoCoder26
|
560c68db89 |
fix(sign): an existing object is judged by the role it had when signed
Rebased #5187 onto
|
||
 Martin Ball
|
ddeca878df |
sign: judge the signed page tree, catalog and form by their signed roles
Rebased onto 102cae3b/9afa1266 for landing (#6358). The signed page tree's nodes, a /Kids array of its own, the catalog by object number and an indirect AcroForm keep the roles they had when signed; a field dropped from the form is no permitted change at any level; a signed widget's /Rect or /F is page content. Co-Authored-By: Martin Ball <martin.ball@ucscaregroup.co.uk> |
||
|
Dylan M. Taylor
|
9afa126653 |
Merge pull request #6364 from benandez/fix/sign-p1-only-invisible-timestamps
sign: under no-changes certification, only invisible document timestamps may follow |
||
|
Dylan M. Taylor
|
102cae3bef |
Merge pull request #6361 from benandez/fix/sign-classify-by-signed-role
sign: classify changes by signed object role |
||
|
Martin BallandClaude Opus 4.8
|
e1a3c033aa |
sign: under no-changes certification, only invisible document timestamps may follow
DocMDP P=1 allowed any change classified as a signature. A signature added after signing now breaks the certification unless it is a document timestamp drawing nothing (no positive-area /Rect, or Hidden/NoView). The added objects are judged directly, so a widget with no /T or one only in a page's /Annots is covered. The document security store and a genuine timestamp still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Martin BallandClaude Opus 5.5
|
de344a4bb1 |
sign: compare annotation lists by what resolves, within a bound
An annotation list is compared by the entries that resolve in each revision, and an unchanged list is no change. Expanding annotation lists is bounded per classification. A damaged slot rewritten with fewer accepted generations is the same slot when nothing signed refers to a generation it gave up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Martin BallandClaude Opus 5.5
|
ce3857c2e5 |
sign: classify changes by signed object role
Judge each object changed after signing by the role it held in the signed revision, keyed by object number, and compare a slot's accepted generations across revisions. Adds `Document::accepted_generations` to cos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9f04acccef |
M10: OCR engines, confidence, preprocessing and a review workflow (#977)
* M10.20: plan Unicode text for OCR output in Type3 faces (fonts, edit) OCR output used to flatten every character beyond WinAnsi to a silent `?`. This adds the two pieces that let the OCR layers write the recognized words as they are; the OCR code that uses them follows in M10.21. pdfcraft-fonts - `TextPlan::for_words` maps each word to the standard Helvetica when WinAnsi covers it, else to Type3 fonts drawn from the craft-fonts faces (the Japanese document face first, then the rest of the build input). Characters no face has a glyph for are counted (`missing`), never dropped silently. At most 240 characters per font and 2400 per plan, so one hostile word cannot balloon. - Which face has which character is asked once per (face, character) for a plan, not once per word. - `face_has`, `craft_glyph`, `win_ansi_char`, `fits_win_ansi` are public for it. pdfcraft-edit - `add_type3_fonts` / `stamp_with_fonts` register a plan's fonts on a page under a free `PCOcr`, `PCOcr1`, ... name, build each font once and share the object with later pages. A shared face is registered under a free name on a later page that already has a font of the name it was given first, instead of replacing that page's own font. - The fonts carry real advances and a ToUnicode CMap, so extraction reads the text back. pdfcraft-redact - A test shows a hand-built Type3 text layer (the shape the OCR output writes) is redacted like any other text and the verifier passes. Tests: pdfcraft-fonts 62 passed, pdfcraft-edit 129 passed, pdfcraft-redact 33 passed; clippy -D warnings clean on the three crates. The name-reuse and the redaction tests build their fonts by hand, so they run without the craft-fonts build input; the tests that plan real faces skip without it and were not run in this commit's own check. * M12: give the hostile-XFA open test more room on a loaded machine The test opens a form whose calculate script sets a value 50 000 times, against the one-second budget of an unasked script and a five-second wall-clock bound on the open. Both sit close to what a machine running several suites at once can take, so the test can fail without anything being wrong. It now runs 20 000 iterations (the total is 19999) and allows 30 seconds, which still fails for a script that hangs or is not capped. Test-only; unrelated to the OCR work in this series, so it is a commit of its own. Tests: both hostile_xfa tests in pdfcraft-engine pass. * M10.21: pluggable OCR engines, preprocessing, review sessions and the OCR tools (ocr, engine, create, automation) Scan & OCR gets a second engine, optional raster clean-up, a review step before anything is applied, editable output and a batch mode, headlessly first: the interface for them follows in M10.22. pdfcraft-ocr - A `Recognizer` trait with two engines: the built-in ocrs (moved out of lib.rs) and the installed tesseract program, run as a fenced external process: fixed argv, allow-listed language codes, a private temp directory (mode 0700 on Unix), 120 s timeout, 64 MiB output cap, 50 000-word reading cap. A location from PDFCRAFT_TESSERACT is validated (absolute, existing, executable) and an invalid one fails closed instead of falling back to the PATH. `--list-langs` and `--version` are read from stdout or, for older versions, stderr. - Strategies: PrimaryOnly, ConfidenceWeighted (the second engine joins when the first reading's mean is under 70 or unknown) and a ROVER merge (higher confidence wins per word; against an ocrs word of unknown confidence a tesseract word wins only at 70 or more; capped at 100 M comparisons; a failing partner degrades to the primary with a note). - Preprocessing: opt-in auto-rotate, deskew, denoise and binarize on the RGBA raster, each with an exact inverse so word boxes map back to the scan. The 300 dpi normalization the steps are calibrated for runs only when a step is on: with every step off the render is read as rendered, as before. (Enlarging a 72 dpi render 4x on its own made the models read "due: €250" as "ue: ?250".) `OcrImage` is validated by construction (side and 64 MP area caps, checked before any buffer is allocated, including by the resize). - Confidence bands (High >= 90, Medium 70-89, Low < 70) drive review. pdfcraft-engine, pdfcraft-create - `OcrSettings` gains engine, strategy, the four steps, output mode, region, the skip flags and the low-confidence threshold; Automatic reads confidence weighted unless a strategy is chosen. - A recognition is a session: the document guard refuses a stale result with its reason, pages can be applied one at a time (the guard is taken again after each applied page), and applying is one undoable step. A per-language dictionary (a plain-text file: words added from any document stay on disk after it closes). - Output is searchable (an invisible layer) or a NEW editable document of visible text at the recognized boxes (`create::from_contents`); words beyond WinAnsi use the Type3 faces of M10.20 in both, with the shared plan across pages. - Engine discovery never blocks the interface: results are cached, a stale answer is shown while a background probe retries, and the models search and `--version` are cached. Batch: pages or files that already have text are skipped unless forced, and inputs sharing a file name are found up front. pdfcraft-automation - ocr_recognize (engine, strategy, steps, region, output mode), ocr_words (with a retained review handle), ocr_review_apply, ocr_dictionary, ocr_status and ocr_recognize_files. The editable `out` must be a new file inside --root and is refused for a searchable run (it used to be ignored), two batch inputs of one file name are refused before anything is read, and the batch folder is checked for existing outputs. pdfcraft-ui-egui - Only what the new API needs: the Recognize Text dialog reads with `recognizers(&settings)`. `ocr.correct` is registered with its label in every catalog; the screen itself is M10.22. Tests: pdfcraft-ocr 78 passed; pdfcraft-create passed; pdfcraft-engine and pdfcraft-automation 450 passed and 1 failed (the automation lib 17, automation.rs 157, hostile_args 2 among them); pdfcraft-ui-egui lib + ocr + control tests 391 passed; clippy -D warnings clean on the five crates. The one failure is in a test this commit does not change: recognize_text_makes_a_scanned_page_searchable fails where the models are installed (they read "quick" as "auick" at that render size); the roadmap on main already records it. The real tesseract binary ran in tesseract::tests::recognize_runs_the_installed_binary (tesseract 5.4.0, PATH). * M10.22: OCR Verify screen, the Recognize Text options and the interface catalogs (ui-egui) Scan & OCR ▸ Correct recognized text: a full-window screen where a recognition is reviewed before anything is applied, plus the dialog and batch options for the engines of M10.21. OCR Verify screen - Four panes: the page list, the source page with confidence-coloured word boxes (click to select, drag to draw a region), the recognized text, and a zoom plus word inspector. Corrections commit on Enter (an empty commit removes the word), Remove/Restore, "+ Dictionary" (a per-language file in the preferences folder, so a word added from a document stays on disk after it closes), per-word and per-page Verified, F8 / Shift+F8 walk the low-confidence words, Ctrl+T marks the page reviewed, Ctrl+Tab moves the pane focus, and review edits undo within the panel. - Re-OCR of the whole page or a drawn region. A region re-read moves the review marks of the words it keeps to their new positions and selects the first new word; a whole-page re-read clears that page's marks (and the undo brings them back). The region outline is scaled onto the pane, not mapped through user space again, so it is right on rotated and cropped pages. - Accept / Reject per page or whole. Pages accepted one at a time each apply once (the document guard is taken again after each applied page; an edit made elsewhere in between is still refused with its reason), and Accept in searchable mode is one undoable step. Editable mode writes a NEW `<name>_ocr.pdf` through a save dialog and never the source. A real Cancel keeps the pages already read. - The engine and strategy menus disable a choice that cannot run and say why; the check never waits on a process (a background probe fills a cache), and choosing Automatic moves "Primary only" to Confidence weighted. Recognize Text dialog and batch - Engine, strategy, the four clean-up steps and the skip flags are remembered with the other choices (a saved Automatic without a strategy reads confidence weighted). The batch refuses two inputs of one file name and existing outputs before reading anything, notes low-confidence pages at the settings' threshold, and its toast has a Cancel (the file in flight finishes). Catalogs and attribution - The 90 new interface strings are in all 21 catalogs (the Lithuanian ones were written without native review, and the Bulgarian ones avoid U+045D, which the bundled monospace face lacks). ATTRIBUTION.toml carries the new hashes, a note on the Lithuanian entries and the README screenshot; ATTRIBUTION.md is regenerated. - README: a screenshot of the Verify screen, taken with `cargo xtask screenshots` from a synthetic scan (`shot --ocr-verify on`) built from allowed assets; the page reads it with the real engines. Tests: pdfcraft-ui-egui lib 373 passed (1 ignored: the Windows hinting measurement probe), tests ocr 6, control 46, fonts 9; clippy -D warnings clean on the engine, ui-egui, pdfcraft and xtask packages; `cargo xtask assets` ok. The review state machine has unit tests for every fix above; each was seen failing with its fix reverted. * M10.23: parity entries, roadmap and localization notes for the OCR workflow Records what M10.20-M10.22 ship, and what they do not. parity/acrobat-features.toml - Status changes against main: ocr.rotation-detection and ocr.deskew planned -> shipped (the opt-in steps of M10.21, with their tests); ocr.correct-recognized- text and ocr.ocr-engines planned -> partial. Other Area I entries get their evidence and notes brought up to date. - ocr.correct-recognized-text: the evidence includes the one-page-at-a-time accept and the region re-read tests; the notes say what the headless tools cover (ocr_words with a retained review, ocr_review_apply through the same stale- document guard, ocr_dictionary) and what is still UI-only (suggestions, per-word Verified marks, region re-reads). - ocr.ocr-engines stays partial rather than shipped: part of its evidence is conditional (the real-binary test runs only where tesseract is installed and skips elsewhere, and the process-level tests are Unix-only). It cites the tests that always run (location validation and the fail-closed rule, --list-langs parsing from stdout or stderr, the merge rules, the default strategy) and says which part is not proven everywhere. - `cargo xtask parity`: area I 32.7% -> 44.2% weighted (5 -> 7 shipped, 7 -> 9 partial); overall 52.8% -> 53.0% shipped and 58.2% -> 58.5% weighted. ROADMAP.md - One progress-log entry (2026-10-13, newest first) and the banner line. It says the catalogs are 21 with the Lithuanian additions not natively reviewed, that the end-to-end redaction test of the Type3 layer needs the craft-fonts build input, and that recognize_text_makes_a_scanned_page_searchable still fails where the models are installed, as main's log already records. docs/localization.md, docs/localization-parity.md - The Lithuanian entry count (2,578 -> 2,668) and the counts of the other catalogs that carry entry counts are re-measured with the 90 new entries; the Lithuanian OCR entries are marked as not natively reviewed. Checked: `cargo xtask parity` (numbers above, evidence tests exist) and `cargo xtask assets` pass on this tree. * M10.24: ocr: apply a late result only to the document it was read from The Recognize Text dialog closes on Go, so the document can be edited, or have text redacted, while a long run is still reading. The result was applied without the stale check the tools and the Verify screen use, so words read before a redaction could be written back as invisible text. The run now keeps its guard and applies through the checked path. Accept with the editable output in the Verify screen had the same gap (Exit keeps the review open): it now refuses when the document changed since the review started. Both new tests fail with their guard switched off. * M10.25: ocr: ocr_dictionary takes dictionary file names only Updating sorts, lower-cases and rewrites the file it is given, so pointed at .env or a README it would corrupt it. The tool now accepts only ocr-dictionary-<language>.txt style names, for reading as well as writing. The size-limit test used a file name the rule now refuses first, so it renamed its fixture to keep testing the size limit. The new test fails with the rule switched off. * M10.26: pt-BR: keep (s) on the word counts and say mecanismo for engine Two review fixes from a native speaker: the two word(s) lines had lost their (s), so a count of 1 read as 1 palavras, and engine was translated both as motor and mecanismo. Mecanismo is the Brazilian word for a software engine and two lines already used it, so the other six now match. * M10: fit the rebased OCR workflow to current main's structure * Fix OCR workflow issues from the review Honor the Recognize dialog's editable-output choice: its result opens in the review screen (whose Accept writes the editable document) instead of an invisible layer. Make page acceptance idempotent: a second Accept entire page never appends the OCR layer twice and keeps later corrections. Move page and region re-OCR onto a worker thread with a status line and a cancel that discards the reading. Skip the Tesseract probe for jobs that name their engines. Bound Tesseract's stderr while the child runs, not only after it exits. Surface the editable output's substitution notes in the tools' and the run's results. Also repair the OCR batch tool against the current structure: inputs are held to the read root again, and a batch whose inputs share an output name is refused before the output folder is created, compared the way the default filesystem compares names; and the editable writer test's word stays inside the fixture's pages. * Apply pre-review fixes to the OCR workflow Hold the automation OCR tools to the document's copying permission, as the other text tools are: the recognized text is left out of the results when copying is forbidden, and the paths that carry the words out of the document (the editable output, the review word list) are refused. Mark the ocr_dictionary tool as overwriting its path. Close the searchable layer's graphics state around its invisible text so the rendering mode cannot leak into later content. Read the OCR user dictionary folder from the portable data folder in portable mode, like the recovery store does. Resolve the attribution manifest's leftover conflict markers and regenerate the generated attribution. Describe the review flow, the searchable layer's Type3 fonts and the parity evidence references accurately, and drop an undeclared copy of the recovery module. * OCR: reject readings that exceed the word limit Assisted-by: OpenAI Codex * OCR: new documents go through the page tree builder main added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OCR: review notes and lints The editable output refuses a file created during recognition. ocr_words takes skip_text_pages and force_ocr. Automatic OCR checks the engine it will use. The stderr test stays below the kill cap, and a failed --ocr-verify shot exits with an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OCR: restore the Verify screenshot attribution, follow the moved constants The merge dropped the screenshot row with the translation rows. The model filename constants live in ocrs.rs now, so the license check reads them there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribution: the Verify screenshot entry goes after the header comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribution: the screenshot row in the table follows the manifest order Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
FredAppsandClaude Opus 5.5
|
250c2623d1 |
Tests: pin the file identifier formula, not a whole rewritten file (#6346)
The written file differs on Windows, so an identifier hashed from it cannot be pinned, and main's Windows gate failed. The formula is pinned on fixed bytes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
60f60fc0a2 |
Give a redacted or sanitized copy a new file identifier (#3734)
* cos: give a redacted or sanitized full save a new file identifier
A full save that the document asked for (Document::require_full_save_with_new_id)
now writes a fresh /ID pair, so a redacted or sanitized copy does not claim to be the
file it came from (ISO 32000-2 section 14.4). Ordinary saves are unchanged.
An encrypted file keeps the first string of /ID, because the file key is derived from
it, and gets a new second string. An encrypted file whose /ID is not a pair of strings
is left as it was, as before. SaveOptions::id_entropy lets a caller or a test supply
entropy; with it the new identifier depends only on that entropy and the written bytes.
Without it the identifier mixes the standard library's random hash keys and a
process-wide counter, the same facility the engine uses for new encryption keys.
Checked: the new tests in crates/cos/tests/file_identifier.rs fail on the clean base
(compile errors for the missing API), fail on assertions when only the API is in place,
and fail again when each pinned line is reverted (the regeneration call, the first-string
keep, the flag, the entropy input, and the ordinary-save guard). cargo test -p
pdfcraft-cos passes, and clippy with warnings denied is clean. A probe that wrote 16
ordinary full and incremental saves gave byte-identical files with and without this change.
* redact: give a redacted or sanitized copy a new file identifier
Applying redactions, sanitizing and removing hidden information now ask the cos
layer for a new file identifier (Document::require_full_save_with_new_id), so the
saved copy no longer carries the /ID of its source. Before this change the
previous revision was dropped but the identifier still linked the copy to the
original.
The request sits in remove_hidden itself, because its per-category callers (the
remove-hidden edit and the optimizer's discard option) call it directly, not
through sanitize(). sanitize() asks only on its nothing-hidden path, where
remove_hidden returned before asking: the rewrite is a new file either way.
Checked: the new tests a_redacted_or_sanitized_copy_gets_a_new_identifier and
a_remove_hidden_copy_gets_a_new_identifier fail before this change (the request
is missing after apply, and after remove_hidden) and fail again when the apply
call, the nothing-hidden branch or the remove_hidden request is reverted. cargo
test -p pdfcraft-redact passes with the tests included (152 passed, 1 ignored);
cargo clippy with warnings denied and cargo fmt --check are clean.
* docs(cos): say when a full save regenerates the file identifier
The writing section of the crate README described full saves without saying what
happens to /ID. It now says that an ordinary save keeps the identifier, that a
document which asks for a new one (redaction and sanitizing) gets both strings
regenerated, that SaveOptions::id_entropy supplies entropy for reproducible output,
and that an encrypted file keeps its first string because the key depends on it.
Checked: the text matches the code added in e1f761cc (regenerate_id, new_id_bytes,
require_full_save_with_new_id) and the tests in crates/cos/tests/file_identifier.rs.
No code changed in this commit.
* cos: hash the whole rewritten file into the new identifier
The identifier of a requested rewrite hashed only the first 64 KiB of the written file.
With caller-supplied id_entropy the identifier is deterministic, so two rewrites of
documents with the same length that differ only after byte 65,536 received the same
/ID, and a later edit of a large document could keep its previous /ID.
Every written byte is now hashed. The output is read once: one loop over 4 KiB chunks
feeds both halves of the 16-byte identifier, so the extra cost is one linear pass over
the output. Measured with a probe in the workspace dev profile (opt-level 1): hashing
64 MiB took 0.024 s, about 24 ms per 64 MiB of output. A release build was not measured.
Ordinary saves never reach this code, and their output is unchanged.
Checked: crates/cos/tests/file_identifier.rs gains
the_identifier_depends_on_the_whole_rewritten_file. It fails on the branch head before
this change ("content past 64 KiB must change the identifier", both identifiers equal)
and passes after it; reverting only the chunk loop to the 64 KiB prefix makes it fail
again. The six existing tests in that file pass, and so do the full cos and redact test
suites.
* cos: keep the key valid when an encrypted rewrite has no usable /ID
A requested rewrite of an encrypted file whose /ID was absent, had one entry, or had a
first entry that is not a string wrote no identifier at all, although the document had
asked for a new one.
The reader takes the empty string as the key input for these shapes (Document::first_id),
so the rewrite now writes the empty string (<>) as the first entry and a new second entry.
The key stays valid. A file whose first entry is a string keeps that string, as before.
Ordinary saves and unencrypted rewrites are unchanged.
Checked: an_encrypted_rewrite_without_a_usable_identifier_still_gets_a_new_one in
crates/cos/tests/file_identifier.rs covers the three shapes for RC4-128, AES-128 and
AES-256. After the rewrite each file opens with its password, the first string is empty
and the second is new. On the branch head the test fails with "the rewrite writes a
pair" (left 0, right 2). Reverting only the guard to the head's early return makes it
fail again. Isolated, the one-entry shape fails with left 1, right 2, and the non-string
shape with left Some(0), right Some(16). The other seven tests in the file pass.
* cos: derive the identifier from caller entropy with a fixed hash
With SaveOptions::id_entropy the identifier was built with std's DefaultHasher, whose
algorithm the standard library does not specify, and with write_usize (4 bytes on wasm32,
8 on 64-bit targets) and native-endian integers. The same entropy and the same file could
therefore give different identifiers on different platforms or toolchains, although the
documentation promised the bytes depend only on the entropy and the written file.
The identifier is now FNV-1a (64 bit, the published constants) over fixed-width big-endian
fields and the whole written file: the salt, the lane, id_seed and the length, then the
bytes. With caller entropy the salt is the FNV-1a hash of that entropy. Without entropy it
is still the standard library's random hash of a process counter, so only the salt differs
between the two paths. Every written byte is hashed, in one pass that feeds both lanes.
Cost: this pass is slower than the SipHash it replaces. A temporary probe in the workspace
dev profile (opt-level 1) hashed 64 MiB through both lanes in 0.129 s (about 500 MiB/s),
against 0.024 s before. A release build was not measured. The cost is paid once per
requested rewrite; ordinary saves do not reach this code.
Checked: the golden test fixed_entropy_rewrite in crates/cos/tests/file_identifier.rs pins
the 16 bytes for a fixed entropy and the standard fixture. On the previous hash it fails
(left [64, 145, ...], right [50, 125, ...]). An independent re-implementation of the same
formula in Python, run over the bytes the writer hashed (everything before startxref),
gives the same 16 bytes. The unit test fnv1a_is_the_published_64_bit_algorithm checks the
constants against the published vectors. FNV is not collision-resistant against chosen
content; that matters only if someone could choose a file's identifier, and the identifier
only has to be unique.
* Repair review branch integration
* cos: take fresh identifier entropy from the browser on wasm
Without SaveOptions::id_entropy the salt of a rewritten file's identifier was
RandomState and a process-wide counter. On wasm32-unknown-unknown those hash
keys are fixed at compile time, so two fresh web apps that followed the same
allocation and call sequence reproduced the same salt and the same identifier
for identical output, and a redacted copy stayed linkable to its source.
The default salt now comes from the browser first: crypto.getRandomValues
(Web Crypto, through js_sys, sixteen bytes folded with the same FNV-1a the
identifier uses), or Math.random where that API is missing, and only when
neither is reachable the previous RandomState salt. js-sys and wasm-bindgen
join pdfcraft-cos as wasm32-only dependencies; both are already in the
dependency tree of the web shell, and native builds compile neither. Caller
entropy is untouched: with id_entropy the identifier is still a pure function
of that entropy, the seed and the written bytes.
Checked: separate_fresh_processes_get_different_identifiers re-runs this test
binary as two fresh child processes over the same fixture and fails when the
default salt is made constant (so does the in-process divergence test);
stand_in_for_a_fresh_process is its no-op child half. Both pass with the real
salt. On WebAssembly no browser test harness exists, so the browser path is
covered by the wasm32 cargo check and by construction, not by a test. cargo
check --target wasm32-unknown-unknown -p pdfcraft-cos passes, cargo test -p
pdfcraft-cos and -p pdfcraft-redact pass, and clippy with warnings denied is
clean on both.
* Extract full-save trailer assembly
Assisted-by: OpenAI Codex
---------
Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com>
Co-authored-by: FredApps <FredApps@users.noreply.github.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
|
||
|
|
f16b5d9331 |
M13 batch: test that results hold no planted secret, and record network-crate reachability (#6297)
* M13 batch: test that results hold no planted secret, and record network-crate reachability * netcheck: list socket2, report a walk past the cap, test by the gate rule socket2 can open sockets, so it belongs in the list. A walk past the cap now fails instead of reading as unreachable. The baseline test follows check_rows, so a dependency update that moves a path no longer fails cargo test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ee39c7e2d0 |
Issue #5081: assembly at revision 3 and later needs bit 11; high-quality print, accessibility and form fill follow the revision (#6309)
* Issue #5081: assembly at revision 3 and later needs bit 11; high-quality print, accessibility and form fill follow the revision * Permissions: at revision 2, bits 11 and 12 mean nothing Table 22 defines them for revision 3 and later. A revision 2 file with bit 4 clear and bit 11 set was given page changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |