9 Commits

Include renames

Author	SHA1	Message	Date
5 people	9f04acccef	M10: OCR engines, confidence, preprocessing and a review workflow (#977) ... * M10.20: plan Unicode text for OCR output in Type3 faces (fonts, edit) OCR output used to flatten every character beyond WinAnsi to a silent `?`. This adds the two pieces that let the OCR layers write the recognized words as they are; the OCR code that uses them follows in M10.21. pdfcraft-fonts - `TextPlan::for_words` maps each word to the standard Helvetica when WinAnsi covers it, else to Type3 fonts drawn from the craft-fonts faces (the Japanese document face first, then the rest of the build input). Characters no face has a glyph for are counted (`missing`), never dropped silently. At most 240 characters per font and 2400 per plan, so one hostile word cannot balloon. - Which face has which character is asked once per (face, character) for a plan, not once per word. - `face_has`, `craft_glyph`, `win_ansi_char`, `fits_win_ansi` are public for it. pdfcraft-edit - `add_type3_fonts` / `stamp_with_fonts` register a plan's fonts on a page under a free `PCOcr`, `PCOcr1`, ... name, build each font once and share the object with later pages. A shared face is registered under a free name on a later page that already has a font of the name it was given first, instead of replacing that page's own font. - The fonts carry real advances and a ToUnicode CMap, so extraction reads the text back. pdfcraft-redact - A test shows a hand-built Type3 text layer (the shape the OCR output writes) is redacted like any other text and the verifier passes. Tests: pdfcraft-fonts 62 passed, pdfcraft-edit 129 passed, pdfcraft-redact 33 passed; clippy -D warnings clean on the three crates. The name-reuse and the redaction tests build their fonts by hand, so they run without the craft-fonts build input; the tests that plan real faces skip without it and were not run in this commit's own check. * M12: give the hostile-XFA open test more room on a loaded machine The test opens a form whose calculate script sets a value 50 000 times, against the one-second budget of an unasked script and a five-second wall-clock bound on the open. Both sit close to what a machine running several suites at once can take, so the test can fail without anything being wrong. It now runs 20 000 iterations (the total is 19999) and allows 30 seconds, which still fails for a script that hangs or is not capped. Test-only; unrelated to the OCR work in this series, so it is a commit of its own. Tests: both hostile_xfa tests in pdfcraft-engine pass. * M10.21: pluggable OCR engines, preprocessing, review sessions and the OCR tools (ocr, engine, create, automation) Scan & OCR gets a second engine, optional raster clean-up, a review step before anything is applied, editable output and a batch mode, headlessly first: the interface for them follows in M10.22. pdfcraft-ocr - A `Recognizer` trait with two engines: the built-in ocrs (moved out of lib.rs) and the installed tesseract program, run as a fenced external process: fixed argv, allow-listed language codes, a private temp directory (mode 0700 on Unix), 120 s timeout, 64 MiB output cap, 50 000-word reading cap. A location from PDFCRAFT_TESSERACT is validated (absolute, existing, executable) and an invalid one fails closed instead of falling back to the PATH. `--list-langs` and `--version` are read from stdout or, for older versions, stderr. - Strategies: PrimaryOnly, ConfidenceWeighted (the second engine joins when the first reading's mean is under 70 or unknown) and a ROVER merge (higher confidence wins per word; against an ocrs word of unknown confidence a tesseract word wins only at 70 or more; capped at 100 M comparisons; a failing partner degrades to the primary with a note). - Preprocessing: opt-in auto-rotate, deskew, denoise and binarize on the RGBA raster, each with an exact inverse so word boxes map back to the scan. The 300 dpi normalization the steps are calibrated for runs only when a step is on: with every step off the render is read as rendered, as before. (Enlarging a 72 dpi render 4x on its own made the models read "due: €250" as "ue: ?250".) `OcrImage` is validated by construction (side and 64 MP area caps, checked before any buffer is allocated, including by the resize). - Confidence bands (High >= 90, Medium 70-89, Low < 70) drive review. pdfcraft-engine, pdfcraft-create - `OcrSettings` gains engine, strategy, the four steps, output mode, region, the skip flags and the low-confidence threshold; Automatic reads confidence weighted unless a strategy is chosen. - A recognition is a session: the document guard refuses a stale result with its reason, pages can be applied one at a time (the guard is taken again after each applied page), and applying is one undoable step. A per-language dictionary (a plain-text file: words added from any document stay on disk after it closes). - Output is searchable (an invisible layer) or a NEW editable document of visible text at the recognized boxes (`create::from_contents`); words beyond WinAnsi use the Type3 faces of M10.20 in both, with the shared plan across pages. - Engine discovery never blocks the interface: results are cached, a stale answer is shown while a background probe retries, and the models search and `--version` are cached. Batch: pages or files that already have text are skipped unless forced, and inputs sharing a file name are found up front. pdfcraft-automation - ocr_recognize (engine, strategy, steps, region, output mode), ocr_words (with a retained review handle), ocr_review_apply, ocr_dictionary, ocr_status and ocr_recognize_files. The editable `out` must be a new file inside --root and is refused for a searchable run (it used to be ignored), two batch inputs of one file name are refused before anything is read, and the batch folder is checked for existing outputs. pdfcraft-ui-egui - Only what the new API needs: the Recognize Text dialog reads with `recognizers(&settings)`. `ocr.correct` is registered with its label in every catalog; the screen itself is M10.22. Tests: pdfcraft-ocr 78 passed; pdfcraft-create passed; pdfcraft-engine and pdfcraft-automation 450 passed and 1 failed (the automation lib 17, automation.rs 157, hostile_args 2 among them); pdfcraft-ui-egui lib + ocr + control tests 391 passed; clippy -D warnings clean on the five crates. The one failure is in a test this commit does not change: recognize_text_makes_a_scanned_page_searchable fails where the models are installed (they read "quick" as "auick" at that render size); the roadmap on main already records it. The real tesseract binary ran in tesseract::tests::recognize_runs_the_installed_binary (tesseract 5.4.0, PATH). * M10.22: OCR Verify screen, the Recognize Text options and the interface catalogs (ui-egui) Scan & OCR ▸ Correct recognized text: a full-window screen where a recognition is reviewed before anything is applied, plus the dialog and batch options for the engines of M10.21. OCR Verify screen - Four panes: the page list, the source page with confidence-coloured word boxes (click to select, drag to draw a region), the recognized text, and a zoom plus word inspector. Corrections commit on Enter (an empty commit removes the word), Remove/Restore, "+ Dictionary" (a per-language file in the preferences folder, so a word added from a document stays on disk after it closes), per-word and per-page Verified, F8 / Shift+F8 walk the low-confidence words, Ctrl+T marks the page reviewed, Ctrl+Tab moves the pane focus, and review edits undo within the panel. - Re-OCR of the whole page or a drawn region. A region re-read moves the review marks of the words it keeps to their new positions and selects the first new word; a whole-page re-read clears that page's marks (and the undo brings them back). The region outline is scaled onto the pane, not mapped through user space again, so it is right on rotated and cropped pages. - Accept / Reject per page or whole. Pages accepted one at a time each apply once (the document guard is taken again after each applied page; an edit made elsewhere in between is still refused with its reason), and Accept in searchable mode is one undoable step. Editable mode writes a NEW `<name>_ocr.pdf` through a save dialog and never the source. A real Cancel keeps the pages already read. - The engine and strategy menus disable a choice that cannot run and say why; the check never waits on a process (a background probe fills a cache), and choosing Automatic moves "Primary only" to Confidence weighted. Recognize Text dialog and batch - Engine, strategy, the four clean-up steps and the skip flags are remembered with the other choices (a saved Automatic without a strategy reads confidence weighted). The batch refuses two inputs of one file name and existing outputs before reading anything, notes low-confidence pages at the settings' threshold, and its toast has a Cancel (the file in flight finishes). Catalogs and attribution - The 90 new interface strings are in all 21 catalogs (the Lithuanian ones were written without native review, and the Bulgarian ones avoid U+045D, which the bundled monospace face lacks). ATTRIBUTION.toml carries the new hashes, a note on the Lithuanian entries and the README screenshot; ATTRIBUTION.md is regenerated. - README: a screenshot of the Verify screen, taken with `cargo xtask screenshots` from a synthetic scan (`shot --ocr-verify on`) built from allowed assets; the page reads it with the real engines. Tests: pdfcraft-ui-egui lib 373 passed (1 ignored: the Windows hinting measurement probe), tests ocr 6, control 46, fonts 9; clippy -D warnings clean on the engine, ui-egui, pdfcraft and xtask packages; `cargo xtask assets` ok. The review state machine has unit tests for every fix above; each was seen failing with its fix reverted. * M10.23: parity entries, roadmap and localization notes for the OCR workflow Records what M10.20-M10.22 ship, and what they do not. parity/acrobat-features.toml - Status changes against main: ocr.rotation-detection and ocr.deskew planned -> shipped (the opt-in steps of M10.21, with their tests); ocr.correct-recognized- text and ocr.ocr-engines planned -> partial. Other Area I entries get their evidence and notes brought up to date. - ocr.correct-recognized-text: the evidence includes the one-page-at-a-time accept and the region re-read tests; the notes say what the headless tools cover (ocr_words with a retained review, ocr_review_apply through the same stale- document guard, ocr_dictionary) and what is still UI-only (suggestions, per-word Verified marks, region re-reads). - ocr.ocr-engines stays partial rather than shipped: part of its evidence is conditional (the real-binary test runs only where tesseract is installed and skips elsewhere, and the process-level tests are Unix-only). It cites the tests that always run (location validation and the fail-closed rule, --list-langs parsing from stdout or stderr, the merge rules, the default strategy) and says which part is not proven everywhere. - `cargo xtask parity`: area I 32.7% -> 44.2% weighted (5 -> 7 shipped, 7 -> 9 partial); overall 52.8% -> 53.0% shipped and 58.2% -> 58.5% weighted. ROADMAP.md - One progress-log entry (2026-10-13, newest first) and the banner line. It says the catalogs are 21 with the Lithuanian additions not natively reviewed, that the end-to-end redaction test of the Type3 layer needs the craft-fonts build input, and that recognize_text_makes_a_scanned_page_searchable still fails where the models are installed, as main's log already records. docs/localization.md, docs/localization-parity.md - The Lithuanian entry count (2,578 -> 2,668) and the counts of the other catalogs that carry entry counts are re-measured with the 90 new entries; the Lithuanian OCR entries are marked as not natively reviewed. Checked: `cargo xtask parity` (numbers above, evidence tests exist) and `cargo xtask assets` pass on this tree. * M10.24: ocr: apply a late result only to the document it was read from The Recognize Text dialog closes on Go, so the document can be edited, or have text redacted, while a long run is still reading. The result was applied without the stale check the tools and the Verify screen use, so words read before a redaction could be written back as invisible text. The run now keeps its guard and applies through the checked path. Accept with the editable output in the Verify screen had the same gap (Exit keeps the review open): it now refuses when the document changed since the review started. Both new tests fail with their guard switched off. * M10.25: ocr: ocr_dictionary takes dictionary file names only Updating sorts, lower-cases and rewrites the file it is given, so pointed at .env or a README it would corrupt it. The tool now accepts only ocr-dictionary-<language>.txt style names, for reading as well as writing. The size-limit test used a file name the rule now refuses first, so it renamed its fixture to keep testing the size limit. The new test fails with the rule switched off. * M10.26: pt-BR: keep (s) on the word counts and say mecanismo for engine Two review fixes from a native speaker: the two word(s) lines had lost their (s), so a count of 1 read as 1 palavras, and engine was translated both as motor and mecanismo. Mecanismo is the Brazilian word for a software engine and two lines already used it, so the other six now match. * M10: fit the rebased OCR workflow to current main's structure * Fix OCR workflow issues from the review Honor the Recognize dialog's editable-output choice: its result opens in the review screen (whose Accept writes the editable document) instead of an invisible layer. Make page acceptance idempotent: a second Accept entire page never appends the OCR layer twice and keeps later corrections. Move page and region re-OCR onto a worker thread with a status line and a cancel that discards the reading. Skip the Tesseract probe for jobs that name their engines. Bound Tesseract's stderr while the child runs, not only after it exits. Surface the editable output's substitution notes in the tools' and the run's results. Also repair the OCR batch tool against the current structure: inputs are held to the read root again, and a batch whose inputs share an output name is refused before the output folder is created, compared the way the default filesystem compares names; and the editable writer test's word stays inside the fixture's pages. * Apply pre-review fixes to the OCR workflow Hold the automation OCR tools to the document's copying permission, as the other text tools are: the recognized text is left out of the results when copying is forbidden, and the paths that carry the words out of the document (the editable output, the review word list) are refused. Mark the ocr_dictionary tool as overwriting its path. Close the searchable layer's graphics state around its invisible text so the rendering mode cannot leak into later content. Read the OCR user dictionary folder from the portable data folder in portable mode, like the recovery store does. Resolve the attribution manifest's leftover conflict markers and regenerate the generated attribution. Describe the review flow, the searchable layer's Type3 fonts and the parity evidence references accurately, and drop an undeclared copy of the recovery module. * OCR: reject readings that exceed the word limit Assisted-by: OpenAI Codex * OCR: new documents go through the page tree builder main added Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OCR: review notes and lints The editable output refuses a file created during recognition. ocr_words takes skip_text_pages and force_ocr. Automatic OCR checks the engine it will use. The stderr test stays below the kill cap, and a failed --ocr-verify shot exits with an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OCR: restore the Verify screenshot attribution, follow the moved constants The merge dropped the screenshot row with the translation rows. The model filename constants live in ocrs.rs now, so the license check reads them there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribution: the Verify screenshot entry goes after the header comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attribution: the screenshot row in the table follows the manifest order Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 23:39:55 +01:00
Zeyu LiandBrandon Thomas	b91d38c073	M14: text the fonts can't draw is kept and reported, never written as ? silently (#1112) ... * M14: text the fonts can't draw is kept and reported, never written as ? silently Form values and captions, text boxes, callouts, Fill & Sign text, measurement labels and stamp by-lines keep text the standard fonts can't draw, draw ? for it, mark the annotation (PCUndrawn) and report the edit (interface notice, tools' warnings); forms set /NeedAppearances (not in PDF/A). Flattening, redaction overlay text, typed and visible signatures and create-from-text refuse instead. Reports escape such characters as [U+XXXX]. * Catalogs: the two new can't-draw messages in Czech, Slovene, Kazakh, Turkish and Greek; refresh hashes --------- Co-authored-by: Brandon Thomas <echelon@gmail.com>	2026-10-11 17:53:55 -04:00
4 people	cc344a26de	Typed signatures draw Cyrillic and Greek letters (#846) (#963) ... Dancing Script, the typed-signature face, is Latin and Vietnamese only, so script_outline skipped every Cyrillic or Greek character and a Serbian Cyrillic name came out empty. Characters the script lacks are now drawn from the bundled Inter Regular, slanted about 12 degrees and scaled to the script's capital height. Characters neither face has are still skipped. Inter's bytes are now exported as pdfcraft_fonts::INTER_REGULAR and the UI uses that static, so the font is still embedded once. Co-authored-by: DWAA1660 <DWAA1660@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Brandon Thomas <echelon@gmail.com>	2026-10-11 13:20:11 -04:00
4 people	21a9624585	M7: Arabic in added text (shaping, bidi order, searchable Type 3 glyphs) (#403) ... Add text wrote Arabic as "?" (standard-14 fonts, WinAnsi). Items with Arabic are now shaped with the craft-fonts Arab face through harfrust (already in the tree via egui) and laid out line by line in display order with unicode-bidi (UAX #9), so Arabic mixed with Latin, digits and brackets reads correctly. Each cluster (a letter with its dots or marks) is one glyph of a Type 3 font drawn from the face's outlines, with a ToUnicode entry, so text extraction, copy and search get every letter once and no font file is embedded. - fonts: shape_arabic / arabic_glyph / arabic_has / document_arabic_font; japanese_glyph's bounded outline code is shared as bounded_outline. - edit: draw_arabic and type3_arabic; wrapping measures shaped words. Fonts get names free on the page (a full save renumbers objects, not resource names) and are removed when the item is updated or deleted. Outlines are read before anything is written. - Without the face, adding Arabic is a clear error that changes nothing; an existing item holding Arabic still moves, drawn as "?" as before. - Non-Arabic items are drawn exactly as before. Refs #250, #280, #369. Co-authored-by: fares <aw674860@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Brandon Thomas <echelon@gmail.com>	2026-10-09 09:22:50 -04:00
3 people	3146e8767e	M7: preserve Japanese Gothic fallback family and weight (#394) ... * M7: select real Japanese fallback faces by family and weight Refs #184. Keep Type 3 font-name reporting separate in #387; missing weights and the small web font input remain explicit limitations. Assisted-by: OpenAI GPT-6.1 (gpt-6.1-sol) * Test the Japanese fallback name for a sans source: the Gothic face since #394 matches the source style Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Brandon Thomas <echelon@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-08 21:48:02 -04:00
4 people	89896af68b	Fix saved signature replacement and long-name clipping (#163) (#178) ... * #163: allow replacing saved signatures and render long names fully Add saved signature and initials previews with change/remove/add controls in Fill & Sign. Preserve placed annotations and saved values on cancellation. Use bounded line outlines for long names and fit typed placements to 150 pt. Co-authored-by: Codex <noreply@openai.com> * cargo fmt after the rename Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Brandon Thomas <echelon@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-07 21:37:28 -04:00
Brandon ThomasandClaude Opus 5.5	1d67833599	Fonts: use craft-fonts as an optional build input; move the Japanese fonts out ... Shippori Mincho (and its OFL file) leaves assets/fonts/: the Japanese fonts now live in storytold/craft-fonts. printcraft-fonts gets craft-fonts' build.rs recipe: with CRAFT_FONTS_DIR=<checkout> it embeds the manifest's fonts as CRAFT_FONTS (wasm32: only BIZ UDPGothic Regular); without it CRAFT_FONTS is empty and the app builds, tests and runs as before, minus the Japanese face. - SHIPPORI_MINCHO is now Option<&[u8]> from CRAFT_FONTS; japanese_glyph (was shippori_glyph) uses document_japanese_font() (Shippori Mincho, then BIZ UDMincho, then any Jpan face) and returns GlyphError::NoFont without one. - Editing PDF text into Japanese without craft-fonts is a clear EditError that leaves the page untouched. - UI: every Jpan craft font (BIZ UDPGothic first) is the last fallback of every egui family, sharing the same static bytes. - Tests: Japanese glyph/edit tests skip without craft-fonts; new UI tests for no-tofu Japanese with craft-fonts and working fonts without it. - CI: the Linux test job checks out craft-fonts at a pinned SHA and runs the tests with CRAFT_FONTS_DIR. Releases build every platform with it (CRAFT_FONTS_REQUIRED=1) and ship each font's OFL as OFL-<family>.txt. - ATTRIBUTION.toml gains [[build_input]] (checked by xtask assets); NOTICE, AGENTS.md, CLAUDE.md, README and docs say fonts live in craft-fonts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 11:36:23 -04:00
4 people	e59c9a793e	Fix text editing layout and preserve source PDF font styles (#36) ... * Comments: a text box's rectangle follows its edited text Editing a text box (or typewriter or callout) rewrote /Contents but kept the old /Rect: the appearance clipped the new text to the old box and silently dropped lines that no longer fit, while shorter text left an oversized box. set_contents now re-measures the text with the same Helvetica widths as at creation and re-fits the box: the wrap width and top edge stay, the height fits the wrapped lines. A callout's leader line keeps its place and re-attaches to the moved box (that geometry moved from set_rect into a shared apply_text_box). A locked box keeps its text editable, so it re-fits too. Covered by annot tests (grow and shrink with nothing clipped, callout leader re-attach, locked box) and a comment_edit round trip through the automation tools; parity entries cite them. Co-Authored-By: GLM <noreply@z.ai> * Polish edit text and images tool * feat(edit-text): select source font style * fix(edit-text): preserve detected PDF font styles * feat(edit-text): support Japanese replacements * fix(forms): escape non-ASCII format script strings * Forms: unwrap_or instead of map_or with identity (clippy map_or_identity) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Brandon Thomas <echelon@gmail.com> Co-authored-by: GLM <noreply@z.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-05 07:48:23 -04:00
Brandon ThomasandClaude Opus 5.5	01d0dc67ba	M5: Fill & Sign typed signatures and initials (Type or Draw; script-font outlines drawn as filled paths); fill_sign_add signature/initials ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-02 09:59:28 -04:00