* M10.20: plan Unicode text for OCR output in Type3 faces (fonts, edit)
OCR output used to flatten every character beyond WinAnsi to a silent `?`. This
adds the two pieces that let the OCR layers write the recognized words as they
are; the OCR code that uses them follows in M10.21.
pdfcraft-fonts
- `TextPlan::for_words` maps each word to the standard Helvetica when WinAnsi
covers it, else to Type3 fonts drawn from the craft-fonts faces (the Japanese
document face first, then the rest of the build input). Characters no face
has a glyph for are counted (`missing`), never dropped silently. At most 240
characters per font and 2400 per plan, so one hostile word cannot balloon.
- Which face has which character is asked once per (face, character) for a plan,
not once per word.
- `face_has`, `craft_glyph`, `win_ansi_char`, `fits_win_ansi` are public for it.
pdfcraft-edit
- `add_type3_fonts` / `stamp_with_fonts` register a plan's fonts on a page under
a free `PCOcr`, `PCOcr1`, ... name, build each font once and share the object
with later pages. A shared face is registered under a free name on a later
page that already has a font of the name it was given first, instead of
replacing that page's own font.
- The fonts carry real advances and a ToUnicode CMap, so extraction reads the
text back.
pdfcraft-redact
- A test shows a hand-built Type3 text layer (the shape the OCR output writes) is
redacted like any other text and the verifier passes.
Tests: pdfcraft-fonts 62 passed, pdfcraft-edit 129 passed, pdfcraft-redact 33
passed; clippy -D warnings clean on the three crates. The name-reuse and the
redaction tests build their fonts by hand, so they run without the craft-fonts
build input; the tests that plan real faces skip without it and were not run in
this commit's own check.
* M12: give the hostile-XFA open test more room on a loaded machine
The test opens a form whose calculate script sets a value 50 000 times, against
the one-second budget of an unasked script and a five-second wall-clock bound
on the open. Both sit close to what a machine running several suites at once
can take, so the test can fail without anything being wrong.
It now runs 20 000 iterations (the total is 19999) and allows 30 seconds, which
still fails for a script that hangs or is not capped. Test-only; unrelated to
the OCR work in this series, so it is a commit of its own.
Tests: both hostile_xfa tests in pdfcraft-engine pass.
* M10.21: pluggable OCR engines, preprocessing, review sessions and the OCR tools (ocr, engine, create, automation)
Scan & OCR gets a second engine, optional raster clean-up, a review step before
anything is applied, editable output and a batch mode, headlessly first: the
interface for them follows in M10.22.
pdfcraft-ocr
- A `Recognizer` trait with two engines: the built-in ocrs (moved out of lib.rs)
and the installed tesseract program, run as a fenced external process: fixed
argv, allow-listed language codes, a private temp directory (mode 0700 on
Unix), 120 s timeout, 64 MiB output cap, 50 000-word reading cap. A location
from PDFCRAFT_TESSERACT is validated (absolute, existing, executable) and an
invalid one fails closed instead of falling back to the PATH. `--list-langs`
and `--version` are read from stdout or, for older versions, stderr.
- Strategies: PrimaryOnly, ConfidenceWeighted (the second engine joins when the
first reading's mean is under 70 or unknown) and a ROVER merge (higher
confidence wins per word; against an ocrs word of unknown confidence a
tesseract word wins only at 70 or more; capped at 100 M comparisons; a failing
partner degrades to the primary with a note).
- Preprocessing: opt-in auto-rotate, deskew, denoise and binarize on the RGBA
raster, each with an exact inverse so word boxes map back to the scan. The
300 dpi normalization the steps are calibrated for runs only when a step is
on: with every step off the render is read as rendered, as before. (Enlarging
a 72 dpi render 4x on its own made the models read "due: €250" as "ue: ?250".)
`OcrImage` is validated by construction (side and 64 MP area caps, checked
before any buffer is allocated, including by the resize).
- Confidence bands (High >= 90, Medium 70-89, Low < 70) drive review.
pdfcraft-engine, pdfcraft-create
- `OcrSettings` gains engine, strategy, the four steps, output mode, region, the
skip flags and the low-confidence threshold; Automatic reads confidence
weighted unless a strategy is chosen.
- A recognition is a session: the document guard refuses a stale result with its
reason, pages can be applied one at a time (the guard is taken again after each
applied page), and applying is one undoable step. A per-language dictionary
(a plain-text file: words added from any document stay on disk after it closes).
- Output is searchable (an invisible layer) or a NEW editable document of visible
text at the recognized boxes (`create::from_contents`); words beyond WinAnsi use
the Type3 faces of M10.20 in both, with the shared plan across pages.
- Engine discovery never blocks the interface: results are cached, a stale
answer is shown while a background probe retries, and the models search and
`--version` are cached. Batch: pages or files that already have text are
skipped unless forced, and inputs sharing a file name are found up front.
pdfcraft-automation
- ocr_recognize (engine, strategy, steps, region, output mode), ocr_words (with a
retained review handle), ocr_review_apply, ocr_dictionary, ocr_status and
ocr_recognize_files. The editable `out` must be a new file inside --root and is
refused for a searchable run (it used to be ignored), two batch inputs of one
file name are refused before anything is read, and the batch folder is checked
for existing outputs.
pdfcraft-ui-egui
- Only what the new API needs: the Recognize Text dialog reads with
`recognizers(&settings)`. `ocr.correct` is registered with its label in every
catalog; the screen itself is M10.22.
Tests: pdfcraft-ocr 78 passed; pdfcraft-create passed; pdfcraft-engine and
pdfcraft-automation 450 passed and 1 failed (the automation lib 17, automation.rs
157, hostile_args 2 among them); pdfcraft-ui-egui lib + ocr + control tests 391
passed; clippy -D warnings clean on the five crates. The one failure is in a
test this commit does not change:
recognize_text_makes_a_scanned_page_searchable fails where the models are
installed (they read "quick" as "auick" at that render size); the roadmap on main
already records it. The real tesseract binary ran in
tesseract::tests::recognize_runs_the_installed_binary (tesseract 5.4.0, PATH).
* M10.22: OCR Verify screen, the Recognize Text options and the interface catalogs (ui-egui)
Scan & OCR ▸ Correct recognized text: a full-window screen where a recognition
is reviewed before anything is applied, plus the dialog and batch options for
the engines of M10.21.
OCR Verify screen
- Four panes: the page list, the source page with confidence-coloured word boxes
(click to select, drag to draw a region), the recognized text, and a zoom plus
word inspector. Corrections commit on Enter (an empty commit removes the word),
Remove/Restore, "+ Dictionary" (a per-language file in the preferences folder,
so a word added from a document stays on disk after it closes), per-word and
per-page Verified, F8 / Shift+F8 walk the low-confidence words, Ctrl+T marks the
page reviewed, Ctrl+Tab moves the pane focus, and review edits undo within the
panel.
- Re-OCR of the whole page or a drawn region. A region re-read moves the review
marks of the words it keeps to their new positions and selects the first new
word; a whole-page re-read clears that page's marks (and the undo brings them
back). The region outline is scaled onto the pane, not mapped through user
space again, so it is right on rotated and cropped pages.
- Accept / Reject per page or whole. Pages accepted one at a time each apply once
(the document guard is taken again after each applied page; an edit made
elsewhere in between is still refused with its reason), and Accept in
searchable mode is one undoable step. Editable mode writes a NEW
`<name>_ocr.pdf` through a save dialog and never the source. A real Cancel
keeps the pages already read.
- The engine and strategy menus disable a choice that cannot run and say why;
the check never waits on a process (a background probe fills a cache), and
choosing Automatic moves "Primary only" to Confidence weighted.
Recognize Text dialog and batch
- Engine, strategy, the four clean-up steps and the skip flags are remembered with
the other choices (a saved Automatic without a strategy reads confidence
weighted). The batch refuses two inputs of one file name and existing outputs
before reading anything, notes low-confidence pages at the settings' threshold,
and its toast has a Cancel (the file in flight finishes).
Catalogs and attribution
- The 90 new interface strings are in all 21 catalogs (the Lithuanian ones were
written without native review, and the Bulgarian ones avoid U+045D, which the
bundled monospace face lacks). ATTRIBUTION.toml carries the new hashes, a
note on the Lithuanian entries and the README screenshot; ATTRIBUTION.md is
regenerated.
- README: a screenshot of the Verify screen, taken with
`cargo xtask screenshots` from a synthetic scan (`shot --ocr-verify on`) built
from allowed assets; the page reads it with the real engines.
Tests: pdfcraft-ui-egui lib 373 passed (1 ignored: the Windows hinting
measurement probe), tests ocr 6, control 46, fonts 9; clippy -D warnings clean on
the engine, ui-egui, pdfcraft and xtask packages; `cargo xtask assets` ok. The
review state machine has unit tests for every fix above; each was seen failing
with its fix reverted.
* M10.23: parity entries, roadmap and localization notes for the OCR workflow
Records what M10.20-M10.22 ship, and what they do not.
parity/acrobat-features.toml
- Status changes against main: ocr.rotation-detection and ocr.deskew planned ->
shipped (the opt-in steps of M10.21, with their tests); ocr.correct-recognized-
text and ocr.ocr-engines planned -> partial. Other Area I entries get their
evidence and notes brought up to date.
- ocr.correct-recognized-text: the evidence includes the one-page-at-a-time accept
and the region re-read tests; the notes say what the headless tools cover
(ocr_words with a retained review, ocr_review_apply through the same stale-
document guard, ocr_dictionary) and what is still UI-only (suggestions,
per-word Verified marks, region re-reads).
- ocr.ocr-engines stays partial rather than shipped: part of its evidence is
conditional (the real-binary test runs only where tesseract is installed and
skips elsewhere, and the process-level tests are Unix-only). It cites the tests
that always run (location validation and the fail-closed rule, --list-langs
parsing from stdout or stderr, the merge rules, the default strategy) and says
which part is not proven everywhere.
- `cargo xtask parity`: area I 32.7% -> 44.2% weighted (5 -> 7 shipped, 7 -> 9
partial); overall 52.8% -> 53.0% shipped and 58.2% -> 58.5% weighted.
ROADMAP.md
- One progress-log entry (2026-10-13, newest first) and the banner line. It says
the catalogs are 21 with the Lithuanian additions not natively reviewed, that
the end-to-end redaction test of the Type3 layer needs the craft-fonts build
input, and that recognize_text_makes_a_scanned_page_searchable still fails where
the models are installed, as main's log already records.
docs/localization.md, docs/localization-parity.md
- The Lithuanian entry count (2,578 -> 2,668) and the counts of the other
catalogs that carry entry counts are re-measured with the 90 new entries; the
Lithuanian OCR entries are marked as not natively reviewed.
Checked: `cargo xtask parity` (numbers above, evidence tests exist) and
`cargo xtask assets` pass on this tree.
* M10.24: ocr: apply a late result only to the document it was read from
The Recognize Text dialog closes on Go, so the document can be edited, or have
text redacted, while a long run is still reading. The result was applied
without the stale check the tools and the Verify screen use, so words read
before a redaction could be written back as invisible text. The run now keeps
its guard and applies through the checked path.
Accept with the editable output in the Verify screen had the same gap (Exit
keeps the review open): it now refuses when the document changed since the
review started.
Both new tests fail with their guard switched off.
* M10.25: ocr: ocr_dictionary takes dictionary file names only
Updating sorts, lower-cases and rewrites the file it is given, so pointed at
.env or a README it would corrupt it. The tool now accepts only
ocr-dictionary-<language>.txt style names, for reading as well as writing.
The size-limit test used a file name the rule now refuses first, so it renamed
its fixture to keep testing the size limit. The new test fails with the rule
switched off.
* M10.26: pt-BR: keep (s) on the word counts and say mecanismo for engine
Two review fixes from a native speaker: the two word(s) lines had lost their
(s), so a count of 1 read as 1 palavras, and engine was translated both as
motor and mecanismo. Mecanismo is the Brazilian word for a software engine and
two lines already used it, so the other six now match.
* M10: fit the rebased OCR workflow to current main's structure
* Fix OCR workflow issues from the review
Honor the Recognize dialog's editable-output choice: its result opens in
the review screen (whose Accept writes the editable document) instead of
an invisible layer. Make page acceptance idempotent: a second Accept
entire page never appends the OCR layer twice and keeps later
corrections. Move page and region re-OCR onto a worker thread with a
status line and a cancel that discards the reading. Skip the Tesseract
probe for jobs that name their engines. Bound Tesseract's stderr while
the child runs, not only after it exits. Surface the editable output's
substitution notes in the tools' and the run's results.
Also repair the OCR batch tool against the current structure: inputs are
held to the read root again, and a batch whose inputs share an output
name is refused before the output folder is created, compared the way
the default filesystem compares names; and the editable writer test's
word stays inside the fixture's pages.
* Apply pre-review fixes to the OCR workflow
Hold the automation OCR tools to the document's copying permission, as
the other text tools are: the recognized text is left out of the
results when copying is forbidden, and the paths that carry the words
out of the document (the editable output, the review word list) are
refused. Mark the ocr_dictionary tool as overwriting its path. Close
the searchable layer's graphics state around its invisible text so the
rendering mode cannot leak into later content. Read the OCR user
dictionary folder from the portable data folder in portable mode, like
the recovery store does. Resolve the attribution manifest's leftover
conflict markers and regenerate the generated attribution. Describe the
review flow, the searchable layer's Type3 fonts and the parity evidence
references accurately, and drop an undeclared copy of the recovery
module.
* OCR: reject readings that exceed the word limit
Assisted-by: OpenAI Codex
* OCR: new documents go through the page tree builder main added
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* OCR: review notes and lints
The editable output refuses a file created during recognition. ocr_words takes
skip_text_pages and force_ocr. Automatic OCR checks the engine it will use. The
stderr test stays below the kill cap, and a failed --ocr-verify shot exits with
an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* OCR: restore the Verify screenshot attribution, follow the moved constants
The merge dropped the screenshot row with the translation rows. The model
filename constants live in ocrs.rs now, so the license check reads them there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Attribution: the Verify screenshot entry goes after the header comment
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Attribution: the screenshot row in the table follows the manifest order
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: eliets2 <287273517+eliets2@users.noreply.github.com>
Co-authored-by: FredApps <FredApps@users.noreply.github.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Create PDF added each page to the page tree through Document::update_dict, which copies the tree's dictionary and its whole /Kids array, so a text of many form feeds took quadratic time: 40,000 pages took 108 s in a debug build. Pages are now collected and /Kids and /Count are written once, for blank, image and text documents.
An end that is missing or isn't a number sorted below any number, so [1],
or [1 7 0 R] with 7 leading nowhere, read as inverted on a JPEG or a stencil
mask, and so did an end too long to hold, which parses as infinite. Such an
array now leaves the default mapping, as it does on the lossless path.
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The shapes in these tests all ran one way. The way round depends on the shapes: 1 × 100,000 to
100,000 × 1 asks for 10 GB if it goes across first, or 30 GB at three components, so that is held to
as well. A mask of its image's size comes through byte for byte, and a larger one scales.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Colour spaces already had tests of this through their bases. `/Decode`, in the renderer and in image
export, `/W`, `/W2` and an Indexed space's table hadn't: each read now has such a reference in its
test, and ends.
These hold that the read ends, not what value it comes to. A reference to a reference in a number
array reads as the first number of that reference in the renderer (`/Decode [11 0 R 0]`, with 11
naming 6 and 6 holding 1, reads as 6). That is older than this branch, in `IndirectObject::read` for
an object stored on its own and `ObjectStream::get` for one in an object stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Scaling goes down first, through a buffer of the source's width by the target's
height at 16 bytes a pixel, and neither image's size limit bounds that product.
A soft mask of 60,000 × 1 on an image of 1 × 60,000, about a kilobyte of PDF,
asked for 57 GB and aborted the process, in image export, Office export, Copy
Image and Optimize at its default settings. The same shapes in smaller sizes
held gigabytes.
Such a picture is scaled across first, which holds no more pixels than one of
the two has, so the size limits bound it after all. Proportions that aren't
far apart scale as they did, to the byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Reading a JPEG's header was free, and it can be most of the file: a mask of
1 MiB of tables that disagreed with its dictionary, shared by 4,000 images,
took 101 s with none refused, and that time grows as the square of the file.
A JPEG now pays 4 steps a byte before its tables are parsed, on every path, and
a mask's header is read once. That is a bound and not a speed-up: the same
file of 4,000 images fits the allowance and still takes 97 s, and 12,000 stop
after 10,233 in 250 s.
A chain that can't run pays nothing for its pixels, and the docs say what past
the allowance isn't counted: LZW streams the image pass didn't read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
A JPEG that was itself a mask paid nothing, and images can share one: 300
images in 66 KB took 135 s with none refused, about 0.45 s for 220 bytes. It
now pays for its samples each time an image reads it.
Export priced a JPEG by its dictionary and decoded the size in its header,
comparing them only afterwards: an image of one pixel over a header of 500
megapixels cost 2 steps and 0.3 s. Both paths now read the header first and
leave a JPEG that disagrees undecoded, as Optimize and the CMYK path do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Four ways round the allowance, each measured by a review of hostile input:
A Flate stream read as zlib that fails at once, and then as raw deflate that
decodes everything before failing, gave nothing back and so was charged
nothing, here and in DecodeBudget: 110 s for a 387 KB file, all decoded. What
the retry decoded now counts, though the data is still dropped as it was.
Fax and 1-bit Flate images paid by the byte for work done by the pixel: 19
bytes can be a blank page of 100 million pixels, about a second each. Every
image decoded here now pays for its pixels first, as JBIG2 already did.
A JPEG paid nothing, and what one asks for doesn't grow with its length: 156
bytes declared 80 megapixels, 72 s for an 8 KB file. A JPEG pays for its
samples before it is decoded, in Optimize and in an export that has to decode
one for its mask.
Merging fragments paid for reading strips and not for the picture it built:
393 s for 1,154 bytes. It pays for the picture before allocating it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Filters were charged only for what they handed a JBIG2 decoder, once they
had handed it over: a chain that failed, or went past its image's limit,
cost its time and nothing else, and images that weren't JBIG2 weren't
charged at all. A 734 KB file of 200 images of Flate inside Flate took an
image export 111 s, and that grew with the file. It now takes 12 s.
Each byte read and each byte a stage decodes is now a step of the same
allowance, paid as it happens, so no stage runs past what is left. That
holds for every image read through ImageBudget, and a fax image pays for
its rows and its code. JBIG2 globals pay when they have to be read, and
kept ones don't. Optimize's two reads that went round the budget go
through it, and merging fragments shares it.
Optimize reads an image twice, to pack it and to resample it, so a refused
image is now counted once a run rather than once a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Without its mask a picture shows what the mask hides, so the page leaves it out, as export does. Any
other unreadable mask still leaves the picture, which is the renderer's tolerance for bad masks.
What an export may hold is a function of its own, tested down to the step and at the top of the range.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From a review of who reads a refusal.
The rule is to leave it out and say so, and it held on some paths only. An image whose JBIG2 mask is
refused no longer comes out opaque. The first image past the allowance reads as that and not as
corrupt. A page reports what it left out. The Office warning stopped blaming forms. Redaction refuses
the run, leaving the document as it was: removing a readable image whole would take what no mark
covers with it, unreported.
The 2 GiB an export holds is twice the document if that is more, so a large honest file loses nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images
made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes
them again.
Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its
page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before
anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes
it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for
each byte of the file, so a long scan is read whole. An image past it spends all that's left, and
later ones aren't read, globals included.
Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it
out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the
picture out and says the page was truncated. Redaction removes it whole, as any image it can't read.
Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts
afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73).
It's a thread-local, unbounded until a page sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Since the renderer now draws an image with /Decode [7 0 R 0], export disagreed with it. Its check of
which way round a stencil mask or a CMYK JPEG is compared the two numbers as given, and a reference
reads as none, so the file came out a negative of what the page draws.
Tests for the function arrays the previous commit's tuple fix also covers: a function's /Domain,
/Range, /Encode and /Decode are read a pair at a time too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
decode_jbig2 sits next to decode_ccitt, through hayro-jbig2 as Export ▸ Images
used it, and shares CCITT's row sink. The JBIG2 page has to be the image's size:
PdfCraft's renderer draws the whole page into the image (black box: a 4-row page
behind /Height 2 fills the image box with all four rows), so the rows of a taller
page aren't the image's. The decoder decodes a page whole, so the page is checked
before it is decoded: that size, within the caller's bound as rows of one bit, and
at most 2^28 pixels (MAX_JBIG2_PIXELS; the decoder keeps a bit a pixel, about 34 MB
there).
Document::decoded_image_within decodes a JBIG2 image as it does a fax image: the
filters before it undone first, then the decoder with the /JBIG2Globals of the
image's own /DecodeParms entry (/Crypt counts in the position; fax_params and the
globals now share last_filter_parms). Globals have a limit of their own,
MAX_JBIG2_GLOBALS (32 MiB, moved from Export), and globals that can't be read are an
error rather than ignored.
Export ▸ Images reads JBIG2 through it, so its own decoder, row sink and DecodeParms
reader go, and hayro-jbig2 leaves pdfcraft-create. JBIG2 behind another filter now
exports, and so copies (Copy Image uses the same function); a page taller than the
image, which Export cut to the image's rows, now skips the image, as one of another
width did. doc_export_all_images' description no longer lists JBIG2 inside other
filters as skipped.
The optimizer's monochrome step read images through decoded_image_within and
relied on JBIG2 failing there: it now leaves JBIG2 images alone explicitly, as
before (a resampled one would come out Flate under Retain), and the README and
parity note list them as not yet handled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Includes #4325#1952#4180#2529#4320#4805, preserving their original histories.
Keep paragraph alignment and hard/soft wrap metadata together; retain CCITT/downsampling/RGB ownership guards with bounded JBIG2 export and exact-copy symbol encoding.
Combined interaction regressions added. Validation pending.
The optimizer now writes JBIG2 images as a symbol dictionary and text
regions, and Export ▸ Images has to read them back. A page of three
repeated letters, symbol coded (smaller than the generic coding),
exports as its picture. The test is added to create.export-all-images'
parity evidence. Asked for in pre-push review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
An image's /JBIG2Globals were decoded with the 1 GiB any stream may decode
to, before the JBIG2 page's size is known, so the page's pixel limit didn't
bound them: a few kilobytes of Flate could make export allocate up to 1 GiB.
They now decode within MAX_GLOBALS (32 MiB); globals past it skip the image,
with the limit in the reason. The test covers one byte over and exactly the
limit, and the image's /DecodeParms direct, by reference and in an array.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Create PDF and the optimizer now store black-and-white scans as Group 4,
and redaction couldn't decode CCITT, so redacting one word on such a
scan removed the whole page image. A CCITT image (the last filter,
after any others) now decodes through Document::decoded_image and is
cleared in a Flate copy like any other image; one that can't be decoded
in full is still removed whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PDF Optimizer and Reduce File Size downsampled colour and grey images
but not black-and-white ones. Settings::mono is now MonoSettings {
downsample, target_ppi, above_ppi, compression }, 300 ppi above 450 ppi
and CCITT Group 4 by default (Monochrome Images ▸ Downsample to;
doc_optimize mono: { downsample, ppi, above_ppi, compression }).
A 1-bit image or image mask (no filter, general-purpose filters, or CCITT
fax after any of them) and eight-bit grey of only black and white are averaged by
area: each new pixel takes the sample covering more of it, a tie 0, so a
dark line half a new pixel wide stays. They are stored as Group 4, or as
1-bit Flate when that is smaller; with Retain a resampled image keeps its
kind of compression. Images with a soft mask keep their size.
CCITT decoding moves from Export ▸ Images into pdfcraft-filters
(decode_ccitt, CcittParams, through hayro-ccitt), and
pdfcraft_cos::Document::decoded_image reads a fax image's /DecodeParms
for it (undoing filters before CCITT, such as ASCII85Decode, with any
/DecodeParms given by reference made direct first), so export
and the optimizer share one reader, and Group 4 scans can be
downsampled. A 600 ppi text page went from 79 KB to 40 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
"Export ▸ Images reads byte-aligned Group 3 whichever way its fill
falls" read byte-aligned rows unaligned whenever /EndOfLine was set.
That broke byte-aligned Group 4 with /EndOfLine (refused); read Group 3
that says /EndOfLine but has no codes unaligned, and wrongly; and read
codes with the fill before them, but no /EndOfLine, on bytes: every row,
the wrong picture, nothing said. Now:
- Group 4, and rows not on bytes, read as the parameters say;
- Group 3 rows on bytes are read by the data: T.4 sends an end-of-line
code before the first row when it sends them at all, and no other code
starts with more than eight zeros. Data that starts with 11 zeros or
more (fill included) and a 1 has codes, and is read unaligned, where
the codes mark the rows whether the fill starts each code on a byte or
ends it on one; other data is read on bytes, whatever /EndOfLine says.
A reading that comes up short is reported rather than tried the other
way, where data cut short would read as other rows.
The Group 3 test exports nine pictures, stripes and eight of random
text-like runs (how a misreading comes out depends on the picture),
coded with T.4's terminating codes at /K 0 and 1: no codes, with
/EndOfLine and without; codes each starting on a byte or ending on one,
with /EndOfLine and without. Each exports as its picture, and each a
byte short is reported; at /K 1, codes with none before the first row
(not as T.4 sends them) export as their picture or are reported.
Byte-aligned Group 4 still reads with and without /EndOfLine. The
create.export-all-images parity note says what is still limited: such
codes may export wrong at /K 0, as the create README says. A rule tried
first, reading both ways and refusing an image that gave every row both
ways differently, failed on the stripes without codes (TIFF's Modified
Huffman layout): its unaligned reading gives every row too, wrongly. So
did one that read the other way when the first reading came up short: it
drew data cut short as other rows. cargo-mutants leaves no mutant of the
new lines alive.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
With /EncodedByteAlign and end-of-line codes, the fill may start each
code on a byte or end it on one (T.4's fill, as TIFF writes it); the
second was misread, and the image refused as short. With end-of-line
codes the decoder now reads unaligned, as the codes mark the rows either
way (probed black-box for /K 0, 1 and 2); codes laid out so under
/EndOfLine false are still misread. The Group 3 test covers both
layouts, /EncodedByteAlign given by reference, and a /BlackIs1 that
inverts the picture given by reference (with_direct_parms resolves both
before the decoder reads them).
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- Eight-bit black-and-white text under a noisy soft mask: the grey
settings win, as they shrink the mask far more than packing saves.
Counting the packed image without its mask passed every test before.
- The packing test checks the report's byte totals exactly.
- Export reads Group 3 with end-of-line codes, with rows on a byte
(EncodedByteAlign), with both, and mixed (/K 1, each row tagged
one-dimensional) with end-of-line codes and without. Decoding /K 1 as
one-dimensional, or ignoring EncodedByteAlign, now fails a test.
/EndOfLine is passed to the decoder, but no input here tells it apart:
the decoder accepts end-of-line codes either way, as ISO 32000-2
(Table 11) requires.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Redaction doesn't decode CCITT yet, so a mark over part of a picture
Create PDF stored as Group 4 removes the whole picture (nothing leaks);
before this PR such pictures were eight-bit Flate and had only the
covered pixels cleared. The create README and the image-to-PDF parity
note say so; #4384 adds CCITT to redaction. The workspace's comment on
hayro-ccitt says Export uses it, not only tests.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- An indexed image's lookup table given as a stream was decoded without
its /DecodeParms when they came by reference, so a PNG predictor's tag
bytes ran into the colours (red and blue came out [2,255,0] and black).
It decodes through with_direct_parms now, as the samples do. This was
already so on main; #4320 makes the same change.
- Tests for what this PR claims and no test held: Group 3 one-
dimensional data exports as its picture (mapping /K 0 to Group 4 gave
all white and passed); CCITT soft and stencil masks export as the same
masks in Flate (decoding them plainly dropped them and passed); a
Flate image's predictor given by reference reads as direct; and a
1-bit TIFF is stored as Group 4 (the TIFF test said "a 1-bit one" but
wrote eight bits; its comment now says what it writes).
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Stream::decoded reads only direct parameter dictionaries, so an image
behind [/FlateDecode /CCITTFaxDecode] whose Flate entry is a reference
lost its predictor, and the CCITT decoder got the predictor's bytes; a
plain Flate image with its predictor given by reference was misread the
same way. Export now decodes through
pdfcraft_cos::Document::with_direct_parms (#4304), as the optimizer
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
A fax image with /Filter [/ASCII85Decode /CCITTFaxDecode], say, was
skipped: only a lone CCITT filter went to the CCITT decoder. Now the
filters before it are undone first, and its parameters are the
/DecodeParms entry at its own place in the array (ISO 32000-2 §7.4.1),
not the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
A CCITT image whose /K, /Columns or /Rows is an indirect integer was read
with the defaults (K 0, 1728 columns), and one whose /Rows is below its
/Height was cut short and skipped. The numbers are now resolved, and the
image is decoded up to its /Height rows whatever /Rows says, still bounded
by the size the export checked (review of #4335).
The doc comment of the optimizer's bilevel step now says what Retain
leaves to the grey settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
A CCITT image whose /DecodeParms is a one-element array (common beside
/Filter [/CCITTFaxDecode]) was decoded with the default parameters: K 0
and 1728 columns. Wider scans were skipped, and a fax 1728 wide could be
misread. The parameters are now read from a dictionary or the one in an
array (ISO 32000-2 §7.4.2). The module doc no longer lists CCITT as
undecodable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
Group 4 images (now written by Create PDF and the optimizer) would
otherwise stop exporting: they were reported as "can't be exported
yet". A CCITT fax image (its only filter) is decoded through
hayro-ccitt with its /DecodeParms (K, Columns, Rows, EndOfBlock,
EndOfLine, EncodedByteAlign, BlackIs1), masks included; /Columns has to
be the image's width, so a hostile value can't size the buffers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
A scan saved as a 1-bit PNG decodes to eight-bit grey and was stored
that way, larger than the PNG it came from; a bilevel TIFF was stored at
one bit but with Flate. Now any image of only pure black and white
(1-bit, eight-bit grey, a two-colour palette, RGB without a profile) is
stored at one bit per pixel, as CCITT Group 4 or Flate, whichever is
smaller as stored. A grey profile stays; RGB with a profile keeps its
colours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
Export ▸ Images skips an image whose JBIG2 page has another width or
more pixels than an export takes; a JBIG2 stencil mask refused the same
way is left out and the image exported opaque, as JPX and CCITT masks
are, which the export test now checks. The README and parity note said
only "is skipped". And redaction doesn't decode JBIG2 or Group 4 yet,
so a mark over part of an image the optimizer stored either way removes
the whole image; the optimizer README and parity note now say so (#4384
adds Group 4). Found in pre-push review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The export checked an image's /Width × /Height against its 100-million-
pixel budget, and the JBIG2 page's width against /Width, but not the
page's height. The decoder decodes the whole page, rows past the
image's height too, and takes pages of up to about 2^28 pixels: a
20,000 × 4 image over a 20,000 × 10,000 JBIG2 page cost 200 million
pixels of decoding to export four rows. The page itself is now held to
the budget before it is decoded. Found by CodeRabbit on #4320.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Export ▸ Images and Save Image As decoded an image with
Stream::decoded, which reads /DecodeParms only when it is direct. An
image whose PNG predictor came by reference (the dictionary, an entry
of an array, or the numbers inside) kept its row tags in the samples,
and the picture came out shifted. The same was true of an indexed
image's lookup table given as a stream, and of a JBIG2 image's
/JBIG2Globals stream, which then couldn't be read, so the image was
skipped. All three now decode through Document::with_direct_parms
(#4304), as the optimizer does, and a test exports each. Found by
CodeRabbit on #4805; the image decode was already this way on main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
JBIG2 images, and JBIG2 stencil masks, were reported as "can't be exported
yet", so an image the Optimizer stores as JBIG2 would stop exporting. They
are now decoded with hayro-jbig2 (already in the tree through hayro), with
their /JBIG2Globals from a DecodeParms dictionary or array, as JBIG2Decode
gives them: black as 0 (ISO 32000-2 §7.4.7). Unreadable globals, a JBIG2
page of another width, or fewer rows than the image skip the image; rows
past its height are dropped, so the output is bounded by the checked size.
docs/pdf-spec-parity.md §7.4 no longer says there is no JBIG2 encoding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
LiveCycle Designer wrote BMPs whose file header gives bfOffBits = 0. image
(0.25) seeks to that offset as given, so the pixels were read from the file
header on and the picture was drawn shifted and wrapped around (#4937).
`pdfcraft_xfa::image::bmp_with_pixel_offset` recomputes the offset from the
info header (its size, BI_BITFIELDS masks in a 40-byte header, the palette)
when the declared one lies inside the headers and the computed one fits in
the file; anything else is passed through unchanged and not copied. XFA
pictures (size and decode) and `create::from_images` both go through it, so
pdfcraft-create gains a dependency on pdfcraft-xfa (L4 -> L3).
Fixes#4937
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqmHYx8FiWM5PcsoiErgp9
Conflicts: ROADMAP.md, docs/roadmap.md: both sides' status, progress and change-log lines; M10 row has the soft-mask sentence added to ours, M11 row from the earlier merge.
* chore: resolve SonarCloud maintainability issues
* fix(packaging): use verify_sha256 helper function
* style: cargo fmt
* fix: restore wildcard imports and fix method-reference type errors
Revert the glob-import expansions (they missed SomIndices and left unused
imports), revert char::is_ascii_digit where the iterator yields char by
value, and pass the content slice to prune_doomed_xobjects as &[Vec<u8>].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep XObject decodability check without binding content in interp
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Dylan Taylor <dylan@craftmaker.dev>
The JPEG decoder re-sorts and rescans every pending Extended XMP chunk after
each marker, so a file of many small chunks cost time quadratic in their
number before any pixel limit applied. Turning a photo upright now counts
the chunks first and leaves a JPEG with more than 1024 as it is.
A one-pixel GIF on a huge logical screen decodes to a canvas that size,
which the conversion then copied before splitting it into colour and alpha,
so a few bytes could hold over a gigabyte. Screens past 16 Mpx are refused
before decoding, and the decoded canvas is converted without a copy.
A TIFF's pages were each capped at 2 GiB but all kept in memory until
embedded, so many small directories sharing one compressed strip could ask
for many times that; embedding one image also decoded every page. The pages
of one TIFF are now refused past 4 GiB together, and image_xobject reads
only the first page.
Image export parsed an Indexed colour space's base before checking whether
that base was itself Indexed. A base that refers back to the same array
recursed until the stack overflowed. The base is now checked first; since
Indexed is the only colour space that recurses, this also ends any cycle.
Image export decoded a stream lookup table under the general 1 GiB
stream limit although a table holds at most 1024 bytes, and every image
sharing the palette decoded it again. The table now decodes within
64 KiB and a larger one is refused for that image.
ROADMAP.md, docs/roadmap.md: union of both sides' log lines and milestone rows, each line once.
parity/acrobat-features.toml, crates/xfdf/README.md: next's text, a superset of main's (#1917, #2686 plus #2688).
crates/edit/src/tests.rs: both sides' new tests (#2558 and #3084).
crates/ui-egui/src/editing.rs: the picked picture is cleared on every edit, and a new drawing commits its raster (#3149).
