* M10: Windows prints a large image whole: drawn as tiles under Windows.Data.Pdf's cap (#5131)
Windows prints each page through Windows.Data.Pdf, which draws any image
of more than 2^22 pixels from a copy shrunk to that size. With the page
at the printer's resolution (craft-libs#396, #5023), a 6000 x 6000 scan
still printed as 2048 x 2048, its one-pixel detail gone.
pdfcraft_optimize::tile_large_images replaces each such image XObject
with a form XObject of the same object number that draws it as tiles of
at most 2000 pixels a side, so whatever drew the image draws the tiles.
Samples are copied as stored (any bit depth and colour space, decode
array, interpolation, colour key); a grey or RGB JPEG is decoded and
stored with Flate; a soft or stencil mask of the image's size is cut
into the same tiles. Tiles overlap by a pixel against drivers that
soften image edges, except under a soft mask, where the overlap would
blend see-through pixels twice. JPEG 2000, CMYK or ambiguous JPEGs,
masks of another size, more than 80 million pixels, short data and
images past the decode allowance are left whole.
pdfcraft_engine::spooler::spooler_copy applies it to the copy that goes
to the spooler on Windows, from the Print dialog's worker and doc_print;
elsewhere, or when the copy can't be read or written, the bytes go as
they are. The saved print-ready PDF is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* M10: a test that the spooler gets the tiles on Windows; tiles are up to 2001 px (#5131)
spooler_copy was only tested with prints that pass through unchanged, so
one that skipped tiling on Windows would have passed. A test now checks
that on Windows it sends the tiled copy and elsewhere the bytes as they
are. The optimize README said tiles have at most 2000 pixels a side;
with the overlap they have 2001.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* M10: the tiles' samples are StoredSamples, apart from the crate's Samples (#5131)
main has since added a public `Samples` (8-bit grey or RGB, for tools
that repaint an image), so two different things in one crate shared a
name. The tiles keep an image's samples as stored, in its own colour
space and depth, so each tile redraws exactly what it replaces; the
type now says so in its name and doc comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: austin62jake <340374338+austin62jake@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
The image pass visits the images pages draw by name. A picture's stencil
`/Mask` and its `/SMask` are images of their own that pages don't name, so
the monochrome step never reached them (the colour and grey settings
resample an eight-bit soft mask with its picture, but store no mask as
Group 4 or JBIG2). In a mixed-raster scan, a colour layer drawn through a
one-bit mask that holds the text, most of the black-and-white data stayed
as it was written (Flate, say).
After the pictures, each picture's stencil mask and one-bit soft mask, once
however many pictures name it, now goes through the monochrome step at its
size (`decide_mask`): stored again as the compression asks (Group 4, JBIG2)
when that is smaller, its bits as they were, so it masks as it did. It
isn't resampled (its resolution isn't measured) and shares no JBIG2
dictionary (only images named directly do, which redaction can store again).
A stencil pages also draw by name stays one of the pictures; a soft mask they
draw is still stored as a mask. A soft mask of
more bits stays, to be resampled with its picture, which wants it at eight.
The report counts masks as images; Retain leaves them as they are.
The pictures and the masks go through one loop (`work_through`), so masks
are worked out on threads and from the run's allowance as pictures are.
Tests: `a_pictures_one_bit_masks_are_stored_again`,
`a_pictures_mask_stored_as_jbig2_shares_no_dictionary` and, through
`doc_optimize`, `doc_optimize_stores_a_pictures_one_bit_soft_mask_as_ccitt_group_4`.
`a_mask_past_the_runs_allowance_stays_and_is_counted` holds a mask past
the run's allowance to staying, counted as refused, and the progress to
counting masks with pictures.
`a_picture_with_a_stencil_mask_is_downsampled_under_its_mask` now checks
the mask's size and bits rather than its bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Merge origin/main. Conflicts: docs/changelog headers keep both histories;
edit/run.rs keeps EnhanceScan beside main's run_edit_page_image; optimize
images.rs keeps main's scaled() and the crate-visible jpeg(); the German
strings in ui-egui/src/i18n/de.tsv go with that file, which main moved to
craft-i18n. decode_samples now takes main's ImageBudget; a scan clean-up
run shares one sized by the file.
Review (CodeRabbit, scan_clean.rs): a page whose own images and a
scan-holding Form only together cover it now cleans the images and reports
the Form; a page whose images alone cover it reports its Form too. New
regression fails on the old classification.
Every colour operator in the pages' content and the Form XObjects it draws is
rewritten in the target space by the standard device formulas, source colours
read through the space in force (edit/src/colour.rs: device, ICCBased, CalRGB/
CalGray, Lab, Indexed, Separation/DeviceN through a Type 2 tint). Preserve black,
pictures too (optimize::recolour_image: 8-bit gray/RGB re-encoded, JPEG stays
JPEG, CMYK lossless), comments' and fields' appearances optionally.
Edit::ConvertColours (one undo step; refuses when nothing changes, so
Session::count_colours counts first), the Convert colours dialog, command
prepress.convert_colors, tool doc_convert_colours, rows in every language
(RGB and CMYK allowed unchanged in the Ukrainian and Kazakh checks), parity
print.convert-colors shipped.
Carries the review fixes:
- ATTRIBUTION.toml: the language files' checksums after the new rows
- Convert colours: keep RGB and CMYK as-is in the Czech and Turkish menu-label checks
- Convert colours: CMYK black start, shared objects copied, streams kept apart, guarded count
- Convert colours tests: drop an unused helper
- Format with rustfmt
- i18n(ui): carry this PR's new wordings in the catalogues that must cover them
- Parity: evidence paths follow the tests moved into tests/it
- fix(colours): comment permission for appearances; page range per document
- fix(convert-colours): say what it is not, and what it leaves behind
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Storing LZW streams with Flate decoded image streams outside the allowance,
with a limit for each stream and no total, and nothing asked whether to go
on: a 593 KB file of 100 images took 113 s with none refused, and images
refused by the image pass were still rewritten. The pass now pays for what
it decodes and for storing and checking it, at 8 steps a byte since Flate
takes up to 78 ns a byte, and streams past that stay LZW.
An image is counted as refused only when it stayed as it was: one packed
before the allowance ran out on its second read was rewritten. doc_reduce
reports images_refused as doc_optimize does, and the app's action says so
in its log. A refused image no longer reads as an unsupported filter:
FilterError::Spent says what happened.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Filters were charged only for what they handed a JBIG2 decoder, once they
had handed it over: a chain that failed, or went past its image's limit,
cost its time and nothing else, and images that weren't JBIG2 weren't
charged at all. A 734 KB file of 200 images of Flate inside Flate took an
image export 111 s, and that grew with the file. It now takes 12 s.
Each byte read and each byte a stage decodes is now a step of the same
allowance, paid as it happens, so no stage runs past what is left. That
holds for every image read through ImageBudget, and a fax image pays for
its rows and its code. JBIG2 globals pay when they have to be read, and
kept ones don't. Optimize's two reads that went round the budget go
through it, and merging fragments shares it.
Optimize reads an image twice, to pack it and to resample it, so a refused
image is now counted once a run rather than once a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images
made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes
them again.
Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its
page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before
anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes
it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for
each byte of the file, so a long scan is read whole. An image past it spends all that's left, and
later ones aren't read, globals included.
Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it
out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the
picture out and says the page was truncated. Redaction removes it whole, as any image it can't read.
Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts
afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73).
It's a thread-local, unbounded until a page sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
#5461 made Document::decoded_image read JBIG2, page exactly the image's size, and
kept the optimizer's monochrome step off it until it could weigh JBIG2 again. Now
one_bit reads JBIG2 last after any filters, within the raster's allowance, and mono
treats it as it does any black-and-white image:
- above the threshold it is averaged, and stored as the compression asks, the
smallest of what that may store; under Retain it is stored as JBIG2 again, its kind
of compression kept;
- not resampled, it is stored again as the compression asks when that is smaller: a
generic region becomes symbols with JBIG2 asked for, and JBIG2 behind ASCIIHex is
stored again alone (as Group 4 where that is smaller);
- a JBIG2 image stored again refers to no /JBIG2Globals; one whose page isn't its
size isn't read, and stays.
jbig2_images_are_left_as_they_are becomes jbig2_scans_are_averaged_and_stored_again:
a JBIG2 scan at 600 ppi, alone, behind ASCIIHex and with globals, averages to the
same samples as its Flate copy under each compression (Retain keeping JBIG2); at 300
ppi a generic region of rendered text stays under Group 4 (larger) and Retain and
becomes symbols under JBIG2 (under half the size); behind ASCIIHex it is stored again
alone, as Group 4 where the hex is spaced out; a taller page stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5426 left indexed images at their resolution rather than guess which sample is
dark. The palette says: dark_is_one now maps samples 0 and 1 through the decode array
to palette entries (rounded and clipped to hival, ISO 32000-2 §8.9.5.2) and compares
how light those entries are (`lightness`: the grey, or Rec. 601 luma over DeviceRGB).
Only DeviceGray and DeviceRGB bases count, the table a string or a stream (within
4 KiB), directly or by reference: there the palette's values are the colour, and
under /DefaultGray or /DefaultRGB no polarity is asked for. Other bases (CMYK,
calibrated, ICC-based, a resource name), two entries alike, an entry the table
doesn't hold or a hival past 255 still say nothing; separations and DeviceN still
aren't averaged, their tint transforms unread. #5426's tone guard applies as to any
image.
#5426's indexed thin-rule test now checks what it was for: with the palette either
way round and the decode array either way, the averaged image holds the averaged
DeviceGray one's samples (flipped where its samples are) and renders the same. A unit
test covers each base, both orders, an inverting decode array, the weights of luma,
the table by reference and in a stream, and every refusal; the older polarity test's
doc says what is known now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The integration in #5426 made downsampling conservative: dark_is_one answers only for
DeviceGray and image masks (a decode array of two different numbers or none), and not
under /DefaultGray or /DefaultRGB, and a square of four new pixels that each cover both
samples keeps the image at its resolution. The README, the crate docs, mono()'s comment
and the optimize.downsample-images parity note still said separations and indexed
images were averaged, a tie going to full ink or entry 0.
Measured on main: a 600 ppi page of the ragged-text fixture is no longer averaged
(it is with the square guard switched off), so the docs say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Includes #4325#1952#4180#2529#4320#4805, preserving their original histories.
Keep paragraph alignment and hard/soft wrap metadata together; retain CCITT/downsampling/RGB ownership guards with bounded JBIG2 export and exact-copy symbol encoding.
Combined interaction regressions added. Validation pending.
pdfcraft_filters::encode_jbig2 stores each distinct mark of a 1-bit
image (8-connected black pixels, at most 256 a side) once, as a symbol
in a symbol dictionary, and places its exact copies with an immediate
lossless text region that refers to it. Larger marks, and in a second
plan marks that occur once, go to a generic region over their box. It
keeps whichever of that and the generic coding (encode_jbig2_generic,
unchanged) is smaller. Only exact copies share a symbol, so the stream
decodes to the same bits.
Written from the JBIG committee's ISO/IEC 14492 Final Committee Draft:
6.4 and 6.5 (text region and symbol dictionary decoding), 7.2, 7.4.2
and 7.4.3 (segment headers and syntax) and Annex A (the integer coders,
checked against its worked example and Table A.1).
On six pages of rendered 300 dpi text the output is 28% of the generic
coding's size and 17% of Group 4's, at about 100 ms a page; hayro-jbig2
and Poppler (pdfimages) both decode it to the original bits. Scans,
whose copies of a letter are seldom identical, gain little: that needs
refinement coding. A dictionary shared between pages (/JBIG2Globals)
is a further step.
The optimizer's Monochrome Images > JBIG2 (lossless) and doc_optimize
mono jbig2 use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The optimize crate doc and MonoSettings named only 1-bit, CCITT and
eight-bit grey images; doc_optimize's mono said Group 4 only; the M11
milestone row stopped at Group 4 compression. They now include
downsampling (halftones aside), Group 3 to Group 4, 1-bit Flate, and
black-and-white RGB and indexed images that use only palette entries 0
and 1, packed to one bit. That last isn't "two-entry", as the roadmap and
the downsampling parity note had it: an indexed image packs whatever its
palette's size, when its samples are 0 and 1. The mono-compression note
says byte-aligned Group 3 with end-of-line codes but none before the
first row is stored again as PdfCraft draws it, which at /K 0 may be
other rows than were coded.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- The optimize README, the crate doc and both parity notes say a
halftone or dither, and an image wider than a fax row, isn't averaged
down.
- The README's "Left alone" list is the colour and grey settings'; image
masks, decode arrays and CCITT go to Monochrome Images now. Its
monochrome "Not yet" names an image's own /SMask or /Mask image again
(only images drawn on a page are visited), as the parity note does;
the parity note's row limit is Group 4's.
- docs/architecture.md says CCITT decodes through pdfcraft-filters for
export, redaction and the optimizer, not only the renderer.
- docs/target-app-parity.md no longer lists monochrome downsampling as
missing.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
The PDF Optimizer and Reduce File Size downsampled colour and grey images
but not black-and-white ones. Settings::mono is now MonoSettings {
downsample, target_ppi, above_ppi, compression }, 300 ppi above 450 ppi
and CCITT Group 4 by default (Monochrome Images ▸ Downsample to;
doc_optimize mono: { downsample, ppi, above_ppi, compression }).
A 1-bit image or image mask (no filter, general-purpose filters, or CCITT
fax after any of them) and eight-bit grey of only black and white are averaged by
area: each new pixel takes the sample covering more of it, a tie 0, so a
dark line half a new pixel wide stays. They are stored as Group 4, or as
1-bit Flate when that is smaller; with Retain a resampled image keeps its
kind of compression. Images with a soft mask keep their size.
CCITT decoding moves from Export ▸ Images into pdfcraft-filters
(decode_ccitt, CcittParams, through hayro-ccitt), and
pdfcraft_cos::Document::decoded_image reads a fax image's /DecodeParms
for it (undoing filters before CCITT, such as ASCII85Decode, with any
/DecodeParms given by reference made direct first), so export
and the optimizer share one reader, and Group 4 scans can be
downsampled. A 600 ppi text page went from 79 KB to 40 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_jbig2_generic writes 1-bit rows as a JBIG2
stream for JBIG2Decode (embedded organisation: a page information
segment and an immediate lossless generic region, no globals): template
0 with its nominal AT pixels, typical prediction, and the MQ arithmetic
coder (mq.rs), written from the JBIG committee's ISO/IEC 14492 Final
Committee Draft. The coder reproduces its Annex H.2 test sequence byte
for byte; output round-trips through hayro-jbig2, and Poppler renders
it the same as the original.
The optimizer's Monochrome Images gets JBIG2 (lossless) beside CCITT
Group 4 (still the default), doc_optimize mono jbig2; a document below
PDF 1.4 gets /Version /1.4 in its catalog. On 300 dpi text JBIG2 is
about a third smaller than Group 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
pdfcraft_filters::encode_g4 encodes packed 1-bit rows as Group 4 (ITU-T
T.6, /K -1) through the fax crate (MIT, already in the tree via tiff),
with rows capped at 32,767 samples so its 16-bit positions can't
overflow; round trips are checked against hayro-ccitt.
The optimizer (Settings::mono, CCITT G4 by default, so Reduce File Size
too) re-encodes a 1-bit image of one colour component, or an image
mask, stored with no filter or general-purpose ones, when that is
smaller as stored. The samples decode to the same bits, so colour
space, decode array and masks stay. PDF Optimizer gets a Monochrome
Images row and doc_optimize a mono option. optimize.mono-compression
goes from planned to partial (JBIG2 and monochrome downsampling not
yet).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfMGHQNgdq5yf5tKZHGR3z
Scan & OCR ▸ Enhance scanned file (Acrobat's Enhance ▸ Scanned document)
now cleans up scanned pages: background removal (the camera clean-up's
level 0-100), edge shadow removal (book gutters, copier borders) and
despeckle, each its own switch.
The camera tool renders the page and replaces it with a picture, which
would drop a scan's invisible OCR text, comments and links. Instead each
scan image (pictures covering at least half the page, each at least a
tenth) is decoded, cleaned and written back into its own image object at
the same size and colour space: a JPEG stays a JPEG (quality 90), other
grey/RGB images are stored losslessly, a 1-bit image stays 1-bit with its
decode array, and every other dictionary entry (SMask, Interpolate, ...)
is kept. An image several pages share is cleaned once. What can't be
cleaned (CMYK, indexed, JPEG 2000, JBIG2, CCITT, colour-key masks, broken
data, oversized images) is reported, never skipped silently; pages that
are no scan are listed too.
The engine works as a job like Optimize (scan_clean_job, run on a worker
with progress and cancel, apply_scan_clean refuses a result for an older
document state) and applies one undoable step, "Enhance scanned file".
pdfcraft-optimize exposes decode_samples/encode_samples for in-place
pixel edits. camera_clean gains clean_samples for grey and RGB samples
with separate switches and an edge-shadow pass; the camera tool's output
is unchanged. page_enhance_scan drives it headlessly.
Not yet an option of Recognize text. Refs #4119.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When an action to another document was removed and its /Next list
started with a null or a reference to something other than an action,
that entry became the head of the chain, and the actions after it, such
as a web link, were dropped with it. Leading entries that are not
action dictionaries are now skipped. The Settings::discard_external_refs
doc comment lists every action type the option removes.
PDF Optimizer > Discard Objects gets "Discard embedded search index", off
by default. It removes the entries of the catalog's /PieceInfo whose
application name contains "index" in any letter case, such as
IBM-ODIndexes, and removes /PieceInfo when nothing else is left in it.
Other application data and the pages' own /PieceInfo stay. An indirect
/PieceInfo is not edited: the catalog gets a direct copy without those
entries, and other objects that refer to the original keep it. The full
save leaves out the objects that only the removed entries refer to.
doc_optimize takes discard_search_index and reports search_indexes.
PDF Optimizer > Discard User Data gets "Discard external cross-references",
off by default. It takes GoToR, GoToE, Launch and ImportData actions, and
Thread actions with an /F, out of the actions of link annotations and
bookmarks, and the actions chained after one through /Next take its place
in order. Other annotations, such as form fields, keep their actions. A
link left with no action and no /Dest is removed from its page; when the
link is an indirect object, that object is rewritten without the action
for anything else that refers to it. A bookmark keeps its title, place and
children, including a bookmark stored directly inside another one. Links
within the document and web links stay. An indirect action that changes
is written as a new indirect object, except the first action left at the
head of an /A, which is copied into the /A directly when other actions
follow it. Objects that use the old action keep it. An /A with more than
100,000 values to check, or one that opens another document and needs
more than 100,000 values to rewrite, stops the optimization with
OptimizeError::ExternalRefLimit, and so does an outline of more than
100,000 bookmarks.
doc_optimize takes discard_external_refs and reports external_links and
external_bookmarks.
ROADMAP.md: kept both progress lines.
crates/optimize/README.md: PR's decoder sentence plus the target-version section from main.
Cargo.lock: auto-merged.
Conflicts: ATTRIBUTION.toml: took main's side (hashes recomputed after the merges). crates/optimize/src/lib.rs, crates/automation/src/lib.rs, tools.rs: Settings and tool schema keep object_streams (#3227) and lzw_to_flate (#4181); doc_optimize description names both options. crates/optimize/tests/optimize.rs: both sides' imports and tests kept.
Optimize decoded DCT images with the generic JPEG decoder, which converts
three components from YCbCr to RGB whenever the JPEG itself doesn't say
otherwise, and then dropped /DecodeParms. An RGB JPEG stored untransformed
(/ColorTransform 0 and no Adobe marker) therefore came out in other colours
when recompressed, resampled or merged from strips.
The samples are now read with the colour transform ISO 32000-2 §7.4.8
gives them: an Adobe APP14 marker's flag, else /ColorTransform (default 1).
Untransformed samples are taken as stored, and the result is written as a
plain YCbCr JPEG (or Flate), which needs no /DecodeParms. When the marker
and /ColorTransform disagree, a value is out of range, the parameters are
not paired with the filter as §7.4 pairs them, or the decoder reads the
components otherwise (named R, G and B), the image is left as it is.
The tests draw each image with PdfCraft's renderer before and after
Optimize: RGB JPEGs with and without an Adobe marker under each
/ColorTransform, CMYK JPEGs (still left alone) and JPEG strips merged into
one picture.
Part of #1201.
FredApps
Grzegorz MartensandClaude Opus 5.5
bflatastic
dotCooCoo
Osvaldo Duran MelendezandClaude Sonnet 5.5
quincy-jc
FredApps
Osvaldo DuranandClaude Opus 5.5
huppifluppandClaude Opus 5.5
aiongg
Marco Arnulfo
jonathanwxh-cell
Guillem Hermida