* perf: reuse large buffers so a pinned glibc trim threshold costs no speed (#10479)
With MALLOC_TRIM_THRESHOLD_ set (#8743), glibc maps every allocation over
128 KiB fresh, so tiles, compositor bands and filter scratch were page-faulted
in on every call: viewport compose 3.6x slower, Gaussian Blur up to 1.8x,
.pcraft save 2.4x.
- photocraft_raster::scratch: a process-wide, capped pool of large Vec
buffers (u8/u16/f32/[f32; 4]). A miss drops the oldest buffers worth its
size down to a 256 MiB floor, so peak memory stays near glibc's defaults.
The engine releases the pool after any command that closes a document,
Headless::close does too, and an idle thread releases it after 2 s unused.
- Tiles, compose Buffer/masks/bands, algo Image and the blur paths, engine
adjustment/filter scratch, GPU tile upload and io import buffers go through
the pool.
- .pcraft save reuses one zstd FrameCompressor per worker (byte-identical).
- interactive_bench reports minor page faults per run.
36 MP, trim threshold set vs glibc defaults: viewport 5.6 vs 6.1 ms (was
22.7), Blur r4 445 vs 437 ms (was 852), Curves CPU proxy 35.0 vs 35.7 ms
(was 50), flatten 146 vs 147 ms (was 237), save .pcraft 251 vs 230 ms (was
545). RSS after closing a 5400x7200 document: 21 MB with trim set.
* fix: corpus tests build again; return remaining pooled buffers (#10479 review)
- Corpus-gated io tests and the Photoshop-oracle test moved `.px` out of a
`Buffer`, which now implements `Drop` (E0509 with `--features corpus`):
use `into_px()`.
- io: masks, the merged image, extra channels, smart-filter masks,
smart-object pixels and patterns give their `interleave` buffers back to
the scratch pool after writing them.
- compose: recycle the mask values of a pass-through group, `effect_shape`
and both `split_parts` callers.
- Styled Free Transform preview copies each band out instead of moving it,
so the band returns to the pool.
- interactive_bench: the fault count covers each whole run, untimed setup
included; say so.
* fix: recycle the masks of adjustment-atop and shape layers (#10479 review)
* fix: recycle effect hiding masks (#10479 review)
* fix: recycle knockout, CMYK and GPU shape masks; release the pool when Split/Merge Channels close a document (#10479 review)
* fix: release the pool when a closed document's cancelled workers exit; idle-release switch for tests; don't pool PSD planes (#10479 review)
* fix: release the pool for workers cancelled before their document closes; take the pool lock for budget and idle-release changes (#10479 review)
* fix: idle release rechecks activity under the lock; try_take_exact releases the pool and retries before failing (#10479 review)
* test: cover a successful try_take_exact miss; name the failed-reservation test for what it checks (#10479 review)
* fix: pool the final GPU effect shape buffer and return it after use (#10479 review)
* docs: describe scratch pool lifecycle regression invariants
* test: verify polling releases cancelled document scratch buffers
* perf: recycle remaining transform selection and PSD scratch buffers
* fix: free PSD gather bands before allocating channel planes
* test: assert exact scratch take reuses pooled storage
* Bound PSD import pixel storage before decoding
* Include ZIP inflater workspace in PSD preflight
* PSD import: budget unknown-mode layers, stop after a fatal layer error, keep best-effort for non-allocation limits (#8593 review)
Merged at the fix-up commit before its main merge (362947695d); the
conflicts with the batch are resolved here: psd_import.rs keeps both
sides' tests, tiff_layers.rs takes ImageData::new.
* chore: resolve SonarCloud maintainability issues
* fix: format imports and fix effect_entry argument type
* fix: restore test closure and wildcard import that broke compilation
* refactor: split composite_to_surface to reduce cognitive complexity
Opening a 2.14 GB PSD (2000×2000, 109 layers, 821 MB of embedded smart objects) peaked at
7.28 GB and took 6.8 s; it now peaks at 3.00 GB and takes 3.6 s (macOS peak memory footprint,
`cargo run --release -p photocraft-io --example psd_open_mem`).
- photocraft-psd: `PsdFile::from_source` parses from a `ReadAt` source (`FileSource`: pread /
seek_read in bounded reads). Channel data and the merged image become `Stored` ranges read
when decoded; the model equals `from_bytes`'s and writes back the same bytes (tests compare
the two on every generated file, including Lr16/Lr32 layer blocks and truncated files).
- `ChannelData::row_decoder` decodes any band of rows: RLE via its row table (a stored channel
reads only the band's bytes), raw by offset, ZIP inflated once.
- photocraft-io: `import_file` opens a path; PSD/PSB through `from_source`, other formats read
whole. Layers decode in bands of whole tile rows (~32 MB) into tiles: no whole-layer plane or
interleaved buffer. Global blocks move into the document instead of being copied; finding an
embedded smart object no longer copies every file in `lnk2`.
- photocraft-raster: `Surface::share_uniform_tiles` + `TileInterner`; the PSD import shares
single-value tiles (backgrounds, fill caches, masks) across layers.
- The background open job and the CLI/MCP (`automation::files::open`) use `import_file`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Limits::banded() lifted the 2^30 px cap, and the only failure left was a
tile allocation error. Linux overcommits, so the tiles were allocated
anyway and writing them had the out-of-memory killer end the app and
every open document. A valid all-black Deflate RGB8 TIFF declaring
100,000 x 100,000 px (30 GB of pixels) is a few tens of MB on disk and
passes the bomb guard.
- io memory.rs: memory::available() is MemAvailable plus free swap on
Linux, capped by the process's cgroup v2 memory.max (and those above
it) and memory.swap.max; None elsewhere (Windows refuses past its
commit limit, which the importer already reports). check_fits refuses
when the need plus a 512 MiB reserve doesn't fit.
- tiff_to_document checks the layer's tiles plus the working copies of
one group before allocating anything: "not enough memory to open this
100000 x 100000 px image: it needs about 41.7 GB and 7.8 GB is free".
- codecs TiffBands::group_bytes gives the largest group; a single-strip
file is one group (the whole image), which the check now counts.
- TiffBands puts Truncated before MorePages, as the whole-image decode.
- raster: try_write_interleaved reports a wrong length as
AllocationError::LengthMismatch { expected, got }, not SizeOverflow.
- Comments and docs no longer name another editor.
Tests: io a_tiff_larger_than_free_memory_is_refused (300,000^2 black
RGB8, Linux) was OOM-killed in a 4 GB cgroup without the check; memory
unit tests for the policy and the meminfo/cgroup parsing; codecs
a_cut_off_page_of_several_warns_in_the_whole_image_order and
group_bytes_is_the_largest_group_handed_over.
Workers that missed the shared-tile memo at the same time each built
and kept a copy, so the number of copies grew with the thread count and
translated_shares_tiles_rebuilt_from_shared_sources failed on machines
with 16 or more threads. Each worker now takes the tile stored first.
* skip redundant smart object resamples and per-pixel reads
* resample smart objects only when their re-render fails
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Rect::tile_count counts a rect's tiles in u64; try_take_tiles uses it (no usize overflow on 32-bit). Surface::translated reuses one rebuilt tile for equal source-tile windows. auto_slices merges vertically adjacent runs through a hash index instead of searching the output.
Keep scan coordinates and tile boundaries in i64 so valid reads at the final tile and large strides do not overflow. Bound each run by the remaining output and return the surface default beyond the i32 coordinate range. Add differential regressions across all sample depths and missing/present tiles.
Co-authored-by: Lucenx9 <185146821+Lucenx9@users.noreply.github.com>
The 117,504 x 22,464 px banner from #8456 failed with "2639609856 pixels
exceed max 1073741824": the decoder built the whole image in one buffer
and the importer copied it into tiles, so the decode was capped at 2^30 px.
- codecs: decode_ifd split into prepare + decode; new TiffBands decodes
~64 MiB groups of bands in parallel and hands each group, upright, to
the caller. Limits::banded(): 300,000 px a side on 64-bit, max_alloc per
group; 32-bit targets keep the whole-image check.
- raster: Surface::try_write_interleaved (fallible, tiles on all cores).
- io: import_tiff_page decodes flat pages through TiffBands straight into
the layer, converting pieces of rows on all cores; layered TIFFs keep
the whole-image path.
Release, 20 threads: 36 MP 220 -> 52 ms; 30,000^2 6.8 s / 6.43 GB peak ->
1.1 s / 3.86 GB; 35,000^2 and the 2.64 GP banner now open (1.5 s, 3.3 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XNm5ZNBFtLZ2cZNaPnYsXV
Move cached keys and compressed buffers between saves, and compare weak
allocation pointers without temporary strong-reference updates. Keep
payload budgets and archive bytes unchanged.
Allow fallible tile edits after a cached save. Move pixel storage when
only weak references remain; use a fallible copy for shared storage and
restore the original tile if the copy fails.
Cover cache key reuse, edits after saves at every depth, and weak/shared
tile ownership. Name the fixed revisions behind the existing timings.
- Replace whole-region Vec<f32> read_region/write_region in CutParts::new
with tile-parallel cut_piece + try_fill_surface/try_clear_surface,
sharing interior Arc<Tile> allocations in O(1) and activating the
solid_tile hole-fill fast path on Background and mask targets.
- Cache piece_bounds on CutParts so CutParts::place avoids scanning tile
content bounds on every drag frame.
- Parallelize composite_over across top tiles with Rayon and share fully
opaque top tiles by Arc<Tile> clone in O(1), preserving COW tile
identity for GPU canvas uploads.
- Parallelize Surface::translated across target tiles with Rayon and
prune cleared tiles in-place inside edit_tiles.
- Skip zero-offset floating preview generation on pointer-down in
move_ui::floating_doc.
- Exclude SnapKind::Selection when moving selected pixels with the Move
tool so small initial drags do not snap back to (0, 0).
- Keep navigator_plain active during Move/Brush/Patch drags on a GPU
canvas so navigator_gpu is not evicted into a synchronous full-document
CPU thumbnail composite on the first drag frame.
- Parallelize row normalization in compose::render_reduced_in_bands and
add a direct read_rgba_into fast path for unmasked Normal layers over
an empty backdrop.
A type render spends most of its time after the glyphs are drawn: composing each
colour's coverage over the premultiplied accumulator, dividing it back to straight
colour, and converting the tile bytes. All three are per pixel - no pixel reads a
slot another wrote - so they ran one after another on one core for no reason.
Measured on a 3.75 MP paragraph (4000 characters, 14 pt at 300 dpi, 1851 glyphs),
release, Windows 11, i5-10300H: rasterize 98.3-103.9 ms before, 60.3-61.8 ms after
(1.6-1.7x). Whole-render stage profile before: ink rect 4.1, draw 18.6-22.9, prefix
sum 7.0-8.1, composite 22.6-33.4, unpremultiply 11.2-14.0, write_region 23.8, prune
3.1 ms.
The glyph drawing, the winding prefix sum and the ink-rect pass stay on one thread:
the first two carry a running accumulator across pixels, and the band split that
would keep them bit-identical costs a second glyph traversal per band.
Native only (rayon, as the other crates): the web build keeps the serial loops.
* raster: a shifted surface copies in one pass per row without pre-filled tiles
Surface::translated's misaligned path allocated every target tile through
Tile::filled (a per-pixel default fill when the default is nonzero)
before overwriting nearly all of it, and addressed spans through
per-row closures. The path now writes each stored row directly: content
spans copy as single memcpys crossing source tiles as needed, vacated
edges fill from one prebuilt default row, and filled() builds its
default rows in bulk. Same output (the read-back oracle covers every
depth, sign and tile crossing), cheaper for nonzero defaults and one
less pass over each tile.
* raster: a missing source tile in the shifted content reads the default pixel
The misaligned path skipped absent tiles inside the content rect,
leaving zeros where the surface's default pixel belongs (a white mask
with sparse tiles turned black after a move). The span now fills from
the prebuilt default row. echelon's snippet is a regression test, and a
randomized oracle compares the whole path against a per-pixel reference
gather across formats, default pixels, sparse tiles and tile-boundary
crossing shifts.
* ci: retrigger wasm32 (the full workspace wasm check is clean locally; the job failed before writing a log and the run is stuck queued)
Surface::translated, under every layer move and the Move drag preview, allocated
every target tile that a moved source tile reached. For a move that isn't a whole
number of tiles, the transparent margins of a layer's edge tiles landed on target
tiles of their own, and those were stored though they hold only the default pixel
(256 KB each at 8-bit RGBA). #4414's shared uniform tiles were also stored when
their value was the default. Moving two 300 px blocks by (100, 56) left 10 of 18
stored tiles empty; nudging back and forth kept adding more.
A target that ends up holding only the default pixel is no longer stored, as
prune() would drop it. A copied tile is compared row by row with a row of the
default pixel after the copy (one memcmp per row; a tile with content stops at
its first row), and a shared uniform tile of the default value is skipped. Moves
by whole tiles still re-share the existing tiles unchanged.
Release, 24 cores, cargo xtask perf --quick (perf_scenarios, p50, HEAD vs this,
two runs each): move layer on 16-bit 15000x10000 + refresh 5.3 -> 3.9 ms; move
layer on 32-bit 20 layers + refresh 11.9 -> 5.9 ms; the other Move rows within
noise. translated() alone on 36 MP: full layer unchanged (27.0 -> 27.6 ms RGBA8);
40 sparse blocks 11.8 -> 10.6 ms RGBA8, 44.0 -> 39 ms RGBA16, 308 -> 190 tiles.
The new test covers RGBA8, RGBA16, RGBA32F and GRAY8 over a zero and a non-zero
default pixel, and both paths; it fails on the old code.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Surface::convert / try_convert and map_color converted tile by
coordinate, and the ICC path (color_cmds::convert_surface) staged and
converted every coordinate's tile. A tile shared across the grid, such
as a filled background (one Arc for all of it), therefore came out as
one fresh tile per coordinate: a white 2048×1024 background went from
1 to 32 tile allocations, a 24 MP one to 384.
Each distinct tile is now converted once and the result shared by
every coordinate that shared the original. Pixels are unchanged; tests
compare against a per-pixel conversion and check the sharing for
convert, try_convert, map_color and Image › Mode › CMYK.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
compose::bounds caches per-tile content bounds and drops entries of dead tiles when it holds
more than 65,536 of them. Once more than 65,536 tiles were alive (a large document plus its
history), every call (every render tile, every GPU plan) walked the whole map again and kept
it all. The sweep now runs past a high-water mark: twice the entries that survived the last
sweep, and at least 65,536.
The per-tile scan moves to raster as Tile::content_bounds, used by both
Surface::content_bounds and the cache instead of two copies of the loop; an empty pixel
yields no content instead of a chunks_exact(0) panic.
Claude-Session: https://claude.ai/code/session_01QUda3K786uop8kvEQoUjSf
Co-authored-by: Claude <noreply@anthropic.com>
Surface::translated copied every target tile of a move that isn't a whole number of tiles,
so a solid fill (one shared tile) became one unique tile per coordinate after its first
nudge: each later move, undo and refresh then rescanned and re-uploaded all of them, and
history kept them all.
A target tile whose window reads only tiles holding one single pixel value (each source
tile is scanned once per call; one tile is kept per value) now shares that tile. Fully
covered targets also skip the default-pixel pre-fill they are about to overwrite.
12 MP RGBA fill nudged by +-(10, 7) twenty times (release, 4-core container): unique tiles
192 -> 76; RGBA16 move p50 55 -> 21 ms; RGBA8 unchanged within noise. Pixels are unchanged
(the test compares every depth with the copied pixels, and checks that a shared but
non-uniform tile is still copied).
Claude-Session: https://claude.ai/code/session_01QUda3K786uop8kvEQoUjSf
Co-authored-by: Claude <noreply@anthropic.com>
PhotoCraft treats 32-bit documents as linear light (Transfer::for_document,
Exposure, Photo Filter, PSD import), but Image > Mode > 32 Bits/Channel only
rescaled the samples, so a converted 8-bit grey 128 stayed 0.502. Photoshop
stores 0.2159 (measured in Photoshop 27.11: 32-bit float TIFF of an sRGB
chart, every sample = the sRGB-to-linear value). The conversion now decodes
colour samples through the document's tone curve after widening to 32 bits,
and encodes them before narrowing (so no linear value is quantised to 8 or
16 bits). Alpha, masks and alpha channels keep their values. RGB and
Grayscale only (the 32-bit modes).
With it, Photoshop's 32-bit results match exactly for the conversion itself,
Levels, Black & White (default), Brightness/Contrast and Colorize (were 40-80
levels off in sRGB terms).
Adds Surface::map_color (colour samples only, including the default pixel)
and makes Transfer::decode/encode public.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Clear (and the background-colour fill it does on a Background layer)
read the whole selection bounds into one float buffer, looked the
selection up in its tile map once per pixel and wrote every tile back,
single-threaded. On an 8000x6000 layer that took about 2 s (3 s on the
Background). It now edits the encoded tiles in parallel, drops fully
cleared tiles, shares one solid tile for fully covered opaque fills and
copies a tile only when a pixel in it changes. Results match the old
pass exactly; 20-110 ms on the same documents.
Fixes#3463
Co-authored-by: Fabio Lussi <198260595+itrycodinganditsbad@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Make large pixel operations allocation-fallible
* Warp: drop the fixed 400 MP cap; fallible reservations already cover large transforms
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Adapt #2530 to main: fallible steep-Distort warp, Result from mask paste in float moves; regenerate scorecard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(compose): Preserve native CMYK fills through PSD round trips
Share depth-quantized ink samples between rendering and PSD caches.
Use the document profile for RGB fill colours and preserve valid imported
caches. Uncached CMYK fills use CPU rendering.
Cover all sample depths, gradient styles and tiled renders. Fix the real
16-bit gradient round trip and raise the psd-tools floor from 307 to 308.
* perf(raster): Reuse exact 8-bit CMYK conversions within rows
Cache the previous ink bytes and ICC result per row, with alpha decoded
independently. Keep the 16/32-bit conversion loops unchanged.
Matched 1 MP reads improve by 77% for smooth ramps, 33% with dithering
and 10% for random colours. Bitwise regression coverage includes all
depths, alpha and tile boundaries.
- A layer about the canvas's size keeps its whole effect region even where it overhangs the
edge, so the region keeps its size while the layer is dragged and the GPU moves its effect
maps with it instead of rebuilding the shape, distance fields and every effect each frame.
- Surface::translated moves a surface by copying encoded tile bytes tile to tile (no float
round trip, no full-size intermediate; whole-tile moves share the tiles): 2.3x faster than
to_interleaved + write_interleaved. Used by layer.translate, Move tool drags and
algo::resample::translate_surface.
- perf_scenarios: Move tool drag frame rows (styled layer, full-canvas layer with and without
a drop shadow).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Surface::fill_rect materialized every tile it touched, so a new document's
background cost one 256x256 tile per tile slot: ~360 GB for the 300000x300000
size the command accepts. Fully covered tiles now share one Arc'd tile, so a
solid fill costs one tile plus the tile index; edge tiles are written as
before, and copy-on-write splits a shared tile off when it is edited.
* Test: retry the GPU startup sentinel check while a forked child briefly shares the lock
monitor_profile tests spawn osascript in the same binary; between fork and exec the child holds
a copy of the marker file descriptor, so flock reports WouldBlock and begin() returns Busy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Background jobs: engine job API, progress and cancel (part of #210) (#289)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- Real crash: build_engine_data took its template from the PSD's text
engine data and did expect("EngineDict"), so a text layer whose engine
data lacked EngineDict (or wasn't a dictionary) crashed PSD export.
The template is now repaired; regression test
engine_data_template_without_engine_dict.
- color: the built-in CMYK<->sRGB transforms fall back to the naive
formulas instead of expect if they ever fail to link.
- doc: insert_above falls back to the top of the root instead of
expect("valid path"); next_layer_name searches a bounded range.
- paint: Mips::new loops with while-let instead of last().expect.
- All eight crates deny unwrap/expect/panic/unimplemented/todo/unreachable.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Rust 1.99's clippy adds chunks_exact_to_as_chunks and some_filter, which
broke the Clippy step on every CI runner (it uses the latest stable). All
sites are mechanical: as_chunks drops the remainder exactly like
chunks_exact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>