Port behavior and regression coverage into current modules; preserve current upstream behavior.
Integrate bounded Sony SR2 metadata decoding and ILCE-6000 image area; preserve newer bit-depth-aware camera profile application and explicit file crop precedence. Keep current parity counts and original metadata/crop tests.
`for_bands_grouped` always split the bands into `min(workers, jobs)` static
chunks, so on the default develop (jobs == workers) a worker held up on the
shared pool kept a fixed share of the bands waiting, where main's
`chunks_mut_limited` runs one rayon task per band. Now, when every worker may
hold a scratch, each band is its own task and borrows a scratch from a pool
(`par::chunks_mut_pooled`, at most one per worker, dropped as soon as no
waiting band can use it); the grouping stays for memory-limited jobs.
The fused bilinear/MHC, X-Trans and LinearRaw paths ran one band per worker
with no memory cap: a 262144 x 455 Bayer frame with bilinear needed 2.92 GB
at 15+ workers and failed with LimitExceeded under the default 2 GiB.
`working_memory` now caps those bands like AHD's and the fused paths use the
returned jobs. The fused LinearRaw and X-Trans band buffers are `band.min(h)`
rows, as charged (CodeRabbit, develop.rs:414).
Tests: a_slow_band_does_not_hold_up_the_bands_after_it and
fused_bands_at_once_are_capped_by_the_memory_limit fail without the fix;
for_bands_keeps_at_most_jobs_scratches_alive, chunks_mut_pooled_makes_one_state_per_worker_at_most,
multi-band and short X-Trans/LinearRaw fused-vs-full cases.
Co-authored-by: Valentin Klein <271991477+vangel76@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main reverted the #9455 merge, so a plain merge would drop the band-by-band
develop; keep this PR's demosaic.rs, develop.rs and development_limits.rs and
take all other upstream changes.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Main reverted the earlier #9455 merge, which also removed the AHD changes
that came in with it; keep this PR's AHD code and add clampf plus the
NaN-plane test from main.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Keep the AHD scratch/band changes; take clampf and the NaN-plane test from main
(main reverted the earlier #9455 merge, so the AHD changes stay on this branch).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* chore: resolve SonarCloud maintainability issues
* fix: format imports and fix effect_entry argument type
* fix: restore test closure and wildcard import that broke compilation
* refactor: split composite_to_surface to reduce cognitive complexity
The bands were split into four groups per worker, clamped to the number
of bands. A 24 MP raw has 126 bands, so with 32 workers every group held
one band and every band allocated and zeroed a fresh 16-plane scratch,
which cost more than the reuse saved. The bands are now split into one
group per worker, so the scratch is allocated once per worker whatever
the thread count. Pixels are unchanged.
Adds a test that runs 35 bands in groups of 9 on a 4-thread pool and
compares the result with the reference bit for bit, so the band index of
later groups and a scratch reused for several bands are checked.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each AHD worker allocated scratch for a full 38-row band while
develop's memory check budgets h.min(32) + 6 rows, so short images
allocated more than was approved: a 262144 x 1 strip was budgeted
~106 MB and allocated ~580 MB (5.4x). The scratch is now sized from
the same row count as the budget, and the budget also counts the two
u32 vertical-sum rows, so budget == allocation (~109 MB there).
Pixels are unchanged.
Tests: ahd_scratch_fits_its_budget (budget equals the real scratch for
h = 1..5000), ahd_matches_reference_with_reused_scratch (1100 rows,
jobs 1 and 2, all four phases: one scratch serves many bands).
Co-authored-by: vangel76 <271991477+vangel76@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 630e32c4238df452d9d5e70357bdb968045ab8c3)
Every lossless-JPEG round-trip wrote a single DHT (id 0) shared by all
components, so the new per-component table mapping (comp_table, the t0/t1
split of the two-component loop) went untested, though real CR2 files
declare one table per component. testgen gains lj92_encode_tables (one DHT
per table id, Td chosen per component, code lengths rotated per table so
the tables differ); lj92_encode delegates to it with output unchanged.
New tests cover 2- and 4-component frames with distinct and mixed tables,
predictors 1, 4, 6 and 7, restart intervals and 16-bit 0/65535 extremes.
They now catch swapping t0/t1, pushing 0 into comp_table, and resolving
every component to tables[0]; they pass on main's decoder as well.
Co-authored-by: vangel76 <271991477+vangel76@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- An opcode list may hold up to 4096 opcodes, and every MapPolynomial is checked for every
sample, so a crafted DNG with thousands of whole-image polynomials (a few hundred KB) hung the
open for minutes. Keep at most 64 per list; the rest are named as not applied.
- A DOUBLE WhiteLevel past f32::MAX became an infinite white, which blacked out that channel.
A non-finite level now falls back to the default, as a zero one does.
- A reduced-resolution LinearRaw IFD (NewSubFileType 1, three samples, JPEG XL) was a JPEG XL
preview candidate, and its undeveloped samples would show as an sRGB picture. Previews now
need PhotometricInterpretation RGB or YCbCr.
- README: floating-point JPEG XL raw data and OpcodeList2 MapPolynomial are supported.
Co-authored-by: SBaksa <18648812+SBaksa@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The full-size part of the tone stage cost as much as the whole AHD
development at 45 MP. Everything changed here keeps the arithmetic and
its order, so the developed output is bit for bit the same:
- The bilinear lookup of the reduced change maps (and of Dehaze's
transmission) worked out the source position with divisions for every
pixel and did it twice, once for each of the two planes. The taps of
every column and row are now computed once per image, and both planes
of a change are stored together.
- Texture's Gaussian bases ran their column pass down strided columns,
one tap at a time with a mirror per tap. Both passes now accumulate
neighbouring samples side by side, in the same tap order.
- The box mean's column pass runs row-wise the same way, blocks in
parallel.
- The luminance reduced for the work copy and for Texture's copy is
worked out once per pixel instead of twice, and the work copy is not
reduced at all when only Texture is set and there is no black point.
- log2 of the luminance is taken once for both changes.
Unit tests keep the replaced code and compare the bits.
Part of #9435.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When neither Dehaze, a local tone slider nor a camera black point is in play, no
statistic of the whole frame is needed, so the demosaic output no longer has to be
kept. Every band is demosaiced into a small band buffer and encoded straight into the
u16 output. The f32 RGB frame (12 bytes a pixel, 545 MB at 45 MP) and the pass that
read it back are gone. A Bayer sensor also skips the padded CFA plane: each band
normalizes the rows it reads (with the mirrored borders) into a small window. X-Trans
keeps its plane and green plane, LinearRaw has none. Tone, Dehaze and black points
keep the full-frame path.
The bands are the ones the full-frame path computes, with the same arithmetic per
pixel, so the output is bit-identical (tests compare both paths for Bayer, X-Trans and
LinearRaw, odd sizes, band borders, orientations and 1, 3 and 16 threads).
working_memory now charges what each path really holds.
Fixes#9434
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
develop_half bins every 2x2 block of a Bayer mosaic (3x3 for X-Trans,
2x2 box for LinearRaw and monochrome) and fuses black/white scaling,
white balance with its clip, gain maps, the colour matrix and the gamma
encode into one pass, with no demosaic and no full-size float frame.
The result is a Developed in the same colour space, type and
orientation as develop_sensor's, at half size, so a viewer can show it
first and swap the full develop in later. Tone sliders and the camera
black point are not applied (a warning says so). Part of #9433.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When neither Dehaze nor a local tone map is active (neutral tone and no
black point; only the Nikon D4 profile sets one), the matrix is applied
into a block of 768 samples and the gamma encode runs over the block:
clamp and square root first (vectorises), then the table with
[value, step] pairs. Same arithmetic, same output; other renderings keep
the old per-sample loop. A test compares the block encode with the
per-sample encode.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The predictor 1 row loop called Bits::diff out of line, indexed the
per-component tables with bounds checks, and kept a private copy of
the 48 KiB Huffman lookup pair per component even when the components
share one DHT (CR2 and DNG two-component frames).
- Bits::diff is always inlined.
- Rows with one or two components carry Ra in registers and walk the
row with chunks of two.
- Components that name the same table share one copy.
The frame size cap, the one-bit-per-sample truncation guard and the
DHT rebuild cap are untouched.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Same arithmetic in the same order, so the output is bit for bit the same
(the old code stays under cfg(test) as ahd_ref and a test compares them).
Scratch planes are allocated once per group of bands instead of per band
(no per-band zeroing or page faults) and never cleared, since every cell
a stage reads was written earlier in the same band. The layout is planar
RGB/Lab, the candidate pixel is built from values instead of indexed
stores (which stalled store forwarding), the 3x3 homogeneity sum is
separable, and the LabF table holds [value, step] pairs. The 32-row band
is kept, as a larger band costs peak memory.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Bit-packed uncompressed raw data was read one sample at a time through
a bit accumulator. Rows of 10, 12 and 14-bit samples are now read a
whole group (4x10, 2x12, 4x14 bits = 5, 3, 7 bytes) at a time; the
row tail still goes through the general reader. A file whose only
segment is the whole image row for row no longer copies it into a
second full-size buffer: the decoded Vec becomes the plane. The
predictor 2 undo works in place instead of on a copy.
The summed-segment charge (#4959) and the per-segment limits are
unchanged. A test compares the grouped reader with the plain bit
reader for 10/12/14 bits, 1 and 3 samples and several widths.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A DNG whose BlackLevel is a DOUBLE beyond f32 reads as inf. The periodic
fill then computed (raw - inf) * 0 = NaN, and AHD's f32::clamp with NaN
bounds asserts. The fill now returns (0, 0) for a non-finite black, as
Scale::value does, and AHD uses a compare-and-select clamp that gives the
same result for every non-NaN bound.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A DNG converter's lossy output carries one MapPolynomial opcode per
plane in OpcodeList2, and its colours and contrast depend on it. The
opcode was reported as not applied, so such a file now opened at full
size from its raw data but with the wrong colours.
opcodes::parse now returns Opcodes { gain_maps, polynomials, skipped }
and reads MapPolynomial (ID 8: area, planes, pitches, degree up to 8,
coefficients). The DNG reader applies OpcodeList2's polynomials to the
samples as it decodes them, on the linear reference values (after the
black and white levels, input and output clipped to 0..1), so
development, white balance and the camera-look fit all see the mapped
data. A LinearizationTable is applied first and is then done with; for
floating-point data, which is linear already, it is ignored.
WhiteLevel holds one value per sample plane, but the float scale came
from the first only. A plane whose white level was higher than the
first's was clipped before the DNG reader applied its own level. The
scale now comes from the highest valid white level (default 1.0).
A compression-7 frame may declare up to four times its segment's samples
(ljpeg::decode is capped at 4x the tile), and read_plane returned the whole
frame, so each retained tile held up to 4x what retained_segment_bytes
charges it: tiles declared 256x256 holding 512x512 frames pass a 1 MiB
budget while keeping 2 MiB. The lossless-JPEG arm moves into
lossless_jpeg(), which now truncates to the segment's rows (all the scatter
reads) and releases the spare capacity.
The PR's test only called the helper; deleting the call in read_plane kept
it green. decode_charges_the_retained_tiles_together drives decode on an
18x18 CFA in four padded 16x16 LJ92 tiles: 1500 B -> LimitExceeded, 2048 B
-> Ok. lossless_jpeg_keeps_only_the_segment pins the truncation. The doc
comment now says the output plane is alive alongside the sum.
Co-authored-by: Muhammad Saad <188063027+saad-mughal435@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- gpu: the test modules share one GPU_LOCK, and timing and vector_mask
take it too, so one process drives one wgpu device at a time.
- ui-egui: the wgpu tests in pattern_preview_canvas, adjust_preview_gpu,
marquee_autoscroll and layout_perf take gpu_lock().
- affinity compound_limits: the child process selects its test by the
module path (compound_limits::<name>); with --exact, the bare name
matches no test and the child passes without running the check.
- xtask scorecard reads crates/io/tests/it/corpus.rs.
- Docs and comments name the new test paths and the --test it filters.
Every tests/*.rs file is its own executable that links the whole crate
graph. Move each crate's test files into tests/it/ as modules of one
tests/it/main.rs; helper modules under tests/common move with them.
These keep their own binary (own process):
- engine panic_hunt: a process-wide panic hook counts every panic.
- engine text_gamma: changes the process-wide text gamma.
- engine smart_filter_stages, mask_cache_purge: assert on global caches.
- codecs pnm_peak_memory: reads the process's peak RSS.
- photocraft mac_menu_appkit: harness = false.
crates/engine/src/file_cmds.rs: batch keeps the save-point closure (takes the output path) and also copies the session's authorize hook into each scratch session, as main now does.
crates/raw/src/sensor.rs: both sides cap the memory of retained tiles; kept the existing retained_segment_bytes check (checked multiplication, same limit) and dropped main's duplicate inline loop.
Resolutions: CLI usage, flag lists and flag parsing carry both --tiff-compression (t15) and
--jpeg-format (t16); file.saveACopy, batch, imageProcessor and layersToFiles param specs carry
both tiffCompression and jpegFormat; file_cmds keeps tiff_compression_param and
jpeg_format_param; atn_cmds keeps flag_set/applies_at_canvas_bounds (t15) and UNHOLDABLE (t16),
edit.transform uses the pixel_target guard (t15) with layer_ref(p)? (t16); ui-egui open paths
build ImportOptions (t16) and apply raw_defaults (t15); docs headers/rows unioned; atn tests keep
both sides; tr.tsv "Keep Using CPU" kept once.
FredApps
Dylan M. Taylor
David Mann
aiongg
vangel76andClaude Sonnet 5.5
aiongg
SBaksa
