LR-IMP-CAMERA-COVERAGE: Sony black levels from the SR2 layout and the lossless M/S pedestal
Part of #535 (items 2 and 3). - SR2SubIFD black level: decrypt the directory's first entry (always the black level, 0x7310) to find where its value sits, and decrypt it with the keystream known for that position (1638, 2166, 2418, 2510). The fixed read at 2510 gave 354-365 on the DSLR-A450/A500/A550 and 975 on the DSLR-A700, where ExifTool and the data floor say 512 (A700 render: dE 20.1 -> 8.8, dL* -9.8 -> 0.0). Keystreams recovered black-box (plain 0x7310 and exiftool -v3 as known plaintext), consistent over 115 ARWs from 97 bodies, 15 layouts. - Downsized lossless M/S ARWs (linear YCbCr): black 1024 and white 16895, i.e. the recorded 512 plus 512, measured against same-scene full-size codings on 12 files from 4 bodies (offset 1015-1052, scale 0.98-1.02). The 512 pedestal lifted shadows by ~3 % of white; M/S renders now match their full-size twins (7RM5 M dE 6.7 -> 3.1). - Only these 16 of 140 Sony files change; render cache version 24. - Tests: synthetic SR2 blocks for every known layout (incl. the A700 trap), YCbCr black/white; corpus test of black levels against the data floor; A500 and A700 samples added to the corpus list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Alex ChambersandClaude Opus 5.5 committed
1 parent
88ea1a804b
commit
3100cc1a5c
5 files changed
+210
-36
No files matched your search
@@ -62,7 +62,7 @@ impl SettingsHashes {
|
||||
}
|
||||
|
||||
/// Bump when the pipeline's output changes, to invalidate cached thumbnails.
|
||||
pub const RENDER_CACHE_VERSION: u64 = 22;
|
||||
pub const RENDER_CACHE_VERSION: u64 = 24;
|
||||
|
||||
/// Thumbnails render at one of these long edges (so window/cell size changes reuse the cache).
|
||||
pub const THUMB_SIZES: [usize; 4] = [128, 256, 384, 512];
|
||||
|
||||
Vendored
+135
-34
@@ -17,8 +17,9 @@
|
||||
//! White balance: the raw IFD's `0x7313` levels when present (bodies from about 2017 on); otherwise the
|
||||
//! `WB_RGBLevels` of the maker note's enciphered `Tag2010` block (see [`DECIPHER`] and [`TAG2010_WB`]). Without
|
||||
//! either, an ARW opened with unit multipliers, i.e. a strong green cast (issue #148). Black level: the raw IFD's
|
||||
//! `0x7310`, else the level stored in the encrypted `SR2SubIFD` (see [`SR2_BLACK_AT`]; 800 rather than the
|
||||
//! default 512 on 1″-sensor bodies such as the RX100 series), else 512 (14-bit) / 128 (12-bit).
|
||||
//! `0x7310`, else the level stored in the encrypted `SR2SubIFD` (see [`SR2_HEAD_KEYSTREAM`]; 800 rather than the
|
||||
//! default 512 on 1″-sensor bodies such as the RX100 series), else 512 (14-bit) / 128 (12-bit). Downsized lossless
|
||||
//! (YCbCr) data sit [`YCBCR_OFFSET`] above that.
|
||||
//!
|
||||
//! Also: uncompressed 16-bit ARW, and lossless-compressed ARW (Compression 7, ILCE-7M4 and later): LJ92 tiles whose
|
||||
//! frames hold one 2×2 CFA cell per four-component sample ([`read_quad_tiles`]). Other lossless-JPEG layouts go
|
||||
@@ -302,6 +303,18 @@ fn read_quad_tiles(bytes: &[u8], info: &ImageInfo) -> Result<Vec<u16>> {
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Sony's downsized lossless ARWs (M and S sizes: linear YCbCr, already white-balanced) are stored this far above the
|
||||
/// sensor black level the raw IFD records (`0x7310`, 512 in every file seen); their reference levels say Y black 0.
|
||||
/// Measured against a full-size coding of the same scene shot seconds apart, on the pixels where both camera JPEGs
|
||||
/// agree: YCbCr = k · (full-size − black) · WB + b, with b 1015–1052 and green k 0.98–1.02, on 11 of 12 files from
|
||||
/// four bodies (ILCE-7M4 M, S and APS-C S; ILCE-7RM5 M, S and S35 S; ILCE-7CR M, S and S35 S; ILCE-9M3 M, S and
|
||||
/// APS-C S; the twelfth's reference was shot 24 s later, 7 % brighter). Their darkest 0.1 % lies at 1047–1127. So
|
||||
/// black and white both move up by 512: the scale stays the full-size raw's, and an M/S file renders like its
|
||||
/// full-size twin. Subtracting only 512 left a pedestal of 3 % of white, lifting the shadows (issue #535).
|
||||
const YCBCR_OFFSET: f32 = 512.0;
|
||||
/// White of the YCbCr data before [`YCBCR_OFFSET`]: the reference levels' Y white (required by [`read_ycbcr_tiles`]).
|
||||
const YCBCR_WHITE: f32 = 16383.0;
|
||||
|
||||
/// Downsized M/S ARWs are linear YCbCr 4:2:0 / 4:2:2, not CFA, despite retaining dummy CFA tags.
|
||||
/// T.81 supplies the lossless coding; TIFF 6.0 supplies the YCbCr coefficients and reference
|
||||
/// levels. Sony's reference chroma black and white are identical (the neutral offset).
|
||||
@@ -371,23 +384,40 @@ fn read_ycbcr_tiles(bytes: &[u8], info: &ImageInfo, raw: &Ifd, mode: Mode) -> Re
|
||||
const SR2_SUBIFD_OFFSET: u16 = 0x7200;
|
||||
const SR2_SUBIFD_LENGTH: u16 = 0x7201;
|
||||
const SR2_SUBIFD_KEY: u16 = 0x7221;
|
||||
/// Black level inside the encrypted `SR2SubIFD` (bodies that do not write `0x7310` in the raw IFD), recovered by
|
||||
/// black-box known-plaintext analysis of CC0 raw.pixls.us samples; no description of Sony's cipher was used.
|
||||
/// The encrypted `SR2SubIFD` (bodies that do not write `0x7310` in the raw IFD keep their black level only there)
|
||||
/// is a TIFF directory whose bytes are XOR-ed with a keystream. Everything below was recovered by black-box
|
||||
/// known-plaintext analysis of CC0 raw.pixls.us samples; no description of Sony's cipher and no decoder source was
|
||||
/// used. Plaintext came from the plain `0x7310` some bodies also write and from the decrypted directory ExifTool
|
||||
/// prints (`exiftool -v3`, used as a tool).
|
||||
///
|
||||
/// Every Sony ARW we examined (16 files, 15 bodies, 2012–2021) stores the same key (`11 22 33 44`), so its
|
||||
/// keystream is the same in every file. The DSC-RX100M6 and ILCE-7M3 also write their black levels in plain form
|
||||
/// (`0x7310`: 800 and 512); XOR-ing their `SR2SubIFD`s against each other matched `pack(512×4) ^ pack(800×4)` at
|
||||
/// exactly one position, byte 2510, which gave the eight keystream bytes there. Applied to the other files at that
|
||||
/// position they decode to four equal levels that agree with each sensor's dark-pixel floor: 800 on every 1″-sensor
|
||||
/// body (RX10, RX100 … RX100M5), 512 on the APS-C/full-frame ones (NEX-5N/6, SLT-A77, ILCE-6000/7/7M2/7RM2), in
|
||||
/// both `SR2SubIFD` sizes seen (29252 and 56958 bytes). Newer layouts (ILCE-7M4, 35398 bytes) put other data there
|
||||
/// and are rejected by the plausibility checks; those bodies write `0x7310` anyway.
|
||||
const SR2_BLACK_AT: usize = 2510;
|
||||
const SR2_BLACK_KEYSTREAM: [u8; 8] = [0x43, 0xcb, 0x86, 0xb6, 0x11, 0xd2, 0x1a, 0x73];
|
||||
/// Every ARW examined stores the same key (`11 22 33 44`), and the keystream depends only on the position in the
|
||||
/// `SR2SubIFD`: the same bytes at each position in 115 files from 97 bodies (DSLR-A200 to ILCE-9M3) and 15 directory layouts
|
||||
/// (lengths 25150 … 62112 bytes). In all of them the directory's first entry is the black level (tag `0x7310`,
|
||||
/// SHORT, count 4), whose value lives at a layout-dependent position (1638 … 2786). So the first entry is
|
||||
/// decrypted ([`SR2_HEAD_KEYSTREAM`]) to find that position, and the value is decrypted with the keystream known
|
||||
/// for it ([`SR2_BLACK_KEYSTREAMS`]). Reading a fixed position instead (2510, right for the 29252, 33210 and
|
||||
/// 56958-byte layouts) read other data on the DSLR-A450/A500/A550 (27152 bytes: 354–365) and the DSLR-A700
|
||||
/// (62112 bytes: 975), whose black is 512 by ExifTool and by the data floor (issue #535).
|
||||
const SR2_KEY: [u8; 4] = [0x11, 0x22, 0x33, 0x44];
|
||||
/// Keystream of the `SR2SubIFD`'s first 14 bytes: entry count and first entry (tag, type, count, value offset).
|
||||
/// The first 16 keystream bytes were identical in all 115 files.
|
||||
const SR2_HEAD_KEYSTREAM: [u8; 14] = [0x54, 0xc2, 0xc4, 0xd6, 0x49, 0xc4, 0x78, 0xbc, 0x34, 0xae, 0x2f, 0x8e, 0x25, 0xd8];
|
||||
/// Keystream at the black-level value, by its position in the `SR2SubIFD`, for the layouts of bodies without a
|
||||
/// plain `0x7310` (bodies with one: 2030, 2102, 2186, 2558 and 2786; not needed). 1638: DSLR-A700 (and the
|
||||
/// undecoded A200/A300/A350); 2166: DSLR-A450/A500/A550 (and the undecoded A230 … A390, A850, A900); 2418:
|
||||
/// DSLR-A560/A580, NEX-3/5/C3, SLT-A33/A35/A55; 2510: NEX-5N … NEX-7, SLT-A37 … A99, ILCE-3000 … 7RM2, ILCA-68/77M2/
|
||||
/// 99M2, RX10/RX100/RX1 series. Each was the same in every file whose black level sits there (4, 10, 8 and 55
|
||||
/// files), and decodes to that file's black level (512, or 800 on the 1″ sensors) at its dark-pixel floor.
|
||||
const SR2_BLACK_KEYSTREAMS: [(usize, [u8; 8]); 4] = [
|
||||
(1638, [0x18, 0x95, 0x65, 0xf0, 0x18, 0x8f, 0x61, 0xc6]),
|
||||
(2166, [0xc6, 0x17, 0x02, 0xcb, 0x89, 0x21, 0xbc, 0x43]),
|
||||
(2418, [0xd1, 0xef, 0xda, 0xfe, 0x50, 0xf3, 0xd0, 0xbe]),
|
||||
(2510, [0x43, 0xcb, 0x86, 0xb6, 0x11, 0xd2, 0x1a, 0x73]),
|
||||
];
|
||||
|
||||
/// The per-channel black level from the encrypted `SR2SubIFD` (see [`SR2_BLACK_AT`]), in 14-bit units; `None`
|
||||
/// unless the file uses the known key and the four decoded levels are equal and plausible.
|
||||
/// The per-channel black level from the encrypted `SR2SubIFD` (see [`SR2_HEAD_KEYSTREAM`]), in 14-bit units; `None`
|
||||
/// unless the file uses the known key, the directory's first entry is a black level at a known position and the
|
||||
/// four decoded levels are equal and plausible.
|
||||
fn sr2_black(bytes: &[u8], ifd0: &Ifd, order: lightcraft_tiff::ByteOrder) -> Option<f32> {
|
||||
let private = ifd0.bytes(t::DNG_PRIVATE_DATA)?;
|
||||
let at = order.read_u32(private, 0)? as u64;
|
||||
@@ -397,17 +427,26 @@ fn sr2_black(bytes: &[u8], ifd0: &Ifd, order: lightcraft_tiff::ByteOrder) -> Opt
|
||||
return None;
|
||||
}
|
||||
let (start, len) = (usize::try_from(sr2.u64(SR2_SUBIFD_OFFSET)?).ok()?, usize::try_from(sr2.u64(SR2_SUBIFD_LENGTH)?).ok()?);
|
||||
if len < SR2_BLACK_AT + 8 {
|
||||
return None;
|
||||
}
|
||||
sr2_black_levels(bytes.get(start.checked_add(SR2_BLACK_AT)?..start.checked_add(SR2_BLACK_AT + 8)?)?, order)
|
||||
sr2_black_in(bytes.get(start..start.checked_add(len)?)?, start, order)
|
||||
}
|
||||
|
||||
/// Decipher the eight `SR2SubIFD` bytes at [`SR2_BLACK_AT`] into one black level (the mean of four near-equal,
|
||||
/// plausible per-channel levels).
|
||||
fn sr2_black_levels(cipher: &[u8], order: lightcraft_tiff::ByteOrder) -> Option<f32> {
|
||||
let plain: Vec<u8> = cipher.iter().zip(SR2_BLACK_KEYSTREAM).map(|(c, k)| c ^ k).collect();
|
||||
let levels: Vec<u16> = (0..4).filter_map(|i| order.read_u16(&plain, 2 * i)).collect();
|
||||
/// [`sr2_black`] on the encrypted `SR2SubIFD` bytes `block`, which start at file offset `start` (the directory's
|
||||
/// value offsets are file offsets).
|
||||
fn sr2_black_in(block: &[u8], start: usize, order: lightcraft_tiff::ByteOrder) -> Option<f32> {
|
||||
let head: Vec<u8> = block.get(..SR2_HEAD_KEYSTREAM.len())?.iter().zip(SR2_HEAD_KEYSTREAM).map(|(c, k)| c ^ k).collect();
|
||||
let (tag, kind, count, value_at) = (order.read_u16(&head, 2)?, order.read_u16(&head, 4)?, order.read_u32(&head, 6)?, order.read_u32(&head, 10)?);
|
||||
if tag != BLACK_LEVEL || kind != 3 || count != 4 {
|
||||
return None;
|
||||
}
|
||||
let pos = usize::try_from(value_at).ok()?.checked_sub(start)?;
|
||||
let &(_, keystream) = SR2_BLACK_KEYSTREAMS.iter().find(|(p, _)| *p == pos)?;
|
||||
let plain: Vec<u8> = block.get(pos..pos.checked_add(8)?)?.iter().zip(keystream).map(|(c, k)| c ^ k).collect();
|
||||
sr2_black_levels(&plain, order)
|
||||
}
|
||||
|
||||
/// One black level from the four deciphered per-channel levels (their mean), when they are near-equal and plausible.
|
||||
fn sr2_black_levels(plain: &[u8], order: lightcraft_tiff::ByteOrder) -> Option<f32> {
|
||||
let levels: Vec<u16> = (0..4).filter_map(|i| order.read_u16(plain, 2 * i)).collect();
|
||||
let (&lo, &hi) = (levels.iter().min()?, levels.iter().max()?);
|
||||
(levels.len() == 4 && (64..=4096).contains(&lo) && hi - lo <= 64).then(|| levels.iter().map(|&v| f32::from(v)).sum::<f32>() / 4.0)
|
||||
}
|
||||
@@ -507,14 +546,15 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
|
||||
};
|
||||
let default_black = if scale_bits >= 14 { 512.0 } else { 128.0 };
|
||||
let black = match raw.f64s(BLACK_LEVEL).as_deref() {
|
||||
Some(v) if linear_rgb && !v.is_empty() => BlackLevel::uniform((v.iter().sum::<f64>() / v.len() as f64) as f32),
|
||||
Some(v) if linear_rgb && !v.is_empty() => BlackLevel::uniform((v.iter().sum::<f64>() / v.len() as f64) as f32 + YCBCR_OFFSET),
|
||||
Some([a, b, c, d]) => {
|
||||
BlackLevel { repeat_rows: 2, repeat_cols: 2, values: vec![*a as f32, *b as f32, *c as f32, *d as f32], ..Default::default() }
|
||||
}
|
||||
_ if linear_rgb => BlackLevel::uniform(default_black + YCBCR_OFFSET),
|
||||
_ => BlackLevel::uniform(sr2_black(bytes, ifd0, tiff.order).filter(|_| scale_bits >= 14).unwrap_or(default_black)),
|
||||
};
|
||||
let white = if linear_rgb {
|
||||
16383.0
|
||||
YCBCR_WHITE + YCBCR_OFFSET
|
||||
} else {
|
||||
raw.f64(t::WHITE_LEVEL).map(|v| v as f32).filter(|v| *v > 0.0).unwrap_or_else(|| super::white_from_data(samples, scale_bits))
|
||||
};
|
||||
@@ -591,6 +631,9 @@ mod tests {
|
||||
assert!(full.cfa.is_none());
|
||||
assert_eq!(full.data.len(), 48);
|
||||
assert_eq!(full.wb_multipliers, Some([1.0; 3]));
|
||||
// the YCbCr data sit 512 above the recorded sensor black level; white moves with it
|
||||
assert_eq!((full.black.mean(), full.white_at(0)), (1024.0, 16895.0));
|
||||
assert_eq!((header.black.mean(), header.white_at(0)), (1024.0, 16895.0));
|
||||
let RawData::U16(ref pixels) = full.data else {
|
||||
panic!("integer ARW");
|
||||
};
|
||||
@@ -735,16 +778,74 @@ mod tests {
|
||||
#[test]
|
||||
fn sr2_black_level_decodes_and_rejects_garbage() {
|
||||
use lightcraft_tiff::ByteOrder::Little;
|
||||
let cipher =
|
||||
|levels: [u16; 4]| -> Vec<u8> { levels.iter().flat_map(|v| v.to_le_bytes()).zip(SR2_BLACK_KEYSTREAM).map(|(p, k)| p ^ k).collect() };
|
||||
assert_eq!(sr2_black_levels(&cipher([800; 4]), Little), Some(800.0));
|
||||
assert_eq!(sr2_black_levels(&cipher([512, 512, 514, 512]), Little), Some(512.5));
|
||||
assert_eq!(sr2_black_levels(&cipher([65320, 64928, 65260, 488]), Little), None); // ILCE-7M4's layout
|
||||
assert_eq!(sr2_black_levels(&cipher([0; 4]), Little), None);
|
||||
assert_eq!(sr2_black_levels(&cipher([800; 4])[..5], Little), None);
|
||||
let plain = |levels: [u16; 4]| -> Vec<u8> { levels.iter().flat_map(|v| v.to_le_bytes()).collect() };
|
||||
assert_eq!(sr2_black_levels(&plain([800; 4]), Little), Some(800.0));
|
||||
assert_eq!(sr2_black_levels(&plain([512, 512, 514, 512]), Little), Some(512.5));
|
||||
assert_eq!(sr2_black_levels(&plain([65320, 64928, 65260, 488]), Little), None);
|
||||
assert_eq!(sr2_black_levels(&plain([0; 4]), Little), None);
|
||||
assert_eq!(sr2_black_levels(&plain([800; 4])[..5], Little), None);
|
||||
assert_eq!(sr2_black_levels(&[], Little), None);
|
||||
}
|
||||
|
||||
/// An encrypted `SR2SubIFD` of `len` bytes at file offset `start` whose first entry is a black level stored at
|
||||
/// `value_pos` (relative), holding `levels`; `other` puts four more levels at position 2510 (where the old
|
||||
/// fixed read looked).
|
||||
fn sr2_block(start: usize, len: usize, tag: u16, value_pos: usize, levels: [u16; 4], other: Option<[u16; 4]>) -> Vec<u8> {
|
||||
let mut plain = vec![0u8; len];
|
||||
plain[0..2].copy_from_slice(&180u16.to_le_bytes());
|
||||
plain[2..4].copy_from_slice(&tag.to_le_bytes());
|
||||
plain[4..6].copy_from_slice(&3u16.to_le_bytes());
|
||||
plain[6..10].copy_from_slice(&4u32.to_le_bytes());
|
||||
plain[10..14].copy_from_slice(&((start + value_pos) as u32).to_le_bytes());
|
||||
for (i, v) in levels.iter().enumerate() {
|
||||
plain[value_pos + 2 * i..value_pos + 2 * i + 2].copy_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
if let Some(o) = other {
|
||||
for (i, v) in o.iter().enumerate() {
|
||||
plain[2510 + 2 * i..2510 + 2 * i + 2].copy_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
}
|
||||
// encrypt the bytes whose keystream is known (the rest stays plain: it is never read)
|
||||
for (b, k) in plain.iter_mut().zip(SR2_HEAD_KEYSTREAM) {
|
||||
*b ^= k;
|
||||
}
|
||||
for (pos, ks) in SR2_BLACK_KEYSTREAMS {
|
||||
if let Some(dst) = plain.get_mut(pos..pos + 8) {
|
||||
for (b, k) in dst.iter_mut().zip(ks) {
|
||||
*b ^= k;
|
||||
}
|
||||
}
|
||||
}
|
||||
plain
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sr2_black_level_follows_the_first_entry_to_its_layout() {
|
||||
use lightcraft_tiff::ByteOrder::Little;
|
||||
let start = 37584;
|
||||
// every layout with a known keystream, e.g. the DSLR-A700's (62112 bytes, value at 1638) and the
|
||||
// DSLR-A450/A500/A550's (27152 bytes, value at 2166)
|
||||
for (pos, _) in SR2_BLACK_KEYSTREAMS {
|
||||
for level in [512, 800] {
|
||||
let block = sr2_block(start, 27152, BLACK_LEVEL, pos, [level; 4], None);
|
||||
assert_eq!(sr2_black_in(&block, start, Little), Some(f32::from(level)), "value at {pos}");
|
||||
}
|
||||
}
|
||||
// the A700's layout holds four other equal values at 2510 (975 each); the old fixed read took those
|
||||
let a700 = sr2_block(start, 62112, BLACK_LEVEL, 1638, [512; 4], Some([975; 4]));
|
||||
assert_eq!(sr2_black_in(&a700, start, Little), Some(512.0));
|
||||
// not a black-level entry, a value at a position without a known keystream, outside the block, before it
|
||||
assert_eq!(sr2_black_in(&sr2_block(start, 27152, 0x7311, 2166, [512; 4], None), start, Little), None);
|
||||
assert_eq!(sr2_black_in(&sr2_block(start, 27152, BLACK_LEVEL, 2030, [512; 4], None), start, Little), None);
|
||||
assert_eq!(sr2_black_in(&sr2_block(start, 27152, BLACK_LEVEL, 2166, [512; 4], None)[..2170], start, Little), None);
|
||||
let block = sr2_block(start, 27152, BLACK_LEVEL, 2166, [512; 4], None);
|
||||
assert_eq!(sr2_black_in(&block, start + 4000, Little), None, "value offset before the block");
|
||||
assert_eq!(sr2_black_in(&block[..10], start, Little), None, "truncated directory");
|
||||
assert_eq!(sr2_black_in(&[], start, Little), None);
|
||||
// implausible levels at a known position
|
||||
assert_eq!(sr2_black_in(&sr2_block(start, 27152, BLACK_LEVEL, 2166, [4; 4], None), start, Little), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quad_tiles_place_cfa_cells() {
|
||||
// 14×12 mosaic in 8×8 tiles (the right and bottom tiles overhang), each tile a 4×4 frame of 2×2 cells,
|
||||
|
||||
@@ -357,6 +357,67 @@ fn corpus_sony_pre2017_colour_metadata() {
|
||||
eprintln!("pre-2017 Sony ARW colour metadata checked on {seen} files");
|
||||
}
|
||||
|
||||
/// Issue #535: Sony black levels against the data. The SR2SubIFD keeps the black level at a position that depends
|
||||
/// on its layout (the DSLR-A500 and A700 read 365 and 975 from a fixed position, where ExifTool and the data say
|
||||
/// 512), and the downsized lossless M and S codings (linear YCbCr) sit 512 above the recorded level (1024, with
|
||||
/// white moved by the same 512). Each file's darkest 0.1 % per channel must lie just above its black level: a black
|
||||
/// level that is too low leaves a pedestal, one that is too high clips the shadows.
|
||||
#[test]
|
||||
fn corpus_sony_black_levels_sit_at_the_data_floor() {
|
||||
use lightcraft_raw::RawData;
|
||||
let dir = corpus_root().join("raw");
|
||||
// (file, black, white): SR2SubIFD black at three layouts, a plain 0x7310, and the lossless L / M / S codings
|
||||
let cases = [
|
||||
("arw-sony-a500.arw", 512.0, None),
|
||||
("arw-sony-a700.arw", 512.0, None),
|
||||
("arw-sony-rx100m3.arw", 800.0, None),
|
||||
("arw-sony-a7m4-lossless-l.arw", 512.0, Some(16383.0)),
|
||||
("arw-sony-a7m4-lossless-m.arw", 1024.0, Some(16895.0)),
|
||||
("arw-sony-a7m4-lossless-s.arw", 1024.0, Some(16895.0)),
|
||||
];
|
||||
let mut seen = 0;
|
||||
for (name, black, white) in cases {
|
||||
let bytes = match std::fs::read(dir.join(name)) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||
eprintln!("skip: {name} absent");
|
||||
continue;
|
||||
}
|
||||
Err(e) => panic!("{name}: {e}"),
|
||||
};
|
||||
let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
|
||||
assert_eq!(img.black.mean(), black, "{name}: black level");
|
||||
if let Some(white) = white {
|
||||
assert_eq!(img.white_at(0), white, "{name}: white level");
|
||||
}
|
||||
let RawData::U16(ref data) = img.data else { panic!("{name}: integer samples expected") };
|
||||
// per CFA position (2×2) or per colour sample, inside the default crop, about a million samples each
|
||||
let channels = if img.cpp == 3 { 3 } else { 4 };
|
||||
let (a, c) = (img.active_area, img.crop);
|
||||
let step = (c.width * c.height / 1_000_000).max(1);
|
||||
let mut samples = vec![Vec::new(); channels];
|
||||
for (k, (x, y)) in (c.y..c.y + c.height).flat_map(|y| (c.x..c.x + c.width).map(move |x| (x + a.x, y + a.y))).enumerate() {
|
||||
if (k / 2) % step != 0 {
|
||||
continue;
|
||||
}
|
||||
if img.cpp == 3 {
|
||||
for (s, v) in samples.iter_mut().enumerate() {
|
||||
v.push(data[(y * img.width + x) * 3 + s]);
|
||||
}
|
||||
} else {
|
||||
samples[(y % 2) * 2 + x % 2].push(data[y * img.width + x]);
|
||||
}
|
||||
}
|
||||
for (i, mut v) in samples.into_iter().enumerate() {
|
||||
v.sort_unstable();
|
||||
let floor = f32::from(v[v.len() / 1000]);
|
||||
assert!((black - 64.0..=black + 128.0).contains(&floor), "{name}: channel {i}: darkest 0.1 % at {floor}, black level {black}");
|
||||
}
|
||||
seen += 1;
|
||||
}
|
||||
eprintln!("Sony black levels checked against the data floor on {seen} files");
|
||||
}
|
||||
|
||||
/// The embedded JPEG as linear RGB, reduced to `gw × gh` cells.
|
||||
fn jpeg_cells(jpeg: &[u8], gw: usize, gh: usize) -> Vec<[f64; 3]> {
|
||||
use zune_core::{bytestream::ZCursor, colorspace::ColorSpace, options::DecoderOptions};
|
||||
|
||||
+1
-1
@@ -144,7 +144,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
||||
| LR-IMP-SIDECAR-SPLIT | Separate XMP sidecar variants | P2 | ⬜ | `cmd:library.xmpPreferences` | sidecar naming option exists (stem/full), no split sidecars |
|
||||
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs`, `crates/raw/src/jxl.rs` | JPEG, PNG, TIFF (1- to 64-bit integer, 16- to 64-bit float, incl. bilevel and CCITT fax), WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG and DNG 1.7 JPEG XL tiles/strips: lossless ones decode bit-exactly on synthetic CFA and LinearRaw files, lossy ones are restored to the declared linear primaries without clipping or gamut mapping, checked on one real file with 108 lossy tiles, and any tile that fails to decode makes the file preview-only; a single-chunk JPEG XL preview IFD is offered as the embedded preview), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW (pre-2017 bodies: white balance and black level from enciphered maker-note / `SR2SubIFD` data, issue #148; downsized Sony lossless YCbCr 4:2:0 / 4:2:2 tiles decoded to linear RGB, verified on private ILCE-7M4 3:2 & 4:3 examples including 16-bit full-resolution export), NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF (uncompressed, lossless/lossy compressed Bayer and X-Trans; `crates/raw/src/vendor/rafc.rs`, full sensor verification on 13 bodies, `docs/raf-compression.md`), RW2 / Leica RWL / Panasonic RAW in every raw format (compressed 4 and 6, the prefix-coded strips of format 8 from the GH6 on, packed 2/5/7, the 16-bit words of the 2005–2007 bodies; black levels, in-camera aspect crops and mapped-out defect pixels; 178 CC0 files from 118 bodies checked, `crates/raw/src/vendor/rw2.rs`), PEF, ORF (uncompressed; colour-filter layout from the file's Exif `CFAPattern`), Hasselblad 3FR with uncompressed raw data (X1D, X1D II, X2D, CFV 100C: a TIFF without a DNG version tag that describes its CFA IFD, levels, crop and white balance with DNG's tags; `RawFormat::CfaTiff`, read by `dng::decode_as`, RGGB when the file states no `CFAPattern`; the lossless-JPEG 3FR and FFF files use scan selection value 8 and stay preview-only), Samsung SRW without compression (NX5, NX10, NX11 and the NX20-generation bodies NX20, NX200, NX210, NX1000 and NX1100 as 12-bit packed samples in either bit order, EX1 and WB2000 as 16-bit words; layout, camera-JPEG framing, levels and white balance from the maker note, `crates/raw/src/vendor/srw.rs`; median ΔE vs the camera JPEG 2.2–7.6 on 11 of the 12 files from 10 bodies checked, 10.7 (5.7 registered) on the backlit WB2000 sample). 12-bit NEF black level: maker note `0x003d` is in 14-bit units (D750, D780, D850, D7500, Z 50 checked). Lossy type 2 NEF curves of version `0x44 0x40` (D780, D6, Z 5, Z 6, Z 7, Z 7_2) span a quarter of the sample range, read from the table's own point spacing. CR3 lossless CRX Bayer and version 0x100/0x200 C-RAW decodes to sensor pixels with timed white balance, black/white levels and file crop (the shot's aspect ratio from the maker note's `AspectInfo`); full sensor samples match the reference on M50/R100/R8 (`crates/raw/tests/cr3_corpus.rs`, `docs/cr3.md`). Unsupported coding variants retain metadata and use the embedded JPEG when available; HEVC preview decoding remains unsupported. Missing: CR3 unverified CRX variants, compressed ORF, compressed Samsung SRW (NX1, NX30, NX300, NX500, NX2000, NX3000, NX3300, NX mini: preview only with the reason), NEFs labelled compressed (34713) but stored uncompressed without a `0x0096` table (Z 6 packed 14-bit, D850 12-bit uncompressed), NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), TIFF-based raws whose raw data sits in a private block (Phase One / Leaf IIQ, Canon EOS-1D / 1Ds and Kodak DCS TIFFs: IFD0 is only a reduced RGB image next to a much larger declared picture) are recognised from that structure and import as `preview_only` with the reason, instead of opening as a thumbnail-sized ordinary image (`thumbnail_shell` in `crates/raw/src/lib.rs`), Recognised raws LightCraft cannot decode open as `preview_only` with any usable embedded JPEG, also Canon CRW, Minolta MRW and Sigma X3F (containers recognised by magic, preview found by a structural JPEG scan), Kodak KDC (IFD0 without `ImageWidth`), Leaf MOS and Epson ERF (scan), and any decode failure of a recognised raw, not only an unsupported variant; ones with no JPEG still fail with "without an embedded preview", AVIF decode (HEIC decodes only behind the non-default `heif` feature: `crates/heif`, heic-rs, same policy as PhotoCraft; 8/10-bit, alpha, irot/imir/clap, ICC/EXIF/XMP), palette (indexed-colour) TIFF (refused by the `tiff` crate) |
|
||||
|
||||
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~55 CC0 corpus files plus 6 exact-sensor CR3 cases (plus ~60 NEFs checked by hand and 178 Panasonic / Leica RW2, RWL and RAW files from 118 bodies, checked against each file's own JPEG) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies; Canon PowerShot and EOS M CR2s whose `ColorData` is stored as bytes opened green until the bytes were read as 16-bit words, 25 files from 25 bodies checked). Compressed RAF sensor arrays verified on 13 Fujifilm bodies (`docs/raf-compression.md`). Grow the CC0 corpus per model and test each sample decodes with plausible colour. Nikon CropArea and untagged-preview colour-space regression tests; private Z 8 crop verified. |
|
||||
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~55 CC0 corpus files plus 6 exact-sensor CR3 cases (plus ~60 NEFs checked by hand and 178 Panasonic / Leica RW2, RWL and RAW files from 118 bodies, checked against each file's own JPEG) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies; pre-2017 Sony ARWs opened green until their scrambled white balance / black level were read, issue #148, checked on 16 ARWs from 15 Sony bodies; the encrypted SR2 black level is now found through its directory's first entry (the DSLR-A450/A500/A550/A700 read 354–975 instead of 512), checked on 115 ARWs from 97 Sony bodies, and downsized lossless M/S ARWs use their measured black level 1024 instead of 512, checked on 12 files from 4 bodies, issue #535; Canon PowerShot and EOS M CR2s whose `ColorData` is stored as bytes opened green until the bytes were read as 16-bit words, 25 files from 25 bodies checked). Compressed RAF sensor arrays verified on 13 Fujifilm bodies (`docs/raf-compression.md`). Grow the CC0 corpus per model and test each sample decodes with plausible colour. Nikon CropArea and untagged-preview colour-space regression tests; private Z 8 crop verified. |
|
||||
| LR-IMP-CULL-AT-IMPORT | Culling analysis at import | P2 | 🟡 | `cmd:photo.analyze` | run Assisted Culling on the imported photos (they're selected after an import); not automatic |
|
||||
| LR-IMP-MOVE | Move on import [Classic] | P1 | ✅ | `cmd:library.import` (`mode` move), `crates/engine/src/import_move.rs`, `crates/engine/src/tests_import_move.rs`, `crates/ui-egui/src/import.rs` | Transfer ▸ Move with Copy's destination, folders (day / month / one folder / custom template), rename template and example destination; XMP sidecars (both namings) move along; each source is removed only after its destination is written (hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate, unchecked files keep their sources, taken names get -1, -2…, never overwritten; files already in the destination/library are added in place; sources that can't be removed (read-only card) are kept and reported (`kept`). Undo removes the photos from the library but leaves the moved files at the destination. No Copy as DNG while moving |
|
||||
| LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched); the raw copy is removed only once its DNG is verified and on disk (issue #106) |
|
||||
|
||||
@@ -356,6 +356,18 @@ const RAW_SAMPLES: &[(&str, &str, &str)] = &[
|
||||
"https://raw.pixls.us/data/Sony/ILCE-7RM2/12-bit-uncompressed.ARW",
|
||||
"71e0888396ef52c7e6a990b4e72ebb4d8413a1fc1c35bada449870539cef6e39",
|
||||
),
|
||||
// older DSLRs whose SR2SubIFD keeps the black level elsewhere than the bodies above (#535): A500 (27152-byte
|
||||
// layout, like the A450/A550), A700 (62112 bytes)
|
||||
(
|
||||
"arw-sony-a500.arw",
|
||||
"https://raw.pixls.us/data/Sony/DSLR-A500/DSC02421.ARW",
|
||||
"1407fb596a391df67c15b90026a1a76702a2e7386dbdff2f0815b903ea49cead",
|
||||
),
|
||||
(
|
||||
"arw-sony-a700.arw",
|
||||
"https://raw.pixls.us/data/Sony/DSLR-A700/DSC07249.ARW",
|
||||
"3159e28892bf0771d01525cd4b6190f9c15bbb19fa2fab6d2515ced0594e8f40",
|
||||
),
|
||||
// CR2 colour-filter layouts differ by model (issue #85): CR2CFAPattern 3 (GBRG) and 1 (RGGB) samples
|
||||
(
|
||||
"cr2-canon-40d.cr2",
|
||||
|
||||
Reference in new issue
Block a user