From 903b816bdfa22f2b415f9639aef85ddb19862edc Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Mon, 26 Oct 2026 17:40:25 -0400 Subject: [PATCH] crw: keep the format record, drop the derivation trail The crate's record should hold what the decoder is built from, not the process that produced it. The starting assumptions, the plan, the container notes, the entropy-derivation log, the chart calibration, the probe scripts and their observational logs are not part of it. The frozen-table JSONs and tools/gen_tables.py stay, so src/tables.rs can still be regenerated and audited. Renumber the surviving documents 01-05 in reading order and update every reference to them, so the record reads in order and cites nothing that is no longer there. AGENTS.md and CLAUDE.md go back to main: repository instructions are not part of this change. --- AGENTS.md | 49 +- ATTRIBUTION.md | 4 +- CLAUDE.md | 7 +- crates/crw/README.md | 6 +- .../crw/clean-room/00-starting-assumptions.md | 111 --- ...payload-layout.md => 01-payload-layout.md} | 21 +- crates/crw/clean-room/01-plan.md | 337 ------- crates/crw/clean-room/02-container.md | 119 --- .../{06-crw-format.md => 02-crw-format.md} | 12 +- ...{07-verification.md => 03-verification.md} | 12 +- .../clean-room/{08-color.md => 04-color.md} | 6 +- crates/crw/clean-room/04-derivation-log.md | 315 ------- ...private-corpus.md => 05-private-corpus.md} | 6 +- crates/crw/clean-room/10-chart-calibration.md | 97 -- crates/crw/clean-room/README.md | 47 +- crates/crw/clean-room/ciff_dump.py | 197 ---- crates/crw/clean-room/logs/chart-captures.tsv | 39 - .../crw/clean-room/logs/p1-payload-public.txt | 195 ---- .../clean-room/logs/p1b-markers-public.txt | 109 --- .../crw/clean-room/logs/p1c-bounds-public.txt | 11 - .../clean-room/logs/structure-d60-public.txt | 73 -- .../crw/clean-room/logs/structure-public.txt | 887 ------------------ .../clean-room/logs/verification-sweep.txt | 22 - crates/crw/clean-room/p1_payload.py | 184 ---- crates/crw/clean-room/p1b_markers.py | 119 --- crates/crw/clean-room/p1c_lowbits_bounds.py | 96 -- crates/crw/clean-room/p2_oracle_lsb.py | 118 --- crates/crw/clean-room/sources.md | 28 +- crates/crw/clean-room/tools/gen_tables.py | 2 +- crates/crw/src/lib.rs | 8 +- crates/crw/src/tables.rs | 2 +- crates/crw/tests/samples.rs | 2 +- 32 files changed, 85 insertions(+), 3156 deletions(-) delete mode 100644 crates/crw/clean-room/00-starting-assumptions.md rename crates/crw/clean-room/{03-payload-layout.md => 01-payload-layout.md} (88%) delete mode 100644 crates/crw/clean-room/01-plan.md delete mode 100644 crates/crw/clean-room/02-container.md rename crates/crw/clean-room/{06-crw-format.md => 02-crw-format.md} (94%) rename crates/crw/clean-room/{07-verification.md => 03-verification.md} (91%) rename crates/crw/clean-room/{08-color.md => 04-color.md} (97%) delete mode 100644 crates/crw/clean-room/04-derivation-log.md rename crates/crw/clean-room/{09-private-corpus.md => 05-private-corpus.md} (95%) delete mode 100644 crates/crw/clean-room/10-chart-calibration.md delete mode 100644 crates/crw/clean-room/ciff_dump.py delete mode 100644 crates/crw/clean-room/logs/chart-captures.tsv delete mode 100644 crates/crw/clean-room/logs/p1-payload-public.txt delete mode 100644 crates/crw/clean-room/logs/p1b-markers-public.txt delete mode 100644 crates/crw/clean-room/logs/p1c-bounds-public.txt delete mode 100644 crates/crw/clean-room/logs/structure-d60-public.txt delete mode 100644 crates/crw/clean-room/logs/structure-public.txt delete mode 100644 crates/crw/clean-room/logs/verification-sweep.txt delete mode 100644 crates/crw/clean-room/p1_payload.py delete mode 100644 crates/crw/clean-room/p1b_markers.py delete mode 100644 crates/crw/clean-room/p1c_lowbits_bounds.py delete mode 100644 crates/crw/clean-room/p2_oracle_lsb.py diff --git a/AGENTS.md b/AGENTS.md index 455051e3..e5adca3b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,54 +1,15 @@ # AGENTS.md -Shared Rust libraries for the Craft Apps (LightCraft, PhotoCraft, PdfCraft, VectorCraft, FilmCraft, -DesignCraft, WordCraft and the others). The shared standards live in +Shared Rust libraries for the Craft Apps. The shared standards live in [craftrules](https://github.com/storytold/craftrules) (`../../craftrules` on a dev machine): read its `AGENTS.md`, then `standards/never-crash.md`, `standards/engineering.md` and `standards/licensing.md`. -## Start every session here - -1. Read this file, then the `README.md` of each crate you touch (its boundaries, origin and tests). -2. A crate that derives or reads a format (`jpeg`, `media`, `crw`, …) follows - [`docs/clean-room.md`](docs/clean-room.md); a crate with a `clean-room/` folder keeps that record - intact (its `README.md` says what is frozen and how a change is reopened). -3. Check `git status`, `git worktree list` and unmerged branches before starting anything new; commit - small and often. - -## Rules - - **Never crash.** Every crate's `lib.rs` starts with `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]`. - No `unsafe` (the workspace forbids it). Input-derived numbers are hostile: `get()` instead of - indexing for offsets that come from a file, checked or saturating arithmetic for lengths and - counts, bounded recursion (depth limits or budgets), allocations capped by a `Limits` the caller - can lower. Every crash fix lands with a regression test that panicked before it. + No `unsafe` (the workspace forbids it). - **New crates** go in `crates/`, named `craft-`, inherit the workspace package fields - and lints, and are listed in `[workspace.dependencies]`, in the root `README.md` (one entry per - library) and in a workflow of their own (`.github/workflows/.yml`: fmt, clippy with - `-D warnings`, tests on Linux, Windows and macOS, and a `wasm32-unknown-unknown` check for a - crate that an app's web build can use). + and lints, and are listed in `[workspace.dependencies]`. - **No app code.** A crate here must be useful to more than one app and must not depend on any app. - App policy (colour models, UI, settings, command handling) stays in the apps; the crate's README - states its boundaries and, when the code came from an app, its origin (repository, paths, - revision) and what changed in the move. `NOTICE` → *Source provenance* records the same. -- **Clean-room.** Format code is written only from published specifications, prose documentation, - usable sample files and black-box observation of reference binaries; never from GPL/LGPL/AGPL/CDDL - sources or Adobe material, in any form. The process and the record a crate keeps are in - [`docs/clean-room.md`](docs/clean-room.md). If you have seen something you should not have, stop - and say so. -- **Assets and samples.** Nothing enters the repository without a row in `ATTRIBUTION.md` (path, - author, source, licence) and a licence text where one is required (`licenses/`). Sample media is - never committed: tests fetch pinned, SHA-256-verified, licence-checked files outside the checkout - (`scripts/fetch-*.py`) and skip cleanly when they are absent. No Adobe assets of any kind, including - Adobe's open-licensed fonts; fonts come from craft-fonts. -- **Time goes through `chrono`** (workspace dependency). Never hand-roll UTC/calendar conversion (days-to-date algorithms, leap-year math); a crate owns only its output format string. The one exception is range extension: past the years chrono can represent, shift the date by whole 400-year Gregorian cycles (146,097 days, after which the calendar repeats), let chrono compute the month and day, and add the cycles back to the year (splitting seconds into hours, minutes and seconds is plain arithmetic). A library that reads a timestamp from a file hands it on as Unix seconds and lets the app convert. -- **Tests.** Unit and property tests next to the code; every feature has a test that would fail - without it; decoders have a hostile-input sweep (truncations, byte mutations, proptest) and, for - real files, a digest-pinned sample test. A test that cannot run (no sample files, no GPU) prints - why it skipped and passes, unless a `*_REQUIRE_*` variable (set in CI) turns the skip into a - failure. +- **Time goes through `chrono`** (workspace dependency). Never hand-roll UTC/calendar conversion (days-to-date algorithms, leap-year math); a crate owns only its output format string. The one exception is range extension: past the years chrono can represent, shift the date by whole 400-year Gregorian cycles (146,097 days, after which the calendar repeats), let chrono compute the month and day, and add the cycles back to the year (splitting seconds into hours, minutes and seconds is plain arithmetic). - **Before merging:** `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, - `cargo test --workspace`. Each crate's workflow runs the same for that crate on every PR. -- **Commits:** one change per commit, only green states, messages that say what and why. Apps - consume a crate by git revision (`craft- = { git = "https://github.com/storytold/craft-libs", rev = "" }`), - so a merged commit is an API contract: note breaking changes in the crate's README. + `cargo test --workspace`. diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index f3a6e899..9585c992 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -23,8 +23,8 @@ author, source and license. Library code and original assets are under the repos | `crates/ui/tests/fonts/NotoSansCJKsc-screens.otf` (subset to printable ASCII and the screen test strings) | The Noto Project Authors (Adobe, Google) | [notofonts/noto-cjk@f8d1575 `Sans/OTF/SimplifiedChinese`](https://github.com/notofonts/noto-cjk/tree/f8d157532fbfaeda587e826d4cd5b21a49186f7c/Sans/OTF/SimplifiedChinese), via craft-fonts | OFL-1.1 (`licenses/ui-test-fonts/OFL-NotoSansCJK.txt`) | | `crates/ui/tests/fonts/NotoSansThai-screens.ttf` (instance at weight 400 and width 100, subset to the screen test strings) | The Noto Project Authors | [google/fonts@8b0a1d0 `ofl/notosansthai`](https://github.com/google/fonts/tree/8b0a1d0f5983c89bc2b93f1b5fb55f9e252744b5/ofl/notosansthai), via craft-fonts | OFL-1.1 (`licenses/ui-test-fonts/OFL-NotoSansThai.txt`) | | `crates/crw/tests/samples.json` (manifest only; the 11 raw files are not bundled, `scripts/fetch-crw-samples.py` downloads them outside the checkout) | The photographers who contributed each file to raw.pixls.us (Canon EOS D30, D60, 10D ×3, 300D ×2, Digital Rebel, PowerShot G1, G5, G6) | [raw.pixls.us](https://raw.pixls.us/), each file's `getfile.php` URL and SHA-256 in the manifest | CC0-1.0 (as published by raw.pixls.us for each file) | -| `crates/crw/src/tables.rs`, `crates/crw/clean-room/logs/table_sel*.json` | craft-crw contributors (derived in LightCraft, 2026-10, by known-plaintext alignment of the sample files' entropy streams; `crates/crw/clean-room/06-crw-format.md`) | this repository | MIT OR Apache-2.0 | -| `crates/crw/clean-room/logs/*.txt`, `*.tsv` (statistics, file names, sizes and SHA-256 of sample files; no image content) | craft-crw contributors (LightCraft derivation record) | this repository | MIT OR Apache-2.0 | +| `crates/crw/src/tables.rs`, `crates/crw/clean-room/logs/table_sel*.json` | craft-crw contributors (derived in LightCraft, 2026-10, by known-plaintext alignment of the sample files' entropy streams; `crates/crw/clean-room/02-crw-format.md`) | this repository | MIT OR Apache-2.0 | +| `crates/crw/clean-room/logs/private-corpus.tsv` (file names, sizes and SHA-256 of the maintainer's own EOS D60 files; no image content) | craft-crw contributors (LightCraft derivation record) | this repository | MIT OR Apache-2.0 | Original assets are covered by the repository licence. diff --git a/CLAUDE.md b/CLAUDE.md index 2fe86b67..11908a75 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1 @@ -# CLAUDE.md - -Read [`AGENTS.md`](AGENTS.md): it is the instruction file for every agent working in this repository -(start-of-session steps, never-crash, new-crate checklist, clean-room, assets, time, tests, merge -gates). Format work also follows [`docs/clean-room.md`](docs/clean-room.md). Suite-wide standards: -[craftrules](https://github.com/storytold/craftrules). +Read `AGENTS.md`. diff --git a/crates/crw/README.md b/crates/crw/README.md index bf987c9a..63b01b30 100644 --- a/crates/crw/README.md +++ b/crates/crw/README.md @@ -14,7 +14,7 @@ tag documentation. The compression was never documented by Canon: it was **deriv maintainer-owned sample files, frozen, and verified sample-exact against a reference decoder's command-line output, treated as a black box. No raw-decoder source was consulted, at any point. The whole record is in [`clean-room/`](clean-room/README.md); the frozen format is -[`clean-room/06-crw-format.md`](clean-room/06-crw-format.md). +[`clean-room/02-crw-format.md`](clean-room/02-crw-format.md). ## What it decodes @@ -29,7 +29,7 @@ The whole record is in [`clean-room/`](clean-room/README.md); the frozen format | PowerShot G5 | 1 | **refused** | its entropy stream diverges from the model from sample 2056 on (`Error::Unsupported`); apps show the embedded JPEG | | PowerShot G2, G3, Pro1, Pro70, S30–S70 | | unverified | no CC0 samples exist; they decode through the same model if their records match, with no guarantee | -"Exact" means every sensor sample equals the reference decode (`clean-room/07-verification.md`). +"Exact" means every sensor sample equals the reference decode (`clean-room/03-verification.md`). Big-endian (`MM`) files are handled from the specification and tested synthetically; none has been seen in the wild. CHDK sensor dumps named `.CRW` are a different format and are not CIFF files (`probe` says no). @@ -141,7 +141,7 @@ Colour (camera matrices, profiles, the JPEG-fitted look) stayed in LightCraft. `src/tables.rs` is generated data. Edit nothing by hand: the source of truth is `clean-room/logs/table_sel{0,1,2}.json`, and `python crates/crw/clean-room/tools/gen_tables.py` renders it (byte-identical to the committed file). A change to the tables is a change to the frozen -format and must follow the reopening rule of `clean-room/06-crw-format.md`. +format and must follow the reopening rule of `clean-room/02-crw-format.md`. ## Boundaries diff --git a/crates/crw/clean-room/00-starting-assumptions.md b/crates/crw/clean-room/00-starting-assumptions.md deleted file mode 100644 index cfea605f..00000000 --- a/crates/crw/clean-room/00-starting-assumptions.md +++ /dev/null @@ -1,111 +0,0 @@ -# CRW clean-room derivation — starting assumptions - -Written 2026-10-12, **before** reading any file in this repository (other than the standing -CLAUDE.md instructions loaded into the session), any web page, any specification, or any sample -file. Everything below is recalled background knowledge of the author (Claude Fable 5) and is the -baseline the derivation starts from. **Every item is: unverified, no source consulted.** - -This file is committed once and never edited. Corrections, confirmations and refutations go in -`01-plan.md` and the derivation documents, each citing the experiment or published document that -established them. A reviewer can diff this file against the final documentation to see which -facts were assumed and which were derived. - -Confidence labels are the author's own recollection strength, not evidence: *firm* = would be -surprised if wrong; *hazy* = plausibly misremembered or confused with another format (CR2/CR3, -other makers' raws). - -## A. Container - -- **A1** (firm, unverified, no source consulted): `.CRW` files are in CIFF, the "Camera Image - File Format", a Canon-proprietary heap-based container. It is not TIFF-based and not - Exif/JPEG-based (unlike the later `.CR2`, which is TIFF). -- **A2** (firm, unverified, no source consulted): the file begins with a 2-byte byte-order mark, - `II` for little-endian (as TIFF does); Canon cameras write little-endian. `MM`/big-endian is - allowed by the format but not expected in camera files. -- **A3** (hazy, unverified, no source consulted): a 32-bit integer at offset 2 gives the length - of the file header, i.e. the offset at which the heap data begins; the common value is 26 - (0x1A). -- **A4** (firm, unverified, no source consulted): the header contains the ASCII signature - `HEAPCCDR` (at offset 6, immediately after the header-length field). -- **A5** (firm, unverified, no source consulted): the body of the file is a "heap". A heap's - directory is located via the **last 4 bytes** of the heap region: a 32-bit offset, relative to - the start of the heap, pointing at the directory. Data blocks come first, directory last. -- **A6** (firm, unverified, no source consulted): a directory is a 16-bit record count followed - by that many fixed-size 10-byte records: tag (u16), size (u32), offset (u32), with offsets - relative to the start of the containing heap. There may be trailing bytes after the records. -- **A7** (hazy, unverified, no source consulted): the tag value's high bits encode storage class - and data type, the low bits the tag identity. In particular, one storage class means "value - stored in the heap" (size/offset meaningful) and another means "value stored in the record - itself" (the 8 bytes that would hold size+offset hold the data inline). Recalled masks: - storage class ≈ `0xC000`, data type ≈ `0x3800`, id ≈ low 11 bits — exact masks uncertain. -- **A8** (firm, unverified, no source consulted): some records are themselves heaps - (sub-directories), giving a tree: e.g. an image-properties heap, a camera-object heap, an - Exif-information heap. Parsing must recurse; recursion depth must be bounded against malformed - files. -- **A9** (hazy, unverified, no source consulted): recalled tag numbers — `0x2005` raw image - data; `0x1835` "decoder table" selector for the raw compression; `0x1031` sensor/image - dimensions (including the margins of masked pixels); `0x2007` embedded JPEG - ("JpgFromRaw"-like); `0x080a` make and model as NUL-separated strings; something like - `0x102a` holding shot info (ISO, etc.) and a white-balance record whose tag I do not recall. - All tag numbers here are suspect. -- **A10** (firm, unverified, no source consulted): cameras writing CRW also write a sibling - `.THM` file: a small Exif/JPEG thumbnail carrying the metadata. The CRW itself may embed a - preview JPEG as well. -- **A11** (hazy, unverified, no source consulted): CRW was written by early Canon EOS DSLRs - (D30, D60, 10D, 300D/Digital Rebel) and PowerShot models of the same era (G1–G6, S-series, - Pro1). Later bodies (20D on) switched to CR2. - -## B. Raw data storage and coding - -- **B1** (firm, unverified, no source consulted): the raw payload is a single plane of Bayer - CFA samples, pattern RGGB at the sensor origin, covering the full readout area including - masked/optically-black border pixels, which is slightly larger than the visible image. -- **B2** (hazy, unverified, no source consulted): sample depth is 10 or 12 bits depending on - the model; the EOS D60 is 12-bit. (10-bit is recalled for some PowerShots.) -- **B3** (firm, unverified, no source consulted): the raw data is losslessly compressed with a - Canon-proprietary scheme. It is **not** a JPEG stream: no JFIF/JPEG markers, no - entropy-coded-segment byte stuffing wrapper — a bare bitstream. -- **B4** (firm, unverified, no source consulted): the scheme is Huffman-coded DPCM in the style - of JPEG's AC coefficient coding: the stream is organised in blocks of 64 difference values; - each Huffman code decodes to a (run-of-zero-differences, magnitude-bit-count) pair; a - magnitude of n bits follows as n raw bits with JPEG-style sign extension (a leading 0 bit - means negative, value = read − (2^n − 1)). -- **B5** (hazy, unverified, no source consulted): there is a small fixed set of Huffman table - pairs (two or three), built into the camera/decoder rather than stored in the file; the - "decoder table" record (see A9) holds an index selecting which set. Within a set, one table - codes the first value of each 64-block and a second table codes the remaining 63 — the - first/rest split is the haziest part of this recollection. -- **B6** (hazy, unverified, no source consulted): a "low bits" mode exists: when enabled, the - 2 least-significant bits of every pixel are stored separately, uncompressed and packed 4 per - byte, at the start of the raw data section, and the Huffman stream codes only the high bits; - the final sample is (decoded << 2) | low_bits. Which cameras/flags enable it is unknown to - me; possibly all 10-bit → 12-bit? Uncertain. -- **B7** (hazy, unverified, no source consulted): decoded differences accumulate into samples - via two running predictors, one for even and one for odd columns (i.e. per CFA column - parity). I do not recall where the predictors reset (per row? per row pair? never after - initialisation to a mid-grey such as 512?) or how the first pixels of the image are formed. -- **B8** (hazy, unverified, no source consulted): bits are consumed MSB-first within each byte; - there is no marker-escaping; I vaguely recall some per-camera quirk where a count of spare - bits or a skipped prefix is involved, but cannot state it. -- **B9** (firm, unverified, no source consulted): the visible image is a crop of the decoded - area; a sensor-info record gives the full decoded width/height and the left/top borders of - the visible region. -- **B10** (hazy, unverified, no source consulted): black level is taken from the masked border - pixels (or a stored value); white-balance coefficients are in a metadata record as per-channel - levels or gains. Formats of those records not recalled. -- **B11** (hazy, unverified, no source consulted): EOS D60 visible size is 3072×2048 (6.3 MP); - the decoded raw area is slightly wider/taller (numbers not recalled). File size around 7–8 MB, - consistent with ~12 bits/pixel compressed losslessly below 2:1. -- **B12** (hazy, unverified, no source consulted): some PowerShot-era CRW variants may store - the raw data uncompressed or in a different packing; unsure whether any in-scope body does. - -## C. Process assumptions - -- **C1** (firm, unverified, no source consulted): ExifTool publishes prose documentation of the - CIFF/CRW structure and tag names (web pages, distinct from its Perl source, which is off - limits under the clean-room rules). -- **C2** (hazy, unverified, no source consulted): Canon once published a CIFF specification - document (circa 1997–98, "CIFF Specification on Image Data File", version ~1.0 r4) that - circulates publicly as a PDF. -- **C3** (hazy, unverified, no source consulted): raw.pixls.us hosts CRW samples for several of - the bodies in A11, some CC0; per-file licences vary, so each file must be checked. diff --git a/crates/crw/clean-room/03-payload-layout.md b/crates/crw/clean-room/01-payload-layout.md similarity index 88% rename from crates/crw/clean-room/03-payload-layout.md rename to crates/crw/clean-room/01-payload-layout.md index eafec02b..14adea06 100644 --- a/crates/crw/clean-room/03-payload-layout.md +++ b/crates/crw/clean-room/01-payload-layout.md @@ -1,10 +1,10 @@ # CRW RawData payload layout (P1) — established by statistics, no decoding -Status: derived 2026-10-12 from the 29-file corpus by `p1_payload.py`, `p1b_markers.py`, -`p1c_lowbits_bounds.py` (logs: `logs/p1-*.txt`, `logs/p1b-*.txt`, `logs/p1c-*.txt`, private -counterparts under `logs/private/`). **The entropy-coded bitstream itself has not been decoded;** -everything here comes from sizes, offsets, byte histograms and ff-digram statistics. -Assumption IDs refer to `00-starting-assumptions.md`; corrections are stated explicitly. +Status: derived 2026-10-12 from the 29-file corpus by statistics alone. The scripts that produced +it (`p1_payload.py`, `p1b_markers.py`, `p1c_lowbits_bounds.py`) and their logs are not part of +this record. **The entropy-coded bitstream itself has not been +decoded;** everything here comes from sizes, offsets, byte histograms and ff-digram statistics. +Corrections are stated against the starting assumptions, by their labels (B2, B3, …). ## 1 The three layouts @@ -17,7 +17,7 @@ dimensions, Q = ⌈w·h/4⌉ (one byte per four pixels — the "low-bits" region | PowerShot 12-bit: G5, G6 | `[514 zero bytes][low bits, Q][stuffed entropy][ff d9 (+00)]` | S−L = 2; first 514 bytes zero; **all** non-stuffed ff (5002 of 5002 on G5, 7928 of 7928 on G6) fall inside [514, 514+Q); the rest of the stream is perfectly stuffed; payload ends `ff d9` | | PowerShot 10-bit: G1 | `[514 zero bytes][stuffed entropy][ff d9 + 00]` | S−L = 3; first 514 bytes zero; zero non-stuffed ff anywhere; payload ends `ff d9 00` | -Private corpus: all 18 D60 files match the EOS row exactly (`logs/private/p1c-bounds-private.txt`). +Private corpus: all 18 D60 files match the EOS row exactly. ## 2 DecoderTable (0x1835) semantics @@ -47,7 +47,7 @@ Private corpus: all 18 D60 files match the EOS row exactly (`logs/private/p1c-bo - Extent Q = w·h/4 exactly, in all 25 files that have the region (B6's arithmetic holds). - D60-class prefix: reshaped at w/4 = 788 bytes/row, the first **7 rows are exactly zero** (5516 = 7×788 bytes) in the public D60 file, and the leading zero run is row-aligned — - strong evidence the region is row-major at 2 bits/pixel (`logs/p1-payload-public.txt`). + strong evidence the region is row-major at 2 bits/pixel. Below the crop-top band the bytes are near-uniform (entropy ≈ 7.99), as LSB noise should be. - D30's region is high-entropy from byte 0 and ends in a run of `ff` bytes (saturated LSBs?); no zero rows — the D30's masked rows evidently carry noise. Placement for D30 is supported @@ -89,8 +89,7 @@ Private corpus: all 18 D60 files match the EOS row exactly (`logs/private/p1c-bo | g5 | 2672×1968 | 4 861 806 | 1 | 4 861 804 | 2 | 1 314 624 (inside L) | — | 514 | | g6 | 3160×2344 | 7 313 156 | 0 | 7 313 154 | 2 | 1 851 760 (inside L) | — | 514 | -(S here is the RawData record size, not the file size. Private D60 table: -`logs/private/p1-payload-private.txt`.) +(S here is the RawData record size, not the file size.) ## 7 Open for P2 (entropy derivation) @@ -102,5 +101,5 @@ Private corpus: all 18 D60 files match the EOS row exactly (`logs/private/p1c-bo explanation to fall out of the geometry work (P3). 5. EOS tail `ff 00 00` vs `00 00` — settle when the stream end is decodable. -P2 requires the approved oracle (LibRaw pre-built binary, run black-box; plan §5a decision 1–2) -for probing and for the low-bits test. **Requested from the maintainer before P2 starts.** +P2 requires the approved oracle (LibRaw pre-built binary, run black-box, per the plan the +maintainer approved) for probing and for the low-bits test. **Requested from the maintainer before P2 starts.** diff --git a/crates/crw/clean-room/01-plan.md b/crates/crw/clean-room/01-plan.md deleted file mode 100644 index 452467f8..00000000 --- a/crates/crw/clean-room/01-plan.md +++ /dev/null @@ -1,337 +0,0 @@ -# Canon CRW (CIFF) raw decoding — clean-room derivation plan - -Written 2026-10-12. **Status: awaiting the maintainer's approval. No decoder code and no table -data exist yet; nothing below §5 phase P1 starts until this plan is approved.** - -Companion documents in this directory: - -- [`00-starting-assumptions.md`](00-starting-assumptions.md) — what was assumed before any - source was consulted (committed first, immutable). Items are cited below as A1–A11 (container), - B1–B12 (raw coding), C1–C3 (process). -- [`sources.md`](sources.md) — every URL opened and search run, with what was taken. -- [`ciff_dump.py`](ciff_dump.py) + [`logs/`](logs/) — the planning-phase container survey. - -Rules in force (unchanged from the task): no raw-decoder source is ever opened, fetched, -searched inside or requested (dcraw, LibRaw, rawspeed, rawloader/rawler, dnglab, darktable, -RawTherapee, libopenraw, ExifTool's Perl code — including licence headers, comments and -mirrors); the earlier `crw-decode` branch is not checked out, diffed, logged or read; public -samples are CC0 files from raw.pixls.us only; the maintainer's private files never enter the -repository and no document ever includes their image content (SHA-256 and size only). - ---- - -## 1 What public documentation establishes (sources, kept separate from assumptions) - -### 1.1 Canon, *CIFF Specification on Image Data File*, v1.0 r4, 1997-12-24 - -Archived PDF (SHA-256 `416d4e49ea3f6ad2ba5ccc76b48ca100605f4a6127816c145e6dd5d6c3b51cf4`, -URL in `sources.md`; local copy in gitignored `plan/crw/`). Canon's -own published specification. It establishes, with exact definitions: - -- **Heap file header** (quoted struct): `byteOrder[2]` (`"II"` little-endian / `"MM"` - big-endian), `headerLength` u32 ("distance from the start of the file to the beginning of the - heap-space"), `type[4]` = `"HEAP"`, `subType[4]` (`"JPGM"` for JPEG-embedded; reserved values - include `"CCDR"`, `"TIFP"`, `"ARCH"`), `version` u32 = 0x00010002, two reserved u32s. -- **Heap** = heap space followed by the `offsetTbl`; a u32 `offsetTblOffset` is stored "in the - final position at the end of the heap", as an offset relative to the start of the heap. -- **offsetTbl** = `numRecords` u16 + that many `DC_RECORD_ENTRY {typeCode u16, length u32, - offset u32}`; offset is "offset value from the beginning of the heap"; `numRecords` may be 0. -- **typeCode bit fields**: bits 15–14 storage (`00` kStg_InHeapSpace, `01` kStg_InRecordEntry — - data held in the 8 bytes of the length+offset fields themselves — `10`/`11` reserved); bits - 13–11 dataType (`000` byte, `001` ASCII string, `010` 2-byte, `011` 4-byte, `100` structure, - `101`/`110` heap record, `111` reserved); bits 10–0 IDCode. Masks (quoted): - `kStgFormatMask 0xc000`, `kDataTypeMask 0x3800`, `kIDCodeMask 0x07ff`. -- Heap records (dataType 101/110) are themselves heaps → the tree structure. -- Null (`kTC_Null` 0x0000) and free (`kTC_Free` 0x0001) records; all records 2-byte aligned, - odd lengths padded. -- Common metadata record structs: ImageSpec (imageWidth, imageHeight, pixelAspectRatio f32, - rotationAngle s32, componentBitDepth, colorBitDepth, colorBW), ImageFormat, CapturedTime - (time_t + zone), ModelName ("two null-terminated strings … manufacturer name … model name"), - and the ImageProps/ShootingRecord/MeasuredInfo/Camera/CameraSpecification hierarchy. -- **The spec does not describe the raw image payload at all.** ImageFormat's fileFormat value 2 - is merely "Reserved (for CanonCRW format)". Everything about the raw coding must come from - sample analysis (and an approved oracle), not from documentation. - -### 1.2 ExifTool prose pages (documentation, not Perl source) - -`exiftool.org/canon_raw.html` (format description) and `TagNames/CanonRaw.html` (tag tables): - -- A `.CRW` file is a heap file: 26-byte header with signature `HEAPCCDR` (= type `HEAP` + - subType `CCDR`), CRWVersion 0x00010002; the root block runs from `headerLength` to EOF; a - subdirectory is a complete block inside its parent's ValueData whose block size is the - entry's Size; offsets are relative to the containing block's ValueData start. -- Tag IDs (TagID = dataType | IDCode, i.e. typeCode & 0x3fff): **0x2005 RawData** ("the bulk - of the CRW file"), **0x2007 JpgFromRaw** (embedded JPEG), 0x2008 ThumbnailImage, **0x1835 - DecoderTable** with sub-values named *DecoderTableNumber, CompressedDataOffset, - CompressedDataLength*, **0x1031 SensorInfo**, 0x1810 ImageInfo, 0x1030 WhiteSample (with - *BlackLevels*), 0x10a9 ColorBalance, 0x10ae ColorTemperature, 0x080a CanonRawMakeModel, - 0x102a CanonShotInfo, 0x1834 CanonModelID, 0x300a ImageProps, 0x2807 CameraObject, 0x300b - ExifInformation, 0x0000 NullRecord, 0x0001 FreeBytes, and others (full list in `sources.md`). -- Neither page describes the raw compression, bit depth or decoder tables' contents. - -### 1.3 Wikipedia, "Camera Image File Format" - -Spec identity and dates; the CRW camera list: EOS D30, D60, 10D, 300D; PowerShot Pro1, G1–G6, -S30–S70. (Matches A11.) - -## 2 Corpus - -### 2.1 Public (CC0, raw.pixls.us; downloaded into gitignored `corpus/crw/`, hashes verified) - -| File | Body | Size (bytes) | SHA-256 | -|---|---|---|---| -| `crw-canon-eos-d30.crw` | EOS D30 | 2 993 260 | `4ee482f2b6f9f1a8d370703ea04e9f2abaa49356973a7f56b229af93a78e8a2e` | -| `crw-canon-eos-d60.crw` | EOS D60 | 8 591 234 | `2b98659ecff170af1dfe44f89b493139e75e995b2979686f6ff0847cfb56b97f` | -| `crw-canon-eos-10d-a.crw` | EOS 10D | 7 787 330 | `b0730eb3a67cc2281a32e0b2ee104199246928d954a68b7e5b72366514c73d02` | -| `crw-canon-eos-10d-b.crw` | EOS 10D | 10 822 040 | `e879c94afdc64dabc88b01ce2d02628aa225259303ec71dd61f7fb0322bfea2d` | -| `crw-canon-eos-10d-c.crw` | EOS 10D | 10 583 702 | `58144118854657df5cd2185df40fecf6b665fc6754fef0416c2f222b68b025c6` | -| `crw-canon-eos-300d-a.crw` | EOS 300D | 9 056 348 | `47da16e1668af5500eea63cf42cf2992d9cb3d32180b8d786f17f1e2a57f39f4` | -| `crw-canon-eos-300d-b.crw` | EOS 300D | 8 389 842 | `9e7f62239fe728046af28d0d4a8a722d35d40146a79794a5bda1775306af4383` | -| `crw-canon-eos-digital-rebel.crw` | EOS 300D (US name) | 10 153 786 | `d6a3131bfac390fde01b96882ec378211f04570c3252708b58f0d05b33a9fdfc` | -| `crw-canon-powershot-g1.crw` | PowerShot G1 | 2 300 702 | `f39be5acf14057ce04527fd8d0574e57ec8ed5fed082d7bfaff763075a44ee8a` | -| `crw-canon-powershot-g5.crw` | PowerShot G5 | 4 894 440 | `bb084223d714871b99de4f549eaf00513e30a7080feeead700252c2008060196` | -| `crw-canon-powershot-g6.crw` | PowerShot G6 | 8 052 032 | `244378e1f5660b6cc6cefc93a75ed4928d0b28aa6304ff327b61c748538f4fef` | - -All 11 are listed CC0 by raw.pixls.us (licence per file verified in the site's data on -2026-10-12; `sources.md`). Each download was checked against the site's published SHA-256. - -### 2.2 Private (maintainer's EOS D60 files, `../RawImages/`, local only — never committed, no -image content in any document) - -18 files, `CRW_0476.crw` … `CRW_2219.crw`, 5 500 958 – 8 628 810 bytes. SHA-256 and sizes -recorded in `sources.md`-adjacent planning logs; full hash list: - -| File | Size | SHA-256 | -|---|---|---| -| CRW_0476.crw | 6 906 290 | `7966aeb5abf3abe80868b8cb1606d671112c42f44d877d3ff79305a52558cacc` | -| CRW_0478.crw | 5 708 018 | `6d07ef616f4b1c7c3025e7af315bb7fb4173ddefea45e2e52d5a4d28e9f2851f` | -| CRW_0486.crw | 8 055 488 | `bf0876f5ba55b2bd120f9588b91f9d4caad54ab73315e542114bf738459d8b3b` | -| CRW_0527.crw | 7 158 482 | `2fd42a69caa20e7c07fe022bd73dfe2626d60a3bacc4228073d9627a5dfa03c1` | -| CRW_0683.crw | 7 453 390 | `c4b320a158b1859fa0709050caddfe2bec0d0a566eafeb17df89cc6b64346a75` | -| CRW_0717.crw | 6 633 986 | `0d6b64935d928e2e0d378eb6efe2b46bed9026ed227111fd47b5fe8b911e22ad` | -| CRW_0765.crw | 5 636 168 | `e8da71d85e07d4157fb5753f850b331dfa6451d7382c4c0d4f9568144e6e8bc5` | -| CRW_0822.crw | 6 478 092 | `817acdc28dd1fe2cd0065b44c63722aaa1f800275e0b0b245ad3b3d964df7c9f` | -| CRW_0879.crw | 6 861 152 | `df66c000ac18b872a69b0fdc171245f89869bd0ad415e117c77e6c27ea93c0ca` | -| CRW_1011.crw | 6 822 000 | `9044f07f30064249cb5ea78bd6ba5d19ae3d2da278dbf7f10aa357872fbbd469` | -| CRW_1153.crw | 7 729 790 | `9cd504ed0991c4c6033d72a2676c778b84d198ddfad202437fcb7f014fa15a17` | -| CRW_1277.crw | 6 850 414 | `c4f0ce8c4ce85d3668b3726c46018d296c784b10a2f0f322acba8b1681fe86f4` | -| CRW_1514.crw | 6 531 360 | `9aaa960461f829b218eaa7f50cf49fa3cb80e92baf9f5aabc03038fddbefe06c` | -| CRW_1646.crw | 6 204 492 | `f9f3dbc2f9aed2cdc2ddccd9a9768a1e02501c337258a211ac9fb663ad78ff74` | -| CRW_1740.crw | 5 500 958 | `033f225fcc8bfdb07b013d971c594bda2adfa0a683c133acc22a171ddcdf95a6` | -| CRW_1776.crw | 6 746 924 | `b07921872c8cf7767841945818a6c8d59553415580d7ebe6a08115964ec6de44` | -| CRW_2035.crw | 6 595 632 | `0ffd9fec448a667b7850ea505b5b117f01c07a98aff5e0b7ec9da7fc5909b4e8` | -| CRW_2219.crw | 8 628 810 | `0460e9ba4884c7e0867570cf71be7a6d2d038fdcecf062fde5cc28422602dd07` | - -## 3 Established from sample files during planning (container level only) - -All from `ciff_dump.py` over the 29 files (logs: `logs/structure-public.txt`, -`logs/private/structure-private.txt`). No raw-data decoding was attempted. - -- **All 29 files parse fully** with the documented container model: `II`, headerLength 26, - `HEAP`+`CCDR`, version 0x00010002; every record resolves in bounds; heap tree ≤ 4 deep - (root → ImageProps → CameraObject/ExifInformation/… → CameraSpecification). Confirms - A1–A8 on real files (details per assumption in §4). -- **Root block layout** in every file: RawData (0x2005) first at heap offset 0, then - JpgFromRaw (0x2007, starts with JPEG SOI), then ImageProps (0x300a). -- **SensorInfo (0x1031)**, 17 u16s beginning with its own length (34): D60/10D/300D all - `(34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0)`; D30 - `(34, 2224, 1456, 1, 0, 56, 12, 2215, 1451, 8, 220, 40, 1244, …)`; G1 - `(34, 2144, 1560, 1, 1, 24, 15, 2071, 1550, 2108, 15, 2111, 1550, …)`; G5 2672×1968; G6 - 3160×2344. For the EOS bodies, fields 5–8 as (left, top, right, bottom) give exactly the - visible size: 3143−72+1 = 3072, 2063−16+1 = 2048, matching ImageInfo (0x1810) = 3072×2048. - (Bears on B1/B9/B11; the remaining fields are candidates for black-measurement windows — to - be established.) -- **ImageInfo (0x1810)** u32s `(3072, 2048, 1.0f, rot, 8, 24, 257)` — the 8/24 bit depths - describe the *full-view image* (per the spec's ImageSpec definition), **not** the raw depth; - rot = 270 on the portrait 300D shots. Corrects the hope that raw bit depth is read from here. -- **DecoderTable (0x1835)** = 4 u32s `(N, 0, 514, L)`: - - N ∈ {0, 1, 2} — all three values observed, varying per shot *on the same body* (private - D60 set has 15× N=1, 2× N=0, 1× N=2; public files: D30 1, D60 0, 10D 0/2/2, 300D 0/2, - Rebel 0, G1 1, G5 1, G6 0). Supports B5's "selector" reading; presumably exposure/ISO - dependent — to be established. - - Third u32 is 514 in all 29 files (meaning unknown; 512+2? — to be established). - - **L vs RawData size S**: on every EOS file (D30, D60, 10D, 300D, 23 files), - S − L = ⌈width×height/4⌉ + 2 or 3 bytes (e.g. D60: 6 694 146 − 5 064 559 = 1 629 587; - 3152×2068/4 = 1 629 584). On every PowerShot file (G1, G5, G6), S − L = 2 or 3 bytes. - Strong container-level evidence for B6 (a 2-bits-per-pixel region, pixels packed 4 per - byte, present on the EOS bodies and absent on these PowerShots), **but its placement, - packing order and meaning are still to be derived** — note the tension with B6's - recollection that low bits appear on PowerShots. -- **RawData byte statistics**: overall entropy 7.84–7.99 bits/byte (dense entropy coding); on - the D60-class files the first 2048 bytes are all zero; on D30 the data is high-entropy from - byte 0; on G1/G5 the first ~522–529 bytes are zero-dominated. No JPEG markers at the start; - JPEG SOI appears only in the separate 0x2007 record. -- The three 10D files and the three 300D/Rebel files are internally consistent per body; - the 18 private D60 files are uniform with the public D60 file. - -## 4 Starting assumptions → experiments - -Status: **spec** = established by §1 documents; **samples** = established by §3 observations; -**open** = still unverified — the listed experiment decides it during derivation. - -| # | Status | Correction / experiment, and what counts as confirmed | -|---|---|---| -| A1 CIFF heap container | spec+samples | Confirmed (header `HEAP`/`CCDR`; all 29 parse). | -| A2 `II` first, LE | spec+samples | Confirmed; `MM` legal per spec — decoder must accept or cleanly refuse it; no `MM` sample exists (§6). | -| A3 u32 header length @2 = 26 | spec+samples | Confirmed (26 in all 29; field is authoritative, not the constant). | -| A4 `HEAPCCDR` @6 | spec+samples | Confirmed, with correction: it is two fields, type `HEAP` + subType `CCDR`; other subTypes exist (`JPGM`…) and must be refused as unsupported. | -| A5 directory via trailing u32 | spec+samples | Confirmed (`offsetTblOffset` in the heap's final position, relative to heap start). | -| A6 u16 count + 10-byte entries | spec+samples | Confirmed (typeCode u16, length u32, offset u32 — in that order). | -| A7 tag bit masks | spec+samples | Confirmed exactly: 0xc000 storage (0x0000 in-heap, 0x4000 in-entry, rest reserved), 0x3800 dataType, 0x07ff IDCode. | -| A8 nested heaps, bounded recursion | spec+samples | Confirmed (dataType 5/6 = heap record); observed depth ≤ 4; parser caps at 8. | -| A9 tag numbers | samples | Mostly right: 0x2005 RawData, 0x1835 DecoderTable, 0x1031 SensorInfo, 0x2007 JpgFromRaw, 0x080a make/model, 0x102a ShotInfo all confirmed; ColorBalance is 0x10a9. DecoderTable is 4×u32, not "16 bytes of table data". | -| A10 .THM sibling + embedded JPEG | part | Embedded JpgFromRaw confirmed (SOI seen); ThumbnailFileName records name a `.THM`, but no `.THM` files are present in `../RawImages` — treat the THM as optional; not needed for decoding. | -| A11 camera list | spec-adjacent | Matches Wikipedia + raw.pixls.us. Later `.CRW`-named CHDK dumps are a different format: detect via header and refuse as unsupported. | -| B1 Bayer CFA, RGGB at origin, full readout incl. masked borders | open | Decode must yield exactly SensorInfo's width×height samples; CFA phase from (i) colour of uniform patches vs the embedded JPEG, (ii) per-channel black/white statistics, (iii) oracle's CFA output. Confirmed when all three agree on all 29 files. | -| B2 10- or 12-bit samples | open | From decoded histograms (max code, gaps), clipped-highlight files, and the low-bits arithmetic (wh/4 = 2 bits/pixel implies the Huffman stream codes depth−2 bits on EOS bodies). Confirmed when the white clip point is stable per body and matches the oracle's scale. | -| B3 not JPEG, bare bitstream | samples (part) + open | No JPEG structure observed; fully confirmed when the derived bitstream model consumes the payload exactly on all files. | -| B4 Huffman run/size DPCM in 64-blocks | open | The central derivation (P2, §5): a candidate code model must decode every file such that (i) the stream is consumed to within the final byte, (ii) every sample lands in [0, 2^depth), (iii) rows have natural image statistics (smoothness, CFA correlation). Confirmed only when one model does this for all 29 files and matches the oracle exactly. | -| B5 small fixed set of table pairs selected by file | samples (part) + open | DecoderTableNumber ∈ {0,1,2} observed varying per shot. Open: the tables' content, whether they live in camera or file (the constant `514` field and the RawData prefix are candidates to rule in/out), first-vs-rest table split. Confirmed when one fixed table set per N decodes every file with that N, across bodies. | -| B6 2 LSBs stored separately, packed 4/byte | samples (part) + open | wh/4-byte region established on EOS bodies — but B6 guessed PowerShots; observed on EOS instead, and placement unknown (D60-class zero prefix suggests the region leads, since masked top rows plausibly have zero LSBs — but D30's leading bytes are high-entropy, so order may differ or D30's masked rows are noisy). Experiments: (i) statistical — LSB region should be near-uniform in lit areas and structured in masked areas consistently across files; (ii) decisive — oracle output codes' 2 LSBs must equal the extracted region bit-for-bit with one fixed packing rule. | -| B7 two column-parity predictors | open | Derived in P2 by fitting predictor variants in the Python model; confirmed when interior and first-row/first-column pixels match the oracle on all files, incl. the masked border. | -| B8 MSB-first bits, no byte-stuffing | open | Both orders tried in P2; the wrong one fails the exact-consumption test almost immediately. Confirmed with B4. | -| B9 visible crop from sensor record | samples (part) + open | Crop rectangle (72,16)–(3143,2063) = 3072×2048 established; confirmed as *the* crop when our visible output aligns with the embedded JPEG (geometry check as in arw6) and the oracle. | -| B10 black from masked pixels / record; WB in metadata | open | Candidates observed: 0x1030 WhiteSample (BlackLevels), 0x10a9 ColorBalance, 0x10a8/0x10aa (u16 arrays that look like per-channel levels). Experiments: correlate record values with decoded masked-area means per channel and with the embedded JPEG's white balance; confirmed when black/WB applied to our decode reproduces the JPEG's grey axis on neutral patches. | -| B11 D60 3072×2048 / slightly larger raw | samples | Confirmed: 3152×2068 raw, 3072×2048 visible. | -| B12 uncompressed CRW variants | open | For the in-scope bodies, none observed (all payloads are high-entropy). CHDK `.CRW` dumps are a different format → refused as unsupported, not decoded. Confirmed by header discrimination tests. | -| C1 ExifTool prose docs exist | confirmed | §1.2. | -| C2 Canon CIFF spec circulates | confirmed | §1.1 (34-page archived PDF). | -| C3 raw.pixls.us CRW samples, mixed licences | confirmed | §2.1 — CC0 for 7 bodies; several classic bodies NC-only (§6). | - -## 5 Derivation plan (phases, freeze, verification) - -Modelled on [`docs/arw6-compression.md`](../../docs/arw6-compression.md): published docs + -samples → Python reference model → **freeze** → black-box verification, no tuning after the -freeze. All phase outputs are documents/scripts in `clean-room/crw/`; each derived fact cites -the experiment (script + log) or document that established it. - -- **P0 — container (mostly done during planning).** Finalize the CIFF parsing notes - (`02-container.md`): header, heap walk, record classes, the records we consume (0x2005, - 0x1835, 0x1031, 0x1810, 0x080a, 0x1834, 0x10a9/0x10a8/0x10aa, 0x1030, 0x180e, 0x5029 …), - endianness rules, malformed-file behaviour. Includes hostile-input rules for the eventual - Rust parser (offsets/lengths checked with `get`, bounded recursion, no panics). -- **P1 — payload partition.** Statistics-only scripts (`p1_*.py`): establish the split of - RawData into (candidate) low-bits region, entropy-coded region, and the 2–3 trailing bytes; - the meaning of DecoderTable fields (incl. the constant 514) by cross-file regression; - per-body low-bits presence. Output: `03-payload-layout.md` with per-file accounting tables. - Still no bitstream decoding. -- **P2 — entropy code derivation.** The core. Hypothesis-driven, in a Python reference model - (`model/`): bit order (B8), code alphabet and block structure (B4), table set per - DecoderTableNumber (B5), predictor/DPCM rules (B7), low-bits recombination (B6), bit depth - (B2). Method: propose a code model; the acceptance tests are *internal* — exact stream - consumption on every file, samples within range, natural-image statistics, cross-file - consistency (same tables for same N across bodies) — iterating over the hypothesis space. - If the oracle (§5a) is approved early, black-box probing is allowed to speed this up - (feeding CC0 files and byte-locally modified CC0 files to the oracle binary and observing - output deltas, as arw6 probed with synthetic tiles); every probe is logged in `sources.md` - style in `04-derivation-log.md`. **No decoder source is consulted regardless.** -- **P3 — geometry, black, white balance, colour.** Visible crop (B9) against the embedded - JPEG at pixel alignment; black level from masked regions + candidate records (B10); white - point per body (B2); WB coefficients from 0x10a9/0x10a8 against the JPEG's neutrals; CFA - phase (B1). Output: `05-geometry-colour.md`. -- **P4 — freeze + verification.** Write `06-crw-format.md`: the complete derived format - (container summary, payload layout, tables as data, algorithm in prose + the Python model's - final hash). **Freeze point: that document's commit.** Only then run the approved oracle - over all 11 public files (and, locally, the 18 private files) and compare decoded samples - exactly; record match/mismatch per file in `07-verification.md` with zero tuning. A - mismatch reopens derivation explicitly: the reopening, its cause and the new freeze commit - are recorded — never silently adjusted. -- **P5 — Rust implementation (separate task(s), after the maintainer approves the derived - format).** `crates/raw` CIFF container + `vendor/crw.rs` decoder; never-crash rules - (hostile offsets, bounded recursion, no unwrap/expect/panic, capped allocations); - unconditional synthetic tests (round-trip via a test-only encoder like `arw6_testenc`, - truncated/hostile input) + corpus-gated tests with pinned sums; `cargo xtask corpus` - entries for 2 pinned CC0 files; docs (`docs/crw-parity.md` or a section in - `docs/raw-parity.md`), parity-checklist/gaps/ROADMAP rows per CLAUDE.md. Perf budget: at - 6 MP these files should decode far inside the 1 s/24 MP export budget; measured in P5. - -### 5a Reference ("oracle") for what the decoded samples should be — **maintainer's decision** - -| Candidate | What it gives | Licence / repo-rule implications | -|---|---|---| -| **(1) LibRaw binary, run as a black box** (as `docs/arw6-compression.md` did: built binary only, source never read) | Exact 16-bit pre-processing codes; the gold standard for B4–B8 | The task forbids fetching LibRaw *source*, so building from source is out for me. LibRaw.org publishes pre-built Windows binaries; I am also forbidden to install/run downloaded programs, so the **maintainer** would download and run it (or approve my running it). Output used as data only; nothing enters the repo but comparison sums. LibRaw is LGPL/CDDL — running a binary copies no code; the house precedent treats this as clean. | -| **(2) Adobe DNG Converter** (free Adobe tool, CRW → DNG; mosaic DNG holds the raw codes losslessly) | Exact codes + Adobe's reading of black/white/WB/crop as DNG tags | Adobe software: CLAUDE.md forbids copying Adobe assets and reading app bundles; running their converter on our CC0/own files and reading the output DNG (an open, documented format) copies no Adobe asset. Residual concerns: Adobe's EULA, and that its interpretation (crop, black) is Adobe's, not the camera's. Same install/run constraint: maintainer runs it. | -| **(3) The embedded JpgFromRaw** | Geometry, orientation, CFA phase, WB neutrals, gross tone sanity | None — it is inside our samples. Lossy and processed, so never a pixel-exact reference. | -| **(4) No-outside-reference checks** | Exact stream consumption, range bounds, masked-area statistics, low-bits cross-checks, Python↔Rust equality, round-trip with our own test encoder | None. Necessary but not sufficient: they validate self-consistency, not correctness of interpretation. | - -**Recommendation:** (3) + (4) always; (1) as the primary oracle, obtained as an official -pre-built binary and run by the maintainer (or by me with explicit approval), used only after -the P4 freeze except for approved black-box probing in P2; (2) optionally as a second opinion -where (1) and our model disagree. **Flagged: the choice, and who runs what, is the -maintainer's.** - -## 6 What cannot be established from the available samples - -- **Bodies with no CC0 file** (raw.pixls.us has them only as CC-BY-NC-SA, which the rules - exclude): PowerShot G2, G3, Pro1, Pro70, S30, S40, S45, S50, S60, S70 (and S2 IS/S3 IS-class - CHDK-era bodies). They can at best be decoded speculatively: if their container metadata - (DecoderTable, SensorInfo, payload accounting) matches a verified body's pattern, the decoder - may accept them with an "unverified" note, as arw6 does for the ILCE-7M5; otherwise they - return unsupported. No G2/G3/S-series verification is possible without new samples. -- **Big-endian (`MM`) files**: none exist here; `MM` handling will be implemented from the spec - and tested only synthetically, or refused — decision in P5. -- **Reserved storage classes (0x8000/0xc000) and dataType 7**: unobserved; refused as - unsupported with a regression test. -- **Per-body table-number coverage**: observed N per body — D30 {1}, D60 {0,1,2}, 10D {0,2}, - 300D {0,2}, G1 {1}, G5 {1}, G6 {0}. Numbers a body never writes can't be verified for it; - cross-body consistency of a given N is testable and part of P2's acceptance. -- **Controlled-exposure pairs** (same scene raw+JPEG at known settings, dark frames, clipped - highlights): none; black/white derivation rests on masked areas, metadata records and the - embedded JPEGs, which is weaker than arw6's lossless/compressed pairs. If the maintainer can - shoot dark-frame and clipped D60 exposures, B2/B10 confirmation strengthens materially. -- **Colour calibration** (camera matrices/profiles): out of scope here, as for every body; - LightCraft's own calibration work, never copied. -- **The remaining SensorInfo fields** (e.g. the `24, 224, 40, 1856` tail on EOS bodies) may - resist interpretation without more bodies; unknowns will be documented as unknown. - -## 7 Open questions for the maintainer — **answered 2026-10-12, plan approved** - -Maintainer's decisions (recorded verbatim in intent, numbered as below): - -1. **Oracle:** no Adobe DNG Converter. LibRaw binary as the arw6 work did; the maintainer will - install it when it is needed. -2. **Black-box probing** before the freeze: allowed, arw6-style (every probe logged). -3. **libopenraw wiki page:** stays unread. -4. **Python reference model:** local-only, like arw6's Phase 0 model (not committed). The - committed evidence set stays as in this commit; the private-file name+size+hash table in - §2.2 is fine to keep. -5. **Scope order:** agreed — D60-class first, then D30, then G1/G5/G6. -6. **NC-licensed bodies:** accept "unverified, header-matched" decoding. The maintainer owns - none of those cameras and has no files from them. -7. **Extra captures:** the maintainer still has the D60 (and a D20) and will provide controlled - shots when the derivation needs them. - -Also noted by the maintainer: the earlier `crw-decode` branch has been deleted and no longer -exists in the repository; the do-not-read rule about it is therefore moot but remains honoured -for any surviving copies. - -### Original questions (for the record) - -1. **Oracle** (§5a): approve candidate (1) (official pre-built LibRaw binary, black box), and - who runs it — you, or me after your explicit approval of the specific download? Allow (2) - Adobe DNG Converter as a secondary reference? -2. **Black-box probing before the freeze** (P2): may the oracle be used interactively on CC0 - and synthetic/modified-CC0 inputs during derivation (arw6-style), with every probe logged — - or only after the freeze? -3. **Prose pages from decoder projects**: the libopenraw freedesktop wiki has a "Canon CRW" - page (surfaced in a search, not opened). It is prose documentation, but authored by a GPL - decoder project and may paraphrase its implementation. Default is to leave it unread; - confirm, or allow it as a documentation source logged in `sources.md`? -4. **Commit policy for `clean-room/crw/`**: plan, sources log, scripts and public-sample logs - are committed (this commit); `logs/private/` is gitignored (may hold personal metadata: - owner name, serial, timestamps). The Python reference model (P2): committed, or local-only - like arw6's Phase 0 model? Private-file hashes in §2.2: confirm you're happy with the - file-name+size+hash table being committed. -5. **Scope order**: derive on the D60-class first (richest corpus: 23 files, one sensor - geometry, all three table numbers), then D30, then G1/G5/G6 — agreed? -6. **NC-licensed bodies** (§6): accept "unverified, header-matched" decoding for G2/G3/Pro1/ - S-series, or return unsupported until samples exist? Do you own any of these cameras, or - CRW files from other bodies (10D/300D/G-series), that could serve as further private - samples? -7. **Extra captures**: if you still have the D60, controlled shots (lens cap dark frame; - clipped sky; grey card at two ISOs, each at all quality settings so all DecoderTableNumbers - appear) would directly confirm B2/B5/B10. Worth doing? - ---- - -*Approved 2026-10-12 (§7). P0 documentation and P1 are under way.* diff --git a/crates/crw/clean-room/02-container.md b/crates/crw/clean-room/02-container.md deleted file mode 100644 index 5b8af04c..00000000 --- a/crates/crw/clean-room/02-container.md +++ /dev/null @@ -1,119 +0,0 @@ -# CRW container (CIFF heap file) — derived parsing notes (P0) - -Status: derived from published documentation and the 29-file corpus; every fact cites its -source. Sources: **[SPEC]** = Canon, *CIFF Specification on Image Data File* v1.0 r4 (see -`sources.md`, hash pinned); **[ET]** = ExifTool's prose pages `canon_raw.html` / -`TagNames/CanonRaw.html`; **[OBS]** = observation of the corpus via `ciff_dump.py` -(`logs/structure-public.txt`, `logs/private/structure-private.txt`). Assumption numbers refer -to `00-starting-assumptions.md`. - -## 1 File header (26 bytes) [SPEC §4.4.1, ET; OBS: all 29 files] - -| Offset | Size | Field | Observed | -|---|---|---|---| -| 0 | 2 | byteOrder: `II` little-endian, `MM` big-endian | `II` in all 29 | -| 2 | 4 | headerLength: file start → heap-space start | 26 in all 29 | -| 6 | 4 | type | `HEAP` in all 29 | -| 10 | 4 | subType | `CCDR` in all 29 (spec reserves `JPGM`, `TIFP`, `ARCH`) | -| 14 | 4 | version: hi16 major, lo16 minor | 0x00010002 in all 29 | -| 18 | 8 | reserved1, reserved2 | — | - -Multi-byte integers throughout the file follow byteOrder [SPEC]. Parser rules: refuse unknown -byteOrder; honour headerLength rather than assuming 26; refuse type ≠ `HEAP`; refuse -subType ≠ `CCDR` as unsupported (a CHDK sensor dump or other container — A11/B12). `MM` files -exist nowhere in the corpus (§6 of the plan); the Rust parser will implement both orders from -[SPEC] and test `MM` synthetically. - -## 2 Heap structure [SPEC §4.1–4.4; ET; OBS] - -- The **root block** spans headerLength..EOF. A **block** = ValueData (heap space), then the - offsetTbl, then optional other data, with a u32 `offsetTblOffset` stored in the final 4 bytes - of the block, relative to the block start [SPEC §4.1.2; ET "DirStart at BlockSize−4"]. -- **offsetTbl** = u16 `numRecords` + numRecords × 10-byte `DC_RECORD_ENTRY {typeCode u16, - length u32, offset u32}`; offsets are relative to the containing block's start [SPEC §4.4.2; - ET]. numRecords = 0 is legal [SPEC §4.1.4]. -- **typeCode bit fields** [SPEC §4.4.2.1, masks quoted in §8.2]: - - bits 15–14, mask 0xc000, storage: `00` in heap space (length/offset meaningful), `01` in - record entry (the 8 bytes of length+offset hold the value, zero-padded; treated as length 8 - unless self-delimiting), `10`/`11` reserved → refuse as unsupported. - - bits 13–11, mask 0x3800, dataType: 0 byte, 1 ASCII (NUL-terminated), 2 u16s, 3 u32s/f32s, - 4 structure (byte-aligned mix), 5/6 heap record (a complete sub-block), 7 reserved → - refuse. - - bits 10–0, mask 0x07ff, IDCode. ExifTool's "TagID" convention = typeCode & 0x3fff - (dataType | IDCode); this document uses that convention for tag numbers. -- **Sub-blocks** (dataType 5/6): the entry's length/offset give the sub-block's extent inside - the parent's ValueData; offsets inside it are relative to its own start [SPEC §4.2; ET; - OBS: all nested directories resolve correctly this way on all 29 files]. -- **Special records** [SPEC §4.4.4]: 0x0000 null (reserves an offsetTbl slot; length/offset - zero), 0x0001 free (reserves heap space). Skip both. -- **Alignment**: record lengths are even; odd-length values are padded [SPEC §4.4.5]. -- Records may appear in any order; data generally precedes its directory ("the natural way to - write information" [ET]). - -## 3 Observed file tree (CRW from cameras) [OBS: all 29 files] - -``` -root (block = headerLength..EOF) -├─ 0x2005 RawData struct, at heap offset 0, the bulk of the file -├─ 0x2007 JpgFromRaw struct, a complete JPEG (starts FF D8) -├─ (G1/G5/G6, D30: 0x2008 ThumbnailImage — JPEG) -└─ 0x300a ImageProps heap - ├─ 0x1803 ImageFormat dword, in-entry - ├─ 0x1810 ImageInfo dword[7] — the FULL-VIEW image: w, h, aspect f32, - │ rotation s32, componentBitDepth, colorBitDepth, colorBW - │ (per [SPEC §5.4.2] ImageSpec; NOT the raw bit depth) - ├─ 0x180e TimeStamp dword[3]: time_t, zone, zoneInfo [SPEC CapturedTime] - ├─ 0x0816/0x0817 file names ascii (e.g. CRW_0099.CRW / CRW_0099.THM) - ├─ 0x0805 description ascii - ├─ 0x3003 MeasuredInfo heap (0x1814 MeasuredEV …) - ├─ 0x2804 ImageDescription heap (0x0805 descr, 0x0815 CanonImageType, e.g. - │ "CRW:EOS D60 CMOS RAW IMAGE") - ├─ 0x2807 CameraObject heap - │ ├─ 0x0810 OwnerName ascii - │ ├─ 0x080a CanonRawMakeModel ascii: "Canon\0\0" [SPEC ModelName] - │ └─ 0x3004 CameraSpecification heap (0x180b SerialNumber, 0x101c BaseISO, - │ 0x080b FirmwareVersion …) - └─ 0x300b ExifInformation heap - ├─ 0x102a CanonShotInfo u16 array (first u16 = byte length) - ├─ 0x102d CanonCameraSettings u16 array - ├─ 0x1031 SensorInfo u16[17]: len(34), rawW, rawH, ?, ?, cropL, cropT, - │ cropR, cropB, then 4+ more u16 (black windows? TBD P3) - ├─ 0x1835 DecoderTable u32[4]: (tableNumber, 0, 514, compressedLength) — see - │ 03-payload-layout.md - ├─ 0x10a9 ColorBalance, 0x10a8, 0x10aa u16 arrays (WB candidates, P3) - ├─ 0x1030 WhiteSample (10D/300D; BlackLevels candidate [ET], P3) - ├─ 0x1834 CanonModelID dword, in-entry - └─ 0x1033 CustomFunctions, 0x10c0/0x10c1/0x10c2, 0x0036 … (not needed for decode) -``` - -Per-body constants observed (rawW×rawH; crop rect, inclusive): - -| Body | Raw | Crop (L,T)–(R,B) | Visible | -|---|---|---|---| -| EOS D30 | 2224×1456 | (56,12)–(2215,1451) | 2160×1440 | -| EOS D60 / 10D / 300D | 3152×2068 | (72,16)–(3143,2063) | 3072×2048 | -| PowerShot G1 | 2144×1560 | (24,15)–(2071,1550) | 2048×1536 | -| PowerShot G5 | 2672×1968 | — (fields differ; P3) | 2592×1944 (ImageInfo) | -| PowerShot G6 | 3160×2344 | — (P3) | 3072×2304 (ImageInfo) | - -(G-series SensorInfo tails differ in shape from the EOS ones — e.g. G1's fields 9–12 are -`2108, 15, 2111, 1550`, a second rectangle to the right of the crop: a plausible black/dark -measurement window. To be established in P3.) - -## 4 Records the decoder will consume - -0x2005 RawData (payload), 0x1835 DecoderTable, 0x1031 SensorInfo, 0x1810 ImageInfo (visible -size cross-check + rotation), 0x080a make/model + 0x1834 CanonModelID (dispatch), 0x10a9 / -0x10a8 / 0x10aa / 0x1030 (black & WB, P3), 0x180e TimeStamp, 0x2007 JpgFromRaw (preview + -geometry verification), 0x102a CanonShotInfo (ISO etc. for metadata display). - -## 5 Hostile-input rules for the eventual Rust parser (never-crash) - -All reads through checked `get`-style access; every offset/length validated against the -containing block before use; offsets/lengths are u32 → promote to u64/usize with checked math, -no wrapping; recursion over sub-blocks bounded (observed depth 4; cap at 16 with a clear -"malformed: heap nesting too deep" error); a record table whose 2 + 10·numRecords exceeds -offsetTblOffset→block-end space is malformed; in-entry values are exactly the 8 bytes of -length+offset; unknown storage class, dataType 7, unknown subType, or missing mandatory -records (0x2005, 0x1031, 0x1835) → typed unsupported/malformed errors, never a panic; -allocations sized by header fields capped against the actual file size. diff --git a/crates/crw/clean-room/06-crw-format.md b/crates/crw/clean-room/02-crw-format.md similarity index 94% rename from crates/crw/clean-room/06-crw-format.md rename to crates/crw/clean-room/02-crw-format.md index b61ecf94..7b5c855b 100644 --- a/crates/crw/clean-room/06-crw-format.md +++ b/crates/crw/clean-room/02-crw-format.md @@ -5,13 +5,13 @@ must reopen it explicitly with a recorded cause and a new freeze commit. **Revision 2 (2026-10-13), cause:** the maintainer supplied 1,516 D60 files, 50 times the first corpus. 223 of them failed on code-table entries the first corpus never exercised, -and 9 were misread as having no low-bits plane. Two corrections, both recorded in -`04-derivation-log.md` and re-verified in `07-verification.md`: the low-bits presence rule +and 9 were misread as having no low-bits plane. Two corrections, both recorded in the +derivation log (not part of this record) and re-verified in `03-verification.md`: the low-bits presence rule (§1) gained the header-position test, and the selector 0 and 1 second tables were completed (§3). Nothing else in the model changed, and every file of the first corpus still verifies. Everything here was derived from the CC0/private sample files and black-box probing of LibRaw 0.22.2 (`unprocessed_raw.exe`), per the approved plan; no -decoder source was consulted. The chronological evidence is `04-derivation-log.md`; the -container and payload groundwork are `02-container.md` and `03-payload-layout.md`. +decoder source was consulted. The chronological evidence is the derivation log, which is not part +of this record; the payload groundwork is `01-payload-layout.md`. ## 1 Payload layout @@ -92,7 +92,7 @@ superset of each file's exercised codes: every row was learned from a real strea position and validated by the decode reaching the end of that frame; rows never exercised by two files may in principle be mergeable duplicates of canonical slots, which the Rust implementation may normalise, provided the verification table of -`07-verification.md` still reproduces exactly. +`03-verification.md` still reproduces exactly. ## 4 Not yet established / out of scope here @@ -110,6 +110,6 @@ which the Rust implementation may normalise, provided the verification table of ## 5 Verification -See `07-verification.md` (written after the freeze): full-frame exact-match decode +See `03-verification.md` (written after the freeze): full-frame exact-match decode against LibRaw 0.22.2 for every sample file, public and private, with zero tuning after the freeze commit. diff --git a/crates/crw/clean-room/07-verification.md b/crates/crw/clean-room/03-verification.md similarity index 91% rename from crates/crw/clean-room/07-verification.md rename to crates/crw/clean-room/03-verification.md index e379e833..3afaeefd 100644 --- a/crates/crw/clean-room/07-verification.md +++ b/crates/crw/clean-room/03-verification.md @@ -1,6 +1,6 @@ # CRW decode verification (P4) — against LibRaw 0.22.2, zero tuning -Method: the frozen model of `06-crw-format.md` plus the frozen tables +Method: the frozen model of `02-crw-format.md` plus the frozen tables (`logs/table_sel{0,1,2}.json`) were run by the local Python reference decoder over every sample file, comparing **every sensor sample** (full values, low bits merged) against `unprocessed_raw.exe -T` output. "EXACT" means zero mismatching samples over the whole @@ -8,8 +8,8 @@ frame. The table extraction itself consumed these same files (known-plaintext le so this is a consistency proof against the oracle, not a blind holdout; the cross-body sharing (tables learned on one body, exact on others) is the strongest independent check. -Private files are the maintainer's own D60 photos (hashes in `01-plan.md` §2.2); only -their verdicts are recorded here, never content. +Private files are the maintainer's own D60 photos (hashes in `logs/private-corpus.tsv`); +only their verdicts are recorded here, never content. ## Results @@ -46,13 +46,13 @@ their verdicts are recorded here, never content. | `private/CRW_2219.crw` | 2 | 3152x2068 | **EXACT** | **28 of 29 files decode EXACT** — all 10 usable public CC0 files and all 18 private -D60 files; the PowerShot G5 is the known exception (`06-crw-format.md` §4). +D60 files; the PowerShot G5 is the known exception (`02-crw-format.md` §4). ## Notes - The final 1–3 samples of each frame follow the stream-end rule (chains persist); every "EXACT" verdict includes them. -- PowerShot G5: see `06-crw-format.md` §4 — detected and excluded; embedded-JPEG +- PowerShot G5: see `02-crw-format.md` §4 — detected and excluded; embedded-JPEG fallback until resolved. - The G1 (10-bit, no low-bits plane) verifies the low-bits presence rule; the G6 (12-bit, different geometry, w not divisible by 64) verifies the row-reset and @@ -62,7 +62,7 @@ D60 files; the PowerShot G5 is the known exception (`06-crw-format.md` §4). The maintainer supplied 1,516 Canon EOS D60 files (private; local use only, nothing copied into the repository, verdicts only recorded here). With the revision-1 decoder, -223 failed on unseen code-table entries and 9 were misread as 10-bit (`06-crw-format.md`, +223 failed on unseen code-table entries and 9 were misread as 10-bit (`02-crw-format.md`, revision 2). After the two corrections every file was compared against LibRaw 0.22.2 `unprocessed_raw.exe -T`, every sensor sample, with the Rust decoder (`plan/crw/model/sweep_all.py`, local): diff --git a/crates/crw/clean-room/08-color.md b/crates/crw/clean-room/04-color.md similarity index 97% rename from crates/crw/clean-room/08-color.md rename to crates/crw/clean-room/04-color.md index 8a04151c..39088859 100644 --- a/crates/crw/clean-room/08-color.md +++ b/crates/crw/clean-room/04-color.md @@ -25,7 +25,7 @@ vs −0.67 GRBG). Unsampled PowerShots default to RGGB, unverified. ## 2 Active area, black level, white level - **Active area**: SensorInfo (0x1031) values 5–8 are an inclusive crop (left, top, right, - bottom); valid on every corpus body (G5/G6 included). See `02-container.md` §3. + bottom); valid on every corpus body (G5/G6 included). - **Black level**: measured from the masked borders. Observed (oracle data, per body class): the left border is dark up to 8 columns before the active area on the EOS bodies (≈127–129 DN on D60/10D, ≈128 on D30, ≈124–127 on 300D) but transitions bright earlier on @@ -105,8 +105,8 @@ vs −0.67 GRBG). Unsampled PowerShots default to RGGB, unverified. on all 10 decodable public corpus files, holdout RMS 0.009–0.043, including the D30 and the PowerShots, whose missing white balance the fitted matrix absorbs. **Update 2026-10-25:** the EOS D60 has dual-illuminant matrices of our own, fitted to - ColorChecker Classic shots under a tungsten bulb and direct sun (`10-chart-calibration.md`, - `crates/raw/src/chart_table.rs`). They give its white balance a Kelvin scale and are its + ColorChecker Classic shots under a tungsten bulb and direct sun (the fitting is part of + LightCraft's colour work, `crates/raw/src/chart_table.rs`). They give its white balance a Kelvin scale and are its colour when the JPEG fit fails; the look stays the JPEG fit. The other CRW bodies have none. ## 4 Orientation and capture time diff --git a/crates/crw/clean-room/04-derivation-log.md b/crates/crw/clean-room/04-derivation-log.md deleted file mode 100644 index 295b9e7a..00000000 --- a/crates/crw/clean-room/04-derivation-log.md +++ /dev/null @@ -1,315 +0,0 @@ -# CRW entropy-coding derivation log (P2) - -Chronological record of experiments and oracle probes, arw6-style. The oracle is LibRaw -0.22.2, official pre-built Windows binaries installed by the maintainer in -a local `LibRaw-0.22.2\bin` folder (2026-10-12); **only `bin\unprocessed_raw.exe` (plus -`libraw.dll`) is executed, as a black box. The LibRaw source tree that ships alongside those -binaries is never opened, read or searched** — all structure below comes from feeding inputs -to the binary and observing outputs. The Python reference model lives in the gitignored -`plan/crw/model/` (maintainer decision #4); findings and probe campaigns are recorded here. - -## 2026-10-12 — oracle baseline runs - -`unprocessed_raw.exe -T ` over 7 public CC0 samples (D30, D60, 10D-a, 300D-a, G1, G5, -G6), each copied to a scratch directory first. Output: 16-bit TIFF of the full raw frame. -Oracle frame sizes equal SensorInfo raw sizes on all 7. Oracle "margins" (active area) differ -from SensorInfo's crop rectangle (e.g. D60: LibRaw top=12/left=64 vs SensorInfo (72,16)) — -LibRaw's own choice, noted, not adopted. - -## 2026-10-12 — low-bits region: CONFIRMED (B6 settled) - -`p2_oracle_lsb.py` (committed) compares the oracle's pixel LSBs against the payload's -candidate low-bits regions, both packing orders: - -| File | Region [0:Q) LSB-first packing | Notes | -|---|---|---| -| D60, 10D-a, 300D-a, D30 | **100.000000 %** (1 629 584/1 629 584 … 809 536/809 536 bytes) | EOS prefix confirmed | -| G6 | **100.000000 %** (1 851 760/1 851 760) | at [0:Q), **not** [514:514+Q) — P1's reading of the G6 layout corrected: G6 low bits precede the 514 block, same as EOS | -| G5 | 47.34 % at [0:Q), 0.39 % at [514:514+Q) | anomaly, parked (scope decision #5: D60-class first) | -| G1 | no region (0.38 % everywhere) | 10-bit body, no low bits ✓ | - -**Packing: LSB-first** — byte b holds pixels 4i..4i+3 with pixel 4i in bits 1:0 and pixel -4i+3 in bits 7:6, row-major over the full raw frame. MSB-first packing scores ~6 %. - -Oracle code ranges: D60 max 4000, 10D max 4003, 300D max 4008, D30 max 3206, G6 max 4095 -(12-bit class); G1 max 936 (10-bit). G5/G6 min ≈ 101/102 (no black subtraction in the oracle). - -Corrections this forces on `03-payload-layout.md`: G6 (and presumably the G5 pattern, once -understood) is `[low bits Q][514-byte block][stuffed entropy][ff d9]`; the G6 "514 zeros at -offset 0" measured in P1 were the low bits of its first 2056 pixels (top masked row), and the -block after the low bits is *not* all-zero on G6. The EOS picture is unchanged. - -## 2026-10-12 — entropy stream: structure findings from known-plaintext analysis - -Using the D60 public file and its oracle decode (scripts `plan/crw/model/align.py`, -`sweep.py`, local): - -- Budget check: unstuffed entropy = 5 049 856 bytes = 6.198 bits/pixel. Under the - two-predictor diff model (B7) E[category] = 3.23 and H(category) ≈ 2.9 bits — model - "code(k) + k magnitude bits" fits the budget almost exactly. Single-predictor E[k] = 5.9 - does not fit. **Two predictors by column parity supported.** -- **Bit order is LSB-first within each byte** (after ff00 unstuffing). Hand-verified at two - anchors: stream start decodes as code `100` (stream order) + 10 magnitude bits = 511 = - JPEG-style magnitude of −512 (first pixel 0 against predictor 512); the first token of row 7 - (byte-aligned at bit 20776) decodes as a 2-bit code + 6 magnitude bits = +55 against - predictor 0 carried from row 6. B8's MSB-first recollection is **refuted**; so is its - "no byte-stuffing" (every 0xff in the entropy data is followed by 0x00, which the decoder - must strip — `03-payload-layout.md`). -- Predictors persist across rows (row 7's first diff is +55 against the row-6 value 0), i.e. - no per-row reset; image start uses 512/512 (10-bit "hi" domain: pixel = hi·4 + lowbits). -- Zero diffs are **run-coded** (a plain per-value zero code is inconsistent with the bits in - the 7 all-zero top rows, which compress to a strictly periodic 7-bit token). -- JPEG-style magnitude convention (negative d stored as d + 2^k − 1) consistent at both anchors. - -## 2026-10-12 — black-box probe campaign 1 (D60 body, patch-in-place) - -Harness `plan/crw/model/probe.py` (local): takes the public D60 file, zeroes the low-bits -region (so TIFF pixel = hi·4), overwrites the entropy region with chosen bits (LSB-first -packed, ff restuffed), optionally patches the DecoderTable selector, runs the oracle, reads -decoded rows. First probes (filler = repeated bit): - -| Entropy bits | Row 0 decoded (hi) | Reading | -|---|---|---| -| all 0 | 497 505 490 498 483 491 476 484 … (parities fall by 7) | repeated all-zero token = k=3 code + mag `000` = −7 per parity; first token longer (−15, k=4) | -| all 1 | 512 512 512 … | all-ones token(s) = zero diffs | -| `100` + 511 + zeros | 16144 505 16137 498 … | no clamping in the oracle; convention probe | - -Then `plan/crw/model/treewalk.py`: adaptive prefix expansion. Three aborted attempts taught -the method its final shape: (1) a race — the first walker's process survived its kill and -fought a second one over a shared probe file (fixed with per-PID work files; both result -sets discarded as corrupt); (2) enumerating magnitude bits explodes 2^k-fold; (3) leaf -detection via an assumed magnitude convention fails because the convention is *not* plain -JPEG (single-bit probes behind a known codeword give non-binary-weight responses). - -## 2026-10-12/13 — truncation probing of the real stream (campaign 2) - -`plan/crw/model/boundary.py` / `boundary2.py`: feed the oracle the real stream truncated at -bit B plus a constant filler. `boundary2`'s agreement scan (decode under 0-filler AND -1-filler; count leading pixels where both agree and match the real decode) yields **exact -token boundaries**: agreement can only grow when a token completes inside [0,B). - -Zero-region result (D60, table 0, bits 0–120 ≈ pixels 0–321, all-zero rows; data in -`bounds2_zero.json`, local): - -| token bits (stream order, LSB-first in bytes) | len | emits pixels | -|---|---|---| -| `1001111111111111` | 16 | 1 (pix 0, d = −512) | -| `011111111111100111111111` | 24 | 2 (zeros) | -| `11111011` | 8 | 61 → ends exactly at pixel 64 (EOB) | -| `0000101111100000` | 16 | 1 (pix 64, first of block 1) | -| `1110000` | 7 | 1 | -| `1` | 1 | 62 → ends exactly at pixel 128 (**1-bit EOB**) | -| `11111011` | 8 | 2 | -| `11111101` | 8 | 63 (ends at pixel 193) | -| `0111111` | 7 | 1 | -| `010111111` | 9 | 63 | -| `11011111` | 8 | 63 | -| `11101111` | 8 | 1 | - -Established: **64-pixel block structure** (B4's "blocks of 64" holds); an EOB-style token -ends a block; the same bit string decodes differently in different blocks (`11111011` = EOB -in block 0 but a 2-pixel token later) → at least two tables/contexts in play (B5's two-table -recollection pointing the right way). First-of-block tokens are long (16 bits here, carrying -the −512 initial diffs). The value-token magnitude convention is still open — it is *not* -plain JPEG — and is being extracted from a second agreement scan over row 7's textured data -(`bounds2_row7.json`). - -## 2026-10-13 — flip probing (campaign 3): code/magnitude split read directly - -`plan/crw/model/flipscan.py`: flip one bit of the real stream at a time (real -continuation), diff the full decode against the real decode. A flipped magnitude bit -changes exactly one pixel by a power of two; a flipped code bit derails the parse. - -First result, the pixel-0 token `1001111111111111` (16 bits): bits 0–7 are the code -(`10011111`), bits 8–15 are an 8-bit magnitude, weights 1,2,4,…,128 in stream order -(LSB-first), and the decoded value responds **linearly** across the would-be JPEG sign -boundary (pixel0 = m − 255 over the whole probed range m = 127…255): the magnitude -convention is offset-style, not JPEG's two-branch rule, at least for this token. A full -flip scan over a textured region (row 11) is mapping code and magnitude bits for ~25 -consecutive tokens. - -## 2026-10-13 — bit order corrected: MSB-first; token structure pinned - -A systematic contradiction (magnitude fields appearing split across bytes, "token -boundaries" clustering at 8 bits) resolved when the flip map was re-plotted in MSB-first -bit coordinates: an early mis-step had locked the analysis into LSB-first reading, which -mirrors every byte. **The stream is MSB-first after ff00 unstuffing.** In MSB coordinates -every magnitude field is contiguous and the JPEG rule is confirmed by exact branch-flip -deltas (e.g. flipping the top magnitude bit of a k=5, m=2 token: −29 → +18, Δ+47 observed). - -Flip-verified token chain (D60 table 0, pixels 34994–34999, bits 20784–20819): - -| bits | table | meaning | pixels | -|---|---|---|---| -| `01` + `00010` | A | k=5, m=2 → −29 | 34994 | -| `11111100` + `11110` | B | run 1 + k=5, m=30 → +30 | 34995 (zero), 34996 | -| `1100` + `1` | A | k=1 → +1 | 34997 | -| `00` + `010` | B | k=3, m=2 → −5 | 34998 | -| `00` + `010` | A | k=3 → −5 | 34999 | - -Established: -- **Two Huffman tables, strictly alternating per token (A, B, A, B, …)** — forced by - prefix-freeness: the verified codes cannot coexist in one table or in parity-assigned - tables, and alternation renders both tables prefix-free. (This refines B5.) -- Tokens carry a **(zero-run r, size s)** pair: r zero-diff pixels plus one value of - size s; s = 0 codes are pure runs/EOB. Predictors: two, by pixel parity, init 512, - carried across rows (flip cascades propagate stride-2 to the image end). -- The 10-bit "hi" plane is nearly constant in dark/flat areas — the reason the low-bits - plane exists: the sensor noise lives in the 2 raw LSBs, keeping the entropy-coded - plane highly compressible. - -Open: exact placement of the bundled zeros (before vs after the value) in corner cases, -EOB/block semantics, the first-token special case at pixel 0, and the complete code -tables for DecoderTableNumber 0, 1, 2 (being extracted by continued flip scans plus the -known-plaintext aligner `plan/crw/model/aligner4.py`). - -## 2026-10-13 — complete token grammar established (D60, table 0) - -The MSB-faithful agreement scan (`bounds3_row11_msb.json`, 187 exact micro-boundaries over -bits 20776–21500) plus a merge analysis settled the remaining structure. Sub-boundary rises -occur inside tokens (fillers can reproduce zero pixels), and merging every zero segment into -the following value yields a token stream that is fully consistent — zero semantic conflicts -and zero prefix violations — under exactly one model: - -- **64-pixel blocks** (image order). The token that starts at a block's first pixel comes - from a **"first" Huffman table**; every other token from a **"second" table**. (The - apparent two-alternating-tables stage was a phantom: fitted as alternating tables, the two - halves come out *identical*, collapsing to one "second" table. B5's recollection — "one - table for the first element of a block, another for the rest" — was exactly right.) -- **Token = code → (run r, size s)**: r zero-diff pixels, then one value of size s coded as - s JPEG-rule magnitude bits (MSB-first). s = 0 codes are EOB (zeros to end of block). -- Two DPCM predictors by pixel parity, init 512, never reset. - -Second-table codes verified so far (D60, DecoderTableNumber 0): `00`=(0,3), `100`=(0,2), -`101`=(0,5), `1100`=(0,1), `111011`=(1,2), `111100`=(1,3), `111101`=(1,1), `11111100`=(1,5), -`1111111000`=(2,1), `1111111100`=(2,3), `11111110111100`=(16,1), and 16-bit codes -(3,1) (4,1) (4,2) (5,1) (5,4) (5,5) (6,3) (6,4) (8,1). First-table: `101`=(0,2), -`111110`=(0,10) at pixel 0. The code set has a canonical Huffman shape. - -A seeded known-plaintext aligner (`aligner6.py`) is now walking the whole 6.5-Mpixel stream -to learn the remaining codes of both tables and verify every pixel; the same harness then -runs on files with DecoderTableNumbers 1 and 2 and on the other bodies. - -## 2026-10-13 — the carry chain, and the first/second table split confirmed - -Synthetic sentinel probes plus the zero-region MSB scan established two more rules: - -- **Carry on block DC**: each 64-pixel block's first diff is coded relative to the - previous block's *combined* first diff (`diff0 += carry; carry = diff0`). This single - rule reconciles pixel 0 (coded −512), pixel 64 (coded +512 → effective 0) and pixel - 128 (coded 0 → effective 0) with plain parity chains init 512 — the earlier - "predictor anomalies" were all carry. -- **The first-of-block token really does use a separate Huffman table** (`101` = (0,2) - there vs (0,5) in the main table, verified by decode-iteration at pixel 35008), i.e. - the classic first/second table structure of B5. -- The S-EOB of the main table is 9 bits (`111111011`); agreement boundaries inside - all-zero blocks proved unreliable (fillers self-heal on zeros), so zero-region token - readings are being re-derived by decode-iteration rather than scans. - -Current state: the main table has ~20 verified codes; a decode-and-learn loop -(`plan/crw/model/aligner8.py`) walks the real stream, decoding deterministically with -known codes and learning first-table/main-table rows at each stall, each verified -against the oracle pixels. - -## 2026-10-13 (cont.) — row reset; full model assembled - -Two refinements completed the structural model, each validated against the stream: - -- **Chains reset to 512 at every row start** (not row padding — a padded-grid variant was - tried and refuted by block alignment at pixel 35008). With the reset, every all-zero row - needs one k=10 "−512" value token at its first pixel, which is exactly what the stream - shows: a 15-bit second-table code (15,10) at the first row crossing, mag 511 → −512. -- The verified second table gains `01` = (0,5) (flip-verified at pixel 34994); both `01` - and `101` map to (0,5) — duplicate leaves are legal and present. - -Model now complete: flat 64-sample blocks; first/second Huffman tables (first = the token -starting each block); (run ≤ 15, size) leaves with JPEG MSB magnitudes; (0,0) mid-block = -EOB; per-block DC carry; parity chains init 512, reset at row starts; low-bits recombined -as the 2 LSBs. Remaining: complete code tables per DecoderTableNumber, extracted by the -known-plaintext walker (`aligner11.py`), then full-corpus verification. - -## 2026-10-13 — SELECTOR-0 TABLES COMPLETE; four bodies decode end-to-end - -The greedy known-plaintext extractor (`plan/crw/model/aligner12.py`: deterministic decode -with deep trial-validation per unknown code, flip-scans resolving the few ambiguous sites) -completed the DecoderTableNumber-0 Huffman tables. Results: - -- **D60 (public): the full 6 518 336-sample frame decodes from bit 0 against the oracle, - pixel-exact,** in seconds, with only the final 3 samples open (stream tail handling). -- **10D, 300D and Digital Rebel (selector 0) decode end-to-end with the same table** — - per-selector tables are shared across bodies, as the plan hoped. Each file exercised a - few additional rare high-k codes (merged). -- Final selector-0 tables: 11 first-table codes, 131 second-table codes (incl. the 16-bit - `11111111xxxxxxxx` run/size family); saved as `logs/table_sel0_d60class.json`. -- The decisive last fix: a hand-guessed 15-bit seed blocked the true 11-bit (1,7) code by - prefix; a flip-scan at the stall site read the real code and magnitude (m = 83) directly. - -Selector 1 (from the D30) and selector 2 (from the second 10D file) extractions running -with the same tooling. Remaining after that: G1/G5/G6 PowerShot handling (10-bit G1; the -G5 low-bits anomaly), the stream-tail rule, then the frozen format document and the -Python reference decoder for the P4 verification pass. - -## 2026-10-13 — selectors 1 and 2; G1, G6, private files; G5 isolated - -- Anchor hunting (truncation-probe agreement at byte boundaries + bit-level pinning) - plus structural bootstrap (reading each file's opening k=10 tokens and the steady - zero-block unit directly off the bits) made extraction routine for new table sets. -- **Selector 1** (extracted from private CRW_0476, topped up from the public D30): - D30 **EXACT**, PowerShot **G1 EXACT** (10-bit, no low-bits plane — detected by the - frozen unstuffed-ff scan), CRW_0476 **EXACT**. Cross-body sharing confirmed again. -- **G6 EXACT** with the selector-0 table once the low-bits rule was frozen (presence - by the ff-scan; entropy always at low-bits-size + 514). -- **G5 remains anomalous**: its low-bits plane matches the oracle's LSBs ~97.5 % in the - top rows (not 100 %), degrading to 0 % in the lower half, and no constant bit-shift - explains it; its entropy stream also diverges from the selector-1 parse within the - first rows. G5 is documented as unresolved and will be refused (or preview-only) by - the decoder pending further work. -- Selector 2 (from the public 10D-b) extraction completing. - -## 2026-10-13 — P2 complete; tables frozen; 28 of 29 files verify EXACT - -A final systematic fix (first-table magnitudes reach s = 11+: the carry-coded block DC -spans ±2046) unlocked selector 2 (extracted from the 300D-b after the saturated 10D-b -proved hostile learning ground). Table top-ups over every private file completed the -three table sets (12 first + 154/161/162 second codes). The final sweep: **28 of 29 -corpus files decode with zero mismatching samples** against LibRaw 0.22.2 — every EOS -body and selector, the 10-bit G1, the G6, and all 18 private D60s. The PowerShot G5 -remains the open anomaly (stream decodes, values diverge from sample 2056; low-bits -plane fits no tested layout) and is excluded pending follow-up. - -Frozen at this commit: `06-crw-format.md` (the format), `logs/table_sel{0,1,2}.json` -(the tables), `07-verification.md` (the evidence). P3 (black/WB/colour records) and P5 -(the Rust decoder) proceed only after maintainer review, per the plan. - -## 2026-10-13 — P5: Rust decoder lands; P3 colour/geometry derived - -The maintainer approved the freeze. `crates/raw/src/vendor/crw.rs` implements the frozen -model (tables generated from `logs/table_sel{0,1,2}.json` into `crw_tables.rs`); the Rust -decoder matched the oracle **sample-exact on all 28 decodable corpus files on its first -run** (`crates/raw/tests/crw_oracle.rs`, ~65 ms per 8 MP file). The G5 is refused by model -string. Colour/geometry work (CFA phases by chroma correlation + green-diagonal tests, -border black levels, 0x10a9/0x10aa white balance, rotation and timestamps) is documented in -`08-color.md`; ExifTool prose pages and a raw.pixls.us repository-listing fetch (to recover -the 11 corpus files' stable URLs for `cargo xtask corpus`) are logged in `sources.md`. - -## 2026-10-13 — Freeze reopened (revision 2): 1,516 D60 files, two corrections - -The maintainer put all his D60 files in `../RawImages` (1,516). The revision-1 decoder -failed on 232 of them, which the 29-file corpus could not have shown: - -- **223 "undecodable entropy code".** The second tables were learned from exercised codes - only: selector 0 had 154 of 162 symbols, selector 1 had 161. Laying out the 16-bit code - range showed exactly eight unassigned slots inside selector 0's range and one inside - selector 1's, against exactly eight and one missing symbols: (7..14, 10) and (14, 10). - For selector 1 the assignment follows by elimination; the learner (`aligner12.py`, same - known-plaintext method, LibRaw as the black box) confirmed it and assigned selector 0's - eight from the failing files. -- **9 files decoded as 10-bit garbage.** Their low-bits plane holds no 0xff byte, because - the lowest bit of nearly every sample is 0, so the revision-1 presence rule said "no - plane". My first reading (an all-zero plane) was wrong: I had counted 0xff bytes, not - nonzero bytes; the plane has 1.5 M nonzero bytes. The rule now also accepts the zero - header found right after the plane. - -Result: 1,516 of 1,516 EXACT, and the first corpus unchanged (`07-verification.md`). -A lesson for the remaining unverified bodies: a table learned from a few files is complete -only when its symbol count is; selector 2 was already at 162. diff --git a/crates/crw/clean-room/09-private-corpus.md b/crates/crw/clean-room/05-private-corpus.md similarity index 95% rename from crates/crw/clean-room/09-private-corpus.md rename to crates/crw/clean-room/05-private-corpus.md index e1b4566b..69ca4c9f 100644 --- a/crates/crw/clean-room/09-private-corpus.md +++ b/crates/crw/clean-room/05-private-corpus.md @@ -7,9 +7,9 @@ sample-exact against LibRaw 0.22.2 `unprocessed_raw.exe -T`. | Set | Files | What it is | What it established | |---|---:|---|---| -| `RawImages/` | 1,516 | Everyday photos, 2005–2014, all Auto white balance; ISO 100 (925), 200 (197), 400 (287), 800 (34), 1000 (73) | Freeze revision 2: two incomplete code tables and the plane-without-0xff case (`06-crw-format.md`); the low-ISO clip level (`08-color.md`); the D60 colour profile (`assets/camera-profiles/Canon_EOS_D60.json`, 1,424 usable photos) | -| `Cap&WhiteOut/` | 4 | Lens cap and blown-out frames, ISO 100 and 1000 (2026-10-13) | Black level at ISO 100 matches the image area; at ISO 1000 the border reads about 4 DN high. Clip level about 4000 at ISO 100, 4095 at ISO 1000 (`08-color.md`) | -| `WBModes/` | 7 | One overcast scene at Auto, Daylight, Cloudy, Tungsten, Fluorescent, Flash, Custom; ISO 100, within a minute (2026-10-14) | White-balance setting n uses ColorBalance slot n (`08-color.md`) | +| `RawImages/` | 1,516 | Everyday photos, 2005–2014, all Auto white balance; ISO 100 (925), 200 (197), 400 (287), 800 (34), 1000 (73) | Freeze revision 2: two incomplete code tables and the plane-without-0xff case (`02-crw-format.md`); the low-ISO clip level (`04-color.md`); the D60 colour profile (`assets/camera-profiles/Canon_EOS_D60.json`, 1,424 usable photos) | +| `Cap&WhiteOut/` | 4 | Lens cap and blown-out frames, ISO 100 and 1000 (2026-10-13) | Black level at ISO 100 matches the image area; at ISO 1000 the border reads about 4 DN high. Clip level about 4000 at ISO 100, 4095 at ISO 1000 (`04-color.md`) | +| `WBModes/` | 7 | One overcast scene at Auto, Daylight, Cloudy, Tungsten, Fluorescent, Flash, Custom; ISO 100, within a minute (2026-10-14) | White-balance setting n uses ColorBalance slot n (`04-color.md`) | | `TripodGrayCard/` | 14 | The same card filling most of the frame, tripod, f/8, ISO 100; two brackets (Auto 1/16–1/6 s, Cloudy 1/12–1/5 s), each starting and ending on the same speed; 14 s each (2026-10-14) | Black level confirmed; sensor linear; the spill strip (below) | | `GrayCard/` | 20 | An 18 % grey card (film-era), handheld, overcast; 5 frames at f/4 with exposure compensation, 15 at f/1.8 stepping 1/640–1/1600 s under Cloudy, Daylight and Auto (2026-10-14) | See below | diff --git a/crates/crw/clean-room/10-chart-calibration.md b/crates/crw/clean-room/10-chart-calibration.md deleted file mode 100644 index beb24d0b..00000000 --- a/crates/crw/clean-room/10-chart-calibration.md +++ /dev/null @@ -1,97 +0,0 @@ -# EOS D60 colour matrices from a ColorChecker (2026-10-25) - -The Canon EOS D60 has no published spectral sensitivities, so it is not in `spectral_table.rs`. -These are its DNG-style matrices (`ColorMatrix1/2`, `ForwardMatrix1/2`), fitted to our own -photographs of a ColorChecker Classic under two lights. They are in -`crates/raw/src/chart_table.rs`. Measurements come from the maintainer's own captures (names, -sizes and SHA-256 in `logs/chart-captures.tsv`; no image content recorded) and the chart's -published reference colours (`sources.md`). Scripts and measurements are local-only, in -`plan/crw/chart/`. - -## Captures - -| Set | Frames used | Light | Notes | -|---|---:|---|---| -| `SunlightColorCard/` | 20 of 22 | Direct afternoon sun, ISO 100 | Calibrite ColorChecker Classic (50th-anniversary edition, post-2014 formulation) leaning on a wall, filling most of the frame. Auto and Daylight white balance, a shutter bracket. Two frames dropped: a white-patch sample ≥ 3900 (clip ≈ 4000 at ISO 100, `08-color.md` §2). | -| `TungstonColorChart/` 2696–2706 | 11 | One household tungsten bulb, ISO 100 | Chart flat on a wall, a shutter bracket. Brightest white patch 2657 of ~4000. | -| `TungstonColorChart/` 2688–2691 | not used | Same bulb, 8 minutes earlier | Steady within the set, but its neutral reads ~3,040 K against ~2,900 K for the later set: the light changed (window light or bulb warm-up). The later, warmer and better-exposed set is used. | - -All 37 frames decode sample-exact against the black-box reference (`crw_oracle.rs`). - -An earlier set on a ColorChecker-layout clone (Pixel Perfect "24 Standard Color Calibration -Chart", 2026-10-15) was not used. The maker publishes no reference values, and its grey patches -are not neutral: against an 18 % grey board they read R/G 0.88–0.94 in shade and B/G up to -1.29 under warm LED light, the darker patches drifting most. That is the sign of greys printed -from coloured inks. - -## Measurement - -- Patches are located on the embedded JPEG (four corner patch centres, bilinear grid), tracked - per frame by phase correlation, and checked visually on drawn boxes. The tungsten-set chart - was upside down in the first four frames; patch order is mapped accordingly. -- Each patch is measured in a central box of about 40 % of its width in the linear raw (black - from the far masked border per CFA site; R, mean of both G, B per 2×2 site). Uniformity - across the box (3×3 sub-blocks): ≤ 4 % in sun, ≤ 7 % under the bulb (light falloff across - the closer chart). -- Frames are normalised by the four middle neutrals (patches 20–23), averaged, and white - balanced on those neutrals. White balance (R, B multipliers, G = 1): sun 2.362, 1.298; bulb - 1.353, 2.213. The neutrals agree on it to ±2 %. - -## Fit - -- The reference is X-Rite's published L\*a\*b\* (D50) for charts made after November 2014. -- The model is a 3×3 matrix from white-balanced camera RGB to XYZ (D50), with neutral (1, 1, 1) - held exactly at the engine's D50 white (`lightcraft_color::D50`): the third column is D50 - minus the first two. That is a DNG `ForwardMatrix`. -- **Chromaticity only.** Each prediction is scaled to the patch's reference luminance before - the CIEDE2000 residual. In the sun the chart warped as it heated: the grey row's brightness - slope swung from +4 % to −7 % across the 70 s of shooting, in time order, and the per-patch - brightness needed was not smooth across the chart. A fit including lightness would have - absorbed that into the matrix. Chromaticity still determines the whole 3×3 up to scale, - which the white constraint fixes. The cost is that luminance of saturated blues and purples - rests on the model, not the measurement. -- `ColorMatrix` per light = diag(1 / multipliers) · ForwardMatrix⁻¹ · Bradford(light → D50), - scaled so the light's white maps to camera values with maximum 1. By construction it maps - the filed illuminant's white to the measured camera neutral. - -| Light | CIEDE2000 on the 24 patches | Leaving each patch out | Worst | -|---|---|---|---| -| Sun (filed as D50) | mean 1.70, median 1.46 | mean 1.97 | white 4.5 | -| Bulb (filed as A) | mean 2.42, median 1.79 | mean 2.74 | bluish green 6.7 | - -- One matrix does not serve both lights: the sun's matrix on the bulb's chart gives mean 3.6 - (cyan 9.0), the bulb's on the sun's 3.4 (red 7.7). Hence the dual-illuminant entry. -- The neutral row reads a\* within ±0.4 for the middle greys; white and the darkest patches - are off by up to 2 (white patch sheen, flare). - -## Which illuminants - -The two lights' colour temperatures come from the camera's own white-balance presets -(ColorBalance slots, `08-color.md` §3) at the temperatures Canon's manuals give them (Daylight -5200 K, Tungsten 3200 K, Fluorescent 4000 K), interpolated in mired on ln(R/B): sun ~4,850 K, -bulb ~2,900 K. They are filed under the nearest DNG illuminants the engine knows, **D50 -(5003 K) and Standard Illuminant A (2856 K)**, so `Camera` now carries its own illuminant -pair (spectral rows: A and D65). - -## Checks in the app - -- `spectral::tests::the_chart_measured_eos_d60_reads_its_presets_in_kelvin`: the camera's - presets, which the fit never saw, read through the matrices as Tungsten 3,264 K, Daylight - 5,603 K, Cloudy 6,613 K, Flash 7,085 K, near the locus. Canon's nominal values are 3200, 5200, - 6000 and 6000 K. Order and spacing hold; the daylight presets read 8–15 % high, as Sony's - dial does against its spectral matrices (#730). -- Real files through the headless app (As Shot): WBModes Fluorescent 4,103 K, tint +26 (green - compensation); LED-lit room, Auto 3,761 K; open shade, Auto 6,596 K; sunlit chart, Auto - 4,762 K. -- `files::tests::crw_probes_name_their_camera_from_the_ciff_records`: CRW imports now take make, - model and capture time from the decoder's CIFF records (no Exif in a CRW; the catalog had an - empty camera, so a D60 stayed on the relative white-balance scale). - -## Limits - -- One body, one bulb, one afternoon. The chart is 24 patches; a 3×3 matrix cannot follow the - sensor's departures from the CIE observer (the bulb fit's worst patches are 5–7 ΔE). -- The lights are filed under standard illuminants they only approximate (~50 K for the bulb, - ~150 K for the sun). -- The D60's look is still the per-file JPEG fit and the pooled profile; these matrices give the - white-balance scale and the fallback colour only. diff --git a/crates/crw/clean-room/README.md b/crates/crw/clean-room/README.md index e50f822e..1174e713 100644 --- a/crates/crw/clean-room/README.md +++ b/crates/crw/clean-room/README.md @@ -5,43 +5,44 @@ produced in LightCraft (`clean-room/crw/`, branch `clean-room-crw-format-decode` `6ca439b5155820d2befeacd81ec0c6a80b2bd3bc`; the record is identical at `70c755e4`, where the decoder was ported) between 2026-10-12 and 2026-10-25 and moved here **verbatim** when the decoder became this shared crate. The documents are the evidence: they are not rewritten after the fact, so paths they mention are LightCraft's paths of that time (the -mapping is below), and `00-starting-assumptions.md` is committed once and never edited. +mapping is below). + +The derivation trail — the starting assumptions, the plan, the container notes, the +entropy-derivation log, the chart calibration, the probe scripts and their observational logs — is +not part of this record. What this record holds is what the decoder is built from and what a +reviewer needs to check it: the frozen format, its verification, the payload-layout and colour +notes, the private-corpus record, the source log and the frozen code tables. Those documents are +renumbered `01`–`05` in reading order. ## Documents | File | What it is | |---|---| -| [`00-starting-assumptions.md`](00-starting-assumptions.md) | What the author believed before consulting any source; every item marked unverified. Immutable. | -| [`01-plan.md`](01-plan.md) | The derivation plan, the corpus (public CC0 files with hashes; private files by name, size and hash), the maintainer's decisions (§7) and approval. | | [`sources.md`](sources.md) | Every URL opened and search run, with the time, what was asked and what was taken. The rules in force are at its top. | -| [`02-container.md`](02-container.md) | CIFF heap parsing from Canon's specification and ExifTool's prose pages, each fact cited. | -| [`03-payload-layout.md`](03-payload-layout.md) | The RawData payload partition established by statistics alone (no decoding): low-bits plane, 514-byte header, ff00-stuffed stream, tail. | -| [`04-derivation-log.md`](04-derivation-log.md) | The entropy-coding derivation, experiment by experiment, against the black-box reference. | -| [`06-crw-format.md`](06-crw-format.md) | **The frozen format** (revision 2) and its reopening rule. The decoder implements this document. | -| [`07-verification.md`](07-verification.md) | Full-frame exact comparison against the reference, zero tuning: 28 of 29 files, then 1,516 of 1,516 D60 files. | -| [`08-color.md`](08-color.md) | CFA phase, active area, black and white levels, white-balance records, orientation and time: what the decoder reports beyond the samples. §3's colour matrices concern LightCraft (below). | -| [`09-private-corpus.md`](09-private-corpus.md) | The maintainer's EOS D60 files and controlled captures (grey card, lens cap, white-balance presets) and what each set established. | -| [`10-chart-calibration.md`](10-chart-calibration.md) | EOS D60 colour matrices from ColorChecker captures. This is LightCraft's colour work (`crates/raw/src/chart_table.rs` there), kept here because it is part of the same record and uses the same captures. | +| [`01-payload-layout.md`](01-payload-layout.md) | The RawData payload partition established by statistics alone (no decoding): low-bits plane, 514-byte header, ff00-stuffed stream, tail. | +| [`02-crw-format.md`](02-crw-format.md) | **The frozen format** (revision 2) and its reopening rule. The decoder implements this document. | +| [`03-verification.md`](03-verification.md) | Full-frame exact comparison against the reference, zero tuning: 28 of 29 files, then 1,516 of 1,516 D60 files. | +| [`04-color.md`](04-color.md) | CFA phase, active area, black and white levels, white-balance records, orientation and time: what the decoder reports beyond the samples. §3's colour matrices concern LightCraft (below). | +| [`05-private-corpus.md`](05-private-corpus.md) | The maintainer's EOS D60 files and controlled captures (grey card, lens cap, white-balance presets) and what each set established. | -There is no `05-geometry-colour.md`: the plan named it, and the material went into `08-color.md`. +## Tables and logs -## Scripts and logs - -- `ciff_dump.py`, `p1_payload.py`, `p1b_markers.py`, `p1c_lowbits_bounds.py`, `p2_oracle_lsb.py`: - the planning and payload-statistics scripts (standard library only; run them with `python -I`). - Their public outputs are in `logs/` (`structure-*.txt`, `p1*.txt`, `verification-sweep.txt`). - `logs/table_sel{0,1,2}.json`: **the frozen code tables**, the source of truth that `tools/gen_tables.py` renders into `../src/tables.rs`. `table_sel0_d60class.json` is the - revision-1 selector-0 table, kept for the record. -- `logs/private-corpus.tsv`, `logs/chart-captures.tsv`: names, sizes and SHA-256 of the - maintainer's own files. No image content, metadata or thumbnails are recorded anywhere here. + revision-1 selector-0 table, kept for the record. These are the only logs that ship with the + crate: without them the frozen tables cannot be regenerated or audited. +- `logs/private-corpus.tsv`: names, sizes and SHA-256 of the maintainer's own files. No image + content, metadata or thumbnails are recorded anywhere here. - `logs/private/` (gitignored): per-file dumps of the private files, which can hold personal metadata. Local only. +- The probe scripts and their observational logs (`structure-*.txt`, `p1*.txt`, + `verification-sweep.txt`, `chart-captures.tsv`) are not part of this record. ## What stayed local - The Python reference model and probe harness (`plan/crw/model/` in LightCraft, gitignored), - by the maintainer's decision #4. Its findings are all in `04-derivation-log.md`. + by the maintainer's decision #4. Its findings are the entropy-coding derivation, which is not + part of this record. - The reference decoder: LibRaw 0.22.2, official pre-built Windows binaries installed by the maintainer; only `unprocessed_raw.exe` was ever run, as a black box. The source tree shipped alongside it was never opened, read or searched. @@ -65,11 +66,11 @@ There is no `05-geometry-colour.md`: the plan named it, and the material went in ## Rules that were in force -From `sources.md` and `01-plan.md`: no raw-decoder source may be opened, fetched, searched inside +From `sources.md`: no raw-decoder source may be opened, fetched, searched inside or requested (dcraw, LibRaw, rawspeed, rawloader/rawler, dnglab, darktable, RawTherapee, libopenraw, ExifTool's Perl code, including licence headers, comments and mirrors). Allowed: published format documentation in prose, published standards, the sample files, and a reference decoder's binary run as a black box with every probe logged. No tuning after the freeze: a mismatch reopens the derivation explicitly, with its cause and a new freeze commit recorded -(`06-crw-format.md`, revision 2, is the one reopening). The general standard this followed is +(`02-crw-format.md`, revision 2, is the one reopening). The general standard this followed is [`docs/clean-room.md`](../../../docs/clean-room.md). diff --git a/crates/crw/clean-room/ciff_dump.py b/crates/crw/clean-room/ciff_dump.py deleted file mode 100644 index 9a69a81f..00000000 --- a/crates/crw/clean-room/ciff_dump.py +++ /dev/null @@ -1,197 +0,0 @@ -"""CIFF/CRW container structure dump (planning phase — no raw-data decoding). - -Prints the heap tree, record entries (tag, storage class, data type, size, offset), -a few interpreted metadata records (make/model, ImageSpec, SensorInfo-ish u16 arrays, -DecoderTable u32s), and byte statistics of the RawData record. It never decodes the -compressed raw bitstream. - -Container layout implemented from published documentation only: -- Canon, "CIFF Specification on Image Data File" v1.0 r4, 1997-12-24 (archived PDF): - heap file header {byteOrder[2], headerLength u32, type[4], subType[4], version u32, - reserved1/2 u32}; heap = heap space + offsetTbl; u32 offsetTblOffset stored in the - final position of the heap; offsetTbl = numRecords u16 + entries {typeCode u16, - length u32, offset u32}; typeCode bits 15-14 storage (00 in heap space, 01 in record - entry), bits 13-11 dataType (000 byte, 001 ASCII, 010 word, 011 dword, 100 struct, - 101/110 heap record), bits 10-0 IDCode. -- ExifTool "canon_raw.html" (prose page): offsets are relative to the start of the - containing block's ValueData; a subdirectory is a complete block inside ValueData - whose length is the entry's Size; the root block runs from headerLength to EOF; - tag IDs incl. 0x2005 RawData, 0x1835 DecoderTable, 0x1031 SensorInfo, 0x080a - make/model, 0x1810 ImageInfo. - -Usage: python -I ciff_dump.py [--no-strings] file.crw [more.crw ...] ---no-strings suppresses record contents (for logs of private files: structure, -sizes, offsets and statistics only; ASCII make/model still shown, pixel data never). -""" - -import math -import struct -import sys -from collections import Counter - -# Tag names from ExifTool's published tag table (documentation page, not source). -TAG_NAMES = { - 0x0000: "NullRecord", 0x0001: "FreeBytes", - 0x0805: "CanonFileDescription", 0x080A: "CanonRawMakeModel", - 0x080B: "CanonFirmwareVersion", 0x0810: "OwnerName", - 0x0815: "CanonImageType", 0x0816: "OriginalFileName", - 0x0817: "ThumbnailFileName", - 0x100A: "TargetImageType", 0x1010: "ShutterReleaseMethod", - 0x1011: "ShutterReleaseTiming", 0x1016: "ReleaseSetting", - 0x101C: "BaseISO", 0x1028: "CanonFlashInfo", - 0x1029: "CanonFocalLength", 0x102A: "CanonShotInfo", - 0x102C: "CanonColorInfo2", 0x102D: "CanonCameraSettings", - 0x1030: "WhiteSample", 0x1031: "SensorInfo", - 0x1033: "CustomFunctions", 0x1038: "CanonAFInfo", - 0x10A9: "ColorBalance", 0x10AE: "ColorTemperature", - 0x10B4: "ColorSpace", 0x10B5: "RawJpgInfo", - 0x1803: "ImageFormat", 0x1804: "RecordID", - 0x1806: "SelfTimerTime", 0x1807: "TargetDistanceSetting", - 0x180B: "SerialNumber", 0x180E: "TimeStamp", - 0x1810: "ImageInfo", 0x1813: "FlashInfo", - 0x1814: "MeasuredEV", 0x1817: "FileNumber", - 0x1818: "ExposureInfo", 0x1834: "CanonModelID", - 0x1835: "DecoderTable", 0x183B: "SerialNumberFormat", - 0x2005: "RawData", 0x2007: "JpgFromRaw", 0x2008: "ThumbnailImage", - 0x2804: "ImageDescription", 0x2807: "CameraObject", - 0x3002: "ShootingRecord", 0x3003: "MeasuredInfo", - 0x3004: "CameraSpecification", 0x300A: "ImageProps", - 0x300B: "ExifInformation", -} - -DATATYPES = {0: "byte", 1: "ascii", 2: "word", 3: "dword", - 4: "struct", 5: "heap1", 6: "heap2", 7: "rsvd7"} - -MAX_DEPTH = 8 - - -def entropy(data): - if not data: - return 0.0 - counts = Counter(data) - n = len(data) - return -sum(c / n * math.log2(c / n) for c in counts.values()) - - -def byte_stats(data, label, out): - n = len(data) - out.append(f" {label}: {n} bytes, entropy {entropy(data):.4f} bits/byte") - if n >= 4096: - for part, name in ((data[:2048], "first 2 KiB"), (data[n // 2:n // 2 + 2048], "middle 2 KiB"), - (data[-2048:], "last 2 KiB")): - c = Counter(part) - top = ", ".join(f"{b:02x}:{cnt}" for b, cnt in c.most_common(4)) - out.append(f" {name}: entropy {entropy(part):.3f}, top bytes {top}, " - f"zeros {c.get(0, 0)}, ff {c.get(255, 0)}") - - -def dump_heap(f, blk_start, blk_len, depth, endian, out, show, path="root"): - """blk_start/blk_len: the block (ValueData + offsetTbl + trailing u32).""" - pre = " " * depth - if depth > MAX_DEPTH: - out.append(f"{pre}!! depth limit at {path}") - return - if blk_len < 6: - out.append(f"{pre}!! block too small ({blk_len}) at {path}") - return - f.seek(blk_start + blk_len - 4) - (tbl_off,) = struct.unpack(endian + "I", f.read(4)) - if tbl_off + 2 > blk_len - 4: - out.append(f"{pre}!! offsetTblOffset {tbl_off:#x} out of range (block {blk_len:#x}) at {path}") - return - f.seek(blk_start + tbl_off) - (nrec,) = struct.unpack(endian + "H", f.read(2)) - out.append(f"{pre}heap {path}: block @{blk_start:#x} len {blk_len:#x}, offsetTbl @+{tbl_off:#x}, {nrec} records") - if 2 + nrec * 10 > blk_len - tbl_off: - out.append(f"{pre}!! record table overruns block at {path}") - return - entries = [] - for i in range(nrec): - tc, length, offset = struct.unpack(endian + "HII", f.read(10)) - entries.append((tc, length, offset)) - for i, (tc, length, offset) in enumerate(entries): - stg = (tc >> 14) & 3 - dt = (tc >> 11) & 7 - tag_id = tc & 0x3FFF # dataType | IDCode, ExifTool's "TagID" - name = TAG_NAMES.get(tag_id, "?") - loc = {0: "heap", 1: "inrec", 2: "stg2?", 3: "stg3?"}[stg] - out.append(f"{pre} [{i}] tag {tc:#06x} id {tag_id:#06x} {name:<22} {DATATYPES[dt]:<6} " - f"{loc:<5} len {length:#x} off {offset:#x}") - if stg == 1: - continue # value lives in the length/offset fields themselves - if offset + length > blk_len: - out.append(f"{pre} !! record overruns block") - continue - abs_off = blk_start + offset - if dt in (5, 6): - dump_heap(f, abs_off, length, depth + 1, endian, out, show, f"{path}/{name or hex(tag_id)}") - elif tag_id == 0x2005: - f.seek(abs_off) - data = f.read(length) - byte_stats(data, "RawData", out) - elif tag_id in (0x2007, 0x2008): - f.seek(abs_off) - head = f.read(4) - out.append(f" starts with {'JPEG SOI' if head[:2] == bytes([0xFF, 0xD8]) else head.hex()}") - elif show and dt == 1 and length < 256: - f.seek(abs_off) - s = f.read(length).rstrip(b"\x00") - out.append(f"{pre} = {s!r}") - elif show and dt == 2 and length <= 64: - f.seek(abs_off) - vals = struct.unpack(endian + "H" * (length // 2), f.read(length & ~1)) - out.append(f"{pre} = u16 {vals}") - elif show and dt == 3 and length <= 64: - f.seek(abs_off) - vals = struct.unpack(endian + "I" * (length // 4), f.read(length & ~3)) - out.append(f"{pre} = u32 {vals}") - elif show and dt == 4 and tag_id == 0x1835 and length <= 64: - f.seek(abs_off) - vals = struct.unpack(endian + "I" * (length // 4), f.read(length & ~3)) - out.append(f"{pre} = DecoderTable u32 {vals}") - - -def dump(fname, show): - out = [f"== {fname}"] - with open(fname, "rb") as f: - f.seek(0, 2) - fsize = f.tell() - f.seek(0) - hdr = f.read(26) - if len(hdr) < 26: - out.append("!! file shorter than header") - return out - bo = hdr[0:2] - endian = {b"II": "<", b"MM": ">"}.get(bo) - if endian is None: - out.append(f"!! unknown byte order {bo!r} — not CIFF? first 16: {hdr[:16].hex()}") - return out - (hdr_len,) = struct.unpack(endian + "I", hdr[2:6]) - sig_type, sig_sub = hdr[6:10], hdr[10:14] - (version,) = struct.unpack(endian + "I", hdr[14:18]) - out.append(f" size {fsize}, byteOrder {bo.decode()}, headerLength {hdr_len}, " - f"type {sig_type!r} subType {sig_sub!r}, version {version:#010x}") - if sig_type != b"HEAP": - out.append("!! not a HEAP file — stopping (CHDK dump or other format?)") - return out - dump_heap(f, hdr_len, fsize - hdr_len, 0, endian, out, show) - return out - - -def main(): - args = sys.argv[1:] - show = True - if args and args[0] == "--no-strings": - show = False - args = args[1:] - for fname in args: - try: - lines = dump(fname, show) - except Exception as e: # planning tool: report, don't die - lines = [f"== {fname}", f"!! exception: {e!r}"] - print("\n".join(lines)) - print() - - -if __name__ == "__main__": - main() diff --git a/crates/crw/clean-room/logs/chart-captures.tsv b/crates/crw/clean-room/logs/chart-captures.tsv deleted file mode 100644 index c8030990..00000000 --- a/crates/crw/clean-room/logs/chart-captures.tsv +++ /dev/null @@ -1,39 +0,0 @@ -# EOS D60 ColorChecker Classic captures (maintainer, local only; never committed). Names, sizes, SHA-256; no image content. Generated 2026-10-25. -file bytes sha256 -SunlightColorCard/CRW_2666.CRW 7791628 371d8e18c12801c72754dd7b277163a866afc422c8edf6ad1ea822009e4170f8 -SunlightColorCard/CRW_2667.CRW 6638578 188d625c31ab44475a6cf54a5dae00fd861e26aafc90f4b5d18a1905c0a3cef5 -SunlightColorCard/CRW_2668.CRW 6832770 1189b8dc79642c57a8c9496b5629e36a1741c545d2b0b864d6e6e20d901510c1 -SunlightColorCard/CRW_2669.CRW 7079102 baf1686b27a9e0a04bf87c0b6cf55168f398d85247f8115a0af69b5e9ea1884c -SunlightColorCard/CRW_2670.CRW 7225298 067ba3f4d3026b8009cfe4babb5e4c3b28f988d95c48024a75c6715de32afccf -SunlightColorCard/CRW_2671.CRW 7504700 cf81cbb96389624318e890b5506221d49f5fba8e2d45447d87e0f3ff3e9bdc72 -SunlightColorCard/CRW_2672.CRW 7830852 7a59446bfb1f00d6eabd0115b8e5451c34f5d3d48f483cb27608ce9508e43ee5 -SunlightColorCard/CRW_2673.CRW 8053180 feeb8f818325d6ddb4e88b5eabe4a74f4cb5e1da9f70135ae3f85938c86695dc -SunlightColorCard/CRW_2674.CRW 8312060 45e7402c888582adadb607599eae80644d657d61c9607b426c411ee306e42daf -SunlightColorCard/CRW_2675.CRW 8565950 be28d2fa049aa4158ae9e3ec49dd28eb56fcd0ffa7af060235b8bda0d89dfd0a -SunlightColorCard/CRW_2676.CRW 7795508 76f7418a82e45e1034b3adacacf4bc756cb65a0d683e5bb3631992784e654f93 -SunlightColorCard/CRW_2677.CRW 7747450 a5e7bc618e6b80ea70f9bad6a1bf41ee63eadbc51df28b82d0864a73434f7964 -SunlightColorCard/CRW_2678.CRW 6815054 28824f0f64b087488d8dc5a66a8bdf717e83a4d4e1660676d07991bf32dffb96 -SunlightColorCard/CRW_2679.CRW 7050606 b5c915fa89542265deb4fa16da3ca7ffda01b1d961903836b0059d024316673a -SunlightColorCard/CRW_2680.CRW 7231410 a5780dfe99a2df2a3ed9de1888e91baa614b2e009a2b6e0a84d442b6bf1af9fd -SunlightColorCard/CRW_2681.CRW 7438088 eb4475919c2770f816d47787656ab7ecb5396f30d2c03628c3ab167ebbc47c65 -SunlightColorCard/CRW_2682.CRW 7584262 a6fe417b90ce41b03552963dee6b21e4f02b59a5c51148edc04af7224e8c59e9 -SunlightColorCard/CRW_2683.CRW 7811474 5402ad40e2689d78ade083d8855ecced5caea9368b9d77ecc4a61a16dc55aa36 -SunlightColorCard/CRW_2684.CRW 8043372 bbef5da457c0b6d6c45c74a99f3e318e6c80a7d3a0ad5c7cb7e13d24205f64fa -SunlightColorCard/CRW_2685.CRW 8242344 8d68d592a5c8933f0f35630114fe7017be808b007867beaf1a8d6d2bd1441f70 -SunlightColorCard/CRW_2686.CRW 8462380 79f7db32b18a834d5408500185c65d316387d1440aa61e26a11be85883fcc382 -SunlightColorCard/CRW_2687.CRW 7690732 0333d8d18390e77128f186239e6de71c0432ab6ee9669a71c210255e4d915d19 -TungstonColorChart/CRW_2688.CRW 6269390 d9c4334edfe3636a8a7e69f29c1d1e89f820b7b704697684d7996faee41f5d43 -TungstonColorChart/CRW_2689.CRW 6001348 1e8b47645d76a71fd9fb4111c4dce805d6dcd41ac82e6b4ce52debbf71565455 -TungstonColorChart/CRW_2690.CRW 6114056 f63fe49a9ed13af33ddb8d8ed8d75de4e304a3bc2c1f1d4275ba7edbc4f459f3 -TungstonColorChart/CRW_2691.CRW 6299730 ad8a2844c00965fb6a12e8fc85a6e8bfbdf1924e6d1a68b55472701cb8ffa115 -TungstonColorChart/CRW_2696.CRW 6169684 105ee515c94da78dedc1f8d777982c4a1e62781b160428524fbe922f62b3ef5c -TungstonColorChart/CRW_2697.CRW 5565982 327eebf702185127feb8a5f4bfac5d7ebad5532f7417b302997b19b97d7ff4a8 -TungstonColorChart/CRW_2698.CRW 5709092 f0fca20f8dcd51046e53bebf9fb9f624677ee674ad739622cdb2578977669705 -TungstonColorChart/CRW_2699.CRW 5873492 57724a1ff507cf4e78c80a34cd4e0b77eac397322228d32e5215fc3869853dd1 -TungstonColorChart/CRW_2700.CRW 5988146 6c97ae2c72114034e2fbe6aed0ef97f475696e43b56ed62273eccbfbba4cbaa0 -TungstonColorChart/CRW_2701.CRW 6202756 c4032e46b753a718d8748e269a9da5d124891d07912542368ea867529143aa62 -TungstonColorChart/CRW_2702.CRW 6409582 89b65c1c07e0685841046db85923b0bf80e747fed141ddf0335ba12dc75f3f3c -TungstonColorChart/CRW_2703.CRW 6407270 50179d0a330c967a74d76bc8342eea1052689c5c544aa1a913769b241ac6f736 -TungstonColorChart/CRW_2704.CRW 6483154 6bac4d8229a8d0fff1445d212ca63e53911dc7bda988743215009257b87787fa -TungstonColorChart/CRW_2705.CRW 7170488 629f94131e7f598964d2635e3039a20c441563201c4f3cbf177d83ad6b061f78 -TungstonColorChart/CRW_2706.CRW 6383026 67657e7a21f6fd8580012fccd2210afccf285e747bc4d39b8ea775811528ce83 diff --git a/crates/crw/clean-room/logs/p1-payload-public.txt b/crates/crw/clean-room/logs/p1-payload-public.txt deleted file mode 100644 index a528740c..00000000 --- a/crates/crw/clean-room/logs/p1-payload-public.txt +++ /dev/null @@ -1,195 +0,0 @@ -== ../../corpus/crw/crw-canon-eos-10d-a.crw - raw 3152x2068 S=7134454 tbl=0 f1=0 f2=514 L=5504868 S-L=1629586 wh/4=1629584 S-L-wh/4=2 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: 95ad8fb002120000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 85.37 ent 5.190 - top..top+32: mean 127.88 ent 7.993 - middle 32: mean 126.86 ent 7.992 - last 32: mean 127.02 ent 7.993 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 132.23 ent 7.944 - top..top+32: mean 131.83 ent 7.953 - middle 32: mean 128.56 ent 7.959 - last 32: mean 126.95 ent 7.956 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9684 (first MiB), 7.9644 (last MiB) - from-0 [0 : L): entropy 7.9960 (first MiB), 7.9530 (last MiB) - stuffing check: ff count 13614 (0.2473%), ff00 digrams 13614, expected-if-independent 64.4 - -== ../../corpus/crw/crw-canon-eos-10d-b.crw - raw 3152x2068 S=7244858 tbl=2 f1=0 f2=514 L=5615272 S-L=1629586 wh/4=1629584 S-L-wh/4=2 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: 0c849b8b12b70000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 64.32 ent 5.152 - top..top+32: mean 121.98 ent 7.958 - middle 32: mean 126.50 ent 7.992 - last 32: mean 127.17 ent 7.993 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 117.92 ent 7.937 - top..top+32: mean 117.33 ent 7.941 - middle 32: mean 114.53 ent 7.905 - last 32: mean 117.63 ent 7.904 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9877 (first MiB), 7.9150 (last MiB) - from-0 [0 : L): entropy 7.9898 (first MiB), 7.9220 (last MiB) - stuffing check: ff count 12542 (0.2234%), ff00 digrams 12542, expected-if-independent 97.9 - -== ../../corpus/crw/crw-canon-eos-10d-c.crw - raw 3152x2068 S=7455524 tbl=2 f1=0 f2=514 L=5825938 S-L=1629586 wh/4=1629584 S-L-wh/4=2 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: 9c12ce7bde990000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 78.12 ent 5.144 - top..top+32: mean 126.89 ent 7.992 - middle 32: mean 127.89 ent 7.993 - last 32: mean 127.04 ent 7.991 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 119.60 ent 7.946 - top..top+32: mean 117.81 ent 7.946 - middle 32: mean 104.15 ent 7.770 - last 32: mean 114.88 ent 7.856 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.8675 (first MiB), 7.8315 (last MiB) - from-0 [0 : L): entropy 7.9961 (first MiB), 7.9712 (last MiB) - stuffing check: ff count 6993 (0.1200%), ff00 digrams 6993, expected-if-independent 47.5 - -== ../../corpus/crw/crw-canon-eos-300d-a.crw - raw 3152x2068 S=6929638 tbl=0 f1=0 f2=514 L=5300052 S-L=1629586 wh/4=1629584 S-L-wh/4=2 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: e438ce510ad90000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 61.27 ent 5.230 - top..top+32: mean 127.52 ent 7.994 - middle 32: mean 127.63 ent 7.993 - last 32: mean 127.31 ent 7.992 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 132.59 ent 7.950 - top..top+32: mean 130.96 ent 7.963 - middle 32: mean 131.10 ent 7.958 - last 32: mean 134.35 ent 7.947 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9405 (first MiB), 7.9600 (last MiB) - from-0 [0 : L): entropy 7.9961 (first MiB), 7.9634 (last MiB) - stuffing check: ff count 17631 (0.3327%), ff00 digrams 17631, expected-if-independent 99.9 - -== ../../corpus/crw/crw-canon-eos-300d-b.crw - raw 3152x2068 S=7206250 tbl=2 f1=0 f2=514 L=5576664 S-L=1629586 wh/4=1629584 S-L-wh/4=2 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: f8ff00eefe7f0000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 61.49 ent 5.196 - top..top+32: mean 127.65 ent 7.989 - middle 32: mean 112.85 ent 7.534 - last 32: mean 119.27 ent 7.834 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 124.00 ent 7.968 - top..top+32: mean 127.27 ent 7.978 - middle 32: mean 127.36 ent 7.985 - last 32: mean 130.81 ent 7.973 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9774 (first MiB), 7.9887 (last MiB) - from-0 [0 : L): entropy 7.8118 (first MiB), 7.9812 (last MiB) - stuffing check: ff count 28275 (0.5070%), ff00 digrams 28275, expected-if-independent 265.2 - -== ../../corpus/crw/crw-canon-eos-d30.crw - raw 2224x1456 S=2824648 tbl=1 f1=0 f2=514 L=2015110 S-L=809538 wh/4=809536 S-L-wh/4=2 - leading zero run: 0 bytes (= 0.00 rows of 2-bit LSBs) - last 8 bytes: fadfd6feb7f50000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 1456 rows of 556 B, leading all-zero rows: 0 - rows 0..top: mean 139.68 ent 7.911 - top..top+32: mean 127.19 ent 7.990 - middle 32: mean 127.59 ent 7.988 - last 32: mean 131.21 ent 7.924 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 1456 rows of 556 B, leading all-zero rows: 0 - rows 0..top: mean 123.77 ent 7.922 - top..top+32: mean 123.76 ent 7.936 - middle 32: mean 116.78 ent 7.895 - last 32: mean 116.20 ent 7.890 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9343 (first MiB), 7.9341 (last MiB) - from-0 [0 : L): entropy 7.9949 (first MiB), 7.9410 (last MiB) - stuffing check: ff count 2576 (0.1278%), ff00 digrams 2576, expected-if-independent 14.1 - -== ../../corpus/crw/crw-canon-eos-d60.crw - raw 3152x2068 S=6694146 tbl=0 f1=0 f2=514 L=5064559 S-L=1629587 wh/4=1629584 S-L-wh/4=3 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: 9385933317ff0000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 55.84 ent 5.113 - top..top+32: mean 127.09 ent 7.992 - middle 32: mean 127.69 ent 7.993 - last 32: mean 127.98 ent 7.992 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 130.50 ent 7.942 - top..top+32: mean 131.88 ent 7.949 - middle 32: mean 131.47 ent 7.951 - last 32: mean 126.89 ent 7.967 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9003 (first MiB), 7.9694 (last MiB) - from-0 [0 : L): entropy 7.9960 (first MiB), 7.9672 (last MiB) - stuffing check: ff count 14189 (0.2802%), ff00 digrams 14189, expected-if-independent 76.1 - -== ../../corpus/crw/crw-canon-eos-digital-rebel.crw - raw 3152x2068 S=6406164 tbl=0 f1=0 f2=514 L=4776577 S-L=1629587 wh/4=1629584 S-L-wh/4=3 - leading zero run: 5516 bytes (= 7.00 rows of 2-bit LSBs) - last 8 bytes: 191093af87ff0000 - prefix hypothesis region [0 : wh/4) as 2-bit rows: - prefix: 2068 rows of 788 B, leading all-zero rows: 7 - rows 0..top: mean 58.54 ent 3.603 - top..top+32: mean 144.80 ent 5.469 - middle 32: mean 163.93 ent 4.986 - last 32: mean 154.41 ent 5.106 - suffix hypothesis region [L : L+wh/4) as 2-bit rows: - suffix: 2068 rows of 788 B, leading all-zero rows: 0 - rows 0..top: mean 128.96 ent 7.960 - top..top+32: mean 129.35 ent 7.966 - middle 32: mean 127.02 ent 7.922 - last 32: mean 122.47 ent 7.926 - compressed-candidate stats: - after-prefix [wh/4 : wh/4+L): entropy 7.9244 (first MiB), 7.9355 (last MiB) - from-0 [0 : L): entropy 5.9926 (first MiB), 7.9542 (last MiB) - stuffing check: ff count 6718 (0.1406%), ff00 digrams 6718, expected-if-independent 36.3 - -== ../../corpus/crw/crw-canon-powershot-g1.crw - raw 2144x1560 S=2257354 tbl=1 f1=0 f2=514 L=2257351 S-L=3 wh/4=836160 S-L-wh/4=-836157 - leading zero run: 514 bytes (= 0.96 rows of 2-bit LSBs) - last 8 bytes: 34e0702c2fffd900 - no wh/4 region (PowerShot class): compressed assumed [0 : L) - entropy first MiB 7.9226, last MiB 7.9396 - stuffing check: ff count 2676 (0.1185%), ff00 digrams 2676, expected-if-independent 11.4 - tail region [L : S): 3 bytes, hex ffd900 - -== ../../corpus/crw/crw-canon-powershot-g5.crw - raw 2672x1968 S=4861806 tbl=1 f1=0 f2=514 L=4861804 S-L=2 wh/4=1314624 S-L-wh/4=-1314622 - leading zero run: 514 bytes (= 0.77 rows of 2-bit LSBs) - last 8 bytes: 71928822fd7fffd9 - no wh/4 region (PowerShot class): compressed assumed [0 : L) - entropy first MiB 7.9958, last MiB 7.9087 - stuffing check: ff count 9974 (0.2052%), ff00 digrams 4958, expected-if-independent 54.4 - tail region [L : S): 2 bytes, hex ffd9 - -== ../../corpus/crw/crw-canon-powershot-g6.crw - raw 3160x2344 S=7313156 tbl=0 f1=0 f2=514 L=7313154 S-L=2 wh/4=1851760 S-L-wh/4=-1851758 - leading zero run: 514 bytes (= 0.65 rows of 2-bit LSBs) - last 8 bytes: dfbf7efdfbf7ffd9 - no wh/4 region (PowerShot class): compressed assumed [0 : L) - entropy first MiB 7.9995, last MiB 7.9596 - stuffing check: ff count 18811 (0.2572%), ff00 digrams 10815, expected-if-independent 103.5 - tail region [L : S): 2 bytes, hex ffd9 - diff --git a/crates/crw/clean-room/logs/p1b-markers-public.txt b/crates/crw/clean-room/logs/p1b-markers-public.txt deleted file mode 100644 index d0442666..00000000 --- a/crates/crw/clean-room/logs/p1b-markers-public.txt +++ /dev/null @@ -1,109 +0,0 @@ -== ../../corpus/crw/crw-canon-eos-10d-a.crw - raw 3152x2068 tbl=0 f2=514 L=5504868 lowbits=yes - ff followed by: 00:13614 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 9df595ad8fb00212 - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : 95ad8fb002120000 - low[-8:] : fc3590b2d228d3dd (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-10d-b.crw - raw 3152x2068 tbl=2 f2=514 L=5615272 lowbits=yes - ff followed by: 00:12542 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 38650c849b8b12b7 - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : 0c849b8b12b70000 - low[-8:] : c6371ccbb9d8dd2c (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-10d-c.crw - raw 3152x2068 tbl=2 f2=514 L=5825938 lowbits=yes - ff followed by: 00:6993 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : e7389c12ce7bde99 - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : 9c12ce7bde990000 - low[-8:] : eb6d0fe72354f02c (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-300d-a.crw - raw 3152x2068 tbl=0 f2=514 L=5300052 lowbits=yes - ff followed by: 00:17631 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 22c9e438ce510ad9 - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : e438ce510ad90000 - low[-8:] : a93331aa85cdb788 (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-300d-b.crw - raw 3152x2068 tbl=2 f2=514 L=5576664 lowbits=yes - ff followed by: 00:28275 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : bbe9f8ff00eefe7f - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : f8ff00eefe7f0000 - low[-8:] : bbb7b7b7bbb777c7 (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-d30.crw - raw 2224x1456 tbl=1 f2=514 L=2015110 lowbits=yes - ff followed by: 00:2576 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 7f5bfadfd6feb7f5 - raw[504:544] : a57fba0edb9db8e63afc7b4eaa3c6dae0efbabaf3fcc8bbee8f93db67aabbee9ce27ba68a9bfaa26 - raw[-8:] : fadfd6feb7f50000 - low[-8:] : ffffffffffffffff (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-d60.crw - raw 3152x2068 tbl=0 f2=514 L=5064559 lowbits=yes - ff followed by: 00:14189 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 51f18a9385933317 - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : 9385933317ff0000 - low[-8:] : 82961c457559780e (end of low-bits region) - -== ../../corpus/crw/crw-canon-eos-digital-rebel.crw - raw 3152x2068 tbl=0 f2=514 L=4776577 lowbits=yes - ff followed by: 00:6718 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : fc86e6191093af87 - raw[504:544] : 00000000000000000000000000000000000000000000000000000000000000000000000000000000 - raw[-8:] : 191093af87ff0000 - low[-8:] : 66ee4e6c4cc6cece (end of low-bits region) - -== ../../corpus/crw/crw-canon-powershot-g1.crw - raw 2144x1560 tbl=1 f2=514 L=2257351 lowbits=no - ff followed by: 00:2676 - markers: none (pure stuffed entropy data) - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 3efec534e0702c2f - raw[504:544] : 00000000000000000000f83df43825313b62bf97ebdc24f476407e600ce3f30b9211928d5f8de276 - raw[-8:] : 34e0702c2fffd900 - -== ../../corpus/crw/crw-canon-powershot-g5.crw - raw 2672x1968 tbl=1 f2=514 L=4861804 lowbits=no - ff followed by: 00:4958, fa:40, e7:35, d8:35, 24:33, c0:32, d3:31, 34:30 - markers: ff01:20, ff02:16, ff03:24, ff04:23, ff05:16, ff06:23, ff07:23, ff08:14, ff09:16, ff0a:21, ff0b:22, ff0c:16, ff0d:22, ff0e:26, ff0f:24, ff10:27, ff11:25, ff12:17, ff13:30, ff14:21, ff15:13, ff16:15, ff17:22, ff18:19, ff19:22, ff1a:18, ff1b:24, ff1c:18, ff1d:9, ff1e:20, ff1f:20, ff20:22, ff21:18, ff22:20, ff23:19, ff24:33, ff25:12, ff26:15, ff27:24, ff28:22, ff29:27, ff2a:18, ff2b:21, ff2c:14, ff2d:23, ff2e:15, ff2f:17, ff30:16, ff31:18, ff32:22, ff33:17, ff34:30, ff35:15, ff36:20, ff37:19, ff38:21, ff39:18, ff3a:16, ff3b:19, ff3c:16, ff3d:27, ff3e:16, ff3f:19, ff40:22, ff41:20, ff42:23, ff43:16, ff44:13, ff45:15, ff46:25, ff47:12, ff48:10, ff49:19, ff4a:18, ff4b:17, ff4c:25, ff4d:17, ff4e:25, ff4f:17, ff50:27, ff51:24, ff52:20, ff53:17, ff54:17, ff55:20, ff56:21, ff57:19, ff58:18, ff59:17, ff5a:13, ff5b:14, ff5c:18, ff5d:27, ff5e:21, ff5f:22, ff60:18, ff61:15, ff62:15, ff63:15, ff64:24, ff65:25, ff66:15, ff67:13, ff68:19, ff69:16, ff6a:13, ff6b:17, ff6c:26, ff6d:24, ff6e:18, ff6f:12, ff70:22, ff71:13, ff72:17, ff73:18, ff74:16, ff75:13, ff76:24, ff77:24, ff78:19, ff79:13, ff7a:26, ff7b:23, ff7c:28, ff7d:14, ff7e:15, ff7f:19, ff80:22, ff81:20, ff82:17, ff83:23, ff84:15, ff85:22, ff86:16, ff87:24, ff88:24, ff89:25, ff8a:18, ff8b:13, ff8c:18, ff8d:18, ff8e:23, ff8f:15, ff90:20, ff91:19, ff92:17, ff93:17, ff94:20, ff95:10, ff96:14, ff97:18, ff98:15, ff99:14, ff9a:22, ff9b:20, ff9c:20, ff9d:21, ff9e:21, ff9f:18, ffa0:21, ffa1:14, ffa2:20, ffa3:18, ffa4:21, ffa5:22, ffa6:24, ffa7:10, ffa8:16, ffa9:22, ffaa:22, ffab:16, ffac:25, ffad:22, ffae:28, ffaf:10, ffb0:18, ffb1:20, ffb2:19, ffb3:20, ffb4:20, ffb5:16, ffb6:18, ffb7:18, ffb8:19, ffb9:25, ffba:24, ffbb:12, ffbc:21, ffbd:12, ffbe:28, ffbf:22, ffc0:32, ffc1:21, ffc2:20, ffc3:14, ffc4:25, ffc5:14, ffc6:24, ffc7:18, ffc8:19, ffc9:17, ffca:23, ffcb:16, ffcc:20, ffcd:18, ffce:21, ffcf:17, ffd0:18, ffd1:24, ffd2:12, ffd3:31, ffd4:23, ffd5:16, ffd6:24, ffd7:23, ffd8:35, ffd9:17, ffda:15, ffdb:30, ffdc:16, ffdd:21, ffde:17, ffdf:12, ffe0:24, ffe1:22, ffe2:16, ffe3:23, ffe4:25, ffe5:24, ffe6:13, ffe7:35, ffe8:19, ffe9:14, ffea:25, ffeb:13, ffec:24, ffed:26, ffee:20, ffef:17, fff0:24, fff1:17, fff2:20, fff3:17, fff4:19, fff5:23, fff6:19, fff7:18, fff8:21, fff9:17, fffa:40, fffb:12, fffc:18, fffd:30, fffe:25, ffff:14 - first markers at: [('0x2b9', '0x43'), ('0x36a', '0x9d'), ('0x669', '0x95'), ('0x6a3', '0x37'), ('0x909', '0x29'), ('0x93a', '0xad'), ('0x9f9', '0x7f'), ('0xa55', '0x68'), ('0xc5f', '0x47'), ('0xf97', '0x65'), ('0x1068', '0x3a'), ('0x1210', '0x51')] - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : 426771928822fd7f - raw[504:544] : 000000000000000000009e16f4366ff2c547b59c5e132991784465d6f539fd37876356e9155982fa - raw[-8:] : 71928822fd7fffd9 - -== ../../corpus/crw/crw-canon-powershot-g6.crw - raw 3160x2344 tbl=0 f2=514 L=7313154 lowbits=no - ff followed by: 00:10815, ff:68, 37:61, 77:60, 7f:59, bb:58, ee:58, 33:56 - markers: ff01:24, ff02:24, ff03:31, ff04:17, ff05:39, ff06:29, ff07:31, ff08:23, ff09:31, ff0a:32, ff0b:26, ff0c:33, ff0d:27, ff0e:34, ff0f:29, ff10:27, ff11:22, ff12:33, ff13:28, ff14:27, ff15:35, ff16:24, ff17:31, ff18:18, ff19:26, ff1a:28, ff1b:28, ff1c:24, ff1d:21, ff1e:30, ff1f:29, ff20:35, ff21:38, ff22:31, ff23:37, ff24:32, ff25:34, ff26:18, ff27:22, ff28:30, ff29:29, ff2a:25, ff2b:23, ff2c:26, ff2d:36, ff2e:44, ff2f:27, ff30:38, ff31:38, ff32:27, ff33:56, ff34:19, ff35:27, ff36:31, ff37:61, ff38:27, ff39:26, ff3a:25, ff3b:41, ff3c:23, ff3d:32, ff3e:24, ff3f:51, ff40:39, ff41:34, ff42:21, ff43:21, ff44:26, ff45:35, ff46:25, ff47:33, ff48:27, ff49:35, ff4a:27, ff4b:31, ff4c:19, ff4d:25, ff4e:34, ff4f:31, ff50:22, ff51:23, ff52:17, ff53:35, ff54:29, ff55:28, ff56:29, ff57:30, ff58:21, ff59:30, ff5a:25, ff5b:35, ff5c:19, ff5d:33, ff5e:25, ff5f:26, ff60:32, ff61:35, ff62:31, ff63:26, ff64:29, ff65:20, ff66:32, ff67:23, ff68:38, ff69:27, ff6a:29, ff6b:27, ff6c:34, ff6d:32, ff6e:43, ff6f:24, ff70:22, ff71:24, ff72:34, ff73:50, ff74:30, ff75:26, ff76:28, ff77:60, ff78:36, ff79:21, ff7a:35, ff7b:47, ff7c:31, ff7d:27, ff7e:27, ff7f:59, ff80:35, ff81:35, ff82:33, ff83:22, ff84:27, ff85:20, ff86:23, ff87:24, ff88:21, ff89:24, ff8a:26, ff8b:31, ff8c:18, ff8d:23, ff8e:27, ff8f:41, ff90:22, ff91:21, ff92:26, ff93:28, ff94:32, ff95:27, ff96:27, ff97:29, ff98:33, ff99:29, ff9a:34, ff9b:32, ff9c:25, ff9d:33, ff9e:18, ff9f:28, ffa0:31, ffa1:25, ffa2:22, ffa3:29, ffa4:19, ffa5:20, ffa6:20, ffa7:32, ffa8:43, ffa9:33, ffaa:33, ffab:34, ffac:18, ffad:32, ffae:26, ffaf:16, ffb0:31, ffb1:23, ffb2:35, ffb3:50, ffb4:38, ffb5:29, ffb6:29, ffb7:43, ffb8:28, ffb9:22, ffba:34, ffbb:58, ffbc:32, ffbd:23, ffbe:34, ffbf:37, ffc0:24, ffc1:32, ffc2:32, ffc3:30, ffc4:28, ffc5:22, ffc6:28, ffc7:29, ffc8:31, ffc9:30, ffca:26, ffcb:32, ffcc:43, ffcd:45, ffce:56, ffcf:42, ffd0:29, ffd1:23, ffd2:24, ffd3:41, ffd4:22, ffd5:29, ffd6:31, ffd7:28, ffd8:27, ffd9:27, ffda:30, ffdb:25, ffdc:49, ffdd:42, ffde:47, ffdf:54, ffe0:25, ffe1:31, ffe2:32, ffe3:32, ffe4:37, ffe5:36, ffe6:36, ffe7:30, ffe8:31, ffe9:44, ffea:24, ffeb:31, ffec:47, ffed:49, ffee:58, ffef:45, fff0:24, fff1:24, fff2:30, fff3:45, fff4:36, fff5:27, fff6:36, fff7:48, fff8:29, fff9:25, fffa:38, fffb:49, fffc:48, fffd:42, fffe:37, ffff:68 - first markers at: [('0x307', '0xd2'), ('0x397', '0xb1'), ('0x428', '0xb0'), ('0x60f', '0xa0'), ('0x66c', '0x23'), ('0x691', '0x8d'), ('0x69e', '0x69'), ('0x79c', '0xd6'), ('0x827', '0x40'), ('0x8f4', '0x6a'), ('0x9fe', '0xa5'), ('0xa09', '0x88')] - comp[0:24] : 000000000000000000000000000000000000000000000000 - comp[-8:] : f7efdfbf7efdfbf7 - raw[504:544] : 00000000000000000000de521366dccd8cfe291a1c468992a578c2b0ad703f581c0b0667c07d2499 - raw[-8:] : dfbf7efdfbf7ffd9 - diff --git a/crates/crw/clean-room/logs/p1c-bounds-public.txt b/crates/crw/clean-room/logs/p1c-bounds-public.txt deleted file mode 100644 index 64aa1d69..00000000 --- a/crates/crw/clean-room/logs/p1c-bounds-public.txt +++ /dev/null @@ -1,11 +0,0 @@ -crw-canon-eos-10d-a.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 6241 total, 6241 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-10d-b.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 6503 total, 6503 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-10d-c.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 6139 total, 6139 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-300d-a.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 6418 total, 6418 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-300d-b.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 7180 total, 7180 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-d30.crw EOS layout: lowbits [0:809536) + zeros(514) + entropy; non-stuffed ff: 3450 total, 3450 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-d60.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 6419 total, 6419 in lowbits, 0 in entropy (expect 0) -crw-canon-eos-digital-rebel.crw EOS layout: lowbits [0:1629584) + zeros(514) + entropy; non-stuffed ff: 23394 total, 23394 in lowbits, 0 in entropy (expect 0) -crw-canon-powershot-g1.crw PS 10-bit layout: zeros(514) + entropy; no non-stuffed ff (pure stuffed stream) -crw-canon-powershot-g5.crw PS 12-bit layout: zeros(514) + lowbits [514:1315138) + entropy; all 5002 non-stuffed ff inside lowbits; first 697, last 1315128 -crw-canon-powershot-g6.crw PS 12-bit layout: zeros(514) + lowbits [514:1852274) + entropy; all 7928 non-stuffed ff inside lowbits; first 775, last 1852264 diff --git a/crates/crw/clean-room/logs/structure-d60-public.txt b/crates/crw/clean-room/logs/structure-d60-public.txt deleted file mode 100644 index 28337ee1..00000000 --- a/crates/crw/clean-room/logs/structure-d60-public.txt +++ /dev/null @@ -1,73 +0,0 @@ -== corpus/crw/crw-canon-eos-d60.crw - size 8591234, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x831768, offsetTbl @+0x831744, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x662502 off 0x0 - RawData: 6694146 bytes, entropy 7.9868 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.864, top bytes 8c:25, cb:21, 5d:18, 63:17, zeros 11, ff 4 - last 2 KiB: entropy 7.890, top bytes f1:16, 6d:16, 76:16, 55:15, zeros 11, ff 5 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x1cdc00 off 0x662502 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x1642 off 0x830102 - heap root/ImageProps: block @0x83011c len 0x1642, offsetTbl @+0x15ba, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x6d8f0000 off 0xcfba0d73 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xd8327aab - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0xf42a3 off 0x9792ef1a - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1467566074, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_0099.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_0099.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x830284 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x3fc00000 off 0xe2577c1e - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x830294 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS D60 CMOS RAW IMAGE' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS D60 CMOS RAW IMAGE' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x8302ee len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS D60\x00\x0e{\xd7\xd8e\x18iG\x03,,c' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x83032e len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x3645276f off 0x5982da24 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xa552febb - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0xfedcfe5 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.0.4' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x1336 off 0x284 - heap root/ImageProps/ExifInformation: block @0x8303a0 len 0x1336, offsetTbl @+0x129a, 15 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x0 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x36 off 0x0 - = u16 (54, 0, 160, 52, 32768, 245, 65504, 0, 3, 0, 8, 8, 0, 0, 12288, 0, 0, 0, 1, 0, 0, 32768, 244, 99, 0, 0, 252) - [3] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x56 off 0x36 - [4] tag 0x0036 id 0x0036 ? byte heap len 0xbd6 off 0x8c - [5] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x1668000 off 0xed8d4a16 - [6] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xc62 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [7] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xc84 - = u32 (0, 0, 514, 5064559) - [8] tag 0x1033 id 0x1033 CustomFunctions word heap len 0x22 off 0xc94 - = u16 (32, 0, 515, 768, 1025, 1280, 1536, 1792, 2048, 2304, 2560, 2818, 3074, 3328, 3584, 3841, 18747) - [9] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xcb6 - = u16 (26, 279, 368, 371, 172, 219, 434, 436, 284, 162, 491, 494, 431) - [10] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xcd0 - = u16 (26, 281, 413, 415, 193, 223, 488, 490, 319, 150, 505, 507, 439) - [11] tag 0x10c2 id 0x10c2 ? word heap len 0x554 off 0xcea - [12] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x123e - = u16 (10, 440, 1021, 1027, 563) - [13] tag 0x10a8 id 0x10a8 ? word heap len 0x10 off 0x1248 - = u16 (16, 6000, 5312, 5989, 3161, 3717, 6503, 4118) - [14] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x42 off 0x1258 - diff --git a/crates/crw/clean-room/logs/structure-public.txt b/crates/crw/clean-room/logs/structure-public.txt deleted file mode 100644 index e5674341..00000000 --- a/crates/crw/clean-room/logs/structure-public.txt +++ /dev/null @@ -1,887 +0,0 @@ -== corpus/crw/crw-canon-eos-10d-a.crw - size 7787330, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x76d328, offsetTbl @+0x76d304, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x6cdcf6 off 0x0 - RawData: 7134454 bytes, entropy 7.9776 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.877, top bytes db:20, d3:16, b6:16, de:16, zeros 6, ff 3 - last 2 KiB: entropy 7.882, top bytes ac:17, d4:17, 91:17, 6c:17, zeros 12, ff 4 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x9dde2 off 0x6cdcf6 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x182c off 0x76bad8 - heap root/ImageProps: block @0x76baf2 len 0x182c, offsetTbl @+0x17a4, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x3af0000 off 0x9b869aeb - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xf5657a8 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x1af8f9 off 0xde23c13d - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1228574022, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_7673.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_7673.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x76bc5a len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x40cc0000 off 0x70138c52 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x76bc6a len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS 10D CMOS RAW IMAGE' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS 10D CMOS RAW IMAGE' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x76bcc4 len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'Horst Kakuschke' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS 10D\x00:g\x8fZ\xb5k\xa7\xea\xca\xda|\x1a' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x76bd04 len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0xdb7145a off 0xde776b76 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0x8762cef2 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0xe63b7d8c - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 2.0.1' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x1520 off 0x284 - heap root/ImageProps/ExifInformation: block @0x76bd76 len 0x1520, offsetTbl @+0x142a, 24 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x180000 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x42 off 0x0 - [3] tag 0x1093 id 0x1093 ? word heap len 0x12 off 0x42 - = u16 (18, 0, 0, 0, 0, 0, 65535, 65535, 65535) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x54 - [5] tag 0x0036 id 0x0036 ? byte heap len 0xe82 off 0xb0 - [6] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x80000168 off 0x4f2612e3 - [7] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xf32 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [8] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xf54 - = u32 (0, 0, 514, 5504868) - [9] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x30 off 0xf64 - = u16 (7, 7, 3072, 2048, 3072, 2048, 151, 151, 1014, 608, 0, 0, 0, 64928, 64522, 0, 0, 65030, 0, 506, 0, 0, 8, 65535) - [10] tag 0x1033 id 0x1033 CustomFunctions word heap len 0x24 off 0xf94 - = u16 (36, 256, 513, 768, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2818, 3072, 3328, 3584, 3841, 4097, 4352) - [11] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xfb8 - = u16 (26, 288, 343, 343, 160, 231, 433, 433, 275, 154, 444, 444, 384) - [12] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xfd2 - = u16 (26, 314, 364, 365, 165, 234, 428, 430, 262, 174, 484, 487, 412) - [13] tag 0x10c2 id 0x10c2 ? word heap len 0x374 off 0xfec - [14] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x1360 - = u16 (10, 418, 1021, 1027, 467) - [15] tag 0x10a8 id 0x10a8 ? word heap len 0x14 off 0x136a - = u16 (20, 4174, 5190, 7000, 6000, 3232, 3934, 6083, 5190, 5200) - [16] tag 0x10ad id 0x10ad ? word heap len 0x3e off 0x137e - = u16 (62, 21468, 8650, 7000, 6000, 5600, 5200, 4700, 4200, 3343, 1668, 64833, 65115, 65180, 65240, 65271, 65307, 65361, 65429, 64, 582, 65341, 65183, 65148, 65121, 65106, 65097, 65083, 65072, 65051, 65251) - [17] tag 0x10ae id 0x10ae ColorTemperature word heap len 0x2 off 0x140e - = u16 (5200,) - [18] tag 0x10af id 0x10af ? word heap len 0x2 off 0x1410 - = u16 (1024,) - [19] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x52 off 0x13bc - [20] tag 0x10b4 id 0x10b4 ColorSpace word heap len 0x2 off 0x1412 - = u16 (1,) - [21] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x1414 - = u16 (10, 2, 2, 1536, 1024) - [22] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x141e - = u16 (0, 159, 7, 112) - [23] tag 0x183b id 0x183b SerialNumberFormat dword heap len 0x4 off 0x1426 - = u32 (2415919104,) - -== corpus/crw/crw-canon-eos-10d-b.crw - size 10822040, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0xa5217e, offsetTbl @+0xa5215a, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x6e8c3a off 0x0 - RawData: 7244858 bytes, entropy 7.9718 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.894, top bytes 8c:18, 23:17, 5b:16, 32:16, zeros 14, ff 2 - last 2 KiB: entropy 7.824, top bytes 52:20, 8c:19, 84:19, 92:18, zeros 17, ff 2 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x367cf4 off 0x6e8c3a - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x182c off 0xa5092e - heap root/ImageProps: block @0xa50948 len 0x182c, offsetTbl @+0x17a4, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x0 off 0x1 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0x80002400 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x127edf off 0x1001000 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1105175848, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_2127.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_2127.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0xa50ab0 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x40a80000 off 0x0 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0xa50ac0 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS 10D CMOS RAW IMAGE' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS 10D CMOS RAW IMAGE' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0xa50b1a len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'Bill Ferguson' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS 10D\x00\x00\x00\x00\x06\x00\x04 \x80\x01\x04' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0xa50b5a len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x54ab324a off 0x200100 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0x1042 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0x1000000 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 2.0.1' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x1520 off 0x284 - heap root/ImageProps/ExifInformation: block @0xa50bcc len 0x1520, offsetTbl @+0x142a, 24 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x550000 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x42 off 0x0 - [3] tag 0x1093 id 0x1093 ? word heap len 0x12 off 0x42 - = u16 (18, 0, 0, 0, 0, 0, 65535, 65535, 65535) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x54 - [5] tag 0x0036 id 0x0036 ? byte heap len 0xe82 off 0xb0 - [6] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x80000168 off 0x20 - [7] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xf32 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [8] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xf54 - = u32 (2, 0, 514, 5615272) - [9] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x30 off 0xf64 - = u16 (7, 7, 3072, 2048, 3072, 2048, 151, 151, 1014, 608, 0, 0, 0, 64928, 64522, 0, 0, 65030, 0, 506, 0, 0, 31, 65535) - [10] tag 0x1033 id 0x1033 CustomFunctions word heap len 0x24 off 0xf94 - = u16 (36, 256, 513, 768, 1024, 1280, 1537, 1792, 2053, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4352) - [11] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xfb8 - = u16 (26, 353, 375, 375, 177, 268, 431, 432, 285, 190, 476, 477, 428) - [12] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xfd2 - = u16 (26, 326, 364, 365, 161, 254, 428, 430, 268, 186, 484, 487, 416) - [13] tag 0x10c2 id 0x10c2 ? word heap len 0x374 off 0xfec - [14] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x1360 - = u16 (10, 345, 1021, 1027, 1301) - [15] tag 0x10a8 id 0x10a8 ? word heap len 0x14 off 0x136a - = u16 (20, 4028, 5190, 7000, 6000, 3232, 3930, 6080, 2818, 5200) - [16] tag 0x10ad id 0x10ad ? word heap len 0x3e off 0x137e - = u16 (62, 21468, 8650, 7000, 6000, 5600, 5200, 4700, 4200, 3343, 1668, 64821, 65116, 65184, 65246, 65279, 65317, 65373, 65444, 83, 613, 65393, 65229, 65191, 65165, 65148, 65139, 65125, 65112, 65100, 65341) - [17] tag 0x10ae id 0x10ae ColorTemperature word heap len 0x2 off 0x140e - = u16 (5200,) - [18] tag 0x10af id 0x10af ? word heap len 0x2 off 0x1410 - = u16 (1024,) - [19] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x52 off 0x13bc - [20] tag 0x10b4 id 0x10b4 ColorSpace word heap len 0x2 off 0x1412 - = u16 (1,) - [21] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x1414 - = u16 (10, 3, 0, 3072, 2048) - [22] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x141e - = u16 (0, 159, 7, 112) - [23] tag 0x183b id 0x183b SerialNumberFormat dword heap len 0x4 off 0x1426 - = u32 (2415919104,) - -== corpus/crw/crw-canon-eos-10d-c.crw - size 10583702, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0xa17e7c, offsetTbl @+0xa17e58, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x71c324 off 0x0 - RawData: 7455524 bytes, entropy 7.9666 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.847, top bytes 82:21, b2:18, 98:18, 84:18, zeros 15, ff 1 - last 2 KiB: entropy 7.785, top bytes 08:23, 9c:20, 38:20, cc:20, zeros 7, ff 0 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x2fa308 off 0x71c324 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x182c off 0xa1662c - heap root/ImageProps: block @0xa16646 len 0x182c, offsetTbl @+0x17a4, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x41020000 off 0xf2c31662 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xc3af7160 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x12f557 off 0x777d1c94 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1109761444, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_2455.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_2455.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0xa167ae len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x41160000 off 0x9a0c5382 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0xa167be len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS 10D CMOS RAW IMAGE' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS 10D CMOS RAW IMAGE' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0xa16818 len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'Bill Ferguson' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS 10D\x00\xfe\x19g\xba1\x94E\x02\x1cc(\xca' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0xa16858 len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x54ab324a off 0x18e17812 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xd734ef03 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0x70fc1863 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 2.0.1' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x1520 off 0x284 - heap root/ImageProps/ExifInformation: block @0xa168ca len 0x1520, offsetTbl @+0x142a, 24 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x180000 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x42 off 0x0 - [3] tag 0x1093 id 0x1093 ? word heap len 0x12 off 0x42 - = u16 (18, 0, 0, 0, 0, 0, 65535, 65535, 65535) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x54 - [5] tag 0x0036 id 0x0036 ? byte heap len 0xe82 off 0xb0 - [6] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x80000168 off 0x1ba7180e - [7] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xf32 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [8] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xf54 - = u32 (2, 0, 514, 5825938) - [9] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x30 off 0xf64 - = u16 (7, 7, 3072, 2048, 3072, 2048, 151, 151, 1014, 608, 0, 0, 0, 64928, 64522, 0, 0, 65030, 0, 506, 0, 0, 113, 65535) - [10] tag 0x1033 id 0x1033 CustomFunctions word heap len 0x24 off 0xf94 - = u16 (36, 256, 513, 768, 1024, 1280, 1537, 1792, 2053, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4352) - [11] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xfb8 - = u16 (26, 353, 375, 375, 177, 268, 431, 432, 285, 190, 476, 477, 428) - [12] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xfd2 - = u16 (26, 326, 364, 365, 161, 254, 428, 430, 268, 186, 484, 487, 416) - [13] tag 0x10c2 id 0x10c2 ? word heap len 0x374 off 0xfec - [14] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x1360 - = u16 (10, 508, 1021, 1027, 856) - [15] tag 0x10a8 id 0x10a8 ? word heap len 0x14 off 0x136a - = u16 (20, 5347, 5190, 7000, 6000, 3232, 3930, 6080, 2818, 5200) - [16] tag 0x10ad id 0x10ad ? word heap len 0x3e off 0x137e - = u16 (62, 21468, 8650, 7000, 6000, 5600, 5200, 4700, 4200, 3343, 1668, 64821, 65116, 65184, 65246, 65279, 65317, 65373, 65444, 83, 613, 65393, 65229, 65191, 65165, 65148, 65139, 65125, 65112, 65100, 65341) - [17] tag 0x10ae id 0x10ae ColorTemperature word heap len 0x2 off 0x140e - = u16 (5200,) - [18] tag 0x10af id 0x10af ? word heap len 0x2 off 0x1410 - = u16 (1024,) - [19] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x52 off 0x13bc - [20] tag 0x10b4 id 0x10b4 ColorSpace word heap len 0x2 off 0x1412 - = u16 (1,) - [21] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x1414 - = u16 (10, 3, 0, 3072, 2048) - [22] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x141e - = u16 (0, 159, 7, 112) - [23] tag 0x183b id 0x183b SerialNumberFormat dword heap len 0x4 off 0x1426 - = u32 (2415919104,) - -== corpus/crw/crw-canon-eos-300d-a.crw - size 9056348, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x8a3042, offsetTbl @+0x8a301e, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x69bce6 off 0x0 - RawData: 6929638 bytes, entropy 7.9804 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.886, top bytes bb:21, 9c:20, 8c:17, 18:14, zeros 13, ff 10 - last 2 KiB: entropy 7.850, top bytes ae:20, bb:18, e3:18, 6e:18, zeros 7, ff 4 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x205b64 off 0x69bce6 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x17d4 off 0x8a184a - heap root/ImageProps: block @0x8a1864 len 0x17d4, offsetTbl @+0x174c, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0x0 - = u32 (3072, 2048, 1065353216, 270, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x9ceb0000 off 0xc97abdef - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xb1dab4b5 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0xfb8bd off 0x0 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x1c - = u32 (1083408238, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_0333.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_0333.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x8a19cc len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x41100000 off 0xf5e6df2f - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x58 off 0x178 - heap root/ImageProps/ImageDescription: block @0x8a19dc len 0x58, offsetTbl @+0x3e, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS 300D DIGITAL CMOS RAW' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x1e off 0x20 - = b'CRW:EOS 300D DIGITAL CMOS RAW' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0x8a off 0x1d0 - heap root/ImageProps/CameraObject: block @0x8a1a34 len 0x8a, offsetTbl @+0x66, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x1 off 0x0 - = b'' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x1d off 0x2 - = b'Canon\x00Canon EOS 300D DIGITAL' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x46 off 0x20 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x8a1a54 len 0x46, offsetTbl @+0x18, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x3d6b01d9 off 0x8afd725d - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xe53a2db9 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0x6efb17d7 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x17 off 0x0 - = b'Firmware Version 1.1.1' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x14f2 off 0x25a - heap root/ImageProps/ExifInformation: block @0x8a1abe len 0x14f2, offsetTbl @+0x1406, 23 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x460000 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x42 off 0x0 - [3] tag 0x1093 id 0x1093 ? word heap len 0x12 off 0x42 - = u16 (18, 0, 0, 0, 0, 0, 65535, 65535, 65535) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x54 - [5] tag 0x0036 id 0x0036 ? byte heap len 0xe82 off 0xb0 - [6] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x80000170 off 0x79d599bf - [7] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xf32 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [8] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xf54 - = u32 (0, 0, 514, 5300052) - [9] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x30 off 0xf64 - = u16 (48, 7, 3072, 2048, 3072, 2048, 151, 151, 1014, 608, 0, 0, 0, 64928, 64522, 0, 0, 65030, 0, 506, 0, 0, 8, 65535) - [10] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xf94 - = u16 (26, 334, 372, 372, 175, 246, 427, 427, 279, 185, 488, 490, 443) - [11] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xfae - = u16 (26, 297, 375, 375, 168, 203, 394, 395, 244, 153, 453, 454, 388) - [12] tag 0x10c2 id 0x10c2 ? word heap len 0x374 off 0xfc8 - [13] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x133c - = u16 (10, 489, 1022, 1026, 668) - [14] tag 0x10a8 id 0x10a8 ? word heap len 0x14 off 0x1346 - = u16 (20, 4709, 5200, 6983, 5987, 3213, 3888, 6196, 2500, 5190) - [15] tag 0x10ad id 0x10ad ? word heap len 0x3e off 0x135a - = u16 (62, 20127, 8480, 7000, 6000, 5600, 5200, 4724, 4200, 3266, 1446, 64813, 65123, 65187, 65248, 65281, 65320, 65371, 65446, 114, 674, 65436, 65225, 65196, 65169, 65152, 65137, 65124, 65112, 65098, 65348) - [16] tag 0x10ae id 0x10ae ColorTemperature word heap len 0x2 off 0x1398 - = u16 (5200,) - [17] tag 0x10af id 0x10af ? word heap len 0x2 off 0x139a - = u16 (1024,) - [18] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x52 off 0x139c - [19] tag 0x10b4 id 0x10b4 ColorSpace word heap len 0x2 off 0x13ee - = u16 (1,) - [20] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x13f0 - = u16 (10, 3, 1, 2048, 1360) - [21] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x13fa - = u16 (0, 159, 7, 112) - [22] tag 0x183b id 0x183b SerialNumberFormat dword heap len 0x4 off 0x1402 - = u32 (2415919104,) - -== corpus/crw/crw-canon-eos-300d-b.crw - size 8389842, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x8004b8, offsetTbl @+0x800494, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x6df56a off 0x0 - RawData: 7206250 bytes, entropy 7.9810 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.903, top bytes d7:20, 7b:18, 32:17, 83:16, zeros 14, ff 6 - last 2 KiB: entropy 7.840, top bytes 00:29, ff:23, fc:21, af:20, zeros 29, ff 23 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x11f72c off 0x6df56a - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x17fe off 0x7fec96 - heap root/ImageProps: block @0x7fecb0 len 0x17fe, offsetTbl @+0x1776, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 270, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x17890000 off 0x637f057b - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0x9cae3253 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x2b4f7b off 0x7aaebd64 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1182188238, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_8395.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_8395.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x7fee18 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x41040000 off 0xde864d39 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x7fee28 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS 300D DIGITAL CMOS RAW' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS 300D DIGITAL CMOS RAW' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x7fee82 len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS 300D DIGITAL\x00\x97\xd4\xc4' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x7feec2 len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x3d6aecf7 off 0x46ec30fc - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0x5aa35384 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0xf1421364 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.1.1' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x14f2 off 0x284 - heap root/ImageProps/ExifInformation: block @0x7fef34 len 0x14f2, offsetTbl @+0x1406, 23 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x150000 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x42 off 0x0 - [3] tag 0x1093 id 0x1093 ? word heap len 0x12 off 0x42 - = u16 (18, 0, 0, 0, 0, 0, 65535, 65535, 65535) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x54 - [5] tag 0x0036 id 0x0036 ? byte heap len 0xe82 off 0xb0 - [6] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x80000170 off 0xedbad5b1 - [7] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xf32 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [8] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xf54 - = u32 (2, 0, 514, 5576664) - [9] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x30 off 0xf64 - = u16 (7, 7, 3072, 2048, 3072, 2048, 151, 151, 1014, 608, 0, 0, 0, 64928, 64522, 0, 0, 65030, 0, 506, 0, 0, 36, 65535) - [10] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xf94 - = u16 (26, 328, 374, 372, 163, 244, 430, 428, 262, 162, 436, 435, 366) - [11] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xfae - = u16 (26, 301, 375, 375, 166, 206, 394, 395, 242, 155, 453, 454, 383) - [12] tag 0x10c2 id 0x10c2 ? word heap len 0x374 off 0xfc8 - [13] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x133c - = u16 (10, 314, 1022, 1026, 1095) - [14] tag 0x10a8 id 0x10a8 ? word heap len 0x14 off 0x1346 - = u16 (20, 5781, 5200, 7000, 5975, 3211, 3882, 6203, 4166, 5190) - [15] tag 0x10ad id 0x10ad ? word heap len 0x3e off 0x135a - = u16 (62, 20127, 8480, 7000, 6000, 5600, 5200, 4724, 4200, 3266, 1446, 64849, 65150, 65212, 65271, 65304, 65342, 65392, 65464, 125, 664, 65367, 65170, 65143, 65117, 65102, 65088, 65078, 65068, 65055, 65302) - [16] tag 0x10ae id 0x10ae ColorTemperature word heap len 0x2 off 0x13ea - = u16 (5200,) - [17] tag 0x10af id 0x10af ? word heap len 0x2 off 0x13ec - = u16 (1024,) - [18] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x52 off 0x1398 - [19] tag 0x10b4 id 0x10b4 ColorSpace word heap len 0x2 off 0x13ee - = u16 (2,) - [20] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x13f0 - = u16 (10, 3, 2, 1536, 1024) - [21] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x13fa - = u16 (0, 159, 7, 112) - [22] tag 0x183b id 0x183b SerialNumberFormat dword heap len 0x4 off 0x1402 - = u32 (2415919104,) - -== corpus/crw/crw-canon-eos-d30.crw - size 2993260, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x2dac52, offsetTbl @+0x2dac2e, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x2b19c8 off 0x0 - RawData: 2824648 bytes, entropy 7.9671 bits/byte - first 2 KiB: entropy 7.540, top bytes ef:35, fb:34, ae:33, bf:31, zeros 1, ff 24 - middle 2 KiB: entropy 7.835, top bytes ac:21, 95:19, b6:18, ab:18, zeros 11, ff 3 - last 2 KiB: entropy 7.792, top bytes 02:22, 4c:21, 00:20, 26:19, zeros 20, ff 2 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x28128 off 0x2b19c8 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x113e off 0x2d9af0 - heap root/ImageProps: block @0x2d9b0a len 0x113e, offsetTbl @+0x10b6, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (2160, 1440, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0xc0200000 off 0x130000 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0x2108c00 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x22378c off 0x7400000 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1482512730, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_2444.CRW\x00\x9a\x02]G\x18\xa6JG\xff\x80 H\x00@\x00\x02\x82' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_2444.THM\x00\x04\x04\x010\x02\x00\x80\x00\x00\x00@\x12\x14\x00\x00\x00 ' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x2d9c72 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0xbec00000 off 0xedfff6ff - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x2d9c82 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS D30 CMOS RAW IMAGE\x00\xff\x9f\xff\xfd\xf7\xfd\xfd\xff\xff' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS D30 CMOS RAW IMAGE\x00\xdf\xff\xde\xff\xff' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x2d9cdc len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'\x00\xdf\xfb\xff\xff\xff\xb5\xef\xff\xff_\xef\xff\xdf\xef\xfb\xff\xff\xff\xfb\xff\xff{\xff\xfe\xff\xff\x7f\xfd?\xdf\xff' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS D30\x00\x00\xff\xff\xff\xff\xff\x7f\xbf\xfd\xff\xffo' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x2d9d1c len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x172700aa off 0xfffaffff - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xfffff1ff - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0xffffff5f - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.02\x00\xbf\xff\xff\xff\xe7\xff\xff\xdf\xff}' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0xe32 off 0x284 - heap root/ImageProps/ExifInformation: block @0x2d9d8e len 0xe32, offsetTbl @+0xdaa, 13 records - [0] tag 0x0001 id 0x0001 FreeBytes byte heap len 0xc off 0x0 - [1] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x400048 - [2] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x320000 off 0x2620392 - [3] tag 0x102a id 0x102a CanonShotInfo word heap len 0x32 off 0xc - = u16 (50, 0, 288, 65484, 31, 181, 0, 0, 3, 1, 8, 8, 0, 0, 12290, 0, 0, 0, 1, 0, 0, 32, 180, 75, 0) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x40 off 0x3e - = u16 (64, 2, 0, 4, 0, 1, 0, 0, 0, 2, 0, 1, 0, 0, 0, 0, 0, 5, 2, 12291, 4, 0, 198, 50, 50, 1, 31, 285, 255, 0, 0, 0) - [5] tag 0x102c id 0x102c CanonColorInfo2 word heap len 0x100 off 0x7e - [6] tag 0x0032 id 0x0032 ? byte heap len 0x300 off 0x17e - [7] tag 0x0036 id 0x0036 ? byte heap len 0x868 off 0x47e - [8] tag 0x1030 id 0x1030 WhiteSample word heap len 0x76 off 0xce6 - [9] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x1140000 off 0x400c - [10] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xd5c - = u16 (34, 2224, 1456, 1, 0, 56, 12, 2215, 1451, 8, 220, 40, 1244, 0, 0, 0, 0) - [11] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xd7e - = u32 (1, 0, 514, 2015110) - [12] tag 0x1033 id 0x1033 CustomFunctions word heap len 0x1c off 0xd8e - = u16 (28, 256, 514, 768, 1025, 1280, 1536, 1794, 2049, 2305, 2560, 2816, 3074, 3328) - -== corpus/crw/crw-canon-eos-d60.crw - size 8591234, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x831768, offsetTbl @+0x831744, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x662502 off 0x0 - RawData: 6694146 bytes, entropy 7.9868 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.864, top bytes 8c:25, cb:21, 5d:18, 63:17, zeros 11, ff 4 - last 2 KiB: entropy 7.890, top bytes f1:16, 6d:16, 76:16, 55:15, zeros 11, ff 5 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x1cdc00 off 0x662502 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x1642 off 0x830102 - heap root/ImageProps: block @0x83011c len 0x1642, offsetTbl @+0x15ba, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0x6d8f0000 off 0xcfba0d73 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xd8327aab - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0xf42a3 off 0x9792ef1a - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1467566074, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_0099.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_0099.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x830284 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x3fc00000 off 0xe2577c1e - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x830294 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS D60 CMOS RAW IMAGE' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS D60 CMOS RAW IMAGE' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x8302ee len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS D60\x00\x0e{\xd7\xd8e\x18iG\x03,,c' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x83032e len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x3645276f off 0x5982da24 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xa552febb - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0xfedcfe5 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.0.4' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x1336 off 0x284 - heap root/ImageProps/ExifInformation: block @0x8303a0 len 0x1336, offsetTbl @+0x129a, 15 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x0 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x36 off 0x0 - = u16 (54, 0, 160, 52, 32768, 245, 65504, 0, 3, 0, 8, 8, 0, 0, 12288, 0, 0, 0, 1, 0, 0, 32768, 244, 99, 0, 0, 252) - [3] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x56 off 0x36 - [4] tag 0x0036 id 0x0036 ? byte heap len 0xbd6 off 0x8c - [5] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x1668000 off 0xed8d4a16 - [6] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xc62 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [7] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xc84 - = u32 (0, 0, 514, 5064559) - [8] tag 0x1033 id 0x1033 CustomFunctions word heap len 0x22 off 0xc94 - = u16 (32, 0, 515, 768, 1025, 1280, 1536, 1792, 2048, 2304, 2560, 2818, 3074, 3328, 3584, 3841, 18747) - [9] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xcb6 - = u16 (26, 279, 368, 371, 172, 219, 434, 436, 284, 162, 491, 494, 431) - [10] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xcd0 - = u16 (26, 281, 413, 415, 193, 223, 488, 490, 319, 150, 505, 507, 439) - [11] tag 0x10c2 id 0x10c2 ? word heap len 0x554 off 0xcea - [12] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x123e - = u16 (10, 440, 1021, 1027, 563) - [13] tag 0x10a8 id 0x10a8 ? word heap len 0x10 off 0x1248 - = u16 (16, 6000, 5312, 5989, 3161, 3717, 6503, 4118) - [14] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x42 off 0x1258 - -== corpus/crw/crw-canon-eos-digital-rebel.crw - size 10153786, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x9aef20, offsetTbl @+0x9aeefc, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x61c014 off 0x0 - RawData: 6406164 bytes, entropy 7.8363 bits/byte - first 2 KiB: entropy -0.000, top bytes 00:2048, zeros 2048, ff 0 - middle 2 KiB: entropy 7.825, top bytes 18:24, 59:18, cd:17, 06:17, zeros 10, ff 6 - last 2 KiB: entropy 7.836, top bytes b5:19, 5a:17, 2c:17, 56:17, zeros 10, ff 2 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x3916ea off 0x61c014 - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x17fe off 0x9ad6fe - heap root/ImageProps: block @0x9ad718 len 0x17fe, offsetTbl @+0x1776, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2048, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0xdfff0000 off 0x7ff757d7 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0x7d5d557f - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x221046 off 0x7df757dd - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1480372394, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_2390.CRW' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_2390.THM' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x9ad880 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0xbec00000 off 0xd7dd55f5 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x9ad890 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'EOS DIGITAL REBEL CMOS RAW' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:EOS DIGITAL REBEL CMOS RAW' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xb2 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x9ad8ea len 0xb2, offsetTbl @+0x8e, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon EOS DIGITAL REBEL\x00w\x7f' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x4e off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x9ad92a len 0x4e, offsetTbl @+0x20, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x5116c2ff off 0xf55d5ff7 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0x5ff555f5 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0x5ddd77ff - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.1.1' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x14f2 off 0x284 - heap root/ImageProps/ExifInformation: block @0x9ad99c len 0x14f2, offsetTbl @+0x1406, 23 records - [0] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x64 off 0x0 - [1] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x320000 off 0x2620392 - [2] tag 0x102a id 0x102a CanonShotInfo word heap len 0x42 off 0x0 - [3] tag 0x1093 id 0x1093 ? word heap len 0x12 off 0x42 - = u16 (18, 0, 0, 0, 0, 0, 65535, 65535, 65535) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x54 - [5] tag 0x0036 id 0x0036 ? byte heap len 0xe82 off 0xb0 - [6] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x80000170 off 0xdf777555 - [7] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0xf32 - = u16 (34, 3152, 2068, 1, 1, 72, 16, 3143, 2063, 24, 224, 40, 1856, 0, 0, 0, 0) - [8] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0xf54 - = u32 (0, 0, 514, 4776577) - [9] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x30 off 0xf64 - = u16 (7, 7, 3072, 2048, 3072, 2048, 151, 151, 1014, 608, 0, 0, 0, 64928, 64522, 0, 0, 65030, 0, 506, 0, 0, 8, 65535) - [10] tag 0x10c0 id 0x10c0 ? word heap len 0x1a off 0xf94 - = u16 (26, 327, 378, 369, 173, 258, 466, 457, 290, 186, 508, 497, 420) - [11] tag 0x10c1 id 0x10c1 ? word heap len 0x1a off 0xfae - = u16 (26, 304, 375, 375, 168, 206, 394, 395, 235, 158, 453, 454, 363) - [12] tag 0x10c2 id 0x10c2 ? word heap len 0x374 off 0xfc8 - [13] tag 0x10aa id 0x10aa ? word heap len 0xa off 0x133c - = u16 (10, 743, 1022, 1026, 483) - [14] tag 0x10a8 id 0x10a8 ? word heap len 0x14 off 0x1346 - = u16 (20, 2993, 5200, 6982, 5987, 3210, 3894, 6220, 2675, 5190) - [15] tag 0x10ad id 0x10ad ? word heap len 0x3e off 0x135a - = u16 (62, 20127, 8480, 7000, 6000, 5600, 5200, 4724, 4200, 3266, 1446, 64839, 65135, 65196, 65255, 65286, 65324, 65373, 65444, 110, 671, 65391, 65193, 65166, 65140, 65124, 65110, 65099, 65088, 65070, 65282) - [16] tag 0x10ae id 0x10ae ColorTemperature word heap len 0x2 off 0x13ea - = u16 (5200,) - [17] tag 0x10af id 0x10af ? word heap len 0x2 off 0x13ec - = u16 (1268,) - [18] tag 0x10a9 id 0x10a9 ColorBalance word heap len 0x52 off 0x1398 - [19] tag 0x10b4 id 0x10b4 ColorSpace word heap len 0x2 off 0x13ee - = u16 (1,) - [20] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x13f0 - = u16 (10, 3, 0, 3072, 2048) - [21] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x13fa - = u16 (0, 159, 7, 112) - [22] tag 0x183b id 0x183b SerialNumberFormat dword heap len 0x4 off 0x1402 - = u32 (2415919104,) - -== corpus/crw/crw-canon-powershot-g1.crw - size 2300702, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x231b04, offsetTbl @+0x231ae0, 3 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x2271ca off 0x0 - RawData: 2257354 bytes, entropy 7.9338 bits/byte - first 2 KiB: entropy 6.516, top bytes 00:522, 22:26, 24:21, 64:18, zeros 522, ff 4 - middle 2 KiB: entropy 7.827, top bytes 5a:23, 6d:23, 99:17, aa:17, zeros 8, ff 1 - last 2 KiB: entropy 7.669, top bytes 44:31, 4c:30, 88:26, 20:24, zeros 16, ff 9 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x9394 off 0x2271ca - starts with JPEG SOI - [2] tag 0x300a id 0x300a ImageProps heap2 heap len 0x1582 off 0x23055e - heap root/ImageProps: block @0x230578 len 0x1582, offsetTbl @+0x14fa, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0x0 - = u32 (2048, 1536, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0xc5600000 off 0xbee19e93 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0x51bd76d5 - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x11b9dd off 0x0 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x1c - = u32 (1065594172, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_1693.CRW\x00\x8e&\xe1>4\xd2\xd2\xa1ko\xddv?\xa5\x0f\xa8C\x0eY' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_1693.THM\x00~,\x07\xef\x19\xef\x88\xe3\xef2t\x89\n\x82\x88\x84\xe97\xe8' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x2306e0 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x0 off 0xab71c23a - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x58 off 0x178 - heap root/ImageProps/ImageDescription: block @0x2306f0 len 0x58, offsetTbl @+0x3e, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'High definition CCD image\x00\xfd\xa31\x18u\x0b' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x1e off 0x20 - = b'CRW:High definition CCD image' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0x8e off 0x1d0 - heap root/ImageProps/CameraObject: block @0x230748 len 0x8e, offsetTbl @+0x6a, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0xc off 0x0 - = b'Kelly Sauke' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x19 off 0xc - = b'Canon\x00Canon PowerShot G1' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x44 off 0x26 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x23076e len 0x44, offsetTbl @+0x16, 4 records - [0] tag 0x580b id 0x180b SerialNumber dword inrec len 0x0 off 0x6eb7053 - [1] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xf13edcd9 - [2] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0xa9d94930 - [3] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x16 off 0x0 - = b'Firmware Version 1.02' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x129c off 0x25e - heap root/ImageProps/ExifInformation: block @0x2307d6 len 0x129c, offsetTbl @+0x121e, 12 records - [0] tag 0x0001 id 0x0001 FreeBytes byte heap len 0xc off 0x0 - [1] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x0 off 0x0 - [2] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x1d30002 off 0xbd00fc - [3] tag 0x102a id 0x102a CanonShotInfo word heap len 0x34 off 0xc - = u16 (52, 0, 128, 115, 73, 170, 0, 0, 0, 0, 24, 0, 0, 0, 0, 0, 0, 0, 1, 494, 0, 79, 164, 0, 0, 0) - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x46 off 0x40 - [5] tag 0x102c id 0x102c CanonColorInfo2 word heap len 0x100 off 0x86 - [6] tag 0x0032 id 0x0032 ? byte heap len 0x800 off 0x186 - [7] tag 0x0036 id 0x0036 ? byte heap len 0x800 off 0x986 - [8] tag 0x1030 id 0x1030 WhiteSample word heap len 0x66 off 0x1186 - [9] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x4040000 off 0x2255b19d - [10] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0x11ec - = u16 (34, 2144, 1560, 1, 1, 24, 15, 2071, 1550, 2108, 15, 2111, 1550, 0, 0, 0, 0) - [11] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0x120e - = u32 (1, 0, 514, 2257351) - -== corpus/crw/crw-canon-powershot-g5.crw - size 4894440, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x4aaece, offsetTbl @+0x4aaea0, 4 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x4a2f6e off 0x0 - RawData: 4861806 bytes, entropy 7.9622 bits/byte - first 2 KiB: entropy 6.656, top bytes 00:529, bb:14, b8:13, 73:13, zeros 529, ff 4 - middle 2 KiB: entropy 7.819, top bytes b2:24, 55:22, 75:21, 57:21, zeros 8, ff 1 - last 2 KiB: entropy 7.680, top bytes 89:34, 44:34, 4c:28, 88:28, zeros 20, ff 6 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0x530e off 0x4a2f6e - starts with JPEG SOI - [2] tag 0x2008 id 0x2008 ThumbnailImage struct heap len 0x9fe off 0x4a827c - starts with JPEG SOI - [3] tag 0x300a id 0x300a ImageProps heap2 heap len 0x2226 off 0x4a8c7a - heap root/ImageProps: block @0x4a8c94 len 0x2226, offsetTbl @+0x219e, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (2592, 1944, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0xc9e90000 off 0x1fba0082 - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xfc48c47b - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x10078a off 0xa0fcb31 - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1059363106, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_0506.CRW\x001Y)\x97\xab\xf9n8\xba)\xd7\x9d\xd9\xc2\x13\x8a 9\x01' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_0506.THM\x00\x05\xc3!\xe5\xbc\xb1<\xf83\xde\xe1E\x93\xb1e5\xab\x81\x92' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x4a8dfc len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x0 off 0xff9410 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x4a8e0c len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'High definition CCD image\x00\xad\xc1<\xb1\x80\xc0' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:High definition CCD image\x00LF' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xa8 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x4a8e66 len 0xa8, offsetTbl @+0x84, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'\x008\x9e!,v\xf7\xc7\x18\xc4c"K\xfe\xe8;\xf4=\x9f\x1e\'9\xe3\x19nq%\xa1\x92\xb3\x853' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon PowerShot G5\x009\x1c\xed\x8f6\xac\xa2' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x44 off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x4a8ea6 len 0x44, offsetTbl @+0x20, 3 records - [0] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0xad6e2c6d - [1] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0x5cc19942 - [2] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.00\x00\x00\x98\xaa\xebP\xf9rhu\xd0' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x1f24 off 0x27a - heap root/ImageProps/ExifInformation: block @0x4a8f0e len 0x1f24, offsetTbl @+0x1e7e, 16 records - [0] tag 0x0001 id 0x0001 FreeBytes byte heap len 0xc off 0x0 - [1] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x400 off 0x0 - [2] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0xe60002 off 0xd7011e - [3] tag 0x102a id 0x102a CanonShotInfo word heap len 0x44 off 0xc - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x50 - [5] tag 0x0032 id 0x0032 ? byte heap len 0x800 off 0xac - [6] tag 0x0036 id 0x0036 ? byte heap len 0x1400 off 0x8ac - [7] tag 0x1030 id 0x1030 WhiteSample word heap len 0x76 off 0x1cac - [8] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x1290000 off 0x725bca88 - [9] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0x1d22 - = u16 (34, 2672, 1968, 1, 1, 24, 14, 2615, 1957, 2640, 14, 2643, 1957, 0, 0, 0, 0) - [10] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0x1d44 - = u32 (1, 0, 514, 4861804) - [11] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x38 off 0x1d54 - = u16 (9, 1, 2592, 1944, 2592, 215, 467, 39, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0) - [12] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x1d8c - = u16 (0, 0, 0, 0) - [13] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x1d94 - = u16 (10, 2, 2, 640, 480) - [14] tag 0x103c id 0x103c ? word heap len 0x9c off 0x1d9e - [15] tag 0x107f id 0x107f ? word heap len 0x44 off 0x1e3a - -== corpus/crw/crw-canon-powershot-g6.crw - size 8052032, byteOrder II, headerLength 26, type b'HEAP' subType b'CCDR', version 0x00010002 -heap root: block @0x1a len 0x7add26, offsetTbl @+0x7adcf8, 4 records - [0] tag 0x2005 id 0x2005 RawData struct heap len 0x6f9704 off 0x0 - RawData: 7313156 bytes, entropy 7.9774 bits/byte - first 2 KiB: entropy 6.684, top bytes 00:520, c1:14, 48:14, ae:13, zeros 520, ff 8 - middle 2 KiB: entropy 7.885, top bytes 12:18, 31:17, 93:16, 64:15, zeros 13, ff 5 - last 2 KiB: entropy 7.355, top bytes f7:42, 63:36, ef:35, 7e:35, zeros 14, ff 15 - [1] tag 0x2007 id 0x2007 JpgFromRaw struct heap len 0xae5f4 off 0x6f9704 - starts with JPEG SOI - [2] tag 0x2008 id 0x2008 ThumbnailImage struct heap len 0x17b8 off 0x7a7cf8 - starts with JPEG SOI - [3] tag 0x300a id 0x300a ImageProps heap2 heap len 0x4848 off 0x7a94b0 - heap root/ImageProps: block @0x7a94ca len 0x4848, offsetTbl @+0x47c0, 13 records - [0] tag 0x5803 id 0x1803 ImageFormat dword inrec len 0x20001 off 0x41200000 - [1] tag 0x1810 id 0x1810 ImageInfo dword heap len 0x1c off 0xc - = u32 (3072, 2304, 1065353216, 0, 8, 24, 257) - [2] tag 0x500a id 0x100a TargetImageType word inrec len 0xffff0000 off 0xffffffff - [3] tag 0x5804 id 0x1804 RecordID dword inrec len 0x0 off 0xffffffff - [4] tag 0x5817 id 0x1817 FileNumber dword inrec len 0x1257af off 0xffffffff - [5] tag 0x180e id 0x180e TimeStamp dword heap len 0xc off 0x0 - = u32 (1607955982, 0, 0) - [6] tag 0x0816 id 0x0816 OriginalFileName ascii heap len 0x20 off 0x28 - = b'CRW_2095.CRW\x00\n\\U\x88JJ\x04\x7f\xff' - [7] tag 0x0817 id 0x0817 ThumbnailFileName ascii heap len 0x20 off 0x48 - = b'CRW_2095.THM\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ' - [8] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x100 off 0x68 - [9] tag 0x3003 id 0x3003 MeasuredInfo heap2 heap len 0x10 off 0x168 - heap root/ImageProps/MeasuredInfo: block @0x7a9632 len 0x10, offsetTbl @+0x0, 1 records - [0] tag 0x5814 id 0x1814 MeasuredEV dword inrec len 0x0 off 0x0 - [10] tag 0x2804 id 0x2804 ImageDescription heap1 heap len 0x5a off 0x178 - heap root/ImageProps/ImageDescription: block @0x7a9642 len 0x5a, offsetTbl @+0x40, 2 records - [0] tag 0x0805 id 0x0805 CanonFileDescription ascii heap len 0x20 off 0x0 - = b'High definition CCD image' - [1] tag 0x0815 id 0x0815 CanonImageType ascii heap len 0x20 off 0x20 - = b'CRW:High definition CCD image' - [11] tag 0x2807 id 0x2807 CameraObject heap1 heap len 0xa8 off 0x1d2 - heap root/ImageProps/CameraObject: block @0x7a969c len 0xa8, offsetTbl @+0x84, 3 records - [0] tag 0x0810 id 0x0810 OwnerName ascii heap len 0x20 off 0x0 - = b'\x00\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08' - [1] tag 0x080a id 0x080a CanonRawMakeModel ascii heap len 0x20 off 0x20 - = b'Canon\x00Canon PowerShot G6' - [2] tag 0x3004 id 0x3004 CameraSpecification heap2 heap len 0x44 off 0x40 - heap root/ImageProps/CameraObject/CameraSpecification: block @0x7a96dc len 0x44, offsetTbl @+0x20, 3 records - [0] tag 0x501c id 0x101c BaseISO word inrec len 0x64 off 0x0 - [1] tag 0x480d id 0x080d ? ascii inrec len 0x415355 off 0x0 - [2] tag 0x080b id 0x080b CanonFirmwareVersion ascii heap len 0x20 off 0x0 - = b'Firmware Version 1.00' - [12] tag 0x300b id 0x300b ExifInformation heap2 heap len 0x4546 off 0x27a - heap root/ImageProps/ExifInformation: block @0x7a9744 len 0x4546, offsetTbl @+0x4482, 19 records - [0] tag 0x0001 id 0x0001 FreeBytes byte heap len 0xc off 0x0 - [1] tag 0x5028 id 0x1028 CanonFlashInfo word inrec len 0x0 off 0x0 - [2] tag 0x5029 id 0x1029 CanonFocalLength word inrec len 0x2960002 off 0xd7011e - [3] tag 0x102a id 0x102a CanonShotInfo word heap len 0x44 off 0xc - [4] tag 0x102d id 0x102d CanonCameraSettings word heap len 0x5c off 0x50 - [5] tag 0x0032 id 0x0032 ? byte heap len 0xc00 off 0xac - [6] tag 0x0041 id 0x0041 ? byte heap len 0x100 off 0xcac - [7] tag 0x0036 id 0x0036 ? byte heap len 0x2000 off 0xdac - [8] tag 0x1030 id 0x1030 WhiteSample word heap len 0x76 off 0x42ac - [9] tag 0x5834 id 0x1834 CanonModelID dword inrec len 0x1400000 off 0xffffffff - [10] tag 0x1031 id 0x1031 SensorInfo word heap len 0x22 off 0x4322 - = u16 (34, 3160, 2344, 1, 1, 64, 24, 3135, 2327, 36, 24, 39, 2327, 0, 0, 0, 0) - [11] tag 0x1835 id 0x1835 DecoderTable dword heap len 0x10 off 0x4344 - = u32 (0, 0, 514, 7313154) - [12] tag 0x1038 id 0x1038 CanonAFInfo word heap len 0x38 off 0x4354 - = u16 (9, 1, 3072, 2304, 3072, 230, 553, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0) - [13] tag 0x1039 id 0x1039 ? word heap len 0x8 off 0x438c - = u16 (0, 0, 0, 0) - [14] tag 0x10b5 id 0x10b5 RawJpgInfo word heap len 0xa off 0x4394 - = u16 (10, 2, 0, 3072, 2304) - [15] tag 0x0040 id 0x0040 ? byte heap len 0x100 off 0x2dac - [16] tag 0x003f id 0x003f ? byte heap len 0x1400 off 0x2eac - [17] tag 0x103c id 0x103c ? word heap len 0x9c off 0x439e - [18] tag 0x107f id 0x107f ? word heap len 0x48 off 0x443a - diff --git a/crates/crw/clean-room/logs/verification-sweep.txt b/crates/crw/clean-room/logs/verification-sweep.txt deleted file mode 100644 index 18992d2d..00000000 --- a/crates/crw/clean-room/logs/verification-sweep.txt +++ /dev/null @@ -1,22 +0,0 @@ -CRW_0476.crw: sel 1, 3152x2068: EXACT -CRW_0478.crw: sel 1, 3152x2068: EXACT -CRW_0486.crw: sel 0, 3152x2068: EXACT -CRW_0527.crw: sel 1, 3152x2068: EXACT -CRW_0683.crw: sel 1, 3152x2068: EXACT -CRW_0717.crw: sel 1, 3152x2068: EXACT -CRW_0765.crw: sel 1, 3152x2068: EXACT -CRW_0822.crw: sel 1, 3152x2068: EXACT -CRW_0879.crw: sel 1, 3152x2068: EXACT -CRW_1011.crw: sel 1, 3152x2068: EXACT -CRW_1153.crw: sel 0, 3152x2068: EXACT -crw-canon-eos-10d-a.crw: sel 0, 3152x2068: EXACT -crw-canon-eos-10d-b.crw: sel 2, 3152x2068: EXACT -crw-canon-eos-10d-c.crw: sel 2, 3152x2068: EXACT -crw-canon-eos-300d-a.crw: sel 0, 3152x2068: EXACT -crw-canon-eos-300d-b.crw: sel 2, 3152x2068: EXACT -crw-canon-eos-d30.crw: sel 1, 2224x1456: EXACT -crw-canon-eos-d60.crw: sel 0, 3152x2068: EXACT -crw-canon-eos-digital-rebel.crw: sel 0, 3152x2068: EXACT -crw-canon-powershot-g1.crw: sel 1, 2144x1560: EXACT -crw-canon-powershot-g5.crw: sel 1, 2672x1968: 2685618 mismatches, first at 0 (row 0, col 0) -crw-canon-powershot-g6.crw: sel 0, 3160x2344: EXACT diff --git a/crates/crw/clean-room/p1_payload.py b/crates/crw/clean-room/p1_payload.py deleted file mode 100644 index d8faf5ad..00000000 --- a/crates/crw/clean-room/p1_payload.py +++ /dev/null @@ -1,184 +0,0 @@ -"""P1 — RawData payload partition statistics (no bitstream decoding). - -Questions (plan 01-plan.md P1): - 1. Exact leading zero run of RawData per file. - 2. Does RawData split as [low-bits wh/4][compressed L][pad] (prefix hypothesis) or - [compressed L][low-bits wh/4][pad] (suffix hypothesis) on EOS bodies? Tested by - row-structure statistics of both candidate regions (2 bits/pixel, rawW/4 bytes/row): - masked sensor rows should show as structured/near-constant rows, the entropy-coded - stream should not align to 788-byte rows. - 3. The constant third DecoderTable u32 (514) and the 2-3 byte remainder. - 4. Byte-stuffing check in the compressed region (ff-digram statistics). - -Prints statistics only; no sample values of image content beyond aggregate counts. - -Usage: python -I p1_payload.py file.crw [more...] -""" - -import math -import struct -import sys -from collections import Counter - - -def read_block_records(buf, start, length, endian): - if length < 6: - return [] - (tbl,) = struct.unpack_from(endian + "I", buf, start + length - 4) - if tbl + 2 > length - 4: - return [] - (n,) = struct.unpack_from(endian + "H", buf, start + tbl) - recs = [] - pos = start + tbl + 2 - if tbl + 2 + n * 10 > length: - return [] - for _ in range(n): - tc, ln, off = struct.unpack_from(endian + "HII", buf, pos) - pos += 10 - recs.append((tc, ln, off)) - return recs - - -def find_records(buf, start, length, endian, depth=0): - """Yield (tag_id, abs_offset, length) for every in-heap record, recursively.""" - if depth > 8: - return - for tc, ln, off in read_block_records(buf, start, length, endian): - stg = (tc >> 14) & 3 - dt = (tc >> 11) & 7 - tag = tc & 0x3FFF - if stg != 0: - yield (tag, None, (ln, off)) # in-entry value - continue - if off + ln > length: - continue - if dt in (5, 6): - yield from find_records(buf, start + off, ln, endian, depth + 1) - else: - yield (tag, start + off, ln) - - -def entropy(data): - if not data: - return 0.0 - c = Counter(data) - n = len(data) - return -sum(v / n * math.log2(v / n) for v in c.values()) - - -def zero_run(data): - i = 0 - while i < len(data) and data[i] == 0: - i += 1 - return i - - -def row_structure(region, row_bytes, crop_top, label, out): - """Reshape region into rows of row_bytes; report zero/constant-row runs and - per-band mean of 2-bit values (both bit orders give the same mean-of-byte/4 scale, - so use byte means as the aggregate).""" - nrows = len(region) // row_bytes - if nrows == 0: - out.append(f" {label}: shorter than one row") - return - zrows = 0 - for r in range(nrows): - row = region[r * row_bytes : (r + 1) * row_bytes] - if any(row): - break - zrows += 1 - bands = [] - for name, lo, hi in (("rows 0..top", 0, min(crop_top, nrows)), - ("top..top+32", min(crop_top, nrows), min(crop_top + 32, nrows)), - ("middle 32", nrows // 2, min(nrows // 2 + 32, nrows)), - ("last 32", max(0, nrows - 32), nrows)): - seg = region[lo * row_bytes : hi * row_bytes] - if seg: - bands.append(f"{name}: mean {sum(seg) / len(seg):7.2f} ent {entropy(seg):5.3f}") - out.append(f" {label}: {nrows} rows of {row_bytes} B, leading all-zero rows: {zrows}") - for b in bands: - out.append(f" {b}") - - -def ff_digrams(data, out): - n = len(data) - c = Counter(data) - ffs = c.get(0xFF, 0) - ff00 = sum(1 for i in range(n - 1) if data[i] == 0xFF and data[i + 1] == 0) - exp = ffs * c.get(0, 0) / max(1, n) # expected if independent - out.append(f" stuffing check: ff count {ffs} ({ffs / max(1, n):.4%}), " - f"ff00 digrams {ff00}, expected-if-independent {exp:.1f}") - - -def analyse(fname, show_bytes): - out = [f"== {fname}"] - buf = open(fname, "rb").read() - endian = {b"II": "<", b"MM": ">"}.get(buf[0:2]) - (hlen,) = struct.unpack_from(endian + "I", buf, 2) - recs = dict() - raw = None - for tag, off, ln in find_records(buf, hlen, len(buf) - hlen, endian): - if tag == 0x2005: - raw = (off, ln) - recs.setdefault(tag, (off, ln)) - if raw is None or raw[0] is None: - out.append(" !! no RawData record") - return out - roff, rlen = raw - data = buf[roff : roff + rlen] - # SensorInfo - soff, sln = recs.get(0x1031, (None, 0)) - w = h = top = 0 - if soff: - vals = struct.unpack_from(endian + "H" * (sln // 2), buf, soff) - w, h, top = vals[1], vals[2], vals[6] - # DecoderTable - doff, dln = recs.get(0x1835, (None, 0)) - dt = struct.unpack_from(endian + "IIII", buf, doff) if doff and dln >= 16 else (None,) * 4 - n_tbl, d1, d2, comp_len = dt - low = (w * h) // 4 - rem = rlen - (comp_len or 0) - out.append(f" raw {w}x{h} S={rlen} tbl={n_tbl} f1={d1} f2={d2} L={comp_len}" - f" S-L={rem} wh/4={low} S-L-wh/4={rem - low}") - out.append(f" leading zero run: {zero_run(data)} bytes" - + (f" (= {zero_run(data) / (w / 4):.2f} rows of 2-bit LSBs)" if w else "")) - if show_bytes: - out.append(f" last 8 bytes: {data[-8:].hex()}") - has_low = rem - low in (0, 1, 2, 3, 4) and low > 4 * (w or 1) - if has_low and w: - out.append(" prefix hypothesis — region [0 : wh/4) as 2-bit rows:") - row_structure(data[:low], w // 4, top, "prefix", out) - out.append(" suffix hypothesis — region [L : L+wh/4) as 2-bit rows:") - row_structure(data[comp_len : comp_len + low], w // 4, top, "suffix", out) - out.append(" compressed-candidate stats:") - for name, seg in (("after-prefix [wh/4 : wh/4+L)", data[low : low + comp_len]), - ("from-0 [0 : L)", data[:comp_len])): - out.append(f" {name}: entropy {entropy(seg[:1 << 20]):.4f} (first MiB), " - f"{entropy(seg[-(1 << 20):]):.4f} (last MiB)") - ff_digrams(data[low : low + comp_len], out) - else: - out.append(" no wh/4 region (PowerShot class): compressed assumed [0 : L)") - out.append(f" entropy first MiB {entropy(data[:1 << 20]):.4f}, " - f"last MiB {entropy(data[max(0, comp_len - (1 << 20)) : comp_len]):.4f}") - ff_digrams(data[:comp_len], out) - out.append(f" tail region [L : S): {rem} bytes" - + (f", hex {data[comp_len:rlen].hex()}" if show_bytes and rem <= 8 else "")) - return out - - -def main(): - args = sys.argv[1:] - show = True - if args and args[0] == "--no-bytes": - show = False - args = args[1:] - for fname in args: - try: - print("\n".join(analyse(fname, show))) - except Exception as e: - print(f"== {fname}\n !! exception: {e!r}") - print() - - -if __name__ == "__main__": - main() diff --git a/crates/crw/clean-room/p1b_markers.py b/crates/crw/clean-room/p1b_markers.py deleted file mode 100644 index 65d9137e..00000000 --- a/crates/crw/clean-room/p1b_markers.py +++ /dev/null @@ -1,119 +0,0 @@ -"""P1b — ff-successor and JPEG-marker statistics of the RawData payload regions. - -Follow-up to p1_payload.py: on EOS bodies every 0xff in the compressed region was followed -by 0x00 (JPEG-style stuffing); on the G5 the payload starts with exactly 514 zero bytes -(= the constant DecoderTable field) and ends in ff d9 (JPEG EOI). This script measures, -per file: the histogram of bytes following 0xff in the compressed region, any JPEG markers -(ff xx, xx not 00), and — for public files — hex of small structural windows (region -starts/ends, bytes around offset 514). Still no entropy decoding. - -Usage: python -I p1b_markers.py [--no-bytes] file.crw ... -""" - -import struct -import sys -from collections import Counter - - -def find_records(buf, start, length, endian, depth=0): - """Yield (tag_id, abs_offset, length) for in-heap records, recursively - (same container model as ciff_dump.py; duplicated so the script runs under python -I).""" - if depth > 8 or length < 6: - return - (tbl,) = struct.unpack_from(endian + "I", buf, start + length - 4) - if tbl + 2 > length - 4: - return - (n,) = struct.unpack_from(endian + "H", buf, start + tbl) - if tbl + 2 + n * 10 > length: - return - for k in range(n): - tc, ln, off = struct.unpack_from(endian + "HII", buf, start + tbl + 2 + k * 10) - stg = (tc >> 14) & 3 - dt = (tc >> 11) & 7 - tag = tc & 0x3FFF - if stg != 0: - yield (tag, None, (ln, off)) - continue - if off + ln > length: - continue - if dt in (5, 6): - yield from find_records(buf, start + off, ln, endian, depth + 1) - else: - yield (tag, start + off, ln) - - -def analyse(fname, show): - out = [f"== {fname}"] - buf = open(fname, "rb").read() - endian = {b"II": "<", b"MM": ">"}.get(buf[0:2]) - (hlen,) = struct.unpack_from(endian + "I", buf, 2) - recs = {} - for tag, off, ln in find_records(buf, hlen, len(buf) - hlen, endian): - if off is not None: - recs.setdefault(tag, (off, ln)) - roff, rlen = recs[0x2005] - data = buf[roff : roff + rlen] - soff, sln = recs.get(0x1031, (None, 0)) - vals = struct.unpack_from(endian + "H" * (sln // 2), buf, soff) if soff else () - w, h = (vals[1], vals[2]) if len(vals) > 2 else (0, 0) - doff, _ = recs.get(0x1835, (None, 0)) - n_tbl, _, f2, comp_len = struct.unpack_from(endian + "IIII", buf, doff) - low = w * h // 4 - has_low = 0 <= rlen - comp_len - low <= 4 - comp = data[low : low + comp_len] if has_low else data[:comp_len] - out.append(f" raw {w}x{h} tbl={n_tbl} f2={f2} L={comp_len} lowbits={'yes' if has_low else 'no'}") - - succ = Counter() - markers = Counter() - first_markers = [] - i = 0 - while True: - i = comp.find(0xFF, i) - if i < 0 or i + 1 >= len(comp): - break - nxt = comp[i + 1] - succ[nxt] += 1 - if nxt != 0: - markers[nxt] += 1 - if len(first_markers) < 12: - first_markers.append((i, nxt)) - i += 2 - else: - i += 2 - top = ", ".join(f"{b:02x}:{c}" for b, c in succ.most_common(8)) - out.append(f" ff followed by: {top}") - if markers: - mk = ", ".join(f"ff{b:02x}:{c}" for b, c in sorted(markers.items())) - out.append(f" markers: {mk}") - out.append(f" first markers at: {[(hex(p), hex(m)) for p, m in first_markers]}") - rst = [p for p, m in first_markers if 0xD0 <= m <= 0xD7] - if len(rst) > 2: - out.append(f" first restart gaps: {[rst[k + 1] - rst[k] for k in range(len(rst) - 1)]}") - else: - out.append(" markers: none (pure stuffed entropy data)") - if show: - out.append(f" comp[0:24] : {comp[:24].hex()}") - out.append(f" comp[-8:] : {comp[-8:].hex()}") - out.append(f" raw[504:544] : {data[504:544].hex()}") - out.append(f" raw[-8:] : {data[-8:].hex()}") - if has_low: - out.append(f" low[-8:] : {data[low - 8 : low].hex()} (end of low-bits region)") - return out - - -def main(): - args = sys.argv[1:] - show = True - if args and args[0] == "--no-bytes": - show = False - args = args[1:] - for fname in args: - try: - print("\n".join(analyse(fname, show))) - except Exception as e: - print(f"== {fname}\n !! exception: {e!r}") - print() - - -if __name__ == "__main__": - main() diff --git a/crates/crw/clean-room/p1c_lowbits_bounds.py b/crates/crw/clean-room/p1c_lowbits_bounds.py deleted file mode 100644 index 8ad04db5..00000000 --- a/crates/crw/clean-room/p1c_lowbits_bounds.py +++ /dev/null @@ -1,96 +0,0 @@ -"""P1c — verify the low-bits region bounds via non-stuffed ff statistics. - -On EOS bodies the entropy region contains no ff followed by non-00 at all (perfect -JPEG-style stuffing); on the 12-bit PowerShots (G5/G6) a region of unstuffed uniform -bytes exists. This script shows that every non-stuffed ff falls inside -[514, 514 + wh/4) of the RawData payload, establishing the layout -[514 zeros][low bits wh/4][stuffed entropy][terminator] for those bodies, and -reports the zero run + stuffing purity for every file class. No entropy decoding. - -Usage: python -I p1c_lowbits_bounds.py file.crw ... -""" - -import struct -import sys - - -def records(buf, start, length, endian, depth=0): - if depth > 8 or length < 6: - return - (tbl,) = struct.unpack_from(endian + "I", buf, start + length - 4) - if tbl + 2 > length - 4: - return - (n,) = struct.unpack_from(endian + "H", buf, start + tbl) - if tbl + 2 + n * 10 > length: - return - for k in range(n): - tc, ln, off = struct.unpack_from(endian + "HII", buf, start + tbl + 2 + k * 10) - if (tc >> 14) & 3: - continue - if off + ln > length: - continue - if ((tc >> 11) & 7) in (5, 6): - yield from records(buf, start + off, ln, endian, depth + 1) - else: - yield (tc & 0x3FFF, start + off, ln) - - -def analyse(fname): - buf = open(fname, "rb").read() - endian = {b"II": "<", b"MM": ">"}.get(buf[0:2]) - (hl,) = struct.unpack_from(endian + "I", buf, 2) - r = {} - for t, o, l in records(buf, hl, len(buf) - hl, endian): - r.setdefault(t, (o, l)) - ro, rl = r[0x2005] - so, sl = r[0x1031] - v = struct.unpack_from(endian + "H" * (sl // 2), buf, so) - w, h = v[1], v[2] - do, _ = r[0x1835] - n_tbl, _, f2, comp_len = struct.unpack_from(endian + "IIII", buf, do) - data = buf[ro : ro + rl] - low = w * h // 4 - eos = 0 <= rl - comp_len - low <= 4 - - # positions of non-stuffed ff (ff followed by non-00), excluding the terminator - pos = [] - i = 0 - while True: - i = data.find(0xFF, i) - if i < 0 or i + 1 >= len(data): - break - if data[i + 1] != 0: - pos.append(i) - i += 2 - pos = [p for p in pos if p < rl - 4] - - comp_start = low if eos else 0 - zr = 0 - while comp_start + zr < rl and data[comp_start + zr] == 0: - zr += 1 - name = fname.replace("\\", "/").rsplit("/", 1)[-1] - if eos: - inside = sum(1 for p in pos if p < low) - print(f"{name:36} EOS layout: lowbits [0:{low}) + zeros({zr}) + entropy; " - f"non-stuffed ff: {len(pos)} total, {inside} in lowbits, " - f"{len(pos) - inside} in entropy (expect 0)") - else: - lo, hi = 514, 514 + low - inside = sum(1 for p in pos if lo <= p < hi) - if pos and inside == len(pos): - print(f"{name:36} PS 12-bit layout: zeros({zr}) + lowbits [{lo}:{hi}) + entropy; " - f"all {len(pos)} non-stuffed ff inside lowbits; " - f"first {pos[0]}, last {pos[-1]}") - elif not pos: - print(f"{name:36} PS 10-bit layout: zeros({zr}) + entropy; " - f"no non-stuffed ff (pure stuffed stream)") - else: - print(f"{name:36} PS ?: {len(pos)} non-stuffed ff, {inside} inside " - f"[{lo}:{hi}); first {pos[0]}, last {pos[-1]} — hypothesis fails") - - -for fn in sys.argv[1:]: - try: - analyse(fn) - except Exception as e: - print(f"{fn}: !! {e!r}") diff --git a/crates/crw/clean-room/p2_oracle_lsb.py b/crates/crw/clean-room/p2_oracle_lsb.py deleted file mode 100644 index e0e9dcfe..00000000 --- a/crates/crw/clean-room/p2_oracle_lsb.py +++ /dev/null @@ -1,118 +0,0 @@ -"""P2 — the decisive low-bits test against the oracle (LibRaw unprocessed_raw, black box). - -Input: a CRW file and the TIFF written by `unprocessed_raw.exe -T` for it. -Tests, per 03-payload-layout.md: - - oracle frame size == SensorInfo raw size; - - oracle code histogram (min/max/percentiles) -> bit depth; - - the low-bits region equals the 2 LSBs of the oracle's codes, row-major, 4 pixels/byte, - under one of the two packing orders (pixel0 in bits 7-6 ... pixel3 in bits 1-0, or the - reverse); reports exact matching byte counts for both orders and both candidate - positions of the region (EOS prefix [0:Q) / PowerShot [514:514+Q)). - -Statistics and equality counts only. Usage: - python -I p2_oracle_lsb.py file.crw file.crw.tiff -""" - -import struct -import sys - - -def records(buf, start, length, endian, depth=0): - if depth > 8 or length < 6: - return - (tbl,) = struct.unpack_from(endian + "I", buf, start + length - 4) - if tbl + 2 > length - 4: - return - (n,) = struct.unpack_from(endian + "H", buf, start + tbl) - if tbl + 2 + n * 10 > length: - return - for k in range(n): - tc, ln, off = struct.unpack_from(endian + "HII", buf, start + tbl + 2 + k * 10) - if (tc >> 14) & 3: - continue - if off + ln > length: - continue - if ((tc >> 11) & 7) in (5, 6): - yield from records(buf, start + off, ln, endian, depth + 1) - else: - yield (tc & 0x3FFF, start + off, ln) - - -def read_tiff_gray16(path): - """Minimal classic-TIFF reader: single image, 16-bit, strips, no compression.""" - buf = open(path, "rb").read() - endian = {b"II": "<", b"MM": ">"}[buf[0:2]] - (magic, ifdoff) = struct.unpack_from(endian + "HI", buf, 2) - assert magic == 42 - (n,) = struct.unpack_from(endian + "H", buf, ifdoff) - tags = {} - for i in range(n): - tag, typ, cnt, val = struct.unpack_from(endian + "HHII", buf, ifdoff + 2 + i * 12) - tags[tag] = (typ, cnt, val, ifdoff + 2 + i * 12 + 8) - def val(tag): - typ, cnt, v, pos = tags[tag] - size = {1: 1, 3: 2, 4: 4}[typ] - if size * cnt <= 4: - if typ == 3 and cnt == 1: - return [struct.unpack_from(endian + "H", buf, pos)[0]] - return [v] - fmt = {1: "B", 3: "H", 4: "I"}[typ] - return list(struct.unpack_from(endian + fmt * cnt, buf, v)) - w = val(256)[0] - h = val(257)[0] - bps = val(258)[0] - comp = val(259)[0] if 259 in tags else 1 - assert bps == 16 and comp == 1, (bps, comp) - offs = val(273) - counts = val(279) - data = b"".join(buf[o : o + c] for o, c in zip(offs, counts)) - assert len(data) >= w * h * 2, (len(data), w * h * 2) - import array - a = array.array("H") - a.frombytes(data[: w * h * 2]) - if (endian == ">") != (sys.byteorder == "big"): - a.byteswap() - return w, h, a - - -def main(): - crw_path, tiff_path = sys.argv[1], sys.argv[2] - buf = open(crw_path, "rb").read() - endian = {b"II": "<", b"MM": ">"}[buf[0:2]] - (hl,) = struct.unpack_from(endian + "I", buf, 2) - r = {} - for t, o, l in records(buf, hl, len(buf) - hl, endian): - r.setdefault(t, (o, l)) - ro, rl = r[0x2005] - so, sl = r[0x1031] - v = struct.unpack_from(endian + "H" * (sl // 2), buf, so) - w, h = v[1], v[2] - q = w * h // 4 - data = buf[ro : ro + rl] - - tw, th, pix = read_tiff_gray16(tiff_path) - print(f"CRW raw {w}x{h}; oracle TIFF {tw}x{th} -> {'MATCH' if (tw, th) == (w, h) else 'MISMATCH'}") - s = sorted(pix[::997]) - print(f"oracle codes: min {min(pix)}, max {max(pix)}, p1 {s[len(s)//100]}, " - f"median {s[len(s)//2]}, p99 {s[-len(s)//100]}") - - # build expected low-bits bytes from oracle LSBs, both packing orders - import array - lsb = array.array("B", (p & 3 for p in pix)) - exp_msb = bytearray(q) - exp_lsb = bytearray(q) - for i in range(q): - a4 = lsb[4 * i : 4 * i + 4] - exp_msb[i] = (a4[0] << 6) | (a4[1] << 4) | (a4[2] << 2) | a4[3] - exp_lsb[i] = a4[0] | (a4[1] << 2) | (a4[2] << 4) | (a4[3] << 6) - for region_name, reg in (("prefix [0:Q)", data[:q]), ("ps [514:514+Q)", data[514 : 514 + q])): - if len(reg) < q: - continue - m1 = sum(1 for a, b in zip(reg, exp_msb) if a == b) - m2 = sum(1 for a, b in zip(reg, exp_lsb) if a == b) - print(f"{region_name}: match MSB-first {m1}/{q} ({m1 / q:.6%}), " - f"LSB-first {m2}/{q} ({m2 / q:.6%})") - - -if __name__ == "__main__": - main() diff --git a/crates/crw/clean-room/sources.md b/crates/crw/clean-room/sources.md index 44933c95..32b7e0a2 100644 --- a/crates/crw/clean-room/sources.md +++ b/crates/crw/clean-room/sources.md @@ -48,7 +48,7 @@ checked out, diffed, logged or read. - 2026-10-12 19:28 — **Search** (WebSearch, standard): `Canon CIFF specification 1.0 rev 4 PDF "CIFF" camera image file format specification download`. Results included Wikipedia "Camera Image File Format", a freedesktop libopenraw wiki page "Canon CRW" (NOT opened — it is - a decoder project's page; left for the maintainer to approve, see plan §7), an unpkg mirror of + a decoder project's page; left for the maintainer to approve), an unpkg mirror of exiftool's canon_raw.html, PRONOM. Taken: pointer to Wikipedia for an archived spec URL. **No decoder algorithm details shown in the result snippets.** - 2026-10-12 19:28 — **Fetch attempt** `http://xyrion.org/ciff/` — failed (site dead). @@ -86,22 +86,23 @@ checked out, diffed, logged or read. different format. **No decoder algorithm details shown.** - 2026-10-12 19:32 — **Download** (Python urllib, stdlib) the 11 CC0 files above from `https://raw.pixls.us/getfile.php/...` into the gitignored `corpus/crw/`; every file verified - against the site's published SHA-256 (all OK; table with hashes in `01-plan.md`). + against the site's published SHA-256 (all OK; the hashes are pinned in + `crates/crw/tests/samples.json`). ## Local analysis (scripts in this directory) - 2026-10-12 19:35–19:45 — `ciff_dump.py` (written here, stdlib only; container layout - implemented from the two documented sources above) run over the 11 public files - (`logs/structure-public.txt`) and the 18 private files - (`logs/private/structure-private.txt`, gitignored — may contain personal metadata). Observed - facts are summarised in `01-plan.md` §3. No raw-data decoding was attempted; statistics only. -- 2026-10-12 ~20:0x — P0/P1 local analysis (post-approval): `02-container.md` written from the + implemented from the two documented sources above) run over the 11 public files and the 18 + private files (whose dumps may contain personal metadata and are never committed). Observed + facts are summarised in `01-payload-layout.md`. No raw-data decoding was attempted; statistics + only. The script and its logs are not part of this record. +- 2026-10-12 ~20:0x — P0/P1 local analysis (post-approval): the container notes written from the already-logged sources + corpus observations. `p1_payload.py`, `p1b_markers.py`, - `p1c_lowbits_bounds.py` run over all 29 files (public logs `logs/p1*`, private under - `logs/private/`). Established statistically (no decoding): the three payload layouts, the + `p1c_lowbits_bounds.py` run over all 29 files. Established statistically (no decoding): the + three payload layouts, the 514-byte zero block at the compressed section start of every file, JPEG-style ff00 stuffing of every entropy stream (zero exceptions), the PowerShot `ff d9` terminator, and the exact - low-bits region bounds on G5/G6. Documented in `03-payload-layout.md`. No external sources + low-bits region bounds on G5/G6. Documented in `01-payload-layout.md`. No external sources consulted in this step. ## Oracle (P2, from 2026-10-12) @@ -110,7 +111,8 @@ checked out, diffed, logged or read. a local `LibRaw-0.22.2\bin` folder. That directory tree also contains LibRaw's source code; **no source file there is ever opened, read or searched** — only `bin\unprocessed_raw.exe` is executed, as a black box, per the approved plan (§5a/§7). - All oracle runs and probe campaigns are logged in `04-derivation-log.md`. + All oracle runs and probe campaigns are logged in the derivation log, which is not part of this + record. - 2026-10-13 — **Fetch** `https://raw.pixls.us/json/getrepository.php?set=all` (and the site root to find that endpoint): recovered the stable `getfile.php` URLs of the 11 CC0 corpus @@ -130,5 +132,5 @@ checked out, diffed, logged or read. - 2026-10-25 — **Fetch** `https://raw.githubusercontent.com/colour-science/colour/v0.4.2/colour/characterisation/datasets/colour_checkers/chromaticity_coordinates.py` (colour-science, BSD-3-Clause; also its rendered docs page): the 24 post-November-2014 ColorChecker Classic L*a*b* values (D50), credited there to X-Rite's release. Reference - data only, the chart's measured colours; no camera or decoder content. Used in - `10-chart-calibration.md`. + data only, the chart's measured colours; no camera or decoder content. Used for the EOS D60 + colour matrices. diff --git a/crates/crw/clean-room/tools/gen_tables.py b/crates/crw/clean-room/tools/gen_tables.py index 666f0b9e..dc8c0ca4 100644 --- a/crates/crw/clean-room/tools/gen_tables.py +++ b/crates/crw/clean-room/tools/gen_tables.py @@ -1,7 +1,7 @@ """Regenerate crates/crw/src/tables.rs from the frozen tables in crates/crw/clean-room/logs/table_sel{0,1,2}.json. Run from anywhere: `python crates/crw/clean-room/tools/gen_tables.py`. The JSON files are the -derivation's frozen data (06-crw-format.md section 3); this script only renders them as Rust +derivation's frozen data (02-crw-format.md section 3); this script only renders them as Rust constants, keeping the file's doc comment. It was a local tool during the derivation in LightCraft (plan/crw/model/gen_tables.py) and moved here so the tables stay reproducible. """ diff --git a/crates/crw/src/lib.rs b/crates/crw/src/lib.rs index f09cd20b..ce6cc633 100644 --- a/crates/crw/src/lib.rs +++ b/crates/crw/src/lib.rs @@ -5,8 +5,8 @@ //! The container rules come from Canon's published CIFF specification and ExifTool's prose tag //! documentation; the compression was derived from CC0 and maintainer-owned sample files by //! black-box analysis against a reference decoder's command-line output, frozen in -//! `clean-room/06-crw-format.md` and verified sample-exact on 28 of 29 corpus files plus 1,516 -//! further EOS D60 files (`clean-room/07-verification.md`). The code tables are frozen data +//! `clean-room/02-crw-format.md` and verified sample-exact on 28 of 29 corpus files plus 1,516 +//! further EOS D60 files (`clean-room/03-verification.md`). The code tables are frozen data //! (`tables`, generated from `clean-room/logs/table_sel{0,1,2}.json`). No other raw decoder's //! source was consulted; the whole derivation record is in the crate's `clean-room/` folder. //! @@ -326,7 +326,7 @@ fn parse(bytes: &[u8], limits: &Limits, full: bool) -> Result { .unwrap_or_default(); if model.contains("PowerShot G5") { return Err(Error::Unsupported( - "Canon PowerShot G5 entropy coding (a documented anomaly of the derivation, clean-room/06-crw-format.md §4); use the embedded JPEG" + "Canon PowerShot G5 entropy coding (a documented anomaly of the derivation, clean-room/02-crw-format.md §4); use the embedded JPEG" .into(), )); } @@ -622,7 +622,7 @@ fn build_lut(codes: &[(u16, u8, u8, u8)]) -> Vec { } /// Decode the ff00-stuffed entropy stream into the `w × h` high-plane samples, per the frozen -/// model (`clean-room/06-crw-format.md` §2): 64-sample blocks, first/main code tables, +/// model (`clean-room/02-crw-format.md` §2): 64-sample blocks, first/main code tables, /// (run, size) leaves with JPEG-rule magnitudes, EOB, block-DC carry, and two parity DPCM chains /// initialised to 512 and reset at every row start. A stream that ends early leaves the remaining /// samples at their chain values (the frozen stream-end rule). diff --git a/crates/crw/src/tables.rs b/crates/crw/src/tables.rs index 60ebb9f9..027329a5 100644 --- a/crates/crw/src/tables.rs +++ b/crates/crw/src/tables.rs @@ -2,7 +2,7 @@ //! //! Generated by `clean-room/tools/gen_tables.py` from `clean-room/logs/table_sel{0,1,2}.json` (the //! tables extracted by known-plaintext alignment and frozen at the CRW-FREEZE commit; derivation: -//! `clean-room/06-crw-format.md` §3). Do not edit by hand; regenerate from the JSON. +//! `clean-room/02-crw-format.md` §3). Do not edit by hand; regenerate from the JSON. //! Each entry is `(code bits, code length, run, size)`; codes are MSB-first. /// One selector's pair of canonical Huffman tables: the code starting each 64-sample diff --git a/crates/crw/tests/samples.rs b/crates/crw/tests/samples.rs index 71828438..1f729f02 100644 --- a/crates/crw/tests/samples.rs +++ b/crates/crw/tests/samples.rs @@ -1,6 +1,6 @@ //! The CC0 sample files (raw.pixls.us) pinned in `samples.json`: each decodes to exactly the //! sensor samples the derivation verified against the black-box reference (`clean-room/ -//! 07-verification.md`), recorded as the SHA-256 of the little-endian sample buffer, with the +//! 03-verification.md`), recorded as the SHA-256 of the little-endian sample buffer, with the //! levels, white balance and metadata the decoder read then. //! //! The files are not in the repository (5–11 MB each): `python scripts/fetch-crw-samples.py