mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-05 01:59:43 +00:00
RWoldbergandClaude Fable 5.1
a0c8a08020
The clean-room Canon CRW decoder written in LightCraft (crates/raw/src/vendor/crw.rs, crw_tables.rs, branch clean-room-crw-format-decode, revision 70c755e4) becomes the shared crate craft-crw: the CIFF heap container, the entropy coding derived by black-box analysis (frozen code tables for selectors 0-2), sensor black and white levels, as-shot white balance, orientation and the make/model/time records. Public types are the crate's own (Error, Limits, Cfa, Rect, BlackLevel, Header, Image); the capture time is handed on as Unix seconds so no calendar code lives here; LightCraft's shared plateau estimate is copied as saturation_plateau. Algorithm, tables and thresholds are unchanged. Tests: the ported synthetic round-trip, table-structure and hostile-input tests; a proptest mutation sweep (tests/robust.rs); and tests/samples.rs, which pins the eleven CC0 raw.pixls.us sample files (tests/samples.json: URL, SHA-256, licence) to the SHA-256 of the verified decode plus levels, white balance and metadata, so the port is proven byte-identical to the LightCraft decode without redistributing the reference TIFFs. scripts/fetch-crw-samples.py downloads the files outside the checkout; CRAFT_CRW_SAMPLES and CRAFT_CRW_ORACLE point the tests at them. The crate builds for wasm32. The whole derivation record (starting assumptions, plan with the maintainer's decisions, source log, container and payload notes, derivation log, frozen format, verification, colour/levels notes, private-corpus and chart notes, scripts, logs and the frozen table JSON) moves verbatim to crates/crw/clean-room/ with a README that indexes it and maps the LightCraft paths it cites; tools/gen_tables.py regenerates src/tables.rs from the JSON, byte for byte. Repository: .github/workflows/crw.yml (fmt, clippy, tests on Linux, Windows and macOS; wasm check; the sample-file job with cached downloads), README and ATTRIBUTION rows, NOTICE source provenance, docs/clean-room.md (the clean-room standard a format crate follows here), and AGENTS.md expanded with the start-of-session steps, the new-crate checklist, clean-room, assets and test rules; CLAUDE.md points at it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
86 lines
5.5 KiB
Markdown
86 lines
5.5 KiB
Markdown
# Clean-room format work in craft-libs
|
|
|
|
Several crates here read or write formats that other open-source projects also implement, often
|
|
under licences the Craft apps cannot take code from (GPL, LGPL, AGPL, CDDL) or under terms that
|
|
forbid it (Adobe's). This document is the standard a format crate follows so that every line of
|
|
it is ours, and so that a reviewer can check that claim from the repository alone. It is the
|
|
written form of what `craft-jpeg`, `craft-media` and `craft-crw` did; the suite-wide rules live in
|
|
[craftrules](https://github.com/storytold/craftrules) (`standards/licensing.md`,
|
|
`standards/never-crash.md`), which this does not replace.
|
|
|
|
## What "clean-room" means here
|
|
|
|
A clean-room crate is written only from:
|
|
|
|
- **published specifications and standards** (ITU-T, ISO/IEC, IETF RFCs, a vendor's own published
|
|
format document, a registry's pages), read as documents;
|
|
- **prose documentation** of a format or its tags (ExifTool's tag-name and format pages, a wiki
|
|
page that describes a container), never a project's code;
|
|
- **sample files** we may use: CC0, or owned by a contributor who consents, or produced by us.
|
|
Not NC-licensed files, not files whose licence is unclear, never anything from inside a
|
|
proprietary application's bundle;
|
|
- **a reference implementation run as a black box**: an official pre-built binary of another
|
|
decoder, fed inputs and observed for outputs, with every run logged. Its source is never
|
|
opened, read, searched, diffed or quoted, including comments, licence headers, mirrors and
|
|
excerpts in forum posts or AI answers.
|
|
|
|
Never consulted, in any form: GPL/LGPL/AGPL/CDDL decoder and encoder sources (dcraw, LibRaw,
|
|
rawspeed, rawloader/rawler, dnglab, darktable, RawTherapee, ART, libopenraw, lensfun, ExifTool's
|
|
Perl, FFmpeg/libav, x264/x265 and the like), Adobe's SDKs, profiles (DCP, LCP), presets, fonts,
|
|
icons or binaries, and any earlier attempt of ours that was itself not clean.
|
|
|
|
## The process (what the record must show)
|
|
|
|
Modelled on LightCraft's ARW compression derivation and repeated for CRW
|
|
(`crates/crw/clean-room/`). A crate that derives an undocumented format commits a `clean-room/`
|
|
folder with:
|
|
|
|
1. **Starting assumptions**, written before any source is consulted, every item marked
|
|
unverified, committed once and never edited. A reviewer diffs them against the result to see
|
|
what was assumed and what was established.
|
|
2. **A plan** naming the sources that will be used, the sample corpus (public files with URL,
|
|
licence and SHA-256; private files by name, size and hash only, never content), the phases,
|
|
the freeze point, and the questions that are the maintainer's to decide. Work past the plan
|
|
waits for the maintainer's recorded approval.
|
|
3. **A source log**: every URL opened and every search run, in order, with the time, what was
|
|
asked and what was taken. If a page exposed implementation details that were not asked for,
|
|
the log says so and what it said.
|
|
4. **Derivation documents** in which every fact cites the experiment (script and log) or the
|
|
document that established it, and corrections are stated, not silently applied. Statistics
|
|
before decoding; hypotheses tested against internal consistency first; black-box probes
|
|
logged one by one.
|
|
5. **A freeze**: one document holding the complete derived format, with its tables as data. Its
|
|
commit is the freeze point. Nothing in it is tuned afterwards. A mismatch found later reopens
|
|
the derivation *explicitly*: the cause, the change and a new freeze revision are recorded in
|
|
that document.
|
|
6. **Verification** after the freeze, with zero tuning: every sample compared exactly against the
|
|
reference, every file's verdict recorded, failures recorded as failures. Cross-body checks
|
|
(tables learned on one body, exact on another) are the strongest independent evidence.
|
|
7. **The implementation** from the frozen document, with: never-crash rules (`get()` for every
|
|
file-derived offset, checked arithmetic, bounded recursion and allocation, no `unwrap`/`panic`),
|
|
synthetic tests that need no sample files (a test-only encoder that inverts the model is the
|
|
usual tool), hostile-input tests, and a sample-file test pinning the verified decode (a digest
|
|
of the output per file) so the crate can prove it still matches without redistributing the
|
|
references.
|
|
8. **Table data** lives in the record as JSON and is rendered into Rust by a committed script; the
|
|
generated file says so and is never edited by hand.
|
|
|
|
Scripts in the record use the Python standard library only and run under `python -I`. Local-only
|
|
material (reference models, oracle binaries, copyrighted specification PDFs, dumps that may carry
|
|
personal metadata) is named in the record with how to obtain it, and is gitignored.
|
|
|
|
## Assets and attribution
|
|
|
|
Sample files are never committed (sizes aside, they are other people's photographs): tests
|
|
download them outside the checkout by pinned URL and SHA-256, licence checked (`scripts/
|
|
fetch-*.py`), and skip cleanly without them. Each manifest, each table derived from samples and
|
|
each log set has a row in `ATTRIBUTION.md`; code and records that came from an app are listed in
|
|
`NOTICE` under *Source provenance* with the repository, paths and revision.
|
|
|
|
## When something is not clean
|
|
|
|
Stop, record what was seen and where (the source log), and tell the maintainer. Code written
|
|
afterwards by the same person for the same part of the format is suspect; the maintainer decides
|
|
whether the part is re-derived by someone else, documented as derived from that source under its
|
|
licence terms, or dropped. Never quietly continue.
|