craft-libs/docs/clean-room.md

# Clean-room format work in craft-libs

Several crates here read or write formats that other open-source projects also implement, often
under licences the Craft apps cannot take code from (GPL, LGPL, AGPL, CDDL) or under terms that
forbid it (Adobe's). This document is the standard a format crate follows so that every line of
it is ours, and so that a reviewer can check that claim from the repository alone. It is the
written form of what `craft-jpeg`, `craft-media` and `craft-crw` did; the suite-wide rules live in
[craftrules](https://github.com/storytold/craftrules) (`standards/licensing.md`,
`standards/never-crash.md`), which this does not replace.

## What "clean-room" means here

A clean-room crate is written only from:

- **published specifications and standards** (ITU-T, ISO/IEC, IETF RFCs, a vendor's own published
  format document, a registry's pages), read as documents;
- **prose documentation** of a format or its tags (ExifTool's tag-name and format pages, a wiki
  page that describes a container), never a project's code;
- **sample files** we may use: CC0, or owned by a contributor who consents, or produced by us.
  Not NC-licensed files, not files whose licence is unclear, never anything from inside a
  proprietary application's bundle;
- **a reference implementation run as a black box**: an official pre-built binary of another
  decoder, fed inputs and observed for outputs, with every run logged. Its source is never
  opened, read, searched, diffed or quoted, including comments, licence headers, mirrors and
  excerpts in forum posts or AI answers.

Never consulted, in any form: GPL/LGPL/AGPL/CDDL decoder and encoder sources (dcraw, LibRaw,
rawspeed, rawloader/rawler, dnglab, darktable, RawTherapee, ART, libopenraw, lensfun, ExifTool's
Perl, FFmpeg/libav, x264/x265 and the like), Adobe's SDKs, profiles (DCP, LCP), presets, fonts,
icons or binaries, and any earlier attempt of ours that was itself not clean.

## The process (what the record must show)

Modelled on LightCraft's ARW compression derivation and repeated for CRW
(`crates/crw/clean-room/`). A crate that derives an undocumented format commits a `clean-room/`
folder with:

1. **Starting assumptions**, written before any source is consulted, every item marked
   unverified, committed once and never edited. A reviewer diffs them against the result to see
   what was assumed and what was established.
2. **A plan** naming the sources that will be used, the sample corpus (public files with URL,
   licence and SHA-256; private files by name, size and hash only, never content), the phases,
   the freeze point, and the questions that are the maintainer's to decide. Work past the plan
   waits for the maintainer's recorded approval.
3. **A source log**: every URL opened and every search run, in order, with the time, what was
   asked and what was taken. If a page exposed implementation details that were not asked for,
   the log says so and what it said.
4. **Derivation documents** in which every fact cites the experiment (script and log) or the
   document that established it, and corrections are stated, not silently applied. Statistics
   before decoding; hypotheses tested against internal consistency first; black-box probes
   logged one by one.
5. **A freeze**: one document holding the complete derived format, with its tables as data. Its
   commit is the freeze point. Nothing in it is tuned afterwards. A mismatch found later reopens
   the derivation *explicitly*: the cause, the change and a new freeze revision are recorded in
   that document.
6. **Verification** after the freeze, with zero tuning: every sample compared exactly against the
   reference, every file's verdict recorded, failures recorded as failures. Cross-body checks
   (tables learned on one body, exact on another) are the strongest independent evidence.
7. **The implementation** from the frozen document, with: never-crash rules (`get()` for every
   file-derived offset, checked arithmetic, bounded recursion and allocation, no `unwrap`/`panic`),
   synthetic tests that need no sample files (a test-only encoder that inverts the model is the
   usual tool), hostile-input tests, and a sample-file test pinning the verified decode (a digest
   of the output per file) so the crate can prove it still matches without redistributing the
   references.
8. **Table data** lives in the record as JSON and is rendered into Rust by a committed script; the
   generated file says so and is never edited by hand.

When the derivation was done in another repository, the crate's `clean-room/` may hold only items
3, 5, 6 and 8, provided it pins that repository, path and revision (in its README and in `NOTICE`)
for the starting assumptions, the plan and the derivation documents, as `craft-crw` does.

Scripts in the record use the Python standard library only and run under `python -I`. Local-only
material (reference models, oracle binaries, copyrighted specification PDFs, dumps that may carry
personal metadata) is named in the record with how to obtain it, and is gitignored.

## Assets and attribution

Sample files are never committed (sizes aside, they are other people's photographs): tests
download them outside the checkout by pinned URL and SHA-256, licence checked (`scripts/
fetch-*.py`), and skip cleanly without them. Each manifest, each table derived from samples and
each log set has a row in `ATTRIBUTION.md`; code and records that came from an app are listed in
`NOTICE` under *Source provenance* with the repository, paths and revision.

## When something is not clean

Stop, record what was seen and where (the source log), and tell the maintainer. Code written
afterwards by the same person for the same part of the format is suspect; the maintainer decides
whether the part is re-derived by someone else, documented as derived from that source under its
licence terms, or dropped. Never quietly continue.