unescape_rbsp pushed every byte into the output Vec one at a time. It now
scans for 00 00 03 with the usual skip rule (a nonzero byte at i rules out
a match ending at i + 1 or i + 2, so the scan advances by 3) and copies the
bytes between matches with extend_from_slice. No unsafe, no indexing that
can panic.
Output is identical: a removed 03 still restarts the zero count. Tests
compare it with a verbatim copy of the old function over 20000 random
buffers (fixed-seed PRNG) and list the edge cases (empty input, 00 00 03 at
the end, 00 00 03 03, runs of zeros).
Closes#518
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add craft-doc: read PhotoCraft, VectorCraft and EffectCraft documents without the apps
A new crate, craft-doc, reads the native files of three apps from bytes:
PhotoCraft .pcraft (a ZIP bundle), VectorCraft .vectorcraft (JSON, plain
or gzip, with the version 4 data after the JSON) and EffectCraft .ecproj
(JSON).
- sniff tells which app wrote a file from its first bytes.
- info gives the format version, the writer the file names, and whether
the version is one the crate reads.
- renditions returns VectorCraft's preview and PDF and PhotoCraft's
thumbnail and composite, when the file has them.
- outline returns PhotoCraft's canvas and layer tree, VectorCraft's
artboards and top-level layers, and EffectCraft's compositions with
their layers.
A file whose version is newer than the crate reads is refused with
Error::TooNew. Limits bound the JSON, each entry, the total bytes
decompressed and the JSON nesting, and each is checked before the bytes
it covers are allocated. ZIP entries are read by a small STORE and
DEFLATE reader in the crate. The crate depends on serde, serde_json,
thiserror and flate2 and builds for wasm32-unknown-unknown.
The tests build every document they read: JSON written by hand, PNG,
ZIP and gzip bytes assembled in the test. doc.yml runs fmt, clippy, the
tests and a wasm32 check.
* ci: do not keep the checkout token in the doc workflow
The checkout step sets persist-credentials: false, as jpeg.yml and media.yml do. No later step in the job uses the token.
* craft-doc: read each format's envelope through its format crate
craft-doc no longer keeps its own copies of the envelope types and
constants. It reads them with craft-pcraft-format (`MANIFEST`,
`FORMAT_VERSION`, `THUMB`, `COMPOSITE`, `check_manifest`),
craft-vectorcraft-format (`Head`, `Image`, `sniff`, `check`, `VERSION`,
`BLOB_MAGIC`, base64) and craft-ecproj-format (`Head::read`,
`SCHEMA_VERSION`). Its own base64 module is removed. A `.pcraft`
manifest is still read in one pass without building a
`serde_json::Value`, with the checks `read_manifest` applies. The
outlines are still read with craft-doc's read-only views of each
document, and the public API (`sniff`, `info`, `renditions`, `outline`,
`Limits`) does not change.
These inputs are now read as the format crates read them:
- A `.pcraft` manifest without `generator`, or with a `thumbnail` other
than `thumb.png` or a `composite` other than `composite/preview.png`,
is malformed for `renditions` and `outline`; `info` still reports its
version. A manifest with more than 4,194,304 JSON items is refused
with `LimitExceeded`.
- A `.vectorcraft` preview or PDF member without `mime` gives no
rendition.
- `sniff` takes `format` only from a top-level member, so
`{"meta":{"format":"vectorcraft"}}` is not a VectorCraft file.
tests/envelopes.rs writes a file of each format with the crates' types
and reads it through every call, and checks the inputs above.
doc.yml also runs when one of the three crates changes.
* craft-doc: sniff reads only top-level members of an EffectCraft project
`sniff` took any JSON object whose first member is `savedBy` for an
EffectCraft project when `"schema"` appeared anywhere in its first
bytes, so `{"savedBy":"1.0","metadata":{"schema":1}}` was claimed. It
now checks the top-level member names only, one at a time, without
keeping them.
* craft-doc: bound list elements with max_items; no rendition without a MIME type
Limits gains max_items (default 1,048,576): EffectCraft project items and
composition layers, VectorCraft artboards and layers, and PhotoCraft layers
at every level are counted before each one is read, and a file that lists
more gives Error::LimitExceeded. A list of empty objects within max_json no
longer expands into gigabytes of layers.
A VectorCraft preview or pdf member with an empty mime gives no rendition, as
one without a mime does; a preview may be PNG or JPEG. An EffectCraft
composition without a duration or a frame rate gets no timing instead of a
duration of 0.
* craft-media: shared codec identifiers, one ByteSource, the testkit
The types a codec crate needs to describe a file and configure a
decoder, ahead of craft-audio:
- `Container`, `CodecId`, `PcmFormat` and `CodecParams` (`id.rs`),
always built, with optional serde.
- `codec_params()` on `isobmff::Track` and `matroska::Track`, both
returning `CodecParams` in the same form; `extradata` follows
Matroska's `CodecPrivate`.
- One `ByteSource` trait (`source.rs`), with `ReadSeekSource`;
`isobmff::ByteSource` and `matroska::ByteSource` are that trait.
- craft-media-testkit (`publish = false`): the ffmpeg oracle helpers
from craft-media's tests, so the codec crates share one copy.
The existing tests prove the refactor; the new types have their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* craft-media: APV extradata is the bare record; reseek after a poisoned lock
An MP4 apvC track gave CodecParams.extradata with the 4-byte FullBox header
in front, where Matroska's V_APV CodecPrivate is the bare
APVDecoderConfigurationRecord. ApvConfig::record_bytes serializes the record
alone and extradata uses it.
ReadSeekSource kept its tracked position after recovering a poisoned lock or
a failed seek, so a reader moved by a panic was read from the wrong offset
with no error. Both now force a seek on the next read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* craft-media: MP4 codec params per sample description
Track::codec_params always described the first stsd entry, while each sample
selects its own through description_index. It now describes the first
sample's entry; codec_params_for gives the parameters of any entry, and
has_mixed_descriptions says whether one set covers every sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* craft-media: test file-backed ByteSources; a File read past i64::MAX is an EOF
A File and a ReadSeekSource over a file now have the buffer's test: the same
bytes at the same offsets, and UnexpectedEof past the end. The test found
that a File read at an offset past i64::MAX failed with the OS's
InvalidInput instead of the UnexpectedEof the trait promises; it now
reports the end of the source like the other implementations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* craft-media: Matroska codec params match MP4 for ALAC, DNx and Opus
Matroska's codec_params gave Unknown for A_ALAC and for DNxHD/DNxHR stored
as V_MS/VFW/FOURCC (AVdn, AVdh), where MP4 gives Alac and Dnxhd; ALAC's
extradata is the CodecPrivate, the ALACSpecificConfig MP4's alac box holds.
Opus now reports 48 kHz from Matroska as from MP4, whatever rate the muxer
declared.
ReadSeekSource clears the poison when it recovers the lock, so one
panicked read costs one recovery instead of a seek on every later read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* craft-media: MP4 reports the MPEG audio layer the first frame names
An MP4 track with MPEG audio reported CodecId::Mp3 whatever the layer, since
mp4a + 0x69/0x6B names none (ffmpeg stores MP2 that way), while Matroska
reports Mp1/Mp2/Mp3 from its CodecID. open() now reads the frame header of
each such entry's first sample and codec_params() reports the layer it
names; without a valid header, QuickTime's .mp2 gives Mp2 and anything else
Mp3 as before. CodecConfig::Mp3 is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Add craft-pcraft-format: the envelope of PhotoCraft's .pcraft bundle
The manifest around the document, with the document as a type parameter
(Manifest<D>), TileRef, the bundle's entry paths and hash check,
FORMAT_VERSION, the manifest decode limits with the nesting and item
scan, and the migration runner as a generic step list. Messages and
serialized field order are PhotoCraft's, so its saved manifests stay
byte-identical when it switches to these types.
Part of #59.
* ci: do not keep the checkout token in the pcraft-format workflow
The checkout step sets persist-credentials: false, as jpeg.yml and media.yml do. No later step in the job uses the token.
* pcraft-format: build no entry name from an invalid hash
tile_path and blob_path return None unless the hash is 64 lowercase hex
digits, so a hash read from a file such as "../x" never becomes an entry
name. TileRef::path gives a tile's entry the same way.
The docs note that MAX_GROUP_DEPTH is a copy of PhotoCraft's constant,
and that check_manifest also refuses a manifest larger than
max_manifest_bytes, which PhotoCraft's check before a save does not.
* ci: pin actions/checkout by commit and Rust to 1.95 in the pcraft-format workflow
* pcraft-format: read_manifest is the one reader of a manifest
read_manifest(json, limits, steps, target) checks the JSON against the
limits, refuses a format_version newer than target, runs the migration
steps, and reads the document as D and the other envelope members.
read_manifest_with reads the document with the caller's function
instead. Manifest no longer implements Deserialize.
Error::TooNew holds the manifest's generator, so a message can name the
writer; its message is unchanged. TooNew, Manifest and TileRef are
#[non_exhaustive], and Manifest::new and TileRef::new build them.
TILES_DIR, BLOBS_DIR, COMPOSITE_DIR and OBJECT_SUFFIX name the bundle's
directories and the object suffix. The docs of max_blob_bytes and
max_total_bytes say the caller enforces them.
* pcraft-format: read_manifest reports errors in serde's struct order
read_manifest reads the envelope members and the document in the order
the parsed object holds them, and reports a missing member after the
members that are present. A manifest with several faults gives the same
error as a derived Deserialize of the whole manifest, as PhotoCraft
reads it today.
* pcraft-format: document that a missing or invalid format_version is Corrupt
read_manifest's docs and the README say that a missing or invalid
format_version gives Error::Corrupt from migrate, and that a missing
generator or document, or any other mistyped member, gives Error::Json.
A test covers a string and a fractional format_version.
* pcraft-format: read_manifest refuses preview names other than its own
A manifest's `thumbnail` and `composite` named any entry the reader then
opened, while tile entries go through `is_valid_hash`. `read_manifest`
now gives `Error::Corrupt` for a `thumbnail` other than `thumb.png` or a
`composite` other than `composite/preview.png`, the only names PhotoCraft
writes. The check runs after the members are read, so JSON errors keep
their order.
The reader's doc now says a missing or invalid `format_version` is
`Error::Corrupt`, from `migrate`, not `Error::Json`.
* Add craft-vectorcraft-format: the .vectorcraft envelope, with the document as a type parameter
File<D>, Head and Image hold the members VectorCraft writes, in its
order, with its skip rules: serde_json output of a File<D> is the JSON
VectorCraft writes, compact or pretty. Tail places entries after the
JSON from version 4 (INLINE_MAX bytes or more, after BLOB_MAGIC), and
split_tail and Image::bytes read them back. The crate also has the
version and name constants, check() for the format name and newer
versions, sniff() for plain and gzip files (the top-level "format"
member must be vectorcraft or drawcraft), and standard base64.
No dependency on any app; builds for wasm32-unknown-unknown.
* ci: do not keep the checkout token in the vectorcraft-format workflow
The checkout step sets persist-credentials: false, as jpeg.yml and media.yml do. No later step in the job uses the token.
* vectorcraft-format: limit the bytes copied from the data after the JSON
Entries::bytes reads a file's entries and returns Error::TooBig once the
blob ranges it has copied add up to more than the data after the JSON
holds, so entries that name the same bytes again and again are refused,
as VectorCraft's reader refuses them.
A generator, preview or pdf of the wrong shape now reads as None in File
and Head, as VectorCraft opens such a file. File::tail gives a Tail for
the file's own version.
* ci: pin actions/checkout by commit and Rust to 1.95 in the vectorcraft-format workflow
* vectorcraft-format: document that base64 decode ignores the bits before the padding
decode reads a group such as Zh== like Zg==, and Zm9= like Zm8=, as
VectorCraft's own base64_decode does. The doc comment says so, and a
test pins both cases to the canonical bytes.
* vectorcraft-format: entries naming the same range share one buffer
VectorCraft's reader keeps the bytes of each [offset, length] it has
read and gives a repeated entry the same buffer without counting it
again (crates/format/src/unpack.rs). Entries::bytes refused such a
file once the repeats passed the size of the data after the JSON. It
now returns Arc<[u8]>, caches each range, and counts it once.
Overlapping ranges that differ are still copied and counted each time.
* vectorcraft-format: a refused entry does not count against the copy limit
Entries::bytes added a refused entry's length to the bytes copied, so a later entry that fit was refused too. It now counts an entry only once it is read.
* vectorcraft-format: Contents reads the document without keeping the preview and PDF
File<D> checks the shape of generator, preview and pdf by holding each
value in memory, and keeps the preview and PDF base64 it reads. Contents<D>
reads format, version, profiles, document and images, and skips the other
three as the JSON is read. It accepts and refuses the same files as File<D>.
* vectorcraft-format: Contents reads a repeated generator, preview or pdf
VectorCraft opens a file whose generator, preview or pdf member is repeated, since its reader does not declare those members, and refuses a repeated format, version, profiles, document or images. Contents now leaves the three members undeclared, so serde skips every copy of them, and still refuses the other repeated members. File and Head are unchanged.
* vectorcraft-format: sniff decodes JSON escapes in the format member
serde_json reads a file whose format key or value is written with escapes, such as \u0066ormat, as a VectorCraft file, and sniff compared the raw bytes and refused it. sniff now decodes the key and the value before comparing them.
* vectorcraft-format: File::read and Contents::read check the version
File::read and Contents::read are the readers of a file's JSON. Both
refuse another format or a version newer than VERSION after the JSON
reads, with ReadError::Envelope holding the Error (Format or TooNew), and
return ReadError::Json otherwise. File and Contents no longer implement
Deserialize, so a reader cannot skip the check.
Both read one implementation and treat repeated members the same way:
a repeated generator, preview or pdf is accepted, and File keeps the last
value, as serde_json reads a repeated key into a Value. A repeated
format, version, document, images or profiles is refused. A top-level
array holds format, version, document, images and profiles in that
order, as VectorCraft reads it; Contents' fields follow that order.
- Image::bytes returns None for an entry of more than MAX_ENTRY (1 GiB,
VectorCraft's limit on one entry of a compressed file), and
Image::bytes_at_most takes another limit. Entries keeps no per-entry
limit, as VectorCraft opens larger entries in uncompressed files.
- Entries::blob reads a range; Entries returns Arc<Vec<u8>>.
- marker_at finds BLOB_MAGIC where the writer puts it in a file's first
bytes; split_tail uses it.
- File, Contents, Head, Image, Part and ReadError are #[non_exhaustive].
* Add craft-ecproj-format: the envelope of EffectCraft project files
An .ecproj file is one JSON object: savedBy, then the saved view's
members, then the project's members, written with two-space indentation.
The crate defines that envelope with the view and the project as type
parameters, so the project model stays in EffectCraft:
- SCHEMA_VERSION (2) and check_schema, which refuses a newer schema
with EffectCraft's own message.
- File { saved_by, view, project }, serialized with both parts
flattened, and File::to_json.
- Head, which reads savedBy and schema from a JSON object, and
saved_by, which reads savedBy alone.
The tests write synthetic projects and compare the text with JSON
written by hand in the shape EffectCraft's writer gives. The workflow
runs fmt, clippy, the tests and a wasm32-unknown-unknown check.
* ci: do not keep the checkout token in the ecproj-format workflow
The checkout step sets persist-credentials: false, as jpeg.yml and media.yml do. No later step in the job uses the token.
* ecproj-format: write compact files, read savedBy as EffectCraft opens it
EffectCraft's Save Compact Project Files setting (effectcraft 07639b5f)
writes .ecproj files on one line with serde_json::to_string.
File::to_json_compact writes the same text, and the docs say the crate
writes and reads both forms.
Head now reads every file EffectCraft's Project::from_json opens: a
savedBy that is not a string, or appears more than once, reads as None,
and a project written as a JSON array reads with its first element as
the schema. saved_by also gives None for a repeated savedBy, as
EffectCraft's saved_by does. File's docs say the crate does not check
values and that serde_json writes NaN and infinity as null.
* ecproj-format: skip a non-string savedBy without keeping it
Head::read and saved_by read savedBy into a serde_json::Value before
dropping any value that is not a string, so a wide or deeply nested
savedBy was held in memory, and one nested past serde_json's recursion
limit made Head::read fail. A non-string savedBy, and every repeated
savedBy, is now skipped element by element with IgnoredAny, as other
members are. It still reads as None.
* ecproj-format: Head::read refuses a newer schema and reads objects only
Head::read is the one reader of a project file's envelope. It returns
ReadError: Json when the text is not one JSON object with a u32 schema
(the array form included), or TooNew when the schema is newer than
SCHEMA_VERSION, holding the file's savedBy so a message can name the
version that wrote it. TooNew reads as before. saved_by() is removed,
and Head no longer implements Deserialize.
TooNew, ReadError and ReadError::TooNew are #[non_exhaustive].
The total flag-set bound (MAX_SET_DATA, 8.4M) stays the hostile-input guard: LibreOffice's
tr_TR loads with 5.2M of it in 0.6 s, and abart, abartmıştım, savaşıyor, inandım, aranıyor,
açılamadı and bilinmiyor are right, as hunspell 1.7 answers. The merged-homonym cap is now
4 × the entry cap (32,768), under the 65,536 distinct flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
hunspell::candidates_exact (folder → exact-locale pack → system, own names only) for English without SCOWL; packs::exact_spelling_dirs. Other languages keep the region-ordered cross-region tiers. Drops has_british_source().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
- build.rs: unset CRAFT_PROOF_EMBED embeds de, pl, pt, it patterns and ru-extra as before (DEFAULT_EMBED); `en` embeds English and our supplements only; `de,pl` and `all` as before. Making `en` the default is CP4.4, in the release that publishes and pins language-packs-1. The default-build data tests cover pl/it/de/pt again; CI's extra step is now the English-only build.
- licence_texts: one Moby entry (the data's, else the built-in en-words.dic) and en-extra.txt (WordCraft contributors, MIT OR Apache-2.0); the deleted core-list notice is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
The goitre failure was not a race: GitHub's Ubuntu runners have a system en_US Hunspell dictionary, which en-GB uses without the Language data (by design), and it marks goitre. Reproduced locally with a fake system en_US; the test now checks British words only when en-GB reads the Moby list or a British dictionary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
P2:
- Every test whose English results depend on installed dictionaries takes DICTIONARY_TESTS (the en-GB goitre CI failure: tiny en_US fixtures raced unlocked en-GB tests).
- hunspell::candidates/find order: dictionaries folder, the first name's region pack (or the language alone), the system's exact-region dictionary, any other region's pack, the system's other names. pt_BR system beats a pt-pt pack; de_DE / de-ch, nl_NL / nl-be, es_ES / es-mx, en_US / en-gb likewise.
P3:
- Pack hyphenation lookups: a map, at most 256 languages (cleared when full); 3000 tags stay bounded and fast.
- A decomposed word (a + U+0301) is hyphenated composed (NFC) and the breaks mapped back; combining marks stay in their word.
- licence_texts checks each file's size against what is left before reading it; never reads past 16 MB.
- German, Bulgarian and pack languages: spelling message in the language where we have it; missing_dictionary/unreadable_dictionary report them like the others.
- build.rs: a missing file no longer says nothing is embedded; ru ignore message says CRAFT_I18N_DIR is enough. Built-in English list notice names the whole Moby list (en-words.dic).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Tables from the Unicode Consortium's 8859-7.TXT (2003, with € ₯ ͺ) and
8859-15.TXT mapping files; unassigned bytes become U+FFFD. These are the
only manifest SETs craft-proof couldn't decode. el now loads and passes its
check words; et loads and next stops at COMPOUNDFLAG (compounding,
storytold/wordcraft#2070).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Recasing a suggestion to capitals turned ß into SS for every dictionary,
so without CHECKSHARPS "STRAßEE" was offered "STRASSE", which check()
rejects. Now ß stays ß (STRAßE); under CHECKSHARPS the SS form is offered
first, and a KEEPCASE word with ß may be capitalized or written in capitals
with SS (FUSBALL → FUSSBALL, fußball; Fusball → Fußball; GRUS → GRUSS,
Gruß). Every expected list is the hunspell 1.7.5 CLI's on the same files
(sharp_s_suggestions_match_hunspell), and every suggestion passes check().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
- An OCONV-written suggestion (colour’s) is mapped to the run's English
spelling: the dialect lists are read with ', and the result keeps the
dictionary's apostrophe (color’s). Straight and curly duplicates (color's,
color’s) are one suggestion. Test: curly_suggestions_are_mapped_to_the_dialect.
- Conv::at looks up each prefix of the text (at most MAX_CONV_BYTES) in a
hash map instead of scanning the pairs under its first character: 65,536
pairs on one character cost 64 lookups per position, not a scan (Korean's
OCONV has 588 pairs on one character, so a per-character cap would not
fit). Test: a_huge_conversion_table_on_one_character_stays_fast.
- The reviewers' OCONV repros are a test (oconv_suggestions_match_hunspell):
OCONV applies after a suggestion's case is set, case-sensitively, as the
hunspell 1.7.5 CLI answers (KBT → KAT; Kbt → Kat under k → q).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
