SCOWL en-AU has 238 words en-GB lacks (billabong, brumby, larrikinism, Australian place
names) and spells program, inquiry, livable, licorice, verandah the Australian way (no
programme, enquiry, liveable, dialog or veranda, as Hunspell en_AU says); -ise/-ize policy and
affix file are the same as en-GB's. craft-i18n already ships language-data/en-au.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
* Add craft-vectorcraft-format: the .vectorcraft envelope, with the document as a type parameter
File<D>, Head and Image hold the members VectorCraft writes, in its
order, with its skip rules: serde_json output of a File<D> is the JSON
VectorCraft writes, compact or pretty. Tail places entries after the
JSON from version 4 (INLINE_MAX bytes or more, after BLOB_MAGIC), and
split_tail and Image::bytes read them back. The crate also has the
version and name constants, check() for the format name and newer
versions, sniff() for plain and gzip files (the top-level "format"
member must be vectorcraft or drawcraft), and standard base64.
No dependency on any app; builds for wasm32-unknown-unknown.
* ci: do not keep the checkout token in the vectorcraft-format workflow
The checkout step sets persist-credentials: false, as jpeg.yml and media.yml do. No later step in the job uses the token.
* vectorcraft-format: limit the bytes copied from the data after the JSON
Entries::bytes reads a file's entries and returns Error::TooBig once the
blob ranges it has copied add up to more than the data after the JSON
holds, so entries that name the same bytes again and again are refused,
as VectorCraft's reader refuses them.
A generator, preview or pdf of the wrong shape now reads as None in File
and Head, as VectorCraft opens such a file. File::tail gives a Tail for
the file's own version.
* ci: pin actions/checkout by commit and Rust to 1.95 in the vectorcraft-format workflow
* vectorcraft-format: document that base64 decode ignores the bits before the padding
decode reads a group such as Zh== like Zg==, and Zm9= like Zm8=, as
VectorCraft's own base64_decode does. The doc comment says so, and a
test pins both cases to the canonical bytes.
* vectorcraft-format: entries naming the same range share one buffer
VectorCraft's reader keeps the bytes of each [offset, length] it has
read and gives a repeated entry the same buffer without counting it
again (crates/format/src/unpack.rs). Entries::bytes refused such a
file once the repeats passed the size of the data after the JSON. It
now returns Arc<[u8]>, caches each range, and counts it once.
Overlapping ranges that differ are still copied and counted each time.
* vectorcraft-format: a refused entry does not count against the copy limit
Entries::bytes added a refused entry's length to the bytes copied, so a later entry that fit was refused too. It now counts an entry only once it is read.
* vectorcraft-format: Contents reads the document without keeping the preview and PDF
File<D> checks the shape of generator, preview and pdf by holding each
value in memory, and keeps the preview and PDF base64 it reads. Contents<D>
reads format, version, profiles, document and images, and skips the other
three as the JSON is read. It accepts and refuses the same files as File<D>.
* vectorcraft-format: Contents reads a repeated generator, preview or pdf
VectorCraft opens a file whose generator, preview or pdf member is repeated, since its reader does not declare those members, and refuses a repeated format, version, profiles, document or images. Contents now leaves the three members undeclared, so serde skips every copy of them, and still refuses the other repeated members. File and Head are unchanged.
* vectorcraft-format: sniff decodes JSON escapes in the format member
serde_json reads a file whose format key or value is written with escapes, such as \u0066ormat, as a VectorCraft file, and sniff compared the raw bytes and refused it. sniff now decodes the key and the value before comparing them.
* vectorcraft-format: File::read and Contents::read check the version
File::read and Contents::read are the readers of a file's JSON. Both
refuse another format or a version newer than VERSION after the JSON
reads, with ReadError::Envelope holding the Error (Format or TooNew), and
return ReadError::Json otherwise. File and Contents no longer implement
Deserialize, so a reader cannot skip the check.
Both read one implementation and treat repeated members the same way:
a repeated generator, preview or pdf is accepted, and File keeps the last
value, as serde_json reads a repeated key into a Value. A repeated
format, version, document, images or profiles is refused. A top-level
array holds format, version, document, images and profiles in that
order, as VectorCraft reads it; Contents' fields follow that order.
- Image::bytes returns None for an entry of more than MAX_ENTRY (1 GiB,
VectorCraft's limit on one entry of a compressed file), and
Image::bytes_at_most takes another limit. Entries keeps no per-entry
limit, as VectorCraft opens larger entries in uncompressed files.
- Entries::blob reads a range; Entries returns Arc<Vec<u8>>.
- marker_at finds BLOB_MAGIC where the writer puts it in a file's first
bytes; split_tail uses it.
- File, Contents, Head, Image, Part and ReadError are #[non_exhaustive].
* Add craft-ecproj-format: the envelope of EffectCraft project files
An .ecproj file is one JSON object: savedBy, then the saved view's
members, then the project's members, written with two-space indentation.
The crate defines that envelope with the view and the project as type
parameters, so the project model stays in EffectCraft:
- SCHEMA_VERSION (2) and check_schema, which refuses a newer schema
with EffectCraft's own message.
- File { saved_by, view, project }, serialized with both parts
flattened, and File::to_json.
- Head, which reads savedBy and schema from a JSON object, and
saved_by, which reads savedBy alone.
The tests write synthetic projects and compare the text with JSON
written by hand in the shape EffectCraft's writer gives. The workflow
runs fmt, clippy, the tests and a wasm32-unknown-unknown check.
* ci: do not keep the checkout token in the ecproj-format workflow
The checkout step sets persist-credentials: false, as jpeg.yml and media.yml do. No later step in the job uses the token.
* ecproj-format: write compact files, read savedBy as EffectCraft opens it
EffectCraft's Save Compact Project Files setting (effectcraft 07639b5f)
writes .ecproj files on one line with serde_json::to_string.
File::to_json_compact writes the same text, and the docs say the crate
writes and reads both forms.
Head now reads every file EffectCraft's Project::from_json opens: a
savedBy that is not a string, or appears more than once, reads as None,
and a project written as a JSON array reads with its first element as
the schema. saved_by also gives None for a repeated savedBy, as
EffectCraft's saved_by does. File's docs say the crate does not check
values and that serde_json writes NaN and infinity as null.
* ecproj-format: skip a non-string savedBy without keeping it
Head::read and saved_by read savedBy into a serde_json::Value before
dropping any value that is not a string, so a wide or deeply nested
savedBy was held in memory, and one nested past serde_json's recursion
limit made Head::read fail. A non-string savedBy, and every repeated
savedBy, is now skipped element by element with IgnoredAny, as other
members are. It still reads as None.
* ecproj-format: Head::read refuses a newer schema and reads objects only
Head::read is the one reader of a project file's envelope. It returns
ReadError: Json when the text is not one JSON object with a u32 schema
(the array form included), or TooNew when the schema is newer than
SCHEMA_VERSION, holding the file's savedBy so a message can name the
version that wrote it. TooNew reads as before. saved_by() is removed,
and Head no longer implements Deserialize.
TooNew, ReadError and ReadError::TooNew are #[non_exhaustive].
The total flag-set bound (MAX_SET_DATA, 8.4M) stays the hostile-input guard: LibreOffice's
tr_TR loads with 5.2M of it in 0.6 s, and abart, abartmıştım, savaşıyor, inandım, aranıyor,
açılamadı and bilinmiyor are right, as hunspell 1.7 answers. The merged-homonym cap is now
4 × the entry cap (32,768), under the 65,536 distinct flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
hunspell::candidates_exact (folder → exact-locale pack → system, own names only) for English without SCOWL; packs::exact_spelling_dirs. Other languages keep the region-ordered cross-region tiers. Drops has_british_source().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
- build.rs: unset CRAFT_PROOF_EMBED embeds de, pl, pt, it patterns and ru-extra as before (DEFAULT_EMBED); `en` embeds English and our supplements only; `de,pl` and `all` as before. Making `en` the default is CP4.4, in the release that publishes and pins language-packs-1. The default-build data tests cover pl/it/de/pt again; CI's extra step is now the English-only build.
- licence_texts: one Moby entry (the data's, else the built-in en-words.dic) and en-extra.txt (WordCraft contributors, MIT OR Apache-2.0); the deleted core-list notice is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
The goitre failure was not a race: GitHub's Ubuntu runners have a system en_US Hunspell dictionary, which en-GB uses without the Language data (by design), and it marks goitre. Reproduced locally with a fake system en_US; the test now checks British words only when en-GB reads the Moby list or a British dictionary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
P2:
- Every test whose English results depend on installed dictionaries takes DICTIONARY_TESTS (the en-GB goitre CI failure: tiny en_US fixtures raced unlocked en-GB tests).
- hunspell::candidates/find order: dictionaries folder, the first name's region pack (or the language alone), the system's exact-region dictionary, any other region's pack, the system's other names. pt_BR system beats a pt-pt pack; de_DE / de-ch, nl_NL / nl-be, es_ES / es-mx, en_US / en-gb likewise.
P3:
- Pack hyphenation lookups: a map, at most 256 languages (cleared when full); 3000 tags stay bounded and fast.
- A decomposed word (a + U+0301) is hyphenated composed (NFC) and the breaks mapped back; combining marks stay in their word.
- licence_texts checks each file's size against what is left before reading it; never reads past 16 MB.
- German, Bulgarian and pack languages: spelling message in the language where we have it; missing_dictionary/unreadable_dictionary report them like the others.
- build.rs: a missing file no longer says nothing is embedded; ru ignore message says CRAFT_I18N_DIR is enough. Built-in English list notice names the whole Moby list (en-words.dic).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Tables from the Unicode Consortium's 8859-7.TXT (2003, with € ₯ ͺ) and
8859-15.TXT mapping files; unassigned bytes become U+FFFD. These are the
only manifest SETs craft-proof couldn't decode. el now loads and passes its
check words; et loads and next stops at COMPOUNDFLAG (compounding,
storytold/wordcraft#2070).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Recasing a suggestion to capitals turned ß into SS for every dictionary,
so without CHECKSHARPS "STRAßEE" was offered "STRASSE", which check()
rejects. Now ß stays ß (STRAßE); under CHECKSHARPS the SS form is offered
first, and a KEEPCASE word with ß may be capitalized or written in capitals
with SS (FUSBALL → FUSSBALL, fußball; Fusball → Fußball; GRUS → GRUSS,
Gruß). Every expected list is the hunspell 1.7.5 CLI's on the same files
(sharp_s_suggestions_match_hunspell), and every suggestion passes check().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
- An OCONV-written suggestion (colour’s) is mapped to the run's English
spelling: the dialect lists are read with ', and the result keeps the
dictionary's apostrophe (color’s). Straight and curly duplicates (color's,
color’s) are one suggestion. Test: curly_suggestions_are_mapped_to_the_dialect.
- Conv::at looks up each prefix of the text (at most MAX_CONV_BYTES) in a
hash map instead of scanning the pairs under its first character: 65,536
pairs on one character cost 64 lookups per position, not a scan (Korean's
OCONV has 588 pairs on one character, so a per-character cap would not
fit). Test: a_huge_conversion_table_on_one_character_stays_fast.
- The reviewers' OCONV repros are a test (oconv_suggestions_match_hunspell):
OCONV applies after a suggestion's case is set, case-sensitively, as the
hunspell 1.7.5 CLI answers (KBT → KAT; Kbt → Kat under k → q).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Every case checked black-box against the hunspell 1.7.5 CLI on the
reviewers' repro files; checker and forms() changed together:
- NEEDAFFIX belongs to its homonym: with foo/NA and foo/B (or a plain foo),
foo is right.
- A prefix's flag may come from the suffix's continuation classes, with
CIRCUMFIX (PFX leg/X + SFX obb/AX: legnagyobb, legeslegnagyobb) or with a
NEEDAFFIX prefix (PFX leg/N + SFX obb/A: legnagyobb).
- The last suffix of a chain may carry NEEDAFFIX (met by the suffix under
it) or CIRCUMFIX (Hunspell ignores it there): work/A, able/B, s/N gives
workables. chained_forms_follow_needaffix_circumfix_and_fullstrip asserted
the opposite and is corrected.
- A prefix and a suffix that both need another affix make no word
(re/N + s/N: reworks is wrong).
Test needaffix_circumfix_and_fullstrip_match_hunspell holds all 32 repro
pairs with Hunspell's answer for each prefix/stem/suffix combination, and
checks that forms() lists exactly the right words. A circumfix prefix with
two suffixes (neworkables) is not supported: its test is #[ignore]d with
the reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
hunspell::candidates (which English now loads from) searches the pack tier between the dictionaries folder and the system, as find() does; pack-loaded English goes through the same reloads guard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
With CHECKSHARPS, a word in capitals may write `SS` for `ß` (`STRASSE`,
`MASSE`), and a KEEPCASE word with `ß` may be capitalized and written in
capitals with `SS` but not with `ß` (`Fußball`, `FUSSBALL`, not
`FUßBALL`). The first five `ss` pairs are tried in every combination.
`ß` counts as neither case when a word's casing is read, as in Hunspell,
so `STRAßE` is in capitals (correct with or without CHECKSHARPS). Every
answer in the test matches Hunspell 1.7's on the same files.
de, de-at and de-ch now stop at compounding (storytold/wordcraft#2070).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
48ed6f8's lenient English path ignored OCONV because craft-proof couldn't
read it. Now that it can (CP6.4), English reads it like every language:
suggestions are shown as the dictionary writes them (`don’t` with the
usual en_US/en_GB `OCONV ' ’`), as Hunspell shows them, and ICONV reads
them back. Compounding is still ignored for English.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
From the rebase onto craft-libs#483: Hunspell rejects a compound with a
NEEDAFFIX part (checked black-box with the hunspell CLI: foo/a + bar/bN under
COMPOUNDRULE ab, foobar is wrong), so compound() skips such stems as it does
FORBIDDENWORD ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
- OCONV (output conversion, as Hunspell 1.7 applies it): read from the
.aff and the compact format, applied to suggestions as they are shown
(`AffixDict::output`); what is correct doesn't change. The French
(Grammalecte) dictionary writes `’` for the `'` its stems use.
- ICONV and OCONV tables hold at most 65,536 pairs of at most 64 bytes a
side (the Korean tables hold 11,172); more is an error. Each table is
indexed by the first character of its `from` texts, so converting a word
looks at the few pairs that can start at each character instead of
scanning the whole table per character (review of storytold/wordcraft#5000).
- Input longer than MAX_INPUT_BYTES (4 × MAX_WORD_BYTES) is no word and
is no longer converted before the length check rejects it (review of
storytold/wordcraft#5000); input made long by IGNORE characters within
that bound still checks.
- Longest match wins, and of equal ones the later pair, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Random stems and words through a dictionary that uses all three
directives (whole-stem strips, empty forms, circumfix pairs, forms that
need another affix): no panic, and every form forms() lists is non-empty
and accepted by the checker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
forms() tested a prefix rule's condition on the stem only, then added
the prefix to every cross-product suffix form as well. contains() tests
the condition on what the prefix comes off: the stem for the prefix
alone, the suffix form in a cross product. So forms() could list a
cross product the checker rejects (q over an empty suffix form under
the condition x) and miss one it accepts.
A small helper, prefixed(), now adds a prefix the way the checker takes
it back off, for the stem and for each cross-product suffix form. The
tests cover a cross product that meets the condition and one that does
not. Over the whole hunspell-lv lv_LV dictionary the listed forms are
unchanged (4,515,511, all accepted).
Ported from storytold/wordcraft#5012 (8d84a49f5) onto craft-proof.
Adapted to flag ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
The previous commit skipped an empty suffix form altogether, so forms()
no longer listed what a continuation class or a prefix adds to it,
although contains() accepts it: with FULLSTRIP, x/A, SFX A x 0/B and
SFX B 0 y, contains("y") is true but forms("x") left "y" out.
An empty form is now only kept out of the list. It still goes into the
cross-product list and the chain queue, as before the previous commit.
The new test covers the chain case and checks that every form listed
for a prefix over an empty suffix form is accepted.
Ported from storytold/wordcraft#5012 (6fa95a695) onto craft-proof.
Adapted to flag ids.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
