Commit Graph
973 Commits
Author SHA1 Message Date
Martin BallandClaude Opus 5.5 2dc51d7866 CP6.13: craft-libs#495 review — a .dic entry may carry up to 8,192 flags (was 1,024), so LibreOffice's Turkish stems with thousands (abart, savaş, inan: up to 3,926) are words again
The total flag-set bound (MAX_SET_DATA, 8.4M) stays the hostile-input guard: LibreOffice's
tr_TR loads with 5.2M of it in 0.6 s, and abart, abartmıştım, savaşıyor, inandım, aranıyor,
açılamadı and bilinmiyor are right, as hunspell 1.7 answers. The merged-homonym cap is now
4 × the entry cap (32,768), under the 65,536 distinct flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:57:01 +00:00
Martin BallandClaude Opus 5.5 f723abff6d CP6.13: tr hyphen — â, î and û are vowels (dük-kân, hi-kâ-ye); the module doc's examples are the ones the rules give (CodeRabbit on craft-libs#484)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 d1cebf89eb CP6.13: tr suggest — suggestions in capitals use Turkish capitals (KİTAP, not KITAP); three table entries that suggested the word itself are gone (CodeRabbit on craft-libs#484/#489)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 8d3a9366db CP6.13: tr grammar — emoticons, decimals and scores aren't stray spaces before punctuation; ordinals (5. sınıf) and file or web names (rapor.txt) don't end a sentence (CodeRabbit on craft-libs#484/#489)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 0b292e955c CP6.13: tr dict — Turkish is read by the shared affix engine: suffix conditions apply (kalelar is no word), homonyms keep each line's flags, and an entry ends before its morphological fields, as hunspell 1.7 reads them (CodeRabbit on craft-libs#484/#489)
TrDict keeps its API (from_hunspell, contains, is_empty) over AffixDict, which reads
FLAG num since #487; the hand-written one-suffix parser and TrSuffixRule are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 6057debe6e CP6.13: hunspell.rs — Windows drives are probed for LibreOffice only when its language folders are wanted (CodeRabbit on craft-libs#489)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 9614ab270d CP6.13: hyphen.rs — a Turkish word with a suffix after an apostrophe (Ankara'dan) is hyphenated by the Turkish rules, part by part, not as English (CodeRabbit on craft-libs#484/#489)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 76f9a55f01 CP6.13: it grammar — a sentence after the first starts with a capital: the rule reads the gap before the next word, not the rest of the paragraph (CodeRabbit on craft-libs#484/#489)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 8f16fe5a13 CP4.2: review — test dictionaries installed in memory hold the dictionary-tests lock until the test ends, so a pack or folder reload in another test can't drop them mid-test (the with-data ru suggestion flake)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 30f0f2a0e8 CP4.2: review — pack licences and licence file names are plain text; missing_dictionary documents that every language is named; past 256 pack names the hint names a generic dictionary
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 9d48d8145f CP4.2: review — the bare es and de packs are Spain's and Germany's; de-CH/de-LI and pt-PT read their own region's dictionary (sharing German's and Portuguese's hyphenation, grammar and messages); a region's text reads the system's exact region before the language's bare pack
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 f67dd7761e CP4.2: review — "Ignore words in UPPERCASE" skips only words with a cased letter, so a misspelt Hebrew or Arabic word is marked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 ba11ccd855 CP4.2: review — English never takes another dialect's dictionary or pack: each English reads its own names in the dictionaries folder, an exactly named pack and the system, else the built-in Moby list; the British-spelling test runs unconditionally again
hunspell::candidates_exact (folder → exact-locale pack → system, own names only) for English without SCOWL; packs::exact_spelling_dirs. Other languages keep the region-ordered cross-region tiers. Drops has_british_source().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 d905406fe3 CP4.2: review — the default build keeps the carry-over locales embedded until Language packs ship; CRAFT_PROOF_EMBED=en is the English-only build (CP4.4); licence notices name the real built-in sources
- build.rs: unset CRAFT_PROOF_EMBED embeds de, pl, pt, it patterns and ru-extra as before (DEFAULT_EMBED); `en` embeds English and our supplements only; `de,pl` and `all` as before. Making `en` the default is CP4.4, in the release that publishes and pins language-packs-1. The default-build data tests cover pl/it/de/pt again; CI's extra step is now the English-only build.
- licence_texts: one Moby entry (the data's, else the built-in en-words.dic) and en-extra.txt (WordCraft contributors, MIT OR Apache-2.0); the deleted core-list notice is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 3893192e88 CP4.2: review — en-GB British-spelling test skips a US-only Hunspell source (the CI goitre failure); German, Bulgarian and pack languages name their missing dictionary
The goitre failure was not a race: GitHub's Ubuntu runners have a system en_US Hunspell dictionary, which en-GB uses without the Language data (by design), and it marks goitre. Reproduced locally with a fake system en_US; the test now checks British words only when en-GB reads the Moby list or a British dictionary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 ee3be01b73 CP4.2: review — English tests hold the dictionaries lock; another region's pack never beats the system's exact region; bounded pack hyphenation lookups; NFC hyphenation; licences budgeted before reading; pack languages report like the others
P2:
- Every test whose English results depend on installed dictionaries takes DICTIONARY_TESTS (the en-GB goitre CI failure: tiny en_US fixtures raced unlocked en-GB tests).
- hunspell::candidates/find order: dictionaries folder, the first name's region pack (or the language alone), the system's exact-region dictionary, any other region's pack, the system's other names. pt_BR system beats a pt-pt pack; de_DE / de-ch, nl_NL / nl-be, es_ES / es-mx, en_US / en-gb likewise.

P3:
- Pack hyphenation lookups: a map, at most 256 languages (cleared when full); 3000 tags stay bounded and fast.
- A decomposed word (a + U+0301) is hyphenated composed (NFC) and the breaks mapped back; combining marks stay in their word.
- licence_texts checks each file's size against what is left before reading it; never reads past 16 MB.
- German, Bulgarian and pack languages: spelling message in the language where we have it; missing_dictionary/unreadable_dictionary report them like the others.
- build.rs: a missing file no longer says nothing is embedded; ru ignore message says CRAFT_I18N_DIR is enough. Built-in English list notice names the whole Moby list (en-words.dic).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 409c8facfd CP4.2: review — ProofLang::tag() names each pack language (its locale) for keys, settings and reported values; code() stays 'static and empty for packs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 664af9bc6f CP4.2: review — on 48ed6f8 English without data looks in the dictionaries folder, then an English Language pack, then the system's Hunspell, then the built-in list
hunspell::candidates (which English now loads from) searches the pack tier between the dictionaries folder and the system, as find() does; pack-loaded English goes through the same reloads guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 66d4dd3a5d CP4.2: review — pack names drop bidi/format characters, licence texts are capped in all, code() documented for Pack
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 867f44a3d8 CP4.2: review — pack hyphenation patterns only take words in the language's script; others fall through as before
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 930365bb6d CP4.2: review — a tag in its language's default script reads the script-less pack (sr-Cyrl → sr, bs-Latn → bs); scripts must be letters
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 e35f35ef76 CP4.2: review — our own *-extra.txt supplements (ru-extra) stay embedded in the default build
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 db341c5e15 CP4.2: review — pack dictionaries share a memory budget and a paragraph never evicts its own, so many languages load once
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 2495fad496 CP4.2: review — German and Bulgarian are spelled from their pack; pack languages are checked in any script
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 afbdede88e CP4.2: from_tag docs name ProofLang::Pack
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 3ad9ea765b CP4.2: ProofLang::Pack(PackLocale) proofs any language from its installed pack or Hunspell dictionary
Languages without tools of their own (nl, fr, sr-Latn…) are spell-checked and given suggestions
through the shared affix engine with default tuning, and hyphenated from their pack's patterns;
no grammar. At most 8 such dictionaries stay loaded. Never another script's pack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 576cf914e3 CP4.2/CP4.4: craft_proof::packs reads installed Language packs; only en-* embedded by default
- packs::set_packs_dir / reload / installed / licence_texts / load_errors / embedded_locales
- spelling lookup: dictionaries folder -> installed pack -> system Hunspell -> (English) core list
- hyphenation: installed pack patterns before the embedded ones, reloaded without a restart
- hostile packs: size caps, no symlinks, pack.toml path traversal rejected, errors reported
- build.rs embeds only en-* unless CRAFT_PROOF_EMBED names more (Language builds)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 16e4ed3115 CP6.1: decode SET ISO8859-7 and ISO8859-15 (Greek and Estonian dictionaries)
Tables from the Unicode Consortium's 8859-7.TXT (2003, with € ₯ ͺ) and
8859-15.TXT mapping files; unassigned bytes become U+FFFD. These are the
only manifest SETs craft-proof couldn't decode. el now loads and passes its
check words; et loads and next stops at COMPOUNDFLAG (compounding,
storytold/wordcraft#2070).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 bb859671cc CP6.12: craft-libs#490 review — ß stays ß when a suggestion is put in capitals
Recasing a suggestion to capitals turned ß into SS for every dictionary,
so without CHECKSHARPS "STRAßEE" was offered "STRASSE", which check()
rejects. Now ß stays ß (STRAßE); under CHECKSHARPS the SS form is offered
first, and a KEEPCASE word with ß may be capitalized or written in capitals
with SS (FUSBALL → FUSSBALL, fußball; Fusball → Fußball; GRUS → GRUSS,
Gruß). Every expected list is the hunspell 1.7.5 CLI's on the same files
(sharp_s_suggestions_match_hunspell), and every suggestion passes check().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 a46457cb7d CP6.5: CHECKSHARPS — SS for ß in German words in capitals
With CHECKSHARPS, a word in capitals may write `SS` for `ß` (`STRASSE`,
`MASSE`), and a KEEPCASE word with `ß` may be capitalized and written in
capitals with `SS` but not with `ß` (`Fußball`, `FUSSBALL`, not
`FUßBALL`). The first five `ss` pairs are tried in every combination.
`ß` counts as neither case when a word's casing is read, as in Hunspell,
so `STRAßE` is in capitals (correct with or without CHECKSHARPS). Every
answer in the test matches Hunspell 1.7's on the same files.

de, de-at and de-ch now stop at compounding (storytold/wordcraft#2070).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 de593dff4d CP6.11: craft-libs#489 review — dialect mapping after OCONV, bounded conversions
- An OCONV-written suggestion (colour’s) is mapped to the run's English
  spelling: the dialect lists are read with ', and the result keeps the
  dictionary's apostrophe (color’s). Straight and curly duplicates (color's,
  color’s) are one suggestion. Test: curly_suggestions_are_mapped_to_the_dialect.
- Conv::at looks up each prefix of the text (at most MAX_CONV_BYTES) in a
  hash map instead of scanning the pairs under its first character: 65,536
  pairs on one character cost 64 lookups per position, not a scan (Korean's
  OCONV has 588 pairs on one character, so a per-character cap would not
  fit). Test: a_huge_conversion_table_on_one_character_stays_fast.
- The reviewers' OCONV repros are a test (oconv_suggestions_match_hunspell):
  OCONV applies after a suggestion's case is set, case-sensitively, as the
  hunspell 1.7.5 CLI answers (KBT → KAT; Kbt → Kat under k → q).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 91d8bd335d CP6.7: English dictionaries' OCONV is applied to suggestions, not ignored
48ed6f8's lenient English path ignored OCONV because craft-proof couldn't
read it. Now that it can (CP6.4), English reads it like every language:
suggestions are shown as the dictionary writes them (`don’t` with the
usual en_US/en_GB `OCONV ' ’`), as Hunspell shows them, and ICONV reads
them back. Compounding is still ignored for English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 1301dc3961 CP6.4: OCONV output conversion; ICONV/OCONV tables capped and indexed; long input rejected before conversion
- OCONV (output conversion, as Hunspell 1.7 applies it): read from the
  .aff and the compact format, applied to suggestions as they are shown
  (`AffixDict::output`); what is correct doesn't change. The French
  (Grammalecte) dictionary writes `’` for the `'` its stems use.
- ICONV and OCONV tables hold at most 65,536 pairs of at most 64 bytes a
  side (the Korean tables hold 11,172); more is an error. Each table is
  indexed by the first character of its `from` texts, so converting a word
  looks at the few pairs that can start at each character instead of
  scanning the whole table per character (review of storytold/wordcraft#5000).
- Input longer than MAX_INPUT_BYTES (4 × MAX_WORD_BYTES) is no word and
  is no longer converted before the length check rejects it (review of
  storytold/wordcraft#5000); input made long by IGNORE characters within
  that bound still checks.
- Longest match wins, and of equal ones the later pair, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 07bc4672b0 CP6.10: craft-libs#488 review — NEEDAFFIX, CIRCUMFIX and chains as hunspell 1.7.5 reads them
Every case checked black-box against the hunspell 1.7.5 CLI on the
reviewers' repro files; checker and forms() changed together:
- NEEDAFFIX belongs to its homonym: with foo/NA and foo/B (or a plain foo),
  foo is right.
- A prefix's flag may come from the suffix's continuation classes, with
  CIRCUMFIX (PFX leg/X + SFX obb/AX: legnagyobb, legeslegnagyobb) or with a
  NEEDAFFIX prefix (PFX leg/N + SFX obb/A: legnagyobb).
- The last suffix of a chain may carry NEEDAFFIX (met by the suffix under
  it) or CIRCUMFIX (Hunspell ignores it there): work/A, able/B, s/N gives
  workables. chained_forms_follow_needaffix_circumfix_and_fullstrip asserted
  the opposite and is corrected.
- A prefix and a suffix that both need another affix make no word
  (re/N + s/N: reworks is wrong).

Test needaffix_circumfix_and_fullstrip_match_hunspell holds all 32 repro
pairs with Hunspell's answer for each prefix/stem/suffix combination, and
checks that forms() lists exactly the right words. A circumfix prefix with
two suffixes (neworkables) is not supported: its test is #[ignore]d with
the reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 fdcebf09f8 CP6.8: a NEEDAFFIX stem is no part of a COMPOUNDRULE compound
From the rebase onto craft-libs#483: Hunspell rejects a compound with a
NEEDAFFIX part (checked black-box with the hunspell CLI: foo/a + bar/bN under
COMPOUNDRULE ab, foobar is wrong), so compound() skips such stems as it does
FORBIDDENWORD ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 2c2dd12417 CP6.3: property test — FULLSTRIP, NEEDAFFIX and CIRCUMFIX never panic and forms() stays checkable
Random stems and words through a dictionary that uses all three
directives (whole-stem strips, empty forms, circumfix pairs, forms that
need another affix): no panic, and every form forms() lists is non-empty
and accepted by the checker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
janis luksandClaude Opus 5.5 d7bdd858bd Hunspell: forms() checks a prefix condition on what the prefix is added to
forms() tested a prefix rule's condition on the stem only, then added
the prefix to every cross-product suffix form as well. contains() tests
the condition on what the prefix comes off: the stem for the prefix
alone, the suffix form in a cross product. So forms() could list a
cross product the checker rejects (q over an empty suffix form under
the condition x) and miss one it accepts.

A small helper, prefixed(), now adds a prefix the way the checker takes
it back off, for the stem and for each cross-product suffix form. The
tests cover a cross product that meets the condition and one that does
not. Over the whole hunspell-lv lv_LV dictionary the listed forms are
unchanged (4,515,511, all accepted).

Ported from storytold/wordcraft#5012 (8d84a49f5) onto craft-proof.
Adapted to flag ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
janis luksandClaude Sonnet 5.5 42680a7e21 Hunspell: an empty suffix form still feeds chains and cross products
The previous commit skipped an empty suffix form altogether, so forms()
no longer listed what a continuation class or a prefix adds to it,
although contains() accepts it: with FULLSTRIP, x/A, SFX A x 0/B and
SFX B 0 y, contains("y") is true but forms("x") left "y" out.

An empty form is now only kept out of the list. It still goes into the
cross-product list and the chain queue, as before the previous commit.
The new test covers the chain case and checks that every form listed
for a prefix over an empty suffix form is accepted.

Ported from storytold/wordcraft#5012 (6fa95a695) onto craft-proof.
Adapted to flag ids.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
janis luksandClaude Sonnet 5.5 425ae605ca Hunspell: no empty forms, and a circumfix suffix inside a chain needs its prefix
Two more cases the forms() checks missed.

- With FULLSTRIP a suffix can strip a stem down to nothing, so forms()
  could list an empty form that contains() never accepts. Empty forms
  are no longer listed, at the first suffix, in a chain or with a prefix.
- A circumfix suffix at the start of a suffix chain was only checked
  for the outermost rule, so work/A, SFX A able/Bc, SFX B s was
  accepted as "workables" without the prefix. A circumfix suffix under
  another suffix is no longer a way to a word without its prefix, and
  forms() does not chain a form that ends in one.

New tests cover both. Over the whole hunspell-lv lv_LV dictionary
(66,480 stems, 4.5 million forms) every form forms() lists is still
accepted by the checker.

Ported from storytold/wordcraft#5012 (d655b3188) onto craft-proof.
Adapted to flag ids; the form_has_flag circumfix filter already came in with the first port.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
janis luksandClaude Sonnet 5.5 2bf2285577 Hunspell: count every cross-product suffix scanned against the forms() budget
The circumfix filter on the prefix cross-product loop of forms() only
counted work for suffix forms that matched the circumfix class of the
prefix. A dictionary with many circumfix prefixes and thousands of
ordinary cross-product suffixes could scan the incompatible ones
without ever reaching MAX_FORMS_WORK.

The loop now counts each suffix form it looks at, then skips the ones
of the wrong class, so the budget bounds the scan as it did before the
filter was added.

Ported from storytold/wordcraft#5012 (bc947e733) onto craft-proof.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
janis luksandClaude Sonnet 5.5 208de70f02 Hunspell: forms() chain loop follows NEEDAFFIX, CIRCUMFIX and FULLSTRIP
The first-level suffix loop of forms() skipped suffix rules that need
another affix or are circumfix ones, and rules that would replace a
whole stem without FULLSTRIP. The chain loop did not, so forms() could
list a chained form the checker rejects (work/A, SFX A able/B,
SFX B s/N with NEEDAFFIX N listed workables).

The chain loop now skips an empty base without FULLSTRIP and does not
list a form whose last suffix needs another affix or is a circumfix
one; such a form is still queued, so it can carry further suffixes.
The new test checks that every listed form is accepted for the three
cases.

Ported from storytold/wordcraft#5012 (2f3a83f6e) onto craft-proof.
Adapted to flag ids (cont_holds).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
janis luksandClaude Sonnet 5.5 a4edc0dce7 Hunspell: support FULLSTRIP, NEEDAFFIX and CIRCUMFIX
The affix engine refused any dictionary that uses these three
directives, so a dictionary such as Latvian lv_LV (hunspell-lv,
LGPL-2.1+) never loaded and its language showed "(dictionary
unreadable)".

- FULLSTRIP: an affix may strip, and so replace, a whole stem.
- NEEDAFFIX: a stem carrying the flag is not a word on its own, and a
  rule with it among its continuation classes gives forms that need one
  more affix (a prefix satisfies it for a suffix and the other way).
- CIRCUMFIX: a prefix or suffix carrying the flag is valid only
  together with one that carries it too; plain affixes never pair with
  circumfix ones.

forms() follows the same rules, so every form it lists is still
accepted by the checker, and the compact format keeps the three
directives. Tests cover each directive, the forms invariant and the
compact round trip.

Ported from storytold/wordcraft#5012 (989e942bc) onto craft-proof.
Adapted to craft-proof's flag ids (this branch sits on the FLAG
num/long port, storytold/wordcraft#5142): NEEDAFFIX and CIRCUMFIX are
flag ids read in the FLAG mode, continuation-class tests are set
lookups, and NEEDAFFIX joins the lookup exclusions. The ROADMAP.md part
and the Latvian language module (c99142ae9) are WordCraft's, not ported.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:32 +00:00
Martin BallandClaude Opus 5.5 280000e22e CP6.9: craft-libs#487 review — bounded lookups, linear homonym merge, Hunspell's reading of bad .dic flags, shared AF aliases
- Lookup cost (P2): form_has_flag tests each distinct continuation set
  once per lookup, against a bitmap when many flags are looked for, so a
  check no longer costs rules × set size. 21,024 rules with 1,024-flag AF
  continuations: 75 s for 40 checks before, well under a second now
  (big_continuation_sets_on_many_rules_stay_fast). Distinct flag sets are
  capped at 8 Mi flags in all, which bounds a lookup.
- Homonym merge (P2): a homonym that adds no flag doesn't re-sort the
  merged set (many_repeated_homonyms_load_fast).
- Malformed .dic flags read as Hunspell 1.7 reads them (black-box, the
  hunspell CLI on the same files), keeping the word: under FLAG num an item
  that is no number from 0 to 65,535 (empty, x2, -1, 70000) is flag 0, and
  000001 is 1; under FLAG long an odd last character is dropped. So
  Nepali's निजामती/I15,22 keeps flag 22. Continuation classes stay strict:
  Hunspell drops such a rule.
- AF: aliases are read once, in build() and in the final FLAG mode; the
  stems using one share its flags instead of copying them. At most 65,536
  aliases and 4 MB of alias text.
- to_bytes returns Result: data over from_bytes' 64 MB inflate limit is
  refused when written.
- ONLYINCOMPOUND sits after COMPOUNDMIN in the compact format again, so
  one-character data is byte-identical. It is also written without compound
  rules, so its stems stay compound-only after a round trip.
- Documented: a FLAG line that changes the mode after flags were read is
  still refused (Hunspell loads it), and long flags are read as characters
  (Hunspell reads bytes). The comment now says three exclusion flags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Martin BallandClaude Opus 5.5 f90ebc5f3b CP6.6: ONLYINCOMPOUND and COMPOUNDRULE read in the dictionary's FLAG mode
From the rebase onto craft-libs#483: the ONLYINCOMPOUND flag is a flag id
read as FLAG long/num/UTF-8 says, so a stem carrying it is a word only
inside a compound in every flag mode (English included). COMPOUNDRULE
patterns with long or numeric flags write each flag in parentheses,
`(aa)*(bb)` / `(1)*(2)`, as Hunspell reads them (checked black-box with the
hunspell CLI); a bare flag in those modes or an unclosed group is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Martin BallandClaude Opus 5.5 35285e27eb CP6.2: FLAG num reads flag 0; AF aliases and SUBSTANDARD with long and numeric flags
Follow-ups to the storytold/wordcraft#5142 port:
- Flag 0 is a flag under `FLAG num`, as Hunspell 1.7 reads it (black-box
  check): the Turkish dictionary's first rule is `SFX 0 N 1`, which the
  port refused. Turkish now loads and passes its check words.
- AF aliases hold flags written in the FLAG mode (`AF AaBb`, `AF 12,7`);
  tested against Hunspell's answers on the same files. An alias whose
  flags don't read leaves its stem out, as a bad .dic field does.
- A homonym past the merged-set cap keeps SUBSTANDARD as it keeps
  FORBIDDENWORD and NOSUGGEST (craft-proof has SUBSTANDARD; the PR's base
  didn't).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Harrison SmithandClaude Opus 5.5 561566c6bb proof: a homonym merge never drops FORBIDDENWORD or NOSUGGEST (storytold/wordcraft#4967)
A homonym that took the merged flag set past MAX_FLAGS_PER_ENTRY was
skipped whole, so if it alone carried FORBIDDENWORD or NOSUGGEST the word
became correct or suggestible.

Merged sets now have their own bound, MAX_FLAGS_PER_STEM (8 x the entry
cap), which the compact reader also accepts for stem lines, so to_bytes ->
from_bytes reloads every set build() makes. A homonym past that bound
drops its other flags but still adds its exclusion flags (evicting a
non-exclusion flag if the set is full), and a .dic entry over the entry
cap is still no word but its exclusions apply to its homonyms.

Ported from storytold/wordcraft#5142 (b9777fc92) onto craft-proof.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Harrison SmithandClaude Opus 5.5 13b6108cb4 proof: cap merged homonym flag sets; leave unused directive flags out of character-mode data (storytold/wordcraft#4967)
Review follow-ups:
- Two .dic entries for the same word could each stay within
  MAX_FLAGS_PER_ENTRY while their merged set went past it; the compact
  data then held a set its reader refuses, and the word was lost on
  reload. An entry that would take the merged set past the cap is now
  skipped, as an oversized entry is.
- In one-character mode a FORBIDDENWORD or NOSUGGEST flag that no stem
  or rule uses was written to the compact data, which the old format
  left out. It is left out again, so those dictionaries' bytes don't
  change.

Ported from storytold/wordcraft#5142 (f10bb61ab) onto craft-proof.
Conflict resolved: the used-flag filter covers SUBSTANDARD and KEEPCASE too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Harrison SmithandClaude Opus 5.5 e4182149a5 proof: read FLAG num (and long) affix dictionaries (storytold/wordcraft#4967)
The affix engine read one-character flags only, at most 128 of them, as
bits of a u128. It now reads `FLAG num` (comma-separated numbers 1-65535,
leading digits as in Hunspell) and `FLAG long` (two characters), with up
to 65,536 distinct flags (storytold/wordcraft#4966):

- One reader (`FlagMode`) parses and writes every flag field: .dic flags,
  PFX/SFX flags and continuation classes, FORBIDDENWORD and NOSUGGEST.
- Each flag gets a dense u16 id; a stem's flags and a rule's continuation
  classes are sorted id sets interned in one flat table (set 0 is empty).
- Hostile input: at most 1,024 flags per entry (Belarusian uses 264), a
  bad .aff flag is an error, a .dic entry whose flags don't read is
  skipped, and a FLAG line after flags were read is refused.
- Lookups skip the continuation walk for groups and flags no continuation
  class can reach, so checks are as fast or faster with the Polish,
  Portuguese, Czech, Italian and Russian dictionaries (same answers).
- The compact format writes a `FLAG num`/`FLAG long` line only for those
  modes, so one-character data is byte-for-byte unchanged and old data
  still loads.

LibreOffice's Serbian (Cyrillic and Latin), Belarusian, Sinhala, Kannada
and Nepali dictionaries now load; Danish, Galician, Icelandic and Korean
still stop at COMPOUNDMIN, NEEDAFFIX, FULLSTRIP and ICONV.

Ported from storytold/wordcraft#5142 (f5778bb76) onto craft-proof.
craft-proof already had storytold/wordcraft#735 (AF aliases, SUBSTANDARD,
KEEPCASE, ICONV, IGNORE, 256 one-character flags as a bitmask), which
this commit predates; conflicts resolved by carrying those through the
flag ids: AF aliases resolve to flag text read in the FLAG mode,
SUBSTANDARD and KEEPCASE are flag ids like FORBIDDENWORD and NOSUGGEST,
and lookups exclude up to four flags (Excluded). The FLAG long test
lines in the language modules and slot.rs now use an unknown mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Martin BallandClaude Opus 5.5 6e45fda67c CP2.2: review — British English takes an -ize spelling only when en-US or en-CA spells the word so
craft-libs#483 re-review (P2): ise_spelling turned any z between i/y and a
vowel into s, so en-GB (and the other British Englishes) accepted
misspellings whose -s- form is a word: wize, rize, noize, dizease, advize,
surprize, exercize… (19 reproduced; Hunspell en-GB rejects them all). The
-ize form now counts only when the word itself is in the en-US or en-CA
SCOWL dictionary and its -ise form in en-GB's (organize, realization,
colourize). English with no dialect no longer needs the swap (it reads all
three). Test: british_ize_needs_a_real_ize_word (the 19, plus colorize, wrong
in en-GB and en-AU; organize, realization, colourize, organizing, realizes
right).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:21:02 +00:00
Martin BallandClaude Opus 5.5 0675d58341 CP2.2: review — each English checks only its own SCOWL dictionary; English suggestions drop junk splits and names
craft-libs#483 review (P2): English spelling checked the union of the three
SCOWL dictionaries, so dialects crossed (en-US took colourblind, greyscale;
en-GB took colorblind, esophagi). Each English now checks its own dictionary
only. British English also takes the -ize spellings of its -ise words
(SCOWL's en-GB is -ise only; Word accepts both), and en-variants.txt widens
each dialect by its pairs; judgment/judgement come from craft-i18n#114, which
CI now pins. English with no dialect (is_correct) still takes a word any
English knows. Test: scowl_dialects_dont_cross samples the stems one
dialect has and another lacks, for every pair.

P3s:
- English suggestions (a Tuning of their own): a missing space only between
  two lower-case words of three letters or more, ranked after every one-word
  candidate (publically: publicly first; no "Thea tr", "Che Que"), and no
  word the dictionary only capitalizes (Col, Wis) for a lower-case word.
- A FORBIDDENWORD whole word is never rebuilt from compound parts.
- to_bytes writes ONLYINCOMPOUND without compound rules too (round-trip test).
- Failed lookups: the compound part table is filled lazily, and a word that
  doesn't start like a compound part (SCOWL: a digit) skips compounding;
  20,000 failed long words take 83 ms, as without compounding (1.09 s before).
- README: en-CA is not a British-spelling English.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 21:44:18 +00:00