P2:
- Every test whose English results depend on installed dictionaries takes DICTIONARY_TESTS (the en-GB goitre CI failure: tiny en_US fixtures raced unlocked en-GB tests).
- hunspell::candidates/find order: dictionaries folder, the first name's region pack (or the language alone), the system's exact-region dictionary, any other region's pack, the system's other names. pt_BR system beats a pt-pt pack; de_DE / de-ch, nl_NL / nl-be, es_ES / es-mx, en_US / en-gb likewise.
P3:
- Pack hyphenation lookups: a map, at most 256 languages (cleared when full); 3000 tags stay bounded and fast.
- A decomposed word (a + U+0301) is hyphenated composed (NFC) and the breaks mapped back; combining marks stay in their word.
- licence_texts checks each file's size against what is left before reading it; never reads past 16 MB.
- German, Bulgarian and pack languages: spelling message in the language where we have it; missing_dictionary/unreadable_dictionary report them like the others.
- build.rs: a missing file no longer says nothing is embedded; ru ignore message says CRAFT_I18N_DIR is enough. Built-in English list notice names the whole Moby list (en-words.dic).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
The affix engine read one-character flags only, at most 128 of them, as
bits of a u128. It now reads `FLAG num` (comma-separated numbers 1-65535,
leading digits as in Hunspell) and `FLAG long` (two characters), with up
to 65,536 distinct flags (storytold/wordcraft#4966):
- One reader (`FlagMode`) parses and writes every flag field: .dic flags,
PFX/SFX flags and continuation classes, FORBIDDENWORD and NOSUGGEST.
- Each flag gets a dense u16 id; a stem's flags and a rule's continuation
classes are sorted id sets interned in one flat table (set 0 is empty).
- Hostile input: at most 1,024 flags per entry (Belarusian uses 264), a
bad .aff flag is an error, a .dic entry whose flags don't read is
skipped, and a FLAG line after flags were read is refused.
- Lookups skip the continuation walk for groups and flags no continuation
class can reach, so checks are as fast or faster with the Polish,
Portuguese, Czech, Italian and Russian dictionaries (same answers).
- The compact format writes a `FLAG num`/`FLAG long` line only for those
modes, so one-character data is byte-for-byte unchanged and old data
still loads.
LibreOffice's Serbian (Cyrillic and Latin), Belarusian, Sinhala, Kannada
and Nepali dictionaries now load; Danish, Galician, Icelandic and Korean
still stop at COMPOUNDMIN, NEEDAFFIX, FULLSTRIP and ICONV.
Ported from storytold/wordcraft#5142 (f5778bb76) onto craft-proof.
craft-proof already had storytold/wordcraft#735 (AF aliases, SUBSTANDARD,
KEEPCASE, ICONV, IGNORE, 256 one-character flags as a bitmask), which
this commit predates; conflicts resolved by carrying those through the
flag ids: AF aliases resolve to flag text read in the FLAG mode,
SUBSTANDARD and KEEPCASE are flag ids like FORBIDDENWORD and NOSUGGEST,
and lookups exclude up to four flags (Excluded). The FLAG long test
lines in the language modules and slot.rs now use an unknown mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
Czech, Croatian, Indonesian, Italian, Polish, Portuguese and Spanish began
their spelling checks with the same thirteen lines, byte for byte: no
dictionary means no marks, single letters and words without letters pass,
words with digits and short words in capitals pass when the File › Options
› Proofing choices say so, and words in another script pass. Each also had
its own copy of the Latin-script test. These were the lines storytold/wordcraft#3966 had to fix
in six copies one by one, and where the Russian and Turkish copies stayed
wrong until the first commit here.
spelling.rs has them once (`word_to_check`), and each language goes on
from there with its own rules (Polish apostrophe endings, Italian elisions,
Indonesian clitics, Uruguayan `vos` forms).
Tests: words_needing_no_lookup. Ignoring the uppercase option there fails
it and all seven languages' proofing-options tests. Proof: 286 tests.
lib.rs named every dictionary language in nine places: reloading for a new
folder, the installed list, load errors, the missing-dictionary hint, the
loaded check, the spelling message, spelling, suggestions and grammar. The
lists had drifted. dictionary_load_error had no Russian or Czech entry, so a
Russian or Czech dictionary that failed to load showed no reason in File ›
Options › Proofing, though the status bar, which reads another list, said
why.
LANGUAGES holds one row per language: its dictionary slot, its spelling
message, and its spelling, suggestion and grammar functions. The nine
functions read the row, so a language now joins with its module, its
ProofLang and one row. The per-language reload_dictionary and
installed_path wrappers had no callers left, inside the crate or out, and
are gone.
Tests: every_dictionary_language_reports_its_load_error, which fails for
Russian on the old code, and the_language_table_covers_the_proofed_languages
(each ProofLang::PROOFED language but English has exactly one row). Proof
285, engine 927 and layout 501 (lib).
Czech's dictionary-driven ranking (case, MAP, REP, missing spaces, KEY,
TRY edits, a stem scan) had five older copies. Polish, Portuguese,
Croatian, Italian and Russian each kept one: 300 to 400 lines of the same
rounds with a few lines of language knowledge mixed in. Fixes reached some
copies only. NOSUGGEST (storytold/wordcraft#1048) and the first-difference tie-break (storytold/wordcraft#797)
never got to all of them, and none had the KEY round or the bound on TRY
letters.
affix_suggest.rs is that ranking once. A language's `Tuning` holds what is
its own: the short words that run into their neighbours (Polish,
Portuguese, Croatian, Italian), Russian's split rule, the letter groups and
confusions its dictionaries lack (Croatian, Russian), letters to try for a
dictionary without a TRY line, and cost hooks for Polish final devoicing
and Russian е/ё. Czech, Indonesian and Spanish use the dictionary's tables
alone. Turkish keeps its own suggester, which reads another dictionary type
with rounds of its own.
The language tables add to the dictionary's instead of replacing them, and
TRY letters are tried in the dictionary's order, commonest first, as the
copies did (the Czech engine sorted them).
Old and new compared on the test dictionaries, with every one-letter typo
of each word plus the tests' misspellings, top eight suggestions: Czech,
Croatian, Italian and Portuguese give the same for all 7,903 inputs.
Polish differs for 44 of 3,244 and Russian for 33 of 2,346, all orderings
among candidates of equal cost, from the tie-break they now share (`ie` →
`nie` before `i`).
New tests pin the moved knowledge: the tuning mechanism, Croatian's groups
without MAP lines, and Russian's costs and split rule. Turning each knob
off fails one. Proof: 283 tests.
Nine languages each kept their own copy of the same code: the installed
dictionary behind a lock, read on first use and logged, replaced by
install_dictionary, forgotten by reload_dictionary, and its file reported
by installed_path. The copies had drifted already. Every one documented that
a dictionary given with install_dictionary has no file and isn't reported,
but only Russian and Czech did that (storytold/wordcraft#3252); the other seven reported the
disk file anyway.
`DictSlot` (proof/src/slot.rs) is that code once. It is generic over the
dictionary type, so Turkish's `TrDict` fits, and keeps the dictionary and
its in-memory origin under one lock as storytold/wordcraft#3252 made it. Each language keeps
its functions as one-line wrappers over its slot, so no caller changes, and
the log lines are the same.
installed_path now follows the documented rule in every language. Nothing
outside tests installs a dictionary in memory, but tests do, concurrently,
so the dictionaries-folder tests in proof and engine could race on the
reported path of a language another test installs. They now accept the
folder's file or none there, and check that the folder is searched first
with hunspell::find, which no install affects.
Tests: proof 279, engine 927, layout 501 (lib); slot.rs has a test of its
own.
Each of these languages accepted every word in capitals of up to six letters and every word
with a digit on its own, so turning off Ignore words in UPPERCASE or Ignore words that contain
numbers had no effect on them for short words. The check now takes the options, as the Spanish
one does; the direct is_correct/is_correct_with keep the defaults.
Co-authored-by: Federico Olivares <FedeOlivares@users.noreply.github.com>
Co-authored-by: aiongg <67543387+aiongg@users.noreply.github.com>
* Proofing: say when a dictionary is installed but unreadable, with file and reason (storytold/wordcraft#2830)
* proof test: use SHIFT_JIS as the unsupported encoding (ISO 8859-1 is decoded since storytold/wordcraft#2904)
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Indonesian spelling reads the installed Hunspell id_ID dictionary at runtime
(Debian/Ubuntu hunspell-id, LibreOffice dict-id, the Dictionaries folder or
WORDCRAFT_DICTIONARIES), like Czech, Polish and Portuguese: nothing is bundled,
and without a dictionary Indonesian text gets no spelling marks.
- ProofLang::Id for id, id-ID, ind and the legacy in tag
- suggestions reuse the dictionary-driven Czech ranking (MAP/REP/TRY/KEY of
the loaded dictionary), nothing Indonesian hard-coded
- hyphenated reduplication accepted whole or by parts (kupu-kupu, anak-anak);
hyphenated clitics after capitals/abbreviations (KTP-nya, se-Indonesia)
- generic grammar only: repeated word (offers anak-anak first), space before
punctuation, double space, with Indonesian messages
- Review > Language, status bar, Set Proofing Language, File > Options >
Proofing list Indonesian; name translated in every catalog
- tests use a hand-written synthetic dictionary; engine end-to-end test
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>