Commit Graph
19 Commits
Author SHA1 Message Date
Martin BallandClaude Opus 5.5 cb82e91428 craft-proof — a run of spaces is one double-space issue over the whole run in every language, through one shared space_runs helper (#496 review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 23:23:15 +00:00
Martin BallandClaude Opus 5.5 bd3db196a5 craft-proof es — a run of spaces is one Espacio doble issue over the whole run (#496 review)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 21:56:44 +00:00
Martin BallandClaude Opus 5.5 8f16fe5a13 CP4.2: review — test dictionaries installed in memory hold the dictionary-tests lock until the test ends, so a pack or folder reload in another test can't drop them mid-test (the with-data ru suggestion flake)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Martin BallandClaude Opus 5.5 ee3be01b73 CP4.2: review — English tests hold the dictionaries lock; another region's pack never beats the system's exact region; bounded pack hyphenation lookups; NFC hyphenation; licences budgeted before reading; pack languages report like the others
P2:
- Every test whose English results depend on installed dictionaries takes DICTIONARY_TESTS (the en-GB goitre CI failure: tiny en_US fixtures raced unlocked en-GB tests).
- hunspell::candidates/find order: dictionaries folder, the first name's region pack (or the language alone), the system's exact-region dictionary, any other region's pack, the system's other names. pt_BR system beats a pt-pt pack; de_DE / de-ch, nl_NL / nl-be, es_ES / es-mx, en_US / en-gb likewise.

P3:
- Pack hyphenation lookups: a map, at most 256 languages (cleared when full); 3000 tags stay bounded and fast.
- A decomposed word (a + U+0301) is hyphenated composed (NFC) and the breaks mapped back; combining marks stay in their word.
- licence_texts checks each file's size against what is left before reading it; never reads past 16 MB.
- German, Bulgarian and pack languages: spelling message in the language where we have it; missing_dictionary/unreadable_dictionary report them like the others.
- build.rs: a missing file no longer says nothing is embedded; ru ignore message says CRAFT_I18N_DIR is enough. Built-in English list notice names the whole Moby list (en-words.dic).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:33 +00:00
Harrison SmithandClaude Opus 5.5 e4182149a5 proof: read FLAG num (and long) affix dictionaries (storytold/wordcraft#4967)
The affix engine read one-character flags only, at most 128 of them, as
bits of a u128. It now reads `FLAG num` (comma-separated numbers 1-65535,
leading digits as in Hunspell) and `FLAG long` (two characters), with up
to 65,536 distinct flags (storytold/wordcraft#4966):

- One reader (`FlagMode`) parses and writes every flag field: .dic flags,
  PFX/SFX flags and continuation classes, FORBIDDENWORD and NOSUGGEST.
- Each flag gets a dense u16 id; a stem's flags and a rule's continuation
  classes are sorted id sets interned in one flat table (set 0 is empty).
- Hostile input: at most 1,024 flags per entry (Belarusian uses 264), a
  bad .aff flag is an error, a .dic entry whose flags don't read is
  skipped, and a FLAG line after flags were read is refused.
- Lookups skip the continuation walk for groups and flags no continuation
  class can reach, so checks are as fast or faster with the Polish,
  Portuguese, Czech, Italian and Russian dictionaries (same answers).
- The compact format writes a `FLAG num`/`FLAG long` line only for those
  modes, so one-character data is byte-for-byte unchanged and old data
  still loads.

LibreOffice's Serbian (Cyrillic and Latin), Belarusian, Sinhala, Kannada
and Nepali dictionaries now load; Danish, Galician, Icelandic and Korean
still stop at COMPOUNDMIN, NEEDAFFIX, FULLSTRIP and ICONV.

Ported from storytold/wordcraft#5142 (f5778bb76) onto craft-proof.
craft-proof already had storytold/wordcraft#735 (AF aliases, SUBSTANDARD,
KEEPCASE, ICONV, IGNORE, 256 one-character flags as a bitmask), which
this commit predates; conflicts resolved by carrying those through the
flag ids: AF aliases resolve to flag text read in the FLAG mode,
SUBSTANDARD and KEEPCASE are flag ids like FORBIDDENWORD and NOSUGGEST,
and lookups exclude up to four flags (Excluded). The FLAG long test
lines in the language modules and slot.rs now use an unknown mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 22:34:28 +00:00
Martin BallandClaude Opus 5.5 ed6bea9b6e CP1.2: Spanish address timing test guards against blowups, not machine load
The bound was 150 ms; at opt-level 0 under a parallel test run the fixed
lookup came close to it and flaked. 1 s still fails a return to the
quadratic scan (storytold/wordcraft#3562) without timing the machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 16:14:44 +00:00
Martin BallandClaude Opus 5.5 547f303095 CP1.2: craft-proof carries wordcraft crates/proof up to storytold/wordcraft@9c72ddf56
Merges the crates/proof history from storytold/wordcraft a88e00525..9c72ddf56
(storytold/wordcraft#5000, storytold/wordcraft#4235: 21 commits and 5 merges),
split with the same git filter-repo run as the first import, so the shared
history has the same commit ids and every commit keeps its author.

Conflicts in src/lib.rs, resolved with the CP1.2 changes kept: the embedded
`data` module beside the new `affix_suggest`; a dictionaries-folder change
reloads every language slot and English; English load errors still come from
the English source; the dictionaries-folder test helpers stay as CP1.2 split
them, with DICTIONARY_TESTS documented for the user-dictionary tests too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 16:12:06 +00:00
Martin BallandClaude Opus 5.5 b758ba6943 CP1.2: craft-proof embeds Language data from CRAFT_I18N_DIR; works without it
- Package renamed wordcraft-proof -> craft-proof; [workspace.dependencies]
  entry; public API unchanged.
- build.rs embeds craft-i18n language-data/ (en-us, de, pl, pt, it, ru)
  instead of include_str!/include_bytes! of WordCraft's assets/: warns when
  CRAFT_I18N_DIR is unset, errors under CRAFT_I18N_REQUIRED=1.
- No data: English spelling marks and suggests nothing, hyphenation that
  needs data is off, grammar and rule-based hyphenation work. no_data.rs runs
  every public entry point without data; data-pinning tests are ignored then
  (cfg craft_proof_data).
- es timing bound 75 -> 150 ms: craft-libs tests run at opt-level 0.
- proof.yml (fmt, clippy -D warnings, tests with and without data, release
  guard, wasm32), README, CODEOWNERS for crates/proof.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
2026-11-01 10:02:10 +00:00
Dylan M. Taylor e8c1c4757a Sonar: use PoisonError::into_inner, explicit test imports, [[ in import-cert
- proof: pass PoisonError::into_inner instead of an identity closure (S1612).
- layout tests: replace glob super::* imports with explicit names (S2208).
- packaging/macos/import-cert.sh: use [[ for the status test (S7688).
2026-10-25 21:28:59 -04:00
VoiceLessQ 140c7041e1 Proofing: the spelling options are read in one place
Czech, Croatian, Indonesian, Italian, Polish, Portuguese and Spanish began
their spelling checks with the same thirteen lines, byte for byte: no
dictionary means no marks, single letters and words without letters pass,
words with digits and short words in capitals pass when the File › Options
› Proofing choices say so, and words in another script pass. Each also had
its own copy of the Latin-script test. These were the lines storytold/wordcraft#3966 had to fix
in six copies one by one, and where the Russian and Turkish copies stayed
wrong until the first commit here.

spelling.rs has them once (`word_to_check`), and each language goes on
from there with its own rules (Polish apostrophe endings, Italian elisions,
Indonesian clitics, Uruguayan `vos` forms).

Tests: words_needing_no_lookup. Ignoring the uppercase option there fails
it and all seven languages' proofing-options tests. Proof: 286 tests.
2026-10-24 19:58:30 -01:00
VoiceLessQ a388c4787c Proofing: one language table instead of nine lists
lib.rs named every dictionary language in nine places: reloading for a new
folder, the installed list, load errors, the missing-dictionary hint, the
loaded check, the spelling message, spelling, suggestions and grammar. The
lists had drifted. dictionary_load_error had no Russian or Czech entry, so a
Russian or Czech dictionary that failed to load showed no reason in File ›
Options › Proofing, though the status bar, which reads another list, said
why.

LANGUAGES holds one row per language: its dictionary slot, its spelling
message, and its spelling, suggestion and grammar functions. The nine
functions read the row, so a language now joins with its module, its
ProofLang and one row. The per-language reload_dictionary and
installed_path wrappers had no callers left, inside the crate or out, and
are gone.

Tests: every_dictionary_language_reports_its_load_error, which fails for
Russian on the old code, and the_language_table_covers_the_proofed_languages
(each ProofLang::PROOFED language but English has exactly one row). Proof
285, engine 927 and layout 501 (lib).
2026-10-24 19:55:43 -01:00
VoiceLessQ c19b1cae73 Proofing: one suggestion engine with per-language tuning
Czech's dictionary-driven ranking (case, MAP, REP, missing spaces, KEY,
TRY edits, a stem scan) had five older copies. Polish, Portuguese,
Croatian, Italian and Russian each kept one: 300 to 400 lines of the same
rounds with a few lines of language knowledge mixed in. Fixes reached some
copies only. NOSUGGEST (storytold/wordcraft#1048) and the first-difference tie-break (storytold/wordcraft#797)
never got to all of them, and none had the KEY round or the bound on TRY
letters.

affix_suggest.rs is that ranking once. A language's `Tuning` holds what is
its own: the short words that run into their neighbours (Polish,
Portuguese, Croatian, Italian), Russian's split rule, the letter groups and
confusions its dictionaries lack (Croatian, Russian), letters to try for a
dictionary without a TRY line, and cost hooks for Polish final devoicing
and Russian е/ё. Czech, Indonesian and Spanish use the dictionary's tables
alone. Turkish keeps its own suggester, which reads another dictionary type
with rounds of its own.

The language tables add to the dictionary's instead of replacing them, and
TRY letters are tried in the dictionary's order, commonest first, as the
copies did (the Czech engine sorted them).

Old and new compared on the test dictionaries, with every one-letter typo
of each word plus the tests' misspellings, top eight suggestions: Czech,
Croatian, Italian and Portuguese give the same for all 7,903 inputs.
Polish differs for 44 of 3,244 and Russian for 33 of 2,346, all orderings
among candidates of equal cost, from the tie-break they now share (`ie` →
`nie` before `i`).

New tests pin the moved knowledge: the tuning mechanism, Croatian's groups
without MAP lines, and Russian's costs and split rule. Turning each knob
off fails one. Proof: 283 tests.
2026-10-24 19:49:22 -01:00
VoiceLessQ 8f4616eece Proofing: one dictionary slot for every language
Nine languages each kept their own copy of the same code: the installed
dictionary behind a lock, read on first use and logged, replaced by
install_dictionary, forgotten by reload_dictionary, and its file reported
by installed_path. The copies had drifted already. Every one documented that
a dictionary given with install_dictionary has no file and isn't reported,
but only Russian and Czech did that (storytold/wordcraft#3252); the other seven reported the
disk file anyway.

`DictSlot` (proof/src/slot.rs) is that code once. It is generic over the
dictionary type, so Turkish's `TrDict` fits, and keeps the dictionary and
its in-memory origin under one lock as storytold/wordcraft#3252 made it. Each language keeps
its functions as one-line wrappers over its slot, so no caller changes, and
the log lines are the same.

installed_path now follows the documented rule in every language. Nothing
outside tests installs a dictionary in memory, but tests do, concurrently,
so the dictionaries-folder tests in proof and engine could race on the
reported path of a language another test installs. They now accept the
folder's file or none there, and check that the folder is searched first
with hunspell::find, which no install affects.

Tests: proof 279, engine 927, layout 501 (lib); slot.rs has a test of its
own.
2026-10-24 19:33:17 -01:00
Federico Olivares a98e00c11a Merge origin/main; fix false missing-opening-mark flags on correctly opened questions; stabilise dictionary test 2026-10-23 20:57:36 -03:00
Federico Olivares 319dec238a storytold/wordcraft#3360: Spanish address lookup is a binary search, not a scan per character
Maintainer review on storytold/wordcraft#3562: opening_marks reverse-scans the characters before a
question or exclamation mark, and ends_clause asked in_address for each one,
which scanned every address range, so a paragraph of thousands of addresses
followed by a question took quadratic time (about 99 KB: hundreds of ms).

address_at() finds the range a byte is in with a binary search over the
sorted, disjoint ranges, and also gives ends_clause the range's end without
another scan. Behaviour is unchanged (the URL and punctuation regressions
still pass).

- address_lookup_finds_the_range_a_byte_is_in: boundaries of the lookup
- many_addresses_before_a_question_stay_linear: 99 KB of addresses before a
  question or exclamation (the review's reproducer and two variants), checked
  for the right issue range and for taking under 75 ms (about 1 to 5 ms now,
  about 300 ms with a scan per character)
2026-10-23 19:27:07 -03:00
Federico Olivares 970bfcf7a8 storytold/wordcraft#3360: Spanish proofing code clean-up (no behaviour change)
- es/mod.rs: one variety() match gives the dictionary slot, log name and file
  names (variety_name() folded in); load_installed() destructures it
- es/grammar.rs: the sentence-start rule for the missing ¿/¡ is one named
  function, ends_clause(), replacing is_boundary() and three closures
  rebuilt per mark; in_address() is shared; repeated_words() builds its
  helpers once instead of per word pair
- es/suggest.rs reuses the Czech ranking's MAX_LETTERS instead of a copy

Same tests pass unchanged; a temporary old-vs-new differential proptest
(20,000+ generated inputs with URLs and punctuation) found identical output.
2026-10-23 09:19:18 -03:00
Federico Olivares ecb4638980 storytold/wordcraft#3360: decomposed accents stay in their word; URLs aren't Spanish questions
Maintainer review on storytold/wordcraft#3562:
- words() keeps combining marks with the word they follow, so a word typed
  decomposed (NFD, `tambie\u{301}n`) is one word with its own byte range;
  is_correct_in_with and suggest_in look it up composed (NFC), so it is
  checked like `también` in every language (unicode-normalization, MIT/Apache)
- the Spanish missing-¿/¡ rule skips marks in web and e-mail addresses
  (`Ver https://example.com/?`), and punctuation inside an address or a word
  (`example.com`, `https:`, `10:30`) no longer starts the sentence the
  opening mark is offered for; a mark ending an address still ends a sentence
- Spanish repeated-word check matches composed and decomposed spellings
- regressions: check_text with the Spanish fixture and NFC/NFD equivalents
  (acceptance and ranges), words() and composed(), URLs ending in /? beside
  ordinary questions

CodeRabbit review: Session::region_spanish takes the ISO 639-2 `spa` as
Spanish, so a `spa-MX` region writes es-MX documents.
2026-10-23 08:43:44 -03:00
Federico Olivares e0abb96729 storytold/wordcraft#3360: Spanish honours the proofing options; es-AR finds es_AR
CodeRabbit review on storytold/wordcraft#3562:
- with Ignore words in UPPERCASE off, short words in capitals are checked
  against the Spanish dictionary instead of accepted as acronyms; likewise
  words with digits with Ignore words that contain numbers off. Direct
  es::is_correct/is_correct_with keep the defaults.
- Latin American Spanish (which es-AR resolves to) also looks for es_AR and
  es-AR, after the neutral es_US and es_MX, so Argentine text is checked when
  only that dictionary is installed
- docs: the exact candidate order per variety in localization-parity.md, and
  es_AR in the spelling-dictionaries table
- the Spain variant's doc lists Uruguay among the regions it doesn't cover
2026-10-22 17:48:42 -03:00
Federico Olivares 3cd7f35193 storytold/wordcraft#3360, storytold/wordcraft#3553: Spanish proofing (es-ES, es-MX, es-419, es-UY)
Spanish text was never checked: ProofLang had no Spanish variant, so every
es-* run fell through to Other (no marks). And the RLA-ES dictionaries
LibreOffice ships were refused by the affix engine for their `☎️` flag
(☎ + variation selector U+FE0F), which Hunspell reads as ☎.

- ProofLang::Es (Spain, plain es, spa), EsMx (es-MX), Es419 (es-419 and
  the other countries of the Americas) and EsUy (es-UY), chosen by region
  like the English dialects
- each variety reads its own installed Hunspell dictionary with fallbacks
  (es_ES; es_MX; es_419/es_US/es_MX; es_UY/es_AR); nothing bundled
- variation selectors in affix flag fields belong to the flag before them
- Uruguayan vos forms accepted when the dictionary has the infinitive:
  imperatives (vení, andá), with pronouns (decime, decímelo), present
  (tenés); a vos form typed without its accent gets it back first
- suggestions use the dictionary-driven Czech ranking (prueva → prueba,
  corector → corrector, tambien → también)
- grammar: repeated word, spacing, missing opening ¿/¡, space after an
  opening mark, with Spanish messages
- Review › Language, status bar, Language dialog and File › Options ›
  Proofing list the four varieties; the two new names translated in every
  catalog; a Spanish interface writes new documents in the region's Spanish
- the dictionaries-folder engine test holds the Italian and Spanish
  fixtures, so parallel proofing tests no longer lose their dictionaries
- tests use hand-written synthetic dictionaries; engine end-to-end tests
2026-10-22 01:53:37 -03:00