The Hungarian Hunspell dictionary declares SET UTF-8 but writes its flags as single bytes, uses AF aliases and 158 distinct flags. Read flag fields byte by byte unless FLAG UTF-8 is set, expand AF aliases, skip AM, apply ICONV and IGNORE, honour KEEPCASE, and widen flag masks to 256 bits. Compounding and NEEDAFFIX stay unsupported for now.
- proof: pass PoisonError::into_inner instead of an identity closure (S1612).
- layout tests: replace glob super::* imports with explicit names (S2208).
- packaging/macos/import-cert.sh: use [[ for the status test (S7688).
Wall-clock bounds of 1-2 s failed on loaded runners. Each bound is now 10 s, at least 10x the
fixed code's time and well under the old code's on the same input (measured by running the
pre-fix code):
- self-continuing rules: 40,000 rules (was 16,000); ~1 ms now, old checker 14 s at 16,000.
- many strips per suffix: 200,000 rules (was 30,000); 0.4 s now, old index 3 s at 30,000 and
11 min at 300,000.
- forms of self-continuing rules: the MAX_FORMS count catches the old code (9 million forms);
the time bound is a backstop.
- overlong Turkish words: ~20 ms now; without the caps it ran past 5 minutes.
- capped Hunspell reads: no clock; the "not a regular file" error is the proof (the old reader
never returned from /dev/zero).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CX1hMKX28764Lrx5moxs1o
Asked to confirm a non-UTF-8 Turkish dictionary still loads: before this PR
tr_TR was read with read_to_string, so a non-UTF-8 file never loaded; now it
goes through read_capped and decode, which didn't know ISO 8859-9 (Latin-5,
the encoding of older tr_TR dictionaries) and refused it as unsupported.
decode now reads `SET ISO8859-9`, and a test writes an ISO 8859-9 .aff/.dic
to disk and finds words with ğ ı ş İ ç ö ü, as stems and with suffixes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CX1hMKX28764Lrx5moxs1o
Audit finding 20.
- tr::suggest_with refuses words over 64 letters (and words with spaces), like the other
languages' suggesters (40 there; Turkish agglutinated words run past 40). Each candidate edit
copies the whole word, so a long word cost O(n^2) per suggestion request on the UI thread.
- TrDict::contains returns false past MAX_WORD_BYTES (120, the affix engine's bound) instead of
collecting the char indices of the whole word.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Audit findings 10 (storytold/wordcraft#3657) and 15 (storytold/wordcraft#3692).
- ru::hyphen::points: the cuts come in increasing order, so a repeat can only be the last one;
compare with `out.last()` instead of `out.contains` per syllable (O(V^2) on a long Cyrillic
word, during layout).
- hyphenate_word_in: walk the sorted break points alongside the chars instead of searching the
point list for every char. Same output.
Test: russian_hyphenation_and_hyphenated_output_are_linear (8x the letters costs ~8x; the old
code measured 31x and 30x).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Audit finding 4, and the rest of storytold/wordcraft#3661.
- dict.rs: `Dictionary::from_bytes` inflates with `decompress_to_vec_with_limit` (64 MiB; the
bundled en-US one is ~1 MB) and caps the words it keeps (bytes and count): front coding lets a
2-byte line spell out a 255-byte word, so the inflated size alone doesn't bound storage.
- hunspell.rs: one `read_capped` reader — regular files only, size checked on the open handle and
the read itself bounded by `take(MAX_FILE + 1)`, so no TOCTOU between metadata and read and no
endless read from a FIFO or device.
- tr: the Turkish loader reads through `read_capped` and decodes with `hunspell::decode` (it
read both files whole, with no cap at all).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Part of storytold/wordcraft#3613 (audit findings 2 and 6).
- Checker: `suffixed` and `form_has_flag` OR the flags of a group's matching rules into one
mask and recurse once, not once per rule. Same answer (the walk is a disjunction over flags);
16k self-continuing empty-append rules: 6 s -> <1 ms per check (debug).
- Finding 2: `forms` expands each (form, continuation) pair once, walks continuation rules through
a per-flag rule list, and stops at MAX_FORMS (4096) forms / MAX_FORMS_WORK rule visits, so a
dictionary of self-continuing rules can't allocate R^2 strings per stem.
- Finding 6: the suffix index finds a rule's (append, strip) group by hash, not by scanning the
append's groups: building with many strips per append is linear.
Not done here (rest of storytold/wordcraft#3613): a work budget for distinct-strip chains in the checker, rule caps
in parse_aff, proofing off the UI thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It came in with the first proofing commit (bcbe09df8) and has never had a caller; `#[allow(dead_code)]` kept the warning quiet. Because it is `pub(crate)`, nothing outside wordcraft-proof can reach it.
SonarCloud's analysis of main at b4da59f0c lists 23 closures such as `|v| v.as_slice()` that hand their argument straight to one method. This passes the method itself: `Vec::as_slice`, `str::to_ascii_lowercase`, `f64::total_cmp` and the like, all forms the repo already uses elsewhere. Behaviour is unchanged.
Five stay. The four in crates/proof/src/es/mod.rs sit in code that storytold/wordcraft#4191 replaces, and its replacement already uses `PoisonError::into_inner`. In crates/engine/src/io/safe_save.rs the path would be `<dyn std::error::Error + Send + Sync>::is::<Guided>`, which reads worse than `|inner| inner.is::<Guided>()`.
The spelling check found the start of the white-space-delimited token around
a word one byte after the last white-space character. After a no-break space
(two bytes in UTF-8) or an ideographic space (three bytes) that offset is
inside the character, so the token could not be read and was never classified
as a web or e-mail address. With Ignore web and file addresses on, the words
of such an address were still marked as misspelled.
The token now starts after the whole white-space character. A proofing test
covers a web address, a www. address and an e-mail address after the two
spaces, with the option on and off.
* chore: resolve SonarCloud maintainability issues
* fix: restore compiling refactors, revert behaviour changes, run cargo fmt
- fontdb::norm: keep the closure for filter (char::is_alphanumeric takes
self, not &self, so it does not fit Iterator::filter).
- story.rs: write_mark_rpr needs &mut self (rev_attrs, rpr_change).
- Revert the viewport meta change and the return 0 added to
workspace_version/sha256, which changed behaviour or masked failures.
- cargo fmt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Import Document in odt page_parts after dropping glob re-export
* fix: apply_kind_properties binds ContentControl; cell_ref keeps char closures
* fix(engine): pass case helpers by value (clippy needless borrows)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Dylan Taylor <dylan@craftmaker.dev>
Each of these languages accepted every word in capitals of up to six letters and every word
with a digit on its own, so turning off Ignore words in UPPERCASE or Ignore words that contain
numbers had no effect on them for short words. The check now takes the options, as the Spanish
one does; the direct is_correct/is_correct_with keep the defaults.
Co-authored-by: Federico Olivares <FedeOlivares@users.noreply.github.com>
Co-authored-by: aiongg <67543387+aiongg@users.noreply.github.com>
Maintainer review on storytold/wordcraft#3562: opening_marks reverse-scans the characters before a
question or exclamation mark, and ends_clause asked in_address for each one,
which scanned every address range, so a paragraph of thousands of addresses
followed by a question took quadratic time (about 99 KB: hundreds of ms).
address_at() finds the range a byte is in with a binary search over the
sorted, disjoint ranges, and also gives ends_clause the range's end without
another scan. Behaviour is unchanged (the URL and punctuation regressions
still pass).
- address_lookup_finds_the_range_a_byte_is_in: boundaries of the lookup
- many_addresses_before_a_question_stay_linear: 99 KB of addresses before a
question or exclamation (the review's reproducer and two variants), checked
for the right issue range and for taking under 75 ms (about 1 to 5 ms now,
about 300 ms with a scan per character)
Upstream merged storytold/wordcraft#3177 (dictionary hints for every Hunspell language) and storytold/wordcraft#3480.
Resolve the conflicts in crates/proof/src/lib.rs by taking storytold/wordcraft#3177's
dictionary_loaded/dictionary_needed structure and adding the four Spanish
varieties to it, and in docs/spelling-dictionaries.md keeping both sides.
Maintainer review on storytold/wordcraft#3562: spelling looks words up composed (NFC), but
add_word, remove_word, ignore_word, custom_word (so loading the dictionary
file) and save_user_dictionary's removed list kept the raw lower-cased
spelling. A word added or ignored decomposed (`zqprue\u{301}ba`) therefore
stayed flagged.
- dictionary_key(): lower case, then composed; every write uses it, and
in_user_dictionary() looks up composed, so either form adds, ignores,
removes and finds the same entry; NFD entries in an existing file load
and save composed
- English's known_core() reuses in_user_dictionary() instead of a copy of it
- issue byte ranges still cover the text as typed
- regressions: proof (add, Ignore All, remove, save/load with an NFD entry,
both forms rechecked) and engine (review.addToDictionary and
review.ignoreAll with the issue's NFD text)
- es/mod.rs: one variety() match gives the dictionary slot, log name and file
names (variety_name() folded in); load_installed() destructures it
- es/grammar.rs: the sentence-start rule for the missing ¿/¡ is one named
function, ends_clause(), replacing is_boundary() and three closures
rebuilt per mark; in_address() is shared; repeated_words() builds its
helpers once instead of per word pair
- es/suggest.rs reuses the Czech ranking's MAX_LETTERS instead of a copy
Same tests pass unchanged; a temporary old-vs-new differential proptest
(20,000+ generated inputs with URLs and punctuation) found identical output.
Maintainer review on storytold/wordcraft#3562:
- words() keeps combining marks with the word they follow, so a word typed
decomposed (NFD, `tambie\u{301}n`) is one word with its own byte range;
is_correct_in_with and suggest_in look it up composed (NFC), so it is
checked like `también` in every language (unicode-normalization, MIT/Apache)
- the Spanish missing-¿/¡ rule skips marks in web and e-mail addresses
(`Ver https://example.com/?`), and punctuation inside an address or a word
(`example.com`, `https:`, `10:30`) no longer starts the sentence the
opening mark is offered for; a mark ending an address still ends a sentence
- Spanish repeated-word check matches composed and decomposed spellings
- regressions: check_text with the Spanish fixture and NFC/NFD equivalents
(acceptance and ranges), words() and composed(), URLs ending in /? beside
ordinary questions
CodeRabbit review: Session::region_spanish takes the ISO 639-2 `spa` as
Spanish, so a `spa-MX` region writes es-MX documents.
CodeRabbit review on storytold/wordcraft#3562:
- with Ignore words in UPPERCASE off, short words in capitals are checked
against the Spanish dictionary instead of accepted as acronyms; likewise
words with digits with Ignore words that contain numbers off. Direct
es::is_correct/is_correct_with keep the defaults.
- Latin American Spanish (which es-AR resolves to) also looks for es_AR and
es-AR, after the neutral es_US and es_MX, so Argentine text is checked when
only that dictionary is installed
- docs: the exact candidate order per variety in localization-parity.md, and
es_AR in the spelling-dictionaries table
- the Spain variant's doc lists Uruguay among the regions it doesn't cover
Resolve conflicts with German hyphenation (ProofLang::De), Russian and Czech
in installed_dictionaries(), and storytold/wordcraft#2164's U+FE0F flag fix: affix.rs takes
main's fix and drops this branch's duplicate. The Danish, Dutch, Romanian and
Slovenian catalogs gain the Spanish (Latin America) and (Uruguay) names; the
language menu tests get room for the four Spanish entries.
Spanish text was never checked: ProofLang had no Spanish variant, so every
es-* run fell through to Other (no marks). And the RLA-ES dictionaries
LibreOffice ships were refused by the affix engine for their `☎️` flag
(☎ + variation selector U+FE0F), which Hunspell reads as ☎.
- ProofLang::Es (Spain, plain es, spa), EsMx (es-MX), Es419 (es-419 and
the other countries of the Americas) and EsUy (es-UY), chosen by region
like the English dialects
- each variety reads its own installed Hunspell dictionary with fallbacks
(es_ES; es_MX; es_419/es_US/es_MX; es_UY/es_AR); nothing bundled
- variation selectors in affix flag fields belong to the flag before them
- Uruguayan vos forms accepted when the dictionary has the infinitive:
imperatives (vení, andá), with pronouns (decime, decímelo), present
(tenés); a vos form typed without its accent gets it back first
- suggestions use the dictionary-driven Czech ranking (prueva → prueba,
corector → corrector, tambien → también)
- grammar: repeated word, spacing, missing opening ¿/¡, space after an
opening mark, with Spanish messages
- Review › Language, status bar, Language dialog and File › Options ›
Proofing list the four varieties; the two new names translated in every
catalog; a Spanish interface writes new documents in the region's Spanish
- the dictionaries-folder engine test holds the Italian and Spanish
fixtures, so parallel proofing tests no longer lose their dictionaries
- tests use hand-written synthetic dictionaries; engine end-to-end tests
crates/proof/src/lang.rs: main's English-dialect helpers kept; code() doc lists `de`.
crates/proof/src/lib.rs: hostile-input test loops over main's languages plus De.
crates/ui-egui/src/options.rs: language_name maps EnGb/EnCa and De to no dictionary name.
ROADMAP.md: main's status lines; both progress-log and revision rows kept.
docs/gaps.md: main's status line; gap 8 row is main's with German hyphenation added; revision rows unioned.
docs/layout-parity.md: main's status line, Tabs and Drop caps rows; the PR's Hyphenation row; revision rows unioned.
docs/localization-parity.md: main's status line; revision rows unioned.
* storytold/wordcraft#1281: English dialects in proofing — en-GB, en-CA, en-AU… flag each other's spellings (storytold/wordcraft#1045)
English was one proofing language: en-GB, en-CA and en-AU all proofed like
US English, and US documents accepted British spellings. Now ProofLang has
three Englishes, picked by the tag's region:
- En (en-US, other regions, no language): British spellings flagged
(colour → color).
- EnGb (en-GB, en-AU, en-NZ, en-IE, en-ZA, en-IN…): American spellings
flagged (color → colour, center → centre, analyze → analyse); -ize and
-ise both right (Oxford spelling).
- EnCa (en-CA): British -our/-re/-ence, American -ize, tire, program.
Data: assets/spelling/en-variants.txt, our own ~280 American/British
pairs with a policy per English (MIT OR Apache-2.0, ATTRIBUTION.md row).
Inflections by rule (crates/proof/src/en_variants.rs). A dialect's form
is right when the shared list knows it or its other spelling, so
"realisation" is right in en-GB. No system dictionary needed.
Suggestions put the dialect's own form first. Review › Language lists
English (Australia) and (Canada); the Language dialog adds India,
Ireland, New Zealand and South Africa (names in every catalog, not
native-reviewed). An English interface on an en-GB/en-CA/en-AU… region
writes new documents in that English. Options › Language shows the
document's language instead of a fixed "English (United States)".
Tests: per-dialect accept/flag/suggest, tag mapping, no-panic proptests,
engine end to end through review.language, DOCX w:lang round trip,
region-based editing language, language menu.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8veLveuUsJJm6UEcQPEuW
* storytold/wordcraft#1281: review — pin the storytold/wordcraft#40 words (judgement, defence, licence, programme, catalogue) in en-GB and en-US
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WEhZXSxzS8orX7f9J3qpr
* storytold/wordcraft#1281: review — harness picks every proofing language in Review › Language and checks it is ticked
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WEhZXSxzS8orX7f9J3qpr
* storytold/wordcraft#1281: review — pin more British spellings in en-GB (goitre, sceptre, enrol, fulfil, instalment, skilful, wilful, encyclopaedia…)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WEhZXSxzS8orX7f9J3qpr
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: aiongg <aiongtaigi@gmail.com>
Review follow-up: the directives that name one flag read the same emoji
form, so they now skip the variation selector like PFX/SFX and stem flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RLA-ES, the Spanish Hunspell dictionaries in LibreOffice and most Linux
distributions, writes one flag as U+260E followed by U+FE0F, in the rules
(PFX ☎️ Y 3) and on stems (banco/S☎️). parse_aff rejected it as "not one
character", so the whole dictionary failed to load. The variation selector
is now dropped wherever flags are read, with a regression test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeRabbit (storytold/wordcraft#3252): reload_dictionary() wrote DICT and DICT_IN_MEMORY
under two separate locks. If install_dictionary() ran between them,
the final state could hold an in-memory dictionary with the flag left
false, so installed_path() would report a same-named disk file
instead of None.
Combined both into one DictState struct behind a single RwLock in
both ru and cs, so install_dictionary() and reload_dictionary() each
update the dictionary and its origin in one critical section.
Didn't add a dedicated concurrency test: genuinely proving a race's
absence needs real multi-threaded stress testing, and the previous
commit on this PR already found that tests directly exercising
install_dictionary()/reload_dictionary() concurrency in this crate's
shared global state cause real, reproducible flakiness in unrelated
tests. The fix is correct by construction (one lock acquisition around
both field writes) and all 222 existing proof tests, run 15 times in
a row, stay green.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CodeRabbit (storytold/wordcraft#3252): installed_path() searched the filesystem
unconditionally, so a dictionary given directly with install_dictionary
(a user-picked one, or a test fixture) was reported as the file of a
same-named dictionary that happened to also be installed, even though
spelling was actually using the in-memory one, not that file.
Added a DICT_IN_MEMORY flag to both modules, set in install_dictionary
and cleared in reload_dictionary; installed_path() now returns None
immediately when it's set, before even searching the filesystem.
Tried adding a dedicated test that calls install_dictionary() and
asserts installed_path() is None. It made dictionaries_folder_is_
searched_first flaky (~15-35% of runs across repeated local testing):
ru's and cs's own test modules already call install_dictionary()
directly in several tests of their own, sharing the same process-wide
static, and dictionaries_folder_is_searched_first's exact-path
assertion for Russian/Czech could observe whichever one last touched
it. Reverted the dedicated test; the production fix alone never
touches that shared state (it's a pure read-only check) and caused no
flakiness by itself across 25 repeated runs. Instead, narrowed
dictionaries_folder_is_searched_first's Russian/Czech assertion to the
language only, not the exact path, since unlike the other four
languages, installed_path() for these two is now sensitive to
concurrently-running tests' legitimate in-memory installs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CodeRabbit (storytold/wordcraft#3252): set_dictionaries_folder() reloaded every other
proofed language but not Russian or Czech, so after a folder change
installed_dictionaries() would report a path from the new folder
while spelling kept using the stale cached dictionary.
Also added Czech's test fixture to the shared dictionaries-folder
test's temp folder (the same content czech_proofing_through_the_
review_commands installs directly): since that folder-change test
now reloads Czech too, it was clearing a concurrently running Czech
test's installed dictionary with nothing to replace it, intermittently
emptying it mid-test. Same fixture content either way keeps the race
harmless, exactly as it already does for Polish/Portuguese/Indonesian.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
installed_dictionaries() never included Russian or Czech at all,
because neither crates/proof/src/ru nor crates/proof/src/cs had an
installed_path() function for it to call (every other proofed
Hunspell language — Polish, Portuguese, Italian, Croatian, Turkish,
Indonesian — already did). File > Options > Proofing's dictionaries
list iterates installed_dictionaries() directly, so Russian and Czech
never showed up there at all, installed or not, matching the report.
Added installed_path() to both modules, mirroring the existing
pattern (crate::hunspell::find(NAMES).map(|(_, dic)| dic)), and added
both to installed_dictionaries().
The actual spell-checking dispatch (is_correct_in_with) already
routed Russian and Czech correctly — confirmed by installing the real
system ru_RU Hunspell dictionary and checking it directly
(ru::is_correct("молоко") true, ru::is_correct("малоко") false), so
that part of the pipeline isn't what's broken. The reporter's other
symptom (Russian text in the document not being checked at all) did
not reproduce with the real dictionary installed; I could not pin
that part down further without more specific repro details from
their setup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Proofing: say when a dictionary is installed but unreadable, with file and reason (storytold/wordcraft#2830)
* proof test: use SHIFT_JIS as the unsupported encoding (ISO 8859-1 is decoded since storytold/wordcraft#2904)
---------
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Skaywalker
Dylan M. Taylor
Martin BallandClaude Opus 5.5
skylar
VoiceLessQ
aiongg
aiongg
Federico Olivares
dotCooCoo
NetterSilverandClaude Haiku 5.5
dmim2502-commitsandClaude Opus 5.5
MichaelandClaude Sonnet 5
MarekSpodBiedryandBrandon Thomas