Checked against the Hunspell 1.7 command line:
- Without a tab or a morphological field the whole line is the entry, so
a phrase with single-byte flags (tök jó/396 in hu_HU) no longer fails
the load or loses its alias; leading blanks belong to the entry.
- With an AF table a flag field is the alias its leading digits number
(fal/2x is alias 2, lab/b has none); an undefined alias, in a .dic
entry or a continuation class, reads as no flags instead of failing
the dictionary; a table without its count line is ignored.
- A REP count line may carry a comment, like AF and ICONV.
normalize copies the input only from its first change on, and SET and
FLAG are found in one pass over the .aff.
Checked against the Hunspell 1.7 command line:
- The first AF and ICONV line is the count even with a trailing comment
(AF 2 # aliases), so alias numbers no longer shift and ICONV 1 # no
longer turns 1 into #.
- A .dic entry ends where Hunspell ends it (dic_entry_end), so ad hoc/B
is one entry and ad alone is no word.
- An undefined alias in a .dic entry reads as no flags: the stem loads
without affixes instead of failing the whole dictionary.
- IGNORE characters are removed from the appended text only; a stripped
text that holds one never matches a stem, as in Hunspell.
- FLAG UTF-8 and SET are found anywhere in the .aff, not only in the
first 8 KB.
Also: normalize borrows the input when no conversion or ignored character
applies, the UTF-8 .aff reader counts fields as it goes and builds its
error context only on error, and the single-byte flag reader is
decode_latin1.
Without FLAG UTF-8, Hunspell reads a field that names one flag (PFX/SFX,
KEEPCASE, FORBIDDENWORD and the like) through decode_flag, which takes its
first byte: SFX é is the flag 0xC3. Such a field no longer fails the load.
SYLLABLENUM lists flags and is read byte by byte like AF. SET and FLAG
read only their first argument, so FLAG UTF-8 with a trailing comment
keeps Unicode flags.
The Hungarian Hunspell dictionary declares SET UTF-8 but writes its flags as single bytes, uses AF aliases and 158 distinct flags. Read flag fields byte by byte unless FLAG UTF-8 is set, expand AF aliases, skip AM, apply ICONV and IGNORE, honour KEEPCASE, and widen flag masks to 256 bits. Compounding and NEEDAFFIX stay unsupported for now.
- proof: pass PoisonError::into_inner instead of an identity closure (S1612).
- layout tests: replace glob super::* imports with explicit names (S2208).
- packaging/macos/import-cert.sh: use [[ for the status test (S7688).
Wall-clock bounds of 1-2 s failed on loaded runners. Each bound is now 10 s, at least 10x the
fixed code's time and well under the old code's on the same input (measured by running the
pre-fix code):
- self-continuing rules: 40,000 rules (was 16,000); ~1 ms now, old checker 14 s at 16,000.
- many strips per suffix: 200,000 rules (was 30,000); 0.4 s now, old index 3 s at 30,000 and
11 min at 300,000.
- forms of self-continuing rules: the MAX_FORMS count catches the old code (9 million forms);
the time bound is a backstop.
- overlong Turkish words: ~20 ms now; without the caps it ran past 5 minutes.
- capped Hunspell reads: no clock; the "not a regular file" error is the proof (the old reader
never returned from /dev/zero).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CX1hMKX28764Lrx5moxs1o
Asked to confirm a non-UTF-8 Turkish dictionary still loads: before this PR
tr_TR was read with read_to_string, so a non-UTF-8 file never loaded; now it
goes through read_capped and decode, which didn't know ISO 8859-9 (Latin-5,
the encoding of older tr_TR dictionaries) and refused it as unsupported.
decode now reads `SET ISO8859-9`, and a test writes an ISO 8859-9 .aff/.dic
to disk and finds words with ğ ı ş İ ç ö ü, as stems and with suffixes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CX1hMKX28764Lrx5moxs1o
Audit finding 20.
- tr::suggest_with refuses words over 64 letters (and words with spaces), like the other
languages' suggesters (40 there; Turkish agglutinated words run past 40). Each candidate edit
copies the whole word, so a long word cost O(n^2) per suggestion request on the UI thread.
- TrDict::contains returns false past MAX_WORD_BYTES (120, the affix engine's bound) instead of
collecting the char indices of the whole word.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Audit findings 10 (storytold/wordcraft#3657) and 15 (storytold/wordcraft#3692).
- ru::hyphen::points: the cuts come in increasing order, so a repeat can only be the last one;
compare with `out.last()` instead of `out.contains` per syllable (O(V^2) on a long Cyrillic
word, during layout).
- hyphenate_word_in: walk the sorted break points alongside the chars instead of searching the
point list for every char. Same output.
Test: russian_hyphenation_and_hyphenated_output_are_linear (8x the letters costs ~8x; the old
code measured 31x and 30x).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Audit finding 4, and the rest of storytold/wordcraft#3661.
- dict.rs: `Dictionary::from_bytes` inflates with `decompress_to_vec_with_limit` (64 MiB; the
bundled en-US one is ~1 MB) and caps the words it keeps (bytes and count): front coding lets a
2-byte line spell out a 255-byte word, so the inflated size alone doesn't bound storage.
- hunspell.rs: one `read_capped` reader — regular files only, size checked on the open handle and
the read itself bounded by `take(MAX_FILE + 1)`, so no TOCTOU between metadata and read and no
endless read from a FIFO or device.
- tr: the Turkish loader reads through `read_capped` and decodes with `hunspell::decode` (it
read both files whole, with no cap at all).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Part of storytold/wordcraft#3613 (audit findings 2 and 6).
- Checker: `suffixed` and `form_has_flag` OR the flags of a group's matching rules into one
mask and recurse once, not once per rule. Same answer (the walk is a disjunction over flags);
16k self-continuing empty-append rules: 6 s -> <1 ms per check (debug).
- Finding 2: `forms` expands each (form, continuation) pair once, walks continuation rules through
a per-flag rule list, and stops at MAX_FORMS (4096) forms / MAX_FORMS_WORK rule visits, so a
dictionary of self-continuing rules can't allocate R^2 strings per stem.
- Finding 6: the suffix index finds a rule's (append, strip) group by hash, not by scanning the
append's groups: building with many strips per append is linear.
Not done here (rest of storytold/wordcraft#3613): a work budget for distinct-strip chains in the checker, rule caps
in parse_aff, proofing off the UI thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It came in with the first proofing commit (bcbe09df8) and has never had a caller; `#[allow(dead_code)]` kept the warning quiet. Because it is `pub(crate)`, nothing outside wordcraft-proof can reach it.
SonarCloud's analysis of main at b4da59f0c lists 23 closures such as `|v| v.as_slice()` that hand their argument straight to one method. This passes the method itself: `Vec::as_slice`, `str::to_ascii_lowercase`, `f64::total_cmp` and the like, all forms the repo already uses elsewhere. Behaviour is unchanged.
Five stay. The four in crates/proof/src/es/mod.rs sit in code that storytold/wordcraft#4191 replaces, and its replacement already uses `PoisonError::into_inner`. In crates/engine/src/io/safe_save.rs the path would be `<dyn std::error::Error + Send + Sync>::is::<Guided>`, which reads worse than `|inner| inner.is::<Guided>()`.
The spelling check found the start of the white-space-delimited token around
a word one byte after the last white-space character. After a no-break space
(two bytes in UTF-8) or an ideographic space (three bytes) that offset is
inside the character, so the token could not be read and was never classified
as a web or e-mail address. With Ignore web and file addresses on, the words
of such an address were still marked as misspelled.
The token now starts after the whole white-space character. A proofing test
covers a web address, a www. address and an e-mail address after the two
spaces, with the option on and off.
* chore: resolve SonarCloud maintainability issues
* fix: restore compiling refactors, revert behaviour changes, run cargo fmt
- fontdb::norm: keep the closure for filter (char::is_alphanumeric takes
self, not &self, so it does not fit Iterator::filter).
- story.rs: write_mark_rpr needs &mut self (rev_attrs, rpr_change).
- Revert the viewport meta change and the return 0 added to
workspace_version/sha256, which changed behaviour or masked failures.
- cargo fmt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Import Document in odt page_parts after dropping glob re-export
* fix: apply_kind_properties binds ContentControl; cell_ref keeps char closures
* fix(engine): pass case helpers by value (clippy needless borrows)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Dylan Taylor <dylan@craftmaker.dev>
Each of these languages accepted every word in capitals of up to six letters and every word
with a digit on its own, so turning off Ignore words in UPPERCASE or Ignore words that contain
numbers had no effect on them for short words. The check now takes the options, as the Spanish
one does; the direct is_correct/is_correct_with keep the defaults.
Co-authored-by: Federico Olivares <FedeOlivares@users.noreply.github.com>
Co-authored-by: aiongg <67543387+aiongg@users.noreply.github.com>
Maintainer review on storytold/wordcraft#3562: opening_marks reverse-scans the characters before a
question or exclamation mark, and ends_clause asked in_address for each one,
which scanned every address range, so a paragraph of thousands of addresses
followed by a question took quadratic time (about 99 KB: hundreds of ms).
address_at() finds the range a byte is in with a binary search over the
sorted, disjoint ranges, and also gives ends_clause the range's end without
another scan. Behaviour is unchanged (the URL and punctuation regressions
still pass).
- address_lookup_finds_the_range_a_byte_is_in: boundaries of the lookup
- many_addresses_before_a_question_stay_linear: 99 KB of addresses before a
question or exclamation (the review's reproducer and two variants), checked
for the right issue range and for taking under 75 ms (about 1 to 5 ms now,
about 300 ms with a scan per character)
Upstream merged storytold/wordcraft#3177 (dictionary hints for every Hunspell language) and storytold/wordcraft#3480.
Resolve the conflicts in crates/proof/src/lib.rs by taking storytold/wordcraft#3177's
dictionary_loaded/dictionary_needed structure and adding the four Spanish
varieties to it, and in docs/spelling-dictionaries.md keeping both sides.
Maintainer review on storytold/wordcraft#3562: spelling looks words up composed (NFC), but
add_word, remove_word, ignore_word, custom_word (so loading the dictionary
file) and save_user_dictionary's removed list kept the raw lower-cased
spelling. A word added or ignored decomposed (`zqprue\u{301}ba`) therefore
stayed flagged.
- dictionary_key(): lower case, then composed; every write uses it, and
in_user_dictionary() looks up composed, so either form adds, ignores,
removes and finds the same entry; NFD entries in an existing file load
and save composed
- English's known_core() reuses in_user_dictionary() instead of a copy of it
- issue byte ranges still cover the text as typed
- regressions: proof (add, Ignore All, remove, save/load with an NFD entry,
both forms rechecked) and engine (review.addToDictionary and
review.ignoreAll with the issue's NFD text)
- es/mod.rs: one variety() match gives the dictionary slot, log name and file
names (variety_name() folded in); load_installed() destructures it
- es/grammar.rs: the sentence-start rule for the missing ¿/¡ is one named
function, ends_clause(), replacing is_boundary() and three closures
rebuilt per mark; in_address() is shared; repeated_words() builds its
helpers once instead of per word pair
- es/suggest.rs reuses the Czech ranking's MAX_LETTERS instead of a copy
Same tests pass unchanged; a temporary old-vs-new differential proptest
(20,000+ generated inputs with URLs and punctuation) found identical output.
Maintainer review on storytold/wordcraft#3562:
- words() keeps combining marks with the word they follow, so a word typed
decomposed (NFD, `tambie\u{301}n`) is one word with its own byte range;
is_correct_in_with and suggest_in look it up composed (NFC), so it is
checked like `también` in every language (unicode-normalization, MIT/Apache)
- the Spanish missing-¿/¡ rule skips marks in web and e-mail addresses
(`Ver https://example.com/?`), and punctuation inside an address or a word
(`example.com`, `https:`, `10:30`) no longer starts the sentence the
opening mark is offered for; a mark ending an address still ends a sentence
- Spanish repeated-word check matches composed and decomposed spellings
- regressions: check_text with the Spanish fixture and NFC/NFD equivalents
(acceptance and ranges), words() and composed(), URLs ending in /? beside
ordinary questions
CodeRabbit review: Session::region_spanish takes the ISO 639-2 `spa` as
Spanish, so a `spa-MX` region writes es-MX documents.
CodeRabbit review on storytold/wordcraft#3562:
- with Ignore words in UPPERCASE off, short words in capitals are checked
against the Spanish dictionary instead of accepted as acronyms; likewise
words with digits with Ignore words that contain numbers off. Direct
es::is_correct/is_correct_with keep the defaults.
- Latin American Spanish (which es-AR resolves to) also looks for es_AR and
es-AR, after the neutral es_US and es_MX, so Argentine text is checked when
only that dictionary is installed
- docs: the exact candidate order per variety in localization-parity.md, and
es_AR in the spelling-dictionaries table
- the Spain variant's doc lists Uruguay among the regions it doesn't cover
Resolve conflicts with German hyphenation (ProofLang::De), Russian and Czech
in installed_dictionaries(), and storytold/wordcraft#2164's U+FE0F flag fix: affix.rs takes
main's fix and drops this branch's duplicate. The Danish, Dutch, Romanian and
Slovenian catalogs gain the Spanish (Latin America) and (Uruguay) names; the
language menu tests get room for the four Spanish entries.
Spanish text was never checked: ProofLang had no Spanish variant, so every
es-* run fell through to Other (no marks). And the RLA-ES dictionaries
LibreOffice ships were refused by the affix engine for their `☎️` flag
(☎ + variation selector U+FE0F), which Hunspell reads as ☎.
- ProofLang::Es (Spain, plain es, spa), EsMx (es-MX), Es419 (es-419 and
the other countries of the Americas) and EsUy (es-UY), chosen by region
like the English dialects
- each variety reads its own installed Hunspell dictionary with fallbacks
(es_ES; es_MX; es_419/es_US/es_MX; es_UY/es_AR); nothing bundled
- variation selectors in affix flag fields belong to the flag before them
- Uruguayan vos forms accepted when the dictionary has the infinitive:
imperatives (vení, andá), with pronouns (decime, decímelo), present
(tenés); a vos form typed without its accent gets it back first
- suggestions use the dictionary-driven Czech ranking (prueva → prueba,
corector → corrector, tambien → también)
- grammar: repeated word, spacing, missing opening ¿/¡, space after an
opening mark, with Spanish messages
- Review › Language, status bar, Language dialog and File › Options ›
Proofing list the four varieties; the two new names translated in every
catalog; a Spanish interface writes new documents in the region's Spanish
- the dictionaries-folder engine test holds the Italian and Spanish
fixtures, so parallel proofing tests no longer lose their dictionaries
- tests use hand-written synthetic dictionaries; engine end-to-end tests
crates/proof/src/lang.rs: main's English-dialect helpers kept; code() doc lists `de`.
crates/proof/src/lib.rs: hostile-input test loops over main's languages plus De.
crates/ui-egui/src/options.rs: language_name maps EnGb/EnCa and De to no dictionary name.
ROADMAP.md: main's status lines; both progress-log and revision rows kept.
docs/gaps.md: main's status line; gap 8 row is main's with German hyphenation added; revision rows unioned.
docs/layout-parity.md: main's status line, Tabs and Drop caps rows; the PR's Hyphenation row; revision rows unioned.
docs/localization-parity.md: main's status line; revision rows unioned.
Skaywalker
Dylan M. Taylor
Martin BallandClaude Opus 5.5
skylar
VoiceLessQ
aiongg
aiongg
Federico Olivares
dotCooCoo
NetterSilverandClaude Haiku 5.5