diff --git a/.github/workflows/proof.yml b/.github/workflows/proof.yml index 8b119ac2..347793b6 100644 --- a/.github/workflows/proof.yml +++ b/.github/workflows/proof.yml @@ -25,10 +25,12 @@ jobs: with: persist-credentials: false # The Language data (craft-i18n `language-data/`), embedded by crates/proof/build.rs. + # Pinned to storytold/craft-i18n#114's head (#109's SCOWL dictionaries + judgment/judgement, + # CP2.1); TODO(CP2.1): move to #114's merge commit once it merges. - uses: actions/checkout@v4 with: repository: storytold/craft-i18n - ref: 062a6375c322c88b96691a17f7b5fed71c51f7a8 + ref: c239f934ace90b7939afaceb23b472ceeafd6b8a path: craft-i18n persist-credentials: false - uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable diff --git a/crates/proof/README.md b/crates/proof/README.md index 194c3797..5d38164a 100644 --- a/crates/proof/README.md +++ b/crates/proof/README.md @@ -13,7 +13,7 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = " | Language | Spelling | Grammar | Hyphenation | |---|---|---|---| -| English (en-US, en-GB and the British-spelling Englishes, en-CA) | Moby word list + `en-extra.txt`, dialect variants (`en-variants.txt`); without them the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic | +| English (en-US, en-CA, en-GB and the British-spelling Englishes such as en-AU, en-NZ) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic | | Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns | | Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise | | German | — | — | TeX patterns | @@ -63,20 +63,41 @@ data folder runs neither set. English spelling uses the first of these that is there; `english_source(lang)` says which. -1. **Embedded:** the Moby word list from `CRAFT_I18N_DIR` (above). Nothing below is used then. -2. **System Hunspell:** the run's own English `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`; - each name in the dictionaries folder and then the system locations before the next name). - Never another English's: an `en_US` dictionary would mark British spellings, so en-GB and - en-CA without their own use the built-in list, which has both spellings. A file that can't be - read is skipped for the next name or folder; one file is read once for every English that uses - it. Compounding (`COMPOUNDMIN`, `COMPOUNDRULE`…, used by English dictionaries only for - ordinals) and `OCONV` are ignored for English rather than refused. None on the web. -3. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB +1. **Dictionaries folder:** the English's own `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`) + that the user put in the folder set with `set_dictionaries_folder`, in `WORDCRAFT_DICTIONARIES` + or in WordCraft's own `dictionaries` folder. The user's choice comes first. +2. **Installed English pack:** reserved for Language packs (craft-libs#485, CP4.2). +3. **Embedded SCOWL:** the English's own SCOWL `rel-2026.02.25` Hunspell dictionary from + `CRAFT_I18N_DIR` (`language-data/en-us`, `en-gb` (British `-ise`) or `en-ca`, `spelling.aff` + + `spelling.dic`), read by the affix engine. en-AU, en-NZ, en-IE and the other British-spelling + Englishes read en-GB; Canadian English reads en-CA. A word is right when the English's own + dictionary knows it (or `en-extra.txt` does), never because another English's does + (`colourblind` is wrong in en-US, `colorblind` in en-GB); suggestions come from the same + dictionary. SCOWL's size-60 lists keep one spelling per dialect, and Word accepts more, so + British English also takes the `-ize` spellings of its `-ise` words that American or Canadian + English spell so (`organize`, `realization`; never `wize` or `advize`), and `en-variants.txt` flags the other dialect's spellings and accepts this + dialect's other spellings of known words (`judgement` in en-US). English with no dialect + (`is_correct`) takes a word any of the three knows. Suggestions offer a missing space only + between two lower-case words of three letters or more, after every one-word candidate, and no + name or abbreviation (`Col`) for a lower-case word. With SCOWL embedded the system locations aren't + searched. +4. **System Hunspell:** the English's own pair in the system locations. Never another English's: + an `en_US` dictionary would mark British spellings, so en-GB and en-CA without their own use + the built-in list, which has both spellings. Each name is looked for in every folder before the + next name; a file that can't be read is skipped for the next name or folder; one file is read + once for every English that uses it. `COMPOUNDRULE`, `ONLYINCOMPOUND` and `COMPOUNDMIN` + (ordinals such as `21st`) are read; flag-based compounding (`COMPOUNDFLAG`…) and `OCONV` are + ignored for English rather than refused. None on the web. +5. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB compressed, words only) built into the crate, with `data/en-extra.txt` (WordCraft's supplement - of modern words, a copy of craft-i18n's) and the same inflection rules as the embedded list. A - word it doesn't know is misspelled. Suggestions are the words one edit away (swap, missing, - added or changed letter), looked up in the list, for words of up to 32 letters: swaps first, - then common words, neighbouring keys, the same first letter. + of modern words, a copy of craft-i18n's) and the inflection rules. A word it doesn't know is + misspelled. Suggestions are the words one edit away (swap, missing, added or changed letter), + looked up in the list, for words of up to 32 letters: swaps first, then common words, + neighbouring keys, the same first letter. + +The embedded Moby list (`language-data/en-us/en-us.dic`) is no longer a spelling source: it is +the same list as the built-in one, and stays as the hyphenation word list. `en-variants.txt` +applies to every source above when it is embedded. A dictionary that is there but can't be read (encoding, size, parse error, no words) is skipped, and `dictionary_load_error(lang)` gives the reason; it never panics. Text in other languages is @@ -91,6 +112,12 @@ Regenerate with `data/en-extra.txt` is WordCraft contributors' own list (MIT OR Apache-2.0); keep it in step with craft-i18n's copy. +The SCOWL notice (SCOWL's licence, the Ispell BSD licence of the affix files and Benjamin Titze's +notice) is embedded with the dictionaries: `craft_proof::scowl::notice()`, for an About window. + +Embedding SCOWL adds 1,542,396 bytes of Language data (the en-US, en-GB and en-CA `.aff`/`.dic` +and the notice); the Moby list (432,905 bytes) stays for hyphenation. + ## Check ```sh diff --git a/crates/proof/build.rs b/crates/proof/build.rs index a0440624..29b45bf0 100644 --- a/crates/proof/build.rs +++ b/crates/proof/build.rs @@ -14,6 +14,10 @@ //! //! `include_str!` takes a string, so a `CRAFT_I18N_DIR` that isn't valid UTF-8 can't be embedded: //! it is treated like a folder without the data. +//! +//! English: SCOWL's en-US, en-GB (-ise) and en-CA dictionaries; en-AU is not embedded, as +//! Australian English is checked as British-spelling English (`ProofLang::EnGb`). The SCOWL +//! notice is embedded once (the four upstream READMEs differ only in their title and build line). use std::fmt::Write as _; use std::path::{Path, PathBuf}; @@ -21,6 +25,13 @@ use std::path::{Path, PathBuf}; /// (constant, path under `language-data/`, bytes rather than text). const FILES: &[(&str, &str, bool)] = &[ ("EN_US_DIC", "en-us/en-us.dic", true), + ("EN_US_AFF", "en-us/spelling.aff", false), + ("EN_US_SPELLING", "en-us/spelling.dic", false), + ("EN_GB_AFF", "en-gb/spelling.aff", false), + ("EN_GB_SPELLING", "en-gb/spelling.dic", false), + ("EN_CA_AFF", "en-ca/spelling.aff", false), + ("EN_CA_SPELLING", "en-ca/spelling.dic", false), + ("SCOWL_NOTICE", "en-us/LICENCE-SCOWL.txt", false), ("EN_US_PAT", "en-us/hyphenation.pat", false), ("EXTRA_EN", "en-us/en-extra.txt", false), ("EN_VARIANTS", "en-us/en-variants.txt", false), diff --git a/crates/proof/src/affix.rs b/crates/proof/src/affix.rs index 701e168c..06433963 100644 --- a/crates/proof/src/affix.rs +++ b/crates/proof/src/affix.rs @@ -17,11 +17,12 @@ //! (at most two suffixes in a row, as in Hunspell), `FORBIDDENWORD`, `NOSUGGEST`, `KEEPCASE` (the //! stem is a word only in its own letter case), input conversion (`ICONV`: ligatures such as `fi` //! read as `fi`) and ignored characters (`IGNORE`), and the suggestion tables `TRY`, `KEY`, `MAP` -//! and `REP`. -//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: compounding, -//! long or numeric flags, `NEEDAFFIX` and the other directives that change which words are correct -//! (an English dictionary's compounding and `OCONV` are ignored instead: -//! [`AffixDict::from_hunspell_english`]). +//! and `REP`. Compounds by `COMPOUNDRULE` (patterns of one-character flags with `*` and `?`, as +//! SCOWL's English dictionaries use for ordinals: `21st`, `112th`), with `ONLYINCOMPOUND` and +//! `COMPOUNDMIN`; the parts of such a compound are stems, without affixes, as in Hunspell. +//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: flag-based +//! compounding (`COMPOUNDFLAG`, `COMPOUNDBEGIN`…), long or numeric flags, `NEEDAFFIX` and the +//! other directives that change which words are correct. //! //! Lookups never panic: stems sit in one string, found through an open-addressing hash table, and //! a probe allocates only small scratch strings for the candidate roots it tries. @@ -48,16 +49,54 @@ const UNSUPPORTED: &[&str] = &[ "COMPOUNDFLAG", "COMPOUNDLAST", "COMPOUNDMIDDLE", - "COMPOUNDMIN", "COMPOUNDPERMITFLAG", - "COMPOUNDRULE", "FULLSTRIP", "NEEDAFFIX", "OCONV", - "ONLYINCOMPOUND", "PSEUDOROOT", ]; +/// Most `COMPOUNDRULE` patterns kept, and most tokens in one: a hostile `.aff` can't make a +/// compound check cost more than a few thousand stem lookups per word. +const MAX_COMPOUND_RULES: usize = 64; +const MAX_COMPOUND_TOKENS: usize = 32; +/// Longest word (in bytes) tried as a compound: each try looks up every substring. +const MAX_COMPOUND_BYTES: usize = 48; + +/// How often a `COMPOUNDRULE` token's flag may repeat. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Repeat { + Once, + /// `?` + Optional, + /// `*` + Any, +} + +/// A `COMPOUNDRULE` pattern as written (`n*1t`): its tokens, each a flag and how often it repeats. +fn parse_compound_rule(pattern: &str) -> Result, String> { + let mut out: Vec<(char, Repeat)> = Vec::new(); + for c in pattern.chars().filter(|&c| c != EMOJI_SELECTOR) { + match c { + '*' | '?' => { + let Some(last) = out.last_mut().filter(|t| t.1 == Repeat::Once) else { + return Err(format!("COMPOUNDRULE `{pattern}`: `{c}` follows no flag")); + }; + last.1 = if c == '*' { Repeat::Any } else { Repeat::Optional }; + } + '(' | ')' => return Err(format!("COMPOUNDRULE `{pattern}`: ( ) groups (long flags) are not supported")), + c => out.push((c, Repeat::Once)), + } + if out.len() > MAX_COMPOUND_TOKENS { + return Err(format!("COMPOUNDRULE `{pattern}`: more than {MAX_COMPOUND_TOKENS} flags")); + } + } + if out.is_empty() { + return Err("COMPOUNDRULE with no flags".into()); + } + Ok(out) +} + /// Most distinct flags a dictionary may use: every single-byte flag (the Hungarian dictionary /// uses 158). const MAX_FLAGS: usize = 256; @@ -272,6 +311,18 @@ pub struct AffixDict { ignore: Box<[char]>, /// Provenance and licence lines, kept through [`AffixDict::to_bytes`]. comments: Vec, + /// `COMPOUNDRULE` patterns: per token, the flag's bit and how often it may repeat. + compound_rules: Vec>, + /// The same patterns as written, for [`AffixDict::to_bytes`]. + compound_text: Vec, + /// The `ONLYINCOMPOUND` flag's bit: such stems are words only as parts of a compound. + only_in_compound: Mask, + /// `COMPOUNDMIN`, in characters. + compound_min: usize, + /// The first characters of the stems a compound part can be (their flags are in a + /// `COMPOUNDRULE`), so a word starting otherwise is no compound: SCOWL's ordinals start with + /// a digit, and a misspelled word costs one lookup here instead of one per cut. + compound_firsts: Vec, } fn hash_str(s: &str) -> u64 { @@ -341,10 +392,14 @@ struct Rules { substandard: Option, /// The `KEEPCASE` flag. keepcase: Option, - /// The `ONLYINCOMPOUND` flag, read for English only ([`AffixDict::from_hunspell_english`]). - only_in_compound: Option, iconv: Vec<(String, String)>, ignore: Vec, + /// `COMPOUNDRULE` patterns, as written (`n*1t`). + compound_rules: Vec, + /// The `ONLYINCOMPOUND` flag. + only_in_compound: Option, + /// `COMPOUNDMIN`: the shortest part of a compound, in characters (Hunspell's default is 3). + compound_min: Option, /// `AF` flag aliases: alias `n` (1-based) stands for `aliases[n - 1]`. aliases: Vec, /// The `AF` table opened with its count line, so flag fields are alias numbers. A table @@ -426,26 +481,15 @@ fn zero_empty(s: &str) -> String { /// but it is not a flag of its own: it is dropped wherever flags are read. const EMOJI_SELECTOR: char = '\u{fe0f}'; -/// Directives an English dictionary may carry that don't change which English words are correct -/// here, ignored by [`AffixDict::from_hunspell_english`]: compounding (English dictionaries use -/// it only for ordinals such as `1st`, which have digits and are skipped by the spelling check) -/// and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`). -const IGNORED_IN_ENGLISH: &[&str] = &[ - "COMPOUNDBEGIN", - "COMPOUNDEND", - "COMPOUNDFIRST", - "COMPOUNDFLAG", - "COMPOUNDLAST", - "COMPOUNDMIDDLE", - "COMPOUNDMIN", - "COMPOUNDPERMITFLAG", - "COMPOUNDRULE", - "OCONV", -]; +/// Directives an English dictionary may carry that this engine doesn't implement but that don't +/// change which English words are right, ignored by [`AffixDict::from_hunspell_english`]: +/// flag-based compounding (English dictionaries compound only ordinals, by `COMPOUNDRULE`, which +/// is read) and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`). +const IGNORED_IN_ENGLISH: &[&str] = + &["COMPOUNDBEGIN", "COMPOUNDEND", "COMPOUNDFIRST", "COMPOUNDFLAG", "COMPOUNDLAST", "COMPOUNDMIDDLE", "COMPOUNDPERMITFLAG", "OCONV"]; /// Read Hunspell `.aff` text (already decoded to UTF-8). `english`: ignore the directives in -/// [`IGNORED_IN_ENGLISH`], and keep the `ONLYINCOMPOUND` flag so the stems that are only parts of -/// compounds can be left out. +/// [`IGNORED_IN_ENGLISH`] instead of refusing them. fn parse_aff(aff: &str, english: bool) -> Result { let mut r = Rules::default(); let mut cross: HashMap<(bool, char), bool> = HashMap::new(); @@ -453,6 +497,7 @@ fn parse_aff(aff: &str, english: bool) -> Result { let mut rep_header = false; let mut af_header = false; let mut iconv_header = false; + let mut compound_header = false; for (n, line) in aff.lines().enumerate() { let f: Vec<&str> = split_fields(line).collect(); let Some(&dir) = f.first() else { continue }; @@ -487,6 +532,23 @@ fn parse_aff(aff: &str, english: bool) -> Result { } "IGNORE" => r.ignore = at(1).chars().collect(), "KEEPCASE" => r.keepcase = Some(one_flag(at(1), dir, n)?), + // The first `COMPOUNDRULE` line is the table's count, as in Hunspell. + "COMPOUNDRULE" => { + if !compound_header && at(1).parse::().is_ok() { + compound_header = true; + } else { + parse_compound_rule(at(1)).map_err(|e| format!("line {}: {e}", n + 1))?; + if r.compound_rules.len() >= MAX_COMPOUND_RULES { + return Err(format!("line {}: more than {MAX_COMPOUND_RULES} COMPOUNDRULE patterns", n + 1)); + } + r.compound_rules.push(at(1).to_string()); + } + } + "ONLYINCOMPOUND" => r.only_in_compound = Some(one_flag(at(1), dir, n)?), + "COMPOUNDMIN" => { + let min = at(1).parse::().map_err(|_| format!("line {}: COMPOUNDMIN needs a number", n + 1))?; + r.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES)); + } // Per-character flag modes (`FLAG UTF-8`, as VERO's pt_BR dictionary uses) work // with the one-character parsing here; only the multi-character encodings differ. "FLAG" if !matches!(at(1), "UTF-8" | "DEFAULT" | "utf-8" | "default" | "UTF8" | "") => { @@ -541,7 +603,6 @@ fn parse_aff(aff: &str, english: bool) -> Result { r.prefixes.push(rule); } } - "ONLYINCOMPOUND" if english => r.only_in_compound = Some(one_flag(at(1), dir, n)?), d if english && IGNORED_IN_ENGLISH.contains(&d) => {} d if UNSUPPORTED.contains(&d) => return Err(format!("line {}: {d} is not supported", n + 1)), _ => {} @@ -570,10 +631,10 @@ impl AffixDict { AffixDict::from_hunspell_in(aff, dic, comments, false) } - /// [`AffixDict::from_hunspell`] for an English dictionary: compounding and output conversion - /// (`COMPOUNDMIN`, `COMPOUNDRULE`, `OCONV`…), which other languages need but English spelling - /// doesn't, are ignored rather than refused, and the stems marked `ONLYINCOMPOUND` (parts of - /// compounds only) are left out. + /// [`AffixDict::from_hunspell`] for an English dictionary: flag-based compounding and output + /// conversion (`COMPOUNDFLAG`, `OCONV`…), which this engine doesn't implement and English + /// spelling doesn't need, are ignored rather than refused. `COMPOUNDRULE`, `ONLYINCOMPOUND` + /// and `COMPOUNDMIN` (ordinals: `21st`) are read, as for every language. pub fn from_hunspell_english(aff: &str, dic: &str, comments: &[&str]) -> Result { AffixDict::from_hunspell_in(aff, dic, comments, true) } @@ -619,9 +680,6 @@ impl AffixDict { continue; } let flags = rules.flags(&flags).to_string(); - if rules.only_in_compound.is_some_and(|f| flags.contains(f)) { - continue; - } stems.push((word, flags)); } AffixDict::build(rules, stems, comments.iter().map(ToString::to_string).collect()) @@ -631,6 +689,7 @@ impl AffixDict { // Flags: every one a rule or a stem uses gets a bit. let mut flag_chars: Vec = rules.prefixes.iter().chain(&rules.suffixes).map(|r| r.0).collect(); flag_chars.extend(stems.iter().flat_map(|s| s.1.chars())); + // A compound rule's flag no stem carries gets no bit: its token matches no part. flag_chars.sort_unstable(); flag_chars.dedup(); if flag_chars.len() > MAX_FLAGS { @@ -656,9 +715,18 @@ impl AffixDict { map: rules.map, rep: rules.rep, comments, + compound_rules: rules + .compound_rules + .iter() + .map(|p| parse_compound_rule(p).map(|t| t.into_iter().map(|(c, rep)| (bit_of(c), rep)).collect())) + .collect::>()?, + compound_text: rules.compound_rules, + only_in_compound: rules.only_in_compound.map_or(Mask::EMPTY, bit_of), + compound_min: rules.compound_min.unwrap_or(3), ..AffixDict::default() }; d.offs.push(0); + let compound_flags = d.compound_rules.iter().flatten().fold(Mask::EMPTY, |m, (bit, _)| m | *bit); let mut set_ids: HashMap = HashMap::new(); let mut i = 0; while i < stems.len() { @@ -676,6 +744,12 @@ impl AffixDict { if usize::from(id) == d.sets.len() { d.sets.push(mask); } + if mask.meets(compound_flags) + && let Some(c) = word.chars().next() + && !d.compound_firsts.contains(&c) + { + d.compound_firsts.push(c); + } d.words.push_str(word); d.offs.push(u32::try_from(d.words.len()).map_err(|_| "stems too large".to_string())?); d.flags.push(id); @@ -797,6 +871,11 @@ impl AffixDict { /// Flag mask of a stem, if `w` is one and carries none of the flags in `deny`. fn stem_flags_except(&self, w: &str, deny: Mask) -> Option { + self.stem_mask(w).filter(|m| !m.meets(self.forbidden | self.only_in_compound | deny)) + } + + /// Flag mask of the stem `w`, whatever flags it carries. + fn stem_mask(&self, w: &str) -> Option { if self.table.is_empty() { return None; } @@ -810,9 +889,9 @@ impl AffixDict { if self.word(t as usize) == Some(w) { let id = *self.flags.get(t as usize)?; let mask = self.sets.get(usize::from(id)).copied(); - // A stem carrying FORBIDDENWORD is listed but is not a word: no direct hit, - // and (this is the only stem lookup) no affixed form is built on it either. - return mask.filter(|m| !m.meets(self.forbidden | deny)); + // [`Self::stem_flags_except`] drops FORBIDDENWORD stems (listed, not words) and + // ONLYINCOMPOUND ones (words only inside a compound, [`Self::compound`]). + return mask; } slot = (slot + 1) & mask; } @@ -928,7 +1007,41 @@ impl AffixDict { return true; } } - false + self.compound(w, deny) + } + + /// Is `w` a compound of two or more stems (no affixes) whose flags spell out one of the + /// `COMPOUNDRULE` patterns, each part at least `COMPOUNDMIN` characters? `21st` is `2/m` + + /// `1st/p` under `n*mp`. A dynamic programme over (byte offset, token, parts so far), so a + /// hostile pattern costs at most a few thousand lookups. A forbidden word is never rebuilt + /// from its parts. + fn compound(&self, w: &str, deny: Mask) -> bool { + if self.compound_rules.is_empty() + || w.len() > MAX_COMPOUND_BYTES + || !w.chars().next().is_some_and(|c| self.compound_firsts.contains(&c)) + || self.stem_mask(w).is_some_and(|m| m.meets(self.forbidden)) + { + return false; + } + // Char boundaries of `w`, `0` and `w.len()` included. + let cuts: Vec = w.char_indices().map(|(i, _)| i).chain(std::iter::once(w.len())).collect(); + let n = cuts.len(); + // Flag mask of each part `w[cuts[a]..cuts[b]]`, by (a, b), looked up when a rule first + // reaches cut `a` (`None`: not looked up yet; `Some(None)`: not a stem). Only the parts + // after a cut a rule reaches are looked up, so a word with no first part costs `n`. + let mut parts: Vec>> = vec![None; n * n]; + let min = self.compound_min; + let mut part = |a: usize, b: usize| -> Option { + if b < a.saturating_add(min) { + return None; + } + let slot = parts.get_mut(a * n + b)?; + *slot.get_or_insert_with(|| { + let (Some(&x), Some(&y)) = (cuts.get(a), cuts.get(b)) else { return None }; + w.get(x..y).and_then(|p| self.stem_mask(p)).filter(|m| !m.meets(self.forbidden | deny)) + }) + }; + self.compound_rules.iter().any(|rule| compound_matches(rule, &mut part, n)) } /// Is `w` a stem with one suffix? With `need` set (a prefix was taken off), the stem must @@ -1166,6 +1279,17 @@ impl AffixDict { if !self.ignore.is_empty() { out.push_str(&format!("IGNORE {}\n", self.ignore.iter().collect::())); } + if !self.only_in_compound.is_empty() + && let Some(c) = self.flag_char(self.only_in_compound) + { + out.push_str(&format!("ONLYINCOMPOUND {c}\n")); + } + if !self.compound_text.is_empty() { + out.push_str(&format!("COMPOUNDMIN {}\n", self.compound_min)); + for p in &self.compound_text { + out.push_str(&format!("COMPOUNDRULE {p}\n")); + } + } let z = |s: &str| if s.is_empty() { "0".to_string() } else { s.to_string() }; for (kind, rules) in [("PFX", &self.prefixes), ("SFX", &self.suffixes)] { for r in rules { @@ -1233,6 +1357,20 @@ impl AffixDict { } "ICONV" if f.len() == 3 => rules.iconv.push((at(1).to_string(), at(2).to_string())), "IGNORE" => rules.ignore = at(1).chars().collect(), + "COMPOUNDMIN" => { + let min = at(1).parse::().map_err(|_| format!("affix dictionary: bad `{line}`"))?; + rules.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES)); + } + "ONLYINCOMPOUND" => { + rules.only_in_compound = Some(one_flag(at(1), "ONLYINCOMPOUND", 0).map_err(|_| format!("affix dictionary: bad `{line}`"))?); + } + "COMPOUNDRULE" => { + parse_compound_rule(at(1)).map_err(|e| format!("affix dictionary: {e}"))?; + if rules.compound_rules.len() >= MAX_COMPOUND_RULES { + return Err("affix dictionary: too many COMPOUNDRULE patterns".into()); + } + rules.compound_rules.push(at(1).to_string()); + } "MAP" => rules.map.push(at(1).chars().collect()), "REP" => rules.rep.push((at(1).replace('_', " "), at(2).replace('_', " "))), kind @ ("PFX" | "SFX") => { @@ -1287,6 +1425,39 @@ impl AffixDict { } } +/// Does a compound rule's token list match a split of the word into two or more parts? `part(a, +/// b)` is the flags of the stem between char cuts `a` and `b` (`None`: not a stem). States are +/// (cut, token, parts so far capped at 2), each visited once. +fn compound_matches(rule: &[(Mask, Repeat)], part: &mut impl FnMut(usize, usize) -> Option, n: usize) -> bool { + let t = rule.len(); + let idx = |cut: usize, tok: usize, k: usize| (cut * (t + 1) + tok) * 3 + k; + let mut seen = vec![false; n * (t + 1) * 3]; + let mut stack = vec![(0usize, 0usize, 0usize)]; + while let Some((cut, tok, k)) = stack.pop() { + match seen.get_mut(idx(cut, tok, k)) { + Some(s) if !*s => *s = true, + _ => continue, + } + if tok == t { + if cut + 1 == n && k >= 2 { + return true; + } + continue; + } + let Some(&(bit, repeat)) = rule.get(tok) else { continue }; + if repeat != Repeat::Once { + stack.push((cut, tok + 1, k)); + } + for next in cut + 1..n { + if part(cut, next).is_some_and(|m| m.meets(bit)) { + let k = (k + 1).min(2); + stack.push((next, if repeat == Repeat::Any { tok } else { tok + 1 }, k)); + } + } + } + false +} + #[cfg(test)] pub(crate) mod tests { use super::*; @@ -1779,6 +1950,73 @@ pub(crate) mod tests { } } + /// SCOWL's ordinal rules (its English `.aff`, verbatim): `21st`, `112th` and `1000th` are + /// compounds of digit stems, `1th` is only a part, and nothing else compounds. + const ORDINALS_AFF: &str = "SET UTF-8\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nWORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s .\n"; + const ORDINALS_DIC: &str = "13\n0/nm\n0th/pt\n1/n1\n1st/p\n1th/tc\n2/nm\n2nd/p\n2th/tc\n3/nm\n3rd/p\n3th/tc\n4/nm\n4th/pt\ncat/S\n"; + + #[test] + fn compound_rules_make_ordinals() { + let d = AffixDict::from_hunspell(ORDINALS_AFF, ORDINALS_DIC, &[]).unwrap(); + for w in ["1st", "2nd", "4th", "21st", "22nd", "23rd", "11th", "12th", "13th", "111th", "1000th", "104th", "cats"] { + assert!(d.check(w), "{w}"); + } + for w in ["1th", "2th", "11st", "12nd", "13rd", "21th", "catcat", "cat1st", "1stcat", "1st1st", "", "x"] { + assert!(!d.check(w), "{w}"); + } + // Only-in-compound parts are never suggested on their own. + assert!(!d.can_suggest("1th") && d.forms("1th").is_empty()); + // The compact format keeps the rules. + let back = AffixDict::from_bytes(&d.to_bytes()).unwrap(); + assert!(back.check("21st") && back.check("111th") && !back.check("1th") && !back.check("21th")); + } + + #[test] + fn compound_rules_need_two_parts_and_compoundmin() { + // `COMPOUNDMIN` defaults to 3; a pattern a lone stem matches still needs two parts. + let aff = "SET UTF-8\nONLYINCOMPOUND c\nCOMPOUNDRULE a*b?\n"; + let d = AffixDict::from_hunspell(aff, "3\nfoo/a\nbar/ab\nqux/c\n", &[]).unwrap(); + assert!(d.check("foobar") && d.check("foofoo") && d.check("foofoofoobar") && d.check("foo")); + assert!(!d.check("qux") && !d.check("quxfoo") && !d.check("fooqux") && !d.check("foobarx")); + let d = AffixDict::from_hunspell(aff, "2\nab/ab\ncd/a\n", &[]).unwrap(); + assert!(!d.check("abcd"), "parts shorter than COMPOUNDMIN 3"); + let d = AffixDict::from_hunspell(&format!("{aff}COMPOUNDMIN 2\n"), "2\nab/ab\ncd/a\n", &[]).unwrap(); + assert!(d.check("cdab")); + } + + /// craft-libs#483 review: a forbidden word isn't rebuilt from its parts. + #[test] + fn compound_never_rebuilds_a_forbidden_word() { + let d = AffixDict::from_hunspell("SET UTF-8\nFORBIDDENWORD X\nCOMPOUNDMIN 1\nCOMPOUNDRULE a*\n", "2\nx/a\nxx/X\n", &[]).unwrap(); + assert!(!d.check("xx") && d.check("xxx") && d.check("x")); + let back = AffixDict::from_bytes(&d.to_bytes()).unwrap(); + assert!(!back.check("xx") && back.check("xxx")); + } + + /// craft-libs#483 review: `ONLYINCOMPOUND` survives a round trip without compound rules too. + #[test] + fn onlyincompound_round_trips_without_compound_rules() { + let d = AffixDict::from_hunspell("SET UTF-8\nONLYINCOMPOUND c\n", "2\nfoo\nqux/c\n", &[]).unwrap(); + assert!(d.check("foo") && !d.check("qux")); + let back = AffixDict::from_bytes(&d.to_bytes()).unwrap(); + assert!(back.check("foo") && !back.check("qux"), "a part-only stem stays a part"); + } + + #[test] + fn hostile_compound_rules_are_refused_or_cheap() { + for bad in ["COMPOUNDRULE *a", "COMPOUNDRULE (ab)", "COMPOUNDRULE a**", "COMPOUNDMIN x", "ONLYINCOMPOUND ab"] { + assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{bad}\n"), DIC, &[]).is_err(), "{bad}"); + } + let many: String = (0..100).map(|_| "COMPOUNDRULE a*\n").collect(); + assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{many}"), DIC, &[]).is_err()); + // Every part a stem and a pattern of stars: still answers at once. + let star: String = std::iter::repeat_n("a*", 16).collect(); + let d = AffixDict::from_hunspell(&format!("SET UTF-8\nCOMPOUNDMIN 1\nCOMPOUNDRULE {star}b\n"), "1\nx/a\n", &[]).unwrap(); + let t = std::time::Instant::now(); + assert!(!d.check(&"x".repeat(48)) && !d.check(&"x".repeat(200))); + assert!(t.elapsed().as_secs() < 2); + } + #[test] fn fields_split_on_spaces_and_tabs_only() { // A flag byte read as U+00A0 (no-break space) or U+0085 is a flag, not a separator. @@ -1796,22 +2034,23 @@ pub(crate) mod tests { } } - /// craft-libs#482 review: an English dictionary with compounding (for ordinals) and output - /// conversion, as the usual en_US and en_GB ones have, loads for English; its `ONLYINCOMPOUND` - /// stems are left out. Other languages still refuse those directives. (A synthetic `.aff` - /// written for this test.) + /// craft-libs#482 review: an English dictionary with compounding and output conversion, as the + /// usual en_US and en_GB ones have, loads for English: `COMPOUNDRULE` makes its ordinals, its + /// `ONLYINCOMPOUND` stems are words only inside them, and `COMPOUNDFLAG` and `OCONV` are + /// ignored. Other languages still refuse those two. (A synthetic `.aff` written for this test.) #[test] fn english_dictionaries_ignore_compounding_and_output_conversion() { let aff = "SET UTF-8\nTRY esianrtolcdugmphbyfvkwz'\nICONV 1\nICONV ’ '\nOCONV 1\nOCONV ' ’\nNOSUGGEST !\n\ COMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nCOMPOUNDFLAG Z\n\ WORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s [^sxzhy]\nSFX D Y 1\nSFX D 0 ed [^ey]\n"; - let dic = "5\nhouse/S\nwalk/SD\n0th/pc\n1st/c\nthe\n"; + let dic = "7\nhouse/S\nwalk/SD\n0th/pc\n1st/p\n1th/tc\n1/n1\nthe\n"; assert!(AffixDict::from_hunspell(aff, dic, &[]).is_err(), "refused for other languages"); let d = AffixDict::from_hunspell_english(aff, dic, &[]).unwrap(); for w in ["house", "houses", "walked", "walks", "the"] { assert!(d.check(w), "{w}"); } - assert!(!d.check("0th") && !d.check("1st"), "parts of compounds only"); + assert!(!d.check("0th") && !d.check("1th"), "parts of compounds only"); + assert!(d.check("1st") && d.check("11th") && d.check("111th") && !d.check("11st"), "ordinals by COMPOUNDRULE"); assert!(!d.check("housed") && !d.check("hous")); // The directives that change which words are right are still refused. for dir in ["NEEDAFFIX", "CIRCUMFIX", "PSEUDOROOT", "FULLSTRIP"] { diff --git a/crates/proof/src/affix_suggest.rs b/crates/proof/src/affix_suggest.rs index 1456f81c..5029c310 100644 --- a/crates/proof/src/affix_suggest.rs +++ b/crates/proof/src/affix_suggest.rs @@ -43,6 +43,8 @@ const COST_SWAP: u32 = 14; const COST_DROP_OR_ADD: u32 = 15; pub(crate) const COST_MISTYPE: u32 = 16; const COST_SCAN: u32 = 20; +/// A missing space ranked after every one-word candidate (English: [`Split::Custom`]). +pub(crate) const COST_SPLIT_LAST: u32 = COST_SCAN + 4 * 3 + 1; /// What a language adds to its dictionary's tables. pub(crate) struct Tuning { @@ -62,6 +64,9 @@ pub(crate) struct Tuning { pub map_cost: fn(char, char) -> u32, /// The cost of typing letter `c` at `i` of the word (`chars`, lower case) instead. pub mistype_cost: fn(&[char], usize, char) -> u32, + /// Offer a word only the dictionary capitalizes (a name, an abbreviation: `Col`, `Wis`) only + /// when the word itself is capitalized, and never as half of a missing space (`Che Que`). + pub strict_case: bool, } /// Where a missing space may go. @@ -78,8 +83,17 @@ pub(crate) enum Split { impl Tuning { /// The dictionary's tables alone (Czech, Indonesian, Spanish). - pub(crate) const DEFAULT: Tuning = - Tuning { split: Split::Anywhere, alphabet: "", map: &[], rep: &[], rep_cost: COST_REP, max_map_changes: 4, map_cost, mistype_cost }; + pub(crate) const DEFAULT: Tuning = Tuning { + split: Split::Anywhere, + alphabet: "", + map: &[], + rep: &[], + rep_cost: COST_REP, + max_map_changes: 4, + map_cost, + mistype_cost, + strict_case: false, + }; } /// The cost of changing letter `from` to its MAP relative `to`. Adding a missing accent is the @@ -107,6 +121,10 @@ struct Collector<'a> { /// Suggestion → (cost, order found). found: HashMap, seq: usize, + /// May a word the dictionary only has capitalized be offered ([`Tuning::strict_case`])? + names: bool, + /// May half of a missing space be one ([`Tuning::strict_case`])? + split_names: bool, } impl Collector<'_> { @@ -125,9 +143,12 @@ impl Collector<'_> { } let mut words = Vec::new(); let mut cost = cost; + let names = if cand.contains(' ') { self.split_names } else { self.names }; for part in cand.split(' ') { if self.d.contains_suggestible(part) { words.push(part.to_string()); + } else if !names { + return; } else { let c = capitalize(part); if !self.d.contains_suggestible(&c) { @@ -278,7 +299,9 @@ pub(crate) fn suggest_in(d: &AffixDict, word: &str, max: usize, t: &Tuning) -> V let case = casing(word); let input = word.to_lowercase(); let chars: Vec = input.chars().collect(); - let mut col = Collector { d, input: input.clone(), tested: HashMap::new(), found: HashMap::new(), seq: 0 }; + let names = !t.strict_case || case != Casing::Lower; + let split_names = !t.strict_case; + let mut col = Collector { d, input: input.clone(), tested: HashMap::new(), found: HashMap::new(), seq: 0, names, split_names }; add_case(&mut col, word, &input); let map = groups(d, t); add_map_slips(&mut col, &chars, &map, t); @@ -479,6 +502,9 @@ fn scan(col: &mut Collector, chars: &[char], map: &[Vec]) { if lf.len().abs_diff(chars.len()) > 2 { continue; } + if !col.names && form.chars().any(char::is_uppercase) { + continue; + } let dist = weighted_distance(&lf, chars, &related, limit); if dist <= limit && lf.as_slice() != chars { col.keep(form, COST_SCAN + dist * 3); @@ -504,7 +530,8 @@ mod tests { #[test] fn a_cheaper_later_offer_wins() { let d = AffixDict::from_hunspell("", "1\ncafe\n", &[]).unwrap(); - let mut col = Collector { d: &d, input: "cafè".into(), tested: HashMap::new(), found: HashMap::new(), seq: 0 }; + let mut col = + Collector { d: &d, input: "cafè".into(), tested: HashMap::new(), found: HashMap::new(), seq: 0, names: true, split_names: true }; col.offer("cafe", COST_DROP_DIACRITIC); col.offer("cafe", COST_REP); col.offer("cafe", COST_SCAN); diff --git a/crates/proof/src/english.rs b/crates/proof/src/english.rs index 67265abf..a18f060f 100644 --- a/crates/proof/src/english.rs +++ b/crates/proof/src/english.rs @@ -1,29 +1,40 @@ //! English spelling never goes unchecked. Its words come from the first of these that is there: //! -//! 1. **Embedded**: the Moby word list embedded from craft-i18n at build time (`CRAFT_I18N_DIR`, -//! see `build.rs`). When it is there, nothing in this module is used. -//! 2. **System Hunspell**: a Hunspell `.aff`/`.dic` pair for the run's own English found by -//! [`crate::hunspell`] (`en_US` for American, `en_GB` for British, `en_CA` for Canadian -//! English; each name in the dictionaries folder and then the system locations before the -//! next name; none on the web). Another English's dictionary is never used: it would mark this -//! English's spellings (`colour` in an `en_US` one), so without its own dictionary an English -//! uses the built-in list, which has both spellings. A file that can't be read is skipped for -//! the next name or folder, and one file is read once for every English that uses it. -//! Directives that only matter for other languages (compounding, output conversion) are -//! ignored ([`crate::hunspell::load_english`]). -//! 3. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into +//! 1. **Dictionaries folder**: a Hunspell `.aff`/`.dic` pair for the run's own English that the +//! user keeps for WordCraft ([`crate::hunspell::user_dirs`]: the folder set in the options, +//! `WORDCRAFT_DICTIONARIES`, WordCraft's own `dictionaries` folder). +//! 2. **Installed English pack**: reserved for Language packs (craft-libs#485, CP4.2). +//! 3. **Embedded SCOWL**: the run's English's SCOWL dictionary (en-US, en-GB or en-CA) embedded +//! from craft-i18n at build time ([`crate::scowl`]); a word it knows (British English: also +//! the `-ize` spelling of one it knows, [`ise_spelling`]), or the supplement ([`EXTRA`]), is +//! right; never one only another English knows. When it is there, the system locations aren't +//! searched. +//! 4. **System Hunspell**: a pair for the run's own English in the system locations (none on the +//! web). +//! 5. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into //! the crate (without its break points), with WordCraft's own supplement of modern words -//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled, as with the -//! embedded list. +//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled. (The Moby list +//! embedded from craft-i18n is the same list; it is used for hyphenation only.) //! -//! A Hunspell dictionary that is there but can't be read (unsupported encoding, too large, a -//! parse error, no words) is skipped, and its reason is reported by +//! Hunspell dictionaries are found by [`crate::hunspell`] by their own English's names only +//! (`en_US` for American, `en_GB` for British, `en_CA` for Canadian English), each name in every +//! folder before the next name. Another English's dictionary is never used: it would mark this +//! English's spellings (`colour` in an `en_US` one). A file that can't be read is skipped for the +//! next name or folder, and one file is read once for every English that uses it. `COMPOUNDRULE` +//! (ordinals) is read; flag-based compounding and `OCONV` are ignored +//! ([`crate::hunspell::load_english`]). +//! +//! Whichever it is, the dialect policy of `en-variants.txt` ([`crate::en_variants`], when it is +//! embedded) flags the other dialect's spellings and accepts this dialect's spellings of known +//! words. A Hunspell dictionary that is there but can't be read (unsupported encoding, too large, +//! a parse error, no words) is skipped, and its reason is reported by //! [`crate::dictionary_load_error`]. Non-English text is never checked against any of these. use std::path::{Path, PathBuf}; use std::sync::{Arc, OnceLock, PoisonError, RwLock}; use crate::affix::AffixDict; +use crate::affix_suggest::{Split, Tuning}; use crate::dict::Dictionary; use crate::{Options, ProofLang}; @@ -47,7 +58,7 @@ const MAX_CANDIDATES: usize = 8; /// Where English spelling comes from now ([`crate::english_source`]). #[derive(Clone, Debug, PartialEq, Eq)] pub enum EnglishSource { - /// The Moby word list embedded from craft-i18n at build time. + /// The English's SCOWL dictionary embedded from craft-i18n at build time. Embedded, /// An English Hunspell dictionary from the dictionaries folder or the system: its `.dic` file. Hunspell(PathBuf), @@ -112,14 +123,9 @@ impl State { static HUNSPELL: RwLock = RwLock::new(State::new()); -/// Why the embedded word list failed to decode, if it did. -static EMBEDDED_ERROR: OnceLock = OnceLock::new(); - -/// Remember that the embedded word list is there but failed to decode (English then uses the -/// next source). +/// The embedded Moby list (hyphenation) failed to decode: log it. Spelling doesn't use it. pub(crate) fn note_embedded_error(e: String) { - log::error!("embedded English word list: {e}; English spelling falls back to Hunspell or the built-in word list"); - let _ = EMBEDDED_ERROR.set(e); + log::error!("embedded English word list: {e}; English hyphenation falls back to the patterns"); } /// The English Hunspell dictionary for `lang` (read on first use). @@ -185,12 +191,14 @@ fn key(lang: ProofLang) -> &'static str { } /// Read the first Hunspell pair of `lang`'s own that loads, trying each name in each folder -/// ([`crate::hunspell::candidates`]), with the first failure before it (the file `lang` would -/// rather use), for [`note_error`]. +/// ([`crate::hunspell::candidates`]; with SCOWL embedded, the dictionaries folder only: +/// [`crate::hunspell::user_candidates`]), with the first failure before it (the file `lang` +/// would rather use), for [`note_error`]. fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) { - let candidates = crate::hunspell::candidates(names(lang)); + let candidates = + if crate::scowl::dict(lang).is_some() { crate::hunspell::user_candidates(names(lang)) } else { crate::hunspell::candidates(names(lang)) }; if candidates.is_empty() { - log::info!("English spelling: no Hunspell {} dictionary installed; using the built-in word list", key(lang)); + log::info!("English spelling: no Hunspell {} dictionary to read first; using SCOWL or the built-in word list", key(lang)); return (None, None); } let mut first_error: Option<(PathBuf, String)> = None; @@ -209,7 +217,7 @@ fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) { } } } - log::warn!("English spelling: no {} Hunspell dictionary could be read; using the built-in word list", key(lang)); + log::warn!("English spelling: no {} Hunspell dictionary could be read; using SCOWL or the built-in word list", key(lang)); (None, first_error) } @@ -232,45 +240,150 @@ pub(crate) fn reload() { } } -/// Where the spelling of `lang` (an English) comes from now. Reads the Hunspell dictionary once -/// when there is no embedded data (it is cached afterwards). +/// Where the spelling of `lang` (an English) comes from now, in the order of the module docs. +/// Reads the Hunspell dictionary once (it is cached afterwards). pub(crate) fn source(lang: ProofLang) -> EnglishSource { - if crate::english_data() { - EnglishSource::Embedded - } else { - hunspell(lang).map_or(EnglishSource::CoreList, |(_, dic)| EnglishSource::Hunspell(dic)) + match speller(lang) { + Speller::Hunspell(_, dic) => EnglishSource::Hunspell(dic), + Speller::Scowl(..) => EnglishSource::Embedded, + Speller::List => EnglishSource::CoreList, } } -/// Why a source of `lang`'s spelling failed to load: the embedded word list, or (without it) the -/// Hunspell dictionary found for it. Reads nothing itself. +/// Why a source of `lang`'s spelling failed to load: its embedded SCOWL dictionary, or the +/// Hunspell dictionary it would rather have used. Reads nothing itself. pub(crate) fn load_error(lang: ProofLang) -> Option { - if let Some(e) = EMBEDDED_ERROR.get() { - return Some(e.clone()); - } - if crate::english_data() { - return None; - } - crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason) + crate::scowl::load_error(lang).or_else(|| crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason)) } -/// Is `word` spelled right in `lang` (an English) when no word list is embedded: against the -/// Hunspell dictionary, or else the built-in word list. -pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options) -> bool { +/// What `lang` is checked against now. +enum Speller { + /// A dictionaries-folder or system Hunspell dictionary, and its `.dic` file. + Hunspell(Arc, PathBuf), + /// `lang`'s own SCOWL dictionary (every English's, for English with no dialect: [`generic`]), + /// and whether British `-ize` spellings are right too ([`ise_spelling`]). + Scowl(Vec<&'static AffixDict>, bool), + /// The built-in word list. + List, +} + +fn speller(lang: ProofLang) -> Speller { + pick(hunspell(lang), lang) +} + +/// The lookup order: the Hunspell dictionary `found` for `lang` (with SCOWL embedded, only one in +/// the dictionaries folder is looked for), then SCOWL, then the built-in list. (The installed +/// English pack, craft-libs#485, goes between the first two.) +fn pick(found: Loaded, lang: ProofLang) -> Speller { + if let Some((d, dic)) = found { + return Speller::Hunspell(d, dic); + } + match crate::scowl::dict(lang) { + Some(d) => Speller::Scowl(vec![d], crate::scowl::is_british(lang)), + None => Speller::List, + } +} + +/// [`speller`] for English with no dialect ([`crate::is_correct`]): a SCOWL word of any English +/// is right, as in the built-in list, which has both spellings. +fn generic(s: Speller) -> Speller { + match s { + Speller::Scowl(..) => { + let all: Vec<_> = [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa].into_iter().filter_map(crate::scowl::dict).collect(); + Speller::Scowl(all, false) + } + other => other, + } +} + +impl Speller { + /// Is `w` (as written, `’` read as `'`) a word: as a whole, or each part of a hyphenated one? + fn knows(&self, w: &str) -> bool { + match self { + Speller::Hunspell(d, _) => affix_correct(&[d.as_ref()], w, false), + Speller::Scowl(ds, ise) => { + affix_correct(ds, w, true) + || (*ise + && ise_spelling(w).is_some_and(|x| affix_correct(ds, &x, true)) + && ize_elsewhere(w, |d, p| affix_correct(&[d], p, false))) + } + Speller::List => list_correct(w), + } + } + + /// Is the lower-case base form `p` a word (for the dialect policy's other spelling)? + fn knows_lower(&self, p: &str) -> bool { + match self { + Speller::Hunspell(d, _) => d.check(p), + Speller::Scowl(ds, ise) => { + ds.iter().any(|d| d.check(p) || (*ise && ise_spelling(p).is_some_and(|x| d.check(&x)) && ize_elsewhere(p, |d, p| d.check(p)))) + } + Speller::List => crate::known_in_word_list(p), + } + } +} + +/// Is `word` spelled right in `lang` (an English), against its own dictionary ([`speller`]); with +/// `policy` false, English with no dialect: any English's SCOWL dictionary ([`generic`]). With +/// `policy`, also by the dialect policy of `en-variants.txt`: the other dialect's spellings are +/// wrong and this dialect's spellings of known words right. The user dictionary overrules both. +pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options, policy: bool) -> bool { let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’').trim_matches(['\'', '’']); if crate::accepted_by_options(w, opts) || !w.chars().any(|c| c.is_ascii_alphabetic()) { return true; // other scripts: not English } let w = w.replace('’', "'"); - match hunspell(lang) { - Some((d, _)) => hunspell_correct(&d, &w), - None => list_correct(&w), + let lower = w.to_lowercase(); + let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect(); + if policy && parts.iter().any(|p| crate::en_variants::is_foreign(p, lang)) { + return false; } + let s = if policy { speller(lang) } else { generic(speller(lang)) }; + s.knows(&w) + || (policy && !parts.is_empty() && parts.iter().all(|p| s.knows_lower(p) || crate::en_variants::known_variant(p, lang, |x| s.knows_lower(x)))) } -/// Against an installed English Hunspell dictionary: the word, or each part of a hyphenated one. -fn hunspell_correct(d: &AffixDict, w: &str) -> bool { - let known = |p: &str| p.chars().count() <= 1 || d.check(p) || crate::in_user_dictionary(&p.to_lowercase()); +/// `w` with its `-ize`/`-yze` endings (and their forms: `-ization`, `-izing`…) spelled `-ise`/ +/// `-yse`, when it has any: British English takes both (Word accepts `organize` and `organise` in +/// en-GB), and SCOWL's en-GB dictionary has only `-ise`. A `z` between `i`/`y` and a vowel is +/// swapped; the `-ize` form counts only when American or Canadian English spells the word so +/// ([`ize_elsewhere`]), or `wize` would pass as `wise`. +pub(crate) fn ise_spelling(w: &str) -> Option { + let b = w.as_bytes(); + let mut out = b.to_vec(); + let mut changed = false; + for i in 1..b.len() { + let after = b.get(i + 1).is_some_and(|c| b"aeiAEI".contains(c)); + if after + && matches!(b.get(i - 1), Some(b'i' | b'y' | b'I' | b'Y')) + && let Some(c) = out.get_mut(i) + { + match *c { + b'z' => *c = b's', + b'Z' => *c = b'S', + _ => continue, + } + changed = true; + } + } + if changed { String::from_utf8(out).ok() } else { None } +} + +/// Is `w` a word in American or Canadian English's SCOWL dictionary (`check` asks one): the +/// `-ize` spellings British English takes are theirs (`organize`, never `advize` or `wize`). +fn ize_elsewhere(w: &str, check: impl Fn(&AffixDict, &str) -> bool) -> bool { + [ProofLang::En, ProofLang::EnCa].into_iter().filter_map(crate::scowl::dict).any(|d| check(d, w)) +} + +/// Against affix dictionaries (any of them): the word, or each part of a hyphenated one; with +/// `supplement`, our supplement's words count too. +fn affix_correct(ds: &[&AffixDict], w: &str, supplement: bool) -> bool { + let known = |p: &str| { + p.chars().count() <= 1 + || ds.iter().any(|d| d.check(p)) + || crate::in_user_dictionary(&p.to_lowercase()) + || (supplement && crate::known_in_supplement(&p.to_lowercase())) + }; known(w) || (w.contains('-') && w.split('-').filter(|p| !p.is_empty()).all(known)) } @@ -285,15 +398,25 @@ fn list_correct(w: &str) -> bool { crate::known_in_word_list(&lower) || (lower.contains('-') && lower.split('-').filter(|p| !p.is_empty()).all(crate::known_in_word_list)) } -/// Spelling suggestions in `lang` (an English) when no word list is embedded: from the Hunspell -/// dictionary, or else the built-in word list. +/// Spelling suggestions in `lang` (an English), from its dictionary ([`speller`]). pub(crate) fn suggest(word: &str, max: usize, lang: ProofLang) -> Vec { - match hunspell(lang) { - Some((d, _)) => crate::cs::suggest::suggest_in(&d, word, max), - None => list_suggest(word, max), + match speller(lang) { + Speller::Hunspell(d, _) => crate::affix_suggest::suggest_in(&d, word, max, &TUNING), + Speller::Scowl(ds, _) => ds.first().map_or_else(Vec::new, |d| crate::affix_suggest::suggest_in(d, word, max, &TUNING)), + Speller::List => list_suggest(word, max), } } +/// English suggestions from an affix dictionary: its own tables, a missing space only between two +/// words of three letters or more and after every one-word candidate (`publically` → `publicly` +/// before `public ally`), and no abbreviation or name offered for a lower-case word (`col`, `Wis`). +const TUNING: Tuning = Tuning { split: Split::Custom(split_cost), strict_case: true, ..Tuning::DEFAULT }; + +/// The cost of a missing space in English ([`TUNING`]). +fn split_cost(a: &str, b: &str) -> Option { + (a.chars().count() >= 3 && b.chars().count() >= 3).then_some(crate::affix_suggest::COST_SPLIT_LAST) +} + /// Letters an edit may add or put in a word's place. const ALPHABET: &str = "abcdefghijklmnopqrstuvwxyz'"; @@ -460,6 +583,46 @@ mod tests { assert!(start.elapsed() < std::time::Duration::from_secs(5), "{:?}", start.elapsed()); } + /// The lookup order (README): a dictionaries-folder English comes before SCOWL, SCOWL before + /// the built-in list; each English looks for its own dictionary only, and with SCOWL embedded + /// the system locations aren't searched. (Checked here without touching the process-wide + /// folder, which every English test running meanwhile would see.) + #[test] + fn lookup_order() { + let tiny = Arc::new(AffixDict::from_hunspell("", "1\nzzyzx\n", &[]).unwrap()); + let file = PathBuf::from("/folder/en_US.dic"); + for lang in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] { + assert!(matches!(pick(Some((tiny.clone(), file.clone())), lang), Speller::Hunspell(_, p) if p == file), "{lang:?}"); + let fallback = pick(None, lang); + if crate::scowl::dict(lang).is_some() { + assert!( + matches!(&fallback, Speller::Scowl(ds, _) if ds.len() == 1 && std::ptr::eq(ds[0], crate::scowl::dict(lang).unwrap())), + "{lang:?}" + ); + assert!(fallback.knows("21st") && !fallback.knows("becuase"), "{lang:?}"); + assert_eq!(fallback.knows("colour"), lang != ProofLang::En, "{lang:?}: its own dictionary only"); + } else { + assert!(matches!(fallback, Speller::List), "{lang:?}"); + } + } + let dir = std::env::temp_dir().join(format!("wc-proof-en-order-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + for name in ["en_US", "en_GB", "en_CA"] { + std::fs::write(dir.join(format!("{name}.aff")), "SET UTF-8\n").unwrap(); + std::fs::write(dir.join(format!("{name}.dic")), "1\nzzyzx\n").unwrap(); + } + let found = |names: &[&str]| -> Vec { + crate::hunspell::candidates_in(std::slice::from_ref(&dir), names) + .iter() + .filter_map(|(_, d)| d.file_stem().map(|n| n.to_string_lossy().into_owned())) + .collect() + }; + assert_eq!(found(names(ProofLang::EnGb)), ["en_GB"], "a user's en_US doesn't stand in for SCOWL's en-GB"); + assert_eq!(found(names(ProofLang::EnCa)), ["en_CA"]); + assert_eq!(found(names(ProofLang::En)), ["en_US"]); + let _ = std::fs::remove_dir_all(&dir); + } + /// craft-libs#482 review: a load that started before a reload (the dictionaries folder /// changed) serves its own check but isn't kept, so the next check reads the new folder; a /// load after it is kept, and shared with the Englishes that read the same file. diff --git a/crates/proof/src/hunspell.rs b/crates/proof/src/hunspell.rs index 98f8c584..6032e1c3 100644 --- a/crates/proof/src/hunspell.rs +++ b/crates/proof/src/hunspell.rs @@ -154,6 +154,25 @@ pub fn candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> { candidates_in(&search_dirs(lo.as_deref()), names) } +/// [`candidates`] in the folders the user keeps dictionaries in for WordCraft ([`user_dirs`]) +/// only, not the system locations. +pub fn user_candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> { + candidates_in(&user_dirs(), names) +} + +/// The folders of [`search_dirs`] the user keeps dictionaries in for WordCraft: the one set with +/// [`set_dictionary_dir`], `WORDCRAFT_DICTIONARIES`, and WordCraft's own `dictionaries` folder +/// under the user's application data. The rest are the system's (and LibreOffice's). +pub fn user_dirs() -> Vec { + let extra = EXTRA_DIR.read().unwrap_or_else(std::sync::PoisonError::into_inner).clone(); + let env = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()).map(PathBuf::from); + let wordcraft_own = |d: &Path| { + d.file_name().is_some_and(|n| n == "dictionaries") + && d.parent().and_then(Path::file_name).is_some_and(|n| n.eq_ignore_ascii_case("wordcraft")) + }; + search_dirs(None).into_iter().filter(|d| Some(d) == extra.as_ref() || Some(d) == env.as_ref() || wordcraft_own(d)).collect() +} + /// [`candidates`] in `dirs`. pub fn candidates_in(dirs: &[PathBuf], names: &[&str]) -> Vec<(PathBuf, PathBuf)> { let mut out: Vec<(PathBuf, PathBuf)> = Vec::new(); diff --git a/crates/proof/src/lang.rs b/crates/proof/src/lang.rs index 8f681408..3d15d224 100644 --- a/crates/proof/src/lang.rs +++ b/crates/proof/src/lang.rs @@ -6,7 +6,7 @@ #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)] pub enum ProofLang { /// American English (`en`, `en-US`, and every region not listed for [`ProofLang::EnGb`] or - /// [`ProofLang::EnCa`]): the public-domain Moby word list, the English grammar rules and the + /// [`ProofLang::EnCa`]): SCOWL's en-US dictionary ([`crate::scowl`]), the English grammar rules and the /// English hyphenation dictionary and patterns. British spellings (`colour`, `centre`, /// `organise`) are flagged, with the American form suggested ([`crate::en_variants`]). Also /// text with no language at all (WordCraft's default editing language is US English, like diff --git a/crates/proof/src/lib.rs b/crates/proof/src/lib.rs index d1aa3f46..ff384792 100644 --- a/crates/proof/src/lib.rs +++ b/crates/proof/src/lib.rs @@ -1,8 +1,9 @@ //! Proofing for the Craft Apps (moved from WordCraft's `wordcraft-proof`), per language. A run's language (its BCP 47 tag) picks the tools through //! [`ProofLang`]; the `*_in` functions take it, the older English-only ones remain. //! -//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the -//! public-domain Moby word list (~160k words, the same list that drives hyphenation) plus our +//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the SCOWL +//! 2026 Hunspell dictionary of the run's English ([`scowl`]; else the public-domain Moby word +//! list, ~160k words, the same list that drives hyphenation) plus our //! supplement of modern words it lacks (craft-i18n's `language-data/en-us/en-extra.txt`: `desktop`, `email`, //! `download`…), with inflections (`-s`, `-es`, `-ed`, `-ing`, `-ly`, `'s`…), a user dictionary and ignore lists; //! suggestions by edit distance. Grammar checks that are cheap and reliable: repeated words, @@ -48,11 +49,12 @@ //! //! [`check_text`] checks a paragraph whose runs are in different languages. //! -//! **Language data** (the Moby word list, the pattern files, our word supplements) is embedded at -//! build time from craft-i18n's `language-data/` (`CRAFT_I18N_DIR`, see `build.rs`). English -//! spelling never goes unchecked: without the embedded word list it uses an English Hunspell -//! dictionary installed on the system, and without one the whole Moby word list built into the -//! crate (see [`english_source`]). Hyphenation that needs the data is off without it; grammar and the +//! **Language data** (the SCOWL English dictionaries, the Moby word list, the pattern files, our +//! word supplements) is embedded at build time from craft-i18n's `language-data/` +//! (`CRAFT_I18N_DIR`, see `build.rs`). English spelling never goes unchecked: a dictionaries-folder +//! English first, then the embedded SCOWL dictionary, then an English Hunspell dictionary +//! installed on the system, then the whole Moby word list built into the crate (see +//! [`english_source`]). Hyphenation that needs the data is off without it; grammar and the //! rule-based hyphenation (Bulgarian, Turkish, Russian) are code and still work. #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] @@ -78,6 +80,7 @@ pub mod patterns; pub mod pl; pub mod pt; pub mod ru; +pub mod scowl; mod slot; mod spelling; pub mod tr; @@ -315,7 +318,7 @@ pub fn installed_dictionaries() -> Vec<(ProofLang, Option)> /// Why the dictionary found for `lang` failed to load (unsupported encoding, too large, a parse /// error), if it was read and failed. Doesn't read anything itself: a dictionary nobody has -/// checked yet reports `None` (#2830). For an English, the embedded word list or (without it) the +/// checked yet reports `None` (#2830). For an English, its embedded SCOWL dictionary or the /// English Hunspell dictionary that failed; English then uses the next source ([`english_source`]). pub fn dictionary_load_error(lang: ProofLang) -> Option { if matches!(lang, ProofLang::En | ProofLang::EnGb | ProofLang::EnCa) { @@ -324,12 +327,11 @@ pub fn dictionary_load_error(lang: ProofLang) -> Option { hunspell::load_error(language(lang)?.slot.name()).map(|(_, reason)| reason) } -/// Where the spelling of an English `lang` comes from now (`None` for other languages): the word -/// list embedded at build time, else an English Hunspell dictionary from the dictionaries folder -/// or the system (the run's English first), else the Moby word list built into the crate. -/// Each English uses only its own Hunspell dictionary (`en_US`, `en_GB`, `en_CA`): without it, -/// the built-in list, which has both spellings. Reads the Hunspell dictionary once when there is no -/// embedded word list (it is cached afterwards). +/// Where the spelling of an English `lang` comes from now (`None` for other languages), in this +/// order: a Hunspell dictionary of its own (`en_US`, `en_GB`, `en_CA`) in the dictionaries +/// folder, the SCOWL dictionary embedded at build time, one of its own on the system, else the +/// Moby word list built into the crate. Another English's dictionary is never used. Reads the +/// Hunspell dictionary once (it is cached afterwards). pub fn english_source(lang: ProofLang) -> Option { lang.is_english().then(|| english::source(lang)) } @@ -467,7 +469,7 @@ pub(crate) fn is_common_word(w: &str) -> bool { } /// Our supplement of common words the Moby list lacks (`desktop`, `email`, `download`…): one -/// lowercase base form per line, `#` comments. Inflections come from [`known_core`]. +/// lowercase base form per line, `#` comments. Inflections come from [`known_with`]. use data::EXTRA_EN; /// The supplement's words: the Language data's copy, else the crate's own ([`english::EXTRA`]). @@ -477,21 +479,18 @@ fn extra_words() -> &'static HashSet<&'static str> { E.get_or_init(|| extra.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect()) } -/// A base form in the dictionary or the supplement. -fn base_known(d: &dict::Dictionary, w: &str) -> bool { - d.get(w).is_some() || extra_words().contains(w) -} - -fn known_core(w: &str) -> bool { - let d = dict::Dictionary::en_us(); - known_with(w, &|s| base_known(d, s)) -} - -/// [`known_core`] against the word list built into the crate (no embedded word list). +/// Is `w` (lower case) a word of the built-in Moby list or the supplement, with the inflection +/// rules ([`known_with`])? pub(crate) fn known_in_word_list(w: &str) -> bool { known_with(w, &|s| english::word_list_contains(s) || extra_words().contains(s)) } +/// Is `w` (lower case) a word of the supplement alone, with the inflection rules (the SCOWL +/// dictionaries take it as an addition)? +pub(crate) fn known_in_supplement(w: &str) -> bool { + known_with(w, &|s| extra_words().contains(s)) +} + /// Is `w` (lower case) a word, by `base_known` (a base form in the word list), the common words, /// the user dictionary and the inflection and prefix rules? fn known_with(w: &str, base_known: &dyn Fn(&str) -> bool) -> bool { @@ -615,66 +614,16 @@ pub(crate) fn accepted_by_options(word: &str, opts: &Options) -> bool { || in_user_dictionary(&w.to_lowercase()) } -/// Is the English word list embedded? Without it English is checked against an installed -/// Hunspell dictionary or the word list built into the crate ([`english`]). -pub(crate) fn english_data() -> bool { - !dict::Dictionary::en_us().is_empty() -} - -/// [`is_correct`] with the File › Options › Proofing choices. +/// [`is_correct`] with the File › Options › Proofing choices: English with no dialect, so a word +/// any English spells so is right (`color` and `colour`; American English's dictionary when it +/// isn't SCOWL: [`english_source`]), without the dialect policy. pub fn is_correct_with(word: &str, opts: &Options) -> bool { - if !english_data() { - return english::is_correct(word, ProofLang::En, opts); - } - let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\''); - let w = w.trim_matches('\''); - if accepted_by_options(w, opts) { - return true; - } - if !w.is_ascii() && !w.chars().any(|c| c.is_ascii_alphabetic()) { - return true; // other scripts: no dictionary - } - let lower = w.to_lowercase().replace('’', "'"); - if known_core(&lower) { - return true; - } - // Hyphenated compounds: each part. - if lower.contains('-') { - return lower.split('-').all(|p| p.is_empty() || known_core(p)); - } - false + english::is_correct(word, ProofLang::En, opts, false) } /// Spelling suggestions for a misspelled word (best first, at most `max`). pub fn suggest(word: &str, max: usize) -> Vec { - if !english_data() { - return english::suggest(word, max, ProofLang::En); - } - suggest_from(word, max, dict::Dictionary::en_us().iter().map(|(w, _)| w)) -} - -/// [`suggest`] from `words` (lower case) and our supplement. -pub(crate) fn suggest_from<'a>(word: &str, max: usize, words: impl Iterator) -> Vec { - let lower = word.to_lowercase(); - let n = lower.chars().count(); - if n == 0 || n > 40 { - return Vec::new(); - } - let mut scored: Vec<(usize, i64, String)> = Vec::new(); - let target: Vec = lower.chars().collect(); - for w in words.chain(extra_words().iter().copied()) { - let wn = w.chars().count(); - if wn + 2 < n || wn > n + 2 { - continue; - } - let dist = damerau(&target, w); - if dist <= 2 { - let same_first = w.chars().next() == target.first().copied(); - scored.push((dist, if same_first { 0 } else { 1 }, w.to_string())); - } - } - scored.sort_by(|a, b| (a.0, a.1, a.2.len().abs_diff(n)).cmp(&(b.0, b.1, b.2.len().abs_diff(n))).then(a.2.cmp(&b.2))); - scored.into_iter().take(max).map(|(_, _, w)| match_case(word, w)).collect() + english::suggest(word, max, ProofLang::En) } /// `w` (lower case) in the case of `word`: all capitals, an initial capital, or as it is. @@ -709,26 +658,11 @@ fn english_preferred(lower: &str, lang: ProofLang) -> Option { } /// Is `word` spelled right in the English `lang` ([`ProofLang::En`], [`ProofLang::EnGb`], -/// [`ProofLang::EnCa`])? The shared word list's words, less the other dialect's spellings +/// [`ProofLang::EnCa`])? Its dictionary's words ([`english`]), less the other dialect's spellings /// (`colour` in American English, `color` in British), plus this dialect's spellings of the -/// list's words ([`en_variants`]). The user dictionary overrules both. +/// dictionary's words ([`en_variants`]). The user dictionary overrules both. fn is_correct_english(word: &str, lang: ProofLang, opts: &Options) -> bool { - if !english_data() { - return english::is_correct(word, lang, opts); - } - let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'').trim_matches('\''); - if accepted_by_options(w, opts) { - return true; - } - let lower = w.to_lowercase().replace('’', "'"); - if in_user_dictionary(&lower) { - return true; - } - let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect(); - if parts.iter().any(|p| en_variants::is_foreign(p, lang)) { - return false; - } - is_correct_with(word, opts) || (!parts.is_empty() && parts.iter().all(|p| known_core(p) || en_variants::known_variant(p, lang, known_core))) + english::is_correct(word, lang, opts, true) } /// Spelling suggestions in the English `lang`: this dialect's spelling first when `word` is the @@ -739,7 +673,7 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec { if let Some(p) = english_preferred(&lower, lang) { out.push(match_case(word, p)); } - let shared = if english_data() { suggest(word, max.saturating_add(2)) } else { english::suggest(word, max.saturating_add(2), lang) }; + let shared = english::suggest(word, max.saturating_add(2), lang); for s in shared { let s = english_preferred(&s.to_lowercase(), lang).map_or(s, |p| match_case(word, p)); if !out.contains(&s) { @@ -750,41 +684,6 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec { out } -/// Optimal string alignment distance, early-exit at 3. -pub(crate) fn damerau(a: &[char], b: &str) -> usize { - let b: Vec = b.chars().collect(); - let (n, m) = (a.len(), b.len()); - let mut prev2 = vec![0usize; m + 1]; - let mut prev: Vec = (0..=m).collect(); - let mut cur = vec![0usize; m + 1]; - for i in 1..=n { - if let Some(c) = cur.get_mut(0) { - *c = i; - } - let mut row_min = i; - for j in 1..=m { - let (ai, bj) = (a.get(i - 1), b.get(j - 1)); - let cost = usize::from(ai != bj); - let mut v = (prev.get(j).copied().unwrap_or(9) + 1) - .min(cur.get(j - 1).copied().unwrap_or(9) + 1) - .min(prev.get(j - 1).copied().unwrap_or(9) + cost); - if i > 1 && j > 1 && a.get(i - 1) == b.get(j - 2) && a.get(i - 2) == b.get(j - 1) { - v = v.min(prev2.get(j - 2).copied().unwrap_or(9) + 1); - } - if let Some(c) = cur.get_mut(j) { - *c = v; - } - row_min = row_min.min(v); - } - if row_min > 2 { - return 3; - } - std::mem::swap(&mut prev2, &mut prev); - std::mem::swap(&mut prev, &mut cur); - } - prev.get(m).copied().unwrap_or(9) -} - /// Words in `text`: (byte start, byte end). Combining marks belong to the word they follow, so a /// word typed decomposed (`tambie\u{301}n`, NFD) is one word, as its precomposed form is. pub fn words(text: &str) -> Vec<(usize, usize)> { @@ -1212,11 +1111,20 @@ mod tests { set_dictionaries_folder(Some(dir.to_path_buf())); } - /// R6.4, English never goes unchecked: with no embedded word list, an English Hunspell - /// dictionary (here in the dictionaries folder) is used before the word list built into the - /// crate; with one, the embedded list stays in use whatever is installed. + /// Language-data builds embed SCOWL: an English dictionary in the dictionaries folder would + /// then come before it for every English test running meanwhile, so the folder tests below + /// run without the data only; `english::tests` checks the order with the data. + fn scowl_embedded() -> bool { + scowl::dict(ProofLang::En).is_some() + } + + /// R6.4, English never goes unchecked: an English Hunspell dictionary in the dictionaries + /// folder is used before the word list built into the crate. #[test] fn english_hunspell_dictionary_is_used_before_the_built_in_list() { + if scowl_embedded() { + return; + } let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner); let dir = std::env::temp_dir().join(format!("wc-proof-en-hunspell-{}", std::process::id())); // A tiny en_US. `hous` and `because` aren't words in it, so a marked `because` is this @@ -1224,12 +1132,7 @@ mod tests { let english: &[(&str, &[u8], &[u8])] = &[("en_US", b"SET UTF-8\nTRY esiarntolcdugmphbyfvkwz\nSFX S Y 1\nSFX S 0 s .\n", b"3\nteh\nthe\nhouse/S\n")]; folder_with_english(&dir, english); - if english_data() { - assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Embedded)); - // (The Moby list has `teh`.) - assert!(!is_correct_in("becuase", ProofLang::En) && is_correct_in("because", ProofLang::En)); - assert_eq!(dictionary_load_error(ProofLang::En), None); - } else { + { let dic = dir.join("en_US.dic"); assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dic.clone()))); for w in ["teh", "Teh", "the", "house", "houses", "House"] { @@ -1258,9 +1161,7 @@ mod tests { assert!(check_text("hous teh", &[(0..8, ProofLang::De)]).is_empty()); assert_eq!(english_source(ProofLang::Pl), None); drop_english_files(&dir, english); - if !english_data() { - assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic")))); - } + assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic")))); } /// R6.4: an English Hunspell dictionary that is there but can't be read (an encoding we don't @@ -1269,16 +1170,16 @@ mod tests { /// still marks `becuase`. #[test] fn unreadable_english_dictionary_is_reported_and_skipped() { + if scowl_embedded() { + return; + } let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner); let dir = std::env::temp_dir().join(format!("wc-proof-en-unreadable-{}", std::process::id())); let english: &[(&str, &[u8], &[u8])] = &[("en_US", b"SET UTF-8\n", b"0\n"), ("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"), ("en_CA", b"SET UTF-8\n", b"1\n\xff\xfe\n")]; folder_with_english(&dir, english); for (lang, reason) in [(ProofLang::En, "no words"), (ProofLang::EnGb, "unsupported encoding"), (ProofLang::EnCa, "not UTF-8")] { - if english_data() { - assert_eq!(english_source(lang), Some(EnglishSource::Embedded)); - assert_eq!(dictionary_load_error(lang), None, "{lang:?}: never read"); - } else { + { // The built-in list, or a dictionary installed on the system (tried next). let source = english_source(lang); assert!(!matches!(&source, Some(EnglishSource::Hunspell(p)) if p.starts_with(&dir)), "{lang:?}: {source:?}"); @@ -1298,6 +1199,9 @@ mod tests { /// American English gets a dictionary that loads: the British tests don't hold the lock.) #[test] fn english_tries_its_next_dictionary_and_reads_real_world_directives() { + if scowl_embedded() { + return; + } let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner); let dir = std::env::temp_dir().join(format!("wc-proof-en-next-{}", std::process::id())); let aff: &[u8] = b"SET UTF-8\nICONV 1\nICONV \xe2\x80\x99 '\nOCONV 1\nOCONV ' \xe2\x80\x99\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\n\ @@ -1308,7 +1212,7 @@ mod tests { ("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"), ]; folder_with_english(&dir, english); - if !english_data() { + { let us = Some(EnglishSource::Hunspell(dir.join("en-US.dic"))); // A real en_US installed elsewhere (WORDCRAFT_DICTIONARIES, the system) comes first: // the same name in every folder before the next name. @@ -1754,6 +1658,149 @@ mod tests { assert!(check_spelling_in(text, gb).is_empty(), "{:?}", check_spelling_in(text, gb)); } + /// English spelling reads SCOWL 2026 through the affix engine: one dictionary per English, + /// its notice embedded, and its ordinals (`COMPOUNDRULE`) understood. + #[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")] + #[test] + fn english_reads_scowl() { + for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] { + let d = scowl::dict(l).unwrap_or_else(|| panic!("{l:?}: {:?}", scowl::load_error(l))); + assert!(d.len() > 40_000, "{l:?}: {} stems", d.len()); + assert_eq!(english_source(l), Some(EnglishSource::Embedded)); + assert_eq!(dictionary_load_error(l), None); + } + assert!(std::ptr::eq(scowl::dict(ProofLang::EnGb).unwrap(), scowl::dict(ProofLang::from_tag(Some("en-AU"))).unwrap()), "en-AU reads en-GB"); + let notice = scowl::notice().unwrap(); + for credit in ["Kevin Atkinson", "Geoff Kuenning", "clause 4 removed", "Benjamin Titze"] { + assert!(notice.contains(credit), "{credit}"); + } + // Ordinals are compounds of digit stems; `1th` is only a part. + let strict = Options { ignore_numbers: false, ..Options::default() }; + for w in ["1st", "21st", "102nd", "33rd", "11th", "112th", "2026th"] { + assert!(is_correct_in_with(w, ProofLang::En, &strict), "{w}"); + } + for w in ["1th", "21th", "11st", "3nd"] { + assert!(!is_correct_in_with(w, ProofLang::En, &strict), "{w}"); + } + // Affixed forms and possessives come from the affix rules. + for w in ["unrecognizable", "reorganization", "children's", "Mississippi's", "can't", "can’t"] { + assert!(is_correct_in(w, ProofLang::En), "{w}"); + } + assert!(!is_correct_in("mississippi", ProofLang::En), "a proper noun in lower case"); + // Suggestions come from the run's own dictionary. + assert!(suggest_in("colur", 5, ProofLang::EnGb).contains(&"colour".to_string()), "{:?}", suggest_in("colur", 5, ProofLang::EnGb)); + assert!(suggest_in("colr", 5, ProofLang::En).contains(&"color".to_string()), "{:?}", suggest_in("colr", 5, ProofLang::En)); + assert_eq!(suggest_in("organisation", 5, ProofLang::En).first().map(String::as_str), Some("organization")); + assert_eq!(suggest_in("color", 5, ProofLang::EnGb).first().map(String::as_str), Some("colour")); + } + + /// craft-libs#483 review: English suggestions offer a missing space only between two + /// lower-case words of three letters or more, after every one-word candidate, and no word the + /// dictionary only has capitalized (`Col`, `Wis`) for a lower-case word. + #[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")] + #[test] + fn english_suggestions_skip_junk_splits_and_names() { + for (w, l) in [ + ("colour", ProofLang::En), + ("theatr", ProofLang::En), + ("publically", ProofLang::En), + ("cheque", ProofLang::En), + ("colr", ProofLang::En), + ("wiss", ProofLang::En), + ("theatr", ProofLang::EnGb), + ] { + let got = suggest_in(w, 8, l); + assert!(!got.is_empty(), "{w}"); + assert!(got.iter().all(|s| !s.chars().any(char::is_uppercase)), "{w} in {l:?}: {got:?}"); + let first_split = got.iter().position(|s| s.contains(' ')).unwrap_or(got.len()); + assert!(got.iter().skip(first_split).all(|s| s.contains(' ')), "{w}: a split before a word: {got:?}"); + assert!(got.iter().filter(|s| s.contains(' ')).all(|s| s.split(' ').all(|p| p.chars().count() >= 3)), "{w}: {got:?}"); + } + assert_eq!(suggest_in("publically", 5, ProofLang::En).first().map(String::as_str), Some("publicly")); + assert_eq!(suggest_in("colour", 5, ProofLang::En).first().map(String::as_str), Some("color")); + assert_eq!(suggest_in("theatr", 5, ProofLang::EnGb).first().map(String::as_str), Some("theatre")); + assert!(suggest_in("Londn", 5, ProofLang::En).contains(&"London".to_string()), "a capitalized word may get a name"); + } + + /// `en-variants.txt` stays with SCOWL: SCOWL's size-60 lists keep one spelling per dialect, + /// and Word accepts more (both `-ize` and `-ise` in British English, `judgement` in American). + #[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")] + #[test] + fn dialect_policy_widens_scowl() { + let gb = scowl::dict(ProofLang::EnGb).unwrap(); + let us = scowl::dict(ProofLang::En).unwrap(); + assert!(!gb.check("organize") && gb.check("organise") && !us.check("judgement"), "SCOWL alone is stricter"); + assert!(is_correct_in("organize", ProofLang::EnGb) && is_correct_in("organise", ProofLang::EnGb)); + assert!(is_correct_in("judgement", ProofLang::En) && !is_correct_in("colour", ProofLang::En)); + assert!(is_correct_in("encyclopaedia", ProofLang::EnGb) && is_correct_in("encyclopedia", ProofLang::EnGb)); + } + + /// craft-libs#483 review: each English checks against its own SCOWL dictionary only. A stem + /// one English's dictionary has and another's lacks (`colourblind`, `greyscale` in en-GB; + /// `colorblind`, `esophagi` in en-US) doesn't cross, unless our dialect policy or the `-ize` + /// spellings of British English say so. Sampled from every pair of Englishes. + #[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")] + #[test] + fn scowl_dialects_dont_cross() { + let langs = [ + (ProofLang::En, crate::data::EN_US_SPELLING), + (ProofLang::EnGb, crate::data::EN_GB_SPELLING), + (ProofLang::EnCa, crate::data::EN_CA_SPELLING), + ]; + for (from, dic) in langs { + for (to, _) in langs { + if from == to { + continue; + } + let d = scowl::dict(to).unwrap(); + let widened = |w: &str| { + let x = english::ise_spelling(w); + (scowl::is_british(to) && x.as_deref().is_some_and(|x| d.check(x))) + || en_variants::known_variant(w, to, |x| d.check(x)) + || known_in_supplement(w) + }; + let only: Vec<&str> = dic + .lines() + .skip(1) + .map(|l| l.split('/').next().unwrap_or_default()) + .filter(|w| w.len() > 3 && w.chars().all(|c| c.is_ascii_lowercase()) && !d.check(w) && !widened(w)) + .collect(); + assert!(only.len() >= 40, "{from:?} → {to:?}: {}", only.len()); + let step = only.len() / 40; + let crossed: Vec<&str> = only.iter().step_by(step.max(1)).copied().filter(|w| is_correct_in(w, to)).collect(); + assert!(crossed.is_empty(), "{from:?} words right in {to:?}: {crossed:?}"); + } + } + for (w, l) in [("colourblind", ProofLang::En), ("greyscale", ProofLang::En), ("colorblind", ProofLang::EnGb), ("esophagi", ProofLang::EnGb)] { + assert!(!is_correct_in(w, l), "{w} in {l:?}"); + } + for (w, l) in + [("colorblind", ProofLang::En), ("colourblind", ProofLang::EnGb), ("organizational", ProofLang::EnGb), ("realized", ProofLang::EnGb)] + { + assert!(is_correct_in(w, l), "{w} in {l:?}"); + } + } + + /// craft-libs#483 re-review: British English takes the `-ize` spelling of an `-ise` word only + /// when American or Canadian English spells it so; a misspelling whose `-s-` form is a word + /// (`wize`, `advize`) stays wrong, as in Hunspell's en-GB. + #[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")] + #[test] + fn british_ize_needs_a_real_ize_word() { + let wrong = [ + "wize", "rize", "noize", "dizease", "advize", "rizing", "praize", "raize", "poize", "wizer", "surprize", "arize", "revize", "devize", + "mizer", "concize", "precize", "exercize", "chaize", + ]; + for l in [ProofLang::EnGb, ProofLang::from_tag(Some("en-AU"))] { + for w in wrong.iter().chain(&["colorize"]) { + assert!(!is_correct_in(w, l), "{w} in {l:?}"); + } + for w in ["organize", "realization", "colourize", "organizing", "realizes"] { + assert!(is_correct_in(w, l), "{w} in {l:?}"); + } + } + } + #[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")] #[test] fn user_dictionary_accepts_the_other_dialects_spelling() { diff --git a/crates/proof/src/no_data.rs b/crates/proof/src/no_data.rs index 0cab5eb9..e050bb60 100644 --- a/crates/proof/src/no_data.rs +++ b/crates/proof/src/no_data.rs @@ -45,6 +45,11 @@ const SAMPLES: [&str; 10] = [ #[test] fn nothing_is_embedded() { assert!(crate::data::EN_US_DIC.is_empty()); + assert!(crate::data::EN_US_SPELLING.is_empty() && crate::data::EN_GB_SPELLING.is_empty() && crate::data::EN_CA_SPELLING.is_empty()); + assert!(crate::scowl::notice().is_none()); + for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] { + assert!(crate::scowl::dict(l).is_none() && crate::scowl::load_error(l).is_none()); + } assert!(dict::Dictionary::en_us().is_empty()); for p in [patterns::Patterns::en_us(), patterns::Patterns::pl(), patterns::Patterns::pt(), patterns::Patterns::it(), patterns::Patterns::de()] { assert!(p.is_empty()); diff --git a/crates/proof/src/scowl.rs b/crates/proof/src/scowl.rs new file mode 100644 index 00000000..9a71188f --- /dev/null +++ b/crates/proof/src/scowl.rs @@ -0,0 +1,68 @@ +//! English spelling from SCOWL (Spell Checker Oriented Word Lists, Kevin Atkinson): the Hunspell +//! dictionaries of craft-i18n's `language-data/en-us`, `en-gb` (British `-ise`) and `en-ca`, +//! embedded unmodified at build time and read by the affix engine ([`crate::affix`]) on first use. +//! +//! Each English has its own: American English reads en-US, the British-spelling Englishes +//! (en-GB, en-AU, en-NZ, en-IE…) read en-GB and Canadian English reads en-CA. Its suggestions +//! come from its own dictionary, and a word is right when its own dictionary knows it: the other +//! dialects' dictionaries are never asked (en-US would take `colourblind` from en-GB's). SCOWL +//! picks one spelling per dialect (en-GB `-ise` has no `organize`, en-US no `judgement`), and Word +//! accepts more, so British English also takes the `-ize` spellings of its `-ise` words that +//! en-US or en-CA spells so, and `en-variants.txt` widens each dialect by its pairs +//! (`crate::english`). An English dictionary in the dictionaries folder comes first; a SCOWL +//! dictionary that isn't embedded, or fails to load, leaves that English to the next source (`crate::english`). +//! +//! SCOWL is under its own permissive licence and the affix files under Ispell's BSD licence; +//! [`notice`] is the upstream text, for an About window. + +use std::sync::OnceLock; + +use crate::ProofLang; +use crate::affix::AffixDict; +use crate::data::{EN_CA_AFF, EN_CA_SPELLING, EN_GB_AFF, EN_GB_SPELLING, EN_US_AFF, EN_US_SPELLING, SCOWL_NOTICE}; + +/// Per English (American, British, Canadian): the loaded dictionary, or why it isn't there. +static DICTS: [OnceLock>; 3] = [OnceLock::new(), OnceLock::new(), OnceLock::new()]; + +/// The slot, `.aff` and `.dic` text of `lang`'s dictionary. +fn source(lang: ProofLang) -> (usize, &'static str, &'static str) { + match lang { + ProofLang::EnGb => (1, EN_GB_AFF, EN_GB_SPELLING), + ProofLang::EnCa => (2, EN_CA_AFF, EN_CA_SPELLING), + _ => (0, EN_US_AFF, EN_US_SPELLING), + } +} + +/// `lang`'s SCOWL dictionary (an English), when it is embedded and loads. +pub(crate) fn dict(lang: ProofLang) -> Option<&'static AffixDict> { + let (i, aff, dic) = source(lang); + if aff.is_empty() || dic.is_empty() { + return None; + } + let loaded = DICTS.get(i)?.get_or_init(|| { + let d = AffixDict::from_hunspell(aff, dic, &["SCOWL (wordlist.aspell.net), embedded from craft-i18n"]) + .and_then(|d| if d.is_empty() { Err("no words".to_string()) } else { Ok(d) }); + if let Err(e) = &d { + log::error!("embedded SCOWL dictionary for {lang:?}: {e}; English spelling falls back to the next source"); + } + d + }); + loaded.as_ref().ok() +} + +/// Does `lang` read the British `-ise` dictionary (en-GB, en-AU, en-NZ…)? +pub(crate) fn is_british(lang: ProofLang) -> bool { + source(lang).0 == 1 +} + +/// Why `lang`'s embedded SCOWL dictionary failed to load, once something tried to load it. +pub(crate) fn load_error(lang: ProofLang) -> Option { + let (i, ..) = source(lang); + DICTS.get(i)?.get()?.as_ref().err().map(|e| format!("embedded SCOWL dictionary: {e}")) +} + +/// The SCOWL copyright and licence notices (the upstream README, with the Ispell BSD licence of +/// the affix files and Benjamin Titze's notice for the Australian data), when SCOWL is embedded. +pub fn notice() -> Option<&'static str> { + (!SCOWL_NOTICE.is_empty() && !EN_US_SPELLING.is_empty()).then_some(SCOWL_NOTICE) +}