Merge pull request #483 from benandez/feat/cp2-scowl

CP2.2: craft-proof reads SCOWL 2026 for English spelling
This commit is contained in:
Dylan M. Taylor authored and GitHub committed 2026-11-02 15:48:22 -05:00
commit 9e1ea140f3
11 files changed
+884 -276

No files matched your search

+3 -1
View File
@@ -25,10 +25,12 @@ jobs:
with:
persist-credentials: false
# The Language data (craft-i18n `language-data/`), embedded by crates/proof/build.rs.
# Pinned to storytold/craft-i18n#114's head (#109's SCOWL dictionaries + judgment/judgement,
# CP2.1); TODO(CP2.1): move to #114's merge commit once it merges.
- uses: actions/checkout@v4
with:
repository: storytold/craft-i18n
ref: 062a6375c322c88b96691a17f7b5fed71c51f7a8
ref: c239f934ace90b7939afaceb23b472ceeafd6b8a
path: craft-i18n
persist-credentials: false
- uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
+41 -14
View File
@@ -13,7 +13,7 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = "<commit>
| Language | Spelling | Grammar | Hyphenation |
|---|---|---|---|
| English (en-US, en-GB and the British-spelling Englishes, en-CA) | Moby word list + `en-extra.txt`, dialect variants (`en-variants.txt`); without them the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| English (en-US, en-CA, en-GB and the British-spelling Englishes such as en-AU, en-NZ) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns |
| Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise |
| German | — | — | TeX patterns |
@@ -63,20 +63,41 @@ data folder runs neither set.
English spelling uses the first of these that is there; `english_source(lang)` says which.
1. **Embedded:** the Moby word list from `CRAFT_I18N_DIR` (above). Nothing below is used then.
2. **System Hunspell:** the run's own English `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`;
each name in the dictionaries folder and then the system locations before the next name).
Never another English's: an `en_US` dictionary would mark British spellings, so en-GB and
en-CA without their own use the built-in list, which has both spellings. A file that can't be
read is skipped for the next name or folder; one file is read once for every English that uses
it. Compounding (`COMPOUNDMIN`, `COMPOUNDRULE`…, used by English dictionaries only for
ordinals) and `OCONV` are ignored for English rather than refused. None on the web.
3. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
1. **Dictionaries folder:** the English's own `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`)
that the user put in the folder set with `set_dictionaries_folder`, in `WORDCRAFT_DICTIONARIES`
or in WordCraft's own `dictionaries` folder. The user's choice comes first.
2. **Installed English pack:** reserved for Language packs (craft-libs#485, CP4.2).
3. **Embedded SCOWL:** the English's own SCOWL `rel-2026.02.25` Hunspell dictionary from
`CRAFT_I18N_DIR` (`language-data/en-us`, `en-gb` (British `-ise`) or `en-ca`, `spelling.aff` +
`spelling.dic`), read by the affix engine. en-AU, en-NZ, en-IE and the other British-spelling
Englishes read en-GB; Canadian English reads en-CA. A word is right when the English's own
dictionary knows it (or `en-extra.txt` does), never because another English's does
(`colourblind` is wrong in en-US, `colorblind` in en-GB); suggestions come from the same
dictionary. SCOWL's size-60 lists keep one spelling per dialect, and Word accepts more, so
British English also takes the `-ize` spellings of its `-ise` words that American or Canadian
English spell so (`organize`, `realization`; never `wize` or `advize`), and `en-variants.txt` flags the other dialect's spellings and accepts this
dialect's other spellings of known words (`judgement` in en-US). English with no dialect
(`is_correct`) takes a word any of the three knows. Suggestions offer a missing space only
between two lower-case words of three letters or more, after every one-word candidate, and no
name or abbreviation (`Col`) for a lower-case word. With SCOWL embedded the system locations aren't
searched.
4. **System Hunspell:** the English's own pair in the system locations. Never another English's:
an `en_US` dictionary would mark British spellings, so en-GB and en-CA without their own use
the built-in list, which has both spellings. Each name is looked for in every folder before the
next name; a file that can't be read is skipped for the next name or folder; one file is read
once for every English that uses it. `COMPOUNDRULE`, `ONLYINCOMPOUND` and `COMPOUNDMIN`
(ordinals such as `21st`) are read; flag-based compounding (`COMPOUNDFLAG`…) and `OCONV` are
ignored for English rather than refused. None on the web.
5. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
compressed, words only) built into the crate, with `data/en-extra.txt` (WordCraft's supplement
of modern words, a copy of craft-i18n's) and the same inflection rules as the embedded list. A
word it doesn't know is misspelled. Suggestions are the words one edit away (swap, missing,
added or changed letter), looked up in the list, for words of up to 32 letters: swaps first,
then common words, neighbouring keys, the same first letter.
of modern words, a copy of craft-i18n's) and the inflection rules. A word it doesn't know is
misspelled. Suggestions are the words one edit away (swap, missing, added or changed letter),
looked up in the list, for words of up to 32 letters: swaps first, then common words,
neighbouring keys, the same first letter.
The embedded Moby list (`language-data/en-us/en-us.dic`) is no longer a spelling source: it is
the same list as the built-in one, and stays as the hyphenation word list. `en-variants.txt`
applies to every source above when it is embedded.
A dictionary that is there but can't be read (encoding, size, parse error, no words) is skipped,
and `dictionary_load_error(lang)` gives the reason; it never panics. Text in other languages is
@@ -91,6 +112,12 @@ Regenerate with
`data/en-extra.txt` is WordCraft contributors' own list (MIT OR Apache-2.0); keep it in step with
craft-i18n's copy.
The SCOWL notice (SCOWL's licence, the Ispell BSD licence of the affix files and Benjamin Titze's
notice) is embedded with the dictionaries: `craft_proof::scowl::notice()`, for an About window.
Embedding SCOWL adds 1,542,396 bytes of Language data (the en-US, en-GB and en-CA `.aff`/`.dic`
and the notice); the Moby list (432,905 bytes) stays for hyphenation.
## Check
```sh
+11
View File
@@ -14,6 +14,10 @@
//!
//! `include_str!` takes a string, so a `CRAFT_I18N_DIR` that isn't valid UTF-8 can't be embedded:
//! it is treated like a folder without the data.
//!
//! English: SCOWL's en-US, en-GB (-ise) and en-CA dictionaries; en-AU is not embedded, as
//! Australian English is checked as British-spelling English (`ProofLang::EnGb`). The SCOWL
//! notice is embedded once (the four upstream READMEs differ only in their title and build line).
use std::fmt::Write as _;
use std::path::{Path, PathBuf};
@@ -21,6 +25,13 @@ use std::path::{Path, PathBuf};
/// (constant, path under `language-data/`, bytes rather than text).
const FILES: &[(&str, &str, bool)] = &[
("EN_US_DIC", "en-us/en-us.dic", true),
("EN_US_AFF", "en-us/spelling.aff", false),
("EN_US_SPELLING", "en-us/spelling.dic", false),
("EN_GB_AFF", "en-gb/spelling.aff", false),
("EN_GB_SPELLING", "en-gb/spelling.dic", false),
("EN_CA_AFF", "en-ca/spelling.aff", false),
("EN_CA_SPELLING", "en-ca/spelling.dic", false),
("SCOWL_NOTICE", "en-us/LICENCE-SCOWL.txt", false),
("EN_US_PAT", "en-us/hyphenation.pat", false),
("EXTRA_EN", "en-us/en-extra.txt", false),
("EN_VARIANTS", "en-us/en-variants.txt", false),
+285 -46
View File
@@ -17,11 +17,12 @@
//! (at most two suffixes in a row, as in Hunspell), `FORBIDDENWORD`, `NOSUGGEST`, `KEEPCASE` (the
//! stem is a word only in its own letter case), input conversion (`ICONV`: ligatures such as `fi`
//! read as `fi`) and ignored characters (`IGNORE`), and the suggestion tables `TRY`, `KEY`, `MAP`
//! and `REP`.
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: compounding,
//! long or numeric flags, `NEEDAFFIX` and the other directives that change which words are correct
//! (an English dictionary's compounding and `OCONV` are ignored instead:
//! [`AffixDict::from_hunspell_english`]).
//! and `REP`. Compounds by `COMPOUNDRULE` (patterns of one-character flags with `*` and `?`, as
//! SCOWL's English dictionaries use for ordinals: `21st`, `112th`), with `ONLYINCOMPOUND` and
//! `COMPOUNDMIN`; the parts of such a compound are stems, without affixes, as in Hunspell.
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: flag-based
//! compounding (`COMPOUNDFLAG`, `COMPOUNDBEGIN`…), long or numeric flags, `NEEDAFFIX` and the
//! other directives that change which words are correct.
//!
//! Lookups never panic: stems sit in one string, found through an open-addressing hash table, and
//! a probe allocates only small scratch strings for the candidate roots it tries.
@@ -48,16 +49,54 @@ const UNSUPPORTED: &[&str] = &[
"COMPOUNDFLAG",
"COMPOUNDLAST",
"COMPOUNDMIDDLE",
"COMPOUNDMIN",
"COMPOUNDPERMITFLAG",
"COMPOUNDRULE",
"FULLSTRIP",
"NEEDAFFIX",
"OCONV",
"ONLYINCOMPOUND",
"PSEUDOROOT",
];
/// Most `COMPOUNDRULE` patterns kept, and most tokens in one: a hostile `.aff` can't make a
/// compound check cost more than a few thousand stem lookups per word.
const MAX_COMPOUND_RULES: usize = 64;
const MAX_COMPOUND_TOKENS: usize = 32;
/// Longest word (in bytes) tried as a compound: each try looks up every substring.
const MAX_COMPOUND_BYTES: usize = 48;
/// How often a `COMPOUNDRULE` token's flag may repeat.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Repeat {
Once,
/// `?`
Optional,
/// `*`
Any,
}
/// A `COMPOUNDRULE` pattern as written (`n*1t`): its tokens, each a flag and how often it repeats.
fn parse_compound_rule(pattern: &str) -> Result<Vec<(char, Repeat)>, String> {
let mut out: Vec<(char, Repeat)> = Vec::new();
for c in pattern.chars().filter(|&c| c != EMOJI_SELECTOR) {
match c {
'*' | '?' => {
let Some(last) = out.last_mut().filter(|t| t.1 == Repeat::Once) else {
return Err(format!("COMPOUNDRULE `{pattern}`: `{c}` follows no flag"));
};
last.1 = if c == '*' { Repeat::Any } else { Repeat::Optional };
}
'(' | ')' => return Err(format!("COMPOUNDRULE `{pattern}`: ( ) groups (long flags) are not supported")),
c => out.push((c, Repeat::Once)),
}
if out.len() > MAX_COMPOUND_TOKENS {
return Err(format!("COMPOUNDRULE `{pattern}`: more than {MAX_COMPOUND_TOKENS} flags"));
}
}
if out.is_empty() {
return Err("COMPOUNDRULE with no flags".into());
}
Ok(out)
}
/// Most distinct flags a dictionary may use: every single-byte flag (the Hungarian dictionary
/// uses 158).
const MAX_FLAGS: usize = 256;
@@ -272,6 +311,18 @@ pub struct AffixDict {
ignore: Box<[char]>,
/// Provenance and licence lines, kept through [`AffixDict::to_bytes`].
comments: Vec<String>,
/// `COMPOUNDRULE` patterns: per token, the flag's bit and how often it may repeat.
compound_rules: Vec<Vec<(Mask, Repeat)>>,
/// The same patterns as written, for [`AffixDict::to_bytes`].
compound_text: Vec<String>,
/// The `ONLYINCOMPOUND` flag's bit: such stems are words only as parts of a compound.
only_in_compound: Mask,
/// `COMPOUNDMIN`, in characters.
compound_min: usize,
/// The first characters of the stems a compound part can be (their flags are in a
/// `COMPOUNDRULE`), so a word starting otherwise is no compound: SCOWL's ordinals start with
/// a digit, and a misspelled word costs one lookup here instead of one per cut.
compound_firsts: Vec<char>,
}
fn hash_str(s: &str) -> u64 {
@@ -341,10 +392,14 @@ struct Rules {
substandard: Option<char>,
/// The `KEEPCASE` flag.
keepcase: Option<char>,
/// The `ONLYINCOMPOUND` flag, read for English only ([`AffixDict::from_hunspell_english`]).
only_in_compound: Option<char>,
iconv: Vec<(String, String)>,
ignore: Vec<char>,
/// `COMPOUNDRULE` patterns, as written (`n*1t`).
compound_rules: Vec<String>,
/// The `ONLYINCOMPOUND` flag.
only_in_compound: Option<char>,
/// `COMPOUNDMIN`: the shortest part of a compound, in characters (Hunspell's default is 3).
compound_min: Option<usize>,
/// `AF` flag aliases: alias `n` (1-based) stands for `aliases[n - 1]`.
aliases: Vec<String>,
/// The `AF` table opened with its count line, so flag fields are alias numbers. A table
@@ -426,26 +481,15 @@ fn zero_empty(s: &str) -> String {
/// but it is not a flag of its own: it is dropped wherever flags are read.
const EMOJI_SELECTOR: char = '\u{fe0f}';
/// Directives an English dictionary may carry that don't change which English words are correct
/// here, ignored by [`AffixDict::from_hunspell_english`]: compounding (English dictionaries use
/// it only for ordinals such as `1st`, which have digits and are skipped by the spelling check)
/// and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
const IGNORED_IN_ENGLISH: &[&str] = &[
"COMPOUNDBEGIN",
"COMPOUNDEND",
"COMPOUNDFIRST",
"COMPOUNDFLAG",
"COMPOUNDLAST",
"COMPOUNDMIDDLE",
"COMPOUNDMIN",
"COMPOUNDPERMITFLAG",
"COMPOUNDRULE",
"OCONV",
];
/// Directives an English dictionary may carry that this engine doesn't implement but that don't
/// change which English words are right, ignored by [`AffixDict::from_hunspell_english`]:
/// flag-based compounding (English dictionaries compound only ordinals, by `COMPOUNDRULE`, which
/// is read) and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
const IGNORED_IN_ENGLISH: &[&str] =
&["COMPOUNDBEGIN", "COMPOUNDEND", "COMPOUNDFIRST", "COMPOUNDFLAG", "COMPOUNDLAST", "COMPOUNDMIDDLE", "COMPOUNDPERMITFLAG", "OCONV"];
/// Read Hunspell `.aff` text (already decoded to UTF-8). `english`: ignore the directives in
/// [`IGNORED_IN_ENGLISH`], and keep the `ONLYINCOMPOUND` flag so the stems that are only parts of
/// compounds can be left out.
/// [`IGNORED_IN_ENGLISH`] instead of refusing them.
fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
let mut r = Rules::default();
let mut cross: HashMap<(bool, char), bool> = HashMap::new();
@@ -453,6 +497,7 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
let mut rep_header = false;
let mut af_header = false;
let mut iconv_header = false;
let mut compound_header = false;
for (n, line) in aff.lines().enumerate() {
let f: Vec<&str> = split_fields(line).collect();
let Some(&dir) = f.first() else { continue };
@@ -487,6 +532,23 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
}
"IGNORE" => r.ignore = at(1).chars().collect(),
"KEEPCASE" => r.keepcase = Some(one_flag(at(1), dir, n)?),
// The first `COMPOUNDRULE` line is the table's count, as in Hunspell.
"COMPOUNDRULE" => {
if !compound_header && at(1).parse::<usize>().is_ok() {
compound_header = true;
} else {
parse_compound_rule(at(1)).map_err(|e| format!("line {}: {e}", n + 1))?;
if r.compound_rules.len() >= MAX_COMPOUND_RULES {
return Err(format!("line {}: more than {MAX_COMPOUND_RULES} COMPOUNDRULE patterns", n + 1));
}
r.compound_rules.push(at(1).to_string());
}
}
"ONLYINCOMPOUND" => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
"COMPOUNDMIN" => {
let min = at(1).parse::<usize>().map_err(|_| format!("line {}: COMPOUNDMIN needs a number", n + 1))?;
r.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES));
}
// Per-character flag modes (`FLAG UTF-8`, as VERO's pt_BR dictionary uses) work
// with the one-character parsing here; only the multi-character encodings differ.
"FLAG" if !matches!(at(1), "UTF-8" | "DEFAULT" | "utf-8" | "default" | "UTF8" | "") => {
@@ -541,7 +603,6 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
r.prefixes.push(rule);
}
}
"ONLYINCOMPOUND" if english => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
d if english && IGNORED_IN_ENGLISH.contains(&d) => {}
d if UNSUPPORTED.contains(&d) => return Err(format!("line {}: {d} is not supported", n + 1)),
_ => {}
@@ -570,10 +631,10 @@ impl AffixDict {
AffixDict::from_hunspell_in(aff, dic, comments, false)
}
/// [`AffixDict::from_hunspell`] for an English dictionary: compounding and output conversion
/// (`COMPOUNDMIN`, `COMPOUNDRULE`, `OCONV`…), which other languages need but English spelling
/// doesn't, are ignored rather than refused, and the stems marked `ONLYINCOMPOUND` (parts of
/// compounds only) are left out.
/// [`AffixDict::from_hunspell`] for an English dictionary: flag-based compounding and output
/// conversion (`COMPOUNDFLAG`, `OCONV`…), which this engine doesn't implement and English
/// spelling doesn't need, are ignored rather than refused. `COMPOUNDRULE`, `ONLYINCOMPOUND`
/// and `COMPOUNDMIN` (ordinals: `21st`) are read, as for every language.
pub fn from_hunspell_english(aff: &str, dic: &str, comments: &[&str]) -> Result<AffixDict, String> {
AffixDict::from_hunspell_in(aff, dic, comments, true)
}
@@ -619,9 +680,6 @@ impl AffixDict {
continue;
}
let flags = rules.flags(&flags).to_string();
if rules.only_in_compound.is_some_and(|f| flags.contains(f)) {
continue;
}
stems.push((word, flags));
}
AffixDict::build(rules, stems, comments.iter().map(ToString::to_string).collect())
@@ -631,6 +689,7 @@ impl AffixDict {
// Flags: every one a rule or a stem uses gets a bit.
let mut flag_chars: Vec<char> = rules.prefixes.iter().chain(&rules.suffixes).map(|r| r.0).collect();
flag_chars.extend(stems.iter().flat_map(|s| s.1.chars()));
// A compound rule's flag no stem carries gets no bit: its token matches no part.
flag_chars.sort_unstable();
flag_chars.dedup();
if flag_chars.len() > MAX_FLAGS {
@@ -656,9 +715,18 @@ impl AffixDict {
map: rules.map,
rep: rules.rep,
comments,
compound_rules: rules
.compound_rules
.iter()
.map(|p| parse_compound_rule(p).map(|t| t.into_iter().map(|(c, rep)| (bit_of(c), rep)).collect()))
.collect::<Result<_, _>>()?,
compound_text: rules.compound_rules,
only_in_compound: rules.only_in_compound.map_or(Mask::EMPTY, bit_of),
compound_min: rules.compound_min.unwrap_or(3),
..AffixDict::default()
};
d.offs.push(0);
let compound_flags = d.compound_rules.iter().flatten().fold(Mask::EMPTY, |m, (bit, _)| m | *bit);
let mut set_ids: HashMap<Mask, u16> = HashMap::new();
let mut i = 0;
while i < stems.len() {
@@ -676,6 +744,12 @@ impl AffixDict {
if usize::from(id) == d.sets.len() {
d.sets.push(mask);
}
if mask.meets(compound_flags)
&& let Some(c) = word.chars().next()
&& !d.compound_firsts.contains(&c)
{
d.compound_firsts.push(c);
}
d.words.push_str(word);
d.offs.push(u32::try_from(d.words.len()).map_err(|_| "stems too large".to_string())?);
d.flags.push(id);
@@ -797,6 +871,11 @@ impl AffixDict {
/// Flag mask of a stem, if `w` is one and carries none of the flags in `deny`.
fn stem_flags_except(&self, w: &str, deny: Mask) -> Option<Mask> {
self.stem_mask(w).filter(|m| !m.meets(self.forbidden | self.only_in_compound | deny))
}
/// Flag mask of the stem `w`, whatever flags it carries.
fn stem_mask(&self, w: &str) -> Option<Mask> {
if self.table.is_empty() {
return None;
}
@@ -810,9 +889,9 @@ impl AffixDict {
if self.word(t as usize) == Some(w) {
let id = *self.flags.get(t as usize)?;
let mask = self.sets.get(usize::from(id)).copied();
// A stem carrying FORBIDDENWORD is listed but is not a word: no direct hit,
// and (this is the only stem lookup) no affixed form is built on it either.
return mask.filter(|m| !m.meets(self.forbidden | deny));
// [`Self::stem_flags_except`] drops FORBIDDENWORD stems (listed, not words) and
// ONLYINCOMPOUND ones (words only inside a compound, [`Self::compound`]).
return mask;
}
slot = (slot + 1) & mask;
}
@@ -928,7 +1007,41 @@ impl AffixDict {
return true;
}
}
false
self.compound(w, deny)
}
/// Is `w` a compound of two or more stems (no affixes) whose flags spell out one of the
/// `COMPOUNDRULE` patterns, each part at least `COMPOUNDMIN` characters? `21st` is `2/m` +
/// `1st/p` under `n*mp`. A dynamic programme over (byte offset, token, parts so far), so a
/// hostile pattern costs at most a few thousand lookups. A forbidden word is never rebuilt
/// from its parts.
fn compound(&self, w: &str, deny: Mask) -> bool {
if self.compound_rules.is_empty()
|| w.len() > MAX_COMPOUND_BYTES
|| !w.chars().next().is_some_and(|c| self.compound_firsts.contains(&c))
|| self.stem_mask(w).is_some_and(|m| m.meets(self.forbidden))
{
return false;
}
// Char boundaries of `w`, `0` and `w.len()` included.
let cuts: Vec<usize> = w.char_indices().map(|(i, _)| i).chain(std::iter::once(w.len())).collect();
let n = cuts.len();
// Flag mask of each part `w[cuts[a]..cuts[b]]`, by (a, b), looked up when a rule first
// reaches cut `a` (`None`: not looked up yet; `Some(None)`: not a stem). Only the parts
// after a cut a rule reaches are looked up, so a word with no first part costs `n`.
let mut parts: Vec<Option<Option<Mask>>> = vec![None; n * n];
let min = self.compound_min;
let mut part = |a: usize, b: usize| -> Option<Mask> {
if b < a.saturating_add(min) {
return None;
}
let slot = parts.get_mut(a * n + b)?;
*slot.get_or_insert_with(|| {
let (Some(&x), Some(&y)) = (cuts.get(a), cuts.get(b)) else { return None };
w.get(x..y).and_then(|p| self.stem_mask(p)).filter(|m| !m.meets(self.forbidden | deny))
})
};
self.compound_rules.iter().any(|rule| compound_matches(rule, &mut part, n))
}
/// Is `w` a stem with one suffix? With `need` set (a prefix was taken off), the stem must
@@ -1166,6 +1279,17 @@ impl AffixDict {
if !self.ignore.is_empty() {
out.push_str(&format!("IGNORE {}\n", self.ignore.iter().collect::<String>()));
}
if !self.only_in_compound.is_empty()
&& let Some(c) = self.flag_char(self.only_in_compound)
{
out.push_str(&format!("ONLYINCOMPOUND {c}\n"));
}
if !self.compound_text.is_empty() {
out.push_str(&format!("COMPOUNDMIN {}\n", self.compound_min));
for p in &self.compound_text {
out.push_str(&format!("COMPOUNDRULE {p}\n"));
}
}
let z = |s: &str| if s.is_empty() { "0".to_string() } else { s.to_string() };
for (kind, rules) in [("PFX", &self.prefixes), ("SFX", &self.suffixes)] {
for r in rules {
@@ -1233,6 +1357,20 @@ impl AffixDict {
}
"ICONV" if f.len() == 3 => rules.iconv.push((at(1).to_string(), at(2).to_string())),
"IGNORE" => rules.ignore = at(1).chars().collect(),
"COMPOUNDMIN" => {
let min = at(1).parse::<usize>().map_err(|_| format!("affix dictionary: bad `{line}`"))?;
rules.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES));
}
"ONLYINCOMPOUND" => {
rules.only_in_compound = Some(one_flag(at(1), "ONLYINCOMPOUND", 0).map_err(|_| format!("affix dictionary: bad `{line}`"))?);
}
"COMPOUNDRULE" => {
parse_compound_rule(at(1)).map_err(|e| format!("affix dictionary: {e}"))?;
if rules.compound_rules.len() >= MAX_COMPOUND_RULES {
return Err("affix dictionary: too many COMPOUNDRULE patterns".into());
}
rules.compound_rules.push(at(1).to_string());
}
"MAP" => rules.map.push(at(1).chars().collect()),
"REP" => rules.rep.push((at(1).replace('_', " "), at(2).replace('_', " "))),
kind @ ("PFX" | "SFX") => {
@@ -1287,6 +1425,39 @@ impl AffixDict {
}
}
/// Does a compound rule's token list match a split of the word into two or more parts? `part(a,
/// b)` is the flags of the stem between char cuts `a` and `b` (`None`: not a stem). States are
/// (cut, token, parts so far capped at 2), each visited once.
fn compound_matches(rule: &[(Mask, Repeat)], part: &mut impl FnMut(usize, usize) -> Option<Mask>, n: usize) -> bool {
let t = rule.len();
let idx = |cut: usize, tok: usize, k: usize| (cut * (t + 1) + tok) * 3 + k;
let mut seen = vec![false; n * (t + 1) * 3];
let mut stack = vec![(0usize, 0usize, 0usize)];
while let Some((cut, tok, k)) = stack.pop() {
match seen.get_mut(idx(cut, tok, k)) {
Some(s) if !*s => *s = true,
_ => continue,
}
if tok == t {
if cut + 1 == n && k >= 2 {
return true;
}
continue;
}
let Some(&(bit, repeat)) = rule.get(tok) else { continue };
if repeat != Repeat::Once {
stack.push((cut, tok + 1, k));
}
for next in cut + 1..n {
if part(cut, next).is_some_and(|m| m.meets(bit)) {
let k = (k + 1).min(2);
stack.push((next, if repeat == Repeat::Any { tok } else { tok + 1 }, k));
}
}
}
false
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
@@ -1779,6 +1950,73 @@ pub(crate) mod tests {
}
}
/// SCOWL's ordinal rules (its English `.aff`, verbatim): `21st`, `112th` and `1000th` are
/// compounds of digit stems, `1th` is only a part, and nothing else compounds.
const ORDINALS_AFF: &str = "SET UTF-8\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nWORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s .\n";
const ORDINALS_DIC: &str = "13\n0/nm\n0th/pt\n1/n1\n1st/p\n1th/tc\n2/nm\n2nd/p\n2th/tc\n3/nm\n3rd/p\n3th/tc\n4/nm\n4th/pt\ncat/S\n";
#[test]
fn compound_rules_make_ordinals() {
let d = AffixDict::from_hunspell(ORDINALS_AFF, ORDINALS_DIC, &[]).unwrap();
for w in ["1st", "2nd", "4th", "21st", "22nd", "23rd", "11th", "12th", "13th", "111th", "1000th", "104th", "cats"] {
assert!(d.check(w), "{w}");
}
for w in ["1th", "2th", "11st", "12nd", "13rd", "21th", "catcat", "cat1st", "1stcat", "1st1st", "", "x"] {
assert!(!d.check(w), "{w}");
}
// Only-in-compound parts are never suggested on their own.
assert!(!d.can_suggest("1th") && d.forms("1th").is_empty());
// The compact format keeps the rules.
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
assert!(back.check("21st") && back.check("111th") && !back.check("1th") && !back.check("21th"));
}
#[test]
fn compound_rules_need_two_parts_and_compoundmin() {
// `COMPOUNDMIN` defaults to 3; a pattern a lone stem matches still needs two parts.
let aff = "SET UTF-8\nONLYINCOMPOUND c\nCOMPOUNDRULE a*b?\n";
let d = AffixDict::from_hunspell(aff, "3\nfoo/a\nbar/ab\nqux/c\n", &[]).unwrap();
assert!(d.check("foobar") && d.check("foofoo") && d.check("foofoofoobar") && d.check("foo"));
assert!(!d.check("qux") && !d.check("quxfoo") && !d.check("fooqux") && !d.check("foobarx"));
let d = AffixDict::from_hunspell(aff, "2\nab/ab\ncd/a\n", &[]).unwrap();
assert!(!d.check("abcd"), "parts shorter than COMPOUNDMIN 3");
let d = AffixDict::from_hunspell(&format!("{aff}COMPOUNDMIN 2\n"), "2\nab/ab\ncd/a\n", &[]).unwrap();
assert!(d.check("cdab"));
}
/// craft-libs#483 review: a forbidden word isn't rebuilt from its parts.
#[test]
fn compound_never_rebuilds_a_forbidden_word() {
let d = AffixDict::from_hunspell("SET UTF-8\nFORBIDDENWORD X\nCOMPOUNDMIN 1\nCOMPOUNDRULE a*\n", "2\nx/a\nxx/X\n", &[]).unwrap();
assert!(!d.check("xx") && d.check("xxx") && d.check("x"));
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
assert!(!back.check("xx") && back.check("xxx"));
}
/// craft-libs#483 review: `ONLYINCOMPOUND` survives a round trip without compound rules too.
#[test]
fn onlyincompound_round_trips_without_compound_rules() {
let d = AffixDict::from_hunspell("SET UTF-8\nONLYINCOMPOUND c\n", "2\nfoo\nqux/c\n", &[]).unwrap();
assert!(d.check("foo") && !d.check("qux"));
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
assert!(back.check("foo") && !back.check("qux"), "a part-only stem stays a part");
}
#[test]
fn hostile_compound_rules_are_refused_or_cheap() {
for bad in ["COMPOUNDRULE *a", "COMPOUNDRULE (ab)", "COMPOUNDRULE a**", "COMPOUNDMIN x", "ONLYINCOMPOUND ab"] {
assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{bad}\n"), DIC, &[]).is_err(), "{bad}");
}
let many: String = (0..100).map(|_| "COMPOUNDRULE a*\n").collect();
assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{many}"), DIC, &[]).is_err());
// Every part a stem and a pattern of stars: still answers at once.
let star: String = std::iter::repeat_n("a*", 16).collect();
let d = AffixDict::from_hunspell(&format!("SET UTF-8\nCOMPOUNDMIN 1\nCOMPOUNDRULE {star}b\n"), "1\nx/a\n", &[]).unwrap();
let t = std::time::Instant::now();
assert!(!d.check(&"x".repeat(48)) && !d.check(&"x".repeat(200)));
assert!(t.elapsed().as_secs() < 2);
}
#[test]
fn fields_split_on_spaces_and_tabs_only() {
// A flag byte read as U+00A0 (no-break space) or U+0085 is a flag, not a separator.
@@ -1796,22 +2034,23 @@ pub(crate) mod tests {
}
}
/// craft-libs#482 review: an English dictionary with compounding (for ordinals) and output
/// conversion, as the usual en_US and en_GB ones have, loads for English; its `ONLYINCOMPOUND`
/// stems are left out. Other languages still refuse those directives. (A synthetic `.aff`
/// written for this test.)
/// craft-libs#482 review: an English dictionary with compounding and output conversion, as the
/// usual en_US and en_GB ones have, loads for English: `COMPOUNDRULE` makes its ordinals, its
/// `ONLYINCOMPOUND` stems are words only inside them, and `COMPOUNDFLAG` and `OCONV` are
/// ignored. Other languages still refuse those two. (A synthetic `.aff` written for this test.)
#[test]
fn english_dictionaries_ignore_compounding_and_output_conversion() {
let aff = "SET UTF-8\nTRY esianrtolcdugmphbyfvkwz'\nICONV 1\nICONV ’ '\nOCONV 1\nOCONV ' ’\nNOSUGGEST !\n\
COMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nCOMPOUNDFLAG Z\n\
WORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s [^sxzhy]\nSFX D Y 1\nSFX D 0 ed [^ey]\n";
let dic = "5\nhouse/S\nwalk/SD\n0th/pc\n1st/c\nthe\n";
let dic = "7\nhouse/S\nwalk/SD\n0th/pc\n1st/p\n1th/tc\n1/n1\nthe\n";
assert!(AffixDict::from_hunspell(aff, dic, &[]).is_err(), "refused for other languages");
let d = AffixDict::from_hunspell_english(aff, dic, &[]).unwrap();
for w in ["house", "houses", "walked", "walks", "the"] {
assert!(d.check(w), "{w}");
}
assert!(!d.check("0th") && !d.check("1st"), "parts of compounds only");
assert!(!d.check("0th") && !d.check("1th"), "parts of compounds only");
assert!(d.check("1st") && d.check("11th") && d.check("111th") && !d.check("11st"), "ordinals by COMPOUNDRULE");
assert!(!d.check("housed") && !d.check("hous"));
// The directives that change which words are right are still refused.
for dir in ["NEEDAFFIX", "CIRCUMFIX", "PSEUDOROOT", "FULLSTRIP"] {
+31 -4
View File
@@ -43,6 +43,8 @@ const COST_SWAP: u32 = 14;
const COST_DROP_OR_ADD: u32 = 15;
pub(crate) const COST_MISTYPE: u32 = 16;
const COST_SCAN: u32 = 20;
/// A missing space ranked after every one-word candidate (English: [`Split::Custom`]).
pub(crate) const COST_SPLIT_LAST: u32 = COST_SCAN + 4 * 3 + 1;
/// What a language adds to its dictionary's tables.
pub(crate) struct Tuning {
@@ -62,6 +64,9 @@ pub(crate) struct Tuning {
pub map_cost: fn(char, char) -> u32,
/// The cost of typing letter `c` at `i` of the word (`chars`, lower case) instead.
pub mistype_cost: fn(&[char], usize, char) -> u32,
/// Offer a word only the dictionary capitalizes (a name, an abbreviation: `Col`, `Wis`) only
/// when the word itself is capitalized, and never as half of a missing space (`Che Que`).
pub strict_case: bool,
}
/// Where a missing space may go.
@@ -78,8 +83,17 @@ pub(crate) enum Split {
impl Tuning {
/// The dictionary's tables alone (Czech, Indonesian, Spanish).
pub(crate) const DEFAULT: Tuning =
Tuning { split: Split::Anywhere, alphabet: "", map: &[], rep: &[], rep_cost: COST_REP, max_map_changes: 4, map_cost, mistype_cost };
pub(crate) const DEFAULT: Tuning = Tuning {
split: Split::Anywhere,
alphabet: "",
map: &[],
rep: &[],
rep_cost: COST_REP,
max_map_changes: 4,
map_cost,
mistype_cost,
strict_case: false,
};
}
/// The cost of changing letter `from` to its MAP relative `to`. Adding a missing accent is the
@@ -107,6 +121,10 @@ struct Collector<'a> {
/// Suggestion → (cost, order found).
found: HashMap<String, (u32, usize)>,
seq: usize,
/// May a word the dictionary only has capitalized be offered ([`Tuning::strict_case`])?
names: bool,
/// May half of a missing space be one ([`Tuning::strict_case`])?
split_names: bool,
}
impl Collector<'_> {
@@ -125,9 +143,12 @@ impl Collector<'_> {
}
let mut words = Vec::new();
let mut cost = cost;
let names = if cand.contains(' ') { self.split_names } else { self.names };
for part in cand.split(' ') {
if self.d.contains_suggestible(part) {
words.push(part.to_string());
} else if !names {
return;
} else {
let c = capitalize(part);
if !self.d.contains_suggestible(&c) {
@@ -278,7 +299,9 @@ pub(crate) fn suggest_in(d: &AffixDict, word: &str, max: usize, t: &Tuning) -> V
let case = casing(word);
let input = word.to_lowercase();
let chars: Vec<char> = input.chars().collect();
let mut col = Collector { d, input: input.clone(), tested: HashMap::new(), found: HashMap::new(), seq: 0 };
let names = !t.strict_case || case != Casing::Lower;
let split_names = !t.strict_case;
let mut col = Collector { d, input: input.clone(), tested: HashMap::new(), found: HashMap::new(), seq: 0, names, split_names };
add_case(&mut col, word, &input);
let map = groups(d, t);
add_map_slips(&mut col, &chars, &map, t);
@@ -479,6 +502,9 @@ fn scan(col: &mut Collector, chars: &[char], map: &[Vec<char>]) {
if lf.len().abs_diff(chars.len()) > 2 {
continue;
}
if !col.names && form.chars().any(char::is_uppercase) {
continue;
}
let dist = weighted_distance(&lf, chars, &related, limit);
if dist <= limit && lf.as_slice() != chars {
col.keep(form, COST_SCAN + dist * 3);
@@ -504,7 +530,8 @@ mod tests {
#[test]
fn a_cheaper_later_offer_wins() {
let d = AffixDict::from_hunspell("", "1\ncafe\n", &[]).unwrap();
let mut col = Collector { d: &d, input: "cafè".into(), tested: HashMap::new(), found: HashMap::new(), seq: 0 };
let mut col =
Collector { d: &d, input: "cafè".into(), tested: HashMap::new(), found: HashMap::new(), seq: 0, names: true, split_names: true };
col.offer("cafe", COST_DROP_DIACRITIC);
col.offer("cafe", COST_REP);
col.offer("cafe", COST_SCAN);
+221 -58
View File
@@ -1,29 +1,40 @@
//! English spelling never goes unchecked. Its words come from the first of these that is there:
//!
//! 1. **Embedded**: the Moby word list embedded from craft-i18n at build time (`CRAFT_I18N_DIR`,
//! see `build.rs`). When it is there, nothing in this module is used.
//! 2. **System Hunspell**: a Hunspell `.aff`/`.dic` pair for the run's own English found by
//! [`crate::hunspell`] (`en_US` for American, `en_GB` for British, `en_CA` for Canadian
//! English; each name in the dictionaries folder and then the system locations before the
//! next name; none on the web). Another English's dictionary is never used: it would mark this
//! English's spellings (`colour` in an `en_US` one), so without its own dictionary an English
//! uses the built-in list, which has both spellings. A file that can't be read is skipped for
//! the next name or folder, and one file is read once for every English that uses it.
//! Directives that only matter for other languages (compounding, output conversion) are
//! ignored ([`crate::hunspell::load_english`]).
//! 3. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
//! 1. **Dictionaries folder**: a Hunspell `.aff`/`.dic` pair for the run's own English that the
//! user keeps for WordCraft ([`crate::hunspell::user_dirs`]: the folder set in the options,
//! `WORDCRAFT_DICTIONARIES`, WordCraft's own `dictionaries` folder).
//! 2. **Installed English pack**: reserved for Language packs (craft-libs#485, CP4.2).
//! 3. **Embedded SCOWL**: the run's English's SCOWL dictionary (en-US, en-GB or en-CA) embedded
//! from craft-i18n at build time ([`crate::scowl`]); a word it knows (British English: also
//! the `-ize` spelling of one it knows, [`ise_spelling`]), or the supplement ([`EXTRA`]), is
//! right; never one only another English knows. When it is there, the system locations aren't
//! searched.
//! 4. **System Hunspell**: a pair for the run's own English in the system locations (none on the
//! web).
//! 5. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
//! the crate (without its break points), with WordCraft's own supplement of modern words
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled, as with the
//! embedded list.
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled. (The Moby list
//! embedded from craft-i18n is the same list; it is used for hyphenation only.)
//!
//! A Hunspell dictionary that is there but can't be read (unsupported encoding, too large, a
//! parse error, no words) is skipped, and its reason is reported by
//! Hunspell dictionaries are found by [`crate::hunspell`] by their own English's names only
//! (`en_US` for American, `en_GB` for British, `en_CA` for Canadian English), each name in every
//! folder before the next name. Another English's dictionary is never used: it would mark this
//! English's spellings (`colour` in an `en_US` one). A file that can't be read is skipped for the
//! next name or folder, and one file is read once for every English that uses it. `COMPOUNDRULE`
//! (ordinals) is read; flag-based compounding and `OCONV` are ignored
//! ([`crate::hunspell::load_english`]).
//!
//! Whichever it is, the dialect policy of `en-variants.txt` ([`crate::en_variants`], when it is
//! embedded) flags the other dialect's spellings and accepts this dialect's spellings of known
//! words. A Hunspell dictionary that is there but can't be read (unsupported encoding, too large,
//! a parse error, no words) is skipped, and its reason is reported by
//! [`crate::dictionary_load_error`]. Non-English text is never checked against any of these.
use std::path::{Path, PathBuf};
use std::sync::{Arc, OnceLock, PoisonError, RwLock};
use crate::affix::AffixDict;
use crate::affix_suggest::{Split, Tuning};
use crate::dict::Dictionary;
use crate::{Options, ProofLang};
@@ -47,7 +58,7 @@ const MAX_CANDIDATES: usize = 8;
/// Where English spelling comes from now ([`crate::english_source`]).
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum EnglishSource {
/// The Moby word list embedded from craft-i18n at build time.
/// The English's SCOWL dictionary embedded from craft-i18n at build time.
Embedded,
/// An English Hunspell dictionary from the dictionaries folder or the system: its `.dic` file.
Hunspell(PathBuf),
@@ -112,14 +123,9 @@ impl State {
static HUNSPELL: RwLock<State> = RwLock::new(State::new());
/// Why the embedded word list failed to decode, if it did.
static EMBEDDED_ERROR: OnceLock<String> = OnceLock::new();
/// Remember that the embedded word list is there but failed to decode (English then uses the
/// next source).
/// The embedded Moby list (hyphenation) failed to decode: log it. Spelling doesn't use it.
pub(crate) fn note_embedded_error(e: String) {
log::error!("embedded English word list: {e}; English spelling falls back to Hunspell or the built-in word list");
let _ = EMBEDDED_ERROR.set(e);
log::error!("embedded English word list: {e}; English hyphenation falls back to the patterns");
}
/// The English Hunspell dictionary for `lang` (read on first use).
@@ -185,12 +191,14 @@ fn key(lang: ProofLang) -> &'static str {
}
/// Read the first Hunspell pair of `lang`'s own that loads, trying each name in each folder
/// ([`crate::hunspell::candidates`]), with the first failure before it (the file `lang` would
/// rather use), for [`note_error`].
/// ([`crate::hunspell::candidates`]; with SCOWL embedded, the dictionaries folder only:
/// [`crate::hunspell::user_candidates`]), with the first failure before it (the file `lang`
/// would rather use), for [`note_error`].
fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) {
let candidates = crate::hunspell::candidates(names(lang));
let candidates =
if crate::scowl::dict(lang).is_some() { crate::hunspell::user_candidates(names(lang)) } else { crate::hunspell::candidates(names(lang)) };
if candidates.is_empty() {
log::info!("English spelling: no Hunspell {} dictionary installed; using the built-in word list", key(lang));
log::info!("English spelling: no Hunspell {} dictionary to read first; using SCOWL or the built-in word list", key(lang));
return (None, None);
}
let mut first_error: Option<(PathBuf, String)> = None;
@@ -209,7 +217,7 @@ fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) {
}
}
}
log::warn!("English spelling: no {} Hunspell dictionary could be read; using the built-in word list", key(lang));
log::warn!("English spelling: no {} Hunspell dictionary could be read; using SCOWL or the built-in word list", key(lang));
(None, first_error)
}
@@ -232,45 +240,150 @@ pub(crate) fn reload() {
}
}
/// Where the spelling of `lang` (an English) comes from now. Reads the Hunspell dictionary once
/// when there is no embedded data (it is cached afterwards).
/// Where the spelling of `lang` (an English) comes from now, in the order of the module docs.
/// Reads the Hunspell dictionary once (it is cached afterwards).
pub(crate) fn source(lang: ProofLang) -> EnglishSource {
if crate::english_data() {
EnglishSource::Embedded
} else {
hunspell(lang).map_or(EnglishSource::CoreList, |(_, dic)| EnglishSource::Hunspell(dic))
match speller(lang) {
Speller::Hunspell(_, dic) => EnglishSource::Hunspell(dic),
Speller::Scowl(..) => EnglishSource::Embedded,
Speller::List => EnglishSource::CoreList,
}
}
/// Why a source of `lang`'s spelling failed to load: the embedded word list, or (without it) the
/// Hunspell dictionary found for it. Reads nothing itself.
/// Why a source of `lang`'s spelling failed to load: its embedded SCOWL dictionary, or the
/// Hunspell dictionary it would rather have used. Reads nothing itself.
pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
if let Some(e) = EMBEDDED_ERROR.get() {
return Some(e.clone());
}
if crate::english_data() {
return None;
}
crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason)
crate::scowl::load_error(lang).or_else(|| crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason))
}
/// Is `word` spelled right in `lang` (an English) when no word list is embedded: against the
/// Hunspell dictionary, or else the built-in word list.
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options) -> bool {
/// What `lang` is checked against now.
enum Speller {
/// A dictionaries-folder or system Hunspell dictionary, and its `.dic` file.
Hunspell(Arc<AffixDict>, PathBuf),
/// `lang`'s own SCOWL dictionary (every English's, for English with no dialect: [`generic`]),
/// and whether British `-ize` spellings are right too ([`ise_spelling`]).
Scowl(Vec<&'static AffixDict>, bool),
/// The built-in word list.
List,
}
fn speller(lang: ProofLang) -> Speller {
pick(hunspell(lang), lang)
}
/// The lookup order: the Hunspell dictionary `found` for `lang` (with SCOWL embedded, only one in
/// the dictionaries folder is looked for), then SCOWL, then the built-in list. (The installed
/// English pack, craft-libs#485, goes between the first two.)
fn pick(found: Loaded, lang: ProofLang) -> Speller {
if let Some((d, dic)) = found {
return Speller::Hunspell(d, dic);
}
match crate::scowl::dict(lang) {
Some(d) => Speller::Scowl(vec![d], crate::scowl::is_british(lang)),
None => Speller::List,
}
}
/// [`speller`] for English with no dialect ([`crate::is_correct`]): a SCOWL word of any English
/// is right, as in the built-in list, which has both spellings.
fn generic(s: Speller) -> Speller {
match s {
Speller::Scowl(..) => {
let all: Vec<_> = [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa].into_iter().filter_map(crate::scowl::dict).collect();
Speller::Scowl(all, false)
}
other => other,
}
}
impl Speller {
/// Is `w` (as written, `’` read as `'`) a word: as a whole, or each part of a hyphenated one?
fn knows(&self, w: &str) -> bool {
match self {
Speller::Hunspell(d, _) => affix_correct(&[d.as_ref()], w, false),
Speller::Scowl(ds, ise) => {
affix_correct(ds, w, true)
|| (*ise
&& ise_spelling(w).is_some_and(|x| affix_correct(ds, &x, true))
&& ize_elsewhere(w, |d, p| affix_correct(&[d], p, false)))
}
Speller::List => list_correct(w),
}
}
/// Is the lower-case base form `p` a word (for the dialect policy's other spelling)?
fn knows_lower(&self, p: &str) -> bool {
match self {
Speller::Hunspell(d, _) => d.check(p),
Speller::Scowl(ds, ise) => {
ds.iter().any(|d| d.check(p) || (*ise && ise_spelling(p).is_some_and(|x| d.check(&x)) && ize_elsewhere(p, |d, p| d.check(p))))
}
Speller::List => crate::known_in_word_list(p),
}
}
}
/// Is `word` spelled right in `lang` (an English), against its own dictionary ([`speller`]); with
/// `policy` false, English with no dialect: any English's SCOWL dictionary ([`generic`]). With
/// `policy`, also by the dialect policy of `en-variants.txt`: the other dialect's spellings are
/// wrong and this dialect's spellings of known words right. The user dictionary overrules both.
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options, policy: bool) -> bool {
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’').trim_matches(['\'', '’']);
if crate::accepted_by_options(w, opts) || !w.chars().any(|c| c.is_ascii_alphabetic()) {
return true; // other scripts: not English
}
let w = w.replace('’', "'");
match hunspell(lang) {
Some((d, _)) => hunspell_correct(&d, &w),
None => list_correct(&w),
let lower = w.to_lowercase();
let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect();
if policy && parts.iter().any(|p| crate::en_variants::is_foreign(p, lang)) {
return false;
}
let s = if policy { speller(lang) } else { generic(speller(lang)) };
s.knows(&w)
|| (policy && !parts.is_empty() && parts.iter().all(|p| s.knows_lower(p) || crate::en_variants::known_variant(p, lang, |x| s.knows_lower(x))))
}
/// Against an installed English Hunspell dictionary: the word, or each part of a hyphenated one.
fn hunspell_correct(d: &AffixDict, w: &str) -> bool {
let known = |p: &str| p.chars().count() <= 1 || d.check(p) || crate::in_user_dictionary(&p.to_lowercase());
/// `w` with its `-ize`/`-yze` endings (and their forms: `-ization`, `-izing`…) spelled `-ise`/
/// `-yse`, when it has any: British English takes both (Word accepts `organize` and `organise` in
/// en-GB), and SCOWL's en-GB dictionary has only `-ise`. A `z` between `i`/`y` and a vowel is
/// swapped; the `-ize` form counts only when American or Canadian English spells the word so
/// ([`ize_elsewhere`]), or `wize` would pass as `wise`.
pub(crate) fn ise_spelling(w: &str) -> Option<String> {
let b = w.as_bytes();
let mut out = b.to_vec();
let mut changed = false;
for i in 1..b.len() {
let after = b.get(i + 1).is_some_and(|c| b"aeiAEI".contains(c));
if after
&& matches!(b.get(i - 1), Some(b'i' | b'y' | b'I' | b'Y'))
&& let Some(c) = out.get_mut(i)
{
match *c {
b'z' => *c = b's',
b'Z' => *c = b'S',
_ => continue,
}
changed = true;
}
}
if changed { String::from_utf8(out).ok() } else { None }
}
/// Is `w` a word in American or Canadian English's SCOWL dictionary (`check` asks one): the
/// `-ize` spellings British English takes are theirs (`organize`, never `advize` or `wize`).
fn ize_elsewhere(w: &str, check: impl Fn(&AffixDict, &str) -> bool) -> bool {
[ProofLang::En, ProofLang::EnCa].into_iter().filter_map(crate::scowl::dict).any(|d| check(d, w))
}
/// Against affix dictionaries (any of them): the word, or each part of a hyphenated one; with
/// `supplement`, our supplement's words count too.
fn affix_correct(ds: &[&AffixDict], w: &str, supplement: bool) -> bool {
let known = |p: &str| {
p.chars().count() <= 1
|| ds.iter().any(|d| d.check(p))
|| crate::in_user_dictionary(&p.to_lowercase())
|| (supplement && crate::known_in_supplement(&p.to_lowercase()))
};
known(w) || (w.contains('-') && w.split('-').filter(|p| !p.is_empty()).all(known))
}
@@ -285,15 +398,25 @@ fn list_correct(w: &str) -> bool {
crate::known_in_word_list(&lower) || (lower.contains('-') && lower.split('-').filter(|p| !p.is_empty()).all(crate::known_in_word_list))
}
/// Spelling suggestions in `lang` (an English) when no word list is embedded: from the Hunspell
/// dictionary, or else the built-in word list.
/// Spelling suggestions in `lang` (an English), from its dictionary ([`speller`]).
pub(crate) fn suggest(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
match hunspell(lang) {
Some((d, _)) => crate::cs::suggest::suggest_in(&d, word, max),
None => list_suggest(word, max),
match speller(lang) {
Speller::Hunspell(d, _) => crate::affix_suggest::suggest_in(&d, word, max, &TUNING),
Speller::Scowl(ds, _) => ds.first().map_or_else(Vec::new, |d| crate::affix_suggest::suggest_in(d, word, max, &TUNING)),
Speller::List => list_suggest(word, max),
}
}
/// English suggestions from an affix dictionary: its own tables, a missing space only between two
/// words of three letters or more and after every one-word candidate (`publically` → `publicly`
/// before `public ally`), and no abbreviation or name offered for a lower-case word (`col`, `Wis`).
const TUNING: Tuning = Tuning { split: Split::Custom(split_cost), strict_case: true, ..Tuning::DEFAULT };
/// The cost of a missing space in English ([`TUNING`]).
fn split_cost(a: &str, b: &str) -> Option<u32> {
(a.chars().count() >= 3 && b.chars().count() >= 3).then_some(crate::affix_suggest::COST_SPLIT_LAST)
}
/// Letters an edit may add or put in a word's place.
const ALPHABET: &str = "abcdefghijklmnopqrstuvwxyz'";
@@ -460,6 +583,46 @@ mod tests {
assert!(start.elapsed() < std::time::Duration::from_secs(5), "{:?}", start.elapsed());
}
/// The lookup order (README): a dictionaries-folder English comes before SCOWL, SCOWL before
/// the built-in list; each English looks for its own dictionary only, and with SCOWL embedded
/// the system locations aren't searched. (Checked here without touching the process-wide
/// folder, which every English test running meanwhile would see.)
#[test]
fn lookup_order() {
let tiny = Arc::new(AffixDict::from_hunspell("", "1\nzzyzx\n", &[]).unwrap());
let file = PathBuf::from("/folder/en_US.dic");
for lang in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
assert!(matches!(pick(Some((tiny.clone(), file.clone())), lang), Speller::Hunspell(_, p) if p == file), "{lang:?}");
let fallback = pick(None, lang);
if crate::scowl::dict(lang).is_some() {
assert!(
matches!(&fallback, Speller::Scowl(ds, _) if ds.len() == 1 && std::ptr::eq(ds[0], crate::scowl::dict(lang).unwrap())),
"{lang:?}"
);
assert!(fallback.knows("21st") && !fallback.knows("becuase"), "{lang:?}");
assert_eq!(fallback.knows("colour"), lang != ProofLang::En, "{lang:?}: its own dictionary only");
} else {
assert!(matches!(fallback, Speller::List), "{lang:?}");
}
}
let dir = std::env::temp_dir().join(format!("wc-proof-en-order-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
for name in ["en_US", "en_GB", "en_CA"] {
std::fs::write(dir.join(format!("{name}.aff")), "SET UTF-8\n").unwrap();
std::fs::write(dir.join(format!("{name}.dic")), "1\nzzyzx\n").unwrap();
}
let found = |names: &[&str]| -> Vec<String> {
crate::hunspell::candidates_in(std::slice::from_ref(&dir), names)
.iter()
.filter_map(|(_, d)| d.file_stem().map(|n| n.to_string_lossy().into_owned()))
.collect()
};
assert_eq!(found(names(ProofLang::EnGb)), ["en_GB"], "a user's en_US doesn't stand in for SCOWL's en-GB");
assert_eq!(found(names(ProofLang::EnCa)), ["en_CA"]);
assert_eq!(found(names(ProofLang::En)), ["en_US"]);
let _ = std::fs::remove_dir_all(&dir);
}
/// craft-libs#482 review: a load that started before a reload (the dictionaries folder
/// changed) serves its own check but isn't kept, so the next check reads the new folder; a
/// load after it is kept, and shared with the Englishes that read the same file.
+19
View File
@@ -154,6 +154,25 @@ pub fn candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
candidates_in(&search_dirs(lo.as_deref()), names)
}
/// [`candidates`] in the folders the user keeps dictionaries in for WordCraft ([`user_dirs`])
/// only, not the system locations.
pub fn user_candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
candidates_in(&user_dirs(), names)
}
/// The folders of [`search_dirs`] the user keeps dictionaries in for WordCraft: the one set with
/// [`set_dictionary_dir`], `WORDCRAFT_DICTIONARIES`, and WordCraft's own `dictionaries` folder
/// under the user's application data. The rest are the system's (and LibreOffice's).
pub fn user_dirs() -> Vec<PathBuf> {
let extra = EXTRA_DIR.read().unwrap_or_else(std::sync::PoisonError::into_inner).clone();
let env = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()).map(PathBuf::from);
let wordcraft_own = |d: &Path| {
d.file_name().is_some_and(|n| n == "dictionaries")
&& d.parent().and_then(Path::file_name).is_some_and(|n| n.eq_ignore_ascii_case("wordcraft"))
};
search_dirs(None).into_iter().filter(|d| Some(d) == extra.as_ref() || Some(d) == env.as_ref() || wordcraft_own(d)).collect()
}
/// [`candidates`] in `dirs`.
pub fn candidates_in(dirs: &[PathBuf], names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
let mut out: Vec<(PathBuf, PathBuf)> = Vec::new();
+1 -1
View File
@@ -6,7 +6,7 @@
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum ProofLang {
/// American English (`en`, `en-US`, and every region not listed for [`ProofLang::EnGb`] or
/// [`ProofLang::EnCa`]): the public-domain Moby word list, the English grammar rules and the
/// [`ProofLang::EnCa`]): SCOWL's en-US dictionary ([`crate::scowl`]), the English grammar rules and the
/// English hyphenation dictionary and patterns. British spellings (`colour`, `centre`,
/// `organise`) are flagged, with the American form suggested ([`crate::en_variants`]). Also
/// text with no language at all (WordCraft's default editing language is US English, like
+199 -152
View File
@@ -1,8 +1,9 @@
//! Proofing for the Craft Apps (moved from WordCraft's `wordcraft-proof`), per language. A run's language (its BCP 47 tag) picks the tools through
//! [`ProofLang`]; the `*_in` functions take it, the older English-only ones remain.
//!
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the
//! public-domain Moby word list (~160k words, the same list that drives hyphenation) plus our
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the SCOWL
//! 2026 Hunspell dictionary of the run's English ([`scowl`]; else the public-domain Moby word
//! list, ~160k words, the same list that drives hyphenation) plus our
//! supplement of modern words it lacks (craft-i18n's `language-data/en-us/en-extra.txt`: `desktop`, `email`,
//! `download`…), with inflections (`-s`, `-es`, `-ed`, `-ing`, `-ly`, `'s`…), a user dictionary and ignore lists;
//! suggestions by edit distance. Grammar checks that are cheap and reliable: repeated words,
@@ -48,11 +49,12 @@
//!
//! [`check_text`] checks a paragraph whose runs are in different languages.
//!
//! **Language data** (the Moby word list, the pattern files, our word supplements) is embedded at
//! build time from craft-i18n's `language-data/` (`CRAFT_I18N_DIR`, see `build.rs`). English
//! spelling never goes unchecked: without the embedded word list it uses an English Hunspell
//! dictionary installed on the system, and without one the whole Moby word list built into the
//! crate (see [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
//! **Language data** (the SCOWL English dictionaries, the Moby word list, the pattern files, our
//! word supplements) is embedded at build time from craft-i18n's `language-data/`
//! (`CRAFT_I18N_DIR`, see `build.rs`). English spelling never goes unchecked: a dictionaries-folder
//! English first, then the embedded SCOWL dictionary, then an English Hunspell dictionary
//! installed on the system, then the whole Moby word list built into the crate (see
//! [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
//! rule-based hyphenation (Bulgarian, Turkish, Russian) are code and still work.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -78,6 +80,7 @@ pub mod patterns;
pub mod pl;
pub mod pt;
pub mod ru;
pub mod scowl;
mod slot;
mod spelling;
pub mod tr;
@@ -315,7 +318,7 @@ pub fn installed_dictionaries() -> Vec<(ProofLang, Option<std::path::PathBuf>)>
/// Why the dictionary found for `lang` failed to load (unsupported encoding, too large, a parse
/// error), if it was read and failed. Doesn't read anything itself: a dictionary nobody has
/// checked yet reports `None` (#2830). For an English, the embedded word list or (without it) the
/// checked yet reports `None` (#2830). For an English, its embedded SCOWL dictionary or the
/// English Hunspell dictionary that failed; English then uses the next source ([`english_source`]).
pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
if matches!(lang, ProofLang::En | ProofLang::EnGb | ProofLang::EnCa) {
@@ -324,12 +327,11 @@ pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
hunspell::load_error(language(lang)?.slot.name()).map(|(_, reason)| reason)
}
/// Where the spelling of an English `lang` comes from now (`None` for other languages): the word
/// list embedded at build time, else an English Hunspell dictionary from the dictionaries folder
/// or the system (the run's English first), else the Moby word list built into the crate.
/// Each English uses only its own Hunspell dictionary (`en_US`, `en_GB`, `en_CA`): without it,
/// the built-in list, which has both spellings. Reads the Hunspell dictionary once when there is no
/// embedded word list (it is cached afterwards).
/// Where the spelling of an English `lang` comes from now (`None` for other languages), in this
/// order: a Hunspell dictionary of its own (`en_US`, `en_GB`, `en_CA`) in the dictionaries
/// folder, the SCOWL dictionary embedded at build time, one of its own on the system, else the
/// Moby word list built into the crate. Another English's dictionary is never used. Reads the
/// Hunspell dictionary once (it is cached afterwards).
pub fn english_source(lang: ProofLang) -> Option<EnglishSource> {
lang.is_english().then(|| english::source(lang))
}
@@ -467,7 +469,7 @@ pub(crate) fn is_common_word(w: &str) -> bool {
}
/// Our supplement of common words the Moby list lacks (`desktop`, `email`, `download`…): one
/// lowercase base form per line, `#` comments. Inflections come from [`known_core`].
/// lowercase base form per line, `#` comments. Inflections come from [`known_with`].
use data::EXTRA_EN;
/// The supplement's words: the Language data's copy, else the crate's own ([`english::EXTRA`]).
@@ -477,21 +479,18 @@ fn extra_words() -> &'static HashSet<&'static str> {
E.get_or_init(|| extra.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect())
}
/// A base form in the dictionary or the supplement.
fn base_known(d: &dict::Dictionary, w: &str) -> bool {
d.get(w).is_some() || extra_words().contains(w)
}
fn known_core(w: &str) -> bool {
let d = dict::Dictionary::en_us();
known_with(w, &|s| base_known(d, s))
}
/// [`known_core`] against the word list built into the crate (no embedded word list).
/// Is `w` (lower case) a word of the built-in Moby list or the supplement, with the inflection
/// rules ([`known_with`])?
pub(crate) fn known_in_word_list(w: &str) -> bool {
known_with(w, &|s| english::word_list_contains(s) || extra_words().contains(s))
}
/// Is `w` (lower case) a word of the supplement alone, with the inflection rules (the SCOWL
/// dictionaries take it as an addition)?
pub(crate) fn known_in_supplement(w: &str) -> bool {
known_with(w, &|s| extra_words().contains(s))
}
/// Is `w` (lower case) a word, by `base_known` (a base form in the word list), the common words,
/// the user dictionary and the inflection and prefix rules?
fn known_with(w: &str, base_known: &dyn Fn(&str) -> bool) -> bool {
@@ -615,66 +614,16 @@ pub(crate) fn accepted_by_options(word: &str, opts: &Options) -> bool {
|| in_user_dictionary(&w.to_lowercase())
}
/// Is the English word list embedded? Without it English is checked against an installed
/// Hunspell dictionary or the word list built into the crate ([`english`]).
pub(crate) fn english_data() -> bool {
!dict::Dictionary::en_us().is_empty()
}
/// [`is_correct`] with the File › Options › Proofing choices.
/// [`is_correct`] with the File › Options › Proofing choices: English with no dialect, so a word
/// any English spells so is right (`color` and `colour`; American English's dictionary when it
/// isn't SCOWL: [`english_source`]), without the dialect policy.
pub fn is_correct_with(word: &str, opts: &Options) -> bool {
if !english_data() {
return english::is_correct(word, ProofLang::En, opts);
}
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'');
let w = w.trim_matches('\'');
if accepted_by_options(w, opts) {
return true;
}
if !w.is_ascii() && !w.chars().any(|c| c.is_ascii_alphabetic()) {
return true; // other scripts: no dictionary
}
let lower = w.to_lowercase().replace('’', "'");
if known_core(&lower) {
return true;
}
// Hyphenated compounds: each part.
if lower.contains('-') {
return lower.split('-').all(|p| p.is_empty() || known_core(p));
}
false
english::is_correct(word, ProofLang::En, opts, false)
}
/// Spelling suggestions for a misspelled word (best first, at most `max`).
pub fn suggest(word: &str, max: usize) -> Vec<String> {
if !english_data() {
return english::suggest(word, max, ProofLang::En);
}
suggest_from(word, max, dict::Dictionary::en_us().iter().map(|(w, _)| w))
}
/// [`suggest`] from `words` (lower case) and our supplement.
pub(crate) fn suggest_from<'a>(word: &str, max: usize, words: impl Iterator<Item = &'a str>) -> Vec<String> {
let lower = word.to_lowercase();
let n = lower.chars().count();
if n == 0 || n > 40 {
return Vec::new();
}
let mut scored: Vec<(usize, i64, String)> = Vec::new();
let target: Vec<char> = lower.chars().collect();
for w in words.chain(extra_words().iter().copied()) {
let wn = w.chars().count();
if wn + 2 < n || wn > n + 2 {
continue;
}
let dist = damerau(&target, w);
if dist <= 2 {
let same_first = w.chars().next() == target.first().copied();
scored.push((dist, if same_first { 0 } else { 1 }, w.to_string()));
}
}
scored.sort_by(|a, b| (a.0, a.1, a.2.len().abs_diff(n)).cmp(&(b.0, b.1, b.2.len().abs_diff(n))).then(a.2.cmp(&b.2)));
scored.into_iter().take(max).map(|(_, _, w)| match_case(word, w)).collect()
english::suggest(word, max, ProofLang::En)
}
/// `w` (lower case) in the case of `word`: all capitals, an initial capital, or as it is.
@@ -709,26 +658,11 @@ fn english_preferred(lower: &str, lang: ProofLang) -> Option<String> {
}
/// Is `word` spelled right in the English `lang` ([`ProofLang::En`], [`ProofLang::EnGb`],
/// [`ProofLang::EnCa`])? The shared word list's words, less the other dialect's spellings
/// [`ProofLang::EnCa`])? Its dictionary's words ([`english`]), less the other dialect's spellings
/// (`colour` in American English, `color` in British), plus this dialect's spellings of the
/// list's words ([`en_variants`]). The user dictionary overrules both.
/// dictionary's words ([`en_variants`]). The user dictionary overrules both.
fn is_correct_english(word: &str, lang: ProofLang, opts: &Options) -> bool {
if !english_data() {
return english::is_correct(word, lang, opts);
}
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'').trim_matches('\'');
if accepted_by_options(w, opts) {
return true;
}
let lower = w.to_lowercase().replace('’', "'");
if in_user_dictionary(&lower) {
return true;
}
let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect();
if parts.iter().any(|p| en_variants::is_foreign(p, lang)) {
return false;
}
is_correct_with(word, opts) || (!parts.is_empty() && parts.iter().all(|p| known_core(p) || en_variants::known_variant(p, lang, known_core)))
english::is_correct(word, lang, opts, true)
}
/// Spelling suggestions in the English `lang`: this dialect's spelling first when `word` is the
@@ -739,7 +673,7 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
if let Some(p) = english_preferred(&lower, lang) {
out.push(match_case(word, p));
}
let shared = if english_data() { suggest(word, max.saturating_add(2)) } else { english::suggest(word, max.saturating_add(2), lang) };
let shared = english::suggest(word, max.saturating_add(2), lang);
for s in shared {
let s = english_preferred(&s.to_lowercase(), lang).map_or(s, |p| match_case(word, p));
if !out.contains(&s) {
@@ -750,41 +684,6 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
out
}
/// Optimal string alignment distance, early-exit at 3.
pub(crate) fn damerau(a: &[char], b: &str) -> usize {
let b: Vec<char> = b.chars().collect();
let (n, m) = (a.len(), b.len());
let mut prev2 = vec![0usize; m + 1];
let mut prev: Vec<usize> = (0..=m).collect();
let mut cur = vec![0usize; m + 1];
for i in 1..=n {
if let Some(c) = cur.get_mut(0) {
*c = i;
}
let mut row_min = i;
for j in 1..=m {
let (ai, bj) = (a.get(i - 1), b.get(j - 1));
let cost = usize::from(ai != bj);
let mut v = (prev.get(j).copied().unwrap_or(9) + 1)
.min(cur.get(j - 1).copied().unwrap_or(9) + 1)
.min(prev.get(j - 1).copied().unwrap_or(9) + cost);
if i > 1 && j > 1 && a.get(i - 1) == b.get(j - 2) && a.get(i - 2) == b.get(j - 1) {
v = v.min(prev2.get(j - 2).copied().unwrap_or(9) + 1);
}
if let Some(c) = cur.get_mut(j) {
*c = v;
}
row_min = row_min.min(v);
}
if row_min > 2 {
return 3;
}
std::mem::swap(&mut prev2, &mut prev);
std::mem::swap(&mut prev, &mut cur);
}
prev.get(m).copied().unwrap_or(9)
}
/// Words in `text`: (byte start, byte end). Combining marks belong to the word they follow, so a
/// word typed decomposed (`tambie\u{301}n`, NFD) is one word, as its precomposed form is.
pub fn words(text: &str) -> Vec<(usize, usize)> {
@@ -1212,11 +1111,20 @@ mod tests {
set_dictionaries_folder(Some(dir.to_path_buf()));
}
/// R6.4, English never goes unchecked: with no embedded word list, an English Hunspell
/// dictionary (here in the dictionaries folder) is used before the word list built into the
/// crate; with one, the embedded list stays in use whatever is installed.
/// Language-data builds embed SCOWL: an English dictionary in the dictionaries folder would
/// then come before it for every English test running meanwhile, so the folder tests below
/// run without the data only; `english::tests` checks the order with the data.
fn scowl_embedded() -> bool {
scowl::dict(ProofLang::En).is_some()
}
/// R6.4, English never goes unchecked: an English Hunspell dictionary in the dictionaries
/// folder is used before the word list built into the crate.
#[test]
fn english_hunspell_dictionary_is_used_before_the_built_in_list() {
if scowl_embedded() {
return;
}
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-hunspell-{}", std::process::id()));
// A tiny en_US. `hous` and `because` aren't words in it, so a marked `because` is this
@@ -1224,12 +1132,7 @@ mod tests {
let english: &[(&str, &[u8], &[u8])] =
&[("en_US", b"SET UTF-8\nTRY esiarntolcdugmphbyfvkwz\nSFX S Y 1\nSFX S 0 s .\n", b"3\nteh\nthe\nhouse/S\n")];
folder_with_english(&dir, english);
if english_data() {
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Embedded));
// (The Moby list has `teh`.)
assert!(!is_correct_in("becuase", ProofLang::En) && is_correct_in("because", ProofLang::En));
assert_eq!(dictionary_load_error(ProofLang::En), None);
} else {
{
let dic = dir.join("en_US.dic");
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dic.clone())));
for w in ["teh", "Teh", "the", "house", "houses", "House"] {
@@ -1258,9 +1161,7 @@ mod tests {
assert!(check_text("hous teh", &[(0..8, ProofLang::De)]).is_empty());
assert_eq!(english_source(ProofLang::Pl), None);
drop_english_files(&dir, english);
if !english_data() {
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
}
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
}
/// R6.4: an English Hunspell dictionary that is there but can't be read (an encoding we don't
@@ -1269,16 +1170,16 @@ mod tests {
/// still marks `becuase`.
#[test]
fn unreadable_english_dictionary_is_reported_and_skipped() {
if scowl_embedded() {
return;
}
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-unreadable-{}", std::process::id()));
let english: &[(&str, &[u8], &[u8])] =
&[("en_US", b"SET UTF-8\n", b"0\n"), ("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"), ("en_CA", b"SET UTF-8\n", b"1\n\xff\xfe\n")];
folder_with_english(&dir, english);
for (lang, reason) in [(ProofLang::En, "no words"), (ProofLang::EnGb, "unsupported encoding"), (ProofLang::EnCa, "not UTF-8")] {
if english_data() {
assert_eq!(english_source(lang), Some(EnglishSource::Embedded));
assert_eq!(dictionary_load_error(lang), None, "{lang:?}: never read");
} else {
{
// The built-in list, or a dictionary installed on the system (tried next).
let source = english_source(lang);
assert!(!matches!(&source, Some(EnglishSource::Hunspell(p)) if p.starts_with(&dir)), "{lang:?}: {source:?}");
@@ -1298,6 +1199,9 @@ mod tests {
/// American English gets a dictionary that loads: the British tests don't hold the lock.)
#[test]
fn english_tries_its_next_dictionary_and_reads_real_world_directives() {
if scowl_embedded() {
return;
}
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-next-{}", std::process::id()));
let aff: &[u8] = b"SET UTF-8\nICONV 1\nICONV \xe2\x80\x99 '\nOCONV 1\nOCONV ' \xe2\x80\x99\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\n\
@@ -1308,7 +1212,7 @@ mod tests {
("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"),
];
folder_with_english(&dir, english);
if !english_data() {
{
let us = Some(EnglishSource::Hunspell(dir.join("en-US.dic")));
// A real en_US installed elsewhere (WORDCRAFT_DICTIONARIES, the system) comes first:
// the same name in every folder before the next name.
@@ -1754,6 +1658,149 @@ mod tests {
assert!(check_spelling_in(text, gb).is_empty(), "{:?}", check_spelling_in(text, gb));
}
/// English spelling reads SCOWL 2026 through the affix engine: one dictionary per English,
/// its notice embedded, and its ordinals (`COMPOUNDRULE`) understood.
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn english_reads_scowl() {
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
let d = scowl::dict(l).unwrap_or_else(|| panic!("{l:?}: {:?}", scowl::load_error(l)));
assert!(d.len() > 40_000, "{l:?}: {} stems", d.len());
assert_eq!(english_source(l), Some(EnglishSource::Embedded));
assert_eq!(dictionary_load_error(l), None);
}
assert!(std::ptr::eq(scowl::dict(ProofLang::EnGb).unwrap(), scowl::dict(ProofLang::from_tag(Some("en-AU"))).unwrap()), "en-AU reads en-GB");
let notice = scowl::notice().unwrap();
for credit in ["Kevin Atkinson", "Geoff Kuenning", "clause 4 removed", "Benjamin Titze"] {
assert!(notice.contains(credit), "{credit}");
}
// Ordinals are compounds of digit stems; `1th` is only a part.
let strict = Options { ignore_numbers: false, ..Options::default() };
for w in ["1st", "21st", "102nd", "33rd", "11th", "112th", "2026th"] {
assert!(is_correct_in_with(w, ProofLang::En, &strict), "{w}");
}
for w in ["1th", "21th", "11st", "3nd"] {
assert!(!is_correct_in_with(w, ProofLang::En, &strict), "{w}");
}
// Affixed forms and possessives come from the affix rules.
for w in ["unrecognizable", "reorganization", "children's", "Mississippi's", "can't", "can’t"] {
assert!(is_correct_in(w, ProofLang::En), "{w}");
}
assert!(!is_correct_in("mississippi", ProofLang::En), "a proper noun in lower case");
// Suggestions come from the run's own dictionary.
assert!(suggest_in("colur", 5, ProofLang::EnGb).contains(&"colour".to_string()), "{:?}", suggest_in("colur", 5, ProofLang::EnGb));
assert!(suggest_in("colr", 5, ProofLang::En).contains(&"color".to_string()), "{:?}", suggest_in("colr", 5, ProofLang::En));
assert_eq!(suggest_in("organisation", 5, ProofLang::En).first().map(String::as_str), Some("organization"));
assert_eq!(suggest_in("color", 5, ProofLang::EnGb).first().map(String::as_str), Some("colour"));
}
/// craft-libs#483 review: English suggestions offer a missing space only between two
/// lower-case words of three letters or more, after every one-word candidate, and no word the
/// dictionary only has capitalized (`Col`, `Wis`) for a lower-case word.
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn english_suggestions_skip_junk_splits_and_names() {
for (w, l) in [
("colour", ProofLang::En),
("theatr", ProofLang::En),
("publically", ProofLang::En),
("cheque", ProofLang::En),
("colr", ProofLang::En),
("wiss", ProofLang::En),
("theatr", ProofLang::EnGb),
] {
let got = suggest_in(w, 8, l);
assert!(!got.is_empty(), "{w}");
assert!(got.iter().all(|s| !s.chars().any(char::is_uppercase)), "{w} in {l:?}: {got:?}");
let first_split = got.iter().position(|s| s.contains(' ')).unwrap_or(got.len());
assert!(got.iter().skip(first_split).all(|s| s.contains(' ')), "{w}: a split before a word: {got:?}");
assert!(got.iter().filter(|s| s.contains(' ')).all(|s| s.split(' ').all(|p| p.chars().count() >= 3)), "{w}: {got:?}");
}
assert_eq!(suggest_in("publically", 5, ProofLang::En).first().map(String::as_str), Some("publicly"));
assert_eq!(suggest_in("colour", 5, ProofLang::En).first().map(String::as_str), Some("color"));
assert_eq!(suggest_in("theatr", 5, ProofLang::EnGb).first().map(String::as_str), Some("theatre"));
assert!(suggest_in("Londn", 5, ProofLang::En).contains(&"London".to_string()), "a capitalized word may get a name");
}
/// `en-variants.txt` stays with SCOWL: SCOWL's size-60 lists keep one spelling per dialect,
/// and Word accepts more (both `-ize` and `-ise` in British English, `judgement` in American).
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn dialect_policy_widens_scowl() {
let gb = scowl::dict(ProofLang::EnGb).unwrap();
let us = scowl::dict(ProofLang::En).unwrap();
assert!(!gb.check("organize") && gb.check("organise") && !us.check("judgement"), "SCOWL alone is stricter");
assert!(is_correct_in("organize", ProofLang::EnGb) && is_correct_in("organise", ProofLang::EnGb));
assert!(is_correct_in("judgement", ProofLang::En) && !is_correct_in("colour", ProofLang::En));
assert!(is_correct_in("encyclopaedia", ProofLang::EnGb) && is_correct_in("encyclopedia", ProofLang::EnGb));
}
/// craft-libs#483 review: each English checks against its own SCOWL dictionary only. A stem
/// one English's dictionary has and another's lacks (`colourblind`, `greyscale` in en-GB;
/// `colorblind`, `esophagi` in en-US) doesn't cross, unless our dialect policy or the `-ize`
/// spellings of British English say so. Sampled from every pair of Englishes.
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn scowl_dialects_dont_cross() {
let langs = [
(ProofLang::En, crate::data::EN_US_SPELLING),
(ProofLang::EnGb, crate::data::EN_GB_SPELLING),
(ProofLang::EnCa, crate::data::EN_CA_SPELLING),
];
for (from, dic) in langs {
for (to, _) in langs {
if from == to {
continue;
}
let d = scowl::dict(to).unwrap();
let widened = |w: &str| {
let x = english::ise_spelling(w);
(scowl::is_british(to) && x.as_deref().is_some_and(|x| d.check(x)))
|| en_variants::known_variant(w, to, |x| d.check(x))
|| known_in_supplement(w)
};
let only: Vec<&str> = dic
.lines()
.skip(1)
.map(|l| l.split('/').next().unwrap_or_default())
.filter(|w| w.len() > 3 && w.chars().all(|c| c.is_ascii_lowercase()) && !d.check(w) && !widened(w))
.collect();
assert!(only.len() >= 40, "{from:?} → {to:?}: {}", only.len());
let step = only.len() / 40;
let crossed: Vec<&str> = only.iter().step_by(step.max(1)).copied().filter(|w| is_correct_in(w, to)).collect();
assert!(crossed.is_empty(), "{from:?} words right in {to:?}: {crossed:?}");
}
}
for (w, l) in [("colourblind", ProofLang::En), ("greyscale", ProofLang::En), ("colorblind", ProofLang::EnGb), ("esophagi", ProofLang::EnGb)] {
assert!(!is_correct_in(w, l), "{w} in {l:?}");
}
for (w, l) in
[("colorblind", ProofLang::En), ("colourblind", ProofLang::EnGb), ("organizational", ProofLang::EnGb), ("realized", ProofLang::EnGb)]
{
assert!(is_correct_in(w, l), "{w} in {l:?}");
}
}
/// craft-libs#483 re-review: British English takes the `-ize` spelling of an `-ise` word only
/// when American or Canadian English spells it so; a misspelling whose `-s-` form is a word
/// (`wize`, `advize`) stays wrong, as in Hunspell's en-GB.
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn british_ize_needs_a_real_ize_word() {
let wrong = [
"wize", "rize", "noize", "dizease", "advize", "rizing", "praize", "raize", "poize", "wizer", "surprize", "arize", "revize", "devize",
"mizer", "concize", "precize", "exercize", "chaize",
];
for l in [ProofLang::EnGb, ProofLang::from_tag(Some("en-AU"))] {
for w in wrong.iter().chain(&["colorize"]) {
assert!(!is_correct_in(w, l), "{w} in {l:?}");
}
for w in ["organize", "realization", "colourize", "organizing", "realizes"] {
assert!(is_correct_in(w, l), "{w} in {l:?}");
}
}
}
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn user_dictionary_accepts_the_other_dialects_spelling() {
+5
View File
@@ -45,6 +45,11 @@ const SAMPLES: [&str; 10] = [
#[test]
fn nothing_is_embedded() {
assert!(crate::data::EN_US_DIC.is_empty());
assert!(crate::data::EN_US_SPELLING.is_empty() && crate::data::EN_GB_SPELLING.is_empty() && crate::data::EN_CA_SPELLING.is_empty());
assert!(crate::scowl::notice().is_none());
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
assert!(crate::scowl::dict(l).is_none() && crate::scowl::load_error(l).is_none());
}
assert!(dict::Dictionary::en_us().is_empty());
for p in [patterns::Patterns::en_us(), patterns::Patterns::pl(), patterns::Patterns::pt(), patterns::Patterns::it(), patterns::Patterns::de()] {
assert!(p.is_empty());
+68
View File
@@ -0,0 +1,68 @@
//! English spelling from SCOWL (Spell Checker Oriented Word Lists, Kevin Atkinson): the Hunspell
//! dictionaries of craft-i18n's `language-data/en-us`, `en-gb` (British `-ise`) and `en-ca`,
//! embedded unmodified at build time and read by the affix engine ([`crate::affix`]) on first use.
//!
//! Each English has its own: American English reads en-US, the British-spelling Englishes
//! (en-GB, en-AU, en-NZ, en-IE…) read en-GB and Canadian English reads en-CA. Its suggestions
//! come from its own dictionary, and a word is right when its own dictionary knows it: the other
//! dialects' dictionaries are never asked (en-US would take `colourblind` from en-GB's). SCOWL
//! picks one spelling per dialect (en-GB `-ise` has no `organize`, en-US no `judgement`), and Word
//! accepts more, so British English also takes the `-ize` spellings of its `-ise` words that
//! en-US or en-CA spells so, and `en-variants.txt` widens each dialect by its pairs
//! (`crate::english`). An English dictionary in the dictionaries folder comes first; a SCOWL
//! dictionary that isn't embedded, or fails to load, leaves that English to the next source (`crate::english`).
//!
//! SCOWL is under its own permissive licence and the affix files under Ispell's BSD licence;
//! [`notice`] is the upstream text, for an About window.
use std::sync::OnceLock;
use crate::ProofLang;
use crate::affix::AffixDict;
use crate::data::{EN_CA_AFF, EN_CA_SPELLING, EN_GB_AFF, EN_GB_SPELLING, EN_US_AFF, EN_US_SPELLING, SCOWL_NOTICE};
/// Per English (American, British, Canadian): the loaded dictionary, or why it isn't there.
static DICTS: [OnceLock<Result<AffixDict, String>>; 3] = [OnceLock::new(), OnceLock::new(), OnceLock::new()];
/// The slot, `.aff` and `.dic` text of `lang`'s dictionary.
fn source(lang: ProofLang) -> (usize, &'static str, &'static str) {
match lang {
ProofLang::EnGb => (1, EN_GB_AFF, EN_GB_SPELLING),
ProofLang::EnCa => (2, EN_CA_AFF, EN_CA_SPELLING),
_ => (0, EN_US_AFF, EN_US_SPELLING),
}
}
/// `lang`'s SCOWL dictionary (an English), when it is embedded and loads.
pub(crate) fn dict(lang: ProofLang) -> Option<&'static AffixDict> {
let (i, aff, dic) = source(lang);
if aff.is_empty() || dic.is_empty() {
return None;
}
let loaded = DICTS.get(i)?.get_or_init(|| {
let d = AffixDict::from_hunspell(aff, dic, &["SCOWL (wordlist.aspell.net), embedded from craft-i18n"])
.and_then(|d| if d.is_empty() { Err("no words".to_string()) } else { Ok(d) });
if let Err(e) = &d {
log::error!("embedded SCOWL dictionary for {lang:?}: {e}; English spelling falls back to the next source");
}
d
});
loaded.as_ref().ok()
}
/// Does `lang` read the British `-ise` dictionary (en-GB, en-AU, en-NZ…)?
pub(crate) fn is_british(lang: ProofLang) -> bool {
source(lang).0 == 1
}
/// Why `lang`'s embedded SCOWL dictionary failed to load, once something tried to load it.
pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
let (i, ..) = source(lang);
DICTS.get(i)?.get()?.as_ref().err().map(|e| format!("embedded SCOWL dictionary: {e}"))
}
/// The SCOWL copyright and licence notices (the upstream README, with the Ispell BSD licence of
/// the affix files and Benjamin Titze's notice for the Australian data), when SCOWL is embedded.
pub fn notice() -> Option<&'static str> {
(!SCOWL_NOTICE.is_empty() && !EN_US_SPELLING.is_empty()).then_some(SCOWL_NOTICE)
}