mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-08 19:42:12 +00:00
Merge pull request #483 from benandez/feat/cp2-scowl
CP2.2: craft-proof reads SCOWL 2026 for English spelling
This commit is contained in:
11 files changed
+884
-276
No files matched your search
@@ -25,10 +25,12 @@ jobs:
|
||||
with:
|
||||
persist-credentials: false
|
||||
# The Language data (craft-i18n `language-data/`), embedded by crates/proof/build.rs.
|
||||
# Pinned to storytold/craft-i18n#114's head (#109's SCOWL dictionaries + judgment/judgement,
|
||||
# CP2.1); TODO(CP2.1): move to #114's merge commit once it merges.
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
repository: storytold/craft-i18n
|
||||
ref: 062a6375c322c88b96691a17f7b5fed71c51f7a8
|
||||
ref: c239f934ace90b7939afaceb23b472ceeafd6b8a
|
||||
path: craft-i18n
|
||||
persist-credentials: false
|
||||
- uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
|
||||
|
||||
+41
-14
@@ -13,7 +13,7 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = "<commit>
|
||||
|
||||
| Language | Spelling | Grammar | Hyphenation |
|
||||
|---|---|---|---|
|
||||
| English (en-US, en-GB and the British-spelling Englishes, en-CA) | Moby word list + `en-extra.txt`, dialect variants (`en-variants.txt`); without them the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
|
||||
| English (en-US, en-CA, en-GB and the British-spelling Englishes such as en-AU, en-NZ) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
|
||||
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns |
|
||||
| Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise |
|
||||
| German | — | — | TeX patterns |
|
||||
@@ -63,20 +63,41 @@ data folder runs neither set.
|
||||
|
||||
English spelling uses the first of these that is there; `english_source(lang)` says which.
|
||||
|
||||
1. **Embedded:** the Moby word list from `CRAFT_I18N_DIR` (above). Nothing below is used then.
|
||||
2. **System Hunspell:** the run's own English `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`;
|
||||
each name in the dictionaries folder and then the system locations before the next name).
|
||||
Never another English's: an `en_US` dictionary would mark British spellings, so en-GB and
|
||||
en-CA without their own use the built-in list, which has both spellings. A file that can't be
|
||||
read is skipped for the next name or folder; one file is read once for every English that uses
|
||||
it. Compounding (`COMPOUNDMIN`, `COMPOUNDRULE`…, used by English dictionaries only for
|
||||
ordinals) and `OCONV` are ignored for English rather than refused. None on the web.
|
||||
3. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
|
||||
1. **Dictionaries folder:** the English's own `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`)
|
||||
that the user put in the folder set with `set_dictionaries_folder`, in `WORDCRAFT_DICTIONARIES`
|
||||
or in WordCraft's own `dictionaries` folder. The user's choice comes first.
|
||||
2. **Installed English pack:** reserved for Language packs (craft-libs#485, CP4.2).
|
||||
3. **Embedded SCOWL:** the English's own SCOWL `rel-2026.02.25` Hunspell dictionary from
|
||||
`CRAFT_I18N_DIR` (`language-data/en-us`, `en-gb` (British `-ise`) or `en-ca`, `spelling.aff` +
|
||||
`spelling.dic`), read by the affix engine. en-AU, en-NZ, en-IE and the other British-spelling
|
||||
Englishes read en-GB; Canadian English reads en-CA. A word is right when the English's own
|
||||
dictionary knows it (or `en-extra.txt` does), never because another English's does
|
||||
(`colourblind` is wrong in en-US, `colorblind` in en-GB); suggestions come from the same
|
||||
dictionary. SCOWL's size-60 lists keep one spelling per dialect, and Word accepts more, so
|
||||
British English also takes the `-ize` spellings of its `-ise` words that American or Canadian
|
||||
English spell so (`organize`, `realization`; never `wize` or `advize`), and `en-variants.txt` flags the other dialect's spellings and accepts this
|
||||
dialect's other spellings of known words (`judgement` in en-US). English with no dialect
|
||||
(`is_correct`) takes a word any of the three knows. Suggestions offer a missing space only
|
||||
between two lower-case words of three letters or more, after every one-word candidate, and no
|
||||
name or abbreviation (`Col`) for a lower-case word. With SCOWL embedded the system locations aren't
|
||||
searched.
|
||||
4. **System Hunspell:** the English's own pair in the system locations. Never another English's:
|
||||
an `en_US` dictionary would mark British spellings, so en-GB and en-CA without their own use
|
||||
the built-in list, which has both spellings. Each name is looked for in every folder before the
|
||||
next name; a file that can't be read is skipped for the next name or folder; one file is read
|
||||
once for every English that uses it. `COMPOUNDRULE`, `ONLYINCOMPOUND` and `COMPOUNDMIN`
|
||||
(ordinals such as `21st`) are read; flag-based compounding (`COMPOUNDFLAG`…) and `OCONV` are
|
||||
ignored for English rather than refused. None on the web.
|
||||
5. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
|
||||
compressed, words only) built into the crate, with `data/en-extra.txt` (WordCraft's supplement
|
||||
of modern words, a copy of craft-i18n's) and the same inflection rules as the embedded list. A
|
||||
word it doesn't know is misspelled. Suggestions are the words one edit away (swap, missing,
|
||||
added or changed letter), looked up in the list, for words of up to 32 letters: swaps first,
|
||||
then common words, neighbouring keys, the same first letter.
|
||||
of modern words, a copy of craft-i18n's) and the inflection rules. A word it doesn't know is
|
||||
misspelled. Suggestions are the words one edit away (swap, missing, added or changed letter),
|
||||
looked up in the list, for words of up to 32 letters: swaps first, then common words,
|
||||
neighbouring keys, the same first letter.
|
||||
|
||||
The embedded Moby list (`language-data/en-us/en-us.dic`) is no longer a spelling source: it is
|
||||
the same list as the built-in one, and stays as the hyphenation word list. `en-variants.txt`
|
||||
applies to every source above when it is embedded.
|
||||
|
||||
A dictionary that is there but can't be read (encoding, size, parse error, no words) is skipped,
|
||||
and `dictionary_load_error(lang)` gives the reason; it never panics. Text in other languages is
|
||||
@@ -91,6 +112,12 @@ Regenerate with
|
||||
`data/en-extra.txt` is WordCraft contributors' own list (MIT OR Apache-2.0); keep it in step with
|
||||
craft-i18n's copy.
|
||||
|
||||
The SCOWL notice (SCOWL's licence, the Ispell BSD licence of the affix files and Benjamin Titze's
|
||||
notice) is embedded with the dictionaries: `craft_proof::scowl::notice()`, for an About window.
|
||||
|
||||
Embedding SCOWL adds 1,542,396 bytes of Language data (the en-US, en-GB and en-CA `.aff`/`.dic`
|
||||
and the notice); the Moby list (432,905 bytes) stays for hyphenation.
|
||||
|
||||
## Check
|
||||
|
||||
```sh
|
||||
|
||||
@@ -14,6 +14,10 @@
|
||||
//!
|
||||
//! `include_str!` takes a string, so a `CRAFT_I18N_DIR` that isn't valid UTF-8 can't be embedded:
|
||||
//! it is treated like a folder without the data.
|
||||
//!
|
||||
//! English: SCOWL's en-US, en-GB (-ise) and en-CA dictionaries; en-AU is not embedded, as
|
||||
//! Australian English is checked as British-spelling English (`ProofLang::EnGb`). The SCOWL
|
||||
//! notice is embedded once (the four upstream READMEs differ only in their title and build line).
|
||||
|
||||
use std::fmt::Write as _;
|
||||
use std::path::{Path, PathBuf};
|
||||
@@ -21,6 +25,13 @@ use std::path::{Path, PathBuf};
|
||||
/// (constant, path under `language-data/`, bytes rather than text).
|
||||
const FILES: &[(&str, &str, bool)] = &[
|
||||
("EN_US_DIC", "en-us/en-us.dic", true),
|
||||
("EN_US_AFF", "en-us/spelling.aff", false),
|
||||
("EN_US_SPELLING", "en-us/spelling.dic", false),
|
||||
("EN_GB_AFF", "en-gb/spelling.aff", false),
|
||||
("EN_GB_SPELLING", "en-gb/spelling.dic", false),
|
||||
("EN_CA_AFF", "en-ca/spelling.aff", false),
|
||||
("EN_CA_SPELLING", "en-ca/spelling.dic", false),
|
||||
("SCOWL_NOTICE", "en-us/LICENCE-SCOWL.txt", false),
|
||||
("EN_US_PAT", "en-us/hyphenation.pat", false),
|
||||
("EXTRA_EN", "en-us/en-extra.txt", false),
|
||||
("EN_VARIANTS", "en-us/en-variants.txt", false),
|
||||
|
||||
+285
-46
@@ -17,11 +17,12 @@
|
||||
//! (at most two suffixes in a row, as in Hunspell), `FORBIDDENWORD`, `NOSUGGEST`, `KEEPCASE` (the
|
||||
//! stem is a word only in its own letter case), input conversion (`ICONV`: ligatures such as `fi`
|
||||
//! read as `fi`) and ignored characters (`IGNORE`), and the suggestion tables `TRY`, `KEY`, `MAP`
|
||||
//! and `REP`.
|
||||
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: compounding,
|
||||
//! long or numeric flags, `NEEDAFFIX` and the other directives that change which words are correct
|
||||
//! (an English dictionary's compounding and `OCONV` are ignored instead:
|
||||
//! [`AffixDict::from_hunspell_english`]).
|
||||
//! and `REP`. Compounds by `COMPOUNDRULE` (patterns of one-character flags with `*` and `?`, as
|
||||
//! SCOWL's English dictionaries use for ordinals: `21st`, `112th`), with `ONLYINCOMPOUND` and
|
||||
//! `COMPOUNDMIN`; the parts of such a compound are stems, without affixes, as in Hunspell.
|
||||
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: flag-based
|
||||
//! compounding (`COMPOUNDFLAG`, `COMPOUNDBEGIN`…), long or numeric flags, `NEEDAFFIX` and the
|
||||
//! other directives that change which words are correct.
|
||||
//!
|
||||
//! Lookups never panic: stems sit in one string, found through an open-addressing hash table, and
|
||||
//! a probe allocates only small scratch strings for the candidate roots it tries.
|
||||
@@ -48,16 +49,54 @@ const UNSUPPORTED: &[&str] = &[
|
||||
"COMPOUNDFLAG",
|
||||
"COMPOUNDLAST",
|
||||
"COMPOUNDMIDDLE",
|
||||
"COMPOUNDMIN",
|
||||
"COMPOUNDPERMITFLAG",
|
||||
"COMPOUNDRULE",
|
||||
"FULLSTRIP",
|
||||
"NEEDAFFIX",
|
||||
"OCONV",
|
||||
"ONLYINCOMPOUND",
|
||||
"PSEUDOROOT",
|
||||
];
|
||||
|
||||
/// Most `COMPOUNDRULE` patterns kept, and most tokens in one: a hostile `.aff` can't make a
|
||||
/// compound check cost more than a few thousand stem lookups per word.
|
||||
const MAX_COMPOUND_RULES: usize = 64;
|
||||
const MAX_COMPOUND_TOKENS: usize = 32;
|
||||
/// Longest word (in bytes) tried as a compound: each try looks up every substring.
|
||||
const MAX_COMPOUND_BYTES: usize = 48;
|
||||
|
||||
/// How often a `COMPOUNDRULE` token's flag may repeat.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum Repeat {
|
||||
Once,
|
||||
/// `?`
|
||||
Optional,
|
||||
/// `*`
|
||||
Any,
|
||||
}
|
||||
|
||||
/// A `COMPOUNDRULE` pattern as written (`n*1t`): its tokens, each a flag and how often it repeats.
|
||||
fn parse_compound_rule(pattern: &str) -> Result<Vec<(char, Repeat)>, String> {
|
||||
let mut out: Vec<(char, Repeat)> = Vec::new();
|
||||
for c in pattern.chars().filter(|&c| c != EMOJI_SELECTOR) {
|
||||
match c {
|
||||
'*' | '?' => {
|
||||
let Some(last) = out.last_mut().filter(|t| t.1 == Repeat::Once) else {
|
||||
return Err(format!("COMPOUNDRULE `{pattern}`: `{c}` follows no flag"));
|
||||
};
|
||||
last.1 = if c == '*' { Repeat::Any } else { Repeat::Optional };
|
||||
}
|
||||
'(' | ')' => return Err(format!("COMPOUNDRULE `{pattern}`: ( ) groups (long flags) are not supported")),
|
||||
c => out.push((c, Repeat::Once)),
|
||||
}
|
||||
if out.len() > MAX_COMPOUND_TOKENS {
|
||||
return Err(format!("COMPOUNDRULE `{pattern}`: more than {MAX_COMPOUND_TOKENS} flags"));
|
||||
}
|
||||
}
|
||||
if out.is_empty() {
|
||||
return Err("COMPOUNDRULE with no flags".into());
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Most distinct flags a dictionary may use: every single-byte flag (the Hungarian dictionary
|
||||
/// uses 158).
|
||||
const MAX_FLAGS: usize = 256;
|
||||
@@ -272,6 +311,18 @@ pub struct AffixDict {
|
||||
ignore: Box<[char]>,
|
||||
/// Provenance and licence lines, kept through [`AffixDict::to_bytes`].
|
||||
comments: Vec<String>,
|
||||
/// `COMPOUNDRULE` patterns: per token, the flag's bit and how often it may repeat.
|
||||
compound_rules: Vec<Vec<(Mask, Repeat)>>,
|
||||
/// The same patterns as written, for [`AffixDict::to_bytes`].
|
||||
compound_text: Vec<String>,
|
||||
/// The `ONLYINCOMPOUND` flag's bit: such stems are words only as parts of a compound.
|
||||
only_in_compound: Mask,
|
||||
/// `COMPOUNDMIN`, in characters.
|
||||
compound_min: usize,
|
||||
/// The first characters of the stems a compound part can be (their flags are in a
|
||||
/// `COMPOUNDRULE`), so a word starting otherwise is no compound: SCOWL's ordinals start with
|
||||
/// a digit, and a misspelled word costs one lookup here instead of one per cut.
|
||||
compound_firsts: Vec<char>,
|
||||
}
|
||||
|
||||
fn hash_str(s: &str) -> u64 {
|
||||
@@ -341,10 +392,14 @@ struct Rules {
|
||||
substandard: Option<char>,
|
||||
/// The `KEEPCASE` flag.
|
||||
keepcase: Option<char>,
|
||||
/// The `ONLYINCOMPOUND` flag, read for English only ([`AffixDict::from_hunspell_english`]).
|
||||
only_in_compound: Option<char>,
|
||||
iconv: Vec<(String, String)>,
|
||||
ignore: Vec<char>,
|
||||
/// `COMPOUNDRULE` patterns, as written (`n*1t`).
|
||||
compound_rules: Vec<String>,
|
||||
/// The `ONLYINCOMPOUND` flag.
|
||||
only_in_compound: Option<char>,
|
||||
/// `COMPOUNDMIN`: the shortest part of a compound, in characters (Hunspell's default is 3).
|
||||
compound_min: Option<usize>,
|
||||
/// `AF` flag aliases: alias `n` (1-based) stands for `aliases[n - 1]`.
|
||||
aliases: Vec<String>,
|
||||
/// The `AF` table opened with its count line, so flag fields are alias numbers. A table
|
||||
@@ -426,26 +481,15 @@ fn zero_empty(s: &str) -> String {
|
||||
/// but it is not a flag of its own: it is dropped wherever flags are read.
|
||||
const EMOJI_SELECTOR: char = '\u{fe0f}';
|
||||
|
||||
/// Directives an English dictionary may carry that don't change which English words are correct
|
||||
/// here, ignored by [`AffixDict::from_hunspell_english`]: compounding (English dictionaries use
|
||||
/// it only for ordinals such as `1st`, which have digits and are skipped by the spelling check)
|
||||
/// and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
|
||||
const IGNORED_IN_ENGLISH: &[&str] = &[
|
||||
"COMPOUNDBEGIN",
|
||||
"COMPOUNDEND",
|
||||
"COMPOUNDFIRST",
|
||||
"COMPOUNDFLAG",
|
||||
"COMPOUNDLAST",
|
||||
"COMPOUNDMIDDLE",
|
||||
"COMPOUNDMIN",
|
||||
"COMPOUNDPERMITFLAG",
|
||||
"COMPOUNDRULE",
|
||||
"OCONV",
|
||||
];
|
||||
/// Directives an English dictionary may carry that this engine doesn't implement but that don't
|
||||
/// change which English words are right, ignored by [`AffixDict::from_hunspell_english`]:
|
||||
/// flag-based compounding (English dictionaries compound only ordinals, by `COMPOUNDRULE`, which
|
||||
/// is read) and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
|
||||
const IGNORED_IN_ENGLISH: &[&str] =
|
||||
&["COMPOUNDBEGIN", "COMPOUNDEND", "COMPOUNDFIRST", "COMPOUNDFLAG", "COMPOUNDLAST", "COMPOUNDMIDDLE", "COMPOUNDPERMITFLAG", "OCONV"];
|
||||
|
||||
/// Read Hunspell `.aff` text (already decoded to UTF-8). `english`: ignore the directives in
|
||||
/// [`IGNORED_IN_ENGLISH`], and keep the `ONLYINCOMPOUND` flag so the stems that are only parts of
|
||||
/// compounds can be left out.
|
||||
/// [`IGNORED_IN_ENGLISH`] instead of refusing them.
|
||||
fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
|
||||
let mut r = Rules::default();
|
||||
let mut cross: HashMap<(bool, char), bool> = HashMap::new();
|
||||
@@ -453,6 +497,7 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
|
||||
let mut rep_header = false;
|
||||
let mut af_header = false;
|
||||
let mut iconv_header = false;
|
||||
let mut compound_header = false;
|
||||
for (n, line) in aff.lines().enumerate() {
|
||||
let f: Vec<&str> = split_fields(line).collect();
|
||||
let Some(&dir) = f.first() else { continue };
|
||||
@@ -487,6 +532,23 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
|
||||
}
|
||||
"IGNORE" => r.ignore = at(1).chars().collect(),
|
||||
"KEEPCASE" => r.keepcase = Some(one_flag(at(1), dir, n)?),
|
||||
// The first `COMPOUNDRULE` line is the table's count, as in Hunspell.
|
||||
"COMPOUNDRULE" => {
|
||||
if !compound_header && at(1).parse::<usize>().is_ok() {
|
||||
compound_header = true;
|
||||
} else {
|
||||
parse_compound_rule(at(1)).map_err(|e| format!("line {}: {e}", n + 1))?;
|
||||
if r.compound_rules.len() >= MAX_COMPOUND_RULES {
|
||||
return Err(format!("line {}: more than {MAX_COMPOUND_RULES} COMPOUNDRULE patterns", n + 1));
|
||||
}
|
||||
r.compound_rules.push(at(1).to_string());
|
||||
}
|
||||
}
|
||||
"ONLYINCOMPOUND" => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
|
||||
"COMPOUNDMIN" => {
|
||||
let min = at(1).parse::<usize>().map_err(|_| format!("line {}: COMPOUNDMIN needs a number", n + 1))?;
|
||||
r.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES));
|
||||
}
|
||||
// Per-character flag modes (`FLAG UTF-8`, as VERO's pt_BR dictionary uses) work
|
||||
// with the one-character parsing here; only the multi-character encodings differ.
|
||||
"FLAG" if !matches!(at(1), "UTF-8" | "DEFAULT" | "utf-8" | "default" | "UTF8" | "") => {
|
||||
@@ -541,7 +603,6 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
|
||||
r.prefixes.push(rule);
|
||||
}
|
||||
}
|
||||
"ONLYINCOMPOUND" if english => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
|
||||
d if english && IGNORED_IN_ENGLISH.contains(&d) => {}
|
||||
d if UNSUPPORTED.contains(&d) => return Err(format!("line {}: {d} is not supported", n + 1)),
|
||||
_ => {}
|
||||
@@ -570,10 +631,10 @@ impl AffixDict {
|
||||
AffixDict::from_hunspell_in(aff, dic, comments, false)
|
||||
}
|
||||
|
||||
/// [`AffixDict::from_hunspell`] for an English dictionary: compounding and output conversion
|
||||
/// (`COMPOUNDMIN`, `COMPOUNDRULE`, `OCONV`…), which other languages need but English spelling
|
||||
/// doesn't, are ignored rather than refused, and the stems marked `ONLYINCOMPOUND` (parts of
|
||||
/// compounds only) are left out.
|
||||
/// [`AffixDict::from_hunspell`] for an English dictionary: flag-based compounding and output
|
||||
/// conversion (`COMPOUNDFLAG`, `OCONV`…), which this engine doesn't implement and English
|
||||
/// spelling doesn't need, are ignored rather than refused. `COMPOUNDRULE`, `ONLYINCOMPOUND`
|
||||
/// and `COMPOUNDMIN` (ordinals: `21st`) are read, as for every language.
|
||||
pub fn from_hunspell_english(aff: &str, dic: &str, comments: &[&str]) -> Result<AffixDict, String> {
|
||||
AffixDict::from_hunspell_in(aff, dic, comments, true)
|
||||
}
|
||||
@@ -619,9 +680,6 @@ impl AffixDict {
|
||||
continue;
|
||||
}
|
||||
let flags = rules.flags(&flags).to_string();
|
||||
if rules.only_in_compound.is_some_and(|f| flags.contains(f)) {
|
||||
continue;
|
||||
}
|
||||
stems.push((word, flags));
|
||||
}
|
||||
AffixDict::build(rules, stems, comments.iter().map(ToString::to_string).collect())
|
||||
@@ -631,6 +689,7 @@ impl AffixDict {
|
||||
// Flags: every one a rule or a stem uses gets a bit.
|
||||
let mut flag_chars: Vec<char> = rules.prefixes.iter().chain(&rules.suffixes).map(|r| r.0).collect();
|
||||
flag_chars.extend(stems.iter().flat_map(|s| s.1.chars()));
|
||||
// A compound rule's flag no stem carries gets no bit: its token matches no part.
|
||||
flag_chars.sort_unstable();
|
||||
flag_chars.dedup();
|
||||
if flag_chars.len() > MAX_FLAGS {
|
||||
@@ -656,9 +715,18 @@ impl AffixDict {
|
||||
map: rules.map,
|
||||
rep: rules.rep,
|
||||
comments,
|
||||
compound_rules: rules
|
||||
.compound_rules
|
||||
.iter()
|
||||
.map(|p| parse_compound_rule(p).map(|t| t.into_iter().map(|(c, rep)| (bit_of(c), rep)).collect()))
|
||||
.collect::<Result<_, _>>()?,
|
||||
compound_text: rules.compound_rules,
|
||||
only_in_compound: rules.only_in_compound.map_or(Mask::EMPTY, bit_of),
|
||||
compound_min: rules.compound_min.unwrap_or(3),
|
||||
..AffixDict::default()
|
||||
};
|
||||
d.offs.push(0);
|
||||
let compound_flags = d.compound_rules.iter().flatten().fold(Mask::EMPTY, |m, (bit, _)| m | *bit);
|
||||
let mut set_ids: HashMap<Mask, u16> = HashMap::new();
|
||||
let mut i = 0;
|
||||
while i < stems.len() {
|
||||
@@ -676,6 +744,12 @@ impl AffixDict {
|
||||
if usize::from(id) == d.sets.len() {
|
||||
d.sets.push(mask);
|
||||
}
|
||||
if mask.meets(compound_flags)
|
||||
&& let Some(c) = word.chars().next()
|
||||
&& !d.compound_firsts.contains(&c)
|
||||
{
|
||||
d.compound_firsts.push(c);
|
||||
}
|
||||
d.words.push_str(word);
|
||||
d.offs.push(u32::try_from(d.words.len()).map_err(|_| "stems too large".to_string())?);
|
||||
d.flags.push(id);
|
||||
@@ -797,6 +871,11 @@ impl AffixDict {
|
||||
|
||||
/// Flag mask of a stem, if `w` is one and carries none of the flags in `deny`.
|
||||
fn stem_flags_except(&self, w: &str, deny: Mask) -> Option<Mask> {
|
||||
self.stem_mask(w).filter(|m| !m.meets(self.forbidden | self.only_in_compound | deny))
|
||||
}
|
||||
|
||||
/// Flag mask of the stem `w`, whatever flags it carries.
|
||||
fn stem_mask(&self, w: &str) -> Option<Mask> {
|
||||
if self.table.is_empty() {
|
||||
return None;
|
||||
}
|
||||
@@ -810,9 +889,9 @@ impl AffixDict {
|
||||
if self.word(t as usize) == Some(w) {
|
||||
let id = *self.flags.get(t as usize)?;
|
||||
let mask = self.sets.get(usize::from(id)).copied();
|
||||
// A stem carrying FORBIDDENWORD is listed but is not a word: no direct hit,
|
||||
// and (this is the only stem lookup) no affixed form is built on it either.
|
||||
return mask.filter(|m| !m.meets(self.forbidden | deny));
|
||||
// [`Self::stem_flags_except`] drops FORBIDDENWORD stems (listed, not words) and
|
||||
// ONLYINCOMPOUND ones (words only inside a compound, [`Self::compound`]).
|
||||
return mask;
|
||||
}
|
||||
slot = (slot + 1) & mask;
|
||||
}
|
||||
@@ -928,7 +1007,41 @@ impl AffixDict {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
self.compound(w, deny)
|
||||
}
|
||||
|
||||
/// Is `w` a compound of two or more stems (no affixes) whose flags spell out one of the
|
||||
/// `COMPOUNDRULE` patterns, each part at least `COMPOUNDMIN` characters? `21st` is `2/m` +
|
||||
/// `1st/p` under `n*mp`. A dynamic programme over (byte offset, token, parts so far), so a
|
||||
/// hostile pattern costs at most a few thousand lookups. A forbidden word is never rebuilt
|
||||
/// from its parts.
|
||||
fn compound(&self, w: &str, deny: Mask) -> bool {
|
||||
if self.compound_rules.is_empty()
|
||||
|| w.len() > MAX_COMPOUND_BYTES
|
||||
|| !w.chars().next().is_some_and(|c| self.compound_firsts.contains(&c))
|
||||
|| self.stem_mask(w).is_some_and(|m| m.meets(self.forbidden))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
// Char boundaries of `w`, `0` and `w.len()` included.
|
||||
let cuts: Vec<usize> = w.char_indices().map(|(i, _)| i).chain(std::iter::once(w.len())).collect();
|
||||
let n = cuts.len();
|
||||
// Flag mask of each part `w[cuts[a]..cuts[b]]`, by (a, b), looked up when a rule first
|
||||
// reaches cut `a` (`None`: not looked up yet; `Some(None)`: not a stem). Only the parts
|
||||
// after a cut a rule reaches are looked up, so a word with no first part costs `n`.
|
||||
let mut parts: Vec<Option<Option<Mask>>> = vec![None; n * n];
|
||||
let min = self.compound_min;
|
||||
let mut part = |a: usize, b: usize| -> Option<Mask> {
|
||||
if b < a.saturating_add(min) {
|
||||
return None;
|
||||
}
|
||||
let slot = parts.get_mut(a * n + b)?;
|
||||
*slot.get_or_insert_with(|| {
|
||||
let (Some(&x), Some(&y)) = (cuts.get(a), cuts.get(b)) else { return None };
|
||||
w.get(x..y).and_then(|p| self.stem_mask(p)).filter(|m| !m.meets(self.forbidden | deny))
|
||||
})
|
||||
};
|
||||
self.compound_rules.iter().any(|rule| compound_matches(rule, &mut part, n))
|
||||
}
|
||||
|
||||
/// Is `w` a stem with one suffix? With `need` set (a prefix was taken off), the stem must
|
||||
@@ -1166,6 +1279,17 @@ impl AffixDict {
|
||||
if !self.ignore.is_empty() {
|
||||
out.push_str(&format!("IGNORE {}\n", self.ignore.iter().collect::<String>()));
|
||||
}
|
||||
if !self.only_in_compound.is_empty()
|
||||
&& let Some(c) = self.flag_char(self.only_in_compound)
|
||||
{
|
||||
out.push_str(&format!("ONLYINCOMPOUND {c}\n"));
|
||||
}
|
||||
if !self.compound_text.is_empty() {
|
||||
out.push_str(&format!("COMPOUNDMIN {}\n", self.compound_min));
|
||||
for p in &self.compound_text {
|
||||
out.push_str(&format!("COMPOUNDRULE {p}\n"));
|
||||
}
|
||||
}
|
||||
let z = |s: &str| if s.is_empty() { "0".to_string() } else { s.to_string() };
|
||||
for (kind, rules) in [("PFX", &self.prefixes), ("SFX", &self.suffixes)] {
|
||||
for r in rules {
|
||||
@@ -1233,6 +1357,20 @@ impl AffixDict {
|
||||
}
|
||||
"ICONV" if f.len() == 3 => rules.iconv.push((at(1).to_string(), at(2).to_string())),
|
||||
"IGNORE" => rules.ignore = at(1).chars().collect(),
|
||||
"COMPOUNDMIN" => {
|
||||
let min = at(1).parse::<usize>().map_err(|_| format!("affix dictionary: bad `{line}`"))?;
|
||||
rules.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES));
|
||||
}
|
||||
"ONLYINCOMPOUND" => {
|
||||
rules.only_in_compound = Some(one_flag(at(1), "ONLYINCOMPOUND", 0).map_err(|_| format!("affix dictionary: bad `{line}`"))?);
|
||||
}
|
||||
"COMPOUNDRULE" => {
|
||||
parse_compound_rule(at(1)).map_err(|e| format!("affix dictionary: {e}"))?;
|
||||
if rules.compound_rules.len() >= MAX_COMPOUND_RULES {
|
||||
return Err("affix dictionary: too many COMPOUNDRULE patterns".into());
|
||||
}
|
||||
rules.compound_rules.push(at(1).to_string());
|
||||
}
|
||||
"MAP" => rules.map.push(at(1).chars().collect()),
|
||||
"REP" => rules.rep.push((at(1).replace('_', " "), at(2).replace('_', " "))),
|
||||
kind @ ("PFX" | "SFX") => {
|
||||
@@ -1287,6 +1425,39 @@ impl AffixDict {
|
||||
}
|
||||
}
|
||||
|
||||
/// Does a compound rule's token list match a split of the word into two or more parts? `part(a,
|
||||
/// b)` is the flags of the stem between char cuts `a` and `b` (`None`: not a stem). States are
|
||||
/// (cut, token, parts so far capped at 2), each visited once.
|
||||
fn compound_matches(rule: &[(Mask, Repeat)], part: &mut impl FnMut(usize, usize) -> Option<Mask>, n: usize) -> bool {
|
||||
let t = rule.len();
|
||||
let idx = |cut: usize, tok: usize, k: usize| (cut * (t + 1) + tok) * 3 + k;
|
||||
let mut seen = vec![false; n * (t + 1) * 3];
|
||||
let mut stack = vec![(0usize, 0usize, 0usize)];
|
||||
while let Some((cut, tok, k)) = stack.pop() {
|
||||
match seen.get_mut(idx(cut, tok, k)) {
|
||||
Some(s) if !*s => *s = true,
|
||||
_ => continue,
|
||||
}
|
||||
if tok == t {
|
||||
if cut + 1 == n && k >= 2 {
|
||||
return true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let Some(&(bit, repeat)) = rule.get(tok) else { continue };
|
||||
if repeat != Repeat::Once {
|
||||
stack.push((cut, tok + 1, k));
|
||||
}
|
||||
for next in cut + 1..n {
|
||||
if part(cut, next).is_some_and(|m| m.meets(bit)) {
|
||||
let k = (k + 1).min(2);
|
||||
stack.push((next, if repeat == Repeat::Any { tok } else { tok + 1 }, k));
|
||||
}
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) mod tests {
|
||||
use super::*;
|
||||
@@ -1779,6 +1950,73 @@ pub(crate) mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// SCOWL's ordinal rules (its English `.aff`, verbatim): `21st`, `112th` and `1000th` are
|
||||
/// compounds of digit stems, `1th` is only a part, and nothing else compounds.
|
||||
const ORDINALS_AFF: &str = "SET UTF-8\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nWORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s .\n";
|
||||
const ORDINALS_DIC: &str = "13\n0/nm\n0th/pt\n1/n1\n1st/p\n1th/tc\n2/nm\n2nd/p\n2th/tc\n3/nm\n3rd/p\n3th/tc\n4/nm\n4th/pt\ncat/S\n";
|
||||
|
||||
#[test]
|
||||
fn compound_rules_make_ordinals() {
|
||||
let d = AffixDict::from_hunspell(ORDINALS_AFF, ORDINALS_DIC, &[]).unwrap();
|
||||
for w in ["1st", "2nd", "4th", "21st", "22nd", "23rd", "11th", "12th", "13th", "111th", "1000th", "104th", "cats"] {
|
||||
assert!(d.check(w), "{w}");
|
||||
}
|
||||
for w in ["1th", "2th", "11st", "12nd", "13rd", "21th", "catcat", "cat1st", "1stcat", "1st1st", "", "x"] {
|
||||
assert!(!d.check(w), "{w}");
|
||||
}
|
||||
// Only-in-compound parts are never suggested on their own.
|
||||
assert!(!d.can_suggest("1th") && d.forms("1th").is_empty());
|
||||
// The compact format keeps the rules.
|
||||
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
|
||||
assert!(back.check("21st") && back.check("111th") && !back.check("1th") && !back.check("21th"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compound_rules_need_two_parts_and_compoundmin() {
|
||||
// `COMPOUNDMIN` defaults to 3; a pattern a lone stem matches still needs two parts.
|
||||
let aff = "SET UTF-8\nONLYINCOMPOUND c\nCOMPOUNDRULE a*b?\n";
|
||||
let d = AffixDict::from_hunspell(aff, "3\nfoo/a\nbar/ab\nqux/c\n", &[]).unwrap();
|
||||
assert!(d.check("foobar") && d.check("foofoo") && d.check("foofoofoobar") && d.check("foo"));
|
||||
assert!(!d.check("qux") && !d.check("quxfoo") && !d.check("fooqux") && !d.check("foobarx"));
|
||||
let d = AffixDict::from_hunspell(aff, "2\nab/ab\ncd/a\n", &[]).unwrap();
|
||||
assert!(!d.check("abcd"), "parts shorter than COMPOUNDMIN 3");
|
||||
let d = AffixDict::from_hunspell(&format!("{aff}COMPOUNDMIN 2\n"), "2\nab/ab\ncd/a\n", &[]).unwrap();
|
||||
assert!(d.check("cdab"));
|
||||
}
|
||||
|
||||
/// craft-libs#483 review: a forbidden word isn't rebuilt from its parts.
|
||||
#[test]
|
||||
fn compound_never_rebuilds_a_forbidden_word() {
|
||||
let d = AffixDict::from_hunspell("SET UTF-8\nFORBIDDENWORD X\nCOMPOUNDMIN 1\nCOMPOUNDRULE a*\n", "2\nx/a\nxx/X\n", &[]).unwrap();
|
||||
assert!(!d.check("xx") && d.check("xxx") && d.check("x"));
|
||||
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
|
||||
assert!(!back.check("xx") && back.check("xxx"));
|
||||
}
|
||||
|
||||
/// craft-libs#483 review: `ONLYINCOMPOUND` survives a round trip without compound rules too.
|
||||
#[test]
|
||||
fn onlyincompound_round_trips_without_compound_rules() {
|
||||
let d = AffixDict::from_hunspell("SET UTF-8\nONLYINCOMPOUND c\n", "2\nfoo\nqux/c\n", &[]).unwrap();
|
||||
assert!(d.check("foo") && !d.check("qux"));
|
||||
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
|
||||
assert!(back.check("foo") && !back.check("qux"), "a part-only stem stays a part");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hostile_compound_rules_are_refused_or_cheap() {
|
||||
for bad in ["COMPOUNDRULE *a", "COMPOUNDRULE (ab)", "COMPOUNDRULE a**", "COMPOUNDMIN x", "ONLYINCOMPOUND ab"] {
|
||||
assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{bad}\n"), DIC, &[]).is_err(), "{bad}");
|
||||
}
|
||||
let many: String = (0..100).map(|_| "COMPOUNDRULE a*\n").collect();
|
||||
assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{many}"), DIC, &[]).is_err());
|
||||
// Every part a stem and a pattern of stars: still answers at once.
|
||||
let star: String = std::iter::repeat_n("a*", 16).collect();
|
||||
let d = AffixDict::from_hunspell(&format!("SET UTF-8\nCOMPOUNDMIN 1\nCOMPOUNDRULE {star}b\n"), "1\nx/a\n", &[]).unwrap();
|
||||
let t = std::time::Instant::now();
|
||||
assert!(!d.check(&"x".repeat(48)) && !d.check(&"x".repeat(200)));
|
||||
assert!(t.elapsed().as_secs() < 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fields_split_on_spaces_and_tabs_only() {
|
||||
// A flag byte read as U+00A0 (no-break space) or U+0085 is a flag, not a separator.
|
||||
@@ -1796,22 +2034,23 @@ pub(crate) mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// craft-libs#482 review: an English dictionary with compounding (for ordinals) and output
|
||||
/// conversion, as the usual en_US and en_GB ones have, loads for English; its `ONLYINCOMPOUND`
|
||||
/// stems are left out. Other languages still refuse those directives. (A synthetic `.aff`
|
||||
/// written for this test.)
|
||||
/// craft-libs#482 review: an English dictionary with compounding and output conversion, as the
|
||||
/// usual en_US and en_GB ones have, loads for English: `COMPOUNDRULE` makes its ordinals, its
|
||||
/// `ONLYINCOMPOUND` stems are words only inside them, and `COMPOUNDFLAG` and `OCONV` are
|
||||
/// ignored. Other languages still refuse those two. (A synthetic `.aff` written for this test.)
|
||||
#[test]
|
||||
fn english_dictionaries_ignore_compounding_and_output_conversion() {
|
||||
let aff = "SET UTF-8\nTRY esianrtolcdugmphbyfvkwz'\nICONV 1\nICONV ’ '\nOCONV 1\nOCONV ' ’\nNOSUGGEST !\n\
|
||||
COMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nCOMPOUNDFLAG Z\n\
|
||||
WORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s [^sxzhy]\nSFX D Y 1\nSFX D 0 ed [^ey]\n";
|
||||
let dic = "5\nhouse/S\nwalk/SD\n0th/pc\n1st/c\nthe\n";
|
||||
let dic = "7\nhouse/S\nwalk/SD\n0th/pc\n1st/p\n1th/tc\n1/n1\nthe\n";
|
||||
assert!(AffixDict::from_hunspell(aff, dic, &[]).is_err(), "refused for other languages");
|
||||
let d = AffixDict::from_hunspell_english(aff, dic, &[]).unwrap();
|
||||
for w in ["house", "houses", "walked", "walks", "the"] {
|
||||
assert!(d.check(w), "{w}");
|
||||
}
|
||||
assert!(!d.check("0th") && !d.check("1st"), "parts of compounds only");
|
||||
assert!(!d.check("0th") && !d.check("1th"), "parts of compounds only");
|
||||
assert!(d.check("1st") && d.check("11th") && d.check("111th") && !d.check("11st"), "ordinals by COMPOUNDRULE");
|
||||
assert!(!d.check("housed") && !d.check("hous"));
|
||||
// The directives that change which words are right are still refused.
|
||||
for dir in ["NEEDAFFIX", "CIRCUMFIX", "PSEUDOROOT", "FULLSTRIP"] {
|
||||
|
||||
@@ -43,6 +43,8 @@ const COST_SWAP: u32 = 14;
|
||||
const COST_DROP_OR_ADD: u32 = 15;
|
||||
pub(crate) const COST_MISTYPE: u32 = 16;
|
||||
const COST_SCAN: u32 = 20;
|
||||
/// A missing space ranked after every one-word candidate (English: [`Split::Custom`]).
|
||||
pub(crate) const COST_SPLIT_LAST: u32 = COST_SCAN + 4 * 3 + 1;
|
||||
|
||||
/// What a language adds to its dictionary's tables.
|
||||
pub(crate) struct Tuning {
|
||||
@@ -62,6 +64,9 @@ pub(crate) struct Tuning {
|
||||
pub map_cost: fn(char, char) -> u32,
|
||||
/// The cost of typing letter `c` at `i` of the word (`chars`, lower case) instead.
|
||||
pub mistype_cost: fn(&[char], usize, char) -> u32,
|
||||
/// Offer a word only the dictionary capitalizes (a name, an abbreviation: `Col`, `Wis`) only
|
||||
/// when the word itself is capitalized, and never as half of a missing space (`Che Que`).
|
||||
pub strict_case: bool,
|
||||
}
|
||||
|
||||
/// Where a missing space may go.
|
||||
@@ -78,8 +83,17 @@ pub(crate) enum Split {
|
||||
|
||||
impl Tuning {
|
||||
/// The dictionary's tables alone (Czech, Indonesian, Spanish).
|
||||
pub(crate) const DEFAULT: Tuning =
|
||||
Tuning { split: Split::Anywhere, alphabet: "", map: &[], rep: &[], rep_cost: COST_REP, max_map_changes: 4, map_cost, mistype_cost };
|
||||
pub(crate) const DEFAULT: Tuning = Tuning {
|
||||
split: Split::Anywhere,
|
||||
alphabet: "",
|
||||
map: &[],
|
||||
rep: &[],
|
||||
rep_cost: COST_REP,
|
||||
max_map_changes: 4,
|
||||
map_cost,
|
||||
mistype_cost,
|
||||
strict_case: false,
|
||||
};
|
||||
}
|
||||
|
||||
/// The cost of changing letter `from` to its MAP relative `to`. Adding a missing accent is the
|
||||
@@ -107,6 +121,10 @@ struct Collector<'a> {
|
||||
/// Suggestion → (cost, order found).
|
||||
found: HashMap<String, (u32, usize)>,
|
||||
seq: usize,
|
||||
/// May a word the dictionary only has capitalized be offered ([`Tuning::strict_case`])?
|
||||
names: bool,
|
||||
/// May half of a missing space be one ([`Tuning::strict_case`])?
|
||||
split_names: bool,
|
||||
}
|
||||
|
||||
impl Collector<'_> {
|
||||
@@ -125,9 +143,12 @@ impl Collector<'_> {
|
||||
}
|
||||
let mut words = Vec::new();
|
||||
let mut cost = cost;
|
||||
let names = if cand.contains(' ') { self.split_names } else { self.names };
|
||||
for part in cand.split(' ') {
|
||||
if self.d.contains_suggestible(part) {
|
||||
words.push(part.to_string());
|
||||
} else if !names {
|
||||
return;
|
||||
} else {
|
||||
let c = capitalize(part);
|
||||
if !self.d.contains_suggestible(&c) {
|
||||
@@ -278,7 +299,9 @@ pub(crate) fn suggest_in(d: &AffixDict, word: &str, max: usize, t: &Tuning) -> V
|
||||
let case = casing(word);
|
||||
let input = word.to_lowercase();
|
||||
let chars: Vec<char> = input.chars().collect();
|
||||
let mut col = Collector { d, input: input.clone(), tested: HashMap::new(), found: HashMap::new(), seq: 0 };
|
||||
let names = !t.strict_case || case != Casing::Lower;
|
||||
let split_names = !t.strict_case;
|
||||
let mut col = Collector { d, input: input.clone(), tested: HashMap::new(), found: HashMap::new(), seq: 0, names, split_names };
|
||||
add_case(&mut col, word, &input);
|
||||
let map = groups(d, t);
|
||||
add_map_slips(&mut col, &chars, &map, t);
|
||||
@@ -479,6 +502,9 @@ fn scan(col: &mut Collector, chars: &[char], map: &[Vec<char>]) {
|
||||
if lf.len().abs_diff(chars.len()) > 2 {
|
||||
continue;
|
||||
}
|
||||
if !col.names && form.chars().any(char::is_uppercase) {
|
||||
continue;
|
||||
}
|
||||
let dist = weighted_distance(&lf, chars, &related, limit);
|
||||
if dist <= limit && lf.as_slice() != chars {
|
||||
col.keep(form, COST_SCAN + dist * 3);
|
||||
@@ -504,7 +530,8 @@ mod tests {
|
||||
#[test]
|
||||
fn a_cheaper_later_offer_wins() {
|
||||
let d = AffixDict::from_hunspell("", "1\ncafe\n", &[]).unwrap();
|
||||
let mut col = Collector { d: &d, input: "cafè".into(), tested: HashMap::new(), found: HashMap::new(), seq: 0 };
|
||||
let mut col =
|
||||
Collector { d: &d, input: "cafè".into(), tested: HashMap::new(), found: HashMap::new(), seq: 0, names: true, split_names: true };
|
||||
col.offer("cafe", COST_DROP_DIACRITIC);
|
||||
col.offer("cafe", COST_REP);
|
||||
col.offer("cafe", COST_SCAN);
|
||||
|
||||
+221
-58
@@ -1,29 +1,40 @@
|
||||
//! English spelling never goes unchecked. Its words come from the first of these that is there:
|
||||
//!
|
||||
//! 1. **Embedded**: the Moby word list embedded from craft-i18n at build time (`CRAFT_I18N_DIR`,
|
||||
//! see `build.rs`). When it is there, nothing in this module is used.
|
||||
//! 2. **System Hunspell**: a Hunspell `.aff`/`.dic` pair for the run's own English found by
|
||||
//! [`crate::hunspell`] (`en_US` for American, `en_GB` for British, `en_CA` for Canadian
|
||||
//! English; each name in the dictionaries folder and then the system locations before the
|
||||
//! next name; none on the web). Another English's dictionary is never used: it would mark this
|
||||
//! English's spellings (`colour` in an `en_US` one), so without its own dictionary an English
|
||||
//! uses the built-in list, which has both spellings. A file that can't be read is skipped for
|
||||
//! the next name or folder, and one file is read once for every English that uses it.
|
||||
//! Directives that only matter for other languages (compounding, output conversion) are
|
||||
//! ignored ([`crate::hunspell::load_english`]).
|
||||
//! 3. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
|
||||
//! 1. **Dictionaries folder**: a Hunspell `.aff`/`.dic` pair for the run's own English that the
|
||||
//! user keeps for WordCraft ([`crate::hunspell::user_dirs`]: the folder set in the options,
|
||||
//! `WORDCRAFT_DICTIONARIES`, WordCraft's own `dictionaries` folder).
|
||||
//! 2. **Installed English pack**: reserved for Language packs (craft-libs#485, CP4.2).
|
||||
//! 3. **Embedded SCOWL**: the run's English's SCOWL dictionary (en-US, en-GB or en-CA) embedded
|
||||
//! from craft-i18n at build time ([`crate::scowl`]); a word it knows (British English: also
|
||||
//! the `-ize` spelling of one it knows, [`ise_spelling`]), or the supplement ([`EXTRA`]), is
|
||||
//! right; never one only another English knows. When it is there, the system locations aren't
|
||||
//! searched.
|
||||
//! 4. **System Hunspell**: a pair for the run's own English in the system locations (none on the
|
||||
//! web).
|
||||
//! 5. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
|
||||
//! the crate (without its break points), with WordCraft's own supplement of modern words
|
||||
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled, as with the
|
||||
//! embedded list.
|
||||
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled. (The Moby list
|
||||
//! embedded from craft-i18n is the same list; it is used for hyphenation only.)
|
||||
//!
|
||||
//! A Hunspell dictionary that is there but can't be read (unsupported encoding, too large, a
|
||||
//! parse error, no words) is skipped, and its reason is reported by
|
||||
//! Hunspell dictionaries are found by [`crate::hunspell`] by their own English's names only
|
||||
//! (`en_US` for American, `en_GB` for British, `en_CA` for Canadian English), each name in every
|
||||
//! folder before the next name. Another English's dictionary is never used: it would mark this
|
||||
//! English's spellings (`colour` in an `en_US` one). A file that can't be read is skipped for the
|
||||
//! next name or folder, and one file is read once for every English that uses it. `COMPOUNDRULE`
|
||||
//! (ordinals) is read; flag-based compounding and `OCONV` are ignored
|
||||
//! ([`crate::hunspell::load_english`]).
|
||||
//!
|
||||
//! Whichever it is, the dialect policy of `en-variants.txt` ([`crate::en_variants`], when it is
|
||||
//! embedded) flags the other dialect's spellings and accepts this dialect's spellings of known
|
||||
//! words. A Hunspell dictionary that is there but can't be read (unsupported encoding, too large,
|
||||
//! a parse error, no words) is skipped, and its reason is reported by
|
||||
//! [`crate::dictionary_load_error`]. Non-English text is never checked against any of these.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{Arc, OnceLock, PoisonError, RwLock};
|
||||
|
||||
use crate::affix::AffixDict;
|
||||
use crate::affix_suggest::{Split, Tuning};
|
||||
use crate::dict::Dictionary;
|
||||
use crate::{Options, ProofLang};
|
||||
|
||||
@@ -47,7 +58,7 @@ const MAX_CANDIDATES: usize = 8;
|
||||
/// Where English spelling comes from now ([`crate::english_source`]).
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum EnglishSource {
|
||||
/// The Moby word list embedded from craft-i18n at build time.
|
||||
/// The English's SCOWL dictionary embedded from craft-i18n at build time.
|
||||
Embedded,
|
||||
/// An English Hunspell dictionary from the dictionaries folder or the system: its `.dic` file.
|
||||
Hunspell(PathBuf),
|
||||
@@ -112,14 +123,9 @@ impl State {
|
||||
|
||||
static HUNSPELL: RwLock<State> = RwLock::new(State::new());
|
||||
|
||||
/// Why the embedded word list failed to decode, if it did.
|
||||
static EMBEDDED_ERROR: OnceLock<String> = OnceLock::new();
|
||||
|
||||
/// Remember that the embedded word list is there but failed to decode (English then uses the
|
||||
/// next source).
|
||||
/// The embedded Moby list (hyphenation) failed to decode: log it. Spelling doesn't use it.
|
||||
pub(crate) fn note_embedded_error(e: String) {
|
||||
log::error!("embedded English word list: {e}; English spelling falls back to Hunspell or the built-in word list");
|
||||
let _ = EMBEDDED_ERROR.set(e);
|
||||
log::error!("embedded English word list: {e}; English hyphenation falls back to the patterns");
|
||||
}
|
||||
|
||||
/// The English Hunspell dictionary for `lang` (read on first use).
|
||||
@@ -185,12 +191,14 @@ fn key(lang: ProofLang) -> &'static str {
|
||||
}
|
||||
|
||||
/// Read the first Hunspell pair of `lang`'s own that loads, trying each name in each folder
|
||||
/// ([`crate::hunspell::candidates`]), with the first failure before it (the file `lang` would
|
||||
/// rather use), for [`note_error`].
|
||||
/// ([`crate::hunspell::candidates`]; with SCOWL embedded, the dictionaries folder only:
|
||||
/// [`crate::hunspell::user_candidates`]), with the first failure before it (the file `lang`
|
||||
/// would rather use), for [`note_error`].
|
||||
fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) {
|
||||
let candidates = crate::hunspell::candidates(names(lang));
|
||||
let candidates =
|
||||
if crate::scowl::dict(lang).is_some() { crate::hunspell::user_candidates(names(lang)) } else { crate::hunspell::candidates(names(lang)) };
|
||||
if candidates.is_empty() {
|
||||
log::info!("English spelling: no Hunspell {} dictionary installed; using the built-in word list", key(lang));
|
||||
log::info!("English spelling: no Hunspell {} dictionary to read first; using SCOWL or the built-in word list", key(lang));
|
||||
return (None, None);
|
||||
}
|
||||
let mut first_error: Option<(PathBuf, String)> = None;
|
||||
@@ -209,7 +217,7 @@ fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) {
|
||||
}
|
||||
}
|
||||
}
|
||||
log::warn!("English spelling: no {} Hunspell dictionary could be read; using the built-in word list", key(lang));
|
||||
log::warn!("English spelling: no {} Hunspell dictionary could be read; using SCOWL or the built-in word list", key(lang));
|
||||
(None, first_error)
|
||||
}
|
||||
|
||||
@@ -232,45 +240,150 @@ pub(crate) fn reload() {
|
||||
}
|
||||
}
|
||||
|
||||
/// Where the spelling of `lang` (an English) comes from now. Reads the Hunspell dictionary once
|
||||
/// when there is no embedded data (it is cached afterwards).
|
||||
/// Where the spelling of `lang` (an English) comes from now, in the order of the module docs.
|
||||
/// Reads the Hunspell dictionary once (it is cached afterwards).
|
||||
pub(crate) fn source(lang: ProofLang) -> EnglishSource {
|
||||
if crate::english_data() {
|
||||
EnglishSource::Embedded
|
||||
} else {
|
||||
hunspell(lang).map_or(EnglishSource::CoreList, |(_, dic)| EnglishSource::Hunspell(dic))
|
||||
match speller(lang) {
|
||||
Speller::Hunspell(_, dic) => EnglishSource::Hunspell(dic),
|
||||
Speller::Scowl(..) => EnglishSource::Embedded,
|
||||
Speller::List => EnglishSource::CoreList,
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a source of `lang`'s spelling failed to load: the embedded word list, or (without it) the
|
||||
/// Hunspell dictionary found for it. Reads nothing itself.
|
||||
/// Why a source of `lang`'s spelling failed to load: its embedded SCOWL dictionary, or the
|
||||
/// Hunspell dictionary it would rather have used. Reads nothing itself.
|
||||
pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
|
||||
if let Some(e) = EMBEDDED_ERROR.get() {
|
||||
return Some(e.clone());
|
||||
}
|
||||
if crate::english_data() {
|
||||
return None;
|
||||
}
|
||||
crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason)
|
||||
crate::scowl::load_error(lang).or_else(|| crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason))
|
||||
}
|
||||
|
||||
/// Is `word` spelled right in `lang` (an English) when no word list is embedded: against the
|
||||
/// Hunspell dictionary, or else the built-in word list.
|
||||
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options) -> bool {
|
||||
/// What `lang` is checked against now.
|
||||
enum Speller {
|
||||
/// A dictionaries-folder or system Hunspell dictionary, and its `.dic` file.
|
||||
Hunspell(Arc<AffixDict>, PathBuf),
|
||||
/// `lang`'s own SCOWL dictionary (every English's, for English with no dialect: [`generic`]),
|
||||
/// and whether British `-ize` spellings are right too ([`ise_spelling`]).
|
||||
Scowl(Vec<&'static AffixDict>, bool),
|
||||
/// The built-in word list.
|
||||
List,
|
||||
}
|
||||
|
||||
fn speller(lang: ProofLang) -> Speller {
|
||||
pick(hunspell(lang), lang)
|
||||
}
|
||||
|
||||
/// The lookup order: the Hunspell dictionary `found` for `lang` (with SCOWL embedded, only one in
|
||||
/// the dictionaries folder is looked for), then SCOWL, then the built-in list. (The installed
|
||||
/// English pack, craft-libs#485, goes between the first two.)
|
||||
fn pick(found: Loaded, lang: ProofLang) -> Speller {
|
||||
if let Some((d, dic)) = found {
|
||||
return Speller::Hunspell(d, dic);
|
||||
}
|
||||
match crate::scowl::dict(lang) {
|
||||
Some(d) => Speller::Scowl(vec![d], crate::scowl::is_british(lang)),
|
||||
None => Speller::List,
|
||||
}
|
||||
}
|
||||
|
||||
/// [`speller`] for English with no dialect ([`crate::is_correct`]): a SCOWL word of any English
|
||||
/// is right, as in the built-in list, which has both spellings.
|
||||
fn generic(s: Speller) -> Speller {
|
||||
match s {
|
||||
Speller::Scowl(..) => {
|
||||
let all: Vec<_> = [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa].into_iter().filter_map(crate::scowl::dict).collect();
|
||||
Speller::Scowl(all, false)
|
||||
}
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
impl Speller {
|
||||
/// Is `w` (as written, `’` read as `'`) a word: as a whole, or each part of a hyphenated one?
|
||||
fn knows(&self, w: &str) -> bool {
|
||||
match self {
|
||||
Speller::Hunspell(d, _) => affix_correct(&[d.as_ref()], w, false),
|
||||
Speller::Scowl(ds, ise) => {
|
||||
affix_correct(ds, w, true)
|
||||
|| (*ise
|
||||
&& ise_spelling(w).is_some_and(|x| affix_correct(ds, &x, true))
|
||||
&& ize_elsewhere(w, |d, p| affix_correct(&[d], p, false)))
|
||||
}
|
||||
Speller::List => list_correct(w),
|
||||
}
|
||||
}
|
||||
|
||||
/// Is the lower-case base form `p` a word (for the dialect policy's other spelling)?
|
||||
fn knows_lower(&self, p: &str) -> bool {
|
||||
match self {
|
||||
Speller::Hunspell(d, _) => d.check(p),
|
||||
Speller::Scowl(ds, ise) => {
|
||||
ds.iter().any(|d| d.check(p) || (*ise && ise_spelling(p).is_some_and(|x| d.check(&x)) && ize_elsewhere(p, |d, p| d.check(p))))
|
||||
}
|
||||
Speller::List => crate::known_in_word_list(p),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Is `word` spelled right in `lang` (an English), against its own dictionary ([`speller`]); with
|
||||
/// `policy` false, English with no dialect: any English's SCOWL dictionary ([`generic`]). With
|
||||
/// `policy`, also by the dialect policy of `en-variants.txt`: the other dialect's spellings are
|
||||
/// wrong and this dialect's spellings of known words right. The user dictionary overrules both.
|
||||
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options, policy: bool) -> bool {
|
||||
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’').trim_matches(['\'', '’']);
|
||||
if crate::accepted_by_options(w, opts) || !w.chars().any(|c| c.is_ascii_alphabetic()) {
|
||||
return true; // other scripts: not English
|
||||
}
|
||||
let w = w.replace('’', "'");
|
||||
match hunspell(lang) {
|
||||
Some((d, _)) => hunspell_correct(&d, &w),
|
||||
None => list_correct(&w),
|
||||
let lower = w.to_lowercase();
|
||||
let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect();
|
||||
if policy && parts.iter().any(|p| crate::en_variants::is_foreign(p, lang)) {
|
||||
return false;
|
||||
}
|
||||
let s = if policy { speller(lang) } else { generic(speller(lang)) };
|
||||
s.knows(&w)
|
||||
|| (policy && !parts.is_empty() && parts.iter().all(|p| s.knows_lower(p) || crate::en_variants::known_variant(p, lang, |x| s.knows_lower(x))))
|
||||
}
|
||||
|
||||
/// Against an installed English Hunspell dictionary: the word, or each part of a hyphenated one.
|
||||
fn hunspell_correct(d: &AffixDict, w: &str) -> bool {
|
||||
let known = |p: &str| p.chars().count() <= 1 || d.check(p) || crate::in_user_dictionary(&p.to_lowercase());
|
||||
/// `w` with its `-ize`/`-yze` endings (and their forms: `-ization`, `-izing`…) spelled `-ise`/
|
||||
/// `-yse`, when it has any: British English takes both (Word accepts `organize` and `organise` in
|
||||
/// en-GB), and SCOWL's en-GB dictionary has only `-ise`. A `z` between `i`/`y` and a vowel is
|
||||
/// swapped; the `-ize` form counts only when American or Canadian English spells the word so
|
||||
/// ([`ize_elsewhere`]), or `wize` would pass as `wise`.
|
||||
pub(crate) fn ise_spelling(w: &str) -> Option<String> {
|
||||
let b = w.as_bytes();
|
||||
let mut out = b.to_vec();
|
||||
let mut changed = false;
|
||||
for i in 1..b.len() {
|
||||
let after = b.get(i + 1).is_some_and(|c| b"aeiAEI".contains(c));
|
||||
if after
|
||||
&& matches!(b.get(i - 1), Some(b'i' | b'y' | b'I' | b'Y'))
|
||||
&& let Some(c) = out.get_mut(i)
|
||||
{
|
||||
match *c {
|
||||
b'z' => *c = b's',
|
||||
b'Z' => *c = b'S',
|
||||
_ => continue,
|
||||
}
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
if changed { String::from_utf8(out).ok() } else { None }
|
||||
}
|
||||
|
||||
/// Is `w` a word in American or Canadian English's SCOWL dictionary (`check` asks one): the
|
||||
/// `-ize` spellings British English takes are theirs (`organize`, never `advize` or `wize`).
|
||||
fn ize_elsewhere(w: &str, check: impl Fn(&AffixDict, &str) -> bool) -> bool {
|
||||
[ProofLang::En, ProofLang::EnCa].into_iter().filter_map(crate::scowl::dict).any(|d| check(d, w))
|
||||
}
|
||||
|
||||
/// Against affix dictionaries (any of them): the word, or each part of a hyphenated one; with
|
||||
/// `supplement`, our supplement's words count too.
|
||||
fn affix_correct(ds: &[&AffixDict], w: &str, supplement: bool) -> bool {
|
||||
let known = |p: &str| {
|
||||
p.chars().count() <= 1
|
||||
|| ds.iter().any(|d| d.check(p))
|
||||
|| crate::in_user_dictionary(&p.to_lowercase())
|
||||
|| (supplement && crate::known_in_supplement(&p.to_lowercase()))
|
||||
};
|
||||
known(w) || (w.contains('-') && w.split('-').filter(|p| !p.is_empty()).all(known))
|
||||
}
|
||||
|
||||
@@ -285,15 +398,25 @@ fn list_correct(w: &str) -> bool {
|
||||
crate::known_in_word_list(&lower) || (lower.contains('-') && lower.split('-').filter(|p| !p.is_empty()).all(crate::known_in_word_list))
|
||||
}
|
||||
|
||||
/// Spelling suggestions in `lang` (an English) when no word list is embedded: from the Hunspell
|
||||
/// dictionary, or else the built-in word list.
|
||||
/// Spelling suggestions in `lang` (an English), from its dictionary ([`speller`]).
|
||||
pub(crate) fn suggest(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
|
||||
match hunspell(lang) {
|
||||
Some((d, _)) => crate::cs::suggest::suggest_in(&d, word, max),
|
||||
None => list_suggest(word, max),
|
||||
match speller(lang) {
|
||||
Speller::Hunspell(d, _) => crate::affix_suggest::suggest_in(&d, word, max, &TUNING),
|
||||
Speller::Scowl(ds, _) => ds.first().map_or_else(Vec::new, |d| crate::affix_suggest::suggest_in(d, word, max, &TUNING)),
|
||||
Speller::List => list_suggest(word, max),
|
||||
}
|
||||
}
|
||||
|
||||
/// English suggestions from an affix dictionary: its own tables, a missing space only between two
|
||||
/// words of three letters or more and after every one-word candidate (`publically` → `publicly`
|
||||
/// before `public ally`), and no abbreviation or name offered for a lower-case word (`col`, `Wis`).
|
||||
const TUNING: Tuning = Tuning { split: Split::Custom(split_cost), strict_case: true, ..Tuning::DEFAULT };
|
||||
|
||||
/// The cost of a missing space in English ([`TUNING`]).
|
||||
fn split_cost(a: &str, b: &str) -> Option<u32> {
|
||||
(a.chars().count() >= 3 && b.chars().count() >= 3).then_some(crate::affix_suggest::COST_SPLIT_LAST)
|
||||
}
|
||||
|
||||
/// Letters an edit may add or put in a word's place.
|
||||
const ALPHABET: &str = "abcdefghijklmnopqrstuvwxyz'";
|
||||
|
||||
@@ -460,6 +583,46 @@ mod tests {
|
||||
assert!(start.elapsed() < std::time::Duration::from_secs(5), "{:?}", start.elapsed());
|
||||
}
|
||||
|
||||
/// The lookup order (README): a dictionaries-folder English comes before SCOWL, SCOWL before
|
||||
/// the built-in list; each English looks for its own dictionary only, and with SCOWL embedded
|
||||
/// the system locations aren't searched. (Checked here without touching the process-wide
|
||||
/// folder, which every English test running meanwhile would see.)
|
||||
#[test]
|
||||
fn lookup_order() {
|
||||
let tiny = Arc::new(AffixDict::from_hunspell("", "1\nzzyzx\n", &[]).unwrap());
|
||||
let file = PathBuf::from("/folder/en_US.dic");
|
||||
for lang in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
|
||||
assert!(matches!(pick(Some((tiny.clone(), file.clone())), lang), Speller::Hunspell(_, p) if p == file), "{lang:?}");
|
||||
let fallback = pick(None, lang);
|
||||
if crate::scowl::dict(lang).is_some() {
|
||||
assert!(
|
||||
matches!(&fallback, Speller::Scowl(ds, _) if ds.len() == 1 && std::ptr::eq(ds[0], crate::scowl::dict(lang).unwrap())),
|
||||
"{lang:?}"
|
||||
);
|
||||
assert!(fallback.knows("21st") && !fallback.knows("becuase"), "{lang:?}");
|
||||
assert_eq!(fallback.knows("colour"), lang != ProofLang::En, "{lang:?}: its own dictionary only");
|
||||
} else {
|
||||
assert!(matches!(fallback, Speller::List), "{lang:?}");
|
||||
}
|
||||
}
|
||||
let dir = std::env::temp_dir().join(format!("wc-proof-en-order-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
for name in ["en_US", "en_GB", "en_CA"] {
|
||||
std::fs::write(dir.join(format!("{name}.aff")), "SET UTF-8\n").unwrap();
|
||||
std::fs::write(dir.join(format!("{name}.dic")), "1\nzzyzx\n").unwrap();
|
||||
}
|
||||
let found = |names: &[&str]| -> Vec<String> {
|
||||
crate::hunspell::candidates_in(std::slice::from_ref(&dir), names)
|
||||
.iter()
|
||||
.filter_map(|(_, d)| d.file_stem().map(|n| n.to_string_lossy().into_owned()))
|
||||
.collect()
|
||||
};
|
||||
assert_eq!(found(names(ProofLang::EnGb)), ["en_GB"], "a user's en_US doesn't stand in for SCOWL's en-GB");
|
||||
assert_eq!(found(names(ProofLang::EnCa)), ["en_CA"]);
|
||||
assert_eq!(found(names(ProofLang::En)), ["en_US"]);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
/// craft-libs#482 review: a load that started before a reload (the dictionaries folder
|
||||
/// changed) serves its own check but isn't kept, so the next check reads the new folder; a
|
||||
/// load after it is kept, and shared with the Englishes that read the same file.
|
||||
|
||||
@@ -154,6 +154,25 @@ pub fn candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
|
||||
candidates_in(&search_dirs(lo.as_deref()), names)
|
||||
}
|
||||
|
||||
/// [`candidates`] in the folders the user keeps dictionaries in for WordCraft ([`user_dirs`])
|
||||
/// only, not the system locations.
|
||||
pub fn user_candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
|
||||
candidates_in(&user_dirs(), names)
|
||||
}
|
||||
|
||||
/// The folders of [`search_dirs`] the user keeps dictionaries in for WordCraft: the one set with
|
||||
/// [`set_dictionary_dir`], `WORDCRAFT_DICTIONARIES`, and WordCraft's own `dictionaries` folder
|
||||
/// under the user's application data. The rest are the system's (and LibreOffice's).
|
||||
pub fn user_dirs() -> Vec<PathBuf> {
|
||||
let extra = EXTRA_DIR.read().unwrap_or_else(std::sync::PoisonError::into_inner).clone();
|
||||
let env = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()).map(PathBuf::from);
|
||||
let wordcraft_own = |d: &Path| {
|
||||
d.file_name().is_some_and(|n| n == "dictionaries")
|
||||
&& d.parent().and_then(Path::file_name).is_some_and(|n| n.eq_ignore_ascii_case("wordcraft"))
|
||||
};
|
||||
search_dirs(None).into_iter().filter(|d| Some(d) == extra.as_ref() || Some(d) == env.as_ref() || wordcraft_own(d)).collect()
|
||||
}
|
||||
|
||||
/// [`candidates`] in `dirs`.
|
||||
pub fn candidates_in(dirs: &[PathBuf], names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
|
||||
let mut out: Vec<(PathBuf, PathBuf)> = Vec::new();
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||
pub enum ProofLang {
|
||||
/// American English (`en`, `en-US`, and every region not listed for [`ProofLang::EnGb`] or
|
||||
/// [`ProofLang::EnCa`]): the public-domain Moby word list, the English grammar rules and the
|
||||
/// [`ProofLang::EnCa`]): SCOWL's en-US dictionary ([`crate::scowl`]), the English grammar rules and the
|
||||
/// English hyphenation dictionary and patterns. British spellings (`colour`, `centre`,
|
||||
/// `organise`) are flagged, with the American form suggested ([`crate::en_variants`]). Also
|
||||
/// text with no language at all (WordCraft's default editing language is US English, like
|
||||
|
||||
+199
-152
@@ -1,8 +1,9 @@
|
||||
//! Proofing for the Craft Apps (moved from WordCraft's `wordcraft-proof`), per language. A run's language (its BCP 47 tag) picks the tools through
|
||||
//! [`ProofLang`]; the `*_in` functions take it, the older English-only ones remain.
|
||||
//!
|
||||
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the
|
||||
//! public-domain Moby word list (~160k words, the same list that drives hyphenation) plus our
|
||||
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the SCOWL
|
||||
//! 2026 Hunspell dictionary of the run's English ([`scowl`]; else the public-domain Moby word
|
||||
//! list, ~160k words, the same list that drives hyphenation) plus our
|
||||
//! supplement of modern words it lacks (craft-i18n's `language-data/en-us/en-extra.txt`: `desktop`, `email`,
|
||||
//! `download`…), with inflections (`-s`, `-es`, `-ed`, `-ing`, `-ly`, `'s`…), a user dictionary and ignore lists;
|
||||
//! suggestions by edit distance. Grammar checks that are cheap and reliable: repeated words,
|
||||
@@ -48,11 +49,12 @@
|
||||
//!
|
||||
//! [`check_text`] checks a paragraph whose runs are in different languages.
|
||||
//!
|
||||
//! **Language data** (the Moby word list, the pattern files, our word supplements) is embedded at
|
||||
//! build time from craft-i18n's `language-data/` (`CRAFT_I18N_DIR`, see `build.rs`). English
|
||||
//! spelling never goes unchecked: without the embedded word list it uses an English Hunspell
|
||||
//! dictionary installed on the system, and without one the whole Moby word list built into the
|
||||
//! crate (see [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
|
||||
//! **Language data** (the SCOWL English dictionaries, the Moby word list, the pattern files, our
|
||||
//! word supplements) is embedded at build time from craft-i18n's `language-data/`
|
||||
//! (`CRAFT_I18N_DIR`, see `build.rs`). English spelling never goes unchecked: a dictionaries-folder
|
||||
//! English first, then the embedded SCOWL dictionary, then an English Hunspell dictionary
|
||||
//! installed on the system, then the whole Moby word list built into the crate (see
|
||||
//! [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
|
||||
//! rule-based hyphenation (Bulgarian, Turkish, Russian) are code and still work.
|
||||
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
|
||||
|
||||
@@ -78,6 +80,7 @@ pub mod patterns;
|
||||
pub mod pl;
|
||||
pub mod pt;
|
||||
pub mod ru;
|
||||
pub mod scowl;
|
||||
mod slot;
|
||||
mod spelling;
|
||||
pub mod tr;
|
||||
@@ -315,7 +318,7 @@ pub fn installed_dictionaries() -> Vec<(ProofLang, Option<std::path::PathBuf>)>
|
||||
|
||||
/// Why the dictionary found for `lang` failed to load (unsupported encoding, too large, a parse
|
||||
/// error), if it was read and failed. Doesn't read anything itself: a dictionary nobody has
|
||||
/// checked yet reports `None` (#2830). For an English, the embedded word list or (without it) the
|
||||
/// checked yet reports `None` (#2830). For an English, its embedded SCOWL dictionary or the
|
||||
/// English Hunspell dictionary that failed; English then uses the next source ([`english_source`]).
|
||||
pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
|
||||
if matches!(lang, ProofLang::En | ProofLang::EnGb | ProofLang::EnCa) {
|
||||
@@ -324,12 +327,11 @@ pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
|
||||
hunspell::load_error(language(lang)?.slot.name()).map(|(_, reason)| reason)
|
||||
}
|
||||
|
||||
/// Where the spelling of an English `lang` comes from now (`None` for other languages): the word
|
||||
/// list embedded at build time, else an English Hunspell dictionary from the dictionaries folder
|
||||
/// or the system (the run's English first), else the Moby word list built into the crate.
|
||||
/// Each English uses only its own Hunspell dictionary (`en_US`, `en_GB`, `en_CA`): without it,
|
||||
/// the built-in list, which has both spellings. Reads the Hunspell dictionary once when there is no
|
||||
/// embedded word list (it is cached afterwards).
|
||||
/// Where the spelling of an English `lang` comes from now (`None` for other languages), in this
|
||||
/// order: a Hunspell dictionary of its own (`en_US`, `en_GB`, `en_CA`) in the dictionaries
|
||||
/// folder, the SCOWL dictionary embedded at build time, one of its own on the system, else the
|
||||
/// Moby word list built into the crate. Another English's dictionary is never used. Reads the
|
||||
/// Hunspell dictionary once (it is cached afterwards).
|
||||
pub fn english_source(lang: ProofLang) -> Option<EnglishSource> {
|
||||
lang.is_english().then(|| english::source(lang))
|
||||
}
|
||||
@@ -467,7 +469,7 @@ pub(crate) fn is_common_word(w: &str) -> bool {
|
||||
}
|
||||
|
||||
/// Our supplement of common words the Moby list lacks (`desktop`, `email`, `download`…): one
|
||||
/// lowercase base form per line, `#` comments. Inflections come from [`known_core`].
|
||||
/// lowercase base form per line, `#` comments. Inflections come from [`known_with`].
|
||||
use data::EXTRA_EN;
|
||||
|
||||
/// The supplement's words: the Language data's copy, else the crate's own ([`english::EXTRA`]).
|
||||
@@ -477,21 +479,18 @@ fn extra_words() -> &'static HashSet<&'static str> {
|
||||
E.get_or_init(|| extra.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect())
|
||||
}
|
||||
|
||||
/// A base form in the dictionary or the supplement.
|
||||
fn base_known(d: &dict::Dictionary, w: &str) -> bool {
|
||||
d.get(w).is_some() || extra_words().contains(w)
|
||||
}
|
||||
|
||||
fn known_core(w: &str) -> bool {
|
||||
let d = dict::Dictionary::en_us();
|
||||
known_with(w, &|s| base_known(d, s))
|
||||
}
|
||||
|
||||
/// [`known_core`] against the word list built into the crate (no embedded word list).
|
||||
/// Is `w` (lower case) a word of the built-in Moby list or the supplement, with the inflection
|
||||
/// rules ([`known_with`])?
|
||||
pub(crate) fn known_in_word_list(w: &str) -> bool {
|
||||
known_with(w, &|s| english::word_list_contains(s) || extra_words().contains(s))
|
||||
}
|
||||
|
||||
/// Is `w` (lower case) a word of the supplement alone, with the inflection rules (the SCOWL
|
||||
/// dictionaries take it as an addition)?
|
||||
pub(crate) fn known_in_supplement(w: &str) -> bool {
|
||||
known_with(w, &|s| extra_words().contains(s))
|
||||
}
|
||||
|
||||
/// Is `w` (lower case) a word, by `base_known` (a base form in the word list), the common words,
|
||||
/// the user dictionary and the inflection and prefix rules?
|
||||
fn known_with(w: &str, base_known: &dyn Fn(&str) -> bool) -> bool {
|
||||
@@ -615,66 +614,16 @@ pub(crate) fn accepted_by_options(word: &str, opts: &Options) -> bool {
|
||||
|| in_user_dictionary(&w.to_lowercase())
|
||||
}
|
||||
|
||||
/// Is the English word list embedded? Without it English is checked against an installed
|
||||
/// Hunspell dictionary or the word list built into the crate ([`english`]).
|
||||
pub(crate) fn english_data() -> bool {
|
||||
!dict::Dictionary::en_us().is_empty()
|
||||
}
|
||||
|
||||
/// [`is_correct`] with the File › Options › Proofing choices.
|
||||
/// [`is_correct`] with the File › Options › Proofing choices: English with no dialect, so a word
|
||||
/// any English spells so is right (`color` and `colour`; American English's dictionary when it
|
||||
/// isn't SCOWL: [`english_source`]), without the dialect policy.
|
||||
pub fn is_correct_with(word: &str, opts: &Options) -> bool {
|
||||
if !english_data() {
|
||||
return english::is_correct(word, ProofLang::En, opts);
|
||||
}
|
||||
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'');
|
||||
let w = w.trim_matches('\'');
|
||||
if accepted_by_options(w, opts) {
|
||||
return true;
|
||||
}
|
||||
if !w.is_ascii() && !w.chars().any(|c| c.is_ascii_alphabetic()) {
|
||||
return true; // other scripts: no dictionary
|
||||
}
|
||||
let lower = w.to_lowercase().replace('’', "'");
|
||||
if known_core(&lower) {
|
||||
return true;
|
||||
}
|
||||
// Hyphenated compounds: each part.
|
||||
if lower.contains('-') {
|
||||
return lower.split('-').all(|p| p.is_empty() || known_core(p));
|
||||
}
|
||||
false
|
||||
english::is_correct(word, ProofLang::En, opts, false)
|
||||
}
|
||||
|
||||
/// Spelling suggestions for a misspelled word (best first, at most `max`).
|
||||
pub fn suggest(word: &str, max: usize) -> Vec<String> {
|
||||
if !english_data() {
|
||||
return english::suggest(word, max, ProofLang::En);
|
||||
}
|
||||
suggest_from(word, max, dict::Dictionary::en_us().iter().map(|(w, _)| w))
|
||||
}
|
||||
|
||||
/// [`suggest`] from `words` (lower case) and our supplement.
|
||||
pub(crate) fn suggest_from<'a>(word: &str, max: usize, words: impl Iterator<Item = &'a str>) -> Vec<String> {
|
||||
let lower = word.to_lowercase();
|
||||
let n = lower.chars().count();
|
||||
if n == 0 || n > 40 {
|
||||
return Vec::new();
|
||||
}
|
||||
let mut scored: Vec<(usize, i64, String)> = Vec::new();
|
||||
let target: Vec<char> = lower.chars().collect();
|
||||
for w in words.chain(extra_words().iter().copied()) {
|
||||
let wn = w.chars().count();
|
||||
if wn + 2 < n || wn > n + 2 {
|
||||
continue;
|
||||
}
|
||||
let dist = damerau(&target, w);
|
||||
if dist <= 2 {
|
||||
let same_first = w.chars().next() == target.first().copied();
|
||||
scored.push((dist, if same_first { 0 } else { 1 }, w.to_string()));
|
||||
}
|
||||
}
|
||||
scored.sort_by(|a, b| (a.0, a.1, a.2.len().abs_diff(n)).cmp(&(b.0, b.1, b.2.len().abs_diff(n))).then(a.2.cmp(&b.2)));
|
||||
scored.into_iter().take(max).map(|(_, _, w)| match_case(word, w)).collect()
|
||||
english::suggest(word, max, ProofLang::En)
|
||||
}
|
||||
|
||||
/// `w` (lower case) in the case of `word`: all capitals, an initial capital, or as it is.
|
||||
@@ -709,26 +658,11 @@ fn english_preferred(lower: &str, lang: ProofLang) -> Option<String> {
|
||||
}
|
||||
|
||||
/// Is `word` spelled right in the English `lang` ([`ProofLang::En`], [`ProofLang::EnGb`],
|
||||
/// [`ProofLang::EnCa`])? The shared word list's words, less the other dialect's spellings
|
||||
/// [`ProofLang::EnCa`])? Its dictionary's words ([`english`]), less the other dialect's spellings
|
||||
/// (`colour` in American English, `color` in British), plus this dialect's spellings of the
|
||||
/// list's words ([`en_variants`]). The user dictionary overrules both.
|
||||
/// dictionary's words ([`en_variants`]). The user dictionary overrules both.
|
||||
fn is_correct_english(word: &str, lang: ProofLang, opts: &Options) -> bool {
|
||||
if !english_data() {
|
||||
return english::is_correct(word, lang, opts);
|
||||
}
|
||||
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'').trim_matches('\'');
|
||||
if accepted_by_options(w, opts) {
|
||||
return true;
|
||||
}
|
||||
let lower = w.to_lowercase().replace('’', "'");
|
||||
if in_user_dictionary(&lower) {
|
||||
return true;
|
||||
}
|
||||
let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect();
|
||||
if parts.iter().any(|p| en_variants::is_foreign(p, lang)) {
|
||||
return false;
|
||||
}
|
||||
is_correct_with(word, opts) || (!parts.is_empty() && parts.iter().all(|p| known_core(p) || en_variants::known_variant(p, lang, known_core)))
|
||||
english::is_correct(word, lang, opts, true)
|
||||
}
|
||||
|
||||
/// Spelling suggestions in the English `lang`: this dialect's spelling first when `word` is the
|
||||
@@ -739,7 +673,7 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
|
||||
if let Some(p) = english_preferred(&lower, lang) {
|
||||
out.push(match_case(word, p));
|
||||
}
|
||||
let shared = if english_data() { suggest(word, max.saturating_add(2)) } else { english::suggest(word, max.saturating_add(2), lang) };
|
||||
let shared = english::suggest(word, max.saturating_add(2), lang);
|
||||
for s in shared {
|
||||
let s = english_preferred(&s.to_lowercase(), lang).map_or(s, |p| match_case(word, p));
|
||||
if !out.contains(&s) {
|
||||
@@ -750,41 +684,6 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
|
||||
out
|
||||
}
|
||||
|
||||
/// Optimal string alignment distance, early-exit at 3.
|
||||
pub(crate) fn damerau(a: &[char], b: &str) -> usize {
|
||||
let b: Vec<char> = b.chars().collect();
|
||||
let (n, m) = (a.len(), b.len());
|
||||
let mut prev2 = vec![0usize; m + 1];
|
||||
let mut prev: Vec<usize> = (0..=m).collect();
|
||||
let mut cur = vec![0usize; m + 1];
|
||||
for i in 1..=n {
|
||||
if let Some(c) = cur.get_mut(0) {
|
||||
*c = i;
|
||||
}
|
||||
let mut row_min = i;
|
||||
for j in 1..=m {
|
||||
let (ai, bj) = (a.get(i - 1), b.get(j - 1));
|
||||
let cost = usize::from(ai != bj);
|
||||
let mut v = (prev.get(j).copied().unwrap_or(9) + 1)
|
||||
.min(cur.get(j - 1).copied().unwrap_or(9) + 1)
|
||||
.min(prev.get(j - 1).copied().unwrap_or(9) + cost);
|
||||
if i > 1 && j > 1 && a.get(i - 1) == b.get(j - 2) && a.get(i - 2) == b.get(j - 1) {
|
||||
v = v.min(prev2.get(j - 2).copied().unwrap_or(9) + 1);
|
||||
}
|
||||
if let Some(c) = cur.get_mut(j) {
|
||||
*c = v;
|
||||
}
|
||||
row_min = row_min.min(v);
|
||||
}
|
||||
if row_min > 2 {
|
||||
return 3;
|
||||
}
|
||||
std::mem::swap(&mut prev2, &mut prev);
|
||||
std::mem::swap(&mut prev, &mut cur);
|
||||
}
|
||||
prev.get(m).copied().unwrap_or(9)
|
||||
}
|
||||
|
||||
/// Words in `text`: (byte start, byte end). Combining marks belong to the word they follow, so a
|
||||
/// word typed decomposed (`tambie\u{301}n`, NFD) is one word, as its precomposed form is.
|
||||
pub fn words(text: &str) -> Vec<(usize, usize)> {
|
||||
@@ -1212,11 +1111,20 @@ mod tests {
|
||||
set_dictionaries_folder(Some(dir.to_path_buf()));
|
||||
}
|
||||
|
||||
/// R6.4, English never goes unchecked: with no embedded word list, an English Hunspell
|
||||
/// dictionary (here in the dictionaries folder) is used before the word list built into the
|
||||
/// crate; with one, the embedded list stays in use whatever is installed.
|
||||
/// Language-data builds embed SCOWL: an English dictionary in the dictionaries folder would
|
||||
/// then come before it for every English test running meanwhile, so the folder tests below
|
||||
/// run without the data only; `english::tests` checks the order with the data.
|
||||
fn scowl_embedded() -> bool {
|
||||
scowl::dict(ProofLang::En).is_some()
|
||||
}
|
||||
|
||||
/// R6.4, English never goes unchecked: an English Hunspell dictionary in the dictionaries
|
||||
/// folder is used before the word list built into the crate.
|
||||
#[test]
|
||||
fn english_hunspell_dictionary_is_used_before_the_built_in_list() {
|
||||
if scowl_embedded() {
|
||||
return;
|
||||
}
|
||||
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let dir = std::env::temp_dir().join(format!("wc-proof-en-hunspell-{}", std::process::id()));
|
||||
// A tiny en_US. `hous` and `because` aren't words in it, so a marked `because` is this
|
||||
@@ -1224,12 +1132,7 @@ mod tests {
|
||||
let english: &[(&str, &[u8], &[u8])] =
|
||||
&[("en_US", b"SET UTF-8\nTRY esiarntolcdugmphbyfvkwz\nSFX S Y 1\nSFX S 0 s .\n", b"3\nteh\nthe\nhouse/S\n")];
|
||||
folder_with_english(&dir, english);
|
||||
if english_data() {
|
||||
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Embedded));
|
||||
// (The Moby list has `teh`.)
|
||||
assert!(!is_correct_in("becuase", ProofLang::En) && is_correct_in("because", ProofLang::En));
|
||||
assert_eq!(dictionary_load_error(ProofLang::En), None);
|
||||
} else {
|
||||
{
|
||||
let dic = dir.join("en_US.dic");
|
||||
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dic.clone())));
|
||||
for w in ["teh", "Teh", "the", "house", "houses", "House"] {
|
||||
@@ -1258,9 +1161,7 @@ mod tests {
|
||||
assert!(check_text("hous teh", &[(0..8, ProofLang::De)]).is_empty());
|
||||
assert_eq!(english_source(ProofLang::Pl), None);
|
||||
drop_english_files(&dir, english);
|
||||
if !english_data() {
|
||||
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
|
||||
}
|
||||
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
|
||||
}
|
||||
|
||||
/// R6.4: an English Hunspell dictionary that is there but can't be read (an encoding we don't
|
||||
@@ -1269,16 +1170,16 @@ mod tests {
|
||||
/// still marks `becuase`.
|
||||
#[test]
|
||||
fn unreadable_english_dictionary_is_reported_and_skipped() {
|
||||
if scowl_embedded() {
|
||||
return;
|
||||
}
|
||||
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let dir = std::env::temp_dir().join(format!("wc-proof-en-unreadable-{}", std::process::id()));
|
||||
let english: &[(&str, &[u8], &[u8])] =
|
||||
&[("en_US", b"SET UTF-8\n", b"0\n"), ("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"), ("en_CA", b"SET UTF-8\n", b"1\n\xff\xfe\n")];
|
||||
folder_with_english(&dir, english);
|
||||
for (lang, reason) in [(ProofLang::En, "no words"), (ProofLang::EnGb, "unsupported encoding"), (ProofLang::EnCa, "not UTF-8")] {
|
||||
if english_data() {
|
||||
assert_eq!(english_source(lang), Some(EnglishSource::Embedded));
|
||||
assert_eq!(dictionary_load_error(lang), None, "{lang:?}: never read");
|
||||
} else {
|
||||
{
|
||||
// The built-in list, or a dictionary installed on the system (tried next).
|
||||
let source = english_source(lang);
|
||||
assert!(!matches!(&source, Some(EnglishSource::Hunspell(p)) if p.starts_with(&dir)), "{lang:?}: {source:?}");
|
||||
@@ -1298,6 +1199,9 @@ mod tests {
|
||||
/// American English gets a dictionary that loads: the British tests don't hold the lock.)
|
||||
#[test]
|
||||
fn english_tries_its_next_dictionary_and_reads_real_world_directives() {
|
||||
if scowl_embedded() {
|
||||
return;
|
||||
}
|
||||
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let dir = std::env::temp_dir().join(format!("wc-proof-en-next-{}", std::process::id()));
|
||||
let aff: &[u8] = b"SET UTF-8\nICONV 1\nICONV \xe2\x80\x99 '\nOCONV 1\nOCONV ' \xe2\x80\x99\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\n\
|
||||
@@ -1308,7 +1212,7 @@ mod tests {
|
||||
("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"),
|
||||
];
|
||||
folder_with_english(&dir, english);
|
||||
if !english_data() {
|
||||
{
|
||||
let us = Some(EnglishSource::Hunspell(dir.join("en-US.dic")));
|
||||
// A real en_US installed elsewhere (WORDCRAFT_DICTIONARIES, the system) comes first:
|
||||
// the same name in every folder before the next name.
|
||||
@@ -1754,6 +1658,149 @@ mod tests {
|
||||
assert!(check_spelling_in(text, gb).is_empty(), "{:?}", check_spelling_in(text, gb));
|
||||
}
|
||||
|
||||
/// English spelling reads SCOWL 2026 through the affix engine: one dictionary per English,
|
||||
/// its notice embedded, and its ordinals (`COMPOUNDRULE`) understood.
|
||||
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
|
||||
#[test]
|
||||
fn english_reads_scowl() {
|
||||
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
|
||||
let d = scowl::dict(l).unwrap_or_else(|| panic!("{l:?}: {:?}", scowl::load_error(l)));
|
||||
assert!(d.len() > 40_000, "{l:?}: {} stems", d.len());
|
||||
assert_eq!(english_source(l), Some(EnglishSource::Embedded));
|
||||
assert_eq!(dictionary_load_error(l), None);
|
||||
}
|
||||
assert!(std::ptr::eq(scowl::dict(ProofLang::EnGb).unwrap(), scowl::dict(ProofLang::from_tag(Some("en-AU"))).unwrap()), "en-AU reads en-GB");
|
||||
let notice = scowl::notice().unwrap();
|
||||
for credit in ["Kevin Atkinson", "Geoff Kuenning", "clause 4 removed", "Benjamin Titze"] {
|
||||
assert!(notice.contains(credit), "{credit}");
|
||||
}
|
||||
// Ordinals are compounds of digit stems; `1th` is only a part.
|
||||
let strict = Options { ignore_numbers: false, ..Options::default() };
|
||||
for w in ["1st", "21st", "102nd", "33rd", "11th", "112th", "2026th"] {
|
||||
assert!(is_correct_in_with(w, ProofLang::En, &strict), "{w}");
|
||||
}
|
||||
for w in ["1th", "21th", "11st", "3nd"] {
|
||||
assert!(!is_correct_in_with(w, ProofLang::En, &strict), "{w}");
|
||||
}
|
||||
// Affixed forms and possessives come from the affix rules.
|
||||
for w in ["unrecognizable", "reorganization", "children's", "Mississippi's", "can't", "can’t"] {
|
||||
assert!(is_correct_in(w, ProofLang::En), "{w}");
|
||||
}
|
||||
assert!(!is_correct_in("mississippi", ProofLang::En), "a proper noun in lower case");
|
||||
// Suggestions come from the run's own dictionary.
|
||||
assert!(suggest_in("colur", 5, ProofLang::EnGb).contains(&"colour".to_string()), "{:?}", suggest_in("colur", 5, ProofLang::EnGb));
|
||||
assert!(suggest_in("colr", 5, ProofLang::En).contains(&"color".to_string()), "{:?}", suggest_in("colr", 5, ProofLang::En));
|
||||
assert_eq!(suggest_in("organisation", 5, ProofLang::En).first().map(String::as_str), Some("organization"));
|
||||
assert_eq!(suggest_in("color", 5, ProofLang::EnGb).first().map(String::as_str), Some("colour"));
|
||||
}
|
||||
|
||||
/// craft-libs#483 review: English suggestions offer a missing space only between two
|
||||
/// lower-case words of three letters or more, after every one-word candidate, and no word the
|
||||
/// dictionary only has capitalized (`Col`, `Wis`) for a lower-case word.
|
||||
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
|
||||
#[test]
|
||||
fn english_suggestions_skip_junk_splits_and_names() {
|
||||
for (w, l) in [
|
||||
("colour", ProofLang::En),
|
||||
("theatr", ProofLang::En),
|
||||
("publically", ProofLang::En),
|
||||
("cheque", ProofLang::En),
|
||||
("colr", ProofLang::En),
|
||||
("wiss", ProofLang::En),
|
||||
("theatr", ProofLang::EnGb),
|
||||
] {
|
||||
let got = suggest_in(w, 8, l);
|
||||
assert!(!got.is_empty(), "{w}");
|
||||
assert!(got.iter().all(|s| !s.chars().any(char::is_uppercase)), "{w} in {l:?}: {got:?}");
|
||||
let first_split = got.iter().position(|s| s.contains(' ')).unwrap_or(got.len());
|
||||
assert!(got.iter().skip(first_split).all(|s| s.contains(' ')), "{w}: a split before a word: {got:?}");
|
||||
assert!(got.iter().filter(|s| s.contains(' ')).all(|s| s.split(' ').all(|p| p.chars().count() >= 3)), "{w}: {got:?}");
|
||||
}
|
||||
assert_eq!(suggest_in("publically", 5, ProofLang::En).first().map(String::as_str), Some("publicly"));
|
||||
assert_eq!(suggest_in("colour", 5, ProofLang::En).first().map(String::as_str), Some("color"));
|
||||
assert_eq!(suggest_in("theatr", 5, ProofLang::EnGb).first().map(String::as_str), Some("theatre"));
|
||||
assert!(suggest_in("Londn", 5, ProofLang::En).contains(&"London".to_string()), "a capitalized word may get a name");
|
||||
}
|
||||
|
||||
/// `en-variants.txt` stays with SCOWL: SCOWL's size-60 lists keep one spelling per dialect,
|
||||
/// and Word accepts more (both `-ize` and `-ise` in British English, `judgement` in American).
|
||||
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
|
||||
#[test]
|
||||
fn dialect_policy_widens_scowl() {
|
||||
let gb = scowl::dict(ProofLang::EnGb).unwrap();
|
||||
let us = scowl::dict(ProofLang::En).unwrap();
|
||||
assert!(!gb.check("organize") && gb.check("organise") && !us.check("judgement"), "SCOWL alone is stricter");
|
||||
assert!(is_correct_in("organize", ProofLang::EnGb) && is_correct_in("organise", ProofLang::EnGb));
|
||||
assert!(is_correct_in("judgement", ProofLang::En) && !is_correct_in("colour", ProofLang::En));
|
||||
assert!(is_correct_in("encyclopaedia", ProofLang::EnGb) && is_correct_in("encyclopedia", ProofLang::EnGb));
|
||||
}
|
||||
|
||||
/// craft-libs#483 review: each English checks against its own SCOWL dictionary only. A stem
|
||||
/// one English's dictionary has and another's lacks (`colourblind`, `greyscale` in en-GB;
|
||||
/// `colorblind`, `esophagi` in en-US) doesn't cross, unless our dialect policy or the `-ize`
|
||||
/// spellings of British English say so. Sampled from every pair of Englishes.
|
||||
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
|
||||
#[test]
|
||||
fn scowl_dialects_dont_cross() {
|
||||
let langs = [
|
||||
(ProofLang::En, crate::data::EN_US_SPELLING),
|
||||
(ProofLang::EnGb, crate::data::EN_GB_SPELLING),
|
||||
(ProofLang::EnCa, crate::data::EN_CA_SPELLING),
|
||||
];
|
||||
for (from, dic) in langs {
|
||||
for (to, _) in langs {
|
||||
if from == to {
|
||||
continue;
|
||||
}
|
||||
let d = scowl::dict(to).unwrap();
|
||||
let widened = |w: &str| {
|
||||
let x = english::ise_spelling(w);
|
||||
(scowl::is_british(to) && x.as_deref().is_some_and(|x| d.check(x)))
|
||||
|| en_variants::known_variant(w, to, |x| d.check(x))
|
||||
|| known_in_supplement(w)
|
||||
};
|
||||
let only: Vec<&str> = dic
|
||||
.lines()
|
||||
.skip(1)
|
||||
.map(|l| l.split('/').next().unwrap_or_default())
|
||||
.filter(|w| w.len() > 3 && w.chars().all(|c| c.is_ascii_lowercase()) && !d.check(w) && !widened(w))
|
||||
.collect();
|
||||
assert!(only.len() >= 40, "{from:?} → {to:?}: {}", only.len());
|
||||
let step = only.len() / 40;
|
||||
let crossed: Vec<&str> = only.iter().step_by(step.max(1)).copied().filter(|w| is_correct_in(w, to)).collect();
|
||||
assert!(crossed.is_empty(), "{from:?} words right in {to:?}: {crossed:?}");
|
||||
}
|
||||
}
|
||||
for (w, l) in [("colourblind", ProofLang::En), ("greyscale", ProofLang::En), ("colorblind", ProofLang::EnGb), ("esophagi", ProofLang::EnGb)] {
|
||||
assert!(!is_correct_in(w, l), "{w} in {l:?}");
|
||||
}
|
||||
for (w, l) in
|
||||
[("colorblind", ProofLang::En), ("colourblind", ProofLang::EnGb), ("organizational", ProofLang::EnGb), ("realized", ProofLang::EnGb)]
|
||||
{
|
||||
assert!(is_correct_in(w, l), "{w} in {l:?}");
|
||||
}
|
||||
}
|
||||
|
||||
/// craft-libs#483 re-review: British English takes the `-ize` spelling of an `-ise` word only
|
||||
/// when American or Canadian English spells it so; a misspelling whose `-s-` form is a word
|
||||
/// (`wize`, `advize`) stays wrong, as in Hunspell's en-GB.
|
||||
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
|
||||
#[test]
|
||||
fn british_ize_needs_a_real_ize_word() {
|
||||
let wrong = [
|
||||
"wize", "rize", "noize", "dizease", "advize", "rizing", "praize", "raize", "poize", "wizer", "surprize", "arize", "revize", "devize",
|
||||
"mizer", "concize", "precize", "exercize", "chaize",
|
||||
];
|
||||
for l in [ProofLang::EnGb, ProofLang::from_tag(Some("en-AU"))] {
|
||||
for w in wrong.iter().chain(&["colorize"]) {
|
||||
assert!(!is_correct_in(w, l), "{w} in {l:?}");
|
||||
}
|
||||
for w in ["organize", "realization", "colourize", "organizing", "realizes"] {
|
||||
assert!(is_correct_in(w, l), "{w} in {l:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
|
||||
#[test]
|
||||
fn user_dictionary_accepts_the_other_dialects_spelling() {
|
||||
|
||||
@@ -45,6 +45,11 @@ const SAMPLES: [&str; 10] = [
|
||||
#[test]
|
||||
fn nothing_is_embedded() {
|
||||
assert!(crate::data::EN_US_DIC.is_empty());
|
||||
assert!(crate::data::EN_US_SPELLING.is_empty() && crate::data::EN_GB_SPELLING.is_empty() && crate::data::EN_CA_SPELLING.is_empty());
|
||||
assert!(crate::scowl::notice().is_none());
|
||||
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
|
||||
assert!(crate::scowl::dict(l).is_none() && crate::scowl::load_error(l).is_none());
|
||||
}
|
||||
assert!(dict::Dictionary::en_us().is_empty());
|
||||
for p in [patterns::Patterns::en_us(), patterns::Patterns::pl(), patterns::Patterns::pt(), patterns::Patterns::it(), patterns::Patterns::de()] {
|
||||
assert!(p.is_empty());
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
//! English spelling from SCOWL (Spell Checker Oriented Word Lists, Kevin Atkinson): the Hunspell
|
||||
//! dictionaries of craft-i18n's `language-data/en-us`, `en-gb` (British `-ise`) and `en-ca`,
|
||||
//! embedded unmodified at build time and read by the affix engine ([`crate::affix`]) on first use.
|
||||
//!
|
||||
//! Each English has its own: American English reads en-US, the British-spelling Englishes
|
||||
//! (en-GB, en-AU, en-NZ, en-IE…) read en-GB and Canadian English reads en-CA. Its suggestions
|
||||
//! come from its own dictionary, and a word is right when its own dictionary knows it: the other
|
||||
//! dialects' dictionaries are never asked (en-US would take `colourblind` from en-GB's). SCOWL
|
||||
//! picks one spelling per dialect (en-GB `-ise` has no `organize`, en-US no `judgement`), and Word
|
||||
//! accepts more, so British English also takes the `-ize` spellings of its `-ise` words that
|
||||
//! en-US or en-CA spells so, and `en-variants.txt` widens each dialect by its pairs
|
||||
//! (`crate::english`). An English dictionary in the dictionaries folder comes first; a SCOWL
|
||||
//! dictionary that isn't embedded, or fails to load, leaves that English to the next source (`crate::english`).
|
||||
//!
|
||||
//! SCOWL is under its own permissive licence and the affix files under Ispell's BSD licence;
|
||||
//! [`notice`] is the upstream text, for an About window.
|
||||
|
||||
use std::sync::OnceLock;
|
||||
|
||||
use crate::ProofLang;
|
||||
use crate::affix::AffixDict;
|
||||
use crate::data::{EN_CA_AFF, EN_CA_SPELLING, EN_GB_AFF, EN_GB_SPELLING, EN_US_AFF, EN_US_SPELLING, SCOWL_NOTICE};
|
||||
|
||||
/// Per English (American, British, Canadian): the loaded dictionary, or why it isn't there.
|
||||
static DICTS: [OnceLock<Result<AffixDict, String>>; 3] = [OnceLock::new(), OnceLock::new(), OnceLock::new()];
|
||||
|
||||
/// The slot, `.aff` and `.dic` text of `lang`'s dictionary.
|
||||
fn source(lang: ProofLang) -> (usize, &'static str, &'static str) {
|
||||
match lang {
|
||||
ProofLang::EnGb => (1, EN_GB_AFF, EN_GB_SPELLING),
|
||||
ProofLang::EnCa => (2, EN_CA_AFF, EN_CA_SPELLING),
|
||||
_ => (0, EN_US_AFF, EN_US_SPELLING),
|
||||
}
|
||||
}
|
||||
|
||||
/// `lang`'s SCOWL dictionary (an English), when it is embedded and loads.
|
||||
pub(crate) fn dict(lang: ProofLang) -> Option<&'static AffixDict> {
|
||||
let (i, aff, dic) = source(lang);
|
||||
if aff.is_empty() || dic.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let loaded = DICTS.get(i)?.get_or_init(|| {
|
||||
let d = AffixDict::from_hunspell(aff, dic, &["SCOWL (wordlist.aspell.net), embedded from craft-i18n"])
|
||||
.and_then(|d| if d.is_empty() { Err("no words".to_string()) } else { Ok(d) });
|
||||
if let Err(e) = &d {
|
||||
log::error!("embedded SCOWL dictionary for {lang:?}: {e}; English spelling falls back to the next source");
|
||||
}
|
||||
d
|
||||
});
|
||||
loaded.as_ref().ok()
|
||||
}
|
||||
|
||||
/// Does `lang` read the British `-ise` dictionary (en-GB, en-AU, en-NZ…)?
|
||||
pub(crate) fn is_british(lang: ProofLang) -> bool {
|
||||
source(lang).0 == 1
|
||||
}
|
||||
|
||||
/// Why `lang`'s embedded SCOWL dictionary failed to load, once something tried to load it.
|
||||
pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
|
||||
let (i, ..) = source(lang);
|
||||
DICTS.get(i)?.get()?.as_ref().err().map(|e| format!("embedded SCOWL dictionary: {e}"))
|
||||
}
|
||||
|
||||
/// The SCOWL copyright and licence notices (the upstream README, with the Ispell BSD licence of
|
||||
/// the affix files and Benjamin Titze's notice for the Australian data), when SCOWL is embedded.
|
||||
pub fn notice() -> Option<&'static str> {
|
||||
(!SCOWL_NOTICE.is_empty() && !EN_US_SPELLING.is_empty()).then_some(SCOWL_NOTICE)
|
||||
}
|
||||
Reference in new issue
Block a user