daniel543x/craft-libs
1
0 Fork 0
mirror of https://github.com/storytold/craft-libs.git synced 2026-11-08 19:42:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

CP2.2: English spelling reads SCOWL 2026 through the affix engine

Browse Source 
craft-i18n's SCOWL rel-2026.02.25 dictionaries (en-us, en-gb -ise, en-ca)
are embedded by build.rs and read per English: en-US, the British-spelling
Englishes (en-AU, en-NZ, en-IE...) through en-GB, en-CA. Suggestions come from
the run's own dictionary; a word is right when any of the three (or
en-extra.txt) knows it, unless en-variants.txt marks it as the other dialect's.
en-variants.txt stays: SCOWL's size-60 lists keep one spelling per dialect and
Word accepts more (organize and organise in en-GB, judgement in en-US).

English lookup order: dictionaries folder (the run's own English only),
[installed English pack, craft-libs#485], embedded SCOWL, system Hunspell
(own English only; not searched while SCOWL is embedded), the built-in Moby
list. The embedded Moby list is no
longer a spelling source (the built-in list is the same list); it stays for
hyphenation. The dialect policy applies to every source.

The affix engine gains COMPOUNDRULE (one-character flags, * and ?),
ONLYINCOMPOUND and COMPOUNDMIN, which SCOWL uses for ordinals (21st, 112th).
Hostile patterns are capped (64 rules, 32 tokens, 48-byte words) and checked
by a memoised walk. load_english now reads them instead of ignoring them (and
keeps ONLYINCOMPOUND stems as compound parts); only flag-based compounding and
OCONV are still ignored for English.

scowl::notice() exposes the SCOWL/Ispell/Titze notice for About. Embedded
Language data grows by 1,542,396 bytes. CI pins craft-i18n#109's merge commit efb1127.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
This commit is contained in:
Martin BallandClaude Opus 5.5 committed 2026-11-01 21:06:00 +00:00
1 parent 9e5862a1c1
commit e6e948f7d5
10 files changed
+649 -272
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/proof.yml
+2 -1
View File
@@ -25,10 +25,11 @@ jobs:
with:
persist-credentials: false
# The Language data (craft-i18n `language-data/`), embedded by crates/proof/build.rs.
# Pinned to storytold/craft-i18n#109's merge commit (the SCOWL dictionaries, CP2.1).
- uses: actions/checkout@v4
with:
repository: storytold/craft-i18n
ref: 062a6375c322c88b96691a17f7b5fed71c51f7a8
ref: efb1127ac65b238ccfe7df1893f33e91b924b96e
path: craft-i18n
persist-credentials: false
- uses: dtolnay/rust-toolchain@b725eda525362e7cf34af9a7243cac1e3f01a7ec # stable
crates/proof/README.md
+36 -14
View File
@@ -13,7 +13,7 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = "<commit>
| Language | Spelling | Grammar | Hyphenation |
|---|---|---|---|
| English (en-US, en-GB and the British-spelling Englishes, en-CA) | Moby word list + `en-extra.txt`, dialect variants (`en-variants.txt`); without them the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| English (en-US, en-GB and the British-spelling Englishes such as en-AU, en-CA) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns |
| Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise |
| German | — | — | TeX patterns |
@@ -63,20 +63,36 @@ data folder runs neither set.
English spelling uses the first of these that is there; `english_source(lang)` says which.
1. **Embedded:** the Moby word list from `CRAFT_I18N_DIR` (above). Nothing below is used then.
2. **System Hunspell:** the run's own English `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`;
each name in the dictionaries folder and then the system locations before the next name).
Never another English's: an `en_US` dictionary would mark British spellings, so en-GB and
en-CA without their own use the built-in list, which has both spellings. A file that can't be
read is skipped for the next name or folder; one file is read once for every English that uses
it. Compounding (`COMPOUNDMIN`, `COMPOUNDRULE`…, used by English dictionaries only for
ordinals) and `OCONV` are ignored for English rather than refused. None on the web.
3. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
1. **Dictionaries folder:** the English's own `.aff`/`.dic` pair (`en_US`, `en_GB` or `en_CA`)
that the user put in the folder set with `set_dictionaries_folder`, in `WORDCRAFT_DICTIONARIES`
or in WordCraft's own `dictionaries` folder. The user's choice comes first.
2. **Installed English pack:** reserved for Language packs (craft-libs#485, CP4.2).
3. **Embedded SCOWL:** the English's own SCOWL `rel-2026.02.25` Hunspell dictionary from
`CRAFT_I18N_DIR` (`language-data/en-us`, `en-gb` (British `-ise`) or `en-ca`, `spelling.aff` +
`spelling.dic`), read by the affix engine. en-AU, en-NZ, en-IE and the other British-spelling
Englishes read en-GB. Suggestions come from the English's own dictionary; a word is right when
any of the three knows it (or `en-extra.txt` does), unless `en-variants.txt` says it is the other
dialect's spelling, and this dialect's spellings of their words are right too. SCOWL's size-60
lists keep one spelling per dialect, and Word accepts more (`organize` and `organise` in
British English), so `en-variants.txt` stays. With SCOWL embedded the system locations aren't
searched.
4. **System Hunspell:** the English's own pair in the system locations. Never another English's:
an `en_US` dictionary would mark British spellings, so en-GB and en-CA without their own use
the built-in list, which has both spellings. Each name is looked for in every folder before the
next name; a file that can't be read is skipped for the next name or folder; one file is read
once for every English that uses it. `COMPOUNDRULE`, `ONLYINCOMPOUND` and `COMPOUNDMIN`
(ordinals such as `21st`) are read; flag-based compounding (`COMPOUNDFLAG`…) and `OCONV` are
ignored for English rather than refused. None on the web.
5. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
compressed, words only) built into the crate, with `data/en-extra.txt` (WordCraft's supplement
of modern words, a copy of craft-i18n's) and the same inflection rules as the embedded list. A
word it doesn't know is misspelled. Suggestions are the words one edit away (swap, missing,
added or changed letter), looked up in the list, for words of up to 32 letters: swaps first,
then common words, neighbouring keys, the same first letter.
of modern words, a copy of craft-i18n's) and the inflection rules. A word it doesn't know is
misspelled. Suggestions are the words one edit away (swap, missing, added or changed letter),
looked up in the list, for words of up to 32 letters: swaps first, then common words,
neighbouring keys, the same first letter.
The embedded Moby list (`language-data/en-us/en-us.dic`) is no longer a spelling source: it is
the same list as the built-in one, and stays as the hyphenation word list. `en-variants.txt`
applies to every source above when it is embedded.
A dictionary that is there but can't be read (encoding, size, parse error, no words) is skipped,
and `dictionary_load_error(lang)` gives the reason; it never panics. Text in other languages is
@@ -91,6 +107,12 @@ Regenerate with
`data/en-extra.txt` is WordCraft contributors' own list (MIT OR Apache-2.0); keep it in step with
craft-i18n's copy.
The SCOWL notice (SCOWL's licence, the Ispell BSD licence of the affix files and Benjamin Titze's
notice) is embedded with the dictionaries: `craft_proof::scowl::notice()`, for an About window.
Embedding SCOWL adds 1,542,396 bytes of Language data (the en-US, en-GB and en-CA `.aff`/`.dic`
and the notice); the Moby list (432,905 bytes) stays for hyphenation.
## Check
```sh
crates/proof/build.rs
+11
View File
@@ -14,6 +14,10 @@
//!
//! `include_str!` takes a string, so a `CRAFT_I18N_DIR` that isn't valid UTF-8 can't be embedded:
//! it is treated like a folder without the data.
//!
//! English: SCOWL's en-US, en-GB (-ise) and en-CA dictionaries; en-AU is not embedded, as
//! Australian English is checked as British-spelling English (`ProofLang::EnGb`). The SCOWL
//! notice is embedded once (the four upstream READMEs differ only in their title and build line).
use std::fmt::Write as _;
use std::path::{Path, PathBuf};
@@ -21,6 +25,13 @@ use std::path::{Path, PathBuf};
/// (constant, path under `language-data/`, bytes rather than text).
const FILES: &[(&str, &str, bool)] = &[
("EN_US_DIC", "en-us/en-us.dic", true),
("EN_US_AFF", "en-us/spelling.aff", false),
("EN_US_SPELLING", "en-us/spelling.dic", false),
("EN_GB_AFF", "en-gb/spelling.aff", false),
("EN_GB_SPELLING", "en-gb/spelling.dic", false),
("EN_CA_AFF", "en-ca/spelling.aff", false),
("EN_CA_SPELLING", "en-ca/spelling.dic", false),
("SCOWL_NOTICE", "en-us/LICENCE-SCOWL.txt", false),
("EN_US_PAT", "en-us/hyphenation.pat", false),
("EXTRA_EN", "en-us/en-extra.txt", false),
("EN_VARIANTS", "en-us/en-variants.txt", false),
crates/proof/src/affix.rs
+246 -46
View File
@@ -17,11 +17,12 @@
//! (at most two suffixes in a row, as in Hunspell), `FORBIDDENWORD`, `NOSUGGEST`, `KEEPCASE` (the
//! stem is a word only in its own letter case), input conversion (`ICONV`: ligatures such as `fi`
//! read as `fi`) and ignored characters (`IGNORE`), and the suggestion tables `TRY`, `KEY`, `MAP`
//! and `REP`.
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: compounding,
//! long or numeric flags, `NEEDAFFIX` and the other directives that change which words are correct
//! (an English dictionary's compounding and `OCONV` are ignored instead:
//! [`AffixDict::from_hunspell_english`]).
//! and `REP`. Compounds by `COMPOUNDRULE` (patterns of one-character flags with `*` and `?`, as
//! SCOWL's English dictionaries use for ordinals: `21st`, `112th`), with `ONLYINCOMPOUND` and
//! `COMPOUNDMIN`; the parts of such a compound are stems, without affixes, as in Hunspell.
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: flag-based
//! compounding (`COMPOUNDFLAG`, `COMPOUNDBEGIN`…), long or numeric flags, `NEEDAFFIX` and the
//! other directives that change which words are correct.
//!
//! Lookups never panic: stems sit in one string, found through an open-addressing hash table, and
//! a probe allocates only small scratch strings for the candidate roots it tries.
@@ -48,16 +49,54 @@ const UNSUPPORTED: &[&str] = &[
"COMPOUNDFLAG",
"COMPOUNDLAST",
"COMPOUNDMIDDLE",
"COMPOUNDMIN",
"COMPOUNDPERMITFLAG",
"COMPOUNDRULE",
"FULLSTRIP",
"NEEDAFFIX",
"OCONV",
"ONLYINCOMPOUND",
"PSEUDOROOT",
];
/// Most `COMPOUNDRULE` patterns kept, and most tokens in one: a hostile `.aff` can't make a
/// compound check cost more than a few thousand stem lookups per word.
const MAX_COMPOUND_RULES: usize = 64;
const MAX_COMPOUND_TOKENS: usize = 32;
/// Longest word (in bytes) tried as a compound: each try looks up every substring.
const MAX_COMPOUND_BYTES: usize = 48;
/// How often a `COMPOUNDRULE` token's flag may repeat.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Repeat {
Once,
/// `?`
Optional,
/// `*`
Any,
}
/// A `COMPOUNDRULE` pattern as written (`n*1t`): its tokens, each a flag and how often it repeats.
fn parse_compound_rule(pattern: &str) -> Result<Vec<(char, Repeat)>, String> {
let mut out: Vec<(char, Repeat)> = Vec::new();
for c in pattern.chars().filter(|&c| c != EMOJI_SELECTOR) {
match c {
'*' | '?' => {
let Some(last) = out.last_mut().filter(|t| t.1 == Repeat::Once) else {
return Err(format!("COMPOUNDRULE `{pattern}`: `{c}` follows no flag"));
};
last.1 = if c == '*' { Repeat::Any } else { Repeat::Optional };
}
'(' | ')' => return Err(format!("COMPOUNDRULE `{pattern}`: ( ) groups (long flags) are not supported")),
c => out.push((c, Repeat::Once)),
}
if out.len() > MAX_COMPOUND_TOKENS {
return Err(format!("COMPOUNDRULE `{pattern}`: more than {MAX_COMPOUND_TOKENS} flags"));
}
}
if out.is_empty() {
return Err("COMPOUNDRULE with no flags".into());
}
Ok(out)
}
/// Most distinct flags a dictionary may use: every single-byte flag (the Hungarian dictionary
/// uses 158).
const MAX_FLAGS: usize = 256;
@@ -272,6 +311,14 @@ pub struct AffixDict {
ignore: Box<[char]>,
/// Provenance and licence lines, kept through [`AffixDict::to_bytes`].
comments: Vec<String>,
/// `COMPOUNDRULE` patterns: per token, the flag's bit and how often it may repeat.
compound_rules: Vec<Vec<(Mask, Repeat)>>,
/// The same patterns as written, for [`AffixDict::to_bytes`].
compound_text: Vec<String>,
/// The `ONLYINCOMPOUND` flag's bit: such stems are words only as parts of a compound.
only_in_compound: Mask,
/// `COMPOUNDMIN`, in characters.
compound_min: usize,
}
fn hash_str(s: &str) -> u64 {
@@ -341,10 +388,14 @@ struct Rules {
substandard: Option<char>,
/// The `KEEPCASE` flag.
keepcase: Option<char>,
/// The `ONLYINCOMPOUND` flag, read for English only ([`AffixDict::from_hunspell_english`]).
only_in_compound: Option<char>,
iconv: Vec<(String, String)>,
ignore: Vec<char>,
/// `COMPOUNDRULE` patterns, as written (`n*1t`).
compound_rules: Vec<String>,
/// The `ONLYINCOMPOUND` flag.
only_in_compound: Option<char>,
/// `COMPOUNDMIN`: the shortest part of a compound, in characters (Hunspell's default is 3).
compound_min: Option<usize>,
/// `AF` flag aliases: alias `n` (1-based) stands for `aliases[n - 1]`.
aliases: Vec<String>,
/// The `AF` table opened with its count line, so flag fields are alias numbers. A table
@@ -426,26 +477,15 @@ fn zero_empty(s: &str) -> String {
/// but it is not a flag of its own: it is dropped wherever flags are read.
const EMOJI_SELECTOR: char = '\u{fe0f}';
/// Directives an English dictionary may carry that don't change which English words are correct
/// here, ignored by [`AffixDict::from_hunspell_english`]: compounding (English dictionaries use
/// it only for ordinals such as `1st`, which have digits and are skipped by the spelling check)
/// and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
const IGNORED_IN_ENGLISH: &[&str] = &[
"COMPOUNDBEGIN",
"COMPOUNDEND",
"COMPOUNDFIRST",
"COMPOUNDFLAG",
"COMPOUNDLAST",
"COMPOUNDMIDDLE",
"COMPOUNDMIN",
"COMPOUNDPERMITFLAG",
"COMPOUNDRULE",
"OCONV",
];
/// Directives an English dictionary may carry that this engine doesn't implement but that don't
/// change which English words are right, ignored by [`AffixDict::from_hunspell_english`]:
/// flag-based compounding (English dictionaries compound only ordinals, by `COMPOUNDRULE`, which
/// is read) and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
const IGNORED_IN_ENGLISH: &[&str] =
&["COMPOUNDBEGIN", "COMPOUNDEND", "COMPOUNDFIRST", "COMPOUNDFLAG", "COMPOUNDLAST", "COMPOUNDMIDDLE", "COMPOUNDPERMITFLAG", "OCONV"];
/// Read Hunspell `.aff` text (already decoded to UTF-8). `english`: ignore the directives in
/// [`IGNORED_IN_ENGLISH`], and keep the `ONLYINCOMPOUND` flag so the stems that are only parts of
/// compounds can be left out.
/// [`IGNORED_IN_ENGLISH`] instead of refusing them.
fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
let mut r = Rules::default();
let mut cross: HashMap<(bool, char), bool> = HashMap::new();
@@ -453,6 +493,7 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
let mut rep_header = false;
let mut af_header = false;
let mut iconv_header = false;
let mut compound_header = false;
for (n, line) in aff.lines().enumerate() {
let f: Vec<&str> = split_fields(line).collect();
let Some(&dir) = f.first() else { continue };
@@ -487,6 +528,23 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
}
"IGNORE" => r.ignore = at(1).chars().collect(),
"KEEPCASE" => r.keepcase = Some(one_flag(at(1), dir, n)?),
// The first `COMPOUNDRULE` line is the table's count, as in Hunspell.
"COMPOUNDRULE" => {
if !compound_header && at(1).parse::<usize>().is_ok() {
compound_header = true;
} else {
parse_compound_rule(at(1)).map_err(|e| format!("line {}: {e}", n + 1))?;
if r.compound_rules.len() >= MAX_COMPOUND_RULES {
return Err(format!("line {}: more than {MAX_COMPOUND_RULES} COMPOUNDRULE patterns", n + 1));
}
r.compound_rules.push(at(1).to_string());
}
}
"ONLYINCOMPOUND" => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
"COMPOUNDMIN" => {
let min = at(1).parse::<usize>().map_err(|_| format!("line {}: COMPOUNDMIN needs a number", n + 1))?;
r.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES));
}
// Per-character flag modes (`FLAG UTF-8`, as VERO's pt_BR dictionary uses) work
// with the one-character parsing here; only the multi-character encodings differ.
"FLAG" if !matches!(at(1), "UTF-8" | "DEFAULT" | "utf-8" | "default" | "UTF8" | "") => {
@@ -541,7 +599,6 @@ fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
r.prefixes.push(rule);
}
}
"ONLYINCOMPOUND" if english => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
d if english && IGNORED_IN_ENGLISH.contains(&d) => {}
d if UNSUPPORTED.contains(&d) => return Err(format!("line {}: {d} is not supported", n + 1)),
_ => {}
@@ -570,10 +627,10 @@ impl AffixDict {
AffixDict::from_hunspell_in(aff, dic, comments, false)
}
/// [`AffixDict::from_hunspell`] for an English dictionary: compounding and output conversion
/// (`COMPOUNDMIN`, `COMPOUNDRULE`, `OCONV`…), which other languages need but English spelling
/// doesn't, are ignored rather than refused, and the stems marked `ONLYINCOMPOUND` (parts of
/// compounds only) are left out.
/// [`AffixDict::from_hunspell`] for an English dictionary: flag-based compounding and output
/// conversion (`COMPOUNDFLAG`, `OCONV`…), which this engine doesn't implement and English
/// spelling doesn't need, are ignored rather than refused. `COMPOUNDRULE`, `ONLYINCOMPOUND`
/// and `COMPOUNDMIN` (ordinals: `21st`) are read, as for every language.
pub fn from_hunspell_english(aff: &str, dic: &str, comments: &[&str]) -> Result<AffixDict, String> {
AffixDict::from_hunspell_in(aff, dic, comments, true)
}
@@ -619,9 +676,6 @@ impl AffixDict {
continue;
}
let flags = rules.flags(&flags).to_string();
if rules.only_in_compound.is_some_and(|f| flags.contains(f)) {
continue;
}
stems.push((word, flags));
}
AffixDict::build(rules, stems, comments.iter().map(ToString::to_string).collect())
@@ -631,6 +685,7 @@ impl AffixDict {
// Flags: every one a rule or a stem uses gets a bit.
let mut flag_chars: Vec<char> = rules.prefixes.iter().chain(&rules.suffixes).map(|r| r.0).collect();
flag_chars.extend(stems.iter().flat_map(|s| s.1.chars()));
// A compound rule's flag no stem carries gets no bit: its token matches no part.
flag_chars.sort_unstable();
flag_chars.dedup();
if flag_chars.len() > MAX_FLAGS {
@@ -656,6 +711,14 @@ impl AffixDict {
map: rules.map,
rep: rules.rep,
comments,
compound_rules: rules
.compound_rules
.iter()
.map(|p| parse_compound_rule(p).map(|t| t.into_iter().map(|(c, rep)| (bit_of(c), rep)).collect()))
.collect::<Result<_, _>>()?,
compound_text: rules.compound_rules,
only_in_compound: rules.only_in_compound.map_or(Mask::EMPTY, bit_of),
compound_min: rules.compound_min.unwrap_or(3),
..AffixDict::default()
};
d.offs.push(0);
@@ -797,6 +860,11 @@ impl AffixDict {
/// Flag mask of a stem, if `w` is one and carries none of the flags in `deny`.
fn stem_flags_except(&self, w: &str, deny: Mask) -> Option<Mask> {
self.stem_mask(w).filter(|m| !m.meets(self.forbidden | self.only_in_compound | deny))
}
/// Flag mask of the stem `w`, whatever flags it carries.
fn stem_mask(&self, w: &str) -> Option<Mask> {
if self.table.is_empty() {
return None;
}
@@ -810,9 +878,9 @@ impl AffixDict {
if self.word(t as usize) == Some(w) {
let id = *self.flags.get(t as usize)?;
let mask = self.sets.get(usize::from(id)).copied();
// A stem carrying FORBIDDENWORD is listed but is not a word: no direct hit,
// and (this is the only stem lookup) no affixed form is built on it either.
return mask.filter(|m| !m.meets(self.forbidden | deny));
// [`Self::stem_flags_except`] drops FORBIDDENWORD stems (listed, not words) and
// ONLYINCOMPOUND ones (words only inside a compound, [`Self::compound`]).
return mask;
}
slot = (slot + 1) & mask;
}
@@ -928,7 +996,32 @@ impl AffixDict {
return true;
}
}
false
self.compound(w, deny)
}
/// Is `w` a compound of two or more stems (no affixes) whose flags spell out one of the
/// `COMPOUNDRULE` patterns, each part at least `COMPOUNDMIN` characters? `21st` is `2/m` +
/// `1st/p` under `n*mp`. A dynamic programme over (byte offset, token, parts so far), so a
/// hostile pattern costs at most a few thousand lookups.
fn compound(&self, w: &str, deny: Mask) -> bool {
if self.compound_rules.is_empty() || w.len() > MAX_COMPOUND_BYTES {
return false;
}
// Char boundaries of `w`, `0` and `w.len()` included.
let cuts: Vec<usize> = w.char_indices().map(|(i, _)| i).chain(std::iter::once(w.len())).collect();
let n = cuts.len();
// Flag mask of each part `w[cuts[a]..cuts[b]]` that is a stem, by (a, b).
let mut parts: Vec<Option<Mask>> = vec![None; n * n];
for a in 0..n {
for b in a.saturating_add(self.compound_min)..n {
let (Some(&x), Some(&y)) = (cuts.get(a), cuts.get(b)) else { continue };
let mask = w.get(x..y).and_then(|p| self.stem_mask(p)).filter(|m| !m.meets(self.forbidden | deny));
if let Some(slot) = parts.get_mut(a * n + b) {
*slot = mask;
}
}
}
self.compound_rules.iter().any(|rule| compound_matches(rule, &parts, n))
}
/// Is `w` a stem with one suffix? With `need` set (a prefix was taken off), the stem must
@@ -1166,6 +1259,15 @@ impl AffixDict {
if !self.ignore.is_empty() {
out.push_str(&format!("IGNORE {}\n", self.ignore.iter().collect::<String>()));
}
if !self.compound_text.is_empty() {
out.push_str(&format!("COMPOUNDMIN {}\n", self.compound_min));
if let Some(c) = self.flag_char(self.only_in_compound) {
out.push_str(&format!("ONLYINCOMPOUND {c}\n"));
}
for p in &self.compound_text {
out.push_str(&format!("COMPOUNDRULE {p}\n"));
}
}
let z = |s: &str| if s.is_empty() { "0".to_string() } else { s.to_string() };
for (kind, rules) in [("PFX", &self.prefixes), ("SFX", &self.suffixes)] {
for r in rules {
@@ -1233,6 +1335,20 @@ impl AffixDict {
}
"ICONV" if f.len() == 3 => rules.iconv.push((at(1).to_string(), at(2).to_string())),
"IGNORE" => rules.ignore = at(1).chars().collect(),
"COMPOUNDMIN" => {
let min = at(1).parse::<usize>().map_err(|_| format!("affix dictionary: bad `{line}`"))?;
rules.compound_min = Some(min.clamp(1, MAX_COMPOUND_BYTES));
}
"ONLYINCOMPOUND" => {
rules.only_in_compound = Some(one_flag(at(1), "ONLYINCOMPOUND", 0).map_err(|_| format!("affix dictionary: bad `{line}`"))?);
}
"COMPOUNDRULE" => {
parse_compound_rule(at(1)).map_err(|e| format!("affix dictionary: {e}"))?;
if rules.compound_rules.len() >= MAX_COMPOUND_RULES {
return Err("affix dictionary: too many COMPOUNDRULE patterns".into());
}
rules.compound_rules.push(at(1).to_string());
}
"MAP" => rules.map.push(at(1).chars().collect()),
"REP" => rules.rep.push((at(1).replace('_', " "), at(2).replace('_', " "))),
kind @ ("PFX" | "SFX") => {
@@ -1287,6 +1403,40 @@ impl AffixDict {
}
}
/// Does a compound rule's token list match a split of the word into two or more parts? `parts`
/// holds, for each pair of char cuts `(a, b)` at `a * n + b`, the flags of the stem
/// `w[a..b]` (`None`: not a stem). States are (cut, token, parts so far capped at 2), each
/// visited once.
fn compound_matches(rule: &[(Mask, Repeat)], parts: &[Option<Mask>], n: usize) -> bool {
let t = rule.len();
let idx = |cut: usize, tok: usize, k: usize| (cut * (t + 1) + tok) * 3 + k;
let mut seen = vec![false; n * (t + 1) * 3];
let mut stack = vec![(0usize, 0usize, 0usize)];
while let Some((cut, tok, k)) = stack.pop() {
match seen.get_mut(idx(cut, tok, k)) {
Some(s) if !*s => *s = true,
_ => continue,
}
if tok == t {
if cut + 1 == n && k >= 2 {
return true;
}
continue;
}
let Some(&(bit, repeat)) = rule.get(tok) else { continue };
if repeat != Repeat::Once {
stack.push((cut, tok + 1, k));
}
for next in cut + 1..n {
if parts.get(cut * n + next).copied().flatten().is_some_and(|m| m.meets(bit)) {
let k = (k + 1).min(2);
stack.push((next, if repeat == Repeat::Any { tok } else { tok + 1 }, k));
}
}
}
false
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
@@ -1779,6 +1929,55 @@ pub(crate) mod tests {
}
}
/// SCOWL's ordinal rules (its English `.aff`, verbatim): `21st`, `112th` and `1000th` are
/// compounds of digit stems, `1th` is only a part, and nothing else compounds.
const ORDINALS_AFF: &str = "SET UTF-8\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nWORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s .\n";
const ORDINALS_DIC: &str = "13\n0/nm\n0th/pt\n1/n1\n1st/p\n1th/tc\n2/nm\n2nd/p\n2th/tc\n3/nm\n3rd/p\n3th/tc\n4/nm\n4th/pt\ncat/S\n";
#[test]
fn compound_rules_make_ordinals() {
let d = AffixDict::from_hunspell(ORDINALS_AFF, ORDINALS_DIC, &[]).unwrap();
for w in ["1st", "2nd", "4th", "21st", "22nd", "23rd", "11th", "12th", "13th", "111th", "1000th", "104th", "cats"] {
assert!(d.check(w), "{w}");
}
for w in ["1th", "2th", "11st", "12nd", "13rd", "21th", "catcat", "cat1st", "1stcat", "1st1st", "", "x"] {
assert!(!d.check(w), "{w}");
}
// Only-in-compound parts are never suggested on their own.
assert!(!d.can_suggest("1th") && d.forms("1th").is_empty());
// The compact format keeps the rules.
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
assert!(back.check("21st") && back.check("111th") && !back.check("1th") && !back.check("21th"));
}
#[test]
fn compound_rules_need_two_parts_and_compoundmin() {
// `COMPOUNDMIN` defaults to 3; a pattern a lone stem matches still needs two parts.
let aff = "SET UTF-8\nONLYINCOMPOUND c\nCOMPOUNDRULE a*b?\n";
let d = AffixDict::from_hunspell(aff, "3\nfoo/a\nbar/ab\nqux/c\n", &[]).unwrap();
assert!(d.check("foobar") && d.check("foofoo") && d.check("foofoofoobar") && d.check("foo"));
assert!(!d.check("qux") && !d.check("quxfoo") && !d.check("fooqux") && !d.check("foobarx"));
let d = AffixDict::from_hunspell(aff, "2\nab/ab\ncd/a\n", &[]).unwrap();
assert!(!d.check("abcd"), "parts shorter than COMPOUNDMIN 3");
let d = AffixDict::from_hunspell(&format!("{aff}COMPOUNDMIN 2\n"), "2\nab/ab\ncd/a\n", &[]).unwrap();
assert!(d.check("cdab"));
}
#[test]
fn hostile_compound_rules_are_refused_or_cheap() {
for bad in ["COMPOUNDRULE *a", "COMPOUNDRULE (ab)", "COMPOUNDRULE a**", "COMPOUNDMIN x", "ONLYINCOMPOUND ab"] {
assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{bad}\n"), DIC, &[]).is_err(), "{bad}");
}
let many: String = (0..100).map(|_| "COMPOUNDRULE a*\n").collect();
assert!(AffixDict::from_hunspell(&format!("SET UTF-8\n{many}"), DIC, &[]).is_err());
// Every part a stem and a pattern of stars: still answers at once.
let star: String = std::iter::repeat_n("a*", 16).collect();
let d = AffixDict::from_hunspell(&format!("SET UTF-8\nCOMPOUNDMIN 1\nCOMPOUNDRULE {star}b\n"), "1\nx/a\n", &[]).unwrap();
let t = std::time::Instant::now();
assert!(!d.check(&"x".repeat(48)) && !d.check(&"x".repeat(200)));
assert!(t.elapsed().as_secs() < 2);
}
#[test]
fn fields_split_on_spaces_and_tabs_only() {
// A flag byte read as U+00A0 (no-break space) or U+0085 is a flag, not a separator.
@@ -1796,22 +1995,23 @@ pub(crate) mod tests {
}
}
/// craft-libs#482 review: an English dictionary with compounding (for ordinals) and output
/// conversion, as the usual en_US and en_GB ones have, loads for English; its `ONLYINCOMPOUND`
/// stems are left out. Other languages still refuse those directives. (A synthetic `.aff`
/// written for this test.)
/// craft-libs#482 review: an English dictionary with compounding and output conversion, as the
/// usual en_US and en_GB ones have, loads for English: `COMPOUNDRULE` makes its ordinals, its
/// `ONLYINCOMPOUND` stems are words only inside them, and `COMPOUNDFLAG` and `OCONV` are
/// ignored. Other languages still refuse those two. (A synthetic `.aff` written for this test.)
#[test]
fn english_dictionaries_ignore_compounding_and_output_conversion() {
let aff = "SET UTF-8\nTRY esianrtolcdugmphbyfvkwz'\nICONV 1\nICONV ’ '\nOCONV 1\nOCONV ' ’\nNOSUGGEST !\n\
COMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nCOMPOUNDFLAG Z\n\
WORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s [^sxzhy]\nSFX D Y 1\nSFX D 0 ed [^ey]\n";
let dic = "5\nhouse/S\nwalk/SD\n0th/pc\n1st/c\nthe\n";
let dic = "7\nhouse/S\nwalk/SD\n0th/pc\n1st/p\n1th/tc\n1/n1\nthe\n";
assert!(AffixDict::from_hunspell(aff, dic, &[]).is_err(), "refused for other languages");
let d = AffixDict::from_hunspell_english(aff, dic, &[]).unwrap();
for w in ["house", "houses", "walked", "walks", "the"] {
assert!(d.check(w), "{w}");
}
assert!(!d.check("0th") && !d.check("1st"), "parts of compounds only");
assert!(!d.check("0th") && !d.check("1th"), "parts of compounds only");
assert!(d.check("1st") && d.check("11th") && d.check("111th") && !d.check("11st"), "ordinals by COMPOUNDRULE");
assert!(!d.check("housed") && !d.check("hous"));
// The directives that change which words are right are still refused.
for dir in ["NEEDAFFIX", "CIRCUMFIX", "PSEUDOROOT", "FULLSTRIP"] {
crates/proof/src/english.rs
+149 -58
View File
@@ -1,23 +1,31 @@
//! English spelling never goes unchecked. Its words come from the first of these that is there:
//!
//! 1. **Embedded**: the Moby word list embedded from craft-i18n at build time (`CRAFT_I18N_DIR`,
//! see `build.rs`). When it is there, nothing in this module is used.
//! 2. **System Hunspell**: a Hunspell `.aff`/`.dic` pair for the run's own English found by
//! [`crate::hunspell`] (`en_US` for American, `en_GB` for British, `en_CA` for Canadian
//! English; each name in the dictionaries folder and then the system locations before the
//! next name; none on the web). Another English's dictionary is never used: it would mark this
//! English's spellings (`colour` in an `en_US` one), so without its own dictionary an English
//! uses the built-in list, which has both spellings. A file that can't be read is skipped for
//! the next name or folder, and one file is read once for every English that uses it.
//! Directives that only matter for other languages (compounding, output conversion) are
//! ignored ([`crate::hunspell::load_english`]).
//! 3. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
//! 1. **Dictionaries folder**: a Hunspell `.aff`/`.dic` pair for the run's own English that the
//! user keeps for WordCraft ([`crate::hunspell::user_dirs`]: the folder set in the options,
//! `WORDCRAFT_DICTIONARIES`, WordCraft's own `dictionaries` folder).
//! 2. **Installed English pack**: reserved for Language packs (craft-libs#485, CP4.2).
//! 3. **Embedded SCOWL**: the run's English's SCOWL dictionary (en-US, en-GB or en-CA) embedded
//! from craft-i18n at build time ([`crate::scowl`]); a word any of the three knows, or the
//! supplement ([`EXTRA`]), is right. When it is there, the system locations aren't searched.
//! 4. **System Hunspell**: a pair for the run's own English in the system locations (none on the
//! web).
//! 5. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
//! the crate (without its break points), with WordCraft's own supplement of modern words
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled, as with the
//! embedded list.
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled. (The Moby list
//! embedded from craft-i18n is the same list; it is used for hyphenation only.)
//!
//! A Hunspell dictionary that is there but can't be read (unsupported encoding, too large, a
//! parse error, no words) is skipped, and its reason is reported by
//! Hunspell dictionaries are found by [`crate::hunspell`] by their own English's names only
//! (`en_US` for American, `en_GB` for British, `en_CA` for Canadian English), each name in every
//! folder before the next name. Another English's dictionary is never used: it would mark this
//! English's spellings (`colour` in an `en_US` one). A file that can't be read is skipped for the
//! next name or folder, and one file is read once for every English that uses it. `COMPOUNDRULE`
//! (ordinals) is read; flag-based compounding and `OCONV` are ignored
//! ([`crate::hunspell::load_english`]).
//!
//! Whichever it is, the dialect policy of `en-variants.txt` ([`crate::en_variants`], when it is
//! embedded) flags the other dialect's spellings and accepts this dialect's spellings of known
//! words. A Hunspell dictionary that is there but can't be read (unsupported encoding, too large,
//! a parse error, no words) is skipped, and its reason is reported by
//! [`crate::dictionary_load_error`]. Non-English text is never checked against any of these.
use std::path::{Path, PathBuf};
@@ -47,7 +55,7 @@ const MAX_CANDIDATES: usize = 8;
/// Where English spelling comes from now ([`crate::english_source`]).
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum EnglishSource {
/// The Moby word list embedded from craft-i18n at build time.
/// The English's SCOWL dictionary embedded from craft-i18n at build time.
Embedded,
/// An English Hunspell dictionary from the dictionaries folder or the system: its `.dic` file.
Hunspell(PathBuf),
@@ -112,14 +120,9 @@ impl State {
static HUNSPELL: RwLock<State> = RwLock::new(State::new());
/// Why the embedded word list failed to decode, if it did.
static EMBEDDED_ERROR: OnceLock<String> = OnceLock::new();
/// Remember that the embedded word list is there but failed to decode (English then uses the
/// next source).
/// The embedded Moby list (hyphenation) failed to decode: log it. Spelling doesn't use it.
pub(crate) fn note_embedded_error(e: String) {
log::error!("embedded English word list: {e}; English spelling falls back to Hunspell or the built-in word list");
let _ = EMBEDDED_ERROR.set(e);
log::error!("embedded English word list: {e}; English hyphenation falls back to the patterns");
}
/// The English Hunspell dictionary for `lang` (read on first use).
@@ -185,12 +188,14 @@ fn key(lang: ProofLang) -> &'static str {
}
/// Read the first Hunspell pair of `lang`'s own that loads, trying each name in each folder
/// ([`crate::hunspell::candidates`]), with the first failure before it (the file `lang` would
/// rather use), for [`note_error`].
/// ([`crate::hunspell::candidates`]; with SCOWL embedded, the dictionaries folder only:
/// [`crate::hunspell::user_candidates`]), with the first failure before it (the file `lang`
/// would rather use), for [`note_error`].
fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) {
let candidates = crate::hunspell::candidates(names(lang));
let candidates =
if crate::scowl::dict(lang).is_some() { crate::hunspell::user_candidates(names(lang)) } else { crate::hunspell::candidates(names(lang)) };
if candidates.is_empty() {
log::info!("English spelling: no Hunspell {} dictionary installed; using the built-in word list", key(lang));
log::info!("English spelling: no Hunspell {} dictionary to read first; using SCOWL or the built-in word list", key(lang));
return (None, None);
}
let mut first_error: Option<(PathBuf, String)> = None;
@@ -209,7 +214,7 @@ fn load(lang: ProofLang) -> (Loaded, Option<(PathBuf, String)>) {
}
}
}
log::warn!("English spelling: no {} Hunspell dictionary could be read; using the built-in word list", key(lang));
log::warn!("English spelling: no {} Hunspell dictionary could be read; using SCOWL or the built-in word list", key(lang));
(None, first_error)
}
@@ -232,45 +237,95 @@ pub(crate) fn reload() {
}
}
/// Where the spelling of `lang` (an English) comes from now. Reads the Hunspell dictionary once
/// when there is no embedded data (it is cached afterwards).
/// Where the spelling of `lang` (an English) comes from now, in the order of the module docs.
/// Reads the Hunspell dictionary once (it is cached afterwards).
pub(crate) fn source(lang: ProofLang) -> EnglishSource {
if crate::english_data() {
EnglishSource::Embedded
} else {
hunspell(lang).map_or(EnglishSource::CoreList, |(_, dic)| EnglishSource::Hunspell(dic))
match speller(lang) {
Speller::Hunspell(_, dic) => EnglishSource::Hunspell(dic),
Speller::Scowl(_) => EnglishSource::Embedded,
Speller::List => EnglishSource::CoreList,
}
}
/// Why a source of `lang`'s spelling failed to load: the embedded word list, or (without it) the
/// Hunspell dictionary found for it. Reads nothing itself.
/// Why a source of `lang`'s spelling failed to load: its embedded SCOWL dictionary, or the
/// Hunspell dictionary it would rather have used. Reads nothing itself.
pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
if let Some(e) = EMBEDDED_ERROR.get() {
return Some(e.clone());
}
if crate::english_data() {
return None;
}
crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason)
crate::scowl::load_error(lang).or_else(|| crate::hunspell::load_error(key(lang)).map(|(_, reason)| reason))
}
/// Is `word` spelled right in `lang` (an English) when no word list is embedded: against the
/// Hunspell dictionary, or else the built-in word list.
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options) -> bool {
/// What `lang` is checked against now.
enum Speller {
/// A dictionaries-folder or system Hunspell dictionary, and its `.dic` file.
Hunspell(Arc<AffixDict>, PathBuf),
/// The SCOWL dictionaries, `lang`'s first ([`crate::scowl::all`]).
Scowl(Vec<&'static AffixDict>),
/// The built-in word list.
List,
}
fn speller(lang: ProofLang) -> Speller {
pick(hunspell(lang), lang)
}
/// The lookup order: the Hunspell dictionary `found` for `lang` (with SCOWL embedded, only one in
/// the dictionaries folder is looked for), then SCOWL, then the built-in list. (The installed
/// English pack, craft-libs#485, goes between the first two.)
fn pick(found: Loaded, lang: ProofLang) -> Speller {
if let Some((d, dic)) = found {
return Speller::Hunspell(d, dic);
}
let scowl = crate::scowl::all(lang);
if crate::scowl::dict(lang).is_some() && !scowl.is_empty() { Speller::Scowl(scowl) } else { Speller::List }
}
impl Speller {
/// Is `w` (as written, `’` read as `'`) a word: as a whole, or each part of a hyphenated one?
fn knows(&self, w: &str) -> bool {
match self {
Speller::Hunspell(d, _) => affix_correct(&[d.as_ref()], w, false),
Speller::Scowl(ds) => affix_correct(ds, w, true),
Speller::List => list_correct(w),
}
}
/// Is the lower-case base form `p` a word (for the dialect policy's other spelling)?
fn knows_lower(&self, p: &str) -> bool {
match self {
Speller::Hunspell(d, _) => d.check(p),
Speller::Scowl(ds) => ds.iter().any(|d| d.check(p)),
Speller::List => crate::known_in_word_list(p),
}
}
}
/// Is `word` spelled right in `lang` (an English), against its dictionary ([`speller`]); with
/// `policy`, also by the dialect policy of `en-variants.txt`: the other dialect's spellings are
/// wrong and this dialect's spellings of known words right. The user dictionary overrules both.
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options, policy: bool) -> bool {
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’').trim_matches(['\'', '’']);
if crate::accepted_by_options(w, opts) || !w.chars().any(|c| c.is_ascii_alphabetic()) {
return true; // other scripts: not English
}
let w = w.replace('’', "'");
match hunspell(lang) {
Some((d, _)) => hunspell_correct(&d, &w),
None => list_correct(&w),
let lower = w.to_lowercase();
let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect();
if policy && parts.iter().any(|p| crate::en_variants::is_foreign(p, lang)) {
return false;
}
let s = speller(lang);
s.knows(&w)
|| (policy && !parts.is_empty() && parts.iter().all(|p| s.knows_lower(p) || crate::en_variants::known_variant(p, lang, |x| s.knows_lower(x))))
}
/// Against an installed English Hunspell dictionary: the word, or each part of a hyphenated one.
fn hunspell_correct(d: &AffixDict, w: &str) -> bool {
let known = |p: &str| p.chars().count() <= 1 || d.check(p) || crate::in_user_dictionary(&p.to_lowercase());
/// Against affix dictionaries (any of them): the word, or each part of a hyphenated one; with
/// `supplement`, our supplement's words count too.
fn affix_correct(ds: &[&AffixDict], w: &str, supplement: bool) -> bool {
let known = |p: &str| {
p.chars().count() <= 1
|| ds.iter().any(|d| d.check(p))
|| crate::in_user_dictionary(&p.to_lowercase())
|| (supplement && crate::known_in_supplement(&p.to_lowercase()))
};
known(w) || (w.contains('-') && w.split('-').filter(|p| !p.is_empty()).all(known))
}
@@ -285,12 +340,12 @@ fn list_correct(w: &str) -> bool {
crate::known_in_word_list(&lower) || (lower.contains('-') && lower.split('-').filter(|p| !p.is_empty()).all(crate::known_in_word_list))
}
/// Spelling suggestions in `lang` (an English) when no word list is embedded: from the Hunspell
/// dictionary, or else the built-in word list.
/// Spelling suggestions in `lang` (an English), from its dictionary ([`speller`]).
pub(crate) fn suggest(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
match hunspell(lang) {
Some((d, _)) => crate::cs::suggest::suggest_in(&d, word, max),
None => list_suggest(word, max),
match speller(lang) {
Speller::Hunspell(d, _) => crate::cs::suggest::suggest_in(&d, word, max),
Speller::Scowl(ds) => ds.first().map_or_else(Vec::new, |d| crate::cs::suggest::suggest_in(d, word, max)),
Speller::List => list_suggest(word, max),
}
}
@@ -460,6 +515,42 @@ mod tests {
assert!(start.elapsed() < std::time::Duration::from_secs(5), "{:?}", start.elapsed());
}
/// The lookup order (README): a dictionaries-folder English comes before SCOWL, SCOWL before
/// the built-in list; each English looks for its own dictionary only, and with SCOWL embedded
/// the system locations aren't searched. (Checked here without touching the process-wide
/// folder, which every English test running meanwhile would see.)
#[test]
fn lookup_order() {
let tiny = Arc::new(AffixDict::from_hunspell("", "1\nzzyzx\n", &[]).unwrap());
let file = PathBuf::from("/folder/en_US.dic");
for lang in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
assert!(matches!(pick(Some((tiny.clone(), file.clone())), lang), Speller::Hunspell(_, p) if p == file), "{lang:?}");
let fallback = pick(None, lang);
if crate::scowl::dict(lang).is_some() {
assert!(matches!(&fallback, Speller::Scowl(ds) if ds.len() == 3), "{lang:?}");
assert!(fallback.knows("colour") && fallback.knows("color") && fallback.knows("21st") && !fallback.knows("becuase"));
} else {
assert!(matches!(fallback, Speller::List), "{lang:?}");
}
}
let dir = std::env::temp_dir().join(format!("wc-proof-en-order-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
for name in ["en_US", "en_GB", "en_CA"] {
std::fs::write(dir.join(format!("{name}.aff")), "SET UTF-8\n").unwrap();
std::fs::write(dir.join(format!("{name}.dic")), "1\nzzyzx\n").unwrap();
}
let found = |names: &[&str]| -> Vec<String> {
crate::hunspell::candidates_in(std::slice::from_ref(&dir), names)
.iter()
.filter_map(|(_, d)| d.file_stem().map(|n| n.to_string_lossy().into_owned()))
.collect()
};
assert_eq!(found(names(ProofLang::EnGb)), ["en_GB"], "a user's en_US doesn't stand in for SCOWL's en-GB");
assert_eq!(found(names(ProofLang::EnCa)), ["en_CA"]);
assert_eq!(found(names(ProofLang::En)), ["en_US"]);
let _ = std::fs::remove_dir_all(&dir);
}
/// craft-libs#482 review: a load that started before a reload (the dictionaries folder
/// changed) serves its own check but isn't kept, so the next check reads the new folder; a
/// load after it is kept, and shared with the Englishes that read the same file.
crates/proof/src/hunspell.rs
+19
View File
@@ -154,6 +154,25 @@ pub fn candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
candidates_in(&search_dirs(lo.as_deref()), names)
}
/// [`candidates`] in the folders the user keeps dictionaries in for WordCraft ([`user_dirs`])
/// only, not the system locations.
pub fn user_candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
candidates_in(&user_dirs(), names)
}
/// The folders of [`search_dirs`] the user keeps dictionaries in for WordCraft: the one set with
/// [`set_dictionary_dir`], `WORDCRAFT_DICTIONARIES`, and WordCraft's own `dictionaries` folder
/// under the user's application data. The rest are the system's (and LibreOffice's).
pub fn user_dirs() -> Vec<PathBuf> {
let extra = EXTRA_DIR.read().unwrap_or_else(std::sync::PoisonError::into_inner).clone();
let env = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()).map(PathBuf::from);
let wordcraft_own = |d: &Path| {
d.file_name().is_some_and(|n| n == "dictionaries")
&& d.parent().and_then(Path::file_name).is_some_and(|n| n.eq_ignore_ascii_case("wordcraft"))
};
search_dirs(None).into_iter().filter(|d| Some(d) == extra.as_ref() || Some(d) == env.as_ref() || wordcraft_own(d)).collect()
}
/// [`candidates`] in `dirs`.
pub fn candidates_in(dirs: &[PathBuf], names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
let mut out: Vec<(PathBuf, PathBuf)> = Vec::new();
crates/proof/src/lang.rs
+1 -1
View File
@@ -6,7 +6,7 @@
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum ProofLang {
/// American English (`en`, `en-US`, and every region not listed for [`ProofLang::EnGb`] or
/// [`ProofLang::EnCa`]): the public-domain Moby word list, the English grammar rules and the
/// [`ProofLang::EnCa`]): SCOWL's en-US dictionary ([`crate::scowl`]), the English grammar rules and the
/// English hyphenation dictionary and patterns. British spellings (`colour`, `centre`,
/// `organise`) are flagged, with the American form suggested ([`crate::en_variants`]). Also
/// text with no language at all (WordCraft's default editing language is US English, like
crates/proof/src/lib.rs
+105 -152
View File
@@ -1,8 +1,9 @@
//! Proofing for the Craft Apps (moved from WordCraft's `wordcraft-proof`), per language. A run's language (its BCP 47 tag) picks the tools through
//! [`ProofLang`]; the `*_in` functions take it, the older English-only ones remain.
//!
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the
//! public-domain Moby word list (~160k words, the same list that drives hyphenation) plus our
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the SCOWL
//! 2026 Hunspell dictionary of the run's English ([`scowl`]; else the public-domain Moby word
//! list, ~160k words, the same list that drives hyphenation) plus our
//! supplement of modern words it lacks (craft-i18n's `language-data/en-us/en-extra.txt`: `desktop`, `email`,
//! `download`…), with inflections (`-s`, `-es`, `-ed`, `-ing`, `-ly`, `'s`…), a user dictionary and ignore lists;
//! suggestions by edit distance. Grammar checks that are cheap and reliable: repeated words,
@@ -48,11 +49,12 @@
//!
//! [`check_text`] checks a paragraph whose runs are in different languages.
//!
//! **Language data** (the Moby word list, the pattern files, our word supplements) is embedded at
//! build time from craft-i18n's `language-data/` (`CRAFT_I18N_DIR`, see `build.rs`). English
//! spelling never goes unchecked: without the embedded word list it uses an English Hunspell
//! dictionary installed on the system, and without one the whole Moby word list built into the
//! crate (see [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
//! **Language data** (the SCOWL English dictionaries, the Moby word list, the pattern files, our
//! word supplements) is embedded at build time from craft-i18n's `language-data/`
//! (`CRAFT_I18N_DIR`, see `build.rs`). English spelling never goes unchecked: a dictionaries-folder
//! English first, then the embedded SCOWL dictionary, then an English Hunspell dictionary
//! installed on the system, then the whole Moby word list built into the crate (see
//! [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
//! rule-based hyphenation (Bulgarian, Turkish, Russian) are code and still work.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -78,6 +80,7 @@ pub mod patterns;
pub mod pl;
pub mod pt;
pub mod ru;
pub mod scowl;
mod slot;
mod spelling;
pub mod tr;
@@ -315,7 +318,7 @@ pub fn installed_dictionaries() -> Vec<(ProofLang, Option<std::path::PathBuf>)>
/// Why the dictionary found for `lang` failed to load (unsupported encoding, too large, a parse
/// error), if it was read and failed. Doesn't read anything itself: a dictionary nobody has
/// checked yet reports `None` (#2830). For an English, the embedded word list or (without it) the
/// checked yet reports `None` (#2830). For an English, its embedded SCOWL dictionary or the
/// English Hunspell dictionary that failed; English then uses the next source ([`english_source`]).
pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
if matches!(lang, ProofLang::En | ProofLang::EnGb | ProofLang::EnCa) {
@@ -324,12 +327,11 @@ pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
hunspell::load_error(language(lang)?.slot.name()).map(|(_, reason)| reason)
}
/// Where the spelling of an English `lang` comes from now (`None` for other languages): the word
/// list embedded at build time, else an English Hunspell dictionary from the dictionaries folder
/// or the system (the run's English first), else the Moby word list built into the crate.
/// Each English uses only its own Hunspell dictionary (`en_US`, `en_GB`, `en_CA`): without it,
/// the built-in list, which has both spellings. Reads the Hunspell dictionary once when there is no
/// embedded word list (it is cached afterwards).
/// Where the spelling of an English `lang` comes from now (`None` for other languages), in this
/// order: a Hunspell dictionary of its own (`en_US`, `en_GB`, `en_CA`) in the dictionaries
/// folder, the SCOWL dictionary embedded at build time, one of its own on the system, else the
/// Moby word list built into the crate. Another English's dictionary is never used. Reads the
/// Hunspell dictionary once (it is cached afterwards).
pub fn english_source(lang: ProofLang) -> Option<EnglishSource> {
lang.is_english().then(|| english::source(lang))
}
@@ -467,7 +469,7 @@ pub(crate) fn is_common_word(w: &str) -> bool {
}
/// Our supplement of common words the Moby list lacks (`desktop`, `email`, `download`…): one
/// lowercase base form per line, `#` comments. Inflections come from [`known_core`].
/// lowercase base form per line, `#` comments. Inflections come from [`known_with`].
use data::EXTRA_EN;
/// The supplement's words: the Language data's copy, else the crate's own ([`english::EXTRA`]).
@@ -477,21 +479,18 @@ fn extra_words() -> &'static HashSet<&'static str> {
E.get_or_init(|| extra.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect())
}
/// A base form in the dictionary or the supplement.
fn base_known(d: &dict::Dictionary, w: &str) -> bool {
d.get(w).is_some() || extra_words().contains(w)
}
fn known_core(w: &str) -> bool {
let d = dict::Dictionary::en_us();
known_with(w, &|s| base_known(d, s))
}
/// [`known_core`] against the word list built into the crate (no embedded word list).
/// Is `w` (lower case) a word of the built-in Moby list or the supplement, with the inflection
/// rules ([`known_with`])?
pub(crate) fn known_in_word_list(w: &str) -> bool {
known_with(w, &|s| english::word_list_contains(s) || extra_words().contains(s))
}
/// Is `w` (lower case) a word of the supplement alone, with the inflection rules (the SCOWL
/// dictionaries take it as an addition)?
pub(crate) fn known_in_supplement(w: &str) -> bool {
known_with(w, &|s| extra_words().contains(s))
}
/// Is `w` (lower case) a word, by `base_known` (a base form in the word list), the common words,
/// the user dictionary and the inflection and prefix rules?
fn known_with(w: &str, base_known: &dyn Fn(&str) -> bool) -> bool {
@@ -615,66 +614,15 @@ pub(crate) fn accepted_by_options(word: &str, opts: &Options) -> bool {
|| in_user_dictionary(&w.to_lowercase())
}
/// Is the English word list embedded? Without it English is checked against an installed
/// Hunspell dictionary or the word list built into the crate ([`english`]).
pub(crate) fn english_data() -> bool {
!dict::Dictionary::en_us().is_empty()
}
/// [`is_correct`] with the File › Options › Proofing choices.
/// [`is_correct`] with the File › Options › Proofing choices: against American English's
/// dictionary ([`english_source`]), without the dialect policy.
pub fn is_correct_with(word: &str, opts: &Options) -> bool {
if !english_data() {
return english::is_correct(word, ProofLang::En, opts);
}
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'');
let w = w.trim_matches('\'');
if accepted_by_options(w, opts) {
return true;
}
if !w.is_ascii() && !w.chars().any(|c| c.is_ascii_alphabetic()) {
return true; // other scripts: no dictionary
}
let lower = w.to_lowercase().replace('’', "'");
if known_core(&lower) {
return true;
}
// Hyphenated compounds: each part.
if lower.contains('-') {
return lower.split('-').all(|p| p.is_empty() || known_core(p));
}
false
english::is_correct(word, ProofLang::En, opts, false)
}
/// Spelling suggestions for a misspelled word (best first, at most `max`).
pub fn suggest(word: &str, max: usize) -> Vec<String> {
if !english_data() {
return english::suggest(word, max, ProofLang::En);
}
suggest_from(word, max, dict::Dictionary::en_us().iter().map(|(w, _)| w))
}
/// [`suggest`] from `words` (lower case) and our supplement.
pub(crate) fn suggest_from<'a>(word: &str, max: usize, words: impl Iterator<Item = &'a str>) -> Vec<String> {
let lower = word.to_lowercase();
let n = lower.chars().count();
if n == 0 || n > 40 {
return Vec::new();
}
let mut scored: Vec<(usize, i64, String)> = Vec::new();
let target: Vec<char> = lower.chars().collect();
for w in words.chain(extra_words().iter().copied()) {
let wn = w.chars().count();
if wn + 2 < n || wn > n + 2 {
continue;
}
let dist = damerau(&target, w);
if dist <= 2 {
let same_first = w.chars().next() == target.first().copied();
scored.push((dist, if same_first { 0 } else { 1 }, w.to_string()));
}
}
scored.sort_by(|a, b| (a.0, a.1, a.2.len().abs_diff(n)).cmp(&(b.0, b.1, b.2.len().abs_diff(n))).then(a.2.cmp(&b.2)));
scored.into_iter().take(max).map(|(_, _, w)| match_case(word, w)).collect()
english::suggest(word, max, ProofLang::En)
}
/// `w` (lower case) in the case of `word`: all capitals, an initial capital, or as it is.
@@ -709,26 +657,11 @@ fn english_preferred(lower: &str, lang: ProofLang) -> Option<String> {
}
/// Is `word` spelled right in the English `lang` ([`ProofLang::En`], [`ProofLang::EnGb`],
/// [`ProofLang::EnCa`])? The shared word list's words, less the other dialect's spellings
/// [`ProofLang::EnCa`])? Its dictionary's words ([`english`]), less the other dialect's spellings
/// (`colour` in American English, `color` in British), plus this dialect's spellings of the
/// list's words ([`en_variants`]). The user dictionary overrules both.
/// dictionary's words ([`en_variants`]). The user dictionary overrules both.
fn is_correct_english(word: &str, lang: ProofLang, opts: &Options) -> bool {
if !english_data() {
return english::is_correct(word, lang, opts);
}
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'').trim_matches('\'');
if accepted_by_options(w, opts) {
return true;
}
let lower = w.to_lowercase().replace('’', "'");
if in_user_dictionary(&lower) {
return true;
}
let parts: Vec<&str> = lower.split('-').filter(|p| !p.is_empty()).collect();
if parts.iter().any(|p| en_variants::is_foreign(p, lang)) {
return false;
}
is_correct_with(word, opts) || (!parts.is_empty() && parts.iter().all(|p| known_core(p) || en_variants::known_variant(p, lang, known_core)))
english::is_correct(word, lang, opts, true)
}
/// Spelling suggestions in the English `lang`: this dialect's spelling first when `word` is the
@@ -739,7 +672,7 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
if let Some(p) = english_preferred(&lower, lang) {
out.push(match_case(word, p));
}
let shared = if english_data() { suggest(word, max.saturating_add(2)) } else { english::suggest(word, max.saturating_add(2), lang) };
let shared = english::suggest(word, max.saturating_add(2), lang);
for s in shared {
let s = english_preferred(&s.to_lowercase(), lang).map_or(s, |p| match_case(word, p));
if !out.contains(&s) {
@@ -750,41 +683,6 @@ fn suggest_english(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
out
}
/// Optimal string alignment distance, early-exit at 3.
pub(crate) fn damerau(a: &[char], b: &str) -> usize {
let b: Vec<char> = b.chars().collect();
let (n, m) = (a.len(), b.len());
let mut prev2 = vec![0usize; m + 1];
let mut prev: Vec<usize> = (0..=m).collect();
let mut cur = vec![0usize; m + 1];
for i in 1..=n {
if let Some(c) = cur.get_mut(0) {
*c = i;
}
let mut row_min = i;
for j in 1..=m {
let (ai, bj) = (a.get(i - 1), b.get(j - 1));
let cost = usize::from(ai != bj);
let mut v = (prev.get(j).copied().unwrap_or(9) + 1)
.min(cur.get(j - 1).copied().unwrap_or(9) + 1)
.min(prev.get(j - 1).copied().unwrap_or(9) + cost);
if i > 1 && j > 1 && a.get(i - 1) == b.get(j - 2) && a.get(i - 2) == b.get(j - 1) {
v = v.min(prev2.get(j - 2).copied().unwrap_or(9) + 1);
}
if let Some(c) = cur.get_mut(j) {
*c = v;
}
row_min = row_min.min(v);
}
if row_min > 2 {
return 3;
}
std::mem::swap(&mut prev2, &mut prev);
std::mem::swap(&mut prev, &mut cur);
}
prev.get(m).copied().unwrap_or(9)
}
/// Words in `text`: (byte start, byte end). Combining marks belong to the word they follow, so a
/// word typed decomposed (`tambie\u{301}n`, NFD) is one word, as its precomposed form is.
pub fn words(text: &str) -> Vec<(usize, usize)> {
@@ -1212,11 +1110,20 @@ mod tests {
set_dictionaries_folder(Some(dir.to_path_buf()));
}
/// R6.4, English never goes unchecked: with no embedded word list, an English Hunspell
/// dictionary (here in the dictionaries folder) is used before the word list built into the
/// crate; with one, the embedded list stays in use whatever is installed.
/// Language-data builds embed SCOWL: an English dictionary in the dictionaries folder would
/// then come before it for every English test running meanwhile, so the folder tests below
/// run without the data only; `english::tests` checks the order with the data.
fn scowl_embedded() -> bool {
scowl::dict(ProofLang::En).is_some()
}
/// R6.4, English never goes unchecked: an English Hunspell dictionary in the dictionaries
/// folder is used before the word list built into the crate.
#[test]
fn english_hunspell_dictionary_is_used_before_the_built_in_list() {
if scowl_embedded() {
return;
}
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-hunspell-{}", std::process::id()));
// A tiny en_US. `hous` and `because` aren't words in it, so a marked `because` is this
@@ -1224,12 +1131,7 @@ mod tests {
let english: &[(&str, &[u8], &[u8])] =
&[("en_US", b"SET UTF-8\nTRY esiarntolcdugmphbyfvkwz\nSFX S Y 1\nSFX S 0 s .\n", b"3\nteh\nthe\nhouse/S\n")];
folder_with_english(&dir, english);
if english_data() {
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Embedded));
// (The Moby list has `teh`.)
assert!(!is_correct_in("becuase", ProofLang::En) && is_correct_in("because", ProofLang::En));
assert_eq!(dictionary_load_error(ProofLang::En), None);
} else {
{
let dic = dir.join("en_US.dic");
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dic.clone())));
for w in ["teh", "Teh", "the", "house", "houses", "House"] {
@@ -1258,9 +1160,7 @@ mod tests {
assert!(check_text("hous teh", &[(0..8, ProofLang::De)]).is_empty());
assert_eq!(english_source(ProofLang::Pl), None);
drop_english_files(&dir, english);
if !english_data() {
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
}
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
}
/// R6.4: an English Hunspell dictionary that is there but can't be read (an encoding we don't
@@ -1269,16 +1169,16 @@ mod tests {
/// still marks `becuase`.
#[test]
fn unreadable_english_dictionary_is_reported_and_skipped() {
if scowl_embedded() {
return;
}
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-unreadable-{}", std::process::id()));
let english: &[(&str, &[u8], &[u8])] =
&[("en_US", b"SET UTF-8\n", b"0\n"), ("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"), ("en_CA", b"SET UTF-8\n", b"1\n\xff\xfe\n")];
folder_with_english(&dir, english);
for (lang, reason) in [(ProofLang::En, "no words"), (ProofLang::EnGb, "unsupported encoding"), (ProofLang::EnCa, "not UTF-8")] {
if english_data() {
assert_eq!(english_source(lang), Some(EnglishSource::Embedded));
assert_eq!(dictionary_load_error(lang), None, "{lang:?}: never read");
} else {
{
// The built-in list, or a dictionary installed on the system (tried next).
let source = english_source(lang);
assert!(!matches!(&source, Some(EnglishSource::Hunspell(p)) if p.starts_with(&dir)), "{lang:?}: {source:?}");
@@ -1298,6 +1198,9 @@ mod tests {
/// American English gets a dictionary that loads: the British tests don't hold the lock.)
#[test]
fn english_tries_its_next_dictionary_and_reads_real_world_directives() {
if scowl_embedded() {
return;
}
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-next-{}", std::process::id()));
let aff: &[u8] = b"SET UTF-8\nICONV 1\nICONV \xe2\x80\x99 '\nOCONV 1\nOCONV ' \xe2\x80\x99\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\n\
@@ -1308,7 +1211,7 @@ mod tests {
("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"),
];
folder_with_english(&dir, english);
if !english_data() {
{
let us = Some(EnglishSource::Hunspell(dir.join("en-US.dic")));
// A real en_US installed elsewhere (WORDCRAFT_DICTIONARIES, the system) comes first:
// the same name in every folder before the next name.
@@ -1754,6 +1657,56 @@ mod tests {
assert!(check_spelling_in(text, gb).is_empty(), "{:?}", check_spelling_in(text, gb));
}
/// English spelling reads SCOWL 2026 through the affix engine: one dictionary per English,
/// its notice embedded, and its ordinals (`COMPOUNDRULE`) understood.
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn english_reads_scowl() {
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
let d = scowl::dict(l).unwrap_or_else(|| panic!("{l:?}: {:?}", scowl::load_error(l)));
assert!(d.len() > 40_000, "{l:?}: {} stems", d.len());
assert_eq!(english_source(l), Some(EnglishSource::Embedded));
assert_eq!(dictionary_load_error(l), None);
assert_eq!(scowl::all(l).len(), 3);
}
assert!(std::ptr::eq(scowl::dict(ProofLang::EnGb).unwrap(), scowl::dict(ProofLang::from_tag(Some("en-AU"))).unwrap()), "en-AU reads en-GB");
let notice = scowl::notice().unwrap();
for credit in ["Kevin Atkinson", "Geoff Kuenning", "clause 4 removed", "Benjamin Titze"] {
assert!(notice.contains(credit), "{credit}");
}
// Ordinals are compounds of digit stems; `1th` is only a part.
let strict = Options { ignore_numbers: false, ..Options::default() };
for w in ["1st", "21st", "102nd", "33rd", "11th", "112th", "2026th"] {
assert!(is_correct_in_with(w, ProofLang::En, &strict), "{w}");
}
for w in ["1th", "21th", "11st", "3nd"] {
assert!(!is_correct_in_with(w, ProofLang::En, &strict), "{w}");
}
// Affixed forms and possessives come from the affix rules.
for w in ["unrecognizable", "reorganization", "children's", "Mississippi's", "can't", "can’t"] {
assert!(is_correct_in(w, ProofLang::En), "{w}");
}
assert!(!is_correct_in("mississippi", ProofLang::En), "a proper noun in lower case");
// Suggestions come from the run's own dictionary.
assert!(suggest_in("colur", 5, ProofLang::EnGb).contains(&"colour".to_string()), "{:?}", suggest_in("colur", 5, ProofLang::EnGb));
assert!(suggest_in("colr", 5, ProofLang::En).contains(&"color".to_string()), "{:?}", suggest_in("colr", 5, ProofLang::En));
assert_eq!(suggest_in("organisation", 5, ProofLang::En).first().map(String::as_str), Some("organization"));
assert_eq!(suggest_in("color", 5, ProofLang::EnGb).first().map(String::as_str), Some("colour"));
}
/// `en-variants.txt` stays with SCOWL: SCOWL's size-60 lists keep one spelling per dialect,
/// and Word accepts more (both `-ize` and `-ise` in British English, `judgement` in American).
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn dialect_policy_widens_scowl() {
let gb = scowl::dict(ProofLang::EnGb).unwrap();
let us = scowl::dict(ProofLang::En).unwrap();
assert!(!gb.check("organize") && gb.check("organise") && !us.check("judgement"), "SCOWL alone is stricter");
assert!(is_correct_in("organize", ProofLang::EnGb) && is_correct_in("organise", ProofLang::EnGb));
assert!(is_correct_in("judgement", ProofLang::En) && !is_correct_in("colour", ProofLang::En));
assert!(is_correct_in("encyclopaedia", ProofLang::EnGb) && is_correct_in("encyclopedia", ProofLang::EnGb));
}
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn user_dictionary_accepts_the_other_dialects_spelling() {
crates/proof/src/no_data.rs
+5
View File
@@ -45,6 +45,11 @@ const SAMPLES: [&str; 10] = [
#[test]
fn nothing_is_embedded() {
assert!(crate::data::EN_US_DIC.is_empty());
assert!(crate::data::EN_US_SPELLING.is_empty() && crate::data::EN_GB_SPELLING.is_empty() && crate::data::EN_CA_SPELLING.is_empty());
assert!(crate::scowl::notice().is_none());
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
assert!(crate::scowl::dict(l).is_none() && crate::scowl::all(l).is_empty() && crate::scowl::load_error(l).is_none());
}
assert!(dict::Dictionary::en_us().is_empty());
for p in [patterns::Patterns::en_us(), patterns::Patterns::pl(), patterns::Patterns::pt(), patterns::Patterns::it(), patterns::Patterns::de()] {
assert!(p.is_empty());
crates/proof/src/scowl.rs
+75
View File
@@ -0,0 +1,75 @@
//! English spelling from SCOWL (Spell Checker Oriented Word Lists, Kevin Atkinson): the Hunspell
//! dictionaries of craft-i18n's `language-data/en-us`, `en-gb` (British `-ise`) and `en-ca`,
//! embedded unmodified at build time and read by the affix engine ([`crate::affix`]) on first use.
//!
//! Each English has its own: American English reads en-US, the British-spelling Englishes
//! (en-GB, en-AU, en-NZ, en-IE…) read en-GB and Canadian English reads en-CA. Its suggestions
//! come from its own dictionary. A word is right when any of the three knows it, unless
//! `en-variants.txt` says it is the other dialect's spelling: SCOWL picks one spelling per
//! dialect (en-GB `-ise` has no `organize`, en-US no `judgement`), and Word accepts more. An
//! English dictionary in the dictionaries folder comes first; a SCOWL dictionary that isn't
//! embedded, or fails to load, leaves that English to the next source (`crate::english`).
//!
//! SCOWL is under its own permissive licence and the affix files under Ispell's BSD licence;
//! [`notice`] is the upstream text, for an About window.
use std::sync::OnceLock;
use crate::ProofLang;
use crate::affix::AffixDict;
use crate::data::{EN_CA_AFF, EN_CA_SPELLING, EN_GB_AFF, EN_GB_SPELLING, EN_US_AFF, EN_US_SPELLING, SCOWL_NOTICE};
/// Per English (American, British, Canadian): the loaded dictionary, or why it isn't there.
static DICTS: [OnceLock<Result<AffixDict, String>>; 3] = [OnceLock::new(), OnceLock::new(), OnceLock::new()];
/// The slot, `.aff` and `.dic` text of `lang`'s dictionary.
fn source(lang: ProofLang) -> (usize, &'static str, &'static str) {
match lang {
ProofLang::EnGb => (1, EN_GB_AFF, EN_GB_SPELLING),
ProofLang::EnCa => (2, EN_CA_AFF, EN_CA_SPELLING),
_ => (0, EN_US_AFF, EN_US_SPELLING),
}
}
/// `lang`'s SCOWL dictionary (an English), when it is embedded and loads.
pub(crate) fn dict(lang: ProofLang) -> Option<&'static AffixDict> {
let (i, aff, dic) = source(lang);
if aff.is_empty() || dic.is_empty() {
return None;
}
let loaded = DICTS.get(i)?.get_or_init(|| {
let d = AffixDict::from_hunspell(aff, dic, &["SCOWL (wordlist.aspell.net), embedded from craft-i18n"])
.and_then(|d| if d.is_empty() { Err("no words".to_string()) } else { Ok(d) });
if let Err(e) = &d {
log::error!("embedded SCOWL dictionary for {lang:?}: {e}; English spelling falls back to the next source");
}
d
});
loaded.as_ref().ok()
}
/// Every embedded SCOWL dictionary that loads, `lang`'s first: the shared English word list the
/// dialect policy (`en-variants.txt`) is applied to.
pub(crate) fn all(lang: ProofLang) -> Vec<&'static AffixDict> {
let mut out: Vec<&'static AffixDict> = dict(lang).into_iter().collect();
for other in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
if source(other).0 != source(lang).0
&& let Some(d) = dict(other)
{
out.push(d);
}
}
out
}
/// Why `lang`'s embedded SCOWL dictionary failed to load, once something tried to load it.
pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
let (i, ..) = source(lang);
DICTS.get(i)?.get()?.as_ref().err().map(|e| format!("embedded SCOWL dictionary: {e}"))
}
/// The SCOWL copyright and licence notices (the upstream README, with the Ispell BSD licence of
/// the affix files and Benjamin Titze's notice for the Australian data), when SCOWL is embedded.
pub fn notice() -> Option<&'static str> {
(!SCOWL_NOTICE.is_empty() && !EN_US_SPELLING.is_empty()).then_some(SCOWL_NOTICE)
}