mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-08 19:42:12 +00:00
Portuguese proofing: pt-BR spelling, suggestions, grammar and hyphenation (storytold/wordcraft#100) (storytold/wordcraft#797)
* M15: Polish interface, and Polish spelling, grammar and hyphenation
Interface
- Polish is the tenth interface language ("Polski", pl.tsv, 1,045 entries);
pl, pl-PL and pl_PL.UTF-8 system locales select it automatically.
- Right-click menu labels now go through translation in every language.
- Review > Language is a menu (English US/UK, Polski, Do not check spelling
or grammar); the status bar shows the language at the caret.
- With a Polish interface, new blank documents and opened .txt/.md files
default to pl-PL.
Proofing
- Each run is checked in its own language (w:lang, resolved through
styles): squiggles, F7, review.issues, suggestions and hyphenation.
Text in languages without proofing tools is no longer flagged as English.
- Polish spelling from the SJP.PL Hunspell dictionary (Apache-2.0 option),
read by a new Rust affix engine (crates/proof/src/affix.rs); 875 KB asset,
~4.5M word forms. Regenerate with the pl_dictionary example.
- Polish suggestions: case, missing diacritics, rz/ż, ó/u, ch/h confusions,
missing spaces (napewno -> na pewno), edit distance.
- Conservative Polish grammar rules: repeated words, spacing, doubled
punctuation, sentence capitals, missing comma before że/który/ponieważ/
aby/żeby/jeśli…, we/ze forms.
- Polish hyphenation with the hyph-utf8 hyph-pl.tex patterns (MIT option);
pattern keys widened to 20 letters, TeX exceptions parsed; the heuristic
no longer treats ł, ż, ś… as vowels.
review.language validates and normalizes tags, can set noProof alone and
the document default; format.state reports lang and noProof.
Tests for every part, docs and ATTRIBUTION.md/NOTICE updated;
cargo xtask ci passes (fmt, clippy, test, assets, layers, wasm).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Layout: rustfmt the test module order left unformatted by storytold/wordcraft#391
cargo fmt --check failed on main after storytold/wordcraft#391 added mod tests_grid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Review fixes for storytold/wordcraft#346: read Polish spelling from the installed Hunspell dictionary
- Polish spelling uses the Hunspell pl_PL dictionary installed on the
system (OS packages, Homebrew, LibreOffice, ~/.local/share/hunspell, or
WORDCRAFT_DICTIONARIES) instead of a bundled 870 KB SJP.PL file;
without one, Polish text gets no spelling marks, like Word without the
language's proofing tools. .aff/.dic are size-capped and decoded from
UTF-8 or ISO 8859-2 (crates/proof/src/hunspell.rs).
- The dictionary can be replaced at runtime (install_dictionary) or
re-searched (reload_dictionary); an empty one accepts every word.
- Tests use small synthetic dictionaries written for them, so they don't
depend on what the machine has installed.
- The bundled pl.dic, its ATTRIBUTION/NOTICE entries and the generator
example are removed; docs describe the installed dictionary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* storytold/wordcraft#100: affix — accept FLAG UTF-8 and DEFAULT flag modes
Per-character flag modes are what the reader already parses; only the
multi-character encodings (long, num) stay errors. VERO pt_BR declares
FLAG UTF-8.
* storytold/wordcraft#100: affix — widen flag masks to u128 (128 flag capacity)
VERO pt_BR needs 106 distinct flags; the u64 bitmask capped at 64.
Serialization is unaffected: WCAFF1 stores flag characters and
re-derives the masks on load.
* storytold/wordcraft#100: affix — support FORBIDDENWORD
VERO pt_BR marks spellings it must reject with a forbidden flag (ý).
Stems carrying it are listed but never correct and never inflect.
* storytold/wordcraft#100: affix — continuation classes in check
SFX A 0 able/B: the form derived by A carries flag B, licensing a
further suffix of class B (workable -> workables) while the bare stem
cannot take one. form_has_flag walks the chain depth-bounded
(MAX_CHAIN = 3); with no continuations declared it degenerates to the
stem lookup, so existing dictionaries are unaffected. VERO pt_BR uses
26 continuation rules.
* storytold/wordcraft#100: affix — continuation classes in forms
Enumerate chained suffix forms (workable -> workables) under the same
MAX_CHAIN bound the checker walks, so every form the checker accepts
is listed and vice versa.
* storytold/wordcraft#100: affix — round-trip continuations and FORBIDDENWORD through WCAFF1
The compact format keeps its WCAFF1 magic: continuation classes ride
on the rule's appended field (as in the .aff source) and the forbidden
flag becomes a FORBIDDENWORD line. Old files read as before.
* storytold/wordcraft#100: affix — strip a UTF-8 BOM from .aff/.dic input
VERO's files open with a BOM; glued to the count line it parsed as a
stem (one garbage entry per load) and to SET it hid the directive.
* storytold/wordcraft#782: affix — the module doc still rejected what the code now reads
Continuation classes and FORBIDDENWORD are supported; the header said
Rejected for both.
* storytold/wordcraft#100: pt — Portuguese spelling, following the Polish module
ProofLang::Pt resolves pt/pt-BR/pt-PT/por tags; the pt module reads the
installed Hunspell pt_BR dictionary (VERO) through the affix engine —
FLAG UTF-8 flags, continuation classes and FORBIDDENWORD spellings all
exercised by the synthetic test dictionary. Hyphenated compounds are
checked part by part (guarda-roupa, ex-marido); without a dictionary
installed, Portuguese text gets no marks. Spelling message in
Portuguese; suggestions and grammar land next.
* storytold/wordcraft#100: pt — Portuguese suggestions
Rounds of rising cost, driven by the dictionary's own tables: missing
accents from MAP (coraçao -> coração, saude -> saúde), the REP
confusions (seje -> seja, excessão -> exceção), a missing space after a
short function word (nãoé -> não é, dagua -> da água), one typo over
TRY, and an accent-folding stem scan — the Polish rounds, retuned for
Portuguese.
* storytold/wordcraft#100: pt — Portuguese grammar flags
Conservative rules with Portuguese messages: repeated words (except
the emphatic doubles Portuguese allows), space before punctuation and
double spaces, sentence capitals knowing the Portuguese abbreviations
and dialogue dashes, and doubled punctuation. No comma rule: que opens
a clause only sometimes, so a wrong flag would cost more than a missed
one.
* storytold/wordcraft#100: pt — Portuguese hyphenation
hyph-utf8's hyph-pt.tex bundled unmodified (BSD 3-clause, ATTRIBUTION
row with source and hash): 426 Liang patterns and 2 exception words;
falls back to the generic heuristic outside their reach.
* storytold/wordcraft#100: pt — Review › Language offers Português (Brasil), and a pt-br interface writes pt-BR
The menu's fourth entry sets the run language; the status bar names
Portuguese; editing_lang is now allowed for pt-br (wordcraft-proof
proofs it), so new documents under a Portuguese interface are
Portuguese by default.
* storytold/wordcraft#100: pt — engine test: Portuguese proofing through the review commands
review.language pt-BR normalizes the tag, review.issues reports
Portuguese spelling and grammar with the lang field, F7 and Change
apply a suggestion, Add to Dictionary works for Portuguese.
* storytold/wordcraft#100: pt — Portuguese in the mixed-language proofing tests and docs
check_text proofs a Portuguese span with the Portuguese message; the
never-panic loops and the module docs cover the third language.
* storytold/wordcraft#100: docs — Portuguese proofing in the gap, parity and roadmap docs
Gap 8 narrowed (English, Polish and Portuguese), localization parity
gains a Portuguese row and a revision entry, target-app-parity's
Proofing row moves 40% -> 42%, and the ROADMAP progress log records
what landed.
* storytold/wordcraft#100: hunspell — LibreOffice extension folders follow the language
The search paths hardcoded LibreOffice's dict-pl; a pt_BR user whose
only dictionary is LibreOffice's dict-pt-BR extension — on Windows the
only realistic on-disk source — got silently no spelling. Both the
primary-language folder and the dashed full name are now tried, so
Polish keeps dict-pl and Portuguese finds dict-pt-BR.
* storytold/wordcraft#100: pt — review fixes
Suggestion ties break toward the change nearest the word's start
(saúde before saudê, proven against a fixture that has both); the
Language menu's label is the English source "Portuguese (Brazil)" with
translations in all twelve catalogs (as Polish is); the attribution
counts pt.pat's 2 exception words; the pt fixture drops a duplicate
stem; docs/roadmap.md catches up with the landing; the throwaway spike
is gone.
* Review fixes for storytold/wordcraft#782: two-suffix chain limit, module doc
- MAX_CHAIN = 2 counts suffixes in a row, matching Hunspell's two-suffix limit
(the old bound of 3 allowed up to four); test pins that a third is rejected.
- Module doc: say probes allocate small scratch strings instead of claiming none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Review fixes for storytold/wordcraft#797: NOTICE for pt.pat, no indexing in the LibreOffice folder helper
- NOTICE reproduces hyph-pt.tex's BSD 3-Clause copyright notice, conditions and
disclaimer, which the licence requires for binary redistributions.
- hunspell::lo_ext_dirs builds its list without indexing (never-crash rule).
- Catalogs: main already translates "Portuguese (Brazil)" in every language, so the
duplicate rows the branch added are dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: witkowski01 <konradwitkowski01@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
This commit is contained in:
1 parent
229f9b32d2
commit
3090bcaef0
8 files changed
+1151
-22
No files matched your search
@@ -26,8 +26,20 @@ pub fn set_dictionary_dir(dir: Option<PathBuf>) {
|
||||
*EXTRA_DIR.write().unwrap_or_else(|e| e.into_inner()) = dir;
|
||||
}
|
||||
|
||||
/// Directories that may hold Hunspell dictionaries, most specific first.
|
||||
pub fn search_dirs() -> Vec<PathBuf> {
|
||||
/// The LibreOffice extension directories a Hunspell dictionary name may live in: LibreOffice's
|
||||
/// IDs are per dictionary — Polish is `dict-pl`, Brazilian Portuguese `dict-pt-BR` — so both
|
||||
/// the primary-language folder and the dashed full name are tried.
|
||||
fn lo_ext_dirs(name: &str) -> Vec<String> {
|
||||
let primary = name.split(['-', '_']).next().unwrap_or(name);
|
||||
let dashed = name.replace('_', "-");
|
||||
let first = format!("dict-{primary}");
|
||||
let full = format!("dict-{dashed}");
|
||||
if full == first { vec![first] } else { vec![first, full] }
|
||||
}
|
||||
|
||||
/// Directories that may hold Hunspell dictionaries, most specific first. `lo_exts`, when a
|
||||
/// language is being searched for, adds LibreOffice's extension folders for it (`dict-pt-BR`).
|
||||
pub fn search_dirs(lo_exts: Option<&[String]>) -> Vec<PathBuf> {
|
||||
let mut dirs: Vec<PathBuf> = Vec::new();
|
||||
if let Some(d) = EXTRA_DIR.read().unwrap_or_else(|e| e.into_inner()).clone() {
|
||||
dirs.push(d);
|
||||
@@ -38,6 +50,9 @@ pub fn search_dirs() -> Vec<PathBuf> {
|
||||
if let Some(d) = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()) {
|
||||
dirs.push(PathBuf::from(d));
|
||||
}
|
||||
// LibreOffice keeps each language under `…/share/extensions/<dict-xx>`.
|
||||
let lo_dirs =
|
||||
|prefix: &str| -> Vec<PathBuf> { lo_exts.map_or_else(Vec::new, |exts| exts.iter().map(|e| PathBuf::from(prefix).join(e)).collect()) };
|
||||
let home = std::env::var_os("HOME").filter(|v| !v.is_empty()).map(PathBuf::from);
|
||||
if cfg!(windows) {
|
||||
for var in ["LOCALAPPDATA", "APPDATA"] {
|
||||
@@ -48,7 +63,7 @@ pub fn search_dirs() -> Vec<PathBuf> {
|
||||
}
|
||||
for var in ["ProgramFiles", "ProgramFiles(x86)"] {
|
||||
if let Some(base) = std::env::var_os(var).filter(|v| !v.is_empty()).map(PathBuf::from) {
|
||||
dirs.push(base.join("LibreOffice").join("share").join("extensions").join("dict-pl"));
|
||||
dirs.extend(lo_dirs(&base.join("LibreOffice").join("share").join("extensions").to_string_lossy()));
|
||||
}
|
||||
}
|
||||
} else if cfg!(target_os = "macos") {
|
||||
@@ -58,30 +73,26 @@ pub fn search_dirs() -> Vec<PathBuf> {
|
||||
dirs.push(PathBuf::from("/Library/Spelling"));
|
||||
dirs.push(PathBuf::from("/opt/homebrew/share/hunspell"));
|
||||
dirs.push(PathBuf::from("/usr/local/share/hunspell"));
|
||||
dirs.push(PathBuf::from("/Applications/LibreOffice.app/Contents/Resources/extensions/dict-pl"));
|
||||
dirs.extend(lo_dirs("/Applications/LibreOffice.app/Contents/Resources/extensions"));
|
||||
} else {
|
||||
if let Some(h) = &home {
|
||||
dirs.push(h.join(".local/share/hunspell"));
|
||||
}
|
||||
for d in [
|
||||
"/usr/share/hunspell",
|
||||
"/usr/share/myspell",
|
||||
"/usr/share/myspell/dicts",
|
||||
"/usr/local/share/hunspell",
|
||||
"/usr/local/share/myspell",
|
||||
"/usr/lib/libreoffice/share/extensions/dict-pl",
|
||||
"/opt/libreoffice/share/extensions/dict-pl",
|
||||
] {
|
||||
for d in ["/usr/share/hunspell", "/usr/share/myspell", "/usr/share/myspell/dicts", "/usr/local/share/hunspell", "/usr/local/share/myspell"] {
|
||||
dirs.push(PathBuf::from(d));
|
||||
}
|
||||
for d in ["/usr/lib/libreoffice/share/extensions", "/opt/libreoffice/share/extensions"] {
|
||||
dirs.extend(lo_dirs(d));
|
||||
}
|
||||
}
|
||||
dirs
|
||||
}
|
||||
|
||||
/// The first `<name>.aff`/`<name>.dic` pair found for any of `names` (`pl_PL`, `pl`), searching
|
||||
/// [`search_dirs`].
|
||||
/// [`search_dirs`] (including LibreOffice's extension folder for the first name).
|
||||
pub fn find(names: &[&str]) -> Option<(PathBuf, PathBuf)> {
|
||||
search_dirs().into_iter().find_map(|dir| find_in(&dir, names))
|
||||
let lo = names.first().copied().map(lo_ext_dirs);
|
||||
search_dirs(lo.as_deref()).into_iter().find_map(|dir| find_in(&dir, names))
|
||||
}
|
||||
|
||||
/// The first `<name>.aff`/`<name>.dic` pair for any of `names` in `dir`.
|
||||
@@ -146,6 +157,18 @@ mod tests {
|
||||
assert!(decode(b"SET UTF-8\n", b"\xff\xfe").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn libreoffice_extension_dirs_follow_the_language() {
|
||||
// LibreOffice's IDs are per dictionary: Polish is `dict-pl`, Brazilian Portuguese
|
||||
// `dict-pt-BR`, so both the primary-language folder and the dashed full name are tried.
|
||||
assert_eq!(super::lo_ext_dirs("pl_PL"), ["dict-pl", "dict-pl-PL"]);
|
||||
assert_eq!(super::lo_ext_dirs("pt_BR"), ["dict-pt", "dict-pt-BR"]);
|
||||
assert_eq!(super::lo_ext_dirs("de-DE"), ["dict-de", "dict-de-DE"]);
|
||||
let dirs = search_dirs(Some(&super::lo_ext_dirs("pt_BR")));
|
||||
assert!(dirs.iter().any(|d| d.ends_with("dict-pt-BR")), "{dirs:?}");
|
||||
assert!(search_dirs(None).iter().all(|d| !d.to_string_lossy().contains("dict-")), "no language: no LibreOffice guess");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn finds_and_loads_a_pair_in_a_directory() {
|
||||
let dir = std::env::temp_dir().join(format!("wordcraft-hunspell-{}", std::process::id()));
|
||||
|
||||
@@ -29,6 +29,8 @@ static EN_US_DIC: &[u8] = include_bytes!("../../../assets/hyphenation/en-us.dic"
|
||||
static EN_US_PAT: &str = include_str!("../../../assets/hyphenation/en-us.pat");
|
||||
/// hyph-utf8's `hyph-pl.tex`, unmodified (its licence header is in the file).
|
||||
static PL_PAT: &str = include_str!("../../../assets/hyphenation/pl.pat");
|
||||
/// hyph-utf8's `hyph-pt.tex`, unmodified (its BSD 3-clause header is in the file).
|
||||
static PT_PAT: &str = include_str!("../../../assets/hyphenation/pt.pat");
|
||||
|
||||
impl Dictionary {
|
||||
/// The bundled US English dictionary (decoded on first use).
|
||||
@@ -58,6 +60,12 @@ impl Patterns {
|
||||
static P: OnceLock<Patterns> = OnceLock::new();
|
||||
P.get_or_init(|| Patterns::parse(PL_PAT))
|
||||
}
|
||||
|
||||
/// The bundled Portuguese patterns and exception words (parsed on first use).
|
||||
pub fn pt() -> &'static Patterns {
|
||||
static P: OnceLock<Patterns> = OnceLock::new();
|
||||
P.get_or_init(|| Patterns::parse(PT_PAT))
|
||||
}
|
||||
}
|
||||
|
||||
/// Hyphenation limits from the paragraph's Hyphenation settings.
|
||||
@@ -101,6 +109,9 @@ fn core_breaks(core: &[char], lang: ProofLang) -> (Vec<usize>, Source) {
|
||||
if lang == ProofLang::Pl {
|
||||
return core_breaks_pl(core);
|
||||
}
|
||||
if lang == ProofLang::Pt {
|
||||
return core_breaks_pt(core);
|
||||
}
|
||||
let dict = Dictionary::en_us();
|
||||
let s: String = core.iter().collect();
|
||||
let from_mask = |m: u64, len: usize| (1..len.min(64)).filter(|&i| m & (1 << i) != 0).collect::<Vec<_>>();
|
||||
@@ -140,6 +151,21 @@ fn core_breaks_pl(core: &[char]) -> (Vec<usize>, Source) {
|
||||
(heuristic_points_pl(core), Source::Heuristic)
|
||||
}
|
||||
|
||||
/// [`core_breaks_pl`] for Portuguese: the bundled patterns, then the generic heuristic.
|
||||
fn core_breaks_pt(core: &[char]) -> (Vec<usize>, Source) {
|
||||
let pats = Patterns::pt();
|
||||
let s: String = core.iter().collect();
|
||||
if let Some(p) = pats.exception(&s) {
|
||||
return (p.to_vec(), Source::Dictionary);
|
||||
}
|
||||
if !pats.is_empty()
|
||||
&& let Some(p) = pats.points(core)
|
||||
{
|
||||
return (p, Source::Patterns);
|
||||
}
|
||||
(heuristic_points(core), Source::Heuristic)
|
||||
}
|
||||
|
||||
/// Unrestricted break points of an English `word` ([`word_breaks_in`] with [`ProofLang::En`]).
|
||||
pub fn word_breaks(word: &str) -> (Vec<usize>, Source) {
|
||||
word_breaks_in(word, ProofLang::En)
|
||||
@@ -667,6 +693,40 @@ mod tests {
|
||||
assert_eq!(p.exception("nigdy"), Some(&[2][..]));
|
||||
}
|
||||
|
||||
fn pt(w: &str) -> String {
|
||||
hyphenate_word_in(w, &Limits::default(), ProofLang::Pt)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn portuguese_patterns_load() {
|
||||
let p = Patterns::pt();
|
||||
assert!(p.len() > 300, "{}", p.len());
|
||||
// `hard-ware`, `soft-ware` — and the file's last pattern line, a bare `1-`, which the
|
||||
// reader counts as a (never-matching) exception because it contains a hyphen.
|
||||
assert_eq!(p.exception_count(), 3);
|
||||
assert_eq!(p.exception("hardware"), Some(&[4][..]));
|
||||
assert_eq!(p.exception("software"), Some(&[4][..]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn portuguese_words_hyphenate_by_portuguese_rules() {
|
||||
for (w, want) in [
|
||||
("problema", "pro-ble-ma"),
|
||||
("coração", "co-ra-ção"),
|
||||
("paralelepípedo", "pa-ra-le-le-pí-pe-do"),
|
||||
("computador", "com-pu-ta-dor"),
|
||||
("escrever", "es-cre-ver"),
|
||||
("saudade", "sau-da-de"),
|
||||
("extraordinário", "ex-tra-or-di-ná-rio"),
|
||||
("automático", "au-to-má-ti-co"),
|
||||
// One syllable: no break.
|
||||
("flor", "flor"),
|
||||
("bem", "bem"),
|
||||
] {
|
||||
assert_eq!(pt(w), want, "{w}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn polish_words_hyphenate_by_polish_rules() {
|
||||
for (w, want) in [
|
||||
|
||||
@@ -13,6 +13,10 @@ pub enum ProofLang {
|
||||
/// Polish (`pl`, `pl-PL`): the installed Hunspell dictionary (when there is one), the Polish grammar rules and the Polish
|
||||
/// hyphenation patterns.
|
||||
Pl,
|
||||
/// Portuguese (`pt`, `pt-BR`, `pt-PT`): the installed Hunspell dictionary (VERO and
|
||||
/// friends, when there is one), the Portuguese grammar rules and the Portuguese hyphenation
|
||||
/// patterns.
|
||||
Pt,
|
||||
/// A language without proofing tools here (`fr-FR`, `ja-JP`, `x-none`…). Like Word without
|
||||
/// the language's proofing tools: no spelling or grammar marks, so its words are never
|
||||
/// flagged as English misspellings. Hyphenation keeps using the English patterns, as before
|
||||
@@ -22,7 +26,7 @@ pub enum ProofLang {
|
||||
|
||||
impl ProofLang {
|
||||
/// Every language with a spelling dictionary, in display order.
|
||||
pub const PROOFED: [ProofLang; 2] = [ProofLang::En, ProofLang::Pl];
|
||||
pub const PROOFED: [ProofLang; 3] = [ProofLang::En, ProofLang::Pl, ProofLang::Pt];
|
||||
|
||||
/// The proofing language of a BCP 47 tag (`pl-PL`, `en_GB`, `PL`…). Only the primary subtag
|
||||
/// counts. No tag (or an empty one) is English, WordCraft's default editing language.
|
||||
@@ -33,6 +37,8 @@ impl ProofLang {
|
||||
ProofLang::En
|
||||
} else if primary.eq_ignore_ascii_case("pl") || primary.eq_ignore_ascii_case("pol") {
|
||||
ProofLang::Pl
|
||||
} else if primary.eq_ignore_ascii_case("pt") || primary.eq_ignore_ascii_case("por") {
|
||||
ProofLang::Pt
|
||||
} else {
|
||||
ProofLang::Other
|
||||
}
|
||||
@@ -43,6 +49,7 @@ impl ProofLang {
|
||||
match self {
|
||||
ProofLang::En => "en",
|
||||
ProofLang::Pl => "pl",
|
||||
ProofLang::Pt => "pt",
|
||||
ProofLang::Other => "",
|
||||
}
|
||||
}
|
||||
@@ -116,6 +123,9 @@ mod tests {
|
||||
for t in ["pl", "pl-PL", "PL-pl", "pl_PL", "pl_PL.UTF-8", " pl-PL ", "pol"] {
|
||||
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pl, "{t}");
|
||||
}
|
||||
for t in ["pt", "pt-BR", "PT_br", "pt-BR.UTF-8", " pt-PT ", "por"] {
|
||||
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pt, "{t}");
|
||||
}
|
||||
for t in ["en", "en-US", "en-GB", "EN_au", "en-Latn-US"] {
|
||||
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::En, "{t}");
|
||||
}
|
||||
@@ -126,6 +136,9 @@ mod tests {
|
||||
assert_eq!(ProofLang::from_tag(Some("")), ProofLang::En);
|
||||
assert_eq!(ProofLang::from_tag(Some(" ")), ProofLang::En);
|
||||
assert!(ProofLang::Pl.is_proofed() && !ProofLang::Other.is_proofed());
|
||||
assert!(ProofLang::Pt.is_proofed() && !ProofLang::Other.is_proofed());
|
||||
assert_eq!(ProofLang::Pl.code(), "pl");
|
||||
assert_eq!(ProofLang::Pt.code(), "pt");
|
||||
assert!(ProofLang::PROOFED.contains(&ProofLang::Pt), "every proofed language is listed");
|
||||
}
|
||||
}
|
||||
+27
-6
@@ -12,9 +12,13 @@
|
||||
//! spelling marks), read by the affix engine ([`affix`]), suggestions tuned for Polish
|
||||
//! (diacritics, `ż`/`rz`, `u`/`ó`, missing spaces), and Polish grammar rules with Polish
|
||||
//! messages.
|
||||
//! - **Portuguese** ([`ProofLang::Pt`], module [`pt`]): spelling against the Hunspell `pt_BR`
|
||||
//! dictionary installed on the system (VERO and friends; not bundled), suggestions tuned for
|
||||
//! Portuguese (missing accents, `seje`/`seja` confusions, missing spaces), conservative
|
||||
//! grammar rules with Portuguese messages, and `hyph-utf8`'s Portuguese patterns.
|
||||
//! - **Other languages** ([`ProofLang::Other`]): no spelling or grammar marks, like Word without
|
||||
//! the language's proofing tools.
|
||||
//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish), heuristic.
|
||||
//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish, Portuguese), heuristic.
|
||||
//!
|
||||
//! [`check_text`] checks a paragraph whose runs are in different languages.
|
||||
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
|
||||
@@ -26,6 +30,7 @@ pub mod hyphen;
|
||||
pub mod lang;
|
||||
pub mod patterns;
|
||||
pub mod pl;
|
||||
pub mod pt;
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::ops::Range;
|
||||
@@ -489,6 +494,7 @@ fn spelling_issue(text: &str, a: usize, b: usize, lang: ProofLang, opts: &Option
|
||||
}
|
||||
let message = match lang {
|
||||
ProofLang::Pl => "Możliwy błąd pisowni",
|
||||
ProofLang::Pt => "Possível erro de ortografia",
|
||||
_ => "Possible spelling mistake",
|
||||
};
|
||||
Some(Issue { start: a, end: b, kind: IssueKind::Spelling, message: message.into(), suggestions: Vec::new(), lang })
|
||||
@@ -504,6 +510,7 @@ pub fn is_correct_in_with(word: &str, lang: ProofLang, opts: &Options) -> bool {
|
||||
match lang {
|
||||
ProofLang::En => is_correct_with(word, opts),
|
||||
ProofLang::Pl => accepted_by_options(word, opts) || pl::is_correct(word),
|
||||
ProofLang::Pt => accepted_by_options(word, opts) || pt::is_correct(word),
|
||||
ProofLang::Other => true,
|
||||
}
|
||||
}
|
||||
@@ -513,16 +520,19 @@ pub fn suggest_in(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
|
||||
match lang {
|
||||
ProofLang::En => suggest(word, max),
|
||||
ProofLang::Pl => pl::suggest(word, max),
|
||||
ProofLang::Pt => pt::suggest(word, max),
|
||||
ProofLang::Other => Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Grammar issues in a paragraph of text in `lang`: the English rules, the Polish ones
|
||||
/// ([`pl::grammar`]), or none for a language WordCraft doesn't proof.
|
||||
/// ([`pl::grammar`]), the Portuguese ones ([`pt::grammar`]), or none for a language WordCraft
|
||||
/// doesn't proof.
|
||||
pub fn check_grammar_in(text: &str, lang: ProofLang) -> Vec<Issue> {
|
||||
match lang {
|
||||
ProofLang::En => check_grammar(text),
|
||||
ProofLang::Pl => pl::grammar::check(text),
|
||||
ProofLang::Pt => pt::grammar::check(text),
|
||||
ProofLang::Other => Vec::new(),
|
||||
}
|
||||
}
|
||||
@@ -994,6 +1004,12 @@ mod tests {
|
||||
assert_eq!(flagged("helo świat", &check_text("helo świat", &[(5..12, ProofLang::Pl)])), ["helo"]);
|
||||
// A language without tools: nothing flagged there.
|
||||
assert!(check_text("Bonjour mes amis", &[(0..16, ProofLang::Other)]).is_empty());
|
||||
// Portuguese checks in their span only, with the Portuguese message.
|
||||
pt::test_dict::install();
|
||||
let t = "Ele disse que coraçao e frequencia existem.";
|
||||
let v = check_text(t, &[(0..t.len(), ProofLang::Pt)]);
|
||||
assert_eq!(flagged(t, &v), ["coraçao", "frequencia"], "{v:?}");
|
||||
assert!(v.iter().all(|i| i.lang == ProofLang::Pt && i.message == "Possível erro de ortografia"), "{v:?}");
|
||||
}
|
||||
|
||||
proptest::proptest! {
|
||||
@@ -1005,7 +1021,7 @@ mod tests {
|
||||
let _ = has_dictionary(Some(&s));
|
||||
let _ = is_correct_for(&s, Some(&s));
|
||||
let _ = hyphen::hyphen_points(&s, &hyphen::Limits::default());
|
||||
for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Other] {
|
||||
for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Pt, ProofLang::Other] {
|
||||
let _ = check_spelling_in(&s, lang);
|
||||
let _ = check_grammar_in(&s, lang);
|
||||
let _ = is_correct_in(&s, lang);
|
||||
@@ -1021,9 +1037,14 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..3), 0..6)) {
|
||||
let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Other];
|
||||
let spans: Vec<(Range<usize>, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 3])).collect();
|
||||
fn portuguese_suggestions_never_panic(s in "[a-záàâãéêíóôõúüçA-ZÁÀÂÃÉÊÍÓÔÕÚÜÇ'’ -]{0,14}") {
|
||||
let _ = suggest_in(&s, 4, ProofLang::Pt);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..4), 0..6)) {
|
||||
let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Pt, ProofLang::Other];
|
||||
let spans: Vec<(Range<usize>, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 4])).collect();
|
||||
for i in check_text(&s, &spans) {
|
||||
proptest::prop_assert!(i.start <= i.end && i.end <= s.len());
|
||||
}
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
//! Portuguese grammar and punctuation checks: cheap, rule-based and tuned to stay quiet when
|
||||
//! unsure (a missed mistake costs less than a wrong mark on correct text). Messages are in
|
||||
//! Portuguese.
|
||||
//!
|
||||
//! - **Repeated word** (`ele ele`), except interjections and words Portuguese doubles on purpose
|
||||
//! (`sim sim`, `não não`, `ora ora`).
|
||||
//! - **Space before punctuation** (`bem ,`) and **double spaces**; emoticons (` :)`) and
|
||||
//! ellipses are left alone.
|
||||
//! - **Capital letter at the start of a sentence**, knowing Portuguese abbreviations (`etc.`,
|
||||
//! `pág.`, `Prof.`, `Dr.`), initials, ordinal numbers (`5. colocado`), ellipses and dialogue
|
||||
//! dashes (`— Vai — disse`), after which lower case is correct.
|
||||
//! - **Doubled punctuation** (`,,`, `..`, `;;`).
|
||||
//!
|
||||
//! No comma rule (`que` opens clauses only sometimes) and no English rule (such as `a`/`an`)
|
||||
//! applies to Portuguese text.
|
||||
|
||||
use crate::{Issue, IssueKind, ProofLang, words};
|
||||
|
||||
/// Words Portuguese repeats on purpose: interjections, onomatopoeia and emphatic doubling.
|
||||
const ALLOWED_REPEATS: &[&str] = &[
|
||||
"sim", "não", "ora", "olha", "já", "bem", "muito", "vai", "lá", "cá", "aí", "ha", "he", "hi", "oh", "ah", "ai", "ui", "uf", "uxa", "oba", "viva",
|
||||
"psiu", "olá", "tchau", "xau", "adeus", "devagar",
|
||||
];
|
||||
|
||||
/// Abbreviations written with a final period after which a sentence goes on in lower case
|
||||
/// (lower case, without the period).
|
||||
const ABBREVIATIONS: &[&str] = &[
|
||||
"etc", "ex", "pág", "págs", "séc", "sécs", "fl", "av", "vol", "p", "pp", "ed", "tel", "min", "max", "art", "par", "prof", "profa", "dr", "dra",
|
||||
"sr", "sra", "srta", "eng", "arq", "op", "cit", "cf", "ibid", "id", "vide", "seg", "ter", "qua", "qui", "sex", "sáb", "dom", "jan", "fev", "mar",
|
||||
"abr", "mai", "jun", "jul", "ago", "set", "out", "nov", "dez", "s.a", "d.c", "a.c", "pg", "ref", "cod", "dept", "univ", "inst", "lat", "ing",
|
||||
"esp", "fr", "it", "al", "obs", "att", "obs",
|
||||
];
|
||||
|
||||
fn issue(start: usize, end: usize, message: String, suggestions: Vec<String>) -> Issue {
|
||||
Issue { start, end, kind: IssueKind::Grammar, message, suggestions, lang: ProofLang::Pt }
|
||||
}
|
||||
|
||||
fn capitalize(w: &str) -> String {
|
||||
crate::affix::capitalize(w)
|
||||
}
|
||||
|
||||
/// Is `prev`, before a period, an abbreviation, an initial, a number or a Roman numeral (after
|
||||
/// which a lower-case word doesn't start a sentence)?
|
||||
fn abbreviation_or_number(prev: &str) -> bool {
|
||||
prev.chars().count() <= 1
|
||||
|| prev.chars().any(|c| c.is_ascii_digit())
|
||||
|| (prev.chars().all(|c| matches!(c, 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M')))
|
||||
|| ABBREVIATIONS.contains(&prev.to_lowercase().as_str())
|
||||
}
|
||||
|
||||
const CLOSING: [char; 8] = ['"', '”', '’', '\'', ')', '»', ']', '\u{1}'];
|
||||
const OPENING: [char; 8] = ['"', '“', '«', '(', '[', '\'', '‚', '—'];
|
||||
|
||||
/// Does the gap between two words end a sentence: a `.`, `!` or `?` (not an ellipsis, not after
|
||||
/// an abbreviation), then space, with nothing but quotes and brackets around?
|
||||
fn ends_sentence(gap: &str, prev: &str) -> bool {
|
||||
let t = gap.trim_start_matches(CLOSING);
|
||||
let Some(end) = t.chars().next() else { return false };
|
||||
if !matches!(end, '.' | '!' | '?') || t.starts_with("..") {
|
||||
return false;
|
||||
}
|
||||
if end == '.' && abbreviation_or_number(prev) {
|
||||
return false;
|
||||
}
|
||||
let rest = t.trim_start_matches(['.', '!', '?']).trim_start_matches(CLOSING);
|
||||
rest.starts_with(char::is_whitespace) && rest.trim_start().chars().all(|c| OPENING.contains(&c))
|
||||
}
|
||||
|
||||
/// Grammar issues in a paragraph of Portuguese text, in order.
|
||||
pub fn check(text: &str) -> Vec<Issue> {
|
||||
let ws: Vec<(usize, usize)> = words(text);
|
||||
let mut v = Vec::new();
|
||||
repeated_words(text, &ws, &mut v);
|
||||
spaces(text, &mut v);
|
||||
sentence_capitals(text, &ws, &mut v);
|
||||
doubled_punctuation(text, &mut v);
|
||||
v.sort_by_key(|i| (i.start, i.end));
|
||||
v
|
||||
}
|
||||
|
||||
fn repeated_words(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
|
||||
for pair in ws.windows(2) {
|
||||
let [(a0, b0), (a1, b1)] = pair else { continue };
|
||||
let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue };
|
||||
if !gap.is_empty()
|
||||
&& gap.chars().all(char::is_whitespace)
|
||||
&& w0.chars().all(char::is_alphabetic)
|
||||
&& w0.to_lowercase() == w1.to_lowercase()
|
||||
&& !ALLOWED_REPEATS.contains(&w0.to_lowercase().as_str())
|
||||
{
|
||||
v.push(issue(*a0, *b1, format!("Palavra repetida: \"{w1}\""), vec![w0.to_string()]));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn spaces(text: &str, v: &mut Vec<Issue>) {
|
||||
for (i, c) in text.char_indices() {
|
||||
if c != ' ' {
|
||||
continue;
|
||||
}
|
||||
let rest = text.get(i + 1..).unwrap_or("");
|
||||
let mut it = rest.chars();
|
||||
let Some(p) = it.next() else { continue };
|
||||
let after = it.next();
|
||||
let flag = match p {
|
||||
',' => true,
|
||||
// `;)` and `;-)` are emoticons.
|
||||
';' => !matches!(after, Some(')' | '(' | '-' | 'P' | 'D')),
|
||||
// ` ...` is an ellipsis, ` .5` a number.
|
||||
'.' => !matches!(after, Some('.')) && !after.is_some_and(|a| a.is_ascii_digit()),
|
||||
// ` :)`, ` :-)`, ` :D` and similar are emoticons; ` ?!` is still a stray space.
|
||||
':' | '!' | '?' => after.is_none_or(|a| a.is_whitespace() || matches!(a, '"' | '”' | '’' | '!' | '?')),
|
||||
_ => false,
|
||||
};
|
||||
if flag {
|
||||
v.push(issue(i, i + 1 + p.len_utf8(), "Espaço antes da pontuação".into(), vec![p.to_string()]));
|
||||
}
|
||||
}
|
||||
for (i, _) in text.match_indices(" ") {
|
||||
if i > 0 && text.get(..i).is_some_and(|t| t.ends_with(' ')) {
|
||||
continue;
|
||||
}
|
||||
v.push(issue(i, i + 2, "Espaço duplo".into(), vec![" ".into()]));
|
||||
}
|
||||
}
|
||||
|
||||
fn sentence_capitals(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
|
||||
// List items (`– maçãs,`) may start in lower case.
|
||||
let list_item = text.trim_end().ends_with([',', ';']);
|
||||
for (k, &(a, b)) in ws.iter().enumerate() {
|
||||
let Some(w) = text.get(a..b) else { continue };
|
||||
let starts = if k == 0 {
|
||||
let before = text.get(..a).unwrap_or("");
|
||||
!list_item && before.chars().all(|c| c.is_whitespace() || OPENING.contains(&c))
|
||||
} else {
|
||||
let Some(&(pa, pb)) = ws.get(k - 1) else { continue };
|
||||
ends_sentence(text.get(pb..a).unwrap_or(""), text.get(pa..pb).unwrap_or(""))
|
||||
};
|
||||
if starts && w.chars().next().is_some_and(char::is_lowercase) && w.chars().all(char::is_alphabetic) {
|
||||
v.push(issue(a, b, "Letra maiúscula no início da frase".into(), vec![capitalize(w)]));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn doubled_punctuation(text: &str, v: &mut Vec<Issue>) {
|
||||
let bytes = text.as_bytes();
|
||||
let mut i = 0;
|
||||
while let Some(&b) = bytes.get(i) {
|
||||
if !matches!(b, b',' | b'.' | b';') {
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
let run = bytes.get(i..).map_or(0, |r| r.iter().take_while(|&&x| x == b).count());
|
||||
let flag = match b {
|
||||
b'.' => run == 2,
|
||||
_ => run >= 2,
|
||||
};
|
||||
if flag {
|
||||
let end = i + run;
|
||||
let suggestions: Vec<String> = match b {
|
||||
b'.' => vec![".".into(), "…".into()],
|
||||
_ => vec![char::from(b).to_string()],
|
||||
};
|
||||
v.push(issue(i, end, "Sinal repetido".into(), suggestions));
|
||||
}
|
||||
i += run.max(1);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn messages(t: &str) -> Vec<String> {
|
||||
check(t).into_iter().map(|i| format!("{}@{}", i.message, &t[i.start..i.end])).collect()
|
||||
}
|
||||
|
||||
fn clean(t: &str) {
|
||||
let m = messages(t);
|
||||
assert!(m.is_empty(), "{t:?}: {m:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn repeated_words() {
|
||||
let v = check("Eu eu vi isso e ele ele saiu.");
|
||||
assert_eq!(v.len(), 2, "{v:?}");
|
||||
assert_eq!(v[0].message, "Palavra repetida: \"eu\"");
|
||||
assert_eq!(v[0].suggestions, ["Eu"]);
|
||||
assert!(v.iter().all(|i| i.lang == ProofLang::Pt && i.kind == IssueKind::Grammar));
|
||||
clean("Sim, sim! Não, não! Ora ora, que surpresa.");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn spaces_and_punctuation() {
|
||||
let m = messages("Olá , mundo . Tudo bem ?");
|
||||
assert!(m.contains(&"Espaço antes da pontuação@ ,".to_string()), "{m:?}");
|
||||
assert!(m.contains(&"Espaço antes da pontuação@ .".to_string()), "{m:?}");
|
||||
assert!(m.contains(&"Espaço antes da pontuação@ ?".to_string()), "{m:?}");
|
||||
assert!(m.contains(&"Espaço duplo@ ".to_string()), "{m:?}");
|
||||
clean("Legal :) Muito bom :-) Ou :D Valor .5 e reticências ...");
|
||||
let m = messages("Falou,, e saiu.. depois;; fim.");
|
||||
assert_eq!(m.iter().filter(|x| x.starts_with("Sinal repetido")).count(), 3, "{m:?}");
|
||||
clean("Esperei… e esperei... até ele voltar.");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sentence_start_capitals() {
|
||||
let m = messages("isso é um teste. a segunda frase! terceira? ok.");
|
||||
assert_eq!(m.iter().filter(|x| x.starts_with("Letra maiúscula")).count(), 4, "{m:?}");
|
||||
assert_eq!(check("ele tem razão.")[0].suggestions, ["Ele"]);
|
||||
// Abbreviations, initials, ordinals, ellipses and dialogue dashes.
|
||||
clean("Comprei frutas, etc. e depois saímos.");
|
||||
clean("Conheci o Dr. Silva na Av. Paulista.");
|
||||
clean("Zhou ficou em 5. lugar no séc. XX.");
|
||||
clean("Esperei... e ele apareceu.");
|
||||
clean("— Vai — disse ele. — Amanhã eu volto.");
|
||||
clean("Ele disse: \"Volto amanhã\". E saiu.");
|
||||
// A list item may start in lower case.
|
||||
clean("maçãs,");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn correct_portuguese_text_is_clean() {
|
||||
clean(
|
||||
"O rato roeu a roupa do rei de Roma. Quem não tem cão caça com gato, \
|
||||
e quem tem paciência colhe o que plantou.",
|
||||
);
|
||||
clean("Brasília, 12 de outubro de 2026 — uma data memorável.");
|
||||
}
|
||||
|
||||
proptest::proptest! {
|
||||
#[test]
|
||||
fn never_panics(s in "\\PC{0,80}") {
|
||||
let _ = check(&s);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn never_panics_on_portuguese_like_text(s in "[a-záàâãéêíóôõúüçA-Z ,.;:!?\"…\\-—]{0,60}") {
|
||||
for i in check(&s) {
|
||||
proptest::prop_assert!(i.start <= i.end && i.end <= s.len());
|
||||
proptest::prop_assert!(s.is_char_boundary(i.start) && s.is_char_boundary(i.end));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
//! Portuguese proofing.
|
||||
//!
|
||||
//! - **Spelling** against the Hunspell `pt_BR` dictionary installed on the system
|
||||
//! ([`dictionary`], found by [`crate::hunspell`]: `pt_BR.aff`/`pt_BR.dic` — VERO and its
|
||||
//! derivatives — from the OS, a package manager or LibreOffice; it isn't bundled). Without it
|
||||
//! Portuguese text gets no spelling marks. Capitalized stems are names (`Brasília`); a sentence
|
||||
//! may start with any word and headings may be in capitals. Hyphenated compounds are checked
|
||||
//! part by part (`guarda-roupa`, `ex-marido`), and the user dictionary applies as in English.
|
||||
//! - **Suggestions** tuned for Portuguese ([`suggest`]): missing accents first (`coraçao` →
|
||||
//! `coração`, `saude` → `saúde`), then the dictionary's `REP` confusions (`seje` → `seja`), a
|
||||
//! missing space, one-letter typos, and a dictionary scan for anything further off.
|
||||
//! - **Grammar** ([`grammar`]): cheap, reliable rules with Portuguese messages.
|
||||
|
||||
pub mod grammar;
|
||||
mod suggest;
|
||||
#[cfg(test)]
|
||||
pub(crate) mod test_dict;
|
||||
|
||||
use std::sync::{Arc, RwLock};
|
||||
|
||||
use crate::affix::AffixDict;
|
||||
|
||||
pub use suggest::suggest;
|
||||
|
||||
/// The Portuguese dictionary: `None` until first use, then the installed one (or an empty one).
|
||||
static DICT: RwLock<Option<Arc<AffixDict>>> = RwLock::new(None);
|
||||
|
||||
/// File names of a Portuguese Hunspell dictionary.
|
||||
const NAMES: &[&str] = &["pt_BR", "pt-BR", "pt"];
|
||||
|
||||
/// The Portuguese dictionary: the installed Hunspell `pt_BR` (read on first use), one given with
|
||||
/// [`install_dictionary`], or an empty one when none is available (Portuguese spelling is off
|
||||
/// then; see [`has_dictionary`]).
|
||||
pub fn dictionary() -> Arc<AffixDict> {
|
||||
if let Some(d) = DICT.read().unwrap_or_else(|e| e.into_inner()).as_ref() {
|
||||
return d.clone();
|
||||
}
|
||||
let loaded = Arc::new(load_installed());
|
||||
let mut slot = DICT.write().unwrap_or_else(|e| e.into_inner());
|
||||
// Another thread (or an install) may have filled it meanwhile: that one wins.
|
||||
slot.get_or_insert(loaded).clone()
|
||||
}
|
||||
|
||||
fn load_installed() -> AffixDict {
|
||||
let Some((aff, dic)) = crate::hunspell::find(NAMES) else {
|
||||
log::info!("Portuguese spelling: no Hunspell pt_BR dictionary installed");
|
||||
return AffixDict::default();
|
||||
};
|
||||
crate::hunspell::load(&aff, &dic).unwrap_or_else(|e| {
|
||||
log::warn!("Portuguese spelling: {e}");
|
||||
AffixDict::default()
|
||||
})
|
||||
}
|
||||
|
||||
/// Use this Portuguese dictionary from now on (Hunspell `.aff` and `.dic` text, decoded to
|
||||
/// UTF-8), for example one the user picked. On error the current dictionary stays.
|
||||
pub fn install_dictionary(aff: &str, dic: &str) -> Result<(), String> {
|
||||
let d = AffixDict::from_hunspell(aff, dic, &[])?;
|
||||
*DICT.write().unwrap_or_else(|e| e.into_inner()) = Some(Arc::new(d));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Forget the loaded dictionary, so the next check looks for an installed one again (after
|
||||
/// [`crate::hunspell::set_dictionary_dir`] or a dictionary was installed).
|
||||
pub fn reload_dictionary() {
|
||||
*DICT.write().unwrap_or_else(|e| e.into_inner()) = None;
|
||||
}
|
||||
|
||||
/// Is a Portuguese dictionary available (spelling is checked)?
|
||||
pub fn has_dictionary() -> bool {
|
||||
!dictionary().is_empty()
|
||||
}
|
||||
|
||||
/// Is `c` a letter of the Latin script (the only one the Portuguese dictionary covers)?
|
||||
fn is_latin_letter(c: char) -> bool {
|
||||
c.is_alphabetic() && (c < '\u{0250}' || ('\u{1E00}'..='\u{1EFF}').contains(&c))
|
||||
}
|
||||
|
||||
/// Is `word` spelled correctly in Portuguese? Numbers, single letters, short words in capitals
|
||||
/// (acronyms: `USP`, `PT`), words with digits and words in other scripts are accepted, and so
|
||||
/// is every word when no Portuguese dictionary is installed.
|
||||
pub fn is_correct(word: &str) -> bool {
|
||||
is_correct_with(&dictionary(), word)
|
||||
}
|
||||
|
||||
/// [`is_correct`] against a given dictionary (an empty one accepts everything).
|
||||
pub fn is_correct_with(d: &AffixDict, word: &str) -> bool {
|
||||
if d.is_empty() {
|
||||
return true; // no Portuguese dictionary installed: no spelling marks
|
||||
}
|
||||
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’');
|
||||
let w = w.trim_matches(['\'', '’']);
|
||||
let n = w.chars().count();
|
||||
if n <= 1 || w.chars().any(|c| c.is_ascii_digit()) || !w.chars().any(char::is_alphabetic) {
|
||||
return true;
|
||||
}
|
||||
if w.chars().all(|c| !c.is_lowercase()) && n <= 6 {
|
||||
return true;
|
||||
}
|
||||
if !w.chars().any(is_latin_letter) {
|
||||
return true; // other scripts: no dictionary
|
||||
}
|
||||
if w.contains('-') {
|
||||
return w.split('-').all(|p| p.chars().count() <= 1 || known(d, p));
|
||||
}
|
||||
known(d, w)
|
||||
}
|
||||
|
||||
/// Is one word (no hyphens) in the dictionary or the user dictionary?
|
||||
fn known(d: &AffixDict, w: &str) -> bool {
|
||||
d.check(w) || crate::in_user_dictionary(&w.to_lowercase())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_dictionary_loads() {
|
||||
test_dict::install();
|
||||
let d = dictionary();
|
||||
assert!(has_dictionary());
|
||||
assert!(d.len() > 50, "{}", d.len());
|
||||
assert!(d.rule_count() >= 5, "the test dictionary's affix rules");
|
||||
assert!(!d.rep().is_empty() && !d.map().is_empty() && !d.try_chars().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn without_a_dictionary_nothing_is_flagged() {
|
||||
let none = AffixDict::default();
|
||||
for w in ["coraçao", "saude", "xqzv", ""] {
|
||||
assert!(is_correct_with(&none, w), "{w}");
|
||||
}
|
||||
assert!(install_dictionary("FLAG long\n", "1\nkot\n").is_err(), "unsupported files are refused");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn portuguese_words_are_correct() {
|
||||
test_dict::install();
|
||||
for w in [
|
||||
"coração",
|
||||
"saudade",
|
||||
"pássaro",
|
||||
"saúde",
|
||||
"árvore",
|
||||
"escrever",
|
||||
"computador",
|
||||
"praia",
|
||||
"conhecer",
|
||||
"Brasília",
|
||||
"São",
|
||||
"autoestima",
|
||||
"frequência",
|
||||
"exceção",
|
||||
"amanhã",
|
||||
"vocês",
|
||||
"coragem",
|
||||
"parabéns",
|
||||
"cantava",
|
||||
"cantando",
|
||||
"provável",
|
||||
"prováveis",
|
||||
"descobrir",
|
||||
"trabalho",
|
||||
"CORAGEM",
|
||||
"USP",
|
||||
"2026",
|
||||
"e-mail",
|
||||
"guarda-roupa",
|
||||
"ex-marido",
|
||||
"vice-presidente",
|
||||
"e",
|
||||
"não",
|
||||
] {
|
||||
assert!(is_correct(w), "{w}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn misspellings_are_flagged() {
|
||||
test_dict::install();
|
||||
for w in ["coraçao", "saude", "trbalho", "excrever", "prsidente", "frequencia", "amnha", "cantaando"] {
|
||||
assert!(!is_correct(w), "{w}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn forbidden_words_are_rejected() {
|
||||
test_dict::install();
|
||||
// `frequencia/F` is listed with FORBIDDENWORD: the old spelling, never correct.
|
||||
assert!(!is_correct("frequencia") && !is_correct("Frequencia") && !is_correct("FREQUENCIA"));
|
||||
assert!(is_correct("frequência"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn user_dictionary_applies() {
|
||||
test_dict::install();
|
||||
assert!(!is_correct("zxqportuguespalavra"));
|
||||
crate::add_word("Zxqportuguespalavra");
|
||||
assert!(is_correct("zxqportuguespalavra"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,444 @@
|
||||
//! Portuguese spelling suggestions, best first.
|
||||
//!
|
||||
//! Candidates come in rounds of rising cost, each checked against the dictionary:
|
||||
//! 1. letter case (`brasília` → `Brasília`);
|
||||
//! 2. missing or wrong accents, from the dictionary's `MAP` groups (`coraçao` → `coração`,
|
||||
//! `saude` → `saúde`), up to four letters changed — Portuguese typed without accents is the
|
||||
//! most common slip, so adding one ranks first, while taking away an accent the writer typed
|
||||
//! ranks after the known confusions;
|
||||
//! 3. the dictionary's `REP` table of common confusions (`seje` → `seja`, `excessão` →
|
||||
//! `exceção`, `previlégio` → `privilégio`);
|
||||
//! 4. a missing space (`nãoé` → `não é`, `paraela` → `para ela`), cheaper when one half is a
|
||||
//! short function word;
|
||||
//! 5. one typo: a letter left out, added, swapped with the next or mistyped (`TRY` letters);
|
||||
//! 6. when that finds fewer than asked for: the stems that start like the word (accents
|
||||
//! folded) and share the most letter pairs with it, whose forms are ranked by an edit distance
|
||||
//! in which an accent slip costs half (two typos at most).
|
||||
//!
|
||||
//! The input is bounded (40 letters), and so is every round, so a call costs ten-odd thousand
|
||||
//! dictionary lookups at most.
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use crate::affix::{AffixDict, Casing, capitalize, casing};
|
||||
|
||||
/// Short words often run into the next one (`não é`, `da água`, `para isso`).
|
||||
const LEADING_WORDS: &[&str] = &[
|
||||
"a", "o", "e", "é", "de", "do", "da", "dos", "das", "em", "no", "na", "nos", "nas", "por", "para", "com", "sem", "sob", "sobre", "ao", "aos",
|
||||
"à", "às", "que", "se", "não", "como", "mas", "mais", "já", "até", "entre", "contra", "perante", "desde", "após", "ante", "durante", "mediante",
|
||||
"segundo", "conforme", "salvo", "fora", "onde", "quando", "bem", "mal", "pois", "logo", "portanto", "porém", "todavia", "contudo", "assim",
|
||||
"tão", "tanto", "tudo", "nada", "algo", "cada", "outro", "mesmo", "próprio", "ainda", "tampouco", "quase", "talvez", "nunca", "sempre",
|
||||
];
|
||||
/// Short words often run into the one before (`amo-te`, `deu-lhe`; in writing without the
|
||||
/// hyphen: `amote`, `deulhe`).
|
||||
const TRAILING_WORDS: &[&str] = &[
|
||||
"me", "te", "se", "lhe", "nos", "vos", "lhes", "o", "a", "os", "as", "ele", "ela", "eles", "elas", "isso", "isto", "aquilo", "mim", "ti", "si",
|
||||
"eu", "tu", "que", "de", "em", "é",
|
||||
];
|
||||
const MAX_LETTERS: usize = 40;
|
||||
const MAX_MAP_CHANGES: usize = 4;
|
||||
const MAX_MAP_VARIANTS: usize = 4096;
|
||||
/// Stems the scan of round 6 expands.
|
||||
const SCAN_STEMS: usize = 48;
|
||||
|
||||
const COST_CASE: u32 = 0;
|
||||
const COST_ADD_DIACRITIC: u32 = 4;
|
||||
const COST_OTHER_MAP: u32 = 7;
|
||||
const COST_DROP_DIACRITIC: u32 = 11;
|
||||
const COST_REP: u32 = 10;
|
||||
const COST_SPLIT: u32 = 12;
|
||||
const COST_SPLIT_RARE: u32 = 18;
|
||||
const COST_SWAP: u32 = 14;
|
||||
const COST_DROP_OR_ADD: u32 = 15;
|
||||
const COST_MISTYPE: u32 = 16;
|
||||
const COST_SCAN: u32 = 20;
|
||||
|
||||
struct Collector<'a> {
|
||||
d: &'a AffixDict,
|
||||
/// The word, lower case.
|
||||
input: String,
|
||||
tested: HashSet<String>,
|
||||
/// Suggestion → (cost, order found).
|
||||
found: HashMap<String, (u32, usize)>,
|
||||
seq: usize,
|
||||
}
|
||||
|
||||
impl Collector<'_> {
|
||||
/// Check a lower-case candidate (a word, or two separated by a space) and keep it if every
|
||||
/// word is in the dictionary, as written or capitalized (names).
|
||||
fn offer(&mut self, cand: &str, cost: u32) {
|
||||
if cand.is_empty() || cand == self.input || !self.tested.insert(cand.to_string()) {
|
||||
return;
|
||||
}
|
||||
let mut words = Vec::new();
|
||||
let mut cost = cost;
|
||||
for part in cand.split(' ') {
|
||||
if self.d.contains(part) {
|
||||
words.push(part.to_string());
|
||||
} else {
|
||||
let c = capitalize(part);
|
||||
if !self.d.contains(&c) {
|
||||
return;
|
||||
}
|
||||
words.push(c);
|
||||
// A name ranks after a common word at the same distance (`moça`, then `Moça`).
|
||||
cost += 1;
|
||||
}
|
||||
}
|
||||
self.keep(words.join(" "), cost);
|
||||
}
|
||||
|
||||
/// Keep a suggestion known to be correct.
|
||||
fn keep(&mut self, text: String, cost: u32) {
|
||||
self.seq += 1;
|
||||
let seq = self.seq;
|
||||
let e = self.found.entry(text).or_insert((cost, seq));
|
||||
if cost < e.0 {
|
||||
*e = (cost, seq);
|
||||
}
|
||||
}
|
||||
|
||||
fn count_within(&self, cost: u32) -> usize {
|
||||
self.found.values().filter(|(c, _)| *c <= cost).count()
|
||||
}
|
||||
}
|
||||
|
||||
/// The MAP group of each letter (its other members are its likely confusions).
|
||||
fn alternatives(c: char, map: &[Vec<char>]) -> Vec<char> {
|
||||
map.iter().filter(|g| g.contains(&c)).flat_map(|g| g.iter().copied()).filter(|&x| x != c).collect()
|
||||
}
|
||||
|
||||
/// The cost of changing letter `from` to its MAP relative `to`. Adding a missing accent is the
|
||||
/// commonest slip (Portuguese typed without accents: `coracao`); an accent the writer did type
|
||||
/// is likelier right, so taking one away costs more than a known confusion from the REP table
|
||||
/// (`seje` → `seja` before `sejé`).
|
||||
fn map_cost(from: char, to: char) -> u32 {
|
||||
match (from.is_ascii(), to.is_ascii()) {
|
||||
(true, false) => COST_ADD_DIACRITIC,
|
||||
(false, true) => COST_DROP_DIACRITIC,
|
||||
_ => COST_OTHER_MAP,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every way of changing up to [`MAX_MAP_CHANGES`] letters to a MAP relative, with its cost.
|
||||
fn map_variants(chars: &[char], alts: &[Vec<char>], i: usize, changes: usize, cost: u32, buf: &mut Vec<char>, out: &mut Vec<(String, u32)>) {
|
||||
if out.len() >= MAX_MAP_VARIANTS {
|
||||
return;
|
||||
}
|
||||
let Some(&c) = chars.get(i) else {
|
||||
if changes > 0 {
|
||||
out.push((buf.iter().collect(), cost));
|
||||
}
|
||||
return;
|
||||
};
|
||||
buf.push(c);
|
||||
map_variants(chars, alts, i + 1, changes, cost, buf, out);
|
||||
buf.pop();
|
||||
if changes < MAX_MAP_CHANGES {
|
||||
for &a in alts.get(i).map(Vec::as_slice).unwrap_or_default() {
|
||||
buf.push(a);
|
||||
map_variants(chars, alts, i + 1, changes + 1, cost + map_cost(c, a), buf, out);
|
||||
buf.pop();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A letter's MAP representative (the first of its group), lower case: `ã` → `a`, `ç` → `c`.
|
||||
fn fold(c: char, map: &[Vec<char>]) -> char {
|
||||
let l = c.to_lowercase().next().unwrap_or(c);
|
||||
map.iter().find(|g| g.contains(&l)).and_then(|g| g.first().copied()).unwrap_or(l)
|
||||
}
|
||||
|
||||
/// Optimal string alignment distance where a MAP slip (`a`/`á`, `c`/`ç`) costs 1 and any other
|
||||
/// edit 2; `limit + 1` once it is certainly above `limit`.
|
||||
fn weighted_distance(a: &[char], b: &[char], related: &dyn Fn(char, char) -> bool, limit: u32) -> u32 {
|
||||
let m = b.len();
|
||||
let over = limit + 1;
|
||||
let mut prev2 = vec![0u32; m + 1];
|
||||
let mut prev: Vec<u32> = (0..=m as u32).map(|j| j * 2).collect();
|
||||
let mut cur = vec![0u32; m + 1];
|
||||
for (i, &ai) in a.iter().enumerate() {
|
||||
if let Some(c) = cur.get_mut(0) {
|
||||
*c = (i as u32 + 1) * 2;
|
||||
}
|
||||
let mut row_min = u32::MAX;
|
||||
for (j, &bj) in b.iter().enumerate() {
|
||||
let sub = if ai == bj {
|
||||
0
|
||||
} else if related(ai, bj) {
|
||||
1
|
||||
} else {
|
||||
2
|
||||
};
|
||||
let at = |v: &[u32], k: usize| v.get(k).copied().unwrap_or(u32::MAX / 4);
|
||||
let mut v = (at(&prev, j + 1) + 2).min(at(&cur, j) + 2).min(at(&prev, j) + sub);
|
||||
if i > 0 && j > 0 && a.get(i - 1) == Some(&bj) && b.get(j - 1) == Some(&ai) {
|
||||
v = v.min(at(&prev2, j - 1) + 2);
|
||||
}
|
||||
if let Some(c) = cur.get_mut(j + 1) {
|
||||
*c = v;
|
||||
}
|
||||
row_min = row_min.min(v);
|
||||
}
|
||||
if m > 0 && row_min > limit {
|
||||
return over;
|
||||
}
|
||||
std::mem::swap(&mut prev2, &mut prev);
|
||||
std::mem::swap(&mut prev, &mut cur);
|
||||
}
|
||||
prev.get(m).copied().unwrap_or(over).min(over)
|
||||
}
|
||||
|
||||
/// Spelling suggestions for a Portuguese word, best first, at most `max`, in the word's letter case.
|
||||
pub fn suggest(word: &str, max: usize) -> Vec<String> {
|
||||
suggest_in(&super::dictionary(), word, max)
|
||||
}
|
||||
|
||||
/// [`suggest`] against a given dictionary.
|
||||
pub fn suggest_in(d: &AffixDict, word: &str, max: usize) -> Vec<String> {
|
||||
let word = word.trim();
|
||||
let n = word.chars().count();
|
||||
if n == 0 || n > MAX_LETTERS || max == 0 || d.is_empty() || word.contains(char::is_whitespace) {
|
||||
return Vec::new();
|
||||
}
|
||||
let case = casing(word);
|
||||
let input = word.to_lowercase();
|
||||
let chars: Vec<char> = input.chars().collect();
|
||||
let mut col = Collector { d, input: input.clone(), tested: HashSet::new(), found: HashMap::new(), seq: 0 };
|
||||
// 1. Letter case.
|
||||
if d.contains(&input) && word != input {
|
||||
col.keep(input.clone(), COST_CASE);
|
||||
} else if d.contains(&capitalize(&input)) && word != capitalize(&input) {
|
||||
col.keep(capitalize(&input), COST_CASE);
|
||||
}
|
||||
// 2. Accents and other MAP slips.
|
||||
let map = d.map();
|
||||
let alts: Vec<Vec<char>> = chars.iter().map(|&c| alternatives(c, map)).collect();
|
||||
let mut variants = Vec::new();
|
||||
map_variants(&chars, &alts, 0, 0, 0, &mut Vec::with_capacity(n), &mut variants);
|
||||
variants.sort_by_key(|v| v.1);
|
||||
for (v, cost) in &variants {
|
||||
col.offer(v, *cost);
|
||||
}
|
||||
// 3. Common confusions.
|
||||
for (from, to) in d.rep() {
|
||||
if from.is_empty() {
|
||||
continue;
|
||||
}
|
||||
for (i, _) in input.match_indices(from.as_str()) {
|
||||
if let (Some(a), Some(b)) = (input.get(..i), input.get(i + from.len()..)) {
|
||||
col.offer(&format!("{a}{to}{b}"), COST_REP);
|
||||
}
|
||||
}
|
||||
}
|
||||
// 4. A missing space: cheap after a short leading word or before a short trailing one,
|
||||
// dearer between two longer words, never leaving a stray short half.
|
||||
for (i, _) in input.char_indices().skip(1) {
|
||||
let (Some(a), Some(b)) = (input.get(..i), input.get(i..)) else { continue };
|
||||
let long = |p: &str| p.chars().count() >= 3;
|
||||
let cost = if (LEADING_WORDS.contains(&a) && long(b)) || (long(a) && TRAILING_WORDS.contains(&b)) {
|
||||
COST_SPLIT
|
||||
} else if long(a) && long(b) {
|
||||
COST_SPLIT_RARE
|
||||
} else {
|
||||
continue;
|
||||
};
|
||||
col.offer(&format!("{a} {b}"), cost);
|
||||
}
|
||||
// 5. One typo.
|
||||
let mut letters: Vec<char> = d.try_chars().chars().filter(|c| c.is_lowercase()).collect();
|
||||
if letters.is_empty() {
|
||||
letters = "aeiousrntmdlcpvgqbzfhjxçãõáéíóúâêôkw".chars().collect();
|
||||
}
|
||||
let s = |v: &[char]| v.iter().collect::<String>();
|
||||
for i in 0..n.saturating_sub(1) {
|
||||
if chars.get(i) != chars.get(i + 1) {
|
||||
let mut v = chars.clone();
|
||||
v.swap(i, i + 1);
|
||||
col.offer(&s(&v), COST_SWAP);
|
||||
}
|
||||
}
|
||||
for i in 0..n {
|
||||
let mut v = chars.clone();
|
||||
v.remove(i);
|
||||
col.offer(&s(&v), COST_DROP_OR_ADD);
|
||||
}
|
||||
for i in 0..=n {
|
||||
for &c in &letters {
|
||||
let mut v = chars.clone();
|
||||
v.insert(i, c);
|
||||
col.offer(&s(&v), COST_DROP_OR_ADD);
|
||||
}
|
||||
}
|
||||
for i in 0..n {
|
||||
for &c in &letters {
|
||||
if chars.get(i) != Some(&c) {
|
||||
let mut v = chars.clone();
|
||||
if let Some(x) = v.get_mut(i) {
|
||||
*x = c;
|
||||
}
|
||||
col.offer(&s(&v), COST_MISTYPE);
|
||||
}
|
||||
}
|
||||
}
|
||||
// 6. Further off: scan the stems that start like the word.
|
||||
if col.count_within(COST_MISTYPE) < max {
|
||||
scan(&mut col, &chars);
|
||||
}
|
||||
let mut found: Vec<(String, (u32, usize))> = col.found.into_iter().collect();
|
||||
// Among equal costs, the change nearest the word's start is the likelier intent
|
||||
// (`saúde` before `saudê`: both add an accent, but at the first difference).
|
||||
let diff_at = |text: &str| chars.iter().zip(text.chars()).position(|(a, b)| *a != b).unwrap_or(n);
|
||||
found.sort_by_key(|(text, (cost, seq))| (*cost, diff_at(text), *seq));
|
||||
let mut out: Vec<String> = Vec::new();
|
||||
for (text, _) in found {
|
||||
let t = match case {
|
||||
Casing::Initial => capitalize(&text),
|
||||
Casing::Upper if n > 1 => text.to_uppercase(),
|
||||
_ => text,
|
||||
};
|
||||
if t != word && !out.contains(&t) {
|
||||
out.push(t);
|
||||
}
|
||||
if out.len() >= max {
|
||||
break;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Round 6: rank the stems starting with the word's first letter (or a MAP relative of it, either
|
||||
/// case) by shared letter pairs, then keep the forms of the best within two typos.
|
||||
fn scan(col: &mut Collector, chars: &[char]) {
|
||||
let d = col.d;
|
||||
let map = d.map();
|
||||
let Some(&first) = chars.first() else { return };
|
||||
let folded: Vec<char> = chars.iter().map(|&c| fold(c, map)).collect();
|
||||
let pairs: HashSet<(char, char)> = std::iter::once(' ').chain(folded.iter().copied()).zip(folded.iter().copied()).collect();
|
||||
let mut firsts: Vec<char> = std::iter::once(first).chain(alternatives(first, map)).collect();
|
||||
let upper: Vec<char> = firsts.iter().flat_map(|c| c.to_uppercase()).collect();
|
||||
firsts.extend(upper);
|
||||
firsts.sort_unstable();
|
||||
firsts.dedup();
|
||||
let n = folded.len();
|
||||
let mut best: Vec<(u32, &str)> = Vec::new();
|
||||
for f in firsts {
|
||||
let prefix = f.to_string();
|
||||
for stem in d.stems_with_prefix(&prefix) {
|
||||
let sf: Vec<char> = stem.chars().map(|c| fold(c, map)).collect();
|
||||
let shared = std::iter::once(' ').chain(sf.iter().copied()).zip(sf.iter().copied()).filter(|p| pairs.contains(p)).count();
|
||||
if shared < 2 {
|
||||
continue;
|
||||
}
|
||||
// Shared pairs, less the stem's letters the word doesn't explain, less a length gap
|
||||
// beyond what an ending explains.
|
||||
let extra = sf.len().saturating_sub(shared);
|
||||
let gap = sf.len().abs_diff(n).saturating_sub(4);
|
||||
let score = (shared * 4).saturating_sub(extra * 3 + gap * 2) as u32;
|
||||
if score == 0 {
|
||||
continue;
|
||||
}
|
||||
if best.len() < SCAN_STEMS {
|
||||
best.push((score, stem));
|
||||
} else if let Some(min) = best.iter_mut().min_by_key(|b| b.0)
|
||||
&& score > min.0
|
||||
{
|
||||
*min = (score, stem);
|
||||
}
|
||||
}
|
||||
}
|
||||
best.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(b.1)));
|
||||
let related = |a: char, b: char| fold(a, map) == fold(b, map);
|
||||
let limit = 4;
|
||||
for (_, stem) in best {
|
||||
for form in d.forms(stem) {
|
||||
let lf: Vec<char> = form.to_lowercase().chars().collect();
|
||||
if lf.len().abs_diff(chars.len()) > 2 {
|
||||
continue;
|
||||
}
|
||||
let dist = weighted_distance(&lf, chars, &related, limit);
|
||||
if dist <= limit && lf.as_slice() != chars {
|
||||
col.keep(form, COST_SCAN + dist * 3);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn has(word: &str, want: &str) {
|
||||
let s = suggest(word, 8);
|
||||
assert!(s.iter().any(|x| x == want), "{word} → {s:?}, want {want}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn portuguese_misspellings_get_the_right_suggestion() {
|
||||
crate::pt::test_dict::install();
|
||||
has("coraçao", "coração");
|
||||
has("saude", "saúde");
|
||||
has("seje", "seja");
|
||||
has("excessão", "exceção");
|
||||
has("previlégio", "privilégio");
|
||||
has("prsidente", "presidente");
|
||||
has("provavel", "provável");
|
||||
has("trbalho", "trabalho");
|
||||
has("nãoé", "não é");
|
||||
has("paraela", "para ela");
|
||||
has("brasília", "Brasília");
|
||||
has("Coraçao", "Coração");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_likeliest_fix_comes_first() {
|
||||
crate::pt::test_dict::install();
|
||||
assert_eq!(suggest("coraçao", 3).first().map(String::as_str), Some("coração"));
|
||||
assert_eq!(
|
||||
suggest("saude", 3).first().map(String::as_str),
|
||||
Some("saúde"),
|
||||
"an accent at the first difference beats one further right (saudê is a word too)"
|
||||
);
|
||||
assert_eq!(suggest("seje", 3).first().map(String::as_str), Some("seja"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn edge_cases_stay_empty_or_bounded() {
|
||||
crate::pt::test_dict::install();
|
||||
assert!(suggest("", 5).is_empty());
|
||||
assert!(suggest("bem", 0).is_empty());
|
||||
assert!(suggest(&"a".repeat(41), 5).is_empty());
|
||||
assert!(suggest("duas palavras", 5).is_empty());
|
||||
assert!(suggest_in(&crate::affix::AffixDict::default(), "coraçao", 5).is_empty());
|
||||
assert!(suggest("coração", 5).len() <= 5);
|
||||
assert!(!suggest("coração", 5).contains(&"coração".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn distance_weighs_accents_half() {
|
||||
let map = vec![vec!['a', 'á', 'à', 'â', 'ã'], vec!['c', 'ç']];
|
||||
let related = |a: char, b: char| fold(a, &map) == fold(b, &map);
|
||||
let c = |s: &str| s.chars().collect::<Vec<_>>();
|
||||
assert_eq!(weighted_distance(&c("coraçao"), &c("coração"), &related, 9), 1);
|
||||
assert_eq!(weighted_distance(&c("coracao"), &c("coração"), &related, 9), 2, "ç and ã are two accent slips");
|
||||
assert_eq!(weighted_distance(&c("coro"), &c("coro"), &related, 9), 0);
|
||||
assert_eq!(weighted_distance(&c("bem"), &c("bom"), &related, 9), 2);
|
||||
assert_eq!(weighted_distance(&c("bem"), &c("bemte"), &related, 9), 4);
|
||||
assert_eq!(weighted_distance(&c("bem"), &c("ver"), &related, 2), 3);
|
||||
assert_eq!(weighted_distance(&c(""), &c("ab"), &related, 9), 4);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn suggestions_are_fast_enough() {
|
||||
crate::pt::test_dict::install();
|
||||
// Warm the dictionary, then time a worst case (a long word nothing near): the scan round.
|
||||
let _ = suggest("bem", 1);
|
||||
let t = std::time::Instant::now();
|
||||
for w in ["xqzvweportuguespalavra", "prsidentes", "paralelepípedo"] {
|
||||
let _ = suggest(w, 6);
|
||||
}
|
||||
let ms = t.elapsed().as_secs_f64() * 1000.0;
|
||||
eprintln!("3 suggestion calls: {ms:.0} ms");
|
||||
assert!(ms < 20_000.0, "{ms} ms");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
//! A tiny synthetic Portuguese Hunspell dictionary for tests (written for these tests; not taken
|
||||
//! from any real dictionary). Real dictionaries are installed on the system, never bundled.
|
||||
//!
|
||||
//! It exercises what VERO-like dictionaries use: `FLAG UTF-8` (multi-byte flag characters —
|
||||
//! `F` marks a forbidden spelling), continuation classes (`provável` → `prováveis`), the
|
||||
//! suggestion tables (`TRY`, `MAP` for accents, `REP` for common confusions) and one prefix.
|
||||
|
||||
pub const AFF: &str = "SET UTF-8
|
||||
FLAG UTF-8
|
||||
FORBIDDENWORD F
|
||||
TRY aeiousrntmdlcpvgqbzfhjxçãõáéíóúâêôkw
|
||||
MAP 6
|
||||
MAP aáàâã
|
||||
MAP eéê
|
||||
MAP ií
|
||||
MAP oóôõ
|
||||
MAP uú
|
||||
MAP cç
|
||||
REP 4
|
||||
REP seje seja
|
||||
REP excessão exceção
|
||||
REP previlégio privilégio
|
||||
REP analize análise
|
||||
SFX A Y 2
|
||||
SFX A a ava a
|
||||
SFX A a ando a
|
||||
SFX B N 1
|
||||
SFX B 0 ável/C .
|
||||
SFX C N 1
|
||||
SFX C vel veis vel
|
||||
PFX X Y 1
|
||||
PFX X 0 des .
|
||||
";
|
||||
|
||||
/// Stems (the count line first, as in Hunspell files).
|
||||
pub const DIC: &str = "76
|
||||
coração
|
||||
saudade
|
||||
saudê
|
||||
pássaro
|
||||
saúde
|
||||
árvore
|
||||
escrever
|
||||
computador
|
||||
praia
|
||||
conhecer
|
||||
Brasília
|
||||
São
|
||||
Paulo
|
||||
guarda
|
||||
roupa
|
||||
mail
|
||||
marido
|
||||
ex
|
||||
vice
|
||||
presidente
|
||||
autoestima
|
||||
frequência
|
||||
frequencia/F
|
||||
exceção
|
||||
privilégio
|
||||
análise
|
||||
seja
|
||||
trabalho
|
||||
canta/A
|
||||
prov/B
|
||||
cobrir/X
|
||||
avião
|
||||
pássaros
|
||||
amanhã
|
||||
vocês
|
||||
coragem
|
||||
parabéns
|
||||
está
|
||||
ele
|
||||
disse
|
||||
existem
|
||||
bem
|
||||
não
|
||||
sim
|
||||
que
|
||||
com
|
||||
para
|
||||
dos
|
||||
das
|
||||
pela
|
||||
pelo
|
||||
muito
|
||||
mais
|
||||
como
|
||||
porque
|
||||
aqui
|
||||
também
|
||||
já
|
||||
ser
|
||||
ter
|
||||
ir
|
||||
ver
|
||||
dar
|
||||
e
|
||||
o
|
||||
é
|
||||
água
|
||||
ela
|
||||
da
|
||||
do
|
||||
de
|
||||
em
|
||||
no
|
||||
na
|
||||
um
|
||||
uma
|
||||
";
|
||||
|
||||
/// Make [`AFF`]/[`DIC`] the Portuguese dictionary (every test that needs one calls this; they all
|
||||
/// install the same one, so tests running in parallel agree).
|
||||
pub fn install() {
|
||||
if let Err(e) = super::install_dictionary(AFF, DIC) {
|
||||
panic!("test dictionary: {e}");
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user