From 3090bcaef0bd109925880f273d530c9458f541ba Mon Sep 17 00:00:00 2001 From: Eduardo Soares Date: Mon, 12 Oct 2026 08:43:24 -0300 Subject: [PATCH] Portuguese proofing: pt-BR spelling, suggestions, grammar and hyphenation (storytold/wordcraft#100) (storytold/wordcraft#797) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * M15: Polish interface, and Polish spelling, grammar and hyphenation Interface - Polish is the tenth interface language ("Polski", pl.tsv, 1,045 entries); pl, pl-PL and pl_PL.UTF-8 system locales select it automatically. - Right-click menu labels now go through translation in every language. - Review > Language is a menu (English US/UK, Polski, Do not check spelling or grammar); the status bar shows the language at the caret. - With a Polish interface, new blank documents and opened .txt/.md files default to pl-PL. Proofing - Each run is checked in its own language (w:lang, resolved through styles): squiggles, F7, review.issues, suggestions and hyphenation. Text in languages without proofing tools is no longer flagged as English. - Polish spelling from the SJP.PL Hunspell dictionary (Apache-2.0 option), read by a new Rust affix engine (crates/proof/src/affix.rs); 875 KB asset, ~4.5M word forms. Regenerate with the pl_dictionary example. - Polish suggestions: case, missing diacritics, rz/ż, ó/u, ch/h confusions, missing spaces (napewno -> na pewno), edit distance. - Conservative Polish grammar rules: repeated words, spacing, doubled punctuation, sentence capitals, missing comma before że/który/ponieważ/ aby/żeby/jeśli…, we/ze forms. - Polish hyphenation with the hyph-utf8 hyph-pl.tex patterns (MIT option); pattern keys widened to 20 letters, TeX exceptions parsed; the heuristic no longer treats ł, ż, ś… as vowels. review.language validates and normalizes tags, can set noProof alone and the document default; format.state reports lang and noProof. Tests for every part, docs and ATTRIBUTION.md/NOTICE updated; cargo xtask ci passes (fmt, clippy, test, assets, layers, wasm). Co-Authored-By: Claude Opus 5.5 * Layout: rustfmt the test module order left unformatted by storytold/wordcraft#391 cargo fmt --check failed on main after storytold/wordcraft#391 added mod tests_grid. Co-Authored-By: Claude Opus 5.5 * Review fixes for storytold/wordcraft#346: read Polish spelling from the installed Hunspell dictionary - Polish spelling uses the Hunspell pl_PL dictionary installed on the system (OS packages, Homebrew, LibreOffice, ~/.local/share/hunspell, or WORDCRAFT_DICTIONARIES) instead of a bundled 870 KB SJP.PL file; without one, Polish text gets no spelling marks, like Word without the language's proofing tools. .aff/.dic are size-capped and decoded from UTF-8 or ISO 8859-2 (crates/proof/src/hunspell.rs). - The dictionary can be replaced at runtime (install_dictionary) or re-searched (reload_dictionary); an empty one accepts every word. - Tests use small synthetic dictionaries written for them, so they don't depend on what the machine has installed. - The bundled pl.dic, its ATTRIBUTION/NOTICE entries and the generator example are removed; docs describe the installed dictionary. Co-Authored-By: Claude Opus 5.5 * storytold/wordcraft#100: affix — accept FLAG UTF-8 and DEFAULT flag modes Per-character flag modes are what the reader already parses; only the multi-character encodings (long, num) stay errors. VERO pt_BR declares FLAG UTF-8. * storytold/wordcraft#100: affix — widen flag masks to u128 (128 flag capacity) VERO pt_BR needs 106 distinct flags; the u64 bitmask capped at 64. Serialization is unaffected: WCAFF1 stores flag characters and re-derives the masks on load. * storytold/wordcraft#100: affix — support FORBIDDENWORD VERO pt_BR marks spellings it must reject with a forbidden flag (ý). Stems carrying it are listed but never correct and never inflect. * storytold/wordcraft#100: affix — continuation classes in check SFX A 0 able/B: the form derived by A carries flag B, licensing a further suffix of class B (workable -> workables) while the bare stem cannot take one. form_has_flag walks the chain depth-bounded (MAX_CHAIN = 3); with no continuations declared it degenerates to the stem lookup, so existing dictionaries are unaffected. VERO pt_BR uses 26 continuation rules. * storytold/wordcraft#100: affix — continuation classes in forms Enumerate chained suffix forms (workable -> workables) under the same MAX_CHAIN bound the checker walks, so every form the checker accepts is listed and vice versa. * storytold/wordcraft#100: affix — round-trip continuations and FORBIDDENWORD through WCAFF1 The compact format keeps its WCAFF1 magic: continuation classes ride on the rule's appended field (as in the .aff source) and the forbidden flag becomes a FORBIDDENWORD line. Old files read as before. * storytold/wordcraft#100: affix — strip a UTF-8 BOM from .aff/.dic input VERO's files open with a BOM; glued to the count line it parsed as a stem (one garbage entry per load) and to SET it hid the directive. * storytold/wordcraft#782: affix — the module doc still rejected what the code now reads Continuation classes and FORBIDDENWORD are supported; the header said Rejected for both. * storytold/wordcraft#100: pt — Portuguese spelling, following the Polish module ProofLang::Pt resolves pt/pt-BR/pt-PT/por tags; the pt module reads the installed Hunspell pt_BR dictionary (VERO) through the affix engine — FLAG UTF-8 flags, continuation classes and FORBIDDENWORD spellings all exercised by the synthetic test dictionary. Hyphenated compounds are checked part by part (guarda-roupa, ex-marido); without a dictionary installed, Portuguese text gets no marks. Spelling message in Portuguese; suggestions and grammar land next. * storytold/wordcraft#100: pt — Portuguese suggestions Rounds of rising cost, driven by the dictionary's own tables: missing accents from MAP (coraçao -> coração, saude -> saúde), the REP confusions (seje -> seja, excessão -> exceção), a missing space after a short function word (nãoé -> não é, dagua -> da água), one typo over TRY, and an accent-folding stem scan — the Polish rounds, retuned for Portuguese. * storytold/wordcraft#100: pt — Portuguese grammar flags Conservative rules with Portuguese messages: repeated words (except the emphatic doubles Portuguese allows), space before punctuation and double spaces, sentence capitals knowing the Portuguese abbreviations and dialogue dashes, and doubled punctuation. No comma rule: que opens a clause only sometimes, so a wrong flag would cost more than a missed one. * storytold/wordcraft#100: pt — Portuguese hyphenation hyph-utf8's hyph-pt.tex bundled unmodified (BSD 3-clause, ATTRIBUTION row with source and hash): 426 Liang patterns and 2 exception words; falls back to the generic heuristic outside their reach. * storytold/wordcraft#100: pt — Review › Language offers Português (Brasil), and a pt-br interface writes pt-BR The menu's fourth entry sets the run language; the status bar names Portuguese; editing_lang is now allowed for pt-br (wordcraft-proof proofs it), so new documents under a Portuguese interface are Portuguese by default. * storytold/wordcraft#100: pt — engine test: Portuguese proofing through the review commands review.language pt-BR normalizes the tag, review.issues reports Portuguese spelling and grammar with the lang field, F7 and Change apply a suggestion, Add to Dictionary works for Portuguese. * storytold/wordcraft#100: pt — Portuguese in the mixed-language proofing tests and docs check_text proofs a Portuguese span with the Portuguese message; the never-panic loops and the module docs cover the third language. * storytold/wordcraft#100: docs — Portuguese proofing in the gap, parity and roadmap docs Gap 8 narrowed (English, Polish and Portuguese), localization parity gains a Portuguese row and a revision entry, target-app-parity's Proofing row moves 40% -> 42%, and the ROADMAP progress log records what landed. * storytold/wordcraft#100: hunspell — LibreOffice extension folders follow the language The search paths hardcoded LibreOffice's dict-pl; a pt_BR user whose only dictionary is LibreOffice's dict-pt-BR extension — on Windows the only realistic on-disk source — got silently no spelling. Both the primary-language folder and the dashed full name are now tried, so Polish keeps dict-pl and Portuguese finds dict-pt-BR. * storytold/wordcraft#100: pt — review fixes Suggestion ties break toward the change nearest the word's start (saúde before saudê, proven against a fixture that has both); the Language menu's label is the English source "Portuguese (Brazil)" with translations in all twelve catalogs (as Polish is); the attribution counts pt.pat's 2 exception words; the pt fixture drops a duplicate stem; docs/roadmap.md catches up with the landing; the throwaway spike is gone. * Review fixes for storytold/wordcraft#782: two-suffix chain limit, module doc - MAX_CHAIN = 2 counts suffixes in a row, matching Hunspell's two-suffix limit (the old bound of 3 allowed up to four); test pins that a third is rejected. - Module doc: say probes allocate small scratch strings instead of claiming none. Co-Authored-By: Claude Opus 5.5 * Review fixes for storytold/wordcraft#797: NOTICE for pt.pat, no indexing in the LibreOffice folder helper - NOTICE reproduces hyph-pt.tex's BSD 3-Clause copyright notice, conditions and disclaimer, which the licence requires for binary redistributions. - hunspell::lo_ext_dirs builds its list without indexing (never-crash rule). - Catalogs: main already translates "Portuguese (Brazil)" in every language, so the duplicate rows the branch added are dropped. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: witkowski01 Co-authored-by: Claude Opus 5.5 Co-authored-by: Brandon Thomas --- crates/proof/src/hunspell.rs | 53 ++-- crates/proof/src/hyphen.rs | 60 +++++ crates/proof/src/lang.rs | 15 +- crates/proof/src/lib.rs | 33 ++- crates/proof/src/pt/grammar.rs | 245 +++++++++++++++++ crates/proof/src/pt/mod.rs | 202 ++++++++++++++ crates/proof/src/pt/suggest.rs | 444 +++++++++++++++++++++++++++++++ crates/proof/src/pt/test_dict.rs | 121 +++++++++ 8 files changed, 1151 insertions(+), 22 deletions(-) create mode 100644 crates/proof/src/pt/grammar.rs create mode 100644 crates/proof/src/pt/mod.rs create mode 100644 crates/proof/src/pt/suggest.rs create mode 100644 crates/proof/src/pt/test_dict.rs diff --git a/crates/proof/src/hunspell.rs b/crates/proof/src/hunspell.rs index 2c09b11d..11342084 100644 --- a/crates/proof/src/hunspell.rs +++ b/crates/proof/src/hunspell.rs @@ -26,8 +26,20 @@ pub fn set_dictionary_dir(dir: Option) { *EXTRA_DIR.write().unwrap_or_else(|e| e.into_inner()) = dir; } -/// Directories that may hold Hunspell dictionaries, most specific first. -pub fn search_dirs() -> Vec { +/// The LibreOffice extension directories a Hunspell dictionary name may live in: LibreOffice's +/// IDs are per dictionary — Polish is `dict-pl`, Brazilian Portuguese `dict-pt-BR` — so both +/// the primary-language folder and the dashed full name are tried. +fn lo_ext_dirs(name: &str) -> Vec { + let primary = name.split(['-', '_']).next().unwrap_or(name); + let dashed = name.replace('_', "-"); + let first = format!("dict-{primary}"); + let full = format!("dict-{dashed}"); + if full == first { vec![first] } else { vec![first, full] } +} + +/// Directories that may hold Hunspell dictionaries, most specific first. `lo_exts`, when a +/// language is being searched for, adds LibreOffice's extension folders for it (`dict-pt-BR`). +pub fn search_dirs(lo_exts: Option<&[String]>) -> Vec { let mut dirs: Vec = Vec::new(); if let Some(d) = EXTRA_DIR.read().unwrap_or_else(|e| e.into_inner()).clone() { dirs.push(d); @@ -38,6 +50,9 @@ pub fn search_dirs() -> Vec { if let Some(d) = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()) { dirs.push(PathBuf::from(d)); } + // LibreOffice keeps each language under `…/share/extensions/`. + let lo_dirs = + |prefix: &str| -> Vec { lo_exts.map_or_else(Vec::new, |exts| exts.iter().map(|e| PathBuf::from(prefix).join(e)).collect()) }; let home = std::env::var_os("HOME").filter(|v| !v.is_empty()).map(PathBuf::from); if cfg!(windows) { for var in ["LOCALAPPDATA", "APPDATA"] { @@ -48,7 +63,7 @@ pub fn search_dirs() -> Vec { } for var in ["ProgramFiles", "ProgramFiles(x86)"] { if let Some(base) = std::env::var_os(var).filter(|v| !v.is_empty()).map(PathBuf::from) { - dirs.push(base.join("LibreOffice").join("share").join("extensions").join("dict-pl")); + dirs.extend(lo_dirs(&base.join("LibreOffice").join("share").join("extensions").to_string_lossy())); } } } else if cfg!(target_os = "macos") { @@ -58,30 +73,26 @@ pub fn search_dirs() -> Vec { dirs.push(PathBuf::from("/Library/Spelling")); dirs.push(PathBuf::from("/opt/homebrew/share/hunspell")); dirs.push(PathBuf::from("/usr/local/share/hunspell")); - dirs.push(PathBuf::from("/Applications/LibreOffice.app/Contents/Resources/extensions/dict-pl")); + dirs.extend(lo_dirs("/Applications/LibreOffice.app/Contents/Resources/extensions")); } else { if let Some(h) = &home { dirs.push(h.join(".local/share/hunspell")); } - for d in [ - "/usr/share/hunspell", - "/usr/share/myspell", - "/usr/share/myspell/dicts", - "/usr/local/share/hunspell", - "/usr/local/share/myspell", - "/usr/lib/libreoffice/share/extensions/dict-pl", - "/opt/libreoffice/share/extensions/dict-pl", - ] { + for d in ["/usr/share/hunspell", "/usr/share/myspell", "/usr/share/myspell/dicts", "/usr/local/share/hunspell", "/usr/local/share/myspell"] { dirs.push(PathBuf::from(d)); } + for d in ["/usr/lib/libreoffice/share/extensions", "/opt/libreoffice/share/extensions"] { + dirs.extend(lo_dirs(d)); + } } dirs } /// The first `.aff`/`.dic` pair found for any of `names` (`pl_PL`, `pl`), searching -/// [`search_dirs`]. +/// [`search_dirs`] (including LibreOffice's extension folder for the first name). pub fn find(names: &[&str]) -> Option<(PathBuf, PathBuf)> { - search_dirs().into_iter().find_map(|dir| find_in(&dir, names)) + let lo = names.first().copied().map(lo_ext_dirs); + search_dirs(lo.as_deref()).into_iter().find_map(|dir| find_in(&dir, names)) } /// The first `.aff`/`.dic` pair for any of `names` in `dir`. @@ -146,6 +157,18 @@ mod tests { assert!(decode(b"SET UTF-8\n", b"\xff\xfe").is_err()); } + #[test] + fn libreoffice_extension_dirs_follow_the_language() { + // LibreOffice's IDs are per dictionary: Polish is `dict-pl`, Brazilian Portuguese + // `dict-pt-BR`, so both the primary-language folder and the dashed full name are tried. + assert_eq!(super::lo_ext_dirs("pl_PL"), ["dict-pl", "dict-pl-PL"]); + assert_eq!(super::lo_ext_dirs("pt_BR"), ["dict-pt", "dict-pt-BR"]); + assert_eq!(super::lo_ext_dirs("de-DE"), ["dict-de", "dict-de-DE"]); + let dirs = search_dirs(Some(&super::lo_ext_dirs("pt_BR"))); + assert!(dirs.iter().any(|d| d.ends_with("dict-pt-BR")), "{dirs:?}"); + assert!(search_dirs(None).iter().all(|d| !d.to_string_lossy().contains("dict-")), "no language: no LibreOffice guess"); + } + #[test] fn finds_and_loads_a_pair_in_a_directory() { let dir = std::env::temp_dir().join(format!("wordcraft-hunspell-{}", std::process::id())); diff --git a/crates/proof/src/hyphen.rs b/crates/proof/src/hyphen.rs index c244455a..9919aa07 100644 --- a/crates/proof/src/hyphen.rs +++ b/crates/proof/src/hyphen.rs @@ -29,6 +29,8 @@ static EN_US_DIC: &[u8] = include_bytes!("../../../assets/hyphenation/en-us.dic" static EN_US_PAT: &str = include_str!("../../../assets/hyphenation/en-us.pat"); /// hyph-utf8's `hyph-pl.tex`, unmodified (its licence header is in the file). static PL_PAT: &str = include_str!("../../../assets/hyphenation/pl.pat"); +/// hyph-utf8's `hyph-pt.tex`, unmodified (its BSD 3-clause header is in the file). +static PT_PAT: &str = include_str!("../../../assets/hyphenation/pt.pat"); impl Dictionary { /// The bundled US English dictionary (decoded on first use). @@ -58,6 +60,12 @@ impl Patterns { static P: OnceLock = OnceLock::new(); P.get_or_init(|| Patterns::parse(PL_PAT)) } + + /// The bundled Portuguese patterns and exception words (parsed on first use). + pub fn pt() -> &'static Patterns { + static P: OnceLock = OnceLock::new(); + P.get_or_init(|| Patterns::parse(PT_PAT)) + } } /// Hyphenation limits from the paragraph's Hyphenation settings. @@ -101,6 +109,9 @@ fn core_breaks(core: &[char], lang: ProofLang) -> (Vec, Source) { if lang == ProofLang::Pl { return core_breaks_pl(core); } + if lang == ProofLang::Pt { + return core_breaks_pt(core); + } let dict = Dictionary::en_us(); let s: String = core.iter().collect(); let from_mask = |m: u64, len: usize| (1..len.min(64)).filter(|&i| m & (1 << i) != 0).collect::>(); @@ -140,6 +151,21 @@ fn core_breaks_pl(core: &[char]) -> (Vec, Source) { (heuristic_points_pl(core), Source::Heuristic) } +/// [`core_breaks_pl`] for Portuguese: the bundled patterns, then the generic heuristic. +fn core_breaks_pt(core: &[char]) -> (Vec, Source) { + let pats = Patterns::pt(); + let s: String = core.iter().collect(); + if let Some(p) = pats.exception(&s) { + return (p.to_vec(), Source::Dictionary); + } + if !pats.is_empty() + && let Some(p) = pats.points(core) + { + return (p, Source::Patterns); + } + (heuristic_points(core), Source::Heuristic) +} + /// Unrestricted break points of an English `word` ([`word_breaks_in`] with [`ProofLang::En`]). pub fn word_breaks(word: &str) -> (Vec, Source) { word_breaks_in(word, ProofLang::En) @@ -667,6 +693,40 @@ mod tests { assert_eq!(p.exception("nigdy"), Some(&[2][..])); } + fn pt(w: &str) -> String { + hyphenate_word_in(w, &Limits::default(), ProofLang::Pt) + } + + #[test] + fn portuguese_patterns_load() { + let p = Patterns::pt(); + assert!(p.len() > 300, "{}", p.len()); + // `hard-ware`, `soft-ware` — and the file's last pattern line, a bare `1-`, which the + // reader counts as a (never-matching) exception because it contains a hyphen. + assert_eq!(p.exception_count(), 3); + assert_eq!(p.exception("hardware"), Some(&[4][..])); + assert_eq!(p.exception("software"), Some(&[4][..])); + } + + #[test] + fn portuguese_words_hyphenate_by_portuguese_rules() { + for (w, want) in [ + ("problema", "pro-ble-ma"), + ("coração", "co-ra-ção"), + ("paralelepípedo", "pa-ra-le-le-pí-pe-do"), + ("computador", "com-pu-ta-dor"), + ("escrever", "es-cre-ver"), + ("saudade", "sau-da-de"), + ("extraordinário", "ex-tra-or-di-ná-rio"), + ("automático", "au-to-má-ti-co"), + // One syllable: no break. + ("flor", "flor"), + ("bem", "bem"), + ] { + assert_eq!(pt(w), want, "{w}"); + } + } + #[test] fn polish_words_hyphenate_by_polish_rules() { for (w, want) in [ diff --git a/crates/proof/src/lang.rs b/crates/proof/src/lang.rs index bfd0bea3..344fc52e 100644 --- a/crates/proof/src/lang.rs +++ b/crates/proof/src/lang.rs @@ -13,6 +13,10 @@ pub enum ProofLang { /// Polish (`pl`, `pl-PL`): the installed Hunspell dictionary (when there is one), the Polish grammar rules and the Polish /// hyphenation patterns. Pl, + /// Portuguese (`pt`, `pt-BR`, `pt-PT`): the installed Hunspell dictionary (VERO and + /// friends, when there is one), the Portuguese grammar rules and the Portuguese hyphenation + /// patterns. + Pt, /// A language without proofing tools here (`fr-FR`, `ja-JP`, `x-none`…). Like Word without /// the language's proofing tools: no spelling or grammar marks, so its words are never /// flagged as English misspellings. Hyphenation keeps using the English patterns, as before @@ -22,7 +26,7 @@ pub enum ProofLang { impl ProofLang { /// Every language with a spelling dictionary, in display order. - pub const PROOFED: [ProofLang; 2] = [ProofLang::En, ProofLang::Pl]; + pub const PROOFED: [ProofLang; 3] = [ProofLang::En, ProofLang::Pl, ProofLang::Pt]; /// The proofing language of a BCP 47 tag (`pl-PL`, `en_GB`, `PL`…). Only the primary subtag /// counts. No tag (or an empty one) is English, WordCraft's default editing language. @@ -33,6 +37,8 @@ impl ProofLang { ProofLang::En } else if primary.eq_ignore_ascii_case("pl") || primary.eq_ignore_ascii_case("pol") { ProofLang::Pl + } else if primary.eq_ignore_ascii_case("pt") || primary.eq_ignore_ascii_case("por") { + ProofLang::Pt } else { ProofLang::Other } @@ -43,6 +49,7 @@ impl ProofLang { match self { ProofLang::En => "en", ProofLang::Pl => "pl", + ProofLang::Pt => "pt", ProofLang::Other => "", } } @@ -116,6 +123,9 @@ mod tests { for t in ["pl", "pl-PL", "PL-pl", "pl_PL", "pl_PL.UTF-8", " pl-PL ", "pol"] { assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pl, "{t}"); } + for t in ["pt", "pt-BR", "PT_br", "pt-BR.UTF-8", " pt-PT ", "por"] { + assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pt, "{t}"); + } for t in ["en", "en-US", "en-GB", "EN_au", "en-Latn-US"] { assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::En, "{t}"); } @@ -126,6 +136,9 @@ mod tests { assert_eq!(ProofLang::from_tag(Some("")), ProofLang::En); assert_eq!(ProofLang::from_tag(Some(" ")), ProofLang::En); assert!(ProofLang::Pl.is_proofed() && !ProofLang::Other.is_proofed()); + assert!(ProofLang::Pt.is_proofed() && !ProofLang::Other.is_proofed()); assert_eq!(ProofLang::Pl.code(), "pl"); + assert_eq!(ProofLang::Pt.code(), "pt"); + assert!(ProofLang::PROOFED.contains(&ProofLang::Pt), "every proofed language is listed"); } } diff --git a/crates/proof/src/lib.rs b/crates/proof/src/lib.rs index 32d1aa7e..736d2164 100644 --- a/crates/proof/src/lib.rs +++ b/crates/proof/src/lib.rs @@ -12,9 +12,13 @@ //! spelling marks), read by the affix engine ([`affix`]), suggestions tuned for Polish //! (diacritics, `ż`/`rz`, `u`/`ó`, missing spaces), and Polish grammar rules with Polish //! messages. +//! - **Portuguese** ([`ProofLang::Pt`], module [`pt`]): spelling against the Hunspell `pt_BR` +//! dictionary installed on the system (VERO and friends; not bundled), suggestions tuned for +//! Portuguese (missing accents, `seje`/`seja` confusions, missing spaces), conservative +//! grammar rules with Portuguese messages, and `hyph-utf8`'s Portuguese patterns. //! - **Other languages** ([`ProofLang::Other`]): no spelling or grammar marks, like Word without //! the language's proofing tools. -//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish), heuristic. +//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish, Portuguese), heuristic. //! //! [`check_text`] checks a paragraph whose runs are in different languages. #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] @@ -26,6 +30,7 @@ pub mod hyphen; pub mod lang; pub mod patterns; pub mod pl; +pub mod pt; use std::collections::HashSet; use std::ops::Range; @@ -489,6 +494,7 @@ fn spelling_issue(text: &str, a: usize, b: usize, lang: ProofLang, opts: &Option } let message = match lang { ProofLang::Pl => "Możliwy błąd pisowni", + ProofLang::Pt => "Possível erro de ortografia", _ => "Possible spelling mistake", }; Some(Issue { start: a, end: b, kind: IssueKind::Spelling, message: message.into(), suggestions: Vec::new(), lang }) @@ -504,6 +510,7 @@ pub fn is_correct_in_with(word: &str, lang: ProofLang, opts: &Options) -> bool { match lang { ProofLang::En => is_correct_with(word, opts), ProofLang::Pl => accepted_by_options(word, opts) || pl::is_correct(word), + ProofLang::Pt => accepted_by_options(word, opts) || pt::is_correct(word), ProofLang::Other => true, } } @@ -513,16 +520,19 @@ pub fn suggest_in(word: &str, max: usize, lang: ProofLang) -> Vec { match lang { ProofLang::En => suggest(word, max), ProofLang::Pl => pl::suggest(word, max), + ProofLang::Pt => pt::suggest(word, max), ProofLang::Other => Vec::new(), } } /// Grammar issues in a paragraph of text in `lang`: the English rules, the Polish ones -/// ([`pl::grammar`]), or none for a language WordCraft doesn't proof. +/// ([`pl::grammar`]), the Portuguese ones ([`pt::grammar`]), or none for a language WordCraft +/// doesn't proof. pub fn check_grammar_in(text: &str, lang: ProofLang) -> Vec { match lang { ProofLang::En => check_grammar(text), ProofLang::Pl => pl::grammar::check(text), + ProofLang::Pt => pt::grammar::check(text), ProofLang::Other => Vec::new(), } } @@ -994,6 +1004,12 @@ mod tests { assert_eq!(flagged("helo świat", &check_text("helo świat", &[(5..12, ProofLang::Pl)])), ["helo"]); // A language without tools: nothing flagged there. assert!(check_text("Bonjour mes amis", &[(0..16, ProofLang::Other)]).is_empty()); + // Portuguese checks in their span only, with the Portuguese message. + pt::test_dict::install(); + let t = "Ele disse que coraçao e frequencia existem."; + let v = check_text(t, &[(0..t.len(), ProofLang::Pt)]); + assert_eq!(flagged(t, &v), ["coraçao", "frequencia"], "{v:?}"); + assert!(v.iter().all(|i| i.lang == ProofLang::Pt && i.message == "Possível erro de ortografia"), "{v:?}"); } proptest::proptest! { @@ -1005,7 +1021,7 @@ mod tests { let _ = has_dictionary(Some(&s)); let _ = is_correct_for(&s, Some(&s)); let _ = hyphen::hyphen_points(&s, &hyphen::Limits::default()); - for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Other] { + for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Pt, ProofLang::Other] { let _ = check_spelling_in(&s, lang); let _ = check_grammar_in(&s, lang); let _ = is_correct_in(&s, lang); @@ -1021,9 +1037,14 @@ mod tests { } #[test] - fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..3), 0..6)) { - let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Other]; - let spans: Vec<(Range, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 3])).collect(); + fn portuguese_suggestions_never_panic(s in "[a-záàâãéêíóôõúüçA-ZÁÀÂÃÉÊÍÓÔÕÚÜÇ'’ -]{0,14}") { + let _ = suggest_in(&s, 4, ProofLang::Pt); + } + + #[test] + fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..4), 0..6)) { + let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Pt, ProofLang::Other]; + let spans: Vec<(Range, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 4])).collect(); for i in check_text(&s, &spans) { proptest::prop_assert!(i.start <= i.end && i.end <= s.len()); } diff --git a/crates/proof/src/pt/grammar.rs b/crates/proof/src/pt/grammar.rs new file mode 100644 index 00000000..c5109de7 --- /dev/null +++ b/crates/proof/src/pt/grammar.rs @@ -0,0 +1,245 @@ +//! Portuguese grammar and punctuation checks: cheap, rule-based and tuned to stay quiet when +//! unsure (a missed mistake costs less than a wrong mark on correct text). Messages are in +//! Portuguese. +//! +//! - **Repeated word** (`ele ele`), except interjections and words Portuguese doubles on purpose +//! (`sim sim`, `não não`, `ora ora`). +//! - **Space before punctuation** (`bem ,`) and **double spaces**; emoticons (` :)`) and +//! ellipses are left alone. +//! - **Capital letter at the start of a sentence**, knowing Portuguese abbreviations (`etc.`, +//! `pág.`, `Prof.`, `Dr.`), initials, ordinal numbers (`5. colocado`), ellipses and dialogue +//! dashes (`— Vai — disse`), after which lower case is correct. +//! - **Doubled punctuation** (`,,`, `..`, `;;`). +//! +//! No comma rule (`que` opens clauses only sometimes) and no English rule (such as `a`/`an`) +//! applies to Portuguese text. + +use crate::{Issue, IssueKind, ProofLang, words}; + +/// Words Portuguese repeats on purpose: interjections, onomatopoeia and emphatic doubling. +const ALLOWED_REPEATS: &[&str] = &[ + "sim", "não", "ora", "olha", "já", "bem", "muito", "vai", "lá", "cá", "aí", "ha", "he", "hi", "oh", "ah", "ai", "ui", "uf", "uxa", "oba", "viva", + "psiu", "olá", "tchau", "xau", "adeus", "devagar", +]; + +/// Abbreviations written with a final period after which a sentence goes on in lower case +/// (lower case, without the period). +const ABBREVIATIONS: &[&str] = &[ + "etc", "ex", "pág", "págs", "séc", "sécs", "fl", "av", "vol", "p", "pp", "ed", "tel", "min", "max", "art", "par", "prof", "profa", "dr", "dra", + "sr", "sra", "srta", "eng", "arq", "op", "cit", "cf", "ibid", "id", "vide", "seg", "ter", "qua", "qui", "sex", "sáb", "dom", "jan", "fev", "mar", + "abr", "mai", "jun", "jul", "ago", "set", "out", "nov", "dez", "s.a", "d.c", "a.c", "pg", "ref", "cod", "dept", "univ", "inst", "lat", "ing", + "esp", "fr", "it", "al", "obs", "att", "obs", +]; + +fn issue(start: usize, end: usize, message: String, suggestions: Vec) -> Issue { + Issue { start, end, kind: IssueKind::Grammar, message, suggestions, lang: ProofLang::Pt } +} + +fn capitalize(w: &str) -> String { + crate::affix::capitalize(w) +} + +/// Is `prev`, before a period, an abbreviation, an initial, a number or a Roman numeral (after +/// which a lower-case word doesn't start a sentence)? +fn abbreviation_or_number(prev: &str) -> bool { + prev.chars().count() <= 1 + || prev.chars().any(|c| c.is_ascii_digit()) + || (prev.chars().all(|c| matches!(c, 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M'))) + || ABBREVIATIONS.contains(&prev.to_lowercase().as_str()) +} + +const CLOSING: [char; 8] = ['"', '”', '’', '\'', ')', '»', ']', '\u{1}']; +const OPENING: [char; 8] = ['"', '“', '«', '(', '[', '\'', '‚', '—']; + +/// Does the gap between two words end a sentence: a `.`, `!` or `?` (not an ellipsis, not after +/// an abbreviation), then space, with nothing but quotes and brackets around? +fn ends_sentence(gap: &str, prev: &str) -> bool { + let t = gap.trim_start_matches(CLOSING); + let Some(end) = t.chars().next() else { return false }; + if !matches!(end, '.' | '!' | '?') || t.starts_with("..") { + return false; + } + if end == '.' && abbreviation_or_number(prev) { + return false; + } + let rest = t.trim_start_matches(['.', '!', '?']).trim_start_matches(CLOSING); + rest.starts_with(char::is_whitespace) && rest.trim_start().chars().all(|c| OPENING.contains(&c)) +} + +/// Grammar issues in a paragraph of Portuguese text, in order. +pub fn check(text: &str) -> Vec { + let ws: Vec<(usize, usize)> = words(text); + let mut v = Vec::new(); + repeated_words(text, &ws, &mut v); + spaces(text, &mut v); + sentence_capitals(text, &ws, &mut v); + doubled_punctuation(text, &mut v); + v.sort_by_key(|i| (i.start, i.end)); + v +} + +fn repeated_words(text: &str, ws: &[(usize, usize)], v: &mut Vec) { + for pair in ws.windows(2) { + let [(a0, b0), (a1, b1)] = pair else { continue }; + let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue }; + if !gap.is_empty() + && gap.chars().all(char::is_whitespace) + && w0.chars().all(char::is_alphabetic) + && w0.to_lowercase() == w1.to_lowercase() + && !ALLOWED_REPEATS.contains(&w0.to_lowercase().as_str()) + { + v.push(issue(*a0, *b1, format!("Palavra repetida: \"{w1}\""), vec![w0.to_string()])); + } + } +} + +fn spaces(text: &str, v: &mut Vec) { + for (i, c) in text.char_indices() { + if c != ' ' { + continue; + } + let rest = text.get(i + 1..).unwrap_or(""); + let mut it = rest.chars(); + let Some(p) = it.next() else { continue }; + let after = it.next(); + let flag = match p { + ',' => true, + // `;)` and `;-)` are emoticons. + ';' => !matches!(after, Some(')' | '(' | '-' | 'P' | 'D')), + // ` ...` is an ellipsis, ` .5` a number. + '.' => !matches!(after, Some('.')) && !after.is_some_and(|a| a.is_ascii_digit()), + // ` :)`, ` :-)`, ` :D` and similar are emoticons; ` ?!` is still a stray space. + ':' | '!' | '?' => after.is_none_or(|a| a.is_whitespace() || matches!(a, '"' | '”' | '’' | '!' | '?')), + _ => false, + }; + if flag { + v.push(issue(i, i + 1 + p.len_utf8(), "Espaço antes da pontuação".into(), vec![p.to_string()])); + } + } + for (i, _) in text.match_indices(" ") { + if i > 0 && text.get(..i).is_some_and(|t| t.ends_with(' ')) { + continue; + } + v.push(issue(i, i + 2, "Espaço duplo".into(), vec![" ".into()])); + } +} + +fn sentence_capitals(text: &str, ws: &[(usize, usize)], v: &mut Vec) { + // List items (`– maçãs,`) may start in lower case. + let list_item = text.trim_end().ends_with([',', ';']); + for (k, &(a, b)) in ws.iter().enumerate() { + let Some(w) = text.get(a..b) else { continue }; + let starts = if k == 0 { + let before = text.get(..a).unwrap_or(""); + !list_item && before.chars().all(|c| c.is_whitespace() || OPENING.contains(&c)) + } else { + let Some(&(pa, pb)) = ws.get(k - 1) else { continue }; + ends_sentence(text.get(pb..a).unwrap_or(""), text.get(pa..pb).unwrap_or("")) + }; + if starts && w.chars().next().is_some_and(char::is_lowercase) && w.chars().all(char::is_alphabetic) { + v.push(issue(a, b, "Letra maiúscula no início da frase".into(), vec![capitalize(w)])); + } + } +} + +fn doubled_punctuation(text: &str, v: &mut Vec) { + let bytes = text.as_bytes(); + let mut i = 0; + while let Some(&b) = bytes.get(i) { + if !matches!(b, b',' | b'.' | b';') { + i += 1; + continue; + } + let run = bytes.get(i..).map_or(0, |r| r.iter().take_while(|&&x| x == b).count()); + let flag = match b { + b'.' => run == 2, + _ => run >= 2, + }; + if flag { + let end = i + run; + let suggestions: Vec = match b { + b'.' => vec![".".into(), "…".into()], + _ => vec![char::from(b).to_string()], + }; + v.push(issue(i, end, "Sinal repetido".into(), suggestions)); + } + i += run.max(1); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn messages(t: &str) -> Vec { + check(t).into_iter().map(|i| format!("{}@{}", i.message, &t[i.start..i.end])).collect() + } + + fn clean(t: &str) { + let m = messages(t); + assert!(m.is_empty(), "{t:?}: {m:?}"); + } + + #[test] + fn repeated_words() { + let v = check("Eu eu vi isso e ele ele saiu."); + assert_eq!(v.len(), 2, "{v:?}"); + assert_eq!(v[0].message, "Palavra repetida: \"eu\""); + assert_eq!(v[0].suggestions, ["Eu"]); + assert!(v.iter().all(|i| i.lang == ProofLang::Pt && i.kind == IssueKind::Grammar)); + clean("Sim, sim! Não, não! Ora ora, que surpresa."); + } + + #[test] + fn spaces_and_punctuation() { + let m = messages("Olá , mundo . Tudo bem ?"); + assert!(m.contains(&"Espaço antes da pontuação@ ,".to_string()), "{m:?}"); + assert!(m.contains(&"Espaço antes da pontuação@ .".to_string()), "{m:?}"); + assert!(m.contains(&"Espaço antes da pontuação@ ?".to_string()), "{m:?}"); + assert!(m.contains(&"Espaço duplo@ ".to_string()), "{m:?}"); + clean("Legal :) Muito bom :-) Ou :D Valor .5 e reticências ..."); + let m = messages("Falou,, e saiu.. depois;; fim."); + assert_eq!(m.iter().filter(|x| x.starts_with("Sinal repetido")).count(), 3, "{m:?}"); + clean("Esperei… e esperei... até ele voltar."); + } + + #[test] + fn sentence_start_capitals() { + let m = messages("isso é um teste. a segunda frase! terceira? ok."); + assert_eq!(m.iter().filter(|x| x.starts_with("Letra maiúscula")).count(), 4, "{m:?}"); + assert_eq!(check("ele tem razão.")[0].suggestions, ["Ele"]); + // Abbreviations, initials, ordinals, ellipses and dialogue dashes. + clean("Comprei frutas, etc. e depois saímos."); + clean("Conheci o Dr. Silva na Av. Paulista."); + clean("Zhou ficou em 5. lugar no séc. XX."); + clean("Esperei... e ele apareceu."); + clean("— Vai — disse ele. — Amanhã eu volto."); + clean("Ele disse: \"Volto amanhã\". E saiu."); + // A list item may start in lower case. + clean("maçãs,"); + } + + #[test] + fn correct_portuguese_text_is_clean() { + clean( + "O rato roeu a roupa do rei de Roma. Quem não tem cão caça com gato, \ + e quem tem paciência colhe o que plantou.", + ); + clean("Brasília, 12 de outubro de 2026 — uma data memorável."); + } + + proptest::proptest! { + #[test] + fn never_panics(s in "\\PC{0,80}") { + let _ = check(&s); + } + + #[test] + fn never_panics_on_portuguese_like_text(s in "[a-záàâãéêíóôõúüçA-Z ,.;:!?\"…\\-—]{0,60}") { + for i in check(&s) { + proptest::prop_assert!(i.start <= i.end && i.end <= s.len()); + proptest::prop_assert!(s.is_char_boundary(i.start) && s.is_char_boundary(i.end)); + } + } + } +} diff --git a/crates/proof/src/pt/mod.rs b/crates/proof/src/pt/mod.rs new file mode 100644 index 00000000..be16c74b --- /dev/null +++ b/crates/proof/src/pt/mod.rs @@ -0,0 +1,202 @@ +//! Portuguese proofing. +//! +//! - **Spelling** against the Hunspell `pt_BR` dictionary installed on the system +//! ([`dictionary`], found by [`crate::hunspell`]: `pt_BR.aff`/`pt_BR.dic` — VERO and its +//! derivatives — from the OS, a package manager or LibreOffice; it isn't bundled). Without it +//! Portuguese text gets no spelling marks. Capitalized stems are names (`Brasília`); a sentence +//! may start with any word and headings may be in capitals. Hyphenated compounds are checked +//! part by part (`guarda-roupa`, `ex-marido`), and the user dictionary applies as in English. +//! - **Suggestions** tuned for Portuguese ([`suggest`]): missing accents first (`coraçao` → +//! `coração`, `saude` → `saúde`), then the dictionary's `REP` confusions (`seje` → `seja`), a +//! missing space, one-letter typos, and a dictionary scan for anything further off. +//! - **Grammar** ([`grammar`]): cheap, reliable rules with Portuguese messages. + +pub mod grammar; +mod suggest; +#[cfg(test)] +pub(crate) mod test_dict; + +use std::sync::{Arc, RwLock}; + +use crate::affix::AffixDict; + +pub use suggest::suggest; + +/// The Portuguese dictionary: `None` until first use, then the installed one (or an empty one). +static DICT: RwLock>> = RwLock::new(None); + +/// File names of a Portuguese Hunspell dictionary. +const NAMES: &[&str] = &["pt_BR", "pt-BR", "pt"]; + +/// The Portuguese dictionary: the installed Hunspell `pt_BR` (read on first use), one given with +/// [`install_dictionary`], or an empty one when none is available (Portuguese spelling is off +/// then; see [`has_dictionary`]). +pub fn dictionary() -> Arc { + if let Some(d) = DICT.read().unwrap_or_else(|e| e.into_inner()).as_ref() { + return d.clone(); + } + let loaded = Arc::new(load_installed()); + let mut slot = DICT.write().unwrap_or_else(|e| e.into_inner()); + // Another thread (or an install) may have filled it meanwhile: that one wins. + slot.get_or_insert(loaded).clone() +} + +fn load_installed() -> AffixDict { + let Some((aff, dic)) = crate::hunspell::find(NAMES) else { + log::info!("Portuguese spelling: no Hunspell pt_BR dictionary installed"); + return AffixDict::default(); + }; + crate::hunspell::load(&aff, &dic).unwrap_or_else(|e| { + log::warn!("Portuguese spelling: {e}"); + AffixDict::default() + }) +} + +/// Use this Portuguese dictionary from now on (Hunspell `.aff` and `.dic` text, decoded to +/// UTF-8), for example one the user picked. On error the current dictionary stays. +pub fn install_dictionary(aff: &str, dic: &str) -> Result<(), String> { + let d = AffixDict::from_hunspell(aff, dic, &[])?; + *DICT.write().unwrap_or_else(|e| e.into_inner()) = Some(Arc::new(d)); + Ok(()) +} + +/// Forget the loaded dictionary, so the next check looks for an installed one again (after +/// [`crate::hunspell::set_dictionary_dir`] or a dictionary was installed). +pub fn reload_dictionary() { + *DICT.write().unwrap_or_else(|e| e.into_inner()) = None; +} + +/// Is a Portuguese dictionary available (spelling is checked)? +pub fn has_dictionary() -> bool { + !dictionary().is_empty() +} + +/// Is `c` a letter of the Latin script (the only one the Portuguese dictionary covers)? +fn is_latin_letter(c: char) -> bool { + c.is_alphabetic() && (c < '\u{0250}' || ('\u{1E00}'..='\u{1EFF}').contains(&c)) +} + +/// Is `word` spelled correctly in Portuguese? Numbers, single letters, short words in capitals +/// (acronyms: `USP`, `PT`), words with digits and words in other scripts are accepted, and so +/// is every word when no Portuguese dictionary is installed. +pub fn is_correct(word: &str) -> bool { + is_correct_with(&dictionary(), word) +} + +/// [`is_correct`] against a given dictionary (an empty one accepts everything). +pub fn is_correct_with(d: &AffixDict, word: &str) -> bool { + if d.is_empty() { + return true; // no Portuguese dictionary installed: no spelling marks + } + let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’'); + let w = w.trim_matches(['\'', '’']); + let n = w.chars().count(); + if n <= 1 || w.chars().any(|c| c.is_ascii_digit()) || !w.chars().any(char::is_alphabetic) { + return true; + } + if w.chars().all(|c| !c.is_lowercase()) && n <= 6 { + return true; + } + if !w.chars().any(is_latin_letter) { + return true; // other scripts: no dictionary + } + if w.contains('-') { + return w.split('-').all(|p| p.chars().count() <= 1 || known(d, p)); + } + known(d, w) +} + +/// Is one word (no hyphens) in the dictionary or the user dictionary? +fn known(d: &AffixDict, w: &str) -> bool { + d.check(w) || crate::in_user_dictionary(&w.to_lowercase()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_dictionary_loads() { + test_dict::install(); + let d = dictionary(); + assert!(has_dictionary()); + assert!(d.len() > 50, "{}", d.len()); + assert!(d.rule_count() >= 5, "the test dictionary's affix rules"); + assert!(!d.rep().is_empty() && !d.map().is_empty() && !d.try_chars().is_empty()); + } + + #[test] + fn without_a_dictionary_nothing_is_flagged() { + let none = AffixDict::default(); + for w in ["coraçao", "saude", "xqzv", ""] { + assert!(is_correct_with(&none, w), "{w}"); + } + assert!(install_dictionary("FLAG long\n", "1\nkot\n").is_err(), "unsupported files are refused"); + } + + #[test] + fn portuguese_words_are_correct() { + test_dict::install(); + for w in [ + "coração", + "saudade", + "pássaro", + "saúde", + "árvore", + "escrever", + "computador", + "praia", + "conhecer", + "Brasília", + "São", + "autoestima", + "frequência", + "exceção", + "amanhã", + "vocês", + "coragem", + "parabéns", + "cantava", + "cantando", + "provável", + "prováveis", + "descobrir", + "trabalho", + "CORAGEM", + "USP", + "2026", + "e-mail", + "guarda-roupa", + "ex-marido", + "vice-presidente", + "e", + "não", + ] { + assert!(is_correct(w), "{w}"); + } + } + + #[test] + fn misspellings_are_flagged() { + test_dict::install(); + for w in ["coraçao", "saude", "trbalho", "excrever", "prsidente", "frequencia", "amnha", "cantaando"] { + assert!(!is_correct(w), "{w}"); + } + } + + #[test] + fn forbidden_words_are_rejected() { + test_dict::install(); + // `frequencia/F` is listed with FORBIDDENWORD: the old spelling, never correct. + assert!(!is_correct("frequencia") && !is_correct("Frequencia") && !is_correct("FREQUENCIA")); + assert!(is_correct("frequência")); + } + + #[test] + fn user_dictionary_applies() { + test_dict::install(); + assert!(!is_correct("zxqportuguespalavra")); + crate::add_word("Zxqportuguespalavra"); + assert!(is_correct("zxqportuguespalavra")); + } +} diff --git a/crates/proof/src/pt/suggest.rs b/crates/proof/src/pt/suggest.rs new file mode 100644 index 00000000..314ef39b --- /dev/null +++ b/crates/proof/src/pt/suggest.rs @@ -0,0 +1,444 @@ +//! Portuguese spelling suggestions, best first. +//! +//! Candidates come in rounds of rising cost, each checked against the dictionary: +//! 1. letter case (`brasília` → `Brasília`); +//! 2. missing or wrong accents, from the dictionary's `MAP` groups (`coraçao` → `coração`, +//! `saude` → `saúde`), up to four letters changed — Portuguese typed without accents is the +//! most common slip, so adding one ranks first, while taking away an accent the writer typed +//! ranks after the known confusions; +//! 3. the dictionary's `REP` table of common confusions (`seje` → `seja`, `excessão` → +//! `exceção`, `previlégio` → `privilégio`); +//! 4. a missing space (`nãoé` → `não é`, `paraela` → `para ela`), cheaper when one half is a +//! short function word; +//! 5. one typo: a letter left out, added, swapped with the next or mistyped (`TRY` letters); +//! 6. when that finds fewer than asked for: the stems that start like the word (accents +//! folded) and share the most letter pairs with it, whose forms are ranked by an edit distance +//! in which an accent slip costs half (two typos at most). +//! +//! The input is bounded (40 letters), and so is every round, so a call costs ten-odd thousand +//! dictionary lookups at most. + +use std::collections::{HashMap, HashSet}; + +use crate::affix::{AffixDict, Casing, capitalize, casing}; + +/// Short words often run into the next one (`não é`, `da água`, `para isso`). +const LEADING_WORDS: &[&str] = &[ + "a", "o", "e", "é", "de", "do", "da", "dos", "das", "em", "no", "na", "nos", "nas", "por", "para", "com", "sem", "sob", "sobre", "ao", "aos", + "à", "às", "que", "se", "não", "como", "mas", "mais", "já", "até", "entre", "contra", "perante", "desde", "após", "ante", "durante", "mediante", + "segundo", "conforme", "salvo", "fora", "onde", "quando", "bem", "mal", "pois", "logo", "portanto", "porém", "todavia", "contudo", "assim", + "tão", "tanto", "tudo", "nada", "algo", "cada", "outro", "mesmo", "próprio", "ainda", "tampouco", "quase", "talvez", "nunca", "sempre", +]; +/// Short words often run into the one before (`amo-te`, `deu-lhe`; in writing without the +/// hyphen: `amote`, `deulhe`). +const TRAILING_WORDS: &[&str] = &[ + "me", "te", "se", "lhe", "nos", "vos", "lhes", "o", "a", "os", "as", "ele", "ela", "eles", "elas", "isso", "isto", "aquilo", "mim", "ti", "si", + "eu", "tu", "que", "de", "em", "é", +]; +const MAX_LETTERS: usize = 40; +const MAX_MAP_CHANGES: usize = 4; +const MAX_MAP_VARIANTS: usize = 4096; +/// Stems the scan of round 6 expands. +const SCAN_STEMS: usize = 48; + +const COST_CASE: u32 = 0; +const COST_ADD_DIACRITIC: u32 = 4; +const COST_OTHER_MAP: u32 = 7; +const COST_DROP_DIACRITIC: u32 = 11; +const COST_REP: u32 = 10; +const COST_SPLIT: u32 = 12; +const COST_SPLIT_RARE: u32 = 18; +const COST_SWAP: u32 = 14; +const COST_DROP_OR_ADD: u32 = 15; +const COST_MISTYPE: u32 = 16; +const COST_SCAN: u32 = 20; + +struct Collector<'a> { + d: &'a AffixDict, + /// The word, lower case. + input: String, + tested: HashSet, + /// Suggestion → (cost, order found). + found: HashMap, + seq: usize, +} + +impl Collector<'_> { + /// Check a lower-case candidate (a word, or two separated by a space) and keep it if every + /// word is in the dictionary, as written or capitalized (names). + fn offer(&mut self, cand: &str, cost: u32) { + if cand.is_empty() || cand == self.input || !self.tested.insert(cand.to_string()) { + return; + } + let mut words = Vec::new(); + let mut cost = cost; + for part in cand.split(' ') { + if self.d.contains(part) { + words.push(part.to_string()); + } else { + let c = capitalize(part); + if !self.d.contains(&c) { + return; + } + words.push(c); + // A name ranks after a common word at the same distance (`moça`, then `Moça`). + cost += 1; + } + } + self.keep(words.join(" "), cost); + } + + /// Keep a suggestion known to be correct. + fn keep(&mut self, text: String, cost: u32) { + self.seq += 1; + let seq = self.seq; + let e = self.found.entry(text).or_insert((cost, seq)); + if cost < e.0 { + *e = (cost, seq); + } + } + + fn count_within(&self, cost: u32) -> usize { + self.found.values().filter(|(c, _)| *c <= cost).count() + } +} + +/// The MAP group of each letter (its other members are its likely confusions). +fn alternatives(c: char, map: &[Vec]) -> Vec { + map.iter().filter(|g| g.contains(&c)).flat_map(|g| g.iter().copied()).filter(|&x| x != c).collect() +} + +/// The cost of changing letter `from` to its MAP relative `to`. Adding a missing accent is the +/// commonest slip (Portuguese typed without accents: `coracao`); an accent the writer did type +/// is likelier right, so taking one away costs more than a known confusion from the REP table +/// (`seje` → `seja` before `sejé`). +fn map_cost(from: char, to: char) -> u32 { + match (from.is_ascii(), to.is_ascii()) { + (true, false) => COST_ADD_DIACRITIC, + (false, true) => COST_DROP_DIACRITIC, + _ => COST_OTHER_MAP, + } +} + +/// Every way of changing up to [`MAX_MAP_CHANGES`] letters to a MAP relative, with its cost. +fn map_variants(chars: &[char], alts: &[Vec], i: usize, changes: usize, cost: u32, buf: &mut Vec, out: &mut Vec<(String, u32)>) { + if out.len() >= MAX_MAP_VARIANTS { + return; + } + let Some(&c) = chars.get(i) else { + if changes > 0 { + out.push((buf.iter().collect(), cost)); + } + return; + }; + buf.push(c); + map_variants(chars, alts, i + 1, changes, cost, buf, out); + buf.pop(); + if changes < MAX_MAP_CHANGES { + for &a in alts.get(i).map(Vec::as_slice).unwrap_or_default() { + buf.push(a); + map_variants(chars, alts, i + 1, changes + 1, cost + map_cost(c, a), buf, out); + buf.pop(); + } + } +} + +/// A letter's MAP representative (the first of its group), lower case: `ã` → `a`, `ç` → `c`. +fn fold(c: char, map: &[Vec]) -> char { + let l = c.to_lowercase().next().unwrap_or(c); + map.iter().find(|g| g.contains(&l)).and_then(|g| g.first().copied()).unwrap_or(l) +} + +/// Optimal string alignment distance where a MAP slip (`a`/`á`, `c`/`ç`) costs 1 and any other +/// edit 2; `limit + 1` once it is certainly above `limit`. +fn weighted_distance(a: &[char], b: &[char], related: &dyn Fn(char, char) -> bool, limit: u32) -> u32 { + let m = b.len(); + let over = limit + 1; + let mut prev2 = vec![0u32; m + 1]; + let mut prev: Vec = (0..=m as u32).map(|j| j * 2).collect(); + let mut cur = vec![0u32; m + 1]; + for (i, &ai) in a.iter().enumerate() { + if let Some(c) = cur.get_mut(0) { + *c = (i as u32 + 1) * 2; + } + let mut row_min = u32::MAX; + for (j, &bj) in b.iter().enumerate() { + let sub = if ai == bj { + 0 + } else if related(ai, bj) { + 1 + } else { + 2 + }; + let at = |v: &[u32], k: usize| v.get(k).copied().unwrap_or(u32::MAX / 4); + let mut v = (at(&prev, j + 1) + 2).min(at(&cur, j) + 2).min(at(&prev, j) + sub); + if i > 0 && j > 0 && a.get(i - 1) == Some(&bj) && b.get(j - 1) == Some(&ai) { + v = v.min(at(&prev2, j - 1) + 2); + } + if let Some(c) = cur.get_mut(j + 1) { + *c = v; + } + row_min = row_min.min(v); + } + if m > 0 && row_min > limit { + return over; + } + std::mem::swap(&mut prev2, &mut prev); + std::mem::swap(&mut prev, &mut cur); + } + prev.get(m).copied().unwrap_or(over).min(over) +} + +/// Spelling suggestions for a Portuguese word, best first, at most `max`, in the word's letter case. +pub fn suggest(word: &str, max: usize) -> Vec { + suggest_in(&super::dictionary(), word, max) +} + +/// [`suggest`] against a given dictionary. +pub fn suggest_in(d: &AffixDict, word: &str, max: usize) -> Vec { + let word = word.trim(); + let n = word.chars().count(); + if n == 0 || n > MAX_LETTERS || max == 0 || d.is_empty() || word.contains(char::is_whitespace) { + return Vec::new(); + } + let case = casing(word); + let input = word.to_lowercase(); + let chars: Vec = input.chars().collect(); + let mut col = Collector { d, input: input.clone(), tested: HashSet::new(), found: HashMap::new(), seq: 0 }; + // 1. Letter case. + if d.contains(&input) && word != input { + col.keep(input.clone(), COST_CASE); + } else if d.contains(&capitalize(&input)) && word != capitalize(&input) { + col.keep(capitalize(&input), COST_CASE); + } + // 2. Accents and other MAP slips. + let map = d.map(); + let alts: Vec> = chars.iter().map(|&c| alternatives(c, map)).collect(); + let mut variants = Vec::new(); + map_variants(&chars, &alts, 0, 0, 0, &mut Vec::with_capacity(n), &mut variants); + variants.sort_by_key(|v| v.1); + for (v, cost) in &variants { + col.offer(v, *cost); + } + // 3. Common confusions. + for (from, to) in d.rep() { + if from.is_empty() { + continue; + } + for (i, _) in input.match_indices(from.as_str()) { + if let (Some(a), Some(b)) = (input.get(..i), input.get(i + from.len()..)) { + col.offer(&format!("{a}{to}{b}"), COST_REP); + } + } + } + // 4. A missing space: cheap after a short leading word or before a short trailing one, + // dearer between two longer words, never leaving a stray short half. + for (i, _) in input.char_indices().skip(1) { + let (Some(a), Some(b)) = (input.get(..i), input.get(i..)) else { continue }; + let long = |p: &str| p.chars().count() >= 3; + let cost = if (LEADING_WORDS.contains(&a) && long(b)) || (long(a) && TRAILING_WORDS.contains(&b)) { + COST_SPLIT + } else if long(a) && long(b) { + COST_SPLIT_RARE + } else { + continue; + }; + col.offer(&format!("{a} {b}"), cost); + } + // 5. One typo. + let mut letters: Vec = d.try_chars().chars().filter(|c| c.is_lowercase()).collect(); + if letters.is_empty() { + letters = "aeiousrntmdlcpvgqbzfhjxçãõáéíóúâêôkw".chars().collect(); + } + let s = |v: &[char]| v.iter().collect::(); + for i in 0..n.saturating_sub(1) { + if chars.get(i) != chars.get(i + 1) { + let mut v = chars.clone(); + v.swap(i, i + 1); + col.offer(&s(&v), COST_SWAP); + } + } + for i in 0..n { + let mut v = chars.clone(); + v.remove(i); + col.offer(&s(&v), COST_DROP_OR_ADD); + } + for i in 0..=n { + for &c in &letters { + let mut v = chars.clone(); + v.insert(i, c); + col.offer(&s(&v), COST_DROP_OR_ADD); + } + } + for i in 0..n { + for &c in &letters { + if chars.get(i) != Some(&c) { + let mut v = chars.clone(); + if let Some(x) = v.get_mut(i) { + *x = c; + } + col.offer(&s(&v), COST_MISTYPE); + } + } + } + // 6. Further off: scan the stems that start like the word. + if col.count_within(COST_MISTYPE) < max { + scan(&mut col, &chars); + } + let mut found: Vec<(String, (u32, usize))> = col.found.into_iter().collect(); + // Among equal costs, the change nearest the word's start is the likelier intent + // (`saúde` before `saudê`: both add an accent, but at the first difference). + let diff_at = |text: &str| chars.iter().zip(text.chars()).position(|(a, b)| *a != b).unwrap_or(n); + found.sort_by_key(|(text, (cost, seq))| (*cost, diff_at(text), *seq)); + let mut out: Vec = Vec::new(); + for (text, _) in found { + let t = match case { + Casing::Initial => capitalize(&text), + Casing::Upper if n > 1 => text.to_uppercase(), + _ => text, + }; + if t != word && !out.contains(&t) { + out.push(t); + } + if out.len() >= max { + break; + } + } + out +} + +/// Round 6: rank the stems starting with the word's first letter (or a MAP relative of it, either +/// case) by shared letter pairs, then keep the forms of the best within two typos. +fn scan(col: &mut Collector, chars: &[char]) { + let d = col.d; + let map = d.map(); + let Some(&first) = chars.first() else { return }; + let folded: Vec = chars.iter().map(|&c| fold(c, map)).collect(); + let pairs: HashSet<(char, char)> = std::iter::once(' ').chain(folded.iter().copied()).zip(folded.iter().copied()).collect(); + let mut firsts: Vec = std::iter::once(first).chain(alternatives(first, map)).collect(); + let upper: Vec = firsts.iter().flat_map(|c| c.to_uppercase()).collect(); + firsts.extend(upper); + firsts.sort_unstable(); + firsts.dedup(); + let n = folded.len(); + let mut best: Vec<(u32, &str)> = Vec::new(); + for f in firsts { + let prefix = f.to_string(); + for stem in d.stems_with_prefix(&prefix) { + let sf: Vec = stem.chars().map(|c| fold(c, map)).collect(); + let shared = std::iter::once(' ').chain(sf.iter().copied()).zip(sf.iter().copied()).filter(|p| pairs.contains(p)).count(); + if shared < 2 { + continue; + } + // Shared pairs, less the stem's letters the word doesn't explain, less a length gap + // beyond what an ending explains. + let extra = sf.len().saturating_sub(shared); + let gap = sf.len().abs_diff(n).saturating_sub(4); + let score = (shared * 4).saturating_sub(extra * 3 + gap * 2) as u32; + if score == 0 { + continue; + } + if best.len() < SCAN_STEMS { + best.push((score, stem)); + } else if let Some(min) = best.iter_mut().min_by_key(|b| b.0) + && score > min.0 + { + *min = (score, stem); + } + } + } + best.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(b.1))); + let related = |a: char, b: char| fold(a, map) == fold(b, map); + let limit = 4; + for (_, stem) in best { + for form in d.forms(stem) { + let lf: Vec = form.to_lowercase().chars().collect(); + if lf.len().abs_diff(chars.len()) > 2 { + continue; + } + let dist = weighted_distance(&lf, chars, &related, limit); + if dist <= limit && lf.as_slice() != chars { + col.keep(form, COST_SCAN + dist * 3); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn has(word: &str, want: &str) { + let s = suggest(word, 8); + assert!(s.iter().any(|x| x == want), "{word} → {s:?}, want {want}"); + } + + #[test] + fn portuguese_misspellings_get_the_right_suggestion() { + crate::pt::test_dict::install(); + has("coraçao", "coração"); + has("saude", "saúde"); + has("seje", "seja"); + has("excessão", "exceção"); + has("previlégio", "privilégio"); + has("prsidente", "presidente"); + has("provavel", "provável"); + has("trbalho", "trabalho"); + has("nãoé", "não é"); + has("paraela", "para ela"); + has("brasília", "Brasília"); + has("Coraçao", "Coração"); + } + + #[test] + fn the_likeliest_fix_comes_first() { + crate::pt::test_dict::install(); + assert_eq!(suggest("coraçao", 3).first().map(String::as_str), Some("coração")); + assert_eq!( + suggest("saude", 3).first().map(String::as_str), + Some("saúde"), + "an accent at the first difference beats one further right (saudê is a word too)" + ); + assert_eq!(suggest("seje", 3).first().map(String::as_str), Some("seja")); + } + + #[test] + fn edge_cases_stay_empty_or_bounded() { + crate::pt::test_dict::install(); + assert!(suggest("", 5).is_empty()); + assert!(suggest("bem", 0).is_empty()); + assert!(suggest(&"a".repeat(41), 5).is_empty()); + assert!(suggest("duas palavras", 5).is_empty()); + assert!(suggest_in(&crate::affix::AffixDict::default(), "coraçao", 5).is_empty()); + assert!(suggest("coração", 5).len() <= 5); + assert!(!suggest("coração", 5).contains(&"coração".to_string())); + } + + #[test] + fn distance_weighs_accents_half() { + let map = vec![vec!['a', 'á', 'à', 'â', 'ã'], vec!['c', 'ç']]; + let related = |a: char, b: char| fold(a, &map) == fold(b, &map); + let c = |s: &str| s.chars().collect::>(); + assert_eq!(weighted_distance(&c("coraçao"), &c("coração"), &related, 9), 1); + assert_eq!(weighted_distance(&c("coracao"), &c("coração"), &related, 9), 2, "ç and ã are two accent slips"); + assert_eq!(weighted_distance(&c("coro"), &c("coro"), &related, 9), 0); + assert_eq!(weighted_distance(&c("bem"), &c("bom"), &related, 9), 2); + assert_eq!(weighted_distance(&c("bem"), &c("bemte"), &related, 9), 4); + assert_eq!(weighted_distance(&c("bem"), &c("ver"), &related, 2), 3); + assert_eq!(weighted_distance(&c(""), &c("ab"), &related, 9), 4); + } + + #[test] + fn suggestions_are_fast_enough() { + crate::pt::test_dict::install(); + // Warm the dictionary, then time a worst case (a long word nothing near): the scan round. + let _ = suggest("bem", 1); + let t = std::time::Instant::now(); + for w in ["xqzvweportuguespalavra", "prsidentes", "paralelepípedo"] { + let _ = suggest(w, 6); + } + let ms = t.elapsed().as_secs_f64() * 1000.0; + eprintln!("3 suggestion calls: {ms:.0} ms"); + assert!(ms < 20_000.0, "{ms} ms"); + } +} diff --git a/crates/proof/src/pt/test_dict.rs b/crates/proof/src/pt/test_dict.rs new file mode 100644 index 00000000..e02be3c1 --- /dev/null +++ b/crates/proof/src/pt/test_dict.rs @@ -0,0 +1,121 @@ +//! A tiny synthetic Portuguese Hunspell dictionary for tests (written for these tests; not taken +//! from any real dictionary). Real dictionaries are installed on the system, never bundled. +//! +//! It exercises what VERO-like dictionaries use: `FLAG UTF-8` (multi-byte flag characters — +//! `F` marks a forbidden spelling), continuation classes (`provável` → `prováveis`), the +//! suggestion tables (`TRY`, `MAP` for accents, `REP` for common confusions) and one prefix. + +pub const AFF: &str = "SET UTF-8 +FLAG UTF-8 +FORBIDDENWORD F +TRY aeiousrntmdlcpvgqbzfhjxçãõáéíóúâêôkw +MAP 6 +MAP aáàâã +MAP eéê +MAP ií +MAP oóôõ +MAP uú +MAP cç +REP 4 +REP seje seja +REP excessão exceção +REP previlégio privilégio +REP analize análise +SFX A Y 2 +SFX A a ava a +SFX A a ando a +SFX B N 1 +SFX B 0 ável/C . +SFX C N 1 +SFX C vel veis vel +PFX X Y 1 +PFX X 0 des . +"; + +/// Stems (the count line first, as in Hunspell files). +pub const DIC: &str = "76 +coração +saudade +saudê +pássaro +saúde +árvore +escrever +computador +praia +conhecer +Brasília +São +Paulo +guarda +roupa +mail +marido +ex +vice +presidente +autoestima +frequência +frequencia/F +exceção +privilégio +análise +seja +trabalho +canta/A +prov/B +cobrir/X +avião +pássaros +amanhã +vocês +coragem +parabéns +está +ele +disse +existem +bem +não +sim +que +com +para +dos +das +pela +pelo +muito +mais +como +porque +aqui +também +já +ser +ter +ir +ver +dar +e +o +é +água +ela +da +do +de +em +no +na +um +uma +"; + +/// Make [`AFF`]/[`DIC`] the Portuguese dictionary (every test that needs one calls this; they all +/// install the same one, so tests running in parallel agree). +pub fn install() { + if let Err(e) = super::install_dictionary(AFF, DIC) { + panic!("test dictionary: {e}"); + } +}