Portuguese proofing: pt-BR spelling, suggestions, grammar and hyphenation (storytold/wordcraft#100) (storytold/wordcraft#797)

* M15: Polish interface, and Polish spelling, grammar and hyphenation

Interface
- Polish is the tenth interface language ("Polski", pl.tsv, 1,045 entries);
  pl, pl-PL and pl_PL.UTF-8 system locales select it automatically.
- Right-click menu labels now go through translation in every language.
- Review > Language is a menu (English US/UK, Polski, Do not check spelling
  or grammar); the status bar shows the language at the caret.
- With a Polish interface, new blank documents and opened .txt/.md files
  default to pl-PL.

Proofing
- Each run is checked in its own language (w:lang, resolved through
  styles): squiggles, F7, review.issues, suggestions and hyphenation.
  Text in languages without proofing tools is no longer flagged as English.
- Polish spelling from the SJP.PL Hunspell dictionary (Apache-2.0 option),
  read by a new Rust affix engine (crates/proof/src/affix.rs); 875 KB asset,
  ~4.5M word forms. Regenerate with the pl_dictionary example.
- Polish suggestions: case, missing diacritics, rz/ż, ó/u, ch/h confusions,
  missing spaces (napewno -> na pewno), edit distance.
- Conservative Polish grammar rules: repeated words, spacing, doubled
  punctuation, sentence capitals, missing comma before że/który/ponieważ/
  aby/żeby/jeśli…, we/ze forms.
- Polish hyphenation with the hyph-utf8 hyph-pl.tex patterns (MIT option);
  pattern keys widened to 20 letters, TeX exceptions parsed; the heuristic
  no longer treats ł, ż, ś… as vowels.

review.language validates and normalizes tags, can set noProof alone and
the document default; format.state reports lang and noProof.

Tests for every part, docs and ATTRIBUTION.md/NOTICE updated;
cargo xtask ci passes (fmt, clippy, test, assets, layers, wasm).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Layout: rustfmt the test module order left unformatted by storytold/wordcraft#391

cargo fmt --check failed on main after storytold/wordcraft#391 added mod tests_grid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Review fixes for storytold/wordcraft#346: read Polish spelling from the installed Hunspell dictionary

- Polish spelling uses the Hunspell pl_PL dictionary installed on the
  system (OS packages, Homebrew, LibreOffice, ~/.local/share/hunspell, or
  WORDCRAFT_DICTIONARIES) instead of a bundled 870 KB SJP.PL file;
  without one, Polish text gets no spelling marks, like Word without the
  language's proofing tools. .aff/.dic are size-capped and decoded from
  UTF-8 or ISO 8859-2 (crates/proof/src/hunspell.rs).
- The dictionary can be replaced at runtime (install_dictionary) or
  re-searched (reload_dictionary); an empty one accepts every word.
- Tests use small synthetic dictionaries written for them, so they don't
  depend on what the machine has installed.
- The bundled pl.dic, its ATTRIBUTION/NOTICE entries and the generator
  example are removed; docs describe the installed dictionary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storytold/wordcraft#100: affix — accept FLAG UTF-8 and DEFAULT flag modes

Per-character flag modes are what the reader already parses; only the
multi-character encodings (long, num) stay errors. VERO pt_BR declares
FLAG UTF-8.

* storytold/wordcraft#100: affix — widen flag masks to u128 (128 flag capacity)

VERO pt_BR needs 106 distinct flags; the u64 bitmask capped at 64.
Serialization is unaffected: WCAFF1 stores flag characters and
re-derives the masks on load.

* storytold/wordcraft#100: affix — support FORBIDDENWORD

VERO pt_BR marks spellings it must reject with a forbidden flag (ý).
Stems carrying it are listed but never correct and never inflect.

* storytold/wordcraft#100: affix — continuation classes in check

SFX A 0 able/B: the form derived by A carries flag B, licensing a
further suffix of class B (workable -> workables) while the bare stem
cannot take one. form_has_flag walks the chain depth-bounded
(MAX_CHAIN = 3); with no continuations declared it degenerates to the
stem lookup, so existing dictionaries are unaffected. VERO pt_BR uses
26 continuation rules.

* storytold/wordcraft#100: affix — continuation classes in forms

Enumerate chained suffix forms (workable -> workables) under the same
MAX_CHAIN bound the checker walks, so every form the checker accepts
is listed and vice versa.

* storytold/wordcraft#100: affix — round-trip continuations and FORBIDDENWORD through WCAFF1

The compact format keeps its WCAFF1 magic: continuation classes ride
on the rule's appended field (as in the .aff source) and the forbidden
flag becomes a FORBIDDENWORD line. Old files read as before.

* storytold/wordcraft#100: affix — strip a UTF-8 BOM from .aff/.dic input

VERO's files open with a BOM; glued to the count line it parsed as a
stem (one garbage entry per load) and to SET it hid the directive.

* storytold/wordcraft#782: affix — the module doc still rejected what the code now reads

Continuation classes and FORBIDDENWORD are supported; the header said
Rejected for both.

* storytold/wordcraft#100: pt — Portuguese spelling, following the Polish module

ProofLang::Pt resolves pt/pt-BR/pt-PT/por tags; the pt module reads the
installed Hunspell pt_BR dictionary (VERO) through the affix engine —
FLAG UTF-8 flags, continuation classes and FORBIDDENWORD spellings all
exercised by the synthetic test dictionary. Hyphenated compounds are
checked part by part (guarda-roupa, ex-marido); without a dictionary
installed, Portuguese text gets no marks. Spelling message in
Portuguese; suggestions and grammar land next.

* storytold/wordcraft#100: pt — Portuguese suggestions

Rounds of rising cost, driven by the dictionary's own tables: missing
accents from MAP (coraçao -> coração, saude -> saúde), the REP
confusions (seje -> seja, excessão -> exceção), a missing space after a
short function word (nãoé -> não é, dagua -> da água), one typo over
TRY, and an accent-folding stem scan — the Polish rounds, retuned for
Portuguese.

* storytold/wordcraft#100: pt — Portuguese grammar flags

Conservative rules with Portuguese messages: repeated words (except
the emphatic doubles Portuguese allows), space before punctuation and
double spaces, sentence capitals knowing the Portuguese abbreviations
and dialogue dashes, and doubled punctuation. No comma rule: que opens
a clause only sometimes, so a wrong flag would cost more than a missed
one.

* storytold/wordcraft#100: pt — Portuguese hyphenation

hyph-utf8's hyph-pt.tex bundled unmodified (BSD 3-clause, ATTRIBUTION
row with source and hash): 426 Liang patterns and 2 exception words;
falls back to the generic heuristic outside their reach.

* storytold/wordcraft#100: pt — Review › Language offers Português (Brasil), and a pt-br interface writes pt-BR

The menu's fourth entry sets the run language; the status bar names
Portuguese; editing_lang is now allowed for pt-br (wordcraft-proof
proofs it), so new documents under a Portuguese interface are
Portuguese by default.

* storytold/wordcraft#100: pt — engine test: Portuguese proofing through the review commands

review.language pt-BR normalizes the tag, review.issues reports
Portuguese spelling and grammar with the lang field, F7 and Change
apply a suggestion, Add to Dictionary works for Portuguese.

* storytold/wordcraft#100: pt — Portuguese in the mixed-language proofing tests and docs

check_text proofs a Portuguese span with the Portuguese message; the
never-panic loops and the module docs cover the third language.

* storytold/wordcraft#100: docs — Portuguese proofing in the gap, parity and roadmap docs

Gap 8 narrowed (English, Polish and Portuguese), localization parity
gains a Portuguese row and a revision entry, target-app-parity's
Proofing row moves 40% -> 42%, and the ROADMAP progress log records
what landed.

* storytold/wordcraft#100: hunspell — LibreOffice extension folders follow the language

The search paths hardcoded LibreOffice's dict-pl; a pt_BR user whose
only dictionary is LibreOffice's dict-pt-BR extension — on Windows the
only realistic on-disk source — got silently no spelling. Both the
primary-language folder and the dashed full name are now tried, so
Polish keeps dict-pl and Portuguese finds dict-pt-BR.

* storytold/wordcraft#100: pt — review fixes

Suggestion ties break toward the change nearest the word's start
(saúde before saudê, proven against a fixture that has both); the
Language menu's label is the English source "Portuguese (Brazil)" with
translations in all twelve catalogs (as Polish is); the attribution
counts pt.pat's 2 exception words; the pt fixture drops a duplicate
stem; docs/roadmap.md catches up with the landing; the throwaway spike
is gone.

* Review fixes for storytold/wordcraft#782: two-suffix chain limit, module doc

- MAX_CHAIN = 2 counts suffixes in a row, matching Hunspell's two-suffix limit
  (the old bound of 3 allowed up to four); test pins that a third is rejected.
- Module doc: say probes allocate small scratch strings instead of claiming none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Review fixes for storytold/wordcraft#797: NOTICE for pt.pat, no indexing in the LibreOffice folder helper

- NOTICE reproduces hyph-pt.tex's BSD 3-Clause copyright notice, conditions and
  disclaimer, which the licence requires for binary redistributions.
- hunspell::lo_ext_dirs builds its list without indexing (never-crash rule).
- Catalogs: main already translates "Portuguese (Brazil)" in every language, so the
  duplicate rows the branch added are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: witkowski01 <konradwitkowski01@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
This commit is contained in:
authored and GitHub committed 2026-10-12 07:43:24 -04:00
1 parent 229f9b32d2
commit 3090bcaef0
8 files changed
+1151 -22

No files matched your search

+38 -15
View File
@@ -26,8 +26,20 @@ pub fn set_dictionary_dir(dir: Option<PathBuf>) {
*EXTRA_DIR.write().unwrap_or_else(|e| e.into_inner()) = dir;
}
/// Directories that may hold Hunspell dictionaries, most specific first.
pub fn search_dirs() -> Vec<PathBuf> {
/// The LibreOffice extension directories a Hunspell dictionary name may live in: LibreOffice's
/// IDs are per dictionary — Polish is `dict-pl`, Brazilian Portuguese `dict-pt-BR` — so both
/// the primary-language folder and the dashed full name are tried.
fn lo_ext_dirs(name: &str) -> Vec<String> {
let primary = name.split(['-', '_']).next().unwrap_or(name);
let dashed = name.replace('_', "-");
let first = format!("dict-{primary}");
let full = format!("dict-{dashed}");
if full == first { vec![first] } else { vec![first, full] }
}
/// Directories that may hold Hunspell dictionaries, most specific first. `lo_exts`, when a
/// language is being searched for, adds LibreOffice's extension folders for it (`dict-pt-BR`).
pub fn search_dirs(lo_exts: Option<&[String]>) -> Vec<PathBuf> {
let mut dirs: Vec<PathBuf> = Vec::new();
if let Some(d) = EXTRA_DIR.read().unwrap_or_else(|e| e.into_inner()).clone() {
dirs.push(d);
@@ -38,6 +50,9 @@ pub fn search_dirs() -> Vec<PathBuf> {
if let Some(d) = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()) {
dirs.push(PathBuf::from(d));
}
// LibreOffice keeps each language under `…/share/extensions/<dict-xx>`.
let lo_dirs =
|prefix: &str| -> Vec<PathBuf> { lo_exts.map_or_else(Vec::new, |exts| exts.iter().map(|e| PathBuf::from(prefix).join(e)).collect()) };
let home = std::env::var_os("HOME").filter(|v| !v.is_empty()).map(PathBuf::from);
if cfg!(windows) {
for var in ["LOCALAPPDATA", "APPDATA"] {
@@ -48,7 +63,7 @@ pub fn search_dirs() -> Vec<PathBuf> {
}
for var in ["ProgramFiles", "ProgramFiles(x86)"] {
if let Some(base) = std::env::var_os(var).filter(|v| !v.is_empty()).map(PathBuf::from) {
dirs.push(base.join("LibreOffice").join("share").join("extensions").join("dict-pl"));
dirs.extend(lo_dirs(&base.join("LibreOffice").join("share").join("extensions").to_string_lossy()));
}
}
} else if cfg!(target_os = "macos") {
@@ -58,30 +73,26 @@ pub fn search_dirs() -> Vec<PathBuf> {
dirs.push(PathBuf::from("/Library/Spelling"));
dirs.push(PathBuf::from("/opt/homebrew/share/hunspell"));
dirs.push(PathBuf::from("/usr/local/share/hunspell"));
dirs.push(PathBuf::from("/Applications/LibreOffice.app/Contents/Resources/extensions/dict-pl"));
dirs.extend(lo_dirs("/Applications/LibreOffice.app/Contents/Resources/extensions"));
} else {
if let Some(h) = &home {
dirs.push(h.join(".local/share/hunspell"));
}
for d in [
"/usr/share/hunspell",
"/usr/share/myspell",
"/usr/share/myspell/dicts",
"/usr/local/share/hunspell",
"/usr/local/share/myspell",
"/usr/lib/libreoffice/share/extensions/dict-pl",
"/opt/libreoffice/share/extensions/dict-pl",
] {
for d in ["/usr/share/hunspell", "/usr/share/myspell", "/usr/share/myspell/dicts", "/usr/local/share/hunspell", "/usr/local/share/myspell"] {
dirs.push(PathBuf::from(d));
}
for d in ["/usr/lib/libreoffice/share/extensions", "/opt/libreoffice/share/extensions"] {
dirs.extend(lo_dirs(d));
}
}
dirs
}
/// The first `<name>.aff`/`<name>.dic` pair found for any of `names` (`pl_PL`, `pl`), searching
/// [`search_dirs`].
/// [`search_dirs`] (including LibreOffice's extension folder for the first name).
pub fn find(names: &[&str]) -> Option<(PathBuf, PathBuf)> {
search_dirs().into_iter().find_map(|dir| find_in(&dir, names))
let lo = names.first().copied().map(lo_ext_dirs);
search_dirs(lo.as_deref()).into_iter().find_map(|dir| find_in(&dir, names))
}
/// The first `<name>.aff`/`<name>.dic` pair for any of `names` in `dir`.
@@ -146,6 +157,18 @@ mod tests {
assert!(decode(b"SET UTF-8\n", b"\xff\xfe").is_err());
}
#[test]
fn libreoffice_extension_dirs_follow_the_language() {
// LibreOffice's IDs are per dictionary: Polish is `dict-pl`, Brazilian Portuguese
// `dict-pt-BR`, so both the primary-language folder and the dashed full name are tried.
assert_eq!(super::lo_ext_dirs("pl_PL"), ["dict-pl", "dict-pl-PL"]);
assert_eq!(super::lo_ext_dirs("pt_BR"), ["dict-pt", "dict-pt-BR"]);
assert_eq!(super::lo_ext_dirs("de-DE"), ["dict-de", "dict-de-DE"]);
let dirs = search_dirs(Some(&super::lo_ext_dirs("pt_BR")));
assert!(dirs.iter().any(|d| d.ends_with("dict-pt-BR")), "{dirs:?}");
assert!(search_dirs(None).iter().all(|d| !d.to_string_lossy().contains("dict-")), "no language: no LibreOffice guess");
}
#[test]
fn finds_and_loads_a_pair_in_a_directory() {
let dir = std::env::temp_dir().join(format!("wordcraft-hunspell-{}", std::process::id()));
+60
View File
@@ -29,6 +29,8 @@ static EN_US_DIC: &[u8] = include_bytes!("../../../assets/hyphenation/en-us.dic"
static EN_US_PAT: &str = include_str!("../../../assets/hyphenation/en-us.pat");
/// hyph-utf8's `hyph-pl.tex`, unmodified (its licence header is in the file).
static PL_PAT: &str = include_str!("../../../assets/hyphenation/pl.pat");
/// hyph-utf8's `hyph-pt.tex`, unmodified (its BSD 3-clause header is in the file).
static PT_PAT: &str = include_str!("../../../assets/hyphenation/pt.pat");
impl Dictionary {
/// The bundled US English dictionary (decoded on first use).
@@ -58,6 +60,12 @@ impl Patterns {
static P: OnceLock<Patterns> = OnceLock::new();
P.get_or_init(|| Patterns::parse(PL_PAT))
}
/// The bundled Portuguese patterns and exception words (parsed on first use).
pub fn pt() -> &'static Patterns {
static P: OnceLock<Patterns> = OnceLock::new();
P.get_or_init(|| Patterns::parse(PT_PAT))
}
}
/// Hyphenation limits from the paragraph's Hyphenation settings.
@@ -101,6 +109,9 @@ fn core_breaks(core: &[char], lang: ProofLang) -> (Vec<usize>, Source) {
if lang == ProofLang::Pl {
return core_breaks_pl(core);
}
if lang == ProofLang::Pt {
return core_breaks_pt(core);
}
let dict = Dictionary::en_us();
let s: String = core.iter().collect();
let from_mask = |m: u64, len: usize| (1..len.min(64)).filter(|&i| m & (1 << i) != 0).collect::<Vec<_>>();
@@ -140,6 +151,21 @@ fn core_breaks_pl(core: &[char]) -> (Vec<usize>, Source) {
(heuristic_points_pl(core), Source::Heuristic)
}
/// [`core_breaks_pl`] for Portuguese: the bundled patterns, then the generic heuristic.
fn core_breaks_pt(core: &[char]) -> (Vec<usize>, Source) {
let pats = Patterns::pt();
let s: String = core.iter().collect();
if let Some(p) = pats.exception(&s) {
return (p.to_vec(), Source::Dictionary);
}
if !pats.is_empty()
&& let Some(p) = pats.points(core)
{
return (p, Source::Patterns);
}
(heuristic_points(core), Source::Heuristic)
}
/// Unrestricted break points of an English `word` ([`word_breaks_in`] with [`ProofLang::En`]).
pub fn word_breaks(word: &str) -> (Vec<usize>, Source) {
word_breaks_in(word, ProofLang::En)
@@ -667,6 +693,40 @@ mod tests {
assert_eq!(p.exception("nigdy"), Some(&[2][..]));
}
fn pt(w: &str) -> String {
hyphenate_word_in(w, &Limits::default(), ProofLang::Pt)
}
#[test]
fn portuguese_patterns_load() {
let p = Patterns::pt();
assert!(p.len() > 300, "{}", p.len());
// `hard-ware`, `soft-ware` — and the file's last pattern line, a bare `1-`, which the
// reader counts as a (never-matching) exception because it contains a hyphen.
assert_eq!(p.exception_count(), 3);
assert_eq!(p.exception("hardware"), Some(&[4][..]));
assert_eq!(p.exception("software"), Some(&[4][..]));
}
#[test]
fn portuguese_words_hyphenate_by_portuguese_rules() {
for (w, want) in [
("problema", "pro-ble-ma"),
("coração", "co-ra-ção"),
("paralelepípedo", "pa-ra-le-le-pí-pe-do"),
("computador", "com-pu-ta-dor"),
("escrever", "es-cre-ver"),
("saudade", "sau-da-de"),
("extraordinário", "ex-tra-or-di-ná-rio"),
("automático", "au-to-má-ti-co"),
// One syllable: no break.
("flor", "flor"),
("bem", "bem"),
] {
assert_eq!(pt(w), want, "{w}");
}
}
#[test]
fn polish_words_hyphenate_by_polish_rules() {
for (w, want) in [
+14 -1
View File
@@ -13,6 +13,10 @@ pub enum ProofLang {
/// Polish (`pl`, `pl-PL`): the installed Hunspell dictionary (when there is one), the Polish grammar rules and the Polish
/// hyphenation patterns.
Pl,
/// Portuguese (`pt`, `pt-BR`, `pt-PT`): the installed Hunspell dictionary (VERO and
/// friends, when there is one), the Portuguese grammar rules and the Portuguese hyphenation
/// patterns.
Pt,
/// A language without proofing tools here (`fr-FR`, `ja-JP`, `x-none`…). Like Word without
/// the language's proofing tools: no spelling or grammar marks, so its words are never
/// flagged as English misspellings. Hyphenation keeps using the English patterns, as before
@@ -22,7 +26,7 @@ pub enum ProofLang {
impl ProofLang {
/// Every language with a spelling dictionary, in display order.
pub const PROOFED: [ProofLang; 2] = [ProofLang::En, ProofLang::Pl];
pub const PROOFED: [ProofLang; 3] = [ProofLang::En, ProofLang::Pl, ProofLang::Pt];
/// The proofing language of a BCP 47 tag (`pl-PL`, `en_GB`, `PL`…). Only the primary subtag
/// counts. No tag (or an empty one) is English, WordCraft's default editing language.
@@ -33,6 +37,8 @@ impl ProofLang {
ProofLang::En
} else if primary.eq_ignore_ascii_case("pl") || primary.eq_ignore_ascii_case("pol") {
ProofLang::Pl
} else if primary.eq_ignore_ascii_case("pt") || primary.eq_ignore_ascii_case("por") {
ProofLang::Pt
} else {
ProofLang::Other
}
@@ -43,6 +49,7 @@ impl ProofLang {
match self {
ProofLang::En => "en",
ProofLang::Pl => "pl",
ProofLang::Pt => "pt",
ProofLang::Other => "",
}
}
@@ -116,6 +123,9 @@ mod tests {
for t in ["pl", "pl-PL", "PL-pl", "pl_PL", "pl_PL.UTF-8", " pl-PL ", "pol"] {
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pl, "{t}");
}
for t in ["pt", "pt-BR", "PT_br", "pt-BR.UTF-8", " pt-PT ", "por"] {
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pt, "{t}");
}
for t in ["en", "en-US", "en-GB", "EN_au", "en-Latn-US"] {
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::En, "{t}");
}
@@ -126,6 +136,9 @@ mod tests {
assert_eq!(ProofLang::from_tag(Some("")), ProofLang::En);
assert_eq!(ProofLang::from_tag(Some(" ")), ProofLang::En);
assert!(ProofLang::Pl.is_proofed() && !ProofLang::Other.is_proofed());
assert!(ProofLang::Pt.is_proofed() && !ProofLang::Other.is_proofed());
assert_eq!(ProofLang::Pl.code(), "pl");
assert_eq!(ProofLang::Pt.code(), "pt");
assert!(ProofLang::PROOFED.contains(&ProofLang::Pt), "every proofed language is listed");
}
}
+27 -6
View File
@@ -12,9 +12,13 @@
//! spelling marks), read by the affix engine ([`affix`]), suggestions tuned for Polish
//! (diacritics, `ż`/`rz`, `u`/`ó`, missing spaces), and Polish grammar rules with Polish
//! messages.
//! - **Portuguese** ([`ProofLang::Pt`], module [`pt`]): spelling against the Hunspell `pt_BR`
//! dictionary installed on the system (VERO and friends; not bundled), suggestions tuned for
//! Portuguese (missing accents, `seje`/`seja` confusions, missing spaces), conservative
//! grammar rules with Portuguese messages, and `hyph-utf8`'s Portuguese patterns.
//! - **Other languages** ([`ProofLang::Other`]): no spelling or grammar marks, like Word without
//! the language's proofing tools.
//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish), heuristic.
//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish, Portuguese), heuristic.
//!
//! [`check_text`] checks a paragraph whose runs are in different languages.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -26,6 +30,7 @@ pub mod hyphen;
pub mod lang;
pub mod patterns;
pub mod pl;
pub mod pt;
use std::collections::HashSet;
use std::ops::Range;
@@ -489,6 +494,7 @@ fn spelling_issue(text: &str, a: usize, b: usize, lang: ProofLang, opts: &Option
}
let message = match lang {
ProofLang::Pl => "Możliwy błąd pisowni",
ProofLang::Pt => "Possível erro de ortografia",
_ => "Possible spelling mistake",
};
Some(Issue { start: a, end: b, kind: IssueKind::Spelling, message: message.into(), suggestions: Vec::new(), lang })
@@ -504,6 +510,7 @@ pub fn is_correct_in_with(word: &str, lang: ProofLang, opts: &Options) -> bool {
match lang {
ProofLang::En => is_correct_with(word, opts),
ProofLang::Pl => accepted_by_options(word, opts) || pl::is_correct(word),
ProofLang::Pt => accepted_by_options(word, opts) || pt::is_correct(word),
ProofLang::Other => true,
}
}
@@ -513,16 +520,19 @@ pub fn suggest_in(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
match lang {
ProofLang::En => suggest(word, max),
ProofLang::Pl => pl::suggest(word, max),
ProofLang::Pt => pt::suggest(word, max),
ProofLang::Other => Vec::new(),
}
}
/// Grammar issues in a paragraph of text in `lang`: the English rules, the Polish ones
/// ([`pl::grammar`]), or none for a language WordCraft doesn't proof.
/// ([`pl::grammar`]), the Portuguese ones ([`pt::grammar`]), or none for a language WordCraft
/// doesn't proof.
pub fn check_grammar_in(text: &str, lang: ProofLang) -> Vec<Issue> {
match lang {
ProofLang::En => check_grammar(text),
ProofLang::Pl => pl::grammar::check(text),
ProofLang::Pt => pt::grammar::check(text),
ProofLang::Other => Vec::new(),
}
}
@@ -994,6 +1004,12 @@ mod tests {
assert_eq!(flagged("helo świat", &check_text("helo świat", &[(5..12, ProofLang::Pl)])), ["helo"]);
// A language without tools: nothing flagged there.
assert!(check_text("Bonjour mes amis", &[(0..16, ProofLang::Other)]).is_empty());
// Portuguese checks in their span only, with the Portuguese message.
pt::test_dict::install();
let t = "Ele disse que coraçao e frequencia existem.";
let v = check_text(t, &[(0..t.len(), ProofLang::Pt)]);
assert_eq!(flagged(t, &v), ["coraçao", "frequencia"], "{v:?}");
assert!(v.iter().all(|i| i.lang == ProofLang::Pt && i.message == "Possível erro de ortografia"), "{v:?}");
}
proptest::proptest! {
@@ -1005,7 +1021,7 @@ mod tests {
let _ = has_dictionary(Some(&s));
let _ = is_correct_for(&s, Some(&s));
let _ = hyphen::hyphen_points(&s, &hyphen::Limits::default());
for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Other] {
for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Pt, ProofLang::Other] {
let _ = check_spelling_in(&s, lang);
let _ = check_grammar_in(&s, lang);
let _ = is_correct_in(&s, lang);
@@ -1021,9 +1037,14 @@ mod tests {
}
#[test]
fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..3), 0..6)) {
let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Other];
let spans: Vec<(Range<usize>, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 3])).collect();
fn portuguese_suggestions_never_panic(s in "[a-záàâãéêíóôõúüçA-ZÁÀÂÃÉÊÍÓÔÕÚÜÇ'’ -]{0,14}") {
let _ = suggest_in(&s, 4, ProofLang::Pt);
}
#[test]
fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..4), 0..6)) {
let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Pt, ProofLang::Other];
let spans: Vec<(Range<usize>, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 4])).collect();
for i in check_text(&s, &spans) {
proptest::prop_assert!(i.start <= i.end && i.end <= s.len());
}
+245
View File
@@ -0,0 +1,245 @@
//! Portuguese grammar and punctuation checks: cheap, rule-based and tuned to stay quiet when
//! unsure (a missed mistake costs less than a wrong mark on correct text). Messages are in
//! Portuguese.
//!
//! - **Repeated word** (`ele ele`), except interjections and words Portuguese doubles on purpose
//! (`sim sim`, `não não`, `ora ora`).
//! - **Space before punctuation** (`bem ,`) and **double spaces**; emoticons (` :)`) and
//! ellipses are left alone.
//! - **Capital letter at the start of a sentence**, knowing Portuguese abbreviations (`etc.`,
//! `pág.`, `Prof.`, `Dr.`), initials, ordinal numbers (`5. colocado`), ellipses and dialogue
//! dashes (`— Vai — disse`), after which lower case is correct.
//! - **Doubled punctuation** (`,,`, `..`, `;;`).
//!
//! No comma rule (`que` opens clauses only sometimes) and no English rule (such as `a`/`an`)
//! applies to Portuguese text.
use crate::{Issue, IssueKind, ProofLang, words};
/// Words Portuguese repeats on purpose: interjections, onomatopoeia and emphatic doubling.
const ALLOWED_REPEATS: &[&str] = &[
"sim", "não", "ora", "olha", "já", "bem", "muito", "vai", "lá", "cá", "aí", "ha", "he", "hi", "oh", "ah", "ai", "ui", "uf", "uxa", "oba", "viva",
"psiu", "olá", "tchau", "xau", "adeus", "devagar",
];
/// Abbreviations written with a final period after which a sentence goes on in lower case
/// (lower case, without the period).
const ABBREVIATIONS: &[&str] = &[
"etc", "ex", "pág", "págs", "séc", "sécs", "fl", "av", "vol", "p", "pp", "ed", "tel", "min", "max", "art", "par", "prof", "profa", "dr", "dra",
"sr", "sra", "srta", "eng", "arq", "op", "cit", "cf", "ibid", "id", "vide", "seg", "ter", "qua", "qui", "sex", "sáb", "dom", "jan", "fev", "mar",
"abr", "mai", "jun", "jul", "ago", "set", "out", "nov", "dez", "s.a", "d.c", "a.c", "pg", "ref", "cod", "dept", "univ", "inst", "lat", "ing",
"esp", "fr", "it", "al", "obs", "att", "obs",
];
fn issue(start: usize, end: usize, message: String, suggestions: Vec<String>) -> Issue {
Issue { start, end, kind: IssueKind::Grammar, message, suggestions, lang: ProofLang::Pt }
}
fn capitalize(w: &str) -> String {
crate::affix::capitalize(w)
}
/// Is `prev`, before a period, an abbreviation, an initial, a number or a Roman numeral (after
/// which a lower-case word doesn't start a sentence)?
fn abbreviation_or_number(prev: &str) -> bool {
prev.chars().count() <= 1
|| prev.chars().any(|c| c.is_ascii_digit())
|| (prev.chars().all(|c| matches!(c, 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M')))
|| ABBREVIATIONS.contains(&prev.to_lowercase().as_str())
}
const CLOSING: [char; 8] = ['"', '”', '’', '\'', ')', '»', ']', '\u{1}'];
const OPENING: [char; 8] = ['"', '“', '«', '(', '[', '\'', '‚', '—'];
/// Does the gap between two words end a sentence: a `.`, `!` or `?` (not an ellipsis, not after
/// an abbreviation), then space, with nothing but quotes and brackets around?
fn ends_sentence(gap: &str, prev: &str) -> bool {
let t = gap.trim_start_matches(CLOSING);
let Some(end) = t.chars().next() else { return false };
if !matches!(end, '.' | '!' | '?') || t.starts_with("..") {
return false;
}
if end == '.' && abbreviation_or_number(prev) {
return false;
}
let rest = t.trim_start_matches(['.', '!', '?']).trim_start_matches(CLOSING);
rest.starts_with(char::is_whitespace) && rest.trim_start().chars().all(|c| OPENING.contains(&c))
}
/// Grammar issues in a paragraph of Portuguese text, in order.
pub fn check(text: &str) -> Vec<Issue> {
let ws: Vec<(usize, usize)> = words(text);
let mut v = Vec::new();
repeated_words(text, &ws, &mut v);
spaces(text, &mut v);
sentence_capitals(text, &ws, &mut v);
doubled_punctuation(text, &mut v);
v.sort_by_key(|i| (i.start, i.end));
v
}
fn repeated_words(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
for pair in ws.windows(2) {
let [(a0, b0), (a1, b1)] = pair else { continue };
let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue };
if !gap.is_empty()
&& gap.chars().all(char::is_whitespace)
&& w0.chars().all(char::is_alphabetic)
&& w0.to_lowercase() == w1.to_lowercase()
&& !ALLOWED_REPEATS.contains(&w0.to_lowercase().as_str())
{
v.push(issue(*a0, *b1, format!("Palavra repetida: \"{w1}\""), vec![w0.to_string()]));
}
}
}
fn spaces(text: &str, v: &mut Vec<Issue>) {
for (i, c) in text.char_indices() {
if c != ' ' {
continue;
}
let rest = text.get(i + 1..).unwrap_or("");
let mut it = rest.chars();
let Some(p) = it.next() else { continue };
let after = it.next();
let flag = match p {
',' => true,
// `;)` and `;-)` are emoticons.
';' => !matches!(after, Some(')' | '(' | '-' | 'P' | 'D')),
// ` ...` is an ellipsis, ` .5` a number.
'.' => !matches!(after, Some('.')) && !after.is_some_and(|a| a.is_ascii_digit()),
// ` :)`, ` :-)`, ` :D` and similar are emoticons; ` ?!` is still a stray space.
':' | '!' | '?' => after.is_none_or(|a| a.is_whitespace() || matches!(a, '"' | '”' | '’' | '!' | '?')),
_ => false,
};
if flag {
v.push(issue(i, i + 1 + p.len_utf8(), "Espaço antes da pontuação".into(), vec![p.to_string()]));
}
}
for (i, _) in text.match_indices(" ") {
if i > 0 && text.get(..i).is_some_and(|t| t.ends_with(' ')) {
continue;
}
v.push(issue(i, i + 2, "Espaço duplo".into(), vec![" ".into()]));
}
}
fn sentence_capitals(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
// List items (`– maçãs,`) may start in lower case.
let list_item = text.trim_end().ends_with([',', ';']);
for (k, &(a, b)) in ws.iter().enumerate() {
let Some(w) = text.get(a..b) else { continue };
let starts = if k == 0 {
let before = text.get(..a).unwrap_or("");
!list_item && before.chars().all(|c| c.is_whitespace() || OPENING.contains(&c))
} else {
let Some(&(pa, pb)) = ws.get(k - 1) else { continue };
ends_sentence(text.get(pb..a).unwrap_or(""), text.get(pa..pb).unwrap_or(""))
};
if starts && w.chars().next().is_some_and(char::is_lowercase) && w.chars().all(char::is_alphabetic) {
v.push(issue(a, b, "Letra maiúscula no início da frase".into(), vec![capitalize(w)]));
}
}
}
fn doubled_punctuation(text: &str, v: &mut Vec<Issue>) {
let bytes = text.as_bytes();
let mut i = 0;
while let Some(&b) = bytes.get(i) {
if !matches!(b, b',' | b'.' | b';') {
i += 1;
continue;
}
let run = bytes.get(i..).map_or(0, |r| r.iter().take_while(|&&x| x == b).count());
let flag = match b {
b'.' => run == 2,
_ => run >= 2,
};
if flag {
let end = i + run;
let suggestions: Vec<String> = match b {
b'.' => vec![".".into(), "…".into()],
_ => vec![char::from(b).to_string()],
};
v.push(issue(i, end, "Sinal repetido".into(), suggestions));
}
i += run.max(1);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn messages(t: &str) -> Vec<String> {
check(t).into_iter().map(|i| format!("{}@{}", i.message, &t[i.start..i.end])).collect()
}
fn clean(t: &str) {
let m = messages(t);
assert!(m.is_empty(), "{t:?}: {m:?}");
}
#[test]
fn repeated_words() {
let v = check("Eu eu vi isso e ele ele saiu.");
assert_eq!(v.len(), 2, "{v:?}");
assert_eq!(v[0].message, "Palavra repetida: \"eu\"");
assert_eq!(v[0].suggestions, ["Eu"]);
assert!(v.iter().all(|i| i.lang == ProofLang::Pt && i.kind == IssueKind::Grammar));
clean("Sim, sim! Não, não! Ora ora, que surpresa.");
}
#[test]
fn spaces_and_punctuation() {
let m = messages("Olá , mundo . Tudo bem ?");
assert!(m.contains(&"Espaço antes da pontuação@ ,".to_string()), "{m:?}");
assert!(m.contains(&"Espaço antes da pontuação@ .".to_string()), "{m:?}");
assert!(m.contains(&"Espaço antes da pontuação@ ?".to_string()), "{m:?}");
assert!(m.contains(&"Espaço duplo@ ".to_string()), "{m:?}");
clean("Legal :) Muito bom :-) Ou :D Valor .5 e reticências ...");
let m = messages("Falou,, e saiu.. depois;; fim.");
assert_eq!(m.iter().filter(|x| x.starts_with("Sinal repetido")).count(), 3, "{m:?}");
clean("Esperei… e esperei... até ele voltar.");
}
#[test]
fn sentence_start_capitals() {
let m = messages("isso é um teste. a segunda frase! terceira? ok.");
assert_eq!(m.iter().filter(|x| x.starts_with("Letra maiúscula")).count(), 4, "{m:?}");
assert_eq!(check("ele tem razão.")[0].suggestions, ["Ele"]);
// Abbreviations, initials, ordinals, ellipses and dialogue dashes.
clean("Comprei frutas, etc. e depois saímos.");
clean("Conheci o Dr. Silva na Av. Paulista.");
clean("Zhou ficou em 5. lugar no séc. XX.");
clean("Esperei... e ele apareceu.");
clean("— Vai — disse ele. — Amanhã eu volto.");
clean("Ele disse: \"Volto amanhã\". E saiu.");
// A list item may start in lower case.
clean("maçãs,");
}
#[test]
fn correct_portuguese_text_is_clean() {
clean(
"O rato roeu a roupa do rei de Roma. Quem não tem cão caça com gato, \
e quem tem paciência colhe o que plantou.",
);
clean("Brasília, 12 de outubro de 2026 — uma data memorável.");
}
proptest::proptest! {
#[test]
fn never_panics(s in "\\PC{0,80}") {
let _ = check(&s);
}
#[test]
fn never_panics_on_portuguese_like_text(s in "[a-záàâãéêíóôõúüçA-Z ,.;:!?\"…\\-—]{0,60}") {
for i in check(&s) {
proptest::prop_assert!(i.start <= i.end && i.end <= s.len());
proptest::prop_assert!(s.is_char_boundary(i.start) && s.is_char_boundary(i.end));
}
}
}
}
+202
View File
@@ -0,0 +1,202 @@
//! Portuguese proofing.
//!
//! - **Spelling** against the Hunspell `pt_BR` dictionary installed on the system
//! ([`dictionary`], found by [`crate::hunspell`]: `pt_BR.aff`/`pt_BR.dic` — VERO and its
//! derivatives — from the OS, a package manager or LibreOffice; it isn't bundled). Without it
//! Portuguese text gets no spelling marks. Capitalized stems are names (`Brasília`); a sentence
//! may start with any word and headings may be in capitals. Hyphenated compounds are checked
//! part by part (`guarda-roupa`, `ex-marido`), and the user dictionary applies as in English.
//! - **Suggestions** tuned for Portuguese ([`suggest`]): missing accents first (`coraçao` →
//! `coração`, `saude` → `saúde`), then the dictionary's `REP` confusions (`seje` → `seja`), a
//! missing space, one-letter typos, and a dictionary scan for anything further off.
//! - **Grammar** ([`grammar`]): cheap, reliable rules with Portuguese messages.
pub mod grammar;
mod suggest;
#[cfg(test)]
pub(crate) mod test_dict;
use std::sync::{Arc, RwLock};
use crate::affix::AffixDict;
pub use suggest::suggest;
/// The Portuguese dictionary: `None` until first use, then the installed one (or an empty one).
static DICT: RwLock<Option<Arc<AffixDict>>> = RwLock::new(None);
/// File names of a Portuguese Hunspell dictionary.
const NAMES: &[&str] = &["pt_BR", "pt-BR", "pt"];
/// The Portuguese dictionary: the installed Hunspell `pt_BR` (read on first use), one given with
/// [`install_dictionary`], or an empty one when none is available (Portuguese spelling is off
/// then; see [`has_dictionary`]).
pub fn dictionary() -> Arc<AffixDict> {
if let Some(d) = DICT.read().unwrap_or_else(|e| e.into_inner()).as_ref() {
return d.clone();
}
let loaded = Arc::new(load_installed());
let mut slot = DICT.write().unwrap_or_else(|e| e.into_inner());
// Another thread (or an install) may have filled it meanwhile: that one wins.
slot.get_or_insert(loaded).clone()
}
fn load_installed() -> AffixDict {
let Some((aff, dic)) = crate::hunspell::find(NAMES) else {
log::info!("Portuguese spelling: no Hunspell pt_BR dictionary installed");
return AffixDict::default();
};
crate::hunspell::load(&aff, &dic).unwrap_or_else(|e| {
log::warn!("Portuguese spelling: {e}");
AffixDict::default()
})
}
/// Use this Portuguese dictionary from now on (Hunspell `.aff` and `.dic` text, decoded to
/// UTF-8), for example one the user picked. On error the current dictionary stays.
pub fn install_dictionary(aff: &str, dic: &str) -> Result<(), String> {
let d = AffixDict::from_hunspell(aff, dic, &[])?;
*DICT.write().unwrap_or_else(|e| e.into_inner()) = Some(Arc::new(d));
Ok(())
}
/// Forget the loaded dictionary, so the next check looks for an installed one again (after
/// [`crate::hunspell::set_dictionary_dir`] or a dictionary was installed).
pub fn reload_dictionary() {
*DICT.write().unwrap_or_else(|e| e.into_inner()) = None;
}
/// Is a Portuguese dictionary available (spelling is checked)?
pub fn has_dictionary() -> bool {
!dictionary().is_empty()
}
/// Is `c` a letter of the Latin script (the only one the Portuguese dictionary covers)?
fn is_latin_letter(c: char) -> bool {
c.is_alphabetic() && (c < '\u{0250}' || ('\u{1E00}'..='\u{1EFF}').contains(&c))
}
/// Is `word` spelled correctly in Portuguese? Numbers, single letters, short words in capitals
/// (acronyms: `USP`, `PT`), words with digits and words in other scripts are accepted, and so
/// is every word when no Portuguese dictionary is installed.
pub fn is_correct(word: &str) -> bool {
is_correct_with(&dictionary(), word)
}
/// [`is_correct`] against a given dictionary (an empty one accepts everything).
pub fn is_correct_with(d: &AffixDict, word: &str) -> bool {
if d.is_empty() {
return true; // no Portuguese dictionary installed: no spelling marks
}
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’');
let w = w.trim_matches(['\'', '’']);
let n = w.chars().count();
if n <= 1 || w.chars().any(|c| c.is_ascii_digit()) || !w.chars().any(char::is_alphabetic) {
return true;
}
if w.chars().all(|c| !c.is_lowercase()) && n <= 6 {
return true;
}
if !w.chars().any(is_latin_letter) {
return true; // other scripts: no dictionary
}
if w.contains('-') {
return w.split('-').all(|p| p.chars().count() <= 1 || known(d, p));
}
known(d, w)
}
/// Is one word (no hyphens) in the dictionary or the user dictionary?
fn known(d: &AffixDict, w: &str) -> bool {
d.check(w) || crate::in_user_dictionary(&w.to_lowercase())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_dictionary_loads() {
test_dict::install();
let d = dictionary();
assert!(has_dictionary());
assert!(d.len() > 50, "{}", d.len());
assert!(d.rule_count() >= 5, "the test dictionary's affix rules");
assert!(!d.rep().is_empty() && !d.map().is_empty() && !d.try_chars().is_empty());
}
#[test]
fn without_a_dictionary_nothing_is_flagged() {
let none = AffixDict::default();
for w in ["coraçao", "saude", "xqzv", ""] {
assert!(is_correct_with(&none, w), "{w}");
}
assert!(install_dictionary("FLAG long\n", "1\nkot\n").is_err(), "unsupported files are refused");
}
#[test]
fn portuguese_words_are_correct() {
test_dict::install();
for w in [
"coração",
"saudade",
"pássaro",
"saúde",
"árvore",
"escrever",
"computador",
"praia",
"conhecer",
"Brasília",
"São",
"autoestima",
"frequência",
"exceção",
"amanhã",
"vocês",
"coragem",
"parabéns",
"cantava",
"cantando",
"provável",
"prováveis",
"descobrir",
"trabalho",
"CORAGEM",
"USP",
"2026",
"e-mail",
"guarda-roupa",
"ex-marido",
"vice-presidente",
"e",
"não",
] {
assert!(is_correct(w), "{w}");
}
}
#[test]
fn misspellings_are_flagged() {
test_dict::install();
for w in ["coraçao", "saude", "trbalho", "excrever", "prsidente", "frequencia", "amnha", "cantaando"] {
assert!(!is_correct(w), "{w}");
}
}
#[test]
fn forbidden_words_are_rejected() {
test_dict::install();
// `frequencia/F` is listed with FORBIDDENWORD: the old spelling, never correct.
assert!(!is_correct("frequencia") && !is_correct("Frequencia") && !is_correct("FREQUENCIA"));
assert!(is_correct("frequência"));
}
#[test]
fn user_dictionary_applies() {
test_dict::install();
assert!(!is_correct("zxqportuguespalavra"));
crate::add_word("Zxqportuguespalavra");
assert!(is_correct("zxqportuguespalavra"));
}
}
+444
View File
@@ -0,0 +1,444 @@
//! Portuguese spelling suggestions, best first.
//!
//! Candidates come in rounds of rising cost, each checked against the dictionary:
//! 1. letter case (`brasília` → `Brasília`);
//! 2. missing or wrong accents, from the dictionary's `MAP` groups (`coraçao` → `coração`,
//! `saude` → `saúde`), up to four letters changed — Portuguese typed without accents is the
//! most common slip, so adding one ranks first, while taking away an accent the writer typed
//! ranks after the known confusions;
//! 3. the dictionary's `REP` table of common confusions (`seje` → `seja`, `excessão` →
//! `exceção`, `previlégio` → `privilégio`);
//! 4. a missing space (`nãoé` → `não é`, `paraela` → `para ela`), cheaper when one half is a
//! short function word;
//! 5. one typo: a letter left out, added, swapped with the next or mistyped (`TRY` letters);
//! 6. when that finds fewer than asked for: the stems that start like the word (accents
//! folded) and share the most letter pairs with it, whose forms are ranked by an edit distance
//! in which an accent slip costs half (two typos at most).
//!
//! The input is bounded (40 letters), and so is every round, so a call costs ten-odd thousand
//! dictionary lookups at most.
use std::collections::{HashMap, HashSet};
use crate::affix::{AffixDict, Casing, capitalize, casing};
/// Short words often run into the next one (`não é`, `da água`, `para isso`).
const LEADING_WORDS: &[&str] = &[
"a", "o", "e", "é", "de", "do", "da", "dos", "das", "em", "no", "na", "nos", "nas", "por", "para", "com", "sem", "sob", "sobre", "ao", "aos",
"à", "às", "que", "se", "não", "como", "mas", "mais", "já", "até", "entre", "contra", "perante", "desde", "após", "ante", "durante", "mediante",
"segundo", "conforme", "salvo", "fora", "onde", "quando", "bem", "mal", "pois", "logo", "portanto", "porém", "todavia", "contudo", "assim",
"tão", "tanto", "tudo", "nada", "algo", "cada", "outro", "mesmo", "próprio", "ainda", "tampouco", "quase", "talvez", "nunca", "sempre",
];
/// Short words often run into the one before (`amo-te`, `deu-lhe`; in writing without the
/// hyphen: `amote`, `deulhe`).
const TRAILING_WORDS: &[&str] = &[
"me", "te", "se", "lhe", "nos", "vos", "lhes", "o", "a", "os", "as", "ele", "ela", "eles", "elas", "isso", "isto", "aquilo", "mim", "ti", "si",
"eu", "tu", "que", "de", "em", "é",
];
const MAX_LETTERS: usize = 40;
const MAX_MAP_CHANGES: usize = 4;
const MAX_MAP_VARIANTS: usize = 4096;
/// Stems the scan of round 6 expands.
const SCAN_STEMS: usize = 48;
const COST_CASE: u32 = 0;
const COST_ADD_DIACRITIC: u32 = 4;
const COST_OTHER_MAP: u32 = 7;
const COST_DROP_DIACRITIC: u32 = 11;
const COST_REP: u32 = 10;
const COST_SPLIT: u32 = 12;
const COST_SPLIT_RARE: u32 = 18;
const COST_SWAP: u32 = 14;
const COST_DROP_OR_ADD: u32 = 15;
const COST_MISTYPE: u32 = 16;
const COST_SCAN: u32 = 20;
struct Collector<'a> {
d: &'a AffixDict,
/// The word, lower case.
input: String,
tested: HashSet<String>,
/// Suggestion → (cost, order found).
found: HashMap<String, (u32, usize)>,
seq: usize,
}
impl Collector<'_> {
/// Check a lower-case candidate (a word, or two separated by a space) and keep it if every
/// word is in the dictionary, as written or capitalized (names).
fn offer(&mut self, cand: &str, cost: u32) {
if cand.is_empty() || cand == self.input || !self.tested.insert(cand.to_string()) {
return;
}
let mut words = Vec::new();
let mut cost = cost;
for part in cand.split(' ') {
if self.d.contains(part) {
words.push(part.to_string());
} else {
let c = capitalize(part);
if !self.d.contains(&c) {
return;
}
words.push(c);
// A name ranks after a common word at the same distance (`moça`, then `Moça`).
cost += 1;
}
}
self.keep(words.join(" "), cost);
}
/// Keep a suggestion known to be correct.
fn keep(&mut self, text: String, cost: u32) {
self.seq += 1;
let seq = self.seq;
let e = self.found.entry(text).or_insert((cost, seq));
if cost < e.0 {
*e = (cost, seq);
}
}
fn count_within(&self, cost: u32) -> usize {
self.found.values().filter(|(c, _)| *c <= cost).count()
}
}
/// The MAP group of each letter (its other members are its likely confusions).
fn alternatives(c: char, map: &[Vec<char>]) -> Vec<char> {
map.iter().filter(|g| g.contains(&c)).flat_map(|g| g.iter().copied()).filter(|&x| x != c).collect()
}
/// The cost of changing letter `from` to its MAP relative `to`. Adding a missing accent is the
/// commonest slip (Portuguese typed without accents: `coracao`); an accent the writer did type
/// is likelier right, so taking one away costs more than a known confusion from the REP table
/// (`seje` → `seja` before `sejé`).
fn map_cost(from: char, to: char) -> u32 {
match (from.is_ascii(), to.is_ascii()) {
(true, false) => COST_ADD_DIACRITIC,
(false, true) => COST_DROP_DIACRITIC,
_ => COST_OTHER_MAP,
}
}
/// Every way of changing up to [`MAX_MAP_CHANGES`] letters to a MAP relative, with its cost.
fn map_variants(chars: &[char], alts: &[Vec<char>], i: usize, changes: usize, cost: u32, buf: &mut Vec<char>, out: &mut Vec<(String, u32)>) {
if out.len() >= MAX_MAP_VARIANTS {
return;
}
let Some(&c) = chars.get(i) else {
if changes > 0 {
out.push((buf.iter().collect(), cost));
}
return;
};
buf.push(c);
map_variants(chars, alts, i + 1, changes, cost, buf, out);
buf.pop();
if changes < MAX_MAP_CHANGES {
for &a in alts.get(i).map(Vec::as_slice).unwrap_or_default() {
buf.push(a);
map_variants(chars, alts, i + 1, changes + 1, cost + map_cost(c, a), buf, out);
buf.pop();
}
}
}
/// A letter's MAP representative (the first of its group), lower case: `ã` → `a`, `ç` → `c`.
fn fold(c: char, map: &[Vec<char>]) -> char {
let l = c.to_lowercase().next().unwrap_or(c);
map.iter().find(|g| g.contains(&l)).and_then(|g| g.first().copied()).unwrap_or(l)
}
/// Optimal string alignment distance where a MAP slip (`a`/`á`, `c`/`ç`) costs 1 and any other
/// edit 2; `limit + 1` once it is certainly above `limit`.
fn weighted_distance(a: &[char], b: &[char], related: &dyn Fn(char, char) -> bool, limit: u32) -> u32 {
let m = b.len();
let over = limit + 1;
let mut prev2 = vec![0u32; m + 1];
let mut prev: Vec<u32> = (0..=m as u32).map(|j| j * 2).collect();
let mut cur = vec![0u32; m + 1];
for (i, &ai) in a.iter().enumerate() {
if let Some(c) = cur.get_mut(0) {
*c = (i as u32 + 1) * 2;
}
let mut row_min = u32::MAX;
for (j, &bj) in b.iter().enumerate() {
let sub = if ai == bj {
0
} else if related(ai, bj) {
1
} else {
2
};
let at = |v: &[u32], k: usize| v.get(k).copied().unwrap_or(u32::MAX / 4);
let mut v = (at(&prev, j + 1) + 2).min(at(&cur, j) + 2).min(at(&prev, j) + sub);
if i > 0 && j > 0 && a.get(i - 1) == Some(&bj) && b.get(j - 1) == Some(&ai) {
v = v.min(at(&prev2, j - 1) + 2);
}
if let Some(c) = cur.get_mut(j + 1) {
*c = v;
}
row_min = row_min.min(v);
}
if m > 0 && row_min > limit {
return over;
}
std::mem::swap(&mut prev2, &mut prev);
std::mem::swap(&mut prev, &mut cur);
}
prev.get(m).copied().unwrap_or(over).min(over)
}
/// Spelling suggestions for a Portuguese word, best first, at most `max`, in the word's letter case.
pub fn suggest(word: &str, max: usize) -> Vec<String> {
suggest_in(&super::dictionary(), word, max)
}
/// [`suggest`] against a given dictionary.
pub fn suggest_in(d: &AffixDict, word: &str, max: usize) -> Vec<String> {
let word = word.trim();
let n = word.chars().count();
if n == 0 || n > MAX_LETTERS || max == 0 || d.is_empty() || word.contains(char::is_whitespace) {
return Vec::new();
}
let case = casing(word);
let input = word.to_lowercase();
let chars: Vec<char> = input.chars().collect();
let mut col = Collector { d, input: input.clone(), tested: HashSet::new(), found: HashMap::new(), seq: 0 };
// 1. Letter case.
if d.contains(&input) && word != input {
col.keep(input.clone(), COST_CASE);
} else if d.contains(&capitalize(&input)) && word != capitalize(&input) {
col.keep(capitalize(&input), COST_CASE);
}
// 2. Accents and other MAP slips.
let map = d.map();
let alts: Vec<Vec<char>> = chars.iter().map(|&c| alternatives(c, map)).collect();
let mut variants = Vec::new();
map_variants(&chars, &alts, 0, 0, 0, &mut Vec::with_capacity(n), &mut variants);
variants.sort_by_key(|v| v.1);
for (v, cost) in &variants {
col.offer(v, *cost);
}
// 3. Common confusions.
for (from, to) in d.rep() {
if from.is_empty() {
continue;
}
for (i, _) in input.match_indices(from.as_str()) {
if let (Some(a), Some(b)) = (input.get(..i), input.get(i + from.len()..)) {
col.offer(&format!("{a}{to}{b}"), COST_REP);
}
}
}
// 4. A missing space: cheap after a short leading word or before a short trailing one,
// dearer between two longer words, never leaving a stray short half.
for (i, _) in input.char_indices().skip(1) {
let (Some(a), Some(b)) = (input.get(..i), input.get(i..)) else { continue };
let long = |p: &str| p.chars().count() >= 3;
let cost = if (LEADING_WORDS.contains(&a) && long(b)) || (long(a) && TRAILING_WORDS.contains(&b)) {
COST_SPLIT
} else if long(a) && long(b) {
COST_SPLIT_RARE
} else {
continue;
};
col.offer(&format!("{a} {b}"), cost);
}
// 5. One typo.
let mut letters: Vec<char> = d.try_chars().chars().filter(|c| c.is_lowercase()).collect();
if letters.is_empty() {
letters = "aeiousrntmdlcpvgqbzfhjxçãõáéíóúâêôkw".chars().collect();
}
let s = |v: &[char]| v.iter().collect::<String>();
for i in 0..n.saturating_sub(1) {
if chars.get(i) != chars.get(i + 1) {
let mut v = chars.clone();
v.swap(i, i + 1);
col.offer(&s(&v), COST_SWAP);
}
}
for i in 0..n {
let mut v = chars.clone();
v.remove(i);
col.offer(&s(&v), COST_DROP_OR_ADD);
}
for i in 0..=n {
for &c in &letters {
let mut v = chars.clone();
v.insert(i, c);
col.offer(&s(&v), COST_DROP_OR_ADD);
}
}
for i in 0..n {
for &c in &letters {
if chars.get(i) != Some(&c) {
let mut v = chars.clone();
if let Some(x) = v.get_mut(i) {
*x = c;
}
col.offer(&s(&v), COST_MISTYPE);
}
}
}
// 6. Further off: scan the stems that start like the word.
if col.count_within(COST_MISTYPE) < max {
scan(&mut col, &chars);
}
let mut found: Vec<(String, (u32, usize))> = col.found.into_iter().collect();
// Among equal costs, the change nearest the word's start is the likelier intent
// (`saúde` before `saudê`: both add an accent, but at the first difference).
let diff_at = |text: &str| chars.iter().zip(text.chars()).position(|(a, b)| *a != b).unwrap_or(n);
found.sort_by_key(|(text, (cost, seq))| (*cost, diff_at(text), *seq));
let mut out: Vec<String> = Vec::new();
for (text, _) in found {
let t = match case {
Casing::Initial => capitalize(&text),
Casing::Upper if n > 1 => text.to_uppercase(),
_ => text,
};
if t != word && !out.contains(&t) {
out.push(t);
}
if out.len() >= max {
break;
}
}
out
}
/// Round 6: rank the stems starting with the word's first letter (or a MAP relative of it, either
/// case) by shared letter pairs, then keep the forms of the best within two typos.
fn scan(col: &mut Collector, chars: &[char]) {
let d = col.d;
let map = d.map();
let Some(&first) = chars.first() else { return };
let folded: Vec<char> = chars.iter().map(|&c| fold(c, map)).collect();
let pairs: HashSet<(char, char)> = std::iter::once(' ').chain(folded.iter().copied()).zip(folded.iter().copied()).collect();
let mut firsts: Vec<char> = std::iter::once(first).chain(alternatives(first, map)).collect();
let upper: Vec<char> = firsts.iter().flat_map(|c| c.to_uppercase()).collect();
firsts.extend(upper);
firsts.sort_unstable();
firsts.dedup();
let n = folded.len();
let mut best: Vec<(u32, &str)> = Vec::new();
for f in firsts {
let prefix = f.to_string();
for stem in d.stems_with_prefix(&prefix) {
let sf: Vec<char> = stem.chars().map(|c| fold(c, map)).collect();
let shared = std::iter::once(' ').chain(sf.iter().copied()).zip(sf.iter().copied()).filter(|p| pairs.contains(p)).count();
if shared < 2 {
continue;
}
// Shared pairs, less the stem's letters the word doesn't explain, less a length gap
// beyond what an ending explains.
let extra = sf.len().saturating_sub(shared);
let gap = sf.len().abs_diff(n).saturating_sub(4);
let score = (shared * 4).saturating_sub(extra * 3 + gap * 2) as u32;
if score == 0 {
continue;
}
if best.len() < SCAN_STEMS {
best.push((score, stem));
} else if let Some(min) = best.iter_mut().min_by_key(|b| b.0)
&& score > min.0
{
*min = (score, stem);
}
}
}
best.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(b.1)));
let related = |a: char, b: char| fold(a, map) == fold(b, map);
let limit = 4;
for (_, stem) in best {
for form in d.forms(stem) {
let lf: Vec<char> = form.to_lowercase().chars().collect();
if lf.len().abs_diff(chars.len()) > 2 {
continue;
}
let dist = weighted_distance(&lf, chars, &related, limit);
if dist <= limit && lf.as_slice() != chars {
col.keep(form, COST_SCAN + dist * 3);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn has(word: &str, want: &str) {
let s = suggest(word, 8);
assert!(s.iter().any(|x| x == want), "{word} → {s:?}, want {want}");
}
#[test]
fn portuguese_misspellings_get_the_right_suggestion() {
crate::pt::test_dict::install();
has("coraçao", "coração");
has("saude", "saúde");
has("seje", "seja");
has("excessão", "exceção");
has("previlégio", "privilégio");
has("prsidente", "presidente");
has("provavel", "provável");
has("trbalho", "trabalho");
has("nãoé", "não é");
has("paraela", "para ela");
has("brasília", "Brasília");
has("Coraçao", "Coração");
}
#[test]
fn the_likeliest_fix_comes_first() {
crate::pt::test_dict::install();
assert_eq!(suggest("coraçao", 3).first().map(String::as_str), Some("coração"));
assert_eq!(
suggest("saude", 3).first().map(String::as_str),
Some("saúde"),
"an accent at the first difference beats one further right (saudê is a word too)"
);
assert_eq!(suggest("seje", 3).first().map(String::as_str), Some("seja"));
}
#[test]
fn edge_cases_stay_empty_or_bounded() {
crate::pt::test_dict::install();
assert!(suggest("", 5).is_empty());
assert!(suggest("bem", 0).is_empty());
assert!(suggest(&"a".repeat(41), 5).is_empty());
assert!(suggest("duas palavras", 5).is_empty());
assert!(suggest_in(&crate::affix::AffixDict::default(), "coraçao", 5).is_empty());
assert!(suggest("coração", 5).len() <= 5);
assert!(!suggest("coração", 5).contains(&"coração".to_string()));
}
#[test]
fn distance_weighs_accents_half() {
let map = vec![vec!['a', 'á', 'à', 'â', 'ã'], vec!['c', 'ç']];
let related = |a: char, b: char| fold(a, &map) == fold(b, &map);
let c = |s: &str| s.chars().collect::<Vec<_>>();
assert_eq!(weighted_distance(&c("coraçao"), &c("coração"), &related, 9), 1);
assert_eq!(weighted_distance(&c("coracao"), &c("coração"), &related, 9), 2, "ç and ã are two accent slips");
assert_eq!(weighted_distance(&c("coro"), &c("coro"), &related, 9), 0);
assert_eq!(weighted_distance(&c("bem"), &c("bom"), &related, 9), 2);
assert_eq!(weighted_distance(&c("bem"), &c("bemte"), &related, 9), 4);
assert_eq!(weighted_distance(&c("bem"), &c("ver"), &related, 2), 3);
assert_eq!(weighted_distance(&c(""), &c("ab"), &related, 9), 4);
}
#[test]
fn suggestions_are_fast_enough() {
crate::pt::test_dict::install();
// Warm the dictionary, then time a worst case (a long word nothing near): the scan round.
let _ = suggest("bem", 1);
let t = std::time::Instant::now();
for w in ["xqzvweportuguespalavra", "prsidentes", "paralelepípedo"] {
let _ = suggest(w, 6);
}
let ms = t.elapsed().as_secs_f64() * 1000.0;
eprintln!("3 suggestion calls: {ms:.0} ms");
assert!(ms < 20_000.0, "{ms} ms");
}
}
+121
View File
@@ -0,0 +1,121 @@
//! A tiny synthetic Portuguese Hunspell dictionary for tests (written for these tests; not taken
//! from any real dictionary). Real dictionaries are installed on the system, never bundled.
//!
//! It exercises what VERO-like dictionaries use: `FLAG UTF-8` (multi-byte flag characters —
//! `F` marks a forbidden spelling), continuation classes (`provável` → `prováveis`), the
//! suggestion tables (`TRY`, `MAP` for accents, `REP` for common confusions) and one prefix.
pub const AFF: &str = "SET UTF-8
FLAG UTF-8
FORBIDDENWORD F
TRY aeiousrntmdlcpvgqbzfhjxçãõáéíóúâêôkw
MAP 6
MAP aáàâã
MAP eéê
MAP ií
MAP oóôõ
MAP uú
MAP cç
REP 4
REP seje seja
REP excessão exceção
REP previlégio privilégio
REP analize análise
SFX A Y 2
SFX A a ava a
SFX A a ando a
SFX B N 1
SFX B 0 ável/C .
SFX C N 1
SFX C vel veis vel
PFX X Y 1
PFX X 0 des .
";
/// Stems (the count line first, as in Hunspell files).
pub const DIC: &str = "76
coração
saudade
saudê
pássaro
saúde
árvore
escrever
computador
praia
conhecer
Brasília
São
Paulo
guarda
roupa
mail
marido
ex
vice
presidente
autoestima
frequência
frequencia/F
exceção
privilégio
análise
seja
trabalho
canta/A
prov/B
cobrir/X
avião
pássaros
amanhã
vocês
coragem
parabéns
está
ele
disse
existem
bem
não
sim
que
com
para
dos
das
pela
pelo
muito
mais
como
porque
aqui
também
já
ser
ter
ir
ver
dar
e
o
é
água
ela
da
do
de
em
no
na
um
uma
";
/// Make [`AFF`]/[`DIC`] the Portuguese dictionary (every test that needs one calls this; they all
/// install the same one, so tests running in parallel agree).
pub fn install() {
if let Err(e) = super::install_dictionary(AFF, DIC) {
panic!("test dictionary: {e}");
}
}