Martin BallandClaude Opus 5.5 23151270ab CP1.2: English never goes unchecked — embedded data, then system Hunspell, then a built-in core word list ...

- No embedded Moby list: English (en-US, en-GB, en-CA) uses an English
  Hunspell .aff/.dic from the dictionaries folder or the system (the run's
  English first, then another English), else data/en-core.txt.
- en-core.txt: 11,367 public-domain Moby Hyphenator II words (Grady Ward),
  ranked by lower-case frequency in 50 public-domain Gutenberg books
  (examples/en_core.rs, book ids recorded). 94 KB. Marks only near misses
  of common words (swapped letters; one edit for lower-case words of 4+).
- Unreadable English dictionary (encoding, not UTF-8, no words, size):
  error via dictionary_load_error, falls through to the core list; a bad
  embedded list is reported the same way. Never a panic.
- Non-English text is never checked against English.
- New: english_source(lang) -> Option<EnglishSource>.
- Embedded-data path unchanged; tests with data unchanged.
- Tests: core list flags/accepts, folder Hunspell preferred over core list,
  unreadable dictionaries reported, no_data English now marks misspellings;
  two data-ignored tests now run without data. README, ATTRIBUTION, NOTICE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2

2026-11-01 10:55:57 +00:00

..

en_core.rs

CP1.2: English never goes unchecked — embedded data, then system Hunspell, then a built-in core word list

2026-11-01 10:55:57 +00:00