CP4.2: review — the default build keeps the carry-over locales embedded until Language packs ship; CRAFT_PROOF_EMBED=en is the English-only build (CP4.4); licence notices name the real built-in sources

- build.rs: unset CRAFT_PROOF_EMBED embeds de, pl, pt, it patterns and ru-extra as before (DEFAULT_EMBED); `en` embeds English and our supplements only; `de,pl` and `all` as before. Making `en` the default is CP4.4, in the release that publishes and pins language-packs-1. The default-build data tests cover pl/it/de/pt again; CI's extra step is now the English-only build.
- licence_texts: one Moby entry (the data's, else the built-in en-words.dic) and en-extra.txt (WordCraft contributors, MIT OR Apache-2.0); the deleted core-list notice is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
This commit is contained in:
Martin BallandClaude Opus 5.5 committed 2026-11-01 22:34:33 +00:00
1 parent 3893192e88
commit d905406fe3
5 files changed
+75 -44

No files matched your search

+2 -2
View File
@@ -60,9 +60,9 @@ jobs:
cargo clippy -p craft-proof --all-targets --locked -- -D warnings
cargo test -p craft-proof --locked
cargo check -p craft-proof --locked --target wasm32-unknown-unknown
- name: A Language build (CRAFT_PROOF_EMBED) with every locale embedded
- name: The English-only build (CRAFT_PROOF_EMBED=en, CP4.4's default once Language packs ship)
env:
CRAFT_I18N_DIR: ${{ github.workspace }}/craft-i18n
CRAFT_I18N_REQUIRED: "1"
CRAFT_PROOF_EMBED: all
CRAFT_PROOF_EMBED: en
run: cargo test -p craft-proof --locked
+10 -8
View File
@@ -14,9 +14,9 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = "<commit>
| Language | Spelling | Grammar | Hyphenation |
|---|---|---|---|
| English (en-US, en-CA, en-GB and the British-spelling Englishes such as en-AU, en-NZ) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns (Language pack, or a Language build) |
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns (embedded, or a Language pack) |
| Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise |
| German | — | — | TeX patterns (Language pack, or a Language build) |
| German | — | — | TeX patterns (embedded, or a Language pack) |
| Bulgarian | — | — | syllable rules |
Grammar rules, suggestion tuning and the rule-based hyphenation are code: they work in every
@@ -59,8 +59,11 @@ Tests that pin results of the data run only when every file the build embeds is
`src/no_data.rs` runs every public entry point when none is (`craft_proof_no_data`). A partial
data folder runs neither set.
- Only English (`en-*`) and our own word supplements (`*-extra.txt`, MIT) are embedded by default. A **Language build** names the others it embeds
with `CRAFT_PROOF_EMBED=de,pl` (or `all`); every other build gets them from a Language pack.
- By default every carry-over locale is embedded: English, the German, Polish, Portuguese and
Italian patterns and our own word supplements (`*-extra.txt`, MIT), as before Language packs.
`CRAFT_PROOF_EMBED` picks the set: `en` (English and our supplements only; the others come from
Language packs), `de,pl`, or `all`. Making `en` the default is **CP4.4**, done in the release
that publishes and pins `language-packs-1`.
`craft_proof::packs::embedded_locales()` lists what a build embeds.
## Language packs
@@ -75,7 +78,7 @@ pub fn packs_dir() -> Option<PathBuf>; // +
pub fn installed() -> Vec<InstalledPack>; // InstalledPack { locale, name, licence, path }, by locale
pub fn licence_texts() -> Vec<(String, String)>; // (title, text): embedded English notices, then each pack's LICENCES/*
pub fn load_errors() -> Vec<(String, String)>; // + (locale or folder, reason): packs skipped, pattern files that failed
pub fn embedded_locales() -> &'static [&'static str]; // + what this build embeds ("en-us"; more in a Language build)
pub fn embedded_locales() -> &'static [&'static str]; // + what this build embeds ("en-us", "de", "pl", "pt", "it", "ru" by default; "en-us" with CRAFT_PROOF_EMBED=en)
// craft_proof (crate root)
pub enum ProofLang { …, Pack(PackLocale), Other } // + Pack: any language without tools of its own (nl, fr, sr-Latn…)
@@ -127,8 +130,7 @@ locale (`nl`, `pt-br`, `sr-latn`):
Hunspell dictionaries, then (English only) the built-in core list. A slot's Hunspell names map to
pack locales (`pt_BR` → `pt-br`, then `pt`, then any other `pt-*` pack). Hyphenation, every
language but English: an installed pack's patterns for a word in the language's script (Russian,
Bulgarian, Turkish: their script; others: letters the patterns know), then the embedded ones (a
Language build), then the language's rules or the English fallback as before.
Bulgarian, Turkish: their script; others: letters the patterns know), then the embedded ones, then the language's rules or the English fallback as before.
**Hostile packs.** Every file is read only if it is a regular file (never through a symbolic
link), size-capped (`pack.toml` 1 MB, licences 4 MB, dictionaries and patterns 64 MB), and a pack
@@ -205,7 +207,7 @@ and the notice); the Moby list (432,905 bytes) stays for hyphenation.
cargo fmt -p craft-proof --check
cargo clippy -p craft-proof --all-targets -- -D warnings
CRAFT_I18N_DIR=/abs/path/craft-i18n cargo test -p craft-proof
CRAFT_PROOF_EMBED=all CRAFT_I18N_DIR=/abs/path/craft-i18n cargo test -p craft-proof # a Language build
CRAFT_PROOF_EMBED=en CRAFT_I18N_DIR=/abs/path/craft-i18n cargo test -p craft-proof # the English-only build (CP4.4)
cargo test -p craft-proof # without Language data
cargo check -p craft-proof --target wasm32-unknown-unknown
```
+13 -6
View File
@@ -8,7 +8,7 @@
//! hyphenation that needs the data is off). With `CRAFT_I18N_REQUIRED=1` (releases) any of those
//! is an error instead.
//!
//! When every file in [`FILES`] this build embeds (English and the `CRAFT_PROOF_EMBED` locales) is
//! When every file in [`FILES`] this build embeds (English and the [`opted_in`] locales) is
//! there, the `craft_proof_data` cfg is set (the tests that pin
//! results of the data use it); when none is, `craft_proof_no_data` (the tests of a build without
//! the data). A partial data folder sets neither, so neither set of tests runs against it.
@@ -24,8 +24,8 @@ use std::fmt::Write as _;
use std::path::{Path, PathBuf};
/// (constant, path under `language-data/`, bytes rather than text). The first path part is the
/// locale: `en-*` and our own `*-extra.txt` supplements are always embedded, any other only in a
/// Language build that names it.
/// locale: `en-*` and our own `*-extra.txt` supplements are always embedded, the others as
/// [`opted_in`] says.
const FILES: &[(&str, &str, bool)] = &[
("EN_US_DIC", "en-us/en-us.dic", true),
("EN_US_AFF", "en-us/spelling.aff", false),
@@ -45,10 +45,17 @@ const FILES: &[(&str, &str, bool)] = &[
("EXTRA_RU", "ru/ru-extra.txt", false),
];
/// The locales a Language build embeds besides English: `CRAFT_PROOF_EMBED`, comma-separated
/// lowercase locales (`de,pl`), or `all`. Unset: English (and our supplements) only; the others come from Language packs.
/// The locales embedded by default besides English: every carry-over locale, until Language
/// packs are published and pinned (CP4.4 then makes the default English only).
const DEFAULT_EMBED: &[&str] = &["de", "pl", "pt", "it", "ru"];
/// The locales this build embeds besides English: `CRAFT_PROOF_EMBED`, comma-separated lowercase
/// locales (`de,pl`), `all`, or `en` (English and our supplements only; the others come from
/// Language packs). Unset: [`DEFAULT_EMBED`].
fn opted_in() -> Vec<String> {
std::env::var("CRAFT_PROOF_EMBED").unwrap_or_default().split(',').map(|l| l.trim().to_ascii_lowercase()).filter(|l| !l.is_empty()).collect()
let v = std::env::var("CRAFT_PROOF_EMBED").unwrap_or_default();
let named: Vec<String> = v.split(',').map(|l| l.trim().to_ascii_lowercase()).filter(|l| !l.is_empty()).collect();
if named.is_empty() { DEFAULT_EMBED.iter().map(|l| (*l).to_string()).collect() } else { named }
}
/// Our own word supplements (`*-extra.txt`): MIT, tiny, embedded in every build whatever the
+12 -12
View File
@@ -19,9 +19,9 @@
//! Swiss text, whose `ss` the patterns split like any double consonant (`Stras-se`).
//!
//! **Installed Language packs** ([`crate::packs`]): a pack's `hyphenation.pat` for the language
//! comes first, for every language but English (embedded). Only English is embedded by default;
//! the Polish, Portuguese, Italian and German patterns are embedded only in a Language build
//! (`CRAFT_PROOF_EMBED`), and otherwise come from their packs.
//! comes first, for every language but English (embedded). The Polish, Portuguese, Italian and
//! German patterns are embedded by default too until Language packs ship (CP4.4); a build with
//! `CRAFT_PROOF_EMBED=en` leaves them to their packs.
//!
//! Then the paragraph's limits ([`Limits`]) are applied.
@@ -819,7 +819,7 @@ mod tests {
hyphenate_word_in(w, &Limits::default(), ProofLang::Pl)
}
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pl)")]
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn polish_patterns_load() {
let p = Patterns::pl();
@@ -836,7 +836,7 @@ mod tests {
hyphenate_word_in(w, &Limits::default(), ProofLang::It)
}
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=it)")]
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn italian_patterns_load() {
let p = Patterns::it();
@@ -845,7 +845,7 @@ mod tests {
assert_eq!(p.exception_count(), 0);
}
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=it)")]
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn italian_words_hyphenate_by_italian_rules() {
for (w, want) in [
@@ -872,7 +872,7 @@ mod tests {
assert_eq!(word_breaks_in("famiglia", ProofLang::It).1, Source::Patterns);
}
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pt)")]
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn portuguese_patterns_load() {
let p = Patterns::pt();
@@ -884,7 +884,7 @@ mod tests {
assert_eq!(p.exception("software"), Some(&[4][..]));
}
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pt)")]
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn portuguese_words_hyphenate_by_portuguese_rules() {
for (w, want) in [
@@ -904,7 +904,7 @@ mod tests {
}
}
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pl)")]
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn polish_words_hyphenate_by_polish_rules() {
for (w, want) in [
@@ -938,7 +938,7 @@ mod tests {
assert_eq!(word_breaks_in("konstytucja", ProofLang::Pl).1, Source::Patterns);
}
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pl)")]
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn polish_heuristic_for_foreign_letters() {
// Letters outside the Polish alphabet: the Polish syllable heuristic, digraphs kept whole.
@@ -960,7 +960,7 @@ mod tests {
}
}
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=de)")]
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn german_patterns_load() {
let p = Patterns::de();
@@ -969,7 +969,7 @@ mod tests {
assert_eq!(p.exception_count(), 0);
}
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=de)")]
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
#[test]
fn german_words_follow_the_german_patterns() {
let de = |w: &str| hyphenate_word_in(w, &Limits::default(), ProofLang::De);
+38 -16
View File
@@ -149,10 +149,13 @@ pub fn load_errors() -> Vec<(String, String)> {
/// embedded at build time, then each installed pack's `LICENCES/` files (About › Language data).
pub fn licence_texts() -> Vec<(String, String)> {
let mut out: Vec<(String, String)> = Vec::new();
if !crate::data::EN_US_DIC.is_empty() {
// One Moby entry: the embedded data's, else the copy built into the crate (`en-words.dic`).
if crate::data::EN_US_DIC.is_empty() {
out.push(("English word list (Moby Hyphenator II)".into(), BUILT_IN_MOBY_NOTICE.into()));
} else {
out.push(("English word list and hyphenation (Moby Hyphenator II)".into(), MOBY_NOTICE.into()));
}
out.push(("English word list built into craft-proof".into(), CORE_LIST_NOTICE.into()));
out.push(("English word supplement (en-extra.txt)".into(), EXTRA_EN_NOTICE.into()));
out.extend(crate::data::EMBEDDED_LICENCES.iter().map(|(t, x)| ((*t).to_string(), (*x).to_string())));
// Each file is checked against what is left before it is read, so no more is ever read.
let mut left: u64 = MAX_LICENCE_TOTAL;
@@ -180,11 +183,16 @@ const MOBY_NOTICE: &str = "The Moby Hyphenator II word list by Grady Ward (Proje
The English hyphenation patterns were trained from it for the Craft apps and are distributed with them under MIT OR Apache-2.0.";
/// The notice of the word list built into the crate (`data/en-words.dic`).
const CORE_LIST_NOTICE: &str = "The English word list built into craft-proof is the Moby Hyphenator II word list by Grady Ward \
const BUILT_IN_MOBY_NOTICE: &str = "The English word list built into craft-proof is the Moby Hyphenator II word list by Grady Ward \
(Project Gutenberg eBook #3204), which is in the public domain.";
/// The locales whose Language data is embedded in this build (`en-us`…, and in a Language build
/// the ones it names; `CRAFT_PROOF_EMBED`).
/// The notice of our English supplement (`data/en-extra.txt`, and craft-i18n's copy).
const EXTRA_EN_NOTICE: &str = "The English word supplement (en-extra.txt), common words the Moby list lacks, is written by \
WordCraft contributors and licensed MIT OR Apache-2.0.";
/// The locales whose Language data is embedded in this build: `en-us`, and by default `de`,
/// `pl`, `pt`, `it`, `ru` (until Language packs ship, CP4.4); a build with `CRAFT_PROOF_EMBED`
/// embeds English and the ones it names (`en`: English only).
pub fn embedded_locales() -> &'static [&'static str] {
crate::data::EMBEDDED_LOCALES
}
@@ -615,24 +623,38 @@ mod tests {
assert!(parse_pack_toml(&deep).is_err());
}
/// Review of #485: our own word supplements (`ru-extra.txt`, MIT) stay embedded in the
/// default build; they aren't upstream data and don't make a locale "embedded".
/// Review of #485: our own word supplements (`ru-extra.txt`, MIT) stay embedded in every
/// build, the English-only one too.
#[test]
fn our_supplements_are_embedded_by_default() {
fn our_supplements_are_always_embedded() {
if cfg!(craft_proof_data) {
assert!(!crate::data::EXTRA_RU.is_empty(), "ru-extra.txt is embedded");
}
let embed = option_env!("CRAFT_PROOF_EMBED").unwrap_or_default();
if embed.is_empty() {
assert!(!embedded_locales().contains(&"ru"));
}
/// #5208 review: until Language packs are published, the default build embeds the
/// carry-over locales (de, pl, pt, it patterns, ru-extra); `CRAFT_PROOF_EMBED=en` is the
/// English-only build (CP4.4's default once language-packs-1 is pinned).
#[test]
fn the_default_build_embeds_the_carry_over_locales() {
let embed = option_env!("CRAFT_PROOF_EMBED").unwrap_or_default().trim().to_ascii_lowercase();
let named: Vec<&str> = if embed.is_empty() { vec!["de", "pl", "pt", "it", "ru"] } else { embed.split(',').map(str::trim).collect() };
for l in embedded_locales() {
assert!(l.starts_with("en-") || *l == "ru" || named.iter().any(|e| e == l || *e == "all"), "{l} is embedded");
}
if cfg!(craft_proof_data) && embed.is_empty() {
for l in ["de", "pl", "pt", "it"] {
assert!(embedded_locales().contains(&l), "{l} is embedded by default");
}
}
}
/// One Moby entry, and the supplement's own licence.
#[test]
fn embedded_locales_are_english_unless_a_language_build_names_more() {
let embed = option_env!("CRAFT_PROOF_EMBED").unwrap_or_default();
for l in embedded_locales() {
assert!(l.starts_with("en-") || embed.split(',').any(|e| e.trim() == *l || e.trim() == "all"), "{l} is embedded");
}
fn built_in_notices_name_the_real_sources() {
let texts = licence_texts();
assert_eq!(texts.iter().filter(|(t, _)| t.contains("Moby")).count(), 1, "{texts:?}");
assert!(texts.iter().any(|(t, x)| t.contains("en-extra.txt") && x.contains("MIT OR Apache-2.0")));
assert!(!texts.iter().any(|(t, _)| t.contains("core")), "the core list is gone");
}
}