mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-08 19:42:12 +00:00
CP4.2: review — the default build keeps the carry-over locales embedded until Language packs ship; CRAFT_PROOF_EMBED=en is the English-only build (CP4.4); licence notices name the real built-in sources
- build.rs: unset CRAFT_PROOF_EMBED embeds de, pl, pt, it patterns and ru-extra as before (DEFAULT_EMBED); `en` embeds English and our supplements only; `de,pl` and `all` as before. Making `en` the default is CP4.4, in the release that publishes and pins language-packs-1. The default-build data tests cover pl/it/de/pt again; CI's extra step is now the English-only build. - licence_texts: one Moby entry (the data's, else the built-in en-words.dic) and en-extra.txt (WordCraft contributors, MIT OR Apache-2.0); the deleted core-list notice is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
This commit is contained in:
1 parent
3893192e88
commit
d905406fe3
5 files changed
+75
-44
No files matched your search
@@ -60,9 +60,9 @@ jobs:
|
||||
cargo clippy -p craft-proof --all-targets --locked -- -D warnings
|
||||
cargo test -p craft-proof --locked
|
||||
cargo check -p craft-proof --locked --target wasm32-unknown-unknown
|
||||
- name: A Language build (CRAFT_PROOF_EMBED) with every locale embedded
|
||||
- name: The English-only build (CRAFT_PROOF_EMBED=en, CP4.4's default once Language packs ship)
|
||||
env:
|
||||
CRAFT_I18N_DIR: ${{ github.workspace }}/craft-i18n
|
||||
CRAFT_I18N_REQUIRED: "1"
|
||||
CRAFT_PROOF_EMBED: all
|
||||
CRAFT_PROOF_EMBED: en
|
||||
run: cargo test -p craft-proof --locked
|
||||
+10
-8
@@ -14,9 +14,9 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = "<commit>
|
||||
| Language | Spelling | Grammar | Hyphenation |
|
||||
|---|---|---|---|
|
||||
| English (en-US, en-CA, en-GB and the British-spelling Englishes such as en-AU, en-NZ) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
|
||||
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns (Language pack, or a Language build) |
|
||||
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns (embedded, or a Language pack) |
|
||||
| Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise |
|
||||
| German | — | — | TeX patterns (Language pack, or a Language build) |
|
||||
| German | — | — | TeX patterns (embedded, or a Language pack) |
|
||||
| Bulgarian | — | — | syllable rules |
|
||||
|
||||
Grammar rules, suggestion tuning and the rule-based hyphenation are code: they work in every
|
||||
@@ -59,8 +59,11 @@ Tests that pin results of the data run only when every file the build embeds is
|
||||
`src/no_data.rs` runs every public entry point when none is (`craft_proof_no_data`). A partial
|
||||
data folder runs neither set.
|
||||
|
||||
- Only English (`en-*`) and our own word supplements (`*-extra.txt`, MIT) are embedded by default. A **Language build** names the others it embeds
|
||||
with `CRAFT_PROOF_EMBED=de,pl` (or `all`); every other build gets them from a Language pack.
|
||||
- By default every carry-over locale is embedded: English, the German, Polish, Portuguese and
|
||||
Italian patterns and our own word supplements (`*-extra.txt`, MIT), as before Language packs.
|
||||
`CRAFT_PROOF_EMBED` picks the set: `en` (English and our supplements only; the others come from
|
||||
Language packs), `de,pl`, or `all`. Making `en` the default is **CP4.4**, done in the release
|
||||
that publishes and pins `language-packs-1`.
|
||||
`craft_proof::packs::embedded_locales()` lists what a build embeds.
|
||||
|
||||
## Language packs
|
||||
@@ -75,7 +78,7 @@ pub fn packs_dir() -> Option<PathBuf>; // +
|
||||
pub fn installed() -> Vec<InstalledPack>; // InstalledPack { locale, name, licence, path }, by locale
|
||||
pub fn licence_texts() -> Vec<(String, String)>; // (title, text): embedded English notices, then each pack's LICENCES/*
|
||||
pub fn load_errors() -> Vec<(String, String)>; // + (locale or folder, reason): packs skipped, pattern files that failed
|
||||
pub fn embedded_locales() -> &'static [&'static str]; // + what this build embeds ("en-us"; more in a Language build)
|
||||
pub fn embedded_locales() -> &'static [&'static str]; // + what this build embeds ("en-us", "de", "pl", "pt", "it", "ru" by default; "en-us" with CRAFT_PROOF_EMBED=en)
|
||||
|
||||
// craft_proof (crate root)
|
||||
pub enum ProofLang { …, Pack(PackLocale), Other } // + Pack: any language without tools of its own (nl, fr, sr-Latn…)
|
||||
@@ -127,8 +130,7 @@ locale (`nl`, `pt-br`, `sr-latn`):
|
||||
Hunspell dictionaries, then (English only) the built-in core list. A slot's Hunspell names map to
|
||||
pack locales (`pt_BR` → `pt-br`, then `pt`, then any other `pt-*` pack). Hyphenation, every
|
||||
language but English: an installed pack's patterns for a word in the language's script (Russian,
|
||||
Bulgarian, Turkish: their script; others: letters the patterns know), then the embedded ones (a
|
||||
Language build), then the language's rules or the English fallback as before.
|
||||
Bulgarian, Turkish: their script; others: letters the patterns know), then the embedded ones, then the language's rules or the English fallback as before.
|
||||
|
||||
**Hostile packs.** Every file is read only if it is a regular file (never through a symbolic
|
||||
link), size-capped (`pack.toml` 1 MB, licences 4 MB, dictionaries and patterns 64 MB), and a pack
|
||||
@@ -205,7 +207,7 @@ and the notice); the Moby list (432,905 bytes) stays for hyphenation.
|
||||
cargo fmt -p craft-proof --check
|
||||
cargo clippy -p craft-proof --all-targets -- -D warnings
|
||||
CRAFT_I18N_DIR=/abs/path/craft-i18n cargo test -p craft-proof
|
||||
CRAFT_PROOF_EMBED=all CRAFT_I18N_DIR=/abs/path/craft-i18n cargo test -p craft-proof # a Language build
|
||||
CRAFT_PROOF_EMBED=en CRAFT_I18N_DIR=/abs/path/craft-i18n cargo test -p craft-proof # the English-only build (CP4.4)
|
||||
cargo test -p craft-proof # without Language data
|
||||
cargo check -p craft-proof --target wasm32-unknown-unknown
|
||||
```
|
||||
|
||||
+13
-6
@@ -8,7 +8,7 @@
|
||||
//! hyphenation that needs the data is off). With `CRAFT_I18N_REQUIRED=1` (releases) any of those
|
||||
//! is an error instead.
|
||||
//!
|
||||
//! When every file in [`FILES`] this build embeds (English and the `CRAFT_PROOF_EMBED` locales) is
|
||||
//! When every file in [`FILES`] this build embeds (English and the [`opted_in`] locales) is
|
||||
//! there, the `craft_proof_data` cfg is set (the tests that pin
|
||||
//! results of the data use it); when none is, `craft_proof_no_data` (the tests of a build without
|
||||
//! the data). A partial data folder sets neither, so neither set of tests runs against it.
|
||||
@@ -24,8 +24,8 @@ use std::fmt::Write as _;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// (constant, path under `language-data/`, bytes rather than text). The first path part is the
|
||||
/// locale: `en-*` and our own `*-extra.txt` supplements are always embedded, any other only in a
|
||||
/// Language build that names it.
|
||||
/// locale: `en-*` and our own `*-extra.txt` supplements are always embedded, the others as
|
||||
/// [`opted_in`] says.
|
||||
const FILES: &[(&str, &str, bool)] = &[
|
||||
("EN_US_DIC", "en-us/en-us.dic", true),
|
||||
("EN_US_AFF", "en-us/spelling.aff", false),
|
||||
@@ -45,10 +45,17 @@ const FILES: &[(&str, &str, bool)] = &[
|
||||
("EXTRA_RU", "ru/ru-extra.txt", false),
|
||||
];
|
||||
|
||||
/// The locales a Language build embeds besides English: `CRAFT_PROOF_EMBED`, comma-separated
|
||||
/// lowercase locales (`de,pl`), or `all`. Unset: English (and our supplements) only; the others come from Language packs.
|
||||
/// The locales embedded by default besides English: every carry-over locale, until Language
|
||||
/// packs are published and pinned (CP4.4 then makes the default English only).
|
||||
const DEFAULT_EMBED: &[&str] = &["de", "pl", "pt", "it", "ru"];
|
||||
|
||||
/// The locales this build embeds besides English: `CRAFT_PROOF_EMBED`, comma-separated lowercase
|
||||
/// locales (`de,pl`), `all`, or `en` (English and our supplements only; the others come from
|
||||
/// Language packs). Unset: [`DEFAULT_EMBED`].
|
||||
fn opted_in() -> Vec<String> {
|
||||
std::env::var("CRAFT_PROOF_EMBED").unwrap_or_default().split(',').map(|l| l.trim().to_ascii_lowercase()).filter(|l| !l.is_empty()).collect()
|
||||
let v = std::env::var("CRAFT_PROOF_EMBED").unwrap_or_default();
|
||||
let named: Vec<String> = v.split(',').map(|l| l.trim().to_ascii_lowercase()).filter(|l| !l.is_empty()).collect();
|
||||
if named.is_empty() { DEFAULT_EMBED.iter().map(|l| (*l).to_string()).collect() } else { named }
|
||||
}
|
||||
|
||||
/// Our own word supplements (`*-extra.txt`): MIT, tiny, embedded in every build whatever the
|
||||
|
||||
+12
-12
@@ -19,9 +19,9 @@
|
||||
//! Swiss text, whose `ss` the patterns split like any double consonant (`Stras-se`).
|
||||
//!
|
||||
//! **Installed Language packs** ([`crate::packs`]): a pack's `hyphenation.pat` for the language
|
||||
//! comes first, for every language but English (embedded). Only English is embedded by default;
|
||||
//! the Polish, Portuguese, Italian and German patterns are embedded only in a Language build
|
||||
//! (`CRAFT_PROOF_EMBED`), and otherwise come from their packs.
|
||||
//! comes first, for every language but English (embedded). The Polish, Portuguese, Italian and
|
||||
//! German patterns are embedded by default too until Language packs ship (CP4.4); a build with
|
||||
//! `CRAFT_PROOF_EMBED=en` leaves them to their packs.
|
||||
//!
|
||||
//! Then the paragraph's limits ([`Limits`]) are applied.
|
||||
|
||||
@@ -819,7 +819,7 @@ mod tests {
|
||||
hyphenate_word_in(w, &Limits::default(), ProofLang::Pl)
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pl)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn polish_patterns_load() {
|
||||
let p = Patterns::pl();
|
||||
@@ -836,7 +836,7 @@ mod tests {
|
||||
hyphenate_word_in(w, &Limits::default(), ProofLang::It)
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=it)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn italian_patterns_load() {
|
||||
let p = Patterns::it();
|
||||
@@ -845,7 +845,7 @@ mod tests {
|
||||
assert_eq!(p.exception_count(), 0);
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=it)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "it"), ignore = "needs the it Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn italian_words_hyphenate_by_italian_rules() {
|
||||
for (w, want) in [
|
||||
@@ -872,7 +872,7 @@ mod tests {
|
||||
assert_eq!(word_breaks_in("famiglia", ProofLang::It).1, Source::Patterns);
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pt)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn portuguese_patterns_load() {
|
||||
let p = Patterns::pt();
|
||||
@@ -884,7 +884,7 @@ mod tests {
|
||||
assert_eq!(p.exception("software"), Some(&[4][..]));
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pt)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "pt"), ignore = "needs the pt Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn portuguese_words_hyphenate_by_portuguese_rules() {
|
||||
for (w, want) in [
|
||||
@@ -904,7 +904,7 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pl)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn polish_words_hyphenate_by_polish_rules() {
|
||||
for (w, want) in [
|
||||
@@ -938,7 +938,7 @@ mod tests {
|
||||
assert_eq!(word_breaks_in("konstytucja", ProofLang::Pl).1, Source::Patterns);
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=pl)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "pl"), ignore = "needs the pl Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn polish_heuristic_for_foreign_letters() {
|
||||
// Letters outside the Polish alphabet: the Polish syllable heuristic, digraphs kept whole.
|
||||
@@ -960,7 +960,7 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=de)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn german_patterns_load() {
|
||||
let p = Patterns::de();
|
||||
@@ -969,7 +969,7 @@ mod tests {
|
||||
assert_eq!(p.exception_count(), 0);
|
||||
}
|
||||
|
||||
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR, CRAFT_PROOF_EMBED=de)")]
|
||||
#[cfg_attr(not(craft_proof_embed = "de"), ignore = "needs the de Language data (CRAFT_I18N_DIR; not in a CRAFT_PROOF_EMBED=en build)")]
|
||||
#[test]
|
||||
fn german_words_follow_the_german_patterns() {
|
||||
let de = |w: &str| hyphenate_word_in(w, &Limits::default(), ProofLang::De);
|
||||
|
||||
+38
-16
@@ -149,10 +149,13 @@ pub fn load_errors() -> Vec<(String, String)> {
|
||||
/// embedded at build time, then each installed pack's `LICENCES/` files (About › Language data).
|
||||
pub fn licence_texts() -> Vec<(String, String)> {
|
||||
let mut out: Vec<(String, String)> = Vec::new();
|
||||
if !crate::data::EN_US_DIC.is_empty() {
|
||||
// One Moby entry: the embedded data's, else the copy built into the crate (`en-words.dic`).
|
||||
if crate::data::EN_US_DIC.is_empty() {
|
||||
out.push(("English word list (Moby Hyphenator II)".into(), BUILT_IN_MOBY_NOTICE.into()));
|
||||
} else {
|
||||
out.push(("English word list and hyphenation (Moby Hyphenator II)".into(), MOBY_NOTICE.into()));
|
||||
}
|
||||
out.push(("English word list built into craft-proof".into(), CORE_LIST_NOTICE.into()));
|
||||
out.push(("English word supplement (en-extra.txt)".into(), EXTRA_EN_NOTICE.into()));
|
||||
out.extend(crate::data::EMBEDDED_LICENCES.iter().map(|(t, x)| ((*t).to_string(), (*x).to_string())));
|
||||
// Each file is checked against what is left before it is read, so no more is ever read.
|
||||
let mut left: u64 = MAX_LICENCE_TOTAL;
|
||||
@@ -180,11 +183,16 @@ const MOBY_NOTICE: &str = "The Moby Hyphenator II word list by Grady Ward (Proje
|
||||
The English hyphenation patterns were trained from it for the Craft apps and are distributed with them under MIT OR Apache-2.0.";
|
||||
|
||||
/// The notice of the word list built into the crate (`data/en-words.dic`).
|
||||
const CORE_LIST_NOTICE: &str = "The English word list built into craft-proof is the Moby Hyphenator II word list by Grady Ward \
|
||||
const BUILT_IN_MOBY_NOTICE: &str = "The English word list built into craft-proof is the Moby Hyphenator II word list by Grady Ward \
|
||||
(Project Gutenberg eBook #3204), which is in the public domain.";
|
||||
|
||||
/// The locales whose Language data is embedded in this build (`en-us`…, and in a Language build
|
||||
/// the ones it names; `CRAFT_PROOF_EMBED`).
|
||||
/// The notice of our English supplement (`data/en-extra.txt`, and craft-i18n's copy).
|
||||
const EXTRA_EN_NOTICE: &str = "The English word supplement (en-extra.txt), common words the Moby list lacks, is written by \
|
||||
WordCraft contributors and licensed MIT OR Apache-2.0.";
|
||||
|
||||
/// The locales whose Language data is embedded in this build: `en-us`, and by default `de`,
|
||||
/// `pl`, `pt`, `it`, `ru` (until Language packs ship, CP4.4); a build with `CRAFT_PROOF_EMBED`
|
||||
/// embeds English and the ones it names (`en`: English only).
|
||||
pub fn embedded_locales() -> &'static [&'static str] {
|
||||
crate::data::EMBEDDED_LOCALES
|
||||
}
|
||||
@@ -615,24 +623,38 @@ mod tests {
|
||||
assert!(parse_pack_toml(&deep).is_err());
|
||||
}
|
||||
|
||||
/// Review of #485: our own word supplements (`ru-extra.txt`, MIT) stay embedded in the
|
||||
/// default build; they aren't upstream data and don't make a locale "embedded".
|
||||
/// Review of #485: our own word supplements (`ru-extra.txt`, MIT) stay embedded in every
|
||||
/// build, the English-only one too.
|
||||
#[test]
|
||||
fn our_supplements_are_embedded_by_default() {
|
||||
fn our_supplements_are_always_embedded() {
|
||||
if cfg!(craft_proof_data) {
|
||||
assert!(!crate::data::EXTRA_RU.is_empty(), "ru-extra.txt is embedded");
|
||||
}
|
||||
let embed = option_env!("CRAFT_PROOF_EMBED").unwrap_or_default();
|
||||
if embed.is_empty() {
|
||||
assert!(!embedded_locales().contains(&"ru"));
|
||||
}
|
||||
|
||||
/// #5208 review: until Language packs are published, the default build embeds the
|
||||
/// carry-over locales (de, pl, pt, it patterns, ru-extra); `CRAFT_PROOF_EMBED=en` is the
|
||||
/// English-only build (CP4.4's default once language-packs-1 is pinned).
|
||||
#[test]
|
||||
fn the_default_build_embeds_the_carry_over_locales() {
|
||||
let embed = option_env!("CRAFT_PROOF_EMBED").unwrap_or_default().trim().to_ascii_lowercase();
|
||||
let named: Vec<&str> = if embed.is_empty() { vec!["de", "pl", "pt", "it", "ru"] } else { embed.split(',').map(str::trim).collect() };
|
||||
for l in embedded_locales() {
|
||||
assert!(l.starts_with("en-") || *l == "ru" || named.iter().any(|e| e == l || *e == "all"), "{l} is embedded");
|
||||
}
|
||||
if cfg!(craft_proof_data) && embed.is_empty() {
|
||||
for l in ["de", "pl", "pt", "it"] {
|
||||
assert!(embedded_locales().contains(&l), "{l} is embedded by default");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One Moby entry, and the supplement's own licence.
|
||||
#[test]
|
||||
fn embedded_locales_are_english_unless_a_language_build_names_more() {
|
||||
let embed = option_env!("CRAFT_PROOF_EMBED").unwrap_or_default();
|
||||
for l in embedded_locales() {
|
||||
assert!(l.starts_with("en-") || embed.split(',').any(|e| e.trim() == *l || e.trim() == "all"), "{l} is embedded");
|
||||
}
|
||||
fn built_in_notices_name_the_real_sources() {
|
||||
let texts = licence_texts();
|
||||
assert_eq!(texts.iter().filter(|(t, _)| t.contains("Moby")).count(), 1, "{texts:?}");
|
||||
assert!(texts.iter().any(|(t, x)| t.contains("en-extra.txt") && x.contains("MIT OR Apache-2.0")));
|
||||
assert!(!texts.iter().any(|(t, _)| t.contains("core")), "the core list is gone");
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user