Merge #3492: M7: break Thai rows between dictionary words

Brings the Thai word-breaking branch up to date with main. The only conflict was the edit.add-text parity entry: main's evidence and notes (print inks) are kept, with the new Thai test added and the Thai wrapping sentence updated.
This commit is contained in:
bflatastic committed 2026-11-08 23:58:18 +07:00
commit c8248e3b6d
17 files changed
+440 -35

No files matched your search

+1 -1
View File
@@ -84,7 +84,7 @@ An **asset** is any non-code material that the project includes, ships, embeds,
### 1.2 Only open, provable licences
Every asset must be one of:
- **Open source:** ISC, MIT, BSD-2-Clause, BSD-3-Clause, Apache-2.0, OFL-1.1, Ubuntu-Font-1.0 or Bitstream-Vera.
- **Open source:** ISC, MIT, BSD-2-Clause, BSD-3-Clause, Apache-2.0, OFL-1.1, Ubuntu-Font-1.0, Bitstream-Vera or Unicode-3.0 (Unicode, Inc. data such as ICU4X's line-breaking dictionaries; already allowed for code in `deny.toml`).
- **Creative Commons:** CC0-1.0, CC-BY-4.0 or CC-BY-SA-4.0.
- **Public domain.**
- **Contributor-original:** created, photographed or generated by a contributor who owns it and licenses it under the project licence (`MIT OR Apache-2.0`).
+2 -1
View File
@@ -267,7 +267,7 @@ Every asset PdfCraft includes, bundles or uses to build its published material,
| `assets/icons/a-large-small.svg` | Lucide icon "a-large-small" | Lucide Contributors (portions Cole Bemis, Feather) | ISC AND MIT | https://github.com/lucide-icons/lucide (npm lucide-static 1.49.0, icons/a-large-small.svg) | UI icon, embedded via crates/ui-egui/src/widgets/icon_data.rs |
| `assets/icons/move-horizontal.svg` | Lucide icon "move-horizontal" | Lucide Contributors (portions Cole Bemis, Feather) | ISC AND MIT | https://github.com/lucide-icons/lucide (npm lucide-static 1.49.0, icons/move-horizontal.svg) | UI icon, embedded via crates/ui-egui/src/icon_data.rs |
## Compiled in through dependencies (6)
## Compiled in through dependencies (7)
| Crate | File | Title | Author | Licence | Source | Used for |
|---|---|---|---|---|---|---|
@@ -277,6 +277,7 @@ Every asset PdfCraft includes, bundles or uses to build its published material,
| `epaint_default_fonts` 0.37.0 | `fonts/emoji-icon-font.ttf` | emoji-icon-font | John Slegers | MIT | https://github.com/jslegers/emoji-icon-font | egui monochrome fallback font (egui feature monochrome_emoji_fonts) |
| `epaint_default_fonts` 0.37.0 | `fonts/egui-icons.ttf` | egui icons (monochrome emoji-icon-font subset) | John Slegers | MIT | https://github.com/jslegers/emoji-icon-font | Small special-icon subset embedded by egui default_fonts; retained in FontDefinitions::default(). Provenance: https://github.com/storytold/egui/blob/6d518fc799681571963842f95b55a6b68431b15a/crates/epaint_default_fonts/fonts/egui-icons.txt; complete MIT copyright and permission text in NOTICE. |
| `hayro-interpret` 0.7.0 | `src/font/generated/metrics.rs` | Standard-14 font metrics and encodings (as Rust tables) (non-visual data, AGENTS.md §1.1) | Adobe (Core 14 AFM metrics, PDF specification encodings); tables by The Hayro Authors | MIT OR Apache-2.0 | ISO 32000-2 Annex D; Adobe Core14 AFM files (via vendored hayro-interpret) | Glyph widths and encodings for non-embedded standard fonts. Non-visual data, allowed by AGENTS.md §1.1 |
| `icu_segmenter_data` 2.3.0 | `data/segmenter_dictionary_extended_v1.rs.data` | Southeast Asian word dictionaries for line breaking (Thai, Lao, Khmer, Burmese) | The ICU4X Project Developers; word lists from the ICU project (Unicode, Inc.) | Unicode-3.0 | https://github.com/unicode-org/icu4x (via icu_segmenter_data) | Word boundaries for wrapping Thai text in Edit text and Add text (crates/edit/src/linebreak.rs). Non-visual data; the line segmenter loads only these dictionaries, not the Chinese and Japanese one |
## Downloaded at build time (22)
+12
View File
@@ -2263,6 +2263,18 @@ adobe_data = true
usage = "Glyph widths and encodings for non-embedded standard fonts. Non-visual data, allowed by AGENTS.md §1.1"
sha256 = "2ec9e608573eda459de1e1d9bda3bb494315e36bee885acc88912a11fe0caa41"
[[bundled]]
crate = "icu_segmenter_data"
version = "2.3.0"
path = "data/segmenter_dictionary_extended_v1.rs.data"
title = "Southeast Asian word dictionaries for line breaking (Thai, Lao, Khmer, Burmese)"
author = "The ICU4X Project Developers; word lists from the ICU project (Unicode, Inc.)"
source = "https://github.com/unicode-org/icu4x (via icu_segmenter_data)"
licence = "Unicode-3.0"
kind = "data"
usage = "Word boundaries for wrapping Thai text in Edit text and Add text (crates/edit/src/linebreak.rs). Non-visual data; the line segmenter loads only these dictionaries, not the Chinese and Japanese one"
sha256 = "cad3821ca44fdb88ed856ff3336444aa200afe7b9d19e34ac28ae6a4459d4c6d"
[[fetched]]
file = "PlayfairDisplay[wght].ttf"
url = "https://raw.githubusercontent.com/google/fonts/9710da1eacb3be272583c3224dcb70f9da6eadbb/ofl/playfairdisplay/PlayfairDisplay%5Bwght%5D.ttf"
Generated
+23
View File
@@ -3133,6 +3133,28 @@ dependencies = [
"zerovec",
]
[[package]]
name = "icu_segmenter"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "82d07aafccd67af15d02512a6adf5896fbc5ed00f2e99b471d2efa14016db3db"
dependencies = [
"icu_collections",
"icu_locale_fallback",
"icu_provider",
"icu_segmenter_data",
"potential_utf",
"smallvec",
"utf8_iter",
"zerovec",
]
[[package]]
name = "icu_segmenter_data"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae293c039020f9ec10710af98d29ce6aa2051486638b49c9a6409f3b4a9e98ad"
[[package]]
name = "idna"
version = "1.1.0"
@@ -4399,6 +4421,7 @@ name = "pdfcraft-edit"
version = "0.18.1"
dependencies = [
"hayro-syntax",
"icu_segmenter",
"kurbo 0.13.1",
"pdfcraft-annot",
"pdfcraft-content",
+14 -1
View File
@@ -16,7 +16,7 @@ sha256 = "cd7032ea33b71427f188e34ffb4aee9d6868a57fd196037c5c8b2be03a2a0352"
[[document]]
source = "NOTICE"
destination = "NOTICE"
sha256 = "9f854f730719af962cd840529324d089e05e68a30bd245c50e33f96e009b4314"
sha256 = "d9ba162ad2bc95d22ab9885065e253bb823bd6795068828e7b39858344b660eb"
[[document]]
source = "assets/app-icon/LICENSE-document.txt"
@@ -60,6 +60,13 @@ destination = "licenses/CMap-LICENSE.txt"
sha256 = "742665db9c8e1bc72603c6d319ca3e90b83bd6d95202f0cd4ef11068a07a9c29"
origin = "https://static.crates.io/crates/hayro-cmap/hayro-cmap-0.1.0.crate#assets/LICENSE.txt"
[[document]]
source = "licenses/ICU4X-LICENSE.txt"
licences = ["Unicode-3.0"]
destination = "licenses/ICU4X-LICENSE.txt"
sha256 = "f367c1b8e1aa262435251e442901da4607b4650e0e63a026f5044473ecfb90f2"
origin = "https://static.crates.io/crates/icu_segmenter_data/icu_segmenter_data-2.3.0.crate#LICENSE"
[[document]]
source = "licenses/egui-Hack-Regular.txt"
licences = ["MIT", "Bitstream-Vera"]
@@ -164,3 +171,9 @@ crate = "hayro-interpret"
path = "src/font/generated/metrics.rs"
licence = "MIT OR Apache-2.0"
documents = ["vendor/hayro-interpret/LICENSE-MIT", "vendor/hayro-interpret/LICENSE-APACHE"]
[[bundled]]
crate = "icu_segmenter_data"
path = "data/segmenter_dictionary_extended_v1.rs.data"
licence = "Unicode-3.0"
documents = ["licenses/ICU4X-LICENSE.txt"]
+29
View File
@@ -36,6 +36,9 @@ list (author, source, license) is in ATTRIBUTION.md.
- egui default fonts (epaint_default_fonts, via eframe) — Ubuntu Light (Ubuntu Font Licence 1.0),
Hack (MIT / Bitstream Vera licence), Noto Emoji (SIL OFL 1.1), emoji-icon-font (MIT).
- Colour profile in hayro-interpret — CGATS001Compat-v2-micro.icc (CC0-1.0, Clinton Ingram).
- ICU4X line-breaking data (icu_segmenter_data, via crates.io) — Unicode-3.0. Its Thai, Lao, Khmer
and Burmese word dictionaries are compiled in for wrapping Thai text; notice below, full licence in
licenses/ICU4X-LICENSE.txt.
Required notices:
@@ -48,6 +51,32 @@ Required notices:
(https://github.com/adobe-type-tools/cmap-resources/blob/master/LICENSE.txt). Neither the name of
Adobe nor the names of its contributors are used to endorse or promote this product.
ICU4X segmenter data (bundled by icu_segmenter_data 2.3.0; word lists, not visual assets):
Copyright © 2020-2024 Unicode, Inc. Portions adapted from ICU4C and/or ICU4J:
ICU 1.8.1 to ICU 57.1 © 1995-2016 International Business Machines Corporation and others.
Permission is hereby granted, free of charge, to any person obtaining a copy of data files and any
associated documentation (the "Data Files") or software and any associated documentation (the
"Software") to deal in the Data Files or Software without restriction, including without
limitation the rights to use, copy, modify, merge, publish, distribute, and/or sell copies of the
Data Files or Software, and to permit persons to whom the Data Files or Software are furnished to
do so, provided that either (a) this copyright and permission notice appear with all copies of
the Data Files or Software, or (b) this copyright and permission notice appear in associated
Documentation.
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS.
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM,
OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA FILES OR SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not be used in
advertising or otherwise to promote the sale, use or other dealings in these Data Files or
Software without prior written authorization of the copyright holder.
Showcase PDF (built by `cargo xtask demo-pdf`, not committed): fonts are downloaded at build time from
google/fonts at a pinned commit, verified by SHA-256, all under the SIL Open Font License 1.1 — see the
"Downloaded at build time" table in ATTRIBUTION.md.
+1
View File
@@ -244,6 +244,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
- **2026-11-03 (M5, #5850):** Fill & Sign Enter inserts a line break in a growing editor; clicking outside commits and Escape cancels. Nonempty text keeps leading, internal and trailing line breaks. The headless fill_sign_add path sizes its typewriter box for the same wrapped lines so saved appearances show every row. Production-order keyboard and engine save/reopen/render regressions exercise the UI and headless callers. Estimates unchanged.
- **2026-11-03 (community, M7, Thai word breaking):** Thai rows end between words, found with ICU4X's dictionary line segmenter (`icu_segmenter` 2.3, compiled dictionary data, no LSTM model; Unicode-3.0), instead of only at spaces when rewrapping and at any cluster in Add text. A row never starts with a following vowel, tone or other mark, Mai Yamok or Paiyannoi, nor ends with a leading vowel; a word wider than the row still breaks between clusters, and the spaces where an Add text row ends are dropped as in Latin text. Rewrapping existing text works out the breaks once per run of non-space characters. Rewriting a paragraph into Thai still needs a source font with the Thai glyphs (there is no Thai fallback face for it yet). The compiled dictionaries (Thai, Lao, Khmer, Burmese) add about 1.8 MB to a stripped release binary. Estimates unchanged.
- **2026-11-03 (M7, #5835):** In image crop mode, a right-click on the image, its crop box or its handles keeps the crop and opens no menu, so the menu no longer switches as the pointer moves. A right-click elsewhere clears the selection and opens the page menu. UI tests cover all four page rotations, overlapping images, a 45° image, and a check that the Done/Cancel/Remove Crop pills at a page edge still keep the crop.
- **2026-11-02 (M12, #1297):** `model_add` embeds a U3D or PRC file, recognised by its header, as a 3D annotation with a chosen activation and a poster (an image or PDF page fitted in the box, or a labelled frame); an older file's catalog `/Version` is raised to what the model needs. Views, a rendered poster and a UI command are later steps. Estimates unchanged.
+3
View File
@@ -21,6 +21,9 @@ kurbo = { workspace = true }
# Display order of right-to-left lines in added text, and reading and writing order in Edit text;
# already in the tree via the interface.
unicode-bidi = "0.3"
# Thai word boundaries for rewrapping and Add text (dictionary data compiled in, no LSTM model);
# the ICU4X 2.3 crates it builds on are already in the tree.
icu_segmenter = { version = "2.3", default-features = false, features = ["compiled_data"] }
[dev-dependencies]
hayro-syntax = { workspace = true }
+2
View File
@@ -29,6 +29,8 @@ use draw::{GLYPHS_PER_FONT, LaidLine, draw_laid_shaped};
use layout::Piece;
#[cfg(test)]
pub(crate) use layout::WRAP_CALLS;
#[cfg(test)]
pub(crate) use layout::shaped_width;
pub use layout::{PASTE_STEP, first_undrawable, lines, paste_place, paste_rect, text_rect, text_width};
pub use store::{add_content, delete_content, isolate_content, list_added, scan_markers, update_content};
pub use types::{Added, AddedImage, AddedText, Align, CHAR_SPACING_LIMIT, Family, Ink, MarkerScan};
+72 -9
View File
@@ -73,8 +73,11 @@ pub(super) fn wrapped(t: &AddedText) -> Vec<(String, bool, usize)> {
out
}
/// Thai wrapping at shaped cluster boundaries never detaches a tone/vowel mark from its base.
/// This is glyph-safe wrapping, not dictionary-based Thai word segmentation.
/// Thai wrapping: a row ends before a word that would overflow it, the words found by the
/// dictionary ([`crate::linebreak`]) inside runs without spaces and at the spaces between them. The
/// spaces where a row ends are dropped, as in the Latin wrapper. Only a word wider than the row is
/// broken inside, at shaped cluster boundaries, which never detach a tone or vowel mark from its
/// base.
fn thai_lines(t: &AddedText, para: &str, width: f64) -> Result<Vec<String>, EditError> {
let (pieces, _) = shaped_layout(t, para, false)?;
let mut atoms = Vec::new();
@@ -88,20 +91,80 @@ fn thai_lines(t: &AddedText, para: &str, width: f64) -> Result<Vec<String>, Edit
})),
}
}
let mut lines = Vec::new();
let words = thai_words(atoms);
let blank = |s: &str| s.chars().all(char::is_whitespace);
let mut lines: Vec<String> = Vec::new();
let (mut line, mut used) = (String::new(), 0.0);
for (text, advance) in atoms {
if !line.is_empty() && used + advance > width {
lines.push(std::mem::take(&mut line));
let end_row = |lines: &mut Vec<String>, line: &mut String| {
let done = std::mem::take(line);
lines.push(done.trim_end().to_string());
};
for word in words {
// The word's own width: the spaces after it may hang past the row's end.
let ink: f64 = word.iter().filter(|(s, _)| !blank(s)).map(|(_, w)| w).sum();
// A row holding only the spaces that open the paragraph has nothing to end yet.
if !blank(&line) && used + ink > width {
end_row(&mut lines, &mut line);
used = 0.0;
}
line.push_str(&text);
used += advance;
// Too wide for what is left of the row, counting the spaces that open the paragraph.
let too_wide = blank(&line) && used + ink > width;
for (text, advance) in word {
if too_wide && !blank(&line) && !blank(&text) && used + advance > width {
end_row(&mut lines, &mut line);
used = 0.0;
}
// A row after a wrap does not begin with the spaces where the last one ended.
if line.is_empty() && !lines.is_empty() && blank(&text) {
continue;
}
line.push_str(&text);
used += advance;
}
}
lines.push(line);
Ok(lines)
}
/// The shaped clusters `atoms` (text and advance) grouped into words: a word starts after spaces,
/// or where [`crate::linebreak::breaks`] lets a row end inside a run without spaces, and keeps the
/// spaces that follow it. A break that falls inside a cluster is not one.
fn thai_words(atoms: Vec<(String, f64)>) -> Vec<Vec<(String, f64)>> {
let text: String = atoms.iter().map(|(s, _)| s.as_str()).collect();
let mut cuts = Vec::new();
let mut run_start: Option<usize> = None;
for (at, c) in text.char_indices().chain(std::iter::once((text.len(), ' '))) {
match (c.is_whitespace(), run_start) {
(false, None) => {
cuts.push(at);
run_start = Some(at);
}
(true, Some(start)) => {
let run = text.get(start..at).unwrap_or_default();
cuts.extend(crate::linebreak::breaks(run).into_iter().map(|b| start.saturating_add(b)));
run_start = None;
}
_ => {}
}
}
// `cuts` is in order: each run's start, then the breaks inside it, then the next run's start.
let mut words: Vec<Vec<(String, f64)>> = Vec::new();
let mut cuts = cuts.into_iter().peekable();
let mut offset = 0usize;
for atom in atoms {
while cuts.peek().is_some_and(|&c| c < offset) {
cuts.next();
}
let starts_word = cuts.peek() == Some(&offset);
offset = offset.saturating_add(atom.0.len());
match words.last_mut() {
Some(word) if !starts_word => word.push(atom),
_ => words.push(vec![atom]),
}
}
words
}
pub(super) fn is_thai(c: char) -> bool {
matches!(u32::from(c), 0x0E00..=0x0E7F)
}
@@ -257,7 +320,7 @@ pub(super) fn shaped_layout(t: &AddedText, line: &str, rtl: bool) -> Result<(Vec
/// The width of `s` in a shaped-text item; the item's font's estimate if it can't be shaped (drawing
/// then reports why).
fn shaped_width(t: &AddedText, s: &str, rtl: bool) -> f64 {
pub(crate) fn shaped_width(t: &AddedText, s: &str, rtl: bool) -> f64 {
shaped_layout(t, s, rtl).map(|(_, w)| w).unwrap_or_else(|_| latin_width(t, s))
}
+142 -14
View File
@@ -6,8 +6,56 @@
//! and quotes, currency signs before an amount): the line-start and line-end prohibitions of
//! JIS X 4051 ("kinsoku"), which UAX #14 gives as classes CL, CP, EX, IS, NS and OP, PR.
//!
//! Korean is written with spaces between words and keeps breaking only there; so does Thai, whose
//! words need a dictionary to find.
//! Thai is also written without spaces between words, but a row may not end between any two of its
//! characters: its words need a dictionary to find. They come from ICU4X's dictionary line
//! segmenter (UAX #14 with dictionary breaking for class SA), kept only where a Thai character is on
//! either side. A row never begins with a following vowel, a tone or other mark, Mai Yamok (ๆ) or
//! Paiyannoi (ฯ), nor ends with a leading vowel (เ แ โ ใ ไ), whatever the dictionary says.
//!
//! Korean is written with spaces between words and keeps breaking only there.
use std::sync::LazyLock;
use icu_segmenter::options::LineBreakOptions;
use icu_segmenter::{LineSegmenter, LineSegmenterBorrowed};
/// The dictionary segmenter, built once: its data is compiled in, so building it cannot fail.
static SEGMENTER: LazyLock<LineSegmenterBorrowed<'static>> = LazyLock::new(|| LineSegmenter::new_dictionary(LineBreakOptions::default()));
/// Letters, vowels, marks, digits and signs of the Thai block.
fn thai(c: char) -> bool {
matches!(c, '\u{0e01}'..='\u{0e5b}')
}
/// Thai that belongs to the character before it: Paiyannoi, the following vowels, Lakkhangyao,
/// Mai Yamok and the tone and other marks.
fn thai_no_start(c: char) -> bool {
matches!(c, '\u{0e2f}' | '\u{0e30}'..='\u{0e3a}' | '\u{0e45}'..='\u{0e4e}')
}
/// The leading vowels, written before the consonant they follow in speech.
fn thai_no_end(c: char) -> bool {
matches!(c, '\u{0e40}'..='\u{0e44}')
}
/// Byte offsets where the dictionary lets a row of Thai end inside `word`.
fn thai_breaks(word: &str) -> Vec<usize> {
if !word.chars().any(thai) {
return Vec::new();
}
SEGMENTER
.segment_str(word)
.filter(|&at| {
let before = word.get(..at).and_then(|s| s.chars().next_back());
let after = word.get(at..).and_then(|s| s.chars().next());
match (before, after) {
(Some(b), Some(a)) => (thai(b) || thai(a)) && !thai_no_start(a) && !thai_no_end(b) && !no_start(a) && !no_end(b),
// The start and end of the word are not breaks inside it.
_ => false,
}
})
.collect()
}
/// Writing with no spaces between words, where a row may end between two characters (UAX #14 ID).
fn ideographic(c: char) -> bool {
@@ -77,29 +125,49 @@ fn no_end(c: char) -> bool {
)
}
/// Whether a row may end between `before` and `after`, neither of them a space.
pub(crate) fn between(before: char, after: char) -> bool {
(ideographic(before) || ideographic(after)) && !no_start(after) && !no_end(before)
/// Whether a row may end between `before` and `after` in Chinese or Japanese, neither of them a
/// space. Thai needs the whole word ([`breaks`]).
fn between(before: char, after: char) -> bool {
(ideographic(before) || ideographic(after)) && !no_start(after) && !no_end(before) && !thai_no_start(after) && !thai_no_end(before)
}
/// Byte offsets inside `word` (text without spaces) where a row may end, in order: never 0 or the
/// word's length, always on a character boundary.
pub(crate) fn breaks(word: &str) -> Vec<usize> {
let mut out: Vec<usize> = Vec::new();
let mut prev: Option<char> = None;
for (at, c) in word.char_indices() {
if let Some(before) = prev
&& between(before, c)
{
out.push(at);
}
prev = Some(c);
}
out.extend(thai_breaks(word));
out.sort_unstable();
out.dedup();
out
}
/// `word` (text without spaces) cut where a row may end inside it; the pieces join with nothing.
pub(crate) fn pieces(word: &str) -> Vec<&str> {
let mut out = Vec::new();
let mut start = 0;
let mut prev: Option<char> = None;
for (at, c) in word.char_indices() {
if let Some(before) = prev
&& between(before, c)
{
out.extend(word.get(start..at));
start = at;
}
prev = Some(c);
for at in breaks(word) {
out.extend(word.get(start..at));
start = at;
}
out.extend(word.get(start..).filter(|rest| !rest.is_empty()));
out
}
/// For each character of `word` (text without spaces), whether a row may end just before it.
pub(crate) fn breaks_before_chars(word: &str) -> Vec<bool> {
let cuts = breaks(word);
word.char_indices().map(|(at, _)| cuts.binary_search(&at).is_ok()).collect()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -121,4 +189,64 @@ mod tests {
assert_eq!(pieces(""), Vec::<&str>::new());
assert!(!between('a', 'b') && between('字', 'a') && !between('字', '。') && !between('「', '字'));
}
#[test]
fn thai_breaks_between_dictionary_words() {
assert_eq!(pieces("ภาษาไทยไม่มีช่องว่างระหว่างคำ"), ["ภาษา", "ไทย", "ไม่มี", "ช่อง", "ว่าง", "ระหว่าง", "คำ"]);
assert_eq!(pieces("น้ำประปาสะอาด"), ["น้ำ", "ประปา", "สะอาด"]);
assert_eq!(pieces("การประปาส่วนภูมิภาค"), ["การ", "ประปา", "ส่วน", "ภูมิภาค"]);
}
#[test]
fn thai_vowels_marks_and_repetition_stay_with_the_word_before_them() {
// The dictionary alone would start a row with Mai Yamok, with the Sara Aa of a decomposed
// Sara Am, or with a closing bracket.
assert_eq!(pieces("เด็กๆชอบกินขนม"), ["เด็กๆ", "ชอบ", "กิน", "ขนม"]);
assert_eq!(pieces("ท\u{0e4d}\u{0e32}งาน"), ["ท\u{0e4d}\u{0e32}", "งาน"]);
assert_eq!(pieces("กรุงเทพฯเป็นเมืองหลวง"), ["กรุงเทพฯ", "เป็น", "เมือง", "หลวง"]);
assert_eq!(pieces("สวัสดีครับ(ทดสอบ)"), ["สวัสดี", "ครับ", "(ทดสอบ)"]);
}
#[test]
fn thai_keeps_numbers_and_latin_whole() {
assert_eq!(pieces("ราคา100บาทต่อเดือน"), ["ราคา", "100บาท", "ต่อ", "เดือน"]);
assert_eq!(pieces("พ.ศ.2569"), ["พ.ศ.2569"]);
assert_eq!(pieces("PDF"), ["PDF"]);
assert_eq!(breaks("ไทย"), Vec::<usize>::new());
}
#[test]
fn breaks_before_chars_marks_the_first_character_of_each_later_piece() {
let flags = breaks_before_chars("น้ำประปา");
let starts: Vec<usize> = flags.iter().enumerate().filter_map(|(i, &b)| b.then_some(i)).collect();
// น ้ ำ | ป ร ะ ป า
assert_eq!(starts, [3]);
assert_eq!(breaks_before_chars(""), Vec::<bool>::new());
}
#[test]
fn pieces_of_arbitrary_thai_rejoin_to_the_input_and_never_start_with_a_mark() {
// A deterministic generator over Thai, Latin, digits and CJK, including mark-only runs and
// unusual orders that no dictionary word has.
let alphabet: Vec<char> = ('\u{0e01}'..='\u{0e5b}').chain(['a', '1', '(', ')', '字', 'の', '.']).collect();
let mut state: u64 = 0x2545_f491_4f6c_dd1d;
for _ in 0..2000 {
let mut word = String::new();
state = state.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);
let len = (state >> 59) as usize + 1;
for _ in 0..len {
state = state.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);
let pick = (state >> 33) as usize % alphabet.len();
word.extend(alphabet.get(pick));
}
let parts = pieces(&word);
assert_eq!(parts.concat(), word);
assert!(parts.iter().all(|p| !p.is_empty()), "{word:?} gave {parts:?}");
for part in parts.iter().skip(1) {
let first = part.chars().next().unwrap_or(' ');
assert!(!thai_no_start(first) && !no_start(first), "{word:?} gave {parts:?}");
}
assert_eq!(breaks_before_chars(&word).len(), word.chars().count());
}
}
}
+51
View File
@@ -6612,6 +6612,57 @@ fn added_thai_text_wraps_clusters_and_survives_reopen() {
assert!(arabic_font_names(&doc, 0).is_empty());
}
#[test]
fn added_thai_text_wraps_between_dictionary_words() {
if pdfcraft_fonts::document_thai_font().is_none() {
eprintln!("skipping the Thai word wrapping checks: built without craft-fonts (set CRAFT_FONTS_DIR)");
return;
}
let text = "การประปาส่วนภูมิภาคให้บริการน้ำประปาที่สะอาดแก่ประชาชนทั่วประเทศ";
let t = AddedText { rect: [0.0, 0.0, 90.0, 400.0], text: text.into(), size: 16.0, ..AddedText::default() };
let rows = super::added::lines(&t);
assert!(rows.len() > 2, "{rows:?}");
assert_eq!(rows.concat(), text, "no space is added or lost between the rows");
// Every row ends where the dictionary lets one end, never inside a word such as ประปา.
let cuts = crate::linebreak::breaks(text);
let mut at = 0;
for row in rows.iter().take(rows.len() - 1) {
at += row.len();
assert!(cuts.contains(&at), "row {row:?} ends inside a word: {rows:?}");
}
// Spaces where a row ends are dropped, as for Latin text, and a word wider than the row is
// still broken between clusters.
let spaced =
AddedText { text: "น้ำประปา สะอาด ปลอดภัย".into(), rect: [0.0, 0.0, 60.0, 400.0], ..t.clone() };
let rows = super::added::lines(&spaced);
assert!(rows.iter().all(|r| !r.starts_with(' ') && !r.ends_with(' ')), "{rows:?}");
assert_eq!(rows.concat(), "น้ำประปาสะอาดปลอดภัย");
let narrow = AddedText { rect: [0.0, 0.0, 16.0, 200.0], text: "กิ่กิ่กิ่".into(), ..t.clone() };
assert_eq!(super::added::lines(&narrow), ["กิ่", "กิ่", "กิ่"]);
// Spaces that open a paragraph are kept, but they are not a row of their own: a first word
// too wide to fit after them stays on the first row (or breaks between clusters there).
for width in [16.0, 40.0, 60.0, 90.0] {
let indented = AddedText { rect: [0.0, 0.0, width, 400.0], text: " ประปาสะอาด".into(), ..t.clone() };
let rows = super::added::lines(&indented);
assert!(rows.iter().all(|r| !r.trim().is_empty()), "an empty row at width {width}: {rows:?}");
assert!(rows[0].starts_with(" "), "the opening spaces are kept at width {width}: {rows:?}");
assert_eq!(rows.concat().trim_start(), "ประปาสะอาด");
}
// A first word that fits a row alone but not after the opening spaces breaks between clusters
// on the first row rather than running past its end.
let word = "ประปา";
let word_w = super::added::shaped_width(&t, word, false);
let indent_w = super::added::shaped_width(&t, " ", false);
let width = word_w + indent_w / 2.0;
let indented = AddedText { rect: [0.0, 0.0, width, 400.0], text: format!(" {word}"), ..t.clone() };
let rows = super::added::lines(&indented);
for row in &rows {
assert!(super::added::shaped_width(&t, row, false) <= width + 0.5, "{row:?} overflows {width}: {rows:?}");
}
assert!(rows.len() > 1 && rows[0].starts_with(" "), "{rows:?}");
assert_eq!(rows.concat().trim_start(), word);
}
#[test]
fn group_moves_preserve_glyph_bytes_and_unselected_text_state() {
for content in [
+12
View File
@@ -276,6 +276,18 @@ mod tests {
assert_eq!(wrapped("一行目 二行目", 4.0), ["一行目 ", "二行目"]);
}
#[test]
fn thai_text_wraps_between_dictionary_words_and_keeps_marks_on_their_word() {
// Without word breaks the whole run is one word and overflows a single row.
assert_eq!(wrapped("ภาษาไทยไม่มีช่องว่าง", 8.0), ["ภาษาไทย", "ไม่มี", "ช่องว่าง"]);
// Mai Yamok stays with the word it repeats; pieces join without a space, words keep theirs.
assert_eq!(wrapped("เด็กๆชอบกิน ขนม", 6.0), ["เด็กๆ", "ชอบกิน", "ขนม"]);
assert_eq!(wrapped("ไฟล์ PDF ประปาส่วนภูมิภาค", 9.0), ["ไฟล์ PDF", "ประปาส่วน", "ภูมิภาค"]);
// A wrap between Thai words consumed no separator, so a rejoin puts back no space.
let rows = wrap("ภาษาไทยไม่มีช่องว่าง", 8.0, |s| s.chars().count() as f64);
assert_eq!(rows.iter().map(|(_, soft, gap)| (*soft, *gap)).collect::<Vec<_>>(), [(false, 0), (true, 0), (true, 0)]);
}
#[test]
fn fallback_face_moves_on_when_the_best_face_lacks_a_character() {
static JPAN: &[&str] = &["Jpan"];
+27 -7
View File
@@ -414,18 +414,41 @@ fn row_step(style: &BlockStyle, size: f64, source_leading: f64) -> f64 {
type WrappedStyledRows = (StyledRows, RowRanges, Vec<usize>, Vec<bool>, Vec<usize>);
/// For each of `chars`, whether a row may end just before it inside a run of non-space characters
/// (Chinese, Japanese and Thai), worked out once per run: Thai words need the whole run, not just
/// two neighbours.
fn styled_break_before(chars: &[StyledChar]) -> Vec<bool> {
let mut break_before = vec![false; chars.len()];
let mut run_start = 0;
while run_start < chars.len() {
let mut run_end = run_start;
while chars.get(run_end).is_some_and(|c| !c.0.is_whitespace()) {
run_end += 1;
}
let run: String = chars.get(run_start..run_end).unwrap_or_default().iter().map(|c| c.0).collect();
for (i, cut) in crate::linebreak::breaks_before_chars(&run).into_iter().enumerate() {
if let Some(slot) = break_before.get_mut(run_start.saturating_add(i)) {
*slot = cut;
}
}
run_start = run_end.max(run_start.saturating_add(1));
}
break_before
}
fn wrap_styled_rows(
chars: &[StyledChar],
width: f64,
measure: impl Fn(&[StyledChar]) -> Result<f64, EditError>,
) -> Result<WrappedStyledRows, EditError> {
let break_before = styled_break_before(chars);
let mut rows: Vec<Vec<StyledChar>> = Vec::new();
let mut row_ranges = Vec::new();
// True when this row starts because the previous word did not fit, not because of a newline.
let mut row_soft = Vec::new();
let mut next_soft = false;
// How many separator characters a wrap trimmed off the row that just ended, for the row
// about to start (0 for a Chinese/Japanese word split, which trims none; meaningless unless
// about to start (0 for a Chinese, Japanese or Thai word split, which trims none; meaningless unless
// the row it belongs to is soft). Mirrors `next_soft`.
let mut row_gap = Vec::new();
let mut next_gap = 0usize;
@@ -458,11 +481,8 @@ fn wrap_styled_rows(
}
let mut end = at + 1;
if !item.0.is_whitespace() {
// A word ends at a space, or where Chinese or Japanese text may break between characters.
while let (Some(before), Some(current)) = (chars.get(end - 1), chars.get(end))
&& !current.0.is_whitespace()
&& !crate::linebreak::between(before.0, current.0)
{
// A word ends at a space, or where Chinese, Japanese or Thai text may break inside it.
while chars.get(end).is_some_and(|c| !c.0.is_whitespace()) && !break_before.get(end).copied().unwrap_or(false) {
end += 1;
}
}
@@ -790,7 +810,7 @@ fn push_whole_paragraph_marker(block_ops: &mut Vec<Op>, marked: bool, op: &str)
/// An empty marker between two rows of one rewrite. `/PCHard` is a newline that was already in
/// the text. `/PCSoft` is a wrap and rejoins when the box widens, with `/N` the exact number of
/// separator characters it consumed (0 for a Chinese/Japanese word split), so a rejoin restores
/// separator characters it consumed (0 for a Chinese, Japanese or Thai word split), so a rejoin restores
/// what was actually there instead of guessing a single space (#2529). Neither is a paragraph
/// region, so the lines stay in one scope.
fn push_row_break_marker(block_ops: &mut Vec<Op>, row_soft: &[bool], row_gap: &[usize], row_index: usize) {
+46
View File
@@ -0,0 +1,46 @@
UNICODE LICENSE V3
COPYRIGHT AND PERMISSION NOTICE
Copyright © 2020-2024 Unicode, Inc.
NOTICE TO USER: Carefully read the following legal agreement. BY
DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
Permission is hereby granted, free of charge, to any person obtaining a
copy of data files and any associated documentation (the "Data Files") or
software and any associated documentation (the "Software") to deal in the
Data Files or Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, and/or sell
copies of the Data Files or Software, and to permit persons to whom the
Data Files or Software are furnished to do so, provided that either (a)
this copyright and permission notice appear with all copies of the Data
Files or Software, or (b) this copyright and permission notice appear in
associated Documentation.
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
THIRD PARTY RIGHTS.
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
FILES OR SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall
not be used in advertising or otherwise to promote the sale, use or other
dealings in these Data Files or Software without prior written
authorization of the copyright holder.
SPDX-License-Identifier: Unicode-3.0
—
Portions of ICU4X may have been adapted from ICU4C and/or ICU4J.
ICU 1.8.1 to ICU 57.1 © 1995-2016 International Business Machines Corporation and others.
+2 -2
View File
@@ -2047,8 +2047,8 @@ milestone = "M7"
status = "shipped"
commands = ["edit.text"]
tools = ["page_add_text", "content_update"]
evidence = ["crates/ui-egui/tests/it/content.rs::typing_moving_styling_and_deleting_added_content", "crates/ui-egui/tests/it/content.rs::clicking_elsewhere_keeps_the_text_and_starts_a_new_box", "crates/automation/tests/it/automation.rs::adding_content_through_tools", "crates/edit/src/tests.rs::added_text_and_images_are_page_content_that_stays_editable", "crates/engine/src/tests.rs::added_text_and_images_render_and_stay_editable", "crates/edit/src/tests.rs::added_arabic_text_is_shaped_in_display_order_and_stays_searchable", "crates/automation/tests/it/automation.rs::thai_text_through_tools_saves_reopens_extracts_and_renders", "crates/ui-egui/tests/it/content.rs::typing_thai_text_keeps_marks_and_commits_to_page", "crates/automation/tests/it/automation.rs::added_arabic_text_keeps_copy_search_and_editor_order_after_retyping_and_reopen", "crates/edit/src/tests.rs::added_hebrew_text_is_drawn_right_to_left_and_stays_searchable", "crates/automation/tests/it/automation.rs::hebrew_text_through_tools_saves_reopens_extracts_and_renders", "crates/edit/src/tests.rs::gray_and_cmyk_added_text_writes_g_and_k_until_it_is_recoloured", "crates/automation/tests/it/automation.rs::added_text_takes_a_cmyk_or_gray_ink_through_tools", "crates/ui-egui/src/editing/tests.rs::recolouring_print_ink_text_in_its_box_drops_the_ink_without_an_empty_step"]
notes = "page content, editable later (retype, move, restyle); standard fonts, Windows-1252 text. The colour is RGB, or for print files gray or CMYK (#6440: page_add_text / content_update gray or cmyk, painted with g / k so black can be 100 K, kept through save and reopen; choosing another colour later, also from the Format panel, drops the ink for good and paints in RGB, while choosing the colour the ink already shows (black on 100 K) keeps it; content_update color always paints in RGB). Arabic text is shaped with the craft-fonts Arab face (Type 3 outlines with ToUnicode) and laid out in bidirectional display order; per-line ActualText keeps copy, search and editor text in typing order through retyping and save/reopen; it is drawn regular whatever the bold/italic setting, and typing in the box is still left to right (misc.rtl-ui). Hebrew (#128) takes the same path with the craft-fonts Hebr face (Noto Sans Hebrew): right-to-left display order, points and cantillation kept on their letter as one glyph, ToUnicode and per-line ActualText; regular style only, and right-to-left scripts other than Arabic and Hebrew are not covered. Thai Add Text uses the craft-fonts Thai face, with shaped clusters and ToUnicode; Thai is regular style, wrapping preserves clusters but is not dictionary-based, and forms/Fill and Sign are not covered by this change. Each click on the page keeps the box being typed and starts a new one; Done and Discard beside the box"
evidence = ["crates/ui-egui/tests/it/content.rs::typing_moving_styling_and_deleting_added_content", "crates/ui-egui/tests/it/content.rs::clicking_elsewhere_keeps_the_text_and_starts_a_new_box", "crates/automation/tests/it/automation.rs::adding_content_through_tools", "crates/edit/src/tests.rs::added_text_and_images_are_page_content_that_stays_editable", "crates/engine/src/tests.rs::added_text_and_images_render_and_stay_editable", "crates/edit/src/tests.rs::added_arabic_text_is_shaped_in_display_order_and_stays_searchable", "crates/automation/tests/it/automation.rs::thai_text_through_tools_saves_reopens_extracts_and_renders", "crates/ui-egui/tests/it/content.rs::typing_thai_text_keeps_marks_and_commits_to_page", "crates/edit/src/tests.rs::added_thai_text_wraps_between_dictionary_words", "crates/automation/tests/it/automation.rs::added_arabic_text_keeps_copy_search_and_editor_order_after_retyping_and_reopen", "crates/edit/src/tests.rs::added_hebrew_text_is_drawn_right_to_left_and_stays_searchable", "crates/automation/tests/it/automation.rs::hebrew_text_through_tools_saves_reopens_extracts_and_renders", "crates/edit/src/tests.rs::gray_and_cmyk_added_text_writes_g_and_k_until_it_is_recoloured", "crates/automation/tests/it/automation.rs::added_text_takes_a_cmyk_or_gray_ink_through_tools", "crates/ui-egui/src/editing/tests.rs::recolouring_print_ink_text_in_its_box_drops_the_ink_without_an_empty_step"]
notes = "page content, editable later (retype, move, restyle); standard fonts, Windows-1252 text. The colour is RGB, or for print files gray or CMYK (#6440: page_add_text / content_update gray or cmyk, painted with g / k so black can be 100 K, kept through save and reopen; choosing another colour later, also from the Format panel, drops the ink for good and paints in RGB, while choosing the colour the ink already shows (black on 100 K) keeps it; content_update color always paints in RGB). Arabic text is shaped with the craft-fonts Arab face (Type 3 outlines with ToUnicode) and laid out in bidirectional display order; per-line ActualText keeps copy, search and editor text in typing order through retyping and save/reopen; it is drawn regular whatever the bold/italic setting, and typing in the box is still left to right (misc.rtl-ui). Hebrew (#128) takes the same path with the craft-fonts Hebr face (Noto Sans Hebrew): right-to-left display order, points and cantillation kept on their letter as one glyph, ToUnicode and per-line ActualText; regular style only, and right-to-left scripts other than Arabic and Hebrew are not covered. Thai Add Text uses the craft-fonts Thai face, with shaped clusters and ToUnicode; Thai is regular style; rows end between dictionary words (ICU4X), and a word wider than the row breaks between clusters; forms/Fill and Sign are not covered. Each click on the page keeps the box being typed and starts a new one; Done and Discard beside the box"
[[feature]]
id = "edit.find-replace"
+1
View File
@@ -29,6 +29,7 @@ pub const ALLOWED_LICENCES: &[&str] = &[
"OFL-1.1",
"Ubuntu-font-1.0",
"Bitstream-Vera",
"Unicode-3.0",
"CC0-1.0",
"CC-BY-4.0",
"CC-BY-SA-4.0",