From 92ff21b9f258d411a9ce82813accfec879545757 Mon Sep 17 00:00:00 2001 From: Mouhand Kaddo Date: Sun, 15 Nov 2026 20:29:22 +0400 Subject: [PATCH] Shape and preserve Arabic text in PDF exports (#940) ## Summary Arabic text reached the PDF writer intact, then WinAnsi encoding replaced its characters with question marks. This fixes #923 through the shared PDF text path, including cells, headers and chart labels. ```diff PDF text - encode Arabic through WinAnsi + shape with harfrust and Unicode bidi + embed licensed glyph outlines with matching weight coordinates + preserve logical source using ToUnicode and ActualText + fall back to the standard and CID fonts for characters the Arabic face lacks ``` Font binaries stay in the optional craft-fonts build input. Native unbundled builds can use installed Noto Sans Arabic. Explicit cell reading-direction integration follows separately. **Web build:** shaped Arabic on web needs the bundled craft fonts. The current release workflow doesn't enable `web-craft-fonts`, so until a maintainer enables it, web sets Arabic in the CID font, as main does today, which has no Arabic glyphs, and shows a message that no Noto Sans Arabic face was found. The same applies to a desktop without Noto Sans Arabic, and to bold Arabic where only the regular face is installed. ## Evidence - **Before:** Arabic exported as `?` (#923, v0.6.1); main now sets it in the CID font, which has no Arabic glyphs, so readers show blanks or boxes. **After:** native and bundled exports show joined letters, marks, Western/Arabic digits, bold/italic, wrapping, rotation and chart labels. Fresh `pdfinfo`, `pdffonts` and `pdftotext -bbox` checks accept the files; eight original Arabic strings match extracted logical text exactly. - At the current head, with Noto Sans Arabic installed and no `CRAFT_FONTS_DIR`: the `gridcraft-pdf`, `gridcraft-engine` and `gridcraft-xlsx` tests, clippy, fmt, layers, assets, parity and the wasm check pass. The full `cargo xtask ci` with the craft-fonts bundle relies on the hosted gate. - Characters the Arabic face lacks (emoji, Cyrillic) are set like non-Arabic text, and without an Arabic face for the text's weight Arabic takes the CID path main uses, with one message per export. Only a broken font or a size limit fails the export. Size limits cover text, glyph subsets, outlines and page content. ## Merge Danger **Door:** two-way **Blast Radius:** printing `PdfDoc::finish` now returns a Result and every caller handles it. Latin-only rendering stays on its existing path. This is a glyph-outline PDF solution; platform printer and physical screen-reader behavior are not claimed. --- ATTRIBUTION.md | 4 +- Cargo.lock | 7 + ROADMAP.md | 2 + crates/engine/Cargo.toml | 1 + crates/engine/src/cmd/print.rs | 29 +- crates/engine/src/cmd/print/tests.rs | 76 ++++ crates/engine/src/cmd/print/text.rs | 119 ++++- crates/pdf/Cargo.toml | 9 +- crates/pdf/README.md | 2 +- crates/pdf/build.rs | 88 ++++ crates/pdf/src/arabic.rs | 637 +++++++++++++++++++++++++++ crates/pdf/src/lib.rs | 98 ++++- crates/pdf/src/tests.rs | 264 +++++++++-- docs/file-format-parity.md | 9 +- docs/localization-parity.md | 11 +- 15 files changed, 1293 insertions(+), 63 deletions(-) create mode 100644 crates/pdf/build.rs create mode 100644 crates/pdf/src/arabic.rs diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index 7dd388ccd..d63b32f16 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -47,5 +47,7 @@ Generated-in-code art is original and has no file to list: the UI icon set (`cra | `docs/images/pivot.png` | GridCraft contributors | screenshot of GridCraft rendered offscreen (`crates/ui-egui/examples/snapshot.rs`) showing an original sample workbook | MIT OR Apache-2.0 | README screenshot; no third-party content | | `docs/images/format-chart.png` | GridCraft contributors | screenshot of GridCraft rendered offscreen (`crates/ui-egui/examples/snapshot.rs`) showing an original sample workbook | MIT OR Apache-2.0 | README screenshot; no third-party content | | `docs/images/page-break.png` | GridCraft contributors | screenshot of GridCraft rendered offscreen (`crates/ui-egui/examples/snapshot.rs`) showing an original sample workbook | MIT OR Apache-2.0 | README screenshot; no third-party content | -| Core 14 AFM metric numbers in `crates/pdf/src/symbol.rs` (Symbol: character codes and advance widths) and `crates/pdf/src/metrics.rs` (Helvetica, Times and Courier advance widths); no font file | Adobe Systems Incorporated | Adobe's Core 14 AFM font metrics files (`Symbol.afm` 001.008 and the Helvetica, Times and Courier AFMs) | Adobe's Core 14 metrics notice: the files may be used, copied and distributed for any purpose and without charge, provided the notice is kept (quoted in `crates/pdf/src/metrics.rs`) | Numbers only, typed in from the public metrics. PDF export embeds no font: Symbol and the Times faces are base-14 fonts every PDF reader supplies, and text outside WinAnsiEncoding is set in a predefined, non-embedded Adobe-GB1 CID font | +| Core 14 AFM metric numbers in `crates/pdf/src/symbol.rs` (Symbol: character codes and advance widths) and `crates/pdf/src/metrics.rs` (Helvetica, Times and Courier advance widths); no font file | Adobe Systems Incorporated | Adobe's Core 14 AFM font metrics files (`Symbol.afm` 001.008 and the Helvetica, Times and Courier AFMs) | Adobe's Core 14 metrics notice: the files may be used, copied and distributed for any purpose and without charge, provided the notice is kept (quoted in `crates/pdf/src/metrics.rs`) | Numbers only, typed in from the public metrics. PDF export embeds none of these fonts: Symbol and the Times faces are base-14 fonts every PDF reader supplies, and non-Arabic text outside WinAnsiEncoding is set in a predefined, non-embedded Adobe-GB1 CID font | | Unicode values of the Symbol encoding in `crates/pdf/src/symbol.rs` (the code-to-character map; numbers only, no file copied) | Unicode, Inc. | `symbol.txt`, "Adobe Symbol Encoding to Unicode", table version 1.0 (2011-07-12), at unicode.org/Public/MAPPINGS/VENDORS/ADOBE/ | Unicode, Inc.'s notice in the file: "Unicode, Inc. hereby grants the right to freely use the information supplied in this file in the creation of products supporting the Unicode Standard, and to make copies of this file in any form for internal or external distribution as long as this notice remains attached." The file is provided as-is, without warranties | Separate from Adobe's Core 14 notice (previous row), which covers the character codes and widths; the Unicode values come from this table | + +Arabic PDF text uses Noto Sans Arabic from the optional [craft-fonts build input](https://github.com/storytold/craft-fonts), pinned by the release workflow. Author: The Noto Project Authors. Licence: SIL Open Font License 1.1. Native unbundled export uses an installed Noto Sans Arabic face. No new asset file is committed. diff --git a/Cargo.lock b/Cargo.lock index 395518b46..53fead994 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2219,6 +2219,7 @@ dependencies = [ "serde", "serde_json", "thiserror 2.0.21", + "unicode-segmentation", "web-time", "zip 3.0.0", ] @@ -2282,6 +2283,12 @@ dependencies = [ [[package]] name = "gridcraft-pdf" version = "0.19.1" +dependencies = [ + "harfrust", + "sha2 0.11.1", + "skrifa", + "unicode-bidi", +] [[package]] name = "gridcraft-sandbox" diff --git a/ROADMAP.md b/ROADMAP.md index fd27ee9eb..bfd19a634 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -149,6 +149,7 @@ Ranked; estimates in Opus 5.5 agent-hours. Detail and the full beta-gate list in | 2026-10-31 | Recently Used lists the functions most recently entered in a formula from any source (ribbon, control channel, MCP, scripts), kept with the preferences and returned by `formulas.recentlyUsed`; the Insert Function dialog takes Excel's title and a working category list (Most Recently Used, All, each category) | | 2026-10-31 | Formulas › Function Library as a big Insert Function button, the AutoSum split button and three columns of labelled small buttons, each category book in its own colour; category menus end with Insert Function… on that category; More Functions cascades into the remaining categories; Home › Editing shares the AutoSum menu; the MR key tip opens Recently Used | | 2026-10-31 | System and embedded CJK fallback text shares the Latin baseline in cells, editors and window titles (#1265, #2621). Symbol and emoji alignment, manual row heights and clipping are unchanged | +| 2026-10-26 | Arabic PDF export shapes licensed glyph outlines and preserves searchable logical text; characters the font lacks, or a missing Arabic font, fall back to the standard fonts instead of failing the export | | 2026-10-26 | Arabic bidi display/editing preserve logical source text; per-cell reading direction is editable and survives undo, XLSX and conditional-format priority | | 2026-10-26 | Arabic text shows in cells and the interface through a system, fontconfig or craft-fonts Arabic face; workbook fonts keep their script fallbacks | | 2026-10-25 | AutoSum over a selection does what Excel does: a selected row gets one total to its right (in its last cell if that is empty), not a one-cell total under each cell; a block whose last column is empty gets each row's total there, one whose last row is empty each column's, and with both the corner gets the total of the block; otherwise totals go under each column as before | @@ -286,6 +287,7 @@ Ranked; estimates in Opus 5.5 agent-hours. Detail and the full beta-gate list in | 2026-10-31 | minor | Progress log: Recently Used and Insert Function categories | | 2026-10-31 | minor | Progress log: Function Library | | 2026-10-31 | minor | Progress log: shared CJK/Latin baselines in cells, editors and window titles | +| 2026-10-26 | minor | Progress log: Arabic PDF text | | 2026-10-26 | minor | Progress log: Arabic bidi display and editing | | 2026-10-26 | minor | Progress log: Arabic font fallback | | 2026-10-25 | minor | Progress log: AutoSum over a selected row, and a selection's empty last row and column | diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml index 9481b0476..c68fb117a 100644 --- a/crates/engine/Cargo.toml +++ b/crates/engine/Cargo.toml @@ -26,6 +26,7 @@ gridcraft-chart = { workspace = true } gridcraft-pdf = { workspace = true } gridcraft-math = { workspace = true } image = { workspace = true } +unicode-segmentation = "=1.13.3" [dev-dependencies] gridcraft-xlsx = { workspace = true, features = ["test-support"] } diff --git a/crates/engine/src/cmd/print.rs b/crates/engine/src/cmd/print.rs index 875ebdc2b..c951a7af4 100644 --- a/crates/engine/src/cmd/print.rs +++ b/crates/engine/src/cmd/print.rs @@ -19,7 +19,9 @@ use gridcraft_model::{ use gridcraft_pdf::{Font, Page, PdfDoc, Point, Rect, Rgb}; use serde_json::{Value as Json, json}; -use super::{CommandSpec, Result, Session, bad, bool_param, cmd, edit, f64_param, has_doc, split_sheet, str_param, target_sheet, u32_param}; +use super::{ + CommandSpec, EngineError, Result, Session, bad, bool_param, cmd, edit, f64_param, has_doc, split_sheet, str_param, target_sheet, u32_param, +}; mod draw; mod header_footer; @@ -151,8 +153,17 @@ pub(crate) const PX_TO_PT: f32 = 0.75; /// Most pages one export produces. pub const MAX_PAGES: usize = 2000; +/// A rendered PDF and whether some of its Arabic text was set without an Arabic face. +pub struct RenderedPdf { + pub bytes: Vec, + pub pages: usize, + pub missing_arabic_font: bool, +} + +const MISSING_ARABIC_FONT: &str = "Some Arabic text could not be drawn in this PDF because no Noto Sans Arabic face was found for it."; + /// Renders laid-out sheets to a PDF. -pub fn render_pdf(wb: &Workbook, layouts: &[SheetLayout], file: &str, path: &str) -> (Vec, usize) { +pub fn render_pdf(wb: &Workbook, layouts: &[SheetLayout], file: &str, path: &str) -> Result { let mut doc = PdfDoc::new(); doc.info.title = if wb.props.title.is_empty() { file.to_string() } else { wb.props.title.clone() }; doc.info.author = wb.props.author.clone(); @@ -227,18 +238,24 @@ pub fn render_pdf(wb: &Workbook, layouts: &[SheetLayout], file: &str, path: &str page.link_page(Rect::new(lx, ly, w, h), target_page, Point::new(x, y)); } } - let n = doc.page_count(); - (doc.finish(), n) + let pages = doc.page_count(); + let missing_arabic_font = doc.missing_arabic_font(); + let bytes = doc.finish().map_err(|error| EngineError::Other(error.to_string()))?; + Ok(RenderedPdf { bytes, pages, missing_arabic_font }) } // ---------------------------------------------------------------- commands -fn build_pdf(s: &Session, p: &Json, cmd: &str) -> Result<(Vec, usize)> { +fn build_pdf(s: &mut Session, p: &Json, cmd: &str) -> Result<(Vec, usize)> { let opts = PrintOptions::from_params(p, cmd)?; let d = s.doc()?; let lays = layouts(&d.wb, &opts); let path = d.path.clone().unwrap_or_default(); - Ok(render_pdf(&d.wb, &lays, &d.display_title(), &path)) + let pdf = render_pdf(&d.wb, &lays, &d.display_title(), &path)?; + if pdf.missing_arabic_font { + s.ui_requests.push(crate::UiRequest::Message(MISSING_ARABIC_FONT.into())); + } + Ok((pdf.bytes, pdf.pages)) } pub(crate) fn export_pdf(s: &mut Session, p: &Json) -> Result { diff --git a/crates/engine/src/cmd/print/tests.rs b/crates/engine/src/cmd/print/tests.rs index 5ebd73301..e38ebf55b 100644 --- a/crates/engine/src/cmd/print/tests.rs +++ b/crates/engine/src/cmd/print/tests.rs @@ -15,6 +15,11 @@ fn pdf_of(s: &mut Session, p: Json) -> (Vec, u64) { fn contains(hay: &[u8], needle: &str) -> bool { hay.windows(needle.len()).any(|w| w == needle.as_bytes()) } +fn arabic_face_available() -> bool { + let mut doc = gridcraft_pdf::PdfDoc::new(); + doc.add_page(100.0, 100.0).text(0.0, 50.0, 10.0, Font::Helvetica, [0, 0, 0], "مرحبا"); + !doc.missing_arabic_font() +} #[test] fn chart_data_table_prints_formatted_values_and_follows_recalculation() { @@ -58,6 +63,65 @@ fn pdf_export_keeps_safe_links_on_their_printed_pages() { assert!(!cropped.contains("/Dest [")); } +#[test] +fn arabic_with_characters_its_font_lacks_still_exports() { + let mut s = s(); + s.execute("cell.set", json!({"cell": "A1", "input": "مرحبا 😀"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "مرحبا Привет"})).unwrap(); + let out = s.execute("file.exportPdf", json!({})).unwrap(); + assert_eq!(out["pages"], 1); +} + +#[test] +fn wrapping_keeps_combining_marks_with_their_letters() { + let word = "a\u{301}a\u{301}a\u{301}a\u{301}"; + let lines = wrap_lines(word, Font::Helvetica, 10.0, 7.0); + assert_eq!(lines.concat(), word); + assert!(lines.iter().all(|line| line == "a\u{301}")); +} + +#[test] +fn arabic_cells_headers_and_charts_export_searchable_shaped_text() { + if !arabic_face_available() && std::env::var_os("CRAFT_FONTS_REQUIRED").is_none() { + return; + } + let mut s = s(); + s.execute("range.setValues", json!({"range": "A1", "values": [["مَرْحَبًا 123 (ABC)", "القيمة"], ["تفاح", 3], ["كمثرى", 5], ["العَرَبِيَّة العَرَبِيَّة العَرَبِيَّة العَرَبِيَّة العَرَبِيَّة", null], ["تقرير ١٢٣ (ABC)", null], ["مرحبا", null]]})).unwrap(); + s.execute("home.columnWidth", json!({"cols": "A:B", "chars": 32})).unwrap(); + s.execute("home.rowHeight", json!({"rows": "1:6", "height": 50})).unwrap(); + s.execute("home.fontSize", json!({"range": "A1:B6", "size": 20})).unwrap(); + s.execute("home.rowHeight", json!({"rows": "4:4", "height": 90})).unwrap(); + s.execute("home.bold", json!({"range": "A2"})).unwrap(); + s.execute("home.italic", json!({"range": "A3"})).unwrap(); + s.execute("home.wrapText", json!({"range": "A4"})).unwrap(); + s.execute("home.orientation", json!({"range": "A6", "angle": 30})).unwrap(); + s.execute("pageLayout.pageSetup", json!({"header": "&Cتقرير المبيعات 2026", "gridlines": true})).unwrap(); + s.execute("insert.chart", json!({"range": "A1:B3", "type": "column"})).unwrap(); + { + let d = s.doc_mut().unwrap(); + let wb = std::sync::Arc::make_mut(&mut d.wb); + for chart in &mut wb.sheet_mut(0).unwrap().charts { + chart.anchor.cell = CellRef::new(8, 0); + chart.anchor.width = 500.0; + chart.anchor.height = 280.0; + } + } + let (pdf, pages) = pdf_of(&mut s, json!({"fitToPage": true})); + assert_eq!(pages, 1); + assert!(contains(&pdf, "/Subtype /Type3")); + for text in ["مَرْحَبًا 123 (ABC)", "تقرير المبيعات 2026", "القيمة", "تفاح", "كمثرى"] { + let utf16 = text.encode_utf16().map(|unit| format!("{unit:04X}")).collect::(); + assert!(contains(&pdf, &format!("/ActualText ")), "missing text {text}"); + } + let lines = wrap_lines("العَرَبِيَّةالعَرَبِيَّة", Font::Helvetica, 20.0, 40.0); + assert_eq!(lines.concat(), "العَرَبِيَّةالعَرَبِيَّة"); + assert!(lines.iter().all(|line| !line.starts_with(['َ', 'ِ', 'ُ', 'ّ', 'ْ']))); + assert!(lines.iter().all(|line| gridcraft_pdf::text_width(Font::Helvetica, 20.0, line) <= 40.0)); + let path = std::env::temp_dir().join(format!("gridcraft-arabic-export-{}.pdf", std::process::id())); + std::fs::write(&path, &pdf).unwrap(); + eprintln!("Arabic PDF artifact: {}", path.display()); +} + #[test] fn header_sections_split_join_and_field_names() { let spec = "&L&\"Arial,Bold\"Sales &&Co&CPage &P of &N&R&D"; @@ -774,6 +838,18 @@ fn wrapping_a_long_word_in_a_narrow_box_stops_at_the_line_limit() { assert_eq!(wrap_lines(¶, Font::Helvetica, 10.0, 1.0e9), vec![para.clone()]); } +#[test] +fn wrapping_a_long_arabic_word_measures_pieces_not_the_rest_of_the_word() { + let word = "مرحبا".repeat(10_000); + if !arabic_face_available() { + return; + } + let lines = wrap_lines(&word, Font::Helvetica, 10.0, 20.0); + assert!(lines.len() > 2000 && lines.len() <= 2002, "{}", lines.len()); + assert!(lines.iter().take(2000).all(|line| gridcraft_pdf::text_width(Font::Helvetica, 10.0, line) <= 20.0)); + assert!(word.starts_with(&lines.concat())); +} + #[test] fn center_across_spans_following_empty_center_across_cells() { let mut styles = StyleTable::default(); diff --git a/crates/engine/src/cmd/print/text.rs b/crates/engine/src/cmd/print/text.rs index 0523aa4ff..8523132fb 100644 --- a/crates/engine/src/cmd/print/text.rs +++ b/crates/engine/src/cmd/print/text.rs @@ -1,5 +1,6 @@ use super::draw::{Block, Look, rgb_of}; use super::{CellRef, Color, Font, HAlign, MAX_COLS, Page, Point, RangeRef, Sheet, Style, StyleTable, Underline, VAlign, Value}; +use unicode_segmentation::UnicodeSegmentation; const PAD_PX: f32 = 2.0; @@ -11,7 +12,11 @@ const MAX_WRAP_LINES: usize = 2000; /// Widths are kept as running sums of glyph widths (the sum `text_width` makes), so each /// character is measured once: re-measuring the line for every word, or the rest of an /// over-long word for every piece cut from it, made a long paragraph or word quadratic. +/// Shaped text is not a sum of glyph widths, so [`wrap_shaped_lines`] measures it as drawn. pub(super) fn wrap_lines(text: &str, font: Font, size: f32, width: f32) -> Vec { + if gridcraft_pdf::is_shaped(font, text) { + return wrap_shaped_lines(text, font, size, width); + } let units_of = |s: &str| -> u64 { s.chars().map(|c| u64::from(font.char_width(c))).sum() }; let fits = |units: u64| units as f32 * size / 1000.0 <= width; let space = units_of(" "); @@ -33,13 +38,11 @@ pub(super) fn wrap_lines(text: &str, font: Font, size: f32, width: f32) -> Vec Vec Vec Vec { + let fits = |s: &str| gridcraft_pdf::text_width(font, size, s) <= width; + let mut out = Vec::new(); + for para in text.split('\n') { + let mut line = String::new(); + for word in para.split(' ') { + if !line.is_empty() { + let candidate = format!("{line} {word}"); + if fits(&candidate) { + line = candidate; + continue; + } + out.push(std::mem::take(&mut line)); + } + line = if fits(word) { word.to_string() } else { break_shaped_word(word, &fits, &mut out) }; + if out.len() > MAX_WRAP_LINES { + return out; + } + } + out.push(line); + } + out +} + +/// Cuts `word` into the longest grapheme prefixes that fit (at least one grapheme each), pushes +/// all but the last to `out` and returns the last. Each cut gallops from one grapheme, so it +/// measures prefixes about as long as the piece rather than the rest of the word. +fn break_shaped_word(word: &str, fits: &impl Fn(&str) -> bool, out: &mut Vec) -> String { + let ends: Vec = word.grapheme_indices(true).map(|(i, g)| i + g.len()).collect(); + let (mut start, mut first) = (0, 0); + loop { + let len = ends.len().saturating_sub(first); + let end = |count: usize| ends.get(first + count.saturating_sub(1)).copied().unwrap_or(word.len()); + let fits_prefix = |count: usize| fits(word.get(start..end(count)).unwrap_or_default()); + let (mut fit, mut over, mut probe) = (1, len + 1, 2); + while fit < len { + let count = probe.min(len); + if fits_prefix(count) { + fit = count; + probe = count.saturating_mul(2); + } else { + over = count; + break; + } + } + if fit >= len { + return word.get(start..).unwrap_or_default().to_string(); + } + while over - fit > 1 { + let mid = fit + (over - fit) / 2; + if fits_prefix(mid) { + fit = mid; + } else { + over = mid; + } + } + let cut = end(fit); + out.push(word.get(start..cut).unwrap_or_default().to_string()); + start = cut; + first += fit; + if out.len() > MAX_WRAP_LINES { + return word.get(start..).unwrap_or_default().to_string(); + } + } +} + fn font_for(style: &Style) -> Font { Font::styled(Font::family_for(&style.font.name), style.font.bold, style.font.italic) } @@ -203,12 +273,25 @@ pub(super) fn draw_text(pg: &mut Page, b: &Block, placement: TextPlacement<'_>) (true, true) => vec![text.lines().next().unwrap_or_default().to_string()], (false, _) => vec![text], }; - let line_h = size * 1.2; + let extents = lines + .iter() + .filter_map(|line| pg.text_vertical_extents(font, size, line)) + .reduce(|a, b| (a.0.max(b.0), a.1.max(b.1))) + .map(|(ascent, descent)| (ascent.max(size * 0.8), descent.max(size * 0.22))); + let line_h = extents.map_or(size * 1.2, |(ascent, descent)| (ascent + descent + size * 0.2).max(size * 1.2)); let n = lines.len().max(1) as f32; - let first_baseline = match st.align.v { - VAlign::Top => y + pad + size * 0.8, - VAlign::Center | VAlign::Justify | VAlign::Distributed => y + (h - n * line_h) / 2.0 + line_h * 0.5 + size * 0.33, - VAlign::Bottom => y + h - pad - size * 0.22 - (n - 1.0) * line_h, + let first_baseline = if let Some((ascent, descent)) = extents { + match st.align.v { + VAlign::Top => y + pad + ascent, + VAlign::Center | VAlign::Justify | VAlign::Distributed => y + (h - (n - 1.0) * line_h - ascent - descent) / 2.0 + ascent, + VAlign::Bottom => y + h - pad - descent - (n - 1.0) * line_h, + } + } else { + match st.align.v { + VAlign::Top => y + pad + size * 0.8, + VAlign::Center | VAlign::Justify | VAlign::Distributed => y + (h - n * line_h) / 2.0 + line_h * 0.5 + size * 0.33, + VAlign::Bottom => y + h - pad - size * 0.22 - (n - 1.0) * line_h, + } }; // Overflow into empty neighbours (single-line text, not merged). // Center Across Selection spans the empty Center Across cells to its right; the text stays within that run. @@ -233,9 +316,9 @@ pub(super) fn draw_text(pg: &mut Page, b: &Block, placement: TextPlacement<'_>) for (i, t) in lines.iter().enumerate() { let i = i as f32; let (ox, oy) = if deg > 0.0 { - (x + pad + size * 0.8 * deg.to_radians().sin() + i * line_h, y + h - pad) + (x + pad + extents.map_or(size * 0.8, |e| e.0) * deg.to_radians().sin() + i * line_h, y + h - pad) } else { - (x + pad + (last - i) * line_h, y + pad + size * 0.8) + (x + pad + (last - i) * line_h, y + pad + extents.map_or(size * 0.8, |e| e.0)) }; pg.text_rotated(Point::new(ox, oy), size, font, color, t, angle); } @@ -244,12 +327,12 @@ pub(super) fn draw_text(pg: &mut Page, b: &Block, placement: TextPlacement<'_>) } if rot == 255 { // Vertical stacked text. - let chars: Vec = lines.concat().chars().take(500).collect(); - let mut yy = y + pad + size * 0.8; + let stacked = lines.concat(); + let chars: Vec<&str> = stacked.graphemes(true).take(500).collect(); + let mut yy = y + pad + extents.map_or(size * 0.8, |e| e.0); for ch in chars { - let s = ch.to_string(); - let cw = gridcraft_pdf::text_width(font, size, &s); - pg.text(x + (w - cw) / 2.0, yy, size, font, color, &s); + let cw = gridcraft_pdf::text_width(font, size, ch); + pg.text(x + (w - cw) / 2.0, yy, size, font, color, ch); yy += line_h; if yy > y + h + line_h { break; diff --git a/crates/pdf/Cargo.toml b/crates/pdf/Cargo.toml index cd63456c2..9a32a8620 100644 --- a/crates/pdf/Cargo.toml +++ b/crates/pdf/Cargo.toml @@ -3,10 +3,17 @@ name = "gridcraft-pdf" version.workspace = true edition.workspace = true license.workspace = true -description = "GridCraft's tiny self-contained PDF 1.7 writer (standard 14 fonts, vector drawing, RGBA images)" +description = "GridCraft PDF 1.7 writer with shaped Arabic text, vector drawing and RGBA images" readme = "README.md" [lints] workspace = true [dependencies] +harfrust = "=0.12.0" +skrifa = { workspace = true } +unicode-bidi = "=0.3.18" + +[build-dependencies] +sha2 = { workspace = true } +skrifa = { workspace = true } diff --git a/crates/pdf/README.md b/crates/pdf/README.md index cdb6308eb..c8ec1c9b3 100644 --- a/crates/pdf/README.md +++ b/crates/pdf/README.md @@ -1,6 +1,6 @@ # gridcraft-pdf -A small, dependency-free PDF 1.7 writer for vector drawing and RGBA images. +A small PDF 1.7 writer for shaped Arabic text, vector drawing and RGBA images. ## API migration diff --git a/crates/pdf/build.rs b/crates/pdf/build.rs new file mode 100644 index 000000000..a5f132434 --- /dev/null +++ b/crates/pdf/build.rs @@ -0,0 +1,88 @@ +use std::fmt::Write as _; + +fn main() -> Result<(), String> { + arabic_fonts() +} + +/// Embed only Arabic fallback faces from the optional, separately licensed font bundle. +fn arabic_fonts() -> Result<(), String> { + use std::path::PathBuf; + println!("cargo::rerun-if-env-changed=CRAFT_FONTS_DIR"); + println!("cargo::rerun-if-env-changed=CRAFT_FONTS_REQUIRED"); + let required = std::env::var_os("CRAFT_FONTS_REQUIRED").is_some(); + let entries = match std::env::var_os("CRAFT_FONTS_DIR").map(PathBuf::from) { + Some(dir) => arabic_font_entries(&dir), + None if required => Err("CRAFT_FONTS_REQUIRED needs CRAFT_FONTS_DIR".into()), + None => Ok(String::new()), + }; + let entries = match entries { + Ok(entries) => entries, + Err(e) if required => return Err(e), + Err(e) => { + println!("cargo::warning=building without bundled Arabic fonts: {e}"); + String::new() + } + }; + let out = PathBuf::from(std::env::var_os("OUT_DIR").unwrap_or_default()).join("arabic_fonts.rs"); + std::fs::write(&out, format!("pub(crate) static ARABIC_FONTS: &[(&str, &[u8])] = &[\n{entries}];\n")) + .map_err(|e| format!("writing {}: {e}", out.display())) +} + +fn arabic_font_entries(dir: &std::path::Path) -> Result { + use sha2::{Digest, Sha256}; + use skrifa::MetadataProvider; + let root = dir.canonicalize().map_err(|e| format!("craft-fonts directory: {e}"))?; + let manifest = root.join("fonts/manifest.txt"); + println!("cargo::rerun-if-changed={}", manifest.display()); + let text = String::from_utf8(read_font_input(&manifest, 1 << 20)?).map_err(|e| e.to_string())?; + let mut out = String::new(); + let mut has_regular = false; + for (line_number, line) in text.lines().enumerate() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let fields: Vec<_> = line.split(" | ").map(str::trim).collect(); + let [family, style, file, scripts, _licence, licence_file, checksum, _source] = fields.as_slice() else { + return Err(format!("malformed craft-fonts manifest line {}", line_number + 1)); + }; + if !scripts.split(',').any(|script| script.trim() == "Arab") { + continue; + } + let path = root.join(file).canonicalize().map_err(|e| format!("{file}: {e}"))?; + let licence = root.join(licence_file).canonicalize().map_err(|e| format!("{licence_file}: {e}"))?; + if !path.starts_with(&root) || !licence.starts_with(&root) { + return Err("craft-fonts paths must stay inside the checkout".into()); + } + println!("cargo::rerun-if-changed={}", path.display()); + println!("cargo::rerun-if-changed={}", licence.display()); + let bytes = read_font_input(&path, 64 << 20)?; + if Sha256::digest(&bytes).iter().map(|byte| format!("{byte:02x}")).collect::() != *checksum { + return Err(format!("checksum mismatch for {file}")); + } + let font = skrifa::FontRef::from_index(&bytes, 0).map_err(|e| format!("invalid Arabic font {file}: {e}"))?; + if !"العربية١٢٣".chars().all(|ch| font.charmap().map(ch).is_some()) { + return Err(format!("{file} lacks Arabic glyphs")); + } + let _ = writeln!(out, " ({:?}, include_bytes!({:?})),", format!("craft:{family}:{style}"), path); + has_regular |= *style != "Bold"; + } + if out.is_empty() { + return Err("craft-fonts manifest contains no Arabic face; update the checkout".into()); + } + if !has_regular { + return Err("craft-fonts manifest has only bold Arabic faces; regular text needs a regular or variable face".into()); + } + Ok(out) +} + +fn read_font_input(path: &std::path::Path, limit: u64) -> Result, String> { + use std::io::Read as _; + let file = std::fs::File::open(path).map_err(|e| format!("{}: {e}", path.display()))?; + let mut bytes = Vec::new(); + file.take(limit + 1).read_to_end(&mut bytes).map_err(|e| format!("{}: {e}", path.display()))?; + if bytes.len() as u64 > limit { + return Err(format!("{} exceeds the font input size limit", path.display())); + } + Ok(bytes) +} diff --git a/crates/pdf/src/arabic.rs b/crates/pdf/src/arabic.rs new file mode 100644 index 000000000..9a298d2f8 --- /dev/null +++ b/crates/pdf/src/arabic.rs @@ -0,0 +1,637 @@ +//! Arabic shaping and subset Type3 fonts. Outlines use the same variation coordinates as the +//! shaper; ActualText preserves the logical source independently of visual glyph order. +use std::{borrow::Cow, collections::BTreeMap, fmt::Write as _, sync::OnceLock}; + +use harfrust::{Direction, ShaperData, ShaperInstance, UnicodeBuffer}; +use skrifa::{ + FontRef, MetadataProvider, + instance::Size, + outline::{DrawSettings, OutlinePen}, +}; +use unicode_bidi::BidiInfo; + +use crate::{CID_FONT, Error, Font, dict_obj, n, needs_cid, pdf_string, pdf_utf16_string, stream_obj}; + +include!(concat!(env!("OUT_DIR"), "/arabic_fonts.rs")); + +const MAX_TEXT_BYTES: usize = 128 * 1024; +const MAX_OUTLINE_BYTES: usize = 256 * 1024; +const MAX_SUBSET_BYTES: usize = 64 * 1024 * 1024; +const MAX_SUBSET_GLYPHS: usize = 16_384; +const SYNTHETIC_BOLD_STROKE: f32 = 30.0; + +/// How `text` set in `font` is drawn. Symbol keeps its own one-run encoding, and Arabic without +/// an installed or bundled face falls back to the standard fonts. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Shaping { + Plain, + Shaped, + NoFont, +} + +pub(crate) fn shaping(font: Font, text: &str) -> Shaping { + if font == Font::Symbol || !needed(text) { + return Shaping::Plain; + } + match face(bold(font)) { + Ok(None) => Shaping::NoFont, + Ok(Some(_)) | Err(_) => Shaping::Shaped, + } +} + +pub(crate) fn shaped(font: Font, text: &str) -> bool { + shaping(font, text) == Shaping::Shaped +} + +pub(crate) fn needed(text: &str) -> bool { + text.chars().any(|ch| matches!(ch as u32, 0x0600..=0x06ff | 0x0750..=0x077f | 0x0870..=0x089f | 0x08a0..=0x08ff | 0xfb50..=0xfdff | 0xfe70..=0xfefc | 0x10e60..=0x10e7f | 0x1ee00..=0x1eeff)) +} + +fn bold(font: Font) -> bool { + matches!( + font, + Font::HelveticaBold | Font::HelveticaBoldOblique | Font::TimesBold | Font::TimesBoldItalic | Font::CourierBold | Font::CourierBoldOblique + ) +} + +fn italic(font: Font) -> bool { + matches!( + font, + Font::HelveticaOblique + | Font::HelveticaBoldOblique + | Font::TimesItalic + | Font::TimesBoldItalic + | Font::CourierOblique + | Font::CourierBoldOblique + ) +} + +struct Face { + bytes: Cow<'static, [u8]>, + data: ShaperData, + instance: ShaperInstance, + synthetic_bold: bool, +} + +impl Face { + fn new(bytes: Cow<'static, [u8]>, bold: bool) -> Result { + let font = FontRef::from_index(&bytes, 0).map_err(|e| Error(format!("Invalid Arabic PDF font: {e}")))?; + if !"العربية١٢٣".chars().all(|ch| font.charmap().map(ch).is_some()) { + return Err(Error("The PDF font lacks Arabic glyphs".into())); + } + let data = ShaperData::new(&font); + let instance = ShaperInstance::from_variations(&font, [("wght", if bold { 700.0 } else { 400.0 })]); + Ok(Self { bytes, data, instance, synthetic_bold: false }) + } + + fn font(&self) -> Result, Error> { + FontRef::from_index(&self.bytes, 0).map_err(|e| Error(format!("Invalid Arabic PDF font: {e}"))) + } +} + +#[cfg(test)] +pub(crate) fn bold_face_is_synthetic() -> bool { + face(true).is_ok_and(|face| face.is_some_and(|face| face.synthetic_bold)) +} + +/// The Arabic face for a weight, or `None` when none is bundled or installed. The search runs +/// once per weight, so measuring text never rescans the font directories. +fn face(is_bold: bool) -> Result, Error> { + static REGULAR: OnceLock, Error>> = OnceLock::new(); + static BOLD: OnceLock, Error>> = OnceLock::new(); + let cache = if is_bold { &BOLD } else { ®ULAR }; + cache.get_or_init(|| find_face(ARABIC_FONTS, &system_font_dirs(), is_bold)).as_ref().map(Option::as_ref).map_err(Error::clone) +} + +fn loaded_face(font: Font) -> Result<&'static Face, Error> { + face(bold(font))?.ok_or_else(|| Error("Arabic PDF export needs Noto Sans Arabic".into())) +} + +fn bundled_match(fonts: &[(&str, &'static [u8])], is_bold: bool) -> Option<&'static [u8]> { + fonts + .iter() + .find(|(name, bytes)| if is_bold { name.ends_with(":Bold") || has_weight_axis(bytes) } else { !name.ends_with(":Bold") }) + .map(|&(_, bytes)| bytes) +} + +#[cfg(not(target_arch = "wasm32"))] +fn system_font_dirs() -> Vec { + use std::path::PathBuf; + let mut dirs: Vec = [ + "/usr/share/fonts/noto", + "/usr/share/fonts/google-noto-vf", + "/usr/share/fonts/truetype/noto", + "/usr/share/fonts/TTF", + "/usr/local/share/fonts", + "/usr/local/share/fonts/noto", + "/usr/local/share/fonts/TTF", + "/Library/Fonts", + ] + .into_iter() + .map(Into::into) + .collect(); + if let Some(home) = std::env::var_os("HOME").or_else(|| std::env::var_os("USERPROFILE")) { + let home = PathBuf::from(home); + dirs.extend([home.join(".local/share/fonts"), home.join(".fonts"), home.join("Library/Fonts")]); + } + if let Some(root) = std::env::var_os("WINDIR") { + dirs.push(PathBuf::from(root).join("Fonts")); + } + if let Some(root) = std::env::var_os("LOCALAPPDATA") { + dirs.push(PathBuf::from(root).join("Microsoft/Windows/Fonts")); + } + dirs +} + +#[cfg(target_arch = "wasm32")] +fn system_font_dirs() -> Vec { + Vec::new() +} + +fn find_face(bundled: &[(&str, &'static [u8])], dirs: &[std::path::PathBuf], is_bold: bool) -> Result, Error> { + use std::io::Read as _; + if let Some(bytes) = bundled_match(bundled, is_bold) { + return Face::new(Cow::Borrowed(bytes), is_bold).map(Some); + } + let filename = if is_bold { "NotoSansArabic-Bold.ttf" } else { "NotoSansArabic-Regular.ttf" }; + for dir in dirs { + for name in ["NotoSansArabic.ttf", "NotoSansArabic[wght].ttf", filename] { + let path = dir.join(name); + let file = match std::fs::File::open(&path) { + Ok(file) => file, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, + Err(e) => return Err(Error(format!("Cannot read Arabic PDF font {}: {e}", path.display()))), + }; + let mut bytes = Vec::new(); + file.take((64 << 20) + 1).read_to_end(&mut bytes).map_err(|e| Error(format!("Cannot read Arabic PDF font: {e}")))?; + if bytes.len() > 64 << 20 { + return Err(Error("Arabic PDF font exceeds 64 MiB".into())); + } + if is_bold && name != filename && !has_weight_axis(&bytes) { + continue; + } + return Face::new(Cow::Owned(bytes), is_bold).map(Some); + } + } + if let Some((name, bytes)) = bundled_fallback(bundled, is_bold) { + let mut face = Face::new(Cow::Borrowed(bytes), is_bold)?; + face.synthetic_bold = needs_synthetic_bold(name, bytes, is_bold); + return Ok(Some(face)); + } + Ok(None) +} + +/// The bundled face to use when no face matches the requested weight: regular text never borrows a bold face. +fn bundled_fallback<'a>(fonts: &[(&'a str, &'static [u8])], is_bold: bool) -> Option<(&'a str, &'static [u8])> { + fonts.iter().find(|(name, _)| is_bold || !name.ends_with(":Bold")).copied() +} + +/// A bold request served by a static regular face is emboldened by stroking its outlines. +fn needs_synthetic_bold(name: &str, bytes: &[u8], is_bold: bool) -> bool { + is_bold && !name.ends_with(":Bold") && !has_weight_axis(bytes) +} + +fn has_weight_axis(bytes: &[u8]) -> bool { + FontRef::from_index(bytes, 0).is_ok_and(|font| font.axes().iter().any(|axis| axis.tag() == skrifa::raw::types::Tag::new(b"wght"))) +} + +#[derive(Debug)] +pub(crate) enum Item { + Glyph { id: u32, x: f32, y: f32, source: String }, + Latin { text: String, x: f32 }, + Cid { text: String, x: f32 }, +} + +pub(crate) struct Layout { + pub items: Vec, + pub width: f32, +} + +/// Positions are in 1/1000 em, matching the Type3 font matrix. +pub(crate) fn layout(font: Font, text: &str) -> Result { + if text.len() > MAX_TEXT_BYTES { + return Err(Error("Arabic PDF text exceeds the 128 KiB per-line limit".into())); + } + let face = loaded_face(font)?; + let raw = face.font()?; + let shaper = face.data.shaper(&raw).instance(Some(&face.instance)).build(); + let units = shaper.units_per_em(); + if units <= 0 { + return Err(Error("Arabic PDF font has invalid units per em".into())); + } + let scale = 1000.0 / units as f32; + let single_line: String = text.chars().map(|ch| if is_paragraph_break(ch) { ' ' } else { ch }).collect(); + let text = single_line.as_str(); + let bidi = BidiInfo::new(text, None); + let mut layout = Layout { items: Vec::new(), width: 0.0 }; + for para in &bidi.paragraphs { + let (levels, runs) = bidi.visual_runs(para, para.range.clone()); + for run in runs { + let rtl = levels.get(run.start).is_some_and(|level| level.is_rtl()); + let run_text = text.get(run.clone()).ok_or_else(|| Error("Invalid PDF text boundary".into()))?; + let mut spans = Vec::new(); + let mut start = 0; + let mut embedded = None; + for (i, ch) in run_text.char_indices() { + let supported = embeddable(raw.charmap().map(ch).is_some(), ch); + if embedded.is_some_and(|v| v != supported) { + spans.push((start..i, embedded.unwrap_or(false))); + start = i; + } + embedded = Some(supported); + } + spans.push((start..run_text.len(), embedded.unwrap_or(false))); + if rtl { + spans.reverse(); + } + for (span, embedded) in spans { + let part = run_text.get(span.clone()).ok_or_else(|| Error("Invalid PDF text boundary".into()))?; + if !embedded { + let part = part.chars().map(|ch| if is_line_break(ch) { ' ' } else { ch }); + let part: Vec = if rtl { part.rev().map(mirror_ansi).collect() } else { part.collect() }; + for run in part.chunk_by(|a, b| needs_cid(*a) == needs_cid(*b)) { + let (text, x) = (run.iter().collect(), layout.width); + layout.width += run.iter().map(|&ch| f32::from(font.char_width(ch))).sum::(); + layout.items.push(if run.first().is_some_and(|&ch| needs_cid(ch)) { Item::Cid { text, x } } else { Item::Latin { text, x } }); + } + continue; + } + let mut buffer = UnicodeBuffer::new(); + buffer.push_str(part); + buffer.set_pre_context(run_text.get(..span.start).unwrap_or_default()); + buffer.set_post_context(run_text.get(span.end..).unwrap_or_default()); + buffer.set_direction(if rtl { Direction::RightToLeft } else { Direction::LeftToRight }); + buffer.set_flags(harfrust::BufferFlags::REMOVE_DEFAULT_IGNORABLES); + if needed(part) { + buffer.set_script(harfrust::script::ARABIC); + } + buffer.guess_segment_properties(); + let shaped = shaper.shape(buffer, harfrust::ShapeOptions::new()); + let mut clusters: Vec<_> = shaped.glyph_infos().iter().map(|g| g.cluster as usize).collect(); + clusters.push(part.len()); + clusters.sort_unstable(); + clusters.dedup(); + let mut previous_cluster = None; + for (info, pos) in shaped.glyph_infos().iter().zip(shaped.glyph_positions()) { + if info.glyph_id == 0 { + return Err(Error("The Arabic PDF font produced a missing glyph".into())); + } + let cluster = info.cluster as usize; + let source = if previous_cluster == Some(cluster) { + String::new() + } else { + let index = clusters.binary_search(&cluster).map_err(|_| Error("Invalid PDF glyph cluster".into()))?; + let end = clusters.get(index + 1).copied().ok_or_else(|| Error("Invalid PDF glyph cluster".into()))?; + part.get(cluster..end).ok_or_else(|| Error("Invalid PDF glyph cluster".into()))?.to_string() + }; + previous_cluster = Some(cluster); + layout.items.push(Item::Glyph { + id: info.glyph_id, + x: layout.width + pos.x_offset as f32 * scale, + y: pos.y_offset as f32 * scale, + source, + }); + layout.width += pos.x_advance as f32 * scale; + } + } + } + } + Ok(layout) +} + +pub(crate) fn vertical_extents(font: Font, text: &str) -> Result<(f32, f32), Error> { + let layout = layout(font, text)?; + let face = loaded_face(font)?; + let raw = face.font()?; + let metrics = raw.glyph_metrics(Size::new(1000.0), skrifa::instance::LocationRef::new(face.instance.coords())); + let (mut ascent, mut descent) = (0.0_f32, 0.0_f32); + for item in layout.items { + match item { + Item::Glyph { id, y, .. } => { + let (bottom, top) = if let Some(bounds) = metrics.bounds(id.into()) { + (bounds.y_min, bounds.y_max) + } else { + let glyph = raw.outline_glyphs().get(id.into()).ok_or_else(|| Error("Missing Arabic PDF glyph outline".into()))?; + let mut pen = Pen::default(); + glyph + .draw(DrawSettings::unhinted(Size::new(1000.0), skrifa::instance::LocationRef::new(face.instance.coords())), &mut pen) + .map_err(|e| Error(format!("Cannot measure Arabic PDF glyph: {e}")))?; + if pen.full { + return Err(Error("Arabic PDF glyph exceeds the outline size limit".into())); + } + (pen.bounds[1], pen.bounds[3]) + }; + ascent = ascent.max(y + top); + descent = descent.max(-y - bottom); + } + Item::Latin { .. } | Item::Cid { .. } => { + ascent = ascent.max(800.0); + descent = descent.max(220.0); + } + } + } + Ok((ascent, descent)) +} + +fn mirror_ansi(ch: char) -> char { + match ch { + '(' => ')', + ')' => '(', + '[' => ']', + ']' => '[', + '{' => '}', + '}' => '{', + '<' => '>', + '>' => '<', + '«' => '»', + '»' => '«', + '‹' => '›', + '›' => '‹', + other => other, + } +} + +/// Whether a character is drawn from the embedded Arabic face. Line breaks never are, so the +/// fallback path draws them as spaces even when the face maps them. +fn embeddable(mapped: bool, ch: char) -> bool { + (mapped || is_control(ch)) && !is_line_break(ch) +} + +fn is_line_break(ch: char) -> bool { + matches!(ch, '\n' | '\r' | '\u{2028}' | '\u{2029}') +} + +/// Bidi paragraph separators (class B). A drawn line is one paragraph: left in, they would split it +/// into paragraphs laid out left to right, swapping the parts of a right-to-left line. +fn is_paragraph_break(ch: char) -> bool { + matches!(ch, '\n' | '\r' | '\u{1C}'..='\u{1E}' | '\u{85}' | '\u{2029}') +} + +fn is_control(ch: char) -> bool { + matches!(ch, '\u{061c}' | '\u{200c}'..='\u{200f}' | '\u{202a}'..='\u{202e}' | '\u{2066}'..='\u{2069}') +} + +#[derive(Default)] +pub(crate) struct Subset { + keys: BTreeMap<(u32, bool, bool, String), usize>, + glyphs: Vec, + bytes: usize, + /// Some text on the page set a run in the CJK CID font, so the page must declare it. + pub(crate) uses_cid: bool, +} + +struct Glyph { + path: String, + width: f32, + source: String, + bounds: [f32; 4], + emboldened: bool, +} + +/// The Type3 glyph procedure: a filled outline, also stroked in the text colour when emboldened. +fn glyph_procedure(glyph: &Glyph) -> String { + if glyph.emboldened { + format!("{} 0 d0\n{} w 1 j\n{}B\n", n(glyph.width), n(SYNTHETIC_BOLD_STROKE), glyph.path) + } else { + format!("{} 0 d0\n{}f\n", n(glyph.width), glyph.path) + } +} + +impl Subset { + fn insert(&mut self, font: Font, id: u32, source: String) -> Result { + let key = (id, bold(font), italic(font), source.clone()); + if let Some(index) = self.keys.get(&key) { + return Ok(*index); + } + if self.glyphs.len() >= MAX_SUBSET_GLYPHS { + return Err(Error("PDF page has too many distinct Arabic glyphs".into())); + } + let face = loaded_face(font)?; + let raw = face.font()?; + let location = skrifa::instance::LocationRef::new(face.instance.coords()); + let glyph = raw.outline_glyphs().get(id.into()).ok_or_else(|| Error("Missing Arabic PDF glyph outline".into()))?; + let width = + raw.glyph_metrics(Size::new(1000.0), location).advance_width(id.into()).ok_or_else(|| Error("Missing Arabic PDF glyph width".into()))?; + let mut pen = Pen { shear: if italic(font) { 0.212_556_56 } else { 0.0 }, ..Default::default() }; + glyph.draw(DrawSettings::unhinted(Size::new(1000.0), location), &mut pen).map_err(|e| Error(format!("Cannot draw Arabic PDF glyph: {e}")))?; + if pen.full || self.bytes.saturating_add(pen.path.len()).saturating_add(source.len().saturating_mul(2)) > MAX_SUBSET_BYTES { + return Err(Error("Arabic PDF outlines exceed the page size limit".into())); + } + let index = self.glyphs.len(); + self.bytes += pen.path.len() + source.len() * 2; + let emboldened = face.synthetic_bold; + let grow = if emboldened { SYNTHETIC_BOLD_STROKE / 2.0 } else { 0.0 }; + let [x0, y0, x1, y1] = pen.bounds; + self.glyphs.push(Glyph { path: pen.path, width, source, bounds: [x0 - grow, y0 - grow, x1 + grow, y1 + grow], emboldened }); + self.keys.insert(key, index); + Ok(index) + } + + pub(crate) fn text(&mut self, font: Font, text: &str, size: f32) -> Result { + let layout = layout(font, text)?; + let mut out = format!("/Span << /ActualText {} >> BDC\nBT\n", unicode_string(text, true)); + for item in layout.items { + match item { + Item::Glyph { id, x, y, source } => { + let index = self.insert(font, id, source)?; + let _ = writeln!( + out, + "/A{} {} Tf 1 0 0 1 {} {} Tm <{:02X}> Tj", + index / 255 + 1, + n(size), + n(x * size / 1000.0), + n(y * size / 1000.0), + index % 255 + 1 + ); + } + Item::Latin { text, x } => { + let _ = writeln!(out, "/F{} {} Tf 1 0 0 1 {} 0 Tm {} Tj", font.index() + 1, n(size), n(x * size / 1000.0), pdf_string(&text)); + } + Item::Cid { text, x } => { + self.uses_cid = true; + let _ = writeln!(out, "/F{} {} Tf 1 0 0 1 {} 0 Tm {} Tj", CID_FONT + 1, n(size), n(x * size / 1000.0), pdf_utf16_string(&text)); + } + } + } + out.push_str("ET\nEMC"); + Ok(out) + } + + /// Writes the subset fonts into `out` and returns their page resource entries. + pub(crate) fn write_objects(&self, next: &mut u32, out: &mut Vec, offsets: &mut Vec) -> String { + let mut resources = String::new(); + for (batch, glyphs) in self.glyphs.chunks(255).enumerate() { + let id = *next; + let cmap_id = id + 1; + *next += 2; + let mut procs = String::new(); + let mut names = String::new(); + let mut widths = String::new(); + let mut mappings = Vec::new(); + let mut bounds = [0.0_f32; 4]; + for (i, glyph) in glyphs.iter().enumerate() { + let code = i + 1; + let _ = write!(procs, "/g{code} {} 0 R ", *next); + let _ = write!(names, "/g{code} "); + let _ = write!(widths, "{} ", n(glyph.width)); + let body = glyph_procedure(glyph); + stream_obj(out, offsets, *next, "", body.as_bytes()); + *next += 1; + bounds[0] = bounds[0].min(glyph.bounds[0]); + bounds[1] = bounds[1].min(glyph.bounds[1]); + bounds[2] = bounds[2].max(glyph.bounds[2]); + bounds[3] = bounds[3].max(glyph.bounds[3]); + if !glyph.source.is_empty() { + mappings.push(format!("<{code:02X}> {}\n", unicode_string(&glyph.source, false))); + } + } + let mut cmap = "/CIDInit /ProcSet findresource begin\n12 dict begin\nbegincmap\n/CIDSystemInfo << /Registry (Adobe) /Ordering (UCS) /Supplement 0 >> def\n/CMapName /GridCraftArabicUnicode def\n/CMapType 2 def\n1 begincodespacerange\n<00> \nendcodespacerange\n".to_string(); + for group in mappings.chunks(100) { + let _ = writeln!(cmap, "{} beginbfchar", group.len()); + for mapping in group { + cmap.push_str(mapping); + } + cmap.push_str("endbfchar\n"); + } + cmap.push_str("endcmap\nCMapName currentdict /CMap defineresource pop\nend\nend\n"); + stream_obj(out, offsets, cmap_id, "", cmap.as_bytes()); + dict_obj( + out, + offsets, + id, + &format!( + "<< /Type /Font /Subtype /Type3 /Name /GridCraftArabic /FontBBox [{} {} {} {}] /FontMatrix [0.001 0 0 0.001 0 0] /CharProcs << {procs}>> /Encoding << /Type /Encoding /Differences [1 {names}] >> /FirstChar 1 /LastChar {} /Widths [{widths}] /Resources << >> /ToUnicode {cmap_id} 0 R >>", + n(bounds[0]), + n(bounds[1]), + n(bounds[2]), + n(bounds[3]), + glyphs.len() + ), + ); + let _ = write!(resources, "/A{} {id} 0 R ", batch + 1); + } + resources + } +} + +pub(crate) fn unicode_string(text: &str, bom: bool) -> String { + let mut out = if bom { "'); + out +} + +#[derive(Default)] +struct Pen { + path: String, + point: [f32; 2], + bounds: [f32; 4], + shear: f32, + full: bool, +} + +impl Pen { + fn point(&mut self, x: f32, y: f32) -> [f32; 2] { + let x = x + self.shear * y; + self.bounds[0] = self.bounds[0].min(x); + self.bounds[1] = self.bounds[1].min(y); + self.bounds[2] = self.bounds[2].max(x); + self.bounds[3] = self.bounds[3].max(y); + [x, y] + } + fn op(&mut self, value: &str) { + if self.path.len().saturating_add(value.len()) > MAX_OUTLINE_BYTES { + self.full = true; + } + if !self.full { + self.path.push_str(value); + } + } +} + +impl OutlinePen for Pen { + fn move_to(&mut self, x: f32, y: f32) { + let [px, py] = self.point(x, y); + self.op(&format!("{} {} m\n", n(px), n(py))); + self.point = [x, y]; + } + fn line_to(&mut self, x: f32, y: f32) { + let [px, py] = self.point(x, y); + self.op(&format!("{} {} l\n", n(px), n(py))); + self.point = [x, y]; + } + fn quad_to(&mut self, cx: f32, cy: f32, x: f32, y: f32) { + self.curve_to( + self.point[0] + (cx - self.point[0]) * 2.0 / 3.0, + self.point[1] + (cy - self.point[1]) * 2.0 / 3.0, + x + (cx - x) * 2.0 / 3.0, + y + (cy - y) * 2.0 / 3.0, + x, + y, + ); + } + fn curve_to(&mut self, cx: f32, cy: f32, dx: f32, dy: f32, x: f32, y: f32) { + let [cx, cy] = self.point(cx, cy); + let [dx, dy] = self.point(dx, dy); + let [px, py] = self.point(x, y); + self.op(&format!("{} {} {} {} {} {} c\n", n(cx), n(cy), n(dx), n(dy), n(px), n(py))); + self.point = [x, y]; + } + fn close(&mut self) { + self.op("h\n"); + } +} + +#[cfg(test)] +mod tests { + #[test] + fn fallback_mirrors_only_paired_ansi_punctuation() { + let visual: String = "(مرحبا)".chars().rev().map(super::mirror_ansi).collect(); + assert_eq!(visual, "(ابحرم)"); + assert_eq!("[(<«‹abc›»>)]".chars().rev().map(super::mirror_ansi).collect::(), "[(<«‹cba›»>)]"); + } + + #[test] + fn a_bold_request_served_by_a_static_regular_face_is_emboldened() { + assert!(super::needs_synthetic_bold("craft:Noto Sans Arabic:Regular", b"static", true)); + assert!(!super::needs_synthetic_bold("craft:Noto Sans Arabic:Regular", b"static", false)); + assert!(!super::needs_synthetic_bold("craft:Noto Sans Arabic:Bold", b"static", true)); + let glyph = + |emboldened| super::Glyph { path: "0 0 m 10 0 l 10 10 l h\n".into(), width: 500.0, source: String::new(), bounds: [0.0; 4], emboldened }; + assert_eq!(super::glyph_procedure(&glyph(false)), "500 0 d0\n0 0 m 10 0 l 10 10 l h\nf\n"); + assert_eq!(super::glyph_procedure(&glyph(true)), "500 0 d0\n30 w 1 j\n0 0 m 10 0 l 10 10 l h\nB\n"); + } + + #[test] + fn line_breaks_never_use_the_embedded_face() { + for ch in ['\n', '\r', '\u{2028}', '\u{2029}'] { + assert!(!super::embeddable(true, ch), "{ch:?}"); + } + assert!(super::embeddable(true, 'م')); + assert!(super::embeddable(false, '\u{200f}')); + assert!(!super::embeddable(false, 'A')); + } + + #[test] + fn no_bundled_or_installed_face_is_not_an_error() { + for bold in [false, true] { + assert!(matches!(super::find_face(&[], &[], bold), Ok(None))); + assert!(matches!(super::find_face(&[], &[std::env::temp_dir().join("gridcraft-no-fonts-here")], bold), Ok(None))); + } + } + + #[test] + fn bundled_faces_match_the_requested_weight() { + const REGULAR: &[u8] = b"regular"; + const BOLD: &[u8] = b"bold"; + let bold_first = [("Noto Sans Arabic:Bold", BOLD), ("Noto Sans Arabic", REGULAR)]; + assert_eq!(super::bundled_match(&bold_first, false), Some(REGULAR)); + assert_eq!(super::bundled_match(&bold_first, true), Some(BOLD)); + assert_eq!(super::bundled_match(&[("Noto Sans Arabic", REGULAR)], true), None); + assert_eq!(super::bundled_match(&[], false), None); + assert_eq!(super::bundled_fallback(&[("Noto Sans Arabic:Bold", BOLD)], false), None); + assert_eq!(super::bundled_fallback(&[("Noto Sans Arabic:Regular", REGULAR)], true), Some(("Noto Sans Arabic:Regular", REGULAR))); + } +} diff --git a/crates/pdf/src/lib.rs b/crates/pdf/src/lib.rs index b94aaf6a7..32e525a4d 100644 --- a/crates/pdf/src/lib.rs +++ b/crates/pdf/src/lib.rs @@ -5,20 +5,32 @@ //! WinAnsiEncoding (CJK and other scripts) are set in a predefined, non-embedded Adobe-GB1 CID //! font with the `UniGB-UTF16-H` CMap instead of being replaced by `?`. Symbol uses its own //! built-in encoding ([`symbol_code`]), for Greek letters and mathematical operators. +//! - Arabic is shaped and set in embedded Type3 subsets, with exact shaped advances. //! - Vector drawing: rectangles, lines (solid or dashed), polygons, circles (Bézier), clipping. //! - RGBA images as RGB image XObjects with a soft mask for alpha (run-length encoded). //! - Coordinates are points with a **top-left origin** (y grows down); text `y` is the baseline. //! -//! Streams are written uncompressed (images use the trivial RunLengthDecode filter) so the crate -//! needs no dependencies. +//! Streams are written uncompressed (images use the RunLengthDecode filter). #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] #![forbid(unsafe_code)] +mod arabic; mod metrics; mod symbol; pub use symbol::{symbol_code, symbol_width}; +/// A PDF export could not preserve the requested text. +#[derive(Clone, Debug)] +pub struct Error(String); + +impl std::fmt::Display for Error { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(&self.0) + } +} +impl std::error::Error for Error {} + use std::collections::HashMap; use std::collections::hash_map::DefaultHasher; use std::fmt::Write as _; @@ -202,8 +214,18 @@ impl Font { } } +/// Whether `text` set in `font` is shaped, so its width is not the sum of its characters' widths. +pub fn is_shaped(font: Font, text: &str) -> bool { + arabic::shaped(font, text) +} + /// Width of `text` in points when set in `font` at `size` points. +/// Arabic uses shaped advances. If its font cannot render the text, measurement returns zero +/// and drawing that text causes [`PdfDoc::finish`] to return the rendering error. pub fn text_width(font: Font, size: f32, text: &str) -> f32 { + if arabic::shaped(font, text) { + return arabic::layout(font, text).map(|layout| layout.width * size / 1000.0).unwrap_or(0.0); + } // u64: a u32 sum overflows for ~8 million characters. let units: u64 = text.chars().map(|c| u64::from(font.char_width(c))).sum(); units as f32 * size / 1000.0 @@ -446,6 +468,9 @@ pub struct Page { /// `save` calls refused at the nesting limit; the same number of `restore` calls are ignored. refused: u32, links: Vec, + arabic: arabic::Subset, + error: Option, + missing_arabic_font: bool, } /// Most bytes of content per page before further drawing is dropped (keeps hostile inputs from @@ -706,6 +731,21 @@ impl Page { self.op(&s); } + /// Arabic ink extents above and below the baseline, in points. Standard font text returns + /// `None`. Font errors are recorded for [`PdfDoc::finish`], just as when drawing text. + pub fn text_vertical_extents(&mut self, font: Font, size: f32, text: &str) -> Option<(f32, f32)> { + if !arabic::shaped(font, text) { + return None; + } + match arabic::vertical_extents(font, text) { + Ok((ascent, descent)) => Some((ascent * size / 1000.0, descent * size / 1000.0)), + Err(error) => { + self.error.get_or_insert(error); + None + } + } + } + /// Text with its baseline starting at `(x, y)`. pub fn text(&mut self, x: f32, y: f32, size: f32, font: Font, color: Rgb, text: &str) { self.text_rotated(Point::new(x, y), size, font, color, text, 0.0); @@ -717,9 +757,39 @@ impl Page { if text.is_empty() || !extent(size) || !finite(&[origin.x, origin.y]) { return; } + let (sn, cs) = if angle.is_finite() { angle.sin_cos() } else { (0.0, 1.0) }; + let shaping = arabic::shaping(font, text); + self.missing_arabic_font |= shaping == arabic::Shaping::NoFont; + if shaping == arabic::Shaping::Shaped { + if self.saved.len() >= MAX_SAVE_DEPTH { + return; + } + match self.arabic.text(font, text, size.min(10_000.0)) { + Ok(body) => { + for slot in [Some(font.index()), self.arabic.uses_cid.then_some(CID_FONT)].into_iter().flatten() { + if let Some(f) = self.fonts.get_mut(slot) { + *f = true; + } + } + let mut s = format!("q\n{} {} {} {} {} {} cm\n", n(cs), n(-sn), n(sn), n(cs), n(origin.x), n(self.y(origin.y))); + push_rgb(&mut s, color, false); + s.push(' '); + push_rgb(&mut s, color, true); + let _ = write!(s, "\n{body}\nQ"); + if self.content.len().saturating_add(s.len()) > MAX_PAGE_CONTENT { + self.error.get_or_insert_with(|| Error("PDF page text exceeds the size limit".into())); + } else { + self.op(&s); + } + } + Err(error) => { + self.error.get_or_insert(error); + } + } + return; + } let fi = font.index(); let size = n(size.min(10_000.0)); - let (sn, cs) = if angle.is_finite() { angle.sin_cos() } else { (0.0, 1.0) }; // Clockwise in y-down = counter-clockwise negated in PDF's y-up space. let tm = format!("{} {} {} {} {} {} Tm\n", n(cs), n(-sn), n(sn), n(cs), n(origin.x), n(self.y(origin.y))); // Symbol text is one run in Symbol's own encoding (the CID font has no maths symbols). @@ -957,6 +1027,9 @@ impl PdfDoc { style: Style::default(), saved: Vec::new(), refused: 0, + arabic: arabic::Subset::default(), + error: None, + missing_arabic_font: false, }); let i = self.pages.len() - 1; &mut self.pages[i] @@ -971,6 +1044,12 @@ impl PdfDoc { self.pages.get_mut(i) } + /// Whether some Arabic text was set in the standard and CID fonts, which have no Arabic glyphs, + /// because no Arabic face was found for its weight. + pub fn missing_arabic_font(&self) -> bool { + self.pages.iter().any(|page| page.missing_arabic_font) + } + /// Serializes the document. A document without pages gets one blank Letter page. /// /// Every object is written straight into the output buffer (nothing is staged in an @@ -978,7 +1057,12 @@ impl PdfDoc { /// the ids are contiguous from 1. Objects are therefore in the file in the order they are /// numbered during writing (fonts, then per page its new images, content stream and page /// dictionary), with the catalog, page tree and info dictionary last. - pub fn finish(mut self) -> Vec { + /// + /// Returns an error if the Arabic font is unusable or Arabic text exceeds a size limit. + pub fn finish(mut self) -> Result, Error> { + if let Some(error) = self.pages.iter().find_map(|page| page.error.clone()) { + return Err(error); + } if self.pages.is_empty() { self.add_page(612.0, 792.0); } @@ -1055,6 +1139,8 @@ impl PdfDoc { // confirmed by comparing the pixels before an object is shared. let mut written: HashMap> = HashMap::new(); for (pi, ((page, content), links)) in self.pages.iter().zip(contents).zip(page_links).enumerate() { + let arabic_fonts = page.arabic.write_objects(&mut next, &mut out, &mut offsets); + let page_fonts = format!("{font_dict}{arabic_fonts}"); let mut xobjects = String::new(); for (k, img) in page.images.iter().enumerate() { let known = written.get(&img.hash).and_then(|c| { @@ -1118,7 +1204,7 @@ impl PdfDoc { "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {} {}] /Resources << /Font << {}>>{} >> /Contents {} 0 R{} >>", n(page.width), n(page.height), - font_dict, + page_fonts, xo, content_id, annots @@ -1163,7 +1249,7 @@ impl PdfDoc { let _ = writeln!(Utf8(&mut out), "{off:010} 00000 n "); } let _ = write!(Utf8(&mut out), "trailer\n<< /Size {} /Root 1 0 R /Info 3 0 R >>\nstartxref\n{}\n%%EOF\n", next, xref); - out + Ok(out) } } diff --git a/crates/pdf/src/tests.rs b/crates/pdf/src/tests.rs index b86300ff0..dfd2ffd02 100644 --- a/crates/pdf/src/tests.rs +++ b/crates/pdf/src/tests.rs @@ -43,7 +43,7 @@ fn check_xref(pdf: &[u8]) -> Result> { #[test] fn minimal_document() -> Result<(), Box> { - let pdf = PdfDoc::new().finish(); + let pdf = PdfDoc::new().finish()?; assert!(pdf.starts_with(b"%PDF-1.7")); assert!(pdf.ends_with(b"%%EOF\n")); assert!(check_xref(&pdf)? >= 5); @@ -69,7 +69,7 @@ fn drawing_and_text() -> Result<(), Box> { let p2 = d.add_page(842.0, 595.0); p2.text(10.0, 20.0, 10.0, Font::CourierBold, [0, 0, 0], "Second"); assert_eq!(d.page_count(), 2); - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; assert!(find(&pdf, b"(Hello \\(world\\) \\\\ caf\\351 \\2005) Tj").is_some()); assert!(find(&pdf, b"(Second) Tj").is_some()); @@ -95,7 +95,7 @@ fn uri_and_page_links_are_real_pdf_annotations() -> Result<(), Box> { first.link_uri(0.0, 0.0, 10.0, 10.0, "javascript:alert(1)"); let second = d.add_page(100.0, 200.0); second.link_page(Rect::new(0.0, 0.0, 10.0, 10.0), 99, Point::new(0.0, 0.0)); - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; let s = String::from_utf8_lossy(&pdf); assert_eq!(s.matches("/Subtype /Link").count(), 3); @@ -119,7 +119,7 @@ fn images_with_alpha() -> Result<(), Box> { p.image_rgba(Rect::new(10.0, 10.0, 40.0, 40.0), &px, [4, 4]); p.image_rgba(Rect::new(10.0, 10.0, 40.0, 40.0), &[255; 16], [2, 2]); // opaque: no mask p.image_rgba(Rect::new(0.0, 0.0, 1.0, 1.0), &[1, 2, 3], [4, 4]); // too short: ignored - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; assert!(find(&pdf, b"/SMask").is_some()); assert!(find(&pdf, b"/Im1 Do").is_some()); @@ -187,7 +187,7 @@ fn hostile_inputs_never_panic() -> Result<(), Box> { p.restore(); p.save(); p.image_rgba(Rect::new(0.0, 0.0, 10.0, 10.0), &[], [u32::MAX, u32::MAX]); - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; Ok(()) } @@ -427,7 +427,7 @@ fn corpus_doc(seed: u64) -> (Vec, Vec) { } old_pages.push(old.content); } - (doc.finish(), old_pages) + (doc.finish().unwrap(), old_pages) } #[test] @@ -471,7 +471,7 @@ fn structure_is_valid_for_all_documents() -> Result<(), Box> { check_xref(&pdf)?; parse_objects(&pdf); // every /Length matches its data } - let empty = PdfDoc::new().finish(); + let empty = PdfDoc::new().finish()?; check_xref(&empty)?; parse_objects(&empty); Ok(()) @@ -487,7 +487,7 @@ fn images_decode_to_their_pixels() { p.image_rgba(Rect::new(1.0, 1.0, 20.0, 20.0), px, [*w, *h]); } } - let pdf = d.finish(); + let pdf = d.finish().unwrap(); let (objs, pages) = parse_pages(&pdf); assert_eq!(pages.len(), 2); for page in &pages { @@ -514,7 +514,7 @@ fn opaque_images_have_no_soft_mask() { let p = d.add_page(100.0, 100.0); p.image_rgba(Rect::new(0.0, 0.0, 10.0, 10.0), &[255; 4 * 12], [4, 3]); assert!(p.images[0].alpha.is_none()); - let pdf = d.finish(); + let pdf = d.finish().unwrap(); assert!(find(&pdf, b"/SMask").is_none()); assert_eq!(pdf.windows(15).filter(|w| *w == b"/Subtype /Image").count(), 1); @@ -523,7 +523,7 @@ fn opaque_images_have_no_soft_mask() { px[translucent * 4 + 3] = 7; let mut d = PdfDoc::new(); d.add_page(100.0, 100.0).image_rgba(Rect::new(0.0, 0.0, 10.0, 10.0), &px, [4, 3]); - let (objs, pages) = parse_pages(&d.finish()); + let (objs, pages) = parse_pages(&d.finish().unwrap()); let (_, _, _, alpha) = decode_image(&objs, pages[0].xobjects[0].1); let mut want = vec![255u8; 12]; want[translucent] = 7; @@ -542,7 +542,7 @@ fn identical_images_are_stored_once() -> Result<(), Box> { p.image_rgba(Rect::new(1.0, 1.0, 20.0, 20.0), px, [*w, *h]); p.image_rgba(Rect::new(30.0, 1.0, 20.0, 20.0), px, [*w, *h]); // twice on one page } - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; assert_eq!(pdf.windows(15).filter(|w| *w == b"/Subtype /Image").count(), 2, "one image + one soft mask"); let (_, pages) = parse_pages(&pdf); @@ -567,7 +567,7 @@ fn different_images_are_not_merged() { p.image_rgba(Rect::new(0.0, 0.0, 5.0, 5.0), &b, [4, 1]); p.image_rgba(Rect::new(0.0, 0.0, 5.0, 5.0), &a, [2, 2]); d.add_page(100.0, 100.0).image_rgba(Rect::new(0.0, 0.0, 5.0, 5.0), &a, [4, 1]); - let pdf = d.finish(); + let pdf = d.finish().unwrap(); assert_eq!(pdf.windows(15).filter(|w| *w == b"/Subtype /Image").count(), 6, "three distinct pictures, each with a soft mask (alpha 50)"); } @@ -587,7 +587,7 @@ fn many_distinct_images_on_one_page_are_not_compared_pairwise() { assert_eq!(p.images.len(), N as usize); assert!(p.content.ends_with("/Im1 Do Q\n"), "the repeated picture reuses /Im1"); assert!(p.image_comparisons <= 1, "{} pixel comparisons for {N} distinct pictures", p.image_comparisons); - let pdf = d.finish(); + let pdf = d.finish().unwrap(); assert_eq!(pdf.windows(15).filter(|w| *w == b"/Subtype /Image").count(), N as usize); } @@ -601,7 +601,7 @@ fn hash_collisions_do_not_merge_images() { for p in 0..2 { d.page_mut(p).unwrap().images[0].hash = 42; // forced collision } - let (objs, pages) = parse_pages(&d.finish()); + let (objs, pages) = parse_pages(&d.finish().unwrap()); let ids: Vec = pages.iter().map(|p| p.xobjects[0].1).collect(); assert_ne!(ids[0], ids[1]); assert_eq!(decode_image(&objs, ids[0]).2, vec![10; 12]); @@ -706,7 +706,7 @@ fn save_nesting_is_limited_to_28() -> Result<(), Box> { p.restore(); p.image_rgba(Rect::new(0.0, 0.0, 5.0, 5.0), &[9; 16], [2, 2]); assert!(p.content.contains(" Do")); - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; Ok(()) } @@ -719,7 +719,7 @@ fn finish_closes_open_saves() -> Result<(), Box> { for _ in 0..33 { p.save(); } - let (_, pages) = parse_pages(&d.finish()); + let (_, pages) = parse_pages(&d.finish()?); let i = interpret(&pages[0].content); assert!(i.balanced); assert_eq!(i.qs, 28); @@ -825,7 +825,7 @@ fn every_draw_method_survives_hostile_numbers() -> Result<(), Box> { p.stroke_line_dashed(Point::new(0.0, 0.0), Point::new(1.0, 1.0), 1.0, [0, 0, 0], &[0.0; 100]); let c = page_content(&d); assert!(!c.contains("NaN") && !c.contains("inf")); - let pdf = d.finish(); + let pdf = d.finish()?; check_xref(&pdf)?; Ok(()) } @@ -848,7 +848,7 @@ fn info_strings_with_non_ascii_are_utf16be() -> Result<(), Box> { doc.add_page(200.0, 200.0); doc.info.title = "Übersicht €5 – “Q1” 日本 😀".into(); doc.info.author = "Müller & Søn".into(); - let pdf = doc.finish(); + let pdf = doc.finish()?; assert_eq!(utf16_entry(&pdf, "/Title")?, "Übersicht €5 – “Q1” 日本 😀"); assert_eq!(utf16_entry(&pdf, "/Author")?, "Müller & Søn"); Ok(()) @@ -859,7 +859,7 @@ fn ascii_info_strings_stay_literal_and_escaped() { let mut doc = PdfDoc::new(); doc.add_page(200.0, 200.0); doc.info.title = r"Q1 (draft) a\b".into(); - let pdf = doc.finish(); + let pdf = doc.finish().unwrap(); assert!(find(&pdf, br"/Title (Q1 \(draft\) a\\b)").is_some()); assert!(find(&pdf, b"/Producer (GridCraft PDF)").is_some()); } @@ -869,7 +869,7 @@ fn info_strings_are_capped() -> Result<(), Box> { let mut doc = PdfDoc::new(); doc.add_page(200.0, 200.0); doc.info.title = "日".repeat(100_000); - let pdf = doc.finish(); + let pdf = doc.finish()?; assert_eq!(utf16_entry(&pdf, "/Title")?.chars().count(), MAX_INFO_CHARS); Ok(()) } @@ -880,7 +880,7 @@ fn cjk_text_is_not_replaced_by_question_marks() -> Result<(), Box> { let p = doc.add_page(612.0, 792.0); p.text(10.0, 20.0, 10.0, Font::Helvetica, [0, 0, 0], "签字清单"); p.text(10.0, 40.0, 10.0, Font::Helvetica, [0, 0, 0], "No. 编号 2"); - let pdf = doc.finish(); + let pdf = doc.finish()?; check_xref(&pdf)?; assert!(find(&pdf, b"(????) Tj").is_none(), "CJK text written as question marks"); assert!(find(&pdf, b"<7B7E5B576E055355> Tj").is_some()); @@ -893,12 +893,211 @@ fn cjk_text_is_not_replaced_by_question_marks() -> Result<(), Box> { Ok(()) } +#[test] +fn arabic_is_shaped_and_preserved_or_falls_back_without_a_font() -> Result<(), Box> { + let text = "مَرْحَبًا 123 (ABC)"; + let mut doc = PdfDoc::new(); + let page = doc.add_page(600.0, 300.0); + page.text(30.0, 70.0, 20.0, Font::Helvetica, [0, 0, 0], text); + let layout = arabic::layout(Font::Helvetica, text); + let Ok(layout) = layout else { + assert!(doc.missing_arabic_font()); + check_xref(&doc.finish()?)?; + return Ok(()); + }; + assert!(layout.width > 0.0); + assert!((text_width(Font::Helvetica, 20.0, text) - layout.width * 0.02).abs() < 0.001); + let pdf = doc.finish()?; + check_xref(&pdf)?; + assert_eq!(utf16_entry(&pdf, "/ActualText")?, text); + assert!(find(&pdf, b"/Subtype /Type3").is_some()); + assert!(find(&pdf, b"/ToUnicode").is_some()); + assert!(find(&pdf, b"beginbfchar").is_some()); + assert!(find(&pdf, b"(?) Tj").is_none()); + assert!(find(&pdf, b"/UniGB-UTF16-H").is_none(), "Arabic is never set in the CJK font"); + Ok(()) +} + +#[test] +fn latin_inside_arabic_text_declares_its_page_font() -> Result<(), Box> { + if arabic::layout(Font::Helvetica, "مرحبا").is_err() && arabic::ARABIC_FONTS.is_empty() { + return Ok(()); + } + let mut doc = PdfDoc::new(); + doc.add_page(600.0, 300.0).text(30.0, 70.0, 20.0, Font::Helvetica, [0, 0, 0], "مرحبا\u{2028}عالم"); + let pdf = doc.finish()?; + check_xref(&pdf)?; + assert!(find(&pdf, b"/F1 20 Tf").is_some(), "the line break is drawn as a Helvetica space"); + assert!(find(&pdf, b"/Font << /F1 ").is_some(), "the page declares the font its Latin run uses"); + Ok(()) +} + +#[test] +fn cjk_inside_arabic_text_uses_the_cid_font() -> Result<(), Box> { + if arabic::layout(Font::Helvetica, "مرحبا").is_err() && arabic::ARABIC_FONTS.is_empty() { + return Ok(()); + } + let text = "مرحبا 编号"; + let mut doc = PdfDoc::new(); + doc.add_page(600.0, 300.0).text(30.0, 70.0, 20.0, Font::Helvetica, [0, 0, 0], text); + let pdf = doc.finish()?; + check_xref(&pdf)?; + assert!(find(&pdf, b"<7F1653F7> Tj").is_some(), "the CJK run is set in the CID font"); + assert!(find(&pdf, b"/Encoding /UniGB-UTF16-H").is_some(), "the page declares the CID font"); + assert_eq!(utf16_entry(&pdf, "/ActualText")?, text); + assert!(text_width(Font::Helvetica, 20.0, text) >= text_width(Font::Helvetica, 20.0, "مرحبا") + 40.0, "two full-width ideographs"); + for (text, cid_run) in [("مرحبا 😀", &b" Tj"[..]), ("مرحبا Привет", &b"<041F04400438043204350442> Tj"[..])] { + let mut uncovered = PdfDoc::new(); + uncovered.add_page(600.0, 300.0).text(30.0, 70.0, 20.0, Font::Helvetica, [0, 0, 0], text); + assert!(!uncovered.missing_arabic_font()); + let pdf = uncovered.finish()?; + check_xref(&pdf)?; + assert!(find(&pdf, cid_run).is_some(), "{text}: characters the Arabic face lacks fall back like non-Arabic text"); + assert_eq!(utf16_entry(&pdf, "/ActualText")?, text); + } + Ok(()) +} + +#[test] +fn oversized_arabic_text_is_an_error_instead_of_partial_pdf() -> Result<(), Box> { + let mut doc = PdfDoc::new(); + doc.add_page(600.0, 300.0).text(0.0, 50.0, 10.0, Font::Helvetica, [0, 0, 0], &"ب".repeat(65_537)); + if doc.missing_arabic_font() { + check_xref(&doc.finish()?)?; + return Ok(()); + } + assert!(doc.finish().unwrap_err().to_string().contains("128 KiB")); + Ok(()) +} + +#[test] +fn arabic_joining_marks_styles_and_rotation() -> Result<(), Box> { + if arabic::layout(Font::Helvetica, "مرحبا").is_err() && arabic::ARABIC_FONTS.is_empty() { + return Ok(()); + } + for (text, expected) in [("مرحبا 123 (ABC)", "(ABC)"), ("مرحبا [ABC]", "[ABC]"), ("مرحبا ", "")] { + let mixed = arabic::layout(Font::Helvetica, text)?; + let latin: String = mixed + .items + .iter() + .filter_map(|item| match item { + arabic::Item::Latin { text, .. } => Some(text.as_str()), + _ => None, + }) + .collect(); + if !latin.is_empty() { + assert_eq!(latin, expected); + } + } + let mut metrics = PdfDoc::new(); + let (ascent, descent) = metrics.add_page(100.0, 100.0).text_vertical_extents(Font::Helvetica, 20.0, "كمثرى").unwrap(); + assert!(ascent > 0.0 && descent > 20.0 * 0.22, "Arabic descenders exceed the old Latin baseline allowance"); + assert!( + metrics.page_mut(0).unwrap().text_vertical_extents(Font::Helvetica, 20.0, "مرحبا 123 (ABC)").is_some(), + "spaces in variable fonts have empty outlines" + ); + metrics.finish()?; + let joined = arabic::layout(Font::Helvetica, "سلام")?; + let separate_width: f32 = "سلام".chars().map(|c| text_width(Font::Helvetica, 1000.0, &c.to_string())).sum(); + assert!((joined.width - separate_width).abs() > 10.0, "joining changes advances"); + let marked = arabic::layout(Font::Helvetica, "مَرْحَبًا")?; + let plain = arabic::layout(Font::Helvetica, "مرحبا")?; + assert!(marked.items.len() > plain.items.len()); + assert!(marked.items.iter().any(|item| matches!(item, arabic::Item::Glyph { y, .. } if *y != 0.0))); + let bold_available = match arabic::layout(Font::HelveticaBold, "مرحبا") { + Ok(bold) if arabic::bold_face_is_synthetic() => { + assert!(bold.width >= plain.width, "synthetic bold keeps the regular advances"); + true + } + Ok(bold) => { + assert!(bold.width > plain.width, "bold metrics use the bold font instance"); + true + } + Err(e) => { + assert!(arabic::ARABIC_FONTS.is_empty(), "a bundled face always serves bold text: {e:?}"); + false + } + }; + let mut regular = PdfDoc::new(); + regular.add_page(500.0, 300.0).text(30.0, 60.0, 20.0, Font::Helvetica, [0, 0, 0], "مرحبا"); + let regular = regular.finish()?; + let mut italic = PdfDoc::new(); + italic.add_page(500.0, 300.0).text(30.0, 60.0, 20.0, Font::HelveticaOblique, [0, 0, 0], "مرحبا"); + let italic = italic.finish()?; + assert_ne!(regular, italic, "italic shears the actual outlines"); + let rotated_font = if bold_available { Font::HelveticaBoldOblique } else { Font::HelveticaOblique }; + let mut rotated = PdfDoc::new(); + rotated.add_page(500.0, 300.0).text_rotated(Point::new(100.0, 100.0), 20.0, rotated_font, [0, 0, 0], "مرحبا 123", -std::f32::consts::FRAC_PI_2); + let rotated = rotated.finish()?; + check_xref(&rotated)?; + assert_eq!(utf16_entry(&rotated, "/ActualText")?, "مرحبا 123"); + assert!(find(&rotated, b"0 1 -1 0 100 200 cm").is_some()); + Ok(()) +} + +#[test] +fn line_breaks_in_arabic_text_draw_as_spaces() -> Result<(), Box> { + if arabic::layout(Font::Helvetica, "مرحبا").is_err() && arabic::ARABIC_FONTS.is_empty() { + return Ok(()); + } + for text in ["مرحبا\r\nعالم", "مرحبا\u{2028}عالم", "مرحبا\u{2029}ABC\r"] { + let layout = arabic::layout(Font::Helvetica, text)?; + for item in &layout.items { + if let arabic::Item::Latin { text: drawn, .. } = item { + assert!(!drawn.chars().any(|ch| matches!(ch, '\n' | '\r' | '\u{2028}' | '\u{2029}')), "{drawn:?} from {text:?}"); + } + } + } + let drawn = |text: &str| format!("{:?}", arabic::layout(Font::Helvetica, text).unwrap().items); + assert_eq!(drawn("مرحبا\u{2029}عالم"), drawn("مرحبا عالم"), "a paragraph separator keeps right-to-left reading order"); + Ok(()) +} + +#[test] +fn arabic_text_respects_the_save_nesting_limit() -> Result<(), Box> { + if arabic::layout(Font::Helvetica, "مرحبا").is_err() && arabic::ARABIC_FONTS.is_empty() { + return Ok(()); + } + let mut d = PdfDoc::new(); + let p = d.add_page(100.0, 100.0); + for _ in 0..28 { + p.save(); + } + p.text(10.0, 50.0, 12.0, Font::Helvetica, [0, 0, 0], "مرحبا"); + assert_eq!(interpret(&p.content).max_depth, 28); + p.restore(); + p.text(10.0, 50.0, 12.0, Font::Helvetica, [0, 0, 0], "مرحبا"); + assert!(p.content.contains("/ActualText")); + check_xref(&d.finish()?)?; + Ok(()) +} + +#[test] +fn arabic_subsets_split_after_255_character_codes() -> Result<(), Box> { + if arabic::layout(Font::Helvetica, "مرحبا").is_err() && arabic::ARABIC_FONTS.is_empty() { + return Ok(()); + } + let mut doc = PdfDoc::new(); + let page = doc.add_page(600.0, 300.0); + let marks = ['َ', 'ُ', 'ِ', 'ّ', 'ْ', 'ً', 'ٌ', 'ٍ']; + for i in 0..260 { + let text = format!("ب{}{}{}", marks[i % 8], marks[(i / 8) % 8], marks[(i / 64) % 8]); + page.text(20.0, 50.0, 10.0, Font::Helvetica, [0, 0, 0], &text); + } + let pdf = doc.finish()?; + check_xref(&pdf)?; + assert!(find(&pdf, b"/A2 ").is_some()); + assert!(find(&pdf, b"/LastChar 255").is_some()); + assert!(find(&pdf, b"<100> Tj").is_none(), "Type3 codes are always one byte"); + Ok(()) +} + #[test] fn many_alternating_runs_share_one_text_matrix() { let mut doc = PdfDoc::new(); let p = doc.add_page(612.0, 792.0); p.text(10.0, 20.0, 10.0, Font::Helvetica, [0, 0, 0], &"a\u{4e2d}".repeat(2000)); - let pdf = doc.finish(); + let pdf = doc.finish().unwrap(); let count = |needle: &[u8]| pdf.windows(needle.len()).filter(|w| *w == needle).count(); assert_eq!(count(b" Tm\n"), 1, "the position is set once, before the first run"); assert_eq!(count(b" Tf\n"), 4000, "every run selects its font"); @@ -906,19 +1105,34 @@ fn many_alternating_runs_share_one_text_matrix() { } #[test] -fn symbol_font_draws_maths_in_its_own_encoding() { +fn symbol_font_draws_maths_in_its_own_encoding() -> Result<(), Box> { let mut doc = PdfDoc::new(); let p = doc.add_page(200.0, 200.0); p.text(10.0, 50.0, 12.0, Font::Symbol, [0, 0, 0], "\u{2211}\u{03B1}\u{2264}\u{221E}x"); p.text(10.0, 80.0, 12.0, Font::TimesItalic, [0, 0, 0], "x"); - let pdf = doc.finish(); + let pdf = doc.finish()?; let s = String::from_utf8_lossy(&pdf); assert!(s.contains("/BaseFont /Symbol >>"), "Symbol has no WinAnsi encoding"); assert!(s.contains("/BaseFont /Times-Italic /Encoding /WinAnsiEncoding")); // The sum, alpha, less-or-equal and infinity codes; x is not in Symbol and becomes `?`. assert!(s.contains("(\\345a\\243\\245?) Tj"), "{s}"); - check_xref(&pdf).unwrap(); + check_xref(&pdf)?; assert!(Font::Symbol.can_encode('\u{2211}') && !Font::Symbol.can_encode('x')); assert!(Font::TimesRoman.can_encode('x') && !Font::TimesRoman.can_encode('\u{2211}')); assert_eq!(text_width(Font::Symbol, 10.0, "\u{2211}"), 7.13); + Ok(()) +} + +#[test] +fn symbol_text_is_never_shaped_as_arabic() -> Result<(), Box> { + let mut doc = PdfDoc::new(); + let page = doc.add_page(200.0, 200.0); + assert_eq!(page.text_vertical_extents(Font::Symbol, 12.0, "\u{03B1}ب"), None); + page.text(10.0, 50.0, 12.0, Font::Symbol, [0, 0, 0], "\u{03B1}ب"); + let pdf = doc.finish()?; + check_xref(&pdf)?; + assert!(find(&pdf, b"(a?) Tj").is_some()); + assert!(find(&pdf, b"/ActualText").is_none()); + assert_eq!(text_width(Font::Symbol, 10.0, "\u{03B1}ب"), f32::from(Font::Symbol.char_width('\u{03B1}') + Font::Symbol.char_width('ب')) / 100.0); + Ok(()) } diff --git a/docs/file-format-parity.md b/docs/file-format-parity.md index 6e5b1a23f..5ec72fd63 100644 --- a/docs/file-format-parity.md +++ b/docs/file-format-parity.md @@ -1,6 +1,6 @@ # File-format parity -> **Last reviewed:** 2026-11-01 · **Last updated:** 2026-11-12 · **Change:** minor (.xls data validation is read; earlier: .xls is read by the shared craft-xls crate: the workbook opens on the sheet it was saved on, and rich-text (RString) cells are read; earlier: form controls: buttons, scroll bars and spin buttons read and written; earlier: a web page's first table opens as a sheet, also when the page is named `.xls`; earlier: `workbookPr filterPrivacy` kept, and honoured on save; earlier: a workbook's R1C1 reference style, `calcPr refMode`, is kept through open and save; earlier: XLSX page margins are written as their decimals, `0.3` rather than `0.30000001192092896`; earlier: Python cells in `.xlsx` as [MS-XLSX] specifies, `_xlfn._xlws.PY(…)` naming code in the Python part, with cached values, and in GridCraft JSON, with limits on open; earlier: equations in xlsx as Office Math with a marker in `descr` so an edit made in Excel reads back, read back only up to the equation source cap, and a brace with a label and a script on its other side keeps both; Symbol font in PDF; earlier: XLSX reads and writes Page Setup's page order, `pageSetup pageOrder`, and PDF export follows it; earlier: chart data tables: native c:dTable import/export; earlier: XLSX workbook-view selection, normalized tab ratios and sheet-edit regressions; fidelity estimates unchanged; earlier: formulas from Google Sheets' XLSX export open as the original formula instead of `IFERROR(__xludf.DUMMYFUNCTION(…),…)` when GridCraft calculates every function in it, and save as it; `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; earlier: `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; CSV and text open detects each column's date order from values that read only one way, warns when every date is ambiguous, and keeps a column with both kinds as text (#2765)) · **Target:** Microsoft Excel (Microsoft 365), Excel for Mac 16.113.4 +> **Last reviewed:** 2026-11-01 · **Last updated:** 2026-11-13 · **Change:** minor (Arabic PDF export no longer fails when a character or the Arabic font is missing; earlier: Arabic PDF text is shaped with embedded Noto Arabic glyphs beside the non-embedded standard, Symbol and CID fonts; earlier: .xls data validation is read; earlier: .xls is read by the shared craft-xls crate: the workbook opens on the sheet it was saved on, and rich-text (RString) cells are read; earlier: form controls: buttons, scroll bars and spin buttons read and written; earlier: a web page's first table opens as a sheet, also when the page is named `.xls`; earlier: `workbookPr filterPrivacy` kept, and honoured on save; earlier: a workbook's R1C1 reference style, `calcPr refMode`, is kept through open and save; earlier: XLSX page margins are written as their decimals, `0.3` rather than `0.30000001192092896`; earlier: Python cells in `.xlsx` as [MS-XLSX] specifies, `_xlfn._xlws.PY(…)` naming code in the Python part, with cached values, and in GridCraft JSON, with limits on open; earlier: equations in xlsx as Office Math with a marker in `descr` so an edit made in Excel reads back, read back only up to the equation source cap, and a brace with a label and a script on its other side keeps both; Symbol font in PDF; earlier: XLSX reads and writes Page Setup's page order, `pageSetup pageOrder`, and PDF export follows it; earlier: chart data tables: native c:dTable import/export; earlier: XLSX workbook-view selection, normalized tab ratios and sheet-edit regressions; fidelity estimates unchanged; earlier: formulas from Google Sheets' XLSX export open as the original formula instead of `IFERROR(__xludf.DUMMYFUNCTION(…),…)` when GridCraft calculates every function in it, and save as it; `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; earlier: `file.exportCsv {values: true}` writes raw values for automation; .xls dialog sheets are skipped instead of read as worksheets; `.xls` import reads each sheet's display options from WINDOW2: right-to-left, gridlines, headings, zeros, formulas; XLSX text boxes and shapes keep their paragraphs' right-to-left direction on save; CSV/TSV/TXT can open with every field kept as text via `file.open {csvAsText}`; formulas over Excel's 8,192 characters, and `%` or call chains past the nesting limit, are parse errors instead of stack overflows when a file is opened; XLSX PivotTables read and write `rowGrandTotals`/`colGrandTotals` the way Excel does, so a table saved with only one grand total opens with the right one; HTML export keeps right-to-left; CSV and text import detects likely Windows-1256 Arabic when invalid UTF-8 has Arabic byte runs; other invalid UTF-8 remains Windows-1252; HTML export places pictures and charts, as inline SVG, at their anchors; CSV and text open detects each column's date order from values that read only one way, warns when every date is ambiguous, and keeps a column with both kinds as text (#2765)) · **Target:** Microsoft Excel (Microsoft 365), Excel for Mac 16.113.4 Every format Excel opens or saves, with GridCraft's read and write support, fidelity and tests. Part of [target-app-parity.md](target-app-parity.md). **File formats: ~50% ready, 80–120 h** @@ -14,6 +14,8 @@ support comes from `crates/engine/src/io.rs` (`FileKind`), `crates/xlsx/src` and Legacy scalar `INDEX(range, MATCH(...))` formulas remain ordinary formulas on save, including when shared formulas are expanded during import (`array_export.rs`, #1505). +Arabic PDF output uses harfrust shaping, Unicode bidi and licensed Noto Arabic glyph outlines with ToUnicode and logical ActualText. Cells, headers and chart labels support marks, digits, rotation and style. Bundled builds use the optional `CRAFT_FONTS_DIR` input; unbundled native builds look for Noto Sans Arabic. Characters the Arabic face lacks fall back to the standard and CID fonts, and without an Arabic face for the text's weight Arabic is set in the CID font, which has no Arabic glyphs, with a message. Explicit cell reading-direction integration is separate. + Cell text now uses shared Unicode bidi layout while retaining logical source indexes for editing, copy and XLSX. Format Cells exposes Context, Left to right and Right to left reading direction, independent of sheet geometry. OOXML Context uses the first non-whitespace character. Optional direction preserves absent versus explicit Context in differential styles. The renderer is pinned to a reviewed egui 0.36.2 backport with real-font, editing, accessibility and WASM checks. Physical screen-reader behavior remains unverified. ## Workbook window settings @@ -122,7 +124,7 @@ past the parser's limit, and a wrapper inside the text. **Saving writes the real | Format | Excel | GridCraft | Notes | |---|---|---|---| -| PDF | Save As / Export | yes (own PDF 1.7 writer; CLI `convert` output) | Printed cell hyperlinks export as clickable HTTP(S)/mailto links or destinations within printed pages (#954); unsafe schemes and destinations outside the print range are omitted. No font is embedded: text uses the standard Helvetica, Times and Courier faces (13 base-14 entries with Symbol), and characters outside WinAnsiEncoding are set in a predefined, non-embedded CID font (STSong-Light, Adobe-GB1, `UniGB-UTF16-H`), so **Simplified Chinese displays where the reader supplies that font, and scripts outside Adobe-GB1 (Arabic, Devanagari, Thai) export as code points but show as the reader's missing-glyph box**; equations use Symbol for the Greek letters and operators it has, the Times faces for letters and digits, and a `[?]` box for the rest; web build exports without a file prompt (#141) | +| PDF | Save As / Export | yes (own PDF 1.7 writer; CLI `convert` output) | Printed cell hyperlinks export as clickable HTTP(S)/mailto links or destinations within printed pages (#954); unsafe schemes and destinations outside the print range are omitted. Latin text uses the standard Helvetica, Times and Courier faces (13 base-14 entries with Symbol), not embedded; Arabic exports as shaped, embedded Noto Arabic glyphs; other characters outside WinAnsiEncoding are set in a predefined, non-embedded CID font (STSong-Light, Adobe-GB1, `UniGB-UTF16-H`, #1912), so **Simplified Chinese displays where the reader supplies that font, and scripts outside Adobe-GB1 (Devanagari, Thai) export as code points but show as the reader's missing-glyph box**; equations use Symbol for the Greek letters and operators it has, the Times faces for letters and digits, and a `[?]` box for the rest; web build exports without a file prompt (#141) | | XPS | Windows | no | Low priority | | Print | yes | yes (via PDF, system dialog on desktop; PR #143 for web) | Pages print in the sheet's page order: down, then over (the default) or over, then down | | Images of ranges/charts (Copy as Picture) | yes | chart raster in `crates/chart` | No range-to-image command | @@ -164,6 +166,8 @@ refresh. Counted under features (Get & Transform, ~10% ready), not here. | 2026-11-12 | minor | `.xls` files open with their data validation (drop-down lists, input messages, error alerts and the rules), read by craft-xls and mapped as the `.xlsx` reader maps the same rules. A rule Excel would refuse, or one past 65,536 in a workbook, is skipped with a warning (#3634, from #3636) | | 2026-11-12 | minor | .xls import reads the file with craft-xls (storytold/craft-libs), the reader that moved there from this repository; GridCraft keeps the mapping onto its model. The workbook now opens on the sheet it was saved on (WINDOW1 `itabCur`), falling back to the first visible sheet as before (from #3551), and cells saved as RString records (text with formatting runs) are read as text instead of skipped | | 2026-11-10 | minor | `.xls` import: every formula except a Ctrl+Shift+Enter array is read as a legacy one with Excel's implicit `@` (a title row `=Other!A1:G1` shows its first cell instead of `#SPILL!`), one-cell Ctrl+Shift+Enter arrays stay arrays, and `@` of a 2-D range meets the formula's row and column (#410, PR #1367) | +| 2026-11-09 | minor | Arabic PDF export no longer fails on a character the Arabic face lacks (it falls back like other text) or when no Arabic font is available (Arabic is set in the CID font, which has no Arabic glyphs, and a message says so) | +| 2026-11-03 | trivial | The PDF row lists Arabic's shaped, embedded Noto glyphs beside the non-embedded standard faces, Symbol and the Adobe-GB1 CID font; Symbol text keeps its own encoding and is never shaped as Arabic | | 2026-11-02 | minor | XLSX: buttons, scroll bars and spin buttons read and written (control properties parts, ``, VML `x:ClientData`); other form controls stay with pass-through; a button's command, fixed input or output cell travels in a hidden defined name and comes back only while the file still assigns the button's macro (or none, for a command) | | 2026-11-03 | minor | SYLK (`.slk`) import: values, R1C1 formulas as A1 (shared formulas too) with cached results, number formats and column widths; import only (#1481) | | 2026-11-03 | minor | Opening a web page (HTML) reads its first table as a sheet, whatever the file is called: a web app's "Export to Excel" saved as `.xls` no longer opens as lines of markup. Values read as in a CSV (or as text with `csvAsText`), `` cells are bold and centred, `` names the sheet, and a page not named `.htm`/`.html` opens with a warning (#3166) | @@ -185,6 +189,7 @@ refresh. Counted under features (Get & Transform, ~10% ready), not here. | 2026-11-01 | trivial | XLSX selects the first visible workbook view as one unit and bounds imported/exported tab ratios; isolated attributes and sheet-edit round trips covered, with firstSheet left to #2264 (#3057) | | 2026-10-31 | minor | CSV and text open detects each column's date order: a value that reads only day first (`31/05/2026`) makes the column day first, a value that reads only month first (`12/31/2026`) makes it month first, an all-ambiguous column keeps month first with a warning, and a column with both kinds stays text with a warning (#2765) | | 2026-10-31 | minor | XLSX open reads Google Sheets' `IFERROR(__xludf.DUMMYFUNCTION("…"),cached)` export wrapper as the formula inside when GridCraft calculates every function in it (plain, shared and array formulas, whole or nested; Google-only functions and unparseable, over-long or over-deep inner text keep the wrapper and the cached value) and keeps the cached value; saving writes the real formula with Excel's `_xlfn.` prefixes (#2871) | +| 2026-10-26 | minor | Arabic PDF export embeds licensed glyph outlines, joins letters and marks, preserves logical text and reports unsupported glyphs before writing; Arabic runs take the embedded faces ahead of the CJK CID font, and CJK inside Arabic text is set in the CID font instead of failing the export | | 2026-10-26 | minor | Alignment readingOrder 0/1/2 survives cell, named-style and differential-style round-trips; Format Cells exposes reading direction and CF priority preserves unspecified inheritance | | 2026-10-21 | minor | HTML (Save as Web Page) writes pictures (`data:` URI ``) and charts (inline SVG from the chart crate's drawing primitives, text included) at their anchors; the table uses the sheet's column widths and row heights when the sheet has either, and output for sheets without pictures or charts is unchanged | | 2026-10-20 | trivial | XLSX shared strings: Japanese phonetic runs (ふりがな, `` with ``) are read into the workbook and written back next to the string with the same text, so an open + save no longer deletes them; bad offsets and readings are dropped with a warning, a `` without runs is kept on its own, and when duplicate `` items share a text but not a reading only the first reading is written (reported on open) (`phonetic_tests.rs`) | diff --git a/docs/localization-parity.md b/docs/localization-parity.md index 1de2927ac..e1cff475c 100644 --- a/docs/localization-parity.md +++ b/docs/localization-parity.md @@ -1,12 +1,14 @@ # Localization parity -> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-11-10 · **Change:** minor (Ukrainian Chart is Діаграма); previously minor (Diagram and Decorative Text labels, Diagram distinct from Chart); previously minor (CJK fallback baseline alignment); previously minor (Arabic bidi display, editing and cell reading direction; missing translations are listed by `cargo xtask i18n-report` and never fail a check); previously minor (Arabic font fallback; missing translations are listed by `cargo xtask i18n-report` and never fail a check); previously minor (number formats honour the `B1`/`B2` calendar prefixes, checked against Excel; earlier: missing translations are listed by `cargo xtask i18n-report` and never fail a check; Arabic text sorts by its letters, not its vowel marks); previously minor (AutoFit estimates Arabic text width from measured glyph widths); previously minor (missing translations are listed by `cargo xtask i18n-report` and never fail a check; Arabic text sorts by its letters, not its vowel marks) · **Target:** Microsoft Excel (Microsoft 365), Excel for Mac 16.113.4 +> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-11-13 · **Change:** minor (Arabic PDF export no longer fails when a character or the Arabic font is missing); previously minor (Ukrainian Chart is Діаграма); previously minor (Diagram and Decorative Text labels, Diagram distinct from Chart); previously minor (CJK fallback baseline alignment); previously minor (Arabic bidi display, editing and cell reading direction; missing translations are listed by `cargo xtask i18n-report` and never fail a check); previously minor (Arabic font fallback; missing translations are listed by `cargo xtask i18n-report` and never fail a check); previously minor (number formats honour the `B1`/`B2` calendar prefixes, checked against Excel; earlier: missing translations are listed by `cargo xtask i18n-report` and never fail a check; Arabic text sorts by its letters, not its vowel marks); previously minor (AutoFit estimates Arabic text width from measured glyph widths); previously minor (missing translations are listed by `cargo xtask i18n-report` and never fail a check; Arabic text sorts by its letters, not its vowel marks) · **Target:** Microsoft Excel (Microsoft 365), Excel for Mac 16.113.4 Per-language status against Excel, which ships its UI in dozens of languages (Windows language packs cover 100+), with localized function names, argument separators, number and date formats, right-to-left sheets and full complex-script support. Part of [target-app-parity.md](target-app-parity.md). **Localization: ~10% ready, 100–165 h** (estimated). +Arabic PDF output uses harfrust shaping, Unicode bidi and licensed Noto Arabic glyph outlines with ToUnicode and logical ActualText. Cells, headers and chart labels support marks, digits, rotation and style. Bundled builds use the optional `CRAFT_FONTS_DIR` input; unbundled native builds look for Noto Sans Arabic. Characters the Arabic face lacks fall back to the standard and CID fonts, and without an Arabic face for the text's weight Arabic is set in the CID font, which has no Arabic glyphs, with a message. Explicit cell reading-direction integration is separate. + Cell text now uses shared Unicode bidi layout while retaining logical source indexes for editing, copy and XLSX. Format Cells exposes Context, Left to right and Right to left reading direction, independent of sheet geometry. OOXML Context uses the first non-whitespace character. Optional direction preserves absent versus explicit Context in differential styles. The renderer is pinned to a reviewed egui 0.36.2 backport with real-font, editing, accessibility and WASM checks. Physical screen-reader behavior remains unverified. ## Where we are (measured from source on `main`, after the 2026-10-10 merges) @@ -81,7 +83,8 @@ Cell text now uses shared Unicode bidi layout while retaining logical source ind - **Names in any script.** Defined names (Define Name, Create from Selection) are limited to 255 characters, counted as characters rather than UTF-8 bytes, so a 255-letter Arabic name is accepted as in Excel. Sheet names already counted characters (31). -- **PDF export** uses only the 14 standard PDF fonts: non-Latin text cannot export. +- **PDF export** shapes Arabic with embedded Noto Arabic glyph outlines; CJK uses a predefined, + non-embedded Adobe-GB1 CID font (#1912); Devanagari and Thai cannot export yet. ## Languages @@ -95,7 +98,7 @@ separators, number/date formats, machine translation with agent review, and scri | Simplified Chinese (Mandarin) | zh-Hans | 266 (~12%) | no | CJK system-font fallback (#38); IME unverified; PDF can't export | no | menus only | 8–13 h | | Spanish | es | 0 (0%) | no | Latin: yes | no | none | 6–10 h | | Hindi | hi | 0 (0%) | no | Devanagari: **no shaping** | no | none | 12–18 h | -| Arabic | ar | 0 (0%) | no | shared shaping/bidi; no Arabic UI catalog or mirrored shell; Sheet RTL mirrors the grid (#146) | no | none | 25–35 h | +| Arabic | ar | 0 (0%) | no | shared shaping/bidi; no Arabic UI catalog or mirrored shell; PDF export: shaped and joined; Sheet RTL mirrors the grid (#146) | no | none | 25–35 h | | French | fr | 0 (0%) | no | Latin: yes | no | none | 6–10 h | | Portuguese | pt-BR | 943 (~43%) | dialogs and tooltips (dialogs take effect with #669) | Latin: yes | no | menus, File view, dialogs | 2–4 h | | Indonesian | id | 0 (0%) | no | Latin: yes | no | none | 5–8 h | @@ -135,11 +138,13 @@ language (none so far) and of localized function names. |---|---|---| | 2026-11-10 | minor | Ukrainian Chart is `Діаграма`, matching [craft-i18n#211](https://github.com/storytold/craft-i18n/pull/211) and distinct from Diagram (`Схема`) | | 2026-11-10 | minor | Diagram and Decorative Text labels in all 18 non-English catalogs; Diagram never repeats the catalog's chart word (German `Schema`, Ukrainian `Схема`, Romanian `Schemă`), see [feature-names.md](feature-names.md) | +| 2026-11-09 | minor | Arabic PDF export no longer fails on a character the Arabic face lacks (it falls back like other text) or when no Arabic font is available (Arabic is set in the CID font, which has no Arabic glyphs, and a message says so) | | 2026-10-31 | minor | System and embedded CJK text fallbacks share the Latin baseline in cells, editors and window titles; other scripts, symbols and emoji keep their alignment | | 2026-10-26 | minor | Shared bidi layout supports Arabic clusters, marks, visual cursor navigation and directional accessibility; cell Context/LTR/RTL is stored separately from sheet RTL and round-trips through XLSX | | 2026-10-26 | minor | Arabic text renders in cells and the interface: a system Arabic face (Tahoma, Arial or Segoe UI on Windows, Geeza Pro on macOS, Noto Sans or Naskh Arabic on Linux) joins the script fallbacks, fontconfig's Arabic choice, when it has Arabic letters, leads the Arabic fallbacks and draws Arabic wherever the UI or cell font has none, such as the default cell fonts (part of #229; a UI font with Arabic of its own, such as DejaVu Sans in the Flatpak, still draws interface Arabic, left for a follow-up), and craft-fonts' Noto Sans Arabic is a required face allowed on the web. Workbook fonts without Arabic keep every script fallback. Shaping and bidirectional layout remain separate work | | 2026-10-27 | minor | Number formats honour the `B1`/`B2` calendar prefixes: never shown; `B2` renders year, month, day and Arabic month and weekday names in the tabular Hijri calendar (1 Jan 2001 = 6 Shawwal 1421; leap years 2, 5, 7, 10, 13, 15, 18, 21, 24, 26 and 29 of the 30-year cycle, which is what Excel shows). Checked against Excel 16.0 (build 20527, en-US) on every serial and about 1,700 format/value pairs: a locale or currency tag before `B2` keeps the date Gregorian, a misplaced `B` and digit or a prefix before a number format is rejected, serial 60 has no Hijri date, serial 0 shows as serial 1, two-word month names use a no-break space. Not matched: Excel's weekday/month-name mix-ups in one `B2` section and its weekday-only quirks | | 2026-10-27 | minor | AutoFit column width, wrapped-row height and PivotTable column fit estimate Arabic letters at ~0.48 em (measured 0.36–0.53 em shaped, Noto Sans/Naskh Arabic and DejaVu Sans) instead of 1 em, Arabic-Indic digits as wide as ASCII digits, and give vowel marks and direction marks no width; vowelled Arabic no longer fits ~3.5× too wide | +| 2026-10-26 | minor | Arabic PDF cells, headers and chart labels support joined letters, combining marks, digits, bold/italic, rotation and grapheme wrapping | | 2026-10-23 | minor | Missing translations are reported, never a failure: `cargo xtask i18n-report` prints, per language, the strings other catalogs translate and it lacks (plus Spanish function help rows keyed by English the function catalog no longer has) and always passes; `cargo xtask ci` runs it first. Coverage and per-language parity tests and tests of one language's exact wording are gone; UI tests take the expected text from the catalog. Duplicate-key, placeholder, context-prefix and line-shape checks stay | | 2026-10-21 | minor | Romanian interface (`ro`, Română): 970 labels covering every key of the other catalogs (`crates/ui-egui/src/i18n/ro.rs`), `ro-*` system locales, the View-tab switcher and `app.language.set`; context keys show no prefix (File › Options test). English formulas and formats; AI-assisted draft pending native-speaker review | | 2026-10-20 | minor | Japanese: furigana (`rPh` phonetic runs and `` in `xl/sharedStrings.xml`) is no longer deleted by an open + save; it is kept on the workbook and written back (a `` without runs is kept too, while duplicate texts with different readings keep only the first, reported on open). Display of the readings is still missing (`crates/xlsx/src/tests/phonetic_tests.rs`) |