diff --git a/Cargo.lock b/Cargo.lock index fd3935a659..a08f650e80 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4703,6 +4703,7 @@ dependencies = [ "serde", "skrifa 0.44.0", "thiserror 2.0.21", + "unicode-bidi", ] [[package]] @@ -4836,10 +4837,12 @@ dependencies = [ name = "pdfcraft-xfdf" version = "0.18.1" dependencies = [ + "base64 0.23.1", "pdfcraft-annot", "pdfcraft-cos", "pdfcraft-forms", "pdfcraft-model", + "proptest", "roxmltree 0.21.2", "thiserror 2.0.21", ] diff --git a/ROADMAP.md b/ROADMAP.md index d200583f59..c2c7c6ad62 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -240,6 +240,12 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md ## Progress log - **2026-10-13 (M8, redact by regex):** `redact_mark` takes `regex` (caller expression, linear-time engine, length/size/nesting caps, empty-matching refused) with optional `validate` (luhn, mod97, iban). Headless only; issue #1886. +- **2026-11-08 (M5, comment exchange):** Comments keep their formatting and appearance through XFDF and FDF. XFDF export and import carry rich text, default style, border width, style and dash pattern, rotation, justification, fringe, line leaders and captions, the caret symbol and redaction overlay text as XFDF 3.0 (ISO 19444-1) names them, and a stamp's appearance as `` (the DICT/STREAM/DATA encoding, bounded); copy and paste carries every comment's appearance. FDF export keeps a stamp's /AP with its forms and images, as XFDF does, and FDF import keeps any /AP the file carries (ISO 32000-2 §12.7.8.3.4). The intent goes out as both `intent` and `IT`. An unreadable XFDF appearance is left out, said in the import report and the repair log, and the comment is drawn by PdfCraft; an FDF past the import budget is refused whole. A redaction mark is always drawn by PdfCraft. + +- **2026-11-07 (M2, #6449):** A non-embedded Times New Roman set as a TrueType CID font (`TimesNewRoman,Bold`, `TimesNewRomanPSMT` and the other faces) draws the glyphs its codes name, as the non-embedded Arial does: its first 258 glyphs follow the same standard Macintosh order, which the embedded Times New Roman subsets of a public French form confirm in 941 of 942 `/ToUnicode` entries (vendored hayro-interpret patch 88). That form drew "Salarié(e) du régime général" as "Salari²*g+f w't²i lo g'ì²p²tcn" and "changement" as "ej angement". Times and Times New Roman MT Std are not read this way. + +- **2026-11-05 (M2, #5889):** A non-embedded Arial set as a TrueType CID font (`CIDFontType2`, `Identity-H`, as Word and PDFMaker write it) draws the glyphs its codes name. Those codes are glyph indexes in the author's Arial, whose first 258 glyphs follow the standard Macintosh order, so the standard substitute now draws each glyph by name (vendored hayro-interpret patch 76). Before, it took each code as one of its own glyph indexes, and "Low Power" drew as "@ck 'Dck Yf". Arial Narrow, Black and Unicode MS, and `CIDFontType0` fonts, are not read this way. Copying or searching those runs still needs a `/ToUnicode`. + - **2026-11-07 (M7, #2964 / #3185):** Supported Form text can be prepared for direct editing; groups left grouped (unsafe compositing, text nested past eight levels) are reported. Legacy Thai keeps the original font programs, Regular/Bold and text settings, also across same-face subsets; the narrow THSarabunPSK map repair is checked glyph by glyph against the font program; cross-family replacement is explicit. Builds on #3067 and #2811. Unsafe groups and mixed contextual styles remain partial; estimates unchanged. - **2026-10-28 (M10, #4119 partial):** Scan & OCR ▸ Enhance scanned file cleans scanned pages in place: background removal (level 0–100, as the camera clean-up), edge shadow removal (book gutters, copier borders) and despeckle. Each scan image is decoded, cleaned and written back into its own object at the same size and colour space (JPEG stays JPEG, others lossless, 1-bit stays 1-bit), so the OCR text layer, comments and links stay; one undo step, shared images once, unsupported images (CMYK, indexed, JPEG 2000, JBIG2, CCITT, broken data) reported. `page_enhance_scan` drives it headlessly. Not yet an option of Recognize text. Estimates unchanged. - **2026-11-07 (M10, `ocr.ocr-web`, port onto #977):** Web OCR rebuilt on main's engines and preprocessing (#977). `OcrJob::run` is now `OcrJob::steps` a page at a time: the page renders, crops to the region and cleans each page, a Web Worker runs ocrs on the result, and `OcrShot::place` maps the words back through the region and preprocessing. Every strategy reads like this when only one engine is present, and the web build has only ocrs. The downloaded models are installed into the engine cache (`ocr::install`), so the Verify screen and the engine-status checks see them too. The batch keeps main's skipped-files bucket, low-confidence notes and cancel. The translations moved to craft-i18n (storytold/craft-i18n#141). The Verify screen starts the model download itself and runs once the models land; before, on a fresh page it failed with the desktop's "set PDFCRAFT_MODELS" message. Known limits: the cleaning steps, and the Verify screen's Run OCR and re-reads, run on the page. Estimates unchanged. @@ -258,6 +264,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md - **2026-11-02 (M10, #5743):** macOS print jobs name their title and owner (`com.apple.print.JobInfo.PMJobName`, `PMJobOwner`), as the system print dialog does, because a Kyocera filter crashes on a job without them. Checked by a test and through `doc_print` against a stand-in `lp`; not yet on a physical printer. - **2026-11-02 (M3, #5708):** Pages shorter than the window sit in its middle, one page or row at a time and when scrolling continuously with every page in view. +- **2026-11-02 (M3, #5716):** A snapshot is drawn at 300 ppi, or at the zoom if that is finer, so one taken zoomed out stays readable; an area too large for one 4096-pixel tile comes out as fine as a tile allows (about 147 ppi for 2000 pt). A choice of resolution and a selection that stays active remain. - **2026-11-02 (M14, #5737, #5460):** Preferences can turn off the update check at start. It stays on by default, and Help ▸ Check for updates still asks. - **2026-11-01 (M7, #5657):** Header/footer, watermark and background additions and replacements stage the complete call before committing. Synthetic later-page failures preserve exact incremental bytes, object count, dirty state and allocation cursor; success controls retain other marks and shared/inherited resources. Estimates unchanged. diff --git a/apps/pdfcraft-cli/src/commands.rs b/apps/pdfcraft-cli/src/commands.rs index c64f9e1c0f..86b31bafd3 100644 --- a/apps/pdfcraft-cli/src/commands.rs +++ b/apps/pdfcraft-cli/src/commands.rs @@ -421,6 +421,12 @@ pub(crate) fn render(args: &[String]) -> Result<(), CliError> { let _ = writeln!(std::io::stderr().lock(), "warning: page {page}: a colour that could not be computed was painted in a fallback colour"); } + pdfcraft_render::RenderWarning::Type3WidthsMissing => { + let _ = writeln!( + std::io::stderr().lock(), + "warning: page {page}: a Type 3 font has no /Widths; its glyphs were spaced by the widths their glyph procedures give" + ); + } } } Ok(()) diff --git a/crates/annot/src/appearance.rs b/crates/annot/src/appearance.rs index 7996567f04..011656c7e7 100644 --- a/crates/annot/src/appearance.rs +++ b/crates/annot/src/appearance.rs @@ -4,7 +4,9 @@ //! objects it points at, so the same code serves new comments and restyled ones. Drawings are in page space with //! `/BBox = /Rect` (identity matrix), except note icons, which are drawn in a 20 × 20 box. //! It returns `None` for anything it cannot draw faithfully (cloudy borders, unknown line -//! endings), so callers never replace an appearance with a worse one. +//! endings), so callers never replace an appearance with a worse one. It resolves `/QuadPoints` +//! itself; for the other values, callers that hold the document follow references first +//! (`draw::with_direct_values`). //! //! The note icons are PdfCraft's own drawings (AGENTS.md §1). Text boxes use the standard //! Helvetica font with WinAnsi encoding, and draw Thai from the craft-fonts Thai face as Type 3 @@ -147,6 +149,29 @@ pub fn smooth_segments(pts: &[(f64, f64)]) -> Vec<[(f64, f64); 3]> { .collect() } +/// The points where the cubic Bézier segment from `p0` (control points `c1`, `c2`) to `p3` turns +/// back in x or in y, strictly between its ends: with its ends, they bound the segment. +fn cubic_extremes(p0: (f64, f64), c1: (f64, f64), c2: (f64, f64), p3: (f64, f64)) -> Vec<(f64, f64)> { + let at = |t: f64| { + let u = 1.0 - t; + let w = [u * u * u, 3.0 * u * u * t, 3.0 * u * t * t, t * t * t]; + (w[0] * p0.0 + w[1] * c1.0 + w[2] * c2.0 + w[3] * p3.0, w[0] * p0.1 + w[1] * c1.1 + w[2] * c2.1 + w[3] * p3.1) + }; + let mut out = Vec::new(); + for (q0, q1, q2, q3) in [(p0.0, c1.0, c2.0, p3.0), (p0.1, c1.1, c2.1, p3.1)] { + // The derivative over 3: a t² + 2 b t + c. + let (a, b, c) = (-q0 + 3.0 * q1 - 3.0 * q2 + q3, q0 - 2.0 * q1 + q2, q1 - q0); + let roots = if a.abs() < 1e-12 { + if b.abs() < 1e-12 { vec![] } else { vec![-c / (2.0 * b)] } + } else { + let disc = b * b - a * c; + if disc < 0.0 { vec![] } else { vec![(-b + disc.sqrt()) / a, (-b - disc.sqrt()) / a] } + }; + out.extend(roots.into_iter().filter(|t| *t > 0.0 && *t < 1.0).map(at)); + } + out +} + /// `c` operators for [`smooth_segments`] (the path already starts at the first point). fn smooth_curve(pts: &[(f64, f64)]) -> String { smooth_segments(pts).iter().map(|[a, b, e]| format!("{} {} {} {} {} {} c\n", n(a.0), n(a.1), n(b.0), n(b.1), n(e.0), n(e.1))).collect() @@ -545,7 +570,15 @@ impl AppearanceDrawing { /// Text the standard fonts can't draw comes out as `?`: [`undrawable_text`] says which, and /// [`crate::set_appearance`] marks the annotation with it (`pdfcraft_fonts::UNDRAWN_MARK`). pub fn build(doc: &Document, d: &Dict) -> Option { - let context = appearance_context(doc, d)?; + build_on_page(doc, d, None) +} + +/// [`build`] for an annotation on a page whose visible region is `page` (`Page::crop` in +/// pdfcraft-model): a text markup, line, polyline, polygon or ink annotation whose points reach +/// past its `/Rect` is drawn through them as far as the page shows ([`point_reach`]), and its +/// appearance's `/BBox` says how far ([`widened_rect`]). Without a page it is drawn within `/Rect`. +pub(crate) fn build_on_page(doc: &Document, d: &Dict, page: Option<[f64; 4]>) -> Option { + let context = appearance_context(doc, d, page)?; let drawing = AppearanceDrawing::new(&context); match context.subtype { b"Text" => return draw_icon(&context, "Note", [1.0, 0.82, 0.0], drawing), @@ -559,15 +592,15 @@ pub fn build(doc: &Document, d: &Dict) -> Option { finish_appearance(&context, drawing) } -fn appearance_context<'a>(doc: &'a Document, d: &'a Dict) -> Option> { +fn appearance_context<'a>(doc: &'a Document, d: &'a Dict, page: Option<[f64; 4]>) -> Option> { // Imported measurement appearances may include leaders, captions and formatting that // this builder cannot reproduce. Preserve them rather than silently losing detail. if d.contains(b"Measure") && !d.contains(b"PCMeasureValue") { return None; } let subtype = d.name(b"Subtype")?; - let rect = nums(d, b"Rect").filter(|r| r.len() == 4)?; - let rect = [rect[0].min(rect[2]), rect[1].min(rect[3]), rect[0].max(rect[2]), rect[1].max(rect[3])]; + let rect = normalised_rect(d)?; + let rect = page.and_then(|page| point_reach(doc, d, rect, page)).unwrap_or(rect); // Cloudy borders (§12.5.4) are drawn for polygons only. let cloudy = d.get(b"BE").and_then(|b| b.as_dict()).is_some_and(|b| b.name(b"S") == Some(b"C")); if cloudy && subtype != b"Polygon" { @@ -586,6 +619,170 @@ fn appearance_context<'a>(doc: &'a Document, d: &'a Dict) -> Option Option<[f64; 4]> { + let [x0, y0, x1, y1] = <[f64; 4]>::try_from(nums(d, b"Rect")?).ok()?; + Some([x0.min(x1), y0.min(y1), x0.max(x1), y0.max(y1)]) +} + +/// The subtypes drawn through points of their own rather than within `/Rect`: text markup over +/// its `/QuadPoints` (ISO 32000-2 §12.5.6.10), lines along `/L` (§12.5.6.7), polygons and +/// polylines through `/Vertices` (§12.5.6.9) and ink through `/InkList` (§12.5.6.13). +pub(crate) fn is_drawn_from_points(d: &Dict) -> bool { + matches!(d.name(b"Subtype"), Some(b"Highlight" | b"Underline" | b"StrikeOut" | b"Squiggly" | b"Line" | b"PolyLine" | b"Polygon" | b"Ink")) +} + +/// At most this many quadrilaterals widen a text markup's box ([`point_reach`]). Each takes in a +/// word or a group of contiguous words (ISO 32000-2 Table 182), usually a line, so even written +/// word by word a markup over a whole dense page stays well below it; a larger one is drawn +/// within its `/Rect`, as before the box was widened. +pub(crate) const MAX_MARKUP_QUADS: usize = 10_000; + +/// At most this many points widen a line's, polyline's, polygon's or ink annotation's box +/// ([`point_reach`]): as many as [`MAX_MARKUP_QUADS`] quadrilaterals have. A larger one is drawn +/// within its `/Rect`. +pub(crate) const MAX_REACH_POINTS: usize = 4 * MAX_MARKUP_QUADS; + +/// The points `d` is drawn through, as `x y` pairs, and how far its strokes reach past them: +/// half the border width, and for a line ending (`/LE`) or a cloudy border (§12.5.4) the size +/// they are drawn at. `None` for other subtypes, more points than the caps allow, and points that +/// aren't all finite numbers. +fn drawn_points(doc: &Document, d: &Dict) -> Option<(Vec, f64)> { + let w = border_width(d); + let has_ending = || d.get(b"LE").and_then(Object::as_array).is_some_and(|e| e.iter().any(|e| e.as_name().is_some_and(|e| e != b"None"))); + let (points, margin) = match d.name(b"Subtype")? { + subtype @ (b"Highlight" | b"Underline" | b"StrikeOut" | b"Squiggly") => { + // The quads `draw_markup` draws (`quads`: resolved, malformed ones skipped), counted + // before the numbers are read. + let array = doc.resolve(d.get(b"QuadPoints")?); + let entries = array.as_array()?; + if entries.len() / 8 > MAX_MARKUP_QUADS { + return None; + } + let quads = pdfcraft_geom::quad_points::usable_quads(entries.iter().map(|x| doc.resolve(x).as_f64())); + if quads.is_empty() { + return None; + } + // How far each quad's stroke reaches past it: half the line's width, and a squiggle's + // peaks above a quad lower than they are (`draw_markup`). + let margin = quads.iter().map(|q| markup_overreach(subtype, (q[0] - q[4]).hypot(q[1] - q[5]))).fold(0.0, f64::max); + (quads.into_iter().flatten().collect(), margin) + } + b"Ink" => { + let strokes = d.get(b"InkList")?.as_array()?; + let count = strokes.iter().try_fold(0usize, |sum, s| sum.checked_add(s.as_array()?.len()))?; + if count / 2 > MAX_REACH_POINTS { + return None; + } + let mut points = Vec::with_capacity(count.saturating_mul(3)); + for s in strokes { + let stroke: Vec = s.as_array()?.iter().map(Object::as_f64).collect::>()?; + // A stroke's odd last number is no point (`draw_ink` skips it too). + let pts: Vec<(f64, f64)> = stroke.as_chunks::<2>().0.iter().map(|p| (p[0], p[1])).collect(); + points.extend(pts.iter().flat_map(|p| [p.0, p.1])); + // The smooth curve through three or more points (`smooth_segments`) can bulge + // past them: its turning points count too. + if pts.len() > 2 { + let mut start = pts.first().copied().unwrap_or_default(); + for [c1, c2, end] in smooth_segments(&pts) { + points.extend(cubic_extremes(start, c1, c2, end).iter().flat_map(|p| [p.0, p.1])); + start = end; + } + } + } + (points, w / 2.0) + } + subtype @ (b"PolyLine" | b"Polygon") => { + let count = d.get(b"Vertices")?.as_array()?.len(); + if count < 4 || count % 2 != 0 || count / 2 > MAX_REACH_POINTS { + return None; + } + let cloudy = d.get(b"BE").and_then(Object::as_dict).is_some_and(|b| b.name(b"S") == Some(b"C")); + // A cloud's bumps are drawn at up to twice their nominal radius (`draw_polygon`). + let reach = match subtype { + b"Polygon" if cloudy => 2.0 * cloud_radius(w), + b"PolyLine" if has_ending() => arrow_size(w), + _ => 0.0, + }; + (nums(d, b"Vertices")?, w / 2.0 + reach) + } + b"Line" => { + let [x0, y0, x1, y1] = <[f64; 4]>::try_from(nums(d, b"L")?).ok()?; + if ![x0, y0, x1, y1].iter().all(|v| v.is_finite()) { + return None; + } + // Leader lines move the line off `/L` (§12.5.6.7, Table 178). + let layout = line_layout(d, [(x0, y0), (x1, y1)], w); + let mut points = vec![x0, y0, x1, y1]; + for (x, y) in layout.line.iter().chain(layout.leaders.iter().flatten()).chain(layout.tick.iter().flatten()) { + points.extend([*x, *y]); + } + // The caption's box: its width along the text matrix, from the descender to the + // cap height and a little over (Helvetica, as `draw_line` sets it). + if let Some((text, m)) = &layout.caption { + let width = text_width(text, CAPTION_SIZE); + for (x, y) in [(0.0, -0.25), (width, -0.25), (0.0, 1.0), (width, 1.0)].map(|(x, y)| (x, y * CAPTION_SIZE)) { + points.extend([m[0] * x + m[2] * y + m[4], m[1] * x + m[3] * y + m[5]]); + } + } + (points, w / 2.0 + if has_ending() { arrow_size(w) } else { 0.0 }) + } + _ => return None, + }; + (points.iter().all(|v| v.is_finite()) && margin.is_finite()).then_some((points, margin)) +} + +/// An annotation's `/Rect` (normalised, `rect`) grown to take in the points it is drawn through +/// ([`drawn_points`]) where they reach past it, as far as the page's visible region `page` goes; +/// `None` when they don't, aren't usable or number more than the caps allow, and for subtypes +/// drawn within `/Rect` ([`is_drawn_from_points`]). The points are what the annotation draws (ISO +/// 32000-2 Table 182 `/QuadPoints`, Table 178 `/L`, Table 181 `/Vertices`, Table 185 `/InkList`), +/// and the appearance is mapped onto `/Rect` (§12.5.5), so a `/Rect` that leaves them out would +/// cut off what is drawn: a highlight or ink stroke whose `/Rect` lies beside it drew nothing. +/// Once a point is outside, the box takes in the strokes' width and line endings too. Nothing +/// outside the crop box is shown or printed (§14.11.2), so the box grows no further than it; it +/// never shrinks `rect`, and a `/Rect` that holds every point is kept as it is. +fn point_reach(doc: &Document, d: &Dict, rect: [f64; 4], page: [f64; 4]) -> Option<[f64; 4]> { + if !rect.iter().all(|v| v.is_finite()) { + return None; + } + let (points, margin) = drawn_points(doc, d)?; + let mut reach = [f64::INFINITY, f64::INFINITY, f64::NEG_INFINITY, f64::NEG_INFINITY]; + for p in points.as_chunks::<2>().0 { + reach = [reach[0].min(p[0]), reach[1].min(p[1]), reach[2].max(p[0]), reach[3].max(p[1])]; + } + let clip = |b: [f64; 4]| [b[0].max(page[0]), b[1].max(page[1]), b[2].min(page[2]), b[3].min(page[3])]; + let shown = clip(reach); + // Strokes wholly off the page widen nothing. Points `/Rect` already holds don't either, and + // points off the page whose strokes reach onto it only as far as `/Rect` holds. + let stroked = clip([reach[0] - margin, reach[1] - margin, reach[2] + margin, reach[3] + margin]); + if stroked[0] > stroked[2] || stroked[1] > stroked[3] { + return None; + } + let inside = |b: [f64; 4]| rect[0] <= b[0] && rect[1] <= b[1] && b[2] <= rect[2] && b[3] <= rect[3]; + let on_page = shown[0] <= shown[2] && shown[1] <= shown[3]; + if inside(if on_page { shown } else { stroked }) { + return None; + } + let reach = stroked; + let out = [rect[0].min(reach[0]), rect[1].min(reach[1]), rect[2].max(reach[2]), rect[3].max(reach[3])]; + // A `/Rect` of its own can still be too wide for a width or height (`-1e308` to `1e308`). + let usable = (out[2] - out[0]).is_finite() && (out[3] - out[1]).is_finite(); + (usable && out != rect).then_some(out) +} + +/// The `/Rect` an annotation takes with its built appearance `stream`: the appearance's `/BBox` +/// where [`point_reach`] widened it past `/Rect`, as the appearance is mapped onto `/Rect` +/// (§12.5.5); `None` (keep `/Rect`) otherwise and for subtypes drawn within `/Rect`. +pub(crate) fn widened_rect(d: &Dict, stream: &Stream) -> Option<[f64; 4]> { + if !is_drawn_from_points(d) { + return None; + } + let bbox = <[f64; 4]>::try_from(nums(&stream.dict, b"BBox")?).ok()?; + (bbox.iter().all(|v| v.is_finite()) && Some(bbox) != normalised_rect(d)).then_some(bbox) +} + fn draw_body(context: &AppearanceContext<'_>, drawing: &mut AppearanceDrawing) -> Option<()> { match context.subtype { b"Highlight" | b"Underline" | b"StrikeOut" | b"Squiggly" => draw_markup(context, drawing), @@ -621,6 +818,30 @@ fn finish_appearance(context: &AppearanceContext<'_>, mut drawing: AppearanceDra Some(form(context.rect, &out, drawing.resources)) } +/// The line width of an underline or strike-out over a quad `h` high. +fn markup_line_width(h: f64) -> f64 { + (h * 0.07).clamp(0.5, 3.0) +} + +/// A squiggle's line width and amplitude over a quad `h` high: its peaks stand twice the +/// amplitude above the quad's bottom edge. +fn squiggle(h: f64) -> (f64, f64) { + ((h * 0.05).clamp(0.5, 2.0), (h * 0.06).max(0.75)) +} + +/// How far a text markup of `subtype` drawn over a quad `h` high reaches past the quad: half its +/// line's width, and for a squiggle, how far its peaks stand above a quad lower than they are. +fn markup_overreach(subtype: &[u8], h: f64) -> f64 { + match subtype { + b"Underline" | b"StrikeOut" => markup_line_width(h) / 2.0, + b"Squiggly" => { + let (lw, amp) = squiggle(h); + lw / 2.0 + (amp * 2.0 - h).max(0.0) + } + _ => 0.0, + } +} + fn draw_markup(context: &AppearanceContext<'_>, drawing: &mut AppearanceDrawing) -> Option<()> { let subtype = context.subtype; let stroke = context.stroke; @@ -639,15 +860,14 @@ fn draw_markup(context: &AppearanceContext<'_>, drawing: &mut AppearanceDrawing) c.push_str(&format!("{} {} m {} {} l {} {} l {} {} l h f\n", n(p1.0), n(p1.1), n(p2.0), n(p2.1), n(p4.0), n(p4.1), n(p3.0), n(p3.1))); } b"Underline" | b"StrikeOut" => { - let lw = (h * 0.07).clamp(0.5, 3.0); + let lw = markup_line_width(h); let t = if subtype == b"Underline" { 0.08 } else { 0.42 }; let (a, b) = (at(p3, p1, t), at(p4, p2, t)); c.push_str(&format!("{}{} w\n{} {} m {} {} l S\n", rg_stroke(col), n(lw), n(a.0), n(a.1), n(b.0), n(b.1))); } _ => { // Squiggly: a zigzag along the bottom of the quad. - let lw = (h * 0.05).clamp(0.5, 2.0); - let amp = (h * 0.06).max(0.75); + let (lw, amp) = squiggle(h); let len = (p4.0 - p3.0).hypot(p4.1 - p3.1); let step = (h / 4.0).max(1.5); let steps = ((len / step).ceil() as usize).clamp(1, 10_000); @@ -1078,7 +1298,11 @@ fn draw_free_text(context: &AppearanceContext<'_>, mut drawing: AppearanceDrawin let mut rect = rect; if d.contains(b"CL") { let cl = nums(d, b"CL").filter(|l| l.len() == 4 || l.len() == 6)?; - let rd = nums(d, b"RD").filter(|r| r.len() == 4 && r.iter().all(|x| *x >= 0.0))?; + // `/RD` is optional (Table 177; null is absent, §7.3.9): without it the text box is `/Rect`. + let rd = match d.get(b"RD") { + None | Some(Object::Null) => vec![0.0; 4], + Some(_) => nums(d, b"RD").filter(|r| r.len() == 4 && r.iter().all(|x| *x >= 0.0))?, + }; rect = [full[0] + rd[0], full[1] + rd[1], full[2] - rd[2], full[3] - rd[3]]; if rect[2] - rect[0] < 1.0 || rect[3] - rect[1] < 1.0 { return None; diff --git a/crates/annot/src/draw.rs b/crates/annot/src/draw.rs index 5c9701ce81..c0e82eb151 100644 --- a/crates/annot/src/draw.rs +++ b/crates/annot/src/draw.rs @@ -33,14 +33,70 @@ pub(crate) fn redraw_or_restore(doc: &mut Document, r: ObjRef, before: Dict) -> pub(crate) fn draw_appearance(doc: &mut Document, r: ObjRef) -> Result<(), AnnotError> { let d = annot_dict(doc, r); let subtype = String::from_utf8_lossy(d.name(b"Subtype").unwrap_or_default()).into_owned(); - let Some(stream) = build_appearance(doc, &d) else { return Err(AnnotError::Unsupported(subtype)) }; - apply_appearance(doc, r, &d, stream) + let page = appearance::is_drawn_from_points(&d).then(|| page_box_listing(doc, r)).flatten(); + let direct = with_direct_values(doc, &d); + let Some(stream) = build_appearance(doc, &d, &direct, page) else { return Err(AnnotError::Unsupported(subtype)) }; + let built = Built::new(stream, &direct); + apply_appearance(doc, r, &d, built) } -/// Draw the appearance for an annotation dictionary: the expensive half of [`set_appearance`]. -/// Pure from the dictionary and the objects it points at, so [`AppearanceCache`] can reuse it. -fn build_appearance(doc: &Document, d: &Dict) -> Option { - let mut stream = appearance::build(doc, d)?; +/// Whether [`set_appearance`] can draw the annotation `d`: read as the display reads it, with +/// references followed and the same fallback. Edits check this before changing anything. +pub(crate) fn can_draw(doc: &Document, d: &Dict) -> bool { + build_appearance(doc, d, &with_direct_values(doc, d), None).is_some() +} + +/// The visible region (`Page::crop`) of the page that lists `r` among its annotations: the page +/// its `/P` names when that page lists it, read up its own `/Parent` chain, else the first page +/// of the tree that lists it. An annotation is listed by one page only (ISO 32000-2 §12.5.2), so +/// both find the same page, and the first reads one page instead of walking the whole tree for +/// every comment redrawn. +fn page_box_listing(doc: &Document, r: ObjRef) -> Option<[f64; 4]> { + let named = annot_dict(doc, r).reference(b"P").filter(|p| annots(doc, *p).iter().any(|a| a.as_ref() == Some(r))); + if let Some(page) = named { + return Some(page_with_inherited_boxes(doc, page).crop(doc)); + } + pdfcraft_model::pages(doc).into_iter().find(|p| annots(doc, p.obj).iter().any(|a| a.as_ref() == Some(r))).map(|p| p.crop(doc)) +} + +/// The page `page` with the `/MediaBox` and `/CropBox` it inherits from its ancestors filled in +/// (ISO 32000-2 §7.7.3.4), enough for [`pdfcraft_model::Page::crop`]. Follows at most 64 +/// `/Parent` links and stops at a loop. +fn page_with_inherited_boxes(doc: &Document, page: ObjRef) -> pdfcraft_model::Page { + let mut dict = doc.get(page).as_dict().cloned().unwrap_or_default(); + let mut seen = std::collections::HashSet::from([page]); + let mut up = dict.reference(b"Parent"); + while let Some(node) = up { + if seen.len() > 64 || !seen.insert(node) { + break; + } + let obj = doc.get(node); + let Some(d) = obj.as_dict() else { break }; + for key in [&b"MediaBox"[..], b"CropBox"] { + if !dict.contains(key) + && let Some(v) = d.get(key) + { + dict.set(key.to_vec(), v.clone()); + } + } + up = d.reference(b"Parent"); + } + pdfcraft_model::Page { obj: page, dict } +} + +/// Draw the appearance for an annotation dictionary on a page whose visible region is `page` +/// (used by annotations drawn through points of their own, [`appearance::build_on_page`]): the +/// expensive half of [`set_appearance`]. `direct` is `d` with references followed +/// ([`with_direct_values`]). Pure from the dictionary, the objects it points at and the page, so +/// [`AppearanceCache`] can reuse it. +fn build_appearance(doc: &Document, d: &Dict, direct: &Dict, page: Option<[f64; 4]>) -> Option { + // References followed first. Where that turns up something the builder doesn't draw (a + // cloudy `/BE` on a square), the drawing made without them stays, rather than none. + let mut stream = match appearance::build_on_page(doc, direct, page) { + Some(stream) => stream, + None if !std::ptr::eq(direct, d) => appearance::build_on_page(doc, d, page)?, + None => return None, + }; // Picture-stamp restyling must retain the placement's page-axis correction. if d.get(b"PCPicture").is_some() { let matrix = d.get(b"AP").and_then(|ap| { @@ -56,10 +112,170 @@ fn build_appearance(doc: &Document, d: &Dict) -> Option { Some(stream) } +/// Entries that link the annotation to other objects rather than describe how it looks: left as +/// references by [`with_direct_values`]. +const LINK_KEYS: [&[u8]; 5] = [b"P", b"Popup", b"IRT", b"Parent", b"AP"]; + +/// How deep inside an entry [`with_direct_values`] follows references: `/InkList` is an array of +/// arrays, `/Border` may end in a dash array. +const INLINE_DEPTH: usize = 4; + +/// Values [`with_direct_values`] copies out of referenced arrays and dictionaries, or walks +/// before finding one too large to copy, for one annotation, so a file can't have a large object +/// copied or scanned for every entry that points at it. +pub(crate) const INLINE_BUDGET: usize = 1 << 16; + +/// Values copied or walked for all the annotations one display pass draws +/// ([`with_missing_appearances_cached`]), so a large object that many annotations point at is +/// not copied, or scanned and found too large, for each of them without bound. Past it, +/// references are left as they are. +pub(crate) const PASS_BUDGET: usize = 1 << 22; + +/// `d` with the references among the values [`appearance::build`] reads followed. Any value may be +/// given by reference (ISO 32000-2 §7.3.10): a border style dictionary (`/BS 5 0 R`, Table 168), a +/// colour, `/RD`, `/CL`, a `/W` inside `/BS`. The builder reads only direct values (apart from +/// `/QuadPoints`, which it resolves itself), so it draws from this copy. The entries in +/// [`LINK_KEYS`] and streams stay references; past [`INLINE_DEPTH`] levels or [`INLINE_BUDGET`] +/// copied values a reference is left as it is, which the builder treats as unreadable. `d` itself +/// when nothing is a reference. +pub(crate) fn with_direct_values<'d>(doc: &Document, d: &'d Dict) -> std::borrow::Cow<'d, Dict> { + with_direct_values_within(doc, d, &mut INLINE_BUDGET.clone()) +} + +/// [`with_direct_values`], copying at most [`INLINE_BUDGET`] values and no more than `pass` has +/// left, which is reduced by what was copied. +pub(crate) fn with_direct_values_within<'d>(doc: &Document, d: &'d Dict, pass: &mut usize) -> std::borrow::Cow<'d, Dict> { + let allowed = INLINE_BUDGET.min(*pass); + let mut budget = allowed; + let mut out: Option = None; + for (key, value) in d.iter() { + if LINK_KEYS.contains(&key.as_slice()) { + continue; + } + if let Some(direct) = direct_value(doc, value, INLINE_DEPTH, &mut budget) { + out.get_or_insert_with(|| d.clone()).set(key.clone(), direct); + } + } + *pass = pass.saturating_sub(allowed.saturating_sub(budget)); + out.map_or(std::borrow::Cow::Borrowed(d), std::borrow::Cow::Owned) +} + +/// The number of values in `o`, counting into its arrays and dictionaries (not through +/// references, which are charged when followed), or, past `limit`, `Err` with the values it +/// visited or queued before finding that out (at least one), so a rejected walk is charged too. +/// Walks at most `limit` values and holds no more than `limit` waiting. +fn weight(o: &Object, limit: usize) -> Result { + let mut count = 0usize; + let mut stack = vec![o]; + while let Some(o) = stack.pop() { + let spent = count.saturating_add(stack.len()).max(1); + count = match count.checked_add(1).filter(|c| *c <= limit) { + Some(c) => c, + None => return Err(spent), + }; + let inner = match o { + Object::Array(items) => items.len(), + Object::Dict(entries) => entries.len(), + _ => continue, + }; + // Every value waiting is counted later, so more than `limit` in all is over it already. + if count.saturating_add(stack.len()).saturating_add(inner) > limit { + return Err(spent.saturating_add(1)); + } + match o { + Object::Array(items) => stack.extend(items.iter()), + Object::Dict(entries) => stack.extend(entries.iter().map(|(_, v)| v)), + _ => {} + } + } + Ok(count) +} + +/// `o` with references followed `depth` levels into arrays and dictionaries, or `None` when there +/// is nothing to follow (or it may not be: a stream, an over-budget object, a chain of references). +fn direct_value(doc: &Document, o: &Object, depth: usize, budget: &mut usize) -> Option { + let held; + let (base, followed) = match o { + Object::Ref(_) => { + // A spent budget follows nothing more, not even to find out what is there. + if *budget == 0 { + return None; + } + held = doc.resolve(o); + if matches!(&*held, Object::Stream(_) | Object::Ref(_)) { + return None; + } + // Everything copied counts, inner arrays and dictionaries too, and so does the walk + // that finds an object too large: many references to one would otherwise each walk + // it again for free. + match weight(&held, *budget) { + Ok(n) => *budget = budget.checked_sub(n)?, + Err(spent) => { + *budget = budget.saturating_sub(spent); + return None; + } + } + (&*held, true) + } + other => (other, false), + }; + let Some(next) = depth.checked_sub(1) else { return followed.then(|| base.clone()) }; + match base { + Object::Array(items) => { + let mut changed: Option> = None; + for (i, item) in items.iter().enumerate() { + if let Some(direct) = direct_value(doc, item, next, budget) { + let copy = changed.get_or_insert_with(|| items.clone()); + if let Some(slot) = copy.get_mut(i) { + *slot = direct; + } + } + } + changed.map(Object::Array).or_else(|| followed.then(|| base.clone())) + } + Object::Dict(entries) => { + let mut changed: Option = None; + for (key, value) in entries.iter() { + if let Some(direct) = direct_value(doc, value, next, budget) { + changed.get_or_insert_with(|| entries.clone()).set(key.clone(), direct); + } + } + changed.map(Object::Dict).or_else(|| followed.then(|| base.clone())) + } + _ => followed.then(|| base.clone()), + } +} + +/// A built appearance and what storing it needs from the dictionary it was drawn from, read +/// with references followed, so a cached one is stored without following them again. +#[derive(Clone, Debug)] +struct Built { + stream: Stream, + /// The first character the standard fonts can't draw (it came out as `?`). + missing: Option, + /// A note's or attachment's `/Rect`, as written. + rect: Option>, + /// Points (/QuadPoints, /L, /Vertices, /InkList) past /Rect: its appearance covers them, so + /// /Rect does too. + markup_rect: Option<[f64; 4]>, +} + +impl Built { + fn new(stream: Stream, direct: &Dict) -> Self { + let missing = appearance::undrawable_text(direct).map(|(_, c)| c); + let rect = matches!(direct.name(b"Subtype"), Some(b"Text" | b"FileAttachment")) + .then(|| direct.get(b"Rect")?.as_array()?.iter().map(|o| o.as_f64()).collect::>>()) + .flatten(); + let markup_rect = appearance::widened_rect(direct, &stream); + Built { stream, missing, rect, markup_rect } + } +} + /// Store a built appearance on the annotation: the cheap half of [`set_appearance`]. -fn apply_appearance(doc: &mut Document, r: ObjRef, d: &Dict, mut stream: Stream) -> Result<(), AnnotError> { - // Characters the standard fonts can't draw came out as `?`: mark the annotation with the first. - let missing = appearance::undrawable_text(d).map(|(_, c)| c); +fn apply_appearance(doc: &mut Document, r: ObjRef, d: &Dict, built: Built) -> Result<(), AnnotError> { + // `missing`: characters the standard fonts can't draw came out as `?`; the annotation is + // marked with the first. + let Built { mut stream, missing, rect, markup_rect } = built; // A text box's Thai glyphs are streams of their own. pdfcraft_fonts::shaped_text::store_streams(doc, &mut stream.dict); let ap = doc.add(Object::Stream(stream)); @@ -72,17 +288,13 @@ fn apply_appearance(doc: &mut Document, r: ObjRef, d: &Dict, mut stream: Stream) // A note icon has a fixed size (`NOTE_SIZE`); viewers draw it at the upper-left corner of `/Rect`. // The appearance's box is mapped onto `/Rect`, so any other `/Rect` (a large one, or one with // no width, as LaTeX `pdfcomment` writes) would stretch the icon or shrink it to nothing. - let icon_rect = matches!(d.name(b"Subtype"), Some(b"Text" | b"FileAttachment")) - .then(|| d.get(b"Rect")?.as_array()?.iter().map(|o| o.as_f64()).collect::>>()) - .flatten() - .filter(|q| q.len() == 4 && finite(q)) - .map(|q| { - let (x, top) = (q[0].min(q[2]), q[1].max(q[3])); - [x, top - NOTE_SIZE, x + NOTE_SIZE, top] - }); + let icon_rect = rect.filter(|q| q.len() == 4 && finite(q)).map(|q| { + let (x, top) = (q[0].min(q[2]), q[1].max(q[3])); + [x, top - NOTE_SIZE, x + NOTE_SIZE, top] + }); doc.update_dict(r, |d| { d.set(b"AP".to_vec(), Object::Dict(apd)); - if let Some(rect) = icon_rect { + if let Some(rect) = icon_rect.or(markup_rect) { d.set(b"Rect".to_vec(), num_array(&rect)); } d.remove(b"AS"); @@ -103,7 +315,9 @@ fn apply_appearance(doc: &mut Document, r: ObjRef, d: &Dict, mut stream: Stream) /// refreshes) and keep it for content-only ones. #[derive(Clone, Debug, Default)] pub struct AppearanceCache { - entries: std::collections::HashMap, + /// Each appearance with the page region it was drawn for (see [`appearance::build_on_page`]): + /// a changed crop box draws it again. + entries: std::collections::HashMap, Built)>, bytes: usize, hits: u64, builds: u64, @@ -118,20 +332,21 @@ impl AppearanceCache { *self = Self::default(); } - fn get(&mut self, r: ObjRef) -> Option { - let stream = self.entries.get(&r)?.clone(); + fn get(&mut self, r: ObjRef, crop: Option<[f64; 4]>) -> Option { + let built = self.entries.get(&r).filter(|(drawn_for, _)| *drawn_for == crop)?.1.clone(); self.hits += 1; - Some(stream) + Some(built) } - fn insert(&mut self, r: ObjRef, stream: &Stream) { + fn insert(&mut self, r: ObjRef, crop: Option<[f64; 4]>, built: &Built) { + let stream = &built.stream; self.builds += 1; if self.entries.len() >= 1_000_000 { self.clear(); } if self.bytes.saturating_add(stream.raw.len()) <= Self::MAX_BYTES { self.bytes += stream.raw.len(); - self.entries.insert(r, stream.clone()); + self.entries.insert(r, (crop, built.clone())); } // Else over budget: keep serving the entries already cached instead of growing. } @@ -165,6 +380,9 @@ pub fn with_missing_appearances_cached(doc: &Document, cache: &mut AppearanceCac // reads `doc`, which never gets the appearance, so every repeat used to add another stream to // the copy, and the display bytes grew with the number of listings. let mut seen = std::collections::HashSet::new(); + // Pages' visible regions, read once and only when an annotation drawn through its points is. + let mut crops: Option> = None; + let mut budget = PASS_BUDGET; for page in page_refs(doc).ok()? { for r in annots(doc, page).iter().filter_map(Object::as_ref) { if !seen.insert(r) { @@ -178,9 +396,12 @@ pub fn with_missing_appearances_cached(doc: &Document, cache: &mut AppearanceCac if has_normal { continue; } - if let Some(stream) = cache.get(r) { + let crop = appearance::is_drawn_from_points(&d) + .then(|| crops.get_or_insert_with(|| pdfcraft_model::pages(doc).iter().map(|p| (p.obj, p.crop(doc))).collect()).get(&page).copied()) + .flatten(); + if let Some(built) = cache.get(r, crop) { // Fails only when `r` isn't a dictionary, which the cached build ruled out. - apply_appearance(copy.get_or_insert_with(|| doc.clone()), r, &d, stream).ok(); + apply_appearance(copy.get_or_insert_with(|| doc.clone()), r, &d, built).ok(); continue; } // Another program's stamp gets a drawing for display only (ISO 32000-2 Table 184). Its @@ -190,10 +411,12 @@ pub fn with_missing_appearances_cached(doc: &Document, cache: &mut AppearanceCac bare.remove(b"AP"); appearance::display_stamp(&bare) }; - let Some(stream) = build_appearance(doc, &d).or_else(other_stamp) else { continue }; - cache.insert(r, &stream); + let direct = with_direct_values_within(doc, &d, &mut budget); + let Some(stream) = build_appearance(doc, &d, &direct, crop).or_else(other_stamp) else { continue }; + let built = Built::new(stream, &direct); + cache.insert(r, crop, &built); // Fails only when `r` isn't a dictionary, which the build just ruled out. - apply_appearance(copy.get_or_insert_with(|| doc.clone()), r, &d, stream).ok(); + apply_appearance(copy.get_or_insert_with(|| doc.clone()), r, &d, built).ok(); } } copy diff --git a/crates/annot/src/info.rs b/crates/annot/src/info.rs index c4913e7ab8..c86b9f8f78 100644 --- a/crates/annot/src/info.rs +++ b/crates/annot/src/info.rs @@ -245,7 +245,10 @@ pub fn props(doc: &Document, page: usize, index: usize) -> Option { let p = page_ref(doc, page).ok()?; let entry = annots(doc, p).get(index).cloned()?; let obj = doc.resolve(&entry); - let d = obj.as_dict()?; + // Values given by reference (`/C 7 0 R`, `/BS 5 0 R`, ISO 32000-2 §7.3.10) read as the display draws them. + let raw = obj.as_dict()?; + let direct = crate::draw::with_direct_values(doc, raw); + let d = direct.as_ref(); let subtype = String::from_utf8_lossy(d.name(b"Subtype")?).into_owned(); let color = if subtype == "FreeText" { Some(appearance::parse_da(d).0) } else { drawn_color(d, b"C") }; let width = matches!(subtype.as_str(), "Square" | "Circle" | "Line" | "Ink" | "Polygon" | "PolyLine" | "FreeText").then(|| { @@ -263,7 +266,7 @@ pub fn props(doc: &Document, page: usize, index: usize) -> Option { width, icon: (subtype == "Text").then(|| d.name(b"Name").and_then(|n| NoteIcon::from_name(&String::from_utf8_lossy(n))).unwrap_or(NoteIcon::Note)), modified: text_value(doc, d, b"M"), - restylable: appearance::build(doc, d).is_some(), + restylable: crate::draw::can_draw(doc, raw), locked: annotation_flags(doc, d) & FLAG_LOCKED != 0, fillable: fillable(&subtype), fill: fill_of(d), diff --git a/crates/annot/src/modify.rs b/crates/annot/src/modify.rs index 6fbb0f35c3..78200f5c16 100644 --- a/crates/annot/src/modify.rs +++ b/crates/annot/src/modify.rs @@ -785,7 +785,7 @@ fn style_one(doc: &mut Document, r: ObjRef, change: StyleChange, meta: &Meta) -> let d = annot_dict(doc, r); let subtype = String::from_utf8_lossy(d.name(b"Subtype").unwrap_or_default()).into_owned(); // Check before changing anything: a stale appearance would contradict the new style. - if appearance::build(doc, &d).is_none() { + if !crate::draw::can_draw(doc, &d) { return Err(AnnotError::Unsupported(subtype.clone())); } if color.is_some_and(|c| !finite(&c)) || opacity.is_some_and(|o| !o.is_finite()) || width.is_some_and(|w| !w.is_finite()) { @@ -805,8 +805,12 @@ fn style_one(doc: &mut Document, r: ObjRef, change: StyleChange, meta: &Meta) -> // So does a text box inside an oval or a circle, which goes around it again. let shape = if free_text { BorderShape::of(&d) } else { BorderShape::Rectangle }; let typewriter = free_text && (d.name(b"IT") == Some(b"FreeTextTypeWriter") || shape != BorderShape::Rectangle) && !d.contains(b"CL"); + // A border style given by reference (`/BS 5 0 R`, ISO 32000-2 §7.3.10): its width and other + // entries, which a new width keeps. + let direct = crate::draw::with_direct_values(doc, &d); + let (old_width, old_bs) = (border_width_of(&direct), direct.get(b"BS").and_then(Object::as_dict).cloned()); let regrow = width.filter(|_| typewriter).and_then(|w| { - let by = w.max(0.0) - if d.contains(b"BS") || d.contains(b"Border") { border_width_of(&d) } else { 0.0 }; + let by = w.max(0.0) - if d.contains(b"BS") || d.contains(b"Border") { old_width } else { 0.0 }; let rect = grow(text_box_of(&d)?, by); (by != 0.0 && finite(&rect) && rect[2] - rect[0] >= 1.0 && rect[3] - rect[1] >= 1.0).then_some((rect, w.max(0.0))) }); @@ -830,9 +834,9 @@ fn style_one(doc: &mut Document, r: ObjRef, change: StyleChange, meta: &Meta) -> set_part_keys(d, appearance::PartOpacity { line: o, fill: o, text: o }); } if let Some(w) = width { - let old_w = border_width_of(d); + let old_w = old_width; let w = w.max(0.0); - let mut bs = d.get(b"BS").and_then(|b| b.as_dict()).cloned().unwrap_or_default(); + let mut bs = old_bs.unwrap_or_default(); bs.set(b"W".to_vec(), Object::Real(w)); d.set(b"BS".to_vec(), Object::Dict(bs)); d.remove(b"Border"); @@ -946,7 +950,7 @@ pub fn set_fill(doc: &mut Document, page: usize, index: usize, fill: Option return Err(AnnotError::Invalid("only rectangles, ovals, polygons and text boxes can be filled".into())); } // Check before changing anything: a stale appearance would contradict the new fill. - if appearance::build(doc, &d).is_none() { + if !crate::draw::can_draw(doc, &d) { return Err(AnnotError::Unsupported(subtype)); } if fill.is_some_and(|c| !finite(&c)) { @@ -987,7 +991,7 @@ pub fn set_part_opacity(doc: &mut Document, page: usize, index: usize, fill: Opt return Err(AnnotError::Invalid("invalid opacity".into())); } // Check before changing anything: a stale appearance would contradict the new opacity. - if appearance::build(doc, &d).is_none() { + if !crate::draw::can_draw(doc, &d) { return Err(AnnotError::Unsupported(subtype)); } refuse_foreign_text(doc, &d)?; diff --git a/crates/annot/src/tests.rs b/crates/annot/src/tests.rs index bfb780dabb..b0220b1023 100644 --- a/crates/annot/src/tests.rs +++ b/crates/annot/src/tests.rs @@ -1724,6 +1724,190 @@ fn malformed_optional_interior_color_does_not_hide_shape_strokes() { assert!(list(&doc, 0)[0].get(b"AP").is_none(), "display generation leaves the source document unchanged"); } +/// Any value may be given by reference (ISO 32000-2 §7.3.10). A Square whose border style +/// dictionary (Table 168), colour and dash array are indirect objects is drawn with them, not +/// with the 1 pt solid default; a reference cycle inside `/BS` is not followed for ever. +#[test] +fn indirect_border_style_and_colour_are_drawn() { + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Square /P 3 0 R /Rect [20 20 180 180] /BS 5 0 R /C 6 0 R >>", + b"<< /Type /Border /W 7 0 R /S /D /D 8 0 R /Self 5 0 R >>", + b"[0 0 1]", + b"6", + b"[4 2]", + ]); + let display = with_missing_appearances(&doc).expect("the square gets a display appearance"); + let shown = list(&display, 0).remove(0); + let content = ap_content(&display, &shown); + assert!(content.contains("0 0 1 RG\n6 w\n[4 2] 0 d\n"), "the indirect /BS, /W, /D and /C are used: {content}"); + assert!(content.contains("23 23 154 154 re"), "the 6 pt border is inset by half its width: {content}"); + assert_eq!(shown.get(b"BS"), Some(&Object::Ref(ObjRef::new(5, 0))), "the display copy keeps the reference"); + assert_eq!(shown.get(b"P"), Some(&Object::Ref(ObjRef::new(3, 0)))); + let props = props(&doc, 0, 0).expect("props"); + assert_eq!(props.width, Some(6.0), "the properties show the width drawn"); +} + +/// A followed reference the builder can't draw (a cloudy border effect on a square, which it +/// leaves alone) keeps the drawing made without it instead of none. +#[test] +fn an_undrawable_indirect_value_keeps_the_plain_drawing() { + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Square /Rect [20 20 180 180] /BE 5 0 R /C [1 0 0] >>", + b"<< /S /C /I 1 >>", + ]); + let display = with_missing_appearances(&doc).expect("the square is still drawn"); + let shown = list(&display, 0).remove(0); + assert!(ap_content(&display, &shown).contains("1 0 0 RG\n1 w\n")); +} + +/// Following references copies a bounded number of values: an indirect array past the budget +/// stays a reference (and the comment is not drawn from it), a chain that never ends is not +/// followed, and links to other objects are never copied. +#[test] +fn direct_values_are_bounded() { + let mut doc = fixture(); + let huge = doc.add(Object::Array(vec![Object::Int(0); (1 << 16) + 1])); + let chain = doc.add(Object::Null); + doc.set(chain, Object::Ref(chain)); + let mut d = Dict::new(); + d.set(b"Subtype".to_vec(), Object::name("Square")); + d.set(b"Rect".to_vec(), Object::Ref(doc.add(num_array(&[0.0, 0.0, 10.0, 10.0])))); + d.set(b"Vertices".to_vec(), Object::Ref(huge)); + d.set(b"IC".to_vec(), Object::Ref(chain)); + d.set(b"Popup".to_vec(), Object::Ref(huge)); + let direct = crate::draw::with_direct_values(&doc, &d); + assert_eq!(direct.get(b"Rect").and_then(Object::as_array).map(Vec::len), Some(4), "a small array is followed"); + assert_eq!(direct.get(b"Vertices"), Some(&Object::Ref(huge)), "over the budget: left as a reference"); + assert_eq!(direct.get(b"IC"), Some(&Object::Ref(chain)), "a chain of references is left alone"); + assert_eq!(direct.get(b"Popup"), Some(&Object::Ref(huge)), "a link is not copied"); + let mut plain = Dict::new(); + plain.set(b"W".to_vec(), Object::Int(2)); + assert!(matches!(crate::draw::with_direct_values(&doc, &plain), std::borrow::Cow::Borrowed(_)), "nothing to follow: no copy"); +} + +/// A new width for a comment whose border style is an indirect object (§7.3.10) keeps that +/// dictionary's other entries (a dashed style stays dashed) and leaves the shared object alone. +#[test] +fn new_width_keeps_an_indirect_border_style() { + let mut doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Square /Rect [20 20 180 180] /BS 5 0 R /C [1 0 0] >>", + b"<< /W 6 /S /D /D [4 2] >>", + ]); + set_style(&mut doc, 0, 0, StyleChange::new(None, None, Some(3.0), None), &meta("")).unwrap(); + let doc = reopen(&doc); + let d = list(&doc, 0).remove(0); + let bs = d.get(b"BS").and_then(Object::as_dict).expect("a direct /BS"); + assert_eq!(bs.get(b"W").and_then(Object::as_f64), Some(3.0)); + assert_eq!(bs.name(b"S"), Some(b"D".as_slice()), "the dashed style is kept"); + assert!(ap_content(&doc, &d).contains("3 w\n[4 2] 0 d\n")); + let shared = doc.get(ObjRef::new(5, 0)); + assert_eq!(shared.as_dict().and_then(|b| b.get(b"W")).and_then(Object::as_f64), Some(6.0), "the shared object is unchanged"); +} + +/// A colour given by reference (`/C 6 0 R`, §7.3.10) is read by the properties and the edits as +/// the display draws it: the properties show it, the appearance can be restyled and filled. +#[test] +fn an_indirect_colour_reads_and_restyles_as_drawn() { + let mut doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Square /Rect [20 20 180 180] /C 5 0 R /IC 6 0 R >>", + b"[0 0 1]", + b"[1 1 0]", + ]); + let p = props(&doc, 0, 0).expect("props"); + assert_eq!((p.color, p.fill, p.restylable), (Some([0.0, 0.0, 1.0]), Some([1.0, 1.0, 0.0]), true)); + set_style(&mut doc, 0, 0, StyleChange::new(None, None, Some(3.0), None), &meta("")).expect("restylable"); + set_fill(&mut doc, 0, 0, Some([0.0, 1.0, 0.0]), &meta("")).expect("fillable"); + let d = list(&doc, 0).remove(0); + let content = ap_content(&doc, &d); + assert!(content.contains("0 0 1 RG") && content.contains("0 1 0 rg") && content.contains("3 w"), "{content}"); +} + +/// References are followed into everything they point at, and all of it is counted: a referenced +/// array of two arrays of 40,000 numbers is over the per-annotation budget. One display pass +/// copies at most [`crate::draw::PASS_BUDGET`] values in all, so an array many annotations point +/// at is not copied again for each of them. +#[test] +fn following_references_is_bounded_per_value_and_per_pass() { + let mut doc = fixture(); + let inner = Object::Array(vec![Object::Int(0); 40_000]); + let nested = doc.add(Object::Array(vec![inner.clone(), inner])); + let mut d = Dict::new(); + d.set(b"Subtype".to_vec(), Object::name("Square")); + d.set(b"Junk".to_vec(), Object::Ref(nested)); + let direct = crate::draw::with_direct_values(&doc, &d); + assert!(direct.get(b"Junk") == Some(&Object::Ref(nested)), "80,003 values: left as a reference"); + + let shared = doc.add(Object::Array(vec![Object::Int(0); 60_000])); + d.set(b"Junk".to_vec(), Object::Ref(shared)); + let mut pass = crate::draw::PASS_BUDGET; + let copies = (0..100).filter(|_| matches!(crate::draw::with_direct_values_within(&doc, &d, &mut pass), std::borrow::Cow::Owned(_))).count(); + assert_eq!(copies, crate::draw::PASS_BUDGET / 60_001, "copies stop when the pass budget is spent"); +} + +/// Finding an object too large to copy costs the walk that found it out: an array of two arrays +/// of 40,000 numbers that many annotations point at is walked about 40,000 values deep before it +/// is rejected, and that is charged to the annotation and to the pass, so after a bounded number +/// of such walks the pass has no room left to walk it again. A reference to it repeated in one annotation +/// is not walked again in full, and a small referenced colour after it is still read. +#[test] +fn a_rejected_large_object_is_charged_to_the_budgets() { + let mut doc = fixture(); + let inner = Object::Array(vec![Object::Int(0); 40_000]); + let nested = doc.add(Object::Array(vec![inner.clone(), inner])); + let red = doc.add(Object::Array(vec![Object::Int(1), Object::Int(0), Object::Int(0)])); + let mut d = Dict::new(); + d.set(b"Subtype".to_vec(), Object::name("Square")); + d.set(b"Junk".to_vec(), Object::Array(vec![Object::Ref(nested); 1_000])); + d.set(b"C".to_vec(), Object::Ref(red)); + let mut pass = crate::draw::PASS_BUDGET; + let direct = crate::draw::with_direct_values_within(&doc, &d, &mut pass); + assert_eq!(direct.get(b"C"), Some(&Object::Array(vec![Object::Int(1), Object::Int(0), Object::Int(0)]))); + let charged = crate::draw::PASS_BUDGET - pass; + assert!((40_000..=crate::draw::INLINE_BUDGET).contains(&charged), "the rejected walk is charged once: {charged}"); + d.set(b"Junk".to_vec(), Object::Ref(nested)); + for _ in 0..crate::draw::PASS_BUDGET / 40_000 { + crate::draw::with_direct_values_within(&doc, &d, &mut pass); + } + assert!(pass < 40_000, "after {} annotations the pass has no room for another full walk: {pass}", crate::draw::PASS_BUDGET / 40_000 + 1); +} + +/// `/RD` is optional (ISO 32000-2 Table 177): a callout without it has its text box at `/Rect` +/// and is drawn, box, text and leader line. A callout line and `/RD` given by reference are read too. +#[test] +fn callout_without_rd_is_drawn() { + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 400 400] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R 5 0 R 6 0 R] >>", + b"<< /Type /Annot /Subtype /FreeText /IT /FreeTextCallout /Rect [100 100 300 160] /CL [20 20 60 60 100 130] /LE /OpenArrow /C [1 1 0] /DA (0 g /Helv 12 Tf) /Contents (callout) >>", + b"<< /Type /Annot /Subtype /FreeText /IT /FreeTextCallout /Rect [10 200 300 300] /CL 7 0 R /RD 8 0 R /C [1 1 0] /DA (0 g /Helv 12 Tf) /Contents (indirect) >>", + b"<< /Type /Annot /Subtype /FreeText /IT /FreeTextCallout /Rect [10 300 300 390] /CL [5 305 10 350] /RD [1 2] /C [1 1 0] /DA (0 g /Helv 12 Tf) /Contents (bad) >>", + b"[15 205 40 250 100 250]", + b"[90 0 0 0]", + ]); + let display = with_missing_appearances(&doc).expect("the callouts get a display appearance"); + let shown = list(&display, 0); + let plain = ap_content(&display, &shown[0]); + assert!(plain.contains("1 1 0 rg\n100 100 200 60 re f"), "the yellow box fills /Rect: {plain}"); + assert!(plain.contains("20 20 m\n60 60 l\n100 130 l\nS"), "the leader line: {plain}"); + assert!(plain.contains("(callout) Tj"), "the text: {plain}"); + let indirect = ap_content(&display, &shown[1]); + assert!(indirect.contains("100 200 200 100 re f") && indirect.contains("15 205 m"), "indirect /CL and /RD: {indirect}"); + assert!(shown[2].get(b"AP").is_none(), "a malformed /RD is still not guessed at"); +} + /// #3175: viewers draw a note without an appearance as a fixed-size icon at the upper-left corner /// of its `/Rect`. The display copy must not stretch the icon over a large `/Rect` or shrink it /// to nothing in one with no width. @@ -3089,8 +3273,10 @@ fn a_referenced_colour_is_listed_like_a_direct_one() { } /// A reference among the four entries is not a non-number read as 0: `/C [1 0 0 6 0 R]` with -/// object 6 the number 1 is CMYK black (§7.3.10), so drawing it cyan would be wrong. The builder -/// doesn't follow it and draws no appearance, as for any colour it can't read. +/// object 6 the number 1 is CMYK black (§7.3.10), so drawing it cyan would be wrong. The display +/// and the properties follow the reference (`draw::with_direct_values`) and draw and +/// read it black; the comment list, which follows only a `/C` given whole by reference, reads no +/// colour rather than cyan. #[test] fn a_four_entry_colour_with_a_reference_is_not_read_as_zero() { let doc = document_of(&[ @@ -3101,11 +3287,374 @@ fn a_four_entry_colour_with_a_reference_is_not_read_as_zero() { b"<< /Type /Annot /Subtype /Square /Rect [55 5 95 35] /C [1 0 0] >>", b"1", ]); - let display = with_missing_appearances(&doc).expect("the RGB square gets a fallback appearance"); + let display = with_missing_appearances(&doc).expect("both squares get a fallback appearance"); let squares = list(&display, 0); - assert!(!squares[0].contains(b"AP"), "no cyan stand-in for CMYK black"); - assert_eq!(props(&doc, 0, 0).and_then(|p| p.color), None); - assert_eq!(summaries(&doc)[0].color, None, "nor in the comment list"); + let black = ap_content(&display, &squares[0]); + assert!(black.contains("0 0 0 RG") && !black.contains("0 1 1 RG"), "CMYK black, no cyan stand-in: {black}"); + assert_eq!(props(&doc, 0, 0).and_then(|p| p.color), Some([0.0, 0.0, 0.0])); + assert_ne!(summaries(&doc)[0].color, Some([0.0, 1.0, 1.0]), "nor cyan in the comment list"); +} + +/// A text markup annotation is drawn over its `/QuadPoints` (ISO 32000-2 Table 182), and an +/// appearance is mapped onto `/Rect` (§12.5.5). A highlight whose `/Rect` lies beside its text +/// (here written with its corners swapped, as a real file has it) got an appearance cut to that +/// `/Rect`, so nothing showed. The display copy's appearance and `/Rect` now take in the +/// quadrilaterals; a markup whose `/Rect` already holds them keeps it. +#[test] +fn text_markup_whose_quads_reach_past_its_rect_is_drawn_over_them() { + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R 5 0 R 6 0 R 7 0 R 8 0 R 9 0 R 10 0 R] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [335.1 370.83 324.6 471.78] /C [1 1 0] /QuadPoints [370 287 471 287 370 277 471 277] >>", + b"<< /Type /Annot /Subtype /Underline /Rect [10 10 20 20] /C [0 0 1] /QuadPoints [100 120 200 120 100 100 200 100] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [50 50 150 70] /C [1 1 0] /QuadPoints [60 68 140 68 60 52 140 52] >>", + b"<< /Type /Annot /Subtype /StrikeOut /Rect [10 10 20 20] /C [1 0 0] /QuadPoints [100 220 200 220 100 200 200 200] >>", + b"<< /Type /Annot /Subtype /Squiggly /Rect [10 10 20 20] /C [0 1 0] /QuadPoints [100 320 200 320 100 300 200 300] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [-1e30 20 1e30 20 -1e30 10 1e30 10] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [-1e308 10 1e308 20] /C [1 1 0] /QuadPoints [100 120 200 120 100 100 200 100] >>", + ]); + let display = with_missing_appearances(&doc).expect("the markups need an appearance"); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + let bbox = |d: &Dict| { + let n = d.get(b"AP").map(|a| display.resolve(a)).and_then(|a| a.as_dict().and_then(|a| a.reference(b"N"))).unwrap(); + let obj = display.get(n); + let Object::Stream(s) = &*obj else { panic!("AP is not a stream") }; + nums_of(&s.dict, b"BBox") + }; + assert_eq!(rect(&shown[0]), [324.6, 277.0, 471.0, 471.78], "the highlight's /Rect takes in its text"); + assert_eq!(bbox(&shown[0]), rect(&shown[0])); + assert!(ap_content(&display, &shown[0]).contains("370 287 m 471 287 l 471 277 l 370 277 l h f"), "the quad is filled"); + // Half the underline's width (0.07 × 20 / 2) past the quad. + assert_eq!(rect(&shown[1]), [10.0, 10.0, 200.7, 120.7], "an underline's too"); + assert_eq!(bbox(&shown[1]), rect(&shown[1])); + assert_eq!(rect(&shown[2]), [50.0, 50.0, 150.0, 70.0], "a /Rect that holds the quads stays as it is"); + assert_eq!(rect(&shown[3]), [10.0, 10.0, 200.7, 220.7], "a strikeout's too"); + assert_eq!(bbox(&shown[3]), rect(&shown[3])); + assert_eq!(rect(&shown[4]), [10.0, 10.0, 200.5, 320.5], "a squiggly's too"); + assert_eq!(bbox(&shown[4]), rect(&shown[4])); + assert_eq!(rect(&shown[5]), [0.0, 10.0, 612.0, 20.0], "quads far past the page reach only as far as it shows"); + // A /Rect too wide for its width to be a number keeps its size: the on-page quad above it + // doesn't make it taller. + assert_eq!(rect(&shown[6]), [-1e308, 10.0, 1e308, 20.0], "an overflowing /Rect is not widened"); + assert_eq!(rect(&list(&doc, 0)[0]), [335.1, 370.83, 324.6, 471.78], "the source document is unchanged"); +} + +/// The box widens over the quads the markup is drawn over, read as every `/QuadPoints` reader +/// reads them (`pdfcraft_geom::quad_points::usable_quads`): an indirect array or entry is +/// followed, and a malformed quad or trailing partial one is skipped while the others still widen +/// the box, so what is drawn is not cut to `/Rect`. +#[test] +fn text_markup_reach_reads_quads_like_the_drawing() { + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R 5 0 R 6 0 R] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints 7 0 R >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [100 120 /oops 120 100 100 200 100 100 220 200 220 100 200 200 200 1 2 3] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [100 120 200 120 100 100 8 0 R 100] >>", + b"[100 120 200 120 100 100 200 100]", + b"200", + ]); + let display = with_missing_appearances(&doc).expect("the markups need an appearance"); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + assert_eq!(rect(&shown[0]), [10.0, 10.0, 200.0, 120.0], "an indirect array widens the box"); + assert_eq!(ap_bbox(&display, &shown[0]), rect(&shown[0])); + assert_eq!(rect(&shown[1]), [10.0, 10.0, 200.0, 220.0], "the usable quad still widens it"); + assert_eq!(ap_bbox(&display, &shown[1]), rect(&shown[1])); + assert_eq!(rect(&shown[2]), [10.0, 10.0, 200.0, 120.0], "an indirect entry is followed"); +} + +/// `n` copies of the quadrilateral `[100 120 200 120 100 100 200 100]`, as `/QuadPoints`. +fn repeated_quads(n: usize) -> String { + format!("[{}]", vec!["100 120 200 120 100 100 200 100"; n].join(" ")) +} + +/// The `/BBox` of `d`'s normal appearance in `doc`. +fn ap_bbox(doc: &Document, d: &Dict) -> Vec { + let n = d.get(b"AP").map(|a| doc.resolve(a)).and_then(|a| a.as_dict().and_then(|a| a.reference(b"N"))).expect("has /AP /N"); + let obj = doc.get(n); + let Object::Stream(s) = &*obj else { panic!("AP is not a stream") }; + nums_of(&s.dict, b"BBox") +} + +/// The widened box stops at the page's visible region (its crop box, ISO 32000-2 §14.11.2), and +/// quadrilaterals wholly outside it widen nothing. A markup with more than +/// `appearance::MAX_MARKUP_QUADS` quadrilaterals keeps its `/Rect`. An annotation with an +/// appearance of its own, and one that isn't text markup, keep theirs whatever `/QuadPoints` say. +#[test] +fn text_markup_reach_stops_at_the_page_and_the_quad_cap() { + let cap = appearance::MAX_MARKUP_QUADS; + let at_cap = format!("<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints {} >>", repeated_quads(cap)); + let past_cap = format!("<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints {} >>", repeated_quads(cap + 1)); + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /CropBox [0 0 300 400] /Annots [4 0 R 5 0 R 6 0 R 7 0 R 8 0 R 9 0 R] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [100 380 500 380 100 360 500 360] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [400 500 450 500 400 480 450 480] >>", + at_cap.as_bytes(), + past_cap.as_bytes(), + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [100 120 200 120 100 100 200 100] /AP << /N 10 0 R >> >>", + b"<< /Type /Annot /Subtype /Square /Rect [10 10 20 20] /C [0 0 1] /QuadPoints [100 120 200 120 100 100 200 100] >>", + b"<< /Type /XObject /Subtype /Form /BBox [0 0 10 10] /Length 0 >>\nstream\n\nendstream", + ]); + let display = with_missing_appearances(&doc).expect("the markups need an appearance"); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + assert_eq!(rect(&shown[0]), [10.0, 10.0, 300.0, 380.0], "the box stops at the crop box"); + assert_eq!(ap_bbox(&display, &shown[0]), rect(&shown[0])); + assert_eq!(rect(&shown[1]), [10.0, 10.0, 20.0, 20.0], "quads off the page widen nothing"); + assert_eq!(rect(&shown[2]), [10.0, 10.0, 200.0, 120.0], "as many quads as the cap widen the box"); + assert_eq!(rect(&shown[3]), [10.0, 10.0, 20.0, 20.0], "more quads than the cap keep the /Rect"); + assert_eq!(ap_bbox(&display, &shown[3]), rect(&shown[3])); + assert_eq!(rect(&shown[4]), [10.0, 10.0, 20.0, 20.0], "a markup with an appearance of its own keeps its /Rect"); + assert_eq!( + shown[4].get(b"AP").map(|a| display.resolve(a)).and_then(|a| a.as_dict().and_then(|a| a.reference(b"N"))), + Some(ObjRef::new(10, 0)), + "and its appearance" + ); + assert_eq!(rect(&shown[5]), [10.0, 10.0, 20.0, 20.0], "a square is not text markup"); + assert_eq!(ap_bbox(&display, &shown[5]), rect(&shown[5])); +} + +/// A restyle (the edit path, `set_appearance`) redraws a text markup over its quadrilaterals and +/// saves the `/Rect` its appearance covers, as far as the page shows, since the appearance is +/// mapped onto `/Rect` (ISO 32000-2 §12.5.5). A markup whose `/Rect` already holds its quads keeps +/// that `/Rect` as written, swapped corners and all. +#[test] +fn restyled_text_markup_saves_the_rect_its_appearance_covers() { + let mut doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R 5 0 R 6 0 R] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [335.1 370.83 324.6 471.78] /C [1 1 0] /QuadPoints [370 287 471 287 370 277 471 277] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [150 70 50 50] /C [1 1 0] /QuadPoints [60 68 140 68 60 52 140 52] >>", + b"<< /Type /Annot /Subtype /Underline /Rect [10 10 20 20] /C [0 0 1] /QuadPoints [500 120 700 120 500 100 700 100] >>", + ]); + for index in 0..3 { + set_style(&mut doc, 0, index, StyleChange::new(Some([1.0, 0.0, 0.0]), None, None, None), &meta("")).unwrap(); + } + let saved = reopen(&doc); + let shown = list(&saved, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + assert_eq!(rect(&shown[0]), [324.6, 277.0, 471.0, 471.78], "the saved /Rect takes in the highlighted text"); + assert_eq!(ap_bbox(&saved, &shown[0]), rect(&shown[0])); + assert!(ap_content(&saved, &shown[0]).contains("370 287 m 471 287 l 471 277 l 370 277 l h f"), "the quad is filled"); + assert_eq!(rect(&shown[1]), [150.0, 70.0, 50.0, 50.0], "a /Rect that holds the quads is kept as written"); + assert_eq!(rect(&shown[2]), [10.0, 10.0, 612.0, 120.7], "the saved /Rect stops at the page"); + assert_eq!(ap_bbox(&saved, &shown[2]), rect(&shown[2])); +} + +/// A one-page document whose page lists `annots` (objects 4 on) and the display copy +/// `with_missing_appearances` draws for it. +fn displayed(annots: &[&[u8]]) -> (Document, Document) { + let kids: Vec = (0..annots.len()).map(|i| format!("{} 0 R", i + 4)).collect(); + let page = format!("<< /Type /Page /Parent 2 0 R /Annots [{}] >>", kids.join(" ")); + let mut objs: Vec<&[u8]> = + vec![b"<< /Type /Catalog /Pages 2 0 R >>", b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", page.as_bytes()]; + objs.extend_from_slice(annots); + let doc = document_of(&objs); + let display = with_missing_appearances(&doc).expect("the annotations need an appearance"); + (doc, display) +} + +/// Ink strokes (`/InkList`, ISO 32000-2 Table 185) outside an ink annotation's `/Rect` are drawn: +/// the display copy's `/Rect` and `/BBox` take in the strokes and half their width. Shaped like +/// the pencil marks of `issue13447.pdf`, whose `/Rect` has x and y swapped. An ink annotation +/// whose `/Rect` holds its points keeps it, even where its wide stroke reaches past it. +#[test] +fn ink_whose_strokes_reach_past_its_rect_is_drawn_through_them() { + let (doc, display) = displayed(&[ + b"<< /Type /Annot /Subtype /Ink /Rect [109.25 260.01 21.25 313.51] /BS << /W 3 >> /C [1 0 0] /InkList [[260 502.75 300 537 330 520] [280 510]] >>", + b"<< /Type /Annot /Subtype /Ink /Rect [0 0 100 100] /BS << /W 30 >> /C [1 0 0] /InkList [[10 10 90 90]] >>", + ]); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + // The smooth curve rises past its highest point (537) to 537.33; half the width (1.5) more. + assert_rect_near(&rect(&shown[0]), [21.25, 260.01, 331.5, 538.825], "the ink's /Rect takes in its strokes"); + assert_eq!(ap_bbox(&display, &shown[0]), rect(&shown[0])); + assert!(ap_content(&display, &shown[0]).contains("260 502.75 m"), "the stroke is drawn"); + assert_eq!(rect(&shown[1]), [0.0, 0.0, 100.0, 100.0], "a /Rect that holds the points stays as it is"); + assert_eq!(rect(&list(&doc, 0)[0]), [109.25, 260.01, 21.25, 313.51], "the source document is unchanged"); +} + +/// A polyline's `/Vertices` (Table 181) outside its `/Rect` are drawn, with its line endings +/// (`/LE`): the box takes in the arrow at the last vertex. +#[test] +fn polyline_whose_vertices_reach_past_its_rect_is_drawn_through_them() { + let (_, display) = displayed(&[ + b"<< /Type /Annot /Subtype /PolyLine /Rect [667.5 250.69 567.75 344.44] /BS << /W 2 >> /C [0 0 1] /Vertices [250 44 350 94 285 138] >>", + b"<< /Type /Annot /Subtype /PolyLine /Rect [10 10 20 20] /BS << /W 2 >> /C [0 0 1] /LE [/None /OpenArrow] /Vertices [100 100 200 100 200 200] >>", + ]); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + assert_eq!(rect(&shown[0]), [249.0, 43.0, 667.5, 344.44], "the polyline's /Rect takes in its vertices"); + assert_eq!(ap_bbox(&display, &shown[0]), rect(&shown[0])); + assert!(ap_content(&display, &shown[0]).contains("250 44 m"), "the polyline is drawn"); + // Half the width (1) and the arrow (6 + 3 × 2) past the vertices. + assert_eq!(rect(&shown[1]), [10.0, 10.0, 213.0, 213.0], "and its arrow"); +} + +/// A polygon's `/Vertices` outside its `/Rect` are drawn, its cloudy border (§12.5.4) included. +#[test] +fn polygon_whose_vertices_reach_past_its_rect_is_drawn_through_them() { + let (_, display) = displayed(&[ + b"<< /Type /Annot /Subtype /Polygon /Rect [10 10 20 20] /BS << /W 1 >> /C [0 0 1] /IC [1 1 0] /Vertices [100 100 200 100 150 180] >>", + b"<< /Type /Annot /Subtype /Polygon /Rect [10 10 20 20] /BS << /W 2 >> /C [0 0 1] /BE << /S /C >> /Vertices [100 100 200 100 150 180] >>", + ]); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + assert_eq!(rect(&shown[0]), [10.0, 10.0, 200.5, 180.5], "the polygon's /Rect takes in its vertices"); + assert_eq!(ap_bbox(&display, &shown[0]), rect(&shown[0])); + assert!(ap_content(&display, &shown[0]).contains("100 100 m"), "the polygon is drawn"); + // Half the width (1) and twice the cloud radius (2 × (6 + 1.5 × 2)) past the vertices. + assert_eq!(rect(&shown[1]), [10.0, 10.0, 219.0, 199.0], "and its cloudy border"); +} + +/// A line's `/L` (Table 178) outside its `/Rect` is drawn, with its line ending and its leader +/// lines. Shaped like the arrows of `issue13447.pdf`. +#[test] +fn line_whose_points_reach_past_its_rect_is_drawn_through_them() { + let (_, display) = displayed(&[ + b"<< /Type /Annot /Subtype /Line /Rect [598.31 146.63 537 316.13] /BS << /W 6 >> /C [0 0 1] /LE [/None /ClosedArrow] /L [176 45 178 154] >>", + b"<< /Type /Annot /Subtype /Line /Rect [10 10 20 20] /BS << /W 2 >> /C [0 0 1] /L [100 100 200 100] /LL 20 /LLE 5 >>", + b"<< /Type /Annot /Subtype /Line /Rect [90 90 210 110] /BS << /W 2 >> /C [0 0 1] /LE [/OpenArrow /OpenArrow] /L [100 100 200 100] >>", + ]); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + // Half the width (3) and the arrow (6 + 3 × 6) past `/L`. + assert_eq!(rect(&shown[0]), [149.0, 18.0, 598.31, 316.13], "the line's /Rect takes in /L and its arrow"); + assert_eq!(ap_bbox(&display, &shown[0]), rect(&shown[0])); + assert!(ap_content(&display, &shown[0]).contains("176 45 m"), "the line is drawn"); + // The leaders rise 20 + 5 above `/L`; half the width (1) past them. + assert_eq!(rect(&shown[1]), [10.0, 10.0, 201.0, 126.0], "and its leader lines"); + assert_eq!(rect(&shown[2]), [90.0, 90.0, 210.0, 110.0], "a /Rect that holds /L stays as it is"); +} + +/// Points past the caps (`appearance::MAX_REACH_POINTS`) leave `/Rect` as it is, as do points +/// that aren't numbers; a restyle (the edit path) saves the widened `/Rect` of an ink annotation. +#[test] +fn path_reach_stops_at_the_point_cap_and_is_saved_on_restyle() { + let cap = appearance::MAX_REACH_POINTS; + let stroke = |n: usize| format!("[{}]", vec!["100 120"; n].join(" ")); + let at_cap = format!("<< /Type /Annot /Subtype /Ink /Rect [10 10 20 20] /BS << /W 0 >> /C [1 0 0] /InkList [{} [200 200]] >>", stroke(cap - 1)); + let past_cap = format!("<< /Type /Annot /Subtype /Ink /Rect [10 10 20 20] /BS << /W 0 >> /C [1 0 0] /InkList [{} [200 200]] >>", stroke(cap)); + let (_, display) = displayed(&[at_cap.as_bytes(), past_cap.as_bytes()]); + let shown = list(&display, 0); + let rect = |d: &Dict| nums_of(d, b"Rect"); + assert_eq!(rect(&shown[0]), [10.0, 10.0, 200.0, 200.0], "as many points as the cap widen the box"); + assert_eq!(rect(&shown[1]), [10.0, 10.0, 20.0, 20.0], "more points than the cap keep the /Rect"); + + let mut doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Ink /Rect [109.25 260.01 21.25 313.51] /BS << /W 3 >> /C [1 0 0] /InkList [[260 502.75 300 537 330 520]] >>", + ]); + set_style(&mut doc, 0, 0, StyleChange::new(Some([0.0, 0.0, 1.0]), None, None, None), &meta("")).unwrap(); + let saved = reopen(&doc); + let shown = list(&saved, 0); + assert_rect_near(&rect(&shown[0]), [21.25, 260.01, 331.5, 538.825], "the saved /Rect takes in the strokes"); + assert_eq!(ap_bbox(&saved, &shown[0]), rect(&shown[0])); +} + +/// A squiggle's peaks stand 1.5 above a quad's bottom edge however low the quad is, so over a quad +/// 1 high they reach past it: the widened box takes them in, with half the line's width. +#[test] +fn squiggly_peaks_over_a_low_quad_are_inside_the_widened_box() { + let (_, display) = + displayed(&[b"<< /Type /Annot /Subtype /Squiggly /Rect [10 10 20 20] /C [0 1 0] /QuadPoints [100 301 200 301 100 300 200 300] >>"]); + let shown = list(&display, 0); + // Peak 300 + 1.5, half the 0.5 line width: 301.75; the sides reach 0.75 past the quad. + assert_eq!(nums_of(&shown[0], b"Rect"), [10.0, 10.0, 200.75, 301.75]); + assert_eq!(ap_bbox(&display, &shown[0]), nums_of(&shown[0], b"Rect")); +} + +/// A line's caption (`/Cap`, ISO 32000-2 Table 178) is drawn beside `/L`: the widened box takes +/// in the caption's text as well as the line. +#[test] +fn a_line_caption_is_inside_the_widened_box() { + let (_, display) = displayed(&[ + b"<< /Type /Annot /Subtype /Line /Rect [10 10 20 20] /BS << /W 2 >> /C [0 0 1] /L [100 100 200 100] /Cap true /CP /Top /Contents (Caption) >>", + ]); + let shown = list(&display, 0); + let rect = nums_of(&shown[0], b"Rect"); + // The caption sits on top of the line: its 10-point text rises well past the stroke's 101. + assert!(rect[3] > 108.0, "{rect:?}"); + assert_eq!(ap_bbox(&display, &shown[0]), rect); +} + +/// A line just above the crop box whose wide stroke reaches down onto the page is drawn as far as +/// the page shows, not cut to its `/Rect`. +#[test] +fn a_stroke_that_reaches_onto_the_page_from_points_off_it_widens_the_box() { + let doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /CropBox [0 0 300 400] /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Line /Rect [10 10 20 20] /BS << /W 6 >> /C [0 0 1] /L [100 401 200 401] >>", + ]); + let display = with_missing_appearances(&doc).expect("the line needs an appearance"); + let shown = list(&display, 0); + // The stroke spans 398 to 404 across and 97 to 203 along; the page shows up to 400. + assert_eq!(nums_of(&shown[0], b"Rect"), [10.0, 10.0, 203.0, 400.0]); +} + +/// A cached display appearance was drawn for its page's crop box: once the crop box grows, the +/// next refresh draws the annotation again for the larger region. +#[test] +fn a_cached_appearance_is_redrawn_for_a_changed_crop_box() { + let mut doc = document_of(&[ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>", + b"<< /Type /Page /Parent 2 0 R /CropBox [0 0 300 400] /Annots [4 0 R] >>", + b"<< /Type /Annot /Subtype /Highlight /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [100 380 500 380 100 360 500 360] >>", + ]); + let mut cache = AppearanceCache::default(); + let display = with_missing_appearances_cached(&doc, &mut cache).expect("drawn"); + assert_eq!(nums_of(&list(&display, 0)[0], b"Rect"), [10.0, 10.0, 300.0, 380.0]); + doc.update_dict(ObjRef::new(3, 0), |d| d.set(b"CropBox".to_vec(), Object::Array([0, 0, 612, 792].map(Object::Int).to_vec()))).unwrap(); + let display = with_missing_appearances_cached(&doc, &mut cache).expect("drawn"); + let shown = list(&display, 0); + assert_eq!(nums_of(&shown[0], b"Rect"), [10.0, 10.0, 500.0, 380.0], "drawn for the larger crop box"); + assert_eq!(ap_bbox(&display, &shown[0]), nums_of(&shown[0], b"Rect")); +} + +/// Redrawing a comment drawn through its points finds its page through `/P` when that page +/// lists it, reading that one page's `/Annots` rather than every page's: here the last of 300. +#[test] +fn restyling_a_comment_drawn_through_its_points_reads_only_its_page() { + let pages = 300; + let mut objs: Vec = vec!["<< /Type /Catalog /Pages 2 0 R >>".into()]; + let kids: Vec = (0..pages).map(|i| format!("{} 0 R", 3 + 2 * i)).collect(); + objs.push(format!("<< /Type /Pages /Kids [{}] /Count {pages} /MediaBox [0 0 612 792] >>", kids.join(" "))); + for i in 0..pages { + let (page, annot) = (3 + 2 * i, 4 + 2 * i); + objs.push(format!("<< /Type /Page /Parent 2 0 R /Annots [{annot} 0 R] >>")); + objs.push(format!( + "<< /Type /Annot /Subtype /Highlight /P {page} 0 R /Rect [10 10 20 20] /C [1 1 0] /QuadPoints [100 120 200 120 100 100 200 100] >>" + )); + } + let bytes: Vec<&[u8]> = objs.iter().map(|o| o.as_bytes()).collect(); + let mut doc = document_of(&bytes); + let last = pages - 1; + ANNOTS_ENTRIES_COPIED.with(|c| c.set(0)); + set_style(&mut doc, last, 0, StyleChange::new(Some([1.0, 0.0, 0.0]), None, None, None), &meta("")).unwrap(); + let copied = ANNOTS_ENTRIES_COPIED.with(std::cell::Cell::get); + assert!(copied < 20, "{copied} annotation entries copied for one restyle"); + assert_eq!(nums_of(&list(&doc, last)[0], b"Rect"), [10.0, 10.0, 200.0, 120.0], "drawn for its page"); +} + +/// `got` is `want` to within 0.001 in every coordinate. +fn assert_rect_near(got: &[f64], want: [f64; 4], what: &str) { + assert!(got.len() == 4 && got.iter().zip(want).all(|(g, w)| (g - w).abs() < 1e-3), "{what}: {got:?}"); +} + +fn nums_of(d: &Dict, key: &[u8]) -> Vec { + d.get(key).and_then(|a| a.as_array()).map(|a| a.iter().filter_map(|o| o.as_f64()).collect()).unwrap_or_default() } /// The `y` of every `1 0 0 1 x y Tm` in `content`: where each piece of text sits. diff --git a/crates/automation/src/bates.rs b/crates/automation/src/bates.rs index 6493df9968..71ae6b3b91 100644 --- a/crates/automation/src/bates.rs +++ b/crates/automation/src/bates.rs @@ -415,7 +415,11 @@ mod tests { let acl = Command::new("icacls.exe").arg(dir.path()).arg("/deny").arg(format!("*{sid}:(AD)")).status().unwrap(); assert!(acl.success()); - assert!(matches!(tempfile::tempdir_in(dir.path()), Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied)); + // Where the deny entry does not bind the test's own account (it did not on CI's Windows image), + // there is no denied folder to probe. + if !matches!(tempfile::tempdir_in(dir.path()), Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied) { + return; + } let direct_probe = dir.path().join("direct-probe"); let file = std::fs::OpenOptions::new().write(true).create_new(true).open(&direct_probe).unwrap(); drop(file); diff --git a/crates/automation/src/json.rs b/crates/automation/src/json.rs index c4ad98efa0..5231132421 100644 --- a/crates/automation/src/json.rs +++ b/crates/automation/src/json.rs @@ -296,7 +296,7 @@ pub(crate) fn info(d: &Document) -> Value { // Custom document properties (Document Properties ▸ Custom): /Info entries beyond the // ones the spec names, set and cleared with doc_set_info. "custom": i.custom.iter().map(|(name, value)| json!({ "name": name, "value": value })).collect::>(), - "creator": i.creator, "producer": i.producer, "created": i.created, "modified": i.modified, + "creator": i.creator, "producer": i.producer, "tagged": i.tagged, "has_javascript": i.has_javascript, // Where the document opens (/OpenAction, named destinations resolved): 1-based page and diff --git a/crates/automation/src/measure.rs b/crates/automation/src/measure.rs index ac3d2d95f1..932f85ad9a 100644 --- a/crates/automation/src/measure.rs +++ b/crates/automation/src/measure.rs @@ -147,7 +147,7 @@ impl Automation { Ok(json!({"point":point,"kind":s.kind,"distance":measure::distance(at,point)})) }) .transpose()?; - Ok(json!({"snap":snap,"truncated":geometry.truncated,"intersection_limited":intersection_limited,"segments":geometry.segments.len()})) + Ok(json!({"snap":snap,"truncated":geometry.truncated,"intersection_limited":intersection_limited,"segments":geometry.segment_count()})) } pub(crate) fn measurement_export(&self, a: &Args) -> Result { let all = self.doc(a)?.measurements().map_err(failed)?; diff --git a/crates/automation/src/pages.rs b/crates/automation/src/pages.rs index 9212b20ce6..04ab8dace4 100644 --- a/crates/automation/src/pages.rs +++ b/crates/automation/src/pages.rs @@ -410,6 +410,7 @@ fn render_warning_text(warning: pdfcraft_render::RenderWarning) -> &'static str match warning { RenderWarning::ContentTruncated => "content past the page's safety budget was skipped", RenderWarning::ColourFallback => "a colour that could not be computed was painted in a fallback colour", + RenderWarning::Type3WidthsMissing => "a Type 3 font has no /Widths; its glyphs were spaced by the widths their glyph procedures give", } } diff --git a/crates/automation/src/tools.rs b/crates/automation/src/tools.rs index 21ec09afee..0cade83abd 100644 --- a/crates/automation/src/tools.rs +++ b/crates/automation/src/tools.rs @@ -1361,7 +1361,7 @@ pub fn tools() -> Vec { t("print_readiness", "Print readiness", "Verification only. Reports effective image resolution where each image is drawn (not against the page size), page boxes and bleed past the trim, output intents, fonts that are not embedded, and the PDF/A-2b structural result. Does not modify the document, embed fonts, invent a press profile, or check PDF/X.") .ro() .with(schema(json!({ "doc": doc() }), &["doc"])), - t("pdfa_convert", "Save as PDF/A", "Standards ▸ Save as PDF/A: fix what can be fixed for PDF/A-2b or 3b (XMP identification and metadata, an sRGB output intent, forbidden actions and JavaScript, annotation print flags, image interpolation, encryption). Returns what was fixed and what remains (e.g. fonts that aren't embedded). Undoable; save the document to keep it.") + t("pdfa_convert", "Save as PDF/A", "Standards ▸ Save as PDF/A: fix what can be fixed for PDF/A-2b or 3b (XMP identification and metadata, an sRGB output intent, forbidden actions and JavaScript, annotation print flags, image interpolation, encryption, the file header, the file identifier). Returns what was fixed and what remains (e.g. fonts that aren't embedded). Undoable; save the document to keep it.") .cmd("standards.pdfa") .with(schema(json!({ "doc": doc(), "level": { "type": "string", "enum": ["2b", "3b"] } }), &["doc"])), t("standards_remove_id", "Remove PDF/A or PDF/X identification", "Remove what identifies the document as PDF/A (its pdfaid: XMP properties) or PDF/X (GTS_PDFXVersion and GTS_PDFXConformance, in the document information and as pdfxid: XMP properties), e.g. after edits it no longer meets. Output intents and other metadata stay. Fails when the document declares nothing of the kind. Undoable.") diff --git a/crates/cos/src/lib.rs b/crates/cos/src/lib.rs index 9e9b6a8764..7f7f32fedc 100644 --- a/crates/cos/src/lib.rs +++ b/crates/cos/src/lib.rs @@ -30,7 +30,7 @@ pub use parser::{Lexer, is_whitespace, parse_indirect}; pub use pdfcraft_crypt::{ Algorithm, Auth, Method as CryptMethod, NewEncryption, Permissions, SecurityHandler, effective_password, pdfdoc_byte, pdfdoc_char, }; -pub use writer::{SaveOptions, civil_from_days, pdf_date, reachable_objects, serialize, write_full, write_incremental}; +pub use writer::{SaveOptions, civil_from_days, new_file_id, pdf_date, reachable_objects, serialize, write_full, write_incremental}; #[derive(Debug, thiserror::Error, Clone, PartialEq)] pub enum CosError { diff --git a/crates/cos/src/writer.rs b/crates/cos/src/writer.rs index 0d6ccba7d2..56731f7e39 100644 --- a/crates/cos/src/writer.rs +++ b/crates/cos/src/writer.rs @@ -789,13 +789,21 @@ fn stamp_mod_date(doc: &mut Document, opts: &SaveOptions) { } } -/// Keep an existing `/ID`, or create one (two equal 16-byte strings, §14.4). +/// Keep any existing two-element `/ID` array as it is, even a malformed one such as `[<> <>]` or `[1 2]` (an ordinary +/// save preserves what it doesn't understand; Save as PDF/A replaces or completes it), or create one with +/// [`new_file_id`] when there is none or it isn't a two-element array. fn ensure_id(trailer: &mut Dict, opts: &SaveOptions, content: &[u8]) { if matches!(trailer.get(b"ID"), Some(Object::Array(a)) if a.len() == 2) { return; } + trailer.set(b"ID".to_vec(), new_file_id(content, opts.id_seed)); +} + +/// A new file identifier for a file whose bytes are `content` (ISO 32000-1 14.4): an array of +/// two equal 16-byte strings, the same for the same content and `seed`. +pub fn new_file_id(content: &[u8], seed: u64) -> Object { // FNV-1a over a sample of the content and the seed: stable, dependency-free. - let mut h: u64 = 0xcbf2_9ce4_8422_2325 ^ opts.id_seed; + let mut h: u64 = 0xcbf2_9ce4_8422_2325 ^ seed; for b in content.iter().step_by((content.len() / 4096).max(1)).chain(content.iter().rev().take(64)) { h ^= *b as u64; h = h.wrapping_mul(0x0100_0000_01b3); @@ -804,7 +812,7 @@ fn ensure_id(trailer: &mut Dict, opts: &SaveOptions, content: &[u8]) { id.extend_from_slice(&h.to_be_bytes()); id.extend_from_slice(&h.rotate_left(29).wrapping_mul(0x9E37_79B9_7F4A_7C15).to_be_bytes()); let s = Object::String(PdfString { bytes: id, hex: true }); - trailer.set(b"ID".to_vec(), Object::Array(vec![s.clone(), s])); + Object::Array(vec![s.clone(), s]) } /// Give a rewritten file a new identifier (§14.4), so that a redacted or sanitized copy does not diff --git a/crates/edit/src/text.rs b/crates/edit/src/text.rs index a3ee880a64..2dbf302685 100644 --- a/crates/edit/src/text.rs +++ b/crates/edit/src/text.rs @@ -25,7 +25,7 @@ mod regions; #[path = "text_session.rs"] mod session; pub use fallback::strip_subset_tag; -use regions::{alignment_frame, group_blocks, is_centred, rewrite_styled_block}; +use regions::{alignment_frame, group_blocks, is_centred, rewrite_changed_words, rewrite_styled_block}; pub use session::TextSession; mod blocks; @@ -42,7 +42,7 @@ pub use blocks::{ BlockKey, BlockStyle, RangeStyle, ReadingBlocks, TextBlock, block_overlapping, clear_block, reading_blocks, reading_blocks_checked, replace_block, text_blocks, }; -use blocks::{paragraph_separator, row_break}; +use blocks::{kept_words, paragraph_separator, row_break}; pub use editor::{TextEditLayout, TextEditorFont, register_text_font, text_edit_layout, text_editor_fonts}; use editor::{builtin_thai_choice, layout_draft}; #[cfg(test)] @@ -55,12 +55,12 @@ use fallback::{ use fallback::{add_type3_font, is_type3_fallback}; pub(crate) use fallback::{glyph_path, type3_font_dict, unique_font_name}; use interp::{Origin, Paint, union}; -use lines::page_dict; pub(crate) use lines::{ ContentStream, FormBudget, Read, content_streams, content_streams_within, dict_in_place, drawn_before_damage, first_op_after_damage, form_call, joined, resource_dict, rewritten_contents, splice, }; pub use lines::{MAX_FORM_DEPTH, MAX_FORM_VISITS, glyph_text_lines, text_lines}; +use lines::{page_dict, word_lines}; pub use rewrite::{LineEdit, replace_line, rewrite_block}; use rewrite::{coincident_ops, rewrite_block_from}; pub(crate) use rewrite::{paragraph_ops, reveal_ops, selectable_blocks, show_starts, translate_blocks}; diff --git a/crates/edit/src/text/blocks.rs b/crates/edit/src/text/blocks.rs index f2da128071..b714bcd592 100644 --- a/crates/edit/src/text/blocks.rs +++ b/crates/edit/src/text/blocks.rs @@ -44,6 +44,12 @@ impl TextBlock { (self.invisible && self.size.is_finite() && self.size > 0.0).then_some(self.size) } + /// Whether `draft` says what the paragraph says, give or take space around it. Spacing inside + /// it has to match: a typed second space is a change. + pub fn same_words(&self, draft: &str) -> bool { + draft.trim() == self.editor_text.trim() + } + /// Preserve styles outside the changed ranges; inserted characters inherit the surrounding /// source run. The texts are aligned word by word, so words left untouched between two edits /// keep their own runs (#2198); past [`MAX_CHANGED_WORDS`] the draft counts as one change. @@ -163,6 +169,23 @@ fn edit_script(a: &[T], b: &[T], max: usize) -> Option> None } +/// For each of `old`'s words, the index in `new` of the word it stays as in the shortest edit from +/// one to the other (`None` when it goes), or `None` overall past [`MAX_CHANGED_WORDS`] changes. +pub(crate) fn kept_words(old: &[T], new: &[T]) -> Option>> { + let mut kept = vec![None; old.len()]; + let (mut o, mut n) = (0usize, 0usize); + for step in edit_script(old, new, MAX_CHANGED_WORDS)? { + if step == Step::Keep + && let Some(slot) = kept.get_mut(o) + { + *slot = Some(n); + } + o = o.saturating_add(usize::from(step != Step::Insert)); + n = n.saturating_add(usize::from(step != Step::Delete)); + } + Some(kept) +} + /// Walk Myers' saved frontiers back from the end to the edit script. fn backtrack(trace: &[Vec], n: isize, m: isize, max: isize) -> Vec { let at = |v: &[isize], k: isize| usize::try_from(k + max + 1).ok().and_then(|i| v.get(i).copied()).unwrap_or(0); @@ -444,6 +467,13 @@ pub struct BlockStyle { /// For an invisible paragraph ([`TextBlock::invisible`]): fill its old area with this colour /// (RGB 0–1) before the new text, hiding the scanned word beneath it. pub cover: Option<[f64; 3]>, + /// For an invisible paragraph whose formatting nobody asked to change: cover and redraw only + /// the words the draft changes, and leave the rest as the scan shows them (#3313). The session + /// sets this; a caller that sets it should also give `cover`. + pub changed_words_only: bool, + /// How far across [`BlockStyle::cover`] may reach, whatever its padding: the edges of the words + /// on either side that stay (user space). + pub cover_between: Option<[f64; 2]>, } #[derive(Clone, Debug, PartialEq)] diff --git a/crates/edit/src/text/lines.rs b/crates/edit/src/text/lines.rs index 1aac66fde4..2755d6bb71 100644 --- a/crates/edit/src/text/lines.rs +++ b/crates/edit/src/text/lines.rs @@ -291,7 +291,13 @@ fn fill_color(fill: &[Op]) -> [f64; 3] { /// The lines of text on a page (0-based), in content order. pub fn text_lines(doc: &Document, page: usize) -> Result, EditError> { - text_lines_in(doc, &page_dict(doc, page)?, true) + text_lines_in(doc, &page_dict(doc, page)?, true, false) +} + +/// [`text_lines`] with every show of invisible text a line of its own, however alike its +/// neighbours: an OCR layer's words, which an edit can keep or change one by one. +pub(crate) fn word_lines(doc: &Document, page: usize) -> Result, EditError> { + text_lines_in(doc, &page_dict(doc, page)?, true, true) } /// The lines `page`'s glyphs show, in content order, without `/ActualText` replacements: what the @@ -299,11 +305,12 @@ pub fn text_lines(doc: &Document, page: usize) -> Result, EditErro /// not trust that metadata (the redaction audit). Takes the page itself, so a caller going /// through every page doesn't walk the page tree again for each. pub fn glyph_text_lines(doc: &Document, page: &pdfcraft_model::Page) -> Result, EditError> { - text_lines_in(doc, page, false) + text_lines_in(doc, page, false, false) } -/// [`text_lines`] of `p`; with `actual_text`, `/ActualText` replaces the glyphs it spans. -fn text_lines_in(doc: &Document, p: &pdfcraft_model::Page, actual_text: bool) -> Result, EditError> { +/// [`text_lines`] of `p`; with `actual_text`, `/ActualText` replaces the glyphs it spans, and with +/// `invisible_apart`, no show of invisible text joins the one before it. +fn text_lines_in(doc: &Document, p: &pdfcraft_model::Page, actual_text: bool, invisible_apart: bool) -> Result, EditError> { let res = p.dict.get(b"Resources").map(|r| doc.resolve(r)).and_then(|r| r.as_dict().cloned()).unwrap_or_default(); let mut cache = HashMap::new(); let mut lines: Vec = Vec::new(); @@ -345,7 +352,7 @@ fn text_lines_in(doc: &Document, p: &pdfcraft_model::Page, actual_text: bool) -> && near // An OCR layer is one text object over the whole scan: its words continue a line // only across a word space, not across a table cell or a tab stop. - && (s.state.render_mode != 3 || gap < pen.size * 0.6) + && (s.state.render_mode != 3 || (gap < pen.size * 0.6 && !invisible_apart)) }); if joins && let Some(l) = lines.last_mut() { let gap = last.map_or(0.0, |(_, pen)| pen.continues(&s, 0.0).1); diff --git a/crates/edit/src/text/rewrite.rs b/crates/edit/src/text/rewrite.rs index 692c5e68de..d9cf616d4b 100644 --- a/crates/edit/src/text/rewrite.rs +++ b/crates/edit/src/text/rewrite.rs @@ -7,7 +7,7 @@ use super::interp::{Carry, Origin, TextState, interpret}; use super::lines::{content_streams, joined, page_dict, rewritten_contents, splice, text_lines}; use super::{ Dict, Document, EditError, HashMap, Matrix, Metrics, Object, Op, PdfString, TextLine, TextSession, alignment_frame, group_blocks, is_centred, - parse, rewrite_styled_block, + parse, rewrite_changed_words, rewrite_styled_block, }; /// What replacing a line did. @@ -851,6 +851,9 @@ pub(crate) fn rewrite_block_from( if b.invisible { crate::check_on_top(doc, &p)?; } + if let Some(edit) = text.and_then(|draft| rewrite_changed_words(doc, page, &b, &lines, draft, style, max_glyphs)) { + return Ok(edit); + } let mut res = p.dict.get(b"Resources").map(|r| doc.resolve(r)).and_then(|r| r.as_dict().cloned()).unwrap_or_default(); let mut fonts_res = res.get(b"Font").map(|f| doc.resolve(f)).and_then(|f| f.as_dict().cloned()).unwrap_or_default(); let dispatch = choose_rewrite_path(doc, &fonts_res, &b, &members, first, text, style)?; @@ -1018,6 +1021,14 @@ pub(crate) fn rewrite_block_from( /// rewritten invisible paragraph, drawn in the space whose CTM is `ctm`. Nothing for visible text. /// The paragraph's `q`/`Q` restores the rendering mode for the invisible text after it; one /// painted over where it was is drawn on top of the page instead ([`crate::draw_on_top`]). +/// What covering the word in `rect` paints: scanned ink reaches a little past the recognised box, +/// but not past `between`, the words on either side that stay. It always covers the word itself. +pub(crate) fn cover_box(rect: [f64; 4], between: Option<[f64; 2]>) -> [f64; 4] { + let pad = ((rect[3] - rect[1]).abs() * 0.1).clamp(0.5, 2.0); + let [left, right] = between.unwrap_or([f64::NEG_INFINITY, f64::INFINITY]); + [(rect[0] - pad).max(left.min(rect[0])), rect[1] - pad, (rect[2] + pad).min(right.max(rect[2])), rect[3] + pad] +} + pub(crate) fn reveal_ops(block: &TextBlock, lines: &[TextLine], style: &BlockStyle, ctm: [f64; 6]) -> Result, EditError> { if !block.invisible { return Ok(Vec::new()); @@ -1031,11 +1042,8 @@ pub(crate) fn reveal_ops(block: &TextBlock, lines: &[TextLine], style: &BlockSty let back = Matrix(ctm).invert().ok_or_else(|| EditError::Invalid("the paragraph is drawn in a space it can't be covered in".into()))?; ops.push(Op::new("q", vec![])); ops.push(Op::new("rg", colour.map(n).to_vec())); - for rect in block.lines.iter().filter_map(|i| lines.get(*i)).map(|l| l.rect) { - // Scanned ink reaches a little past the recognised box. - let pad = ((rect[3] - rect[1]).abs() * 0.1).clamp(0.5, 2.0); - let corners = - [[rect[0] - pad, rect[1] - pad], [rect[2] + pad, rect[1] - pad], [rect[2] + pad, rect[3] + pad], [rect[0] - pad, rect[3] + pad]]; + for rect in block.lines.iter().filter_map(|i| lines.get(*i)).map(|l| cover_box(l.rect, style.cover_between)) { + let corners = [[rect[0], rect[1]], [rect[2], rect[1]], [rect[2], rect[3]], [rect[0], rect[3]]]; for (i, [x, y]) in corners.into_iter().enumerate() { let (x, y) = back.apply(x, y); if !(x.is_finite() && y.is_finite()) { diff --git a/crates/edit/src/text_regions.rs b/crates/edit/src/text_regions.rs index ab9a055429..ee808a8dec 100644 --- a/crates/edit/src/text_regions.rs +++ b/crates/edit/src/text_regions.rs @@ -3,15 +3,18 @@ use super::{ BlockStyle, CraftFont, Dict, Document, EditError, HashMap, LineEdit, MAX_TYPE3_GLYPHS, Matrix, Metrics, Object, Op, PdfString, Pieces, RangeStyle, TextBlock, TextLine, Type3Glyph, builtin_thai_choice, coincident_ops, content_streams, fallback_face, japanese_glyph_from, - needs_type3, no_fallback_font, no_glyph_for, page_dict, paragraph_separator, reveal_ops, rewritten_contents, row_break, same_base_font, - same_font, source_encodes, source_family, source_program, source_show, source_space, source_width, splice, strip_subset_tag, type3_faces, - type3_glyph, union, write_type3_glyphs, + kept_words, needs_type3, no_fallback_font, no_glyph_for, page_dict, paragraph_separator, reveal_ops, rewritten_contents, row_break, + same_base_font, same_font, source_encodes, source_family, source_program, source_show, source_space, source_width, splice, strip_subset_tag, + type3_faces, type3_glyph, union, word_lines, write_type3_glyphs, }; #[path = "text_regions/grouping.rs"] mod grouping; +#[path = "text_regions/ocr_words.rs"] +mod ocr_words; #[path = "text_regions/styled.rs"] mod styled; pub(super) use grouping::{alignment_frame, group_blocks, group_reading_blocks, is_centred}; +pub(super) use ocr_words::rewrite_changed_words; pub(super) use styled::rewrite_styled_block; diff --git a/crates/edit/src/text_regions/ocr_words.rs b/crates/edit/src/text_regions/ocr_words.rs new file mode 100644 index 0000000000..d511ccdf47 --- /dev/null +++ b/crates/edit/src/text_regions/ocr_words.rs @@ -0,0 +1,436 @@ +//! Edit text on a scanned page, word by word: a text edit of an OCR phrase covers and redraws only +//! the words it changes, and the rest stay as the scan shows them (#3313). Each changed stretch is +//! rewritten as a phrase of its own, which is how a phrase beside others in one OCR layer already +//! rewrites. A deleted word leaves its gap: the words after it are pixels, and don't move. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::ops::Range; + +use super::grouping::row_rect; +use super::styled::rewrite_styled_block; +use super::{BlockStyle, Document, EditError, LineEdit, TextBlock, TextLine, kept_words, word_lines}; + +/// How many times a replacement may be refitted to the room it has before the whole phrase is +/// redrawn instead. Each try rewrites a copy of the document. +const MAX_TRIES: usize = 16; + +/// The narrowest a replacement is squeezed to stay in its word's place. Most replacements are +/// only a little wider than the word they replace, and taking over the next word pushes that one +/// along too, and so on down the line. Past this it would be hard to read, so it takes the word over. +const MIN_SQUEEZE: f64 = 0.75; + +/// A stretch of the phrase's word runs that changes, the text that replaces it (empty when it is +/// deleted), and how much that text is squeezed across to fit. +#[derive(Debug, PartialEq)] +struct Change { + runs: Range, + /// Which of the draft's words replace them. + words: Range, + text: String, + squeeze: Option, +} + +/// The stretches `new` changes, given how many words each run holds and which word of `new` each +/// old word stays as ([`kept_words`]). A run in `redrawn` counts as changed, and new words with no +/// run of their own to replace add the run after them to it (or the last one, at the end). +fn changes(runs: &[usize], kept_as: &[Option], new: &[&str], redrawn: &mut BTreeSet) -> Vec { + // Each pass either returns or adds a run to `redrawn`, so this ends. + 'plan: loop { + let mut out = Vec::new(); + let (mut first_run, mut first_new, mut token) = (0usize, 0usize, 0usize); + for (run, count) in runs.iter().enumerate() { + let words = kept_as.get(token..token.saturating_add(*count)).unwrap_or_default(); + token = token.saturating_add(*count); + let at = words.first().copied().flatten(); + let kept = !redrawn.contains(&run) && !words.is_empty() && words.iter().enumerate().all(|(i, w)| *w == at.map(|a| a.saturating_add(i))); + let (true, Some(at)) = (kept, at) else { continue }; + if first_run == run && first_new < at { + redrawn.insert(run); + continue 'plan; + } + if first_run < run { + out.push(Change { + runs: first_run..run, + words: first_new..at, + text: new.get(first_new..at).unwrap_or_default().join(" "), + squeeze: None, + }); + } + first_run = run.saturating_add(1); + first_new = at.saturating_add(*count); + } + if first_run < runs.len() { + out.push(Change { + runs: first_run..runs.len(), + words: first_new..new.len(), + text: new.get(first_new..).unwrap_or_default().join(" "), + squeeze: None, + }); + } else if first_new < new.len() + && let Some(last) = runs.len().checked_sub(1) + { + redrawn.insert(last); + continue 'plan; + } + return out; + } +} + +/// Each word of `text` as a byte range. +fn word_ranges(text: &str) -> Vec> { + let mut out = Vec::new(); + let mut start = None; + for (at, c) in text.char_indices() { + match (c.is_whitespace(), start) { + (true, Some(from)) => { + out.push(from..at); + start = None; + } + (false, None) => start = Some(at), + _ => {} + } + } + if let Some(from) = start { + out.push(from..text.len()); + } + out +} + +/// Whether `plan` can stand for the spacing in `draft`, whose words are `words`. Spaces beside a +/// change are nothing to draw: Backspacing over a word leaves one either side of where it was, and +/// the words around it stay where the scan has them. Anything else that isn't one space has to be +/// drawn, inside the new words or between two words nothing else changed, which only the whole +/// rewrite can do. +fn spacing_fits(draft: &str, words: &[Range], plan: &[Change]) -> bool { + words.windows(2).enumerate().all(|(index, pair)| { + let [before, after] = pair else { return true }; + let gap = draft.get(before.end..after.start).unwrap_or(" "); + let boundary = index.saturating_add(1); + gap == " " || (gap.chars().all(|c| c == ' ') && plan.iter().any(|change| change.words.start == boundary || change.words.end == boundary)) + }) +} + +/// The phrase's stretch `indices` of `lines`, as a paragraph of its own. It keeps the whole +/// phrase's size, so what is redrawn is set alike wherever in the phrase it is. +fn stretch(phrase: &TextBlock, lines: &[TextLine], indices: &[usize]) -> Result { + let first = indices.first().and_then(|i| lines.get(*i)).ok_or_else(|| EditError::Invalid("the phrase has no such words".into()))?; + let rect = row_rect(indices, lines).ok_or_else(|| EditError::Invalid("the phrase has no such words".into()))?; + let mut text = String::new(); + let mut spans = Vec::with_capacity(indices.len()); + for index in indices { + let Some(line) = lines.get(*index) else { continue }; + if !text.is_empty() && !text.ends_with(char::is_whitespace) && !line.text.starts_with(char::is_whitespace) { + text.push(' '); + } + let start = text.len(); + text.push_str(&line.text); + spans.push((start..text.len(), *index)); + } + Ok(TextBlock { + reflow_text: text.clone(), + editor_text: text.clone(), + text, + spans, + rect, + base_font: first.base_font.clone(), + size: phrase.size, + bold: first.bold, + italic: first.italic, + color: first.color, + rotation: first.rotation(), + lines: indices.to_vec(), + rows: vec![indices.to_vec()], + invisible: true, + }) +} + +fn key(line: &TextLine) -> (&str, [i64; 4]) { + // Saturating casts: a non-finite edge only makes the comparison below fail, safely. + (line.text.as_str(), line.rect.map(|v| (v * 100.0).round() as i64)) +} + +/// How far right a rewrite of the runs `indices` drew (`None` when it drew nothing), or an error +/// when it moved or lost any other text on the page: text positioned after the end of the text +/// before it, say, depends on the words that went. +fn drawn_right(before: &[TextLine], indices: &[usize], after: &[TextLine]) -> Result, EditError> { + let mut unmatched: HashMap<_, usize> = HashMap::new(); + for line in before { + let count = unmatched.entry(key(line)).or_default(); + *count = count.saturating_add(1); + } + let mut drawn = Vec::new(); + for line in after { + match unmatched.get_mut(&key(line)).filter(|count| **count > 0) { + Some(count) => *count -= 1, + None => drawn.push(line), + } + } + // Visible text beside what was drawn reads as one line with it. It is still where it was. + let joined = |line: &TextLine| { + let [x0, y0, x1, y1] = line.rect; + drawn.iter().any(|d| { + d.text.contains(line.text.trim()) && d.rect[0] <= x0 + 0.01 && d.rect[1] <= y0 + 0.01 && d.rect[2] >= x1 - 0.01 && d.rect[3] >= y1 - 0.01 + }) + }; + let moved = before + .iter() + .enumerate() + .any(|(index, line)| !indices.contains(&index) && unmatched.get(&key(line)).is_some_and(|c| *c > 0) && !joined(line)); + if moved { + return Err(EditError::Invalid("rewriting these words alone would move the text around them".into())); + } + Ok(drawn.iter().map(|line| line.rect[2]).reduce(f64::max)) +} + +/// Rewrite each change of `plan` in `doc`, last first: a rewrite goes where its words were or at +/// the end of the page, so the words before it keep their indexes (an error if they don't). Gives +/// how far right each drew. +fn apply( + doc: &mut Document, + page: usize, + phrase: &TextBlock, + members: &[usize], + plan: &[Change], + style: &BlockStyle, + max_glyphs: usize, +) -> Result<(Vec>, Option), EditError> { + // The stretch starts where its words did. Its own box would centre a word that happens to + // sit in the middle of the page. + let mut style = BlockStyle { align: Some(crate::added::Align::Left), changed_words_only: false, ..style.clone() }; + let mut rights = vec![None; plan.len()]; + let mut substituted = None; + let original = word_lines(doc, page)?; + for (at, change) in plan.iter().enumerate().rev() { + let indices = members.get(change.runs.clone()).ok_or_else(|| EditError::Invalid("the phrase has no such words".into()))?; + let before = word_lines(doc, page)?; + // The words up to the next that stays must still be where they were. A rewrite also takes + // away copies of its words, and a copy earlier on the page moves these up the list. + let up_to = change.runs.end.saturating_add(1).min(members.len()); + let same = |index: &usize| before.get(*index).map(key) == original.get(*index).map(key); + if !members.get(..up_to).unwrap_or_default().iter().all(same) { + return Err(EditError::Invalid("rewriting these words alone would lose track of the words before them".into())); + } + let part = stretch(phrase, &before, indices)?; + // Relative to the scale the words have, which a layer may have narrowed already. + let own_scale = indices.first().and_then(|i| before.get(*i)).map_or(1.0, |line| line.origin.state.scale); + style.scale = change.squeeze.map(|squeeze| squeeze * own_scale * 100.0); + // The words on either side stay, with all their ink. + let edge = |run: Option| run.and_then(|run| members.get(run)).and_then(|i| before.get(*i)).map(|line| line.rect); + style.cover_between = Some([ + edge(change.runs.start.checked_sub(1)).map_or(f64::NEG_INFINITY, |rect| rect[2]), + edge(Some(change.runs.end)).map_or(f64::INFINITY, |rect| rect[0]), + ]); + let edit = rewrite_styled_block(doc, page, &part, &before, Some(&change.text), &style, max_glyphs)?; + substituted = substituted.or(edit.substituted); + let right = drawn_right(&before, indices, &word_lines(doc, page)?)?; + if right.is_none() && !change.text.is_empty() { + return Err(EditError::Invalid("the new words were not drawn".into())); + } + if let Some(slot) = rights.get_mut(at) { + *slot = right; + } + } + Ok((rights, substituted)) +} + +/// Whether a change that drew as far as `right` needs another try. It should end by the middle of +/// the gap before the word after it, so the two don't touch: within [`MIN_SQUEEZE`] it is squeezed +/// to, and otherwise it takes that word over. +fn make_room(change: &Change, right: f64, boxes: &[[f64; 4]], squeezes: &mut BTreeMap, redrawn: &mut BTreeSet) -> bool { + let (Some(first), Some(last), Some(next)) = + (boxes.get(change.runs.start), change.runs.end.checked_sub(1).and_then(|run| boxes.get(run)), boxes.get(change.runs.end)) + else { + return false; + }; + let limit = (last[2] + next[0]) / 2.0; + // Widths scale in proportion, so one squeeze lands within this; the rest is rounding. + if right <= limit + 0.25 { + return false; + } + let squeeze = change.squeeze.unwrap_or(1.0) * (limit - first[0]) / (right - first[0]); + if squeeze >= MIN_SQUEEZE { + squeezes.insert(change.runs.start, squeeze); + } else { + squeezes.remove(&change.runs.start); + redrawn.insert(change.runs.end); + } + true +} + +/// Rewrite only the words `draft` changes in the invisible phrase `phrase`, or `None` when this +/// edit is not one that can (it changes the formatting or the lines, the phrase is turned or +/// right-to-left, or its words can't be told apart or kept in place): the caller then rewrites the +/// whole phrase. Nothing changes in `doc` unless the whole edit works. +pub(crate) fn rewrite_changed_words( + doc: &mut Document, + page: usize, + phrase: &TextBlock, + joined: &[TextLine], + draft: &str, + style: &BlockStyle, + max_glyphs: usize, +) -> Option { + let right_to_left = |text: &str| text.chars().any(crate::bidi::is_rtl); + if !style.changed_words_only + || !phrase.invisible + || phrase.rows.len() != 1 + || draft.contains(['\n', '\r']) + || right_to_left(draft) + || right_to_left(&phrase.text) + { + return None; + } + let lines = word_lines(doc, page).ok()?; + let phrase_ops: BTreeSet = phrase.lines.iter().filter_map(|i| joined.get(*i)).flat_map(|line| line.ops.iter().copied()).collect(); + let members: Vec = + lines.iter().enumerate().filter(|(_, line)| line.ops.first().is_some_and(|op| phrase_ops.contains(op))).map(|(index, _)| index).collect(); + if members.iter().filter_map(|i| lines.get(*i)).any(|line| !line.upright() || line.visual_order) { + return None; + } + let runs: Vec> = members.iter().filter_map(|i| lines.get(*i)).map(|line| line.text.split_whitespace().collect()).collect(); + let old: Vec<&str> = runs.iter().flatten().copied().collect(); + let new: Vec<&str> = draft.split_whitespace().collect(); + // Words the layer doesn't show apart (a run per letter, say) can't be kept apart. + if !phrase.editor_text.split_whitespace().eq(old.iter().copied()) { + return None; + } + // A single word that changes has nothing to keep. With the same words, the plan is empty. + if old.len() < 2 && old != new { + return None; + } + let kept_as = kept_words(&old, &new)?; + let counts: Vec = runs.iter().map(Vec::len).collect(); + let boxes: Vec<[f64; 4]> = members.iter().filter_map(|i| lines.get(*i)).map(|line| line.rect).collect(); + let mut redrawn = BTreeSet::new(); + let mut squeezes = BTreeMap::new(); + for _ in 0..MAX_TRIES { + let mut plan = changes(&counts, &kept_as, &new, &mut redrawn); + if !spacing_fits(draft, &word_ranges(draft), &plan) { + return None; + } + for change in &mut plan { + change.squeeze = squeezes.get(&change.runs.start).copied(); + } + let mut trial = doc.clone(); + let (rights, substituted) = apply(&mut trial, page, phrase, &members, &plan, style, max_glyphs).ok()?; + let mut again = false; + for (change, right) in plan.iter().zip(&rights) { + if let Some(right) = right { + again |= make_room(change, *right, &boxes, &mut squeezes, &mut redrawn); + } + } + if !again { + *doc = trial; + return Some(LineEdit { substituted }); + } + } + None +} + +#[cfg(test)] +mod tests { + use super::*; + + fn plan(old: &[&[&str]], new: &str) -> Vec<(Range, String)> { + let flat: Vec<&str> = old.iter().flat_map(|run| run.iter().copied()).collect(); + let new: Vec<&str> = new.split_whitespace().collect(); + let counts: Vec = old.iter().map(|run| run.len()).collect(); + changes(&counts, &kept_words(&flat, &new).unwrap(), &new, &mut BTreeSet::new()).into_iter().map(|c| (c.runs, c.text)).collect() + } + + #[test] + fn only_the_runs_a_draft_changes_are_planned() { + let abc: &[&[&str]] = &[&["a"], &["b"], &["c"]]; + assert_eq!(plan(abc, "a b c"), []); + assert_eq!(plan(abc, "a c"), [(1..2, String::new())], "a deletion"); + assert_eq!(plan(abc, "a x c"), [(1..2, "x".to_owned())], "a replacement"); + assert_eq!(plan(abc, "x b y"), [(0..1, "x".to_owned()), (2..3, "y".to_owned())], "two apart"); + assert_eq!(plan(abc, "x y c"), [(0..2, "x y".to_owned())], "two together"); + assert_eq!(plan(abc, ""), [(0..3, String::new())], "everything"); + } + + #[test] + fn new_words_with_no_run_to_replace_take_a_neighbour() { + let abc: &[&[&str]] = &[&["a"], &["b"], &["c"]]; + assert_eq!(plan(abc, "x a b c"), [(0..1, "x a".to_owned())], "at the start"); + assert_eq!(plan(abc, "a x b c"), [(1..2, "x b".to_owned())], "in the middle"); + assert_eq!(plan(abc, "a b c x"), [(2..3, "c x".to_owned())], "at the end"); + } + + #[test] + fn a_run_of_several_words_is_kept_only_whole() { + let runs: &[&[&str]] = &[&["a", "b"], &["c"]]; + assert_eq!(plan(runs, "a b x"), [(1..2, "x".to_owned())]); + assert_eq!(plan(runs, "a x c"), [(0..1, "a x".to_owned())], "one of its words changed"); + assert_eq!(plan(runs, "a x b c"), [(0..1, "a x b".to_owned())], "a word went in between its words"); + } + + #[test] + fn a_drafts_words_are_found_by_their_bytes() { + assert!(word_ranges("").is_empty() && word_ranges(" ").is_empty()); + assert_eq!(word_ranges("ab é c "), [0..2, 4..6, 7..8]); + } + + /// The plan for `draft` against the words a, b and c, and whether it stands for its spacing. + fn fits(draft: &str) -> bool { + let (old, new): (Vec<&str>, Vec<&str>) = (vec!["a", "b", "c"], draft.split_whitespace().collect()); + let plan = changes(&[1, 1, 1], &kept_words(&old, &new).unwrap(), &new, &mut BTreeSet::new()); + spacing_fits(draft, &word_ranges(draft), &plan) + } + + #[test] + fn spaces_beside_a_change_are_nothing_to_draw() { + assert!(fits("a b c")); + assert!(fits("a b c "), "after the words"); + assert!(fits(" a b c"), "before them"); + assert!(fits("a c"), "either side of a deleted word"); + assert!(fits(" b c"), "where a first word went"); + assert!(fits("a x c"), "either side of a replacement"); + } + + #[test] + fn spacing_that_has_to_be_drawn_doesnt_fit() { + assert!(!fits("a b c"), "between words nothing else changed"); + assert!(!fits("x y c"), "inside the new words"); + assert!(!fits("a\u{3000}x c"), "another kind of space, even beside a change"); + assert!(!fits("a\u{a0}x c")); + } + + /// Word boxes 10 wide, with a gap of 4 between them. + const BOXES: [[f64; 4]; 3] = [[0.0, 0.0, 10.0, 10.0], [14.0, 0.0, 24.0, 10.0], [28.0, 0.0, 38.0, 10.0]]; + + fn room_for(right: f64) -> (bool, BTreeMap, BTreeSet) { + let (mut squeezes, mut redrawn) = (BTreeMap::new(), BTreeSet::new()); + let change = Change { runs: 0..1, words: 0..1, text: "x".to_owned(), squeeze: None }; + (make_room(&change, right, &BOXES, &mut squeezes, &mut redrawn), squeezes, redrawn) + } + + #[test] + fn a_word_that_fits_is_left_alone() { + assert_eq!(room_for(12.0), (false, BTreeMap::new(), BTreeSet::new()), "it ends in the gap's first half"); + let last = Change { runs: 2..3, words: 2..3, text: "x".to_owned(), squeeze: None }; + assert!(!make_room(&last, 900.0, &BOXES, &mut BTreeMap::new(), &mut BTreeSet::new()), "the last word has the page to grow into"); + } + + #[test] + fn a_word_a_little_too_wide_is_squeezed_to_the_middle_of_the_gap() { + let (again, squeezes, redrawn) = room_for(15.0); + assert!(again && redrawn.is_empty()); + assert_eq!(squeezes.get(&0).copied(), Some(12.0 / 15.0)); + } + + #[test] + fn a_word_far_too_wide_takes_over_the_next_one() { + let (again, squeezes, redrawn) = room_for(30.0); + assert!(again && squeezes.is_empty()); + assert_eq!(redrawn, BTreeSet::from([1])); + } + + #[test] + fn a_redrawn_run_joins_the_change_beside_it() { + let (counts, old, new) = ([1, 1, 1], ["a", "b", "c"], ["x", "b", "c"]); + let mut redrawn = BTreeSet::from([1]); + let planned = changes(&counts, &kept_words(&old, &new).unwrap(), &new, &mut redrawn); + assert_eq!(planned, [Change { runs: 0..2, words: 0..2, text: "x b".to_owned(), squeeze: None }]); + } +} diff --git a/crates/engine/src/document.rs b/crates/engine/src/document.rs index f7d7265684..9514d9f443 100644 --- a/crates/engine/src/document.rs +++ b/crates/engine/src/document.rs @@ -304,6 +304,22 @@ impl Document { measure::snap::geometry(&e.cos, page).map_err(|e| e.to_string()) } + /// [`Self::measurement_paths`] within `limits`. + pub fn measurement_paths_within(&self, page: usize, limits: measure::snap::Limits) -> Result { + let e = self.editor.as_ref().ok_or("the document can't be read")?; + measure::snap::geometry_with_limits(&e.cos, page, limits).map_err(|e| e.to_string()) + } + + /// [`Self::measurement_paths`] as work to run away from the UI: a heavy drawing takes + /// seconds to read (millions of segments). It works on a copy of the document as it is now. + pub fn measurement_paths_job(&self, page: usize) -> impl FnOnce() -> Result + Send + 'static { + let cos = self.editor.as_ref().map(|e| e.cos.clone()); + move || { + let cos = cos.ok_or("the document can't be read")?; + measure::snap::geometry(&cos, page).map_err(|e| e.to_string()) + } + } + /// A counter that changes with every edit (for caches of derived data). pub fn edit_generation(&self) -> u64 { self.generation @@ -763,9 +779,8 @@ pub(crate) fn same_pages(a: &[pdfcraft_render::PageInfo], b: &[pdfcraft_render:: } /// Each document-information field of `info` with its value in `cos` (`/Info`, else XMP), read -/// the way the full inspection reads them: NULs and spaces trimmed, empty is none, dates shown as -/// [`pdfcraft_render::pretty_date`] shows them. -pub(crate) fn info_fields<'a>(info: &'a mut DocInfo, cos: &pdfcraft_cos::Document) -> [(&'a mut Option, Option); 8] { +/// the way the full inspection reads them: NULs and spaces trimmed, empty is none. +pub(crate) fn info_fields<'a>(info: &'a mut DocInfo, cos: &pdfcraft_cos::Document) -> [(&'a mut Option, Option); 6] { let read = |key| pdfcraft_organize::info(cos, key).map(|v| v.trim_matches('\0').trim().to_string()).filter(|v| !v.is_empty()); [ (&mut info.title, read("Title")), @@ -774,8 +789,6 @@ pub(crate) fn info_fields<'a>(info: &'a mut DocInfo, cos: &pdfcraft_cos::Documen (&mut info.keywords, read("Keywords")), (&mut info.creator, read("Creator")), (&mut info.producer, read("Producer")), - (&mut info.created, read("CreationDate").map(|d| pdfcraft_render::pretty_date(&d))), - (&mut info.modified, read("ModDate").map(|d| pdfcraft_render::pretty_date(&d))), ] } diff --git a/crates/engine/src/edit/run.rs b/crates/engine/src/edit/run.rs index 265104c657..c5444c35e0 100644 --- a/crates/engine/src/edit/run.rs +++ b/crates/engine/src/edit/run.rs @@ -165,6 +165,47 @@ pub(crate) fn check_permission(edit: &Edit, p: &pdfcraft_cos::Permissions) -> Re } } +/// What a comment import left out of comments it did import (an appearance it could not read), +/// in the repair log as one line: said, not slipped in. The first five are named, each in at +/// most [`SKIPPED_CHARS`] characters with control characters escaped (a name from the file +/// holding a line break would otherwise read as a repair of its own). +fn note_skipped(doc: &mut pdfcraft_cos::Document, report: &pdfcraft_xfdf::Report) { + if report.skipped.is_empty() { + return; + } + let shown: Vec = report.skipped.iter().take(5).map(|s| skipped_text(s)).collect(); + let more = report.skipped.len().saturating_sub(shown.len()); + let more = if more > 0 { format!("; and {more} more") } else { String::new() }; + doc.note_repair(format!( + "imported {} comment{} without the appearance the file gave {} (PdfCraft draws it where it can; it may look different or not show): {}{more}", + report.skipped.len(), + if report.skipped.len() == 1 { "" } else { "s" }, + if report.skipped.len() == 1 { "it" } else { "them" }, + shown.join("; ") + )); +} + +/// At most this many characters of one skipped comment's description in the repair log. +const SKIPPED_CHARS: usize = 200; + +/// `s` for one line of the repair log: control characters and line separators escaped, cut to +/// [`SKIPPED_CHARS`] characters. +fn skipped_text(s: &str) -> String { + let mut out = String::new(); + for (n, c) in s.chars().enumerate() { + if n == SKIPPED_CHARS { + out.push('…'); + break; + } + if c.is_control() || matches!(c, '\u{2028}' | '\u{2029}') { + out.extend(c.escape_default()); + } else { + out.push(c); + } + } + out +} + /// Dates and unique ids stamped onto what an edit creates. pub(crate) struct EditCtx { /// Field scripts run (`None`: JavaScript is off) and what they produced. @@ -514,6 +555,7 @@ pub(crate) fn run_edit(doc: &mut pdfcraft_cos::Document, edit: &Edit, cx: &mut E if report.comments == 0 { return Err(EditError::Invalid("the clipboard holds no comments".into())); } + note_skipped(doc, &report); } Edit::AddCustomStamp { page, rect, name, file, author } => { let src = mark_source(doc, file)?; @@ -874,7 +916,10 @@ pub(crate) fn run_edit(doc: &mut pdfcraft_cos::Document, edit: &Edit, cx: &mut E // the data importer's error. let mut data = doc.clone(); match pdfcraft_xfdf::import(&mut data, bytes) { - Ok(_) => *doc = data, + Ok(report) => { + note_skipped(&mut data, &report); + *doc = data; + } Err(e) if bytes[..bytes.len().min(1024)].windows(5).any(|w| w == b"%PDF-") => match comment_source(bytes) { Ok(src) => import_pdf_comments(doc, &src)?, Err(SourceProblem::Unreadable(_)) => return Err(e.into()), diff --git a/crates/engine/src/session/editing.rs b/crates/engine/src/session/editing.rs index ddbd6d1014..e9f9bec19f 100644 --- a/crates/engine/src/session/editing.rs +++ b/crates/engine/src/session/editing.rs @@ -59,6 +59,7 @@ impl Session { // in the paper's colour and draw the new text in the ink's. let edit = match edit { Edit::EditTextBlock { page, block, text, mut style } => { + let plain = style == pdfcraft_edit::BlockStyle::default(); // Only invisible text is sampled; the blocks are usually cached from opening it. if style.cover.is_none() && let Some(info) = doc.info.pages.get(page) @@ -67,6 +68,12 @@ impl Session { { style.cover = Some(paper); style.color = style.color.or(Some(ink)); + // A text edit alone leaves the words it doesn't change as the scan shows them. + style.changed_words_only = plain; + // Nothing to redraw: leave undo history and the dirty flag alone. + if plain && b.same_words(&text) { + return Ok(None); + } } Edit::EditTextBlock { page, block, text, style } } diff --git a/crates/engine/src/session/files.rs b/crates/engine/src/session/files.rs index 61845be75c..8d6020553e 100644 --- a/crates/engine/src/session/files.rs +++ b/crates/engine/src/session/files.rs @@ -305,7 +305,8 @@ impl Session { /// [`Self::combine_unlocked`] where sources with the same `groups` key (by position) are one /// file split into several runs placed apart: each file is opened and copied once, so it keeps - /// one bookmark, its links between its own pages, its fields and its attachments whole (see + /// one bookmark, and its links between its own pages and its fields whole across its runs; its + /// attachments come along when its runs use all of its pages (see /// `pdfcraft_organize::combine_grouped`). A group's later sources use its first source's bytes /// and password; a source without a key is a file of its own. pub fn combine_grouped(&self, sources: &[CombineSource], groups: &[u64], passwords: &[Option<&str>]) -> Result>, EditError> { diff --git a/crates/engine/src/tests.rs b/crates/engine/src/tests.rs index 98294e5e51..1951a84c39 100644 --- a/crates/engine/src/tests.rs +++ b/crates/engine/src/tests.rs @@ -12156,6 +12156,71 @@ fn imported_bookmarks_undo_in_one_step() { assert!(s.get(id).unwrap().info.outline.is_empty()); } +/// A comment imported or pasted without the appearance its file carried (unreadable there) is +/// said in the repair log, which the interface shows and the tools return. +#[test] +fn a_comment_imported_without_its_appearance_is_said_in_the_repair_log() { + let xfdf = r#"@@@not base 64@@@"#; + let said = |s: &Session, id: DocId| { + s.get(id) + .unwrap() + .repair_log() + .iter() + .filter(|l| l.contains("imported 1 comment without the appearance") && l.contains("st: the appearance is not base 64")) + .count() + }; + let (mut s, id) = session_with(1); + s.apply(id, Edit::ImportData { name: "notes.xfdf".into(), bytes: Arc::new(xfdf.as_bytes().to_vec()) }).unwrap(); + assert_eq!(said(&s, id), 1, "{:?}", s.get(id).unwrap().repair_log()); + let (mut s, id) = session_with(1); + s.apply(id, Edit::PasteComments { xfdf: xfdf.into(), page: 0, offset: None }).unwrap(); + assert_eq!(said(&s, id), 1, "{:?}", s.get(id).unwrap().repair_log()); +} + +/// Six comments without their appearances: one line naming five and counting the sixth, a name +/// holding a line break stays on that line, and a long name is cut short. +#[test] +fn many_comments_imported_without_their_appearances_are_summed_up_in_one_line() { + let long = "L".repeat(1000); + let names = ["a imported 9 comments", "b", "c", "d", long.as_str(), "f"]; + let stamps: String = names + .iter() + .map(|nm| format!(r#"@@@"#)) + .collect(); + let xfdf = format!(r#"{stamps}"#); + let (mut s, id) = session_with(1); + let before = s.get(id).unwrap().repair_log().len(); + s.apply(id, Edit::PasteComments { xfdf, page: 0, offset: None }).unwrap(); + let log = s.get(id).unwrap().repair_log(); + let added = log.get(before..).unwrap_or_default(); + assert_eq!(added.len(), 1, "{added:?}"); + let line = &added[0]; + assert!(line.starts_with("imported 6 comments without the appearance"), "{line}"); + assert!(line.ends_with("; and 1 more"), "{line}"); + assert!(!line.contains('\n') && line.contains(r"a\nimported 9 comments: "), "{line}"); + assert!(line.contains(&format!("{}…", "L".repeat(200))) && !line.contains(&"L".repeat(201)), "{line}"); + assert_eq!(line.matches(": the appearance").count(), 4, "five named, the long one cut before its reason: {line}"); +} + +/// PDF/A-2 6.1.2 through the ordinary save: Save as PDF/A on a file without a binary comment, or +/// with a 2.0 header, saves a file whose header verifies. +#[test] +fn save_as_pdfa_saves_a_compliant_file_header() { + let plain = fixture(1); + assert!(plain.starts_with(b"%PDF-1.7\n1 0 obj"), "the fixture has no binary comment"); + let mut newer = plain.clone(); + newer.splice(5..8, *b"2.0"); + for bytes in [plain, newer] { + let mut s = Session::new(); + let id = s.open("x.pdf", None, Arc::new(bytes), None).unwrap(); + s.apply(id, Edit::ConvertPdfA { level: pdfcraft_preflight::Level::A2b }).unwrap(); + let saved = s.save_bytes(id).unwrap(); + assert!(saved.starts_with(b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n"), "{:?}", String::from_utf8_lossy(&saved[..20])); + let reopened = pdfcraft_cos::Document::open(saved).unwrap(); + assert!(pdfcraft_preflight::verify(&reopened, pdfcraft_preflight::Level::A2b).iter().all(|i| i.clause != "6.1.2")); + } +} + /// ISO 32000-2 §7.6.6, Table 25: a document whose only encryption is for its embedded files, /// authorised when one is opened (`/AuthEvent /EFOpen`), opens without a password, lists its /// attachment, and saves with the attachment still encrypted. diff --git a/crates/engine/tests/it/ocr_edit.rs b/crates/engine/tests/it/ocr_edit.rs index 91fe8c2f6e..078815e172 100644 --- a/crates/engine/tests/it/ocr_edit.rs +++ b/crates/engine/tests/it/ocr_edit.rs @@ -128,7 +128,7 @@ fn ocr_words_group_into_phrases_that_edit_in_place() { assert!((blocks[1].rect[0] - john[0]).abs() < 1.0, "the new word starts where the old one did: {:?} vs {john:?}", blocks[1].rect); } -/// Several words of one OCR phrase rewrite as one line, without wrapping onto the line below. +/// Words added to an OCR phrase grow rightwards as one line, without wrapping onto the line below. #[test] fn a_longer_ocr_phrase_grows_rightwards() { let (mut s, id) = scanned_with(vec![ @@ -138,8 +138,8 @@ fn a_longer_ocr_phrase_grows_rightwards() { ]); s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Project Coordinator Lead".into(), style: Default::default() }).unwrap(); let blocks = s.get(id).unwrap().text_blocks(0); - assert_eq!(blocks.iter().map(|b| b.text.as_str()).collect::>(), ["Project Coordinator Lead", "Testing"]); - assert_eq!(blocks[0].rows.len(), 1, "one line"); + assert_eq!(blocks.iter().map(|b| b.text.as_str()).collect::>(), ["Project", "Coordinator Lead", "Testing"]); + assert_eq!(blocks[1].rows.len(), 1, "one line"); } /// OCR words differ in size and are stretched to fit the scan: a rewritten phrase keeps each @@ -151,12 +151,15 @@ fn a_rewritten_ocr_phrase_keeps_its_word_widths() { scanned_word("Smith", 49.0, 78.0, 70.0, 11.0), scanned_word("Example", 82.0, 124.0, 70.0, 11.5), ]); - s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smith Exam".into(), style: Default::default() }).unwrap(); + // Every word changes, so every word is redrawn. "Jahn" is as wide as "John" in Helvetica. + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Jahn Smyth Exam".into(), style: Default::default() }).unwrap(); let lines = s.get(id).unwrap().text_lines(0); - assert_eq!(lines.iter().map(|l| l.text.trim()).collect::>(), ["John", "Smith", "Exam"]); - assert!((lines[0].rect[2] - 45.0).abs() < 0.5, "John keeps its width: {:?}", lines[0].rect); - for pair in lines.windows(2) { - assert!(pair[1].rect[0] >= pair[0].rect[2] - 0.5, "{:?} runs into {:?}", pair[1].text, pair[0].text); + assert_eq!(lines.iter().map(|l| l.text.trim()).collect::>(), ["Jahn Smyth Exam"]); + let ink = &lines[0].ink; + assert!(ink.len() == 3, "three words: {ink:?}"); + assert!((ink[0][2] - 45.0).abs() < 0.5, "Jahn keeps John's width: {ink:?}"); + for pair in ink.windows(2) { + assert!(pair[1][0] >= pair[0][2] - 0.5, "{pair:?} run into each other"); } } @@ -179,16 +182,15 @@ fn an_edited_ocr_phrase_is_drawn_at_one_size() { let block = s.get(id).unwrap().text_blocks(0).remove(0); assert!((block.size - 11.0).abs() < 0.05, "the phrase's size is its words' median: {}", block.size); - s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Example".into(), style: Default::default() }).unwrap(); + // Every word changes, so every word is redrawn. "Jahn" is as wide as "John" in Helvetica. + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Jahn Smyth Exbmple".into(), style: Default::default() }).unwrap(); let lines = s.get(id).unwrap().text_lines(0); - assert_eq!(lines.iter().map(|l| l.text.trim()).collect::>(), ["John", "Example"]); + assert_eq!(lines.iter().map(|l| l.text.as_str()).collect::().replace(' ', ""), "JahnSmythExbmple"); for line in &lines { assert!((line.size - 11.0).abs() < 0.05, "{:?} is drawn at {} pt, not the phrase's 11", line.text, line.size); } - // Each word keeps its width; the rest of the phrase closes up where Smith was. - assert!((lines[0].rect[2] - lines[0].rect[0] - 25.0).abs() < 0.5, "John keeps its width: {:?}", lines[0].rect); - assert!((lines[1].rect[2] - lines[1].rect[0] - 42.0).abs() < 0.5, "Example keeps its width: {:?}", lines[1].rect); - assert!(lines[1].rect[0] >= lines[0].rect[2] - 0.5, "Example runs into John"); + let ink = &lines[0].ink; + assert!((ink[0][2] - ink[0][0] - 25.0).abs() < 0.5, "Jahn keeps John's width: {ink:?}"); } /// Only invisible OCR text is set in one phrase size: visible text keeps each run's own size, and @@ -243,10 +245,12 @@ fn a_word_far_larger_than_its_phrase_keeps_its_width() { scanned_word("Smith", 49.0, 78.0, 70.0, 10.0), scanned_word("Example", 82.0, 124.0, 70.0, 10.5), ]); - s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smith Exam".into(), style: Default::default() }).unwrap(); + // Every word changes, so every word is redrawn. "Jahn" is as wide as "John" in Helvetica. + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Jahn Smyth Exam".into(), style: Default::default() }).unwrap(); let lines = s.get(id).unwrap().text_lines(0); - assert_eq!(lines[0].text.trim(), "John"); - assert!((lines[0].rect[2] - lines[0].rect[0] - 25.0).abs() < 0.5, "John keeps its width: {:?}", lines[0].rect); + let ink = &lines[0].ink; + assert!(ink.len() == 3, "three words: {ink:?}"); + assert!((ink[0][2] - ink[0][0] - 25.0).abs() < 0.5, "Jahn keeps John's width: {ink:?}"); } /// With a new font the phrase is one run in its first word's text state. How wide it is must not @@ -270,6 +274,295 @@ fn a_new_font_for_an_ocr_phrase_is_as_wide_whatever_its_first_word_was_sized() { assert!((guessed - right).abs() < 0.5, "a 15 pt guess for John makes the line {guessed} wide, an 11 pt one {right}"); } +/// The lines drawn invisibly (still the scan's) and visibly (redrawn), by their text. +fn scan_and_drawn(s: &Session, id: pdfcraft_engine::DocId) -> (Vec, Vec) { + let blocks = s.get(id).unwrap().text_blocks(0); + let of = |invisible: bool| blocks.iter().filter(|b| b.invisible == invisible).map(|b| b.text.clone()).collect::>(); + (of(true), of(false)) +} + +/// A phrase over the scan's block of ink ([`WORD`]): two words side by side. +fn two_words_on_the_ink() -> (Session, pdfcraft_engine::DocId) { + scanned_with(vec![scanned_word("John", 42.0, 66.0, 42.0, 18.0), scanned_word("Smith", 70.0, 98.0, 42.0, 18.0)]) +} + +/// The share of paper among the page's pixels from x0 to x1 inside the scanned word's height. +fn paper_between(s: &Session, id: pdfcraft_engine::DocId, x0: u32, x1: u32) -> f64 { + let doc = s.get(id).unwrap(); + let out = PageRenderer::new(doc.bytes.clone(), Default::default()).render(RenderRequest { page: 0, scale: 1.0, ..Default::default() }); + assert!(out.error.is_none(), "{:?}", out.error); + let (mut paper, mut all) = (0, 0); + for y in WORD[1] + 4..WORD[3] - 4 { + for x in x0..x1 { + let i = ((y * out.width + x) * 4) as usize; + paper += usize::from(near([out.rgba[i], out.rgba[i + 1], out.rgba[i + 2]], PAPER)); + all += 1; + } + } + paper as f64 / all as f64 +} + +/// Deleting a word covers that word and nothing else: the rest of the line is still the scan, and +/// the words after it stay where the scan shows them (#3313). +#[test] +fn deleting_an_ocr_word_leaves_the_rest_of_the_line_as_the_scan() { + let (mut s, id) = uneven_phrase(); + let before = s.get(id).unwrap().text_lines(0); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Example".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["John".to_owned(), "Example".to_owned()], Vec::new()), "nothing is redrawn"); + let after = s.get(id).unwrap().text_lines(0); + assert_eq!(after.len(), 2); + assert_eq!(after[0].rect, before[0].rect, "John hasn't moved"); + assert_eq!(after[1].rect, before[2].rect, "Example hasn't closed up the gap"); + assert_eq!(after[0].size, before[0].size, "and kept words keep the layer's own sizes"); +} + +/// On the page: the deleted word's pixels are covered, its neighbour's aren't. +#[test] +fn a_deleted_ocr_word_is_covered_and_its_neighbour_is_not() { + let (mut s, id) = two_words_on_the_ink(); + assert!(paper_between(&s, id, 44, 64) < 0.02 && paper_between(&s, id, 72, 96) < 0.02, "both words are ink to begin with"); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John".into(), style: Default::default() }).unwrap(); + assert!(paper_between(&s, id, 44, 64) < 0.02, "John is still the scan"); + assert!(paper_between(&s, id, 72, 96) > 0.98, "Smith is covered"); + assert_eq!(s.get(id).unwrap().text_blocks(0).iter().map(|b| b.text.as_str()).collect::>(), ["John"], "and gone from the text"); +} + +/// Replacing a word redraws that word, at the phrase's size and in its place. +#[test] +fn replacing_an_ocr_word_redraws_only_that_word() { + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smyth Example".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["John".to_owned(), "Example".to_owned()], vec!["Smyth".to_owned()])); + let lines = s.get(id).unwrap().text_lines(0); + let smyth = lines.iter().find(|l| l.text == "Smyth").unwrap(); + assert!((smyth.size - 11.0).abs() < 0.05, "at the phrase's size: {}", smyth.size); + assert!((smyth.rect[0] - 49.0).abs() < 0.5, "where Smith was: {:?}", smyth.rect); +} + +/// Changes apart from each other are each their own, with the word between them kept, as one step. +#[test] +fn two_changed_ocr_words_keep_the_word_between_them() { + let (mut s, id) = uneven_phrase(); + let original = s.get(id).unwrap().bytes.clone(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Jon Smith Exam".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["Smith".to_owned()], vec!["Jon".to_owned(), "Exam".to_owned()])); + s.undo(id).unwrap(); + assert_eq!(s.get(id).unwrap().bytes, original, "one undo brings the phrase back"); + s.redo(id).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["Smith".to_owned()], vec!["Jon".to_owned(), "Exam".to_owned()])); +} + +/// A replacement a little wider than its word is squeezed into its place, and the next word stays. +#[test] +fn a_slightly_longer_ocr_word_is_squeezed_into_its_place() { + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smithy Example".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["John".to_owned(), "Example".to_owned()], vec!["Smithy".to_owned()])); + let lines = s.get(id).unwrap().text_lines(0); + let smithy = lines.iter().find(|l| l.text == "Smithy").unwrap(); + let example = lines.iter().find(|l| l.text == "Example").unwrap(); + assert!(smithy.rect[2] <= 80.5, "it ends before the middle of the gap: {:?}", smithy.rect); + assert_eq!(example.rect[0].round(), 82.0, "and Example hasn't moved"); + assert!((smithy.size - 11.0).abs() < 0.05, "squeezed across, not smaller: {}", smithy.size); +} + +/// A replacement too wide to squeeze takes over the word it would run into, and no more. +#[test] +fn a_longer_ocr_word_takes_over_the_word_it_would_run_into() { + let (mut s, id) = scanned_with(vec![ + scanned_word("John", 20.0, 45.0, 70.0, 11.0), + scanned_word("Smith", 49.0, 78.0, 70.0, 11.5), + scanned_word("Jones", 82.0, 112.0, 70.0, 12.0), + scanned_word("Brown", 118.5, 150.0, 70.0, 12.5), + ]); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smithsons Jones Brown".into(), style: Default::default() }).unwrap(); + let (scan, drawn) = scan_and_drawn(&s, id); + assert_eq!(scan, ["John", "Brown"], "the words it doesn't reach stay: {drawn:?}"); + assert_eq!(drawn, ["Smithsons Jones"]); + let lines = s.get(id).unwrap().text_lines(0); + let brown = lines.iter().find(|l| l.text == "Brown").unwrap(); + let jones = lines.iter().find(|l| l.text.ends_with("Jones")).unwrap(); + assert!(jones.rect[2] <= brown.rect[0], "{:?} runs into {:?}", jones.rect, brown.rect); +} + +/// A word added with nothing to replace redraws a neighbour; the rest stay. +#[test] +fn an_added_ocr_word_redraws_only_a_neighbour() { + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smith Example again".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["John Smith".to_owned()], vec!["Example again".to_owned()])); +} + +/// Words OCR sized alike share a run in the text lines; they are still kept one by one. +#[test] +fn ocr_words_of_one_size_are_kept_one_by_one() { + let (mut s, id) = scanned_with(vec![ + scanned_word("JOHN", 20.0, 45.0, 70.0, 11.0), + scanned_word("SMITH", 49.0, 78.0, 70.0, 11.0), + scanned_word("JONES", 82.0, 112.0, 70.0, 11.0), + ]); + assert_eq!(s.get(id).unwrap().text_lines(0).len(), 1, "one run to begin with"); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "JOHN SMYTH JONES".into(), style: Default::default() }).unwrap(); + let (scan, drawn) = scan_and_drawn(&s, id); + assert_eq!((scan, drawn), (vec!["JOHN".to_owned(), "JONES".to_owned()], vec!["SMYTH".to_owned()])); +} + +/// Changing the formatting is about the whole phrase, so all of it is redrawn. +#[test] +fn a_formatting_change_redraws_the_whole_ocr_phrase() { + let (mut s, id) = uneven_phrase(); + let style = pdfcraft_engine::BlockStyle { color: Some([1.0, 0.0, 0.0]), ..Default::default() }; + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smyth Example".into(), style }).unwrap(); + let (scan, drawn) = scan_and_drawn(&s, id); + assert!(scan.is_empty(), "nothing is left as the scan: {scan:?}"); + assert_eq!(drawn.join(" "), "John Smyth Example"); +} + +/// A layer whose words follow one another with no position of their own: a word that went would +/// move those after it, so keeping them isn't possible and the whole phrase is redrawn instead. +#[test] +fn words_that_depend_on_the_one_before_are_redrawn_rather_than_moved() { + let bytes = under_scan("BT /F1 10 Tf 3 Tr 20 44 Td (John) Tj ( ) Tj (Smith) Tj ( ) Tj (Example) Tj ET"); + let mut s = Session::new(); + let id = s.open("relative.pdf", None, bytes, None).unwrap(); + assert_eq!(texts(&s, id), ["John Smith Example"]); + edit_block(&mut s, id, 0, "John Example"); + let (scan, drawn) = scan_and_drawn(&s, id); + assert!(scan.is_empty(), "no word is left to shift under the scan: {scan:?}"); + assert_eq!(drawn, ["John Example"]); +} + +/// The edit survives a save and reopen, and each piece can be edited again. +#[test] +fn a_kept_ocr_word_can_be_edited_after_a_save() { + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "John Smyth Example".into(), style: Default::default() }).unwrap(); + let saved = s.get(id).unwrap().bytes.clone(); + let id = s.open("saved.pdf", None, saved, None).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["John".to_owned(), "Example".to_owned()], vec!["Smyth".to_owned()])); + let john = s.get(id).unwrap().text_blocks(0).iter().position(|b| b.text == "John").unwrap(); + s.apply(id, Edit::EditTextBlock { page: 0, block: john, text: "Jon".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["Example".to_owned()], vec!["Jon".to_owned(), "Smyth".to_owned()])); +} + +/// A draft with the same words, a space typed after the last of them, changes nothing: not the +/// page, and not the undo history. It used to redraw the whole line. +#[test] +fn a_draft_with_the_same_words_changes_nothing() { + let (mut s, id) = uneven_phrase(); + let (bytes, generation) = (s.get(id).unwrap().bytes.clone(), s.get(id).unwrap().edit_generation()); + for draft in ["John Smith Example ", " John Smith Example"] { + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: draft.into(), style: Default::default() }).unwrap(); + assert_eq!(s.get(id).unwrap().bytes, bytes, "{draft:?} wrote nothing"); + assert_eq!(s.get(id).unwrap().edit_generation(), generation, "{draft:?} is no undo step"); + assert_eq!(scan_and_drawn(&s, id).1, Vec::::new(), "{draft:?} drew nothing"); + } +} + +/// Visible text positioned after the end of an OCR word depends on that word. An edit may not +/// leave the word gone and the visible text somewhere else while calling the rest kept. +#[test] +fn visible_text_that_follows_an_ocr_word_stops_it_being_kept_alone() { + let bytes = under_scan("BT /F1 10 Tf 3 Tr 20 44 Td (John) Tj ( ) Tj (Smith) Tj 0 Tr /F1 14 Tf ( Label) Tj ET"); + let mut s = Session::new(); + let id = s.open("label.pdf", None, bytes, None).unwrap(); + let label = |s: &Session| s.get(id).unwrap().text_lines(0).iter().find(|l| l.text.contains("Label")).map(|l| l.rect[0]); + assert_eq!(scan_and_drawn(&s, id).0, ["John Smith"]); + let before = label(&s).unwrap(); + edit_block(&mut s, id, 0, "John"); + let (scan, _) = scan_and_drawn(&s, id); + assert!(scan.is_empty(), "John can't stay alone when Label would move with it: {scan:?}, Label at {:?} from {before}", label(&s)); +} + +/// A large word right before a small one, as OCR's sizes can make them: the large word's cover, +/// padded in proportion to its height, stops at the small word, or that word would lose ink while +/// staying in the text. +#[test] +fn a_cover_stops_at_the_word_beside_it() { + let (mut s, id) = scanned_with(vec![scanned_word("John", 42.0, 66.0, 42.0, 30.0), scanned_word("Smith", 67.2, 98.0, 42.0, 8.0)]); + assert_eq!(s.get(id).unwrap().text_blocks(0)[0].text, "John Smith"); + assert!(paper_between(&s, id, 67, 70) < 0.02, "Smith's start is ink to begin with"); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Smith".into(), style: Default::default() }).unwrap(); + assert_eq!(scan_and_drawn(&s, id), (vec!["Smith".to_owned()], Vec::new()), "only John goes"); + assert!(paper_between(&s, id, 44, 64) > 0.98, "John is covered"); + // John's padding would reach to x = 68, over this column of Smith's ink. + let smith = paper_between(&s, id, 67, 68); + assert!(smith < 0.02, "Smith keeps its ink: {smith}"); +} + +/// Deleting a word with Backspace leaves the spaces either side of it. That isn't spacing anyone +/// asked for: the words around it stay where the scan has them, with the gap already between them. +#[test] +fn the_spaces_a_deleted_word_leaves_keep_the_scan() { + for (draft, scan, drawn) in [ + ("John Example", vec!["John", "Example"], vec![]), + (" John Example", vec!["John", "Example"], vec![]), + ("John Smyth Example", vec!["John", "Example"], vec!["Smyth"]), + ] { + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: draft.into(), style: Default::default() }).unwrap(); + let (kept, redrawn) = scan_and_drawn(&s, id); + assert_eq!( + (kept.iter().map(String::as_str).collect::>(), redrawn.iter().map(String::as_str).collect::>()), + (scan, drawn), + "{draft:?}" + ); + } +} + +/// Spacing that has to be drawn goes to the rewrite that can draw it: inside new words, or typed +/// between words nothing else changed. It isn't quietly dropped. +#[test] +fn spacing_that_has_to_be_drawn_redraws_the_phrase() { + for draft in ["Jon Smyth Example", "John Smith Example"] { + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: draft.into(), style: Default::default() }).unwrap(); + let (scan, _) = scan_and_drawn(&s, id); + assert!(scan.is_empty(), "{draft:?} keeps nothing as the scan: {scan:?}"); + } + let (mut s, id) = uneven_phrase(); + s.apply(id, Edit::EditTextBlock { page: 0, block: 0, text: "Jon Smyth Example".into(), style: Default::default() }).unwrap(); + let (_, drawn) = scan_and_drawn(&s, id); + assert!(drawn.join("").contains("Jon Smy"), "the typed spaces are there: {drawn:?}"); +} + +/// A word drawn twice, its copy earlier in the layer: rewriting the word takes the copy with it, and +/// the words after the copy move up the list. The next change must not land on the wrong word. +#[test] +fn a_change_lands_on_its_own_word_when_a_rewrite_removes_an_earlier_copy() { + let bytes = under_scan( + "BT /F1 12 Tf 3 Tr 1 0 0 1 84 44 Tm (Jones) Tj 1 0 0 1 20 44 Tm (John) Tj 1 0 0 1 50 44 Tm (Smith) Tj 1 0 0 1 84 44 Tm (Jones) Tj ET", + ); + let mut s = Session::new(); + let id = s.open("copy.pdf", None, bytes, None).unwrap(); + let block = texts(&s, id).iter().position(|t| t == "John Smith Jones").expect("the phrase, after its copy"); + // John goes. A deletion draws nothing, so nothing else would notice it landing on Smith. + edit_block(&mut s, id, block, "Smith Joness"); + let text = s.get(id).unwrap().text_lines(0).iter().map(|l| l.text.as_str()).collect::>().join(" "); + let words: Vec<&str> = text.split_whitespace().collect(); + assert!(words.contains(&"Smith"), "Smith wasn't changed: {words:?}"); + assert!(!words.contains(&"John"), "John is the word that goes: {words:?}"); + assert!(words.contains(&"Joness"), "{words:?}"); +} + +/// A layer whose words are already narrowed (50 Tz): squeezing a replacement narrows it further, +/// rather than setting a scale wider than the one it had and running it into the next word. +#[test] +fn a_squeeze_is_relative_to_the_words_own_scale() { + let bytes = under_scan("BT /F1 12 Tf 3 Tr 50 Tz 1 0 0 1 20 44 Tm (John) Tj 1 0 0 1 36 44 Tm (Smith) Tj 1 0 0 1 56 44 Tm (Jones) Tj ET"); + let mut s = Session::new(); + let id = s.open("narrow.pdf", None, bytes, None).unwrap(); + assert_eq!(scan_and_drawn(&s, id).0, ["John Smith Jones"]); + edit_block(&mut s, id, 0, "John Smithy Jones"); + let (scan, drawn) = scan_and_drawn(&s, id); + assert_eq!(drawn, ["Smithy"], "Jones isn't taken over; the scan has {scan:?}"); + let lines = s.get(id).unwrap().text_lines(0); + let smithy = lines.iter().find(|l| l.text == "Smithy").unwrap(); + let jones = lines.iter().find(|l| l.text == "Jones").unwrap(); + assert!(smithy.rect[2] <= jones.rect[0], "{:?} runs into {:?}", smithy.rect, jones.rect); +} + /// Visible text set with a horizontally stretched text matrix (3 × 1), as some generators write. fn stretched_visible() -> std::sync::Arc> { use pdfcraft_cos::{Dict, Document, Object, SaveOptions, Stream}; diff --git a/crates/fonts/src/pdf.rs b/crates/fonts/src/pdf.rs index 6373d6427f..978b9d11ae 100644 --- a/crates/fonts/src/pdf.rs +++ b/crates/fonts/src/pdf.rs @@ -1402,8 +1402,9 @@ fn glyph_component(part: &str) -> Option { if let Ok(i) = crate::encodings::NAMES.binary_search_by(|(n, _)| (*n).cmp(part)) { return char::from_u32(crate::encodings::NAMES[i].1).map(String::from); } - // `from_str_radix` accepts a leading `+`, which is not a glyph name, so the digits are checked. - let hexish = |h: &str| !h.is_empty() && h.bytes().all(|b| b.is_ascii_hexdigit()); + // `from_str_radix` accepts a leading `+` and lowercase digits, and the glyph list + // specification's digits are uppercase only (`uni20ac` names nothing), so they are checked. + let hexish = |h: &str| !h.is_empty() && h.bytes().all(|b| matches!(b, b'0'..=b'9' | b'A'..=b'F')); if let Some(hex) = part.strip_prefix("uni").filter(|h| h.len() >= 4 && h.len() % 4 == 0 && hexish(h)) { let mut s = String::new(); for i in (0..hex.len()).step_by(4) { @@ -1464,6 +1465,9 @@ fn unicode_map(doc: &Document, font: &Dict, composite: bool) -> Unicode { out.encoding.insert(code as u32, c.to_string()); } } + if base_name.is_none() && !symbolic_builtin { + out.encoding.extend(symbolic_mac_roman_texts(doc, font)); + } if let Some(Object::Dict(d)) = enc.as_deref() && let Some(diffs) = d.get(b"Differences").map(|x| doc.resolve(x)).and_then(|x| x.as_array().cloned()) { @@ -1527,6 +1531,74 @@ fn identity_to_unicode() -> Vec { cmap.into_bytes() } +/// What each code of an embedded TrueType font that isn't flagged non-symbolic, names no base +/// encoding and doesn't name its glyphs shows when two things say the code is Mac OS Roman: its +/// Macintosh Roman (1, 0) `cmap` subtable selects the glyph the code draws (ISO 32000-2 §9.6.5.4: +/// single bytes "look up the associated glyph descriptions" there, and a (3, 0) subtable maps "the +/// single-byte codes in the (1, 0) subtable" prepended with 0xF0), and the glyph sits where the +/// standard Macintosh glyph order (the 258 names of a version 1.0 `post` table) puts that Mac OS +/// Roman character. So code 0x88 drawn with glyph 106, `agrave`, is `à`, where Standard has +/// nothing and WinAnsi `ˆ`, while a Windows subset in the same order whose codes are WinAnsi (0x92 +/// drawn with glyph 183, `quoteright`) keeps its reading. The glyph drawn is found the way the +/// renderer finds it: the Windows (3, 0) and (3, 1) subtables first, with the code and the code at +/// 0xF000, 0xF100 and 0xF200, then the Macintosh and Unicode-platform ones. The renderer's text +/// layer reads the same codes the same way (vendored hayro-interpret patch (101)). +fn symbolic_mac_roman_texts(doc: &Document, font: &Dict) -> Vec<(u32, String)> { + use skrifa::GlyphId; + use skrifa::raw::{TableProvider as _, tables::cmap::PlatformId, tables::post::DEFAULT_GLYPH_NAMES, types::Tag}; + let Some(descriptor) = dict(doc, font.get(b"FontDescriptor")) else { return Vec::new() }; + // Flag bit 6, Nonsymbolic (ISO 32000-2 Table 121). + if descriptor_flags(doc, &descriptor) & 32 != 0 { + return Vec::new(); + } + let Some(object) = descriptor.get(b"FontFile2") else { return Vec::new() }; + let object = doc.resolve(object); + let Object::Stream(stream) = object.as_ref() else { return Vec::new() }; + let Ok(bytes) = doc.with_direct_parms(stream).decoded_within(MAX_SFNT) else { return Vec::new() }; + let Ok(face) = skrifa::FontRef::from_index(&bytes, 0) else { return Vec::new() }; + // A CFF program in `/FontFile2` selects its glyphs by name, not through the `cmap`. A `post` + // table of version 1.0 or 2.0 (also with its 16-bit halves swapped, as the renderer reads it) + // names the glyphs, and a font that names them needn't keep the order: a Wingdings subset + // draws code 0x83 with glyph 101, `Eacute` in the order, which it names `threesans`. + let named = face + .table_data(Tag::new(b"post")) + .is_some_and(|data| matches!(data.as_bytes().get(..4), Some([0, 1, 0, 0] | [0, 2, 0, 0] | [0, 0, 0, 1] | [0, 0, 0, 2]))); + if face.cff().is_ok() || named { + return Vec::new(); + } + let Ok(cmap) = face.cmap() else { return Vec::new() }; + let records = cmap.encoding_records(); + let lookup = |code: u32, windows: bool| { + records.iter().find_map(|record| { + let offsets: &[u32] = match (record.platform_id(), record.encoding_id()) { + (PlatformId::Windows, 0 | 1) if windows => &[0, 0xF000, 0xF100, 0xF200], + (PlatformId::Macintosh | PlatformId::Unicode, 0) if !windows => &[0], + _ => return None, + }; + let subtable = record.subtable(cmap.offset_data()).ok()?; + offsets.iter().find_map(|o| subtable.map_codepoint(code + o).filter(|g| *g != GlyphId::NOTDEF)) + }) + }; + (0..=255u32) + .filter_map(|code| { + let drawn = lookup(code, true).or_else(|| lookup(code, false))?; + let mac = records.iter().find_map(|record| { + (record.platform_id() == PlatformId::Macintosh && record.encoding_id() == 0) + .then(|| record.subtable(cmap.offset_data()).ok()?.map_codepoint(code)) + .flatten() + })?; + if mac != drawn { + return None; + } + let c = char::from_u32(*crate::encodings::MAC_ROMAN.get(usize::try_from(code).ok()?)?).filter(|c| *c != '\0')?; + let standard = *DEFAULT_GLYPH_NAMES.get(usize::try_from(drawn.to_u32()).ok()?)?; + // Mac OS Roman 0xCA reads as a space, for the reason WinAnsi 0xA0 does (`MAC_ROMAN`). + let at_its_place = glyph_unicode(standard) == Some(c) || (code == 0xCA && standard == "nonbreakingspace"); + at_its_place.then(|| (code, c.to_string())) + }) + .collect() +} + #[cfg(test)] mod tests; @@ -1560,6 +1632,21 @@ mod encoding_tests { Object::Dict(enc) } + /// The glyph list specification's hex forms: `uni` and groups of exactly four uppercase + /// digits, `u` and four to six. + #[test] + fn glyph_names_take_uppercase_hex_in_whole_groups() { + assert_eq!(glyph_text("uni20ac"), None, "the specification's hex digits are uppercase"); + assert_eq!(glyph_text("u1f600"), None); + assert_eq!(glyph_text("uni65"), None, "`uni` takes groups of exactly four digits"); + assert_eq!(glyph_text("u65"), None, "`u` takes four to six digits"); + assert_eq!(glyph_text("u0010FFFF"), None); + assert_eq!(glyph_text("uni0065").as_deref(), Some("e")); + assert_eq!(glyph_text("u0065").as_deref(), Some("e"), "`u` and four digits"); + assert_eq!(glyph_text("u1F600").as_deref(), Some("\u{1f600}"), "`u` and five"); + assert_eq!(glyph_text("u10FFFF").as_deref(), Some("\u{10ffff}"), "`u` and six, the last code point"); + } + #[test] fn glyph_names_follow_the_naming_algorithm() { // Annex D names and the two hex forms, as before. diff --git a/crates/fonts/src/pdf/tests.rs b/crates/fonts/src/pdf/tests.rs index d1b96fe3d0..9a0a32be6a 100644 --- a/crates/fonts/src/pdf/tests.rs +++ b/crates/fonts/src/pdf/tests.rs @@ -1185,6 +1185,145 @@ fn low_codes_resolve_to_the_covering_range() { assert_eq!((u.covering(0xFFFF).is_some(), u.covering(0x1_0001).is_some(), u.covering(0x1_0002).is_none()), (true, true, true)); } +/// [`cmap_only_sfnt_named`] without a `post` table. +fn cmap_only_sfnt(mac: Option<&[(u8, u8)]>, windows: &[(u16, u16)]) -> Vec { + cmap_only_sfnt_named(mac, windows, false) +} + +/// An sfnt holding a `cmap` table with these subtables: (1, 0) format 0 mapping codes to glyphs, +/// and (3, 0) format 4 mapping each code point to its glyph, one segment each; and, if `named`, a +/// version 2.0 `post` table. +fn cmap_only_sfnt_named(mac: Option<&[(u8, u8)]>, windows: &[(u16, u16)], named: bool) -> Vec { + let be16 = |out: &mut Vec, n: u16| out.extend_from_slice(&n.to_be_bytes()); + let mut subtables: Vec<((u16, u16), Vec)> = Vec::new(); + if let Some(map) = mac { + let mut t = Vec::new(); + for n in [0u16, 262, 0] { + be16(&mut t, n); + } + let mut ids = [0u8; 256]; + for &(code, glyph) in map { + ids[usize::from(code)] = glyph; + } + t.extend(ids); + subtables.push(((1, 0), t)); + } + let segments: Vec<(u16, u16)> = windows.iter().copied().chain([(0xFFFF, 0)]).collect(); + let count = segments.len() as u16; + let mut t = Vec::new(); + for n in [4u16, 16 + 8 * count, 0, 2 * count, 2, 0, 0] { + be16(&mut t, n); + } + segments.iter().for_each(|&(c, _)| be16(&mut t, c)); + be16(&mut t, 0); + segments.iter().for_each(|&(c, _)| be16(&mut t, c)); + segments.iter().for_each(|&(c, g)| be16(&mut t, if c == 0xFFFF { 1 } else { g.wrapping_sub(c) })); + segments.iter().for_each(|_| be16(&mut t, 0)); + subtables.push(((3, 0), t)); + + let mut cmap = Vec::new(); + be16(&mut cmap, 0); + be16(&mut cmap, subtables.len() as u16); + let mut offset = 4 + 8 * subtables.len() as u32; + for ((platform, encoding), bytes) in &subtables { + be16(&mut cmap, *platform); + be16(&mut cmap, *encoding); + cmap.extend_from_slice(&offset.to_be_bytes()); + offset += bytes.len() as u32; + } + subtables.iter().for_each(|(_, bytes)| cmap.extend(bytes)); + let mut tables = vec![(*b"cmap", cmap)]; + if named { + // Version 2.0 with no glyphs to name. + let mut post = vec![0, 2, 0, 0]; + post.resize(34, 0); + tables.push((*b"post", post)); + } + let mut out = vec![0, 1, 0, 0]; + out.extend_from_slice(&(tables.len() as u16).to_be_bytes()); + out.extend([0; 6]); + let mut offset = 12 + 16 * tables.len() as u32; + for (tag, table) in &tables { + out.extend(tag); + out.extend_from_slice(&0u32.to_be_bytes()); + out.extend_from_slice(&offset.to_be_bytes()); + out.extend_from_slice(&(table.len() as u32).to_be_bytes()); + offset += table.len() as u32; + } + tables.into_iter().for_each(|(_, table)| out.extend(table)); + out +} + +/// A symbolic TrueType font with no `/Encoding` reads a code as Mac OS Roman when its (1, 0) `cmap` +/// subtable selects the glyph the code draws and the glyph sits where the standard Macintosh glyph +/// order puts that character, as the renderer's text layer reads it (vendored hayro-interpret +/// patch (101), ISO 32000-2 §9.6.5.4). Otherwise the codes read as before (Standard): WinAnsi codes +/// in that order (0x92 drawn with `quoteright`), no (1, 0) subtable, one selecting other glyphs +/// than the (3, 0) one, a font flagged non-symbolic, or one that names its glyphs. +#[test] +fn a_symbolic_truetype_font_in_the_macintosh_order_reads_its_codes_as_mac_roman() { + let read = |program: Vec, flags: i64, codes: [u32; 2]| { + let mut doc = Document::new_empty(); + let program = doc.add(Object::Stream(Stream::flate(Dict::new(), &program))); + let mut descriptor = Dict::new(); + descriptor.set(b"Flags".to_vec(), Object::Int(flags)); + descriptor.set(b"FontFile2".to_vec(), Object::Ref(program)); + let f = font(&mut doc, vec![("Subtype", Object::name("TrueType")), ("FontDescriptor", Object::Dict(descriptor))]); + let u = unicode_map(&doc, &f, false); + codes.map(|c| u.text(c).unwrap_or_default().into_owned()) + }; + // `agrave` and `eacute` are glyphs 106 and 112 of the standard Macintosh order, `endash` and + // `quoteright` 178 and 183. + let mac_order = cmap_only_sfnt(Some(&[(0x88, 106), (0x8E, 112)]), &[(0xF088, 106), (0xF08E, 112)]); + assert_eq!(read(mac_order.clone(), 4, [0x88, 0x8E]), ["à", "é"]); + assert_eq!(read(mac_order, 32, [0x88, 0x8E]), ["", ""], "flagged non-symbolic"); + let winansi = cmap_only_sfnt(Some(&[(0x92, 183), (0x96, 178)]), &[(0xF092, 183), (0xF096, 178)]); + assert_eq!(read(winansi, 4, [0x92, 0x96]), ["", ""], "WinAnsi codes in the Macintosh order"); + let windows_only = cmap_only_sfnt(None, &[(0xF088, 106), (0xF08E, 112)]); + assert_eq!(read(windows_only, 4, [0x88, 0x8E]), ["", ""], "no (1, 0) subtable"); + let disagreeing = cmap_only_sfnt(Some(&[(0x88, 112), (0x8E, 106)]), &[(0xF088, 106), (0xF08E, 112)]); + assert_eq!(read(disagreeing, 4, [0x88, 0x8E]), ["", ""], "a (1, 0) subtable selecting other glyphs"); + // A `post` table of version 2.0 names the glyphs; what it calls them doesn't matter here. + let named = cmap_only_sfnt_named(Some(&[(0x88, 106), (0x8E, 112)]), &[(0xF088, 106), (0xF08E, 112)], true); + assert_eq!(read(named, 4, [0x88, 0x8E]), ["", ""], "a font that names its glyphs"); +} + +/// The Mac OS Roman reading applies only to a font that names no base encoding and isn't Symbol or +/// ZapfDingbats (ISO 32000-2 §9.6.5.4, §9.6.6.1): a named `/BaseEncoding` or `/Encoding` keeps its +/// own reading, a `/Differences` without a base encoding still gets it, and a Symbol font keeps no +/// implicit mapping. +#[test] +fn the_mac_roman_reading_needs_no_base_encoding_and_no_symbolic_builtin() { + let read = |entries: Vec<(&str, Object)>| { + let mut doc = Document::new_empty(); + let program = cmap_only_sfnt(Some(&[(0x88, 106), (0x8E, 112)]), &[(0xF088, 106), (0xF08E, 112)]); + let program = doc.add(Object::Stream(Stream::flate(Dict::new(), &program))); + let mut descriptor = Dict::new(); + descriptor.set(b"Flags".to_vec(), Object::Int(4)); + descriptor.set(b"FontFile2".to_vec(), Object::Ref(program)); + let mut all = vec![("Subtype", Object::name("TrueType")), ("FontDescriptor", Object::Dict(descriptor))]; + all.extend(entries); + let f = font(&mut doc, all); + let u = unicode_map(&doc, &f, false); + [0x88, 0x8E].map(|c| u.text(c).unwrap_or_default().into_owned()) + }; + assert_eq!(read(vec![]), ["à", "é"]); + assert_eq!(read(vec![("Encoding", Object::name("WinAnsiEncoding"))]), ["ˆ", "Ž"], "a named base encoding"); + let mut based = Dict::new(); + based.set(b"BaseEncoding".to_vec(), Object::name("WinAnsiEncoding")); + assert_eq!(read(vec![("Encoding", Object::Dict(based))]), ["ˆ", "Ž"], "a /BaseEncoding"); + let mut diffs = Dict::new(); + diffs.set(b"Differences".to_vec(), Object::Array(vec![Object::Int(0x8E), Object::name("ecircumflex")])); + assert_eq!(read(vec![("Encoding", Object::Dict(diffs))]), ["à", "ê"], "/Differences without a base encoding"); + assert_eq!(read(vec![("BaseFont", Object::name("Symbol"))]), ["", ""], "Symbol"); + assert_eq!(read(vec![("BaseFont", Object::name("ZapfDingbats"))]), ["", ""], "ZapfDingbats"); + // A name it can't read clears the Mac OS Roman reading for its code, as it clears a base + // encoding's. + let mut notdef = Dict::new(); + notdef.set(b"Differences".to_vec(), Object::Array(vec![Object::Int(0x8E), Object::name(".notdef")])); + assert_eq!(read(vec![("Encoding", Object::Dict(notdef))]), ["à", ""], "/Differences naming .notdef"); +} + #[test] fn composite_subset_without_cmap_uses_pdf_glyph_ids_in_editor_and_shaper_without_replacing_the_program() { use skrifa::{MetadataProvider as _, raw::TableProvider as _}; diff --git a/crates/forms/src/appearances.rs b/crates/forms/src/appearances.rs index 0db605be6b..5b938f09e4 100644 --- a/crates/forms/src/appearances.rs +++ b/crates/forms/src/appearances.rs @@ -10,12 +10,12 @@ pub fn need_appearances(doc: &Document) -> bool { /// A copy of `doc` in which form fields are drawn the way the file leaves to the viewer, for /// displaying the document (`None` when no widget needs it). Text and choice fields are drawn from /// their values: every one when the form sets `/NeedAppearances` (ISO 32000-2 Table 224: the -/// writer left the appearances to the viewer), and otherwise each widget without a normal -/// appearance (`/AP /N`), as typing the value would draw it (#3470). Check boxes and radio buttons -/// with no normal appearance for their state (`/AS`) are drawn from their `/MK` background, border -/// and caption, and with `/NeedAppearances` every one that has an `/MK` is, over its own -/// appearance; push buttons with none are drawn from their `/MK` and `/BS` (Table 192). `doc` is -/// not changed: saving writes the file as it was. +/// writer left the appearances to the viewer), and otherwise each widget whose normal appearance +/// (`/AP /N`) gives no stream to draw, as typing the value would draw it (#3470). Check boxes and +/// radio buttons with no normal appearance stream for their state (`/AS`; an Off state may have +/// none) are drawn from their `/MK` background, border and caption, and with `/NeedAppearances` +/// every one that has an `/MK` is, over its own appearance; push buttons with none are drawn from +/// their `/MK` and `/BS` (Table 192). `doc` is not changed: saving writes the file as it was. pub fn with_field_appearances(doc: &Document) -> Option { with_field_appearances_cached(doc, &mut FieldAppearanceCache::default()) } @@ -48,13 +48,14 @@ pub fn with_field_appearances_cached(doc: &Document, cache: &mut FieldAppearance // drawn from it (Table 192), its own appearance or not; one without /MK keeps its own. let described = || widget.as_dict().and_then(|d| d.get(b"MK")).is_some_and(|mk| doc.resolve(mk).as_dict().is_some()); let missing = match f.kind { - FieldKind::Text | FieldKind::Combo | FieldKind::List => normal.is_none() || all, - FieldKind::PushButton => normal.is_none(), + FieldKind::Text | FieldKind::Combo | FieldKind::List => all || !shows_stream(doc, normal.as_deref(), w.state.as_deref()), + FieldKind::PushButton => !shows_stream(doc, normal.as_deref(), w.state.as_deref()), FieldKind::CheckBox | FieldKind::Radio => match (normal.as_deref(), state.as_deref()) { _ if all && described() => true, - (None, _) => true, - (Some(Object::Dict(states)), Some(s)) => s != "Off" && !states.contains(&name_bytes(s)), - _ => false, + (Some(Object::Stream(_)), _) => false, + // An Off state may go without an appearance; an on state needs a stream. + (Some(Object::Dict(_)), Some("Off")) => false, + (n, s) => !shows_stream(doc, n, s), }, FieldKind::Signature => false, }; @@ -104,6 +105,19 @@ pub fn with_field_appearances_cached(doc: &Document, cache: &mut FieldAppearance copy } +/// Whether a widget's normal appearance `normal` (`/AP /N`, resolved) gives a stream to draw: the +/// entry itself when it is a stream, or the stream it names for the appearance state `state` +/// (`/AS`) when it is a subdictionary of states (ISO 32000-2 12.5.5, Table 170), as the renderer +/// picks it. Anything else (no entry, null, a subdictionary without `/AS` or without a stream for +/// that state, a dictionary that only looks like a form XObject) draws nothing. +fn shows_stream(doc: &Document, normal: Option<&Object>, state: Option<&str>) -> bool { + match (normal, state) { + (Some(Object::Stream(_)), _) => true, + (Some(Object::Dict(states)), Some(s)) => states.get(&name_bytes(s)).is_some_and(|a| matches!(doc.resolve(a).as_ref(), Object::Stream(_))), + _ => false, + } +} + /// The objects a widget's drawn appearance is built from, as the document holds them: the widget /// and the fields above it (value, `/DA`, `/Q`, flags, options, `/MK`, formatting actions), the /// form dictionary (`/DA`) and the fonts of its `/DR`, each with the objects its entries refer to diff --git a/crates/forms/src/read.rs b/crates/forms/src/read.rs index 4cb8da3b0f..27dbc2d224 100644 --- a/crates/forms/src/read.rs +++ b/crates/forms/src/read.rs @@ -93,7 +93,8 @@ fn qualified_name(doc: &Document, r: ObjRef) -> Option { if seen.len() > 64 || !seen.insert(c) { break; } - let obj = doc.get(c); + // An object that is only a reference to another is followed, as `Document::resolve` does. + let obj = doc.resolve(&Object::Ref(c)); let d = obj.as_dict()?; if let Some(t) = d.get(b"T").and_then(|o| text_of(&doc.resolve(o))) { parts.push(t); @@ -111,6 +112,38 @@ fn parent_of(doc: &Document, d: &Dict) -> Option { d.reference(b"Parent").filter(|p| doc.get(*p).as_dict().is_some()) } +/// The fields of the form's tree whose `/Kids` list one of `kids` and whose own fully qualified +/// name is `name`: a field listing kids that don't name it as their `/Parent`, and so share its +/// name rather than extend it. Each node is visited once, at most 64 levels down. +pub(crate) fn listing_fields_named(doc: &Document, name: &str, kids: &[ObjRef]) -> Vec { + let mut out = Vec::new(); + let Some(af) = acroform(doc) else { return out }; + let top = af.get(b"Fields").map(|f| doc.resolve(f)).and_then(|f| f.as_array().cloned()).unwrap_or_default(); + let mut stack: Vec<(ObjRef, usize)> = top.iter().filter_map(Object::as_ref).map(|r| (r, 0)).collect(); + let mut seen = std::collections::HashSet::new(); + while let Some((r, depth)) = stack.pop() { + if depth > 64 || !seen.insert(r) { + continue; + } + let obj = doc.get(r); + let Some(d) = obj.as_dict() else { continue }; + let list: Vec = d + .get(b"Kids") + .map(|k| doc.resolve(k)) + .and_then(|k| k.as_array().cloned()) + .unwrap_or_default() + .iter() + .filter_map(Object::as_ref) + .collect(); + if list.iter().any(|k| kids.contains(k)) && qualified_name(doc, r).as_deref() == Some(name) { + out.push(r); + } + let next = depth.saturating_add(1); + stack.extend(list.into_iter().filter(|k| doc.get(*k).as_dict().is_some_and(|kd| kd.contains(b"T"))).map(|k| (k, next))); + } + out +} + /// JavaScript streams one reading of the form (its fields, or its document scripts) may decode /// in all, however many actions name the same stream. pub(crate) const MAX_SCRIPT_BYTES: usize = 64 << 20; @@ -373,6 +406,19 @@ pub(crate) fn enumerate(doc: &Document) -> (Vec, usize) { values: DecodeBudget::new(MAX_VALUE_BYTES, 0), }; for r in listed_roots { + // A `/Fields` entry that is a nameless kid of a field `/Fields` doesn't list is read + // through that field, which names it (ISO 32000-2 §12.7.4.2). A kid a field already read + // has claimed as its widget is not read again as a nameless field; one its parent's + // `/Kids` omits is still read on its own, as before. + let nameless_kid = doc.get(r).as_dict().is_some_and(|d| !d.contains(b"T") && d.contains(b"Parent")); + if nameless_kid { + if let Some(root) = unlisted_root(doc, r, &seen.fields) { + walk(doc, root, &base, &page_of, &mut seen, &mut out, 0); + } + if seen.widgets.contains(&r) { + continue; + } + } walk(doc, r, &base, &page_of, &mut seen, &mut out, 0); } let listed = out.len(); @@ -520,15 +566,34 @@ fn walk( let kids: Vec = d.get(b"Kids").map(|k| doc.resolve(k)).and_then(|k| k.as_array().cloned()).unwrap_or_default().iter().filter_map(|k| k.as_ref()).collect(); // Kids with /T are fields; kids without are this field's widgets. - let (field_kids, kids): (Vec, Vec) = kids.into_iter().partition(|k| doc.get(*k).as_dict().is_some_and(|kd| kd.contains(b"T"))); + let (field_kids, mut kids): (Vec, Vec) = + kids.into_iter().partition(|k| doc.get(*k).as_dict().is_some_and(|kd| kd.contains(b"T"))); let field_kids: Vec = field_kids.into_iter().filter(|k| !not_a_child(doc, seen, r, *k)).collect(); if !field_kids.is_empty() { seen.path.push(r); for k in field_kids { - walk(doc, k, &inh, page_of, seen, out, depth + 1); + // A fully qualified name is built up the field's own `/Parent` chain (ISO 32000-2 + // §12.7.4.2). A kid that doesn't name this field as its `/Parent` is named from its + // own chain, as other readers name it; it still inherits this field's attributes. + // The `/Parent` as written: `qualified_name` follows an object that only refers on, + // and names a missing one as no parent. + let parent = doc.get(k).as_dict().and_then(|kd| kd.reference(b"Parent")); + if parent == Some(r) { + walk(doc, k, &inh, page_of, seen, out, depth + 1); + } else { + let mut own = inh.clone(); + own.name = parent.and_then(|p| qualified_name(doc, p)).unwrap_or_default(); + walk(doc, k, &own, page_of, seen, out, depth + 1); + } } seen.path.pop(); - return; + // "A field's children in the hierarchy may also include widget annotations" (ISO 32000-2 + // §12.7.4): widget annotations listed beside named kids are still this field's widgets, + // read as a field of this name next to the named kids. + kids.retain(|k| doc.get(*k).as_dict().is_some_and(|kd| kd.name(b"Subtype") == Some(b"Widget"))); + if kids.is_empty() { + return; + } } // A field of no kind this reads is skipped before it claims widgets or scripts, which a later // field listing them then reads. @@ -545,8 +610,16 @@ fn walk( }; // A child names its parent (ISO 32000-2 §12.7.4.1, Table 226): a page widget naming this field // is one of its widgets even when its `/Kids` leave it out, as MuPDF and PDFium read it. - let widget_refs: Vec = - if d.contains(b"Rect") { vec![r] } else { kids.into_iter().chain(seen.by_parent.get(&r).into_iter().flatten().copied()).collect() }; + // A field's widgets are its widget kids. The field dictionary is its own widget only when it + // has none (no kid whose `/Subtype` is `/Widget`): field and widget are merged "when a field has only a single associated widget + // annotation" (ISO 32000-2 §12.7.4), so a merged dictionary that also lists widget kids is a + // field whose buttons are those kids, not one more (often zero-area) button. + let has_widget_kids = kids.iter().any(|k| doc.get(*k).as_dict().is_some_and(|kd| kd.name(b"Subtype") == Some(b"Widget"))); + let widget_refs: Vec = if d.contains(b"Rect") && !has_widget_kids { + vec![r] + } else { + kids.into_iter().chain(seen.by_parent.get(&r).into_iter().flatten().copied()).collect() + }; let widget_refs: Vec = widget_refs.into_iter().filter(|w| seen.widgets.insert(*w)).collect(); let actions = actions_of(doc, d, &widget_refs, &mut seen.scripts); let button = (inh.ft.as_deref() == Some(b"Btn") && inh.ff.unwrap_or(0) & flags::PUSH_BUTTON != 0) diff --git a/crates/forms/src/tests.rs b/crates/forms/src/tests.rs index d1b94f361d..4a798ee8f0 100644 --- a/crates/forms/src/tests.rs +++ b/crates/forms/src/tests.rs @@ -2493,6 +2493,332 @@ fn fields_sharing_a_name_take_the_same_value() { } } +/// A radio group `Group1` whose two kids are fields of their own: each has `/T` (`kid_t`) and, +/// with `parent`, a `/Parent` naming the group. +fn radio_kids_with_names(kid_t: [&str; 2], parent: bool) -> Document { + let parent = if parent { " /Parent 4 0 R" } else { "" }; + let kid = |t: &str, state: &str, x: u32| { + format!( + "<< /Type /Annot /Subtype /Widget /FT /Btn /Ff 49152 /T ({t}){parent} /V /Off /AS /Off /Rect [{x} 700 {} 715] /P 3 0 R /AP << /N << /Off 7 0 R /{state} 7 0 R >> >> >>", + x + 15 + ) + }; + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R 6 0 R] >>".into(), // 3 + "<< /FT /Btn /Ff 49152 /T (Group1) /V /Choice2 /Kids [6 0 R 5 0 R] >>".into(), // 4 + kid(kid_t[0], "Choice1", 50), // 5 + kid(kid_t[1], "Choice2", 100), // 6 + "<< /Length 0 >>\nstream\n\nendstream".into(), // 7 + ]; + document(&objs) +} + +/// A field's fully qualified name is built from its own `/Parent` chain (ISO 32000-2 §12.7.4.2, +/// Table 226): a kid with `/T` that a field's `/Kids` lists but that has no `/Parent` naming +/// that field is named on its own, not under the field that lists it. +#[test] +fn a_named_kid_without_a_parent_keeps_its_own_name() { + let names = |doc: &Document| fields(doc).iter().map(|f| (f.name.clone(), f.obj.num)).collect::>(); + let same = radio_kids_with_names(["Group1", "Group1"], false); + assert_eq!(names(&same), [("Group1".to_string(), 6), ("Group1".to_string(), 5)]); + let other = radio_kids_with_names(["A", "B"], false); + assert_eq!(names(&other), [("B".to_string(), 6), ("A".to_string(), 5)]); + // With /Parent the kids are the group's children, as before. + let nested = radio_kids_with_names(["A", "B"], true); + assert_eq!(names(&nested), [("Group1.B".to_string(), 6), ("Group1.A".to_string(), 5)]); +} + +/// Fields with the same fully qualified name are one field (ISO 32000-2 §12.7.4.2): a radio +/// option that only a later one of them draws can be chosen by the name, and the others turn off. +#[test] +fn a_radio_option_of_any_field_sharing_the_name_can_be_chosen() { + for parent in [false, true] { + let name = if parent { "Group1.Group1" } else { "Group1" }; + let mut doc = radio_kids_with_names(["Group1", "Group1"], parent); + for (choice, on, off) in [("Choice1", 5, 6), ("Choice2", 6, 5), ("Choice1", 5, 6)] { + set_value(&mut doc, name, &FieldValue::Radio(Some(choice.into()))).unwrap_or_else(|e| panic!("{name} = {choice}: {e}")); + doc = reopen(&doc); + let st = |n: u32| doc.get(ObjRef { num: n, generation: 0 }).as_dict().and_then(|d| d.name(b"AS").map(<[u8]>::to_vec)); + assert_eq!((st(on), st(off)), (Some(choice.as_bytes().to_vec()), Some(b"Off".to_vec())), "{name} = {choice}"); + } + let err = set_value(&mut doc, name, &FieldValue::Radio(Some("Choice3".into()))).unwrap_err(); + assert!(err.to_string().contains("Choice3"), "{err}"); + } +} + +/// The same group as a broken save leaves it: the kids carry no `/P` (their page is found through +/// the page's `/Annots`), and both are drawn on with values that disagree (`/V /Off` with +/// `/AS /Choice1`, `/V /Choice2` with `/AS /Choice2`). Choosing by the name leaves exactly one +/// button on, across a save, whichever option it is, and gives every field of the name the +/// chosen value: both kids and the group listing them. +#[test] +fn a_radio_choice_by_the_name_turns_one_button_on_in_kids_without_a_page_that_disagree() { + let kid = |state: &str, v: &str, x: u32| { + format!( + "<< /Type /Annot /Subtype /Widget /FT /Btn /Ff 49152 /T (Group1) /V /{v} /AS /{state} /Rect [{x} 700 {} 715] /AP << /N << /Off 7 0 R /{state} 7 0 R >> >> >>", + x + 15 + ) + }; + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R] >> >>".into(), + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R 6 0 R] >>".into(), + "<< /FT /Btn /Ff 49152 /T (Group1) /V /Choice2 /Kids [6 0 R 5 0 R] >>".into(), + kid("Choice1", "Off", 50), + kid("Choice2", "Choice2", 100), + "<< /Length 0 >>\nstream\n\nendstream".into(), + ]; + let mut doc = document(&objs); + let st = |doc: &Document, n: u32| doc.get(ObjRef { num: n, generation: 0 }).as_dict().and_then(|d| d.name(b"AS").map(<[u8]>::to_vec)); + let all = fields(&doc); + assert_eq!(all.iter().map(|f| f.name.as_str()).collect::>(), ["Group1", "Group1"]); + assert!(all.iter().flat_map(|f| &f.widgets).all(|w| w.page == Some(0)), "found on the page without /P"); + assert_eq!((st(&doc, 5), st(&doc, 6)), (Some(b"Choice1".to_vec()), Some(b"Choice2".to_vec())), "both drawn on as read"); + for (choice, on, off) in [("Choice1", 5, 6), ("Choice2", 6, 5), ("Choice1", 5, 6)] { + set_value(&mut doc, "Group1", &FieldValue::Radio(Some(choice.into()))).unwrap_or_else(|e| panic!("Group1 = {choice}: {e}")); + doc = reopen(&doc); + assert_eq!((st(&doc, on), st(&doc, off)), (Some(choice.as_bytes().to_vec()), Some(b"Off".to_vec())), "Group1 = {choice}"); + // Every field named Group1 has the same value (ISO 32000-2 §12.7.4.2): both kids and the + // field listing them, whose /V was a stale Choice2. + let v = |n: u32| doc.get(ObjRef { num: n, generation: 0 }).as_dict().and_then(|d| d.name(b"V").map(<[u8]>::to_vec)); + assert_eq!( + [v(4), v(5), v(6)], + [Some(choice.as_bytes().to_vec()), Some(choice.as_bytes().to_vec()), Some(choice.as_bytes().to_vec())], + "Group1 = {choice}" + ); + let values: Vec> = fields(&doc).into_iter().map(|f| f.value).collect(); + assert_eq!(values, [vec![choice.to_string()], vec![choice.to_string()]], "Group1 = {choice}"); + } +} + +/// A radio group whose field dictionary is merged with a zero-area widget and also lists four +/// widget kids, as some form exporters write it: field and widget may be merged only "when a field +/// has only a single associated widget annotation" (ISO 32000-2 §12.7.4), so the kids are the +/// buttons. Two such groups: each lists its four options, and choosing one by the name turns on +/// exactly that button, across a save, and leaves the other group alone. +#[test] +fn a_field_merged_with_a_widget_that_lists_widget_kids_has_the_kids_as_buttons() { + const OPTIONS: [&str; 4] = ["Not#20Satisfied", "Somewhat", "Satisfied", "Very#20Satisfied"]; + let group = |t: &str, first: u32| { + let kids: Vec = (first..first + 4).map(|n| format!("{n} 0 R")).collect(); + format!("<< /Type /Annot /Subtype /Widget /Rect [0 0 0 0] /FT /Btn /Ff 49152 /Kids [{}] /V /Off /T ({t}) >>", kids.join(" ")) + }; + let kid = |parent: u32, i: usize, y: u32| { + let state = OPTIONS.get(i).copied().unwrap_or("X"); + let x = 100 + 100 * u32::try_from(i).unwrap_or(0); + format!( + "<< /Type /Annot /Subtype /Widget /F 4 /FT /Btn /Ff 49152 /Parent {parent} 0 R /Rect [{x} {y} {} {}] /P 3 0 R /AS /Off /AP << /N << /{state} 14 0 R /Off 14 0 R >> >> >>", + x + 12, + y + 12 + ) + }; + let mut objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R 5 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [6 0 R 7 0 R 8 0 R 9 0 R 10 0 R 11 0 R 12 0 R 13 0 R] >>".into(), // 3 + group("typeA15[0]", 6), // 4 + group("typeA15[1]", 10), // 5 + ]; + objs.extend((0..4).map(|i| kid(4, i, 600))); // 6-9 + objs.extend((0..4).map(|i| kid(5, i, 560))); // 10-13 + objs.push("<< /Length 0 >>\nstream\n\nendstream".into()); // 14 + let mut doc = document(&objs); + let all = fields(&doc); + assert_eq!(all.iter().map(|f| f.name.as_str()).collect::>(), ["typeA15[0]", "typeA15[1]"]); + let want: Vec = ["Not Satisfied", "Somewhat", "Satisfied", "Very Satisfied"].map(String::from).to_vec(); + for f in &all { + assert_eq!(f.widgets.iter().filter_map(|w| w.on_state.clone()).collect::>(), want, "{}", f.name); + assert!(f.widgets.iter().all(|w| w.page == Some(0) && w.rect[2] > w.rect[0]), "{}: the kids, on the page", f.name); + } + let st = |doc: &Document, n: u32| doc.get(ObjRef { num: n, generation: 0 }).as_dict().and_then(|d| d.name(b"AS").map(<[u8]>::to_vec)); + for (name, choice, on) in [("typeA15[0]", "Satisfied", 8), ("typeA15[1]", "Very Satisfied", 13), ("typeA15[0]", "Not Satisfied", 6)] { + set_value(&mut doc, name, &FieldValue::Radio(Some(choice.into()))).unwrap_or_else(|e| panic!("{name} = {choice}: {e}")); + doc = reopen(&doc); + let first = if name == "typeA15[0]" { 6 } else { 10 }; + for n in first..first + 4 { + let want = if n == on { choice.as_bytes().to_vec() } else { b"Off".to_vec() }; + assert_eq!(st(&doc, n), Some(want), "{name} = {choice}: widget {n}"); + } + } + assert_eq!(st(&doc, 13), Some(b"Very Satisfied".to_vec()), "the other group keeps its choice"); + let values: Vec> = fields(&doc).into_iter().map(|f| f.value).collect(); + assert_eq!(values, [vec!["Not Satisfied".to_string()], vec!["Very Satisfied".to_string()]]); +} + +/// Radio buttons named `2` listed straight in `/Fields`, some fields of their own (`/T (2)`, no +/// `/Parent`) and some nameless widgets of an unlisted field `2` whose `/Kids` mix both: "A +/// field's children in the hierarchy may also include widget annotations" (ISO 32000-2 §12.7.4). +/// Every option is listed under the name `2`, and each, including the nameless widgets', can be +/// chosen by the name, leaving exactly that button on across a save. +#[test] +fn nameless_widgets_of_an_unlisted_field_are_options_of_its_name() { + let named = |state: &str, x: u32| { + format!( + "<< /Type /Annot /Subtype /Widget /FT /Btn /Ff 49152 /T (2) /V /Off /AS /Off /Rect [{x} 700 {} 712] /P 3 0 R /AP << /N << /{state} 9 0 R /Off 9 0 R >> >> >>", + x + 12 + ) + }; + let nameless = |state: &str, x: u32| { + format!( + "<< /Type /Annot /Subtype /Widget /Parent 4 0 R /AS /Off /Rect [{x} 700 {} 712] /P 3 0 R /AP << /N << /{state} 9 0 R /Off 9 0 R >> >> >>", + x + 12 + ) + }; + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [5 0 R 6 0 R 7 0 R 8 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R 6 0 R 7 0 R 8 0 R] >>".into(), // 3 + "<< /FT /Btn /Ff 49152 /T (2) /Kids [5 0 R 6 0 R 7 0 R 8 0 R] >>".into(), // 4 unlisted + named("0", 50), // 5 + named("1", 100), // 6 + nameless("2", 150), // 7 + nameless("6", 200), // 8 + "<< /Length 0 >>\nstream\n\nendstream".into(), // 9 + ]; + let mut doc = document(&objs); + let all = fields(&doc); + assert!(all.iter().all(|f| f.name == "2"), "{:?}", all.iter().map(|f| &f.name).collect::>()); + let mut options: Vec = all.iter().flat_map(|f| &f.widgets).filter_map(|w| w.on_state.clone()).collect(); + options.sort(); + assert_eq!(options, ["0", "1", "2", "6"]); + let st = |doc: &Document, n: u32| doc.get(ObjRef { num: n, generation: 0 }).as_dict().and_then(|d| d.name(b"AS").map(<[u8]>::to_vec)); + for (choice, on) in [("6", 8), ("1", 6), ("2", 7), ("0", 5)] { + set_value(&mut doc, "2", &FieldValue::Radio(Some(choice.into()))).unwrap_or_else(|e| panic!("2 = {choice}: {e}")); + doc = reopen(&doc); + for n in 5..=8 { + let want = if n == on { choice.as_bytes().to_vec() } else { b"Off".to_vec() }; + assert_eq!(st(&doc, n), Some(want), "2 = {choice}: widget {n}"); + } + } +} + +/// A kid with `/T` that a field's `/Kids` lists but whose `/Parent` names a different field is +/// named up that field's chain (ISO 32000-2 §12.7.4.2), not under the field listing it. +#[test] +fn a_named_kid_whose_parent_is_another_field_is_named_under_it() { + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>".into(), // 3 + "<< /FT /Tx /T (Listing) /Kids [5 0 R] >>".into(), // 4 + "<< /Type /Annot /Subtype /Widget /T (k) /Parent 6 0 R /Rect [50 700 150 715] /P 3 0 R >>".into(), // 5 + "<< /T (Other) /Kids [5 0 R] >>".into(), // 6 unlisted + ]; + let names: Vec = fields(&document(&objs)).into_iter().map(|f| f.name).collect(); + assert_eq!(names, ["Other.k"]); +} + +/// A `/Parent` naming an object that is only a reference to the parent field is followed to it, +/// as document inspection follows it, so both name the kid the same. +#[test] +fn a_parent_given_through_an_object_that_is_a_reference_names_the_kid() { + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>".into(), // 3 + "<< /FT /Tx /T (Listing) /Kids [5 0 R] >>".into(), // 4 + "<< /Type /Annot /Subtype /Widget /T (k) /Parent 6 0 R /Rect [50 700 150 715] /P 3 0 R >>".into(), // 5 + "7 0 R".into(), // 6 + "<< /T (Other) /Kids [5 0 R] >>".into(), // 7 + ]; + let names: Vec = fields(&document(&objs)).into_iter().map(|f| f.name).collect(); + assert_eq!(names, ["Other.k"]); +} + +/// The two kids of [`radio_kids_with_names`] (`Group1`, no `/Parent`), with the one drawing +/// `Choice2` (6, read first) or the one drawing `Choice1` (5) read-only. +fn radio_kids_one_read_only(read_only: u32) -> Document { + let kid = |n: u32, state: &str, x: u32| { + let ff = if n == read_only { 49153 } else { 49152 }; + format!( + "<< /Type /Annot /Subtype /Widget /FT /Btn /Ff {ff} /T (Group1) /V /Off /AS /Off /Rect [{x} 700 {} 715] /P 3 0 R /AP << /N << /Off 7 0 R /{state} 7 0 R >> >> >>", + x + 15 + ) + }; + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R] >> >>".into(), + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R 6 0 R] >>".into(), + "<< /FT /Btn /Ff 49152 /T (Group1) /Kids [6 0 R 5 0 R] >>".into(), + kid(5, "Choice1", 50), + kid(6, "Choice2", 100), + "<< /Length 0 >> +stream + +endstream" + .into(), + ]; + document(&objs) +} + +/// A radio choice by the name reaches every radio field of that name (ISO 32000-2 §12.7.4.2), so +/// it is refused when any of them is read-only: the first one, even when a later one draws the +/// choice, or a later one, even when the first draws it. Nothing is written. +#[test] +fn a_radio_choice_by_the_name_is_refused_when_a_field_of_the_name_is_read_only() { + for (read_only, choice) in [(6, "Choice1"), (5, "Choice2")] { + let mut doc = radio_kids_one_read_only(read_only); + let state = |doc: &Document| { + [4, 5, 6].map(|n| { + doc.get(ObjRef { num: n, generation: 0 }).as_dict().map(|d| (d.name(b"V").map(<[u8]>::to_vec), d.name(b"AS").map(<[u8]>::to_vec))) + }) + }; + let before = state(&doc); + let err = set_value(&mut doc, "Group1", &FieldValue::Radio(Some(choice.into()))).unwrap_err(); + assert!(matches!(err, FormError::ReadOnly(_)), "{read_only} read-only, {choice}: {err}"); + assert_eq!(state(&doc), before, "{read_only} read-only, {choice}: nothing written"); + } +} + +/// A nameless widget listed in `/Fields` that the field before it already claimed is not read a +/// second time as a nameless field with no buttons. +#[test] +fn a_listed_widget_its_parent_claimed_is_read_once() { + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R 5 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>".into(), // 3 + "<< /FT /Tx /T (p) /Kids [5 0 R] >>".into(), // 4 + "<< /Type /Annot /Subtype /Widget /FT /Tx /Parent 4 0 R /Rect [50 700 150 715] /P 3 0 R >>".into(), // 5 + ]; + let all = fields(&document(&objs)); + assert_eq!(all.iter().map(|f| (f.name.as_str(), f.widgets.len())).collect::>(), [("p", 1)]); +} + +/// A nameless widget listed in `/Fields` whose `/Parent` doesn't list it in its `/Kids` is still +/// read, as it was before such widgets were read through their parent. +#[test] +fn a_listed_widget_its_parent_omits_is_still_read() { + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [5 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>".into(), // 3 + "<< /FT /Tx /T (p) /Kids [] >>".into(), // 4 unlisted + "<< /Type /Annot /Subtype /Widget /FT /Tx /Parent 4 0 R /Rect [50 700 150 715] /P 3 0 R >>".into(), // 5 + ]; + let all = fields(&document(&objs)); + let widgets: Vec = all.iter().flat_map(|f| &f.widgets).map(|w| w.obj.num).collect(); + assert_eq!(widgets, [5]); +} + +/// A field merged with its widget whose `/Kids` holds a dictionary that isn't a widget keeps +/// itself as its one widget: only widget kids replace it (ISO 32000-2 §12.7.4). +#[test] +fn a_merged_field_with_a_kid_that_is_not_a_widget_keeps_itself_as_widget() { + let objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [4 0 R] >> >>".into(), // 1 + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), // 2 + "<< /Type /Page /Parent 2 0 R /Annots [4 0 R] >>".into(), // 3 + "<< /Type /Annot /Subtype /Widget /FT /Tx /T (t) /Rect [50 700 150 715] /P 3 0 R /Kids [5 0 R] >>".into(), // 4 + "<< /Foo 1 >>".into(), // 5 + ]; + let all = fields(&document(&objs)); + let widgets: Vec<(u32, [f64; 4])> = all.iter().flat_map(|f| &f.widgets).map(|w| (w.obj.num, w.rect)).collect(); + assert_eq!(widgets, [(4, [50.0, 700.0, 150.0, 715.0])]); +} + #[test] fn locked_fields_only_take_unlocking() { let mut doc = one_page(); @@ -3761,6 +4087,55 @@ fn without_need_appearances_only_missing_appearances_are_drawn() { assert!(with_field_appearances(©).is_none(), "every widget has one now"); } +/// ISO 32000-2 12.5.5 and Table 170: a normal appearance is a stream or a subdictionary of +/// streams chosen by `/AS`. A form XObject's dictionary without its stream (`x`: object 6 says +/// it is one, but 7.5.7 keeps streams out of object streams, so a writer that packed it there +/// lost the data), a subdictionary with no `/AS` (`s`), or a state whose entry is no stream +/// (`u`, the push button `p`, the check box `k`) draws nothing, so these are drawn like widgets +/// without one: `x` with its `/MK /BC` border at its `/BS /W` of 3, given by reference. A +/// subdictionary whose `/AS` names a stream (`t`) is the file's own. +fn appearances_without_a_stream() -> Document { + let widget = |n: &str, rest: &str| format!("<< /Type /Annot /Subtype /Widget /T ({n}) /F 4 /P 3 0 R {rest} >>"); + let tx = "/FT /Tx /V (v) /DA (/Helv 12 Tf 0 g)"; + document(&[ + "<< /Type /Catalog /Pages 2 0 R /AcroForm 4 0 R >>".into(), + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(), + "<< /Type /Page /Parent 2 0 R /Annots [9 0 R 10 0 R 11 0 R 12 0 R 13 0 R 14 0 R] >>".into(), + "<< /Fields [9 0 R 10 0 R 11 0 R 12 0 R 13 0 R 14 0 R] /DA (/Helv 0 Tf 0 g) /DR << /Font << /Helv 5 0 R >> >> >>".into(), + "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>".into(), + "<< /Type /XObject /Subtype /Form /FormType 1 /BBox [0 0 106 14] >>".into(), // 6 + "<< /W 3 >>".into(), + "<< /Length 18 >>\nstream\nBT (old T) Tj ET\nendstream".into(), // 8 + widget("x", &format!("{tx} /Rect [54 705 160 719] /MK << /BC [0 0 0] /BG [0.88 0.88 0.88] >> /BS 7 0 R /AP << /N 6 0 R >>")), + widget("s", &format!("{tx} /Rect [54 680 160 694] /AP << /N << /On 8 0 R >> >>")), // 10 + widget("t", &format!("{tx} /Rect [54 655 160 669] /AS /On /AP << /N << /On 8 0 R >> >>")), + widget("u", &format!("{tx} /Rect [54 630 160 644] /AS /On /AP << /N << /On 6 0 R >> >>")), + widget("p", "/FT /Btn /Ff 65536 /DA (/Helv 12 Tf 0 g) /MK << /CA (Go) /BC [0 0 0] >> /Rect [54 600 160 620] /AP << /N 6 0 R >>"), + widget( + "k", + "/FT /Btn /V /Yes /AS /Yes /DA (/ZaDb 0 Tf 0 g) /MK << /CA (4) /BC [0 0 0] >> /Rect [200 700 224 724] /AP << /N << /Yes 6 0 R /Off 8 0 R >> >>", + ), + ]) +} + +#[test] +fn normal_appearances_that_give_no_stream_are_drawn() { + let doc = appearances_without_a_stream(); + let copy = with_field_appearances(&doc).expect("x, s, u, p and k draw nothing of their own"); + let all = fields(©); + let x = ap(©, &field(&all, "x").widgets[0]); + assert!(x.contains("0.88 0.88 0.88 rg\n0 0 106 14 re f") && x.contains("0 0 0 RG\n3 w\n1.5 1.5 103 11 re S"), "{x}"); + assert!(x.contains("(v) Tj"), "{x}"); + for name in ["s", "u"] { + assert!(ap(©, &field(&all, name).widgets[0]).contains("(v) Tj"), "{name}"); + } + assert!(ap(©, &field(&all, "p").widgets[0]).contains("(Go) Tj")); + assert!(state_ap(©, &field(&all, "k").widgets[0], "Yes").contains("(4) Tj")); + let t = |d: &Document| d.get(field(&fields(d), "t").widgets[0].obj).as_dict().and_then(|w| w.get(b"AP").cloned()); + assert_eq!(t(©), t(&doc), "a state with a stream is the file's own"); + assert!(with_field_appearances(©).is_none(), "every widget has a stream now"); +} + /// #3470: refreshing the display copy redraws only the widgets whose inputs changed. Typing in /// one field of a form that leaves every appearance to the viewer otherwise redrew all of them. #[test] diff --git a/crates/forms/src/values.rs b/crates/forms/src/values.rs index 0981404bcc..46538d5e39 100644 --- a/crates/forms/src/values.rs +++ b/crates/forms/src/values.rs @@ -1,5 +1,5 @@ use super::appearances::{declares_pdfa, need_appearances}; -use super::read::{acroform, field_flags, fields, page_refs}; +use super::read::{acroform, field_flags, fields, listing_fields_named, page_refs}; use super::{ Dict, Document, Field, FieldEvent, FieldKind, FieldValue, FormError, NoScripts, ObjRef, Object, PdfString, Scripts, Widget, af, appearance, author, calc_order, flags, invalid, name_obj, scripting, @@ -89,8 +89,23 @@ impl NamedFields { /// Write `value` into the field `name`, without recalculating: `named` are the positions in /// `all` (the form as read before this write) of the fields with that name, in field order. fn fill(doc: &mut Document, all: &[Field], named: &[usize], name: &str, value: &FieldValue, scripts: &mut dyn Scripts) -> Result<(), FormError> { - let f = named.first().and_then(|&i| all.get(i)).ok_or_else(|| FormError::NoSuchField(name.into()))?; - if f.read_only() { + let mut same = named.iter().filter_map(|&i| all.get(i)); + let first = same.clone().next().ok_or_else(|| FormError::NoSuchField(name.into()))?; + // Fields with the same fully qualified name are one field (ISO 32000-2 §12.7.4.2), so a radio + // option any of them draws is an option of the name: it is written through the first one + // that draws it, and the others turn off below. + let choice = match value { + FieldValue::Radio(Some(c)) | FieldValue::Text(c) => Some(c.as_str()), + _ => None, + }; + let f = match choice { + Some(c) if first.kind == FieldKind::Radio => same.find(|t| t.kind == FieldKind::Radio && t.state_for(c).is_some()).unwrap_or(first), + _ => first, + }; + // The name is read-only when the first field of it is, whichever field draws the choice; and + // a radio choice reaches every radio field of the name, so none of them may be read-only. + let radio_locked = f.kind == FieldKind::Radio && named.iter().filter_map(|&i| all.get(i)).any(|t| t.kind == FieldKind::Radio && t.read_only()); + if f.read_only() || first.read_only() || radio_locked { return Err(FormError::ReadOnly(name.into())); } write_value(doc, f, value, scripts)?; @@ -105,6 +120,16 @@ fn fill(doc: &mut Document, all: &[Field], named: &[usize], name: &str, value: & share_value(doc, f.obj, t, &taken, scripts)?; } } + // A field whose `/Kids` lists radio fields that don't name it as their `/Parent` has their + // name too, so it takes the same value (§12.7.4.2) rather than keep a stale choice. + if f.kind == FieldKind::Radio { + let objs: Vec = named.iter().filter_map(|&i| all.get(i)).map(|t| t.obj).collect(); + if let Some(v) = doc.get(f.obj).as_dict().and_then(|d| d.get(b"V").cloned()) { + for p in listing_fields_named(doc, name, &objs) { + doc.update_dict(p, |d| d.set(b"V".to_vec(), v.clone()))?; + } + } + } Ok(()) } @@ -118,7 +143,13 @@ fn share_value(doc: &mut Document, source: ObjRef, twin: &Field, value: &[String } FieldKind::Radio => { let choice = value.first().filter(|c| twin.widgets.iter().any(|w| w.on_state.as_deref() == Some(c.as_str()))); - set_states(doc, twin, choice.map(String::as_str)) + set_states(doc, twin, choice.map(String::as_str))?; + // The same value (§12.7.4.2), even when none of the twin's buttons draws it: they all + // stay off, and only the field that draws the choice shows it. + match value.first() { + Some(v) if choice.is_none() => Ok(doc.update_dict(twin.obj, |d| d.set(b"V".to_vec(), name_obj(v)))?), + _ => Ok(()), + } } FieldKind::Text | FieldKind::Combo | FieldKind::List => { let src = doc.get(source).as_dict().cloned().unwrap_or_default(); diff --git a/crates/measure/README.md b/crates/measure/README.md index 95a04a2b64..2e438d7a1e 100644 --- a/crates/measure/README.md +++ b/crates/measure/README.md @@ -8,7 +8,7 @@ Scale::calibrate derives a conversion from two points and a known real-world dis The desktop Measure objects panel exposes all three tools, live values and coordinate deltas, 45-degree constraints, vertex cancellation, snapping, calibration and CSV export. Automation provides measure_distance, measure_perimeter, measure_area, measure_info, measure_list, measure_scale, measure_snap and measure_export. The panel's drawing scale and measure_scale's `page_distance`/`page_unit` give the page side in a unit, by default the real-world one. Tool coordinates and explicit units-per-point scales use physical display points. -Snapping interprets painted paths and nested Form XObjects, including graphics-state transforms. Cubic paths are flattened to a maximum deviation of 0.2 user units. Raster edges, glyph outlines, clipping and optional-content visibility are not interpreted. Extraction stops at 20,000 segments, 40,000 endpoint and midpoint targets, 16 MiB of decoded content, 4,096 streams, 32 nested forms or 256 nested `q`; intersection searches use at most 256 nearby segments. Extraction is lenient: a content stream that can't be decoded is skipped and counted, and the rest of the page still snaps. The UI and automation report these limits. +Snapping interprets painted paths and nested Form XObjects, including graphics-state transforms. Cubic paths are flattened to a maximum deviation of 0.2 user units. Raster edges, glyph outlines, clipping and optional-content visibility are not interpreted. Extraction stops at 8,000,000 segments, 24,000,000 endpoint and midpoint targets, 512 MiB of decoded content, 4,096 streams, 32 nested forms or 256 nested `q`; intersection searches use at most 256 nearby segments. Extraction is lenient: a content stream that can't be decoded is skipped and counted, and the rest of the page still snaps. The UI and automation report these limits. Imported single-unit decimal rectilinear scales support differing X and Y conversion factors when CYX is present. Unit labels are text strings and may be non-ASCII ("m²"). Compound units (such as feet and inches), fraction formats and non-rectilinear scales are not supported: `list` reports such measurements, and ones with invalid geometry, under `unsupported` with a reason instead of failing, and leaves their PDF data untouched. A polygon whose last vertex repeats the first is treated as closed. Interoperability with Acrobat-authored measurements is untested. Geospatial and 3D measuring are outside this component. diff --git a/crates/measure/examples/snap_timing.rs b/crates/measure/examples/snap_timing.rs new file mode 100644 index 0000000000..319434f8cd --- /dev/null +++ b/crates/measure/examples/snap_timing.rs @@ -0,0 +1,32 @@ +//! Times extracting a page's snapping geometry and snapping at points on it. +//! +//! Usage: `snap_timing FILE PAGE` (PAGE is 1-based). + +use std::time::Instant; + +fn main() -> Result<(), Box> { + let args: Vec = std::env::args().collect(); + let (Some(path), Some(page)) = (args.get(1), args.get(2)) else { + return Err("usage: snap_timing FILE PAGE".into()); + }; + let page: usize = page.parse::()?.checked_sub(1).ok_or("pages are numbered from 1")?; + let bytes = std::sync::Arc::new(std::fs::read(path)?); + let doc = pdfcraft_cos::Document::open(bytes).map_err(|e| format!("{e:?}"))?; + let start = Instant::now(); + let g = pdfcraft_measure::snap::geometry(&doc, page)?; + let extracted = start.elapsed(); + println!("extracted {} segments in {} ms: truncated {} unreadable {}", g.segment_count(), extracted.as_millis(), g.truncated, g.unreadable); + // Snaps at points spread over the drawing, with a 6-unit tolerance (a few pixels on screen). + let points: Vec<[f64; 2]> = + g.segments().step_by((g.segment_count() / 2000).max(1)).map(|[a, b]| [(a[0] + b[0]) / 2.0 + 0.7, (a[1] + b[1]) / 2.0 - 0.4]).collect(); + let start = Instant::now(); + let mut hits = 0; + for &p in &points { + if g.snap(p, 6.0, pdfcraft_measure::snap::SnapOptions::default())?.is_some() { + hits += 1; + } + } + let per = start.elapsed().as_secs_f64() * 1e6 / points.len().max(1) as f64; + println!("{} snaps, {hits} hits, {per:.0} us per snap", points.len()); + Ok(()) +} diff --git a/crates/measure/src/snap.rs b/crates/measure/src/snap.rs index c39d811d9f..0554a1f017 100644 --- a/crates/measure/src/snap.rs +++ b/crates/measure/src/snap.rs @@ -5,19 +5,50 @@ use pdfcraft_cos::{DecodeBudget, Dict, Document, ObjRef, Object}; use serde::Serialize; use std::collections::HashSet; -const MAX_SEGMENTS: usize = 20_000; -/// Endpoints and midpoints together, across the whole page. -const MAX_TARGETS: usize = 2 * MAX_SEGMENTS; -const MAX_BYTES: usize = 16 * 1024 * 1024; +/// Open subpaths remembered for a fill, which closes each of them. +const MAX_OPEN_SUBPATHS: usize = Limits::DEFAULT.segments; +/// How much of a page extraction reads and keeps. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Limits { + /// Straight segments (curves are split into several). + pub segments: usize, + /// Endpoints and midpoints together, across the whole page. + pub targets: usize, + /// Decoded content bytes, across every stream read. + pub bytes: usize, +} +impl Limits { + /// Enough for heavy CAD sheets (several million segments), and + /// the renderer's own content limit. At about 40 bytes a segment, the cap holds ~320 MB, plus + /// up to [`MAX_INDEX_ENTRIES`] in the index. + pub const DEFAULT: Self = Self { segments: 8_000_000, targets: 24_000_000, bytes: 512 << 20 }; + /// What a window can read without missing frames that matter. + pub const WITHIN_A_FRAME: Self = Self { segments: 20_000, targets: 40_000, bytes: 16 << 20 }; + /// Within what the index can number. + fn indexable(self) -> Self { + let most = u32::MAX as usize / 2; + Self { segments: self.segments.min(most), targets: self.targets.min(most), ..self } + } +} +impl Default for Limits { + fn default() -> Self { + Self::DEFAULT + } +} +/// The page's drawn geometry in user space, compactly (`f32`: about 0.0003 units of precision at +/// A0 sizes), with a grid index so that a snap looks only near the pointer. #[derive(Clone, Debug, Default)] pub struct Geometry { - pub segments: Vec<[Point; 2]>, - pub endpoints: Vec, - pub midpoints: Vec, + /// Segments as `[ax, ay, bx, by]`. + segs: Vec<[f32; 4]>, + /// Snap targets: a straight edge's or curve's two ends, then its midpoint. + tgts: Vec<[f32; 6]>, /// A limit stopped extraction early (segments, targets, bytes, streams, nesting). pub truncated: bool, /// Content streams that couldn't be decoded and were skipped. pub unreadable: usize, + limits: Limits, + grid: Grid, } #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] #[serde(rename_all = "lowercase")] @@ -80,7 +111,7 @@ fn remember_close(closures: &mut Vec<[Point; 2]>, current: Option, start: if let (Some(a), Some(b)) = (current, start) && distance(a, b) > 1e-9 { - if closures.len() >= MAX_SEGMENTS { + if closures.len() >= MAX_OPEN_SUBPATHS { return false; } closures.push([a, b]); @@ -88,10 +119,45 @@ fn remember_close(closures: &mut Vec<[Point; 2]>, current: Option, start: true } +fn seg(s: &[f32; 4]) -> [Point; 2] { + [[f64::from(s[0]), f64::from(s[1])], [f64::from(s[2]), f64::from(s[3])]] +} +fn pt(x: f32, y: f32) -> Point { + [f64::from(x), f64::from(y)] +} impl Geometry { + fn with_limits(limits: Limits) -> Self { + Self { limits, ..Self::default() } + } + /// How many segments were kept. + pub fn segment_count(&self) -> usize { + self.segs.len() + } + /// The segments, in drawing order. + pub fn segments(&self) -> impl ExactSizeIterator + '_ { + self.segs.iter().map(seg) + } + /// Snap endpoints: each edge's or curve's two ends, in drawing order. + pub fn endpoints(&self) -> impl Iterator + '_ { + self.tgts.iter().flat_map(|t| [pt(t[0], t[1]), pt(t[2], t[3])]) + } + /// Snap midpoints: each edge's or curve's middle, in drawing order. + pub fn midpoints(&self) -> impl Iterator + '_ { + self.tgts.iter().map(|t| pt(t[4], t[5])) + } /// Dense intersection searches are capped independently from path extraction. pub fn intersection_limited(&self, at: Point, tolerance: f64) -> bool { - self.segments.iter().filter(|[a, b]| distance(at, nearest(at, *a, *b)) <= tolerance).take(257).count() > 256 + if tolerance.is_nan() || tolerance < 0.0 { + return false; + } + let (segs, _) = self.grid.near(self, at, tolerance); + segs.iter() + .filter_map(|&i| self.segs.get(i as usize)) + .map(seg) + .filter(|[a, b]| distance(at, nearest(at, *a, *b)) <= tolerance) + .take(257) + .count() + > 256 } pub fn snap(&self, at: Point, tolerance: f64, options: SnapOptions) -> Result> { check_points(&[at])?; @@ -106,19 +172,25 @@ impl Geometry { *best = Some(Snap { point: p, kind, distance }); } }; + // Only what the grid finds near the pointer: everything within `tolerance` is there, in + // drawing order, so the result is the one a scan of the whole page would give. + let (segs, tgts) = self.grid.near(self, at, tolerance); + let targets: Vec<&[f32; 6]> = tgts.iter().filter_map(|&i| self.tgts.get(i as usize)).collect(); if options.endpoints { - for &p in &self.endpoints { - consider(&mut best, p, SnapKind::Endpoint); + for t in &targets { + consider(&mut best, pt(t[0], t[1]), SnapKind::Endpoint); + consider(&mut best, pt(t[2], t[3]), SnapKind::Endpoint); } } if options.midpoints { - for &p in &self.midpoints { - consider(&mut best, p, SnapKind::Midpoint); + for t in &targets { + consider(&mut best, pt(t[4], t[5]), SnapKind::Midpoint); } } + let near: Vec<[Point; 2]> = segs.iter().filter_map(|&i| self.segs.get(i as usize)).map(seg).collect(); // Only segments in the pointer's neighbourhood participate in intersection tests. // Cap the local quadratic work even on a malicious densely overlapping drawing. - let local: Vec<_> = self.segments.iter().filter(|[a, b]| distance(at, nearest(at, *a, *b)) <= tolerance).take(256).collect(); + let local: Vec<_> = near.iter().filter(|[a, b]| distance(at, nearest(at, *a, *b)) <= tolerance).take(256).collect(); if options.intersections { for (i, s) in local.iter().enumerate() { for t in local.iter().skip(i + 1) { @@ -129,35 +201,36 @@ impl Geometry { } } if options.paths && best.is_none() { - for [a, b] in &self.segments { + for [a, b] in &near { consider(&mut best, nearest(at, *a, *b), SnapKind::Path); } } Ok(best) } + /// Endpoints and midpoints together. fn targets(&self) -> usize { - self.endpoints.len().saturating_add(self.midpoints.len()) + self.tgts.len().saturating_mul(3) } /// Whether the segment was accepted (finite, non-degenerate and within the cap). fn segment(&mut self, a: Point, b: Point) -> bool { - if self.segments.len() >= MAX_SEGMENTS { + if self.segs.len() >= self.limits.segments { self.truncated = true; return false; } if check_points(&[a, b]).is_ok() && distance(a, b) > 1e-9 { - self.segments.push([a, b]); + self.segs.push([a[0] as f32, a[1] as f32, b[0] as f32, b[1] as f32]); return true; } false } /// Snap targets are recorded only for accepted geometry, and are capped. fn target(&mut self, ends: [Point; 2], mid: Point) { - if self.targets() >= MAX_TARGETS { + if self.targets().saturating_add(3) > self.limits.targets { self.truncated = true; return; } - self.endpoints.extend(ends); - self.midpoints.push(mid); + let [a, b] = ends; + self.tgts.push([a[0] as f32, a[1] as f32, b[0] as f32, b[1] as f32, mid[0] as f32, mid[1] as f32]); } fn edge(&mut self, a: Point, b: Point) { if self.segment(a, b) { @@ -166,7 +239,7 @@ impl Geometry { } fn curve(&mut self, points: [Point; 4], depth: usize) { let [a, b, c, d] = points; - if self.segments.len() >= MAX_SEGMENTS { + if self.segs.len() >= self.limits.segments { self.truncated = true; return; } @@ -203,6 +276,7 @@ struct Walker<'a> { /// expands and then shrinks pays for the expansion each time it is drawn. budget: DecodeBudget, streams: usize, + limits: Limits, } impl Walker<'_> { fn walk(&mut self, object: &Object, resources: &Dict, initial: Matrix, depth: usize) -> Result<()> { @@ -261,7 +335,7 @@ impl Walker<'_> { let mut start = None; let mut current = None; let mut closures = Vec::new(); - let mut path = Geometry::default(); + let mut path = Geometry::with_limits(self.limits); // Read on the first `Do` and kept: resolving a direct dictionary copies it. let mut xobjects = None; let point = |m: Matrix, x: f64, y: f64| { @@ -270,8 +344,8 @@ impl Walker<'_> { }; // One operator at a time: a stream of millions of short operators is never held whole. for op in ops(data) { - if self.geometry.segments.len().saturating_add(path.segments.len()) >= MAX_SEGMENTS - || self.geometry.targets().saturating_add(path.targets()) >= MAX_TARGETS + if self.geometry.segs.len().saturating_add(path.segs.len()) >= self.limits.segments + || self.geometry.targets().saturating_add(path.targets()) >= self.limits.targets { self.geometry.truncated = true; break; @@ -327,9 +401,9 @@ impl Walker<'_> { if let Some([a, b, c, d]) = curve && check_points(&[a, b, c, d]).is_ok() { - let before = path.segments.len(); + let before = path.segs.len(); path.curve([a, b, c, d], 0); - if path.segments.len() > before { + if path.segs.len() > before { path.target([a, d], [(a[0] + 3.0 * b[0] + 3.0 * c[0] + d[0]) / 8.0, (a[1] + 3.0 * b[1] + 3.0 * c[1] + d[1]) / 8.0]); } current = Some(d); @@ -369,8 +443,8 @@ impl Walker<'_> { // Filling implicitly closes every open subpath, not just the last one. // Stroke-only S leaves its open subpaths unchanged. for [a, b] in closures.drain(..) { - if self.geometry.segments.len().saturating_add(path.segments.len()) >= MAX_SEGMENTS - || self.geometry.targets().saturating_add(path.targets()) >= MAX_TARGETS + if self.geometry.segs.len().saturating_add(path.segs.len()) >= self.limits.segments + || self.geometry.targets().saturating_add(path.targets()) >= self.limits.targets { path.truncated = true; break; @@ -382,9 +456,8 @@ impl Walker<'_> { { path.edge(a, b); } - self.geometry.segments.append(&mut path.segments); - self.geometry.endpoints.append(&mut path.endpoints); - self.geometry.midpoints.append(&mut path.midpoints); + self.geometry.segs.append(&mut path.segs); + self.geometry.tgts.append(&mut path.tgts); self.geometry.truncated |= path.truncated; closures.clear(); current = None; @@ -392,7 +465,7 @@ impl Walker<'_> { } "n" => { self.geometry.truncated |= path.truncated; - path = Geometry::default(); + path = Geometry::with_limits(self.limits); closures.clear(); current = None; start = None; @@ -450,16 +523,296 @@ thread_local! { pub(crate) static XOBJECT_DICT_READS: std::cell::Cell = const { std::cell::Cell::new(0) }; } +/// The page's geometry for snapping, within [`Limits::DEFAULT`]. pub fn geometry(doc: &Document, page_index: usize) -> Result { + geometry_with_limits(doc, page_index, Limits::DEFAULT) +} + +/// The page's geometry for snapping, within `limits`. +pub fn geometry_with_limits(doc: &Document, page_index: usize, limits: Limits) -> Result { + let limits = limits.indexable(); let p = crate::page(doc, page_index)?; let resources = p.dict.get(b"Resources").map(|o| doc.resolve(o)); let empty = Dict::new(); let resources = resources.as_ref().and_then(|o| o.as_dict()).unwrap_or(&empty); - let mut walker = - Walker { doc, geometry: Geometry::default(), seen: HashSet::new(), budget: DecodeBudget::new(MAX_BYTES, usize::MAX), streams: 0 }; + let mut walker = Walker { + doc, + geometry: Geometry::with_limits(limits), + seen: HashSet::new(), + budget: DecodeBudget::new(limits.bytes, usize::MAX), + streams: 0, + limits, + }; // Extraction is lenient: limits and unreadable streams are reported in the geometry. if let Some(contents) = p.dict.get(b"Contents") { walker.walk(contents, resources, Matrix::IDENTITY, 0)?; } - Ok(walker.geometry) + let mut geometry = walker.geometry; + geometry.segs.shrink_to_fit(); + geometry.tgts.shrink_to_fit(); + geometry.grid = Grid::build(&geometry, MAX_INDEX_ENTRIES); + Ok(geometry) +} + +/// The most cells a side of the grid has. +const GRID_SIDE: usize = 1024; +/// A segment whose box covers more cells than this is kept in one list checked at every snap +/// (a long line would otherwise fill a whole row of cells). +const WIDE_CELLS: usize = 64; +/// The most cells one snap reads; a larger search area (a huge tolerance) scans everything. +const MAX_QUERY_CELLS: usize = 4096; +/// The most entries the segment lists of the cells hold (4 bytes each). Past it, shorter segments +/// go in the list checked at every snap: snaps get slower, and find the same. +const MAX_INDEX_ENTRIES: u64 = 64_000_000; + +/// A uniform grid over the geometry's box: per cell, the segments and targets touching it. +#[derive(Clone, Debug, Default)] +struct Grid { + x0: f32, + y0: f32, + cell: f32, + cols: usize, + rows: usize, + /// Per cell, where its segment indices start in `seg_items` (one more entry than cells). + seg_start: Vec, + seg_items: Vec, + tgt_start: Vec, + tgt_items: Vec, + /// Segments too long for the cells. + wide: Vec, +} + +impl Grid { + fn build(g: &Geometry, max_entries: u64) -> Self { + let mut bounds: Option<[f32; 4]> = None; + let mut grow = |x: f32, y: f32| { + let b = bounds.get_or_insert([x, y, x, y]); + *b = [b[0].min(x), b[1].min(y), b[2].max(x), b[3].max(y)]; + }; + for s in &g.segs { + grow(s[0], s[1]); + grow(s[2], s[3]); + } + for t in &g.tgts { + grow(t[0], t[1]); + grow(t[2], t[3]); + grow(t[4], t[5]); + } + let Some([x0, y0, x1, y1]) = bounds else { return Self::default() }; + let items = g.segs.len().max(g.tgts.len()).max(1); + // About eight items a cell. + let side = ((items as f64 / 8.0).sqrt().ceil() as usize).clamp(1, GRID_SIDE); + let extent = (x1 - x0).max(y1 - y0).max(1e-3); + let cell = extent / side as f32; + let cols = (((x1 - x0) / cell) as usize + 1).min(GRID_SIDE); + let rows = (((y1 - y0) / cell) as usize + 1).min(GRID_SIDE); + let mut grid = Self { x0, y0, cell, cols, rows, ..Self::default() }; + let cells = cols * rows; + // Segments: count, then fill (compressed rows). + let seg_cells = |s: &[f32; 4]| grid.range(s[0].min(s[2]), s[1].min(s[3]), s[0].max(s[2]), s[1].max(s[3])); + let covered = |s: &[f32; 4]| { + let (c0, r0, c1, r1) = seg_cells(s); + (c1 - c0 + 1) * (r1 - r0 + 1) + }; + // A segment of one cell adds one entry, so the last step fits whatever the sheet. + let most_cells = [WIDE_CELLS, 16, 4] + .into_iter() + .find(|&most| g.segs.iter().map(covered).filter(|&n| n <= most).map(|n| n as u64).sum::() <= max_entries) + .unwrap_or(1); + let mut counts = vec![0u32; cells]; + let mut wide = Vec::new(); + for (i, s) in g.segs.iter().enumerate() { + let (c0, r0, c1, r1) = seg_cells(s); + if (c1 - c0 + 1) * (r1 - r0 + 1) > most_cells { + wide.push(i as u32); + continue; + } + for r in r0..=r1 { + for c in c0..=c1 { + counts[r * cols + c] += 1; + } + } + } + let (seg_start, mut fill) = Self::starts(&counts); + let mut seg_items = vec![0u32; seg_start.last().copied().unwrap_or(0) as usize]; + for (i, s) in g.segs.iter().enumerate() { + let (c0, r0, c1, r1) = seg_cells(s); + if (c1 - c0 + 1) * (r1 - r0 + 1) > most_cells { + continue; + } + for r in r0..=r1 { + for c in c0..=c1 { + let k = r * cols + c; + seg_items[fill[k] as usize] = i as u32; + fill[k] += 1; + } + } + } + // Targets: the cells of their three points. + let tgt_cells = |t: &[f32; 6]| { + let mut cs = [grid.cell_of(t[0], t[1]), grid.cell_of(t[2], t[3]), grid.cell_of(t[4], t[5])]; + cs.sort_unstable(); + cs + }; + let mut counts = vec![0u32; cells]; + for t in &g.tgts { + let cs = tgt_cells(t); + for (j, &k) in cs.iter().enumerate() { + if j == 0 || cs[j - 1] != k { + counts[k] += 1; + } + } + } + let (tgt_start, mut fill) = Self::starts(&counts); + let mut tgt_items = vec![0u32; tgt_start.last().copied().unwrap_or(0) as usize]; + for (i, t) in g.tgts.iter().enumerate() { + let cs = tgt_cells(t); + for (j, &k) in cs.iter().enumerate() { + if j == 0 || cs[j - 1] != k { + tgt_items[fill[k] as usize] = i as u32; + fill[k] += 1; + } + } + } + grid.seg_start = seg_start; + grid.seg_items = seg_items; + grid.tgt_start = tgt_start; + grid.tgt_items = tgt_items; + grid.wide = wide; + grid + } + + /// Prefix sums of `counts` (the starts), and a copy to fill from. + fn starts(counts: &[u32]) -> (Vec, Vec) { + let mut start = Vec::with_capacity(counts.len() + 1); + let mut total = 0u32; + start.push(0); + for &c in counts { + total = total.saturating_add(c); + start.push(total); + } + let fill = start[..counts.len()].to_vec(); + (start, fill) + } + + fn col(&self, x: f32) -> usize { + (((x - self.x0) / self.cell).max(0.0) as usize).min(self.cols.saturating_sub(1)) + } + fn row(&self, y: f32) -> usize { + (((y - self.y0) / self.cell).max(0.0) as usize).min(self.rows.saturating_sub(1)) + } + fn cell_of(&self, x: f32, y: f32) -> usize { + self.row(y) * self.cols + self.col(x) + } + /// The cells (first and last column, first and last row) a box covers. + fn range(&self, x0: f32, y0: f32, x1: f32, y1: f32) -> (usize, usize, usize, usize) { + (self.col(x0), self.row(y0), self.col(x1), self.row(y1)) + } + + /// The segments and targets near `at` (all within `tolerance` of it, and maybe more), each + /// list in drawing order without repeats. + fn near(&self, g: &Geometry, at: Point, tolerance: f64) -> (Vec, Vec) { + let all = || ((0..g.segs.len() as u32).collect(), (0..g.tgts.len() as u32).collect()); + if self.cols == 0 || self.rows == 0 { + return if g.segs.is_empty() && g.tgts.is_empty() { (Vec::new(), Vec::new()) } else { all() }; + } + // Rounding to `f32` can stop the box short of a candidate's cell: this much wider can't. + let t = (tolerance + (at[0].abs().max(at[1].abs()) + tolerance) * 2.0 * f64::from(f32::EPSILON)) as f32; + let (x, y) = (at[0] as f32, at[1] as f32); + if !(x.is_finite() && y.is_finite() && t.is_finite()) { + return all(); + } + let (c0, r0, c1, r1) = self.range(x - t, y - t, x + t, y + t); + // Entirely outside the grid: nothing is near (except wide segments). + let outside = + x + t < self.x0 || y + t < self.y0 || x - t > self.x0 + self.cell * self.cols as f32 || y - t > self.y0 + self.cell * self.rows as f32; + if (c1 - c0 + 1) * (r1 - r0 + 1) > MAX_QUERY_CELLS { + return all(); + } + let mut segs: Vec = self.wide.clone(); + let mut tgts: Vec = Vec::new(); + if !outside { + for r in r0..=r1 { + for c in c0..=c1 { + let k = r * self.cols + c; + let range = |start: &[u32]| (start[k] as usize)..(start[k + 1] as usize); + segs.extend_from_slice(self.seg_items.get(range(&self.seg_start)).unwrap_or(&[])); + tgts.extend_from_slice(self.tgt_items.get(range(&self.tgt_start)).unwrap_or(&[])); + } + } + } + segs.sort_unstable(); + segs.dedup(); + tgts.sort_unstable(); + tgts.dedup(); + (segs, tgts) + } +} + +#[cfg(test)] +mod grid_tests { + use super::*; + + fn geometry(segs: Vec<[f32; 4]>, tgts: Vec<[f32; 6]>, max_entries: u64) -> Geometry { + let mut g = Geometry { segs, tgts, ..Geometry::default() }; + g.grid = Grid::build(&g, max_entries); + g + } + + /// Past the allowance the index holds fewer entries and finds the same. + #[test] + fn a_bounded_index_finds_what_a_full_one_does() { + let mut segs = Vec::new(); + for i in 0..4000u16 { + let (x, y) = (f32::from(i % 64) * 15.0, f32::from(i / 64) * 15.0); + segs.push([x, y, x + 37.0, y + 41.0]); + } + let tgts = Vec::new(); + let full = geometry(segs.clone(), tgts.clone(), u64::MAX); + let budget = segs.len() as u64; + let bounded = geometry(segs, tgts, budget); + assert!(full.grid.seg_items.len() as u64 > budget, "the sheet needs more than the allowance"); + assert!(bounded.grid.seg_items.len() as u64 <= budget, "{} entries", bounded.grid.seg_items.len()); + let options = SnapOptions { endpoints: false, midpoints: false, intersections: true, paths: true }; + for i in 0..400u16 { + let at = [f64::from(i) * 2.37, f64::from(i) * 1.13]; + assert_eq!(bounded.snap(at, 6.0, options).unwrap(), full.snap(at, 6.0, options).unwrap(), "at {at:?}"); + } + } + + #[test] + fn limits_stay_within_what_the_index_can_number() { + let limits = Limits { segments: usize::MAX, targets: usize::MAX, bytes: 1 }.indexable(); + assert!(limits.segments <= u32::MAX as usize && limits.targets <= u32::MAX as usize); + assert_eq!(Limits::DEFAULT.indexable(), Limits::DEFAULT); + } + + /// A tolerance below zero finds nothing; it must not reverse the cells searched. + #[test] + fn a_negative_tolerance_finds_nothing() { + let segs = (0..4000u16) + .map(|i| [f32::from(i % 64) * 15.0, f32::from(i / 64) * 15.0, f32::from(i % 64) * 15.0 + 5.0, f32::from(i / 64) * 15.0]) + .collect(); + let g = geometry(segs, Vec::new(), u64::MAX); + assert!(g.grid.cols > 1 && g.grid.rows > 1); + assert!(!g.intersection_limited([400.0, 400.0], -200.0)); + } + + /// Near 1e8 one `f32` step is 8 units, which must not hide an end just within tolerance. + #[test] + fn rounding_doesnt_hide_a_target_at_the_edge_of_the_grid() { + let g = geometry(Vec::new(), vec![[100_000_008.0, 0.0, 100_000_016.0, 0.0, 100_000_012.0, 0.0]], u64::MAX); + let options = SnapOptions { endpoints: true, midpoints: false, intersections: false, paths: false }; + let hit = g.snap([100_000_003.999_999_9, 0.0], 4.000_000_11, options).unwrap(); + assert_eq!(hit.map(|h| h.point), Some([100_000_008.0, 0.0])); + } + + /// A target is three points, and all three must fit. + #[test] + fn a_target_is_kept_only_if_all_its_points_fit() { + let mut g = Geometry::with_limits(Limits { segments: 10, targets: 2, bytes: 1 }); + g.target([[0.0, 0.0], [1.0, 0.0]], [0.5, 0.0]); + assert_eq!(g.targets(), 0); + assert!(g.truncated); + } } diff --git a/crates/measure/src/tests.rs b/crates/measure/src/tests.rs index f265f086ec..b039f7c57f 100644 --- a/crates/measure/src/tests.rs +++ b/crates/measure/src/tests.rs @@ -112,7 +112,7 @@ fn filled_open_triangle_snaps_to_closing_edge() { for (op, closed) in [("f", true), ("F", true), ("f*", true), ("B", true), ("B*", true), ("h f", true), ("S", false)] { let d = fixture(&format!("40 50 m 160 50 l 100 170 l {op}"), "", &[]); let paths = snap::geometry(&d, 0).unwrap(); - assert_eq!(paths.segments.len(), if closed { 3 } else { 2 }, "{op}"); + assert_eq!(paths.segment_count(), if closed { 3 } else { 2 }, "{op}"); let hit = paths.snap([70.0, 110.0], 5.0, snap::SnapOptions::default()).unwrap(); assert_eq!(hit.is_some(), closed, "{op}"); } @@ -121,7 +121,7 @@ fn filled_open_triangle_snaps_to_closing_edge() { fn snap_paths_endpoints_midpoints_and_intersections() { let d = fixture("10 20 m 110 20 l S 60 0 m 60 80 l S 10 100 100 40 re S", "", &[]); let paths = snap::geometry(&d, 0).unwrap(); - assert_eq!(paths.segments.len(), 6); + assert_eq!(paths.segment_count(), 6); let opts = snap::SnapOptions::default(); let hit = paths.snap([11.0, 20.0], 3.0, opts).unwrap().unwrap(); assert_eq!(hit.kind, snap::SnapKind::Endpoint); @@ -147,9 +147,9 @@ fn filled_open_subpath_offers_the_same_snap_edges_as_explicit_close() { let explicit = snap::geometry(&closed, 0).unwrap(); let stroked = snap::geometry(&stroke, 0).unwrap(); - assert_eq!(filled.segments, explicit.segments); - assert_eq!(filled.segments.len(), 3); - assert_eq!(stroked.segments.len(), 2); + assert!(filled.segments().eq(explicit.segments())); + assert_eq!(filled.segment_count(), 3); + assert_eq!(stroked.segment_count(), 2); let paths_only = snap::SnapOptions { endpoints: false, midpoints: false, intersections: false, paths: true }; let hit = filled.snap([55.0, 80.0], 5.0, paths_only).unwrap().unwrap(); assert_eq!(hit.kind, snap::SnapKind::Path); @@ -163,12 +163,12 @@ fn fills_close_all_open_subpaths_but_strokes_do_not() { for paint in ["f", "F", "f*", "B", "B*", "b", "b*"] { let doc = fixture(&format!("{triangles} {paint}"), "", &[]); let geometry = snap::geometry(&doc, 0).unwrap(); - assert_eq!(geometry.segments.len(), 6, "{paint}"); - assert!(geometry.segments.contains(&[[10.0, 20.0], [0.0, 0.0]]), "{paint}"); - assert!(geometry.segments.contains(&[[50.0, 20.0], [40.0, 0.0]]), "{paint}"); + assert_eq!(geometry.segment_count(), 6, "{paint}"); + assert!(geometry.segments().any(|s| s == [[10.0, 20.0], [0.0, 0.0]]), "{paint}"); + assert!(geometry.segments().any(|s| s == [[50.0, 20.0], [40.0, 0.0]]), "{paint}"); } let stroked = fixture(&format!("{triangles} S"), "", &[]); - assert_eq!(snap::geometry(&stroked, 0).unwrap().segments.len(), 4); + assert_eq!(snap::geometry(&stroked, 0).unwrap().segment_count(), 4); } #[test] @@ -180,12 +180,12 @@ fn nested_forms_transforms_curves_and_cycles_are_bounded() { ); let d = fixture("q 1 0 0 1 100 200 cm /Nested Do Q", "", &[&form]); let geometry = snap::geometry(&d, 0).unwrap(); - assert_eq!(geometry.segments[0], [[100.0, 200.0], [120.0, 200.0]]); - assert!(geometry.endpoints.contains(&[120.0, 200.0])); - assert!(geometry.midpoints.contains(&[110.0, 222.5])); - assert!(geometry.segments.len() > 4 && geometry.segments.len() < 200); + assert_eq!(geometry.segments().next(), Some([[100.0, 200.0], [120.0, 200.0]])); + assert!(geometry.endpoints().any(|p| p == [120.0, 200.0])); + assert!(geometry.midpoints().any(|p| p == [110.0, 222.5])); + assert!(geometry.segment_count() > 4 && geometry.segment_count() < 200); let d = fixture(&"0 0 m 1 1 l S ".repeat(20_001), "", &[]); - assert!(snap::geometry(&d, 0).unwrap().truncated); + assert!(snap::geometry_with_limits(&d, 0, SMALL).unwrap().truncated); } #[test] fn measurement_csv_quotes_newlines_and_blocks_formula_cells() { @@ -242,35 +242,39 @@ fn imported_measurement_appearance_is_not_silently_replaced() { close(list(&d).measurements[0].reading.value, 100.0); } +/// The limits snapping had before heavy sheets, which the cap tests below were written for. +const SMALL: snap::Limits = snap::Limits { segments: 20_000, targets: 40_000, bytes: 16 << 20 }; + #[test] fn degenerate_and_dense_paths_keep_snap_targets_bounded() { // Rejected (zero-length) segments record no endpoints or midpoints at all. let d = fixture(&"0 0 0 0 re f ".repeat(100_000), "", &[]); - let g = snap::geometry(&d, 0).unwrap(); - assert!(g.segments.is_empty() && g.endpoints.is_empty() && g.midpoints.is_empty(), "{} {}", g.endpoints.len(), g.midpoints.len()); + let g = snap::geometry_with_limits(&d, 0, SMALL).unwrap(); + assert!(g.segment_count() == 0 && g.endpoints().next().is_none() && g.midpoints().next().is_none()); // Every paint drains into the page geometry; the page-wide target cap still holds. let d = fixture(&"0 0 1 1 re f ".repeat(30_000), "", &[]); - let g = snap::geometry(&d, 0).unwrap(); + let g = snap::geometry_with_limits(&d, 0, SMALL).unwrap(); assert!(g.truncated); - assert!(g.endpoints.len() + g.midpoints.len() <= 40_000 + 12, "{}", g.endpoints.len() + g.midpoints.len()); - assert!(g.segments.len() <= 20_000 + 4); + let targets = g.endpoints().count() + g.midpoints().count(); + assert!(targets <= 40_000 + 12, "{targets}"); + assert!(g.segment_count() <= 20_000 + 4); // Unpainted curves and lines are capped too. let d = fixture(&"0 0 m 5 9 1 9 7 0 c 0 3 l ".repeat(30_000), "", &[]); - let g = snap::geometry(&d, 0).unwrap(); - assert!(g.truncated && g.endpoints.is_empty()); + let g = snap::geometry_with_limits(&d, 0, SMALL).unwrap(); + assert!(g.truncated && g.endpoints().next().is_none()); } #[test] fn deep_graphics_state_and_unreadable_streams_are_skipped_not_fatal() { let d = fixture(&format!("0 0 m 10 0 l S {}", "q ".repeat(300)), "", &[]); let g = snap::geometry(&d, 0).unwrap(); assert!(g.truncated); - assert_eq!(g.segments.len(), 1); + assert_eq!(g.segment_count(), 1); // An unsupported filter cannot be decoded at all (corrupt Flate is decoded tolerantly). let broken = "<< /Length 10 /Filter /NoSuchDecode >>\nstream\nnot-zlib!!\nendstream"; let d = fixture_with("[4 0 R 5 0 R]", "0 0 m 10 0 l S", "", &[broken]); let g = snap::geometry(&d, 0).unwrap(); assert_eq!(g.unreadable, 1); - assert_eq!(g.segments.len(), 1); + assert_eq!(g.segment_count(), 1); let d = fixture_with("5 0 R", "", "", &[broken]); assert_eq!(snap::geometry(&d, 0).unwrap().unreadable, 1); } @@ -410,7 +414,7 @@ fn xobject_resources_are_read_once_per_stream() { let d = fixture(&"/Missing Do\n".repeat(10_000), "", &[]); snap::XOBJECT_DICT_READS.with(|c| c.set(0)); let paths = snap::geometry(&d, 0).unwrap(); - assert!(paths.segments.is_empty()); + assert_eq!(paths.segment_count(), 0); let reads = snap::XOBJECT_DICT_READS.with(std::cell::Cell::get); assert!(reads <= 1, "{reads} reads of the page's XObject resources"); } @@ -466,10 +470,10 @@ fn a_stream_over_the_byte_budget_uses_it_up_once() { let bomb = String::from_utf8(pdfcraft_testkit::stream_obj("/Filter [/ASCIIHexDecode /FlateDecode]", hex)).unwrap(); let contents = format!("[4 0 R{}]", " 5 0 R".repeat(50)); let d = fixture_with(&contents, "0 0 m 10 0 l S", "", &[bomb.as_str()]); - let g = snap::geometry(&d, 0).unwrap(); + let g = snap::geometry_with_limits(&d, 0, SMALL).unwrap(); assert_eq!(g.unreadable, 1); assert!(g.truncated); - assert_eq!(g.segments.len(), 1, "the parts before it still snap"); + assert_eq!(g.segment_count(), 1, "the parts before it still snap"); } /// A content stream drawn again and again pays for every filter stage's output each time, not @@ -484,7 +488,7 @@ fn repeated_content_pays_for_every_filter_stage() { .obj("<< /Type /Page /Parent 2 0 R /Contents [4 0 R 4 0 R 4 0 R 4 0 R 4 0 R 4 0 R 4 0 R 4 0 R] >>") .stream("/Filter [/ASCIIHexDecode /RunLengthDecode]", &hex); let d = Document::open(Arc::new(pdf.build())).unwrap(); - let g = snap::geometry(&d, 0).unwrap(); + let g = snap::geometry_with_limits(&d, 0, SMALL).unwrap(); assert!(g.truncated, "eight draws decode past the page's bound"); } diff --git a/crates/measure/tests/snap_property.rs b/crates/measure/tests/snap_property.rs index a38bb80aaf..ddb071dc33 100644 --- a/crates/measure/tests/snap_property.rs +++ b/crates/measure/tests/snap_property.rs @@ -131,18 +131,20 @@ fn finite(point: [f64; 2]) -> bool { fn geometry_of(content: &[u8]) -> Geometry { let doc = Document::open(Arc::new(one_page_pdf(content))).unwrap(); - snap::geometry(&doc, 0).unwrap() + // The caps these properties were written for. + let limits = snap::Limits { segments: MAX_GEOMETRY_SEGMENTS, targets: 2 * MAX_GEOMETRY_SEGMENTS, bytes: 16 << 20 }; + snap::geometry_with_limits(&doc, 0, limits).unwrap() } /// Checks the bounds and finiteness that snapping relies on, and returns the extracted geometry. fn check_snap_invariants(content: &[u8]) -> Geometry { assert!(content.len() <= MAX_CONTENT_BYTES, "generated content exceeded the byte cap"); let geometry = geometry_of(content); - assert!(geometry.segments.len() <= MAX_GEOMETRY_SEGMENTS, "geometry exceeded the segment cap: {}", geometry.segments.len()); - assert!(geometry.endpoints.len().saturating_add(geometry.midpoints.len()) <= MAX_SNAP_TARGETS, "geometry exceeded the snap-target cap"); - assert!(geometry.segments.iter().flatten().copied().all(finite), "geometry contains a non-finite segment coordinate"); - assert!(geometry.endpoints.iter().copied().all(finite), "geometry contains a non-finite endpoint"); - assert!(geometry.midpoints.iter().copied().all(finite), "geometry contains a non-finite midpoint"); + assert!(geometry.segment_count() <= MAX_GEOMETRY_SEGMENTS, "geometry exceeded the segment cap: {}", geometry.segment_count()); + assert!(geometry.endpoints().count().saturating_add(geometry.midpoints().count()) <= MAX_SNAP_TARGETS, "geometry exceeded the snap-target cap"); + assert!(geometry.segments().flatten().all(finite), "geometry contains a non-finite segment coordinate"); + assert!(geometry.endpoints().all(finite), "geometry contains a non-finite endpoint"); + assert!(geometry.midpoints().all(finite), "geometry contains a non-finite midpoint"); geometry.intersection_limited([0.0, 0.0], 10_000.0); let snap = geometry.snap([0.0, 0.0], 10_000.0, SnapOptions::default()).unwrap(); @@ -173,7 +175,7 @@ fn mutated_snap_geometry_stays_bounded_and_finite() { fn high_curvature_geometry_hits_segment_cap() { let stress_curve = b"0 0 m 0 100000000 100000000 100000000 100000000 0 c S ".repeat(6); let geometry = check_snap_invariants(&stress_curve); - assert!(geometry.truncated, "high-curvature geometry did not report its extraction cap ({} segments)", geometry.segments.len()); + assert!(geometry.truncated, "high-curvature geometry did not report its extraction cap ({} segments)", geometry.segment_count()); } fn two_piece_pdf(first: &[u8], second: &[u8]) -> Vec { @@ -216,3 +218,95 @@ fn trailing_comment_in_previous_piece_hides_next_piece_first_line() { assert!(snap_at_box_corner(b"q % tail comment, no end of line").is_none(), "snapped to a rectangle the renderer comments out"); assert!(snap_at_box_corner(b"q % tail comment\n").is_some(), "control: the rectangle is drawn"); } + +/// A snap computed by scanning the whole page, as snapping did before its grid index: the +/// reference the indexed snap must equal. +fn snap_by_scan(g: &Geometry, at: [f64; 2], tolerance: f64, options: SnapOptions) -> Option { + let distance = |a: [f64; 2], b: [f64; 2]| (a[0] - b[0]).hypot(a[1] - b[1]); + let nearest = |p: [f64; 2], a: [f64; 2], b: [f64; 2]| { + let (dx, dy) = (b[0] - a[0], b[1] - a[1]); + let len2 = dx * dx + dy * dy; + if len2 <= f64::EPSILON { + return a; + } + let t = (((p[0] - a[0]) * dx + (p[1] - a[1]) * dy) / len2).clamp(0.0, 1.0); + [a[0] + t * dx, a[1] + t * dy] + }; + let mut best: Option = None; + let consider = |best: &mut Option, p: [f64; 2], kind: snap::SnapKind| { + let d = distance(at, p); + if d <= tolerance && best.is_none_or(|b| d < b.distance - 1e-9) { + *best = Some(snap::Snap { point: p, kind, distance: d }); + } + }; + if options.endpoints { + g.endpoints().for_each(|p| consider(&mut best, p, snap::SnapKind::Endpoint)); + } + if options.midpoints { + g.midpoints().for_each(|p| consider(&mut best, p, snap::SnapKind::Midpoint)); + } + let local: Vec<[[f64; 2]; 2]> = g.segments().filter(|[a, b]| distance(at, nearest(at, *a, *b)) <= tolerance).take(256).collect(); + if options.intersections { + for (i, s) in local.iter().enumerate() { + for t in local.iter().skip(i + 1) { + if let Some(p) = snap::intersection(s[0], s[1], t[0], t[1]) { + consider(&mut best, p, snap::SnapKind::Intersection); + } + } + } + } + if options.paths && best.is_none() { + g.segments().for_each(|[a, b]| consider(&mut best, nearest(at, a, b), snap::SnapKind::Path)); + } + best +} + +/// Whether more than 256 segments pass within `tolerance` of `at`, by a scan of the whole page. +fn limited_by_scan(g: &Geometry, at: [f64; 2], tolerance: f64) -> bool { + g.segments() + .filter(|[a, b]| { + let (dx, dy) = (b[0] - a[0], b[1] - a[1]); + let len2 = dx * dx + dy * dy; + let t = if len2 <= f64::EPSILON { 0.0 } else { (((at[0] - a[0]) * dx + (at[1] - a[1]) * dy) / len2).clamp(0.0, 1.0) }; + (at[0] - (a[0] + t * dx)).hypot(at[1] - (a[1] + t * dy)) <= tolerance + }) + .count() + > 256 +} + +#[test] +fn indexed_snapping_matches_a_scan_of_the_whole_page() { + let mut rng = Rng(SEED ^ 0x5eed); + let mut checked = 0usize; + for case in 0..200 { + let seed = SEEDS[rng.below(SEEDS.len())]; + // A drawing with many short lines, some long ones (kept outside the cells) and curves. + let mut content = mutate(&mut rng, seed); + for i in 0..200 + rng.below(800) { + let (x, y) = (rng.below(600) as f64, rng.below(800) as f64); + let line = match i % 7 { + 0 => format!("{x} {y} m {} {} l S ", x + rng.below(600) as f64, y), + 1 => format!("{x} {y} m {} {} {} {} {} {} c S ", x + 5.0, y + 9.0, x + 9.0, y + 5.0, x + 12.0, y), + _ => format!("{x} {y} m {} {} l S ", x + rng.below(20) as f64, y + rng.below(20) as f64), + }; + content.extend_from_slice(line.as_bytes()); + } + let g = geometry_of(&content); + for _ in 0..40 { + let at = [rng.below(700) as f64 - 50.0 + 0.37, rng.below(900) as f64 - 50.0 + 0.61]; + let tolerance = [0.5, 2.0, 7.5, 40.0, 400.0][rng.below(5)]; + let options = SnapOptions { + endpoints: rng.below(4) != 0, + midpoints: rng.below(4) != 0, + intersections: rng.below(3) != 0, + paths: rng.below(2) == 0, + }; + let indexed = g.snap(at, tolerance, options).unwrap(); + let scanned = snap_by_scan(&g, at, tolerance, options); + assert_eq!(indexed, scanned, "case {case} at {at:?} tolerance {tolerance} {options:?}"); + assert_eq!(g.intersection_limited(at, tolerance), limited_by_scan(&g, at, tolerance), "case {case} at {at:?} tolerance {tolerance}"); + checked += 1; + } + } + assert!(checked >= 8000); +} diff --git a/crates/organize/src/import.rs b/crates/organize/src/import.rs index 6ec4d3bd92..c9b4b85a67 100644 --- a/crates/organize/src/import.rs +++ b/crates/organize/src/import.rs @@ -103,8 +103,8 @@ fn added_objects(dst: &Document, before: &HashSet) -> Vec { /// Insert Pages from File: [`import_pages_with`], and the inserted file's bookmarks come along /// under one new bookmark titled `title` (the file's name) at its first inserted page, as Combine /// Files does for each file. That bookmark goes among the top-level bookmarks where the pages went: -/// before the first one that points after them. Bookmarks to pages left out lose their destination; -/// a file without bookmarks adds none. +/// before the first one that points after them. Bookmarks to pages left out stay behind, unless a +/// bookmark below them leads to an inserted page; a file without bookmarks adds none. pub fn insert_file_pages( dst: &mut Document, src: &Document, diff --git a/crates/organize/src/import/carry.rs b/crates/organize/src/import/carry.rs index 15a412e231..00fb9bd0b8 100644 --- a/crates/organize/src/import/carry.rs +++ b/crates/organize/src/import/carry.rs @@ -98,9 +98,13 @@ pub(super) fn source_outline(src: &Document, pages: HashMap) -> .and_then(|r| src.get(r).as_dict().cloned()) .and_then(|c| c.get(b"Outlines").map(|o| src.resolve(o))) .and_then(|o| o.as_dict().and_then(|d| d.reference(b"First"))); + let whole = crate::page_count(src).is_ok_and(|n| pages.len() >= n); let mapper = Copier::new(src, HashMap::new(), pages); let mut budget = 10_000usize; - outline_level(&mapper, first, 0, &mut budget) + let outline = outline_level(&mapper, first, 0, &mut budget); + // Of some of the pages, only the bookmarks that lead to a copied page and those above them: + // the others would carry their titles along into a file without their pages. + if whole { outline } else { kept_bookmarks(outline) } } /// `nodes` without the bookmarks whose page was not copied (no destination left), unless a bookmark diff --git a/crates/organize/src/import/combine.rs b/crates/organize/src/import/combine.rs index 06972ae101..7782e5de18 100644 --- a/crates/organize/src/import/combine.rs +++ b/crates/organize/src/import/combine.rs @@ -1,5 +1,5 @@ use super::carry::{Mark, add_outline, collect_attachments, kept_bookmarks, set_attachments, source_outline, write_outline_level}; -use super::{Dict, Document, FieldConflicts, HashMap, ObjRef, Object, OrganizeError, import_pages_mapped, rebuild}; +use super::{Dict, Document, FieldConflicts, HashMap, HashSet, ObjRef, Object, OrganizeError, import_pages_mapped, rebuild}; /// A new document containing copies of `pages` from `src` (Extract Pages / Split). /// Document information (title, author…) is carried over, and so is the print standard the @@ -95,7 +95,8 @@ pub fn combine_with(sources: &[(&str, &Document)], conflicts: FieldConflicts) -> } /// Combine Files with chosen pages: each source contributes `pages` (0-based, in that order; -/// `None` for all of them). Bookmarks that point at pages left out lose their destination. +/// `None` for all of them). A source whose pages are not all used brings only the bookmarks that +/// lead to its copied pages (and those above them), and none of its attachments. pub fn combine_selected(sources: &[(&str, &Document, Option<&[usize]>)]) -> Result { combine_selected_with(sources, FieldConflicts::Rename).map(|o| o.doc) } @@ -187,7 +188,11 @@ impl Combiner { let outline = source_outline(src, imported.page_map); self.marks.push(Mark { title: title.to_string(), page: *first, outline }); } - collect_attachments(&mut self.out, src, &mut self.attachments)?; + // Attachments belong to the whole file: they come along with all of its pages, not with a + // selection of them. + if pages.iter().collect::>().len() == n { + collect_attachments(&mut self.out, src, &mut self.attachments)?; + } let declared = crate::pdfx::PrintStandard::of(src); match self.standard.as_ref().map(|first| first.declared.same_as(&declared)) { Some(same) => self.agreed &= same, @@ -227,9 +232,9 @@ pub type Run<'a> = (usize, &'a str, &'a Document, Option<&'a [usize]>); /// Combine Files where a file's pages may be split into several runs with other files' pages /// between them. Each file (group) is copied once, all its pages together, then the pages are put -/// in the runs' order: its links between its own pages, its form fields and its attachments stay -/// whole, and it gets one top-level bookmark (the title of its first run) at its first page, its -/// own bookmarks nested. Otherwise as [`combine_selected`] (print standard, shared resources). +/// in the runs' order: its links between its own pages and its form fields stay whole across its +/// runs, its attachments come along when its runs use all of its pages, and it gets one top-level +/// bookmark (the title of its first run) at its first page, its own bookmarks nested. Otherwise as [`combine_selected`] (print standard, shared resources). pub fn combine_grouped(runs: &[Run<'_>]) -> Result { combine_grouped_with(runs, FieldConflicts::Rename).map(|o| o.doc) } @@ -281,7 +286,10 @@ pub fn combine_grouped_with(runs: &[Run<'_>], conflicts: FieldConflicts) -> Resu if let Some(first) = imported.pages.first() { marks.push(Mark { title: file.title.to_string(), page: *first, outline: source_outline(file.src, imported.page_map) }); } - collect_attachments(&mut out, file.src, &mut attachments)?; + // Attachments come along with all of a file's pages, not with a selection of them. + if file.pages.iter().collect::>().len() == crate::page_count(file.src)? { + collect_attachments(&mut out, file.src, &mut attachments)?; + } let declared = crate::pdfx::PrintStandard::of(file.src); match &standard { None => standard = Some((declared, file.src, imported.objects)), diff --git a/crates/organize/src/tests.rs b/crates/organize/src/tests.rs index 6e8f2df4be..ed95321006 100644 --- a/crates/organize/src/tests.rs +++ b/crates/organize/src/tests.rs @@ -1591,13 +1591,12 @@ fn inserting_a_file_brings_its_bookmarks_under_one_named_after_it() { let page_of = |title: &str| marks.iter().find(|b| b.title == title).and_then(|b| b.page); assert_eq!((page_of("annex"), page_of("Chapter one"), page_of("Section"), page_of("End")), (Some(1), Some(1), Some(2), Some(4))); assert!(!marks.iter().find(|b| b.title == "annex").unwrap().open, "the file's bookmarks start collapsed"); - // Inserted last, after every bookmark; only the pages chosen keep a destination. + // Inserted last, after every bookmark; a bookmark to a page left out stays behind with it. let mut host = open(fixture()); add_bookmark(&mut host, &[], 0, "Start", 0).unwrap(); insert_file_pages(&mut host, &doc_c(), "annex", &[0], 9, FieldConflicts::Rename).unwrap(); let marks = bookmarks(&host); - assert_eq!(titles(&marks), ["Start", "annex[Chapter one[Section]]"]); - assert_eq!(marks.iter().find(|b| b.title == "Section").unwrap().page, None, "its page was left out"); + assert_eq!(titles(&marks), ["Start", "annex[Chapter one]"], "Section's page was left out"); // A file without bookmarks adds none, and a host without an outline gets one only when needed. let mut host = open(fixture()); insert_file_pages(&mut host, &doc_b(), "plain", &[0], 0, FieldConflicts::Rename).unwrap(); @@ -4453,3 +4452,42 @@ fn a_form_action_brings_no_field_from_a_page_left_behind() { // With every page, the actions still list it. assert!(holds_value(&full_roundtrip(&extract_pages(&src, &[0, 1]).unwrap()), "90000")); } + +/// The titles of a bookmark's children, each with its own children's titles. +fn bookmark_titles(doc: &Document, first: Option) -> Vec<(String, Vec)> { + let mut out = Vec::new(); + let mut at = first; + while let Some(r) = at { + let item = doc.get(r).as_dict().cloned().unwrap(); + let title = item.get(b"Title").and_then(|t| t.as_string()).map(|t| t.to_text()).unwrap_or_default(); + let children = bookmark_titles(doc, item.reference(b"First")).into_iter().map(|(t, _)| t).collect(); + out.push((title, children)); + at = item.reference(b"Next"); + } + out +} + +/// Combining some of a file's pages brings only the bookmarks that lead to them (and those above +/// them), and none of the file's attachments. +#[test] +fn combining_some_pages_brings_only_their_bookmarks_and_no_attachments() { + let source_bookmarks = |out: &Document| { + let outlines = out.get(catalog(out).reference(b"Outlines").unwrap()).as_dict().cloned().unwrap(); + let file = out.get(outlines.reference(b"First").unwrap()).as_dict().cloned().unwrap(); + bookmark_titles(out, file.reference(b"First")) + }; + let attachments = |out: &Document| { + catalog(out).get(b"Names").map(|n| out.resolve(n)).is_some_and(|n| n.as_dict().is_some_and(|d| d.contains(b"EmbeddedFiles"))) + }; + // Page 1 only: "Section" leads to page 2. + let first = full_roundtrip(&combine_selected(&[("C", &doc_c(), Some(&[0]))]).unwrap()); + assert_eq!(source_bookmarks(&first), [("Chapter one".to_string(), vec![])]); + assert!(!attachments(&first), "an attachment came along with a selection of pages"); + // Page 2 only: "Chapter one" stays, as the way to "Section". + let second = full_roundtrip(&combine_selected(&[("C", &doc_c(), Some(&[1]))]).unwrap()); + assert_eq!(source_bookmarks(&second), [("Chapter one".to_string(), vec!["Section".to_string()])]); + // Every page: all of it, attachments too. + let all = full_roundtrip(&combine_selected(&[("C", &doc_c(), Some(&[0, 1]))]).unwrap()); + assert_eq!(source_bookmarks(&all), [("Chapter one".to_string(), vec!["Section".to_string()])]); + assert!(attachments(&all)); +} diff --git a/crates/preflight/src/gstate.rs b/crates/preflight/src/gstate.rs new file mode 100644 index 0000000000..64c19de116 --- /dev/null +++ b/crates/preflight/src/gstate.rs @@ -0,0 +1,235 @@ +//! PDF/A-2 6.2.5, the extended graphics state: device-dependent settings a conforming file may not +//! rely on. +//! +//! - no transfer function: no `/TR` entry, and `/TR2` only as `/Default` (ISO 32000-1 10.4); +//! - no `/HTP` (halftone phase) entry, a PDF 1.2 key ISO 32000-1 no longer defines; +//! - halftones of type 1 or 5 only, without a `/HalftoneName` (ISO 32000-1 10.5.5); the halftones +//! a type 5 halftone holds for its colorants are checked the same way; +//! - a halftone's `/TransferFunction` only where one is needed: a type 5 halftone's component for +//! a non-primary colorant has one, and no other halftone does. The primary colour components +//! are those of the standard device spaces: Gray; Red, Green and Blue; Cyan, Magenta, Yellow +//! and Black (ISO 32000-1 10.5.5.6; ISO 32000-2 10.6.5.6). The `Default` component is left as +//! it is (ISO 32000-1 Table 130); +//! - a rendering intent (`/RI`) that is one of the four of ISO 32000-1 8.6.5.8, Table 70. +//! +//! Graphics state dictionaries are found as the `/ExtGState` entries of resource dictionaries +//! (`/Resources` of pages, forms, Type 3 fonts, patterns and annotation appearances, and an +//! interactive form's `/DR`), and as the `/ExtGState` of a shading pattern (ISO 32000-1 8.7.4.3, +//! Table 76). They are looked for in what the document reaches from its trailer, following +//! references, but not into private application data (`/PieceInfo`), so an object only that data +//! refers to isn't read. A map shared by reference is read once. + +use std::collections::{BTreeSet, HashSet, VecDeque}; + +use pdfcraft_cos::{Dict, Document, ObjRef, Object}; + +/// How deep a direct object is followed: as deep as the parser nests them. +const MAX_DEPTH: u8 = 128; +/// The rendering intents of ISO 32000-1 Table 70. +const INTENTS: [&[u8]; 4] = [b"AbsoluteColorimetric", b"RelativeColorimetric", b"Saturation", b"Perceptual"]; +/// The entries of a type 5 halftone that aren't colorant halftones. +const HALFTONE_KEYS: [&[u8]; 3] = [b"Type", b"HalftoneType", b"HalftoneName"]; +/// The primary colour components of DeviceGray, DeviceRGB and DeviceCMYK, whose halftones take +/// the graphics state's transfer function. +const PRIMARY: [&[u8]; 8] = [b"Gray", b"Red", b"Green", b"Blue", b"Cyan", b"Magenta", b"Yellow", b"Black"]; +/// The entries that hold resource dictionaries. +const RESOURCES: [&[u8]; 2] = [b"Resources", b"DR"]; + +/// What the graphics states of a document break of 6.2.5. +#[derive(Debug, Default, PartialEq, Eq)] +pub(crate) struct Findings { + /// A graphics state has a `/TR` entry. + pub(crate) transfer: bool, + /// A graphics state has a `/TR2` other than `/Default`. + pub(crate) transfer2: bool, + /// A graphics state has an `/HTP`. + pub(crate) phase: bool, + /// Halftone types other than 1 and 5, in order. + pub(crate) halftone_types: BTreeSet, + /// A halftone's `/HalftoneType` is missing or isn't an integer. + pub(crate) untyped: bool, + /// A halftone has a `/HalftoneName`. + pub(crate) halftone_name: bool, + /// A halftone that may not have a `/TransferFunction` has one. + pub(crate) halftone_transfer: bool, + /// Non-primary colorants whose halftone has no `/TransferFunction`, in name order. + pub(crate) missing_transfer: BTreeSet, + /// Rendering intents outside Table 70, in name order. + pub(crate) intents: BTreeSet, + /// A rendering intent isn't a name. + pub(crate) intent_not_name: bool, +} + +/// The 6.2.5 findings over what `doc` reaches from its trailer. +pub(crate) fn check(doc: &Document) -> Findings { + scan(doc).findings +} + +/// How many objects [`check`] reads. +#[cfg(test)] +pub(crate) fn reached(doc: &Document) -> usize { + scan(doc).reached.len() +} + +fn scan(doc: &Document) -> Scan<'_> { + let mut scan = Scan { doc, read: HashSet::new(), reached: HashSet::new(), queue: VecDeque::new(), findings: Findings::default() }; + scan.walk(&Object::Dict(doc.trailer().clone()), 0); + // References are followed from a queue, not by recursion: a chain of them can be as long as + // the file. + while let Some(r) = scan.queue.pop_front() { + let o = doc.get(r); + scan.walk(&o, 0); + } + scan +} + +struct Scan<'a> { + doc: &'a Document, + /// Resource dictionaries, `/ExtGState` maps and graphics states read so far, by object number. + read: HashSet, + /// Object numbers reached so far, queued or walked. Both sets go by number, as + /// [`Document::get`] does: references differing only in generation reach the same object. + reached: HashSet, + queue: VecDeque, + findings: Findings, +} + +impl Scan<'_> { + fn walk(&mut self, o: &Object, depth: u8) { + if depth > MAX_DEPTH { + return; + } + let d = match o { + Object::Ref(r) => { + if self.reached.insert(r.num) { + self.queue.push_back(*r); + } + return; + } + Object::Array(a) => { + for v in a { + self.walk(v, depth.saturating_add(1)); + } + return; + } + Object::Dict(d) => d, + Object::Stream(s) => &s.dict, + _ => return, + }; + self.holder(d); + for (k, v) in d.iter() { + if k.as_slice() != b"PieceInfo" { + self.walk(v, depth.saturating_add(1)); + } + } + } + + /// Whether `o` is a reference read before; marks it read. + fn read_before(&mut self, o: &Object) -> bool { + matches!(o, Object::Ref(r) if !self.read.insert(r.num)) + } + + /// A dictionary that holds graphics states: through its resources, or as a shading pattern. + fn holder(&mut self, d: &Dict) { + for key in RESOURCES { + if let Some(resources) = d.get(key) { + self.resources(resources); + } + } + if d.get(b"PatternType").is_some() + && let Some(gs) = d.get(b"ExtGState") + && !self.read_before(gs) + && let Some(gs) = self.doc.resolve(gs).as_dict() + { + self.gstate(gs); + } + } + + fn resources(&mut self, resources: &Object) { + if self.read_before(resources) { + return; + } + let resources = self.doc.resolve(resources); + let Some(states) = resources.as_dict().and_then(|r| r.get(b"ExtGState")) else { return }; + if self.read_before(states) { + return; + } + let states = self.doc.resolve(states); + let Some(states) = states.as_dict() else { return }; + for (_, gs) in states.iter() { + if !self.read_before(gs) + && let Some(gs) = self.doc.resolve(gs).as_dict() + { + self.gstate(gs); + } + } + } + + /// `key` of `d`, resolved, unless it is missing or null (ISO 32000-1 7.3.9). + fn present(&self, d: &Dict, key: &[u8]) -> Option> { + d.get(key).map(|v| self.doc.resolve(v)).filter(|v| !matches!(**v, Object::Null)) + } + + fn gstate(&mut self, gs: &Dict) { + if self.present(gs, b"TR").is_some() { + self.findings.transfer = true; + } + if self.present(gs, b"TR2").is_some_and(|f| f.as_name() != Some(b"Default")) { + self.findings.transfer2 = true; + } + if self.present(gs, b"HTP").is_some() { + self.findings.phase = true; + } + if let Some(ht) = self.present(gs, b"HT") + && let Some(ht) = ht.as_dict() + { + self.halftone(ht, None); + } + if let Some(ri) = self.present(gs, b"RI") { + match ri.as_name() { + Some(ri) if !INTENTS.contains(&ri) => { + self.findings.intents.insert(String::from_utf8_lossy(ri).into_owned()); + } + Some(_) => {} + None => self.findings.intent_not_name = true, + } + } + } + + /// A halftone dictionary or stream: the graphics state's own (`colorant` is `None`), or the + /// one a type 5 halftone holds for `colorant`. + fn halftone(&mut self, ht: &Dict, colorant: Option<&[u8]>) { + // An integer (ISO 32000-1 Table 128), not a real of the same value. + let kind = self.present(ht, b"HalftoneType").and_then(|t| if let Object::Int(t) = *t { Some(t) } else { None }); + match kind { + None => self.findings.untyped = true, + Some(1 | 5) => {} + Some(kind) => { + self.findings.halftone_types.insert(kind); + } + } + if self.present(ht, b"HalftoneName").is_some() { + self.findings.halftone_name = true; + } + let transfer = self.present(ht, b"TransferFunction").is_some(); + match colorant { + Some(b"Default") => {} + Some(c) if !PRIMARY.contains(&c) => { + if !transfer { + self.findings.missing_transfer.insert(String::from_utf8_lossy(c).into_owned()); + } + } + _ => self.findings.halftone_transfer |= transfer, + } + if colorant.is_none() && kind == Some(5) { + for (k, v) in ht.iter() { + if HALFTONE_KEYS.contains(&k.as_slice()) { + continue; + } + if let Some(component) = self.doc.resolve(v).as_dict() { + self.halftone(component, Some(k)); + } + } + } + } +} diff --git a/crates/preflight/src/header.rs b/crates/preflight/src/header.rs new file mode 100644 index 0000000000..3e9263ef28 --- /dev/null +++ b/crates/preflight/src/header.rs @@ -0,0 +1,71 @@ +//! PDF/A-2 6.1.2, the file header: the file starts at its first byte with `%PDF-1.n`, `n` from 0 +//! to 7, and an end-of-line marker; the next line is a comment, `%` and at least four bytes each +//! above 127, which marks the file as binary (ISO 32000-1 7.5.2). +//! +//! A document whose next save rewrites it in full is judged by the header that save writes: the +//! version it keeps, followed by the writer's own binary comment. + +use pdfcraft_cos::Document; + +/// What the header of `doc`, as it will be saved, breaks of 6.1.2. +#[derive(Debug, Default, PartialEq, Eq)] +pub(crate) struct Header { + /// The first line isn't `%PDF-1.0` to `%PDF-1.7` at the start of the file. + pub(crate) version: bool, + /// The second line isn't a comment of at least four bytes above 127. + pub(crate) comment: bool, +} + +/// What the header of `doc` breaks of 6.1.2: as stored, or, when the next save rewrites the +/// file, as that save will write it. +pub(crate) fn check(doc: &Document) -> Header { + if doc.full_save_required() || doc.encryption_changed() { + let written = doc.version().get(..3).unwrap_or_default(); + return Header { version: !allowed(written.as_bytes()), comment: false }; + } + let data = doc.bytes(); + let Some(rest) = data.strip_prefix(b"%PDF-") else { return Header { version: true, comment: false } }; + let version = rest.get(..3).unwrap_or_default(); + let Some(rest) = rest.get(3..).and_then(eol) else { + return Header { version: true, comment: !binary_comment(after_line(rest)) }; + }; + Header { version: !allowed(version), comment: !binary_comment(rest) } +} + +/// Whether a version is `1.0` to `1.7`. +fn allowed(version: &[u8]) -> bool { + matches!(version, [b'1', b'.', b'0'..=b'7']) +} + +/// What follows an end-of-line marker (CR, LF or CR LF) at the start of `data`. +fn eol(data: &[u8]) -> Option<&[u8]> { + match data { + [b'\r', b'\n', rest @ ..] | [b'\r', rest @ ..] | [b'\n', rest @ ..] => Some(rest), + _ => None, + } +} + +/// What follows the first end-of-line marker in `data`, or nothing. +fn after_line(data: &[u8]) -> &[u8] { + let end = data.iter().position(|&b| b == b'\r' || b == b'\n').unwrap_or(data.len()); + data.get(end..).and_then(eol).unwrap_or_default() +} + +/// Whether `data` starts with `%` and four bytes above 127. +fn binary_comment(data: &[u8]) -> bool { + matches!(data, [b'%', a, b, c, d, ..] if [a, b, c, d].iter().all(|&&x| x > 127)) +} + +/// Makes the next save of `doc` write a header 6.1.2 allows, when the current one breaks it: +/// a full save, at version 1.7 when the version kept isn't 1.0 to 1.7. What it fixed, if anything. +pub(crate) fn rewrite(doc: &mut Document) -> Option { + let header = check(doc); + if !header.version && !header.comment { + return None; + } + if !allowed(doc.version().as_bytes().get(..3).unwrap_or_default()) { + doc.set_version("1.7"); + } + doc.require_full_save(); + Some("Rewrote the file header".into()) +} diff --git a/crates/preflight/src/lib.rs b/crates/preflight/src/lib.rs index 1cbe9a6ed3..d5a6d80b04 100644 --- a/crates/preflight/src/lib.rs +++ b/crates/preflight/src/lib.rs @@ -10,7 +10,9 @@ //! - every font embedded; //! - annotations: allowed types, printable, not hidden, with appearance streams; //! - actions: none of the forbidden kinds (JavaScript, Launch, Sound, Movie, ResetForm, -//! ImportData, Hide, SetOCGState, Rendition, Trans, GoTo3DView); no additional actions on the +//! ImportData, Hide, SetOCGState, Rendition, Trans, GoTo3DView, and the PDF 1.2 SetState and +//! NOP that ISO 32000-1 no longer defines); Named actions only for NextPage, PrevPage, +//! FirstPage and LastPage (ISO 32000-1 12.6.4.11, Table 212); no additional actions on the //! catalog, pages, fields or widgets; `NeedAppearances` not true; //! - images: no `Interpolate`, no `Alternates`, no OPI; no PostScript XObjects; //! - embedded files: PDF/A-2 only allows embedded PDF/A files; PDF/A-3 needs `AFRelationship`. @@ -21,8 +23,15 @@ #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] +mod gstate; +mod header; pub mod icc; +mod limits; +mod spot; +mod trailer; +mod transparency; mod xmp; +mod xmp_extension; mod xmp_types; use pdfcraft_cos::{Dict, Document, ObjRef, Object, PdfString, Stream}; @@ -153,14 +162,46 @@ pub const WORDINGS: &[&str] = &[ "The title in the metadata and the document information differ", "The XMP property {property} in metadata stream {object} should have the value type {type}", "The XMP property {property} in metadata stream {object}, written {written}, should have the value type {type}", + "The XMP extension schema {schema} in metadata stream {object} has no {entry} entry", + "The XMP extension schema {schema} in metadata stream {object} describes {item} without a {entry} entry", + "The XMP extension schema {schema} in metadata stream {object} has an empty {entry} entry", + "The XMP extension schema {schema} in metadata stream {object} describes {item} with an empty {entry} entry", + "The XMP extension schema container in metadata stream {object} holds no rdf:Bag", + "The XMP extension schema container in metadata stream {object} holds {found}, not one rdf:Bag", + "Metadata stream {object} writes the namespace {namespace} without a prefix, not with {prefix}", + "Metadata stream {object} writes the namespace {namespace} with the prefix {written}, not {prefix}", // The document as a whole. + "The file doesn't start with a %PDF-1.0 to %PDF-1.7 header line", + "The file header isn't followed by a comment of four bytes above 127", + "The trailer has no file identifier (ID)", + "The file identifier (ID) in the trailer isn't two non-empty strings", "The document is encrypted", + "An integer is outside the 32-bit range", + "A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸", + "A CMap maps a character code to a CID above 65535", + "A string is longer than 32767 bytes", + "A name is longer than 127 bytes", + "Page {} has a boundary smaller than 3 or larger than 14400 units", "Device colour is used, but there is no PDF/A output intent", + "Page {} has transparency but no device-independent blending colour space or PDF/A output intent", + "A graphics state uses the blend mode {}, which ISO 32000-1 doesn't define", "DeviceCMYK is used: it needs a CMYK output intent (not added automatically)", "DeviceCMYK is used, but the PDF/A output intent isn't a CMYK profile", "DeviceRGB is used, but the PDF/A output intent isn't an RGB profile", + "Separation colour spaces named {} differ in their alternate space or tint transform", + "The spot colour {} of a DeviceN colour space isn't in its Colorants dictionary", "The document has additional actions (/AA)", "The document has document-level JavaScript", + "A graphics state has a transfer function (TR)", + "A graphics state has a transfer function (TR2) other than Default", + "A graphics state has a halftone phase (HTP)", + "A halftone has type {}: only types 1 and 5 are allowed", + "A halftone's HalftoneType is missing or isn't an integer", + "A halftone has a HalftoneName", + "A halftone has a transfer function, which only a non-primary colorant's halftone may have", + "The halftone for the colorant {} has no transfer function", + "A graphics state has the rendering intent {}, which isn't a standard one", + "A graphics state has a rendering intent that isn't a name", "The form asks viewers to make appearances (NeedAppearances)", // Pages, actions, annotations and fields. "Page {} has additional actions", @@ -192,8 +233,23 @@ pub struct Declared { pub output_intents: Vec, } -const FORBIDDEN_ACTIONS: &[&[u8]] = - &[b"JavaScript", b"Launch", b"Sound", b"Movie", b"ResetForm", b"ImportData", b"Hide", b"SetOCGState", b"Rendition", b"Trans", b"GoTo3DView"]; +const FORBIDDEN_ACTIONS: &[&[u8]] = &[ + b"JavaScript", + b"Launch", + b"Sound", + b"Movie", + b"ResetForm", + b"ImportData", + b"Hide", + b"SetOCGState", + b"Rendition", + b"Trans", + b"GoTo3DView", + b"SetState", + b"NOP", +]; +/// The named actions of ISO 32000-1 Table 212, the only ones a Named action may name. +const NAMED_ACTIONS: &[&[u8]] = &[b"NextPage", b"PrevPage", b"FirstPage", b"LastPage"]; const FORBIDDEN_ANNOTS: &[&[u8]] = &[b"Sound", b"Movie", b"Screen", b"3D", b"RichMedia"]; fn catalog(doc: &Document) -> Dict { @@ -250,6 +306,19 @@ fn dict_of(o: &Object) -> Option<&Dict> { } } +/// Whether the document has a PDF/A output intent whose `/DestOutputProfile` is a stream: one with +/// a null or missing profile describes no colour, so it doesn't stand in for a blending colour +/// space (6.2.10). +fn pdfa_intent_has_profile(doc: &Document) -> bool { + let cat = catalog(doc); + let list = cat.get(b"OutputIntents").map(|o| doc.resolve(o)).and_then(|o| o.as_array().cloned()).unwrap_or_default(); + list.iter().any(|oi| { + let oi = doc.resolve(oi); + let Some(d) = oi.as_dict() else { return false }; + d.name(b"S") == Some(b"GTS_PDFA1") && d.get(b"DestOutputProfile").is_some_and(|p| matches!(&*doc.resolve(p), Object::Stream(_))) + }) +} + fn has_pdfa_intent(doc: &Document) -> bool { pdfa_intent_profile(doc).is_some() } @@ -369,6 +438,37 @@ fn referenced_content(doc: &Document, objs: &[(ObjRef, std::sync::Arc)]) out } +/// The fonts the document draws with: those a resource dictionary names (`/Resources`, of a page or +/// a page tree node, a form XObject or an annotation's appearance stream, a pattern or a Type 3 +/// font). PDF/A asks that every font "used for rendering" be embedded (ISO 19005-2, 6.2.11.4.1). A +/// font that only the interactive form's default resources hold (`/AcroForm /DR`, ISO 32000-1 +/// 12.7.2) draws nothing, so it needn't be: an appearance stream that does draw with it names it in +/// its own `/Resources`. +fn drawing_fonts(doc: &Document, objs: &[(ObjRef, std::sync::Arc)]) -> std::collections::HashSet { + let mut out = std::collections::HashSet::new(); + for (_, o) in objs { + let Some(resources) = dict_of(o).and_then(|d| d.get(b"Resources")).map(|r| doc.resolve(r)) else { continue }; + let Some(fonts) = resources.as_dict().and_then(|r| r.get(b"Font")).map(|f| doc.resolve(f)) else { continue }; + if let Some(fonts) = fonts.as_dict() { + out.extend(fonts.iter().filter_map(|(_, f)| f.as_ref()).map(|r| font_object(doc, r))); + } + } + out +} + +/// How many references [`font_object`] follows, as the document's own `resolve` does. +const MAX_REFERENCE_HOPS: usize = 32; + +/// The object a font entry's reference `r` ends at: an object can be nothing but a reference to +/// another (ISO 32000-1 7.3.10), and the font dictionary is the last one in that chain. +fn font_object(doc: &Document, mut r: ObjRef) -> ObjRef { + for _ in 0..MAX_REFERENCE_HOPS { + let Object::Ref(next) = &*doc.get(r) else { break }; + r = *next; + } + r +} + /// Whether a stream's own dictionary says it holds content: a form XObject, or a stream with a /// `/BBox` and no `/Subtype` (a tiling pattern, or an appearance stream that leaves out /// `/Subtype /Form`). Mesh shadings may carry a `/BBox` too, but their data isn't content. @@ -486,7 +586,48 @@ pub fn verify(doc: &Document, level: Level) -> Vec { let message = fill(pattern, None, &values); push(Issue { clause, message, pattern, value: None, values, page: None, object: Some(object), fixable }); }; - let cat = catalog(doc); + check_identification(doc, level, &mut issue); + let objs = objects(doc); + check_xmp_packets(&objs, &located_issue); + check_output_intent(doc, &objs, &mut issue); + check_catalog_actions(doc, &mut issue); + // Pages: annotations and page actions. + let pages = pdfcraft_model::pages(doc); + // Each annotation's page: the first page that lists it. + let mut annot_pages: std::collections::HashMap = std::collections::HashMap::new(); + for (i, p) in pages.iter().enumerate() { + if let Some(a) = p.dict.get(b"Annots").map(|a| doc.resolve(a)) + && let Some(a) = a.as_array() + { + #[cfg(test)] + ANNOT_ENTRIES_READ.with(|n| n.set(n.get() + a.len())); + for r in a.iter().filter_map(Object::as_ref) { + annot_pages.entry(r).or_insert(i); + } + } + } + let page_of = |r: ObjRef| annot_pages.get(&r).copied(); + for (i, p) in pages.iter().enumerate() { + if p.dict.get(b"AA").is_some() { + issue("6.5.2", "Page {} has additional actions", Some((i + 1).to_string()), Some(i), true); + } + } + check_transparency(doc, &pages, &mut issue); + let drawing = drawing_fonts(doc, &objs); + check_objects(doc, level, &objs, &drawing, &page_of, &mut issue); + check_limits(doc, &objs, &pages, &mut issue); + check_spots(doc, &objs, &mut issue); + check_graphics_states(doc, &mut issue); + // One message per kind of problem is enough for repeated ones (as preflight summaries do). + let mut out = out.into_inner(); + let mut seen = std::collections::HashSet::new(); + out.retain(|i| seen.insert((i.clause, i.message.clone(), i.page))); + out +} + +/// 6.1.2, 6.1.3 and 6.6: identification and metadata, the file header, the file identifier and +/// encryption. +fn check_identification(doc: &Document, level: Level, issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool)) { // 6.1.3 / 6.6: identification and metadata. match metadata(doc) { None => issue("6.6.2.1", "The document has no XMP metadata", None, None, true), @@ -503,16 +644,37 @@ pub fn verify(doc: &Document, level: Level) -> Vec { } } } - // 6.1.3: encryption. + // 6.1.2: the file header. + let header = header::check(doc); + if header.version { + issue("6.1.2", "The file doesn't start with a %PDF-1.0 to %PDF-1.7 header line", None, None, true); + } + if header.comment { + issue("6.1.2", "The file header isn't followed by a comment of four bytes above 127", None, None, true); + } + // 6.1.3: the file identifier and encryption. + match trailer::check(doc) { + Some(trailer::IdProblem::Missing) => issue("6.1.3", "The trailer has no file identifier (ID)", None, None, true), + Some(trailer::IdProblem::Malformed) => { + issue("6.1.3", "The file identifier (ID) in the trailer isn't two non-empty strings", None, None, true) + } + None => {} + } if doc.trailer().get(b"Encrypt").is_some() { issue("6.1.3", "The document is encrypted", None, None, true); } - // 6.2.2: output intent when device colour is used. - let objs = objects(doc); - // 6.6.2.3.1: the value types of predefined XMP properties, in every metadata stream. - let mut mistyped = Vec::new(); - xmp_types::for_each_packet(&objs, |object, packet| { +} + +/// 6.6.2.3.1 and 6.6.2.3.3: predefined XMP value types and extension schema descriptions, in +/// every metadata stream. +fn check_xmp_packets( + objs: &[(ObjRef, std::sync::Arc)], + located_issue: &impl Fn(&'static str, &'static str, Vec<(&'static str, String)>, ObjRef, bool), +) { + let (mut mistyped, mut extension) = (Vec::new(), xmp_extension::Findings::default()); + xmp_types::for_each_packet(objs, |object, packet| { mistyped.extend(xmp_types::mistyped(packet).into_iter().map(|((name, written), ty)| (object, name, written, ty))); + extension.add(object, packet); }); for (object, name, written, ty) in mistyped { let mut values = vec![("{property}", name.clone()), ("{object}", object.to_string()), ("{type}", ty)]; @@ -535,7 +697,104 @@ pub fn verify(doc: &Document, level: Level) -> Vec { ); } } - let used = uses_device_colour(doc, &objs); + report_xmp_extension(extension, located_issue); +} + +/// 6.6.2.3.3: what the extension schema descriptions of the metadata streams lack. +fn report_xmp_extension( + extension: xmp_extension::Findings, + located_issue: &impl Fn(&'static str, &'static str, Vec<(&'static str, String)>, ObjRef, bool), +) { + for m in extension.missing { + let mut values = vec![("{schema}", m.schema), ("{object}", m.object.to_string()), ("{entry}", m.entry)]; + match (m.item, m.empty) { + (None, false) => located_issue( + "6.6.2.3.3", + "The XMP extension schema {schema} in metadata stream {object} has no {entry} entry", + values, + m.object, + false, + ), + (None, true) => located_issue( + "6.6.2.3.3", + "The XMP extension schema {schema} in metadata stream {object} has an empty {entry} entry", + values, + m.object, + false, + ), + (Some(item), false) => { + values.push(("{item}", item)); + located_issue( + "6.6.2.3.3", + "The XMP extension schema {schema} in metadata stream {object} describes {item} without a {entry} entry", + values, + m.object, + false, + ); + } + (Some(item), true) => { + values.push(("{item}", item)); + located_issue( + "6.6.2.3.3", + "The XMP extension schema {schema} in metadata stream {object} describes {item} with an empty {entry} entry", + values, + m.object, + false, + ); + } + } + } + for c in extension.not_bag { + let object = c.object; + if c.found.is_empty() { + located_issue( + "6.6.2.3.3", + "The XMP extension schema container in metadata stream {object} holds no rdf:Bag", + vec![("{object}", object.to_string())], + object, + false, + ); + } else { + let values = vec![("{object}", object.to_string()), ("{found}", c.found)]; + located_issue( + "6.6.2.3.3", + "The XMP extension schema container in metadata stream {object} holds {found}, not one rdf:Bag", + values, + object, + false, + ); + } + } + for p in extension.prefixes { + let mut values = vec![("{object}", p.object.to_string()), ("{namespace}", p.namespace.to_owned()), ("{prefix}", p.own.to_owned())]; + if p.written.is_empty() { + located_issue( + "6.6.2.3.3", + "Metadata stream {object} writes the namespace {namespace} without a prefix, not with {prefix}", + values, + p.object, + false, + ); + } else { + values.push(("{written}", p.written)); + located_issue( + "6.6.2.3.3", + "Metadata stream {object} writes the namespace {namespace} with the prefix {written}, not {prefix}", + values, + p.object, + false, + ); + } + } +} + +/// 6.2.3 and 6.2.4.3: the output intent device colour needs. +fn check_output_intent( + doc: &Document, + objs: &[(ObjRef, std::sync::Arc)], + issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool), +) { + let used = uses_device_colour(doc, objs); match pdfa_intent_profile(doc) { None if used.any() => { issue("6.2.3", "Device colour is used, but there is no PDF/A output intent", None, None, !used.cmyk); @@ -554,7 +813,11 @@ pub fn verify(doc: &Document, level: Level) -> Vec { } None => {} } +} + +fn check_catalog_actions(doc: &Document, issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool)) { // 6.1.3 / 6.6.1: catalog actions. + let cat = catalog(doc); if cat.get(b"AA").is_some() { issue("6.5.2", "The document has additional actions (/AA)", None, None, true); } @@ -568,61 +831,137 @@ pub fn verify(doc: &Document, level: Level) -> Vec { { issue("6.4.1", "The form asks viewers to make appearances (NeedAppearances)", None, None, true); } - // Pages: annotations and page actions. - let pages = pdfcraft_model::pages(doc); - // Each annotation's page: the first page that lists it. - let mut annot_pages: std::collections::HashMap = std::collections::HashMap::new(); +} + +fn check_transparency( + doc: &Document, + pages: &[pdfcraft_model::Page], + issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool), +) { + // 6.2.10: blend modes are those ISO 32000-1 defines, and without a PDF/A output intent (one with + // a profile) a page with transparency names a blending colour space. + let intent = pdfa_intent_has_profile(doc); + // Pages share resources, forms and fonts: what one page found under them, the next takes as is. + let mut shared = transparency::Shared::default(); for (i, p) in pages.iter().enumerate() { - if let Some(a) = p.dict.get(b"Annots").map(|a| doc.resolve(a)) - && let Some(a) = a.as_array() - { - #[cfg(test)] - ANNOT_ENTRIES_READ.with(|n| n.set(n.get() + a.len())); - for r in a.iter().filter_map(Object::as_ref) { - annot_pages.entry(r).or_insert(i); - } + let painting = transparency::page_painting(doc, &p.dict, &mut shared); + for name in painting.unknown_blend_modes { + issue("6.2.10", "A graphics state uses the blend mode {}, which ISO 32000-1 doesn't define", Some(name), Some(i), false); + } + if painting.transparency && !intent && !transparency::page_names_blending_space(doc, &p.dict) { + let n = Some((i + 1).to_string()); + issue("6.2.10", "Page {} has transparency but no device-independent blending colour space or PDF/A output intent", n, Some(i), false); } } - let page_of = |r: ObjRef| annot_pages.get(&r).copied(); +} + +fn check_limits( + doc: &Document, + objs: &[(ObjRef, std::sync::Arc)], + pages: &[pdfcraft_model::Page], + issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool), +) { + // 6.1.13: the implementation limits of ISO 32000-1 Annex C. + let over = limits::exceeded(doc, objs, &referenced_content(doc, objs)); + if over.int { + issue("6.1.13", "An integer is outside the 32-bit range", None, None, false); + } + if over.real { + issue("6.1.13", "A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸", None, None, false); + } + if over.string { + issue("6.1.13", "A string is longer than 32767 bytes", None, None, false); + } + if limits::cid_over_limit(doc, objs) { + issue("6.1.13", "A CMap maps a character code to a CID above 65535", None, None, false); + } + if over.name { + issue("6.1.13", "A name is longer than 127 bytes", None, None, false); + } for (i, p) in pages.iter().enumerate() { - if p.dict.get(b"AA").is_some() { - issue("6.5.2", "Page {} has additional actions", Some((i + 1).to_string()), Some(i), true); + if limits::page_box_out_of_range(doc, &p.dict) { + issue("6.1.13", "Page {} has a boundary smaller than 3 or larger than 14400 units", Some((i + 1).to_string()), Some(i), false); } } - check_objects(doc, level, &objs, &page_of, &mut issue); - // One message per kind of problem is enough for repeated ones (as preflight summaries do). - let mut out = out.into_inner(); - let mut seen = std::collections::HashSet::new(); - out.retain(|i| seen.insert((i.clause, i.message.clone(), i.page))); - out +} + +fn check_spots( + doc: &Document, + objs: &[(ObjRef, std::sync::Arc)], + issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool), +) { + // 6.2.4.4: Separation and DeviceN colour spaces. + let spots = spot::check(doc, objs); + for name in spots.differing { + issue("6.2.4.4", "Separation colour spaces named {} differ in their alternate space or tint transform", Some(name), None, false); + } + for name in spots.undefined { + issue("6.2.4.4", "The spot colour {} of a DeviceN colour space isn't in its Colorants dictionary", Some(name), None, false); + } +} + +fn check_graphics_states(doc: &Document, issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool)) { + // 6.2.5: the extended graphics state. + let gs = gstate::check(doc); + if gs.transfer { + issue("6.2.5", "A graphics state has a transfer function (TR)", None, None, false); + } + if gs.transfer2 { + issue("6.2.5", "A graphics state has a transfer function (TR2) other than Default", None, None, false); + } + if gs.phase { + issue("6.2.5", "A graphics state has a halftone phase (HTP)", None, None, false); + } + for kind in gs.halftone_types { + issue("6.2.5", "A halftone has type {}: only types 1 and 5 are allowed", Some(kind.to_string()), None, false); + } + if gs.untyped { + issue("6.2.5", "A halftone's HalftoneType is missing or isn't an integer", None, None, false); + } + if gs.halftone_name { + issue("6.2.5", "A halftone has a HalftoneName", None, None, false); + } + if gs.halftone_transfer { + issue("6.2.5", "A halftone has a transfer function, which only a non-primary colorant's halftone may have", None, None, false); + } + for colorant in gs.missing_transfer { + issue("6.2.5", "The halftone for the colorant {} has no transfer function", Some(colorant), None, false); + } + for intent in gs.intents { + issue("6.2.5", "A graphics state has the rendering intent {}, which isn't a standard one", Some(intent), None, false); + } + if gs.intent_not_name { + issue("6.2.5", "A graphics state has a rendering intent that isn't a name", None, None, false); + } } fn check_objects( doc: &Document, level: Level, objects: &[(ObjRef, std::sync::Arc)], + drawing: &std::collections::HashSet, page_of: &impl Fn(ObjRef) -> Option, issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool), ) { for (r, o) in objects { let Some(d) = dict_of(o) else { continue }; - check_one_object(doc, level, *r, d, o, page_of, issue); + check_one_object(doc, level, d, o, drawing.contains(r), page_of(*r), issue); } } fn check_one_object( doc: &Document, level: Level, - reference: ObjRef, dict: &Dict, object: &Object, - page_of: &impl Fn(ObjRef) -> Option, + draws: bool, + page: Option, issue: &mut impl FnMut(&'static str, &'static str, Option, Option, bool), ) { let ty = dict.name(b"Type"); let subtype = dict.name(b"Subtype"); check_actions(doc, dict, issue); - if is_annotation(dict, ty, subtype) && check_annotation(doc, dict, subtype, page_of(reference), issue) { + if is_annotation(dict, ty, subtype) && check_annotation(doc, dict, subtype, page, issue) { // A forbidden annotation stops every later check for its object, as the scan did before // it was split into helpers: the rest of its own checks and the form field, font, stream // and embedded file below all came after that `continue`. @@ -631,7 +970,8 @@ fn check_one_object( if dict.get(b"FT").is_some() && dict.get(b"AA").is_some() && ty != Some(b"Annot") { issue("6.5.2", "A form field has additional actions", None, None, true); } - if ty == Some(b"Font") || dict.get(b"BaseFont").is_some() && subtype.is_some() { + // A Type 0 font's descendant is checked through it. + if draws && (ty == Some(b"Font") || dict.get(b"BaseFont").is_some() && subtype.is_some()) { check_font(doc, dict, issue); } if let Object::Stream(stream) = object { @@ -829,10 +1169,8 @@ fn walk_chain(doc: &Document, a: &Dict, read: &mut usize, seen: &mut std::collec if *read > MAX_ACTION_CHAIN { return Some("too deeply chained".into()); } - if let Some(s) = d.name(b"S") - && FORBIDDEN_ACTIONS.contains(&s) - { - return Some(String::from_utf8_lossy(s).into_owned()); + if let Some(s) = action_forbidden(doc, d) { + return Some(s); } match d.get(b"Next") { Some(Object::Dict(next)) => stack.push(next), @@ -849,6 +1187,24 @@ fn walk_chain(doc: &Document, a: &Dict, read: &mut usize, seen: &mut std::collec None } +/// The forbidden kind of the action `d` itself, if it is one: a forbidden type, or a Named +/// action for anything but the four of [`NAMED_ACTIONS`]. The type and the name may each be given +/// directly or by reference. +fn action_forbidden(doc: &Document, d: &Dict) -> Option { + let s = doc.resolve(d.get(b"S")?); + let s = s.as_name()?; + if FORBIDDEN_ACTIONS.contains(&s) { + return Some(String::from_utf8_lossy(s).into_owned()); + } + let n = d.get(b"N").map(|n| doc.resolve(n)); + match n.as_deref().and_then(Object::as_name) { + _ if s != b"Named" => None, + Some(n) if NAMED_ACTIONS.contains(&n) => None, + Some(n) => Some(format!("Named {}", String::from_utf8_lossy(n))), + None => Some("Named".into()), + } +} + /// Whether `o` is (or refers to) a forbidden action or one chaining to it, or to nothing. fn forbidden(doc: &Document, o: &Object) -> bool { match &*doc.resolve(o) { @@ -874,6 +1230,9 @@ pub fn convert(doc: &mut Document, level: Level) -> Result bool { + self.int && self.real && self.string && self.name + } + + fn name(&mut self, n: &[u8]) { + self.name |= n.len() > MAX_NAME; + } + + /// One value and, within [`MAX_DEPTH`], what it holds: a dictionary's keys count as names. + fn value(&mut self, o: &Object, depth: u8) { + if self.all() || depth > MAX_DEPTH { + return; + } + match o { + Object::Int(i) => self.int |= !(MIN_INT..=MAX_INT).contains(i), + Object::Real(r) => self.real |= !r.is_finite() || r.abs() > MAX_REAL || (*r != 0.0 && r.abs() < MIN_REAL), + Object::String(s) => self.string |= s.bytes.len() > MAX_STRING, + Object::Name(n) => self.name(n), + Object::Array(a) => { + for v in a { + self.value(v, depth.saturating_add(1)); + } + } + Object::Dict(d) => self.dict(d, depth), + Object::Stream(s) => self.dict(&s.dict, depth), + Object::Null | Object::Bool(_) | Object::Ref(_) => {} + } + } + + /// The operands of one content stream, inline images' dictionaries included. + fn content(&mut self, data: &[u8]) { + for op in pdfcraft_content::ops(data) { + if self.all() { + return; + } + for v in &op.operands { + self.value(v, 0); + } + if let Some((image, _)) = &op.inline { + self.dict(image, 0); + } + } + } + + fn dict(&mut self, d: &Dict, depth: u8) { + for (k, v) in d.iter() { + self.name(k); + self.value(v, depth.saturating_add(1)); + } + } +} + +/// The limits that the trailer, the indirect objects `objs` and the content streams go past. +/// `content` holds the content streams only a reference identifies (a page's `/Contents`, Type 3 +/// glyph procedures), besides form XObjects and appearance streams, which say so themselves. +pub(crate) fn exceeded(doc: &Document, objs: &[(ObjRef, std::sync::Arc)], content: &HashSet) -> Exceeded { + let mut found = Exceeded::default(); + found.dict(doc.trailer(), 0); + for (_, o) in objs { + if found.all() { + return found; + } + found.value(o, 0); + } + let mut budget = DecodeBudget::new(MAX_CONTENT_BYTES, 0); + let mut read: HashSet = HashSet::new(); + for page in pdfcraft_model::pages(doc) { + if found.all() || budget.remaining() == 0 { + return found; + } + let refs = contents(doc, &page.dict); + let mut joined = Vec::new(); + for r in &refs { + if let Object::Stream(s) = &*doc.get(*r) + && let Ok(data) = budget.decode(s) + { + joined.extend_from_slice(&data); + joined.push(b' '); + } + } + read.extend(refs); + found.content(&joined); + } + for (r, o) in objs { + if found.all() || budget.remaining() == 0 { + break; + } + if let Object::Stream(s) = &**o + && !read.contains(r) + && (content.contains(r) || crate::is_content(&s.dict)) + && let Ok(data) = budget.decode(s) + { + found.content(&data); + } + } + found +} + +/// A page's `/Contents` streams, in order. +fn contents(doc: &Document, page: &Dict) -> Vec { + match page.get(b"Contents") { + Some(Object::Ref(r)) => match &*doc.get(*r) { + Object::Array(a) => a.iter().filter_map(Object::as_ref).collect(), + _ => vec![*r], + }, + Some(Object::Array(a)) => a.iter().filter_map(Object::as_ref).collect(), + _ => Vec::new(), + } +} + +/// Whether one of a page's boundaries (with the media and crop boxes it inherits) is narrower or +/// lower than 3 units, or wider or higher than 14400. +pub(crate) fn page_box_out_of_range(doc: &Document, page: &Dict) -> bool { + BOXES.iter().filter_map(|k| page.get(k)).any(|b| { + let b = doc.resolve(b); + let v: Vec = b.as_array().map(|a| a.iter().filter_map(|x| doc.resolve(x).as_f64()).collect()).unwrap_or_default(); + let [x0, y0, x1, y1] = v.as_slice() else { return false }; + let (w, h) = ((x1 - x0).abs(), (y1 - y0).abs()); + // A size past `f64` (corners near ±1e308) is as much too large as any other. + [w, h].iter().any(|s| !(MIN_PAGE..=MAX_PAGE).contains(s)) + }) +} + +/// Whether an embedded CMap of a Type 0 font among `objs`, or one it uses through `/UseCMap`, maps +/// a code to a CID above 65535: a `begincidrange` entry ` cid` reaches +/// `cid + high − low`, a `begincidchar` entry ` cid` reaches `cid` (ISO 32000-1 9.7.5.4). A +/// CMap named rather than embedded is a predefined one, within the limit. The CMaps are decoded +/// within one allowance for the document. +pub(crate) fn cid_over_limit(doc: &Document, objs: &[(ObjRef, std::sync::Arc)]) -> bool { + let mut budget = DecodeBudget::new(MAX_CMAP_BYTES, 0); + let mut read: HashSet = HashSet::new(); + for (_, o) in objs { + let Some(d) = o.as_dict() else { continue }; + if d.name(b"Subtype") != Some(b"Type0") { + continue; + } + let mut next = d.get(b"Encoding").and_then(Object::as_ref); + for _ in 0..MAX_CMAP_CHAIN { + let Some(r) = next.filter(|r| read.insert(*r)) else { break }; + if budget.remaining() == 0 { + return false; + } + let Object::Stream(cmap) = &*doc.get(r) else { break }; + if budget.decode(cmap).is_ok_and(|data| maps_past_limit(&data)) { + return true; + } + next = cmap.dict.get(b"UseCMap").and_then(Object::as_ref); + } + } + false +} + +/// Whether the CMap program `data` maps a code to a CID above [`MAX_CID`]. The entries of a +/// `begincidrange` … `endcidrange` block are the operands of its `endcidrange`, read as content is. +fn maps_past_limit(data: &[u8]) -> bool { + let value = |o: &Object| -> Option { + match o { + Object::String(s) if s.bytes.len() <= 4 => Some(s.bytes.iter().fold(0u64, |v, b| (v << 8) | u64::from(*b))), + Object::Int(i) => u64::try_from(*i).ok(), + _ => None, + } + }; + pdfcraft_content::ops(data).any(|op| match op.op.as_slice() { + b"endcidrange" => op.operands.as_chunks::<3>().0.iter().any(|[low, high, cid]| match (value(low), value(high), value(cid)) { + (Some(low), Some(high), Some(cid)) => cid.saturating_add(high.saturating_sub(low)) > MAX_CID, + _ => false, + }), + b"endcidchar" => op.operands.as_chunks::<2>().0.iter().any(|[_, cid]| value(cid).is_some_and(|cid| cid > MAX_CID)), + _ => false, + }) +} diff --git a/crates/preflight/src/spot.rs b/crates/preflight/src/spot.rs new file mode 100644 index 0000000000..8367d8d28a --- /dev/null +++ b/crates/preflight/src/spot.rs @@ -0,0 +1,212 @@ +//! PDF/A-2 6.2.4.4, Separation and DeviceN colour spaces: +//! +//! - Separation spaces of the same colorant name agree on their alternate space and tint +//! transform, also where a DeviceN space's `/Colorants` dictionary holds them. They are compared +//! as objects, whether stored directly or by reference, at any depth (ISO 32000-1 7.3.10); +//! - a DeviceN space names each spot colorant it uses in the `/Colorants` dictionary of its +//! attributes (ISO 32000-1 8.6.6.5, Table 71); a null or missing entry defines nothing. The +//! process colorants (`Cyan`, `Magenta`, `Yellow`, `Black`, and those an NChannel space's +//! `/Process` lists) and `None` are not spot colorants. +//! +//! Colour spaces are found wherever they are written: in resource dictionaries, images, shadings +//! and other colour spaces, direct or by reference. + +use std::collections::{BTreeSet, HashMap, HashSet}; + +use pdfcraft_cos::{DecodeBudget, Dict, Document, ObjRef, Object, Stream}; + +/// How deep a direct object is followed (arrays and dictionaries inside one another): as deep as +/// the parser nests them. +const MAX_DEPTH: u8 = 128; +/// How deep two definitions are compared, references followed. +const MAX_COMPARE_DEPTH: u8 = 32; +/// How many object pairs the comparisons of one document visit in all. Past it, a comparison +/// claims no difference. +const MAX_COMPARE_STEPS: u32 = 1 << 20; +/// The decoded bytes the stream comparisons of one document may spend in all. +const MAX_COMPARE_BYTES: usize = 64 << 20; +/// The stream entries that say how the data is stored, not what it is. +const ENCODING: [&[u8]; 3] = [b"Length", b"Filter", b"DecodeParms"]; +/// The colorants of DeviceCMYK, which a DeviceN space may name without defining them. +const PROCESS: [&[u8]; 4] = [b"Cyan", b"Magenta", b"Yellow", b"Black"]; + +/// What the colour spaces of a document break of 6.2.4.4. +#[derive(Debug, Default, PartialEq, Eq)] +pub(crate) struct Findings { + /// Colorant names whose Separation spaces differ, in name order. + pub(crate) differing: BTreeSet, + /// Spot colorants a DeviceN space uses without a `/Colorants` entry, in name order. + pub(crate) undefined: BTreeSet, +} + +/// The 6.2.4.4 findings over the indirect objects `objs`. +pub(crate) fn check(doc: &Document, objs: &[(ObjRef, std::sync::Arc)]) -> Findings { + let compare = Compare { doc, equal: HashSet::new(), steps: MAX_COMPARE_STEPS, budget: DecodeBudget::new(MAX_COMPARE_BYTES, 0) }; + let mut scan = Scan { doc, compare, separations: HashMap::new(), findings: Findings::default() }; + for (_, o) in objs { + scan.walk(o, 0); + } + scan.findings +} + +struct Scan<'a> { + doc: &'a Document, + compare: Compare<'a>, + /// Each colorant name's first Separation space: its alternate space and tint transform. + separations: HashMap, (Object, Object)>, + findings: Findings, +} + +impl Scan<'_> { + fn walk(&mut self, o: &Object, depth: u8) { + if depth > MAX_DEPTH { + return; + } + match o { + Object::Array(a) => { + match a.first().and_then(Object::as_name) { + Some(b"Separation") => self.separation(a), + Some(b"DeviceN") => self.devicen(a), + _ => {} + } + for v in a { + self.walk(v, depth.saturating_add(1)); + } + } + Object::Dict(d) => { + for (_, v) in d.iter() { + self.walk(v, depth.saturating_add(1)); + } + } + Object::Stream(s) => { + for (_, v) in s.dict.iter() { + self.walk(v, depth.saturating_add(1)); + } + } + _ => {} + } + } + + /// `[/Separation name alternateSpace tintTransform]`: the same name must keep the same + /// alternate space and tint transform. + fn separation(&mut self, a: &[Object]) { + let [_, name, alternate, tint, ..] = a else { return }; + let name = self.doc.resolve(name); + let Some(name) = name.as_name() else { return }; + let this = (alternate.clone(), tint.clone()); + match self.separations.get(name) { + Some((alt, fun)) => { + if !self.compare.same(alt, &this.0, 0) || !self.compare.same(fun, &this.1, 0) { + self.findings.differing.insert(String::from_utf8_lossy(name).into_owned()); + } + } + None => { + self.separations.insert(name.to_vec(), this); + } + } + } + + /// `[/DeviceN names alternateSpace tintTransform attributes]`: each spot colorant in `names` + /// needs an entry in the attributes' `/Colorants`. + fn devicen(&mut self, a: &[Object]) { + let Some(names) = a.get(1).map(|n| self.doc.resolve(n)) else { return }; + let Some(names) = names.as_array() else { return }; + let attributes = a.get(4).map(|d| self.doc.resolve(d)); + let attributes = attributes.as_deref().and_then(Object::as_dict); + let colorants = attributes.and_then(|d| d.get(b"Colorants")).map(|c| self.doc.resolve(c)); + let colorants = colorants.as_deref().and_then(Object::as_dict); + let nchannel = attributes.and_then(|d| d.get(b"Subtype")).map(|t| self.doc.resolve(t)).is_some_and(|t| t.as_name() == Some(b"NChannel")); + let process = attributes + .filter(|_| nchannel) + .and_then(|d| d.get(b"Process")) + .map(|p| self.doc.resolve(p)) + .and_then(|p| p.as_dict().and_then(|p| p.get(b"Components")).map(|c| self.doc.resolve(c))); + let process: Vec> = process + .as_deref() + .and_then(Object::as_array) + .map(|c| c.iter().filter_map(|n| self.doc.resolve(n).as_name().map(<[u8]>::to_vec)).collect()) + .unwrap_or_default(); + for n in names.iter().map(|n| self.doc.resolve(n)) { + let Some(n) = n.as_name() else { continue }; + let spot = n != b"None" && !PROCESS.contains(&n) && !process.iter().any(|p| p == n); + // Each entry is a Separation space for the colorant it is keyed by (ISO 32000-1 8.6.6.5, + // Table 71); a null entry, or one referring to no object, defines nothing (7.3.9). + let defined = colorants.and_then(|c| c.get(n)).is_some_and(|e| self.separation_named(e, n)); + if spot && !defined { + self.findings.undefined.insert(String::from_utf8_lossy(n).into_owned()); + } + } + } + + /// Whether `o` is `[/Separation name ...]` for the colorant `name`. + fn separation_named(&self, o: &Object, name: &[u8]) -> bool { + let o = self.doc.resolve(o); + let Some([kind, n, ..]) = o.as_array().map(Vec::as_slice) else { return false }; + kind.as_name() == Some(b"Separation") && self.doc.resolve(n).as_name() == Some(name) + } +} + +/// Compares definitions, remembering which reference pairs it found equal, so a structure shared +/// by many definitions is compared once, within [`MAX_COMPARE_STEPS`] and [`MAX_COMPARE_BYTES`]. +struct Compare<'a> { + doc: &'a Document, + /// Reference pairs compared equal. A depth cut-off only ever answers "different", so an + /// equal answer holds at any depth. + equal: HashSet<(ObjRef, ObjRef)>, + steps: u32, + budget: DecodeBudget, +} + +impl Compare<'_> { + /// Whether `a` and `b` are the same object, wherever their parts are stored: references are + /// followed (to [`MAX_COMPARE_DEPTH`]), so two identical streams under different object + /// numbers are the same, and streams compare by their decoded data. Once the steps or the + /// decode allowance run out, nothing more is called different. + fn same(&mut self, a: &Object, b: &Object, depth: u8) -> bool { + let pair = match (a, b) { + (Object::Ref(x), Object::Ref(y)) if x == y || self.equal.contains(&(*x, *y)) => return true, + (Object::Ref(x), Object::Ref(y)) => Some((*x, *y)), + _ => None, + }; + let Some(steps) = self.steps.checked_sub(1) else { return true }; + self.steps = steps; + if depth > MAX_COMPARE_DEPTH { + return false; + } + let (a, b) = (self.doc.resolve(a), self.doc.resolve(b)); + let deeper = depth.saturating_add(1); + let same = match (&*a, &*b) { + (Object::Array(x), Object::Array(y)) => x.len() == y.len() && x.iter().zip(y).all(|(p, q)| self.same(p, q, deeper)), + (Object::Dict(x), Object::Dict(y)) => self.same_dict(x, y, &[], deeper), + (Object::Stream(x), Object::Stream(y)) => self.same_data(x, y, deeper) && self.same_dict(&x.dict, &y.dict, &ENCODING, deeper), + (x, y) => x == y, + }; + if same && let Some(pair) = pair { + self.equal.insert(pair); + } + same + } + + /// Whether two dictionaries hold the same entries, those keyed in `skip` aside. + fn same_dict(&mut self, a: &Dict, b: &Dict, skip: &[&[u8]], depth: u8) -> bool { + let kept = |d: &Dict| d.iter().filter(|(k, _)| !skip.contains(&k.as_slice())).count(); + kept(a) == kept(b) && a.iter().filter(|(k, _)| !skip.contains(&k.as_slice())).all(|(k, v)| b.get(k).is_some_and(|w| self.same(v, w, depth))) + } + + /// Whether two streams hold the same data, however each is encoded. Data that can't be + /// decoded within the allowance is not called different. + fn same_data(&mut self, a: &Stream, b: &Stream, depth: u8) -> bool { + let stored_alike = |c: &mut Self, k: &[u8]| match (a.dict.get(k), b.dict.get(k)) { + (None, None) => true, + (Some(x), Some(y)) => c.same(x, y, depth), + _ => false, + }; + if a.raw == b.raw && stored_alike(self, b"Filter") && stored_alike(self, b"DecodeParms") { + return true; + } + match (self.budget.decode(a), self.budget.decode(b)) { + (Ok(x), Ok(y)) => x == y, + _ => true, + } + } +} diff --git a/crates/preflight/src/tests.rs b/crates/preflight/src/tests.rs index e691adae84..e0107d9f21 100644 --- a/crates/preflight/src/tests.rs +++ b/crates/preflight/src/tests.rs @@ -1,7 +1,11 @@ use super::*; fn build(objs: &[&str], trailer_extra: &str) -> Document { - let mut out = b"%PDF-1.7\n".to_vec(); + build_with_header(b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n", objs, trailer_extra) +} + +fn build_with_header(header: &[u8], objs: &[&str], trailer_extra: &str) -> Document { + let mut out = header.to_vec(); let mut offsets = Vec::new(); for (i, o) in objs.iter().enumerate() { offsets.push(out.len()); @@ -491,12 +495,279 @@ fn an_output_intent_profile_header_is_read_within_a_size_cap() { } } +/// PDF/A-2 6.1.13 holds a file to the implementation limits of ISO 32000-1 Annex C (Table C.1): +/// an integer past 32 bits, a string over 32767 bytes or a name over 127 bytes (decoded: a name +/// written with `#xx` escapes counts its bytes once read), in an object or in content; and a page +/// boundary under 3 or over 14400 units, read after inheritance, so a page that overrides its +/// parent's box is judged by its own. +#[test] +fn implementation_limits_are_reported() { + let limits = + |doc: &Document| -> Vec { verify(doc, Level::A2b).into_iter().filter(|i| i.clause == "6.1.13").map(|i| i.message).collect() }; + let doc = |extra: &str, content: &str, parent_box: &str, page_box: &str| { + let objects = [ + format!("<< /Type /Catalog /Pages 2 0 R {extra} >>"), + format!("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox {parent_box} >>"), + format!("<< /Type /Page /Parent 2 0 R /Contents 4 0 R {page_box} >>"), + format!("<< /Length {} >>\nstream\n{content}\nendstream", content.len()), + ]; + let refs: Vec<&str> = objects.iter().map(String::as_str).collect(); + build(&refs, "") + }; + let ok = "[0 0 200 200]"; + assert!(limits(&doc("", "", ok, "")).is_empty(), "{:?}", limits(&doc("", "", ok, ""))); + assert_eq!(limits(&doc("/Big 2147483648", "", ok, "")), ["An integer is outside the 32-bit range"]); + assert_eq!(limits(&doc("/Small -2147483649", "", ok, "")), ["An integer is outside the 32-bit range"]); + assert!(limits(&doc("/Edge -2147483648 /Top 2147483647", "", ok, "")).is_empty()); + assert_eq!(limits(&doc("", "2147483648 0 Td", ok, "")), ["An integer is outside the 32-bit range"]); + let long = "x".repeat(32768); + assert_eq!(limits(&doc(&format!("/S ({long})"), "", ok, "")), ["A string is longer than 32767 bytes"]); + assert_eq!(limits(&doc("", &format!("BT ({long}) Tj ET"), ok, "")), ["A string is longer than 32767 bytes"]); + assert!(limits(&doc(&format!("/S ({})", "x".repeat(32767)), "", ok, "")).is_empty()); + let name = "n".repeat(128); + assert_eq!(limits(&doc(&format!("/V /{name}"), "", ok, "")), ["A name is longer than 127 bytes"]); + assert_eq!(limits(&doc(&format!("/{name} 1"), "", ok, "")), ["A name is longer than 127 bytes"]); + assert_eq!(limits(&doc("", &format!("/{name} Do"), ok, "")), ["A name is longer than 127 bytes"]); + // 200 bytes as written, 66 once the escapes are read. + let escaped = format!("A{}", "#41".repeat(65) + &"B".repeat(4)); + assert!(limits(&doc(&format!("/V /{escaped}"), "", ok, "")).is_empty()); + assert_eq!(limits(&doc("", "", "[0 0 2 200]", "")), ["Page 1 has a boundary smaller than 3 or larger than 14400 units"]); + assert_eq!(limits(&doc("", "", ok, "/CropBox [0 0 14401 200]")), ["Page 1 has a boundary smaller than 3 or larger than 14400 units"]); + assert!(limits(&doc("", "", "[0 0 2 2]", "/MediaBox [0 0 3 14400]")).is_empty(), "the page's own box overrides its parent's"); +} + +/// 6.1.13 also reads the trailer, a page's `/Contents` streams joined (operands may end one +/// stream and their operator start the next, ISO 32000-1 7.8.2), values nested as deep as the +/// parser allows, and a page box whose size overflows to infinity. +#[test] +fn implementation_limits_reach_the_trailer_split_content_deep_values_and_huge_boxes() { + let limits = + |doc: &Document| -> Vec { verify(doc, Level::A2b).into_iter().filter(|i| i.clause == "6.1.13").map(|i| i.message).collect() }; + let stream = |data: &str| { + format!( + "<< /Length {} >> +stream +{data} +endstream", + data.len() + ) + }; + let page = |contents: &str, catalog_extra: &str, media: &str, trailer: &str| { + let (first, second) = contents.split_once('|').unwrap_or((contents, "")); + build( + &[ + &format!("<< /Type /Catalog /Pages 2 0 R {catalog_extra} >>"), + &format!("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox {media} >>"), + "<< /Type /Page /Parent 2 0 R /Contents [4 0 R 5 0 R] >>", + &stream(first), + &stream(second), + ], + trailer, + ) + }; + let ok = "[0 0 200 200]"; + assert!(limits(&page("|", "", ok, "")).is_empty()); + assert_eq!(limits(&page("|", "", ok, "/Custom 2147483648")), ["An integer is outside the 32-bit range"]); + let long = "x".repeat(32768); + assert_eq!(limits(&page(&format!("BT ({long})|Tj ET"), "", ok, "")), ["A string is longer than 32767 bytes"]); + let deep = format!("/Deep {}2147483648{}", "[".repeat(40), "]".repeat(40)); + assert_eq!(limits(&page("|", &deep, ok, "")), ["An integer is outside the 32-bit range"]); + // Corners near ±1e308 are reals past the limit too, and the width overflows to infinity. + assert_eq!( + limits(&page("|", "", "[-1e308 0 1e308 200]", "")), + [ + "A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸", + "Page 1 has a boundary smaller than 3 or larger than 14400 units" + ] + ); +} + +/// 6.1.13's two remaining limits of ISO 32000-1 Annex C: a real number beyond ±3.403 × 10³⁸ (or +/// nonzero and nearer zero than ±1.175 × 10⁻³⁸), and +/// a CID above 65535 that an embedded CMap of a Type 0 font maps to (a `begincidrange` reaching +/// past it, a `begincidchar` naming one, or a CMap it uses through `/UseCMap`). +#[test] +fn implementation_limits_cover_reals_and_cids() { + let limits = + |doc: &Document| -> Vec { verify(doc, Level::A2b).into_iter().filter(|i| i.clause == "6.1.13").map(|i| i.message).collect() }; + let stream = |dict: &str, data: &str| format!("<< {dict} /Length {} >>\nstream\n{data}\nendstream", data.len()); + let doc = |catalog_extra: &str, cmap: &str, used: &str| { + let first = stream("/Type /CMap /CMapName /Test-A /UseCMap 6 0 R", cmap); + let second = stream("/Type /CMap /CMapName /Test-B", used); + build( + &[ + &format!("<< /Type /Catalog /Pages 2 0 R {catalog_extra} >>"), + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + "<< /Type /Page /Parent 2 0 R /Resources << /Font << /F 4 0 R >> >> >>", + "<< /Type /Font /Subtype /Type0 /BaseFont /Test /Encoding 5 0 R /DescendantFonts [] >>", + &first, + &second, + ], + "", + ) + }; + let range = |low: &str, high: &str, cid: u32| format!("1 begincidrange <{low}> <{high}> {cid} endcidrange"); + let near = format!("34{}.0", "0".repeat(37)); + let far = format!("4{}.0", "0".repeat(38)); + assert!(limits(&doc(&format!("/R {near} /S -{near}"), &range("0000", "00ff", 65280), "")).is_empty(), "both at their limits"); + assert_eq!(limits(&doc(&format!("/R {far}"), "", "")), ["A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸"]); + assert_eq!(limits(&doc(&format!("/R -{far}"), "", "")), ["A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸"]); + // A nonzero real nearer zero than ±1.175 × 10⁻³⁸ is not allowed either; zero and the + // smallest allowed magnitude are. + let tiny = format!("0.{}1", "0".repeat(39)); + let smallest = format!("0.{}1175", "0".repeat(37)); + assert!(limits(&doc(&format!("/Z 0.0 /M -0.0 /S {smallest} /T -{smallest}"), "", "")).is_empty(), "zero and the smallest real"); + assert_eq!(limits(&doc(&format!("/R {tiny}"), "", "")), ["A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸"]); + assert_eq!(limits(&doc(&format!("/R -{tiny}"), "", "")), ["A real number is outside ±3.403 × 10³⁸, or nonzero but within ±1.175 × 10⁻³⁸"]); + let cid = ["A CMap maps a character code to a CID above 65535"]; + assert_eq!(limits(&doc("", &range("3f00", "3fff", 65536), "")), cid); + assert_eq!(limits(&doc("", &range("0000", "0100", 65280), "")), cid, "the range reaches 65536"); + assert_eq!(limits(&doc("", "1 begincidchar <01> 70000 endcidchar", "")), cid); + assert_eq!(limits(&doc("", &range("0000", "00ff", 0), &range("3f00", "3fff", 65536))), cid, "through /UseCMap"); +} + +/// PDF/A-2 6.2.4.4: Separation spaces of one colorant name agree on their alternate space and tint +/// transform, also inside a DeviceN space's `/Colorants`; and a DeviceN space names each spot +/// colorant it uses in its `/Colorants` (ISO 32000-1 8.6.6.5, Table 71). Process colorants (the +/// CMYK four, and an NChannel space's `/Process` components) and `None` need no entry. +#[test] +fn separation_and_devicen_spaces_are_checked() { + let spots = |spaces: &str| -> Vec { + let page = format!("<< /Type /Page /Parent 2 0 R /Resources << /ColorSpace << {spaces} >> >> >>"); + let doc = build( + &[ + "<< /Type /Catalog /Pages 2 0 R >>", + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + &page, + "<< /FunctionType 2 /Domain [0 1] /C0 [1 1 1] /C1 [1 0 0] /N 1 >>", + "<< /FunctionType 2 /Domain [0 1] /C0 [1 1 1] /C1 [0 1 0] /N 1 >>", + "<< /FunctionType 2 /Domain [0 1] /C0 [1 1 1] /C1 [1 0 0] /N 1 >>", + ], + "", + ); + verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.2.4.4").map(|i| i.message).collect() + }; + let red = |alt: &str, f: u32| format!("[/Separation /Red /{alt} {f} 0 R]"); + // The same name, the same alternate and an identical tint transform (another object): fine. + assert!(spots(&format!("/A {} /B {}", red("DeviceRGB", 4), red("DeviceRGB", 6))).is_empty()); + assert_eq!( + spots(&format!("/A {} /B {}", red("DeviceRGB", 4), red("DeviceRGB", 5))), + ["Separation colour spaces named Red differ in their alternate space or tint transform"] + ); + assert_eq!( + spots(&format!("/A {} /B {}", red("DeviceRGB", 4), red("DeviceGray", 4))), + ["Separation colour spaces named Red differ in their alternate space or tint transform"] + ); + // Inside two DeviceN spaces' Colorants. + let nchannel = |f: u32| format!("[/DeviceN [/Red] /DeviceRGB 4 0 R << /Subtype /NChannel /Colorants << /Red {} >> >>]", red("DeviceRGB", f)); + assert_eq!( + spots(&format!("/A {} /B {}", nchannel(4), nchannel(5))), + ["Separation colour spaces named Red differ in their alternate space or tint transform"] + ); + // Spot colorants need a Colorants entry; process colorants and None don't. + assert_eq!( + spots("/A [/DeviceN [/Spot1 /Cyan /None] /DeviceCMYK 4 0 R]"), + ["The spot colour Spot1 of a DeviceN colour space isn't in its Colorants dictionary"] + ); + assert_eq!( + spots("/A [/DeviceN [/Spot1] /DeviceRGB 4 0 R << /Colorants << >> >>]"), + ["The spot colour Spot1 of a DeviceN colour space isn't in its Colorants dictionary"] + ); + assert!(spots(&format!("/A [/DeviceN [/Red /Black] /DeviceRGB 4 0 R << /Colorants << /Red {} >> >>]", red("DeviceRGB", 4))).is_empty()); + assert!( + spots("/A [/DeviceN [/PrCyan /Black] /DeviceCMYK 4 0 R << /Subtype /NChannel /Process << /ColorSpace /DeviceCMYK /Components [/PrCyan /M /Y /Black] >> >>]") + .is_empty() + ); +} + +/// From the review of the 6.2.4.4 checks: nested parts compare by content, `All` and `None` get +/// no exemption, names may be given by reference, `/Process` exempts only in an NChannel space, a +/// null `/Colorants` entry defines nothing, and a colour space nested deep is still found. +#[test] +fn separation_and_devicen_checks_follow_references_depth_and_nchannel() { + let spots = |spaces: &str| -> Vec { + let page = format!("<< /Type /Page /Parent 2 0 R /Resources << /ColorSpace << {spaces} >> >> >>"); + let icc = "<< /N 3 /Length 4 >>\nstream\nabcd\nendstream"; + let doc = build( + &[ + "<< /Type /Catalog /Pages 2 0 R >>", + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + &page, + "<< /FunctionType 2 /Domain [0 1] /C0 [1 1 1] /C1 [1 0 0] /N 1 >>", + "<< /FunctionType 2 /Domain [0 1] /C0 [1 1 1] /C1 [0 1 0] /N 1 >>", + icc, + icc, + "/Red", + "null", + ], + "", + ); + verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.2.4.4").map(|i| i.message).collect() + }; + let differ = |n: &str| vec![format!("Separation colour spaces named {n} differ in their alternate space or tint transform")]; + let undefined = |n: &str| vec![format!("The spot colour {n} of a DeviceN colour space isn't in its Colorants dictionary")]; + // Identical ICC streams under different numbers are the same alternate space. + assert!(spots("/A [/Separation /Red [/ICCBased 6 0 R] 4 0 R] /B [/Separation /Red [/ICCBased 7 0 R] 4 0 R]").is_empty()); + assert_eq!(spots("/A [/Separation /All /DeviceRGB 4 0 R] /B [/Separation /All /DeviceRGB 5 0 R]"), differ("All")); + assert_eq!(spots("/A [/Separation 8 0 R /DeviceRGB 4 0 R] /B [/Separation /Red /DeviceRGB 5 0 R]"), differ("Red")); + assert_eq!(spots("/A [/DeviceN [/Spot] /DeviceRGB 4 0 R << /Process << /Components [/Spot] >> >>]"), undefined("Spot")); + assert_eq!(spots("/A [/DeviceN [/Spot] /DeviceRGB 4 0 R << /Colorants << /Spot null >> >>]"), undefined("Spot")); + assert_eq!(spots("/A [/DeviceN [/Spot] /DeviceRGB 4 0 R << /Colorants << /Spot 9 0 R >> >>]"), undefined("Spot")); + let deep = format!("/A {}[/DeviceN [/Spot] /DeviceRGB 4 0 R]{}", "[".repeat(40), "]".repeat(40)); + assert_eq!(spots(&deep), undefined("Spot")); +} + +/// From the second review of the 6.2.4.4 checks: a `/Colorants` entry is a Separation space of +/// its own colorant, tint-transform streams compare by decoded data, and a structure shared by +/// many definitions is compared once rather than once per path through it. +#[test] +fn separation_checks_match_colorant_entries_decode_streams_and_share_comparisons() { + // Two copies of a chain of arrays, each level naming the next twice: comparing them path by + // path takes about 2^19 steps. + const LEVELS: usize = 18; + let chain = |first: usize| -> Vec { + (0..LEVELS).map(|k| if k + 1 < LEVELS { format!("[{n} 0 R {n} 0 R]", n = first + k + 1) } else { "[1 2 3]".into() }).collect() + }; + let (a, b) = (5, 5 + LEVELS); + let spots = |spaces: &str| -> Vec { + let page = format!("<< /Type /Page /Parent 2 0 R /Resources << /ColorSpace << {spaces} >> >> >>"); + let mut objects = vec![ + "<< /Type /Catalog /Pages 2 0 R >>".to_string(), + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>".into(), + page, + "<< /FunctionType 2 /Domain [0 1] /C0 [1 1 1] /C1 [1 0 0] /N 1 >>".into(), + ]; + objects.extend(chain(a)); + objects.extend(chain(b)); + objects.push("<< /FunctionType 4 /Domain [0 1] /Range [0 1 0 1 0 1] /Length 15 >>\nstream\n{ dup dup pop }\nendstream".into()); + objects.push("<< /FunctionType 4 /Domain [0 1] /Range [0 1 0 1 0 1] /Filter /ASCIIHexDecode /Length 31 >>\nstream\n7B206475702064757020706F70207D>\nendstream".into()); + objects.push("<< /FunctionType 4 /Domain [0 1] /Range [0 1 0 1 0 1] /Length 15 >>\nstream\n{ dup pop dup }\nendstream".into()); + let objects: Vec<&str> = objects.iter().map(String::as_str).collect(); + let doc = build(&objects, ""); + verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.2.4.4").map(|i| i.message).collect() + }; + let differ = vec!["Separation colour spaces named Red differ in their alternate space or tint transform".to_string()]; + let undefined = vec!["The spot colour Spot of a DeviceN colour space isn't in its Colorants dictionary".to_string()]; + // A Colorants entry that isn't a Separation space of the same colorant defines nothing. + assert_eq!(spots("/A [/DeviceN [/Spot] /DeviceRGB 4 0 R << /Colorants << /Spot /DeviceRGB >> >>]"), undefined); + assert_eq!(spots("/A [/DeviceN [/Spot] /DeviceRGB 4 0 R << /Colorants << /Spot [/Separation /Other /DeviceRGB 4 0 R] >> >>]"), undefined); + assert!(spots("/A [/DeviceN [/Spot] /DeviceRGB 4 0 R << /Colorants << /Spot [/Separation /Spot /DeviceRGB 4 0 R] >> >>]").is_empty()); + // The same PostScript function, once plain and once hex-encoded, is the same tint transform. + let (plain, hex, other) = (5 + 2 * LEVELS, 6 + 2 * LEVELS, 7 + 2 * LEVELS); + assert!(spots(&format!("/A [/Separation /Red /DeviceRGB {plain} 0 R] /B [/Separation /Red /DeviceRGB {hex} 0 R]")).is_empty()); + assert_eq!(spots(&format!("/A [/Separation /Red /DeviceRGB {plain} 0 R] /B [/Separation /Red /DeviceRGB {other} 0 R]")), differ); + // Equal copies compared again and again still leave room to find the one that differs. + let sep = |f: usize| format!("[/Separation /Red /DeviceRGB {f} 0 R]"); + assert_eq!(spots(&format!("/A {} /B {} /C {} /D {} /E {}", sep(a), sep(b), sep(b), sep(b), sep(4))), differ); +} + const XMP_NS: &str = concat!( r#"xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" "#, r#"xmlns:xmp="http://ns.adobe.com/xap/1.0/" xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/" "#, r#"xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#" xmlns:xmpDM="http://ns.adobe.com/xmp/1.0/DynamicMedia/" "#, r#"xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/" xmlns:xmpTPg="http://ns.adobe.com/xap/1.0/t/pg/" "#, - r#"xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#" xmlns:pdf="http://ns.adobe.com/pdf/1.3/""# + r#"xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#" xmlns:pdf="http://ns.adobe.com/pdf/1.3/" "#, + r#"xmlns:tiff="http://ns.adobe.com/tiff/1.0/" xmlns:exif="http://ns.adobe.com/exif/1.0/" xmlns:aux="http://ns.adobe.com/exif/1.0/aux/""# ); /// The mistyped predefined properties of a packet whose one description holds `body` (its @@ -720,6 +991,400 @@ fn xmp_value_types_are_reported_by_verify() { assert_eq!(found.iter().map(|i| i.object).collect::>(), [Some(ObjRef::new(5, 0))]); } +/// PDF/A-2 6.6.2.3.1 for the TIFF, EXIF and Additional EXIF schemas of the September 2005 XMP +/// Specification (pages 58–70), with the EXIF value types of pages 83–87. +#[test] +fn xmp_exif_schemas_have_their_value_types() { + // TIFF: integers, closed choices of integer, rationals, sequences, language alternatives. + assert!(element("tiff:ImageWidth", "256").is_empty()); + assert_eq!(element("tiff:ImageWidth", "256 mm"), ["tiff:ImageWidth"]); + assert_eq!(element("tiff:Orientation", "4/1"), ["tiff:Orientation"]); + assert!(element("tiff:XResolution", "72/1").is_empty()); + assert_eq!(element("tiff:XResolution", "1/2"), ["tiff:XResolution"]); + assert!(element("tiff:BitsPerSample", "88").is_empty()); + assert_eq!(element("tiff:BitsPerSample", "88.0"), ["tiff:BitsPerSample"]); + assert_eq!(element("tiff:TransferFunction", "0"), ["tiff:TransferFunction"]); + assert_eq!(element("tiff:ImageDescription", "A picture"), ["tiff:ImageDescription"]); + // EXIF: GPS coordinates (page 85). + for good in ["51,30,26N", "0,5.25W", "120,5,0E", "8,0.000001S"] { + assert!(element("exif:GPSLatitude", good).is_empty(), "{good}"); + } + for bad in ["51.5N", "51,30,26", "51,30.N", "51,30,26X", "N", "51,30,2.6N", "a,30,26N", "51,30N", "51.5,30,26N"] { + assert_eq!(element("exif:GPSLatitude", bad), ["exif:GPSLatitude"], "{bad}"); + } + assert!(element("exif:GPSTimeStamp", "2016-02-01T13:19:21").is_empty()); + assert_eq!(element("exif:ISOSpeedRatings", "100"), ["exif:ISOSpeedRatings"]); + // EXIF structures: Flash, OECF/SFR, CFAPattern, DeviceSettings (pages 85–87). + assert!(mistyped(r#">True3"#).is_empty()); + assert_eq!(mistyped(r#">yes"#), ["exif:Flash"]); + assert_eq!(mistyped(r#">"#), ["exif:Flash"]); + assert_eq!(element("exif:Flash", "fired"), ["exif:Flash"]); + assert!( + mistyped(r#">1/2"#).is_empty() + ); + assert_eq!( + mistyped(r#">0.5"#), + ["exif:OECF"] + ); + assert_eq!( + mistyped( + r#">1/2"# + ), + ["exif:CFAPattern"] + ); + assert_eq!( + mistyped(r#">2.0"#), + ["exif:DeviceSettingDescription"] + ); + assert_eq!( + mistyped(r#">a"#), + ["exif:DeviceSettingDescription"] + ); + // Additional EXIF properties (page 69). + assert!(element("aux:Lens", "50mm").is_empty()); + assert_eq!(element("aux:Lens", "50mm"), ["aux:Lens"]); + assert_eq!(element("aux:SerialNumber", "1"), ["aux:SerialNumber"]); +} + +/// The 6.6.2.3.3 findings of a packet whose extension schema bag holds `schema`, with the five +/// description namespaces bound to the prefixes in `prefixes` (`pdfaSchema` and so on by default). +fn extension(schema: &str, prefixes: [&str; 4]) -> (Vec, Vec<&'static str>) { + let found = extension_findings(schema, prefixes); + (found.missing.into_iter().map(|m| m.entry).collect(), found.prefixes.into_iter().map(|p| p.own).collect()) +} + +/// [`extension`]'s findings in full, as if from the metadata stream `3 0 R`. +fn extension_findings(schema: &str, prefixes: [&str; 4]) -> xmp_extension::Findings { + let [s, p, t, f] = prefixes; + let xmp = format!( + concat!( + r#""#, + r#""#, + r#"{schema}"#, + r#""# + ), + s = s, + p = p, + t = t, + f = f, + schema = schema + ); + let parsed = xmp::parse(&xmp).expect("parses"); + let mut found = xmp_extension::Findings::default(); + found.add(ObjRef::new(3, 0), &parsed); + found +} + +const OWN: [&str; 4] = ["pdfaSchema", "pdfaProperty", "pdfaType", "pdfaField"]; + +/// A complete schema description, with `drop` left out of it (`pdfaSchema:schema` and so on). +fn schema_without(drop: &str) -> String { + let entry = |name: &str, value: &str| if name == drop { String::new() } else { format!("<{name}>{value}") }; + let property = ["pdfaProperty:name", "pdfaProperty:valueType", "pdfaProperty:category", "pdfaProperty:description"] + .iter() + .map(|n| entry(n, "x")) + .collect::(); + let field = ["pdfaField:name", "pdfaField:valueType", "pdfaField:description"].iter().map(|n| entry(n, "x")).collect::(); + let field = entry("pdfaType:field", &format!(r#"{field}"#)); + let value_type = + ["pdfaType:type", "pdfaType:namespaceURI", "pdfaType:prefix", "pdfaType:description"].iter().map(|n| entry(n, "x")).collect::(); + [ + entry("pdfaSchema:schema", "Some schema"), + entry("pdfaSchema:namespaceURI", "http://example.com/ns/"), + entry("pdfaSchema:prefix", "ex"), + entry("pdfaSchema:property", &format!(r#"{property}"#)), + entry("pdfaSchema:valueType", &format!(r#"{value_type}{field}"#)), + ] + .concat() +} + +/// PDF/A-2 6.6.2.3.3: every extension schema description has its required entries, in each of its +/// four kinds, and writes each description namespace with its own prefix. +#[test] +fn xmp_extension_schema_descriptions_are_complete() { + assert_eq!(extension(&schema_without(""), OWN), (vec![], vec![])); + // The property and value type lists of a schema are optional. + assert_eq!(extension(&schema_without("pdfaSchema:property"), OWN), (vec![], vec![])); + assert_eq!(extension(&schema_without("pdfaSchema:valueType"), OWN), (vec![], vec![])); + // Every other entry is required. + for required in [ + "pdfaSchema:schema", + "pdfaSchema:namespaceURI", + "pdfaSchema:prefix", + "pdfaProperty:name", + "pdfaProperty:valueType", + "pdfaProperty:category", + "pdfaProperty:description", + "pdfaType:type", + "pdfaType:namespaceURI", + "pdfaType:prefix", + "pdfaType:description", + "pdfaType:field", + "pdfaField:name", + "pdfaField:valueType", + "pdfaField:description", + ] { + assert_eq!(extension(&schema_without(required), OWN), (vec![required.to_string()], vec![]), "{required}"); + } + // An entry may be an attribute of an rdf:Description item. + let as_attribute = r#""#; + assert_eq!(extension(&format!("{}{as_attribute}", schema_without("")), OWN), (vec![], vec![])); + let missing_attribute = r#""#; + assert_eq!(extension(&format!("{}{missing_attribute}", schema_without("")), OWN).0, ["pdfaSchema:namespaceURI"]); + // Each namespace with its own prefix: the right URI under another prefix breaks the rule. + for (i, own) in OWN.iter().enumerate() { + let mut prefixes = OWN; + prefixes[i] = "other"; + let schema = schema_without("").replace(&format!("{own}:"), "other:"); + assert_eq!(extension(&schema, prefixes), (vec![], vec![*own]), "{own}"); + } + // A second property list is read as well as the first. + let complete = schema_without(""); + let second = complete.replacen( + "", + r#"p"#, + 1, + ); + assert_eq!(extension(&second, OWN).0, ["pdfaProperty:valueType", "pdfaProperty:category", "pdfaProperty:description"]); + // The container is an rdf:Bag (TechNote 0009, 4.2). + let container = |inner: &str| { + let xmp = format!( + r#"{inner}"# + ); + let mut found = xmp_extension::Findings::default(); + found.add(ObjRef::new(3, 0), &xmp::parse(&xmp).expect("parses")); + found.not_bag.into_iter().map(|c| c.found).collect::>() + }; + let schema = r#"Shttp://e/e"#; + assert!(container(&format!("{schema}")).is_empty()); + // What it holds instead is named as the packet writes it. + assert_eq!(container(&format!("{schema}")), ["rdf:Seq"]); + assert_eq!(container(r#""#), ["rdf:Description"]); + assert_eq!(container(&format!("{schema}")), ["rdf:Bag, rdf:Bag"]); + assert_eq!(container(""), [""]); + // The container itself is written as pdfaExtension too (TechNote 0009, 4.1). + let xmp = concat!( + r#""#, + r#""#, + r#"Shttp://e/"#, + r#"e"# + ); + let mut found = xmp_extension::Findings::default(); + found.add(ObjRef::new(3, 0), &xmp::parse(xmp).expect("parses")); + assert!(found.missing.is_empty()); + let written: Vec<(&str, &str, &str)> = found.prefixes.iter().map(|p| (p.namespace, p.own, p.written.as_str())).collect(); + assert_eq!(written, [("http://www.aiim.org/pdfa/ns/extension/", "pdfaExtension", "ext")]); + // Through verify, under 6.6.2.3.3 and not fixable. + let xmp = concat!( + r#""#, + r#""#, + r#"Se"#, + r#""# + ); + let stream = format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{xmp}\nendstream", xmp.len()); + let doc = build(&["<< /Type /Catalog /Pages 2 0 R /Metadata 3 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>", &stream], ""); + let issues: Vec = verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.6.2.3.3").collect(); + assert_eq!(issues.len(), 1); + assert_eq!(issues[0].message, "The XMP extension schema #1 in metadata stream 3 0 R has no pdfaSchema:namespaceURI entry"); + assert_eq!(issues[0].object, Some(ObjRef::new(3, 0))); + assert!(!issues[0].fixable); +} + +/// PDF/A-2 6.6.2.3.3: an entry that names something is incomplete when it is empty, while the +/// human-readable descriptions may be empty (TechNote 0009, 4.3 to 4.6). +#[test] +fn xmp_extension_naming_entries_are_not_empty() { + let complete = schema_without(""); + let emptied = |name: &str, value: &str, with: &str| { + let element = format!("<{name}>{value}"); + assert_eq!(complete.matches(&element).count(), 1, "{name}"); + complete.replace(&element, &format!("<{name}>{with}")) + }; + let said = |schema: &str| -> Vec<(String, bool)> { extension_findings(schema, OWN).missing.into_iter().map(|m| (m.entry, m.empty)).collect() }; + for (name, value) in [ + ("pdfaSchema:namespaceURI", "http://example.com/ns/"), + ("pdfaSchema:prefix", "ex"), + ("pdfaProperty:name", "x"), + ("pdfaProperty:valueType", "x"), + ("pdfaProperty:category", "x"), + ("pdfaType:type", "x"), + ("pdfaType:namespaceURI", "x"), + ("pdfaType:prefix", "x"), + ("pdfaField:name", "x"), + ("pdfaField:valueType", "x"), + ] { + assert_eq!(said(&emptied(name, value, "")), [(name.to_owned(), true)], "{name}"); + assert_eq!(said(&emptied(name, value, " \n\t ")), [(name.to_owned(), true)], "{name} blank"); + } + for (name, value) in + [("pdfaSchema:schema", "Some schema"), ("pdfaProperty:description", "x"), ("pdfaType:description", "x"), ("pdfaField:description", "x")] + { + assert_eq!(said(&emptied(name, value, "")), [], "{name}"); + } + // An empty element written shut, and an empty attribute, are empty too. + let shut = complete.replace("x", ""); + assert_eq!(said(&shut), [("pdfaType:prefix".to_owned(), true)]); + let attribute = r#""#; + assert_eq!(said(&format!("{complete}{attribute}")), [("pdfaSchema:prefix".to_owned(), true)]); + // A URI given as an rdf:resource, or a value given as rdf:value, is a value. + let resource = + complete.replace("x", r#""#); + assert_eq!(said(&resource), []); + let value = complete.replace("x", r#""#); + assert_eq!(said(&value), []); + // A comment is not a value. + assert_eq!(said(&emptied("pdfaType:prefix", "x", "")), [("pdfaType:prefix".to_owned(), true)]); + + // Through verify, with their own wordings. + let xmp = concat!( + r#""#, + r#""#, + r#""#, + r#"Shttp://a.example/"#, + r#""#, + r#"Thttp://t.example/"#, + r#" d"#, + r#""#, + r#""# + ); + let stream = format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{xmp}\nendstream", xmp.len()); + let doc = build(&["<< /Type /Catalog /Pages 2 0 R /Metadata 3 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>", &stream], ""); + let messages: Vec = verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.6.2.3.3").map(|i| i.message).collect(); + assert_eq!( + messages, + [ + "The XMP extension schema http://a.example/ in metadata stream 3 0 R has an empty pdfaSchema:prefix entry", + "The XMP extension schema http://a.example/ in metadata stream 3 0 R describes T with an empty pdfaType:prefix entry", + ] + ); +} + +/// PDF/A-2 6.6.2.3.3: each incomplete description is reported by itself, named by its schema's +/// namespace URI, its own name and its metadata stream, and so is each misused prefix and container. +#[test] +fn xmp_extension_findings_say_where_they_are() { + // Two schemas lacking the same entries are two findings, each with its schema. + let schema = |uri: &str, property: &str| { + format!( + concat!( + r#"S{uri}"#, + r#"e{property}"#, + r#""# + ), + uri = uri, + property = property + ) + }; + let named = "pText"; + let two = format!(r#"{}{}"#, schema("http://a.example/", named), schema("http://b.example/", "")); + let found = extension_findings(&two, OWN); + let said: Vec<(&str, Option<&str>, &str)> = found.missing.iter().map(|m| (m.schema.as_str(), m.item.as_deref(), m.entry.as_str())).collect(); + assert_eq!( + said, + [ + ("http://a.example/", Some("p"), "pdfaProperty:category"), + ("http://a.example/", Some("p"), "pdfaProperty:description"), + ("http://b.example/", Some("#1"), "pdfaProperty:name"), + ("http://b.example/", Some("#1"), "pdfaProperty:valueType"), + ("http://b.example/", Some("#1"), "pdfaProperty:category"), + ("http://b.example/", Some("#1"), "pdfaProperty:description"), + ] + ); + assert!(found.missing.iter().all(|m| m.object == ObjRef::new(3, 0))); + // A field is named by its value type and its own name. + let field = schema_without("pdfaField:description") + .replace(">x", ">Dim") + .replace(">x", ">w"); + let found = extension_findings(&field, OWN); + assert_eq!(found.missing.iter().map(|m| m.item.as_deref()).collect::>(), [Some("Dim/w")]); + // A name the packet gives is repeated only so far. + let long = format!("http://example.com/{}", "a".repeat(400)); + let found = extension_findings(&schema(&long, ""), OWN); + assert!(found.missing[0].schema.ends_with('…') && found.missing[0].schema.chars().count() == 201, "{}", found.missing[0].schema); + // So many findings are enough. + let many = vec![schema("http://m.example/", ""); 400].join(r#""#); + assert_eq!(extension_findings(&many, OWN).missing.len(), 1000); + + // Through verify: each wording, with the stream it is in. + let verify_packet = |description: &str| -> Vec { + let xmp = format!( + concat!( + r#""#, + r#""# + ), + description + ); + let stream = format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{xmp}\nendstream", xmp.len()); + let doc = build(&["<< /Type /Catalog /Pages 2 0 R /Metadata 3 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>", &stream], ""); + verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.6.2.3.3").map(|i| i.message).collect() + }; + let ns = r#"xmlns:pdfaExtension="http://www.aiim.org/pdfa/ns/extension/" xmlns:pdfaSchema="http://www.aiim.org/pdfa/ns/schema#" xmlns:pdfaProperty="http://www.aiim.org/pdfa/ns/property#""#; + assert_eq!( + verify_packet(&format!( + r#"{ns}>{}"#, + schema("http://a.example/", named) + )), + [ + "The XMP extension schema http://a.example/ in metadata stream 3 0 R describes p without a pdfaProperty:category entry", + "The XMP extension schema http://a.example/ in metadata stream 3 0 R describes p without a pdfaProperty:description entry", + ] + ); + assert_eq!( + verify_packet(&format!( + r#"{ns}>{}"#, + schema( + "http://a.example/", + &format!("{named}externald") + ) + )), + ["The XMP extension schema container in metadata stream 3 0 R holds rdf:Seq, not one rdf:Bag"] + ); + assert_eq!( + verify_packet(&format!(r#"{ns}>"#)), + ["The XMP extension schema container in metadata stream 3 0 R holds no rdf:Bag"] + ); + let other = r#"xmlns:pdfaExtension="http://www.aiim.org/pdfa/ns/extension/" xmlns:s="http://www.aiim.org/pdfa/ns/schema#""#; + assert_eq!( + verify_packet(&format!( + r#"{other}>Shttp://a.example/e"# + )), + ["Metadata stream 3 0 R writes the namespace http://www.aiim.org/pdfa/ns/schema# with the prefix s, not pdfaSchema"] + ); + let default = r#"xmlns:pdfaExtension="http://www.aiim.org/pdfa/ns/extension/""#; + assert_eq!( + verify_packet(&format!( + r#"{default}>Shttp://a.example/e"# + )), + ["Metadata stream 3 0 R writes the namespace http://www.aiim.org/pdfa/ns/schema# without a prefix, not with pdfaSchema"] + ); +} + +/// PDF/A-2 6.6.2.3.1 for the TIFF, EXIF and Additional EXIF schemas: a property is known by its +/// namespace URI, whatever prefix the packet binds to it, and is named by the namespace's +/// preferred prefix. +#[test] +fn xmp_exif_properties_are_known_by_namespace_uri() { + let tiff = r#"xmlns:foo="http://ns.adobe.com/tiff/1.0/""#; + assert_eq!(mistyped(&format!(r#">256 mm"#)), ["tiff:ImageWidth"]); + assert!(mistyped(&format!(r#">256"#)).is_empty()); + assert_eq!(mistyped(r#"xmlns:bar="http://ns.adobe.com/exif/1.0/" bar:GPSLatitude="51.5N">"#), ["exif:GPSLatitude"]); + assert_eq!( + mistyped(r#">50mm"#), + ["aux:Lens"] + ); + // The same local names in another namespace, even under the preferred prefixes, are not these + // properties. + let other = r#"xmlns:tiff="http://example.com/not-tiff/""#; + assert!(mistyped(&format!(r#">256 mm"#)).is_empty()); + assert!(mistyped(r#"xmlns:exif="http://example.com/not-exif/" exif:GPSLatitude="51.5N">"#).is_empty()); + assert!(mistyped(r#">50mm"#).is_empty()); +} + /// Holes are filled in one pass: the first `{}` from `value`, named holes from `values`, and what /// has nothing for it, a value's own braces and a brace left open stay as they are. #[test] @@ -755,6 +1420,12 @@ fn xmp_mistyped_properties_name_their_type() { [t("xmpMM:History", "seq ResourceEvent"), t("xmpTPg:MaxPageSize", "Dimensions")] ); assert_eq!(types(">p"), [t("xmpMM:Pantry", "bag struct")]); + // The TIFF and EXIF types, as the 2005 edition writes them. + assert_eq!(types(r#"tiff:XResolution="0.5" exif:GPSLatitude="N">"#), [t("exif:GPSLatitude", "GPSCoordinate"), t("tiff:XResolution", "Rational")]); + assert_eq!( + types(">8firedo"), + [t("exif:Flash", "Flash"), t("exif:OECF", "OECF/SFR"), t("tiff:BitsPerSample", "seq Integer")] + ); } /// PDF/A-2 6.6.2.3.1: a property is known by its namespace URI, whatever prefix the packet binds @@ -831,3 +1502,661 @@ fn xmp_value_forms_are_those_xmp_allows() { ["xmpMM:DerivedFrom"] ); } + +/// PDF/A asks for the fonts "used for rendering" to be embedded (ISO 19005-2, 6.2.11.4.1). A font +/// that only the form's default resources hold (`/AcroForm /DR`, ISO 32000-1 12.7.2) draws +/// nothing, as in veraPDF's 6-4-1-t02-pass-a; one that a widget's appearance stream draws with is +/// named in that stream's `/Resources`, and still has to be embedded. +#[test] +fn only_fonts_something_draws_with_must_be_embedded() { + let fonts = + |doc: &Document| -> Vec { verify(doc, Level::A2b).into_iter().filter(|i| i.clause == "6.2.11.4").map(|i| i.message).collect() }; + let form = |appearance_fonts: &str| { + build( + &[ + "<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [6 0 R] /DA (/Helv 0 Tf 0 g) /DR << /Font << /Helv 4 0 R /ZaDb 5 0 R >> >> >> >>", + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + "<< /Type /Page /Parent 2 0 R /Resources << >> /Annots [6 0 R] >>", + "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>", + "<< /Type /Font /Subtype /Type1 /BaseFont /ZapfDingbats >>", + "<< /Type /Annot /Subtype /Widget /FT /Tx /T (name) /F 4 /Rect [10 10 90 30] /P 3 0 R /AP << /N 7 0 R >> >>", + &format!("<< /Type /XObject /Subtype /Form /BBox [0 0 80 20] /Resources << {appearance_fonts} >> /Length 0 >>\nstream\n\nendstream"), + // An object that is nothing but a reference to the Helvetica font (ISO 32000-1 7.3.10). + "4 0 R", + ], + "", + ) + }; + assert!(fonts(&form("")).is_empty(), "{:?}", fonts(&form(""))); + assert_eq!(fonts(&form("/Font << /Helv 4 0 R >>")), ["The font Helvetica isn't embedded"]); + // Named through a reference to it, it is still the font something draws with. + assert_eq!(fonts(&form("/Font << /Helv 8 0 R >>")), ["The font Helvetica isn't embedded"]); + // A page's own font is checked as before. + assert_eq!(fonts(&messy()), ["The font Helvetica isn't embedded"]); +} + +/// PDF/A-2 6.1.2: the file starts with `%PDF-1.0` to `%PDF-1.7` and an end-of-line marker, then a +/// comment of `%` and four bytes above 127; Save as PDF/A rewrites a header that doesn't. +#[test] +fn file_header_is_checked_and_rewritten() { + let objs = ["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>"]; + let header = |h: &[u8]| -> Vec { + let doc = build_with_header(h, &objs, ""); + verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.1.2").map(|i| i.message).collect() + }; + let version = || vec!["The file doesn't start with a %PDF-1.0 to %PDF-1.7 header line".to_string()]; + let comment = || vec!["The file header isn't followed by a comment of four bytes above 127".to_string()]; + for fine in [&b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n"[..], b"%PDF-1.4\r\n%\x80\x81\x82\x83af\r\n", b"%PDF-1.0\r%\xFF\xFF\xFF\xFF\r"] { + assert!(header(fine).is_empty(), "{}", String::from_utf8_lossy(fine)); + } + let wrong_version = [ + &b"%PDF-2.0\n%\xE2\xE3\xCF\xD3\n"[..], + b"%PDF-1.9\n%\xE2\xE3\xCF\xD3\n", + b"%PDF-1.70\n%\xE2\xE3\xCF\xD3\n", + b"%PDF-1.7 \n%\xE2\xE3\xCF\xD3\n", + b"SomeData\n%PDF-1.7\n%\xE2\xE3\xCF\xD3\n", + ]; + for wrong in wrong_version { + assert_eq!(header(wrong), version(), "{}", String::from_utf8_lossy(wrong)); + } + let wrong_comment = [ + &b"%PDF-1.7\n%\xE2\xE3\xCF\n"[..], + b"%PDF-1.7\ninvalidData\n", + b"%PDF-1.7\n%a\xE3\xCF\xD3\n", + b"%PDF-1.7\n\n%\xE2\xE3\xCF\xD3\n", + b"%PDF-1.7\n", + ]; + for wrong in wrong_comment { + assert_eq!(header(wrong), comment(), "{}", String::from_utf8_lossy(wrong)); + } + // Save as PDF/A rewrites the header: the version drops to 1.7 and the next save is full. + let mut doc = build_with_header(b"%PDF-2.0\n%ab\n", &objs, ""); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(report.fixed.iter().any(|f| f == "Rewrote the file header")); + assert_eq!(doc.version(), "1.7"); + assert!(verify(&doc, Level::A2b).iter().all(|i| i.clause != "6.1.2")); + let saved = pdfcraft_cos::write_full(&doc, &Default::default()).unwrap(); + assert!(saved.starts_with(b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n")); + // A good version with a short comment is rewritten too, keeping its version. + let mut doc = build_with_header(b"%PDF-1.4\n%ab\n", &objs, ""); + assert!(!doc.full_save_required()); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(report.fixed.iter().any(|f| f == "Rewrote the file header")); + assert!(doc.full_save_required()); + assert_eq!(doc.version(), "1.4"); + // A good header is left as it is. + let mut doc = build(&objs, ""); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(!report.fixed.iter().any(|f| f == "Rewrote the file header")); +} + +/// PDF/A-2 6.2.5: a graphics state has no transfer function, halftones of type 1 or 5 without a +/// name, and a standard rendering intent, wherever it is kept. +#[test] +fn extended_graphics_states_are_checked() { + let gs = |states: &str| -> Vec { + let page = format!("<< /Type /Page /Parent 2 0 R /Resources << /ExtGState << {states} >> >> >>"); + let doc = build( + &[ + "<< /Type /Catalog /Pages 2 0 R >>", + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", + &page, + "<< /FunctionType 2 /Domain [0 1] /C0 [0] /C1 [1] /N 1 >>", + "<< /HalftoneType 1 /Frequency 60 /Angle 45 /SpotFunction /Round >>", + "<< /TR 4 0 R >>", + "<< /Type /Halftone /HalftoneType 6 /Width 2 /Height 2 /Length 4 >>\nstream\nabcd\nendstream", + "<< /Resources << /ExtGState << /X << /TR 4 0 R >> >> >> >>", + ], + "", + ); + let issues = verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.2.5"); + issues.inspect(|i| assert!(!i.fixable, "{}", i.message)).map(|i| i.message).collect() + }; + let tr = "A graphics state has a transfer function (TR)"; + let tr2 = "A graphics state has a transfer function (TR2) other than Default"; + let name = "A halftone has a HalftoneName"; + // What 6.2.5 allows. + assert!(gs("/A << /TR2 /Default /HT 5 0 R /RI /Perceptual >> /B << /HT /Default /RI /AbsoluteColorimetric >>").is_empty()); + assert!(gs("/A << /HT << /HalftoneType 5 /Default 5 0 R /Cyan 5 0 R >> /RI /Saturation >> /B << /TR null >>").is_empty()); + // Transfer functions. + assert_eq!(gs("/A << /TR 4 0 R >>"), [tr]); + assert_eq!(gs("/A << /TR /Identity >>"), [tr]); + assert_eq!(gs("/A 6 0 R"), [tr]); + assert_eq!(gs("/A << /TR2 4 0 R >>"), [tr2]); + assert_eq!(gs("/A << /TR2 /Identity >>"), [tr2]); + assert_eq!(gs("/A << /HTP [0 1] >>"), ["A graphics state has a halftone phase (HTP)"]); + // Halftones: the type, also of a type 5 halftone's colorants and of a stream, and the name. + assert_eq!(gs("/A << /HT << /HalftoneType 6 >> >>"), ["A halftone has type 6: only types 1 and 5 are allowed"]); + assert_eq!(gs("/A << /HT 7 0 R >>"), ["A halftone has type 6: only types 1 and 5 are allowed"]); + assert_eq!( + gs("/A << /HT << /HalftoneType 5 /Default 5 0 R /Magenta << /HalftoneType 10 >> >> >>"), + ["A halftone has type 10: only types 1 and 5 are allowed"] + ); + assert_eq!(gs("/A << /HT << /HalftoneType 1 /HalftoneName (Screen) >> >>"), [name]); + assert_eq!(gs("/A << /HT << /HalftoneType 5 /Default << /HalftoneType 1 /HalftoneName (S) >> >> >>"), [name]); + // Rendering intents. + assert_eq!(gs("/A << /RI /Custom >>"), ["A graphics state has the rendering intent Custom, which isn't a standard one"]); + assert_eq!(gs("/A << /RI (Perceptual) >>"), ["A graphics state has a rendering intent that isn't a name"]); + // A halftone's type is required; a null name is no name. + let untyped = ["A halftone's HalftoneType is missing or isn't an integer"]; + assert_eq!(gs("/A << /HT << /Frequency 60 >> >>"), untyped); + assert_eq!(gs("/A << /HT << /HalftoneType 1.0 >> >>"), untyped); + assert!(gs("/A << /HT << /HalftoneType 1 /HalftoneName null >> >>").is_empty()); + // Transfer functions in halftones: only, and always, for a non-primary colorant. + let in_halftone = "A halftone has a transfer function, which only a non-primary colorant's halftone may have"; + assert_eq!(gs("/A << /HT << /HalftoneType 1 /TransferFunction 4 0 R >> >>"), [in_halftone]); + assert_eq!(gs("/A << /HT << /HalftoneType 5 /Default 5 0 R /Cyan << /HalftoneType 1 /TransferFunction /Identity >> >> >>"), [in_halftone]); + // The primary components are those of DeviceGray, DeviceRGB and DeviceCMYK (ISO 32000-1 10.5.5.6). + assert_eq!(gs("/A << /HT << /HalftoneType 5 /Default 5 0 R /Gray << /HalftoneType 1 /TransferFunction /Identity >> >> >>"), [in_halftone]); + assert_eq!( + gs("/A << /HT << /HalftoneType 5 /Default 5 0 R /Spot 5 0 R /Red 5 0 R /Gray 5 0 R >> >>"), + ["The halftone for the colorant Spot has no transfer function"] + ); + assert!(gs("/A << /HT << /HalftoneType 5 /Default 5 0 R /Black 5 0 R /Spot << /HalftoneType 1 /TransferFunction 4 0 R >> >> >>").is_empty()); + // Private application data isn't graphics resources. + assert!( + gs("/A << /RI /Perceptual >> >> /PieceInfo << /App << /Private << /Resources << /ExtGState << /X << /TR 4 0 R >> >> >> >> >>").is_empty() + ); + // Nor is an object (8) only private data refers to; one the page refers to is read. + assert!(gs("/A << /RI /Perceptual >> >> /PieceInfo << /App << /Private 8 0 R >> >> /Z <<").is_empty()); + assert_eq!(gs("/A << /RI /Perceptual >> >> /Other 8 0 R /Z <<"), [tr]); + // A shading pattern holds one graphics state, not a dictionary of them. + let pattern = |gs: &str| format!("/A << /RI /Perceptual >> >> /Pattern << /P << /PatternType 2 /Shading << /ShadingType 2 >> /ExtGState {gs} >>"); + assert_eq!(gs(&pattern("<< /TR 4 0 R >>")), [tr]); + assert!(gs(&pattern("<< /HT << /HalftoneType 1 >> >>")).is_empty()); +} + +/// From the review of the 6.2.5 check: references that differ only in generation reach the same +/// object, which is read once. +#[test] +fn graphics_state_scan_reads_each_object_once_whatever_the_generation() { + let gens: String = (0..10).map(|g| format!("4 {g} R ")).collect(); + let page = format!("<< /Type /Page /Parent 2 0 R /Same [{gens}] >>"); + let doc = + build(&["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", &page, "[1 2 3]"], ""); + // The catalog, the page tree, the page and object 4. + assert_eq!(gstate::reached(&doc), 4); +} + +/// PDF/A-2 6.1.3: the trailer `startxref` points to has a file identifier, two non-empty +/// strings; Save as PDF/A gives a file one. +#[test] +fn trailer_file_identifier_is_checked_and_added() { + let objs = ["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>"]; + let id = |extra: &str| -> Vec { + let doc = build(&objs, extra); + verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.1.3").map(|i| i.message).collect() + }; + let missing = ["The trailer has no file identifier (ID)"]; + let malformed = ["The file identifier (ID) in the trailer isn't two non-empty strings"]; + assert!(id("/ID [<0011> (ab)]").is_empty()); + assert_eq!(id(""), missing); + assert_eq!(id("/ID null"), missing); + for wrong in ["/ID [<><>]", "/ID [<0011>]", "/ID [<0011> <0011> <0011>]", "/ID [1 2]", "/ID (ab)"] { + assert_eq!(id(wrong), malformed, "{wrong}"); + } + // An update without an /ID has none, whatever the trailer it replaces had. + let base = build(&objs, "/ID [<0011> <0011>]"); + let mut bytes = base.bytes().to_vec(); + let prev = bytes.windows(9).rposition(|w| w == b"startxref").unwrap(); + let prev: usize = String::from_utf8_lossy(&bytes[prev + 10..]).lines().next().unwrap().trim().parse().unwrap(); + let at = bytes.len(); + bytes.extend_from_slice( + format!("xref\n0 1\n0000000000 65535 f \ntrailer\n<< /Size 3 /Root 1 0 R /Prev {prev} >>\nstartxref\n{at}\n%%EOF\n").as_bytes(), + ); + let updated = Document::open(std::sync::Arc::new(bytes)).unwrap(); + assert!(verify(&updated, Level::A2b).iter().any(|i| i.message == missing[0])); + // Save as PDF/A adds one, which a save keeps. + for extra in ["", "/ID [<><>]"] { + let mut doc = build(&objs, extra); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(report.fixed.iter().any(|f| f == "Added a file identifier (ID)"), "{extra}"); + assert!(verify(&doc, Level::A2b).iter().all(|i| i.clause != "6.1.3"), "{extra}"); + let saved = pdfcraft_cos::write_incremental(&doc, &Default::default()).unwrap(); + let saved = Document::open(std::sync::Arc::new(saved)).unwrap(); + assert!(verify(&saved, Level::A2b).iter().all(|i| i.clause != "6.1.3"), "{extra}"); + } + // A good one is kept as it is. + let mut doc = build(&objs, "/ID [<0011> <0022>]"); + let before = doc.trailer().get(b"ID").cloned(); + assert!(!convert(&mut doc, Level::A2b).unwrap().fixed.iter().any(|f| f.contains("identifier"))); + assert_eq!(doc.trailer().get(b"ID").cloned(), before); + // A malformed one keeps its non-empty first string, the permanent identifier (14.4). + let first = |doc: &Document| doc.trailer().get(b"ID").and_then(|id| id.as_array()?.first()?.as_string().map(|s| s.bytes.clone())); + let mut doc = build(&objs, "/ID [<0011> <>]"); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(report.fixed.iter().any(|f| f == "Completed the file identifier (ID)")); + assert_eq!(first(&doc), Some(vec![0x00, 0x11])); + assert!(verify(&doc, Level::A2b).iter().all(|i| i.clause != "6.1.3")); + let mut doc = build(&objs, "/ID [<> <0011>]"); + convert(&mut doc, Level::A2b).unwrap(); + assert_eq!(first(&doc).map(|b| b.len()), Some(16)); + // A first string longer than a conforming file may hold (Annex C) is made afresh too. + let long = format!("/ID [<{}> <>]", "ab".repeat(32768)); + let mut doc = build(&objs, &long); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(report.fixed.iter().any(|f| f == "Added a file identifier (ID)")); + assert_eq!(first(&doc).map(|b| b.len()), Some(16)); + let longest = format!("/ID [<{}> <>]", "ab".repeat(32767)); + let mut doc = build(&objs, &longest); + convert(&mut doc, Level::A2b).unwrap(); + assert_eq!(first(&doc).map(|b| b.len()), Some(32767)); +} + +/// PDF/A-2 6.5.1: the PDF 1.2 SetState and NOP actions aren't allowed, and a Named action may +/// only name NextPage, PrevPage, FirstPage or LastPage; Save as PDF/A removes the others. +#[test] +fn set_state_no_op_and_named_actions_are_checked() { + let doc_with = |action: &str| { + let outline = format!("<< /Title (x) /Parent 4 0 R /A {action} >>"); + build( + &[ + "<< /Type /Catalog /Pages 2 0 R /Outlines 4 0 R >>", + "<< /Type /Pages /Kids [] /Count 0 >>", + &outline, + "<< /Type /Outlines /First 3 0 R /Last 3 0 R /Count 1 >>", + "/NextPage", + "/Named", + "/NOP", + ], + "", + ) + }; + let actions = |action: &str| -> Vec { + verify(&doc_with(action), Level::A2b).into_iter().filter(|i| i.clause == "6.5.1").map(|i| i.message).collect() + }; + assert_eq!(actions("<< /Type /Action /S /SetState /T [] >>"), ["A SetState action isn't allowed"]); + assert_eq!(actions("<< /S /NOP >>"), ["A NOP action isn't allowed"]); + assert_eq!(actions("<< /S /Named /N /Print >>"), ["A Named Print action isn't allowed"]); + assert_eq!(actions("<< /S /Named >>"), ["A Named action isn't allowed"]); + // Behind an allowed action, in its /Next. + assert_eq!(actions("<< /S /GoTo /D [0 /Fit] /Next << /S /NOP >> >>"), ["A NOP action isn't allowed"]); + for allowed in ["NextPage", "PrevPage", "FirstPage", "LastPage"] { + assert!(actions(&format!("<< /S /Named /N /{allowed} >>")).is_empty(), "{allowed}"); + } + // The name given by reference (object 5 is /NextPage). + assert!(actions("<< /S /Named /N 5 0 R >>").is_empty()); + // The type given by reference (object 6 is /Named, 7 is /NOP). + assert_eq!(actions("<< /S 6 0 R /N /Print >>"), ["A Named Print action isn't allowed"]); + assert_eq!(actions("<< /S 7 0 R >>"), ["A NOP action isn't allowed"]); + assert!(actions("<< /S 6 0 R /N 5 0 R >>").is_empty()); + // Save as PDF/A removes them. + for action in ["<< /S /SetState /T [] >>", "<< /S /NOP >>", "<< /S /Named /N /Print >>"] { + let mut doc = doc_with(action); + convert(&mut doc, Level::A2b).unwrap(); + assert!(verify(&doc, Level::A2b).iter().all(|i| i.clause != "6.5.1"), "{action}"); + } + // A permitted Named action survives it, also one naming its page move by reference. + for action in ["<< /S /Named /N /NextPage >>", "<< /S /Named /N 5 0 R >>"] { + let mut doc = doc_with(action); + convert(&mut doc, Level::A2b).unwrap(); + let outline = doc.get(ObjRef { num: 3, generation: 0 }); + let kept = outline + .as_dict() + .and_then(|d| d.get(b"A")) + .and_then(Object::as_dict) + .map(|a| (a.name(b"S").map(<[u8]>::to_vec), a.get(b"N").map(|n| doc.resolve(n)))); + let (s, n) = kept.unwrap_or_else(|| panic!("{action}: the action is gone")); + assert_eq!(s.as_deref(), Some(&b"Named"[..]), "{action}"); + assert_eq!(n.as_deref().and_then(Object::as_name), Some(&b"NextPage"[..]), "{action}"); + } +} + +/// The SetState, NOP and Named checks of 6.5.1 apply wherever an action can be: the catalog's +/// `/OpenAction`, an action standing alone as an object of its own (here one a page's `/AA` +/// refers to), and a link's `/A`, not only an outline item's. A Named action's `/N` given by +/// reference to a forbidden name is reported by that name. Save as PDF/A removes each one and +/// keeps a Named page move in the same places. +#[test] +fn set_state_no_op_and_named_actions_are_checked_wherever_actions_are() { + // Object 5 is the standalone action, 6 is /Print and 7 is /NextPage. + let doc_with = |open_action: &str, page_aa: &str, link_action: &str, standalone: &str| { + let catalog = format!("<< /Type /Catalog /Pages 2 0 R {open_action} >>"); + let page = format!("<< /Type /Page /Parent 2 0 R /Annots [4 0 R] {page_aa} >>"); + let link = format!("<< /Type /Annot /Subtype /Link /Rect [0 0 0 0] /P 3 0 R /F 4 {link_action} >>"); + build(&[&catalog, "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", &page, &link, standalone, "/Print", "/NextPage"], "") + }; + let goto = "<< /S /GoTo /D [3 0 R /Fit] >>"; + // Where the action is, as (open action, page /AA, link /A, object 5). + let placed = |action: &str| { + [ + ("an open action", doc_with(&format!("/OpenAction {action}"), "", "", goto)), + ("an open action by reference", doc_with("/OpenAction 5 0 R", "", "", action)), + ("a page's /AA", doc_with("", "/AA << /O 5 0 R >>", "", action)), + ("a link's /A", doc_with("", "", &format!("/A {action}"), goto)), + ] + }; + let forbidden = [ + ("<< /S /SetState /T [] >>", "A SetState action isn't allowed"), + ("<< /S /NOP >>", "A NOP action isn't allowed"), + ("<< /S /Named /N /Print >>", "A Named Print action isn't allowed"), + ("<< /S /Named /N 6 0 R >>", "A Named Print action isn't allowed"), + ]; + let plain = pdfcraft_cos::SaveOptions { object_streams: false, ..Default::default() }; + for (action, message) in forbidden { + for (place, mut doc) in placed(action) { + let issues = verify(&doc, Level::A2b); + let found: Vec<&str> = issues.iter().filter(|i| i.clause == "6.5.1").map(|i| i.message.as_str()).collect(); + assert_eq!(found, [message], "{action} in {place}: {issues:#?}"); + let r = convert(&mut doc, Level::A2b).unwrap(); + let left: Vec<_> = r.remaining.iter().filter(|i| i.clause.starts_with("6.5")).collect(); + assert!(left.is_empty(), "{action} in {place}: {left:#?}"); + let bytes = pdfcraft_cos::write_full(&doc, &plain).unwrap(); + for kind in [&b"/SetState"[..], b"/NOP", b"/Named"] { + assert!(!bytes.windows(kind.len()).any(|w| w == kind), "{action} in {place}: {}", String::from_utf8_lossy(&bytes)); + } + } + } + // A Named page move, with its name given by reference, is allowed and kept outside /AA. + for (place, mut doc) in placed("<< /S /Named /N 7 0 R >>") { + let issues = verify(&doc, Level::A2b); + assert!(!issues.iter().any(|i| i.clause == "6.5.1"), "{place}: {issues:#?}"); + convert(&mut doc, Level::A2b).unwrap(); + let bytes = pdfcraft_cos::write_full(&doc, &plain).unwrap(); + let kept = bytes.windows(6).any(|w| w == b"/Named"); + assert_eq!(kept, place != "a page's /AA", "{place}: {}", String::from_utf8_lossy(&bytes)); + } +} + +/// 6.2.10: without a PDF/A output intent, a page whose resources can paint transparently names its +/// blending colour space in its `/Group`. Transparency is found through graphics states, images, +/// transparency groups, nested form XObjects, Type 3 fonts, patterns and annotation appearances. +#[test] +fn a_page_with_transparency_and_no_output_intent_names_its_blending_space() { + // With `intent`, the catalog has a PDF/A output intent, its profile the last object. + let blending = |intent: bool, page_extra: &str, resources: &str, extra: &[&str]| -> Vec { + let profile = extra.len() + 4; + let intents = + format!("/OutputIntents [<< /Type /OutputIntent /S /GTS_PDFA1 /OutputConditionIdentifier (sRGB) /DestOutputProfile {profile} 0 R >>]"); + let catalog = format!("<< /Type /Catalog /Pages 2 0 R {} >>", if intent { intents.as_str() } else { "" }); + let page = format!("<< /Type /Page /Parent 2 0 R /Resources << {resources} >> {page_extra} >>"); + let mut objs: Vec<&str> = vec![&catalog, "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", &page]; + objs.extend_from_slice(extra); + if intent { + objs.push( + "<< /N 3 /Length 0 >> +stream + +endstream", + ); + } + verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| i.message).collect() + }; + let said = ["Page 1 has transparency but no device-independent blending colour space or PDF/A output intent"]; + let form = |dict: &str| format!("<< /Type /XObject /Subtype /Form /BBox [0 0 10 10] {dict} /Length 0 >>\nstream\n\nendstream"); + let gs = |entries: &str| format!("/ExtGState << /G << {entries} >> >>"); + for (what, resources, extra) in [ + ("fill alpha", gs("/ca 0.5"), vec![]), + ("stroke alpha", gs("/CA 0.25"), vec![]), + ("blend mode", gs("/BM /Darken"), vec![]), + ("blend mode array", gs("/BM [/Multiply /Normal]"), vec![]), + ("soft mask", gs("/SMask << /S /Luminosity /G 4 0 R >>"), vec![form("")]), + ( + "image soft mask", + "/XObject << /Im 4 0 R >>".to_owned(), + vec![ + "<< /Subtype /Image /Width 1 /Height 1 /SMask 5 0 R /Length 0 >>\nstream\n\nendstream".to_owned(), + "<< /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length 0 >>\nstream\n\nendstream".to_owned(), + ], + ), + ( + "SMaskInData", + "/XObject << /Im 4 0 R >>".to_owned(), + vec!["<< /Subtype /Image /Width 1 /Height 1 /SMaskInData 1 /Length 0 >>\nstream\n\nendstream".to_owned()], + ), + ("transparency group", "/XObject << /Fm 4 0 R >>".to_owned(), vec![form("/Group << /S /Transparency >>")]), + ( + "nested form", + "/XObject << /Fm 4 0 R >>".to_owned(), + vec![form("/Resources << /XObject << /In 5 0 R >> >>"), form("/Resources << /ExtGState << /G << /ca 0.5 >> >> >>")], + ), + ( + "Type 3 font", + "/Font << /T3 4 0 R >>".to_owned(), + vec!["<< /Type /Font /Subtype /Type3 /Resources << /ExtGState << /G << /CA 0.5 >> >> >> >>".to_owned()], + ), + ("tiling pattern", "/Pattern << /P 4 0 R >>".to_owned(), vec![form("/PatternType 1 /Resources << /ExtGState << /G << /BM /Screen >> >> >>")]), + ("shading pattern", "/Pattern << /P << /PatternType 2 /ExtGState << /ca 0.5 >> >> >>".to_owned(), vec![]), + ] { + let extra: Vec<&str> = extra.iter().map(String::as_str).collect(); + assert_eq!(blending(false, "", &resources, &extra), said, "{what}"); + // A page group with a blending colour space, or a PDF/A output intent, satisfies the rule. + assert!( + blending(false, "/Group << /S /Transparency /CS [/CalRGB << /WhitePoint [0.9505 1 1.089] >>] >>", &resources, &extra).is_empty(), + "{what}: page /CS" + ); + assert_eq!(blending(false, "/Group << /S /Transparency >>", &resources, &extra), said, "{what}: page group without /CS"); + assert!(blending(true, "", &resources, &extra).is_empty(), "{what}: output intent"); + } + // Annotation appearances, as one stream or a dictionary of streams by state. + let annot = |ap: &str| format!("<< /Type /Annot /Subtype /Widget /Rect [0 0 10 10] /AP << /N {ap} >> >>"); + assert_eq!(blending(false, "/Annots [4 0 R]", "", &[&annot("5 0 R"), &form("/Group << /S /Transparency >>")]), said, "appearance"); + assert_eq!( + blending(false, "/Annots [4 0 R]", "", &[&annot("<< /On 5 0 R /Off 6 0 R >>"), &form(""), &form("/Group << /S /Transparency >>")]), + said, + "appearance states" + ); + // Opaque painting needs no blending colour space: alpha 1, Normal or Compatible blending, no + // soft mask, an image without one, a form that is no transparency group. + for (what, resources, extra) in [ + ("opaque", gs("/CA 1 /ca 1.0 /BM /Normal /SMask /None"), vec![]), + ("compatible", gs("/BM [/Compatible /Darken]"), vec![]), + ( + "image", + "/XObject << /Im 4 0 R >>".to_owned(), + vec!["<< /Subtype /Image /Width 1 /Height 1 /SMaskInData 0 /Length 0 >>\nstream\n\nendstream".to_owned()], + ), + ( + "plain form, its resources cycling back to itself", + "/XObject << /Fm 4 0 R >>".to_owned(), + vec![form("/Resources << /XObject << /Me 4 0 R >> >>")], + ), + ( + "Type 1 font", + "/Font << /F 4 0 R >>".to_owned(), + vec!["<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Resources << /ExtGState << /G << /ca 0.5 >> >> >> >>".to_owned()], + ), + ] { + let extra: Vec<&str> = extra.iter().map(String::as_str).collect(); + assert!(blending(false, "", &resources, &extra).is_empty(), "{what}"); + } +} + +/// 6.2.10: a graphics state's blend mode, or each name of a blend mode array, is one ISO 32000-1 +/// 11.3.5 defines, whether the graphics state is an object of its own or written in a page's +/// resources. +#[test] +fn blend_modes_are_the_ones_iso_32000_1_defines() { + let modes = |resources: &str, extra: &[&str]| -> Vec { + let page = format!( + "<< /Type /Page /Parent 2 0 R /Group << /S /Transparency /CS [/CalRGB << /WhitePoint [0.9505 1 1.089] >>] >> /Resources << {resources} >> >>" + ); + let mut objs: Vec<&str> = + vec!["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", &page]; + objs.extend_from_slice(extra); + verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| i.message).collect() + }; + let custom = ["A graphics state uses the blend mode Custom, which ISO 32000-1 doesn't define"]; + assert_eq!(modes("/ExtGState << /G 4 0 R >>", &["<< /BM /Custom >>"]), custom, "an object of its own"); + assert_eq!(modes("/ExtGState << /G << /BM /Custom >> >>", &[]), custom, "written in the resources"); + assert_eq!(modes("/ExtGState << /G << /BM [/Multiply /Custom] >> >>", &[]), custom, "in an array"); + let all = "/Normal /Compatible /Multiply /Screen /Overlay /Darken /Lighten /ColorDodge /ColorBurn /HardLight /SoftLight /Difference /Exclusion /Hue /Saturation /Color /Luminosity"; + assert!(modes(&format!("/ExtGState << /G << /BM [{all}] >> >>"), &[]).is_empty()); +} + +/// 6.2.10's blending colour space is one PDF/A accepts without an output intent: not null, and +/// not a device space unless the page gives it a default (6.2.4.3). An output intent stands in +/// for it only with a profile. +#[test] +fn a_blending_space_is_device_independent_and_an_intent_has_a_profile() { + let page_said = ["Page 1 has transparency but no device-independent blending colour space or PDF/A output intent"]; + let check = |catalog_extra: &str, page_extra: &str, extra: &[&str]| -> Vec { + let catalog = format!("<< /Type /Catalog /Pages 2 0 R {catalog_extra} >>"); + let page = format!("<< /Type /Page /Parent 2 0 R {page_extra} >>"); + let mut objs: Vec<&str> = vec![&catalog, "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", &page]; + objs.extend_from_slice(extra); + verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| i.message).collect() + }; + let alpha = "/ExtGState << /G << /ca 0.5 >> >>"; + let page = |group: &str, colour_spaces: &str| format!("/Group << /S /Transparency {group} >> /Resources << {alpha} {colour_spaces} >>"); + assert_eq!(check("", &page("/CS null", ""), &[]), page_said, "a null /CS"); + assert_eq!(check("", &page("/CS 9 0 R", ""), &[]), page_said, "a /CS naming no object"); + // A device space needs a default of as many components (ISO 32000-1 8.6.5.6). + let cmyk_profile = "<< /N 4 /Length 0 >>\nstream\n\nendstream"; + let cal = |family: &str| format!("[/{family} << /WhitePoint [0.9505 1 1.089] >>]"); + for (device, default, space) in [ + ("DeviceGray", "DefaultGray", cal("CalGray")), + ("DeviceRGB", "DefaultRGB", cal("CalRGB")), + ("DeviceCMYK", "DefaultCMYK", "[/ICCBased 4 0 R]".to_owned()), + ] { + assert_eq!(check("", &page(&format!("/CS /{device}"), ""), &[cmyk_profile]), page_said, "{device}"); + let defaults = format!("/ColorSpace << /{default} {space} >>"); + assert!(check("", &page(&format!("/CS /{device}"), &defaults), &[cmyk_profile]).is_empty(), "{device} with {default}"); + let null = format!("/ColorSpace << /{default} null >>"); + assert_eq!(check("", &page(&format!("/CS /{device}"), &null), &[cmyk_profile]), page_said, "{device} with a null {default}"); + } + let wrong = format!("/ColorSpace << /DefaultGray {} >>", cal("CalRGB")); + assert_eq!(check("", &page("/CS /DeviceGray", &wrong), &[]), page_said, "DeviceGray with a CalRGB default"); + // Only CalGray, CalRGB and gray, RGB or CMYK ICCBased spaces blend (ISO 32000-1 11.3.4). + assert!(check("", &page("/CS [/CalGray << /WhitePoint [0.9505 1 1.089] >>]", ""), &[]).is_empty(), "CalGray"); + let icc = |n: u8, data: &str| format!("<< /N {n} /Length {} >>\nstream\n{data}\nendstream", data.len()); + assert!(check("", &page("/CS [/ICCBased 4 0 R]", ""), &[&icc(3, "")]).is_empty(), "an ICC profile of three components"); + for (what, cs, extra) in [ + ("Indexed", "/CS [/Indexed /DeviceRGB 1 <000000FFFFFF>]".to_owned(), vec![]), + ("CalRGB without its dictionary", "/CS [/CalRGB]".to_owned(), vec![]), + ("CalGray without a white point", "/CS [/CalGray << >>]".to_owned(), vec![]), + ("CalRGB with a short white point", "/CS [/CalRGB << /WhitePoint [0.9505 1] >>]".to_owned(), vec![]), + ("CalRGB with a zero white point", "/CS [/CalRGB << /WhitePoint [0 0 0] >>]".to_owned(), vec![]), + ("CalGray with a white point whose Y isn't 1", "/CS [/CalGray << /WhitePoint [0.9505 0.5 1.089] >>]".to_owned(), vec![]), + ("CalRGB with a non-number in its white point", "/CS [/CalRGB << /WhitePoint [0.9505 1 /Z 1.089] >>]".to_owned(), vec![]), + ("Lab", "/CS [/Lab << /WhitePoint [0.9505 1 1.089] >>]".to_owned(), vec![]), + ( + "Separation", + "/CS [/Separation /Spot /DeviceCMYK 4 0 R]".to_owned(), + vec!["<< /FunctionType 2 /Domain [0 1] /C1 [0 0 0 1] /N 1 >>".to_owned()], + ), + ("an ICC profile of two components", "/CS [/ICCBased 4 0 R]".to_owned(), vec![icc(2, "")]), + ("a Lab ICC profile", "/CS [/ICCBased 4 0 R]".to_owned(), vec![icc(3, &format!("{}Lab ", "\0".repeat(16)))]), + ( + "an ICC profile of one component whose header says RGB", + "/CS [/ICCBased 4 0 R]".to_owned(), + vec![icc(1, &format!("{}RGB ", "\0".repeat(16)))], + ), + ("a pattern space", "/CS /Pattern".to_owned(), vec![]), + ] { + let extra: Vec<&str> = extra.iter().map(String::as_str).collect(); + assert_eq!(check("", &page(&cs, ""), &extra), page_said, "{what}"); + } + // A null soft mask is none: an otherwise opaque graphics state is not transparency. + assert!(check("", "/Resources << /ExtGState << /G << /SMask null >> >> >>", &[]).is_empty(), "/SMask null"); + let intent = |profile: &str| format!("/OutputIntents [<< /Type /OutputIntent /S /GTS_PDFA1 /OutputConditionIdentifier (sRGB) {profile} >>]"); + let opaque_group = format!("/Resources << {alpha} >>"); + assert_eq!(check(&intent("/DestOutputProfile null"), &opaque_group, &[]), page_said, "a null profile"); + assert_eq!(check(&intent(""), &opaque_group, &[]), page_said, "no profile"); + assert!(check(&intent("/DestOutputProfile 4 0 R"), &opaque_group, &["<< /N 3 /Length 0 >>\nstream\n\nendstream"]).is_empty(), "a profile"); + // Private data is no graphics state: a /BM there is not a blend mode. + assert!(check("/PieceInfo << /App << /Private << /BM /Bookkeeping >> >> >>", "", &[]).is_empty(), "/PieceInfo"); +} + +/// A form XObject first reached at the depth limit, where its resources go unread, is read again +/// when a shallower place names it, so what it holds is not missed. +#[test] +fn transparency_reached_first_too_deep_is_found_from_a_shallower_place() { + let form = |dict: &str| format!("<< /Type /XObject /Subtype /Form /BBox [0 0 10 10] {dict} /Length 0 >>\nstream\n\nendstream"); + // A chain of 32 forms, objects 4 to 35, the last naming form 36, which holds a translucent + // graphics state; the page names the chain first and form 36 after it. + let mut objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R >>".to_owned(), + "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>".to_owned(), + "<< /Type /Page /Parent 2 0 R /Resources << /XObject << /A 4 0 R /B 36 0 R >> >> >>".to_owned(), + ]; + for n in 4..=35 { + objs.push(form(&format!("/Resources << /XObject << /X {} 0 R >> >>", n + 1))); + } + objs.push(form("/Resources << /ExtGState << /G << /ca 0.5 >> >> >>")); + let objs: Vec<&str> = objs.iter().map(String::as_str).collect(); + let said: Vec = verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| i.message).collect(); + assert_eq!(said, ["Page 1 has transparency but no device-independent blending colour space or PDF/A output intent"]); +} + +/// 6.2.10: an unknown blend mode is reported on the page whose resources reach it, as the +/// transparency issue is, each page once. +#[test] +fn an_unknown_blend_mode_names_its_page() { + let page = "<< /Type /Page /Parent 2 0 R /Group << /S /Transparency /CS [/CalRGB << /WhitePoint [0.9505 1 1.089] >>] >> /Resources << /ExtGState << /G 5 0 R >> >> >>"; + let objs = [ + "<< /Type /Catalog /Pages 2 0 R >>", + "<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 /MediaBox [0 0 200 200] >>", + "<< /Type /Page /Parent 2 0 R >>", + page, + "<< /BM /Custom >>", + ]; + let found: Vec<(String, Option)> = + verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| (i.message, i.page)).collect(); + assert_eq!(found, [("A graphics state uses the blend mode Custom, which ISO 32000-1 doesn't define".to_owned(), Some(1))]); +} + +/// A `/BM` array of many unknown names gathers a bounded number of them. +#[test] +fn unknown_blend_mode_names_are_capped() { + let names: String = (0..1000).map(|k| format!("/Custom{k} ")).collect(); + let page = format!( + "<< /Type /Page /Parent 2 0 R /Group << /S /Transparency /CS [/CalRGB << /WhitePoint [0.9505 1 1.089] >>] >> /Resources << /ExtGState << /G << /BM [/Multiply {names}] >> >> >> >>" + ); + let objs = ["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>", &page]; + let said: Vec = verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| i.message).collect(); + assert_eq!(said.len(), 32); + assert_eq!(said.first().map(String::as_str), Some("A graphics state uses the blend mode Custom0, which ISO 32000-1 doesn't define")); +} + +/// Pages that share a form read it once: the next page takes what was found under it from +/// [`transparency::Shared`] in one read, and still reports it. +#[test] +fn pages_sharing_a_form_read_it_once() { + let form = |dict: &str| format!("<< /Type /XObject /Subtype /Form /BBox [0 0 10 10] {dict} /Length 0 >>\nstream\n\nendstream"); + let pages = 50; + let first_form = 3 + pages; + let kids: String = (0..pages).map(|k| format!("{} 0 R ", 3 + k)).collect(); + let mut objs = + vec!["<< /Type /Catalog /Pages 2 0 R >>".to_owned(), format!("<< /Type /Pages /Kids [{kids}] /Count {pages} /MediaBox [0 0 200 200] >>")]; + for _ in 0..pages { + objs.push(format!("<< /Type /Page /Parent 2 0 R /Resources << /XObject << /F {first_form} 0 R >> >> >>")); + } + // A chain of ten forms, each with twenty graphics states, the last one translucent. + let gstates: String = (0..20).map(|k| format!("/G{k} << /CA 1 >> ")).collect(); + for n in 0..10 { + let next = if n < 9 { format!("/XObject << /X {} 0 R >>", first_form + n + 1) } else { "/ExtGState << /T << /ca 0.5 >> >>".to_owned() }; + objs.push(form(&format!("/Resources << /ExtGState << {gstates} >> {next} >>"))); + } + let objs: Vec<&str> = objs.iter().map(String::as_str).collect(); + let doc = build(&objs, ""); + let list = pdfcraft_model::pages(&doc); + let mut shared = transparency::Shared::default(); + let reads: Vec = list.iter().map(|p| transparency::page_reads(&doc, &p.dict, &mut shared)).collect(); + assert!(reads.first().is_some_and(|&n| n > 200), "the first page reads the chain: {reads:?}"); + assert!(reads.iter().skip(1).all(|&n| n == 2), "the others its resources and the form: {reads:?}"); + let said = verify(&doc, Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").count(); + assert_eq!(said, pages, "every page has transparency"); +} + +/// A form whose reading stopped at the depth limit on one page is not kept as read: a page that +/// names it from a shallower place reads what lies below it. +#[test] +fn a_form_cut_short_on_one_page_is_read_again_on_the_next() { + let form = |dict: &str| format!("<< /Type /XObject /Subtype /Form /BBox [0 0 10 10] {dict} /Length 0 >>\nstream\n\nendstream"); + // Page 1 (object 3) names a chain of 33 forms, objects 5 to 37, whose last holds a translucent + // graphics state beyond the depth limit; page 2 (object 4) names form 20, from which it is near. + let mut objs: Vec = vec![ + "<< /Type /Catalog /Pages 2 0 R >>".to_owned(), + "<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 /MediaBox [0 0 200 200] >>".to_owned(), + "<< /Type /Page /Parent 2 0 R /Resources << /XObject << /A 5 0 R >> >> >>".to_owned(), + "<< /Type /Page /Parent 2 0 R /Resources << /XObject << /A 20 0 R >> >> >>".to_owned(), + ]; + for n in 5..=36 { + objs.push(form(&format!("/Resources << /XObject << /X {} 0 R >> >>", n + 1))); + } + objs.push(form("/Resources << /ExtGState << /G << /ca 0.5 >> >> >>")); + let objs: Vec<&str> = objs.iter().map(String::as_str).collect(); + let said: Vec = verify(&build(&objs, ""), Level::A2b).into_iter().filter(|i| i.clause == "6.2.10").map(|i| i.message).collect(); + assert_eq!(said, ["Page 2 has transparency but no device-independent blending colour space or PDF/A output intent"]); +} diff --git a/crates/preflight/src/trailer.rs b/crates/preflight/src/trailer.rs new file mode 100644 index 0000000000..7c9d134bec --- /dev/null +++ b/crates/preflight/src/trailer.rs @@ -0,0 +1,55 @@ +//! PDF/A-2 6.1.3, the file trailer: it has an `/ID`, a file identifier as ISO 32000-1 14.4 +//! defines it, an array of two byte strings. +//! +//! The trailer read is the one the file's last `startxref` points to: the latest update's, or a +//! linearized file's first-page trailer (ISO 32000-1 F.3.4). Trailers earlier in the chain don't +//! stand in for it. + +use pdfcraft_cos::{Document, Object}; + +/// The longest string a conforming file may hold (ISO 32000-1 Annex C, Table C.1). +const MAX_STRING: usize = 32767; + +/// What is wrong with the file identifier. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum IdProblem { + /// The trailer has no `/ID`, or a null one. + Missing, + /// The `/ID` isn't an array of two non-empty strings. + Malformed, +} + +/// What 6.1.3 finds wrong with the file identifier of `doc`, if anything. +pub(crate) fn check(doc: &Document) -> Option { + let id = doc.trailer().get(b"ID").map(|id| doc.resolve(id)).filter(|id| !matches!(**id, Object::Null)); + let Some(id) = id else { return Some(IdProblem::Missing) }; + let strings = |parts: &[Object]| parts.iter().all(|p| doc.resolve(p).as_string().is_some_and(|s| !s.bytes.is_empty())); + match id.as_array() { + Some(parts) if parts.len() == 2 && strings(parts) => None, + _ => Some(IdProblem::Malformed), + } +} + +/// Gives `doc` a file identifier when it lacks a good one; what it fixed, if anything. A first +/// string a conforming file can keep (non-empty, within [`MAX_STRING`]) is the permanent +/// identifier (ISO 32000-1 14.4): it is kept, and only the second is made afresh. Otherwise both +/// are new, and the report says the identifier was added rather than completed. +pub(crate) fn fix(doc: &mut Document) -> Option { + check(doc)?; + let Object::Array(mut id) = pdfcraft_cos::new_file_id(doc.bytes(), 0) else { return None }; + let permanent = doc + .trailer() + .get(b"ID") + .map(|old| doc.resolve(old)) + .and_then(|old| old.as_array().and_then(|parts| parts.first().map(|p| doc.resolve(p)))) + .filter(|first| first.as_string().is_some_and(|s| (1..=MAX_STRING).contains(&s.bytes.len()))); + let kept = match (permanent, id.first_mut()) { + (Some(permanent), Some(slot)) => { + *slot = (*permanent).clone(); + true + } + _ => false, + }; + doc.trailer_mut().set(b"ID".to_vec(), Object::Array(id)); + Some(if kept { "Completed the file identifier (ID)" } else { "Added a file identifier (ID)" }.into()) +} diff --git a/crates/preflight/src/transparency.rs b/crates/preflight/src/transparency.rs new file mode 100644 index 0000000000..605bf4dfae --- /dev/null +++ b/crates/preflight/src/transparency.rs @@ -0,0 +1,391 @@ +//! PDF/A-2 6.2.10: a graphics state's blend mode is one ISO 32000-1 defines, and a page that +//! contains transparency, in a file without a PDF/A output intent, names its blending colour +//! space: its `/Group` dictionary has a `/CS` entry, which as any colour space in such a file is +//! not a device space without a default (6.2.4.3). +//! +//! A page contains transparency when something its resources reach can paint transparently (ISO +//! 32000-1 11.6.4.4 to 11.6.6): a graphics state with a constant alpha (`/CA`, `/ca`) below 1, +//! a blend mode other than `Normal` or `Compatible`, or a soft mask; an image with a soft mask +//! (`/SMask`, or `/SMaskInData` above 0); or a transparency group XObject. The resources looked +//! through are the page's own, those of the form XObjects, tiling patterns and Type 3 fonts they +//! hold, and those of the appearance streams of the page's annotations. Whether the content uses +//! a resource is not asked: a resource a page holds is one it may paint with. Blend modes are read +//! from the same graphics states, so private data (`/PieceInfo`) is never taken for one. + +use std::collections::HashMap; +use std::sync::Arc; + +use pdfcraft_cos::{Dict, Document, ObjRef, Object}; + +/// How many dictionaries one page's search reads, across its resources, XObjects, patterns, Type +/// 3 fonts and appearance streams. An object [`Shared`] holds counts as one read. +const MAX_VISITS: usize = 4096; +/// How deep resources nest (a form XObject in a form XObject, ...). +const MAX_DEPTH: usize = 32; +/// The resource categories that hold what a page paints with, read in this order. +const CATEGORIES: [&[u8]; 4] = [b"ExtGState", b"XObject", b"Pattern", b"Font"]; +/// How many unknown blend mode names one search gathers: enough to name them in the report, +/// without a `/BM` array of a million names comparing each with all the others. +const MAX_BLEND_NAMES: usize = 32; +/// How many objects [`Shared`] keeps what was found under. +const MAX_SHARED: usize = 1 << 16; + +/// What the graphics a page's resources reach paint with. +#[derive(Clone, Debug, Default)] +pub(crate) struct Painting { + /// Something paints transparently. + pub(crate) transparency: bool, + /// The blend modes graphics states name that ISO 32000-1 doesn't define. + pub(crate) unknown_blend_modes: Vec, +} + +impl Painting { + fn add(&mut self, other: &Painting) { + self.transparency |= other.transparency; + for name in &other.unknown_blend_modes { + self.add_blend_mode(name); + } + } + + /// Whether no more unknown blend mode names are gathered. + fn blend_modes_full(&self) -> bool { + self.unknown_blend_modes.len() >= MAX_BLEND_NAMES + } + + fn add_blend_mode(&mut self, name: &str) { + if !self.blend_modes_full() && !self.unknown_blend_modes.iter().any(|n| n == name) { + self.unknown_blend_modes.push(name.to_owned()); + } + } +} + +/// What was found under the indirect objects (resources, form XObjects, patterns, Type 3 fonts, +/// graphics states) that pages share, so the next page that names one takes it from here instead +/// of walking it again. Only an object read to the end goes in: none whose reading stopped below +/// it at the budget, at the depth limit, or at an object already read on that page (a cycle). +#[derive(Default)] +pub(crate) struct Shared(HashMap); + +/// An object read in place, or the object a reference names as the document holds it: either way +/// nothing is copied. +enum Held<'o> { + Direct(&'o Object), + Resolved(Arc), +} + +impl std::ops::Deref for Held<'_> { + type Target = Object; + + fn deref(&self) -> &Object { + match self { + Held::Direct(o) => o, + Held::Resolved(o) => o, + } + } +} + +/// `o`, or the object it refers to, without copying either ([`Document::resolve`] copies a direct +/// object). +fn held<'o>(doc: &Document, o: &'o Object) -> Held<'o> { + match o { + Object::Ref(_) => Held::Resolved(doc.resolve(o)), + o => Held::Direct(o), + } +} + +/// What the page `page` paints with: its resources, and the appearance streams of its +/// annotations. `shared` carries what earlier pages found under the objects they share. +pub(crate) fn page_painting(doc: &Document, page: &Dict, shared: &mut Shared) -> Painting { + walk_page(doc, page, shared).found +} + +/// How many reads the search of `page` took, each object taken from `shared` counting as one. +#[cfg(test)] +pub(crate) fn page_reads(doc: &Document, page: &Dict, shared: &mut Shared) -> usize { + walk_page(doc, page, shared).visits +} + +fn walk_page<'a>(doc: &'a Document, page: &Dict, shared: &'a mut Shared) -> Walk<'a> { + let mut walk = Walk { doc, shared, seen: HashMap::new(), visits: 0, cuts: 0, found: Painting::default() }; + if let Some(r) = page.get(b"Resources") { + walk.visit(r, 0, Walk::resources); + } + let annots = page.get(b"Annots").map(|a| held(doc, a)); + for a in annots.as_deref().and_then(Object::as_array).into_iter().flatten() { + let a = held(doc, a); + let Some(ap) = a.as_dict().and_then(|a| a.get(b"AP")).map(|ap| held(doc, ap)) else { continue }; + let Some(ap) = ap.as_dict() else { continue }; + // Each appearance (normal, rollover, down) is a stream, or a dictionary of streams by state. + for (_, v) in ap.iter() { + match &*held(doc, v) { + Object::Stream(_) => walk.visit(v, 0, Walk::xobject), + Object::Dict(states) => { + for (_, s) in states.iter() { + walk.visit(s, 0, Walk::xobject); + } + } + _ => {} + } + } + } + walk +} + +/// Whether `page` names a blending colour space PDF/A accepts without an output intent: its +/// `/Group` has a `/CS` that is a blending colour space (ISO 32000-1 and 32000-2 11.3.4: +/// `CalGray`, `CalRGB`, or an `ICCBased` space of 1, 3 or 4 components whose profile describes +/// gray, RGB or CMYK; not Lab, Indexed, Pattern, Separation or DeviceN), and not a device colour +/// space unless the page's resources give that device space a default (`DefaultGray`, +/// `DefaultRGB`, `DefaultCMYK`) that is itself such a space with as many components (ISO +/// 32000-1 8.6.5.6), as PDF/A-2 6.2.4.3 asks of any device colour. +pub(crate) fn page_names_blending_space(doc: &Document, page: &Dict) -> bool { + let group = page.get(b"Group").map(|g| doc.resolve(g)); + let Some(cs) = group.as_deref().and_then(Object::as_dict).and_then(|g| g.get(b"CS")).map(|c| doc.resolve(c)) else { return false }; + let (default, components): (&[u8], i64) = match &*cs { + Object::Name(n) if n == b"DeviceGray" => (b"DefaultGray", 1), + Object::Name(n) if n == b"DeviceRGB" => (b"DefaultRGB", 3), + Object::Name(n) if n == b"DeviceCMYK" => (b"DefaultCMYK", 4), + Object::Array(a) => return independent_blending_space(doc, a).is_some(), + _ => return false, + }; + let resources = page.get(b"Resources").map(|r| doc.resolve(r)); + let spaces = resources.as_deref().and_then(Object::as_dict).and_then(|r| r.get(b"ColorSpace")).map(|c| doc.resolve(c)); + let space = spaces.as_deref().and_then(Object::as_dict).and_then(|s| s.get(default)).map(|d| doc.resolve(d)); + space.as_deref().and_then(Object::as_array).and_then(|a| independent_blending_space(doc, a)) == Some(components) +} + +/// The number of components of the colour space array `a` when it is a device-independent +/// blending colour space: `CalGray` (1), `CalRGB` (3), or `ICCBased` with 1, 3 or 4 components +/// whose profile header, where it can be read (within the output intent's profile cap), says +/// gray, RGB or CMYK to match. +fn independent_blending_space(doc: &Document, a: &[Object]) -> Option { + let family = a.first().map(|f| doc.resolve(f)); + // A CalGray or CalRGB space needs its dictionary, with the required `/WhitePoint` [X Y Z]: X + // and Z positive, Y 1 (ISO 32000-1 Tables 63 and 64). + let calibrated = || { + let params = a.get(1).map(|p| doc.resolve(p)); + let white = params.as_deref().and_then(Object::as_dict).and_then(|d| d.get(b"WhitePoint")).map(|w| doc.resolve(w)); + let values: Option> = white.as_deref().and_then(Object::as_array).and_then(|w| w.iter().map(|v| doc.resolve(v).as_f64()).collect()); + matches!(values.as_deref(), Some(&[x, y, z]) if x > 0.0 && y == 1.0 && z > 0.0) + }; + match family.as_deref().and_then(Object::as_name)? { + b"CalGray" => calibrated().then_some(1), + b"CalRGB" => calibrated().then_some(3), + b"ICCBased" => { + let profile = a.get(1).map(|p| doc.resolve(p)); + let Some(Object::Stream(s)) = profile.as_deref() else { return None }; + let components = s.dict.get(b"N").and_then(|n| doc.resolve(n).as_int())?; + // ICC.1 header: the data colour space is bytes 16–19. + let header: Option = match s.decoded_within(crate::MAX_ICC_PROFILE).ok().and_then(|data| data.get(16..20).map(<[u8]>::to_vec)) { + Some(space) => match space.as_slice() { + b"GRAY" => Some(1), + b"RGB " => Some(3), + b"CMYK" => Some(4), + _ => return None, + }, + None => None, + }; + (matches!(components, 1 | 3 | 4) && header.is_none_or(|h| h == components)).then_some(components) + } + _ => None, + } +} + +struct Walk<'a> { + doc: &'a Document, + shared: &'a mut Shared, + /// The indirect objects already read on this page, each with the shallowest depth it was read + /// at: an object is read again only from a shallower place, so a cycle ends and a deep first + /// visit doesn't hide what a shallower one would read. + seen: HashMap, + visits: usize, + /// How many times reading stopped short (the budget, the depth limit, an object already read + /// on this page). An object with a stop below it doesn't go in [`Shared`]: what was found + /// under it may be incomplete. + cuts: usize, + found: Painting, +} + +impl Walk<'_> { + /// Read `o`, reached `depth` levels down, with `read`: from [`Shared`] when an earlier page + /// read it to the end, else when it is within the budget and the depth, and not an indirect + /// object read before on this page from as shallow a place. + fn visit(&mut self, o: &Object, depth: usize, read: fn(&mut Self, &Object, usize)) { + if self.visits >= MAX_VISITS || depth > MAX_DEPTH { + self.cuts = self.cuts.saturating_add(1); + return; + } + let Object::Ref(r) = o else { + self.visits += 1; + read(self, o, depth); + return; + }; + if let Some(found) = self.shared.0.get(r) { + self.visits += 1; + self.found.add(found); + return; + } + if self.seen.get(r).is_some_and(|&d| d <= depth) { + self.cuts = self.cuts.saturating_add(1); + return; + } + self.seen.insert(*r, depth); + self.visits += 1; + // Gather what this object holds on its own, to keep for the next page, then add it to what + // the page holds. + let outer = std::mem::take(&mut self.found); + let cuts = self.cuts; + read(self, o, depth); + let below = std::mem::replace(&mut self.found, outer); + self.found.add(&below); + if self.cuts == cuts && self.shared.0.len() < MAX_SHARED { + self.shared.0.insert(*r, below); + } + } + + fn resources(&mut self, r: &Object, depth: usize) { + let doc = self.doc; + let res = held(doc, r); + let Some(res) = res.as_dict() else { return }; + for category in CATEGORIES { + let entries = res.get(category).map(|d| held(doc, d)); + let Some(entries) = entries.as_deref().and_then(Object::as_dict) else { continue }; + for (_, v) in entries.iter() { + // Once the budget is spent nothing more is read, so nothing more is gathered either. + if self.visits >= MAX_VISITS { + self.cuts = self.cuts.saturating_add(1); + return; + } + match category { + b"ExtGState" => self.visit(v, depth, Self::ext_gstate), + b"XObject" => self.visit(v, depth, Self::xobject), + b"Pattern" => self.visit(v, depth, Self::pattern), + _ => self.visit(v, depth, Self::type3), + } + } + } + } + + /// A graphics state (ISO 32000-1 Table 58): whether it paints transparently, and the blend + /// modes it names that ISO 32000-1 doesn't define. + fn ext_gstate(&mut self, g: &Object, _depth: usize) { + let doc = self.doc; + let g = held(doc, g); + let Some(g) = g.as_dict() else { return }; + let below_one = |key: &[u8]| g.get(key).and_then(|v| held(doc, v).as_f64()).is_some_and(|a| a < 1.0); + let bm = g.get(b"BM").map(|b| held(doc, b)); + let blend = bm.as_deref().is_some_and(|b| blends(doc, b)); + // A soft mask is a dictionary; `/None`, null or anything else is none (ISO 32000-1 Table 58). + let mask = g.get(b"SMask").map(|m| held(doc, m)).is_some_and(|m| matches!(&*m, Object::Dict(_))); + self.found.transparency |= below_one(b"CA") || below_one(b"ca") || blend || mask; + if let Some(bm) = bm.as_deref() { + unknown_blend_modes(doc, bm, &mut self.found); + } + } + + /// An image with a soft mask, or a form XObject that is a transparency group or holds + /// transparency in its resources. + fn xobject(&mut self, x: &Object, depth: usize) { + let doc = self.doc; + let x = held(doc, x); + let Object::Stream(s) = &*x else { return }; + let d = &s.dict; + if d.name(b"Subtype") == Some(b"Image") { + let soft_mask = d.get(b"SMask").is_some_and(|m| matches!(&*held(doc, m), Object::Stream(_))); + let in_data = d.get(b"SMaskInData").and_then(|v| held(doc, v).as_int()).is_some_and(|v| v > 0); + self.found.transparency |= soft_mask || in_data; + return; + } + let group = d.get(b"Group").map(|g| held(doc, g)).is_some_and(|g| g.as_dict().is_some_and(|g| g.name(b"S") == Some(b"Transparency"))); + self.found.transparency |= group; + if let Some(r) = d.get(b"Resources") { + self.visit(r, depth + 1, Self::resources); + } + } + + /// A tiling pattern's resources, or a shading pattern's graphics state. + fn pattern(&mut self, p: &Object, depth: usize) { + let p = held(self.doc, p); + let Some(d) = (match &*p { + Object::Stream(s) => Some(&s.dict), + Object::Dict(d) => Some(d), + _ => None, + }) else { + return; + }; + if let Some(g) = d.get(b"ExtGState") { + self.visit(g, depth + 1, Self::ext_gstate); + } + if let Some(r) = d.get(b"Resources") { + self.visit(r, depth + 1, Self::resources); + } + } + + /// A Type 3 font's resources, which its glyph procedures paint with. + fn type3(&mut self, f: &Object, depth: usize) { + let f = held(self.doc, f); + let Some(d) = f.as_dict() else { return }; + if d.name(b"Subtype") != Some(b"Type3") { + return; + } + if let Some(r) = d.get(b"Resources") { + self.visit(r, depth + 1, Self::resources); + } + } +} + +/// The blend modes of ISO 32000-1 11.3.5 (Tables 136 and 137), and `Compatible` (Table 58). +const BLEND_MODES: &[&[u8]] = &[ + b"Normal", + b"Compatible", + b"Multiply", + b"Screen", + b"Overlay", + b"Darken", + b"Lighten", + b"ColorDodge", + b"ColorBurn", + b"HardLight", + b"SoftLight", + b"Difference", + b"Exclusion", + b"Hue", + b"Saturation", + b"Color", + b"Luminosity", +]; + +/// Add to `found` the names in a `/BM` value (a name, or an array of them) that are not blend +/// modes ISO 32000-1 defines, up to [`MAX_BLEND_NAMES`] of them. +fn unknown_blend_modes(doc: &Document, bm: &Object, found: &mut Painting) { + let one = |found: &mut Painting, n: &[u8]| { + if !BLEND_MODES.contains(&n) { + found.add_blend_mode(&String::from_utf8_lossy(n)); + } + }; + match bm { + Object::Name(n) => one(found, n), + Object::Array(a) => { + for m in a { + if found.blend_modes_full() { + return; + } + if let Some(n) = held(doc, m).as_name() { + one(found, n); + } + } + } + _ => {} + } +} + +/// Whether a `/BM` value blends: a name other than `Normal` or `Compatible`, or an array whose +/// first name is not one of those. +fn blends(doc: &Document, b: &Object) -> bool { + let normal = |n: &[u8]| n == b"Normal" || n == b"Compatible"; + match b { + Object::Name(n) => !normal(n), + Object::Array(a) => a.iter().find_map(|m| doc.resolve(m).as_name().map(|n| !normal(n))).unwrap_or(false), + _ => false, + } +} diff --git a/crates/preflight/src/xmp_extension.rs b/crates/preflight/src/xmp_extension.rs new file mode 100644 index 0000000000..92b3afa38e --- /dev/null +++ b/crates/preflight/src/xmp_extension.rs @@ -0,0 +1,251 @@ +//! PDF/A-2 6.6.2.3.3, the extension schema container schema: a packet describes each schema it +//! extends XMP with in `pdfaExtension:schemas`, a bag of schema descriptions, and each description +//! is complete. +//! +//! The schemas, their entries and their prefixes are those of PDF Association TechNote 0009, "XMP +//! Extension Schemas in PDF/A-1" (2008), section 4, for PDF/A-2 6.6.2.3.3: +//! - a schema (`pdfaSchema`, 4.3) has `schema`, `namespaceURI` and `prefix`; its `property` and +//! `valueType` lists are optional. TechNote 0009, written for PDF/A-1, calls `schema` an +//! optional description; PDF/A-2 requires it, as its conformance tests show; +//! - a property (`pdfaProperty`, 4.4) has `name`, `valueType`, `category` and `description`; +//! - a value type (`pdfaType`, 4.5) has `type`, `namespaceURI`, `prefix`, `description` and +//! `field`. TechNote 0009 calls `field` optional; PDF/A-2 requires it, as its conformance tests +//! show; +//! - a field (`pdfaField`, 4.6) has `name`, `valueType` and `description`; +//! - each of the five namespaces is written with its own prefix (`pdfaExtension`, `pdfaSchema`, +//! `pdfaProperty`, `pdfaType`, `pdfaField`): "Unlike predefined XMP schemas, the namespace +//! prefixes are not only preferred, but required" (4.1). +//! +//! An entry counts as present written as an element or as an attribute of the description. An +//! entry that names something is empty when it holds no value: a namespace URI or prefix, a +//! property, value type or field name, a value type reference, a property category. TechNote 0009 +//! asks for names that are valid XML names (4.4, 4.6) and a category that is internal or external +//! (4.4), and an empty prefix or namespace binds nothing. The human-readable descriptions +//! (`pdfaSchema:schema` and the `description` entries) are Text and may be empty. + +use pdfcraft_cos::ObjRef; +use roxmltree::{Document, Node}; + +const RDF: &str = "http://www.w3.org/1999/02/22-rdf-syntax-ns#"; +const EXTENSION: &str = "http://www.aiim.org/pdfa/ns/extension/"; +const SCHEMA: &str = "http://www.aiim.org/pdfa/ns/schema#"; +const PROPERTY: &str = "http://www.aiim.org/pdfa/ns/property#"; +const TYPE: &str = "http://www.aiim.org/pdfa/ns/type#"; +const FIELD: &str = "http://www.aiim.org/pdfa/ns/field#"; + +/// Each container namespace with its required prefix (TechNote 0009, 4.1). +const PREFIXES: [(&str, &str); 5] = + [(EXTENSION, "pdfaExtension"), (SCHEMA, "pdfaSchema"), (PROPERTY, "pdfaProperty"), (TYPE, "pdfaType"), (FIELD, "pdfaField")]; + +/// How many findings a document's packets are read for. A document past this many has plainly +/// failed the clause, and listing every one of a hostile packet's descriptions helps nobody. +const MAX_FINDINGS: usize = 1000; +/// How much of a name the packet gives (a namespace URI, a property) a finding repeats. +const MAX_SHOWN_CHARS: usize = 200; + +/// What the extension schema descriptions of a document break of 6.6.2.3.3, in the order the +/// metadata streams and their descriptions come. +#[derive(Debug, Default, PartialEq, Eq)] +pub(crate) struct Findings { + /// Entries descriptions lack. + pub(crate) missing: Vec, + /// Description namespaces written with another prefix, each once per stream and prefix. + pub(crate) prefixes: Vec, + /// `pdfaExtension:schemas` containers that aren't one `rdf:Bag` (TechNote 0009, 4.2: "bag + /// Schema"). + pub(crate) not_bag: Vec, +} + +/// A description without an entry it needs, or with the entry empty. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct Missing { + /// The metadata stream. + pub(crate) object: ObjRef, + /// The schema the description is in: its `pdfaSchema:namespaceURI`, or `#n` for the n-th + /// schema of its container when it has none. + pub(crate) schema: String, + /// The property, value type or field description, when it is one of those and not the schema + /// description itself: its `pdfaProperty:name` or `pdfaType:type`, a field as `type/name`, or + /// `#n` for the n-th of its list when it has no name. + pub(crate) item: Option, + /// The entry it lacks or holds empty, as `prefix:name`. + pub(crate) entry: String, + /// Whether the entry is there but empty, rather than absent. + pub(crate) empty: bool, +} + +/// A description namespace written with another prefix than its own. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct Prefix { + pub(crate) object: ObjRef, + pub(crate) namespace: &'static str, + /// The prefix it needs. + pub(crate) own: &'static str, + /// The prefix it is written with, empty when it is written as the default namespace. + pub(crate) written: String, +} + +/// A `pdfaExtension:schemas` container that isn't one `rdf:Bag`. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct NotBag { + pub(crate) object: ObjRef, + /// The elements it holds instead, by their names as the packet writes them, comma-separated; + /// empty when it holds none. + pub(crate) found: String, +} + +/// Where a description is: its stream, its schema and, below the schema, its own name. +struct Place { + object: ObjRef, + schema: String, + item: Option, +} + +impl Findings { + /// Adds what the packet `doc`, from the metadata stream `object`, breaks. + pub(crate) fn add(&mut self, object: ObjRef, doc: &Document<'_>) { + let containers = doc.descendants().filter(|n| is(n, EXTENSION, "schemas")); + for container in containers { + self.prefixes_in(object, doc, &container); + let children: Vec> = container.children().filter(Node::is_element).collect(); + if !matches!(children.as_slice(), [bag] if is(bag, RDF, "Bag")) && self.room() { + let found = children.iter().take(MAX_SHOWN_CHARS).map(|c| written_name(doc, c)).collect::>().join(", "); + self.not_bag.push(NotBag { object, found: shown(&found) }); + } + for (n, schema) in items(&container).iter().enumerate() { + let id = value(schema, SCHEMA, "namespaceURI").unwrap_or_else(|| position(n)); + let at = |item: Option| Place { object, schema: id.clone(), item }; + self.need(&at(None), schema, SCHEMA, "pdfaSchema", &["schema", "namespaceURI", "prefix"]); + let properties = entries(schema, SCHEMA, "property").flat_map(|e| items(&e)); + for (n, property) in properties.enumerate() { + let name = value(&property, PROPERTY, "name").unwrap_or_else(|| position(n)); + self.need(&at(Some(name)), &property, PROPERTY, "pdfaProperty", &["name", "valueType", "category", "description"]); + } + let value_types = entries(schema, SCHEMA, "valueType").flat_map(|e| items(&e)); + for (n, value_type) in value_types.enumerate() { + let name = value(&value_type, TYPE, "type").unwrap_or_else(|| position(n)); + self.need(&at(Some(name.clone())), &value_type, TYPE, "pdfaType", &["type", "namespaceURI", "prefix", "description", "field"]); + for (n, field) in entries(&value_type, TYPE, "field").flat_map(|e| items(&e)).enumerate() { + let field_name = value(&field, FIELD, "name").unwrap_or_else(|| position(n)); + self.need(&at(Some(format!("{name}/{field_name}"))), &field, FIELD, "pdfaField", &["name", "valueType", "description"]); + } + } + } + } + } + + /// Whether another finding may be recorded (see [`MAX_FINDINGS`]). + fn room(&self) -> bool { + self.missing.len().saturating_add(self.prefixes.len()).saturating_add(self.not_bag.len()) < MAX_FINDINGS + } + + /// Notes the entries of `ns` in `names` that the description `d`, at `at`, lacks, or holds + /// empty when they are entries that name something (see [`NAMING`]). + fn need(&mut self, at: &Place, d: &Node<'_, '_>, ns: &str, prefix: &str, names: &[&str]) { + for name in names { + let element = entry(d, ns, name); + let attribute = d.attribute((ns, *name)); + let empty = match (element, attribute) { + (None, None) => false, + (Some(e), _) => NAMING.contains(name) && is_empty(&e), + (None, Some(a)) => NAMING.contains(name) && a.trim().is_empty(), + }; + if (empty || (element.is_none() && attribute.is_none())) && self.room() { + let Place { object, schema, item } = at; + let entry = format!("{prefix}:{name}"); + self.missing.push(Missing { object: *object, schema: schema.clone(), item: item.clone(), entry, empty }); + } + } + } + + /// Notes container namespaces written in `container` (itself included) with another prefix + /// than their own. + fn prefixes_in(&mut self, object: ObjRef, doc: &Document<'_>, container: &Node<'_, '_>) { + let text = doc.input_text(); + for n in container.descendants().filter(Node::is_element) { + let element = (n.tag_name().namespace(), text.get(n.range()).unwrap_or_default().strip_prefix('<').unwrap_or_default()); + let attributes = n.attributes().map(|a| (a.namespace(), text.get(a.range_qname()).unwrap_or_default())); + for (ns, qname) in std::iter::once(element).chain(attributes) { + let Some(&(namespace, own)) = PREFIXES.iter().find(|(uri, _)| Some(*uri) == ns) else { continue }; + let written = shown(written_prefix(qname)); + if written != own + && self.room() + && !self.prefixes.iter().any(|p| p.object == object && p.namespace == namespace && p.written == written) + { + self.prefixes.push(Prefix { object, namespace, own, written }); + } + } + } + } +} + +/// The entries that name something, which an empty value leaves without a name: namespace URIs +/// and prefixes, property, value type and field names, value type references and the property +/// category. +const NAMING: [&str; 6] = ["namespaceURI", "prefix", "name", "type", "valueType", "category"]; + +/// Whether the element entry `e` holds no value: no element, no text but white space, and no +/// `rdf:resource` or `rdf:value` attribute with a value. +fn is_empty(e: &Node<'_, '_>) -> bool { + let blank = |v: Option<&str>| v.is_none_or(|v| v.trim().is_empty()); + !e.children().any(|c| c.is_element() || (c.is_text() && c.text().is_some_and(|t| !t.trim().is_empty()))) + && blank(e.attribute((RDF, "resource"))) + && blank(e.attribute((RDF, "value"))) +} + +/// `#n` for the 0-based position `n`, counted from 1. +fn position(n: usize) -> String { + format!("#{}", n.saturating_add(1)) +} + +/// At most [`MAX_SHOWN_CHARS`] of `s`, cut at a char boundary, with `…` when cut. +fn shown(s: &str) -> String { + let mut out: String = s.chars().take(MAX_SHOWN_CHARS).collect(); + if out.len() < s.len() { + out.push('…'); + } + out +} + +/// The text of the entry `name` of `ns` in the description `d`, written as an element or an +/// attribute, trimmed and [`shown`]; `None` when it is absent or blank. +fn value(d: &Node<'_, '_>, ns: &str, name: &str) -> Option { + let text = entry(d, ns, name).and_then(|e| e.text()).or_else(|| d.attribute((ns, name)))?.trim(); + (!text.is_empty()).then(|| shown(text)) +} + +/// The name of the element `n` as the packet writes it, prefix included. +fn written_name<'i>(doc: &Document<'i>, n: &Node<'_, '_>) -> &'i str { + qualified_name(doc.input_text().get(n.range()).unwrap_or_default().strip_prefix('<').unwrap_or_default()) +} + +/// The qualified name the text of a start tag (after its `<`) or of an attribute begins with. +fn qualified_name(text: &str) -> &str { + text.split(|c: char| c.is_whitespace() || c == '>' || c == '/' || c == '=').next().unwrap_or_default() +} + +/// The prefix of the qualified name `text` begins with, empty when it has none. +fn written_prefix(text: &str) -> &str { + qualified_name(text).split_once(':').map_or("", |(p, _)| p) +} + +fn is(n: &Node<'_, '_>, ns: &str, name: &str) -> bool { + n.is_element() && n.tag_name().namespace() == Some(ns) && n.tag_name().name() == name +} + +/// Every element entry `name` of `ns` in the description `d`: a list written twice is read twice. +fn entries<'a, 'i>(d: &Node<'a, 'i>, ns: &'a str, name: &'a str) -> impl Iterator> { + d.children().filter(move |c| is(c, ns, name)) +} + +/// The element entry `name` of `ns` in the description `d`. +fn entry<'a, 'i>(d: &Node<'a, 'i>, ns: &str, name: &str) -> Option> { + d.children().find(|c| is(c, ns, name)) +} + +/// The descriptions in the array `e` holds: each item's `rdf:parseType="Resource"` node or +/// `rdf:Description`. +fn items<'a, 'i>(e: &Node<'a, 'i>) -> Vec> { + let Some(array) = e.children().find(|c| is(c, RDF, "Bag") || is(c, RDF, "Seq") || is(c, RDF, "Alt")) else { return Vec::new() }; + array.children().filter(|i| is(i, RDF, "li")).map(|li| li.children().find(|c| is(c, RDF, "Description")).unwrap_or(li)).collect() +} diff --git a/crates/preflight/src/xmp_types.rs b/crates/preflight/src/xmp_types.rs index 3563a671db..cd24e59a13 100644 --- a/crates/preflight/src/xmp_types.rs +++ b/crates/preflight/src/xmp_types.rs @@ -4,7 +4,10 @@ //! The types come from ISO 16684-1:2011 (XMP Part 1) clause 8 for the Dublin Core, XMP Basic and //! XMP Rights Management namespaces, and from XMP Specification Part 2 (Additional Properties) for //! the Media Management, Basic Job Ticket, Paged-Text, Dynamic Media, Adobe PDF, Photoshop and -//! Camera Raw namespaces, with the structures they use. A value is read as RDF/XML serializes it +//! Camera Raw namespaces, with the structures they use. The TIFF, EXIF and Additional EXIF (`aux`) +//! namespaces, which the current Part 2 leaves to CIPA DC-010, come from the September 2005 XMP +//! Specification that PDF/A-2 cites: "EXIF Schemas" (pages 58–70) and "EXIF Schema Value Types" +//! (pages 83–87). A value is read as RDF/XML serializes it //! (ISO 16684-1 7.9): a simple value as text or an attribute, an array as `rdf:Seq`, `rdf:Bag` or //! `rdf:Alt`, a structure as `rdf:parseType="Resource"`, an `rdf:Description` or attributes, and a //! qualified value through its `rdf:value` (element or attribute). Every metadata stream of the @@ -12,7 +15,8 @@ //! //! The simple types are checked by their lexical form (ISO 16684-1 8.2.1): Boolean is `True` or //! `False`; Integer an optionally signed run of digits; Real a decimal; Date one of the six forms -//! of 8.2.1.2; Rational two integers around a `/`, as Part 2's `xmpDM:scale` example writes it. +//! of 8.2.1.2; Rational two integers around a `/`, as Part 2's `xmpDM:scale` example writes it and +//! the 2005 edition defines it (page 84); GPSCoordinate `DDD,MM,SSk` or `DDD,MM.mmk` (page 85). //! The other simple types are text. A field a structure doesn't define, and a property a //! namespace doesn't define, are left to the clauses about them. @@ -43,6 +47,8 @@ enum Ty { Boolean, Date, Rational, + /// `DDD,MM,SSk` or `DDD,MM.mmk`, `k` one of N, S, E, W (2005 edition, page 85). + GpsCoordinate, /// An alternative array of text items, each with an `xml:lang` (ISO 16684-1 8.2.2.4). LangAlt, Seq(&'static Ty), @@ -203,6 +209,19 @@ const ANCESTOR: &[Field] = &[(PHOTOSHOP, "AncestorID", Text)]; /// Part 2 Table 32. const LAYER: &[Field] = &[(PHOTOSHOP, "LayerName", Text), (PHOTOSHOP, "LayerText", Text)]; +const EXIF: &str = "http://ns.adobe.com/exif/1.0/"; + +/// Flash (2005 edition, page 85). +const FLASH: &[Field] = + &[(EXIF, "Fired", Boolean), (EXIF, "Return", Integer), (EXIF, "Mode", Integer), (EXIF, "Function", Boolean), (EXIF, "RedEyeMode", Boolean)]; +/// OECF/SFR (2005 edition, page 86). +const OECF_SFR: &[Field] = + &[(EXIF, "Columns", Integer), (EXIF, "Rows", Integer), (EXIF, "Names", Ty::Seq(&Text)), (EXIF, "Values", Ty::Seq(&Rational))]; +/// CFAPattern (2005 edition, page 87). +const CFA_PATTERN: &[Field] = &[(EXIF, "Columns", Integer), (EXIF, "Rows", Integer), (EXIF, "Values", Ty::Seq(&Integer))]; +/// DeviceSettings (2005 edition, page 87). +const DEVICE_SETTINGS: &[Field] = &[(EXIF, "Columns", Integer), (EXIF, "Rows", Integer), (EXIF, "Settings", Ty::Seq(&Text))]; + const SCHEMAS: &[Schema] = &[ // ISO 16684-1 Table 4. Schema { @@ -452,6 +471,126 @@ const SCHEMAS: &[Schema] = &[ ("WhiteBalance", Text), ], }, + // 2005 edition, "EXIF Schema for TIFF Properties", pages 58–60. A Closed Choice of Integer is an + // Integer. + Schema { + uri: "http://ns.adobe.com/tiff/1.0/", + prefix: "tiff", + properties: &[ + ("ImageWidth", Integer), + ("ImageLength", Integer), + ("BitsPerSample", Ty::Seq(&Integer)), + ("Compression", Integer), + ("PhotometricInterpretation", Integer), + ("Orientation", Integer), + ("SamplesPerPixel", Integer), + ("PlanarConfiguration", Integer), + ("YCbCrSubSampling", Ty::Seq(&Integer)), + ("YCbCrPositioning", Integer), + ("XResolution", Rational), + ("YResolution", Rational), + ("ResolutionUnit", Integer), + ("TransferFunction", Ty::Seq(&Integer)), + ("WhitePoint", Ty::Seq(&Rational)), + ("PrimaryChromaticities", Ty::Seq(&Rational)), + ("YCbCrCoefficients", Ty::Seq(&Rational)), + ("ReferenceBlackWhite", Ty::Seq(&Rational)), + ("DateTime", Date), + ("ImageDescription", LangAlt), + ("Make", Text), + ("Model", Text), + ("Software", Text), + ("Artist", Text), + ("Copyright", LangAlt), + ], + }, + // 2005 edition, "EXIF Schema for EXIF-specific Properties", pages 60–69. + Schema { + uri: EXIF, + prefix: "exif", + properties: &[ + ("ExifVersion", Text), + ("FlashpixVersion", Text), + ("ColorSpace", Integer), + ("ComponentsConfiguration", Ty::Seq(&Integer)), + ("CompressedBitsPerPixel", Rational), + ("PixelXDimension", Integer), + ("PixelYDimension", Integer), + ("UserComment", LangAlt), + ("RelatedSoundFile", Text), + ("DateTimeOriginal", Date), + ("DateTimeDigitized", Date), + ("ExposureTime", Rational), + ("FNumber", Rational), + ("ExposureProgram", Integer), + ("SpectralSensitivity", Text), + ("ISOSpeedRatings", Ty::Seq(&Integer)), + ("OECF", Ty::Struct("OECF/SFR", OECF_SFR)), + ("ShutterSpeedValue", Rational), + ("ApertureValue", Rational), + ("BrightnessValue", Rational), + ("ExposureBiasValue", Rational), + ("MaxApertureValue", Rational), + ("SubjectDistance", Rational), + ("MeteringMode", Integer), + ("LightSource", Integer), + ("Flash", Ty::Struct("Flash", FLASH)), + ("FocalLength", Rational), + ("SubjectArea", Ty::Seq(&Integer)), + ("FlashEnergy", Rational), + ("SpatialFrequencyResponse", Ty::Struct("OECF/SFR", OECF_SFR)), + ("FocalPlaneXResolution", Rational), + ("FocalPlaneYResolution", Rational), + ("FocalPlaneResolutionUnit", Integer), + ("SubjectLocation", Ty::Seq(&Integer)), + ("ExposureIndex", Rational), + ("SensingMethod", Integer), + ("FileSource", Integer), + ("SceneType", Integer), + ("CFAPattern", Ty::Struct("CFAPattern", CFA_PATTERN)), + ("CustomRendered", Integer), + ("ExposureMode", Integer), + ("WhiteBalance", Integer), + ("DigitalZoomRatio", Rational), + ("FocalLengthIn35mmFilm", Integer), + ("SceneCaptureType", Integer), + ("GainControl", Integer), + ("Contrast", Integer), + ("Saturation", Integer), + ("Sharpness", Integer), + ("DeviceSettingDescription", Ty::Struct("DeviceSettings", DEVICE_SETTINGS)), + ("SubjectDistanceRange", Integer), + ("ImageUniqueID", Text), + ("GPSVersionID", Text), + ("GPSLatitude", Ty::GpsCoordinate), + ("GPSLongitude", Ty::GpsCoordinate), + ("GPSAltitudeRef", Integer), + ("GPSAltitude", Rational), + ("GPSTimeStamp", Date), + ("GPSSatellites", Text), + ("GPSStatus", Text), + ("GPSMeasureMode", Text), + ("GPSDOP", Rational), + ("GPSSpeedRef", Text), + ("GPSSpeed", Rational), + ("GPSTrackRef", Text), + ("GPSTrack", Rational), + ("GPSImgDirectionRef", Text), + ("GPSImgDirection", Rational), + ("GPSMapDatum", Text), + ("GPSDestLatitude", Ty::GpsCoordinate), + ("GPSDestLongitude", Ty::GpsCoordinate), + ("GPSDestBearingRef", Text), + ("GPSDestBearing", Rational), + ("GPSDestDistanceRef", Text), + ("GPSDestDistance", Rational), + ("GPSProcessingMethod", Text), + ("GPSAreaInformation", Text), + ("GPSDifferential", Integer), + ], + }, + // 2005 edition, "EXIF Schema for Additional EXIF Properties", page 69. + Schema { uri: "http://ns.adobe.com/exif/1.0/aux/", prefix: "aux", properties: &[("Lens", Text), ("SerialNumber", Text)] }, ]; /// Calls `f` with every metadata stream among the indirect objects `objs` (ISO 32000-1 14.3.2) that @@ -535,8 +674,8 @@ fn written(qname: &str, local: &str) -> String { format!("{out}:{local}") } -/// `ty` as the XMP Specification writes value types: `Integer`, `Lang Alt`, `seq Rational`, a -/// structure by its type's name, such as `Dimensions`. +/// `ty` as the XMP Specification writes value types: `Integer`, `GPSCoordinate`, `Lang Alt`, +/// `seq Rational`, a structure by its type's name, such as `Flash`. fn named(ty: Ty) -> String { let array = |kind: &str, item: &Ty| format!("{kind} {}", named(*item)); match ty { @@ -547,6 +686,7 @@ fn named(ty: Ty) -> String { Boolean => "Boolean".into(), Date => "Date".into(), Rational => "Rational".into(), + Ty::GpsCoordinate => "GPSCoordinate".into(), LangAlt => "Lang Alt".into(), Ty::Seq(item) => array("seq", item), Ty::Bag(item) => array("bag", item), @@ -608,7 +748,7 @@ fn fits(ty: Ty, e: &Node<'_, '_>, depth: usize) -> bool { return blank(e) && blank(&n) && simple(ty, v) && predefined_ok(&n, deeper); } match ty { - Ty::Text | Ty::Guid | Ty::Integer | Ty::Real | Ty::Boolean | Ty::Date | Ty::Rational => { + Ty::Text | Ty::Guid | Ty::Integer | Ty::Real | Ty::Boolean | Ty::Date | Ty::Rational | Ty::GpsCoordinate => { // `rdf:resource` makes the value a URI, which only a text type takes. // The element is then empty. if e.attribute((RDF, "resource")).is_some() { @@ -716,12 +856,25 @@ fn simple(ty: Ty, v: &str) -> bool { Ty::Boolean => matches!(v, "True" | "False"), Ty::Date => date(v), Ty::Rational => v.split_once('/').is_some_and(|(n, d)| integer(n) && integer(d)), + Ty::GpsCoordinate => gps_coordinate(v), Ty::Text | Ty::Guid => true, Ty::OneOf(types) => types.iter().any(|t| simple(*t, v)), _ => false, } } +/// `DDD,MM,SSk` or `DDD,MM.mmk`: degrees, minutes and seconds or minutes with any number of +/// fractional digits, then N, S, E or W (2005 edition, page 85). +fn gps_coordinate(v: &str) -> bool { + let Some(v) = v.strip_suffix(['N', 'S', 'E', 'W']) else { return false }; + let Some((degrees, rest)) = v.split_once(',') else { return false }; + let minutes_ok = match rest.split_once(',') { + Some((minutes, seconds)) => digits(minutes) && digits(seconds), + None => rest.split_once('.').is_some_and(|(m, f)| digits(m) && digits(f)), + }; + digits(degrees) && minutes_ok +} + fn digits(s: &str) -> bool { !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()) } diff --git a/crates/preflight/tests/generation_repairs.rs b/crates/preflight/tests/generation_repairs.rs index c57ef141d2..68c2f2b663 100644 --- a/crates/preflight/tests/generation_repairs.rs +++ b/crates/preflight/tests/generation_repairs.rs @@ -4,7 +4,9 @@ use pdfcraft_preflight::{Level, convert}; use std::sync::Arc; fn image_file(damaged: bool) -> Arc> { - let mut bytes = b"%PDF-1.7\n".to_vec(); + // A PDF/A-2 header (6.1.2), so that Save as PDF/A has no header to rewrite and only the + // repaired references decide whether the next save is full. + let mut bytes = b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n".to_vec(); let bodies = [ "<< /Type /Catalog /Pages 2 0 R /HeaderImage 5 1 R /RowImage 5 0 R /StaleImage 5 2 R >>", "<< /Type /Pages /Kids [3 0 R] /Count 1 >>", diff --git a/crates/preflight/tests/known_answers.rs b/crates/preflight/tests/known_answers.rs index 23d503dc62..bb92e3dd83 100644 --- a/crates/preflight/tests/known_answers.rs +++ b/crates/preflight/tests/known_answers.rs @@ -22,7 +22,8 @@ use pdfcraft_preflight::{Level, convert, declared, icc, verify}; /// A document from hand-written objects and extra trailer entries (copied from the crate's unit /// tests). fn build(objs: &[&str], trailer_extra: &str) -> Document { - let mut out = b"%PDF-1.7\n".to_vec(); + // ISO 19005-2 6.1.2: the header line is followed by a comment of four bytes above 127. + let mut out = b"%PDF-1.7\n%\xE2\xE3\xCF\xD3\n".to_vec(); let mut offsets = Vec::new(); for (i, o) in objs.iter().enumerate() { offsets.push(out.len()); @@ -33,7 +34,10 @@ fn build(objs: &[&str], trailer_extra: &str) -> Document { for o in offsets { out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes()); } - out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R {trailer_extra} >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes()); + // ISO 19005-2 6.1.3: the trailer names a file identifier; a test about another clause gets one + // unless it gives its own. + let id = if trailer_extra.contains("/ID") { "" } else { "/ID [<00112233445566778899AABBCCDDEEFF> <00112233445566778899AABBCCDDEEFF>]" }; + out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R {id} {trailer_extra} >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes()); Document::open(std::sync::Arc::new(out)).unwrap() } diff --git a/crates/redact/src/interp.rs b/crates/redact/src/interp.rs index 0ff960466e..1ac7cb6714 100644 --- a/crates/redact/src/interp.rs +++ b/crates/redact/src/interp.rs @@ -31,7 +31,7 @@ mod prune; use layers::strip_hidden_layers; use prune::{clip_out, extgstate_font, prune_form_resources}; -pub(crate) use prune::{prune, prune_patterns, prune_properties, resource_ops, unlink_doomed, unlink_pruned_patterns}; +pub(crate) use prune::{prune, prune_patterns, prune_properties, resource_ops, unlink_doomed, unlink_pruned_patterns, unlink_undrawn}; /// How deep this pass follows form XObjects. A form past it is removed rather than walked, which /// is the fail-safe choice for a redaction -- marked content inside a form we cannot read would @@ -1097,7 +1097,9 @@ fn xobject( out.error = Some(error); return None; } - if let Err(error) = crate::type3::prune_type3_from_resources(doc, &mut cleaned_res, content) { + // No page: annotation appearances draw through a page's resources, never through a + // form's. The form's own soft masks are still counted. + if let Err(error) = crate::type3::prune_type3_from_resources(doc, &mut cleaned_res, content, None) { out.error = Some(error); return None; } diff --git a/crates/redact/src/interp/prune.rs b/crates/redact/src/interp/prune.rs index 188af03abe..2bfae6391f 100644 --- a/crates/redact/src/interp/prune.rs +++ b/crates/redact/src/interp/prune.rs @@ -1,6 +1,6 @@ use super::{ - CosError, Dict, Document, HashMap, HashSet, MAX_DEPTH, Matrix, Metrics, ObjRef, Object, Op, Pieces, Rc, RedactError, Scope, is_vertical, num, - overlaps, page_streams, res_dict, + CosError, DecodeBudget, Dict, Document, HashMap, HashSet, MAX_DEPTH, Matrix, Metrics, ObjRef, Object, Op, Pieces, Rc, RedactError, Scope, + is_vertical, num, overlaps, page_streams, res_dict, }; /// Drop every `replaced` resource name whose `Do` no longer appears in the rewritten `content`, @@ -20,33 +20,231 @@ pub(crate) fn prune(xo: &mut Dict, replaced: &[Vec], content: &[Vec]) { } } -/// Whether `content` draws the XObject `name` of the resource dictionary `xo`: directly, or -/// through a form that has no /Resources of its own (such a form draws through its caller's). -/// `None` when that cannot be told (a form that will not decode, or nesting past the limit). -fn content_draws(doc: &Document, content: &[Vec], xo: &Dict, name: &[u8], depth: usize, seen: &mut HashSet) -> Option { - let joined = Pieces::join(content); - let ops = joined.parse().ops; - for op in ops.iter().filter(|op| op.is("Do")) { - let Some(n) = op.last_name() else { continue }; - if n == name { - return Some(true); - } - let Some(r) = xo.get(n).and_then(Object::as_ref) else { continue }; - let obj = doc.get(r); - let Object::Stream(s) = &*obj else { continue }; - if s.dict.name(b"Subtype") != Some(b"Form") || s.dict.contains(b"Resources") || !seen.insert(r) { - continue; - } - if depth >= MAX_DEPTH { - return None; - } - let Ok(data) = s.decoded() else { return None }; - match content_draws(doc, std::slice::from_ref(&data), xo, name, depth + 1, seen) { - Some(false) => {} - other => return other, +/// The XObject names of the resource dictionary `xo` that `content` draws: directly, or through +/// a form that has no /Resources of its own (such a form draws through its caller's). Each such +/// form is read once, however many names are asked about. The flag says whether some of what +/// `content` draws could not be read (a form that will not decode, or nesting past the limit), so +/// a name missing from the set may still be drawn. +fn names_content_draws(doc: &Document, content: &[Vec], xo: &Dict) -> (HashSet>, bool) { + let mut drawn = HashSet::new(); + let mut seen = HashSet::new(); + let mut pending = vec![(Pieces::join(content).parse().ops, 0)]; + // A form that will not decode makes the answer uncertain, but it does not stop the forms + // already queued from being read: a name drawn only inside one of those need not be unknown. + let mut unknown = false; + while let Some((ops, depth)) = pending.pop() { + for n in ops.iter().filter(|op| op.is("Do")).filter_map(Op::last_name) { + if !drawn.insert(n.to_vec()) { + continue; + } + let Some(r) = xo.get(n).and_then(Object::as_ref) else { continue }; + let obj = doc.get(r); + let Object::Stream(s) = &*obj else { continue }; + if s.dict.name(b"Subtype") != Some(b"Form") || s.dict.contains(b"Resources") || !seen.insert(r) { + continue; + } + match s.decoded() { + Ok(data) if depth < MAX_DEPTH => pending.push((pdfcraft_content::parse(&data).ops, depth + 1)), + _ => unknown = true, + } } } - Some(false) + (drawn, unknown) +} + +/// Whether any of `drawn` (from [`names_content_draws`]) draws `name`: `None` when none is known to and +/// one could not be read in full. +fn any_draws(drawn: &[(HashSet>, bool)], name: &[u8]) -> Option { + if drawn.iter().any(|(names, _)| names.contains(name)) { + Some(true) + } else if drawn.iter().any(|(_, unknown)| *unknown) { + None + } else { + Some(false) + } +} + +/// Every page-tree node with a /Resources of its own, and the pages that draw through it (the +/// nearest such node above a page, or the page itself). Cycles are cut as in +/// `pdfcraft_model::pages`. The flag is false when a node was reached twice (listed under two +/// parents, or in a cycle) or the walk stopped early: a page may then draw through a holder it +/// isn't listed under. +fn resource_holders(doc: &Document) -> (Vec<(ObjRef, Vec)>, bool) { + let mut holders: Vec<(ObjRef, Vec)> = Vec::new(); + let mut whole = true; + let Some(top) = doc.root().and_then(|root| doc.get(root).as_dict().and_then(|d| d.reference(b"Pages"))) else { return (holders, whole) }; + let mut seen = HashSet::new(); + let mut stack: Vec<(ObjRef, Option)> = vec![(top, None)]; + while let Some((node, mut holder)) = stack.pop() { + if !seen.insert(node) || seen.len() > 1_000_000 { + whole = false; + continue; + } + let obj = doc.get(node); + let Some(d) = obj.as_dict() else { continue }; + if d.contains(b"Resources") { + holders.push((node, Vec::new())); + holder = Some(holders.len() - 1); + } + let is_pages = d.name(b"Type") == Some(b"Pages") || (d.contains(b"Kids") && d.name(b"Type") != Some(b"Page")); + if is_pages { + let kids = d.get(b"Kids").map(|k| doc.resolve(k)).and_then(|k| k.as_array().cloned()).unwrap_or_default(); + stack.extend(kids.iter().filter_map(Object::as_ref).map(|k| (k, holder))); + } else if let Some(h) = holder.and_then(|i| holders.get_mut(i)) { + h.1.push(node); + } + } + (holders, whole) +} + +/// Unlink the image and form XObjects nothing draws from the page-tree resource dictionaries +/// that still name them: Delete Image and Replace Image take an image off the page but leave it +/// listed there, and so in a full-rewrite save. A name stays when anything that draws through the +/// dictionary uses it ([`names_drawn`]), and a dictionary whose use can't be told is left as it +/// is, as is every one when a page is listed twice in the page tree. A dictionary that loses an +/// entry is replaced by a private copy on its node, as in [`unlink_doomed`]. Returns how many +/// names went. +pub(crate) fn unlink_undrawn(doc: &mut Document) -> Result { + let index: HashMap = pdfcraft_model::pages(doc).iter().enumerate().map(|(i, p)| (p.obj, i)).collect(); + let (holders, whole) = resource_holders(doc); + if !whole { + return Ok(0); + } + let mut n = 0; + for (node, users) in holders { + let Some(mut res) = doc.get(node).as_dict().and_then(|d| d.get(b"Resources")).and_then(|r| doc.resolve(r).as_dict().cloned()) else { + continue; + }; + let mut xo = res_dict(doc, &res, b"XObject"); + let listed: Vec> = xo + .iter() + .filter(|(_, v)| { + v.as_ref().is_some_and(|r| matches!(&*doc.get(r), Object::Stream(s) if matches!(s.dict.name(b"Subtype"), Some(b"Image" | b"Form")))) + }) + .map(|(name, _)| name.to_vec()) + .collect(); + if listed.is_empty() { + continue; + } + let mut pruned = false; + // A form that goes may have been all that drew another name: look again while names go. + for _ in 0..MAX_DEPTH { + let Some(drawn) = names_drawn(doc, &res, &users, &index) else { break }; + let mut gone = 0; + for name in listed.iter().filter(|name| !drawn.contains(*name)) { + gone += usize::from(xo.remove(name).is_some()); + } + if gone == 0 { + break; + } + n += gone; + pruned = true; + res.set(b"XObject".to_vec(), Object::Dict(xo.clone())); + } + if pruned { + doc.update_dict(node, |d| d.set(b"Resources".to_vec(), Object::Dict(res)))?; + } + } + Ok(n) +} + +/// Every name used with `Do` by what draws through `res`, the resource dictionary of `pages`: +/// their content, their annotations' appearances, and every form, tiling pattern, Type 3 glyph +/// and soft mask those can reach, with /Resources of their own or not. A name such a stream +/// doesn't define is looked up in the resources of whatever draws it, up to the page's, so a `Do` +/// anywhere in there may draw from `res`. Going by name keeps more than it must, never less. +/// Each stream is read once, against one budget. `None` when that can't be told: a page or +/// stream that won't decode, or the budget spent. +fn names_drawn(doc: &Document, res: &Dict, pages: &[ObjRef], index: &HashMap) -> Option>> { + let mut drawn = HashSet::new(); + let mut note = |data: &[u8]| { + for op in pdfcraft_content::parse(data).ops.iter().filter(|op| op.is("Do")) { + if let Some(name) = op.last_name() { + drawn.insert(name.to_vec()); + } + } + }; + // Streams still to read, and resource dictionaries still to look through. + let mut streams: Vec = Vec::new(); + let mut dicts: Vec = vec![res.clone()]; + for page in pages { + let dict = doc.get(*page).as_dict().cloned().unwrap_or_default(); + let (_, content) = page_streams(doc, &dict, index.get(page).copied().unwrap_or(0)).ok()?; + // One stream may end with a name its `Do` begins the next with: read them as one. + note(Pieces::join(&content).data()); + let annots = dict.get(b"Annots").map(|a| doc.resolve(a)).and_then(|a| a.as_array().cloned()).unwrap_or_default(); + for annot in annots { + let ap = doc.resolve(&annot).as_dict().and_then(|a| a.get(b"AP")).map(|ap| doc.resolve(ap)).and_then(|ap| ap.as_dict().cloned()); + // Each of /N, /R and /D is an appearance, or a dictionary of them by state. + for (_, look) in ap.unwrap_or_default().iter() { + match look.as_ref() { + Some(r) if matches!(&*doc.resolve(look), Object::Stream(_)) => streams.push(r), + _ => streams.extend(doc.resolve(look).as_dict().into_iter().flat_map(|states| states.iter().filter_map(|(_, v)| v.as_ref()))), + } + } + } + } + let mut budget = DecodeBudget::page(); + let mut read = HashSet::new(); + let mut looked = 0; + loop { + while let Some(dict) = dicts.get(looked) { + let (found, fonts) = drawn_through(doc, dict); + looked += 1; + streams.extend(found); + // Type 3 fonts' own resources, each look charged like a form so fonts naming each + // other can't go round for ever. + for font_res in fonts { + if !budget.enter_form() { + return None; + } + dicts.push(font_res); + } + } + let Some(r) = streams.pop() else { break }; + if !read.insert(r) { + continue; + } + let obj = doc.resolve(&Object::Ref(r)); + let Object::Stream(s) = &*obj else { continue }; + if !budget.enter_form() { + return None; + } + note(&budget.decode_strict(doc, s).ok()?); + if let Some(own) = s.dict.get(b"Resources").map(|o| doc.resolve(o)).and_then(|o| o.as_dict().cloned()) { + dicts.push(own); + } + } + Some(drawn) +} + +/// The content streams the resource dictionary `res` lists: its form XObjects, its tiling +/// patterns, its Type 3 fonts' glyph procedures and its soft masks' groups. Then the /Resources +/// of those Type 3 fonts, which their glyphs look names up in first. An object that holds only a +/// reference stands for what it names, as the renderer reads it. +fn drawn_through(doc: &Document, res: &Dict) -> (Vec, Vec) { + let is_stream = |r: &ObjRef| matches!(&*doc.resolve(&Object::Ref(*r)), Object::Stream(_)); + let mut found: Vec = Vec::new(); + let mut fonts: Vec = Vec::new(); + for (_, x) in res_dict(doc, res, b"XObject").iter() { + found.extend(x.as_ref().filter(|_| matches!(&*doc.resolve(x), Object::Stream(s) if s.dict.name(b"Subtype") == Some(b"Form")))); + } + for (_, p) in res_dict(doc, res, b"Pattern").iter() { + found.extend(p.as_ref().filter(|_| matches!(&*doc.resolve(p), Object::Stream(s) if s.dict.int(b"PatternType") == Some(1)))); + } + for (_, f) in res_dict(doc, res, b"Font").iter() { + let font = doc.resolve(f); + let Some(font) = font.as_dict().filter(|f| f.name(b"Subtype") == Some(b"Type3")) else { continue }; + let procs = font.get(b"CharProcs").map(|c| doc.resolve(c)).and_then(|c| c.as_dict().cloned()).unwrap_or_default(); + found.extend(procs.iter().filter_map(|(_, v)| v.as_ref()).filter(is_stream)); + fonts.extend(font.get(b"Resources").map(|r| doc.resolve(r)).and_then(|r| r.as_dict().cloned())); + } + for (_, g) in res_dict(doc, res, b"ExtGState").iter() { + let gs = doc.resolve(g); + let mask = gs.as_dict().and_then(|g| g.get(b"SMask")).map(|m| doc.resolve(m)); + found.extend(mask.as_ref().and_then(|m| m.as_dict()).and_then(|m| m.reference(b"G")).filter(is_stream)); + } + (found, fonts) } /// Unlink the XObjects a rewrite dropped or replaced (`doomed`) from every page-tree resource @@ -67,31 +265,7 @@ pub(crate) fn unlink_doomed(doc: &mut Document, doomed: &[ObjRef], retired_prope return Ok(()); } let doomed: HashSet = doomed.iter().copied().collect(); - let Some(top) = doc.root().and_then(|root| doc.get(root).as_dict().and_then(|d| d.reference(b"Pages"))) else { return Ok(()) }; - // Every page-tree node with a /Resources of its own, and the pages that draw through it - // (the nearest such node above a page, or the page itself). Cycles are cut as in - // `pdfcraft_model::pages`. - let mut holders: Vec<(ObjRef, Vec)> = Vec::new(); - let mut seen = HashSet::new(); - let mut stack: Vec<(ObjRef, Option)> = vec![(top, None)]; - while let Some((node, mut holder)) = stack.pop() { - if !seen.insert(node) || seen.len() > 1_000_000 { - continue; - } - let obj = doc.get(node); - let Some(d) = obj.as_dict() else { continue }; - if d.contains(b"Resources") { - holders.push((node, Vec::new())); - holder = Some(holders.len() - 1); - } - let is_pages = d.name(b"Type") == Some(b"Pages") || (d.contains(b"Kids") && d.name(b"Type") != Some(b"Page")); - if is_pages { - let kids = d.get(b"Kids").map(|k| doc.resolve(k)).and_then(|k| k.as_array().cloned()).unwrap_or_default(); - stack.extend(kids.iter().filter_map(Object::as_ref).map(|k| (k, holder))); - } else if let Some(h) = holder.and_then(|i| holders.get_mut(i)) { - h.1.push(node); - } - } + let (holders, _) = resource_holders(doc); let index: HashMap = pdfcraft_model::pages(doc).iter().enumerate().map(|(i, p)| (p.obj, i)).collect(); let holder_nodes: Vec = holders.iter().map(|(node, _)| *node).collect(); for (node, users) in holders { @@ -112,15 +286,11 @@ pub(crate) fn unlink_doomed(doc: &mut Document, doomed: &[ObjRef], retired_prope contents.push(page_streams(doc, &page, index.get(user).copied().unwrap_or(0))?.1); } let mut pruned = false; + // What each page draws through this dictionary, worked out once for all the names. + let drawn: Vec<_> = + if names.is_empty() { Vec::new() } else { contents.iter().map(|content| names_content_draws(doc, content, &xo)).collect() }; for n in names { - let mut drawn = Some(false); - for content in &contents { - drawn = content_draws(doc, content, &xo, &n, 0, &mut HashSet::new()); - if drawn != Some(false) { - break; - } - } - match drawn { + match any_draws(&drawn, &n) { Some(true) => {} Some(false) => { xo.remove(&n); @@ -230,8 +400,12 @@ fn unlink_from_forms(doc: &mut Document, holders: &[ObjRef], doomed: &HashSet], xo: &mut Dict, doomed_names: &[Vec]) -> Result { let mut pruned = false; + if doomed_names.is_empty() { + return Ok(false); + } + let drawn = [names_content_draws(doc, content, xo)]; for name in doomed_names { - match content_draws(doc, content, xo, name, 0, &mut HashSet::new()) { + match any_draws(&drawn, name) { Some(true) => {} Some(false) => pruned |= xo.remove(name).is_some(), None => return Err(CosError::Syntax { offset: 0, detail: "redaction could not fully inspect shared form resources".into() }), diff --git a/crates/redact/src/lib.rs b/crates/redact/src/lib.rs index ce5ce01d9e..12cd3f6232 100644 --- a/crates/redact/src/lib.rs +++ b/crates/redact/src/lib.rs @@ -855,7 +855,7 @@ fn redact_page_content( let painted = if out.keeps_all_patterns { &out.patterns_seen } else { &out.patterns_painted }; report.patterns_removed += interp::prune_patterns(doc, &mut res, &out.patterns_seen, painted, pruned_patterns); retired_properties.extend(interp::prune_properties(doc, &mut res, &cleaned_data)?); - type3::prune_type3_from_resources(doc, &mut res, &cleaned_data)?; + type3::prune_type3_from_resources(doc, &mut res, &cleaned_data, Some(&page.dict))?; doc.update_dict(page.obj, |d| { d.set(b"Contents".to_vec(), Object::Array(new_list)); d.set(b"Resources".to_vec(), Object::Dict(res)); diff --git a/crates/redact/src/sanitize.rs b/crates/redact/src/sanitize.rs index 7661b83ddb..1ef9a1f6de 100644 --- a/crates/redact/src/sanitize.rs +++ b/crates/redact/src/sanitize.rs @@ -410,22 +410,153 @@ fn is_comment(subtype: &[u8]) -> bool { !matches!(subtype, b"FileAttachment" | b"Redact") && pdfcraft_annot::is_comment_subtype(&String::from_utf8_lossy(subtype)) } -/// The layers that are off in the default configuration. +/// The layers that are off in the default configuration: listed off (or not listed on, under +/// `/BaseState /OFF`), or turned off on screen by one of its usage rules (`/AS`, ISO 32000-2, +/// 8.11.4.4): a rule for the View event in the View category hides a layer whose +/// `/Usage /View /ViewState` is OFF. Rules that turn a layer on, or apply only when printing or +/// exporting, leave it as the lists say. fn hidden_layers(doc: &Document) -> Vec { let Some((_, cat)) = catalog(doc) else { return Vec::new() }; let Some(oc) = sub(doc, &cat, b"OCProperties") else { return Vec::new() }; let Some(d) = sub(doc, &oc, b"D") else { return Vec::new() }; - let all: Vec = - oc.get(b"OCGs").and_then(|g| doc.resolve(g).as_array().map(|a| a.iter().filter_map(Object::as_ref).collect())).unwrap_or_default(); - let list = |k: &[u8]| -> Vec { - d.get(k).and_then(|g| doc.resolve(g).as_array().map(|a| a.iter().filter_map(Object::as_ref).collect())).unwrap_or_default() + let refs = |o: Option<&Object>| -> Vec { + o.and_then(|g| doc.resolve(g).as_array().map(|a| a.iter().filter_map(Object::as_ref).collect())).unwrap_or_default() }; - let (on, off) = (list(b"ON"), list(b"OFF")); - if d.name(b"BaseState") == Some(b"OFF") { - all.into_iter().filter(|g| !on.contains(g)).collect() + let all = refs(oc.get(b"OCGs")); + let (on, off) = (refs(d.get(b"ON")), refs(d.get(b"OFF"))); + let mut hidden: Vec = if d.name(b"BaseState") == Some(b"OFF") { + all.iter().copied().filter(|g| !on.contains(g)).collect() } else { off.into_iter().filter(|g| all.contains(g)).collect() + }; + // Each rule, layer list and layer is read once, however often the rules name them: many rules + // can share one long list. + let known: std::collections::HashSet = all.iter().copied().collect(); + let mut looked: std::collections::HashSet = hidden.iter().copied().collect(); + let mut read = std::collections::HashSet::new(); + let rules = d.get(b"AS").and_then(|a| doc.resolve(a).as_array().cloned()).unwrap_or_default(); + for rule in &rules { + if let Some(r) = rule.as_ref() + && !read.insert(r) + { + continue; + } + let Some(rule) = doc.resolve(rule).as_dict().cloned() else { continue }; + let categories = rule.get(b"Category").and_then(|c| doc.resolve(c).as_array().cloned()).unwrap_or_default(); + if rule.name(b"Event") != Some(b"View") || !categories.iter().any(|c| c.as_name() == Some(b"View")) { + continue; + } + if let Some(r) = rule.get(b"OCGs").and_then(Object::as_ref) + && !read.insert(r) + { + continue; + } + for g in refs(rule.get(b"OCGs")) { + if !known.contains(&g) || !looked.insert(g) { + continue; + } + let usage = doc.get(g).as_dict().and_then(|layer| sub(doc, layer, b"Usage")); + if usage.and_then(|u| sub(doc, &u, b"View")).is_some_and(|v| v.name(b"ViewState") == Some(b"OFF")) { + hidden.push(g); + } + } } + hidden +} + +/// The objects a layer cleanup has reached by reference, and whether the lists naming each keep it. +type Cleaned = std::collections::HashMap; + +/// `o` cleaned by `clean`: in its list when written there, and when written by reference, once and +/// where it is, however many lists name it (they keep the reference, or drop it when `clean` gives +/// `None`). A list that names one object many times costs one step per object, not one per path, +/// and a cycle stops where it began. +fn once( + doc: &mut Document, + o: &Object, + cleaned: &mut Cleaned, + clean: impl FnOnce(&mut Document, &Object, &mut Cleaned) -> Result, RedactError>, +) -> Result, RedactError> { + let Object::Ref(r) = o else { return clean(doc, o, cleaned) }; + if let Some(&kept) = cleaned.get(r) { + return Ok(kept.then(|| o.clone())); + } + cleaned.insert(*r, true); + let held: &Object = &doc.resolve(o); + let Some(new) = clean(doc, held, cleaned)? else { + cleaned.insert(*r, false); + return Ok(None); + }; + if new != *held { + doc.set_existing(*r, new)?; + } + Ok(Some(o.clone())) +} + +/// `o` without the entries `keep` refuses, through nested arrays (an `/Order` holds sub-lists, and +/// `/RBGroups` holds groups), each array written by reference cleaned [`once`]. `None` when `keep` +/// refuses `o` itself, or when `o` is a sub-list (`depth` above 0) that named a layer and names +/// none now: it goes, with the heading that labelled it. +fn filtered( + doc: &mut Document, + o: &Object, + keep: &impl Fn(&Object) -> bool, + depth: usize, + cleaned: &mut Cleaned, +) -> Result, RedactError> { + if depth > MAX_DIRECT_DEPTH { + return Err(RedactError::TooDeep); + } + if !keep(o) { + return Ok(None); + } + once(doc, o, cleaned, |doc, o, cleaned| { + let Object::Array(a) = o else { return Ok(Some(o.clone())) }; + let mut out = Vec::new(); + for x in a { + out.extend(filtered(doc, x, keep, depth + 1, cleaned)?); + } + // Anything but a text string (a heading) is a layer or a sub-list. + let names = |l: &[Object]| l.iter().any(|x| !matches!(x, Object::String(_))); + let emptied = depth > 0 && names(a.as_slice()) && !names(out.as_slice()); + Ok((!emptied).then_some(Object::Array(out))) + }) +} + +/// An optional-content configuration without the layers `keep` refuses, in every list that names +/// them: on, off, order, locked, radio-button groups and usage rules (a rule, group or sub-list left +/// with no layer goes). A removed layer still named in any of them would stay in the file, its name +/// included. +fn config_without(doc: &mut Document, mut d: Dict, keep: &impl Fn(&Object) -> bool, cleaned: &mut Cleaned) -> Result { + for k in [&b"ON"[..], b"OFF", b"Order", b"Locked", b"RBGroups"] { + if let Some(v) = d.get(k).cloned() { + let v = filtered(doc, &v, keep, 0, cleaned)?.unwrap_or_else(|| Object::Array(Vec::new())); + d.set(k.to_vec(), v); + } + } + let Some(rules) = d.get(b"AS").cloned() else { return Ok(d) }; + let rules = once(doc, &rules, cleaned, |doc, rules, cleaned| { + let Object::Array(rules) = rules else { return Ok(Some(rules.clone())) }; + let mut kept = Vec::new(); + for rule in rules { + kept.extend(once(doc, rule, cleaned, |doc, rule, cleaned| { + let Some(mut rule) = rule.as_dict().cloned() else { return Ok(None) }; + if let Some(layers) = rule.get(b"OCGs").cloned() { + let Some(layers) = filtered(doc, &layers, keep, 0, cleaned)? else { return Ok(None) }; + if doc.resolve(&layers).as_array().is_some_and(Vec::is_empty) { + return Ok(None); + } + rule.set(b"OCGs".to_vec(), layers); + } + Ok(Some(Object::Dict(rule))) + })?); + } + Ok(Some(Object::Array(kept))) + })?; + if let Some(rules) = rules { + d.set(b"AS".to_vec(), rules); + } + Ok(d) } /// Glyphs of hidden text, or content of hidden layers, on every page (`write` = remove them). @@ -511,7 +642,7 @@ fn content_pass(doc: &mut Document, text: bool, layers: bool, write: bool) -> Re res.set(b"XObject".to_vec(), Object::Dict(xo)); } retired_properties.extend(crate::interp::prune_properties(doc, &mut res, &page_content)?); - crate::type3::prune_type3_from_resources(doc, &mut res, &page_content)?; + crate::type3::prune_type3_from_resources(doc, &mut res, &page_content, Some(&page.dict))?; doc.update_dict(page.obj, |d| { d.set(b"Contents".to_vec(), Object::Array(new_list)); d.set(b"Resources".to_vec(), Object::Dict(res)); @@ -755,13 +886,19 @@ pub fn remove_hidden(doc: &mut Document, which: &[Hidden]) -> Result Ok(Some(Object::Dict(config_without(doc, c.clone(), &keep, cleaned)?))), + None => Ok(None), + })?); } - oc.set(b"D".to_vec(), Object::Dict(d)); + oc.set(b"Configs".to_vec(), Object::Array(kept)); } cat.set(b"OCProperties".to_vec(), Object::Dict(oc)); } @@ -790,7 +927,10 @@ pub fn remove_hidden(doc: &mut Document, which: &[Hidden]) -> Result