diff --git a/README.md b/README.md index 3c1b67936a..f9a41cb3ca 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Real-world typography: world scripts, vertical Japanese, colour emoji, gradients ### Fearless -Every save appends your changes and leaves the original bytes untouched. Writes are atomic, undo runs deep, and nothing is lost if you close by mistake. +Ordinary saves append your changes and leave the original bytes untouched. Edits to damaged objects whose xref and header generations disagree use a full rewrite, keeping both accepted references connected to the same repaired object. Writes are atomic, undo runs deep, and nothing is lost if you close by mistake. diff --git a/ROADMAP.md b/ROADMAP.md index a1432f1120..1f753c500f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -21,6 +21,11 @@ > **Last reviewed:** 2026-10-23 · **Last updated:** 2026-10-23 · **Change:** minor (M2, PR #4172: extend on-screen CJK fallback to Combine thumbnails and figure previews; recognise FangSong and subset-prefixed Founder names; estimates unchanged) > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (typed small Edit font sizes and Fill & Sign text/date colour, #2444/#2496; estimates unchanged) +> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** patch (M1, #1143: repaired aliases survive preflight, snapshots and normalized saves; retained materialized revisions report generation disagreements; estimates unchanged) + +> **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M1, #1143: explicit live xref/header generation repair policy and bounded diagnostics; estimates unchanged) + +> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M1: refresh object-generation reference identity against current upstream; estimates unchanged) **Stage: alpha** · next: beta, ~25% (ready for real work 50% → 75%) and ~400–700 h away > **Last reviewed:** 2026-10-29 | **Last updated:** 2026-10-29 | **Change:** minor (M2: scans keep their strokes at screen size and are smoothed when zoomed; M7 #2517: a narrowed then widened paragraph keeps the typed spaces the wrap consumed, after save and reopen; estimates unchanged) @@ -339,6 +344,8 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md - **2026-10-20 (M5, arrow keys move comments):** With a comment selected, the arrow keys move it a screen pixel and ⌘/Ctrl or Shift and an arrow ten, instead of scrolling the view or turning the page; up is up on screen at any zoom and rotation, and each press is one undo step. Text markup and locked comments stay put. Estimates unchanged. +- **2026-10-20 (M1, PR #1143 review):** Repair-aware dictionary/preflight edits and snapshot restoration preserve both accepted generations of a damaged live slot. Such edits use the existing full-save fallback; classic and packed rewrites coalesce aliases so page-tree parent links retain one object identity. Explicit generation replacement remains single-generation. Materialized xref diagnostics inspect retained primary and hybrid rows before folding, retaining the 256-object cap plus summary and discarding abandoned-index notes. Synthetic red/green controls cover preflight, undo restoration, equal values, explicit narrowing, encryption packing and four save variants. Estimates unchanged. + - **2026-10-19 (M14, #3373):** The main menu and its long submenus scroll in short windows, keeping their commands reachable. Estimates unchanged. - **2026-10-19 (M3, #3387):** Save-prompt shortcuts yield to later consent and utility dialogs; a synthetic UI regression checks that an edited document stays open. Estimates unchanged. @@ -402,6 +409,8 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md - **2026-10-15 (community, M6, #2608):** In Prepare a form the selected field moves with the arrow keys: a point per press, ten with Shift, as the object nudge and dragging it do. Each press is one undo step ("Change field properties"), and the keys no longer turn the page while a field is selected. Estimates unchanged. - **2026-10-15 (M3, closable community card):** The ArtCraft community card on Home has a close button that hides it for the rest of the run; it returns on the next launch, and its links stay in the top bar's Discord button and the Help menu. A UI regression covers closing and the fresh-launch default. Estimates unchanged. - **2026-10-13 (community, M3, #1569 review):** Pixel-only Wayland wheel, wheel-tilt and continuous input use standalone moved events, so Ctrl zoom does not remain latched after the modifier is released (#991); vendored winit patch (13), with a policy test. Estimates unchanged. +- **2026-10-18 (M1, PR #1143 repair policy):** A live in-file reference resolves when its generation matches the xref row or a same-number header at that indexed offset. Neither-match and free slots remain null, including when old object bytes remain. Opening records bounded xref/header disagreements without parsing object bodies or expanding compact xref rows. Synthetic controls cover read order, clones, overlays, dictionary edits, full classic/packed saves and an independent reader of incremental output. **Correction to the 2026-10-15 note below:** its async-opening-test sentence was not evidenced by this COS change; no fresh async-opening or GUI validation is claimed. The full-reference stream-length recursion key is defensive hardening, not an independently isolated regression fix. Estimates unchanged. +- **2026-10-15 (M1, PR #1143 refresh):** COS reference lookups and full saves distinguish complete object references, so stale generations cannot expose replacement content. Snapshot restoration also keeps edits to objects with nonzero generations before incremental saves. Synthetic regressions cover caches, overlays, snapshots, stream lengths and classic/packed saves. Current upstream async-opening test corrections are included. Estimates unchanged. - **2026-10-15 (M14, #2383 candidate):** Device-limit requests now clamp unsupported maxima and minimum alignments before enabling the adapter's full 2D texture extent. This addresses the reported V3D request of 16 shader variables on an adapter supporting 15. Synthetic regressions fail before the clamp and pass afterward, along with the existing HiDPI case and Windows GPU-required application tests. Raspberry Pi startup remains unverified; milestone and overall estimates are unchanged. - **2026-10-12 (M6):** XFDF field import and data-file extraction join all text nodes in a value, preserving text after XML comments and processing instructions. Synthetic regressions cover Unicode, CDATA, empty values and PDF save/reopen. Estimates unchanged. - **2026-10-12 (M7, picture right-click menu):** The Edit text & images menu of a picture gains Cut Image, Redact Image (a redaction mark over the picture, applied from Redact a PDF), Create Link… and Recognize Text… (for that page), in all 21 catalogs. The menu was drawn only while the pointer was over the picture, so its items outside the picture (most of a logo's menu, and Delete below a page) could not be clicked; an open menu now stays drawn. Still missing from Acrobat's menu: Arrange (bring to front, send backward), Remove crop, Draw freehand and Edit image using… (opening an external editor). @@ -780,6 +789,7 @@ Newest first: the date, what moved, and the new overall estimate. | 2026-10-20 | minor | M6, #3502: form detection preserves comments across content streams; estimates unchanged. | | 2026-10-20 | minor | M10, #3532 partial: vector printing retains existing appearances with absent Subtype; estimates unchanged. | | 2026-10-20 | minor | M11 #3507: image merging preserves comments and state across content streams. Estimates unchanged. | +| 2026-10-18 | minor | M1, #1143: a live reference may match the xref generation or a same-number header at its indexed offset; neither-match and free slots remain null. Bounded repair diagnostics and synthetic cache, free-slot, edit and classic/packed save regressions; estimates unchanged. | | 2026-10-18 | minor | M5, PR #1971: nested source `/Page` and `/Pages` dictionaries export as null without traversing page content or the page tree. Synthetic saved/reopened three-page regression retains unrelated metadata and aliases. FDF import semantics, copied-graph limits and estimates unchanged. | | 2026-10-15 | minor | M5: budget retained FDF entries before cloning; discovery keeps lazy array owners and annotation positions, with alias/cache-eviction stress fixtures. Limits cover the copied graph; source parsing/page discovery retain existing COS/parser limits, with no total source-memory bound. FDF import semantics and estimates unchanged. | | 2026-10-15 | minor | M7: the Edit text caret is as tall as the row it is on in a region with several sizes; estimates unchanged. | @@ -790,6 +800,7 @@ Newest first: the date, what moved, and the new overall estimate. | 2026-10-14 | minor | M1: raw-deflate fallback preserves decoding-limit errors in strict and tolerant decoding; synthetic filter and COS stream regressions. Estimates unchanged. | | 2026-10-16 | minor | M14: Czech catalog completed (2,674 entries) and "tag" terminology unified. Estimates unchanged. | | 2026-10-13 | minor | M9: separate page validation overlays preserve signature artwork, and snapshot tabs reuse instead of multiplying; baked captions remain unchanged (#79 open); estimates unchanged | +| 2026-10-15 | minor | M1: refresh object-generation reference identity, retaining current upstream progress and regression tests. Estimates unchanged. | | 2026-10-16 | minor | M10 #1201 partial: lossless image export applies component Decode pairs and Indexed mapping. Estimates unchanged. | | 2026-10-15 | minor | M6: CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities. Estimates unchanged. | | 2026-10-13 | minor | M4: rotation queries and edits resolve indirect page-tree values; inherited, negative and large rotations verified through both saves. Estimates unchanged. | diff --git a/crates/cos/README.md b/crates/cos/README.md index 7b576f293f..398b161625 100644 --- a/crates/cos/README.md +++ b/crates/cos/README.md @@ -53,7 +53,18 @@ another. Edits never touch the original bytes. - **Every cross-reference form:** classic tables with their quirky subsections, cross-reference streams, hybrid-reference files (`/XRefStm`), and object streams. - **Repair.** A file whose xref is wrong or missing is reconstructed by scanning for `obj` - headers. Repairs are *recorded*, never silent, so the UI can say the file was damaged. + headers. Repairs are *recorded*, never silent, so the UI can say the file was damaged. A live + in-file reference may match either its xref generation or a same-number object header at + that indexed offset; a missing, free, or neither-matching slot still resolves to null. + Header probes inspect at most 128 bytes at the indexed and header-relative offsets, without + parsing object bodies. On open, disagreements in retained xref revisions are recorded once + per object, up to 256 objects plus an omitted-details summary. This diagnostic scan does + not expand compact xref ranges; the winning xref entry remains authoritative for free + slots. Compressed objects require generation zero. Explicit `set` overlays accept only the + supplied generation; `set_existing` and `update_dict` retain both accepted generations of a + damaged live slot, including through snapshots, and require full rewrite. Headers beyond + this bounded probe are not additionally recovered. Materialized primary and hybrid rows are + diagnosed before folding, so a newer row does not hide a retained disagreement. - **Tolerant, never fatal.** The lexer works on a byte slice and a position; it allocates nothing for skipped data, bounds-checks every index, and caps nesting depth. Malformed input yields `CosError`, never a panic — this crate is the first thing a hostile file meets. Exhausted xref entries are skipped with a repair log; exhausted /Size hints allocate unused numbers without overflowing or replacing existing objects. @@ -66,14 +77,17 @@ another. Edits never touch the original bytes. ## Writing -`write_incremental` appends only the edited objects, a cross-reference section **in the same -style as the file's last one**, and a trailer with `/Prev`. The original bytes are never modified, -which is what keeps a signature over an earlier revision valid. +`write_incremental` ordinarily appends only the edited objects, a cross-reference section **in +the same style as the file's last one**, and a trailer with `/Prev`, preserving the original +bytes and signatures over earlier revisions. It delegates to `write_full` when reconstruction, +encryption changes or `full_save_required()` demand a rewrite. Repair-aware edits to a live +xref/header generation disagreement use this fallback rather than append a contradictory row. `write_full` writes only what the trailer can reach, renumbered densely — the Save As and garbage-collecting path. By default (`SaveOptions::object_streams`) it packs non-stream objects into compressed object streams with a cross-reference stream (PDF 1.5+); turning that off writes a -classic table any reader accepts. +classic table any reader accepts. Both forms map accepted xref/header aliases to one output +object, retaining shared identity such as page-tree parent links; unrelated generations stay null. Dictionaries keep their key order and streams keep their *encoded* bytes, so rewritten objects stay recognisable in a diff and byte-stable when re-serialized. diff --git a/crates/cos/src/document/objects.rs b/crates/cos/src/document/objects.rs index 62ef1e74cc..8d9917d749 100644 --- a/crates/cos/src/document/objects.rs +++ b/crates/cos/src/document/objects.rs @@ -1,6 +1,8 @@ +use std::collections::HashSet; + use super::reader::find; use super::state::{LENGTHS, LoadGuard, ObjStm, ObjectReader, ReadState, Slot, SourceIdentity, held_bytes, objstm_member}; -use super::xref::{Revision, XrefEntry}; +use super::xref::{Revision, XrefEntry, XrefIndex}; use super::{ Arc, CosError, Cow, Dict, Document, Lexer, MAX_DECODED, MAX_JBIG2_GLOBALS, ObjRef, Object, Stream, is_whitespace, parse_indirect_shared, }; @@ -129,8 +131,8 @@ impl Document { pub fn xref_entry_is_exact(&self, num: u32) -> bool { match self.xref_entry(num) { None | Some(XrefEntry::Free { .. }) => true, - Some(XrefEntry::InFile { offset, .. }) => { - let at = |off: usize| matches!(parse_indirect_shared(&self.data, off, &|_| None, &self.endstreams), Ok((id, _)) if id.num == num); + Some(XrefEntry::InFile { offset, generation }) => { + let at = |off: usize| matches!(parse_indirect_shared(&self.data, off, &|_| None, &self.endstreams), Ok((id, _)) if id == ObjRef::new(num, generation)); // Offsets may count from a header that does not start the file. usize::try_from(offset).is_ok_and(|off| at(off) || (self.header_offset != 0 && at(off.saturating_add(self.header_offset)))) } @@ -151,9 +153,93 @@ impl Document { // ── object access ─────────────────────────────────────────────────────────────────────── - /// Fetch an object by reference. Missing objects are `Null` (ISO 32000-2 §7.3.10). + /// Fetch an object by reference. Undefined references are `Null` (ISO 32000-2 §7.3.10). + /// A live file slot also accepts its actual header generation when a damaged xref row + /// disagrees; the disagreement is recorded in [`Self::repair_log`]. Free slots never do. pub fn get(&self, r: ObjRef) -> Arc { - self.try_get(r.num).unwrap_or_else(|_| Arc::new(Object::Null)) + self.try_get_reference(r).unwrap_or_else(|_| Arc::new(Object::Null)) + } + + pub(super) fn reference_matches(&self, r: ObjRef) -> bool { + match self.overlay.get(&r.num) { + Some(Slot::Set(generation, _)) => r.generation == *generation, + Some(Slot::RepairedSet(generation, alias, _)) => r.generation == *generation || r.generation == *alias, + Some(Slot::Freed(_)) => false, + None => match self.entries.lookup(r.num, &self.data) { + Some(XrefEntry::InFile { offset, generation }) => { + r.generation == generation || self.indexed_header_reference(r.num, offset) == Some(r) + } + Some(XrefEntry::InStream { .. }) => r.generation == 0, + None | Some(XrefEntry::Free { .. }) => false, + }, + } + } + + pub(super) fn try_get_reference(&self, r: ObjRef) -> Result, CosError> { + if !self.reference_matches(r) { + return Ok(Arc::new(Object::Null)); + } + self.try_get(r.num) + } + + /// A repair probe, not a whole-file search: only the indexed header (or the same offset + /// relative to a shifted PDF header) can establish this additional generation. A small + /// window bounds hostile whitespace/tokens; ordinary xref-matching reads are unchanged. + pub(super) fn indexed_header_reference(&self, num: u32, offset: u64) -> Option { + let at = |off: usize| { + let bytes = self.data.get(off..off.saturating_add(128).min(self.data.len()))?; + let mut lexer = Lexer::new(bytes, 0); + lexer.skip_ws(); + let number = std::str::from_utf8(lexer.token()).ok()?.parse::().ok()?; + lexer.skip_ws(); + let generation = std::str::from_utf8(lexer.token()).ok()?.parse::().ok()?; + (number == num && lexer.eat_keyword(b"obj")).then_some(ObjRef::new(number, generation)) + }; + let off = usize::try_from(offset).ok()?; + at(off).or_else(|| if self.header_offset == 0 { None } else { off.checked_add(self.header_offset).and_then(at) }) + } + + /// Add at most one bounded diagnostic per object. False stops diagnostic probing only; + /// parsers must still read the remaining xref rows needed to build their index. + pub(super) fn inspect_xref_generation_repair(&self, num: u32, entry: XrefEntry, recorded: &mut HashSet, log: &mut Vec) -> bool { + let XrefEntry::InFile { offset, generation } = entry else { return true }; + if recorded.contains(&num) { + return true; + } + let Some(header) = self.indexed_header_reference(num, offset) else { return true }; + if header.generation == generation { + return true; + } + if recorded.len() == 256 { + log.push("additional xref/header generation disagreements omitted after 256 objects".into()); + return false; + } + recorded.insert(num); + log.push(format!("object {num}: xref generation {generation} disagrees with object header generation {}", header.generation)); + true + } + + /// Diagnose once at open without expanding compact ranges or decoding an xref twice. + /// Materialized readers retain bounded notes while they read accepted sections, before + /// newer rows hide older ones. Discarded parse attempts/rebuilt indexes carry no notes. + pub(super) fn note_xref_generation_repairs(&self, log: &mut Vec) { + match &*self.entries { + XrefIndex::Materialized(_, notes) => log.extend(notes.iter().cloned()), + XrefIndex::Compact { revisions, .. } => { + let mut recorded = HashSet::new(); + for revision in revisions { + for section in std::iter::once(&revision.primary).chain(revision.hybrid.iter()) { + for num in section.numbers() { + if let Some(entry) = section.lookup(num, &self.data) + && !self.inspect_xref_generation_repair(num, entry, &mut recorded, log) + { + return; + } + } + } + } + } + } } /// Follow a reference if `o` is one; otherwise return `o` itself. An object can be nothing but @@ -388,7 +474,7 @@ impl Document { pub fn try_get(&self, num: u32) -> Result, CosError> { match self.overlay.get(&num) { - Some(Slot::Set(_, o)) => return Ok(o.clone()), + Some(Slot::Set(_, o) | Slot::RepairedSet(_, _, o)) => return Ok(o.clone()), Some(Slot::Freed(_)) => return Ok(Arc::new(Object::Null)), None => {} } @@ -519,11 +605,11 @@ impl Document { None | Some(XrefEntry::Free { .. }) => Ok(Object::Null), Some(XrefEntry::InFile { offset, .. }) => { let resolve = |r: ObjRef| { - if let Some(known) = LENGTHS.with(|m| m.borrow().get(&r.num).copied()) { + if let Some(known) = LENGTHS.with(|m| m.borrow().get(&r).copied()) { return known; } - let length = self.try_get(r.num).ok().and_then(|o| o.as_int()); - LENGTHS.with(|m| m.borrow_mut().insert(r.num, length)); + let length = self.try_get_reference(r).ok().and_then(|o| o.as_int()); + LENGTHS.with(|m| m.borrow_mut().insert(r, length)); length }; let off = offset as usize; @@ -640,6 +726,38 @@ impl Document { } } + /// Replace an accepted existing reference without losing a damaged source slot's aliases. + /// Unlike [`Self::set`], this preserves both xref/header generations when they disagree. + /// Such an edit forces a full rewrite so the output has consistent references, rather than + /// appending a new contradictory xref row. Missing, free and stale references are errors. + pub fn set_existing(&mut self, r: ObjRef, obj: impl Into) -> Result<(), CosError> { + if !self.reference_matches(r) { + return Err(CosError::MissingObject(r.num)); + } + let alias = self.alternate_generation(r); + if let Some(alias) = alias { + let obj = self.adopt(obj.into()); + self.overlay.insert(r.num, Slot::RepairedSet(r.generation, alias, Arc::new(obj))); + } else { + self.set(r, obj); + } + Ok(()) + } + + /// Other accepted generation for a known-live reference to a damaged slot. + pub(super) fn alternate_generation(&self, r: ObjRef) -> Option { + match self.overlay.get(&r.num) { + Some(Slot::RepairedSet(generation, alias, _)) => Some(if r.generation == *generation { *alias } else { *generation }), + Some(_) => None, + None => match self.entries.lookup(r.num, &self.data) { + Some(XrefEntry::InFile { offset, generation }) => self.indexed_header_reference(r.num, offset).and_then(|header| { + (header.generation != generation).then_some(if r.generation == generation { header.generation } else { generation }) + }), + _ => None, + }, + } + } + /// Add a new indirect object and return its reference. pub fn add(&mut self, obj: impl Into) -> ObjRef { let r = ObjRef::new(self.allocate_num(), 0); @@ -699,7 +817,8 @@ impl Document { let generation = current.generation(num); let same = saved_nums.binary_search(&num).is_ok() && self.generation(num) == generation - && saved_reader.try_get(ObjRef::new(num, 0))? == current_reader.try_get(ObjRef::new(num, 0))?; + && self.alternate_generation(ObjRef::new(num, generation)) == current.alternate_generation(ObjRef::new(num, generation)) + && saved_reader.try_get(ObjRef::new(num, generation))? == current_reader.try_get(ObjRef::new(num, generation))?; if !same { changed.push(num); } @@ -728,8 +847,14 @@ impl Document { let mut current_reader = current.object_reader(); for num in changed { let generation = current.generation(num); - let object = current_reader.try_get(ObjRef::new(num, 0))?; - self.set(ObjRef::new(num, generation), (*object).clone()); + let object = current_reader.try_get(ObjRef::new(num, generation))?; + if let Some(alias) = current.alternate_generation(ObjRef::new(num, generation)) { + let object = self.adopt((*object).clone()); + self.overlay.insert(num, Slot::RepairedSet(generation, alias, Arc::new(object))); + self.next_num = self.next_num.max(num.saturating_add(1)); + } else { + self.set(ObjRef::new(num, generation), (*object).clone()); + } } *self.trailer_mut() = current.trailer().clone(); Ok(()) @@ -740,14 +865,13 @@ impl Document { let mut obj = (*self.get(r)).clone(); let d = obj.as_dict_mut().ok_or(CosError::NotADictionary(r))?; f(d); - self.set(r, obj); - Ok(()) + self.set_existing(r, obj) } /// Current generation of an object number. pub fn generation(&self, num: u32) -> u16 { match self.overlay.get(&num) { - Some(Slot::Set(g, _)) => *g, + Some(Slot::Set(g, _) | Slot::RepairedSet(g, _, _)) => *g, Some(Slot::Freed(g)) => *g, None => match self.entries.lookup(num, &self.data) { Some(XrefEntry::InFile { generation, .. }) => generation, @@ -759,7 +883,7 @@ impl Document { pub(crate) fn overlay_entries(&self) -> impl Iterator>)> + '_ { self.overlay.iter().map(|(n, s)| match s { - Slot::Set(g, o) => (*n, *g, Some(o.clone())), + Slot::Set(g, o) | Slot::RepairedSet(g, _, o) => (*n, *g, Some(o.clone())), Slot::Freed(g) => (*n, *g, None), }) } @@ -782,7 +906,7 @@ impl Document { if source.peek().is_some_and(|n| *n == num) { source.next(); } - if matches!(slot, Slot::Set(..)) { + if matches!(slot, Slot::Set(..) | Slot::RepairedSet(..)) { numbers.push(num); } } @@ -800,7 +924,7 @@ impl Document { pub(super) fn root_is_catalog(&self) -> bool { let Some(r) = self.root() else { return false }; - matches!(self.try_get(r.num).as_deref(), Ok(Object::Dict(d)) if d.name(b"Type") == Some(b"Catalog") || d.contains(b"Pages")) + matches!(self.try_get_reference(r).as_deref(), Ok(Object::Dict(d)) if d.name(b"Type") == Some(b"Catalog") || d.contains(b"Pages")) } } diff --git a/crates/cos/src/document/open.rs b/crates/cos/src/document/open.rs index 70bbcc313e..7a4337c8da 100644 --- a/crates/cos/src/document/open.rs +++ b/crates/cos/src/document/open.rs @@ -1,5 +1,5 @@ use super::reader::find; -use super::state::Credentials; +use super::state::{Credentials, Slot}; use super::xref::XrefIndex; use super::{Arc, BTreeMap, CosError, Dict, Document, HashSet, ObjRef, Object}; @@ -156,6 +156,9 @@ impl Document { } doc.reconstruct(&mut log)?; } + // Header generations are plaintext, including in encrypted files. Inspect only the + // bounded headers, not object bodies, and retain the compact xref representation. + doc.note_xref_generation_repairs(&mut log); doc.repair_log = Arc::new(log); // Reconstruction may have found a different trailer: (re)authenticate against it. if (!ok || !authenticated) @@ -288,8 +291,9 @@ impl Document { } /// The next save must rewrite the whole file (see [`Document::require_full_save`]). + /// Repair-aware edits retaining two accepted generations cannot use one incremental slot. pub fn full_save_required(&self) -> bool { - self.full_save + self.full_save || self.overlay.values().any(|slot| matches!(slot, Slot::RepairedSet(..))) } /// Security was set or removed since the document was opened (the next save applies it). diff --git a/crates/cos/src/document/reader.rs b/crates/cos/src/document/reader.rs index d21bf9403f..e1691ace90 100644 --- a/crates/cos/src/document/reader.rs +++ b/crates/cos/src/document/reader.rs @@ -17,8 +17,8 @@ impl Document { return Ok((index, trailer, revisions, Vec::new())); } } - let (entries, trailer, revisions, oddities) = self.read_xref_chain_materialized(log)?; - Ok((XrefIndex::Materialized(Arc::new(entries)), trailer, revisions, oddities)) + let (entries, trailer, revisions, oddities, generation_repairs) = self.read_xref_chain_materialized(log)?; + Ok((XrefIndex::Materialized(Arc::new(entries), Arc::new(generation_repairs)), trailer, revisions, oddities)) } fn read_compact_xref_chain(&self, log: &mut Vec) -> Result)>, CosError> { @@ -323,7 +323,7 @@ impl Document { pub(super) fn read_xref_chain_materialized( &self, log: &mut Vec, - ) -> Result<(BTreeMap, Dict, Vec, Vec<(u64, u32)>), CosError> { + ) -> Result<(BTreeMap, Dict, Vec, Vec<(u64, u32)>, Vec), CosError> { let data = &self.data; let tail_start = data.len().saturating_sub(4096); let sx = rfind(data, b"startxref", tail_start).ok_or(CosError::Syntax { offset: data.len(), detail: "no startxref".into() })?; @@ -343,6 +343,9 @@ impl Document { // At most MAX_XREF_OBJECTS entries are kept in all; a stream past that is read again. let mut hybrids: HashMap>> = HashMap::new(); let (mut hybrid_rows, mut noted) = (0usize, HashSet::new()); + let mut generation_repairs = Vec::new(); + let mut recorded = HashSet::new(); + let mut diagnose = true; let mut next = Some(first); while let Some(off) = next.take() { if !visited.insert(off) || visited.len() > 10_000 { @@ -371,6 +374,12 @@ impl Document { } }; revisions.push(Revision { xref_offset: off, is_stream, header_relative }); + for (&num, &entry) in § { + if !diagnose { + break; + } + diagnose = self.inspect_xref_generation_repair(num, entry, &mut recorded, &mut generation_repairs); + } // Hybrid files (§7.5.8.4): the /XRefStm entries supplement the table and replace the // free entries it gives objects only a cross-reference stream can locate. let hybrid = sect_trailer.int(b"XRefStm").filter(|x| *x >= 0); @@ -390,6 +399,12 @@ impl Document { let mut stm = BTreeMap::new(); let read = match self.read_section(x as usize, &mut stm, &mut odd, log) { Ok(_) => { + for (&num, &entry) in &stm { + if !diagnose { + break; + } + diagnose = self.inspect_xref_generation_repair(num, entry, &mut recorded, &mut generation_repairs); + } for (&num, &e) in &stm { match sect.get(&num) { None | Some(XrefEntry::Free { .. }) => { @@ -427,7 +442,7 @@ impl Document { for k in [&b"Prev"[..], b"XRefStm", b"Type", b"W", b"Index", b"Filter", b"DecodeParms", b"Length"] { trailer.remove(k); } - Ok((entries, trailer, revisions, oddities)) + Ok((entries, trailer, revisions, oddities, generation_repairs)) } /// Read one section at `off`, adding entries not already known (newer sections win). `odd` @@ -606,7 +621,7 @@ impl Document { } if found > 0 { log.push(format!("the cross-reference data leaves out {found} objects the file contains; located them by their object headers")); - self.entries = Arc::new(XrefIndex::Materialized(Arc::new(filled))); + self.entries = Arc::new(XrefIndex::Materialized(Arc::new(filled), Arc::default())); } self.located = Arc::new(located); } @@ -631,7 +646,7 @@ impl Document { if entries.is_empty() { return Err(CosError::Syntax { offset: 0, detail: "no objects found".into() }); } - self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries.clone()))); + self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries.clone()), Arc::default())); self.cache.lock().map_err(|_| CosError::Poisoned)?.clear(); // Objects inside object streams. for num in objstms { @@ -648,7 +663,7 @@ impl Document { } } let object_count = entries.len(); - self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries))); + self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries), Arc::default())); self.cache.lock().map_err(|_| CosError::Poisoned)?.clear(); // Trailer: the last `trailer` dictionary or cross-reference stream dictionary with a // usable /Root (last in the file, whichever kind), else the catalog itself. One whose @@ -658,7 +673,7 @@ impl Document { // A file whose only trailers are cross-reference streams (and whose `startxref` is gone // or points nowhere) keeps its /Info, /ID and /Encrypt this way. let rank = |at: usize, d: &Dict| { - let root = self.try_get(d.reference(b"Root")?.num).ok()?; + let root = self.try_get_reference(d.reference(b"Root")?).ok()?; let Object::Dict(c) = &*root else { return None }; Some((c.name(b"Type") == Some(b"Catalog") || c.contains(b"Pages"), at)) }; diff --git a/crates/cos/src/document/state.rs b/crates/cos/src/document/state.rs index 96dde01c0f..f68a483290 100644 --- a/crates/cos/src/document/state.rs +++ b/crates/cos/src/document/state.rs @@ -7,7 +7,7 @@ thread_local! { /// Indirect `/Length` values resolved during the current outermost load: a damaged file /// retries a parse several ways, and each retry would resolve the same lengths again /// (exponential in the chain of streams whose lengths point at further streams). - pub(super) static LENGTHS: std::cell::RefCell>> = std::cell::RefCell::new(HashMap::new()); + pub(super) static LENGTHS: std::cell::RefCell>> = std::cell::RefCell::new(HashMap::new()); } /// Marks an object as being loaded on this thread. Loading can re-enter (indirect stream @@ -65,6 +65,9 @@ pub(super) fn objstm_member(num: i64, offset: i64, first: Option, len: us #[derive(Clone, Debug, PartialEq)] pub(super) enum Slot { Set(u16, Arc), + /// Both generations accepted from a damaged live source slot survive a repair-aware edit. + /// A full rewrite is required: a new xref slot cannot encode two generations. + RepairedSet(u16, u16, Arc), Freed(u16), } @@ -103,6 +106,7 @@ impl ReadState { a == b && match (x, y) { (Slot::Set(g, o), Slot::Set(h, p)) => g == h && Arc::ptr_eq(o, p), + (Slot::RepairedSet(g, a, o), Slot::RepairedSet(h, b, p)) => g == h && a == b && Arc::ptr_eq(o, p), (Slot::Freed(g), Slot::Freed(h)) => g == h, _ => false, } @@ -133,7 +137,7 @@ impl ObjectReader { } pub(crate) fn try_get(&mut self, reference: ObjRef) -> Result, CosError> { - let object = self.document.try_get(reference.num); + let object = self.document.try_get_reference(reference); // A missed eviction here only costs a re-parse, so never fail the read on a poisoned // lock. The editing-cache evictors propagate instead; they guard user edits. if let Ok(mut streams) = self.document.objstms.lock() { diff --git a/crates/cos/src/document/tests.rs b/crates/cos/src/document/tests.rs index f12437baaf..ad8f86fdea 100644 --- a/crates/cos/src/document/tests.rs +++ b/crates/cos/src/document/tests.rs @@ -599,7 +599,7 @@ pub(crate) fn build(bodies: &[&str], trailer: &str) -> Vec { fn assert_compact_matches_materialized(doc: &Document) { assert!(matches!(doc.entries.as_ref(), XrefIndex::Compact { .. }), "valid xrefs should not retain a per-entry map"); - let (entries, _, revisions, _) = doc.read_xref_chain_materialized(&mut Vec::new()).unwrap(); + let (entries, _, revisions, _, _) = doc.read_xref_chain_materialized(&mut Vec::new()).unwrap(); assert_eq!(revisions.as_slice(), doc.revisions()); let max = entries.keys().next_back().copied().unwrap_or(0); assert_eq!(doc.entries.max_num(), max); @@ -687,7 +687,7 @@ fn malformed_size_uses_the_materialized_compatibility_path() { let at = bytes.windows(marker.len()).position(|window| window == marker).unwrap(); bytes[at..at + marker.len()].copy_from_slice(b"/Size 2"); let doc = Document::open(Arc::new(bytes)).unwrap(); - assert!(matches!(doc.entries.as_ref(), XrefIndex::Materialized(_))); + assert!(matches!(doc.entries.as_ref(), XrefIndex::Materialized(..))); assert_eq!(doc.next_num, 4); assert_eq!(doc.object_numbers(), vec![1, 2, 3]); } @@ -1181,15 +1181,18 @@ fn reads_classic_revision_and_edits_are_copy_on_write() { fn object_numbers_merge_source_and_overlay_in_order() { let mut doc = Document::new_empty(); doc.overlay.clear(); - doc.entries = Arc::new(XrefIndex::Materialized(Arc::new(BTreeMap::from([ - (0, XrefEntry::Free { next_generation: 65535 }), - (2, XrefEntry::InFile { offset: 10, generation: 7 }), - (4, XrefEntry::Free { next_generation: 9 }), - (6, XrefEntry::InStream { stream: 80, index: 1 }), - (8, XrefEntry::InFile { offset: 20, generation: 5 }), - (10, XrefEntry::InFile { offset: 30, generation: 3 }), - (u32::MAX, XrefEntry::InFile { offset: 40, generation: 1 }), - ])))); + doc.entries = Arc::new(XrefIndex::Materialized( + Arc::new(BTreeMap::from([ + (0, XrefEntry::Free { next_generation: 65535 }), + (2, XrefEntry::InFile { offset: 10, generation: 7 }), + (4, XrefEntry::Free { next_generation: 9 }), + (6, XrefEntry::InStream { stream: 80, index: 1 }), + (8, XrefEntry::InFile { offset: 20, generation: 5 }), + (10, XrefEntry::InFile { offset: 30, generation: 3 }), + (u32::MAX, XrefEntry::InFile { offset: 40, generation: 1 }), + ])), + Arc::default(), + )); doc.set(ObjRef::new(1, 2), Object::Int(1)); // before the first live source slot doc.set(ObjRef::new(2, 8), Object::Int(2)); // override, without duplication doc.set(ObjRef::new(4, 9), Object::Int(4)); // reuse a source free slot diff --git a/crates/cos/src/document/xref.rs b/crates/cos/src/document/xref.rs index 02390ff7da..771df28b80 100644 --- a/crates/cos/src/document/xref.rs +++ b/crates/cos/src/document/xref.rs @@ -41,7 +41,7 @@ pub(super) struct XrefRevisionSection { #[derive(Clone, Debug)] pub(super) enum XrefIndex { Compact { revisions: Vec, max_num: u32, has_entries: bool }, - Materialized(Arc>), + Materialized(Arc>, Arc>), } pub(super) const MAX_COMPACT_XREF_ROWS: usize = 2_000_000; @@ -149,8 +149,8 @@ impl XrefSection { } } - pub(super) fn extend_numbers(&self, numbers: &mut Vec) { - for range in &self.ranges { + pub(super) fn numbers(&self) -> impl Iterator + '_ { + self.ranges.iter().flat_map(|range| { let available = match &self.data { XrefSectionData::Table => range.count as usize, XrefSectionData::Stream { bytes, widths } => { @@ -158,14 +158,12 @@ impl XrefSection { bytes.len().saturating_sub(range.offset).checked_div(row_width).unwrap_or(0) } }; - for i in 0..available.min(range.count as usize) { - if let Ok(i) = u32::try_from(i) - && let Some(num) = range.start.checked_add(i) - { - numbers.push(num); - } - } - } + (0..available.min(range.count as usize)).filter_map(move |i| u32::try_from(i).ok().and_then(|i| range.start.checked_add(i))) + }) + } + + pub(super) fn extend_numbers(&self, numbers: &mut Vec) { + numbers.extend(self.numbers()); } } @@ -189,20 +187,20 @@ impl XrefRevisionSection { impl XrefIndex { pub(super) fn empty() -> Self { - Self::Materialized(Arc::new(BTreeMap::new())) + Self::Materialized(Arc::new(BTreeMap::new()), Arc::default()) } pub(super) fn lookup(&self, num: u32, file: &[u8]) -> Option { match self { Self::Compact { revisions, .. } => revisions.iter().find_map(|revision| revision.lookup(num, file)), - Self::Materialized(entries) => entries.get(&num).copied(), + Self::Materialized(entries, _) => entries.get(&num).copied(), } } pub(super) fn is_empty(&self) -> bool { match self { Self::Compact { has_entries, .. } => !has_entries, - Self::Materialized(entries) => entries.is_empty(), + Self::Materialized(entries, _) => entries.is_empty(), } } @@ -211,14 +209,14 @@ impl XrefIndex { pub(super) fn remove_exhausted(&mut self) -> bool { match self { Self::Compact { .. } => false, - Self::Materialized(entries) => Arc::make_mut(entries).remove(&u32::MAX).is_some(), + Self::Materialized(entries, _) => Arc::make_mut(entries).remove(&u32::MAX).is_some(), } } /// Every entry, newest section first, as the materialized reader keeps them. pub(super) fn to_map(&self, file: &[u8]) -> BTreeMap { match self { - Self::Materialized(entries) => entries.as_ref().clone(), + Self::Materialized(entries, _) => entries.as_ref().clone(), Self::Compact { revisions, .. } => { let mut numbers = Vec::new(); for revision in revisions { @@ -232,13 +230,13 @@ impl XrefIndex { pub(super) fn max_num(&self) -> u32 { match self { Self::Compact { max_num, .. } => *max_num, - Self::Materialized(entries) => entries.keys().next_back().copied().unwrap_or(0), + Self::Materialized(entries, _) => entries.keys().next_back().copied().unwrap_or(0), } } pub(super) fn object_numbers(&self, file: &[u8]) -> Vec { match self { - Self::Materialized(entries) => { + Self::Materialized(entries, _) => { entries.iter().filter_map(|(&num, entry)| (!matches!(entry, XrefEntry::Free { .. })).then_some(num)).collect() } Self::Compact { revisions, .. } => { diff --git a/crates/cos/src/writer.rs b/crates/cos/src/writer.rs index dcfcb97997..6c4c71580a 100644 --- a/crates/cos/src/writer.rs +++ b/crates/cos/src/writer.rs @@ -307,8 +307,17 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result, CosErro if matches!(*o, Object::Null) { continue; // dangling reference: written as null by renumbering below } - map.insert(r, output_number(order.len())?); - order.push((r, matches!(*o, Object::Stream(_)))); + // Accepted xref/header aliases denote one slot, including page-tree cycles. + // Check for null first: an unrelated or freed generation must remain dangling. + let canonical = ObjRef::new(r.num, reader.document.generation(r.num)); + if let Some(&num) = map.get(&canonical) { + map.insert(r, num); + continue; + } + let num = output_number(order.len())?; + map.insert(r, num); + map.insert(canonical, num); + order.push((canonical, matches!(*o, Object::Stream(_)))); visit_refs(&o, &mut queue); } let version = reader.document.version().chars().take(3).collect::(); @@ -319,13 +328,15 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result, CosErro out.extend_from_slice(&[b'%', 0xE2, 0xE3, 0xCF, 0xD3, b'\n']); // The /Encrypt dictionary and everything it references (e.g. indirect /CF crypt-filter // dictionaries) are needed before anything can be decrypted: never inside object streams. - let mut encryption_objects: std::collections::HashSet = std::collections::HashSet::new(); + let mut encryption_objects: std::collections::HashSet = std::collections::HashSet::new(); let mut pending: Vec = Vec::new(); if let Some(encrypt) = trailer_in.get(b"Encrypt") { collect_refs(encrypt, &mut |r| pending.push(r)); } while let Some(r) = pending.pop() { - if encryption_objects.insert(r) { + if let Some(&num) = map.get(&r) + && encryption_objects.insert(num) + { collect_refs(&reader.get(r), &mut |x| pending.push(x)); } } @@ -333,14 +344,15 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result, CosErro // damaged and could not be repaired", before it even asks for the password), while other // readers accept it. qpdf keeps the catalog out of object streams too. Verified with // Acrobat DC on macOS (#774). - let root_outside = if reader.document.output_security().0.is_some() { trailer_in.reference(b"Root") } else { None }; + let root_outside = + if reader.document.output_security().0.is_some() { trailer_in.reference(b"Root").and_then(|r| map.get(&r)).copied() } else { None }; let mut rows: BTreeMap = BTreeMap::new(); // Keep only references, not renamed copies of the entire object graph. Standalone // objects still precede object streams, preserving the established output order. let mut packed: Vec<(u32, ObjRef)> = Vec::new(); for (i, (r, is_stream)) in order.iter().enumerate() { let num = output_number(i)?; - let in_stream = opts.object_streams && !is_stream && !encryption_objects.contains(r) && root_outside != Some(*r); + let in_stream = opts.object_streams && !is_stream && !encryption_objects.contains(&num) && root_outside != Some(num); if in_stream { packed.push((num, *r)); } else { diff --git a/crates/cos/tests/it/generations.rs b/crates/cos/tests/it/generations.rs new file mode 100644 index 0000000000..3d56628c93 --- /dev/null +++ b/crates/cos/tests/it/generations.rs @@ -0,0 +1,475 @@ +//! An indirect reference identifies an object number AND its generation (ISO 32000-2 §7.3.10). + +use std::sync::Arc; + +use hayro_syntax::object::{Dict as HayroDict, ObjectIdentifier}; +use pdfcraft_cos::{Document, ObjRef, Object, SaveOptions, XrefEntry, write_full, write_incremental}; + +/// Object 3 was reused: only 3 1 R refers to the current value, never 3 0 R. +fn reused_object() -> Arc> { + object_file(&[ + (1, 0, "<< /Type /Catalog /Pages 2 0 R /Stale 3 0 R /Current 3 1 R >>"), + (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), + (3, 1, "42"), + ]) +} + +fn object_file(objects: &[(u32, u16, &str)]) -> Arc> { + object_file_with_entry(objects, None) +} + +fn object_file_with_entry(objects: &[(u32, u16, &str)], changed: Option<(u32, u16, bool)>) -> Arc> { + let mut bytes = b"%PDF-1.7\n".to_vec(); + let mut offsets = Vec::new(); + for (num, generation, value) in objects { + offsets.push((*num, bytes.len(), *generation)); + bytes.extend_from_slice(format!("{num} {generation} obj\n{value}\nendobj\n").as_bytes()); + } + let xref = bytes.len(); + let size = objects.len() + 1; + bytes.extend_from_slice(format!("xref\n0 {size}\n0000000000 65535 f \n").as_bytes()); + for (num, offset, generation) in offsets { + let (generation, free) = changed.filter(|(n, _, _)| *n == num).map_or((generation, false), |(_, g, free)| (g, free)); + let (offset, kind) = if free { (0, 'f') } else { (offset, 'n') }; + bytes.extend_from_slice(format!("{offset:010} {generation:05} {kind} \n").as_bytes()); + } + bytes.extend_from_slice(format!("trailer\n<< /Size {size} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes()); + Arc::new(bytes) +} + +fn contradictory_entry(free: bool) -> Arc> { + object_file_with_entry( + &[(1, 0, "<< /Type /Catalog /Pages 2 0 R /Stale 3 0 R /Current 3 1 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), (3, 1, "42")], + Some((3, if free { 2 } else { 0 }, free)), + ) +} + +#[test] +fn contradictory_xref_generation_recovers_the_matching_header_and_records_repair() { + for warm in [false, true] { + let doc = Document::open(contradictory_entry(false)).unwrap(); + if warm { + assert_eq!(*doc.try_get(3).unwrap(), Object::Int(42)); + } + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42)); + assert_eq!(*doc.resolve(&Object::Ref(ObjRef::new(3, 1))), Object::Int(42)); + // Preserve the existing xref-matching behavior, but never accept an unrelated generation. + assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Int(42)); + assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Null); + assert!(!doc.xref_entry_is_exact(3), "a contradictory generation is not an exact xref entry"); + assert!( + doc.repair_log() + .iter() + .any(|line| { line.contains("object 3") && line.contains("xref generation 0") && line.contains("header generation 1") }), + "{:?}", + doc.repair_log() + ); + let repairs = doc.repair_log().to_vec(); + for _ in 0..3 { + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42)); + } + assert_eq!(doc.repair_log(), repairs); + + let mut edited = doc.clone(); + edited.free(ObjRef::new(3, 1)); + for generation in [0, 1, 2] { + assert_eq!(*edited.get(ObjRef::new(3, generation)), Object::Null); + } + edited.set(ObjRef::new(3, 2), Object::Int(99)); + assert_eq!(*edited.get(ObjRef::new(3, 1)), Object::Null); + assert_eq!(*edited.get(ObjRef::new(3, 2)), Object::Int(99)); + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42)); + } +} + +/// A damaged row may name generation zero while the retained object and catalog use one. +fn contradictory_dictionary() -> Arc> { + object_file_with_entry( + &[(1, 0, "<< /Type /Catalog /Pages 2 0 R /Current 3 1 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), (3, 1, "<< /Value 42 >>")], + Some((3, 0, false)), + ) +} + +fn assert_repaired_dictionary(bytes: Vec, value: i32) { + let bytes = Arc::new(bytes); + let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap(); + let catalog = oracle.xref().get::>(ObjectIdentifier::new(1, 0)).unwrap(); + assert_eq!(catalog.get::>(b"Current").unwrap().get::(b"Value"), Some(value)); + let back = Document::open(bytes).unwrap(); + let catalog = back.get(back.root().unwrap()); + let current = back.resolve(catalog.as_dict().unwrap().get(b"Current").unwrap()); + assert_eq!(current.as_dict().unwrap().int(b"Value"), Some(i64::from(value))); +} + +#[test] +fn repaired_header_references_survive_full_saves_and_dictionary_edits() { + let original = contradictory_dictionary(); + let doc = Document::open(original.clone()).unwrap(); + for object_streams in [false, true] { + assert_repaired_dictionary(write_full(&doc, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap(), 42); + } + let mut edited = doc.clone(); + edited.update_dict(ObjRef::new(3, 1), |dict| dict.set(b"Value".to_vec(), Object::Int(77))).unwrap(); + assert_eq!(edited.generation(3), 1); + let incremental = write_incremental(&edited, &SaveOptions::default()).unwrap(); + assert!(edited.full_save_required(), "repair-aware dictionary edits normalize on full rewrite"); + assert_repaired_dictionary(incremental, 77); + for object_streams in [false, true] { + assert_repaired_dictionary(write_full(&edited, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap(), 77); + } + let mut restored = doc.clone(); + restored.overlay_changes_from(&edited).unwrap(); + assert_repaired_dictionary(write_incremental(&restored, &SaveOptions::default()).unwrap(), 77); +} + +#[test] +fn free_xref_slots_never_revive_retained_object_headers() { + for warm in [false, true] { + let doc = Document::open(contradictory_entry(true)).unwrap(); + assert!(matches!(doc.xref_entry(3), Some(XrefEntry::Free { next_generation: 2 }))); + if warm { + assert_eq!(*doc.try_get(3).unwrap(), Object::Null); + } + for generation in [0, 1, 2] { + assert_eq!(*doc.get(ObjRef::new(3, generation)), Object::Null); + } + assert_eq!(*doc.resolve(&Object::Ref(ObjRef::new(3, 1))), Object::Null); + assert!(doc.repair_log().is_empty(), "{:?}", doc.repair_log()); + } + // Control: the retained bytes really are a readable object when its slot is live. + let live = Document::open(contradictory_entry(false)).unwrap(); + assert_eq!(*live.try_get(3).unwrap(), Object::Int(42)); +} + +#[test] +fn incremental_generation_identity_is_visible_to_an_independent_reader() { + let original = reused_object(); + let mut doc = Document::open(original.clone()).unwrap(); + doc.set(ObjRef::new(3, 1), Object::Int(77)); + let bytes = write_incremental(&doc, &SaveOptions::default()).unwrap(); + assert!(bytes.starts_with(&original)); + let oracle = hayro_syntax::Pdf::new(Arc::new(bytes)).unwrap(); + let catalog = oracle.xref().get::>(ObjectIdentifier::new(1, 0)).unwrap(); + assert_eq!(catalog.get::(b"Current"), Some(77)); + assert_eq!(catalog.get::(b"Stale"), None); +} + +#[test] +fn generation_repair_diagnostics_are_bounded_without_disabling_recovery() { + let mut objects = vec![(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>")]; + objects.extend((3..303).map(|num| (num, 1, "42"))); + let bytes = object_file(&objects); + let bytes = String::from_utf8(bytes.as_ref().clone()).unwrap().replace("00001 n \n", "00000 n \n"); + for materialized in [false, true] { + let bytes = if materialized { bytes.replace("/Size 303", "/Size 302") } else { bytes.clone() }; + let doc = Document::open(Arc::new(bytes.into_bytes())).unwrap(); + assert_eq!(doc.repair_log().len(), 257); + assert!(doc.repair_log().last().unwrap().contains("omitted after 256 objects")); + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42)); + assert_eq!(*doc.get(ObjRef::new(302, 1)), Object::Int(42)); + assert_eq!(*doc.get(ObjRef::new(302, 2)), Object::Null); + assert_eq!(doc.repair_log().len(), 257); + } +} + +#[test] +fn references_require_the_current_generation_even_with_a_warm_cache() { + let bytes = reused_object(); + let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap(); + assert_eq!(oracle.xref().get::(ObjectIdentifier::new(3, 0)), None); + assert_eq!(oracle.xref().get::(ObjectIdentifier::new(3, 1)), Some(42)); + + let doc = Document::open(bytes).unwrap(); + assert!(doc.repair_log().is_empty()); + assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null); + assert_eq!(*doc.try_get(3).unwrap(), Object::Int(42)); + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42)); + assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null); + assert_eq!(*doc.resolve(&Object::Ref(ObjRef::new(3, 0))), Object::Null); + assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Null); +} + +#[test] +fn overlay_generation_changes_do_not_retarget_old_references_or_clones() { + let mut doc = Document::open(reused_object()).unwrap(); + let snapshot = doc.clone(); + let old = ObjRef::new(3, 1); + let new = ObjRef::new(3, 2); + doc.free(old); + assert_eq!(*doc.get(old), Object::Null); + doc.set(new, Object::Int(99)); + assert_eq!(*doc.get(old), Object::Null); + assert_eq!(*doc.get(new), Object::Int(99)); + assert_eq!(*snapshot.get(old), Object::Int(42)); + assert_eq!(*snapshot.get(new), Object::Null); + + let saved = write_incremental(&doc, &SaveOptions::default()).unwrap(); + let back = Document::open(Arc::new(saved)).unwrap(); + assert!(back.repair_log().is_empty()); + assert_eq!(*back.get(old), Object::Null); + assert_eq!(*back.get(new), Object::Int(99)); +} + +#[test] +fn restoring_snapshot_preserves_changes_to_nonzero_generation_objects() { + for generation in [1, 2] { + let mut saved = Document::open(reused_object()).unwrap(); + let mut current = saved.clone(); + let reference = ObjRef::new(3, generation); + current.set(reference, Object::Int(99)); + current.update_dict(current.root().unwrap(), |catalog| catalog.set(b"Current".to_vec(), Object::Ref(reference))).unwrap(); + + saved.overlay_changes_from(¤t).unwrap(); + assert_eq!(*saved.get(reference), Object::Int(99), "generation {generation}"); + let back = Document::open(Arc::new(write_incremental(&saved, &SaveOptions::default()).unwrap())).unwrap(); + assert!(back.repair_log().is_empty()); + assert_eq!(*back.get(reference), Object::Int(99), "generation {generation}"); + let catalog = back.get(back.root().unwrap()); + assert_eq!(*back.resolve(catalog.as_dict().unwrap().get(b"Current").unwrap()), Object::Int(99)); + } +} + +#[test] +fn full_save_does_not_turn_stale_references_into_live_ones() { + let doc = Document::open(reused_object()).unwrap(); + for object_streams in [false, true] { + let saved = write_full(&doc, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap(); + let saved = Arc::new(saved); + let oracle = hayro_syntax::Pdf::new(saved.clone()).unwrap(); + let catalog = oracle.xref().get::>(ObjectIdentifier::new(1, 0)).unwrap(); + assert_eq!(catalog.get::(b"Current"), Some(42)); + assert_eq!(catalog.get::(b"Stale"), None); + let back = Document::open(saved).unwrap(); + assert!(back.repair_log().is_empty()); + let catalog = back.get(back.root().unwrap()); + let catalog = catalog.as_dict().unwrap(); + assert_eq!(catalog.get(b"Stale"), Some(&Object::Null)); + assert_eq!(*back.resolve(catalog.get(b"Current").unwrap()), Object::Int(42)); + } +} + +#[test] +fn compressed_objects_only_accept_generation_zero() { + let doc = Document::open(reused_object()).unwrap(); + let saved = write_full(&doc, &SaveOptions::default()).unwrap(); + let back = Document::open(Arc::new(saved)).unwrap(); + let catalog = back.get(back.root().unwrap()); + let current = catalog.as_dict().unwrap().reference(b"Current").unwrap(); + assert_eq!(current.generation, 0); + assert!(matches!(back.xref_entry(current.num), Some(XrefEntry::InStream { .. }))); + assert_eq!(*back.get(current), Object::Int(42)); + assert_eq!(*back.get(ObjRef::new(current.num, 1)), Object::Null); +} + +#[test] +fn indirect_stream_lengths_require_the_referenced_generation() { + for generation in [0, 1] { + // The embedded endstream is data when /Length resolves, but the existing missing- + // length recovery stops there when the reference is stale. + let stream = format!("<< /Length 4 {generation} R >>\nstream\nA\nendstream\nB\nendstream"); + let doc = Document::open(object_file(&[ + (1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), + (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), + (3, 0, &stream), + (4, 1, "13"), + ])) + .unwrap(); + let object = doc.get(ObjRef::new(3, 0)); + let Object::Stream(stream) = &*object else { panic!("expected stream") }; + assert_eq!(&stream.raw[..], if generation == 1 { b"A\nendstream\nB".as_slice() } else { b"A".as_slice() }); + } +} + +/// A newer valid row must not hide damage in a retained older revision. +fn shadowed_generation_disagreement(materialized: bool, shifted: bool) -> Arc> { + let original = contradictory_dictionary(); + let text = std::str::from_utf8(&original).unwrap(); + let previous: usize = text.rsplit("startxref\n").next().unwrap().lines().next().unwrap().parse().unwrap(); + let mut bytes = (*original).clone(); + let offset = bytes.len(); + bytes.extend_from_slice(b"3 1 obj\n<< /Value 77 >>\nendobj\n"); + let xref = bytes.len(); + let size = if materialized { 3 } else { 4 }; + bytes.extend_from_slice( + format!("xref\n3 1\n{offset:010} 00001 n \ntrailer\n<< /Size {size} /Root 1 0 R /Prev {previous} >>\nstartxref\n{xref}\n%%EOF\n").as_bytes(), + ); + if shifted { + bytes.splice(..0, b"junk before the header\n".iter().copied()); + } + Arc::new(bytes) +} + +#[test] +fn retained_generation_disagreements_are_reported_in_materialized_mode_too() { + for materialized in [false, true] { + for shifted in [false, true] { + let doc = Document::open(shadowed_generation_disagreement(materialized, shifted)).unwrap(); + assert_eq!(doc.revisions().len(), 2, "the xref chain is retained"); + assert!(!doc.reconstructed()); + assert_eq!(doc.get(ObjRef::new(3, 1)).as_dict().unwrap().int(b"Value"), Some(77)); + assert!( + doc.repair_log().iter().any(|line| line.contains("object 3: xref generation 0 disagrees with object header generation 1")), + "materialized={materialized}, shifted={shifted}: {:?}", + doc.repair_log() + ); + } + } +} + +#[test] +fn repair_aware_edits_keep_both_aliases_and_full_save_normalizes_them() { + let mut doc = Document::open(contradictory_entry(false)).unwrap(); + let snapshot = doc.clone(); + doc.set_existing(ObjRef::new(3, 0), Object::Int(77)).unwrap(); + for generation in [0, 1] { + assert_eq!(*doc.get(ObjRef::new(3, generation)), Object::Int(77)); + assert_eq!(*snapshot.get(ObjRef::new(3, generation)), Object::Int(42)); + } + let state = doc.read_state(); + assert!(doc.set_existing(ObjRef::new(3, 2), Object::Int(99)).is_err()); + assert!(state.matches(&doc)); + assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Null); + let mut restored = snapshot.clone(); + restored.overlay_changes_from(&doc).unwrap(); + for changed in [&doc, &restored] { + assert!(changed.full_save_required()); + for bytes in [ + write_incremental(changed, &SaveOptions::default()).unwrap(), + write_full(changed, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap(), + ] { + let bytes = Arc::new(bytes); + let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap(); + let catalog = oracle.xref().get::>(ObjectIdentifier::new(1, 0)).unwrap(); + assert_eq!(catalog.get::(b"Current"), Some(77)); + assert_eq!(catalog.get::(b"Stale"), Some(77), "both source aliases stay live"); + let back = Document::open(bytes).unwrap(); + assert!(back.repair_log().is_empty(), "normalized output has no generation repairs"); + } + } + // Explicit replacement keeps its narrower, caller-supplied generation contract. + doc.set(ObjRef::new(3, 2), Object::Int(99)); + assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null); + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Null); + assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Int(99)); +} + +#[test] +fn hybrid_generation_diagnostics_observe_rows_before_the_table_hides_them() { + let mut bytes = b"%PDF-1.7\n".to_vec(); + let mut offsets = Vec::new(); + for (num, generation, body) in [(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), (3, 1, "42")] { + offsets.push(bytes.len()); + bytes.extend_from_slice(format!("{num} {generation} obj\n{body}\nendobj\n").as_bytes()); + } + let hybrid = bytes.len(); + offsets.push(hybrid); + bytes.extend_from_slice(b"4 0 obj\n<< /Type /XRef /Size 5 /W [1 4 2] /Index [3 1] /Length 7 >>\nstream\n"); + bytes.push(1); + bytes.extend_from_slice(&(offsets[2] as u32).to_be_bytes()); + bytes.extend_from_slice(&0u16.to_be_bytes()); + bytes.extend_from_slice(b"\nendstream\nendobj\n"); + let xref = bytes.len(); + bytes.extend_from_slice(b"xref\n0 5\n0000000000 65535 f \n"); + for (i, offset) in offsets.into_iter().enumerate() { + let generation = u16::from(i == 2); + bytes.extend_from_slice(format!("{offset:010} {generation:05} n \n").as_bytes()); + } + bytes.extend_from_slice(format!("trailer\n<< /Size 5 /Root 1 0 R /XRefStm {hybrid} >>\nstartxref\n{xref}\n%%EOF\n").as_bytes()); + let doc = Document::open(Arc::new(bytes)).unwrap(); + assert!(!doc.reconstructed()); + assert!(doc.xref_oddities().iter().any(|(_, num)| *num == 3), "the conflicting hybrid row is set aside"); + assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42)); + assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null, "the table still determines accepted references"); + assert!(doc.repair_log().iter().any(|line| line.contains("object 3: xref generation 0 disagrees with object header generation 1"))); +} + +#[test] +fn discarded_xref_repair_notes_do_not_survive_object_header_reconstruction() { + let bytes = + String::from_utf8((*contradictory_dictionary()).clone()).unwrap().replace("/Root 1 0 R", "/Root 99 0 R").replace("/Size 4", "/Size 3"); + let doc = Document::open(Arc::new(bytes.into_bytes())).unwrap(); + assert!(doc.reconstructed()); + assert!(doc.repair_log().iter().all(|line| !line.contains("xref generation")), "{:?}", doc.repair_log()); +} + +#[test] +fn restoring_equal_values_still_restores_the_accepted_generation_aliases() { + for edited in [false, true] { + for value in [42, 77] { + let mut current = Document::open(contradictory_entry(false)).unwrap(); + if edited { + current.set_existing(ObjRef::new(3, 0), Object::Int(42)).unwrap(); + } + let mut saved = Document::open(object_file(&[ + (1, 0, "<< /Type /Catalog /Pages 2 0 R /Stale 3 0 R /Current 3 1 R >>"), + (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), + (3, 0, &value.to_string()), + ])) + .unwrap(); + assert_eq!(*saved.get(ObjRef::new(3, 1)), Object::Null); + saved.overlay_changes_from(¤t).unwrap(); + for generation in [0, 1] { + assert_eq!(*saved.get(ObjRef::new(3, generation)), Object::Int(42), "edited={edited}, value={value}"); + } + assert!(saved.full_save_required()); + } + } +} + +#[test] +fn repaired_page_tree_aliases_keep_one_identity_in_full_saves() { + let source = object_file_with_entry( + &[ + (1, 0, "<< /Type /Catalog /Pages 2 1 R /Stale 2 2 R >>"), + (2, 1, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, 0, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 10 10] >>"), + ], + Some((2, 0, false)), + ); + for edited in [false, true] { + let mut doc = Document::open(source.clone()).unwrap(); + if edited { + doc.update_dict(ObjRef::new(2, 0), |dict| dict.set(b"Edited".to_vec(), Object::Bool(true))).unwrap(); + } + let mut restored = Document::open(source.clone()).unwrap(); + restored.overlay_changes_from(&doc).unwrap(); + for current in [&doc, &restored] { + for object_streams in [false, true] { + let opts = SaveOptions { object_streams, ..SaveOptions::default() }; + let mut saves = vec![write_full(current, &opts).unwrap()]; + if edited { + saves.push(write_incremental(current, &opts).unwrap()); + } + for bytes in saves { + let back = Document::open(Arc::new(bytes)).unwrap(); + let catalog = back.get(back.root().unwrap()); + let catalog = catalog.as_dict().unwrap(); + let pages_ref = catalog.reference(b"Pages").unwrap(); + let pages = back.get(pages_ref); + let child_ref = pages.as_dict().unwrap().get(b"Kids").unwrap().as_array().unwrap()[0].as_ref().unwrap(); + let child = back.get(child_ref); + assert_eq!(child.as_dict().unwrap().reference(b"Parent"), Some(pages_ref), "edited={edited}, packed={object_streams}"); + assert!(matches!(catalog.get(b"Stale"), Some(Object::Null)), "unrelated generations stay null"); + assert!(back.repair_log().is_empty()); + } + } + } + } +} + +#[test] +fn restoring_explicit_sets_removes_old_generation_aliases() { + for value in [42, 77] { + let mut current = Document::open(contradictory_entry(false)).unwrap(); + current.set(ObjRef::new(3, 0), Object::Int(42)); + for repaired_overlay in [false, true] { + let mut saved = Document::open(contradictory_entry(false)).unwrap(); + if repaired_overlay { + saved.set_existing(ObjRef::new(3, 0), Object::Int(value)).unwrap(); + } + saved.overlay_changes_from(¤t).unwrap(); + assert_eq!(*saved.get(ObjRef::new(3, 0)), Object::Int(42)); + assert_eq!(*saved.get(ObjRef::new(3, 1)), Object::Null, "value={value}, overlay={repaired_overlay}"); + } + } +} diff --git a/crates/cos/tests/it/main.rs b/crates/cos/tests/it/main.rs index 52cf05d6c2..1bd7ee1fd6 100644 --- a/crates/cos/tests/it/main.rs +++ b/crates/cos/tests/it/main.rs @@ -3,6 +3,7 @@ mod bounded_writer; mod corpus; mod encryption; +mod generations; mod proptests; mod shared_streams; mod writer; diff --git a/crates/cos/tests/it/writer.rs b/crates/cos/tests/it/writer.rs index eb03523189..2716b9242e 100644 --- a/crates/cos/tests/it/writer.rs +++ b/crates/cos/tests/it/writer.rs @@ -358,11 +358,32 @@ fn objects_referenced_by_the_encrypt_dictionary_stay_outside_object_streams() { let cf = doc.get(enc_ref).as_dict().unwrap().get(b"CF").cloned().unwrap(); let cf_ref = doc.add(cf); doc.update_dict(enc_ref, |d| d.set(b"CF".to_vec(), Object::Ref(cf_ref))).unwrap(); - let packed = write_full(&doc, &SaveOptions::default()).unwrap(); - let back = Document::open(Arc::new(packed.clone())).expect("reopens"); - assert!(back.repair_log().is_empty(), "{:?}", back.repair_log()); - assert_eq!(title(&back), b"Caf\xe9 (draft)"); - assert_eq!(hayro_pages(&packed), 4); + // Visit this alias from Root before the encryption walk reaches /CF's other alias. + doc.update_dict(doc.root().unwrap(), |d| d.set(b"CFProbe".to_vec(), Object::Ref(cf_ref))).unwrap(); + let classic = write_full(&doc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap(); + let saved = Document::open(Arc::new(classic.clone())).unwrap(); + let cf_num = saved.get(saved.root().unwrap()).as_dict().unwrap().reference(b"CFProbe").unwrap().num; + for damaged in [false, true] { + let mut bytes = classic.clone(); + if damaged { + // Equal-length edits preserve the xref offsets and its generation-zero row. + for (from, to) in [(format!("\n{cf_num} 0 obj"), format!("\n{cf_num} 1 obj")), (format!("/CF {cf_num} 0 R"), format!("/CF {cf_num} 1 R"))] + { + let start = bytes.windows(from.len()).position(|w| w == from.as_bytes()).unwrap(); + bytes[start..start + from.len()].copy_from_slice(to.as_bytes()); + } + } + let doc = Document::open(Arc::new(bytes)).unwrap(); + let packed = write_full(&doc, &SaveOptions::default()).unwrap(); + let back = Document::open(Arc::new(packed.clone())).expect("reopens"); + assert!(back.repair_log().is_empty(), "{:?}", back.repair_log()); + assert_eq!(title(&back), b"Caf\xe9 (draft)"); + assert_eq!(hayro_pages(&packed), 4); + let probe = back.get(back.root().unwrap()).as_dict().unwrap().reference(b"CFProbe").unwrap(); + let encrypt = back.get(back.trailer().reference(b"Encrypt").unwrap()); + assert_eq!(encrypt.as_dict().unwrap().reference(b"CF"), Some(probe)); + assert!(matches!(back.xref_entry(probe.num), Some(pdfcraft_cos::XrefEntry::InFile { .. })), "crypt filters must stay outside object streams"); + } } #[test] diff --git a/crates/preflight/src/lib.rs b/crates/preflight/src/lib.rs index 1df39ebf05..1fb7272b33 100644 --- a/crates/preflight/src/lib.rs +++ b/crates/preflight/src/lib.rs @@ -894,7 +894,7 @@ pub fn convert(doc: &mut Document, level: Level) -> Result Result Result Result Arc> { + let mut bytes = b"%PDF-1.7\n".to_vec(); + let bodies = [ + "<< /Type /Catalog /Pages 2 0 R /HeaderImage 5 1 R /RowImage 5 0 R /StaleImage 5 2 R >>", + "<< /Type /Pages /Kids [3 0 R] /Count 1 >>", + "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 10 10] /Resources << /XObject << /Im 5 1 R >> >> >>", + "null", + "<< /Type /XObject /Subtype /Image /Width 1 /Height 1 /BitsPerComponent 8 /ColorSpace /DeviceGray /Interpolate true /Length 1 >>\nstream\nX\nendstream", + ]; + let mut offsets = Vec::new(); + for (i, body) in bodies.iter().enumerate() { + offsets.push(bytes.len()); + let generation = u16::from(i == 4); + bytes.extend_from_slice(format!("{} {generation} obj\n{body}\nendobj\n", i + 1).as_bytes()); + } + let xref = bytes.len(); + bytes.extend_from_slice(b"xref\n0 6\n0000000000 65535 f \n"); + for (i, offset) in offsets.into_iter().enumerate() { + let generation = u16::from(i == 4 && !damaged); + bytes.extend_from_slice(format!("{offset:010} {generation:05} n \n").as_bytes()); + } + bytes.extend_from_slice(format!("trailer\n<< /Size 6 /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes()); + Arc::new(bytes) +} + +fn assert_images(doc: &Document, converted: bool, dual: bool) { + let root = doc.get(doc.root().unwrap()); + let catalog = root.as_dict().unwrap(); + for key in [b"HeaderImage".as_slice(), b"RowImage".as_slice()] { + let value = doc.resolve(catalog.get(key).unwrap()); + if key == b"RowImage" && !dual { + assert_eq!(*value, Object::Null); + } else { + let Object::Stream(image) = &*value else { panic!("{key:?} became {value:?}") }; + assert_eq!(image.dict.get(b"Interpolate"), Some(&Object::Bool(!converted))); + assert_eq!(&image.raw[..], b"X"); + } + } + assert_eq!(*doc.resolve(catalog.get(b"StaleImage").unwrap()), Object::Null); +} + +#[test] +fn preflight_preserves_repaired_image_references_in_snapshots_and_saves() { + for damaged in [false, true] { + let original = image_file(damaged); + let mut doc = Document::open(original.clone()).unwrap(); + assert_images(&doc, false, damaged); + let snapshot = doc.clone(); + let report = convert(&mut doc, Level::A2b).unwrap(); + assert!(report.fixed.iter().any(|line| line.contains("cleared image interpolation"))); + assert_images(&doc, true, damaged); + assert_eq!(doc.full_save_required(), damaged); + assert_images(&snapshot, false, damaged); + let mut restored = snapshot.clone(); + restored.overlay_changes_from(&doc).unwrap(); + assert_images(&restored, true, damaged); + for changed in [&doc, &restored] { + for object_streams in [false, true] { + let opts = SaveOptions { object_streams, ..SaveOptions::default() }; + let incremental = write_incremental(changed, &opts).unwrap(); + if !damaged { + assert!(incremental.starts_with(&original)); + } + assert_images(&Document::open(Arc::new(incremental)).unwrap(), true, damaged); + let full = write_full(changed, &opts).unwrap(); + let reopened = Document::open(Arc::new(full)).unwrap(); + assert_images(&reopened, true, damaged); + if damaged { + let catalog = reopened.get(reopened.root().unwrap()); + let catalog = catalog.as_dict().unwrap(); + assert_eq!(catalog.reference(b"HeaderImage"), catalog.reference(b"RowImage"), "saved aliases must share one object"); + } + } + } + } +} diff --git a/docs/roadmap.md b/docs/roadmap.md index 8de636a500..25c0da7e00 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -63,6 +63,10 @@ > **Last reviewed:** 2026-10-17 · **Last updated:** 2026-10-17 · **Change:** minor (M8, gap 6 #2662: the redaction audit counts a cover over one word of a long line; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-13 · **Last updated:** 2026-10-13 · **Change:** minor (M10: dotted-leader contents entries export as complete paragraphs; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-22 · **Last updated:** 2026-10-22 · **Change:** minor (M6: calculation scripts read their notation only from a comment, and a regular expression after a condition as one; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) + +> **Last reviewed:** 2026-10-20 · **Last updated:** 2026-10-20 · **Change:** patch (M1, #1143: repaired aliases survive preflight, snapshots and normalized saves; retained materialized revisions report generation disagreements; estimates unchanged) + +> **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M1, #1143: explicit live xref/header generation repair policy and bounded diagnostics; estimates unchanged) > **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** minor (M5, PR #1971: bounded nested FDF export excludes source page/page-tree dictionaries; refreshed against upstream 4db6ac4a; estimates unchanged) > **Last reviewed:** 2026-10-18 · **Last updated:** 2026-10-18 · **Change:** patch (M2 #3086: dotted TOC leaders no longer hide word spaces; estimates unchanged) > @@ -78,6 +82,7 @@ > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M6: AcroForm XML text; estimates unchanged) > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M6: preserve XML/XFDF attribute whitespace; refreshed upstream; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M1: raw-deflate fallback preserves decoding-limit errors; refreshed upstream; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) +> **Last reviewed:** 2026-10-15 · **Last updated:** 2026-10-15 · **Change:** minor (M1: refresh object-generation reference identity against current upstream; estimates unchanged) > **Last reviewed:** 2026-10-14 · **Last updated:** 2026-10-15 · **Change:** minor (M2: objects in one large object stream are read in linear time; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS) > **Last reviewed:** 2026-10-16; **Last updated:** 2026-10-16; **Change:** minor (M5 #2669, item 5: reply groups and page root counts per frame; estimates unchanged) > @@ -204,7 +209,7 @@ total matches the parity estimate. | M | Milestone | Plan size (h) | Done | Remaining (h, 2026-10-10) | Notes | |---|---|---|---|---|---| | M0 | Skeleton: workspace, xtask gates, CI | 15–30 | 98% | 1–2 | GitHub workflow, `deny.toml` (licence audit of every dependency), parity checklist (826 features, `xtask parity`) done. Missing: remaining crate stubs, testkit/oracle crates | -| M1 | COS: filters, crypt, parser, xref, writer | 120–200 | 80% | 20–40 | Done:
- filters and crypt: every standard-security revision R2–R6 (RC4, AES-128/256), SASLprep, permissions, creating encryption;
- cos: parse and repair (including invalid xref object-number ranges), decoded-stream limits including unfiltered data, decrypt on load, re-encrypt on save, incremental and full writing.
Corpus: open/edit/save passes on 958 files, and all 7 password-protected files open.
Full saves now pack objects into compressed object streams. An unchanged incremental save of a classic table writes a zero-length cross-reference subsection instead of a section with no subsections (#1339). A full rewrite keeps unknown trailer entries, including objects reachable only from them (#1338). Fuzzing runs nightly (`xtask fuzz`). Raw-deflate fallback preserves decoding-limit errors, including through tolerant COS stream decoding. Done since: #2658: macOS atomic saves retain destination extended attributes, including Finder tags, and reject partial resource-fork reads after size changes. Metadata failures retain the original PDF and unsaved edit. Missing: ≥ 250 tests, own image codecs | +| M1 | COS: filters, crypt, parser, xref, writer | 120–200 | 80% | 20–40 | Done:
- filters and crypt: every standard-security revision R2–R6 (RC4, AES-128/256), SASLprep, permissions, creating encryption;
- cos: parse and repair (including invalid xref object-number ranges), decoded-stream limits including unfiltered data, decrypt on load, re-encrypt on save, incremental and full writing.
Corpus: open/edit/save passes on 958 files, and all 7 password-protected files open.
Full saves now pack objects into compressed object streams. An unchanged incremental save of a classic table writes a zero-length cross-reference subsection instead of a section with no subsections (#1339). A full rewrite keeps unknown trailer entries, including objects reachable only from them (#1338). Fuzzing runs nightly (`xtask fuzz`). Raw-deflate fallback preserves decoding-limit errors, including through tolerant COS stream decoding. Refreshed against upstream `99941daa` (2026-10-15). COS reference lookups and full saves distinguish complete object references, so stale generations cannot expose replacement content. Snapshot restoration preserves edited nonzero-generation objects through incremental save/reopen. Synthetic regressions cover caches, overlays, snapshots, stream lengths and classic/packed saves. For damaged live rows, either the xref generation or a same-number header at its indexed offset may identify the object; neither-match and free slots remain null. Header probes read at most 128 bytes per candidate offset, and opening records at most 256 object disagreements plus a summary without expanding compact xref rows. Synthetic dictionary-edit and independent save/reopen controls cover this repair policy. Repair-aware preflight edits and restored snapshots retain both accepted aliases, with full rewrite for repaired edits; classic/packed output coalesces them to one object identity. Materialized diagnostics inspect retained primary/hybrid rows before folding, with the same cap. Done since: #2658: macOS atomic saves retain destination extended attributes, including Finder tags, and reject partial resource-fork reads after size changes. Metadata failures retain the original PDF and unsaved edit. Missing: ≥ 250 tests, own image codecs | | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. Page marks, added content and print output retain small page-unit transform coefficients when serialized. Forms without Resources use page Properties for ActualText and layer visibility; own dictionaries stop fallback. Oversized missing inline-image samples, including header and terminator overhead, no longer pin the content window (#3107 / PR #3438); normal large streamed images retain deterministic sample boundaries. Radial luminosity fades have synthetic coverage for the existing #2132 mask guard, including per-pixel references, antialiased clips, previews, scales and tiles; no additional production fix was needed. Dotted TOC leaders no longer inflate letter tracking or hide word spaces (#3086); phrase search and selection are covered. Short tracked ellipses retain their gap samples and remain searchable. Missing spacing glyphs in non-embedded Japanese CID substitutes render blank without changing PDF advances or Unicode (#3601). The desktop viewer can use installed CJK fonts, including Windows GBK Song/Hei/Kai/FangSong and subset-prefixed Founder names; Combine thumbnails and accessibility figure previews use the same display-only fallback, without changing export or print font policy (#234, continuation of #1567, review #4172). Print previews exclude installed faces and isolate their font cache; GBK regressions cover view/print switching for inline and pooled whole-page/tile rendering. Installed-font helpers and their regression contracts have function documentation. GBK regression rasters match explicit-Unicode references with independently selected installed faces; wrong-glyph and wrong-family negative controls fail. Integration with main fa5f9f8c retains viewer fallback across the split engine session modules, renderer tests and Combine thumbnail workers. | | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings. Packed TIFF-predictor RGB rows retain later rows in decoding and rendering. The renderer keeps its one-point crop-size floor in physical points after UserUnit scaling, with regression coverage for crop origins, rotations and f32 reciprocal rounding. Existing unit policy and nearly-zero fallback are retained. | | M2 | Model, render, text | 200–350 | 15% | 130–230 | hayro bootstrap renderer (vendored patches); inspector predictor scratch is bounded by available input, and JPX sample rescaling handles 1–32 bits safely. Text extraction reaches word-F1 0.98 against pdftotext. Short same-line redraws split before placement rather than interleaving. Missing: model crate, fonts, DisplayList, renderer independent of hayro. Inspector page-label traversal and output are resource-bounded, with physical-number fallback warnings | @@ -352,6 +357,8 @@ Estimates unchanged. | 2026-10-20 | minor | M10, #3532 partial: vector printing retains existing appearances with absent Subtype; estimates unchanged. | | 2026-10-19 | minor | M4 #3309: named-destination pruning keeps large documents usable; 60,000-name and deep-tree regressions verify targets through undo/redo and both save modes. Estimates unchanged. | | 2026-10-19 | minor | M6: Text field table (rows and columns of text fields as one edit, `form_add_field_table`); estimates unchanged | +| 2026-10-20 | patch | M1, #1143 review: preserve accepted aliases through preflight/snapshots, normalize them to one output identity with full-save fallback for repaired edits, and report retained materialized/hybrid generation disagreements before folding. Synthetic red/green and four-save controls; estimates unchanged. | +| 2026-10-18 | minor | M1, #1143: a live reference may match the xref generation or a same-number header at its indexed offset; neither-match and free slots remain null. Bounded repair diagnostics and synthetic cache, free-slot, edit and classic/packed save regressions; estimates unchanged. | | 2026-10-18 | minor | M5, PR #1971: nested source `/Page` and `/Pages` dictionaries export as null without traversing page content or the page tree. Synthetic saved/reopened three-page regression retains unrelated metadata and aliases. FDF import semantics, copied-graph limits and estimates unchanged. | | 2026-10-15 | minor | M5: budget retained FDF entries before cloning; discovery keeps lazy array owners and annotation positions, with alias/cache-eviction stress fixtures. Limits cover the copied graph; source parsing/page discovery retain existing COS/parser limits, with no total source-memory bound. FDF import semantics and estimates unchanged. | | 2026-10-15 | minor | M7: Edit text caret height follows its row, with a kittest regression; estimates unchanged | @@ -364,6 +371,7 @@ Estimates unchanged. | 2026-10-14 | minor | M1: raw-deflate fallback preserves decoding-limit errors in strict and tolerant decoding; synthetic filter and COS stream regressions. Estimates unchanged. | | 2026-10-16 | minor | M5 #2669, item 5: the Comments panel groups replies and counts all page roots once per frame. Estimates unchanged. | | 2026-10-15 | minor | M9 #1895: generic appearance rewriting deferred; separate validation overlays preserve original artwork, exports and printing. Baked captions remain unchanged (#79 open). Estimates unchanged. | +| 2026-10-15 | minor | M1: refresh object-generation reference identity, retaining current upstream progress and regression tests. Estimates unchanged. | | 2026-10-16 | minor | M10 #1201 partial: lossless image export applies component Decode pairs and Indexed mapping. Estimates unchanged. | | 2026-10-15 | minor | M6: CSV export and spreadsheet merging quote tabs, and import distinguishes quoted tabs from TSV delimiters without changing field identities. Estimates unchanged. | | 2026-10-13 | minor | M4: rotation queries and edits resolve indirect page-tree values; inherited, negative and large rotations verified through both saves. Estimates unchanged. | diff --git a/parity/acrobat-features.toml b/parity/acrobat-features.toml index 223fc66850..af283bed6e 100644 --- a/parity/acrobat-features.toml +++ b/parity/acrobat-features.toml @@ -70,8 +70,8 @@ milestone = "M1" status = "shipped" commands = [] tools = ["doc_info"] -evidence = ["crates/ui-egui/tests/it/viewer.rs::damaged_files_say_they_were_repaired"] -notes = "a notice says the file was repaired; Document Properties ▸ Advanced lists the repairs; doc_info reports them" +evidence = ["crates/ui-egui/tests/it/viewer.rs::damaged_files_say_they_were_repaired", "crates/cos/tests/it/generations.rs::contradictory_xref_generation_recovers_the_matching_header_and_records_repair", "crates/cos/tests/it/generations.rs::generation_repair_diagnostics_are_bounded_without_disabling_recovery", "crates/cos/tests/it/generations.rs::retained_generation_disagreements_are_reported_in_materialized_mode_too", "crates/cos/tests/it/generations.rs::hybrid_generation_diagnostics_observe_rows_before_the_table_hides_them", "crates/cos/tests/it/generations.rs::discarded_xref_repair_notes_do_not_survive_object_header_reconstruction"] +notes = "a notice says the file was repaired; Document Properties ▸ Advanced lists the repairs; doc_info reports them. COS records bounded live xref/header generation disagreements on open, including retained materialized and hybrid rows before folding; this follow-up validates the COS policy, without a fresh GUI claim." [[feature]] id = "core.xref-streams" @@ -82,8 +82,8 @@ milestone = "M1" status = "shipped" commands = [] tools = [] -evidence = ["crates/cos/tests/it/writer.rs::incremental_saves_stack_on_an_xref_stream_file", "crates/cos/tests/it/writer.rs::page_tree_loops_in_object_streams_do_not_crash_readers"] -notes = "" +evidence = ["crates/cos/tests/it/generations.rs::indirect_stream_lengths_require_the_referenced_generation", "crates/cos/tests/it/generations.rs::compressed_objects_only_accept_generation_zero", "crates/cos/tests/it/writer.rs::incremental_saves_stack_on_an_xref_stream_file", "crates/cos/tests/it/writer.rs::page_tree_loops_in_object_streams_do_not_crash_readers"] +notes = "Compressed objects accept generation zero only; indirect stream lengths are resolved by their complete reference. Synthetic regression coverage." [[feature]] id = "core.hybrid-xref" @@ -298,8 +298,8 @@ milestone = "M1" status = "shipped" commands = ["file.save_as"] tools = ["doc_save"] -evidence = ["crates/organize/src/tests.rs::full_save_drops_unreachable_objects", "crates/engine/src/tests.rs::full_save_is_readable_and_smaller_after_deletes"] -notes = "" +evidence = ["crates/cos/tests/it/generations.rs::full_save_does_not_turn_stale_references_into_live_ones", "crates/cos/tests/it/generations.rs::repaired_page_tree_aliases_keep_one_identity_in_full_saves", "crates/organize/src/tests.rs::full_save_drops_unreachable_objects", "crates/engine/src/tests.rs::full_save_is_readable_and_smaller_after_deletes"] +notes = "Full rewrites coalesce accepted repair aliases to one output object, preserving page-tree parent identity. They preserve reference-generation identity: stale references become null while live replacement objects remain reachable. Synthetic classic and object-stream save tests; this does not establish general rendering fidelity." [[feature]] id = "core.object-stream-compression" @@ -334,8 +334,8 @@ milestone = "M1" status = "shipped" commands = [] tools = [] -evidence = ["crates/cos/src/document/tests.rs::reads_classic_revision_and_edits_are_copy_on_write"] -notes = "" +evidence = ["crates/cos/tests/it/generations.rs::free_xref_slots_never_revive_retained_object_headers", "crates/cos/tests/it/generations.rs::restoring_equal_values_still_restores_the_accepted_generation_aliases", "crates/cos/tests/it/generations.rs::restoring_explicit_sets_removes_old_generation_aliases", "crates/preflight/tests/generation_repairs.rs::preflight_preserves_repaired_image_references_in_snapshots_and_saves", "crates/cos/tests/it/generations.rs::repaired_header_references_survive_full_saves_and_dictionary_edits", "crates/cos/tests/it/generations.rs::incremental_generation_identity_is_visible_to_an_independent_reader", "crates/cos/tests/it/generations.rs::restoring_snapshot_preserves_changes_to_nonzero_generation_objects", "crates/cos/tests/it/generations.rs::overlay_generation_changes_do_not_retarget_old_references_or_clones", "crates/cos/tests/it/generations.rs::references_require_the_current_generation_even_with_a_warm_cache", "crates/cos/src/document/tests.rs::reads_classic_revision_and_edits_are_copy_on_write"] +notes = "Generation-aware references stay distinct across warm caches, edit overlays and snapshots, including restoration and incremental save/reopen of edited nonzero-generation objects. A damaged live xref row may additionally resolve the matching same-number object header; unrelated generations and free slots remain null. Repair-aware edits preserve both aliases and require full rewrite; explicit replacement keeps only its supplied generation. Synthetic preflight/snapshot and classic/packed full/incremental controls include an independent reader. Broader real-file fidelity remains tracked separately." [[feature]] id = "core.write-validity"