mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-17 16:16:34 +00:00
## What this changes lopdf's fallback loader (used by `inspect` when the lazy reader fails, and always by `attachment_data`) parsed every object stream eagerly, capping only decoded bytes. A stream of `0 0 0 …` inside the 256 MiB limit built ~120M `Object`s (10+ GB) and aborted. This bounds what it builds: - **Per top-level object:** at most 4,194,304 objects (`parser::MAX_OBJECT_NODES`, the same as cos's `MAX_NODES`), counting every array element and dictionary value at any depth. One budget is threaded through `array` / `_dictionary` / `_direct_object`. An object is charged once it is built, so failed `alt` branches cost nothing, and an exhausted budget is a nom `Failure`. - **Per object stream:** at most 1,000,000 index entries (cos's `/N` clamp), and the index list is no longer allocated from the whole block. - **Per load:** at most 8,388,608 objects across all object streams, in an atomic `NodeBudget` shared by rayon workers. A member reserves up to its own byte length (every object takes at least one byte) and refunds what it doesn't use, so parallel members can't hold the budget between them. Once the budget is spent, the remaining members are skipped with a warning. - **`convert` in `crates/render/src/structure.rs`:** the depth limit is now 128, one level per array or dictionary, matching cos's parser. Before, it was 100 with two levels per dictionary, so ~51 nested dictionaries alone forced the lopdf fallback. Logged as lopdf patch (7) in `vendor/README.md`. Fixes #5083 ## How you verified it `cargo fmt --check` on `pdfcraft-render` and `cargo clippy -p pdfcraft-render --all-targets -- -D warnings` (which compiles the vendored lopdf). Regression tests in `crates/render/src/inspect/tests.rs`, where lopdf patches are tested in-tree (the vendored lopdf's own tests need its missing `assets/example.pdf`): - `object_stream_members_over_the_node_cap_are_skipped` (croquete1): a member that is one array of 4.3M elements is skipped at the per-object cap; a small member is kept. - `object_stream_index_entries_past_the_clamp_are_ignored`: of 1,000,001 distinct members, the one past the 1,000,000-entry clamp is never built. - `object_streams_of_one_load_share_their_node_budget`, `object_streams_loaded_in_parallel_share_their_node_budget` and `a_malformed_object_stream_member_refunds_its_reservation` (Harrison-Booth): a small shared `NodeBudget` bounds the members of several streams, serially and across threads, and a malformed member refunds what it didn't build. `NodeBudget` and `ObjectStream::new_within` are now public for this. - [x] `cargo fmt --check` - [x] clippy with `-D warnings` on the crates I changed - [x] tests pass on the crates I changed (`cargo test -p pdfcraft-render --lib object_stream`: 9 passed, including the five new tests) ## Never-crash - [x] No new `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!` outside tests - [x] No indexing or slicing with input-derived positions; the budget uses `checked_sub` and a `min`-bounded reservation; index entries and objects built are capped; nesting stays bounded (128 in `convert`) - [x] No `unsafe` - [x] The abort this fixes has synthetic regression tests (above) ## Provenance - [x] No assets, fonts or copied code; the limits follow `pdfcraft_cos`'s own caps ## Security - [x] No change to `--root`, MCP, the control channel or network access ## Parity and reach - Not applicable: an internal robustness fix to the fallback loader, with no user-facing feature or parity entry. The vendored-patch log (`vendor/README.md`, lopdf (7)) is updated. ## Anything a reviewer should look at first - `vendor/README.md` now lists the tests above for patch (7). - Ordinary (not compressed) indirect objects are capped at 4M objects each but aren't charged to the per-load budget; their total is bounded only by file size. - `get_compressed_object` still decodes and re-parses the whole stream per lookup (pre-existing); it now gets one object's budget. - Objects nested 101–128 deep now reach the Inspector through the lazy reader rather than being skipped by the fallback. The Inspector's own walkers cap depth at 32 or less. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Added limits to PDF object parsing and object-stream processing. Loading now parses only selected members, while lookups parse only the requested member. Members that exceed available limits are skipped in a deterministic order. * Increased the supported nesting depth for dictionary and object conversion, allowing more deeply nested content to be processed. * Documented that conversion failures can trigger eager loading as a fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
349 lines
15 KiB
Rust
349 lines
15 KiB
Rust
//! Read-only structure inspection backed by the lazy COS reader. Keep the existing
|
|
//! inspector's lopdf object vocabulary while loading only the objects its queries use.
|
|
|
|
use std::collections::{BTreeMap, HashSet};
|
|
use std::sync::{
|
|
Arc, OnceLock,
|
|
atomic::{AtomicBool, Ordering},
|
|
};
|
|
|
|
use lopdf::{Dictionary, Error, Object, ObjectId, Result};
|
|
use pdfcraft_cos::{Document, Object as CosObject};
|
|
|
|
pub(crate) trait Structure {
|
|
fn trailer(&self) -> &Dictionary;
|
|
fn get_object(&self, id: ObjectId) -> Result<&Object>;
|
|
fn encrypted(&self) -> bool;
|
|
|
|
fn dereference<'a>(&'a self, mut object: &'a Object) -> Result<(Option<ObjectId>, &'a Object)> {
|
|
let mut id = None;
|
|
for _ in 0..128 {
|
|
match object {
|
|
Object::Reference(reference) => {
|
|
id = Some(*reference);
|
|
object = self.get_object(*reference)?;
|
|
}
|
|
_ => return Ok((id, object)),
|
|
}
|
|
}
|
|
Err(Error::ReferenceLimit)
|
|
}
|
|
|
|
fn get_dictionary(&self, id: ObjectId) -> Result<&Dictionary> {
|
|
self.get_object(id).and_then(|o| self.dereference(o)).and_then(|(_, o)| o.as_dict())
|
|
}
|
|
|
|
fn catalog(&self) -> Result<&Dictionary> {
|
|
self.trailer().get(b"Root").and_then(Object::as_reference).and_then(|id| self.get_dictionary(id))
|
|
}
|
|
|
|
fn page_ids(&self) -> Vec<ObjectId> {
|
|
let mut pages = Vec::new();
|
|
let Ok(root) = self.catalog().and_then(|c| c.get(b"Pages")).and_then(Object::as_reference) else { return pages };
|
|
let mut pending = vec![(root, 0usize)];
|
|
let mut seen = HashSet::new();
|
|
while let Some((id, depth)) = pending.pop() {
|
|
if depth > 256 || !seen.insert(id) {
|
|
continue;
|
|
}
|
|
let Ok(dict) = self.get_dictionary(id) else { continue };
|
|
match dict.get_type() {
|
|
Ok(b"Page") => {
|
|
// Keep the effective u32 page-count bound without trusting /Count.
|
|
if u32::try_from(pages.len()).ok().and_then(|n| n.checked_add(1)).is_none() {
|
|
return pages;
|
|
}
|
|
pages.push(id);
|
|
}
|
|
Ok(b"Pages") => {
|
|
if let Ok(kids) = dict.get(b"Kids").and_then(|o| self.dereference(o)).and_then(|(_, o)| o.as_array()) {
|
|
pending.extend(kids.iter().rev().filter_map(|k| k.as_reference().ok()).map(|id| (id, depth + 1)));
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
pages
|
|
}
|
|
|
|
// Match lopdf's font-resource precedence: a page's direct resources, then its
|
|
// indirect resources and ancestors. Stop cycles and excessive inheritance depth.
|
|
fn get_page_fonts(&self, page: ObjectId) -> Result<BTreeMap<Vec<u8>, &Dictionary>> {
|
|
let mut resources = Vec::new();
|
|
let mut current = Some(page);
|
|
let mut seen = HashSet::new();
|
|
while let Some(id) = current {
|
|
if !seen.insert(id) {
|
|
return Err(Error::ReferenceCycle(id));
|
|
}
|
|
if seen.len() > 100 {
|
|
return Err(Error::RecursionLimit);
|
|
}
|
|
let dict = self.get_dictionary(id)?;
|
|
if let Ok(resource) = dict.get(b"Resources") {
|
|
match resource {
|
|
Object::Dictionary(dict) if id == page => resources.push(dict),
|
|
Object::Reference(id) => {
|
|
if let Ok(dict) = self.get_dictionary(*id) {
|
|
resources.push(dict);
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
current = dict.get(b"Parent").and_then(Object::as_reference).ok();
|
|
}
|
|
let mut fonts = BTreeMap::new();
|
|
for resources in resources {
|
|
if let Ok(dict) = resources.get(b"Font").and_then(|o| self.dereference(o)).and_then(|(_, o)| o.as_dict()) {
|
|
for (name, font) in dict.iter() {
|
|
if let Ok(font) = self.dereference(font).and_then(|(_, o)| o.as_dict()) {
|
|
fonts.entry(name.clone()).or_insert(font);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Ok(fonts)
|
|
}
|
|
}
|
|
|
|
impl Structure for lopdf::Document {
|
|
fn trailer(&self) -> &Dictionary {
|
|
&self.trailer
|
|
}
|
|
fn get_object(&self, id: ObjectId) -> Result<&Object> {
|
|
self.get_object(id)
|
|
}
|
|
// lopdf's `get_object` already follows an indirect object that holds a reference, so the
|
|
// trait's walk would report the first object of a chain. lopdf's own walk reports the last.
|
|
fn dereference<'a>(&'a self, object: &'a Object) -> Result<(Option<ObjectId>, &'a Object)> {
|
|
lopdf::Document::dereference(self, object)
|
|
}
|
|
fn encrypted(&self) -> bool {
|
|
self.is_encrypted() || self.was_encrypted() || self.trailer.has(b"Encrypt")
|
|
}
|
|
// `page_ids` is not overridden: lopdf's own page-tree walk trusts /Count and pre-allocates
|
|
// by it, so a hostile /Count (a fuzz finding: 4294967295) aborted the process with a 48 GiB
|
|
// allocation. The trait's bounded default walk counts the real pages instead.
|
|
fn get_page_fonts(&self, page: ObjectId) -> Result<BTreeMap<Vec<u8>, &Dictionary>> {
|
|
self.get_page_fonts(page)
|
|
}
|
|
}
|
|
|
|
type Slot = (u32, OnceLock<Option<Box<Object>>>);
|
|
|
|
pub(crate) struct LazyStructure {
|
|
source: Document,
|
|
trailer: Dictionary,
|
|
// Sorted slots keep returned references stable, without allocating by the largest
|
|
// input object number (which can be sparse or hostile). Only queried objects fill slots.
|
|
objects: Vec<Slot>,
|
|
failed: AtomicBool,
|
|
}
|
|
|
|
impl LazyStructure {
|
|
pub(crate) fn new(bytes: Arc<Vec<u8>>, password: Option<&str>) -> std::result::Result<Self, String> {
|
|
let source = Document::open_with_stream_limit(bytes, password, 256 << 20).map_err(|e| e.to_string())?;
|
|
// A file rebuilt because its last update has no `/Prev` is read here as rebuilt: the
|
|
// renderer rebuilds it too, and the fallback reader would follow the cut-off chain.
|
|
if !source.repair_log().is_empty() && !source.rebuilt_unlinked_update() {
|
|
return Err("the structure requires compatibility repair".into());
|
|
}
|
|
let trailer = convert_dict(source.trailer(), 0).map_err(|e| e.to_string())?;
|
|
let objects = source.object_numbers().into_iter().map(|n| (n, OnceLock::new())).collect();
|
|
Ok(Self { source, trailer, objects, failed: AtomicBool::new(false) })
|
|
}
|
|
|
|
pub(crate) fn failed(&self) -> bool {
|
|
self.failed.load(Ordering::Relaxed)
|
|
}
|
|
}
|
|
|
|
impl Structure for LazyStructure {
|
|
fn trailer(&self) -> &Dictionary {
|
|
&self.trailer
|
|
}
|
|
fn encrypted(&self) -> bool {
|
|
self.source.security().is_some() || self.trailer.has(b"Encrypt")
|
|
}
|
|
/// The object `id`, read from the lazy source and converted once, then kept; a failed
|
|
/// conversion marks the structure so the caller can fall back to the eager loader.
|
|
fn get_object(&self, id: ObjectId) -> Result<&Object> {
|
|
let missing = || Error::ObjectNotFound(id);
|
|
let slot = self.objects.binary_search_by_key(&id.0, |(n, _)| *n).ok().and_then(|i| self.objects.get(i)).ok_or_else(missing)?;
|
|
if self.source.generation(id.0) != id.1 {
|
|
return Err(missing());
|
|
}
|
|
slot.1
|
|
.get_or_init(|| {
|
|
let result = self.source.try_get(id.0).ok().and_then(|o| convert(&o, 0).ok()).map(Box::new);
|
|
if result.is_none() {
|
|
self.failed.store(true, Ordering::Relaxed);
|
|
}
|
|
result
|
|
})
|
|
.as_deref()
|
|
.ok_or_else(missing)
|
|
}
|
|
}
|
|
|
|
/// Nesting `convert` accepts: the same as `pdfcraft_cos`'s parser (each array or dictionary
|
|
/// one level), so every object cos parsed converts. A lower limit would turn deep but valid
|
|
/// nesting into a lazy-reader failure and force the eager lopdf fallback.
|
|
const CONVERT_DEPTH: usize = 128;
|
|
|
|
/// Convert a cos dictionary to lopdf's, without `/PieceInfo`. `depth` is the nesting level of
|
|
/// the dictionary itself; its values sit one level deeper.
|
|
fn convert_dict(dict: &pdfcraft_cos::Dict, depth: usize) -> Result<Dictionary> {
|
|
if depth > CONVERT_DEPTH {
|
|
return Err(Error::RecursionLimit);
|
|
}
|
|
dict.iter()
|
|
.filter(|(key, _)| key.as_slice() != PRIVATE_DATA)
|
|
.map(|(key, value)| Ok((key.clone(), convert(value, depth + 1)?)))
|
|
.collect::<Result<Vec<_>>>()
|
|
.map(|items| items.into_iter().collect())
|
|
}
|
|
|
|
/// `/PieceInfo`: an application's private data about a page, form or document (ISO 32000-1
|
|
/// §14.5), left out of what inspection keeps. Nothing here reads it, and every object read is
|
|
/// kept converted for the inspection's lifetime: InDesign writes kilobytes of it into every page,
|
|
/// and a 1,412-page book held over a gigabyte of it (#5873). The file and the editing document are
|
|
/// untouched; this is the read-only view inspection walks.
|
|
const PRIVATE_DATA: &[u8] = b"PieceInfo";
|
|
|
|
/// Convert a cos object to lopdf's, failing past [`CONVERT_DEPTH`] levels of arrays and
|
|
/// dictionaries.
|
|
fn convert(object: &CosObject, depth: usize) -> Result<Object> {
|
|
if depth > CONVERT_DEPTH {
|
|
return Err(Error::RecursionLimit);
|
|
}
|
|
Ok(match object {
|
|
CosObject::Null => Object::Null,
|
|
CosObject::Bool(value) => Object::Boolean(*value),
|
|
CosObject::Int(value) => Object::Integer(*value),
|
|
CosObject::Real(value) => Object::Real(*value as f32),
|
|
CosObject::Name(value) => Object::Name(value.clone()),
|
|
CosObject::String(value) => {
|
|
Object::String(value.bytes.clone(), if value.hex { lopdf::StringFormat::Hexadecimal } else { lopdf::StringFormat::Literal })
|
|
}
|
|
CosObject::Ref(value) => Object::Reference((value.num, value.generation)),
|
|
CosObject::Array(value) => Object::Array(value.iter().map(|o| convert(o, depth + 1)).collect::<Result<_>>()?),
|
|
CosObject::Dict(value) => Object::Dictionary(convert_dict(value, depth)?),
|
|
CosObject::Stream(value) => Object::Stream(lopdf::Stream::new(convert_dict(&value.dict, depth)?, value.raw.to_vec())),
|
|
})
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
/// A hostile page-tree /Count must not abort the process: lopdf's own `get_pages`
|
|
/// pre-allocates by it (fuzz finding: /Count 4294967295 -> a 48 GiB allocation),
|
|
/// the bounded default walk counts the real pages instead.
|
|
#[test]
|
|
fn a_hostile_page_count_does_not_abort_inspection() {
|
|
let pdf = b"%PDF-1.7
|
|
1 0 obj
|
|
<< /Type /Catalog /Pages 2 0 R >>
|
|
endobj
|
|
2 0 obj
|
|
<< /Type /Pages /Kids [3 0 R] /Count 4294967295 >>
|
|
endobj
|
|
3 0 obj
|
|
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] >>
|
|
endobj
|
|
trailer
|
|
<< /Size 4 /Root 1 0 R >>
|
|
%%EOF";
|
|
let doc = lopdf::Document::load_mem(&pdf[..]).unwrap();
|
|
let pages = Structure::page_ids(&doc);
|
|
assert_eq!(pages, [(3, 0)], "the walk counts the one real page, not /Count");
|
|
}
|
|
|
|
fn nested_sparse_page_tree(repeated_and_cyclic: bool) -> lopdf::Document {
|
|
let mut doc = lopdf::Document::with_version("1.7");
|
|
let mut catalog = Dictionary::new();
|
|
catalog.set("Type", Object::Name(b"Catalog".to_vec()));
|
|
catalog.set("Pages", Object::Reference((5, 0)));
|
|
doc.objects.insert((17, 0), Object::Dictionary(catalog));
|
|
|
|
let mut pages = Dictionary::new();
|
|
pages.set("Type", Object::Name(b"Pages".to_vec()));
|
|
pages.set("Kids", vec![Object::Reference((31, 0)), Object::Reference((88, 0))]);
|
|
pages.set("Count", Object::Integer(if repeated_and_cyclic { 5 } else { 3 }));
|
|
doc.objects.insert((5, 0), Object::Dictionary(pages));
|
|
|
|
let mut nested = Dictionary::new();
|
|
nested.set("Type", Object::Name(b"Pages".to_vec()));
|
|
nested.set("Parent", Object::Reference((5, 0)));
|
|
let kids = if repeated_and_cyclic {
|
|
vec![Object::Reference((12, 0)), Object::Reference((5, 0)), Object::Reference((31, 0)), Object::Reference((42, 0))]
|
|
} else {
|
|
vec![Object::Reference((12, 0)), Object::Reference((42, 0))]
|
|
};
|
|
nested.set("Kids", kids);
|
|
nested.set("Count", Object::Integer(if repeated_and_cyclic { 4 } else { 2 }));
|
|
doc.objects.insert((88, 0), Object::Dictionary(nested));
|
|
|
|
for (id, parent) in [(31, 5), (12, 88), (42, 88)] {
|
|
let mut page = Dictionary::new();
|
|
page.set("Type", Object::Name(b"Page".to_vec()));
|
|
page.set("Parent", Object::Reference((parent, 0)));
|
|
doc.objects.insert((id, 0), Object::Dictionary(page));
|
|
}
|
|
doc.max_id = 88;
|
|
doc.trailer.set("Root", Object::Reference((17, 0)));
|
|
doc
|
|
}
|
|
|
|
#[test]
|
|
fn page_walk_keeps_nested_sparse_order_and_first_visit_semantics() {
|
|
let doc = nested_sparse_page_tree(true);
|
|
assert_eq!(Structure::page_ids(&doc), [(31, 0), (12, 0), (42, 0)]);
|
|
}
|
|
|
|
#[test]
|
|
fn lazy_and_compatibility_walks_keep_nested_sparse_order() {
|
|
let mut source = nested_sparse_page_tree(false);
|
|
let mut bytes = Vec::new();
|
|
source.save_to(&mut bytes).unwrap();
|
|
let expected = [(31, 0), (12, 0), (42, 0)];
|
|
let eager = lopdf::Document::load_mem(&bytes).unwrap();
|
|
let lazy = LazyStructure::new(Arc::new(bytes), None).unwrap();
|
|
assert_eq!(Structure::page_ids(&eager), expected);
|
|
assert_eq!(Structure::page_ids(&lazy), expected);
|
|
}
|
|
|
|
#[test]
|
|
fn inspection_loads_only_requested_objects_in_a_compressed_document() {
|
|
let mut source = lopdf::Document::with_version("1.7");
|
|
let catalog = source.add_object(lopdf::dictionary! { "Type" => "Catalog", "Pages" => (2, 0) });
|
|
source.add_object(lopdf::dictionary! { "Type" => "Pages", "Kids" => vec![Object::Reference((3, 0))], "Count" => 1 });
|
|
source
|
|
.add_object(lopdf::dictionary! { "Type" => "Page", "Parent" => (2, 0), "MediaBox" => vec![0.into(), 0.into(), 100.into(), 100.into()] });
|
|
source.trailer.set("Root", catalog);
|
|
let unused: Vec<_> = (0..300).map(|i| Object::Reference(source.add_object(lopdf::dictionary! { "Unused" => i }))).collect();
|
|
// Keep these objects reachable when the full writer removes orphan objects,
|
|
// through a catalog extension that the panel inspector has no reason to read.
|
|
source.get_dictionary_mut(catalog).unwrap().set("Uninspected", unused);
|
|
let mut bytes = Vec::new();
|
|
source.save_to(&mut bytes).unwrap();
|
|
let cos = Document::open(Arc::new(bytes)).unwrap();
|
|
let compressed = pdfcraft_cos::write_full(&cos, &pdfcraft_cos::SaveOptions::default()).unwrap();
|
|
let eager = lopdf::Document::load_mem(&compressed).unwrap();
|
|
let expected = Structure::page_ids(&eager);
|
|
let view = LazyStructure::new(Arc::new(compressed), None).unwrap();
|
|
assert_eq!(view.page_ids().len(), 1);
|
|
assert_eq!(view.page_ids(), expected);
|
|
assert!(!view.failed());
|
|
assert!(
|
|
view.objects.iter().filter(|(_, slot)| slot.get().is_some()).count() <= 3,
|
|
"unused objects must not be materialized to inspect a page tree"
|
|
);
|
|
let unused = view.catalog().unwrap().get(b"Uninspected").unwrap().as_array().unwrap();
|
|
let last = unused.last().unwrap().as_reference().unwrap();
|
|
assert!(matches!(view.get_dictionary(last).unwrap().get(b"Unused"), Ok(Object::Integer(299))));
|
|
}
|
|
}
|