pdfcraft

6,461 Commits 192 Branches 25 Tags

2 Commits

Include renames

Author	SHA1	Message	Date
Michael Higgins	f51ce3cfcc	fix(render): bound the objects lopdf's fallback parses from object streams and arrays (#5083 ) ## What this changes lopdf's fallback loader (used by `inspect` when the lazy reader fails, and always by `attachment_data`) parsed every object stream eagerly, capping only decoded bytes. A stream of `0 0 0 …` inside the 256 MiB limit built ~120M `Object`s (10+ GB) and aborted. This bounds what it builds: - Per top-level object: at most 4,194,304 objects (`parser::MAX_OBJECT_NODES`, the same as cos's `MAX_NODES`), counting every array element and dictionary value at any depth. One budget is threaded through `array` / `_dictionary` / `_direct_object`. An object is charged once it is built, so failed `alt` branches cost nothing, and an exhausted budget is a nom `Failure`. - Per object stream: at most 1,000,000 index entries (cos's `/N` clamp), and the index list is no longer allocated from the whole block. - Per load: at most 8,388,608 objects across all object streams, in an atomic `NodeBudget` shared by rayon workers. A member reserves up to its own byte length (every object takes at least one byte) and refunds what it doesn't use, so parallel members can't hold the budget between them. Once the budget is spent, the remaining members are skipped with a warning. - `convert` in `crates/render/src/structure.rs`: the depth limit is now 128, one level per array or dictionary, matching cos's parser. Before, it was 100 with two levels per dictionary, so ~51 nested dictionaries alone forced the lopdf fallback. Logged as lopdf patch (7) in `vendor/README.md`. Fixes #5083 ## How you verified it `cargo fmt --check` on `pdfcraft-render` and `cargo clippy -p pdfcraft-render --all-targets -- -D warnings` (which compiles the vendored lopdf). Regression tests in `crates/render/src/inspect/tests.rs`, where lopdf patches are tested in-tree (the vendored lopdf's own tests need its missing `assets/example.pdf`): - `object_stream_members_over_the_node_cap_are_skipped` (croquete1): a member that is one array of 4.3M elements is skipped at the per-object cap; a small member is kept. - `object_stream_index_entries_past_the_clamp_are_ignored`: of 1,000,001 distinct members, the one past the 1,000,000-entry clamp is never built. - `object_streams_of_one_load_share_their_node_budget`, `object_streams_loaded_in_parallel_share_their_node_budget` and `a_malformed_object_stream_member_refunds_its_reservation` (Harrison-Booth): a small shared `NodeBudget` bounds the members of several streams, serially and across threads, and a malformed member refunds what it didn't build. `NodeBudget` and `ObjectStream::new_within` are now public for this. - [x] `cargo fmt --check` - [x] clippy with `-D warnings` on the crates I changed - [x] tests pass on the crates I changed (`cargo test -p pdfcraft-render --lib object_stream`: 9 passed, including the five new tests) ## Never-crash - [x] No new `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!` outside tests - [x] No indexing or slicing with input-derived positions; the budget uses `checked_sub` and a `min`-bounded reservation; index entries and objects built are capped; nesting stays bounded (128 in `convert`) - [x] No `unsafe` - [x] The abort this fixes has synthetic regression tests (above) ## Provenance - [x] No assets, fonts or copied code; the limits follow `pdfcraft_cos`'s own caps ## Security - [x] No change to `--root`, MCP, the control channel or network access ## Parity and reach - Not applicable: an internal robustness fix to the fallback loader, with no user-facing feature or parity entry. The vendored-patch log (`vendor/README.md`, lopdf (7)) is updated. ## Anything a reviewer should look at first - `vendor/README.md` now lists the tests above for patch (7). - Ordinary (not compressed) indirect objects are capped at 4M objects each but aren't charged to the per-load budget; their total is bounded only by file size. - `get_compressed_object` still decodes and re-parses the whole stream per lookup (pre-existing); it now gets one object's budget. - Objects nested 101–128 deep now reach the Inspector through the lazy reader rather than being skipped by the fallback. The Inspector's own walkers cap depth at 32 or less. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * Bug Fixes * Added limits to PDF object parsing and object-stream processing. Loading now parses only selected members, while lookups parse only the requested member. Members that exceed available limits are skipped in a deterministic order. * Increased the supported nesting depth for dictionary and object conversion, allowing more deeply nested content to be processed. * Documented that conversion failures can trigger eager loading as a fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-11-11 07:48:14 +01:00
Brandon ThomasandClaude Opus 5.5	1449b71a19	Never crash: bound the allocations behind the nightly fuzz runner's out-of-memory deaths Fuzzed inputs drove one check-one process to 2-6 GB (the 16 GB CI runner running two of them was killed every night). Three causes, each with a test: - lopdf (bootstrap inspector) sized PNG predictor rows from /Columns before reading data: /Columns 4294967295 allocated two 4 GiB rows. Vendored lopdf 0.45.0 refuses rows longer than the data (and uses saturating row widths); its load-time stream decoding also gets a 256 MiB limit (it had none). - hayro sized a tiling pattern's cell pixmap from /XStep x scale, bounded only by u16: a step far beyond /BBox allocated 2 GB. The vendored patch keeps the cell within 3000 px a side, lowering the resolution instead. - cos decoded cross-reference streams up to the general 1 GiB limit; they now stop at what 8,388,607 objects' rows can hold (~200 MB at most), and the section is reconstructed (logged) instead. On the nightly's seed (41) the worst captured input fell from 6.2 GB to 6 MB, and a 300 s fuzz run had no process above 300 MB (before: many at 2-5 GB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-04 17:59:25 -04:00

Author

SHA1

Message

Date

Michael Higgins

f51ce3cfcc

fix(render): bound the objects lopdf's fallback parses from object streams and arrays (#5083 )

## What this changes

lopdf's fallback loader (used by `inspect` when the lazy reader fails, and always by `attachment_data`) parsed every object stream eagerly, capping only decoded bytes. A stream of `0 0 0 …` inside the 256 MiB limit built ~120M `Object`s (10+ GB) and aborted. This bounds what it builds:

- **Per top-level object:** at most 4,194,304 objects (`parser::MAX_OBJECT_NODES`, the same as cos's `MAX_NODES`), counting every array element and dictionary value at any depth. One budget is threaded through `array` / `_dictionary` / `_direct_object`. An object is charged once it is built, so failed `alt` branches cost nothing, and an exhausted budget is a nom `Failure`.
- **Per object stream:** at most 1,000,000 index entries (cos's `/N` clamp), and the index list is no longer allocated from the whole block.
- **Per load:** at most 8,388,608 objects across all object streams, in an atomic `NodeBudget` shared by rayon workers. A member reserves up to its own byte length (every object takes at least one byte) and refunds what it doesn't use, so parallel members can't hold the budget between them. Once the budget is spent, the remaining members are skipped with a warning.
- **`convert` in `crates/render/src/structure.rs`:** the depth limit is now 128, one level per array or dictionary, matching cos's parser. Before, it was 100 with two levels per dictionary, so ~51 nested dictionaries alone forced the lopdf fallback.

Logged as lopdf patch (7) in `vendor/README.md`.

Fixes #5083

## How you verified it

`cargo fmt --check` on `pdfcraft-render` and `cargo clippy -p pdfcraft-render --all-targets -- -D warnings` (which compiles the vendored lopdf). Regression tests in `crates/render/src/inspect/tests.rs`, where lopdf patches are tested in-tree (the vendored lopdf's own tests need its missing `assets/example.pdf`):

- `object_stream_members_over_the_node_cap_are_skipped` (croquete1): a member that is one array of 4.3M elements is skipped at the per-object cap; a small member is kept.
- `object_stream_index_entries_past_the_clamp_are_ignored`: of 1,000,001 distinct members, the one past the 1,000,000-entry clamp is never built.
- `object_streams_of_one_load_share_their_node_budget`, `object_streams_loaded_in_parallel_share_their_node_budget` and `a_malformed_object_stream_member_refunds_its_reservation` (Harrison-Booth): a small shared `NodeBudget` bounds the members of several streams, serially and across threads, and a malformed member refunds what it didn't build. `NodeBudget` and `ObjectStream::new_within` are now public for this.

- [x] `cargo fmt --check`
- [x] clippy with `-D warnings` on the crates I changed
- [x] tests pass on the crates I changed (`cargo test -p pdfcraft-render --lib object_stream`: 9 passed, including the five new tests)

## Never-crash

- [x] No new `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!` outside tests
- [x] No indexing or slicing with input-derived positions; the budget uses `checked_sub` and a `min`-bounded reservation; index entries and objects built are capped; nesting stays bounded (128 in `convert`)
- [x] No `unsafe`
- [x] The abort this fixes has synthetic regression tests (above)

## Provenance

- [x] No assets, fonts or copied code; the limits follow `pdfcraft_cos`'s own caps

## Security

- [x] No change to `--root`, MCP, the control channel or network access

## Parity and reach

- Not applicable: an internal robustness fix to the fallback loader, with no user-facing feature or parity entry. The vendored-patch log (`vendor/README.md`, lopdf (7)) is updated.

## Anything a reviewer should look at first

- `vendor/README.md` now lists the tests above for patch (7).
- Ordinary (not compressed) indirect objects are capped at 4M objects each but aren't charged to the per-load budget; their total is bounded only by file size.
- `get_compressed_object` still decodes and re-parses the whole stream per lookup (pre-existing); it now gets one object's budget.
- Objects nested 101–128 deep now reach the Inspector through the lazy reader rather than being skipped by the fallback. The Inspector's own walkers cap depth at 32 or less.


<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
  * Added limits to PDF object parsing and object-stream processing. Loading now parses only selected members, while lookups parse only the requested member. Members that exceed available limits are skipped in a deterministic order.
  * Increased the supported nesting depth for dictionary and object conversion, allowing more deeply nested content to be processed.
  * Documented that conversion failures can trigger eager loading as a fallback.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-11-11 07:48:14 +01:00

Brandon ThomasandClaude Opus 5.5

1449b71a19

Never crash: bound the allocations behind the nightly fuzz runner's out-of-memory deaths

Fuzzed inputs drove one check-one process to 2-6 GB (the 16 GB CI runner
running two of them was killed every night). Three causes, each with a test:
- lopdf (bootstrap inspector) sized PNG predictor rows from /Columns before
  reading data: /Columns 4294967295 allocated two 4 GiB rows. Vendored lopdf
  0.45.0 refuses rows longer than the data (and uses saturating row widths);
  its load-time stream decoding also gets a 256 MiB limit (it had none).
- hayro sized a tiling pattern's cell pixmap from /XStep x scale, bounded only
  by u16: a step far beyond /BBox allocated 2 GB. The vendored patch keeps
  the cell within 3000 px a side, lowering the resolution instead.
- cos decoded cross-reference streams up to the general 1 GiB limit; they now
  stop at what 8,388,607 objects' rows can hold (~200 MB at most), and the
  section is reconstructed (logged) instead.

On the nightly's seed (41) the worst captured input fell from 6.2 GB to 6 MB,
and a 300 s fuzz run had no process above 300 MB (before: many at 2-5 GB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-04 17:59:25 -04:00