Files
pdfcraft/vendor/hayro-syntax/src/object/stream.rs
T
e88def3465 Never crash: decompression bombs can't exhaust the renderer's memory (#411)
* Never crash: decompression bombs can't exhaust the renderer's memory

The renderer's parser (vendored hayro-syntax) decoded Flate, LZW and
RunLength streams with no output limit. A 2 MB page whose content
stream inflates to 2 GB, or whose 8 x 8 image does, took `pdfcraft-cli
render` and `text` past 3 GB in under a second, and a few more MB of
file scale it further. On the desktop that is the first-page render of
an emailed PDF, and the allocation-failure abort takes every open
document with it. Our own layers already bound single streams (lopdf's
load-time decoding at 256 MiB, cos at 1 GiB), so `info` stays at a few
MB; the renderer path didn't. Random mutation almost never makes a
high-ratio stream, so `cargo xtask fuzz` doesn't find this; found by
hand while triaging the fuzz run.

Vendored hayro-syntax patch (6):
- `decode_limit` gives each filter an output limit, and Flate (both
  flate2 and the fallback inflater), LZW and RunLength never write past
  it (the buffer can still grow to under twice it):
  `MAX_DECODED_STREAM`, 256 MiB, lopdf's figure; or, for the last filter
  of an image whose size, bits per component and components are known,
  twice its pixel bytes (a PNG predictor tag byte per byte at most),
  capped at 1 GiB. Earlier filters in a chain produce compressed data
  and get the general limit.
- `Stream::decoded_within(max)` for callers that budget.
- A page's `/Contents` array shares one `MAX_DECODED_STREAM` (naming one
  bomb twenty times decoded 5 GB), reserving exactly what it appends.

Vendored hayro-interpret patch (18):
- The content a page holds at once (its contents, and the forms, tiling
  patterns and Type 3 glyph procedures being painted) stays within
  `MAX_PAGE_CONTENT`, 512 MiB, reset per page like the existing paint
  budget. Forms nest fifty deep and each level kept its content while
  the next ran: a form painting itself around a 256 MiB stream would
  have held over 12 GB. Each level gives its bytes back when it is
  dropped, so content painted again and again keeps drawing, and a
  stream without filters (borrowed from the file) costs nothing.
- Type 0 sampled functions read every sample into a 4-byte value and
  then a table entry: 256 MiB of 1-bit samples asked for over 8 GB. They
  now take at most 2^21 table points and 2^23 values (the largest real
  tables, DeviceN tint transforms such as 33^4 points x 3 outputs, are
  1.2 million and 3.6 million), counted with checked arithmetic, and
  read no further than `/Size` asks. `/Size 0` no longer underflows
  building the default `/Encode`.

Three rounds of independent review found the aggregate cases, buffer
doubling past the limit, a chain/predictor shape the first image limit
would have truncated, the Type 0 amplifier, and a first, cumulative
budget that would have dropped legitimately repeated content, and a
first Type 0 cap low enough to refuse real tint transforms; all fixed
here.

Measured in a release build (render; OS peak commit):
- content stream inflating to 2 GB: 389 MB, 0.3 s
- 8 x 8 image inflating to 2 GB: 4 MB, 0.2 s
- /Contents naming it twenty times: 516 MB, 0.3 s
- a form painting itself around it: 644 MB, 0.4 s

Synthetic regression tests in render (flate2 is a new dev-dependency
there, already in the workspace):
- `decompression_bombs_stop_at_the_stream_limit`: 288 MiB bombs through
  flate2, the fallback inflater (a stored block with a wrong length
  check, which flate2 refuses and the fallback accepts), LZW and
  RunLength each decode exactly 256 MiB, and at most 128 bytes as an
  8 x 8 image.
- `page_contents_and_nested_forms_share_one_budget`: three /Contents
  references decode one stream's worth; a self-painting form finishes.
- `content_painted_again_and_again_keeps_drawing`: a 2 MiB form painted
  300 times draws the 300th time.
- `sampled_functions_with_billions_of_samples_are_refused`.
Ten mutations each turn them red: no limit at all; no limit in flate2,
the fallback, LZW, RunLength or for images; an unshared /Contents limit;
no page budget; no refund on drop; no Type 0 sample cap. (Taking a form
out of the budget no longer compiles: its content is the budget guard.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* M2: normalize JPEG2000 samples without overflowing

Use bounded integer sample scaling for 1-32-bit JPX images and reject empty dimensions before chunking. Cover public decoding and continued page rendering with synthetic regressions.

Co-Authored-By: OpenAI Codex <codex@openai.com>

* Never crash: decompression cut at its limit is logged, not silent; vendor README renumbered after #575

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Streetblock <github@streetblock.de>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
2026-10-09 14:10:50 -04:00

425 lines
13 KiB
Rust

//! Streams.
use crate::crypto::DecryptionTarget;
use crate::filter::Filter;
use crate::object;
use crate::object::Dict;
use crate::object::Name;
use crate::object::dict::keys::{DECODE_PARMS, DP, F, FILTER, LENGTH, TYPE};
use crate::object::{Array, ObjectIdentifier};
use crate::object::{Object, ObjectLike, ObjectRefLike};
use crate::reader::Reader;
use crate::reader::{Readable, ReaderContext, ReaderExt, Skippable};
use crate::trivia::is_white_space_character;
use crate::util::{OptionLog, find_needle};
use alloc::borrow::Cow;
use alloc::vec::Vec;
use core::fmt::{Debug, Formatter};
use smallvec::SmallVec;
struct FiltersAndParams<'a> {
filters: SmallVec<[Filter; 2]>,
params: SmallVec<[Dict<'a>; 2]>,
}
/// A stream of arbitrary data.
#[derive(Clone)]
pub struct Stream<'a> {
dict: Dict<'a>,
data: &'a [u8],
}
impl PartialEq for Stream<'_> {
fn eq(&self, other: &Self) -> bool {
self.dict == other.dict && self.data == other.data
}
}
/// Additional parameters for decoding images.
#[derive(Clone, PartialEq, Default)]
pub struct ImageDecodeParams {
/// Whether the color space of the image is an indexed color space.
pub is_indexed: bool,
/// The bits per component of the image, if that information is available.
pub bpc: Option<u8>,
/// The components per channel of the image, if that information is available.
pub num_components: Option<u8>,
/// A target resolution for the image. Note that this is only a hint so that
/// in case it's possible, a version of the image will be extracted that
/// is as close as possible to the hinted dimension.
pub target_dimension: Option<(u32, u32)>,
/// The width of the image as indicated by the image dictionary.
pub width: u32,
/// The height of the image as indicated by the image dictionary.
pub height: u32,
}
impl<'a> Stream<'a> {
pub(crate) fn new(data: &'a [u8], dict: Dict<'a>) -> Self {
Self { dict, data }
}
fn filters_and_params(&self) -> FiltersAndParams<'a> {
let mut collected_filters = SmallVec::new();
let mut collected_params = SmallVec::new();
if let Some(filter) = self
.dict
.get::<Name<'_>>(F)
.or_else(|| self.dict.get::<Name<'_>>(FILTER))
.and_then(Filter::from_name)
{
let params = self
.dict
.get::<Dict<'_>>(DP)
.or_else(|| self.dict.get::<Dict<'_>>(DECODE_PARMS))
.unwrap_or_default();
collected_filters.push(filter);
collected_params.push(params);
} else if let Some(filters) = self
.dict
.get::<Array<'_>>(F)
.or_else(|| self.dict.get::<Array<'_>>(FILTER))
{
let filters = filters.iter::<Name<'_>>().map(Filter::from_name);
let mut params = self
.dict
.get::<Array<'_>>(DP)
.or_else(|| self.dict.get::<Array<'_>>(DECODE_PARMS))
.map(|a| a.iter::<Object<'_>>());
for filter in filters {
let params = params
.as_mut()
.and_then(|p| p.next())
.and_then(|p| p.into_dict())
.unwrap_or_default();
if let Some(filter) = filter {
collected_filters.push(filter);
collected_params.push(params);
}
}
}
FiltersAndParams {
filters: collected_filters,
params: collected_params,
}
}
/// Return the raw, decrypted data of the stream.
///
/// Stream filters will not be applied.
pub fn raw_data(&self) -> Cow<'a, [u8]> {
let ctx = self.dict.ctx();
if ctx.xref().needs_decryption(ctx)
&& self
.dict
.get::<object::String<'_>>(TYPE)
.map(|t| t.as_ref() != b"XRef")
.unwrap_or(true)
{
Cow::Owned(
ctx.xref()
.decrypt(self.obj_id(), self.data, DecryptionTarget::Stream)
// TODO: MAybe an error would be better?
.unwrap_or_default(),
)
} else {
Cow::Borrowed(self.data)
}
}
/// Return the raw, underlying dictionary of the stream.
pub fn dict(&self) -> &Dict<'a> {
&self.dict
}
/// Return the object identifier of the stream.
pub fn obj_id(&self) -> ObjectIdentifier {
self.dict.obj_id().unwrap_or_else(|| {
// In theory shouldn't ever happen, but could theoretically be
// triggered in a crafted PDF.
ObjectIdentifier::new(0, 0)
})
}
/// Return the filters that are applied to the stream.
pub fn filters(&self) -> SmallVec<[Filter; 2]> {
self.filters_and_params().filters
}
/// Return the decoded data of the stream.
///
/// Note that the result of this method will not be cached, so calling it multiple
/// times is expensive.
pub fn decoded(&self) -> Result<Cow<'a, [u8]>, DecodeFailure> {
self.decoded_image(&ImageDecodeParams::default())
.map(|r| r.data)
}
/// Return the decoded data of the stream, and return image metadata
/// if available.
pub fn decoded_image(
&self,
image_params: &ImageDecodeParams,
) -> Result<FilterResult<'a>, DecodeFailure> {
self.decode_within(image_params, usize::MAX)
}
/// PdfCraft patch: the decoded data of the stream, for callers that budget what a page
/// holds. Flate, LZW and RunLength stop at `max` bytes (and at
/// [`crate::filter::MAX_DECODED_STREAM`] in any case). A stream without filters is returned
/// as it is; ASCIIHex and ASCII85 barely expand, and the image codecs stop at their own pixel
/// limits.
pub fn decoded_within(&self, max: usize) -> Result<Cow<'a, [u8]>, DecodeFailure> {
self.decode_within(&ImageDecodeParams::default(), max)
.map(|r| r.data)
}
fn decode_within(
&self,
image_params: &ImageDecodeParams,
max: usize,
) -> Result<FilterResult<'a>, DecodeFailure> {
let data = self.raw_data();
let filters_and_params = self.filters_and_params();
let count = filters_and_params.filters.len();
let mut current: Option<FilterResult<'a>> = None;
for (i, (filter, params)) in filters_and_params
.filters
.iter()
.zip(filters_and_params.params.iter())
.enumerate()
{
// PdfCraft patch: see `filter::decode_limit`.
let limit = crate::filter::decode_limit(image_params, i + 1 == count).min(max);
let new = filter.apply(
current.as_ref().map(|c| c.data.as_ref()).unwrap_or(&data),
params,
image_params,
limit,
)?;
current = Some(new);
}
Ok(current.unwrap_or(FilterResult {
data,
image_data: None,
}))
}
}
impl Debug for Stream<'_> {
fn fmt(&self, f: &mut Formatter<'_>) -> core::fmt::Result {
write!(f, "Stream (len: {:?})", self.data.len())
}
}
impl Skippable for Stream<'_> {
fn skip(_: &mut Reader<'_>, _: bool) -> Option<()> {
// A stream can never appear in a dict/array, so it should never be skipped.
warn!("attempted to skip a stream object");
None
}
}
impl<'a> Readable<'a> for Stream<'a> {
fn read(r: &mut Reader<'a>, ctx: &ReaderContext<'a>) -> Option<Self> {
let dict = r.read_with_context::<Dict<'_>>(ctx)?;
if dict.contains_key(F) {
warn!("encountered stream referencing external file, which is unsupported");
return None;
}
let offset = r.offset();
parse_proper(r, &dict)
.or_else(|| {
warn!("failed to parse stream, trying to parse it manually");
r.jump(offset);
parse_fallback(r, &dict)
})
.error_none("was unable to manually parse the stream")
}
}
#[derive(Debug, Copy, Clone)]
/// A failure that can occur during decoding a data stream.
pub enum DecodeFailure {
/// An image stream failed to decode.
ImageDecode,
/// A data stream failed to decode.
StreamDecode,
/// A failure occurred while decrypting a file.
Decryption,
/// An unknown failure occurred.
Unknown,
}
/// An image color space.
#[derive(Debug, Copy, Clone)]
pub enum ImageColorSpace {
/// Grayscale color space.
Gray,
/// RGB color space.
Rgb,
/// CMYK color space.
Cmyk,
/// An unknown color space.
Unknown(u8),
}
/// Additional data that is extracted from some image streams.
pub struct ImageData {
/// An optional alpha channel of the image.
pub alpha: Option<Vec<u8>>,
/// The color space of the image.
pub color_space: Option<ImageColorSpace>,
/// The bits per component of the image.
pub bits_per_component: u8,
/// The width of the image.
pub width: u32,
/// The height of the image.
pub height: u32,
}
/// The result of applying a filter.
pub struct FilterResult<'a> {
/// The decoded data.
pub data: Cow<'a, [u8]>,
/// Additional data that is extracted from JPX image streams.
pub image_data: Option<ImageData>,
}
impl FilterResult<'_> {
pub(crate) fn from_data(data: Vec<u8>) -> Self {
Self {
data: Cow::Owned(data),
image_data: None,
}
}
}
fn parse_proper<'a>(r: &mut Reader<'a>, dict: &Dict<'a>) -> Option<Stream<'a>> {
let length = dict.get::<u32>(LENGTH)?;
r.skip_white_spaces_and_comments();
r.forward_tag(b"stream")?;
r.forward_tag(b"\n")
.or_else(|| r.forward_tag(b"\r\n"))
.or_else(|| r.forward_tag(b"\r"))?;
let data = r.read_bytes(length as usize)?;
r.skip_white_spaces();
r.forward_tag(b"endstream")?;
Some(Stream::new(data, dict.clone()))
}
fn parse_fallback<'a>(r: &mut Reader<'a>, dict: &Dict<'a>) -> Option<Stream<'a>> {
// PdfCraft patch: `stream` must be the next token. Scanning ahead to a later
// `stream` made a dictionary look like a stream — in particular an indirect
// `/AP /N` appearance-state dictionary (`<< /Yes 5 0 R /Off 6 0 R >>`, as Quartz
// writes checkboxes). The widget then had no form to draw, so the box disappeared.
// A wrong `/Length` still recovers, because that keyword sits directly after the dict.
r.skip_white_spaces_and_comments();
r.forward_tag(b"stream")?;
r.forward_tag(b"\n")
.or_else(|| r.forward_tag(b"\r\n"))
// Technically not allowed, but no reason to not try it.
.or_else(|| r.forward_tag(b"\r"))?;
let tail = r.tail()?;
let endstream_offset = find_needle(tail, b"endstream")?;
let data_end = trim_trailing_ascii_whitespace(&tail[..endstream_offset]);
let data = tail.get(..data_end)?;
r.read_bytes(endstream_offset)?;
r.skip_white_spaces();
r.forward_tag(b"endstream")?;
Some(Stream::new(data, dict.clone()))
}
fn trim_trailing_ascii_whitespace(data: &[u8]) -> usize {
let mut end = data.len();
while data
.get(end.wrapping_sub(1))
.copied()
.is_some_and(is_white_space_character)
{
end -= 1;
}
end
}
impl<'a> TryFrom<Object<'a>> for Stream<'a> {
type Error = ();
fn try_from(value: Object<'a>) -> Result<Self, Self::Error> {
match value {
Object::Stream(s) => Ok(s),
_ => Err(()),
}
}
}
impl<'a> ObjectLike<'a> for Stream<'a> {}
impl<'a> ObjectRefLike<'a> for Stream<'a> {
fn cast_ref<'b>(obj: &'b Object<'a>) -> Option<&'b Self> {
match obj {
Object::Stream(stream) => Some(stream),
_ => None,
}
}
}
#[cfg(test)]
mod tests {
use crate::object::Stream;
use crate::reader::Reader;
use crate::reader::{ReaderContext, ReaderExt};
#[test]
fn stream() {
let data = b"<< /Length 10 >> stream\nabcdefghij\nendstream";
let mut r = Reader::new(data);
let stream = r
.read_with_context::<Stream<'_>>(&ReaderContext::dummy())
.unwrap();
assert_eq!(stream.data, b"abcdefghij");
}
#[test]
fn stream_fallback() {
let data = b"<< /Length 999 >> stream\nabcdefghij\nendstream";
let mut r = Reader::new(data);
let stream = r
.read_with_context::<Stream<'_>>(&ReaderContext::dummy())
.unwrap();
assert_eq!(stream.data, b"abcdefghij");
}
/// A dictionary followed by some other object's stream is not itself a stream.
/// `parse_fallback` used to latch onto that later `stream` keyword.
#[test]
fn dictionary_is_not_a_later_objects_stream() {
let data = b"<< /Yes 5 0 R /Off 6 0 R >>\nendobj\n11 0 obj << /Length 5 >> stream\nhello\nendstream";
let mut r = Reader::new(data);
assert!(r.read_with_context::<Stream<'_>>(&ReaderContext::dummy()).is_none());
}
}