mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-11 21:32:39 +00:00
M2.6: say what was measured, not why, and test the other block zlib-rs refuses
The Claims review of the branch: the docs said miniz_oxide "builds a block's tables for every block", which a timing doesn't show. They now say what was timed: about 1.7 us for every fixed-Huffman block, empty or not. "The worst comes to about 7 ns a step" was per byte read; a stage's start, about 0.25 us under zlib-rs (1.6 us for eight over an empty stream, release), isn't priced, since every way to repeat one takes bytes of the file. The letterhead test's doc promised more than it checks. The same review found a second shape zlib-rs refuses where miniz_oxide read on: a dynamic block whose code has no end-of-block symbol. flate.rs's quirks and the README row say so, and a_dynamic_code_without_an_end_is_damage checks that nothing of it is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
This commit is contained in:
1 parent
f8788dff11
commit
05300ea5bc
5 files changed
+50
-8
No files matched your search
+1
-1
@@ -64,7 +64,7 @@ pdfcraft-automation = { path = "crates/automation", version = "0.17.0", default-
|
||||
hayro = "0.7.1"
|
||||
hayro-syntax = "0.7.2"
|
||||
lopdf = "0.45"
|
||||
# zlib-rs: miniz_oxide spends about 1.7 µs on every deflate block, empty ones included
|
||||
# zlib-rs: miniz_oxide took about 1.7 µs for every fixed-Huffman block, empty or not
|
||||
# (`pdfcraft_filters::FLATE_READ_WORK`). Other dependencies still build flate2's default
|
||||
# miniz_oxide backend; flate2 prefers zlib-rs when both are built, as the Windows build already did
|
||||
# through craft-update. `flate::tests::a_copy_from_before_the_start_is_damage` shows which runs.
|
||||
|
||||
@@ -594,7 +594,8 @@ mod image_tests {
|
||||
/// What Flate reads is priced for blocks that make nothing, and an image an encoder made pays
|
||||
/// little for it: a scan-like picture stored in under half its size costs no more than half
|
||||
/// again its bytes, samples and pixels. Redaction decodes a picture once for each page that draws it, so
|
||||
/// a letterhead on hundreds of pages has to fit as it did unpriced.
|
||||
/// a letterhead on hundreds of pages has to fit about as often as it did unpriced (here, at
|
||||
/// most half again the cost).
|
||||
#[test]
|
||||
fn an_image_an_encoder_made_pays_little_for_its_reading() {
|
||||
let doc = Document::new_empty();
|
||||
|
||||
@@ -44,7 +44,7 @@ encode_jbig2(data, columns, rows) -> Result<Vec<u8>, FilterError> // 1-bit row
|
||||
|
||||
| Filter | Decode | Encode |
|
||||
|---|---|---|
|
||||
| FlateDecode | zlib header checked, then inflated raw: a missing or wrong Adler-32 is tolerated; no valid header → raw deflate; trailing bytes ignored; preset dictionary → Corrupt; truncated, or a back-reference to before the start of the output → Corrupt (partial in tolerant mode) | zlib level 6 |
|
||||
| FlateDecode | zlib header checked, then inflated raw: a missing or wrong Adler-32 is tolerated; no valid header → raw deflate; trailing bytes ignored; preset dictionary → Corrupt; truncated, a back-reference to before the start of the output, or a dynamic block without an end-of-block code → Corrupt (partial in tolerant mode) | zlib level 6 |
|
||||
| LZWDecode | 9–12-bit MSB-first codes, clear (256), EOD (257), `EarlyChange` 0/1; missing EOD/initial clear tolerated; a full table stops growing | greedy, leading clear, clear-table when the table fills, EOD |
|
||||
| Predictors (Flate/LZW) | 2 = TIFF (1/2/4/8/16 bpc), 10–15 = PNG with per-row tags 0–4; partial final rows decoded; bad params or tags → Corrupt | 2, 10 None, 11 Sub, 12 Up, 13 Average, 14 Paeth, 15 per-row optimum (min sum of signed residuals) |
|
||||
| ASCIIHexDecode | white space ignored, odd digit padded with 0, `>` EOD (optional), other characters → Corrupt | upper case, newline every 64 digits, `>` |
|
||||
|
||||
@@ -8,7 +8,8 @@
|
||||
//!
|
||||
//! A back-reference to before the start of the output points at nothing, and is damage, as zlib
|
||||
//! has it: read strictly the stream is refused, and read tolerantly what came before it is kept.
|
||||
//! (miniz_oxide, flate2's backend here before zlib-rs, copied zeros and went on.)
|
||||
//! (miniz_oxide, flate2's backend here before zlib-rs, copied zeros and went on.) So is a dynamic
|
||||
//! block whose code has no end-of-block symbol, and nothing of that block is kept.
|
||||
|
||||
use std::io::Write;
|
||||
|
||||
@@ -235,6 +236,44 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// A dynamic block whose literal and length code has no end-of-block symbol can't end (RFC 1951
|
||||
/// §3.2.5): it is refused, and nothing of it is kept.
|
||||
#[test]
|
||||
fn a_dynamic_code_without_an_end_is_damage() {
|
||||
let mut bits = Bits::default();
|
||||
bits.put(1, 1); // the final block,
|
||||
bits.put(2, 2); // dynamic Huffman codes (§3.2.7)
|
||||
bits.put(0, 5); // 257 literal and length codes
|
||||
bits.put(0, 5); // 1 distance code
|
||||
bits.put(14, 4); // 18 code length codes, in their order up to the one for 1
|
||||
for symbol in [16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1] {
|
||||
bits.put(u32::from(symbol == 18 || symbol == 1), 3);
|
||||
}
|
||||
// Code length codes: 1 is 0, 18 is 1. 'A' and 'B' get a bit each, 256 none.
|
||||
let zeros = |bits: &mut Bits, n: u32| {
|
||||
bits.code(1, 1);
|
||||
bits.put(n - 11, 7);
|
||||
};
|
||||
zeros(&mut bits, 65);
|
||||
bits.code(0, 1);
|
||||
bits.code(0, 1);
|
||||
zeros(&mut bits, 138);
|
||||
zeros(&mut bits, 52);
|
||||
bits.code(0, 1); // the distance code: a bit
|
||||
for _ in 0..4 {
|
||||
bits.code(0, 1); // 'A'
|
||||
}
|
||||
let data = bits.finish();
|
||||
assert!(zlib_header(&data).is_none(), "read as raw deflate");
|
||||
match decode(&data, 1 << 20, Some(1 << 20)).0 {
|
||||
Err(f) => {
|
||||
assert!(matches!(f.error, FilterError::Corrupt { .. }), "{}", f.error);
|
||||
assert_eq!(f.partial, b"");
|
||||
}
|
||||
Ok(out) => panic!("read as {out:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// A back-reference to before the start of the output points at nothing: read strictly the
|
||||
/// stream is refused, and what came before it is what a tolerant reading keeps. miniz_oxide,
|
||||
/// flate2's backend here before zlib-rs, copied zeros and went on, so this also shows which
|
||||
|
||||
@@ -263,10 +263,12 @@ pub const MAX_CHAIN: usize = 8;
|
||||
/// fixed-Huffman block is ten bits (RFC 1951 §3.2.6). With flate2's zlib-rs backend (zlib-rs
|
||||
/// 0.6.8, release build) the slowest blocks measured, empty dynamic ones, took 21 ns a byte read
|
||||
/// to make nothing, empty fixed ones 9 and fixed ones of a literal each 6, so at this price the
|
||||
/// worst comes to about 7 ns a step, inside what a step of JBIG2 decoding takes. (miniz_oxide
|
||||
/// 0.9.1, flate2's backend here before, builds a block's tables for every block, and took 1,383 ns
|
||||
/// a byte read on empty fixed blocks.) Data a real encoder makes took 0.2 to 2.3 ns a byte made, its
|
||||
/// reading included, so an image pays twice its stored length more than it would unpriced.
|
||||
/// worst per byte read comes to about 7 ns a step, inside what a step of JBIG2 decoding takes.
|
||||
/// (miniz_oxide 0.9.1, flate2's backend here before, took about 1.7 µs for every fixed-Huffman
|
||||
/// block, empty or not: 1,383 ns a byte read on empty ones.) A stage's start isn't priced, about
|
||||
/// 0.25 µs: every way to repeat one takes bytes of the file, so what starts cost is bounded by the
|
||||
/// file's length. Data a real encoder makes took 0.2 to 2.3 ns a byte made, its reading included, so
|
||||
/// an image pays twice its stored length more than it would unpriced.
|
||||
pub const FLATE_READ_WORK: u64 = 2;
|
||||
|
||||
/// Whether a chain is one [`run`] takes, and if not, the error.
|
||||
|
||||
Reference in new issue
Block a user