mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-13 14:22:25 +00:00
This commit is contained in:
bflatastic committed
commit
af67bf1d61
28 files changed
+2612
-39
No files matched your search
Generated
+2
@@ -4681,8 +4681,10 @@ dependencies = [
|
||||
"pdfcraft-testkit",
|
||||
"pdfcraft-xfa",
|
||||
"regex",
|
||||
"roxmltree 0.21.2",
|
||||
"sha2 0.11.1",
|
||||
"thiserror 2.0.21",
|
||||
"unicode-normalization",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -456,6 +456,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
|
||||
- **2026-10-15 (M3, closable community card):** The ArtCraft community card on Home has a close button that hides it for the rest of the run; it returns on the next launch, and its links stay in the top bar's Discord button and the Help menu. A UI regression covers closing and the fresh-launch default. Estimates unchanged.
|
||||
- **2026-10-15 (M14, Combine files from File Explorer, #585):** The Windows MSI adds "Combine files in PdfCraft" to PDF files' right-click menu (with the Edit verb, same switch); each selected file starts `pdfcraft --combine`, and the single-instance hand-off (`"combine": true`, `OsEvent::Combine`) collects them into the running PdfCraft's Combine files tab. Tests: the request round trip and hand-off in `single_instance.rs`, the OS event in `tests/ui.rs`, the verb rows in `test-msi.ps1`. Estimates unchanged.
|
||||
- **2026-10-15 (M14, MSI install folder, #1139):** The Windows installer's welcome dialog has an "Install to" row with a native folder browser, starting on the folder chosen for the per-user or per-machine scope (the recorded one on an update). `test-msi.ps1` checks the dialog wiring: both Install buttons set the folder before the dialog closes, and Browse's OK and Cancel. Estimates unchanged.
|
||||
- **2026-10-15 (M8, #2448):** Applying redactions also removes the document information entries and the document's and pages' XMP metadata that repeat the removed text, so a redacted word can't survive in `/Title`, `/Keywords` or the XMP that mirrors them. Estimates unchanged.
|
||||
- **2026-10-13 (community, M3, #1569 review):** Pixel-only Wayland wheel, wheel-tilt and continuous input use standalone moved events, so Ctrl zoom does not remain latched after the modifier is released (#991); vendored winit patch (13), with a policy test. Estimates unchanged.
|
||||
- **2026-10-18 (M1, PR #1143 repair policy):** A live in-file reference resolves when its generation matches the xref row or a same-number header at that indexed offset. Neither-match and free slots remain null, including when old object bytes remain. Opening records bounded xref/header disagreements without parsing object bodies or expanding compact xref rows. Synthetic controls cover read order, clones, overlays, dictionary edits, full classic/packed saves and an independent reader of incremental output. **Correction to the 2026-10-15 note below:** its async-opening-test sentence was not evidenced by this COS change; no fresh async-opening or GUI validation is claimed. The full-reference stream-length recursion key is defensive hardening, not an independently isolated regression fix. Estimates unchanged.
|
||||
- **2026-10-15 (M1, PR #1143 refresh):** COS reference lookups and full saves distinguish complete object references, so stale generations cannot expose replacement content. Snapshot restoration also keeps edits to objects with nonzero generations before incremental saves. Synthetic regressions cover caches, overlays, snapshots, stream lengths and classic/packed saves. Current upstream async-opening test corrections are included. Estimates unchanged.
|
||||
|
||||
@@ -200,6 +200,24 @@ impl Automation {
|
||||
let after = self.doc(a)?.redaction_marks();
|
||||
out["applied"] = json!(before - after);
|
||||
out["marks_pending"] = json!(after);
|
||||
// Where the removed text is still held that applying leaves as it is.
|
||||
let left = self.doc(a)?.redaction_elsewhere().unwrap_or_default();
|
||||
out["still_holding"] = json!({
|
||||
"bookmarks": left.bookmarks,
|
||||
"attachments": left.attachments,
|
||||
"comments": left.comments,
|
||||
"xmp": left.xmp,
|
||||
"total": left.total(),
|
||||
"unsearched": left.unsearched,
|
||||
});
|
||||
// What became of a PDF/A or PDF/UA claim in the document XMP the scrub removed.
|
||||
let claim = |c: pdfcraft_engine::RedactionClaim| match c {
|
||||
pdfcraft_engine::RedactionClaim::None => "none",
|
||||
pdfcraft_engine::RedactionClaim::Kept => "kept",
|
||||
pdfcraft_engine::RedactionClaim::Lost => "lost",
|
||||
};
|
||||
let id = self.doc(a)?.redaction_identification().unwrap_or_default();
|
||||
out["identification"] = json!({ "pdfa": claim(id.pdfa), "pdfua": claim(id.pdfua) });
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
|
||||
@@ -688,7 +688,7 @@ pub fn tools() -> Vec<ToolDef> {
|
||||
t(
|
||||
"redact_apply",
|
||||
"Apply redactions",
|
||||
"Apply the redaction marks (all, or those on pages): text, images, vectors, comments and form fields under them are removed for good and boxes are drawn in their place. A verification pass fails the operation if anything readable remains. Undoable until saved; the saved file no longer contains the content.",
|
||||
"Apply the redaction marks (all, or those on pages): text, images, vectors, comments and form fields under them are removed for good and boxes are drawn in their place. A verification pass fails the operation if anything readable remains. Document metadata (Info entries, the document's and pages' XMP) that repeats the removed text is removed too; when the document XMP goes, identification says what became of its PDF/A and PDF/UA claims (pdfa, pdfua: none, kept when written back in a packet holding only the identification, or lost). still_holding counts the places applying leaves as they are that still hold the removed text: bookmarks (titles), attachments (names, file names, descriptions), comments (text, rich text, subject or author of those that remain) and xmp (XMP on images, fonts, forms, annotations and other objects), with their total; doc_remove_hidden removes them. unsearched: true means the removed text couldn't be searched for, so every such place holding any text is counted. Undoable until saved; the saved file no longer contains the content.",
|
||||
)
|
||||
.destructive()
|
||||
.with(schema(json!({ "doc": doc(), "pages": pages("whose marks to apply (default: all)") }), &["doc"])),
|
||||
|
||||
@@ -5162,6 +5162,73 @@ fn redacting_through_tools() {
|
||||
assert!(matches!(a.call("redact_mark", &json!({ "doc": doc })), Err(ToolError::InvalidArgs(_))));
|
||||
}
|
||||
|
||||
/// #2539 review: `redact_apply` says where the removed text is still held that applying leaves as
|
||||
/// it is (a bookmark title and a note's text here), and changes neither.
|
||||
#[test]
|
||||
fn redact_apply_reports_places_that_still_hold_the_removed_text() {
|
||||
let dir = workdir("redact-still-holding");
|
||||
std::fs::write(dir.join("memo.txt"), "Codename Bluefin report\nPublic line").unwrap();
|
||||
let mut a = auto(&dir);
|
||||
let doc = ok(&mut a, "doc_create", json!({ "from": "text", "path": "memo.txt" }))["doc"].as_u64().unwrap();
|
||||
ok(&mut a, "bookmark_add", json!({ "doc": doc, "title": "Bluefin plans", "page": 1 }));
|
||||
ok(&mut a, "bookmark_add", json!({ "doc": doc, "title": "Annual figures", "page": 1 }));
|
||||
ok(&mut a, "comment_add", json!({ "doc": doc, "page": 1, "type": "note", "at": [300, 400], "contents": "Ask about bluefin" }));
|
||||
ok(&mut a, "comment_add", json!({ "doc": doc, "page": 1, "type": "note", "at": [300, 500], "contents": "Looks good" }));
|
||||
ok(&mut a, "redact_mark", json!({ "doc": doc, "find": "Bluefin" }));
|
||||
let r = ok(&mut a, "redact_apply", json!({ "doc": doc }));
|
||||
assert_eq!(r["still_holding"], json!({ "bookmarks": 1, "attachments": 0, "comments": 1, "xmp": 0, "total": 2, "unsearched": false }), "{r}");
|
||||
// Said in the warnings too, with the other edit notices.
|
||||
assert_eq!(
|
||||
r["warnings"],
|
||||
json!([
|
||||
"the redacted text also appears in 1 bookmark title and 1 comment, which redaction doesn't change (Remove hidden information removes them)"
|
||||
]),
|
||||
"{r}"
|
||||
);
|
||||
assert!(!page_text(&mut a, doc)[0].contains("Bluefin"));
|
||||
let titles: Vec<String> = ok(&mut a, "bookmark_list", json!({ "doc": doc }))["bookmarks"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|b| b["title"].as_str().unwrap().to_string())
|
||||
.collect();
|
||||
assert_eq!(titles, ["Bluefin plans", "Annual figures"], "reported, not changed");
|
||||
assert_eq!(r["identification"], json!({ "pdfa": "none", "pdfua": "none" }), "{r}");
|
||||
}
|
||||
|
||||
/// #2539 review: when `redact_apply` removes the document's XMP packet because it names the
|
||||
/// redacted word, `identification` says the PDF/A-2b and PDF/UA-1 identification it held was
|
||||
/// written back, and the warnings say it too.
|
||||
#[test]
|
||||
fn redact_apply_reports_the_identification_written_back() {
|
||||
let dir = workdir("redact-identification");
|
||||
let xmp = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" xmlns:pdfuaid="http://www.aiim.org/pdfua/ns/id/" pdfaid:part="2" pdfaid:conformance="B" pdfuaid:part="1"/><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Bluefin report</dc:title></rdf:Description></rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#;
|
||||
let body = "BT /F1 18 Tf 20 150 Td (Codename BLUEFIN report) Tj ET";
|
||||
let content = format!("<< /Length {} >>\nstream\n{body}\nendstream", body.len());
|
||||
let metadata = format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{xmp}\nendstream", xmp.len());
|
||||
let objs = [
|
||||
"<< /Type /Catalog /Pages 2 0 R /Metadata 6 0 R >>",
|
||||
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
|
||||
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
|
||||
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
|
||||
&content,
|
||||
&metadata,
|
||||
];
|
||||
std::fs::write(dir.join("pdfa.pdf"), pdf_of(&objs)).unwrap();
|
||||
let mut a = auto(&dir);
|
||||
let doc = ok(&mut a, "doc_open", json!({ "path": "pdfa.pdf" }))["doc"].as_u64().unwrap();
|
||||
ok(&mut a, "redact_mark", json!({ "doc": doc, "find": "BLUEFIN" }));
|
||||
let r = ok(&mut a, "redact_apply", json!({ "doc": doc }));
|
||||
assert_eq!(r["identification"], json!({ "pdfa": "kept", "pdfua": "kept" }), "{r}");
|
||||
assert_eq!(
|
||||
r["warnings"],
|
||||
json!([
|
||||
"the document's XMP metadata held the redacted text and was removed; its PDF/A and PDF/UA identification was written back on its own"
|
||||
]),
|
||||
"{r}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redacting_with_a_caller_regex_through_tools() {
|
||||
let dir = workdir("redact-regex");
|
||||
|
||||
@@ -66,8 +66,9 @@ pub struct Document {
|
||||
/// (also in `info.warnings`, kept there when the document is re-read).
|
||||
pub xfa_warnings: Vec<String>,
|
||||
/// What the last successful [`Session::apply`](crate::Session::apply) wants said (see
|
||||
/// [`EditNotice`]): text its fonts can't draw, pages whose own backdrop stays off. The
|
||||
/// interface shows it; tools return it.
|
||||
/// [`EditNotice`]): text its fonts can't draw, pages whose own backdrop stays off, places
|
||||
/// applied redactions left that still hold the removed text. The interface shows it; tools
|
||||
/// return it.
|
||||
pub edit_notices: Vec<EditNotice>,
|
||||
}
|
||||
|
||||
@@ -82,6 +83,21 @@ pub enum EditNotice {
|
||||
/// turning it on could paint over something (`pdfcraft_edit::backdrops_left_off`). They look
|
||||
/// the same on white paper.
|
||||
BackdropsLeftOff { pages: Vec<usize> },
|
||||
/// Applying redactions left these places as they were, and they still hold the removed text
|
||||
/// (bookmark titles, attachments, the comments that remain, XMP on other objects; see
|
||||
/// [`RedactionElsewhere`](crate::RedactionElsewhere)). Only when one is counted, or the text
|
||||
/// couldn't be searched for.
|
||||
RedactionElsewhere(pdfcraft_redact::Elsewhere),
|
||||
/// Applying redactions removed the document's XMP packet, which held the removed text and
|
||||
/// said the file conforms to PDF/A or PDF/UA: each claim was written back in a packet of its
|
||||
/// own, or lost (see [`RedactionIdentification`](crate::RedactionIdentification)).
|
||||
RedactionIdentification(pdfcraft_redact::Identification),
|
||||
}
|
||||
|
||||
/// The standards whose identification `id` has as `claim`: "PDF/A", "PDF/UA" or both, joined.
|
||||
pub(crate) fn standards_with(id: &pdfcraft_redact::Identification, claim: pdfcraft_redact::Claim) -> Option<String> {
|
||||
let names: Vec<&str> = [("PDF/A", id.pdfa), ("PDF/UA", id.pdfua)].into_iter().filter(|(_, c)| *c == claim).map(|(n, _)| n).collect();
|
||||
(!names.is_empty()).then(|| names.join(" and "))
|
||||
}
|
||||
|
||||
impl std::fmt::Display for EditNotice {
|
||||
@@ -96,10 +112,50 @@ impl std::fmt::Display for EditNotice {
|
||||
page_runs(pages)
|
||||
)
|
||||
}
|
||||
EditNotice::RedactionElsewhere(e) => elsewhere_text(e, f),
|
||||
EditNotice::RedactionIdentification(id) => {
|
||||
use pdfcraft_redact::Claim;
|
||||
write!(f, "the document's XMP metadata held the redacted text and was removed")?;
|
||||
if let Some(kept) = standards_with(id, Claim::Kept) {
|
||||
write!(f, "; its {kept} identification was written back on its own")?;
|
||||
}
|
||||
match standards_with(id, Claim::Lost) {
|
||||
Some(lost) => write!(f, "; its {lost} identification couldn't be kept, so the file no longer says it conforms to {lost}"),
|
||||
None => Ok(()),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// [`EditNotice::RedactionElsewhere`] in words (English; the interface has its own wording).
|
||||
fn elsewhere_text(e: &pdfcraft_redact::Elsewhere, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
let count = |n: usize, one: &str, many: &str| match n {
|
||||
0 => None,
|
||||
1 => Some(format!("1 {one}")),
|
||||
n => Some(format!("{n} {many}")),
|
||||
};
|
||||
let places: Vec<String> = [
|
||||
count(e.bookmarks, "bookmark title", "bookmark titles"),
|
||||
count(e.attachments, "attachment", "attachments"),
|
||||
count(e.comments, "comment", "comments"),
|
||||
count(e.xmp, "other metadata stream", "other metadata streams"),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.collect();
|
||||
let list = match places.split_last() {
|
||||
None => "no place counted".to_string(),
|
||||
Some((last, [])) => last.clone(),
|
||||
Some((last, rest)) => format!("{} and {last}", rest.join(", ")),
|
||||
};
|
||||
if e.unsearched {
|
||||
write!(f, "the redacted text couldn't be searched for, so {list} may still hold it (Remove hidden information removes them)")
|
||||
} else {
|
||||
write!(f, "the redacted text also appears in {list}, which redaction doesn't change (Remove hidden information removes them)")
|
||||
}
|
||||
}
|
||||
|
||||
/// 0-based pages as 1-based runs: `[0, 1, 2, 5]` is "1-3, 6".
|
||||
pub(crate) fn page_runs(pages: &[usize]) -> String {
|
||||
let mut runs: Vec<(usize, usize)> = Vec::new();
|
||||
@@ -186,6 +242,26 @@ pub(crate) fn undrawn_notices(prev: &pdfcraft_cos::Document, next: &pdfcraft_cos
|
||||
}
|
||||
|
||||
impl Document {
|
||||
/// The places the last edit's applied redactions left that still hold the removed text
|
||||
/// ([`EditNotice::RedactionElsewhere`]), or `None` when there are none or the last edit
|
||||
/// applied none.
|
||||
pub fn redaction_elsewhere(&self) -> Option<pdfcraft_redact::Elsewhere> {
|
||||
self.edit_notices.iter().find_map(|n| match n {
|
||||
EditNotice::RedactionElsewhere(e) => Some(*e),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
/// What became of the PDF/A and PDF/UA identification when the last edit's applied
|
||||
/// redactions removed the document's XMP packet ([`EditNotice::RedactionIdentification`]),
|
||||
/// or `None` when no packet that made such a claim was removed.
|
||||
pub fn redaction_identification(&self) -> Option<pdfcraft_redact::Identification> {
|
||||
self.edit_notices.iter().find_map(|n| match n {
|
||||
EditNotice::RedactionIdentification(id) => Some(*id),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
/// How the document opens (Document Properties ▸ Initial View).
|
||||
/// Accessibility ▸ Check for accessibility: the full check (`None` if the document can't be
|
||||
/// read for editing).
|
||||
|
||||
@@ -246,6 +246,11 @@ pub(crate) struct EditCtx {
|
||||
/// The overlay streams the edit's applied redactions added, which alone the renderer's check
|
||||
/// leaves out.
|
||||
pub(crate) redaction_overlays: Vec<pdfcraft_cos::ObjRef>,
|
||||
/// What the edit's applied redactions left as it was that still holds the removed text.
|
||||
pub(crate) redaction_elsewhere: Option<pdfcraft_redact::Elsewhere>,
|
||||
/// What became of the PDF/A and PDF/UA identification when the edit's applied redactions
|
||||
/// removed the catalog's XMP packet.
|
||||
pub(crate) redaction_identification: pdfcraft_redact::Identification,
|
||||
/// What [`Edit::PrepareText`] prepared, repaired and left grouped, for
|
||||
/// [`Session::prepare_text`](crate::Session::prepare_text).
|
||||
pub(crate) prepared: Option<pdfcraft_edit::text::forms::Preparation>,
|
||||
@@ -278,6 +283,8 @@ impl EditCtx {
|
||||
rng: getrandom::fill,
|
||||
renamed_fields: Vec::new(),
|
||||
redaction_overlays: Vec::new(),
|
||||
redaction_elsewhere: None,
|
||||
redaction_identification: pdfcraft_redact::Identification::default(),
|
||||
prepared: None,
|
||||
}
|
||||
}
|
||||
@@ -901,7 +908,10 @@ pub(crate) fn run_edit(doc: &mut pdfcraft_cos::Document, edit: &Edit, cx: &mut E
|
||||
pdfcraft_edit::add_content(doc, *page, &content)?;
|
||||
}
|
||||
Edit::ApplyRedactions { pages } => {
|
||||
cx.redaction_overlays = pdfcraft_redact::apply(doc, pages.as_deref())?.overlays;
|
||||
let report = pdfcraft_redact::apply(doc, pages.as_deref())?;
|
||||
cx.redaction_overlays = report.overlays;
|
||||
cx.redaction_elsewhere = Some(report.still_holding);
|
||||
cx.redaction_identification = report.identification;
|
||||
}
|
||||
Edit::ClearRedactions => {
|
||||
pdfcraft_redact::clear_marks(doc, None)?;
|
||||
|
||||
@@ -128,12 +128,14 @@ pub use pdfcraft_forms::detect;
|
||||
pub use pdfcraft_optimize as optimize;
|
||||
pub use pdfcraft_preflight as pdfa;
|
||||
pub use pdfcraft_print as print;
|
||||
pub use pdfcraft_redact::Elsewhere as RedactionElsewhere;
|
||||
pub use pdfcraft_redact::codes::{CODE_SETS as REDACTION_CODE_SETS, CodeSet as RedactionCodeSet};
|
||||
pub use pdfcraft_redact::patterns::{
|
||||
MAX_WORDS as REDACT_MAX_WORDS, PATTERNS as REDACT_PATTERNS, Pattern as RedactPattern, find as find_pattern, word_list as redact_word_list,
|
||||
};
|
||||
pub use pdfcraft_redact::sanitize::{HIDDEN, Hidden};
|
||||
pub use pdfcraft_redact::user_regex::{UserRegex as RedactRegex, VALIDATORS as REDACT_VALIDATORS, Validator as RedactValidator};
|
||||
pub use pdfcraft_redact::{Claim as RedactionClaim, Identification as RedactionIdentification};
|
||||
pub use pdfcraft_sign as sign;
|
||||
pub use pdfcraft_sign::{SignOptions, SignatureInfo, Status as SignatureStatus, StatusCaptions, TrustStore};
|
||||
pub use pdfcraft_xfdf::Format as DataFormat;
|
||||
|
||||
@@ -201,7 +201,13 @@ impl Session {
|
||||
if signed && ((rewrites(&next) && !rewrites(&editor.cos)) || (next.is_modified() && next.revisions().is_empty())) {
|
||||
return Err(EditError::SignedRewrite(edit.label().to_string()));
|
||||
}
|
||||
let notices = edit_notices(&edit, &editor.cos, &next);
|
||||
let mut notices = edit_notices(&edit, &editor.cos, &next);
|
||||
// Places applied redactions left that still hold the removed text, and what became of
|
||||
// the PDF/A or PDF/UA identification in the XMP they removed: said, not silent.
|
||||
notices.extend(cx.redaction_elsewhere.filter(|e| e.total() > 0 || e.unsearched).map(EditNotice::RedactionElsewhere));
|
||||
if cx.redaction_identification != pdfcraft_redact::Identification::default() {
|
||||
notices.push(EditNotice::RedactionIdentification(cx.redaction_identification));
|
||||
}
|
||||
let previous = std::mem::replace(&mut editor.cos, next);
|
||||
let keys = keys_after(&edit).unwrap_or_else(|| editor.keys.clone());
|
||||
let previous_keys = std::mem::replace(&mut editor.keys, keys);
|
||||
|
||||
+50
-13
@@ -3303,19 +3303,7 @@ fn one_line_page_with(resources: &str, stream_entries: &str, body: &str) -> Vec<
|
||||
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>".to_string(),
|
||||
format!("<< /Length {} {stream_entries} >>\nstream\n{body}\nendstream", body.len()),
|
||||
];
|
||||
let mut out = b"%PDF-1.7\n".to_vec();
|
||||
let mut offsets = Vec::new();
|
||||
for (i, o) in objs.iter().enumerate() {
|
||||
offsets.push(out.len());
|
||||
out.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
|
||||
}
|
||||
let xref = out.len();
|
||||
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
|
||||
for o in offsets {
|
||||
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
|
||||
}
|
||||
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
|
||||
out
|
||||
pdf_of(&objs)
|
||||
}
|
||||
|
||||
/// Put a redaction mark over `needle` where page 1 shows it, as Find text and redact does.
|
||||
@@ -3382,6 +3370,55 @@ fn text_after_an_inline_image_is_redacted_where_the_page_shows_it() {
|
||||
);
|
||||
}
|
||||
|
||||
/// #2539 review: Apply redactions tells the caller where the removed text is still held that
|
||||
/// it leaves as it is (here a bookmark title), and the next edit, or an undo, forgets it.
|
||||
#[test]
|
||||
fn applied_redactions_report_a_bookmark_that_still_holds_the_removed_text() {
|
||||
let mut s = Session::new().with_clock(|| 1_700_000_000);
|
||||
let page = one_line_page("/Font << /F1 4 0 R >>", "BT /F1 10 Tf 20 150 Td (Codename BLUEFIN report) Tj ET");
|
||||
let id = s.open("bookmark.pdf", None, Arc::new(page), None).unwrap();
|
||||
s.apply(id, Edit::AddBookmark { parent: vec![], index: 0, title: "Bluefin plans".into(), page: 0, left: None, top: None }).unwrap();
|
||||
s.apply(id, Edit::AddBookmark { parent: vec![], index: 1, title: "Annual figures".into(), page: 0, left: None, top: None }).unwrap();
|
||||
mark_shown_text(&mut s, id, "BLUEFIN");
|
||||
assert_eq!(s.get(id).unwrap().redaction_elsewhere(), None, "only applying redactions reports it");
|
||||
s.apply(id, Edit::ApplyRedactions { pages: None }).unwrap();
|
||||
let found = s.get(id).unwrap().redaction_elsewhere().expect("applying reports what it left");
|
||||
assert_eq!(found, crate::RedactionElsewhere { bookmarks: 1, ..Default::default() });
|
||||
// One of the edit notices, so tools and the interface say it with the others.
|
||||
let said: Vec<String> = s.get(id).unwrap().edit_notices.iter().map(ToString::to_string).collect();
|
||||
assert_eq!(said, ["the redacted text also appears in 1 bookmark title, which redaction doesn't change (Remove hidden information removes them)"]);
|
||||
assert_eq!(outline_titles(&s.get(id).unwrap().info.outline), ["Bluefin plans→1", "Annual figures→1"], "the bookmark is reported, not changed");
|
||||
s.undo(id).unwrap();
|
||||
assert_eq!(s.get(id).unwrap().redaction_elsewhere(), None, "an undo forgets it");
|
||||
}
|
||||
|
||||
/// #2539 review: applying redactions that remove the document's XMP packet, which named the
|
||||
/// redacted word and said the file conforms to PDF/A-2b, writes the identification back and
|
||||
/// says so; tools list it with the other edit notices, and an undo forgets it.
|
||||
#[test]
|
||||
fn applied_redactions_report_the_pdfa_identification_written_back() {
|
||||
let xmp = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" pdfaid:part="2" pdfaid:conformance="B"/><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Bluefin report</dc:title></rdf:Description></rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#;
|
||||
let body = "BT /F1 10 Tf 20 150 Td (Codename BLUEFIN report) Tj ET";
|
||||
let objs = [
|
||||
"<< /Type /Catalog /Pages 2 0 R /Metadata 6 0 R >>".to_string(),
|
||||
"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 300] >>".to_string(),
|
||||
"<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>".to_string(),
|
||||
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>".to_string(),
|
||||
format!("<< /Length {} >>\nstream\n{body}\nendstream", body.len()),
|
||||
format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{xmp}\nendstream", xmp.len()),
|
||||
];
|
||||
let mut s = Session::new().with_clock(|| 1_700_000_000);
|
||||
let id = s.open("pdfa.pdf", None, Arc::new(pdf_of(&objs)), None).unwrap();
|
||||
mark_shown_text(&mut s, id, "BLUEFIN");
|
||||
s.apply(id, Edit::ApplyRedactions { pages: None }).unwrap();
|
||||
let kept = crate::RedactionIdentification { pdfa: crate::RedactionClaim::Kept, pdfua: crate::RedactionClaim::None };
|
||||
assert_eq!(s.get(id).unwrap().redaction_identification(), Some(kept));
|
||||
let said: Vec<String> = s.get(id).unwrap().edit_notices.iter().map(ToString::to_string).collect();
|
||||
assert_eq!(said, ["the document's XMP metadata held the redacted text and was removed; its PDF/A identification was written back on its own"]);
|
||||
s.undo(id).unwrap();
|
||||
assert_eq!(s.get(id).unwrap().redaction_identification(), None, "an undo forgets it");
|
||||
}
|
||||
|
||||
/// A word whose font and size an ExtGState /Font entry (`gs`) sets is placed where the renderer
|
||||
/// draws it, so a mark drawn over it as displayed takes it out of the file.
|
||||
#[test]
|
||||
|
||||
@@ -24,6 +24,10 @@ pdfcraft-xfa = { workspace = true }
|
||||
pdfcraft-filters = { workspace = true }
|
||||
regex = { workspace = true }
|
||||
thiserror = { workspace = true }
|
||||
# NFKC for the metadata check; already in the build through pdfcraft-crypt (stringprep).
|
||||
unicode-normalization = "0.1"
|
||||
# Reads the PDF/A and PDF/UA identification by namespace; already in the build through pdfcraft-xfa.
|
||||
roxmltree = "0.21"
|
||||
sha2 = "0.11"
|
||||
|
||||
[dev-dependencies]
|
||||
|
||||
+55
-1
@@ -47,7 +47,61 @@ clear_marks(&mut doc, None)?; // remove marks without app
|
||||
- **annotations** whose rectangle overlaps (with their pop-ups and their replies, `/IRT`), and **form fields** with a
|
||||
widget under a mark;
|
||||
- the marks themselves, replaced by boxes in their fill colour (and overlay text) drawn into the
|
||||
page as a `/PCMark /Redaction` stream.
|
||||
page as a `/PCMark /Redaction` stream;
|
||||
- **metadata** that repeats the removed text (#2448): `/Info` entries containing it or wholly
|
||||
part of it as whole words (three letters or digits at least), and the catalog's and pages'
|
||||
XMP streams whose text (element content, attribute values other than `xmlns` declarations,
|
||||
comments and processing instructions, entities decoded) contains it or has a value wholly
|
||||
part of it, removed whole. When the catalog's packet goes, its PDF/A (`pdfaid`) and PDF/UA
|
||||
(`pdfuaid`) identification, read by namespace, is written back in a packet holding nothing
|
||||
else: the part number, the PDF/A conformance letter, amendment and revision, and PDF/UA's
|
||||
part and revision, each kept only if it is the short code it must be and doesn't itself hold
|
||||
the removed text (a one-character code can't, so it is kept even when the text couldn't be
|
||||
searched for), plus a fixed extension schema for `pdfuaid` under PDF/A-1 to -3, left out
|
||||
with the PDF/UA claim if the redacted words are among its words. `Report::identification`
|
||||
says, for each, whether the claim was kept or lost (a value that isn't a valid code or holds
|
||||
the text, or a packet that can't be read); the notification and `redact_apply` say it too.
|
||||
`/Info` entries that remain aren't mirrored into the new packet, so a PDF/A validator may
|
||||
still report them. The removed
|
||||
glyphs' text is collected in runs by position (character and word spacing included, so
|
||||
letter-spaced text stays one word), and each run (two or more letters or digits) and each of
|
||||
its words of three or more are matched without regard to case or spacing, and also by their
|
||||
letters and digits alone when they have three or more: "Blue fin" and "BLUE-FIN" hold
|
||||
"Bluefin", and so does a heading whose letters `TJ` numbers spread apart (collected as
|
||||
"B L U E F I N", since a `TJ` gap can't be told from a word space). A value is wholly part of
|
||||
a run when its letters and digits are, starting and ending where the run's words do, so
|
||||
"True" isn't part of "untrue". This over-matches only towards removal: "is a" also holds
|
||||
"Lisa". Both sides are compared in Unicode compatibility form (NFKC) and lower case, so a
|
||||
decomposed "Müller" (u and U+0308), full-width letters and a "fi" ligature from a
|
||||
`ToUnicode` map match their plain forms. A string behind an `FF FE` mark, which ISO 32000-2
|
||||
§7.9.2.2.1 doesn't define, is read as UTF-16LE, here and in the places reported below, rather
|
||||
than through PDFDocEncoding, in which it could never match. A value that shares only words of one or two
|
||||
letters with the removed text, and isn't wholly part of it, stays:
|
||||
Sanitize's metadata category removes the rest. `Report::metadata` counts what went.
|
||||
- **Fail-closed.** When the check can't know what the removed text was, every `/Info`
|
||||
string and every catalog and page XMP stream goes: more than 100,000 runs on a page, a run
|
||||
over 64 KiB, or more than 16 MiB in the whole document removed (the collection limits); a removed glyph whose text isn't known (no
|
||||
`/ToUnicode` or encoding gives it); a form XObject removed without being read, because it
|
||||
is nested past the depth limit (`MAX_DEPTH`, 50) or its content can't be read; or more
|
||||
than 20,000 terms, or a run longer than 4,096 characters, to search for. Once the
|
||||
search's work limit (2^28 steps) runs out, every entry and packet not yet checked goes
|
||||
too. An XMP packet that isn't valid UTF-8 or UTF-16, can't be decoded strictly (damaged or
|
||||
cut-short filter data, or more than one filter), or decodes to more than
|
||||
16 MiB goes without being read, and a value whose normalized form would be more than four
|
||||
times as long (U+FDFA alone is 18 letters) counts as holding the text.
|
||||
- **Not scrubbed, but reported** (Remove hidden information's categories remove them): bookmark
|
||||
titles, attachments (names in the document's list of embedded files, file names and
|
||||
descriptions, and the files of file attachment comments), the comments that remain (their
|
||||
text, rich text, subject and author; not form fields or pop-ups), and XMP streams on anything
|
||||
but the catalog and the pages (images, fonts, form XObjects, annotations and other objects).
|
||||
`apply` changes none of them, but counts those that still hold the removed text, matched as
|
||||
above, in `Report::still_holding` (`Elsewhere`), so the interface and the `redact_apply` tool
|
||||
can say where it is; this search has a work limit of its own, past which every place counts.
|
||||
When the removed text couldn't be searched for (the cases above in which every `/Info`
|
||||
string goes), every one of them that holds any text is counted and `unsearched` is set; an XMP
|
||||
stream or rich text stream that can't be read counts as holding it. The walks are bounded: at
|
||||
most 100,000 bookmarks and 100,000 name-tree nodes (32 deep) are read, each once, and a walk
|
||||
cut short counts one more place.
|
||||
|
||||
A verification pass re-reads each redacted page and fails the operation if any glyph or inline
|
||||
image is still under an area. It checks every content stream the page had, whatever its
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
//! Places [`crate::apply`] leaves as they are that may still hold the removed text (#2448
|
||||
//! review): bookmark titles, attachments, the comments that remain, and XMP on objects other than
|
||||
//! the catalog and the pages. Removing them is Remove hidden information's job (its bookmarks,
|
||||
//! attachments, comments and metadata categories); applying redactions only counts them, so the
|
||||
//! user can be told. Nothing here changes the document.
|
||||
//!
|
||||
//! The text is matched as the metadata check matches it ([`crate::metadata::Search`]), with a
|
||||
//! work limit of its own. Every walk is bounded: outline items and name-tree nodes are each read
|
||||
//! once (a seen-set for indirect ones) up to a cap, and the XMP walk is Remove hidden
|
||||
//! information's own.
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use pdfcraft_cos::{Document, ObjRef, Object};
|
||||
|
||||
use crate::metadata::Search;
|
||||
|
||||
/// Places that may still hold the removed text after [`crate::apply`], which it doesn't change.
|
||||
/// Remove hidden information removes them. Each count is of places, not of occurrences.
|
||||
///
|
||||
/// When the removed text couldn't be searched for (`unsearched`: the text of a removed glyph
|
||||
/// isn't known, a form was removed unread, or there was too much removed text to collect or
|
||||
/// search for, as for the metadata check), every place holding any text is counted, since any
|
||||
/// of them may hold it. A count may also include places that couldn't be read or that lie past
|
||||
/// a walk's cap (one for each walk cut short), which count as holding it, and, once the search's
|
||||
/// work limit runs out, every place not yet checked that holds any text.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Elsewhere {
|
||||
/// Bookmarks (outline items) whose title holds it.
|
||||
pub bookmarks: usize,
|
||||
/// Embedded files whose name in the document's list of attachments (`/EmbeddedFiles`),
|
||||
/// file name (`/F`, `/UF`) or description (`/Desc`) holds it, and file attachment
|
||||
/// annotations whose file's name or description does.
|
||||
pub attachments: usize,
|
||||
/// Annotations left on the pages (not form field widgets or pop-ups) whose text
|
||||
/// (`/Contents`, rich text `/RC`, subject `/Subj`) or author (`/T`) holds it.
|
||||
pub comments: usize,
|
||||
/// XMP streams on objects other than the catalog and the pages (images, fonts, form
|
||||
/// XObjects, annotations…) whose text holds it, or that can't be read.
|
||||
pub xmp: usize,
|
||||
/// The removed text couldn't be searched for, so the counts are of every place that holds
|
||||
/// any text.
|
||||
pub unsearched: bool,
|
||||
}
|
||||
|
||||
impl Elsewhere {
|
||||
/// All the places counted.
|
||||
pub fn total(&self) -> usize {
|
||||
self.bookmarks.saturating_add(self.attachments).saturating_add(self.comments).saturating_add(self.xmp)
|
||||
}
|
||||
}
|
||||
|
||||
/// The most outline items read.
|
||||
const MAX_OUTLINE_ITEMS: usize = 100_000;
|
||||
/// The most name-tree nodes read, and how deep the tree is followed.
|
||||
const MAX_NAME_TREE_NODES: usize = 100_000;
|
||||
const MAX_NAME_TREE_DEPTH: usize = 32;
|
||||
|
||||
/// Count the places that hold the `removed` text (see [`Elsewhere`]). `complete` is false when
|
||||
/// the collection of removed text was cut short.
|
||||
pub(crate) fn find(doc: &Document, removed: &[String], complete: bool) -> Elsewhere {
|
||||
let search = Search::of(removed, complete);
|
||||
if search.is_nothing() {
|
||||
return Elsewhere::default();
|
||||
}
|
||||
// A blank value holds nothing, even when everything counts.
|
||||
let hit = |o: Option<&Object>| {
|
||||
o.and_then(|o| doc.resolve(o).as_string().map(crate::metadata::text_string)).is_some_and(|t| !t.trim().is_empty() && search.text(&t))
|
||||
};
|
||||
Elsewhere {
|
||||
bookmarks: bookmarks(doc, &hit),
|
||||
attachments: attachments(doc, &hit),
|
||||
comments: comments(doc, &search, &hit),
|
||||
xmp: xmp(doc, &search),
|
||||
unsearched: search.is_everything(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Outline items whose `/Title` holds the text.
|
||||
fn bookmarks(doc: &Document, hit: &dyn Fn(Option<&Object>) -> bool) -> usize {
|
||||
let Some(root) = doc.root() else { return 0 };
|
||||
let Some(first) =
|
||||
doc.get(root).as_dict().and_then(|c| c.get(b"Outlines").cloned()).and_then(|o| doc.resolve(&o).as_dict()?.get(b"First").cloned())
|
||||
else {
|
||||
return 0;
|
||||
};
|
||||
let (mut n, mut read) = (0usize, 0usize);
|
||||
let mut seen = HashSet::new();
|
||||
let mut next = vec![first];
|
||||
while let Some(o) = next.pop() {
|
||||
if let Object::Ref(r) = &o
|
||||
&& !seen.insert(*r)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(item) = doc.resolve(&o).as_dict().cloned() else { continue };
|
||||
if read >= MAX_OUTLINE_ITEMS {
|
||||
// Items past the cap aren't read: count one for them.
|
||||
return n.saturating_add(1);
|
||||
}
|
||||
read += 1;
|
||||
n += usize::from(hit(item.get(b"Title")));
|
||||
next.extend([b"Next".as_slice(), b"First"].into_iter().filter_map(|k| item.get(k).cloned()));
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// Embedded files in the `/EmbeddedFiles` name tree whose name, file name or description holds
|
||||
/// the text, and file attachment annotations whose file's do.
|
||||
fn attachments(doc: &Document, hit: &dyn Fn(Option<&Object>) -> bool) -> usize {
|
||||
let file = |spec: Option<&Object>| {
|
||||
spec.and_then(|s| doc.resolve(s).as_dict().cloned()).is_some_and(|d| [b"F".as_slice(), b"UF", b"Desc"].iter().any(|k| hit(d.get(k))))
|
||||
};
|
||||
let mut n = 0usize;
|
||||
let tree = doc
|
||||
.root()
|
||||
.and_then(|r| doc.get(r).as_dict()?.get(b"Names").cloned())
|
||||
.and_then(|names| doc.resolve(&names).as_dict()?.get(b"EmbeddedFiles").cloned());
|
||||
if let Some(tree) = tree {
|
||||
let mut seen = HashSet::new();
|
||||
let mut next = vec![(tree, 0usize)];
|
||||
let mut read = 0usize;
|
||||
while let Some((o, depth)) = next.pop() {
|
||||
if let Object::Ref(r) = &o
|
||||
&& !seen.insert(*r)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(node) = doc.resolve(&o).as_dict().cloned() else { continue };
|
||||
if read >= MAX_NAME_TREE_NODES || depth > MAX_NAME_TREE_DEPTH {
|
||||
// A node past the cap isn't read: count one for what it holds.
|
||||
n = n.saturating_add(1);
|
||||
break;
|
||||
}
|
||||
read += 1;
|
||||
if let Some(leaves) = node.get(b"Names").map(|l| doc.resolve(l)).and_then(|l| l.as_array().cloned()) {
|
||||
for pair in leaves.chunks(2) {
|
||||
n += usize::from(hit(pair.first()) || file(pair.get(1)));
|
||||
}
|
||||
}
|
||||
if let Some(kids) = node.get(b"Kids").map(|k| doc.resolve(k)).and_then(|k| k.as_array().cloned()) {
|
||||
next.extend(kids.into_iter().map(|k| (k, depth.saturating_add(1))));
|
||||
}
|
||||
}
|
||||
}
|
||||
for page in pdfcraft_model::pages(doc) {
|
||||
for a in crate::annots_of(doc, &page.dict) {
|
||||
let Some(a) = doc.resolve(&a).as_dict().cloned() else { continue };
|
||||
if a.name(b"Subtype") == Some(b"FileAttachment".as_slice()) {
|
||||
n += usize::from(file(a.get(b"FS")));
|
||||
}
|
||||
}
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// Annotations on the pages, other than form field widgets and pop-ups (which show their
|
||||
/// parent's text), whose text or author holds it.
|
||||
fn comments(doc: &Document, search: &Search, hit: &dyn Fn(Option<&Object>) -> bool) -> usize {
|
||||
let mut seen = HashSet::new();
|
||||
let mut n = 0usize;
|
||||
for page in pdfcraft_model::pages(doc) {
|
||||
for a in crate::annots_of(doc, &page.dict) {
|
||||
if let Some(r) = a.as_ref()
|
||||
&& !seen.insert(r)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(a) = doc.resolve(&a).as_dict().cloned() else { continue };
|
||||
if matches!(a.name(b"Subtype"), Some(b"Widget" | b"Popup")) {
|
||||
continue;
|
||||
}
|
||||
// Rich text is XHTML: its text is searched, not its markup. A rich text stream that
|
||||
// can't be read counts as holding it.
|
||||
let rich = a.get(b"RC").is_some_and(|o| rich_text(doc, o).is_none_or(|t| !t.trim().is_empty() && search.text(&t)));
|
||||
n += usize::from(rich || [b"Contents".as_slice(), b"T", b"Subj"].iter().any(|k| hit(a.get(k))));
|
||||
}
|
||||
}
|
||||
n
|
||||
}
|
||||
|
||||
/// The text of a rich text string or stream (`/RC`), markup left out; `None` when a stream
|
||||
/// can't be read (strictly, as UTF-8, within the XMP limit).
|
||||
fn rich_text(doc: &Document, o: &Object) -> Option<String> {
|
||||
let xml = match &*doc.resolve(o) {
|
||||
Object::String(s) => crate::metadata::text_string(s),
|
||||
Object::Stream(s) if s.filters().len() <= 1 => {
|
||||
String::from_utf8(doc.with_direct_parms(s).decoded_first_filter_strict(crate::metadata::MAX_XMP_BYTES).ok()?).ok()?
|
||||
}
|
||||
Object::Stream(_) => return None,
|
||||
_ => return Some(String::new()),
|
||||
};
|
||||
Some(crate::metadata::xml_values(&xml).join(" "))
|
||||
}
|
||||
|
||||
/// XMP streams on anything reachable from the catalog other than the catalog's and the pages'
|
||||
/// own, each counted once, that hold the text or can't be read.
|
||||
fn xmp(doc: &Document, search: &Search) -> usize {
|
||||
let scrubbed: HashSet<ObjRef> = crate::metadata::catalog_and_pages(doc).into_iter().collect();
|
||||
let Ok(values) = crate::sanitize::owned_values(doc, b"Metadata") else {
|
||||
// A graph too deep to walk: some XMP may be there, unread.
|
||||
return 1;
|
||||
};
|
||||
let mut seen = HashSet::new();
|
||||
let mut n = 0usize;
|
||||
for (holder, own, v) in values {
|
||||
if own && scrubbed.contains(&holder) {
|
||||
continue;
|
||||
}
|
||||
if let Some(r) = v.as_ref()
|
||||
&& !seen.insert(r)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
// An entry that isn't a stream holds no XMP.
|
||||
if !matches!(&*doc.resolve(&v), Object::Stream(_)) {
|
||||
continue;
|
||||
}
|
||||
n += usize::from(search.xmp(doc, &v));
|
||||
}
|
||||
n
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
//! The PDF/A and PDF/UA identification of a document XMP packet the metadata check removes
|
||||
//! (#2539 review).
|
||||
//!
|
||||
//! The scrub removes the catalog's XMP packet whole when it holds the removed text, and the
|
||||
//! packet is where a file says it conforms to PDF/A (ISO 19005, the `pdfaid` schema) or PDF/UA
|
||||
//! (ISO 14289, `pdfuaid`). Rather than drop that claim silently, the scrub writes back a packet
|
||||
//! holding only those identification properties, read by namespace (not by prefix), each
|
||||
//! checked to be the short code it must be (a part number, a conformance letter, a year), and
|
||||
//! none of them holding the removed text. Nothing else of the old packet is copied. What became
|
||||
//! of each claim is reported ([`Identification`]).
|
||||
|
||||
use pdfcraft_cos::{Dict, Document, Object, Stream};
|
||||
|
||||
use crate::metadata::Search;
|
||||
|
||||
const PDFA_ID: &str = "http://www.aiim.org/pdfa/ns/id/";
|
||||
const PDFUA_ID: &str = "http://www.aiim.org/pdfua/ns/id/";
|
||||
/// Packets with more `=` than this (attributes, roughly) aren't parsed: the parser checks each
|
||||
/// element's attributes against each other, so a huge attribute list would be slow.
|
||||
const MAX_EQUALS: usize = 4096;
|
||||
/// The most XML nodes parsed.
|
||||
const MAX_NODES: u32 = 100_000;
|
||||
/// The PDF/A extension schema describing `pdfuaid:part`, fixed text.
|
||||
const PDFUA_EXTENSION: &str = concat!(
|
||||
r#"<rdf:Description rdf:about="" xmlns:pdfaExtension="http://www.aiim.org/pdfa/ns/extension/" xmlns:pdfaSchema="http://www.aiim.org/pdfa/ns/schema#" xmlns:pdfaProperty="http://www.aiim.org/pdfa/ns/property#">"#,
|
||||
r#"<pdfaExtension:schemas><rdf:Bag><rdf:li rdf:parseType="Resource">"#,
|
||||
r#"<pdfaSchema:schema>PDF/UA identification schema</pdfaSchema:schema><pdfaSchema:namespaceURI>http://www.aiim.org/pdfua/ns/id/</pdfaSchema:namespaceURI><pdfaSchema:prefix>pdfuaid</pdfaSchema:prefix>"#,
|
||||
r#"<pdfaSchema:property><rdf:Seq><rdf:li rdf:parseType="Resource">"#,
|
||||
r#"<pdfaProperty:name>part</pdfaProperty:name><pdfaProperty:valueType>Integer</pdfaProperty:valueType><pdfaProperty:category>internal</pdfaProperty:category><pdfaProperty:description>The part of ISO 14289 the file conforms to</pdfaProperty:description>"#,
|
||||
r#"</rdf:li></rdf:Seq></pdfaSchema:property></rdf:li></rdf:Bag></pdfaExtension:schemas></rdf:Description>"#,
|
||||
);
|
||||
|
||||
/// What became of a conformance claim when the packet holding it was removed.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub enum Claim {
|
||||
/// No removed packet made the claim.
|
||||
#[default]
|
||||
None,
|
||||
/// The packet written in its place makes it again.
|
||||
Kept,
|
||||
/// The removed packet made it (or, when it couldn't be read, seems to have), and it couldn't
|
||||
/// be written back: a property wasn't a valid code, held the removed text, or the packet
|
||||
/// couldn't be read.
|
||||
Lost,
|
||||
}
|
||||
|
||||
/// What became of the document's PDF/A and PDF/UA identification when the metadata check
|
||||
/// removed the catalog's XMP packet.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Identification {
|
||||
/// PDF/A (`pdfaid`: part, conformance, and amendment and revision when present).
|
||||
pub pdfa: Claim,
|
||||
/// PDF/UA (`pdfuaid`: part, and revision when present).
|
||||
pub pdfua: Claim,
|
||||
}
|
||||
|
||||
impl Identification {
|
||||
/// A claim was lost.
|
||||
pub fn lost(&self) -> bool {
|
||||
self.pdfa == Claim::Lost || self.pdfua == Claim::Lost
|
||||
}
|
||||
}
|
||||
|
||||
/// The identification of the removed catalog packet `xmp`, and the packet to write in its place
|
||||
/// (`None` when nothing is kept).
|
||||
pub(crate) fn keep(doc: &Document, xmp: &Object, search: &Search) -> (Identification, Option<Stream>) {
|
||||
let Some(text) = crate::metadata::xmp_text(doc, xmp) else {
|
||||
// Unreadable: a claim it seems to make, by its namespace or usual prefix in the raw
|
||||
// bytes, is lost.
|
||||
let seen = |raw: &[u8], ns: &str, prefix: &[u8]| {
|
||||
let has = |n: &[u8]| raw.windows(n.len()).any(|w| w == n);
|
||||
if has(ns.as_bytes()) || has(prefix) { Claim::Lost } else { Claim::None }
|
||||
};
|
||||
let id = match &*doc.resolve(xmp) {
|
||||
Object::Stream(s) => Identification { pdfa: seen(&s.raw, PDFA_ID, b"pdfaid:"), pdfua: seen(&s.raw, PDFUA_ID, b"pdfuaid:") },
|
||||
_ => Identification::default(),
|
||||
};
|
||||
return (id, None);
|
||||
};
|
||||
let Some(found) = read(&text) else {
|
||||
let seen = |ns: &str, prefix: &str| if text.contains(ns) || text.contains(prefix) { Claim::Lost } else { Claim::None };
|
||||
return (Identification { pdfa: seen(PDFA_ID, "pdfaid:"), pdfua: seen(PDFUA_ID, "pdfuaid:") }, None);
|
||||
};
|
||||
// A value is kept when it is a valid code and doesn't hold the removed text. One letter or
|
||||
// digit can't be a term, so it is kept even when everything counts as holding the text.
|
||||
let ok = |v: &Option<String>, valid: fn(&str) -> bool| -> Option<String> {
|
||||
let v = v.as_deref()?;
|
||||
let short = v.chars().filter(|c| c.is_alphanumeric()).count() < 2;
|
||||
(valid(v) && (short || !search.text(v))).then(|| v.to_string())
|
||||
};
|
||||
let mut id = Identification::default();
|
||||
let mut body = String::new();
|
||||
|
||||
let a = &found.pdfa;
|
||||
if a.any() {
|
||||
let part = ok(&a.part, part);
|
||||
let conformance = ok(&a.conformance, |v| v.len() == 1 && v.bytes().all(|b| b.is_ascii_alphabetic()));
|
||||
// PDF/A-1 to -3 require a conformance level; PDF/A-4 has one only for -4e and -4f.
|
||||
if part.is_some() && (conformance.is_some() || a.conformance.is_none()) {
|
||||
id.pdfa = Claim::Kept;
|
||||
body.push_str(r#"<rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/">"#);
|
||||
let props = [("part", part), ("conformance", conformance), ("amd", ok(&a.amd, amendment)), ("rev", ok(&a.rev, year))];
|
||||
for (name, v) in props {
|
||||
if let Some(v) = v {
|
||||
body.push_str(&format!("<pdfaid:{name}>{v}</pdfaid:{name}>"));
|
||||
}
|
||||
}
|
||||
body.push_str("</rdf:Description>");
|
||||
} else {
|
||||
id.pdfa = Claim::Lost;
|
||||
}
|
||||
}
|
||||
if found.pdfua.any() {
|
||||
// PDF/A-1 to -3 accept only the schemas they define or that the packet describes in an
|
||||
// extension schema, which pdfuaid needs there. Its fixed text is ours, but a file whose
|
||||
// redacted words it happens to use doesn't get it (nor, then, the PDF/UA claim).
|
||||
let extension = id.pdfa == Claim::Kept && a.part.as_deref().is_some_and(|p| matches!(p, "1" | "2" | "3"));
|
||||
let extension_ok = !extension || !crate::metadata::xml_values(PDFUA_EXTENSION).iter().any(|v| search.text(v));
|
||||
match ok(&found.pdfua.part, part).filter(|_| extension_ok) {
|
||||
Some(p) => {
|
||||
id.pdfua = Claim::Kept;
|
||||
// PDF/UA-2 adds a revision year (PDF 2.0, so never beside PDF/A-1 to -3).
|
||||
let rev = ok(&found.pdfua.rev, year).map(|r| format!("<pdfuaid:rev>{r}</pdfuaid:rev>")).unwrap_or_default();
|
||||
body.push_str(&format!(r#"<rdf:Description rdf:about="" xmlns:pdfuaid="http://www.aiim.org/pdfua/ns/id/"><pdfuaid:part>{p}</pdfuaid:part>{rev}</rdf:Description>"#));
|
||||
if extension {
|
||||
body.push_str(PDFUA_EXTENSION);
|
||||
}
|
||||
}
|
||||
None => id.pdfua = Claim::Lost,
|
||||
}
|
||||
}
|
||||
if body.is_empty() {
|
||||
return (id, None);
|
||||
}
|
||||
let packet = format!(
|
||||
"<?xpacket begin=\"\u{FEFF}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>\n<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">{body}</rdf:RDF></x:xmpmeta>\n<?xpacket end=\"w\"?>"
|
||||
);
|
||||
let mut dict = Dict::new();
|
||||
dict.set(b"Type".to_vec(), Object::name("Metadata"));
|
||||
dict.set(b"Subtype".to_vec(), Object::name("XML"));
|
||||
// Unfiltered, as PDF/A requires of metadata streams.
|
||||
(id, Some(Stream::from_raw(dict, packet.into_bytes())))
|
||||
}
|
||||
|
||||
/// A part number: one or two digits.
|
||||
fn part(v: &str) -> bool {
|
||||
(1..=2).contains(&v.len()) && v.bytes().all(|b| b.is_ascii_digit())
|
||||
}
|
||||
|
||||
/// An amendment identifier, such as "2005" or "1:2005".
|
||||
fn amendment(v: &str) -> bool {
|
||||
(1..=16).contains(&v.len()) && v.bytes().all(|b| b.is_ascii_digit() || b == b':' || b == b'.' || b == b'-')
|
||||
}
|
||||
|
||||
/// A revision year.
|
||||
fn year(v: &str) -> bool {
|
||||
v.len() == 4 && v.bytes().all(|b| b.is_ascii_digit())
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct PdfaId {
|
||||
part: Option<String>,
|
||||
conformance: Option<String>,
|
||||
amd: Option<String>,
|
||||
rev: Option<String>,
|
||||
}
|
||||
|
||||
impl PdfaId {
|
||||
fn any(&self) -> bool {
|
||||
self.part.is_some() || self.conformance.is_some() || self.amd.is_some() || self.rev.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Found {
|
||||
pdfa: PdfaId,
|
||||
pdfua: PdfaId,
|
||||
}
|
||||
|
||||
/// The identification properties of a packet, by namespace, written as attributes or as
|
||||
/// elements; the first of each wins. `None` when the packet can't be parsed.
|
||||
fn read(text: &str) -> Option<Found> {
|
||||
if text.bytes().filter(|&b| b == b'=').count() > MAX_EQUALS {
|
||||
return None;
|
||||
}
|
||||
let options = roxmltree::ParsingOptions { allow_dtd: false, nodes_limit: MAX_NODES, ..Default::default() };
|
||||
let xml = roxmltree::Document::parse_with_options(text, options).ok()?;
|
||||
let mut found = Found::default();
|
||||
for node in xml.descendants().filter(roxmltree::Node::is_element) {
|
||||
for (ns, id) in [(PDFA_ID, &mut found.pdfa), (PDFUA_ID, &mut found.pdfua)] {
|
||||
for (local, slot) in [("part", &mut id.part), ("conformance", &mut id.conformance), ("amd", &mut id.amd), ("rev", &mut id.rev)] {
|
||||
if slot.is_some() {
|
||||
continue;
|
||||
}
|
||||
if let Some(v) = node.attribute((ns, local)) {
|
||||
*slot = Some(v.trim().to_string());
|
||||
} else if node.tag_name().namespace() == Some(ns) && node.tag_name().name() == local {
|
||||
*slot = Some(node.text().unwrap_or("").trim().to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(found)
|
||||
}
|
||||
@@ -46,6 +46,14 @@ pub(crate) const MAX_DEPTH: usize = 50;
|
||||
// A form a reader still shows must be walkable here, or a redaction removes content that is
|
||||
// visible; the build says so rather than a test, so the two limits cannot drift apart.
|
||||
const _: () = assert!(MAX_DEPTH >= pdfcraft_edit::text::MAX_FORM_DEPTH);
|
||||
/// Bounds on the removed text collected for the metadata check (#2448).
|
||||
const MAX_RUNS: usize = 100_000;
|
||||
const MAX_RUN_BYTES: usize = 64 * 1024;
|
||||
/// The removed text kept for the whole document, in bytes. A glyph's `/ToUnicode` text has no
|
||||
/// length limit, so a short content stream can make many runs of nearly [`MAX_RUN_BYTES`] each,
|
||||
/// on every page; past this the rest is not kept and the metadata check fails closed, as for
|
||||
/// any other text it couldn't keep (review on #2539).
|
||||
pub(crate) const MAX_REMOVED_BYTES: usize = 16 * 1024 * 1024;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Mode {
|
||||
@@ -162,6 +170,19 @@ pub(crate) struct Scope<'a> {
|
||||
xobject_oc_cache: HashMap<ObjRef, bool>,
|
||||
/// Content removed from hidden layers (blocks and XObjects).
|
||||
pub layer_blocks: usize,
|
||||
/// Apply: the text of the glyphs removed under the regions, one entry per run of adjacent
|
||||
/// glyphs (#2448), so the document's metadata can be checked for it. Call [`Scope::flush`]
|
||||
/// after processing to close the last run.
|
||||
pub removed_text: Vec<String>,
|
||||
/// Some removed text was not kept (too many runs, or a run too long): the metadata check
|
||||
/// can't search for it and must fail closed.
|
||||
pub removed_cut: bool,
|
||||
/// Bytes of removed text this scope may still keep: what is left of the document's
|
||||
/// [`MAX_REMOVED_BYTES`] (see [`Scope::with_removed_room`]).
|
||||
pub removed_room: usize,
|
||||
/// The run being collected, and where its last glyph ended (device space) with its height.
|
||||
run: String,
|
||||
run_end: Option<((f64, f64), f64)>,
|
||||
/// Collects the glyph runs removed under a region in apply mode (`None` = not collected).
|
||||
/// The redaction proof's snapshot is what sets it: it must know what applying is about to
|
||||
/// remove, by the interpreter's own judgement, without trusting anything else to it.
|
||||
@@ -193,6 +214,11 @@ impl<'a> Scope<'a> {
|
||||
ocmd_cache: HashMap::new(),
|
||||
xobject_oc_cache: HashMap::new(),
|
||||
layer_blocks: 0,
|
||||
removed_text: Vec::new(),
|
||||
removed_cut: false,
|
||||
removed_room: MAX_REMOVED_BYTES,
|
||||
run: String::new(),
|
||||
run_end: None,
|
||||
removed: None,
|
||||
shown: 0,
|
||||
fonts: HashMap::new(),
|
||||
@@ -202,6 +228,49 @@ impl<'a> Scope<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Keep at most `room` bytes of removed text: what earlier pages left of the document's
|
||||
/// [`MAX_REMOVED_BYTES`].
|
||||
pub fn with_removed_room(mut self, room: usize) -> Self {
|
||||
self.removed_room = room;
|
||||
self
|
||||
}
|
||||
|
||||
/// Close the run of removed text being collected.
|
||||
pub fn flush(&mut self) {
|
||||
self.run_end = None;
|
||||
let run = std::mem::take(&mut self.run);
|
||||
if !run.trim().is_empty() {
|
||||
if self.removed_text.len() < MAX_RUNS && run.len() <= self.removed_room {
|
||||
self.removed_room -= run.len();
|
||||
self.removed_text.push(run);
|
||||
} else {
|
||||
self.removed_cut = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Note a removed glyph's text: `origin` and `end` are where it starts and ends in device
|
||||
/// space, `height` its size there. A glyph that starts where the last one ended continues the
|
||||
/// run, one a little further along the line is a new word of it, and anything else starts a
|
||||
/// new run, so separate marks don't join their text.
|
||||
fn removed(&mut self, text: &str, origin: (f64, f64), end: (f64, f64), height: f64) {
|
||||
if let Some((last, h)) = self.run_end {
|
||||
let gap = (origin.0 - last.0).hypot(origin.1 - last.1);
|
||||
let h = h.max(height);
|
||||
if !(gap.is_finite() && h.is_finite()) || gap > 1.5 * h {
|
||||
self.flush();
|
||||
} else if gap > 0.15 * h && !self.run.ends_with(char::is_whitespace) {
|
||||
self.run.push(' ');
|
||||
}
|
||||
}
|
||||
if self.run.len().saturating_add(text.len()) <= MAX_RUN_BYTES {
|
||||
self.run.push_str(text);
|
||||
} else {
|
||||
self.removed_cut = true;
|
||||
}
|
||||
self.run_end = Some((end, height));
|
||||
}
|
||||
|
||||
/// Sanitizing visits every form XObject (not only those under a region).
|
||||
fn everywhere(&self) -> bool {
|
||||
self.hidden_text.is_some() || self.layers_enabled
|
||||
@@ -856,6 +925,26 @@ fn show(scope: &mut Scope<'_>, gs: &Gs, tm: &mut Matrix, items: &[Object]) -> (V
|
||||
!rects.is_empty()
|
||||
}
|
||||
};
|
||||
if scope.mode == Mode::Apply && scope.hidden_text.is_none() {
|
||||
if hit {
|
||||
// The glyph ends after its character and word spacing, in `trm`'s
|
||||
// units (the size is in `trm`), so letter-spaced text stays one word.
|
||||
let spaced = if size != 0.0 { w0 + spacing / size } else { w0 };
|
||||
let end = trm.apply(spaced, 0.0);
|
||||
let top = trm.apply(0.0, 1.0);
|
||||
let height = (top.0 - origin.0).hypot(top.1 - origin.1);
|
||||
let text = f.text_of(code);
|
||||
// A glyph whose text isn't known could stand for anything in the
|
||||
// metadata, so the metadata check can't rely on the text it has
|
||||
// (review on #2539): it fails closed, as for text it couldn't keep.
|
||||
if text.is_none() {
|
||||
scope.removed_cut = true;
|
||||
}
|
||||
scope.removed(text.as_deref().unwrap_or(""), origin, end, height);
|
||||
} else {
|
||||
scope.flush();
|
||||
}
|
||||
}
|
||||
glyphs.push(Glyph { bytes: pos..pos + len, advance, hit, rects });
|
||||
*tm = Matrix::translate(advance * gs.scale, 0.0).then(tm);
|
||||
pos += len;
|
||||
@@ -1001,6 +1090,8 @@ fn xobject(
|
||||
return None;
|
||||
}
|
||||
scope.report.forms_removed += 1;
|
||||
// Its text was never read, so the metadata check can't know what went with it.
|
||||
scope.removed_cut = true;
|
||||
out.replaced.push(name);
|
||||
out.doomed.push(r);
|
||||
return Some(None);
|
||||
@@ -1027,6 +1118,7 @@ fn xobject(
|
||||
}
|
||||
Err(_) if scope.mode == Mode::Apply => {
|
||||
scope.report.forms_removed += 1;
|
||||
scope.removed_cut = true;
|
||||
out.replaced.push(name);
|
||||
out.doomed.push(r);
|
||||
return Some(None);
|
||||
|
||||
@@ -11,6 +11,9 @@
|
||||
//!
|
||||
//! A verification pass then re-reads every redacted page; if any glyph or inline image is still
|
||||
//! under a region the whole operation fails (callers keep the previous document: fail-closed).
|
||||
//! Finally, document metadata (`/Info` entries, XMP) that repeats the removed text is removed.
|
||||
//! Places it leaves as they are (bookmark titles, attachments, remaining comments, other XMP)
|
||||
//! are only counted when they still hold that text ([`Report::still_holding`]).
|
||||
|
||||
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
|
||||
|
||||
@@ -19,8 +22,11 @@ use std::collections::{HashMap, HashSet};
|
||||
use pdfcraft_cos::{Dict, Document, ObjRef, Object, Stream};
|
||||
|
||||
pub mod codes;
|
||||
mod elsewhere;
|
||||
mod identification;
|
||||
mod image;
|
||||
mod interp;
|
||||
mod metadata;
|
||||
mod overlay;
|
||||
mod pattern_guard;
|
||||
pub mod patterns;
|
||||
@@ -35,6 +41,8 @@ pub(crate) mod type3_tables;
|
||||
pub mod user_regex;
|
||||
pub mod verify;
|
||||
|
||||
pub use elsewhere::Elsewhere;
|
||||
pub use identification::{Claim, Identification};
|
||||
use interp::{Mode, Scope, process};
|
||||
pub use pattern_guard::PatternProblem;
|
||||
pub use verify::{Proof, ProofOptions, Snapshot};
|
||||
@@ -178,6 +186,16 @@ pub struct Report {
|
||||
/// only streams a check for text left under the marks may skip: an input stream labelled
|
||||
/// like an overlay (`/PCMark /Redaction`) is checked like any other.
|
||||
pub overlays: Vec<ObjRef>,
|
||||
/// Document information entries and XMP streams removed because they held redacted text.
|
||||
pub metadata: usize,
|
||||
/// What became of the PDF/A and PDF/UA identification when the catalog's XMP packet was
|
||||
/// removed: written back in a packet of its own, or lost (see [`Identification`]).
|
||||
pub identification: Identification,
|
||||
/// Places this apply leaves as they are that still hold the removed text (bookmark titles,
|
||||
/// attachments, the comments that remain, XMP on other objects), counted so the user can be
|
||||
/// told; Remove hidden information removes them. See [`Elsewhere`] for when the counts are
|
||||
/// of every place holding any text.
|
||||
pub still_holding: Elsewhere,
|
||||
}
|
||||
|
||||
pub(crate) fn rect_of(doc: &Document, o: Option<&Object>) -> Option<[f64; 4]> {
|
||||
@@ -498,6 +516,9 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
|
||||
doomed_xobjects: Vec::new(),
|
||||
retired_properties: Vec::new(),
|
||||
widget_owner,
|
||||
removed_text: Vec::new(),
|
||||
removed_cut: false,
|
||||
removed_room: interp::MAX_REMOVED_BYTES,
|
||||
removed_everywhere: HashSet::new(),
|
||||
images,
|
||||
removed_appearances: HashSet::new(),
|
||||
@@ -519,6 +540,8 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
|
||||
doomed_xobjects,
|
||||
retired_properties,
|
||||
mut removed_everywhere,
|
||||
removed_text,
|
||||
removed_cut,
|
||||
mut images,
|
||||
mut removed_appearances,
|
||||
mut appearance_scan_limit,
|
||||
@@ -667,6 +690,12 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
|
||||
// Whatever reference keeps a pruned pattern in the file -- one no walk above follows -- the
|
||||
// full save would write it: prove it safe or refuse.
|
||||
pattern_guard::check_pruned_unreachable(doc, &pruned_patterns, first_page)?;
|
||||
// 6. The document's metadata must not keep what the pages no longer show (#2448).
|
||||
let scrubbed = metadata::scrub(doc, &removed_text, !removed_cut)?;
|
||||
report.metadata = scrubbed.removed;
|
||||
report.identification = scrubbed.identification;
|
||||
// What it doesn't scrub is counted, never changed.
|
||||
report.still_holding = elsewhere::find(doc, &removed_text, !removed_cut);
|
||||
// The previous revision still holds the removed content: the next save must rewrite, and the
|
||||
// rewritten file is a different file from its source, so it gets a new identifier.
|
||||
doc.require_full_save_with_new_id();
|
||||
@@ -752,6 +781,13 @@ struct Applying {
|
||||
retired_properties: Vec<Object>,
|
||||
widget_owner: Vec<(ObjRef, ObjRef)>,
|
||||
removed_everywhere: HashSet<ObjRef>,
|
||||
/// The text of the removed glyphs, for the metadata check at the end.
|
||||
removed_text: Vec<String>,
|
||||
/// Some removed text was too long to keep whole for that check.
|
||||
removed_cut: bool,
|
||||
/// Bytes of removed text later pages may still keep (`interp::MAX_REMOVED_BYTES` for the
|
||||
/// document).
|
||||
removed_room: usize,
|
||||
/// What clearing images may spend, across the run's pages.
|
||||
images: pdfcraft_cos::ImageBudget,
|
||||
/// Appearance streams of the annotations removed, on any page: one shared with a kept
|
||||
@@ -771,14 +807,19 @@ fn redact_page_content(
|
||||
page: &pdfcraft_model::Page,
|
||||
rects: &[[f64; 4]],
|
||||
) -> Result<(), RedactError> {
|
||||
let Applying { report, doomed_xobjects, retired_properties, images, pruned_patterns, .. } = applying;
|
||||
let Applying { report, doomed_xobjects, retired_properties, removed_text, removed_cut, removed_room, images, pruned_patterns, .. } = applying;
|
||||
// 1. Content.
|
||||
let mut budget = pdfcraft_cos::DecodeBudget::page();
|
||||
let (list, data) = page_streams_within(doc, &page.dict, pi, &mut budget)?;
|
||||
let resources = page.dict.get(b"Resources").and_then(|r| doc.resolve(r).as_dict().cloned()).unwrap_or_default();
|
||||
let out = {
|
||||
let mut scope = Scope::new(rects, Mode::Apply, report, budget, images);
|
||||
process(doc, &mut scope, &data, &resources, pdfcraft_content::Matrix::IDENTITY)
|
||||
let mut scope = Scope::new(rects, Mode::Apply, report, budget, images).with_removed_room(*removed_room);
|
||||
let out = process(doc, &mut scope, &data, &resources, pdfcraft_content::Matrix::IDENTITY);
|
||||
scope.flush();
|
||||
removed_text.append(&mut scope.removed_text);
|
||||
*removed_cut |= scope.removed_cut;
|
||||
*removed_room = scope.removed_room;
|
||||
out
|
||||
};
|
||||
// Fail closed: the interpreter lays vertical-writing text out horizontally, so it cannot
|
||||
// tell whether those glyphs lie under a mark. Rather than report success and possibly
|
||||
|
||||
@@ -0,0 +1,547 @@
|
||||
//! Metadata that repeats redacted text (#2448).
|
||||
//!
|
||||
//! [`crate::apply`] removes the glyphs under the marks, but the same words are often in the
|
||||
//! document information dictionary (`/Title`, `/Subject`, `/Keywords`…) and the XMP metadata
|
||||
//! that mirrors it. Once the page no longer shows them, the metadata must not keep them either.
|
||||
//!
|
||||
//! An `/Info` entry whose text contains a redacted term is removed, and so is one whose whole
|
||||
//! value is part of a removed run (an author "J. Li" redacted from a sentence, whose words are too
|
||||
//! short to be terms; the reverse check #2598 also proposed). An XMP stream whose text
|
||||
//! (element content, attribute values other than namespace declarations, comments and
|
||||
//! processing instructions, entities decoded) contains one, or has a value that is part of a
|
||||
//! removed run, is removed whole: editing the XML in place could leave it malformed or keep the
|
||||
//! term in another spelling. A PDF/A identification in that packet goes with it.
|
||||
//!
|
||||
//! Fail-closed: XMP that isn't UTF-8 or UTF-16, or whose filter data is damaged or behind more than
|
||||
//! one filter (it is decoded strictly, so nothing after damage goes unsearched), goes; text too
|
||||
//! large to search against every term counts as matching, and when the removed text itself was cut short (too many runs or terms,
|
||||
//! or a run too long to keep) every `/Info` string and every XMP stream goes, since what was
|
||||
//! never collected can't be searched for. Terms are matched without regard to case or spacing,
|
||||
//! and by their letters and digits alone, so "Blue fin", "BLUE-FIN" and a tracked heading
|
||||
//! collected as "B L U E F I N" all match "Bluefin" (#2539 review).
|
||||
|
||||
use std::collections::HashSet;
|
||||
|
||||
use pdfcraft_cos::{Document, ObjRef, Object};
|
||||
use unicode_normalization::UnicodeNormalization;
|
||||
|
||||
/// Terms longer than this are cut (matching still finds the prefix).
|
||||
const MAX_TERM_CHARS: usize = 4096;
|
||||
const MAX_TERMS: usize = 20_000;
|
||||
/// XMP packets larger than this, decoded, are removed without reading them.
|
||||
pub(crate) const MAX_XMP_BYTES: usize = 16 * 1024 * 1024;
|
||||
/// The searching (text length times terms, or value length times removed text) one scrub may
|
||||
/// do in all; past it every entry and packet counts as matching (fail-closed), so a page of
|
||||
/// removed text against a large packet, or against very many small values, stays quick.
|
||||
const MAX_WORK: usize = 1 << 28;
|
||||
/// Normalized text longer than this many times its input (plus 64 bytes) isn't searched: it
|
||||
/// counts as matching. Ordinary text barely changes length; a few compatibility characters
|
||||
/// (U+FDFA, the square katakana words) decompose to many.
|
||||
const MAX_GROWTH: usize = 4;
|
||||
|
||||
/// `s` in Unicode compatibility form (NFKC), lower case, every run of whitespace made one space,
|
||||
/// trimmed: "Mu\u{308}ller" (decomposed), "ＭÜＬＬＥＲ" (full width) and "Müller" are all
|
||||
/// "müller", and the ligature "fi" is "fi" (#2539 review). `None` when it would grow past
|
||||
/// [`MAX_GROWTH`] times `s`.
|
||||
fn normalize(s: &str) -> Option<String> {
|
||||
let cap = s.len().saturating_mul(MAX_GROWTH).saturating_add(64);
|
||||
let mut out = String::new();
|
||||
let mut space = false;
|
||||
for c in s.nfkc().flat_map(char::to_lowercase).nfkc() {
|
||||
if c.is_whitespace() {
|
||||
space = !out.is_empty();
|
||||
continue;
|
||||
}
|
||||
if space {
|
||||
out.push(' ');
|
||||
space = false;
|
||||
}
|
||||
out.push(c);
|
||||
if out.len() > cap {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
|
||||
fn significant(s: &str, min: usize) -> bool {
|
||||
s.chars().filter(|c| c.is_alphanumeric()).count() >= min
|
||||
}
|
||||
|
||||
/// The letters and digits of `s` alone: "blue-fin" and "b l u e f i n" are both "bluefin".
|
||||
fn squeeze(s: &str) -> String {
|
||||
s.chars().filter(|c| c.is_alphanumeric()).collect()
|
||||
}
|
||||
|
||||
/// A removed run (normalized), and its letters and digits alone with where its words begin and
|
||||
/// end among them, so a value can be found in it whatever its spacing but still as whole words.
|
||||
struct Run {
|
||||
text: String,
|
||||
letters: String,
|
||||
/// `edges[i]`: byte `i` of `letters` begins a word (something other than a letter or digit
|
||||
/// comes before it in the run, or nothing does), or `i` is the end. One entry per byte of
|
||||
/// `letters`, and one for the end.
|
||||
edges: Vec<bool>,
|
||||
}
|
||||
|
||||
impl Run {
|
||||
fn new(text: String) -> Self {
|
||||
let mut letters = String::new();
|
||||
let mut edges = Vec::new();
|
||||
let mut gap = true;
|
||||
for c in text.chars() {
|
||||
if c.is_alphanumeric() {
|
||||
edges.resize(letters.len(), false);
|
||||
edges.push(gap);
|
||||
letters.push(c);
|
||||
gap = false;
|
||||
} else {
|
||||
gap = true;
|
||||
}
|
||||
}
|
||||
edges.resize(letters.len(), false);
|
||||
edges.push(true);
|
||||
Self { text, letters, edges }
|
||||
}
|
||||
|
||||
/// Are the letters and digits `value` (squeezed) among the run's, beginning and ending at
|
||||
/// word edges? "bluefin" is in "b l u e f i n" and "bluef in"; "true" isn't in "untrue".
|
||||
fn holds_letters(&self, value: &str) -> bool {
|
||||
let edge = |i: usize| self.edges.get(i).copied().unwrap_or(false);
|
||||
!value.is_empty() && self.letters.match_indices(value).any(|(at, m)| edge(at) && edge(at.saturating_add(m.len())))
|
||||
}
|
||||
}
|
||||
|
||||
/// The terms to look for: each removed run (at least two letters or digits), and each of its
|
||||
/// words with at least three, so "Smith, John" in `/Author` is caught when "John Smith" was
|
||||
/// redacted. Shorter pieces would match nearly any text. `None` when a run is longer than
|
||||
/// [`MAX_TERM_CHARS`] (or grows past [`MAX_GROWTH`] when normalized) or there are more than
|
||||
/// [`MAX_TERMS`] terms: the caller then can't search
|
||||
/// for all of it and must clear the metadata instead.
|
||||
pub(crate) fn terms(removed: &[String]) -> Option<Vec<String>> {
|
||||
let mut seen = HashSet::new();
|
||||
let mut out = Vec::new();
|
||||
for run in removed {
|
||||
let run = normalize(run)?;
|
||||
if run.chars().nth(MAX_TERM_CHARS).is_some() {
|
||||
return None;
|
||||
}
|
||||
let words = run.split(|c: char| !c.is_alphanumeric()).filter(|w| significant(w, 3)).map(str::to_string).collect::<Vec<_>>();
|
||||
for t in std::iter::once(run.clone()).filter(|r| significant(r, 2)).chain(words) {
|
||||
if seen.insert(t.clone()) {
|
||||
if out.len() >= MAX_TERMS {
|
||||
return None;
|
||||
}
|
||||
out.push(t);
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
|
||||
/// The terms ([`terms`]) with three letters or digits or more, by their letters and digits
|
||||
/// alone, each once: "blue fin" holds the term "bluefin" this way, and "bluefin" the run
|
||||
/// "b l u e f i n".
|
||||
fn squeezed_terms(terms: &[String]) -> Vec<String> {
|
||||
let mut seen = HashSet::new();
|
||||
terms.iter().filter(|t| significant(t, 3)).map(|t| squeeze(t)).filter(|t| seen.insert(t.clone())).collect()
|
||||
}
|
||||
|
||||
/// The searching one scrub may do, across every `/Info` entry and XMP value: past it, each
|
||||
/// further question is answered "yes" (fail-closed), so a packet of many small values can't add
|
||||
/// up to more than one large one.
|
||||
struct Work(std::cell::Cell<usize>);
|
||||
|
||||
impl Work {
|
||||
fn new(limit: usize) -> Self {
|
||||
Self(std::cell::Cell::new(limit))
|
||||
}
|
||||
|
||||
/// Take `cost` from what is left; `false` (and nothing left) when it doesn't fit.
|
||||
fn spend(&self, cost: usize) -> bool {
|
||||
match self.0.get().checked_sub(cost) {
|
||||
Some(left) => {
|
||||
self.0.set(left);
|
||||
true
|
||||
}
|
||||
None => {
|
||||
self.0.set(0);
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Does `text` hold a term, as written or, for `squeezed` terms, by its letters and digits alone?
|
||||
fn mentions(text: &str, terms: &[String], squeezed: &[String], work: &Work) -> bool {
|
||||
let Some(text) = normalize(text) else { return true };
|
||||
let letters = squeeze(&text);
|
||||
!work.spend(text.len().saturating_mul(terms.len().saturating_add(squeezed.len())))
|
||||
|| terms.iter().any(|t| text.contains(t.as_str()))
|
||||
|| squeezed.iter().any(|t| letters.contains(t.as_str()))
|
||||
}
|
||||
|
||||
/// Is `value` itself part of one of the removed `runs`, as whole words, compared as written or by
|
||||
/// letters and digits alone ("Bluefin" is part of "b l u e f i n")? It needs three letters or
|
||||
/// digits, so an entry such as "by" doesn't go with every sentence that contains it, and word
|
||||
/// edges, so an XMP value "True" doesn't go with "untrue". Too much text to search counts as
|
||||
/// part (fail-closed).
|
||||
fn within(value: &str, runs: &[Run], work: &Work) -> bool {
|
||||
let Some(value) = normalize(value) else { return true };
|
||||
if !significant(&value, 3) {
|
||||
return false;
|
||||
}
|
||||
let letters = squeeze(&value);
|
||||
let total = runs.iter().map(|r| r.text.len().saturating_add(r.letters.len())).fold(0usize, usize::saturating_add);
|
||||
let word = |c: Option<char>| c.is_some_and(char::is_alphanumeric);
|
||||
let whole = |r: &Run| {
|
||||
r.holds_letters(&letters)
|
||||
|| r.text.match_indices(value.as_str()).any(|(at, m)| {
|
||||
let before = r.text.get(..at).and_then(|s| s.chars().next_back());
|
||||
let after = r.text.get(at.saturating_add(m.len())..).and_then(|s| s.chars().next());
|
||||
!(word(before) && value.starts_with(char::is_alphanumeric)) && !(word(after) && value.ends_with(char::is_alphanumeric))
|
||||
})
|
||||
};
|
||||
!work.spend(value.len().saturating_mul(total)) || runs.iter().any(whole)
|
||||
}
|
||||
|
||||
/// Decode the five predefined XML entities and character references; anything else stays.
|
||||
fn unescape(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
let mut rest = s;
|
||||
while let Some(at) = rest.find('&') {
|
||||
out.push_str(rest.get(..at).unwrap_or(""));
|
||||
let tail = rest.get(at..).unwrap_or("");
|
||||
// A named entity ends within 12 bytes; looking further made a text of bare `&`s
|
||||
// quadratic. A character reference may carry any number of leading zeros, so its digits
|
||||
// are read to their end: each digit belongs to one `&` only, which keeps it linear.
|
||||
let end = match tail.strip_prefix("&#") {
|
||||
Some(digits) => {
|
||||
let hex = digits.starts_with(['x', 'X']);
|
||||
let from = 2 + usize::from(hex);
|
||||
let run = tail.get(from..).unwrap_or("").bytes().take_while(|b| if hex { b.is_ascii_hexdigit() } else { b.is_ascii_digit() }).count();
|
||||
let e = from + run;
|
||||
(tail.as_bytes().get(e) == Some(&b';')).then_some(e)
|
||||
}
|
||||
None => tail.char_indices().take(13).find(|&(_, c)| c == ';').map(|(e, _)| e),
|
||||
};
|
||||
let entity = end.and_then(|e| tail.get(1..e));
|
||||
let decoded = entity.and_then(|e| match e {
|
||||
"amp" => Some('&'),
|
||||
"lt" => Some('<'),
|
||||
"gt" => Some('>'),
|
||||
"quot" => Some('"'),
|
||||
"apos" => Some('\''),
|
||||
_ => {
|
||||
let n = match e.strip_prefix("#x").or_else(|| e.strip_prefix("#X")) {
|
||||
// Leading zeros dropped, so they can't overflow the number; `�` isn't a
|
||||
// character XML allows, and stays as written.
|
||||
Some(h) => u32::from_str_radix(h.trim_start_matches('0'), 16).ok(),
|
||||
None => e.strip_prefix('#').and_then(|d| d.trim_start_matches('0').parse().ok()),
|
||||
};
|
||||
n.and_then(char::from_u32)
|
||||
}
|
||||
});
|
||||
match (decoded, end) {
|
||||
(Some(c), Some(e)) => {
|
||||
out.push(c);
|
||||
rest = tail.get(e + 1..).unwrap_or("");
|
||||
}
|
||||
_ => {
|
||||
out.push('&');
|
||||
rest = tail.get(1..).unwrap_or("");
|
||||
}
|
||||
}
|
||||
}
|
||||
out.push_str(rest);
|
||||
out
|
||||
}
|
||||
|
||||
/// The text values an XML document carries, entities decoded: element content, attribute values
|
||||
/// (XMP stores simple properties as attributes), CDATA, and the text of comments and processing
|
||||
/// instructions other than the `xpacket` wrapper, which a reader can still see. Names and markup are left out, so a term can't
|
||||
/// match a tag such as `dc:title`, and so are namespace declarations (`xmlns`), whose URIs
|
||||
/// would match words such as "adobe" or "application" in every packet.
|
||||
pub(crate) fn xml_values(xml: &str) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
let mut push = |s: String| {
|
||||
if !s.trim().is_empty() {
|
||||
out.push(s);
|
||||
}
|
||||
};
|
||||
let mut rest = xml;
|
||||
while !rest.is_empty() {
|
||||
let Some(open) = rest.find('<') else {
|
||||
push(unescape(rest));
|
||||
break;
|
||||
};
|
||||
push(unescape(rest.get(..open).unwrap_or("")));
|
||||
let tag = rest.get(open..).unwrap_or("");
|
||||
let delimited = [("<![CDATA[", "]]>"), ("<!--", "-->"), ("<?", "?>")].into_iter().find(|(start, _)| tag.starts_with(start));
|
||||
if let Some((start, stop)) = delimited {
|
||||
let body = tag.get(start.len()..).unwrap_or("");
|
||||
let end = body.find(stop).unwrap_or(body.len());
|
||||
let text = body.get(..end).unwrap_or("");
|
||||
// The packet wrapper's fixed `begin`/`id`/`end` would match words such as "end".
|
||||
if !(start == "<?" && text.split_whitespace().next() == Some("xpacket")) {
|
||||
push(text.to_string());
|
||||
}
|
||||
rest = body.get(end.saturating_add(stop.len())..).unwrap_or("");
|
||||
continue;
|
||||
}
|
||||
// Attribute values inside the tag, up to its closing `>` (which can't be inside quotes).
|
||||
let mut chars = tag.char_indices().skip(1);
|
||||
let mut end = tag.len();
|
||||
// The attribute name being read, and the last one read before whitespace: XML allows
|
||||
// whitespace around `=` (`xmlns:x = "…"`).
|
||||
let mut name = String::new();
|
||||
let mut last = String::new();
|
||||
while let Some((i, c)) = chars.next() {
|
||||
match c {
|
||||
'>' => {
|
||||
end = i + 1;
|
||||
break;
|
||||
}
|
||||
'"' | '\'' => {
|
||||
let mut value = String::new();
|
||||
for (_, d) in chars.by_ref() {
|
||||
if d == c {
|
||||
break;
|
||||
}
|
||||
value.push(d);
|
||||
}
|
||||
let attr = if name.is_empty() { &last } else { &name };
|
||||
let declaration = attr == "xmlns" || attr.starts_with("xmlns:");
|
||||
if !declaration {
|
||||
push(unescape(&value));
|
||||
}
|
||||
name.clear();
|
||||
last.clear();
|
||||
}
|
||||
'=' => {}
|
||||
'/' => {
|
||||
name.clear();
|
||||
last.clear();
|
||||
}
|
||||
c if c.is_whitespace() => {
|
||||
if !name.is_empty() {
|
||||
last = std::mem::take(&mut name);
|
||||
}
|
||||
}
|
||||
c => name.push(c),
|
||||
}
|
||||
}
|
||||
rest = tag.get(end..).unwrap_or("");
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Decode an XMP packet: UTF-8 (with or without a byte-order mark), or UTF-16 told by its mark
|
||||
/// or, as a packet usually begins `<?xpacket` with the mark inside the attribute, by the zero
|
||||
/// byte of its first character. `None` for anything else or any invalid sequence, so text that
|
||||
/// can't be read faithfully is never searched lossily (the caller removes it).
|
||||
pub(crate) fn decode_xmp(bytes: &[u8]) -> Option<String> {
|
||||
let utf16 = |body: &[u8], be: bool| -> Option<String> {
|
||||
let (units, rest) = body.as_chunks::<2>();
|
||||
if !rest.is_empty() {
|
||||
return None;
|
||||
}
|
||||
char::decode_utf16(units.iter().map(|&p| if be { u16::from_be_bytes(p) } else { u16::from_le_bytes(p) })).collect::<Result<String, _>>().ok()
|
||||
};
|
||||
match bytes {
|
||||
[0xEF, 0xBB, 0xBF, rest @ ..] => std::str::from_utf8(rest).ok().map(str::to_string),
|
||||
[0xFE, 0xFF, rest @ ..] => utf16(rest, true),
|
||||
[0xFF, 0xFE, rest @ ..] => utf16(rest, false),
|
||||
[0, a, ..] if *a != 0 => utf16(bytes, true),
|
||||
[a, 0, ..] if *a != 0 => utf16(bytes, false),
|
||||
_ => std::str::from_utf8(bytes).ok().map(str::to_string),
|
||||
}
|
||||
}
|
||||
|
||||
/// The text of a text string, as searched for the removed text. ISO 32000-2 §7.9.2.2.1 defines
|
||||
/// PDFDocEncoding, UTF-16BE (`FE FF`) and UTF-8 (`EF BB BF`) only, but a string behind an `FF FE`
|
||||
/// mark is UTF-16LE to any reader that takes it as such, and read through PDFDocEncoding it
|
||||
/// could never match: it is read as UTF-16LE (an odd last byte is left out, a lone surrogate
|
||||
/// becomes U+FFFD), so the search stays fail-closed (#2539 review).
|
||||
pub(crate) fn text_string(s: &pdfcraft_cos::PdfString) -> String {
|
||||
match s.bytes.as_slice() {
|
||||
[0xFF, 0xFE, rest @ ..] => {
|
||||
let units = rest.as_chunks::<2>().0.iter().map(|&p| u16::from_le_bytes(p));
|
||||
char::decode_utf16(units).map(|c| c.unwrap_or(char::REPLACEMENT_CHARACTER)).collect()
|
||||
}
|
||||
_ => s.to_text(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The text values of an XMP stream, or `None` when it can't be read.
|
||||
fn xmp_values(doc: &Document, o: &Object) -> Option<Vec<String>> {
|
||||
xmp_text(doc, o).map(|t| xml_values(&t))
|
||||
}
|
||||
|
||||
/// The decoded text of an XMP stream, or `None` when it can't be read.
|
||||
pub(crate) fn xmp_text(doc: &Document, o: &Object) -> Option<String> {
|
||||
let Object::Stream(s) = &*doc.resolve(o) else { return None };
|
||||
// Decoded strictly: a tolerant decode returns whatever came before the damage, and the part
|
||||
// after it would go unsearched. A chain of more than one filter has no strict decoder here,
|
||||
// so it isn't read either (fail-closed; XMP is rarely filtered at all). Decoding stops at the
|
||||
// cap, so a small stream that inflates hugely costs no more memory.
|
||||
if s.filters().len() > 1 {
|
||||
return None;
|
||||
}
|
||||
// `/DecodeParms` given by reference are read too ([`Document::with_direct_parms`]).
|
||||
let bytes = doc.with_direct_parms(s).decoded_first_filter_strict(MAX_XMP_BYTES).ok()?;
|
||||
decode_xmp(&bytes)
|
||||
}
|
||||
|
||||
/// What the removed text is looked for with, and the searching left to do it.
|
||||
pub(crate) struct Search {
|
||||
lookup: Lookup,
|
||||
work: Work,
|
||||
}
|
||||
|
||||
enum Lookup {
|
||||
/// Nothing with text was removed: nothing can hold it.
|
||||
Nothing,
|
||||
/// The removed text couldn't be collected or searched for in full: everything counts as
|
||||
/// holding it (fail-closed).
|
||||
Everything,
|
||||
/// The terms ([`terms`]), the same by their letters and digits alone ([`squeezed_terms`]),
|
||||
/// and the distinct runs, normalized.
|
||||
Terms(Vec<String>, Vec<String>, Vec<Run>),
|
||||
}
|
||||
|
||||
impl Search {
|
||||
/// The search for `removed` text, with `limit` steps of searching ([`MAX_WORK`] outside
|
||||
/// tests). `complete` is false when the collection of removed text was cut short; then, as
|
||||
/// when the terms exceed their limits, everything matches.
|
||||
pub(crate) fn new(removed: &[String], complete: bool, limit: usize) -> Self {
|
||||
let runs: Option<Vec<String>> = removed.iter().map(|r| normalize(r)).collect();
|
||||
let lookup = match (terms(removed).filter(|_| complete), runs) {
|
||||
(Some(t), _) if t.is_empty() => Lookup::Nothing,
|
||||
(Some(t), Some(runs)) => {
|
||||
// The same run removed many times is searched for once.
|
||||
let mut seen = HashSet::new();
|
||||
let runs: Vec<Run> = runs.into_iter().filter(|r| seen.insert(r.clone())).map(Run::new).collect();
|
||||
let squeezed = squeezed_terms(&t);
|
||||
Lookup::Terms(t, squeezed, runs)
|
||||
}
|
||||
_ => Lookup::Everything,
|
||||
};
|
||||
Self { lookup, work: Work::new(limit) }
|
||||
}
|
||||
|
||||
/// [`Search::new`] with the whole work limit.
|
||||
pub(crate) fn of(removed: &[String], complete: bool) -> Self {
|
||||
Self::new(removed, complete, MAX_WORK)
|
||||
}
|
||||
|
||||
/// Nothing with text was removed.
|
||||
pub(crate) fn is_nothing(&self) -> bool {
|
||||
matches!(self.lookup, Lookup::Nothing)
|
||||
}
|
||||
|
||||
/// The removed text couldn't be searched for: everything matches.
|
||||
pub(crate) fn is_everything(&self) -> bool {
|
||||
matches!(self.lookup, Lookup::Everything)
|
||||
}
|
||||
|
||||
/// Does `text` hold the removed text: a term, or wholly part of a removed run? Past the
|
||||
/// work limit, yes (fail-closed).
|
||||
pub(crate) fn text(&self, text: &str) -> bool {
|
||||
match &self.lookup {
|
||||
Lookup::Nothing => false,
|
||||
Lookup::Everything => true,
|
||||
Lookup::Terms(terms, squeezed, runs) => mentions(text, terms, squeezed, &self.work) || within(text, runs, &self.work),
|
||||
}
|
||||
}
|
||||
|
||||
/// Does the XMP stream `o` hold the removed text? A stream that can't be read does
|
||||
/// (fail-closed), unless nothing with text was removed.
|
||||
pub(crate) fn xmp(&self, doc: &Document, o: &Object) -> bool {
|
||||
match &self.lookup {
|
||||
Lookup::Nothing => false,
|
||||
Lookup::Everything => true,
|
||||
Lookup::Terms(terms, squeezed, runs) => xmp_values(doc, o)
|
||||
.is_none_or(|values| mentions(&values.join(" "), terms, squeezed, &self.work) || values.iter().any(|v| within(v, runs, &self.work))),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What [`scrub`] did.
|
||||
#[derive(Debug, Default)]
|
||||
pub(crate) struct Scrubbed {
|
||||
/// `/Info` entries and XMP streams removed.
|
||||
pub(crate) removed: usize,
|
||||
/// What became of the PDF/A and PDF/UA identification of a removed catalog packet.
|
||||
pub(crate) identification: crate::Identification,
|
||||
}
|
||||
|
||||
/// Remove the metadata that still holds `removed` text: matching `/Info` entries, and the
|
||||
/// catalog's and pages' XMP streams that mention it. `complete` is false when the collection of
|
||||
/// removed text was cut short; then, as when the terms themselves exceed their limits, every
|
||||
/// `/Info` string and every XMP stream goes. A removed catalog packet's PDF/A and PDF/UA
|
||||
/// identification is written back in a packet of its own ([`crate::identification`]).
|
||||
pub(crate) fn scrub(doc: &mut Document, removed: &[String], complete: bool) -> Result<Scrubbed, pdfcraft_cos::CosError> {
|
||||
scrub_within(doc, removed, complete, MAX_WORK)
|
||||
}
|
||||
|
||||
/// [`scrub`] with the searching it may do given, so tests can run out of it.
|
||||
pub(crate) fn scrub_within(doc: &mut Document, removed: &[String], complete: bool, limit: usize) -> Result<Scrubbed, pdfcraft_cos::CosError> {
|
||||
let search = Search::new(removed, complete, limit);
|
||||
let mut out = Scrubbed::default();
|
||||
if search.is_nothing() {
|
||||
return Ok(out);
|
||||
}
|
||||
let mut n = 0;
|
||||
|
||||
// The information dictionary, direct in the trailer or an object of its own.
|
||||
let info_obj = doc.trailer().get(b"Info").cloned();
|
||||
if let Some(info) = info_obj.as_ref().and_then(|i| doc.resolve(i).as_dict().cloned()) {
|
||||
let doomed: Vec<Vec<u8>> =
|
||||
info.iter().filter(|(_, v)| doc.resolve(v).as_string().is_some_and(|s| search.text(&text_string(s)))).map(|(k, _)| k.to_vec()).collect();
|
||||
if !doomed.is_empty() {
|
||||
n += doomed.len();
|
||||
let mut kept = info.clone();
|
||||
for k in &doomed {
|
||||
kept.remove(k);
|
||||
}
|
||||
match info_obj.as_ref().and_then(Object::as_ref) {
|
||||
Some(r) => doc.update_dict(r, |d| *d = kept)?,
|
||||
None => doc.trailer_mut().set(b"Info".to_vec(), Object::Dict(kept)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// XMP on the catalog and on every page.
|
||||
let root = doc.root();
|
||||
for holder in catalog_and_pages(doc) {
|
||||
let Some(xmp) = doc.get(holder).as_dict().and_then(|d| d.get(b"Metadata").cloned()) else { continue };
|
||||
if search.xmp(doc, &xmp) {
|
||||
// The catalog's packet says whether the file conforms to PDF/A or PDF/UA: that much
|
||||
// is written back, in a packet of its own.
|
||||
let replacement = if Some(holder) == root {
|
||||
let (id, packet) = crate::identification::keep(doc, &xmp, &search);
|
||||
out.identification = id;
|
||||
packet.map(|p| doc.add(Object::Stream(p)))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
doc.update_dict(holder, |d| match replacement {
|
||||
Some(r) => d.set(b"Metadata".to_vec(), Object::Ref(r)),
|
||||
None => {
|
||||
d.remove(b"Metadata");
|
||||
}
|
||||
})?;
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
out.removed = n;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// The catalog and the pages: the objects whose XMP [`scrub`] checks.
|
||||
pub(crate) fn catalog_and_pages(doc: &Document) -> Vec<ObjRef> {
|
||||
let mut holders: Vec<ObjRef> = doc.root().into_iter().collect();
|
||||
holders.extend(pdfcraft_model::pages(doc).iter().map(|p| p.obj));
|
||||
holders
|
||||
}
|
||||
@@ -419,6 +419,47 @@ fn strip_owned(o: &mut Object, role: Held, keys: &[&[u8]], depth: usize) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Every value of `key` held outside a name map, as [`owners`] finds them: with the indirect
|
||||
/// object that holds it and whether it is that object's own entry (rather than one in a
|
||||
/// dictionary written inside it). Fails as [`owners`] does for a graph nested too deeply.
|
||||
pub(crate) fn owned_values(doc: &Document, key: &[u8]) -> Result<Vec<(ObjRef, bool, Object)>, RedactError> {
|
||||
let (_, holders) = owners(doc, &[key])?;
|
||||
let mut out = Vec::new();
|
||||
for (r, role) in holders {
|
||||
collect_owned(&doc.get(r), role, key, 0, &mut |own, v| out.push((r, own, v.clone())));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Calls `found` with each value of `key` in `o` and what it holds in place, outside name maps
|
||||
/// (the reading counterpart of [`strip_owned`]), and whether it is `o`'s own entry.
|
||||
fn collect_owned(o: &Object, role: Held, key: &[u8], depth: usize, found: &mut dyn FnMut(bool, &Object)) {
|
||||
if depth > MAX_DIRECT_DEPTH {
|
||||
return;
|
||||
}
|
||||
let (d, role) = match o {
|
||||
Object::Array(a) => {
|
||||
for v in a {
|
||||
collect_owned(v, Held::Owner, key, depth + 1, found);
|
||||
}
|
||||
return;
|
||||
}
|
||||
Object::Dict(d) => (d, role),
|
||||
Object::Stream(s) => (&s.dict, Held::Owner),
|
||||
_ => return,
|
||||
};
|
||||
if role != Held::Map
|
||||
&& let Some(v) = d.get(key)
|
||||
{
|
||||
found(depth == 0, v);
|
||||
}
|
||||
for (k, v) in d.iter() {
|
||||
if matches!(v, Object::Array(_) | Object::Dict(_)) {
|
||||
collect_owned(v, role.child(k), key, depth + 1, found);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A comment as the comments list sees it, less file attachments (removed with Attachments) and
|
||||
/// redaction marks.
|
||||
fn is_comment(subtype: &[u8]) -> bool {
|
||||
|
||||
@@ -252,6 +252,406 @@ fn a_whole_line_under_a_mark_goes_even_without_widths() {
|
||||
assert_eq!(under(&mut doc, 0, &[[0.0, 0.0, 300.0, 300.0]]), 0, "{}", content(&doc, 0));
|
||||
}
|
||||
|
||||
/// Issue #2448: the redacted words are often in the document information dictionary and the XMP
|
||||
/// metadata too. Applying removes the entries and streams that repeat them, and keeps the rest.
|
||||
#[test]
|
||||
fn redacted_text_does_not_survive_in_info_or_xmp() {
|
||||
let xmp = |body: &str| {
|
||||
stream(
|
||||
"/Type /Metadata /Subtype /XML",
|
||||
format!(
|
||||
r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">{body}</rdf:RDF></x:xmpmeta>"#
|
||||
)
|
||||
.as_bytes(),
|
||||
)
|
||||
};
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Metadata 9 0 R >>".to_vec(),
|
||||
// "BLUEFIN" spans x 55-90.
|
||||
stream("", b"BT /F1 10 Tf 10 100 Td (Codename BLUEFIN report) Tj ET"),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
// 7: the document's XMP names it in an attribute, through a character reference.
|
||||
xmp(r#"<rdf:Description rdf:about="" xmlns:pdf="http://ns.adobe.com/pdf/1.3/" pdf:Keywords="tuna, BlueFin"/>"#),
|
||||
// 8: the information dictionary.
|
||||
b"<< /Title (Codename BLUEFIN report) /Keywords (tuna, bluefin) /Author (Ada Lovelace) /Producer (Test) >>".to_vec(),
|
||||
// 9: the page's XMP doesn't mention it.
|
||||
xmp(r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Annual figures</dc:title></rdf:Description>"#),
|
||||
];
|
||||
let mut doc = Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 8 0 R").build())).unwrap();
|
||||
mark(&mut doc, 0, &[[55.0, 95.0, 90.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!((r.glyphs, r.metadata), (7, 3), "{r:?}");
|
||||
// Without object streams, so the byte search below can see the `/Info` dictionary.
|
||||
let bytes = write_full(&doc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap();
|
||||
let has = |needle: &[u8]| bytes.windows(needle.len()).any(|w| w.eq_ignore_ascii_case(needle));
|
||||
assert!(has(b"Ada Lovelace"), "the kept /Info entry is visible to the search");
|
||||
assert!(!has(b"bluefin") && !has(b"BlueFin"), "the redacted word survived in the metadata");
|
||||
let saved = Document::open(Arc::new(bytes)).unwrap();
|
||||
let info = saved.trailer().get(b"Info").and_then(|i| saved.resolve(i).as_dict().cloned()).unwrap();
|
||||
let text = |k: &[u8]| info.get(k).and_then(|v| saved.resolve(v).as_string().map(|s| s.to_text()));
|
||||
assert_eq!((text(b"Title"), text(b"Keywords")), (None, None));
|
||||
assert_eq!((text(b"Author").as_deref(), text(b"Producer").as_deref()), (Some("Ada Lovelace"), Some("Test")));
|
||||
let catalog = saved.get(saved.root().unwrap()).as_dict().cloned().unwrap();
|
||||
assert!(!catalog.contains(b"Metadata"), "the document XMP named the redacted word");
|
||||
assert!(pdfcraft_model::pages(&saved)[0].dict.contains(b"Metadata"), "the page XMP didn't, and stays");
|
||||
}
|
||||
|
||||
/// An `/Info` value that is itself part of the removed text goes, though its words are too short
|
||||
/// to be terms ("J. Li"); a value with fewer than three letters or digits stays.
|
||||
#[test]
|
||||
fn an_info_value_inside_the_redacted_text_goes() {
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>".to_vec(),
|
||||
stream("", b"BT /F1 10 Tf 10 100 Td (Prepared by J. Li) Tj ET"),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
b"<< /Author (J. li) /Subject (by) /Producer (Test) >>".to_vec(),
|
||||
];
|
||||
let mut doc = Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 7 0 R").build())).unwrap();
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.metadata, 1, "{r:?}");
|
||||
let info = doc.trailer().get(b"Info").and_then(|i| doc.resolve(i).as_dict().cloned()).unwrap();
|
||||
assert!(!info.contains(b"Author"), "the author's name was redacted from the page");
|
||||
assert!(info.contains(b"Subject") && info.contains(b"Producer"));
|
||||
}
|
||||
|
||||
/// Removed glyphs whose text isn't known (no `/ToUnicode`, glyph names outside the glyph list)
|
||||
/// leave nothing to search the metadata for, so every `/Info` string goes, as when the collected
|
||||
/// text was cut short (review on #2539). Glyphs whose text is known keep the targeted check.
|
||||
#[test]
|
||||
fn removed_glyphs_with_no_known_text_clear_the_metadata() {
|
||||
let page = |codes: &[u8], font: &str| -> Document {
|
||||
let content = [&b"BT /F1 10 Tf 10 100 Td ("[..], codes, b") Tj ET"].concat();
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>".to_vec(),
|
||||
stream("", &content),
|
||||
font.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
b"<< /Title (Project Nightjar) /Producer (Test) >>".to_vec(),
|
||||
];
|
||||
Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 7 0 R").build())).unwrap()
|
||||
};
|
||||
let unnamed = FONT.replace(" /Widths", " /Encoding << /Differences [65 /g17 /g18 /g19] >> /Widths");
|
||||
let mut doc = page(b"ABC", &unnamed);
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
let info = doc.trailer().get(b"Info").and_then(|i| doc.resolve(i).as_dict().cloned()).unwrap();
|
||||
assert!(!info.contains(b"Title") && !info.contains(b"Producer"), "fail closed: {r:?}");
|
||||
// Control: the same glyphs with known text don't touch unrelated metadata.
|
||||
let mut doc = page(b"ABC", FONT);
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
apply(&mut doc, None).unwrap();
|
||||
let info = doc.trailer().get(b"Info").and_then(|i| doc.resolve(i).as_dict().cloned()).unwrap();
|
||||
assert!(info.contains(b"Title") && info.contains(b"Producer"));
|
||||
}
|
||||
|
||||
/// One page showing `content`, with `/Info` `info` and catalog XMP whose `rdf:RDF` holds
|
||||
/// `xmp_body`.
|
||||
fn with_metadata(content: &[u8], info: &str, xmp_body: &str) -> Document {
|
||||
let xmp = format!(
|
||||
r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">{xmp_body}</rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#
|
||||
);
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>".to_vec(),
|
||||
stream("", content),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
stream("/Type /Metadata /Subtype /XML", xmp.as_bytes()),
|
||||
info.as_bytes().to_vec(),
|
||||
];
|
||||
Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 8 0 R").build())).unwrap()
|
||||
}
|
||||
|
||||
fn info_keys(doc: &Document) -> Vec<String> {
|
||||
let info = doc.trailer().get(b"Info").and_then(|i| doc.resolve(i).as_dict().cloned()).unwrap();
|
||||
info.iter().map(|(k, _)| String::from_utf8_lossy(k).into_owned()).collect()
|
||||
}
|
||||
|
||||
fn has_xmp(doc: &Document) -> bool {
|
||||
doc.get(doc.root().unwrap()).as_dict().is_some_and(|c| c.contains(b"Metadata"))
|
||||
}
|
||||
|
||||
/// Letter-spaced text (character spacing wider than a word gap between plain glyphs) is still
|
||||
/// one word, so `/Title (Bluefin)` goes when "B L U E F I N" is redacted (#2539 review).
|
||||
#[test]
|
||||
fn letter_spaced_redacted_text_still_matches_metadata() {
|
||||
let body = r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Bluefin</dc:title></rdf:Description>"#;
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 3 Tc 10 100 Td (BLUEFIN) Tj ET", "<< /Title (Bluefin) /Producer (Test) >>", body);
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.metadata, 2, "{r:?}");
|
||||
assert_eq!(info_keys(&doc), ["Producer"]);
|
||||
assert!(!has_xmp(&doc));
|
||||
}
|
||||
|
||||
/// An XMP value that is part of the removed text goes, as an `/Info` value does, though its
|
||||
/// words are too short to be terms ("J. Li"). Values are compared as whole words, so "True"
|
||||
/// doesn't go with "untrue", and namespace URIs and the `xpacket` wrapper aren't searched, so
|
||||
/// redacting "Adobe … end" keeps a packet declared under `adobe:ns:meta/` and closed by
|
||||
/// `<?xpacket end="w"?>`.
|
||||
#[test]
|
||||
fn xmp_values_inside_the_redacted_text_go_as_whole_words() {
|
||||
let creator = r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:creator>J. Li</dc:creator></rdf:Description>"#;
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 10 100 Td (Prepared by J. Li) Tj ET", "<< /Producer (Test) >>", creator);
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.metadata, 1, "{r:?}");
|
||||
assert!(!has_xmp(&doc), "the packet named J. Li");
|
||||
|
||||
let trapped = r#"<rdf:Description rdf:about="" xmlns:pdf="http://ns.adobe.com/pdf/1.3/" pdf:Trapped="True"/>"#;
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 10 100 Td (Adobe said untrue at the end) Tj ET", "<< /Subject (True) /Producer (Test) >>", trapped);
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.metadata, 0, "{r:?}");
|
||||
assert!(has_xmp(&doc));
|
||||
assert_eq!(info_keys(&doc), ["Subject", "Producer"]);
|
||||
}
|
||||
|
||||
/// When the removed text was too much to collect, what wasn't collected can't be searched for:
|
||||
/// every `/Info` string and the XMP go (fail-closed), even though the runs that were kept match
|
||||
/// nothing.
|
||||
#[test]
|
||||
fn metadata_is_cleared_when_the_removed_text_was_cut_short() {
|
||||
// More separate runs of one letter than are kept: each `A` is too far from the last to join.
|
||||
let mut content = b"BT /F1 10 Tf 10 100 Td".to_vec();
|
||||
for _ in 0..50_001 {
|
||||
content.extend_from_slice(b" (A) Tj 30 0 Td (A) Tj -30 0 Td");
|
||||
}
|
||||
content.extend_from_slice(b" ET");
|
||||
let body = r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Annual figures</dc:title></rdf:Description>"#;
|
||||
let mut doc = with_metadata(&content, "<< /Title (Annual figures) /Producer (Test) /Trapped /False >>", body);
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.metadata, 3, "{r:?}");
|
||||
assert_eq!(info_keys(&doc), ["Trapped"], "strings go, the name stays");
|
||||
assert!(!has_xmp(&doc));
|
||||
}
|
||||
|
||||
/// #2539 review: the removed text kept for the metadata check is bounded for the whole document,
|
||||
/// not per page. One glyph code can stand for 256 UTF-16 units of `/ToUnicode` text (768 bytes of
|
||||
/// UTF-8), so here each page makes 150 runs of 80 such glyphs (9.2 MB, under the page's run and
|
||||
/// byte limits); two pages pass `MAX_REMOVED_BYTES`, so the rest isn't kept and the check fails
|
||||
/// closed.
|
||||
#[test]
|
||||
fn removed_text_kept_for_the_metadata_check_is_bounded_per_document() {
|
||||
let long = "4E00".repeat(256);
|
||||
let to_unicode = format!("/CIDInit /ProcSet findresource begin 1 beginbfchar <41> <{long}> endbfchar end");
|
||||
let run = "A".repeat(80);
|
||||
let mut content = b"BT /F1 1 Tf 10 100 Td".to_vec();
|
||||
for _ in 0..75 {
|
||||
content.extend_from_slice(format!(" ({run}) Tj 30 0 Td ({run}) Tj -30 0 Td").as_bytes());
|
||||
}
|
||||
content.extend_from_slice(b" ET");
|
||||
let page = "/Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >>";
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R 8 0 R] /Count 2 >>".to_vec(),
|
||||
format!("<< {page} >>").into_bytes(),
|
||||
stream("", &content),
|
||||
FONT.replace("95 0 R", "6 0 R").replace(" /Widths", " /ToUnicode 7 0 R /Widths").into_bytes(),
|
||||
widths(),
|
||||
stream("", to_unicode.as_bytes()),
|
||||
format!("<< {page} >>").into_bytes(),
|
||||
];
|
||||
let mut doc = pdf(objs);
|
||||
let mut applying = Applying {
|
||||
report: Report::default(),
|
||||
doomed_fields: Vec::new(),
|
||||
doomed_xobjects: Vec::new(),
|
||||
retired_properties: Vec::new(),
|
||||
widget_owner: Vec::new(),
|
||||
removed_text: Vec::new(),
|
||||
removed_cut: false,
|
||||
removed_room: interp::MAX_REMOVED_BYTES,
|
||||
removed_everywhere: HashSet::new(),
|
||||
images: pdfcraft_cos::ImageBudget::new(&doc),
|
||||
removed_appearances: HashSet::new(),
|
||||
appearance_scan_limit: false,
|
||||
pruned_patterns: Vec::new(),
|
||||
};
|
||||
let rects = [[5.0, 99.0, 200.0, 102.0]];
|
||||
let kept = |a: &Applying| a.removed_text.iter().map(String::len).sum::<usize>();
|
||||
let first = pdfcraft_model::pages(&doc).swap_remove(0);
|
||||
redact_page_content(&mut doc, &mut applying, 0, &first, &rects).unwrap();
|
||||
assert_eq!((kept(&applying), applying.removed_cut), (9_216_000, false), "one page fits");
|
||||
let second = pdfcraft_model::pages(&doc).swap_remove(1);
|
||||
redact_page_content(&mut doc, &mut applying, 1, &second, &rects).unwrap();
|
||||
assert!(applying.removed_cut, "the second page passes the document's limit");
|
||||
assert!(kept(&applying) <= interp::MAX_REMOVED_BYTES, "kept {} bytes", kept(&applying));
|
||||
}
|
||||
|
||||
/// #2539 review: the searching a scrub may do is one budget for all of it, not a check per
|
||||
/// value: very many small `/Info` values each fit, but together they don't, and past the budget
|
||||
/// the rest counts as matching (fail-closed). With room to search, nothing that doesn't mention
|
||||
/// the removed text goes.
|
||||
#[test]
|
||||
fn metadata_search_has_one_budget_for_every_value() {
|
||||
let entries: String = (0..50).map(|i| format!("/K{i} (an unrelated value number {i:02} in the list) ")).collect();
|
||||
let info = format!("<< {entries}/Producer (Test) >>");
|
||||
let fresh = || with_metadata(b"BT /F1 10 Tf 10 100 Td (Bluefin) Tj ET", &info, "");
|
||||
let mut doc = fresh();
|
||||
assert_eq!(metadata::scrub_within(&mut doc, &["Bluefin".into()], true, 1 << 28).unwrap().removed, 0, "room to search: nothing goes");
|
||||
let mut doc = fresh();
|
||||
// Each value costs about 40; fifty of them don't fit in 1000.
|
||||
let n = metadata::scrub_within(&mut doc, &["Bluefin".into()], true, 1000).unwrap().removed;
|
||||
assert!(n > 20, "past the budget the rest goes ({n} went)");
|
||||
}
|
||||
|
||||
/// #2539 review: an entity is looked for only in the 12 bytes after its `&`, so a packet of bare
|
||||
/// `&` characters is read in linear time, and decoding is unchanged.
|
||||
#[test]
|
||||
fn xml_entities_are_found_near_their_ampersand_only() {
|
||||
assert_eq!(metadata::xml_values("<a>Résumé & AA �</a>"), ["Résumé & AA �"]);
|
||||
// #2539 review: a character reference with many leading zeros still decodes, so it can't
|
||||
// hide a redacted word from the search; a reference that never ends stays as written.
|
||||
assert_eq!(metadata::xml_values("<a>BLUEFIN BLUEFIN �</a>"), ["BLUEFIN BLUEFIN �"]);
|
||||
let bare = "&".repeat(200_000);
|
||||
let started = std::time::Instant::now();
|
||||
let values = metadata::xml_values(&format!("<a>{bare}</a>"));
|
||||
assert_eq!(values.concat().len(), bare.len());
|
||||
let zeros = format!("&#{}", "0".repeat(10)).repeat(20_000);
|
||||
assert_eq!(metadata::xml_values(&format!("<a>{zeros}</a>")).concat().len(), zeros.len());
|
||||
eprintln!("200000 bare ampersands, 20000 unended references: {:?}", started.elapsed());
|
||||
}
|
||||
|
||||
/// #2539 review: Apply removes a form XObject whole when it is nested past the depth limit (or
|
||||
/// its stream can't be decoded), without reading its text, so what it held is unknown: the
|
||||
/// metadata check fails closed.
|
||||
#[test]
|
||||
fn a_form_removed_unread_clears_the_metadata() {
|
||||
// Forms 5, 6, ... each draw the next, one more than the depth limit deep, and the last one
|
||||
// paints. The page draws form 5.
|
||||
let last = 5 + crate::interp::MAX_DEPTH;
|
||||
let mut objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /XObject << /Fm 5 0 R >> >> >>".to_vec(),
|
||||
stream("", b"q /Fm Do Q"),
|
||||
];
|
||||
for n in 5..=last {
|
||||
objs.push(stream(
|
||||
&format!("/Type /XObject /Subtype /Form /BBox [0 0 300 300] /Resources << /XObject << /Fm {} 0 R >> >>", n + 1),
|
||||
b"q /Fm Do Q",
|
||||
));
|
||||
}
|
||||
objs.push(stream("/Type /XObject /Subtype /Form /BBox [0 0 300 300]", b"0 g 10 10 50 50 re f"));
|
||||
objs.push(b"<< /Title (Annual figures) /Producer (Test) >>".to_vec());
|
||||
let info = objs.len();
|
||||
let mut doc = Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry(&format!("/Info {info} 0 R")).build())).unwrap();
|
||||
mark(&mut doc, 0, &[[0.0, 0.0, 300.0, 300.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert!(r.forms_removed >= 1, "{r:?}");
|
||||
assert!(!info_keys(&doc).contains(&"Title".to_string()), "the deep form's text was never read: {:?}", info_keys(&doc));
|
||||
}
|
||||
|
||||
/// A value shorter than the redacted sentence goes from XMP too, in a custom schema's element or
|
||||
/// attribute, as from `/Info` (the regression #2598 suggested keeping): the packets holding it are
|
||||
/// removed, one that doesn't stays.
|
||||
#[test]
|
||||
fn a_shorter_value_in_custom_xmp_goes() {
|
||||
let xmp = |body: &str| {
|
||||
stream(
|
||||
"/Type /Metadata /Subtype /XML",
|
||||
format!(
|
||||
r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">{body}</rdf:RDF></x:xmpmeta>"#
|
||||
)
|
||||
.as_bytes(),
|
||||
)
|
||||
};
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Metadata 8 0 R >>".to_vec(),
|
||||
stream("", b"BT /F1 10 Tf 10 100 Td (JAMES BOND IS A SPY) Tj ET"),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
// 7: an element of a custom schema.
|
||||
xmp(r#"<rdf:Description rdf:about="" xmlns:acme="https://example.org/acme/"><acme:Agent>James Bond</acme:Agent></rdf:Description>"#),
|
||||
// 8: an attribute of a custom schema.
|
||||
xmp(r#"<rdf:Description rdf:about="" xmlns:acme="https://example.org/acme/" acme:code="James Bond"/>"#),
|
||||
// 9: a third packet, on no holder yet; swapped onto the page below to show a clean one stays.
|
||||
xmp(r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Annual figures</dc:title></rdf:Description>"#),
|
||||
b"<< /Author (James Bond) /Subject (Annual figures) >>".to_vec(),
|
||||
];
|
||||
for (holder_of_8, expected) in [("8 0 R", 3), ("9 0 R", 2)] {
|
||||
let objs: Vec<Vec<u8>> =
|
||||
objs.iter().map(|o| String::from_utf8_lossy(o).replace("/Metadata 8 0 R", &format!("/Metadata {holder_of_8}")).into_bytes()).collect();
|
||||
let mut doc = Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 10 0 R").build())).unwrap();
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.metadata, expected, "{holder_of_8}: {r:?}");
|
||||
let catalog = doc.get(doc.root().unwrap()).as_dict().cloned().unwrap();
|
||||
assert!(!catalog.contains(b"Metadata"), "the custom element named him");
|
||||
let page_keeps = pdfcraft_model::pages(&doc)[0].dict.contains(b"Metadata");
|
||||
assert_eq!(page_keeps, holder_of_8 == "9 0 R", "the attribute packet goes, a clean one stays");
|
||||
let info = doc.trailer().get(b"Info").and_then(|i| doc.resolve(i).as_dict().cloned()).unwrap();
|
||||
assert!(!info.contains(b"Author") && info.contains(b"Subject"), "the author goes, the unrelated subject stays");
|
||||
}
|
||||
}
|
||||
|
||||
/// The removed text is collected in runs by position, not by operator: glyphs shown one operator
|
||||
/// at a time still make one word, a gap makes a space, and separate marks make separate runs.
|
||||
#[test]
|
||||
fn removed_text_is_collected_in_runs() {
|
||||
let mut doc = one_page(
|
||||
b"BT /F1 10 Tf 10 100 Td (BLU) Tj 15 0 Td (E) Tj 5 0 Td [(F) -300 (IN)] TJ ( 7) Tj ET BT /F1 10 Tf 10 200 Td (Gary) Tj ET",
|
||||
"",
|
||||
vec![],
|
||||
);
|
||||
let page = pdfcraft_model::pages(&doc).swap_remove(0);
|
||||
let (_, data) = page_streams(&doc, &page.dict, 0).unwrap();
|
||||
let res = page.dict.get(b"Resources").and_then(|r| doc.resolve(r).as_dict().cloned()).unwrap_or_default();
|
||||
let mut rep = Report::default();
|
||||
let rects = [[10.0, 95.0, 300.0, 110.0], [10.0, 195.0, 30.0, 210.0]];
|
||||
let mut images = pdfcraft_cos::ImageBudget::new(&doc);
|
||||
let mut scope = Scope::new(&rects, Mode::Apply, &mut rep, pdfcraft_cos::DecodeBudget::page(), &mut images);
|
||||
process(&mut doc, &mut scope, &data, &res, pdfcraft_content::Matrix::IDENTITY);
|
||||
scope.flush();
|
||||
assert_eq!(scope.removed_text, ["BLUEF IN 7", "Gary"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_terms_and_xmp_text() {
|
||||
// Runs of two letters or digits or more are matched whole, and by their words of three or
|
||||
// more, ignoring case and spacing.
|
||||
let t = metadata::terms(&["John Smith".into(), "7".into(), "of".into(), "A-12".into()]);
|
||||
assert_eq!(t.as_deref(), Some(&["john smith", "john", "smith", "of", "a-12"].map(String::from)[..]));
|
||||
// A run too long to search for whole can't be matched: the caller clears everything.
|
||||
assert_eq!(metadata::terms(&["x".repeat(5000)]), None);
|
||||
// XMP values: element content, attribute values, CDATA, comments and processing instructions,
|
||||
// entities decoded; not names, and not namespace declarations.
|
||||
let x = metadata::xml_values(
|
||||
r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><?audit James Bond?><?xpacketAudit SECRET?><x:xmpmeta xmlns:x="adobe:ns:meta/" xmlns="http://example.org/" xmlns:y = "adobe:ns:y/" b = "spaced"><dc:title a="x & y"><!-- hidden --><rdf:li>Résumé <![CDATA[<raw>]]></rdf:li></dc:title></x:xmpmeta>"#,
|
||||
);
|
||||
// Only the `xpacket` wrapper itself is skipped, and a declaration is one with spaces around
|
||||
// `=` too.
|
||||
assert_eq!(x, ["audit James Bond", "xpacketAudit SECRET", "spaced", "x & y", " hidden ", "Résumé ", "<raw>"]);
|
||||
}
|
||||
|
||||
/// XMP is decoded from UTF-8 or UTF-16, the latter also without a byte-order mark at the start
|
||||
/// (a packet's mark sits inside `<?xpacket begin="…"`); anything that isn't valid is not read.
|
||||
#[test]
|
||||
fn xmp_packets_decode_or_fail_closed() {
|
||||
let text = "<?xpacket begin=\"\u{feff}\"?><a>Bond</a>";
|
||||
let utf16 = |be: bool| text.encode_utf16().flat_map(|u| if be { u.to_be_bytes() } else { u.to_le_bytes() }).collect::<Vec<u8>>();
|
||||
assert_eq!(metadata::decode_xmp(text.as_bytes()).as_deref(), Some(text));
|
||||
assert_eq!(metadata::decode_xmp(&utf16(true)).as_deref(), Some(text));
|
||||
assert_eq!(metadata::decode_xmp(&utf16(false)).as_deref(), Some(text));
|
||||
assert_eq!(metadata::decode_xmp(&[[0xFF, 0xFE].as_slice(), &utf16(false)].concat()).as_deref(), Some(text));
|
||||
assert_eq!(metadata::decode_xmp(b"<a>Bond \xE9</a>"), None, "Latin-1 isn't searched lossily");
|
||||
assert_eq!(metadata::decode_xmp(&utf16(false)[1..]), None, "odd length");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn added_text_parameters_do_not_keep_redacted_text() {
|
||||
// An Edit ▸ Add content item: its stream dictionary keeps the source text under /PCAdded.
|
||||
@@ -4113,9 +4513,13 @@ fn unused_descriptions_in_shared_or_inherited_resources_are_retired() {
|
||||
}
|
||||
|
||||
mod descriptions;
|
||||
mod elsewhere;
|
||||
mod hidden_scope;
|
||||
mod identification;
|
||||
mod inline_images;
|
||||
mod limits;
|
||||
mod metadata_fail_closed;
|
||||
mod metadata_matching;
|
||||
mod property_retirement;
|
||||
mod surviving_references;
|
||||
mod text_state;
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
//! #2539 review: what applying leaves as it is but still holds the removed text is counted in
|
||||
//! [`Report::still_holding`], and nothing of it is changed.
|
||||
|
||||
use super::*;
|
||||
|
||||
const XMP_BLUEFIN: &str = r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:description>Photo for Bluefin</dc:description></rdf:Description></rdf:RDF></x:xmpmeta>"#;
|
||||
const XMP_OTHER: &str = r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Annual figures</dc:title></rdf:Description></rdf:RDF></x:xmpmeta>"#;
|
||||
|
||||
/// The objects whose text the check reads, by number.
|
||||
const HELD: [u32; 20] = [7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26];
|
||||
|
||||
/// A page showing "Codename BLUEFIN report" (BLUEFIN at x 55–90) with `font`, and the word, or
|
||||
/// other text, in a bookmark, attachments, comments and XMP the mark doesn't touch:
|
||||
/// - bookmarks: "Bluefin plans", "Annual figures", and under it "BLUEFIN appendix";
|
||||
/// - attachments: one named `bluefin.txt`, one described "About Bluefin", one with neither, and
|
||||
/// a file attachment annotation for `Bluefin.pdf`;
|
||||
/// - comments: one whose text mentions it, one by "Bluefin Team", one whose rich text does, one
|
||||
/// that doesn't, and a pop-up repeating it (pop-ups aren't counted);
|
||||
/// - XMP: an image's mentions it, a font's doesn't.
|
||||
fn page_with_places(font: &str) -> Document {
|
||||
let annot = |rest: &str| format!("<< /Type /Annot /Rect [200 200 220 220] {rest} >>").into_bytes();
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Outlines 7 0 R /Names << /EmbeddedFiles 11 0 R >> >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> /XObject << /Im 24 0 R >> >> /Annots [16 0 R 17 0 R 18 0 R 19 0 R 20 0 R 21 0 R] >>".to_vec(),
|
||||
stream("", b"BT /F1 10 Tf 10 100 Td (Codename BLUEFIN report) Tj ET q 10 0 0 10 200 250 cm /Im Do Q"),
|
||||
format!("{} /Metadata 26 0 R >>", font.replace("95 0 R", "6 0 R").strip_suffix(">>").unwrap()).into_bytes(),
|
||||
widths(),
|
||||
// 7–10: the outline.
|
||||
b"<< /Type /Outlines /First 8 0 R /Last 9 0 R /Count 3 >>".to_vec(),
|
||||
b"<< /Title (Bluefin plans) /Parent 7 0 R /Next 9 0 R >>".to_vec(),
|
||||
b"<< /Title (Annual figures) /Parent 7 0 R /Prev 8 0 R /First 10 0 R /Last 10 0 R /Count 1 >>".to_vec(),
|
||||
b"<< /Title (BLUEFIN appendix) /Parent 9 0 R >>".to_vec(),
|
||||
// 11–15: the attachments' name tree and file specifications.
|
||||
b"<< /Kids [12 0 R] >>".to_vec(),
|
||||
b"<< /Limits [(a.txt) (bluefin.txt)] /Names [(a.txt) 13 0 R (b.txt) 14 0 R (bluefin.txt) 15 0 R] >>".to_vec(),
|
||||
b"<< /Type /Filespec /F (a.txt) /UF (a.txt) >>".to_vec(),
|
||||
b"<< /Type /Filespec /F (b.txt) /Desc (About Bluefin) >>".to_vec(),
|
||||
b"<< /Type /Filespec /F (c.txt) >>".to_vec(),
|
||||
// 16–21: the annotations.
|
||||
annot("/Subtype /Text /Contents (Ask about bluefin) /T (Ada)"),
|
||||
annot("/Subtype /Text /Contents (Seen) /T (Bluefin Team)"),
|
||||
annot("/Subtype /FreeText /Contents (plain) /RC (<body><p>BLUEFIN</p></body>) /DA (/F1 10 Tf)"),
|
||||
annot("/Subtype /Text /Contents (Looks good) /T (Ada) /Popup 21 0 R"),
|
||||
annot("/Subtype /FileAttachment /Contents (Attached) /T (Ada) /FS << /Type /Filespec /UF (Bluefin.pdf) >>"),
|
||||
annot("/Subtype /Popup /Parent 19 0 R /Contents (bluefin)"),
|
||||
// 22–23: unused.
|
||||
b"null".to_vec(),
|
||||
b"null".to_vec(),
|
||||
// 24–26: an image with XMP that mentions it, and the two packets.
|
||||
stream("/Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray /BitsPerComponent 8 /Metadata 25 0 R", b"\x80"),
|
||||
stream("/Type /Metadata /Subtype /XML", XMP_BLUEFIN.as_bytes()),
|
||||
stream("/Type /Metadata /Subtype /XML", XMP_OTHER.as_bytes()),
|
||||
];
|
||||
pdf(objs)
|
||||
}
|
||||
|
||||
fn held(doc: &Document) -> Vec<Object> {
|
||||
HELD.iter().map(|&n| (*doc.get(ObjRef::new(n, 0))).clone()).collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn places_apply_leaves_that_still_hold_the_removed_text_are_counted_and_kept() {
|
||||
let mut doc = page_with_places(FONT);
|
||||
let before = held(&doc);
|
||||
mark(&mut doc, 0, &[[55.0, 95.0, 90.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.glyphs, 7, "{r:?}");
|
||||
assert_eq!(r.still_holding, Elsewhere { bookmarks: 2, attachments: 3, comments: 3, xmp: 1, unsearched: false }, "{r:?}");
|
||||
assert_eq!(r.still_holding.total(), 9);
|
||||
assert_eq!(held(&doc), before, "counting changes nothing");
|
||||
let catalog = doc.get(doc.root().unwrap()).as_dict().cloned().unwrap();
|
||||
assert!(catalog.contains(b"Outlines") && catalog.contains(b"Names"));
|
||||
assert_eq!(annots_of(&doc, &pdfcraft_model::pages(&doc)[0].dict).len(), 6, "the comments stay");
|
||||
}
|
||||
|
||||
/// Removed text that is unrelated to every place counts nothing.
|
||||
#[test]
|
||||
fn places_without_the_removed_text_are_not_counted() {
|
||||
let mut doc = page_with_places(FONT);
|
||||
// "Codename" alone, at x 10–50.
|
||||
mark(&mut doc, 0, &[[8.0, 95.0, 50.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.glyphs, 8, "{r:?}");
|
||||
assert_eq!(r.still_holding, Elsewhere::default(), "{r:?}");
|
||||
}
|
||||
|
||||
/// When the removed text can't be known (a removed glyph whose text isn't known), every place
|
||||
/// that holds any text counts, and says why.
|
||||
#[test]
|
||||
fn places_count_whenever_they_hold_text_if_the_removed_text_is_unknown() {
|
||||
let unnamed = FONT.replace(" /Widths", " /Encoding << /Differences [66 /g17 /g18 /g19 /g20 /g21 /g22 /g23 /g24 /g25 /g26 /g27 /g28 /g29 /g30 /g31 /g32 /g33 /g34 /g35 /g36 /g37 /g38 /g39 /g40 /g41 /g42] >> /Widths");
|
||||
let mut doc = page_with_places(&unnamed);
|
||||
let before = held(&doc);
|
||||
mark(&mut doc, 0, &[[55.0, 95.0, 90.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
// Every bookmark title, every attachment (each has a name), every comment but the pop-up,
|
||||
// and both XMP packets (the font's too).
|
||||
assert_eq!(r.still_holding, Elsewhere { bookmarks: 3, attachments: 4, comments: 5, xmp: 2, unsearched: true }, "{r:?}");
|
||||
assert_eq!(held(&doc), before);
|
||||
}
|
||||
|
||||
/// A bookmark list that loops back on itself is walked once.
|
||||
#[test]
|
||||
fn a_looping_outline_is_read_once() {
|
||||
let mut doc = page_with_places(FONT);
|
||||
// The last item's /Next goes back to the first.
|
||||
doc.update_dict(ObjRef::new(9, 0), |d| d.set(b"Next".to_vec(), Object::Ref(ObjRef::new(8, 0)))).unwrap();
|
||||
doc.update_dict(ObjRef::new(10, 0), |d| d.set(b"First".to_vec(), Object::Ref(ObjRef::new(7, 0)))).unwrap();
|
||||
mark(&mut doc, 0, &[[55.0, 95.0, 90.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.still_holding.bookmarks, 2, "{r:?}");
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
//! #2539 review: the PDF/A and PDF/UA identification of a catalog XMP packet the metadata check
|
||||
//! removes is written back in a packet of its own, or reported lost.
|
||||
|
||||
use super::*;
|
||||
use crate::{Claim, Identification};
|
||||
|
||||
const DOC_XMP_HEAD: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">"#;
|
||||
const DOC_XMP_TAIL: &str = r#"</rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#;
|
||||
|
||||
/// One page showing "Codename BLUEFIN report", whose catalog XMP is `xmp` (raw bytes).
|
||||
fn page(xmp: &[u8]) -> Document {
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>".to_vec(),
|
||||
stream("", b"BT /F1 10 Tf 10 100 Td (Codename BLUEFIN report) Tj ET"),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
stream("/Type /Metadata /Subtype /XML", xmp),
|
||||
b"<< /Producer (Test) >>".to_vec(),
|
||||
];
|
||||
Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 8 0 R").build())).unwrap()
|
||||
}
|
||||
|
||||
/// Apply a mark over "BLUEFIN" and return the report and the catalog's XMP text afterwards.
|
||||
fn redact(doc: &mut Document) -> (Report, Option<String>) {
|
||||
mark(doc, 0, &[[55.0, 95.0, 90.0, 110.0]], "");
|
||||
let r = apply(doc, None).unwrap();
|
||||
let xmp = doc.get(doc.root().unwrap()).as_dict().and_then(|c| c.get(b"Metadata").cloned());
|
||||
let text = xmp.and_then(|x| crate::metadata::xmp_text(doc, &x));
|
||||
(r, text)
|
||||
}
|
||||
|
||||
/// The packet that names the redacted word goes, and one holding only its PDF/A and PDF/UA
|
||||
/// identification takes its place: the PDF/A properties written as attributes, PDF/UA's as an
|
||||
/// element under a prefix of its own, both found by namespace. Nothing else is copied.
|
||||
#[test]
|
||||
fn the_identification_of_a_removed_packet_is_written_back() {
|
||||
let body = r#"<rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" pdfaid:part="2" pdfaid:conformance="B"/><rdf:Description rdf:about="" xmlns:ua="http://www.aiim.org/pdfua/ns/id/"><ua:part>1</ua:part></rdf:Description><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Bluefin report</dc:title><dc:creator>Ada Lovelace</dc:creator></rdf:Description>"#;
|
||||
let mut doc = page(format!("{DOC_XMP_HEAD}{body}{DOC_XMP_TAIL}").as_bytes());
|
||||
let (r, text) = redact(&mut doc);
|
||||
assert_eq!(r.identification, Identification { pdfa: Claim::Kept, pdfua: Claim::Kept }, "{r:?}");
|
||||
assert_eq!(r.metadata, 1, "{r:?}");
|
||||
let text = text.expect("a packet is written back");
|
||||
assert!(!text.to_lowercase().contains("bluefin") && !text.contains("Lovelace"), "{text}");
|
||||
let xml = roxmltree::Document::parse(&text).expect("well-formed");
|
||||
let value = |ns: &str, local: &str| {
|
||||
xml.descendants().find(|n| n.tag_name().namespace() == Some(ns) && n.tag_name().name() == local).and_then(|n| n.text()).map(str::to_string)
|
||||
};
|
||||
let a = "http://www.aiim.org/pdfa/ns/id/";
|
||||
let ua = "http://www.aiim.org/pdfua/ns/id/";
|
||||
assert_eq!((value(a, "part").as_deref(), value(a, "conformance").as_deref(), value(ua, "part").as_deref()), (Some("2"), Some("B"), Some("1")));
|
||||
// PDF/A-2 accepts pdfuaid only through an extension schema describing it.
|
||||
assert_eq!(value("http://www.aiim.org/pdfa/ns/schema#", "prefix").as_deref(), Some("pdfuaid"), "{text}");
|
||||
}
|
||||
|
||||
/// A packet without identification is removed as before, and nothing is said about it.
|
||||
#[test]
|
||||
fn a_packet_without_identification_is_only_removed() {
|
||||
let body = r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Bluefin report</dc:title></rdf:Description>"#;
|
||||
let mut doc = page(format!("{DOC_XMP_HEAD}{body}{DOC_XMP_TAIL}").as_bytes());
|
||||
let (r, text) = redact(&mut doc);
|
||||
assert_eq!((r.identification, text), (Identification::default(), None), "{r:?}");
|
||||
}
|
||||
|
||||
/// The identification is lost, and said to be, when its value isn't the code it must be (a
|
||||
/// conformance level that is a word) or when the packet can't be read (here it isn't UTF-8).
|
||||
#[test]
|
||||
fn an_identification_that_cannot_be_kept_is_reported_lost() {
|
||||
let wordy = r#"<rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" pdfaid:part="2" pdfaid:conformance="Bluefin"/>"#;
|
||||
let mut doc = page(format!("{DOC_XMP_HEAD}{wordy}{DOC_XMP_TAIL}").as_bytes());
|
||||
let (r, text) = redact(&mut doc);
|
||||
assert_eq!((r.identification, text), (Identification { pdfa: Claim::Lost, pdfua: Claim::None }, None), "{r:?}");
|
||||
|
||||
let latin1 = [
|
||||
DOC_XMP_HEAD.as_bytes(),
|
||||
br#"<rdf:Description rdf:about="" xmlns:pdfuaid="http://www.aiim.org/pdfua/ns/id/" pdfuaid:part="1" dc:title="Bluefin caf\xE9"/>"#,
|
||||
DOC_XMP_TAIL.as_bytes(),
|
||||
]
|
||||
.concat();
|
||||
let mut doc = page(&latin1);
|
||||
let (r, text) = redact(&mut doc);
|
||||
assert_eq!((r.identification, text), (Identification { pdfa: Claim::None, pdfua: Claim::Lost }, None), "{r:?}");
|
||||
assert!(r.identification.lost());
|
||||
}
|
||||
|
||||
/// When the removed text couldn't be searched for, every catalog packet goes, but a part number
|
||||
/// and a conformance letter, one letter or digit each, can't hold it and are written back; a
|
||||
/// revision year could, and isn't.
|
||||
#[test]
|
||||
fn unsearched_text_keeps_only_one_character_codes() {
|
||||
let body = r#"<rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" pdfaid:part="4" pdfaid:rev="2020"/><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Annual figures</dc:title></rdf:Description>"#;
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>".to_vec(),
|
||||
stream("", b"BT /F1 10 Tf 10 100 Td (ABC) Tj ET"),
|
||||
// Glyph names outside the glyph list: the removed text isn't known.
|
||||
FONT.replace("95 0 R", "6 0 R").replace(" /Widths", " /Encoding << /Differences [65 /g17 /g18 /g19] >> /Widths").into_bytes(),
|
||||
widths(),
|
||||
stream("/Type /Metadata /Subtype /XML", format!("{DOC_XMP_HEAD}{body}{DOC_XMP_TAIL}").as_bytes()),
|
||||
];
|
||||
let mut doc = Document::open(Arc::new(Pdf::new().objs(objs).build())).unwrap();
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.identification, Identification { pdfa: Claim::Kept, pdfua: Claim::None }, "{r:?}");
|
||||
let xmp = doc.get(doc.root().unwrap()).as_dict().and_then(|c| c.get(b"Metadata").cloned()).unwrap();
|
||||
let text = crate::metadata::xmp_text(&doc, &xmp).unwrap();
|
||||
assert!(text.contains("<pdfaid:part>4</pdfaid:part>") && !text.contains("2020") && !text.contains("Annual"), "{text}");
|
||||
}
|
||||
|
||||
/// The extension schema PDF/A-2 needs for pdfuaid is fixed text, but if the redacted words are
|
||||
/// among its words ("schema" here) it isn't written, and the PDF/UA claim is reported lost.
|
||||
#[test]
|
||||
fn the_extension_schema_is_left_out_when_it_uses_a_redacted_word() {
|
||||
let body = r#"<rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" pdfaid:part="2" pdfaid:conformance="U"/><rdf:Description rdf:about="" xmlns:pdfuaid="http://www.aiim.org/pdfua/ns/id/" pdfuaid:part="1"/><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Schema notes</dc:title></rdf:Description>"#;
|
||||
let mut doc = page(format!("{DOC_XMP_HEAD}{body}{DOC_XMP_TAIL}").as_bytes());
|
||||
let s = crate::metadata::scrub_within(&mut doc, &["SCHEMA".into()], true, 1 << 28).unwrap();
|
||||
assert_eq!(s.identification, Identification { pdfa: Claim::Kept, pdfua: Claim::Lost });
|
||||
let xmp = doc.get(doc.root().unwrap()).as_dict().and_then(|c| c.get(b"Metadata").cloned()).unwrap();
|
||||
let text = crate::metadata::xmp_text(&doc, &xmp).unwrap();
|
||||
assert!(!text.to_lowercase().contains("schema") && text.contains("<pdfaid:conformance>U</pdfaid:conformance>"), "{text}");
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
//! #2539 review: each way the metadata check fails closed, end to end. When the removed text
|
||||
//! can't be known in full, every `/Info` string and both the catalog's and the page's XMP go;
|
||||
//! when one XMP packet can't be read, that packet goes and the rest of the metadata is checked
|
||||
//! as usual.
|
||||
|
||||
use super::*;
|
||||
|
||||
const CLEAN_XMP: &str = r#"<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Annual figures</dc:title></rdf:Description></rdf:RDF></x:xmpmeta>"#;
|
||||
|
||||
/// One page showing `content`, with `/Info` (two strings and a name), catalog XMP (object 7)
|
||||
/// and page XMP (object 8). None of the metadata mentions anything the tests remove.
|
||||
fn page_with_metadata(content: &[u8], catalog_xmp: Vec<u8>) -> Document {
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Metadata 8 0 R >>".to_vec(),
|
||||
stream("", content),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
catalog_xmp,
|
||||
stream("/Type /Metadata /Subtype /XML", CLEAN_XMP.as_bytes()),
|
||||
b"<< /Title (Annual figures) /Producer (Test) /Trapped /False >>".to_vec(),
|
||||
];
|
||||
Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 9 0 R").build())).unwrap()
|
||||
}
|
||||
|
||||
fn clean_xmp() -> Vec<u8> {
|
||||
stream("/Type /Metadata /Subtype /XML", CLEAN_XMP.as_bytes())
|
||||
}
|
||||
|
||||
fn catalog_has_xmp(doc: &Document) -> bool {
|
||||
doc.get(doc.root().unwrap()).as_dict().is_some_and(|c| c.contains(b"Metadata"))
|
||||
}
|
||||
|
||||
fn page_has_xmp(doc: &Document) -> bool {
|
||||
pdfcraft_model::pages(doc)[0].dict.contains(b"Metadata")
|
||||
}
|
||||
|
||||
/// Mark the line at y 100 across the page and apply.
|
||||
fn redact_line(doc: &mut Document) -> Report {
|
||||
mark(doc, 0, &[[0.0, 95.0, 300.0, 110.0]], "");
|
||||
apply(doc, None).unwrap()
|
||||
}
|
||||
|
||||
/// Every `/Info` string and both XMP streams went, though none of them mentions what was removed.
|
||||
fn assert_cleared(doc: &Document, r: &Report) {
|
||||
assert_eq!(r.metadata, 4, "{r:?}");
|
||||
assert_eq!(info_keys(doc), ["Trapped"], "strings go, the name stays");
|
||||
assert!(!catalog_has_xmp(doc) && !page_has_xmp(doc), "both XMP streams go");
|
||||
}
|
||||
|
||||
/// Control for the tests below: removing a short word that the metadata doesn't mention keeps
|
||||
/// all of it.
|
||||
#[test]
|
||||
fn metadata_that_does_not_mention_the_removed_text_stays() {
|
||||
let mut doc = page_with_metadata(b"BT /F1 10 Tf 10 100 Td (Bluefin) Tj ET", clean_xmp());
|
||||
let r = redact_line(&mut doc);
|
||||
assert_eq!(r.metadata, 0, "{r:?}");
|
||||
assert_eq!(info_keys(&doc), ["Title", "Producer", "Trapped"]);
|
||||
assert!(catalog_has_xmp(&doc) && page_has_xmp(&doc));
|
||||
}
|
||||
|
||||
/// A single removed run longer than the collection keeps (64 KiB): the part past the limit is
|
||||
/// never seen. The run is one letter, 70,000 spaces (one space once normalized) and another
|
||||
/// letter, so what would be collected without the limit is short enough to search, and only
|
||||
/// the limit itself clears the metadata.
|
||||
#[test]
|
||||
fn a_removed_run_over_64_kib_clears_the_metadata() {
|
||||
// At 0.004 pt each glyph advances 0.002 pt, so all 70,002 fit in 141 pt of one line.
|
||||
let text = format!("A{}B", " ".repeat(70_000));
|
||||
let content = format!("BT /F1 0.004 Tf 10 100 Td ({text}) Tj ET");
|
||||
let mut doc = page_with_metadata(content.as_bytes(), clean_xmp());
|
||||
let r = redact_line(&mut doc);
|
||||
assert_eq!(r.glyphs, 70_002, "{r:?}");
|
||||
assert_cleared(&doc, &r);
|
||||
}
|
||||
|
||||
/// More than 20,000 distinct terms: each removed run is one word, far enough from the last to
|
||||
/// start a run of its own, and each word is different.
|
||||
#[test]
|
||||
fn more_than_20000_terms_clear_the_metadata() {
|
||||
let mut content = b"BT /F1 10 Tf 10 100 Td".to_vec();
|
||||
// Words alternate between x 10 and x 70; each is 30 pt wide, so the gap between two is
|
||||
// 30 pt, more than a run spans.
|
||||
for i in 0..20_001u32 {
|
||||
let x = if i % 2 == 0 { 60 } else { -60 };
|
||||
content.extend_from_slice(format!(" (q{i:05}) Tj {x} 0 Td").as_bytes());
|
||||
}
|
||||
content.extend_from_slice(b" ET");
|
||||
let mut doc = page_with_metadata(&content, clean_xmp());
|
||||
let r = redact_line(&mut doc);
|
||||
assert_eq!(r.glyphs, 20_001 * 6, "{r:?}");
|
||||
assert_cleared(&doc, &r);
|
||||
}
|
||||
|
||||
/// A removed run longer than 4,096 characters can't be searched for whole.
|
||||
#[test]
|
||||
fn a_removed_run_over_4096_characters_clears_the_metadata() {
|
||||
// 4,100 letters at 0.05 pt advance 0.025 pt each: 103 pt in all, one run, 4 KiB.
|
||||
let content = format!("BT /F1 0.05 Tf 10 100 Td ({}) Tj ET", "Z".repeat(4_100));
|
||||
let mut doc = page_with_metadata(content.as_bytes(), clean_xmp());
|
||||
let r = redact_line(&mut doc);
|
||||
assert_eq!(r.glyphs, 4_100, "{r:?}");
|
||||
assert_cleared(&doc, &r);
|
||||
}
|
||||
|
||||
/// The catalog's packet can't be read: that packet goes, unread, while the page's packet and
|
||||
/// `/Info`, which don't mention the removed word, stay.
|
||||
fn assert_only_the_catalog_packet_went(catalog_xmp: Vec<u8>) {
|
||||
let mut doc = page_with_metadata(b"BT /F1 10 Tf 10 100 Td (Bluefin) Tj ET", catalog_xmp);
|
||||
let r = redact_line(&mut doc);
|
||||
assert_eq!(r.metadata, 1, "{r:?}");
|
||||
assert!(!catalog_has_xmp(&doc), "the packet that couldn't be read goes");
|
||||
assert!(page_has_xmp(&doc), "the readable packet is checked, and stays");
|
||||
assert_eq!(info_keys(&doc), ["Title", "Producer", "Trapped"]);
|
||||
}
|
||||
|
||||
/// XMP whose Flate data is corrupt can't be decoded, so it can't be searched.
|
||||
#[test]
|
||||
fn an_xmp_stream_whose_filter_data_is_corrupt_goes() {
|
||||
assert_only_the_catalog_packet_went(stream("/Type /Metadata /Subtype /XML /Filter /FlateDecode", b"this is not deflate data at all"));
|
||||
}
|
||||
|
||||
/// A valid Flate stream cut off half way decodes, leniently, to the first part of the packet;
|
||||
/// the rest was never seen, so the packet isn't searched but goes.
|
||||
#[test]
|
||||
fn an_xmp_stream_whose_filter_data_is_cut_short_goes() {
|
||||
let full = Stream::flate(Dict::new(), CLEAN_XMP.as_bytes());
|
||||
let cut = full.raw.get(..full.raw.len() / 2).unwrap();
|
||||
assert_only_the_catalog_packet_went(stream("/Type /Metadata /Subtype /XML /Filter /FlateDecode", cut));
|
||||
}
|
||||
|
||||
/// XMP that decodes to more than 16 MiB isn't read at all.
|
||||
#[test]
|
||||
fn an_xmp_stream_over_16_mib_goes() {
|
||||
// A valid packet, padded with whitespace past the limit.
|
||||
let padded = CLEAN_XMP.replace("</x:xmpmeta>", &format!("{}</x:xmpmeta>", " ".repeat(16 * 1024 * 1024)));
|
||||
assert_only_the_catalog_packet_went(stream("/Type /Metadata /Subtype /XML", padded.as_bytes()));
|
||||
}
|
||||
|
||||
/// XMP that is neither UTF-8 nor UTF-16 (here Latin-1) isn't searched lossily.
|
||||
#[test]
|
||||
fn an_xmp_stream_in_another_encoding_goes() {
|
||||
let latin1 = CLEAN_XMP.replace("Annual figures", "Ann\u{e9}e").replace('\u{e9}', "\u{0}");
|
||||
let bytes: Vec<u8> = latin1.bytes().map(|b| if b == 0 { 0xE9 } else { b }).collect();
|
||||
assert!(std::str::from_utf8(&bytes).is_err());
|
||||
assert_only_the_catalog_packet_went(stream("/Type /Metadata /Subtype /XML", &bytes));
|
||||
}
|
||||
|
||||
/// XMP behind two filters has no strict decoder here, so it isn't read either. The first filter
|
||||
/// alone decodes to valid ASCII (the packet in hex), which must not be searched as the packet.
|
||||
#[test]
|
||||
fn an_xmp_stream_behind_two_filters_goes() {
|
||||
let hex: String = CLEAN_XMP.bytes().map(|b| format!("{b:02x}")).chain(std::iter::once(">".to_string())).collect();
|
||||
let Stream { raw, .. } = Stream::flate(Dict::new(), hex.as_bytes());
|
||||
assert_only_the_catalog_packet_went(stream("/Type /Metadata /Subtype /XML /Filter [/FlateDecode /ASCIIHexDecode]", &raw));
|
||||
}
|
||||
|
||||
/// A form XObject under a mark whose content can't be decoded is removed without being read, so
|
||||
/// what it showed is unknown (the deep-form case is `a_form_removed_unread_clears_the_metadata`).
|
||||
#[test]
|
||||
fn a_form_whose_content_cant_be_read_clears_the_metadata() {
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Metadata 7 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> /XObject << /Fm 9 0 R >> >> /Metadata 8 0 R >>".to_vec(),
|
||||
stream("", b"q 1 0 0 1 0 0 cm /Fm Do Q"),
|
||||
FONT.replace("95 0 R", "6 0 R").into_bytes(),
|
||||
widths(),
|
||||
clean_xmp(),
|
||||
clean_xmp(),
|
||||
// 9: a form whose data no filter here decodes.
|
||||
stream("/Type /XObject /Subtype /Form /BBox [0 0 300 300] /Filter /NoSuchDecode", b"BT /F1 10 Tf 10 100 Td (Bluefin) Tj ET"),
|
||||
b"<< /Title (Annual figures) /Producer (Test) /Trapped /False >>".to_vec(),
|
||||
];
|
||||
let mut doc = Document::open(Arc::new(Pdf::new().objs(objs).trailer_entry("/Info 10 0 R").build())).unwrap();
|
||||
let r = redact_line(&mut doc);
|
||||
assert_eq!(r.forms_removed, 1, "{r:?}");
|
||||
assert_cleared(&doc, &r);
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
//! #2539 review (second round): spellings of the removed text the metadata check has to match.
|
||||
|
||||
use super::*;
|
||||
|
||||
fn dc_title(title: &str) -> String {
|
||||
format!(r#"<rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>{title}</dc:title></rdf:Description>"#)
|
||||
}
|
||||
|
||||
/// A tracked heading, its letters spread by `TJ` numbers rather than character spacing, is
|
||||
/// collected as "B L U E F I N": its letters and digits still match `/Title (Bluefin)` and the
|
||||
/// XMP title, while `/Producer` stays.
|
||||
#[test]
|
||||
fn a_heading_tracked_with_tj_numbers_matches_the_metadata() {
|
||||
let content = b"BT /F1 10 Tf 10 100 Td [(B) -300 (L) -300 (U) -300 (E) -300 (F) -300 (I) -300 (N)] TJ ET";
|
||||
let mut doc = with_metadata(content, "<< /Title (Bluefin) /Producer (Test) >>", &dc_title("Bluefin"));
|
||||
mark(&mut doc, 0, &[[5.0, 95.0, 200.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(r.glyphs, 7, "{r:?}");
|
||||
assert_eq!(info_keys(&doc), ["Producer"], "{r:?}");
|
||||
assert!(!has_xmp(&doc), "the XMP title named it");
|
||||
assert_eq!(r.metadata, 2, "{r:?}");
|
||||
}
|
||||
|
||||
/// A value that spells the removed word with a space or a hyphen in it still holds it: "Blue fin"
|
||||
/// and "BLUE-FIN" go when "BLUEFIN" is redacted, while "Bluefish", which only shares letters
|
||||
/// with it, stays.
|
||||
#[test]
|
||||
fn a_value_spaced_or_hyphenated_otherwise_matches() {
|
||||
let info = "<< /Title (Blue fin) /Keywords (tuna, BLUE-FIN) /Subject (Bluefish) /Producer (Test) >>";
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 10 100 Td (Codename BLUEFIN report) Tj ET", info, &dc_title("Blue Fin"));
|
||||
// "BLUEFIN" spans x 55-90.
|
||||
mark(&mut doc, 0, &[[55.0, 95.0, 90.0, 110.0]], "");
|
||||
let r = apply(&mut doc, None).unwrap();
|
||||
assert_eq!(info_keys(&doc), ["Subject", "Producer"], "{r:?}");
|
||||
assert!(!has_xmp(&doc), "the XMP title spelled it with a space");
|
||||
assert_eq!(r.metadata, 3, "{r:?}");
|
||||
}
|
||||
|
||||
/// `s` as a UTF-16BE text string in hex, byte order mark first.
|
||||
fn utf16be(s: &str) -> String {
|
||||
format!("<FEFF{}>", s.encode_utf16().map(|u| format!("{u:04X}")).collect::<String>())
|
||||
}
|
||||
|
||||
/// The removed text and the metadata are compared after Unicode compatibility normalization
|
||||
/// (NFKC) and lower case on both sides: a decomposed "Müller" (u + U+0308) in a UTF-16BE `/Title`
|
||||
/// matches the composed one removed from the page, a "fi" ligature from a `ToUnicode` map matches
|
||||
/// "fi", and full-width letters match their ASCII forms.
|
||||
#[test]
|
||||
fn unicode_forms_of_the_removed_text_match() {
|
||||
let info = format!(
|
||||
"<< /Title {} /Subject (Finance plan) /Keywords {} /Author (Ada Lovelace) /Producer (Test) >>",
|
||||
utf16be("Mu\u{308}ller"),
|
||||
utf16be("\u{FF22}\u{FF2C}\u{FF35}\u{FF25}\u{FF26}\u{FF29}\u{FF2E}")
|
||||
);
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 10 100 Td (x) Tj ET", &info, "");
|
||||
let removed = ["M\u{FC}ller".to_string(), "\u{FB01}nance".to_string(), "Bluefin".to_string()];
|
||||
let n = metadata::scrub_within(&mut doc, &removed, true, 1 << 28).unwrap().removed;
|
||||
assert_eq!(info_keys(&doc), ["Author", "Producer"]);
|
||||
assert_eq!(n, 3);
|
||||
}
|
||||
|
||||
/// A value whose normalized form would be far longer than itself (U+FDFA is 18 letters in NFKC)
|
||||
/// isn't normalized in full: it counts as holding the removed text (fail-closed).
|
||||
#[test]
|
||||
fn a_value_that_normalizes_to_far_more_text_counts_as_holding_it() {
|
||||
let info = format!("<< /Subject {} /Producer (Test) >>", utf16be(&"\u{FDFA}".repeat(50)));
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 10 100 Td (x) Tj ET", &info, "");
|
||||
assert_eq!(metadata::scrub_within(&mut doc, &["Bluefin".into()], true, 1 << 28).unwrap().removed, 1);
|
||||
assert_eq!(info_keys(&doc), ["Producer"]);
|
||||
}
|
||||
|
||||
/// `s` as UTF-16LE bytes in hex, behind an `FF FE` byte order mark.
|
||||
fn utf16le(s: &str) -> String {
|
||||
format!("<FFFE{}>", s.encode_utf16().map(|u| format!("{:02X}{:02X}", u & 0xFF, u >> 8)).collect::<String>())
|
||||
}
|
||||
|
||||
/// ISO 32000-2 §7.9.2.2.1 defines text strings in PDFDocEncoding, UTF-16BE (`FE FF`) and UTF-8
|
||||
/// (`EF BB BF`) only, but a string behind an `FF FE` mark is UTF-16LE to anyone who reads it as
|
||||
/// such; read through PDFDocEncoding, Cyrillic or CJK text in it could never match. It is
|
||||
/// searched as UTF-16LE in `/Info` and in the places applying leaves (here a bookmark title).
|
||||
#[test]
|
||||
fn a_utf16le_string_is_searched_as_utf16le() {
|
||||
let info = format!("<< /Title {} /Subject {} /Producer (Test) >>", utf16le("Синий план"), utf16le("Годовой отчёт"));
|
||||
let mut doc = with_metadata(b"BT /F1 10 Tf 10 100 Td (x) Tj ET", &info, "");
|
||||
assert_eq!(metadata::scrub_within(&mut doc, &["СИНИЙ".into()], true, 1 << 28).unwrap().removed, 1);
|
||||
assert_eq!(info_keys(&doc), ["Subject", "Producer"]);
|
||||
|
||||
// A bookmark titled the same way is counted as still holding it.
|
||||
let objs: Vec<Vec<u8>> = vec![
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Outlines 4 0 R >>".to_vec(),
|
||||
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] >>".to_vec(),
|
||||
b"<< /Type /Outlines /First 5 0 R /Last 5 0 R /Count 1 >>".to_vec(),
|
||||
format!("<< /Title {} /Parent 4 0 R /Dest [3 0 R /Fit] >>", utf16le("Синий план")).into_bytes(),
|
||||
];
|
||||
let doc = Document::open(Arc::new(Pdf::new().objs(objs).build())).unwrap();
|
||||
assert_eq!(crate::elsewhere::find(&doc, &["СИНИЙ".into()], true), Elsewhere { bookmarks: 1, ..Default::default() });
|
||||
}
|
||||
@@ -115,6 +115,37 @@ pub(super) fn run(doc: &mut Document) -> (Result<Report, RedactError>, Proof) {
|
||||
(result, proof)
|
||||
}
|
||||
|
||||
/// Metadata a redaction that missed it would have left: applying removes the metadata holding
|
||||
/// the removed text (#2448), so a proof test of those surfaces puts it back afterwards.
|
||||
enum Left<'a> {
|
||||
Info(&'a [u8]),
|
||||
Xmp(&'a [u8]),
|
||||
}
|
||||
|
||||
/// Capture, apply, put `left` back, prove.
|
||||
fn run_leaving(doc: &mut Document, left: Left<'_>) -> (Result<Report, RedactError>, Proof) {
|
||||
let mut snap = Snapshot::capture(doc, None).unwrap();
|
||||
let result = applied(doc, &mut snap);
|
||||
match left {
|
||||
Left::Info(title) => {
|
||||
let mut info = Dict::new();
|
||||
info.set(b"Title".to_vec(), Object::String(pdfcraft_cos::PdfString::literal(title.to_vec())));
|
||||
let info = doc.add(Object::Dict(info));
|
||||
doc.trailer_mut().set(b"Info".to_vec(), Object::Ref(info));
|
||||
}
|
||||
Left::Xmp(packet) => {
|
||||
let mut dict = Dict::new();
|
||||
dict.set(b"Type".to_vec(), Object::Name(b"Metadata".to_vec()));
|
||||
dict.set(b"Subtype".to_vec(), Object::Name(b"XML".to_vec()));
|
||||
let xmp = doc.add(Object::Stream(Stream::from_raw(dict, packet.to_vec())));
|
||||
let root = doc.root().unwrap();
|
||||
doc.update_dict(root, |c| c.set(b"Metadata".to_vec(), Object::Ref(xmp))).unwrap();
|
||||
}
|
||||
}
|
||||
let proof = snap.prove(doc, None, &ProofOptions::default());
|
||||
(result, proof)
|
||||
}
|
||||
|
||||
/// Prove a document the redaction was never applied to (as a full save would write it).
|
||||
pub(super) fn prove_unapplied(doc: &mut Document) -> Proof {
|
||||
let snap = Snapshot::capture(doc, None).unwrap();
|
||||
@@ -207,7 +238,7 @@ fn the_manifest_has_digests_and_counts_but_no_plaintext() {
|
||||
#[test]
|
||||
fn survivors_do_not_echo_the_text_either() {
|
||||
let mut doc = marked(Spec { trailer: "/Info 7 0 R", extra: vec![b"<< /Title (SECRETVALUE) >>".to_vec()], ..Spec::default() });
|
||||
let (_, proof) = run(&mut doc);
|
||||
let (_, proof) = run_leaving(&mut doc, Left::Info(b"SECRETVALUE"));
|
||||
assert!(!proof.passed());
|
||||
for shown in [proof.to_text(), format!("{proof:?}")] {
|
||||
assert!(!shown.contains(SECRET), "{shown}");
|
||||
@@ -224,7 +255,7 @@ fn numeric_only_tj_is_not_glyph_carrying() {
|
||||
fn a_survivor_in_the_info_dictionary() {
|
||||
let mut doc =
|
||||
marked(Spec { trailer: "/Info 7 0 R", extra: vec![b"<< /Title (Report SECRETVALUE) /Author (Someone) >>".to_vec()], ..Spec::default() });
|
||||
let (_, proof) = run(&mut doc);
|
||||
let (_, proof) = run_leaving(&mut doc, Left::Info(b"Report SECRETVALUE"));
|
||||
assert_eq!(surface(&proof, Surface::Info).verdict, Verdict::Survivor);
|
||||
assert_eq!(proof.entries[0].status, Status::Failed);
|
||||
assert!(!proof.passed());
|
||||
@@ -232,20 +263,38 @@ fn a_survivor_in_the_info_dictionary() {
|
||||
|
||||
#[test]
|
||||
fn a_survivor_in_xmp() {
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", b"<x:xmpmeta><dc:title>Quarterly & SECRETVALUE</dc:title></x:xmpmeta>");
|
||||
let packet: &[u8] = b"<x:xmpmeta><dc:title>Quarterly & SECRETVALUE</dc:title></x:xmpmeta>";
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", packet);
|
||||
let mut doc = marked(Spec { catalog: "/Metadata 7 0 R", extra: vec![xmp], ..Spec::default() });
|
||||
let (_, proof) = run(&mut doc);
|
||||
let (_, proof) = run_leaving(&mut doc, Left::Xmp(packet));
|
||||
assert_eq!(surface(&proof, Surface::Xmp).verdict, Verdict::Survivor, "{}", proof.to_text());
|
||||
assert!(!proof.passed());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn applying_removes_the_metadata_survivors_the_proof_finds() {
|
||||
// The same files as the two tests above, applied without putting anything back: the
|
||||
// metadata holding the removed text is gone (#2448), and the proof finds nothing there.
|
||||
let mut doc =
|
||||
marked(Spec { trailer: "/Info 7 0 R", extra: vec![b"<< /Title (Report SECRETVALUE) /Author (Someone) >>".to_vec()], ..Spec::default() });
|
||||
let (r, proof) = run(&mut doc);
|
||||
assert_eq!(r.unwrap().metadata, 1);
|
||||
assert_ne!(surface(&proof, Surface::Info).verdict, Verdict::Survivor, "{}", proof.to_text());
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", b"<x:xmpmeta><dc:title>Quarterly & SECRETVALUE</dc:title></x:xmpmeta>");
|
||||
let mut doc = marked(Spec { catalog: "/Metadata 7 0 R", extra: vec![xmp], ..Spec::default() });
|
||||
let (r, proof) = run(&mut doc);
|
||||
assert_eq!(r.unwrap().metadata, 1);
|
||||
assert_ne!(surface(&proof, Surface::Xmp).verdict, Verdict::Survivor, "{}", proof.to_text());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_long_numeric_reference_in_xmp_is_a_survivor() {
|
||||
// A character reference may carry any number of leading zeros: the padded form reads the
|
||||
// same, and is swept the same.
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", b"<x:xmpmeta><dc:title>SECRETVALUE</dc:title></x:xmpmeta>");
|
||||
let packet: &[u8] = b"<x:xmpmeta><dc:title>SECRETVALUE</dc:title></x:xmpmeta>";
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", packet);
|
||||
let mut doc = marked(Spec { catalog: "/Metadata 7 0 R", extra: vec![xmp], ..Spec::default() });
|
||||
let (_, proof) = run(&mut doc);
|
||||
let (_, proof) = run_leaving(&mut doc, Left::Xmp(packet));
|
||||
assert_eq!(surface(&proof, Surface::Xmp).verdict, Verdict::Survivor, "{}", proof.to_text());
|
||||
assert!(!proof.passed());
|
||||
}
|
||||
@@ -855,14 +904,15 @@ fn an_ambiguous_needle_is_still_swept_outside_page_content() {
|
||||
let content = b"BT /F1 10 Tf 10 200 Td (SECRETVALUE) Tj ET BT /F1 10 Tf 10 100 Td (SECRETVALUE) Tj ET";
|
||||
let spec = Spec { trailer: "/Info 7 0 R", extra: vec![b"<< /Title (SECRETVALUE) >>".to_vec()], ..Spec::default() };
|
||||
let mut doc = marked_content(content, spec);
|
||||
let (_, proof) = run(&mut doc);
|
||||
let (_, proof) = run_leaving(&mut doc, Left::Info(b"SECRETVALUE"));
|
||||
assert_eq!(surface(&proof, Surface::Info).verdict, Verdict::Survivor, "{}", proof.to_text());
|
||||
assert!(!proof.passed());
|
||||
|
||||
// Likewise an XMP packet and an annotation string.
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", b"<x:xmpmeta>SECRETVALUE</x:xmpmeta>");
|
||||
let packet: &[u8] = b"<x:xmpmeta>SECRETVALUE</x:xmpmeta>";
|
||||
let xmp = stream("/Type /Metadata /Subtype /XML", packet);
|
||||
let mut doc = marked_content(content, Spec { catalog: "/Metadata 7 0 R", extra: vec![xmp], ..Spec::default() });
|
||||
assert_eq!(surface(&run(&mut doc).1, Surface::Xmp).verdict, Verdict::Survivor);
|
||||
assert_eq!(surface(&run_leaving(&mut doc, Left::Xmp(packet)).1, Surface::Xmp).verdict, Verdict::Survivor);
|
||||
let annot = b"<< /Type /Annot /Subtype /Text /Rect [200 200 220 220] /Contents (SECRETVALUE) >>".to_vec();
|
||||
let mut doc = marked_content(content, Spec { page: "/Annots [7 0 R]", extra: vec![annot], ..Spec::default() });
|
||||
assert_eq!(surface(&run(&mut doc).1, Surface::ObjectStrings).verdict, Verdict::Survivor);
|
||||
|
||||
@@ -1904,11 +1904,19 @@ fn finish_redactions_and_fields(app: &mut PdfCraftApp, req: &mut Pending) {
|
||||
if let Some((_, id)) = app.active_ids() {
|
||||
app.renew_recovery(id);
|
||||
}
|
||||
if marks == 1 {
|
||||
app.notify_tr("Applied 1 redaction mark. Save to remove the content from the file.");
|
||||
let applied = if marks == 1 {
|
||||
tl!("Applied 1 redaction mark. Save to remove the content from the file.").to_string()
|
||||
} else {
|
||||
app.notify_fmt("Applied {n} redaction marks. Save to remove the content from the file.", &[("n", &marks.to_string())]);
|
||||
}
|
||||
crate::i18n::fmt(tl!("Applied {n} redaction marks. Save to remove the content from the file."), &[("n", &marks.to_string())])
|
||||
};
|
||||
// Where the removed text is still held that applying leaves as it is, and what
|
||||
// became of a PDF/A or PDF/UA claim in the document XMP it removed: said in the
|
||||
// same notification, so neither can replace the first sentence.
|
||||
let doc = app.active_ids().and_then(|(_, id)| app.session.get(id));
|
||||
let left = doc.and_then(|d| d.redaction_elsewhere()).and_then(|e| still_holding_note(&e));
|
||||
let id = doc.and_then(|d| d.redaction_identification()).and_then(|id| identification_note(&id));
|
||||
let message = [Some(applied), left, id].into_iter().flatten().collect::<Vec<_>>().join(" ");
|
||||
app.notify(message);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
@@ -1957,6 +1965,61 @@ fn finish_redactions_and_fields(app: &mut PdfCraftApp, req: &mut Pending) {
|
||||
}
|
||||
}
|
||||
|
||||
/// The sentence telling the user where the removed text is still held that applying redactions
|
||||
/// leaves as it is, or `None` when it is nowhere else. When the text couldn't be searched for,
|
||||
/// the places are those holding any text, and the sentence says they may hold it.
|
||||
pub(crate) fn still_holding_note(e: &pdfcraft_engine::RedactionElsewhere) -> Option<String> {
|
||||
let count = |n: usize, one: &str, many: &str| match n {
|
||||
0 => None,
|
||||
1 => Some(tl!(one).to_string()),
|
||||
n => Some(crate::i18n::fmt(tl!(many), &[("n", &n.to_string())])),
|
||||
};
|
||||
let places: Vec<String> = [
|
||||
count(e.bookmarks, "1 bookmark title", "{n} bookmark titles"),
|
||||
count(e.attachments, "1 attachment", "{n} attachments"),
|
||||
count(e.comments, "1 comment", "{n} comments"),
|
||||
count(e.xmp, "1 other metadata stream", "{n} other metadata streams"),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.collect();
|
||||
let (last, rest) = places.split_last()?;
|
||||
let list = if rest.is_empty() { last.clone() } else { crate::i18n::fmt(tl!("{list} and {last}"), &[("list", &rest.join(", ")), ("last", last)]) };
|
||||
let template = if e.unsearched {
|
||||
"The redacted text couldn't be searched for, so {places} may still hold it. Remove hidden information removes them."
|
||||
} else {
|
||||
"The redacted text also appears in {places}, which redaction doesn't change. Remove hidden information removes them."
|
||||
};
|
||||
Some(crate::i18n::fmt(tl!(template), &[("places", &list)]))
|
||||
}
|
||||
|
||||
/// The sentence telling the user what became of the document's PDF/A or PDF/UA identification
|
||||
/// when applying redactions removed the XMP packet that held it (and the redacted text), or
|
||||
/// `None` when no such packet was removed.
|
||||
pub(crate) fn identification_note(id: &pdfcraft_engine::RedactionIdentification) -> Option<String> {
|
||||
use pdfcraft_engine::RedactionClaim as Claim;
|
||||
let named = |claim: Claim| -> Option<String> {
|
||||
match (id.pdfa == claim, id.pdfua == claim) {
|
||||
(true, true) => Some(tl!("PDF/A and PDF/UA").to_string()),
|
||||
(true, false) => Some("PDF/A".to_string()),
|
||||
(false, true) => Some("PDF/UA".to_string()),
|
||||
(false, false) => None,
|
||||
}
|
||||
};
|
||||
let kept = named(Claim::Kept).map(|s| crate::i18n::fmt(tl!("Its {standard} identification was written back on its own."), &[("standard", &s)]));
|
||||
let lost = named(Claim::Lost).map(|s| {
|
||||
crate::i18n::fmt(
|
||||
tl!("Its {standard} identification couldn't be kept, so the file no longer says it conforms to {standard}."),
|
||||
&[("standard", &s)],
|
||||
)
|
||||
});
|
||||
if kept.is_none() && lost.is_none() {
|
||||
return None;
|
||||
}
|
||||
let head = tl!("The document's XMP metadata held the redacted text and was removed.").to_string();
|
||||
Some([Some(head), kept, lost].into_iter().flatten().collect::<Vec<_>>().join(" "))
|
||||
}
|
||||
|
||||
/// Edits built from the dialog's drafts.
|
||||
fn finish_edits(app: &mut PdfCraftApp, dialog: Dialog, req: &mut Pending) {
|
||||
if req.props_now
|
||||
|
||||
@@ -233,12 +233,17 @@ impl PdfCraftApp {
|
||||
self.quick_tool = crate::QuickTool::Select;
|
||||
}
|
||||
// Text saved as typed but drawn with `?` (its font can't draw it), or pages whose
|
||||
// own backdrop stays off: said, not silent. Before the scripts' output, so an alert
|
||||
// they raise still shows.
|
||||
// own backdrop stays off, or places applied redactions left holding the removed text
|
||||
// (the Apply redactions dialog then says it after its count): said, not silent.
|
||||
// Before the scripts' output, so an alert they raise still shows.
|
||||
if let Some(n) = self.session.get(id).and_then(|d| d.edit_notices.first()).cloned() {
|
||||
self.notify(match &n {
|
||||
pdfcraft_engine::EditNotice::Undrawn(t) => crate::content_ui::undrawn_message(t),
|
||||
pdfcraft_engine::EditNotice::BackdropsLeftOff { pages } => crate::marks_ui::backdrops_left_off_message(pages),
|
||||
pdfcraft_engine::EditNotice::RedactionElsewhere(e) => crate::dialogs::still_holding_note(e).unwrap_or_else(|| n.to_string()),
|
||||
pdfcraft_engine::EditNotice::RedactionIdentification(id) => {
|
||||
crate::dialogs::identification_note(id).unwrap_or_else(|| n.to_string())
|
||||
}
|
||||
});
|
||||
}
|
||||
let out = self.session.take_js_output(id);
|
||||
|
||||
@@ -265,3 +265,116 @@ fn removing_hidden_information_and_sanitizing() {
|
||||
let s = h.state();
|
||||
assert_eq!(s.session.get(s.views[0].id).unwrap().can_undo(), Some("Sanitize document"));
|
||||
}
|
||||
|
||||
/// A one-page PDF showing "Codename BLUEFIN report", with bookmarks "Bluefin plans" and
|
||||
/// "Annual figures".
|
||||
fn bookmarked_pdf() -> Vec<u8> {
|
||||
let content = "BT /F1 18 Tf 20 150 Td (Codename BLUEFIN report) Tj ET";
|
||||
let objs = [
|
||||
"<< /Type /Catalog /Pages 2 0 R /Outlines 6 0 R >>".to_string(),
|
||||
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_string(),
|
||||
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>".to_string(),
|
||||
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>".to_string(),
|
||||
format!("<< /Length {} >>\nstream\n{content}\nendstream", content.len()),
|
||||
"<< /Type /Outlines /First 7 0 R /Last 8 0 R /Count 2 >>".to_string(),
|
||||
"<< /Title (Bluefin plans) /Parent 6 0 R /Next 8 0 R /Dest [3 0 R /Fit] >>".to_string(),
|
||||
"<< /Title (Annual figures) /Parent 6 0 R /Prev 7 0 R /Dest [3 0 R /Fit] >>".to_string(),
|
||||
];
|
||||
let mut out = b"%PDF-1.7\n".to_vec();
|
||||
let mut offsets = Vec::new();
|
||||
for (i, o) in objs.iter().enumerate() {
|
||||
offsets.push(out.len());
|
||||
out.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
|
||||
}
|
||||
let xref = out.len();
|
||||
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
|
||||
for o in offsets {
|
||||
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
|
||||
}
|
||||
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
|
||||
out
|
||||
}
|
||||
|
||||
/// #2539 review: applying redactions says when the removed text is still in a place it leaves
|
||||
/// as it is (here a bookmark title), in the same notification as the count of marks applied.
|
||||
#[test]
|
||||
fn applying_says_where_the_removed_text_is_still_held() {
|
||||
let mut h = Harness::builder().with_size(egui::vec2(1400.0, 900.0)).build_eframe(|_cc| {
|
||||
let mut app = PdfCraftApp::new();
|
||||
app.set_option("language", "en").unwrap();
|
||||
app.open_bytes("bluefin.pdf", None, bookmarked_pdf()).unwrap();
|
||||
app
|
||||
});
|
||||
crate::support::settle(&mut h);
|
||||
assert!(h.state_mut().execute("redact.mark"));
|
||||
h.run_steps(2);
|
||||
// Redact pages ▸ current page, then Redact all ▸ Apply.
|
||||
h.state_mut().execute("redact.pages");
|
||||
h.run_steps(2);
|
||||
h.get_by_label("OK").click();
|
||||
h.run_steps(3);
|
||||
assert_eq!(marks(&h), 1);
|
||||
h.get_by_label("Redact all").click();
|
||||
h.run_steps(3);
|
||||
h.get_by_label("Apply").click();
|
||||
h.run_steps(4);
|
||||
assert_eq!(marks(&h), 0);
|
||||
let toast = h.state().toast.as_ref().map(|(m, _)| m.clone()).unwrap_or_default();
|
||||
assert_eq!(
|
||||
toast,
|
||||
"Applied 1 redaction mark. Save to remove the content from the file. The redacted text also appears in 1 bookmark title, which redaction doesn't change. Remove hidden information removes them."
|
||||
);
|
||||
}
|
||||
|
||||
/// #2539 review: applying redactions that remove the document's XMP packet, which named the
|
||||
/// redacted word and said the file conforms to PDF/A, says that the identification was written
|
||||
/// back, in the same notification as the count of marks applied.
|
||||
#[test]
|
||||
fn applying_says_the_pdfa_identification_was_written_back() {
|
||||
let xmp = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" pdfaid:part="2" pdfaid:conformance="B"/><rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Bluefin report</dc:title></rdf:Description></rdf:RDF></x:xmpmeta><?xpacket end="w"?>"#;
|
||||
let content = "BT /F1 18 Tf 20 150 Td (Codename BLUEFIN report) Tj ET";
|
||||
let objs = [
|
||||
"<< /Type /Catalog /Pages 2 0 R /Metadata 6 0 R >>".to_string(),
|
||||
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_string(),
|
||||
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>".to_string(),
|
||||
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>".to_string(),
|
||||
format!("<< /Length {} >>\nstream\n{content}\nendstream", content.len()),
|
||||
format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{xmp}\nendstream", xmp.len()),
|
||||
];
|
||||
let mut bytes = b"%PDF-1.7\n".to_vec();
|
||||
let mut offsets = Vec::new();
|
||||
for (i, o) in objs.iter().enumerate() {
|
||||
offsets.push(bytes.len());
|
||||
bytes.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
|
||||
}
|
||||
let xref = bytes.len();
|
||||
bytes.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
|
||||
for o in offsets {
|
||||
bytes.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
|
||||
}
|
||||
bytes.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
|
||||
let mut h = Harness::builder().with_size(egui::vec2(1400.0, 900.0)).build_eframe(|_cc| {
|
||||
let mut app = PdfCraftApp::new();
|
||||
app.set_option("language", "en").unwrap();
|
||||
app.open_bytes("pdfa.pdf", None, bytes).unwrap();
|
||||
app
|
||||
});
|
||||
crate::support::settle(&mut h);
|
||||
assert!(h.state_mut().execute("redact.mark"));
|
||||
h.run_steps(2);
|
||||
h.state_mut().execute("redact.pages");
|
||||
h.run_steps(2);
|
||||
h.get_by_label("OK").click();
|
||||
h.run_steps(3);
|
||||
assert_eq!(marks(&h), 1);
|
||||
h.get_by_label("Redact all").click();
|
||||
h.run_steps(3);
|
||||
h.get_by_label("Apply").click();
|
||||
h.run_steps(4);
|
||||
assert_eq!(marks(&h), 0);
|
||||
let toast = h.state().toast.as_ref().map(|(m, _)| m.clone()).unwrap_or_default();
|
||||
assert_eq!(
|
||||
toast,
|
||||
"Applied 1 redaction mark. Save to remove the content from the file. The document's XMP metadata held the redacted text and was removed. Its PDF/A identification was written back on its own."
|
||||
);
|
||||
}
|
||||
@@ -5919,8 +5919,8 @@ milestone = "M8"
|
||||
status = "shipped"
|
||||
commands = ["redact.apply"]
|
||||
tools = ["redact_apply"]
|
||||
evidence = ["crates/ui-egui/tests/it/redact.rs::marking_applying_and_clearing", "crates/engine/src/tests.rs::redaction_marks_apply_for_good_and_undo", "crates/automation/tests/it/automation.rs::redacting_through_tools", "crates/engine/src/tests.rs::the_renderer_check_refuses_text_still_shown_under_a_mark", "crates/redact/src/tests.rs::redacted_xobjects_do_not_survive_through_inherited_or_shared_resources"]
|
||||
notes = "confirmation; one undo step; next save is a full rewrite. Refused, with nothing changed, when the page still shows text under a mark afterwards: text in some predefined CMaps and rotated Type 3 fonts is not removed yet, so those documents cannot be redacted. Also refused on a page with an inline image the renderer reads no image for, since the operators after it can't be known. Removed images and forms are unlinked from page-tree resources (own, inherited or shared) and from surviving forms' resources, not yet from annotation appearances or patterns"
|
||||
evidence = ["crates/ui-egui/tests/it/redact.rs::marking_applying_and_clearing", "crates/engine/src/tests.rs::redaction_marks_apply_for_good_and_undo", "crates/automation/tests/it/automation.rs::redacting_through_tools", "crates/engine/src/tests.rs::the_renderer_check_refuses_text_still_shown_under_a_mark", "crates/redact/src/tests.rs::redacted_xobjects_do_not_survive_through_inherited_or_shared_resources", "crates/redact/src/tests.rs::redacted_text_does_not_survive_in_info_or_xmp", "crates/redact/src/tests.rs::metadata_is_cleared_when_the_removed_text_was_cut_short", "crates/redact/src/tests/metadata_fail_closed.rs::an_xmp_stream_whose_filter_data_is_corrupt_goes", "crates/redact/src/tests/elsewhere.rs::places_apply_leaves_that_still_hold_the_removed_text_are_counted_and_kept", "crates/engine/src/tests.rs::applied_redactions_report_a_bookmark_that_still_holds_the_removed_text", "crates/automation/tests/it/automation.rs::redact_apply_reports_places_that_still_hold_the_removed_text", "crates/ui-egui/tests/it/redact.rs::applying_says_where_the_removed_text_is_still_held", "crates/redact/src/tests/metadata_matching.rs::a_heading_tracked_with_tj_numbers_matches_the_metadata", "crates/redact/src/tests/metadata_matching.rs::unicode_forms_of_the_removed_text_match", "crates/redact/src/tests/metadata_matching.rs::a_utf16le_string_is_searched_as_utf16le", "crates/redact/src/tests/identification.rs::the_identification_of_a_removed_packet_is_written_back", "crates/automation/tests/it/automation.rs::redact_apply_reports_the_identification_written_back", "crates/ui-egui/tests/it/redact.rs::applying_says_the_pdfa_identification_was_written_back"]
|
||||
notes = "confirmation; one undo step; next save is a full rewrite. Document information entries that contain the removed text, and the document's and pages' XMP streams that do, are removed (#2448; matched without regard to case or spacing, by the whole run of removed text and by its words of three letters or digits or more, and a value wholly part of a run as whole words, each also compared by letters and digits alone (so spacing, hyphens and TJ tracking don't hide it), after NFKC and lower case, with FF FE strings read as UTF-16LE; fail-closed: all of it goes when the removed text can't be known in full (too much to collect or search for, a glyph whose text isn't known, a form removed unread), and an XMP packet goes when it can't be decoded strictly, is over 16 MiB or isn't UTF-8 or UTF-16). When the document's XMP goes, its PDF/A and PDF/UA identification is written back in a packet holding only that, value by value when each is a valid code that doesn't hold the removed text, else reported lost (Report::identification, the notification, redact_apply's identification); /Info entries that remain aren't mirrored into that packet. Bookmark titles, attachments (names, file names, descriptions), the comments that remain and other objects' XMP are not changed: applying counts those that still hold the removed text, and the notification and redact_apply's still_holding say where (Remove hidden information removes them); when the removed text couldn't be searched for, every such place holding any text is counted. Refused, with nothing changed, when the page still shows text under a mark afterwards: text in some predefined CMaps and rotated Type 3 fonts is not removed yet, so those documents cannot be redacted. Also refused on a page with an inline image the renderer reads no image for, since the operators after it can't be known. Removed images and forms are unlinked from page-tree resources (own, inherited or shared) and from surviving forms' resources, not yet from annotation appearances or patterns"
|
||||
|
||||
[[feature]]
|
||||
id = "protect.redact-glyph-removal"
|
||||
|
||||
Reference in new issue
Block a user