Add craft-markdown: Markdown read and written, on craft-flow (stacked on #1025) (#1030)

* Add craft-wordmodel: WordCraft's document model without its editing code

WordCraft's crates/doc (wordcraft-doc, at 07f97737) moved as it is, so the DOCX,
RTF, ODT and other text-format crates can read into and write from one model.
Module paths, type names and serde attributes are unchanged: an app that
re-exports the crate keeps its native file format.

Left in the app: range editing (edit.rs), IME text and equation editing. Not
model, and already in craft-libs: code pages (craft-rtf), the ZIP entry-count
check (craft-opc), the compound-file guard (craft-cfb). The paragraph walks
moved to walk.rs and the unit helpers of wordcraft-geom to geom.rs.

* craft-wordmodel: math_tree, the arguments of an equation node and a walk over its runs

The DOCX reader and writer walk an equation's runs. The walk and the child
accessors it uses were in WordCraft's equation editor (math_edit.rs), which
stays in the app; they are structure, not editing.

* craft-wordmodel: keep the document operations (edit.rs) with the model

The DOCX and RTF readers use them: alternative-format chunks are merged into
a document with adopt_blocks, and a reader formats ranges with format_range
and inserts objects. They operate on the document, not on a view, so they
belong beside it. The tests that drove them come back with them; the
paragraph walks stay in walk.rs and edit re-exports them.

base64 0.23, as WordCraft and craft-offcrypto use.

* Add craft-flow: the flow model the text formats map to, on craft-wordmodel

WordCraft's crates/formats/src/model (07f97737) moved as it is: Flow,
FBlock, Para, Inline, Fmt, tables and cells, pictures, to_doc and
from_doc, picture sniffing, colours, base64 and the Dropped report. Only
the paths changed (wordcraft_doc is craft_wordmodel); the report helpers
are public so the format crates can call them; the image re-encoding
helper stays with WordCraft's RTF writer.

* Add craft-markdown: Markdown read and written, on craft-flow

WordCraft's crates/formats/src/markdown (07f97737) with its 35 tests,
moved onto craft-flow and craft-wordmodel. Character references come from
craft-html. The table cell budget lives in craft-flow, which names it in
its warning. Two test lines that wrote plain text through WordCraft's txt
module stay in WordCraft.
This commit is contained in:
Dylan M. Taylor authored and GitHub committed 2026-11-11 20:46:15 -05:00
1 parent 9eb25f7139
commit 1e3d3968c1
10 files changed
+2663

No files matched your search

Generated
+9
View File
@@ -1379,6 +1379,15 @@ dependencies = [
"thiserror 2.0.21",
]
[[package]]
name = "craft-markdown"
version = "0.1.0"
dependencies = [
"craft-flow",
"craft-html",
"craft-wordmodel",
]
[[package]]
name = "craft-math"
version = "0.1.0"
+1
View File
@@ -70,6 +70,7 @@ craft-assuan = { path = "crates/assuan" }
craft-kra = { path = "crates/kra" }
craft-audio = { path = "crates/audio" }
# No default features, so craft-audio's `encode` builds craft-media without `isobmff`.
craft-markdown = { path = "crates/markdown" }
craft-math = { path = "crates/math" }
craft-media = { path = "crates/media", default-features = false }
craft-media-testkit = { path = "crates/media-testkit" }
+1
View File
@@ -210,6 +210,7 @@ craft-<name> = { git = "https://github.com/storytold/craft-libs", rev = "<commit
- [`craft-audio`](crates/audio/README.md): audio file I/O and codecs (WAV, AIFF, FLAC, AAC, Opus,
AC-3/E-AC-3, MP1/MP2, MP3 encoding), streaming and seeking, and packet decoding for any demuxer. See its
README for formats and features.
- [`craft-markdown`](crates/markdown/README.md): Markdown read and written (a CommonMark subset with GitHub tables and strikethrough), to and from the craft-flow model.
- [`craft-math`](crates/math/README.md): equations as one tree (the Office Math object model) with its syntaxes, read and written: OMML, presentation MathML, LaTeX math and the UnicodeMath linear format.
- [`craft-media`](crates/media/README.md): bit-level I/O (`bitstream`), clean-room MP4/MOV
(`isobmff`), Matroska/WebM (`matroska`) and MXF (`mxf`) demuxers and muxers, Ogg (`ogg`), AVI (`avi`) and
+16
View File
@@ -0,0 +1,16 @@
[package]
name = "craft-markdown"
description = "Markdown read and written: a CommonMark subset with GitHub tables and strikethrough, to and from the craft-flow model and a craft-wordmodel Document."
version.workspace = true
edition.workspace = true
license.workspace = true
rust-version.workspace = true
repository.workspace = true
[lints]
workspace = true
[dependencies]
craft-flow = { workspace = true }
craft-html = { workspace = true }
craft-wordmodel = { workspace = true }
+23
View File
@@ -0,0 +1,23 @@
# craft-markdown
Markdown, read and written: a CommonMark subset plus GitHub pipe tables and strikethrough, to and
from the [craft-flow](../flow/README.md) model and a craft-wordmodel `Document`.
- Read (`import`, `import_on`, `parse` for the flow): ATX and setext headings, paragraphs with
soft and hard breaks, block quotes, fenced and indented code, thematic breaks, bullet and
ordered lists nested by indentation (with their start number), GFM tables, link reference
definitions; inlines by the CommonMark delimiter algorithm: emphasis and strong, `~~strike~~`,
code spans, inline, reference and autolinks, images (`data:` URIs become pictures, others their
alt text), backslash escapes, character references, HTML comments, `<br>` and `<a id>` anchors.
- Write (`export`): the same constructs; pictures become `![alt](name)`.
```rust
let doc = craft_markdown::import("# Title\n\nSome *emphasis* and a [link](https://example.com).\n");
assert_eq!(craft_markdown::export(&doc).trim_end(), "# Title\n\nSome *emphasis* and a [link](https://example.com).");
```
Input is hostile. Nesting is bounded, and table cells are capped (`MAX_TABLE_CELLS` in all, plus
the model's row and column limits), with one warning per kind of thing left out.
Moved from WordCraft (`crates/formats/src/markdown`, `07f97737`) with its tests; character
references come from craft-html.
+727
View File
@@ -0,0 +1,727 @@
use super::inlines::parse_inlines;
use super::{Align, Arc, Cell, Document, FBlock, FTable, Fmt, HashMap, Kind, ListInfo, MAX_COLS, MAX_ROWS, Para, model};
/// Nesting limit for block containers (quotes, list items) and inline links.
pub(super) const MAX_NEST: usize = 32;
// ---------------------------------------------------------------------------------------------
// Import
/// Parse Markdown text into a document.
pub fn import(text: &str) -> Document {
model::begin_import();
model::to_doc(&parse(text))
}
/// [`import`] onto `page`: Markdown gives no page size (#1381).
pub fn import_on(text: &str, page: &craft_wordmodel::SectionProps) -> Document {
model::begin_import();
model::to_doc_in(&parse(text), page.clone())
}
/// Link reference definitions: the normalised label to its destination.
/// A destination is shared by every reference to it, not copied into each (#3713).
pub type Refs = HashMap<String, Arc<str>>;
/// A label as CommonMark compares them: case-folded, and any run of white space one space
/// (§4.7, "matches"). `[FOO bar]` and `[foo BAR]` are the same label.
pub(super) fn ref_label(raw: &str) -> String {
raw.split_whitespace().collect::<Vec<_>>().join(" ").to_lowercase()
}
/// A link reference definition on its own line: `[label]: destination "optional title"`. The
/// destination may be wrapped in `<...>`. Returns the label and the destination.
///
/// CommonMark allows the title on the following line and the label to span lines; a definition
/// written across lines is left as text here, which is what it was before these were read at all.
fn link_definition(line: &str) -> Option<(String, String)> {
let t = line.trim_start();
// Up to three spaces of indentation; four makes it indented code.
if line.len() - t.len() > 3 || !t.starts_with('[') {
return None;
}
let bytes = t.as_bytes();
// The label ends at the first unescaped `]`, and cannot be empty or hold an unescaped `[`.
let mut i = 1;
let mut close = None;
while i < bytes.len() {
match bytes[i] {
b'\\' => i += 1,
b'[' => return None,
b']' => {
close = Some(i);
break;
}
_ => {}
}
i += 1;
}
let close = close?;
let label = ref_label(t.get(1..close)?);
// `[^label]: ...` is a footnote definition, which this importer leaves as text.
if label.starts_with('^') {
return None;
}
if label.is_empty() || t.as_bytes().get(close + 1) != Some(&b':') {
return None;
}
let rest = t.get(close + 2..)?.trim_start();
if rest.is_empty() {
return None;
}
// The destination: `<...>`, else up to the next space. What follows may only be a title.
let (dest, after) = match rest.strip_prefix('<') {
Some(inner) => {
let end = inner.find('>')?;
(inner.get(..end)?.to_string(), inner.get(end + 1..)?)
}
None => {
let end = rest.find(char::is_whitespace).unwrap_or(rest.len());
(rest.get(..end)?.to_string(), rest.get(end..)?)
}
};
if dest.is_empty() {
return None;
}
let title = after.trim();
let titled =
title.is_empty() || (title.len() >= 2 && matches!((title.chars().next()?, title.chars().last()?), ('"', '"') | ('\'', '\'') | ('(', ')')));
titled.then_some((label, dest))
}
/// Take the link reference definitions out of `lines`, leaving the rest. Lines inside a fenced
/// code block are left alone: a definition there is code, not a definition. Fences open and
/// close by the same rules [`parse_blocks`] applies, or a line it reads as code would be read
/// here as a definition and deleted.
///
/// A definition has to start a block: CommonMark §4.7 forbids one from interrupting a paragraph,
/// and a line like `[Warning]: unstable` in the middle of one is the author's text. This pass
/// sees lines without the block structure around them, so it takes a definition only after a
/// blank line, at the start of the document, after a heading or thematic break, or after another
/// definition. A definition anywhere else stays in the document and reads as text, which is what
/// it did before definitions were understood at all; deleting a line that was part of a
/// paragraph would lose it silently.
fn take_definitions(lines: &mut Vec<String>) -> Refs {
let mut refs = Refs::new();
let mut open: Option<(char, usize)> = None;
// The start of the document begins a block.
let mut starts_block = true;
lines.retain(|line| {
let t = line.trim_start_matches([' ', '\t']);
// Four spaces of indentation is code, not a fence (`parse_blocks`), on both sides.
let marker = (indent(line) < 4).then(|| fence(t)).flatten();
match open {
Some((c, n)) => {
// A closing fence holds nothing but its own run of fence characters.
if marker.is_some_and(|(fc, fl)| fc == c && fl >= n) && t.trim_start_matches(c).trim().is_empty() {
open = None;
starts_block = true;
}
return true;
}
None => {
if let Some(f) = marker {
open = Some(f);
return true;
}
}
}
if starts_block && let Some((label, dest)) = link_definition(line) {
// The first definition of a label wins, as CommonMark says, and definitions may
// follow one another.
refs.entry(label).or_insert(dest.into());
return false;
}
starts_block = is_blank(line) || atx_heading(t).is_some() || thematic_break(t);
true
});
refs
}
/// Parse Markdown text into the flow model.
pub fn parse(text: &str) -> model::Flow {
let text = text.strip_prefix('\u{FEFF}').unwrap_or(text);
let mut lines: Vec<String> = text.split('\n').map(|l| expand_tabs(l.trim_end_matches('\r'))).collect();
let refs = take_definitions(&mut lines);
let budget = std::cell::Cell::new(super::MAX_TABLE_CELLS);
let mut blocks = parse_blocks(&lines, 0, 0, &refs, &budget);
// A leading "Title" from front matter isn't supported; a first-level heading stays a heading.
if blocks.is_empty() {
blocks.push(FBlock::Para(Para::default()));
}
model::Flow { blocks, meta: Default::default() }
}
/// Leading tabs → spaces (tab stops of 4); the rest of the line is kept.
fn expand_tabs(l: &str) -> String {
if !l.starts_with([' ', '\t']) || !l.contains('\t') {
return l.to_string();
}
let mut out = String::with_capacity(l.len() + 8);
let mut col = 0usize;
let mut rest = l;
while let Some(c) = rest.chars().next() {
match c {
' ' => {
out.push(' ');
col += 1;
}
'\t' => {
let n = 4 - col % 4;
out.push_str(&" ".repeat(n));
col += n;
}
_ => break,
}
rest = &rest[c.len_utf8()..];
}
out.push_str(rest);
out
}
fn indent(l: &str) -> usize {
l.len() - l.trim_start_matches(' ').len()
}
fn is_blank(l: &str) -> bool {
l.trim().is_empty()
}
/// Strip up to `n` leading spaces.
fn strip_indent(l: &str, n: usize) -> &str {
let k = indent(l).min(n);
l.get(k..).unwrap_or("")
}
fn atx_heading(t: &str) -> Option<(u8, &str)> {
let n = t.chars().take_while(|c| *c == '#').count();
if !(1..=6).contains(&n) {
return None;
}
let rest = t.get(n..)?;
if !rest.is_empty() && !rest.starts_with([' ', '\t']) {
return None;
}
let mut body = rest.trim();
// Optional closing sequence.
let trimmed = body.trim_end_matches('#');
if trimmed.len() != body.len() && (trimmed.is_empty() || trimmed.ends_with(' ')) {
body = trimmed.trim_end();
}
Some((n as u8, body))
}
pub(super) fn thematic_break(t: &str) -> bool {
let mut ch = None;
let mut n = 0;
for c in t.chars() {
match c {
' ' | '\t' => {}
'*' | '-' | '_' => {
if ch.is_some_and(|x| x != c) {
return false;
}
ch = Some(c);
n += 1;
}
_ => return false,
}
}
n >= 3
}
fn fence(t: &str) -> Option<(char, usize)> {
let c = t.chars().next().filter(|c| *c == '`' || *c == '~')?;
let n = t.chars().take_while(|x| *x == c).count();
if n < 3 {
return None;
}
if c == '`' && t.get(n..).is_some_and(|info| info.contains('`')) {
return None;
}
Some((c, n))
}
#[derive(Clone, Copy, PartialEq)]
struct Marker {
ordered: bool,
/// Bullet char or ordered delimiter.
ch: char,
/// Bytes of the marker itself (after the indent).
len: usize,
start: u32,
}
fn list_marker(t: &str) -> Option<Marker> {
let first = t.chars().next()?;
let (m, len) = if matches!(first, '-' | '+' | '*') {
(Marker { ordered: false, ch: first, len: 1, start: 1 }, 1)
} else {
let digits = t.chars().take_while(char::is_ascii_digit).count();
if !(1..=9).contains(&digits) {
return None;
}
let d = t.get(digits..)?.chars().next()?;
if d != '.' && d != ')' {
return None;
}
let start = t.get(..digits)?.parse::<u32>().unwrap_or(1);
(Marker { ordered: true, ch: d, len: digits + 1, start }, digits + 1)
};
let rest = t.get(len..)?;
if rest.is_empty() || rest.starts_with([' ', '\t']) { Some(m) } else { None }
}
/// A table row without its leading and trailing pipes (an escaped last pipe stays).
fn row_body(l: &str) -> &str {
let t = l.trim();
let t = t.strip_prefix('|').unwrap_or(t);
if let Some(before) = t.strip_suffix('|') {
let num_bs = before.chars().rev().take_while(|&c| c == '\\').count();
if num_bs % 2 == 0 { before } else { t }
} else {
t
}
}
/// The first `keep` cells of a table row, and how many cells it has: the rest are only counted.
pub(super) fn split_row(l: &str, keep: usize) -> (Vec<String>, usize) {
let mut cells = Vec::new();
let mut count = 0usize;
let mut cur = String::new();
let mut chars = row_body(l).chars().peekable();
while let Some(c) = chars.next() {
let full = cells.len() >= keep;
match c {
'\\' => match chars.peek() {
Some(&'|') => {
if !full {
cur.push('|');
}
chars.next();
}
Some(&'\\') => {
if !full {
cur.push_str("\\\\");
}
chars.next();
}
_ if !full => cur.push('\\'),
_ => {}
},
'|' => {
if !full {
cells.push(std::mem::take(&mut cur).trim().to_string());
}
count += 1;
}
_ if !full => cur.push(c),
_ => {}
}
}
if cells.len() < keep {
cells.push(cur.trim().to_string());
}
(cells, count + 1)
}
/// The alignments of the first `keep` cells of a delimiter row, and how many cells it has; every
/// cell is checked, so a bad one past `keep` still means it isn't a delimiter row.
fn delimiter_row(l: &str, keep: usize) -> Option<(Vec<Option<Align>>, usize)> {
if !l.contains('-') {
return None;
}
let mut out = Vec::new();
let mut count = 0usize;
// A valid cell is only dashes, colons and spaces, so an escaped pipe can't be in one.
for c in row_body(l).split('|') {
let c = c.trim();
let left = c.starts_with(':');
let right = c.ends_with(':');
let core = c.trim_matches(':');
if core.is_empty() || !core.chars().all(|x| x == '-') {
return None;
}
if out.len() < keep {
out.push(match (left, right) {
(true, true) => Some(Align::Center),
(false, true) => Some(Align::Right),
(true, false) => Some(Align::Left),
_ => None,
});
}
count += 1;
}
Some((out, count))
}
/// Does this line start a block that interrupts a paragraph?
fn interrupts(l: &str) -> bool {
if indent(l) >= 4 {
return false;
}
let t = l.trim_start_matches([' ', '\t']);
atx_heading(t).is_some()
|| thematic_break(t)
|| fence(t).is_some()
|| t.starts_with('>')
|| t.starts_with("<!--")
|| list_marker(t).is_some_and(|m| (!m.ordered || m.start == 1) && !is_blank(t.get(m.len..).unwrap_or("")))
}
fn flush_para(lines: &mut Vec<String>, out: &mut Vec<FBlock>, depth: usize, refs: &Refs) {
if lines.is_empty() {
return;
}
let text = join_lines(lines);
lines.clear();
let inlines = parse_inlines(&text, &Fmt::default(), depth, refs);
if inlines.is_empty() {
return;
}
let p = Para { inlines, ..Default::default() };
out.push(FBlock::Para(p));
}
/// Paragraph lines → text with `\n` for hard breaks and spaces for soft ones.
fn join_lines(lines: &[String]) -> String {
let mut s = String::new();
let n = lines.len();
for (i, l) in lines.iter().enumerate() {
let l = l.trim_start_matches([' ', '\t']);
if i + 1 == n {
s.push_str(l.trim_end());
} else if l.ends_with(" ") {
s.push_str(l.trim_end());
s.push('\n');
} else if l.ends_with('\\') && !l.ends_with("\\\\") {
s.push_str(&l[..l.len() - 1]);
s.push('\n');
} else {
s.push_str(l.trim_end());
s.push(' ');
}
}
s
}
fn code_para(line: &str) -> FBlock {
let mut p = Para::new(Kind::Code);
p.push_text(line, &Fmt::default());
FBlock::Para(p)
}
/// Parse block structure. `level` is the list nesting level for lists found here.
/// `budget`: the table cells the document may still make ([`super::MAX_TABLE_CELLS`]).
fn parse_blocks(lines: &[String], depth: usize, level: u8, refs: &Refs, budget: &std::cell::Cell<usize>) -> Vec<FBlock> {
let mut out = Vec::new();
let mut para: Vec<String> = Vec::new();
let mut i = 0;
while let Some(line) = lines.get(i) {
if is_blank(line) {
flush_para(&mut para, &mut out, depth, refs);
i += 1;
continue;
}
let ind = indent(line);
if ind >= 4 {
if !para.is_empty() {
para.push(line.clone());
i += 1;
continue;
}
parse_indented_code(lines, &mut i, &mut out);
continue;
}
let t = line.trim_start_matches([' ', '\t']);
// Setext heading underline.
if !para.is_empty() {
let u = t.trim_end();
if !u.is_empty() && (u.chars().all(|c| c == '=') || (u.chars().all(|c| c == '-'))) {
let lvl = if u.starts_with('=') { 1 } else { 2 };
let text = join_lines(&para);
para.clear();
let mut p = Para::new(Kind::Heading(lvl));
p.inlines = parse_inlines(&text, &Fmt::default(), depth, refs);
out.push(FBlock::Para(p));
i += 1;
continue;
}
}
if let Some((n, body)) = atx_heading(t) {
flush_para(&mut para, &mut out, depth, refs);
let mut p = Para::new(Kind::Heading(n));
p.inlines = parse_inlines(body, &Fmt::default(), depth, refs);
out.push(FBlock::Para(p));
i += 1;
continue;
}
if thematic_break(t) {
flush_para(&mut para, &mut out, depth, refs);
out.push(FBlock::Para(Para::new(Kind::Rule)));
i += 1;
continue;
}
if let Some((fc, fl)) = fence(t) {
flush_para(&mut para, &mut out, depth, refs);
i += 1;
while let Some(l) = lines.get(i) {
i += 1;
let lt = l.trim_start_matches([' ', '\t']);
if indent(l) < 4 && fence(lt).is_some_and(|(c, n)| c == fc && n >= fl) && lt.trim_start_matches(fc).trim().is_empty() {
break;
}
out.push(code_para(strip_indent(l, ind)));
}
continue;
}
if t.starts_with("<!--") {
flush_para(&mut para, &mut out, depth, refs);
while let Some(l) = lines.get(i) {
i += 1;
if l.contains("-->") {
break;
}
}
continue;
}
if t.starts_with('>') {
flush_para(&mut para, &mut out, depth, refs);
parse_quote(lines, &mut i, depth, level, refs, budget, &mut out);
continue;
}
if para.is_empty()
&& t.contains('|')
&& let Some((aligns, width)) = lines.get(i + 1).and_then(|l| delimiter_row(l, MAX_COLS))
{
let (head, head_width) = split_row(t, MAX_COLS);
if head_width == width {
let mut table = FTable::default();
let cell = |s: &str, header: bool, a: Option<Align>| {
let p = Para { align: a, inlines: parse_inlines(s, &Fmt::default(), depth, refs), ..Default::default() };
Cell { blocks: vec![FBlock::Para(p)], header, ..Default::default() }
};
// The document keeps no more columns or rows than this, nor cells past its budget,
// so those aren't built; their lines are still the table's.
let cols = aligns.len();
let fits = |rows: usize| rows < MAX_ROWS && budget.get() >= cols;
let mut cut = head_width > cols;
if fits(0) {
budget.set(budget.get() - cols);
table.rows.push(head.iter().zip(&aligns).map(|(s, a)| cell(s, true, *a)).collect());
} else {
cut = true;
}
i += 2;
while let Some(l) = lines.get(i) {
if is_blank(l) || !l.contains('|') || interrupts(l) {
break;
}
if !table.rows.is_empty() && fits(table.rows.len()) {
budget.set(budget.get() - cols);
let (mut cells, _) = split_row(l, cols);
cells.resize(cols, String::new());
table.rows.push(cells.iter().zip(&aligns).map(|(s, a)| cell(s, false, *a)).collect());
} else {
cut = true;
}
i += 1;
}
if cut {
model::dropped(model::Dropped::TableCells);
}
if !table.rows.is_empty() {
out.push(FBlock::Table(table));
}
continue;
}
}
if let Some(m) = list_marker(t)
&& (para.is_empty() || ((!m.ordered || m.start == 1) && !is_blank(t.get(m.len..).unwrap_or(""))))
{
flush_para(&mut para, &mut out, depth, refs);
i = parse_list(lines, i, depth, level, &mut out, refs, budget);
continue;
}
para.push(line.clone());
i += 1;
}
flush_para(&mut para, &mut out, depth, refs);
out
}
fn parse_indented_code(lines: &[String], i: &mut usize, out: &mut Vec<FBlock>) {
let mut code: Vec<&str> = Vec::new();
while let Some(line) = lines.get(*i) {
if is_blank(line) {
code.push("");
} else if indent(line) >= 4 {
code.push(strip_indent(line, 4));
} else {
break;
}
*i += 1;
}
while code.last().is_some_and(|line| line.is_empty()) {
code.pop();
}
out.extend(code.into_iter().map(code_para));
}
fn parse_quote(lines: &[String], i: &mut usize, depth: usize, level: u8, refs: &Refs, budget: &std::cell::Cell<usize>, out: &mut Vec<FBlock>) {
let mut inner = Vec::new();
while let Some(line) = lines.get(*i) {
let trimmed = line.trim_start_matches([' ', '\t']);
if indent(line) < 4 && trimmed.starts_with('>') {
let content = &trimmed[1..];
inner.push(content.strip_prefix(' ').unwrap_or(content).to_string());
} else if !is_blank(line) && !interrupts(line) && inner.last().is_some_and(|line| !is_blank(line)) {
inner.push(line.clone());
} else {
break;
}
*i += 1;
}
if depth >= MAX_NEST {
let mut p = Para::new(Kind::Quote);
p.push_text(&inner.join(" "), &Fmt::default());
out.push(FBlock::Para(p));
return;
}
for mut block in parse_blocks(&inner, depth + 1, level, refs, budget) {
if let FBlock::Para(p) = &mut block
&& p.kind == Kind::Normal
{
p.kind = Kind::Quote;
}
out.push(block);
}
}
fn marker_space(ind: usize, marker_len: usize, after: &str) -> (usize, usize) {
let mut cols = 0;
let mut bytes = 0;
let mut rest = after;
while let Some(c) = rest.chars().next() {
match c {
' ' => {
cols += 1;
bytes += 1;
rest = &rest[1..];
if cols >= 4 {
break;
}
}
'\t' => {
let col = ind + marker_len + cols;
let n = 4 - col % 4;
cols += n;
bytes += 1;
break;
}
_ => break,
}
}
(cols, bytes)
}
/// Parse a list starting at `lines[i]`; returns the index after it.
fn parse_list(lines: &[String], mut i: usize, depth: usize, level: u8, out: &mut Vec<FBlock>, refs: &Refs, budget: &std::cell::Cell<usize>) -> usize {
let Some(first) = lines.get(i).and_then(|l| list_marker(l.trim_start_matches([' ', '\t']))) else { return i + 1 };
while let Some(line) = lines.get(i) {
let ind = indent(line);
if ind >= 4 {
break;
}
let t = line.trim_start_matches([' ', '\t']);
let Some(m) = list_marker(t) else { break };
if m.ordered != first.ordered || m.ch != first.ch {
break;
}
let after = t.get(m.len..).unwrap_or("");
let (spaces, skip) = if indent(after) > 4 {
(1, 1)
} else if after.starts_with([' ', '\t']) {
marker_space(ind, m.len, after)
} else {
(0, 0)
};
let content_indent = if is_blank(after) { ind + m.len + 1 } else { ind + m.len + spaces };
let item0 = if is_blank(after) { String::new() } else { after.get(skip.min(after.len())..).unwrap_or("").to_string() };
let mut item: Vec<String> = vec![item0];
i += 1;
while let Some(l) = lines.get(i) {
if is_blank(l) {
item.push(String::new());
} else if indent(l) >= content_indent {
item.push(strip_indent(l, content_indent).to_string());
} else if item.last().is_some_and(|x| !is_blank(x)) && !interrupts(l) && list_marker(l.trim_start_matches([' ', '\t'])).is_none() {
// Lazy continuation of the item's paragraph.
item.push(l.trim_start_matches([' ', '\t']).to_string());
} else {
break;
}
i += 1;
}
// Blank lines at the end of an item belong between items.
while item.len() > 1 && item.last().is_some_and(String::is_empty) {
item.pop();
}
let li = ListInfo { ordered: first.ordered, level: level.min(8), start: first.start, ..Default::default() };
let inner = if depth >= MAX_NEST {
let mut p = Para::default();
p.push_text(&item.join(" "), &Fmt::default());
vec![FBlock::Para(p)]
} else {
parse_blocks(&item, depth + 1, level.saturating_add(1).min(8), refs, budget)
};
let mut placed = false;
if inner.is_empty() {
out.push(FBlock::Para(Para { list: Some(li), ..Default::default() }));
placed = true;
}
for b in inner {
match b {
FBlock::Para(mut p) if p.list.is_none() => {
if !placed {
p.list = Some(li);
placed = true;
out.push(FBlock::Para(p));
} else if let Some(FBlock::Para(prev)) = out.last_mut()
&& prev.list.is_some_and(|x| x.level == li.level)
{
// A continuation paragraph joins the item with a line break.
prev.push_text("\n", &Fmt::default());
prev.inlines.extend(p.inlines);
} else {
p.list = Some(li);
out.push(FBlock::Para(p));
}
}
FBlock::Para(p) => {
if !placed {
// The item starts with a nested list: give it an empty first line.
out.push(FBlock::Para(Para { list: Some(li), ..Default::default() }));
placed = true;
}
out.push(FBlock::Para(p));
}
other => {
placed = true;
out.push(other);
}
}
}
// Blank lines between items.
while lines.get(i).is_some_and(|l| is_blank(l)) {
let next_is_item = lines.get(i + 1).is_some_and(|l| {
indent(l) < 4 && list_marker(l.trim_start_matches([' ', '\t'])).is_some_and(|m| m.ordered == first.ordered && m.ch == first.ch)
});
if !next_is_item {
return i;
}
i += 1;
}
}
i
}
+448
View File
@@ -0,0 +1,448 @@
use super::blocks::thematic_break;
use super::inlines::{is_punct, within};
use super::{Align, Arc, Cell, Document, FBlock, FTable, Fmt, Inline, Kind, model};
// ---------------------------------------------------------------------------------------------
// Export
/// Escape text for Markdown (inline context).
pub fn escape(s: &str) -> String {
let mut out = String::with_capacity(s.len() + 8);
for c in s.chars() {
match c {
'\\' | '`' | '*' | '_' | '[' | ']' | '<' | '>' | '~' | '|' | '&' => {
out.push('\\');
out.push(c);
}
'\u{000C}' | '\u{000E}' => {}
_ => out.push(c),
}
}
out
}
/// Escape what would start a block at the beginning of a line. Leading spaces are kept, but at
/// most three (a tab counts as four): four would turn the line into an indented code block.
pub(super) fn escape_line_start(s: String) -> String {
let t = s.trim_start_matches([' ', '\t']);
let lead = s.get(..s.len() - t.len()).unwrap_or("");
let cols: usize = lead.chars().map(|c| if c == '\t' { 4 } else { 1 }).sum();
let indent = " ".repeat(cols.min(3));
let needs = t.starts_with('#')
|| t.starts_with('>')
|| t.starts_with("- ")
|| t.starts_with("+ ")
|| t.starts_with("-\t")
|| t.starts_with("+\t")
|| t == "-"
|| t == "+"
|| t.starts_with('=')
|| (t.starts_with('-') && t.trim_end_matches([' ', '\t']).chars().all(|c| c == '-'))
|| thematic_break(t)
|| {
let d = t.chars().take_while(char::is_ascii_digit).count();
d > 0
&& t.get(d..).is_some_and(|r| {
r.starts_with(". ") || r.starts_with(") ") || r.starts_with(".\t") || r.starts_with(")\t") || r == "." || r == ")"
})
};
if !needs {
return if cols > 3 || lead.contains('\t') { format!("{indent}{t}") } else { s };
}
if let Some(d) = t.find(['.', ')']).filter(|_| t.starts_with(|c: char| c.is_ascii_digit())) {
return format!("{indent}{}\\{}", &t[..d], &t[d..]);
}
format!("{indent}\\{t}")
}
/// [`escape_line_start`] for every line of a body, including those after a manual line break.
fn escape_lines(body: String) -> String {
if !body.contains("\\\n") {
return escape_line_start(body);
}
body.split("\\\n").map(|l| escape_line_start(l.to_string())).collect::<Vec<_>>().join("\\\n")
}
fn code_span(t: &str) -> String {
let t = t.replace("\r\n", " ").replace(['\r', '\n'], " ");
let mut longest = 0;
let mut cur = 0;
for c in t.chars() {
if c == '`' {
cur += 1;
longest = longest.max(cur);
} else {
cur = 0;
}
}
let ticks = "`".repeat(longest + 1);
let pad = if t.starts_with('`') || t.ends_with('`') || (t.starts_with(' ') && t.ends_with(' ') && !t.trim().is_empty()) { " " } else { "" };
format!("{ticks}{pad}{t}{pad}{ticks}")
}
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug)]
enum Mark {
Bold,
Italic,
Strike,
}
/// Inline spans → Markdown with properly nested markers.
pub(super) fn inlines_md(inlines: &[Inline]) -> String {
inlines_md_impl(inlines, false)
}
fn inlines_md_table(inlines: &[Inline]) -> String {
inlines_md_impl(inlines, true)
}
fn inlines_md_impl(inlines: &[Inline], in_table: bool) -> String {
// Split into (text, fmt) pieces with whitespace moved outside emphasis.
let mut pieces: Vec<(String, Fmt, Option<String>)> = Vec::new();
let eq = craft_flow::equations_as_text(inlines);
// Markdown keeps no ruby: its base text goes through the text path below.
let inlines = craft_flow::ruby_as_text(&eq);
for i in inlines.iter() {
match i {
Inline::Text(t, f) => {
let has_emph = f.bold || f.italic || f.strike;
if has_emph && !f.code {
let core = t.trim_matches([' ', '\t', '\n']);
let lead = &t[..t.len() - t.trim_start_matches([' ', '\t', '\n']).len()];
let trail = &t[t.trim_end_matches([' ', '\t', '\n']).len()..];
let plain = Fmt { bold: false, italic: false, strike: false, ..f.clone() };
if !lead.is_empty() {
pieces.push((lead.to_string(), plain.clone(), None));
}
if !core.is_empty() {
pieces.push((core.to_string(), f.clone(), None));
}
if !trail.is_empty() && !core.is_empty() {
pieces.push((trail.to_string(), plain, None));
}
} else {
pieces.push((t.clone(), f.clone(), None));
}
}
Inline::Image(img) => {
let name = format!("image.{}", img.ext);
pieces.push((String::new(), Fmt::default(), Some(format!("![{}]({})", escape(&img.alt), name))));
}
Inline::Anchor(a) => {
let safe: String = a.chars().filter(|c| c.is_alphanumeric() || matches!(c, '_' | '-' | '.' | ':')).collect();
pieces.push((String::new(), Fmt::default(), Some(format!("<a id=\"{safe}\"></a>"))));
}
Inline::Figure(alt) | Inline::Chart(craft_flow::Chart { alt, .. }) => {
pieces.push((String::new(), Fmt::default(), Some(format!("![{}]()", escape(alt)))))
}
Inline::Equation { .. } => {}
// ASK and SET show nothing (see `model::shows_nothing`).
Inline::Field { instr, .. } if model::shows_nothing(instr) => {}
Inline::Field { result, fmt, .. } | Inline::Control { text: result, fmt, .. } => pieces.push((result.clone(), fmt.clone(), None)),
Inline::Note { .. }
| Inline::Comment { .. }
| Inline::CommentEnd(_)
| Inline::Shape { .. }
| Inline::Group { .. }
| Inline::FieldStart(_)
| Inline::FieldEnd
| Inline::AnchorEnd(_) => {}
// Read as its base text by `ruby_as_text` above.
Inline::Ruby { .. } => {}
}
}
let mut out = String::new();
let mut stack: Vec<Mark> = Vec::new();
let mut link: Option<Arc<str>> = None;
let close_all = |out: &mut String, stack: &mut Vec<Mark>| {
while let Some(m) = stack.pop() {
out.push_str(marker(m));
}
};
for (text, f, raw) in pieces {
// Links are the outermost construct; a script or other non-web target is plain text.
let f_link = f.link.clone().filter(|l| craft_wordmodel::link::is_safe_href(l));
if f_link != link || raw.is_some() {
close_all(&mut out, &mut stack);
if let Some(l) = &link {
out.push_str(&format!("]({})", link_target(l)));
}
link = None;
if let Some(r) = raw {
out.push_str(&r);
continue;
}
if let Some(l) = f_link {
out.push('[');
link = Some(l);
}
}
let want: Vec<Mark> =
[(f.bold, Mark::Bold), (f.italic, Mark::Italic), (f.strike, Mark::Strike)].iter().filter(|x| x.0).map(|x| x.1).collect();
let body = if f.code {
if in_table { code_span(&text.replace('|', "\\|")) } else { code_span(&text) }
} else {
escape(&text).replace('\n', "\\\n").replace('\t', " ")
};
// Close until the stack is a prefix of what we want.
let mut close = String::new();
while let Some(pos) = stack.iter().position(|m| !want.contains(m)) {
while stack.len() > pos {
if let Some(m) = stack.pop() {
close.push_str(marker(m));
}
}
}
let mut open = String::new();
for m in want {
if !stack.contains(&m) {
open.push_str(marker(m));
stack.push(m);
}
}
// CommonMark only reads a closing run after punctuation as closing when whitespace or
// punctuation follows it, and an opening run before punctuation as opening when whitespace
// or punctuation precedes it. Where a letter sits there instead, or an opening run would
// directly follow a closing run (the two would merge into one run), an empty HTML comment (which readers drop) supplies
// the punctuation.
let prev = out.chars().next_back();
let next = body.chars().next();
let word = |c: Option<char>| c.is_some_and(|c| !c.is_whitespace() && !is_punct(c));
out.push_str(&close);
let close_needs = !close.is_empty() && prev.is_some_and(is_punct) && (!open.is_empty() || word(next));
let open_needs = !open.is_empty() && next.is_some_and(is_punct) && (!close.is_empty() || word(prev));
if close_needs || open_needs {
out.push_str(EMPTY_COMMENT);
}
out.push_str(&open);
out.push_str(&body);
}
close_all(&mut out, &mut stack);
if let Some(l) = &link {
out.push_str(&format!("]({})", link_target(l)));
}
out
}
/// `l` as a Markdown link destination that reads back as exactly `l`, the target
/// `is_safe_href` passed. What Markdown would decode (a backslash escape, an entity reference)
/// or end the destination on (`<`, `>`) is backslash-escaped, never deleted: deleting `<` or `>`
/// could turn an accepted relative target into `javascript:…` or `//host`. Control characters
/// can't be written in a destination and are left out, as the safety check ignores them.
fn link_target(l: &str) -> String {
let l: String = l.chars().filter(|c| !c.is_ascii_control()).collect();
let mut out = String::with_capacity(l.len() + 2);
for (i, c) in l.char_indices() {
let rest = l.get(i + c.len_utf8()..).unwrap_or("");
match c {
'<' | '>' => out.push('\\'),
// A backslash is literal unless punctuation (or the closing `)`/`>`) follows it.
'\\' if rest.chars().next().is_none_or(|n| n.is_ascii_punctuation()) => out.push('\\'),
'&' if entity_like(rest) => out.push('\\'),
_ => {}
}
out.push(c);
}
if l.contains(|c: char| c.is_whitespace() || matches!(c, '(' | ')' | '<' | '>')) { format!("<{out}>") } else { out }
}
/// Whether `rest` (the text after a `&`) starts like an entity reference (`amp;`, `#106;`,
/// `#x6A;`), which a Markdown reader would decode.
fn entity_like(rest: &str) -> bool {
let Some(end) = within(rest, 34).find(';') else { return false };
let body = rest.get(..end).unwrap_or("");
let body = body.strip_prefix('#').map_or(body, |n| n.strip_prefix(['x', 'X']).unwrap_or(n));
!body.is_empty() && body.chars().all(|c| c.is_ascii_alphanumeric())
}
/// Inline HTML that Markdown readers drop, used to separate emphasis markers from letters.
const EMPTY_COMMENT: &str = "<!-- -->";
fn marker(m: Mark) -> &'static str {
match m {
Mark::Bold => "**",
Mark::Italic => "*",
Mark::Strike => "~~",
}
}
pub(super) fn table_md(t: &FTable, out: &mut Vec<String>) {
let cols = t.cols().max(1);
let cell_md = |c: &Cell| -> String {
let mut parts = Vec::new();
for b in &c.blocks {
match b {
FBlock::Para(p) => parts.push(inlines_md_table(&p.inlines).replace("\\\n", "<br>")),
FBlock::Table(inner) => parts.push(escape(&inner.rows.iter().flat_map(|r| r.iter().map(Cell::text)).collect::<Vec<_>>().join(" "))),
}
}
parts.join("<br>").replace('\n', " ")
};
let align_of = |c: &Cell| c.blocks.iter().find_map(|b| if let FBlock::Para(p) = b { Some(p.align) } else { None }).flatten();
let row_md = |r: &Vec<Cell>| -> Vec<String> {
let mut v: Vec<String> = Vec::new();
for c in r {
v.push(if c.covered { String::new() } else { cell_md(c) });
for _ in 1..c.colspan.clamp(1, 63) {
v.push(String::new());
}
}
v.resize(cols, String::new());
v
};
let Some(head) = t.rows.first() else { return };
out.push(format!("| {} |", row_md(head).join(" | ")));
let mut aligns: Vec<Option<Align>> = Vec::new();
for c in head {
aligns.push(align_of(c));
for _ in 1..c.colspan.clamp(1, 63) {
aligns.push(None);
}
}
aligns.resize(cols, None);
let delim: Vec<&str> = aligns
.iter()
.map(|a| match a {
Some(Align::Center) => ":---:",
Some(Align::Right) => "---:",
Some(Align::Left) => ":---",
_ => "---",
})
.collect();
out.push(format!("| {} |", delim.join(" | ")));
for r in t.rows.iter().skip(1) {
out.push(format!("| {} |", row_md(r).join(" | ")));
}
}
/// Markdown text of a document.
pub fn export(doc: &Document) -> String {
let flow = model::from_doc(doc);
let mut out: Vec<String> = Vec::new();
write_blocks(&flow.blocks, &mut out);
let mut s = out.join("\n");
while s.ends_with("\n\n") {
s.pop();
}
if !s.ends_with('\n') {
s.push('\n');
}
s
}
/// A heading's text on one line, with a trailing run of '#' escaped so a reader does not take it
/// for the optional closing sequence (`## Item #` reads back as "Item").
fn heading_text(body: &str) -> String {
let mut s = body.replace("\\\n", " ");
let t = s.trim_end();
let core = t.trim_end_matches('#');
if core.len() != t.len() && (core.is_empty() || core.ends_with([' ', '\t'])) {
s.insert(core.len(), '\\');
}
s
}
pub(super) fn write_blocks(blocks: &[FBlock], out: &mut Vec<String>) {
let mut i = 0;
let mut counters = [0u32; 9];
let mut prev_list = false;
let mut prev_top_ordered = false;
while let Some(b) = blocks.get(i) {
match b {
FBlock::Table(t) => {
table_md(t, out);
out.push(String::new());
prev_list = false;
}
FBlock::Para(p) if p.kind == Kind::Code && p.list.is_none() => {
let mut lines = Vec::new();
while let Some(FBlock::Para(q)) = blocks.get(i) {
if q.kind != Kind::Code || q.list.is_some() {
break;
}
lines.push(q.text());
i += 1;
}
let longest = lines.iter().map(|l| l.chars().filter(|c| *c == '`').count()).max().unwrap_or(0);
// The fence must be longer than the longest run in the content, or a content
// line is read back as the closing fence (the `.min(64)` cap did exactly that).
let fence = "`".repeat(3.max(longest + 1));
out.push(fence.clone());
out.extend(lines);
out.push(fence);
out.push(String::new());
prev_list = false;
continue;
}
FBlock::Para(p) => {
let body = inlines_md(&p.inlines);
match p.list {
Some(li) => {
let lv = li.level.min(8) as usize;
if !prev_list || (lv == 0 && prev_top_ordered != li.ordered) {
counters = [0; 9];
}
if lv == 0 {
prev_top_ordered = li.ordered;
}
let n = counters.get_mut(lv).map(|c| {
if *c == 0 {
*c = li.start.saturating_sub(1);
}
*c = c.saturating_add(1);
*c
});
for d in counters.iter_mut().skip(lv + 1) {
*d = 0;
}
let marker = if li.ordered { format!("{}.", n.unwrap_or(1)) } else { "-".to_string() };
let pad = " ".repeat(4 * lv);
let cont = " ".repeat(4 * lv + marker.len() + 1);
let body = if p.kind == Kind::Code {
code_span(&p.text())
} else {
let body = escape_lines(body).replace("\\\n", &format!("\\\n{cont}"));
// The bullet marker can turn a short run of hyphens into a rule.
if thematic_break(&format!("{marker} {body}")) { format!("\\{body}") } else { body }
};
let line = format!("{pad}{marker} {body}").trim_end().to_string();
// A list inside a block quote stays in it: every line of the item, its
// continuation lines included, takes the quote marker.
let line = if p.kind == Kind::Quote { format!("> {}", line.replace('\n', "\n> ")) } else { line };
out.push(line);
prev_list = true;
// A list going into or out of a block quote is a new block: without the
// blank line, an unquoted item would be read back into the quote.
let quoted = |k: Kind| k == Kind::Quote;
let next_is_list = matches!(blocks.get(i + 1), Some(FBlock::Para(q)) if q.list.is_some() && quoted(q.kind) == quoted(p.kind));
if !next_is_list {
out.push(String::new());
}
}
None => {
prev_list = false;
let line = match p.kind {
Kind::Heading(n) => format!("{} {}", "#".repeat(n.clamp(1, 6) as usize), heading_text(&body)),
Kind::Title => format!("# {}", heading_text(&body)),
Kind::Rule => "---".to_string(),
Kind::Quote => {
let b = escape_lines(body);
format!("> {}", b.replace("\\\n", "\\\n> "))
}
_ => {
if body.is_empty() {
i += 1;
continue;
}
escape_lines(body)
}
};
out.push(line);
out.push(String::new());
}
}
}
}
i += 1;
}
}
+595
View File
@@ -0,0 +1,595 @@
use super::blocks::{MAX_NEST, Refs, ref_label};
use super::{Arc, Fmt, HashMap, Inline, decode_entity, make_img, model};
// ---- inlines ----
#[derive(Clone, Copy, PartialEq, Eq)]
enum Emph {
Em,
Strong,
Strike,
}
enum Node {
Text(String, Fmt),
Inl(Inline),
Delim { ch: char, orig: usize, left: usize, open: bool, close: bool, opens: Vec<Emph>, closes: Vec<Emph> },
}
pub(super) fn is_punct(c: char) -> bool {
c.is_ascii_punctuation() || (!c.is_alphanumeric() && !c.is_whitespace() && !c.is_control())
}
/// Parse inline Markdown into formatted spans (on top of `base`).
pub fn parse_inlines(s: &str, base: &Fmt, depth: usize, refs: &Refs) -> Vec<Inline> {
let nodes = tokenize(s, base, depth, refs);
emit(nodes)
}
fn push_text(nodes: &mut Vec<Node>, s: &str, f: &Fmt) {
if s.is_empty() {
return;
}
if let Some(Node::Text(t, pf)) = nodes.last_mut()
&& pf == f
{
t.push_str(s);
return;
}
nodes.push(Node::Text(s.to_string(), f.clone()));
}
/// The closing `]` matching the `[` at byte `open` (escapes and code spans respected). Gives up
/// after looking `max` bytes ahead. Records in `known` every bracket pair it passes, and every `[`
/// still open at the end of the text: a scan from one of those would take the same path.
fn matching_bracket(s: &str, open: usize, max: usize, ticks: &Backticks, known: &mut HashMap<usize, Option<usize>>) -> Option<usize> {
let b = s.as_bytes();
let mut opens: Vec<usize> = Vec::new();
let mut i = open;
while let Some(&c) = b.get(i) {
if i - open > max {
return None;
}
match c {
b'\\' => i += 1,
b'`' => {
let n = b.get(i..).map(|r| r.iter().take_while(|x| **x == b'`').count()).unwrap_or(1);
if let Some(end) = ticks.find(i + n, n) {
i = end + n - 1;
} else {
i += n - 1;
}
}
b'[' => opens.push(i),
b']' => {
if let Some(o) = opens.pop() {
known.insert(o, Some(i));
}
if opens.is_empty() {
return Some(i);
}
}
_ => {}
}
i += 1;
}
for o in opens {
known.insert(o, None);
}
None
}
/// Where a text's runs of backticks start, by run length. Finding the run that closes a code span
/// is then a lookup: scanning the rest of the text for it, again for every run nothing closes
/// (each of a different length) and from every link bracket over them, was superlinear.
struct Backticks(HashMap<usize, Vec<usize>>);
impl Backticks {
fn new(s: &str) -> Self {
let b = s.as_bytes();
let mut runs: HashMap<usize, Vec<usize>> = HashMap::new();
let mut i = 0;
while let Some(&c) = b.get(i) {
if c == b'`' {
let k = b.get(i..).map(|r| r.iter().take_while(|x| **x == b'`').count()).unwrap_or(1);
runs.entry(k).or_default().push(i);
i += k;
} else {
i += 1;
}
}
Self(runs)
}
/// Byte index of a run of exactly `n` backticks at or after `from` (which is not inside a run).
fn find(&self, from: usize, n: usize) -> Option<usize> {
let starts = self.0.get(&n)?;
starts.get(starts.partition_point(|&p| p < from)).copied()
}
}
struct LinkScan<'a> {
text: &'a str,
refs: &'a Refs,
ticks: &'a Backticks,
known: HashMap<usize, Option<usize>>,
budget: usize,
}
impl<'a> LinkScan<'a> {
fn parse_at(&mut self, open: usize) -> Option<(&'a str, Arc<str>, usize)> {
let bytes = self.text.as_bytes();
let close = match self.known.get(&open).copied() {
Some(found) => found,
None if self.budget > 0 => {
let found = matching_bracket(self.text, open, self.budget, self.ticks, &mut self.known);
self.budget = self.budget.saturating_sub(found.unwrap_or(self.text.len()).saturating_sub(open));
found
}
None => None,
}?;
if self.budget == 0 {
return None;
}
let label_text = self.text.get(open + 1..close)?;
if bytes.get(close + 1) == Some(&b'(') {
let destination = link_dest(self.text, close + 1);
self.budget = self.budget.saturating_sub(destination.as_ref().map_or(self.text.len(), |d| d.1).saturating_sub(close));
let (destination, end) = destination?;
return Some((label_text, Arc::from(destination), end));
}
// Full and shortcut reference links use definitions collected before inline parsing.
if self.refs.is_empty() {
return None;
}
let (label, end) = match self.text.get(close + 1..).and_then(|rest| rest.strip_prefix('[')) {
Some(rest) => {
let ahead = within(rest, self.budget);
let shut = ahead.find(']');
self.budget = self.budget.saturating_sub(shut.map_or(ahead.len(), |n| n + 1));
let shut = shut?;
let inner = rest.get(..shut)?;
let label = if inner.trim().is_empty() { label_text } else { inner };
(label, close + 2 + shut + 1)
}
None => (label_text, close + 1),
};
// A cached bracket pair still costs a pass over its label. Without charging it,
// nested undefined references normalize every suffix again.
if label.len() > self.budget {
self.budget = 0;
return None;
}
self.budget -= label.len();
Some((label_text, self.refs.get(&ref_label(label))?.clone(), end))
}
}
/// Parse `(dest "title")` at byte `at` (which holds `(`): (destination, index after `)`).
fn link_dest(s: &str, at: usize) -> Option<(String, usize)> {
let rest = s.get(at..)?.strip_prefix('(')?;
let mut i = at + 1;
let ws = rest.len() - rest.trim_start().len();
i += ws;
let r = s.get(i..)?;
let (dest, used) = if let Some(inner) = r.strip_prefix('<') {
// The first `>` not escaped by a backslash ends it; a line break first means no link.
let b = inner.as_bytes();
let mut k = 0;
let end = loop {
match b.get(k)? {
b'\\' => k += 1,
b'>' => break k,
b'\n' => return None,
_ => {}
}
k += 1;
};
(inner.get(..end)?.to_string(), end + 2)
} else {
let mut depth = 0i32;
let mut end = r.len();
let mut chars = r.char_indices().peekable();
while let Some((k, c)) = chars.next() {
match c {
'\\' => {
chars.next();
}
'(' => depth += 1,
')' if depth == 0 => {
end = k;
break;
}
')' => depth -= 1,
c if c.is_whitespace() => {
end = k;
break;
}
_ => {}
}
}
(r.get(..end)?.to_string(), end)
};
i += used;
let r = s.get(i..)?;
let ws = r.len() - r.trim_start().len();
i += ws;
let r = s.get(i..)?;
if let Some(q) = r.chars().next().filter(|c| matches!(c, '"' | '\'' | '(')) {
let close = if q == '(' { ')' } else { q };
let end = r.get(1..)?.find(close)?;
i += end + 2;
let r = s.get(i..)?;
i += r.len() - r.trim_start().len();
}
if s.get(i..)?.starts_with(')') { Some((unescape(&dest), i + 1)) } else { None }
}
fn unescape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut chars = s.chars().peekable();
while let Some(c) = chars.next() {
if c == '\\'
&& let Some(n) = chars.peek().copied().filter(char::is_ascii_punctuation)
{
out.push(n);
chars.next();
} else {
out.push(c);
}
}
out
}
/// The value of attribute `name` in an HTML tag's text.
fn tag_attr(tag: &str, name: &str) -> Option<String> {
let lower = tag.to_ascii_lowercase();
let mut from = 0;
while let Some(k) = lower.get(from..)?.find(name) {
let at = from + k;
let before = lower.get(..at)?.chars().next_back();
let after = lower.get(at + name.len()..)?.trim_start();
if before.is_some_and(char::is_whitespace) && after.starts_with('=') {
let v = after.get(1..)?.trim_start();
let q = v.chars().next()?;
return Some(if q == '"' || q == '\'' {
let inner = v.get(1..)?;
let off = tag.len() - inner.len();
let end = inner.find(q)?;
tag.get(off..off + end)?.to_string()
} else {
let end = v.find(|c: char| c.is_whitespace() || c == '>').unwrap_or(v.len());
let off = tag.len() - v.len();
tag.get(off..off + end)?.to_string()
});
}
from = at + name.len();
}
None
}
fn tokenize(s: &str, base: &Fmt, depth: usize, refs: &Refs) -> Vec<Node> {
let mut nodes: Vec<Node> = Vec::new();
// Bytes the link/image scans and reference-label lookups may read in total. Unmatched
// brackets and nested undefined references would otherwise rescan the rest of the text,
// which is quadratic on hostile input. Once the budget is gone, further brackets are plain
// text. A scan pays for what it reads; earlier bracket matches are looked up in `known`.
let ticks = Backticks::new(s);
let mut links = LinkScan { text: s, refs, ticks: &ticks, known: HashMap::new(), budget: s.len().saturating_mul(8).saturating_add(8192) };
let b = s.as_bytes();
let mut i = 0usize;
let mut text_start = 0usize;
macro_rules! flush {
() => {
if text_start < i {
push_text(&mut nodes, s.get(text_start..i).unwrap_or(""), base);
}
};
}
while let Some(&c) = b.get(i) {
match c {
b'\\' => {
flush!();
match s.get(i + 1..).and_then(|r| r.chars().next()) {
Some(n) if n.is_ascii_punctuation() => {
push_text(&mut nodes, &n.to_string(), base);
i += 2;
}
_ => {
push_text(&mut nodes, "\\", base);
i += 1;
}
}
text_start = i;
}
b'`' => {
flush!();
let (code, fmt, end) = code_span(s, i, base, &ticks);
push_text(&mut nodes, &code, &fmt);
i = end;
text_start = i;
}
b'*' | b'_' | b'~' => {
flush!();
let ch = c as char;
let n = b.get(i..).map(|r| r.iter().take_while(|x| **x == c).count()).unwrap_or(1);
let before = s.get(..i).and_then(|r| r.chars().next_back()).unwrap_or(' ');
let after = s.get(i + n..).and_then(|r| r.chars().next()).unwrap_or(' ');
let lf = !after.is_whitespace() && (!is_punct(after) || before.is_whitespace() || is_punct(before));
let rf = !before.is_whitespace() && (!is_punct(before) || after.is_whitespace() || is_punct(after));
let (open, close) = match ch {
'_' => (lf && (!rf || is_punct(before)), rf && (!lf || is_punct(after))),
'~' if n != 2 => (false, false),
_ => (lf, rf),
};
if open || close {
nodes.push(Node::Delim { ch, orig: n, left: n, open, close, opens: Vec::new(), closes: Vec::new() });
} else {
push_text(&mut nodes, s.get(i..i + n).unwrap_or(""), base);
}
i += n;
text_start = i;
}
b'!' | b'[' if depth < MAX_NEST => {
let img = c == b'!';
let open = if img { i + 1 } else { i };
let parsed = if b.get(open) == Some(&b'[') { links.parse_at(open) } else { None };
match parsed {
Some((inner, dest, end)) if img => {
flush!();
let alt: String = parse_inlines(inner, &Fmt::default(), depth + 1, refs)
.iter()
.filter_map(|x| if let Inline::Text(t, _) = x { Some(t.as_str()) } else { None })
.collect();
match model::data_uri(&dest).and_then(|d| make_img(d, None, None, &alt)) {
Some(im) => nodes.push(Node::Inl(Inline::Image(im))),
None => push_text(&mut nodes, &alt, base),
}
i = end;
text_start = i;
}
Some((inner, dest, end)) if base.link.is_none() => {
flush!();
let mut f = base.clone();
// A destination too long to keep leaves its text unlinked.
f.link = (dest.len() <= model::MAX_LINK).then_some(dest);
if f.link.is_none() {
model::dropped(model::Dropped::Link);
}
for x in parse_inlines(inner, &f, depth + 1, refs) {
nodes.push(Node::Inl(x));
}
i = end;
text_start = i;
}
_ => i += 1,
}
}
b'<' => {
if let Some((end, parsed)) = angle_inlines(s, i, base) {
flush!();
nodes.extend(parsed);
i = end;
text_start = end;
} else {
i += 1;
}
}
b'&' => {
let rest = s.get(i..).unwrap_or("");
match within(rest, 33).find(';').and_then(|k| decode_entity(rest.get(1..k)?).map(|ch| (ch, k))) {
Some((ch, k)) => {
flush!();
push_text(&mut nodes, &ch, base);
i += k + 1;
text_start = i;
}
None => i += 1,
}
}
_ => i += 1,
}
}
flush!();
nodes
}
fn angle_inlines(s: &str, start: usize, base: &Fmt) -> Option<(usize, Vec<Node>)> {
if let Some(pos) = s.get(start..).and_then(|rest| rest.strip_prefix("<!--")).and_then(|rest| rest.find("-->")) {
return Some((start + 4 + pos + 3, Vec::new()));
}
let rest = s.get(start + 1..).unwrap_or("");
let tag_end = within(rest, MAX_TAG).find('>')?;
let inner = rest.get(..tag_end)?;
let lower = inner.to_ascii_lowercase();
let end = start + inner.len() + 2;
let mut nodes = Vec::new();
if !inner.is_empty()
&& !inner.contains([' ', '<', '\n'])
&& (inner.contains("://") || lower.starts_with("mailto:") || (inner.contains('@') && !inner.contains('/')))
{
let mut fmt = base.clone();
fmt.link = Some(if inner.contains('@') && !inner.contains(':') { format!("mailto:{inner}").into() } else { inner.into() });
push_text(&mut nodes, inner, &fmt);
Some((end, nodes))
} else if matches!(lower.trim_end_matches('/').trim(), "br") {
push_text(&mut nodes, "\n", base);
Some((end, nodes))
} else if lower.starts_with("a ") && rest.get(inner.len() + 1..).is_some_and(|tail| tail.to_ascii_lowercase().starts_with("</a>")) {
if let Some(name) = tag_attr(inner, "id").or_else(|| tag_attr(inner, "name")) {
nodes.push(Node::Inl(Inline::Anchor(name)));
}
Some((end + 4, nodes))
} else {
None
}
}
fn code_span(s: &str, start: usize, base: &Fmt, ticks: &Backticks) -> (String, Fmt, usize) {
let bytes = s.as_bytes();
let count = bytes.get(start..).map(|tail| tail.iter().take_while(|byte| **byte == b'`').count()).unwrap_or(1);
let Some(end) = ticks.find(start + count, count) else {
return (s.get(start..start + count).unwrap_or("").to_string(), base.clone(), start + count);
};
let mut code = s.get(start + count..end).unwrap_or("").replace('\n', " ");
if code.len() >= 2 && code.starts_with(' ') && code.ends_with(' ') && !code.trim().is_empty() {
code = code[1..code.len() - 1].to_string();
}
let mut fmt = base.clone();
fmt.code = true;
(code, fmt, end + count)
}
/// Longest `<...>` tag or autolink taken as inline HTML.
const MAX_TAG: usize = 4096;
/// The first `n` bytes of `s` (fewer if that would cut a character), so a search for a closing
/// character stays short however long the text is.
pub(super) fn within(s: &str, n: usize) -> &str {
let mut end = n.min(s.len());
while !s.is_char_boundary(end) {
end -= 1;
}
s.get(..end).unwrap_or("")
}
fn emph_index(ch: char) -> usize {
match ch {
'*' => 0,
'_' => 1,
_ => 2,
}
}
/// The CommonMark "process emphasis" pass over the delimiter runs, then flattening into spans.
fn emit(mut nodes: Vec<Node>) -> Vec<Inline> {
// Delimiter stack of node indexes; bottoms per (char, closer length % 3, closer can open).
let mut stack: Vec<usize> = Vec::new();
let mut bottom = [[[None::<usize>; 2]; 3]; 3];
for c in 0..nodes.len() {
let (ch, corig, copen, cclose) = match nodes.get(c) {
Some(Node::Delim { ch, orig, open, close, .. }) => (*ch, *orig, *open, *close),
_ => continue,
};
if cclose {
loop {
let cleft = match nodes.get(c) {
Some(Node::Delim { left, .. }) => *left,
_ => 0,
};
if cleft == 0 {
break;
}
let bot = bottom.get(emph_index(ch)).and_then(|x| x.get(corig % 3)).and_then(|x| x.get(copen as usize)).copied().flatten();
let mut found = None;
for (k, &o) in stack.iter().enumerate().rev() {
if bot.is_some_and(|b| o <= b) {
break;
}
if let Some(Node::Delim { ch: och, orig: oorig, left: oleft, open: oopen, close: oclose, .. }) = nodes.get(o) {
if *och != ch || !*oopen || *oleft == 0 {
continue;
}
if ch == '~' && (*oleft < 2 || cleft < 2) {
continue;
}
let odd = (*oclose || copen) && (oorig + corig) % 3 == 0 && !(oorig % 3 == 0 && corig % 3 == 0);
if odd {
continue;
}
found = Some((k, o, *oleft));
break;
}
}
let Some((k, o, oleft)) = found else {
if let Some(slot) = bottom.get_mut(emph_index(ch)).and_then(|x| x.get_mut(corig % 3)).and_then(|x| x.get_mut(copen as usize)) {
*slot = Some(c.saturating_sub(1));
}
break;
};
let (used, kind) = if ch == '~' {
(2, Emph::Strike)
} else if oleft >= 2 && cleft >= 2 {
(2, Emph::Strong)
} else {
(1, Emph::Em)
};
if let Some(Node::Delim { left, opens, .. }) = nodes.get_mut(o) {
*left -= used;
opens.push(kind);
}
if let Some(Node::Delim { left, closes, .. }) = nodes.get_mut(c) {
*left -= used;
closes.push(kind);
}
// Delimiters between opener and closer can no longer match.
stack.truncate(k + 1);
if oleft - used == 0 {
stack.truncate(k);
}
}
}
let left = match nodes.get(c) {
Some(Node::Delim { left, .. }) => *left,
_ => 0,
};
if copen && left > 0 {
stack.push(c);
}
}
// Flatten.
let mut out: Vec<Inline> = Vec::new();
let (mut em, mut strong, mut strike) = (0usize, 0usize, 0usize);
let push = |out: &mut Vec<Inline>, t: &str, f: &Fmt, em: usize, strong: usize, strike: usize| {
if t.is_empty() {
return;
}
let mut f = f.clone();
f.italic |= em > 0;
f.bold |= strong > 0;
f.strike |= strike > 0;
if let Some(Inline::Text(pt, pf)) = out.last_mut()
&& *pf == f
{
pt.push_str(t);
return;
}
out.push(Inline::Text(t.to_string(), f));
};
// Delimiter literals take the formatting (link, …) of the text before them: the last text
// node, tracked here rather than searched for backwards from every delimiter (quadratic on
// a paragraph of delimiters).
let mut last_text: Option<&Fmt> = None;
for n in &nodes {
match n {
Node::Text(t, f) => {
push(&mut out, t, f, em, strong, strike);
last_text = Some(f);
}
Node::Inl(Inline::Text(t, f)) => push(&mut out, t, f, em, strong, strike),
Node::Inl(other) => out.push(other.clone()),
Node::Delim { ch, left, opens, closes, .. } => {
for e in closes {
match e {
Emph::Em => em = em.saturating_sub(1),
Emph::Strong => strong = strong.saturating_sub(1),
Emph::Strike => strike = strike.saturating_sub(1),
}
}
if *left > 0 {
let lit: String = std::iter::repeat_n(*ch, *left).collect();
let f = last_text.map_or_else(Fmt::default, |f| Fmt { bold: false, italic: false, strike: false, ..f.clone() });
push(&mut out, &lit, &f, em, strong, strike);
}
for e in opens {
match e {
Emph::Em => em += 1,
Emph::Strong => strong += 1,
Emph::Strike => strike += 1,
}
}
}
}
}
out
}
+48
View File
@@ -0,0 +1,48 @@
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
// The code came from an app whose lints allow this.
#![allow(clippy::type_complexity)]
//! Markdown: a CommonMark subset plus GitHub tables and strikethrough.
//!
//! Import: ATX and setext headings, paragraphs with soft/hard breaks, block quotes (→ Quote
//! style), fenced and indented code (→ one Code paragraph per line), thematic breaks (→ a
//! bottom-border paragraph), bullet and ordered lists (nested by indentation), GFM pipe tables,
//! and inlines: emphasis/strong (the CommonMark delimiter algorithm), `~~strike~~`, code spans,
//! inline links, autolinks, images (`data:` URIs become pictures, others their alt text),
//! backslash escapes, entities, HTML comments (`<!-- ... -->`), `<br>` and `<a id>` anchors.
//!
//! Export writes the same constructs; pictures become `![alt](media-name)` placeholders.
#[cfg(doctest)]
#[doc = include_str!("../README.md")]
struct ReadmeExamples;
use std::collections::HashMap;
use std::sync::Arc;
use craft_wordmodel::table::{MAX_COLS, MAX_ROWS};
use craft_wordmodel::{Align, Document};
use craft_flow::{self as model, Cell, FBlock, FTable, Fmt, Inline, Kind, ListInfo, Para, make_img};
mod blocks;
mod export;
mod inlines;
pub use craft_flow::MAX_TABLE_CELLS;
/// The text of an entity body (`amp`, `#38`, `#x26`).
fn decode_entity(body: &str) -> Option<String> {
craft_html::decode_entity(body).map(String::from)
}
pub use blocks::{Refs, import, import_on, parse};
pub use export::escape;
pub use export::export;
#[cfg(test)]
use export::{escape_line_start, inlines_md, table_md, write_blocks};
pub use inlines::parse_inlines;
#[cfg(test)]
use blocks::split_row;
#[cfg(test)]
mod tests;
+795
View File
@@ -0,0 +1,795 @@
use super::*;
fn inl(s: &str) -> Vec<Inline> {
parse_inlines(s, &Fmt::default(), 0, &Refs::new())
}
fn spans(s: &str) -> Vec<(String, bool, bool)> {
inl(s).into_iter().filter_map(|i| if let Inline::Text(t, f) = i { Some((t, f.bold, f.italic)) } else { None }).collect()
}
#[test]
fn emphasis() {
assert_eq!(spans("a *b* c"), vec![("a ".into(), false, false), ("b".into(), false, true), (" c".into(), false, false)]);
assert_eq!(spans("**b**"), vec![("b".into(), true, false)]);
assert_eq!(spans("***x***"), vec![("x".into(), true, true)]);
assert_eq!(spans("*a **b** c*"), vec![("a ".into(), false, true), ("b".into(), true, true), (" c".into(), false, true)]);
assert_eq!(spans("snake_case_name"), vec![("snake_case_name".into(), false, false)]);
assert_eq!(spans("2 * 3 * 4"), vec![("2 * 3 * 4".into(), false, false)]);
assert_eq!(spans("**unclosed"), vec![("**unclosed".into(), false, false)]);
let s = inl("~~gone~~");
assert!(matches!(&s[0], Inline::Text(t, f) if t == "gone" && f.strike));
}
#[test]
fn links_code_escapes() {
let s = inl("see [the *site*](http://x.org \"T\") and `a*b*` \\*lit\\* &amp; <https://y.com>");
let link: Vec<_> = s
.iter()
.filter_map(|i| if let Inline::Text(t, f) = i { f.link.as_ref().map(|l| (t.clone(), l.to_string(), f.italic)) } else { None })
.collect();
assert_eq!(link[0], ("the ".into(), "http://x.org".into(), false));
assert_eq!(link[1], ("site".into(), "http://x.org".into(), true));
assert!(link.iter().any(|l| l.1 == "https://y.com"));
assert!(s.iter().any(|i| matches!(i, Inline::Text(t, f) if t == "a*b*" && f.code)));
let text: String = s.iter().filter_map(|i| if let Inline::Text(t, _) = i { Some(t.as_str()) } else { None }).collect();
assert!(text.contains("*lit* & "), "{text}");
}
/// Reference links: `[text][label]`, `[text][]` and `[text]` take their destination from a
/// definition line, and the definition itself leaves the document. Before, none of the three
/// was a link and the definition stayed as a paragraph of its own (#3238).
#[test]
fn reference_links_resolve_and_their_definitions_leave_the_body() {
let links = |md: &str| -> Vec<(String, String)> {
match &*parse(md).blocks {
[FBlock::Para(p), ..] => p
.inlines
.iter()
.filter_map(|x| match x {
Inline::Text(t, f) => f.link.as_deref().map(|l| (t.clone(), l.to_string())),
_ => None,
})
.collect(),
_ => vec![],
}
};
let def = "\n\n[ref1]: https://example.com/ref\n";
let to_ref = |text: &str| vec![(text.to_string(), "https://example.com/ref".to_string())];
assert_eq!(links(&format!("[click here][ref1]{def}")), to_ref("click here"), "full");
assert_eq!(links(&format!("[ref1][]{def}")), to_ref("ref1"), "collapsed");
assert_eq!(links(&format!("[ref1]{def}")), to_ref("ref1"), "shortcut");
// Labels match case-folded with runs of white space collapsed (CommonMark §4.7).
assert_eq!(links(&format!("[click here][ REF1 ]{def}")), to_ref("click here"), "folded label");
// An undefined label is not a link: an ordinary bracketed phrase stays text.
assert_eq!(links(&format!("[nope]{def}")), vec![], "undefined");
// Inline links are unaffected.
assert_eq!(links("[x](https://inline.example)"), vec![("x".into(), "https://inline.example".into())], "inline");
// The definition is not a paragraph, and its text is nowhere in the body.
let doc = import(&format!("Reference link: [click here][ref1]{def}"));
let text = doc.plain_text(craft_wordmodel::StoryRef::Body);
assert_eq!(text, "Reference link: click here", "{text:?}");
// A definition inside a fence is code, not a definition.
let fenced = "```\n[ref1]: https://example.com/ref\n```\n\n[ref1]\n";
let body = import(fenced).plain_text(craft_wordmodel::StoryRef::Body);
assert!(body.contains("[ref1]: https://example.com/ref"), "the fenced definition is still code: {body:?}");
assert_eq!(links(fenced), vec![], "a fenced definition defines nothing");
}
/// Cached bracket pairs must not make undefined shortcut references normalize every nested
/// suffix again. One unrelated definition keeps this on the reference-lookup path.
#[test]
fn nested_undefined_reference_labels_have_bounded_work() {
let literal = format!("prefix {}x{} suffix", "[".repeat(128_000), "]".repeat(128_000));
let src = format!("{literal}\n\n[other]: https://example.com/other\n");
let started = std::time::Instant::now();
let doc = import(&src);
let elapsed = started.elapsed();
assert_eq!(doc.plain_text(craft_wordmodel::StoryRef::Body), literal);
assert!(elapsed < std::time::Duration::from_secs(10), "nested undefined references took {elapsed:?}");
}
#[test]
fn reference_lookup_keeps_long_unicode_labels_and_nested_text() {
// The work guard is not a label-length cap. Collapsed/full/shortcut references keep their
// case and whitespace matching, even when lowercasing changes the number of UTF-8 bytes.
let label = format!("{}İ END", "Écho ".repeat(1500));
let key = label.to_lowercase();
let src = format!("[{label}] [{label}][] [full][{label}]\n\n[{key}]: https://example.com/long\n");
let flow = parse(&src);
let FBlock::Para(p) = &flow.blocks[0] else { panic!("paragraph") };
let links: Vec<_> = p
.inlines
.iter()
.filter_map(|x| match x {
Inline::Text(t, f) if f.link.is_some() => Some((t.as_str(), f.link.as_deref())),
_ => None,
})
.collect();
assert_eq!(
links,
vec![
(label.as_str(), Some("https://example.com/long")),
(label.as_str(), Some("https://example.com/long")),
("full", Some("https://example.com/long"))
]
);
let refs = Refs::from([("known".into(), Arc::from("https://example.com/known"))]);
let out = parse_inlines("[outer [known]] tail [inline](https://example.com/inline)", &Fmt::default(), 0, &refs);
let text: String = out.iter().filter_map(|x| if let Inline::Text(t, _) = x { Some(t.as_str()) } else { None }).collect();
assert_eq!(text, "[outer known] tail inline");
assert!(out.iter().any(|x| matches!(x, Inline::Text(t, f) if t == "known" && f.link.as_deref() == Some("https://example.com/known"))));
assert!(out.iter().any(|x| matches!(x, Inline::Text(t, f) if t == "inline" && f.link.as_deref() == Some("https://example.com/inline"))));
}
/// A definition has to start a block (CommonMark §4.7), and a fence opens and closes by the
/// same rules the block parser uses. Reading a line without that context deleted text: a
/// definition-shaped line inside a paragraph, and every such line inside a code block whose
/// fence this pass thought had closed.
#[test]
fn a_definition_shaped_line_that_is_not_a_definition_stays_in_the_document() {
let body = |md: &str| import(md).plain_text(craft_wordmodel::StoryRef::Body);
// A definition cannot interrupt a paragraph: this is the author's text, lines and all.
let para = "Status update\n[Warning]: unstable\nmore text\n";
assert_eq!(body(para), "Status update [Warning]: unstable more text", "{:?}", body(para));
// `` ```rust `` holds an info string, so it does not close a fence: the definition-shaped
// line after it is still code.
let code = "```\n```rust\n[foo]: /bar\n```\n\nSee [foo].\n";
assert!(body(code).contains("[foo]: /bar"), "{:?}", body(code));
assert!(body(code).ends_with("See [foo]."), "nothing defines foo: {:?}", body(code));
// Four spaces of indentation is code, not a fence, so the definition that follows is
// taken and the reference resolves.
let indented = " ```\n\n[x]: /y\n\nSee [x].\n";
let doc = import(indented);
let text = doc.plain_text(craft_wordmodel::StoryRef::Body);
assert!(text.contains("```") && text.ends_with("See x."), "{text:?}");
// A definition still gives up its line after a blank line, at the start of the document,
// after a heading, and after another definition.
let starts = "[a]: /a\n[b]: /b\n\n# Head\n[c]: /c\n\nSee [a], [b], [c].\n";
assert_eq!(body(starts), "Head\nSee a, b, c.", "{:?}", body(starts));
// A footnote definition is not a link definition: it stays in the body.
let footnote = "x[^1]\n\n[^1]: Note\n";
assert!(body(footnote).contains("[^1]: Note"), "{:?}", body(footnote));
}
#[test]
fn blocks() {
let f = parse(
"# Title\n\nPara one\ncontinues.\n\n> quoted\n\n- a\n- b\n 1. nested\n\n```\ncode *x*\n```\n\n---\n\n| A | B |\n|---|:-:|\n| 1 | 2 |\n",
);
let kinds: Vec<String> = f
.blocks
.iter()
.map(|b| match b {
FBlock::Para(p) => format!("{:?}{:?}:{}", p.kind, p.list.map(|l| (l.ordered, l.level)), p.text()),
FBlock::Table(t) => format!("table{}", t.rows.len()),
})
.collect();
assert_eq!(
kinds,
vec![
"Heading(1)None:Title",
"NormalNone:Para one continues.",
"QuoteNone:quoted",
"NormalSome((false, 0)):a",
"NormalSome((false, 0)):b",
"NormalSome((true, 1)):nested",
"CodeNone:code *x*",
"RuleNone:",
"table2",
]
);
}
#[test]
fn export_escapes_round_trip() {
let mut p = Para::default();
p.push_text("1. not a list *really*", &Fmt::default());
let md = escape_line_start(inlines_md(&p.inlines));
let back = parse(&md);
let FBlock::Para(q) = &back.blocks[0] else { panic!() };
assert_eq!(q.text(), "1. not a list *really*");
assert!(q.list.is_none());
}
#[test]
fn export_emphasis_next_to_punctuation_round_trips() {
let marks: [(&str, fn(&mut Fmt)); 3] = [("bold", |f| f.bold = true), ("italic", |f| f.italic = true), ("strike", |f| f.strike = true)];
let cases: [&[(&str, bool)]; 4] = [
&[("注意:", true), ("请阅读", false)],
&[("a.", true), ("b", false)],
&[("a", false), ("!", true), ("b", false)],
&[("a", false), ("(x)", true), ("b", false)],
];
for (name, set) in marks {
for runs in cases {
let mut p = Para::default();
for (t, on) in runs {
let mut f = Fmt::default();
if *on {
set(&mut f);
}
p.push_text(t, &f);
}
let md = inlines_md(&p.inlines);
let back = parse(&md);
let FBlock::Para(q) = &back.blocks[0] else { panic!() };
let want: String = runs.iter().map(|r| r.0).collect();
assert_eq!(q.text(), want, "{name} exported as {md:?}");
let got: Vec<(String, bool)> = q
.inlines
.iter()
.filter_map(|i| if let Inline::Text(t, f) = i { Some((t.clone(), f.bold || f.italic || f.strike)) } else { None })
.collect();
let expect: Vec<(String, bool)> = runs.iter().map(|(t, on)| (t.to_string(), *on)).collect();
assert_eq!(got, expect, "{name} exported as {md:?}");
}
}
}
#[test]
fn export_adjacent_emphasis_next_to_punctuation_round_trips() {
let b = Fmt { bold: true, ..Fmt::default() };
let i = Fmt { italic: true, ..Fmt::default() };
let s = Fmt { strike: true, ..Fmt::default() };
let cases: [&[(&str, &Fmt)]; 6] = [
&[("ab", &b), ("!c", &i)],
&[("ab", &i), ("!c", &b)],
&[("ab", &b), ("!c", &s)],
&[("ab", &s), ("(c)", &b)],
&[("a.", &b), ("b", &i)],
&[("a.", &b), ("!b", &i)],
];
for runs in cases {
let mut p = Para::default();
for (t, f) in runs {
p.push_text(t, f);
}
let md = inlines_md(&p.inlines);
let back = parse(&md);
let FBlock::Para(q) = &back.blocks[0] else { panic!() };
let got: Vec<(String, bool, bool, bool)> =
q.inlines.iter().filter_map(|x| if let Inline::Text(t, f) = x { Some((t.clone(), f.bold, f.italic, f.strike)) } else { None }).collect();
let expect: Vec<(String, bool, bool, bool)> = runs.iter().map(|(t, f)| (t.to_string(), f.bold, f.italic, f.strike)).collect();
assert_eq!(got, expect, "exported as {md:?}");
}
}
#[test]
fn escape_line_start_preserves_indentation() {
assert_eq!(escape_line_start(" indented text".to_string()), " indented text");
assert_eq!(escape_line_start(" # not a heading".to_string()), " \\# not a heading");
assert_eq!(escape_line_start(" 1. not a list".to_string()), " 1\\. not a list");
assert_eq!(escape_line_start(" > quote".to_string()), " \\> quote");
// Four spaces or a tab would make an indented code block: the indent is capped at three.
assert_eq!(escape_line_start(" deep".to_string()), " deep");
assert_eq!(escape_line_start("\tTabbed # text".to_string()), " Tabbed # text");
assert_eq!(escape_line_start(" # not a heading".to_string()), " \\# not a heading");
assert_eq!(escape_line_start(" \t1. not a list".to_string()), " 1\\. not a list");
}
#[test]
fn leading_unicode_whitespace_is_kept_and_does_not_start_blocks() {
let kinds_text = |md: &str| -> Vec<(Kind, bool, String)> {
parse(md).blocks.iter().filter_map(|b| if let FBlock::Para(p) = b { Some((p.kind, p.list.is_some(), p.text())) } else { None }).collect()
};
assert_eq!(kinds_text("\u{3000}\u{3000}日本語の段落\n"), vec![(Kind::Normal, false, "\u{3000}\u{3000}日本語の段落".to_string())]);
for src in ["\u{a0}# not a heading", "\u{3000}1. not a list", "\u{a0}- x", "\u{a0}> not a quote"] {
let mut p = Para::default();
p.push_text(src, &Fmt::default());
let md = escape_line_start(inlines_md(&p.inlines));
assert_eq!(kinds_text(&md), vec![(Kind::Normal, false, src.to_string())], "{src:?} via {md:?}");
}
}
#[test]
fn single_backtick_code_span_escaped_pipe() {
let md = "| Command | Status |\n| --- | --- |\n| `cmd \\| grep` | pass |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
assert_eq!(t.rows[1].len(), 2);
let FBlock::Para(p0) = &t.rows[1][0].blocks[0] else { panic!() };
assert_eq!(p0.inlines.len(), 1);
assert!(matches!(&p0.inlines[0], Inline::Text(t, f) if t == "cmd | grep" && f.code));
let FBlock::Para(p1) = &t.rows[1][1].blocks[0] else { panic!() };
assert_eq!(p1.text(), "pass");
let mut out = Vec::new();
table_md(t, &mut out);
let exported = out.join("\n");
assert!(exported.contains("`cmd \\| grep`"), "Exported: {exported}");
}
#[test]
fn double_backtick_code_span_escaped_pipe() {
let md = "| Syntax | Note |\n| --- | --- |\n| ``a \\| b`` | second |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
assert_eq!(t.rows[1].len(), 2);
let FBlock::Para(p0) = &t.rows[1][0].blocks[0] else { panic!() };
assert_eq!(p0.inlines.len(), 1);
assert!(matches!(&p0.inlines[0], Inline::Text(t, f) if t == "a | b" && f.code));
let FBlock::Para(p1) = &t.rows[1][1].blocks[0] else { panic!() };
assert_eq!(p1.text(), "second");
}
#[test]
fn code_span_literal_backtick_and_escaped_pipe() {
let md = "| Expression | Detail |\n| --- | --- |\n| ``a`b\\|c`` | distinct |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
assert_eq!(t.rows[1].len(), 2);
let FBlock::Para(p0) = &t.rows[1][0].blocks[0] else { panic!() };
assert_eq!(p0.inlines.len(), 1);
assert!(matches!(&p0.inlines[0], Inline::Text(t, f) if t == "a`b|c" && f.code));
let FBlock::Para(p1) = &t.rows[1][1].blocks[0] else { panic!() };
assert_eq!(p1.text(), "distinct");
}
#[test]
fn three_column_table_with_distinctive_text() {
let md = "| Col1 | Col2 | Col3 |\n| --- | --- | --- |\n| `p1\\|q1` | Distinctive Two | Distinctive Three |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
assert_eq!(t.rows[1].len(), 3);
assert_eq!(t.rows[1][0].text(), "p1|q1");
assert_eq!(t.rows[1][1].text(), "Distinctive Two");
assert_eq!(t.rows[1][2].text(), "Distinctive Three");
let FBlock::Para(p0) = &t.rows[1][0].blocks[0] else { panic!() };
assert!(matches!(&p0.inlines[0], Inline::Text(_, f) if f.code));
let FBlock::Para(p1) = &t.rows[1][1].blocks[0] else { panic!() };
assert!(matches!(&p1.inlines[0], Inline::Text(_, f) if !f.code));
let FBlock::Para(p2) = &t.rows[1][2].blocks[0] else { panic!() };
assert!(matches!(&p2.inlines[0], Inline::Text(_, f) if !f.code));
}
#[test]
fn table_cells_past_the_document_limits_are_not_built() {
// Every body row was padded to the header's width before any cap (#3701).
let wide = format!("{}|\n{}|\n|\n| b |\n", "|a".repeat(50_000), "|-".repeat(50_000));
craft_flow::begin_import();
let flow = parse(&wide);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 3);
assert!(t.rows.iter().all(|r| r.len() == MAX_COLS));
assert_eq!(t.rows[2][0].text(), "b");
assert!(craft_flow::Dropped::TableCells.reported(), "the columns left out are logged");
// Rows past the limit still belong to the table, so the text after it reads as before.
let tall = format!("| a |\n| - |\n{}\nafter\n", "| x |\n".repeat(MAX_ROWS + 10));
craft_flow::begin_import();
let flow = parse(&tall);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), MAX_ROWS);
let FBlock::Para(p) = &flow.blocks[1] else { panic!("expected paragraph") };
assert_eq!(p.text(), "after");
assert_eq!(flow.blocks.len(), 2);
assert!(craft_flow::Dropped::TableCells.reported(), "the rows left out are logged");
}
/// A short row is padded to its header's width, so a small file of one-cell rows under wide
/// headers made millions of cells (CodeRabbit on #3862). The document's tables share one cell
/// budget: rows past it are left out and logged, and their lines make no paragraphs.
#[test]
fn tables_share_one_cell_budget() {
let header = |n: usize| format!("|{}|\n|{}|\n", vec![" h "; n].join("|"), vec!["-"; n].join("|"));
let fits = crate::MAX_TABLE_CELLS / MAX_COLS;
let src = format!("{}{}\nafter\n\n{}| 1 | 2 |\n", header(MAX_COLS), "| x |\n".repeat(fits + 100), header(2));
craft_flow::begin_import();
let flow = parse(&src);
let tables: Vec<&FTable> = flow.blocks.iter().filter_map(|b| if let FBlock::Table(t) = b { Some(t) } else { None }).collect();
let cells: usize = tables.iter().flat_map(|t| &t.rows).map(Vec::len).sum();
assert!(cells <= crate::MAX_TABLE_CELLS, "{cells} cells");
assert_eq!(tables.iter().map(|t| t.rows.len()).collect::<Vec<_>>(), [fits], "the second table is past the budget");
let texts: Vec<String> = flow.blocks.iter().filter_map(|b| if let FBlock::Para(p) = b { Some(p.text()) } else { None }).collect();
assert_eq!(texts, ["after"]);
assert!(craft_flow::Dropped::TableCells.reported());
// Within the budget a table is whole and nothing is logged.
craft_flow::begin_import();
let flow = parse(&format!("{}| 1 | 2 |\n", header(2)));
let [FBlock::Table(t)] = flow.blocks.as_slice() else { panic!("{:?}", flow.blocks) };
assert_eq!(t.rows.len(), 2);
assert!(!craft_flow::Dropped::TableCells.reported());
}
#[test]
fn a_wide_table_still_checks_every_header_and_delimiter_cell() {
// Only the first 63 cells are kept, but every one is counted and checked.
let head = "|a".repeat(100) + "|\n";
let delim = |bad: Option<usize>| (0..100).map(|i| if Some(i) == bad { "|x" } else { "|-" }).collect::<String>() + "|\n";
let is_table = |md: String| matches!(parse(&md).blocks.first(), Some(FBlock::Table(_)));
assert!(is_table(format!("{head}{}| b |\n", delim(None))));
assert!(!is_table(format!("{head}{}| b |\n", delim(Some(80)))), "a bad delimiter cell past the kept ones");
assert!(!is_table(format!("{}|\n{}| b |\n", "|a".repeat(99), delim(None))), "99 header cells, 100 delimiter cells");
let (cells, n) = split_row(r"| a | b \| c | d |", 2);
assert_eq!((cells, n), (vec!["a".to_string(), "b | c".to_string()], 3));
}
#[test]
fn regular_table_cells_containing_escaped_pipes() {
let md = "| Option | Description |\n| --- | --- |\n| Regular \\| Pipe | Normal text cell |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
assert_eq!(t.rows[1].len(), 2);
assert_eq!(t.rows[1][0].text(), "Regular | Pipe");
assert_eq!(t.rows[1][1].text(), "Normal text cell");
let FBlock::Para(p0) = &t.rows[1][0].blocks[0] else { panic!() };
assert!(matches!(&p0.inlines[0], Inline::Text(_, f) if !f.code));
let mut out = Vec::new();
table_md(t, &mut out);
let exported = out.join("\n");
assert!(exported.contains("Regular \\| Pipe"), "Exported: {exported}");
assert!(!exported.contains("Regular \\\\| Pipe"), "Exported: {exported}");
}
#[test]
fn unescaped_pipes_behave_as_gfm_column_separators() {
let md = "| Header 1 | Header 2 |\n| --- | --- |\n| `raw|pipe` | Extra Column |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
assert_eq!(t.rows[1].len(), 2);
assert_eq!(t.rows[1][0].text(), "`raw");
assert_eq!(t.rows[1][1].text(), "pipe`");
}
#[test]
fn ordinary_tables_and_standalone_code_spans_unchanged() {
let md = "| Alpha | Beta |\n| --- | --- |\n| 1 | 2 |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!() };
let mut out = Vec::new();
table_md(t, &mut out);
assert_eq!(out.join("\n"), "| Alpha | Beta |\n| --- | --- |\n| 1 | 2 |");
let mut p = Para::default();
let f = Fmt { code: true, ..Default::default() };
p.push_text("cmd | grep", &f);
let exported = inlines_md(&p.inlines);
assert_eq!(exported, "`cmd | grep`");
}
#[test]
fn code_span_line_endings_stay_literal_on_export() {
let cases = [
("a\n# b", "a # b"),
("a\n<div>b</div>", "a <div>b</div>"),
("a\n- b", "a - b"),
("a\n```", "a ```"),
("a\r# b", "a # b"),
("a\r\n# b", "a # b"),
("a\n\n> b", "a > b"),
("\na\n", " a "),
("`a\n# b`", "`a # b`"),
("cmd | grep", "cmd | grep"),
];
let shapes = [
(Kind::Normal, None),
(Kind::Normal, Some(ListInfo { ordered: false, level: 0, start: 1, ..Default::default() })),
(Kind::Code, Some(ListInfo { ordered: true, level: 0, start: 1, ..Default::default() })),
];
for (source, expected) in cases {
for (kind, list) in shapes {
let mut p = Para::new(kind);
p.list = list;
p.push_text(source, &Fmt { code: true, ..Default::default() });
let mut doc = model::to_doc(&model::Flow { blocks: vec![FBlock::Para(p)], ..Default::default() });
// The flow builder removes CR controls; keep the native document's raw input.
let p = doc.para_mut(craft_wordmodel::StoryRef::Body, &craft_wordmodel::Path::top(0)).unwrap();
p.text = source.to_string();
p.runs[0].len = source.len();
let md = export(&doc);
assert_eq!(md.trim_end().lines().count(), 1, "{source:?} {kind:?} {list:?} -> {md:?}");
assert!(!md.contains('\r'), "{md:?}");
let back = import(&md);
let flow = model::from_doc(&back);
let [FBlock::Para(q)] = flow.blocks.as_slice() else { panic!("{md:?}") };
assert_eq!(q.text(), expected, "{source:?} {kind:?} {list:?} -> {md:?}");
assert_eq!(q.list, list, "{md:?}");
assert!(q.inlines.iter().all(|i| matches!(i, Inline::Text(_, f) if f.code)), "{md:?}");
}
}
let fenced = "```\na\n# b\n```";
assert_eq!(export(&import(fenced)).trim_end(), fenced);
}
#[test]
fn code_span_line_endings_in_table_cells_stay_literal() {
for ending in ["\n", "\r", "\r\n"] {
let source = format!("a{ending}# b|c");
let mut p = Para::default();
p.push_text(&source, &Fmt { code: true, ..Default::default() });
let cell = Cell { blocks: vec![FBlock::Para(p)], ..Default::default() };
let table = FTable { rows: vec![vec![cell, Cell::default()], vec![Cell::default(), Cell::default()]], ..Default::default() };
let mut doc = model::to_doc(&model::Flow { blocks: vec![FBlock::Table(table)], ..Default::default() });
let p = doc.para_mut(craft_wordmodel::StoryRef::Body, &craft_wordmodel::Path(vec![0, 0, 0, 0])).unwrap();
p.text = source.clone();
p.runs[0].len = source.len();
let md = export(&doc);
let back = model::from_doc(&import(&md));
let [FBlock::Table(table), ..] = back.blocks.as_slice() else { panic!("{md:?}") };
assert!(back.blocks.iter().skip(1).all(|b| matches!(b, FBlock::Para(p) if p.text().is_empty())), "{md:?}");
assert_eq!(table.rows.len(), 2, "{md:?}");
assert_eq!(table.rows[0].len(), 2, "{md:?}");
let [FBlock::Para(p)] = table.rows[0][0].blocks.as_slice() else { panic!("{md:?}") };
assert_eq!(p.text(), "a # b|c", "{md:?}");
assert!(p.inlines.iter().all(|i| matches!(i, Inline::Text(_, f) if f.code)), "{md:?}");
}
}
#[test]
fn table_code_pipe_export_import_round_trip() {
let mut table = FTable::default();
let mut cell_a = Cell::default();
let mut para_a = Para::default();
let fmt_a = Fmt { code: true, ..Default::default() };
para_a.push_text("a`b|c", &fmt_a);
cell_a.blocks.push(FBlock::Para(para_a));
let mut cell_b = Cell::default();
let mut para_b = Para::default();
para_b.push_text("CRITICAL_DISTINCTIVE_INFO", &Fmt::default());
cell_b.blocks.push(FBlock::Para(para_b));
let mut head_a = Cell::default();
let mut head_para_a = Para::default();
head_para_a.push_text("Col A", &Fmt::default());
head_a.blocks.push(FBlock::Para(head_para_a));
let mut head_b = Cell::default();
let mut head_para_b = Para::default();
head_para_b.push_text("Col B", &Fmt::default());
head_b.blocks.push(FBlock::Para(head_para_b));
table.rows.push(vec![head_a, head_b]);
table.rows.push(vec![cell_a, cell_b]);
let mut out = Vec::new();
table_md(&table, &mut out);
let exported_md = out.join("\n");
assert!(exported_md.contains("``a`b\\|c``"), "Exported: {exported_md}");
assert!(exported_md.contains("CRITICAL\\_DISTINCTIVE\\_INFO"), "Exported: {exported_md}");
let back_flow = parse(&exported_md);
let FBlock::Table(back_table) = &back_flow.blocks[0] else { panic!("expected table") };
assert_eq!(back_table.rows.len(), 2, "Both rows must exist");
let row1 = &back_table.rows[1];
assert_eq!(row1.len(), 2, "Both cells must exist in row 1");
let back_cell_a = &row1[0];
let back_cell_b = &row1[1];
assert_eq!(back_cell_a.text(), "a`b|c", "Cell A text must be preserved");
assert_eq!(back_cell_b.text(), "CRITICAL_DISTINCTIVE_INFO", "Cell B text must be preserved");
let FBlock::Para(back_p_a) = &back_cell_a.blocks[0] else { panic!() };
assert_eq!(back_p_a.inlines.len(), 1);
assert!(
matches!(&back_p_a.inlines[0], Inline::Text(t, f) if t == "a`b|c" && f.code),
"Code formatting must remain intact on Cell A: {:?}",
back_p_a.inlines
);
let FBlock::Para(back_p_b) = &back_cell_b.blocks[0] else { panic!() };
assert_eq!(back_p_b.inlines.len(), 1);
assert!(
matches!(&back_p_b.inlines[0], Inline::Text(t, f) if t == "CRITICAL_DISTINCTIVE_INFO" && !f.code),
"Cell B formatting must remain plain text: {:?}",
back_p_b.inlines
);
assert_eq!(back_cell_a.text(), "a`b|c");
assert_eq!(back_cell_b.text(), "CRITICAL_DISTINCTIVE_INFO");
assert!(!back_cell_a.text().contains("CRITICAL_DISTINCTIVE_INFO"));
assert!(!back_cell_b.text().contains("a`b|c"));
let doc = import(&exported_md);
let re_exported = export(&doc);
assert!(re_exported.contains("``a`b\\|c``"), "Re-exported: {re_exported}");
assert!(re_exported.contains("CRITICAL\\_DISTINCTIVE\\_INFO"), "Re-exported: {re_exported}");
let doc2 = import(&re_exported);
let back_flow2 = model::from_doc(&doc2);
let FBlock::Table(back_table2) = &back_flow2.blocks[0] else { panic!() };
assert_eq!(back_table2.rows[1][0].text(), "a`b|c");
assert_eq!(back_table2.rows[1][1].text(), "CRITICAL_DISTINCTIVE_INFO");
}
#[test]
fn ordered_list_keeps_start_number() {
assert_eq!(export(&import("5. a\n6. b")).trim_end(), "5. a\n6. b");
}
/// A list inside a block quote stays in the quote: its items were imported as quoted list
/// paragraphs but written back without `>`, so the list moved out of the quote.
#[test]
fn a_list_inside_a_quote_stays_quoted() {
let src = "> Intro\n>\n> - one\n> - two\n> - nested\n> 1. first\n\nAfter.\n";
let want = "> Intro\n\n> - one\n> - two\n> - nested\n> 1. first\n\nAfter.";
let md = export(&import(src));
assert_eq!(md.trim_end(), want);
assert_eq!(export(&import(&md)).trim_end(), want, "and stays so when read back");
// An item's continuation line is quoted too.
assert_eq!(export(&import("> - a\\\n> b\n")).trim_end(), "> - a\\\n> b");
// A list outside a quote is unchanged.
assert_eq!(export(&import("- one\n- two\n")).trim_end(), "- one\n- two");
// A list after a quoted list stays outside the quote.
let md = export(&import("> - a\n\n2. b\n"));
assert_eq!(md.trim_end(), "> - a\n\n2. b");
assert_eq!(export(&import(&md)).trim_end(), "> - a\n\n2. b", "and stays so when read back");
}
#[test]
fn lines_after_line_break_stay_in_paragraph() {
use craft_wordmodel::{CharProps, Document, Paragraph, StoryRef, para_block};
let cases = ["Shopping\n- milk\n# not a heading", "a\n1. x", "a\n2) x", "a\n+ x", "a\n---", "a\n***", "a\n==", "a\n--", "a\n> q"];
for s in cases {
let mut d = Document::new();
d.body = vec![para_block(Paragraph::with_text(s, CharProps::default()))];
let md = export(&d);
let back = import(&md);
assert_eq!(back.body.len(), 1, "{s:?} -> {md:?}");
assert_eq!(back.plain_text(StoryRef::Body), s, "{md:?}");
}
}
#[test]
fn lines_after_line_break_stay_in_list_item_and_quote() {
let cases = ["a\n- x", "a\n# h", "a\n1. x", "a\n--", "a\n---", "a\n==", "a\n> q"];
let shapes: [(Kind, Option<ListInfo>); 5] = [
(Kind::Normal, Some(ListInfo { ordered: false, level: 0, start: 1, ..Default::default() })),
(Kind::Normal, Some(ListInfo { ordered: true, level: 0, start: 1, ..Default::default() })),
(Kind::Normal, Some(ListInfo { ordered: false, level: 1, start: 1, ..Default::default() })),
(Kind::Normal, Some(ListInfo { ordered: true, level: 1, start: 1, ..Default::default() })),
(Kind::Quote, None),
];
for s in cases {
for (kind, list) in shapes {
let mut blocks = Vec::new();
if let Some(li) = list.filter(|l| l.level > 0) {
let mut parent = Para::new(Kind::Normal);
parent.list = Some(ListInfo { ordered: li.ordered, level: 0, start: 1, ..Default::default() });
parent.push_text("top", &Fmt::default());
blocks.push(FBlock::Para(parent));
}
let mut p = Para::new(kind);
p.list = list;
p.push_text(s, &Fmt::default());
blocks.push(FBlock::Para(p));
let mut out = Vec::new();
write_blocks(&blocks, &mut out);
let md = out.join("\n");
let back = parse(&md);
assert_eq!(back.blocks.len(), blocks.len(), "{s:?} {kind:?} {list:?} -> {md:?}");
let Some(FBlock::Para(q)) = back.blocks.last() else { panic!("{md:?}") };
assert_eq!(q.kind, kind, "{md:?}");
assert_eq!(q.list, list, "{md:?}");
assert_eq!(q.text(), s, "{md:?}");
}
}
}
#[test]
fn table_column_alignments_import_and_export() {
let md = "| Left | Center | Right | Default |\n| :--- | :---: | ---: | --- |\n| 1 | 2 | 3 | 4 |\n";
let flow = parse(md);
let FBlock::Table(t) = &flow.blocks[0] else { panic!("expected table") };
assert_eq!(t.rows.len(), 2);
let aligns: Vec<Option<Align>> =
t.rows[0].iter().map(|c| c.blocks.iter().find_map(|b| if let FBlock::Para(p) = b { Some(p.align) } else { None }).flatten()).collect();
assert_eq!(aligns, vec![Some(Align::Left), Some(Align::Center), Some(Align::Right), None]);
let mut out = Vec::new();
table_md(t, &mut out);
let exported = out.join("\n");
assert!(exported.contains("| :--- | :---: | ---: | --- |"), "Exported: {exported}");
}
#[test]
fn tab_separated_list_markers() {
let f = parse("-\tbullet item\n*\tstar item\n+\tplus item\n1.\tone item\n2)\ttwo item\n");
assert_eq!(f.blocks.len(), 5);
for (i, b) in f.blocks.iter().enumerate() {
let FBlock::Para(p) = b else { panic!("block {i} not a para") };
assert!(p.list.is_some(), "block {i} should be a list item: {:?}", p);
}
let FBlock::Para(p0) = &f.blocks[0] else { panic!() };
assert_eq!(p0.text(), "bullet item");
assert!(!p0.list.unwrap().ordered);
let FBlock::Para(p3) = &f.blocks[3] else { panic!() };
assert_eq!(p3.text(), "one item");
assert!(p3.list.unwrap().ordered);
}
#[test]
fn html_comments_are_ignored_in_blocks_and_inlines() {
let md = "<!-- block comment -->\n\nHello <!-- inline comment -->world!\n\n<!--\nmultiline\ncomment\n-->\n\nAfter comment.";
let flow = parse(md);
assert_eq!(flow.blocks.len(), 2);
let texts: Vec<String> = flow.blocks.iter().filter_map(|b| if let FBlock::Para(p) = b { Some(p.text()) } else { None }).collect();
assert_eq!(texts, ["Hello world!", "After comment."]);
}
#[test]
fn a_paragraph_of_delimiters_parses_in_linear_time() {
// #3714: each delimiter's literal looked back for the text before it, to the start of
// the paragraph when there was none (quadratic).
let src = "*_".repeat(200_000);
let t = std::time::Instant::now();
let out = inl(&src);
// Under a second on a loaded machine; the backwards search ran past 6 minutes.
assert!(t.elapsed() < std::time::Duration::from_secs(10), "{:?}", t.elapsed());
let text: String = out.iter().filter_map(|i| if let Inline::Text(t, _) = i { Some(t.as_str()) } else { None }).collect();
assert!(text.chars().all(|c| matches!(c, '*' | '_')), "only delimiters come out");
// Literal delimiters still take the link of the text before them.
let s = inl("[a](http://x.org) [b*](http://y.org)");
assert!(s.iter().any(|i| matches!(i, Inline::Text(t, f) if t == "b*" && f.link.as_deref() == Some("http://y.org"))), "{s:?}");
}
#[test]
fn exported_link_destinations_read_back_unchanged() {
// Removing `<` and `>` from a destination after the safety check could make a relative
// target a script or network-path one (`java<script:…` → `javascript:…`). They are
// escaped instead, as are backslash escapes and entity references a reader would decode.
let md = |l: &str| inlines_md(&[Inline::Text("x".into(), Fmt { link: Some(l.into()), ..Default::default() })]);
assert_eq!(md("java<script:alert(1)"), "[x](<java\\<script:alert(1)>)");
for l in [
"java<script:alert(1)",
"/</evil.example/(x)",
"a>b c",
"&#106;avascript:alert(1)",
"&#x6A;avascript:x",
"&amp;",
"javascript\\:x",
"dir\\",
"dir\\(1)\\",
"q?a=1&b=2",
] {
let out = md(l);
let back = inl(&out);
let got: Vec<_> = back.iter().filter_map(|i| if let Inline::Text(_, f) = i { f.link.as_deref().map(str::to_string) } else { None }).collect();
assert_eq!(got, [l.to_string()], "{l:?} → {out:?}");
}
// Ordinary destinations are written as before.
assert_eq!(md("https://example.com/a?b=1&c=2"), "[x](https://example.com/a?b=1&c=2)");
assert_eq!(md("https://example.com/a b"), "[x](<https://example.com/a b>)");
assert_eq!(md("https://e.com/x\ny"), "[x](https://e.com/xy)");
}
#[test]
fn heading_trailing_hash_survives_round_trip() {
let heading = |md: &str| {
let f = parse(md);
let Some(FBlock::Para(p)) = f.blocks.first() else { panic!("no paragraph in {md:?}") };
assert_eq!(p.kind, Kind::Heading(2), "{md:?}");
p.text()
};
for (src, text, want) in [
("## Item \\#\n", "Item #", "## Item \\#\n"),
("## Issue \\##\n", "Issue ##", "## Issue \\##\n"),
("## \\#\n", "#", "## \\#\n"),
("## C#\n", "C#", "## C#\n"),
] {
assert_eq!(heading(src), text, "{src:?}");
let md = export(&import(src));
assert_eq!(md, want, "first export of {src:?}");
assert_eq!(heading(&md), text, "md was {md:?}");
assert_eq!(export(&import(&md)), want, "second export of {src:?}");
}
}