mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-13 22:12:12 +00:00
* Add craft-wordmodel: WordCraft's document model without its editing code
WordCraft's crates/doc (wordcraft-doc, at 07f97737) moved as it is, so the DOCX,
RTF, ODT and other text-format crates can read into and write from one model.
Module paths, type names and serde attributes are unchanged: an app that
re-exports the crate keeps its native file format.
Left in the app: range editing (edit.rs), IME text and equation editing. Not
model, and already in craft-libs: code pages (craft-rtf), the ZIP entry-count
check (craft-opc), the compound-file guard (craft-cfb). The paragraph walks
moved to walk.rs and the unit helpers of wordcraft-geom to geom.rs.
* craft-wordmodel: math_tree, the arguments of an equation node and a walk over its runs
The DOCX reader and writer walk an equation's runs. The walk and the child
accessors it uses were in WordCraft's equation editor (math_edit.rs), which
stays in the app; they are structure, not editing.
* craft-mime: detect, for bytes that name no character set
is_text, decode (BOM, unmarked UTF-16, UTF-8, else the legacy encoding
the bytes look like), legacy and legacy_label, behind a detect feature
that brings in chardetng. Moved from WordCraft's crates/formats/src/txt.rs
at 06c2ae6f with its text-sniffing tests. Also charset::is_known, so a
caller can tell a label decode would read as UTF-8 from one it knows.
* craft-wordmodel: keep the document operations (edit.rs) with the model
The DOCX and RTF readers use them: alternative-format chunks are merged into
a document with adopt_blocks, and a reader formats ranges with format_range
and inserts objects. They operate on the document, not on a view, so they
belong beside it. The tests that drove them come back with them; the
paragraph walks stay in walk.rs and edit re-exports them.
base64 0.23, as WordCraft and craft-offcrypto use.
* craft-codepage: one table per legacy code page, for craft-rtf, craft-dxf and craft-mime
craft-rtf and craft-dxf each carried tables for DOS 437, 737, 850, 852 and
857 (and craft-rtf one for 866). They are now one set in craft-codepage,
which takes craft-rtf's codepage module (Windows and Mac tables, OEM glyphs,
font charsets) and craft-dxf's DOS tables. craft_rtf::codepage re-exports it
unchanged; craft-dxf keeps its $DWGCODEPAGE names and Decoder and looks the
DOS tables up there.
Every byte of every code page either crate decodes is unchanged: the
public decoders of both crates were dumped over all 256 bytes of each page
(and the East Asian pairs) before and after, and the dumps are identical.
craft-mime now reads the DOS pages by their IANA names (IBM437, cp850,
ibm-852, 857...), which it read as UTF-8 before.
* Add craft-math: the equation tree with OMML, MathML, LaTeX and UnicodeMath
The math tree, LaTeX, linear-format, symbol and tree-walk modules come
from craft-wordmodel (storytold/craft-libs#963, WordCraft's crates/doc at
07f97737) unchanged, with the run colour (Rgb) and the symbol-font tables
they use. MathML is WordCraft's crates/formats/src/mathml.rs and OMML is
WordCraft's crates/docx/src/read/math.rs and write/math.rs (06c2ae6f /
07f97737), moved onto craft-xml. The HTML-specific MathML tests stay in
WordCraft; the rest moved with the code.
mathml::parse and omml::parse/read_str/write_str are new: string entry
points for callers that have no tree of their own.
* craft-math: read MathLive's and other spellings of LaTeX math commands
\abs{...} (an absolute value), \lvert \rvert \lVert \rVert, \nparallel,
\backprime, \infinity, \cross, \differentialD, \exponentialE and
\imaginaryI (MathLive's d, upright e and i), \emph, \textsf, \hbox,
\mathnormal and \underleftarrow / \underrightarrow /
\underleftrightarrow were unknown and read as literal text. They are read
as equaSHion's typed-input parser reads them (storytold/craft-libs#29).
* craft-math: equaSHion's equation corpus as test data
corpus/equations.tsv, corpus/latex.tsv and the OMML samples of
storytold/craft-libs#29 (7a6ef6b9), unmodified.
* craft-math: corpus round trips, GridCraft's OMML cases, entry points and limits tested; README
The corpus equations go through OMML, MathML and LaTeX and back; the OMML
samples are written back as kept and from the tree. GridCraft's OMML to
LaTeX test cases (crates/xlsx/src/omml.rs) give LaTeX that reads back the
same through the tree. omml::parse and read_str are tested against other
prefixes, the strict namespace, kept OMML, junk, the element cap and deep
nesting.
* Add craft-flow: the flow model the text formats map to, on craft-wordmodel
WordCraft's crates/formats/src/model (07f97737) moved as it is: Flow,
FBlock, Para, Inline, Fmt, tables and cells, pictures, to_doc and
from_doc, picture sniffing, colours, base64 and the Dropped report. Only
the paths changed (wordcraft_doc is craft_wordmodel); the report helpers
are public so the format crates can call them; the image re-encoding
helper stays with WordCraft's RTF writer.
* Add craft-markdown: Markdown read and written, on craft-flow
WordCraft's crates/formats/src/markdown (07f97737) with its 35 tests,
moved onto craft-flow and craft-wordmodel. Character references come from
craft-html. The table cell budget lives in craft-flow, which names it in
its warning. Two test lines that wrote plain text through WordCraft's txt
module stay in WordCraft.
* WIP: craft-latex sources from WordCraft (unmodified)
* Add craft-latex: LaTeX documents read and written, on craft-flow
WordCraft's crates/formats/src/latex (07f97737) with its 21 tests, moved
onto craft-flow and craft-wordmodel; bytes are decoded by craft-mime's
detect. The LaTeX parts of WordCraft's cross-format tests (nesting,
column specs, labels, no TeX commands from the document, the fuzz and
round-trip properties) come along as tests/hostile.rs. The ligature test
builds its three-font paragraph as a flow instead of reading HTML.
* craft-math: latex_text, LaTeX math and the linear format converted as text
WordCraft's crates/formats/src/latex/math.rs (07f97737): the converter its
LaTeX document reader and writer use, which keeps the source's spacing.
It moves here so craft-math is the one place that converts LaTeX math;
craft-latex calls it, and its output does not change. Its tests (from
WordCraft's latex tests and cross-format tests) come along.
* craft-math: latex_text's group finders are public, for craft-latex's length parser
* craft-latex: math through craft_math::latex_text; label timing test by ratio
The document math converter moved to craft-math (#1040) unchanged; this
crate calls it there (and its length parser uses its group finders), so
export output is the same. The label test compared one read with a
ten-second wall clock and failed on a loaded machine; it now compares the
fastest of three reads of 25,000 and 100,000 labels, which a linear read
keeps near 4x and a quadratic one near 16x.
* Cargo.lock after the merge
* WIP: WordCraft's HTML document reader and writer, unmodified (crates/formats/src/html at 07f97737)
* craft-html: HTML documents on craft-flow, behind a flow feature
WordCraft's crates/formats/src/html (07f97737) as craft_html::doc: the
tag-soup builder, CSS subset, ruby and MathML reading, page decoding and
the writer, with its 44 tests. It reads tokens from this crate's
tokenizer through a small adapter (decoded text and attributes, names in
lower case) instead of its own lexer, and its character references are
this crate's. Lexer::set_pos is new: the ruby reader hands a token back.
The HTML parts of WordCraft's MathML and cross-format tests come along as
tests/doc.rs, the linear-time check as a ratio.
* craft-math: operator names take their LaTeX scripts directly again (restores #1019)
#1019 (22161d2d) wrote \sin^2\theta as \sin ^{2}{\theta } so it reads back as
the same tree. The squash merge of #1038, whose branch predated it, put
math_latex.rs and the corpus test back as they were before. This applies
#1019's change again, unchanged, with its test.
* Cargo.lock: craft-html's flow dependencies
---------
Co-authored-by: shuff57 <62350898+shuff57@users.noreply.github.com>
157 lines
5.7 KiB
Rust
157 lines
5.7 KiB
Rust
use super::builder::block_kind;
|
|
use super::lexer::{Lexer, Tok};
|
|
|
|
/// A `<math>` element (#816): its tokens up to the matching end tag, as a tree. Nesting and size
|
|
/// are bounded; an unclosed element ends with the input.
|
|
pub(super) fn read_math(lx: &mut Lexer, name: String, attrs: Vec<(String, String)>) -> craft_math::mathml::El {
|
|
use craft_math::mathml::{El, Item, MAX_NODES, local};
|
|
let mut stack: Vec<El> = vec![El { name, attrs, children: Vec::new() }];
|
|
let mut count = 0usize;
|
|
let close = |stack: &mut Vec<El>| {
|
|
if stack.len() > 1
|
|
&& let Some(done) = stack.pop()
|
|
&& let Some(parent) = stack.last_mut()
|
|
{
|
|
parent.children.push(Item::El(done));
|
|
}
|
|
};
|
|
while let Some(t) = lx.next() {
|
|
count += 1;
|
|
if count > MAX_NODES * 4 {
|
|
// Over the cap: drop the rest of this element so its tokens aren't read as HTML text.
|
|
skip_math(lx, t);
|
|
break;
|
|
}
|
|
match t {
|
|
Tok::Text(t) => {
|
|
if let Some(top) = stack.last_mut()
|
|
&& !t.is_empty()
|
|
{
|
|
top.children.push(Item::Text(t));
|
|
}
|
|
}
|
|
Tok::Start { name, attrs, self_close } => {
|
|
let el = El { name, attrs, children: Vec::new() };
|
|
if self_close || stack.len() > craft_wordmodel::math::MAX_DEPTH {
|
|
if let Some(top) = stack.last_mut() {
|
|
top.children.push(Item::El(el));
|
|
}
|
|
} else {
|
|
stack.push(el);
|
|
}
|
|
}
|
|
Tok::End(name) => {
|
|
let name = local(&name).to_string();
|
|
// Close up to the matching open element; a stray end tag is ignored.
|
|
if let Some(k) = stack.iter().rposition(|e| local(&e.name) == name) {
|
|
if k == 0 {
|
|
break;
|
|
}
|
|
while stack.len() > k {
|
|
close(&mut stack);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
while stack.len() > 1 {
|
|
close(&mut stack);
|
|
}
|
|
stack.pop().unwrap_or_default()
|
|
}
|
|
|
|
/// Skip tokens (starting with `first`) up to and including the `</math>` that closes the element
|
|
/// being read; nested `<math>` elements are counted. An unclosed element ends with the input.
|
|
fn skip_math(lx: &mut Lexer, first: Tok) {
|
|
use craft_math::mathml::local;
|
|
let mut depth = 0usize;
|
|
let mut t = Some(first);
|
|
while let Some(tok) = t {
|
|
match tok {
|
|
Tok::Start { name, self_close: false, .. } if local(&name) == "math" => depth = depth.saturating_add(1),
|
|
Tok::End(name) if local(&name) == "math" => {
|
|
if depth == 0 {
|
|
return;
|
|
}
|
|
depth = depth.saturating_sub(1);
|
|
}
|
|
_ => {}
|
|
}
|
|
t = lx.next();
|
|
}
|
|
}
|
|
|
|
/// A `<ruby>` element's base text and reading (#1681).
|
|
#[derive(Default)]
|
|
pub(super) struct RubyText {
|
|
pub(super) base: String,
|
|
pub(super) ruby: String,
|
|
}
|
|
|
|
/// Which part of a `<ruby>` element the text being read belongs to.
|
|
#[derive(Clone, Copy, PartialEq, Eq)]
|
|
enum RubyPart {
|
|
/// The text being annotated: an `<rb>` child, or the text outside `<rt>`/`<rp>`.
|
|
Base,
|
|
/// The reading: an `<rt>` child.
|
|
Ruby,
|
|
/// `<rp>` fallback parentheses, which are not text: browsers show them only where ruby isn't
|
|
/// supported.
|
|
Fallback,
|
|
}
|
|
|
|
/// A block-level tag ends an open `<ruby>` implicitly, as it does in a browser.
|
|
fn ends_ruby(t: &Tok) -> bool {
|
|
match t {
|
|
Tok::Start { name, .. } | Tok::End(name) => block_kind(name).is_some(),
|
|
Tok::Text(_) => false,
|
|
}
|
|
}
|
|
|
|
/// Read a `<ruby>` element's content: its base text from `<rb>` children (or the text outside
|
|
/// them), its reading from `<rt>` children, and `<rp>` fallback parentheses dropped. Tolerant,
|
|
/// like the rest of the reader: tags inside are ignored, an unclosed element ends at the next
|
|
/// block-level tag (handed back to the caller) or with the input, and the tokens read are capped.
|
|
pub(super) fn read_ruby(lx: &mut Lexer) -> RubyText {
|
|
/// Most tokens one `<ruby>` element is read from: a missing end tag must not scan the rest of
|
|
/// a hostile file here.
|
|
const MAX_TOKENS: usize = 4096;
|
|
let mut out = RubyText::default();
|
|
let mut part = RubyPart::Base;
|
|
let mut nested = 0usize;
|
|
let mut count = 0usize;
|
|
while count < MAX_TOKENS {
|
|
count += 1;
|
|
let start = lx.pos();
|
|
let Some(t) = lx.next() else { break };
|
|
if nested == 0 && ends_ruby(&t) {
|
|
// The boundary ends the element: give the token back to the main reader.
|
|
lx.set_pos(start);
|
|
break;
|
|
}
|
|
match t {
|
|
Tok::Text(t) => match part {
|
|
RubyPart::Base => out.base.push_str(&t),
|
|
RubyPart::Ruby => out.ruby.push_str(&t),
|
|
RubyPart::Fallback => {}
|
|
},
|
|
// The self-closing slash is ignored on HTML elements (`<rt/>` opens an `rt`), as in a
|
|
// browser's parser.
|
|
Tok::Start { name, self_close, .. } => match name.as_str() {
|
|
"rb" => part = RubyPart::Base,
|
|
"rt" => part = RubyPart::Ruby,
|
|
"rp" => part = RubyPart::Fallback,
|
|
"ruby" if !self_close => nested += 1,
|
|
_ => {}
|
|
},
|
|
Tok::End(name) => match name.as_str() {
|
|
"rb" | "rt" | "rp" => part = RubyPart::Base,
|
|
"ruby" if nested > 0 => nested -= 1,
|
|
"ruby" => break,
|
|
_ => {}
|
|
},
|
|
}
|
|
}
|
|
out
|
|
}
|