Commit Graph
11 Commits
Author SHA1 Message Date
dotCooCoo 976e4ef8ea craft-csv: rows of empty cells decide the delimiter when nothing else does
sniff_delimiter skipped every record without a non-whitespace character
other than the delimiter, so a record such as ";" or ";;;" was blank to
the semicolon candidate. A semicolon export of a sheet whose only value
is far down (B40001) starts with 40,000 ";" rows, which fill the 64 KiB
sample, and the file was read as comma-separated with one column.

Records that hold only a candidate's delimiters and whitespace are now
kept apart, at most 20 per candidate. When no candidate separates the
records with other text, the same rules pick among the candidates whose
sample holds only such records. Every result that did not fall back to
comma stays the same: a row of tabs alone is still blank to the other
candidates and does not outweigh them, as #1209 set out.
2026-11-16 19:17:49 -08:00
Dylan M. Taylor 4f3bf746bb Merge pull request #1886 from Hamza0101/fix/sylk-escaped-characters
craft-csv: decode Excel's character escapes in SYLK text
2026-11-16 19:48:42 -05:00
dotCooCoo 52cca84204 fix(csv): count a first record longer than the delimiter sample
sniff_delimiter reads the first 64 KiB of the text and leaves out a
record cut off at that point (#1209). When the first record is longer
than the sample, no record is counted and the function returns comma,
so a semicolon or tab file with a very wide first row opens as one
column.

When no candidate delimiter finds coherent complete records with a
delimiter in each, a candidate that finds no complete record now
counts the delimiters read so far in the first record. Delimiters
inside quoted fields are not counted. A quote inside a field that does
not start with one makes that record incoherent, and the coherent
candidate is picked. When one of those candidates ends the sample
inside a quoted field, each of them counts only up to that field's
opening quote. The per-candidate loop moved into a scan_delimiter
helper, which runs a second scan over the text before that quote.
2026-11-16 11:29:40 -08:00
Hamza 503caef9c4 craft-csv: a SYLK diacritic on a space is the spacing accent
ISO 6937 writes a spacing accent as its diacritic followed by SPACE.
ESC N B SPACE was read as a space and a combining acute; it is now the
acute accent itself, and likewise for the other nine marks glibc's
ISO_6937 map gives a spacing form.
2026-11-16 19:20:31 +05:00
Hamza e096b5eb9b craft-csv: a diacritic before another escape keeps it as written
An escape's diacritic took the next character as its letter even when that was the next escape (ESC N H ESC N H o gave ESC + U+0308 + NHo). A control character is no letter: the first escape is kept as written and the next one decodes on its own (ESC N H ö).
2026-11-16 19:07:27 +05:00
Hamza ab87149855 craft-csv: decode Excel's character escapes in SYLK text
Excel writes a character outside ASCII in a SYLK text value as ESC N and one byte, the character's ISO 6937 code less 0x80; a diacritic's code is followed by the letter it goes on. The reader kept those bytes, so Größe für Köln opened as Gr<ESC>NHo<ESC>N{e f<ESC>NHur K<ESC>NHoln (storytold/gridcraft#4321). Text values now decode them: a diacritic and letter become the precomposed letter (or the letter and its combining mark when Unicode has no single character), a spacing code its character; a code ISO 6937 doesn't define is kept as written.
2026-11-16 18:51:40 +05:00
Dylan M. Taylor 69e457a8eb Add strict UTF-8 text decoder 2026-11-13 21:38:11 +00:00
Dylan M. Taylor 415dc3204c fix(csv): preserve bounded delimiter sampling 2026-11-13 19:25:10 +00:00
Dylan M. Taylor 5d46565f01 test(csv): lock delimiter tie priority 2026-11-13 19:03:26 +00:00
Dylan M. Taylor 8263ccd054 fix(csv): preserve quoted delimiter coherence 2026-11-13 18:28:50 +00:00
Dylan M. Taylor 991ea7dbd6 Add craft-csv: delimited text and SYLK, from GridCraft (#889)
Moved from GridCraft at 5cf52c5f: the encoding detection, delimiter
sniffing, record parser and field quoting of crates/xlsx/src/csv.rs,
and the SYLK reader of crates/engine/src/io/sylk.rs.

Fields stay text and the SYLK reader returns its own Sheet; typed
values, dates and the workbook stay in GridCraft.
2026-11-11 18:07:00 -05:00