sniff_delimiter skipped every record without a non-whitespace character
other than the delimiter, so a record such as ";" or ";;;" was blank to
the semicolon candidate. A semicolon export of a sheet whose only value
is far down (B40001) starts with 40,000 ";" rows, which fill the 64 KiB
sample, and the file was read as comma-separated with one column.
Records that hold only a candidate's delimiters and whitespace are now
kept apart, at most 20 per candidate. When no candidate separates the
records with other text, the same rules pick among the candidates whose
sample holds only such records. Every result that did not fall back to
comma stays the same: a row of tabs alone is still blank to the other
candidates and does not outweigh them, as #1209 set out.
sniff_delimiter reads the first 64 KiB of the text and leaves out a
record cut off at that point (#1209). When the first record is longer
than the sample, no record is counted and the function returns comma,
so a semicolon or tab file with a very wide first row opens as one
column.
When no candidate delimiter finds coherent complete records with a
delimiter in each, a candidate that finds no complete record now
counts the delimiters read so far in the first record. Delimiters
inside quoted fields are not counted. A quote inside a field that does
not start with one makes that record incoherent, and the coherent
candidate is picked. When one of those candidates ends the sample
inside a quoted field, each of them counts only up to that field's
opening quote. The per-candidate loop moved into a scan_delimiter
helper, which runs a second scan over the text before that quote.
ISO 6937 writes a spacing accent as its diacritic followed by SPACE.
ESC N B SPACE was read as a space and a combining acute; it is now the
acute accent itself, and likewise for the other nine marks glibc's
ISO_6937 map gives a spacing form.
An escape's diacritic took the next character as its letter even when that was the next escape (ESC N H ESC N H o gave ESC + U+0308 + NHo). A control character is no letter: the first escape is kept as written and the next one decodes on its own (ESC N H ö).
Excel writes a character outside ASCII in a SYLK text value as ESC N and one byte, the character's ISO 6937 code less 0x80; a diacritic's code is followed by the letter it goes on. The reader kept those bytes, so Größe für Köln opened as Gr<ESC>NHo<ESC>N{e f<ESC>NHur K<ESC>NHoln (storytold/gridcraft#4321). Text values now decode them: a diacritic and letter become the precomposed letter (or the letter and its combining mark when Unicode has no single character), a spacing code its character; a code ISO 6937 doesn't define is kept as written.
Moved from GridCraft at 5cf52c5f: the encoding detection, delimiter
sniffing, record parser and field quoting of crates/xlsx/src/csv.rs,
and the SYLK reader of crates/engine/src/io/sylk.rs.
Fields stay text and the SYLK reader returns its own Sheet; typed
values, dates and the workbook stay in GridCraft.