A Python cell's spill range was recorded only when the file held a record for
one of its other cells, so a result whose other cells are blank, which a file
may leave out, opened with no spill and no rebuilt result. The range is now
recorded when the anchor is read, if a result can fill it.
A Python range larger than a result may be (more than 1,000,000 cells, but
within what the reader tracks) emptied the cells it covered, although it is
no spill and file-format-parity.md says such cells keep the values the file
shows. They are now kept as plain values; a formula's spilled values are still
dropped, since recalculation fills them again.
Tests: a_python_spill_whose_blank_cells_have_no_records_is_still_a_spill and
a_python_spill_range_larger_than_a_result_keeps_its_values_as_plain_values;
both fail without the fix.
python.set kept the stored result whenever the code was unchanged, also when
the call switched the return type between an Excel value and a Python object.
The cell then showed a value and preview of the other type, and a save wrote
the new type with that old cached value. The result is now kept only when both
the code and the return type match; otherwise the cell is new code, which
shows #BLOCKED! until it runs, and its old spill goes with its readers
recalculated.
When a file is opened, a Python cell falls back to keeping only its value in
four cases: its code is missing from the Python part or the part is damaged
(the xlsx reader), its code is too long for a cell, its kept result is
malformed, or its result is past the adoption budget. The first kept the
values the cell spilled as plain values; the others dropped them, so values
the file showed were lost. All of them now follow one rule, through
Sheet::settle_spill: the anchor and every spilled cell keep their cached
values as plain values, with no live spill range; a cell's own content and
format are never overwritten; and the work is bounded by the spilled values
stored, not by the range. The values move from the spill into the cells, so
nothing is copied. The cut-by-budget test is renamed for what it now checks,
the dropped-result test now expects the values kept, and the adoption warning
and the file-format docs say so.
A script's code was cut at the character limit before its ST_Xstring escapes
were decoded, so a script whose escapes (`_x0009_`, `_x000D_`) made the file's
text longer than the limit lost its end without a warning, although its code
fits. The code is now decoded first and then cut one character past the limit,
so a script that is too long once decoded still fails the check and its cell
keeps only its value, as before. Decoding is linear and reads at most a bounded
number of encoded characters, enough for any code that fits.
A PY formula whose script the file doesn't hold (the Python part missing or
damaged, or the script over the code budget) keeps only its cached value, but
its array range stayed recorded, so the cached values over the rest of its
spill were dropped as a formula's spill would be, and nothing filled them
again. The range of such a cell is now forgotten when it is read: the anchor
and the cells of its spill keep their cached values as plain values, as the
file shows them, with no live spill. An ordinary formula's dynamic-array spill
is handled as before.
A Python part whose zip entry failed to read (a bad checksum or compressed
stream) made the whole workbook fail to open: the error was passed up, and the
failed read had also ended the package's read budget, so even a warning would
have left the sheets unread. The part is now read with
Package::read_optional_limited, which charges a failed read only that part's
allowance, and every error other than TooLarge is a warning. The Python cells
then keep only their cached values, as when the part is missing. Malformed XML
in the part was already a warning; the test covers it too.
When a file's cached Python spills were larger than the adoption budget, the
cells past it kept only their own value, but their spill ranges and spilled
values stayed on the sheet, which went on showing values no result held. The
spill of such a cell is now cleared, as for a dropped result; the warning is
unchanged.
A Python cell is written to xlsx as [MS-XLSX] 2.3.10 specifies: its
formula is _xlfn._xlws.PY(i,0|1), where i is the zero-based index of the
cell's script in the workbook's Python part and 0|1 the return type, 0
for an Excel value and 1 for a Python object. The part (2.1.23) is
xl/python.xml, content type application/vnd.ms-excel.python+xml,
related from the workbook part with
http://schemas.microsoft.com/office/2023/09/relationships/Python; its
root python holds pythonScripts, one pythonScript per Python cell in the
order the cells are written, with the code in code. The code keeps its
xl() references as text, so the formula has no further arguments. The
cell keeps its cached value. A spilling result is written like a dynamic
array (cm="1", t="array" ref=...) with the spilled values cached, so
other applications show the values (and #NAME? if they recalculate).
The reader turns a PY( formula with or without the _xlfn. and _xlws.
prefixes into a Python cell. An index names a script of the Python part,
and each %Px% in the script becomes the text of the formula's argument x
(2.3.10.4). The string-literal form, PY("chunk"&"chunk",0|1), still
reads, from the formula's own text so line breaks in the code survive. A
formula naming a script the part doesn't hold keeps only its cached
value, with a warning: no calculation can evaluate it. The reader keeps
the cached spilled values as the cell's result, so a workbook from Excel
shows its arrays without running anything. Python errors are cached as
#VALUE!, as #SPILL! and #CALC! already are. Native JSON keeps Cell.py
whole: code, return type, the last result and the exact error.
Calc lays out a Python cell's kept result: Python cells are nodes of the
dependency graph that read nothing, and a pass takes a dirty Python
cell's result instead of evaluating it, so its array spills, is blocked
(#SPILL!) and unblocked, and feeds its dependents exactly like a
formula's. An array that is malformed or has more than 1,000,000 cells
(only a crafted file holds one) shows #SPILL!. A dirty Python cell is
pending in the pass, as a dirty formula is, so a formula, a range read
or a cell-by-cell range walk that reaches it before its turn lays it out
first rather than reading its old value.
Opening a file can't make the app loop over a claimed size: the reader
keeps a Python cell's spill range only when it has at most 1,000,000
cells (a range claiming the whole sheet is dropped with its values, as a
formula's spilled values are), and io::adopt_python rebuilds results for
at most 4,000,000 cells over the whole workbook, with a warning for the
cells it skips. The Python part is read up to 64 MB and 1,000,000
scripts, a script is kept up to one character past what a cell may
hold, and the cells take at most 64 MB of code from the part in all,
since several formulas can name one script. io::enforce_python_limits
drops the code of a cell whose code doesn't fit in 8,192 characters as
PY(...) (the cell keeps its cached value, with a warning) and a kept
result that isn't a well-formed array within the cap, each with its
spill range and spilled values, so the sheet and a save no longer show
the old spill. Formats that keep PY() as an ordinary formula (xls, xlsb,
ods) are adopted the same way. Opening a file runs nothing.
The command-sequence fuzzer sets Python cells (code with xl()
references, quotes, line breaks, and code at and over the size limit),
and its xlsx round-trip check allows the Python errors becoming
#VALUE!, which is what the file format stores.
Tests: the written formula naming its script, the Python part with its
content type and relationship, return types, a spilling table saved as a
dynamic array, Python errors as #VALUE!, the round trip of values, code,
long code with quotes and line breaks and the error, and no Python part
without Python cells; a workbook laid out as the format specifies (an
index, a parameter named as %P2%) read as Python cells with their
spilled values and saved back with its code in a Python part; the
string-literal form; a formula naming a missing script, with and without
a Python part; 20,000 cells naming one long script against the code
budget; the part's reader and writer (escaping, odd %P tokens, a long
script, a repeated parameter); a blocker in the spill area giving #SPILL!
and clearing it spilling again; the native JSON round trip; a spill
range claiming the whole sheet; the workbook-wide budget; hostile JSON
(code over the limit, a result claiming far more cells than it holds); a
malformed result that never spills; PY() formulas at 8,079 and 8,080
characters of code. Calc: formulas, MAX, SUM, COUNTA and SUMPRODUCT read
a dirty Python cell whose array is blocked before its turn, in a full
and an incremental recalculation, and see the array once it is
unblocked; enforce_python_limits removes the spill of dropped code and
of a dropped result, and keeps a sound one.
prxnvt