Commit Graph
198 Commits
Author SHA1 Message Date
vangel76andClaude Opus 5.5 0f5a860d74 Merge perf/calc-spill-index-2683 (with upstream/main) into perf/calc-spill-unblock-2684
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 12:23:30 +01:00
vangel76andClaude Opus 5.5 17131fb0e5 Merge upstream/main into perf/calc-spill-index-2683
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 12:18:31 +01:00
vangel76andClaude Sonnet 5.5 865a7f6a06 Merge upstream/main into perf/calc-spill-unblock-2684
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-07 19:09:51 +01:00
vangel76andClaude Sonnet 5.5 1458f45214 Merge upstream/main into perf/calc-spill-index-2683
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-07 19:09:31 +01:00
vangel76andClaude Sonnet 5.5 42370639e0 Merge upstream/main into perf/calc-range-memo-2681
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-07 17:29:04 +01:00
vangel76andClaude Sonnet 5.5 76002fe0bd Reuse remembered range sums for every COUNT block
COUNT never reads the running sum, so the zero-sum gate that keeps SUM
and AVERAGE additions in order does not apply to it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-07 16:58:19 +01:00
FredApps ee1b5df3db Merge pull request #2693 from vangel76/perf/calc-expr-arc-2685
calc: rebuild the dependency graph from shared syntax trees
2026-11-07 16:51:36 +01:00
Harrison BoothandClaude Sonnet 5.5 292b784f02 fix(calc): apply "Set precision as displayed" in calculation (#1366)
* ft-121.7.9: apply Set precision as displayed in calculation (#1343)

With the option on, a formula result (and each element of a spill) is
rounded to its cell's number format before it is stored and before other
formulas read it. Numbers typed, pasted or reformatted while it is on are
rounded at once, and turning it on through formulas.calculationOptions
{precisionAsDisplayed} rounds every number constant, as Excel does.

The rounding is NumberFormat::round_as_displayed in the numfmt crate,
which shares the renderer's digit logic (fixed, scientific, fraction).
General, date/time, text, digitless and conditional formats keep the full
value, as observed in Excel through COM. Nothing changes when the flag is
off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ft-121.7.9: round conditional formats by the sign's section under precision as displayed (#1343)

Excel does round formats with a condition when "Set precision as
displayed" is on; the PR kept them at full value. Observed through COM
with synthetic values (and in gridcraft-corpus
excel/precision-displayed/pad-on-conditional-sections.xlsx): conditions
are ignored and the sign picks the section, the first for positives and
the second for negatives. `[<1]0.00;0` shows 2+1/3 as 2 but stores 2.33,
and stores -10/3 as -3.

Two more observed details: with a condition and a single number section,
negatives keep the full value (`[<1]0.00`), unless an `@` section follows
(`[<1]0.00;@` rounds them by the first section); and a completely empty
section passes to the next one (`0.0;;0.000` stores -2.333 as -2.333,
`;0.0` stores 2.333 as 2.3), for formats with or without a condition.

Display is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* engine: recalculate spill anchors when a spill cell is reformatted under precision as displayed

round_edited_as_displayed only looked at stored-cell changes, but a spill
destination lives in Sheet::spill, so reformatting it (or its row or column) left
the anchor out of the changed set and the element unrounded. Add each spill anchor
whose range holds a changed cell or a changed row/column style to the changed set.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* numfmt: drop a cast the fraction merge made redundant

* precision as displayed: keep locked constants on protected sheets, round new sheets, fix manual-mode comment

* precision as displayed: name the input sheet in the new-sheet histogram test

* precision as displayed: use six values in the new-sheet histogram test

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:24:49 +01:00
Dylan M. Taylor 632e4a1bc5 formula, calc: LET/LAMBDA names looked up through a map; ragged array literals refused before reserving (#3312) 2026-11-03 22:57:52 -05:00
Dylan M. Taylor db05400e67 calc: list each dependent once when a change spreads (#3190)
* calc: list each dependent once when a change spreads

* calc: check the dependency closure against a walk of every cell

Random graphs of single-cell, narrow and wide range reads (some removed or unsettled), random seeds and changed areas: the closure with shared taken records and area lookups equals the one listing every dependent of every changed cell.
2026-11-03 22:51:29 -05:00
Dylan M. Taylor c0ea297615 calc: cap the text one & over arrays may build in all (#3189)
* calc: cap the text one & over arrays may build in all

* address review: concatenation over arrays stops at the byte budget without reserving the whole broadcast
2026-11-03 22:51:11 -05:00
Hector VillarrealandClaude Opus 5.5 2caf435a38 #2012: split a parsed sheet-qualified name at the last !, sheet as is
Addresses review comments 4455115540 and 4455115557. The parser stores
`'Sheet'!Name` as `Sheet!Name` with the sheet already unquoted:

- Evaluator::name split it at the first `!`, so a series named
  `'a!b'!Title` showed #NAME?.
- chart_name unquoted it again, trimming spaces, so a chart reference
  to `' Sales '!Data` was saved as `Sales!Data`.

Both now split at the last `!` and look the sheet up as is.

Tests: chart tests::a_series_named_by_a_name_on_a_sheet_with_a_bang_resolves,
xlsx tests::chart_book_names::chart_names_keep_spaces_around_a_quoted_sheet_name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:21:22 -06:00
vangel76andClaude Sonnet 5.5 bcad4a75cf Merge calc: register a Python result's spill in the spill index (#2694) into perf/calc-spill-unblock-2684
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 20:32:11 +01:00
vangel76andClaude Sonnet 5.5 e85b24df57 calc: register a Python result's spill in the spill index
python_result wrote into spills but not into spill_index, so indexed readers
(cell_value, range_values, visit_range) missed a Python array laid out in the
same pass. Insert on spill, remove when blocked, as the formula path does.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 20:31:52 +01:00
vangel76andClaude Sonnet 5.5 00f565bc11 Factor out types clippy flags as too complex
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 19:44:16 +01:00
vangel76andClaude Sonnet 5.5 4b98b11996 Merge upstream/main into perf/calc-range-memo-2681
A Python cell is not a constant for the range_sum block memo.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 19:41:44 +01:00
vangel76andClaude Sonnet 5.5 21504859c8 Merge upstream/main into perf/calc-expr-arc-2685
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 19:36:41 +01:00
vangel76andClaude Sonnet 5.5 a9db1da90d Merge upstream/main into perf/calc-spill-index-2683
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 19:34:16 +01:00
vangel76andClaude Sonnet 5.5 96672e8ad5 Drop helpers the merged range reads no longer use
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 19:33:53 +01:00
vangel76andClaude Sonnet 5.5 628da5117a Merge upstream/main into perf/calc-spill-unblock-2684
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-02 19:28:52 +01:00
prxnvt 11b55e0c71 Calc: new Python code recalculates the readers of the cell's old spill
python.set, and Replace in a Python cell's code, removed the cell's spill
registration and its spilled values before the recalculation ran, so the
recalculation never saw the freed range and formulas reading those cells kept
their old values until a full recalculation. The commands now leave the spill
to the recalculation. When a Python cell's content changes and it has no array
result to lay out, cells_changed removes the spill and queues the freed range,
as it does for a formula replaced by a constant, so its readers are
recalculated; a cell that keeps its array result (the same code again) keeps
its spill and lays it out again. Clearing, pasting over and blocking a Python
spill already recalculated its readers, and undo and redo restore the values
along with the cells; a test covers them.
2026-11-01 14:41:48 -05:00
prxnvt cd51fe110d Calc: a range read walks only the spills it overlaps
A range read during recalculation walked every array laid out in the pass on
the sheet, whatever the range, and a block read (visit_range) gave up its fast
path as soon as any array was laid out there. A Python cell's cached result
can be up to 1,000,000 cells, so 1,000 formulas reading small ranges beside it
could visit about a billion values in one full recalculation; a large formula
array costs the same. Both reads now look only at the arrays whose area meets
the range, and only at the part inside it. The spilled values the sheet stores
from the last calculation are found row by row when the range has fewer rows
than there are spilled values, so a read beside them doesn't scan them either.
2026-11-01 14:06:02 -05:00
prxnvt b3a3590db6 Merge main into Python cells as data 2026-11-01 14:03:09 -05:00
prxnvt c03c61fd05 Merge main into Python cells as data 2026-11-01 12:00:09 -05:00
Max Murphy 81aee9e075 calc: form the MAP result size in u64 so wasm32 cannot wrap the cap (#2165)
`broadcast_len`, the MAX_CELLS guard shared by MAP, the binary operators,
TEXT and array IF, multiplied `rows` and `cols` as `usize`. On wasm32 a
broadcast such as 65536 x 65536 (2^32 cells) wraps that product to 0 and
slips past the cap, after which the allocation and loop run far past
MAX_CELLS. The product is now formed in u64 and converted back only after
the cap check, so the guard means the same on every target.

`single`, the per-cell scalar check in MAP and MAKEARRAY, compared
`rows * cols` with 1. A constructed array stores exactly `rows * cols`
values, so it now tests `data.len()` directly: the same answer without an
unchecked product.

Tests cover MAX_CELLS exactly, one cell over, the 2^32 pair that wraps to
zero on a 32-bit usize, a full column x row (2^34), and MAP itself just
over the cap and at 16384 x 16384.

Closes #2165
2026-10-31 16:41:24 +01:00
Brandon Thomas c305315610 Merge PR #2719 2026-10-30 15:42:10 -04:00
vangel76 95ba076778 Merge remote-tracking branch 'origin/perf/calc-spill-index-2683' into HEAD 2026-10-30 00:04:34 +01:00
vangel76andClaude Sonnet 5.5 bdbd29e623 perf(calc): size the spill-index decision by the sheet's own changed cells
A batch spread over several sheets used its total length for every sheet, so a
one-cell sheet with many spills built the block index as if it had the whole
batch's lookups.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-30 00:04:25 +01:00
vangel76andClaude Sonnet 5.5 c1e1c7bdc9 Merge origin/perf/calc-spill-index-2683 into perf/calc-spill-unblock-2684
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 23:49:01 +01:00
vangel76andClaude Sonnet 5.5 6c912d2eb1 perf(calc): keep tall and wide spill areas out of the block index
A full-height column spill took 65,536 block registrations, on every pass
that produced it. Areas over 1024 blocks are now tested directly by each
lookup instead. Adds a randomized comparison against a scan of all areas.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 23:48:11 +01:00
vangel76 1801f4a7cf Merge upstream/main into perf/calc-spill-unblock-2684 2026-10-29 23:09:25 +01:00
vangel76 c3865b5dbd Merge upstream/main into perf/calc-spill-index-2683 2026-10-29 23:07:21 +01:00
vangel76andClaude Sonnet 5.5 b6bbdcd5cf calc: build precedents from a borrowed syntax tree, no Arc for text-only formulas
Formula::with_expr runs a closure on the cached tree, or on a freshly parsed
one that is dropped afterwards, so rebuild and cells_changed need neither a
tree copy nor an Arc allocation / reference count.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 00:51:45 +01:00
vangel76 d71cf10e70 Merge remote-tracking branch 'upstream/main' into perf/calc-expr-arc-2685 2026-10-29 00:48:45 +01:00
Dylan M. Taylor 5a84d9bed8 Refactor calc for strict Clippy checks 2026-10-28 10:52:53 +00:00
vangel76andClaude Sonnet 5.5 f404a08357 Merge upstream/main into perf/calc-range-memo-2681
eval.rs and recalc.rs were split into submodules; re-applied the range_sum hunks in eval.rs, eval/call.rs and recalc/pass_host.rs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-28 00:54:49 +01:00
vangel76andClaude Sonnet 5.5 5f3fcc8140 perf(calc): look up #SPILL! cells by row and column range, not by row band
The spill-error set was ranged by rows only and filtered by column afterwards, so an edit in
column A walked every #SPILL! cell to its right in the same rows. Each row now keeps its own
column set and a window lookup takes one column range per row that has an entry. Results are
unchanged. Also covers an array spilling right over the cleared cell.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 19:45:28 +01:00
vangel76andClaude Sonnet 5.5 356ccdcf1a perf(calc): index spills for a paste only when registering them costs less than testing them
A full-height column spill takes 65,536 blocks to register, which a 17-cell paste elsewhere
never needed. A sheet's spill areas are now indexed only when their block count is below what
testing each area against each changed cell would cost; otherwise the plain containment check
runs, as before. Results are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 19:43:47 +01:00
vangel76 0b7f674925 Merge upstream/main 2026-10-27 19:40:32 +01:00
Higgs32584andClaude Opus 5.5 7d828c01f7 Security: cap the work of lifted lookups, needles x lookup vector, with #NUM! (#2716)
A lookup lifted over an array of needles searched its whole lookup vector once per needle
with no total cap, so =ROWS(XMATCH(SEQUENCE(1048576),-SEQUENCE(1048576))) in a 1.6 KB file
ran for hours and hung file.open. VLOOKUP, HLOOKUP, LOOKUP, MATCH, XLOOKUP and XMATCH now
return #NUM! past 5e8 needle x entry steps, the pattern MMULT uses. Other lifted functions
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 00:40:18 -04:00
vangel76andClaude Sonnet 5.5 72236e4aab perf(calc): find the formulas a cleared cell may unblock from a set of #SPILL! cells (#2684)
Clearing a cell walked every formula in the graph, and each changed cell
scanned a 64 x 64 window of stored cells, to find formulas showing
#SPILL!. Calc now keeps the set of cells whose value is #SPILL! (filled
in rebuild, kept up to date by cells_changed and the write-back of a
run) and takes those candidates from it. The same set replaces the walk
over the graph in run for areas a formula stopped spilling over. Each
entry is checked against the workbook before it is used.

Fixes #2684

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 22:26:15 +01:00
vangel76andClaude Sonnet 5.5 6c37a63704 perf(calc): index the pass's spill areas so reads don't scan every spill (#2683)
A cell read and a range read looked at every spill computed so far in
the pass, and cells_changed looked at every spill range of the sheet for
each changed cell. A spatial index (16 x 16 cell blocks) now finds the
areas covering a cell or meeting a range. Where areas overlap, the one
that counted before (first/last in the pass's spill map order) still
counts.

Fixes #2683

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 22:22:22 +01:00
vangel76andClaude Sonnet 5.5 c567511cf1 calc: rebuild the dependency graph from shared syntax trees
Calc::rebuild and Calc::cells_changed called Formula::expr(), which deep-copies each formula's
syntax tree only to read its references. Use expr_arc() (already used when formulas are
calculated), which hands out the cached Arc without copying.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 22:21:15 +01:00
vangel76andClaude Sonnet 5.5 96bfba93f8 perf(calc): remember SUM/COUNT/AVERAGE of constant-only blocks per pass (#2681)
A block that holds only constants can't change during a pass, so its sum
and count are worked out once and reused by every formula that reads the
same block. A block that is the same rows from the top but longer (a
running total) continues the shorter block's sum, which is the same
additions in the same order as one walk from the start.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-26 22:17:29 +01:00
aiongg c5098a0765 refactor(calc): move recalc.rs items to recalc/ 2026-10-26 19:04:52 +00:00
aiongg b0818bccc3 refactor(calc): move eval.rs items to eval/ 2026-10-26 19:04:52 +00:00
prxnvt fcbbfa5142 XLSX: Python cells as _xlfn._xlws.PY() formulas, code in the Python part, cached values; opening limits
A Python cell is written to xlsx as [MS-XLSX] 2.3.10 specifies: its
formula is _xlfn._xlws.PY(i,0|1), where i is the zero-based index of the
cell's script in the workbook's Python part and 0|1 the return type, 0
for an Excel value and 1 for a Python object. The part (2.1.23) is
xl/python.xml, content type application/vnd.ms-excel.python+xml,
related from the workbook part with
http://schemas.microsoft.com/office/2023/09/relationships/Python; its
root python holds pythonScripts, one pythonScript per Python cell in the
order the cells are written, with the code in code. The code keeps its
xl() references as text, so the formula has no further arguments. The
cell keeps its cached value. A spilling result is written like a dynamic
array (cm="1", t="array" ref=...) with the spilled values cached, so
other applications show the values (and #NAME? if they recalculate).

The reader turns a PY( formula with or without the _xlfn. and _xlws.
prefixes into a Python cell. An index names a script of the Python part,
and each %Px% in the script becomes the text of the formula's argument x
(2.3.10.4). The string-literal form, PY("chunk"&"chunk",0|1), still
reads, from the formula's own text so line breaks in the code survive. A
formula naming a script the part doesn't hold keeps only its cached
value, with a warning: no calculation can evaluate it. The reader keeps
the cached spilled values as the cell's result, so a workbook from Excel
shows its arrays without running anything. Python errors are cached as
#VALUE!, as #SPILL! and #CALC! already are. Native JSON keeps Cell.py
whole: code, return type, the last result and the exact error.

Calc lays out a Python cell's kept result: Python cells are nodes of the
dependency graph that read nothing, and a pass takes a dirty Python
cell's result instead of evaluating it, so its array spills, is blocked
(#SPILL!) and unblocked, and feeds its dependents exactly like a
formula's. An array that is malformed or has more than 1,000,000 cells
(only a crafted file holds one) shows #SPILL!. A dirty Python cell is
pending in the pass, as a dirty formula is, so a formula, a range read
or a cell-by-cell range walk that reaches it before its turn lays it out
first rather than reading its old value.

Opening a file can't make the app loop over a claimed size: the reader
keeps a Python cell's spill range only when it has at most 1,000,000
cells (a range claiming the whole sheet is dropped with its values, as a
formula's spilled values are), and io::adopt_python rebuilds results for
at most 4,000,000 cells over the whole workbook, with a warning for the
cells it skips. The Python part is read up to 64 MB and 1,000,000
scripts, a script is kept up to one character past what a cell may
hold, and the cells take at most 64 MB of code from the part in all,
since several formulas can name one script. io::enforce_python_limits
drops the code of a cell whose code doesn't fit in 8,192 characters as
PY(...) (the cell keeps its cached value, with a warning) and a kept
result that isn't a well-formed array within the cap, each with its
spill range and spilled values, so the sheet and a save no longer show
the old spill. Formats that keep PY() as an ordinary formula (xls, xlsb,
ods) are adopted the same way. Opening a file runs nothing.

The command-sequence fuzzer sets Python cells (code with xl()
references, quotes, line breaks, and code at and over the size limit),
and its xlsx round-trip check allows the Python errors becoming
#VALUE!, which is what the file format stores.

Tests: the written formula naming its script, the Python part with its
content type and relationship, return types, a spilling table saved as a
dynamic array, Python errors as #VALUE!, the round trip of values, code,
long code with quotes and line breaks and the error, and no Python part
without Python cells; a workbook laid out as the format specifies (an
index, a parameter named as %P2%) read as Python cells with their
spilled values and saved back with its code in a Python part; the
string-literal form; a formula naming a missing script, with and without
a Python part; 20,000 cells naming one long script against the code
budget; the part's reader and writer (escaping, odd %P tokens, a long
script, a repeated parameter); a blocker in the spill area giving #SPILL!
and clearing it spilling again; the native JSON round trip; a spill
range claiming the whole sheet; the workbook-wide budget; hostile JSON
(code over the limit, a result claiming far more cells than it holds); a
malformed result that never spills; PY() formulas at 8,079 and 8,080
characters of code. Calc: formulas, MAX, SUM, COUNTA and SUMPRODUCT read
a dirty Python cell whose array is blocked before its turn, in a full
and an incremental recalculation, and see the array once it is
unblocked; enforce_python_limits removes the spill of dropped code and
of a dropped result, and keeps a sound one.
2026-10-26 10:18:38 -04:00
Dylan M. TaylorandCodebuff d8c5f5885f refactor: reduce cognitive complexity in options, python highlighter, i18n scanner, loop search (#2611)
* refactor: split complex functions for Sonar S3776

Extract helpers from backstage options (general, calculation and iteration
sections), code_editor python (string scanning), xtask i18n literals
(escape and string readers) and calc recalc loops_among (SCC search).
No behaviour change.

* Restore the loop count that loops_among's refactor dropped

The S3776 split moved the loop-detection body into loops_among without
the 'let n = circ.len()' it reads, so calc failed to compile and the
Sonar build broke. Restore the binding from main.

* Extract the SCC pop step from components() to reduce cognitive complexity (S3776)

Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>

---------

Co-authored-by: Codebuff <noreply@codebuff.com>
2026-10-26 09:48:03 -04:00
aiongg 50c7ee3971 Batch: undo, redo and rollbacks list the circular references of the restored cells (#2501) 2026-10-25 03:50:42 +00:00
aiongg 2660973286 Merge storytold/gridcraft#2501: Error Checking: its arrow opens a Circular References submenu listing a cell on each loop 2026-10-25 02:24:49 +00:00