* Functions: GETPIVOTDATA (#284)
The engine records each PivotTable report value cell (value field,
row/column items, value) on refresh and when a workbook is opened;
GETPIVOTDATA is an evaluator special form that looks the value up by
value field and field/item pairs, #REF! when the report doesn't show it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* GETPIVOTDATA: shared lazy index, report values, hash lookup (review of #290)
- PivotTable::data_index is an Arc-shared PivotLookup (serde-skipped), so the
per-command workbook clone no longer copies the index; refresh sets it, clear resets it.
- No index work on open: the first GETPIVOTDATA on a pivot without one builds it from the
source via the engine-registered PIVOT_INDEXER; a failure is #REF!, not an empty index.
- The result is the value the sheet shows at the cell's report position, so formula and
report cell agree (test with a source edited after the last refresh).
- Lookup by a pre-lowercased hash key; numbers/booleans fall back to a scan on a miss.
- GETPIVOTDATA is no longer volatile: it depends on its pivot's whole report area.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* GETPIVOTDATA: keep the index of the stored report; collision-free keys; align estimates
The lookup index describes the report as last refreshed and is saved with
the pivot. A pivot loaded without one (XLSX) only builds it when the
recomputed layout matches the stored labels. Lookup keys are length-prefixed
and distinguish Boolean from text items. Function-parity estimates match.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* pivot: import PivotDataCell, PivotItemRef and PivotLookup explicitly after upstream dropped the glob import
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* pivot: add an ignored measurement of edits on a sheet with a large report
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* pivot: keep a failed lazy GETPIVOTDATA build retryable, never save an empty index
A failed lazy index build stored unwrap_or_default() into the shared,
first-set-wins PivotLookup, so a valid state coming later (undo, a
source fix, a reopen) could never rebuild it, and the empty list was
saved as a restored index. Cache only validated builds, treat an empty
list as no index in serialization, reject reports with more value
cells than the index cap instead of truncating, register the pivot
indexer in gridcraft-cli check, and add two regression tests.
* GETPIVOTDATA: move cached offsets with edits inside the report; exact numeric items first
Inserting or deleting rows or columns inside a report remapped the cells
and last_range but kept the cached value-cell offsets, so a lookup could
read another item's cell. shift_sheet_features now maps each indexed
cell through the edit into a new index (the shared one stays with the
undo snapshots); a value cell deleted with its line drops out (#REF!).
A numeric item was accepted on a formatted-label hash hit, so 0.6 and 1
both shown as "1" returned the first. A number now prefers the cell
whose item has exactly that value, before any label match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix pivot chart test for the new Output.data field
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What
When two array formulas want the same cells, the one that already has its spill keeps them, whichever anchor comes first (this is what Excel does, measured on this PR by Harrison-Booth). The other formula shows `#SPILL!` until the winner is cleared, then it spills. When neither has a spill yet (a first calculation with no remembered layout), the anchor first in reading order (row, then column) wins, so the result never depends on evaluation order.
Before, an incremental recalculation let the existing spill keep its cells, while a full recalculation laid everything out from scratch and gave them to the earlier anchor. The two disagreed. A full recalculation now starts from the layout the sheet held, so both agree.
Also fixed: clearing a spilling formula did not unblock an array blocked by its old area when that array sat below or to the right of it.
## Why
Issue #2480: on the sales sample, `cell.set C14 =TRANSPOSE(B18:B24)` followed by `range.fill =D14:E18` over E13:F13 left F13 as `#SPILL!` in memory but `0` after a full recalc (the fuzzer, `command_sequence_fuzz` seed 0, hits this on #795). Now C14 keeps its cells, F13 is `#SPILL!`, and a full recalc gives the same.
Known limits: the xlsx reader does not restore spill layouts, so a workbook opened from a file is laid out by reading order only (Excel keeps the layout the file records); and when both anchors already held a spill and one grows into the other, reading order decides (Excel keeps the cells the other owns). The rule is documented on `Standing` in `crates/calc/src/recalc.rs`.
## Test
In `crates/engine/src/tests.rs`: the two-command sequence from the issue; both entry orders of an overlap, plus clearing the winner; and a no-history layout, where the earlier anchor wins in both entry orders. All compare cells with a full recalc.
## How verified
- `cargo fmt --check`, `cargo clippy -p gridcraft-calc -p gridcraft-engine --all-targets -- -D warnings`, `cargo test -p gridcraft-calc -p gridcraft-engine` (includes the fuzz test): all pass.
Fixes#2480
* calc: linear dependency extraction for range-operator chains (#2183)
precedents() handled a range operator with non-reference operands by recursing into both operands and then letting Expr::walk descend into them again, so a left-associative chain of k operands cost 2^k (a 60-operand INDEX(...):INDEX(...) chain would hang on open). It is now a hand-written collector that visits every node once, growing the per-sheet bounding boxes of the enclosing range operator as it goes. Past a depth (4096) or node (1,000,000) bound it marks the formula dynamic (always recalculated) instead of recursing further.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* calc: drop wall-clock timing from precedents tests (clippy)
std::time::Instant::now is a disallowed method (clippy.toml: traps on wasm).
The regression test only needs to finish; the old 2^k walk would never return.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* calc: bound precedents to 100 ms and box plain-reference range chains (#2183)
- range_operator_chain_precedents_are_linear times the direct precedents
call against the 100 ms bound in the issue.
- A Range node with two plain Ref operands (A1:B2:C3, which the parser
can't fold) now collects its bounding box like any other range node, so
an edit to C1 recalculates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* calc: INDEX decides a multi-cell position from its shape before reading it
* calc: charge each formula's evaluation against one cumulative work budget
* address review: a database function whose computed criterion ran out of work is #CALC!
* address review: give the test's criteria cell its stored value
* address review: stop matching database records once a computed criterion exhausts the work budget
* address review: skip the criteria-row scan once work is exhausted
* address review: stop between conditions once the work budget is spent
* FORMULATEXT and the cell editor keep a hidden formula's source hidden
* address review: recalculate formulas when sheet protection changes
* address review: skip the full recalculation for a no-op unprotect
* address review: recalculate on protect/unprotect without marking the edit structural
.xls formulas are read as legacy formulas with Excel's implicit @ (apply_legacy via read_xls_with), one-cell Ctrl+Shift+Enter arrays keep their range, @ of a 2-D range meets the formula's row and column, and defined-name analyses are shared across the formulas of a pass and replayed exactly.
Closes#410
* XLSX: legacy formulas (no dynamic-array cm) open with Excel's implicit @ and calculate like Excel; implied @ saved back without cm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* XLSX: long operator chains analysed in full and unfinished legacy analysis leaves the formula unchanged (never saved as SINGLE); @ of a union keeps its parentheses; IF's condition intersected inside SUMPRODUCT
* XLSX: a legacy union already in parentheses saves back without a second pair (never as SINGLE); a name's definition analysed as an array throughout, inserting no @
* XLSX: drop the unreachable keep-unions retry; an @ of a bare union saves as a dynamic formula
* XLSX: an @ keeps parentheses the reader would not put back (redundant ones, a union's)
* XLSX: a redundant @ keeps a legacy formula legacy; IFERROR and IFNA intersected inside SUMPRODUCT; file parentheses and unfinished analyses saved as read
An edit could leave an implied @ on a single cell: deleting rows 2:3 under the legacy SQRT(A1:A3), read as SQRT(@A1:A3), gives SQRT(@A1:A1), and a name redefined to one cell does the same to SQRT(@Rng). The writer kept an @ only when re-analysing the formula without it put the same @ back, which it no longer does for one cell, so the formula was saved as SQRT(_xlfn.SINGLE(A1:A1)) with cm, readable only by dynamic-array Excel. The writer now analyses the formula with its own @ in place: an @ is accepted when the analysis implies it at that position or when its operand is a single value there (such an @ changes nothing), and any @ inserted elsewhere or any other @ still makes the formula dynamic. SQRT(@A1:A1) saves as SQRT(A1:A1), SQRT(@A1) as SQRT(A1) and @SUM(A1:A3) as SUM(A1:A3), all without cm; SUMPRODUCT(@SQRT(A1:A3)) stays dynamic because SQRT(A1:A3) is an array there.
Legacy Excel intersects IFERROR's and IFNA's arguments inside SUMPRODUCT unless the formula is entered with Ctrl+Shift+Enter, like IF's condition: =SUMPRODUCT(IFERROR(VALUE(C19:C21),0)) returns one row's value or #VALUE!, and the accepted fix is CSE or N(+range) (https://stackoverflow.com/questions/59228707). They now get the same single-value parameter class as IF's condition (Param::Single), so the legacy SUMPRODUCT(IFERROR(A1:A3/A1:A3,0)) reads as SUMPRODUCT(IFERROR(@A1:A3/@A1:A3,0)), and a typed SUMPRODUCT(IFERROR(VALUE(A1:A3),0)), which dynamic-array Excel evaluates as an array, is saved with cm. IF's result arguments are left as arrays: in https://stackoverflow.com/questions/29682328, a non-CSE =SUMPRODUCT(--(B4=IF(NOT($C$4:$C$11),$B$4:$B$11))) in row 4 returns 2 (both "apple" rows of B4:B11 counted, so the result branch was not intersected), 0 in a row where C is TRUE and #VALUE! outside rows 4:11 (the condition was intersected).
The reader's @ in front of a whole-formula array result took no extra parentheses when the file already had them, so ({1,2}+1) and {1,2}+1 both read as @({1,2}+1), and the writer dropped the file's parentheses. An operand already in parentheses now gets a second pair, so ({1,2}+1) saves as written.
A legacy formula the analysis could not finish (node budget, nesting) was read unchanged, evaluated as a dynamic-array formula and saved with cm, so Excel spilled it. Names were analysed again at every use, so a 399-term name used six times exhausted the 4096-node budget; their kinds are now remembered per formula, and the budget is 16384 nodes. The nesting limit is 128, so Excel's 64 levels of nested functions are analysed. A formula that still exceeds the limits is saved as it was read, without cm.
* XLSX: an implied @ is dropped and any other @ of a single value kept as _xlfn.SINGLE in a legacy formula (as Excel writes it); a legacy formula the analysis cannot finish or whose semantics it does not know opens with a whole-formula @ and never spills; analysis depth follows the parser's limit
* XLSX: a legacy union given to a function's single-value parameter keeps its implied @, so it calculates to #VALUE! instead of spilling and saves as read; checked on a calculated workbook
* XLSX: only a one-cell range an @ applies to counts as one an edit shrank; a reading past the parser's nesting limit leaves the formula as read, and the writer saves the text a formula it cannot analyse was read from
* XLSX: a formula nested past the parser's limit for its reading is left as read (documented: it may spill); test covers it
* XLSX: a name's definition counts its one-cell ranges as collapsed only when it is itself a reference, so CHOOSE's index or IF's condition in it keeps the name legacy
* XLSX: an @ on a name or one-cell range counts as implied when ranges an edit may have shrunk, anywhere the name returns them (CHOOSE's choices, parentheses), make it so; deleting rows keeps such formulas legacy
* XLSX: the collapsed-range retry of an @ runs only when the first pass reached a one-cell range written as a range (tracked through cached names)
* docs: describe unfinished formula export fallback
* docs: the unfinished-analysis fallback differs for formulas with an explicit @
* Preserve main formula analyzer fixes
* Order file-format revision history
---------
Co-authored-by: neilcawse <8085959+neilcawse@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Local contributor <contributor@localhost>
CEILING.MATH(6.7,VLOOKUP({"k","k"},A1:B1,2,FALSE)) with B1 empty now gives 0 per
element, as the scalar call does, instead of treating the significance as left out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FILTER(1,FALSE,VLOOKUP(...)) with a blank lookup result returned #CALC!
because util::has read the blank value as a left-out argument. Arguments
written in the formula that compute a blank are now marked as given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
COUNT never reads the running sum, so the zero-sum gate that keeps SUM
and AVERAGE additions in order does not apply to it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* ft-121.7.9: apply Set precision as displayed in calculation (#1343)
With the option on, a formula result (and each element of a spill) is
rounded to its cell's number format before it is stored and before other
formulas read it. Numbers typed, pasted or reformatted while it is on are
rounded at once, and turning it on through formulas.calculationOptions
{precisionAsDisplayed} rounds every number constant, as Excel does.
The rounding is NumberFormat::round_as_displayed in the numfmt crate,
which shares the renderer's digit logic (fixed, scientific, fraction).
General, date/time, text, digitless and conditional formats keep the full
value, as observed in Excel through COM. Nothing changes when the flag is
off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ft-121.7.9: round conditional formats by the sign's section under precision as displayed (#1343)
Excel does round formats with a condition when "Set precision as
displayed" is on; the PR kept them at full value. Observed through COM
with synthetic values (and in gridcraft-corpus
excel/precision-displayed/pad-on-conditional-sections.xlsx): conditions
are ignored and the sign picks the section, the first for positives and
the second for negatives. `[<1]0.00;0` shows 2+1/3 as 2 but stores 2.33,
and stores -10/3 as -3.
Two more observed details: with a condition and a single number section,
negatives keep the full value (`[<1]0.00`), unless an `@` section follows
(`[<1]0.00;@` rounds them by the first section); and a completely empty
section passes to the next one (`0.0;;0.000` stores -2.333 as -2.333,
`;0.0` stores 2.333 as 2.3), for formats with or without a condition.
Display is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* engine: recalculate spill anchors when a spill cell is reformatted under precision as displayed
round_edited_as_displayed only looked at stored-cell changes, but a spill
destination lives in Sheet::spill, so reformatting it (or its row or column) left
the anchor out of the changed set and the element unrounded. Add each spill anchor
whose range holds a changed cell or a changed row/column style to the changed set.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* numfmt: drop a cast the fraction merge made redundant
* precision as displayed: keep locked constants on protected sheets, round new sheets, fix manual-mode comment
* precision as displayed: name the input sheet in the new-sheet histogram test
* precision as displayed: use six values in the new-sheet histogram test
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* calc: list each dependent once when a change spreads
* calc: check the dependency closure against a walk of every cell
Random graphs of single-cell, narrow and wide range reads (some removed or unsettled), random seeds and changed areas: the closure with shared taken records and area lookups equals the one listing every dependent of every changed cell.
* calc: cap the text one & over arrays may build in all
* address review: concatenation over arrays stops at the byte budget without reserving the whole broadcast
Addresses review comments 4455115540 and 4455115557. The parser stores
`'Sheet'!Name` as `Sheet!Name` with the sheet already unquoted:
- Evaluator::name split it at the first `!`, so a series named
`'a!b'!Title` showed #NAME?.
- chart_name unquoted it again, trimming spaces, so a chart reference
to `' Sales '!Data` was saved as `Sales!Data`.
Both now split at the last `!` and look the sheet up as is.
Tests: chart tests::a_series_named_by_a_name_on_a_sheet_with_a_bang_resolves,
xlsx tests::chart_book_names::chart_names_keep_spaces_around_a_quoted_sheet_name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
python_result wrote into spills but not into spill_index, so indexed readers
(cell_value, range_values, visit_range) missed a Python array laid out in the
same pass. Insert on spill, remove when blocked, as the formula path does.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
python.set, and Replace in a Python cell's code, removed the cell's spill
registration and its spilled values before the recalculation ran, so the
recalculation never saw the freed range and formulas reading those cells kept
their old values until a full recalculation. The commands now leave the spill
to the recalculation. When a Python cell's content changes and it has no array
result to lay out, cells_changed removes the spill and queues the freed range,
as it does for a formula replaced by a constant, so its readers are
recalculated; a cell that keeps its array result (the same code again) keeps
its spill and lays it out again. Clearing, pasting over and blocking a Python
spill already recalculated its readers, and undo and redo restore the values
along with the cells; a test covers them.
A range read during recalculation walked every array laid out in the pass on
the sheet, whatever the range, and a block read (visit_range) gave up its fast
path as soon as any array was laid out there. A Python cell's cached result
can be up to 1,000,000 cells, so 1,000 formulas reading small ranges beside it
could visit about a billion values in one full recalculation; a large formula
array costs the same. Both reads now look only at the arrays whose area meets
the range, and only at the part inside it. The spilled values the sheet stores
from the last calculation are found row by row when the range has fewer rows
than there are spilled values, so a read beside them doesn't scan them either.
`broadcast_len`, the MAX_CELLS guard shared by MAP, the binary operators,
TEXT and array IF, multiplied `rows` and `cols` as `usize`. On wasm32 a
broadcast such as 65536 x 65536 (2^32 cells) wraps that product to 0 and
slips past the cap, after which the allocation and loop run far past
MAX_CELLS. The product is now formed in u64 and converted back only after
the cap check, so the guard means the same on every target.
`single`, the per-cell scalar check in MAP and MAKEARRAY, compared
`rows * cols` with 1. A constructed array stores exactly `rows * cols`
values, so it now tests `data.len()` directly: the same answer without an
unchecked product.
Tests cover MAX_CELLS exactly, one cell over, the 2^32 pair that wraps to
zero on a 32-bit usize, a full column x row (2^34), and MAP itself just
over the cap and at 16384 x 16384.
Closes#2165
A batch spread over several sheets used its total length for every sheet, so a
one-cell sheet with many spills built the block index as if it had the whole
batch's lookups.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A full-height column spill took 65,536 block registrations, on every pass
that produced it. Areas over 1024 blocks are now tested directly by each
lookup instead. Adds a randomized comparison against a scan of all areas.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Formula::with_expr runs a closure on the cached tree, or on a freshly parsed
one that is dropped afterwards, so rebuild and cells_changed need neither a
tree copy nor an Arc allocation / reference count.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>