Functions: GETPIVOTDATA (#284) (#290)

* Functions: GETPIVOTDATA (#284)

The engine records each PivotTable report value cell (value field,
row/column items, value) on refresh and when a workbook is opened;
GETPIVOTDATA is an evaluator special form that looks the value up by
value field and field/item pairs, #REF! when the report doesn't show it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* GETPIVOTDATA: shared lazy index, report values, hash lookup (review of #290)

- PivotTable::data_index is an Arc-shared PivotLookup (serde-skipped), so the
  per-command workbook clone no longer copies the index; refresh sets it, clear resets it.
- No index work on open: the first GETPIVOTDATA on a pivot without one builds it from the
  source via the engine-registered PIVOT_INDEXER; a failure is #REF!, not an empty index.
- The result is the value the sheet shows at the cell's report position, so formula and
  report cell agree (test with a source edited after the last refresh).
- Lookup by a pre-lowercased hash key; numbers/booleans fall back to a scan on a miss.
- GETPIVOTDATA is no longer volatile: it depends on its pivot's whole report area.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* GETPIVOTDATA: keep the index of the stored report; collision-free keys; align estimates

The lookup index describes the report as last refreshed and is saved with
the pivot. A pivot loaded without one (XLSX) only builds it when the
recomputed layout matches the stored labels. Lookup keys are length-prefixed
and distinguish Boolean from text items. Function-parity estimates match.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: import PivotDataCell, PivotItemRef and PivotLookup explicitly after upstream dropped the glob import

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: add an ignored measurement of edits on a sheet with a large report

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: keep a failed lazy GETPIVOTDATA build retryable, never save an empty index

A failed lazy index build stored unwrap_or_default() into the shared,
first-set-wins PivotLookup, so a valid state coming later (undo, a
source fix, a reopen) could never rebuild it, and the empty list was
saved as a restored index. Cache only validated builds, treat an empty
list as no index in serialization, reject reports with more value
cells than the index cap instead of truncating, register the pivot
indexer in gridcraft-cli check, and add two regression tests.

* GETPIVOTDATA: move cached offsets with edits inside the report; exact numeric items first

Inserting or deleting rows or columns inside a report remapped the cells
and last_range but kept the cached value-cell offsets, so a lookup could
read another item's cell. shift_sheet_features now maps each indexed
cell through the edit into a new index (the shared one stays with the
undo snapshots); a value cell deleted with its line drops out (#REF!).

A numeric item was accepted on a formatted-label hash hit, so 0.6 and 1
both shown as "1" returned the first. A number now prefers the cell
whose item has exactly that value, before any label match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix pivot chart test for the new Output.data field

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valentin KleinandClaude Opus 5.5 authored and GitHub committed 2026-11-16 08:59:28 +01:00
1 parent 19f07f3a25
commit ef99f1e51f
21 files changed
+846 -37

No files matched your search

+10 -13
View File
@@ -12,13 +12,9 @@ Excel's worksheet functions, category by category, against GridCraft's function
them), plus 14 text functions the page lists under combined entries (FIND/FINDB, LEFT/LEFTB,
LEN/LENB, MID/MIDB, REPLACE/REPLACEB, RIGHT/RIGHTB, SEARCH/SEARCHB), BETA.INV, and COPILOT and
PY (Microsoft 365 functions not on that page when fetched).
- **Ours (505):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (503):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (502):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (502):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (505):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (514):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
evaluator's special forms in `crates/calc/src/eval.rs` (`SPECIAL_FUNCTIONS`: IF, IFS, CHOOSE,
OFFSET, INDIRECT, INDEX, SUBTOTAL, AGGREGATE, TEXT, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL,
OFFSET, INDIRECT, INDEX, GETPIVOTDATA, SUBTOTAL, AGGREGATE, TEXT, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL,
MAKEARRAY…), excluding the test-only `ADD2`. Every one is an Excel name.
- **Not on Microsoft's list:** USDOLLAR and ECMA.CEILING, legacy functions Excel still evaluates (like DOLLAR),
are in our library as Compatibility functions for files that use them. They are not counted above.
@@ -29,7 +25,7 @@ Excel's worksheet functions, category by category, against GridCraft's function
## Coverage
**Measured: 513 / 523 by name (98.1%); 504 / 523 working (96.4%).**
**Measured: 514 / 523 by name (98.3%); 505 / 523 working (96.6%).**
| Category | Excel | Ours | Working | % working | Missing |
|---|---|---|---|---|---|
@@ -38,7 +34,7 @@ Excel's worksheet functions, category by category, against GridCraft's function
| Financial | 55 | 55 | 55 | 100% | — |
| Engineering | 54 | 54 | 54 | 100% | — |
| Text | 50 | 48 | 48 | 96% | DETECTLANGUAGE, TRANSLATE |
| Lookup and reference | 40 | 36 | 36 | 90% | GETPIVOTDATA, IMAGE, RTD, TRIMRANGE |
| Lookup and reference | 40 | 38 | 38 | 95% | IMAGE, RTD |
| Compatibility | 40 | 40 | 40 | 100% | — |
| Date and time | 25 | 25 | 25 | 100% | — |
| Information | 22 | 21 | 21 | 95% | STOCKHISTORY |
@@ -48,10 +44,10 @@ Excel's worksheet functions, category by category, against GridCraft's function
| Web | 3 | 3 | 1 | 33% | (WEBSERVICE, FILTERXML are stubs; ENCODEURL works) |
| Add-in and Automation | 3 | 1 | 1 | 33% | CALL, REGISTER.ID |
| AI and Python | 2 | 0 | 0 | 0% | COPILOT, PY |
| **Total** | **523** | **505** | **496** | **94.8%** | 18 missing, 9 stubs |
| **Total** | **523** | **514** | **505** | **96.6%** | 9 missing, 9 stubs |
By the Excel version that introduced them (working / total): before 2010 346 / 360, 2010
56 / 56, 2013 49 / 51, 2016 1 / 5, 2019 6 / 6, 2021 11 / 11, 2024 22 / 23, Microsoft 365 5 / 11.
By the Excel version that introduced them (working / total): before 2010 349 / 360, 2010
56 / 56, 2013 49 / 51, 2016 5 / 5, 2019 6 / 6, 2021 11 / 11, 2024 22 / 23, Microsoft 365 7 / 11.
Dynamic-array and LAMBDA-era functions (FILTER, SORT, UNIQUE, XLOOKUP, VSTACK, TAKE, GROUPBY,
PIVOTBY, REGEXTEST…) are all present.
@@ -66,7 +62,7 @@ This improves series preparation; it does not change the model-fitting or season
|---|---|---|---|
| FORECAST.ETS family (4) | yes | 4–6 h | AAA exponential smoothing with seasonality detection; needed for Forecast Sheet too |
| ODDFPRICE, ODDFYIELD, ODDLPRICE, ODDLYIELD | yes | 2–4 h | Odd first/last period bond maths |
| GETPIVOTDATA | yes | 2–3 h | Needs the pivot engine's result cache; also the "Generate GetPivotData" option |
| Generate GetPivotData | yes | 1–2 h | GETPIVOTDATA works; clicking a PivotTable value while typing a formula doesn't insert it yet |
| IMAGE | yes | 3–5 h | Pictures in cells (see PR #105); fetching URLs needs a privacy decision |
| CUBE* (7), WEBSERVICE, FILTERXML | partly | 4–8 h | FILTERXML can work offline; CUBE needs an OLAP/data-model source |
| TRANSLATE, DETECTLANGUAGE, STOCKHISTORY, COPILOT, PY | no (cloud) | — | Microsoft cloud services; open equivalents only by owner decision |
@@ -128,7 +124,7 @@ inputs, so large finite middle values do not become `#NUM!` merely because their
Sorting, argument coercion and error propagation are unchanged; this is a numerical correction
under the documented median rule, not an extreme-value comparison against desktop Excel.
Work to close (16–26 h in total): the in-scope missing functions (10–16 h with the table above),
Work to close (15–25 h in total): the in-scope missing functions (10–16 h with the table above),
an oracle corpus of Excel-computed results per function (owner runs Excel on generated
workbooks; results checked in as data, not Excel files) and fixing what it finds (6–10 h).
@@ -183,6 +179,7 @@ Positive fractional inputs still truncate, and signed zero remains valid. Covere
| 2026-10-14 | minor | Bounded references up to 1,048,576 cells (`D3:AZ400`) keep their full shape instead of stopping at the used range, so `SUMPRODUCT(D3:AZ400*B3:B400,(A3:A400=x)*(D1:AZ1=y))` broadcasts like Excel and `ROWS`, `COUNTBLANK` and `--(r="")` count the blanks; references that run to the edge of the sheet (`A:A`) still stop at the used range (#1010) |
| 2026-10-13 | minor | Legacy formulas from files (no dynamic-array metadata) evaluate with implicit intersection: cells differing from Excel's cached values 68,964 → 49,866 across 4,224 test-corpus workbooks (none gets more) |
| 2026-10-13 | minor | Clear reports changed stored cells sparsely, including those beyond the first million coordinates; dependent formulas and removed spills update, while clearing empty merges still unblocks spills (#1032) |
| 2026-10-12 | minor | GETPIVOTDATA added: 514 / 523 by name, 505 working |
| 2026-10-12 | minor | USDOLLAR added as a Compatibility function (not on Microsoft's list, so the counts don't change); MIRR with no positive flow is -1, COUNTIF's bare `"<="` / `">="` skip empty strings, INDIRECT with `a1` = FALSE gives `#REF!` for A1-style text (#738) |
| 2026-10-11 | minor | TEXT / number formats fixed against Excel-cached results: fraction literals and zero fractions, `0` denominators zero padded, `?` pads the thousands separator, booleans use the text section (226 mismatches → 0 in 352 TEXT cells) |
| 2026-10-11 | minor | Iterative calculation: circular references iterate up to Maximum Iterations / Maximum Change when enabled |