Functions: GETPIVOTDATA (#284) (#290)

* Functions: GETPIVOTDATA (#284)

The engine records each PivotTable report value cell (value field,
row/column items, value) on refresh and when a workbook is opened;
GETPIVOTDATA is an evaluator special form that looks the value up by
value field and field/item pairs, #REF! when the report doesn't show it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* GETPIVOTDATA: shared lazy index, report values, hash lookup (review of #290)

- PivotTable::data_index is an Arc-shared PivotLookup (serde-skipped), so the
  per-command workbook clone no longer copies the index; refresh sets it, clear resets it.
- No index work on open: the first GETPIVOTDATA on a pivot without one builds it from the
  source via the engine-registered PIVOT_INDEXER; a failure is #REF!, not an empty index.
- The result is the value the sheet shows at the cell's report position, so formula and
  report cell agree (test with a source edited after the last refresh).
- Lookup by a pre-lowercased hash key; numbers/booleans fall back to a scan on a miss.
- GETPIVOTDATA is no longer volatile: it depends on its pivot's whole report area.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* GETPIVOTDATA: keep the index of the stored report; collision-free keys; align estimates

The lookup index describes the report as last refreshed and is saved with
the pivot. A pivot loaded without one (XLSX) only builds it when the
recomputed layout matches the stored labels. Lookup keys are length-prefixed
and distinguish Boolean from text items. Function-parity estimates match.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: import PivotDataCell, PivotItemRef and PivotLookup explicitly after upstream dropped the glob import

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: add an ignored measurement of edits on a sheet with a large report

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: keep a failed lazy GETPIVOTDATA build retryable, never save an empty index

A failed lazy index build stored unwrap_or_default() into the shared,
first-set-wins PivotLookup, so a valid state coming later (undo, a
source fix, a reopen) could never rebuild it, and the empty list was
saved as a restored index. Cache only validated builds, treat an empty
list as no index in serialization, reject reports with more value
cells than the index cap instead of truncating, register the pivot
indexer in gridcraft-cli check, and add two regression tests.

* GETPIVOTDATA: move cached offsets with edits inside the report; exact numeric items first

Inserting or deleting rows or columns inside a report remapped the cells
and last_range but kept the cached value-cell offsets, so a lookup could
read another item's cell. shift_sheet_features now maps each indexed
cell through the edit into a new index (the shared one stays with the
undo snapshots); a value cell deleted with its line drops out (#REF!).

A numeric item was accepted on a formatted-label hash hit, so 0.6 and 1
both shown as "1" returned the first. A number now prefers the cell
whose item has exactly that value, before any label match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix pivot chart test for the new Output.data field

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Valentin KleinandClaude Opus 5.5 authored and GitHub committed 2026-11-16 08:59:28 +01:00
1 parent 19f07f3a25
commit ef99f1e51f
21 files changed
+846 -37

No files matched your search

+10 -13
View File
@@ -12,13 +12,9 @@ Excel's worksheet functions, category by category, against GridCraft's function
them), plus 14 text functions the page lists under combined entries (FIND/FINDB, LEFT/LEFTB,
LEN/LENB, MID/MIDB, REPLACE/REPLACEB, RIGHT/RIGHTB, SEARCH/SEARCHB), BETA.INV, and COPILOT and
PY (Microsoft 365 functions not on that page when fetched).
- **Ours (505):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (503):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (502):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (502):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (505):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
- **Ours (514):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the
evaluator's special forms in `crates/calc/src/eval.rs` (`SPECIAL_FUNCTIONS`: IF, IFS, CHOOSE,
OFFSET, INDIRECT, INDEX, SUBTOTAL, AGGREGATE, TEXT, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL,
OFFSET, INDIRECT, INDEX, GETPIVOTDATA, SUBTOTAL, AGGREGATE, TEXT, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL,
MAKEARRAY…), excluding the test-only `ADD2`. Every one is an Excel name.
- **Not on Microsoft's list:** USDOLLAR and ECMA.CEILING, legacy functions Excel still evaluates (like DOLLAR),
are in our library as Compatibility functions for files that use them. They are not counted above.
@@ -29,7 +25,7 @@ Excel's worksheet functions, category by category, against GridCraft's function
## Coverage
**Measured: 513 / 523 by name (98.1%); 504 / 523 working (96.4%).**
**Measured: 514 / 523 by name (98.3%); 505 / 523 working (96.6%).**
| Category | Excel | Ours | Working | % working | Missing |
|---|---|---|---|---|---|
@@ -38,7 +34,7 @@ Excel's worksheet functions, category by category, against GridCraft's function
| Financial | 55 | 55 | 55 | 100% | — |
| Engineering | 54 | 54 | 54 | 100% | — |
| Text | 50 | 48 | 48 | 96% | DETECTLANGUAGE, TRANSLATE |
| Lookup and reference | 40 | 36 | 36 | 90% | GETPIVOTDATA, IMAGE, RTD, TRIMRANGE |
| Lookup and reference | 40 | 38 | 38 | 95% | IMAGE, RTD |
| Compatibility | 40 | 40 | 40 | 100% | — |
| Date and time | 25 | 25 | 25 | 100% | — |
| Information | 22 | 21 | 21 | 95% | STOCKHISTORY |
@@ -48,10 +44,10 @@ Excel's worksheet functions, category by category, against GridCraft's function
| Web | 3 | 3 | 1 | 33% | (WEBSERVICE, FILTERXML are stubs; ENCODEURL works) |
| Add-in and Automation | 3 | 1 | 1 | 33% | CALL, REGISTER.ID |
| AI and Python | 2 | 0 | 0 | 0% | COPILOT, PY |
| **Total** | **523** | **505** | **496** | **94.8%** | 18 missing, 9 stubs |
| **Total** | **523** | **514** | **505** | **96.6%** | 9 missing, 9 stubs |
By the Excel version that introduced them (working / total): before 2010 346 / 360, 2010
56 / 56, 2013 49 / 51, 2016 1 / 5, 2019 6 / 6, 2021 11 / 11, 2024 22 / 23, Microsoft 365 5 / 11.
By the Excel version that introduced them (working / total): before 2010 349 / 360, 2010
56 / 56, 2013 49 / 51, 2016 5 / 5, 2019 6 / 6, 2021 11 / 11, 2024 22 / 23, Microsoft 365 7 / 11.
Dynamic-array and LAMBDA-era functions (FILTER, SORT, UNIQUE, XLOOKUP, VSTACK, TAKE, GROUPBY,
PIVOTBY, REGEXTEST…) are all present.
@@ -66,7 +62,7 @@ This improves series preparation; it does not change the model-fitting or season
|---|---|---|---|
| FORECAST.ETS family (4) | yes | 4–6 h | AAA exponential smoothing with seasonality detection; needed for Forecast Sheet too |
| ODDFPRICE, ODDFYIELD, ODDLPRICE, ODDLYIELD | yes | 2–4 h | Odd first/last period bond maths |
| GETPIVOTDATA | yes | 2–3 h | Needs the pivot engine's result cache; also the "Generate GetPivotData" option |
| Generate GetPivotData | yes | 1–2 h | GETPIVOTDATA works; clicking a PivotTable value while typing a formula doesn't insert it yet |
| IMAGE | yes | 3–5 h | Pictures in cells (see PR #105); fetching URLs needs a privacy decision |
| CUBE* (7), WEBSERVICE, FILTERXML | partly | 4–8 h | FILTERXML can work offline; CUBE needs an OLAP/data-model source |
| TRANSLATE, DETECTLANGUAGE, STOCKHISTORY, COPILOT, PY | no (cloud) | — | Microsoft cloud services; open equivalents only by owner decision |
@@ -128,7 +124,7 @@ inputs, so large finite middle values do not become `#NUM!` merely because their
Sorting, argument coercion and error propagation are unchanged; this is a numerical correction
under the documented median rule, not an extreme-value comparison against desktop Excel.
Work to close (16–26 h in total): the in-scope missing functions (10–16 h with the table above),
Work to close (15–25 h in total): the in-scope missing functions (10–16 h with the table above),
an oracle corpus of Excel-computed results per function (owner runs Excel on generated
workbooks; results checked in as data, not Excel files) and fixing what it finds (6–10 h).
@@ -183,6 +179,7 @@ Positive fractional inputs still truncate, and signed zero remains valid. Covere
| 2026-10-14 | minor | Bounded references up to 1,048,576 cells (`D3:AZ400`) keep their full shape instead of stopping at the used range, so `SUMPRODUCT(D3:AZ400*B3:B400,(A3:A400=x)*(D1:AZ1=y))` broadcasts like Excel and `ROWS`, `COUNTBLANK` and `--(r="")` count the blanks; references that run to the edge of the sheet (`A:A`) still stop at the used range (#1010) |
| 2026-10-13 | minor | Legacy formulas from files (no dynamic-array metadata) evaluate with implicit intersection: cells differing from Excel's cached values 68,964 → 49,866 across 4,224 test-corpus workbooks (none gets more) |
| 2026-10-13 | minor | Clear reports changed stored cells sparsely, including those beyond the first million coordinates; dependent formulas and removed spills update, while clearing empty merges still unblocks spills (#1032) |
| 2026-10-12 | minor | GETPIVOTDATA added: 514 / 523 by name, 505 working |
| 2026-10-12 | minor | USDOLLAR added as a Compatibility function (not on Microsoft's list, so the counts don't change); MIRR with no positive flow is -1, COUNTIF's bare `"<="` / `">="` skip empty strings, INDIRECT with `a1` = FALSE gives `#REF!` for A1-style text (#738) |
| 2026-10-11 | minor | TEXT / number formats fixed against Excel-cached results: fraction literals and zero fractions, `0` denominators zero padded, `?` pads the thousands separator, booleans use the text section (226 mismatches → 0 in 352 TEXT cells) |
| 2026-10-11 | minor | Iterative calculation: circular references iterate up to Maximum Iterations / Maximum Change when enabled |
+4 -6
View File
@@ -91,8 +91,8 @@ No alpha blockers: every core workflow passes the [alpha gate](roadmap.md#alpha-
### 8. PivotTables: no slicers, calculated items, PivotCharts; unverified in Excel (F)
- **Evidence:** `parity-checklist.md` missing `insert.slicer`, `insert.timeline`,
`insert.pivotChart`, `table.insertSlicer`; calculated items and GETPIVOTDATA missing (calculated fields
and values on rows done). Show Details (double-click a value, subtotal or grand total cell) lists
`insert.pivotChart`, `table.insertSlicer`; calculated items and Generate GetPivotData missing (calculated fields,
values on rows and GETPIVOTDATA done). Show Details (double-click a value, subtotal or grand total cell) lists
its source records on a new sheet. The "Enable show details" option (`pivot.options`
`enableShowDetails`, saved as `enableDrill`) turns it off; it has no PivotTable Options dialog
yet (command only).
@@ -142,10 +142,7 @@ No alpha blockers: every core workflow passes the [alpha gate](roadmap.md#alpha-
Margins are dragged on the rulers (#792). Remaining: pictures in headers/footers, Different First Page / Odd & Even pages. Desktop now submits directly through craft-print with printer choice, copies, duplex and grayscale; browser printing stays available. Print ranges and driver tray/options UI remain. **Estimate:** 5–10 h.
### 15. Remaining functions and an oracle suite (F)
- 18 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 16–26 h.
- 20 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 17–29 h.
- 21 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 18–30 h.
- 18 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 18–30 h.
- 9 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 15–25 h.
[function-parity.md](function-parity.md)
### 16. Collaboration and review (F)
@@ -212,6 +209,7 @@ No alpha blockers: every core workflow passes the [alpha gate](roadmap.md#alpha-
| 2026-10-15 | minor | §13: Solver prepares reports before committing the solution, model and report sheets together; report refusal preserves the workbook and Undo/Redo (#1050) |
| 2026-10-13 | trivial | §4: multi-level category axes landed |
| 2026-10-12 | minor | §9: next/previous sheet commands retain the current visible sheet and selection when there is no visible destination |
| 2026-10-12 | minor | §8, §15: GETPIVOTDATA added (9 functions missing) |
| 2026-10-12 | minor | rows auto-fit oversized fonts on open and whole-row/column formatting (#833) |
| 2026-10-12 | minor | Page Layout view: dragging margins on the rulers (#792) removed from the §14 remainder |
| 2026-10-12 | minor | §13: Forecast Sheet Forecast Start (hindcast) landed (#785) |
+5 -4
View File
@@ -14,8 +14,8 @@ generated ribbon checklist is [`parity-checklist.md`](parity-checklist.md).
|---|---|---|
| Feature breadth (weighted by use) | **~75%** | estimated from the area table below; anchored on two measurements |
| Ribbon and menu catalog | **281 / 296 (94.9%)** | measured: `cargo xtask parity` (`crates/engine/src/catalog.rs` ∩ registered command ids) |
| Worksheet functions, by name | **513 / 523 (98.1%)** | measured: registered functions vs Microsoft's published function list (method below) |
| Worksheet functions that do real work | **504 / 523 (96.4%)** | measured: excludes 9 stubs (7 CUBE functions, WEBSERVICE, FILTERXML) |
| Worksheet functions, by name | **514 / 523 (98.3%)** | measured: registered functions vs Microsoft's published function list (method below) |
| Worksheet functions that do real work | **505 / 523 (96.6%)** | measured: excludes 9 stubs (7 CUBE functions, WEBSERVICE, FILTERXML) |
| **Ready for real work** | **~50%** | estimated; written weights below (51.6 computed) |
| **Mainstream practitioner** | **~50%** | estimated; [method](#mainstream-practitioner) (69.5 depth × 0.90 × 0.92 × 0.88 = 50.6) |
| **Essentials user** | **~63%** | estimated; [method](#essentials-user) (80.2 depth × 0.92 × 0.95 × 0.90 = 63.1) |
@@ -81,11 +81,11 @@ performance 8, stability 8, localization 4, hardware 2, platforms 2, ecosystem 2
| Area | Weight | Breadth | Ready | Hours | Evidence |
|---|---|---|---|---|---|
| Formula language and calculation | 18 | ~92% | ~72% | 20–30 | A1/R1C1 parser, reference adjustment, dependency-graph recalc, dynamic arrays and `#SPILL!`, `A1#`, LET/LAMBDA and helpers, structured references, names, 3-D references. Open correctness bugs: 15-digit equality (#57), stale spills after structural edits and cleared blockers (#133, #139), 3-D span endpoint deletion (#140), sheet rename and qualified names (#134). Single-threaded |
| Worksheet functions | 12 | 98.1% (measured) | ~82% | 16–26 | 513 / 523 names, 504 working. Missing: GETPIVOTDATA, IMAGE plus cloud functions. No conformance suite against Excel-computed oracles; edge-case bugs keep surfacing (#137, #138). [function-parity.md](function-parity.md) |
| Worksheet functions | 12 | 98.3% (measured) | ~82% | 15–25 | 514 / 523 names, 505 working. Missing: IMAGE plus cloud functions. No conformance suite against Excel-computed oracles; edge-case bugs keep surfacing (#137, #138). [function-parity.md](function-parity.md) |
| Editing, navigation, clipboard, fill | 12 | ~90% | ~70% | 15–25 | Point mode, F4, AutoComplete, Paste Special, AutoFill, Flash Fill, Find/Replace, Go To Special. Bugs: rectangular drag-select (#28), monthly fill (#132), cut across sheets (#128), Insert Copied Cells missing (#21), key tips only on part of the Home tab (#43; #33, #49). [ui-parity.md](ui-parity.md) |
| Cell formatting, styles, conditional formatting | 10 | ~95% | ~80% | 8–14 | Full number-format language, borders, 47 cell styles, 60 table styles, all CF rule kinds including data bars, colour scales and icon sets. Gaps: font weights beyond regular/bold (#168), system fonts list (#4), themes |
| Sort, filter, tables, validation, outline | 8 | ~82% | ~70% | 10–16 | Multi-level sort, AutoFilter, Advanced Filter, tables with totals, validation, outline, subtotals, consolidate. Advanced Filter copy-to preserves cells outside its result and rejects worksheet overflow (#1031). Bugs: subtotal header (#131), validation edits (#129), table style panic caught (#130); no slicers on tables |
| PivotTables | 7 | ~65% | ~45% | 25–40 | Insert/Recommended, field list, three layouts, values on rows or columns, date grouping, show-values-as, refresh, calculated fields, XLSX round-trip. Missing: slicers, timelines, calculated items, PivotCharts, value/label filters, number grouping, GETPIVOTDATA, data-model pivots. Round-trip never checked in Excel |
| PivotTables | 7 | ~65% | ~45% | 25–40 | Insert/Recommended, field list, three layouts, values on rows or columns, date grouping, show-values-as, refresh, calculated fields, GETPIVOTDATA, XLSX round-trip. Missing: slicers, timelines, calculated items, PivotCharts, value/label filters, number grouping, Generate GetPivotData, data-model pivots. Round-trip never checked in Excel |
| Charts and sparklines | 8 | ~50% | ~30% | 40–60 | 25 chart kinds, all 2-D. Doughnut charts draw every series as a concentric ring (first series innermost), with editable hole size and first slice angle. The chart model holds title, legend position, a data-labels flag, gridlines flag, axis titles and chart data tables (legend keys, borders, XLSX round-trip and print/PDF); no axis scaling, error bars, per-point formatting, chart sheets, 3-D, surface, map, Pareto. Kinds survive XLSX round trip (#158, #161), except stock charts: only 3–4 series are a stock chart, and any other count is written and reimported as a line chart. [chart-parity.md](chart-parity.md) |
| Page layout, print, PDF | 5 | ~80% | ~55% | 12–20 | Page setup, print titles, breaks, page order, Page Break Preview, headers/footers, PDF export and printing. Paper sizes match Japanese Excel: B4/B5/B6 are the JIS sizes, ISO B4/B5/B6 stay reachable by name, and Japanese Postcard (はがき), Japanese Double Postcard (往復はがき), the chou/kaku envelopes and A6 are available. No true Page Layout view; the PDF writer uses only the 14 standard fonts without embedding (no CJK or other non-Latin text in PDF); web print only exports (#141, #143) |
| What-if and analysis | 4 | ~50% | ~40% | 22–35 | Goal Seek, Scenario Manager, data tables, Analysis ToolPak (all 19 tools, no charts), Solver (Simplex LP, a local nonlinear method and Evolutionary; Answer, Sensitivity and Limits reports; Load/Save Model, Reset All and Solver Options; no Population report), Forecast Sheet. Missing: Analyze Data |
@@ -351,6 +351,7 @@ The inventory carried over from the 2026-10-07 ROADMAP.md, updated for what land
| 2026-10-18 | trivial | Stale chart notes: all chart kinds survive XLSX round-trip (PRs #158, #161) and trendlines landed |
| 2026-10-16 | minor | Advanced Filter copy-to clears only its actual output and rejects overflow without changing cells (#1031) |
| 2026-10-15 | minor | Row auto-fit: line breaks count only in wrapping cells (Excel: wrap off shows one line, AutoFit gives one line); unwrapped multi-line cells from files no longer grow rows on open (no score change) |
| 2026-10-12 | minor | Worksheet functions 514 / 523 (98.3%; 505 working): GETPIVOTDATA |
| 2026-10-21 | minor | Automation: typed macros in the engine and `macro.*` commands, no interface yet (no score change); catalog 274 / 290 → 275 / 291 (View › Macros › Record Macro) |
| 2026-10-12 | minor | Essentials AutoFit evidence: rows auto-fit oversized fonts on open and row formatting (#833) |
| 2026-10-12 | minor | Sheet tabs: reveal the selected tab on navigation and resize while preserving manual scrolling (#681); no score change |