diff --git a/README.md b/README.md index f2d9a930e..a58f4fb19 100644 --- a/README.md +++ b/README.md @@ -205,7 +205,7 @@ egui frontend is a separate crate, so the UI can be swapped without touching the GridCraft is in **alpha**: the core workflow works end to end, but depth, fidelity and file compatibility are still rough. It covers 89% of standard ribbon commands in our catalog -([`docs/parity-checklist.md`](docs/parity-checklist.md)) and 513 standard worksheet functions +([`docs/parity-checklist.md`](docs/parity-checklist.md)) and 514 standard worksheet functions ([`docs/function-parity.md`](docs/function-parity.md)); this is a parity inventory, not a count of working implementations. Weighted by what spreadsheet users rely on, we estimate it is about 50% ready for real work. Next up: XLSX fidelity on real-world files, very large workbooks and multi-threaded recalc, chart depth, text in every script, and `.xls` import. Stage, diff --git a/ROADMAP.md b/ROADMAP.md index 85e83feb9..6632e6989 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -22,7 +22,7 @@ This page is the summary. The detail is in: | Number | Value | Kind | |---|---|---| | 2026-10-14 | Editing a cell with text that would read back as a number, boolean, date or formula (entered with a leading apostrophe) shows the apostrophe in the editor and formula bar, so Enter unchanged keeps it text (#1362) | -| **Feature breadth** (weighted) | **~75%** | estimated, anchored on two measurements: ribbon catalog 281 / 296 (94.9%), worksheet functions 513 / 523 (98.1%; 504 working, 96.4%) | +| **Feature breadth** (weighted) | **~75%** | estimated, anchored on two measurements: ribbon catalog 281 / 296 (94.9%), worksheet functions 514 / 523 (98.3%; 505 working, 96.6%) | | **Ready for real work** | **~50%** | estimated from written weights (51.6); [target-app-parity.md](docs/target-app-parity.md) | | **Mainstream practitioner** | **~50%** | estimated: typical analyst/finance user, weekly features, × 0.90 interaction × 0.92 stability × 0.88 file exchange; [method](docs/target-app-parity.md#mainstream-practitioner) | | **Essentials user** | **~63%** | estimated: core features only, × 0.92 launch/stability × 0.95 clarity × 0.90 opening sent files; [method](docs/target-app-parity.md#essentials-user) | @@ -74,7 +74,7 @@ collaboration, mobile, AI). | Area | Breadth | Ready | Hours | Doc | |---|---|---|---|---| | Formula language and calculation | ~92% | ~72% | 20–30 | [target-app-parity.md](docs/target-app-parity.md#feature-areas) | -| Worksheet functions | 96.6% measured | ~82% | 16–26 | [function-parity.md](docs/function-parity.md) | +| Worksheet functions | 96.6% measured | ~82% | 15–25 | [function-parity.md](docs/function-parity.md) | | Editing, navigation, clipboard, fill | ~90% | ~70% | 15–25 | [ui-parity.md](docs/ui-parity.md) | | Cell formatting, styles, conditional formatting | ~95% | ~80% | 8–14 | [target-app-parity.md](docs/target-app-parity.md#feature-areas) | | Sort, filter, tables, validation, outline | ~82% | ~70% | 10–16 | [target-app-parity.md](docs/target-app-parity.md#feature-areas) | @@ -202,6 +202,7 @@ Ranked; estimates in Opus 5.5 agent-hours. Detail and the full beta-gate list in | 2026-10-13 | Conditional formatting: a rule gathers range statistics only when it reads them (top/bottom, above average, duplicates, scales, bars, icons) and borrows them instead of copying them per cell, so the per-frame lookup of an expression or cell-value rule no longer visits its whole range (#1098, part 1) | | 2026-10-13 | Grid: resizing several selected columns or rows previews on the dragged one and applies the size to all on release; Esc cancels (#1082) | | 2026-10-12 | Next/previous sheet navigation stops at the first/last visible worksheet instead of activating a hidden boundary sheet; the current selection is retained | +| 2026-10-12 | GETPIVOTDATA reads PivotTable values by field and items (#284) | | 2026-10-12 | PDF writer: objects are written straight to the output, identical pictures are stored once, style operators are emitted only when they change, `q` nesting is limited to 28 and polyline joins no longer leak into later strokes (#816) | | 2026-10-12 | Insert › Text › WordArt (`insert.wordArt`, `wordArt.set`): floating decorative text with 10 original styles, outline and shadow, in-place editing, XLSX and print (#784) | | 2026-10-12 | Number formatting: one table of built-in formats (xlsx); numbers, dates and General render with one scratch buffer instead of per-token strings; output unchanged, checked against digests recorded from the old code (#817) | @@ -321,6 +322,7 @@ Ranked; estimates in Opus 5.5 agent-hours. Detail and the full beta-gate list in | 2026-10-13 | trivial | Progress log: shared pictures written once on XLSX export (#985) | | 2026-10-13 | minor | Progress log: conditional-format lookups no longer scan the range for rules without statistics (#1098) | | 2026-10-12 | minor | Progress log: hidden-sheet navigation at workbook boundaries | +| 2026-10-12 | minor | Worksheet functions 514 / 523 (GETPIVOTDATA); progress log row | | 2026-10-13 | trivial | Progress log: charts cached between frames (#1139) | | 2026-10-12 | minor | Progress log: PDF writer cleanup (#816) | | 2026-10-12 | minor | Catalog 276 → 277 / 292: Insert › WordArt (#784) | diff --git a/apps/gridcraft-cli/src/check.rs b/apps/gridcraft-cli/src/check.rs index cfd583baf..1c14812db 100644 --- a/apps/gridcraft-cli/src/check.rs +++ b/apps/gridcraft-cli/src/check.rs @@ -368,6 +368,8 @@ pub fn check_bytes(path: &str, bytes: &[u8], opts: &Options) -> FileReport { } fn check_inner(path: &str, bytes: &[u8], opts: &Options) -> FileReport { + // GETPIVOTDATA needs the engine's pivot indexer; without it every such cell reads #REF!. + let _ = gridcraft_engine::model::PIVOT_INDEXER.set(gridcraft_engine::pivot::index_cells); let mut report = FileReport { path: path.to_string(), ..Default::default() }; let (cached, warnings) = match gridcraft_engine::io::open_bytes(path, bytes) { Ok(x) => x, diff --git a/crates/calc/src/eval/call.rs b/crates/calc/src/eval/call.rs index eb426398a..25c194758 100644 --- a/crates/calc/src/eval/call.rs +++ b/crates/calc/src/eval/call.rs @@ -72,6 +72,7 @@ impl<'h> Evaluator<'h> { "XLOOKUP" => self.xlookup(args), "CELL" => self.cell_info(args), "TRIMRANGE" => self.trimrange(args), + "GETPIVOTDATA" => self.getpivotdata(args), "SUBTOTAL" | "AGGREGATE" => self.subtotal(name, args), "TEXT" => { if args.len() != 2 { diff --git a/crates/calc/src/eval/special.rs b/crates/calc/src/eval/special.rs index 2e7083457..c75d489d8 100644 --- a/crates/calc/src/eval/special.rs +++ b/crates/calc/src/eval/special.rs @@ -136,6 +136,25 @@ impl Precedents<'_> { (_, Some(Expr::Number(_) | Expr::Text(_) | Expr::Bool(_) | Expr::Error(_) | Expr::Array(_) | Expr::Missing)) => {} _ => self.dynamic = true, }, + // GETPIVOTDATA reads the report cells, so it depends on the whole report its reference + // points into (pivot commands recalculate everything). Any other reference is dynamic. + Expr::Call(n, args) if n == "GETPIVOTDATA" => match args.get(1) { + Some(Expr::Ref(r)) => { + let si = match &r.sheet { + SheetSel::Current => Some(self.sheet), + SheetSel::Named(name) => self.wb.sheet_index(name), + SheetSel::Span(..) => None, + }; + let report = si + .and_then(|i| self.wb.sheet(i)) + .and_then(|sh| sh.pivots.iter().find_map(|p| p.last_range.filter(|lr| lr.intersects(&r.range())))); + match report { + Some(lr) => self.push(r, lr, boxes), + None => self.dynamic = true, + } + } + _ => self.dynamic = true, + }, Expr::Call(n, _) if matches!(n.as_str(), "INDIRECT" | "OFFSET" | "NOW" | "TODAY" | "RAND" | "RANDBETWEEN" | "RANDARRAY" | "CELL" | "INFO") => { @@ -199,6 +218,7 @@ fn is_special(n: &str) -> bool { | "INDEX" | "CELL" | "TRIMRANGE" + | "GETPIVOTDATA" | "SUBTOTAL" | "AGGREGATE" | "TEXT" @@ -237,6 +257,7 @@ pub const SPECIAL_FUNCTIONS: &[&str] = &[ "INDEX", "CELL", "TRIMRANGE", + "GETPIVOTDATA", "SUBTOTAL", "AGGREGATE", "TEXT", diff --git a/crates/calc/src/lib.rs b/crates/calc/src/lib.rs index e30cff29f..2f7933d71 100644 --- a/crates/calc/src/lib.rs +++ b/crates/calc/src/lib.rs @@ -4,6 +4,7 @@ #![forbid(unsafe_code)] pub mod eval; +mod pivotdata; mod precision; pub mod recalc; mod spill_index; diff --git a/crates/calc/src/pivotdata.rs b/crates/calc/src/pivotdata.rs new file mode 100644 index 000000000..90296cd43 --- /dev/null +++ b/crates/calc/src/pivotdata.rs @@ -0,0 +1,149 @@ +//! GETPIVOTDATA: a value from a PivotTable report, found by its value field and the row and +//! column items that locate it (Excel's documented semantics). Each PivotTable's value cells are +//! indexed in `PivotTable::data_index` (set on refresh and saved with the report; built on first +//! use only for a pivot loaded without one, and only when it matches the stored report); the +//! result is read from the report cell the sheet shows. + +use gridcraft_core::{CellError, CellRef, Value}; +use gridcraft_formula::Expr; +use gridcraft_model::{PIVOT_INDEXER, PivotDataCell, PivotItemRef, PivotLookup, PivotTable}; + +use crate::eval::{Ev, Evaluator}; + +/// Excel allows 126 field/item pairs. +const MAX_PAIRS: usize = 126; + +fn same_text(a: &str, b: &str) -> bool { + a.trim().to_lowercase() == b.trim().to_lowercase() +} + +/// A number matches an item with that value (dates as serials) or with that label; anything +/// else matches the label as text. With `exact`, a number matches only an item of that value. +fn item_matches(item: &PivotItemRef, v: &Value, exact: bool) -> bool { + if let Value::Number(n) = v { + if let Value::Number(x) = &item.value + && n == x + { + return true; + } + if exact { + return false; + } + } + if let Value::Bool(b) = v { + return item.value == Value::Bool(*b); + } + v.to_text().is_ok_and(|t| same_text(&t, &item.label)) +} + +/// Whether `cell` is the value field asked for with exactly these items. +fn cell_matches(cell: &PivotDataCell, data_field: &str, pairs: &[(String, Value)], exact: bool) -> bool { + (same_text(&cell.data_field, data_field) || same_text(&cell.source_col, data_field)) + && cell.items.len() == pairs.len() + && pairs.iter().all(|(f, v)| cell.items.iter().any(|it| same_text(&it.field, f) && item_matches(it, v, exact))) +} + +fn is_axis_field(pt: &PivotTable, field: &str) -> bool { + pt.rows.iter().chain(pt.columns.iter()).any(|f| same_text(&f.source_col, field)) +} + +/// The value cell whose value field and items are exactly those asked for. Text items hit the +/// hash index; numbers and booleans also match by value, which a miss falls back to scanning for. +/// A number prefers the item of that value: two numeric items can show the same label (0.6 and 1 +/// formatted as `0`), so a label hit counts only when no item has the exact value. +fn find<'a>(cells: &'a PivotLookup, data_field: &str, pairs: &[(String, Value)]) -> Option<&'a PivotDataCell> { + let labels: Option> = + pairs.iter().map(|(f, v)| v.to_text().ok().map(|t| (f.as_str(), t, matches!(v, Value::Bool(_))))).collect(); + let hit = labels.as_ref().and_then(|labels| { + let refs: Vec<(&str, &str, bool)> = labels.iter().map(|(f, t, b)| (*f, t.as_str(), *b)).collect(); + cells.find(data_field, &refs) + }); + if pairs.iter().all(|(_, v)| matches!(v, Value::Text(_))) { + return hit; + } + if let Some(c) = hit.filter(|c| cell_matches(c, data_field, pairs, true)) { + return Some(c); + } + let all = cells.cells()?; + all.iter() + .find(|cell| cell_matches(cell, data_field, pairs, true)) + .or(hit) + .or_else(|| all.iter().find(|cell| cell_matches(cell, data_field, pairs, false))) +} + +impl Evaluator<'_> { + pub(crate) fn getpivotdata(&mut self, args: &[Expr]) -> Ev { + match self.pivot_value(args) { + Ok(v) => Ev::V(v), + Err(e) => Ev::V(Value::Error(e)), + } + } + + fn scalar_arg(&mut self, e: &Expr) -> Result { + match self.value(e).scalar() { + Value::Error(e) => Err(e), + v => Ok(v), + } + } + + fn pivot_value(&mut self, args: &[Expr]) -> Result { + let (Some(field_expr), Some(table_expr)) = (args.first(), args.get(1)) else { return Err(CellError::Value) }; + let rest = args.get(2..).unwrap_or_default(); + if rest.len() % 2 != 0 || rest.len() / 2 > MAX_PAIRS { + return Err(CellError::Ref); + } + let data_field = self.scalar_arg(field_expr)?.to_text()?; + let area = match self.eval(table_expr) { + Ev::R(areas) => areas.first().copied().ok_or(CellError::Ref)?, + Ev::V(Value::Error(e)) => return Err(e), + _ => return Err(CellError::Ref), + }; + let mut pairs: Vec<(String, Value)> = Vec::with_capacity(rest.len() / 2); + for pair in rest.chunks(2) { + let (Some(f), Some(i)) = (pair.first(), pair.get(1)) else { return Err(CellError::Ref) }; + let field = self.scalar_arg(f)?.to_text()?; + let item = self.scalar_arg(i)?; + if pairs.iter().any(|(g, _)| same_text(g, &field)) { + return Err(CellError::Ref); + } + pairs.push((field, item)); + } + let wb = self.host.workbook(); + let sheet = wb.sheet(area.sheet).ok_or(CellError::Ref)?; + let pt = sheet.pivots.iter().find(|p| p.last_range.is_some_and(|r| r.intersects(&area.range))).ok_or(CellError::Ref)?; + // A report filter field may be named when the report shows exactly that one item. + let mut axis_pairs = Vec::with_capacity(pairs.len()); + for (field, item) in pairs { + if is_axis_field(pt, &field) { + axis_pairs.push((field, item)); + continue; + } + let filter = pt.filters.iter().find(|f| same_text(&f.source_col, &field)).ok_or(CellError::Ref)?; + let shown = match filter.selected.as_deref() { + Some([one]) => item.to_text().is_ok_and(|t| same_text(&t, one)), + _ => false, + }; + if !shown { + return Err(CellError::Ref); + } + } + // The report's value cells. A pivot loaded without them builds them once; a stored report + // that no longer matches its source gets no index at all, so the lookup is #REF! until a + // refresh — never an index cached from a failed build, because the shared lookup keeps + // the first one set and undo or reopen could then never rebuild it. + if pt.data_index.cells().is_none() { + let build = PIVOT_INDEXER.get().ok_or(CellError::Ref)?; + if let Ok(cells) = build(wb, area.sheet, pt) { + pt.data_index.set_cells(cells); + } + } + let cell = find(&pt.data_index, &data_field, &axis_pairs).ok_or(CellError::Ref)?; + // The value is the one the sheet shows, so the formula and the report cell always agree. + let report = pt.last_range.ok_or(CellError::Ref)?; + let at = CellRef::new(report.start.row.saturating_add(cell.row), report.start.col.saturating_add(cell.col)); + if !report.contains(at) { + return Err(CellError::Ref); + } + Ok(sheet.value(at)) + } +} diff --git a/crates/engine/src/cmd/formulas.rs b/crates/engine/src/cmd/formulas.rs index 7c0904599..6afd14768 100644 --- a/crates/engine/src/cmd/formulas.rs +++ b/crates/engine/src/cmd/formulas.rs @@ -290,6 +290,12 @@ pub fn function_list() -> Vec { ("SHEETS", "Information", "SHEETS([reference])", "Number of sheets."), ("AREAS", "Lookup", "AREAS(reference)", "Number of areas in a reference."), ("TRIMRANGE", "Lookup", "TRIMRANGE(range, [trim_rows], [trim_cols])", "A range or array without its blank outer rows and columns."), + ( + "GETPIVOTDATA", + "Lookup", + "GETPIVOTDATA(data_field, pivot_table, [field1, item1], ...)", + "A value from a PivotTable report, by value field and items.", + ), ("ISOMITTED", "Information", "ISOMITTED(argument)", "TRUE when a LAMBDA argument was left out."), ]; for (n, c, sig, d) in specials { diff --git a/crates/engine/src/cmd/pivot/tests.rs b/crates/engine/src/cmd/pivot/tests.rs index c40ddf1be..03f2d7a5f 100644 --- a/crates/engine/src/cmd/pivot/tests.rs +++ b/crates/engine/src/cmd/pivot/tests.rs @@ -628,6 +628,312 @@ fn calculated_field_errors() { assert!(matches!(s.ui_requests.last(), Some(crate::UiRequest::Dialog(n, _)) if n == "pivotCalculatedField")); } +#[test] +fn getpivotdata_reads_the_report() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "columns": ["Product"], "values": ["Sales"]})).unwrap(); + assert_eq!(pt(&s).last_range.unwrap().a1(), "A3:E8"); + let gp = |s: &mut Session, cell: &str, f: &str| { + s.execute("cell.set", json!({"cell": cell, "input": f})).unwrap(); + val(s, cell) + }; + // Item cell, row subtotal by any cell of the report, grand total; names and items ignore case. + assert_eq!(gp(&mut s, "G1", r#"=GETPIVOTDATA("Sales",$A$3,"Region","East","Product","Apples")"#), Value::Number(160.0)); + assert_eq!(gp(&mut s, "G2", r#"=GETPIVOTDATA("sales",C6,"region","east")"#), Value::Number(415.0)); + assert_eq!(gp(&mut s, "G3", r#"=GETPIVOTDATA("Sum of Sales",A3)"#), Value::Number(TOTAL)); + assert_eq!(gp(&mut s, "G4", r#"=GETPIVOTDATA("Sales",A3,"Product","Pears")"#), Value::Number(270.0)); + // Not in the report: unknown item, value field, field, unpaired field, not a pivot cell. + for (cell, f) in [ + ("H1", r#"=GETPIVOTDATA("Sales",A3,"Region","South")"#), + ("H2", r#"=GETPIVOTDATA("Units",A3)"#), + ("H3", r#"=GETPIVOTDATA("Sales",A3,"Date","East")"#), + ("H4", r#"=GETPIVOTDATA("Sales",A3,"Region")"#), + ("H5", r#"=GETPIVOTDATA("Sales",Z99)"#), + ("H6", r#"=GETPIVOTDATA("Sales",A3,"Region","East","Region","East")"#), + ] { + assert_eq!(gp(&mut s, cell, f), Value::Error(CellError::Ref), "{f}"); + } + assert_eq!(gp(&mut s, "H7", r#"=GETPIVOTDATA("Sales",A3,"Region",NA())"#), Value::Error(CellError::NA)); + // A refresh after a source edit updates the result. + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "E2", "input": "1100"})).unwrap(); + s.execute("data.refreshAll", json!({})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(val(&s, "G1"), Value::Number(1160.0)); + // Opening a saved workbook starts without an index (files don't carry it); the first + // GETPIVOTDATA builds it. + let wb = (*s.doc().unwrap().wb).clone(); + let (back, _) = gridcraft_xlsx::read_xlsx(&gridcraft_xlsx::write_xlsx(&wb).unwrap()).unwrap(); + s.add_document(crate::DocState::new(back, None, "Reopened".into(), crate::Trust::FromFile)); + let d = s.doc().unwrap(); + let sh = d.wb.sheet(d.wb.sheet_index("Sheet2").unwrap()).unwrap(); + assert!(sh.pivots.first().unwrap().data_index.cells().is_some()); + assert_eq!(sh.value(CellRef::parse("G1").unwrap()), Value::Number(1160.0)); + assert_eq!(sh.value(CellRef::parse("G3").unwrap()), Value::Number(TOTAL + 1000.0)); +} + +#[test] +fn getpivotdata_reads_the_displayed_report() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + // The source changes after the last refresh, then the workbook is saved and reopened: + // the report still shows the old numbers, and the formula agrees with the cell. + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "E2", "input": "1100"})).unwrap(); + let wb = (*s.doc().unwrap().wb).clone(); + let (back, _) = gridcraft_xlsx::read_xlsx(&gridcraft_xlsx::write_xlsx(&wb).unwrap()).unwrap(); + s.add_document(crate::DocState::new(back, None, "Reopened".into(), crate::Trust::FromFile)); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + let r = pt(&s).last_range.unwrap(); + let at = r.start.a1(); + s.execute("cell.set", json!({"cell": "K1", "input": format!(r#"=GETPIVOTDATA("Sales",{at},"Region","East")"#)})).unwrap(); + let d = s.doc().unwrap(); + let sh = d.wb.sheet(d.wb.sheet_index("Sheet2").unwrap()).unwrap(); + let east = (r.start.row..=r.end.row).find(|row| sh.value(CellRef::new(*row, r.start.col)) == Value::Text("East".into())).unwrap(); + let shown = sh.value(CellRef::new(east, r.start.col + 1)); + assert_eq!(shown, Value::Number(415.0)); + assert_eq!(val(&s, "K1"), shown); +} + +/// `=GETPIVOTDATA("Sales", , "Region", )` evaluated in K1 of the active sheet. +fn region_sales(s: &mut Session, region: &str) -> Value { + let at = pt(s).last_range.unwrap().start.a1(); + s.execute("cell.set", json!({"cell": "K1", "input": format!(r#"=GETPIVOTDATA("Sales",{at},"Region","{region}")"#)})).unwrap(); + val(s, "K1") +} + +fn reopen(s: &mut Session) { + let wb = (*s.doc().unwrap().wb).clone(); + let (back, _) = gridcraft_xlsx::read_xlsx(&gridcraft_xlsx::write_xlsx(&wb).unwrap()).unwrap(); + s.add_document(crate::DocState::new(back, None, "Reopened".into(), crate::Trust::FromFile)); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); +} + +#[test] +fn getpivotdata_retries_after_a_lazy_build_fails() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + // A pivot read from a file carries no index, so the first lookup builds one. + reopen(&mut s); + assert!(pt(&s).data_index.cells().is_none()); + // The source changes, so the stored report is stale and the build finds nothing to trust. + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "Alpha"})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "West"), Value::Error(CellError::Ref)); + assert!(pt(&s).data_index.cells().is_none(), "the failed build caches nothing"); + // The lookup is shared, so caching a failure as an empty index would leave every state that + // shares it #REF! for good. Restoring the source lets the lookup build the index. + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "East"})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); + assert!(pt(&s).data_index.cells().is_some(), "the lookup rebuilt after the failure"); +} + +#[test] +fn getpivotdata_saves_no_index_when_the_lazy_build_fails() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + // A pivot read from a file carries no index; a source change makes the stored report stale, + // so the lazy build fails and the lookup is #REF!. + reopen(&mut s); + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "Alpha"})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "West"), Value::Error(CellError::Ref)); + // Saving the pivot must save no index at all, not an empty one: restoring an empty index + // would count as an index and lock the reopened pivot against the lazy rebuild. + let json = serde_json::to_string(&pt(&s)).unwrap(); + let back: gridcraft_model::PivotTable = serde_json::from_str(&json).unwrap(); + assert!(json.contains(r#""data_index":null"#), "a failed build saves no index, not an empty one: {json}"); + assert!(back.data_index.cells().is_none()); + // Reopened with the source restored, the lookup builds the index. + reopen(&mut s); + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "East"})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); +} + +#[test] +fn getpivotdata_keeps_the_refreshed_report_when_an_item_is_inserted() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); + // A new first item in the source shifts every item of a fresh report down one row. + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "Alpha"})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + // Before the refresh the report still shows the old items, and the lookup reads that report. + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); + assert_eq!(region_sales(&mut s, "Alpha"), Value::Error(CellError::Ref)); + // Saved and reopened unrefreshed: the stored report no longer matches the source, so there + // is no index to trust (never the shifted cell of a different item). + reopen(&mut s); + assert_eq!(region_sales(&mut s, "West"), Value::Error(CellError::Ref)); + assert_eq!(region_sales(&mut s, "Alpha"), Value::Error(CellError::Ref)); + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("data.refreshAll", json!({})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "Alpha"), Value::Number(100.0)); + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); + assert_eq!(region_sales(&mut s, "East"), Value::Number(315.0)); +} + +#[test] +fn getpivotdata_keeps_the_refreshed_report_when_an_item_is_removed() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + assert_eq!(region_sales(&mut s, "North"), Value::Number(135.0)); + // Every North row becomes East: North leaves the report and West moves up. + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + for a in ["A5", "A8", "A11"] { + s.execute("cell.set", json!({"cell": a, "input": "East"})).unwrap(); + } + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "North"), Value::Number(135.0)); + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); + reopen(&mut s); + assert_eq!(region_sales(&mut s, "North"), Value::Error(CellError::Ref)); + assert_eq!(region_sales(&mut s, "West"), Value::Error(CellError::Ref)); + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("data.refreshAll", json!({})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + assert_eq!(region_sales(&mut s, "North"), Value::Error(CellError::Ref)); + assert_eq!(region_sales(&mut s, "West"), Value::Number(285.0)); + assert_eq!(region_sales(&mut s, "East"), Value::Number(550.0)); +} + +#[test] +fn getpivotdata_index_is_saved_with_an_unrefreshed_pivot() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet1"})).unwrap(); + s.execute("cell.set", json!({"cell": "A2", "input": "Alpha"})).unwrap(); + s.execute("sheet.activate", json!({"sheet": "Sheet2"})).unwrap(); + // The pivot as saved by the native format: the index of the stored report comes back with + // it even though the source has since changed. + let back: gridcraft_model::PivotTable = serde_json::from_str(&serde_json::to_string(&pt(&s)).unwrap()).unwrap(); + let cells = back.data_index.cells().expect("index saved"); + assert_eq!(cells, pt(&s).data_index.cells().unwrap()); + let west = back.data_index.find("Sales", &[("Region", "West", false)]).unwrap(); + assert_eq!((west.row, west.col), (3, 1)); +} + +#[test] +fn getpivotdata_calculated_field() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Region"], "values": ["Sales"]})).unwrap(); + s.execute("pivot.calculatedField", json!({"name": "Double", "formula": "=Sales*2"})).unwrap(); + let at = pt(&s).last_range.unwrap().start.a1(); + s.execute("cell.set", json!({"cell": "K1", "input": format!(r#"=GETPIVOTDATA("Double",{at},"Region","East")"#)})).unwrap(); + assert_eq!(val(&s, "K1"), Value::Number(830.0)); +} + +#[test] +fn getpivotdata_report_filter() { + let mut s = sales(); + s.execute("insert.pivotTable", json!({"rows": ["Product"], "values": ["Sales"], "filters": ["Region"]})).unwrap(); + let at = pt(&s).last_range.unwrap().start.a1(); + let f = |region: &str| format!(r#"=GETPIVOTDATA("Sales",{at},"Region","{region}","Product","Apples")"#); + // With every item shown, naming the filter field is #REF!. + s.execute("cell.set", json!({"cell": "H1", "input": f("East")})).unwrap(); + assert_eq!(val(&s, "H1"), Value::Error(CellError::Ref)); + // Filtered to East: East matches, West doesn't. + s.execute("pivot.filter", json!({"field": "Region", "selected": ["East"]})).unwrap(); + assert_eq!(val(&s, "H1"), Value::Number(160.0)); + s.execute("cell.set", json!({"cell": "H2", "input": f("West")})).unwrap(); + assert_eq!(val(&s, "H2"), Value::Error(CellError::Ref)); +} + +/// A workbook whose Sheet1 holds `rows` under a two-column header, with a PivotTable of the first +/// column (rows) and the second (sum) on a new sheet; the report sheet is active. +fn two_column_pivot(header: [&str; 2], rows: Json, key_format: Option<&str>) -> Session { + let mut s = Session::new(); + s.new_workbook(); + let n = rows.as_array().map_or(0, Vec::len); + let mut all = vec![json!(header)]; + all.extend(rows.as_array().cloned().unwrap_or_default()); + s.execute("range.setValues", json!({"range": "A1", "values": all})).unwrap(); + if let Some(code) = key_format { + s.execute("home.numberFormat", json!({"range": format!("A2:A{}", n + 1), "code": code})).unwrap(); + } + s.execute("selection.set", json!({"range": "A1"})).unwrap(); + s.execute("insert.pivotTable", json!({"rows": [header[0]], "values": [header[1]]})).unwrap(); + s +} + +/// The value the report shows for the row item `label` (one value field: the report's last column). +fn shown(s: &Session, label: &str) -> Value { + let r = pt(s).last_range.unwrap(); + let row = find_row(s, label).unwrap_or_else(|| panic!("{label} is not in the report")); + val(s, &CellRef::new(row, r.end.col).a1()) +} + +/// Inserting or deleting rows or columns inside a report keeps GETPIVOTDATA on the requested +/// item's cell (the stored offsets move with the cells), and undo restores the old positions. +#[test] +fn getpivotdata_follows_rows_and_columns_edited_inside_the_report() { + let mut s = two_column_pivot(["Region", "Sales"], json!([["East", 10], ["North", 30], ["West", 20]]), None); + assert_eq!(pt(&s).last_range.unwrap().start.a1(), "A3"); + s.execute("cell.set", json!({"cell": "K1", "input": r#"=GETPIVOTDATA("Sales",A3,"Region","West")"#})).unwrap(); + s.execute("cell.set", json!({"cell": "K2", "input": r#"=GETPIVOTDATA("Sales",A3,"Region","North")"#})).unwrap(); + assert_eq!(val(&s, "K1"), Value::Number(20.0)); + // Row 4 splits the header from the items; column B splits the labels from the values (and + // moves the formulas from K to L). + for (cmd, lines, west, north) in + [("home.insertRows", json!({"rows": "4:4"}), "K1", "K2"), ("home.insertColumns", json!({"cols": "B:B"}), "L1", "L2")] + { + s.execute(cmd, lines).unwrap(); + assert_eq!(shown(&s, "West"), Value::Number(20.0), "{cmd}"); + assert_eq!(val(&s, west), shown(&s, "West"), "{cmd}"); + assert_eq!(val(&s, north), shown(&s, "North"), "{cmd}"); + } + s.execute("edit.undo", json!({})).unwrap(); + s.execute("edit.undo", json!({})).unwrap(); + assert_eq!(pt(&s).last_range.unwrap().start.a1(), "A3"); + s.execute("cell.set", json!({"cell": "K3", "input": r#"=GETPIVOTDATA("Sales",A3,"Region","West")"#})).unwrap(); + assert_eq!(val(&s, "K3"), Value::Number(20.0), "undo restores the report and its offsets"); + // Deleting North's row moves West up; North itself is gone from the report. + let north = find_row(&s, "North").unwrap() + 1; + s.execute("home.deleteRows", json!({"rows": format!("{north}:{north}")})).unwrap(); + s.execute("cell.set", json!({"cell": "K1", "input": r#"=GETPIVOTDATA("Sales",A3,"Region","West")"#})).unwrap(); + s.execute("cell.set", json!({"cell": "K2", "input": r#"=GETPIVOTDATA("Sales",A3,"Region","North")"#})).unwrap(); + assert_eq!(val(&s, "K1"), Value::Number(20.0)); + assert_eq!(val(&s, "K1"), shown(&s, "West")); + assert_eq!(val(&s, "K2"), Value::Error(CellError::Ref), "a deleted value cell is not read from its old place"); + // Lines inserted at the report's first line move the whole report, and lines right after it + // leave it alone; either way the offsets stay what they were. + for rows in ["3:3", "9:9"] { + s.execute("home.insertRows", json!({"rows": rows})).unwrap(); + let at = pt(&s).last_range.unwrap().start.a1(); + s.execute("cell.set", json!({"cell": "K1", "input": format!(r#"=GETPIVOTDATA("Sales",{at},"Region","West")"#)})).unwrap(); + assert_eq!(val(&s, "K1"), Value::Number(20.0), "insert {rows}"); + assert_eq!(val(&s, "K1"), shown(&s, "West"), "insert {rows}"); + } +} + +/// Numeric items whose formatted labels collide are told apart by value: an exact numeric item +/// wins over another item that only displays the same label. +#[test] +fn getpivotdata_matches_numeric_items_by_value_before_label() { + let mut s = two_column_pivot(["Key", "Sales"], json!([[0.6, 10], [1, 20]]), Some("0")); + let p = pt(&s); + let labels: Vec<&str> = p.data_index.cells().unwrap().iter().flat_map(|c| c.items.iter().map(|i| i.label.as_str())).collect(); + assert_eq!(labels, ["1", "1"], "both items display as 1"); + s.execute("cell.set", json!({"cell": "K1", "input": r#"=GETPIVOTDATA("Sales",A3,"Key",1)"#})).unwrap(); + s.execute("cell.set", json!({"cell": "K2", "input": r#"=GETPIVOTDATA("Sales",A3,"Key",0.6)"#})).unwrap(); + assert_eq!(val(&s, "K1"), Value::Number(20.0)); + assert_eq!(val(&s, "K2"), Value::Number(10.0)); + // Dates: two times on one day show the same date label; each serial finds its own item. + let mut s = two_column_pivot(["Day", "Sales"], json!([[45000, 10], [45000.25, 20]]), Some("m/d/yyyy")); + s.execute("cell.set", json!({"cell": "K1", "input": r#"=GETPIVOTDATA("Sales",A3,"Day",45000.25)"#})).unwrap(); + s.execute("cell.set", json!({"cell": "K2", "input": r#"=GETPIVOTDATA("Sales",A3,"Day",45000)"#})).unwrap(); + assert_eq!(val(&s, "K1"), Value::Number(20.0)); + assert_eq!(val(&s, "K2"), Value::Number(10.0)); +} + #[test] fn replace_protection() { let mut s = sales(); @@ -1062,6 +1368,34 @@ fn detail_cells_are_the_cells_show_details_takes() { assert!(crate::pivot::detail_cells(&d.wb, pt, d.wb.active_sheet()).unwrap_err().contains("out of date")); } +/// Measurement, not a check: `perf stat -e instructions:u` of this test with `PIVOT_BENCH_EDITS=0` +/// and `=20` (and `PIVOT_BENCH_REOPEN=0/1`) gives the cost of edits on the report's sheet and of +/// opening a saved workbook, with the setup subtracted. `PIVOT_BENCH_ROWS` is the source size. +#[test] +#[ignore = "measurement only"] +fn bench_edit_on_sheet_with_large_report() { + let env = |k: &str, d: usize| std::env::var(k).ok().and_then(|v| v.parse().ok()).unwrap_or(d); + let (n, edits) = (env("PIVOT_BENCH_ROWS", 100_000), env("PIVOT_BENCH_EDITS", 0)); + let mut s = Session::new(); + s.new_workbook(); + let mut rows = vec![json!(["Id", "Product", "Sales"])]; + rows.extend((0..n).map(|i| { + let (id, product) = (format!("id{i:07}"), ["Apples", "Pears", "Plums"][i % 3]); + json!([id, product, i]) + })); + s.execute("range.setValues", json!({"range": "A1", "values": rows})).unwrap(); + s.execute("selection.set", json!({"range": "A1"})).unwrap(); + s.execute("insert.pivotTable", json!({"rows": ["Id"], "columns": ["Product"], "values": ["Sales"]})).unwrap(); + for i in 0..edits { + s.execute("cell.set", json!({"cell": "K1", "input": i.to_string()})).unwrap(); + } + if env("PIVOT_BENCH_REOPEN", 0) == 1 { + let wb = (*s.doc().unwrap().wb).clone(); + let (back, _) = gridcraft_xlsx::read_xlsx(&gridcraft_xlsx::write_xlsx(&wb).unwrap()).unwrap(); + s.add_document(crate::DocState::new(back, None, "Reopened".into(), crate::Trust::FromFile)); + } +} + #[test] fn rank_ties_are_dense_and_count_numbers_is_captioned_count_of() { // #1448 items 10 and 12: Excel ranks 5, 4, 3, 3, 4 as 1, 2, 3, 3, 2 (GridCraft gave 1, 2, 4, diff --git a/crates/engine/src/cmd/sheet.rs b/crates/engine/src/cmd/sheet.rs index 95a0c93f5..7fda97983 100644 --- a/crates/engine/src/cmd/sheet.rs +++ b/crates/engine/src/cmd/sheet.rs @@ -4,7 +4,7 @@ use std::sync::Arc; use gridcraft_core::{CellRef, MAX_COLS, MAX_ROWS, RangeRef}; use gridcraft_formula::adjust::{Axis, Edit}; -use gridcraft_model::{Anchor, AnchorMode, Sheet, Visibility}; +use gridcraft_model::{Anchor, AnchorMode, PivotDataCell, PivotLookup, Sheet, Visibility}; use serde_json::{Value as Json, json}; use super::{ @@ -211,7 +211,31 @@ fn shift_sheet_features(sh: &mut Sheet, axis: Axis, at: u32, count: u32, insert: Axis::Rows => CellRef::new(at, pt.anchor.col), Axis::Cols => CellRef::new(pt.anchor.row, at), }); - pt.last_range = pt.last_range.and_then(map_range); + let old = pt.last_range; + pt.last_range = old.and_then(map_range); + // GETPIVOTDATA's value cells are stored as offsets from the report's top-left cell, so + // lines inserted or deleted inside the report move them as well. The moved cells go into + // a new index (the shared one stays with the undo snapshots); a value cell deleted with + // its line drops out, so it is #REF! rather than read from whatever took its place. + let index = pt.data_index.clone(); + if let (Some(o), Some(n), Some(cells)) = (old, pt.last_range, index.cells()) { + let (lo, hi) = if axis == Axis::Rows { (o.start.row, o.end.row) } else { (o.start.col, o.end.col) }; + let inside = at <= hi && if insert { at > lo } else { at.saturating_add(count) > lo }; + if inside { + let moved: Vec = cells + .iter() + .filter_map(|c| { + let to = map_cell(CellRef::new(o.start.row.checked_add(c.row)?, o.start.col.checked_add(c.col)?))?; + Some(PivotDataCell { row: to.row.checked_sub(n.start.row)?, col: to.col.checked_sub(n.start.col)?, ..c.clone() }) + }) + .collect(); + let lookup = PivotLookup::default(); + if !moved.is_empty() { + lookup.set_cells(moved); + } + pt.data_index = lookup; + } + } } } diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index e068f7a1e..99fb64efd 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -323,6 +323,7 @@ impl DocState { /// read from bytes (a file, a download, a recovery copy), [`Trust::Local`] only for a workbook /// made in this session. pub fn new(mut wb: Workbook, path: Option, title: String, trust: Trust) -> DocState { + let _ = gridcraft_model::PIVOT_INDEXER.set(pivot::index_cells); let mut calc = Calc::new(); calc.recalc_all(&mut wb); Self::with_calc(wb, calc, path, title, trust) diff --git a/crates/engine/src/pivot.rs b/crates/engine/src/pivot.rs index 1b5d48155..37a8b3527 100644 --- a/crates/engine/src/pivot.rs +++ b/crates/engine/src/pivot.rs @@ -12,8 +12,8 @@ use gridcraft_core::date::datetime_from_serial; pub use gridcraft_core::quote_sheet; use gridcraft_core::{CellError, CellRef, MAX_COLS, MAX_ROWS, RangeRef, Value, split_sheet}; use gridcraft_model::{ - BorderLine, BorderStyle, Cell, Chart, Color, Fill, HAlign, NumFmt, PivotDateGroup, PivotField, PivotFilter, PivotFunc, PivotLayout, PivotShowAs, - PivotSort, PivotTable, PivotValue, Series, Style, StyleId, Workbook, + BorderLine, BorderStyle, Cell, Chart, Color, Fill, HAlign, NumFmt, PivotDataCell, PivotDateGroup, PivotField, PivotFilter, PivotFunc, + PivotItemRef, PivotLayout, PivotLookup, PivotShowAs, PivotSort, PivotTable, PivotValue, Series, Style, StyleId, Workbook, }; use crate::pivot_calc::CalcSet; @@ -41,6 +41,9 @@ pub const MAX_OUTPUT_CELLS: u64 = 2_000_000; pub const MAX_AXIS_FIELDS: usize = 32; /// Most fields in the Values area. pub const MAX_VALUE_FIELDS: usize = 256; +/// Most report value cells a report may have (every one is indexed for GETPIVOTDATA; bigger +/// reports are rejected). +pub const MAX_INDEXED_CELLS: usize = 1_000_000; const MONTHS: [&str; 12] = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; @@ -314,9 +317,42 @@ pub fn refresh(wb: &mut Workbook, sheet: usize, idx: usize, replace: bool) -> Re } let sheet_name = shm.name.clone(); rebind_charts(wb, &sheet_name, &pt.name, origin, &out); + // The index of the new report; its lookup hash is built on the first GETPIVOTDATA. + if let Some(p) = wb.sheet_mut(sheet).and_then(|s| s.pivots.get_mut(idx)) { + let index = PivotLookup::default(); + index.set_cells(out.data); + p.data_index = index; + } Ok(area) } +/// The value cells of a PivotTable's report, for GETPIVOTDATA when a refresh did not set the +/// index (a workbook just opened from XLSX). The positions must describe the report as it is +/// stored on the sheet, not as the current source would render it: the source is recomputed only +/// to learn the layout, and the result is accepted when every label and header cell of that +/// layout equals the stored cell at the same place. A source changed since the last refresh gives +/// a different layout, so there is no trustworthy index and the lookup fails until a refresh. +pub fn index_cells(wb: &Workbook, sheet: usize, pt: &PivotTable) -> Result, String> { + const STALE: &str = "The stored report no longer matches the source. Refresh the PivotTable."; + let report = pt.last_range.ok_or("The PivotTable has no report.")?; + let out = compute(wb, pt, sheet)?; + let sh = wb.sheet(sheet).ok_or("There's no such sheet.")?; + let rows = u64::from(report.end.row) - u64::from(report.start.row) + 1; + let cols = u64::from(report.end.col) - u64::from(report.start.col) + 1; + if rows != u64::from(out.height().max(1)) || cols != u64::from(out.width.max(1)) { + return Err(STALE.into()); + } + let labels = [CellKind::Header, CellKind::RowLabel, CellKind::ColLabel, CellKind::Filter]; + for (r, row) in out.grid.iter().enumerate() { + for (c, oc) in row.iter().enumerate() { + if labels.contains(&oc.kind) && sh.value(CellRef::new(report.start.row + r as u32, report.start.col + c as u32)) != oc.value { + return Err(STALE.into()); + } + } + } + Ok(out.data) +} + #[cfg(test)] mod fit_tests { use super::data_cols_exceed_sheet; diff --git a/crates/engine/src/pivot/charts.rs b/crates/engine/src/pivot/charts.rs index 97fea4f42..554f83d9d 100644 --- a/crates/engine/src/pivot/charts.rs +++ b/crates/engine/src/pivot/charts.rs @@ -1,5 +1,5 @@ use super::output::Output; -use super::{CellRef, Chart, Series, Workbook, quote_sheet}; +use super::{CellRef, Chart, PivotLookup, Series, Workbook, quote_sheet}; // ---------------------------------------------------------------- PivotCharts @@ -109,6 +109,7 @@ pub fn clear_output(wb: &mut Workbook, sheet: usize, idx: usize) { shm.take_cells(r); if let Some(p) = shm.pivots.get_mut(idx) { p.last_range = None; + p.data_index = PivotLookup::default(); } } @@ -126,6 +127,7 @@ mod tests { chart_rows: vec![1, 2, 3], chart_cols: (1..1000).map(|c| (c, format!("S{c}"))).collect(), label_col: Some(0), + data: vec![], }; assert_eq!(chart_series("Sheet1", CellRef::new(0, 0), &out).len(), MAX_CHART_SERIES); } diff --git a/crates/engine/src/pivot/compute.rs b/crates/engine/src/pivot/compute.rs index 2bb6bdbc6..2f282c167 100644 --- a/crates/engine/src/pivot/compute.rs +++ b/crates/engine/src/pivot/compute.rs @@ -3,8 +3,8 @@ use super::output::{ AggMap, CellKind, ColKind, ColSpec, Ctx, Grid, Line, LineKind, Output, RowKind, ValSrc, caption, data_cols_exceed_sheet, flatten_cols, }; use super::{ - CalcSet, CellError, CellRef, HashMap, HashSet, MAX_AXIS_FIELDS, MAX_COLS, MAX_VALUE_FIELDS, PivotDateGroup, PivotField, PivotFilter, PivotFunc, - PivotLayout, PivotShowAs, PivotTable, PivotValue, Source, Value, Workbook, read_source, + CalcSet, CellError, CellRef, HashMap, HashSet, MAX_AXIS_FIELDS, MAX_COLS, MAX_INDEXED_CELLS, MAX_VALUE_FIELDS, PivotDataCell, PivotDateGroup, + PivotField, PivotFilter, PivotFunc, PivotItemRef, PivotLayout, PivotShowAs, PivotTable, PivotValue, Source, Value, Workbook, read_source, }; /// Why Show Details turns a cell down when it isn't a value cell of a PivotTable (the front end @@ -653,6 +653,7 @@ pub(super) fn compute_from(wb: &Workbook, pt: &PivotTable, src: &Source, mut pro chart_rows: vec![], chart_cols: vec![], label_col: None, + data: vec![], }); } @@ -728,6 +729,16 @@ pub(super) fn compute_from(wb: &Workbook, pt: &PivotTable, src: &Source, mut pro } else { Some(row_levels.saturating_sub(1) as u32) }; + let item_refs = |fields: &[&PivotField], items: &[FieldItems], path: &[u32]| -> Vec { + path.iter() + .enumerate() + .filter_map(|(d, it)| { + let (f, item) = (fields.get(d)?, items.get(d)?.get(*it)?); + Some(PivotItemRef { field: f.source_col.clone(), label: item.label.clone(), value: item.value.clone() }) + }) + .collect() + }; + let mut data: Vec = Vec::new(); for line in &lines { let rk = match line.kind { LineKind::Item(b) => RowKind::Item(b), @@ -758,6 +769,21 @@ pub(super) fn compute_from(wb: &Workbook, pt: &PivotTable, src: &Source, mut pro let Some(raw) = lookup.get(rp, cp, *vi) else { continue }; let context = ShowContext { row_path: rp, col_path: cp, value_index: *vi, level0: &level0, siblings: &siblings, cache: &show_cache }; let shown = show_value(vdef, raw, &lookup, &context); + // Every value cell of the report is indexed, so a report that needs more cells than + // the index cap is rejected outright — never written with a truncated index that + // would answer #REF! for the values the sheet shows. + if data.len() >= MAX_INDEXED_CELLS { + return Err(format!("A PivotTable report can have at most {MAX_INDEXED_CELLS} value cells.")); + } + let mut items = item_refs(&ctx.rows, &ctx.row_items, rp); + items.extend(item_refs(&col_fields, &col_items, cp)); + data.push(PivotDataCell { + data_field: vdef.name.clone(), + source_col: vdef.source_col.clone(), + items, + row: u32::try_from(r).unwrap_or(u32::MAX), + col: u32::try_from(l + j).unwrap_or(u32::MAX), + }); if let Some(pr) = probe.as_deref_mut() { if pr.at == Some((r, l + j)) { pr.hit = Some((rp.to_vec(), cp.to_vec())); @@ -769,5 +795,5 @@ pub(super) fn compute_from(wb: &Workbook, pt: &PivotTable, src: &Source, mut pro g.set(r, l + j, shown, ck.1, 0, data_fmt.get(*vi).cloned().flatten()); } } - Ok(Output { width: g.width as u32, grid: g.rows, row_kinds: g.kinds, body_offset, chart_rows, chart_cols, label_col }) + Ok(Output { width: g.width as u32, grid: g.rows, row_kinds: g.kinds, body_offset, chart_rows, chart_cols, label_col, data }) } diff --git a/crates/engine/src/pivot/output.rs b/crates/engine/src/pivot/output.rs index ca680335b..2d57fc21a 100644 --- a/crates/engine/src/pivot/output.rs +++ b/crates/engine/src/pivot/output.rs @@ -1,7 +1,7 @@ use super::items::{Acc, FieldItems, Node, is_collapsed}; use super::{ - BorderLine, BorderStyle, Color, Fill, HAlign, HashMap, MAX_COLS, MAX_OUTPUT_CELLS, MAX_ROWS, NumFmt, PivotField, PivotLayout, PivotTable, Style, - Value, + BorderLine, BorderStyle, Color, Fill, HAlign, HashMap, MAX_COLS, MAX_OUTPUT_CELLS, MAX_ROWS, NumFmt, PivotDataCell, PivotField, PivotLayout, + PivotTable, Style, Value, }; // ---------------------------------------------------------------- output @@ -61,6 +61,8 @@ pub struct Output { pub chart_cols: Vec<(u32, String)>, /// Grid column holding the innermost row labels (`None` without row fields). pub label_col: Option, + /// Every value cell with the items that locate it (for GETPIVOTDATA). + pub data: Vec, } impl Output { diff --git a/crates/model/src/features.rs b/crates/model/src/features.rs index 5c9f961d7..099bd440a 100644 --- a/crates/model/src/features.rs +++ b/crates/model/src/features.rs @@ -9,7 +9,10 @@ use crate::style::{Color, Style}; mod pivot; mod print; -pub use pivot::{PivotCalcField, PivotDateGroup, PivotField, PivotFilter, PivotFunc, PivotLayout, PivotShowAs, PivotSort, PivotTable, PivotValue}; +pub use pivot::{ + PIVOT_INDEXER, PivotCalcField, PivotDataCell, PivotDateGroup, PivotField, PivotFilter, PivotFunc, PivotIndexer, PivotItemRef, PivotLayout, + PivotLookup, PivotShowAs, PivotSort, PivotTable, PivotValue, pivot_key, +}; pub use print::{EditRange, HfPage, LineInfo, Orientation, PageOrder, PrintSettings, SheetProtection, password_hash}; // ---------------------------------------------------------------- tables diff --git a/crates/model/src/features/pivot.rs b/crates/model/src/features/pivot.rs index b34015f4c..9a630ef44 100644 --- a/crates/model/src/features/pivot.rs +++ b/crates/model/src/features/pivot.rs @@ -1,3 +1,5 @@ +use gridcraft_core::Value; + use super::{CellRef, RangeRef}; use serde::{Deserialize, Serialize}; @@ -131,6 +133,11 @@ pub struct PivotTable { pub last_range: Option, /// Calculated fields, usable in the Values area by name. pub calculated_fields: Vec, + /// Where GETPIVOTDATA finds the report's value cells, as of the last refresh. Shared between + /// workbook clones (so editing the pivot's sheet doesn't copy it), set on refresh and saved + /// with the report, so a reopened unrefreshed pivot still reads the report it stored. Only a + /// pivot loaded without one (XLSX) builds it on first use, checked against the stored report. + pub data_index: PivotLookup, /// The Σ Values pseudo-field (it exists with two or more value fields) sits in Rows rather /// than Columns (Excel's "values on rows"). pub values_on_rows: bool, @@ -149,6 +156,143 @@ pub struct PivotCalcField { pub formula: String, } +/// One value cell of a computed PivotTable report: the value field, the row and column items +/// that locate it (fewer for subtotals, none for the grand total) and where the report shows it. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct PivotDataCell { + /// The value field's caption ("Sum of Sales"). + pub data_field: String, + /// The value field's source column ("Sales"). + pub source_col: String, + pub items: Vec, + /// Row and column of the cell, counted from the report's top-left cell. + pub row: u32, + pub col: u32, +} + +/// Builds a PivotTable's value cells from its source (`sheet` is the pivot's sheet). The engine +/// sets it; the model can't compute reports. +pub type PivotIndexer = fn(&crate::Workbook, usize, &PivotTable) -> Result, String>; +pub static PIVOT_INDEXER: std::sync::OnceLock = std::sync::OnceLock::new(); + +fn norm(s: &str) -> String { + s.trim().to_lowercase() +} + +/// One part of a lookup key: length-prefixed, so no label text can imitate a separator. +fn key_part(out: &mut String, s: &str) { + out.push_str(&s.len().to_string()); + out.push(':'); + out.push_str(s); +} + +/// The lookup key of a value field with field/item labels (names ignore case and outer blanks). +/// Each item is `(field, label, is_bool)`: a Boolean item and a text item with the same label +/// are different items. +pub fn pivot_key(data_field: &str, items: &[(&str, &str, bool)]) -> String { + let mut parts: Vec = items + .iter() + .map(|(f, l, b)| { + let mut p = String::new(); + key_part(&mut p, &norm(f)); + key_part(&mut p, &norm(l)); + p.push(if *b { 'b' } else { 't' }); + p + }) + .collect(); + parts.sort_unstable(); + let mut key = String::new(); + key_part(&mut key, &norm(data_field)); + key.push_str(&parts.len().to_string()); + key.push('|'); + for p in parts { + key.push_str(&p); + } + key +} + +#[derive(Default)] +struct LookupInner { + cells: std::sync::OnceLock>, + keys: std::sync::OnceLock>, +} + +/// A PivotTable's GETPIVOTDATA index. Clones share it, so `Workbook` clones per command stay +/// cheap however large the report is. The cells are set on refresh or on first use; the hash of +/// them is built on the first lookup. +#[derive(Clone, Default)] +pub struct PivotLookup(std::sync::Arc); + +impl PivotLookup { + /// The value cells, if they were set. + pub fn cells(&self) -> Option<&[PivotDataCell]> { + self.0.cells.get().map(Vec::as_slice) + } + + /// Sets the value cells (the first set wins) and returns them. + pub fn set_cells(&self, cells: Vec) -> &[PivotDataCell] { + self.0.cells.get_or_init(|| cells) + } + + /// The cell with this value field and exactly these items, by hash. + pub fn find(&self, data_field: &str, items: &[(&str, &str, bool)]) -> Option<&PivotDataCell> { + let cells = self.0.cells.get()?; + let keys = self.0.keys.get_or_init(|| { + let mut m = std::collections::HashMap::with_capacity(cells.len()); + for (i, c) in cells.iter().enumerate() { + let items: Vec<(&str, &str, bool)> = + c.items.iter().map(|it| (it.field.as_str(), it.label.as_str(), matches!(it.value, Value::Bool(_)))).collect(); + m.entry(pivot_key(&c.data_field, &items)).or_insert(i); + m.entry(pivot_key(&c.source_col, &items)).or_insert(i); + } + m + }); + keys.get(&pivot_key(data_field, items)).and_then(|i| cells.get(*i)) + } +} + +impl std::fmt::Debug for PivotLookup { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "PivotLookup({} cells)", self.cells().map_or(0, <[PivotDataCell]>::len)) + } +} + +/// Saved as the list of value cells (or nothing before the first refresh); the hash is rebuilt. +/// An empty list is never saved: it is no index, and restoring one would lock the shared lookup +/// against the lazy rebuild that a reopened workbook needs. +impl Serialize for PivotLookup { + fn serialize(&self, s: S) -> Result { + self.cells().filter(|cells| !cells.is_empty()).serialize(s) + } +} + +impl<'de> Deserialize<'de> for PivotLookup { + fn deserialize>(d: D) -> Result { + let l = PivotLookup::default(); + if let Some(cells) = Option::>::deserialize(d)? + && !cells.is_empty() + { + l.set_cells(cells); + } + Ok(l) + } +} + +/// The index is a cache, not part of a PivotTable's identity. +impl PartialEq for PivotLookup { + fn eq(&self, _: &Self) -> bool { + true + } +} + +/// A row or column field item: the field's source column, the item's label and its value. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct PivotItemRef { + pub field: String, + pub label: String, + pub value: Value, +} + impl Default for PivotTable { fn default() -> Self { PivotTable { @@ -170,8 +314,65 @@ impl Default for PivotTable { show_headers: true, last_range: None, calculated_fields: vec![], + data_index: PivotLookup::default(), values_on_rows: false, values_position: None, } } } + +#[cfg(test)] +mod pivot_lookup_tests { + use super::*; + + fn cell(field: &str, label: &str, value: Value, row: u32) -> PivotDataCell { + PivotDataCell { + data_field: "Sum of Sales".into(), + source_col: "Sales".into(), + items: vec![PivotItemRef { field: field.into(), label: label.into(), value }], + row, + col: 1, + } + } + + #[test] + fn labels_with_separator_characters_do_not_collide() { + let l = PivotLookup::default(); + // Two different (field, label) pairs that a separator-joined key would make identical. + l.set_cells(vec![cell("A\u{2}B", "C", Value::Text("C".into()), 1), cell("A", "B\u{2}C", Value::Text("B\u{2}C".into()), 2)]); + assert_eq!(l.find("Sales", &[("A\u{2}B", "C", false)]).unwrap().row, 1); + assert_eq!(l.find("Sales", &[("A", "B\u{2}C", false)]).unwrap().row, 2); + // A label carrying the old pair separator cannot stand for two items. + assert!(l.find("Sales", &[("A", "B", false), ("C", "D", false)]).is_none()); + } + + #[test] + fn boolean_and_text_items_with_one_label_are_different_items() { + let l = PivotLookup::default(); + l.set_cells(vec![cell("Flag", "TRUE", Value::Text("TRUE".into()), 1), cell("Flag", "TRUE", Value::Bool(true), 2)]); + assert_eq!(l.find("Sales", &[("Flag", "TRUE", true)]).unwrap().row, 2); + assert_eq!(l.find("Sales", &[("Flag", "TRUE", false)]).unwrap().row, 1); + } + + #[test] + fn index_round_trips_through_serde() { + let l = PivotLookup::default(); + l.set_cells(vec![cell("Region", "East", Value::Text("East".into()), 1)]); + let back: PivotLookup = serde_json::from_str(&serde_json::to_string(&l).unwrap()).unwrap(); + assert_eq!(back.find("Sales", &[("Region", "East", false)]).unwrap().row, 1); + let none: PivotLookup = serde_json::from_str(&serde_json::to_string(&PivotLookup::default()).unwrap()).unwrap(); + assert!(none.cells().is_none()); + } + + #[test] + fn an_empty_lookup_is_saved_as_no_index() { + // An empty cell list is no index, so it must not be saved as an empty one: restoring an + // empty index counts as an initialized lookup, which blocks the lazy rebuild. + let l = PivotLookup::default(); + l.set_cells(vec![]); + let saved = serde_json::to_string(&l).unwrap(); + assert_eq!(saved, "null", "an empty lookup saves no index: {saved}"); + let back: PivotLookup = serde_json::from_str("[]").unwrap(); + assert!(back.cells().is_none(), "an empty saved list restores no index"); + } +} diff --git a/crates/xlsx/src/tests/pivot_tests.rs b/crates/xlsx/src/tests/pivot_tests.rs index d0a8e0bfc..155e27940 100644 --- a/crates/xlsx/src/tests/pivot_tests.rs +++ b/crates/xlsx/src/tests/pivot_tests.rs @@ -116,6 +116,7 @@ fn workbook() -> Workbook { show_headers: false, last_range: Some(rr("A2:H12")), calculated_fields: vec![], + data_index: Default::default(), values_on_rows: false, values_position: None, }; @@ -176,6 +177,7 @@ fn workbook() -> Workbook { show_headers: true, last_range: Some(rr("K2:Z30")), calculated_fields: vec![], + data_index: Default::default(), values_on_rows: false, values_position: None, }; diff --git a/docs/function-parity.md b/docs/function-parity.md index 272056909..54866d54e 100644 --- a/docs/function-parity.md +++ b/docs/function-parity.md @@ -12,13 +12,9 @@ Excel's worksheet functions, category by category, against GridCraft's function them), plus 14 text functions the page lists under combined entries (FIND/FINDB, LEFT/LEFTB, LEN/LENB, MID/MIDB, REPLACE/REPLACEB, RIGHT/RIGHTB, SEARCH/SEARCHB), BETA.INV, and COPILOT and PY (Microsoft 365 functions not on that page when fetched). -- **Ours (505):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the -- **Ours (503):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the -- **Ours (502):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the -- **Ours (502):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the -- **Ours (505):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the +- **Ours (514):** every function spec in `crates/functions/src/*.rs` (`f!("NAME", …)`) plus the evaluator's special forms in `crates/calc/src/eval.rs` (`SPECIAL_FUNCTIONS`: IF, IFS, CHOOSE, - OFFSET, INDIRECT, INDEX, SUBTOTAL, AGGREGATE, TEXT, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL, + OFFSET, INDIRECT, INDEX, GETPIVOTDATA, SUBTOTAL, AGGREGATE, TEXT, LET, LAMBDA, MAP, REDUCE, SCAN, BYROW, BYCOL, MAKEARRAY…), excluding the test-only `ADD2`. Every one is an Excel name. - **Not on Microsoft's list:** USDOLLAR and ECMA.CEILING, legacy functions Excel still evaluates (like DOLLAR), are in our library as Compatibility functions for files that use them. They are not counted above. @@ -29,7 +25,7 @@ Excel's worksheet functions, category by category, against GridCraft's function ## Coverage -**Measured: 513 / 523 by name (98.1%); 504 / 523 working (96.4%).** +**Measured: 514 / 523 by name (98.3%); 505 / 523 working (96.6%).** | Category | Excel | Ours | Working | % working | Missing | |---|---|---|---|---|---| @@ -38,7 +34,7 @@ Excel's worksheet functions, category by category, against GridCraft's function | Financial | 55 | 55 | 55 | 100% | — | | Engineering | 54 | 54 | 54 | 100% | — | | Text | 50 | 48 | 48 | 96% | DETECTLANGUAGE, TRANSLATE | -| Lookup and reference | 40 | 36 | 36 | 90% | GETPIVOTDATA, IMAGE, RTD, TRIMRANGE | +| Lookup and reference | 40 | 38 | 38 | 95% | IMAGE, RTD | | Compatibility | 40 | 40 | 40 | 100% | — | | Date and time | 25 | 25 | 25 | 100% | — | | Information | 22 | 21 | 21 | 95% | STOCKHISTORY | @@ -48,10 +44,10 @@ Excel's worksheet functions, category by category, against GridCraft's function | Web | 3 | 3 | 1 | 33% | (WEBSERVICE, FILTERXML are stubs; ENCODEURL works) | | Add-in and Automation | 3 | 1 | 1 | 33% | CALL, REGISTER.ID | | AI and Python | 2 | 0 | 0 | 0% | COPILOT, PY | -| **Total** | **523** | **505** | **496** | **94.8%** | 18 missing, 9 stubs | +| **Total** | **523** | **514** | **505** | **96.6%** | 9 missing, 9 stubs | -By the Excel version that introduced them (working / total): before 2010 346 / 360, 2010 -56 / 56, 2013 49 / 51, 2016 1 / 5, 2019 6 / 6, 2021 11 / 11, 2024 22 / 23, Microsoft 365 5 / 11. +By the Excel version that introduced them (working / total): before 2010 349 / 360, 2010 +56 / 56, 2013 49 / 51, 2016 5 / 5, 2019 6 / 6, 2021 11 / 11, 2024 22 / 23, Microsoft 365 7 / 11. Dynamic-array and LAMBDA-era functions (FILTER, SORT, UNIQUE, XLOOKUP, VSTACK, TAKE, GROUPBY, PIVOTBY, REGEXTEST…) are all present. @@ -66,7 +62,7 @@ This improves series preparation; it does not change the model-fitting or season |---|---|---|---| | FORECAST.ETS family (4) | yes | 4–6 h | AAA exponential smoothing with seasonality detection; needed for Forecast Sheet too | | ODDFPRICE, ODDFYIELD, ODDLPRICE, ODDLYIELD | yes | 2–4 h | Odd first/last period bond maths | -| GETPIVOTDATA | yes | 2–3 h | Needs the pivot engine's result cache; also the "Generate GetPivotData" option | +| Generate GetPivotData | yes | 1–2 h | GETPIVOTDATA works; clicking a PivotTable value while typing a formula doesn't insert it yet | | IMAGE | yes | 3–5 h | Pictures in cells (see PR #105); fetching URLs needs a privacy decision | | CUBE* (7), WEBSERVICE, FILTERXML | partly | 4–8 h | FILTERXML can work offline; CUBE needs an OLAP/data-model source | | TRANSLATE, DETECTLANGUAGE, STOCKHISTORY, COPILOT, PY | no (cloud) | — | Microsoft cloud services; open equivalents only by owner decision | @@ -128,7 +124,7 @@ inputs, so large finite middle values do not become `#NUM!` merely because their Sorting, argument coercion and error propagation are unchanged; this is a numerical correction under the documented median rule, not an extreme-value comparison against desktop Excel. -Work to close (16–26 h in total): the in-scope missing functions (10–16 h with the table above), +Work to close (15–25 h in total): the in-scope missing functions (10–16 h with the table above), an oracle corpus of Excel-computed results per function (owner runs Excel on generated workbooks; results checked in as data, not Excel files) and fixing what it finds (6–10 h). @@ -183,6 +179,7 @@ Positive fractional inputs still truncate, and signed zero remains valid. Covere | 2026-10-14 | minor | Bounded references up to 1,048,576 cells (`D3:AZ400`) keep their full shape instead of stopping at the used range, so `SUMPRODUCT(D3:AZ400*B3:B400,(A3:A400=x)*(D1:AZ1=y))` broadcasts like Excel and `ROWS`, `COUNTBLANK` and `--(r="")` count the blanks; references that run to the edge of the sheet (`A:A`) still stop at the used range (#1010) | | 2026-10-13 | minor | Legacy formulas from files (no dynamic-array metadata) evaluate with implicit intersection: cells differing from Excel's cached values 68,964 → 49,866 across 4,224 test-corpus workbooks (none gets more) | | 2026-10-13 | minor | Clear reports changed stored cells sparsely, including those beyond the first million coordinates; dependent formulas and removed spills update, while clearing empty merges still unblocks spills (#1032) | +| 2026-10-12 | minor | GETPIVOTDATA added: 514 / 523 by name, 505 working | | 2026-10-12 | minor | USDOLLAR added as a Compatibility function (not on Microsoft's list, so the counts don't change); MIRR with no positive flow is -1, COUNTIF's bare `"<="` / `">="` skip empty strings, INDIRECT with `a1` = FALSE gives `#REF!` for A1-style text (#738) | | 2026-10-11 | minor | TEXT / number formats fixed against Excel-cached results: fraction literals and zero fractions, `0` denominators zero padded, `?` pads the thousands separator, booleans use the text section (226 mismatches → 0 in 352 TEXT cells) | | 2026-10-11 | minor | Iterative calculation: circular references iterate up to Maximum Iterations / Maximum Change when enabled | diff --git a/docs/gaps.md b/docs/gaps.md index 3b21efc69..8a22aac9e 100644 --- a/docs/gaps.md +++ b/docs/gaps.md @@ -91,8 +91,8 @@ No alpha blockers: every core workflow passes the [alpha gate](roadmap.md#alpha- ### 8. PivotTables: no slicers, calculated items, PivotCharts; unverified in Excel (F) - **Evidence:** `parity-checklist.md` missing `insert.slicer`, `insert.timeline`, - `insert.pivotChart`, `table.insertSlicer`; calculated items and GETPIVOTDATA missing (calculated fields - and values on rows done). Show Details (double-click a value, subtotal or grand total cell) lists + `insert.pivotChart`, `table.insertSlicer`; calculated items and Generate GetPivotData missing (calculated fields, + values on rows and GETPIVOTDATA done). Show Details (double-click a value, subtotal or grand total cell) lists its source records on a new sheet. The "Enable show details" option (`pivot.options` `enableShowDetails`, saved as `enableDrill`) turns it off; it has no PivotTable Options dialog yet (command only). @@ -142,10 +142,7 @@ No alpha blockers: every core workflow passes the [alpha gate](roadmap.md#alpha- Margins are dragged on the rulers (#792). Remaining: pictures in headers/footers, Different First Page / Odd & Even pages. Desktop now submits directly through craft-print with printer choice, copies, duplex and grayscale; browser printing stays available. Print ranges and driver tray/options UI remain. **Estimate:** 5–10 h. ### 15. Remaining functions and an oracle suite (F) -- 18 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 16–26 h. -- 20 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 17–29 h. -- 21 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 18–30 h. -- 18 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 18–30 h. +- 9 missing, 9 stubs; no Excel-computed oracle corpus; #137, #138. **Estimate:** 15–25 h. [function-parity.md](function-parity.md) ### 16. Collaboration and review (F) @@ -212,6 +209,7 @@ No alpha blockers: every core workflow passes the [alpha gate](roadmap.md#alpha- | 2026-10-15 | minor | §13: Solver prepares reports before committing the solution, model and report sheets together; report refusal preserves the workbook and Undo/Redo (#1050) | | 2026-10-13 | trivial | §4: multi-level category axes landed | | 2026-10-12 | minor | §9: next/previous sheet commands retain the current visible sheet and selection when there is no visible destination | +| 2026-10-12 | minor | §8, §15: GETPIVOTDATA added (9 functions missing) | | 2026-10-12 | minor | rows auto-fit oversized fonts on open and whole-row/column formatting (#833) | | 2026-10-12 | minor | Page Layout view: dragging margins on the rulers (#792) removed from the §14 remainder | | 2026-10-12 | minor | §13: Forecast Sheet Forecast Start (hindcast) landed (#785) | diff --git a/docs/target-app-parity.md b/docs/target-app-parity.md index a2243bc92..557c29df9 100644 --- a/docs/target-app-parity.md +++ b/docs/target-app-parity.md @@ -14,8 +14,8 @@ generated ribbon checklist is [`parity-checklist.md`](parity-checklist.md). |---|---|---| | Feature breadth (weighted by use) | **~75%** | estimated from the area table below; anchored on two measurements | | Ribbon and menu catalog | **281 / 296 (94.9%)** | measured: `cargo xtask parity` (`crates/engine/src/catalog.rs` ∩ registered command ids) | -| Worksheet functions, by name | **513 / 523 (98.1%)** | measured: registered functions vs Microsoft's published function list (method below) | -| Worksheet functions that do real work | **504 / 523 (96.4%)** | measured: excludes 9 stubs (7 CUBE functions, WEBSERVICE, FILTERXML) | +| Worksheet functions, by name | **514 / 523 (98.3%)** | measured: registered functions vs Microsoft's published function list (method below) | +| Worksheet functions that do real work | **505 / 523 (96.6%)** | measured: excludes 9 stubs (7 CUBE functions, WEBSERVICE, FILTERXML) | | **Ready for real work** | **~50%** | estimated; written weights below (51.6 computed) | | **Mainstream practitioner** | **~50%** | estimated; [method](#mainstream-practitioner) (69.5 depth × 0.90 × 0.92 × 0.88 = 50.6) | | **Essentials user** | **~63%** | estimated; [method](#essentials-user) (80.2 depth × 0.92 × 0.95 × 0.90 = 63.1) | @@ -81,11 +81,11 @@ performance 8, stability 8, localization 4, hardware 2, platforms 2, ecosystem 2 | Area | Weight | Breadth | Ready | Hours | Evidence | |---|---|---|---|---|---| | Formula language and calculation | 18 | ~92% | ~72% | 20–30 | A1/R1C1 parser, reference adjustment, dependency-graph recalc, dynamic arrays and `#SPILL!`, `A1#`, LET/LAMBDA and helpers, structured references, names, 3-D references. Open correctness bugs: 15-digit equality (#57), stale spills after structural edits and cleared blockers (#133, #139), 3-D span endpoint deletion (#140), sheet rename and qualified names (#134). Single-threaded | -| Worksheet functions | 12 | 98.1% (measured) | ~82% | 16–26 | 513 / 523 names, 504 working. Missing: GETPIVOTDATA, IMAGE plus cloud functions. No conformance suite against Excel-computed oracles; edge-case bugs keep surfacing (#137, #138). [function-parity.md](function-parity.md) | +| Worksheet functions | 12 | 98.3% (measured) | ~82% | 15–25 | 514 / 523 names, 505 working. Missing: IMAGE plus cloud functions. No conformance suite against Excel-computed oracles; edge-case bugs keep surfacing (#137, #138). [function-parity.md](function-parity.md) | | Editing, navigation, clipboard, fill | 12 | ~90% | ~70% | 15–25 | Point mode, F4, AutoComplete, Paste Special, AutoFill, Flash Fill, Find/Replace, Go To Special. Bugs: rectangular drag-select (#28), monthly fill (#132), cut across sheets (#128), Insert Copied Cells missing (#21), key tips only on part of the Home tab (#43; #33, #49). [ui-parity.md](ui-parity.md) | | Cell formatting, styles, conditional formatting | 10 | ~95% | ~80% | 8–14 | Full number-format language, borders, 47 cell styles, 60 table styles, all CF rule kinds including data bars, colour scales and icon sets. Gaps: font weights beyond regular/bold (#168), system fonts list (#4), themes | | Sort, filter, tables, validation, outline | 8 | ~82% | ~70% | 10–16 | Multi-level sort, AutoFilter, Advanced Filter, tables with totals, validation, outline, subtotals, consolidate. Advanced Filter copy-to preserves cells outside its result and rejects worksheet overflow (#1031). Bugs: subtotal header (#131), validation edits (#129), table style panic caught (#130); no slicers on tables | -| PivotTables | 7 | ~65% | ~45% | 25–40 | Insert/Recommended, field list, three layouts, values on rows or columns, date grouping, show-values-as, refresh, calculated fields, XLSX round-trip. Missing: slicers, timelines, calculated items, PivotCharts, value/label filters, number grouping, GETPIVOTDATA, data-model pivots. Round-trip never checked in Excel | +| PivotTables | 7 | ~65% | ~45% | 25–40 | Insert/Recommended, field list, three layouts, values on rows or columns, date grouping, show-values-as, refresh, calculated fields, GETPIVOTDATA, XLSX round-trip. Missing: slicers, timelines, calculated items, PivotCharts, value/label filters, number grouping, Generate GetPivotData, data-model pivots. Round-trip never checked in Excel | | Charts and sparklines | 8 | ~50% | ~30% | 40–60 | 25 chart kinds, all 2-D. Doughnut charts draw every series as a concentric ring (first series innermost), with editable hole size and first slice angle. The chart model holds title, legend position, a data-labels flag, gridlines flag, axis titles and chart data tables (legend keys, borders, XLSX round-trip and print/PDF); no axis scaling, error bars, per-point formatting, chart sheets, 3-D, surface, map, Pareto. Kinds survive XLSX round trip (#158, #161), except stock charts: only 3–4 series are a stock chart, and any other count is written and reimported as a line chart. [chart-parity.md](chart-parity.md) | | Page layout, print, PDF | 5 | ~80% | ~55% | 12–20 | Page setup, print titles, breaks, page order, Page Break Preview, headers/footers, PDF export and printing. Paper sizes match Japanese Excel: B4/B5/B6 are the JIS sizes, ISO B4/B5/B6 stay reachable by name, and Japanese Postcard (はがき), Japanese Double Postcard (往復はがき), the chou/kaku envelopes and A6 are available. No true Page Layout view; the PDF writer uses only the 14 standard fonts without embedding (no CJK or other non-Latin text in PDF); web print only exports (#141, #143) | | What-if and analysis | 4 | ~50% | ~40% | 22–35 | Goal Seek, Scenario Manager, data tables, Analysis ToolPak (all 19 tools, no charts), Solver (Simplex LP, a local nonlinear method and Evolutionary; Answer, Sensitivity and Limits reports; Load/Save Model, Reset All and Solver Options; no Population report), Forecast Sheet. Missing: Analyze Data | @@ -351,6 +351,7 @@ The inventory carried over from the 2026-10-07 ROADMAP.md, updated for what land | 2026-10-18 | trivial | Stale chart notes: all chart kinds survive XLSX round-trip (PRs #158, #161) and trendlines landed | | 2026-10-16 | minor | Advanced Filter copy-to clears only its actual output and rejects overflow without changing cells (#1031) | | 2026-10-15 | minor | Row auto-fit: line breaks count only in wrapping cells (Excel: wrap off shows one line, AutoFit gives one line); unwrapped multi-line cells from files no longer grow rows on open (no score change) | +| 2026-10-12 | minor | Worksheet functions 514 / 523 (98.3%; 505 working): GETPIVOTDATA | | 2026-10-21 | minor | Automation: typed macros in the engine and `macro.*` commands, no interface yet (no score change); catalog 274 / 290 → 275 / 291 (View › Macros › Record Macro) | | 2026-10-12 | minor | Essentials AutoFit evidence: rows auto-fit oversized fonts on open and row formatting (#833) | | 2026-10-12 | minor | Sheet tabs: reveal the selected tab on navigation and resize while preserving manual scrolling (#681); no score change |