Files
gridcraft/crates/engine/src/pivot.rs
T
Valentin KleinandClaude Opus 5.5 ef99f1e51f Functions: GETPIVOTDATA (#284) (#290)
* Functions: GETPIVOTDATA (#284)

The engine records each PivotTable report value cell (value field,
row/column items, value) on refresh and when a workbook is opened;
GETPIVOTDATA is an evaluator special form that looks the value up by
value field and field/item pairs, #REF! when the report doesn't show it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* GETPIVOTDATA: shared lazy index, report values, hash lookup (review of #290)

- PivotTable::data_index is an Arc-shared PivotLookup (serde-skipped), so the
  per-command workbook clone no longer copies the index; refresh sets it, clear resets it.
- No index work on open: the first GETPIVOTDATA on a pivot without one builds it from the
  source via the engine-registered PIVOT_INDEXER; a failure is #REF!, not an empty index.
- The result is the value the sheet shows at the cell's report position, so formula and
  report cell agree (test with a source edited after the last refresh).
- Lookup by a pre-lowercased hash key; numbers/booleans fall back to a scan on a miss.
- GETPIVOTDATA is no longer volatile: it depends on its pivot's whole report area.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* GETPIVOTDATA: keep the index of the stored report; collision-free keys; align estimates

The lookup index describes the report as last refreshed and is saved with
the pivot. A pivot loaded without one (XLSX) only builds it when the
recomputed layout matches the stored labels. Lookup keys are length-prefixed
and distinguish Boolean from text items. Function-parity estimates match.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: import PivotDataCell, PivotItemRef and PivotLookup explicitly after upstream dropped the glob import

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: add an ignored measurement of edits on a sheet with a large report

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* pivot: keep a failed lazy GETPIVOTDATA build retryable, never save an empty index

A failed lazy index build stored unwrap_or_default() into the shared,
first-set-wins PivotLookup, so a valid state coming later (undo, a
source fix, a reopen) could never rebuild it, and the empty list was
saved as a restored index. Cache only validated builds, treat an empty
list as no index in serialization, reject reports with more value
cells than the index cap instead of truncating, register the pivot
indexer in gridcraft-cli check, and add two regression tests.

* GETPIVOTDATA: move cached offsets with edits inside the report; exact numeric items first

Inserting or deleting rows or columns inside a report remapped the cells
and last_range but kept the cached value-cell offsets, so a lookup could
read another item's cell. shift_sheet_features now maps each indexed
cell through the edit into a new index (the shared one stays with the
undo snapshots); a value cell deleted with its line drops out (#REF!).

A numeric item was accepted on a formatted-label hash hit, so 0.6 and 1
both shown as "1" returned the first. A number now prefers the cell
whose item has exactly that value, before any label match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix pivot chart test for the new Output.data field

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 08:59:28 +01:00

369 lines
15 KiB
Rust

//! PivotTables: reading the source list, grouping and aggregating, laying out the report and
//! writing it onto the sheet as static, styled cells.
//!
//! Behaviour follows Excel's documented PivotTable model (rows/columns/values/filters areas,
//! Compact/Outline/Tabular layouts, subtotals, grand totals, show-values-as); the code and the
//! look (our own palette) are original.
use std::cmp::Ordering;
use std::collections::{HashMap, HashSet};
use gridcraft_core::date::datetime_from_serial;
pub use gridcraft_core::quote_sheet;
use gridcraft_core::{CellError, CellRef, MAX_COLS, MAX_ROWS, RangeRef, Value, split_sheet};
use gridcraft_model::{
BorderLine, BorderStyle, Cell, Chart, Color, Fill, HAlign, NumFmt, PivotDataCell, PivotDateGroup, PivotField, PivotFilter, PivotFunc,
PivotItemRef, PivotLayout, PivotLookup, PivotShowAs, PivotSort, PivotTable, PivotValue, Series, Style, StyleId, Workbook,
};
use crate::pivot_calc::CalcSet;
mod charts;
mod compute;
mod items;
mod output;
use charts::rebind_charts;
pub use charts::{chart_link, chart_series, clear_output, links_to, parse_chart_link};
use compute::compute_from;
pub use compute::{NOT_A_VALUE_CELL, detail_cells, detail_records};
pub use items::{distinct_labels, func_caption, item_label};
#[cfg(test)]
use output::data_cols_exceed_sheet;
pub use output::{CellKind, OutCell, Output, RowKind};
use output::{cell_style, look};
/// Largest source block read (cells).
pub const MAX_SOURCE_CELLS: u64 = 10_000_000;
/// Largest report written (cells).
pub const MAX_OUTPUT_CELLS: u64 = 2_000_000;
/// Most fields in the Rows plus Columns areas.
pub const MAX_AXIS_FIELDS: usize = 32;
/// Most fields in the Values area.
pub const MAX_VALUE_FIELDS: usize = 256;
/// Most report value cells a report may have (every one is indexed for GETPIVOTDATA; bigger
/// reports are rejected).
pub const MAX_INDEXED_CELLS: usize = 1_000_000;
const MONTHS: [&str; 12] = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
// ---------------------------------------------------------------- source
/// The source list: header names, per-column number formats and the records.
#[derive(Clone, Debug)]
pub struct Source {
pub sheet: usize,
pub range: RangeRef,
pub headers: Vec<String>,
pub formats: Vec<String>,
/// Every non-blank value in the column is a number.
pub numeric: Vec<bool>,
/// Numeric with a date number format.
pub dates: Vec<bool>,
pub rows: Vec<Vec<Value>>,
}
impl Source {
pub fn col(&self, name: &str) -> Option<usize> {
self.headers.iter().position(|h| h.eq_ignore_ascii_case(name))
}
}
/// Absolute reference text for a range on a sheet: `Sheet1!$A$1:$E$20`.
pub fn range_text(sheet_name: &str, r: RangeRef) -> String {
let abs = |c: CellRef| format!("${}${}", gridcraft_core::col_to_letters(c.col), c.row as u64 + 1);
format!("{}!{}:{}", quote_sheet(sheet_name), abs(r.start), abs(r.end))
}
/// Resolves source text (range, table name or defined name) to (sheet, range including header).
pub fn resolve_source(wb: &Workbook, text: &str, default_sheet: usize) -> Result<(usize, RangeRef), String> {
resolve_inner(wb, text, default_sheet, 0)
}
fn resolve_inner(wb: &Workbook, text: &str, default_sheet: usize, depth: u32) -> Result<(usize, RangeRef), String> {
const INVALID: &str = "The reference isn't valid.";
let t = text.trim().trim_start_matches('=').trim();
if t.is_empty() || depth > 4 {
return Err(INVALID.into());
}
if let Some((si, ti)) = wb.table(t)
&& let Some(tb) = wb.sheet(si).and_then(|s| s.tables.get(ti))
{
if !tb.header_row {
return Err("The table needs a header row to be used as a PivotTable source.".into());
}
let end_row = tb.range.end.row.saturating_sub(tb.totals_row as u32).max(tb.range.start.row);
return Ok((si, RangeRef::new(tb.range.start, CellRef::new(end_row, tb.range.end.col))));
}
let (sheet, body) = split_sheet(t);
if let Some(r) = RangeRef::parse(body) {
let si = match sheet {
Some(n) => wb.sheet_index(&n).ok_or_else(|| INVALID.to_string())?,
None => default_sheet,
};
let sh = wb.sheet(si).ok_or_else(|| INVALID.to_string())?;
let r = if r.is_full_cols() || r.is_full_rows() {
let used = sh.used_range().ok_or_else(|| INVALID.to_string())?;
r.intersection(&used).ok_or_else(|| INVALID.to_string())?
} else {
r
};
return Ok((si, r));
}
if sheet.is_none()
&& let Some(n) = wb.name(t, default_sheet)
{
let f = n.formula.clone();
return resolve_inner(wb, &f, default_sheet, depth + 1);
}
Err(INVALID.into())
}
const FIELD_NAME_ERR: &str =
"The PivotTable field name is not valid. To create a PivotTable, you must use data that is organized as a list with labeled columns.";
/// Reads the source list.
pub fn read_source(wb: &Workbook, text: &str, default_sheet: usize) -> Result<Source, String> {
let (si, r) = resolve_source(wb, text, default_sheet)?;
let sh = wb.sheet(si).ok_or("The reference isn't valid.")?;
if r.count() > MAX_SOURCE_CELLS {
return Err("The source range is too large for a PivotTable.".into());
}
let mut headers: Vec<String> = Vec::new();
// Names taken so far (lower-case), and the next suffix to try for each repeated header, so
// that thousands of equal headers don't take cubic time.
let mut taken: HashSet<String> = HashSet::new();
let mut next_suffix: HashMap<String, usize> = HashMap::new();
let mut formats = Vec::new();
for c in r.start.col..=r.end.col {
let h = crate::display::cell_text(wb, sh, CellRef::new(r.start.row, c)).trim().to_string();
if h.is_empty() {
return Err(FIELD_NAME_ERR.into());
}
let key = h.to_ascii_lowercase();
let mut n = h.clone();
let mut k = next_suffix.get(&key).copied().unwrap_or(2);
while taken.contains(&n.to_ascii_lowercase()) {
n = format!("{h}{k}");
k += 1;
}
next_suffix.insert(key, k);
taken.insert(n.to_ascii_lowercase());
headers.push(n);
let fmt = if r.end.row > r.start.row {
wb.styles.get(sh.style_id(CellRef::new(r.start.row + 1, c))).num_fmt.as_str().to_string()
} else {
"General".into()
};
formats.push(fmt);
}
let w = headers.len();
let mut rows = Vec::new();
let mut numeric = vec![true; w];
let mut any = vec![false; w];
if r.end.row > r.start.row {
for row in r.start.row + 1..=r.end.row {
let rec: Vec<Value> = (r.start.col..=r.end.col).map(|c| sh.value(CellRef::new(row, c)).scalar()).collect();
if rec.iter().all(Value::is_empty) {
continue;
}
for (i, v) in rec.iter().enumerate() {
if !v.is_empty() {
if let Some(a) = any.get_mut(i) {
*a = true;
}
if !v.is_number()
&& let Some(n) = numeric.get_mut(i)
{
*n = false;
}
}
}
rows.push(rec);
}
}
for (n, a) in numeric.iter_mut().zip(any.iter()) {
*n = *n && *a;
}
let dates = numeric.iter().zip(formats.iter()).map(|(n, f)| *n && crate::display::parsed(f).is_date()).collect();
Ok(Source { sheet: si, range: r, headers, formats, numeric, dates, rows })
}
/// Computes a PivotTable report from the workbook.
pub fn compute(wb: &Workbook, pt: &PivotTable, pivot_sheet: usize) -> Result<Output, String> {
let src = read_source(wb, &pt.source, pivot_sheet)?;
compute_from(wb, pt, &src, None)
}
/// Recomputes a pivot and writes its report. With `replace` false, refuses when the target area
/// holds other data. Returns the written area.
pub fn refresh(wb: &mut Workbook, sheet: usize, idx: usize, replace: bool) -> Result<RangeRef, String> {
let pt = wb.sheet(sheet).and_then(|s| s.pivots.get(idx)).cloned().ok_or("There's no such PivotTable.")?;
let out = compute(wb, &pt, sheet)?;
let h = out.height().max(1);
let w = out.width.max(1);
let k = out.body_offset;
let origin_row = if pt.anchor.row >= k { pt.anchor.row - k } else { pt.anchor.row };
let origin = CellRef::new(origin_row, pt.anchor.col);
let end_row = origin.row as u64 + h as u64 - 1;
let end_col = origin.col as u64 + w as u64 - 1;
if end_row >= MAX_ROWS as u64 || end_col >= MAX_COLS as u64 {
return Err("The PivotTable report will not fit on the sheet. Choose a different location.".into());
}
let area = RangeRef::new(origin, CellRef::new(end_row as u32, end_col as u32));
let sh = wb.sheet(sheet).ok_or("There's no such sheet.")?;
for (j, other) in sh.pivots.iter().enumerate() {
if j != idx && other.last_range.is_some_and(|r| r.intersects(&area)) {
return Err("A PivotTable report cannot overlap another PivotTable report.".into());
}
}
if sh.is_protected() {
return Err("The cell or chart you're trying to change is on a protected sheet.".into());
}
if let Ok((ssi, sr)) = resolve_source(wb, &pt.source, sheet)
&& ssi == sheet
&& sr.intersects(&area)
{
return Err("The PivotTable report can't overlap its source data. Choose a different location.".into());
}
if sh.tables.iter().any(|t| t.range.intersects(&area)) {
return Err("A PivotTable report cannot overlap a table.".into());
}
if !replace {
for (c, cell) in sh.cells.iter_range(area) {
if cell.has_content() && !pt.last_range.is_some_and(|r| r.contains(c)) {
return Err(format!(
"There's already data in {}!{}. Do you want to replace it? (Run again with replace: true to overwrite.)",
quote_sheet(&sh.name),
c.a1()
));
}
}
if let Some(c) = sh.spill.keys().find(|c| area.contains(**c) && !pt.last_range.is_some_and(|r| r.contains(**c))) {
return Err(format!(
"There's already data in {}!{}. Do you want to replace it? (Run again with replace: true to overwrite.)",
quote_sheet(&sh.name),
c.a1()
));
}
}
// Styles first (the style table is workbook-wide).
let lk = look(&pt.style);
let header_last = out.row_kinds.iter().rposition(|k| *k == RowKind::Header);
let mut cache: HashMap<Style, StyleId> = HashMap::new();
let mut cells: Vec<(CellRef, Cell)> = Vec::new();
for (ri, row) in out.grid.iter().enumerate() {
let rk = out.row_kinds.get(ri).copied().unwrap_or(RowKind::Blank);
if rk == RowKind::Blank {
continue;
}
for (ci, oc) in row.iter().enumerate() {
if rk == RowKind::Filter && oc.value.is_empty() {
continue;
}
if rk == RowKind::Placeholder && oc.value.is_empty() {
continue;
}
let flag = match rk {
RowKind::Header => Some(ri) == header_last,
RowKind::Filter => ci == 0,
_ => false,
};
let st = cell_style(&lk, rk, oc, flag);
let id = match cache.get(&st) {
Some(id) => *id,
None => {
let id = wb.styles.intern(st.clone());
cache.insert(st, id);
id
}
};
if oc.value.is_empty() && id == StyleId::default() {
continue;
}
cells.push((
CellRef::new(origin.row + ri as u32, origin.col + ci as u32),
Cell { value: oc.value.clone(), formula: None, style: id, py: None },
));
}
}
// Column widths that fit the labels.
let mut widths: HashMap<u32, f32> = HashMap::new();
for (c, cell) in &cells {
if cell.value.is_empty() {
continue;
}
let st = wb.styles.get(cell.style);
let text = crate::display::format(&cell.value, st.num_fmt.as_str(), wb).text;
let wpt = crate::cmd::format::approx_text_width(&text, st.font.size, st.font.bold) + st.align.indent as f32 * 9.0 + 14.0;
let e = widths.entry(c.col).or_insert(0.0);
*e = e.max(wpt.min(320.0));
}
let shm = wb.sheet_mut(sheet).ok_or("There's no such sheet.")?;
if let Some(old) = pt.last_range {
shm.take_cells(old);
}
shm.take_cells(area);
for (c, cell) in cells {
shm.set_cell(c, cell);
}
for (col, wpt) in widths {
if wpt > shm.col_width(col) && !shm.is_col_hidden(col) {
shm.cols.entry(col).or_default().size = Some(wpt);
}
}
if let Some(p) = shm.pivots.get_mut(idx) {
p.last_range = Some(area);
}
let sheet_name = shm.name.clone();
rebind_charts(wb, &sheet_name, &pt.name, origin, &out);
// The index of the new report; its lookup hash is built on the first GETPIVOTDATA.
if let Some(p) = wb.sheet_mut(sheet).and_then(|s| s.pivots.get_mut(idx)) {
let index = PivotLookup::default();
index.set_cells(out.data);
p.data_index = index;
}
Ok(area)
}
/// The value cells of a PivotTable's report, for GETPIVOTDATA when a refresh did not set the
/// index (a workbook just opened from XLSX). The positions must describe the report as it is
/// stored on the sheet, not as the current source would render it: the source is recomputed only
/// to learn the layout, and the result is accepted when every label and header cell of that
/// layout equals the stored cell at the same place. A source changed since the last refresh gives
/// a different layout, so there is no trustworthy index and the lookup fails until a refresh.
pub fn index_cells(wb: &Workbook, sheet: usize, pt: &PivotTable) -> Result<Vec<PivotDataCell>, String> {
const STALE: &str = "The stored report no longer matches the source. Refresh the PivotTable.";
let report = pt.last_range.ok_or("The PivotTable has no report.")?;
let out = compute(wb, pt, sheet)?;
let sh = wb.sheet(sheet).ok_or("There's no such sheet.")?;
let rows = u64::from(report.end.row) - u64::from(report.start.row) + 1;
let cols = u64::from(report.end.col) - u64::from(report.start.col) + 1;
if rows != u64::from(out.height().max(1)) || cols != u64::from(out.width.max(1)) {
return Err(STALE.into());
}
let labels = [CellKind::Header, CellKind::RowLabel, CellKind::ColLabel, CellKind::Filter];
for (r, row) in out.grid.iter().enumerate() {
for (c, oc) in row.iter().enumerate() {
if labels.contains(&oc.kind) && sh.value(CellRef::new(report.start.row + r as u32, report.start.col + c as u32)) != oc.value {
return Err(STALE.into());
}
}
}
Ok(out.data)
}
#[cfg(test)]
mod fit_tests {
use super::data_cols_exceed_sheet;
#[test]
fn column_budget_counts_value_fields_and_grand_total() {
assert!(!data_cols_exceed_sheet(64, true, 128));
assert!(data_cols_exceed_sheet(64, true, 256));
assert!(!data_cols_exceed_sheet(64, false, 256));
assert!(!data_cols_exceed_sheet(16_384, false, 0));
assert!(data_cols_exceed_sheet(16_385, false, 0));
}
}