* Functions: GETPIVOTDATA (#284)
The engine records each PivotTable report value cell (value field,
row/column items, value) on refresh and when a workbook is opened;
GETPIVOTDATA is an evaluator special form that looks the value up by
value field and field/item pairs, #REF! when the report doesn't show it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* GETPIVOTDATA: shared lazy index, report values, hash lookup (review of #290)
- PivotTable::data_index is an Arc-shared PivotLookup (serde-skipped), so the
per-command workbook clone no longer copies the index; refresh sets it, clear resets it.
- No index work on open: the first GETPIVOTDATA on a pivot without one builds it from the
source via the engine-registered PIVOT_INDEXER; a failure is #REF!, not an empty index.
- The result is the value the sheet shows at the cell's report position, so formula and
report cell agree (test with a source edited after the last refresh).
- Lookup by a pre-lowercased hash key; numbers/booleans fall back to a scan on a miss.
- GETPIVOTDATA is no longer volatile: it depends on its pivot's whole report area.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* GETPIVOTDATA: keep the index of the stored report; collision-free keys; align estimates
The lookup index describes the report as last refreshed and is saved with
the pivot. A pivot loaded without one (XLSX) only builds it when the
recomputed layout matches the stored labels. Lookup keys are length-prefixed
and distinguish Boolean from text items. Function-parity estimates match.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* pivot: import PivotDataCell, PivotItemRef and PivotLookup explicitly after upstream dropped the glob import
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* pivot: add an ignored measurement of edits on a sheet with a large report
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* pivot: keep a failed lazy GETPIVOTDATA build retryable, never save an empty index
A failed lazy index build stored unwrap_or_default() into the shared,
first-set-wins PivotLookup, so a valid state coming later (undo, a
source fix, a reopen) could never rebuild it, and the empty list was
saved as a restored index. Cache only validated builds, treat an empty
list as no index in serialization, reject reports with more value
cells than the index cap instead of truncating, register the pivot
indexer in gridcraft-cli check, and add two regression tests.
* GETPIVOTDATA: move cached offsets with edits inside the report; exact numeric items first
Inserting or deleting rows or columns inside a report remapped the cells
and last_range but kept the cached value-cell offsets, so a lookup could
read another item's cell. shift_sheet_features now maps each indexed
cell through the edit into a new index (the shared one stays with the
undo snapshots); a value cell deleted with its line drops out (#REF!).
A numeric item was accepted on a formatted-label hash hit, so 0.6 and 1
both shown as "1" returned the first. A number now prefers the cell
whose item has exactly that value, before any label match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix pivot chart test for the new Output.data field
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Native JSON workbook: golden bytes before the move to craft-libs
A workbook with text, numbers, a formula, bold, a number format and a macro is
saved as native JSON. Its bytes, the bytes saved after a newer-format macro is
kept, and the error for a non-workbook file are golden files written by the
current code. Opening the golden file as .json and .scjson and saving it again
gives it back.
* Native JSON workbook: read and write through craft-gridcraft-format
The `.json`/`.scjson` extension check, the reader and its error text, and the
pretty writer with newer-format macros written back into `macros` now come from
craft-gridcraft-format (storytold/craft-libs#1749), pinned at its PR tip. The
Workbook model, the style-index and cell-band rebuilds after a read, and macro
reading and checks stay here.
The golden tests from the previous commit pass unchanged.
Part of storytold/craft-libs#778.
* Native JSON workbook: pin craft-gridcraft-format to the craft-libs merge commit 9a6d8942 (storytold/craft-libs#1749)
---------
Co-authored-by: FredApps <FredApps@users.noreply.github.com>
**Stacked on #940.** It contains all of #940 (Arabic shaping in PDF) and adds cell direction and right-to-left sheet geometry (`M12.AR5`). Review the [diff against #940's branch](https://github.com/Mouhand-Kaddo/gridcraft/compare/fix/arabic-pdf-text...fix/arabic-pdf-reading-direction); once #940 merges, this PR is up to date with it.
## Summary
PDF export now honors cell Context, Left-to-Right and Right-to-Left settings. Wrapped lines keep their paragraph direction. Right-to-left sheets place columns, repeated headings, merged cells and objects in the same order as the grid, and explicit Left and Right alignment stay on the side the grid shows them.
```diff
PDF export
- infer each line separately; place every sheet left to right
+ preserve cell direction across lines; mirror RTL sheet positions
+ reject oversized text before appending an incomplete PDF block
```
Depends on #940 (Arabic PDF glyphs). The cell reading direction it reads landed with #947.
## Evidence
- **Before:** PDF ignored explicit reading order and RTL sheet layout. A public size-limit probe returned a PDF with an unmatched marked-content block.
**After:** Direction, wrapped paragraphs, numeric brackets, RTL columns, headings, objects, clipping and sparklines have focused regressions. The same size-limit probe returns an error; its regression failed before the fix and passes afterward.
- At the current head, with Noto Sans Arabic installed and no `CRAFT_FONTS_DIR`: the `gridcraft-pdf` and `gridcraft-engine` tests, workspace clippy and fmt pass. The full `cargo xtask ci` with the craft-fonts bundle relies on the hosted gate.
- Poppler rendering inspected for LTR and RTL sheets. Bounding boxes confirm reversed column positions; logical Arabic strings remain searchable. No physical printer test.
## Merge Danger
**Door:** Two-way. Revert the commit to restore prior export behavior.
**Blast Radius:** PDF. Explicit cell direction is shared with XLSX persistence. Latin text retains the standard-font path, including forced RTL numeric text without an Arabic font.
## Summary
Arabic text reached the PDF writer intact, then WinAnsi encoding replaced its characters with question marks. This fixes#923 through the shared PDF text path, including cells, headers and chart labels.
```diff
PDF text
- encode Arabic through WinAnsi
+ shape with harfrust and Unicode bidi
+ embed licensed glyph outlines with matching weight coordinates
+ preserve logical source using ToUnicode and ActualText
+ fall back to the standard and CID fonts for characters the Arabic face lacks
```
Font binaries stay in the optional craft-fonts build input. Native unbundled builds can use installed Noto Sans Arabic. Explicit cell reading-direction integration follows separately.
**Web build:** shaped Arabic on web needs the bundled craft fonts. The current release workflow doesn't enable `web-craft-fonts`, so until a maintainer enables it, web sets Arabic in the CID font, as main does today, which has no Arabic glyphs, and shows a message that no Noto Sans Arabic face was found. The same applies to a desktop without Noto Sans Arabic, and to bold Arabic where only the regular face is installed.
## Evidence
- **Before:** Arabic exported as `?` (#923, v0.6.1); main now sets it in the CID font, which has no Arabic glyphs, so readers show blanks or boxes.
**After:** native and bundled exports show joined letters, marks, Western/Arabic digits, bold/italic, wrapping, rotation and chart labels. Fresh `pdfinfo`, `pdffonts` and `pdftotext -bbox` checks accept the files; eight original Arabic strings match extracted logical text exactly.
- At the current head, with Noto Sans Arabic installed and no `CRAFT_FONTS_DIR`: the `gridcraft-pdf`, `gridcraft-engine` and `gridcraft-xlsx` tests, clippy, fmt, layers, assets, parity and the wasm check pass. The full `cargo xtask ci` with the craft-fonts bundle relies on the hosted gate.
- Characters the Arabic face lacks (emoji, Cyrillic) are set like non-Arabic text, and without an Arabic face for the text's weight Arabic takes the CID path main uses, with one message per export. Only a broken font or a size limit fails the export. Size limits cover text, glyph subsets, outlines and page content.
## Merge Danger
**Door:** two-way
**Blast Radius:** printing
`PdfDoc::finish` now returns a Result and every caller handles it. Latin-only rendering stays on its existing path. This is a glyph-outline PDF solution; platform printer and physical screen-reader behavior are not claimed.
## What
When two array formulas want the same cells, the one that already has its spill keeps them, whichever anchor comes first (this is what Excel does, measured on this PR by Harrison-Booth). The other formula shows `#SPILL!` until the winner is cleared, then it spills. When neither has a spill yet (a first calculation with no remembered layout), the anchor first in reading order (row, then column) wins, so the result never depends on evaluation order.
Before, an incremental recalculation let the existing spill keep its cells, while a full recalculation laid everything out from scratch and gave them to the earlier anchor. The two disagreed. A full recalculation now starts from the layout the sheet held, so both agree.
Also fixed: clearing a spilling formula did not unblock an array blocked by its old area when that array sat below or to the right of it.
## Why
Issue #2480: on the sales sample, `cell.set C14 =TRANSPOSE(B18:B24)` followed by `range.fill =D14:E18` over E13:F13 left F13 as `#SPILL!` in memory but `0` after a full recalc (the fuzzer, `command_sequence_fuzz` seed 0, hits this on #795). Now C14 keeps its cells, F13 is `#SPILL!`, and a full recalc gives the same.
Known limits: the xlsx reader does not restore spill layouts, so a workbook opened from a file is laid out by reading order only (Excel keeps the layout the file records); and when both anchors already held a spill and one grows into the other, reading order decides (Excel keeps the cells the other owns). The rule is documented on `Standing` in `crates/calc/src/recalc.rs`.
## Test
In `crates/engine/src/tests.rs`: the two-command sequence from the issue; both entry orders of an overlap, plus clearing the winner; and a no-history layout, where the earlier anchor wins in both entry orders. All compare cells with a full recalc.
## How verified
- `cargo fmt --check`, `cargo clippy -p gridcraft-calc -p gridcraft-engine --all-targets -- -D warnings`, `cargo test -p gridcraft-calc -p gridcraft-engine` (includes the fuzz test): all pass.
Fixes#2480
* Sandbox: refuse fcntl's asynchronous notification on Linux
* Sandbox: tests for fcntl's notification rules and the new probe
* Sandbox: the socket ioctls that notify another process, too
The package reader reports a part over the limit as "expands to more
than <limit> bytes"; the test added in #2491 expected the declared size
and failed on main.
In a descending custom-list key a blank tied with values not in the list,
which come first, while it still went after the listed values, so the order
depended on the input. It now goes last.
Comparing two values not in the list changed descending custom-list keys too:
they no longer tied, so a later key could not order them. Only an ascending
key sorts them now; with a descending one they tie, as before.
Repeating a copy down a selection narrower than it writes the copy's width on
every row, so the 4,000,000-cell limit on a tiled paste now counts that area
rather than the selection's. Past it the copy is pasted once, as a tiled
paste over too large a selection already is.
Data Subtotal grouped each group's detail rows at level 1 and left the
subtotal rows at level 0, so outline level 1 still showed every
subtotal. Excel adds an outer group over all data and subtotal rows,
summarised by the grand total: details at level 2, subtotals at
level 1, the grand total at level 0. Subtotal now adds that group after
writing the grand total.
Two tests that read the old levels now expect Excel's: the summary-row
test shows level 2 to keep the subtotal visible, and the sheet-prefix
test expects a detail row at level 2.
Paste tiled the copy only over a selection whose height and width were
both multiples of the copy's. A row copied into a narrower selection
several rows tall (A1:B1 into A3:A6) was pasted once. Excel repeats it
down every row of the selection at the copy's width. A selection
narrower than the copy now tiles down when its height is a multiple of
the copy's, and one shorter than the copy tiles across when its width
is a multiple.
A custom-list sort key compared only each value's position in the list.
Every value outside the list got the same position, so the stable sort
left them in their original order. Excel sorts them after the listed
values in the usual order. Two unlisted values now compare as a sort
without a list does (matchCase included).
Measured with crates/engine/examples/bench.rs (perf/xtask-bench), 100k rows, M2 Pro:
enter.formulas 465 -> 420 ms (23829d8c: 375), xlsx.save 540 -> 488 ms (466).
- number_to_text: the round-up-to-1E+15 check (f29ac7d9e) formatted every number with
exact {:.14e} and parsed it back; formula printing calls it once per literal. Run it
only for magnitudes that can round up (>= 9.9999999999999e14). Same output; boundary
tests added.
- Expr clone/map, xlsx prefix_params: a binary node whose left operand is not a chain
no longer allocates a spine Vec (2aeb21d4d); map rewrites the operands in their boxes.
Expr::walk reserves room for a typical formula up front.
- grow_autofilter checks the sheet has an AutoFilter before looking up the cell.
- retained_bytes_against looks up the newer band's row once per row, not per cell.
- The journal measures an entry's JSON size with a counting writer instead of building
the text (a big paste's params are MBs).
The rest of the enter.formulas gap and all of edit.fanout's (~+12%) is Undo's byte
budget (72db972de, #1227): sizing each snapshot and freeing evicted ones, an intended
cost of bounded memory.
Co-authored-by: nihilist-narwhal <240798405+nihilist-narwhal@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
read_cache turned every pivot cache error into absence with
optional_xml(part).ok().flatten(), so a cache over the part size limit
dropped its PivotTable with no warning, and a malformed or missing cache
only left a generic "skipped part" line. Each case now adds a warning that
the PivotTable was kept as plain cells, and the workbook keeps opening
(the oversized part is refused before it inflates).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Found while testing crates/engine/src/cmd/data.rs:992 (split_delimited, used by text_to_columns)
Co-authored-by: Claude Haiku 5.5 <noreply@anthropic.com>
* calc: linear dependency extraction for range-operator chains (#2183)
precedents() handled a range operator with non-reference operands by recursing into both operands and then letting Expr::walk descend into them again, so a left-associative chain of k operands cost 2^k (a 60-operand INDEX(...):INDEX(...) chain would hang on open). It is now a hand-written collector that visits every node once, growing the per-sheet bounding boxes of the enclosing range operator as it goes. Past a depth (4096) or node (1,000,000) bound it marks the formula dynamic (always recalculated) instead of recursing further.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* calc: drop wall-clock timing from precedents tests (clippy)
std::time::Instant::now is a disallowed method (clippy.toml: traps on wasm).
The regression test only needs to finish; the old 2^k walk would never return.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* calc: bound precedents to 100 ms and box plain-reference range chains (#2183)
- range_operator_chain_precedents_are_linear times the direct precedents
call against the 100 ms bound in the issue.
- A Range node with two plain Ref operands (A1:B2:C3, which the parser
can't fold) now collects its bounding box like any other range node, so
an edit to C1 recalculates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* M0: reject negative fractional GCD and LCM arguments
Validate the original input before truncation. Add scalar and range regressions, including signed zero, positive fractions and zero-first LCM validation.
* M0: document negative-fraction GCD and LCM regression
Record tested edge-case behavior and update the parity revision history. Function counts and readiness estimates remain unchanged.
* M0: preserve GCD and LCM regression after math test extraction
Move the original negative-fraction regression into the existing extracted math tests. Scalar and range cases, zero-first validation, signed zero and positive truncation coverage are unchanged.
ACCRINTM returned #NUM! whenever issue >= settlement. Excel rejects only
an issue date after settlement: when the two dates are equal after
truncation to whole days (including distinct times on the same day) it
returns 0 in every basis and both date systems. Relax the guard to
issue > settle; zero rate, zero par and an invalid basis stay #NUM!.