* engine: bound Advanced Filter, Remove Duplicates, AutoFilter, Circle Invalid Data; one-pass AutoFit; saturating range dimensions
* address review: AutoFit scans each run of adjacent selected columns separately
* address review: report a truncated circle-invalid scan, no spill scan for plain cells in Replace, one pass for many AutoFit runs
* address review: warn in the app when Circle Invalid Data stopped early; Replace test requires success
* address review: the scan note joins an open-time notice; say cell visits
* engine: index merges and bound the page in HTML export and clipboard HTML
Save as Web Page and the clipboard's HTML scanned every merge of the
sheet for every cell, counted a merge's span over all the rows the file
gave it, placed each picture by summing every custom row before it, and
grew the page without a byte limit. Merges are now indexed once over
the exported rectangle (overlapping merges past a few visits per cell
are refused), spans use prefix counts clipped to the table, anchors use
LineIndex, and the page stops at 256 MiB with an error.
* address review: check the web page size after each chart and before returning it
* address review: bound the merge index, escape cell text within the page's room, size images before encoding
* address review: escape image attributes within the page's room
* Directory walkers key visited directories by (dev, inode); stats skips non-regular or oversized .rs targets
* address review: record a walked directory only within the depth limit; stats skips symlinked sources and bounds the read
* address review: rescan a directory reached from a shallower path, skipping files already collected
* xtask stats: test the directory walk (links, identity, size limit)
* address review: create the test directories atomically and privately
* xtask stats tests: no unreachable! in the directory helper
* gridcraft check: unit-test that each workbook is listed once whatever path reaches it
* functions: one work budget across every call of a lifted function
A function lifted over a computed array runs once per element, and only the
six lookups counted their repeated work against LIFTED_SCAN_BUDGET. Criteria
aggregates rescanned their ranges per criterion, CUMIPMT/CUMPRINC stepped
through up to 1e7 periods per element, and array-producing functions such as
MUNIT built a whole result per element before keeping its first cell.
The budget now covers all of it: COUNTIF/SUMIF/AVERAGEIF(S)/MAXIFS/MINIFS
charge their range cells per call, the period loops of every element are
added up before any runs, and array results charge their cells as they are
built. Past the budget the call is #NUM!, as lifted lookups already are.
* address review: charge numeric-text periods of lifted CUMIPMT/CUMPRINC to the work budget
* address review: charge only calls that reach the cumulative period loop
* address review: charge the numeric text a lifted period call parses
* address review: SUMIF/AVERAGEIF charge only the sum-range cells they read; weight cumulative period work
* print: hold the print area to the grid; render each chart once and budget picture and chart output across pages
* address review: budget chart path points and drop charts that do not fit the budget
Adds step-security/harden-runner (audit, pinned to v2.22.4 like
craft-common-actions) as the first step of each job that did not have it,
so every workflow in the org's public repos reports its egress to
StepSecurity. Audit, not block: block-mode allow lists come from each
job's observed traffic.
The seccomp list named libc::SYS_kexec_file_load, which libc (0.2.190) defines for x86_64 and
aarch64 but not riscv64, so the release's Linux riscv64 build stopped with E0425. riscv64 takes
the number from the kernel's generic table, 294, as aarch64 does; the test checks libc's aarch64
value against it.
* xlsx: share each custom number format across cell formats, cap font names at 31 characters
Every cell format record copied its font (name included) and built its
own copy of its number-format code, so many short records naming one
long font or code multiplied it in memory. Custom codes are now built
once and shared, and font names are cut to Excel's 31 characters.
* xlsx: drop stray worktree entries added by mistake
* address review: compare truncated theme font for the scheme link; intern inline dxf number formats
* Charts: one point budget per chart and a byte budget for web-page SVG
A chart resolved every series reference into its own vector of up to
100,000 points, so 255 short whole-column series retained and drew
25.5 million points, and Save as Web Page wrote one SVG element per
mark without any output limit.
- resolve() shares MAX_CHART_POINTS (1,000,000) across the values,
X values and bubble sizes of all series; a series past the budget
gets what is left.
- chart_svg() stops writing once a chart passes the bytes left of a
64 MiB per-page budget and leaves that chart out.
* address review: charge X values to the chart point budget only when they are kept
* address review: charge scatter Y and X values to the chart point budget as a pair
* xls: charge expanded shared-formula text to the import's text allowance
Shared-formula followers each kept their own printed formula text, but
none of it counted against the 256 MB text allowance, so compact PtgExp
records could expand into gigabytes of retained formulas. Each attach
now charges its text; past the allowance the formula is dropped and the
cell keeps its cached value.
* address review: keep resolving array formulas after the shared-formula text allowance is spent
* address review: keep the shared-formula text cut-off across worksheets
* address review: expand shared formulas only after every sheet's mandatory text is charged
* address review: refuse duplicate sheet names at import; adapt tests to read_xls_budgeted
* pivot: cached Rank and Running Total, accumulator-width limit, capped subtotal captions and PivotChart series
* address review: store running-total entries only for items with a number
* address review: running totals visit only the top-level items that hold the path
* xlsx: resolve cell pictures through a per-structure layout and a relationship index
Every rich value rebuilt its structure's whole key list before checking
its value count, and scanned the image relationship table for a
duplicate id. Structures are now laid out once and relationships are
indexed by id (a repeated id stays invalid), so reading cell-picture
metadata is linear in its records.
* address review: wide-structure fixture uses the right count and loads a valid picture
* engine: bound Scenario Summary size and solver report name searches; charge comment and link text to undo history
* address review: count result rows in the scenario summary limit and reply/date vector storage in the history estimate
* Row auto-fit: sweep merges once per sheet instead of per row
auto_row_height() picked the merges touching a row by scanning every
merge on the sheet, and opening a workbook (fit_imported_rows) or a
commit fitting many rows called it per row, so rows x merges
comparisons ran synchronously on open.
Rows are fitted in ascending order, so a MergeSweep sorts the sheet's
merges by first row once and keeps only those active on the current
row. All comparisons count against MERGE_FIT_WORK (50,000,000) per
pass; past it the remaining rows keep the heights they have.
auto_row_height keeps its signature for single rows.
* address review: share the merge-comparison budget across the whole auto-fit pass
* address review: build the merge sweep lazily, after the row fast paths
* address review: charge the merge sort to the fit budget, keep row heights once it is spent
* address review: a pass that fits one row scans the merges instead of sorting them
* calc: INDEX decides a multi-cell position from its shape before reading it
* calc: charge each formula's evaluation against one cumulative work budget
* address review: a database function whose computed criterion ran out of work is #CALC!
* address review: give the test's criteria cell its stored value
* address review: stop matching database records once a computed criterion exhausts the work budget
* address review: skip the criteria-row scan once work is exhausted
* address review: stop between conditions once the work budget is spent
* xlsx: analyse each kept chart sheet's graph once per save
chart_sheet_usable walked a chart sheet's relationship graph and parsed
its charts for every workbook entry, so many entries naming one shared
graph multiplied the work. The referenced sheet names are now cached by
relationship id.
* address review: cache the chart-sheet usability verdict per relationship id instead of every reference
* address review: cache chart-sheet analysis by the graph root, not the relationship id
* numfmt: TEXT returns #VALUE! when its result passes a cell's 32,767 characters
A text section repeats its input once per @ placeholder, and TEXT handed
back whatever that built, so nested TEXT calls could multiply a long
text until memory ran out. Results past the cell text limit are now
#VALUE!.
* address review: stop TEXT rendering past the cell limit and count UTF-16 units
* xlsx: find a reused picture through a content hash instead of comparing every media part
Out::image_part compared each new picture byte by byte with every
media part written so far, so a workbook of many distinct pictures
sharing a long prefix was exported in quadratic time. Media parts are
now indexed by extension and SHA-256.
* address review: index preserved media parts when they are added
* address review: do not hash an image twice when it is first added
* xlsx: one hyperlink allowance per sheet, and a copy limit for unparsable shared masters
Every <hyperlinks> element of a worksheet started a fresh cell and byte
allowance although all of them fill the same map. The sheet now keeps
one allowance across its elements.
Followers of a shared master the parser refuses each copied its full
text; a sheet full of followers of a long master multiplied that text
without bound. The copies are now charged against a 64 MiB per-sheet
limit, past which the followers keep their cached values.
* address review: keep the cached value of a follower dropped for the shared-formula copy limit inside array ranges
* FORMULATEXT and the cell editor keep a hidden formula's source hidden
* address review: recalculate formulas when sheet protection changes
* address review: skip the full recalculation for a no-op unprotect
* address review: recalculate on protect/unprotect without marking the edit structural
* Analysis ToolPak: stop building a report once it outgrows the output limit
Report builders filled their whole output map before write_report
checked its size, so Descriptive Statistics grouped by rows over a
tall, nearly empty column allocated about 16 entries for each of up
to a million variables before refusing the report.
Grid now tracks its extent as values are stored and keeps nothing
once the report's rows x columns pass MAX_OUTPUT_CELLS; write_report
refuses such a report. extent() no longer rescans every entry.
* address review: stop the descriptive builder once the grid is too large
* address review: descriptive statistics reads variables one at a time
* Wildcards: match * and ? patterns in one pass over the text
After a mismatch the matcher restarted the pattern after the last
star at the next text position, so a long repetitive cell against a
near-matching pattern cost text x pattern comparisons. AutoFilter
custom criteria, Find and Replace all call it once per cell.
wildcard() now runs the pattern as a bit-parallel automaton: one pass
over the text, one step per 64 pattern positions for each character.
Results are unchanged; a test compares it with the old matcher on
every short text and pattern over a small alphabet.
* address review: build wildcard literal masks lazily with a bounded cache
* address review: keep masks of the most frequent wildcard literals past the cache cap
* chart: bound scan conversion, spline points and category label bytes
- The rasteriser tested every edge of a path at every sub-sample of every
covered row. It now keeps an active-edge list (edges sorted by top, joining
and leaving as the scan passes them), and one rasterize call has a total
scan-work allowance after which remaining fills are skipped.
- Smoothed lines budgeted interpolated points per run, so gap-separated short
runs escaped both decimation and the cap. The budget is now shared by all
runs of a line; once spent, runs are drawn through their data points.
- Category labels copied every text value in full. Labels are clipped to 255
characters before copying, within an 8 MiB budget per series.
* address review: chart raster and label budget fixes
* address review: bound polyline point count before stroke_paths
* functions: TEXTJOIN builds its result piece by piece within the text cap
TEXTJOIN converted every input and delimiter into an owned String before
joining, so many references to one long shared text copied gigabytes before
the joined length was ever checked. Errors are now found first without
building text (so they still win in argument order), then each piece and
delimiter is converted only as it is appended, stopping at the cap.
* address review: check the cap before copying each text piece
* functions: stop WORKDAY, PIVOTBY, SORT and the D-functions rescanning their input
- WORKDAY/WORKDAY.INTL counted the holidays each step passed by scanning every
holiday, once per step; a run of consecutive holidays made that quadratic.
The working-day holidays are kept sorted and counted by binary search.
- PIVOTBY totals rescanned the whole source (rebuilding keys) once per total
row and column. The grouping pass now also indexes rows by row key and by
column key, and totals read from those indexes in the same order.
- SORT copied one key per sort_index entry for every row, so a long
sort_index with repeated columns multiplied the source. Repeats can't break
a tie their first use left, so only first uses are kept.
- DSUM and the other database functions tested every record against every
criteria row. A selection whose records x conditions exceed 5e8 is #NUM!,
and each criteria label's database column is found once, not per cell.
* address review: bounds-check SORT indices before narrowing; empty criteria rows are not charged
* Forecast Sheet: count hindcast rows against the work limit
MAX_WORK bounded history rows x forecast rows, but a Forecast Start
inside the history adds a hindcast row for every history point after
it, each holding the same ETS formulas that refit the model over the
history before the start. Those rows were not counted, so a start in
the middle of 10,000 rows of history built millions of uncharged
model fits per recalculation.
The forecast row cap is now computed from the rows the model is fit
to, after subtracting the hindcast rows; a start so early that the
hindcast alone passes the limit is refused, and the default number of
periods stays within the cap.
* address review: base the in-history forecast cap on the slots the fit range spans
* address review: document the hindcast-aware forecast work limit
* address review: document the fitted-row minimum in the forecast work limit
* address review: say the forecast start row counts toward the hindcast work limit
* Find and Replace: bound the text a replacement builds
Each occurrence copied the whole replacement string, so a cell full of
short matches and a long replacement expanded tens of kilobytes of
input into about a gigabyte, in both the case-sensitive and the
case-insensitive paths.
Replacement now builds the new text only while it stays within
Excel's 32,767 characters per cell (counted as UTF-8 bytes, 4 each at
most) and within 256 MiB for the whole command, checking before each
copy. Past either limit the command fails and changes nothing.
* address review: enforce the 32,767-character limit on replacements and borrow the replacement text
* Get Data: refuse an import whose dense rows would be too large
data.getData and data.fromTextCsv built every row densely before
writing it: JSON records were padded to the union of all keys (found
with a quadratic Vec::contains scan), and CSV cells were expanded over
the whole used rectangle. A few sparse records or one far-away CSV
cell therefore allocated billions of slots.
The importer now collects JSON keys with a HashSet, refuses more keys
than a sheet has columns, and checks the projected rows x columns
against MAX_IMPORT_CELLS (10,000,000) before building any row.
* address review: stop collecting JSON keys at the column limit and reject oversized row counts before building rows
* address review: enforce the CSV import cell limit while parsing
* Conditional formatting: one scan budget per rule, and no thresholds for unsupported colour scales
numbers_in() scanned each of a rule's ranges separately with its own
2,000,000-cell limit, so a rule listing the same (or overlapping)
large range many times multiplied the scan while painting a single
cell. Colour scales also evaluated every stop's threshold, formulas
included, before discarding scales that are not 2- or 3-colour.
- A rule's statistics now share STATS_SCAN_LIMIT across all its ranges,
and an exact repeat of a range is scanned once.
- A colour scale with a stop count other than 2 or 3 paints nothing
and evaluates none of its thresholds.
* address review: count stored and spilled cells of sparse ranges, skip statistics for unsupported colour scales
* address review: skip unrelated spills a row at a time in the statistics scan
* Numfmt: one built-in format table, no dead render arms, fewer allocations (#817)
S1: the built-in format table lives only in gridcraft-xlsx (file formats do not
depend on numfmt); numfmt's unused copy and its exports are removed, xlsx gets
tests pinning its lookup rules.
S3: drop the render_number arm that printed what the fallback prints; literal
text is appended to the output instead of built as a String per call.
O3: render_number keeps pieces, text-before and literal replacements in one
buffer (Parts), fill_int writes digits left to right, Section stores the date
flags (sub-second digits, calendar, AM/PM) computed at parse time, render_date
writes into one buffer.
O4: significant_digits formats into a stack buffer; format_general_fit re-rounds
the 15 digits per width into reused strings.
An equality test compares format_value, TEXT() and format_general_fit with
digests recorded from the previous implementation.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* xlsx test: builtin ids 42 and 44 are written as custom formats like 5-8
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* numfmt: push_str for short month/day names (clippy)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
## Stack
The Python cells work is split by layer. It merges in this order, each with a merge commit rather than a squash. Nobody can switch Python on before #3481.
1. #3475 depends on `main`
2. #3476 depends on `main` **(this pull request)**
3. #3477 depends on [#3475](https://github.com/storytold/gridcraft/pull/3475) and [#3476](https://github.com/storytold/gridcraft/pull/3476)
4. #3478 depends on [#3477](https://github.com/storytold/gridcraft/pull/3477)
5. #3479 depends on [#3478](https://github.com/storytold/gridcraft/pull/3478)
6. #3480 depends on [#3479](https://github.com/storytold/gridcraft/pull/3479)
7. #3481 depends on [#3480](https://github.com/storytold/gridcraft/pull/3480)
## Summary
`crates/sandbox` runs a program that can read only the paths its policy lists, write nothing except `/dev/null`, start no process, and use no network. An application re-executes itself with a launcher flag, closes every inherited descriptor above standard error, applies the confinement, checks that it holds, and only then runs the program. If any of that fails, the program does not run.
## Changes
- **`crates/sandbox`:** the policy, the Linux backend (Landlock, and seccomp that denies all sockets, with an x32 filter), the macOS backend (`/usr/bin/sandbox-exec` with a deny-default profile), the probes and a canary.
- **`apps/gridcraft`, `apps/gridcraft-cli`:** the launcher flag comes first in `main`, before any window or log.
- **`gridcraft-cli sandbox-check`:** says whether this system confines programs, and can run a confined self-test.
- **`docs/sandbox.md`:** what is refused and how.
- **`nix`:** a new dependency on Linux and macOS, for closing descriptors without `unsafe`.
## Compatibility
Windows and FreeBSD have no backend here. There the crate reports that it can't confine, and nothing that needs it runs. On Linux a launch needs Landlock ABI 3, which is Linux 6.2 and later, so an older kernel is treated the same way: this excludes, for example, Debian 12's kernel (6.1) and Ubuntu 22.04's original kernel (5.15). Nothing changes for a process started without the launcher flag.
## Review fixes
- **File metadata on Linux.** Landlock has no right for a file's permissions, owner, extended attributes, times or flags, and the seccomp filter allows what it doesn't name, so a confined program could change them on any file its user owns, outside the policy too. The filter now refuses the `chmod`, `chown`, `setxattr`, `removexattr` and timestamp families with their `*at` variants, by path and by descriptor, and any `ioctl` whose number says it hands the kernel data, as `FS_IOC_SETFLAGS` and `FS_IOC_FSSETXATTR` do on a descriptor opened only to read. A filter allows what it doesn't name, so a call a later kernel adds is allowed until it is named; the newest named is `file_setattr` (Linux 6.17).
- **Truncation on Linux.** Landlock covers truncation only from ABI 3, so on ABI 1 or 2 `os.truncate(path, 0)` emptied any file its user can write, and `open` with `O_RDONLY | O_TRUNC` emptied one it may read. A launch now requires that right. A filter can't take its place: `openat2` hides its flags from seccomp, and refusing `O_TRUNC` would refuse opening `/dev/null` to write, as `open(os.devnull, "w")` does.
- **Standard streams.** They stay open with the rights they were opened with, so a stream that is a file can be written and truncated, and one that is a socket can still send. The page and `launch_command` say so, and that a parent gives the program pipes.
- **The enforcement check, again.** It now also tries to change the canary's permissions and to truncate it, asking for what it already has, so nothing changes if one works.
- **Other processes on Linux.** On a kernel without Landlock's signal scope, a confined program could signal any process of the same user, and `kill(-1, 9)` would reach them all. The seccomp filter now refuses signals to any process but the program itself, whose ID `exec` keeps; changes to the limits, priority or scheduling of any other process; `pidfd_open`, `pidfd_send_signal`, `kcmp`, `get_robust_list`, `migrate_pages` and `move_pages`; every System V shared memory, message queue and semaphore call and every POSIX message queue call, which no path rule covers; and `syslog`.
- **The terminal on Linux.** A program started from a shell could type into its terminal. `TIOCSTI` and `TIOCLINUX` are refused, compared in 32 bits as the kernel reads them, so high bits set on the argument don't get past the rule.
- **The enforcement check.** It now also tries signal 0 on the launcher's parent, which delivers nothing, so `sandbox-check` reports a gap rather than assuming none. The launcher tries the same signal before it confines itself and refuses if that fails, and the check refuses if its parent has changed since, since a parent that refuses anyone would prove nothing. A launcher's arguments must include the current executable, which that check starts; otherwise the file rules refused the start and the check proved nothing about new processes.
- **A refused launch.** The launcher read its arguments only after closing every inherited descriptor, so a refused launch had already closed them. It reads them first.
- **A kernel without Landlock.** The refusal says so in that case too, which the tests that skip on such a machine look for. Only the first ABI's rights being refused, or `ENOSYS` or `EOPNOTSUPP` from creating the ruleset, says so. Any other failure, such as `EINVAL` or too many layers, is reported as what it is and doesn't skip the tests.
- **Folders in the exec list on macOS.** A folder got a rule that matches only the folder, so a launcher under it failed to start where the same policy works on Linux. It now covers what is in it, in both rules.
- **The sandbox page** names only what this layer has: only `gridcraft-cli sandbox-check` starts the launcher here.
## Testing
All 10 steps of `cargo xtask ci` pass on macOS (arm64), and `cargo clippy` passes for this crate for both Linux architectures. The gate on ubuntu-24.04 is the hosted one on this pull request, and it executes the Linux backend through the real binary, including `a_confined_program_leaves_other_processes_alone` and `a_confined_program_changes_no_file`, in which a confined Python tries each refused call on a file inside the policy, one outside it and a descriptor, and the test then compares contents, permissions and modification time.
A separate run on the fork, on Linux 7.0 with Landlock ABI 8, covered the same commit's code with parts of the fix taken out. Without the metadata rules, the launch check refuses with `change the canary's permissions: it worked`. Without those rules or the new checks, `a_confined_program_changes_no_file` fails with `permissions of ...: it worked`. With the Landlock rights limited to ABI 2, as on an older kernel, it fails with `truncate ...: it worked`. A kernel that is actually older than ABI 3, or that has no Landlock at all, has not been run. On macOS, terminal input was checked by hand on a pseudo-terminal: `TIOCSTI` is refused when confined and works when not.
The commits group the change by area for reading. Only the last commit of the branch is expected to build.
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Added sandbox support for running commands with policy-limited file access and restrictions on network and process operations. The sandbox is available on supported Linux and macOS systems; launches are refused if confinement cannot be established or verified.
* Added the `sandbox-check` command to check sandbox support and enforcement, or run a specified program in the sandbox. It reports when sandboxing is unsupported.
* **Documentation**
* Added guidance on sandbox behavior, platform support, access restrictions, launch checks, and `sandbox-check` usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* io: refuse a FIFO before opening it; keep the replaced file's group on save
* address review: create the replacement owner-only, then restore group and mode; never widen it when the group cannot be restored
* address review: keep the old owner and other bits when the group cannot be restored
Dylan M. Taylor
FredApps
Valentin KleinandClaude Sonnet 5.5
Pranav