Add affected CI and replayable crash repros (#1224)

* AG4: xtask ci --affected

cargo xtask ci --affected [--base REV] [--dry-run] runs fmt, clippy, tests (nextest + doctests when available), layers, assets (only on asset changes) and wasm (only for affected L0-L4/web crates) for the packages a change can affect: changed files map to the owning package and to packages whose sources read them via ../ paths, expanded to reverse dependents. Workspace-wide inputs (root Cargo.toml, Cargo.lock, .cargo/, xtask/, toolchain, clippy.toml) fall back to full CI. Documented in docs/testing.md, AGENTS.md and CLAUDE.md.

(cherry picked from commit f3ca84fc2ee3948582cbc94e4c99ee510e829540)

(cherry picked from commit 8a2cdcadf447eafa57ab14016beae4420f002af0)
(cherry picked from commit d8a4301bc1)

(cherry picked from commit 36402c61ac2f7e29ecd76a6cfc89365cc2572e0a)

* AO1: fast CI profile, looser timing tests

- [profile.ci] (release without LTO, 16 codegen units); `xtask ci` runs clippy
  and tests with --profile ci instead of --release.
- Wall-clock asserts in engine memory_tick, gpu OpenGL decline and script
  socket tests keep their intent with bounds that survive a loaded machine.
- docs/testing.md and CONTRIBUTING.md mention the profile.

(cherry picked from commit b45c8a379351ebdfda0d7076ad4b740baf6f2087)

(cherry picked from commit fa1ea6a107c4c1936cde89b518a05adc73f2b4b0)
(cherry picked from commit e5d5ed80e9)

(cherry picked from commit d2c67660c70764f9c56880a642ad7f0d004a7f92)

* AG6: journal replay repros

Bug reports become replay scripts and crash regression tests become data files:
- engine::replay: a JSON-lines repro format (start line, command steps with
  expect_error, expect queries with JSON pointer + equals, expect_no_error),
  a never-panicking runner that reports per-step ok/error and the first failure,
  and journal export (from the last project open; saves left out; size-capped).
- journal.export {path?} and help.exportRepro {path?} (not journaled).
- Crash capture: when the last-resort guard catches a panic in the outermost
  command, the journal plus that command is saved to Repros/crash-*.jsonl in the
  settings store (newest 20 kept) and the error names the file.
- Commands run by a journaled command (script steps) are no longer journaled twice.
- effectcraft-cli replay SCRIPT.jsonl [--json] [--keep-going]: exits 1 on a failed
  step, unmet expectation or panic.
- crates/host/tests/repros.rs replays every tests/repros/*.jsonl; three samples
  from past crash fixes (zero frame rate, Layer Settings with no selection,
  New Comp from Selection with hostile overlap).
- Docs: docs/agents.md "Repro scripts"; AGENTS.md: crash fix regression test =
  a .jsonl in tests/repros.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

(cherry picked from commit c38509d7dafe507856c48c78db65e00b94b0cebc)

(cherry picked from commit e55b56dbf76716adfcbfe4cc773df19a74edc419)
(cherry picked from commit 96a63855a9)

(cherry picked from commit c59351ab3bd697f0868e41230dd883960d94a540)

* xtask: ci and ci --affected run cargo test and nextest with --no-fail-fast

One run now lists every failing crate instead of stopping at the first.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qXL3GPNnY9BSFF2XoU2Yu

(cherry picked from commit e46c7debe738eca62b35bf7392ab77637f6a33ae)

* Agents: confinement write-checks journal.export, help.exportRepro and file.exportTimeline

Review on #859 (split into this PR): their output paths were only read-checked (relative ones
not at all). Test: confinement_covers_every_command_that_writes_a_file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014d65YdNxQmjRkbXRtP2NwH

* #1224: handle bin-only CI targets and BOM repros without weakening timing tests

* #1224: run the full gate when nextest policy changes

---------

Co-authored-by: Elias <hakifx91@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: FredApps <FredApps@users.noreply.github.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-10 14:57:32 +01:00
1 parent f88e7980ae
commit 13470ca161
20 files changed
+1970 -24

No files matched your search

+69
View File
@@ -865,6 +865,75 @@ not read or written.
`<comp>` is an id or name, or `-` for the active comp. `<value>` is JSON (`50`, `[960,540]`,
`"#ff0000"`) or a bare string.
## Repro scripts
Every journaled command the app runs (menus, panels, shortcuts, agents, scripts) goes into the session's
command journal. A **repro script** is that journal as JSON lines, so a bug or crash becomes
something an agent can replay and a regression test becomes a data file.
```text
// comments: lines starting with // or #; blank lines are skipped
{"start": "new" | "demo" | {"project": "rel/or/abs.ecproj"}, "allow_errors": false} optional, first line
{"command": "comp.new", "params": {"name": "Main", "frameRate": 30}}
{"command": "comp.settings", "params": {"frameRate": 0}, "expect_error": "frameRate"}
{"expect": {"query": "comp.info", "pointer": "/frameRate", "equals": 30}}
{"expect_no_error": true}
```
- `start`: the project to begin with (default `new`). A relative `project` path is relative to the
script. Other header keys (`version`, `app`, `os`, `crash`, `note`) are for people.
- `command`: run like the app runs it. It must succeed, unless it has `expect_error` (`true`, or a
text the error must contain) or the header says `allow_errors` (errors are reported but
tolerated). A panic always fails the replay.
- `expect`: run a command, normally a query (`prop.get`, `comp.info`, `layer.tree`, `project.summary`),
and compare the value at the JSON `pointer` of its result with `equals` (numbers within 1e-6).
Without `equals` it only checks that the query succeeds.
- `expect_no_error`: the previous command succeeded.
Where scripts come from:
- `journal.export {path?}` returns `{script, steps}` (or writes `path`): the commands since the
project was last replaced (`file.newProject`, `file.open`, `file.openDemoProject`). Queries and
saves are not in it; commands a script or `file.runScript` ran are covered by that one step.
A successful Revert starts a new replay boundary at the saved file it reopened.
- `help.exportRepro {path?}` saves the same for a bug report, by default to `Repros/` in the
settings folder (macOS `~/Library/Application Support/EffectCraft/Repros`, next to `Logs/`).
Confined automation must give an explicit path inside a write root; `journal.export {}`
remains available because it only returns the script in memory.
- **Crash repros**: when the last-resort guard catches a panic in a command, the app writes
`Repros/crash-<unix time>-<command>.jsonl` there automatically (the newest 20 are kept): the
journal, then the command that panicked, with `"allow_errors": true` and the error in the header.
The error the user or agent sees names the file. Headless sessions without a settings folder
(the CLI, tests) don't write one.
Privacy: a repro holds command ids and parameters only, never project, image or media file
contents. Parameters can include file paths, layer and item names and text typed into text
layers, so read a repro before sharing it.
Replay one headless (a fully wired session, like the CLI's other subcommands):
```sh
effectcraft-cli replay crash.jsonl # per-step ok/FAIL lines, then the first failure
effectcraft-cli replay crash.jsonl --json # {ok, panicked, steps, first_failure, skipped}
effectcraft-cli replay crash.jsonl --keep-going
```
Replaying a repro runs its commands with your privileges and file access, including commands
that write files or run scripts. Read repros from untrusted sources before running them, or use
a disposable environment.
It exits with status 1 when a step fails, an expectation is not met or a command panics, and
stops at the first failure unless `--keep-going`. Steps that work on the user's files (`file.open`
of their project, imports) need those files; replace them with a `start` project or generated
content when you turn a report into a test. A repro only sees commands: UI state changed
without one (a hover, a half-finished drag) is not in it.
**Regression tests are data files.** A crash fix's regression test is the repro, trimmed to the
steps that matter, with an `expect_error` or `expect` line where the fix changed the outcome,
dropped into [`tests/repros/`](../tests/repros). `crates/host/tests/it/repros.rs` replays every
`*.jsonl` there and fails on any panic or unmet expectation
(`cargo nextest run -p effectcraft-host --test it -E 'test(/^repros::/)'`).
## Seeing the UI
To work on the UI, start the app with `cargo run -p effectcraft -- --control 9877` (add `--demo` to
+42
View File
@@ -3,6 +3,7 @@
```sh
cargo test --workspace # everything
cargo xtask ci # fmt, clippy -D warnings, tests, pdf-deps, lock-patches, layers, assets, wasm
cargo xtask ci --affected # the same gates, scoped to crates your change can affect
```
The After Effects test projects of `crates/aep` are not in the repository; without them the
@@ -32,6 +33,47 @@ Each crate with several integration-test files has one test binary, `tests/it/ma
modules are the files in `tests/it/`. Run one file with `cargo test -p <crate> --test it <module>::`
or `cargo nextest run -p <crate> -E 'binary(it) and test(/^<module>::/)'`.
## Affected-only CI
Use `cargo xtask ci --affected` to run the gates for the crates a change can affect. The full
`cargo xtask ci` remains the required check before a release and when affected mode falls back
because a workspace-wide input changed.
```sh
cargo xtask ci --affected # base: the merge-base of HEAD with main
cargo xtask ci --affected --base HEAD~3 # any git revision
cargo xtask ci --affected --dry-run # print the plan only
```
It collects files that differ from the base (committed, staged, unstaged and untracked), preserving
Unicode and whitespace in their names. Each file maps to the workspace package whose directory
holds it and every package whose sources point at it with a `../` literal, such as
`include_bytes!("../../../assets/fonts/Inter-Regular.ttf")` or
`concat!(env!("CARGO_MANIFEST_DIR"), "/../../docs/preferences.md")`. Deleted and renamed inputs
still select their readers, including when the input's entire directory no longer exists.
The affected set is those packages plus everything that depends on them (transitively through
normal and build dependencies) and every package that uses one of them as a dev-dependency.
Clippy and tests use the same dev profile as full CI. Both modes use the same test runner:
nextest plus doctests when installed, otherwise cargo test plus doctests. GPU tests run in one
libtest process, and the translation report reruns with visible output when ui-egui is affected.
The audio-backend probe runs before building affected crates.
Formatting, PDF dependency policy, lock-patch and layer checks always run. Assets are checked
when an asset, an `.attribution` sidecar or `ATTRIBUTION.md` changes. The wasm check selects
affected L0–L4 and web crates. Every step runs even when an earlier step fails; the final result
names failed steps and includes timings.
Changes to the root `Cargo.toml`, `Cargo.lock`, `.cargo/`, `xtask/`, `rust-toolchain(.toml)`,
`clippy.toml` or `.config/nextest.toml` can change every crate's build or test policy, so they run
the full CI instead and say why. Changes no crate reads (most of `docs/`, `README.md`, `.github/`,
`packaging/`) only get the cheap checks.
Limits: a crate that reads a repo file at run time through a path that is not a `../` literal in its
sources is not detected, and building a subset of packages can unify features differently from the
whole workspace. Non-UTF-8 Git paths stop the command rather than silently omitting checks.
Run the full `cargo xtask ci` before a release.
## What the tests cover
- **Engine commands** (`crates/engine/src/tests*.rs`): every command family, including undo and