mirror of
https://github.com/storytold/effectcraft.git
synced 2026-11-10 20:41:25 +00:00
Add affected CI and replayable crash repros (#1224)
* AG4: xtask ci --affected cargo xtask ci --affected [--base REV] [--dry-run] runs fmt, clippy, tests (nextest + doctests when available), layers, assets (only on asset changes) and wasm (only for affected L0-L4/web crates) for the packages a change can affect: changed files map to the owning package and to packages whose sources read them via ../ paths, expanded to reverse dependents. Workspace-wide inputs (root Cargo.toml, Cargo.lock, .cargo/, xtask/, toolchain, clippy.toml) fall back to full CI. Documented in docs/testing.md, AGENTS.md and CLAUDE.md. (cherry picked from commit f3ca84fc2ee3948582cbc94e4c99ee510e829540) (cherry picked from commit 8a2cdcadf447eafa57ab14016beae4420f002af0) (cherry picked from commitd8a4301bc1) (cherry picked from commit 36402c61ac2f7e29ecd76a6cfc89365cc2572e0a) * AO1: fast CI profile, looser timing tests - [profile.ci] (release without LTO, 16 codegen units); `xtask ci` runs clippy and tests with --profile ci instead of --release. - Wall-clock asserts in engine memory_tick, gpu OpenGL decline and script socket tests keep their intent with bounds that survive a loaded machine. - docs/testing.md and CONTRIBUTING.md mention the profile. (cherry picked from commit b45c8a379351ebdfda0d7076ad4b740baf6f2087) (cherry picked from commit fa1ea6a107c4c1936cde89b518a05adc73f2b4b0) (cherry picked from commite5d5ed80e9) (cherry picked from commit d2c67660c70764f9c56880a642ad7f0d004a7f92) * AG6: journal replay repros Bug reports become replay scripts and crash regression tests become data files: - engine::replay: a JSON-lines repro format (start line, command steps with expect_error, expect queries with JSON pointer + equals, expect_no_error), a never-panicking runner that reports per-step ok/error and the first failure, and journal export (from the last project open; saves left out; size-capped). - journal.export {path?} and help.exportRepro {path?} (not journaled). - Crash capture: when the last-resort guard catches a panic in the outermost command, the journal plus that command is saved to Repros/crash-*.jsonl in the settings store (newest 20 kept) and the error names the file. - Commands run by a journaled command (script steps) are no longer journaled twice. - effectcraft-cli replay SCRIPT.jsonl [--json] [--keep-going]: exits 1 on a failed step, unmet expectation or panic. - crates/host/tests/repros.rs replays every tests/repros/*.jsonl; three samples from past crash fixes (zero frame rate, Layer Settings with no selection, New Comp from Selection with hostile overlap). - Docs: docs/agents.md "Repro scripts"; AGENTS.md: crash fix regression test = a .jsonl in tests/repros. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c38509d7dafe507856c48c78db65e00b94b0cebc) (cherry picked from commit e55b56dbf76716adfcbfe4cc773df19a74edc419) (cherry picked from commit96a63855a9) (cherry picked from commit c59351ab3bd697f0868e41230dd883960d94a540) * xtask: ci and ci --affected run cargo test and nextest with --no-fail-fast One run now lists every failing crate instead of stopping at the first. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qXL3GPNnY9BSFF2XoU2Yu (cherry picked from commit e46c7debe738eca62b35bf7392ab77637f6a33ae) * Agents: confinement write-checks journal.export, help.exportRepro and file.exportTimeline Review on #859 (split into this PR): their output paths were only read-checked (relative ones not at all). Test: confinement_covers_every_command_that_writes_a_file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014d65YdNxQmjRkbXRtP2NwH * #1224: handle bin-only CI targets and BOM repros without weakening timing tests * #1224: run the full gate when nextest policy changes --------- Co-authored-by: Elias <hakifx91@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: FredApps <FredApps@users.noreply.github.com> Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
This commit is contained in:
20 files changed
+1970
-24
No files matched your search
@@ -865,6 +865,75 @@ not read or written.
|
||||
`<comp>` is an id or name, or `-` for the active comp. `<value>` is JSON (`50`, `[960,540]`,
|
||||
`"#ff0000"`) or a bare string.
|
||||
|
||||
## Repro scripts
|
||||
|
||||
Every journaled command the app runs (menus, panels, shortcuts, agents, scripts) goes into the session's
|
||||
command journal. A **repro script** is that journal as JSON lines, so a bug or crash becomes
|
||||
something an agent can replay and a regression test becomes a data file.
|
||||
|
||||
```text
|
||||
// comments: lines starting with // or #; blank lines are skipped
|
||||
{"start": "new" | "demo" | {"project": "rel/or/abs.ecproj"}, "allow_errors": false} optional, first line
|
||||
{"command": "comp.new", "params": {"name": "Main", "frameRate": 30}}
|
||||
{"command": "comp.settings", "params": {"frameRate": 0}, "expect_error": "frameRate"}
|
||||
{"expect": {"query": "comp.info", "pointer": "/frameRate", "equals": 30}}
|
||||
{"expect_no_error": true}
|
||||
```
|
||||
|
||||
- `start`: the project to begin with (default `new`). A relative `project` path is relative to the
|
||||
script. Other header keys (`version`, `app`, `os`, `crash`, `note`) are for people.
|
||||
- `command`: run like the app runs it. It must succeed, unless it has `expect_error` (`true`, or a
|
||||
text the error must contain) or the header says `allow_errors` (errors are reported but
|
||||
tolerated). A panic always fails the replay.
|
||||
- `expect`: run a command, normally a query (`prop.get`, `comp.info`, `layer.tree`, `project.summary`),
|
||||
and compare the value at the JSON `pointer` of its result with `equals` (numbers within 1e-6).
|
||||
Without `equals` it only checks that the query succeeds.
|
||||
- `expect_no_error`: the previous command succeeded.
|
||||
|
||||
Where scripts come from:
|
||||
|
||||
- `journal.export {path?}` returns `{script, steps}` (or writes `path`): the commands since the
|
||||
project was last replaced (`file.newProject`, `file.open`, `file.openDemoProject`). Queries and
|
||||
saves are not in it; commands a script or `file.runScript` ran are covered by that one step.
|
||||
A successful Revert starts a new replay boundary at the saved file it reopened.
|
||||
- `help.exportRepro {path?}` saves the same for a bug report, by default to `Repros/` in the
|
||||
settings folder (macOS `~/Library/Application Support/EffectCraft/Repros`, next to `Logs/`).
|
||||
Confined automation must give an explicit path inside a write root; `journal.export {}`
|
||||
remains available because it only returns the script in memory.
|
||||
- **Crash repros**: when the last-resort guard catches a panic in a command, the app writes
|
||||
`Repros/crash-<unix time>-<command>.jsonl` there automatically (the newest 20 are kept): the
|
||||
journal, then the command that panicked, with `"allow_errors": true` and the error in the header.
|
||||
The error the user or agent sees names the file. Headless sessions without a settings folder
|
||||
(the CLI, tests) don't write one.
|
||||
|
||||
Privacy: a repro holds command ids and parameters only, never project, image or media file
|
||||
contents. Parameters can include file paths, layer and item names and text typed into text
|
||||
layers, so read a repro before sharing it.
|
||||
|
||||
Replay one headless (a fully wired session, like the CLI's other subcommands):
|
||||
|
||||
```sh
|
||||
effectcraft-cli replay crash.jsonl # per-step ok/FAIL lines, then the first failure
|
||||
effectcraft-cli replay crash.jsonl --json # {ok, panicked, steps, first_failure, skipped}
|
||||
effectcraft-cli replay crash.jsonl --keep-going
|
||||
```
|
||||
|
||||
Replaying a repro runs its commands with your privileges and file access, including commands
|
||||
that write files or run scripts. Read repros from untrusted sources before running them, or use
|
||||
a disposable environment.
|
||||
|
||||
It exits with status 1 when a step fails, an expectation is not met or a command panics, and
|
||||
stops at the first failure unless `--keep-going`. Steps that work on the user's files (`file.open`
|
||||
of their project, imports) need those files; replace them with a `start` project or generated
|
||||
content when you turn a report into a test. A repro only sees commands: UI state changed
|
||||
without one (a hover, a half-finished drag) is not in it.
|
||||
|
||||
**Regression tests are data files.** A crash fix's regression test is the repro, trimmed to the
|
||||
steps that matter, with an `expect_error` or `expect` line where the fix changed the outcome,
|
||||
dropped into [`tests/repros/`](../tests/repros). `crates/host/tests/it/repros.rs` replays every
|
||||
`*.jsonl` there and fails on any panic or unmet expectation
|
||||
(`cargo nextest run -p effectcraft-host --test it -E 'test(/^repros::/)'`).
|
||||
|
||||
## Seeing the UI
|
||||
|
||||
To work on the UI, start the app with `cargo run -p effectcraft -- --control 9877` (add `--demo` to
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
```sh
|
||||
cargo test --workspace # everything
|
||||
cargo xtask ci # fmt, clippy -D warnings, tests, pdf-deps, lock-patches, layers, assets, wasm
|
||||
cargo xtask ci --affected # the same gates, scoped to crates your change can affect
|
||||
```
|
||||
|
||||
The After Effects test projects of `crates/aep` are not in the repository; without them the
|
||||
@@ -32,6 +33,47 @@ Each crate with several integration-test files has one test binary, `tests/it/ma
|
||||
modules are the files in `tests/it/`. Run one file with `cargo test -p <crate> --test it <module>::`
|
||||
or `cargo nextest run -p <crate> -E 'binary(it) and test(/^<module>::/)'`.
|
||||
|
||||
## Affected-only CI
|
||||
|
||||
Use `cargo xtask ci --affected` to run the gates for the crates a change can affect. The full
|
||||
`cargo xtask ci` remains the required check before a release and when affected mode falls back
|
||||
because a workspace-wide input changed.
|
||||
|
||||
```sh
|
||||
cargo xtask ci --affected # base: the merge-base of HEAD with main
|
||||
cargo xtask ci --affected --base HEAD~3 # any git revision
|
||||
cargo xtask ci --affected --dry-run # print the plan only
|
||||
```
|
||||
|
||||
It collects files that differ from the base (committed, staged, unstaged and untracked), preserving
|
||||
Unicode and whitespace in their names. Each file maps to the workspace package whose directory
|
||||
holds it and every package whose sources point at it with a `../` literal, such as
|
||||
`include_bytes!("../../../assets/fonts/Inter-Regular.ttf")` or
|
||||
`concat!(env!("CARGO_MANIFEST_DIR"), "/../../docs/preferences.md")`. Deleted and renamed inputs
|
||||
still select their readers, including when the input's entire directory no longer exists.
|
||||
|
||||
The affected set is those packages plus everything that depends on them (transitively through
|
||||
normal and build dependencies) and every package that uses one of them as a dev-dependency.
|
||||
Clippy and tests use the same dev profile as full CI. Both modes use the same test runner:
|
||||
nextest plus doctests when installed, otherwise cargo test plus doctests. GPU tests run in one
|
||||
libtest process, and the translation report reruns with visible output when ui-egui is affected.
|
||||
The audio-backend probe runs before building affected crates.
|
||||
|
||||
Formatting, PDF dependency policy, lock-patch and layer checks always run. Assets are checked
|
||||
when an asset, an `.attribution` sidecar or `ATTRIBUTION.md` changes. The wasm check selects
|
||||
affected L0–L4 and web crates. Every step runs even when an earlier step fails; the final result
|
||||
names failed steps and includes timings.
|
||||
|
||||
Changes to the root `Cargo.toml`, `Cargo.lock`, `.cargo/`, `xtask/`, `rust-toolchain(.toml)`,
|
||||
`clippy.toml` or `.config/nextest.toml` can change every crate's build or test policy, so they run
|
||||
the full CI instead and say why. Changes no crate reads (most of `docs/`, `README.md`, `.github/`,
|
||||
`packaging/`) only get the cheap checks.
|
||||
|
||||
Limits: a crate that reads a repo file at run time through a path that is not a `../` literal in its
|
||||
sources is not detected, and building a subset of packages can unify features differently from the
|
||||
whole workspace. Non-UTF-8 Git paths stop the command rather than silently omitting checks.
|
||||
Run the full `cargo xtask ci` before a release.
|
||||
|
||||
## What the tests cover
|
||||
|
||||
- **Engine commands** (`crates/engine/src/tests*.rs`): every command family, including undo and
|
||||
|
||||
Reference in new issue
Block a user