Files
effectcraft/xtask/src/main.rs
T
611a282bea Plug-ins: external effect helpers (safe, out-of-process native effects) + plug-in API 2 (#889)
* Plug-ins: API 2 (PluginHost, padding, layer_span)

Split out of the OpenFX work so it can be reviewed on its own. Additive; the WebAssembly ABI
stays at 1.

- point3 / layer / text / hidden parameters, integer sliders, parameter groups.
- EffectPlugin::padding (transparent pixels around the buffer, at most 4096 per side).
- EffectPlugin::render_v2 with a PluginHost: the input at other times, other layers, parameters
  at other times, frame rate, per-instance key, the layer's source range.
- EffectEnv::layer_span from the renderer; MAX_PLUGINS 1024.
- plugin_error: a plug-in effect's last render error, for the UI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxacwTpUwqyfpcLgKW8yjS

* Plug-ins: external effect helpers (crates/helper-host)

Native plug-in formats run in separate helper programs, never in EffectCraft's process
(docs/adr/0001-external-effect-helpers.md). New safe-Rust crate effectcraft-helper-host (L3,
native only, unsafe forbidden): it finds helper programs, starts each as a child process and
registers its effects through plug-in API 2, so the UI and command surface need nothing new.

- Protocol v1 (docs/external-helpers.md): JSON lines over stdin / stdout (helper replies are
  prefixed so plug-in chatter is ignored); pixels go through a frame exchange file (/dev/shm on
  Linux), never through the pipe. Render-time callbacks for the input at other times, other layers
  and parameters at other times, with nested renders through the same helper.
- A helper that exits, hangs past its timeout or breaks the protocol is stopped. The render fails
  with a message, the frame is left as it was, and the helper restarts for the next render. An
  effect that stops its helper 3 times is turned off (helper_blocklist.txt).
- Opt-in: nothing starts helpers unless the user asks (Effect > Scan for Effect Helpers), or opts
  in to starting them at launch with EFFECTCRAFT_HELPERS=1, an environment variable, so writing
  to the settings folder alone can't run code. effect.helpers.scan / load are refused for
  confined agent sessions (confine.rs), like file.executeFile.
- Discovery: helpers/ next to the app (Contents/Helpers on macOS), helpers/ in the settings
  folder, EFFECTCRAFT_HELPER_PATH.
- Commands effect.helpers.list / scan / load; Effect menu entries (translated); the last render
  error of an effect shows as a warning sign after its name in Effect Controls.
- serve(): the helper side for Rust helpers. tests/it/main.rs runs the whole protocol (render,
  padding, callbacks, nesting, threads, crash, restart, block list, hang) against a fake helper,
  with nothing to install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxacwTpUwqyfpcLgKW8yjS

* Effect helpers: address review (start failures, frame file mode, nextest listing)

- A helper that can't start (not installed any more, restarting too often, no frame exchange
  file) is reported but no longer charged to the effect being rendered: it is Failure::Refused,
  not Failure::Broken, so it never counts toward turning that effect off. New check in
  tests/it: once the helper is out of restarts, renders of an effect fail without blocking it
  (fails without the fix: the effect was turned off after 3 such renders).
- The frame exchange file is created owner-only (0o600) on Unix: it holds the user's frames and
  /dev/shm is shared. Asserted in the frames unit test.
- tests/it (harness = false) speaks the libtest listing protocol cargo-nextest uses:
  `--list --format terse` prints `helper_protocol: test` (nothing with `--ignored`), and
  `helper_protocol --exact` runs the checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KxacwTpUwqyfpcLgKW8yjS

* Effect helpers: bound writes to a helper, drain stderr, keep callbacks in step

- Writes to a helper go through a writer thread with the request timeout: a
  helper that stops reading its stdin (hung with a large callback answer or
  render request pending) no longer blocks the render thread forever while
  holding the helper.
- stderr is drained as bytes (lossy, in 64 KiB pieces) to the end: output that
  isn't UTF-8 stopped the reader, closing the pipe under the helper (EPIPE /
  SIGPIPE on its next write); lines were unbounded.
- Callback images that can't be written are answered with `null` instead of
  failing the render while the helper waits (which left the protocol out of
  step); a frame that can't be written before the request is sent is not
  charged to the effect.
- Mid-render calls use the render's own process: no respawn or heartbeat ping
  while the helper is waiting for an answer.
- Waiting for a busy helper is bounded (load timeout) instead of forever;
  status() no longer blocks on a running render.
- Reap the child when start-up fails after spawning; cap the block list read.
- Scripts start helpers only with Allow Scripts to Write Files and Access
  Network on (they run native code), as for File.execute().
- helper-host declares the workspace's no-panic crate lints; the desktop
  main() keeps its doc comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Effect helpers: rustfmt the fake provider, build the test session in its initializer (clippy)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Plug-in Library: offer WebAssembly API 1 releases only; a helper already stopped mid-render isn't charged again

- WasmPlugin loads only modules at the WebAssembly ABI version, which stays 1 while
  PLUGIN_API_VERSION is 2: share it as effects::plugin::WASM_API_VERSION and have the
  library's compatibility check use it, so it never installs an API 2 release that
  can't load.
- with_running: a process that is already gone was stopped by what broke it (a nested
  render, charged there), so this render fails without charging its own effect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* frames: plug-in effects count as reading other frames (plug-in API 2)

With main's narrowed frame invalidation (#2795), an edit inside a precomp under an
effect that reads its layer at other times must reach the layer's whole span.
Plug-in API 2 gives every plug-in effect the input at other times, so any effect
outside ec. counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
2026-11-05 11:11:57 +01:00

605 lines
26 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Workspace automation: `cargo xtask <layers|assets|pdf-deps|aep-corpus|wasm|web|ico|ci>`.
//!
//! - `layers`: enforces the dependency layering of `docs/architecture.md` §1 (downward-only edges,
//! listed same-layer edges, no UI/OS crates below L5).
//! - `wasm`: `cargo check --target wasm32-unknown-unknown` for every crate in L0–L4, the egui UI
//! and the web app.
//! - `web [--dev] [--serve PORT]`: build the web app (`apps/effectcraft-web`) into
//! `<target>/web/dist` with `wasm-bindgen` (docs/web.md); `--serve` serves it on localhost.
//! - `assets`: every asset file (image, icon, font, LUT, audio, video…) has a complete
//! `<file>.attribution` sidecar and an entry in `ATTRIBUTION.md` (AGENTS.md §1).
//! - `aep-corpus [--dir DIR]`: fetch the After Effects test corpus at the commit pinned in
//! `crates/aep/tests/fixtures/corpus.lock`, verify every file's SHA-256, install into `corpus/aep`.
//! - `ico <out.ico> <in.png>…`: pack PNGs into a Windows `.ico` (used by `packaging/icons.sh`).
//! - `ci`: fmt check, clippy -D warnings, tests (`cargo nextest run` when installed) and doctests,
//! PDF dependency policy, layers, assets, wasm. Everything builds in the dev profile.
//! - `bench [--bin CLI] [--quick] [--compare OLD.jsonl] …`: render benchmark of generated scenarios
//! (memory, CPU, wall time) into `target/bench/results.jsonl` (`bench.rs`, docs/testing.md).
mod aep_corpus;
mod bench;
mod ico;
mod pdf_deps;
mod version;
use std::process::{Command, ExitCode};
use serde_json::Value;
/// (crate name without the `effectcraft-` prefix, layer). See `docs/architecture.md` §1.
const LAYERS: &[(&str, u8)] = &[
("time", 0),
("geom", 0),
("color", 0),
("vp9enc", 0),
("testkit", 0),
("opusenc", 0),
("hevcenc", 0),
("av1enc", 0),
("raster", 1),
("keyframe", 1),
("path", 1),
("psd", 1),
("aep", 3),
("segment", 1),
("project", 2),
("text", 2),
("effects", 2),
("track", 2),
("svg", 2),
("pdf", 2),
("model", 2),
("render", 3),
("media", 3),
("expr", 3),
("export", 3),
("gpu", 3),
("sam2", 3),
("lottie", 3),
("format", 3),
("plugin", 3),
("helper-host", 3),
("interchange", 3),
("engine", 4),
("script", 4),
("host", 4),
("ui-egui", 5),
("automation", 5),
("effectcraft", 6),
("cli", 6),
("web", 6),
];
/// Allowed same-layer edges (from, to).
const SAME_LAYER: &[(&str, &str)] = &[
("path", "keyframe"),
("path", "raster"),
("text", "path"),
("pdf", "svg"),
("pdf", "text"),
("effects", "project"),
("effects", "text"),
("effects", "path"),
("effects", "track"),
("media", "render"),
("expr", "render"),
("export", "render"),
("export", "media"),
("gpu", "render"),
("lottie", "format"),
("host", "engine"),
("script", "engine"),
("host", "script"),
("cli", "effectcraft"),
];
/// Crates that must not appear below L5 (UI toolkits, windowing, OS audio/menus).
const UI_ONLY: &[&str] = &["craft-ui", "egui", "eframe", "egui-wgpu", "winit", "rfd", "cpal", "muda"];
fn short(name: &str) -> &str {
name.strip_prefix("effectcraft-").unwrap_or(name)
}
fn layer_of(name: &str) -> Option<u8> {
LAYERS.iter().find(|(n, _)| *n == short(name)).map(|(_, l)| *l)
}
fn metadata() -> Result<Value, String> {
let out = Command::new(env!("CARGO")).args(["metadata", "--format-version", "1", "--no-deps"]).output().map_err(|e| e.to_string())?;
if !out.status.success() {
return Err(String::from_utf8_lossy(&out.stderr).into());
}
serde_json::from_slice(&out.stdout).map_err(|e| e.to_string())
}
fn workspace_crates(md: &Value) -> Vec<(String, Vec<String>)> {
md["packages"]
.as_array()
.into_iter()
.flatten()
.filter(|p| p["name"] != "xtask")
.map(|p| {
let deps = p["dependencies"]
.as_array()
.into_iter()
.flatten()
.filter(|d| d["kind"].is_null() || d["kind"] == "build")
.filter_map(|d| d["name"].as_str().map(str::to_string))
.collect();
(p["name"].as_str().unwrap_or_default().to_string(), deps)
})
.collect()
}
fn layers() -> Result<(), String> {
let md = metadata()?;
let mut errors = Vec::new();
for (name, deps) in workspace_crates(&md) {
let Some(l) = layer_of(&name) else {
errors.push(format!("{name}: not assigned a layer (add it to xtask LAYERS and docs/architecture.md §1)"));
continue;
};
for d in &deps {
if l < 5 && UI_ONLY.contains(&d.as_str()) {
errors.push(format!("{name} (L{l}) depends on UI/OS crate `{d}`"));
}
if !d.starts_with("effectcraft-") {
continue;
}
let Some(dl) = layer_of(d) else { continue };
let (a, b) = (short(&name), short(d));
if dl > l {
errors.push(format!("{a} (L{l}) depends upward on {b} (L{dl})"));
} else if dl == l && l > 0 && !SAME_LAYER.contains(&(a, b)) {
errors.push(format!("{a} → {b}: same-layer edge (L{l}) not in the allowed list"));
}
}
}
if errors.is_empty() {
println!("layers: ok");
Ok(())
} else {
Err(errors.join("\n"))
}
}
/// File extensions that count as assets (AGENTS.md §1).
const ASSET_EXT: &[&str] = &[
"png", "jpg", "jpeg", "gif", "webp", "svg", "ico", "icns", "bmp", "tif", "tiff", "heic", "avif", "exr", "ttf", "otf", "ttc", "woff", "woff2", "cube",
"3dl", "lut", "look", "wav", "mp3", "aac", "flac", "ogg", "opus", "m4a", "aif", "aiff", "mp4", "mov", "m4v", "mkv", "webm", "avi", "mxf", "psd", "ai",
"eps", "pdf", "prproj", "ffx", "prfpset", "mogrt", "aep", "aepx",
];
/// Sidecar fields that must be present and non-empty.
const REQUIRED_FIELDS: &[&str] = &["asset", "title", "author", "source", "license", "added"];
fn repo_files() -> Result<Vec<String>, String> {
let out = Command::new("git").args(["ls-files", "--cached", "--others", "--exclude-standard"]).output().map_err(|e| e.to_string())?;
if !out.status.success() {
return Err(String::from_utf8_lossy(&out.stderr).into());
}
Ok(String::from_utf8_lossy(&out.stdout).lines().map(str::to_string).filter(|f| std::path::Path::new(f).exists()).collect())
}
fn is_asset(path: &str) -> bool {
std::path::Path::new(path).extension().and_then(std::ffi::OsStr::to_str).is_some_and(|e| ASSET_EXT.contains(&e.to_ascii_lowercase().as_str()))
}
fn assets() -> Result<(), String> {
let files = repo_files()?;
let index = std::fs::read_to_string("ATTRIBUTION.md").map_err(|e| format!("ATTRIBUTION.md: {e}"))?;
let mut errors = Vec::new();
let mut n = 0;
for f in files.iter().filter(|f| is_asset(f)) {
n += 1;
let lower = f.to_ascii_lowercase();
if lower.contains("adobe") || lower.contains("aftereffects") || lower.contains("after-effects") {
errors.push(format!("{f}: asset paths must not reference Adobe/After Effects (AGENTS.md §1)"));
}
let side = format!("{f}.attribution");
match std::fs::read_to_string(&side) {
Err(_) => errors.push(format!("{f}: missing attribution sidecar {side}")),
Ok(text) => {
for field in REQUIRED_FIELDS {
let ok = text.lines().any(|l| l.split_once(':').is_some_and(|(k, v)| k.trim() == *field && !v.trim().is_empty()));
if !ok {
errors.push(format!("{side}: field `{field}` missing or empty"));
}
}
let lic = text.lines().find_map(|l| l.split_once(':').filter(|(k, _)| k.trim() == "license").map(|(_, v)| v.trim().to_ascii_lowercase()));
if lic.is_some_and(|l| l.contains("-nc") || l.contains("-nd") || l.contains("adobe") || l.contains("proprietary")) {
errors.push(format!("{side}: licence not allowed (no NC/ND, Adobe or proprietary licences)"));
}
}
}
if !index.contains(&format!("`{f}`")) {
errors.push(format!("{f}: not listed in ATTRIBUTION.md"));
}
}
for f in files.iter().filter(|f| f.ends_with(".attribution")) {
let asset = f.trim_end_matches(".attribution");
if !std::path::Path::new(asset).exists() {
errors.push(format!("{f}: sidecar for a file that does not exist"));
}
}
if errors.is_empty() {
println!("assets: ok ({n} assets attributed)");
Ok(())
} else {
Err(errors.join("\n"))
}
}
fn run(cmd: &mut Command) -> Result<(), String> {
eprintln!("$ {cmd:?}");
let st = cmd.status().map_err(|e| e.to_string())?;
if st.success() { Ok(()) } else { Err(format!("failed: {cmd:?}")) }
}
/// Crates above L4 that must also build for the web.
const WEB_CRATES: &[&str] = &["effectcraft-ui-egui", "effectcraft-web"];
fn wasm() -> Result<(), String> {
let md = metadata()?;
let mut cmd = Command::new(env!("CARGO"));
cmd.args(["check", "--target", "wasm32-unknown-unknown", "--keep-going"]);
let mut n = 0;
for (name, _) in workspace_crates(&md) {
if layer_of(&name).is_some_and(|l| l <= 4) || WEB_CRATES.contains(&name.as_str()) {
cmd.args(["-p", &name]);
n += 1;
}
}
if n == 0 {
return Ok(());
}
run(&mut cmd)?;
println!("wasm: ok ({n} crates)");
Ok(())
}
/// The wasm-bindgen CLI must match the `wasm-bindgen` crate version exactly.
const WASM_BINDGEN: &str = "0.2.129";
fn target_dir() -> std::path::PathBuf {
std::env::var_os("CARGO_TARGET_DIR").map(std::path::PathBuf::from).unwrap_or_else(|| std::path::PathBuf::from("target"))
}
/// Build the web app into `<target>/web/dist` (see docs/web.md).
fn web(args: &[String]) -> Result<(), String> {
let dev = args.iter().any(|a| a == "--dev");
let serve = args.iter().position(|a| a == "--serve").map(|i| args.get(i + 1).and_then(|p| p.parse::<u16>().ok()).unwrap_or(8765));
let profile = if dev { "dev" } else { "release" };
let mut build = Command::new(env!("CARGO"));
build.args(["build", "--target", "wasm32-unknown-unknown", "-p", "effectcraft-web", "--profile", profile]);
run(&mut build)?;
let out = Command::new("wasm-bindgen")
.arg("--version")
.output()
.map_err(|_| format!("wasm-bindgen CLI not found: cargo install wasm-bindgen-cli --version {WASM_BINDGEN} --locked"))?;
let v = String::from_utf8_lossy(&out.stdout);
if !v.contains(WASM_BINDGEN) {
return Err(format!(
"wasm-bindgen CLI is {} but the crate is {WASM_BINDGEN}: cargo install wasm-bindgen-cli --version {WASM_BINDGEN} --locked",
v.trim()
));
}
let dir = if dev { "debug" } else { "release" };
let wasm = target_dir().join("wasm32-unknown-unknown").join(dir).join("effectcraft_web.wasm");
let dist = target_dir().join("web").join("dist");
let _ = std::fs::remove_dir_all(&dist);
std::fs::create_dir_all(&dist).map_err(|e| e.to_string())?;
run(Command::new("wasm-bindgen").args(["--target", "web", "--no-typescript", "--out-dir"]).arg(&dist).arg(&wasm))?;
let bg = dist.join("effectcraft_web_bg.wasm");
if !dev && Command::new("wasm-opt").arg("--version").output().is_ok() {
// optional: smaller and faster (binaryen); skipped when not installed
let opt = dist.join("effectcraft_web_opt.wasm");
run(Command::new("wasm-opt")
.args(["-O2", "--enable-bulk-memory", "--enable-nontrapping-float-to-int", "--enable-sign-ext", "--enable-mutable-globals"])
.arg(&bg)
.arg("-o")
.arg(&opt))?;
std::fs::rename(&opt, &bg).map_err(|e| e.to_string())?;
}
let web = std::path::Path::new("apps/effectcraft-web/web");
for e in std::fs::read_dir(web).map_err(|e| e.to_string())?.flatten() {
let name = e.file_name().to_string_lossy().to_string();
if !name.ends_with(".attribution") {
std::fs::copy(e.path(), dist.join(&name)).map_err(|e| format!("{name}: {e}"))?;
}
}
// PWA icons: the app icon (assets/app-icon, attributed there).
for n in [256, 512] {
let src = format!("assets/app-icon/hicolor/{n}x{n}/apps/ai.storyteller.effectcraft.png");
std::fs::copy(&src, dist.join(format!("icon-{n}.png"))).map_err(|e| format!("{src}: {e}"))?;
}
service_worker(&dist)?;
let size = |p: &std::path::Path| std::fs::metadata(p).map(|m| m.len()).unwrap_or(0);
let total: u64 = dist_files(&dist).iter().map(|f| size(&dist.join(f))).sum();
println!("web: {} ({:.1} MB wasm, {:.1} MB total)", dist.display(), size(&bg) as f64 / 1e6, total as f64 / 1e6);
if let Some(port) = serve {
serve_dir(&dist, port)?;
}
Ok(())
}
/// Every file under `dist`, as `/`-separated relative paths, sorted.
fn dist_files(dist: &std::path::Path) -> Vec<String> {
fn walk(dir: &std::path::Path, rel: &str, out: &mut Vec<String>) {
for e in std::fs::read_dir(dir).into_iter().flatten().flatten() {
let name = e.file_name().to_string_lossy().to_string();
let r = if rel.is_empty() { name } else { format!("{rel}/{name}") };
if e.path().is_dir() {
walk(&e.path(), &r, out);
} else {
out.push(r);
}
}
}
let mut v = vec![];
walk(dist, "", &mut v);
v.sort();
v
}
/// Fill the service worker's precache list and version (a hash of the build's files).
fn service_worker(dist: &std::path::Path) -> Result<(), String> {
let sw = dist.join("sw.js");
let text = std::fs::read_to_string(&sw).map_err(|e| format!("sw.js: {e}"))?;
let files: Vec<String> = dist_files(dist).into_iter().filter(|f| f != "sw.js").collect();
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
for f in &files {
for b in f.bytes().chain(std::fs::read(dist.join(f)).map_err(|e| format!("{f}: {e}"))?) {
h ^= b as u64;
h = h.wrapping_mul(0x0100_0000_01b3);
}
}
let list: Vec<String> = std::iter::once("./".to_string()).chain(files.iter().map(|f| format!("./{f}"))).map(|f| format!("{f:?}")).collect();
let text = text.replace("__EC_VERSION__", &format!("{h:016x}")).replace("__EC_FILES__", &format!("[{}]", list.join(", ")));
std::fs::write(&sw, text).map_err(|e| format!("sw.js: {e}"))
}
/// A tiny static file server for the web build (localhost only). Sends the cross-origin
/// isolation headers so `crossOriginIsolated` is true, like a production deployment would.
fn serve_dir(dir: &std::path::Path, port: u16) -> Result<(), String> {
use std::io::{BufRead, BufReader, Write};
let listener = std::net::TcpListener::bind(("127.0.0.1", port)).map_err(|e| format!("127.0.0.1:{port}: {e}"))?;
println!("serving {} on http://127.0.0.1:{port}/", dir.display());
for stream in listener.incoming().flatten() {
let dir = dir.to_path_buf();
std::thread::spawn(move || {
let mut stream = stream;
let mut line = String::new();
let mut reader = BufReader::new(match stream.try_clone() {
Ok(s) => s,
Err(_) => return,
});
if reader.read_line(&mut line).is_err() {
return;
}
loop {
let mut h = String::new();
if reader.read_line(&mut h).map(|n| n == 0).unwrap_or(true) || h.trim().is_empty() {
break;
}
}
let path = line.split_whitespace().nth(1).unwrap_or("/").split(['?', '#']).next().unwrap_or("/").to_string();
let rel = if path == "/" { "index.html".to_string() } else { path.trim_start_matches('/').replace("..", "") };
let file = dir.join(&rel);
let (status, body) = match std::fs::read(&file) {
Ok(b) => ("200 OK", b),
Err(_) => ("404 Not Found", b"not found".to_vec()),
};
let mime = match file.extension().and_then(std::ffi::OsStr::to_str).unwrap_or("") {
"html" => "text/html; charset=utf-8",
"js" => "text/javascript",
"wasm" => "application/wasm",
"svg" => "image/svg+xml",
"json" | "webmanifest" => "application/json",
"png" => "image/png",
"gif" => "image/gif",
"ecproj" => "application/json",
"mp4" => "video/mp4",
"webm" => "video/webm",
_ => "application/octet-stream",
};
let head = format!(
"HTTP/1.1 {status}\r\nContent-Type: {mime}\r\nContent-Length: {}\r\nCross-Origin-Opener-Policy: same-origin\r\nCross-Origin-Embedder-Policy: require-corp\r\nCache-Control: no-cache\r\nConnection: close\r\n\r\n",
body.len()
);
let _ = stream.write_all(head.as_bytes()).and_then(|_| stream.write_all(&body));
});
}
Ok(())
}
/// The guidance printed when the ALSA pkg-config package is missing. Split out so the tests can
/// check the package names without needing a machine that is actually missing them.
#[cfg(target_os = "linux")]
fn audio_backend_missing_hint(pkg_config_stderr: Option<&str>) -> String {
let mut hint = String::from(
"Linux audio needs the ALSA development package, and pkg-config cannot find it.\n\
The desktop app plays preview sound through cpal, whose alsa-sys dependency asks\n\
pkg-config for `alsa`; without the package the build dies minutes later with an\n\
unrelated-looking linker error.\n\
\n\
Fedora: sudo dnf install alsa-lib-devel\n\
Debian/Ubuntu: sudo apt install libasound2-dev\n\
Arch: sudo pacman -S alsa-lib\n\
\n\
Install one of those and run `cargo xtask ci` again. If you cannot install it, point\n\
PKG_CONFIG_PATH at a directory holding an `alsa.pc` whose Libs line names your system's\n\
libasound (usually `libasound.so.2`); every CI run will need that variable set.",
);
if let Some(stderr) = pkg_config_stderr {
hint.push_str("\n\npkg-config said: ");
hint.push_str(stderr);
}
hint
}
/// `cargo xtask ci` builds the desktop app on the host, so on Linux it needs a usable ALSA
/// pkg-config package before the build dies minutes later at link time. The probe goes through
/// the same `pkg-config` crate alsa-sys's build script uses, so the executable and search-path
/// lookup match by construction: `PKG_CONFIG` and its target-scoped variants, then `pkg-config`
/// on `PATH` with a `pkgconf` fallback, plus the scoped `PKG_CONFIG_PATH`/`LIBDIR`/`SYSROOT_DIR`.
/// A probe that ran and said no is "ALSA missing"; one that could not run is reported separately,
/// so a machine with only `pkgconf` does not fail a build that would succeed.
#[cfg(target_os = "linux")]
fn audio_backend() -> Result<(), String> {
// cargo_metadata off: xtask is not a build script, so the crate's rerun-if-env lines would
// only clutter the ci log.
match pkg_config::Config::new().cargo_metadata(false).probe("alsa") {
Ok(_) => Ok(()),
Err(pkg_config::Error::ProbeFailure { output, .. }) | Err(pkg_config::Error::Failure { output, .. }) => {
let stderr = String::from_utf8_lossy(&output.stderr);
let stderr = stderr.trim();
Err(audio_backend_missing_hint((!stderr.is_empty()).then_some(stderr)))
}
Err(e) => Err(format!(
"The pkg-config probe for `alsa` could not be run: {e}. Install pkg-config (or pkgconf), \
or point PKG_CONFIG at a working one — it is the same lookup alsa-sys's pkg-config \
dependency performs."
)),
}
}
#[cfg(not(target_os = "linux"))]
fn audio_backend() -> Result<(), String> {
Ok(())
}
/// Lib, bin and integration tests (examples aren't linked; clippy checked them), then doctests,
/// which nextest doesn't run.
///
/// `effectcraft-gpu`'s tests share one device and serialise on a lock inside their process (several
/// devices at once crash software Vulkan and GL), so they run in one `cargo test` process instead of
/// nextest's process per test.
///
/// Each run goes ahead when an earlier one fails; the error names every run that failed.
fn tests() -> Result<(), String> {
let cargo = env!("CARGO");
let targets = ["--workspace", "--lib", "--bins", "--tests", "--no-fail-fast"];
let nextest = Command::new(cargo).args(["nextest", "--version"]).output().is_ok_and(|o| o.status.success());
let mut runs = Vec::new();
if nextest {
runs.push(run(Command::new(cargo).args(["nextest", "run"]).args(targets).args(["-E", "not package(effectcraft-gpu)"])));
// effectcraft-gpu's tests share one device, so they run in one libtest process: the
// binary nextest just built (`cargo test -p effectcraft-gpu` would rebuild ~2 minutes).
runs.push(lib_test_binary("effectcraft-gpu").and_then(|mut c| run(&mut c)));
} else {
println!("cargo-nextest not found, running cargo test (install: cargo install cargo-nextest --locked)");
runs.push(run(Command::new(cargo).arg("test").args(targets)));
}
// The test runs above hide a passing test's output, so the always-passing report runs again
// with --nocapture, from the binary already built.
runs.push(lib_test_binary("effectcraft-ui-egui").and_then(|mut c| run(c.args(["missing_translations_report", "--nocapture"]))));
runs.push(run(Command::new(cargo).args(["test", "--workspace", "--doc", "--no-fail-fast"])));
let failed: Vec<String> = runs.into_iter().filter_map(Result::err).collect();
if failed.is_empty() { Ok(()) } else { Err(failed.join("; ")) }
}
/// A command running `package`'s lib test binary from the test build above, the way `cargo test`
/// runs it (from the package directory), found with `--no-run` and the same workspace selection
/// so nothing rebuilds: `cargo test -p <package>` resolves features for that package alone and
/// recompiles much of the workspace.
fn lib_test_binary(package: &str) -> Result<Command, String> {
let out = Command::new(env!("CARGO"))
.args(["test", "--workspace", "--lib", "--bins", "--tests", "--no-run", "--message-format=json-render-diagnostics"])
.stderr(std::process::Stdio::inherit())
.output()
.map_err(|e| format!("cargo test --no-run: {e}"))?;
if !out.status.success() {
return Err(format!("cargo test --no-run: exited with {}", out.status));
}
let lib = package.replace('-', "_");
String::from_utf8_lossy(&out.stdout)
.lines()
.filter_map(|line| serde_json::from_str::<Value>(line).ok())
.find(|m| {
m["reason"] == "compiler-artifact"
&& m["profile"]["test"] == true
&& m["target"]["name"] == lib.as_str()
&& m["target"]["kind"].as_array().is_some_and(|k| k.iter().any(|k| k == "lib"))
})
.and_then(|m| {
let mut c = Command::new(m["executable"].as_str()?);
c.current_dir(std::path::Path::new(m["manifest_path"].as_str()?).parent()?);
Some(c)
})
.ok_or_else(|| format!("no {package} lib test binary in the test build"))
}
type Stage = fn() -> Result<(), String>;
/// Every stage runs even when an earlier one fails, so one run reports all the problems.
fn ci() -> Result<(), String> {
let stages: [(&str, Stage); 8] = [
("audio backend", audio_backend),
("fmt", || run(Command::new(env!("CARGO")).args(["fmt", "--check"]))),
("clippy", || run(Command::new(env!("CARGO")).args(["clippy", "--workspace", "--all-targets", "--", "-D", "warnings"]))),
("tests", tests),
("pdf-deps", || pdf_deps::run(&root())),
("layers", layers),
("assets", assets),
("wasm", wasm),
];
let failed: Vec<&str> = stages.iter().filter_map(|(name, stage)| stage().map_err(|e| eprintln!("ci: {name} failed: {e}")).err().map(|()| *name)).collect();
if failed.is_empty() { Ok(()) } else { Err(format!("ci failed: {}", failed.join(", "))) }
}
fn main() -> ExitCode {
let task = std::env::args().nth(1).unwrap_or_default();
let r = match task.as_str() {
"layers" => layers(),
"wasm" => wasm(),
"assets" => assets(),
"pdf-deps" => pdf_deps::run(&root()),
"aep-corpus" => aep_corpus::run(&root(), &std::env::args().skip(2).collect::<Vec<_>>()),
"ci" => ci(),
"audio-backend" => audio_backend(),
"web" => web(&std::env::args().skip(2).collect::<Vec<_>>()),
"bench" => bench::run(&root(), &std::env::args().skip(2).collect::<Vec<_>>()),
"ico" => {
let rest: Vec<String> = std::env::args().skip(2).collect();
ico::run(&rest.iter().map(String::as_str).collect::<Vec<_>>())
}
"version" => {
let rest: Vec<String> = std::env::args().skip(2).collect();
version::run(&root(), &rest.iter().map(String::as_str).collect::<Vec<_>>())
}
_ => Err("usage: cargo xtask <layers|assets|pdf-deps|aep-corpus [--dir DIR]|wasm|web [--dev] [--serve PORT]|ico OUT.ico IN.png…|version [set X.Y.Z]|ci|audio-backend|bench [--bin CLI] [--quick] [--project F --comp C] [--compare OLD.jsonl] (--help)>".into()),
};
match r {
Ok(()) => ExitCode::SUCCESS,
Err(e) => {
eprintln!("{e}");
ExitCode::FAILURE
}
}
}
/// The workspace root (the parent of `xtask/`).
pub fn root() -> std::path::PathBuf {
std::path::Path::new(env!("CARGO_MANIFEST_DIR")).parent().expect("xtask has a parent dir").to_path_buf()
}
#[cfg(all(test, target_os = "linux"))]
mod tests {
use super::audio_backend_missing_hint;
#[test]
fn alsa_missing_names_the_package_for_each_distro() {
let e = audio_backend_missing_hint(None);
for want in ["alsa-lib-devel", "libasound2-dev", "alsa-lib", "pkg-config", "PKG_CONFIG_PATH"] {
assert!(e.contains(want), "hint lost `{want}`:\n{e}");
}
}
#[test]
fn alsa_missing_appends_pkg_config_s_own_detail() {
let e = audio_backend_missing_hint(Some("Package 'alsa' not found"));
assert!(e.contains("pkg-config said: Package 'alsa' not found"), "detail lost:\n{e}");
}
}