17 KiB
craft-testkit
Source-level platform regression guards and verification utilities for the Craft application suite.
Overview
craft-testkit provides lightweight, non-panicking source guards that prevent applications in the Craft ecosystem from repeating platform defects that were previously diagnosed and resolved in peer applications:
predefined_quit: Flags direct calls tomuda'sPredefinedMenuItem::quitoutside comments, strings and character literals, also with whitespace or a line break around::, in the qualified form<muda::PredefinedMenuItem>::quit, and through ause muda::PredefinedMenuItem as ...rename. A type of the same name reached through another crate or module, such asother::PredefinedMenuItem::quit, is not flagged. On macOS, this predefined menu item routes directly toNSApplication terminate:, completely bypassing application dirty document checks and unsaved changes save prompts. Applications should instead wire Quit throughcraft-quitlifecycle interceptors.wgsl_float_literals/wgsl_float_literals_in: Scans WGSL shader sources and flags decimal and hexadecimal float literals whose value, rounded tof64, exceedsf32::MAX(such as3.40282347e38or0x1p+500f), or 65504, the largest finitef16, for a literal with anhsuffix. A literal that exceeds the maximum by less thanf64precision rounds to the maximum and is not reported. Integer literals are not checked. Some native compilers round such literals down, but browser WebGPU compilers reject them with pipeline compilation errors, forcing applications into CPU fallback rendering.f32::MAXcan be written as the hex float0x1.fffffep+127f, which equals it, or as the decimal3.4028234663852886e38, which rounds to it.std_instant: Flagsstd::time::Instantoutside comments and strings: the full path,<m>::Instantwhere ausebindsmtostd::time, and eachusethat importsstd::time::Instantorstd::time::*, under any name. The parts of a path may be split by whitespace, comments or line breaks. Other crates'Instant(such asweb_time::Instantortokio::time::Instant) is not flagged, and neither is an item whosecfgattribute is false onwasm32-unknown-unknown; code before that attribute or after its item on the same line is still checked. Amod name;inside such an item also skips the module's file (name.rsorname/mod.rs, or the file a#[path]attribute names) and the files of the modules declared in it, when they are among the files passed in and no declaration outside such items also reaches them. On that target,std::time::Instant::now()panics at runtime with"time not implemented on this platform". Code targeting the web must useweb_time::Instant.rust_sources: Recursively discovers a crate's.rssource files while excluding integration tests (tests/), test modules (tests.rs,tests_*.rs,*_tests.rs,*_test.rs), build artifacts (target/, any folder holding aCACHEDIR.TAGfile that starts with the lineSignature: 8a477f597d28d172789f06886806bc55,.git/and other folders whose name starts with.) and symbolic links, which it does not follow. A file over 8 MiB, more than 256 MiB in all or more than 50,000 files returns an error;rust_sources_with_limitstakes other limits. It is not built forwasm32.
Epistemic Scope: Syntactic Guards vs Runtime Validation
Source-level guards provide deterministic syntactic evidence about codebase structure:
What They Establish
- Checked source files do not contain forbidden symbol invocations, known out-of-range literal representations, or platform-hazardous imports.
- Reusable, deterministic checks that run in milliseconds as regular Cargo tests without external runners.
What They Cannot Establish
- Runtime execution safety: Syntactic checks do not verify whether code branches actually execute at runtime.
- Semantic type resolution: The scanners follow
userenames within one file only. They do not resolve paths through other modules or re-exports, a rename ofstditself, or expand macros. - Complex target predicates:
std_instantevaluatescfgpredicates (all,any,not,unix,windowsand thetarget_*keys) forwasm32-unknown-unknown, on outer attributes for the item that follows and on inner attributes (#![cfg(...)]) for the rest of the module, and onmod name;declarations for the module's file and its descendants. It compares file paths as text, so the files must come from onerust_sourcescall or be named the same way. A predicate that depends on a feature or ontestis not known from the source, so its item is checked. Applications should scopestd_instanttests to the specific crates that compile into their web distribution. - Driver and host OS behavior: Syntactic checks do not verify WebGPU driver compatibility or host window manager event dispatch.
Source guards serve as fast, zero-dependency regression gates in application test suites, complementing platform integration tests and CI workflows.
Consumer Adoption Examples
1. PhotoCraft: Adopt wgsl_float_literals and predefined_quit
In photocraft/crates/gpu/src/lib.rs, replace the in-tree decimal_float_literals helper with craft_testkit::wgsl_float_literals:
#[test]
fn shader_float_literals_fit_f32() {
let bad = craft_testkit::wgsl_float_literals(SHADER);
assert!(
bad.is_empty(),
"float literals outside f32 finite range reject on WebGPU (Tint):\n{}",
bad.iter().map(|f| f.to_string()).collect::<Vec<_>>().join("\n")
);
}
In photocraft/apps/photocraft/tests/native_menu_tests.rs:
#[test]
fn no_unintercepted_predefined_quit_in_menu() {
let sources = craft_testkit::rust_sources("src").expect("scan sources");
let findings = craft_testkit::predefined_quit(&sources);
assert!(
findings.is_empty(),
"predefined muda Quit bypasses dirty document save prompts:\n{}",
findings.iter().map(|f| f.to_string()).collect::<Vec<_>>().join("\n")
);
}
2. FilmCraft: Adopt std_instant and wgsl_float_literals
In filmcraft/crates/engine/src/wasm_clock_tests.rs, replace in-tree textual search with craft_testkit::std_instant:
#[test]
fn no_std_instant_in_code_the_web_build_runs() {
let sources = craft_testkit::rust_sources(Path::new(env!("CARGO_MANIFEST_DIR")).join("src"))
.expect("scan engine sources");
let findings = craft_testkit::std_instant(&sources);
assert!(
findings.is_empty(),
"std::time::Instant panics on wasm32; use web_time::Instant:\n{}",
findings.iter().map(|f| f.to_string()).collect::<Vec<_>>().join("\n")
);
}
deadline
The deadline module is not built for wasm32.
with_deadline(limit, label, f) runs f on its own thread and returns its value. When f does not
finish within the limit, the test fails with a message that names label, the limit and the variable
that sets it. A panic in f is raised again in the test with its original payload. The limit counts
from the call, including the time spent starting the thread. When f returns or panics after the
limit has passed, the test fails with the label and the limit, and with the panic's message if it has
one.
wait_until(limit, what, cond) calls cond on the test's thread until it returns true. It calls
cond at once, then after pauses that start at 1 ms and double up to 50 ms, and does not call it
again once the limit has passed. When the limit passes first, or cond returns true only after the
limit has passed, the test fails with a message that names what.
use std::time::Duration;
use craft_testkit::deadline::{DeadlineLimit, wait_until, with_deadline};
// The default of 30 s; CRAFT_TEST_TIMEOUT_SECS replaces it.
let pixels = with_deadline(Duration::from_secs(30), "decode seed 7", move || decode(&bytes));
// The app's own variable replaces the default instead.
let limit = DeadlineLimit::new(Duration::from_secs(30)).with_variable("FILMCRAFT_HOSTILE_TIMEOUT_SECS");
with_deadline(limit, "hostile input 12", move || parse(&input));
wait_until(Duration::from_secs(10), "the export to finish", || job.is_done());
The limit variable
A Duration reads CRAFT_TEST_TIMEOUT_SECS. DeadlineLimit::with_variable names another variable.
The variable is read on every call; neither function sets or caches it.
| Value | Result |
|---|---|
| Not set | The default limit applies. |
Decimal digits above zero, such as 30 or 007 |
That many seconds replace the default. |
Empty, signed (+5, -5), fractional (1.5), with spaces or units (5s), zero, or larger than a u64 |
The test fails with the variable's name and value, before f runs or cond is called. |
| Not Unicode | The test fails with the variable's name, before f runs or cond is called. |
A zero default, a variable name that is empty or holds = or a NUL character, and a limit that ends past
the latest time the platform's clock can represent also fail the test before any work starts.
FilmCraft's filmcraft-testkit uses the default when the variable holds something other than a
number or holds zero. This module fails the test instead. craft-media-testkit's with_deadline
calls this module's.
Limits
- A body that runs past its limit fails the test, but its thread is left running until the test process exits. A body that never returns can keep the test binary alive after its tests finish.
wait_untilchecks its limit only between calls ofcond. Acondthat blocks is not interrupted, so it must return promptly.- Neither function returns elapsed time. Tests of this crate use channel handshakes, and the only time bounds in them are the limits under test.
hostile
mutations(seed, rounds) yields rounds damaged copies of a valid seed. Each round makes one to four
edits: a bit flip, a random byte, a 1, 2, 4 or 8-byte overwrite with an extreme value
(0, 1, 0x7F, 0x80, 0xFF, 0xFFFF, 0x8000_0000 or u64::MAX) in either byte order, a deletion of up to
16 bytes, or a truncation. Each copy carries a label that lists its edits, such as
flip[3].5 del[7..9]. The xorshift64* generator behind it is seeded from the seed's length, so the
same seed length and round count give the same edits on every run and every platform. An empty seed
yields no copies. mutations and Rng build for wasm32.
hammer(&config, seed, parse) runs parse on the unmodified seed, on every truncation of it
(lengths 0 to one less than its length), and on mutations(seed, rounds). The cases run one after
another on one worker thread, and the test's thread fails the test when a case runs past its limit.
A case that finishes after its limit also fails the test. The test fails on the first case that
panics or does not finish, with a message that names the config's label, the case
(the unmodified seed, truncation to N or the edits), the input's length and the panic's message.
parse must be Fn(&[u8]) + Send + Sync + 'static, since a case that does not finish keeps running
on the worker thread after the test fails. hammer is not built for wasm32.
use std::time::Duration;
use craft_testkit::DeadlineLimit;
use craft_testkit::hostile::{HammerConfig, hammer};
// 2,000 rounds times CRAFT_HOSTILE_MULT; 5 s per case, or CRAFT_TEST_TIMEOUT_SECS.
hammer(&HammerConfig::new("png seed 3", 2_000), &seed, |data| {
let _ = decode(data);
});
// The app's own variables instead.
let config = HammerConfig::new("mxf primer", 5_000)
.with_multiplier_variable("CRAFT_MEDIA_HOSTILE_MULT")
.with_timeout(DeadlineLimit::new(Duration::from_secs(5)).with_variable("FILMCRAFT_HOSTILE_TIMEOUT_SECS"));
hammer(&config, &primer, |data| {
let _ = parse_primer(data);
});
The rounds
The rounds are the config's iterations times the multiplier variable, CRAFT_HOSTILE_MULT unless
with_multiplier_variable names another. The limit for each case is 5 s, replaced by
CRAFT_TEST_TIMEOUT_SECS or the variable that with_timeout names, as in deadline. Both variables
are read once, before any case runs. HammerConfig::rounds returns the rounds or the error that hammer would fail with.
| Multiplier value | Result |
|---|---|
| Not set | The multiplier is 1. |
Decimal digits above zero, such as 10 |
The rounds are multiplied by that number. |
Empty, signed, fractional, with spaces, zero, or larger than a usize |
The test fails with the variable's name and value before parse is called. |
| Not Unicode | The test fails with the variable's name before parse is called. |
A product of the iterations and the multiplier that does not fit in a usize, a call that would run
more than MAX_CASES (10,000,000) cases in all, and a limit variable that deadline would reject
also fail the test before parse is called.
Changes from craft-media-testkit's hammer:
- A case that does not finish fails the test after its limit with the case's label. Before, a case was timed after it returned and failed only when it took longer than 5 s.
- A panic's message is part of the failure.
- Every truncation runs. Before, a seed of 8,192 bytes or more ran every
len / 4096th truncation. - The caller names the multiplier variable. Before, it was always
CRAFT_MEDIA_HOSTILE_MULT, and a value that was not a number counted as 1. - An empty seed runs once. Before, it ran once more for each round.
- A mutation's label joins its edits with single spaces. Before, every edit ended with a space, so each label ended with one.
parseis owned and runs on a worker thread. A closure that borrows from the test needs its data moved or shared with anArc.
Origin
guards
| Module | Source | Path | Commit |
|---|---|---|---|
guards |
storytold/craft-libs#163 (zachshallbetter), with the rules of PhotoCraft's shader float test and FilmCraft's web clock test | crates/testkit/src/guards.rs |
e92825e427 |
deadline
| Code | Repository | Path | SHA |
|---|---|---|---|
with_deadline, its panic and timeout tests (a_slow_body_passes, a_panic_in_the_body_is_re_raised, a_hang_fails_and_names_the_input) |
storytold/filmcraft | crates/testkit/src/deadline.rs |
cff7e144 |
with_deadline under CRAFT_TEST_TIMEOUT_SECS, tests with_deadline_completes and with_deadline_times_out |
storytold/craft-libs | crates/media-testkit/src/lib.rs |
75a654f4 |
within, a deadline that returns the body's value |
storytold/lightcraft | crates/engine/src/demo/tests.rs |
487833ad |
within, a deadline that returns the body's value |
storytold/gridcraft | crates/math/src/lib.rs |
c2aba7a3 |
wait_until |
storytold/photocraft | crates/ui-egui/src/canvas/paint_thread_tests.rs |
0d3b1e85 |
wait_until |
storytold/lightcraft | crates/engine/src/availability.rs |
7ebbb17a |
wait_until |
storytold/effectcraft | crates/engine/src/footage/footage_reload.rs |
53576d22 |
wait_until |
storytold/filmcraft | crates/ui-egui/tests/it/export_ui.rs |
cff7e144 |
hostile
| Code | Repository | Path | SHA |
|---|---|---|---|
Rng, the edits of mutations and hammer |
storytold/craft-libs | crates/media-testkit/src/hostile.rs |
75a654f4 |
Migration
guards
| Consumer | Guard | Replaces | State |
|---|---|---|---|
| PhotoCraft | wgsl_float_literals |
crates/gpu/src/tests.rs scanner |
Not started |
| FilmCraft | std_instant |
crates/engine/src/wasm_clock_tests.rs search |
Not started |
deadline
| Consumer | Code it replaces | State |
|---|---|---|
| FilmCraft | filmcraft-testkit's with_deadline (FILMCRAFT_HOSTILE_TIMEOUT_SECS) and the export UI tests' wait_until |
Pending |
| LightCraft | within in the demo tests and wait_until in the availability tests |
Pending |
| GridCraft | within in the tests of its crates/math |
Pending |
| PhotoCraft | wait_until in the paint thread tests |
Pending |
| EffectCraft | wait_until in the footage reload tests |
Pending |
craft-media-testkit |
with_deadline (CRAFT_TEST_TIMEOUT_SECS) |
Done |
An adoption keeps the variable name the app uses today.
hostile
| Consumer | Code it replaces | State |
|---|---|---|
craft-media-testkit |
hostile::hammer (CRAFT_MEDIA_HOSTILE_MULT), called by the ISO BMFF, AVI, Matroska, MXF and MPEG TS hostile tests |
Done |
craft-cfb |
its hostile_input_never_panics test, through craft-media-testkit |
Done |
| CADCraft | the mutation loop of hostile_files_never_panic in crates/fonts/src/shx/tests.rs |
Pending |
| TextureCraft | the hammer of malformed_meshes_are_errors in crates/formats/src/mesh_import/tests.rs |
Pending |
CADCraft's and TextureCraft's loops make other edits than mutations. Their adoptions say which
cases change, and keep the test names.