Hiera configs and size detection from the weights, registry entries pinned to
commits with verified SHA-256, the loader picks the size by checksum. The GPU
encoder submits once per block so Large fits in memory, and uploads no longer
panic on a lost device. Reference tests run per size on the CPU and GPU.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
egui_kittest test: a scripted tracker (sure, doubtful, unsure, absent, sure again)
tracks a layer and the Layer panel is rendered; the strip's pixels are green, amber,
red, slate and green at the right frames. ROTO_SPAN_SNAPSHOT saves the panel image.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
--clip FRAMES_DIR --truth MASKS_DIR runs the trackers on image sequences with true
mattes (strokes made from the truth; nothing is bundled or copied), tested end to end.
Trackers report the most memory their banks held (peakMB column).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Simulated user paints a foreground stroke through the true object and a background
stroke on the biggest false positive on the first frame under IoU 0.9, up to 6
rounds. SAM 2 needs none on every clip; classical and MobileSAM need more than 6 on
occlusion and re-entry. All trackers now take strokes on any frame.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Memory attention (1.8e-5 vs PyTorch) and the memory encoder (2.6e-6) as wgpu kernels,
convolution and layer-norm+GELU kernels, VideoModel::load_auto picks the GPU when
there is one (EFFECTCRAFT_SAM2_DEVICE overrides) and falls back to the CPU on any GPU
error; disable_gpu switches it off mid-clip. One process-wide device (several at once
crashed the Vulkan loader). A tracked frame: 4.3 s CPU, 0.7 s on the RX 7900 GRE.
Clips tracked on the GPU match Sam2VideoModel (worst logit error 0.0032).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Strided batched matmul with bias, GELU/ReLU and residual in the epilogue, layer
norm, softmax, window partition, pooling, patch embedding, RoPE and neck kernels
(WGSL, all tested against CPU definitions). GpuEncoder keeps activations on the
GPU and returns the 64x64 level and the decoder's skip features. On the RX 7900
GRE one frame encodes in 0.29 s (debug build) against 2.1 s on the CPU in release;
errors 6.8e-6 vs PyTorch, 1.2e-5 vs the CPU path.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Candle adds a bias on one thread; the Linear layers now add it (and apply GELU
for the MLP in) in one parallel pass. Encode 2.4 s -> 2.1 s, memory attention
2.4 -> 2.3 s; reference tests unchanged. gpu_matmul example: a tiled WGSL
matmul on the RX 7900 GRE reaches 1.9-3.6 TFLOPS against 0.16-0.47 for the CPU.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A tracked frame is the encode (2.4 s) plus memory attention (2.4 s with a full
bank), about 5 s on 16 threads, not one encode as the docs said. Docs corrected;
chunked attention tried and reverted.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sine position encodings depend only on the sizes: built once (150 ms of a frame).
do_pool takes the maximum over grouped cells instead of permute + max_pool2d +
permute (tested equal, odd sizes included). Encode 2.6 s -> 2.4 s per frame in
release on 16 threads (3.4 s at the start of the speed work); reference tests
unchanged. Chunked global attention was tried and did not help.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Candle's gelu_erf runs on one thread and was a third of the Hiera block time.
A rayon CustomOp1 calls the same erf per element (bit-identical, tested). All
reference tests unchanged. op_bench example times the block's operations.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MaskModel::idle_secs (SAM 2 tracks its use); the engine frees a model idle for
ten minutes when no job runs, keeps the choice, and reloads it before a stroke
or a Propagate needs it. Docs: memory and limits.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
VideoTracker::set_budget (default 2 GiB of state, optional seconds per
propagate call) stops with Error::Budget; state_bytes reports what it holds.
Two runs of the same prompts give bit-identical masks.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Antialiased bilinear resize as PyTorch does it, the reference's mask-as-output
path (mask is the frame's output, the decoder makes the object pointer),
VideoTracker::prompt_mask, and mask-only prompts through the tracker interface.
Prompt frame logits exact, tracking forwards and backwards within 0.0019.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Frames a tracker followed are coloured by its confidence (green, amber, red;
slate where the object was absent) and drawn taller; roto_trust reports them.
Not checked visually: no headless way to track without weights in a snapshot.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Chain::track tracks the frames of the span without a tracked result, in a window
that restarts at the nearest prompted frame; unchanged spans are not tracked
again. Test with a recording tracker: first pass 0..20, no second pass, then a
correction on frame 8 tracks 8..20 with that prompt only.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TrackRequest takes a list of objects (each with its prompts on its frames) and
the tracker reports (object, frame, result); frame encodings are shared, objects
may start on different frames. VideoTracker::propagate_objects runs a subset.
Limits: 32 objects. Test: two discs, one prompted on frame 0 and one on frame 2.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
On real footage the stroke frame came out partial (clicks make SAM 2 pick a
part of the object). Measured on a multi-part figure: clicks only IoU 0.41,
with a box around the figure IoU 1.00. Chain::track now adds the box of the
classic engine's matte of the strokes to each prompt; a box stroke wins.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The tracked frames lived only in memory, so a saved project lost them: a render in another
process (effectcraft-cli render-frame) segmented the frames again with the classic engine.
A propagate with a tracker now saves the frames it followed with the instance, in a hidden
parameter beside Freeze's (TrackCache: each frame's matte, Refine Edge band, colour models and
confidence, keyed by the inputs it was tracked from).
- They apply while the strokes and span they came from are unchanged, whichever model Settings
choose, so a project renders the same when it reopens, without the model loaded or chosen.
- A stroke or span change makes them stale until the next Propagate. A Propagate replaces them;
with the classic engine it leaves none.
- roto.status reports tracker.saved (the model they came from) and a notice when Settings
choose another model.
- The save at job completion is not an undo step (the frames are a cache of the strokes) and is
skipped if the strokes changed while tracking. Decoding checks the saved size, so a corrupt
project is ignored rather than overflowing.
Object Matte saves nothing. Projects saved before this change open unchanged (the parameter is
filled in on open).
Tests: a saved project reopens with the model unloaded and renders the saved frames through
Chain::ensure; choosing the classic engine keeps them until a classic Propagate replaces them; a
stroke change makes them stale. The SAM 2 tests and the failing-tracker test take the Roto test
lock.
Claude-Session: https://claude.ai/code/session_0136kZtXrYo5VUWEFxWduwUD
With SAM 2 (or MobileSAM) chosen, a propagate that started before the model had loaded ran
the whole job on the classic engine and reported nothing. The chosen model is now in use
before a job that segments with it starts (Roto Brush 2.0 / 3.0 and Object Matte):
- roto.propagate and the other Roto jobs wait for a load or download under way when called with
wait: true (bounded to 10 minutes). Without wait they refuse while the model is still loading,
and start the load in the background. A model that is not installed, or failed to load, is
refused either way.
- roto.status reports model {selected, installed, active, loading, error} and a notice while the
chosen model is not the one in use. source is "model" for a frame the model segmented on its own,
instead of "classical".
- Auto-propagate keeps edited instances pending while the model loads, and shows errors as toasts
instead of dropping them.
- Docs: what propagate does now; tracked frames live in the session that ran propagate and are not
saved in the project file, so a separate render process does not show them; roto.model.select
lists sam2.1-hiera-tiny.
Tests: a selected model that is not installed refuses to propagate (no weights needed); SAM 2
selected without wait is loaded before propagate (EFFECTCRAFT_SAM2). The SAM 2 tests and the
failing-tracker test take the Roto test lock, since they share the process-wide model slot.
Claude-Session: https://claude.ai/code/session_0136kZtXrYo5VUWEFxWduwUD
Bit-for-bit test: every frame of the Roto Brush chain equals a direct
segment_with / propagate_with call with no model.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
--mobilesam FILE; results recorded in docs/roto-brush-sam2.md (re-entry clip:
0.632 classical, 0.499 MobileSAM, 0.975 SAM 2).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Test with a tracker that always errors: roto.propagate completes, every frame
comes from the classic engine (status source: classical), IoU unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run_tracker follows the object with the model's video tracker on the same
synthetic clips (--sam2 FILE or the installed weights). 12 frames, dev build:
re-entry clip mean IoU 0.632 classical vs 0.975 SAM 2 (0.000 vs 0.967 after
the object returns); the other clips 0.98 vs 0.99.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When the chosen model has a tracker, roto.propagate runs one tracking pass over
the span: prompts from the strokes (a handful of points, which SAM 2 trusts),
the clip tracked both ways, each frame's probability map refined by the same
graph cut with the strokes as hard constraints. Results are cached under their
own keys with a per-frame confidence and an occluded flag; roto.status reports
them (source: tracker|classical). Without a tracker nothing changes.
Real weights: IoU 0.98 on all 21 frames, confidence 0.99.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ObjectTracker interface in the segment crate (prompts on some frames, the
clip tracked both ways, each frame streamed with confidence and an occluded
flag), implemented by SAM 2. MaskModel::tracker() is None for other models.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sam2Mask wraps the image path behind the Roto Brush's MaskModel interface
(cached frame encoding, bilinear mask resize). The engine loads the verified
weights natively (not on wasm). Checked with the real file: install, notice,
load, use by Roto Brush 3.0, deselect.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Apache-2.0 entry pinned to a Hugging Face commit with its SHA-256 and size,
listed in Settings > Roto Brush and installable with the existing verified
install flow. No weights are bundled.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Click on frame 2 of a 7-frame clip with an occlusion, tracked forwards and
backwards: IoU 1.0000 on all 9 frame results, worst mask-logit error 0.0038,
object scores equal to 3 decimals.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Per-object memory banks, prompts on any frame, forward and backward
propagation with streaming results, cancellation, object pointers with
temporal encoding, pruning of memories no later frame can use, and checked
input handling. Hostile-input and cancel tests.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Self and cross attention with axial 2-D RoPE (keys tiled over memory frames,
object-pointer tokens left unrotated). Max abs error 2.6e-5.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Mask down-sampler, feature fuser and projection, sine position encoding,
occlusion embedding and bfloat16 rounding of stored memories. Max abs error
3.5e-6 on the memory features, 6e-8 on the position encoding.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Encode 5.0 s -> 3.4 s per 1024x1024 frame in release on 16 threads (fused
softmax, queries scaled before the matmul, flat 2-D linear layers); a click
on an encoded frame is about 120 ms. Reference tests unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A click on the synthetic image: mask logits within 1.1e-4 (range 11.6), IoU
head 1e-6, object score 2e-5, prompt embeddings 1e-7.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Weights loader with shape/finite validation (S2.10), patch embed, bicubic
positional embedding, window/global attention, query pooling. Reference test
(skips without weights): max abs error 2.6e-5 over all 12 blocks.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Synthetic Roto Brush clips with known mattes (movement, lighting change,
occlusion, leaving and re-entering) and a runner that compares the classical
engine with MobileSAM when installed. Needs no weights; SAM 2 plugs in later.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
#1339 was written before ui-egui's integration tests became modules of one binary
(`tests/it/main.rs`), so its six test files landed at the top of `tests/`, each its own crate
and link. They move into `tests/it/` as modules; the a11y module's doc names the new path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
