Strokes carry an object (0 for older projects, nothing written for 0). The
tracker follows every object in one pass and the mattes are united; the
confidence of a frame is the least sure object's. Object picker in the Layer
panel's bar; roto.options/stroke/clearStrokes/status take or report objects.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
--clip FRAMES_DIR --truth MASKS_DIR runs the trackers on image sequences with true
mattes (strokes made from the truth; nothing is bundled or copied), tested end to end.
Trackers report the most memory their banks held (peakMB column).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Simulated user paints a foreground stroke through the true object and a background
stroke on the biggest false positive on the first frame under IoU 0.9, up to 6
rounds. SAM 2 needs none on every clip; classical and MobileSAM need more than 6 on
occlusion and re-entry. All trackers now take strokes on any frame.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TrackRequest takes a list of objects (each with its prompts on its frames) and
the tracker reports (object, frame, result); frame encodings are shared, objects
may start on different frames. VideoTracker::propagate_objects runs a subset.
Limits: 32 objects. Test: two discs, one prompted on frame 0 and one on frame 2.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
On real footage the stroke frame came out partial (clicks make SAM 2 pick a
part of the object). Measured on a multi-part figure: clicks only IoU 0.41,
with a box around the figure IoU 1.00. Chain::track now adds the box of the
classic engine's matte of the strokes to each prompt; a box stroke wins.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run_tracker follows the object with the model's video tracker on the same
synthetic clips (--sam2 FILE or the installed weights). 12 frames, dev build:
re-entry clip mean IoU 0.632 classical vs 0.975 SAM 2 (0.000 vs 0.967 after
the object returns); the other clips 0.98 vs 0.99.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
When the chosen model has a tracker, roto.propagate runs one tracking pass over
the span: prompts from the strokes (a handful of points, which SAM 2 trusts),
the clip tracked both ways, each frame's probability map refined by the same
graph cut with the strokes as hard constraints. Results are cached under their
own keys with a per-frame confidence and an occluded flag; roto.status reports
them (source: tracker|classical). Without a tracker nothing changes.
Real weights: IoU 0.98 on all 21 frames, confidence 0.99.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Synthetic Roto Brush clips with known mattes (movement, lighting change,
occlusion, leaving and re-entering) and a runner that compares the classical
engine with MobileSAM when installed. Needs no weights; SAM 2 plugs in later.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflict in crates/track/src/camtrack/bundle.rs: kept both the branch's
PtSys/preconditioner/iterative_schur block and upstream's new
add_focal_prior helper. Adapted to upstream's explicit linalg imports
(conjugate_gradient, and I3/log_rot/mt/norm in the tests) and moved the
explicit point elimination into reduce_points so adjust_impl stays under the
cognitive-complexity lint.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflict in crates/track/src/camtrack/bundle.rs (imports): upstream replaced
the linalg glob import with an explicit list; kept that list and added the
branch's SolveError import (use super::{Distortion, SolveError}). Input
validation, size budget and Result-returning adjust/cost are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflict in crates/track/src/face.rs: upstream moved the feature-blob search
into FaceTracker::feature_blobs. Re-applied the branch there: the per-pixel
point-in-polygon loop is replaced by non_skin_inside (one crossing pass per
window row) and the roll rotation is hoisted with face_mapper.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Dropping the points a later stroke of the other kind painted over could leave
a stroke with none, even though most of it was still painted: a small Alt tap
in the middle of a large foreground dot removed its only point, and the
selected model was skipped for the classical cut. Points are now kept by the
resolved ownership under them, and a stroke that loses every sampled point
but still owns pixels is prompted from the one deepest inside them. A stroke
painted over completely still prompts nothing.
A trained model is no longer prompted with both labels at one spot: points
along a stroke that a later stroke of the other kind painted over are
dropped, and the propagated matte's points, box and mask take this frame's
strokes over it.
A Refine Edge stroke over an earlier Alt (erase) stroke now restores the band
there; the band carried from the previous frame stays under this frame's
strokes.
A green stroke over an earlier Alt (red) stroke now restores the area: where
strokes overlap, the one painted last sets the hard constraint, for the cut,
the colour samples and the cleanup seeds alike.
Upstream split lib.rs files into modules and moved engine commands/ to cmd/; this branch's change merges without textual conflicts.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The fine stage searched a 5x5 neighbourhood per pyramid level and seed.
When the best coarse match is reliable (NCC score >= 0.8) and lies within
one top-level pixel of the motion prediction, a 3x3 neighbourhood is
enough; otherwise the 5x5 search stays.
Fixes#2349
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Reuse consecutive caller-owned pyramids for regular Warp motion and trajectory tracking. Preserve the original three-level motion search by setting aside the trajectory-only extra level, without cloning planes. Keep detailed analysis at its existing distinct resolutions and test bitwise motion and complete JSON equality against separate pyramids.
Fixes#2322
Solve expensive camera systems with matrix-free Schur products and a block-preconditioned iterative solver. Check the true linear residual and fall back to the direct solve when convergence fails. Keep short systems on the existing path and compare reprojections, camera parameters and outlier decisions against direct solves.
Fixes#2328
Share the unchanged Jacobi factors between full SVD and null-vector callers. Select the same right singular vector directly instead of normalizing U and sorting copies that the caller discards.
Fixes#2329
Detect narrow lower-triangle bandwidth and skip structural zeros while preserving dot-product lanes and signed-zero arithmetic. Keep a dense fallback for wide systems and compare factors and solutions against the original algorithm.
Fixes#2325
Return errors for invalid bundle observations and oversized workspaces, reject non-finite projections and mismatched correspondence slices, and prevent track frame overflow. Cover the reported failures with synthetic regression tests.
Fixes#2330
Use a bounded inline array for the current camera parameters and preserve the existing accumulation order. Extra parameters spill safely if the parameter set grows.
Fixes#2327
The normalized cross-correlation only had a fast path for an exactly
identity shape, which tracked footage never has after the first frame, so
every candidate went through Plane::sample. All candidates of a search are
the reference centre plus integer offsets, so each template pixel has the
same fractional offsets for all of them. Build one table of taps per
(level, seed, shape) and score each candidate with integer-indexed lerps,
with one bounding-box check up front (outside the plane the old path is
used).
lk_refine only reads the level-0 gradients around the match, so compute
them for that window (with an interior fast path in Plane::gradients_in)
and fall back to the full gradients if the iteration leaves it.
The results are unchanged: same lerp form and accumulation order.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
step() cloned the whole RGBA frame to remember it (33 MB at 1080p,
133 MB at 4K) and then cropped that clone, although only the region
around the outline is ever read. The tracker now stores the previous
frame already cropped to the region the next step will use, computed
from the updated outline with the same margin and rounding as before,
so the crops, and therefore the tracking result, are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Store the two four-element DLT rows in arrays while preserving arithmetic and accumulation order. This removes two heap allocations for every triangulated view.
Fixes#2326
The feature mask tested every non-skin window pixel against the 48-edge
face outline (about 25 M edge tests per frame). A scanline hits the same
edges for every pixel of a row, so the crossings are now computed once
per row with the same expression and comparisons as inside_polygon, and
a pixel's parity follows from how many lie to its right. The face-frame
rotation is also worked out once per frame instead of once per blob
pixel.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Every pixel of a (2r+1)^2 window shares one fractional offset per level
and iteration, but lk_point called Plane::sample (with a libm floor, a
bounds test and the clamp path) up to three times per pixel. The offset
and the clamped column/row index tables are now computed once per
window and the pixels run the same lerps on integer indices.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The nightly FreeBSD run measured 119 ms/frame against a 100 ms limit: the mean included
Tracker::new and any scheduling stalls on the two-core VM. The best single step with a
250 ms bound still catches an order-of-magnitude regression.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: with a trained model installed, Object Matte's optional box reached
MobileSAM only as background points along the box's edges (an approximation),
although the model's prompt encoder takes a box prompt directly.
Fix: a new Roto Brush stroke kind, Box (two opposite corners), that
stroke_prompt turns into the model prompt's bbox (scaled to the matte grid,
corners in any order, degenerate or non-finite boxes ignored). The graph cut
never rasterizes it. Object Matte emits it instead of the edge strokes when a
model segments; the classical fallback keeps the edge strokes. A prior matte's
box no longer replaces a box the user drew on that frame.
Tests: effects object_matte::the_box_reaches_the_model_as_a_box_prompt (a
stand-in model sees the box, at full and half resolution, and no edge points);
track roto::tests::box_stroke_is_the_model_box_prompt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- effectcraft-segment: a FaceModel interface (find a face in a region, follow it; a Topology
maps the model's points to the tracker's landmarks and outline), a TensorFlow Lite reader and
interpreter (float ops, float16 weights folded; matches ai-edge-litert to ~1e-6), and
MediaPipe Face Landmarker (Google, Apache-2.0: BlazeFace + Face Mesh V2, 3.8 MB, pinned
SHA-256). The registry gains a task and authors per model; installed weights get a NOTICE.
- effectcraft-track: FaceTracker::new_with uses the model, falls back to the classical fitter
when it finds no face; chin and jaw come from the outline either way so measurements agree.
- Engine: roto_models becomes models (both tasks); face.models / face.model.* commands,
Settings ▸ Face Tracking, `wait` on select/install/download for scripts; track.mask reports
faceModel.
- Credits: ATTRIBUTION.md lists the optional models and the Apache-2.0 specs the code follows.
- Docs: README, ROADMAP (G7), gaps, parity (TRK-3 score unchanged pending G1), agents,
preferences, architecture, segment README.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New L1 crate effectcraft-segment: the MaskModel interface (prompts → foreground probability),
a registry of open-licensed models (licence must be MIT/Apache/BSD/…; URL, size, SHA-256),
a PyTorch checkpoint reader (zip + pickle subset), dependency-free SHA-256, and MobileSAM
(TinyViT-5M + SAM prompt encoder / two-way transformer / mask decoder) on rayon and ndarray's
safe GEMM. Loads the official mobile_sam.pt as published; checked against candle
(embedding 0.3% RMS, 99.9% equal mask pixels); ~0.7 s/frame encoder at 32 threads, cached
per frame, ~0.1 s decoder.
- track::roto: segment_with / propagate_with prompt a model with the strokes or the flow-warped
matte (box, core points, mask); its probability is a strong, unblurred prior for the same
graph cut (strokes stay hard constraints); a failing model or one disagreeing with the flow
(IoU < 0.5) is ignored. Prior gets a weight.
- effects::roto: Version 2.0 / 3.0 use the model chosen in Settings (set_model / model_for);
the model id is part of the chain seed.
- engine: roto_models (roto.models, roto.model.select / install / download / remove), weights in
the models folder (host::config_dir()/models, shared by the app, CLI and MCP), background
loading, downloads through the system curl, verified; Settings ▸ Roto Brush page (prefs
roto.model). host::config_dir moved from the desktop app.
- Tests: segment (kernels vs direct sums, checkpoint reader, SHA-256 vectors, registry), track
(fake model: prior shapes the cut, failing / disagreeing models ignored; real weights with
EFFECTCRAFT_MOBILESAM: base IoU 0.989, propagated ≥ 0.980 vs classic 0.973), engine (registry,
verification, selection; real install + load), MCP.
- Docs: README, ROADMAP, parity (MSK-4 done: 90/2/0), preferences, agents, architecture,
crates/segment/README.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kristian Myram
vangel76andClaude Sonnet 5.5
BFlat
Dylan M. Taylor
tylerjenningsw
aiongg
Brandon ThomasandClaude Opus 5.5
Dexter