From 7f56fd87097bf1ae01e633c1d68106a482ca2ec4 Mon Sep 17 00:00:00 2001 From: Kristian Myram Date: Thu, 15 Oct 2026 21:53:57 +0200 Subject: [PATCH] S2.52: refreshed benchmark (24 frames, SAM 2 on the GPU) Occlusion 0.491 classical / 0.494 MobileSAM / 0.989 SAM 2; re-entry 0.618 / 0.620 / 0.982. Co-Authored-By: Claude Sonnet 5.5 --- docs/roto-brush-sam2.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/roto-brush-sam2.md b/docs/roto-brush-sam2.md index b3ceba2ad..d36d2086c 100644 --- a/docs/roto-brush-sam2.md +++ b/docs/roto-brush-sam2.md @@ -88,18 +88,18 @@ encoder's 64 × 64 level matches PyTorch to 7e-6, the memory attention to 1.8e-5 times the pieces on both devices (`EFFECTCRAFT_SAM2_PROFILE=1` prints a frame's breakdown). ## Benchmark -`effectcraft-cli bench --roto [--frames N] [--sam2 model.safetensors] [--mobilesam mobile_sam.pt]` runs synthetic clips -with known mattes through all three. 12 frames, debug build, this PC; mean IoU (mean IoU after the object returns), seconds +`effectcraft-cli bench --roto [--frames N] [--sam2 model.safetensors] [--mobilesam mobile_sam.pt]` runs synthetic clips with +known mattes through all three. 24 frames, debug build, SAM 2 on the GPU; mean IoU (mean IoU after the object returns), seconds per frame: | Clip | Classical | MobileSAM | SAM 2.1 Tiny | |---|---|---|---| -| move | 0.982 | 0.992 | 0.993 | -| light | 0.981 | 0.992 | 0.992 | -| occlusion | 0.978 | 0.881 | 0.988 | -| reenter | 0.632 (0.000) | 0.499 (0.725) | 0.975 (0.967) | -| s/frame | 0.01 | 1.1 | 8.2 | +| move | 0.980 | 0.989 | 0.992 | +| light | 0.979 | 0.990 | 0.992 | +| occlusion (a bar hides the disk, then clears it) | 0.491 (0.000) | 0.494 (0.000) | 0.989 (0.988) | +| reenter (the disk leaves and comes back) | 0.618 (0.000) | 0.620 (0.000) | 0.982 (0.976) | +| s/frame | 0.01 | 0.5–1.3 | 0.8 | MobileSAM prompts every frame independently from the flow-warped matte, so it follows the object while the flow does and -loses it when it is hidden; SAM 2's streaming memory is what lets it re-acquire the object. The synthetic clips are easy -(one textured disc), so the "move" and "light" rows mostly show all three working; the re-entry row is the difference. +loses it when it is hidden; SAM 2's streaming memory is what lets it re-acquire the object. The synthetic clips are easy (one +textured disc), so the "move" and "light" rows mostly show all three working; the last two rows are the difference.