Publish fixed model evaluation and register contributor real name

This commit is contained in:
mighty-programmer committed 2026-10-10 22:37:09 +03:00
1 parent 7f272aacd6
commit 58fceda01f
2 files changed
+47 -7

No files matched your search

+1
View File
@@ -92,4 +92,5 @@ sync_github_name = false
sync_github_name = true
[people.mighty-programmer]
real_name = "John Pantrakis"
sync_github_name = true
+46 -7
View File
@@ -2,8 +2,9 @@
Recorded 2026-10-08 on an Apple M3 Max with 36 GiB RAM, Rust 1.98.0, native CPU inference
and the immutable model exports listed in [local-models.md](local-models.md). No learned weights
are included in this repository. This is an initial integration and performance check, not a
labelled accuracy benchmark or evidence that a model always improves on the classical method.
are included in this repository. The integration/performance checks below are supplemented by a
fixed 22-case evaluation. Its 20 labelled COCO targets are exploratory evidence, not a representative
or held-out benchmark, a soft-alpha matting assessment, or evidence of a universal winner.
## Preferences before and after
@@ -41,8 +42,8 @@ selection, SAM object selection.
![Actual command outputs](images/local-models-comparison.png)
The classical method works well on this example. BiRefNet supplies continuous edge alpha;
SAM leaves some unwanted holes. These results support keeping both choices available and
requiring a labelled multi-image benchmark before a general accuracy claim.
SAM leaves some unwanted holes. These results support keeping both choices available. The
exploratory multi-image evaluation below also includes failures and cases where Classical is better.
Repeated BiRefNet runs initially exhausted memory with ONNX Runtime's default CPU arena and
memory-pattern caching. Disabling both while retaining only the current model's weights let
@@ -55,6 +56,44 @@ cargo run --release -p photocraft-engine --features local-ml --example local_mod
/absolute/model-cache /absolute/image.png /absolute/comparison-directory
```
## Fixed multi-image evaluation
The [33-page comparison PDF and public reproducibility supplement](https://github.com/mighty-programmer/photocraft/releases/tag/model-evaluation-2026-10-08)
contain all 22 distinct cases: 20 COCO val2017 images with independent binary instance references,
plus two user-supplied LEGO photos for visual inspection. There are 44 paired comparisons,
88 final method outcomes and 86 actual exported cutouts. BiRefNet's no-subject chair result and
Classical's empty zebra object selection are explicit failures, counted as zero rather than omitted.
| Feature | Classical mean target IoU | Learned mean target IoU | Learned wins / losses / near-ties |
|---|---:|---:|---:|
| Remove Background | 31.5% | 51.5% (BiRefNet HR Matting) | 15 / 2 / 3 |
| Box Object Selection | 52.5% | 79.1% (SAM 2.1 Large) | 16 / 3 / 1 |
Cases were selected using a fixed hash seed before inference, one from each of 20 categories.
Alpha is thresholded at 128/255, and a win/loss requires over one percentage point of IoU change.
Object boxes are derived from the annotation with 10% padding and reused for both methods.
The two unlabelled LEGO photos are excluded from the means. Each background-removal reference
contains one instance; keeping other plausible foreground subjects counts as a false positive.
COCO polygons cannot establish hair, soft-alpha or transparent-glass matting quality. This selected
pilot may overlap training data, and its annotation-derived boxes are easier than imperfect user boxes.
Median warm command times on the M3 Max / 36 GiB were 0.19 / 26.63 seconds for Classical /
BiRefNet removal and 0.20 / 5.59 seconds for Classical / SAM selection. These single-pass observations
exclude image import/export and initial/model-loading cases; they are not Windows/Linux timings.
Storage pressure interrupted two exports, which were rerun; the supplement preserves all 90 raw
attempts, 88 final outcomes and the recovery notes. Sampled resumed-run peak RSS was 13.35 GiB;
initial memory samples were not saved.
All methods ran in the same release engine at benchmark commit
`6e2b7bc9b20938787cf1f7af9718a832a2edf716`. The Classical algorithm sources match stock 0.3.0
`60224d3fb7d4006bcfcc97603c1611b9b756aebd`; the installed stock GUI was not timed separately.
Later UI/account changes were not re-benchmarked. The release tag identifies the tested engine,
and the supplement supplies the exact rectangles, annotation polygons/reference masks, source
hashes, command logs, CSV scores, model revisions and Rust harness. Its helper downloads and
hash-checks the 20 public inputs; model installation stays explicit. The report retains photo and
annotation attribution. Standalone user-photo originals, full-resolution cutouts and model weights
are not bundled in the public supplement or software repository.
## Checks
- ML unit tests execute both contracts through the actual CPU runtime using original tiny ONNX
@@ -79,11 +118,11 @@ cargo run --release -p photocraft-engine --features local-ml --example local_mod
it does not establish a sub-100 ms budget for the optional models.
The dedicated `local-models.yml` workflow runs Linux, Windows and macOS CPU runtime tests and
native build checks. The [final cross-platform run](https://github.com/mighty-programmer/photocraft/actions/runs/37783114338)
native build checks. The [cross-platform run at `1904396`](https://github.com/mighty-programmer/photocraft/actions/runs/37849240044)
passes on all three platforms, including the final document-state and input regression tests.
The contribution is ready for review. Upstream PR CI needs maintainer approval for
this first contribution. Quantitative accuracy evaluation, packaged installer / sandbox download
checks, web inference, point prompts and GPU acceleration remain separate work.
this first contribution. Representative held-out/matting evaluation, packaged installer / sandbox
download checks, web inference, point prompts and GPU acceleration remain separate work.
Image sources and licences: [SOURCES.md](images/SOURCES.md),
[LICENSE-local-models.txt](images/LICENSE-local-models.txt).