Files
photocraft/docs/local-models.md
T

12 KiB
Raw Blame History

Optional local selection models

PhotoCraft keeps its classical Object Selection, Select Subject and Remove Background methods as the defaults. A build with local-ml offers two separately downloadable alternatives for cases where a larger learned model is useful. Images are processed on the device; downloading weights is the only network operation. This first backend uses CPU inference on native desktops, including Windows and Linux, without Python, CUDA or a GPU requirement. The web build retains the classical methods.

Model Use Download Training input Licence
BiRefNet HR Matting Select Subject / Remove Background, preserving soft alpha 932,150,975 bytes (889 MiB) 2048 × 2048 RGB MIT
SAM 2.1 Large Object Selection with a dragged rectangle 911,109,265 bytes (869 MiB) 1024 × 1024 RGB Apache-2.0

These FP32 models can take tens of seconds and use several GB of RAM. They are optional methods, not a guarantee of better results on every image. SAM is an object segmentation model, not a hair/transparency matting model. BiRefNet may pick several foreground objects. Difficult edges can still need Select and Mask or manual mask edits. HDR values are converted and clipped for the model input; original pixels remain unchanged. Canvases are currently limited to 64 MP for model commands, and output coverage uses PhotoCraft's existing 8-bit selection/mask format.

Build and use

cargo run --release -p photocraft --features local-ml -- photo.png
  1. Open Edit › Preferences › Integrations.
  2. Download either model. Progress and Cancel use the existing status bar. Each download is independent and does not select a method automatically.
  3. Choose a downloaded model for Object selection or Subject / background removal, then Apply or OK. These preferences are independent; Classical remains available.
  4. Drag an Object Selection rectangle, use Select › Subject, or use the existing Remove Background Quick Action in the layer Properties panel.

Remove Background replaces the layer's mask, converts a Background to a normal layer when needed, and clears the selection in one undo step. Source pixels are preserved. Its refine option affects the classical method only: applying classical refinement to BiRefNet's alpha would change the matte. Existing layer/parent locks still apply. Selection commands retain replace/add/subtract/intersect and active-layer/composite sampling.

Object Finder hover previews and point clicks use the built-in classical method, including when SAM is selected for boxes. Recorded point actions store Classical. An explicit SAM request with a point is rejected: this model export supports rectangle prompts only. The classical Select Subject result cache remains available independently of model inference.

Missing or corrupt weights produce an actionable error; inference never downloads or silently changes methods. Remove download removes only the selected model and returns its preference to Classical. A cancelled job never applies its document result. Cancellation during model loading is checked after session creation; network cleanup may wait for a stalled read timeout.

The desktop cache is Models/ under the existing per-user configuration directory (including PHOTOCRAFT_CONFIG_DIR or portable PhotoCraftData). It is separate from image files and preferences. Each immutable model revision gets its own directory with LICENSE.txt and SOURCE.txt; inference verifies the hashes again, including SAM's external weights.

Linux uses the existing XDG configuration path, so an AppImage stores models outside its read-only mounted image and a Flatpak uses its sandbox's XDG configuration directory. The current Flatpak manifests omit network permission and do not enable local-ml. A distributor opting in must also choose how explicit downloads receive network access (or prepopulate a verified cache); this backend does not alter sandbox permissions. Windows uses %APPDATA%\\Photocraft\\Models outside portable/override mode. Installer packaging and sandbox download checks remain separate from the native CPU build matrix.

Commands and CLI

Command Parameters
models.list {}; returns availability, installed/busy state, sizes, licences and provenance
models.download {"id":"birefnet-hr-matting"} or {"id":"sam2.1-large"}
models.remove Same ids; no document required
select.subject, layer.removeBackground "model":"classical" or "birefnet-hr-matting"; omitted uses Preferences
select.object "rect":[x,y,width,height], plus "model":"classical" or "sam2.1-large"; "point":[x,y] uses Classical Object Finder

Downloads/removals do not enter recorded actions or document history. Commands accept catalogue ids, never arbitrary URLs or model paths. Recorded selection/removal actions store the chosen method, so changing Preferences does not change playback. Batch and droplets reuse a configured backend across inputs. Automation hosts must explicitly inject a backend capability; an untrusted headless MCP session does not gain cache/network authority by enabling the feature. The trusted CLI accepts an explicit PHOTOCRAFT_MODEL_DIR:

# Build once. No model weights are downloaded at build time.
cargo build --release -p photocraft-cli --features local-ml
export PHOTOCRAFT_MODEL_DIR="/absolute/path/to/model-cache"

# Explicit installation, without an input photo (the empty document is only CLI run context).
target/release/photocraft-cli run --new '{"width":1,"height":1}' \
  --cmd models.download --params '{"id":"birefnet-hr-matting"}'

target/release/photocraft-cli run photo.png \
  --cmd layer.removeBackground --params '{"model":"birefnet-hr-matting"}' \
  --out cutout.png

target/release/photocraft-cli run photo.png \
  --cmd select.object --params '{"rect":[40,20,500,700],"model":"sam2.1-large"}' \
  --out selected.pcraft

PowerShell: set $env:PHOTOCRAFT_MODEL_DIR = 'C:\\path\\to\\model-cache' and use the .exe binary. Distributors opt in with local-ml; existing/default installers do not acquire the feature from downloading these weights. ort = 2.0.0-rc.13 downloads its pinned ONNX Runtime 1.28 redistribution at build time. Ship that runtime's MIT licence and third-party notices with enabled builds; the copies in crates/ml/licenses/ document this obligation. Release packaging is a distributor decision, and this change does not enable the feature in official release workflows.

Provenance and review

Why these community exports

Source review on 2026-10-10 found no ONNX files in the exact official checkpoint trees: BiRefNet HR Matting at 5d6b6f8 publishes Safetensors; Meta SAM 2.1 Large at 665f8e2 publishes Safetensors and a PyTorch checkpoint. This Rust backend needs ONNX graphs, so it retains the already evaluated FP32 exports below rather than introducing Python/PyTorch or changing the benchmarked model variant. No Hugging Face Python or trust_remote_code runs.

The pinned PinkPixel card identifies the original BiRefNet checkpoint and documents its 2048-pixel input/alpha contract. The pinned onnx-community configuration specifies FP32 SAM 2 at 1024 pixels; that revision has no model card or conversion recipe. Our review therefore relies on graph/IO inspection, actual CPU execution of both downloaded models and the published 22-case output evidence, with failures retained. It does not claim independent verification of the conversion's equivalence to Meta's checkpoint.

Every selected artifact's LFS byte count and SHA-256 was rechecked against its immutable export revision on 2026-10-10 and still matches the catalog. Exact pins, streaming integrity checks, explicit opt-in, original-model licences and evaluated outputs make these community artifacts reviewable; hashes establish identity, not training provenance or upstream endorsement. A compatible author-published ONNX export can replace one through a future reviewed catalog/adapter update and new output comparisons.

All URLs, exact byte counts and SHA-256 hashes are in crates/ml/src/catalog.rs, derived from the pinned exports' Hugging Face LFS metadata. A model update requires a code change and review; there is no mutable remote manifest. Original checkpoint licences apply to their converted weights:

Model Original checkpoint / code Reviewed ONNX export and immutable revision
BiRefNet ZhengPeng7/BiRefNet_HR-matting, BiRefNet PinkPixel/birefnet-hr-matting-onnx
SAM 2.1 Large facebook/sam2.1-hiera-large, facebookresearch/sam2 onnx-community/sam2.1-hiera-large-ONNX

The BiRefNet export has image input and alpha output with sigmoid already included. SAM uses three image embeddings, an ignored padding point with int64 label -1, and normalized box corners scaled to 1024. It returns three logits masks and predicted IoU scores. These contracts are tested through the real runtime with small original synthetic graphs, and the downloaded graphs are inspected/run separately. Community exports should remain part of upstream review: file hashes establish identity, not an independent audit of model training or conversion.

cargo test -p photocraft-ml --features onnx
cargo test -p photocraft-engine -p photocraft-cli --features local-ml
cargo clippy -p photocraft-ml -p photocraft-engine --features photocraft-ml/onnx,local-ml --all-targets -- -D warnings

.github/workflows/local-models.yml is manual-only (workflow_dispatch): no push or pull-request trigger. An explicit run exercises CPU runtime contracts, commands, Preferences, catalogs and native build paths on Windows, Linux and macOS without downloading learned weights. The opt-in full-model smoke test is separate; the download flag is explicit and transfers about 1.8 GB:

PHOTOCRAFT_MODEL_TEST_DIR=/absolute/path/to/test-cache \
PHOTOCRAFT_MODEL_TEST_DOWNLOAD=1 \
  cargo test -p photocraft-ml --features onnx real_models_cpu_smoke -- --ignored --nocapture

Omit PHOTOCRAFT_MODEL_TEST_DOWNLOAD to test existing verified files offline. The 22-case PDF and public supplement include an exploratory comparison on 20 fixed COCO polygon targets plus two visual-only photos. The protocol, model failures, measured scores and reproducible harness are documented in local-models-validation.md. These binary instance references do not establish labelled hair/soft-alpha/transparency matting quality. Representative quality evaluation and web inference remain open, so the smart-features scorecard stays partial.

See validation and comparison for the measured 24 MP timings, before/after Preferences screenshots, actual exported masks, completed three-platform native CI and outstanding packaged-installer/production checks.