Compare commits

...
9 Commits
Author SHA1 Message Date
Brandon ThomasandClaude Opus 5.5 e9a0c49d15 Docs: hardware decode on macOS and 31 GPU effects landed (#30)
ROADMAP honest assessment, Where we are lacking and Running now; README status and codec note. 35 crates (platform).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:50:45 -04:00
Brandon ThomasandClaude Opus 5.5 4aea06e2ae GPU2: standard effects on the GPU compositor (#30)
A media clip whose enabled standard effects all have a GPU implementation
plans as a GPU layer carrying them (render::plan::LayerFx): parameters are
evaluated on the CPU at the frame's time (render::gpufx::FxOp::eval, shared
with the CPU path so both clamp hostile values identically), the source is
drawn into an Rgba32Float working image at the size the CPU decodes it, the
effects run as WGSL compute passes (gpu::fx, fx.wgsl) with the CPU
reference's math, and the result is placed with Motion, opacity and blend
mode as before. Layers without effects keep the single draw.

GPU effects: Brightness & Contrast, ProcAmp, Tint, Black & White, Color
Balance, Leave Color, Change to Color, Color Pass, Color Replace, Channel
Mix, ASC CDL, Gamma Correction, Levels, Extract, Invert, Posterize, Alpha
Adjust, Gaussian / Camera / Directional Blur (+ legacy aliases), Sharpen,
Unsharp Mask, Crop, Edge Feather, Transform, H/V Flip, Mirror, Offset.
Masked effects, opacity masks, non-finite parameters and minifying
Transform/Crop zooms stay on the CPU. Devices without compute shaders run
the effects on the CPU inside the compositor.

Also fixes a debug-build overflow in Image::transformed for non-finite maps.
Parity tests (exact f32 checks and composited plans), a golden test that
plans effect chains as GPU layers, and the stack3-fx playback benchmark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:50:20 -04:00
Brandon Thomas e9a01dbf38 Merge pull request #33 from storytold/pr/hw-decode-videotoolbox
HW1: platform crate + VideoToolbox H.264/HEVC hardware decode (#30)
2026-10-05 07:26:47 -04:00
Brandon Thomas 2e8df0541a Merge pull request #34 from storytold/pr/honest-assessment
Docs: honest parity assessment (ROADMAP, README, agent docs)
2026-10-05 06:32:58 -04:00
Brandon ThomasandClaude Opus 5.5 3d4e6d77bb Docs: honest parity assessment in ROADMAP, README and agent docs
ROADMAP now reports two numbers: the feature checklist (~87%, what exists)
and ready for real work (~50-60%, an honest estimate). Adds a by-dimension
assessment with evidence, a prioritised "Where we are lacking" list (hardware
acceleration, a measured parity tool, real-world media corpus, workflow
acceptance tests, Windows/Linux runtime CI, delivery codecs, plugins, AI, web),
and where we're going. Fixes stale rows (M5 effect count, Running now) and logs
2026-10-05.

README: the stale "45%" status replaced with both numbers and the biggest
gaps; the codec note now lists what imports today and what's missing.

docs/agents.md: agents read the honest assessment first, prefer its gaps,
and report progress as measured vs estimated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:04:09 -04:00
Brandon Thomas 98792afd1c Merge pull request #32 from storytold/pr/gpu-blend-modes
GPU1: blend modes on the GPU compositor (#30)
2026-10-05 05:28:35 -04:00
Brandon ThomasandClaude Opus 5.5 03f5439d9a HW1: plane views borrow the pixel-buffer lock guard (#30)
PlaneView's slices were tied to the pixel buffer's lifetime, not the lock's, so nothing stopped a later edit from reading plane memory after CVPixelBufferUnlockBaseAddress. They now borrow the Locked guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 04:53:28 -04:00
Brandon ThomasandClaude Opus 5.5 c3ec0e85d3 HW1: platform crate + VideoToolbox H.264/HEVC hardware decode (#30)
- New crates/platform (filmcraft-platform, L5): OS media FFI only. The one
  crate allowed `unsafe` (own [lints] with unsafe_code = "deny",
  #[allow(unsafe_code)] only on the videotoolbox module, SAFETY comment on
  every block, safe Result-returning public API). AGENTS.md §0.3 names the
  exception; docs/adr/0001-platform-ffi.md records why and the rules.
- VideoToolbox decoder for avcC / hvcC, 8/10-bit 4:2:0 and 4:2:2: session from
  the sample entry's parameter sets with hardware required, asynchronous
  decode (2 in flight), biplanar CVPixelBuffer copied to planar Yuv8/Yuv16,
  reorder buffer of the stream's own depth, RASL skipped after a starting CRA.
  Output callback under catch_unwind.
- HybridDecoder: transparent mid-stream fallback to the software decoder
  (replays samples since the last restart point, drops already-returned
  pictures). New codecs::software_video_decoder (built-in factories only) and
  codecs::hw (switch, counters, NalStreamInfo shared with software decoders;
  vui_color / sar_par helpers now used by both).
- Settings ▸ Playback ▸ Hardware decoding: Auto / Off (engine applies it to
  the codecs switch); desktop app, CLI/MCP and bench call register().
- perf.stats decode.framesDecoded and decode.hardware counters.
- Tests: bit-exact parity with the software decoders (H.264 High, HEVC Main
  open GOP, Main 10; resets, reseeks, flushes), forced-failure fallback,
  mutation fuzz of samples and parameter sets, cross-platform fallback with a
  stand-in decoder, setting Off, HEVC 4:2:2 vs ffmpeg.
- Bench --hw auto|off; results in docs/performance.md (4K CPU/frame
  ~86-119 ms -> ~4 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 04:53:28 -04:00
Brandon ThomasandClaude Opus 5.5 6102a3794a GPU1: blend modes on the GPU compositor (#30)
All 27 blend modes now composite on the GPU instead of sending the
whole frame to the CPU.

- render::plan: PlanLayer carries its Blend; plan_frame no longer falls
  back to a CPU frame for non-Normal modes. Media, graphic clips,
  multicam angles and CPU-rendered layer images (standard effects) keep
  their clip's mode; inside a transition the mode is Normal, as in the
  CPU reference. execute_cpu composites with each layer's mode.
- gpu: Normal keeps the single fixed-function pass. Dissolve also uses
  fixed-function "over": the CPU's 64-bit pixel hash is reproduced in
  WGSL on u32 pairs (pattern bit-exact). The other 25 modes copy the
  accumulator region under the layer's quad into a backdrop texture and
  composite in fs_blend with the CPU math (sRGB-encoded straight colour,
  W3C formulas, back to linear, same premultiplied mix and clamping).
- Tests: per-mode GPU-vs-CPU parity (gpu crate: overlapping transformed
  layers, partial opacity/alpha, empty/partial backdrop; golden: every
  mode through plan_frame incl. an effect layer, blended clips in a
  cross dissolve), exact-value edge cases (black/white, Divide by 0,
  Dodge/Burn at 0 and 1, zero alpha) within 4.9e-4, Dissolve pattern
  exact, off-output layers, quad bounds.
- Bench: new stack3-blend playback scenario (V2 Screen, V3 Overlay);
  CPU ms/frame 151-214 -> 86-89, Full 19/173 and 138/54 -> 191/1, 192/0.
- Docs: architecture §5, gpu crate docs, testing, performance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 03:25:16 -04:00
57 changed files with 5982 additions and 552 deletions

No files matched your search

+6 -1
View File
@@ -19,7 +19,12 @@ panic path, and fix a crash before building on top of it. The cross-app standard
fallback that is truly correct (never one that silently corrupts a project). An unfinished feature
returns an "unsupported" error. The only exception is a call that is provably infallible from local
code alone (a literal that always parses): a single-item `#[allow(clippy::expect_used)]` with the reason.
3. **No `unsafe`** (`unsafe_code = "forbid"` for the workspace).
3. **No `unsafe`** (`unsafe_code = "forbid"` for the workspace). The single exception is
`crates/platform`, which needs `unsafe` to call the operating system's media APIs (hardware
video decoding through VideoToolbox; later encoding and other OSes) and does nothing else. It
uses `unsafe_code = "deny"` with `#[allow(unsafe_code)]` only on its FFI modules, a
`// SAFETY:` comment on every `unsafe` block, a safe `Result`-returning public API, and the
pure-Rust decoder as the tested fallback ([ADR 0001](docs/adr/0001-platform-ffi.md)).
4. **Every input-derived number is hostile.** Media files, project and interchange files, presets, fonts,
CLI / MCP / control-channel parameters and UI state are untrusted. Index and slice with `get()` (or
validate bounds once, up front, for a hot loop); slice strings only at char boundaries; use
+1 -1
View File
@@ -12,7 +12,7 @@ FilmCraft is a clean-room, open-source, pure-Rust non-linear video editor target
**Read [`AGENTS.md`](AGENTS.md) first; its rules override everything here.**
- **Never crash** ([`AGENTS.md`](AGENTS.md) §0, standard: [`craftrules/standards/never-crash.md`](https://github.com/storytold/craftrules/blob/main/standards/never-crash.md)). People depend on FilmCraft; a crash loses their work, so this outranks feature work. No `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!`/`unimplemented!` or `unsafe` outside tests: return `Result<T, E>` and propagate with `?` (`ok_or`, `let … else`, `if let`). Treat every number from media, project files, presets or CLI/MCP/control params as hostile: `get()` instead of `[i]`, checked/saturating math, no division by possibly-zero values, capped allocations, bounded recursion. `lock().unwrap_or_else(PoisonError::into_inner)`; background jobs run under `catch_unwind`. The panic hook + UI `catch_unwind` (`crates/ui-egui/src/crash.rs`) are the last-resort guard, not a licence. Every crash fix gets a regression test that panicked before. In particular (§1): never use Adobe iconography, images or any other Adobe asset. Every asset must be openly licensed (OSS / public domain / Creative Commons, or original work by a contributor) and must have a `<file>.attribution` sidecar plus an entry in `ATTRIBUTION.md`. `cargo xtask assets` enforces this.
- **Clean-room.** Never read/disassemble anything inside Adobe app bundles (names/listings only). Never copy Adobe icons, presets, LUTs, fonts. Never copy GPL/LGPL/AGPL code (FFmpeg, x264, x265, MLT, Kdenlive, Shotcut, Olive, LAME…). ffmpeg/ffprobe run only as external test oracles / fixture generators — never linked or shipped.
- **Pure Rust** in the product. OS media APIs only via Rust bindings in `crates/platform` (planned; not created yet), optional, behind traits.
- **Pure Rust** in the product. OS media APIs only via Rust bindings in `crates/platform` (hardware decoding; the one crate allowed `unsafe`, see `AGENTS.md` §0.3 and `docs/adr/0001-platform-ffi.md`), optional, behind traits, with our decoders as the fallback.
- **Layering** ([`docs/architecture.md`](docs/architecture.md) §1, enforced by `cargo xtask layers`): nothing below L5 depends on egui/eframe/winit/rfd/cpal. L0 codec/container crates depend only on `filmcraft-bitstream`.
- **Exact time:** all time is `filmcraft_time::Tick` (254 016 000 000/s). Never use f64 seconds for edit math.
- **Everything is a command** (`crates/engine`): id, label, menu path, shortcut, params, enabled(), run(). UI, CLI, control channel and MCP all dispatch by id.
Generated
+71
View File
@@ -1662,6 +1662,7 @@ dependencies = [
"eframe",
"egui",
"filmcraft-engine",
"filmcraft-platform",
"filmcraft-ui-egui",
"log",
"muda",
@@ -1761,6 +1762,7 @@ dependencies = [
"filmcraft-mpegts",
"filmcraft-mxf",
"filmcraft-ogg",
"filmcraft-platform",
"filmcraft-time",
"serde_json",
"tokio",
@@ -2069,6 +2071,24 @@ dependencies = [
"filmcraft-testkit",
]
[[package]]
name = "filmcraft-platform"
version = "0.2.0"
dependencies = [
"filmcraft-codecs",
"filmcraft-color",
"filmcraft-frame",
"filmcraft-isobmff",
"filmcraft-media",
"filmcraft-testkit",
"filmcraft-time",
"log",
"objc2-core-foundation",
"objc2-core-media",
"objc2-core-video",
"objc2-video-toolbox",
]
[[package]]
name = "filmcraft-project"
version = "0.2.0"
@@ -2183,6 +2203,7 @@ dependencies = [
"filmcraft-gpu",
"filmcraft-interchange",
"filmcraft-media",
"filmcraft-platform",
"filmcraft-project",
"filmcraft-render",
"filmcraft-scopes",
@@ -3779,7 +3800,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
dependencies = [
"bitflags 2.13.2",
"block2 0.6.2",
"dispatch2",
"libc",
"objc2 0.6.4",
]
@@ -3820,6 +3843,37 @@ dependencies = [
"objc2-foundation 0.2.2",
]
[[package]]
name = "objc2-core-media"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05ec576860167a15dd9fce7fbee7512beb4e31f532159d3482d1f9c6caedf31d"
dependencies = [
"bitflags 2.13.2",
"block2 0.6.2",
"dispatch2",
"objc2 0.6.4",
"objc2-core-audio",
"objc2-core-audio-types",
"objc2-core-foundation",
"objc2-core-video",
]
[[package]]
name = "objc2-core-video"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d425caf1df73233f29fd8a5c3e5edbc30d2d4307870f802d18f00d83dc5141a6"
dependencies = [
"bitflags 2.13.2",
"block2 0.6.2",
"objc2 0.6.4",
"objc2-core-foundation",
"objc2-core-graphics",
"objc2-io-surface",
"objc2-metal 0.3.2",
]
[[package]]
name = "objc2-encode"
version = "4.1.0"
@@ -3992,6 +4046,23 @@ dependencies = [
"objc2-foundation 0.2.2",
]
[[package]]
name = "objc2-video-toolbox"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05bf9a3c14831a7d9641b0d81d87dd913ee238a012b2fde27db5a84b56f5df3e"
dependencies = [
"bitflags 2.13.2",
"block2 0.6.2",
"objc2 0.6.4",
"objc2-core-foundation",
"objc2-core-graphics",
"objc2-core-media",
"objc2-core-video",
"objc2-foundation 0.3.2",
"objc2-metal 0.3.2",
]
[[package]]
name = "once_cell"
version = "1.21.4"
+1
View File
@@ -51,6 +51,7 @@ filmcraft-format = { path = "crates/format" }
filmcraft-engine = { path = "crates/engine" }
filmcraft-ui-egui = { path = "crates/ui-egui" }
filmcraft-automation = { path = "crates/automation" }
filmcraft-platform = { path = "crates/platform" }
filmcraft-testkit = { path = "crates/testkit" }
# foundation
+18 -3
View File
@@ -184,7 +184,7 @@ No FFmpeg inside. The video codecs, AAC, Opus and the containers are our own Rus
| **Matroska / WebM** | ✓ | | Lacing, Cues, header stripping, HDR colour metadata |
| **Stills** | ✓ | ✓ | Import PNG, JPEG, GIF, WebP, TIFF and BMP; export PNG sequences and animated GIF |
MXF and hardware decode are next ([roadmap](ROADMAP.md)).
Also imported: MPEG-2 / MPEG-1 video, AC-3, MP2, MXF (OP1a / OP-Atom), MPEG transport and program streams (AVCHD, broadcast, DVD), Ogg and image sequences. Not yet: HEVC and AV1 export, camera RAW, E-AC-3. Hardware decoding works on macOS (VideoToolbox), with Linux and Windows to follow ([#30](https://github.com/storytold/filmcraft/issues/30)).
<br>
@@ -266,11 +266,26 @@ The control protocol is documented in [docs/control-protocol.md](docs/control-pr
| [docs/control-protocol.md](docs/control-protocol.md) | Control-channel and MCP reference |
| [docs/project-files.md](docs/project-files.md) | `.fcproj` format, schema migrations, auto-save and crash recovery |
| [docs/graphics.md](docs/graphics.md) · [docs/captions.md](docs/captions.md) | Text engine, graphic clips and tools; caption tracks and formats |
| [ROADMAP.md](ROADMAP.md) | Milestones and estimates |
| [ROADMAP.md](ROADMAP.md) | Honest assessment, what's missing, milestones and estimates |
## Status
FilmCraft is young and moving fast: roughly 45% of the way to Premiere Pro parity, with editing, trimming, colour, keyframes, titles, captions, mixing, codecs and export working today. [ROADMAP.md](ROADMAP.md) tracks every milestone with estimates.
FilmCraft is young and moving fast. Editing, trimming, multicam, colour, keyframes, effects, titles, captions, mixing, codecs and export work today.
We track two numbers ([ROADMAP.md](ROADMAP.md#honest-assessment-2026-10-05)):
- **Feature checklist: ~87%.** Premiere Pro's menu items, effects, transitions, panels and formats that exist in FilmCraft.
- **Ready for real work: ~50–60%.** Our honest estimate of how close FilmCraft is to replacing Premiere on real projects.
The biggest gaps today:
- **Speed on big footage.** Hardware decoding works on macOS; Windows and Linux have no hardware path yet. Blend modes and the most common effects run on the GPU, but Lumetri, keys, export and encoding still run on the CPU ([#30](https://github.com/storytold/filmcraft/issues/30)).
- **No plugins.** No VST3 / Audio Units or OpenFX hosting.
- **Delivery codecs.** H.264 is our only delivery-codec export; no HEVC or AV1 export yet.
- **Real-world media and platforms.** Our decoders are bit-exact on conformance streams, but camera and phone files in the wild are less tested. Windows and Linux get far less testing than macOS.
- **AI features.** Few so far; speech to text is optional and off by default.
Bug reports with real footage are the most useful thing you can send us: [open an issue](https://github.com/storytold/filmcraft/issues) or tell us in [Discord](https://discord.gg/artcraft).
## Architecture
+90 -6
View File
@@ -1,10 +1,88 @@
# FilmCraft Roadmap
Progress toward feature parity with Adobe Premiere Pro, with estimates. Updated as milestones land.
Progress toward parity with Adobe Premiere Pro, with estimates. Updated as milestones land.
**Last updated:** 2026-10-02 · **Overall parity:** ~87% (measured scorecard below) · **Code:** 34 crates, 1605 tests
**Last updated:** 2026-10-05 · **Feature checklist:** ~87% · **Ready for real work:** ~50–60% · **Code:** 35 crates, 1605+ tests
## Parity scorecard
Two numbers, because they answer different questions. The **feature checklist** counts whether
Premiere's menu items, effects, panels and formats exist in FilmCraft. **Ready for real work** is
our honest estimate of how close FilmCraft is to replacing Premiere for an editor on real
projects: depth, correctness, robustness, speed and the pro ecosystem. The gap between the two
is the work that matters most now. Agents: read [Where we are lacking](#where-we-are-lacking)
before choosing work.
## Honest assessment (2026-10-05)
### How the checklist number is made, and what it doesn't tell you
- **Partly measured.** The menu figure comes from diffing a dump of Premiere 26.5's menus against
`filmcraft-cli commands`. The effects, transitions and audio-effects counts are checked by unit
tests against Premiere's Effects panel tree (`premiere_26_video_effects_catalogue`,
`premiere_audio_set_is_complete_and_foldered`, `premiere_26_transition_tree`).
- **Partly agent-reported.** The other areas' percentages are written by the agents doing the work.
No automated parity check runs in CI (PhotoCraft has `cargo xtask parity`; we don't yet), so
nothing stops a number drifting upward.
- **Presence, not quality.** An item counts once it exists. An effect described as an
"approximation" counts the same as an exact one. Nothing compares our rendering with
Premiere's.
- **Weights understate what users hit first.** Performance carries 5%. Plugins and AI features
aren't in the scorecard at all.
### By dimension
| Dimension | Checklist says | Honest estimate | Evidence |
|---|---|---|---|
| **Feature breadth** (menus, commands, panels) | ~86–95% | **~85%** | Menu diff against Premiere 26.5; effect, transition and audio-effect catalogues complete by test. The long tail is mostly there. |
| **Correctness in depth** | not tracked | **~60–70%** | A first-time contributor found four real bugs in core paths within hours (#5–#8: AIFF import failed, Slide left linked audio behind, transitions didn't follow ripple trims, export ignored start/end times). User reports: Color Matte always grey with no picker (#29); ALSA audio error on Linux (#23). Settings are "most wired", not all. |
| **Codecs and media** | ~90% | **~75%** | Our decoders are bit-exact on conformance streams. Real camera and phone media (variable frame rate, damaged files, unusual containers) is far less tested. No camera RAW (RED, BRAW, ARRIRAW), no E-AC-3. |
| **Export** | ~92% | **~70%** | Delivery: H.264 only (ProRes, DNxHR and image sequences cover mastering). No HEVC or AV1 export. AAF and OMF have never been validated in Avid Media Composer or Pro Tools. Export renders and encodes on the CPU only. |
| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS only (VideoToolbox, #33). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects, export and encoding are CPU. Linux and Windows have no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. |
| **Stability** | not tracked | **improving, unproven** | The never-crash pass (no panics in product code, last-resort guards) landed on 2026-10-04. There's no field record from real users' projects yet. |
| **Plugins and pro ecosystem** | not tracked | **~0–10%** | No audio plugin hosting (VST3 / Audio Units) and no third-party video effects (OpenFX). For many professional editors this alone rules FilmCraft out. Team Projects and Productions are out of scope by design. |
| **AI features** | partial | **~25–35%** | Speech to text exists but is off in default builds (the `whisper` feature). Enhance Speech is a DSP chain, not a model. Auto Reframe is approximate. No Generative Extend, and no media-intelligence search or auto colour. |
| **Platforms** | not tracked | **macOS solid; Windows and Linux thin** | Development and testing happen on macOS. Windows and Linux are built for releases but barely exercised at runtime. The web build is a compile target; the browser app shell isn't finished. |
| **UI fidelity** | ~85% | **~80%** | Agents compare with Premiere reference screenshots; this hasn't been checked independently. |
### Where we are lacking
In priority order. Agents should prefer this work over adding more checklist items.
1. **Hardware acceleration** (#30), the most visible gap to users:
- hardware decode on Linux (VA-API) and Windows (Media Foundation);
- zero-copy decoded frames into wgpu;
- the remaining effects on the GPU: Lumetri, keys, Vignette, Video Limiter, masks (31 common effects are done);
- GPU export, then hardware encode (H.264 / HEVC).
2. **A measured parity number.** Add `cargo xtask parity`, run in CI. It should cover menus,
commands, effects, transitions, panels, preferences and formats against the Premiere snapshot in
`plan/premiere/`. It should report presence and fidelity separately (exact, approximate, stub).
3. **Real-world media corpus.** Phone, camera and screen recordings, variable frame rate, damaged
files, long GOPs, odd containers. Each gets an import → edit → export round-trip test with an
ffprobe / ffmpeg oracle.
4. **Workflow acceptance tests.** Scripted MCP sessions that do real jobs end to end: assemble and
trim a scene, colour it, mix and loudness-normalise it, caption it, deliver it, round-trip
through XML / AAF.
5. **Windows and Linux at runtime.** CI that runs the test suite and a headless playback/export
smoke test on both, not just release builds. Fix the Linux audio path (#23).
6. **Delivery codecs.** HEVC and AV1 export: hardware encoders first, and `rav1e` (BSD-2) for AV1.
7. **Plugin hosting** (VST3 / Audio Units, then OpenFX). Needs FFI, so it falls under the isolated
`unsafe` crate rule (AGENTS.md §0.3). **Owner decision needed** before starting.
8. **AI features** with openly licensed local models: transcription on by default, text-based
editing, speech enhancement, reframing.
9. **The web app shell**: file access, WebCodecs, audio.
### Where we're going
**Next:** close items 1–5. Hardware acceleration fixes the gap users notice first (CPU-bound 4K playback and export). The
parity tool, media corpus and workflow tests make the progress number trustworthy. Windows and
Linux CI makes "runs everywhere" true.
**After that:** delivery codecs, plugin hosting (pending the owner's decision) and AI features,
measured with the same tools.
**Goal:** "ready for real work" at 85%+, measured rather than estimated, on all three desktop
platforms.
## Feature checklist scorecard
Measured against Premiere Pro 26.5 on this machine. Menu items: the native menu bar dump, minus
Adobe-cloud-only items (Team Projects, Productions, Firefly, Stock, Dynamic Link, account/help pages),
@@ -22,11 +100,14 @@ matched against `filmcraft-cli commands` and the UI command table. Effects: the
| Graphics and captions | 8% | text engine, shapes, per-character styles, our own graphics templates (.fcgt, 8 built-ins, export/install/edit), rolls/crawls, responsive pins + time, captions SRT/VTT/SCC/MCC/STL/TTML/DFXP, transcripts | ~85% |
| Export | 8% | own H.264/AAC, ProRes, DNxHR, MXF OP1a/OP-Atom, image sequences, GIF, WAV, 5.1; preset library + user presets, full Export-mode settings, queue, Quick Export; AAF (Edit Protocol) import/export, OMF 2.0 export; no AAF/OMF validation against Avid/Pro Tools yet | ~92% |
| Preferences and project management | 5% | Settings dialog with 16 categories (most settings wired), project settings, scratch disks, search bins, templates | ~80% |
| Performance | 5% | 1080p real-time, 3×1080p; 4K H.264 real-time at load ≤140 (−23% decoder cycles, opt-in draft decoding at 1/2–1/4); HEVC 2.5× cheaper, catch-up decoding, `cargo xtask bench` + `perf.stats`; 8K and AV1 not yet real-time on a loaded machine | ~62% |
| Performance | 5% | 1080p real-time, 3×1080p; 4K H.264 real-time at load ≤140 (−23% decoder cycles, opt-in draft decoding at 1/2–1/4); HEVC 2.5× cheaper, catch-up decoding, `cargo xtask bench` + `perf.stats`; blend modes on the GPU (#32); 8K and AV1 not yet real-time on a loaded machine; no hardware path on Windows/Linux (see the honest assessment: ~35–40% against Premiere) | ~62% |
| **Weighted total** | | | **~87%** |
## Estimate to parity
This estimate is for the **feature checklist**. Closing the honest-assessment gaps (above) is the
"Robust on real-world material" row, plus plugins and AI features, which it doesn't include.
Measured throughput in the last work block (2026-10-01 night → 10-02): five Opus 5.5 agents in
parallel for ~4.5 wall-clock hours (~18 agent-hours including integration) moved parity by ~4 points
(≈4–5 agent-hours per point), on a machine that was heavily overloaded (load 150–300 on 14 cores) and
@@ -62,7 +143,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
| M2 | H.264 decoder | ✅ | Own decoder, bit-exact on 37+ streams, 500–600 fps 1080p | — | — |
| M3 | Editing core | 🟡 | Edit algebra (insert/overwrite/razor/lift/extract/ripple/roll/slip/slide/rate-stretch/nest/paste), tools, markers, trim mode + Trim Monitor + dynamic J/K/L trimming, Keyboard Shortcuts editor with FilmCraft/Premiere/FCP/Avid presets; Edit/Clip/File menu commands (Label colours and Select Label Group, Paste/Remove Attributes, Select All Matching, Remove Unused, Consolidate Duplicates, Sequence From Clip, Bin From Selection, Offline File, Close Project, Make/Edit Subclip, Modify Audio Channels/Timecode, Frame Hold Options/Add Frame Hold/Insert Frame Hold Segment, Time Interpolation with frame blending, Fit/Fill frame, Breakout to Mono, Extract Audio, Replace With Clip; M3.11: Scene Edit Detection (pure-Rust cut detection, background job), Normalize Mix Track, Simplify Sequence, Transcribe Sequence, Find/Find Next and search bins, Automate to Sequence, Edit Original, Edit Offline, Source Settings, Update Metadata (XMP), Generate Audio Waveform, Project Settings General/Scratch Disks, Get Media File Properties, Save as Template, Selection as FilmCraft Project, Avid Log Exchange export, Flash Cue markers, Dynamic Audio Waveforms, Reveal Log Files, System Compatibility Report); M3.12 keyboard parity: Premiere's default keyboard (~115 keyboard-only commands: edit-point navigation on targeted/any track, select clip at playhead/next/previous, extend edit to playhead, nudge/slip/slide selection, target and source-patch toggles, clip volume ±1 dB/many, frame maximize/full screen/panel cycling, monitor zoom, track heights, Project and Text panel keyboard navigation, text size/leading/alignment, Export Frame, poster frames; see docs/keyboard.md); multicam (Create Multi-Camera Source Sequence, Multi-Camera view with live switching on 1–9, angle switching, Enable/Flatten, Edit Cameras) and sync (Synchronize, Merge Clips; In/Out/timecode/marker/audio — GCC-PHAT, sample-accurate) | Multicam paging >16 angles and grid thumbnails, optical flow (renders as frame blending), 39 Premiere default shortcuts skipped with reasons in docs/keyboard.md (Productions, AI/cloud tools, work area bar, Production/Search panels) | 6–10 |
| M4 | Playback | 🟡 | Audio-clock master, prefetch with cancellation, J/K/L, correct dropped-frame stats, playback resolution, render bar + content-hashed render previews, App Nap opt-out; 1080p H.264 and 3 stacked 1080p streams play with 0 dropped frames | 4K under load, 8K, frame-threaded AV1 decode, reduced-resolution decode for multicam grids | 6–10 |
| M5 | Effects, keyframes, GPU | 🟡 | ~60 CPU effects, 30 transitions, keyframes + value/velocity graphs, wgpu compositor, effect + opacity masks (ellipse/polygon/pen, feather, expansion, modes; CPU/WGSL parity), mask tracking (Lucas–Kanade + RANSAC), adjustment layers, effect presets (built-in + user, JSON import/export) | Full ~150-effect catalogue, WGSL parity for all effects and masks in the live GPU path, Warp Stabilizer, Morph Cut | 20–30 |
| M5 | Effects, keyframes, GPU | 🟡 | All 93 Premiere 26 video effects and 84 transitions (CPU), blend modes and 31 common effects on the GPU (#32, GPU2), keyframes + value/velocity graphs, wgpu compositor, effect + opacity masks (ellipse/polygon/pen, feather, expansion, modes; CPU/WGSL parity), mask tracking (Lucas–Kanade + RANSAC), adjustment layers, effect presets (built-in + user, JSON import/export) | Effects on the GPU (in progress, #30), GPU masks in the live path, GPU export, exact Warp Stabilizer (3-D) and Morph Cut | 20–30 |
| M6 | Export | ✅ | Own H.264 encoder (High/Main/Baseline, B-frames, VBR/CBR/2-pass) → MP4 or QuickTime + own AAC; ProRes, DNxHR, MJPEG, PNG/TIFF/BMP sequences (Premiere-style numbering), GIF, WAV, AIFF; background jobs; M6.5: Export mode parity (frame size / rate / scaling / pixel aspect, profile / level, CBR / VBR 1- and 2-pass, keyframe distance, audio codec / rate / channels / bitrate / sample size, multiplexer, burn-in or sidecar captions, image / name / timecode overlays, video limiter, loudness normalization with true-peak limiter, metadata, ranges, estimated size, summary), 24 built-in presets + user presets with favourites and import/export (Preset Manager), export queue (reorder, cancel, retry, several sequences / ranges), Quick Export; `export.*` commands and `filmcraft-cli export --preset` | Interlaced encoding, 5.1 audio, smart render, publishing destinations | 6–8 |
| M7 | Audio | 🟡 | Mixer graph (tracks → submixes → Mix, pre/post-fader inserts and sends, latency-compensated, sample-accurate, ~6× realtime for 24 tracks × 3 effects on one core), Audio Track Mixer + Audio Clip Mixer panels, track automation (Off/Read/Latch/Touch/Write, recorded live while playing, thinned to keyframes, timeline lanes with pen editing), solo/solo-safe, channel mapping basics, peak + BS.1770 loudness meters (match ffmpeg), DSP crate with 16 clip/track effects, Audio Gain (set/adjust/normalize), Constant Power / Constant Gain / Exponential Fade, Essential Sound (types, Loudness auto-match, Repair incl. DeEss/DeReverb, Clarity, Creative, Ducking keyframes, presets; full Dialogue chain 22× realtime) | Music duration remix, ML speech enhancement, 5.1 panner and multichannel buses, voice-over record, effect editor windows (EQ curve), remaining effects (multiband, convolution reverb), clip-mixer automation recording | 6–9 |
| M8 | Colour | 🟡 | Lumetri: basic, creative + looks, RGB & hue curves, wheels, HSL secondary, vignette, section bypass; Input LUT / Look LUT (.cube 1D/3D/shaper, .3dl; tetrahedral CPU + WGSL; project LUT library; built-in camera conversions); Colour Match (Oklab tonal-range statistics, skin protection, solved in Lumetri wheels); colour management: Rec. 709 / Rec. 2100 PQ / HLG working spaces + wide gamut, Interpret Footage colour space (S-Log3, V-Log, Canon Log 2/3, LogC3/4, Apple Log, D-Log from published specs), metadata auto-detect, BT.2390 tone mapping, gamut mapping, HDR export signalling (VUI/colr/mdcv/clli/SEI, ffprobe-verified); Lumetri Scopes: Vectorscope YUV (75/100 % targets, skin-tone line) and HLS, Histogram, Parade (RGB/YUV/RGB-White), Waveform (RGB/Luma/YC/YC no Chroma), presets, Rec. 601/709/2100, 8-bit/float/HDR (cd/m²) scales, multi-scope grid, `scopes.read` | HDR-aware Lumetri maths, macOS EDR monitors, mastering metadata → tone-map peak, D-Log M (no published formula), HSL Secondary refine | 3–5 |
@@ -73,10 +154,13 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
## Running now
- Nothing; next: remaining Premiere menu items (Settings categories, Scene Edit Detection, Normalize Mix Track, OMF/AAF, Find), AV1 single-thread re-measure on an idle machine
- **Hardware acceleration (#30):** landed: blend modes on the GPU (#32); VideoToolbox hardware decode + the `platform` FFI crate (#33); 31 common effects on the GPU (GPU2). Next: VA-API (Linux) and Media Foundation (Windows) decode, zero-copy upload, Lumetri on the GPU.
- **Next:** [Where we are lacking](#where-we-are-lacking) items 1–5. GPU export waits for the export-crate work (#19 and the never-crash follow-up) to land.
## Log
- **2026-10-05:** honest assessment added (checklist ~87% vs ready for real work ~50–60%). Hardware acceleration started (#30): blend modes on the GPU (#32); VideoToolbox decode + `platform` crate, the one crate allowed `unsafe` (#33). Community fixes landed: AIFF import, export start/end ranges, transitions following ripple trims, Slide moving linked audio (#5–#8).
- **2026-10-04 (later):** interchange + MXF export (M11.4/M6.6): own compound-file container (`crates/cfb`), AAF Edit Protocol import/export (embedded/linked/consolidated audio, video mixdown, handles, breakout to mono), OMF 2.0 export (Bento), MXF OP1a/OP-Atom writer (DNxHR/ProRes/H.264/PCM) as export formats. 1605 tests.
- **2026-10-04:** MPEG-2 (M9.15): own MPEG-2/MPEG-1 video decoder (4:2:2, interlaced, field pictures; IEEE 1180 IDCT), MPEG TS/PS demux (AVCHD .mts, .m2ts, .ts, .mpg/.vob/.mod), own AC-3 decoder (ATSC A/52), MP2, LATM AAC, Blu-ray/DVD LPCM, MPEG-2 in MXF/MOV/MP4/MKV. 1562 tests.
+1
View File
@@ -21,6 +21,7 @@ whisper = ["filmcraft-engine/whisper", "filmcraft-engine/speech-download"]
filmcraft-engine = { workspace = true }
filmcraft-media = { workspace = true }
filmcraft-codecs = { workspace = true }
filmcraft-platform = { workspace = true }
filmcraft-mxf = { workspace = true }
filmcraft-mpegts = { workspace = true }
filmcraft-ogg = { workspace = true }
+2
View File
@@ -155,6 +155,8 @@ async fn main() {
println!("filmcraft-cli {}", env!("CARGO_PKG_VERSION"));
return;
}
// OS hardware video decoders (VideoToolbox on macOS) in front of our own, as in the desktop app.
let _ = filmcraft_platform::register();
let a = Args::parse(std::env::args().skip(1));
let Some(cmd) = a.pos(0) else { usage("missing subcommand") };
match cmd {
+1
View File
@@ -18,6 +18,7 @@ whisper = ["filmcraft-engine/whisper", "filmcraft-engine/speech-download"]
[dependencies]
filmcraft-engine = { workspace = true }
filmcraft-ui-egui = { workspace = true }
filmcraft-platform = { workspace = true }
egui = { workspace = true }
eframe = { workspace = true, features = ["default_fonts", "wgpu", "accesskit", "wayland", "x11"] }
serde_json = { workspace = true }
+3
View File
@@ -73,6 +73,9 @@ fn main() -> eframe::Result {
// Panics anywhere go to <data dir>/Logs/crash-<day>.log with a backtrace; the UI pass and
// frame workers catch them and keep running (see filmcraft_ui_egui::crash).
filmcraft_ui_egui::crash::install(data_dir.clone().or_else(default_data_dir).map(|d| d.join("Logs")));
// OS hardware video decoders (VideoToolbox on macOS) in front of our own; Settings ▸ Playback ▸
// Hardware decoding switches them off. Unsupported streams and failures use our decoders.
log::info!("hardware decoding: {:?}", filmcraft_platform::register());
let options = eframe::NativeOptions {
viewport: egui::ViewportBuilder::default()
.with_title("FilmCraft")
+19 -2
View File
@@ -33,6 +33,9 @@ pub struct GopStats {
pub skipped: u64,
/// Frames decoded in draft mode (reduced-resolution playback, [`VideoDecoder::set_draft`]).
pub draft: u64,
/// Pictures decoders output (hardware and software; [`crate::hw::hw_stats`] counts the
/// hardware ones).
pub frames: u64,
}
impl GopStats {
@@ -56,6 +59,12 @@ static EVICTED: AtomicU64 = AtomicU64::new(0);
static DECODE_NS: AtomicU64 = AtomicU64::new(0);
static SKIPPED: AtomicU64 = AtomicU64::new(0);
static DRAFT: AtomicU64 = AtomicU64::new(0);
static FRAMES: AtomicU64 = AtomicU64::new(0);
/// Count pictures a decoder call output.
fn count_frames<T>(out: &[T]) {
FRAMES.fetch_add(out.len() as u64, Ordering::Relaxed);
}
/// Run a decoder call, adding its wall time to the process-wide counter.
fn timed<R>(f: impl FnOnce() -> R) -> R {
@@ -76,6 +85,7 @@ pub fn gop_stats() -> GopStats {
decode_ns: DECODE_NS.load(Ordering::Relaxed),
skipped: SKIPPED.load(Ordering::Relaxed),
draft: DRAFT.load(Ordering::Relaxed),
frames: FRAMES.load(Ordering::Relaxed),
}
}
@@ -91,6 +101,7 @@ impl std::ops::Sub for GopStats {
decode_ns: self.decode_ns - o.decode_ns,
skipped: self.skipped - o.skipped,
draft: self.draft - o.draft,
frames: self.frames - o.frames,
}
}
}
@@ -252,6 +263,7 @@ impl GopCache {
/// Store decoder output (in presentation order) and advance `out_max`.
fn store_output(&self, st: &mut State, out: Vec<crate::video::DecodedFrame>) {
count_frames(&out);
for d in out {
st.out_max = st.out_max.max(d.pts);
self.store(st, d.pts, d.frame, d.draft);
@@ -419,6 +431,7 @@ impl GopCache {
let k = *next;
let data = s.read(k)?;
let pics = timed(|| d.decode(&data, s.pts(k)))?;
count_frames(&pics);
*next += 1;
found |= pics.iter().any(|p| p.pts == want_pts);
out.extend(pics);
@@ -427,7 +440,9 @@ impl GopCache {
}
}
if !found {
out.extend(timed(|| d.flush()));
let pics = timed(|| d.flush());
count_frames(&pics);
out.extend(pics);
*next = usize::MAX;
}
Ok(out)
@@ -451,7 +466,9 @@ impl GopCache {
if st.spare.len() < 16 {
st.spare.push(dec);
}
for d in res? {
let res = res?;
count_frames(&res);
for d in res {
self.store(&mut st, d.pts, d.frame, d.draft);
}
st.at_or_before(want_pts, true).ok_or_else(|| CodecError::Decode("frame not produced".into()))
+350
View File
@@ -0,0 +1,350 @@
//! Support for hardware (OS) video decoders registered by `filmcraft-platform`: the
//! Settings ▸ Playback ▸ Hardware decoding switch, the hardware counters `perf.stats` reports, and
//! what such a decoder needs from an `avcC` / `hvcC` sample entry to behave exactly like our
//! software decoder for the same stream (parameter sets, cropping, colour, pixel aspect, reorder
//! depth, random-access and disposable samples).
//!
//! The software decoders stay the reference: a hardware factory declines streams it cannot
//! decode, and a hardware decoder that fails mid-stream switches to
//! [`crate::software_video_decoder`], so registering one never makes a file undecodable.
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use filmcraft_isobmff::{CodecConfig, SampleEntry};
use crate::video::{avcc_length_size, h264_disposable, hevc_disposable, hvcc_length_size_and_tid, sar_par, vui_color};
use crate::{CodecError, Result};
static ENABLED: AtomicBool = AtomicBool::new(true);
/// Settings ▸ Playback ▸ Hardware decoding: `Auto` (true, the default) lets registered hardware
/// decoders take streams they support; `Off` (false) makes every new decoder a software one.
/// Decoders already created keep running: a source picks the change up when it is reopened.
pub fn set_hardware_decoding(on: bool) {
ENABLED.store(on, Ordering::Relaxed);
}
/// Whether hardware decoders may be used for new decoders ([`set_hardware_decoding`]).
pub fn hardware_decoding() -> bool {
ENABLED.load(Ordering::Relaxed)
}
/// Process-wide hardware decoding counters (they only grow; subtract two snapshots to measure an
/// interval). Frames decoded in software are `GopStats::frames - frames`.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct HwStats {
/// Pictures output by hardware decoders.
pub frames: u64,
/// Hardware decoders created (one per decoder instance; a seek reuses it).
pub sessions: u64,
/// Streams a hardware factory handed to the software decoder (unsupported profile, size,
/// chroma format or bit depth, no hardware, or the switch is Off).
pub declined: u64,
/// Hardware decoders that failed mid-stream and continued in software.
pub fallbacks: u64,
}
static FRAMES: AtomicU64 = AtomicU64::new(0);
static SESSIONS: AtomicU64 = AtomicU64::new(0);
static DECLINED: AtomicU64 = AtomicU64::new(0);
static FALLBACKS: AtomicU64 = AtomicU64::new(0);
/// The counters so far.
pub fn hw_stats() -> HwStats {
HwStats {
frames: FRAMES.load(Ordering::Relaxed),
sessions: SESSIONS.load(Ordering::Relaxed),
declined: DECLINED.load(Ordering::Relaxed),
fallbacks: FALLBACKS.load(Ordering::Relaxed),
}
}
/// A hardware decoder output `n` pictures.
pub fn note_hw_frames(n: usize) {
FRAMES.fetch_add(n as u64, Ordering::Relaxed);
}
/// A hardware decoder was created.
pub fn note_hw_session() {
SESSIONS.fetch_add(1, Ordering::Relaxed);
}
/// A hardware factory declined a stream it is responsible for.
pub fn note_hw_declined() {
DECLINED.fetch_add(1, Ordering::Relaxed);
}
/// A hardware decoder switched to software mid-stream.
pub fn note_hw_fallback() {
FALLBACKS.fetch_add(1, Ordering::Relaxed);
}
/// The two codecs whose `avcC` / `hvcC` streams hardware decoders take.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum NalCodec {
H264,
Hevc,
}
/// What a decoder needs to know about a length-prefixed H.264 / HEVC stream, from its sample
/// entry's configuration record and active sequence parameter set (parsed with the software
/// decoders' own parsers, so both decoders derive the same values).
#[derive(Clone, Debug)]
pub struct NalStreamInfo {
pub codec: NalCodec,
/// NAL unit length-prefix size of the samples (1, 2 or 4).
pub length_size: usize,
/// HEVC: highest TemporalId (numTemporalLayers - 1), when signalled.
pub highest_tid: Option<u8>,
/// The parameter-set NAL units of the record (header included, emulation prevention kept),
/// in record order: H.264 SPS then PPS; HEVC VPS, SPS, PPS. SEI arrays are left out.
pub parameter_sets: Vec<Vec<u8>>,
/// Coded (decoded) picture size in luma samples.
pub coded: (u32, u32),
/// Output (cropped) rectangle in luma samples: x, y, width, height.
pub crop: (u32, u32, u32, u32),
/// 0 monochrome, 1 4:2:0, 2 4:2:2, 3 4:4:4.
pub chroma_format_idc: u32,
pub bit_depth_luma: u32,
pub bit_depth_chroma: u32,
/// Field (PAFF / MBAFF) coding (H.264 `frame_mbs_only_flag` 0).
pub interlaced: bool,
/// H.264 `profile_idc` / HEVC `general_profile_idc`.
pub profile_idc: u8,
/// Colour of every output picture, as the software decoder reports it.
pub color: filmcraft_color::ColorInfo,
/// Pixel aspect ratio, as the software decoder reports it.
pub par: (u32, u32),
/// Pictures that may precede a picture in decoding order and follow it in output order
/// (H.264 `max_num_reorder_frames`, HEVC `sps_max_num_reorder_pics`).
pub reorder: usize,
}
/// Length-prefixed parameter-set entries (`u16` length + bytes) starting at `pos`.
fn read_sets(rec: &[u8], pos: &mut usize, count: usize, out: &mut Vec<Vec<u8>>) -> Result<()> {
for _ in 0..count {
let len = rec
.get(*pos..*pos + 2)
.map(|b| u16::from_be_bytes([b[0], b[1]]) as usize)
.ok_or_else(|| CodecError::Decode("configuration record truncated".into()))?;
*pos += 2;
let nal = rec.get(*pos..*pos + len).ok_or_else(|| CodecError::Decode("configuration record truncated".into()))?;
*pos += len;
if !nal.is_empty() {
out.push(nal.to_vec());
}
}
Ok(())
}
impl NalStreamInfo {
/// The stream info of an `avcC` / `hvcC` sample entry (`None` for other codecs).
pub fn from_entry(e: &SampleEntry) -> Option<Result<Self>> {
match &e.codec {
CodecConfig::Avc(a) => Some(Self::from_avcc(&a.to_bytes())),
CodecConfig::Hevc(c) => Some(Self::from_hvcc(&c.to_bytes())),
_ => None,
}
}
/// From an `avcC` (AVCDecoderConfigurationRecord) payload.
pub fn from_avcc(avcc: &[u8]) -> Result<Self> {
if avcc.len() < 7 || avcc[0] != 1 {
return Err(CodecError::Decode("bad avcC record".into()));
}
let mut sets = Vec::new();
let mut pos = 6;
read_sets(avcc, &mut pos, (avcc[5] & 0x1f) as usize, &mut sets)?;
let nsps = sets.len();
let npps = *avcc.get(pos).ok_or_else(|| CodecError::Decode("avcC truncated".into()))? as usize;
pos += 1;
read_sets(avcc, &mut pos, npps, &mut sets)?;
let sps_nal = sets.iter().take(nsps).find(|n| n.first().is_some_and(|h| h & 0x1f == 7)).ok_or_else(|| CodecError::Decode("avcC has no SPS".into()))?;
if sets.len() == nsps {
return Err(CodecError::Decode("avcC has no PPS".into()));
}
let sps = filmcraft_h264::params::Sps::parse(&filmcraft_bitstream::unescape_rbsp(sps_nal.get(1..).unwrap_or_default()))
.map_err(|e| CodecError::Decode(e.to_string()))?;
let crop = sps.crop_rect();
let vui = sps.vui.clone().unwrap_or_default();
Ok(Self {
codec: NalCodec::H264,
length_size: avcc_length_size(avcc),
highest_tid: None,
parameter_sets: sets,
coded: (sps.width(), sps.height()),
crop,
chroma_format_idc: sps.chroma_format_idc,
bit_depth_luma: sps.bit_depth_luma,
bit_depth_chroma: sps.bit_depth_chroma,
interlaced: !sps.frame_mbs_only,
profile_idc: sps.profile_idc,
color: vui_color(crop.2, crop.3, vui.matrix_coefficients, vui.transfer_characteristics, vui.full_range),
par: sar_par(vui.sar),
// as the software decoder's DPB (`max_reorder`)
reorder: sps.max_num_reorder_frames().min(sps.max_dpb_frames()),
})
}
/// From an `hvcC` (HEVCDecoderConfigurationRecord) payload.
pub fn from_hvcc(hvcc: &[u8]) -> Result<Self> {
if hvcc.len() < 23 || hvcc[0] != 1 {
return Err(CodecError::Decode("bad hvcC record".into()));
}
let (length_size, highest_tid) = hvcc_length_size_and_tid(hvcc);
let mut sets = Vec::new();
let mut pos = 23;
for _ in 0..hvcc[22] {
let hdr = hvcc.get(pos..pos + 3).ok_or_else(|| CodecError::Decode("hvcC truncated".into()))?;
let (kind, n) = (hdr[0] & 0x3f, u16::from_be_bytes([hdr[1], hdr[2]]) as usize);
pos += 3;
let mut arr = Vec::new();
read_sets(hvcc, &mut pos, n, &mut arr)?;
if (32..=34).contains(&kind) {
sets.extend(arr);
}
}
let nal_type = |n: &[u8]| n.first().map(|h| (h >> 1) & 0x3f);
let sps_nal = sets.iter().find(|n| nal_type(n) == Some(33)).ok_or_else(|| CodecError::Decode("hvcC has no SPS".into()))?;
if !sets.iter().any(|n| nal_type(n) == Some(32)) || !sets.iter().any(|n| nal_type(n) == Some(34)) {
return Err(CodecError::Decode("hvcC lacks a VPS or PPS".into()));
}
let sps = filmcraft_hevc::params::Sps::parse(&filmcraft_bitstream::unescape_rbsp(sps_nal.get(2..).unwrap_or_default()))
.map_err(|e| CodecError::Decode(e.to_string()))?;
let crop = sps.crop_rect();
let vui = sps.vui.clone().unwrap_or_default();
Ok(Self {
codec: NalCodec::Hevc,
length_size,
highest_tid,
parameter_sets: sets,
coded: (sps.width, sps.height),
crop,
chroma_format_idc: sps.chroma_format_idc,
bit_depth_luma: sps.bit_depth_luma,
bit_depth_chroma: sps.bit_depth_chroma,
interlaced: false,
profile_idc: sps.ptl.profile_idc,
color: vui_color(crop.2, crop.3, vui.matrix_coefficients, vui.transfer_characteristics, vui.full_range),
par: sar_par(vui.sar),
reorder: sps.max_num_reorder as usize,
})
}
/// NAL unit types of a sample (stops at the first malformed length prefix).
pub fn nal_types(&self, sample: &[u8]) -> Vec<u8> {
self.nals(sample).into_iter().filter_map(|n| n.first().map(|&h| self.nal_type(h))).collect()
}
/// The NAL units of a length-prefixed sample (stops at the first malformed length prefix).
pub fn nals<'a>(&self, sample: &'a [u8]) -> Vec<&'a [u8]> {
let ls = self.length_size;
let mut out = Vec::new();
let mut pos = 0usize;
while let Some(pre) = sample.get(pos..pos.saturating_add(ls)) {
if !(1..=4).contains(&ls) {
break;
}
let len = pre.iter().fold(0usize, |a, &b| (a << 8) | b as usize);
pos += ls;
let Some(n) = sample.get(pos..pos.saturating_add(len)) else { break };
pos += len;
out.push(n);
}
out
}
/// The NAL unit type in a NAL header byte.
pub fn nal_type(&self, header: u8) -> u8 {
match self.codec {
NalCodec::H264 => header & 0x1f,
NalCodec::Hevc => (header >> 1) & 0x3f,
}
}
/// Whether `t` is a parameter-set NAL unit type (H.264 SPS / PPS / SPS extension, HEVC VPS /
/// SPS / PPS).
pub fn is_parameter_set(&self, t: u8) -> bool {
match self.codec {
NalCodec::H264 => matches!(t, 7 | 8 | 13 | 15),
NalCodec::Hevc => (32..=34).contains(&t),
}
}
/// Whether decoding can restart at `sample` with no earlier state: an IDR access unit (H.264)
/// or an IRAP one (HEVC IDR / CRA / BLA).
pub fn is_irap(&self, sample: &[u8]) -> bool {
self.nal_types(sample).iter().any(|&t| match self.codec {
NalCodec::H264 => t == 5,
NalCodec::Hevc => (16..=23).contains(&t),
})
}
/// As the software decoder's [`crate::VideoDecoder::is_random_access`] for this stream
/// (length-prefixed samples: unknown, the container's sync flags are trusted).
pub fn is_random_access(&self, _sample: &[u8]) -> Option<bool> {
None
}
/// As the software decoder's [`crate::VideoDecoder::is_disposable`] for this stream.
pub fn is_disposable(&self, sample: &[u8]) -> bool {
match self.codec {
NalCodec::H264 => h264_disposable(sample, self.length_size),
NalCodec::Hevc => hevc_disposable(sample, self.length_size, self.highest_tid),
}
}
/// Output picture size (cropped).
pub fn size(&self) -> (u32, u32) {
(self.crop.2, self.crop.3)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn switch_defaults_to_auto() {
// other tests in this process may flip it; only check the round trip
let was = hardware_decoding();
set_hardware_decoding(false);
assert!(!hardware_decoding());
set_hardware_decoding(true);
assert!(hardware_decoding());
set_hardware_decoding(was);
}
#[test]
fn hostile_records_are_errors() {
for rec in [&[][..], &[1, 2, 3], &[1, 0x64, 0, 0x1f, 0xff, 0xe1, 0, 9, 0x67], &[0; 30]] {
assert!(NalStreamInfo::from_avcc(rec).is_err());
assert!(NalStreamInfo::from_hvcc(rec).is_err());
}
}
#[test]
fn nal_splitting_is_bounded() {
let info = NalStreamInfo {
codec: NalCodec::Hevc,
length_size: 4,
highest_tid: None,
parameter_sets: Vec::new(),
coded: (16, 16),
crop: (0, 0, 16, 16),
chroma_format_idc: 1,
bit_depth_luma: 8,
bit_depth_chroma: 8,
interlaced: false,
profile_idc: 1,
color: filmcraft_color::ColorInfo::REC709,
par: (1, 1),
reorder: 0,
};
// CRA (21) then a length that runs past the end
let s = [0, 0, 0, 2, 21 << 1, 1, 0xff, 0xff, 0xff, 0xff, 1];
assert_eq!(info.nal_types(&s), vec![21]);
assert!(info.is_irap(&s));
assert!(!info.is_irap(&[0, 0, 0, 2, 1 << 1, 1]));
}
}
+25 -12
View File
@@ -20,6 +20,7 @@
pub mod audio;
pub mod gop;
pub mod hw;
pub mod mkv;
pub mod mp4;
pub mod mpeg;
@@ -63,20 +64,21 @@ impl From<CodecError> for filmcraft_media::MediaError {
}
}
/// Our own (pure-Rust) decoders, in the order they are tried.
const BUILTIN_FACTORIES: [VideoDecoderFactory; 8] = [
video::h264_factory,
video::hevc_factory,
video::vp9_factory,
video::av1_factory,
video::prores_factory,
video::dnx_factory,
video::mjpeg_factory,
video::mpeg2_factory,
];
fn factories() -> &'static RwLock<Vec<VideoDecoderFactory>> {
static F: std::sync::OnceLock<RwLock<Vec<VideoDecoderFactory>>> = std::sync::OnceLock::new();
F.get_or_init(|| {
RwLock::new(vec![
video::h264_factory,
video::hevc_factory,
video::vp9_factory,
video::av1_factory,
video::prores_factory,
video::dnx_factory,
video::mjpeg_factory,
video::mpeg2_factory,
])
})
F.get_or_init(|| RwLock::new(BUILTIN_FACTORIES.to_vec()))
}
/// Register a video decoder factory (tried before previously registered ones).
@@ -98,6 +100,17 @@ pub fn make_video_decoder(entry: &filmcraft_isobmff::SampleEntry) -> Result<Box<
Err(CodecError::Unsupported(format!("no decoder for {} video", entry.codec.name())))
}
/// Create one of our own (software) decoders for a sample entry, skipping registered factories:
/// what a hardware decoder falls back to when it fails mid-stream.
pub fn software_video_decoder(entry: &filmcraft_isobmff::SampleEntry) -> Result<Box<dyn VideoDecoder>> {
for f in BUILTIN_FACTORIES {
if let Some(r) = f(entry) {
return r;
}
}
Err(CodecError::Unsupported(format!("no decoder for {} video", entry.codec.name())))
}
/// Openers for the engine's media pool (MP4/MOV, Matroska/WebM, MXF, Ogg Opus/Vorbis, MPEG TS/PS and
/// MPEG-1/2 video elementary streams, standalone audio).
pub fn openers() -> Vec<filmcraft_media::Opener> {
+42 -26
View File
@@ -128,6 +128,42 @@ pub(crate) fn tight_plane<T: Copy>(src: Vec<T>, stride: usize, w: usize, h: usiz
out
}
/// Colour of an H.264 / HEVC picture from its VUI code points (ITU-T H.273), as the decoders
/// report it: BT.709 by default with the matrix guessed from the size, the signalled matrix and
/// transfer when known, and full range when flagged. Shared by the software and hardware
/// decoders so the two are interchangeable.
pub fn vui_color(width: u32, height: u32, matrix: u8, transfer: u8, full_range: bool) -> filmcraft_color::ColorInfo {
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(width, height), ..filmcraft_color::ColorInfo::REC709 };
if let Some(m) = filmcraft_color::Matrix::from_code(matrix) {
color.matrix = m;
}
if let Some(t) = filmcraft_color::Transfer::from_code(transfer) {
color.transfer = t;
}
if full_range {
color.range = filmcraft_color::Range::Full;
}
color
}
/// Pixel aspect ratio from a VUI sample aspect ratio ((0, 0) = unspecified = square).
pub fn sar_par(sar: (u16, u16)) -> (u32, u32) {
if sar.0 > 0 && sar.1 > 0 { (sar.0 as u32, sar.1 as u32) } else { (1, 1) }
}
/// NAL length-prefix size of an `avcC` record (byte 4: lengthSizeMinusOne).
pub fn avcc_length_size(avcc: &[u8]) -> usize {
avcc.get(4).map_or(4, |b| (b & 3) as usize + 1)
}
/// NAL length-prefix size and highest TemporalId of an `hvcC` record.
pub fn hvcc_length_size_and_tid(hvcc: &[u8]) -> (usize, Option<u8>) {
// hvcC byte 21: constantFrameRate(2) numTemporalLayers(3) temporalIdNested(1) lengthSizeMinusOne(2)
let length_size = hvcc.get(21).map_or(4, |b| (b & 3) as usize + 1);
let highest_tid = hvcc.get(21).map(|b| (b >> 3) & 7).filter(|&n| n > 0).map(|n| n - 1);
(length_size, highest_tid)
}
/// A factory returns `None` when it does not handle the entry.
pub type VideoDecoderFactory = fn(&SampleEntry) -> Option<Result<Box<dyn VideoDecoder>>>;
@@ -169,7 +205,7 @@ pub struct H264Decoder {
impl H264Decoder {
pub fn new(avcc: Vec<u8>) -> Result<Self> {
let dec = filmcraft_h264::Decoder::from_avcc(&avcc).map_err(|e| CodecError::Decode(e.to_string()))?;
let length_size = avcc.get(4).map_or(4, |b| (b & 3) as usize + 1);
let length_size = avcc_length_size(&avcc);
Ok(Self { avcc, dec, length_size, draft: false })
}
/// A decoder for Annex B byte-stream samples (start codes, in-band parameter sets: MXF, TS).
@@ -184,17 +220,8 @@ impl H264Decoder {
let y = tight_plane(p.y, p.y_stride, w, h);
let u = tight_plane(p.u, p.uv_stride, cw, ch);
let v = tight_plane(p.v, p.uv_stride, cw, ch);
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(p.width, p.height), ..filmcraft_color::ColorInfo::REC709 };
if let Some(m) = filmcraft_color::Matrix::from_code(p.color.matrix) {
color.matrix = m;
}
if let Some(t) = filmcraft_color::Transfer::from_code(p.color.transfer) {
color.transfer = t;
}
if p.color.full_range {
color.range = filmcraft_color::Range::Full;
}
let par = if p.sar.0 > 0 && p.sar.1 > 0 { (p.sar.0 as u32, p.sar.1 as u32) } else { (1, 1) };
let color = vui_color(p.width, p.height, p.color.matrix, p.color.transfer, p.color.full_range);
let par = sar_par(p.sar);
let frame = VideoFrame {
width: p.width,
height: p.height,
@@ -265,9 +292,7 @@ pub struct HevcDecoder {
impl HevcDecoder {
pub fn new(hvcc: Vec<u8>) -> Result<Self> {
let dec = filmcraft_hevc::Decoder::from_hvcc(&hvcc).map_err(|e| CodecError::Decode(e.to_string()))?;
// hvcC byte 21: constantFrameRate(2) numTemporalLayers(3) temporalIdNested(1) lengthSizeMinusOne(2)
let length_size = hvcc.get(21).map_or(4, |b| (b & 3) as usize + 1);
let highest_tid = hvcc.get(21).map(|b| (b >> 3) & 7).filter(|&n| n > 0).map(|n| n - 1);
let (length_size, highest_tid) = hvcc_length_size_and_tid(&hvcc);
Ok(Self { hvcc, dec, length_size, highest_tid })
}
/// A decoder for Annex B byte-stream samples (start codes, in-band parameter sets: TS).
@@ -299,17 +324,8 @@ impl HevcDecoder {
alpha: None,
},
};
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(p.width, p.height), ..filmcraft_color::ColorInfo::REC709 };
if let Some(m) = filmcraft_color::Matrix::from_code(p.color.matrix) {
color.matrix = m;
}
if let Some(t) = filmcraft_color::Transfer::from_code(p.color.transfer) {
color.transfer = t;
}
if p.color.full_range {
color.range = filmcraft_color::Range::Full;
}
let par = if p.sar.0 > 0 && p.sar.1 > 0 { (p.sar.0 as u32, p.sar.1 as u32) } else { (1, 1) };
let color = vui_color(p.width, p.height, p.color.matrix, p.color.transfer, p.color.full_range);
let par = sar_par(p.sar);
let frame = VideoFrame { width: p.width, height: p.height, data, color, par, pts: filmcraft_time::Tick::ZERO };
DecodedFrame { pts: p.pts, frame, draft: p.draft }
}
+6
View File
@@ -82,6 +82,11 @@ pub struct PlaybackPrefs {
/// skip deblocking (faster; those frames are approximate). Never used for exports, renders or
/// a paused frame. Off by default.
pub draft_decode: bool,
/// "Hardware decoding": `auto` (default: the system's hardware decoder for streams it
/// supports, e.g. VideoToolbox H.264 / HEVC on macOS, falling back to FilmCraft's own decoder)
/// or `off` (always FilmCraft's own decoders). Media already open keeps its decoder until it
/// is reopened.
pub hardware_decoding: String,
}
impl Default for PlaybackPrefs {
@@ -94,6 +99,7 @@ impl Default for PlaybackPrefs {
enable_transmit: false,
disable_video_in_background: true,
draft_decode: false,
hardware_decoding: "auto".into(),
}
}
}
+2
View File
@@ -477,6 +477,7 @@ impl Session {
pub fn start_autosave(&mut self, cfg: autosave::AutosaveConfig) -> std::io::Result<()> {
let prefs_path = cfg.data_dir.join("preferences.json");
self.prefs = autosave::Preferences::load(&prefs_path);
self.prefs.apply_hardware_decoding();
self.media.set_use_proxies(self.prefs.media.enable_proxies);
self.shortcuts.set_dir(&cfg.data_dir);
self.presets.set_dir(&cfg.data_dir);
@@ -548,6 +549,7 @@ impl Session {
pub fn set_prefs(&mut self, p: autosave::Preferences) -> std::io::Result<()> {
let cache_changed = self.prefs.media_cache != p.media_cache;
self.prefs = p;
self.prefs.apply_hardware_decoding();
if cache_changed {
self.apply_media_cache();
}
+28 -1
View File
@@ -9,6 +9,7 @@ use crate::Session;
/// measure an interval).
pub fn decode_json() -> Value {
let g = filmcraft_codecs::gop_stats();
let hw = filmcraft_codecs::hw::hw_stats();
json!({
"requests": g.hits + g.misses,
"cacheHits": g.hits,
@@ -22,6 +23,17 @@ pub fn decode_json() -> Value {
"evicted": g.evicted,
"decodeMs": g.decode_ns as f64 / 1e6,
"decodeMsPerSample": g.decode_ms_per_sample(),
"framesDecoded": g.frames,
// Settings ▸ Playback ▸ Hardware decoding: pictures from OS hardware decoders vs ours,
// hardware decoders created, streams handed to software up front, mid-stream fallbacks.
"hardware": {
"enabled": filmcraft_codecs::hw::hardware_decoding(),
"frames": hw.frames,
"softwareFrames": g.frames.saturating_sub(hw.frames),
"sessions": hw.sessions,
"declined": hw.declined,
"fallbacks": hw.fallbacks,
},
})
}
@@ -48,9 +60,24 @@ mod tests {
s.execute("file.openDemoProject", json!({})).unwrap();
let undo = s.history.undo.len();
let v = s.execute("perf.stats", json!({})).unwrap();
for k in ["requests", "cacheHitRate", "seeks", "samplesDecoded", "samplesSkipped", "draftFrames", "h264Threads", "decodeMs", "decodeMsPerSample"] {
for k in [
"requests",
"cacheHitRate",
"seeks",
"samplesDecoded",
"samplesSkipped",
"draftFrames",
"h264Threads",
"decodeMs",
"decodeMsPerSample",
"framesDecoded",
] {
assert!(v["decode"][k].is_number(), "decode.{k} in {v}");
}
for k in ["frames", "softwareFrames", "sessions", "declined", "fallbacks"] {
assert!(v["decode"]["hardware"][k].is_number(), "decode.hardware.{k} in {v}");
}
assert!(v["decode"]["hardware"]["enabled"].is_boolean());
assert!(v["media"]["openSources"].is_number());
assert_eq!(v["jobs"]["running"], json!(0));
assert_eq!(s.history.undo.len(), undo, "a query adds no undo step");
+11
View File
@@ -653,6 +653,7 @@ const CACHE_MGMT: &[(&str, &str)] = &[
];
const AUTO_SCROLL: &[(&str, &str)] = &[("noScroll", "No Scroll"), ("pageScroll", "Page Scroll"), ("smoothScroll", "Smooth Scroll")];
const MOUSE_SCROLL: &[(&str, &str)] = &[("vertical", "Vertical"), ("horizontal", "Horizontal")];
const HW_DECODE: &[(&str, &str)] = &[("auto", "Auto"), ("off", "Off")];
const TRACKS: &[(&str, &str)] = &[("useFile", "Use File"), ("mono", "Mono"), ("stereo", "Stereo"), ("5.1", "5.1"), ("adaptive", "Adaptive")];
pub const DURATION_UNITS: &[(&str, &str)] = &[("frames", "Frames"), ("seconds", "Seconds")];
@@ -952,6 +953,10 @@ static CATEGORIES: &[Category] = &[
b("playback.enableTransmit", "Enable Mercury Transmit", false),
b("playback.disableVideoInBackground", "Disable video output when in the background", false),
b("playback.draftDecode", "Draft decoding at reduced playback resolution (H.264: faster, some frames less filtered)", true),
f("playback.hardwareDecoding", "Hardware decoding", Kind::Choice(HW_DECODE), true),
Row::Note(
"Hardware decoding: Auto uses the system's video decoder (VideoToolbox on macOS) for the H.264 and HEVC streams it supports, and FilmCraft's own decoder for everything else or if the hardware fails. Media that is already open keeps its decoder until it is reopened.",
),
],
},
Category {
@@ -1122,6 +1127,12 @@ pub fn migrate(v: &mut Value) {
}
impl crate::autosave::Preferences {
/// Settings ▸ Playback ▸ Hardware decoding as the process-wide switch the hardware decoder
/// factories consult (new decoders only).
pub fn apply_hardware_decoding(&self) {
filmcraft_codecs::hw::set_hardware_decoding(self.playback.hardware_decoding != "off");
}
/// Values of one category as a JSON object.
pub fn category_value(&self, id: &str) -> Option<Value> {
self.to_value().get(id).cloned()
+12
View File
@@ -480,3 +480,15 @@ fn auto_transcribe_on_import_and_transcription_defaults() {
set(&mut s, "mediaAnalysis.whisperModel", json!("whisper-tiny"));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn hardware_decoding_setting_drives_the_decoder_switch() {
let mut s = Session::default();
assert_eq!(s.prefs.playback.hardware_decoding, "auto", "Auto by default");
set(&mut s, "playback.hardwareDecoding", json!("off"));
assert!(!filmcraft_codecs::hw::hardware_decoding(), "Off reaches the decoder registry");
assert!(s.execute("prefs.set", json!({"key": "playback.hardwareDecoding", "value": "gpu"})).is_err());
set(&mut s, "playback.hardwareDecoding", json!("auto"));
assert!(filmcraft_codecs::hw::hardware_decoding());
assert!(settings::field("playback.hardwareDecoding").is_some_and(|f| f.wired && matches!(f.kind, Kind::Choice(_))));
}
+198
View File
@@ -775,3 +775,201 @@ fn gpu_matches_cpu_on_golden_scenes() {
}
assert!(failures.is_empty(), "GPU differs from the CPU reference:\n{}", failures.join("\n"));
}
/// Per-pixel max RGB difference (99th percentile, mean) over the pixels at least 1.5 px from every
/// plan layer's quad outline: the CPU resampler fades a layer's edge over one more pixel than the
/// GPU rasterises, and with blend modes those edge pixels are a visible share of a 320×180 frame.
fn interior_diff(plan: &filmcraft_render::plan::FramePlan, a: &Rgba8, b: &Rgba8) -> (u32, f64) {
let filmcraft_render::plan::FramePlan::Layers { width, layers, .. } = plan else { return (u32::MAX, f64::MAX) };
let inv: Vec<_> = layers.iter().filter_map(|l| Some((l.matrix.inverse()?, l.size().0 as f64, l.size().1 as f64))).collect();
let mut d: Vec<u32> = Vec::new();
for (i, (pa, pb)) in a.px.chunks(4).zip(b.px.chunks(4)).enumerate() {
let (x, y) = ((i % width) as f64 + 0.5, (i / width) as f64 + 0.5);
let edge = inv.iter().any(|(m, fw, fh)| {
let c = [(-1.5, -1.5), (1.5, -1.5), (-1.5, 1.5), (1.5, 1.5)].map(|(dx, dy)| {
let p = m.apply(Vec2::new(x + dx, y + dy));
p.x >= 0.0 && p.y >= 0.0 && p.x <= *fw && p.y <= *fh
});
c.iter().any(|v| *v != c[0])
});
if !edge {
d.push((0..3).map(|k| (pa[k] as i32 - pb[k] as i32).unsigned_abs()).max().unwrap_or(0));
}
}
d.sort_unstable();
(d[d.len() * 99 / 100], d.iter().sum::<u32>() as f64 / d.len() as f64)
}
/// Every blend mode composited by the GPU from `plan_frame` (no CPU fallback): two overlapping
/// transformed clips at partial opacity in the mode over footage, plus a clip with a standard
/// effect (rendered on the CPU as a layer image) in the same mode, match the CPU render with the
/// GPU parity tolerance (p99 ≤ 6, mean < 1.5) away from layer outlines; whole-frame numbers are
/// printed too.
#[test]
fn gpu_blend_modes_match_cpu_render() {
let Some((dev, q)) = device() else {
eprintln!("SKIPPED (gpu blend parity): no GPU adapter");
return;
};
let mut c = filmcraft_gpu::GpuCompositor::new(&dev, &q);
let mut failures = Vec::new();
for mode in filmcraft_project::effect::BLEND_MODES {
let mut b = Builder::new(4);
let bg = b.demo(DemoScene::OceanSunset);
let (dunes, aurora, bars) = (b.demo(DemoScene::Dunes), b.demo(DemoScene::Aurora), b.media(Generator::BarsAndTone));
b.place(0, bg, 0, 48);
let v2 = b.place(1, dunes, 0, 48);
b.fixed(v2, "motion", &[("scale", fl(60.0)), ("rotation", fl(12.0)), ("position", pt(130.0, 80.0))]);
b.fixed(v2, "opacity", &[("opacity", fl(70.0)), ("blend", blend(mode))]);
let v3 = b.place(2, aurora, 0, 48);
b.fixed(v3, "motion", &[("scale", fl(45.0)), ("rotation", fl(-20.0)), ("position", pt(210.0, 110.0))]);
b.fixed(v3, "opacity", &[("opacity", fl(60.0)), ("blend", blend(mode))]);
let v4 = b.place(3, bars, 0, 48);
b.fixed(v4, "motion", &[("scale", fl(30.0)), ("position", pt(250.0, 50.0))]);
b.fixed(v4, "opacity", &[("opacity", fl(85.0)), ("blend", blend(mode))]);
b.effect(v4, "crop", &[("left", fl(10.0)), ("bottom", fl(20.0))]);
let s = b.at(12);
let t = RATE.tick_of(s.frame);
let cpu = render_cpu(&s);
let plan = filmcraft_render::plan::plan_frame(&s.project, s.seq, t, RenderOptions::default(), &s.sources);
let filmcraft_render::plan::FramePlan::Layers { layers, .. } = &plan else {
failures.push(format!("{mode}: planned as a CPU image"));
continue;
};
assert_eq!(layers.len(), 4, "{mode}");
c.composite(&plan);
let (gw, gh, mut px) = c.read_output().expect("GPU readback");
px.as_chunks_mut::<4>().0.iter_mut().for_each(|p| p[3] = 255);
let gpu = Rgba8::new(gw, gh, px);
let d = diff(&cpu, &gpu).unwrap();
let (p99, mean) = interior_diff(&plan, &cpu, &gpu);
eprintln!("{mode}: GPU vs CPU interior p99 {p99}, mean {mean:.3}; whole frame {d}");
if p99 > 6 || mean >= 1.5 {
failures.push(format!("{mode}: interior p99 {p99}, mean {mean}; {d}"));
}
}
assert!(failures.is_empty(), "GPU blend modes differ from the CPU render:\n{}", failures.join("\n"));
}
/// Inside a transition the CPU mixes the clips and composites the result Normal, ignoring their
/// blend modes: the plan does the same (Screen clips cross-dissolving on V2 over V1).
#[test]
fn gpu_transition_of_blended_clips_matches_cpu_render() {
let Some((dev, q)) = device() else {
eprintln!("SKIPPED (gpu blend parity): no GPU adapter");
return;
};
let mut b = Builder::new(2);
let (bg, x, y) = (b.demo(DemoScene::OceanSunset), b.demo(DemoScene::Aurora), b.demo(DemoScene::Dunes));
b.place(0, bg, 0, 48);
let ca = b.place(1, x, 0, 24);
let cb = b.place(1, y, 24, 24);
for id in [ca, cb] {
b.fixed(id, "opacity", &[("blend", blend("Screen"))]);
}
b.transition(1, "cross_dissolve", ca, cb, 24, 12);
let s = b.at(22);
let plan = filmcraft_render::plan::plan_frame(&s.project, s.seq, RATE.tick_of(s.frame), RenderOptions::default(), &s.sources);
let filmcraft_render::plan::FramePlan::Layers { layers, .. } = &plan else { panic!("planned as a CPU image") };
assert!(layers.iter().all(|l| l.blend == filmcraft_render::Blend::Normal));
let mut c = filmcraft_gpu::GpuCompositor::new(&dev, &q);
c.composite(&plan);
let (gw, gh, mut px) = c.read_output().expect("GPU readback");
px.as_chunks_mut::<4>().0.iter_mut().for_each(|p| p[3] = 255);
let d = diff(&render_cpu(&s), &Rgba8::new(gw, gh, px)).unwrap();
eprintln!("Screen clips in a cross dissolve: GPU vs CPU {d}");
assert!(d.p99 <= 6 && d.mean_abs * 4.0 / 3.0 < 1.5, "{d}");
}
/// Standard effect chains run on the GPU: every clip below plans as a GPU layer with its effects
/// (no CPU-rendered image), including keyframed parameters, transformed clips, partial opacity
/// and blend modes, and matches the CPU render with the GPU parity tolerance (p99 ≤ 6, mean < 1.5
/// away from layer outlines) at three times. A clip with an effect the GPU stage lacks (Emboss)
/// still falls back to a CPU layer image.
#[test]
fn gpu_effect_chains_match_cpu_render() {
let mut b = Builder::new(5);
let (bg, dunes, aurora, bars, sunset) =
(b.demo(DemoScene::OceanSunset), b.demo(DemoScene::Dunes), b.demo(DemoScene::Aurora), b.media(Generator::BarsAndTone), b.demo(DemoScene::OceanSunset));
let v1 = b.place(0, bg, 0, 48);
b.effect(v1, "brightness_contrast", &[("brightness", fl(-10.0)), ("contrast", fl(20.0))]);
b.effect(v1, "gaussian_blur", &[("repeat_edge", ParamValue::Bool(true))]);
// keyframed blur: evaluated at the frame's time
let p = b.clip(v1).effects.last_mut().and_then(|e| e.params.get_mut("blurriness")).expect("blurriness");
p.put_keyframe(Tick::ZERO, fl(2.0));
p.put_keyframe(RATE.tick_of(24), fl(20.0));
let v2 = b.place(1, dunes, 0, 48);
b.fixed(v2, "motion", &[("scale", fl(60.0)), ("rotation", fl(12.0)), ("position", pt(130.0, 80.0))]);
b.fixed(v2, "opacity", &[("opacity", fl(70.0)), ("blend", blend("Screen"))]);
b.effect(v2, "tint", &[("amount", fl(60.0))]);
b.effect(v2, "unsharp_mask", &[("amount", fl(120.0)), ("radius", fl(2.0))]);
b.effect(v2, "crop", &[("left", fl(8.0)), ("bottom", fl(12.0)), ("feather", fl(6.0))]);
let v3 = b.place(2, aurora, 0, 48);
b.fixed(v3, "motion", &[("scale", fl(45.0)), ("rotation", fl(-20.0)), ("position", pt(210.0, 110.0))]);
b.fixed(v3, "opacity", &[("opacity", fl(85.0))]);
b.effect(v3, "levels", &[("in_black", fl(20.0)), ("gamma", fl(130.0))]);
b.effect(v3, "mirror", &[("angle", fl(60.0))]);
b.effect(v3, "directional_blur", &[("length", fl(4.0)), ("direction", fl(45.0))]);
b.effect(v3, "transform", &[("rotation", fl(10.0)), ("scale_height", fl(110.0))]);
let v4 = b.place(3, bars, 0, 48);
b.fixed(v4, "motion", &[("scale", fl(30.0)), ("position", pt(250.0, 50.0))]);
b.fixed(v4, "opacity", &[("blend", blend("Multiply"))]);
b.effect(v4, "posterize", &[("levels", fl(5.0))]);
b.effect(v4, "invert", &[("blend", fl(40.0))]);
b.effect(v4, "offset", &[("shift", pt(400.0, 120.0))]);
b.effect(v4, "horizontal_flip", &[]);
b.effect(v4, "black_white", &[]);
let v5 = b.place(4, sunset, 0, 48);
b.fixed(v5, "motion", &[("scale", fl(25.0)), ("position", pt(60.0, 140.0))]);
b.effect(v5, "emboss", &[]);
let mut s = b.at(12);
let plan = filmcraft_render::plan::plan_frame(&s.project, s.seq, RATE.tick_of(s.frame), RenderOptions::default(), &s.sources);
let filmcraft_render::plan::FramePlan::Layers { layers, .. } = &plan else { panic!("planned as a CPU image") };
assert_eq!(layers.len(), 5);
for (i, l) in layers.iter().take(4).enumerate() {
let fx = l.fx.as_ref().unwrap_or_else(|| panic!("V{}: not a GPU effect layer", i + 1));
assert!(!fx.ops.is_empty());
}
assert!(layers[4].fx.is_none(), "Emboss is rendered on the CPU");
let Some((dev, q)) = device() else {
eprintln!("SKIPPED (gpu effect parity): no GPU adapter");
return;
};
let mut c = filmcraft_gpu::GpuCompositor::new(&dev, &q);
let mut failures = Vec::new();
for frame in [0, 12, 30] {
s.frame = frame;
let plan = filmcraft_render::plan::plan_frame(&s.project, s.seq, RATE.tick_of(frame), RenderOptions::default(), &s.sources);
c.composite(&plan);
let (gw, gh, mut px) = c.read_output().expect("GPU readback");
px.as_chunks_mut::<4>().0.iter_mut().for_each(|p| p[3] = 255);
let gpu = Rgba8::new(gw, gh, px);
let cpu = render_cpu(&s);
let d = diff(&cpu, &gpu).unwrap();
let (p99, mean) = interior_diff(&plan, &cpu, &gpu);
eprintln!("effect chains @ {frame}: GPU vs CPU interior p99 {p99}, mean {mean:.3}; whole frame {d}");
if p99 > 6 || mean >= 1.5 {
let dir = filmcraft_testkit::golden::failures_dir();
let _ = filmcraft_testkit::golden::write_png(&dir.join(format!("fx_chains_{frame}.gpu.png")), &gpu);
let _ = filmcraft_testkit::golden::write_png(&dir.join(format!("fx_chains_{frame}.cpu.png")), &cpu);
failures.push(format!("frame {frame}: interior p99 {p99}, mean {mean}; {d}; images in {}", dir.display()));
}
}
// ½ playback resolution: the working images (and the effects' pixel radii) shrink with it
let opts = RenderOptions { scale: 0.5, ..RenderOptions::default() };
let plan = filmcraft_render::plan::plan_frame(&s.project, s.seq, RATE.tick_of(s.frame), opts, &s.sources);
let filmcraft_render::plan::FramePlan::Layers { layers, .. } = &plan else { panic!("½: planned as a CPU image") };
assert!(layers.iter().take(4).all(|l| l.fx.is_some()));
c.composite(&plan);
let (gw, gh, mut px) = c.read_output().expect("GPU readback");
px.as_chunks_mut::<4>().0.iter_mut().for_each(|p| p[3] = 255);
let gpu = Rgba8::new(gw, gh, px);
let img = render_sequence(&s.project, s.seq, RATE.tick_of(s.frame), opts, &s.sources);
let cpu = Rgba8::new(img.w as u32, img.h as u32, img.over_black_rgba8());
let (p99, mean) = interior_diff(&plan, &cpu, &gpu);
eprintln!("effect chains @ ½: GPU vs CPU interior p99 {p99}, mean {mean:.3}; whole frame {}", diff(&cpu, &gpu).unwrap());
if p99 > 6 || mean >= 1.5 {
failures.push(format!("½ resolution: interior p99 {p99}, mean {mean}"));
}
assert!(failures.is_empty(), "GPU effect chains differ from the CPU render:\n{}", failures.join("\n"));
}
+338
View File
@@ -0,0 +1,338 @@
//! GPU-vs-CPU parity of the blend modes (`filmcraft_render::blend::composite` is the reference).
//!
//! Tolerances: the composited 8-bit output uses the compositor's parity criterion (99th percentile
//! of the per-pixel max channel difference ≤ 6, mean < 1.5; antialiased edges and half-float
//! textures differ slightly). Exact inputs drawn 1:1 (no resampling) must agree to 2e-3 in linear
//! premultiplied values (half-float accumulator), except Hard Mix, a step function, which may
//! flip a pixel whose blend lands on its 0.5 threshold.
use super::*;
use filmcraft_geom::{Affine, Vec2};
use filmcraft_render::plan::execute_cpu;
pub(crate) fn device() -> Option<(wgpu::Device, wgpu::Queue)> {
let instance = wgpu::Instance::default();
let adapter = pollster::block_on(instance.request_adapter(&wgpu::RequestAdapterOptions::default())).ok()?;
pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor::default())).ok()
}
pub(crate) fn yuv_frame(w: u32, h: u32) -> Arc<VideoFrame> {
let (cw, ch) = (w.div_ceil(2), h.div_ceil(2));
let y: Vec<u8> = (0..w * h).map(|i| (16 + ((i % w) * 219 / w)) as u8).collect();
let u: Vec<u8> = (0..cw * ch).map(|i| (64 + (i / cw) * 128 / ch) as u8).collect();
let v: Vec<u8> = (0..cw * ch).map(|i| (200 - (i % cw) * 100 / cw) as u8).collect();
Arc::new(VideoFrame {
width: w,
height: h,
data: PixelData::Yuv8 { planes: [Arc::new(y), Arc::new(u), Arc::new(v)], chroma: Chroma::C420, alpha: None },
color: filmcraft_color::ColorInfo::REC709,
par: (1, 1),
pts: Default::default(),
})
}
/// Decode an IEEE half.
pub(crate) fn f16_to_f32(h: u16) -> f32 {
let s = if h & 0x8000 != 0 { -1.0 } else { 1.0 };
let e = ((h >> 10) & 0x1f) as i32;
let m = (h & 0x3ff) as f32;
match e {
0 => s * m * 2f32.powi(-24),
31 => s * f32::INFINITY,
_ => s * (1.0 + m / 1024.0) * 2f32.powi(e - 15),
}
}
/// Read the linear premultiplied accumulator back as f32 RGBA (what the CPU plan executor returns).
pub(crate) fn read_accum(c: &GpuCompositor) -> Vec<f32> {
let (tex, _, (w, h)) = c.accum.as_ref().expect("accumulator");
let row = (w * 8).div_ceil(256) * 256;
let buf = c.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("accum-readback"),
size: (row * h) as u64,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
let mut enc = c.device.create_command_encoder(&Default::default());
enc.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo { texture: tex, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
wgpu::TexelCopyBufferInfo { buffer: &buf, layout: wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(row), rows_per_image: Some(*h) } },
wgpu::Extent3d { width: *w, height: *h, depth_or_array_layers: 1 },
);
c.queue.submit([enc.finish()]);
let slice = buf.slice(..);
slice.map_async(wgpu::MapMode::Read, |_| {});
let _ = c.device.poll(wgpu::PollType::wait_indefinitely());
let data = slice.get_mapped_range().expect("mapped");
let mut out = Vec::with_capacity((w * h * 4) as usize);
for y in 0..*h {
let r = &data[(y * row) as usize..(y * row + w * 8) as usize];
out.extend(r.as_chunks::<2>().0.iter().map(|b| f16_to_f32(u16::from_le_bytes(*b))));
}
out
}
/// Per-pixel max channel difference of two RGBA8 images over the pixels `keep` selects: (p99,
/// mean, max).
pub(crate) fn stats8(a: &[u8], b: &[u8], keep: &[bool]) -> (u32, f64, u32) {
let mut d: Vec<u32> = a
.chunks(4)
.zip(b.chunks(4))
.zip(keep)
.filter(|(_, k)| **k)
.map(|((a, b), _)| (0..3).map(|k| (a[k] as i32 - b[k] as i32).unsigned_abs()).max().unwrap_or(0))
.collect();
d.sort_unstable();
(d[d.len() * 99 / 100], d.iter().sum::<u32>() as f64 / d.len() as f64, d[d.len() - 1])
}
/// Pixels at least 1.5 px away from every layer's quad outline (the CPU resampler fades a layer's
/// edge over a pixel beyond it, the GPU rasterises the quad: compared separately).
pub(crate) fn interior(plan: &FramePlan) -> Vec<bool> {
let FramePlan::Layers { width, height, layers } = plan else { return Vec::new() };
let inv: Vec<(Affine, f64, f64)> = layers.iter().filter_map(|l| Some((l.matrix.inverse()?, l.size().0 as f64, l.size().1 as f64))).collect();
let mut keep = vec![true; width * height];
for (i, k) in keep.iter_mut().enumerate() {
let (x, y) = ((i % width) as f64 + 0.5, (i / width) as f64 + 0.5);
*k = inv.iter().all(|(m, fw, fh)| {
let inside = |dx: f64, dy: f64| {
let p = m.apply(Vec2::new(x + dx, y + dy));
p.x >= 0.0 && p.y >= 0.0 && p.x <= *fw && p.y <= *fh
};
let c = [inside(-1.5, -1.5), inside(1.5, -1.5), inside(-1.5, 1.5), inside(1.5, 1.5)];
c.iter().all(|v| *v == c[0])
});
}
keep
}
/// Linear premultiplied RGBA f32 test frame: a colour ramp with black and white bands; alpha 0 in
/// the left tenth, 1 in the top third, partial below.
pub(crate) fn ramp_layer(w: u32, h: u32, phase: u32) -> Arc<VideoFrame> {
let mut px = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
let fx = x as f32 / (w - 1) as f32;
let fy = y as f32 / (h - 1) as f32;
let c = match (x + phase) / 8 % 6 {
0 => [0.0, 0.0, 0.0],
1 => [1.0, 1.0, 1.0],
_ => [fx, (fy * 1.3 + phase as f32 * 0.1).fract(), ((fx + fy) * 0.7).fract()],
};
let a = if x < w / 10 {
0.0
} else if y < h / 3 {
1.0
} else {
0.25 + 0.75 * fy
};
px.extend_from_slice(&[c[0] * a, c[1] * a, c[2] * a, a]);
}
}
Arc::new(VideoFrame::rgba_f32(w, h, px))
}
/// Every blend mode over a backdrop with opaque, partially transparent and empty regions, on two
/// overlapping transformed layers (rotated, scaled, partly off the output) with partial opacity
/// and partial alpha: the GPU matches the CPU plan executor within the compositor's tolerance,
/// away from the layers' outlines (which are reported: the CPU fades them over an extra pixel).
#[test]
fn gpu_blend_modes_match_cpu() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let (w, h) = (320usize, 180usize);
let pixels = filmcraft_media::generators::render(
&filmcraft_media::Generator::Demo(filmcraft_media::DemoScene::Dunes),
320,
180,
1.0,
24,
filmcraft_time::FrameRate::FPS_24,
);
let demo = Arc::new(VideoFrame::rgba8(320, 180, pixels));
let base = ramp_layer(200, 180, 3);
let top = ramp_layer(160, 120, 0);
for mode in Blend::ALL {
let plan = FramePlan::Layers {
width: w,
height: h,
layers: vec![
// opaque YUV on the left, a partially transparent ramp on the right; the top-right
// corner stays empty (destination alpha 0)
PlanLayer { frame: yuv_frame(640, 360), matrix: Affine::scale(0.35, 0.5), opacity: 1.0, blend: Blend::Normal, fx: None },
PlanLayer { frame: base.clone(), matrix: Affine::translate(200.0, 40.0), opacity: 0.9, blend: Blend::Normal, fx: None },
PlanLayer {
frame: top.clone(),
matrix: Affine::motion(Vec2::new(170.0, 95.0), Vec2::new(1.1, 1.1), 17.0, Vec2::new(80.0, 60.0)),
opacity: 0.8,
blend: mode,
fx: None,
},
PlanLayer {
frame: demo.clone(),
matrix: Affine::motion(Vec2::new(250.0, 70.0), Vec2::new(0.45, 0.45), -9.0, Vec2::new(160.0, 90.0)),
opacity: 0.6,
blend: mode,
fx: None,
},
],
};
let cpu_img = execute_cpu(&plan);
let cpu = cpu_img.over_black_rgba8();
c.composite(&plan);
let (_, _, gpu) = c.read_output().expect("readback");
let keep = interior(&plan);
let (p99, mean, max) = stats8(&cpu, &gpu, &keep);
let all = vec![true; keep.len()];
let (ap99, amean, _) = stats8(&cpu, &gpu, &all);
// linear premultiplied accumulator, alpha included
let acc = read_accum(&c);
let mut fd: Vec<f32> =
cpu_img.px.chunks(4).zip(acc.chunks(4)).zip(&keep).filter(|(_, k)| **k).flat_map(|((a, b), _)| (0..4).map(|k| (a[k] - b[k]).abs())).collect();
fd.sort_unstable_by(f32::total_cmp);
let (fp99, fmax) = (fd[fd.len() * 99 / 100], fd[fd.len() - 1]);
eprintln!("{mode:?}: interior 8-bit p99 {p99} mean {mean:.3} max {max}; linear p99 {fp99:.2e} max {fmax:.2e}; with edges p99 {ap99} mean {amean:.3}");
assert!(p99 <= 6 && mean < 1.5, "{mode:?}: p99 {p99}, mean {mean}");
}
assert!(c.backdrop.is_some(), "blend modes use the backdrop copy");
}
/// Formula edge cases on exact (half-float representable) values, drawn 1:1 so no resampling is
/// involved: black / white / mid inputs, Divide by zero, Color Dodge and Burn at 0 and 1, zero and
/// partial alpha on either side.
#[test]
fn gpu_blend_edge_cases_match_cpu() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let vals = [0.0f32, 1.0, 0.5, 0.25, 0.75, 0.125, 0.875, 0.0625];
let alphas = [1.0f32, 0.5, 0.0, 0.25];
let n = vals.len() * alphas.len();
// backdrop varies along x, source along y: every (b, s, alpha) combination; channels are
// shifted along the other axis so greys, primaries and mixed colours all occur
let make = |along_x: bool| {
let mut px = Vec::with_capacity(n * n * 4);
for y in 0..n {
for x in 0..n {
let (i, o) = if along_x { (x, y) } else { (y, x) };
let (v, a) = (i % vals.len(), alphas[i / vals.len()]);
let j = o % 3;
let c = [vals[v], vals[(v + j) % vals.len()], vals[(v + 2 * j) % vals.len()]];
px.extend_from_slice(&[c[0] * a, c[1] * a, c[2] * a, a]);
}
}
Arc::new(VideoFrame::rgba_f32(n as u32, n as u32, px))
};
let (back, src) = (make(true), make(false));
let mut c = GpuCompositor::new(&dev, &q);
for mode in Blend::ALL {
for opacity in [1.0f32, 0.5] {
let plan = FramePlan::Layers {
width: n,
height: n,
layers: vec![
PlanLayer { frame: back.clone(), matrix: Affine::IDENTITY, opacity: 1.0, blend: Blend::Normal, fx: None },
PlanLayer { frame: src.clone(), matrix: Affine::IDENTITY, opacity, blend: mode, fx: None },
],
};
let cpu = execute_cpu(&plan);
c.composite(&plan);
let gpu = read_accum(&c);
let mut worst = (0f32, 0usize);
let mut steps = 0;
for (i, (a, b)) in cpu.px.iter().zip(&gpu).enumerate() {
assert!(b.is_finite(), "{mode:?}: non-finite GPU value at {i}");
let d = (a - b).abs();
if d > 0.05 {
steps += 1;
}
if d > worst.0 {
worst = (d, i);
}
}
eprintln!("{mode:?} @ {opacity}: max |cpu − gpu| = {:.2e} (value {} of {}), {steps} step flips", worst.0, worst.1, gpu.len());
assert!(worst.0 < 2e-3 || mode == Blend::HardMix && steps <= 2, "{mode:?} @ {opacity}: {worst:?}, {steps}");
}
}
}
/// Dissolve on the GPU reproduces the CPU's pixel pattern exactly (opaque layer at partial opacity).
#[test]
fn gpu_dissolve_pattern_is_exact() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let (w, h) = (257usize, 131usize);
let white = Arc::new(VideoFrame::rgba_f32(1, 1, vec![1.0; 4]));
let red = Arc::new(VideoFrame::rgba_f32(1, 1, vec![1.0, 0.0, 0.0, 1.0]));
let mut c = GpuCompositor::new(&dev, &q);
for op in [0.1f32, 0.5, 0.9] {
let plan = FramePlan::Layers {
width: w,
height: h,
layers: vec![
PlanLayer { frame: red.clone(), matrix: Affine::scale(w as f64, h as f64), opacity: 1.0, blend: Blend::Normal, fx: None },
PlanLayer { frame: white.clone(), matrix: Affine::scale(w as f64, h as f64), opacity: op, blend: Blend::Dissolve, fx: None },
],
};
let cpu = execute_cpu(&plan).over_black_rgba8();
c.composite(&plan);
let (_, _, gpu) = c.read_output().expect("readback");
let differ = cpu.chunks(4).zip(gpu.chunks(4)).filter(|(a, b)| a[..3] != b[..3]).count();
let kept = gpu.chunks(4).filter(|p| p[1] > 128).count() as f32 / (w * h) as f32;
eprintln!("Dissolve @ {op}: {differ} differing pixels, {kept:.3} kept");
assert_eq!(differ, 0, "opacity {op}");
assert!((kept - op).abs() < 0.02);
}
// fixed-function only: no backdrop copy
assert!(c.backdrop.is_none());
}
/// Normal-only plans keep the single fixed-function pass (no backdrop texture), and blend-mode
/// layers entirely off the output draw nothing.
#[test]
fn normal_fast_path_and_blend_layers_off_output() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let grey = Arc::new(VideoFrame::rgba_f32(4, 4, vec![0.2; 64]));
let layer = |m: Affine, blend: Blend| PlanLayer { frame: grey.clone(), matrix: m, opacity: 1.0, blend, fx: None };
let normal = FramePlan::Layers { width: 32, height: 16, layers: vec![layer(Affine::scale(8.0, 4.0), Blend::Normal)] };
c.composite(&normal);
assert!(c.backdrop.is_none());
let (_, _, before) = c.read_output().expect("readback");
let off = FramePlan::Layers {
width: 32,
height: 16,
layers: vec![
layer(Affine::scale(8.0, 4.0), Blend::Normal),
layer(Affine::translate(100.0, 0.0), Blend::Multiply),
layer(Affine::translate(-10.0, -10.0), Blend::Screen),
],
};
c.composite(&off);
let (_, _, after) = c.read_output().expect("readback");
assert_eq!(before, after);
// only blend-mode layers first: the accumulator is still cleared
let only = FramePlan::Layers { width: 32, height: 16, layers: vec![layer(Affine::translate(100.0, 0.0), Blend::Multiply)] };
c.composite(&only);
let (_, _, empty) = c.read_output().expect("readback");
assert!(empty.chunks(4).all(|p| p[..3] == [0, 0, 0]));
}
#[test]
fn quad_bounds_clamps_and_handles_non_finite() {
let l = |m: Affine| PlanLayer { frame: Arc::new(VideoFrame::rgba_f32(10, 10, vec![0.0; 400])), matrix: m, opacity: 1.0, blend: Blend::Screen, fx: None };
assert_eq!(quad_bounds(&l(Affine::translate(5.0, 5.0)), 100, 100), Some((4, 4, 12, 12)));
assert_eq!(quad_bounds(&l(Affine::translate(-5.0, 95.0)), 100, 100), Some((0, 94, 6, 6)));
assert_eq!(quad_bounds(&l(Affine::translate(f64::NAN, 0.0)), 100, 100), Some((0, 0, 100, 100)));
assert_eq!(quad_bounds(&l(Affine::scale(f64::INFINITY, 1.0)), 100, 100), Some((0, 0, 100, 100)));
assert_eq!(quad_bounds(&l(Affine::translate(-50.0, 0.0)), 100, 100), None);
}
+176 -4
View File
@@ -2,7 +2,10 @@
// Layers are drawn as transformed quads; the fragment shader samples the source (YUV planes or
// RGBA) with manual bilinear filtering (textureLoad, so any float format works) and N×N
// supersampling over the pixel footprint when minifying, converts to linear light, premultiplies
// and scales by opacity. Blending into the Rgba16Float accumulator is premultiplied "over".
// and scales by opacity. Normal layers blend into the Rgba16Float accumulator with fixed-function
// premultiplied "over"; Dissolve does too (each pixel is either dropped or drawn opaque). The other
// blend modes (`fs_blend`) read the accumulator under the layer from a copy (`backdrop`) and write
// the composited result, with the math of `filmcraft_render::blend::composite`.
struct U {
m0: vec4<f32>, // a b c d
@@ -11,12 +14,15 @@ struct U {
p0: vec4<f32>, // opacity, kind (0 rgba8 srgb straight, 1 rgba16f premul linear, 2 yuv), taps, transfer (0 srgb, 1 linear, 2 pq, 3 hlg)
p1: vec4<f32>, // y_off y_scale c_off c_scale (code units)
p2: vec4<f32>, // kr kb code_scale footprint
p3: vec4<f32>, // blend mode (index into filmcraft_render::Blend::ALL), unused ×3
};
@group(0) @binding(0) var<uniform> u: U;
@group(0) @binding(1) var tex0: texture_2d<f32>;
@group(0) @binding(2) var tex1: texture_2d<f32>;
@group(0) @binding(3) var tex2: texture_2d<f32>;
// The accumulator under the layer (blend modes other than Normal / Dissolve only).
@group(0) @binding(4) var backdrop: texture_2d<f32>;
struct VOut {
@builtin(position) pos: vec4<f32>,
@@ -100,20 +106,186 @@ fn sample(p: vec2<f32>) -> vec4<f32> {
return vec4(to_linear(vec3(R, G, B)), 1.0);
}
@fragment
fn fs(in: VOut) -> @location(0) vec4<f32> {
// The layer's premultiplied linear colour at this pixel, opacity applied.
fn layer_color(sp: vec2<f32>) -> vec4<f32> {
let n = max(u32(u.p0.z), 1u);
let fp = u.p2.w;
var acc = vec4(0.0);
for (var j = 0u; j < n; j++) {
for (var i = 0u; i < n; i++) {
let off = (vec2(f32(i), f32(j)) + 0.5) / f32(n) - 0.5;
acc += sample(in.sp + off * fp);
acc += sample(sp + off * fp);
}
}
return acc / f32(n * n) * u.p0.x;
}
// ---- Dissolve: the CPU's 64-bit pixel hash (`blend::hash2`), on 32-bit halves (lo, hi).
// Full 64-bit product of two u32.
fn mul32(a: u32, b: u32) -> vec2<u32> {
let a0 = a & 0xffffu; let a1 = a >> 16u; let b0 = b & 0xffffu; let b1 = b >> 16u;
let p00 = a0 * b0; let p01 = a0 * b1; let p10 = a1 * b0; let p11 = a1 * b1;
let mid = (p00 >> 16u) + (p01 & 0xffffu) + (p10 & 0xffffu);
return vec2((mid << 16u) | (p00 & 0xffffu), p11 + (p01 >> 16u) + (p10 >> 16u) + (mid >> 16u));
}
// Wrapping 64-bit multiply.
fn mul64(a: vec2<u32>, b: vec2<u32>) -> vec2<u32> {
let p = mul32(a.x, b.x);
return vec2(p.x, p.y + a.x * b.y + a.y * b.x);
}
// 64-bit shift right by 0 < n < 32.
fn shr64(a: vec2<u32>, n: u32) -> vec2<u32> {
return vec2((a.x >> n) | (a.y << (32u - n)), a.y >> n);
}
fn hash2(x: u32, y: u32) -> f32 {
var h = mul64(vec2(x, 0u), vec2(0x7F4A7C15u, 0x9E3779B9u)) ^ mul64(vec2(y, 0u), vec2(0x27D4EB4Fu, 0xC2B2AE3Du));
h = h ^ shr64(h, 31u);
h = mul64(h, vec2(0x1CE4E5B9u, 0xBF58476Du));
h = h ^ shr64(h, 29u);
return f32(h.y >> 8u) / 16777216.0;
}
// Normal and Dissolve (fixed-function premultiplied "over").
@fragment
fn fs(in: VOut) -> @location(0) vec4<f32> {
let c = layer_color(in.sp);
if u32(u.p3.x) == 1u {
// Dissolve: keep the pixel with probability alpha, then draw it opaque.
let sa = c.a;
if hash2(u32(in.pos.x), u32(in.pos.y)) >= sa {
discard;
}
return vec4(c.rgb / sa, 1.0);
}
return c;
}
// ---- the other blend modes: W3C Compositing and Blending formulas on straight, sRGB-encoded
// colour, exactly as `filmcraft_render::blend` (no recursion in WGSL, so helpers are split out).
fn screen(b: f32, s: f32) -> f32 { return b + s - b * s; }
fn color_burn(b: f32, s: f32) -> f32 {
if b >= 1.0 { return 1.0; }
if s <= 0.0 { return 0.0; }
return 1.0 - min((1.0 - b) / s, 1.0);
}
fn color_dodge(b: f32, s: f32) -> f32 {
if b <= 0.0 { return 0.0; }
if s >= 1.0 { return 1.0; }
return min(b / (1.0 - s), 1.0);
}
fn hard_light(b: f32, s: f32) -> f32 {
if s <= 0.5 { return b * 2.0 * s; }
return screen(b, 2.0 * s - 1.0);
}
fn vivid_light(b: f32, s: f32) -> f32 {
if s <= 0.5 { return color_burn(b, 2.0 * s); }
return color_dodge(b, 2.0 * s - 1.0);
}
fn sep(mode: u32, b: f32, s: f32) -> f32 {
switch mode {
case 2u: { return min(b, s); } // Darken
case 3u: { return b * s; } // Multiply
case 4u: { return color_burn(b, s); }
case 5u: { return max(b + s - 1.0, 0.0); } // Linear Burn
case 7u: { return max(b, s); } // Lighten
case 8u: { return screen(b, s); }
case 9u: { return color_dodge(b, s); }
case 10u: { return min(b + s, 1.0); } // Linear Dodge (Add)
case 12u: { return hard_light(s, b); } // Overlay
case 13u: { // Soft Light
if s <= 0.5 { return b - (1.0 - 2.0 * s) * b * (1.0 - b); }
var d = sqrt(b);
if b <= 0.25 { d = ((16.0 * b - 12.0) * b + 4.0) * b; }
return b + (2.0 * s - 1.0) * (d - b);
}
case 14u: { return hard_light(b, s); }
case 15u: { return vivid_light(b, s); }
case 16u: { return clamp(b + 2.0 * s - 1.0, 0.0, 1.0); } // Linear Light
case 17u: { // Pin Light
if s <= 0.5 { return min(b, 2.0 * s); }
return max(b, 2.0 * s - 1.0);
}
case 18u: { return select(0.0, 1.0, vivid_light(b, s) >= 0.5); } // Hard Mix
case 19u: { return abs(b - s); } // Difference
case 20u: { return b + s - 2.0 * b * s; } // Exclusion
case 21u: { return max(b - s, 0.0); } // Subtract
case 22u: { // Divide
if s <= 0.0 { return 1.0; }
return min(b / s, 1.0);
}
default: { return s; }
}
}
fn lum(c: vec3<f32>) -> f32 { return 0.3 * c.r + 0.59 * c.g + 0.11 * c.b; }
fn clip_color(c: vec3<f32>) -> vec3<f32> {
let l = lum(c);
let n = min(min(c.r, c.g), c.b);
let x = max(max(c.r, c.g), c.b);
var o = c;
if n < 0.0 { o = l + (o - l) * l / max(l - n, 1e-6); }
if x > 1.0 { o = l + (o - l) * (1.0 - l) / max(x - l, 1e-6); }
return o;
}
fn set_lum(c: vec3<f32>, l: f32) -> vec3<f32> { return clip_color(c + (l - lum(c))); }
fn sat(c: vec3<f32>) -> f32 { return max(max(c.r, c.g), c.b) - min(min(c.r, c.g), c.b); }
fn set_sat(c: vec3<f32>, s: f32) -> vec3<f32> {
let mx = max(max(c.r, c.g), c.b);
let mn = min(min(c.r, c.g), c.b);
if mx - mn <= 1e-6 { return vec3(0.0); }
return (c - mn) * s / (mx - mn);
}
fn blend_rgb(mode: u32, b: vec3<f32>, s: vec3<f32>) -> vec3<f32> {
switch mode {
case 23u: { return set_lum(set_sat(s, sat(b)), lum(b)); } // Hue
case 24u: { return set_lum(set_sat(b, sat(s)), lum(b)); } // Saturation
case 25u: { return set_lum(s, lum(b)); } // Color
case 26u: { return set_lum(b, lum(s)); } // Luminosity
case 6u: { return select(b, s, lum(s) < lum(b)); } // Darker Color
case 11u: { return select(b, s, lum(s) > lum(b)); } // Lighter Color
default: { return vec3(sep(mode, b.r, s.r), sep(mode, b.g, s.g), sep(mode, b.b, s.b)); }
}
}
fn linear_to_srgb(v: vec3<f32>) -> vec3<f32> {
let hi = 1.055 * pow(max(v, vec3(0.0)), vec3(1.0 / 2.4)) - 0.055;
return select(hi, v * 12.92, v <= vec3(0.0031308));
}
// Blend modes that need the destination: written without fixed-function blending.
@fragment
fn fs_blend(in: VOut) -> @location(0) vec4<f32> {
let sp = layer_color(in.sp);
let sa = sp.a;
if sa <= 0.0 {
discard;
}
let d = textureLoad(backdrop, vec2<i32>(in.pos.xy), 0);
let da = d.a;
if da <= 0.0 {
return sp + d * (1.0 - sa);
}
let cs = linear_to_srgb(sp.rgb / sa);
let cb = linear_to_srgb(d.rgb / da);
let mixed = srgb_to_linear(clamp(blend_rgb(u32(u.p3.x), cb, cs), vec3(0.0), vec3(1.0)));
return vec4(sp.rgb * (1.0 - da) + d.rgb * (1.0 - sa) + sa * da * mixed, sa + da - sa * da);
}
// ---- final pass: accumulator (linear premul) over black → sRGB target
@group(0) @binding(0) var accum: texture_2d<f32>;
+418
View File
@@ -0,0 +1,418 @@
//! The GPU effect stage: standard effects of a plan layer ([`LayerFx`]) run in WGSL (`fx.wgsl`).
//!
//! A layer with effects is first drawn from its source (YUV planes or RGBA, through the
//! compositor's own `fs` shader, so decoding is the same as for a plain layer) into a working
//! texture of the size the CPU decodes it at; each [`FxOp`] then runs as compute passes that
//! ping-pong between two working textures (three for Unsharp Mask, which keeps the original);
//! the last one is drawn into the accumulator with the layer's matrix, opacity and blend mode
//! like any RGBA layer. Working textures are `Rgba32Float` (the CPU reference's precision) and
//! pooled per size; layers without effects never touch any of this.
//!
//! Box blurs (Gaussian Blur, Camera Blur, the blur inside Unsharp / Sharpen) run per pixel for
//! radii up to [`BOX_PER_PIXEL_MAX`] and as a running sum per row / column above, so a huge
//! radius costs O(1) per pixel like on the CPU (radii are capped by `gaussian_boxes`).
use std::collections::{HashMap, HashSet};
use filmcraft_render::gpufx::FxOp;
use filmcraft_render::plan::LayerFx;
/// Working texture format.
pub(crate) const FX_FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba32Float;
/// Box radii up to this many pixels are summed per output pixel; larger ones as a running sum.
pub const BOX_PER_PIXEL_MAX: u32 = 32;
const OP_BRIGHTNESS_CONTRAST: u32 = 1;
const OP_PROC_AMP: u32 = 2;
const OP_TINT: u32 = 3;
const OP_BLACK_WHITE: u32 = 4;
const OP_COLOR_BALANCE: u32 = 5;
const OP_LEAVE_COLOR: u32 = 6;
const OP_CHANGE_TO_COLOR: u32 = 7;
const OP_COLOR_PASS: u32 = 8;
const OP_GAMMA: u32 = 9;
const OP_LEVELS: u32 = 10;
const OP_EXTRACT: u32 = 11;
const OP_INVERT: u32 = 12;
const OP_INVERT_ALPHA: u32 = 13;
const OP_POSTERIZE: u32 = 14;
const OP_ASC_CDL: u32 = 15;
const OP_CHANNEL_MIX: u32 = 16;
const OP_COLOR_REPLACE: u32 = 17;
const OP_ALPHA_ADJUST: u32 = 18;
const OP_BOX: u32 = 20;
const OP_DIRECTIONAL: u32 = 22;
const OP_UNSHARP: u32 = 23;
const OP_CROP: u32 = 24;
const OP_RESAMPLE: u32 = 25;
const OP_HFLIP: u32 = 26;
const OP_VFLIP: u32 = 27;
const OP_MIRROR: u32 = 28;
const OP_OFFSET: u32 = 29;
type Target = (wgpu::Texture, wgpu::TextureView);
pub(crate) struct FxStage {
px: wgpu::ComputePipeline,
run: wgpu::ComputePipeline,
bgl: wgpu::BindGroupLayout,
/// Draws a layer's source into a working texture (`composite.wgsl` `fs`, no blending).
source: wgpu::RenderPipeline,
pool: HashMap<(u32, u32), Vec<Target>>,
used: HashSet<(u32, u32)>,
}
/// One compute pass of a job.
struct Dispatch {
run: bool,
bg: wgpu::BindGroup,
groups: (u32, u32),
}
/// The recorded work of one layer's effects (bind groups made; recorded into the frame's encoder
/// right before the layer is drawn, so layers can share pooled textures).
pub(crate) struct FxJob {
source_bg: wgpu::BindGroup,
source_target: wgpu::TextureView,
dispatches: Vec<Dispatch>,
/// The texture holding the result.
pub(crate) result: wgpu::TextureView,
pub(crate) result_texture: wgpu::Texture,
}
/// One pass before texture routing: entry point, op code, integer and float parameters.
struct Step {
run: bool,
code: u32,
i1: [u32; 4],
p: [f32; 16],
/// Unsharp combine: reads the blurred image and the original.
combine: bool,
/// A blur pass of Unsharp Mask (the original stays untouched until the combine).
unsharp_blur: bool,
}
fn step(code: u32, i1: [u32; 4], p: &[f32]) -> Step {
let mut q = [0.0; 16];
for (d, s) in q.iter_mut().zip(p) {
*d = *s;
}
Step { run: false, code, i1, p: q, combine: false, unsharp_blur: false }
}
/// Box passes along x (vertical = false) or y for the radii (radius 0 is a no-op on the CPU).
fn boxes(radii: &[u32], vertical: bool, repeat: bool, out: &mut Vec<Step>) {
for &r in radii.iter().filter(|r| **r > 0) {
let mut s = step(OP_BOX, [r, repeat as u32, vertical as u32, 0], &[]);
s.run = r > BOX_PER_PIXEL_MAX;
out.push(s);
}
}
/// The passes of an op (empty: no-op). Unsharp's blur passes come first, then the combine.
fn steps(op: &FxOp) -> Vec<Step> {
let mut out = Vec::new();
match op {
FxOp::BrightnessContrast { br, co } => out.push(step(OP_BRIGHTNESS_CONTRAST, [0; 4], &[*br, *co])),
FxOp::ProcAmp { br, co, hue, sat } => out.push(step(OP_PROC_AMP, [0; 4], &[*br, *co, *hue, *sat])),
FxOp::Tint { black, white, amount } => {
out.push(step(OP_TINT, [0; 4], &[black[0], black[1], black[2], *amount, white[0], white[1], white[2]]));
}
FxOp::BlackWhite => out.push(step(OP_BLACK_WHITE, [0; 4], &[])),
FxOp::ColorBalance { sh, md, hi, preserve } => {
out.push(step(OP_COLOR_BALANCE, [*preserve as u32, 0, 0, 0], &[sh[0], sh[1], sh[2], 0.0, md[0], md[1], md[2], 0.0, hi[0], hi[1], hi[2]]))
}
FxOp::LeaveColor { amount, key_hue, tol, soft } => out.push(step(OP_LEAVE_COLOR, [0; 4], &[*amount, *key_hue, *tol, *soft])),
FxOp::ChangeToColor { from_hue, to_hue, tol, soft } => out.push(step(OP_CHANGE_TO_COLOR, [0; 4], &[*from_hue, *to_hue, *tol, *soft])),
FxOp::ColorPass { key, sim, reverse } => out.push(step(OP_COLOR_PASS, [*reverse as u32, 0, 0, 0], &[key[0], key[1], key[2], *sim])),
FxOp::Gamma { g } => out.push(step(OP_GAMMA, [0; 4], &[*g])),
FxOp::Levels { ib, iw, ob, ow, g } => out.push(step(OP_LEVELS, [0; 4], &[*ib, *iw, *ob, *ow, *g])),
FxOp::Extract { lo, hi, soft, invert } => out.push(step(OP_EXTRACT, [*invert as u32, 0, 0, 0], &[*lo, *hi, *soft])),
FxOp::Invert { channel: 4, blend } => out.push(step(OP_INVERT_ALPHA, [0; 4], &[*blend])),
FxOp::Invert { channel, blend } => out.push(step(OP_INVERT, [*channel, 0, 0, 0], &[*blend])),
FxOp::Posterize { n } => out.push(step(OP_POSTERIZE, [0; 4], &[*n])),
FxOp::Gaussian { rx, ry, repeat } => {
boxes(rx, false, *repeat, &mut out);
boxes(ry, true, *repeat, &mut out);
}
FxOp::DirectionalBlur { dx, dy, steps } => {
if *steps >= 2 {
out.push(step(OP_DIRECTIONAL, [*steps, 0, 0, 0], &[*dx, *dy]));
}
}
FxOp::Unsharp { rx, ry, amount, threshold } => {
boxes(rx, false, true, &mut out);
boxes(ry, true, true, &mut out);
for s in &mut out {
s.unsharp_blur = true;
}
let mut s = step(OP_UNSHARP, [0; 4], &[*amount, *threshold]);
s.combine = true;
out.push(s);
}
FxOp::Crop { x0, x1, y0, y1, feather } => out.push(step(OP_CROP, [0; 4], &[*x0, *x1, *y0, *y1, *feather])),
FxOp::Resample(r) => {
let op = if (r.opacity - 1.0).abs() < 1e-6 { 1.0 } else { r.opacity };
match &r.inv {
Some(m) => out.push(step(OP_RESAMPLE, r.rect, &[m.a as f32, m.b as f32, m.c as f32, m.d as f32, m.e as f32, m.f as f32, op, 1.0])),
None => out.push(step(OP_RESAMPLE, [0; 4], &[0.0; 8])),
}
}
FxOp::HFlip => out.push(step(OP_HFLIP, [0; 4], &[])),
FxOp::VFlip => out.push(step(OP_VFLIP, [0; 4], &[])),
FxOp::Mirror { cx, cy, nx, ny } => out.push(step(OP_MIRROR, [0; 4], &[*cx, *cy, *nx, *ny])),
FxOp::Offset { dx, dy, blend } => out.push(step(OP_OFFSET, [0; 4], &[*dx, *dy, *blend])),
FxOp::AscCdl { slope: s, offset: o, power: p, sat } => {
out.push(step(OP_ASC_CDL, [0; 4], &[s[0], s[1], s[2], *sat, o[0], o[1], o[2], 0.0, p[0], p[1], p[2]]));
}
FxOp::ChannelMix { m } => {
let p: Vec<f32> = m.iter().flatten().copied().collect();
out.push(step(OP_CHANNEL_MIX, [0; 4], &p));
}
FxOp::ColorReplace { sim, solid, target: t, replace: r, replace_hsl: h } => {
out.push(step(OP_COLOR_REPLACE, [*solid as u32, 0, 0, 0], &[t[0], t[1], t[2], *sim, r[0], r[1], r[2], 0.0, h[0], h[1], h[2]]))
}
FxOp::AlphaAdjust { opacity, ignore, invert, mask_only } => {
out.push(step(OP_ALPHA_ADJUST, [*ignore as u32, *invert as u32, *mask_only as u32, 0], &[*opacity]));
}
}
out
}
impl FxStage {
/// Whether `device` can run the stage: compute shaders with 16×16 workgroups and a storage
/// texture (WebGL2-class devices can't; they render effects on the CPU).
pub(crate) fn supported(device: &wgpu::Device) -> bool {
let l = device.limits();
l.max_storage_textures_per_shader_stage >= 1
&& l.max_compute_invocations_per_workgroup >= 256
&& l.max_compute_workgroup_size_x >= 64
&& l.max_compute_workgroup_size_y >= 16
&& l.max_compute_workgroups_per_dimension >= 1
}
pub(crate) fn new(device: &wgpu::Device, composite: &wgpu::ShaderModule, layer_bgl: &wgpu::BindGroupLayout) -> Self {
let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
label: Some("filmcraft-fx"),
source: wgpu::ShaderSource::Wgsl(include_str!("fx.wgsl").into()),
});
let tex = |binding| wgpu::BindGroupLayoutEntry {
binding,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Texture {
sample_type: wgpu::TextureSampleType::Float { filterable: false },
view_dimension: wgpu::TextureViewDimension::D2,
multisampled: false,
},
count: None,
};
let bgl = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("fx"),
entries: &[
wgpu::BindGroupLayoutEntry {
binding: 0,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Buffer { ty: wgpu::BufferBindingType::Uniform, has_dynamic_offset: false, min_binding_size: None },
count: None,
},
tex(1),
wgpu::BindGroupLayoutEntry {
binding: 2,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: FX_FORMAT,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
},
tex(3),
],
});
let pl = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor { label: Some("fx"), bind_group_layouts: &[Some(&bgl)], immediate_size: 0 });
let compute = |entry: &str| {
device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor {
label: Some("fx"),
layout: Some(&pl),
module: &shader,
entry_point: Some(entry),
compilation_options: Default::default(),
cache: None,
})
};
let spl = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("fx-source"),
bind_group_layouts: &[Some(layer_bgl)],
immediate_size: 0,
});
let source = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
label: Some("fx-source"),
layout: Some(&spl),
vertex: wgpu::VertexState { module: composite, entry_point: Some("vs"), compilation_options: Default::default(), buffers: &[] },
primitive: wgpu::PrimitiveState::default(),
depth_stencil: None,
multisample: wgpu::MultisampleState::default(),
fragment: Some(wgpu::FragmentState {
module: composite,
entry_point: Some("fs"),
compilation_options: Default::default(),
targets: &[Some(wgpu::ColorTargetState { format: FX_FORMAT, blend: None, write_mask: wgpu::ColorWrites::ALL })],
}),
multiview_mask: None,
cache: None,
});
Self { px: compute("fx_px"), run: compute("fx_run"), bgl, source, pool: HashMap::new(), used: HashSet::new() }
}
/// Start a frame: pooled textures of sizes no layer uses by [`end_frame`](Self::end_frame)
/// are released then.
pub(crate) fn begin_frame(&mut self) {
self.used.clear();
}
pub(crate) fn end_frame(&mut self) {
let used = &self.used;
self.pool.retain(|k, _| used.contains(k));
}
/// No working textures held.
#[cfg(test)]
pub(crate) fn is_idle(&self) -> bool {
self.pool.is_empty()
}
fn targets(&mut self, device: &wgpu::Device, size: (u32, u32), n: usize) -> Vec<Target> {
self.used.insert(size);
let v = self.pool.entry(size).or_default();
while v.len() < n {
let t = device.create_texture(&wgpu::TextureDescriptor {
label: Some("filmcraft-fx"),
size: wgpu::Extent3d { width: size.0, height: size.1, depth_or_array_layers: 1 },
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: FX_FORMAT,
usage: wgpu::TextureUsages::RENDER_ATTACHMENT
| wgpu::TextureUsages::TEXTURE_BINDING
| wgpu::TextureUsages::STORAGE_BINDING
| wgpu::TextureUsages::COPY_SRC,
view_formats: &[],
});
let view = t.create_view(&Default::default());
v.push((t, view));
}
v.iter().take(n).cloned().collect()
}
/// Prepare the passes of `fx`, whose source draw uses `source_bg` (a layer bind group whose
/// uniforms map the frame onto the working image). None when the working image does not fit
/// in a texture (the layer is then drawn without its effects — never larger than the source
/// frame, which would not have uploaded either).
pub(crate) fn job(
&mut self,
device: &wgpu::Device,
queue: &wgpu::Queue,
fx: &LayerFx,
source_bg: wgpu::BindGroup,
dummy: &wgpu::TextureView,
) -> Option<FxJob> {
let (w, h) = (fx.size.0.max(1), fx.size.1.max(1));
let max = device.limits().max_texture_dimension_2d;
if w > max || h > max {
return None;
}
let all: Vec<Step> = fx.ops.iter().flat_map(steps).collect();
let n = if all.iter().any(|s| s.combine) { 3 } else { 2 };
let targets = self.targets(device, (w, h), n);
let views: Vec<wgpu::TextureView> = targets.iter().map(|t| t.1.clone()).collect();
let mut dispatches = Vec::with_capacity(all.len());
let mut cur = 0usize;
// Unsharp: the image its blur started from (kept while its blur passes run)
let mut orig: Option<usize> = None;
for s in &all {
// (an Unsharp without blur passes, radius 0, combines the original with itself)
if (s.unsharp_blur || s.combine) && orig.is_none() {
orig = Some(cur);
}
let keep = if s.unsharp_blur || s.combine { orig } else { None };
let dst = (0..n).find(|k| *k != cur && Some(*k) != keep)?;
let aux = match (s.combine, orig) {
(true, Some(o)) => views.get(o)?,
_ => dummy,
};
let mut bytes = Vec::with_capacity(96);
for v in [s.code, w, h, 0].iter().chain(&s.i1) {
bytes.extend_from_slice(&v.to_le_bytes());
}
for v in &s.p {
bytes.extend_from_slice(&v.to_le_bytes());
}
let buf = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("fx-u"),
size: bytes.len() as u64,
usage: wgpu::BufferUsages::UNIFORM | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
});
queue.write_buffer(&buf, 0, &bytes);
let bg = device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("fx"),
layout: &self.bgl,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: buf.as_entire_binding() },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(views.get(cur)?) },
wgpu::BindGroupEntry { binding: 2, resource: wgpu::BindingResource::TextureView(views.get(dst)?) },
wgpu::BindGroupEntry { binding: 3, resource: wgpu::BindingResource::TextureView(aux) },
],
});
let groups = if s.run {
let lines = if s.i1[2] != 0 { w } else { h };
(lines.div_ceil(64), 1)
} else {
(w.div_ceil(16), h.div_ceil(16))
};
dispatches.push(Dispatch { run: s.run, bg, groups });
cur = dst;
if s.combine {
orig = None;
}
}
Some(FxJob {
source_bg,
source_target: views.first()?.clone(),
dispatches,
result: views.get(cur)?.clone(),
result_texture: targets.get(cur)?.0.clone(),
})
}
/// Record a job (source draw, then its compute passes) into `enc`.
pub(crate) fn record(&self, enc: &mut wgpu::CommandEncoder, job: &FxJob) {
{
let mut pass = enc.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("fx-source"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view: &job.source_target,
depth_slice: None,
resolve_target: None,
ops: wgpu::Operations { load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT), store: wgpu::StoreOp::Store },
})],
depth_stencil_attachment: None,
timestamp_writes: None,
occlusion_query_set: None,
multiview_mask: None,
});
pass.set_pipeline(&self.source);
pass.set_bind_group(0, &job.source_bg, &[]);
pass.draw(0..6, 0..1);
}
if job.dispatches.is_empty() {
return;
}
let mut pass = enc.begin_compute_pass(&wgpu::ComputePassDescriptor { label: Some("fx"), timestamp_writes: None });
for d in &job.dispatches {
pass.set_pipeline(if d.run { &self.run } else { &self.px });
pass.set_bind_group(0, &d.bg, &[]);
pass.dispatch_workgroups(d.groups.0, d.groups.1, 1);
}
}
}
+525
View File
@@ -0,0 +1,525 @@
// FilmCraft GPU effect stage: the standard effects of `filmcraft_render::gpufx::FxOp`, one
// compute pass each, reading the working image (`src`, linear premultiplied RGBA f32) and writing
// the next one (`dst`). The math mirrors `FxOp::apply` (the CPU reference) operation by
// operation; parameters arrive evaluated (keyframes, defaults and clamps applied on the CPU).
struct U {
i0: vec4<u32>, // op, width, height, unused
i1: vec4<u32>, // integer parameters (box radius, repeat, steps, channel, rect…)
p0: vec4<f32>,
p1: vec4<f32>,
p2: vec4<f32>,
p3: vec4<f32>,
};
@group(0) @binding(0) var<uniform> u: U;
@group(0) @binding(1) var src: texture_2d<f32>;
@group(0) @binding(2) var dst: texture_storage_2d<rgba32float, write>;
// Unsharp Mask: the image before blurring.
@group(0) @binding(3) var aux: texture_2d<f32>;
const OP_BRIGHTNESS_CONTRAST: u32 = 1u;
const OP_PROC_AMP: u32 = 2u;
const OP_TINT: u32 = 3u;
const OP_BLACK_WHITE: u32 = 4u;
const OP_COLOR_BALANCE: u32 = 5u;
const OP_LEAVE_COLOR: u32 = 6u;
const OP_CHANGE_TO_COLOR: u32 = 7u;
const OP_COLOR_PASS: u32 = 8u;
const OP_GAMMA: u32 = 9u;
const OP_LEVELS: u32 = 10u;
const OP_EXTRACT: u32 = 11u;
const OP_INVERT: u32 = 12u;
const OP_INVERT_ALPHA: u32 = 13u;
const OP_POSTERIZE: u32 = 14u;
const OP_ASC_CDL: u32 = 15u;
const OP_CHANNEL_MIX: u32 = 16u;
const OP_COLOR_REPLACE: u32 = 17u;
const OP_ALPHA_ADJUST: u32 = 18u;
const OP_BOX: u32 = 20u;
const OP_DIRECTIONAL: u32 = 22u;
const OP_UNSHARP: u32 = 23u;
const OP_CROP: u32 = 24u;
const OP_RESAMPLE: u32 = 25u;
const OP_HFLIP: u32 = 26u;
const OP_VFLIP: u32 = 27u;
const OP_MIRROR: u32 = 28u;
const OP_OFFSET: u32 = 29u;
fn size() -> vec2<i32> {
return vec2<i32>(i32(u.i0.y), i32(u.i0.z));
}
fn ld(p: vec2<i32>) -> vec4<f32> {
return textureLoad(src, p, 0);
}
// `Image::get_or_clear`
fn ld_or_clear(p: vec2<i32>) -> vec4<f32> {
let s = size();
if p.x < 0 || p.y < 0 || p.x >= s.x || p.y >= s.y {
return vec4(0.0);
}
return textureLoad(src, p, 0);
}
// `Image::sample_bilinear`: pixel centres at +0.5, transparent outside.
fn sample_bilinear(x: f32, y: f32) -> vec4<f32> {
let fx = x - 0.5;
let fy = y - 0.5;
let x0 = floor(fx);
let y0 = floor(fy);
let tx = fx - x0;
let ty = fy - y0;
let i = vec2<i32>(i32(x0), i32(y0));
let a = ld_or_clear(i);
let b = ld_or_clear(i + vec2(1, 0));
let c = ld_or_clear(i + vec2(0, 1));
let d = ld_or_clear(i + vec2(1, 1));
let top = a + (b - a) * tx;
let bot = c + (d - c) * tx;
return top + (bot - top) * ty;
}
// `Image::sample_bilinear_clamped`
fn sample_clamped(x: f32, y: f32) -> vec4<f32> {
let s = size();
let fx = clamp(x - 0.5, 0.0, f32(s.x) - 1.0);
let fy = clamp(y - 0.5, 0.0, f32(s.y) - 1.0);
let x0 = i32(floor(fx));
let y0 = i32(floor(fy));
let x1 = min(x0 + 1, s.x - 1);
let y1 = min(y0 + 1, s.y - 1);
let tx = fx - f32(x0);
let ty = fy - f32(y0);
let a = ld(vec2(x0, y0));
let b = ld(vec2(x1, y0));
let c = ld(vec2(x0, y1));
let d = ld(vec2(x1, y1));
let top = a + (b - a) * tx;
let bot = c + (d - c) * tx;
return top + (bot - top) * ty;
}
// ---- colour helpers (`filmcraft_color`, `effects::enc` / `dec`)
fn linear_to_srgb1(v: f32) -> f32 {
if v <= 0.0031308 {
return v * 12.92;
}
return 1.055 * pow(v, 1.0 / 2.4) - 0.055;
}
fn srgb_to_linear1(v: f32) -> f32 {
if v <= 0.04045 {
return v / 12.92;
}
return pow((v + 0.055) / 1.055, 2.4);
}
fn enc(c: vec3<f32>) -> vec3<f32> {
let m = max(c, vec3(0.0));
return vec3(linear_to_srgb1(m.x), linear_to_srgb1(m.y), linear_to_srgb1(m.z));
}
fn dec(c: vec3<f32>) -> vec3<f32> {
let m = clamp(c, vec3(0.0), vec3(1.0));
return vec3(srgb_to_linear1(m.x), srgb_to_linear1(m.y), srgb_to_linear1(m.z));
}
fn luma709(c: vec3<f32>) -> f32 {
return 0.2126 * c.x + 0.7152 * c.y + 0.0722 * c.z;
}
// `f32::powf` for x >= 0 (WGSL leaves pow(0, y) undefined).
fn powf(x: f32, y: f32) -> f32 {
if x == 0.0 {
if y == 0.0 {
return 1.0;
}
if y > 0.0 {
return 0.0;
}
return 3.0e38;
}
return pow(x, y);
}
// `f32::round`: halves away from zero (WGSL `round` rounds them to even).
fn round_away(x: f32) -> f32 {
let a = abs(x);
let f = floor(a);
let r = select(f, f + 1.0, a - f >= 0.5);
return select(r, -r, x < 0.0);
}
// `rem_euclid(1.0)`
fn fract_euclid(x: f32) -> f32 {
return x - floor(x);
}
// `vfx::smoothstep` (not WGSL's: a degenerate edge pair divides by 1e-6)
fn smoothstep_fx(e0: f32, e1: f32, x: f32) -> f32 {
let t = clamp((x - e0) / max(e1 - e0, 1e-6), 0.0, 1.0);
return t * t * (3.0 - 2.0 * t);
}
fn rgb_to_hsl(c: vec3<f32>) -> vec3<f32> {
let r = c.x; let g = c.y; let b = c.z;
let mx = max(max(r, g), b);
let mn = min(min(r, g), b);
let l = (mx + mn) / 2.0;
if abs(mx - mn) < 1e-7 {
return vec3(0.0, 0.0, l);
}
let d = mx - mn;
var s: f32;
if l > 0.5 {
s = d / (2.0 - mx - mn);
} else {
s = d / (mx + mn);
}
var h: f32;
if mx == r {
h = (g - b) / d + select(0.0, 6.0, g < b);
} else if mx == g {
h = (b - r) / d + 2.0;
} else {
h = (r - g) / d + 4.0;
}
return vec3(h / 6.0, s, l);
}
fn hsl_channel(p: f32, q: f32, t0: f32) -> f32 {
let t = fract_euclid(t0);
if t < 1.0 / 6.0 {
return p + (q - p) * 6.0 * t;
}
if t < 0.5 {
return q;
}
if t < 2.0 / 3.0 {
return p + (q - p) * (2.0 / 3.0 - t) * 6.0;
}
return p;
}
fn hsl_to_rgb(h: f32, s: f32, l: f32) -> vec3<f32> {
if s <= 0.0 {
return vec3(l);
}
var q: f32;
if l < 0.5 {
q = l * (1.0 + s);
} else {
q = l + s - l * s;
}
let p = 2.0 * l - q;
return vec3(hsl_channel(p, q, h + 1.0 / 3.0), hsl_channel(p, q, h), hsl_channel(p, q, h - 1.0 / 3.0));
}
// ---- per-pixel colour effects on straight colour (`Image::map_rgb`)
fn color_op(op: u32, c: vec3<f32>) -> vec3<f32> {
switch op {
case OP_BRIGHTNESS_CONTRAST: {
let br = u.p0.x; let co = u.p0.y;
return dec((enc(c) - 0.5) * co + 0.5 + br);
}
case OP_PROC_AMP: {
let br = u.p0.x; let co = u.p0.y; let hue = u.p0.z; let sat = u.p0.w;
var hsl = rgb_to_hsl(enc(c));
hsl.x = fract_euclid(hsl.x + hue);
hsl.y = clamp(hsl.y * sat, 0.0, 1.0);
return dec((hsl_to_rgb(hsl.x, hsl.y, hsl.z) - 0.5) * co + 0.5 + br);
}
case OP_TINT: {
let bl = u.p0.xyz; let wh = u.p1.xyz; let amt = u.p0.w;
let l = linear_to_srgb1(max(luma709(c), 0.0));
let t = bl + (wh - bl) * l;
let e = enc(c);
return dec(e + (t - e) * amt);
}
case OP_BLACK_WHITE: {
return vec3(luma709(c));
}
case OP_COLOR_BALANCE: {
let e = enc(c);
let l = 0.2126 * e.x + 0.7152 * e.y + 0.0722 * e.z;
let ws = (1.0 - l) * (1.0 - l);
let wh = l * l;
let wm = 1.0 - ws - wh;
var o = e + u.p0.xyz * ws + u.p1.xyz * max(wm, 0.0) + u.p2.xyz * wh;
if u.i1.x != 0u {
let l2 = 0.2126 * o.x + 0.7152 * o.y + 0.0722 * o.z;
o = o + (l - l2);
}
return dec(o);
}
case OP_LEAVE_COLOR: {
let amt = u.p0.x; let kh = u.p0.y; let tol = u.p0.z; let soft = u.p0.w;
let h = rgb_to_hsl(enc(c)).x;
let d = min(abs(h - kh), 1.0 - abs(h - kh)) * 2.0;
let keep = 1.0 - clamp((d - tol) / soft, 0.0, 1.0);
let l = luma709(c);
let k = amt * (1.0 - keep);
return c + (l - c) * k;
}
case OP_CHANGE_TO_COLOR: {
let fh = u.p0.x; let th = u.p0.y; let tol = u.p0.z; let soft = u.p0.w;
var hsl = rgb_to_hsl(enc(c));
let d = min(abs(hsl.x - fh), 1.0 - abs(hsl.x - fh));
let w = 1.0 - clamp((d - tol) / soft, 0.0, 1.0);
hsl.x = fract_euclid(hsl.x + (th - fh) * w);
return dec(hsl_to_rgb(hsl.x, hsl.y, hsl.z));
}
case OP_COLOR_PASS: {
let e = enc(c);
let k = e - u.p0.xyz;
let d = sqrt(k.x * k.x + k.y * k.y + k.z * k.z);
let passes = (d <= u.p0.w * 1.2) != (u.i1.x != 0u);
if passes {
return c;
}
return vec3(luma709(c));
}
case OP_GAMMA: {
let e = enc(c);
let g = u.p0.x;
return dec(vec3(powf(max(e.x, 0.0), g), powf(max(e.y, 0.0), g), powf(max(e.z, 0.0), g)));
}
case OP_LEVELS: {
let ib = u.p0.x; let iw = u.p0.y; let ob = u.p0.z; let ow = u.p0.w; let g = u.p1.x;
let v = clamp((enc(c) - ib) / (iw - ib), vec3(0.0), vec3(1.0));
return dec(ob + vec3(powf(v.x, g), powf(v.y, g), powf(v.z, g)) * (ow - ob));
}
case OP_EXTRACT: {
let lo = u.p0.x; let hi = u.p0.y; let soft = u.p0.z;
let l = linear_to_srgb1(max(luma709(c), 0.0));
let inside = min(clamp((l - lo) / soft, 0.0, 1.0), clamp((hi - l) / soft, 0.0, 1.0));
return vec3(select(inside, 1.0 - inside, u.i1.x != 0u));
}
case OP_INVERT: {
let ch = u.i1.x;
let blend = u.p0.x;
let e = enc(c);
var o = e;
if ch == 0u || ch == 1u { o.x = 1.0 - e.x; }
if ch == 0u || ch == 2u { o.y = 1.0 - e.y; }
if ch == 0u || ch == 3u { o.z = 1.0 - e.z; }
return dec(o + (e - o) * blend);
}
case OP_POSTERIZE: {
let n = u.p0.x;
let e = enc(c) * n;
return dec(vec3(round_away(e.x), round_away(e.y), round_away(e.z)) / n);
}
case OP_ASC_CDL: {
let e = enc(c);
let s = u.p0.xyz; let o = u.p1.xyz; let pw = max(u.p2.xyz, vec3(0.0)); let sat = u.p0.w;
let q = clamp(e * s + o, vec3(0.0), vec3(1.0));
let v = vec3(powf(q.x, pw.x), powf(q.y, pw.y), powf(q.z, pw.z));
let l = luma709(v);
return dec(clamp(l + sat * (v - l), vec3(0.0), vec3(1.0)));
}
case OP_CHANNEL_MIX: {
let v = enc(c);
let r = u.p0; let g = u.p1; let b = u.p2;
return dec(clamp(vec3(r.x * v.x + r.y * v.y + r.z * v.z + r.w, g.x * v.x + g.y * v.y + g.z * v.z + g.w, b.x * v.x + b.y * v.y + b.z * v.z + b.w), vec3(0.0), vec3(1.0)));
}
case OP_COLOR_REPLACE: {
let v = enc(c);
let t = u.p0.xyz; let sim = u.p0.w;
let dd = v - t;
let d = sqrt(dd.x * dd.x + dd.y * dd.y + dd.z * dd.z);
let k = 1.0 - smoothstep_fx(sim * 0.85, max(sim, 1e-4), d);
if k <= 0.0 {
return c;
}
var repl = u.p1.xyz;
if u.i1.x == 0u {
let rh = u.p2.xyz;
repl = hsl_to_rgb(rh.x, rh.y, rgb_to_hsl(v).z);
}
return dec(v + (repl - v) * k);
}
default: {
return c;
}
}
}
// One output pixel of every op but the running-sum box blur.
fn pixel(op: u32, p: vec2<i32>) -> vec4<f32> {
let s = size();
let pc = vec2<f32>(p) + 0.5;
switch op {
case OP_INVERT_ALPHA: {
let o = ld(p);
let a = o.a;
let na = 1.0 - a;
let k = select(0.0, na / a, a > 1e-6);
let blend = u.p0.x;
return vec4(o.rgb * k, na * (1.0 - blend) + a * blend);
}
case OP_ALPHA_ADJUST: {
let o = ld(p);
let c = select(o.rgb / o.a, vec3(0.0), o.a <= 1e-6);
var a = select(o.a, 1.0, u.i1.x != 0u);
if u.i1.y != 0u {
a = 1.0 - a;
}
a = clamp(a * u.p0.x, 0.0, 1.0);
if u.i1.z != 0u {
return vec4(vec3(srgb_to_linear1(a)), 1.0);
}
return vec4(c * a, a);
}
case OP_BOX: {
// one box pass along x (i1.z = 0) or y (1): mean over 2r+1 pixels, clamped to the
// edge (repeat) or skipping pixels outside, as `effects::box_rows`
let r = i32(u.i1.x);
let repeat = u.i1.y != 0u;
let vertical = u.i1.z != 0u;
let n = select(s.x, s.y, vertical);
let x = select(p.x, p.y, vertical);
var acc = vec4(0.0);
for (var i = x - r; i <= x + r; i++) {
if repeat || (i >= 0 && i < n) {
let j = clamp(i, 0, n - 1);
acc += ld(select(vec2(j, p.y), vec2(p.x, j), vertical));
}
}
return acc * (1.0 / f32(2 * r + 1));
}
case OP_DIRECTIONAL: {
let steps = u.i1.x;
let dx = u.p0.x; let dy = u.p0.y;
var acc = vec4(0.0);
for (var i = 0u; i < steps; i++) {
let t = f32(i) / f32(steps - 1u) - 0.5;
acc += sample_clamped(pc.x + dx * t, pc.y + dy * t);
}
return acc / f32(steps);
}
case OP_UNSHARP: {
// src: the blurred image, aux: the original
var o = textureLoad(aux, p, 0);
let bq = ld(p);
let amount = u.p0.x; let th = u.p0.y;
for (var k = 0; k < 3; k++) {
let d = o[k] - bq[k];
if abs(d) >= th * o.a {
o[k] = max(o[k] + d * amount, 0.0);
}
}
return o;
}
case OP_CROP: {
let x0 = u.p0.x; let x1 = u.p0.y; let y0 = u.p0.z; let y1 = u.p0.w; let fe = u.p1.x;
let d = min(min(pc.x - x0, x1 - pc.x), min(pc.y - y0, y1 - pc.y));
var a: f32;
if fe > 0.0 {
a = clamp(d / fe, 0.0, 1.0);
} else {
a = clamp(d + 0.5, 0.0, 1.0);
}
let o = ld(p);
return select(o, o * a, a < 1.0);
}
case OP_RESAMPLE: {
// inverse map p0 = (a b c d), p1 = (e f opacity valid); rect i1 = x0 x1 y0 y1
if u.p1.w == 0.0 || u32(p.x) < u.i1.x || u32(p.x) >= u.i1.y || u32(p.y) < u.i1.z || u32(p.y) >= u.i1.w {
return vec4(0.0);
}
let m0 = u.p0; let m1 = u.p1;
let uu = m0.x * pc.x + m0.z * pc.y + m1.x;
let vv = m0.y * pc.x + m0.w * pc.y + m1.y;
if uu < -1.0 || vv < -1.0 || uu > f32(s.x) + 1.0 || vv > f32(s.y) + 1.0 {
return vec4(0.0);
}
return sample_bilinear(uu, vv) * m1.z;
}
case OP_HFLIP: {
return ld(vec2(s.x - 1 - p.x, p.y));
}
case OP_VFLIP: {
return ld(vec2(p.x, s.y - 1 - p.y));
}
case OP_MIRROR: {
let c = u.p0.xy; let nrm = u.p0.zw;
let d = (pc.x - c.x) * nrm.x + (pc.y - c.y) * nrm.y;
if d > 0.0 {
return sample_bilinear(pc.x - 2.0 * d * nrm.x, pc.y - 2.0 * d * nrm.y);
}
return ld(p);
}
case OP_OFFSET: {
let w = f32(s.x); let h = f32(s.y);
// dx, dy in 0..w / 0..h, so the shifted coordinate is within (-w, w + 0.5)
var uu = pc.x - u.p0.x;
var vv = pc.y - u.p0.y;
if uu < 0.0 { uu += w; }
if uu >= w { uu -= w; }
if vv < 0.0 { vv += h; }
if vv >= h { vv -= h; }
let q = sample_clamped(uu, vv);
let o = ld(p);
return q + (o - q) * u.p0.z;
}
default: {
let o = ld(p);
let a = o.a;
if a <= 1e-6 {
return o;
}
return vec4(color_op(op, o.rgb / a) * a, a);
}
}
}
@compute @workgroup_size(16, 16)
fn fx_px(@builtin(global_invocation_id) id: vec3<u32>) {
if id.x >= u.i0.y || id.y >= u.i0.z {
return;
}
let p = vec2<i32>(id.xy);
textureStore(dst, p, pixel(u.i0.x, p));
}
// A box pass as a running sum along one row (i1.z = 0) or column (1) per invocation: O(1) per
// pixel for any radius, the arithmetic of `effects::box_rows`.
@compute @workgroup_size(64)
fn fx_run(@builtin(global_invocation_id) id: vec3<u32>) {
let s = size();
let vertical = u.i1.z != 0u;
let n = select(s.x, s.y, vertical);
let lines = select(s.y, s.x, vertical);
let line = i32(id.x);
if line >= lines {
return;
}
let r = i32(u.i1.x);
let repeat = u.i1.y != 0u;
let inv = 1.0 / f32(2 * r + 1);
var acc = vec4(0.0);
for (var i = -r; i <= r; i++) {
if repeat || (i >= 0 && i < n) {
let j = clamp(i, 0, n - 1);
acc += ld(select(vec2(j, line), vec2(line, j), vertical));
}
}
for (var x = 0; x < n; x++) {
textureStore(dst, select(vec2(x, line), vec2(line, x), vertical), acc * inv);
let out_i = x - r;
let in_i = x + r + 1;
if repeat || out_i >= 0 {
let j = clamp(out_i, 0, n - 1);
acc -= ld(select(vec2(j, line), vec2(line, j), vertical));
}
if repeat || in_i < n {
let j = clamp(in_i, 0, n - 1);
acc += ld(select(vec2(j, line), vec2(line, j), vertical));
}
}
}
+491
View File
@@ -0,0 +1,491 @@
//! GPU-vs-CPU parity of the effect stage (`filmcraft_render::gpufx::FxOp::apply` is the
//! reference).
//!
//! Tolerances:
//! - Exact checks ([`gpu_effects_match_cpu_exactly`]): the working image read back as f32 from a
//! half-float-representable source (so the source draw is lossless) must match the CPU op to
//! [`EXACT`] per value (f32 arithmetic and the GPU's `pow` / `sqrt` differ in the last bits).
//! Step functions (Posterize, Color Pass, Leave Color and Change to Color keys, Extract) may
//! flip a pixel that lands on a threshold; at most [`FLIPS`] of the values may differ by more.
//! - Composited plans ([`gpu_effect_layers_match_cpu_plan`]): the compositor's criterion (p99 of
//! the per-pixel max 8-bit channel difference ≤ 6, mean < 1.5) away from layer outlines, as for
//! the blend modes.
use super::blend_tests::{device, interior, ramp_layer, stats8, yuv_frame};
use super::*;
use filmcraft_geom::{Affine, Vec2};
use filmcraft_project::{EffectInstance, ParamValue, find_effect};
use filmcraft_render::effects::FxCtx;
use filmcraft_render::gpufx::FxOp;
use filmcraft_render::plan::{LayerFx, execute_cpu};
use filmcraft_time::Tick;
const EXACT: f32 = 1e-4;
const FLIPS: f64 = 0.01;
fn cx(t: Tick, px_scale: f32) -> FxCtx<'static> {
FxCtx { t, px_scale, seconds: 0.0, timecode: "", clip_name: "", project: None, env: None, working: filmcraft_color::WorkingSpace::Rec709 }
}
fn effect(id: &str, params: &[(&str, ParamValue)]) -> EffectInstance {
let mut e = find_effect(id).unwrap_or_else(|| panic!("{id}")).instance();
for (k, v) in params {
e.params.get_mut(*k).unwrap_or_else(|| panic!("{id}.{k}")).value = v.clone();
}
e
}
fn fl(v: f64) -> ParamValue {
ParamValue::Float(v)
}
fn col(r: f32, g: f32, b: f32) -> ParamValue {
ParamValue::Color([r, g, b, 1.0])
}
fn pt(x: f64, y: f64) -> ParamValue {
ParamValue::Vec2(Vec2::new(x, y))
}
fn half(v: f32) -> f32 {
super::blend_tests::f16_to_f32(f32_to_f16(v))
}
/// Linear premultiplied test picture, every value representable in half floats: hue sweep along
/// x, lightness along y, a grey ramp, black and white columns, super-whites (up to 1.6), and
/// alpha 1 / partial / 0 bands.
fn picture(w: u32, h: u32) -> Vec<f32> {
let mut px = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
let fx = x as f32 / (w - 1) as f32;
let fy = y as f32 / (h - 1) as f32;
let c = match x % 11 {
0 => [0.0, 0.0, 0.0],
1 => [1.0, 1.0, 1.0],
2 => [fy, fy, fy],
3 => [1.6 * fy, 1.2 * fy, 0.3],
_ => {
let rgb = filmcraft_color::hsl_to_rgb(fx, 0.25 + 0.75 * ((y % 5) as f32 / 4.0), 0.1 + 0.8 * fy);
rgb.map(filmcraft_color::srgb_to_linear)
}
};
let a = match y % 7 {
0 => 0.0,
1 | 2 => 0.5 + 0.4 * fx,
3 => 0.25,
_ => 1.0,
};
px.extend_from_slice(&[c[0] * a, c[1] * a, c[2] * a, a].map(half));
}
}
px
}
/// Max |cpu − gpu| (finite values required) and the share of values off by more than 0.02.
fn compare(cpu: &[f32], gpu: &[f32]) -> (f32, f64) {
assert_eq!(cpu.len(), gpu.len());
let mut worst = 0f32;
let mut flips = 0usize;
for (a, b) in cpu.iter().zip(gpu) {
assert!(b.is_finite(), "non-finite GPU value");
let d = (a - b).abs() / (1.0 + a.abs());
if d > 0.02 {
flips += 1;
} else {
worst = worst.max(d);
}
}
(worst, flips as f64 / cpu.len() as f64)
}
/// Effect cases: (effect id, parameters, whether a threshold may flip pixels).
fn cases() -> Vec<(&'static str, Vec<(&'static str, ParamValue)>, bool)> {
vec![
("brightness_contrast", vec![("brightness", fl(30.0)), ("contrast", fl(40.0))], false),
("brightness_contrast", vec![("brightness", fl(-100.0)), ("contrast", fl(-100.0))], false),
("brightness_contrast", vec![("brightness", fl(100.0)), ("contrast", fl(100.0))], false),
("brightness_contrast", vec![("brightness", fl(-20.0)), ("contrast", fl(900.0))], false),
("proc_amp", vec![("brightness", fl(10.0)), ("contrast", fl(130.0)), ("hue", fl(200.0)), ("saturation", fl(150.0))], false),
("proc_amp", vec![("hue", fl(-400.0)), ("saturation", fl(0.0))], false),
("tint", vec![], false),
("tint", vec![("black", col(0.2, 0.1, 0.5)), ("white", col(1.0, 0.9, 0.3)), ("amount", fl(60.0))], false),
("black_white", vec![], false),
("color_balance", vec![("shadow_r", fl(50.0)), ("mid_g", fl(-40.0)), ("hi_b", fl(80.0)), ("shadow_b", fl(-100.0)), ("hi_r", fl(100.0))], false),
("color_balance", vec![("mid_r", fl(70.0)), ("hi_g", fl(-60.0)), ("preserve", ParamValue::Bool(true))], false),
("leave_color", vec![("amount", fl(80.0)), ("tolerance", fl(20.0)), ("softness", fl(30.0))], true),
("leave_color", vec![("amount", fl(100.0)), ("color", col(0.1, 0.8, 0.2)), ("tolerance", fl(0.0)), ("softness", fl(0.0))], true),
("change_to_color", vec![], true),
("change_to_color", vec![("hue_tol", fl(30.0)), ("softness", fl(10.0)), ("to", col(0.9, 0.9, 0.1))], true),
("color_pass", vec![("similarity", fl(30.0))], true),
("color_pass", vec![("similarity", fl(45.0)), ("reverse", ParamValue::Bool(true)), ("color", col(0.2, 0.4, 0.9))], true),
("gamma_correction", vec![("gamma", fl(5.0))], false),
("gamma_correction", vec![("gamma", fl(28.0))], false),
("gamma_correction", vec![("gamma", fl(0.0))], false),
("gamma_correction", vec![("gamma", fl(-3.0))], false),
("levels", vec![("in_black", fl(20.0)), ("in_white", fl(230.0)), ("out_black", fl(10.0)), ("out_white", fl(240.0)), ("gamma", fl(150.0))], false),
("levels", vec![("in_black", fl(200.0)), ("in_white", fl(0.0)), ("gamma", fl(5.0)), ("out_white", fl(-50.0))], true),
("extract", vec![("black", fl(40.0)), ("white", fl(200.0)), ("softness", fl(30.0))], true),
("extract", vec![("black", fl(90.0)), ("white", fl(160.0)), ("invert", ParamValue::Bool(true))], true),
("invert", vec![("blend", fl(30.0))], false),
("invert", vec![("channel", ParamValue::Choice(1))], false),
("invert", vec![("channel", ParamValue::Choice(2)), ("blend", fl(70.0))], false),
("invert", vec![("channel", ParamValue::Choice(3))], false),
("invert", vec![("channel", ParamValue::Choice(4)), ("blend", fl(20.0))], false),
("invert", vec![("channel", ParamValue::Choice(9))], false),
("posterize", vec![("levels", fl(2.0))], true),
("posterize", vec![("levels", fl(7.0))], true),
("posterize", vec![("levels", fl(255.0))], true),
("gaussian_blur", vec![("blurriness", fl(0.0))], false),
("gaussian_blur", vec![("blurriness", fl(5.0))], false),
("gaussian_blur", vec![("blurriness", fl(5.0)), ("repeat_edge", ParamValue::Bool(true))], false),
("gaussian_blur", vec![("blurriness", fl(80.0))], false),
("gaussian_blur", vec![("blurriness", fl(80.0)), ("repeat_edge", ParamValue::Bool(true))], false),
("gaussian_blur", vec![("blurriness", fl(12.0)), ("dimensions", ParamValue::Choice(1))], false),
("gaussian_blur", vec![("blurriness", fl(12.0)), ("dimensions", ParamValue::Choice(2)), ("repeat_edge", ParamValue::Bool(true))], false),
("gaussian_blur", vec![("blurriness", fl(3000.0)), ("repeat_edge", ParamValue::Bool(true))], false),
("gaussian_blur", vec![("blurriness", fl(1e9))], false),
("gaussian_blur", vec![("blurriness", fl(-50.0))], false),
("camera_blur", vec![("percent", fl(20.0))], false),
("directional_blur", vec![("length", fl(10.0)), ("direction", fl(30.0))], false),
("directional_blur", vec![("length", fl(0.2))], false),
("directional_blur", vec![("length", fl(1000.0)), ("direction", fl(-100.0))], false),
("sharpen", vec![("amount", fl(80.0))], false),
("sharpen", vec![("amount", fl(4000.0))], false),
("unsharp_mask", vec![("amount", fl(150.0)), ("radius", fl(3.0)), ("threshold", fl(10.0))], true),
("unsharp_mask", vec![("radius", fl(0.1))], false),
("unsharp_mask", vec![("amount", fl(300.0)), ("radius", fl(250.0))], false),
("crop", vec![("left", fl(10.0)), ("top", fl(5.0)), ("right", fl(20.0)), ("bottom", fl(15.0))], false),
("crop", vec![("left", fl(10.0)), ("top", fl(5.0)), ("right", fl(20.0)), ("bottom", fl(15.0)), ("feather", fl(12.0))], false),
("crop", vec![("left", fl(12.0)), ("top", fl(5.0)), ("right", fl(20.0)), ("zoom", ParamValue::Bool(true))], false),
("crop", vec![("left", fl(70.0)), ("right", fl(70.0)), ("feather", fl(5000.0))], false),
("edge_feather", vec![("amount", fl(30.0))], false),
("transform", vec![("scale_height", fl(150.0)), ("rotation", fl(30.0)), ("position", pt(30.0, 20.0)), ("opacity", fl(60.0))], false),
(
"transform",
vec![
("uniform_scale", ParamValue::Bool(false)),
("scale_width", fl(80.0)),
("scale_height", fl(120.0)),
("skew", fl(20.0)),
("skew_axis", fl(45.0)),
("anchor", pt(10.0, 30.0)),
],
false,
),
("horizontal_flip", vec![], false),
("vertical_flip", vec![], false),
("mirror", vec![("center", pt(20.0, 10.0)), ("angle", fl(30.0))], false),
("mirror", vec![("angle", fl(200.0))], false),
("offset", vec![("shift", pt(10.0, 30.0)), ("blend", fl(25.0))], false),
("offset", vec![("shift", pt(-1e5, 7.25))], false),
("asc_cdl", vec![("r_slope", fl(1.4)), ("g_offset", fl(-0.1)), ("b_power", fl(2.2)), ("saturation", fl(1.6))], false),
("asc_cdl", vec![("r_power", fl(0.0)), ("g_slope", fl(0.0)), ("saturation", fl(0.0))], false),
("channel_mix", vec![("rr", fl(50.0)), ("rg", fl(60.0)), ("gb", fl(-80.0)), ("bc", fl(30.0))], false),
("channel_mix", vec![("rb", fl(120.0)), ("monochrome", ParamValue::Bool(true))], false),
("color_replace", vec![("similarity", fl(40.0))], true),
("color_replace", vec![("similarity", fl(25.0)), ("solid", ParamValue::Bool(true)), ("target", col(0.3, 0.6, 0.2))], true),
("color_replace", vec![("similarity", fl(0.0))], true),
("alpha_adjust", vec![("opacity", fl(60.0))], false),
("alpha_adjust", vec![("ignore", ParamValue::Bool(true)), ("invert", ParamValue::Bool(true))], false),
("alpha_adjust", vec![("invert", ParamValue::Bool(true)), ("mask_only", ParamValue::Bool(true)), ("opacity", fl(150.0))], false),
]
}
/// Every effect case on the GPU against its CPU reference, on the working image read back as f32.
#[test]
fn gpu_effects_match_cpu_exactly() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let (w, h) = (67u32, 41u32);
let px = picture(w, h);
let frame = VideoFrame::rgba_f32(w, h, px.clone());
let mut failures = Vec::new();
for (id, params, steps) in cases() {
let e = effect(id, &params);
let op = FxOp::eval(&e, &cx(Tick::ZERO, 1.0), w as usize, h as usize).unwrap_or_else(|| panic!("{id} has no GPU op"));
assert!(op.gpu_ok(), "{id} {params:?}: not GPU-capable");
let mut cpu = filmcraft_render::Image { w: w as usize, h: h as usize, px: px.clone() };
op.apply(&mut cpu);
let fx = LayerFx { size: (w, h), decimation: 1, ops: vec![op] };
let (_, _, gpu) = c.effect_image(&frame, &fx).expect("effect image");
let (worst, flips) = compare(&cpu.px, &gpu);
eprintln!("{id} {params:?}: max rel diff {worst:.2e}, {:.3}% flipped", flips * 100.0);
if worst > EXACT || flips > if steps { FLIPS } else { 0.0 } {
failures.push(format!("{id} {params:?}: {worst:.2e}, flips {flips:.4}"));
}
}
assert!(failures.is_empty(), "GPU effects differ from the CPU:\n{}", failures.join("\n"));
}
/// Chains run in order (and Unsharp keeps its original while blurring, between other blurs).
#[test]
fn gpu_effect_chains_match_cpu() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let (w, h) = (96u32, 54u32);
let px = picture(w, h);
let frame = VideoFrame::rgba_f32(w, h, px.clone());
let chains: Vec<Vec<EffectInstance>> = vec![
vec![
effect("brightness_contrast", &[("brightness", fl(20.0)), ("contrast", fl(30.0))]),
effect("gaussian_blur", &[("blurriness", fl(8.0))]),
effect("tint", &[("amount", fl(50.0))]),
],
vec![
effect("gaussian_blur", &[("blurriness", fl(90.0)), ("repeat_edge", ParamValue::Bool(true))]),
effect("unsharp_mask", &[("amount", fl(200.0)), ("radius", fl(4.0))]),
effect("gaussian_blur", &[("blurriness", fl(3.0)), ("dimensions", ParamValue::Choice(1))]),
effect("sharpen", &[("amount", fl(50.0))]),
],
vec![
effect("crop", &[("left", fl(10.0)), ("bottom", fl(20.0)), ("feather", fl(4.0))]),
effect("transform", &[("rotation", fl(-15.0)), ("scale_height", fl(110.0))]),
effect("horizontal_flip", &[]),
effect("levels", &[("in_black", fl(15.0)), ("gamma", fl(80.0))]),
effect("offset", &[("shift", pt(70.0, 10.0))]),
effect("invert", &[("channel", ParamValue::Choice(4))]),
effect("directional_blur", &[("length", fl(6.0)), ("direction", fl(90.0))]),
],
];
for chain in chains {
let ops: Vec<FxOp> = chain.iter().filter_map(|e| FxOp::eval(e, &cx(Tick::ZERO, 1.0), w as usize, h as usize)).collect();
assert!(ops.iter().all(FxOp::gpu_ok));
let mut cpu = filmcraft_render::Image { w: w as usize, h: h as usize, px: px.clone() };
for op in &ops {
op.apply(&mut cpu);
}
let names: Vec<&str> = chain.iter().map(|e| e.effect.as_str()).collect();
let (_, _, gpu) = c.effect_image(&frame, &LayerFx { size: (w, h), decimation: 1, ops }).expect("effect image");
let (worst, flips) = compare(&cpu.px, &gpu);
eprintln!("{names:?}: max rel diff {worst:.2e}, {:.3}% flipped", flips * 100.0);
assert!(worst < EXACT * 2.0 && flips == 0.0, "{names:?}: {worst}, {flips}");
}
}
/// Keyframed parameters are evaluated at the layer's time on the CPU: the GPU result follows them.
#[test]
fn keyframed_effects_follow_time() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let (w, h) = (48u32, 32u32);
let px = picture(w, h);
let frame = VideoFrame::rgba_f32(w, h, px.clone());
let mut blur = effect("gaussian_blur", &[]);
let mut bc = effect("brightness_contrast", &[]);
let p = blur.params.get_mut("blurriness").expect("blurriness");
p.put_keyframe(Tick::ZERO, fl(0.0));
p.put_keyframe(Tick::from_seconds_f64(1.0), fl(30.0));
let p = bc.params.get_mut("brightness").expect("brightness");
p.put_keyframe(Tick::ZERO, fl(-50.0));
p.put_keyframe(Tick::from_seconds_f64(1.0), fl(60.0));
let mut last: Option<Vec<f32>> = None;
for s in [0.0, 0.3, 0.75, 1.0] {
let cx = cx(Tick::from_seconds_f64(s), 1.0);
let ops: Vec<FxOp> = [&bc, &blur].iter().filter_map(|e| FxOp::eval(e, &cx, w as usize, h as usize)).collect();
let mut cpu = filmcraft_render::Image { w: w as usize, h: h as usize, px: px.clone() };
for op in &ops {
op.apply(&mut cpu);
}
let (_, _, gpu) = c.effect_image(&frame, &LayerFx { size: (w, h), decimation: 1, ops }).expect("effect image");
let (worst, flips) = compare(&cpu.px, &gpu);
assert!(worst < EXACT && flips == 0.0, "t = {s}: {worst}, {flips}");
assert_ne!(last.as_ref(), Some(&gpu), "t = {s}: the keyframes change the picture");
last = Some(gpu);
}
}
/// Hostile values: non-finite parameters stay on the CPU (`gpu_ok` is false), huge radii and
/// sizes are capped as on the CPU and run in bounded time on the GPU.
#[test]
fn hostile_parameters_are_bounded() {
let (w, h) = (40usize, 24usize);
let nan = [
effect("brightness_contrast", &[("brightness", fl(f64::NAN))]),
effect("gaussian_blur", &[("blurriness", fl(f64::INFINITY))]),
effect("directional_blur", &[("length", fl(f64::NAN))]),
effect("transform", &[("rotation", fl(f64::NAN))]),
effect("crop", &[("left", fl(f64::NAN))]),
effect("offset", &[("shift", pt(f64::INFINITY, 0.0))]),
effect("levels", &[("in_black", fl(f64::NEG_INFINITY))]),
effect("transform", &[("scale_height", fl(1e300))]),
// minifying resamples (the CPU's mip path) stay on the CPU
effect("transform", &[("scale_height", fl(20.0))]),
effect("crop", &[("left", fl(-300.0)), ("zoom", ParamValue::Bool(true))]),
];
for e in &nan {
let op = FxOp::eval(e, &cx(Tick::ZERO, 1.0), w, h);
// gaussian with an infinite radius is capped (no blur), which the GPU runs
if e.effect == "gaussian_blur" {
assert!(op.as_ref().is_some_and(FxOp::gpu_ok));
continue;
}
assert!(op.as_ref().is_some_and(|o| !o.gpu_ok()), "{}: {op:?}", e.effect);
// and the CPU reference copes
let mut img = filmcraft_render::Image::filled(w, h, [0.3, 0.2, 0.1, 1.0]);
if let Some(op) = op {
op.apply(&mut img);
}
}
// huge radii: box radii capped by the image size
let op = FxOp::eval(&effect("gaussian_blur", &[("blurriness", fl(1e30))]), &cx(Tick::ZERO, 1.0), w, h);
let Some(FxOp::Gaussian { rx, ry, .. }) = op else { panic!("{op:?}") };
assert!(rx.iter().chain(&ry).all(|r| *r as usize <= 4 * w.max(h).max(8)), "{rx:?} {ry:?}");
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let frame = VideoFrame::rgba_f32(w as u32, h as u32, picture(w as u32, h as u32));
let ops = vec![
FxOp::eval(&effect("gaussian_blur", &[("blurriness", fl(1e30))]), &cx(Tick::ZERO, 1.0), w, h).expect("op"),
FxOp::eval(&effect("unsharp_mask", &[("radius", fl(1e30)), ("amount", fl(1e30))]), &cx(Tick::ZERO, 1.0), w, h).expect("op"),
FxOp::eval(&effect("directional_blur", &[("length", fl(1e30))]), &cx(Tick::ZERO, 1.0), w, h).expect("op"),
];
let t0 = std::time::Instant::now();
let (_, _, gpu) = c.effect_image(&frame, &LayerFx { size: (w as u32, h as u32), decimation: 1, ops }).expect("effect image");
assert!(t0.elapsed().as_secs() < 30);
assert_eq!(gpu.len(), w * h * 4);
// a working image larger than any texture is refused, not attempted
assert!(c.effect_image(&frame, &LayerFx { size: (1 << 20, 4), decimation: 1, ops: vec![FxOp::BlackWhite] }).is_none());
}
/// Layers with effect chains in a composited plan — YUV and RGBA sources, a decimated working
/// image, rotated / scaled placement, partial opacity and alpha, blend modes — match the CPU plan
/// executor.
#[test]
fn gpu_effect_layers_match_cpu_plan() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let (w, h) = (320usize, 180usize);
let ops = |list: &[EffectInstance], lw: u32, lh: u32, px_scale: f32| -> Vec<FxOp> {
list.iter().filter_map(|e| FxOp::eval(e, &cx(Tick::ZERO, px_scale), lw as usize, lh as usize)).collect()
};
let chain_a = [
effect("brightness_contrast", &[("brightness", fl(15.0)), ("contrast", fl(25.0))]),
effect("gaussian_blur", &[("blurriness", fl(6.0))]),
effect("tint", &[("amount", fl(40.0))]),
];
let chain_b = [
effect("crop", &[("left", fl(8.0)), ("right", fl(12.0)), ("feather", fl(6.0))]),
effect("unsharp_mask", &[("amount", fl(120.0)), ("radius", fl(2.0))]),
effect("color_balance", &[("hi_b", fl(60.0)), ("shadow_r", fl(40.0))]),
];
let chain_c = [effect("mirror", &[("angle", fl(90.0))]), effect("gaussian_blur", &[("blurriness", fl(50.0)), ("dimensions", ParamValue::Choice(1))])];
let ramp = ramp_layer(160, 120, 0);
for mode in [Blend::Normal, Blend::Screen, Blend::Multiply, Blend::Overlay] {
let plan = FramePlan::Layers {
width: w,
height: h,
layers: vec![
// YUV source decoded at half size (decimation 2) with effects, filling the output
PlanLayer {
frame: yuv_frame(640, 360),
matrix: Affine::IDENTITY,
opacity: 1.0,
blend: Blend::Normal,
fx: Some(Arc::new(LayerFx { size: (320, 180), decimation: 2, ops: ops(&chain_c, 320, 180, 0.5) })),
},
PlanLayer {
frame: ramp.clone(),
matrix: Affine::motion(Vec2::new(170.0, 95.0), Vec2::new(1.1, 1.1), 17.0, Vec2::new(80.0, 60.0)),
opacity: 0.8,
blend: mode,
fx: Some(Arc::new(LayerFx { size: (160, 120), decimation: 1, ops: ops(&chain_a, 160, 120, 1.0) })),
},
PlanLayer::new(ramp.clone(), Affine::translate(10.0, 20.0), 0.5, Blend::Normal),
PlanLayer {
frame: yuv_frame(320, 180),
matrix: Affine::motion(Vec2::new(250.0, 70.0), Vec2::new(0.45, 0.45), -9.0, Vec2::new(160.0, 90.0)),
opacity: 0.7,
blend: mode,
fx: Some(Arc::new(LayerFx { size: (320, 180), decimation: 1, ops: ops(&chain_b, 320, 180, 1.0) })),
},
],
};
let cpu = execute_cpu(&plan).over_black_rgba8();
c.composite(&plan);
let (_, _, gpu) = c.read_output().expect("readback");
let keep = interior(&plan);
let (p99, mean, max) = stats8(&cpu, &gpu, &keep);
let (ap99, amean, _) = stats8(&cpu, &gpu, &vec![true; keep.len()]);
eprintln!("{mode:?}: interior 8-bit p99 {p99} mean {mean:.3} max {max}; with edges p99 {ap99} mean {amean:.3}");
assert!(p99 <= 6 && mean < 1.5, "{mode:?}: p99 {p99}, mean {mean}");
}
}
/// Plans without effects never create working textures.
#[test]
fn plain_layers_skip_the_effect_stage() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let mut c = GpuCompositor::new(&dev, &q);
let plan = FramePlan::Layers { width: 64, height: 36, layers: vec![PlanLayer::new(yuv_frame(64, 36), Affine::IDENTITY, 1.0, Blend::Normal)] };
c.composite(&plan);
assert!(c.fx.as_ref().is_some_and(fx::FxStage::is_idle));
let fx = LayerFx { size: (64, 36), decimation: 1, ops: vec![FxOp::BlackWhite] };
let with = FramePlan::Layers {
width: 64,
height: 36,
layers: vec![PlanLayer { fx: Some(Arc::new(fx)), ..PlanLayer::new(yuv_frame(64, 36), Affine::IDENTITY, 1.0, Blend::Normal) }],
};
c.composite(&with);
assert!(!c.fx.as_ref().is_some_and(fx::FxStage::is_idle));
// and they are released once no layer needs them
c.composite(&plan);
assert!(c.fx.as_ref().is_some_and(fx::FxStage::is_idle));
}
/// Without an effect stage (devices without compute shaders) the compositor renders a layer's
/// effects on the CPU itself: the picture is the same.
#[test]
fn layers_with_effects_fall_back_to_the_cpu_without_a_stage() {
let Some((dev, q)) = device() else {
eprintln!("no GPU adapter; skipping");
return;
};
let ops: Vec<FxOp> = [effect("tint", &[("amount", fl(70.0))]), effect("gaussian_blur", &[("blurriness", fl(10.0))])]
.iter()
.filter_map(|e| FxOp::eval(e, &cx(Tick::ZERO, 1.0), 160, 120))
.collect();
let plan = FramePlan::Layers {
width: 200,
height: 150,
layers: vec![
PlanLayer::new(yuv_frame(200, 150), Affine::IDENTITY, 1.0, Blend::Normal),
PlanLayer {
fx: Some(Arc::new(LayerFx { size: (160, 120), decimation: 1, ops })),
..PlanLayer::new(
ramp_layer(160, 120, 2),
Affine::motion(Vec2::new(100.0, 75.0), Vec2::new(0.9, 0.9), 8.0, Vec2::new(80.0, 60.0)),
0.8,
Blend::Screen,
)
},
],
};
let cpu = execute_cpu(&plan).over_black_rgba8();
for stage in [true, false] {
let mut c = GpuCompositor::new(&dev, &q);
if !stage {
c.fx = None;
}
c.composite(&plan);
let (_, _, gpu) = c.read_output().expect("readback");
let (p99, mean, _) = stats8(&cpu, &gpu, &interior(&plan));
eprintln!("effect stage {stage}: interior p99 {p99}, mean {mean:.3}");
assert!(p99 <= 6 && mean < 1.5, "stage {stage}: p99 {p99}, mean {mean}");
}
}
+351 -53
View File
@@ -7,6 +7,19 @@
//! registers as a native texture. Uploads are cached by pixel-buffer identity, so a paused frame or
//! a still costs nothing.
//!
//! Every blend mode of [`filmcraft_render::Blend`] runs here. Normal and Dissolve (whose per-pixel
//! pattern is the CPU's 64-bit hash, reproduced in WGSL) use fixed-function "over" blending. The
//! other 25 modes need the colour under the layer: before such a layer is drawn, the accumulator
//! region under its quad is copied into a backdrop texture, and the fragment shader composites
//! with the CPU reference's math (`filmcraft_render::blend::composite`: sRGB-encoded straight
//! colour, W3C formulas, back to linear) and writes the result. Only those layers pay the copy.
//!
//! A layer carrying standard effects ([`filmcraft_render::plan::LayerFx`]) goes through the
//! effect stage ([`fx`]) first: its source is drawn into an `Rgba32Float` working image, the
//! effects run as compute passes in WGSL with the CPU reference's math
//! (`filmcraft_render::gpufx::FxOp::apply`), and the result is drawn like any RGBA layer. Layers
//! without effects take the single draw above.
//!
//! Layers that need converting before upload (linear f32 RGBA from CPU-rendered layers, 16-bit
//! YUV such as ProRes) are converted to half floats by [`prepare`], which frame workers run off the
//! UI thread; [`GpuCompositor::composite_prepared`] then only copies bytes into textures.
@@ -22,8 +35,10 @@ use rayon::prelude::*;
use filmcraft_color::{Matrix, Range, Transfer};
use filmcraft_frame::{Chroma, PixelData, VideoFrame};
use filmcraft_render::Blend;
use filmcraft_render::plan::{FramePlan, PlanLayer};
pub mod fx;
pub mod lut;
pub mod mask;
pub use lut::GpuLut;
@@ -49,14 +64,22 @@ pub struct GpuCompositor {
device: wgpu::Device,
queue: wgpu::Queue,
layer_pipeline: wgpu::RenderPipeline,
/// Blend modes that read the destination: `fs_blend`, no fixed-function blending.
blend_pipeline: wgpu::RenderPipeline,
final_pipeline: wgpu::RenderPipeline,
layer_bgl: wgpu::BindGroupLayout,
blend_bgl: wgpu::BindGroupLayout,
final_bgl: wgpu::BindGroupLayout,
accum: Option<(wgpu::Texture, wgpu::TextureView, (u32, u32))>,
/// Copy of the accumulator under a blend-mode layer (allocated on first use).
backdrop: Option<(wgpu::Texture, wgpu::TextureView, (u32, u32))>,
output: Option<(wgpu::Texture, wgpu::TextureView, (u32, u32))>,
uploads: HashMap<(usize, u32, u32), Uploaded>,
clock: u64,
dummy: wgpu::TextureView,
/// The effect stage (None on devices without compute shaders or storage textures: layers
/// with effects are then rendered on the CPU here).
fx: Option<fx::FxStage>,
/// Total bytes uploaded (stats).
pub uploaded_bytes: u64,
}
@@ -203,6 +226,21 @@ impl GpuCompositor {
tex_entry(3),
],
});
let blend_bgl = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("blend-layer"),
entries: &[
wgpu::BindGroupLayoutEntry {
binding: 0,
visibility: wgpu::ShaderStages::VERTEX_FRAGMENT,
ty: wgpu::BindingType::Buffer { ty: wgpu::BufferBindingType::Uniform, has_dynamic_offset: false, min_binding_size: None },
count: None,
},
tex_entry(1),
tex_entry(2),
tex_entry(3),
tex_entry(4),
],
});
let final_bgl = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
label: Some("final"),
entries: &[wgpu::BindGroupLayoutEntry { binding: 0, ..tex_entry(0) }],
@@ -229,6 +267,27 @@ impl GpuCompositor {
multiview_mask: None,
cache: None,
});
let bpl = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
label: Some("blend-layer"),
bind_group_layouts: &[Some(&blend_bgl)],
immediate_size: 0,
});
let blend_pipeline = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
label: Some("blend-layer"),
layout: Some(&bpl),
vertex: wgpu::VertexState { module: &shader, entry_point: Some("vs"), compilation_options: Default::default(), buffers: &[] },
primitive: wgpu::PrimitiveState::default(),
depth_stencil: None,
multisample: wgpu::MultisampleState::default(),
fragment: Some(wgpu::FragmentState {
module: &shader,
entry_point: Some("fs_blend"),
compilation_options: Default::default(),
targets: &[Some(wgpu::ColorTargetState { format: ACCUM_FORMAT, blend: None, write_mask: wgpu::ColorWrites::ALL })],
}),
multiview_mask: None,
cache: None,
});
let fpl =
device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor { label: Some("final"), bind_group_layouts: &[Some(&final_bgl)], immediate_size: 0 });
let final_pipeline = device.create_render_pipeline(&wgpu::RenderPipelineDescriptor {
@@ -258,18 +317,23 @@ impl GpuCompositor {
view_formats: &[],
});
let dummy = dummy_tex.create_view(&Default::default());
let fx = fx::FxStage::supported(device).then(|| fx::FxStage::new(device, &shader, &layer_bgl));
Self {
device: device.clone(),
queue: queue.clone(),
layer_pipeline,
blend_pipeline,
final_pipeline,
layer_bgl,
blend_bgl,
final_bgl,
accum: None,
backdrop: None,
output: None,
uploads: HashMap::new(),
clock: 0,
dummy,
fx,
uploaded_bytes: 0,
}
}
@@ -394,18 +458,21 @@ impl GpuCompositor {
key
}
fn uniforms(&self, l: &PlanLayer, key: (usize, u32, u32), out: (u32, u32)) -> [f32; 24] {
let up = &self.uploads[&key];
let f = &l.frame;
let m = &l.matrix;
let (kr, kb) = f.color.matrix.kr_kb();
let bits_scale = up.code_scale / 255.0; // code units relative to 8-bit
let (yo, ys, co, cs) = match (f.color.range, up.kind) {
/// The source description of an uploaded frame.
fn src_info(&self, f: &VideoFrame, key: (usize, u32, u32)) -> Option<SrcInfo> {
let up = self.uploads.get(&key)?;
Some(SrcInfo { kind: up.kind, code_scale: up.code_scale, chroma: up.chroma, size: (f.width, f.height), color: f.color })
}
fn uniforms(src: &SrcInfo, m: &filmcraft_geom::Affine, opacity: f32, blend: Blend, out: (u32, u32)) -> [f32; 28] {
let (kr, kb) = src.color.matrix.kr_kb();
let bits_scale = src.code_scale / 255.0; // code units relative to 8-bit
let (yo, ys, co, cs) = match (src.color.range, src.kind) {
(Range::Limited, 2) => (16.0 * bits_scale, 219.0 * bits_scale, 128.0 * bits_scale, 224.0 * bits_scale),
(Range::Full, 2) => (0.0, up.code_scale - 1.0, up.code_scale / 2.0, up.code_scale - 1.0),
(Range::Full, 2) => (0.0, src.code_scale - 1.0, src.code_scale / 2.0, src.code_scale - 1.0),
_ => (0.0, 1.0, 0.0, 1.0),
};
let transfer = match f.color.transfer {
let transfer = match src.color.transfer {
Transfer::Linear => 1.0,
Transfer::Pq => 2.0,
Transfer::Hlg => 3.0,
@@ -426,12 +493,12 @@ impl GpuCompositor {
m.f as f32,
out.0 as f32,
out.1 as f32,
f.width as f32,
f.height as f32,
up.chroma.0 as f32,
up.chroma.1 as f32,
l.opacity,
up.kind as f32,
src.size.0 as f32,
src.size.1 as f32,
src.chroma.0 as f32,
src.chroma.1 as f32,
opacity,
src.kind as f32,
taps,
transfer,
yo,
@@ -440,11 +507,27 @@ impl GpuCompositor {
cs,
kr,
kb,
up.code_scale,
src.code_scale,
footprint.max(1.0),
blend.index() as f32,
0.0,
0.0,
0.0,
]
}
fn uniform_buffer(&self, u: &[f32; 28]) -> wgpu::Buffer {
let bytes: Vec<u8> = u.iter().flat_map(|v| v.to_le_bytes()).collect();
let buf = self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("layer-u"),
size: bytes.len() as u64,
usage: wgpu::BufferUsages::UNIFORM | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
});
self.queue.write_buffer(&buf, 0, &bytes);
buf
}
/// Composite a plan; returns the output view (sRGB, over black) and its size.
pub fn composite(&mut self, plan: &FramePlan) -> (wgpu::TextureView, (u32, u32)) {
self.composite_prepared(plan, None)
@@ -467,44 +550,125 @@ impl GpuCompositor {
},
None => VideoFrame::rgba_f32(img.w as u32, img.h as u32, img.px.clone()),
};
owned = [PlanLayer { frame: Arc::new(frame), matrix: filmcraft_geom::Affine::IDENTITY, opacity: 1.0 }];
owned = [PlanLayer { frame: Arc::new(frame), matrix: filmcraft_geom::Affine::IDENTITY, opacity: 1.0, blend: Blend::Normal, fx: None }];
(img.w as u32, img.h as u32, &owned[..])
}
};
let (w, h) = (w.max(1), h.max(1));
let accum_view = Self::target(&self.device, &mut self.accum, w, h, ACCUM_FORMAT, wgpu::TextureUsages::empty());
let accum_view = Self::target(&self.device, &mut self.accum, w, h, ACCUM_FORMAT, wgpu::TextureUsages::COPY_SRC);
let backdrop_view = if layers.iter().any(|l| l.blend.reads_destination()) {
Some(Self::target(&self.device, &mut self.backdrop, w, h, ACCUM_FORMAT, wgpu::TextureUsages::COPY_DST))
} else {
None
};
let out_view = Self::target(&self.device, &mut self.output, w, h, OUTPUT_FORMAT, wgpu::TextureUsages::COPY_SRC);
// Layers whose effects the GPU stage can't run (no stage on this device, or a working image
// larger than a texture) are rendered on the CPU here and drawn as plain RGBA layers.
let max_side = self.device.limits().max_texture_dimension_2d;
let cpu_fx: Vec<Option<PlanLayer>> = layers
.iter()
.map(|l| {
let lfx = l.fx.as_ref()?;
if self.fx.is_some() && lfx.size.0.max(lfx.size.1) <= max_side {
return None;
}
let img = filmcraft_render::plan::effect_image(&l.frame, lfx);
Some(PlanLayer::new(Arc::new(VideoFrame::rgba_f32(img.w as u32, img.h as u32, img.px)), l.matrix, l.opacity, l.blend))
})
.collect();
let resolved: Vec<&PlanLayer> = layers.iter().zip(&cpu_fx).map(|(l, c)| c.as_ref().unwrap_or(l)).collect();
let layers = resolved;
let keys: Vec<(usize, u32, u32)> = layers
.iter()
.enumerate()
.map(|(i, l)| {
let p = prep.and_then(|p| if matches!(plan, FramePlan::Image(_)) { p.image.as_ref() } else { p.layers.get(i).and_then(Option::as_ref) });
// (texels converted beforehand belong to the plan's own frame)
let p = if cpu_fx.get(i).is_some_and(Option::is_some) { None } else { p };
self.upload(&l.frame, p)
})
.collect();
// (bind group, None for a fixed-function layer or Some(region of the accumulator to copy
// into the backdrop) for a layer that reads the destination; Some(None): off the output)
let mut bind_groups = Vec::with_capacity(layers.len());
// the effect stage of each layer with effects (recorded right before its draw)
let mut jobs: Vec<Option<fx::FxJob>> = Vec::with_capacity(layers.len());
if let Some(f) = self.fx.as_mut() {
f.begin_frame();
}
for (l, k) in layers.iter().zip(&keys) {
let u = self.uniforms(l, *k, (w, h));
let bytes: Vec<u8> = u.iter().flat_map(|v| v.to_le_bytes()).collect();
let buf = self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("layer-u"),
size: bytes.len() as u64,
usage: wgpu::BufferUsages::UNIFORM | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
// (every frame was just uploaded; a missing upload draws transparent texels)
let src = self.src_info(&l.frame, *k).unwrap_or(SrcInfo {
kind: 1,
code_scale: 1.0,
chroma: (1, 1),
size: (l.frame.width, l.frame.height),
color: l.frame.color,
});
self.queue.write_buffer(&buf, 0, &bytes);
let up = &self.uploads[k];
let bg = self.device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("layer"),
layout: &self.layer_bgl,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: buf.as_entire_binding() },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&up.views[0]) },
wgpu::BindGroupEntry { binding: 2, resource: wgpu::BindingResource::TextureView(&up.views[1]) },
wgpu::BindGroupEntry { binding: 3, resource: wgpu::BindingResource::TextureView(&up.views[2]) },
],
});
bind_groups.push(bg);
let views = self.uploads.get(k).map_or_else(|| [self.dummy.clone(), self.dummy.clone(), self.dummy.clone()], |up| up.views.clone());
let job = match &l.fx {
Some(lfx) => {
// the source drawn onto the working image (box-decimated by n, as the CPU decodes)
let n = lfx.decimation.max(1) as f64;
let su = Self::uniforms(&src, &filmcraft_geom::Affine::scale(1.0 / n, 1.0 / n), 1.0, Blend::Normal, (lfx.size.0.max(1), lfx.size.1.max(1)));
let sbuf = self.uniform_buffer(&su);
let sbg = self.device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("fx-source"),
layout: &self.layer_bgl,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: sbuf.as_entire_binding() },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&views[0]) },
wgpu::BindGroupEntry { binding: 2, resource: wgpu::BindingResource::TextureView(&views[1]) },
wgpu::BindGroupEntry { binding: 3, resource: wgpu::BindingResource::TextureView(&views[2]) },
],
});
match self.fx.as_mut() {
Some(f) => f.job(&self.device, &self.queue, lfx, sbg, &self.dummy),
None => None,
}
}
None => None,
};
// what the layer draws: the effect result (linear premultiplied RGBA) or the frame
let (u, tex) = match &job {
Some(j) => {
let s = SrcInfo { kind: 1, code_scale: 1.0, chroma: l.size(), size: l.size(), color: l.frame.color };
(Self::uniforms(&s, &l.matrix, l.opacity, l.blend, (w, h)), [j.result.clone(), self.dummy.clone(), self.dummy.clone()])
}
None => {
// (a layer with effects only gets here if its job failed, which the CPU
// fallback above rules out; it is then drawn without them, where the working
// image would be)
let m = match &l.fx {
Some(lfx) => {
let n = 1.0 / lfx.decimation.max(1) as f64;
l.matrix.then_apply(&filmcraft_geom::Affine::scale(n, n))
}
None => l.matrix,
};
(Self::uniforms(&src, &m, l.opacity, l.blend, (w, h)), views)
}
};
jobs.push(job);
let buf = self.uniform_buffer(&u);
let mut entries = vec![
wgpu::BindGroupEntry { binding: 0, resource: buf.as_entire_binding() },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&tex[0]) },
wgpu::BindGroupEntry { binding: 2, resource: wgpu::BindingResource::TextureView(&tex[1]) },
wgpu::BindGroupEntry { binding: 3, resource: wgpu::BindingResource::TextureView(&tex[2]) },
];
let (layout, region) = match (&backdrop_view, l.blend.reads_destination()) {
(Some(b), true) => {
entries.push(wgpu::BindGroupEntry { binding: 4, resource: wgpu::BindingResource::TextureView(b) });
(&self.blend_bgl, Some(quad_bounds(l, w, h)))
}
_ => (&self.layer_bgl, None),
};
let bg = self.device.create_bind_group(&wgpu::BindGroupDescriptor { label: Some("layer"), layout, entries: &entries });
bind_groups.push((bg, region));
}
if let Some(f) = self.fx.as_mut() {
f.end_frame();
}
let final_bg = self.device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("final"),
@@ -512,25 +676,50 @@ impl GpuCompositor {
entries: &[wgpu::BindGroupEntry { binding: 0, resource: wgpu::BindingResource::TextureView(&accum_view) }],
});
let mut enc = self.device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("filmcraft-composite") });
{
let mut pass = enc.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("layers"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view: &accum_view,
depth_slice: None,
resolve_target: None,
ops: wgpu::Operations { load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT), store: wgpu::StoreOp::Store },
})],
depth_stencil_attachment: None,
timestamp_writes: None,
occlusion_query_set: None,
multiview_mask: None,
});
// Runs of fixed-function layers draw in one pass. A layer that reads the destination
// ends it, copies the accumulator under its quad into the backdrop and draws on its own.
let accum_tex = self.accum.as_ref().map(|a| a.0.clone());
let backdrop_tex = self.backdrop.as_ref().map(|b| b.0.clone());
let mut cleared = false;
let mut i = 0;
while i < bind_groups.len() || !cleared {
// a layer's effects run right before it is drawn (pooled working textures are reused
// by the next layer with effects)
if let (Some(Some(job)), Some(f)) = (jobs.get(i), self.fx.as_ref()) {
f.record(&mut enc, job);
}
if let Some((bg, Some(region))) = bind_groups.get(i) {
i += 1;
let (Some(rect), Some(src), Some(dst)) = (region, &accum_tex, &backdrop_tex) else { continue };
if !cleared {
accum_pass(&mut enc, &accum_view, wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT));
cleared = true;
}
let origin = wgpu::Origin3d { x: rect.0, y: rect.1, z: 0 };
enc.copy_texture_to_texture(
wgpu::TexelCopyTextureInfo { texture: src, mip_level: 0, origin, aspect: wgpu::TextureAspect::All },
wgpu::TexelCopyTextureInfo { texture: dst, mip_level: 0, origin, aspect: wgpu::TextureAspect::All },
wgpu::Extent3d { width: rect.2, height: rect.3, depth_or_array_layers: 1 },
);
let mut pass = accum_pass(&mut enc, &accum_view, wgpu::LoadOp::Load);
pass.set_pipeline(&self.blend_pipeline);
pass.set_bind_group(0, bg, &[]);
pass.draw(0..6, 0..1);
continue;
}
// a run of fixed-function layers, up to the next one that reads the destination or
// has effects to run first
let breaks = |j: &usize| bind_groups.get(*j).is_some_and(|(_, r)| r.is_some()) || jobs.get(*j).is_some_and(Option::is_some);
let end = (i + 1..bind_groups.len()).find(breaks).unwrap_or(bind_groups.len()).max(i);
let load = if cleared { wgpu::LoadOp::Load } else { wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT) };
let mut pass = accum_pass(&mut enc, &accum_view, load);
cleared = true;
pass.set_pipeline(&self.layer_pipeline);
for bg in &bind_groups {
for (bg, _) in bind_groups.get(i..end).unwrap_or_default() {
pass.set_bind_group(0, bg, &[]);
pass.draw(0..6, 0..1);
}
i = end;
}
{
let mut pass = enc.begin_render_pass(&wgpu::RenderPassDescriptor {
@@ -554,6 +743,64 @@ impl GpuCompositor {
(out_view, (w, h))
}
/// Run the effect stage of a layer alone: `frame` decoded into its working image and `fx`
/// applied, read back as linear premultiplied RGBA f32 (what `LayerFx` ops produce on the
/// CPU from the same frame). None when the working image does not fit in a texture.
pub fn effect_image(&mut self, frame: &VideoFrame, fx: &filmcraft_render::plan::LayerFx) -> Option<(u32, u32, Vec<f32>)> {
let key = self.upload(frame, None);
let src = self.src_info(frame, key)?;
let views = self.uploads.get(&key)?.views.clone();
let n = fx.decimation.max(1) as f64;
let (w, h) = (fx.size.0.max(1), fx.size.1.max(1));
let su = Self::uniforms(&src, &filmcraft_geom::Affine::scale(1.0 / n, 1.0 / n), 1.0, Blend::Normal, (w, h));
let sbuf = self.uniform_buffer(&su);
let sbg = self.device.create_bind_group(&wgpu::BindGroupDescriptor {
label: Some("fx-source"),
layout: &self.layer_bgl,
entries: &[
wgpu::BindGroupEntry { binding: 0, resource: sbuf.as_entire_binding() },
wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&views[0]) },
wgpu::BindGroupEntry { binding: 2, resource: wgpu::BindingResource::TextureView(&views[1]) },
wgpu::BindGroupEntry { binding: 3, resource: wgpu::BindingResource::TextureView(&views[2]) },
],
});
let stage = self.fx.as_mut()?;
stage.begin_frame();
let job = stage.job(&self.device, &self.queue, fx, sbg, &self.dummy)?;
let row = (w * 16).div_ceil(256) * 256;
let buf = self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("fx-readback"),
size: (row * h) as u64,
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
let mut enc = self.device.create_command_encoder(&Default::default());
stage.record(&mut enc, &job);
enc.copy_texture_to_buffer(
wgpu::TexelCopyTextureInfo { texture: &job.result_texture, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All },
wgpu::TexelCopyBufferInfo { buffer: &buf, layout: wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(row), rows_per_image: Some(h) } },
wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 },
);
self.queue.submit([enc.finish()]);
stage.end_frame();
let slice = buf.slice(..);
let (tx, rx) = std::sync::mpsc::channel();
slice.map_async(wgpu::MapMode::Read, move |r| {
let _ = tx.send(r);
});
let _ = self.device.poll(wgpu::PollType::wait_indefinitely());
rx.recv().ok()?.ok()?;
let data = slice.get_mapped_range().ok()?;
let mut out = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
let r = data.get((y * row) as usize..(y * row + w * 16) as usize)?;
out.extend(r.as_chunks::<4>().0.iter().map(|b| f32::from_le_bytes(*b)));
}
drop(data);
buf.unmap();
Some((w, h, out))
}
/// Read the output back as RGBA8 (tests / screenshots / thumbnails).
pub fn read_output(&self) -> Option<(u32, u32, Vec<u8>)> {
let (tex, _, (w, h)) = self.output.as_ref()?;
@@ -589,5 +836,56 @@ impl GpuCompositor {
}
}
/// What a layer's shader samples: the texture kind (0 RGBA8 sRGB, 1 linear premultiplied RGBA
/// float, 2 YUV planes), code scale, chroma plane size, picture size and colour description.
struct SrcInfo {
kind: u32,
code_scale: f32,
chroma: (u32, u32),
size: (u32, u32),
color: filmcraft_color::ColorInfo,
}
/// A render pass drawing into the accumulator.
fn accum_pass<'e>(enc: &'e mut wgpu::CommandEncoder, view: &wgpu::TextureView, load: wgpu::LoadOp<wgpu::Color>) -> wgpu::RenderPass<'e> {
enc.begin_render_pass(&wgpu::RenderPassDescriptor {
label: Some("layers"),
color_attachments: &[Some(wgpu::RenderPassColorAttachment {
view,
depth_slice: None,
resolve_target: None,
ops: wgpu::Operations { load, store: wgpu::StoreOp::Store },
})],
depth_stencil_attachment: None,
timestamp_writes: None,
occlusion_query_set: None,
multiview_mask: None,
})
}
/// The output pixels a layer's quad can touch, as (x, y, width, height) clamped to the `w`×`h`
/// output with a one-pixel margin; None when it touches none. A non-finite matrix covers it all.
fn quad_bounds(l: &PlanLayer, w: u32, h: u32) -> Option<(u32, u32, u32, u32)> {
let m = &l.matrix;
let (fw, fh) = (l.size().0 as f64, l.size().1 as f64);
let pts = [(0.0, 0.0), (fw, 0.0), (0.0, fh), (fw, fh)].map(|(x, y)| (m.a * x + m.c * y + m.e, m.b * x + m.d * y + m.f));
if pts.iter().any(|p| !p.0.is_finite() || !p.1.is_finite()) {
return Some((0, 0, w, h));
}
let (mut x0, mut y0, mut x1, mut y1) = (f64::INFINITY, f64::INFINITY, f64::NEG_INFINITY, f64::NEG_INFINITY);
for (x, y) in pts {
(x0, y0, x1, y1) = (x0.min(x), y0.min(y), x1.max(x), y1.max(y));
}
// in range before the cast (`as` saturates anyway; NaN is excluded above)
let px = |v: f64, hi: u32| v.max(0.0).min(hi as f64) as u32;
let (x0, y0) = (px(x0.floor() - 1.0, w), px(y0.floor() - 1.0, h));
let (x1, y1) = (px(x1.ceil() + 1.0, w), px(y1.ceil() + 1.0, h));
(x1 > x0 && y1 > y0).then(|| (x0, y0, x1 - x0, y1 - y0))
}
#[cfg(test)]
mod blend_tests;
#[cfg(test)]
mod fx_tests;
#[cfg(test)]
mod tests;
+13 -5
View File
@@ -1,5 +1,6 @@
use super::*;
use filmcraft_geom::{Affine, Vec2};
use filmcraft_render::Blend;
use filmcraft_render::plan::execute_cpu;
fn device() -> Option<(wgpu::Device, wgpu::Queue)> {
@@ -44,8 +45,14 @@ fn gpu_matches_cpu_plan() {
width: w,
height: h,
layers: vec![
PlanLayer { frame: yuv_frame(640, 360), matrix: Affine::scale(0.5, 0.5), opacity: 1.0 },
PlanLayer { frame: rgba, matrix: Affine::motion(Vec2::new(200.0, 100.0), Vec2::new(0.4, 0.4), 12.0, Vec2::new(160.0, 90.0)), opacity: 0.7 },
PlanLayer { frame: yuv_frame(640, 360), matrix: Affine::scale(0.5, 0.5), opacity: 1.0, blend: Blend::Normal, fx: None },
PlanLayer {
frame: rgba,
matrix: Affine::motion(Vec2::new(200.0, 100.0), Vec2::new(0.4, 0.4), 12.0, Vec2::new(160.0, 90.0)),
opacity: 0.7,
blend: Blend::Normal,
fx: None,
},
],
};
let cpu = execute_cpu(&plan).over_black_rgba8();
@@ -108,8 +115,8 @@ fn prepared_upload_matches_inline_conversion() {
width: w as usize,
height: h as usize,
layers: vec![
PlanLayer { frame: yuv16, matrix: Affine::IDENTITY, opacity: 1.0 },
PlanLayer { frame: rgbaf, matrix: Affine::scale(0.5, 0.5), opacity: 0.8 },
PlanLayer { frame: yuv16, matrix: Affine::IDENTITY, opacity: 1.0, blend: Blend::Normal, fx: None },
PlanLayer { frame: rgbaf, matrix: Affine::scale(0.5, 0.5), opacity: 0.8, blend: Blend::Normal, fx: None },
],
};
let image = FramePlan::Image(filmcraft_render::Image { w: w as usize, h: h as usize, px: f32_layer });
@@ -138,7 +145,8 @@ fn upload_cache_keeps_buffers_alive() {
let mut c = GpuCompositor::new(&dev, &q);
let px = Arc::new(vec![200u8; 16 * 8 * 4]);
let frame = Arc::new(VideoFrame { width: 16, height: 8, data: PixelData::Rgba8(px.clone()), ..(*yuv_frame(16, 8)).clone() });
let plan = FramePlan::Layers { width: 16, height: 8, layers: vec![PlanLayer { frame, matrix: Affine::IDENTITY, opacity: 1.0 }] };
let plan =
FramePlan::Layers { width: 16, height: 8, layers: vec![PlanLayer { frame, matrix: Affine::IDENTITY, opacity: 1.0, blend: Blend::Normal, fx: None }] };
c.composite(&plan);
drop(plan);
assert!(Arc::strong_count(&px) > 1, "cached upload must own its pixel buffer");
+40
View File
@@ -0,0 +1,40 @@
[package]
name = "filmcraft-platform"
description = "OS media integration: hardware video decoding through the operating system's codecs (VideoToolbox on macOS), with transparent fallback to the pure-Rust decoders"
version.workspace = true
edition.workspace = true
license.workspace = true
rust-version.workspace = true
# The one crate allowed to contain `unsafe` (OS media FFI; AGENTS.md §0.3, docs/adr/0001-platform-ffi.md).
# The workspace lints, except that `unsafe_code` is `deny` instead of `forbid`: only the FFI modules
# opt in with `#[allow(unsafe_code)]`, and every `unsafe` block carries a `// SAFETY:` comment.
[lints.rust]
unsafe_code = "deny"
[lints.clippy]
dbg_macro = "warn"
todo = "warn"
too_many_arguments = "allow"
needless_range_loop = "allow"
type_complexity = "allow"
undocumented_unsafe_blocks = "deny"
[dependencies]
filmcraft-codecs = { workspace = true }
filmcraft-isobmff = { workspace = true }
log = { workspace = true }
[target.'cfg(target_os = "macos")'.dependencies]
filmcraft-color = { workspace = true }
filmcraft-frame = { workspace = true }
filmcraft-time = { workspace = true }
objc2-core-foundation = "0.3.2"
objc2-core-media = "0.3.2"
objc2-core-video = "0.3.2"
objc2-video-toolbox = "0.3.2"
[dev-dependencies]
filmcraft-testkit = { workspace = true }
filmcraft-media = { workspace = true }
filmcraft-frame = { workspace = true }
+69
View File
@@ -0,0 +1,69 @@
# filmcraft-platform
OS media integration for FilmCraft (layer L5): hardware video decoding through the operating
system's codecs, behind `filmcraft_codecs::VideoDecoder`. It holds OS media FFI and nothing else.
It is the one crate of the workspace allowed to contain `unsafe`, under the rules of
[ADR 0001](../../docs/adr/0001-platform-ffi.md) and [AGENTS.md](../../AGENTS.md) §0.3.
```rust
// at startup (the desktop app, filmcraft-cli, the bench)
let availability = filmcraft_platform::register(); // Available("VideoToolbox") on macOS
```
## What it does
- **macOS: VideoToolbox H.264 (`avcC`) and HEVC (`hvcC`)**, 8- and 10-bit, 4:2:0 and 4:2:2
(`videotoolbox.rs`). The session is created from the sample entry's parameter sets with a
hardware decoder *required*; samples go in as `CMSampleBuffer`s with asynchronous decompression
(two access units in flight); the output callback copies each NV12 / P010-style biplanar
`CVPixelBuffer` into planar `Yuv8` / `Yuv16` (chroma deinterleaved, 10-bit samples shifted down
from the high bits, cropped to the conformance window when the buffer is the coded size). A
reorder buffer of the stream's own depth (`max_num_reorder_frames` /
`sps_max_num_reorder_pics`) restores presentation order; a run starting at an HEVC CRA leaves
out its RASL pictures, as our decoder does. Each seek (`reset`) starts a fresh session.
- **Other systems:** `register()` does nothing and returns `Availability::Unavailable`.
- **`HybridDecoder`** (`hybrid.rs`, safe code): the hardware decoder plus the means to build our
software decoder for the same `SampleEntry` (`filmcraft_codecs::software_video_decoder`). On a
mid-stream failure (decode error, invalidated session, changed in-band parameter sets) it replays
the samples since the last restart point (IDR / IRAP; for an HEVC CRA the one before, so its RASL
pictures decode) through the software decoder, drops pictures already returned, keeps the ones
the hardware had decoded but not returned, and stays in software for that instance. The replay
log is bounded (600 samples / 256 MB); beyond it one error is returned and the next seek restarts
in software. Streams our decoders cannot decode (HEVC 4:2:2) have no fallback: the error stands.
## Guarantees
- **Never undecodable:** the factory declines (returns `None`, so the software decoder is used)
when Settings ▸ Playback ▸ Hardware decoding is Off, for formats it does not take (field-coded
H.264, bit depths other than 8 / 10, 4:4:4 or monochrome, luma / chroma depth mismatch, larger
than 8192×8192) and when VideoToolbox cannot create a hardware session.
- **Interchangeable:** colour, pixel aspect, pts, presentation order, `is_random_access` and
`is_disposable` come from the software decoders' own helpers (`filmcraft_codecs::hw`,
`video::vui_color`, `sar_par`).
- **Never crash:** no `unwrap` / `expect` / `panic!` outside tests; the output callback runs under
`catch_unwind`; every `unsafe` block has a `// SAFETY:` comment; the public API is safe.
- **Counted:** `perf.stats` `decode.hardware` (frames, software frames, sessions, declined,
fallbacks; `filmcraft_codecs::hw::hw_stats`).
## Tests
| test | what |
|---|---|
| `tests/videotoolbox.rs` (macOS) | H.264 High, HEVC Main (open GOP: CRA + RASL) and HEVC Main 10, 640×360 (coded 368: cropping) with B-frames: every picture **bit-exact** with our software decoder, same pts order, count, colour and aspect, also after `reset` + reseek to every later sync sample, mid-stream `flush`, and a full pass after resets; forced mid-stream failures (`VtDecoder::fail_after`) at five points continue with the software decoder's exact output; seeded mutation of samples and parameter sets (bit flips, truncation, corrupt length prefixes) never panics or hangs; HEVC 4:2:2 10-bit is bit-exact with ffmpeg's decode |
| `tests/fallback.rs` (every OS) | `HybridDecoder` with a stand-in hardware decoder failing after N samples (every sync sample ± a few, first / last sample, after a seek): output identical to the software decoder; in-band parameter sets identical to the sample entry's stay in hardware, different ones switch to software |
| `tests/setting.rs` | Hardware decoding Off gives the software decoder through `make_video_decoder` and the media stack (no hardware frames); Auto gives VideoToolbox where available |
Fixtures are made with ffmpeg into `target/fixtures/platform/` (generator only, never linked);
tests skip without ffmpeg or without a hardware decoder.
## Performance
M4 Pro, load 150–190 (`cargo xtask bench --hw off|auto`): CPU per decoded frame H.264 2160p
119 → 4.2 ms, HEVC 2160p 86 → 3.7 ms; decode 35 → 107 fps and 49 → 217 fps; 4K H.264 and HEVC
playback with no dropped frames at Full, 1/2 and 1/4. Details in
[docs/performance.md](../../docs/performance.md).
## Not yet
Zero-copy upload of `CVPixelBuffer`s into wgpu textures; hardware encoding; Media Foundation /
D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264.
+235
View File
@@ -0,0 +1,235 @@
//! [`HybridDecoder`]: a hardware decoder that turns into the software decoder for the same stream
//! when it fails mid-stream (decode error, session lost to a GPU change or sleep, an in-band
//! parameter-set change it was not set up for), without the caller noticing.
//!
//! The hybrid keeps the samples fed since the last point decoding can restart from (the run's
//! first sample, then every IDR / IRAP access unit). On a failure it builds the software decoder
//! ([`filmcraft_codecs::software_video_decoder`], skipping registered factories), replays those
//! samples through it and carries on, dropping pictures the hardware already returned and keeping
//! the ones it had decoded but not yet returned, so the output is the software decoder's own
//! sequence. That replay log is bounded; past the bound a failure is reported as an error once
//! and the instance continues in software from the next seek ([`VideoDecoder::reset`]).
use std::collections::{BTreeMap, BTreeSet};
use filmcraft_codecs::hw::NalStreamInfo;
use filmcraft_codecs::{CodecError, DecodedFrame, Result, VideoDecoder};
use filmcraft_isobmff::SampleEntry;
/// Replay log bounds: samples and bytes since the last restart point.
const LOG_MAX_SAMPLES: usize = 600;
const LOG_MAX_BYTES: usize = 256 << 20;
/// Presentation times remembered as returned (older ones are forgotten first).
const EMITTED_MAX: usize = 4096;
/// A hardware decoder with a transparent software fallback.
pub struct HybridDecoder {
hw: Option<Box<dyn VideoDecoder>>,
sw: Option<Box<dyn VideoDecoder>>,
entry: SampleEntry,
info: NalStreamInfo,
draft: bool,
/// Samples (and pts) since the last restart point; `None` once the bound was passed.
log: Option<Vec<(Vec<u8>, i64)>>,
log_bytes: usize,
/// Index in the log of the most recent restart point.
last_irap: usize,
/// Pictures returned since the last reset (by pts).
emitted: BTreeSet<i64>,
/// Pictures the hardware had decoded but not returned when it failed.
carry: BTreeMap<i64, DecodedFrame>,
}
impl HybridDecoder {
/// Wrap `hw`, a decoder for `entry` (whose stream is described by `info`).
pub fn new(hw: Box<dyn VideoDecoder>, entry: SampleEntry, info: NalStreamInfo) -> Self {
filmcraft_codecs::hw::note_hw_session();
Self {
hw: Some(hw),
sw: None,
entry,
info,
draft: false,
log: Some(Vec::new()),
log_bytes: 0,
last_irap: 0,
emitted: BTreeSet::new(),
carry: BTreeMap::new(),
}
}
/// Whether the hardware decoder is still in use (false after a fallback).
pub fn is_hardware(&self) -> bool {
self.hw.is_some()
}
fn remember(&mut self, sample: &[u8], pts: i64) {
if self.info.is_irap(sample) {
// An HEVC CRA keeps the previous restart point: replaying from the CRA itself would
// treat it as a first picture and drop its RASL pictures, which the hardware decodes.
let cra = self.info.codec == filmcraft_codecs::hw::NalCodec::Hevc && self.info.nal_types(sample).contains(&21);
match self.log.as_mut() {
Some(log) if cra => {
log.drain(..self.last_irap.min(log.len()));
self.log_bytes = log.iter().map(|s| s.0.len()).sum();
}
_ => {
self.log = Some(Vec::new());
self.log_bytes = 0;
}
}
self.last_irap = self.log.as_ref().map_or(0, Vec::len);
}
let Some(log) = self.log.as_mut() else { return };
self.log_bytes = self.log_bytes.saturating_add(sample.len());
if log.len() >= LOG_MAX_SAMPLES || self.log_bytes > LOG_MAX_BYTES {
self.log = None;
return;
}
log.push((sample.to_vec(), pts));
}
fn note_emitted(&mut self, out: &[DecodedFrame]) {
for f in out {
self.emitted.insert(f.pts);
}
while self.emitted.len() > EMITTED_MAX {
self.emitted.pop_first();
}
}
/// In-band parameter sets that differ from the sample entry's (the hardware session was set
/// up from those).
fn parameter_sets_changed(&self, sample: &[u8]) -> bool {
self.info
.nals(sample)
.into_iter()
.any(|n| n.first().is_some_and(|&h| self.info.is_parameter_set(self.info.nal_type(h))) && !self.info.parameter_sets.iter().any(|p| p == n))
}
/// Software output after a fallback: pictures already returned are dropped, carried hardware
/// pictures are merged in presentation order (a software picture replaces the carried one with
/// the same pts).
fn merge(&mut self, frames: Vec<DecodedFrame>) -> Vec<DecodedFrame> {
let mut out = Vec::with_capacity(frames.len());
for f in frames {
if self.emitted.contains(&f.pts) {
self.carry.remove(&f.pts);
continue;
}
while let Some(e) = self.carry.first_entry() {
if *e.key() >= f.pts {
break;
}
out.push(e.remove());
}
self.carry.remove(&f.pts);
out.push(f);
}
self.note_emitted(&out);
out
}
/// Switch to the software decoder after `err`, replaying the log (which ends with the sample
/// that failed).
fn fall_back(&mut self, err: CodecError) -> Result<Vec<DecodedFrame>> {
let Some(mut hw) = self.hw.take() else { return Err(err) };
log::warn!("{} failed ({err}); continuing with the software decoder", hw.name());
filmcraft_codecs::hw::note_hw_fallback();
for f in hw.flush() {
if !self.emitted.contains(&f.pts) {
self.carry.insert(f.pts, f);
}
}
drop(hw);
let mut sw = filmcraft_codecs::software_video_decoder(&self.entry).map_err(|e| CodecError::Decode(format!("{err}; no software decoder: {e}")))?;
sw.set_draft(self.draft);
self.sw = Some(sw);
let Some(log) = self.log.take() else {
// Too far from a restart point to replay: the next seek restarts in software.
self.carry.clear();
return Err(CodecError::Decode(format!("{err} (continuing in software after the next seek)")));
};
let mut frames = Vec::new();
for (s, p) in &log {
let Some(sw) = self.sw.as_mut() else { break };
frames.extend(sw.decode(s, *p)?);
}
self.log = Some(log);
Ok(self.merge(frames))
}
}
impl VideoDecoder for HybridDecoder {
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
if let Some(sw) = self.sw.as_mut() {
let frames = sw.decode(sample, pts)?;
return Ok(self.merge(frames));
}
self.remember(sample, pts);
if self.parameter_sets_changed(sample) {
return self.fall_back(CodecError::Decode("in-band parameter sets differ from the sample entry".into()));
}
let Some(hw) = self.hw.as_mut() else {
return Err(CodecError::Decode("no decoder".into()));
};
match hw.decode(sample, pts) {
Ok(out) => {
filmcraft_codecs::hw::note_hw_frames(out.len());
self.note_emitted(&out);
Ok(out)
}
Err(e) => self.fall_back(e),
}
}
fn flush(&mut self) -> Vec<DecodedFrame> {
if let Some(hw) = self.hw.as_mut() {
let out = hw.flush();
filmcraft_codecs::hw::note_hw_frames(out.len());
self.note_emitted(&out);
return out;
}
let frames = self.sw.as_mut().map(|d| d.flush()).unwrap_or_default();
let mut out = self.merge(frames);
out.extend(std::mem::take(&mut self.carry).into_values());
out
}
fn reset(&mut self) {
if let Some(hw) = self.hw.as_mut() {
hw.reset();
}
if let Some(sw) = self.sw.as_mut() {
sw.reset();
}
self.log = Some(Vec::new());
self.log_bytes = 0;
self.last_irap = 0;
self.emitted.clear();
self.carry.clear();
}
fn name(&self) -> &str {
match (&self.hw, &self.sw) {
(Some(hw), _) => hw.name(),
(None, Some(sw)) => sw.name(),
(None, None) => "hardware decoder",
}
}
fn is_random_access(&self, sample: &[u8]) -> Option<bool> {
self.info.is_random_access(sample)
}
fn is_disposable(&self, sample: &[u8]) -> bool {
self.info.is_disposable(sample)
}
fn set_draft(&mut self, on: bool) {
self.draft = on;
if let Some(sw) = self.sw.as_mut() {
sw.set_draft(on);
}
}
}
+86
View File
@@ -0,0 +1,86 @@
//! OS media integration (layer L5): hardware video decoding through the operating system's codecs.
//!
//! [`register`] puts the platform's hardware decoder factory in front of FilmCraft's own decoders
//! (`filmcraft_codecs::register_video_decoder`). Today that is VideoToolbox on macOS for H.264
//! (`avcC`) and HEVC (`hvcC`) streams, 8- and 10-bit, 4:2:0 and 4:2:2; on other systems
//! registration does nothing and reports [`Availability::Unavailable`].
//!
//! Hardware decoding never makes a file undecodable:
//!
//! - the factory declines (falls through to the software decoder) when Settings ▸ Playback ▸
//! Hardware decoding is Off (`filmcraft_codecs::hw::set_hardware_decoding`), when the stream's
//! format is one the hardware path does not take, or when the OS cannot create a hardware
//! session for it (profile, size, no hardware decoder);
//! - a decoder that fails mid-stream switches to the software decoder transparently
//! ([`HybridDecoder`]) and logs it.
//!
//! Pictures are the software decoder's: same planes (bit-exact on the parity fixtures), colour,
//! pixel aspect, pts and presentation order, so the two are interchangeable.
//!
//! This is the one crate allowed to use `unsafe` (OS FFI), and only in its FFI modules
//! (docs/adr/0001-platform-ffi.md).
#![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable))]
pub mod hybrid;
#[cfg(target_os = "macos")]
#[allow(unsafe_code)]
pub mod videotoolbox;
pub use hybrid::HybridDecoder;
/// What [`register`] made available.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Availability {
/// A hardware decoder factory was registered (its name).
Available(&'static str),
/// Nothing to register on this system (why).
Unavailable(&'static str),
}
/// Register the platform's hardware video decoders (call once at startup; repeated calls are
/// harmless). Streams they do not take, and every stream while hardware decoding is Off, keep
/// using FilmCraft's own decoders.
pub fn register() -> Availability {
#[cfg(target_os = "macos")]
{
filmcraft_codecs::register_video_decoder(videotoolbox_factory);
Availability::Available("VideoToolbox")
}
#[cfg(not(target_os = "macos"))]
{
Availability::Unavailable("no hardware video decoder for this system yet")
}
}
/// Whether this system's hardware decoder takes the stream of `entry` (whatever the Hardware
/// decoding setting says): diagnostics and tests.
pub fn hardware_decoder_for(entry: &filmcraft_isobmff::SampleEntry) -> bool {
#[cfg(target_os = "macos")]
{
filmcraft_codecs::hw::NalStreamInfo::from_entry(entry).and_then(|r| r.ok()).is_some_and(|info| videotoolbox::VtDecoder::new(info).is_ok())
}
#[cfg(not(target_os = "macos"))]
{
let _ = entry;
false
}
}
/// The VideoToolbox factory: a [`HybridDecoder`] around [`videotoolbox::VtDecoder`] for `avcC` /
/// `hvcC` streams VideoToolbox can decode in hardware, `None` otherwise.
#[cfg(target_os = "macos")]
pub fn videotoolbox_factory(entry: &filmcraft_isobmff::SampleEntry) -> Option<filmcraft_codecs::Result<Box<dyn filmcraft_codecs::VideoDecoder>>> {
if !filmcraft_codecs::hw::hardware_decoding() {
return None;
}
let info = filmcraft_codecs::hw::NalStreamInfo::from_entry(entry)?.ok()?;
match videotoolbox::VtDecoder::new(info.clone()) {
Ok(vt) => Some(Ok(Box::new(HybridDecoder::new(Box::new(vt), entry.clone(), info)))),
Err(why) => {
log::info!("hardware decoding declined for {} video: {why}", entry.codec.name());
filmcraft_codecs::hw::note_hw_declined();
None
}
}
}
+571
View File
@@ -0,0 +1,571 @@
//! VideoToolbox (macOS) hardware H.264 / HEVC decoding.
//!
//! The only module of the workspace with `unsafe` code besides the other FFI modules of this crate
//! (docs/adr/0001-platform-ffi.md). Rules: every `unsafe` block has a `// SAFETY:` comment; no
//! panic may unwind into VideoToolbox (the output callback runs under `catch_unwind`); every
//! failure is a `Result` the caller ([`crate::hybrid::HybridDecoder`]) answers by switching to the
//! software decoder.
//!
//! A session is created from the parameter sets of the sample entry (`avcC` / `hvcC`) with
//! hardware decoding required, so streams the hardware cannot take are declined up front. Samples
//! are fed as `CMSampleBuffer`s with asynchronous decompression (a couple in flight); the output
//! callback (on a VideoToolbox thread) copies each decoded
//! `CVPixelBuffer` (NV12 / P010-style biplanar) into planar [`PixelData::Yuv8`] /
//! [`PixelData::Yuv16`]. VideoToolbox emits pictures in decoding order, so a reorder buffer of the
//! stream's own depth (`max_num_reorder_frames` / `sps_max_num_reorder_pics`) puts them in
//! presentation order, as the software decoders do.
use std::ffi::c_void;
use std::panic::AssertUnwindSafe;
use std::ptr::NonNull;
use std::sync::{Arc, Mutex, PoisonError};
use filmcraft_codecs::hw::{NalCodec, NalStreamInfo};
use filmcraft_codecs::{CodecError, DecodedFrame, Result, VideoDecoder};
use filmcraft_frame::{Chroma, PixelData, VideoFrame};
use objc2_core_foundation::{CFBoolean, CFDictionary, CFNumber, CFRetained, CFString, CFType};
use objc2_core_media::{
CMBlockBuffer, CMFormatDescription, CMSampleBuffer, CMSampleTimingInfo, CMTime, CMTimeFlags, CMVideoFormatDescriptionCreateFromH264ParameterSets,
CMVideoFormatDescriptionCreateFromHEVCParameterSets, kCMBlockBufferAssureMemoryNowFlag,
};
use objc2_core_video::{
CVImageBuffer, CVPixelBufferGetBaseAddressOfPlane, CVPixelBufferGetBytesPerRowOfPlane, CVPixelBufferGetHeight, CVPixelBufferGetHeightOfPlane,
CVPixelBufferGetPixelFormatType, CVPixelBufferGetPlaneCount, CVPixelBufferGetWidth, CVPixelBufferGetWidthOfPlane, CVPixelBufferLockBaseAddress,
CVPixelBufferLockFlags, CVPixelBufferUnlockBaseAddress, kCVPixelBufferPixelFormatTypeKey,
};
use objc2_video_toolbox::{
VTDecodeFrameFlags, VTDecodeInfoFlags, VTDecompressionOutputCallbackRecord, VTDecompressionSession,
kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder,
};
/// `CVPixelBuffer` formats we ask VideoToolbox for: biplanar Y + interleaved CbCr, 8-bit or
/// 10-bit in the high bits of 16-bit words.
const NV12_VIDEO: u32 = u32::from_be_bytes(*b"420v");
const NV12_FULL: u32 = u32::from_be_bytes(*b"420f");
const P010_VIDEO: u32 = u32::from_be_bytes(*b"x420");
const P010_FULL: u32 = u32::from_be_bytes(*b"xf20");
const NV16_VIDEO: u32 = u32::from_be_bytes(*b"422v");
const NV16_FULL: u32 = u32::from_be_bytes(*b"422f");
const P210_VIDEO: u32 = u32::from_be_bytes(*b"x422");
const P210_FULL: u32 = u32::from_be_bytes(*b"xf22");
/// How decoded buffers are laid out and what each output picture carries.
#[derive(Clone, Debug)]
struct Layout {
pixel_format: u32,
/// Coded size and output rectangle (luma samples).
coded: (u32, u32),
crop: (u32, u32, u32, u32),
chroma: Chroma,
/// Significant bits per sample (8 = `Yuv8`).
bits: u32,
color: filmcraft_color::ColorInfo,
par: (u32, u32),
}
/// One output-callback result.
enum Output {
Frame(DecodedFrame),
Failed(String),
}
/// State shared with the output callback (its `decompressionOutputRefCon`). Boxed by the decoder
/// and kept alive until the session is invalidated.
struct Shared {
layout: Layout,
out: Mutex<Vec<Output>>,
}
impl Shared {
fn push(&self, o: Output) {
self.out.lock().unwrap_or_else(PoisonError::into_inner).push(o);
}
}
/// A VideoToolbox session with its callback state. Dropping it invalidates the session before the
/// callback state is freed.
struct Session {
session: CFRetained<VTDecompressionSession>,
format: CFRetained<CMFormatDescription>,
/// Pointed to by the session's callback record: must outlive `session`.
shared: Box<Shared>,
}
// SAFETY: a VTDecompressionSession and the CoreMedia objects it uses may be used from any thread
// (Apple: "thread safe"); the decoder owning a `Session` is used by one thread at a time (`&mut
// self`), and the callback state is behind a `Mutex`.
unsafe impl Send for Session {}
impl Drop for Session {
fn drop(&mut self) {
// SAFETY: the session is valid until released; invalidating it guarantees no callback runs
// afterwards, so `shared` can be freed when this function returns.
unsafe {
self.session.wait_for_asynchronous_frames();
self.session.invalidate();
}
}
}
/// The output callback. Never unwinds: panics are caught and reported as a failed frame.
unsafe extern "C-unwind" fn output_callback(
refcon: *mut c_void,
frame_refcon: *mut c_void,
status: i32,
_flags: VTDecodeInfoFlags,
image: *mut CVImageBuffer,
_pts: CMTime,
_duration: CMTime,
) {
if refcon.is_null() {
return;
}
// SAFETY: `refcon` is the `Shared` the session was created with; it lives in a `Box` owned by
// the `Session`, which invalidates the session (no more callbacks) before freeing it.
let shared = unsafe { &*(refcon as *const Shared) };
let pts = frame_refcon as isize as i64;
let r = std::panic::catch_unwind(AssertUnwindSafe(|| {
if status != 0 {
return Output::Failed(format!("VideoToolbox decode error {status}"));
}
let Some(image) = NonNull::new(image) else {
return Output::Failed("VideoToolbox dropped a picture".into());
};
// SAFETY: VideoToolbox passes a valid image buffer that stays alive for the duration of
// the callback; we only borrow it here.
let pb = unsafe { image.as_ref() };
match copy_out(pb, &shared.layout) {
Ok(frame) => Output::Frame(DecodedFrame { pts, frame, draft: false }),
Err(e) => Output::Failed(e),
}
}));
shared.push(r.unwrap_or_else(|_| Output::Failed("panic while reading a decoded picture".into())));
}
/// Unlocks a pixel buffer's base address when dropped.
struct Locked<'a>(&'a CVImageBuffer);
impl Drop for Locked<'_> {
fn drop(&mut self) {
// SAFETY: `copy_out` locked the buffer with the same flags, and it is still alive (borrowed).
unsafe { CVPixelBufferUnlockBaseAddress(self.0, CVPixelBufferLockFlags::ReadOnly) };
}
}
/// One plane of a locked pixel buffer.
struct PlaneView<'a> {
data: &'a [u8],
stride: usize,
width: usize,
height: usize,
}
impl<'a> PlaneView<'a> {
/// `len` bytes of row `y` from byte `x`.
fn row(&self, y: usize, x: usize, len: usize) -> std::result::Result<&'a [u8], String> {
let start = y * self.stride + x;
self.data.get(start..start + len).ok_or_else(|| "plane row out of range".to_string())
}
}
/// The returned view borrows the lock guard, so it cannot outlive the locked base address.
fn plane<'l>(pb: &CVImageBuffer, _lock: &'l Locked<'_>, i: usize) -> std::result::Result<PlaneView<'l>, String> {
let base = CVPixelBufferGetBaseAddressOfPlane(pb, i) as *const u8;
let stride = CVPixelBufferGetBytesPerRowOfPlane(pb, i);
let (width, height) = (CVPixelBufferGetWidthOfPlane(pb, i), CVPixelBufferGetHeightOfPlane(pb, i));
if base.is_null() || stride == 0 || height == 0 {
return Err(format!("decoded picture plane {i} is not mapped"));
}
let len = stride.checked_mul(height).ok_or("plane size overflows")?;
// SAFETY: the buffer's base address is locked while `_lock` lives, and the returned slice
// borrows `_lock` ('l), so it cannot be used after the unlock; CoreVideo maps
// `bytes_per_row * height` bytes from each plane's base address.
let data = unsafe { std::slice::from_raw_parts(base, len) };
Ok(PlaneView { data, stride, width, height })
}
/// Copy a decoded biplanar buffer into a planar frame (deinterleaving chroma, cropping to the
/// output rectangle, shifting 10-bit samples down from the high bits).
fn copy_out(pb: &CVImageBuffer, l: &Layout) -> std::result::Result<VideoFrame, String> {
let fmt = CVPixelBufferGetPixelFormatType(pb);
if fmt != l.pixel_format || CVPixelBufferGetPlaneCount(pb) != 2 {
return Err(format!("unexpected decoded pixel format {fmt:#x}"));
}
let (bw, bh) = (CVPixelBufferGetWidth(pb) as u32, CVPixelBufferGetHeight(pb) as u32);
let (cx, cy, w, h) = l.crop;
// VideoToolbox outputs the cropped picture; a buffer of the coded size is cropped here.
let (ox, oy) = if (bw, bh) == (w, h) {
(0, 0)
} else if (bw, bh) == l.coded && cx + w <= bw && cy + h <= bh {
(cx, cy)
} else {
return Err(format!("decoded picture is {bw}x{bh}, expected {w}x{h}"));
};
// SAFETY: `pb` is a valid pixel buffer for the duration of this call (see the callback).
if unsafe { CVPixelBufferLockBaseAddress(pb, CVPixelBufferLockFlags::ReadOnly) } != 0 {
return Err("cannot lock the decoded picture".into());
}
let lock = Locked(pb);
let (ys, cs) = (plane(pb, &lock, 0)?, plane(pb, &lock, 1)?);
let (w, h, ox, oy) = (w as usize, h as usize, ox as usize, oy as usize);
let sub_y = if l.chroma == Chroma::C420 { 2 } else { 1 };
let (cw, ch) = (w.div_ceil(2), h.div_ceil(sub_y));
let (cox, coy) = (ox / 2, oy / sub_y);
let bps = if l.bits > 8 { 2 } else { 1 };
if ys.width < ox + w || ys.height < oy + h || ys.stride < ys.width * bps || cs.width < cox + cw || cs.height < coy + ch || cs.stride < cs.width * 2 * bps {
return Err("decoded picture planes are smaller than the picture".into());
}
let data = if bps == 1 {
let mut yp = Vec::with_capacity(w * h);
for y in 0..h {
yp.extend_from_slice(ys.row(oy + y, ox * bps, w * bps)?);
}
let (mut u, mut v) = (Vec::with_capacity(cw * ch), Vec::with_capacity(cw * ch));
for y in 0..ch {
for pair in cs.row(coy + y, cox * 2 * bps, cw * 2 * bps)?.as_chunks::<2>().0 {
u.push(pair[0]);
v.push(pair[1]);
}
}
PixelData::Yuv8 { planes: [Arc::new(yp), Arc::new(u), Arc::new(v)], chroma: l.chroma, alpha: None }
} else {
let shift = 16 - l.bits;
let mut yp = Vec::with_capacity(w * h);
for y in 0..h {
yp.extend(ys.row(oy + y, ox * bps, w * bps)?.as_chunks::<2>().0.iter().map(|b| u16::from_ne_bytes([b[0], b[1]]) >> shift));
}
let (mut u, mut v) = (Vec::with_capacity(cw * ch), Vec::with_capacity(cw * ch));
for y in 0..ch {
for q in cs.row(coy + y, cox * 2 * bps, cw * 2 * bps)?.as_chunks::<4>().0 {
u.push(u16::from_ne_bytes([q[0], q[1]]) >> shift);
v.push(u16::from_ne_bytes([q[2], q[3]]) >> shift);
}
}
PixelData::Yuv16 { planes: [Arc::new(yp), Arc::new(u), Arc::new(v)], chroma: l.chroma, bits: l.bits, alpha: None }
};
drop(lock);
Ok(VideoFrame { width: w as u32, height: h as u32, data, color: l.color, par: l.par, pts: filmcraft_time::Tick::ZERO })
}
/// The layout for a stream, or why VideoToolbox is not used for it.
fn layout(info: &NalStreamInfo) -> std::result::Result<Layout, String> {
if info.interlaced {
return Err("field-coded H.264".into());
}
if info.bit_depth_luma != info.bit_depth_chroma || !matches!(info.bit_depth_luma, 8 | 10) {
return Err(format!("{}-bit luma / {}-bit chroma", info.bit_depth_luma, info.bit_depth_chroma));
}
let full = info.color.range == filmcraft_color::Range::Full;
let ten = info.bit_depth_luma == 10;
let (chroma, pixel_format) = match (info.chroma_format_idc, ten, full) {
(1, false, false) => (Chroma::C420, NV12_VIDEO),
(1, false, true) => (Chroma::C420, NV12_FULL),
(1, true, false) => (Chroma::C420, P010_VIDEO),
(1, true, true) => (Chroma::C420, P010_FULL),
(2, false, false) => (Chroma::C422, NV16_VIDEO),
(2, false, true) => (Chroma::C422, NV16_FULL),
(2, true, false) => (Chroma::C422, P210_VIDEO),
(2, true, true) => (Chroma::C422, P210_FULL),
(c, ..) => return Err(format!("chroma_format_idc {c}")),
};
let (cx, cy, w, h) = info.crop;
if w == 0 || h == 0 || w > 8192 || h > 8192 || cx.saturating_add(w) > info.coded.0 || cy.saturating_add(h) > info.coded.1 {
return Err(format!("picture size {w}x{h}"));
}
Ok(Layout { pixel_format, coded: info.coded, crop: info.crop, chroma, bits: info.bit_depth_luma, color: info.color, par: info.par })
}
impl Session {
fn new(info: &NalStreamInfo, layout: Layout) -> std::result::Result<Session, String> {
let format = format_description(info)?;
// SAFETY: reading an immutable framework constant.
let key: &CFString = unsafe { kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder };
let spec = CFDictionary::<CFString, CFType>::from_slices(&[key], &[CFBoolean::new(true).as_ref()]);
let pf = CFNumber::new_i32(layout.pixel_format as i32);
// SAFETY: reading an immutable framework constant.
let pf_key: &CFString = unsafe { kCVPixelBufferPixelFormatTypeKey };
let attrs = CFDictionary::<CFString, CFType>::from_slices(&[pf_key], &[pf.as_ref()]);
let shared = Box::new(Shared { layout, out: Mutex::new(Vec::new()) });
let record = VTDecompressionOutputCallbackRecord {
decompressionOutputCallback: Some(output_callback),
decompressionOutputRefCon: &*shared as *const Shared as *mut c_void,
};
let mut out: *mut VTDecompressionSession = std::ptr::null_mut();
// SAFETY: every pointer is valid for the call (VideoToolbox copies the callback record);
// the refcon points into `shared`, which the returned `Session` keeps alive until the
// session is invalidated.
let status = unsafe { VTDecompressionSession::create(None, &format, Some(spec.as_ref()), Some(attrs.as_ref()), &record, NonNull::from(&mut out)) };
let session = NonNull::new(out).filter(|_| status == 0).ok_or_else(|| format!("VTDecompressionSessionCreate failed ({status})"))?;
// SAFETY: a created session is returned retained (+1); `CFRetained` takes that reference over.
let session = unsafe { CFRetained::from_raw(session) };
Ok(Session { session, format, shared })
}
/// Submit one length-prefixed access unit (asynchronous decompression: its picture arrives
/// through the callback, possibly after this returns).
fn submit(&self, sample: &[u8], pts: i64) -> std::result::Result<(), String> {
let buffer = sample_buffer(sample, pts, &self.format)?;
let mut info = VTDecodeInfoFlags(0);
// SAFETY: the session and sample buffer are valid (VideoToolbox retains the sample buffer
// for as long as it needs it); the frame refcon is not a pointer, it carries the pts.
let status =
unsafe { self.session.decode_frame(&buffer, VTDecodeFrameFlags::Frame_EnableAsynchronousDecompression, pts as isize as *mut c_void, &mut info) };
if status != 0 {
return Err(format!("VTDecompressionSessionDecodeFrame failed ({status})"));
}
Ok(())
}
/// Wait until every submitted picture has been emitted.
fn wait(&self) {
// SAFETY: the session is valid.
unsafe { self.session.wait_for_asynchronous_frames() };
}
/// What the callback produced so far.
fn take(&self) -> Vec<Output> {
std::mem::take(&mut *self.shared.out.lock().unwrap_or_else(PoisonError::into_inner))
}
}
/// Access units decoding at once (asynchronous decompression overlaps decoding a picture with
/// copying out the previous one).
const MAX_IN_FLIGHT: usize = 2;
/// A `CMVideoFormatDescription` from the record's parameter sets.
fn format_description(info: &NalStreamInfo) -> std::result::Result<CFRetained<CMFormatDescription>, String> {
let mut ptrs: Vec<NonNull<u8>> = Vec::with_capacity(info.parameter_sets.len());
let mut sizes: Vec<usize> = Vec::with_capacity(info.parameter_sets.len());
for p in &info.parameter_sets {
ptrs.push(NonNull::new(p.as_ptr() as *mut u8).ok_or("empty parameter set")?);
sizes.push(p.len());
}
let ptr_array = NonNull::new(ptrs.as_mut_ptr()).ok_or("no parameter sets")?;
let size_array = NonNull::new(sizes.as_mut_ptr()).ok_or("no parameter sets")?;
let mut out: *const CMFormatDescription = std::ptr::null();
let ls = info.length_size as i32;
// SAFETY: `ptrs` / `sizes` hold `len` valid (pointer, size) pairs into `info.parameter_sets`,
// which outlive the call; CoreMedia copies the parameter sets.
let status = unsafe {
match info.codec {
NalCodec::H264 => CMVideoFormatDescriptionCreateFromH264ParameterSets(None, ptrs.len(), ptr_array, size_array, ls, NonNull::from(&mut out)),
NalCodec::Hevc => CMVideoFormatDescriptionCreateFromHEVCParameterSets(None, ptrs.len(), ptr_array, size_array, ls, None, NonNull::from(&mut out)),
}
};
let fd = NonNull::new(out as *mut CMFormatDescription).filter(|_| status == 0).ok_or_else(|| format!("bad parameter sets for VideoToolbox ({status})"))?;
// SAFETY: a "Create" function returns the description retained (+1); `CFRetained` takes it over.
Ok(unsafe { CFRetained::from_raw(fd) })
}
/// A ready `CMSampleBuffer` holding a copy of `sample`.
fn sample_buffer(sample: &[u8], pts: i64, format: &CMFormatDescription) -> std::result::Result<CFRetained<CMSampleBuffer>, String> {
if sample.is_empty() {
return Err("empty sample".into());
}
let mut block: *mut CMBlockBuffer = std::ptr::null_mut();
// SAFETY: CoreMedia allocates (AssureMemoryNow) a block of `len` bytes with the default
// allocator; no external memory is referenced.
let status = unsafe {
CMBlockBuffer::create_with_memory_block(
None,
std::ptr::null_mut(),
sample.len(),
None,
std::ptr::null(),
0,
sample.len(),
kCMBlockBufferAssureMemoryNowFlag,
NonNull::from(&mut block),
)
};
let block = NonNull::new(block).filter(|_| status == 0).ok_or_else(|| format!("CMBlockBufferCreateWithMemoryBlock failed ({status})"))?;
// SAFETY: created retained (+1).
let block = unsafe { CFRetained::from_raw(block) };
let src = NonNull::new(sample.as_ptr() as *mut c_void).ok_or("empty sample")?;
// SAFETY: copies `len` bytes from `sample` into the block, which holds exactly `len` bytes.
let status = unsafe { CMBlockBuffer::replace_data_bytes(src, &block, 0, sample.len()) };
if status != 0 {
return Err(format!("CMBlockBufferReplaceDataBytes failed ({status})"));
}
let invalid = CMTime { value: 0, timescale: 0, flags: CMTimeFlags(0), epoch: 0 };
let timing = CMSampleTimingInfo {
duration: invalid,
presentationTimeStamp: CMTime { value: pts, timescale: 1, flags: CMTimeFlags::Valid, epoch: 0 },
decodeTimeStamp: invalid,
};
let size = sample.len();
let mut out: *mut CMSampleBuffer = std::ptr::null_mut();
// SAFETY: one sample with one timing entry and one size entry, all pointing at locals that
// outlive the call; the sample buffer retains the block buffer and format description.
let status = unsafe { CMSampleBuffer::create_ready(None, Some(&block), Some(format), 1, 1, &timing, 1, &size, NonNull::from(&mut out)) };
let out = NonNull::new(out).filter(|_| status == 0).ok_or_else(|| format!("CMSampleBufferCreateReady failed ({status})"))?;
// SAFETY: created retained (+1).
Ok(unsafe { CFRetained::from_raw(out) })
}
/// Whether VideoToolbox reports a hardware decoder for the codec at all.
pub fn hardware_available(codec: NalCodec) -> bool {
let t = match codec {
NalCodec::H264 => u32::from_be_bytes(*b"avc1"),
NalCodec::Hevc => u32::from_be_bytes(*b"hvc1"),
};
// SAFETY: a pure query with no pointer arguments.
unsafe { objc2_video_toolbox::VTIsHardwareDecodeSupported(t) }
}
/// The VideoToolbox decoder for one `avcC` / `hvcC` stream.
pub struct VtDecoder {
info: NalStreamInfo,
layout: Layout,
/// `None` after a reset until the next sample (a fresh session per decoding run).
session: Option<Session>,
/// Decoded pictures waiting for presentation order (sorted by pts).
pending: Vec<DecodedFrame>,
/// Presentation times of submitted access units whose picture has not arrived yet.
in_flight: Vec<i64>,
/// The run started at a CRA / BLA picture: its RASL pictures are not decodable and are left
/// out (as the software decoder does).
skip_rasl: bool,
first: bool,
/// Test hook: fail every decode once this many samples were fed.
fail_after: Option<u64>,
fed: u64,
name: &'static str,
}
impl VtDecoder {
/// A decoder for the stream, or why VideoToolbox does not take it (unsupported format, no
/// hardware decoder, session creation failed).
pub fn new(info: NalStreamInfo) -> std::result::Result<Self, String> {
let layout = layout(&info)?;
if !hardware_available(info.codec) {
return Err("no hardware decoder".into());
}
// Create the session now so unsupported streams are declined up front.
let session = Session::new(&info, layout.clone())?;
let name = match info.codec {
NalCodec::H264 => "VideoToolbox H.264",
NalCodec::Hevc => "VideoToolbox HEVC",
};
Ok(Self {
info,
layout,
session: Some(session),
pending: Vec::new(),
in_flight: Vec::new(),
skip_rasl: false,
first: true,
fail_after: None,
fed: 0,
name,
})
}
/// Test hook: from the `n`-th sample on, every decode fails as a lost session would.
pub fn fail_after(&mut self, n: u64) {
self.fail_after = Some(n);
}
fn insert(&mut self, f: DecodedFrame) {
let at = self.pending.partition_point(|p| p.pts <= f.pts);
self.pending.insert(at, f);
}
/// Pictures that are due: everything beyond the reorder depth, smallest pts first.
fn bump(&mut self) -> Vec<DecodedFrame> {
let mut out = Vec::new();
// A picture still decoding counts towards the depth and may come before the waiting ones.
while self.pending.len() + self.in_flight.len() > self.info.reorder {
let first = self.pending.first().map(|f| f.pts);
match (first, self.in_flight.iter().min()) {
(Some(p), Some(&q)) if q < p => break,
(Some(_), _) => out.push(self.pending.remove(0)),
(None, _) => break,
}
}
out
}
/// Move the callback's pictures into the reorder buffer.
fn collect(&mut self) -> std::result::Result<(), String> {
let Some(session) = self.session.as_ref() else { return Ok(()) };
for o in session.take() {
match o {
Output::Frame(f) => {
if let Some(i) = self.in_flight.iter().position(|&p| p == f.pts) {
self.in_flight.swap_remove(i);
}
self.insert(f);
}
Output::Failed(e) => return Err(e),
}
}
Ok(())
}
}
impl VideoDecoder for VtDecoder {
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
self.fed += 1;
if self.fail_after.is_some_and(|n| self.fed > n) {
return Err(CodecError::Decode("VideoToolbox session failed (test hook)".into()));
}
if self.info.codec == NalCodec::Hevc {
let types = self.info.nal_types(sample);
let irap = types.iter().any(|t| (16..=23).contains(t));
if irap {
// CRA / BLA starting a run: its RASL pictures reference pictures we never decoded.
self.skip_rasl = self.first && types.iter().any(|t| (16..=21).contains(t) && !(19..=20).contains(t));
} else if self.skip_rasl && types.iter().any(|t| matches!(t, 8 | 9)) {
self.first = false;
return Ok(Vec::new());
}
}
self.first = false;
if self.session.is_none() {
self.session = Some(Session::new(&self.info, self.layout.clone()).map_err(CodecError::Decode)?);
}
let Some(session) = self.session.as_ref() else {
return Err(CodecError::Decode("no VideoToolbox session".into()));
};
session.submit(sample, pts).map_err(CodecError::Decode)?;
self.in_flight.push(pts);
if self.in_flight.len() > MAX_IN_FLIGHT {
session.wait();
}
self.collect().map_err(CodecError::Decode)?;
Ok(self.bump())
}
fn flush(&mut self) -> Vec<DecodedFrame> {
if let Some(s) = self.session.as_ref() {
s.wait();
}
if let Err(e) = self.collect() {
log::warn!("{}: {e} while flushing", self.name);
}
self.in_flight.clear();
std::mem::take(&mut self.pending)
}
fn reset(&mut self) {
// A fresh session per run (made on the next sample): no reference pictures survive a seek.
self.session = None;
self.pending.clear();
self.in_flight.clear();
self.skip_rasl = false;
self.first = true;
}
fn name(&self) -> &str {
self.name
}
fn is_random_access(&self, sample: &[u8]) -> Option<bool> {
self.info.is_random_access(sample)
}
fn is_disposable(&self, sample: &[u8]) -> bool {
self.info.is_disposable(sample)
}
}
+174
View File
@@ -0,0 +1,174 @@
//! Shared helpers for the platform tests: ffmpeg-made fixtures (external generator only, never
//! linked), MP4 sample reading, and decoding a run of samples to compare decoders.
#![allow(dead_code)]
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use filmcraft_codecs::{DecodedFrame, VideoDecoder};
use filmcraft_frame::PixelData;
use filmcraft_isobmff::SampleEntry;
/// The parity fixtures: (file, ffmpeg arguments). B-frames everywhere (reordering), a 360-line
/// picture (coded 368: cropping), two GOPs or more (seeks), and an open-GOP HEVC stream (CRA with
/// RASL pictures).
pub const FIXTURES: &[(&str, &[&str])] = &[
(
"h264_high.mp4",
&[
"-f",
"lavfi",
"-i",
"testsrc2=s=640x360:r=24:d=3,noise=alls=12:allf=t",
"-c:v",
"libx264",
"-profile:v",
"high",
"-preset",
"fast",
"-x264-params",
"bframes=3:b-pyramid=normal:keyint=24:min-keyint=24:scenecut=0",
"-pix_fmt",
"yuv420p",
],
),
(
"hevc_main.mp4",
&[
"-f",
"lavfi",
"-i",
"testsrc2=s=640x360:r=24:d=3,noise=alls=12:allf=t",
"-c:v",
"libx265",
"-preset",
"fast",
"-tag:v",
"hvc1",
"-x265-params",
"log-level=error:bframes=4:keyint=24:min-keyint=24:scenecut=0:open-gop=1",
"-pix_fmt",
"yuv420p",
],
),
(
"hevc_main10.mp4",
&[
"-f",
"lavfi",
"-i",
"testsrc2=s=640x360:r=24:d=3,noise=alls=12:allf=t",
"-c:v",
"libx265",
"-preset",
"fast",
"-profile:v",
"main10",
"-tag:v",
"hvc1",
"-x265-params",
"log-level=error:bframes=4:keyint=24:min-keyint=24:scenecut=0",
"-pix_fmt",
"yuv420p10le",
],
),
];
pub fn dir() -> PathBuf {
filmcraft_testkit::fixtures_dir("platform")
}
/// Generate `name` with `ffmpeg -y -v error <args> <out>` (cached).
pub fn fixture(ff: &Path, name: &str, args: &[&str]) -> Option<PathBuf> {
let out = dir().join(name);
filmcraft_testkit::fixtures::generate(&out, |tmp| {
let st = Command::new(ff).args(["-y", "-v", "error"]).args(args).arg(tmp).stdin(Stdio::null()).status();
match st {
Ok(s) if s.success() => true,
other => {
eprintln!("fixture {name}: ffmpeg failed: {other:?}");
false
}
}
})
}
/// The fixture by name.
pub fn named(ff: &Path, name: &str) -> Option<PathBuf> {
let (_, args) = FIXTURES.iter().find(|f| f.0 == name)?;
fixture(ff, name, args)
}
/// A video track's sample entry, samples (decode order) with pts, and sync flags.
pub struct Stream {
pub entry: SampleEntry,
pub samples: Vec<(Vec<u8>, i64)>,
pub sync: Vec<bool>,
}
pub fn read_stream(path: &Path) -> Stream {
let bytes = std::fs::read(path).unwrap();
let file = filmcraft_isobmff::open(bytes.clone()).unwrap();
let t = file.track_of_kind(filmcraft_isobmff::TrackKind::Video).unwrap();
let track = &file.tracks[t];
let samples = (0..track.samples.len()).map(|i| (file.read_sample(&bytes, t, i).unwrap(), track.samples[i].pts)).collect();
Stream { entry: track.entries[0].clone(), samples, sync: track.samples.iter().map(|s| s.is_sync).collect() }
}
/// Feed `samples` then flush.
pub fn decode_all(d: &mut dyn VideoDecoder, samples: &[(Vec<u8>, i64)]) -> Vec<DecodedFrame> {
let mut out = Vec::new();
for (s, p) in samples {
out.extend(d.decode(s, *p).unwrap_or_else(|e| panic!("{}: {e}", d.name())));
}
out.extend(d.flush());
out
}
/// Assert two decoders' outputs are the same pictures (pts order, size, format, colour, aspect
/// and every sample).
pub fn assert_same(what: &str, a: &[DecodedFrame], b: &[DecodedFrame]) {
let pa: Vec<i64> = a.iter().map(|f| f.pts).collect();
let pb: Vec<i64> = b.iter().map(|f| f.pts).collect();
assert_eq!(pa, pb, "{what}: pts sequence");
for (x, y) in a.iter().zip(b) {
let (fx, fy) = (&x.frame, &y.frame);
assert_eq!((fx.width, fx.height, fx.par, fx.color), (fy.width, fy.height, fy.par, fy.color), "{what}: pts {} geometry / colour", x.pts);
match (&fx.data, &fy.data) {
(PixelData::Yuv8 { planes: p, chroma: c, .. }, PixelData::Yuv8 { planes: q, chroma: d, .. }) => {
assert_eq!(c, d, "{what}: chroma");
for i in 0..3 {
assert!(p[i] == q[i], "{what}: pts {} plane {i} differs (max diff {})", x.pts, max_diff8(&p[i], &q[i]));
}
}
(PixelData::Yuv16 { planes: p, chroma: c, bits: bx, .. }, PixelData::Yuv16 { planes: q, chroma: d, bits: by, .. }) => {
assert_eq!((c, bx), (d, by), "{what}: chroma / bits");
for i in 0..3 {
assert!(p[i] == q[i], "{what}: pts {} plane {i} differs (max diff {})", x.pts, max_diff16(&p[i], &q[i]));
}
}
_ => panic!("{what}: pts {}: pixel formats differ", x.pts),
}
}
}
fn max_diff8(a: &[u8], b: &[u8]) -> String {
if a.len() != b.len() {
return format!("sizes {} vs {}", a.len(), b.len());
}
a.iter().zip(b).map(|(x, y)| x.abs_diff(*y)).max().unwrap_or(0).to_string()
}
fn max_diff16(a: &[u16], b: &[u16]) -> String {
if a.len() != b.len() {
return format!("sizes {} vs {}", a.len(), b.len());
}
a.iter().zip(b).map(|(x, y)| x.abs_diff(*y)).max().unwrap_or(0).to_string()
}
pub fn xorshift(seed: &mut u64) -> u64 {
*seed ^= *seed << 13;
*seed ^= *seed >> 7;
*seed ^= *seed << 17;
*seed
}
+107
View File
@@ -0,0 +1,107 @@
//! [`HybridDecoder`] fallback on every platform, with a stand-in "hardware" decoder (our software
//! decoder that starts failing after N samples, as a lost hardware session does): the output must
//! be exactly the software decoder's, whatever sample the failure hits. Also: in-band parameter
//! sets that differ from the sample entry's switch to software; identical ones do not.
mod common;
use common::*;
use filmcraft_codecs::hw::NalStreamInfo;
use filmcraft_codecs::{DecodedFrame, Result, VideoDecoder};
use filmcraft_platform::HybridDecoder;
/// Decodes like the software decoder until `fail_after` samples were fed, then fails every call
/// (its pending pictures stay retrievable through `flush`, as with a real session).
struct Failing {
inner: Box<dyn VideoDecoder>,
fail_after: usize,
fed: usize,
}
impl VideoDecoder for Failing {
fn decode(&mut self, sample: &[u8], pts: i64) -> Result<Vec<DecodedFrame>> {
self.fed += 1;
if self.fed > self.fail_after {
return Err(filmcraft_codecs::CodecError::Decode("session invalidated (test)".into()));
}
self.inner.decode(sample, pts)
}
fn flush(&mut self) -> Vec<DecodedFrame> {
self.inner.flush()
}
fn reset(&mut self) {
self.inner.reset();
}
fn name(&self) -> &str {
"stand-in hardware"
}
}
fn hybrid(s: &Stream, fail_after: usize) -> HybridDecoder {
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
let inner = filmcraft_codecs::software_video_decoder(&s.entry).unwrap();
HybridDecoder::new(Box::new(Failing { inner, fail_after, fed: 0 }), s.entry.clone(), info)
}
#[test]
fn failures_anywhere_continue_with_the_software_output() {
let ff = filmcraft_testkit::require_ffmpeg!();
for (name, _) in FIXTURES {
let Some(path) = named(&ff, name) else { continue };
let s = read_stream(&path);
let reference = decode_all(filmcraft_codecs::software_video_decoder(&s.entry).unwrap().as_mut(), &s.samples);
let syncs: Vec<usize> = (0..s.samples.len()).filter(|&i| s.sync[i]).collect();
let mut points: Vec<usize> = vec![0, 1, 3, 7, s.samples.len() - 1, s.samples.len()];
for &k in &syncs {
points.extend([k, k + 1, k + 2, k + 5]);
}
for fail_after in points {
let mut d = hybrid(&s, fail_after);
let out = decode_all(&mut d, &s.samples);
assert_eq!(d.is_hardware(), fail_after >= s.samples.len(), "{name}: fell back at {fail_after}");
assert_same(&format!("{name} failing after {fail_after} samples"), &out, &reference);
}
// a failure after a seek (reset) into the second GOP
if let Some(&k) = syncs.get(1) {
let tail = &s.samples[k..];
let expect = {
let mut sw = filmcraft_codecs::software_video_decoder(&s.entry).unwrap();
decode_all(sw.as_mut(), tail)
};
let mut d = hybrid(&s, k + 4);
decode_all(&mut d, &s.samples[..k]);
d.reset();
assert_same(&format!("{name} failing after a seek"), &decode_all(&mut d, tail), &expect);
}
}
}
#[test]
fn changed_in_band_parameter_sets_switch_to_software() {
let ff = filmcraft_testkit::require_ffmpeg!();
let Some(path) = named(&ff, "h264_high.mp4") else { return };
let s = read_stream(&path);
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
let sps = info.parameter_sets[0].clone();
let with_sps = |sps: &[u8], sample: &[u8]| {
let mut v = (sps.len() as u32).to_be_bytes().to_vec();
v.extend_from_slice(sps);
v.extend_from_slice(sample);
v
};
// the same SPS repeated in band at every sync sample: stays in "hardware"
let same: Vec<_> = s.samples.iter().zip(&s.sync).map(|((x, p), &k)| (if k { with_sps(&sps, x) } else { x.clone() }, *p)).collect();
let mut d = hybrid(&s, usize::MAX);
let out = decode_all(&mut d, &same);
assert!(d.is_hardware());
assert_same("repeated SPS", &out, &decode_all(filmcraft_codecs::software_video_decoder(&s.entry).unwrap().as_mut(), &same));
// a different SPS (same values, trailing zero byte) at the second sync sample: software
let mut changed_sps = sps.clone();
changed_sps.push(0);
let k = (1..s.samples.len()).find(|&i| s.sync[i]).unwrap();
let changed: Vec<_> = s.samples.iter().enumerate().map(|(i, (x, p))| (if i == k { with_sps(&changed_sps, x) } else { x.clone() }, *p)).collect();
let mut d = hybrid(&s, usize::MAX);
let out = decode_all(&mut d, &changed);
assert!(!d.is_hardware(), "changed SPS: software");
assert_same("changed SPS", &out, &decode_all(filmcraft_codecs::software_video_decoder(&s.entry).unwrap().as_mut(), &changed));
}
+47
View File
@@ -0,0 +1,47 @@
//! Settings ▸ Playback ▸ Hardware decoding through the decoder registry and the media stack:
//! Off gives the software decoder (and no hardware frames), Auto the hardware one where the
//! system has it. Its own test binary: the switch is process-wide.
mod common;
use common::*;
use filmcraft_media::FrameRequest;
#[test]
fn off_uses_the_software_decoder() {
let ff = filmcraft_testkit::require_ffmpeg!();
let Some(path) = named(&ff, "hevc_main.mp4") else { return };
let s = read_stream(&path);
let available = filmcraft_platform::register();
assert_eq!(filmcraft_platform::register(), available, "registering twice is harmless");
filmcraft_codecs::hw::set_hardware_decoding(false);
let d = filmcraft_codecs::make_video_decoder(&s.entry).unwrap();
assert_eq!(d.name(), "FilmCraft HEVC");
let bytes: std::sync::Arc<[u8]> = std::fs::read(&path).unwrap().into();
let (g0, h0) = (filmcraft_codecs::gop_stats(), filmcraft_codecs::hw::hw_stats());
let src = filmcraft_codecs::open_bytes("hevc_main.mp4", bytes.clone()).unwrap();
let rate = src.info().frame_rate();
for i in 0..30 {
src.video_frame(FrameRequest::full(rate.tick_of(i))).unwrap();
}
assert!(filmcraft_codecs::gop_stats().frames > g0.frames, "frames counted");
assert_eq!(filmcraft_codecs::hw::hw_stats().frames, h0.frames, "Off: no hardware frames");
filmcraft_codecs::hw::set_hardware_decoding(true);
let d = filmcraft_codecs::make_video_decoder(&s.entry).unwrap();
let hw = filmcraft_platform::hardware_decoder_for(&s.entry);
if !hw {
eprintln!("SKIPPED (Auto half): no hardware decoder ({available:?})");
assert_eq!(d.name(), "FilmCraft HEVC");
return;
}
assert_eq!(d.name(), "VideoToolbox HEVC");
let h1 = filmcraft_codecs::hw::hw_stats();
let src = filmcraft_codecs::open_bytes("hevc_main.mp4", bytes).unwrap();
for i in 0..30 {
src.video_frame(FrameRequest::full(rate.tick_of(i))).unwrap();
}
let h2 = filmcraft_codecs::hw::hw_stats();
assert!(h2.frames > h1.frames && h2.sessions > h1.sessions, "Auto: hardware frames {h1:?} → {h2:?}");
}
+232
View File
@@ -0,0 +1,232 @@
//! VideoToolbox decoding against our software decoders (macOS): every picture of H.264 High,
//! HEVC Main and HEVC Main 10 fixtures with B-frames must be identical (bit-exact planes, colour,
//! pixel aspect, pts and presentation order), also after `reset` + reseek and a mid-stream
//! `flush`; a forced mid-stream failure must continue in software with the software decoder's
//! exact output; damaged samples and parameter sets must give errors or fall back, never crash or
//! hang. Skips without ffmpeg (fixture generator) or without a hardware decoder.
#![cfg(target_os = "macos")]
mod common;
use std::time::Duration;
use common::*;
use filmcraft_codecs::VideoDecoder;
use filmcraft_codecs::hw::NalStreamInfo;
use filmcraft_platform::HybridDecoder;
use filmcraft_platform::videotoolbox::VtDecoder;
/// The hardware decoder for a stream, or `None` (skip) when this machine has none for it.
fn hardware(s: &Stream) -> Option<Box<dyn VideoDecoder>> {
match filmcraft_platform::videotoolbox_factory(&s.entry) {
Some(Ok(d)) => {
assert!(d.name().starts_with("VideoToolbox"), "{}", d.name());
Some(d)
}
Some(Err(e)) => panic!("factory error: {e}"),
None => {
eprintln!("SKIPPED: no VideoToolbox hardware decoder for {}", s.entry.codec.name());
None
}
}
}
fn software(s: &Stream) -> Box<dyn VideoDecoder> {
filmcraft_codecs::software_video_decoder(&s.entry).unwrap()
}
#[test]
fn bit_exact_with_the_software_decoders() {
let ff = filmcraft_testkit::require_ffmpeg!();
for (name, _) in FIXTURES {
let Some(path) = named(&ff, name) else { continue };
let s = read_stream(&path);
let Some(mut hw) = hardware(&s) else { continue };
let mut sw = software(&s);
let a = decode_all(hw.as_mut(), &s.samples);
let b = decode_all(sw.as_mut(), &s.samples);
assert!(a.len() >= 60, "{name}: {} pictures", a.len());
assert_same(name, &a, &b);
assert!(hw.name().starts_with("VideoToolbox"), "{name}: stayed in hardware ({})", hw.name());
// reset + reseek to each later sync sample, decode a stretch, flush mid-stream
let syncs: Vec<usize> = (1..s.samples.len()).filter(|&i| s.sync[i]).collect();
assert!(!syncs.is_empty(), "{name}: more than one GOP");
for &k in syncs.iter().rev() {
let end = (k + 17).min(s.samples.len());
for d in [&mut hw, &mut sw] {
d.reset();
}
let a = decode_all(hw.as_mut(), &s.samples[k..end]);
let b = decode_all(sw.as_mut(), &s.samples[k..end]);
assert!(!a.is_empty(), "{name}: pictures after seeking to {k}");
assert_same(&format!("{name} from sample {k}"), &a, &b);
}
// back to the start after seeking around: the whole stream again
hw.reset();
sw.reset();
assert_same(&format!("{name} after resets"), &decode_all(hw.as_mut(), &s.samples), &decode_all(sw.as_mut(), &s.samples));
// random-access / disposable answers match the software decoder's
for (smp, _) in &s.samples {
assert_eq!(hw.is_random_access(smp), sw.is_random_access(smp), "{name}");
assert_eq!(hw.is_disposable(smp), sw.is_disposable(smp), "{name}");
}
}
}
#[test]
fn mid_stream_failure_continues_in_software() {
let ff = filmcraft_testkit::require_ffmpeg!();
for (name, _) in FIXTURES {
let Some(path) = named(&ff, name) else { continue };
let s = read_stream(&path);
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
let Ok(_) = VtDecoder::new(info.clone()) else {
eprintln!("SKIPPED: no VideoToolbox hardware decoder for {name}");
continue;
};
let reference = decode_all(software(&s).as_mut(), &s.samples);
// fail at the first sample, inside the first GOP, at a sync sample and just after one
let k = (1..s.samples.len()).find(|&i| s.sync[i]).unwrap();
for fail_at in [0, 5, k, k + 1, k + 9] {
let mut vt = VtDecoder::new(info.clone()).unwrap();
vt.fail_after(fail_at as u64);
let mut d = HybridDecoder::new(Box::new(vt), s.entry.clone(), info.clone());
let before = filmcraft_codecs::hw::hw_stats().fallbacks;
let out = decode_all(&mut d, &s.samples);
assert!(!d.is_hardware(), "{name}: switched to software");
assert!(filmcraft_codecs::hw::hw_stats().fallbacks > before, "{name}: fallback counted");
assert_same(&format!("{name} failing at sample {fail_at}"), &out, &reference);
// the instance stays in software across seeks
d.reset();
assert_same(&format!("{name} after the fallback and a reset"), &decode_all(&mut d, &s.samples), &reference);
}
}
}
/// Run `f` on a thread; fail if it panics or takes longer than `limit`.
fn bounded(what: &str, limit: Duration, f: impl FnOnce() + Send + 'static) {
let (tx, rx) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(f));
let _ = tx.send(r.is_ok());
});
match rx.recv_timeout(limit) {
Ok(true) => {}
Ok(false) => panic!("{what}: panicked"),
Err(_) => panic!("{what}: hung"),
}
}
#[test]
fn damaged_samples_and_parameter_sets_never_crash_or_hang() {
let ff = filmcraft_testkit::require_ffmpeg!();
for (name, _) in FIXTURES {
let Some(path) = named(&ff, name) else { continue };
let s = read_stream(&path);
if hardware(&s).is_none() {
continue;
}
// damaged samples (bit flips, truncation, corrupt length prefixes) through the hybrid
for seed in 1..=6u64 {
let samples = s.samples.clone();
let entry = s.entry.clone();
bounded(&format!("{name} seed {seed}"), Duration::from_secs(60), move || {
let mut seed = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
let mut d = filmcraft_platform::videotoolbox_factory(&entry).unwrap().unwrap();
for (i, (smp, pts)) in samples.iter().enumerate().take(40) {
let mut m = smp.clone();
match xorshift(&mut seed) % 4 {
0 => {
for _ in 0..1 + xorshift(&mut seed) % 8 {
let at = (xorshift(&mut seed) as usize) % m.len().max(1);
if let Some(b) = m.get_mut(at) {
*b ^= 1 << (xorshift(&mut seed) % 8);
}
}
}
1 => m.truncate((xorshift(&mut seed) as usize) % m.len().max(1)),
2 if m.len() > 4 => m[..4].copy_from_slice(&(xorshift(&mut seed) as u32).to_be_bytes()),
_ => {}
}
let _ = d.decode(&m, *pts);
if i % 13 == 12 {
d.reset();
}
}
let _ = d.flush();
});
}
// damaged parameter sets: the factory declines or the decoder errors / falls back
let rec = match &s.entry.codec {
filmcraft_isobmff::CodecConfig::Avc(a) => a.to_bytes(),
filmcraft_isobmff::CodecConfig::Hevc(c) => c.to_bytes(),
_ => continue,
};
let hevc = matches!(s.entry.codec, filmcraft_isobmff::CodecConfig::Hevc(_));
let mut seed = 0x1234_5678u64;
for round in 0..24 {
let mut r = rec.clone();
if round % 3 == 2 {
r.truncate(r.len() * (round + 1) / 30);
} else {
for _ in 0..1 + round % 4 {
let at = 6 + (xorshift(&mut seed) as usize) % r.len().saturating_sub(6).max(1);
if let Some(b) = r.get_mut(at) {
*b ^= 1 << (xorshift(&mut seed) % 8);
}
}
}
let samples: Vec<_> = s.samples.iter().take(12).cloned().collect();
bounded(&format!("{name} record round {round}"), Duration::from_secs(60), move || {
let Some(info) = (if hevc { NalStreamInfo::from_hvcc(&r) } else { NalStreamInfo::from_avcc(&r) }).ok() else { return };
let Ok(mut vt) = VtDecoder::new(info) else { return };
for (smp, pts) in &samples {
if vt.decode(smp, *pts).is_err() {
break;
}
}
let _ = vt.flush();
});
}
}
}
/// 4:2:2 (which our HEVC decoder does not decode): when VideoToolbox takes it, its pictures must
/// match ffmpeg's decode exactly; when it does not, the factory declines.
#[test]
fn hevc_422_10bit_matches_ffmpeg_when_the_hardware_takes_it() {
let ff = filmcraft_testkit::require_ffmpeg!();
let args = [
"-f",
"lavfi",
"-i",
"testsrc2=s=640x360:r=24:d=1,noise=alls=12:allf=t",
"-c:v",
"libx265",
"-preset",
"fast",
"-tag:v",
"hvc1",
"-x265-params",
"log-level=error:bframes=3:keyint=12:min-keyint=12:scenecut=0",
"-pix_fmt",
"yuv422p10le",
];
let Some(path) = fixture(&ff, "hevc_422_10.mp4", &args) else { return };
let s = read_stream(&path);
let Some(mut hw) = hardware(&s) else { return };
let out = decode_all(hw.as_mut(), &s.samples);
let o =
std::process::Command::new(&ff).args(["-v", "error", "-i", path.to_str().unwrap(), "-f", "rawvideo", "-pix_fmt", "yuv422p10le", "-"]).output().unwrap();
let (w, h) = (640usize, 360usize);
let fsize = (w * h + 2 * (w / 2) * h) * 2;
assert_eq!(out.len(), o.stdout.len() / fsize, "frame count");
for (i, f) in out.iter().enumerate() {
let raw: Vec<u16> = o.stdout[i * fsize..(i + 1) * fsize].as_chunks::<2>().0.iter().map(|b| u16::from_le_bytes([b[0], b[1]])).collect();
let filmcraft_frame::PixelData::Yuv16 { planes, chroma, bits, .. } = &f.frame.data else { panic!("16-bit planes") };
assert_eq!((*chroma, *bits), (filmcraft_frame::Chroma::C422, 10));
let (y, c) = (w * h, (w / 2) * h);
assert!(planes[0][..] == raw[..y] && planes[1][..] == raw[y..y + c] && planes[2][..] == raw[y + c..], "frame {i} differs from ffmpeg");
}
}
+9
View File
@@ -70,6 +70,15 @@ impl Blend {
Blend::Color,
Blend::Luminosity,
];
/// Position in [`Blend::ALL`] (= `filmcraft_project::effect::BLEND_MODES`).
pub fn index(self) -> u32 {
Self::ALL.iter().position(|b| *b == self).unwrap_or(0) as u32
}
/// Whether compositing needs the destination colour (every mode but Normal and Dissolve,
/// which are plain "over" per pixel).
pub fn reads_destination(self) -> bool {
!matches!(self, Blend::Normal | Blend::Dissolve)
}
/// Index into `filmcraft_project::effect::BLEND_MODES`.
pub fn from_index(i: u32) -> Blend {
Self::ALL.get(i as usize).copied().unwrap_or(Blend::Normal)
+46 -303
View File
@@ -34,10 +34,10 @@ pub struct FxCtx<'a> {
pub working: filmcraft_color::WorkingSpace,
}
fn f(e: &EffectInstance, id: &str, cx: &FxCtx) -> f32 {
pub(crate) fn f(e: &EffectInstance, id: &str, cx: &FxCtx) -> f32 {
e.f64_at(id, cx.t) as f32
}
fn b(e: &EffectInstance, id: &str) -> bool {
pub(crate) fn b(e: &EffectInstance, id: &str) -> bool {
e.param(id).and_then(|p| p.value.as_bool()).unwrap_or(false)
}
/// A section switch (missing in older projects = on).
@@ -50,17 +50,17 @@ fn text<'e>(e: &'e EffectInstance, id: &str) -> &'e str {
_ => "",
}
}
fn choice(e: &EffectInstance, id: &str) -> u32 {
pub(crate) fn choice(e: &EffectInstance, id: &str) -> u32 {
match e.param(id).map(|p| &p.value) {
Some(ParamValue::Choice(c)) => *c,
_ => 0,
}
}
fn color(e: &EffectInstance, id: &str, cx: &FxCtx) -> [f32; 4] {
pub(crate) fn color(e: &EffectInstance, id: &str, cx: &FxCtx) -> [f32; 4] {
e.param(id).and_then(|p| p.value_at(cx.t).as_color()).unwrap_or([1.0; 4])
}
/// A point param in working-image pixels; NaN components default to the image centre.
fn point(e: &EffectInstance, id: &str, cx: &FxCtx, img: &Image) -> Vec2 {
pub(crate) fn point(e: &EffectInstance, id: &str, cx: &FxCtx, img: &Image) -> Vec2 {
let v = e.param(id).map(|p| p.vec2_at(cx.t)).unwrap_or(Vec2::new(f64::NAN, f64::NAN));
Vec2::new(
if v.x.is_nan() { img.w as f64 / 2.0 } else { v.x * cx.px_scale as f64 },
@@ -93,196 +93,13 @@ pub fn apply(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
if crate::vfx::apply(img, e, cx) {
return;
}
// effects with a GPU implementation: evaluated parameters + CPU reference (`gpufx`)
if let Some(op) = crate::gpufx::FxOp::eval(e, cx, img.w, img.h) {
op.apply(img);
return;
}
match e.effect.as_str() {
"brightness_contrast" => {
let br = f(e, "brightness", cx) / 100.0 * 0.4;
let co = 1.0 + f(e, "contrast", cx) / 100.0;
img.map_rgb(|c, _, _| {
let c = enc(c);
dec(c.map(|v| (v - 0.5) * co + 0.5 + br))
});
}
"proc_amp" => {
let br = f(e, "brightness", cx) / 100.0 * 0.4;
let co = f(e, "contrast", cx) / 100.0;
let hue = f(e, "hue", cx) / 360.0;
let sat = f(e, "saturation", cx) / 100.0;
img.map_rgb(|c, _, _| {
let c = enc(c);
let mut hsl = rgb_to_hsl(c[0], c[1], c[2]);
hsl[0] = (hsl[0] + hue).rem_euclid(1.0);
hsl[1] = (hsl[1] * sat).clamp(0.0, 1.0);
let c = hsl_to_rgb(hsl[0], hsl[1], hsl[2]);
dec(c.map(|v| (v - 0.5) * co + 0.5 + br))
});
}
"tint" => {
let bl = color(e, "black", cx);
let wh = color(e, "white", cx);
let amt = f(e, "amount", cx) / 100.0;
img.map_rgb(|c, _, _| {
let l = linear_to_srgb(luma709(c[0], c[1], c[2]).max(0.0));
let t = [bl[0] + (wh[0] - bl[0]) * l, bl[1] + (wh[1] - bl[1]) * l, bl[2] + (wh[2] - bl[2]) * l];
let e = enc(c);
dec([e[0] + (t[0] - e[0]) * amt, e[1] + (t[1] - e[1]) * amt, e[2] + (t[2] - e[2]) * amt])
});
}
"black_white" => img.map_rgb(|c, _, _| {
let l = luma709(c[0], c[1], c[2]);
[l, l, l]
}),
"color_balance" => {
let g = |k: &str| f(e, k, cx) / 100.0 * 0.25;
let sh = [g("shadow_r"), g("shadow_g"), g("shadow_b")];
let md = [g("mid_r"), g("mid_g"), g("mid_b")];
let hi = [g("hi_r"), g("hi_g"), g("hi_b")];
let preserve = b(e, "preserve");
img.map_rgb(|c, _, _| {
let c = enc(c);
let l = 0.2126 * c[0] + 0.7152 * c[1] + 0.0722 * c[2];
let ws = (1.0 - l).powi(2);
let wh = l.powi(2);
let wm = 1.0 - ws - wh;
let mut o = [0.0; 3];
for k in 0..3 {
o[k] = c[k] + sh[k] * ws + md[k] * wm.max(0.0) + hi[k] * wh;
}
if preserve {
let l2 = 0.2126 * o[0] + 0.7152 * o[1] + 0.0722 * o[2];
let d = l - l2;
o = o.map(|v| v + d);
}
dec(o)
});
}
"leave_color" => {
let amt = f(e, "amount", cx) / 100.0;
let key = color(e, "color", cx);
let tol = f(e, "tolerance", cx) / 100.0;
let soft = f(e, "softness", cx) / 100.0 + 1e-4;
let kh = rgb_to_hsl(key[0], key[1], key[2])[0];
img.map_rgb(|c, _, _| {
let ec = enc(c);
let h = rgb_to_hsl(ec[0], ec[1], ec[2])[0];
let d = (h - kh).abs().min(1.0 - (h - kh).abs()) * 2.0;
let keep = 1.0 - ((d - tol) / soft).clamp(0.0, 1.0);
let l = luma709(c[0], c[1], c[2]);
let k = amt * (1.0 - keep);
[c[0] + (l - c[0]) * k, c[1] + (l - c[1]) * k, c[2] + (l - c[2]) * k]
});
}
"change_to_color" => {
let from = color(e, "from", cx);
let to = color(e, "to", cx);
let tol = f(e, "hue_tol", cx) / 100.0;
let soft = f(e, "softness", cx) / 100.0 * 0.3 + 1e-4;
let fh = rgb_to_hsl(from[0], from[1], from[2])[0];
let th = rgb_to_hsl(to[0], to[1], to[2])[0];
img.map_rgb(|c, _, _| {
let ec = enc(c);
let mut hsl = rgb_to_hsl(ec[0], ec[1], ec[2]);
let d = (hsl[0] - fh).abs().min(1.0 - (hsl[0] - fh).abs());
let w = 1.0 - ((d - tol) / soft).clamp(0.0, 1.0);
hsl[0] = (hsl[0] + (th - fh) * w).rem_euclid(1.0);
dec(hsl_to_rgb(hsl[0], hsl[1], hsl[2]))
});
}
"color_pass" => {
let key = color(e, "color", cx);
let sim = f(e, "similarity", cx) / 100.0;
let rev = b(e, "reverse");
img.map_rgb(|c, _, _| {
let ec = enc(c);
let d = ((ec[0] - key[0]).powi(2) + (ec[1] - key[1]).powi(2) + (ec[2] - key[2]).powi(2)).sqrt();
let pass = (d <= sim * 1.2) != rev;
if pass {
c
} else {
let l = luma709(c[0], c[1], c[2]);
[l, l, l]
}
});
}
"gamma_correction" => {
let g = f(e, "gamma", cx) / 10.0;
img.map_rgb(|c, _, _| dec(enc(c).map(|v| v.max(0.0).powf(g))));
}
"levels" => {
let ib = f(e, "in_black", cx) / 255.0;
let iw = (f(e, "in_white", cx) / 255.0).max(ib + 1e-3);
let ob = f(e, "out_black", cx) / 255.0;
let ow = f(e, "out_white", cx) / 255.0;
let g = 100.0 / f(e, "gamma", cx).max(1.0);
img.map_rgb(|c, _, _| dec(enc(c).map(|v| ob + (((v - ib) / (iw - ib)).clamp(0.0, 1.0)).powf(g) * (ow - ob))));
}
"extract" => {
let lo = f(e, "black", cx) / 255.0;
let hi = f(e, "white", cx) / 255.0;
let soft = f(e, "softness", cx) / 100.0 * 0.2 + 1e-4;
let inv = b(e, "invert");
img.map_rgb(|c, _, _| {
let l = linear_to_srgb(luma709(c[0], c[1], c[2]).max(0.0));
let inside = ((l - lo) / soft).clamp(0.0, 1.0).min(((hi - l) / soft).clamp(0.0, 1.0));
let v = if inv { 1.0 - inside } else { inside };
[v, v, v]
});
}
"invert" => {
let ch = choice(e, "channel");
let blend = f(e, "blend", cx) / 100.0;
if ch == 4 {
img.px.par_chunks_mut(4).for_each(|p| {
let a = p[3];
let na = 1.0 - a;
let k = if a > 1e-6 { na / a } else { 0.0 };
for c in &mut p[..3] {
*c *= k;
}
p[3] = na * (1.0 - blend) + a * blend;
});
} else {
img.map_rgb(|c, _, _| {
let ec = enc(c);
let mut o = ec;
for k in 0..3 {
if ch == 0 || ch as usize == k + 1 {
o[k] = 1.0 - ec[k];
}
}
dec([o[0] + (ec[0] - o[0]) * blend, o[1] + (ec[1] - o[1]) * blend, o[2] + (ec[2] - o[2]) * blend])
});
}
}
"posterize" => {
let n = f(e, "levels", cx).max(2.0) - 1.0;
img.map_rgb(|c, _, _| dec(enc(c).map(|v| (v * n).round() / n)));
}
"lumetri" => lumetri(img, e, cx),
"gaussian_blur" => {
let r = f(e, "blurriness", cx) * cx.px_scale * 0.5;
let dims = choice(e, "dimensions");
let repeat = b(e, "repeat_edge");
gaussian(img, if dims == 2 { 0.0 } else { r }, if dims == 1 { 0.0 } else { r }, repeat);
}
"camera_blur" => {
let r = f(e, "percent", cx) * cx.px_scale * 0.3;
gaussian(img, r, r, true);
}
"directional_blur" => {
let len = f(e, "length", cx) * cx.px_scale * 2.0;
let dir = (f(e, "direction", cx) as f64).to_radians();
directional_blur(img, len, dir);
}
"sharpen" => {
let amt = f(e, "amount", cx) / 100.0;
unsharp(img, 1.0 * cx.px_scale.max(0.35), amt, 0.0);
}
"unsharp_mask" => {
let amt = f(e, "amount", cx) / 100.0;
let r = f(e, "radius", cx) * cx.px_scale;
let th = f(e, "threshold", cx) / 255.0;
unsharp(img, r, amt, th);
}
"median" => {
let r = (f(e, "radius", cx) * cx.px_scale).round().clamp(0.0, 4.0) as isize;
if r > 0 {
@@ -389,100 +206,6 @@ pub fn apply(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
img.map_rgb(|o, _, _| [lc[0] + (o[0] - lc[0]) * blend, lc[1] + (o[1] - lc[1]) * blend, lc[2] + (o[2] - lc[2]) * blend]);
}
}
"crop" => {
let l = f(e, "left", cx) / 100.0;
let t = f(e, "top", cx) / 100.0;
let r = f(e, "right", cx) / 100.0;
let bt = f(e, "bottom", cx) / 100.0;
let feather = f(e, "feather", cx) * cx.px_scale;
if b(e, "zoom") && l + r < 0.99 && t + bt < 0.99 {
let src = img.clone();
let (w, h) = (img.w as f64, img.h as f64);
let m =
Affine::scale(1.0 / (1.0 - (l + r) as f64), 1.0 / (1.0 - (t + bt) as f64)).then_apply(&Affine::translate(-(l as f64) * w, -(t as f64) * h));
*img = src.transformed(img.w, img.h, &m);
} else {
crop(img, l, t, r, bt, feather);
}
}
"horizontal_flip" => {
let w = img.w;
img.px.par_chunks_mut(w * 4).for_each(|row| {
for x in 0..w / 2 {
for k in 0..4 {
row.swap(x * 4 + k, (w - 1 - x) * 4 + k);
}
}
});
}
"vertical_flip" => {
let (w, h) = (img.w, img.h);
for y in 0..h / 2 {
let (a, bb) = img.px.split_at_mut((h - 1 - y) * w * 4);
a[y * w * 4..(y + 1) * w * 4].swap_with_slice(&mut bb[..w * 4]);
}
}
"edge_feather" => {
let amt = f(e, "amount", cx) / 100.0 * (img.w.min(img.h) as f32) * 0.5;
crop(img, 0.0, 0.0, 0.0, 0.0, amt);
}
"transform" => {
let anchor = point(e, "anchor", cx, img);
let pos = point(e, "position", cx, img);
let sh = f(e, "scale_height", cx) as f64 / 100.0;
let sw = if b(e, "uniform_scale") { sh } else { f(e, "scale_width", cx) as f64 / 100.0 };
let rot = f(e, "rotation", cx) as f64;
let skew = (f(e, "skew", cx) as f64).to_radians().tan();
let skew_axis = f(e, "skew_axis", cx) as f64;
let op = f(e, "opacity", cx) / 100.0;
let sk = Affine::rotate_deg(skew_axis)
.then_apply(&Affine { a: 1.0, b: 0.0, c: skew, d: 1.0, e: 0.0, f: 0.0 })
.then_apply(&Affine::rotate_deg(-skew_axis));
let m = Affine::translate(pos.x, pos.y)
.then_apply(&Affine::rotate_deg(rot))
.then_apply(&sk)
.then_apply(&Affine::scale(sw, sh))
.then_apply(&Affine::translate(-anchor.x, -anchor.y));
let mut out = img.transformed(img.w, img.h, &m);
out.scale_alpha(op);
*img = out;
}
"mirror" => {
let c = point(e, "center", cx, img);
let ang = (f(e, "angle", cx) as f64).to_radians();
let (nx, ny) = (ang.cos(), ang.sin());
let src = img.clone();
let w = img.w;
img.px.par_chunks_mut(w * 4).enumerate().for_each(|(y, row)| {
for x in 0..w {
let (px, py) = (x as f64 + 0.5, y as f64 + 0.5);
let d = (px - c.x) * nx + (py - c.y) * ny;
if d > 0.0 {
let (rx, ry) = (px - 2.0 * d * nx, py - 2.0 * d * ny);
row[x * 4..x * 4 + 4].copy_from_slice(&src.sample_bilinear(rx as f32, ry as f32));
}
}
});
}
"offset" => {
let s = point(e, "shift", cx, img);
let (dx, dy) = (s.x - img.w as f64 / 2.0, s.y - img.h as f64 / 2.0);
let blend = f(e, "blend", cx) / 100.0;
let src = img.clone();
let (w, h) = (img.w as f64, img.h as f64);
let wi = img.w;
img.px.par_chunks_mut(wi * 4).enumerate().for_each(|(y, row)| {
for x in 0..wi {
let u = (x as f64 + 0.5 - dx).rem_euclid(w);
let v = (y as f64 + 0.5 - dy).rem_euclid(h);
let p = src.sample_bilinear_clamped(u as f32, v as f32);
let o = src.get(x, y);
for k in 0..4 {
row[x * 4 + k] = p[k] + (o[k] - p[k]) * blend;
}
}
});
}
"twirl" => {
let c = point(e, "center", cx, img);
let ang = (f(e, "angle", cx) as f64).to_radians();
@@ -782,8 +505,12 @@ pub(crate) fn warp_opt(img: &mut Image, f: impl Fn(f64, f64) -> Option<(f64, f64
pub(crate) fn crop(img: &mut Image, l: f32, t: f32, r: f32, b: f32, feather: f32) {
let (w, h) = (img.w as f32, img.h as f32);
let (x0, x1, y0, y1) = (l * w, w * (1.0 - r), t * h, h * (1.0 - b));
let fe = feather.max(0.0);
crop_px(img, l * w, w * (1.0 - r), t * h, h * (1.0 - b), feather.max(0.0));
}
/// Fade alpha towards the edges of the rectangle x0..x1 × y0..y1 (working pixels): over `fe`
/// pixels inside it, or a half-pixel antialiased edge when `fe` is 0.
pub(crate) fn crop_px(img: &mut Image, x0: f32, x1: f32, y0: f32, y1: f32, fe: f32) {
let wi = img.w;
img.px.par_chunks_mut(wi * 4).enumerate().for_each(|(y, row)| {
let py = y as f32 + 0.5;
@@ -1047,26 +774,42 @@ fn transpose(img: &Image) -> Image {
/// Gaussian blur via 3 box passes per axis (O(1) per pixel for any radius).
pub fn gaussian(img: &mut Image, sigma_x: f32, sigma_y: f32, repeat_edge: bool) {
let (rx, ry) = gaussian_boxes(img.w, img.h, sigma_x, sigma_y);
box_blur(img, &rx, &ry, repeat_edge);
}
/// The box radii [`gaussian`] runs on a `w`×`h` image, per axis (empty: that axis is untouched).
pub(crate) fn gaussian_boxes(w: usize, h: usize, sigma_x: f32, sigma_y: f32) -> (Vec<u32>, Vec<u32>) {
// beyond a few image sizes every radius gives the same (flat) result; cap to keep box sizes sane
let sigma_x = if sigma_x.is_finite() { sigma_x.min(img.w.max(8) as f32 * 2.0) } else { 0.0 };
let sigma_y = if sigma_y.is_finite() { sigma_y.min(img.h.max(8) as f32 * 2.0) } else { 0.0 };
if sigma_x > 0.3 {
for r in boxes_for_gauss(sigma_x, 3) {
box_rows(&mut img.px, img.w, r, repeat_edge);
}
let sigma_x = if sigma_x.is_finite() { sigma_x.min(w.max(8) as f32 * 2.0) } else { 0.0 };
let sigma_y = if sigma_y.is_finite() { sigma_y.min(h.max(8) as f32 * 2.0) } else { 0.0 };
let radii = |s: f32| if s > 0.3 { boxes_for_gauss(s, 3).into_iter().map(|r| r.min(u32::MAX as usize) as u32).collect() } else { Vec::new() };
(radii(sigma_x), radii(sigma_y))
}
/// Box passes of radii `rx` along rows, then `ry` along columns.
pub(crate) fn box_blur(img: &mut Image, rx: &[u32], ry: &[u32], repeat_edge: bool) {
for r in rx {
box_rows(&mut img.px, img.w, *r as usize, repeat_edge);
}
if sigma_y > 0.3 {
if !ry.is_empty() {
let mut t = transpose(img);
for r in boxes_for_gauss(sigma_y, 3) {
box_rows(&mut t.px, t.w, r, repeat_edge);
for r in ry {
box_rows(&mut t.px, t.w, *r as usize, repeat_edge);
}
*img = transpose(&t);
}
}
pub(crate) fn unsharp(img: &mut Image, radius: f32, amount: f32, threshold: f32) {
let (rx, ry) = gaussian_boxes(img.w, img.h, radius, radius);
unsharp_boxes(img, &rx, &ry, amount, threshold);
}
/// Unsharp mask against a repeat-edge box-Gaussian of radii `rx` / `ry`.
pub(crate) fn unsharp_boxes(img: &mut Image, rx: &[u32], ry: &[u32], amount: f32, threshold: f32) {
let mut blurred = img.clone();
gaussian(&mut blurred, radius, radius, true);
box_blur(&mut blurred, rx, ry, true);
img.px.par_chunks_mut(4).zip(blurred.px.par_chunks(4)).for_each(|(p, bq)| {
for k in 0..3 {
let d = p[k] - bq[k];
@@ -1078,12 +821,12 @@ pub(crate) fn unsharp(img: &mut Image, radius: f32, amount: f32, threshold: f32)
});
}
pub(crate) fn directional_blur(img: &mut Image, len: f32, dir: f64) {
if len < 0.5 {
/// Average of `steps` edge-clamped bilinear taps along (dx, dy), centred on each pixel (no-op
/// below 2 steps).
pub(crate) fn directional_taps(img: &mut Image, dx: f32, dy: f32, steps: usize) {
if steps < 2 {
return;
}
let steps = (len.ceil() as usize).clamp(2, 64);
let (dx, dy) = ((dir.sin() as f32) * len, (-dir.cos() as f32) * len);
let src = img.clone();
let w = img.w;
img.px.par_chunks_mut(w * 4).enumerate().for_each(|(y, row)| {
+737
View File
@@ -0,0 +1,737 @@
//! Standard effects with a GPU implementation (`filmcraft_gpu` runs them in WGSL).
//!
//! Each effect here is evaluated in two steps: [`FxOp::eval`] reads the effect's parameters at a
//! time (keyframes, defaults, `px_scale`, the working image size) into plain numbers, and
//! [`FxOp::apply`] runs the CPU reference on a working [`Image`]. `effects::apply` uses exactly
//! these two steps, so the CPU render and the GPU plan agree on every parameter (and on how
//! hostile values are clamped) by construction. The GPU compositor receives the evaluated
//! [`FxOp`]s in a [`crate::plan::LayerFx`] and reproduces `apply` per pixel; the plan only hands an
//! op to the GPU when [`FxOp::gpu_ok`] says the shader covers it (finite numbers, no minifying
//! resample), otherwise the clip is rendered on the CPU as before.
use filmcraft_color::{hsl_to_rgb, linear_to_srgb, luma709, rgb_to_hsl};
use filmcraft_geom::Affine;
use filmcraft_project::EffectInstance;
use rayon::prelude::*;
use crate::effects::{FxCtx, b, choice, color, dec, enc, f, gaussian_boxes, point};
use crate::image::Image;
/// A bilinear resample of the working image through an affine map (`Image::transformed` at a
/// magnification or mild minification: no mip pre-filter).
#[derive(Clone, Debug, PartialEq)]
pub struct Resample {
/// Destination pixel → source pixel (None: singular map, the result is transparent).
pub inv: Option<Affine>,
/// Destination pixels written: x0..x1, y0..y1 (the rest becomes transparent).
pub rect: [u32; 4],
/// Alpha scale applied afterwards (Transform's Opacity; 1 = none).
pub opacity: f32,
}
/// One evaluated effect.
#[derive(Clone, Debug, PartialEq)]
pub enum FxOp {
BrightnessContrast {
br: f32,
co: f32,
},
ProcAmp {
br: f32,
co: f32,
hue: f32,
sat: f32,
},
Tint {
black: [f32; 3],
white: [f32; 3],
amount: f32,
},
BlackWhite,
ColorBalance {
sh: [f32; 3],
md: [f32; 3],
hi: [f32; 3],
preserve: bool,
},
LeaveColor {
amount: f32,
key_hue: f32,
tol: f32,
soft: f32,
},
ChangeToColor {
from_hue: f32,
to_hue: f32,
tol: f32,
soft: f32,
},
ColorPass {
key: [f32; 3],
sim: f32,
reverse: bool,
},
Gamma {
g: f32,
},
Levels {
ib: f32,
iw: f32,
ob: f32,
ow: f32,
g: f32,
},
Extract {
lo: f32,
hi: f32,
soft: f32,
invert: bool,
},
Invert {
channel: u32,
blend: f32,
},
Posterize {
n: f32,
},
/// Three box passes per axis approximating a Gaussian (radii per pass; empty: axis untouched).
Gaussian {
rx: Vec<u32>,
ry: Vec<u32>,
repeat: bool,
},
/// `steps` bilinear (edge-clamped) taps along `(dx, dy)` centred on the pixel.
DirectionalBlur {
dx: f32,
dy: f32,
steps: u32,
},
/// Unsharp mask over a repeat-edge Gaussian of the given box radii (per axis).
Unsharp {
rx: Vec<u32>,
ry: Vec<u32>,
amount: f32,
threshold: f32,
},
/// Crop / Edge Feather: alpha falloff `feather` px inside the rectangle (0: half-pixel AA).
Crop {
x0: f32,
x1: f32,
y0: f32,
y1: f32,
feather: f32,
},
Resample(Resample),
HFlip,
VFlip,
Mirror {
cx: f32,
cy: f32,
nx: f32,
ny: f32,
},
/// Wrap-around shift (dx, dy already reduced into 0..w, 0..h), mixed with the original.
Offset {
dx: f32,
dy: f32,
blend: f32,
},
/// ASC CDL on display-encoded colour (slope, offset, power, saturation).
AscCdl {
slope: [f32; 3],
offset: [f32; 3],
power: [f32; 3],
sat: f32,
},
/// Channel Mixer: rows (r, g, b, constant) per output channel, on display-encoded colour.
ChannelMix {
m: [[f32; 4]; 3],
},
/// Color Replace: `sim` already ×1.2; `replace_hsl` the HSL of the replacement colour.
ColorReplace {
sim: f32,
solid: bool,
target: [f32; 3],
replace: [f32; 3],
replace_hsl: [f32; 3],
},
AlphaAdjust {
opacity: f32,
ignore: bool,
invert: bool,
mask_only: bool,
},
}
/// Effect ids [`FxOp::eval`] understands (the GPU-capable standard effects).
pub const GPU_EFFECTS: &[&str] = &[
"brightness_contrast",
"proc_amp",
"tint",
"black_white",
"color_balance",
"leave_color",
"change_to_color",
"color_pass",
"gamma_correction",
"levels",
"extract",
"invert",
"posterize",
"gaussian_blur",
"gaussian_blur_legacy",
"camera_blur",
"directional_blur",
"directional_blur_legacy",
"sharpen",
"unsharp_mask",
"crop",
"edge_feather",
"transform",
"horizontal_flip",
"vertical_flip",
"mirror",
"offset",
"asc_cdl",
"channel_mix",
"color_replace",
"alpha_adjust",
];
/// `Image::transformed` without the mip path: the destination rectangle it writes and the inverse.
fn resample(w: usize, h: usize, m: &Affine, opacity: f32) -> Resample {
let Some(inv) = m.inverse() else { return Resample { inv: None, rect: [0; 4], opacity } };
let b = m.bounds(&filmcraft_geom::Rect::new(0.0, 0.0, w as f64, h as f64));
let y0 = (b.y.floor().max(0.0) as usize).min(h);
let y1 = (b.bottom().ceil().max(0.0) as usize).min(h);
let x0 = (b.x.floor().max(0.0) as usize).min(w);
let x1 = (b.right().ceil().max(0.0) as usize).min(w);
Resample { inv: Some(inv), rect: [x0 as u32, x1 as u32, y0.min(y1) as u32, y1 as u32], opacity }
}
/// Whether `Image::transformed` would take its mip (downsample) path for `m` on a `w`×`h` image.
fn minifies(m: &Affine, w: usize, h: usize) -> bool {
let sx = (m.a * m.a + m.b * m.b).sqrt();
let sy = (m.c * m.c + m.d * m.d).sqrt();
let minify = 1.0 / sx.min(sy).max(1e-6);
// (a non-finite factor: keep such maps off the GPU; `apply` still hands them to the CPU's
// `Image::transformed`, which decides as before)
!minify.is_finite() || minify >= 2.0 && w >= 4 && h >= 4
}
impl FxOp {
/// Evaluate `e` at `cx.t` for a `w`×`h` working image. None for effects without a GPU
/// implementation (and for disabled effects).
pub fn eval(e: &EffectInstance, cx: &FxCtx, w: usize, h: usize) -> Option<FxOp> {
if !e.enabled {
return None;
}
Some(match e.effect.as_str() {
"brightness_contrast" => FxOp::BrightnessContrast { br: f(e, "brightness", cx) / 100.0 * 0.4, co: 1.0 + f(e, "contrast", cx) / 100.0 },
"proc_amp" => FxOp::ProcAmp {
br: f(e, "brightness", cx) / 100.0 * 0.4,
co: f(e, "contrast", cx) / 100.0,
hue: f(e, "hue", cx) / 360.0,
sat: f(e, "saturation", cx) / 100.0,
},
"tint" => {
let (bl, wh) = (color(e, "black", cx), color(e, "white", cx));
FxOp::Tint { black: [bl[0], bl[1], bl[2]], white: [wh[0], wh[1], wh[2]], amount: f(e, "amount", cx) / 100.0 }
}
"black_white" => FxOp::BlackWhite,
"color_balance" => {
let g = |k: &str| f(e, k, cx) / 100.0 * 0.25;
FxOp::ColorBalance {
sh: [g("shadow_r"), g("shadow_g"), g("shadow_b")],
md: [g("mid_r"), g("mid_g"), g("mid_b")],
hi: [g("hi_r"), g("hi_g"), g("hi_b")],
preserve: b(e, "preserve"),
}
}
"leave_color" => {
let key = color(e, "color", cx);
FxOp::LeaveColor {
amount: f(e, "amount", cx) / 100.0,
key_hue: rgb_to_hsl(key[0], key[1], key[2])[0],
tol: f(e, "tolerance", cx) / 100.0,
soft: f(e, "softness", cx) / 100.0 + 1e-4,
}
}
"change_to_color" => {
let (from, to) = (color(e, "from", cx), color(e, "to", cx));
FxOp::ChangeToColor {
from_hue: rgb_to_hsl(from[0], from[1], from[2])[0],
to_hue: rgb_to_hsl(to[0], to[1], to[2])[0],
tol: f(e, "hue_tol", cx) / 100.0,
soft: f(e, "softness", cx) / 100.0 * 0.3 + 1e-4,
}
}
"color_pass" => {
let key = color(e, "color", cx);
FxOp::ColorPass { key: [key[0], key[1], key[2]], sim: f(e, "similarity", cx) / 100.0, reverse: b(e, "reverse") }
}
"gamma_correction" => FxOp::Gamma { g: f(e, "gamma", cx) / 10.0 },
"levels" => {
let ib = f(e, "in_black", cx) / 255.0;
FxOp::Levels {
ib,
iw: (f(e, "in_white", cx) / 255.0).max(ib + 1e-3),
ob: f(e, "out_black", cx) / 255.0,
ow: f(e, "out_white", cx) / 255.0,
g: 100.0 / f(e, "gamma", cx).max(1.0),
}
}
"extract" => FxOp::Extract {
lo: f(e, "black", cx) / 255.0,
hi: f(e, "white", cx) / 255.0,
soft: f(e, "softness", cx) / 100.0 * 0.2 + 1e-4,
invert: b(e, "invert"),
},
"invert" => FxOp::Invert { channel: choice(e, "channel"), blend: f(e, "blend", cx) / 100.0 },
"posterize" => FxOp::Posterize { n: f(e, "levels", cx).max(2.0) - 1.0 },
"gaussian_blur" | "gaussian_blur_legacy" => {
let r = f(e, "blurriness", cx) * cx.px_scale * 0.5;
let dims = choice(e, "dimensions");
let (rx, ry) = gaussian_boxes(w, h, if dims == 2 { 0.0 } else { r }, if dims == 1 { 0.0 } else { r });
FxOp::Gaussian { rx, ry, repeat: b(e, "repeat_edge") }
}
"camera_blur" => {
let r = f(e, "percent", cx) * cx.px_scale * 0.3;
let (rx, ry) = gaussian_boxes(w, h, r, r);
FxOp::Gaussian { rx, ry, repeat: true }
}
"directional_blur" | "directional_blur_legacy" => {
let len = f(e, "length", cx) * cx.px_scale * 2.0;
let dir = (f(e, "direction", cx) as f64).to_radians();
if len < 0.5 {
FxOp::DirectionalBlur { dx: 0.0, dy: 0.0, steps: 0 }
} else {
let steps = (len.ceil() as usize).clamp(2, 64) as u32;
FxOp::DirectionalBlur { dx: (dir.sin() as f32) * len, dy: (-dir.cos() as f32) * len, steps }
}
}
"sharpen" => {
let r = cx.px_scale.max(0.35);
let (rx, ry) = gaussian_boxes(w, h, r, r);
FxOp::Unsharp { rx, ry, amount: f(e, "amount", cx) / 100.0, threshold: 0.0 }
}
"unsharp_mask" => {
let r = f(e, "radius", cx) * cx.px_scale;
let (rx, ry) = gaussian_boxes(w, h, r, r);
FxOp::Unsharp { rx, ry, amount: f(e, "amount", cx) / 100.0, threshold: f(e, "threshold", cx) / 255.0 }
}
"crop" => {
let l = f(e, "left", cx) / 100.0;
let t = f(e, "top", cx) / 100.0;
let r = f(e, "right", cx) / 100.0;
let bt = f(e, "bottom", cx) / 100.0;
if b(e, "zoom") && l + r < 0.99 && t + bt < 0.99 {
let (fw, fh) = (w as f64, h as f64);
let m = Affine::scale(1.0 / (1.0 - (l + r) as f64), 1.0 / (1.0 - (t + bt) as f64))
.then_apply(&Affine::translate(-(l as f64) * fw, -(t as f64) * fh));
if minifies(&m, w, h) {
// negative crops zoom out: the CPU's mip path (see `transform`)
return Some(FxOp::Resample(Resample { inv: Some(m), rect: [u32::MAX; 4], opacity: 1.0 }));
}
FxOp::Resample(resample(w, h, &m, 1.0))
} else {
FxOp::crop_rect(w, h, l, t, r, bt, f(e, "feather", cx) * cx.px_scale)
}
}
"edge_feather" => {
let amt = f(e, "amount", cx) / 100.0 * (w.min(h) as f32) * 0.5;
FxOp::crop_rect(w, h, 0.0, 0.0, 0.0, 0.0, amt)
}
"transform" => {
let img = Image { w, h, px: Vec::new() };
let anchor = point(e, "anchor", cx, &img);
let pos = point(e, "position", cx, &img);
let sh = f(e, "scale_height", cx) as f64 / 100.0;
let sw = if b(e, "uniform_scale") { sh } else { f(e, "scale_width", cx) as f64 / 100.0 };
let rot = f(e, "rotation", cx) as f64;
let skew = (f(e, "skew", cx) as f64).to_radians().tan();
let skew_axis = f(e, "skew_axis", cx) as f64;
let op = f(e, "opacity", cx) / 100.0;
let sk = Affine::rotate_deg(skew_axis)
.then_apply(&Affine { a: 1.0, b: 0.0, c: skew, d: 1.0, e: 0.0, f: 0.0 })
.then_apply(&Affine::rotate_deg(-skew_axis));
let m = Affine::translate(pos.x, pos.y)
.then_apply(&Affine::rotate_deg(rot))
.then_apply(&sk)
.then_apply(&Affine::scale(sw, sh))
.then_apply(&Affine::translate(-anchor.x, -anchor.y));
if minifies(&m, w, h) {
// the CPU's mip path (`Image::transformed` with the forward map; `rect`
// u32::MAX marks it): CPU only
return Some(FxOp::Resample(Resample { inv: Some(m), rect: [u32::MAX; 4], opacity: op }));
}
FxOp::Resample(resample(w, h, &m, op))
}
"horizontal_flip" => FxOp::HFlip,
"vertical_flip" => FxOp::VFlip,
"mirror" => {
let img = Image { w, h, px: Vec::new() };
let c = point(e, "center", cx, &img);
let ang = (f(e, "angle", cx) as f64).to_radians();
FxOp::Mirror { cx: c.x as f32, cy: c.y as f32, nx: ang.cos() as f32, ny: ang.sin() as f32 }
}
"offset" => {
let img = Image { w, h, px: Vec::new() };
let s = point(e, "shift", cx, &img);
let (dx, dy) = (s.x - w as f64 / 2.0, s.y - h as f64 / 2.0);
FxOp::Offset { dx: dx.rem_euclid(w.max(1) as f64) as f32, dy: dy.rem_euclid(h.max(1) as f64) as f32, blend: f(e, "blend", cx) / 100.0 }
}
"asc_cdl" => {
use crate::vfx::fv;
FxOp::AscCdl {
slope: [fv(e, "r_slope", cx), fv(e, "g_slope", cx), fv(e, "b_slope", cx)],
offset: [fv(e, "r_offset", cx), fv(e, "g_offset", cx), fv(e, "b_offset", cx)],
power: [fv(e, "r_power", cx), fv(e, "g_power", cx), fv(e, "b_power", cx)],
sat: fv(e, "saturation", cx),
}
}
"channel_mix" => {
use crate::vfx::{bv, fv};
let g = |k: &str| fv(e, k, cx) / 100.0;
let mut m = [[g("rr"), g("rg"), g("rb"), g("rc")], [g("gr"), g("gg"), g("gb"), g("gc")], [g("br"), g("bg"), g("bb"), g("bc")]];
if bv(e, "monochrome") {
m = [m[0]; 3];
}
FxOp::ChannelMix { m }
}
"color_replace" => {
use crate::vfx::{bv, cv, fv};
let t = cv(e, "target", cx);
let r = cv(e, "replace", cx);
FxOp::ColorReplace {
sim: fv(e, "similarity", cx) / 100.0 * 1.2,
solid: bv(e, "solid"),
target: [t[0], t[1], t[2]],
replace: [r[0], r[1], r[2]],
replace_hsl: rgb_to_hsl(r[0], r[1], r[2]),
}
}
"alpha_adjust" => {
use crate::vfx::{bv, fv};
FxOp::AlphaAdjust { opacity: fv(e, "opacity", cx) / 100.0, ignore: bv(e, "ignore"), invert: bv(e, "invert"), mask_only: bv(e, "mask_only") }
}
_ => return None,
})
}
fn crop_rect(w: usize, h: usize, l: f32, t: f32, r: f32, b: f32, feather: f32) -> FxOp {
let (w, h) = (w as f32, h as f32);
FxOp::Crop { x0: l * w, x1: w * (1.0 - r), y0: t * h, y1: h * (1.0 - b), feather: feather.max(0.0) }
}
/// Whether the GPU shader reproduces [`apply`](Self::apply) for this op: every number finite
/// (the CPU's NaN / infinity behaviour is left to the CPU) and resamples that the CPU does
/// not pre-filter.
pub fn gpu_ok(&self) -> bool {
let fin = |v: &[f32]| v.iter().all(|x| x.is_finite());
match self {
FxOp::BrightnessContrast { br, co } => fin(&[*br, *co]),
FxOp::ProcAmp { br, co, hue, sat } => fin(&[*br, *co, *hue, *sat]),
FxOp::Tint { black, white, amount } => fin(black) && fin(white) && amount.is_finite(),
FxOp::BlackWhite | FxOp::HFlip | FxOp::VFlip => true,
FxOp::ColorBalance { sh, md, hi, .. } => fin(sh) && fin(md) && fin(hi),
FxOp::LeaveColor { amount, key_hue, tol, soft } => fin(&[*amount, *key_hue, *tol, *soft]),
FxOp::ChangeToColor { from_hue, to_hue, tol, soft } => fin(&[*from_hue, *to_hue, *tol, *soft]),
FxOp::ColorPass { key, sim, .. } => fin(key) && sim.is_finite(),
FxOp::Gamma { g } => g.is_finite(),
FxOp::Levels { ib, iw, ob, ow, g } => fin(&[*ib, *iw, *ob, *ow, *g]),
FxOp::Extract { lo, hi, soft, .. } => fin(&[*lo, *hi, *soft]),
FxOp::Invert { blend, .. } => blend.is_finite(),
FxOp::Posterize { n } => n.is_finite(),
FxOp::Gaussian { .. } => true,
FxOp::DirectionalBlur { dx, dy, .. } => fin(&[*dx, *dy]),
FxOp::Unsharp { amount, threshold, .. } => fin(&[*amount, *threshold]),
FxOp::Crop { x0, x1, y0, y1, feather } => fin(&[*x0, *x1, *y0, *y1, *feather]),
FxOp::Resample(r) => {
r.rect[0] != u32::MAX
&& r.opacity.is_finite()
&& r.inv.as_ref().is_none_or(|m| [m.a, m.b, m.c, m.d, m.e, m.f].iter().all(|v| v.is_finite() && v.abs() < 1e7))
}
FxOp::Mirror { cx, cy, nx, ny } => fin(&[*cx, *cy, *nx, *ny]),
FxOp::Offset { dx, dy, blend } => fin(&[*dx, *dy, *blend]),
FxOp::AscCdl { slope, offset, power, sat } => fin(slope) && fin(offset) && fin(power) && sat.is_finite(),
FxOp::ChannelMix { m } => m.iter().all(|r| fin(r)),
FxOp::ColorReplace { sim, target, replace, replace_hsl, .. } => sim.is_finite() && fin(target) && fin(replace) && fin(replace_hsl),
FxOp::AlphaAdjust { opacity, .. } => opacity.is_finite(),
}
}
/// The CPU reference.
pub fn apply(&self, img: &mut Image) {
if img.w == 0 || img.h == 0 {
return;
}
match self {
FxOp::BrightnessContrast { br, co } => {
let (br, co) = (*br, *co);
img.map_rgb(|c, _, _| {
let c = enc(c);
dec(c.map(|v| (v - 0.5) * co + 0.5 + br))
});
}
FxOp::ProcAmp { br, co, hue, sat } => {
let (br, co, hue, sat) = (*br, *co, *hue, *sat);
img.map_rgb(|c, _, _| {
let c = enc(c);
let mut hsl = rgb_to_hsl(c[0], c[1], c[2]);
hsl[0] = (hsl[0] + hue).rem_euclid(1.0);
hsl[1] = (hsl[1] * sat).clamp(0.0, 1.0);
let c = hsl_to_rgb(hsl[0], hsl[1], hsl[2]);
dec(c.map(|v| (v - 0.5) * co + 0.5 + br))
});
}
FxOp::Tint { black: bl, white: wh, amount } => {
let amt = *amount;
img.map_rgb(|c, _, _| {
let l = linear_to_srgb(luma709(c[0], c[1], c[2]).max(0.0));
let t = [bl[0] + (wh[0] - bl[0]) * l, bl[1] + (wh[1] - bl[1]) * l, bl[2] + (wh[2] - bl[2]) * l];
let e = enc(c);
dec([e[0] + (t[0] - e[0]) * amt, e[1] + (t[1] - e[1]) * amt, e[2] + (t[2] - e[2]) * amt])
});
}
FxOp::BlackWhite => img.map_rgb(|c, _, _| {
let l = luma709(c[0], c[1], c[2]);
[l, l, l]
}),
FxOp::ColorBalance { sh, md, hi, preserve } => {
let preserve = *preserve;
img.map_rgb(|c, _, _| {
let c = enc(c);
let l = 0.2126 * c[0] + 0.7152 * c[1] + 0.0722 * c[2];
let ws = (1.0 - l).powi(2);
let wh = l.powi(2);
let wm = 1.0 - ws - wh;
let mut o = [0.0; 3];
for k in 0..3 {
o[k] = c[k] + sh[k] * ws + md[k] * wm.max(0.0) + hi[k] * wh;
}
if preserve {
let l2 = 0.2126 * o[0] + 0.7152 * o[1] + 0.0722 * o[2];
let d = l - l2;
o = o.map(|v| v + d);
}
dec(o)
});
}
FxOp::LeaveColor { amount, key_hue, tol, soft } => {
let (amt, kh, tol, soft) = (*amount, *key_hue, *tol, *soft);
img.map_rgb(|c, _, _| {
let ec = enc(c);
let h = rgb_to_hsl(ec[0], ec[1], ec[2])[0];
let d = (h - kh).abs().min(1.0 - (h - kh).abs()) * 2.0;
let keep = 1.0 - ((d - tol) / soft).clamp(0.0, 1.0);
let l = luma709(c[0], c[1], c[2]);
let k = amt * (1.0 - keep);
[c[0] + (l - c[0]) * k, c[1] + (l - c[1]) * k, c[2] + (l - c[2]) * k]
});
}
FxOp::ChangeToColor { from_hue: fh, to_hue: th, tol, soft } => {
let (fh, th, tol, soft) = (*fh, *th, *tol, *soft);
img.map_rgb(|c, _, _| {
let ec = enc(c);
let mut hsl = rgb_to_hsl(ec[0], ec[1], ec[2]);
let d = (hsl[0] - fh).abs().min(1.0 - (hsl[0] - fh).abs());
let w = 1.0 - ((d - tol) / soft).clamp(0.0, 1.0);
hsl[0] = (hsl[0] + (th - fh) * w).rem_euclid(1.0);
dec(hsl_to_rgb(hsl[0], hsl[1], hsl[2]))
});
}
FxOp::ColorPass { key, sim, reverse } => {
let (sim, rev) = (*sim, *reverse);
img.map_rgb(|c, _, _| {
let ec = enc(c);
let d = ((ec[0] - key[0]).powi(2) + (ec[1] - key[1]).powi(2) + (ec[2] - key[2]).powi(2)).sqrt();
let pass = (d <= sim * 1.2) != rev;
if pass {
c
} else {
let l = luma709(c[0], c[1], c[2]);
[l, l, l]
}
});
}
FxOp::Gamma { g } => {
let g = *g;
img.map_rgb(|c, _, _| dec(enc(c).map(|v| v.max(0.0).powf(g))));
}
FxOp::Levels { ib, iw, ob, ow, g } => {
let (ib, iw, ob, ow, g) = (*ib, *iw, *ob, *ow, *g);
img.map_rgb(|c, _, _| dec(enc(c).map(|v| ob + (((v - ib) / (iw - ib)).clamp(0.0, 1.0)).powf(g) * (ow - ob))));
}
FxOp::Extract { lo, hi, soft, invert } => {
let (lo, hi, soft, inv) = (*lo, *hi, *soft, *invert);
img.map_rgb(|c, _, _| {
let l = linear_to_srgb(luma709(c[0], c[1], c[2]).max(0.0));
let inside = ((l - lo) / soft).clamp(0.0, 1.0).min(((hi - l) / soft).clamp(0.0, 1.0));
let v = if inv { 1.0 - inside } else { inside };
[v, v, v]
});
}
FxOp::Invert { channel, blend } => {
let (ch, blend) = (*channel, *blend);
if ch == 4 {
img.px.par_chunks_mut(4).for_each(|p| {
let a = p[3];
let na = 1.0 - a;
let k = if a > 1e-6 { na / a } else { 0.0 };
for c in &mut p[..3] {
*c *= k;
}
p[3] = na * (1.0 - blend) + a * blend;
});
} else {
img.map_rgb(|c, _, _| {
let ec = enc(c);
let mut o = ec;
for k in 0..3 {
if ch == 0 || ch as usize == k + 1 {
o[k] = 1.0 - ec[k];
}
}
dec([o[0] + (ec[0] - o[0]) * blend, o[1] + (ec[1] - o[1]) * blend, o[2] + (ec[2] - o[2]) * blend])
});
}
}
FxOp::Posterize { n } => {
let n = *n;
img.map_rgb(|c, _, _| dec(enc(c).map(|v| (v * n).round() / n)));
}
FxOp::Gaussian { rx, ry, repeat } => crate::effects::box_blur(img, rx, ry, *repeat),
FxOp::DirectionalBlur { dx, dy, steps } => crate::effects::directional_taps(img, *dx, *dy, *steps as usize),
FxOp::Unsharp { rx, ry, amount, threshold } => crate::effects::unsharp_boxes(img, rx, ry, *amount, *threshold),
FxOp::Crop { x0, x1, y0, y1, feather } => crate::effects::crop_px(img, *x0, *x1, *y0, *y1, *feather),
FxOp::Resample(r) => {
let out = match (&r.inv, r.rect[0]) {
(Some(m), u32::MAX) => img.transformed(img.w, img.h, m), // mip path (CPU only)
(Some(inv), _) => resample_cpu(img, inv, r.rect),
(None, _) => Image::new(img.w, img.h),
};
*img = out;
img.scale_alpha(r.opacity);
}
FxOp::HFlip => {
let w = img.w;
img.px.par_chunks_mut(w * 4).for_each(|row| {
for x in 0..w / 2 {
for k in 0..4 {
row.swap(x * 4 + k, (w - 1 - x) * 4 + k);
}
}
});
}
FxOp::VFlip => {
let (w, h) = (img.w, img.h);
for y in 0..h / 2 {
let (a, bb) = img.px.split_at_mut((h - 1 - y) * w * 4);
a[y * w * 4..(y + 1) * w * 4].swap_with_slice(&mut bb[..w * 4]);
}
}
FxOp::Mirror { cx, cy, nx, ny } => {
let (c, n) = ((*cx as f64, *cy as f64), (*nx as f64, *ny as f64));
let src = img.clone();
let w = img.w;
img.px.par_chunks_mut(w * 4).enumerate().for_each(|(y, row)| {
for x in 0..w {
let (px, py) = (x as f64 + 0.5, y as f64 + 0.5);
let d = (px - c.0) * n.0 + (py - c.1) * n.1;
if d > 0.0 {
let (rx, ry) = (px - 2.0 * d * n.0, py - 2.0 * d * n.1);
row[x * 4..x * 4 + 4].copy_from_slice(&src.sample_bilinear(rx as f32, ry as f32));
}
}
});
}
FxOp::Offset { dx, dy, blend } => {
let (dx, dy, blend) = (*dx as f64, *dy as f64, *blend);
let src = img.clone();
let (w, h) = (img.w as f64, img.h as f64);
let wi = img.w;
img.px.par_chunks_mut(wi * 4).enumerate().for_each(|(y, row)| {
for x in 0..wi {
let u = (x as f64 + 0.5 - dx).rem_euclid(w);
let v = (y as f64 + 0.5 - dy).rem_euclid(h);
let p = src.sample_bilinear_clamped(u as f32, v as f32);
let o = src.get(x, y);
for k in 0..4 {
row[x * 4 + k] = p[k] + (o[k] - p[k]) * blend;
}
}
});
}
FxOp::AscCdl { slope, offset, power, sat } => {
let (s, o, pw, sat) = (*slope, *offset, *power, *sat);
img.map_rgb(|c, _, _| dec(crate::vfx::cdl(enc(c), s, o, pw, sat)));
}
FxOp::ChannelMix { m } => {
let m = *m;
img.map_rgb(|c, _, _| {
let v = enc(c);
dec([0, 1, 2].map(|k| (m[k][0] * v[0] + m[k][1] * v[1] + m[k][2] * v[2] + m[k][3]).clamp(0.0, 1.0)))
});
}
FxOp::ColorReplace { sim, solid, target: t, replace: r, replace_hsl: rh } => {
let (sim, solid) = (*sim, *solid);
img.map_rgb(|c, _, _| {
let v = enc(c);
let d = ((v[0] - t[0]).powi(2) + (v[1] - t[1]).powi(2) + (v[2] - t[2]).powi(2)).sqrt();
let k = 1.0 - crate::vfx::smoothstep(sim * 0.85, sim.max(1e-4), d);
if k <= 0.0 {
return c;
}
let target = if solid {
[r[0], r[1], r[2]]
} else {
let l = rgb_to_hsl(v[0], v[1], v[2])[2];
hsl_to_rgb(rh[0], rh[1], l)
};
dec(crate::vfx::lerp3(v, target, k))
});
}
FxOp::AlphaAdjust { opacity, ignore, invert, mask_only } => {
let (op, ignore, invert, mask_only) = (*opacity, *ignore, *invert, *mask_only);
img.px.par_chunks_mut(4).for_each(|p| {
let c = Image::unpremul([p[0], p[1], p[2], p[3]]);
let mut a = if ignore { 1.0 } else { p[3] };
if invert {
a = 1.0 - a;
}
a = (a * op).clamp(0.0, 1.0);
if mask_only {
let g = filmcraft_color::srgb_to_linear(a);
p.copy_from_slice(&[g, g, g, 1.0]);
} else {
p.copy_from_slice(&[c[0] * a, c[1] * a, c[2] * a, a]);
}
});
}
}
}
}
/// The body of `Image::transformed` (no mip path) for a precomputed inverse and rectangle.
fn resample_cpu(src: &Image, inv: &Affine, rect: [u32; 4]) -> Image {
let (w, h) = (src.w, src.h);
let mut out = Image::new(w, h);
let [x0, x1, y0, y1] = rect.map(|v| v as usize);
let (x1, y1) = (x1.min(w), y1.min(h));
let (x0, y0) = (x0.min(x1), y0.min(y1));
let axis_aligned = inv.b == 0.0 && inv.c == 0.0;
out.px.par_chunks_mut(w * 4).enumerate().skip(y0).take(y1 - y0).for_each(|(y, row)| {
let py = y as f64 + 0.5;
for x in x0..x1 {
let px = x as f64 + 0.5;
let (u, v) =
if axis_aligned { (inv.a * px + inv.e, inv.d * py + inv.f) } else { (inv.a * px + inv.c * py + inv.e, inv.b * px + inv.d * py + inv.f) };
if u < -1.0 || v < -1.0 || u > w as f64 + 1.0 || v > h as f64 + 1.0 {
continue;
}
row[x * 4..x * 4 + 4].copy_from_slice(&src.sample_bilinear(u as f32, v as f32));
}
});
out
}
+1 -1
View File
@@ -101,7 +101,7 @@ impl Image {
let x0 = (b.x.floor().max(0.0) as usize).min(w);
let x1 = (b.right().ceil().max(0.0) as usize).min(w);
let axis_aligned = m.b == 0.0 && m.c == 0.0;
out.px.par_chunks_mut(w * 4).enumerate().skip(y0).take(y1 - y0).for_each(|(y, row)| {
out.px.par_chunks_mut(w * 4).enumerate().skip(y0).take(y1.saturating_sub(y0)).for_each(|(y, row)| {
let py = y as f64 + 0.5;
for x in x0..x1 {
let px = x as f64 + 0.5;
+1
View File
@@ -15,6 +15,7 @@ pub mod blend;
pub mod color_match;
pub mod colorman;
pub mod effects;
pub mod gpufx;
pub mod graphic_clip;
pub mod graphics;
pub mod image;
+133 -40
View File
@@ -2,10 +2,14 @@
//!
//! [`plan_frame`] resolves what is visible at a time into a list of layers the GPU can draw
//! directly: a decoded source frame (YUV planes or RGBA), the matrix from source pixels to output
//! pixels, and an opacity. Anything the shaders don't cover yet — non-Normal blend modes,
//! standard effects, adjustment layers, nested sequences, non-dissolve transitions — is rendered
//! on the CPU for that layer (or the whole frame) and handed over as a pre-composited image, so
//! the GPU path is always exact with respect to the CPU reference.
//! pixels, an opacity and a blend mode (all 27 of [`Blend`] are composited by the GPU). A media
//! clip whose enabled standard effects all have a GPU implementation ([`crate::gpufx`]) carries
//! them as a [`LayerFx`]: their parameters evaluated at the frame's time, run by the GPU on the
//! clip's working image before Motion places it, in the CPU's order (effects → Motion → Opacity /
//! blend). Anything the shaders don't cover yet — other standard effects, masks, adjustment
//! layers, nested sequences, non-dissolve transitions — is rendered on the CPU for that layer (or
//! the whole frame) and handed over as a pre-composited image, so the GPU path is always exact
//! with respect to the CPU reference.
use std::sync::Arc;
@@ -15,15 +19,46 @@ use filmcraft_media::FrameRequest;
use filmcraft_project::{ItemId, ItemKind, Project, Sequence, TrackItem};
use filmcraft_time::Tick;
use crate::gpufx::FxOp;
use crate::{Blend, RenderOptions, SourceProvider, motion_matrix, output_size};
/// One layer for the GPU, bottom to top.
#[derive(Clone)]
pub struct PlanLayer {
pub frame: Arc<VideoFrame>,
/// Maps frame pixels (0..w, 0..h) to output pixels.
/// Maps frame pixels (0..w, 0..h) to output pixels — or, with [`fx`](Self::fx), working-image
/// pixels (0..fx.size).
pub matrix: Affine,
pub opacity: f32,
/// How the layer combines with what is under it ([`crate::blend::composite`]).
pub blend: Blend,
/// Standard effects to run on the frame before it is placed (None: draw the frame directly).
pub fx: Option<Arc<LayerFx>>,
}
/// The GPU effect stage of a layer: the frame is decoded into a working image of `size` (the
/// frame box-decimated by `decimation`, as the CPU decodes it), `ops` run on it in order, and the
/// result is placed with the layer's matrix, opacity and blend mode.
#[derive(Clone, Debug, PartialEq)]
pub struct LayerFx {
pub size: (u32, u32),
pub decimation: u32,
pub ops: Vec<FxOp>,
}
impl PlanLayer {
/// A layer drawing `frame` directly.
pub fn new(frame: Arc<VideoFrame>, matrix: Affine, opacity: f32, blend: Blend) -> Self {
Self { frame, matrix, opacity, blend, fx: None }
}
/// Size of the picture the matrix places: the working image with effects, else the frame.
pub fn size(&self) -> (u32, u32) {
match &self.fx {
Some(fx) => fx.size,
None => (self.frame.width, self.frame.height),
}
}
}
#[derive(Clone)]
@@ -42,14 +77,29 @@ fn simple_transition(id: &str) -> bool {
matches!(id, "cross_dissolve" | "dip_to_black" | "dip_to_white" | "morph_cut")
}
/// Whether a track item can be drawn by the GPU as-is (no standard effects, Normal blend).
fn gpu_simple(project: &Project, item: &TrackItem, mt: Tick) -> bool {
/// The standard effects of a media clip if the GPU can draw it (any blend mode, no opacity masks,
/// every enabled standard effect unmasked and with a GPU implementation; empty without effects or
/// with effects off). None: the clip is rendered on the CPU.
fn gpu_chain<'a>(project: &Project, item: &'a TrackItem, opts: RenderOptions) -> Option<Vec<&'a filmcraft_project::EffectInstance>> {
let is_media = project.item(item.item).is_some_and(|p| matches!(p.kind, ItemKind::Media(_) | ItemKind::Subclip { .. }));
let no_fx = !item.has_standard_effects() && !item.has_opacity_masks();
let normal =
item.effect("opacity").is_none_or(|e| !e.enabled || e.param("blend").is_none_or(|p| matches!(p.value, filmcraft_project::ParamValue::Choice(0))));
let _ = mt;
is_media && no_fx && normal
if !is_media || item.has_opacity_masks() {
return None;
}
if !opts.effects {
return Some(Vec::new());
}
let mut chain = Vec::new();
for e in item.effects.iter().filter(|e| e.def().is_some_and(|d| !d.intrinsic) && !filmcraft_project::graphic::is_layer(e)) {
if !e.enabled {
continue;
}
let masked = e.masks.iter().any(|m| m.mode != filmcraft_project::MaskMode::None);
if masked || !crate::gpufx::GPU_EFFECTS.contains(&e.effect.as_str()) {
return None;
}
chain.push(e);
}
Some(chain)
}
/// Plan the frame at timeline `t`.
@@ -71,8 +121,7 @@ pub fn plan_frame(project: &Project, seq_id: ItemId, t: Tick, opts: RenderOption
return FramePlan::Image(crate::render_sequence(project, seq_id, t, opts, sources));
}
if let Some(it) = tr.item_at(t)
&& (project.item(it.item).is_some_and(|p| matches!(p.kind, ItemKind::AdjustmentLayer { .. }))
|| crate::opacity_blend(it, it.source_time_at(t)).1 != Blend::Normal)
&& project.item(it.item).is_some_and(|p| matches!(p.kind, ItemKind::AdjustmentLayer { .. }))
{
return FramePlan::Image(crate::render_sequence(project, seq_id, t, opts, sources));
}
@@ -90,23 +139,19 @@ pub fn plan_frame(project: &Project, seq_id: ItemId, t: Tick, opts: RenderOption
match trn.effect.effect.as_str() {
"dip_to_black" | "dip_to_white" => {
let col = if trn.effect.effect == "dip_to_black" { [0.0, 0.0, 0.0, 1.0] } else { [1.0, 1.0, 1.0, 1.0] };
layers.push(PlanLayer {
frame: Arc::new(VideoFrame::rgba_f32(1, 1, col.to_vec())),
matrix: Affine::scale(w as f64, h as f64),
opacity: 1.0,
});
layers.push(PlanLayer::new(Arc::new(VideoFrame::rgba_f32(1, 1, col.to_vec())), Affine::scale(w as f64, h as f64), 1.0, Blend::Normal));
let (it, k) = if p < 0.5 { (a, 1.0 - p * 2.0) } else { (b, (p - 0.5) * 2.0) };
if let Some(it) = it {
push_item(project, seq, it, t, opts, sources, k, &mut layers);
push_item(project, seq, it, t, opts, sources, k, Some(Blend::Normal), &mut layers);
}
}
_ => {
// cross dissolve: A at full, B over it at p (premultiplied over == linear mix when A is opaque)
if let Some(it) = a {
push_item(project, seq, it, t, opts, sources, 1.0 - if b.is_none() { p } else { 0.0 }, &mut layers);
push_item(project, seq, it, t, opts, sources, 1.0 - if b.is_none() { p } else { 0.0 }, Some(Blend::Normal), &mut layers);
}
if let Some(it) = b {
push_item(project, seq, it, t, opts, sources, p, &mut layers);
push_item(project, seq, it, t, opts, sources, p, Some(Blend::Normal), &mut layers);
}
}
}
@@ -116,20 +161,23 @@ pub fn plan_frame(project: &Project, seq_id: ItemId, t: Tick, opts: RenderOption
if !item.enabled {
continue;
}
push_item(project, seq, item, t, opts, sources, 1.0, &mut layers);
push_item(project, seq, item, t, opts, sources, 1.0, None, &mut layers);
}
if opts.captions {
for o in crate::caption_overlays(seq, t, w, h) {
layers.push(PlanLayer {
frame: Arc::new(VideoFrame::rgba_f32(o.w as u32, o.h as u32, o.px)),
matrix: Affine::translate(o.x as f64, o.y as f64),
opacity: 1.0,
});
layers.push(PlanLayer::new(
Arc::new(VideoFrame::rgba_f32(o.w as u32, o.h as u32, o.px)),
Affine::translate(o.x as f64, o.y as f64),
1.0,
Blend::Normal,
));
}
}
FramePlan::Layers { width: w, height: h, layers }
}
/// Push the layer(s) of `item`. `blend` overrides the item's own blend mode: inside a transition
/// the CPU reference mixes the clips and composites the result Normal, ignoring their modes.
#[allow(clippy::too_many_arguments)]
fn push_item(
project: &Project,
@@ -139,32 +187,35 @@ fn push_item(
opts: RenderOptions,
sources: &dyn SourceProvider,
extra_opacity: f32,
blend: Option<Blend>,
out: &mut Vec<PlanLayer>,
) {
// frame time (`ft`) vs. effect time (`mt`): they differ inside a frame hold without Hold Filters
let ft = item.source_time_at(t);
let mt = item.effect_time_at(t);
let (op, bl) = crate::opacity_blend(item, mt);
let (op, own) = crate::opacity_blend(item, mt);
let bl = blend.unwrap_or(own);
// A multi-camera clip that only shows its angle (no effects, untransformed, same frame size)
// draws the angle's clip directly: no CPU pass over the nested sequence.
if let Some(ItemKind::Sequence(nested)) = project.item(item.item).map(|p| &p.kind)
&& let Some(angle) = item.multicam_angle(nested)
&& bl == Blend::Normal
&& !(opts.effects && item.has_standard_effects())
&& (nested.settings.width, nested.settings.height) == (seq.settings.width, seq.settings.height)
&& near_identity(&motion_matrix(seq, item, (nested.settings.width, nested.settings.height), mt))
&& let Some(tr) = nested.angle_video_track_index(angle).and_then(|i| nested.video_tracks.get(i))
&& !tr.transitions.iter().any(|x| x.range().contains(ft))
&& tr.item_at(ft).is_none_or(|i| crate::opacity_blend(i, i.effect_time_at(ft)).1 != Blend::Dissolve)
{
// Inside the nested sequence the angle's clip is composited onto an empty canvas, where
// every mode but Dissolve is Normal; the multicam clip's own mode then applies to it.
if let Some(inner) = tr.item_at(ft).filter(|i| i.enabled) {
push_item(project, nested, inner, ft, opts, sources, extra_opacity * op, out);
push_item(project, nested, inner, ft, opts, sources, extra_opacity * op, Some(bl), out);
}
return;
}
// Graphic clips without standard effects: the layers are rasterised (cached) into one tight
// image the GPU places as a layer.
if bl == Blend::Normal
&& !(opts.effects && item.has_standard_effects())
if !(opts.effects && item.has_standard_effects())
&& !item.has_opacity_masks()
&& project.item(item.item).is_some_and(|p| matches!(p.kind, ItemKind::Graphic { .. }))
{
@@ -172,11 +223,11 @@ fn push_item(
let (w, h) = output_size(seq, opts.scale);
let m = Affine::scale(opts.scale as f64, opts.scale as f64).then_apply(&motion_matrix(seq, item, size, mt));
if let Some((img, x, y)) = crate::graphic_clip::render_graphic_tight(item, mt, size, &m, w, h) {
out.push(PlanLayer { frame: cpu_frame(img), matrix: Affine::translate(x as f64, y as f64), opacity: op * extra_opacity });
out.push(PlanLayer::new(cpu_frame(img), Affine::translate(x as f64, y as f64), op * extra_opacity, bl));
}
return;
}
if gpu_simple(project, item, mt) && bl == Blend::Normal {
if let Some(chain) = gpu_chain(project, item, opts) {
let Some(src) = sources.source(item.item) else { return };
let Some(size) = crate::source_size(project, item.item) else { return };
let motion = motion_matrix(seq, item, size, mt);
@@ -186,7 +237,35 @@ fn push_item(
let cs = crate::colorman::source_space(project, item.item, &frame);
// log / HDR / wide-gamut media is converted on the CPU (below), and so are blended
// in-between frames (Frame Blending / Optical Flow on speed-changed clips)
if !crate::colorman::needs_management(&seq.settings.color, cs, &frame) && crate::interpolation_blend(item, t, src.info().frame_rate()).is_none() {
let plain =
!crate::colorman::needs_management(&seq.settings.color, cs, &frame) && crate::interpolation_blend(item, t, src.info().frame_rate()).is_none();
if plain && !chain.is_empty() {
// GPU effect stage: the working image the CPU would decode (`base_layer`), the
// effects evaluated for it, placed as `item_layer` places it
let n = crate::decimation(frame.width as f32, size.0 as f32 * want);
let (lw, lh) = ((frame.width as usize / n).max(1), (frame.height as usize / n).max(1));
let px_scale = lw as f32 / size.0.max(1) as f32;
let tc = "";
let cx = crate::effects::FxCtx {
t: mt,
px_scale,
seconds: (t - item.start).seconds(),
timecode: tc,
clip_name: &item.name,
project: Some(project),
env: None,
working: seq.settings.color.working,
};
let ops: Option<Vec<FxOp>> = chain.iter().map(|e| FxOp::eval(e, &cx, lw, lh).filter(FxOp::gpu_ok)).collect();
if let Some(ops) = ops {
let m = Affine::scale(opts.scale as f64, opts.scale as f64)
.then_apply(&motion)
.then_apply(&Affine::scale(1.0 / px_scale as f64, 1.0 / px_scale as f64));
let fx = LayerFx { size: (lw as u32, lh as u32), decimation: n as u32, ops };
out.push(PlanLayer { frame, matrix: m, opacity: op * extra_opacity, blend: bl, fx: Some(Arc::new(fx)) });
return;
}
} else if plain {
// Draft playback at reduced resolution: hand over planes box-filtered to the size
// drawn instead of the full picture (a quarter at 1/2, a sixteenth at 1/4 of the
// upload and sampling). The mean is taken over Y'CbCr codes, not linear light as
@@ -198,14 +277,14 @@ fn push_item(
};
let px_scale = frame.width as f64 / size.0.max(1) as f64;
let m = Affine::scale(opts.scale as f64, opts.scale as f64).then_apply(&motion).then_apply(&Affine::scale(1.0 / px_scale, 1.0 / px_scale));
out.push(PlanLayer { frame, matrix: m, opacity: op * extra_opacity });
out.push(PlanLayer::new(frame, m, op * extra_opacity, bl));
return;
}
}
// CPU-rendered layer (standard effects): drawn by the GPU as a pre-rendered canvas image.
let tc = filmcraft_time::format_time(t, seq.settings.frame_rate, seq.settings.drop_frame, filmcraft_time::TimeDisplay::Timecode, 48_000);
if let Some((img, op2, _)) = crate::item_layer(project, seq, item, t, opts, sources, &tc) {
out.push(PlanLayer { frame: cpu_frame(img), matrix: Affine::IDENTITY, opacity: op2 * extra_opacity });
out.push(PlanLayer::new(cpu_frame(img), Affine::IDENTITY, op2 * extra_opacity, bl));
}
}
@@ -213,6 +292,17 @@ fn near_identity(m: &Affine) -> bool {
(m.a - 1.0).abs() < 1e-9 && (m.d - 1.0).abs() < 1e-9 && m.b.abs() < 1e-9 && m.c.abs() < 1e-9 && m.e.abs() < 1e-6 && m.f.abs() < 1e-6
}
/// A layer's working image with its effects applied, on the CPU (the reference for the GPU
/// effect stage).
pub fn effect_image(frame: &VideoFrame, fx: &LayerFx) -> crate::Image {
let (w, h, px) = frame.to_linear_f32_decimated(fx.decimation.max(1) as usize);
let mut img = crate::Image { w, h, px };
for op in &fx.ops {
op.apply(&mut img);
}
img
}
/// Execute a plan on the CPU (reference for the GPU compositor).
pub fn execute_cpu(plan: &FramePlan) -> crate::Image {
match plan {
@@ -220,13 +310,16 @@ pub fn execute_cpu(plan: &FramePlan) -> crate::Image {
FramePlan::Layers { width, height, layers } => {
let mut canvas = crate::Image::new(*width, *height);
for l in layers {
let src = crate::Image { w: l.frame.width as usize, h: l.frame.height as usize, px: l.frame.to_linear_f32() };
let src = match &l.fx {
Some(fx) => effect_image(&l.frame, fx),
None => crate::Image { w: l.frame.width as usize, h: l.frame.height as usize, px: l.frame.to_linear_f32() },
};
let placed = if l.matrix == Affine::scale(*width as f64, *height as f64) && src.w == 1 && src.h == 1 {
crate::Image::filled(*width, *height, src.get(0, 0))
} else {
src.transformed(*width, *height, &l.matrix)
};
crate::blend::composite(&mut canvas, &placed, l.opacity, Blend::Normal);
crate::blend::composite(&mut canvas, &placed, l.opacity, l.blend);
}
canvas
}
+2 -68
View File
@@ -99,16 +99,8 @@ pub fn lighting(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
});
}
/// ASC CDL: out = clamp(in × slope + offset)^power per channel, then saturation (Rec. 709 luma),
/// on display-encoded values (ASC CDL v1.2).
pub fn asc_cdl(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
let s = [fv(e, "r_slope", cx), fv(e, "g_slope", cx), fv(e, "b_slope", cx)];
let o = [fv(e, "r_offset", cx), fv(e, "g_offset", cx), fv(e, "b_offset", cx)];
let pw = [fv(e, "r_power", cx), fv(e, "g_power", cx), fv(e, "b_power", cx)];
let sat = fv(e, "saturation", cx);
img.map_rgb(|c, _, _| dec(cdl(enc(c), s, o, pw, sat)));
}
/// ASC CDL (v1.2): clamp(in × slope + offset)^power per channel, then saturation (Rec. 709 luma),
/// on display-encoded values (`gpufx::FxOp::AscCdl`).
pub(crate) fn cdl(c: [f32; 3], s: [f32; 3], o: [f32; 3], p: [f32; 3], sat: f32) -> [f32; 3] {
let mut v = [0.0; 3];
for k in 0..3 {
@@ -225,64 +217,6 @@ pub fn vignette(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
});
}
/// Channel Mix: each output channel = Σ input × weight / 100 + constant / 100 (display-encoded).
pub fn channel_mix(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
let g = |k: &str| fv(e, k, cx) / 100.0;
let mut m = [[g("rr"), g("rg"), g("rb"), g("rc")], [g("gr"), g("gg"), g("gb"), g("gc")], [g("br"), g("bg"), g("bb"), g("bc")]];
if bv(e, "monochrome") {
m = [m[0]; 3];
}
img.map_rgb(|c, _, _| {
let v = enc(c);
dec([0, 1, 2].map(|k| (m[k][0] * v[0] + m[k][1] * v[1] + m[k][2] * v[2] + m[k][3]).clamp(0.0, 1.0)))
});
}
/// Color Replace: pixels within Similarity of Target take the Replace colour (keeping their
/// lightness unless Solid Colors).
pub fn color_replace(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
let sim = fv(e, "similarity", cx) / 100.0 * 1.2;
let solid = bv(e, "solid");
let t = cv(e, "target", cx);
let r = cv(e, "replace", cx);
let rh = rgb_to_hsl(r[0], r[1], r[2]);
img.map_rgb(|c, _, _| {
let v = enc(c);
let d = ((v[0] - t[0]).powi(2) + (v[1] - t[1]).powi(2) + (v[2] - t[2]).powi(2)).sqrt();
let k = 1.0 - smoothstep(sim * 0.85, sim.max(1e-4), d);
if k <= 0.0 {
return c;
}
let target = if solid {
[r[0], r[1], r[2]]
} else {
let l = rgb_to_hsl(v[0], v[1], v[2])[2];
hsl_to_rgb(rh[0], rh[1], l)
};
dec(lerp3(v, target, k))
});
}
/// Alpha Adjust: opacity, ignore / invert alpha, show the matte only.
pub fn alpha_adjust(img: &mut Image, e: &EffectInstance, cx: &FxCtx) {
let op = fv(e, "opacity", cx) / 100.0;
let (ignore, invert, mask_only) = (bv(e, "ignore"), bv(e, "invert"), bv(e, "mask_only"));
img.px.par_chunks_mut(4).for_each(|p| {
let c = Image::unpremul([p[0], p[1], p[2], p[3]]);
let mut a = if ignore { 1.0 } else { p[3] };
if invert {
a = 1.0 - a;
}
a = (a * op).clamp(0.0, 1.0);
if mask_only {
let g = filmcraft_color::srgb_to_linear(a);
p.copy_from_slice(&[g, g, g, 1.0]);
} else {
p.copy_from_slice(&[c[0] * a, c[1] * a, c[2] * a, a]);
}
});
}
/// Logo Cutout: removes a flat white/black/custom background by un-multiplying it — the alpha of
/// a pixel is how far it is from the background towards the gamut edge, and the colour is
/// recovered as if it had been composited over that background.
+1 -4
View File
@@ -13,6 +13,7 @@
mod blur;
mod color;
pub(crate) use color::cdl;
mod distort;
mod immersive;
mod lights;
@@ -459,12 +460,8 @@ pub fn apply(img: &mut Image, e: &EffectInstance, cx: &FxCtx) -> bool {
match e.effect.as_str() {
// Adjust / Color / Image Control / Keying / Utility colour
"lighting_effects" => color::lighting(img, e, cx),
"asc_cdl" => color::asc_cdl(img, e, cx),
"video_limiter" => color::video_limiter(img, e, cx),
"vignette" => color::vignette(img, e, cx),
"channel_mix" => color::channel_mix(img, e, cx),
"color_replace" => color::color_replace(img, e, cx),
"alpha_adjust" => color::alpha_adjust(img, e, cx),
"logo_cutout" => color::logo_cutout(img, e, cx),
"ultra_key" => color::ultra_key(img, e, cx),
"track_matte" => color::track_matte(img, e, cx),
+3
View File
@@ -40,5 +40,8 @@ rustix = { version = "1", default-features = false, features = ["std", "time"] }
[dev-dependencies]
pollster = "0.4"
filmcraft-codecs = { workspace = true }
# the bench registers OS hardware decoders as the desktop app does (dev-dependency only: ui-egui
# itself stays OS-independent and web-portable)
filmcraft-platform = { workspace = true }
filmcraft-interchange = { workspace = true }
egui_kittest = { workspace = true, features = ["eframe", "wgpu"] }
+10 -2
View File
@@ -5,6 +5,7 @@
//! ```sh
//! cargo xtask bench # every section, writes target/bench/bench-<label>.{json,md}
//! cargo xtask bench --sections decode,scrub --repeat 3 --label after
//! cargo xtask bench --sections decode --hw off --label hw-off # FilmCraft's own decoders only
//! cargo run --release -p filmcraft-ui-egui --example bench -- --section timeline # one section, this process
//! ```
//!
@@ -33,6 +34,8 @@ pub struct Opts {
pub gpu: bool,
/// Only cases whose name contains this (decode fixtures, playback / scrub scenarios).
pub only: Option<String>,
/// Settings ▸ Playback ▸ Hardware decoding: `auto` (default, as the app) or `off`.
pub hw: String,
}
impl Opts {
@@ -49,7 +52,7 @@ fn main() {
let args: Vec<String> = std::env::args().skip(1).collect();
if args.iter().any(|a| a == "--help" || a == "-h") {
println!(
"bench [--sections {}] [--repeat N] [--quick] [--cpu] [--only SUBSTRING] [--label NAME] [--out DIR]\n bench --section NAME [--json FILE] (one section in this process)",
"bench [--sections {}] [--repeat N] [--quick] [--cpu] [--only SUBSTRING] [--hw auto|off] [--label NAME] [--out DIR]\n bench --section NAME [--json FILE] (one section in this process)",
SECTIONS.join(",")
);
return;
@@ -59,7 +62,11 @@ fn main() {
quick: args.iter().any(|a| a == "--quick"),
gpu: !args.iter().any(|a| a == "--cpu"),
only: arg_value(&args, "--only"),
hw: arg_value(&args, "--hw").unwrap_or_else(|| "auto".into()),
};
// as the desktop app: OS hardware decoders in front of ours, unless Hardware decoding is Off
filmcraft_platform::register();
filmcraft_codecs::hw::set_hardware_decoding(opts.hw != "off");
if let Some(section) = arg_value(&args, "--section") {
let v = run_section(&section, &opts);
let text = serde_json::to_string_pretty(&v).unwrap_or_default();
@@ -113,6 +120,7 @@ fn orchestrate(args: &[String], o: &Opts) {
if let Some(only) = &o.only {
cmd.args(["--only", only]);
}
cmd.args(["--hw", &o.hw]);
eprintln!("== {name} (load {})", playback::load_avg());
let output = cmd.stderr(std::process::Stdio::piped()).output().expect("run section");
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
@@ -131,7 +139,7 @@ fn orchestrate(args: &[String], o: &Opts) {
println!("{}", section_markdown(&v));
results.push(v);
}
let doc = json!({"label": label, "machine": machine, "repeat": o.repeat, "quick": o.quick, "gpu": o.gpu, "sections": results});
let doc = json!({"label": label, "machine": machine, "repeat": o.repeat, "quick": o.quick, "gpu": o.gpu, "hw": o.hw, "sections": results});
let jp = out.join(format!("bench-{label}.json"));
let mp = out.join(format!("bench-{label}.md"));
std::fs::write(&jp, serde_json::to_string_pretty(&doc).unwrap_or_default()).expect("write json");
+49 -5
View File
@@ -626,7 +626,7 @@ fn parse_args() -> Args {
"--json" => a.json = it.next(),
"--help" | "-h" => {
println!(
"bench_playback [--scenario h264-1080,stack3,h264-2160,hevc-2160,vp9-2160,av1-2160,demo,after-preview,seek-storm] [--res full,half] [--cpu|--gpu] [--seconds 8] [--refresh 60] [--workers N] [--repeat N] [--json out.json]"
"bench_playback [--scenario h264-1080,stack3,stack3-blend,stack3-fx,h264-2160,hevc-2160,vp9-2160,av1-2160,demo,after-preview,seek-storm] [--res full,half] [--cpu|--gpu] [--seconds 8] [--refresh 60] [--workers N] [--repeat N] [--json out.json]"
);
std::process::exit(0);
}
@@ -634,7 +634,10 @@ fn parse_args() -> Args {
}
}
if a.scenarios.is_empty() {
a.scenarios = ["h264-1080", "stack3", "h264-2160", "hevc-2160", "demo", "after-preview", "seek-storm"].iter().map(|s| s.to_string()).collect();
a.scenarios = ["h264-1080", "stack3", "stack3-blend", "stack3-fx", "h264-2160", "hevc-2160", "demo", "after-preview", "seek-storm"]
.iter()
.map(|s| s.to_string())
.collect();
}
a
}
@@ -647,17 +650,58 @@ pub fn scenario_session(name: &str) -> Option<(Session, ItemId)> {
let a = import(&mut s, &fixture("a1080.mp4")?);
build_sequence(&mut s, 1920, 1080, &[(a, 100.0, None, 0.0, 100.0)], &[], 20.0)
}
"stack3" => {
"stack3" | "stack3-blend" | "stack3-fx" => {
let (a, b, c) = (fixture("a1080.mp4")?, fixture("b1080.mp4")?, fixture("c1080.mp4")?);
let (a, b, c) = (import(&mut s, &a), import(&mut s, &b), import(&mut s, &c));
build_sequence(
let seq = build_sequence(
&mut s,
1920,
1080,
&[(a, 100.0, None, 0.0, 100.0), (b, 60.0, Some((700.0, 420.0)), 0.0, 70.0), (c, 40.0, Some((1450.0, 760.0)), 12.0, 85.0)],
&[],
20.0,
)
);
if name == "stack3-blend" {
// V2 in Screen, V3 in Overlay
s.edit("blend modes", |p, _| {
let q = p.sequence_mut(seq).expect("seq");
for (track, mode) in [(1, "Screen"), (2, "Overlay")] {
let i = filmcraft_project::effect::BLEND_MODES.iter().position(|b| *b == mode).expect("blend mode") as u32;
for it in &mut q.video_tracks[track].items {
if let Some(b) = it.effect_mut("opacity").and_then(|o| o.params.get_mut("blend")) {
b.value = ParamValue::Choice(i);
}
}
}
Ok(())
})
.ok()?;
}
if name == "stack3-fx" {
// V2: Brightness & Contrast + Gaussian Blur; V3: Tint
s.edit("effects", |p, _| {
let q = p.sequence_mut(seq).expect("seq");
let fx: [(usize, &str, &[(&str, f64)]); 3] = [
(1, "brightness_contrast", &[("brightness", 12.0), ("contrast", 25.0)]),
(1, "gaussian_blur", &[("blurriness", 8.0)]),
(2, "tint", &[("amount", 70.0)]),
];
for (track, id, params) in fx {
for it in &mut q.video_tracks[track].items {
let mut e = find_effect(id).expect("effect").instance();
for (k, v) in params {
if let Some(x) = e.params.get_mut(*k) {
x.value = ParamValue::Float(*v);
}
}
it.effects.push(e);
}
}
Ok(())
})
.ok()?;
}
seq
}
"h264-2160" | "seek-storm-2160" => {
let a = import(&mut s, &fixture("a2160.mp4")?);
@@ -87,6 +87,7 @@ pub fn decode(o: &Opts) -> Vec<Value> {
"fps": best, "fps_min": fps.iter().cloned().fold(f64::MAX, f64::min), "fps_runs": fps,
"cpu_ms_per_frame": median(&cpu), "first_frame_ms": median(&first), "realtime": best / rate.as_f64(),
"decode_threads": rayon::current_num_threads(), "load": load_avg(),
"hw_frames": filmcraft_codecs::hw::hw_stats().frames,
});
eprintln!("decode {name}: {best:.1} fps");
rows.push(row);
@@ -160,6 +161,8 @@ pub fn playback(o: &Opts) -> Vec<Value> {
let cases: &[(&str, &str, f32, bool)] = &[
("h264-1080", "full", 1.0, false),
("stack3", "full", 1.0, false),
("stack3-blend", "full", 1.0, false),
("stack3-fx", "full", 1.0, false),
("h264-2160", "full", 1.0, false),
("h264-2160", "half", 0.5, false),
("h264-2160", "half draft", 0.5, true),
+3 -1
View File
@@ -883,12 +883,14 @@ fn plan_job(job: &Job, seq: ItemId, pool: &Arc<MediaPool>, services: &Arc<dyn Se
// captions (they are not part of the preview hash), so captions are layered on live.
let (w, h) = filmcraft_render::output_size(q, job.scale);
let matrix = filmcraft_geom::Affine::scale(w as f64 / frame.width.max(1) as f64, h as f64 / frame.height.max(1) as f64);
let mut layers = vec![filmcraft_render::plan::PlanLayer { frame, matrix, opacity: 1.0 }];
let mut layers = vec![filmcraft_render::plan::PlanLayer { frame, matrix, opacity: 1.0, blend: filmcraft_render::Blend::Normal, fx: None }];
for o in filmcraft_render::caption_overlays(q, job.time, w, h) {
layers.push(filmcraft_render::plan::PlanLayer {
frame: Arc::new(filmcraft_frame::VideoFrame::rgba_f32(o.w as u32, o.h as u32, o.px)),
matrix: filmcraft_geom::Affine::translate(o.x as f64, o.y as f64),
opacity: 1.0,
blend: filmcraft_render::Blend::Normal,
fx: None,
});
}
return (filmcraft_render::plan::FramePlan::Layers { width: w, height: h, layers }, true);
+2 -1
View File
@@ -1314,7 +1314,8 @@ mod gpu_fallback_tests {
#[test]
fn plan_side_covers_output_and_every_layer() {
let frame = |w: u32, h: u32| std::sync::Arc::new(filmcraft_frame::VideoFrame::rgba_f32(w, h, vec![0.0; (w * h * 4) as usize]));
let layer = |w, h| PlanLayer { frame: frame(w, h), matrix: filmcraft_geom::Affine::IDENTITY, opacity: 1.0 };
let layer =
|w, h| PlanLayer { frame: frame(w, h), matrix: filmcraft_geom::Affine::IDENTITY, opacity: 1.0, blend: filmcraft_render::Blend::Normal, fx: None };
// a 4K source in an HD sequence still needs a 3840-wide texture on the GPU
let p = FramePlan::Layers { width: 1920, height: 1080, layers: vec![layer(64, 64), layer(3840, 2160)] };
assert_eq!(super::plan_side(&p), 3840);
+65
View File
@@ -0,0 +1,65 @@
# ADR 0001: `unsafe` OS media FFI in `crates/platform`, and nowhere else
- **Status:** accepted (2026-10-05, approved by the project owner; recorded in
[AGENTS.md](../../AGENTS.md) §0 item 3 and the craftrules never-crash standard)
- **Issue:** #30 (hardware acceleration)
## Context
FilmCraft is pure Rust and the workspace sets `unsafe_code = "forbid"`. Every codec has a
clean-room software decoder, and those remain the reference. But decoding 4K H.264 / HEVC in
software costs 85–120 ms of CPU per frame on an M4 Pro, so 4K playback drops frames under load,
while every current computer has a hardware video decoder the operating system exposes
(VideoToolbox on macOS; Media Foundation / D3D11 on Windows and VA-API on Linux later).
Using it needs FFI into C / Objective-C APIs, which is `unsafe` Rust. The bindings we use (the
`objc2-*` crates, MIT / Apache-2.0 / Zlib) are generated declarations; nothing GPL/LGPL is linked.
## Decision
Allow `unsafe` in exactly one crate, `crates/platform` (`filmcraft-platform`, layer L5), for one
job: OS media FFI (hardware decoding now, hardware encoding later). Audio output, dialogs, menus
and other OS integration do not go in it.
Containment rules:
1. The crate does not use `lints.workspace = true`. Its own `[lints]` table copies the workspace
lints except `unsafe_code = "deny"` (not `forbid`), and adds
`clippy::undocumented_unsafe_blocks = "deny"`. Only the FFI modules (`videotoolbox`) carry
`#[allow(unsafe_code)]`; the rest of the crate (the fallback logic in `hybrid`) has no `unsafe`.
2. Every `unsafe` block has a `// SAFETY:` comment saying why it is sound.
3. The public API is safe: no `pub unsafe fn`, no raw pointers or FFI types in public signatures;
failures are `Result`s. The crate keeps the never-crash `deny(clippy::unwrap_used, …)`
attribute of every clean crate.
4. No panic crosses the FFI boundary: the VideoToolbox output callback runs under `catch_unwind`
and reports a panic as a failed frame.
5. The crate compiles on every target. OS bindings are target-specific dependencies; elsewhere
`register()` reports `Unavailable` and does nothing. It is not part of the wasm check (L5).
## The fallback guarantee
Registering a hardware decoder never makes a file undecodable, and never changes a picture:
- The factory declines (the software decoder is used) when Settings ▸ Playback ▸ Hardware
decoding is Off, for formats the hardware path does not take (field-coded H.264, bit depths
other than 8 / 10, 4:4:4, mismatched luma / chroma depth), and when the OS cannot create a
hardware session for the stream (hardware decoding is *required*, so the OS's own software
decoder is never used instead of ours).
- A hardware decoder that fails mid-stream (decode error, session invalidated by a GPU change or
sleep, in-band parameter sets that differ from the sample entry) becomes the software decoder
for that stream without the caller noticing: `HybridDecoder` replays the samples since the last
restart point through it and drops pictures already returned. It is logged and counted
(`perf.stats` `decode.hardware.fallbacks`).
- Pictures are interchangeable with the software decoder's: same planes (bit-exact on the H.264
High, HEVC Main and Main 10 parity fixtures, including after seeks and flushes), colour, pixel
aspect, pts and presentation order. Tests compare the two on every run on macOS, and the
fallback logic is tested on every platform with a stand-in hardware decoder.
## Consequences
- 4K H.264 / HEVC decode costs ~4 ms of CPU per frame instead of 85–120 ms
([performance.md](../performance.md)).
- Better patent posture for the most encumbered formats: H.264 / HEVC decoding is done by the
operating system, which carries the licences, wherever hardware decoding is used.
- A new review burden: changes to `crates/platform` need the same scrutiny as any FFI code.
Anything else that wants `unsafe` needs a new decision, not an extension of this one.
+17 -2
View File
@@ -140,7 +140,8 @@ media, sidecar and `ATTRIBUTION.md` entry.
1. [AGENTS.md](../AGENTS.md): absolute rules (assets, clean-room, licences).
2. [CLAUDE.md](../CLAUDE.md): working instructions and non-negotiables.
3. [ROADMAP.md](../ROADMAP.md): milestones, what's done, what's running.
3. [ROADMAP.md](../ROADMAP.md): read the **honest assessment** and **Where we are lacking** first, then
milestones, what's done and what's running.
4. [architecture.md](architecture.md), then the README and tests of the crate you'll touch.
5. [contributing.md](contributing.md) (how to add things, gates) and [testing.md](testing.md).
@@ -151,7 +152,9 @@ everything you need to contribute is in the public docs above. Ask a maintainer
## 5. Autonomous work loop
1. **Orient.** Pick the next task: the next unchecked task in the maintainer status file, or an open
ROADMAP item. Read the relevant architecture section and crate README.
ROADMAP item. Prefer the gaps in ROADMAP's *Where we are lacking* (speed, correctness on real
media, measurement, Windows/Linux) over adding more checklist items. Read the relevant
architecture section and crate README.
2. **Plan tests first.** Write down the acceptance test before writing code.
3. **Implement and test.**
4. **Verify.** Run all gates (`cargo xtask ci`). For UI work, run the app with `--control`, drive it
@@ -190,3 +193,15 @@ A feature is done when:
- [ ] all gates pass (`cargo xtask ci`), and new assets have sidecars;
- [ ] the crate README and ROADMAP.md are updated where relevant;
- [ ] it is committed with its task id.
### Reporting progress honestly
ROADMAP.md keeps two numbers: the **feature checklist** (does it exist?) and **ready for real
work** (does it hold up?). When you update either:
- say how a number was obtained: *measured* (a test, a diff, a benchmark you ran) or *estimated*;
- count an approximation, a stub or a partly wired setting as such, never as done;
- record known bugs and limitations in the crate README and the honest assessment, not only
what works;
- benchmarks state the machine, the load average and before/after on the same build;
- don't raise a percentage without evidence a reviewer can rerun.
+45 -8
View File
@@ -21,7 +21,7 @@ Design principles:
```text
L6 apps/filmcraft · apps/filmcraft-cli · apps/filmcraft-web
L5 ui-egui · automation
L5 ui-egui · automation · platform
L4 engine
L3 render · gpu · export · golden (test-only)
L2 edit · codecs · interchange · captions · speech
@@ -71,6 +71,7 @@ and `filmcraft-cli`.
| `engine` | L4 | `Session`, command registry, undo history, media pool, jobs, interchange glue |
| `ui-egui` | L5 | the egui frontend: docking, panels, timeline, monitors, playback, control-channel handlers |
| `automation` | L5 | MCP server (`rmcp`, stdio), headless or bridged to the running app |
| `platform` | L5 | OS media FFI only: hardware video decoding (VideoToolbox H.264 / HEVC on macOS; a no-op elsewhere) behind `codecs::VideoDecoder`, with transparent fallback to our decoders. The one crate allowed `unsafe` ([ADR 0001](adr/0001-platform-ffi.md), [README](../crates/platform/README.md)) |
| `filmcraft` | L6 | desktop binary: eframe/wgpu window, cpal audio output, file dialogs, native macOS menu, TCP control server |
| `filmcraft-cli` | L6 | headless CLI: `exec`, `run`, `inspect`, `describe`, `commands`, `import`, `export`, `render`, `probe`, `mcp`; `--bridge` targets the running app |
| `filmcraft-web` | L6 | the browser app (wasm32): eframe web runner on WebGPU/WebGL2, Blob-backed services, OPFS recovery, WebAudio, WebCodecs, `window.filmcraft` API ([web.md](web.md)) |
@@ -91,7 +92,8 @@ are exempt):
`cargo xtask wasm` runs `cargo check --target wasm32-unknown-unknown` on every L0–L4 crate, the
egui UI and the web app, so everything up to the engine stays web-portable and the web app builds
([web.md](web.md)). `unsafe_code = "deny"` applies workspace-wide.
([web.md](web.md)). `unsafe_code = "forbid"` applies workspace-wide, except in `platform`
(`deny`, allowed only on its FFI modules; [ADR 0001](adr/0001-platform-ffi.md)).
## 2. Time base
@@ -196,7 +198,8 @@ pub struct CommandSpec {
```text
file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware seek
│ decoder registry: h264, hevc, vp9, av1, prores, dnx, mjpeg (+ any registered first)
│ decoder registry: h264, hevc, vp9, av1, prores, dnx, mjpeg (+ any registered first:
│ platform's VideoToolbox H.264 / HEVC on macOS, falling back to ours)
▼
media::MediaSource ──► frame cache (byte-budgeted LRU, shared)
▼
@@ -206,7 +209,7 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
in linear-light premultiplied f32
│
└─ render::plan::plan_frame → gpu::GpuCompositor GPU path
layers = decoded YUV/RGBA frames + matrix + opacity;
layers = decoded YUV/RGBA frames + matrix + opacity + blend mode;
anything the shaders don't cover is pre-rendered on the CPU
▼
ui-egui frames.rs worker pool ──► monitors (program/source), thumbnails, prefetch
@@ -225,13 +228,46 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
every frame of the GOP. Sequential playback reuses the decoder. Decoders implement
`codecs::VideoDecoder`. `register_video_decoder` puts a factory in front of the built-in ones, so a
hardware decoder can take precedence. The GOP cache never holds its lock while decoding.
- **Hardware decoding.** The apps (desktop, CLI / headless MCP, bench) call
`filmcraft_platform::register()` at startup, which on macOS registers a VideoToolbox factory for
`avcC` / `hvcC` streams (8 / 10-bit, 4:2:0 and 4:2:2; elsewhere it does nothing). The factory
declines (our decoder is used) when Settings ▸ Playback ▸ Hardware decoding is Off
(`codecs::hw::set_hardware_decoding`, applied by the engine whenever preferences change), for
formats it does not take, and when the OS cannot create a *hardware* session for the stream.
Its decoder is a `platform::HybridDecoder`: VideoToolbox decodes asynchronously (two access
units in flight), a reorder buffer of the stream's own depth (`max_num_reorder_frames` /
`sps_max_num_reorder_pics`, from `codecs::hw::NalStreamInfo`) restores presentation order, and
the decoded biplanar `CVPixelBuffer` is copied into planar `Yuv8` / `Yuv16`. Colour, pixel
aspect, random-access and disposable answers come from the same helpers as the software
decoders (`video::vui_color`, `sar_par`, `h264_disposable`, `hevc_disposable`), so the two are
interchangeable (bit-exact in the parity tests). If the hardware fails mid-stream (decode error,
session lost to a GPU change or sleep, changed in-band parameter sets), the hybrid builds
`codecs::software_video_decoder` (built-in factories only), replays the samples since the last
restart point and continues in software for that instance; it is logged and counted. A source
already open keeps its decoder when the setting changes, until it is reopened. `perf.stats`
`decode.hardware` reports hardware vs software frames, sessions, declines and fallbacks.
Decoders run slices on rayon, so an export worker waiting inside a decode can pick up another
frame of the same source. A request that finds the shared decoder busy decodes with a private
decoder.
- **Compositor.** `render` is the reference for monitors, thumbnails and export. `render::plan`
turns a frame into GPU layers. Non-Normal blend modes, standard effects, adjustment layers, nested
sequences and non-dissolve transitions are rendered on the CPU for that layer or frame and handed to
the GPU as an image, so both paths give the same picture. Setting `FILMCRAFT_CPU_COMPOSITE=1`
turns a frame into GPU layers, each with its opacity and blend mode. All 27 blend modes run on the
GPU (`filmcraft-gpu`): Normal and Dissolve with fixed-function "over" blending, the others by
copying the accumulator under the layer into a backdrop texture and compositing in the fragment
shader with the CPU reference's formulas. The common standard effects run on the GPU too
(`render::gpufx`, `gpu::fx`): Brightness & Contrast, ProcAmp, Tint, Black & White, Color
Balance, Leave Color, Change to Color, Color Pass, Color Replace, Channel Mix, ASC CDL, Gamma
Correction, Levels, Extract, Invert, Posterize, Alpha Adjust, Gaussian Blur and Directional Blur
(and their legacy aliases), Camera Blur, Sharpen, Unsharp Mask, Crop, Edge Feather, Transform,
Horizontal / Vertical Flip, Mirror and Offset. When every enabled effect of a media clip is in
that set (unmasked, with finite parameters, and no Transform shrinking the picture below half
size, which the CPU pre-filters), the plan hands the GPU the clip's source with
the effects' parameters evaluated at that time: the source is drawn into an `Rgba32Float`
working image at the size the CPU decodes it, each effect runs as compute passes with the CPU
reference's math (effects, then Motion, then Opacity / blend, as on the CPU), and the result is
placed like any layer. Other standard effects, effect and opacity masks, adjustment layers,
nested sequences and non-dissolve transitions are rendered on the CPU for that layer or frame
and handed to the GPU as an image (a layer image keeps its clip's blend mode), so both paths
give the same picture. Setting `FILMCRAFT_CPU_COMPOSITE=1`
forces the CPU path in the desktop app.
- **Frame scheduling.** `crates/ui-egui/src/frames.rs` runs a small pool of worker threads with
prioritised jobs: the frame on screen first, then playback prefetch, then thumbnails. The UI never
@@ -636,7 +672,8 @@ Protocol reference: [control-protocol.md](control-protocol.md). Agent guide: [ag
## 8. Not built yet
The layer table reserves names for crates that don't exist yet: `riff`, `mjpeg`,
`keyframe`, `effects`, `audio`, `playback` and `platform`.
`keyframe`, `effects`, `audio` and `playback`. (`platform` exists but holds only OS media FFI;
other OS integration stays in `apps/filmcraft`.)
Until they exist, that work lives elsewhere: keyframes and effect definitions in `project`, effects
and the audio mix in `render`, playback in `ui-egui`, and OS integration (cpal, rfd,
native menus) in `apps/filmcraft`. [ROADMAP.md](../ROADMAP.md) has the milestone status.
+4
View File
@@ -133,6 +133,10 @@ export CARGO_TARGET_DIR=target/agent-hevc # separate build dir, no lock figh
so reduced-resolution playback matches full resolution. Unknown ids are a no-op.
3. Add a test in `crates/render/src/tests.rs`.
4. The GPU path needs no change: `render::plan` pre-renders layers with standard effects on the CPU.
To run it on the GPU too, make it an `FxOp` in `crates/render/src/gpufx.rs` instead (parameter
evaluation in `FxOp::eval`, the CPU reference in `FxOp::apply`, the id in `GPU_EFFECTS`), add
its passes to `crates/gpu/src/fx.rs` and its math to `fx.wgsl`, and a parity case to
`crates/gpu/src/fx_tests.rs`.
Audio effects implement `AudioEffect` in `crates/audio-dsp` and register in its `effects()`
registry. Then add an `audio(…)` definition in `effect.rs` and a `Mapping` from project parameters
+1 -1
View File
@@ -29,7 +29,7 @@ Methods (handlers in `crates/ui-egui/src/control.rs`):
| `ui.playback` | `{action: play|stop|toggle, speed?}` | |
| `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) |
| `ui.resize`, `ui.focus`, `app.quit` | | `ui.focus` is the only request that activates the app and takes keyboard focus |
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), decoder ms (total and per sample); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
| `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), decoder ms (total and per sample), `framesDecoded`, `hardware` (Settings ▸ Playback ▸ Hardware decoding: `enabled`, pictures from hardware decoders `frames` vs `softwareFrames`, `sessions`, `declined` streams, mid-stream `fallbacks`); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) |
**Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens
without activating itself. UI requests and screenshots that need a rendered frame bring the window
+87
View File
@@ -13,6 +13,93 @@ the before/after comparison relies on. Each milestone below alternated base / af
each); the most load-independent figure is the decoder's own cycle count (`proc_pid_rusage`
instructions and cycles of a process, all threads summed).
## Results (GPU2: standard effects on the GPU compositor, #30, before → after)
Before = this change with clips that carry standard effects sent back to the CPU layer path in
`render::plan` (same binary otherwise); after = GPU2. New `stack3-fx` scenario: `stack3` with
Brightness & Contrast + Gaussian Blur (blurriness 8) on V2 and Tint (70 %) on V3. `bench_playback
--scenario stack3-fx,stack3 --res full|half --gpu` (release), 8 s plays, two rounds alternating
before / after on 2026-10-05 at load average 107–170 (1-minute; 5-minute 141–152). `stack3` in
the same runs is the control; it shows how much the load alone moved the numbers.
| case | shown/dropped before (2 runs) | after (2 runs) | CPU ms/frame before → after | worker svc p50 before → after |
|---|---|---|---|---|
| stack3-fx Full | 0/192, 0/192 | **192/0, 132/60** | 84, 92 → 94, 89 | 245, 220 → **1.8, 93** ms |
| stack3-fx 1/2 | 9/183, 15/177 | **181/11, 166/26** | 111, 110 → 92, 92 | 295, 307 → **6.5, 28** ms |
| stack3 Full (control) | 180/12, 189/3 | 172/19, 80/112 | 92, 92 → 89, 80 | 7.2, 5.1 → 17.8, 185 ms |
| stack3 1/2 (control) | 138/54, 181/11 | 116/76, 134/53 | 91, 92 → 83, 91 | 61, 15 → 123, 106 ms |
Before, every frame of the effect stack was rendered on the CPU (Gaussian Blur and the two colour
effects on 1080p layers): ≈ 250 ms of worker time per frame, so playback showed nothing at Full
and 5–8 % of frames at ½ (CPU ms/frame stays near 90 only because most jobs were cancelled
before finishing). After, the workers only decode, as for `stack3`: the effect stage runs on the
GPU at present time (UI present p50 11–24 ms vs 11–39 ms for the control in the same runs). The
second "after Full" run coincided with a load spike (load 155 → 170; the control dropped 112
frames in the same minute).
## Results (HW1: VideoToolbox hardware decoding, Off → Auto)
Same commit, Settings ▸ Playback ▸ Hardware decoding switched with the bench flag `--hw off` /
`--hw auto` (`cargo xtask bench --sections decode --only dec_h --repeat 3 --hw …`, then
`--sections playback --only h264` and `--only hevc-2160`), 2026-10-05, Apple M4 Pro, load average
**150–190** throughout (shared with other agent builds). The decoded pictures are identical either
way (bit-exact parity tests, `crates/platform/tests/videotoolbox.rs`); only who decodes changes.
### Decode (every frame through the media stack)
| codec | size | CPU ms/frame Off → **Auto** | fps Off → **Auto** (best of 3) |
|---|---|---|---|
| H.264 | 1080p | 31.8 → **1.2** | 153 → **389** |
| H.264 | 2160p | 119.3 → **4.2** | 35 → **107** |
| HEVC | 1080p | 22.2 → **1.1** | 185 → **708** |
| HEVC | 2160p | 85.7 → **3.7** | 49 → **217** |
CPU time per frame is what is left on the CPU: the sample copy into a `CMSampleBuffer`, copying
the decoded biplanar picture out into planar Y'CbCr (chroma deinterleaved; 10-bit shifted down),
and the GOP cache. An earlier synchronous version (one picture decoding at a time) reached only
155 / 51 / 449 / 149 fps; asynchronous decompression with two access units in flight overlaps
decoding with the copy-out.
### Program-monitor playback (8 s, GPU path)
| case | shown/dropped Off → **Auto** | CPU ms/frame Off → **Auto** |
|---|---|---|
| H.264 1080p Full | 192/0 → **192/0** | 38.1 → **3.8** |
| H.264 2160p Full | 104/88 → **192/0** | 119.8 → **9.2** |
| H.264 2160p 1/2 | 1/191 → **192/0** | 95.5 → **9.0** |
| H.264 2160p 1/4 | 0/192 → **192/0** | 94.3 → **8.8** |
| HEVC 2160p Full | 175/17 → **192/0** | 105.6 → **9.1** |
| HEVC 2160p 1/2 | 190/2 → **192/0** | 97.4 → **8.8** |
| HEVC 2160p 1/2 draft | 190/2 → **191/1** | 107.0 → **22.3** |
At this load the software decoders drop most 4K H.264 frames; with hardware decoding every case
plays in real time at a tenth of the CPU. Draft playback costs more CPU than full-quality
playback with hardware decoding: the hardware ignores draft mode (`set_draft` is a no-op) and
the draft plan's box decimation of the planes, cheap next to software decoding, is now the main
CPU cost. Zero-copy upload of the decoded `CVPixelBuffer` into wgpu (no copy-out at all) is the
next step (issue #30).
## Results (GPU1: blend modes on the GPU compositor, #30, before → after)
Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back
in `render::plan` (same binary otherwise); after = GPU1. New `stack3-blend` scenario: `stack3`
(three 1080p23.976 H.264 clips, V2/V3 scaled, positioned, rotated, 70–85 % opacity) with V2 in
Screen and V3 in Overlay. `cargo xtask bench-playback --scenario stack3-blend,stack3 --res
full,half`, GPU path, 8 s plays, two alternating rounds each on 2026-10-05 at load average
33–63 (1-minute; 5-minute 67–81).
| case | shown/dropped before (2 runs) | after (2 runs) | CPU ms/frame before → after | worker svc p50 before → after |
|---|---|---|---|---|
| stack3-blend Full | 19/173, 138/54 | **191/1, 192/0** | 168, 214 → **86, 87** | 258, 219 → 1.2, 1.2 ms |
| stack3-blend 1/2 | 192/0, 192/0 | 191/1, 192/0 | 158, 151 → **89, 88** | 66, 99 → 1.1, 1.0 ms |
| stack3 Full (control, Normal) | 192/0, 192/0 | 191/1, 192/0 | 88, 88 → 91, 88 | 1.3, 1.6 → 0.9, 1.5 ms |
| stack3 1/2 (control, Normal) | 191/1, 192/0 | 192/0, 192/0 | 82, 90 → 90, 89 | |
A stack with blend modes now costs what the same stack in Normal does: the frame workers only
decode, and the UI-thread present (upload and draw, including the backdrop copies for the two
blended layers) stays at 9–10 ms p50, as for `stack3`. Before, every frame was composited on the
CPU (≈ 2 cores more at 23.976 fps) and Full resolution could not keep up at this load.
## Results (M4.10: 4K VP9, AV1 and HEVC, before → after)
Baseline = commit `a814ff6` (decoders unchanged since M4.8); after = M4.10. Interleaved runs on
+3 -2
View File
@@ -238,10 +238,10 @@ own process under `/usr/bin/time` so peak RSS is per section: `decode` (every fr
(the real app under `egui_kittest` with a 1000-clip / 20-track sequence: update, tessellation and
wgpu render ms per frame), `export` (H.264 + AAC and ProRes) and `project` (save / open 5000 clips).
Options: `--sections decode,scrub`, `--only <substring>` (fixture or scenario), `--repeat N`,
`--quick`, `--cpu`, `--label NAME` (output `target/bench/bench-<label>.{json,md}`),
`--quick`, `--cpu`, `--hw auto|off` (Settings ▸ Playback ▸ Hardware decoding; default `auto`, as the app), `--label NAME` (output `target/bench/bench-<label>.{json,md}`),
`--section NAME --json FILE` (one section in-process, e.g. under a profiler). Fixtures are made
with ffmpeg in `target/fixtures/playback/` (or `$FILMCRAFT_FIXTURES/playback`). The suite's
`playback` section plays `h264-1080` and `stack3` at Full, `h264-2160` at Full, 1/2 and 1/4 and with
`playback` section plays `h264-1080`, `stack3` and `stack3-blend` at Full, `h264-2160` at Full, 1/2 and 1/4 and with
draft decoding (Settings ▸ Playback ▸ Draft decoding) at 1/2 and 1/4, and `hevc-2160`, `vp9-2160`
and `av1-2160` at Full, 1/2 and 1/2 draft; its `draft` column counts the frames decoded in draft
mode.
@@ -268,6 +268,7 @@ cargo xtask bench-playback --json target/bench-playback.json # machine-readable
|---|---|
| `h264-1080` | one 1080p23.976 H.264 clip (testsrc2 + grain, ~45 Mbit/s, 250-frame GOP) |
| `stack3` | three 1080p H.264 clips on V1–V3; V2/V3 scaled, positioned, rotated, 70–85 % opacity |
| `stack3-blend` | `stack3` with V2 in Screen and V3 in Overlay (blend modes on the GPU) |
| `h264-2160` | one 2160p23.976 H.264 clip |
| `demo` | the built-in demo project (procedural footage, transitions, effects) |
| `after-preview` | 1080p H.264 + Lumetri/Sharpen/Levels/Tint: a live play, Render Effects In to Out, then two plays of the green segment |