mirror of
https://github.com/storytold/effectcraft.git
synced 2026-11-06 10:31:05 +00:00
gmrmad1andFredApps
389894e69b
* gpu: run Content-Aware Fill's PatchMatch on the device Content-Aware Fill's synthesis is the stage that makes the tool slow: at 1920×1080 with a 300×200 hole, PatchMatch costs ≈ 1.1 s per frame on the CPU, and it is what Object, Surface and every refining frame spend their time in. Add a `FillBackend` trait to `effectcraft-raster` whose stages (the membrane solve, the flow fields, PatchMatch) default to the CPU reference, and `Accelerator::inpaint()` on the compositor, which is how the engine asks for a device backend. `effectcraft_gpu::inpaint` implements PatchMatch: the multi-scale pyramid, the EM rounds, the propagation (a two-pass checkerboard instead of the CPU's scan order), the random search, the SSD, the histogram-based σ² and the weighted vote, as compute kernels in `shaders/inpaint.wgsl`. The fill's frames are read back to the CPU before it runs (analyses composite on the CPU by design), so the backend is about the fill's own work; the orchestration, the flow-guided propagation and the per-frame flow control stay on the CPU. Only the hole's bounds cross back to the CPU, which is what makes the offload pay. Measured on an Intel Arc (MTL) iGPU, 1920×1080, 300×200 hole: CPU 1087 ms GPU 397 ms (2.7×, 41.0 dB PSNR against the CPU's synthesis) The membrane (harmonic) solve deliberately stays on the CPU: the same measurement has it at ≈ 90 ms on the device against ≈ 55 ms on the CPU, because a fill's full-frame data movement (upload, the known pyramid, a copy) does not shrink with the hole, while the relaxation does. The device takes only the stage where its parallel search pays; `membrane_tex` still runs on the device inside PatchMatch, where it is the coarsest level's start and never crosses back. Closes #1839. * gpu: fix the fill's vote histogram and run its flow on the device Review follow-ups on the Content-Aware Fill GPU port, plus the fill's flow. Fixes: * imp_hist kept about one sample in 256: each of the 256 threads scanned a strided subset of the pixels and retained only those whose bin selected it, so the vote's sigma² came from a sample correlated with the pixel index (and a small hole usually fell back to 1.0, making the weights near-uniform). One workgroup now accumulates the whole population through shared atomic bins, with a test that pins the sample count. * imp_pm_prop and imp_pm_rand were dispatched over grow(near, 1) but with near's origin, so the ring's left column and top row were never written (NaN under test, stale pooled data in production) while a strip past it was. `over` now derives the shader's origin from the box it sizes the dispatch from, so a dispatch's region and its origin can no longer disagree. * Gpu::inpaint() requires its kernels to have built. Otherwise every stage paid an upload and a failed dispatch to fall back to the CPU, while the fill still reported the GPU path. * imp_rnd's XOR chain parenthesises its multiplicative terms: WGSL's grammar makes each XOR operand a unary expression, so `a * b ^ c * d` is rejected by spec-compliant parsers (naga accepts it). * FillBackend documents that its stages may be called concurrently. * The coarsest membrane's sweep count (64, against the CPU's 200) is now justified: matching 200 measured ~6% slower over the fill with no change in the result. And the flow: Gpu::flow runs block_flow's coarse-to-fine search, sub-pixel fit and vector median on the device. The luma is the CPU's (flow::luma_pair), so the field matches block_flow's to a float rounding and the propagation sees the same motion; the hole completion stays on the CPU with the membrane. A four-frame 1920x1080 fill goes from 1.27x to 1.42x against the CPU, and the device path is covered for holes from 16x16 to 300x200. * gpu: keep the fill vote's histogram range on HDR input imp_hist binned every distance at a fixed 256 bins per unit and clamped to 1023, so the vote's sigma2 topped out at 3.998. A 32-bpc frame's channels can exceed 1, and masked_frame forwards them unclamped: a patch distance of 1600 then reported sigma2 ~ 4, exp(-1600/(2*sigma2)) rounded to zero in f32, and imp_pm_vote kept the prior image instead of synthesising. The histogram now reports the largest finite distance it saw (atomicMax on the float bits, carried in bin 0's unused .g channel). SDR input keeps the fixed 0..4 range with identical bins; HDR input is re-binned over the observed range and percentile() maps the bin centre back through it. Nothing clamps the input. Also group over/sweeps/pm_level's arguments into small structs: the workspace denies too_many_arguments and the lint policy (#2496) does not allow new #[allow]s. No behaviour change on those paths. * engine: don't report Edge Blend fills as device fills Edge Blend calls only the membrane, which the GPU backend leaves to the CPU reference, so run_plan's "Filling (GPU)" status was wrong for it whenever a backend was present. Object and Surface still run the flow and PatchMatch on the device and keep the label. CodeRabbit's #1856 finding. * gpu: run the fill's device stages inside error scopes An allocation or validation failure inside patch_image/flow_image left an invalid texture whose later use raised an uncaptured error, retiring the shared compositor instead of falling back to the CPU for that frame. Wrap both stages in Gpu::within_memory, whose scopes are native thread-local, so the fill's rayon workers do not share them. --------- Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
5.6 KiB
5.6 KiB
Hardware parity
Last reviewed: 2026-10-29 · Last updated: 2026-10-29 · Change: minor (Content-Aware Fill's PatchMatch and flow search on the GPU) · Target: Adobe After Effects 2026
Each piece of hardware After Effects uses, per platform, against EffectCraft. Ours comes from the
code at origin/main 1446e27 (crates/gpu, crates/render, crates/ui-egui) and the issue
tracker; After Effects' from its documentation (Mercury GPU Acceleration, Multi-Frame Rendering,
hardware-accelerated decode and encode, Mercury Transmit) and its bundle.
Overall ≈ 40% (estimated). The GPU compositor and 280 GPU effects are a real strength and run in the browser too; hardware video decode and encode, video-out devices, tablets and multiple monitors are missing, and two GPU start-up failures are open.
| Feature | After Effects 2026 | EffectCraft macOS | Windows | Linux | Web | Notes / evidence |
|---|---|---|---|---|---|---|
| GPU compositing and effects | Mercury GPU (Metal; CUDA / OpenCL / DirectX on Windows) | yes, Metal via wgpu | Vulkan yes; Direct3D 12 fails (FXC can't compile the pointwise kernel, #613) |
Vulkan | WebGPU, in frame workers | 280 of 306 effects on the GPU, checked against the CPU renderer; Fractal and the analysis/tool effects stay on the CPU (effects-parity.md) |
| GPU analysis | GPU-accelerated Content-Aware Fill | Content-Aware Fill's PatchMatch and flow block matching on the GPU (Gpu::inpaint); every other stage on the CPU |
same | same | no (needs synchronous readback) | On an Intel Arc iGPU a four-frame 1920 × 1080 fill is ≈ 1.4× faster than the CPU's (PatchMatch is the dominant win, the flow search a smaller one), and even a 16 × 16 hole at 1920 × 1080 is ≈ 1.9× faster — the CPU's own stages are O(frame), not O(hole). The device's flow field matches block_flow to a float rounding. The membrane and the flow's hole completion stay on the CPU: both are bandwidth-bound, and the device's full-frame transfers cost more than they save — see effectcraft_gpu::inpaint |
| CPU fallback and device-loss recovery | yes | yes | yes; Windows ARM64 Adreno panic after device loss (#591, closed) | yes | yes (#225 fixed) | Out of video memory falls back to the CPU (#106) |
| GPU detection and choice | GPU info, renderer choice | Backend Auto per comp | "GPU isn't being detected" (#596, closed); Intel HD 5500 fix unverified (#198) | yes | yes | No settings page listing adapters yet |
| Multi-core rendering | Multi-Frame Rendering | rayon pools; preview scheduler sized to the render pool | same | same | Web Workers, one engine per worker | No benchmark against Multi-Frame Rendering |
| Hardware video decode | H.264 / HEVC / ProRes (Apple silicon) | none (software, FilmCraft) | none | none | none | FilmCraft has VideoToolbox, Media Foundation/DXVA and VA-API paths to borrow (gap 15) |
| Hardware video encode | H.264 / HEVC via VideoToolbox, NVENC, Quick Sync (render queue and Media Encoder) | none | none | none | none | #498 asks for NVENC (answered: Nvidia's SDK needs FFI, which the no-unsafe rule forbids; open PR #819 adds hardware decoding); our software encoders are slower and lack B-frames |
| Video output devices | Mercury Transmit to AJA, Blackmagic, second display | second preview window only (Video Preview) | same | same | — | No SDI/HDMI device output |
| HDR display | HDR monitoring on EDR / HDR displays | tone-mapped SDR view; no EDR output | no HDR output | no | no | Colour engine handles HDR maths (Rec. 2100 PQ/HLG output files) |
| Pen tablets (pressure, tilt) | yes, for Paint and Roto | Brushes panel has pressure options, but no tablet pressure is read | same | same | same | Paint strokes store pressure per vertex; input never fills it |
| Multiple monitors | panels and windows across screens | panels move into their own native windows on any screen (Move Panel to New Window; desktop builds) | same | same (#486) | — | gap 20 |
| Touch / trackpad gestures | pinch zoom in viewers | Composition and Layer viewers: two-finger scroll pans, pinch zooms around the pointer (Alt + scroll and the mouse wheel zoom too); Timeline pinch zooms time | two-finger scroll pans once it moves diagonally (a pixel-precise wheel keeps zooming); pinch arrives as Ctrl + scroll and zooms | same | same | No rotate or swipe gestures; tested in ui_viewer_trackpad.rs (UI logic, not on hardware) |
| Audio hardware | Core Audio / ASIO / WASAPI, device choice, latency | cpal output; preview sound late after a start offset (#504 open; #554 closed) | cpal | cpal (ALSA) | Web Audio | No input or device latency settings |
| Machine learning acceleration | Apple Neural Engine / GPU for Object Matte, Roto Brush 3 | CPU inference (pure Rust, rayon) | CPU | CPU | CPU | No GPU inference for MobileSAM or MediaPipe |
| Memory management | RAM reserve, disk cache | RAM Reserved, Reduce Cache When Low on Memory, disk cache | same | same | OPFS disk cache | good |
Estimate
≈ 60–120 h: Direct3D 12 and adapter start-up fixes 6–12 h; hardware decode on three platforms 20–35 h and encode 15–30 h (borrowing FilmCraft's work); multiple monitors 10–20 h; tablet pressure 4–8 h; HDR output 8–15 h. Needs Windows and Linux GPUs we don't have locally, and a human for the Adreno and Intel cases.
Revision history
| Date | Change | Summary |
|---|---|---|
| 2026-10-29 | minor | Content-Aware Fill's PatchMatch and flow block matching run on the GPU (FillBackend/Accelerator::inpaint, effectcraft_gpu::inpaint) |
| 2026-10-12 | minor | #591, #596 and #554 marked closed; #498 answer and open PR #819 |
| 2026-10-10 | major | First hardware parity document |