mirror of
https://github.com/storytold/effectcraft.git
synced 2026-11-18 16:14:44 +00:00
* gpu: run Content-Aware Fill's PatchMatch on the device Content-Aware Fill's synthesis is the stage that makes the tool slow: at 1920×1080 with a 300×200 hole, PatchMatch costs ≈ 1.1 s per frame on the CPU, and it is what Object, Surface and every refining frame spend their time in. Add a `FillBackend` trait to `effectcraft-raster` whose stages (the membrane solve, the flow fields, PatchMatch) default to the CPU reference, and `Accelerator::inpaint()` on the compositor, which is how the engine asks for a device backend. `effectcraft_gpu::inpaint` implements PatchMatch: the multi-scale pyramid, the EM rounds, the propagation (a two-pass checkerboard instead of the CPU's scan order), the random search, the SSD, the histogram-based σ² and the weighted vote, as compute kernels in `shaders/inpaint.wgsl`. The fill's frames are read back to the CPU before it runs (analyses composite on the CPU by design), so the backend is about the fill's own work; the orchestration, the flow-guided propagation and the per-frame flow control stay on the CPU. Only the hole's bounds cross back to the CPU, which is what makes the offload pay. Measured on an Intel Arc (MTL) iGPU, 1920×1080, 300×200 hole: CPU 1087 ms GPU 397 ms (2.7×, 41.0 dB PSNR against the CPU's synthesis) The membrane (harmonic) solve deliberately stays on the CPU: the same measurement has it at ≈ 90 ms on the device against ≈ 55 ms on the CPU, because a fill's full-frame data movement (upload, the known pyramid, a copy) does not shrink with the hole, while the relaxation does. The device takes only the stage where its parallel search pays; `membrane_tex` still runs on the device inside PatchMatch, where it is the coarsest level's start and never crosses back. Closes #1839. * gpu: fix the fill's vote histogram and run its flow on the device Review follow-ups on the Content-Aware Fill GPU port, plus the fill's flow. Fixes: * imp_hist kept about one sample in 256: each of the 256 threads scanned a strided subset of the pixels and retained only those whose bin selected it, so the vote's sigma² came from a sample correlated with the pixel index (and a small hole usually fell back to 1.0, making the weights near-uniform). One workgroup now accumulates the whole population through shared atomic bins, with a test that pins the sample count. * imp_pm_prop and imp_pm_rand were dispatched over grow(near, 1) but with near's origin, so the ring's left column and top row were never written (NaN under test, stale pooled data in production) while a strip past it was. `over` now derives the shader's origin from the box it sizes the dispatch from, so a dispatch's region and its origin can no longer disagree. * Gpu::inpaint() requires its kernels to have built. Otherwise every stage paid an upload and a failed dispatch to fall back to the CPU, while the fill still reported the GPU path. * imp_rnd's XOR chain parenthesises its multiplicative terms: WGSL's grammar makes each XOR operand a unary expression, so `a * b ^ c * d` is rejected by spec-compliant parsers (naga accepts it). * FillBackend documents that its stages may be called concurrently. * The coarsest membrane's sweep count (64, against the CPU's 200) is now justified: matching 200 measured ~6% slower over the fill with no change in the result. And the flow: Gpu::flow runs block_flow's coarse-to-fine search, sub-pixel fit and vector median on the device. The luma is the CPU's (flow::luma_pair), so the field matches block_flow's to a float rounding and the propagation sees the same motion; the hole completion stays on the CPU with the membrane. A four-frame 1920x1080 fill goes from 1.27x to 1.42x against the CPU, and the device path is covered for holes from 16x16 to 300x200. * gpu: keep the fill vote's histogram range on HDR input imp_hist binned every distance at a fixed 256 bins per unit and clamped to 1023, so the vote's sigma2 topped out at 3.998. A 32-bpc frame's channels can exceed 1, and masked_frame forwards them unclamped: a patch distance of 1600 then reported sigma2 ~ 4, exp(-1600/(2*sigma2)) rounded to zero in f32, and imp_pm_vote kept the prior image instead of synthesising. The histogram now reports the largest finite distance it saw (atomicMax on the float bits, carried in bin 0's unused .g channel). SDR input keeps the fixed 0..4 range with identical bins; HDR input is re-binned over the observed range and percentile() maps the bin centre back through it. Nothing clamps the input. Also group over/sweeps/pm_level's arguments into small structs: the workspace denies too_many_arguments and the lint policy (#2496) does not allow new #[allow]s. No behaviour change on those paths. * engine: don't report Edge Blend fills as device fills Edge Blend calls only the membrane, which the GPU backend leaves to the CPU reference, so run_plan's "Filling (GPU)" status was wrong for it whenever a backend was present. Object and Surface still run the flow and PatchMatch on the device and keep the label. CodeRabbit's #1856 finding. * gpu: run the fill's device stages inside error scopes An allocation or validation failure inside patch_image/flow_image left an invalid texture whose later use raised an uncaptured error, retiring the shared compositor instead of falling back to the CPU for that frame. Wrap both stages in Gpu::within_memory, whose scopes are native thread-local, so the fill's rayon workers do not share them. --------- Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
317 lines
13 KiB
Rust
317 lines
13 KiB
Rust
//! Optical flow by block matching, and motion-compensated frame interpolation (the Pixel Motion
|
||
//! mode of frame blending).
|
||
//!
|
||
//! The method is the textbook hierarchical block matcher used in video coding and frame-rate
|
||
//! conversion: build a luma pyramid of both frames; at the coarsest level, find for each block of
|
||
//! frame A the integer displacement into frame B with the smallest sum of absolute differences
|
||
//! (SAD) in a search window; at each finer level, start from the doubled coarse vectors (and the
|
||
//! neighbours' vectors) and refine in a small window; finish with a sub-pixel parabola fit of the
|
||
//! SAD around the best vector and a 3×3 vector median to remove outliers.
|
||
//!
|
||
//! Interpolation at a fraction `w` between the frames warps both towards the in-between time —
|
||
//! A forward by `w·F`, B back by `(1 − w)·F` — and cross-fades them, so moving detail stays sharp
|
||
//! instead of double-exposing as with a plain frame mix.
|
||
|
||
use rayon::prelude::*;
|
||
|
||
use crate::Image;
|
||
|
||
/// A block motion field: one vector per `block × block` tile of frame A (in pixels, A → B).
|
||
#[derive(Clone, Debug, Default, PartialEq)]
|
||
pub struct Flow {
|
||
pub width: u32,
|
||
pub height: u32,
|
||
pub block: u32,
|
||
/// Tiles across / down.
|
||
pub cols: usize,
|
||
pub rows: usize,
|
||
pub v: Vec<[f32; 2]>,
|
||
}
|
||
|
||
impl Flow {
|
||
/// The motion at pixel position (x, y), bilinearly interpolated between tile centres.
|
||
pub fn at(&self, x: f64, y: f64) -> [f32; 2] {
|
||
if self.v.is_empty() {
|
||
return [0.0; 2];
|
||
}
|
||
let b = self.block as f64;
|
||
let fx = (x / b - 0.5).clamp(0.0, (self.cols - 1) as f64);
|
||
let fy = (y / b - 0.5).clamp(0.0, (self.rows - 1) as f64);
|
||
let (x0, y0) = (fx.floor() as usize, fy.floor() as usize);
|
||
let (x1, y1) = ((x0 + 1).min(self.cols - 1), (y0 + 1).min(self.rows - 1));
|
||
let (tx, ty) = ((fx - x0 as f64) as f32, (fy - y0 as f64) as f32);
|
||
let g = |c: usize, r: usize| self.v[r * self.cols + c];
|
||
let mut o = [0.0; 2];
|
||
for i in 0..2 {
|
||
let top = g(x0, y0)[i] + (g(x1, y0)[i] - g(x0, y0)[i]) * tx;
|
||
let bot = g(x0, y1)[i] + (g(x1, y1)[i] - g(x0, y1)[i]) * tx;
|
||
o[i] = top + (bot - top) * ty;
|
||
}
|
||
o
|
||
}
|
||
}
|
||
|
||
/// The `Plane::luma` of `a` and `b` packed into one image (`a` in the red channel, `b` in the
|
||
/// green; the rest zero). A device block matcher takes this as its input rather than computing
|
||
/// the luma itself: the weighted sum is not reproducible on a device, which may contract
|
||
/// `a·b + c` into a fused multiply-add, and the search's near-tie choices would then differ.
|
||
pub fn luma_pair(a: &Image, b: &Image) -> Image {
|
||
let luma = |p: &[f32; 4]| 0.2126 * p[0] + 0.7152 * p[1] + 0.0722 * p[2] + 0.25 * p[3];
|
||
let mut out = Image::new(a.width, a.height);
|
||
out.data.par_iter_mut().enumerate().for_each(|(i, q)| {
|
||
let la = a.data.get(i).map(luma).unwrap_or(0.0);
|
||
let lb = b.data.get(i).map(luma).unwrap_or(0.0);
|
||
*q = [la, lb, 0.0, 0.0];
|
||
});
|
||
out
|
||
}
|
||
|
||
/// A single-channel plane.
|
||
struct Plane {
|
||
w: usize,
|
||
h: usize,
|
||
d: Vec<f32>,
|
||
}
|
||
|
||
impl Plane {
|
||
fn luma(img: &Image) -> Plane {
|
||
let d = img.data.par_iter().map(|p| 0.2126 * p[0] + 0.7152 * p[1] + 0.0722 * p[2] + 0.25 * p[3]).collect();
|
||
Plane { w: img.width as usize, h: img.height as usize, d }
|
||
}
|
||
fn half(&self) -> Plane {
|
||
let (w, h) = ((self.w / 2).max(1), (self.h / 2).max(1));
|
||
let d = (0..w * h)
|
||
.into_par_iter()
|
||
.map(|i| {
|
||
let (x, y) = (i % w * 2, i / w * 2);
|
||
let g = |dx: usize, dy: usize| self.d[(y + dy).min(self.h - 1) * self.w + (x + dx).min(self.w - 1)];
|
||
(g(0, 0) + g(1, 0) + g(0, 1) + g(1, 1)) * 0.25
|
||
})
|
||
.collect();
|
||
Plane { w, h, d }
|
||
}
|
||
#[inline]
|
||
fn get(&self, x: i64, y: i64) -> f32 {
|
||
self.d[y.clamp(0, self.h as i64 - 1) as usize * self.w + x.clamp(0, self.w as i64 - 1) as usize]
|
||
}
|
||
}
|
||
|
||
/// SAD of the `bs` block at (bx, by) in `a` against `b` displaced by (dx, dy).
|
||
fn sad(a: &Plane, b: &Plane, bx: i64, by: i64, bs: i64, dx: i64, dy: i64) -> f32 {
|
||
let mut s = 0.0;
|
||
for y in by..by + bs {
|
||
for x in bx..bx + bs {
|
||
s += (a.get(x, y) - b.get(x + dx, y + dy)).abs();
|
||
}
|
||
}
|
||
s
|
||
}
|
||
|
||
/// Block-matching flow from `a` to `b` (same size). `block` is the tile size at full resolution;
|
||
/// `radius` the search range at the coarsest level (motion up to about `radius · 2^levels`).
|
||
pub fn block_flow(a: &Image, b: &Image, block: u32, radius: i32) -> Flow {
|
||
block_flow_window(a, b, block, block, radius)
|
||
}
|
||
|
||
/// [`block_flow`] with separate vector spacing and matching window: one vector per
|
||
/// `tile × tile` tile, each matched with the `window × window` block centred on its tile
|
||
/// (Timewarp's Vector Detail and Block Size).
|
||
pub fn block_flow_window(a: &Image, b: &Image, tile: u32, window: u32, radius: i32) -> Flow {
|
||
let bs = tile.max(2) as usize;
|
||
let win = window.max(4) as i64;
|
||
// Window origin relative to the tile origin.
|
||
let wo = bs as i64 / 2 - win / 2;
|
||
let (w, h) = (a.width as usize, a.height as usize);
|
||
let cols = w.div_ceil(bs).max(1);
|
||
let rows = h.div_ceil(bs).max(1);
|
||
let mut flow = Flow { width: a.width, height: a.height, block: bs as u32, cols, rows, v: vec![[0.0; 2]; cols * rows] };
|
||
if a.width != b.width || a.height != b.height || w < bs || h < bs || (w as i64) < win || (h as i64) < win {
|
||
return flow;
|
||
}
|
||
// Pyramids (level 0 = full resolution); stop while blocks still fit.
|
||
let mut pa = vec![Plane::luma(a)];
|
||
let mut pb = vec![Plane::luma(b)];
|
||
while pa.len() < 4 && pa.last().is_some_and(|p| p.w / 2 >= (bs as i64).max(win) as usize * 2 && p.h / 2 >= (bs as i64).max(win) as usize * 2) {
|
||
let (na, nb) = (pa.last().map(Plane::half), pb.last().map(Plane::half));
|
||
pa.extend(na);
|
||
pb.extend(nb);
|
||
}
|
||
let top = pa.len() - 1;
|
||
// Integer vectors at the current level, per full-resolution tile.
|
||
let mut cur: Vec<[i64; 2]> = vec![[0, 0]; cols * rows];
|
||
for level in (0..=top).rev() {
|
||
let (la, lb) = (&pa[level], &pb[level]);
|
||
let k = 1i64 << level;
|
||
// The block covers the tile's footprint at this level (at least 4 pixels).
|
||
let lbs = (win / k).max(4);
|
||
let r = if level == top { radius as i64 } else { 1 };
|
||
let prev = cur.clone();
|
||
cur = (0..cols * rows)
|
||
.into_par_iter()
|
||
.map(|i| {
|
||
let (c, rr) = ((i % cols) as i64, (i / cols) as i64);
|
||
let bx = (c * bs as i64 + wo) / k;
|
||
let by = (rr * bs as i64 + wo) / k;
|
||
// Candidates: this tile's and its neighbours' vectors from the coarser level.
|
||
let mut seeds: Vec<[i64; 2]> = vec![[0, 0]];
|
||
if level != top {
|
||
for (dc, dr) in [(0, 0), (-1, 0), (1, 0), (0, -1), (0, 1)] {
|
||
let (nc, nr) = (c + dc, rr + dr);
|
||
if nc >= 0 && nr >= 0 && (nc as usize) < cols && (nr as usize) < rows {
|
||
let v = prev[nr as usize * cols + nc as usize];
|
||
seeds.push([v[0] * 2, v[1] * 2]);
|
||
}
|
||
}
|
||
}
|
||
seeds.sort_unstable();
|
||
seeds.dedup();
|
||
let mut best = ([0i64, 0i64], f32::INFINITY);
|
||
for s in seeds {
|
||
for dy in -r..=r {
|
||
for dx in -r..=r {
|
||
let d = [s[0] + dx, s[1] + dy];
|
||
// A tiny bias towards short vectors breaks ties in flat areas.
|
||
let e = sad(la, lb, bx, by, lbs, d[0], d[1]) + 1e-4 * (d[0].abs() + d[1].abs()) as f32;
|
||
if e < best.1 {
|
||
best = (d, e);
|
||
}
|
||
}
|
||
}
|
||
}
|
||
best.0
|
||
})
|
||
.collect();
|
||
}
|
||
// Sub-pixel refinement (parabola through the SAD of the neighbours) at full resolution.
|
||
let (la, lb) = (&pa[0], &pb[0]);
|
||
let bsi = bs as i64;
|
||
let wsi = win;
|
||
let fine: Vec<[f32; 2]> = (0..cols * rows)
|
||
.into_par_iter()
|
||
.map(|i| {
|
||
let (bx, by) = ((i % cols) as i64 * bsi + wo, (i / cols) as i64 * bsi + wo);
|
||
let d = cur[i];
|
||
let e0 = sad(la, lb, bx, by, wsi, d[0], d[1]);
|
||
let fit = |em: f32, ep: f32| {
|
||
let den = em - 2.0 * e0 + ep;
|
||
if den > 1e-6 { (0.5 * (em - ep) / den).clamp(-0.5, 0.5) } else { 0.0 }
|
||
};
|
||
let sx = fit(sad(la, lb, bx, by, wsi, d[0] - 1, d[1]), sad(la, lb, bx, by, wsi, d[0] + 1, d[1]));
|
||
let sy = fit(sad(la, lb, bx, by, wsi, d[0], d[1] - 1), sad(la, lb, bx, by, wsi, d[0], d[1] + 1));
|
||
[d[0] as f32 + sx, d[1] as f32 + sy]
|
||
})
|
||
.collect();
|
||
// 3×3 vector median (component-wise) against outliers.
|
||
flow.v = (0..cols * rows)
|
||
.into_par_iter()
|
||
.map(|i| {
|
||
let (c, r) = ((i % cols) as i64, (i / cols) as i64);
|
||
let mut xs = Vec::with_capacity(9);
|
||
let mut ys = Vec::with_capacity(9);
|
||
for dr in -1..=1 {
|
||
for dc in -1..=1 {
|
||
let (nc, nr) = (c + dc, r + dr);
|
||
if nc >= 0 && nr >= 0 && (nc as usize) < cols && (nr as usize) < rows {
|
||
let v = fine[nr as usize * cols + nc as usize];
|
||
xs.push(v[0]);
|
||
ys.push(v[1]);
|
||
}
|
||
}
|
||
}
|
||
xs.sort_by(f32::total_cmp);
|
||
ys.sort_by(f32::total_cmp);
|
||
[xs[xs.len() / 2], ys[ys.len() / 2]]
|
||
})
|
||
.collect();
|
||
flow
|
||
}
|
||
|
||
/// Cross-fade of two frames: `a·(1 − w) + b·w` (Frame Mix).
|
||
pub fn mix(a: &Image, b: &Image, w: f32) -> Image {
|
||
if a.width != b.width || a.height != b.height {
|
||
return if w < 0.5 { a.clone() } else { b.clone() };
|
||
}
|
||
let mut out = a.clone();
|
||
out.data.par_iter_mut().zip(b.data.par_iter()).for_each(|(p, q)| {
|
||
for c in 0..4 {
|
||
p[c] += (q[c] - p[c]) * w;
|
||
}
|
||
});
|
||
out
|
||
}
|
||
|
||
/// Motion-compensated frame at fraction `w` (0 = `a`, 1 = `b`) between two frames (Pixel Motion).
|
||
pub fn interpolate(a: &Image, b: &Image, w: f32) -> Image {
|
||
if a.width != b.width || a.height != b.height {
|
||
return mix(a, b, w);
|
||
}
|
||
let flow = block_flow(a, b, 8, 4);
|
||
let mut out = Image::new(a.width, a.height);
|
||
let wd = w as f64;
|
||
out.rows_mut().for_each(|(y, row)| {
|
||
let cy = y as f64 + 0.5;
|
||
for (x, px) in row.iter_mut().enumerate() {
|
||
let cx = x as f64 + 0.5;
|
||
let f = flow.at(cx, cy);
|
||
let (fx, fy) = (f[0] as f64, f[1] as f64);
|
||
let pa = a.sample_bilinear_clamped(cx - wd * fx, cy - wd * fy);
|
||
let pb = b.sample_bilinear_clamped(cx + (1.0 - wd) * fx, cy + (1.0 - wd) * fy);
|
||
for c in 0..4 {
|
||
px[c] = pa[c] + (pb[c] - pa[c]) * w;
|
||
}
|
||
}
|
||
});
|
||
out
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
/// A textured image shifted right by `dx` pixels.
|
||
fn texture(w: u32, h: u32, dx: f64) -> Image {
|
||
let mut img = Image::new(w, h);
|
||
img.rows_mut().for_each(|(y, row)| {
|
||
for (x, p) in row.iter_mut().enumerate() {
|
||
let (u, v) = (x as f64 - dx, y as f64);
|
||
let s = 0.5 + 0.2 * (u * 0.21 + v * 0.07).sin() + 0.15 * (u * 0.05 - v * 0.19).cos() + 0.1 * (u * 0.013 * v * 0.011).sin();
|
||
let s = s as f32;
|
||
*p = [s, s * 0.8, 1.0 - s, 1.0];
|
||
}
|
||
});
|
||
img
|
||
}
|
||
|
||
fn mae(a: &Image, b: &Image, margin: u32) -> f32 {
|
||
let mut s = 0.0;
|
||
let mut n = 0;
|
||
for y in margin..a.height - margin {
|
||
for x in margin..a.width - margin {
|
||
let (p, q) = (a.get(x as i64, y as i64), b.get(x as i64, y as i64));
|
||
s += (0..3).map(|c| (p[c] - q[c]).abs()).sum::<f32>();
|
||
n += 3;
|
||
}
|
||
}
|
||
s / n as f32
|
||
}
|
||
|
||
#[test]
|
||
fn flow_finds_translation() {
|
||
let a = texture(128, 96, 0.0);
|
||
let b = texture(128, 96, 6.0);
|
||
let f = block_flow(&a, &b, 8, 4);
|
||
let m = f.at(64.0, 48.0);
|
||
assert!((m[0] - 6.0).abs() < 0.3 && m[1].abs() < 0.3, "{m:?}");
|
||
}
|
||
|
||
#[test]
|
||
fn pixel_motion_beats_frame_mix_on_translation() {
|
||
let a = texture(128, 96, 0.0);
|
||
let b = texture(128, 96, 6.0);
|
||
let truth = texture(128, 96, 3.0);
|
||
let pm = mae(&interpolate(&a, &b, 0.5), &truth, 12);
|
||
let fm = mae(&mix(&a, &b, 0.5), &truth, 12);
|
||
assert!(pm < fm * 0.25, "pixel motion {pm} vs frame mix {fm}");
|
||
}
|
||
}
|