Merge pull request #1140 from vangel76/perf/av1-wavefront-3106

AV1 decoding about 1.7x as fast with many threads
This commit is contained in:
Dylan M. Taylor authored and GitHub committed 2026-11-15 18:28:38 -05:00
commit faa36fc2ea
14 files changed
+1992 -197

No files matched your search

+49 -5
View File
@@ -73,11 +73,17 @@ with libdav1d: **bit-exact** on all 6 fixtures.
`Decoder::new()` uses all cores (`Decoder::with_threads(1)` decodes on the calling thread; on
wasm32, or without the default `threads` feature, everything runs on the calling thread). Headers are parsed in order; each frame is
then decoded by a frame worker once the reference frames it reads are finished, so independent
frames (hierarchical GOP layers) decode concurrently; tiles of a frame decode in parallel into
private buffers that are merged afterwards; the in-loop filters (deblocking, CDEF, loop
restoration) each run in parallel bands of 64 rows (`postfilter.rs` explains why the bands are
independent). Results never depend on scheduling (the oracle tests compare 1 thread, many threads
then decoded by a frame worker. With worker threads it first reads the symbols of all its tiles,
which needs only the parse results of its references (CDFs, segment ids, saved motion vectors),
publishes its own, and then reconstructs from the recorded blocks. Frames with several tiles
read the symbols of their tiles in parallel (one after another with intra block copy). A frame reconstructs superblock row by superblock
row (that row of every tile), each row waiting only for the reference rows its motion vectors
reach (row wavefront), and the in-loop filters (deblocking, CDEF, loop restoration) follow a few
64-row bands behind and publish each final band for the frames predicting from it
(`decoder.rs` and `postfilter.rs` give the dependency rules). Super-resolution frames wait for
their whole references and filter the whole frame. There is one frame worker per thread (at most
16), and the frames in flight are bounded by their size. Independent frames (hierarchical GOP
layers) decode concurrently. Results never depend on scheduling (the oracle tests compare 1 thread, many threads
and libdav1d). With frame threads, pictures may be returned by a
later `decode_pts` call or by `flush`; each carries the `pts` of the temporal unit that showed it.
@@ -130,6 +136,44 @@ motion vectors nor CDFs reach another frame — skip deblocking, CDEF and loop r
(super-resolution still runs, it sets the output size) and are flagged `Picture::draft`. In
SVT-AV1's hierarchical GOPs that is the top layer, half of the frames.
Frame threading on a Ryzen 7800X3D (8 cores / 16 threads, release `examples/av1_av1dec`, median of 7
alternating runs against Filmcraft `6909e461`, 2026-11-08), wall time for the whole file:
| Fixture | Threads | Before | After | Speed-up |
|---|---|---|---|---|
| `perf_1080p_gop` (60 frames) | 1 | 69.2 fps | 69.1 fps | 1.00x |
| `perf_1080p_gop` | 16 | 159.6 fps | 227.3 fps | 1.42x |
| `perf_2160p_gop` (30 frames) | 1 | 17.9 fps | 17.8 fps | 1.00x |
| `perf_2160p_gop` | 16 | 40.2 fps | 57.7 fps | 1.43x |
Peak RSS at 16 threads goes from 278 to 365 MB (1080p) and from 1031 to 1217 MB (2160p, +18%):
every frame in flight holds its parse record and its final samples are kept in 64-row bands.
With one frame worker per thread, the row wavefront for frames with several tiles and parse
waits limited to what parsing reads (same machine, median of 5 alternating runs, 2026-11-08):
| Fixture | Threads | Before (`6909e461`) | After | Speed-up | Peak RSS before / after |
|---|---|---|---|---|---|
| `perf_1080p_gop` | 4 | 95.7 fps | 185.2 fps | 1.94x | 148 / 236 MB |
| `perf_1080p_gop` | 16 | 155.0 fps | 259.7 fps | 1.68x | 274 / 560 MB |
| `perf_2160p_gop` | 4 | 23.6 fps | 47.1 fps | 2.00x | 512 / 798 MB |
| `perf_2160p_gop` | 16 | 39.9 fps | 65.8 fps | 1.65x | 994 / 1353 MB |
| `perf_1080p_gop_tiles` (4x2 tiles) | 16 | 186.3 fps | 237.2 fps | 1.27x | 328 / 617 MB |
| SVT-AV1 2160p, 4x2 tiles (30 frames) | 16 | 57.1 fps | 70.8 fps | 1.24x | 1197 / 1502 MB |
Known limitations of the frame threading (the output is the same in every case, only the
overlap is smaller):
- A frame whose parse record would pass 32 bytes per luma sample of a tile (pathological streams:
tiny blocks, large palettes, every coefficient coded), or that cannot reserve its record from
the decoder-wide budget of 1 GiB shared by the frames in flight, is decoded again in one pass.
- A superblock row using local warped motion, global motion beyond translation or a scaled
reference waits for the whole reference instead of the rows its motion vectors reach.
- Super-resolution frames do not use the row wavefront: they wait for their whole references
and filter the whole frame.
- The tile columns of a superblock row are reconstructed one after another on the frame's worker
(their symbols are still read in parallel).
## Limits
Frames beyond the AV1 level 6.x limits (Annex A: at most 16384 wide, 8704 high and 35 651 584 luma samples
+21 -9
View File
@@ -10,19 +10,33 @@ use crate::av1::decode::state::FrameState;
/// equals CurrFrame). Bands of 64 luma rows (whole 64x64 filter blocks) are filtered in
/// parallel: every block reads only CurrFrame and writes only its own samples.
pub(crate) fn apply(fs: &FrameState, pool: &Pool) -> Option<FrameBuf> {
if !active(fs) {
return None;
}
let mut out = fs.cur.clone();
let ranges = bands(pool, fs.fh.mi_rows as usize * 4);
filter_bands(fs, pool, out.row_bands(&ranges), &ranges);
Some(out)
}
/// Whether CDEF filters any block of the frame (it reads only parse results).
pub(crate) fn active(fs: &FrameState) -> bool {
let fh = &fs.fh;
if fh.coded_lossless || fh.allow_intrabc || !fs.seq.enable_cdef {
return None;
return false;
}
let (mi_rows, mi_cols) = (fh.mi_rows as usize, fh.mi_cols as usize);
let fb_rows = mi_rows.div_ceil(16);
let fb_cols = mi_cols.div_ceil(16);
if !(0..fb_rows).any(|fr| (0..fb_cols).any(|fc| fb_skip_mask(fs, fr * 16, fc * 16).is_some())) {
return None;
}
let mut out = fs.cur.clone();
let ranges = bands(pool, mi_rows * 4);
let mut planes = out.row_bands(&ranges);
(0..fb_rows).any(|fr| (0..fb_cols).any(|fc| fb_skip_mask(fs, fr * 16, fc * 16).is_some()))
}
/// Filter the bands `planes` covering luma rows `ranges` (whole 64x64 filter block rows), which
/// hold CurrFrame; reads CurrFrame up to 2 rows beyond each band.
pub(crate) fn filter_bands(fs: &FrameState, pool: &Pool, mut planes: Vec<[RowBand; 3]>, ranges: &[(usize, usize)]) {
let (mi_rows, mi_cols) = (fs.fh.mi_rows as usize, fs.fh.mi_cols as usize);
let fb_rows = mi_rows.div_ceil(16);
let fb_cols = mi_cols.div_ceil(16);
pool.map_mut(&mut planes, |b, band| {
let (start, end) = ranges[b];
let mut buf = [0i16; BUF_LEN];
@@ -34,8 +48,6 @@ pub(crate) fn apply(fs: &FrameState, pool: &Pool) -> Option<FrameBuf> {
}
}
});
drop(planes);
Some(out)
}
/// The filtered 8x8 blocks of the 64x64 filter block at (r, c): bit `(i << 3) | j` is set for
File diff suppressed because it is too large. Load diff
+166
View File
@@ -197,6 +197,50 @@ impl FrameBuf {
}
}
/// Copy luma rows [y0, y1) (chroma rows scaled down; clipped to the stored rows) of `src`, a
/// frame of the same geometry.
pub fn copy_rows_from(&mut self, src: &FrameBuf, y0: usize, y1: usize) {
for p in 0..self.num_planes {
let sy = if p == 0 { 0 } else { self.subsampling_y };
let (d, s) = (&mut self.planes[p], &src.planes[p]);
if d.stride != s.stride {
continue;
}
let a = (y0 >> sy).min(d.rows).min(s.rows) * d.stride;
let b = (y1 >> sy).min(d.rows).min(s.rows) * d.stride;
if let (Some(dst), Some(src)) = (d.data.get_mut(a..b), s.data.get(a..b)) {
dst.copy_from_slice(src);
}
}
}
/// Copies of luma rows [y0, y1) (chroma rows scaled down; clipped to the stored rows), one
/// plane region each.
pub fn rows_copy(&self, y0: usize, y1: usize) -> [Plane; 3] {
std::array::from_fn(|p| {
if p >= self.num_planes {
return Plane::default();
}
let sy = if p == 0 { 0 } else { self.subsampling_y };
let s = &self.planes[p];
let (a, b) = ((y0 >> sy).min(s.rows), (y1 >> sy).min(s.rows));
let mut out = Plane::region(0, a, s.stride, b - a);
if let Some(src) = s.data.get(a * s.stride..b * s.stride) {
out.data.copy_from_slice(src);
}
out
})
}
/// A frame of this geometry (samples not copied: zero).
pub fn same_geometry(&self) -> FrameBuf {
let planes = std::array::from_fn(|p| {
let pl = &self.planes[p];
Plane { data: zeroed(pl.data.len()), stride: pl.stride, rows: pl.rows, ox: pl.ox, oy: pl.oy }
});
FrameBuf { planes, ..*self }
}
/// [`Plane::row_bands`] of all three planes for ranges of luma rows (chroma rows scaled
/// down): element `i` holds the bands of range `i`.
pub fn row_bands(&mut self, luma: &[(usize, usize)]) -> Vec<[RowBand<'_>; 3]> {
@@ -208,6 +252,128 @@ impl FrameBuf {
}
}
/// Luma rows per published band of a [`BandFrame`] (log2).
pub const BAND_LOG2: u32 = 6;
/// One plane of a frame's final samples, published band by band while the rest of the frame
/// is still being decoded (see [`BandFrame`]). A band is written once and then only read.
pub struct BandPlane {
bands: Vec<std::sync::OnceLock<Plane>>,
stride: usize,
rows: usize,
log2: u32,
/// Returned for rows of a band that is not published: zeros instead of a panic. Readers wait
/// for the row progress first and references of another chroma format are refused, so
/// this only happens through a scheduling bug.
missing: Box<[u16]>,
}
impl BandPlane {
fn new(stride: usize, rows: usize, log2: u32) -> BandPlane {
let n = rows.div_ceil(1 << log2);
BandPlane { bands: (0..n).map(|_| std::sync::OnceLock::new()).collect(), stride, rows, log2, missing: vec![0; stride].into_boxed_slice() }
}
/// Row `y` (`stride` samples) of a published band.
#[inline(always)]
pub fn row(&self, y: usize) -> &[u16] {
let o = (y & ((1 << self.log2) - 1)) * self.stride;
match self.bands.get(y >> self.log2).and_then(|b| b.get()).and_then(|b| b.data.get(o..o + self.stride)) {
Some(r) => r,
None => {
debug_assert!(false, "row {y} read before its band was published");
&self.missing
}
}
}
/// Publish band `band` copied from the rows of the whole plane `src` (same geometry); a
/// band is published once.
fn publish(&self, band: usize, src: &Plane) {
let y0 = band << self.log2;
let y1 = ((band + 1) << self.log2).min(self.rows);
let mut p = Plane::region(0, y0, self.stride, y1.saturating_sub(y0));
let w = self.stride.min(src.stride);
for (y, out) in (y0..y1).zip(p.data.chunks_exact_mut(self.stride.max(1))) {
if let Some(r) = src.data.get(y * src.stride..y * src.stride + w) {
out[..w].copy_from_slice(r);
}
}
self.publish_owned(band, p);
}
/// Publish band `band` as `p`, whose rows start at the band's first row (stride `stride`;
/// rows past the band are ignored).
fn publish_owned(&self, band: usize, p: Plane) {
if let Some(slot) = self.bands.get(band)
&& p.stride == self.stride
{
let _ = slot.set(p);
}
}
}
/// A frame's final samples (every in-loop filter applied) in bands of 64 luma rows, so later
/// frames can predict from the top of a reference while its bottom is still being decoded.
/// Each band is published once, before the frame's row progress says it is final.
pub struct BandFrame {
pub planes: [BandPlane; 3],
pub num_planes: usize,
pub subsampling_x: usize,
pub subsampling_y: usize,
}
impl BandFrame {
/// Bands for the visible rows of a `width` x `height` frame stored like [`FrameBuf::new`].
pub fn new(width: usize, height: usize, num_planes: usize, subsampling_x: usize, subsampling_y: usize) -> BandFrame {
let aw = width.div_ceil(128) * 128 + 160;
let planes = std::array::from_fn(|p| {
if p >= num_planes {
return BandPlane::new(0, 0, BAND_LOG2);
}
let (sx, sy) = if p == 0 { (0, 0) } else { (subsampling_x, subsampling_y) };
BandPlane::new(aw >> sx, (height + sy) >> sy, BAND_LOG2 - sy as u32)
});
BandFrame { planes, num_planes, subsampling_x, subsampling_y }
}
/// Number of bands (of 64 luma rows).
pub fn bands(&self) -> usize {
self.planes[0].bands.len()
}
/// Publish luma band `band` (and the chroma rows it covers) copied from `src`.
pub fn publish(&self, band: usize, src: &FrameBuf) {
for p in 0..self.num_planes {
self.planes[p].publish(band, &src.planes[p]);
}
}
/// Publish luma band `band` from per-plane buffers starting at the band's first row.
pub fn publish_owned(&self, band: usize, planes: [Plane; 3]) {
for (p, pl) in planes.into_iter().enumerate().take(self.num_planes) {
self.planes[p].publish_owned(band, pl);
}
}
}
/// A plane to predict from: a whole stored plane or the published bands of a reference.
#[derive(Clone, Copy)]
pub enum PlaneRef<'a> {
Whole(&'a Plane),
Bands(&'a BandPlane),
}
impl<'a> PlaneRef<'a> {
#[inline(always)]
pub fn row(&self, y: usize) -> &'a [u16] {
match *self {
PlaneRef::Whole(p) => p.row(y),
PlaneRef::Bands(b) => b.row(y),
}
}
}
/// A motion vector (row, col) in 1/8 sample units.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct Mv {
+17 -12
View File
@@ -3,7 +3,7 @@
use std::sync::OnceLock;
use crate::av1::decode::frame::Plane;
use crate::av1::decode::frame::{Plane, PlaneRef};
use crate::av1::decode::mvpred::round2_signed64;
use crate::av1::decode::spec_tables::{
BLOCK_8X8, BLOCK_SIZES, COMPOUND_AVERAGE, COMPOUND_DIFFWTD, COMPOUND_DISTANCE, COMPOUND_INTRA, COMPOUND_WEDGE, DIV_LUT, DIV_LUT_BITS,
@@ -227,12 +227,9 @@ impl TileDecoder<'_, '_> {
(start_x as i32, start_y as i32, step_x as i32, step_y as i32)
}
fn ref_plane(&self, ref_idx: i32, plane: usize) -> &Plane {
fn ref_plane(&self, ref_idx: i32, plane: usize) -> PlaneRef<'_> {
// A missing reference (rejected earlier for valid streams) predicts from the current frame.
match usize::try_from(ref_idx).ok().and_then(|i| self.fs.refs.get(i)).and_then(|r| r.as_ref()) {
Some(r) => &r.buf.planes[plane],
None => &self.t.cur.planes[plane],
}
self.fs.ref_samples(ref_idx, plane).unwrap_or(PlaneRef::Whole(&self.t.cur.planes[plane]))
}
/// Block inter prediction process (7.11.3.4) into `pred` (stride PRED_STRIDE); `tmp` holds
@@ -512,12 +509,12 @@ impl TileDecoder<'_, '_> {
return false;
}
let fs = self.fs;
let Some(r) = usize::try_from(ref_idx).ok().and_then(|i| fs.refs.get(i)).and_then(|r| r.as_ref()) else {
let Some(refp) = fs.ref_samples(ref_idx, plane) else {
return false;
};
let dims = self.ref_plane_dims(ref_idx, plane);
let max = ((1u32 << fs.bit_depth) - 1) as u16;
copy_fullpel(&r.buf.planes[plane], dims, McBlock { x: sx, y: sy, w, h }, &mut self.t.cur.planes[plane], (x, y), max);
copy_fullpel(refp, dims, McBlock { x: sx, y: sy, w, h }, &mut self.t.cur.planes[plane], (x, y), max);
true
}
@@ -763,7 +760,7 @@ impl TileDecoder<'_, '_> {
/// Unscaled single-reference prediction at a whole-sample position: both filter phases are the
/// identity and Round2(128 * Round2(128 * v, InterRound0), InterRound1) == v for every bit
/// depth, so the prediction is the (clamped) reference sample.
fn copy_fullpel(refp: &Plane, (last_x, last_y): (i32, i32), src_blk: McBlock, dst: &mut Plane, (dx, dy): (usize, usize), max: u16) {
fn copy_fullpel(refp: PlaneRef, (last_x, last_y): (i32, i32), src_blk: McBlock, dst: &mut Plane, (dx, dy): (usize, usize), max: u16) {
let McBlock { x, y, w, h } = src_blk;
let x0 = x >> 10;
let y0 = y >> 10;
@@ -831,7 +828,7 @@ fn vpass<const W: usize>(rows: &[i32], vf: &[i16; 8], out: &mut [i32], half: i32
/// Block inter prediction without reference scaling: every column / row uses the same filter
/// phase, so the filters run over whole rows (bit-identical to the per-sample process).
fn mc_unscaled(
refp: &Plane,
refp: PlaneRef,
(last_x, last_y): (i32, i32),
blk: McBlock,
(hf, vf): (&[i16; 8], &[i16; 8]),
@@ -982,9 +979,17 @@ mod tests {
[(0, 0), (-5, -3), (7, 9), (last_x - w as i32 / 2, last_y - h as i32 / 2), (last_x + 3, last_y + 2), (-(w as i32) - 4, 3)]
{
let (x, y) = ((px << 10) | 32, (py << 10) | 32);
mc_unscaled(&refp, (last_x, last_y), McBlock { x, y, w, h }, (&IDENTITY_TAPS, &IDENTITY_TAPS), rnd, &mut pred, &mut tmp);
mc_unscaled(
PlaneRef::Whole(&refp),
(last_x, last_y),
McBlock { x, y, w, h },
(&IDENTITY_TAPS, &IDENTITY_TAPS),
rnd,
&mut pred,
&mut tmp,
);
let mut dst = Plane::new(w + 3, h);
copy_fullpel(&refp, (last_x, last_y), McBlock { x, y, w, h }, &mut dst, (3, 0), max);
copy_fullpel(PlaneRef::Whole(&refp), (last_x, last_y), McBlock { x, y, w, h }, &mut dst, (3, 0), max);
for i in 0..h {
for j in 0..w {
let want = spec_phase0(&refp, (last_x, last_y), (px, py), (j, i), (w * 3 + h) % SUBPEL_FILTERS.len(), rnd)
+146 -14
View File
@@ -1,7 +1,7 @@
//! In-loop post filters run by decode_frame_wrapup: loop filter (7.14), CDEF (7.15),
//! super-resolution upscaling (7.16) and loop restoration (7.17).
use crate::av1::decode::frame::RowBand;
use crate::av1::decode::frame::{FrameBuf, Plane, RowBand};
use crate::av1::decode::spec_tables::{
GLOBAL_GLOBALMV, GLOBALMV, INTRA_FRAME, MAX_LOOP_FILTER, NEARESTMV, NUM_4X4_BLOCKS_HIGH, NUM_4X4_BLOCKS_WIDE, SEG_LVL_ALT_LF_Y_V, TX_HEIGHT,
TX_WIDTH,
@@ -48,6 +48,121 @@ pub(crate) fn apply(fs: &mut FrameState, stats: &mut DecodeStats, pool: &crate::
stats.add(Stage::Restoration, t.lap());
}
/// The in-loop filters run band by band behind reconstruction (row wavefront): each stage of a
/// band starts once the stage before has finished the band below it, which bounds what the
/// stage reads beyond the band (the loop filter changes up to 7 rows above a band and intra
/// prediction of the next superblock row reads the unfiltered row above it; CDEF reads 2 rows
/// and loop restoration 3 rows beyond a band). Bands are 64 luma rows. The result is the same
/// as running each filter over the whole frame. Super-resolution frames use [`apply`].
///
/// The last active stage writes each band into its own buffers, which are handed out as the
/// band's final samples without a copy.
pub(crate) struct Pipeline {
bands: Vec<(usize, usize)>,
/// Luma rows of the coded frame (mi_rows * 4).
rows: usize,
lf_on: bool,
cdef_on: bool,
lr_on: bool,
/// Bands finished by each stage.
lf: usize,
cdef: usize,
lr: usize,
pending: [Vec<u16>; 3],
/// CdefFrame when loop restoration reads it.
cdef_out: Option<FrameBuf>,
}
/// Final bands from [`Pipeline::advance`]: (band, its samples starting at the band's first row).
pub(crate) type FinalBands = Vec<(usize, [Plane; 3])>;
impl Pipeline {
pub fn new(fs: &FrameState, draft: bool) -> Pipeline {
let rows = fs.fh.mi_rows as usize * 4;
let n = rows.div_ceil(BAND_ROWS).max(1);
let bands = (0..n).map(|b| (b * BAND_ROWS, if b + 1 == n { usize::MAX } else { (b + 1) * BAND_ROWS })).collect();
let lvl = fs.fh.lf.level;
let cdef_on = !draft && crate::av1::decode::cdef::active(fs);
let lr_on = fs.fh.lr.uses_lr && !draft;
let cdef_out = (cdef_on && lr_on).then(|| fs.cur.same_geometry());
Pipeline {
bands,
rows,
lf_on: (lvl[0] != 0 || lvl[1] != 0) && !draft,
cdef_on,
lr_on,
lf: 0,
cdef: 0,
lr: 0,
pending: Default::default(),
cdef_out,
}
}
/// Filter what luma rows [0, `recon_rows`) of reconstructed samples allow (everything once
/// they cover the frame); returns the bands whose samples became final.
pub fn advance(&mut self, fs: &mut FrameState, stats: &mut DecodeStats, pool: &crate::av1::decode::par::Pool, recon_rows: usize) -> FinalBands {
let n = self.bands.len();
let all = recon_rows >= self.rows;
let mut out = FinalBands::new();
let mut t = Timer::start();
// The loop filter of a band waits for the band below it to be reconstructed.
let lf_to = if all { n } else { (recon_rows / BAND_ROWS).saturating_sub(1) };
if lf_to > self.lf {
if self.lf_on {
loop_filter_bands(fs, pool, &self.bands, self.lf..lf_to, &mut self.pending);
}
self.lf = lf_to;
}
stats.add(Stage::LoopFilter, t.lap());
let cdef_to = if all { n } else { self.lf.saturating_sub(1) };
if cdef_to > self.cdef {
let r = &self.bands[self.cdef..cdef_to];
if self.cdef_on {
if let Some(cdef) = &mut self.cdef_out {
cdef.copy_rows_from(&fs.cur, r[0].0, r[r.len() - 1].1);
crate::av1::decode::cdef::filter_bands(fs, pool, cdef.row_bands(r), r);
} else {
let mut own: Vec<[Plane; 3]> = r.iter().map(|&(a, b)| fs.cur.rows_copy(a, b)).collect();
crate::av1::decode::cdef::filter_bands(fs, pool, own_bands(&mut own), r);
out.extend((self.cdef..cdef_to).zip(own));
}
} else if !self.lr_on {
// The loop filter is the last stage: the band is final once the band below it
// is filtered.
out.extend((self.cdef..cdef_to).zip(r.iter().map(|&(a, b)| fs.cur.rows_copy(a, b))));
}
self.cdef = cdef_to;
}
stats.add(Stage::Cdef, t.lap());
let lr_to = if all { n } else { self.cdef.saturating_sub(1) };
if self.lr_on && lr_to > self.lr {
let r = &self.bands[self.lr..lr_to];
let cdef = self.cdef_out.as_ref().unwrap_or(&fs.cur);
let mut own: Vec<[Plane; 3]> = r.iter().map(|&(a, b)| cdef.rows_copy(a, b)).collect();
crate::av1::decode::restoration::restore_bands(fs, &fs.cur, cdef, own_bands(&mut own), pool, r);
out.extend((self.lr..lr_to).zip(own));
self.lr = lr_to;
}
stats.add(Stage::Restoration, t.lap());
out
}
}
/// Row bands over per-band buffers made by [`FrameBuf::rows_copy`].
fn own_bands(own: &mut [[Plane; 3]]) -> Vec<[RowBand<'_>; 3]> {
fn band(p: &mut Plane) -> RowBand<'_> {
let (stride, y0) = (p.stride.max(1), p.oy);
RowBand { data: p.data.as_mut_slice(), stride, y0 }
}
own.iter_mut()
.map(|planes| {
let [a, b, c] = planes;
[band(a), band(b), band(c)]
})
.collect()
}
// ---------------------------------------------------------------------------------------------
// Loop filter (7.14)
@@ -69,23 +184,40 @@ const LF_MARGIN: usize = 8;
/// written back; the band above never touches them.
fn loop_filter(fs: &mut FrameState, pool: &crate::av1::decode::par::Pool) {
let luma = bands(pool, fs.fh.mi_rows as usize * 4);
loop_filter_bands(fs, pool, &luma, 0..luma.len(), &mut Default::default());
}
/// The loop filter on bands `range` of `all` (the frame's bands). The bands above the range
/// are filtered already: `pending` holds the unfiltered rows above the first band of the range
/// (empty for band 0), saved before the band above was filtered, and is set to those of the
/// band after the range.
fn loop_filter_bands(
fs: &mut FrameState,
pool: &crate::av1::decode::par::Pool,
all: &[(usize, usize)],
range: std::ops::Range<usize>,
pending: &mut [Vec<u16>; 3],
) {
let Some(luma) = all.get(range.clone()) else { return };
let planes: Vec<usize> = (0..fs.num_planes).filter(|&p| p == 0 || fs.fh.lf.level[1 + p] != 0).collect();
let mut cur = std::mem::take(&mut fs.cur);
let f: &FrameState = fs;
// Unfiltered copies of the rows above each band but the first.
let margins: Vec<[Vec<u16>; 3]> = luma
.iter()
.map(|&(start, _)| {
std::array::from_fn(|p| {
let sub_y = if p == 0 { 0 } else { f.ssy };
let pl = &cur.planes[p];
let (y0, y1) = ((start.saturating_sub(LF_MARGIN)) >> sub_y, start >> sub_y);
let rows = pl.data.get(y0 * pl.stride..y1 * pl.stride);
if start == 0 || !planes.contains(&p) { Vec::new() } else { rows.map(<[u16]>::to_vec).unwrap_or_default() }
})
// Unfiltered copies of the rows above a band (none above the first).
let margin = |cur: &crate::av1::decode::frame::FrameBuf, start: usize| -> [Vec<u16>; 3] {
std::array::from_fn(|p| {
let sub_y = if p == 0 { 0 } else { f.ssy };
let pl = &cur.planes[p];
let (y0, y1) = ((start.saturating_sub(LF_MARGIN)) >> sub_y, start >> sub_y);
let rows = pl.data.get(y0 * pl.stride..y1 * pl.stride);
if start == 0 || !planes.contains(&p) { Vec::new() } else { rows.map(<[u16]>::to_vec).unwrap_or_default() }
})
.collect();
let mut bands: Vec<([RowBand; 3], [Vec<u16>; 3])> = cur.row_bands(&luma).into_iter().zip(margins).collect();
};
let margins: Vec<[Vec<u16>; 3]> =
luma.iter().enumerate().map(|(i, &(start, _))| if i == 0 { std::mem::take(pending) } else { margin(&cur, start) }).collect();
if let Some(&(next, _)) = all.get(range.end) {
*pending = margin(&cur, next);
}
let mut bands: Vec<([RowBand; 3], [Vec<u16>; 3])> = cur.row_bands(luma).into_iter().zip(margins).collect();
let changed = pool.map_mut(&mut bands, |b, (band, margin)| {
let mut changed: [Vec<(usize, u16)>; 3] = Default::default();
for &p in &planes {
+9 -5
View File
@@ -70,10 +70,16 @@ struct LrPlane<'a> {
/// Bands of 64 luma rows are restored in parallel: every unit reads only the two input frames
/// and writes only its own samples.
pub(crate) fn loop_restoration(fs: &FrameState, cur: &FrameBuf, cdef: &FrameBuf, pool: &Pool) -> FrameBuf {
let fh = &fs.fh;
let mut lr = cdef.clone();
let ranges = bands(pool, fh.frame_height as usize);
let mut planes = lr.row_bands(&ranges);
let ranges = bands(pool, fs.fh.frame_height as usize);
restore_bands(fs, cur, cdef, lr.row_bands(&ranges), pool, &ranges);
lr
}
/// Restore the bands `planes` covering luma rows `ranges`, which hold UpscaledCdefFrame; reads
/// both inputs up to 3 rows beyond each band.
pub(crate) fn restore_bands(fs: &FrameState, cur: &FrameBuf, cdef: &FrameBuf, mut planes: Vec<[RowBand; 3]>, pool: &Pool, ranges: &[(usize, usize)]) {
let fh = &fs.fh;
pool.map_mut(&mut planes, |b, band| {
let (start, end) = ranges[b];
let mut scratch = Scratch::default();
@@ -100,8 +106,6 @@ pub(crate) fn loop_restoration(fs: &FrameState, cur: &FrameBuf, cdef: &FrameBuf,
p.restore(out, start >> sub_y, end >> sub_y, &mut scratch);
}
});
drop(planes);
lr
}
/// Row buffers reused across stripes.
+27 -2
View File
@@ -6,8 +6,8 @@
//! with several, each tile decodes into private buffers covering its area (tiles never read
//! outside themselves), which are then copied into the frame.
use crate::av1::decode::decoder::RefFrame;
use crate::av1::decode::frame::{FrameBuf, MiInfo};
use crate::av1::decode::decoder::{Parsed, RefFrame};
use crate::av1::decode::frame::{BandFrame, FrameBuf, MiInfo, PlaneRef};
use crate::av1::decode::header::{FrameHeader, RefInfo, SequenceHeader};
use crate::av1::decode::spec_tables::{DEFAULT_INTRA_FRAME_Y_MODE_CDF, NUM_REF_FRAMES, RESTORE_NONE, SUBSAMPLED_SIZE};
use std::ops::{Deref, DerefMut};
@@ -26,6 +26,11 @@ pub(crate) struct FrameShared {
pub cdef_cols: usize,
/// Reference slots (FrameStore and saved state) used by this frame.
pub refs: [Option<Arc<RefFrame>>; NUM_REF_FRAMES],
/// Parse results (CDFs, segment ids, saved motion vectors) of the same slots.
pub parsed: [Option<Arc<Parsed>>; NUM_REF_FRAMES],
/// The published bands of references that are still being decoded (row wavefront): read
/// only below the row progress that was waited for.
pub ref_rows: [Option<Arc<BandFrame>>; NUM_REF_FRAMES],
pub ref_info: [RefInfo; NUM_REF_FRAMES],
/// MotionFieldMvs[ ref ][ y8 * w8 + x8 ] (7.9); empty when unused.
pub motion_field: [Vec<[i32; 2]>; 8],
@@ -34,6 +39,16 @@ pub(crate) struct FrameShared {
}
impl FrameShared {
/// The samples of reference slot `ref_idx` to predict from (None for a missing reference).
#[inline]
pub fn ref_samples(&self, ref_idx: i32, plane: usize) -> Option<PlaneRef<'_>> {
let i = usize::try_from(ref_idx).ok()?;
if let Some(b) = self.ref_rows.get(i)?.as_ref() {
return b.planes.get(plane).map(PlaneRef::Bands);
}
self.refs.get(i)?.as_ref().map(|r| r.plane(plane))
}
pub fn plane_residual_size(&self, subsize: usize, plane: usize) -> usize {
let sx = if plane > 0 { self.ssx } else { 0 };
let sy = if plane > 0 { self.ssy } else { 0 };
@@ -210,6 +225,8 @@ impl FrameState {
lr_unit_cols,
cdef_cols,
refs: Default::default(),
parsed: Default::default(),
ref_rows: Default::default(),
ref_info: Default::default(),
motion_field: Default::default(),
prev_segment_ids: vec![0; mi_cols * mi_rows],
@@ -283,6 +300,14 @@ impl FrameState {
pub fn merge_tile_state(&mut self, t: &TileState) {
for p in 0..self.num_planes {
t.cur.planes[p].copy_into(&mut self.cur.planes[p]);
}
self.merge_tile_modes(t);
}
/// Copy a tile's private buffers except the samples (after parsing, before reconstruction)
/// into the frame.
pub fn merge_tile_modes(&mut self, t: &TileState) {
for p in 0..self.num_planes {
t.lf_tx_size[p].copy_into(&mut self.lf_tx_size[p]);
for &u in &t.lr_written[p] {
self.lr.lr_type[p][u] = t.lr.lr_type[p][u];
+303 -11
View File
@@ -94,6 +94,191 @@ pub(crate) struct Block {
pub local_warp_params: [i32; 6],
}
/// What a tile decoder does with the blocks it visits.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub(crate) enum Pass {
/// Read the symbols and reconstruct each block right away (single-threaded decoding).
Full,
/// Read the symbols only, recording what reconstruction needs in a [`TileRecord`].
Parse,
/// Reconstruct from a [`TileRecord`], reading no symbols.
Reconstruct,
}
/// The per-block values reconstruction needs beyond the mode info (which is final after
/// parsing): the block variables as parsed, its quantizer index and the warp samples / palette
/// color maps when used.
#[derive(Clone)]
pub(crate) struct BlockRecord {
b: Block,
current_q_index: i32,
warp_samples: Option<Box<(usize, [[i32; 4]; crate::av1::decode::spec_tables::LEAST_SQUARES_SAMPLES_MAX])>>,
color_maps: Option<Box<(Vec<u8>, Vec<u8>)>>,
}
/// A parsed tile: its blocks in decoding order, how many of them each superblock holds and the
/// coefficients of every coded transform block as a stream of
/// `eob, tx_type, count, (position, value) * count` (only non-zero values).
#[derive(Default)]
pub(crate) struct TileRecord {
blocks: Vec<BlockRecord>,
sb_blocks: Vec<u32>,
coeffs: Vec<i32>,
/// Read positions while reconstructing.
next_block: usize,
next_coeff: usize,
next_sb: usize,
next_sb_row: usize,
/// Bytes recorded and the most allowed.
pub bytes: usize,
pub limit: usize,
/// The decoder-wide budget the recorded bytes are reserved from, and how much is reserved.
pub budget: Option<std::sync::Arc<RecordBudget>>,
reserved: usize,
}
impl Drop for TileRecord {
fn drop(&mut self) {
if let Some(b) = &self.budget {
b.release(self.reserved);
}
}
}
/// The parse record bytes all frames of a decoder may hold at once (frames in flight decode in
/// one pass once it is used up).
pub(crate) struct RecordBudget {
used: std::sync::atomic::AtomicUsize,
max: usize,
/// Most bytes ever reserved at once (tests).
#[cfg(test)]
pub peak: std::sync::atomic::AtomicUsize,
}
/// Bytes reserved from a [`RecordBudget`] at a time.
const RESERVE_STEP: usize = 1 << 20;
impl RecordBudget {
pub fn new(max: usize) -> RecordBudget {
RecordBudget {
used: Default::default(),
max,
#[cfg(test)]
peak: Default::default(),
}
}
/// Reserve `n` bytes; false (nothing reserved) if that would go over the budget.
fn reserve(&self, n: usize) -> bool {
use std::sync::atomic::Ordering;
let ok = self.used.try_update(Ordering::AcqRel, Ordering::Acquire, |u| u.checked_add(n).filter(|&v| v <= self.max)).is_ok();
#[cfg(test)]
if ok {
self.peak.fetch_max(self.used.load(Ordering::Acquire), Ordering::AcqRel);
}
ok
}
fn release(&self, n: usize) {
self.used.fetch_sub(n, std::sync::atomic::Ordering::AcqRel);
}
/// Bytes reserved now (tests).
#[cfg(test)]
pub fn used(&self) -> usize {
self.used.load(std::sync::atomic::Ordering::Acquire)
}
}
/// The motion of a recorded block (what a reference row wait depends on).
pub(crate) struct BlockMotion {
pub mi_row: usize,
pub bh4: usize,
/// Inter prediction from reference frames (not intra block copy).
pub inter: bool,
pub ref_frame: [i8; 2],
/// Mv[ list ] (row, col) in 1/8 samples.
pub mv: [[i32; 2]; 2],
pub motion_mode: u8,
pub y_mode: usize,
}
impl TileRecord {
/// Whether recording stopped at [`TileRecord::limit`] (the record is unusable).
pub fn over_limit(&self) -> bool {
self.bytes > self.limit
}
/// Account for `n` more recorded bytes; false once over the limit (nothing more is kept).
fn grow(&mut self, n: usize) -> bool {
self.bytes = self.bytes.saturating_add(n);
if self.bytes > self.limit {
return false;
}
if self.bytes > self.reserved
&& let Some(b) = &self.budget
{
let step = (self.bytes - self.reserved).max(RESERVE_STEP);
if !b.reserve(step) {
// The decoder-wide budget is used up: this frame is decoded in one pass.
self.limit = 0;
return false;
}
self.reserved += step;
}
true
}
/// The recorded blocks of superblock rows [first, end), for a tile `sbs_per_row` superblocks wide.
pub fn row_blocks(&self, sbs_per_row: usize, first: usize, end: usize) -> impl Iterator<Item = BlockMotion> + '_ {
let start: usize = self.sb_blocks.iter().take(first * sbs_per_row).map(|&n| n as usize).sum();
let count: usize = self.sb_blocks.iter().skip(first * sbs_per_row).take((end - first) * sbs_per_row).map(|&n| n as usize).sum();
self.blocks.iter().skip(start).take(count).map(|r| BlockMotion {
mi_row: r.b.mi_row,
bh4: r.b.bh4,
inter: r.b.is_inter && !r.b.use_intrabc,
ref_frame: r.b.ref_frame,
mv: r.b.mv_i,
motion_mode: r.b.motion_mode,
y_mode: r.b.y_mode,
})
}
fn next_coeff(&mut self) -> Result<i32> {
let v = *self.coeffs.get(self.next_coeff).ok_or(crate::av1::decode::Error::Invalid("coefficient record exhausted"))?;
self.next_coeff += 1;
Ok(v)
}
}
/// The work buffers of a [`TileDecoder`] (`quant` all zero between transform blocks).
pub(crate) struct Buffers {
quant: Vec<i32>,
dequant: Vec<i32>,
residual: Vec<i32>,
color_map_y: Vec<u8>,
color_map_uv: Vec<u8>,
pred_buf: [Vec<i32>; 2],
mask: Vec<i32>,
mc_tmp: Vec<i32>,
levels: Vec<u8>,
}
impl Buffers {
pub fn new() -> Buffers {
Buffers {
quant: vec![0; 1024],
dequant: vec![0; 64 * 64],
residual: vec![0; 64 * 64],
color_map_y: vec![0; 64 * 64],
color_map_uv: vec![0; 64 * 64],
pred_buf: [vec![0; 128 * 128], vec![0; 128 * 128]],
mask: vec![0; 128 * 128],
mc_tmp: vec![0; (2 * 128 + 8) * 128],
levels: vec![0; (32 + TX_PAD) * (32 + TX_PAD)],
}
}
}
/// Decoder for one tile.
pub(crate) struct TileDecoder<'a, 'f> {
pub fs: &'f FrameShared,
@@ -128,6 +313,9 @@ pub(crate) struct TileDecoder<'a, 'f> {
pub mc_tmp: Vec<i32>,
/// Coefficient levels of the current transform block, padded (see TX_PAD).
levels: Vec<u8>,
pub pass: Pass,
/// Written by [`Pass::Parse`], read by [`Pass::Reconstruct`].
pub record: TileRecord,
}
const SB_MAX: usize = 32; // superblock size in 4x4 units (128 / 4)
@@ -141,6 +329,21 @@ fn block_height(bs: usize) -> usize {
impl<'a, 'f> TileDecoder<'a, 'f> {
pub fn new(fs: &'f FrameShared, t: &'f mut TileState, data: &'a [u8], cdf: Box<Cdfs>, tile_row: usize, tile_col: usize) -> Self {
Self::with_buffers(fs, t, data, cdf, tile_row, tile_col, Buffers::new())
}
/// [`TileDecoder::new`] reusing the work buffers of an earlier decoder of the same frame
/// (see [`TileDecoder::into_buffers`]).
pub fn with_buffers(
fs: &'f FrameShared,
t: &'f mut TileState,
data: &'a [u8],
cdf: Box<Cdfs>,
tile_row: usize,
tile_col: usize,
bufs: Buffers,
) -> Self {
let Buffers { quant, dequant, residual, color_map_y, color_map_uv, pred_buf, mask, mc_tmp, levels } = bufs;
let ti = &fs.fh.tile_info;
let (rs, re) = (ti.mi_row_starts[tile_row] as usize, ti.mi_row_starts[tile_row + 1] as usize);
let (cs, ce) = (ti.mi_col_starts[tile_col] as usize, ti.mi_col_starts[tile_col + 1] as usize);
@@ -164,19 +367,27 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
b: Block::default(),
mvs: Default::default(),
nb: Default::default(),
quant: vec![0; 1024],
dequant: vec![0; 64 * 64],
residual: vec![0; 64 * 64],
color_map_y: vec![0; 64 * 64],
color_map_uv: vec![0; 64 * 64],
quant,
dequant,
residual,
color_map_y,
color_map_uv,
plane_tx_type: 0,
pred_buf: [vec![0; 128 * 128], vec![0; 128 * 128]],
mask: vec![0; 128 * 128],
mc_tmp: vec![0; (2 * 128 + 8) * 128],
levels: vec![0; (32 + TX_PAD) * (32 + TX_PAD)],
pred_buf,
mask,
mc_tmp,
levels,
pass: Pass::Full,
record: TileRecord::default(),
}
}
/// The work buffers and CDFs, for [`TileDecoder::with_buffers`].
pub fn into_buffers(self) -> (Buffers, Box<Cdfs>) {
let TileDecoder { quant, dequant, residual, color_map_y, color_map_uv, pred_buf, mask, mc_tmp, levels, cdf, .. } = self;
(Buffers { quant, dequant, residual, color_map_y, color_map_uv, pred_buf, mask, mc_tmp, levels }, cdf)
}
fn seq(&self) -> &SequenceHeader {
&self.fs.seq
}
@@ -189,8 +400,11 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
c >= self.mi_col_start as isize && c < self.mi_col_end as isize && r >= self.mi_row_start as isize && r < self.mi_row_end as isize
}
/// decode_tile( )
/// decode_tile( ), or its reconstruction from [`TileDecoder::record`] in [`Pass::Reconstruct`].
pub fn decode_tile(&mut self) -> Result<()> {
if self.pass == Pass::Reconstruct {
return self.reconstruct_tile();
}
// clear_above_context( )
let t = &mut *self.t;
for p in 0..3 {
@@ -223,7 +437,11 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
self.clear_cdef(r, c);
self.clear_block_decoded_flags(r, c, sb4);
self.read_lr(r, c, sb_size);
let before = self.record.blocks.len();
self.decode_partition(r, c, sb_size)?;
if self.pass == Pass::Parse {
self.record.sb_blocks.push((self.record.blocks.len() - before) as u32);
}
c += sb4;
}
r += sb4;
@@ -231,6 +449,76 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
Ok(())
}
/// The reconstruction half of decode_tile( ): the superblocks in the same order, each block
/// predicted and reconstructed from its record.
fn reconstruct_tile(&mut self) -> Result<()> {
self.reconstruct_sb_rows(usize::MAX)
}
/// Reconstruct the next `rows` superblock rows of the tile (the record keeps the position).
pub fn reconstruct_sb_rows(&mut self, rows: usize) -> Result<()> {
let sb4 = if self.seq().use_128x128_superblock { 32 } else { 16 };
let mut r = self.mi_row_start + self.record.next_sb_row * sb4;
let mut done = 0;
while r < self.mi_row_end && done < rows {
let mut c = self.mi_col_start;
while c < self.mi_col_end {
self.clear_block_decoded_flags(r, c, sb4);
let n = *self.record.sb_blocks.get(self.record.next_sb).ok_or(crate::av1::decode::Error::Invalid("block record exhausted"))?;
self.record.next_sb += 1;
for _ in 0..n {
self.reconstruct_block()?;
}
c += sb4;
}
r += sb4;
done += 1;
self.record.next_sb_row += 1;
}
Ok(())
}
fn reconstruct_block(&mut self) -> Result<()> {
let rec = self.record.blocks.get(self.record.next_block).ok_or(crate::av1::decode::Error::Invalid("block record exhausted"))?;
self.record.next_block += 1;
self.b = rec.b.clone();
self.current_q_index = rec.current_q_index;
if let Some(w) = &rec.warp_samples {
self.mvs.num_samples = w.0;
self.mvs.cand_list = w.1;
}
if let Some(maps) = &rec.color_maps {
if let Some(d) = self.color_map_y.get_mut(..maps.0.len()) {
d.copy_from_slice(&maps.0);
}
if let Some(d) = self.color_map_uv.get_mut(..maps.1.len()) {
d.copy_from_slice(&maps.1);
}
}
self.compute_prediction()?;
self.residual()
}
/// Records the block for [`Pass::Reconstruct`] (its variables as parsed).
fn record_block(&mut self) {
let b = &self.b;
// The rows of the color maps (stride 64) the block's palette prediction reads.
let map_rows = |n: usize, h: usize| if n > 0 { (h + 2).min(64) * 64 } else { 0 };
let rows_y = map_rows(b.palette_size_y, block_height(b.mi_size));
let rows_uv = map_rows(b.palette_size_uv, block_height(b.mi_size) >> self.fs.ssy);
let size = std::mem::size_of::<BlockRecord>()
+ rows_y
+ rows_uv
+ if b.motion_mode as usize == crate::av1::decode::spec_tables::LOCALWARP { 160 } else { 0 };
if !self.record.grow(size) {
return;
}
let warp_samples =
(b.motion_mode as usize == crate::av1::decode::spec_tables::LOCALWARP).then(|| Box::new((self.mvs.num_samples, self.mvs.cand_list)));
let color_maps = (rows_y + rows_uv > 0).then(|| Box::new((self.color_map_y[..rows_y].to_vec(), self.color_map_uv[..rows_uv].to_vec())));
self.record.blocks.push(BlockRecord { b: b.clone(), current_q_index: self.current_q_index, warp_samples, color_maps });
}
fn clear_cdef(&mut self, r: usize, c: usize) {
let t = &mut *self.t;
t.set_cdef_idx(r, c, -1);
@@ -461,7 +749,11 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
}
}
}
self.compute_prediction()?;
if self.pass == Pass::Parse {
self.record_block();
} else {
self.compute_prediction()?;
}
self.residual()?;
let b = &self.b;
let delta_lf = self.delta_lf;
@@ -459,6 +459,58 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
}
}
/// Moves the coefficients `coeffs` left in `quant` into the record (leaving `quant` zero, as
/// reconstruction does).
pub(super) fn record_coeffs(&mut self, eob: usize, tx_sz: usize) {
let scan = self.get_scan(tx_sz);
let scan = scan.get(..eob).unwrap_or(scan);
if !self.record.grow(4 * (3 + 2 * scan.len())) {
// Over the limit: the record is dropped (the frame is decoded again in one pass);
// `quant` must still be left zero.
for &pos in scan {
if let Some(q) = self.quant.get_mut(pos as usize) {
*q = 0;
}
}
return;
}
let rec = &mut self.record.coeffs;
rec.push(eob as i32);
if eob == 0 {
return;
}
rec.push(self.plane_tx_type as i32);
let count_at = rec.len();
rec.push(0);
let mut count = 0;
for &pos in scan {
let v = std::mem::take(&mut self.quant[pos as usize]);
if v != 0 {
rec.push(pos as i32);
rec.push(v);
count += 1;
}
}
rec[count_at] = count;
}
/// The next transform block's coefficients from the record into `quant` (and its type);
/// returns its eob.
pub(super) fn replay_coeffs(&mut self) -> crate::av1::decode::Result<usize> {
let eob = self.record.next_coeff()?;
if eob <= 0 {
return Ok(0);
}
self.plane_tx_type = usize::try_from(self.record.next_coeff()?).map_err(|_| crate::av1::decode::Error::Invalid("coefficient record"))?;
let count = self.record.next_coeff()?;
for _ in 0..count {
let pos = self.record.next_coeff()? as usize;
let v = self.record.next_coeff()?;
*self.quant.get_mut(pos).ok_or(crate::av1::decode::Error::Invalid("coefficient record"))? = v;
}
Ok(eob as usize)
}
/// Reconstruct process (7.12.3).
pub(super) fn reconstruct(&mut self, plane: usize, x: usize, y: usize, tx_sz: usize, dc_only: bool) {
// dqDenom (7.12.3) is 1, 2 or 4 and the dividend is non-negative.
+19 -4
View File
@@ -213,7 +213,8 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
if start_x >= max_x || start_y >= max_y {
return Ok(());
}
if !self.b.is_inter {
let parse_only = self.pass == super::Pass::Parse;
if !self.b.is_inter && !parse_only {
let palette = if plane == 0 { self.b.palette_size_y } else { self.b.palette_size_uv };
if palette > 0 {
self.predict_palette(plane, start_x, start_y, x, y, tx_sz);
@@ -266,9 +267,23 @@ impl<'a, 'f> TileDecoder<'a, 'f> {
}
}
if !self.b.skip {
let eob = self.coeffs(plane, start_x, start_y, tx_sz);
if eob > 0 {
self.reconstruct(plane, start_x, start_y, tx_sz, eob == 1);
match self.pass {
super::Pass::Full => {
let eob = self.coeffs(plane, start_x, start_y, tx_sz);
if eob > 0 {
self.reconstruct(plane, start_x, start_y, tx_sz, eob == 1);
}
}
super::Pass::Parse => {
let eob = self.coeffs(plane, start_x, start_y, tx_sz);
self.record_coeffs(eob, tx_sz);
}
super::Pass::Reconstruct => {
let eob = self.replay_coeffs()?;
if eob > 0 {
self.reconstruct(plane, start_x, start_y, tx_sz, eob == 1);
}
}
}
}
for i in 0..step_y {
+45 -25
View File
@@ -1,7 +1,6 @@
//! Hostile-input robustness: truncation, header-region sweeps and random mutation of small
//! ffmpeg-generated AV1 streams (frame by frame) must never panic or run away. Skips without ffmpeg.
use std::panic::{AssertUnwindSafe, catch_unwind};
use std::process::{Command, Stdio};
use std::sync::atomic::{AtomicUsize, Ordering};
@@ -20,8 +19,15 @@ impl Rng {
}
}
/// (name, encoder args, pix_fmt)
const SEEDS: &[(&str, &str, &str)] = &[("av1_fz_default", "-c:v libsvtav1 -g 4", "yuv420p"), ("av1_fz_10bit", "-c:v libsvtav1 -g 6", "yuv420p10le")];
// Seeds are at least 64x64: SVT-AV1 before 3.0 (the Ubuntu runners have 2.3) refuses a source under 64 lines.
/// (name, encoder args, pix_fmt, size, decoder thread counts)
const SEEDS: &[(&str, &str, &str, &str, &[usize])] = &[
("av1_fz_default", "-c:v libsvtav1 -g 4", "yuv420p", "64x64", &[1, 3]),
("av1_fz_10bit", "-c:v libsvtav1 -g 6", "yuv420p10le", "64x64", &[1, 3]),
// Several superblock rows, so corrupt references meet frames already predicting from their
// top rows (row wavefront with frame threads).
("av1_fz_rows", "-c:v libsvtav1 -g 6", "yuv420p", "64x136", &[1, 8]),
];
fn ivf_frames(data: &[u8]) -> Vec<Vec<u8>> {
let mut out = Vec::new();
@@ -37,14 +43,13 @@ fn ivf_frames(data: &[u8]) -> Vec<Vec<u8>> {
out
}
// Seeds are 64x64: SVT-AV1 before 3.0 (the Ubuntu runners have 2.3) refuses a source under 64 lines.
fn make(ff: &std::path::Path, name: &str, args: &str, pix: &str) -> Option<Vec<Vec<u8>>> {
fn make(ff: &std::path::Path, name: &str, args: &str, pix: &str, size: &str) -> Option<Vec<Vec<u8>>> {
let out = craft_media_testkit::fixtures_dir("av1").join(format!("{name}.ivf"));
let p = craft_media_testkit::fixtures::generate(&out, |tmp| {
let a: Vec<&str> = args.split_whitespace().collect();
matches!(
Command::new(ff)
.args(["-hide_banner", "-loglevel", "error", "-y", "-f", "lavfi", "-i", "testsrc2=size=64x64:rate=25", "-frames:v", "6", "-pix_fmt", pix])
.args(["-hide_banner", "-loglevel", "error", "-y", "-f", "lavfi", "-i", &format!("testsrc2=size={size}:rate=25"), "-frames:v", "6", "-pix_fmt", pix])
.args(a)
.args(["-f", "ivf"])
.arg(tmp)
@@ -69,20 +74,29 @@ fn count_panics() {
});
}
fn run_all(frames: &[Vec<u8>], label: &str) {
fn run_all(frames: &[Vec<u8>], threads: &[usize], label: &str) {
craft_media_testkit::set_progress(label);
count_panics();
let before = PANICS.load(Ordering::SeqCst);
let res = catch_unwind(AssertUnwindSafe(|| {
for threads in [1, 3] {
let mut dec = Decoder::with_threads(threads);
for (i, f) in frames.iter().enumerate() {
let _ = dec.decode_pts(f, i as i64);
}
let _ = dec.flush_result();
}
}));
assert!(res.is_ok() && PANICS.load(Ordering::SeqCst) == before, "panic: {label}");
// Every thread count decodes the stream at the same time, each on a thread of its own.
let ok = std::thread::scope(|s| {
let runs: Vec<_> = threads
.iter()
.map(|&threads| {
s.spawn(move || {
let mut dec = Decoder::with_threads(threads);
for (i, f) in frames.iter().enumerate() {
let _ = dec.decode_pts(f, i as i64);
}
let _ = dec.flush_result();
})
})
.collect();
// Join every run (no short cut), then check them all.
let joined: Vec<bool> = runs.into_iter().map(|r| r.join().is_ok()).collect();
joined.into_iter().all(|ok| ok)
});
assert!(ok && PANICS.load(Ordering::SeqCst) == before, "panic: {label}");
}
/// Random-mutation rounds per seed (raise with FILMCRAFT_FUZZ_ITERS for a long hunt).
@@ -95,17 +109,23 @@ const DEADLINE_SECS: u64 = 600;
#[test]
fn hostile_streams_never_panic() {
craft_media_testkit::with_deadline(DEADLINE_SECS, sweep);
craft_media_testkit::with_deadline(DEADLINE_SECS, || sweep(&SEEDS[..2]));
}
fn sweep() {
/// The seed with several superblock rows, in a test of its own so it runs beside the others.
#[test]
fn hostile_streams_with_superblock_rows_never_panic() {
craft_media_testkit::with_deadline(DEADLINE_SECS, || sweep(&SEEDS[2..]));
}
fn sweep(seeds: &[(&str, &str, &str, &str, &[usize])]) {
let Some(ff) = craft_media_testkit::ffmpeg_or_skip("AV1 fuzz") else { return };
for &(name, args, pix) in SEEDS {
for &(name, args, pix, size, threads) in seeds {
if !craft_media_testkit::ffmpeg_has_encoder(&ff, "libsvtav1") {
craft_media_testkit::skip_test("AV1 fuzz", &format!("this ffmpeg has no libsvtav1 encoder (seed {name})"));
continue;
}
let Some(seed) = make(&ff, name, args, pix) else {
let Some(seed) = make(&ff, name, args, pix, size) else {
panic!("ffmpeg has libsvtav1 but could not encode the {name} seed");
};
if seed.is_empty() {
@@ -118,7 +138,7 @@ fn sweep() {
for n in (0..len).step_by((len / 200).max(1)) {
let mut d = seed.clone();
d[fi].truncate(n);
run_all(&d, &format!("{name} frame {fi} truncate {n}"));
run_all(&d, threads, &format!("{name} frame {fi} truncate {n}"));
}
}
// header region of every frame: every byte to a few values, every bit
@@ -127,12 +147,12 @@ fn sweep() {
for v in [0u8, 0xff, 0x80, 0x01, 0x7f] {
let mut d = seed.clone();
d[fi][p] = v;
run_all(&d, &format!("{name} frame {fi} byte {p}={v:#x}"));
run_all(&d, threads, &format!("{name} frame {fi} byte {p}={v:#x}"));
}
for b in 0..8 {
let mut d = seed.clone();
d[fi][p] ^= 1 << b;
run_all(&d, &format!("{name} frame {fi} flip {p}.{b}"));
run_all(&d, threads, &format!("{name} frame {fi} flip {p}.{b}"));
}
}
}
@@ -151,7 +171,7 @@ fn sweep() {
d[fi][p] = rng.next() as u8;
}
}
run_all(&d, &format!("{name} random {i}"));
run_all(&d, threads, &format!("{name} random {i}"));
}
}
}
+1
View File
@@ -9,3 +9,4 @@ mod hostile_dims;
mod oracle_inter;
mod oracle_intra;
mod perf;
mod wavefront;
@@ -0,0 +1,93 @@
//! Stress tests of the frame threading (parse / reconstruct split and the row wavefront across
//! frames): every thread count gives the single-thread pictures, run after run, and dropping
//! or flushing a busy decoder never hangs. A hang fails the test through the deadline.
use crate::common;
use common::*;
use craft_video::av1::decode::{Decoder, Picture};
/// A GOP with many superblock rows (the wavefront has rows to overlap), hierarchical
/// references and motion reaching across rows.
fn spec() -> Spec {
Spec::new("wavefront_640x480", "testsrc2=s=640x480:r=25,rotate=a=t*0.5:c=black", 24, "yuv420p", &["-preset", "8", "-crf", "38"])
}
/// Seconds a test may take before it counts as a hang (override: FILMCRAFT_HOSTILE_TIMEOUT_SECS).
const DEADLINE_SECS: u64 = 300;
fn decode(tus: &[Vec<u8>], threads: usize) -> Vec<Picture> {
let mut dec = Decoder::with_threads(threads);
let mut pics = Vec::new();
for (i, tu) in tus.iter().enumerate() {
pics.extend(dec.decode_pts(tu, i as i64).unwrap_or_else(|e| panic!("threads {threads}, unit {i}: {e}")));
}
pics.extend(dec.flush_result().unwrap_or_else(|e| panic!("threads {threads}, flush: {e}")));
pics
}
#[test]
fn every_thread_count_gives_the_single_thread_pictures_repeatedly() {
let Some(ff) = ffmpeg() else { return };
let tus = ivf_frames(&make(&ff, &spec()));
craft_media_testkit::with_deadline(DEADLINE_SECS, move || {
let want = decode(&tus, 1);
assert_eq!(want.len(), 24);
for threads in [2, 3, 4, 5, 8, 16] {
for round in 0..4 {
assert!(decode(&tus, threads) == want, "threads {threads}, round {round}: pictures differ");
}
}
});
}
/// 128x128 superblocks in two tile columns (libaom), with motion field estimation: the
/// wavefront over the superblock rows of every tile column.
fn spec_sb128_tiles() -> Spec {
let enc = ["-cpu-used", "8", "-crf", "40", "-tile-columns", "1", "-lag-in-frames", "16", "-aom-params", "sb-size=128:enable-ref-frame-mvs=1"];
Spec::new("wavefront_sb128_tiles", "testsrc2=s=640x400:r=25,rotate=a=t*0.5:c=black", 16, "yuv420p", &enc).encoder("libaom-av1")
}
#[test]
fn superblocks_of_128_in_tile_columns_give_the_single_thread_pictures() {
let Some(ff) = ffmpeg_libaom() else { return };
let tus = ivf_frames(&make(&ff, &spec_sb128_tiles()));
craft_media_testkit::with_deadline(DEADLINE_SECS, move || {
let want = decode(&tus, 1);
assert_eq!(want.len(), 16);
for threads in [2, 3, 4, 8, 16] {
for round in 0..2 {
assert!(decode(&tus, threads) == want, "threads {threads}, round {round}: pictures differ");
}
}
});
}
#[test]
fn dropping_or_flushing_a_busy_decoder_never_hangs() {
let Some(ff) = ffmpeg() else { return };
let tus = ivf_frames(&make(&ff, &spec()));
craft_media_testkit::with_deadline(DEADLINE_SECS, move || {
let want = decode(&tus, 1);
for threads in [2, 3, 8, 16] {
for cut in [1, 2, 7, 12, 23] {
// Dropped with frames in flight (no flush).
let mut dec = Decoder::with_threads(threads);
for (i, tu) in tus.iter().take(cut).enumerate() {
dec.decode_pts(tu, i as i64).expect("decode");
}
drop(dec);
// Flushed mid-stream (as before a seek), then continued: the same pictures.
let mut dec = Decoder::with_threads(threads);
let mut pics = Vec::new();
for (i, tu) in tus.iter().enumerate() {
if i == cut {
pics.extend(dec.flush_result().expect("flush"));
}
pics.extend(dec.decode_pts(tu, i as i64).expect("decode"));
}
pics.extend(dec.flush_result().expect("flush"));
assert!(pics == want, "threads {threads}, flush before unit {cut}: pictures differ");
}
}
});
}