Commit Graph
367 Commits
Author SHA1 Message Date
vangel76 3494ef8552 Caching: an oversized Cryptomatte plane stays beside held ones, disk copies are made a job at a time, and the review tests prove what they claim (#3750 review) 2026-11-17 17:40:16 +01:00
vangel76 5d0bfe7869 Caching: memory is let go only where it frees memory, Purge keeps nothing decoding meanwhile, and regions read the rows they show (#3750 review)
- Cryptomatte planes: a plane a render holds is not offered for its gone channels, and the entry
  limit holds while renders hold every plane.
- Footage frames and whole stills held elsewhere are not offered to the memory accountant.
- Purge (clear frames) makes decodes in flight stale: their frames are not kept or joined.
- Plane store: a size refusal of a merged decode is not handed to a request that fits alone;
  slimming a shared set and copies for the disk cache take memory fallibly, and planes the disk
  cache can't take are not copied.
- Disk cache: the plane encoder's scratch is taken fallibly; no memory skips the entry.
- Region of interest: rows follow the projected transform of a 2D layer under a 3D parent, and
  a frame of another size than its footage is read whole.
- The box filter's work bound is checked without a wall clock.
2026-11-17 17:00:29 +01:00
vangel76 fc4eddf2d5 Merge upstream/main into the plane store branch (#3301)
# Conflicts:
#	Cargo.toml
2026-11-17 11:10:07 +01:00
vangel76 c205773d92 Caching: memory is counted where it is held, and whole layers stay whole under a viewer region (#3301 review)
- The memory accountant is offered only layer buffers and preview frames nobody
  else holds, so what it counts as freed is freed; the frame cache's recycled
  buffers count and go first.
- A Cryptomatte plane over the matte cache's own limit is kept only where the
  common budget trims it.
- Disk cache entries read back take their buffers fallibly: memory the system
  can't give is a miss.
- The Layer panel, sampleImage and effect inputs read whole footage frames under
  a viewer region of interest.
- A 16K Cryptomatte source at half resolution is box-filtered, not looked up
  nearest-neighbour.
- The System Compatibility Report shows the cache budget in force.
- A Project panel thumbnail render never starts under a key held back from an
  earlier state.
- Convert Expression to Keyframes makes no keys on a property that can't be
  animated.
2026-11-17 11:09:13 +01:00
Dylan M. Taylor 6d908d1301 Merge pull request #3215 from vangel76/cc-ball-action-params-3197
Ball Grid effect has all of After Effects' CC Ball Action controls, including flat balls without shading
2026-11-16 20:39:25 -05:00
vangel76 9ff031c5f1 Merge upstream/main into the plane store branch (#3301)
Conflicts: Cargo manifests (craft-libs pins from main, plus memory-stats, exr and
half for effectcraft-media), exr_channels.rs (the plane store's version; main's
craft-exr channel decode is replaced by the plane store until craft-exr reads
from a seekable source), pool.rs (frame_px takes the rows of #3271 and the
forget generation of #2042), session open/forget (main's forget_files plus the
reload counter), Project panel thumbnails (main's Thumb types, keyed by content).
2026-11-16 20:57:45 +01:00
Dylan M. Taylor 600db3717f Merge remote-tracking branch 'origin/main' into fix/effect-gpu-integrated 2026-11-16 00:32:26 -05:00
Dylan M. Taylor f0a5783a82 fix(gpu): preserve CSP shaper channels on software Vulkan 2026-11-15 23:56:13 -05:00
Dylan M. Taylor 3678cc9bec fix(gpu): bound inverse LUT search and write transfer components explicitly 2026-11-15 23:25:31 -05:00
Dylan M. Taylor 5a7b4e903f test(gpu): isolate inverse LUT and profile readbacks 2026-11-15 23:03:51 -05:00
vangel76 f4a8c88349 Merge upstream/main into cache-defaults-3298 2026-11-15 20:28:43 +01:00
.CooCoo d6c9ad6643 Cineon Converter: Gamma 1 gives linear output and saved projects keep their render (#2312)
## Behavior

A new Cineon Converter opens with Gamma 1.7. Its Version parameter is 2 by default. A converter saved before Version existed opens as version 1, retaining its Gamma value, keyframes, and expression text and rendering on its original scale. Reset keeps the instance's Version.

In version 2:

- Log to Linear raises the exposure to `1 / gamma` before subtracting the black offset and normalizing: `(E^(1/g) - O^(1/g)) / (1 - O^(1/g))`, with `E = 10^((code - white) * 0.002 / 0.6)` and `O` the exposure at the black point. Gamma 1 gives linear output, and a larger Gamma lightens the midtones. Highlight Rolloff then applies as before, and Linear to Log is the inverse.
- Log to Log returns code values unchanged at Gamma 1 and raises the normalized code value to `1 / gamma` at other values. With custom black or white points, the code values first go through the Gamma 1 conversion.

The CPU and GPU paths match. Version 1 instances render as before, and Interpret Footage ▸ Cineon Settings keeps its existing formula, which agrees with the version 2 Converter at Gamma 1.

The public Cineon Converter documentation (https://helpx.adobe.com/after-effects/using/utility-effects.html) describes Gamma as a midtone control and gives no defaults or formula. The default, the Log to Linear curve and the Log to Log behavior come from running and scripting After Effects 26.5 on my own machine and reading back the rendered pixels. In After Effects, choosing Log to Log also sets Gamma to 1 and Highlight Rolloff to 0; EffectCraft does not do that yet, because no command sets one parameter from another.

Animation preset format 3 records `cineonVersion` for a saved Gamma property and stores effect key times relative to the first key. Presets in formats 1 and 2 without that tag contain version 1 Gamma values; format 2 already has relative effect key times. Applying a whole old Cineon Converter also gives it version 1. Exposed Essential Properties Gamma values convert to the target converter version where their target is unambiguous (the conversion keeps the linear point; other Gamma values follow each version's own curve); enabled expressions on another scale are skipped instead of rewriting their source.

A saved Gamma records the Version its converter has at the current time. Applied to a converter whose Version is keyframed, a Gamma preset keeps those keys: the Gamma is converted to the one version they hold, or skipped when they hold several or an expression sets the Version. A preset that also holds the Version sets it from that property.

The project schema remains 2. `docs/architecture.md` describes the behavior above.

## Validation

Effect tests check version 2 Log to Linear at Gamma 1.7 (codes 95 to 685) and Gamma 2.2 (codes 180 to 620), and Log to Log at Gamma 1 (Rolloff 0 and 20) and Gamma 1.7, against the measured 8-bit values within one or two codes, plus round trips, version 1 rendering and the Interpret Footage agreement at Gamma 1. GPU tests compare the version 2 paths with the CPU. Engine tests cover old projects, Reset, presets, keyframes, and Essential Properties.

Commands run on `74d398c2`:

- `cargo fmt --all -- --check`
- `cargo clippy -p effectcraft-effects -p effectcraft-gpu -p effectcraft-project -p effectcraft-engine --all-targets -- -D warnings`
- `cargo test -p effectcraft-effects`: 546 passed (including `effects_doc_is_current`, so `docs/effects.md` needs no change), 1 ignored, plus 6
- `cargo test -p effectcraft-project`: 47 passed
- `cargo test -p effectcraft-gpu`: 253 passed, 1 ignored
- `cargo test -p effectcraft-engine`: 1,007 passed, 5 ignored, plus 27
2026-11-15 12:11:07 +01:00
Dylan M. Taylor 26fa70007a fix(gpu): serialize readback mapping cleanup 2026-11-15 09:21:33 +00:00
vangel76 363e6db494 Merge main 2026-11-14 22:28:58 +01:00
vangel76 3ad49f8f7b Merge upstream/main into cache-defaults-3298
Conflict in crates/render/src/pipeline.rs: kept both the footage-rows
reader of this branch and the frame-mix span reader from main.
Adapted three menus tests to DialogAnswer.paths now being a Result.
2026-11-14 19:53:16 +01:00
FredApps 05801ee111 Merge pull request #3210 from vangel76/grid-mask-3121
Effects: Grid, Checkerboard and Circle keep the layer's mask with Blending Mode None
2026-11-14 17:00:32 +01:00
Dylan M. Taylor e8b188fdb0 fix(gpu): sort selective color channels on llvmpipe 2026-11-13 17:45:15 +00:00
vangel76 4829a83d30 Merge upstream/main into cache-defaults-3298 2026-11-13 12:39:55 +01:00
vangel76 7f13d1baed Merge upstream/main into Ball Grid branch 2026-11-12 12:47:55 +01:00
arnav-hiwarkar f88d0d0a8d #3487, #3488, #3489, #3490, #3491: Levels (Individual Controls): skip the overflowing pow with a branch
WGSL's select evaluates both of its arguments, so the previous commit still
ran the overflowing pow and only discarded it. An explicit branch now skips
the power for a curve past the bound.
2026-11-12 16:37:09 +05:30
arnav-hiwarkar 047b912fc5 #3487, #3488, #3489, #3490, #3491: Levels (Individual Controls): GPU saturates the curve before pow can overflow
WGSL leaves the result of an overflowing pow indeterminate, so clamping
after it did not guarantee the CPU's 1e30 bound on every backend. A curve
past the bound (log2|t| / gamma > 99.7) now saturates to +-1e30 without
calling pow; between 1e30 and 2^99.7 the clamp gives the same 1e30, so the
GPU matches the CPU everywhere.
2026-11-12 16:35:08 +05:30
arnav-hiwarkar c0387ea18a #3470, #3472, #3473, #3474, #3475, #3476, #3477, #3480, #3483, #3484, #3487-#3491: test the ends of the new ranges on the GPU
CPU / GPU comparisons at the new ends for Venetian Blinds (width 1, width
and feather 32000), Radial Wipe (feather 32000), Grid (border 2000),
Scatter (1000), Beam (4000), Change Color (1710 degrees) and Turbulent
Displace (seed 100000). The comparison counts infinity on both sides as
equal, so Levels (Individual Controls) gets a direct check that the GPU's
output is finite at the ends of its ranges. The Noise HLS test asserts the
spec's ranges, so it fails without them.
2026-11-12 16:24:17 +05:30
arnav-hiwarkar acf60eeaf3 #3487, #3488, #3489, #3490, #3491: Levels (Individual Controls): bound the curve only where it would overflow
The 1e6 bound also capped curves that were finite before (gamma 0.1 on an
HDR value 4 x Input White is 4^10, about 1.05e6). The bound is now 1e30:
1e30 times the widest output range (2e4) is still finite, and every curve
below it renders as before. A test pins 4^10.
2026-11-12 16:24:17 +05:30
arnav-hiwarkar 244f9bd846 #3483, #3484: Noise HLS: amounts and Grain Size take After Effects' ranges
Hue, Lightness and Saturation go up to 1000 % and Grain Size down to 0 (it
keeps its wider maximum, 100), After Effects' recorded valid ranges in the
same units. Noise HLS Auto shares the controls and widens with them.
Defaults and sliders are unchanged.

The renderers need no change: hue wraps, lightness and saturation clamp per
pixel after the noise is added, and Grain Size is floored at 0.1 px on the
CPU and the GPU, so the new values cost the same and stay finite.

The amounts were written as "percent" in keyframe text because their range
was 0..100, and Grain Size would become "percent" at 0..100; UNITS rows keep
the units they had.
2026-11-12 15:46:48 +05:30
arnav-hiwarkar 363a800d3f #3478, #3479: 4-Color Gradient: Blend and Jitter take After Effects' ranges
Blend goes up to 10000 (it keeps its minimum of 1, below After Effects' 5)
and Jitter up to 500 %, After Effects' recorded valid ranges in the same
units. Defaults and sliders are unchanged.

The CPU and GPU renderers capped Jitter's dither at 100 %, so higher values
would be stored but render as 100. Both now take it from one function,
four_color_jitter, which keeps the dither linear up to 500 %. Blend 10000
is a finite softening (Blend squared) at the same cost.

Jitter was written as "percent" in keyframe text because its range was
0..100; a UNITS row keeps that unit.
2026-11-12 15:14:34 +05:30
arnav-hiwarkar 39a3b565d8 #3487, #3488, #3489, #3490, #3491: Levels (Individual Controls): levels and gamma take After Effects' ranges
Input/Output Black and White of every channel take -10000..10000 and Gamma
goes down to 0 (it keeps its wider maximum, 10), After Effects' recorded
valid ranges in the same units (the .aep mapping is Conv::Plain). Defaults
and the 0..1 / 0.1..3 sliders are unchanged.

Gamma 0 renders as the existing 0.01 floor, an exponent of 100, so an HDR
value a few times Input White would overflow f32 and an infinite curve
times an empty output range is NaN. The curve is now bounded at +-1e6 on
the CPU (Lv::apply) and the GPU (fxt_lv), so the output stays finite over
the whole range; values below the bound are unchanged.

The engine regression (tests/effect_ranges.rs) sets both ends of each row
through prop.set, checks undo/redo, save and reopen, and that a project
saved with the narrower ranges opens with the new ones and its values.
2026-11-12 15:11:21 +05:30
vangel76 9924ebb1e0 Merge remote-tracking branch 'upstream/main' into cache-defaults-3298 2026-11-12 00:05:19 +01:00
Dylan M. Taylor e339a5b375 Effects: cap Camera Lens Blur and median radii (#2951)
* Effects: cap Camera Lens Blur and median radii

Camera Lens Blur built its iris table (one row per pixel of radius) from an
uncapped Blur Radius before any size check, and the median window is filled
(2r + 1)^2 per row from an uncapped Radius, on the CPU and the GPU. The
radius is now capped where the work is built: 2000 buffer pixels for the
lens iris (UI max 500) and 512 for the median window (UI max 255), on both
paths.

* address review: cap camera lens reach with the spans
2026-11-11 15:53:34 -05:00
vangel76 7db82732c7 Merge upstream/main into cc-ball-action-params-3197 2026-11-10 22:39:48 +01:00
vangel76 0bc46981f5 Merge upstream/main into cache-defaults-3298 (one RAM budget kept: the pool holds float and 8-bit frames, rows reads and flights; the disk cache writer encodes off the render thread) 2026-11-10 22:35:36 +01:00
Matej Kozlovský 513db8f421 gpu: assert box-blur row parity with diff(), exact only on llvmpipe
Reuse the existing diff() helper so a non-finite pixel fails the test
(f32::max ignored NaN). Keep exact equality on llvmpipe, which still
fails without the row-block fix; other adapters use the usual Bpc32
tolerance, because a 2e-6 bound can fail on unmeasured hardware and
does not catch the regression there.
2026-11-10 20:50:18 +01:00
Matej Kozlovský 5d1b6ce3f2 gpu: match CPU rounding in box blur rows
Horizontal GPU box passes restarted the running sum every 256 pixels, while
the CPU keeps one sum per row; use the full row width as the block so f32
rounding matches. The regression test requires exact equality on llvmpipe
(CI); other adapters only get an absolute 2e-6 sanity bound.
2026-11-10 20:50:18 +01:00
FredApps 8b9f9e6943 Capture observed process and GPU failure evidence (#3409)
* #3357: retain process, memory and typed GPU failure evidence

* #3357: tighten reporter protocol regression cases
2026-11-10 14:05:57 +01:00
vangel76 c8bea7cf94 Merge remote-tracking branch 'upstream/main' into cache-defaults-3298 2026-11-09 19:29:28 +01:00
vangel76 b0730f0cb4 Merge upstream/main into cc-ball-action-params-3197 2026-11-09 19:19:43 +01:00
vangel76 d761e064b4 Merge remote-tracking branch 'upstream/main' into cache-defaults-3298
# Conflicts:
#	crates/gpu/src/tests.rs
#	crates/render/src/lib.rs
#	crates/render/src/pipeline.rs
#	docs/preferences.md
2026-11-09 19:12:36 +01:00
FredApps 1197414255 Merge pull request #3363 from FredApps/fix/3357-gpu-fallback-exit
#3357: release retired GPU resources before CPU fallback
2026-11-09 19:10:21 +01:00
FredApps e8634b1f4b #3357: release retired GPU resources before CPU fallback 2026-11-09 18:41:39 +01:00
vangel76 74ac5a2f26 #3335: choose the GPU frames in flight from measured GPU memory, with a Memory & CPU setting 2026-11-09 02:04:15 +01:00
vangel76 dab57e6982 #3334: the matte cache's memory candidates after the merge (Candidate::new) 2026-11-09 01:34:25 +01:00
vangel76 5b23f6fe60 #3334: Cryptomatte uploads only the matte, computed at source resolution
The matte (or the Colors display's pixels) is made from the unscaled ID and coverage channels at source resolution, box-filtered below render scale 1, kept per footage frame, layer, selection, output and scale in a cache in the common memory budget, and uploaded to the GPU as one R32F plane. OpenEXR file identity checks are remembered for 1 s; Reload Footage forgets them.
2026-11-09 01:23:16 +01:00
vangel76 885ee302d7 Merge remote-tracking branch 'origin/gpu-upload-once-3333' into rq-readback-3335 2026-11-09 00:55:36 +01:00
vangel76 4649c27ef9 #3335: Render Queue reads frames back once, pipelines GPU frames and picks the faster renderer 2026-11-09 00:55:30 +01:00
vangel76 2320255c8b #3333: upload each footage frame once per frame and share it across layers 2026-11-09 00:47:50 +01:00
vangel76 cbaa68321c Merge remote-tracking branch 'upstream/main' into footage-cache-3271
# Conflicts:
#	crates/effects/src/lib.rs
2026-11-08 16:16:21 +01:00
FredApps 7ca249d085 Merge pull request #3120 from vangel76/aep-import-thirdparty
.aep import: map CC Color Neutralizer, EXtractoR and IDentifier
2026-11-08 15:45:08 +01:00
vangel76 67a39d9a2b .aep import: EXtractoR (copy) reads the effect input, not the EXR plane of that name
After Effects' Channel Info kind 2 "(copy)" was imported as R, G, B or A.
EffectCraft's EXtractoR resolves those names through the footage's aux planes
first, and an OpenEXR keeps its own R/G/B/A planes, so "(copy)" read the
original file plane and dropped what an earlier effect did to the channel.

"(copy)" is now a reserved EXtractoR channel name (EXTRACTOR_COPY) that the
CPU and GPU renderers never look up in the aux planes: it is the input's own
channel of the same position. The importer writes it for kind 2.

Render regression: a modified input buffer with same-named R/G/B/A aux planes
of other values; GPU parity case with (copy) on an aux scene with R/G/B.
2026-11-08 02:32:11 +01:00
Valentin KleinandClaude Opus 5.5 84382487bb Disk cache: entries keep the files they read, keys from memory (#3184) (#3191)
* Disk cache: record the source files each frame reads

Part of #3184 (step 1): renders now report which footage files they read,
each with its identity (size and modification time) as read. Nothing uses
the list yet: images, cache keys and cache behaviour are unchanged.

- render: `reads` module with `FileStamp`, `ReadFile`, `ReadFiles` (sorted,
  deduplicated, bounded; marked incomplete when it can't vouch for every
  file) and the thread-safe `ReadLog`. `Renderer::reads` collects a frame's
  files; `FootageSource::recording` lets a source note what it reads.
  `file_stamp` uses the same identity.
- Layer cache entries keep the files their buffer was made of and report
  them on every hit; derived entries (styles, blend space, flattened) keep
  their base's. Buffers from the disk cache or the browser store don't know
  their files yet and mark the list incomplete.
- Media pool: every file read notes its identity, taken from the file it
  opens. Decoded frames, opened movies, GIFs, models and EXR channels keep
  it with the cache entry, so a frame served from memory reports its file
  without touching the disk.
- Viewer frame worker collects each frame's list and carries it to where
  the disk entry is written (step 2 stores it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Automation: confined footage passes on the files it reads

Part of #3184: ConfinedFootage forwards FootageSource::recording to the
source it wraps, so renders of an automation session list their files
instead of coming back incomplete. The read tests also render through the
viewer's CancellableFootage wrapper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: frame entries keep their source files, comp key from memory

Part of #3184 (step 2). Fixes #3175: opening a comp no longer stamps every
footage file of the project before its first frame can show.

- The comp's content key (`disk_cache::comp_content_key`) is taken from the
  project in memory alone: no footage or proxy file is stamped, no
  `VirtualFileId` is asked. `comp_content_key_with` is gone; `frame_key`
  hashes a new entry version, so entries of the earlier format are never
  looked up again (LRU evicts them).
- Frame entries (formats 2 and 3) store the files the frame read
  (`EntryFiles`: size and modification time as read; size and content hash
  through `VirtualFileId` for files from memory; missing files), then the
  pixels. Formats 0 and 1 read as misses and are deleted.
- Check on hit: `get_frame`/`get_frame_f16` look only at the listed files
  (one metadata call each) on the frame worker. A changed, back or unreadable
  file, or a damaged entry, is a miss: the entry is deleted and the frame
  rendered and stored again.
- A frame whose file list is incomplete is shown but not stored. The RAM
  preview keeps each frame's files, so `persist_missing` writes only frames
  with a complete list.
- Remote frame worker: `FrameServer` records each frame's reads
  (`take_files`); the web worker stores them with the entry and the page
  checks them through the file table before using it. The browser's media
  epoch existed only for the old key and is removed. Export does not use
  frame entries.
- The cache indicator still only asks whether an entry exists.

Tests: entry format round trip and damage, the check per file, and in the
viewer: a changed file re-renders exactly its frames (and a reopen reads the
rest from disk); a frame checks only its own files in a project with a large
unused sequence and the key looks at no file; missing, unreadable files and
damaged entries never serve a stale frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: layer entries keep their source files, no footage salt

Part of #3184 (step 3 of 4).

Layer buffers on disk (and in the browser's layer store) now carry the
files they were made of, like frame entries since step 2, and are checked
against exactly those files when used. The project-wide footage salt is
gone: nothing stamps every footage file any more.

- Layer entry format 4: the buffer's EntryFiles, then the pixels. A hit
  checks only the listed files (metadata, or the VirtualFileId hook for
  files from memory; Missing must still be missing); a change, a damaged
  entry or the old format 0 is a miss and the entry is deleted, so the
  re-rendered buffer replaces it. A hit reports its file list, so frames
  made of buffers from disk are complete and get stored again.
- Buffers whose files can't all be named are not written.
- Layer disk keys: layer_disk_key = hash of the layer entry version, the
  renderer version and the layer cache key (which names the footage items
  with their paths and settings, from memory).
- PrefetchStore checks on use; an entry made of changed files is dropped
  and forgotten, so the new buffer is queued over it. FrameServer passes
  its virtual-file hook to its layer cache.
- Removed: footage_salt(_with), file_stamp, footage_stamp,
  set_disk_salt, set_store_salt, Session::refresh_disk_salt,
  poll_disk_salt, footage_fingerprint, disk_salt, disk_salt_background,
  disk_salt_job, DiskSaltJob, the UI's per-frame poll, the salt in
  FrameServer and in the availability worker (with its cache_enabled
  plumbing). The disk cache is attached as soon as it is configured.
- Docs: architecture.md, web.md.

Tests: layer entries check their files (size, time, missing/back,
folder, damaged, old format, incomplete); virtual files; prefetch store;
media tests with a real pool across restarts (only the changed file is
read, projects with the same item ids never share an entry, missing,
unreadable and damaged never serve stale buffers); engine tests that the
cache is attached at once and edits do no file work (#1926, #3106).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: no-salt worker test, sequence paths in the frame-blend key

Part of #3184 (step 3 of 4).

- A frame-blended footage frame's layer-cache key now hashes the
  sequence's file list: with no project-wide salt, two projects whose
  items share an id and a first frame must not share its disk entry.
- Test: a frame worker syncing, patching and rendering a project with an
  unused 5000-frame sequence asks its file hook only about the file its
  buffer reads (the salt asked about every file at every sync).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: the comp key covers what its expressions can read

With an expression anywhere in the project, every comp's disk key hashed
every item of the project, so any edit (another comp, an unused footage
item) threw away every comp's frames on disk. The key now covers what the
comp can draw plus what the expressions among it can reach
(Project::pixel_reach):

- without expressions: the items the comp draws, without silent layers,
  plus the footage its precomp layers swap in (Media Replacement);
- with expressions: every layer of the comps they reach, their markers,
  the comps and footage they name with string literals (comp("Name"),
  footage("Name")) together with which items have those names now, and
  the footage named like a reached source (its data is read by name);
- the whole project when an expression computes a name, uses the object
  model's constructors or internals, dynamic code, computed keys or
  writes, or text the scanner can't read for sure (regex literals,
  template strings, escapes): expression_reach errs towards reaching more.

Markers count only where an expression can read them (#1083). Memory
only, no file is looked at.

Tests: scanner and reach unit tests in the project crate; engine tests
for the key (own-comp expressions keep the key through edits elsewhere,
comp("Other") follows Other and not a third comp, a second comp taking
the name counts, a computed name counts every item, markers read from
another comp); a ui-egui test that the disk frames outside an expression
layer are carried when the comp it names changes.

Part of #3184. Part of #1492.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: Reload Footage drops the entries of its files, docs

File > Reload Footage now deletes the frame and layer entries whose file
lists name the reloaded item's files (DiskCache::drop_files), so a file
replaced by one with the same size and modification time, which the
check on hit can't tell apart, is read again. The index knows keys, not
files: the writer thread reads the files section at the start of each
frame and layer entry (not the pixels) after the writes queued before
it, and deletes the entries that list one of the files. Until it is
done, an entry made of one of them is a miss and frames aren't linked to
new keys. The entries are gone for good, so they don't come back after a
restart. Automatic reloads don't drop anything: they follow a changed
stamp, which the check on hit catches. The browser needs nothing (its
file identities are content hashes).

Removed: disk_cache::seal, which steps 2 and 3 left without callers (the
tests use seal_format). Updated the module, seal_format and forget_media
docs, and docs/architecture.md / docs/web.md describe the final design:
what is cached, keys from memory, file lists checked on hit, nothing
scans the project, incomplete lists stay in memory, Reload Footage drops
entries, edits keep the frames they can't change.

Tests: render unit tests (same size and time passes the check but not
the drop; other files' entries stay; an entry queued before the drop is
dropped; a layer entry too; written again afterwards it is used; the
drop survives a restart; a pending drop is a miss), and an engine test
through file.reloadFootage.

Closes #3184. Part of #1492.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: file stamps keep the whole modification time

A file's modification time was nanoseconds since 1970 as u64: a time before 1970
read as "no time" (two such files of one size looked alike) and times after 2554
were truncated. It is now i128 nanoseconds, negative before the epoch; None only
means the file system has no time. The content-key hash writes a presence tag and
the full value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Automation: a refused footage read leaves the render's file list incomplete

Footage outside the read roots shows nothing and was noted as no file at all, so a
picture made without it had a complete, empty list and could be kept on disk for
a later session with other roots. A refused read during `recording` now marks the
list incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render: a buffer kept by another worker comes with its own files

LayerCache::insert returns the files of the buffer it keeps, and Renderer::keep
notes them in the render's log and returns them, so a render that uses another
worker's buffer lists that buffer's files (a file changed between the two reads
then has two identities and the list is incomplete).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render: sampleImage and text fonts note the files they read

EvalCtx carries the render's read log. A sampleImage render records its footage
reads, keeps them with its per-thread cached buffer and notes them in the
evaluating render's log on every use. Text notes the font files it is drawn with,
with the identity their bytes were read with; a family that isn't installed, a
fallback or missing glyph, or a font from memory leaves the list incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: frame entries store the whole modification time

A frame entry's file list stores a file's modification time as i128 nanoseconds
since 1970 (negative before), as the file stamps now keep it; the frame entry
format version goes to frame-entry-files-2 so entries in the old layout are not
read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: layer entries store the whole modification time

Layer entries share the frame entries' file list layout, which now stores the
full i128 modification time: the layer entry format version goes to
layer-entry-files-2. The module doc names the font files text is drawn with
among an entry's source files, and what keeps a list incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Layer cache: buffers that read other layers are keyed by the switches passed down

A layer whose effect reads another layer (Gradient Wipe, Set Matte, Blend…) gets
that layer drawn by its own renderer. Through a precomp layer, Switches Affect
Nested Comps, the switches inherited from the precomp layers above and motion
blur on change those pixels, but the key held only the layers' own switches: a
buffer made with the setting on was reused (from memory or the disk cache) with
it off. Such layers now derive their key with what the renderer passes down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Layer cache: masked buffers are keyed by the mask motion blur gate

A mask with Mask Motion Blur On blurs even when the layer's own switch is off,
as long as the render's gate is open (Render Settings motion blur, and with
Switches Affect Nested Comps the precomp layers above). The key held the
layer's own motion blur state only, so a buffer blurred with the gate open was
reused with it closed (and the other way round), also from the disk cache.
Masked layer, input and footprint keys now tell a closed gate apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render: cached buffers list what their expressions and substituted characters read

- Buffers made under `Renderer::recorded` (layer content, a Time effect's input, an adjustment
  layer's footprint) evaluate their masks, shapes and effects with the context that notes into
  the buffer's own read log, so a file an expression reads (`sampleImage`, also at shutter times
  mask motion blur adds and the cache key doesn't see) is in the buffer's files.
- A styled layer served from the cache lists the files of the cached body and passes too, so
  the buffers made of it (flattened, blending space) don't get a complete list when one of them
  came with an incomplete one.
- A character put in by Character Offset or Character Value without a relayout notes the font
  it is drawn with; a fallback face (or none) leaves the list incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render: text lists the font file of a run that draws no glyph; tests clean their temp dirs

An empty first paragraph draws no glyph, but in a box its face sets the
first baseline that places every later line. The read list took faces only
from glyphs, so it could be complete without that file. Note each style
run's resolved face as well.

The font-reads and confine tests use tempfile::tempdir(): a fresh
directory per run, removed on panic too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: Reload Footage closes the gaps around its drop (#3184)

- A reload while the disk cache is off is kept on the session
  (pending_disk_drops) and applied when the cache is switched on, before
  anything is read from it.
- Each render takes the cache's drop generation before it reads (frame
  worker; layer cache at the miss). An entry listing a file dropped after
  that is refused under the queue's lock, so a render that was under way
  during the reload can't store pixels from before it. Such a frame is
  shown with an incomplete file list, so persist_missing skips it too.
  The per-file record is bounded (past it, everything older is refused).
- An entry read while its drop was pending is a miss even when the
  writer ends the sweep between the read and the check (a stale hit
  found under load): the pending drops and the generation are taken
  before the entry is read.
- Test: a reopened project serves its frame from disk with no source file
  opened or decoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Disk cache: RAM frames keep the drop generation their render started at

Reload Footage empties the RAM preview, but an idle pass of
persist_missing may already hold a frame from before the reload. It
took the cache's drop generation when it ran, after the drop, so the old
pixels could be written back and, with the file's size and time
unchanged, served again after a restart.

Each RAM frame now keeps the disk cache its render started with and
that cache's drop generation then. persist_missing writes a frame only
to that cache, with that generation, and skips it once one of its files
was dropped since; a frame rendered without that cache is not written
from there.

Tests use tempfile directories (the layer drop test, the frame_files
folders under target/).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Doc comments for the functions this PR touches

* Disk cache: cap the free-space reserve at 10 GB, skip encodes the limit refuses

The reserve the cache leaves free on its disk was max(2 GB, 5% of the
disk) (#2551). On a 4 TB disk with 190 GB free that is 200 GB, so a
cache set to 20 GB had a limit of 0: every write was refused, the cache
never held anything, and only a log warning said so.

The reserve is now clamp(5% of the disk, 2 GB, 10 GB). It is there to
keep the cache from filling the disk, not to keep a share of a large
disk free: 10 GB leaves the system room to work on any disk, a small
disk still keeps 2 GB, and a 100 GB disk keeps 5 GB as before. After
Effects has only a maximum cache size and no reserve; a clamp keeps
the floor for small disks, where a fixed 10 GB would take most of the
space.

Writes the limit refuses cost no work: DiskCache::accepts_writes (one
atomic read) is checked before a frame is converted (put_disk_frame)
and before frames and layers are compressed (put_frame, put_frame_f16,
put_layer). The frame workers spent about 8 ms per frame and 14 ms per
layer on encodes that were then refused. persist_missing returns at
once at limit 0 instead of converting the RAM frames on every idle pass.

Settings > Disk shows the limit in effect in MB under Maximum Disk
Cache Size, with a note when free disk space lowers it
(settings.disk.cacheLimit).

* Disk cache: refuse writes below 64 MB of room, show this disk's reserve

Review follow-ups to the reserve fix:

- accepts_writes also refuses while free disk space leaves less than
  64 MB of room (a limit the setting asks for is always used): a few
  frames would fill it, and the rest would be compressed only to be
  refused.
- While writes are refused the idle passes ask for no repaint, so the
  UI keeps a frame coming every 5 s: the free space is read again and
  the RAM frames are written once there is room.
- Settings > Disk names the space kept free on this disk (the reserve
  the last reading applied; the rule from MIN/MAX_DISK_RESERVE when it
  is unknown), and the "no room" note follows accepts_writes.
- The settings.disk.cacheLimit automation label is machine-readable
  and the same in every language: limitMb, maxMb, reserveMb,
  spaceLimited, writes.
- cargo fmt.

* Disk cache: follow 3D model sources in the comp's reach

The reach walk followed comp, footage and solid sources but not
LayerSource::Model. An expression reading a model layer's source
(thisComp.layer("Model").source.name) left the model item out of the
comp's content key, so renaming the model kept the key and the old
frames on disk were served. Every source item is followed now
(LayerSource::item), which also counts the model when no expression
reads it (Advanced 3D draws it).

The same omission is fixed in the RAM preview's content walk
(frames.rs) and in the layer key's walk of referenced layers
(cache/dependencies.rs).

* Disk cache: a render's drop generation travels with its reads; no disk entries for Expression Selectors

Drop generation per render: the layer cache recovered a buffer's drop
generation from the latest miss of its key, so a second render of the
key that started after File > Reload Footage lent its generation to a
render still running from before, and that render's buffer (the file as
it was) was written after the sweep. Now Renderer::recorded starts each
buffer's read log at the store's drop generation before anything is
read (ReadLog::starting_at); ReadFiles carries the earliest generation
through merges, so derived buffers (styles, the blend-space copy, the
flattened layer) and frames that use a buffer from memory carry it too.
LayerCache::insert keeps a buffer made of a file dropped since its
reads began neither on disk nor in memory; a list that doesn't say
counts as made before every drop. DiskCache::dropped_since takes the
list's own generation too. missed_at keeps only the miss time.

Expression Selectors: the layer key evaluates a text Expression
Selector's amount for the first character only, so a per-character
dependency such as textIndex == 1 ? 0 : footage("ref.png").width was
not in the key, and after the footage item's width changed a reopened
session reused the old layer entry. A text layer with an Expression
Selector now marks its reads unkeyed (ReadFiles::is_unkeyed), which
merges like the generation; such buffers stay in memory and are never
written to disk. Frame entries are unaffected: the comp's frame key
hashes what the expressions name. The equality and hash of ReadFiles
ignore both new fields.

* Disk cache: never serve a RAM layer buffer made before a drop of its files

A render that passed the drop check could sit in put_layer's encode
while Reload Footage cleared the layer cache and dropped the file, then
publish its old pixels to memory after the purge; get, and a fresh
render's insert, then served them. The disk-hit promotion went through
the same insert_mem.

Every way the cache hands out a RAM buffer (get, the kept-first path of
insert, an existing entry in insert_mem) now checks the entry's files
against the store's drops from their own generation (ReadFiles::since):
an entry made before a drop of one of its files is forgotten and the
lookup misses, so a fresh render's buffer replaces it. insert_mem also
re-checks a buffer just before publishing it.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 02:30:59 +01:00
vangel76 14e0665ef4 Footage cache (#3271): 3D Channel effects declare the aux channels they read
AuxNeed / AuxLookup describe what an effect reads as the lookups it makes on the planes
(get, find, a Cryptomatte layer by Layer Name and slot); effects::aux_needs gives them for
EXtractoR, Cryptomatte and the 3D Channel effects, resolved against a file's channel names.
AuxChannels gains aliases (a multi-part file's stored channel names, Crypto_Material00.R in
part Crypto_Material) and listed (every name when only some planes are held); the Cryptomatte
layer slot rules move to raster so a declared need resolves the layer as the effect does.
2026-11-08 01:55:41 +01:00
vangel76 250e52cdc3 Merge remote-tracking branch 'upstream/main' into disk-cache-comp-scope 2026-11-07 21:38:56 +01:00