mirror of
https://github.com/storytold/effectcraft.git
synced 2026-11-08 11:31:25 +00:00
Valentin KleinandClaude Opus 5.5
84382487bb
* Disk cache: record the source files each frame reads Part of #3184 (step 1): renders now report which footage files they read, each with its identity (size and modification time) as read. Nothing uses the list yet: images, cache keys and cache behaviour are unchanged. - render: `reads` module with `FileStamp`, `ReadFile`, `ReadFiles` (sorted, deduplicated, bounded; marked incomplete when it can't vouch for every file) and the thread-safe `ReadLog`. `Renderer::reads` collects a frame's files; `FootageSource::recording` lets a source note what it reads. `file_stamp` uses the same identity. - Layer cache entries keep the files their buffer was made of and report them on every hit; derived entries (styles, blend space, flattened) keep their base's. Buffers from the disk cache or the browser store don't know their files yet and mark the list incomplete. - Media pool: every file read notes its identity, taken from the file it opens. Decoded frames, opened movies, GIFs, models and EXR channels keep it with the cache entry, so a frame served from memory reports its file without touching the disk. - Viewer frame worker collects each frame's list and carries it to where the disk entry is written (step 2 stores it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Automation: confined footage passes on the files it reads Part of #3184: ConfinedFootage forwards FootageSource::recording to the source it wraps, so renders of an automation session list their files instead of coming back incomplete. The read tests also render through the viewer's CancellableFootage wrapper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: frame entries keep their source files, comp key from memory Part of #3184 (step 2). Fixes #3175: opening a comp no longer stamps every footage file of the project before its first frame can show. - The comp's content key (`disk_cache::comp_content_key`) is taken from the project in memory alone: no footage or proxy file is stamped, no `VirtualFileId` is asked. `comp_content_key_with` is gone; `frame_key` hashes a new entry version, so entries of the earlier format are never looked up again (LRU evicts them). - Frame entries (formats 2 and 3) store the files the frame read (`EntryFiles`: size and modification time as read; size and content hash through `VirtualFileId` for files from memory; missing files), then the pixels. Formats 0 and 1 read as misses and are deleted. - Check on hit: `get_frame`/`get_frame_f16` look only at the listed files (one metadata call each) on the frame worker. A changed, back or unreadable file, or a damaged entry, is a miss: the entry is deleted and the frame rendered and stored again. - A frame whose file list is incomplete is shown but not stored. The RAM preview keeps each frame's files, so `persist_missing` writes only frames with a complete list. - Remote frame worker: `FrameServer` records each frame's reads (`take_files`); the web worker stores them with the entry and the page checks them through the file table before using it. The browser's media epoch existed only for the old key and is removed. Export does not use frame entries. - The cache indicator still only asks whether an entry exists. Tests: entry format round trip and damage, the check per file, and in the viewer: a changed file re-renders exactly its frames (and a reopen reads the rest from disk); a frame checks only its own files in a project with a large unused sequence and the key looks at no file; missing, unreadable files and damaged entries never serve a stale frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: layer entries keep their source files, no footage salt Part of #3184 (step 3 of 4). Layer buffers on disk (and in the browser's layer store) now carry the files they were made of, like frame entries since step 2, and are checked against exactly those files when used. The project-wide footage salt is gone: nothing stamps every footage file any more. - Layer entry format 4: the buffer's EntryFiles, then the pixels. A hit checks only the listed files (metadata, or the VirtualFileId hook for files from memory; Missing must still be missing); a change, a damaged entry or the old format 0 is a miss and the entry is deleted, so the re-rendered buffer replaces it. A hit reports its file list, so frames made of buffers from disk are complete and get stored again. - Buffers whose files can't all be named are not written. - Layer disk keys: layer_disk_key = hash of the layer entry version, the renderer version and the layer cache key (which names the footage items with their paths and settings, from memory). - PrefetchStore checks on use; an entry made of changed files is dropped and forgotten, so the new buffer is queued over it. FrameServer passes its virtual-file hook to its layer cache. - Removed: footage_salt(_with), file_stamp, footage_stamp, set_disk_salt, set_store_salt, Session::refresh_disk_salt, poll_disk_salt, footage_fingerprint, disk_salt, disk_salt_background, disk_salt_job, DiskSaltJob, the UI's per-frame poll, the salt in FrameServer and in the availability worker (with its cache_enabled plumbing). The disk cache is attached as soon as it is configured. - Docs: architecture.md, web.md. Tests: layer entries check their files (size, time, missing/back, folder, damaged, old format, incomplete); virtual files; prefetch store; media tests with a real pool across restarts (only the changed file is read, projects with the same item ids never share an entry, missing, unreadable and damaged never serve stale buffers); engine tests that the cache is attached at once and edits do no file work (#1926, #3106). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: no-salt worker test, sequence paths in the frame-blend key Part of #3184 (step 3 of 4). - A frame-blended footage frame's layer-cache key now hashes the sequence's file list: with no project-wide salt, two projects whose items share an id and a first frame must not share its disk entry. - Test: a frame worker syncing, patching and rendering a project with an unused 5000-frame sequence asks its file hook only about the file its buffer reads (the salt asked about every file at every sync). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: the comp key covers what its expressions can read With an expression anywhere in the project, every comp's disk key hashed every item of the project, so any edit (another comp, an unused footage item) threw away every comp's frames on disk. The key now covers what the comp can draw plus what the expressions among it can reach (Project::pixel_reach): - without expressions: the items the comp draws, without silent layers, plus the footage its precomp layers swap in (Media Replacement); - with expressions: every layer of the comps they reach, their markers, the comps and footage they name with string literals (comp("Name"), footage("Name")) together with which items have those names now, and the footage named like a reached source (its data is read by name); - the whole project when an expression computes a name, uses the object model's constructors or internals, dynamic code, computed keys or writes, or text the scanner can't read for sure (regex literals, template strings, escapes): expression_reach errs towards reaching more. Markers count only where an expression can read them (#1083). Memory only, no file is looked at. Tests: scanner and reach unit tests in the project crate; engine tests for the key (own-comp expressions keep the key through edits elsewhere, comp("Other") follows Other and not a third comp, a second comp taking the name counts, a computed name counts every item, markers read from another comp); a ui-egui test that the disk frames outside an expression layer are carried when the comp it names changes. Part of #3184. Part of #1492. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: Reload Footage drops the entries of its files, docs File > Reload Footage now deletes the frame and layer entries whose file lists name the reloaded item's files (DiskCache::drop_files), so a file replaced by one with the same size and modification time, which the check on hit can't tell apart, is read again. The index knows keys, not files: the writer thread reads the files section at the start of each frame and layer entry (not the pixels) after the writes queued before it, and deletes the entries that list one of the files. Until it is done, an entry made of one of them is a miss and frames aren't linked to new keys. The entries are gone for good, so they don't come back after a restart. Automatic reloads don't drop anything: they follow a changed stamp, which the check on hit catches. The browser needs nothing (its file identities are content hashes). Removed: disk_cache::seal, which steps 2 and 3 left without callers (the tests use seal_format). Updated the module, seal_format and forget_media docs, and docs/architecture.md / docs/web.md describe the final design: what is cached, keys from memory, file lists checked on hit, nothing scans the project, incomplete lists stay in memory, Reload Footage drops entries, edits keep the frames they can't change. Tests: render unit tests (same size and time passes the check but not the drop; other files' entries stay; an entry queued before the drop is dropped; a layer entry too; written again afterwards it is used; the drop survives a restart; a pending drop is a miss), and an engine test through file.reloadFootage. Closes #3184. Part of #1492. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: file stamps keep the whole modification time A file's modification time was nanoseconds since 1970 as u64: a time before 1970 read as "no time" (two such files of one size looked alike) and times after 2554 were truncated. It is now i128 nanoseconds, negative before the epoch; None only means the file system has no time. The content-key hash writes a presence tag and the full value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Automation: a refused footage read leaves the render's file list incomplete Footage outside the read roots shows nothing and was noted as no file at all, so a picture made without it had a complete, empty list and could be kept on disk for a later session with other roots. A refused read during `recording` now marks the list incomplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render: a buffer kept by another worker comes with its own files LayerCache::insert returns the files of the buffer it keeps, and Renderer::keep notes them in the render's log and returns them, so a render that uses another worker's buffer lists that buffer's files (a file changed between the two reads then has two identities and the list is incomplete). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render: sampleImage and text fonts note the files they read EvalCtx carries the render's read log. A sampleImage render records its footage reads, keeps them with its per-thread cached buffer and notes them in the evaluating render's log on every use. Text notes the font files it is drawn with, with the identity their bytes were read with; a family that isn't installed, a fallback or missing glyph, or a font from memory leaves the list incomplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: frame entries store the whole modification time A frame entry's file list stores a file's modification time as i128 nanoseconds since 1970 (negative before), as the file stamps now keep it; the frame entry format version goes to frame-entry-files-2 so entries in the old layout are not read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: layer entries store the whole modification time Layer entries share the frame entries' file list layout, which now stores the full i128 modification time: the layer entry format version goes to layer-entry-files-2. The module doc names the font files text is drawn with among an entry's source files, and what keeps a list incomplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Layer cache: buffers that read other layers are keyed by the switches passed down A layer whose effect reads another layer (Gradient Wipe, Set Matte, Blend…) gets that layer drawn by its own renderer. Through a precomp layer, Switches Affect Nested Comps, the switches inherited from the precomp layers above and motion blur on change those pixels, but the key held only the layers' own switches: a buffer made with the setting on was reused (from memory or the disk cache) with it off. Such layers now derive their key with what the renderer passes down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Layer cache: masked buffers are keyed by the mask motion blur gate A mask with Mask Motion Blur On blurs even when the layer's own switch is off, as long as the render's gate is open (Render Settings motion blur, and with Switches Affect Nested Comps the precomp layers above). The key held the layer's own motion blur state only, so a buffer blurred with the gate open was reused with it closed (and the other way round), also from the disk cache. Masked layer, input and footprint keys now tell a closed gate apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render: cached buffers list what their expressions and substituted characters read - Buffers made under `Renderer::recorded` (layer content, a Time effect's input, an adjustment layer's footprint) evaluate their masks, shapes and effects with the context that notes into the buffer's own read log, so a file an expression reads (`sampleImage`, also at shutter times mask motion blur adds and the cache key doesn't see) is in the buffer's files. - A styled layer served from the cache lists the files of the cached body and passes too, so the buffers made of it (flattened, blending space) don't get a complete list when one of them came with an incomplete one. - A character put in by Character Offset or Character Value without a relayout notes the font it is drawn with; a fallback face (or none) leaves the list incomplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render: text lists the font file of a run that draws no glyph; tests clean their temp dirs An empty first paragraph draws no glyph, but in a box its face sets the first baseline that places every later line. The read list took faces only from glyphs, so it could be complete without that file. Note each style run's resolved face as well. The font-reads and confine tests use tempfile::tempdir(): a fresh directory per run, removed on panic too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: Reload Footage closes the gaps around its drop (#3184) - A reload while the disk cache is off is kept on the session (pending_disk_drops) and applied when the cache is switched on, before anything is read from it. - Each render takes the cache's drop generation before it reads (frame worker; layer cache at the miss). An entry listing a file dropped after that is refused under the queue's lock, so a render that was under way during the reload can't store pixels from before it. Such a frame is shown with an incomplete file list, so persist_missing skips it too. The per-file record is bounded (past it, everything older is refused). - An entry read while its drop was pending is a miss even when the writer ends the sweep between the read and the check (a stale hit found under load): the pending drops and the generation are taken before the entry is read. - Test: a reopened project serves its frame from disk with no source file opened or decoded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Disk cache: RAM frames keep the drop generation their render started at Reload Footage empties the RAM preview, but an idle pass of persist_missing may already hold a frame from before the reload. It took the cache's drop generation when it ran, after the drop, so the old pixels could be written back and, with the file's size and time unchanged, served again after a restart. Each RAM frame now keeps the disk cache its render started with and that cache's drop generation then. persist_missing writes a frame only to that cache, with that generation, and skips it once one of its files was dropped since; a frame rendered without that cache is not written from there. Tests use tempfile directories (the layer drop test, the frame_files folders under target/). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Doc comments for the functions this PR touches * Disk cache: cap the free-space reserve at 10 GB, skip encodes the limit refuses The reserve the cache leaves free on its disk was max(2 GB, 5% of the disk) (#2551). On a 4 TB disk with 190 GB free that is 200 GB, so a cache set to 20 GB had a limit of 0: every write was refused, the cache never held anything, and only a log warning said so. The reserve is now clamp(5% of the disk, 2 GB, 10 GB). It is there to keep the cache from filling the disk, not to keep a share of a large disk free: 10 GB leaves the system room to work on any disk, a small disk still keeps 2 GB, and a 100 GB disk keeps 5 GB as before. After Effects has only a maximum cache size and no reserve; a clamp keeps the floor for small disks, where a fixed 10 GB would take most of the space. Writes the limit refuses cost no work: DiskCache::accepts_writes (one atomic read) is checked before a frame is converted (put_disk_frame) and before frames and layers are compressed (put_frame, put_frame_f16, put_layer). The frame workers spent about 8 ms per frame and 14 ms per layer on encodes that were then refused. persist_missing returns at once at limit 0 instead of converting the RAM frames on every idle pass. Settings > Disk shows the limit in effect in MB under Maximum Disk Cache Size, with a note when free disk space lowers it (settings.disk.cacheLimit). * Disk cache: refuse writes below 64 MB of room, show this disk's reserve Review follow-ups to the reserve fix: - accepts_writes also refuses while free disk space leaves less than 64 MB of room (a limit the setting asks for is always used): a few frames would fill it, and the rest would be compressed only to be refused. - While writes are refused the idle passes ask for no repaint, so the UI keeps a frame coming every 5 s: the free space is read again and the RAM frames are written once there is room. - Settings > Disk names the space kept free on this disk (the reserve the last reading applied; the rule from MIN/MAX_DISK_RESERVE when it is unknown), and the "no room" note follows accepts_writes. - The settings.disk.cacheLimit automation label is machine-readable and the same in every language: limitMb, maxMb, reserveMb, spaceLimited, writes. - cargo fmt. * Disk cache: follow 3D model sources in the comp's reach The reach walk followed comp, footage and solid sources but not LayerSource::Model. An expression reading a model layer's source (thisComp.layer("Model").source.name) left the model item out of the comp's content key, so renaming the model kept the key and the old frames on disk were served. Every source item is followed now (LayerSource::item), which also counts the model when no expression reads it (Advanced 3D draws it). The same omission is fixed in the RAM preview's content walk (frames.rs) and in the layer key's walk of referenced layers (cache/dependencies.rs). * Disk cache: a render's drop generation travels with its reads; no disk entries for Expression Selectors Drop generation per render: the layer cache recovered a buffer's drop generation from the latest miss of its key, so a second render of the key that started after File > Reload Footage lent its generation to a render still running from before, and that render's buffer (the file as it was) was written after the sweep. Now Renderer::recorded starts each buffer's read log at the store's drop generation before anything is read (ReadLog::starting_at); ReadFiles carries the earliest generation through merges, so derived buffers (styles, the blend-space copy, the flattened layer) and frames that use a buffer from memory carry it too. LayerCache::insert keeps a buffer made of a file dropped since its reads began neither on disk nor in memory; a list that doesn't say counts as made before every drop. DiskCache::dropped_since takes the list's own generation too. missed_at keeps only the miss time. Expression Selectors: the layer key evaluates a text Expression Selector's amount for the first character only, so a per-character dependency such as textIndex == 1 ? 0 : footage("ref.png").width was not in the key, and after the footage item's width changed a reopened session reused the old layer entry. A text layer with an Expression Selector now marks its reads unkeyed (ReadFiles::is_unkeyed), which merges like the generation; such buffers stay in memory and are never written to disk. Frame entries are unaffected: the comp's frame key hashes what the expressions name. The equality and hash of ReadFiles ignore both new fields. * Disk cache: never serve a RAM layer buffer made before a drop of its files A render that passed the drop check could sit in put_layer's encode while Reload Footage cleared the layer cache and dropped the file, then publish its old pixels to memory after the purge; get, and a fresh render's insert, then served them. The disk-hit promotion went through the same insert_mem. Every way the cache hands out a RAM buffer (get, the kept-first path of insert, an existing entry in insert_mem) now checks the entry's files against the store's drops from their own generation (ReadFiles::since): an entry made before a drop of one of its files is forgotten and the lookup misses, so a fresh render's buffer replaces it. insert_mem also re-checks a buffer just before publishing it. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
815 lines
69 KiB
Markdown
815 lines
69 KiB
Markdown
# Architecture
|
||
|
||
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-29 · **Change:** minor (SAM 2 crate added to the layer table) · **Target:** Adobe After Effects 2026
|
||
|
||
EffectCraft is a stack of small Rust crates. The engine knows nothing about the user interface;
|
||
the egui frontend, the command-line tool and the MCP server all sit on top of the same `Session`
|
||
and drive it through the same command registry.
|
||
|
||
## 1. Crates and layering
|
||
|
||
Dependencies only point downward (or along the few same-layer edges listed below). `cargo xtask
|
||
layers` enforces this, and fails the build if a crate below L5 depends on egui, eframe, winit, rfd,
|
||
cpal or muda. Everything in L0 to L4, the egui UI and the web app also build for
|
||
`wasm32-unknown-unknown` (`cargo xtask wasm`); see [web.md](web.md).
|
||
|
||
| Layer | Crate (`effectcraft-…`) | Responsibility |
|
||
|---|---|---|
|
||
| L0 | `time` | `Tick` (254 016 000 000 per second), rational frame rates incl. NTSC, SMPTE and drop-frame timecode |
|
||
| L0 | `geom` | Vectors, matrices, quaternions, the layer transform (anchor, position, scale, orientation, rotation) |
|
||
| L0 | `color` | sRGB and linear, HSL/HSV, luminance, the 38 blend modes, label colors |
|
||
| L0 | `vp9enc` | VP9 encoder (profile 0, 8-bit 4:2:0, key and inter frames with motion search, loop filter and rate control; lossless at quality 100) for WebM export, from the VP9 bitstream specification |
|
||
| L0 | `opusenc` | Opus encoder: RFC 6716 SILK (NB/MB/WB), hybrid (SWB/FB) and CELT (FB) modes chosen by bitrate and application (audio / voice), 48 kHz 20 ms packets, mono/stereo, and the RFC 7845 `OpusHead`, for WebM export audio |
|
||
| L0 | `hevcenc` | HEVC (H.265) encoder from ITU-T H.265: Main / Main 10 4:2:0, IDR + P slices (quarter-pel motion, merge/AMVP), CABAC, deblocking, bitrate or constant-QP rate control, for MP4 (`hvc1`) export |
|
||
| L0 | `av1enc` | AV1 encoder from the AV1 bitstream specification: Main profile 8/10-bit 4:2:0, key + inter frames (quarter-pel motion), deblocking, for MP4 (`av01`) and WebM export |
|
||
| L1 | `raster` | Premultiplied float images, sampling, affine and projective warps, blurs, compositing (parallel with rayon) |
|
||
| L1 | `keyframe` | Animated values, keyframes with temporal ease and spatial Bezier, roving, hold, velocity |
|
||
| L1 | `segment` | Swappable trained models for Roto Brush and face tracking: the `MaskModel` and `FaceModel` interfaces, a registry of open-licensed models (authors, licence, URL, size, SHA-256), PyTorch checkpoint and TensorFlow Lite readers (with an interpreter), SHA-256, MobileSAM (TinyViT + Segment Anything decoder) and MediaPipe Face Landmarker (BlazeFace + Face Mesh V2) on rayon and ndarray's safe GEMM |
|
||
| L1 | `psd` | Photoshop PSD/PSB reader (layers, groups, masks, blend modes, text, layer effects, adjustment layers; 8/16/32-bit; RGB/CMYK/Gray/Lab) and a minimal writer, from Adobe's published format specification |
|
||
| L1 | `path` | Bezier paths, path operators (trim, offset, round corners, zig zag, twist, merge…), stroking, coverage masks |
|
||
| L2 | `project` | The document: items, compositions, layers, the property tree, render queue model, `.ecproj` serde |
|
||
| L2 | `text` | Fonts, shaping, layout, per-glyph geometry, text animators and selectors |
|
||
| L2 | `effects` | The effect registry (306 effects) and their CPU implementations |
|
||
| L2 | `svg` | SVG import (W3C SVG 1.1/2 static subset: shapes, paths, transforms, `use`, CSS, gradients) and rasterisation at any scale |
|
||
| L2 | `pdf` | PDF, PDF-compatible Illustrator (`.ai`) and EPS (PostScript subset) vector footage into the `svg` render tree: paths, fills, strokes, shadings, tiling patterns, clipping, text (embedded TrueType / CFF / Type 1 / Type 3 fonts; the bundled `text` fonts for the standard 14), images, blend modes, soft masks, optional-content layers |
|
||
| L2 | `model` | 3D models for Advanced 3D: glTF 2.0 (`.gltf`/`.glb`) and OBJ/MTL import (meshes, PBR metallic-roughness materials and textures, node hierarchy, skins, animations), parametric primitives, extruded/bevelled outline meshes and polygon triangulation |
|
||
| L2 | `track` | Motion tracking: feature/search region point tracking (pyramid normalized cross-correlation, Lucas–Kanade sub-pixel refinement), confidence, homography/affine/similarity solves |
|
||
| L3 | `render` | Evaluation and compositing: sources, masks, effects, transforms, 3D, motion blur, mattes, blending, layer cache, audio mixdown |
|
||
| L3 | `media` | Footage decoding (FilmCraft's pure-Rust codecs), image sequences, Photoshop and SVG stills, frame cache |
|
||
| L3 | `expr` | The expression engine (JavaScript via boa) with the After Effects object model |
|
||
| L3 | `export` | Render queue encoding: H.264, HEVC and AV1 MP4, ProRes, WebM (VP9 + alpha or AV1, Opus), PNG/JPEG/TIFF/EXR/Targa sequences, GIF, WAV/AIFF |
|
||
| L3 | `gpu` | The GPU compositor and GPU effects on wgpu compute shaders (Metal, Vulkan, Direct3D 12, WebGPU), checked against the CPU renderer |
|
||
| L3 | `sam2` | SAM 2.1 image segmentation and video object tracking for Roto Brush, with candle CPU inference and optional wgpu acceleration; native only, empty on wasm32, with user-installed weights |
|
||
| L3 | `interchange` | Premiere Pro interop through timeline interchange (FCP7 XML, FCPXML, OTIO, EDL, AAF, OMF via FilmCraft's `filmcraft-interchange`): sequences ↔ compositions (§6a) |
|
||
| L3 | `aep` | After Effects projects in one crate (like `lottie`): `container` reads and writes the `.aep` container (RIFX `Egg!`) as a lossless chunk tree with a byte-identical round trip, bounded for hostile input, reads the `.aepx` XML form into the same tree, and has a typed view of items, layers, properties, effects, masks, shapes, text, markers, footage, cameras, lights and layer styles, from our own observations of generated files ([aep-format.md](aep-format.md)); `import` maps the typed view into a project (items, settings, compositions, every layer kind, Transform, keyframes, expressions, masks, text, shapes, cameras, lights, mattes, markers, time remapping, layer styles, the effects of a shared table, footage) with a report of everything not imported ([aep-import.md](aep-import.md)); `export` builds every chunk in code (prototypes in the layout After Effects writes, cloned and edited only at verified fields) for the same scope, with a report of everything not exported, verified by opening the output in After Effects ([aep-export.md](aep-export.md)) |
|
||
| L3 | `lottie` | Lottie JSON / dotLottie import and export (layers, precomps, eased and spatial keyframes, shapes, masks, mattes) with a warnings list for what Lottie cannot express |
|
||
| L3 | `plugin` | Effect plug-ins: loads sandboxed WebAssembly effect modules (plug-in API v1; the wasmi interpreter behind the `wasm` feature, on for native hosts) into the effect registry ([plugins.md](plugins.md)) |
|
||
| L3 | `helper-host` | External effect helpers: finds helper programs, runs each as a child process and registers its effects (plug-in API 2) over a versioned pipe + shared-file protocol; a crashing helper is an error on the effect, never a crash ([external-helpers.md](external-helpers.md), [ADR 0001](adr/0001-external-effect-helpers.md)). Native only |
|
||
| L4 | `engine` | `Session`: project, branching undo history, editor state, the command registry and menus, the ScriptUI window model (`scriptui`) |
|
||
| L4 | `script` | Scripting: JavaScript (boa) with an After Effects-style object model (`app.project`, comps, layers, properties, render queue) and ScriptUI (dialogs, palettes, dockable panels; script-host threads so dialogs block `show()`), whose edits run engine commands; AE match names |
|
||
| L4 | `host` | A fully wired `Session` (media, expressions, scripting, WebAssembly plug-ins, exporter) for the frontends |
|
||
| L5 | `ui-egui` | The desktop interface: docking, panels, viewer, timeline, graph editor, dialogs, control channel |
|
||
| L5 | `automation` | The MCP server, headless or bridged to the running app |
|
||
| L6 | apps `effectcraft`, `effectcraft-cli`, `effectcraft-web` | Desktop app; command-line tool (render, exec, get/set, MCP); the browser app (wasm32, [web.md](web.md)) |
|
||
|
||
Allowed same-layer edges: `path → keyframe, raster`, `text → path`, `pdf → svg, text`, `effects → project, text, path, track`,
|
||
`media / expr / export / gpu → render`, `export → media`, `lottie → format`, `host → engine, script`,
|
||
`script → engine`.
|
||
|
||
The frontend uses `craft-ui::docking` for atomic layout transactions and pointer drop targets,
|
||
and `craft-ui::layout` for bounded split geometry and resize interactions. A small adapter
|
||
preserves EffectCraft's saved dock tree, stacked entries, floating rectangles and dynamic
|
||
viewer/ScriptUI identities. Tab moves, edge splits, reordering, float/redock, activation and
|
||
close operations use the shared model. The app retains composition-tab behavior, accordion
|
||
height allocation and ScriptUI close callbacks. Lower layers cannot depend on `craft-ui`.
|
||
|
||
Viewer controls retain their native widgets inside a horizontal scroll area, so narrow panels
|
||
keep dropdowns, exposure dragging and snapshot holds reachable. Keyboard focus scrolls the
|
||
focused control into view. Active composition breadcrumbs use the selected-row theme color.
|
||
The `viewer_geometry` tests inspect actual painted contrast, scroll geometry and pointer/keyboard
|
||
reachability across the three themes.
|
||
|
||
## 2. Time
|
||
|
||
All time is an integer `Tick`, 254 016 000 000 per second, the least common multiple of every
|
||
broadcast frame rate (×1001) and audio sample rate, so frame and sample positions are exact.
|
||
Keyframe times are in **layer time**; `layer_time = (comp_time - start_time) / stretch`, and a
|
||
Time Remap property overrides the source time. Commands snap layer times, keyframe times and
|
||
comp durations to whole frames, as After Effects does, except Quick Offset (`layer.quickOffset`,
|
||
`keys.quickOffset`), which staggers layers and keys onto subframes, as After Effects 25.4 does.
|
||
Timecode is only a display.
|
||
|
||
## 3. Document model
|
||
|
||
```
|
||
Project { settings, items, render_queue }
|
||
Item { id, name, label, parent folder, kind: Folder | Comp | Footage | Solid }
|
||
Comp { size, pixel aspect, frame rate, duration, start timecode, background, work area,
|
||
layers (index 0 = layer #1), markers, motion blur shutter, renderer }
|
||
Layer { id, name, source (footage | comp | solid | text | shape | null | camera | light),
|
||
in/out/start, stretch, switches, blend mode, track matte, parent, props }
|
||
```
|
||
|
||
Everything animatable is a `Property` in the layer's `PropGroup` tree, addressed by a path such as
|
||
`transform/position`, `effects/#1/blurriness`, `masks/#1/feather`, or by `@uid`. Effects, masks,
|
||
text animators and shape contents are groups in the same tree.
|
||
|
||
A project file (`.ecproj`) is the project as readable JSON, headed by `savedBy` (the EffectCraft
|
||
version that wrote it) and `schema`; `.ecprojx` is the same data as XML (File ▸ Save a Copy As
|
||
XML). Saving goes through `Project::to_file_json`, which checks that the text reads back before
|
||
anything is written (JSON can't hold NaN or infinity), and files are written atomically. Opening
|
||
a file that a newer EffectCraft saved shows a warning: fields this version doesn't know are
|
||
dropped if it saves the project. Save, Save As, Save a Copy and Increment and Save keep the
|
||
file's format (`.ecprojx` stays XML). Next to `savedBy` the file records the active comp and the
|
||
open comp tabs (`activeComp`, `openComps`), which opening restores; they are editor state, not
|
||
part of `Project`, so older files (and ids that no longer name a comp) open on the first comp and
|
||
older versions ignore them.
|
||
|
||
`schema` changes when a saved value changes meaning; a project from an older schema is converted
|
||
when it is opened (`effectcraft_engine::upgrade_effects`, which also refreshes effect instances
|
||
from the registry) and saved with the current one, which older versions refuse to open rather than
|
||
render wrongly. Schema 2 put effect positions on shape and text layers in effect space (§4,
|
||
#227). Opening a schema 1 project moves what an effect on a shape or text layer (not an adjustment
|
||
layer, whose effects see the comp below as before) holds as a position by half the comp size,
|
||
so points that were set keep their place in the comp: point parameters with their keyframes, Paint
|
||
stroke paths and clone positions, Puppet mesh seeds, pin rest points and positions, and Liquify
|
||
and Roto Brush strokes. A point parameter still at its default and not animated is left alone:
|
||
the old default sat at the comp's bottom-right corner, where most effects did nothing, and the
|
||
same value is now the layer centre, as in After Effects. Expression controls (Point Control…)
|
||
keep their values, because their meaning is up to the expressions that read them, and expressions
|
||
are not rewritten, so an expression that computes an effect point on a shape or text layer from
|
||
other values may need `+ [thisComp.width / 2, thisComp.height / 2]`. Effects that use the layer
|
||
bounds (edge pinning, wipes, Offset, generators) now cover the whole comp-sized bounds instead of
|
||
their bottom-right quarter. Tracking data of effects that analyse footage (Warp Stabilizer, 3D
|
||
Camera Tracker, Mocha shape) is left as it is: footage has a source rectangle, whose effect space
|
||
did not change.
|
||
|
||
## 4. Rendering
|
||
|
||
`Renderer::comp_frame(comp, t)` walks the layers bottom to top. For each visible layer it renders
|
||
the **source** (solid, footage frame, text, shape contents, or a nested comp), applies **masks**,
|
||
then **effects** in order, then **layer styles**, then the **transform** into comp space (with motion blur sub-samples
|
||
when enabled), the **track matte**, and finally **blends** into the accumulator. Adjustment layers
|
||
apply their effects to the accumulator.
|
||
|
||
Effects run in **effect space** (`effectcraft_render::effect_bounds`): layer pixels measured from
|
||
the top-left of the layer's effect bounds, which are the source rectangle, or for layers without
|
||
one (shape, text) a comp-sized rectangle centred on the layer's origin that their content
|
||
surrounds. A shape layer's default effect point (the bounds' centre) is therefore its content's
|
||
origin, as in After Effects. The renderer moves the layer buffer into effect space for the whole
|
||
stack (`Buf::rebase`, on the CPU and the GPU alike) and hands effects everything else in it too:
|
||
masks, other layers' pixels for layer parameters (each in its own effect space), the layer's own
|
||
frames at other times (`Renderer::layer_input`, also what Puppet, Roto Brush and the Layer panel
|
||
read) and the comp camera. Effect point handles, Puppet pins and the Layer panel map effect space
|
||
to the comp with `EvalCtx::effect_to_comp`. Adjustment layers run their effects on the comp below
|
||
as it is.
|
||
|
||
Runs of 3D layers are composited per pixel through the
|
||
active camera, with lights, shadows and depth of field (Classic 3D), or rasterised as meshes by
|
||
the Advanced 3D renderer (below).
|
||
|
||
**Advanced 3D** (Composition Settings ▸ 3D Renderer ▸ Advanced 3D, `comp.renderer`;
|
||
`crates/render/src/three_d/adv`, `crates/model`). A run of 3D layers becomes one scene of
|
||
world-space triangles: 3D model layers (glTF 2.0 / OBJ footage, mapped from +Y-up model units
|
||
into the layer by Geometry Options ▸ Model Scale; skins and animation clips played along layer
|
||
time), primitive layers (Layer ▸ New ▸ Cube/Sphere/Plane/Torus/Cone/Cylinder, parametric in
|
||
Geometry Options), text and shape layers with Geometry Options ▸ Extrusion Depth (glyph and
|
||
fill outlines flattened, triangulated with holes and extruded with Angular/Concave/Convex
|
||
bevels, per character so per-character 3D carries over), and a textured card for every other
|
||
3D layer. Classic 3D doesn't draw model or primitive layers, so adding one switches a Classic 3D
|
||
comp to Advanced 3D in the same undo step, as in After Effects. Shading is physically based in linear light (glTF metallic-roughness; Cook–Torrance
|
||
GGX specular, Smith–Schlick geometry, Schlick Fresnel, Lambert diffuse, after Karis 2013):
|
||
parallel, spot and point lights with After Effects' falloffs, ambient lights, image-based
|
||
light from an Environment light whose Source (or the comp's Environment Layer, Layer ▸
|
||
Environment Layer, or a default studio environment — bright above and toward the initial
|
||
camera, dark below and behind, at a dim 0.3 mean level — of the light's colour × intensity
|
||
when there is neither) is an equirectangular image (cosine irradiance map; roughness-indexed
|
||
mip chain with the split-sum environment BRDF fit), and percentage-closer-filtered shadow maps
|
||
(spot: perspective, parallel: orthographic over the scene, point: six cube faces, environment:
|
||
one orthographic from the image's dominant bright region — none only when the image lacks a
|
||
sufficiently strong dominant direction (a uniform one, or opposing bright regions that
|
||
cancel) — the region weighted by the light's colour, so a coloured light shadows from the
|
||
regions it actually lights (opposing red and blue regions cancel to a white light, but a red
|
||
light's shadow comes from the red one) — occluding only the image's directional key fraction of
|
||
the light, so the shadow stays partial like After Effects' area-light shadows, with a soft
|
||
default penumbra of at least eight pixels or four map texels). A Source or Environment Layer
|
||
that fails to render — a deleted layer, missing footage — lights nothing: no silent fallback
|
||
to the default studio environment. An Environment light only lights Advanced 3D comps,
|
||
so adding one (or changing a light's type to it) switches a Classic 3D comp to Advanced 3D,
|
||
as model layers do. Cards keep
|
||
Material Options (Accepts Lights/Shadows, Casts Shadows On/Only, Ambient, Diffuse, Specular);
|
||
models and primitives add Base Color, Metallic, Roughness, Emissive. Rasterisation runs at 2×2
|
||
supersampling with a reversed-Z depth buffer; opaque triangles resolve visibility first, the
|
||
transparent ones are sorted back to front and blended. The software rasteriser
|
||
(`adv::raster`) is the reference and the headless/CI path; `effectcraft-gpu` implements
|
||
`Accelerator::raster_3d` with a wgpu render pipeline (`advanced3d.wgsl`) that repeats the
|
||
shading step for step, and tests compare the two. Resolve, depth of field (a gather blur by
|
||
each pixel's circle of confusion from the camera's Depth of Field settings) and the conversion
|
||
back to the working encoding run on the CPU for both. **Motion blur** renders the scene at
|
||
sub-samples spread over the comp's shutter (angle, phase, samples per frame) — layers with the
|
||
Motion Blur switch, the camera and the lights at the sub-sample time, other layers at the frame
|
||
time — and averages them. Layers with a **blend mode, a track matte or Preserve Transparency**
|
||
are rendered on their own through the same camera and lights, hidden where the rest of the run
|
||
is nearer, and composited from the farthest to the nearest through the 2D path (a 3D matte is
|
||
itself drawn through the camera). **Environment Light Background** layers (Layer ▸ Light) draw
|
||
their equirectangular image behind the run, looked up by each pixel's view direction. Not yet:
|
||
collapsed 3D precomps (drawn flattened), morph targets, extruded strokes.
|
||
|
||
A **layer cache** keeps each layer's finished pixels (source, masks and effects) keyed by a hash of
|
||
its evaluated inputs, excluding the transform. Static and transform-only layers render once;
|
||
editing one layer re-renders only that layer. Each Render Queue job keeps its own layer cache
|
||
for its frames. Effects that read the clock directly are declared in
|
||
`effects::TIME_DEPENDENT`, and a test checks every registered effect against that list.
|
||
|
||
The viewer's **RAM preview** (`ui-egui/src/frames.rs`) keeps finished frames keyed by comp
|
||
content, frame, scale, 3D view / region of interest and a hash of the other render options
|
||
(Draft / Fast Previews, shadows, nested switches). Edits preserve unaffected comps' frames,
|
||
and undo can reuse earlier content. During rapid edits at the same time and view, the viewer
|
||
shows the newest completed revision while the exact requested revision renders; out-of-order
|
||
completions cannot move it backwards. This avoids freezing until a slider or layer drag stops.
|
||
Frames at other times or with other render options are never substituted. Obsolete queued jobs
|
||
are dropped; running jobs finish into the cache. When the budget (Settings ▸ Memory & CPU) is
|
||
full the least recently shown frames go first. The timeline's green bar counts cached frames
|
||
of the current content and viewer settings.
|
||
Resolution Auto renders the pixels the magnification needs, as in After Effects (Full above
|
||
50 %, Half down to 33.3 %, Third down to 25 %, then Quarter). The Resolution is the comp's
|
||
(`Comp::resolution`, saved with the project): setting it is no undo step (undo, redo and history
|
||
jumps carry the current values over) and, like the Shy switches, it is outside
|
||
`Comp::same_pixels`, so cached frames stay valid. Below 100 % the viewer averages
|
||
instead of skipping pixels (Viewer Zoom Quality More Accurate): GPU frames carry a mip chain
|
||
sampled trilinearly, CPU frames go up averaged down by the whole factor that leaves about one
|
||
texel per screen pixel.
|
||
|
||
32 bpc projects keep **half-float frames** on the desktop (`frames::F16Frame`, 8 bytes a pixel:
|
||
the RAM preview holds half as many as of 8-bit frames). They are rendered without Project
|
||
Settings ▸ HDR (`RenderOpts::skip_hdr_step`), so the values above 1.0 stay. The viewer's
|
||
**display pass** (`engine::viewer::DisplayPass`; on the GPU `ui-egui/src/display_gpu.rs` +
|
||
`display_gpu.wgsl`) then turns a frame into what the panel shows, in this order: decode with the
|
||
output space's curve, the viewer's exposure in linear light, the HDR step the frame was rendered
|
||
without (Compand / Tone Map), encode, clamp to 0..1, the display colour conversion (Use Display
|
||
Color Management, Simulate Output, LUT profiles), Show Channel, premultiply, round to 8 bits.
|
||
So exposure reaches the full float range, and tone mapping and the monitor conversion see the
|
||
exposed light. The pass runs on egui-wgpu's device (it needs no GPU compositor) into one RGBA8
|
||
texture with mips per viewer slot (main viewer, snapshot, each passive viewer: about 110 MB of
|
||
video memory per slot for a 4K frame, input included); without a device the same steps run on
|
||
the CPU (`DisplayPass::pixel`, which is also the shader's test reference). 8 and 16 bpc frames
|
||
stay 8-bit with the HDR step applied when rendered; they go through the same pass when exposure,
|
||
a channel view or a display conversion applies, else they are drawn as they are. Panels that
|
||
read the viewer's pixels as 8-bit (Video Preview, keyers, thumbnails) get the half-float frame
|
||
converted on demand with the HDR step and without the exposure; the Info panel shows the
|
||
frame's float values. The browser keeps 8-bit frames for now (its frame workers send RGBA8).
|
||
|
||
A persistent **disk cache** (`render::disk_cache`, Settings ▸ Media & Disk Cache) backs both
|
||
the layer cache and the viewer's RAM preview: layer buffers that were slow to render, every
|
||
preview frame and Roto Brush / Object Matte segmentations are written (LZ4, atomic rename,
|
||
checksummed) and survive restarts. The design (#3184):
|
||
|
||
- **Keys from memory.** A frame's key is its comp's content key plus frame, scale, view and
|
||
render options; the content key hashes the project settings and the items the comp can draw
|
||
(its layers but those that can't change the picture, nested comps, footage with its settings
|
||
and paths, solids, Media Replacement footage, proxies: `Project::pixel_reach`). Expressions
|
||
among them add what they can read: every layer of their comps, the markers, the comps and
|
||
footage they name with literal names (`comp("Name")`, `footage("Name")`, and which items have
|
||
those names), or the whole project when they compute a name or reach items another way
|
||
(`expression_reach`). An edit to anything else keeps the key. A layer buffer's key is the
|
||
layer cache's content key (which names the footage it uses with its paths) and the renderer
|
||
version. Nothing here looks at a file, and nothing ever scans the project's files.
|
||
- **File lists checked on hit.** Every frame and layer entry keeps the source files it was
|
||
made of, with the size and modification time they were read with (the browser: size and
|
||
content hash). An entry is used only while they are unchanged: one `metadata` call per listed
|
||
file when it is read. A changed, missing or unreadable file, a damaged entry or an older
|
||
format is a miss; the entry is deleted and the frame renders again and replaces it.
|
||
- **Incomplete lists stay in memory.** A frame or buffer whose files can't all be named (a
|
||
source that doesn't say what it read, more than 4096 files) is shown but never written.
|
||
- **Reload Footage drops entries.** File ▸ Reload Footage deletes the frame and layer entries
|
||
whose lists name the item's files (the writer thread reads each entry's list, not its
|
||
pixels), so a file replaced by one with the same size and time is read again. A reload while
|
||
the cache is off is remembered for the session and applied when it is switched on, before it
|
||
is read. A render already under way offers entries made of the files as they were: each
|
||
render takes the cache's drop generation before it reads, and an entry listing a file dropped
|
||
after that is refused (`DiskCache::drop_generation`); the frame is shown, never stored.
|
||
- **Edits keep what they can't change.** RAM frames outside an edit's dirty spans are kept
|
||
(`Project::dirty_spans`), and their disk entries are linked to the comp's new key.
|
||
|
||
Frame entries are 8-bit RGBA, or for 32 bpc viewer frames half floats with their bytes shuffled
|
||
into planes before LZ4 (the entry header's format byte tells them apart; the render options in
|
||
the key keep the two kinds apart). LRU eviction keeps the cache under the size limit; Edit ▸
|
||
Purge empties it, `cache.diskStats` reports it, and the timeline draws disk-only frames in blue.
|
||
In the browser the frames live in the Origin Private File System: frame workers write them, the
|
||
page keeps the index (`DiskIndex`) and reads hits ([web.md](web.md#disk-cache)).
|
||
|
||
**Deferred readbacks** (`effectcraft-gpu` `deferred`): where a readback can't be waited for (WebGPU
|
||
in a browser worker) a frame renders in passes. `Accelerator::frame_begin` / `pass_begin` /
|
||
`pass_missed` / `miss_gate`: a readback not there yet starts asynchronously and the step returns a
|
||
placeholder; the pass is missed, the layer cache's gate drops inserts until the pass ends, and the
|
||
caller (`remote::FrameServer::resume`) renders again once the device delivered. Readbacks are keyed
|
||
by content (a chain's input pixels and parameters; the frame's comp, time and options), so later
|
||
passes are served from them; a pass without a miss is the frame.
|
||
|
||
**Time effects** (Echo, Posterize Time, Timewarp…) read the layer at other times through
|
||
`EffectHost::self_at`: the renderer renders the layer's source and masks (plus, optionally, the
|
||
effects before it) at that layer time via `Renderer::layer_input`, cached under separate *input*
|
||
keys, with a nesting-depth guard.
|
||
|
||
**Audio** is mixed by `render::audio::mix_comp` in blocks: Audio switch, solo, Audio Levels and
|
||
the Effect > Audio effects (`effects::audio_fx`, applied per layer with a pre-roll so blocks are
|
||
independent). Export and preview playback share it; the desktop app plays it through cpal and the
|
||
audio clock drives preview playback (`ui-egui::audio`). The sound starts once the frames ahead
|
||
are cached; until then, and whenever playback reaches a frame that isn't cached, every frame
|
||
shows as it renders, silently, rather than being skipped. Layer and frame caches ignore the
|
||
switches that don't change pixels (Audio, Lock, Shy, Hide Shy Layers: `Switches::pixels`,
|
||
`Comp::same_pixels`), so toggling them keeps the cached frames.
|
||
|
||
**Layer styles** (Layer ▸ Layer Styles; `crates/render/src/styles.rs`) render in layer space and
|
||
may grow the layer's bounds. Drop Shadow and Outer Glow become separate passes composited below the
|
||
layer with their own blend modes; the interior styles, Stroke and Bevel and Emboss are baked into
|
||
the layer body. Interior styles are clipped to the layer: they blend onto its colour inside its
|
||
shape and keep its alpha, so an anti-aliased edge mixes the style and the background only. Layer
|
||
opacity fades the whole stack; Knockout and the R/G/B channel switches apply at composite time. Styled pixels are cached separately from the layer content, so editing a style
|
||
reuses the cached source/masks/effects. Global Light is one setting per comp, mirrored into every
|
||
layer's Blending Options and kept in step after each edit. Blending Options can be edited in the Layer Style dialog
|
||
before any style is added: the first change creates the Layer Styles group with Blending Options
|
||
only (`layer.style.add` with `blendingOptions`), in the same undo step as the change. Removing
|
||
the last style keeps the group when its Advanced Blending is not at the defaults.
|
||
|
||
**Paint and Puppet** are effects whose instances carry nested groups. Paint (`effects::paint`)
|
||
holds Brush / Clone / Eraser strokes (Path, Stroke Options, Transform, hidden Duration span);
|
||
strokes rasterize as brush-tip dabs and composite in order, and the layer cache key includes which
|
||
strokes are visible at the frame. Puppet (`effects::puppet`) holds meshes and pins: the mesh is
|
||
traced from the input alpha, expanded, and triangulated (Delaunay + constraint recovery, cached
|
||
by content); pins drive an as-rigid-as-possible solve (Igarashi et al. 2005) each frame and the
|
||
input is texture-mapped through the deformed triangles. Pins are selected like properties
|
||
(`puppet.selectPins`, marquee, Select All by kind) and rigged with expressions: Points Follow
|
||
Nulls / Nulls Follow Points take pins as well as paths, and `puppet.follow` gives follower pins a
|
||
Position expression that trails the leader by a delay. The renderer flattens nested effect
|
||
groups into `Params` keys (`effects::flatten_params`). Commands: `paint.*`, `puppet.*`.
|
||
|
||
Half, Third and Quarter resolution render proportionally fewer pixels end to end. Footage
|
||
starts at the render's size where the source can make it more cheaply than the full frame
|
||
(`FootageSource::frame_at_size`): movies in opaque 4:2:0 Y'CbCr are box-filtered to half size
|
||
before conversion (a quarter of the conversions; the result differs from converting first only
|
||
in blocks where colours clip) and resampled from there, as other frames are from full size.
|
||
|
||
**GPU compositor** (`crates/gpu`; Project Settings ▸ Video Rendering and Effects ▸ Mercury GPU
|
||
Acceleration, the default, or Mercury Software Only; `render.backend`). The CPU renderer is the
|
||
reference and keeps rendering layer content (sources, masks, CPU effects, layer styles) into the
|
||
layer cache. The render crate defines an `Accelerator` trait; `effectcraft_gpu::Gpu` implements it
|
||
on wgpu compute shaders. `RenderOpts::backend` picks `Cpu`, `Gpu` or `Auto` (an accelerator
|
||
attached and the project's renderer the GPU). Auto renders each comp's top-level frames on
|
||
whichever compositor measured faster for it (`render::auto::AutoPick`, kept by the accelerator):
|
||
per comp, output scale and path (readback or viewer display) it averages CPU and GPU frame times
|
||
(warm-up on both sides, the fastest warm-up frame starting the average, single stalls capped),
|
||
picks the GPU unless the CPU is clearly faster, and re-measures the other side every 48 frames.
|
||
Render Queue jobs don't use Auto: every frame of a job renders on the project's renderer (the
|
||
GPU compositor with Mercury GPU Acceleration), so runs of a render don't differ by timing.
|
||
Light comps (a few small layers, which the CPU composites only within their bounds) then stay on
|
||
the CPU instead of paying for full-frame passes and a readback. A GPU frame walks the comp like
|
||
`draw_comp`: cached layer buffers are uploaded once per buffer, then transformed with the CPU's
|
||
sampling (nearest, bilinear, Catmull-Rom bicubic, the same minification pre-filter), motion-blur
|
||
sub-samples are accumulated, and track mattes, Preserve Transparency, layer style passes,
|
||
knockout, all 38 blend modes (a WGSL port of `color::blend`), the 8/16 bpc clamp-and-quantise steps
|
||
and colour-space conversions run on the GPU. Classic 3D runs composite on the GPU too
|
||
(`gpu::classic3d`): the render crate prepares the planes (`Renderer::prepare_3d_run`: layer buffers
|
||
with their bokeh depth of field, homographies per motion-blur sub-sample, materials, lights, track
|
||
mattes), the GPU packs the buffers into one atlas texture and one kernel does the per-pixel
|
||
fragment sort (intersecting planes, coplanar stack order), Blinn-Phong lighting, ray-cast shadows
|
||
against the caster planes (Shadow Diffusion, Light Transmission) and blending. Adjustment layers
|
||
run their effect stacks on the GPU-resident comp (`Renderer::run_effects_on` with an `FxTarget`):
|
||
runs of GPU effects stay on the device and only non-GPU effects read back and upload. Advanced 3D
|
||
runs render on the GPU end to end (`gpu::adv3d`, `Renderer::prepare_adv_run` /
|
||
`Accelerator::render_3d`): each motion-blur sub-sample's scene is rasterised (`advanced3d.wgsl`),
|
||
then compute kernels (`adv3d.wgsl`) resolve the 2×2 supersampling, average the sub-samples
|
||
(nearest depth), apply the depth-based iris depth of field with the Classic 3D bokeh spans
|
||
(`three_d::bokeh::kernel_spans`, highlight boost, progressive blur levels) and composite over the
|
||
GPU canvas; only the depth of field reads back its 8-byte radius range. Runs whose layers need
|
||
the 2D compositing path (blend modes, track mattes, Preserve Transparency) come to the GPU as a
|
||
`three_d::adv::SplitRun` (`Renderer::split_adv_run`, `draw_run`'s steps as data): the plain
|
||
layers render as one scene over the canvas, then each special layer, farthest first, renders on
|
||
its own, is hidden where the main scene is nearer (`occlude`), and goes through the 2D matte /
|
||
Preserve Transparency / blend kernels (a 3D track matte renders solo through the camera).
|
||
Environment Light Background layers draw their sky in a kernel (`adv_sky`, from
|
||
`Renderer::sky_draw`) before the rest of the run, in Classic and Advanced 3D comps. Wireframe
|
||
outlines draw on the GPU from the CPU's pixel list (`Renderer::wireframe_pixels`). GPU effects
|
||
(`effects::GPU_EFFECTS`, 280 of them: blurs, colour correction, keying incl. Key Light, mattes,
|
||
channel, stylize, distortion and warps (Warp, Bezier Warp, Smear, Reshape, CC Bend It, CC Page
|
||
Turn, CC Bender, CC Blobbylize), Cartoon, bevels, shapes, the transitions (CC light family, CC
|
||
transitions, Block Dissolve), perspective (Radial Shadow, CC Cylinder / Sphere / Spotlight /
|
||
Environment, 3D Glasses), generators, noise, grain, text (Numbers, Timecode), time, 3D Channel (aux channels as an
|
||
extra texture), Immersive Video, colour management (colour programs from
|
||
`effects::color_program`, LUTs in a storage buffer) and the simulations' render passes (the
|
||
simulation stays on the CPU and hands its plan — sprites, pieces, blobs, grids — to a tiled
|
||
rasteriser; the particle effects too, `gpu::fx_particles`: CC Particle World / Systems II,
|
||
Particle Playground with its Layer Map frames in one atlas, CC Ball Action, CC Pixel Polly, CC
|
||
Scatterize; plans too large for one item table draw in several passes over the previous one),
|
||
and the geometry generators (Lightning, Advanced Lightning, Radio Waves, Stroke, Scribble,
|
||
Vegas, Write-on, Audio Spectrum / Waveform, Basic / Path Text: bolts, strokes, waves, audio
|
||
marks and glyphs are planned on the CPU, shared with the CPU effect, and rasterised in tiles
|
||
with the CPU's coverage functions); see
|
||
[effects.md](effects.md)) repeat the CPU effect's steps (padding, box radii, parameters, hashes)
|
||
as kernels, in one module per family (`gpu::fx_*` with `shaders/fx_*.wgsl`); consecutive GPU
|
||
effects run as one chain with one upload and one readback. Statistics that need the whole frame
|
||
(Auto Levels / Contrast / Color, Equalize, Shadow/Highlight, Color Stabilizer, Remove Grain's
|
||
noise level) are measured on the CPU from one readback and applied on the GPU; effects reading
|
||
other frames (Echo, Posterize Time, Time Difference, Time Displacement, CC Force Motion Blur, CC
|
||
Wide Time, Pixel Motion Blur, Timewarp) upload the frames the host renders (`time_frames`) and
|
||
combine them on the GPU; Time Displacement's per-pixel times and Timewarp's motion vectors
|
||
(`time_fx::timewarp_plan`) are computed on the CPU. Numbers and Timecode rasterise their glyph
|
||
coverage on the CPU (`textfx::text_layer`) and composite it on the GPU. Other layers an effect
|
||
reads (gradients, reveals, environments) are fitted on the CPU (`util::fit_layer`) and uploaded.
|
||
Where f32 cannot follow the CPU's f64, the CPU solves the part that needs it and the GPU uses
|
||
the result: Warp's Fisheye and Twist past 50 % or with a distortion (a Newton inverse that
|
||
wanders chaotically near the crease) sample the CPU's inverse map (`warp_inverse_map`). Settings
|
||
a kernel cannot match still render on the CPU (`effects::catalog::gpu_supported`: Cell
|
||
Pattern's HQ variants, Turbulent Displace's locked pinning and the controls in
|
||
`CPU_ONLY_CONTROLS`). Tests render
|
||
scenes on both paths and compare them (≤ 1/255 at 8 bpc, ≤ 1e-3 at 32 bpc); they skip without an
|
||
adapter. Working textures (premultiplied RGBA f32) come from a pool: a released texture is reused
|
||
once every encoder that could still read it has been submitted, transparent inputs share one
|
||
zero texture per size, readback staging buffers are reused, and the 8/16 bpc quantisation after
|
||
each layer is fused into the layer's composite kernel; under test, pooled scratch images start
|
||
as NaNs (a kernel that skips pixels fails the oracle) and validation errors panic. The desktop viewer builds the
|
||
`Gpu` on egui-wgpu's device and shows frames from GPU textures without reading them back
|
||
(`ui-egui::frames`); headless renders, the CLI (unless `--gpu`) and CI use the CPU. Outside
|
||
tests, device errors are logged rather than fatal. A viewer frame that runs out of video memory
|
||
(`Gpu::within_memory`) renders on the CPU, the uploaded layers and pooled textures are freed, and
|
||
the RAM preview keeps half as many GPU frames from then on. The same guard catches a frame's
|
||
validation errors (an invalid texture, often left by a failed allocation): that frame renders on
|
||
the CPU and the kept textures are freed, but the compositor stays on and, with no allocation
|
||
failure before it, the RAM preview keeps its GPU frames (#1319). A frame whose render panics is
|
||
rendered again on the CPU or released, never left in progress. Native readbacks and timing waits
|
||
share a finite deadline; failure retires pending readbacks exactly once, and late results are
|
||
discarded. The desktop executable and browser page own their shared device's error/loss handlers and forward
|
||
failures to the compositor through a weak notifier. Libraries borrowing a device preserve its
|
||
host's handlers. A host fault disables shared-device preview acceleration without editing the project or its
|
||
renderer preference; materialized CPU frames remain usable and late GPU textures cannot return
|
||
to the preview cache. Scoped allocation failures retain the adaptive budget/CPU retry behavior.
|
||
|
||
**Recovery boundary:** the host installs handlers through eframe's CreationContext, after egui's
|
||
presentation pipelines but before EffectCraft's pipelines. This does not cover that earlier egui
|
||
startup window. eframe 0.37 supports lost-surface recreation, not replacing its live device and
|
||
all presentation resources. Device loss therefore requires restarting the presentation backend;
|
||
CPU compositing alone cannot restore that display. No automatic restart or persistent safe-mode
|
||
sentinel is implemented here. Browser scope futures are not synchronously awaited; the browser page
|
||
forwards shared-device errors, while owned worker devices install their own handlers. On the web
|
||
(WebGPU) the GPU composites viewer frames; steps that need a readback fall back to the CPU (the
|
||
Info panel's pixel readout reads GPU frames back asynchronously). **GPU particles**
|
||
(`effects::psim`, `gpu::particles`): CC Particle World, CC Particle Systems II and Particle
|
||
Playground cannons without interacting forces evolve every particle independently, so the GPU
|
||
backend (reached through `EffectHost::particles` when the GPU compositor is active) simulates one
|
||
particle per invocation from the CPU's birth schedule and keeps per-key state checkpoints on the
|
||
GPU, like `SimCache`; the CPU simulation stays the oracle (tests compare ids and positions).
|
||
CC Particle World, CC Particle Systems II and CC Mr. Mercury sample their Birth Rate at every
|
||
step (`EffectHost::param_at`, #1373), so a keyed rate only decides the births of its own time;
|
||
the rate is left out of the simulation key, and a checkpoint is resumed only with the same rate history (the CPU state's `hist`, the GPU
|
||
checkpoint's births). The rate's keyframes are read at the step's exact layer time
|
||
(`EvalCtx::value_at_layer_time`; mapping it to comp time and back rounds under a fractional
|
||
stretch), a rate without keyframes or an expression isn't sampled, and a layer of one of them with
|
||
expressions isn't layer cached (its key can't see what an expression returned at earlier times).
|
||
`effectcraft-cli bench --gpu` reports CPU vs GPU ms/frame, speed-up, Auto's ms/frame, pixel
|
||
agreement and the steady-state upload / readback MB per frame for every comp plus an
|
||
adjustment-layer comp.
|
||
|
||
**Colour and bit depth** (`crates/render/src/color.rs`, `crates/color/src/space.rs`). Pixels are
|
||
`f32`, but 8 and 16 bpc projects clamp and quantise each layer after its source and masks and
|
||
after every effect, and the comp after every layer (16 bpc uses 0..32768), so over-range values
|
||
(Add, Screen, Exposure…) only survive in 32 bpc; blend modes without HDR support clamp their
|
||
inputs in 32 bpc. With a working space (sRGB, Rec. 709, Rec. 2020, Display P3; matrices derived
|
||
from the published primaries), footage is converted from its colour profile (stream metadata or
|
||
Interpret Footage, else sRGB) and the top-level comp is converted to the sRGB display. Linearize
|
||
Working Space runs everything in linear light; Blend Colors Using 1.0 Gamma linearises only for
|
||
blending. Layer cache keys include these settings. Project Settings ▸ HDR (compand / tone
|
||
mapping) is the last top-level step, for standard-dynamic-range displays and outputs; OpenEXR
|
||
renders (sequences and ProEXR) skip it (`RenderOpts::skip_hdr_step`) and decode the frame with the
|
||
output space's own curve (`Pipe::output_linear`), so they hold linear light in the output space's
|
||
primaries; unmanaged projects decode with the Working Gamma's curve (sRGB at 2.2, a pure 2.4 power
|
||
curve at 2.4). Every OpenEXR file (sequences and ProEXR) names those primaries and the white point in its
|
||
`chromaticities` attribute (`ColorSpace::exr_chromaticities`; Rec. 709 / sRGB primaries for the default
|
||
and for unmanaged projects, Rec. 2100 PQ and HLG as BT.2020, the ACES spaces with the ACES white). Unmanaged projects and Rec. 2100 PQ / HLG outputs never get the HDR step.
|
||
|
||
**Frame blending**: footage and precomp layers whose source time falls between source frames
|
||
(rate conform, stretch, remap) blend the two neighbouring frames, by cross-fade (Frame Mix) or by
|
||
hierarchical block-matching optical flow and a bidirectional warp (Pixel Motion,
|
||
`raster::flow`).
|
||
|
||
**Collapse Transformations**: a collapsed precomp layer without masks, effects or styles draws
|
||
its nested layers straight into the parent with concatenated transforms (one resample), so
|
||
nested blend modes and adjustment layers act on the parent's layers; nested 3D layers use the
|
||
parent's camera and lights and, when the precomp layer is 3D, are depth-sorted with the parent's
|
||
3D run. Essential Properties overrides (properties, Font and Media Replacement) travel with the
|
||
collapsed context and are applied while the nested layers are evaluated, so they do not force a
|
||
flattened render or clip at the precomp's frame. Masks, effects or styles force a flattened
|
||
render. Text layers are always rasterised at
|
||
their on-screen scale, as text is vector in After Effects; on shape layers and vector footage the
|
||
switch is Continuously Rasterize and does the same. Quality: Draft samples
|
||
nearest-neighbour, Wireframe draws the layer bounds. Slip edit (`layer.slip`, Alt+PageUp/Down,
|
||
dragging the source bar in the timeline) moves the source under fixed in/out points.
|
||
Dragging a layer bar's in or out edge with several layers selected trims every selected layer by
|
||
the same amount from its own in/out point (`layer.timing {inDelta | outDelta}`), clamped to the
|
||
comp and to one frame of layer, as one undo step.
|
||
|
||
**Nested comps** show nothing before they start or after they end (`Comp::covers`), even when
|
||
the precomp layer is trimmed or remapped past them. A comp with **Preserve frame rate when nested
|
||
or in render queue** shows only its own frames when nested (`EvalCtx::nested_time` floors the
|
||
precomp layer's source time to its frame rate) and renders at its own rate in the Render Queue;
|
||
one with **Preserve resolution when nested** renders at full size inside a comp drawn at a lower
|
||
resolution (the buffer's scale places it).
|
||
|
||
**Motion blur** has one gate, `Renderer::mb_on`: the render options (Render Settings ▸ Motion
|
||
Blur, previews), the comp's switch and the layer's switch, limited by the precomp layers above
|
||
when switches affect nested comps. Layer motion, mask motion blur (the comp's Samples Per Frame,
|
||
four in Draft), effects that read the shutter and Advanced 3D all follow it, and the layer cache
|
||
key of a layer with masks or effects records it.
|
||
|
||
**Motion tracking** (Animation ▸ Track Motion / Stabilize Motion, Window ▸ Tracker): a tracker
|
||
is a `Tracker` group under the layer's Motion Trackers group, with Track Point groups (Feature
|
||
Center, Feature Size, Search Offset, Search Size, Confidence, Attach Point, Attach Point Offset)
|
||
keyframed per analysed frame. `track.analyze` renders the layer's source frames
|
||
(`Renderer::layer_source`) on a background thread (`engine::tracking`, polled like the render
|
||
queue; blocking with `wait`), and `effectcraft-track` matches each point in parallel. One undo step
|
||
covers an analysis. `track.apply` keys the target's Position/Rotation/Scale, the tracked layer's
|
||
Anchor Point and Position (Stabilize), or a Corner Pin effect (Parallel / Perspective).
|
||
|
||
**Mask tracking** (`track.mask`, `engine::mask_track`) runs the same kind of background job over
|
||
the layer's source frames: `effectcraft-track`'s mask tracker detects features inside the mask,
|
||
tracks them with pyramidal Lucas–Kanade and fits the chosen model (position … perspective) with
|
||
RANSAC; the motion moves the Mask Path's vertices and tangents and is keyed per frame. **Mask
|
||
Interpolation** (`mask.interpolate`) uses `effectcraft-path`'s smart interpolation (arc-length
|
||
vertex insertion, shape-context matching, rigid in-betweens) to key in-between shapes.
|
||
The two **Face Tracking** methods (`effectcraft_track::face`) fit a face inside the mask instead:
|
||
a skin colour model from the first frame, the face outline along rays from the skin component's
|
||
centre (an area-moment ellipse gives centre, size and roll), facial features as the non-skin
|
||
components inside it, and an active-shape point distribution model trained on synthetic face
|
||
shapes (our own generator, no external weights) that fills and checks the landmarks. The outline
|
||
keys the Mask Path; Detailed Features keys a Face Track Points effect, and
|
||
`track.extractFaceMeasurements` derives a keyed Face Measurements effect (and copies its keys).
|
||
With a trained face model chosen (`effectcraft_segment::face::FaceModel`, Settings ▸ Face
|
||
Tracking; `Session::models` hands it to the mask track), `FaceTracker::new_with` lets the model
|
||
find the face in the mask and follow it; the model's outline and named points replace the
|
||
classical steps, and chin and jaw still come from the outline so the measurements agree.
|
||
|
||
**Warp Stabilizer** (`effects::warp_stab`, `engine::warp`): `warp.analyze` renders the layer's
|
||
input to the effect (`Renderer::layer_input`: source, masks and the effects above it) for every
|
||
frame on a background thread and stores `effectcraft-track`'s `WarpAnalysis` (per-frame
|
||
translation / similarity / homography fits) as JSON in the effect's hidden Analysis parameter,
|
||
with a key of the layer's source, In/Out, start, stretch and Time Remap. `Session::edit` clears
|
||
analyses whose key no longer matches and queues them; the desktop app re-analyses them in the
|
||
background. Rendering derives the stabilization plan (smoothed camera path or No Motion, framing,
|
||
auto-scale; cached per analysis and settings) and warps each frame. Subspace Warp also stores long
|
||
feature trajectories in the analysis, smooths them in a low-rank subspace (Liu et al. 2011) and
|
||
warps each frame with a content-preserving mesh (Liu et al. 2009) fitted to the smoothed
|
||
positions, rendered as a per-pixel mesh lookup; Synthesize Edges fills the
|
||
borders from neighbouring frames read with `EffectHost::self_at`. The effect lives in the
|
||
effects crate, hence the `effects → track` edge.
|
||
|
||
**Essential Graphics** (`project::essential`, `engine::commands::essential`): a comp's
|
||
`essential` field lists its exposed controls (properties, Media Replacement, groups, comments).
|
||
Every precomp layer of such a comp gets an Essential Properties group (`GroupKind::Essential`,
|
||
children `eg<control id>`), kept in step by `Session::edit`; editing a child overrides it for
|
||
that instance. A collapsed instance carries its property, Font and Media Replacement overrides
|
||
into the nested render context (`Collapsed`, `EvalCtx::{property_overrides, media_overrides}`);
|
||
other instances are drawn from a copy of the project with the overridden source properties set
|
||
(`essential::with_overrides`). Templates are `.ectemplate`
|
||
ZIPs: `manifest.json` (controls), `project.ecproj` (the comp and its dependencies), `media/…` and
|
||
`poster.png`; importing offsets every id past the project's (`essential::offset_ids`).
|
||
**Responsive Design — Time**: a time-stretched precomp maps time piecewise so its protected
|
||
marker regions play at 100 % (`eval::responsive_source_time`).
|
||
|
||
**Proxies and interpretation**: an item's `proxy` (footage or comp) is decoded in its place at
|
||
the source's nominal size when `RenderOpts::proxy` (Render Settings ▸ Proxy Use) allows it.
|
||
Interpret Footage's Separate Fields turns each field into a frame at twice the rate (the other
|
||
lines interpolated); pixel aspect stretches footage, solids and precomps in the comp
|
||
(`EvalCtx::par_ratio`, not inherited by children); Invert Alpha and Interpret As Linear Light
|
||
apply after decoding.
|
||
|
||
**3D Camera Tracker** (`effects::camera_tracker`, `engine::camera_track`, `effectcraft-track`'s
|
||
`camtrack`): `camera.analyze` / `track.camera` render the layer's input to the effect on a
|
||
background thread and run structure from motion in two steps. Step 1 follows Shi–Tomasi features
|
||
through the clip with pyramidal Lucas–Kanade (forward–backward checked, re-detected where the frame
|
||
has none) into long 2D tracks. Step 2 solves the camera: parallax keyframes; a two-view start from
|
||
the essential matrix (normalised 8-point in RANSAC) or a planar homography decomposition; incremental
|
||
resection and triangulation; a sparse Levenberg–Marquardt bundle adjustment (Schur complement over
|
||
the points, Huber loss) with the focal length fixed, shared, or per frame; a log-spaced and
|
||
golden-section focal search for Fixed Angle of View / Variable Zoom; a rotation-only model for tripod
|
||
pans, chosen by Auto Detect when it explains the tracks as well; with Solve Lens Distortion (or
|
||
Detailed Analysis) radial distortion `k1, k2` is adjusted jointly in a final bundle adjustment on
|
||
the raw tracks and the solve repeated on tracks undistorted with that estimate; Undistort Footage
|
||
renders the input through the inverse model. Tracks and solve are stored as JSON
|
||
in hidden effect parameters with keys: changing the layer's frames clears both, changing Shot Type,
|
||
Angle of View, Solve Method or deleting points only re-solves. The solve's canonical frame maps to
|
||
comp space so the first frame's camera is the default comp camera (or so a chosen ground plane is
|
||
the X-Z plane at the origin); `camera.createFromSolve` keys a one-node "3D Tracker Camera" on every
|
||
frame and places text, solids, nulls or a shadow catcher and light on the target plane.
|
||
|
||
**Roto Brush & Refine Edge** (`effectcraft_track::roto`, `effects::roto`, `engine::roto`):
|
||
strokes (foreground, background, Refine Edge) are stored as JSON in the effect's hidden Strokes
|
||
parameter with the base frame and segmentation span. Each frame is segmented by graph cut
|
||
(Boykov–Jolly hard constraints, GrabCut colour mixtures, our own Boykov–Kolmogorov max-flow,
|
||
coarse to fine) and propagated to the next frame by warping the matte with block optical flow and
|
||
re-cutting in a Search Radius band; frames with correction strokes are re-cut with the warped
|
||
matte as a soft prior. With Version 2.0 / 3.0 and a trained model chosen (`effectcraft_segment`,
|
||
Settings ▸ Roto Brush), the model's foreground probability, prompted by the strokes or by the
|
||
warped matte, becomes a strong prior for the same cut (falling back to the classic result when it
|
||
disagrees with the flow); the model's id is part of the chain seed. Refine Edge bands get guided-filter + closed-form matting and
|
||
foreground-colour decontamination. Segmentations are derived data: each frame's chain key (Input
|
||
Key, settings, strokes from the base frame out) indexes a process-wide cache that renders fill on
|
||
demand (`self_at`) and `roto.propagate` fills in the background; `Session::edit` keeps the Input
|
||
Key in step with the layer's source, masks and upstream effects. Freeze stores final 8-bit mattes
|
||
in the effect.
|
||
|
||
**Content-Aware Fill** (`raster::inpaint`, `engine::commands::content_fill`): the hole is
|
||
where the layer's masked source (`Renderer::layer_input` with no effects) is under half
|
||
opacity, grown by Alpha Expansion. Optical flow between neighbouring membrane-filled frames is
|
||
*completed* inside the holes (membrane interpolation of the surrounding tiles' motion); each
|
||
missing pixel follows it forwards and backwards to the nearest frame (or reference frame) where
|
||
it is visible, the two candidates blended by temporal distance, with Lighting Correction adding
|
||
the membrane of the border mismatch (Poisson-style). What no frame shows is synthesised by
|
||
PatchMatch (Barnes et al. 2009) inside Wexler et al.'s (2007) multi-scale EM: Object refines each
|
||
frame starting from the previous result warped along the flow; Surface carries one synthesised
|
||
fill through the range; Edge Blend is the membrane fill alone. The frames are written as a PNG
|
||
sequence and imported into a "Fill" layer above the source. It runs as a generic **background
|
||
task** (`engine::jobs`: a closure with progress and cancel on a thread, inline on wasm32 or
|
||
with `wait`, whose result is applied on the UI thread as one undo step); the Progress panel and
|
||
`jobs.list` / `jobs.cancel` show these tasks together with the render queue and the analyses.
|
||
|
||
**Scene Edit Detection** (`raster::cuts`) compares consecutive frames by colour-histogram
|
||
distance and motion-compensated (block-flow warped) luma error, takes their geometric mean and
|
||
marks frames that exceed a threshold and several times their neighbours' median; the cuts
|
||
become layer markers, split layers, or split-and-precomposed layers. **Auto-trace**
|
||
(`path::trace`) runs marching squares on the chosen channel, Douglas–Peucker simplification and
|
||
cubic fitting with Corner Roundness blending chord and smooth tangents; holes become Subtract
|
||
masks, and a work-area trace keys Mask Path per frame.
|
||
|
||
**Align Video to Data** reads data footage (File ▸ Import of `.json`, `.csv`, `.tsv`, kept as
|
||
text in the project). Supported data: JSON as an array of objects or an object holding one
|
||
(`{"samples": [...]}`), CSV/TSV with a header row (quoted fields allowed). The time key is the
|
||
`key` parameter or the first field named `time`, `timestamp`, `t`, `date`, `datetime`, `utc`,
|
||
`gps_time`, `seconds` or `time_s`; values may be ISO 8601 date-times (with `Z` or `±hh:mm`),
|
||
times of day (`hh:mm:ss.sss`), timecode (`hh:mm:ss:ff` at the footage rate), Unix seconds or
|
||
Unix milliseconds. The layer's Start Time becomes `dataStart + videoStart − firstSample`
|
||
(`videoStart` given, or the file's creation date on the desktop; reduced to a time of day when
|
||
the data has no dates).
|
||
|
||
**Lumetri Scopes** (`raster::scopes`) are density plots of the viewer frame (rendered over the
|
||
comp background) with BT.601 / BT.709 / BT.2020 luma and colour-difference coefficients.
|
||
|
||
## 5. Expressions
|
||
|
||
Expressions are JavaScript, run by boa, with After Effects' object model (`thisComp`, `thisLayer`,
|
||
`time`, `value`, `wiggle`, `loopOut`, vector maths on arrays, and so on). A syntax error keeps the
|
||
text, disables the expression and shows a warning, like After Effects. `sampleImage` renders the
|
||
sampled layer through the evaluating renderer's footage source (`EvalCtx::footage`), cached per
|
||
thread and frame; `footage(name)` reads data footage (JSON, CSV, TSV imported with File ▸ Import,
|
||
text kept in the project) through `sourceData`, `sourceText` and `dataValue`. `expr.errors`
|
||
lists failing expressions on demand. The viewer's error bar uses the same checks against an
|
||
immutable project snapshot, after 150 ms without editing, pointer interaction or playback.
|
||
Native builds run these automatic checks on one persistent worker; layout and paint only read
|
||
completed results. Requests are coalesced per comp, cancelled between properties when obsolete,
|
||
and bounded to eight comp entries. Results must match the current project, revision and time.
|
||
Disabling Show Expression Error Banner cancels automatic checks. Non-solo layers still receive
|
||
diagnostics, including disabled expressions with syntax errors. The single-threaded web build
|
||
shares the cache and debounce but evaluates one scan on the UI thread after interaction stops.
|
||
Each thread keeps one boa context; boa
|
||
leaves values on its stack whenever an exception unwinds through call frames (every pending
|
||
host request does), so a run that hits the stack limit gets a fresh context and is retried once.
|
||
|
||
Snapping memoizes expression values while collecting one immutable set of targets. Bounds,
|
||
vertices and transforms share those values; the cache is discarded before the next query, so
|
||
time changes and edits always use current geometry. Native timeline disk-cache indicators also
|
||
request missing content hashes in the background, coalescing edits per comp. Their blue bars
|
||
appear after the key is ready instead of serializing composition content during painting.
|
||
|
||
## 5a. Scripting
|
||
|
||
`effectcraft-script` runs JavaScript (boa, a fresh context per run; the Script Console keeps one)
|
||
with an object model written from the behaviour the After Effects Scripting Guide documents:
|
||
`app`, `Project`, `ItemCollection`, `CompItem`/`FootageItem`/`FolderItem`, `LayerCollection`,
|
||
`AVLayer`/`TextLayer`/`ShapeLayer`/`CameraLayer`/`LightLayer`, `PropertyGroup`/`Property`
|
||
(values, keyframes, eases, expressions), `TextDocument`, `Shape`, `KeyframeEase`, `MarkerValue`,
|
||
`RenderQueue`/`RenderQueueItem`/`OutputModule`, `File`/`Folder` and the enums. The model lives in
|
||
`prelude.js`; it reads the session through `__query` (JSON snapshots of items, layers, property
|
||
nodes, values and keys) and edits only through `__exec`, which runs engine commands, so every
|
||
scripted edit is undoable and identical to the UI's. While a script runs the session is moved
|
||
into a thread-local the natives borrow (nothing borrowed enters the JS heap); the comp a call
|
||
targets is made active for that command only. `app.beginUndoGroup`/`endUndoGroup` fold the undo
|
||
steps in between into one named step. Match names (`ADBE Transform Group`, `ADBE Position`,
|
||
`ADBE Gaussian Blur 2`, `ADBE Vector Shape - Rect`…) come from a table in `matchnames.rs`; effect
|
||
parameters are `<effect match name>-0001…`. `File` reads are limited to the project's folder, and
|
||
writes and the network are refused, unless Preferences ▸ Scripting & Expressions ▸ Allow Scripts
|
||
to Write Files and Access Network is on. Entry points: the `script.run` command
|
||
(`Session::script`, set by the host), File ▸ Scripts ▸ Run Script File… (`.jsx`/`.js`; `.json`
|
||
command scripts still run as steps), Window ▸ Script Console, `effectcraft-cli script` and the
|
||
MCP `run_script` tool.
|
||
|
||
## 6. Commands, the UI seam and automation
|
||
|
||
Every action is a command: an id, a label, a menu path, a shortcut, a parameter description,
|
||
`enabled()` and `run()`. The menu bar is a tree of command ids. The desktop UI, the command line,
|
||
the JSON control channel and the MCP server all dispatch by id, so anything a person can do from a
|
||
menu, an agent can do too. Every interactive widget registers an automation id, so agents can also
|
||
inspect and click the interface. See [agents.md](agents.md) and
|
||
[control-protocol.md](control-protocol.md).
|
||
|
||
Settings (`engine::prefs`), keyboard shortcut presets (`engine::shortcuts`) and auto-save /
|
||
crash recovery (`engine::autosave`) live in the engine too; they persist through a
|
||
`ConfigStore` the frontend provides (a directory on the desktop; the Origin Private File System,
|
||
or IndexedDB, on the web). Auto-saves go through the store's `FileOps` (the file system by
|
||
default, the browser storage on the web).
|
||
|
||
Renders and analyses run on background threads on the desktop. Where there are no threads (the
|
||
browser), `engine::offload` sends them to a second engine instance instead: the session's
|
||
`Offload` ships the serialized project, the footage it reads and the job (`WorkerRequest`) to a
|
||
Web Worker, and the worker's replies (progress, item status, the analysed property group) are
|
||
applied on the UI thread.
|
||
The shortcut dispatcher and the menus read the active preset. See
|
||
[preferences.md](preferences.md).
|
||
|
||
## 6a. Premiere Pro interop (timeline interchange)
|
||
|
||
After Effects' File ▸ Import / Export ▸ Adobe Premiere Pro Project… go through the public
|
||
interchange formats Premiere Pro itself reads and writes, not the native `.prproj` file (it has no
|
||
public specification; supporting it awaits a decision). `crates/interchange` (L3, no file I/O)
|
||
maps FilmCraft's timeline model (`filmcraft-project`, parsed and written by `filmcraft-interchange`,
|
||
pinned at the same git revision as the other `filmcraft-*` crates; pure Rust, builds for wasm32)
|
||
to compositions and back:
|
||
|
||
- **Import** (`file.importTimeline {path, format?, edlFrameRate?}`): Final Cut Pro 7 XML (`xmeml`,
|
||
Premiere's File ▸ Export ▸ Final Cut Pro XML), FCPXML 1.9–1.11, OpenTimelineIO, CMX 3600 EDL, AAF and OMF.
|
||
Sequences become comps (size, rate, duration, pixel aspect, start timecode, markers); video
|
||
tracks become layers stacked by track (top track = top layer, later clips of a track above
|
||
earlier ones); clips become footage layers with their in/out/start, speed and reverse (time
|
||
stretch) and frame holds (Time Remap); Motion (position, scale / scale width, rotation, anchor
|
||
point, Scale to Frame Size) and Opacity (blend mode) with keyframes, keyframe times converted
|
||
from clip time to layer time; cross dissolves and dips become overlapping layers with opacity
|
||
keyframes; nested sequences become precomps; audio tracks become audio-only layers with Audio
|
||
Levels (clip volume + gain + track volume, crossfades as level keyframes); colour mattes and
|
||
adjustment layers become solids; bins become Project panel folders under a folder named after
|
||
the document; media that cannot be found become placeholders with the document's metadata
|
||
(`missing` in the result). Effects, effect masks, titles/graphics and speed ramps are reported
|
||
in `warnings`.
|
||
- **Export** (`file.exportTimeline {comp?, path, format?, prerender?, precomps?}`): one sequence
|
||
per comp, one track per layer (bottom layer = V1), footage clips with their timing, Motion,
|
||
Opacity and audio levels (linked audio clips for layers with sound), precomps as nested
|
||
sequences, full-frame solids as colour mattes. Layers a timeline cannot represent (text, shapes,
|
||
3D, effects, masks, track mattes, parenting, expressions, animated time remapping, non-uniform
|
||
scale, blend modes Premiere lacks) are pre-rendered to ProRes 4444 with alpha next to the
|
||
document (`prerender: unsupported`, the default; `all` renders every visual layer; `none` leaves
|
||
them out with a warning) and referenced as media. The menu keeps After Effects' label "Adobe
|
||
Premiere Pro Project…", but the file written is Final Cut Pro XML (`.xml`), which Premiere Pro
|
||
imports with File ▸ Import; the dialog says so. FCPXML, OTIO, EDL, AAF and OMF are the other formats.
|
||
|
||
AAF and OMF (FilmCraft's structured-storage and Bento readers/writers) import and export too;
|
||
media embedded in them is not extracted, and the AAF writer keeps neither speed nor nesting.
|
||
|
||
## 7. Performance of everyday operations
|
||
|
||
`effectcraft-cli bench --ops [--small] [--layers N] [--comps N] [--footage N]` measures the
|
||
operations people do all day on a large generated project (`engine::perf::large_project`: 200
|
||
comps, 5,000 layers in the main comp plus 10 in each other comp, 300 footage items — 100 image
|
||
sequences of 240 frames, 100 movies, 100 stills — a 20-deep precomp chain, expressions on every
|
||
4th layer, Gaussian Blur on every 6th, parenting). The engine side is `engine::perf::ops_bench`;
|
||
the UI side (`ui-egui::bench`) drives the real `EffectcraftApp` headless: layout, painting and
|
||
tessellation, no window or GPU upload.
|
||
|
||
What keeps it fast:
|
||
|
||
- **Lazy open.** A project file carries its footage metadata, so File ▸ Open is read + parse +
|
||
swap. Nothing is decoded until a frame needs it, expressions compile on first evaluation
|
||
(cached by text), and system fonts are scanned on first need (a text layer asking for a family
|
||
that isn't bundled, or a font menu or `text.fonts` listing the families); the desktop app starts
|
||
that scan on a background thread at launch, reading name tables only. The footage files are checked afterwards in the background (`footage.check`, a
|
||
"Checking footage" job in the Progress panel): missing items are flagged, items saved without
|
||
metadata are probed, and the project is not marked modified. The desktop app does this after
|
||
every open (`Session::check_footage_on_open`); headless sessions run `footage.check {wait:true}`.
|
||
- **No deep copies per frame.** Panels hold `Arc<Comp>` snapshots (`Session::active_comp_arc`,
|
||
`Project::comp_arc`) instead of cloning a comp's layers every frame.
|
||
- **Virtualised lists.** The Timeline and the Project panel draw (and register automation ids
|
||
for) only the rows in view; the Project panel scrolls vertically and reveals an item selected
|
||
elsewhere. Menus listing every layer (track matte, parent) are built only while open.
|
||
- **Off the UI thread.** Project panel thumbnails render on a background thread (the previous
|
||
thumbnail stays up meanwhile); auto-save snapshots the project (an `Arc` clone) and serialises
|
||
and writes compact JSON on a background thread (`AutoSaveState::background`, on in the desktop
|
||
app; inline on wasm32).
|
||
- **Undo is a pointer swap.** History stores `Arc<Project>` snapshots and comps are `Arc`s, so
|
||
undo and redo of any edit are constant time. An edit copies the comps it changes (a one-layer
|
||
edit in a 5,000-layer comp copies that comp: ~8 ms). An edit that changes nothing (a value
|
||
set to what it already was) records no step: `Project::same_content` compares the comps the
|
||
edit left shared by pointer, so the check costs about as much as the copy.
|
||
- **Modified follows undo.** The session keeps the snapshot it last opened or saved
|
||
(`Session::saved_project`); the project is unmodified whenever the current snapshot is that
|
||
one, so undoing (or jumping in the History panel, or rolling back a batch) to the saved state
|
||
clears the modified mark. The window title shows the project and `*` while it is modified, and
|
||
closing a modified project (Quit, the window's close button, New Project, Open, Open Recent,
|
||
Close Project, Revert, the demo project, a Home template) asks Save / Don't Save / Cancel first
|
||
(`ui-egui/src/panels/unsaved.rs`). Agents' `engine.execute` calls are never asked.
|
||
|
||
Numbers (Apple Silicon, 14 cores, release build, best of 3–20 runs; the machine was shared with
|
||
other builds, load average 150–220, so treat them as orders of magnitude). "Before" is the
|
||
commit before M13.14 with the same benchmark code:
|
||
|
||
| Operation | Before | After |
|
||
|---|---|---|
|
||
| Startup to first full UI frame (open + app + 2 frames) | 2,119 ms | 432 ms (1,482 ms at load 220) |
|
||
| File ▸ Open (125 MB pretty JSON; parse is ~95% of it) | 1,049 ms | 269–733 ms |
|
||
| Save (pretty JSON, 125 MB) | 845 ms | 275–359 ms |
|
||
| Auto-save, UI-thread cost | 530 ms (pretty, synchronous) | 0.1–0.2 ms (compact, background; 69–172 ms off-thread) |
|
||
| Steady frame, Standard workspace | 862 ms | 0.5–6.9 ms |
|
||
| Timeline frame scrolling 5,000 layers | 1,725 ms | 1.5–8.1 ms |
|
||
| Timeline frame, 500 layers twirled open | 1,508 ms | 1.8–5.5 ms |
|
||
| Project panel frame, 503 items, all folders open | 1.2 ms (first screenful only, no scrolling) | 0.4–2.8 ms (scrolling) |
|
||
| Small edit (one property) / undo + redo | 14.7 ms / 0.01 ms | 8.4–19.7 ms / 0.01 ms |
|
||
| Large edit (switch on 5,000 layers) / undo / redo | 62 / 0.03 / 0.01 ms | 18–57 / 0.02 / 0.01 ms |
|
||
| First rendered frame of the 5,000-layer comp at ¼ (viewer thread) | 5.7 s | 2.7–3.6 s (unchanged code; load) |
|
||
|
||
The before frame times were dominated by an O(layers²) menu build per visible timeline row and
|
||
by every panel deep-cloning the active comp each frame. Left for later: per-layer structural
|
||
sharing (so a one-layer edit doesn't copy its comp), a binary or compact project format (pretty
|
||
JSON parsing is now the bulk of open), and rendering 5,000-layer comps faster.
|
||
|
||
## Revision history
|
||
|
||
| Date | Change | Summary |
|
||
|---|---|---|
|
||
| 2026-10-10 | minor | Status line added; crate table checked against `crates/` (every crate listed); effect count and VP9 encoder description corrected |
|
||
|
||
PDF document structure uses the `craft-pdf` read feature through an app-owned bounded object adapter. EffectCraft retains its drawing tree, fonts, content/image filters and EPS interpreter; ordinary object/xref limits remain 256 MiB on 64-bit and 64 MiB on 32-bit/WASM, including repair.
|