photocraft/docs/scratch-disks.md

Scratch disks and memory residency

Native PhotoCraft keeps pixel tiles in RAM and spills cold tiles to a configured scratch directory. A read or write guard keeps a tile resident while a consumer uses its bytes. Undo snapshots keep their copy-on-write tiles, including disk-backed tiles, until History States retires them. Web builds retain their explicit memory limit (capped like Memory Usage) and do not use a filesystem.

New preferences use 70% of physical RAM as the process ceiling. This is configurable from 1–95%; older preferences without performance.memoryUsagePercent retain their absolute performance.memoryUsageMb limit. Other applications, non-tile process memory and bounded working buffers reduce the live tile allowance. The separate performance.gpuMemoryMb setting limits graphics resources. Memory virtual address-space usage is diagnostic data, not a measurement of committed memory or a substitute for scratch storage.

Enable a directory in Preferences → Scratch Disks. (system temp) selects the operating system's temporary directory. Only the first enabled, nonempty directory is used. Changing directories creates a new scratch file; old disk-backed tiles keep their old file alive. The development override PHOTOCRAFT_TILE_BUDGET_MB fixes the tile allowance for stress tests.

Scratch extents are LZ4 compressed, CRC-checked and reused after a tile is released. Closing the last tile truncates its file. No scratch file outlives the process, including a crash or an exit without Rust destructors: Windows opens it with delete-on-close, and Unix creates it with mode 0600 and unlinks it immediately after opening, reading and writing through the one open handle with positioned I/O (pread/pwrite). The space is returned when the last handle closes. Configuring a directory also sweeps photocraft-<pid>-<nonce>.scratch files of dead processes (older builds, or a crash between creating and unlinking). It preserves live-process files, other users' files, symlinks and unrelated names; unavailable process enumeration leaves files untouched. PID reuse conservatively retains a stale file, and a unique launch nonce avoids a collision. A filesystem that refuses the unlink keeps the name until the file is dropped or the next sweep.

The engine owns a shared, stoppable residency supervisor and joins it when its last session is dropped. photocraft-raster only exposes synchronous measurement and bounded eviction operations; standalone callers work without a background thread. sysinfo is native-only with its default features disabled and only the system (RAM) feature enabled, plus disk on macOS only, for purgeable space.

An unavailable or full scratch disk never loses pixels or turns spilling off for the session. A failed scratch write or file creation, or a volume about to run low (free space is checked when configuring, every 64 MB of scratch growth and on each retry; spilling stops while less than max(1 GiB, 5% of the volume) would stay free), pauses spilling: tiles stay in memory, the status bar shows the reason, and History's Memory Usage byte limit applies again from the next command, as without a scratch disk. The pause is retried after 1 s, doubling up to 60 s; spilling resumes on its own once the directory is usable and has room, and changing preferences retries at once. Preferences → Scratch Disks → Scratch Size Limit (scratchDisks.sizeLimitMb, 0 = no limit; spill::set_size_cap) caps the scratch file: reaching it pauses spilling the same way, and the free-space reserve applies regardless. Free space is asked of the scratch folder itself (GetDiskFreeSpaceEx on Windows, statvfs on Unix), so a network folder (UNC path, mapped drive, NFS or CIFS mount) reports its own volume's numbers, for both the large-import allowance and the reserve. On macOS a volume sysinfo lists also counts its purgeable space. A mount that doesn't report its space (the query fails, or the total is zero, as with some FUSE and WebDAV mounts) counts as unknown: the free-space reserve doesn't apply, imports earn a scratch allowance only up to the Scratch Size Limit, and Preferences → Scratch Disks and the hard-limit refusal say so, with the hint to set a limit or use a local disk; spill::stats exposes it as freeSpaceUnmeasured. With neither free space nor a size limit known, imports get no scratch allowance. Opening and measuring the folder happen off every lock, on a short-lived thread (at most one per folder and question, 16 overall), and callers wait up to 2 s for the answer. A query that doesn't answer in time counts as unknown (warned as above) and, when the scratch file couldn't be opened, pauses spilling with "not responding" and retries later. A system call can't be cancelled, so a stuck query keeps its thread until the mount answers and no second one starts; a late answer applies if that folder is still the current one and is discarded otherwise. The free-space check every 64 MB of growth never waits, and with an engine session tile allocation leaves retries to its supervisor. The wait is on the calling thread: changing Scratch Disks preferences or checking room for an import may wait up to those 2 s once for a newly hung folder. A whole folder change waits at most 2 s in total, the new folder first, with closing the old scratch file getting what is left; old files close on at most four background threads, so one stuck close doesn't hold up the others. On macOS the purgeable-space lookup asks every mounted volume, so it runs separately for at most 0.5 s, and an unrelated dead mount only loses the purgeable figure, never the folder's own numbers. Limit: reading or writing tile pixels already spilled to a hung folder still blocks the thread doing it; only opening and measuring are bounded. A failed read retains its backing extent and blocks editing, saving and exporting of the document that holds the tile until it is readable again; other open documents keep working. Each commit checks the health of its own document (before and after an edit, the document a job captured, the imported result, the document being saved or exported) and refuses a result if any tile failed or recovered while it ran; the retry succeeds. Derived-pixel caches never admit results built from unreadable tiles and are purged once per new failure. Recovery can retry the original bytes after the storage becomes available; zeros from a failed display read must never become saved pixels. Limitation: a tile that was already unreadable and is read from outside the edited document (for example a clipboard copy) is not detected.

Scratch storage alone does not make every format, filter or renderer streamable. See EXR streaming for the block-by-block EXR import. Viewport rendering is a separate change built on these guards and memory reservations.

Available RAM is measured in one place, photocraft_raster::memory::system (cgroup v1/v2 limits on Linux, reclaimable pages on macOS, the commit limit on Windows; swap never counts as RAM, and an unreadable container limit fails closed). The residency budget and the PSD/PSB and layered-TIFF import admission (photocraft_io::memory) both read it. While spilling is on and not paused, import admission counts the tile pixels of a file against RAM plus the scratch room (spill::scratch_capacity: the free space above the reserve, within the Scratch Size Limit); decode buffers, conversion copies and merged planes still need RAM. With spilling off or paused, and on the web, the RAM-only limits apply unchanged.