Compare commits

...
Author SHA1 Message Date
Brandon Thomas 0f49ebf8f0 Merge main into fix/issue-101-torn-append
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:52:48 -04:00
Brandon Thomas ab86862011 Merge pull request #123 from storytold/fix/issue-92-xmp-merge
Issue #92: XMP: merge into an existing sidecar instead of replacing it
2026-10-05 18:52:25 -04:00
Brandon Thomas 577330a08b Merge main into fix/issue-92-xmp-merge
# Conflicts:
#	crates/engine/src/sidecar.rs
#	docs/whats-new.md
2026-10-05 18:50:42 -04:00
Brandon Thomas 30697cfc26 Merge pull request #122 from storytold/fix/issue-92-rename-shared-sidecar
Issue #92: Rename Photos copies a stem sidecar another file still shares
2026-10-05 18:50:26 -04:00
Brandon Thomas ddbf139e5e Merge main into fix/issue-92-rename-shared-sidecar 2026-10-05 18:49:29 -04:00
Brandon Thomas ee77490dd2 Merge pull request #121 from storytold/fix/issue-105-rename-rollback
Issue #105: Rename Photos reports failed rollbacks; Find Missing matches by content
2026-10-05 18:49:15 -04:00
Brandon Thomas f31a55c8a7 Merge main into fix/issue-105-rename-rollback
# Conflicts:
#	docs/parity.md
2026-10-05 18:48:27 -04:00
Brandon Thomas 7218b990f5 Merge pull request #120 from storytold/fix/issue-95-case-only-rename
Issue #95: Rename Photos: case-only renames never overwrite another file
2026-10-05 18:48:06 -04:00
Brandon Thomas d16d9aadeb Merge main into fix/issue-95-case-only-rename 2026-10-05 18:47:18 -04:00
Brandon Thomas 1caa04acbc Merge pull request #119 from storytold/fix/issue-98-thumb-cache-own-files
Issue #98: thumbnail cache only lists, counts and deletes its own files
2026-10-05 18:47:03 -04:00
Brandon Thomas bef2060b24 Merge main into fix/issue-98-thumb-cache-own-files 2026-10-05 18:46:59 -04:00
Brandon Thomas 40303b999c Merge pull request #118 from storytold/fix/issue-96-verified-copy-import
Issue #96: Import > Copy verifies each copy like Move
2026-10-05 18:46:42 -04:00
Brandon Thomas ce5488ad99 Merge main into fix/issue-96-verified-copy-import
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:46:20 -04:00
Brandon Thomas f6bc1ae5ae Merge pull request #117 from storytold/fix/issue-106-dng-smart-previews
Issue #106: Convert to DNG and smart previews: verified, atomic writes
2026-10-05 18:46:06 -04:00
Brandon Thomas 6c8a65c337 Merge main into fix/issue-106-dng-smart-previews 2026-10-05 18:45:48 -04:00
Brandon Thomas fe386f4f15 Merge pull request #116 from storytold/fix/issue-93-export-overwrite
Issue #93: Export: never overwrite an original; atomic writes
2026-10-05 18:45:31 -04:00
Brandon Thomas a6f421aafb Merge main into fix/issue-93-export-overwrite 2026-10-05 18:44:02 -04:00
Brandon ThomasandClaude Opus 5.5 08c2acda26 Issue #101: catalog.log: cut a partial append back before retrying
A failed append (disk full, dropped share) could leave part of its batch in
catalog.log; the retry (same seqs) then shared a line with the fragment and
the next load treated it as mid-log damage, setting later edits aside.

- Journal::append cuts the log back to the last whole record on error (or,
  if that fails too, before the next append).
- FsStore drops a failed append handle so the next attempt reopens the file;
  truncate of a never-created file is a no-op.
- The loader recognises a CRC-valid record glued after a fragment (logs
  written by older builds) and skips only the fragment.
- log_bytes now counts the newline added to an unterminated last record.
- close_library writes the closing snapshot even when appending fails; the
  snapshot absorbs the queued ops (Journal::snapshot_with_unlogged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 e96a2750a1 Issue #96: Import > Copy verifies each copy like Move
Copy import used exists() then std::fs::copy: no sync, no comparison, and
a file appearing at the chosen name in between was replaced. It now goes
through import_move::copy_new, which reuses Move's copy_verified: a new
file (create_new, retrying -1, -2… on AlreadyExists), synced and compared
byte for byte with the source; a copy that fails or differs is removed and
reported as a failed import. Copy as DNG's copy step uses the same path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:22:35 -04:00
Brandon ThomasandClaude Opus 5.5 771b0b98d7 Issue #106: Convert to DNG and smart previews: verified, atomic writes
- Convert to DNG / Copy as DNG: the encoded DNG must decode to the same raw
  data, then goes through safe_file::write_new_unique (temp file, sync,
  read-back, hard-linked to the first free <stem>.dng / -2 / -3 name, never
  replacing a file). On failure no DNG is left, the photo is not relinked
  and the raw copy is kept.
- Photo Merge writes its DNG the same way.
- Smart previews are written atomically; smart::is_valid (header + complete
  JPEG) decides "built", has_smart_preview and the offline fallback, so a
  proxy cut short is rebuilt (reported as `repaired`); moving proxies to a
  new folder never deletes a good one in favour of a damaged copy and
  copies across drives atomically.

Stacked on the #93 branch (shared lightcraft_catalog::safe_file helper).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:17:41 -04:00
Brandon ThomasandClaude Opus 5.5 f4df914731 Issue #92: Rename Photos copies a stem sidecar another file still shares
Renaming IMG_1.CR3 alone took the shared IMG_1.xmp along and left
IMG_1.JPG without it. A Stem-named sidecar that a sibling photo file
still uses is now copied (independently, never linked) instead of moved;
the last file renamed takes it. A rollback removes the copy.

Stacked on fix/issue-105-rename-rollback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:12:25 -04:00
Brandon ThomasandClaude Opus 5.5 882a059fac Issue #92: XMP: merge into an existing sidecar instead of replacing it
Saving (Cmd+S or auto-write) now merges LightCraft's properties into an
existing sidecar (lightcraft_meta::merge_xmp): owned properties (rating,
label, dc/IPTC/rights fields, lc:*) are replaced or removed, capture time
and GPS only when LightCraft has a value; everything else (crs:,
xmpMM:History, unknown namespaces, packet wrapper) is kept byte for byte.
The result is re-parsed and checked before it is written (temp + fsync +
rename). A sidecar that isn't valid XMP is copied to .xmp.bak-<time>
(never overwriting) before it is replaced; one that can't be read at all
is left alone and the save fails.

Shared stems: when several catalogued files map to one stem sidecar
(IMG.CR3 + IMG.JPG), it belongs to the raw (else the first by name) and
the others write and read <file>.xmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:08:18 -04:00
Brandon ThomasandClaude Opus 5.5 46732cb0fe Issue #93: Export: never overwrite an original; atomic writes
- lightcraft_catalog::safe_file: one tested helper for crash-safe writes
  (temp file in the same folder, sync, rename, folder sync; temp removed on
  failure), write_new/write_new_unique that never replace a file and read
  the result back, same_file, and a per-thread fault injection for tests.
- engine::originals::OriginalGuard: refuses a write whose target is a
  catalogued photo's original or XMP sidecar (by file identity), whatever
  the conflict policy or exact path. Used by run_batch (export, Export with
  Previous, control/MCP exact paths), ui.render/ui.screenshot paths, merge
  previewPath, MCP render_photo/screenshot paths (new query command
  export.checkTarget) and lightcraft-cli render.
- The conflict policy now covers an export's sidecars together with the
  file (Unique gives both the same free name, Skip skips when either is
  taken).
- export::write_file replaces the plain std::fs::write in the desktop app,
  CLI, MCP and engine writers; XMP sidecar writes use the shared helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:03:07 -04:00
Brandon Thomas 572b55a44a Merge pull request #110 from midasdf/feat/japanese-interface-and-watermarks
Add Japanese menus and vertical Japanese text watermarks
2026-10-05 16:49:18 -04:00
Brandon Thomas bd10f89d02 Merge main into PR #110 2026-10-05 16:48:57 -04:00
Brandon ThomasandClaude Opus 5.5 fb5b18843f Issue #105: Rename Photos reports failed rollbacks; Find Missing matches by content
Rollbacks in move_file, apply_rename and undo/redo no longer ignore
errors: every move that can't be undone is reported (old -> new), and the
catalog follows the files that kept their new names (an undoable partial
rename; a logged change for undo/redo), so nothing shows as missing.
A file that can't go back keeps the sidecars that moved with it.

library.findMissing: same name + size confirmed by the stored content
hash first, then renamed files by size + content hash, then a unique
name + size match; several look-alikes are skipped and reported in
'ambiguous' instead of guessed. Files already used by a photo are skipped.

Stacked on fix/issue-95-case-only-rename (same functions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:47:50 -04:00
Brandon ThomasandClaude Opus 5.5 dfbb1ad935 Issue #95: Rename Photos: case-only renames never overwrite another file
Whether an existing target is the photo itself is decided by file identity
(dev+inode on Unix; an exact directory entry check elsewhere), not by a
lower-case name comparison, so on case-sensitive volumes img_1.JPG next to
IMG_1.JPG is a collision (-1 suffix). Moves use a no-replace rename (hard
link + unlink where supported) and case changes go through a unique temp
name. Tests model case-sensitive and case-insensitive volumes and check
the real disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:36:35 -04:00
Brandon ThomasandClaude Opus 5.5 0fd0ffb783 Issue #98: thumbnail cache only lists, counts and deletes its own files
scan() now accepts only <2 hex>/<32 hex>.jpg (matching shard) and the
cache's own <32 hex>.tmp<id> temp files, so prune/clear never delete and
size() never counts foreign files in a shared thumbs/ folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:23:32 -04:00
Brandon Thomas e99aa61fc4 Merge pull request #111 from storybored379/fix/unicode-timezone
Metadata: gracefully ignore invalid Unicode timezone offsets
2026-10-05 16:22:21 -04:00
Brandon ThomasandClaude Opus 5.5 58626b982d Fonts: replace Shippori Mincho with BIZ UDMincho (no Adobe-derived glyphs)
Shippori Mincho's name table credits 'FONTDASU / Google Inc. / but / Adobe' — its kanji
come from Adobe's Source Han Serif, which our asset rules exclude even under the OFL
(CLAUDE.md: no Source Han fonts). BIZ UDMincho (Morisawa Inc., design TypeBank Co., Ltd.;
OFL-1.1, from google/fonts at 63833b7) is a Mincho of its own with no Adobe glyphs, and
smaller (6.2 MB vs 8.7 MB). Attribution, licence text and NOTICE updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:38 -04:00
midasdf e00025aaf6 i18n: add Japanese menus and vertical Japanese watermarks 2026-10-06 01:07:26 +09:00
Brandon Thomas 294012742e Merge pull request #108 from storytold/pr/freebsd-ci-sync
FreeBSD CI: match the shared template
2026-10-05 10:19:07 -04:00
Brandon ThomasandClaude Opus 5.5 4d13061580 FreeBSD CI: match the shared template (git + safe.directory, 4 vCPUs, pinned vmactions/freebsd-vm)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:45:24 -04:00
Brandon Thomas 63380829b1 Merge pull request #91 from storytold/pr/freebsd
FreeBSD: build and test the workspace in a FreeBSD VM
2026-10-05 09:30:12 -04:00
Brandon ThomasandClaude Opus 5.5 62e16d7adc FreeBSD CI: 4 vCPUs, git in the VM, pinned vmactions/freebsd-vm
The headless UI snapshot test settles for at most 120 s; on a 2-vCPU VM
under the full test load it timed out mid-render.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:58:05 -04:00
Brandon ThomasandClaude Opus 5.5 d47e941a0c docs: point agents to the shared test corpora (#90)
Real-file test oracles live in storytold/photocraft-corpus, explained in
craftrules standards/test-corpora.md. Never commit large binary fixtures
here; fetch them pinned and sha256-verified.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:24:23 -04:00
Brandon ThomasandClaude Opus 5.5 83fb206a29 FreeBSD: build and test the workspace in a FreeBSD VM
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:23:26 -04:00
Brandon Thomas 3567980442 Merge pull request #89 from storytold/chore/ignore-target-dirs
Ignore /target-*/ build dirs
2026-10-05 07:38:00 -04:00
Brandon ThomasandClaude Opus 5.5 8cd791100b Ignore /target-*/ build dirs
Agents sometimes build into target-<name>/ next to target/; ignore those too so a
'git add -A' can never commit build artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:37:49 -04:00
Brandon Thomas ff1e51e4d4 Merge pull request #88 from storytold/fix/issue-85-cr2-cfa
Issue #85: Raw: CR2 colour pattern from the file's CR2CFAPattern tag, not a constant
2026-10-05 07:37:19 -04:00
Brandon Thomas c05f2de251 Merge main into fix/issue-85-cr2-cfa
# Conflicts:
#	docs/parity.md
2026-10-05 07:37:00 -04:00
Brandon ThomasandClaude Opus 5.5 e87ee8e36d Issue #85: Raw: CR2 colour pattern from the file's CR2CFAPattern tag, not a constant
The Bayer layout of Canon CR2 files differs by model (GBRG on the 7D, 50D,
60D, 550D, 600D, 1200D, 1300D, 5D Mark II, 1D Mark IV; RGGB on the 40D,
5D Mark III, 6D, 5DS R and others). Read it from raw IFD tag 0xc5e0
(CR2CFAPattern: 1 RGGB, 2 BGGR, 3 GBRG, 4 GRBG), anchored at the full
decoded sensor; fall back to RGGB when the tag is missing. No model table.

Checked on 35 decoded CC0 raw.pixls.us CR2s (35 models): the tag agrees
with the mosaic's green-diagonal statistic on every file; the SensorInfo
border parity does not predict the layout. Corpus manifest gains the
reporter's samples; a corpus test asserts each layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:24:26 -04:00
Brandon Thomas efef3a5df1 Merge pull request #87 from storytold/docs/roadmap-assessment
Docs: honest parity assessment — where we stand, where we're going
2026-10-05 06:49:16 -04:00
Brandon ThomasandClaude Opus 5.5 b50fbf1285 Docs: honest parity assessment — where we stand, where we're going
ROADMAP.md: new 'Where we stand' (by dimension and by kind of user; checklist 79% vs
~60-70% as a day-to-day Lightroom replacement) and 'Where we're going' (ordered priorities:
camera colour calibration, CR3 and compressed raws, verified camera coverage, render fidelity
suite, lens profiles, AI model strategy, then HDR / Classic modules / video). Current
parity numbers, M11 status, risks (raw-source decision of 2026-10-05), log.

docs/parity.md: new rows LR-PROF-CAMERACOLOR, LR-IMP-CAMERA-COVERAGE (P0) and
LR-BEHAV-RENDER-FIDELITY (P1) so quality gaps are tracked; Top gaps reordered by user impact.

README.md: honest status summary above the feature table; stale rows fixed (compressed NEF,
DNG/original export, image watermark, Move import), camera colour and AI/HDR/video rows.

AGENTS.md / CLAUDE.md: agents read the assessment before picking work, and downgrade
✅ rows they find wrong or incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:48:54 -04:00
Brandon Thomas 92bd3aa688 Merge pull request #86 from storytold/fix/issue-10-nef-compressed
Issue #10: Raw: Nikon compressed NEF (lossless + lossy)
2026-10-05 06:08:28 -04:00
Brandon ThomasandClaude Opus 5.5 6592e62912 Issue #10: Raw: Nikon compressed NEF (lossless + lossy)
Decode Nikon Huffman-compressed NEF (compression 34713): lossless and lossy
(type 1 and 2) at 12 and 14 bits, with the maker-note 0x96 linearization
curve, predictor seeds and either byte order. Clean-room: 0x96 layout and
encoding-table sizes from prose write-ups (Clevy, Claff); the fixed Huffman
tables were recovered by black-box analysis of CC0 raw.pixls.us samples
(compressed vs uncompressed shots of the same scene) and verified on ~45
files from ~35 bodies. Lossy-after-split files stay preview only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 05:58:31 -04:00
Brandon Thomas 8532d4ea56 Merge pull request #84 from storytold/fix/issue-78-gpu-black-export
Issue #78: GPU export: respect device limits, fall back to CPU on GPU errors
2026-10-05 05:09:57 -04:00
Brandon ThomasandClaude Opus 5.5 2218ca5a4c Issue #78: GPU export: respect device limits, fall back to CPU on GPU errors
A GPU render no longer hands back an image it cannot vouch for:
- set the wgpu device-lost callback (wgpu never reports a lost device as an
  error, so a driver reset went unnoticed); readback waits with a timeout and
  checks the poll result and map callback instead of blocking forever
- each render runs in OutOfMemory/Validation/Internal error scopes; errors
  fail that render (OOM and limit failures for that render only, device
  errors for the process) and it is redone on the CPU
- every buffer is checked against the device limit when created (no
  validation error, no process-wide disable); LIGHTCRAFT_GPU_LIMITS=webgpu|<MiB>
  creates the device with lower limits to reproduce small adapters
- the output buffer is cleared and the per-pixel kernel writes alpha 255:
  any unwritten pixel (work silently dropped, the issue's all-black export)
  fails the render; an all-black result is re-rendered on the CPU
- renders submit per stage and every 16 M invocations, the per-pixel stage
  runs in ~4 MP bands: no long submission for a slow iGPU's watchdog to reset
- ui.inspect perf.gpuReason / perf.gpuFallback, app.gpu reason/lastFallback,
  System Info and Settings say why the GPU is off (e.g. llvmpipe skipped) and
  why the last render fell back

Tests: crates/gpu/tests/fallback.rs (limit, mid-render limit, validation
error, dropped work, black photo), engine export falls back to the CPU image,
banded 3000x2000 GPU/CPU equivalence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 05:06:24 -04:00
Brandon Thomas 14053b2e40 Merge pull request #83 from storytold/fix/local-record-eviction
Local: forget untouched browse records of folders not browsed for 30 days
2026-10-05 02:59:23 -04:00
Brandon Thomas 1f2f25356a Merge main into fix/local-record-eviction
# Conflicts:
#	crates/engine/src/cmd/library.rs
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 02:59:05 -04:00
Brandon Thomas 2ed6af155d Merge pull request #82 from storytold/fix/persist-errors
Persistence: a failed save fails the command (change kept in memory, queued, retried)
2026-10-05 02:58:49 -04:00
Brandon ThomasandClaude Opus 5.5 b14cf58630 Local: forget untouched browse records of folders not browsed for 30 days
Local browse records can be most of a large catalog (85k photos, 90 %
Local: 313.5 MB snapshot). Untouched Local records of folders whose last
browse is older than library.preferences forgetLocalDays (default 30,
0 = never) are now removed from the catalog when the library opens (and by
library.forgetLocal {dryRun?, days?}), as journaled, non-undoable ops, then
compacted in the background. Files and XMP sidecars are never touched;
browsing the folder again catalogues them afresh.

Untouched = a Local file record, not deleted, not a virtual copy, whose
state still hashes to the fingerprint taken when the browse catalogued it
(any change keeps it); records from before fingerprints need every
user-editable field empty/default. Kept too: records in albums (or as
cover), stacks, sources of virtual copies, visible/selected/edited ones,
ones referenced by undo/redo, ones with a smart preview.

Per-folder last-browsed times live in the catalog (Op::SetBrowsed,
normalised with folder_key; re-stamped at most hourly). Old catalogs load
unchanged; folders without a time are stamped now, so nothing is forgotten
on the first open after the upgrade.

persist_bench (85k photos, 76.5k Local, every 7th folder recent, 1 in 50
rated): 63.8k forgotten, plan 162 ms, snapshot 313.5 MB -> 94.9 MB.
Settings -> Performance -> Local folders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 02:26:49 -04:00
Brandon ThomasandClaude Opus 5.5 104444edcb Persistence: a failed save fails the command (change kept in memory, queued, retried)
A command whose journal append fails now returns EngineError::NotSaved
("saved in memory but not written to disk: <reason>; LightCraft will retry")
instead of OK. The change stays applied and undoable, its ops stay queued and
are written by the next save (any command, or the frame loop with a 2 s
backoff). Queries and commands that change nothing are unaffected; a failed
compaction is reported in lastError, not as a failed command (and a failed
synchronous snapshot now backs off like a background one).

Session::unsaved(), library.info unsavedOps/unsavedError, ui.inspect unsaved;
the desktop UI toasts the failure and shows a warning on the top bar's cloud
icon until a save succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 02:26:49 -04:00
storybored379andOpenAI Codex 35e38661a7 Metadata: gracefully reject invalid Unicode timezone offsets
Co-Authored-By: OpenAI Codex <noreply@openai.com>
2026-10-04 21:11:55 -07:00
96 changed files with 6416 additions and 452 deletions

No files matched your search

+48
View File
@@ -0,0 +1,48 @@
# Builds and tests the workspace on FreeBSD, inside a FreeBSD VM on a Linux runner
# (GitHub has no FreeBSD runners). GPU tests skip themselves when the VM has no adapter.
name: FreeBSD
on:
push:
branches: [main]
pull_request:
paths:
- Cargo.toml
- Cargo.lock
- "**/Cargo.toml"
- .github/workflows/freebsd.yml
workflow_dispatch:
concurrency:
group: freebsd-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
freebsd:
name: FreeBSD 14 (build + test)
runs-on: ubuntu-latest
timeout-minutes: 300
steps:
- uses: actions/checkout@v4
- name: Build and test in a FreeBSD VM
uses: vmactions/freebsd-vm@c46abacb49f09938ca4e1702d15d836285d694cc # v1
with:
release: "14.3"
usesh: true
cpu: 4
mem: 12288
# Nothing to bring back; copying the target dir out of the VM fails on large builds.
copyback: false
prepare: |
pkg install -y git curl pkgconf cmake python3 libxkbcommon wayland libX11 libXcursor libXrandr libXi libxcb mesa-libs vulkan-loader gtk3 fontconfig freetype2 alsa-lib
curl -fsSL https://sh.rustup.rs | sh -s -- -y --profile minimal
run: |
. "$HOME/.cargo/env"
# The checkout is synced in under another owner; let tests that call git read it.
git config --global --add safe.directory "*"
rustc --version
CARGO_PROFILE_DEV_DEBUG=0 cargo build --workspace --all-targets --locked
CARGO_PROFILE_DEV_DEBUG=0 cargo test --workspace --locked
+1
View File
@@ -1,6 +1,7 @@
*~
plan/
/target
/target-*/
/dist
/corpus
/log
+8 -2
View File
@@ -5,8 +5,13 @@ LightCraft is a clean-room, open-source, pure-Rust photo library + non-destructi
## Start every session here
1. Read `plan/STATUS.md` (current milestone, next unchecked task, blockers).
2. Read the task in `plan/execution-plan.md` §3, the relevant section of `plan/architecture.md`, and the README/docs of the crate you touch. Behaviour/visual reference: `plan/lightroom/` (incl. `10-observed-ui.md` + screenshots).
3. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (Lightroom parity tracker: one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
4. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
3. **Know where we stand:** read [`ROADMAP.md`](ROADMAP.md) → *Where we stand* (honest assessment by dimension and
by kind of user) and *Where we're going* (ordered priorities). The checklist counts features that *exist*; the real
gaps are quality and coverage: camera colour calibration, CR3 / compressed raws, per-model verification, render
fidelity against Lightroom, AI models, HDR / video / Classic modules. A ✅ row is not proof of parity — if you find a
✅ feature that is wrong or incomplete, downgrade it to 🟡 with a note.
4. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (ordered by user impact; the tracker has one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit, and the *Where we stand* / *Where we're going* sections of `ROADMAP.md` when a listed gap closes; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
5. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
`plan/` is gitignored (local-only).
@@ -86,6 +91,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
- Shell gotcha: `mv`/`cp` are aliased interactive here — use `/bin/mv -f` / `/bin/cp -f`.
- Parallel agents: separate git worktrees and `CARGO_TARGET_DIR=target/agent-<name>`; each agent uses its **own control port** (pick one in 18000–19999, never the default 7980) and its own scratch subfolder (`<scratch>/<agent-name>/`) — never `rm -rf` shared paths; delete your target dir when done (disk is shared); keep every `Cargo.toml` valid at all times (the `crates/*` glob means one broken manifest breaks everyone).
- Test corpora: `cargo xtask corpus --download` into `corpus/` (gitignored, CC0 only). Never commit media.
- Shared real-file test corpora (Photoshop-authored PSDs, etc.) live in [`storytold/photocraft-corpus`](https://github.com/storytold/photocraft-corpus), explained in [craftrules `standards/test-corpora.md`](https://github.com/storytold/craftrules/blob/main/standards/test-corpora.md). Never commit large binary fixtures to this repo; fetch them pinned by commit and sha256-verified, as PhotoCraft does with `cargo xtask corpus`.
## Testing & performance (do this often)
- Unit/property tests next to the code; end-to-end tests drive real binaries (`crates/mcp/tests/e2e.rs`,
+8 -2
View File
@@ -5,8 +5,13 @@ LightCraft is a clean-room, open-source, pure-Rust photo library + non-destructi
## Start every session here
1. Read `plan/STATUS.md` (current milestone, next unchecked task, blockers).
2. Read the task in `plan/execution-plan.md` §3, the relevant section of `plan/architecture.md`, and the README/docs of the crate you touch. Behaviour/visual reference: `plan/lightroom/` (incl. `10-observed-ui.md` + screenshots).
3. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (Lightroom parity tracker: one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
4. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
3. **Know where we stand:** read [`ROADMAP.md`](ROADMAP.md) → *Where we stand* (honest assessment by dimension and
by kind of user) and *Where we're going* (ordered priorities). The checklist counts features that *exist*; the real
gaps are quality and coverage: camera colour calibration, CR3 / compressed raws, per-model verification, render
fidelity against Lightroom, AI models, HDR / video / Classic modules. A ✅ row is not proof of parity — if you find a
✅ feature that is wrong or incomplete, downgrade it to 🟡 with a note.
4. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (ordered by user impact; the tracker has one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit, and the *Where we stand* / *Where we're going* sections of `ROADMAP.md` when a listed gap closes; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
5. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
`plan/` is gitignored (local-only).
@@ -86,6 +91,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
- Shell gotcha: `mv`/`cp` are aliased interactive here — use `/bin/mv -f` / `/bin/cp -f`.
- Parallel agents: separate git worktrees and `CARGO_TARGET_DIR=target/agent-<name>`; each agent uses its **own control port** (pick one in 18000–19999, never the default 7980) and its own scratch subfolder (`<scratch>/<agent-name>/`) — never `rm -rf` shared paths; delete your target dir when done (disk is shared); keep every `Cargo.toml` valid at all times (the `crates/*` glob means one broken manifest breaks everyone).
- Test corpora: `cargo xtask corpus --download` into `corpus/` (gitignored, CC0 only). Never commit media.
- Shared real-file test corpora (Photoshop-authored PSDs, etc.) live in [`storytold/photocraft-corpus`](https://github.com/storytold/photocraft-corpus), explained in [craftrules `standards/test-corpora.md`](https://github.com/storytold/craftrules/blob/main/standards/test-corpora.md). Never commit large binary fixtures to this repo; fetch them pinned by commit and sha256-verified, as PhotoCraft does with `cargo xtask corpus`.
## Testing & performance (do this often)
- Unit/property tests next to the code; end-to-end tests drive real binaries (`crates/mcp/tests/e2e.rs`,
Generated
+1
View File
@@ -2188,6 +2188,7 @@ dependencies = [
"serde",
"serde_json",
"thiserror 2.0.21",
"web-time",
]
[[package]]
+4
View File
@@ -14,3 +14,7 @@ list (author, source, license) is in assets/ATTRIBUTION.md.
Inter (assets/fonts/Inter-*.ttf) — SIL Open Font License 1.1. See assets/fonts/OFL-Inter.txt.
Showcase photographs by Ansel Adams, Dorothea Lange and NASA (docs/images/) — public domain. See assets/ATTRIBUTION.md.
BIZ UDMincho: Copyright 2022 The BIZ UDMincho Project Authors
(https://github.com/googlefonts/morisawa-biz-ud-mincho). SIL Open Font License 1.1;
see assets/fonts/OFL-BIZUDMincho.txt.
+19 -7
View File
@@ -280,9 +280,18 @@ lightcraft --control 7980 ~/Pictures/trip
## Feature status
LightCraft is young and moving fast; see the [roadmap](ROADMAP.md) for estimates. The detailed, row-by-row
Lightroom parity tracker (every feature, menu item and shortcut with its status, evidence and the top gaps) is
[docs/parity.md](docs/parity.md).
LightCraft is young and moving fast. **Where we honestly stand** (details in the [roadmap](ROADMAP.md#where-we-stand)):
- **By feature count we're at ~79%** of Lightroom (core features 98%), tracked row by row in
[docs/parity.md](docs/parity.md).
- **As a day-to-day Lightroom replacement we're nearer 60–70%.** It's great for JPEG/DNG and most Nikon / Sony /
older-Canon raws on one machine.
- **The biggest gaps:**
- **camera colour calibration:** raws other than DNG develop with a neutral colour matrix today, so colour is muted;
- **CR3 and compressed Fujifilm / Olympus raws:** these open as embedded previews only;
- **AI masks and denoise:** subject and sky selection are classical heuristics;
- **HDR, video and the Classic Print / Book / Map modules.**
- **What's next:** see [where we're going](ROADMAP.md#where-were-going).
| Area | Status |
|---|---|
@@ -292,17 +301,20 @@ Lightroom parity tracker (every feature, menu item and shortcut with its status,
| Masking: brush, linear, radial, luminance/colour range, add/subtract/intersect | ✅ (AI subject/sky use classical heuristics for now) |
| Crop, straighten tool + auto straighten, flip, rotate, aspect ratios, overlays | ✅ |
| Profiles (Color, Neutral, Vivid, Landscape, Portrait, Monochrome: our own looks), presets, versions, history, copy/paste/sync settings | ✅ |
| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; neutral fallback today) |
| Native macOS menu bar (generated from the command registry), control channel + every widget addressable, headless UI snapshots | ✅ |
| RAW: DNG, CR2, ARW, NEF (uncompressed), Fujifilm RAF (Bayer + X-Trans), Panasonic RW2, Pentax PEF, Olympus ORF; embedded previews for every format incl. CR3 | ✅ · compressed NEF/RAF/ORF, CR3 decode 🚧 |
| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2, Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
| Detail: sharpening, luminance + colour noise reduction | ✅ · AI Denoise, Super Resolution ⬜ |
| Remove / Heal / Clone spots (auto source), Visualize Spots, Red Eye and Pet Eye (auto pupil detection, catchlight) | ✅ · content-aware fill, spot pin editing 🚧 |
| Export: JPEG / PNG / TIFF / WebP / AVIF, sizing, file-size limit, output sharpening, naming, batch, metadata policy, text watermark | ✅ · DNG export, image watermark ⬜ |
| Library persistence (crash-safe op log + snapshots), disk thumbnail cache, import with duplicate detection | ✅ |
| Export: JPEG / PNG / TIFF / WebP / AVIF / DNG / original, sizing, file-size limit, output sharpening, naming templates, batch, metadata policy, text or image watermark | ✅ · HDR export ⬜ |
| Library persistence (crash-safe op log + snapshots, background compaction, failed saves reported), disk thumbnail cache | ✅ |
| Import: Add in place / Copy / Move, rename and folder templates, devices, duplicate detection, watched folders; Local folder browsing | ✅ |
| MCP server (headless or live app, persistent libraries), CLI, control channel | ✅ |
| XMP sidecars (read/write, auto-write), reading `crs:` develop settings, preset files (`.lcpreset`, XMP presets) | ✅ |
| Optics (distortion, vignetting, auto + manual CA, defringe, DNG-embedded lens corrections), Geometry (transforms, Constrain Crop), Upright (Auto/Level/Vertical/Full/Guided) | ✅ · camera lens profiles (our own) ⬜ |
| Photo Merge: HDR (auto-align, deghost), Panorama (spherical/cylindrical/perspective, boundary warp, auto crop), HDR Panorama → DNG | ✅ |
| GPU pipeline (wgpu compute, CPU-exact within 1/255) with CPU fallback | ✅ · WebGPU in the browser 🚧 |
| GPU pipeline (wgpu compute, CPU-exact within 1/255), CPU fallback on device limits / errors | ✅ · WebGPU in the browser 🚧 |
| AI: segmentation masks, AI denoise, super resolution, faces; HDR editing; video | ⬜ (see [roadmap](ROADMAP.md#where-were-going)) |
| Web build (same UI in the browser via WASM): persistent library in OPFS/IndexedDB, Web Worker rendering, export downloads | ✅ · WebGPU, Safari/Firefox testing 🚧 |
<sub>✅ works today · 🚧 in progress · ⬜ not started</sub>
+98 -18
View File
@@ -4,6 +4,68 @@ Milestones toward full Adobe Lightroom parity (cloud Lightroom first, then every
estimates for continuous (24/7) agent-driven development with 4–6 parallel agents. Estimates are calibrated on the sibling
projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revised as milestones land.
## Where we stand
*Honest assessment, 2026-10-05. Agents: read this before picking work. The checklist in
[`docs/parity.md`](docs/parity.md) counts features that **exist**; this section is about whether a photographer can
**switch** from Lightroom. Update it when a gap below closes.*
**In one line:** the checklist says **79%** (P0 98.5%, P1 95.9%, P2 39%), but measured by whether a working
photographer could replace Lightroom without noticing, we are at roughly **60–70%**. The remaining gap is mostly
**quality of results and camera coverage**, not missing buttons.
Caveat on the checklist: ✅ is set by whoever lands a feature, and nobody has systematically checked rows against
Lightroom's behaviour or output. Bugs keep turning up in ✅ areas (CR2 colour-filter phase on some Canon models #85,
duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
### By dimension
| Dimension | Estimate | What's true today | Biggest gaps |
|---|---:|---|---|
| **Feature checklist** | 79% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
| **RAW coverage** (formats people shoot) | ~50% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, packed RW2, PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, RW2 v4, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. Per-model verification is thin (~40 corpus files vs >1,000 models) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No camera colour calibration**: non-DNG raws use a neutral matrix, so colour is muted/inaccurate. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
| **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |
| **Platform & robustness** | ~70% | macOS native; Windows/Linux builds; web via WASM; no-panic lints workspace-wide, `unsafe` confined to `crates/sysmem`; failed saves are reported; GPU errors fall back to CPU | Windows installer UI unverified on Windows (PR #79); GPU path proven only on Apple + user reports; no localisation; accessibility partial; headless UI tests time out under machine load |
### By kind of user
| User | Readiness | What blocks them |
|---|---:|---|
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour (muted colour on every photo) |
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only |
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
| Relies on AI (masks, denoise) | ~25% | No segmentation or AI denoise models |
## Where we're going
Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/parity.md) → *Top gaps*.
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
matrices the files carry themselves, then chart shots. Raises colour quality for every non-DNG raw at once.
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, RW2 v4, NEF
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
no LGPL dependency); compressed NEF (#86) is the template.
3. **Verified camera coverage** (LR-IMP-CAMERA-COVERAGE, P0): a CC0 sample per model in the corpus, each decoded and
checked for plausible colour; fix per-model bugs (#85).
4. **Render fidelity suite** (LR-BEHAV-RENDER-FIDELITY, P1): measure our output against Lightroom on the same CC0 raws
(references stay in the local `plan/`), then tune against the numbers.
5. **Lens profiles of our own** (LR-EDIT-OPTICS-PROFILE, P1).
6. **AI model strategy** (maintainer decision): which permissively licensed models (or our own training) for
segmentation masks and denoise; then pure-Rust inference. Unblocks M12 and Enhance.
7. **Then:** HDR (Q), the Classic output modules (Print first, then Map view, Book, Slideshow), video (R), localisation
and accessibility.
Already closed in the week of 2026-10-03: compressed NEF (#10), Move import (#29), per-library smart-preview folder,
Local roots / cleanup, grid and catalog performance at 85k photos (#35, #37), failed-save reporting, GPU export
hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help and folder templates (#31, #32).
## Milestones
**Status legend:** ✅ done · 🚧 in progress · ⬜ not started
| # | Milestone | Scope (summary) | Estimate (h) | Status |
@@ -11,7 +73,7 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi
| M0 | Skeleton + visual shell | workspace, xtask CI + layering, geom/color/raster, develop model, pipeline v0, catalog v0, engine commands, Lightroom-look UI (grid, loupe, filmstrip, Edit panel), control channel, MCP, web build | 3–5 | ✅ |
| M1 | Library core | import (JPEG/PNG/TIFF/WebP), EXIF/XMP, persistent catalog (op log + snapshots), albums, ratings/flags/labels, filter/search/sort, thumbnail cache, 100k-photo grid | 6–10 | ✅ (100k-photo grid scale test pending) |
| M2 | Pipeline v1 (quality) | WB temp/tint, profiles, local tone mapping (highlights/shadows), curves, HSL, point colour, colour grading, texture/clarity/dehaze, vignette, grain, B&W, auto tone/WB, histogram, before/after | 10–15 | ✅ (look tuning vs our references ongoing) |
| M3 | RAW I | TIFF/DNG (LJ92, deflate, tiles, opcodes), demosaic (AHD/PPG/bilinear), highlight recovery, DNG colour model, CR2, NEF, ARW, embedded previews | 10–15 | 🚧 (DNG, CR2, ARW, NEF uncompressed, embedded previews ✅; NEF Huffman ⬜) |
| M3 | RAW I | TIFF/DNG (LJ92, deflate, tiles, opcodes), demosaic (AHD/PPG/bilinear), highlight recovery, DNG colour model, CR2, NEF, ARW, embedded previews | 10–15 | 🚧 (DNG, CR2, ARW, NEF uncompressed + Huffman lossless/lossy, embedded previews ✅; NEF lossy-after-split ⬜) |
| M4 | Crop, geometry, optics | crop tool + overlays, straighten, Upright (auto/level/vertical/full/guided), manual transforms, CA, defringe, manual lens corrections | 6–10 | ✅ |
| M5 | Performance | source pyramids, wgpu compute pipeline (CPU oracle), draft/full renders, prefetch, budgets (16 ms slider updates on 24 MP) | 10–15 | 🚧 (stage cache, source pyramid, wgpu pipeline, prefetch, memory budget ✅; colour NR at half resolution, GPU histogram ⬜) |
| M6 | Masking | brush, linear/radial gradients, colour/luminance/depth range, add/subtract/intersect/invert, all local adjustments, masks panel | 8–12 | 🚧 (brush/linear/radial/colour/luminance range, add/subtract/intersect, masks panel ✅; depth range, AI masks ⬜) |
@@ -19,24 +81,27 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi
| M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) |
| M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ |
| M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) |
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; CR3, compressed NEF/ORF/RAF ⬜) |
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; **camera colour calibration** ⬜ (top priority), CR3, compressed ORF/RAF ⬜) |
| M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ |
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
| M15 | Classic modules | Map, Book, Slideshow, Print, Web; smart collections, stacks, virtual copies, publish services, tethering | 25–40 | 🚧 (smart albums, stacks, virtual copies, compare/survey ✅; Map/Book/Slideshow/Print/Web ⬜) |
| M16 | 1.0 polish | preferences, shortcut editor, accessibility, localization, packaging (dmg/msi/AppImage/web), hardening | 10–20 | 🚧 (settings, keyboard shortcuts sheet, packaging basics ✅; accessibility, localisation ⬜) |
## Parity estimate (2026-10-02, evening update)
## Parity estimate (feature count updated 2026-10-05; effort estimate from 2026-10-02)
**By feature count** — from `docs/parity.md` (one row per Lightroom feature, menu item and shortcut; `cargo xtask
parity` prints this line on every run, so it stays current):
| Scope | Weighted completion | Rows | Morning |
| Scope | Weighted completion | Rows | 2026-10-02 |
|---|---:|---:|---:|
| P0 (core) | **98.7%** | 198 | 98.5% |
| P1 (important parity) | **94.4%** | 144 | 79.9% |
| P2 (later / AI / niche, incl. Classic modules) | **29.4%** | 158 | 11.7% |
| **All in-scope rows** | **75.6%** | 500 | 65.7% |
| P0 (core) | **98.5%** | 200 | 98.7% |
| P1 (important parity) | **95.9%** | 147 | 94.4% |
| P2 (later / AI / niche, incl. Classic modules) | **39.2%** | 158 | 29.4% |
| **All in-scope rows** | **79.2%** | 505 | 75.6% |
The 2026-10-05 count includes three new 🟡 rows that make quality gaps visible (camera colour calibration, verified
camera coverage, render fidelity), which is why P0 dipped slightly.
✅ counts 1, 🟡 ½, ⬜ 0; out-of-scope rows (cloud sharing, Adobe accounts…) are left out.
@@ -50,7 +115,7 @@ trip, slideshow, auto import…) in ≈ 12 h on 2026-10-02; four to six parallel
| Work package | Tracker rows | Agent-hours | Risk |
|---|---|---:|---|
| Remaining P0/P1 UI and library features (folder rename/move, keyword painter, people view…) | ≈ 8 | 5–10 | low |
| Raw codecs: CR3 (CRX), compressed NEF / ORF / RAF, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
| Lens-profile database of our own (calibration targets, fitting, data) | LR-EDIT-OPTICS-PROFILE | 15–30 | data collection |
| Video: playback, trim, edits, export (pure-Rust decode, ideally shared with FilmCraft) | R. Video | 20–40 | medium |
| AI: subject / sky / background / people / object masks, object-aware remove, AI denoise, super resolution, lens blur, people & faces, natural-language search, culling | ≈ 30 | 80–150 | **high** — permissively licensed weights, pure-Rust inference, maybe training |
@@ -86,24 +151,25 @@ The milestone estimates in the table above were made before work started and are
fallbacks first. No permissively licensed sky-segmentation or raw-denoise model was found — we may need to train our own.
- **Camera colour and lens data** is a data problem: we never use Adobe's matrices, DCPs or LCPs. DNG-embedded data first,
then our own calibration; long-tail camera/lens coverage grows over time.
- **Legal decisions pending:** whether GPL-licensed *prose* format descriptions (e.g. the public CR3 write-up) may be read
by a designated engineer to produce an internal spec; freedom-to-operate review for local Laplacian filters, PatchMatch and
HEVC (HEIC).
- **Look parity** with Adobe's default rendering is subjective tuning against our own reference targets.
- **Raw-format sources:** decided 2026-10-05: decoders are written from *prose* format descriptions (even ones
published alongside GPL code); decoder source is never read. Still open: freedom-to-operate review for local
Laplacian filters, PatchMatch and HEVC (HEIC).
- **Look parity** with Adobe's default rendering is tuned by eye today; the planned fidelity suite (LR-BEHAV-RENDER-FIDELITY)
turns it into measured comparisons against local-only Lightroom references.
## Raw format coverage and known gaps
Decoded (CC0 corpus from raw.pixls.us, `cargo xtask corpus --download`, `crates/raw/tests/corpus.rs`): DNG (uncompressed,
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92), NEF/NRW uncompressed, RAF uncompressed (Bayer and
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed (Bayer and
X-Trans), RW2 packed 12/14-bit, PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
supported container also yields its embedded JPEG preview (CR3 too), and the engine shows that preview for raw variants
it can't decode yet.
Not decoded yet — preview only (no permissively licensed description; black-box analysis incomplete):
- **Nikon Huffman NEF** (lossless / lossy). Black-box findings so far: maker note `0x0096` holds per-parity predictor
seeds (2048 for 14-bit lossless); ≈ 9 bits per pixel; the category-0 code word is a 6-bit rotation of `011111`. The
Huffman tables are not stored in the files, and an automatic table search (beam search over prefix codes scored by
smoothness and bit rate, validated on Pentax files whose tables *are* stored) did not converge.
- **Nikon "lossy after split" NEF** (non-zero split row in maker note `0x0096`, e.g. some D3400/D5000/D5200/D5500/D5600
files): rows above the split decode with the regular lossy table; from the split row on a different code is used
that our black-box analysis has not recovered yet (none of the four regular tables fits, also not with byte
alignment or reset predictors at the split row). The other Nikon Huffman variants are decoded (`crates/raw/src/vendor/nefc.rs` documents the analysis).
- **Panasonic RW2 raw format 4** (quantised): the block layout is known (0x4000-byte chunks rotated by 0x1ff8; 128-bit
blocks of two 12-bit seeds plus four groups of a 2-bit scale and three 8-bit codes; scales 0/1 are ×1/×2 differences
from the same-colour pixel two to the left), the reconstruction rule for scales 2/3 is not established.
@@ -119,3 +185,17 @@ own chart-based calibration (M11.4). Adobe matrices are never used.
- 2026-09-30 (later): app running with the full Lightroom-style UI; pipeline v0; DNG/CR2/ARW; README showcase. ≈ 8 h elapsed.
- 2026-10-02: parity estimate added (xtask parity prints weighted completion); milestone statuses refreshed.
- 2026-10-01: RAW II formats: RAF (uncompressed Bayer + X-Trans), RW2 (packed), PEF (incl. Huffman), ORF (uncompressed); embedded previews for every container incl. CR3; raw corpus test with 37 CC0 samples.
- 2026-10-03 – 10-05: community PRs (#42–#51, #60) and 25+ issues landed: compressed NEF, Move import, folder templates,
Local roots and cleanup, 85k-photo grid and catalog performance, background compaction, failed-save errors, GPU
export hardening, clippy 1.99. Added the honest *Where we stand* assessment and *Where we're going* priorities; added
tracker rows for camera colour, camera coverage and render fidelity.
## Japanese interface and text watermarks
English/Japanese interface language is persisted in UI state. Core menus have Japanese
translations; untranslated panels and dialogs retain English. BIZ UDMincho is bundled
under OFL for Japanese glyph coverage without system fonts. Text watermarks now accept
`vertical: true` in export JSON/presets and expose an orientation selector. Japanese
characters stay upright in top-to-bottom columns, with newlines starting columns to the
left. This is basic lettering, without tate-chu-yoko, ruby, kinsoku, or general vertical
OpenType shaping. The same coverage renderer serves 8/16/32-bit exports.
+13 -17
View File
@@ -533,11 +533,17 @@ fn render(args: &[String]) -> Result<(), String> {
.ok_or_else(|| format!("{output}: unknown extension (use .jpg .png .tif .webp .avif .dng or --opt format=…)"))?;
}
let e = export_photo(&mut s, lightcraft_engine::catalog::PhotoId(id), &o, 1)?;
std::fs::write(&output, &e.bytes).map_err(|err| format!("{output}: {err}"))?;
for (ext, bytes) in &e.sidecars {
let sc = Path::new(&output).with_extension(ext);
std::fs::write(&sc, bytes).map_err(|err| format!("{}: {err}", sc.display()))?;
eprintln!("lightcraft-cli: wrote {}", sc.display());
// never over the input (or its sidecar), however it is spelled: `render IMG.jpg -o IMG.jpg`
let sidecars: Vec<(String, &Vec<u8>)> =
e.sidecars.iter().map(|(ext, bytes)| (Path::new(&output).with_extension(ext).to_string_lossy().to_string(), bytes)).collect();
let guard = s.original_guard();
for p in std::iter::once(&output).chain(sidecars.iter().map(|(p, _)| p)) {
guard.check(Path::new(p)).map_err(|err| format!("render: {err}"))?;
}
lightcraft_engine::export::write_file(&output, &e.bytes)?;
for (sc, bytes) in &sidecars {
lightcraft_engine::export::write_file(sc, bytes)?;
eprintln!("lightcraft-cli: wrote {sc}");
}
eprintln!("lightcraft-cli: wrote {output} ({}×{})", e.width, e.height);
Ok(())
@@ -593,18 +599,8 @@ fn snapshot(args: &[String]) -> Result<(), String> {
eprintln!("lightcraft-cli snapshot: imported {n} files in {:.0} ms", ti.elapsed().as_secs_f64() * 1e3);
}
let services = lightcraft_ui_egui::Services {
write_shared: Some(std::sync::Arc::new(|p: &str, b: &[u8]| {
if let Some(dir) = Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| format!("{p}: {e}"))
})),
write: Some(Box::new(|p: &str, b: &[u8]| {
if let Some(dir) = Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| format!("{p}: {e}"))
})),
write_shared: Some(std::sync::Arc::new(lightcraft_engine::export::write_file)),
write: Some(Box::new(lightcraft_engine::export::write_file)),
png: Some(Box::new(|img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
})),
+18
View File
@@ -86,6 +86,24 @@ fn render_subcommand() {
assert!(String::from_utf8_lossy(&o.stderr).contains("unknown control"));
}
/// Issue #93: `render IMG -o IMG` replaced its own input with the render.
#[test]
fn render_refuses_to_overwrite_its_input() {
let input = tmp("self-in.png");
gradient_png(&input);
let before = std::fs::read(&input).unwrap();
let dir = input.parent().unwrap();
std::fs::create_dir_all(dir.join("sub")).unwrap();
let other_spelling = dir.join("sub/../self-in.png");
for out in [&input, &other_spelling] {
let o =
Command::new(BIN).args(["render", input.to_str().unwrap(), "-o", out.to_str().unwrap(), "--set", "light.exposure=1"]).output().unwrap();
assert!(!o.status.success());
assert!(String::from_utf8_lossy(&o.stderr).contains("never writes over an original"), "{}", String::from_utf8_lossy(&o.stderr));
}
assert_eq!(std::fs::read(&input).unwrap(), before, "the input is untouched");
}
#[test]
fn render_export_options() {
let input = tmp("o-in.png");
+3 -12
View File
@@ -209,18 +209,9 @@ fn services() -> Services {
.save_file()
.map(|p| p.to_string_lossy().to_string())
})),
write_shared: Some(std::sync::Arc::new(|p: &str, b: &[u8]| {
if let Some(dir) = std::path::Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| e.to_string())
})),
write: Some(Box::new(|p: &str, b: &[u8]| {
if let Some(dir) = std::path::Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| e.to_string())
})),
// atomic (temp file + sync + rename): a failed write never leaves a truncated file
write_shared: Some(std::sync::Arc::new(lightcraft_engine::export::write_file)),
write: Some(Box::new(lightcraft_engine::export::write_file)),
png: Some(Box::new(|img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
})),
+5 -5
View File
@@ -211,7 +211,7 @@ impl NativeMenu {
// the application menu
let app_menu = Submenu::new("LightCraft", true);
let about = MenuItem::with_id("app.about", "About LightCraft", true, None);
let settings = MenuItem::with_id(SETTINGS, "Settings…", true, accelerator(SETTINGS_KEY));
let settings = MenuItem::with_id(SETTINGS, app.ui.language.tr("Settings…"), true, accelerator(SETTINGS_KEY));
let quit = MenuItem::with_id(QUIT, "Quit LightCraft", true, accelerator("Cmd+Q"));
let _ = app_menu.append_items(&[
&about,
@@ -230,7 +230,7 @@ impl NativeMenu {
for (title, nodes) in &bar {
let sub = Submenu::new(title, true);
if title == "Window" {
if title == app.ui.language.tr("Window") {
let _ = sub.append_items(&[
&PredefinedMenuItem::minimize(None),
&PredefinedMenuItem::maximize(Some("Zoom")),
@@ -245,15 +245,15 @@ impl NativeMenu {
.collect();
let nodes = tidy_separators(nodes);
self.append_nodes(&sub, &nodes);
if title == "File" {
if title == app.ui.language.tr("File") {
// ⌘W, the system's own item
let _ = sub.append_items(&[&PredefinedMenuItem::separator(), &PredefinedMenuItem::close_window(None)]);
}
if title == "Window" {
if title == app.ui.language.tr("Window") {
let _ = sub.append_items(&[&PredefinedMenuItem::separator(), &PredefinedMenuItem::bring_all_to_front(None)]);
sub.set_as_windows_menu_for_nsapp();
}
if title == "Help" {
if title == app.ui.language.tr("Help") {
sub.set_as_help_menu_for_nsapp();
}
let _ = self.menu.append(&sub);
+2
View File
@@ -9,6 +9,8 @@ covers a whole directory), or if a licence file referenced in the Licence column
| Path | Asset | Creator | Source | Licence | Added | Modifications |
|---|---|---|---|---|---|---|
| `assets/fonts/Inter-*.ttf` (Regular, Medium, SemiBold) | Inter | Rasmus Andersson | https://github.com/rsms/inter | SIL Open Font License 1.1 (`assets/fonts/OFL-Inter.txt`) | 2026-09-30 | none |
| `assets/fonts/BIZUDMincho-Regular.ttf` | BIZ UDMincho Regular | The BIZ UDMincho Project Authors (Morisawa Inc., design TypeBank Co., Ltd.) | https://github.com/google/fonts/tree/63833b7d10bb1f86a8f0b91cba2d3ae1f68d1aa3/ofl/bizudmincho | OFL-1.1 (`assets/fonts/OFL-BIZUDMincho.txt`) | 2026-10-05 | none |
| `assets/fonts/OFL-BIZUDMincho.txt` | BIZ UDMincho licence | The BIZ UDMincho Project Authors | https://github.com/google/fonts/blob/63833b7d10bb1f86a8f0b91cba2d3ae1f68d1aa3/ofl/bizudmincho/OFL.txt | OFL-1.1 | 2026-10-05 | none |
| `assets/app-icon/` (`lightcraft.svg` and the PNG/ICNS/ICO/SVG files derived from it) | LightCraft app icon (lynx) | LightCraft project owner (drawn in ArtCraft) | original work (vectorised from the owner's drawing; see `assets/app-icon/README.md`) | MIT OR Apache-2.0 (`assets/app-icon/LICENSE.txt`) | 2026-10-02 | vectorised to SVG; rendered to PNG/ICNS/ICO by `packaging/icons.sh` |
| `crates/scenes/` (generated images) | Procedural demo photographs | LightCraft contributors | original work (generated at runtime, no source imagery) | MIT OR Apache-2.0 | 2026-09-30 | n/a |
| `crates/ui-egui/src/icons.rs` | UI icons drawn as vector paths in code | LightCraft contributors | original work | MIT OR Apache-2.0 | 2026-09-30 | n/a |
Binary file not shown.
+93
View File
@@ -0,0 +1,93 @@
Copyright 2022 The BIZ UDMincho Project Authors (https://github.com/googlefonts/morisawa-biz-ud-mincho)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
+46
View File
@@ -44,6 +44,11 @@ fn photo(c: &mut Catalog, i: u64) -> Photo {
p.meta.iso = Some(400);
p.as_shot_wb = Some((5200.0, 3.0));
p.local = !i.is_multiple_of(10);
if p.local {
// as a browse catalogues it: import defaults, a baseline fingerprint
p.develop = Arc::new(p.import_defaults());
p.set_local_baseline();
}
if !p.local {
let mut d = DevelopSettings::default();
d.light.exposure = (i % 7) as f64 * 0.1;
@@ -153,6 +158,47 @@ fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
}
println!(" background compaction blocking: {}", stats(blocking));
// forget the untouched Local records (folders last browsed long ago; every 7th folder
// recently; every 50th Local record rated), then compact
let folders: std::collections::BTreeSet<String> = cat.photos().filter(|p| p.local).filter_map(|p| folder_of(p)).collect();
let mut ops: Vec<Op> = folders
.iter()
.enumerate()
.map(|(k, f)| Op::SetBrowsed {
folder: f.clone(),
at: Some(if k % 7 == 0 { "2026-09-30T00:00:00" } else { "2026-06-01T00:00:00" }.into()),
})
.collect();
ops.extend(cat.photos().filter(|p| p.local && p.id.0.is_multiple_of(50)).map(|p| Op::SetRating { id: p.id, rating: 2 }).collect::<Vec<_>>());
for op in &ops {
cat.apply(op.clone())?;
}
j.append(&ops)?;
j.snapshot(&cat)?;
let before = j.stats().last_snapshot.bytes;
let t = Instant::now();
let plan = cat.forget_local_plan("2026-10-05T00:00:00", DEFAULT_FORGET_DAYS, &|_| false);
let plan_ms = ms(t);
let t = Instant::now();
let ops = plan.ops();
for op in &ops {
cat.apply(op.clone())?;
}
j.append(&ops)?;
let apply_ms = ms(t);
j.snapshot(&cat)?;
let after = j.stats().last_snapshot.bytes;
println!(
" forget Local: {} of {} local records forgotten (kept: {} recent, {} touched, {} in use); plan {plan_ms:.1} ms, apply+append {apply_ms:.1} ms; snapshot {:.1} MB -> {:.1} MB",
plan.evict.len(),
plan.local,
plan.kept_recent,
plan.kept_touched,
plan.kept_in_use,
before as f64 / 1e6,
after as f64 / 1e6
);
drop(j);
if std::env::var_os("PERSIST_BENCH_NO_LOAD").is_some() {
// peak memory of building + compacting only
+78 -5
View File
@@ -10,6 +10,12 @@
//! Loading = snapshot + replay of the records with `seq > snapshot seq`. Recovery rules:
//! - a **torn final record** (crash mid-append: truncated line or bad CRC at the end) is dropped
//! and the file is cut back to the last good record, so later appends start on a clean line;
//! - a **failed append** (disk full, a dropped network share) may leave part of its batch in the
//! file: [`Journal::append`] cuts the log back to the last whole record before returning the
//! error (or, if that fails too, before the next append), so the retry — the same ops with the
//! same `seq`s — starts on a clean line. Logs written by builds without that cut-back hold the
//! fragment glued to the first retried record on one line; the loader recognises the whole,
//! CRC-valid record at the end of such a line and skips only the fragment;
//! - records already covered by the snapshot (crash between writing the snapshot and resetting
//! the log) are skipped;
//! - a bad record **followed by good ones** is real damage: replay stops there, the log is kept
@@ -62,6 +68,9 @@ pub struct LoadReport {
pub failed: usize,
/// Bytes of a torn final record that were dropped.
pub torn_bytes: u64,
/// Partial records left mid-log by a failed append that was retried (written by builds that
/// didn't cut the log back after a failed write): skipped, the retried records were replayed.
pub torn_fragments: usize,
/// The damaged log was preserved under this name.
pub damaged: Option<String>,
/// No catalog files existed (a new library).
@@ -140,6 +149,9 @@ pub struct Journal {
pending: Option<Pending>,
/// After a failed background snapshot: don't retry before the log has this many records.
retry_at_records: u64,
/// A failed append may have left part of its records after `log_bytes`, and cutting them off
/// failed too: the next append cuts the log back to `log_bytes` first.
tail_dirty: bool,
}
/// A background snapshot in flight.
@@ -182,6 +194,18 @@ pub fn decode_record(line: &str) -> Option<(u64, Op)> {
serde_json::from_str(body).ok().map(|op| (seq, op))
}
/// A line that doesn't decode but ends with a whole record: a fragment of a failed append followed
/// by the record that retried it (the retry re-encodes the same ops with the same `seq`s). Returns
/// the fragment's length and the record, if that record continues the log (`seq <= last + 1`).
/// The record's CRC must match, so damage is never mistaken for it.
fn resync(line: &[u8], last: u64) -> Option<(usize, (u64, Op))> {
const START: &[u8] = b"{\"seq\":";
(1..line.len().saturating_sub(START.len() - 1)).filter(|&i| line[i..].starts_with(START)).find_map(|i| {
let rec = std::str::from_utf8(&line[i..]).ok().map(|l| l.trim_end_matches('\r')).and_then(decode_record)?;
(rec.0 <= last + 1).then_some((i, rec))
})
}
/// Write `catalog.snap` for the state after op `seq`, streaming the JSON into the store (no
/// whole-file string). The bytes are exactly
/// `{"format":"lightcraft-catalog","version":1,"seq":N,"catalog":<serde_json of the catalog>}\n`,
@@ -242,6 +266,7 @@ impl Journal {
stats: PersistStats::default(),
pending: None,
retry_at_records: 0,
tail_dirty: false,
};
let log = log.unwrap_or_default();
@@ -261,7 +286,17 @@ impl Journal {
good_end = next;
continue;
}
match line.and_then(decode_record) {
let mut record = line.and_then(decode_record);
if record.is_none()
&& let Some((at, r)) = resync(&log[pos..line_end], j.seq)
{
// the torn start of a failed append, then the retried records (older builds
// appended them right after the fragment)
log::warn!("catalog log: skipped a partial record ({at} bytes) left by a failed write");
report.torn_fragments += 1;
record = Some(r);
}
match record {
Some((seq, op)) if seq <= j.seq => {
// already in the snapshot
let _ = op;
@@ -301,10 +336,13 @@ impl Journal {
report.torn_bytes = (log.len() - good_end) as u64;
log::warn!("catalog log: dropped a torn final record ({} bytes)", report.torn_bytes);
j.store.truncate(LOG, good_end as u64).map_err(io)?;
j.log_bytes = good_end as u64;
} else if needs_newline {
j.store.append(LOG, b"\n").map_err(io)?;
j.log_bytes = good_end as u64 + 1;
} else {
j.log_bytes = good_end as u64;
}
j.log_bytes = good_end as u64;
}
catalog.revision = 0;
Ok((j, catalog, report))
@@ -323,7 +361,20 @@ impl Journal {
buf.push_str(&encode_record(seq, op));
buf.push('\n');
}
self.store.append(LOG, buf.as_bytes()).map_err(io)?;
if self.tail_dirty {
self.store.truncate(LOG, self.log_bytes).map_err(io)?;
self.tail_dirty = false;
}
if let Err(e) = self.store.append(LOG, buf.as_bytes()) {
// Part of the batch may be in the file. Cut it back to the last whole record, so the
// retry (same seqs) starts on a clean line instead of after a fragment that would
// read back as damage.
if let Err(t) = self.store.truncate(LOG, self.log_bytes) {
log::error!("catalog: can't cut the log back after a failed append: {t}");
self.tail_dirty = true;
}
return Err(io(e));
}
if let Some(p) = self.pending.as_mut() {
p.tail.extend_from_slice(buf.as_bytes());
p.tail_records += ops.len() as u64;
@@ -343,12 +394,33 @@ impl Journal {
/// Write a snapshot of `catalog` (which must reflect every appended op) and reset the log.
pub fn snapshot(&mut self, catalog: &Catalog) -> Result<()> {
self.snapshot_with_unlogged(catalog, 0)
}
/// [`Journal::snapshot`] of a catalog that also reflects `unlogged` ops applied after the
/// last appended one but never written to the log (their append failed). The snapshot counts
/// them (it holds the state after op `seq + unlogged`), so they are saved and must not be
/// appended afterwards. Once the snapshot is durable [`Journal::seq`] includes them, even if
/// resetting the log then fails (the error is still returned).
pub fn snapshot_with_unlogged(&mut self, catalog: &Catalog, unlogged: u64) -> Result<()> {
// never two snapshot writers; the one in flight is older, so it must land first
if let Err(e) = self.wait() {
log::warn!("catalog: background snapshot failed ({e}); writing one now");
}
let t0 = web_time::Instant::now();
let mut timing = write_snapshot(self.store.as_mut(), self.seq, catalog).map_err(io)?;
let seq = self.seq + unlogged;
let mut timing = match write_snapshot(self.store.as_mut(), seq, catalog) {
Ok(t) => t,
Err(e) => {
// nothing lost (the log is whole); don't retry on every append
self.stats.failed_snapshots += 1;
self.retry_at_records = self.log_records + self.policy.max_records;
return Err(io(e));
}
};
// the snapshot is durable: it holds every op up to `seq`
self.seq = seq;
self.snapshot_seq = seq;
let t2 = web_time::Instant::now();
// A crash here leaves old records in the log; they are skipped by seq on load.
self.store.write_atomic(LOG, b"").map_err(io)?;
@@ -357,10 +429,10 @@ impl Journal {
timing.blocking_ms = timing.total_ms;
timing.records = self.log_records;
self.record_snapshot(timing);
self.snapshot_seq = self.seq;
self.log_records = 0;
self.log_bytes = 0;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
@@ -451,6 +523,7 @@ impl Journal {
self.log_records = p.tail_records;
self.log_bytes = p.tail.len() as u64;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
+24
View File
@@ -11,9 +11,11 @@
pub mod dates;
pub mod journal;
pub mod keywords;
pub mod local;
pub mod model;
pub mod query;
pub mod rules;
pub mod safe_file;
pub mod stacks;
pub mod store;
@@ -24,6 +26,7 @@ pub use dates::{DateRun, GroupBy};
pub use journal::{Journal, LoadReport, PersistStats, SnapshotPolicy, SnapshotTiming};
pub use keywords::KeywordNode;
use lightcraft_develop::DevelopSettings;
pub use local::{DEFAULT_FORGET_DAYS, ForgetPlan, folder_of};
pub use model::*;
pub use query::{DateGroup, Filter, RatingOp, Sort, SortKey};
pub use rules::{Match, Rule, RuleSet};
@@ -190,6 +193,12 @@ pub enum Op {
label: ColorLabel,
name: Option<String>,
},
/// When a Local folder was last browsed (ISO 8601; `None` = forget the time). Not an undo
/// step: it drives forgetting untouched Local records (see [`local`]).
SetBrowsed {
folder: String,
at: Option<String>,
},
/// Several ops as one step (undo applies the inverses in reverse).
Batch {
ops: Vec<Op>,
@@ -209,6 +218,9 @@ pub struct Catalog {
/// Custom colour label names.
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
label_names: BTreeMap<ColorLabel, String>,
/// When each Local folder was last browsed (folder path → ISO 8601), see [`local`].
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
browsed: BTreeMap<String, String>,
/// Increments on every applied op.
#[serde(skip)]
pub revision: u64,
@@ -547,6 +559,14 @@ impl Catalog {
};
Op::SetLabelName { label, name: old }
}
Op::SetBrowsed { folder, at } => {
let folder = crate::query::folder_key(&folder);
let old = match at {
Some(t) => self.browsed.insert(folder.clone(), t),
None => self.browsed.remove(&folder),
};
Op::SetBrowsed { folder, at: old }
}
Op::Batch { ops } => {
let mut inverses = Vec::with_capacity(ops.len());
for op in ops {
@@ -625,3 +645,7 @@ mod tests;
mod tests_background;
#[cfg(test)]
mod tests_journal;
#[cfg(test)]
mod tests_local;
#[cfg(test)]
mod tests_torn_append;
+304
View File
@@ -0,0 +1,304 @@
//! Forgetting untouched Local browse records.
//!
//! Browsing a folder (Local) catalogues its photos as `local` records so they can be shown,
//! rated and edited in place. Most are only ever looked at, and on a large disk they can be most
//! of the catalog. A record is **forgotten** (removed from the catalog, never from the disk —
//! files and XMP sidecars are not touched) when all of these hold:
//!
//! - it is a Local record ([`Photo::local`]) of a file, not deleted, not a virtual copy;
//! - its folder was last browsed more than `days` ago ([`Catalog::last_browsed`]). A folder with
//! no recorded time (a catalog from before this existed) counts as browsed *now*: it is stamped
//! with the current time and nothing in it is forgotten this time;
//! - it is **untouched** ([`Photo::untouched_local`]): nothing about it changed since it was
//! browsed — see below;
//! - nothing refers to it: no album holds it or shows it as cover, no stack holds it, no virtual
//! copy was made of it; and the caller (the engine) says it isn't in use (visible, selected,
//! in the undo history, has a smart preview…).
//!
//! **Untouched.** A record catalogued by a browse carries a fingerprint of its whole state at
//! that moment ([`Photo::local_baseline`]). It is untouched while its state still hashes to that
//! fingerprint, i.e. *any* change since — develop settings or history, versions, rating, flag,
//! label, any metadata (title, caption, keywords, location, copyright…), capture time, crop,
//! rename or relink, culling scores, file content — keeps it. Records without a fingerprint
//! (browsed before fingerprints existed) are judged by their data alone and kept unless every
//! user-editable field is still empty / default: no rating, flag or label; develop settings equal
//! to the import defaults with no history, versions or edit time; no title, caption, alt text,
//! description, keywords, creator, copyright, rights, location or GPS; no culling scores.
//! When in doubt a record is kept.
//!
//! Browsing the folder again catalogues its files afresh (new records, new fingerprints).
use std::collections::{BTreeMap, BTreeSet, HashSet};
use serde::Serialize;
use crate::{Catalog, Op, Photo, PhotoId, Source};
/// Default for how long a Local folder may go unbrowsed before its untouched records are
/// forgotten (days).
pub const DEFAULT_FORGET_DAYS: u32 = 30;
/// Records removed per logged batch (keeps log lines a reasonable size).
const BATCH: usize = 2000;
/// 64-bit FNV-1a over everything written to it (stable across builds and platforms, unlike
/// `std`'s hasher), so a photo is hashed as it serialises, without a buffer.
struct Fnv(u64);
impl std::io::Write for Fnv {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
for b in buf {
self.0 ^= u64::from(*b);
self.0 = self.0.wrapping_mul(0x0100_0000_01b3);
}
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// The folder a file-backed photo is in (`None` for demo photos).
pub fn folder_of(p: &Photo) -> Option<String> {
match &p.source {
// `/` and `\` are both separators whatever the platform (a catalog can move between them)
Source::File { path } => {
let key = crate::query::folder_key(path);
key.rsplit_once('/').map(|(dir, _)| if dir.is_empty() { "/".to_string() } else { dir.to_string() })
}
Source::Demo { .. } => None,
}
}
impl Photo {
/// A hash of everything about the photo except [`Photo::local_baseline`] itself.
pub fn fingerprint(&self) -> u64 {
let mut h = Fnv(0xcbf2_9ce4_8422_2325);
let r = if self.local_baseline.is_some() {
let mut p = self.clone();
p.local_baseline = None;
serde_json::to_writer(&mut h, &p)
} else {
serde_json::to_writer(&mut h, self)
};
// (serialising a photo can't fail; if it ever did, no baseline would match: kept)
if r.is_err() { 0 } else { h.0 }
}
/// Record the current state as the browse baseline (done when a browse catalogues it).
pub fn set_local_baseline(&mut self) {
self.local_baseline = None;
self.local_baseline = Some(self.fingerprint());
}
/// A Local record nothing was changed on since it was browsed (see the module docs). Says
/// nothing about albums, stacks or virtual copies referring to it.
pub fn untouched_local(&self) -> bool {
if !self.local || self.deleted || self.copy_of.is_some() || self.copy_name.is_some() || folder_of(self).is_none() {
return false;
}
match self.local_baseline {
Some(b) => self.fingerprint() == b,
None => self.looks_unedited(),
}
}
/// Every user-editable field is still empty / default (records without a baseline).
fn looks_unedited(&self) -> bool {
let m = &self.meta;
self.rating == 0
&& self.flag == crate::Flag::None
&& self.label.is_none()
&& self.edited.is_none()
&& self.history.is_empty()
&& self.versions.is_empty()
&& *self.develop == self.import_defaults()
&& self.analysis.is_none()
&& m.title.is_empty()
&& m.caption.is_empty()
&& m.alt_text.is_empty()
&& m.extended_description.is_empty()
&& m.keywords.is_empty()
&& m.creator.is_empty()
&& m.copyright.is_empty()
&& m.copyright_status.is_unknown()
&& m.usage_terms.is_empty()
&& m.copyright_url.is_empty()
&& m.location.is_empty()
&& m.city.is_empty()
&& m.state.is_empty()
&& m.country.is_empty()
&& m.gps.is_none()
}
}
/// What forgetting Local records would do / did.
#[derive(Clone, Debug, Default, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ForgetPlan {
/// Local records looked at.
pub local: usize,
/// Records to forget.
pub evict: Vec<PhotoId>,
/// Kept: their folder was browsed within the period (or has no time yet).
pub kept_recent: usize,
/// Kept: changed by the user since they were browsed.
pub kept_touched: usize,
/// Kept: in an album, a stack, the source of a virtual copy, or in use.
pub kept_in_use: usize,
/// Folders with Local records but no last-browsed time: stamped now.
pub stamp: Vec<String>,
/// Folders whose last Local record goes: their time is dropped.
pub unstamp: Vec<String>,
/// The time stamps are given.
#[serde(skip)]
pub now: String,
}
impl ForgetPlan {
/// The ops that carry the plan out (empty when there is nothing to do).
pub fn ops(&self) -> Vec<Op> {
let mut ops: Vec<Op> = self.stamp.iter().map(|f| Op::SetBrowsed { folder: f.clone(), at: Some(self.now.clone()) }).collect();
ops.extend(self.evict.chunks(BATCH).map(|c| Op::Batch { ops: c.iter().map(|id| Op::RemovePhoto { id: *id }).collect() }));
ops.extend(self.unstamp.iter().map(|f| Op::SetBrowsed { folder: f.clone(), at: None }));
ops
}
}
impl Catalog {
/// When `folder` was last browsed (ISO 8601), if known.
pub fn last_browsed(&self, folder: &str) -> Option<&str> {
self.browsed.get(&crate::query::folder_key(folder)).map(String::as_str)
}
/// Folders with a last-browsed time.
pub fn browsed_folders(&self) -> &BTreeMap<String, String> {
&self.browsed
}
/// Plan forgetting untouched Local records whose folder wasn't browsed for `days` days (as of
/// `now`, ISO 8601). `in_use` keeps records the caller still needs. `days == 0` (never) or an
/// unreadable `now` forgets nothing. See the module docs.
pub fn forget_local_plan(&self, now: &str, days: u32, in_use: &dyn Fn(&Photo) -> bool) -> ForgetPlan {
let mut plan = ForgetPlan { now: now.to_string(), ..Default::default() };
let now_s = crate::stacks::iso_seconds(now);
let max_age = i64::from(days) * 86_400;
let mut referenced: HashSet<PhotoId> = HashSet::new();
for a in self.albums.values() {
referenced.extend(a.photos.iter().copied());
referenced.extend(a.cover);
}
for s in self.stacks.values() {
referenced.extend(s.photos.iter().copied());
}
referenced.extend(self.photos.values().filter_map(|p| p.copy_of));
// the fingerprints are the costly part: work them out for the candidates in parallel
let candidates: Vec<&Photo> = self
.photos
.values()
.filter(|p| p.local && !referenced.contains(&p.id))
.filter(|p| folder_of(p).and_then(|f| self.browsed.get(&f)).is_some_and(|at| is_old(at, now_s, days, max_age)))
.map(|p| &**p)
.collect();
let untouched: HashSet<PhotoId> = untouched_of(&candidates);
let mut stamp = BTreeSet::new();
let mut remaining: BTreeSet<String> = BTreeSet::new();
for p in self.photos.values().filter(|p| p.local) {
plan.local += 1;
let Some(folder) = folder_of(p) else {
plan.kept_in_use += 1;
continue;
};
let recent = match (self.browsed.get(&folder), now_s) {
(None, _) => {
stamp.insert(folder.clone());
true
}
(Some(at), _) => !is_old(at, now_s, days, max_age),
};
let keep = if recent {
plan.kept_recent += 1;
true
} else if referenced.contains(&p.id) || in_use(p) {
plan.kept_in_use += 1;
true
} else if !untouched.contains(&p.id) {
plan.kept_touched += 1;
true
} else {
plan.evict.push(p.id);
false
};
if keep {
remaining.insert(folder);
}
}
plan.stamp = stamp.into_iter().collect();
plan.unstamp = self.browsed.keys().filter(|f| !remaining.contains(*f)).cloned().collect();
plan
}
}
/// A folder browsed at `at` is past the period (never when `days` is 0 or a time is unreadable).
fn is_old(at: &str, now_s: Option<i64>, days: u32, max_age: i64) -> bool {
days > 0 && matches!((now_s, crate::stacks::iso_seconds(at)), (Some(n), Some(a)) if n - a >= max_age)
}
/// The untouched ones among `photos` ([`Photo::untouched_local`]), checked on several threads
/// where there are many.
fn untouched_of(photos: &[&Photo]) -> HashSet<PhotoId> {
let check = |chunk: &[&Photo]| chunk.iter().filter(|p| p.untouched_local()).map(|p| p.id).collect::<Vec<_>>();
#[cfg(not(target_arch = "wasm32"))]
{
let threads = std::thread::available_parallelism().map_or(1, |n| n.get()).min(8);
if threads > 1 && photos.len() >= 256 {
let size = photos.len().div_ceil(threads);
return std::thread::scope(|sc| {
let jobs: Vec<_> = photos.chunks(size).map(|c| sc.spawn(move || check(c))).collect();
// a job that panicked (it can't) leaves its photos out: they are kept
jobs.into_iter().filter_map(|j| j.join().ok()).flatten().collect()
});
}
}
check(photos).into_iter().collect()
}
impl Op {
/// Every photo the op refers to.
pub fn photo_ids(&self, out: &mut impl FnMut(PhotoId)) {
match self {
Op::AddPhoto { photo } => out(photo.id),
Op::RemovePhoto { id }
| Op::SetRating { id, .. }
| Op::SetFlag { id, .. }
| Op::SetLabel { id, .. }
| Op::SetDevelop { id, .. }
| Op::SetMeta { id, .. }
| Op::SetDeleted { id, .. }
| Op::SetLocal { id, .. }
| Op::SetVersions { id, .. }
| Op::SetHistory { id, .. }
| Op::PushHistory { id, .. }
| Op::SetCaptured { id, .. }
| Op::SetAnalysis { id, .. }
| Op::SetFile { id, .. }
| Op::Relink { id, .. }
| Op::SetContent { id, .. } => out(*id),
Op::AddAlbum { album } => {
album.photos.iter().for_each(|p| out(*p));
album.cover.into_iter().for_each(&mut *out);
}
Op::SetAlbumPhotos { photos, .. } | Op::SetStack { photos, .. } => photos.iter().for_each(|p| out(*p)),
Op::SetAlbumCover { cover, .. } => cover.iter().for_each(|p| out(*p)),
Op::AddStack { stack } => stack.photos.iter().for_each(|p| out(*p)),
Op::Batch { ops } => ops.iter().for_each(|o| o.photo_ids(out)),
Op::RemoveAlbum { .. }
| Op::RenameAlbum { .. }
| Op::MoveAlbum { .. }
| Op::SetAlbumRules { .. }
| Op::RemoveStack { .. }
| Op::SetLabelName { .. }
| Op::SetBrowsed { .. } => {}
}
}
}
+5
View File
@@ -272,6 +272,10 @@ pub struct Photo {
/// picture style baked in — so it is treated as a rendered (non-raw) source.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preview_only: Option<String>,
/// A Local record: the fingerprint of its state when the browse catalogued it, to tell
/// whether the user changed anything since (see [`crate::local`]).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub local_baseline: Option<u64>,
}
/// What assisted culling measured on a photo.
@@ -319,6 +323,7 @@ impl Photo {
import_look: None,
analysis: None,
preview_only: None,
local_baseline: None,
}
}
/// A raw file developed from its sensor data: not a rendered image, and not a raw shown from
+317
View File
@@ -0,0 +1,317 @@
//! Crash-safe writes of user-visible files: exports, renders, DNG conversions, smart previews and
//! XMP sidecars all go through here (the catalog's own files use [`crate::store::FsStore`]).
//!
//! - [`write_atomic`] replaces a file so that after a crash, a full disk or an unplugged drive
//! either the old or the complete new content is there — never a truncated mix. The content
//! goes to a new temp file in the same folder (unique name, `create_new`), is synced to disk and
//! renamed over the target; then the folder is synced (POSIX). Any failure removes the temp
//! file and leaves the target as it was.
//! - [`write_new`] / [`write_new_unique`] write a file that must not exist yet: the same temp file
//! and sync, plus a read-back check, and the file then appears under its final name **without
//! replacing anything** (a hard link, which fails on a taken name; a check-then-rename where the
//! volume has no hard links).
//! - [`same_file`] tells whether two paths name the same file on disk (device + inode on Unix,
//! resolved paths elsewhere), for "is this target an original?" checks.
use std::cell::Cell;
use std::fs::{self, File, OpenOptions};
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
thread_local! {
static FAIL_AFTER: Cell<Option<u64>> = const { Cell::new(None) };
}
/// Fault injection for tests (in any crate): while the returned guard lives, every write made by
/// this module **on the calling thread** fails once `bytes` bytes of a file were written — as a
/// full disk or an unplugged drive would.
#[doc(hidden)]
pub fn fail_writes_after(bytes: u64) -> FaultGuard {
FAIL_AFTER.with(|c| c.set(Some(bytes)));
FaultGuard(())
}
/// Ends a [`fail_writes_after`] when dropped.
#[doc(hidden)]
pub struct FaultGuard(());
impl Drop for FaultGuard {
fn drop(&mut self) {
FAIL_AFTER.with(|c| c.set(None));
}
}
/// Counts what is written; fails past an injected limit.
struct Limited<W> {
inner: W,
n: u64,
limit: Option<u64>,
}
impl<W: Write> Write for Limited<W> {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let buf = match self.limit {
Some(l) => {
let room = usize::try_from(l.saturating_sub(self.n)).unwrap_or(usize::MAX);
if room == 0 && !buf.is_empty() {
return Err(io::Error::other("write failed (injected fault)"));
}
buf.get(..room.min(buf.len())).unwrap_or(buf)
}
None => buf,
};
let k = self.inner.write(buf)?;
self.n += k as u64;
Ok(k)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
fn parent_of(path: &Path) -> &Path {
path.parent().filter(|d| !d.as_os_str().is_empty()).unwrap_or(Path::new("."))
}
/// Make a rename or a new name in `dir` durable (POSIX). Opening a directory fails on Windows,
/// where NTFS journals the rename itself, so errors are ignored.
pub fn sync_dir(dir: &Path) {
if let Ok(d) = File::open(dir) {
let _ = d.sync_all();
}
}
/// A new, empty temp file next to `path` (hidden, unique in this process and folder).
fn create_temp(path: &Path) -> io::Result<(PathBuf, File)> {
static N: AtomicU64 = AtomicU64::new(0);
if path.file_name().is_none() {
return Err(io::Error::new(io::ErrorKind::InvalidInput, format!("{} is not a file path", path.display())));
}
let dir = parent_of(path);
for _ in 0..1000 {
let tmp = dir.join(format!(".lightcraft-{}-{}.tmp", std::process::id(), N.fetch_add(1, Ordering::Relaxed)));
match OpenOptions::new().write(true).create_new(true).open(&tmp) {
Ok(f) => return Ok((tmp, f)),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("no free temp file name in {}", dir.display())))
}
/// Write a temp file next to `path` with what `fill` writes, synced to disk. On failure the temp
/// file is removed. Returns its path and length.
fn write_temp(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<(PathBuf, u64)> {
let (tmp, file) = create_temp(path)?;
let r = (|| {
let limit = FAIL_AFTER.with(Cell::get);
let mut w = Limited { inner: io::BufWriter::with_capacity(1 << 20, file), n: 0, limit };
fill(&mut w)?;
w.flush()?;
let n = w.n;
let f = w.inner.into_inner().map_err(|e| e.into_error())?;
f.sync_all()?;
Ok(n)
})();
match r {
Ok(n) => Ok((tmp, n)),
Err(e) => {
let _ = fs::remove_file(&tmp);
Err(e)
}
}
}
/// Replace (or create) `path` with `data` atomically and durably; see the module docs. The
/// folder must exist.
pub fn write_atomic(path: &Path, data: &[u8]) -> io::Result<()> {
write_atomic_with(path, &mut |w| w.write_all(data)).map(|_| ())
}
/// [`write_atomic`] with streamed content: `fill` writes it (an error aborts and leaves the old
/// file). Returns the length written.
pub fn write_atomic_with(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<u64> {
let (tmp, n) = write_temp(path, fill)?;
if let Err(e) = fs::rename(&tmp, path) {
let _ = fs::remove_file(&tmp);
return Err(e);
}
sync_dir(parent_of(path));
Ok(n)
}
/// Write `data` as the new file `path`: never replaces an existing file (`AlreadyExists`), and
/// the file only appears once it is complete, synced and read back identical.
pub fn write_new(path: &Path, data: &[u8]) -> io::Result<()> {
write_new_unique(&mut std::iter::once(path.to_path_buf()), data).map(|_| ())
}
/// [`write_new`] under the first free name of `candidates` (all in the same folder, e.g.
/// `IMG_1.dng`, `IMG_1-2.dng`, …): the file is written once and then given the first name that
/// is not taken. Returns that name; `AlreadyExists` when every candidate is taken.
pub fn write_new_unique(candidates: &mut dyn Iterator<Item = PathBuf>, data: &[u8]) -> io::Result<PathBuf> {
let mut candidates = candidates.peekable();
let Some(first) = candidates.peek().cloned() else {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "no file name to write"));
};
let (tmp, _) = write_temp(&first, &mut |w| w.write_all(data))?;
let r = (|| {
if !same_content(&tmp, data)? {
return Err(io::Error::other("the written file reads back different (failing drive or connection?)"));
}
for path in candidates.by_ref().take(100_000) {
match publish_new(&tmp, &path) {
Ok(()) => {
sync_dir(parent_of(&path));
return Ok(path);
}
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("{}: no free file name", first.display())))
})();
// after a hard link the temp name is a second name of the new file; after a rename it's gone
let _ = fs::remove_file(&tmp);
r
}
/// Give the complete temp file `tmp` the name `path`, unless that is taken.
fn publish_new(tmp: &Path, path: &Path) -> io::Result<()> {
match fs::hard_link(tmp, path) {
Ok(()) => Ok(()),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => Err(e),
// no hard links on this volume (FAT/exFAT, some shares): check, then rename
Err(_) => {
if fs::symlink_metadata(path).is_ok() {
return Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("{} exists", path.display())));
}
fs::rename(tmp, path)
}
}
}
/// Does the file at `path` hold exactly `data`?
fn same_content(path: &Path, data: &[u8]) -> io::Result<bool> {
use std::io::Read;
let mut f = File::open(path)?;
if f.metadata()?.len() != data.len() as u64 {
return Ok(false);
}
let mut buf = vec![0u8; 1 << 20];
let mut at = 0usize;
loop {
let k = match f.read(&mut buf) {
Ok(0) => return Ok(at == data.len()),
Ok(k) => k,
Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
Err(e) => return Err(e),
};
if data.get(at..at + k) != buf.get(..k) {
return Ok(false);
}
at += k;
}
}
/// Do `a` and `b` name the same existing file (through links, `..`, case-insensitive names)?
pub fn same_file(a: &Path, b: &Path) -> bool {
let (Ok(ma), Ok(mb)) = (fs::metadata(a), fs::metadata(b)) else { return false };
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
ma.dev() == mb.dev() && ma.ino() == mb.ino()
}
#[cfg(not(unix))]
{
if ma.len() != mb.len() {
return false;
}
match (fs::canonicalize(a), fs::canonicalize(b)) {
(Ok(x), Ok(y)) => x == y || x.to_string_lossy().to_lowercase() == y.to_string_lossy().to_lowercase(),
_ => false,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn temp(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-safe-file-{tag}-{}", std::process::id()));
let _ = fs::remove_dir_all(&d);
fs::create_dir_all(&d).unwrap();
d
}
fn names(dir: &Path) -> Vec<String> {
let mut v: Vec<String> = fs::read_dir(dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
v.sort();
v
}
#[test]
fn atomic_write_replaces_and_leaves_no_temp() {
let d = temp("replace");
let p = d.join("out.jpg");
write_atomic(&p, b"one").unwrap();
write_atomic(&p, b"two two").unwrap();
assert_eq!(fs::read(&p).unwrap(), b"two two");
assert_eq!(names(&d), vec!["out.jpg"]);
let _ = fs::remove_dir_all(&d);
}
#[test]
fn a_failure_mid_write_keeps_the_old_file() {
let d = temp("midwrite");
let p = d.join("photo.jpg");
fs::write(&p, b"the original bytes").unwrap();
{
let _f = fail_writes_after(4);
let e = write_atomic(&p, &[7u8; 10_000]).unwrap_err();
assert!(e.to_string().contains("injected"), "{e}");
assert!(write_new(&d.join("new.dng"), &[1u8; 100]).is_err());
}
assert_eq!(fs::read(&p).unwrap(), b"the original bytes", "the target is untouched");
assert_eq!(names(&d), vec!["photo.jpg"], "no temp file and no partial new file");
// a fill error aborts the same way
let r = write_atomic_with(&p, &mut |w| {
w.write_all(b"half")?;
Err(io::Error::other("encoder failed"))
});
assert!(r.is_err());
assert_eq!(fs::read(&p).unwrap(), b"the original bytes");
assert_eq!(names(&d), vec!["photo.jpg"]);
let _ = fs::remove_dir_all(&d);
}
#[test]
fn new_files_never_replace_anything() {
let d = temp("new");
let p = d.join("a.dng");
fs::write(&p, b"keep").unwrap();
assert_eq!(write_new(&p, b"other").unwrap_err().kind(), io::ErrorKind::AlreadyExists);
assert_eq!(fs::read(&p).unwrap(), b"keep");
let mut c = (1..).map(|k| if k == 1 { d.join("a.dng") } else { d.join(format!("a-{k}.dng")) });
let got = write_new_unique(&mut c, b"fresh").unwrap();
assert_eq!(got, d.join("a-2.dng"));
assert_eq!(fs::read(&got).unwrap(), b"fresh");
assert_eq!(names(&d), vec!["a-2.dng", "a.dng"], "the temp file is gone");
let _ = fs::remove_dir_all(&d);
}
#[test]
fn same_file_sees_through_other_spellings() {
let d = temp("same");
let p = d.join("x.jpg");
fs::write(&p, b"x").unwrap();
fs::write(d.join("y.jpg"), b"x").unwrap();
fs::create_dir_all(d.join("sub")).unwrap();
assert!(same_file(&p, &d.join("sub/../x.jpg")));
assert!(!same_file(&p, &d.join("y.jpg")));
assert!(!same_file(&p, &d.join("missing.jpg")));
let _ = fs::remove_dir_all(&d);
}
}
+19 -5
View File
@@ -28,9 +28,12 @@ pub trait Store: Send {
self.write_atomic(name, &buf)?;
Ok(buf.len() as u64)
}
/// Append and make durable (fsync) before returning.
/// Append and make durable (fsync) before returning. On an error, part of `data` may have
/// been written (a full disk, a dropped network share): the journal cuts the file back to its
/// previous length with [`Store::truncate`] before appending again.
fn append(&mut self, name: &str, data: &[u8]) -> io::Result<()>;
/// Cut a file to `len` bytes (drop a torn tail before appending again).
/// Cut a file to `len` bytes (drop a torn tail before appending again). A missing file is
/// left missing.
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()>;
/// Human-readable location (diagnostics).
fn describe(&self) -> String;
@@ -132,15 +135,26 @@ impl Store for FsStore {
self.appender = Some((name.to_string(), f));
}
let Some((_, f)) = self.appender.as_mut() else { return Err(io::Error::other("appender not open")) };
f.write_all(data)?;
f.sync_data()
let written = f.write_all(data).and_then(|()| f.sync_data());
if written.is_err() {
// The handle may be dead (a network share that reconnected, a removed volume): the next
// attempt reopens the file. Part of `data` may be in the file; the caller cuts it back
// ([`Store::truncate`]) before appending again.
self.appender = None;
}
written
}
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()> {
if self.appender.as_ref().is_some_and(|(n, _)| n == name) {
self.appender = None;
}
let f = std::fs::OpenOptions::new().write(true).open(self.path(name))?;
let f = match std::fs::OpenOptions::new().write(true).open(self.path(name)) {
Ok(f) => f,
// nothing to cut (an append that failed before creating the file)
Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()),
Err(e) => return Err(e),
};
f.set_len(len)?;
f.sync_all()
}
+267
View File
@@ -0,0 +1,267 @@
//! Forgetting untouched Local browse records (see [`crate::local`]).
use std::sync::Arc;
use lightcraft_develop::DevelopSettings;
use crate::*;
const OLD: &str = "2026-08-01T10:00:00";
const NOW: &str = "2026-10-05T10:00:00";
/// A Local record of `/pics/<folder>/IMG_<i>.JPG` as a browse catalogues it (with its baseline).
fn local(c: &mut Catalog, folder: &str, i: u32) -> PhotoId {
let id = c.alloc_photo_id();
let mut p = Photo::new(id, Source::File { path: format!("/pics/{folder}/IMG_{i}.JPG") }, &format!("IMG_{i}.JPG"), "JPEG", 60, 40, OLD);
p.meta.camera = "Cam".into();
p.meta.creator = "From EXIF".into(); // file metadata is part of the baseline, not a user change
p.focal_and_wb_for_test();
p.local = true;
p.set_local_baseline();
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
id
}
impl Photo {
fn focal_and_wb_for_test(&mut self) {
self.meta.focal_mm = Some(35.3);
self.as_shot_wb = Some((5123.4, 3.1));
}
}
fn browsed(c: &mut Catalog, folder: &str, at: &str) {
c.apply(Op::SetBrowsed { folder: format!("/pics/{folder}"), at: Some(at.into()) }).unwrap();
}
fn plan(c: &Catalog, days: u32) -> ForgetPlan {
c.forget_local_plan(NOW, days, &|_| false)
}
fn run(c: &mut Catalog, days: u32) -> ForgetPlan {
let p = plan(c, days);
for op in p.ops() {
c.apply(op).unwrap();
}
p
}
#[test]
fn untouched_records_of_old_folders_are_forgotten() {
let mut c = Catalog::new();
let old: Vec<_> = (0..5).map(|i| local(&mut c, "old", i)).collect();
let recent: Vec<_> = (0..3).map(|i| local(&mut c, "recent", i)).collect();
browsed(&mut c, "old", OLD);
browsed(&mut c, "recent", "2026-10-01T09:00:00");
// a library photo in the old folder is never a candidate
let lib = c.alloc_photo_id();
let mut p = Photo::new(lib, Source::File { path: "/pics/old/LIB.JPG".into() }, "LIB.JPG", "JPEG", 6, 4, OLD);
p.set_local_baseline();
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
let dry = plan(&c, 30);
assert_eq!((dry.local, dry.evict.len(), dry.kept_recent), (8, 5, 3), "{dry:?}");
assert_eq!(dry.evict, old);
assert_eq!(c.len(), 9, "a plan changes nothing");
let done = run(&mut c, 30);
assert_eq!(done.evict.len(), 5);
assert!(old.iter().all(|id| c.photo(*id).is_none()));
assert!(recent.iter().all(|id| c.photo(*id).is_some()));
assert!(c.photo(lib).is_some());
// the folder has no Local records left: its time goes
assert_eq!(done.unstamp, vec!["/pics/old".to_string()]);
assert_eq!(c.last_browsed("/pics/old"), None);
assert_eq!(c.last_browsed("/pics/recent/"), Some("2026-10-01T09:00:00"));
// 0 days = never
browsed(&mut c, "recent", OLD);
assert!(plan(&c, 0).evict.is_empty());
assert_eq!(plan(&c, 30).evict.len(), 3);
}
#[test]
fn any_user_change_keeps_a_record() {
type Change = Box<dyn Fn(&mut Catalog, PhotoId)>;
let set = |f: fn(&mut Photo)| -> Change {
Box::new(move |c: &mut Catalog, id: PhotoId| {
let mut p = (**c.photo(id).unwrap()).clone();
f(&mut p);
c.apply(Op::RemovePhoto { id }).unwrap();
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
})
};
let op = |f: fn(PhotoId) -> Op| -> Change { Box::new(move |c: &mut Catalog, id: PhotoId| drop(c.apply(f(id)).unwrap())) };
let edited = || {
let mut d = DevelopSettings::default();
d.light.exposure = 0.3;
Arc::new(d)
};
let changes: Vec<(&str, Change)> = vec![
("rating", op(|id| Op::SetRating { id, rating: 3 })),
("flag", op(|id| Op::SetFlag { id, flag: Flag::Reject })),
("label", op(|id| Op::SetLabel { id, label: Some(ColorLabel::Red) })),
("develop", Box::new(move |c, id| drop(c.apply(Op::SetDevelop { id, settings: edited(), label: "Exposure".into(), edited: None }).unwrap()))),
("crop", set(|p| Arc::make_mut(&mut p.develop).crop.flip_h = true)),
(
"history only",
Box::new(|c, id| {
let step = HistoryStep { label: "Reset".into(), settings: Arc::new(DevelopSettings::default()) };
drop(c.apply(Op::PushHistory { id, step }).unwrap())
}),
),
("versions", set(|p| p.versions.push(Version { name: "v".into(), created: NOW.into(), settings: p.develop.clone(), auto: false }))),
("title", set(|p| p.meta.title = "T".into())),
("caption", set(|p| p.meta.caption = "C".into())),
("keywords", set(|p| p.meta.keywords.push("k".into()))),
("location", set(|p| p.meta.city = "Oslo".into())),
("gps", set(|p| p.meta.gps = Some((1.0, 2.0)))),
("copyright", set(|p| p.meta.copyright = "© me".into())),
("creator edit", set(|p| p.meta.creator = "Me".into())),
("capture time", op(|id| Op::SetCaptured { id, captured: Some("2020-01-01T00:00:00".into()) })),
("rename", set(|p| p.file_name = "renamed.JPG".into())),
("culling", op(|id| Op::SetAnalysis { id, analysis: Some(Analysis::default()) })),
("deleted", op(|id| Op::SetDeleted { id, deleted: true })),
("pick", op(|id| Op::SetFlag { id, flag: Flag::Pick })),
(
"album",
Box::new(|c, id| {
let a = c.alloc_album_id();
let mut al = Album::new(a, "A");
al.photos.push(id);
drop(c.apply(Op::AddAlbum { album: al }).unwrap())
}),
),
(
"stack",
Box::new(|c, id| {
let other = local(c, "elsewhere", 99);
let s = c.alloc_stack_id();
drop(c.apply(Op::AddStack { stack: Stack { id: s, photos: vec![id, other], collapsed: false } }).unwrap())
}),
),
(
"virtual copy",
Box::new(|c, id| {
let mut v = (**c.photo(id).unwrap()).clone();
v.id = c.alloc_photo_id();
v.copy_of = Some(id);
v.copy_name = Some("Copy 1".into());
v.local = false;
drop(c.apply(Op::AddPhoto { photo: Box::new(v) }).unwrap())
}),
),
];
for (what, change) in changes {
let mut c = Catalog::new();
let id = local(&mut c, "old", 1);
let control = local(&mut c, "old", 2);
browsed(&mut c, "old", OLD);
change(&mut c, id);
let p = plan(&c, 30);
assert_eq!(p.evict, vec![control], "{what}: {p:?}");
assert_eq!(p.kept_touched + p.kept_in_use, 1, "{what}: {p:?}");
// the stamp stays: a record in the folder remains
assert!(p.unstamp.is_empty(), "{what}");
}
// a change undone (back to the browse state) is untouched again
let mut c = Catalog::new();
let id = local(&mut c, "old", 1);
browsed(&mut c, "old", OLD);
let inv = c.apply(Op::SetRating { id, rating: 2 }).unwrap();
assert!(plan(&c, 30).evict.is_empty());
c.apply(inv).unwrap();
assert_eq!(plan(&c, 30).evict, vec![id]);
}
/// Catalogs from before last-browsed times (and baselines): nothing is forgotten on the first
/// run — the folders are stamped now — and baseline-less records are judged by their data.
#[test]
fn upgrade_without_times_forgets_nothing_then_counts_from_then() {
let mut c = Catalog::new();
let a = local(&mut c, "a", 1);
// legacy records: no baseline
let legacy = |c: &mut Catalog, i: u32, f: fn(&mut Photo)| {
let id = c.alloc_photo_id();
let mut p = Photo::new(id, Source::File { path: format!("/pics/a/L{i}.JPG") }, "L.JPG", "JPEG", 6, 4, OLD);
p.meta.camera = "Cam".into();
p.local = true;
f(&mut p);
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
id
};
let clean = legacy(&mut c, 1, |_| {});
let rated = legacy(&mut c, 2, |p| p.rating = 1);
let with_creator = legacy(&mut c, 3, |p| p.meta.creator = "maybe the user".into());
let p = run(&mut c, 30);
assert!(p.evict.is_empty(), "{p:?}");
assert_eq!(p.stamp, vec!["/pics/a".to_string()]);
assert_eq!(c.last_browsed("/pics/a"), Some(NOW));
// 30+ days later
let later = c.forget_local_plan("2026-11-06T10:00:00", 30, &|_| false);
let mut evict = later.evict.clone();
evict.sort();
assert_eq!(evict, vec![a, clean]);
assert_eq!(later.kept_touched, 2, "{rated:?} {with_creator:?}");
}
#[test]
fn in_use_records_are_kept() {
let mut c = Catalog::new();
let a = local(&mut c, "old", 1);
let b = local(&mut c, "old", 2);
browsed(&mut c, "old", OLD);
let p = c.forget_local_plan(NOW, 30, &|p| p.id == a);
assert_eq!((p.evict.clone(), p.kept_in_use), (vec![b], 1));
}
/// The ops are journaled: a crash right after forgetting reloads the same catalog, and a
/// fingerprint survives the snapshot round trip (a reloaded record is still untouched).
#[test]
fn forgetting_is_journaled_and_replays() {
let m = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let mut ops = Vec::new();
for i in 0..10 {
local(&mut c, if i % 2 == 0 { "old" } else { "new" }, i);
}
// log what was built so far: the ops are the records as added
for p in c.photos() {
ops.push(Op::AddPhoto { photo: Box::new((**p).clone()) });
}
ops.push(Op::SetBrowsed { folder: "/pics/old".into(), at: Some(OLD.into()) });
ops.push(Op::SetBrowsed { folder: "/pics/new".into(), at: Some(NOW.into()) });
browsed(&mut c, "old", OLD);
browsed(&mut c, "new", NOW);
j.append(&ops).unwrap();
j.snapshot(&c).unwrap();
drop(j);
// reload from the snapshot: baselines still match
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let p = c.forget_local_plan(NOW, 30, &|_| false);
assert_eq!(p.evict.len(), 5, "{p:?}");
let ops = p.ops();
for op in &ops {
c.apply(op.clone()).unwrap();
}
j.append(&ops).unwrap();
let expect = c.to_snapshot();
drop(j); // crash: replay the log
let (_, c2, r) = Journal::open(Box::new(m)).unwrap();
assert_eq!(r.failed, 0);
assert_eq!(c2.to_snapshot(), expect);
assert_eq!(c2.len(), 5);
// nothing left to forget
assert!(c2.forget_local_plan(NOW, 30, &|_| false).evict.is_empty());
}
#[test]
fn op_photo_ids_cover_nested_ops() {
let op = Op::Batch {
ops: vec![
Op::SetRating { id: PhotoId(1), rating: 1 },
Op::SetAlbumPhotos { id: AlbumId(1), photos: vec![PhotoId(2), PhotoId(3)] },
Op::Batch { ops: vec![Op::RemovePhoto { id: PhotoId(4) }] },
],
};
let mut ids = Vec::new();
op.photo_ids(&mut |id| ids.push(id.0));
assert_eq!(ids, vec![1, 2, 3, 4]);
}
+223
View File
@@ -0,0 +1,223 @@
//! Issue #101: an append that fails part-way (disk full, a dropped network share) and is then
//! retried must not read back as log damage, and must lose nothing.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use crate::journal::LOG;
use crate::*;
/// A store whose next append writes only a prefix of its data, then fails.
#[derive(Clone, Default)]
struct Torn {
files: MemStore,
/// The next append writes this many bytes, then returns an error.
torn_at: Arc<Mutex<Option<usize>>>,
/// `truncate` fails (the cut-back after a failed append can't be done right away).
truncate_fails: Arc<AtomicBool>,
}
impl Torn {
fn tear_next_append(&self, bytes: usize) {
*self.torn_at.lock().unwrap() = Some(bytes);
}
}
impl Store for Torn {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if let Some(n) = self.torn_at.lock().unwrap().take() {
self.files.append(name, &data[..n.min(data.len())])?;
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
if self.truncate_fails.load(Ordering::SeqCst) {
return Err(std::io::Error::other("share unavailable"));
}
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"torn".into()
}
}
fn add_photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// Apply `ops` to the live catalog (like a command) and return them for the journal.
fn apply(c: &mut Catalog, ops: Vec<Op>) -> Vec<Op> {
for op in &ops {
c.apply(op.clone()).unwrap();
}
ops
}
/// Every tear position of a batch, then the retry (same ops, same seqs, like the engine's queue)
/// and more appends: reloading replays everything, reports no damage and no torn bytes.
#[test]
fn partial_append_then_retry_loads_cleanly() {
let probe = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(probe.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
let before = probe.get(LOG).unwrap().len();
j.append(&batch).unwrap();
let batch_len = probe.get(LOG).unwrap().len() - before;
for tear in 1..batch_len {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
t.tear_next_append(tear);
assert!(j.append(&batch).is_err(), "tear at {tear}");
// cut back to the last whole record
assert_eq!(t.files.get(LOG).unwrap().len(), good_len, "tear at {tear}");
j.append(&batch).unwrap();
let more = apply(&mut c, vec![Op::SetLabel { id, label: Some(ColorLabel::ALL[0]) }]);
j.append(&more).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!((r.torn_bytes, r.torn_fragments, r.replayed), (0, 0, 5), "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
}
/// When cutting back fails too (the share is gone), the next append cuts back first; until that
/// works nothing more is appended after the fragment.
#[test]
fn failed_cut_back_is_done_before_the_next_append() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetRating { id, rating: 5 }]);
t.truncate_fails.store(true, Ordering::SeqCst);
t.tear_next_append(30);
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30, "the fragment is still there");
// still unreachable: the retry fails without writing anything after the fragment
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30);
t.truncate_fails.store(false, Ordering::SeqCst);
j.append(&batch).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "{r:?}");
assert_eq!(r.replayed, 3);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// Logs written before the fix: the fragment of the failed append is glued to the first retried
/// record on one line, followed by the rest of the retry. They load in full (the fragment is
/// skipped), instead of stopping at the fragment and setting the later records aside as damage.
#[test]
fn legacy_fragment_glued_to_the_retry_is_skipped() {
let m = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good = m.get(LOG).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(
&mut c,
vec![
Op::SetRating { id, rating: 3 },
Op::SetFlag { id, flag: Flag::Pick },
Op::SetLabelName { label: ColorLabel::ALL[1], name: Some("Größe ✓".into()) },
],
);
j.append(&batch).unwrap();
let full = m.get(LOG).unwrap();
let batch_bytes = &full[good.len()..];
let later = apply(&mut c, vec![Op::SetRating { id, rating: 1 }]);
j.append(&later).unwrap();
let later_bytes = m.get(LOG).unwrap()[full.len()..].to_vec();
drop(j);
// every tear (inside the first record, after it, inside a multi-byte character…)
for tear in 1..batch_bytes.len() {
if batch_bytes[tear - 1] == b'\n' {
continue; // a tear on a record boundary leaves no fragment
}
let mut log = good.clone();
log.extend_from_slice(&batch_bytes[..tear]);
log.extend_from_slice(batch_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, loaded, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!(r.torn_fragments, 1, "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
// a damaged record whose tail happens to be a whole record from the future is still damage
let mut log = good.clone();
log.extend_from_slice(b"garbage");
log.extend_from_slice(&later_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, _, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_some(), "{r:?}");
}
/// A snapshot can absorb ops whose append failed: it records them as saved (`seq` moves past
/// them), and appending after it continues from there.
#[test]
fn snapshot_with_unlogged_ops_saves_them() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let queued = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetFlag { id, flag: Flag::Reject }]);
t.tear_next_append(10);
assert!(j.append(&queued).is_err());
j.snapshot_with_unlogged(&c, queued.len() as u64).unwrap();
assert_eq!(j.seq(), 3);
let after = apply(&mut c, vec![Op::SetRating { id, rating: 5 }]);
j.append(&after).unwrap();
drop(j);
let (j, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert_eq!((r.snapshot_seq, r.replayed), (3, 1));
assert_eq!(j.seq(), 4);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// `FsStore::truncate` on a file that was never created (an append that failed first) is a no-op.
#[test]
fn fs_store_truncates_a_missing_file_quietly() {
let dir = std::env::temp_dir().join(format!("lc-torn-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
let mut s = FsStore::open(&dir).unwrap();
s.truncate(LOG, 0).unwrap();
assert!(!dir.join(LOG).exists());
s.append(LOG, b"abc\n").unwrap();
s.truncate(LOG, 0).unwrap();
assert_eq!(std::fs::read(dir.join(LOG)).unwrap(), b"");
let _ = std::fs::remove_dir_all(&dir);
}
+1
View File
@@ -17,6 +17,7 @@ lightcraft-scenes = { workspace = true }
lightcraft-pipeline = { workspace = true }
lightcraft-gpu = { workspace = true }
lightcraft-catalog = { workspace = true }
web-time = { workspace = true }
lightcraft-preview = { workspace = true }
lightcraft-codecs = { workspace = true }
lightcraft-raw = { workspace = true }
+115
View File
@@ -2,6 +2,11 @@
//! The folder's photos are probed in place as `local` photos — only folder views list them —
//! and edits go to the files' XMP sidecars like any photo added in place. "Add to My Photos"
//! makes them ordinary library photos.
//!
//! Each browse records when the folders it listed were browsed. Untouched records of folders
//! not browsed for [`Session::forget_local_days`] are forgotten (when the library opens, or with
//! `library.forgetLocal`); see [`lightcraft_catalog::local`] for what "untouched" means. Files
//! and sidecars on disk are never touched, and browsing the folder again brings them back.
use std::path::Path;
@@ -35,12 +40,113 @@ fn browse(s: &mut Session, p: &Value) -> Result<Value> {
v
};
let report = import_with(s, &files, &ImportOptions { mode: ImportMode::Add, local: true, ..Default::default() })?;
stamp_browsed(s, &dir_s, &files);
s.browse = Some(Browse { path: dir_s.clone(), subfolders });
s.source = LibrarySource::Folder;
let shown = s.visible().len();
Ok(json!({"path": dir_s, "subfolders": subfolders, "photos": shown, "new": report.imported.len(), "failed": report.failed.len()}))
}
/// Folders whose browse time is refreshed at most this often (keeps repeated browsing from
/// growing the log; far finer than the days the eviction counts in).
const RESTAMP_SECS: i64 = 3600;
/// Record that `root` and the folders of `files` were browsed now (journaled, not an undo step).
fn stamp_browsed(s: &mut Session, root: &str, files: &[String]) {
let now = (s.clock)();
let now_s = lightcraft_catalog::stacks::iso_seconds(&now);
let mut folders: std::collections::BTreeSet<String> =
files.iter().filter_map(|f| Path::new(f).parent().map(|d| d.to_string_lossy().trim_end_matches(['/', '\\']).to_string())).collect();
folders.insert(root.to_string());
let ops: Vec<Op> = folders
.into_iter()
.filter(|f| {
let at = s.catalog.last_browsed(f).and_then(lightcraft_catalog::stacks::iso_seconds);
match (at, now_s) {
(Some(at), Some(n)) => n - at >= RESTAMP_SECS || n < at,
_ => true,
}
})
.map(|folder| Op::SetBrowsed { folder, at: Some(now.clone()) })
.collect();
if !ops.is_empty() {
let op = Op::Batch { ops };
if s.catalog.apply(op.clone()).is_ok() {
s.pending_log.push(op);
}
}
}
/// Photos the session still needs, whatever their state: shown, selected, being edited, or
/// referred to by an undo / redo step.
fn in_use(s: &mut Session) -> std::collections::HashSet<PhotoId> {
let mut keep: std::collections::HashSet<PhotoId> = s.visible().iter().copied().collect();
keep.extend(s.selection.ids.iter().copied());
keep.extend(s.selection.active);
keep.extend(s.previous_active);
keep.extend(s.interaction.as_ref().map(|i| i.photo));
keep.extend(s.before.keys().copied());
for e in s.undo.iter().chain(s.redo.iter()) {
e.op.photo_ids(&mut |id| {
keep.insert(id);
});
}
keep
}
impl Session {
/// Forget untouched Local records whose folder wasn't browsed for `days` (default
/// [`Session::forget_local_days`]; 0 = never). With `dry_run`, only report what would go.
/// Journaled (durable, replayed after a crash) but not an undo step; nothing on disk changes.
pub fn forget_local(&mut self, dry_run: bool, days: Option<u32>) -> lightcraft_catalog::ForgetPlan {
let days = days.unwrap_or(self.forget_local_days);
let now = (self.clock)();
let keep = in_use(self);
let smart = self.media.smart_dir.clone();
let plan = self
.catalog
.forget_local_plan(&now, days, &|p| keep.contains(&p.id) || smart.as_ref().is_some_and(|d| d.join(crate::smart::file_name(p)).exists()));
if dry_run {
return plan;
}
for op in plan.ops() {
match self.catalog.apply(op.clone()) {
Ok(_) => self.pending_log.push(op),
Err(e) => log::warn!("forget Local records: {e}"),
}
}
if !plan.evict.is_empty() {
log::info!("forgot {} untouched Local record(s) of folders not browsed for {days} days", plan.evict.len());
}
plan
}
}
fn forget_local(s: &mut Session, p: &Value) -> Result<Value> {
let dry_run = bool_or(p, "dryRun", false);
let days = match p.get("days") {
None | Some(Value::Null) => None,
Some(v) => {
Some(v.as_u64().ok_or_else(|| bad("library.forgetLocal", "days must be a whole number (0 = never)"))?.min(u64::from(u32::MAX)) as u32)
}
};
let used = days.unwrap_or(s.forget_local_days);
let plan = s.forget_local(dry_run, days);
if !dry_run && !plan.evict.is_empty() {
s.compact_soon();
}
Ok(json!({
"dryRun": dry_run,
"days": used,
"local": plan.local,
"forgotten": plan.evict.len(),
"keptRecent": plan.kept_recent,
"keptTouched": plan.kept_touched,
"keptInUse": plan.kept_in_use,
"foldersStamped": plan.stamp.len(),
}))
}
/// The browsed photos among `ids` (default: the selection) become library photos.
fn add_to_library(s: &mut Session, p: &Value) -> Result<Value> {
let ids: Vec<PhotoId> = s.targets(p).into_iter().filter(|id| s.catalog.photo(*id).is_some_and(|ph| ph.local)).collect();
@@ -98,6 +204,15 @@ pub fn specs() -> Vec<CommandSpec> {
always,
browse
),
cmd!(
"library.forgetLocal",
"Forget Unchanged Local Photos",
[],
None,
"{dryRun?: bool, days?: n (default: the library preference, 0 = never)} — remove Local (browsed, not added) photos nobody changed, of folders not browsed for `days`, from the catalog; files and sidecars stay, browsing the folder again brings them back; not an undo step → {local, forgotten, keptRecent, keptTouched, keptInUse, foldersStamped, days, dryRun}",
always,
forget_local
),
cmd!("photo.addToLibrary", "Add to My Photos", ["Photo"], None, "{ids?} — browsed (Local) photos join the library", always, add_to_library),
]
}
+27 -16
View File
@@ -10,22 +10,21 @@ use serde_json::{Value, json};
use super::{CommandSpec, bad, cmd, has_active, has_selection};
use crate::{Result, Session};
/// A free `<stem>.dng` (then `<stem>-2.dng`…) next to `path`.
fn dng_path(path: &str) -> String {
/// `<stem>.dng`, then `<stem>-2.dng`, `<stem>-3.dng`… next to `path`.
fn dng_names(path: &str) -> impl Iterator<Item = std::path::PathBuf> {
let p = Path::new(path);
let dir = p.parent().unwrap_or(Path::new(""));
let dir = p.parent().unwrap_or(Path::new("")).to_path_buf();
let stem = p.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| "photo".into());
let mut out = dir.join(format!("{stem}.dng"));
let mut n = 2;
while out.exists() {
out = dir.join(format!("{stem}-{n}.dng"));
n += 1;
}
out.to_string_lossy().to_string()
(1u64..).map(move |n| if n == 1 { dir.join(format!("{stem}.dng")) } else { dir.join(format!("{stem}-{n}.dng")) })
}
/// Write the DNG for raw file `path` (develop settings in `packet`); returns the new path.
/// An empty `packet` writes no XMP.
/// Write the DNG for raw file `path` (develop settings in `packet`) next to it under a free name;
/// returns the new path. An empty `packet` writes no XMP.
///
/// The DNG is checked before anyone relies on it (Convert to DNG relinks the photo to it, Copy as
/// DNG deletes the raw copy): the encoded file must decode to the same raw data, and it is
/// written to a temp file, synced, read back identical and only then given its name — never
/// replacing an existing file. On any failure no DNG is left behind and the raw stays in use.
pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::result::Result<String, String> {
let bytes = match &s.media.file_bytes {
Some(r) => r(path)?,
@@ -35,9 +34,21 @@ pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::res
drop(bytes);
let dng = lightcraft_raw::write_dng(&raw, &lightcraft_raw::DngWriteOptions { xmp: (!packet.is_empty()).then_some(packet), ..Default::default() })
.map_err(|e| e.to_string())?;
let out = dng_path(path);
std::fs::write(&out, dng).map_err(|e| format!("{out}: {e}"))?;
Ok(out)
let back = lightcraft_raw::decode(&dng).map_err(|e| format!("{path}: the DNG written for it doesn't decode ({e}); the raw is kept"))?;
let same = match (&back.data, &raw.data) {
(lightcraft_raw::RawData::U16(a), lightcraft_raw::RawData::U16(b)) => a == b,
(lightcraft_raw::RawData::F32(a), lightcraft_raw::RawData::F32(b)) => {
a.len() == b.len() && a.iter().zip(b).all(|(x, y)| x.to_bits() == y.to_bits())
}
_ => false,
};
if (back.width, back.height, back.cpp) != (raw.width, raw.height, raw.cpp) || !same {
return Err(format!("{path}: the DNG written for it doesn't hold the same raw data; the raw is kept"));
}
drop((raw, back));
let out = lightcraft_catalog::safe_file::write_new_unique(&mut dng_names(path), &dng)
.map_err(|e| format!("{path}: could not write its DNG ({e}); the raw is kept"))?;
Ok(out.to_string_lossy().to_string())
}
fn convert(s: &mut Session, p: &Value) -> Result<Value> {
@@ -133,7 +144,7 @@ fn edit_external(s: &mut Session, p: &Value) -> Result<Value> {
};
let mut written = Vec::new();
let mut write = |path: &str, bytes: &[u8]| -> std::result::Result<(), String> {
std::fs::write(path, bytes).map_err(|e| format!("{path}: {e}"))?;
crate::export::write_file(path, bytes)?;
written.push(path.to_string());
Ok(())
};
+13
View File
@@ -96,5 +96,18 @@ pub fn specs() -> Vec<CommandSpec> {
save
),
cmd!("export.deletePreset", "Delete Export Preset", [], None, "{name} — removes a user preset → presets", always, delete),
cmd!(
query "export.checkTarget",
"Check Output Path",
[],
None,
"{path} — fails when writing `path` would replace a photo's original (or its XMP sidecar) in the library; front ends call it before saving a render or screenshot to a user-given path → {path}",
always,
|s, p| {
let path = p.get("path").and_then(Value::as_str).ok_or_else(|| bad("export.checkTarget", "missing `path`"))?;
s.check_write_target(path).map_err(|e| bad("export.checkTarget", e))?;
Ok(json!({"path": path}))
}
),
]
}
+5
View File
@@ -867,6 +867,11 @@ pub fn specs() -> Vec<CommandSpec> {
"logRecords": j.log_records(),
"logBytes": j.log_bytes(),
"lastError": lib.last_error,
// changes applied in memory whose write failed (retried by every save)
"unsavedOps": s.unsaved().map_or(0, |u| u.0),
"unsavedError": s.unsaved().map(|u| u.1),
// untouched Local records forgotten when the library opened
"forgotLocal": lib.forgot_local.as_ref().map(|p| json!({"forgotten": p.evict.len(), "local": p.local, "keptRecent": p.kept_recent, "keptTouched": p.kept_touched, "keptInUse": p.kept_in_use})),
"persistence": j.stats(),
"cache": cache,
"load": {
+2 -1
View File
@@ -30,7 +30,8 @@ fn run(s: &mut Session, id: &str, p: &Value) -> Result<Value> {
if let (Some(path), Some(img)) = (str_param(p, "previewPath"), &out.preview) {
let png = lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default())
.map_err(|e| bad(id, e.to_string()))?;
std::fs::write(path, png).map_err(|e| bad(id, format!("{path}: {e}")))?;
s.check_write_target(path).map_err(|e| bad(id, e))?;
crate::export::write_file(path, &png).map_err(|e| bad(id, e))?;
info["previewPath"] = json!(path);
}
if let Some(img) = &out.preview {
+85 -18
View File
@@ -1,8 +1,9 @@
//! Missing originals: photos whose file is no longer where the library expects it (moved,
//! renamed, on an unplugged drive), and relinking them — one at a time (`photo.relink`) or by
//! searching a folder for files with the same name and size (`library.findMissing`).
//! searching a folder for files with the same name and size, or the same content
//! (`library.findMissing`).
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::path::Path;
use lightcraft_catalog::{Catalog, Op, Photo, PhotoId, Source};
@@ -83,8 +84,29 @@ fn relink(s: &mut Session, p: &Value) -> Result<Value> {
Ok(json!({"id": id.0, "path": abs}))
}
/// Search `folder` (recursively) for each missing photo's file name — and its size when the
/// library knows it — and relink every match in one undoable step.
/// The stored content hash of a photo, without the `:dupN` suffix duplicates get.
fn stored_hash(p: &Photo) -> Option<&str> {
p.content_hash.as_deref().map(|h| h.split(':').next().unwrap_or(h)).filter(|h| !h.is_empty())
}
/// The content hash of the file at `path` (as import computes it), if it can be read.
fn file_hash(path: &str) -> Option<String> {
std::fs::read(path).ok().map(|b| lightcraft_preview::hash_bytes(&b).to_string())
}
/// Search `folder` (recursively) for each missing photo's file and relink every match in one
/// undoable step.
///
/// Matching, per photo:
/// 1. a file with the same name (any letter case), the same size and the same content hash;
/// 2. renamed: a file of the same size with the same content hash (only same-size files are
/// read, so this stays cheap);
/// 3. without a hash to confirm (or the file was edited in place): the one file with the same
/// name and size. Several such files are ambiguous: the photo is skipped and reported in
/// `ambiguous`, never guessed (two cameras' `IMG_0001.CR3` of one size are different photos).
///
/// Files that are already some photo's original, or that another photo was just matched with,
/// are not used.
fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.findMissing";
let folder = str_param(p, "folder").ok_or_else(|| bad(C, "missing `folder`"))?;
@@ -93,27 +115,72 @@ fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
}
let lost = missing(s);
if lost.is_empty() {
return Ok(json!({"found": [], "missing": 0}));
return Ok(json!({"found": [], "missing": 0, "ambiguous": []}));
}
// file name (lower case) → candidate paths
let in_use: HashSet<String> = s.catalog.photos().filter_map(|p| checked_path(p).map(str::to_string)).collect();
// file name (lower case) → candidate paths; size → paths (filled lazily)
let mut by_name: HashMap<String, Vec<String>> = HashMap::new();
for f in crate::import::expand(&[folder.to_string()], None) {
if let Some(n) = Path::new(&f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f);
let files: Vec<String> = crate::import::expand(&[folder.to_string()], None).into_iter().filter(|f| !in_use.contains(f)).collect();
for f in &files {
if let Some(n) = Path::new(f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f.clone());
}
}
let mut by_size: Option<HashMap<u64, Vec<String>>> = None;
let mut hashes: HashMap<String, Option<String>> = HashMap::new();
let mut hash_of = |f: &str| hashes.entry(f.to_string()).or_insert_with(|| file_hash(f)).clone();
let size_of = |f: &str| std::fs::metadata(f).map(|m| m.len()).ok();
let mut used: HashSet<String> = HashSet::new();
let mut ops = Vec::new();
let mut found = Vec::new();
let mut ambiguous = Vec::new();
for (id, old) in &lost {
let Some(ph) = s.catalog.photo(*id) else { continue };
let name = Path::new(old).file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default();
let size = s.catalog.photo(*id).map(|p| p.file_size).unwrap_or(0);
let hit = by_name.get(&name).and_then(|c| {
c.iter().find(|f| size == 0 || std::fs::metadata(f).is_ok_and(|m| m.len() == size)).or_else(|| (c.len() == 1 && size == 0).then(|| &c[0]))
});
if let Some(path) = hit {
ops.push(relink_op(*id, path));
found.push(json!({"id": id.0, "from": old, "to": path}));
let size = ph.file_size;
let hash = stored_hash(ph).map(str::to_string);
let named: Vec<&String> = by_name
.get(&name)
.map(|c| c.iter().filter(|f| !used.contains(*f) && (size == 0 || size_of(f) == Some(size))).collect())
.unwrap_or_default();
// 1. same name and size, confirmed by the content hash
let mut hit: Option<(String, &str)> =
hash.as_deref().and_then(|h| named.iter().find(|f| hash_of(f).as_deref() == Some(h))).map(|f| (f.to_string(), "name"));
// 2. renamed: same size and content
if hit.is_none()
&& let Some(h) = &hash
&& size > 0
{
let sizes = by_size.get_or_insert_with(|| {
let mut m: HashMap<u64, Vec<String>> = HashMap::new();
for f in &files {
if let Some(n) = size_of(f) {
m.entry(n).or_default().push(f.clone());
}
}
m
});
hit = sizes
.get(&size)
.and_then(|c| c.iter().find(|f| !used.contains(*f) && hash_of(f).as_deref() == Some(h.as_str())))
.map(|f| (f.clone(), "content"));
}
// 3. same name and size only (no hash, or the file was edited in place): one candidate
// is taken, several are ambiguous and skipped rather than guessed
if hit.is_none() {
match named.len() {
1 => hit = Some((named[0].clone(), "name")),
0 => {}
_ => {
ambiguous.push(json!({"id": id.0, "from": old, "candidates": named}));
continue;
}
}
}
let Some((path, by)) = hit else { continue };
ops.push(relink_op(*id, &path));
found.push(json!({"id": id.0, "from": old, "to": path, "by": by}));
used.insert(path);
}
let still = lost.len() - found.len();
if !ops.is_empty() {
@@ -124,7 +191,7 @@ fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
}
}
}
Ok(json!({"found": found, "missing": still}))
Ok(json!({"found": found, "missing": still, "ambiguous": ambiguous}))
}
pub fn specs() -> Vec<CommandSpec> {
@@ -146,7 +213,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Find Missing Photos",
[],
None,
"{folder} — relink every missing photo whose file (same name and size) is somewhere in the folder → {found: [{id, from, to}], missing}",
"{folder} — relink every missing photo whose file is somewhere in the folder: same name and size (and content, when the library knows its hash), or — renamed — same size and content → {found: [{id, from, to, by: name|content}], missing, ambiguous: [{id, from, candidates}] (several same-name candidates and no hash to tell them apart: skipped)}",
always,
find_missing
),
+7 -1
View File
@@ -14,6 +14,7 @@ fn current(s: &Session) -> Value {
"xmp": s.xmp,
"import": s.import_defaults,
"cacheMb": if s.cache_mb == 0 { s.cache_bytes() >> 20 } else { u64::from(s.cache_mb) },
"forgetLocalDays": s.forget_local_days,
"persistent": s.library.is_some(),
})
}
@@ -86,6 +87,11 @@ fn prefs(s: &mut Session, p: &Value) -> Result<Value> {
changed = true;
}
s.import_defaults = d;
if let Some(v) = p.get("forgetLocalDays") {
let days = v.as_u64().ok_or_else(|| bad(ID, "forgetLocalDays must be a whole number of days (0 = never)"))?;
s.forget_local_days = days.min(36_500) as u32;
changed = true;
}
if let Some(mb) = p.get("cacheMb") {
let mb = mb.as_u64().ok_or_else(|| bad(ID, "cacheMb must be a number of megabytes"))?;
s.set_cache_mb(mb.clamp(0, 1 << 20) as u32)?;
@@ -101,7 +107,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Library Preferences",
[],
None,
"{import?: {rawPreset?: presetId|\"default\", otherPreset?: presetId|\"default\", perCamera?: bool, cameras?: [{camera: \"Make Model\", preset: presetId|null}], copyright?: text, creator?: text (given to imported photos without one)}, camera?: {camera, preset?, remove?: bool}, cacheMb?: n (0 = default)} — develop defaults applied on import (raws / other images / per camera) and the thumbnail cache size, saved with the library → {xmp, import, cacheMb, persistent}",
"{import?: {rawPreset?: presetId|\"default\", otherPreset?: presetId|\"default\", perCamera?: bool, cameras?: [{camera: \"Make Model\", preset: presetId|null}], copyright?: text, creator?: text (given to imported photos without one)}, camera?: {camera, preset?, remove?: bool}, cacheMb?: n (0 = default), forgetLocalDays?: n (forget untouched Local photos of folders not browsed for n days; 0 = never)} — develop defaults applied on import (raws / other images / per camera) and the thumbnail cache size, saved with the library → {xmp, import, cacheMb, forgetLocalDays, persistent}",
always,
prefs
)]
+14 -7
View File
@@ -102,7 +102,7 @@ fn smart(s: &mut Session, p: &Value) -> Result<Value> {
let dir = s.media.smart_dir.clone().ok_or_else(|| bad(C, "smart previews need a library on disk"))?;
let discard = p.get("discard").and_then(Value::as_bool).unwrap_or(false);
let ids = if p.get("ids").is_some() || !s.selection.ids.is_empty() { s.targets(p) } else { s.visible_cloned() };
let (mut built, mut removed, mut failed) = (0usize, 0usize, Vec::new());
let (mut built, mut repaired, mut removed, mut failed) = (0usize, 0usize, 0usize, Vec::new());
if !discard {
// a chosen folder on another drive must be there: never recreate it on this one
if s.smart_previews_dir.is_some() {
@@ -123,20 +123,27 @@ fn smart(s: &mut Session, p: &Value) -> Result<Value> {
}
continue;
}
if path.exists() {
// a complete proxy is kept; a missing or damaged one (cut short by a crash or a full
// drive) is (re)built
let damaged = path.exists();
if damaged && crate::smart::is_valid(&path) {
built += 1;
continue;
}
// atomic: a failed write leaves no partial proxy that would pass for a built one
let r = s
.source_now(id, crate::media::SourceLevel::Preview)
.and_then(|src| crate::smart::encode(&src))
.and_then(|b| std::fs::write(&path, b).map_err(|e| e.to_string()));
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&path, &b).map_err(|e| format!("{}: {e}", path.display())));
match r {
Ok(()) => built += 1,
Ok(()) => {
built += 1;
repaired += usize::from(damaged);
}
Err(e) => failed.push(json!([id.0, e])),
}
}
Ok(json!({"built": built, "removed": removed, "failed": failed}))
Ok(json!({"built": built, "repaired": repaired, "removed": removed, "failed": failed}))
}
/// The smart previews folder: where it is, whether it is custom, what it holds, whether it can be
@@ -205,7 +212,7 @@ fn smart_location(s: &mut Session, p: &Value) -> Result<Value> {
/// Whether photo `id` has a smart preview.
pub fn has_smart_preview(s: &Session, id: lightcraft_catalog::PhotoId) -> bool {
match (&s.media.smart_dir, s.catalog.photo(id)) {
(Some(dir), Some(p)) => dir.join(crate::smart::file_name(p)).exists(),
(Some(dir), Some(p)) => crate::smart::is_valid(&dir.join(crate::smart::file_name(p))),
_ => false,
}
}
@@ -218,7 +225,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Build Smart Previews",
[],
None,
"{ids?, discard?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline → {built, removed, failed}",
"{ids?, discard?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline; a damaged proxy (cut short) is rebuilt → {built, repaired, removed, failed}",
always,
smart
),
+9 -3
View File
@@ -113,7 +113,7 @@ pub fn specs() -> Vec<CommandSpec> {
.map(|r| json!({"group": r.group, "tag": r.tag, "name": r.name, "value": r.value}))
.collect();
// XMP: the sidecar if there is one, else the file's own packet
let packet = crate::sidecar::read_packet(path, ph.kind, s.xmp.naming).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let packet = crate::sidecar::read_packet(path, ph.kind, s.sidecar_naming(ph.id)).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let xmp: Vec<Value> = packet
.and_then(|x| lightcraft_meta::parse_xmp(&x).ok())
.map(|d| d.properties.into_iter().filter(|(k, _)| !k.starts_with("lc:")).map(|(k, v)| json!({"name": k, "value": v.join("; ")})).collect())
@@ -186,11 +186,17 @@ pub fn specs() -> Vec<CommandSpec> {
"versions": ph.versions.iter().map(|v| json!({"name": v.name, "created": v.created})).collect::<Vec<_>>(),
}))
}),
cmd!(query "app.gpu", "GPU Rendering", [], None, "{enabled?: bool} — allow/forbid GPU rendering (CPU fallback; LIGHTCRAFT_GPU=0 forbids it for the process)", always, |_, p| {
cmd!(query "app.gpu", "GPU Rendering", [], None, "{enabled?: bool} — allow/forbid GPU rendering (CPU fallback; LIGHTCRAFT_GPU=0 forbids it for the process); returns {enabled, available, adapter, reason (why the GPU is off), lastFallback (latest render redone on the CPU, and why)}", always, |_, p| {
if let Some(on) = p.get("enabled").and_then(Value::as_bool) {
lightcraft_gpu::set_enabled(on);
}
Ok(json!({"enabled": lightcraft_gpu::enabled(), "available": lightcraft_gpu::available(), "adapter": lightcraft_gpu::adapter_name()}))
Ok(json!({
"enabled": lightcraft_gpu::enabled(),
"available": lightcraft_gpu::available(),
"adapter": lightcraft_gpu::adapter_name(),
"reason": lightcraft_gpu::unavailable_reason(),
"lastFallback": lightcraft_gpu::last_fallback(),
}))
}),
cmd!(query "library.memory", "Memory Usage", [], None, "{} — bytes held by each cache (decoded sources, rendered previews, GPU buffers; heap when instrumented)", always, |s, _| {
Ok(serde_json::to_value(s.memory_report()).unwrap_or_default())
+16 -4
View File
@@ -9,14 +9,26 @@ use crate::{Result, Session};
fn save(s: &mut Session, p: &Value) -> Result<Value> {
let mut written = Vec::new();
let mut merged = Vec::new();
let mut backups = Vec::new();
let mut failed = Vec::new();
let owners = crate::sidecar::StemOwners::of(&s.catalog);
for id in s.targets(p) {
match s.save_sidecar(id) {
Ok(path) => written.push(path.display().to_string()),
match s.save_sidecar_with(id, &owners) {
Ok(r) => {
let path = r.path.display().to_string();
if r.merged {
merged.push(path.clone());
}
if let Some(b) = r.backup {
backups.push(json!({"path": path, "backup": b.display().to_string()}));
}
written.push(path);
}
Err(e) => failed.push(json!({"id": id.0, "error": e.to_string()})),
}
}
Ok(json!({"written": written, "failed": failed}))
Ok(json!({"written": written, "merged": merged, "backups": backups, "failed": failed}))
}
fn read(s: &mut Session, p: &Value) -> Result<Value> {
@@ -63,7 +75,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Save Metadata to File",
["Photo"],
Some("Cmd+S"),
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original → {written: [path], failed}",
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original, merged into an existing one (other apps' data kept; an unreadable one is backed up first) → {written: [path], merged: [path], backups: [{path, backup}], failed}",
has_selection,
save
),
+3 -1
View File
@@ -35,7 +35,9 @@ fn mount_parents() -> Vec<std::path::PathBuf> {
let mut out = Vec::new();
if cfg!(target_os = "macos") {
out.push("/Volumes".into());
} else if cfg!(target_os = "linux") {
} else if cfg!(any(target_os = "linux", target_os = "freebsd")) {
// Linux desktops mount under /media/$USER or /run/media/$USER; FreeBSD's automounter
// (automount/autofs, or a desktop's) under /media. Missing folders are skipped.
if let Ok(user) = std::env::var("USER") {
out.push(format!("/media/{user}").into());
out.push(format!("/run/media/{user}").into());
+184 -39
View File
@@ -202,6 +202,8 @@ pub enum Anchor {
#[serde(default, rename_all = "camelCase")]
pub struct Watermark {
pub text: String,
/// Basic Japanese vertical lettering: upright glyphs in columns from right to left.
pub vertical: bool,
/// Text height as a fraction of the image's short edge.
pub size: f32,
/// 0..1.
@@ -226,6 +228,7 @@ impl Default for Watermark {
fn default() -> Self {
Self {
text: String::new(),
vertical: false,
size: 0.035,
opacity: 0.7,
anchor: Anchor::BottomRight,
@@ -241,6 +244,9 @@ impl Default for Watermark {
/// Inter SemiBold (OFL, see assets/ATTRIBUTION.md).
static WATERMARK_FONT: &[u8] = include_bytes!("../../../assets/fonts/Inter-SemiBold.ttf");
/// BIZ UDMincho (OFL), Japanese fallback on both native and web.
/// Bundled OFL Japanese font bytes, shared by watermarks and UI glyph fallback.
pub static WATERMARK_JAPANESE_FONT: &[u8] = include_bytes!("../../../assets/fonts/BIZUDMincho-Regular.ttf");
/// Draw `wm` onto `img` (straight alpha blending of the encoded values).
pub fn draw_watermark(img: &mut Rgba8, wm: &Watermark) {
@@ -288,23 +294,67 @@ fn watermark_coverage(width: usize, height: usize, wm: &Watermark, mut blend_px:
return;
}
let Ok(font) = FontRef::try_from_slice(WATERMARK_FONT) else { return };
let Ok(japanese) = FontRef::try_from_slice(WATERMARK_JAPANESE_FONT) else { return };
let fonts = [font, japanese];
let short = width.min(height) as f32;
let px = (wm.size.clamp(0.005, 0.5) * short).max(6.0);
let sf = font.as_scaled(PxScale::from(px));
// Lay out one line.
let mut glyphs = Vec::new();
let mut x = 0.0f32;
let mut y = 0.0f32;
let mut tw = 0.0f32;
let mut prev = None;
for ch in text.chars() {
let columns = text.split('\n').count();
let mut column = 0usize;
for original in text.chars() {
if original == '\n' {
if wm.vertical {
column = column.saturating_add(1);
tw = tw.max(y);
y = 0.0;
} else {
tw = tw.max(x);
x = 0.0;
y += px;
}
prev = None;
continue;
}
let vertical_form = match original {
'、' => '︑',
'。' => '︒',
'(' => '︵',
')' => '︶',
'「' => '﹁',
'」' => '﹂',
'『' => '﹃',
'』' => '﹄',
'【' => '︻',
'】' => '︼',
'…' => '︙',
other => other,
};
let ch = if wm.vertical && fonts[1].glyph_id(vertical_form).0 != 0 { vertical_form } else { original };
let face = usize::from(fonts[0].glyph_id(ch).0 == 0);
let sf = fonts[face].as_scaled(PxScale::from(px));
let id = sf.glyph_id(ch);
if let Some(p) = prev {
x += sf.kern(p, id);
if wm.vertical {
let left = columns.saturating_sub(column.saturating_add(1)) as f32 * px;
let pen = point(left + (px - sf.h_advance(id)) / 2.0, y + sf.ascent());
let rotation = matches!(original, 'ー' | '—' | '–').then_some((left + px / 2.0, y + px / 2.0));
glyphs.push((face, id.with_scale_and_position(px, pen), rotation));
y += px;
} else {
if let Some((previous_face, previous)) = prev
&& previous_face == face
{
x += sf.kern(previous, id);
}
glyphs.push((face, id.with_scale_and_position(px, point(x, y + sf.ascent())), None));
x += sf.h_advance(id);
prev = Some((face, id));
}
glyphs.push(id.with_scale_and_position(px, point(x, sf.ascent())));
x += sf.h_advance(id);
prev = Some(id);
}
let (tw, th) = (x, sf.ascent() - sf.descent());
let (tw, th) = if wm.vertical { (columns as f32 * px, tw.max(y)) } else { (tw.max(x), y + px) };
let inset = wm.inset.clamp(0.0, 0.4) * short;
let (w, h) = (width as f32, height as f32);
use Anchor::*;
@@ -328,10 +378,17 @@ fn watermark_coverage(width: usize, height: usize, wm: &Watermark, mut blend_px:
let passes: &[(f32, [u8; 3], f32)] =
if wm.shadow { &[((px * 0.05).max(1.0), [0, 0, 0], 0.45), (0.0, wm.color, 1.0)] } else { &[(0.0, wm.color, 1.0)] };
for &(off, col, a) in passes {
for g in &glyphs {
if let Some(o) = font.outline_glyph(g.clone()) {
for (face, g, rotation) in &glyphs {
if let Some(o) = fonts.get(*face).and_then(|font| font.outline_glyph(g.clone())) {
let b = o.px_bounds();
o.draw(|gx, gy, cov| blend((ox + off + b.min.x) as i32 + gx as i32, (oy + off + b.min.y) as i32 + gy as i32, cov, col, a * alpha));
o.draw(|gx, gy, cov| {
let (sx, sy) = (b.min.x + gx as f32, b.min.y + gy as f32);
let (sx, sy) = match rotation {
Some((cx, cy)) => (cx - (sy - cy), cy + (sx - cx)),
None => (sx, sy),
};
blend((ox + off + sx) as i32, (oy + off + sy) as i32, cov, col, a * alpha);
});
}
}
}
@@ -930,6 +987,8 @@ pub struct PreparedExport {
pub photo: lightcraft_catalog::PhotoId,
pub file_name: String,
work: Work,
/// The library's originals, which [`run_batch`] never writes over (shared by a batch).
guard: std::sync::Arc<crate::originals::OriginalGuard>,
}
struct RenderWork {
@@ -952,12 +1011,30 @@ enum Work {
},
}
/// Set up the export of photo `id` at 1-based position `seq` of a batch.
/// Set up the export of photo `id` at 1-based position `seq` of a batch. (For a whole batch use
/// [`prepare_batch`]: it looks at the library's originals once.)
pub fn prepare_export(
session: &mut crate::Session,
id: lightcraft_catalog::PhotoId,
o: &ExportOptions,
seq: usize,
) -> Result<PreparedExport, String> {
let guard = std::sync::Arc::new(session.original_guard());
prepare_guarded(session, id, o, seq, guard)
}
/// [`prepare_export`] for each of `ids` in order (`{seq}` = position, from 1).
pub fn prepare_batch(session: &mut crate::Session, ids: &[lightcraft_catalog::PhotoId], o: &ExportOptions) -> Result<Vec<PreparedExport>, String> {
let guard = std::sync::Arc::new(session.original_guard());
ids.iter().enumerate().map(|(i, id)| prepare_guarded(session, *id, o, i + 1, guard.clone())).collect()
}
fn prepare_guarded(
session: &mut crate::Session,
id: lightcraft_catalog::PhotoId,
o: &ExportOptions,
seq: usize,
guard: std::sync::Arc<crate::originals::OriginalGuard>,
) -> Result<PreparedExport, String> {
let p = session.catalog.photo(id).ok_or("no such photo")?;
let file_name = o.file_name_for(p, seq);
@@ -979,7 +1056,7 @@ pub fn prepare_export(
size: (p.width as usize, p.height as usize),
}
};
Ok(PreparedExport { photo: id, file_name, work })
Ok(PreparedExport { photo: id, file_name, work, guard })
}
impl PreparedExport {
@@ -1031,14 +1108,16 @@ pub enum DngCompression {
}
/// The destination of a batch: a folder (with `ExportOptions::subfolder` and the conflict policy
/// applied), or one exact file path (single photo; overwritten).
/// applied to each file together with its sidecars), or one exact file path (single photo; an
/// ordinary file already there is replaced). Either way a catalogued original (or its sidecar)
/// is never written over: see [`crate::originals`].
#[derive(Clone, Debug, Default)]
pub struct Destination {
pub dir: String,
pub exact: Option<String>,
}
/// Export `ids` in order ([`prepare_export`] + [`run_batch`]), stopping at the first error.
/// Export `ids` in order ([`prepare_batch`] + [`run_batch`]), stopping at the first error.
pub fn export_batch(
session: &mut crate::Session,
ids: &[lightcraft_catalog::PhotoId],
@@ -1047,15 +1126,34 @@ pub fn export_batch(
write: &mut dyn FnMut(&str, &[u8]) -> Result<(), String>,
exists: &dyn Fn(&str) -> bool,
) -> Result<Vec<serde_json::Value>, String> {
let items = ids.iter().enumerate().map(|(i, id)| prepare_export(session, *id, o, i + 1)).collect::<Result<Vec<_>, _>>()?;
let items = prepare_batch(session, ids, o)?;
run_batch(items, o, to, write, exists, true, &mut |_, _| true)
}
/// Write an exported or rendered file on disk: its folder is created if needed, and the file is
/// replaced atomically ([`lightcraft_catalog::safe_file::write_atomic`]: a temp file, synced, then
/// renamed), so a failure part-way leaves any previous file intact and no truncated one. The
/// native writer behind exports, renders and screenshots (app, CLI, MCP).
pub fn write_file(path: &str, bytes: &[u8]) -> Result<(), String> {
let p = std::path::Path::new(path);
if let Some(dir) = p.parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
}
lightcraft_catalog::safe_file::write_atomic(p, bytes).map_err(|e| format!("{path}: {e}"))
}
/// The path of the sidecar with extension `ext` of the exported file `main`.
fn sidecar_path(main: &str, ext: &str) -> String {
std::path::Path::new(main).with_extension(ext).to_string_lossy().to_string()
}
/// Run prepared exports in order: pick each one's path, and hand the bytes (and sidecars) to
/// `write`. `exists` tells whether a path is taken (conflict policy). `progress(done, next file)`
/// is called before each photo; returning false cancels the rest. Returns one JSON object per
/// photo: `{path, width, height, bytes, sidecars}`, `{skipped: path}` or (unless
/// `stop_on_error`) `{photo, file, error}`.
/// `write`. `exists` tells whether a path is taken. The conflict policy applies to a file and its
/// sidecars as one: with Unique both get the same free name, with Skip the photo is skipped when
/// either is taken. A path that is a catalogued original (or its sidecar) is refused whatever the
/// policy. `progress(done, next file)` is called before each photo; returning false cancels the
/// rest. Returns one JSON object per photo: `{path, width, height, bytes, sidecars}`,
/// `{skipped: path}` or (unless `stop_on_error`) `{photo, file, error}`.
pub fn run_batch(
items: Vec<PreparedExport>,
o: &ExportOptions,
@@ -1075,7 +1173,7 @@ pub fn run_batch(
if !progress(i, &item.file_name) {
break;
}
let (photo, name) = (item.photo, item.file_name.clone());
let (photo, name, guard) = (item.photo, item.file_name.clone(), item.guard.clone());
let e = match item.run() {
Ok(e) => e,
Err(err) if stop_on_error => return Err(err),
@@ -1084,14 +1182,18 @@ pub fn run_batch(
continue;
}
};
// the exported file and its sidecars
let group = |main: &str| std::iter::once(main.to_string()).chain(e.sidecars.iter().map(|(x, _)| sidecar_path(main, x))).collect::<Vec<_>>();
let path = match to.exact.as_deref().filter(|_| single) {
Some(p) => p.to_string(),
Some(p) => Ok(p.to_string()),
None => {
let mut path = join(&dir, &e.file_name);
let busy = |p: &str, taken: &std::collections::HashSet<String>| taken.contains(p) || exists(p);
if busy(&path, &taken) {
let path = join(&dir, &e.file_name);
let busy = |main: &str| group(main).iter().any(|p| taken.contains(p) || exists(p));
if !busy(&path) {
Ok(path)
} else {
match o.conflict {
Conflict::Overwrite => {}
Conflict::Overwrite => Ok(path),
Conflict::Skip => {
out.push(json!({"skipped": path}));
continue;
@@ -1099,29 +1201,34 @@ pub fn run_batch(
Conflict::Unique => {
let (stem, ext) = e.file_name.rsplit_once('.').map_or((e.file_name.as_str(), None), |(a, b)| (a, Some(b)));
let name = |n: usize| join(&dir, &ext.map_or(format!("{stem}-{n}"), |x| format!("{stem}-{n}.{x}")));
path = (2..).map(name).find(|p| !busy(p, &taken)).expect("a free name");
(2..1_000_000).map(name).find(|p| !busy(p)).ok_or_else(|| format!("{path}: no free file name"))
}
}
}
path
}
};
let written = write(&path, &e.bytes).and_then(|()| {
let mut sidecars = Vec::new();
for (ext, bytes) in &e.sidecars {
let sc = std::path::Path::new(&path).with_extension(ext).to_string_lossy().to_string();
write(&sc, bytes)?;
sidecars.push(sc);
let file = path.clone().unwrap_or_else(|_| name.clone());
let written = path.and_then(|path| {
let files = group(&path);
// never over an original, whatever the conflict policy or the exact path said
for f in &files {
guard.check(std::path::Path::new(f))?;
}
Ok(sidecars)
write(&path, &e.bytes)?;
let mut sidecars = Vec::new();
for ((_, bytes), sc) in e.sidecars.iter().zip(files.iter().skip(1)) {
write(sc, bytes)?;
sidecars.push(sc.clone());
}
Ok((path, files, sidecars))
});
match written {
Ok(sidecars) => {
taken.insert(path.clone());
Ok((path, files, sidecars)) => {
taken.extend(files);
out.push(json!({"path": path, "width": e.width, "height": e.height, "bytes": e.bytes.len(), "sidecars": sidecars}));
}
Err(err) if stop_on_error => return Err(err),
Err(err) => out.push(json!({"photo": photo.0, "file": path, "error": err})),
Err(err) => out.push(json!({"photo": photo.0, "file": file, "error": err})),
}
}
Ok(out)
@@ -1227,6 +1334,44 @@ mod tests {
assert!(ExportOptions::from_json(&serde_json::json!({"watermark": ""})).watermark.is_none());
}
#[test]
fn japanese_watermarks_support_vertical_columns_and_legacy_defaults() {
use ab_glyph::Font;
let font = ab_glyph::FontRef::try_from_slice(WATERMARK_JAPANESE_FONT).unwrap();
for c in "日本語".chars() {
assert_ne!(font.glyph_id(c).0, 0);
}
let old = ExportOptions::from_json(&json!({"watermark": {"text": "日本語"}}));
assert!(!old.watermark.unwrap().vertical);
let options =
ExportOptions::from_json(&json!({"watermark": {"text": "日本語", "vertical": true, "size": 0.1, "shadow": false, "opacity": 1.0}}));
let wm = options.watermark.unwrap();
assert!(wm.vertical);
let round: Watermark = serde_json::from_value(serde_json::to_value(&wm).unwrap()).unwrap();
assert!(round.vertical);
let mut img = Rgba8::new(400, 300);
img.data.fill([0, 0, 0, 255]);
draw_watermark(&mut img, &wm);
let lit: Vec<_> = (0..300).flat_map(|y| (0..400).map(move |x| (x, y))).filter(|&(x, y)| img.get(x, y)[0] > 64).collect();
assert!(lit.len() > 100);
let w = lit.iter().map(|p| p.0).max().unwrap() - lit.iter().map(|p| p.0).min().unwrap();
let h = lit.iter().map(|p| p.1).max().unwrap() - lit.iter().map(|p| p.1).min().unwrap();
assert!(h > w * 2, "vertical Japanese must extend down the column");
let two = Watermark { text: "日\n本".into(), anchor: Anchor::TopLeft, inset: 0.0, ..wm };
img.data.fill([0, 0, 0, 255]);
draw_watermark(&mut img, &two);
for (text, dx) in [("本", 0), ("日", 30)] {
let mut single = Rgba8::new(400, 300);
single.data.fill([0, 0, 0, 255]);
draw_watermark(&mut single, &Watermark { text: text.into(), ..two.clone() });
for y in 0..30 {
for x in 0..30 {
assert_eq!(img.get(x + dx, y), single.get(x, y), "newlines move columns left");
}
}
}
}
#[test]
fn graphic_watermark_is_placed_scaled_and_blended() {
let dir = std::env::temp_dir().join(format!("lc-wm-logo-{}", std::process::id()));
+7 -14
View File
@@ -374,25 +374,14 @@ fn probe_all(probe: Option<&crate::media::FileProbe>, paths: &[String], progress
}
/// Copy `src` into `root`/`folders` as `name` (default: its own name), made unique with -1, -2…;
/// returns the new path.
/// returns the new path. The copy is verified ([`crate::import_move::copy_new`]: a new file,
/// synced and compared byte for byte); a bad one is removed and the photo reported as failed.
fn copy_into(root: &Path, src: &str, folders: &[String], name: Option<&str>) -> Result<String, String> {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
std::fs::create_dir_all(&dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let name = name
.map(str::to_string)
.unwrap_or_else(|| Path::new(src).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| "photo".into()));
let (stem, ext) = match name.rsplit_once('.') {
Some((s, e)) => (s.to_string(), format!(".{e}")),
None => (name.clone(), String::new()),
};
let mut dst = dir.join(&name);
let mut i = 1;
while dst.exists() {
dst = dir.join(format!("{stem}-{i}{ext}"));
i += 1;
}
std::fs::copy(src, &dst).map_err(|e| format!("copy {src}: {e}"))?;
Ok(dst.to_string_lossy().to_string())
crate::import_move::copy_new(Path::new(src), &dir, &name).map(|p| p.to_string_lossy().to_string())
}
/// What a [`scan`] needs from the session, so it can run on another thread (a folder on a network
@@ -722,6 +711,10 @@ pub fn import_with(s: &mut Session, paths: &[String], opts: &ImportOptions) -> c
}
report.imported.push(id.0);
p.local = opts.local;
if opts.local {
// the state as browsed: a later change keeps the record from being forgotten
p.set_local_baseline();
}
ops.push(Op::AddPhoto { photo: Box::new(p) });
}
let log0 = s.pending_log.len();
+22 -1
View File
@@ -169,6 +169,27 @@ pub(crate) fn place(src: &Path, dir: &Path, name: &str) -> Result<Placed, String
Ok(placed)
}
/// Import → **Copy**: copy `src` into `dir` as `name` (-1, -2… when taken) with the same checks as
/// a move's copy — a new file (never overwriting one, even one that appears meanwhile), synced to
/// disk and compared byte for byte with the source; a bad copy is removed and reported, so a card
/// is never wiped on the strength of it. Returns the new path.
pub(crate) fn copy_new(src: &Path, dir: &Path, name: &str) -> Result<PathBuf, String> {
fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let (stem, ext) = match name.rsplit_once('.') {
Some((s, e)) if !s.is_empty() => (s.to_string(), format!(".{e}")),
_ => (name.to_string(), String::new()),
};
for i in 0..100_000u32 {
let d = if i == 0 { dir.join(name) } else { dir.join(format!("{stem}-{i}{ext}")) };
match copy_verified(src, &d) {
Ok(()) => return Ok(d),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(format!("copy {}: {e} (nothing was imported from it; the original is untouched)", src.display())),
}
}
Err(format!("copy {}: no free name in {}", src.display(), dir.display()))
}
/// Undo a [`place`]: remove what it created at the destination (the sources were never touched).
pub(crate) fn rollback(p: &Placed) {
for sc in p.sidecars.iter().filter(|s| s.created) {
@@ -213,7 +234,7 @@ pub(crate) fn finish(p: &Placed) -> Result<Vec<(String, String)>, String> {
}
/// A photo (by extension) still beside a stem-named sidecar, sharing its stem.
fn sibling_photo(sidecar: &Path) -> Option<String> {
pub(crate) fn sibling_photo(sidecar: &Path) -> Option<String> {
let dir = sidecar.parent()?;
let stem = sidecar.file_stem()?.to_string_lossy().to_string();
fs::read_dir(dir).ok()?.flatten().map(|e| e.path()).find_map(|f| {
+35 -6
View File
@@ -24,6 +24,7 @@ pub mod library;
pub mod media;
pub mod memory;
pub mod merge;
pub mod originals;
pub mod preset_import;
pub mod preset_luminar;
pub mod presets;
@@ -52,6 +53,10 @@ pub enum EngineError {
BadParams { cmd: String, msg: String },
#[error("{0}")]
Catalog(#[from] lightcraft_catalog::CatalogError),
/// The command's change is applied (in memory, undoable) but its journal records could not
/// be written. They stay queued and are written by the next successful save.
#[error("saved in memory but not written to disk: {0}; LightCraft will retry")]
NotSaved(String),
#[error("{0}")]
Other(String),
}
@@ -168,6 +173,9 @@ pub struct Session {
pub cache_mb: u32,
/// Where smart previews are kept when not in the library folder (persisted in prefs.json).
pub smart_previews_dir: Option<std::path::PathBuf>,
/// Untouched Local records of folders not browsed for this many days are forgotten when the
/// library opens (0 = never; persisted in prefs.json). See `cmd/browse.rs`.
pub forget_local_days: u32,
}
impl Default for Session {
@@ -228,6 +236,7 @@ impl Session {
import_defaults: import::ImportDefaults::default(),
cache_mb: 0,
smart_previews_dir: None,
forget_local_days: lightcraft_catalog::DEFAULT_FORGET_DAYS,
}
}
@@ -266,9 +275,18 @@ impl Session {
self.auto_write_sidecars(&self.pending_log[log_start..]);
}
if self.depth == 0 && self.library.is_some() {
// Make the command durable before reporting success (on failure the ops stay pending,
// are retried after the next command, and `library.info` reports the error).
let _ = self.persist();
// Make the command durable before reporting success. When the command's own records
// can't be appended, it fails with `NotSaved`: the change stays applied in memory and
// queued, and the next save (any later command, or the frame loop) retries it. Other
// persistence trouble (a failed compaction, an older queued change still unwritten by
// a command that changed nothing) doesn't fail the command; `library.info` reports it.
let produced = self.pending_log.len() > log_start;
if let Err(e @ EngineError::NotSaved(_)) = self.persist()
&& produced
&& r.is_ok()
{
return Err(e);
}
}
r
}
@@ -361,15 +379,22 @@ impl Session {
Ok(e.label)
}
/// Apply an undo/redo op, first moving the files its renames imply (all or nothing).
/// Apply an undo/redo op, first moving the files its renames imply (all or nothing). Files
/// that can't be moved back after a failure are reported and the library follows them (a
/// logged change outside the undo history), so none goes missing.
fn apply_with_files(&mut self, op: &Op) -> Result<Op> {
let fs = rename::RealFs;
let moves = self.file_moves(op);
Session::move_files(&moves)?;
if let Err(e) = rename::move_all(&fs, &moves) {
return Err(EngineError::Other(format!("can't move the files back: {}", self.follow_stuck(op, e, false))));
}
match self.catalog.apply(op.clone()) {
Ok(inv) => Ok(inv),
Err(e) => {
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
let _ = Session::move_files(&back);
if let Err(be) = rename::move_all(&fs, &back) {
return Err(EngineError::Other(format!("{e}; the files could not all be moved back: {}", be.message)));
}
Err(e.into())
}
}
@@ -599,6 +624,8 @@ mod tests_color;
#[cfg(test)]
mod tests_export;
#[cfg(test)]
mod tests_forget_local;
#[cfg(test)]
mod tests_import;
#[cfg(test)]
mod tests_import_move;
@@ -611,6 +638,8 @@ mod tests_merge;
#[cfg(test)]
mod tests_organize;
#[cfg(test)]
mod tests_persist;
#[cfg(test)]
mod tests_prefs;
#[cfg(test)]
mod tests_spots;
+119 -13
View File
@@ -14,7 +14,10 @@
//! via [`Session::open_library_in`] (the browser build keeps them in OPFS or IndexedDB).
//!
//! Every top-level [`Session::execute`] persists the ops it produced (fsynced) before returning,
//! so a crash loses at most the command in flight. The log is compacted into a snapshot when it
//! so a crash loses at most the command in flight. When that write fails the command returns
//! [`EngineError::NotSaved`]: its change stays applied in memory and queued, and every later save
//! retries the queue ([`Session::unsaved`] reports it meanwhile), so nothing is lost once the
//! disk is writable again. The log is compacted into a snapshot when it
//! grows (see [`lightcraft_catalog::SnapshotPolicy`]; written by a worker thread on native, see
//! [`lightcraft_catalog::journal`]) and on [`Session::close_library`].
@@ -52,6 +55,13 @@ pub struct Library {
pub report: LoadReport,
/// Last persistence error (shown by the UI; ops stay pending and are retried).
pub last_error: Option<String>,
/// Why the last append of queued ops failed; `None` once a save succeeds. While set, changes
/// exist only in memory ([`Session::unsaved`]).
pub unsaved_error: Option<String>,
/// When the frame loop may retry a failed append ([`Session::persist_if_dirty`] backs off).
retry_at: Option<web_time::Instant>,
/// What forgetting untouched Local records did when the library opened.
pub forgot_local: Option<lightcraft_catalog::ForgetPlan>,
presets_written: String,
view_written: Vec<u8>,
}
@@ -133,6 +143,9 @@ struct PrefsFile {
cache_mb: u32,
/// Folder for smart previews (default: `Smart Previews` in the library).
smart_previews_dir: Option<String>,
/// Days after which untouched Local records of unbrowsed folders are forgotten (missing =
/// the default, 0 = never).
forget_local_days: Option<u32>,
}
fn presets_json(s: &Session) -> String {
@@ -155,6 +168,9 @@ fn poll_compaction(lib: &mut Library) {
}
}
/// How long the frame loop waits before retrying a failed append (commands retry at once).
const RETRY_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2);
fn read_json<T: serde::de::DeserializeOwned>(store: &mut dyn Store, name: &str) -> Option<T> {
store.read(name).ok().flatten().and_then(|b| serde_json::from_slice(&b).ok())
}
@@ -222,6 +238,7 @@ impl Session {
self.recent_keywords = prefs.recent_keywords;
self.import_defaults = prefs.import;
self.cache_mb = prefs.cache_mb;
self.forget_local_days = prefs.forget_local_days.unwrap_or(lightcraft_catalog::DEFAULT_FORGET_DAYS);
self.smart_previews_dir = prefs.smart_previews_dir.filter(|_| on_disk).map(PathBuf::from);
if let Some(d) = &self.smart_previews_dir {
self.media.smart_dir = Some(d.clone());
@@ -248,28 +265,70 @@ impl Session {
self.media.attach_disk_cache(&dir.join("thumbs"), self.cache_bytes());
}
let view_written = self.view_json();
let library = self.library.insert(Library { dir, on_disk, journal, files, report, last_error: None, presets_written, view_written });
Ok(&library.report)
self.library = Some(Library {
dir,
on_disk,
journal,
files,
report,
last_error: None,
unsaved_error: None,
retry_at: None,
forgot_local: None,
presets_written,
view_written,
});
// forget untouched Local records of folders not browsed for a while (journaled at once)
if self.forget_local_days > 0 {
let plan = self.forget_local(false, None);
if !plan.evict.is_empty() {
self.compact_soon();
} else if let Err(e) = self.persist() {
log::error!("library: {e}");
}
if let Some(lib) = self.library.as_mut() {
lib.forgot_local = Some(plan);
}
}
match &self.library {
Some(lib) => Ok(&lib.report),
None => Err(EngineError::Other("library closed while opening".into())),
}
}
/// Write pending ops to the log (fsynced), compact when due, and save changed presets.
/// Called after every top-level command; cheap when nothing changed.
/// Called after every top-level command; cheap when nothing changed. Fails (with
/// [`EngineError::NotSaved`]) only when the queued ops couldn't be written; they stay queued.
pub fn persist(&mut self) -> Result<()> {
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if !self.pending_log.is_empty() {
if let Err(e) = lib.journal.append(&self.pending_log) {
lib.last_error = Some(e.to_string());
log::error!("library: {e}");
return Err(e.into());
// the ops stay queued (and applied in memory): the next persist retries them
let reason = e.to_string();
log::error!("library: {} change(s) not written to disk: {reason}", self.pending_log.len());
lib.last_error = Some(reason.clone());
lib.unsaved_error = Some(reason.clone());
lib.retry_at = Some(web_time::Instant::now() + RETRY_BACKOFF);
return Err(EngineError::NotSaved(reason));
}
if lib.unsaved_error.take().is_some() {
log::info!("library: queued changes written to disk");
}
lib.retry_at = None;
self.pending_log.clear();
lib.last_error = None;
}
poll_compaction(lib);
// Never snapshot mid-interaction: the catalog then holds an uncommitted preview value.
// The snapshot is written by a worker thread (natively); appends go on meanwhile.
if lib.journal.wants_snapshot() && self.interaction.is_none() {
lib.journal.snapshot_in_background(&self.catalog)?;
// A failed compaction loses nothing (the log is kept whole and compacted later): it is
// reported (`library.info` → `lastError`), not returned.
if lib.journal.wants_snapshot()
&& self.interaction.is_none()
&& let Err(e) = lib.journal.snapshot_in_background(&self.catalog)
{
log::error!("library: compaction: {e}");
lib.last_error = Some(format!("compaction: {e}"));
}
let presets = presets_json(self);
let Some(lib) = self.library.as_mut() else { return Ok(()) };
@@ -286,24 +345,52 @@ impl Session {
/// Persist if commands left ops pending (cheap; frontends call it once per frame for state
/// changed outside [`Session::execute`]).
pub fn persist_if_dirty(&mut self) {
if self.library.is_some() && !self.pending_log.is_empty() {
let backing_off = self.library.as_ref().and_then(|l| l.retry_at).is_some_and(|t| web_time::Instant::now() < t);
if self.library.is_some() && !self.pending_log.is_empty() && !backing_off {
let _ = self.persist();
} else if let Some(lib) = self.library.as_mut() {
poll_compaction(lib);
}
}
/// Changes applied in memory but not yet written to disk because saving failed: the number of
/// queued ops and why (`None` when everything is saved, or no library is open). Retried by
/// every command and by [`Session::persist_if_dirty`].
pub fn unsaved(&self) -> Option<(usize, &str)> {
let lib = self.library.as_ref()?;
lib.unsaved_error.as_deref().filter(|_| !self.pending_log.is_empty()).map(|e| (self.pending_log.len(), e))
}
/// Flush everything and write a snapshot (on quit). Ends an open interaction first.
///
/// The snapshot is written even when appending the queued ops fails: it is a fresh file
/// holding everything in memory, so it saves those ops too (the log handle may be the only
/// thing that's broken). Fails only if nothing could be saved; the ops then stay queued.
pub fn close_library(&mut self) -> Result<()> {
if self.library.is_none() {
return Ok(());
}
let _ = self.end_interaction();
self.persist()?;
let persisted = self.persist();
self.save_view();
let unlogged = if persisted.is_err() { self.pending_log.len() as u64 } else { 0 };
let Some(lib) = self.library.as_mut() else { return Ok(()) };
lib.journal.snapshot(&self.catalog)?;
Ok(())
let before = lib.journal.seq();
let snapshot = lib.journal.snapshot_with_unlogged(&self.catalog, unlogged);
if unlogged > 0 && lib.journal.seq() == before + unlogged {
// the snapshot holds the queued ops: they are saved, never append them again
log::info!("library: {unlogged} queued change(s) saved in the closing snapshot");
self.pending_log.clear();
lib.unsaved_error = None;
lib.retry_at = None;
lib.last_error = None;
}
match (persisted, snapshot) {
(_, Ok(())) => Ok(()),
// neither the log nor the snapshot took the queued ops
(Err(EngineError::NotSaved(e)), Err(s)) if lib.journal.seq() == before => Err(EngineError::NotSaved(format!("{e}; snapshot: {s}"))),
(_, Err(s)) => Err(s.into()),
}
}
fn view_json(&self) -> Vec<u8> {
@@ -351,6 +438,7 @@ impl Session {
import: self.import_defaults.clone(),
cache_mb: self.cache_mb,
smart_previews_dir: self.smart_previews_dir.as_ref().map(|d| d.to_string_lossy().to_string()),
forget_local_days: Some(self.forget_local_days),
})
.unwrap_or_default();
let Some(lib) = self.library.as_mut() else { return Ok(()) };
@@ -373,6 +461,24 @@ impl Session {
self.save_prefs()
}
/// Save, then start compacting in the background (natively; synchronously elsewhere): after
/// a change that shrinks the catalog a lot, so the smaller snapshot replaces the old one.
/// Errors are logged and reported like other persistence errors (nothing is lost).
pub fn compact_soon(&mut self) {
if let Err(e) = self.persist() {
log::error!("library: {e}");
return;
}
if self.interaction.is_some() {
return;
}
let Some(lib) = self.library.as_mut() else { return };
if let Err(e) = lib.journal.snapshot_in_background(&self.catalog) {
log::error!("library: compaction: {e}");
lib.last_error = Some(format!("compaction: {e}"));
}
}
/// Compact the log into a snapshot now.
pub fn compact_library(&mut self) -> Result<()> {
self.persist()?;
+1 -1
View File
@@ -290,7 +290,7 @@ impl MediaCache {
&& !std::path::Path::new(path).exists()
{
let sp = dir.join(crate::smart::file_name(p));
if sp.exists() {
if crate::smart::is_valid(&sp) {
return SourceRef::Smart { path: sp };
}
}
+6 -1
View File
@@ -44,7 +44,12 @@ fn total_ram() -> Option<usize> {
let out = std::process::Command::new("/usr/sbin/sysctl").args(["-n", "hw.memsize"]).output().ok()?;
String::from_utf8_lossy(&out.stdout).trim().parse().ok()
}
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
#[cfg(target_os = "freebsd")]
{
let out = std::process::Command::new("/sbin/sysctl").args(["-n", "hw.physmem"]).output().ok()?;
String::from_utf8_lossy(&out.stdout).trim().parse().ok()
}
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")))]
{
None
}
+9 -8
View File
@@ -304,15 +304,13 @@ fn apply_auto(s: &mut DevelopSettings, src: &Rgb32f, info: &SourceInfo) {
s.color.saturation = a.saturation;
}
/// A free file name `dir/<stem>-<suffix>.dng` (`-2`, `-3`… when taken).
fn output_path(first: &str, suffix: &str) -> String {
/// `dir/<stem>-<suffix>.dng`, then `-2`, `-3`… (the candidates for a merge's output).
fn output_names(first: &str, suffix: &str) -> impl Iterator<Item = std::path::PathBuf> {
let p = std::path::Path::new(first);
let dir = p.parent().map(|d| d.to_path_buf()).unwrap_or_default();
let stem = p.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| "Merge".into());
let cand = |k: u64| dir.join(if k == 1 { format!("{stem}-{suffix}.dng") } else { format!("{stem}-{suffix}-{k}.dng") });
// (`u64::MAX` taken names can't exist; the last candidate is the fallback)
let free = (1..u64::MAX).map(cand).find(|c| !c.exists()).unwrap_or_else(|| cand(u64::MAX));
free.to_string_lossy().to_string()
let suffix = suffix.to_string();
(1u64..).map(move |k| dir.join(if k == 1 { format!("{stem}-{suffix}.dng") } else { format!("{stem}-{suffix}-{k}.dng") }))
}
/// Parse `merge.*` command parameters into a kind and finishing options.
@@ -367,8 +365,11 @@ impl Session {
/// apply Auto Settings / the auto crop. Returns `{id, path, …info}`.
pub fn finish_merge(&mut self, job: &MergeJob, out: MergeOutput) -> Result<Value> {
let first = &job.sources.first().ok_or_else(|| EngineError::Other("nothing merged".into()))?.1;
let path = output_path(first, job.kind.suffix());
std::fs::write(&path, &out.dng).map_err(|e| EngineError::Other(format!("{path}: {e}")))?;
// a new file under a free name (never replacing one), complete and synced before it appears
let path = lightcraft_catalog::safe_file::write_new_unique(&mut output_names(first, job.kind.suffix()), &out.dng)
.map_err(|e| EngineError::Other(format!("could not write the merged DNG next to {first}: {e}")))?
.to_string_lossy()
.to_string();
let report = crate::import::import(self, std::slice::from_ref(&path), crate::import::ImportMode::Add)?;
let id = report
.imported
+102
View File
@@ -0,0 +1,102 @@
//! Never write over an original: exports, renders and other files LightCraft writes for the user
//! must not land on a catalogued photo's own file (or its XMP sidecar), whatever the conflict
//! setting or exact path asked for. [`OriginalGuard`] is a snapshot of the library's originals
//! that answers "would writing here replace one?" by file identity (the same file under another
//! spelling, through `..`, a link or a case-insensitive volume counts), not by comparing strings.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use lightcraft_catalog::safe_file::same_file;
use lightcraft_catalog::{Catalog, Source};
/// The library's originals and their sidecars, looked up by file name.
#[derive(Debug, Default)]
pub struct OriginalGuard {
/// lower-case file name → (path on disk, what it is: "the original of IMG_1.CR3", …)
by_name: HashMap<String, Vec<(PathBuf, String)>>,
}
impl OriginalGuard {
/// The originals of every photo in `catalog`.
pub fn new(catalog: &Catalog) -> Self {
let mut g = Self::default();
for p in catalog.photos() {
if let Source::File { path } = &p.source {
g.add(Path::new(path), &p.file_name);
}
}
g
}
/// Protect `path` (an original named `label`) and its sidecars (`IMG_1.xmp`, `IMG_1.CR3.xmp`).
pub fn add(&mut self, path: &Path, label: &str) {
let Some(name) = path.file_name().map(|n| n.to_string_lossy().to_string()) else { return };
let mut put = |file: String, p: PathBuf, what: String| self.by_name.entry(file.to_lowercase()).or_default().push((p, what));
put(name.clone(), path.to_path_buf(), format!("the original of {label}"));
let stem = path.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_default();
put(format!("{stem}.xmp"), path.with_extension("xmp"), format!("the XMP sidecar of {label}"));
put(format!("{name}.xmp"), PathBuf::from(format!("{}.xmp", path.to_string_lossy())), format!("the XMP sidecar of {label}"));
}
/// `Err` (a message for the user) when writing `target` would replace a protected file.
pub fn check(&self, target: &Path) -> Result<(), String> {
if std::fs::symlink_metadata(target).is_err() {
return Ok(()); // nothing there to replace
}
let mut names: Vec<String> = target.file_name().map(|n| n.to_string_lossy().to_lowercase()).into_iter().collect();
if let Some(n) = std::fs::canonicalize(target).ok().and_then(|c| c.file_name().map(|n| n.to_string_lossy().to_lowercase())) {
names.push(n);
}
for n in &names {
for (p, what) in self.by_name.get(n).into_iter().flatten() {
if same_file(p, target) {
return Err(format!(
"{} is {what} in the library: LightCraft never writes over an original (choose another folder or file name)",
target.display()
));
}
}
}
Ok(())
}
}
impl crate::Session {
/// A guard over this library's originals (see [`OriginalGuard`]).
pub fn original_guard(&self) -> OriginalGuard {
OriginalGuard::new(&self.catalog)
}
/// `Err` when writing `path` would replace a catalogued photo's original or sidecar.
pub fn check_write_target(&self, path: &str) -> Result<(), String> {
self.original_guard().check(Path::new(path))
}
}
#[cfg(test)]
mod tests {
use super::*;
use lightcraft_catalog::{Op, Photo, PhotoId};
#[test]
fn originals_and_their_sidecars_are_protected_by_identity() {
let d = std::env::temp_dir().join(format!("lc-originals-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(d.join("sub")).unwrap();
let orig = d.join("IMG_1.jpg");
std::fs::write(&orig, b"original").unwrap();
std::fs::write(d.join("IMG_1.xmp"), b"<x/>").unwrap();
std::fs::write(d.join("other.jpg"), b"other").unwrap();
let mut c = Catalog::default();
let p = Photo::new(PhotoId(1), Source::File { path: orig.to_string_lossy().to_string() }, "IMG_1.jpg", "JPEG", 10, 10, "2026-01-01");
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
let g = OriginalGuard::new(&c);
assert!(g.check(&orig).unwrap_err().contains("original of IMG_1.jpg"));
assert!(g.check(&d.join("sub/../IMG_1.jpg")).is_err(), "another spelling of the same file");
assert!(g.check(&d.join("IMG_1.xmp")).unwrap_err().contains("sidecar"));
assert!(g.check(&d.join("other.jpg")).is_ok(), "an ordinary existing file");
assert!(g.check(&d.join("new.jpg")).is_ok(), "a free name");
let _ = std::fs::remove_dir_all(&d);
}
}
+522 -65
View File
@@ -11,7 +11,11 @@
//!
//! Safety:
//! - a target that exists on disk, or that another photo in the batch gets, receives a `-1`, `-2`…
//! suffix — no file is ever overwritten;
//! suffix — no file is ever overwritten. Whether an existing target is the photo itself (a
//! change of letter case on a case-insensitive volume) is decided by file identity, not by
//! comparing names, so on a case-sensitive volume `img_1.jpg` next to `IMG_1.JPG` is a collision
//! (issue #95); moves never replace a file even if one appears after the check (hard link +
//! unlink where the volume supports it);
//! - the moves happen before the catalog changes; if one fails, the ones already done are moved
//! back and nothing is committed;
//! - the catalog op ([`Op::SetFile`]) is undoable: undo/redo move the files back and forth (see
@@ -297,58 +301,251 @@ fn sidecars(path: &str) -> Vec<(PathBuf, SidecarNaming, bool)> {
v
}
/// Move `from` to `to` (never over an existing file) together with its sidecars. On error nothing
/// is left moved.
pub fn move_file(from: &str, to: &str) -> std::result::Result<(), String> {
/// The file-system primitives renaming uses. [`RealFs`] is the disk; tests substitute a model of
/// a case-insensitive or case-sensitive volume (and inject failures).
pub(crate) trait MoveFs {
/// Something (file, folder, link) answers to `p`.
fn exists(&self, p: &Path) -> bool;
fn is_file(&self, p: &Path) -> bool;
/// `a` and `b` name the same file (e.g. `IMG.JPG` and `img.jpg` on a case-insensitive
/// volume). Decided by file identity, never by comparing names: on a case-sensitive volume
/// they are two different photos. When in doubt: `false`.
fn same_file(&self, a: &Path, b: &Path) -> bool;
/// Rename `a` to `b`, failing (never replacing) when `b` exists.
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()>;
/// Copy `a` to a new file `b` (an independent copy, never a link), failing when `b` exists.
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()>;
fn remove_file(&self, p: &Path) -> std::io::Result<()>;
/// Another photo file beside the stem-named sidecar `sidecar` that shares its stem (e.g.
/// `IMG_1.JPG` for `IMG_1.xmp`), which still uses that sidecar.
fn stem_sibling(&self, sidecar: &Path) -> Option<String>;
}
/// The real file system.
pub(crate) struct RealFs;
impl MoveFs for RealFs {
fn exists(&self, p: &Path) -> bool {
std::fs::symlink_metadata(p).is_ok()
}
fn is_file(&self, p: &Path) -> bool {
p.is_file()
}
fn same_file(&self, a: &Path, b: &Path) -> bool {
if a == b {
return true;
}
let (Ok(ma), Ok(mb)) = (std::fs::metadata(a), std::fs::metadata(b)) else { return false };
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
ma.dev() == mb.dev() && ma.ino() == mb.ino()
}
#[cfg(not(unix))]
{
// no stable file id in std here: `b` is `a` under another spelling when they look
// alike and the folder has no entry spelled exactly like `b` (on a case-sensitive
// folder holding both `IMG.JPG` and `img.jpg`, both entries are listed)
let (Some(dir), Some(name)) = (b.parent(), b.file_name()) else { return false };
if a.parent() != Some(dir) || ma.len() != mb.len() || ma.modified().ok() != mb.modified().ok() {
return false;
}
let Ok(entries) = std::fs::read_dir(dir) else { return false };
!entries.flatten().any(|e| e.file_name() == name)
}
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, format!("{} already exists", b.display())));
}
// A hard link fails when `b` exists — even if it appeared since the check above — so
// nothing is ever replaced. Volumes without hard links (FAT, some network shares) fall back
// to a plain rename after the check.
match std::fs::hard_link(a, b) {
Ok(()) => std::fs::remove_file(a).inspect_err(|_| {
// both names are the same file: dropping the new one loses nothing
let _ = std::fs::remove_file(b);
}),
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => Err(e),
Err(_) => std::fs::rename(a, b),
}
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
use std::io::Write;
let bytes = std::fs::read(a)?;
let mut f = std::fs::OpenOptions::new().write(true).create_new(true).open(b)?;
f.write_all(&bytes).and_then(|_| f.sync_all()).inspect_err(|_| {
let _ = std::fs::remove_file(b);
})
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
std::fs::remove_file(p)
}
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
crate::import_move::sibling_photo(sidecar)
}
}
/// `from` → `to` differs only in letter case and `to` is that very file (a case-insensitive
/// volume): the rename must go through a temporary name.
fn is_respelling(fs: &dyn MoveFs, from: &Path, to: &Path) -> bool {
from != to && from.to_string_lossy().to_lowercase() == to.to_string_lossy().to_lowercase() && fs.exists(to) && fs.same_file(from, to)
}
/// Rename `a` to `b` without ever replacing a file; a change of letter case only goes through a
/// unique temporary name so case-insensitive volumes see a real change.
fn move_one(fs: &dyn MoveFs, a: &Path, b: &Path) -> std::io::Result<()> {
if !is_respelling(fs, a, b) {
return fs.rename_no_replace(a, b);
}
let tmp = (0..1000u32)
.map(|k| a.with_file_name(format!(".lc-rename-{}-{k}", std::process::id())))
.find(|t| !fs.exists(t))
.ok_or_else(|| std::io::Error::other("no free temporary name"))?;
fs.rename_no_replace(a, &tmp)?;
fs.rename_no_replace(&tmp, b).map_err(|e| match fs.rename_no_replace(&tmp, a) {
Ok(()) => e,
Err(e2) => std::io::Error::other(format!("{e}; it could not be given its old name back either ({e2}): it is now {}", tmp.display())),
})
}
/// Why [`move_file`] failed, and what it left behind.
#[derive(Clone, Debug, PartialEq)]
pub struct MoveError {
/// What went wrong, including every step that could not be undone and where that file is now.
pub message: String,
/// The file itself ended up at the new path: it moved, a sidecar didn't, and moving the file
/// back failed too. Its sidecars that did move stay with it.
pub moved: bool,
}
impl std::fmt::Display for MoveError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.message)
}
}
/// Move `from` to `to` (never over an existing file) together with its sidecars. On error the
/// moves are undone; whatever can't be undone is named in the error (see [`MoveError`]).
pub fn move_file(from: &str, to: &str) -> std::result::Result<(), MoveError> {
move_file_with(&RealFs, from, to)
}
/// [`move_file`] on the file system `fs`.
pub(crate) fn move_file_with(fs: &dyn MoveFs, from: &str, to: &str) -> std::result::Result<(), MoveError> {
if from == to {
return Ok(());
}
let fail = |message: String| MoveError { message, moved: false };
let (f, t) = (Path::new(from), Path::new(to));
// a case-only change on a case-insensitive file system "exists" already: that is this file
let case_only = from.to_lowercase() == to.to_lowercase();
if t.exists() && !case_only {
return Err(format!("{to} already exists"));
// `to` may be this very file spelled differently (case-insensitive volume); any other file
// there — e.g. `img.jpg` next to `IMG.JPG` on a case-sensitive volume — is never replaced
if fs.exists(t) && !is_respelling(fs, f, t) {
return Err(fail(format!("{to} already exists")));
}
let mut done: Vec<(PathBuf, PathBuf)> = Vec::new();
let mv = |a: &Path, b: &Path| -> std::io::Result<()> {
if case_only {
// via a temporary name, so case-insensitive file systems see a real change
let tmp = a.with_file_name(format!(".lc-rename-{}", std::process::id()));
std::fs::rename(a, &tmp)?;
std::fs::rename(&tmp, b).inspect_err(|_| {
let _ = std::fs::rename(&tmp, a);
})
} else {
std::fs::rename(a, b)
}
};
mv(f, t).map_err(|e| format!("rename {from}: {e}"))?;
done.push((f.to_path_buf(), t.to_path_buf()));
move_one(fs, f, t).map_err(|e| fail(format!("rename {from}: {e}")))?;
// sidecars moved (or copied: `true`)
let mut done: Vec<(PathBuf, PathBuf, bool)> = Vec::new();
for (sc, naming, upper) in sidecars(from) {
if !sc.is_file() {
if !fs.is_file(&sc) {
continue;
}
let mut dst = crate::sidecar::sidecar_path(to, naming);
if upper {
dst = dst.with_extension("XMP");
}
if dst.exists() && !case_only {
// a stem sidecar another file still uses (`IMG_1.xmp` of `IMG_1.CR3` and `IMG_1.JPG`)
// is copied, not taken away from it (issue #92)
let shared = naming == SidecarNaming::Stem && fs.stem_sibling(&sc).is_some();
if fs.exists(&dst) && (shared || !is_respelling(fs, &sc, &dst)) {
continue; // never overwrite: the old sidecar stays where it was
}
if let Err(e) = mv(&sc, &dst) {
for (a, b) in done.iter().rev() {
let _ = std::fs::rename(b, a);
let r = if shared { fs.copy_no_replace(&sc, &dst) } else { move_one(fs, &sc, &dst) };
if let Err(e) = r {
let mut message = format!("rename {}: {e}", sc.display());
// the file first: if it can't go back, the sidecars that moved stay with it
if let Err(e) = move_one(fs, t, f) {
message.push_str(&format!("; {from} could not be moved back ({e}): it is now {to}"));
if !done.is_empty() {
let names: Vec<String> = done.iter().map(|(_, b, _)| b.display().to_string()).collect();
message.push_str(&format!(" with its sidecar {}", names.join(", ")));
}
return Err(MoveError { message, moved: true });
}
return Err(format!("rename {}: {e}", sc.display()));
for (a, b, copied) in done.iter().rev() {
if *copied {
// the original is still there: the copy just goes (a leftover copy loses nothing)
let _ = fs.remove_file(b);
} else if let Err(e) = move_one(fs, b, a) {
message.push_str(&format!("; sidecar {} could not be moved back ({e}): it is now {}", a.display(), b.display()));
}
}
return Err(fail(message));
}
done.push((sc, dst));
done.push((sc, dst, shared));
}
Ok(())
}
/// What a failed batch of moves ([`move_all`]) left behind.
#[derive(Debug)]
pub(crate) struct BatchMoveError {
/// The first failure and every move that could not be undone.
pub message: String,
/// Moves that stayed done, (from, to): those files are at `to` now.
pub stuck: Vec<(String, String)>,
}
/// Move the files `moves` names, all or nothing: after a failure the earlier moves are undone.
/// Moves that can't be undone (e.g. the network share went away) are reported, not ignored.
pub(crate) fn move_all(fs: &dyn MoveFs, moves: &[(String, String)]) -> std::result::Result<(), BatchMoveError> {
let mut done: Vec<&(String, String)> = Vec::new();
for m in moves {
let Err(e) = move_file_with(fs, &m.0, &m.1) else {
done.push(m);
continue;
};
let mut message = e.message;
let mut stuck = Vec::new();
if e.moved {
stuck.push(m.clone());
}
for d in done.iter().rev() {
if let Err(e) = move_file_with(fs, &d.1, &d.0) {
message.push_str(&format!("; {} could not be moved back to {}: {}", d.1, d.0, e.message));
if !e.moved {
stuck.push((*d).clone());
}
}
}
return Err(BatchMoveError { message, stuck });
}
Ok(())
}
/// The `SetFile` ops in `op` (nested batches included) that point a photo at one of `to`.
fn set_file_ops_to(op: &Op, to: &HashSet<&str>, out: &mut Vec<Op>) {
match op {
Op::Batch { ops } => ops.iter().for_each(|o| set_file_ops_to(o, to, out)),
Op::SetFile { source: Source::File { path }, .. } if to.contains(path.as_str()) => out.push(op.clone()),
_ => {}
}
}
impl Session {
/// Plan renaming `ids` with `template` (sequence numbers from `start`), resolving collisions.
pub fn plan_rename(&self, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
self.plan_rename_with(&RealFs, ids, template, start)
}
/// [`Session::plan_rename`] on the file system `fs`.
pub(crate) fn plan_rename_with(&self, fs: &dyn MoveFs, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
// photos sharing one file (virtual copies) rename once
let mut seen_paths: HashSet<String> = HashSet::new();
let mut taken: HashSet<String> = HashSet::new(); // lower-case target paths/names claimed in this batch
@@ -382,10 +579,12 @@ impl Session {
let name = candidate(k);
let tp = dir.join(&name).to_string_lossy().to_string();
let key = tp.to_lowercase();
let same = key == path.to_lowercase();
// this very file, maybe spelled differently (case-insensitive volume)? By
// identity: on a case-sensitive volume `img_1.jpg` may be another photo
let same = tp == *path || is_respelling(fs, Path::new(path), Path::new(&tp));
// free: not claimed in this batch and not on disk (unless it is this very file).
// A file this batch moves away still counts as taken: simple and safe.
let on_disk = Path::new(&tp).exists() && !same;
let on_disk = fs.exists(Path::new(&tp)) && !same;
if !taken.contains(&key) && !on_disk {
taken.insert(key);
break (name, Some(tp));
@@ -415,21 +614,16 @@ impl Session {
/// Carry out `plans`: move the files (rolled back on failure), then commit one undoable op
/// that updates the photos (and their virtual copies).
///
/// When a failure can't be fully rolled back, the files that kept their new names are
/// committed as a smaller undoable rename, so the library points at them (nothing goes
/// missing), and the error lists them (old → new).
pub fn apply_rename(&mut self, plans: &[RenamePlan]) -> Result<usize> {
let mut moved: Vec<(String, String)> = Vec::new();
for pl in plans {
if let (Some(a), Some(b)) = (&pl.from_path, &pl.to_path)
&& a != b
{
if let Err(e) = move_file(a, b) {
for (a, b) in moved.iter().rev() {
let _ = move_file(b, a);
}
return Err(EngineError::Other(e));
}
moved.push((a.clone(), b.clone()));
}
}
self.apply_rename_with(&RealFs, plans)
}
/// [`Session::apply_rename`] on the file system `fs`.
pub(crate) fn apply_rename_with(&mut self, fs: &dyn MoveFs, plans: &[RenamePlan]) -> Result<usize> {
let mut ops = Vec::new();
let by_path: HashMap<&str, &RenamePlan> = plans.iter().filter_map(|p| p.from_path.as_deref().map(|f| (f, p))).collect();
for p in self.catalog.photos() {
@@ -447,17 +641,66 @@ impl Session {
}
}
let n = ops.len();
let op = Op::Batch { ops };
let moves: Vec<(String, String)> = plans
.iter()
.filter_map(|pl| match (&pl.from_path, &pl.to_path) {
(Some(a), Some(b)) if a != b => Some((a.clone(), b.clone())),
_ => None,
})
.collect();
if let Err(e) = move_all(fs, &moves) {
return Err(EngineError::Other(self.follow_stuck(&op, e, true)));
}
if n > 0
&& let Err(e) = self.commit(&format!("Rename {n} Photo{}", if n == 1 { "" } else { "s" }), Op::Batch { ops })
&& let Err(err) = self.commit(&format!("Rename {n} Photo{}", if n == 1 { "" } else { "s" }), op)
{
for (a, b) in moved.iter().rev() {
let _ = move_file(b, a);
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
if let Err(e) = move_all(fs, &back) {
return Err(EngineError::Other(format!("{err}; the renamed files could not all be moved back: {}", e.message)));
}
return Err(e);
return Err(err);
}
Ok(n)
}
/// After a failed batch of moves for `op`: point the photos whose files stayed moved at
/// their new paths (an undoable "partly" rename when `undoable`, otherwise a plain logged
/// change), so the library matches the disk. Returns the message for the user.
pub(crate) fn follow_stuck(&mut self, op: &Op, e: BatchMoveError, undoable: bool) -> String {
let mut msg = e.message;
if e.stuck.is_empty() {
return msg;
}
let to: HashSet<&str> = e.stuck.iter().map(|m| m.1.as_str()).collect();
let mut keep = Vec::new();
set_file_ops_to(op, &to, &mut keep);
let list: Vec<String> = e.stuck.iter().map(|(a, b)| format!("{a} → {b}")).collect();
let n = e.stuck.len();
let s = if n == 1 { "" } else { "s" };
let fix = Op::Batch { ops: keep };
let applied = if undoable {
self.commit(&format!("Rename {n} Photo{s} (partly)"), fix)
} else {
match self.catalog.apply(fix.clone()) {
Ok(_) => {
self.pending_log.push(fix);
Ok(())
}
Err(e) => Err(e.into()),
}
};
match applied {
Ok(()) => msg
.push_str(&format!(". {n} file{s} could not be moved back and kept the new name; the library now points there: {}", list.join(", "))),
Err(err) => msg.push_str(&format!(
". {n} file{s} could not be moved back and the library could not follow ({err}); relink with Find Missing Photos: {}",
list.join(", ")
)),
}
msg
}
/// File moves an undo/redo op implies: `SetFile` ops whose source path differs from the photo's
/// current one.
pub(crate) fn file_moves(&self, op: &Op) -> Vec<(String, String)> {
@@ -479,26 +722,240 @@ impl Session {
rec(self, op, &mut v);
v
}
/// Move the files `moves` names; all or nothing.
pub(crate) fn move_files(moves: &[(String, String)]) -> Result<()> {
let mut done: Vec<&(String, String)> = Vec::new();
for m in moves {
if let Err(e) = move_file(&m.0, &m.1) {
for (a, b) in done.iter().rev().map(|m| (&m.0, &m.1)) {
let _ = move_file(b, a);
}
return Err(EngineError::Other(format!("can't undo the rename: {e}")));
}
done.push(m);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::cell::RefCell;
/// A model volume: files by path with contents; case-insensitive (default APFS/NTFS) or
/// case-sensitive (Linux, case-sensitive APFS). `fail` makes renames from a matching source fail.
pub(crate) struct FakeFs {
pub ci: bool,
pub files: RefCell<Vec<(String, Vec<u8>)>>,
pub fail: RefCell<Vec<String>>,
}
impl FakeFs {
pub fn new(ci: bool, files: &[(&str, &str)]) -> FakeFs {
FakeFs {
ci,
files: RefCell::new(files.iter().map(|(p, c)| (p.to_string(), c.as_bytes().to_vec())).collect()),
fail: RefCell::new(Vec::new()),
}
}
fn idx(&self, p: &Path) -> Option<usize> {
let p = p.to_string_lossy();
self.files.borrow().iter().position(|(f, _)| if self.ci { f.to_lowercase() == p.to_lowercase() } else { *f == p })
}
/// The listing: (exact path, contents), sorted.
pub fn listing(&self) -> Vec<(String, String)> {
let mut v: Vec<_> = self.files.borrow().iter().map(|(p, c)| (p.clone(), String::from_utf8_lossy(c).to_string())).collect();
v.sort();
v
}
}
impl MoveFs for FakeFs {
fn exists(&self, p: &Path) -> bool {
self.idx(p).is_some()
}
fn is_file(&self, p: &Path) -> bool {
self.exists(p)
}
fn same_file(&self, a: &Path, b: &Path) -> bool {
self.idx(a).is_some() && self.idx(a) == self.idx(b)
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.fail.borrow().iter().any(|f| a.to_string_lossy().contains(f.as_str())) {
return Err(std::io::Error::other("injected failure"));
}
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
self.files.borrow_mut()[i].0 = b.to_string_lossy().to_string();
Ok(())
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
let c = self.files.borrow()[i].1.clone();
self.files.borrow_mut().push((b.to_string_lossy().to_string(), c));
Ok(())
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
let i = self.idx(p).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
self.files.borrow_mut().remove(i);
Ok(())
}
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
let stem = sidecar.with_extension("").to_string_lossy().to_string();
self.files.borrow().iter().map(|(f, _)| f.clone()).find(|f| {
let p = Path::new(f);
p.with_extension("").to_string_lossy() == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
})
}
}
fn file_photo(id: u64, path: &str) -> Photo {
let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
}
/// Issue #95: a case-only rename is a real change on a case-insensitive volume and never
/// replaces a different file that differs only by case on a case-sensitive one.
#[test]
fn case_only_renames_never_replace_another_file() {
// case-sensitive: IMG_1.JPG and img_1.JPG are two photos
let fs = FakeFs::new(false, &[("/p/IMG_1.JPG", "first"), ("/p/img_1.JPG", "second"), ("/p/IMG_1.xmp", "sc1"), ("/p/img_1.xmp", "sc2")]);
assert!(move_file_with(&fs, "/p/IMG_1.JPG", "/p/img_1.JPG").unwrap_err().message.contains("already exists"));
assert_eq!(fs.listing().len(), 4);
assert!(fs.listing().contains(&("/p/img_1.JPG".into(), "second".into())));
// the planner picks a free name instead
let mut s = Session::new();
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(1, "/p/IMG_1.JPG")) }).unwrap();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "img_1", 1);
assert_eq!(plans[0].to, "img_1-1.JPG");
move_file_with(&fs, "/p/IMG_1.JPG", plans[0].to_path.as_deref().unwrap()).unwrap();
assert_eq!(
fs.listing(),
vec![
("/p/img_1-1.JPG".into(), "first".into()),
("/p/img_1-1.xmp".into(), "sc1".into()),
("/p/img_1.JPG".into(), "second".into()),
("/p/img_1.xmp".into(), "sc2".into())
]
);
// case-sensitive, target free: a plain rename
let fs = FakeFs::new(false, &[("/p/IMG_2.JPG", "x")]);
move_file_with(&fs, "/p/IMG_2.JPG", "/p/img_2.JPG").unwrap();
assert_eq!(fs.listing(), vec![("/p/img_2.JPG".into(), "x".into())]);
// case-insensitive: the "existing" target is the photo itself → renamed via a temp name
let fs = FakeFs::new(true, &[("/p/IMG_1.JPG", "first"), ("/p/IMG_1.xmp", "sc1")]);
let mut s = Session::new();
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(1, "/p/IMG_1.JPG")) }).unwrap();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "img_1", 1);
assert_eq!(plans[0].to, "img_1.JPG", "not a collision with itself");
move_file_with(&fs, "/p/IMG_1.JPG", "/p/img_1.JPG").unwrap();
assert_eq!(fs.listing(), vec![("/p/img_1.JPG".into(), "first".into()), ("/p/img_1.xmp".into(), "sc1".into())]);
// case-insensitive, another file under another name: still a collision
let fs = FakeFs::new(true, &[("/p/A.JPG", "a"), ("/p/b.JPG", "b")]);
assert!(move_file_with(&fs, "/p/A.JPG", "/p/B.JPG").is_err());
}
/// Issue #105: a rollback that fails is reported, and the library follows the files that kept
/// their new names (an undoable partial rename), so nothing goes missing.
#[test]
fn failed_rollbacks_are_reported_and_followed() {
let fs = FakeFs::new(false, &[("/p/a.jpg", "A"), ("/p/a.xmp", "As"), ("/p/b.jpg", "B"), ("/p/c.jpg", "C")]);
let mut s = Session::new();
for (i, f) in ["/p/a.jpg", "/p/b.jpg", "/p/c.jpg"].iter().enumerate() {
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(i as u64 + 1, f)) }).unwrap();
}
let ids = [PhotoId(1), PhotoId(2), PhotoId(3)];
let plans = s.plan_rename_with(&fs, &ids, "Trip-{seq}", 1);
// c can't be renamed (the share went away), and neither can Trip-1 be moved back
fs.fail.borrow_mut().extend(["c.jpg".to_string(), "Trip-1".to_string()]);
let err = s.apply_rename_with(&fs, &plans).unwrap_err().to_string();
assert!(err.contains("rename /p/c.jpg"), "{err}");
assert!(err.contains("/p/Trip-1.jpg could not be moved back"), "{err}");
assert!(err.contains("/p/a.jpg → /p/Trip-1.jpg"), "lists what stayed renamed: {err}");
assert_eq!(
fs.listing(),
vec![
("/p/Trip-1.jpg".into(), "A".into()),
("/p/Trip-1.xmp".into(), "As".into()),
("/p/b.jpg".into(), "B".into()),
("/p/c.jpg".into(), "C".into())
]
);
// the catalog matches the disk
let path = |s: &Session, id: u64| match &s.catalog.photo(PhotoId(id)).unwrap().source {
Source::File { path } => path.clone(),
Source::Demo { .. } => String::new(),
};
assert_eq!(path(&s, 1), "/p/Trip-1.jpg");
assert_eq!(s.catalog.photo(PhotoId(1)).unwrap().file_name, "Trip-1.jpg");
assert_eq!((path(&s, 2), path(&s, 3)), ("/p/b.jpg".into(), "/p/c.jpg".into()));
assert_eq!(s.undo.last().map(|e| e.label.as_str()), Some("Rename 1 Photo (partly)"));
// a sidecar that won't move, and a file that won't go back: the file keeps its new name
// with the sidecars that did move, and says so
let fs = FakeFs::new(false, &[("/p/d.jpg", "D"), ("/p/d.xmp", "Ds"), ("/p/d.jpg.xmp", "Df")]);
fs.fail.borrow_mut().extend(["d.jpg.xmp".to_string(), "/p/e.jpg".to_string()]);
let e = move_file_with(&fs, "/p/d.jpg", "/p/e.jpg").unwrap_err();
assert!(e.moved, "{e}");
assert!(e.message.contains("it is now /p/e.jpg with its sidecar /p/e.xmp"), "{e}");
assert_eq!(fs.listing(), vec![("/p/d.jpg.xmp".into(), "Df".into()), ("/p/e.jpg".into(), "D".into()), ("/p/e.xmp".into(), "Ds".into())]);
// … and when the file does go back, its sidecars follow it
let fs = FakeFs::new(false, &[("/p/d.jpg", "D"), ("/p/d.xmp", "Ds"), ("/p/d.jpg.xmp", "Df")]);
fs.fail.borrow_mut().push("d.jpg.xmp".to_string());
let e = move_file_with(&fs, "/p/d.jpg", "/p/e.jpg").unwrap_err();
assert!(!e.moved);
assert_eq!(fs.listing().len(), 3);
assert!(fs.listing().iter().all(|(p, _)| p.starts_with("/p/d.")));
}
/// Issue #92 (comment): renaming one of `IMG_1.CR3` + `IMG_1.JPG` copies their shared
/// `IMG_1.xmp` instead of taking it away from the other; the last one renamed takes it.
#[test]
fn shared_stem_sidecar_is_copied_not_taken() {
let fs = FakeFs::new(false, &[("/p/IMG_1.CR3", "raw"), ("/p/IMG_1.JPG", "jpg"), ("/p/IMG_1.xmp", "shared"), ("/p/IMG_1.CR3.xmp", "own")]);
move_file_with(&fs, "/p/IMG_1.CR3", "/p/Trip_1.CR3").unwrap();
assert_eq!(
fs.listing(),
vec![
("/p/IMG_1.JPG".into(), "jpg".into()),
("/p/IMG_1.xmp".into(), "shared".into()),
("/p/Trip_1.CR3".into(), "raw".into()),
("/p/Trip_1.CR3.xmp".into(), "own".into()),
("/p/Trip_1.xmp".into(), "shared".into()),
]
);
// the JPEG is the last one using it: it moves
move_file_with(&fs, "/p/IMG_1.JPG", "/p/Trip_2.JPG").unwrap();
assert!(fs.listing().contains(&("/p/Trip_2.xmp".into(), "shared".into())));
assert!(!fs.exists(Path::new("/p/IMG_1.xmp")));
// a failure after the copy removes the copy and keeps the original
let fs = FakeFs::new(false, &[("/p/A.CR3", "raw"), ("/p/A.JPG", "jpg"), ("/p/A.xmp", "shared"), ("/p/A.CR3.xmp", "own")]);
fs.fail.borrow_mut().push("A.CR3.xmp".into());
assert!(move_file_with(&fs, "/p/A.CR3", "/p/B.CR3").is_err());
assert_eq!(fs.listing().len(), 4);
assert!(fs.listing().contains(&("/p/A.xmp".into(), "shared".into())) && !fs.exists(Path::new("/p/B.xmp")));
}
/// Issue #95 on the real disk, whichever kind of volume the temp folder is on.
#[test]
fn case_only_rename_on_disk() {
let dir = std::env::temp_dir().join(format!("lc-rename-case-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let (upper, lower) = (dir.join("IMG_1.JPG"), dir.join("img_1.JPG"));
std::fs::write(&upper, b"first").unwrap();
let case_sensitive = !lower.exists();
let p = |q: &Path| q.to_string_lossy().to_string();
if case_sensitive {
std::fs::write(&lower, b"second").unwrap();
assert!(move_file(&p(&upper), &p(&lower)).is_err());
assert_eq!(std::fs::read(&lower).unwrap(), b"second");
assert_eq!(std::fs::read(&upper).unwrap(), b"first");
} else {
move_file(&p(&upper), &p(&lower)).unwrap();
let names: Vec<String> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
assert_eq!(names, vec!["img_1.JPG".to_string()]);
assert_eq!(std::fs::read(&lower).unwrap(), b"first");
}
// a plain rename never replaces
std::fs::write(dir.join("other.JPG"), b"other").unwrap();
assert!(move_file(&p(&dir.join("other.JPG")), &p(&lower)).is_err());
assert_eq!(std::fs::read(dir.join("other.JPG")).unwrap(), b"other");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn folder_templates_stay_inside_the_destination() {
+162 -13
View File
@@ -5,12 +5,25 @@
//! creator, keywords, capture time and GPS in the standard namespaces, plus our complete develop
//! settings (`lc:settings`, JSON), the pick/reject flag (`lc:flag`) and location (`lc:location`).
//!
//! An existing sidecar is never replaced wholesale (issue #92): LightCraft's properties
//! ([`OWNED`], [`OWNED_IF_STATED`]) are merged into it ([`lightcraft_meta::merge_xmp`]) and
//! everything else — another editor's `crs:` develop settings, `xmpMM` history, unknown
//! namespaces — is kept byte for byte. A sidecar that can't be read as XMP is first copied to
//! `<name>.xmp.bak-<time>` (never overwritten), then replaced. Writes are atomic (temp file,
//! fsync, rename).
//!
//! **Shared names** (Stem naming): `IMG_0001.CR3` and `IMG_0001.JPG` would share `IMG_0001.xmp`.
//! When two catalogued files map to one stem sidecar, it belongs to one of them (a raw first, then
//! the first by file name — [`Session::sidecar_naming`]); the others use `<file>.xmp`
//! (`IMG_0001.JPG.xmp`) for writing and reading, so neither overwrites the other.
//!
//! **Read** (on import, and `photo.readMetadataFromFile`): the sidecar wins for metadata, except
//! the capture time: a time embedded in the file wins, and the sidecar's (`exif:DateTimeOriginal`,
//! else `photoshop:DateCreated`, else `xmp:CreateDate`) fills in only when the file has none; develop
//! settings are restored from `lc:settings` when present, else mapped from interoperable `crs:`
//! fields ([`crate::crs`]). For raw/DNG files without a sidecar the file's embedded XMP is used.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -258,13 +271,99 @@ pub fn sidecar_packet(p: &Photo, cat: &lightcraft_catalog::Catalog) -> String {
lightcraft_meta::write_xmp_lc(&meta, &lc)
}
/// Write `data` to `path` atomically (temp file + rename).
/// The properties LightCraft writes and owns in a sidecar: replaced on every save (removed when
/// LightCraft has no value, so clearing a field clears it in the file too). All of them are read
/// back on import and by Read Metadata from File.
pub const OWNED: &[&str] = &[
"xmp:Rating",
"xmp:Label",
"dc:title",
"dc:description",
"dc:rights",
"dc:creator",
"dc:subject",
"Iptc4xmpCore:Location",
"Iptc4xmpCore:AltTextAccessibility",
"Iptc4xmpCore:ExtDescrAccessibility",
"photoshop:City",
"photoshop:State",
"photoshop:Country",
"xmpRights:Marked",
"xmpRights:UsageTerms",
"xmpRights:WebStatement",
"lc:*",
];
/// Properties replaced only when LightCraft has a value (capture time; GPS, which LightCraft
/// doesn't read from sidecars): otherwise the sidecar's own value stays. Anything else LightCraft
/// writes (`xmp:CreatorTool`) goes only into new sidecars.
pub const OWNED_IF_STATED: &[&str] = &["exif:DateTimeOriginal", "photoshop:DateCreated", "exif:GPSLatitude", "exif:GPSLongitude"];
/// Write `data` to `path` atomically: a temp file next to it, synced, then renamed over it.
fn write_atomic(path: &Path, data: &[u8]) -> std::io::Result<()> {
let tmp = path.with_extension(format!("xmp.tmp-{}", std::process::id()));
std::fs::write(&tmp, data)?;
std::fs::rename(&tmp, path).inspect_err(|_| {
let _ = std::fs::remove_file(&tmp);
})
lightcraft_catalog::safe_file::write_atomic(path, data)
}
/// Copy `path` to a new `<name>.bak-<stamp>` (`-1`, `-2`… if taken; never overwriting).
fn backup(path: &Path, stamp: &str) -> std::io::Result<PathBuf> {
use std::io::Write;
let bytes = std::fs::read(path)?;
let stamp: String = stamp.chars().filter(char::is_ascii_alphanumeric).collect();
let name = path.file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
for k in 0..1000 {
let b = path.with_file_name(if k == 0 { format!("{name}.bak-{stamp}") } else { format!("{name}.bak-{stamp}-{k}") });
match std::fs::OpenOptions::new().write(true).create_new(true).open(&b) {
Ok(mut f) => {
f.write_all(&bytes)?;
f.sync_all()?;
return Ok(b);
}
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(std::io::Error::other("no free backup name"))
}
/// What [`Session::save_sidecar`] did.
#[derive(Clone, Debug, PartialEq)]
pub struct SidecarSaved {
pub path: PathBuf,
/// An existing sidecar was merged into (its other contents kept).
pub merged: bool,
/// An existing sidecar that couldn't be read as XMP was copied here before being replaced.
pub backup: Option<PathBuf>,
}
/// The new contents for the sidecar at `path`: `fresh` merged into the existing file, if any.
/// Returns (contents, merged, backup).
fn sidecar_contents(path: &Path, fresh: String, stamp: &str) -> std::io::Result<(String, bool, Option<PathBuf>)> {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((fresh, false, None)),
// can't see what is there: never overwrite it blindly
Err(e) => return Err(e),
};
if bytes.iter().all(u8::is_ascii_whitespace) {
return Ok((fresh, false, None));
}
let rules = lightcraft_meta::MergeRules { owned: OWNED, owned_if_present: OWNED_IF_STATED };
match std::str::from_utf8(&bytes)
.map_err(|e| e.to_string())
.and_then(|old| lightcraft_meta::merge_xmp(old, &fresh, rules).map_err(|e| e.to_string()))
{
Ok(merged) => Ok((merged, true, None)),
Err(why) => {
let b = backup(path, stamp)?;
log::warn!("{}: not readable as XMP ({why}); kept a copy as {} before replacing it", path.display(), b.display());
Ok((fresh, false, Some(b)))
}
}
}
/// Key of the stem sidecar a file maps to (case-insensitive).
fn stem_key(original: &str) -> String {
sidecar_path(original, SidecarNaming::Stem).to_string_lossy().to_lowercase()
}
fn file_path(p: &Photo) -> Option<&str> {
@@ -302,24 +401,70 @@ pub(crate) fn op_photos(op: &Op, out: &mut Vec<PhotoId>) {
}
}
/// Which photo owns each stem sidecar that several catalogued files map to (see
/// [`Session::sidecar_naming`]).
pub(crate) struct StemOwners(HashMap<String, PhotoId>);
impl StemOwners {
pub(crate) fn of(cat: &lightcraft_catalog::Catalog) -> StemOwners {
// per stem: (raw first, then file name, then id) of the best candidate; and how many
let mut best: HashMap<String, ((bool, String, u64), PhotoId, usize)> = HashMap::new();
for p in cat.photos().filter(|p| p.copy_of.is_none()) {
let Some(path) = file_path(p) else { continue };
let rank = (p.kind != MediaKind::Raw, path.to_lowercase(), p.id.0);
let e = best.entry(stem_key(path)).or_insert_with(|| (rank.clone(), p.id, 0));
e.2 += 1;
if rank < e.0 {
(e.0, e.1) = (rank, p.id);
}
}
StemOwners(best.into_iter().filter(|(_, v)| v.2 > 1).map(|(k, v)| (k, v.1)).collect())
}
/// `naming` for photo `p`, except that a stem sidecar another file owns becomes Full.
fn naming(&self, p: &Photo, naming: SidecarNaming) -> SidecarNaming {
match (naming, file_path(p)) {
(SidecarNaming::Stem, Some(path)) if self.0.get(&stem_key(path)).is_some_and(|owner| *owner != p.id) => SidecarNaming::Full,
_ => naming,
}
}
}
impl Session {
/// Write the photo's XMP sidecar; returns its path.
pub fn save_sidecar(&self, id: PhotoId) -> Result<PathBuf> {
/// The sidecar naming for photo `id`: the library's, except under Stem naming when another
/// catalogued file shares the stem (`IMG_0001.CR3` + `IMG_0001.JPG`) and owns the stem
/// sidecar — a raw first, else the first by file name. The others use Full naming
/// (`IMG_0001.JPG.xmp`), so their metadata never overwrites each other.
pub fn sidecar_naming(&self, id: PhotoId) -> SidecarNaming {
match self.catalog.photo(id) {
Some(p) if self.xmp.naming == SidecarNaming::Stem => StemOwners::of(&self.catalog).naming(p, self.xmp.naming),
_ => self.xmp.naming,
}
}
/// Write the photo's XMP sidecar, merged into an existing one (see the module docs).
pub fn save_sidecar(&self, id: PhotoId) -> Result<SidecarSaved> {
self.save_sidecar_with(id, &StemOwners::of(&self.catalog))
}
pub(crate) fn save_sidecar_with(&self, id: PhotoId, owners: &StemOwners) -> Result<SidecarSaved> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
if p.copy_of.is_some() {
return Err(EngineError::Other(format!("{} is a virtual copy: its settings live only in the library", p.file_name)));
}
let orig = file_path(p).ok_or_else(|| EngineError::Other(format!("{} is not a file on disk", p.file_name)))?;
let path = sidecar_path(orig, self.xmp.naming);
write_atomic(&path, sidecar_packet(p, &self.catalog).as_bytes()).map_err(|e| EngineError::Other(format!("{}: {e}", path.display())))?;
Ok(path)
let path = sidecar_path(orig, owners.naming(p, self.xmp.naming));
let err = |e: std::io::Error| EngineError::Other(format!("{}: {e}", path.display()));
let (data, merged, backup) = sidecar_contents(&path, sidecar_packet(p, &self.catalog), &(self.clock)()).map_err(err)?;
write_atomic(&path, data.as_bytes()).map_err(err)?;
Ok(SidecarSaved { path, merged, backup })
}
/// The op that applies a photo's sidecar (or embedded XMP) to the catalog, if there is one.
pub fn read_sidecar_op(&self, id: PhotoId) -> Result<Option<(Op, PathBuf)>> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
let Some(orig) = file_path(p) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.xmp.naming) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.sidecar_naming(id)) else { return Ok(None) };
let sc = parse_sidecar(&packet, p.kind == MediaKind::Raw)
.map_err(|e| EngineError::Other(format!("{}: {e}", from.display())))?
.resolve_label(&self.catalog);
@@ -344,9 +489,13 @@ impl Session {
pub(crate) fn auto_write_sidecars(&self, ops: &[Op]) {
let mut ids = Vec::new();
ops.iter().for_each(|o| op_photos(o, &mut ids));
if ids.is_empty() {
return;
}
let owners = StemOwners::of(&self.catalog);
for id in ids {
if self.catalog.photo(id).is_some_and(|p| file_path(p).is_some() && p.copy_of.is_none())
&& let Err(e) = self.save_sidecar(id)
&& let Err(e) = self.save_sidecar_with(id, &owners)
{
log::warn!("auto-write XMP: {e}");
}
+52 -3
View File
@@ -130,11 +130,16 @@ pub fn migrate(from: &Path, to: &Path, what: Existing) -> (usize, Vec<String>) {
Existing::Discard => std::fs::remove_file(&src),
_ => {
let dst = to.join(e.file_name());
if dst.exists() {
// named by content: the copy already there is the same preview
if is_valid(&dst) {
// named by content: the complete copy already there is the same preview
std::fs::remove_file(&src)
} else {
std::fs::rename(&src, &dst).or_else(|_| std::fs::copy(&src, &dst).and_then(|_| std::fs::remove_file(&src)))
// across drives: copied atomically (never a partial proxy at the new place)
std::fs::rename(&src, &dst).or_else(|_| {
std::fs::read(&src)
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&dst, &b))
.and_then(|()| std::fs::remove_file(&src))
})
}
}
};
@@ -146,6 +151,29 @@ pub fn migrate(from: &Path, to: &Path, what: Existing) -> (usize, Vec<String>) {
(done, failed)
}
/// A cheap check that the smart preview at `path` is complete: our header, then a JPEG from its
/// start marker to its end marker. A file cut short by a crash, a full drive or an unplugged one
/// fails it (and is then rebuilt by Build Smart Previews, not counted as there).
pub fn is_valid(path: &Path) -> bool {
use std::io::{Read, Seek, SeekFrom};
let Ok(mut f) = std::fs::File::open(path) else { return false };
let mut head = Vec::with_capacity(256);
if (&mut f).take(256).read_to_end(&mut head).is_err() {
return false;
}
let Some(rest) = head.strip_prefix(MAGIC) else { return false };
let Some(nl) = rest.iter().position(|b| *b == b'\n') else { return false };
let size_ok = rest
.get(..nl)
.and_then(|h| serde_json::from_slice::<serde_json::Value>(h).ok())
.is_some_and(|h| h["w"].as_u64().is_some_and(|w| w > 0) && h["h"].as_u64().is_some_and(|h| h > 0));
if !size_ok || rest.get(nl + 1..nl + 3) != Some(&[0xFF, 0xD8][..]) {
return false;
}
let mut end = [0u8; 2];
f.seek(SeekFrom::End(-2)).is_ok() && f.read_exact(&mut end).is_ok() && end == [0xFF, 0xD9]
}
/// Load the proxy at `path`.
pub fn load(path: &Path) -> Result<Arc<Rgb32f>, String> {
let b = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?;
@@ -193,6 +221,27 @@ mod tests {
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn damaged_proxies_are_invalid_and_never_kept_over_good_ones() {
let base = temp("valid");
let (a, b) = (base.join("a"), base.join("b"));
std::fs::create_dir_all(&a).unwrap();
std::fs::create_dir_all(&b).unwrap();
let img = lightcraft_scenes::demo_library()[0].render(64, 40);
let good = encode(&img).unwrap();
std::fs::write(a.join("1.lcsp"), &good).unwrap();
assert!(is_valid(&a.join("1.lcsp")));
for cut in [good.len() - 1, good.len() / 2, 10, 0] {
std::fs::write(b.join("1.lcsp"), &good[..cut]).unwrap();
assert!(!is_valid(&b.join("1.lcsp")), "cut at {cut}");
}
assert!(!is_valid(&b.join("missing.lcsp")));
// moving onto a damaged copy replaces it instead of deleting the good one
assert_eq!(migrate(&a, &b, Existing::Move).0, 1);
assert_eq!(std::fs::read(b.join("1.lcsp")).unwrap(), good);
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn roundtrip_keeps_the_picture() {
let img = lightcraft_scenes::demo_library()[0].render(160, 100);
+149
View File
@@ -174,3 +174,152 @@ fn prepared_exports_run_on_another_thread() {
assert_eq!(files[0]["width"], 48);
assert_eq!(seen.lock().unwrap().len(), 2);
}
/// Issue #78: a GPU render whose work never ran (a driver that drops a submission without an
/// error) gave an all-black export. The export must come from the CPU instead — the same
/// image — with the reason recorded.
#[test]
fn export_falls_back_to_the_cpu_when_gpu_work_is_lost() {
let mut s = Session::with_demo();
let id = s.active().unwrap();
s.execute("develop.set", &json!({"control": "light.exposure", "value": 0.4})).unwrap();
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 20})).unwrap();
let o = ExportOptions::from_json(&json!({"format": "png"}));
let mean = |b: &[u8]| {
let d = lightcraft_codecs::decode(b, Default::default()).unwrap().image;
d.data.iter().map(|p| (p[0] + p[1] + p[2]) as f64).sum::<f64>() / (3 * d.data.len()) as f64
};
let gpu = lightcraft_gpu::available();
let healthy = mean(&export_photo(&mut s, id, &o, 1).unwrap().bytes);
lightcraft_gpu::inject_fault(lightcraft_gpu::Fault::DropWork);
let faulted = export_photo(&mut s, id, &o, 1).unwrap().bytes;
if gpu {
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("incomplete"), "{why}");
// the GPU is off for the process now: this export renders on the CPU
assert!(!lightcraft_gpu::available());
}
let cpu = export_photo(&mut s, id, &o, 1).unwrap().bytes;
lightcraft_gpu::reset_failures();
let px = |b: &[u8]| lightcraft_codecs::decode(b, Default::default()).unwrap().image.data;
assert!(px(&faulted) == px(&cpu), "the fallback is the CPU render");
let m = mean(&faulted);
assert!(m > 0.02, "not black: mean {m}");
assert!((m - healthy).abs() < 0.01, "GPU {healthy} vs CPU {m}");
}
/// A JPEG original `dir/IMG_1.jpg` imported into a library on disk.
fn library_with_jpeg(tag: &str) -> (Session, lightcraft_catalog::PhotoId, std::path::PathBuf, std::path::PathBuf, Vec<u8>) {
let dir = temp_dir(tag);
let src = dir.join("IMG_1.jpg");
let img = lightcraft_raster::Rgba8::from_fn(48, 32, |x, y| [(x * 5) as u8, (y * 7) as u8, 120, 255]);
let bytes = crate::export::encode_image(&img, &ExportOptions::default()).unwrap();
std::fs::write(&src, &bytes).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = s.catalog.photos().next().unwrap().id;
s.execute("develop.set", &json!({"ids": [id.0], "control": "light.exposure", "value": 1.0})).unwrap();
(s, id, dir, src, bytes)
}
fn disk_batch(
s: &mut Session,
id: lightcraft_catalog::PhotoId,
o: &ExportOptions,
to: &crate::export::Destination,
) -> Result<Vec<serde_json::Value>, String> {
crate::export::export_batch(s, &[id], o, to, &mut crate::export::write_file, &|p| std::path::Path::new(p).exists())
}
/// Issue #93: exporting into the photo's own folder as `{name}` with "Overwrite" replaced the
/// original with the re-encoded render. It is refused now, whatever the policy or exact path.
#[test]
fn export_never_overwrites_an_original() {
use crate::export::{Conflict, Destination};
let (mut s, id, dir, src, original) = library_with_jpeg("export-guard");
let folder = Destination { dir: dir.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { naming: "{name}".into(), conflict: Conflict::Overwrite, ..Default::default() };
let err = disk_batch(&mut s, id, &o, &folder).unwrap_err();
assert!(err.contains("original of IMG_1.jpg"), "{err}");
assert_eq!(std::fs::read(&src).unwrap(), original, "the original is byte-identical");
// the exact path of the control channel / MCP, also spelled another way
std::fs::create_dir_all(dir.join("sub")).unwrap();
for exact in [src.to_string_lossy().to_string(), dir.join("sub/../IMG_1.jpg").to_string_lossy().to_string()] {
let to = Destination { dir: String::new(), exact: Some(exact) };
assert!(disk_batch(&mut s, id, &o, &to).unwrap_err().contains("never writes over an original"));
}
// the "Original" format re-writing the file onto itself (and its sidecar)
let orig = ExportOptions { format: ExportFormat::Original, ..o.clone() };
assert!(disk_batch(&mut s, id, &orig, &folder).is_err());
assert_eq!(std::fs::read(&src).unwrap(), original);
assert!(s.execute("export.checkTarget", &json!({"path": src.to_string_lossy()})).is_err());
assert!(s.execute("export.checkTarget", &json!({"path": dir.join("free.jpg").to_string_lossy()})).is_ok());
// Unique moves past it; Overwrite still replaces an ordinary earlier export
let unique = ExportOptions { conflict: Conflict::Unique, ..o.clone() };
let files = disk_batch(&mut s, id, &unique, &folder).unwrap();
assert!(files[0]["path"].as_str().unwrap().ends_with("IMG_1-2.jpg"), "{files:?}");
let earlier = dir.join("IMG_1-2.jpg");
std::fs::write(&earlier, b"an earlier export").unwrap();
let named = ExportOptions { naming: "{name}-2".into(), ..o };
disk_batch(&mut s, id, &named, &folder).unwrap();
assert_ne!(std::fs::read(&earlier).unwrap(), b"an earlier export", "an ordinary file is overwritten");
assert_eq!(std::fs::read(&src).unwrap(), original);
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #93: the XMP sidecar of an "Original" export ignored the conflict policy.
#[test]
fn export_sidecars_follow_the_conflict_policy() {
use crate::export::{Conflict, Destination};
let (mut s, id, dir, _src, original) = library_with_jpeg("export-sidecar");
let out = dir.join("out");
std::fs::create_dir_all(&out).unwrap();
let theirs = out.join("IMG_1.xmp");
std::fs::write(&theirs, b"someone else's sidecar").unwrap();
let to = Destination { dir: out.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { format: ExportFormat::Original, naming: "{name}".into(), ..Default::default() };
// Unique: the photo and its sidecar both move to the next free name
let files = disk_batch(&mut s, id, &ExportOptions { conflict: Conflict::Unique, ..o.clone() }, &to).unwrap();
assert!(files[0]["path"].as_str().unwrap().ends_with("IMG_1-2.jpg"), "{files:?}");
assert!(files[0]["sidecars"][0].as_str().unwrap().ends_with("IMG_1-2.xmp"), "{files:?}");
assert_eq!(std::fs::read(&theirs).unwrap(), b"someone else's sidecar");
assert_eq!(std::fs::read(out.join("IMG_1-2.jpg")).unwrap(), original);
// Skip: a taken sidecar name skips the photo
std::fs::remove_file(out.join("IMG_1-2.jpg")).unwrap();
std::fs::remove_file(out.join("IMG_1-2.xmp")).unwrap();
let files = disk_batch(&mut s, id, &ExportOptions { conflict: Conflict::Skip, ..o.clone() }, &to).unwrap();
assert!(files[0]["skipped"].as_str().is_some(), "{files:?}");
assert!(!out.join("IMG_1.jpg").exists());
assert_eq!(std::fs::read(&theirs).unwrap(), b"someone else's sidecar");
// Overwrite: replaced, as asked
disk_batch(&mut s, id, &ExportOptions { conflict: Conflict::Overwrite, ..o }, &to).unwrap();
assert!(std::fs::read_to_string(&theirs).unwrap().contains("xmpmeta"));
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #93: the native writer truncated the target first; a failure mid-write (full disk,
/// unplugged drive) now leaves the previous file intact and no partial one.
#[test]
fn a_failed_export_write_keeps_the_previous_file() {
use crate::export::{Conflict, Destination};
let (mut s, id, dir, _src, _) = library_with_jpeg("export-midwrite");
let out = dir.join("out");
std::fs::create_dir_all(&out).unwrap();
let prev = out.join("IMG_1.jpg");
std::fs::write(&prev, b"last week's export").unwrap();
let to = Destination { dir: out.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { conflict: Conflict::Overwrite, ..Default::default() };
{
let _fault = lightcraft_catalog::safe_file::fail_writes_after(100);
let err = disk_batch(&mut s, id, &o, &to).unwrap_err();
assert!(err.contains("injected"), "{err}");
}
assert_eq!(std::fs::read(&prev).unwrap(), b"last week's export");
assert_eq!(std::fs::read_dir(&out).unwrap().count(), 1, "no temp file left behind");
disk_batch(&mut s, id, &o, &to).unwrap();
assert_ne!(std::fs::read(&prev).unwrap(), b"last week's export");
let _ = std::fs::remove_dir_all(&dir);
}
+218
View File
@@ -0,0 +1,218 @@
//! Forgetting untouched Local browse records: by command and when the library opens; changed,
//! recent and visible records stay; a crash replays it; browsing again brings records back.
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use serde_json::json;
use crate::{LibrarySource, Selection, Session};
fn temp_dir(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-forget-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
fn png(p: &Path, seed: u8) {
let img = lightcraft_raster::Rgba8::from_fn(16, 12, |x, y| [(x * 9) as u8, (y * 11) as u8, seed, 255]);
let b =
crate::export::encode_image(&img, &crate::export::ExportOptions { format: crate::export::ExportFormat::Png, ..Default::default() }).unwrap();
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, b).unwrap();
}
fn open(lib: &Path, clock: &Arc<Mutex<String>>) -> Session {
let mut s = Session::new().with_fs();
let c = clock.clone();
s.clock = Box::new(move || c.lock().unwrap().clone());
s.open_library(lib, false).unwrap();
s
}
fn set(clock: &Arc<Mutex<String>>, t: &str) {
*clock.lock().unwrap() = t.to_string();
}
fn local_paths(s: &Session) -> Vec<String> {
let mut v: Vec<String> = s
.catalog
.photos()
.filter(|p| p.local)
.filter_map(|p| match &p.source {
lightcraft_catalog::Source::File { path } => Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()),
_ => None,
})
.collect();
v.sort();
v
}
/// Leave the folder view (nothing browsed is visible or selected any more).
fn look_elsewhere(s: &mut Session) {
s.execute("library.source", &json!({"kind": "all"})).unwrap();
s.selection = Selection::default();
s.previous_active = None;
s.undo.clear();
}
#[test]
fn untouched_records_of_unbrowsed_folders_are_forgotten_and_come_back() {
let root = temp_dir("cmd");
let (a, b, lib) = (root.join("a"), root.join("b"), root.join("lib"));
for (i, n) in ["a1.png", "a2.png", "a3.png"].iter().enumerate() {
png(&a.join(n), i as u8);
}
png(&b.join("b1.png"), 7);
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
// the user rates a2 (a change: kept)
let a2 = s.catalog.photos().find(|p| p.file_name == "a2.png").unwrap().id;
s.selection = Selection::single(a2);
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
set(&clock, "2026-08-25T10:00:00");
s.execute("library.browse", &json!({"path": b.to_string_lossy()})).unwrap();
look_elsewhere(&mut s);
// 40 days after a was browsed, 16 after b
set(&clock, "2026-09-10T10:00:00");
let dry = s.execute("library.forgetLocal", &json!({"dryRun": true})).unwrap();
assert_eq!(
(dry["local"].as_u64(), dry["forgotten"].as_u64(), dry["keptTouched"].as_u64(), dry["keptRecent"].as_u64()),
(Some(4), Some(2), Some(1), Some(1)),
"{dry}"
);
assert_eq!(local_paths(&s).len(), 4, "a dry run changes nothing");
let r = s.execute("library.forgetLocal", &json!({})).unwrap();
assert_eq!(r["forgotten"], 2, "{r}");
assert_eq!(local_paths(&s), vec!["a2.png", "b1.png"]);
// files on disk are untouched
assert!(["a1.png", "a2.png", "a3.png"].iter().all(|n| a.join(n).exists()));
// not an undo step
assert!(s.undo.is_empty());
// a crash now: the log replays the same catalog
let expect = s.catalog.to_snapshot();
drop(s);
let mut s = open(&lib, &clock);
assert_eq!(s.library.as_ref().unwrap().report.failed, 0);
assert_eq!(s.catalog.to_snapshot(), expect, "nothing more to forget on open, same state after replay");
// browsing a again brings the forgotten files back, fresh; the rated one is the same record
let r = s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
assert_eq!((r["photos"].as_u64(), r["new"].as_u64()), (Some(3), Some(2)), "{r}");
assert_eq!(s.catalog.photo(a2).unwrap().rating, 3);
// while a is shown, nothing in it goes, whatever the time
set(&clock, "2027-09-10T10:00:00");
assert_eq!(s.execute("library.forgetLocal", &json!({"dryRun": true})).unwrap()["forgotten"], 1, "only b1 (not visible)");
let _ = std::fs::remove_dir_all(&root);
}
#[test]
fn opening_the_library_forgets_per_the_preference() {
let root = temp_dir("open");
let (a, lib) = (root.join("a"), root.join("lib"));
png(&a.join("x.png"), 1);
png(&a.join("y.png"), 2);
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
look_elsewhere(&mut s);
// 0 = never
let r = s.execute("library.preferences", &json!({"forgetLocalDays": 0})).unwrap();
assert_eq!(r["forgetLocalDays"], 0);
s.close_library().unwrap();
drop(s);
set(&clock, "2027-01-01T10:00:00");
let mut s = open(&lib, &clock);
assert_eq!(local_paths(&s).len(), 2, "never");
assert!(s.library.as_ref().unwrap().forgot_local.is_none());
// back to 30 days: the next open forgets them
s.execute("library.preferences", &json!({"forgetLocalDays": 30})).unwrap();
s.source = LibrarySource::All;
s.close_library().unwrap();
drop(s);
let s = open(&lib, &clock);
assert!(local_paths(&s).is_empty());
let info = {
let mut s = s;
s.execute("library.info", &json!({})).unwrap()
};
assert_eq!(info["forgotLocal"]["forgotten"], 2, "{info}");
assert!(a.join("x.png").exists() && a.join("y.png").exists());
let _ = std::fs::remove_dir_all(&root);
}
/// A file with an XMP sidecar comes in with the sidecar's rating and keywords: that is its
/// browse state, not a user change, so the record is forgotten like any other — and the sidecar
/// stays byte for byte, so browsing again reads the same values back.
#[test]
fn sidecar_values_are_not_user_changes_and_sidecars_stay() {
let root = temp_dir("xmp");
let (a, lib) = (root.join("a"), root.join("lib"));
png(&a.join("x.png"), 1);
let x = a.join("x.png").to_string_lossy().to_string();
let mut p = lightcraft_catalog::Photo::new(
lightcraft_catalog::PhotoId(1),
lightcraft_catalog::Source::File { path: x.clone() },
"x.png",
"PNG",
16,
12,
"2026-01-01T00:00:00",
);
p.rating = 4;
p.meta.keywords = vec!["dunes".into()];
let side = crate::sidecar::sidecar_path(&x, Default::default());
std::fs::write(&side, crate::sidecar::sidecar_packet(&p, &lightcraft_catalog::Catalog::new())).unwrap();
let packet = std::fs::read(&side).unwrap();
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
let rec = s.catalog.photos().find(|p| p.local).unwrap().clone();
assert_eq!((rec.rating, rec.meta.keywords.clone()), (4, vec!["dunes".to_string()]), "read from the sidecar");
look_elsewhere(&mut s);
set(&clock, "2026-10-01T10:00:00");
assert_eq!(s.execute("library.forgetLocal", &json!({})).unwrap()["forgotten"], 1);
assert_eq!(std::fs::read(&side).unwrap(), packet, "the sidecar is untouched");
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
let back = s.catalog.photos().find(|p| p.local).unwrap();
assert_ne!(back.id, rec.id, "a fresh record");
assert_eq!((back.rating, back.meta.keywords.clone()), (4, vec!["dunes".to_string()]));
let _ = std::fs::remove_dir_all(&root);
}
/// A catalog written before browse times existed: its Local records are kept on the first open
/// (their folders are stamped then), and forgotten only once the period has passed since.
#[test]
fn upgraded_catalogs_forget_nothing_at_first() {
let root = temp_dir("upgrade");
let (a, lib) = (root.join("a"), root.join("lib"));
png(&a.join("x.png"), 1);
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
look_elsewhere(&mut s);
s.close_library().unwrap();
drop(s);
// strip the times and baselines, as an older version would have written the snapshot
let snap = std::fs::read_to_string(lib.join("catalog.snap")).unwrap();
let mut v: serde_json::Value = serde_json::from_str(&snap).unwrap();
v["catalog"].as_object_mut().unwrap().remove("browsed");
for p in v["catalog"]["photos"].as_object_mut().unwrap().values_mut() {
p.as_object_mut().unwrap().remove("local_baseline");
}
std::fs::write(lib.join("catalog.snap"), serde_json::to_vec(&v).unwrap()).unwrap();
set(&clock, "2027-01-01T10:00:00");
let s = open(&lib, &clock);
assert_eq!(local_paths(&s).len(), 1, "kept on the first open after the upgrade");
assert_eq!(s.catalog.last_browsed(&a.to_string_lossy()), Some("2027-01-01T10:00:00"));
drop(s);
set(&clock, "2027-02-15T10:00:00");
let s = open(&lib, &clock);
assert!(local_paths(&s).is_empty(), "the untouched legacy record goes once 30 days passed");
let _ = std::fs::remove_dir_all(&root);
}
+77
View File
@@ -144,6 +144,45 @@ fn browsing_a_folder_lists_its_photos_without_adding_them() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #105: Find Missing finds renamed files by content, prefers the content match over a
/// same-name same-size impostor, and skips (reports) photos it can't tell apart.
#[test]
fn find_missing_matches_renamed_files_by_content() {
let dir = temp_dir("missing-content");
std::fs::create_dir_all(dir.join("old")).unwrap();
std::fs::create_dir_all(dir.join("moved/real")).unwrap();
for (n, seed) in [("a.png", 1), ("b.png", 2), ("c.png", 3)] {
write_png(&dir.join("old").join(n), seed);
}
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [dir.join("old").to_string_lossy()]})).unwrap();
// an impostor: same name and size, other bytes
let impostor = |src: &Path, dst: &Path| {
let mut b = std::fs::read(src).unwrap();
let k = b.len() - 20;
b[k] ^= 0xff;
std::fs::write(dst, b).unwrap();
};
// a: renamed; b: renamed, with an impostor under its old name; c: gone, two impostors
std::fs::rename(dir.join("old/a.png"), dir.join("moved/Trip-001.png")).unwrap();
impostor(&dir.join("old/b.png"), &dir.join("moved/b.png"));
std::fs::rename(dir.join("old/b.png"), dir.join("moved/real/Trip-002.png")).unwrap();
impostor(&dir.join("old/c.png"), &dir.join("moved/c.png"));
impostor(&dir.join("old/c.png"), &dir.join("moved/real/c.png"));
std::fs::remove_file(dir.join("old/c.png")).unwrap();
let r = s.execute("library.findMissing", &json!({"folder": dir.join("moved").to_string_lossy()})).unwrap();
let found = r["found"].as_array().unwrap();
assert_eq!(found.len(), 2, "{r}");
let to = |name: &str| {
found.iter().find(|f| f["from"].as_str().unwrap().ends_with(name)).map(|f| (f["to"].as_str().unwrap().to_string(), f["by"].clone()))
};
assert_eq!(to("a.png"), Some((dir.join("moved/Trip-001.png").to_string_lossy().to_string(), json!("content"))));
assert_eq!(to("b.png"), Some((dir.join("moved/real/Trip-002.png").to_string_lossy().to_string(), json!("content"))), "not the impostor");
assert_eq!(r["missing"], 1);
assert_eq!(r["ambiguous"][0]["candidates"].as_array().map(Vec::len), Some(2), "{r}");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn missing_files_are_found_and_relinked() {
let dir = std::env::temp_dir().join(format!("lc-missing-{}", std::process::id()));
@@ -352,6 +391,44 @@ fn smart_previews_stand_in_for_offline_originals() {
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #106: a smart preview cut short (crash, full drive) counted as built forever and failed
/// exactly when the original went offline. Writes are atomic now, and a damaged proxy is not
/// reported as present and is rebuilt.
#[test]
fn damaged_smart_previews_are_rebuilt_and_failed_writes_leave_none() {
let src = temp_dir("smartdmg-src");
let lib = temp_dir("smartdmg-lib");
write_png(&src.join("a.png"), 5);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("a.png").to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
let dir = s.media.smart_dir.clone().unwrap();
let file = dir.join(crate::smart::file_name(s.catalog.photo(id).unwrap()));
{
let _fault = lightcraft_catalog::safe_file::fail_writes_after(64);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["failed"].as_array().map(Vec::len)), (Some(0), Some(1)), "{r}");
}
assert!(!file.exists(), "no partial proxy");
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 0, "no temp file left");
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], false);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(0)), "{r}");
assert!(crate::smart::is_valid(&file));
// cut short, as by a crash before the fix
let full = std::fs::read(&file).unwrap();
std::fs::write(&file, &full[..full.len() / 2]).unwrap();
assert!(!crate::smart::is_valid(&file));
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], false, "not counted as there");
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// LUT profiles: a .cube imports as a creative profile (into the library), renders differently
/// (Amount 0 = no change), is listed in the profile browser and comes back with the library.
#[test]
+44
View File
@@ -163,6 +163,50 @@ fn a_failed_or_corrupt_copy_keeps_the_source() {
}
}
/// Issue #96: Import → Copy used a plain unverified copy (and check-then-copy naming that could
/// replace a file). It now gets the same verified copy as Move: a copy that fails or comes out
/// different is removed and reported as a failed import, nothing is catalogued, and the card
/// is untouched; a good copy never replaces an existing file.
#[test]
fn copy_import_verifies_each_copy() {
let import_copy = |s: &mut Session, card: &Path, dest: &Path| {
s.execute(
"library.import",
&json!({"paths": [card.to_string_lossy()], "mode": "copy", "destination": dest.to_string_lossy(), "organize": "flat"}),
)
.unwrap()
};
for fault in [Fault::FailWrite, Fault::Corrupt] {
let base = temp_dir(&format!("copyfail-{fault:?}"));
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 3);
let bytes = std::fs::read(card.join("a.png")).unwrap();
let mut s = session();
inject(fault);
let r = import_copy(&mut s, &card, &dest);
inject(Fault::None);
assert_eq!(len(&r, "failed"), 1, "{fault:?}: {r}");
assert_eq!(len(&r, "imported"), 0, "{fault:?}: {r}");
assert!(r.to_string().contains("the original is untouched"), "{r}");
assert_eq!(s.catalog.len(), 0);
assert_eq!(std::fs::read(card.join("a.png")).unwrap(), bytes, "{fault:?}: source intact");
assert_eq!(files_under(&dest), Vec::<String>::new(), "{fault:?}: no bad copy left");
let _ = std::fs::remove_dir_all(&base);
}
// a good copy: verified, and a taken name gets -1 instead of being replaced
let base = temp_dir("copyok");
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 4);
std::fs::create_dir_all(&dest).unwrap();
std::fs::write(dest.join("a.png"), b"someone else's file").unwrap();
let mut s = session();
let r = import_copy(&mut s, &card, &dest);
assert_eq!(len(&r, "imported"), 1, "{r}");
assert_eq!(std::fs::read(dest.join("a.png")).unwrap(), b"someone else's file");
assert_eq!(std::fs::read(dest.join("a-1.png")).unwrap(), std::fs::read(card.join("a.png")).unwrap());
let _ = std::fs::remove_dir_all(&base);
}
/// A source that can't be removed (a read-only card) stays and is reported; the photo is
/// imported from its copy.
#[test]
+193
View File
@@ -0,0 +1,193 @@
//! A failed save is an error, not a silent success: a command whose journal append fails returns
//! [`EngineError::NotSaved`], its change stays applied and queued, and the next save writes it.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use lightcraft_catalog::{Flag, MemStore, SnapshotPolicy, Store};
use serde_json::json;
use crate::library::LibraryStores;
use crate::{EngineError, Session};
/// A store whose writes fail while `fail` is set (a full or unplugged disk).
#[derive(Clone, Default)]
struct Flaky {
files: MemStore,
fail: Arc<AtomicBool>,
/// Only snapshots fail (appends work).
snapshots_only: bool,
/// Only appends fail (a dead log handle; fresh files can still be written).
appends_only: bool,
}
impl Flaky {
fn failing(&self) -> bool {
self.fail.load(Ordering::SeqCst)
}
fn set_failing(&self, on: bool) {
self.fail.store(on, Ordering::SeqCst);
}
/// The files as a crash at this moment would leave them.
fn crash_copy(&self) -> Flaky {
let m = MemStore::new();
*m.files.lock().unwrap() = self.files.files.lock().unwrap().clone();
Flaky { files: m, ..Default::default() }
}
}
impl Store for Flaky {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.failing() && !self.appends_only {
return Err(std::io::Error::other("disk full"));
}
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.failing() && !self.snapshots_only {
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"flaky".into()
}
}
fn open(store: &Flaky, seed: bool) -> Session {
let mut s = Session::new();
let stores = LibraryStores { dir: "flaky".into(), catalog: Box::new(store.clone()), files: Box::new(MemStore::new()), on_disk: false };
s.open_library_in(stores, seed).unwrap();
s
}
#[test]
fn failed_append_fails_the_command_and_is_retried() {
let store = Flaky::default();
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
s.execute("photo.rate", &json!({"rating": 1})).unwrap();
store.set_failing(true);
let e = s.execute("photo.rate", &json!({"rating": 4})).unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)), "{e:?}");
let msg = e.to_string();
assert!(
msg.starts_with("saved in memory but not written to disk: ") && msg.contains("disk full") && msg.ends_with("LightCraft will retry"),
"{msg}"
);
// applied in memory, undoable, queued
assert_eq!(s.catalog.photo(id).unwrap().rating, 4);
assert!(!s.undo.is_empty());
assert_eq!(s.unsaved().map(|u| u.0), Some(1));
// queries and commands that change nothing are unaffected (they retry the queue, but report
// their own result)
let info = s.execute("library.info", &json!({})).unwrap();
assert_eq!(info["unsavedOps"], 1, "{info}");
assert!(info["unsavedError"].as_str().unwrap().contains("disk full"), "{info}");
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
// a second change fails too; both stay queued, in order
let e = s.execute("photo.flag", &json!({"flag": "pick"})).unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)));
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
// the frame loop backs off instead of retrying every frame
s.persist_if_dirty();
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
// a crash now loses only the unsaved changes; the earlier ones are on disk
assert_eq!(open(&store.crash_copy(), true).catalog.photo(id).unwrap().rating, 1);
// the disk comes back: the next save writes the whole queue
store.set_failing(false);
s.execute("library.info", &json!({})).unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.library.as_ref().unwrap().last_error.is_none());
let expect = s.catalog.to_snapshot();
drop(s); // no close: like a crash
let s2 = open(&store, true);
assert_eq!(s2.catalog.to_snapshot(), expect);
let p = s2.catalog.photo(id).unwrap();
assert_eq!((p.rating, p.flag), (4, Flag::Pick));
}
/// Undo is a command like any other: a failed write of its op is reported, and retried.
#[test]
fn failed_append_of_undo_is_reported_and_retried() {
let store = Flaky::default();
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
let before = s.catalog.photo(id).unwrap().rating;
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
store.set_failing(true);
assert!(matches!(s.execute("edit.undo", &json!({})), Err(EngineError::NotSaved(_))));
assert_eq!(s.catalog.photo(id).unwrap().rating, before);
store.set_failing(false);
s.persist().unwrap();
drop(s);
assert_eq!(open(&store, true).catalog.photo(id).unwrap().rating, before);
}
/// A failed compaction is not a failed command: the command's own records were appended.
#[test]
fn failed_snapshot_does_not_fail_the_command() {
let store = Flaky { snapshots_only: true, ..Default::default() };
let mut s = open(&store, false);
s.library.as_mut().unwrap().journal_mut().policy = SnapshotPolicy { max_records: 1, max_bytes: u64::MAX };
store.set_failing(true);
s.execute("album.create", &json!({"name": "A"})).unwrap();
s.execute("album.create", &json!({"name": "B"})).unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.library.as_ref().unwrap().last_error.is_some(), "the compaction failure is reported");
drop(s);
store.set_failing(false);
assert_eq!(open(&store, false).catalog.albums().count(), 2);
}
/// Issue #101: quitting while the log can't be appended to still saves the queued changes in the
/// closing snapshot (a fresh file), and reports success; they are not appended again later.
#[test]
fn close_saves_queued_changes_in_the_snapshot_when_appends_fail() {
let store = Flaky { appends_only: true, ..Default::default() };
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
store.set_failing(true);
assert!(matches!(s.execute("photo.rate", &json!({"rating": 4})), Err(EngineError::NotSaved(_))));
assert!(matches!(s.execute("photo.flag", &json!({"flag": "pick"})), Err(EngineError::NotSaved(_))));
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.pending_log.is_empty());
let expect = s.catalog.to_snapshot();
drop(s);
store.set_failing(false);
let mut s2 = open(&store, true);
assert_eq!(s2.catalog.to_snapshot(), expect);
let p = s2.catalog.photo(id).unwrap();
assert_eq!((p.rating, p.flag), (4, Flag::Pick));
// later edits continue after the absorbed ops
s2.execute("photo.rate", &json!({"rating": 2})).unwrap();
let expect = s2.catalog.to_snapshot();
drop(s2);
assert_eq!(open(&store, true).catalog.to_snapshot(), expect);
}
/// When neither the log nor the snapshot can be written, closing fails with `NotSaved` and the
/// changes stay queued (a later retry can still save them).
#[test]
fn close_fails_when_nothing_can_be_saved() {
let store = Flaky::default();
let mut s = open(&store, true);
store.set_failing(true);
assert!(s.execute("photo.rate", &json!({"rating": 4})).is_err());
let e = s.close_library().unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)), "{e:?}");
assert_eq!(s.unsaved().map(|u| u.0), Some(1));
store.set_failing(false);
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
}
+178
View File
@@ -329,6 +329,55 @@ fn convert_raw_to_dng() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #106: Convert to DNG wrote straight to the final name (no temp file, sync or check) and
/// relinked the photo even when the write was cut short. A failed write now leaves no DNG and
/// the photo on its raw; a good one is verified and never replaces an existing file.
#[test]
fn convert_to_dng_is_verified_and_atomic() {
let dir = temp_dir("todng-safe");
// a synthetic raw (stored as DNG, catalogued as a NEF so Convert to DNG takes it)
let raw = dir.join("shot.dng");
let bytes = synthetic_dng_with(None, Default::default());
std::fs::write(&raw, &bytes).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [raw.to_string_lossy()]})).unwrap();
let id = s.catalog.photos().next().unwrap().id;
let path = raw.to_string_lossy().to_string();
s.catalog
.apply(lightcraft_catalog::Op::Relink {
id,
file_name: "shot.dng".into(),
source: lightcraft_catalog::Source::File { path: path.clone() },
format: Some("NEF".into()),
})
.unwrap();
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
let names = || {
let mut v: Vec<String> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
v.sort();
v
};
{
// the drive fills up part-way
let _fault = lightcraft_catalog::safe_file::fail_writes_after(200);
let r = s.execute("photo.convertToDng", &json!({})).unwrap();
assert_eq!(r["converted"].as_array().unwrap().len(), 0, "{r}");
assert!(r["skipped"][0][1].as_str().unwrap().contains("the raw is kept"), "{r}");
}
assert_eq!(names(), ["shot.dng"], "no partial DNG");
assert_eq!(s.catalog.photo(id).unwrap().source, lightcraft_catalog::Source::File { path: path.clone() }, "not relinked");
// a good conversion: a new name (the existing file is not replaced), decodable, relinked
let r = s.execute("photo.convertToDng", &json!({})).unwrap();
let out = r["converted"][0]["path"].as_str().unwrap().to_string();
assert!(out.ends_with("shot-2.dng"), "{r}");
assert_eq!(std::fs::read(&raw).unwrap(), bytes, "the raw is untouched");
let back = lightcraft_raw::decode(&std::fs::read(&out).unwrap()).unwrap();
assert_eq!(back.data, lightcraft_raw::decode(&bytes).unwrap().data);
assert_eq!(s.catalog.photo(id).unwrap().source, lightcraft_catalog::Source::File { path: out });
assert_eq!(names(), ["shot-2.dng", "shot.dng"], "no temp file left");
let _ = std::fs::remove_dir_all(&dir);
}
/// Edit in External Editor (engine half): a 16-bit TIFF `-Edit` copy with the edits, next to
/// the original, added and stacked on top of it; a second one doesn't overwrite the first.
#[test]
@@ -576,3 +625,132 @@ fn copyright_status_usage_terms_and_url_round_trip() {
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// A sidecar another raw developer wrote: develop settings in attributes and elements, a
/// structured edit history, an unknown namespace.
const OTHER_APP_SIDECAR: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Other Toolkit">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:xmp="http://ns.adobe.com/xap/1.0/"
xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#"
xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
xmlns:other="http://example.com/other/1.0/"
xmp:Rating="2"
crs:Version="99.0"
crs:Exposure2012="+0.65"
crs:HasSettings="True"
other:Secret="keep me">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action="derived" stEvt:parameters="converted from image/x-raw to image/png"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>128, 140</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>"#;
/// Issue #92: saving merges into another app's sidecar instead of replacing it.
#[test]
fn saving_merges_into_another_apps_sidecar() {
let src = temp_dir("merge");
let lib = temp_dir("merge-lib");
write_png(&src.join("shot.png"), 1);
std::fs::write(src.join("shot.xmp"), OTHER_APP_SIDECAR).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("shot.png").to_string_lossy()]})).unwrap();
let id = only(&s);
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
s.execute("photo.rate", &json!({"rating": 5})).unwrap();
s.execute("photo.setMeta", &json!({"title": "Harbour", "keywords": ["boats"]})).unwrap();
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 30})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
assert_eq!(r["merged"][0], src.join("shot.xmp").to_string_lossy().as_ref(), "{r}");
let out = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
// the other app's data, byte for byte
for part in [
"crs:Exposure2012=\"+0.65\"",
"other:Secret=\"keep me\"",
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"derived\" stEvt:parameters=\"converted from image/x-raw to image/png\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>\n <rdf:li>128, 140</rdf:li>",
] {
assert!(out.contains(part), "lost {part:?}:\n{out}");
}
let d = lightcraft_meta::parse_xmp(&out).unwrap();
assert_eq!(d.metadata.rating, Some(5), "ours replaces theirs");
assert_eq!(d.metadata.title.as_deref(), Some("Harbour"));
assert_eq!(d.properties.get("crs:Version"), Some(&vec!["99.0".to_string()]));
let saved = s.catalog.photo(id).unwrap().develop.clone();
// reading it back restores our settings (lc:settings wins over crs:)
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().develop, saved);
// saving again keeps one LightCraft description and the other app's data
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let again = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert!(again.contains("other:Secret=\"keep me\"") && again.contains("crs:Exposure2012=\"+0.65\""));
assert_eq!(lightcraft_meta::parse_xmp(&again).unwrap().metadata.rating, Some(3));
// a sidecar that isn't XMP is kept as a backup before it is replaced
std::fs::write(src.join("shot.xmp"), b"<not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let backup = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert!(backup.contains("shot.xmp.bak-"), "{r}");
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join("shot.xmp")).unwrap()).unwrap().metadata.rating, Some(3));
// … and another one never overwrites the first backup
std::fs::write(src.join("shot.xmp"), b"<still not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let second = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert_ne!(second, backup);
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(std::fs::read(&second).unwrap(), b"<still not xmp");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #92: two catalogued files with one stem (`IMG_1.jpg` + `IMG_1.png`) don't share a
/// Stem-named sidecar: the first by name keeps `IMG_1.xmp`, the other writes and reads
/// `IMG_1.png.xmp`.
#[test]
fn files_sharing_a_stem_get_their_own_sidecars() {
let src = temp_dir("stem");
let lib = temp_dir("stem-lib");
write_png(&src.join("IMG_1.png"), 1);
let img = lightcraft_raster::Rgba8::from_fn(40, 24, |x, y| [(x * 6) as u8, (y * 9) as u8, 7, 255]);
let jpg =
crate::export::encode_image(&img, &crate::export::ExportOptions { format: crate::export::ExportFormat::Jpeg, ..Default::default() }).unwrap();
std::fs::write(src.join("IMG_1.jpg"), jpg).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = |s: &Session, name: &str| s.catalog.photos().find(|p| p.file_name == name).unwrap().id;
let (j, p) = (id(&s, "IMG_1.jpg"), id(&s, "IMG_1.png"));
assert_eq!(s.sidecar_naming(j), crate::sidecar::SidecarNaming::Stem);
assert_eq!(s.sidecar_naming(p), crate::sidecar::SidecarNaming::Full);
s.execute("photo.rate", &json!({"ids": [j.0], "rating": 2})).unwrap();
s.execute("photo.rate", &json!({"ids": [p.0], "rating": 5})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!(r["written"].as_array().unwrap().len(), 2, "{r}");
let rating = |f: &str| lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join(f)).unwrap()).unwrap().metadata.rating;
assert_eq!(rating("IMG_1.xmp"), Some(2));
assert_eq!(rating("IMG_1.png.xmp"), Some(5));
// each reads its own back
s.execute("photo.rate", &json!({"ids": [j.0, p.0], "rating": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!((s.catalog.photo(j).unwrap().rating, s.catalog.photo(p).unwrap().rating), (2, 5));
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
+222 -24
View File
@@ -275,37 +275,47 @@ fn module_source(m: &Module, consts: &str) -> String {
}
impl Gpu {
/// Create a device on the best available adapter (no software fallback), or `None`.
pub fn new() -> Option<Gpu> {
/// Create a device on the best available adapter (no software fallback), or why there is none.
pub fn new() -> Result<Gpu, String> {
let mut desc = wgpu::InstanceDescriptor::new_without_display_handle();
desc.backends = wgpu::Backends::PRIMARY;
let instance = wgpu::Instance::new(desc);
let adapter = pollster::block_on(
instance.request_adapter(&wgpu::RequestAdapterOptions { power_preference: wgpu::PowerPreference::HighPerformance, ..Default::default() }),
)
.ok()?;
.map_err(|e| format!("no GPU adapter ({e})"))?;
let info = adapter.get_info();
if info.device_type == wgpu::DeviceType::Cpu {
log::info!("gpu: only a software adapter ({}), using the CPU pipeline", info.name);
return None;
return Err(format!("software adapter ({}) skipped: the CPU pipeline is faster", info.name));
}
let limits = adapter.limits();
let mut limits = adapter.limits();
if limits.max_storage_buffers_per_shader_stage < 10 {
log::info!("gpu: {} has too few storage buffers per stage", info.name);
return None;
return Err(format!("{} has too few storage buffers per shader stage ({} < 10)", info.name, limits.max_storage_buffers_per_shader_stage));
}
if let Some((binding, buffer)) = limits_override() {
limits.max_storage_buffer_binding_size = limits.max_storage_buffer_binding_size.min(binding);
limits.max_buffer_size = limits.max_buffer_size.min(buffer);
log::info!("gpu: LIGHTCRAFT_GPU_LIMITS: storage bindings ≤ {} MiB, buffers ≤ {} MiB", binding >> 20, buffer >> 20);
}
let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor {
label: Some("lightcraft"),
required_limits: limits.clone(),
..Default::default()
}))
.ok()?;
// Errors outside an error scope (e.g. out of memory during a render): log, and stop using
// the GPU (the render in flight is redone on the CPU).
.map_err(|e| format!("{}: device creation failed ({e})", info.name))?;
// Errors outside a render's error scopes (see `ErrorScopes`): log, and stop using the GPU.
device.on_uncaptured_error(std::sync::Arc::new(|e| {
log::error!("gpu: {e}");
crate::mark_broken();
crate::mark_broken(&format!("device error: {e}"));
}));
// wgpu reports a lost device (driver reset, GPU hang, watchdog) only here — never as an
// error — so without this callback a lost device went unnoticed.
device.set_device_lost_callback(|reason, msg| {
log::error!("gpu: device lost ({reason:?}): {msg}");
crate::mark_broken(&format!("device lost ({reason:?}): {msg}"));
});
let scope = device.push_error_scope(wgpu::ErrorFilter::Validation);
let consts = constants();
let mut kernels = HashMap::new();
@@ -334,7 +344,7 @@ impl Gpu {
}
if let Some(e) = pollster::block_on(scope.pop()) {
log::error!("gpu: kernels failed to build on {}: {e}", info.name);
return None;
return Err(format!("{}: kernels failed to build ({e})", info.name));
}
let dummy = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("dummy"),
@@ -343,13 +353,20 @@ impl Gpu {
mapped_at_creation: false,
});
let max_buffer = limits.max_storage_buffer_binding_size.min(limits.max_buffer_size);
log::info!("gpu: {} ({:?})", info.name, info.backend);
Some(Gpu { device, queue, info, kernels, dummy, free: Default::default(), max_buffer })
log::info!("gpu: {} ({:?}), buffers ≤ {} MiB", info.name, info.backend, max_buffer >> 20);
Ok(Gpu { device, queue, info, kernels, dummy, free: Default::default(), max_buffer })
}
/// A buffer of `len` 32-bit values with undefined contents (a recycled one when possible).
///
/// A buffer over the device limit fails the render ([`FailKind::Limit`]) instead of raising a
/// validation error: the caller gets a placeholder and the render is redone on the CPU.
pub fn buffer(&self, len: usize) -> Buf {
let size = (len.max(1) * 4) as u64;
if size > self.limit() {
self.over_limit(size);
return Buf { buf: Some(Tracked::new(self.placeholder())), len };
}
let recycled = {
let mut f = self.free.lock().unwrap_or_else(|e| e.into_inner());
// the smallest pooled buffer that holds `size` with at most 25 % to spare: photos of
@@ -426,6 +443,10 @@ impl Gpu {
if bytes.is_empty() {
return self.buffer(1);
}
if bytes.len() as u64 > self.limit() {
self.over_limit(bytes.len() as u64);
return Buf { buf: Some(Tracked::new(self.placeholder())), len };
}
let buf = self.device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
label: None,
contents: bytes,
@@ -434,9 +455,32 @@ impl Gpu {
Buf { buf: Some(Tracked::new(buf)), len }
}
fn over_limit(&self, size: u64) {
fail(
FailKind::Limit,
format!("a {} MiB buffer exceeds the device's {} MiB storage-buffer limit", size.div_ceil(1 << 20), self.limit() >> 20),
);
}
/// Stands in for a buffer that could not be created (the render has failed already).
fn placeholder(&self) -> wgpu::Buffer {
self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("placeholder"),
size: 16,
usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
})
}
/// Whether a buffer of `len` 32-bit values fits the device limits.
pub fn fits(&self, len: usize) -> bool {
(len as u64) * 4 <= self.max_buffer
(len as u64) * 4 <= self.limit()
}
/// Largest buffer a render may use (bytes): the device limit, or a lower one set for the
/// render on this thread ([`LimitOverride`]).
pub fn limit(&self) -> u64 {
LIMIT.try_with(|l| l.get()).ok().flatten().map_or(self.max_buffer, |l| l.min(self.max_buffer))
}
pub fn encoder(&self) -> wgpu::CommandEncoder {
@@ -445,10 +489,12 @@ impl Gpu {
/// Record kernel `name` with parameters `p` over buffers `bufs` (None = unused binding).
pub fn run(&self, enc: &mut wgpu::CommandEncoder, name: &str, p: &[u32], bufs: &[Option<&Buf>], groups: [u32; 3]) {
if failed() {
return; // the render is redone on the CPU: don't spend the GPU on it
}
let Some(k) = self.kernels.get(name) else {
// a kernel name typo: fail this render over to the CPU instead of crashing
log::error!("gpu: unknown kernel {name}");
crate::mark_broken();
fail(FailKind::Fatal, format!("unknown kernel {name}"));
return;
};
assert_eq!(bufs.len(), k.nbuf, "{name}: binding count");
@@ -470,8 +516,16 @@ impl Gpu {
}
/// Submit `enc`, then read back the first `len` 32-bit values of `src`.
///
/// A failed readback (device lost, map error, timeout) fails the render ([`FailKind::Fatal`])
/// and returns zeros, which the caller discards; after an earlier failure in the same render
/// nothing more is submitted.
pub fn finish_and_read<T: bytemuck::Pod>(&self, mut enc: wgpu::CommandEncoder, src: &Buf, len: usize) -> Vec<T> {
assert!(len <= src.len, "readback of {len} values from a buffer of {}", src.len);
let zeros = || vec![<T as bytemuck::Zeroable>::zeroed(); len];
if failed() {
return zeros();
}
let size = (len * 4) as u64;
let staging = self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("readback"),
@@ -486,17 +540,20 @@ impl Gpu {
slice.map_async(wgpu::MapMode::Read, move |r| {
let _ = tx.send(r);
});
let _ = self.device.poll(wgpu::PollType::wait_indefinitely());
// A failed readback (device lost, out of memory) marks the GPU broken: the caller
// (`crate::render`) then discards this render's zeroed result and renders on the CPU.
let failed = |why: String| {
log::error!("gpu readback failed: {why}");
crate::mark_broken();
vec![<T as bytemuck::Zeroable>::zeroed(); len]
fail(FailKind::Fatal, format!("readback failed: {why}"));
zeros()
};
match rx.recv() {
// A lost device makes the wait fail and the map callback never run: checking both keeps
// this from blocking forever (or handing back an unwritten buffer as the image).
if let Err(e) = self.device.poll(wgpu::PollType::Wait { submission_index: None, timeout: Some(READBACK_TIMEOUT) }) {
return failed(format!("{e}"));
}
// another thread's poll may be the one that runs the callback: allow it a moment
match rx.recv_timeout(std::time::Duration::from_secs(10)) {
Ok(Ok(())) => {}
other => return failed(format!("{other:?}")),
Ok(Err(e)) => return failed(format!("{e}")),
Err(e) => return failed(format!("map callback: {e}")),
}
let data = match slice.get_mapped_range() {
Ok(d) => d,
@@ -509,6 +566,127 @@ impl Gpu {
}
}
/// Longest wait for a render's GPU work: beyond it the render fails over to the CPU (drivers
/// normally reset a hung GPU much sooner).
const READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
/// `LIGHTCRAFT_GPU_LIMITS`: lower the device limits, e.g. to reproduce a smaller adapter.
/// `webgpu` / `downlevel` = 128 MiB storage bindings and 256 MiB buffers (the WebGPU defaults),
/// `<n>` = n MiB storage bindings and 2n MiB buffers. Returns (binding bytes, buffer bytes).
fn limits_override() -> Option<(u64, u64)> {
parse_limits(&std::env::var("LIGHTCRAFT_GPU_LIMITS").ok()?)
}
fn parse_limits(v: &str) -> Option<(u64, u64)> {
let v = v.trim().to_ascii_lowercase();
match v.as_str() {
"" => None,
"webgpu" | "downlevel" | "default" => {
let d = wgpu::Limits::default();
Some((d.max_storage_buffer_binding_size, d.max_buffer_size))
}
n => {
let mb: u64 = n.trim_end_matches("mib").trim_end_matches("mb").trim().parse().ok()?;
let b = mb.max(1) << 20;
Some((b, 2 * b))
}
}
}
/// How a GPU render failed (it is then redone on the CPU).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum FailKind {
/// A buffer over the device limits: this render only.
Limit,
/// The device ran out of memory: this render only (the buffer pool is emptied).
OutOfMemory,
/// The image came back entirely black: redone on the CPU in case the GPU dropped work.
Blank,
/// Device error or loss, readback failure, incomplete image: the GPU is not used again.
Fatal,
}
#[derive(Clone, Debug)]
pub(crate) struct Failure {
pub kind: FailKind,
pub reason: String,
}
thread_local! {
/// A lower buffer limit for the render on this thread (fault injection in tests).
static LIMIT: std::cell::Cell<Option<u64>> = const { std::cell::Cell::new(None) };
}
/// Lowers [`Gpu::limit`] on this thread while it lives.
pub(crate) struct LimitOverride;
impl LimitOverride {
pub fn new(bytes: u64) -> LimitOverride {
LIMIT.with(|l| l.set(Some(bytes)));
LimitOverride
}
}
impl Drop for LimitOverride {
fn drop(&mut self) {
let _ = LIMIT.try_with(|l| l.set(None));
}
}
thread_local! {
/// The first failure of the render running on this thread.
static FAILURE: std::cell::RefCell<Option<Failure>> = const { std::cell::RefCell::new(None) };
}
/// Record a failure of the render running on this thread (the first one is kept).
pub(crate) fn fail(kind: FailKind, reason: String) {
let _ = FAILURE.try_with(|f| {
let mut f = f.borrow_mut();
if f.is_none() {
*f = Some(Failure { kind, reason });
}
});
}
/// Has the render on this thread failed?
pub(crate) fn failed() -> bool {
FAILURE.try_with(|f| f.borrow().is_some()).unwrap_or(false)
}
/// The render's failure, clearing it.
pub(crate) fn take_failure() -> Option<Failure> {
FAILURE.try_with(|f| f.borrow_mut().take()).ok().flatten()
}
/// wgpu error scopes around a render: out-of-memory, validation and internal errors raised by
/// this thread's calls are captured (instead of reaching the uncaptured-error handler) and fail
/// the render.
pub(crate) struct ErrorScopes([wgpu::ErrorScopeGuard; 3]);
impl ErrorScopes {
pub fn push(g: &Gpu) -> ErrorScopes {
ErrorScopes([
g.device.push_error_scope(wgpu::ErrorFilter::OutOfMemory),
g.device.push_error_scope(wgpu::ErrorFilter::Validation),
g.device.push_error_scope(wgpu::ErrorFilter::Internal),
])
}
/// Pop the scopes: the most telling error captured (out of memory first — an allocation
/// failure makes later commands invalid too — then internal, then validation).
pub fn pop(self) -> Option<Failure> {
let [oom, validation, internal] = self.0;
// scopes pop in reverse order of their creation
let internal = pollster::block_on(internal.pop());
let validation = pollster::block_on(validation.pop());
let oom = pollster::block_on(oom.pop());
if let Some(e) = oom {
return Some(Failure { kind: FailKind::OutOfMemory, reason: format!("out of device memory: {e}") });
}
internal.or(validation).map(|e| Failure { kind: FailKind::Fatal, reason: format!("device error: {e}") })
}
}
fn storage_entry(binding: u32, read_only: bool) -> wgpu::BindGroupLayoutEntry {
wgpu::BindGroupLayoutEntry {
binding,
@@ -529,3 +707,23 @@ pub fn groups1(n: usize) -> [u32; 3] {
let x = g.min(32768);
[x as u32, g.div_ceil(x) as u32, 1]
}
#[cfg(test)]
mod tests {
#[test]
fn limits_override_parses() {
assert_eq!(super::parse_limits("webgpu"), Some((128 << 20, 256 << 20)));
assert_eq!(super::parse_limits(" Downlevel "), Some((128 << 20, 256 << 20)));
assert_eq!(super::parse_limits("64"), Some((64 << 20, 128 << 20)));
assert_eq!(super::parse_limits("512MiB"), Some((512 << 20, 1024 << 20)));
assert_eq!(super::parse_limits(""), None);
assert_eq!(super::parse_limits("lots"), None);
}
#[test]
fn dispatches_stay_within_the_workgroup_limit() {
// 1-D kernels over a 100 MP image and 2-D ones over 16k × 16k stay under 65535 per axis
assert!(super::groups1(100_000_000 * 3).iter().all(|g| *g <= 65535));
assert!(super::groups2(16384, 16384, [16, 16]).iter().all(|g| *g <= 65535));
}
}
+141 -15
View File
@@ -7,14 +7,16 @@
//! CPU inside the same render (per-stage hybrid); see `docs/gpu-pipeline.md`.
//!
//! Use [`render`]: it returns `None` when the GPU is unavailable, disabled (`LIGHTCRAFT_GPU=0` or
//! [`set_enabled`]), or the render does not fit the device — callers then render on the CPU.
//! [`set_enabled`]), the render does not fit the device, or the device reported an error, ran out
//! of memory or returned an incomplete image — callers then render on the CPU. [`unavailable_reason`]
//! and [`last_fallback`] say why (`ui.inspect` → `perf.gpuReason` / `perf.gpuFallback`).
//! The browser build has no GPU path yet (WebGPU device creation is asynchronous): everything here
//! compiles to the CPU fallback on wasm32.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use lightcraft_develop::DevelopSettings;
use lightcraft_pipeline::{RenderRequest, Rendered, SourceInfo, StageCache};
@@ -34,12 +36,95 @@ static ENABLED: AtomicBool = AtomicBool::new(true);
/// Set when a GPU render failed: the process stays on the CPU from then on.
static BROKEN: AtomicBool = AtomicBool::new(false);
/// Why the GPU stopped being used (the first fatal failure).
static BROKEN_REASON: Mutex<Option<String>> = Mutex::new(None);
/// The latest render that fell back to the CPU, and why.
static LAST_FALLBACK: Mutex<Option<String>> = Mutex::new(None);
/// Stop using the GPU for the rest of the process (after a device error).
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
fn mark_broken() {
fn mark_broken(reason: &str) {
let mut r = BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner());
if r.is_none() {
*r = Some(reason.to_string());
}
BROKEN.store(true, Ordering::Relaxed);
}
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
fn record_fallback(reason: String) {
*LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()) = Some(reason);
}
/// Why renders do not use the GPU, or `None` when they do (or will, once the device is created).
/// E.g. "disabled by LIGHTCRAFT_GPU=0", "software adapter (llvmpipe …) skipped", "device lost …".
/// Never blocks.
pub fn unavailable_reason() -> Option<String> {
if env_disabled() {
return Some("disabled by LIGHTCRAFT_GPU=0".into());
}
if !ENABLED.load(Ordering::Relaxed) {
return Some("disabled by the GPU rendering preference (app.gpu)".into());
}
if BROKEN.load(Ordering::Relaxed) {
let r = BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner()).clone().unwrap_or_else(|| "device error".into());
return Some(format!("stopped after a GPU failure: {r}"));
}
#[cfg(not(target_arch = "wasm32"))]
{
match GPU.get() {
Some(Err(e)) => Some(e.clone()),
_ => None,
}
}
#[cfg(target_arch = "wasm32")]
{
Some("the browser build has no GPU path yet".into())
}
}
/// The latest render that fell back from the GPU to the CPU and why (device limit, out of memory,
/// device error, incomplete or blank result), if any.
pub fn last_fallback() -> Option<String> {
LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
/// A simulated GPU failure for the next [`render`] on this thread (tests of the CPU fallback).
#[doc(hidden)]
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Fault {
/// A command the device rejects (validation error).
Validation,
/// The per-pixel stage's work never runs (what a driver that silently drops or resets a
/// submission looks like: the readback holds no image).
DropWork,
/// A storage-buffer limit of this many bytes for this render.
Limit(u64),
}
thread_local! {
static FAULT: std::cell::Cell<Option<Fault>> = const { std::cell::Cell::new(None) };
}
/// Make the next [`render`] on this thread fail with `f` (tests).
#[doc(hidden)]
pub fn inject_fault(f: Fault) {
FAULT.with(|c| c.set(Some(f)));
}
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
fn take_fault() -> Option<Fault> {
FAULT.with(|c| c.take())
}
/// Use the GPU again after a failure (tests that inject faults).
#[doc(hidden)]
pub fn reset_failures() {
BROKEN.store(false, Ordering::Relaxed);
*BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner()) = None;
*LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()) = None;
}
/// Allow or forbid GPU rendering at runtime (a preference). `LIGHTCRAFT_GPU=0` forbids it for the
/// whole process regardless.
pub fn set_enabled(on: bool) {
@@ -57,20 +142,20 @@ fn env_disabled() -> bool {
}
#[cfg(not(target_arch = "wasm32"))]
static GPU: std::sync::OnceLock<Option<ctx::Gpu>> = std::sync::OnceLock::new();
static GPU: std::sync::OnceLock<Result<ctx::Gpu, String>> = std::sync::OnceLock::new();
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn device() -> Option<&'static ctx::Gpu> {
if env_disabled() {
return None;
}
GPU.get_or_init(|| std::panic::catch_unwind(ctx::Gpu::new).ok().flatten()).as_ref()
GPU.get_or_init(|| std::panic::catch_unwind(ctx::Gpu::new).unwrap_or_else(|_| Err("device creation panicked".into()))).as_ref().ok()
}
/// The device if it has been created (never creates it).
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn existing_device() -> Option<&'static ctx::Gpu> {
GPU.get().and_then(|g| g.as_ref())
GPU.get().and_then(|g| g.as_ref().ok())
}
/// Create the device and compile the kernels on a background thread now (app start), so the
@@ -160,7 +245,7 @@ pub fn set_pool_limit(bytes: u64) {
/// Free recycled buffers until at most `keep` bytes stay pooled (e.g. when the app goes idle).
pub fn trim_pool(keep: u64) {
#[cfg(not(target_arch = "wasm32"))]
if let Some(g) = GPU.get().and_then(|g| g.as_ref()) {
if let Some(g) = existing_device() {
g.trim(keep);
}
#[cfg(target_arch = "wasm32")]
@@ -191,19 +276,60 @@ pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &R
}
let gpu = device()?;
let ext = stages.map(|c| c.extension::<GpuStages>());
let r = {
let fault = take_fault();
let _ = ctx::take_failure(); // nothing left over from an earlier render on this thread
let (r, scoped) = {
let _scope = ctx::RenderScope::new(gpu);
std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| render::render(gpu, src, info, s, req, ext.as_deref())))
let errors = ctx::ErrorScopes::push(gpu);
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| render::render(gpu, src, info, s, req, ext.as_deref(), fault)));
(r, errors.pop())
};
match r {
// a device error during the render: its result is not trustworthy
Ok(_) if BROKEN.load(Ordering::Relaxed) => None,
Ok(r) => r,
Err(_) => {
// An error the device raised explains the readback failures that follow it; a buffer
// over the limit explains the errors its placeholder raises.
let failure = match (ctx::take_failure(), scoped) {
(Some(f), _) if f.kind == ctx::FailKind::Limit => Some(f),
(f, scoped) => scoped.or(f),
};
match (r, failure) {
(Err(_), _) => {
log::error!("gpu: render failed; using the CPU pipeline from now on");
BROKEN.store(true, Ordering::Relaxed);
mark_broken("a GPU render panicked");
record_fallback("a GPU render panicked; the GPU is not used again".into());
None
}
(Ok(r), Some(f)) => {
// the view's cached device stages may hold results of the failed passes
if let Some(e) = &ext {
e.clear();
}
let what = match &r {
Some(r) => format!("{}×{}", r.image.width, r.image.height),
None => "GPU render".into(),
};
let reason = match f.kind {
ctx::FailKind::Fatal => {
mark_broken(&f.reason);
format!("{what}: {}; the GPU is not used again", f.reason)
}
ctx::FailKind::OutOfMemory => {
gpu.trim(0);
format!("{what}: {}", f.reason)
}
ctx::FailKind::Limit | ctx::FailKind::Blank => format!("{what}: {}", f.reason),
};
log::warn!("gpu: rendering on the CPU instead: {reason}");
if render::profiling() {
eprintln!(" gpu fallback to the CPU: {reason}");
}
record_fallback(reason);
None
}
// a device error on another thread or a lost device during the render
(Ok(Some(_)), None) if BROKEN.load(Ordering::Relaxed) => {
record_fallback(format!("device failure during the render: {}", unavailable_reason().unwrap_or_default()));
None
}
(Ok(r), None) => r,
}
}
#[cfg(target_arch = "wasm32")]
+3 -1
View File
@@ -71,6 +71,8 @@ const FIELDS: &[(&str, usize)] = &[
("OUT_Y", 3),
("OUT_TRC", 1),
("OUT_GAMMA", 1),
// first row of a band dispatch (the kernel runs over rows Y0.., see `render`)
("Y0", 1),
];
/// `(name, index)` of every field (for the WGSL constants).
@@ -87,7 +89,7 @@ pub fn finish_fields() -> Vec<(&'static str, usize)> {
}
/// Offset of field `name`; `None` (and a debug assertion) for a name not in [`FIELDS`].
fn index(name: &str) -> Option<usize> {
pub fn index(name: &str) -> Option<usize> {
let mut i = 0;
for (n, len) in FIELDS {
if *n == name {
+98 -24
View File
@@ -12,7 +12,7 @@ use lightcraft_pipeline::{Plan, RenderRequest, Rendered, SourceInfo, local};
use lightcraft_raster::resample::Filter;
use lightcraft_raster::{Histogram, Plane, Rgb32f, Rgba8};
use crate::ctx::{Buf, Gpu, groups1, groups2};
use crate::ctx::{Buf, FailKind, Gpu, fail, groups1, groups2};
use crate::params::{Present, finish_block};
/// Device-resident stages of one view, kept next to the CPU [`lightcraft_pipeline::StageCache`]
@@ -128,27 +128,54 @@ impl GpuStages {
pub(crate) struct Cx<'a> {
pub gpu: &'a Gpu,
enc: Option<wgpu::CommandEncoder>,
/// Kernel invocations recorded since the last submit.
pending: u64,
}
/// Kernel invocations recorded before a render submits them (under one full-size pass over a
/// 24 MP export, several passes over a preview): no single submission runs long.
const FLUSH_INVOCATIONS: u64 = 16 << 20;
/// Pixels per dispatch of the per-pixel stage (the heaviest kernel).
const BAND_PIXELS: usize = 4 << 20;
impl<'a> Cx<'a> {
pub fn new(gpu: &'a Gpu) -> Cx<'a> {
Cx { gpu, enc: Some(gpu.encoder()) }
Cx { gpu, enc: Some(gpu.encoder()), pending: 0 }
}
/// Record kernel `name` (see [`Gpu::run`]).
pub fn run(&mut self, name: &str, p: &[u32], bufs: &[Option<&Buf>], groups: [u32; 3]) {
let enc = self.enc.get_or_insert_with(|| self.gpu.encoder());
self.gpu.run(enc, name, p, bufs, groups);
// every kernel runs 256 invocations per workgroup
self.pending += groups.iter().map(|g| *g as u64).product::<u64>() * 256;
if self.pending >= FLUSH_INVOCATIONS {
self.flush();
}
}
/// Submit what is recorded and read back `len` values of `b`.
pub fn read<T: bytemuck::Pod>(&mut self, b: &Buf, len: usize) -> Vec<T> {
self.pending = 0;
let enc = self.enc.take().unwrap_or_else(|| self.gpu.encoder());
self.gpu.finish_and_read(enc, b, len)
}
/// Submit what is recorded so far without waiting. Renders submit stage by stage and every
/// [`FLUSH_INVOCATIONS`]: long submissions are what GPU watchdogs reset (Windows TDR, the
/// i915 hang check; issue #78's iGPU spent seconds on an export recorded as one submission),
/// and buffers dropped by earlier work become reusable by later work (lower peak memory).
pub fn flush(&mut self) {
self.pending = 0;
if let Some(enc) = self.enc.take() {
self.gpu.submit([enc.finish()]);
}
}
/// Submit what is recorded and wait for it (profiling only: stage timings).
pub fn sync(&mut self) {
self.pending = 0;
if let Some(enc) = self.enc.take() {
self.gpu.submit([enc.finish()]);
}
@@ -434,12 +461,13 @@ struct Host {
log_l: Option<Arc<Plane>>,
}
fn profiling() -> bool {
pub(crate) fn profiling() -> bool {
static ON: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*ON.get_or_init(|| std::env::var_os("LIGHTCRAFT_PROFILE").is_some())
}
/// Render on `gpu`, reusing `stages` (if given). `None` when the render does not fit the device.
/// Render on `gpu`, reusing `stages` (if given). `None` (or a result the caller discards) when the
/// render failed: the reason is recorded with [`crate::ctx::fail`].
pub fn render(
gpu: &Gpu,
src: &Arc<Rgb32f>,
@@ -447,20 +475,31 @@ pub fn render(
s: &DevelopSettings,
req: &RenderRequest,
stages: Option<&GpuStages>,
fault: Option<crate::Fault>,
) -> Option<Rendered> {
let mut t = profiling().then(std::time::Instant::now);
// Under `LIGHTCRAFT_PROFILE` each stage is submitted and waited for, so the timings are real.
let lap = |what: &str, t: &mut Option<std::time::Instant>, cx: &mut Cx<'_>| {
if let Some(t) = t {
// Each stage is submitted on its own; under `LIGHTCRAFT_PROFILE` also waited for, so the
// timings are real.
let lap = |what: &str, t: &mut Option<std::time::Instant>, cx: &mut Cx<'_>| match t {
Some(t) => {
cx.sync();
eprintln!(" gpu {what}: {:.1} ms", t.elapsed().as_secs_f64() * 1e3);
*t = std::time::Instant::now();
}
None => cx.flush(),
};
let plan = lightcraft_pipeline::plan(src, info, s, req);
let (w, h) = (plan.w, plan.h);
let n = w * h;
let _limit = match fault {
Some(crate::Fault::Limit(bytes)) => Some(crate::ctx::LimitOverride::new(bytes)),
_ => None,
};
if !gpu.fits(n * 3) {
fail(
FailKind::Limit,
format!("{w}×{h} needs {} MiB buffers, over the device's {} MiB storage-buffer limit", (n * 12).div_ceil(1 << 20), gpu.limit() >> 20),
);
return None;
}
let s = &*plan.settings;
@@ -521,23 +560,35 @@ pub fn render(
};
let (p, aux) = finish_block(&fp, &terms, &present);
let aux = gpu.upload(&aux);
let out = gpu.buffer(n);
cx.run(
"main",
&p,
&[
Some(&lin),
Some(&prep.log_l),
Some(&prep.base),
prep.clarity.as_deref(),
prep.texture.as_deref(),
prep.dark.as_deref(),
masks.as_ref(),
Some(&aux),
Some(&out),
],
groups2(w, h, [16, 16]),
);
// cleared, so that work which never ran cannot pass for an image (see the check below)
let out = cx.zeroed(n);
if fault == Some(crate::Fault::Validation) {
// a dispatch the device rejects (too many workgroups)
cx.run("box_h", &[0; 5], &[Some(&aux), Some(&out)], [gpu.device.limits().max_compute_workgroups_per_dimension + 1, 1, 1]);
}
// in bands of rows: each dispatch (and submission) stays short on a slow GPU
let rows = (BAND_PIXELS / w.max(1) / 16 * 16).max(16); // whole workgroups: bands never overlap
let y0_at = crate::params::index("Y0").unwrap_or(0);
let mut p = p;
for y0 in (0..h).step_by(rows).filter(|_| fault != Some(crate::Fault::DropWork)) {
p[y0_at] = y0 as u32;
cx.run(
"main",
&p,
&[
Some(&lin),
Some(&prep.log_l),
Some(&prep.base),
prep.clarity.as_deref(),
prep.texture.as_deref(),
prep.dark.as_deref(),
masks.as_ref(),
Some(&aux),
Some(&out),
],
groups2(w, rows.min(h - y0), [16, 16]),
);
}
// the alpha a mask overlay shows: copied out before the readback below submits
let overlay_mask = req.overlay.mask(s).map(|m| {
let list = s.masks.iter().filter(|m| m.visible && !m.components.is_empty());
@@ -549,8 +600,25 @@ pub fn render(
let data: Vec<[u8; 4]> = cx.read(&out, n);
let image = Rgba8 { width: w, height: h, data };
lap("finish + readback", &mut t, &mut cx);
if crate::ctx::failed() {
return None;
}
// The per-pixel kernel writes alpha 255 everywhere and `out` was cleared: a pixel with any
// other alpha means work silently did not run (a driver that reset or dropped a submission
// without reporting it — issue #78's all-black exports).
let unwritten = unwritten(&image.data);
if unwritten > 0 {
fail(FailKind::Fatal, format!("the GPU returned an incomplete image ({unwritten} of {n} pixels unwritten)"));
return None;
}
let histogram = Histogram::of_srgb8(&image);
lap("histogram", &mut t, &mut cx);
// An entirely black result (upstream work that did not run would give that too): redo it on
// the CPU, which costs time only for the rare photo that really is black.
if n >= 4096 && [&histogram.r, &histogram.g, &histogram.b].iter().all(|c| c.first() == Some(&histogram.total)) {
fail(FailKind::Blank, "the GPU result is entirely black; rendering it on the CPU to be sure".into());
return None;
}
let mut image = image;
let overlay_mask = overlay_mask.map(|r| match r {
Ok(b) => cx.read_plane(&b, w, h),
@@ -565,6 +633,11 @@ pub fn render(
Some(Rendered { image, histogram, deep: None })
}
/// Pixels whose alpha is not 255 (cheap: no early exit, so it vectorizes).
fn unwritten(data: &[[u8; 4]]) -> usize {
data.chunks(4096).filter(|c| c.iter().fold(255u8, |a, p| a & p[3]) != 255).map(|c| c.iter().filter(|p| p[3] != 255).count()).sum()
}
/// The white-balanced, retouched, denoised image.
fn linear(cx: &mut Cx<'_>, sampled: &Buf, info: &SourceInfo, plan: &Plan<'_>, host: &mut Host) -> Buf {
let (w, h) = (plan.w, plan.h);
@@ -679,6 +752,7 @@ fn prepare(cx: &mut Cx<'_>, lin: &Buf, plan: &Plan<'_>, req: &RenderRequest, pla
let sub = cx.gpu.buffer(m);
map(cx, "subsample", m, &[local::AIRLIGHT_STEP as u32], [Some(&d), None, None], &sub);
let air = local::airlight_of(cx.read(&sub, m));
cx.flush();
let b = Arc::new(d);
planes.dark = Some((sg.to_bits(), b.clone(), air));
(Some(b), air)
+3 -3
View File
@@ -243,11 +243,11 @@ fn out_encode(v: f32) -> f32 {
fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
let w = pu(F_W);
let h = pu(F_H);
if (gid.x >= w || gid.y >= h) {
let x = gid.x;
let y = gid.y + pu(F_Y0);
if (x >= w || y >= h) {
return;
}
let x = gid.x;
let y = gid.y;
let i = y * w + x;
let n = w * h;
let gain = pf(F_GAIN);
+26
View File
@@ -65,6 +65,32 @@ fn typical(s: &mut DevelopSettings) {
s.detail.sharpen_amount = 40.0;
}
/// A render big enough for the per-pixel stage to run in several bands of rows (and the render
/// in many submissions), with grain (position-dependent) and masks.
#[test]
fn banded_full_size_render_matches() {
if !gpu() {
return;
}
let (w, h) = (3000, 2000);
let src = scene(3, w, h);
let mut s = DevelopSettings::default();
typical(&mut s);
s.grain.amount = 30.0;
s.vignette.amount = -30.0;
s.masks = vec![Mask {
components: vec![MaskComponent {
name: None,
op: MaskOp::Add,
invert: false,
shape: MaskShape::Linear { start: Point::new(0.5, 0.0), end: Point::new(0.5, 0.9) },
}],
adjust: lightcraft_develop::LocalAdjustments { exposure: -0.6, ..Default::default() },
..Default::default()
}];
check("banded 3000×2000", &src, &SourceInfo { raw: true, ..Default::default() }, &s, &RenderRequest::fit(w, h));
}
fn cases() -> Vec<(&'static str, Edit)> {
vec![
("default", |_| {}),
+103
View File
@@ -0,0 +1,103 @@
//! Issue #78: a GPU render that cannot be trusted — over the device's buffer limits, a device
//! error, work that silently did not run — must come back as `None` (the caller renders on the
//! CPU) with the reason recorded, never as an image of zeros. One test, sequential: the fatal
//! faults stop GPU use for the whole process until `reset_failures`.
//! Skips (passes with a note) when no GPU adapter exists.
use std::sync::Arc;
use lightcraft_develop::{DevelopSettings, Mask, MaskComponent, MaskOp, MaskShape};
use lightcraft_geom::Point;
use lightcraft_gpu::Fault;
use lightcraft_pipeline::{RenderRequest, SourceInfo, render};
use lightcraft_raster::Rgba8;
fn mean_abs_diff(a: &Rgba8, b: &Rgba8) -> f64 {
let sum: u64 = a.data.iter().zip(&b.data).map(|(p, q)| (0..3).map(|c| p[c].abs_diff(q[c]) as u64).sum::<u64>()).sum();
sum as f64 / (a.data.len() * 3) as f64
}
fn masked(s: &mut DevelopSettings, k: usize) {
for i in 0..k {
let c = Point::new(0.2 + 0.2 * i as f64, 0.5);
s.masks.push(Mask {
id: i as u32 + 1,
components: vec![MaskComponent {
name: None,
op: MaskOp::Add,
invert: false,
shape: MaskShape::Radial { center: c, rx: 0.2, ry: 0.15, angle: 0.0, feather: 50.0, invert: false },
}],
..Default::default()
});
s.masks[i].adjust.exposure = 0.5;
}
}
#[test]
fn failed_gpu_renders_fall_back_with_a_reason() {
if !lightcraft_gpu::available() {
eprintln!("skipped: no GPU adapter ({:?})", lightcraft_gpu::unavailable_reason());
return;
}
assert_eq!(lightcraft_gpu::unavailable_reason(), None);
let (w, h) = (1200, 800);
let src = Arc::new(lightcraft_scenes::demo_library()[0].render(w, h));
let info = SourceInfo { raw: true, ..Default::default() };
let mut s = DevelopSettings::default();
s.light.exposure = 0.3;
s.effects.clarity = 20.0;
masked(&mut s, 4);
let req = RenderRequest::fit(w, h);
let cpu = render(&src, &info, &s, &req).image;
let ok = |what: &str| {
let g = lightcraft_gpu::render(&src, &info, &s, &req, None)
.unwrap_or_else(|| panic!("{what}: gpu render ({:?})", lightcraft_gpu::last_fallback()));
let d = mean_abs_diff(&cpu, &g.image);
assert!(d < 0.5, "{what}: GPU differs from the CPU by {d:.3} LSB");
};
ok("healthy");
// the output itself is over the limit: refused up front, the GPU stays in use
lightcraft_gpu::inject_fault(Fault::Limit(1 << 20));
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("limit"), "{why}");
assert!(lightcraft_gpu::available(), "a limit is no device failure");
// the output fits but the mask planes (4 masks: 4 planes in one buffer) don't: caught when the
// buffer is created, without a device error, and the GPU stays in use
lightcraft_gpu::inject_fault(Fault::Limit((w * h * 3 * 4 + 1024) as u64));
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("exceeds"), "{why}");
assert!(lightcraft_gpu::available(), "{:?}", lightcraft_gpu::unavailable_reason());
ok("after the limit faults");
// a device (validation) error: captured by the render's error scope; the GPU is not used again
lightcraft_gpu::inject_fault(Fault::Validation);
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("device error"), "{why}");
assert!(!lightcraft_gpu::available());
assert!(lightcraft_gpu::unavailable_reason().is_some_and(|r| r.contains("device error")), "{:?}", lightcraft_gpu::unavailable_reason());
lightcraft_gpu::reset_failures();
ok("after a device error");
// the per-pixel work never runs (a driver dropping a submission without an error): before
// the fix this came back as a fully black image
lightcraft_gpu::inject_fault(Fault::DropWork);
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none(), "an unwritten image must not be returned");
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("incomplete"), "{why}");
assert!(!lightcraft_gpu::available());
lightcraft_gpu::reset_failures();
ok("after dropped work");
// a black photo is still rendered (on the CPU, to be sure)
let black = Arc::new(lightcraft_raster::Rgb32f::from_fn(256, 256, |_, _| [0.0; 3]));
let s0 = DevelopSettings::default();
assert!(lightcraft_gpu::render(&black, &info, &s0, &RenderRequest::fit(256, 256), None).is_none());
assert!(lightcraft_gpu::last_fallback().unwrap_or_default().contains("black"));
assert!(lightcraft_gpu::available());
}
+3 -7
View File
@@ -78,12 +78,7 @@ impl Headless {
None => vec![self.photo_or_active(p)?],
};
let dir = p.get("dir").and_then(Value::as_str).unwrap_or("");
let write = &mut |path: &str, bytes: &[u8]| {
if let Some(parent) = Path::new(path).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(parent).map_err(|err| err.to_string())?;
}
std::fs::write(path, bytes).map_err(|err| format!("{path}: {err}"))
};
let write = &mut lightcraft_engine::export::write_file;
let files =
export_batch(&mut self.session, &ids, &opts, &Destination { dir: dir.to_string(), exact: exact.map(str::to_string) }, write, &|path| {
Path::new(path).exists()
@@ -118,6 +113,7 @@ impl Backend for Headless {
let img = self.render(&p, 1600)?;
match p.get("path").and_then(Value::as_str) {
Some(path) => {
self.session.check_write_target(path)?;
write_image(Path::new(path), &img, 92)?;
Ok(json!({"path": path, "width": img.width, "height": img.height}))
}
@@ -183,7 +179,7 @@ pub fn encode_image(ext: &str, img: &Rgba8, quality: u8) -> Result<Vec<u8>, Stri
pub fn write_image(path: &Path, img: &Rgba8, quality: u8) -> Result<(), String> {
let ext = path.extension().map(|e| e.to_string_lossy().to_string()).unwrap_or_default();
let bytes = encode_image(&ext, img, quality)?;
std::fs::write(path, bytes).map_err(|e| format!("{}: {e}", path.display()))
lightcraft_engine::export::write_file(&path.to_string_lossy(), &bytes)
}
#[cfg(test)]
+29 -1
View File
@@ -1,6 +1,6 @@
use serde_json::{Value, json};
use crate::{Headless, PROTOCOL_VERSION, Server, call_tool, command_tool_name};
use crate::{Backend, Headless, PROTOCOL_VERSION, Server, call_tool, command_tool_name};
fn server() -> Server {
Server::new(Box::new(Headless::demo()))
@@ -92,6 +92,34 @@ fn import_tool_copies_with_a_folder_template() {
let _ = std::fs::remove_dir_all(&base);
}
/// Issue #93: `render_photo`, `ui.render` and `app.export` with an exact `path` must not replace
/// a photo's original.
#[test]
fn path_writes_never_replace_an_original() {
let base = std::env::temp_dir().join(format!("lc-mcp-guard-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).unwrap();
let img = lightcraft_raster::Rgba8 { width: 8, height: 8, data: vec![[90, 30, 9, 255]; 64] };
let png = lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(&img), &lightcraft_codecs::EncodeMeta::default()).unwrap();
let orig = base.join("a.png");
std::fs::write(&orig, &png).unwrap();
let mut b = Headless::demo();
let r = call_tool(&mut b, "import", &json!({"paths": [orig.to_string_lossy()]}));
assert!(!r.is_error, "{r:?}");
let id = b.session.catalog.photos().find(|p| p.file_name == "a.png").unwrap().id.0;
let path = orig.to_string_lossy().to_string();
let r = call_tool(&mut b, "render_photo", &json!({"id": id, "path": path}));
assert!(r.is_error, "{r:?}");
assert!(b.call("ui.render", json!({"id": id, "path": path})).unwrap_err().contains("original"));
let e = b.call("app.export", json!({"ids": [id], "path": path})).unwrap_err();
assert!(e.contains("never writes over an original"), "{e}");
assert_eq!(std::fs::read(&orig).unwrap(), png, "the original is untouched");
// another path still works
let r = call_tool(&mut b, "render_photo", &json!({"id": id, "path": base.join("render.png").to_string_lossy()}));
assert!(!r.is_error, "{r:?}");
let _ = std::fs::remove_dir_all(&base);
}
/// The `import` helper moves: renamed into the folder template, the source removed.
#[test]
fn import_tool_moves_with_a_folder_template() {
+17 -3
View File
@@ -205,7 +205,7 @@ pub fn helper_tools(has_ui: bool) -> Vec<Value> {
"removeLocation": {"type": "boolean"},
"colorSpace": {"type": "string", "enum": ["srgb", "displayP3", "adobeRgb", "proPhoto", "rec2020"], "description": "Output colour space (default sRGB; AVIF is always sRGB). adobeRgb = Adobe RGB (1998) compatible; the embedded ICC profile is generated from the published primaries"},
"bitDepth": {"type": "integer", "enum": [8, 10, 16, 32], "description": "Bits per channel: PNG 8|16 (default 8), TIFF 8|16|32 (default 16; 32 = linear float with a linear profile), AVIF 8|10; JPEG/WebP are 8-bit"},
"watermark": {"description": "Text, or {text, size (fraction of short edge), opacity, anchor (topLeft|top|topRight|left|center|right|bottomLeft|bottom|bottomRight), inset, color [r,g,b], shadow}"}
"watermark": {"description": "Text, or {text, vertical (boolean; defaults to false), size (fraction of short edge), opacity, anchor (topLeft|top|topRight|left|center|right|bottomLeft|bottom|bottomRight), inset, color [r,g,b], shadow}"}
}),
&[],
),
@@ -352,9 +352,9 @@ fn image_result(file: &std::path::Path, max: Option<u32>, format: &str, save_to:
(bytes, w, h)
};
if let Some(p) = save_to
&& let Err(e) = std::fs::write(p, &bytes)
&& let Err(e) = lightcraft_engine::export::write_file(p, &bytes)
{
return ToolResult::error(format!("{p}: {e}"));
return ToolResult::error(e);
}
let mime = if jpeg { "image/jpeg" } else { "image/png" };
let mut info = meta;
@@ -372,7 +372,18 @@ fn image_result(file: &std::path::Path, max: Option<u32>, format: &str, save_to:
}
}
/// A user-given `path` to save to must not be a photo's original (or its sidecar).
fn check_save_path(b: &mut dyn Backend, args: &Value) -> Result<(), String> {
match args.get("path").and_then(Value::as_str) {
Some(p) => exec(b, "export.checkTarget", json!({"path": p})).map(|_| ()),
None => Ok(()),
}
}
fn render_photo(b: &mut dyn Backend, args: &Value) -> ToolResult {
if let Err(e) = check_save_path(b, args) {
return ToolResult::error(e);
}
let size = args.get("size").and_then(Value::as_u64).unwrap_or(1024).clamp(16, 4096);
let id = match args.get("id").and_then(Value::as_u64) {
Some(id) => Some(id),
@@ -394,6 +405,9 @@ fn render_photo(b: &mut dyn Backend, args: &Value) -> ToolResult {
}
fn screenshot(b: &mut dyn Backend, args: &Value) -> ToolResult {
if let Err(e) = check_save_path(b, args) {
return ToolResult::error(e);
}
let file = temp_path("screenshot");
let path = file.to_string_lossy().to_string();
match b.call("ui.screenshot", json!({"path": path})) {
+12 -1
View File
@@ -71,7 +71,7 @@ impl DateTime {
};
let (h, m) = match rest.split_once(':') {
Some((h, m)) => (num(h)?, num(m)?),
None if rest.len() == 4 => (num(&rest[..2])?, num(&rest[2..])?),
None if rest.len() == 4 => (num(rest.get(..2)?)?, num(rest.get(2..)?)?),
None => (num(rest)?, 0),
};
if h > 18 || m > 59 {
@@ -180,6 +180,17 @@ mod tests {
assert_eq!(DateTime::parse_offset("+99:00"), None);
}
#[test]
fn unicode_offsets_are_rejected_without_panicking() {
for offset in ["+1€", "-1€", "+€1", "-€1"] {
assert_eq!(DateTime::parse_offset(offset), None);
let date = format!("2024-06-01T13:45:59{offset}");
let metadata = crate::Metadata { capture_time: DateTime::parse_iso("2024-06-01T13:45:59Z"), ..Default::default() };
let xmp = crate::write_xmp(&metadata, None).replace("2024-06-01T13:45:59+00:00", &date);
assert!(crate::parse_xmp(&xmp).unwrap().metadata.capture_time.is_none());
}
}
#[test]
fn iso_roundtrip() {
for s in ["2024-06-01T13:45:59", "2024-06-01T13:45:59.250+02:00", "1999-12-31T23:59:00-05:30", "2024-02-29T00:00:00Z"] {
+2
View File
@@ -17,6 +17,7 @@ mod gpx;
mod iptc;
pub mod tags;
mod xmp;
mod xmp_merge;
pub use container::{Embedded, embedded, jpeg_segments, png_chunks, webp_chunks};
pub use datetime::DateTime;
@@ -26,6 +27,7 @@ pub use iptc::parse_iptc;
pub use lightcraft_geom::Orientation;
pub use tags::{TagRow, file_tag_rows, tag_rows};
pub use xmp::{CRS_NS, LC_NS, XmpData, XmpError, XmpValue, parse_xmp, write_xmp, write_xmp_lc};
pub use xmp_merge::{MergeRules, merge_xmp};
use serde::{Deserialize, Serialize};
+5
View File
@@ -145,6 +145,11 @@ impl Node {
const MAX_DEPTH: usize = 64;
const MAX_NODES: usize = 200_000;
/// `prefix:local` with the prefix canonicalised by namespace URI (see [`NAMESPACES`]).
pub(crate) fn canonical_name(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
canonical(qname, scopes)
}
fn canonical(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
let (prefix, local) = qname.split_once(':').unwrap_or(("", qname));
if prefix == "xml" || prefix == "xmlns" {
+403
View File
@@ -0,0 +1,403 @@
//! Merging a freshly written XMP packet into an existing one (a sidecar another application
//! wrote, e.g. with its develop settings and edit history), keeping everything the writer does not
//! own byte for byte.
//!
//! The existing packet is scanned once (namespace-resolved, with byte positions). Only top-level
//! properties of the `rdf:Description` elements directly under `rdf:RDF` are considered, in
//! attribute form (`xmp:Rating="3"`) and element form (`<dc:subject>…</dc:subject>`):
//! - properties the writer owns ([`MergeRules::owned`]) are removed, and written from the fresh
//! packet when it has them (so a cleared field is cleared in the file too);
//! - properties owned only when stated ([`MergeRules::owned_if_present`]) are replaced when the
//! fresh packet has them and kept otherwise;
//! - any other property of the fresh packet (e.g. `xmp:CreatorTool`) is not written: the file
//! keeps describing itself as its creator left it;
//! - everything else — other namespaces (e.g. `crs:`), `xmpMM:History`, unknown properties,
//! comments, the packet wrapper and padding — is copied unchanged.
//!
//! The written properties go into one new `rdf:Description` (declaring its own namespaces, so
//! the existing prefix bindings never matter) just before `</rdf:RDF>`. A description left empty
//! by the removals is dropped, so repeated saves don't accumulate. The result is checked by
//! parsing it again: every written property must read back as in the fresh packet.
use std::collections::BTreeSet;
use quick_xml::events::Event;
use crate::xmp::{XmpError, canonical_name, parse_xmp};
/// Which properties [`merge_xmp`] takes from the fresh packet. Names are canonical
/// `prefix:name` (prefixes as [`crate::parse_xmp`] reports them); `prefix:*` is a whole namespace.
#[derive(Clone, Copy, Debug, Default)]
pub struct MergeRules<'a> {
/// Always replaced: removed from the existing packet, written from the fresh one if stated.
pub owned: &'a [&'a str],
/// Replaced only when the fresh packet states them; otherwise the existing value stays.
pub owned_if_present: &'a [&'a str],
}
fn listed(list: &[&str], name: &str) -> bool {
list.iter().any(|p| match p.strip_suffix('*') {
Some(ns) => name.starts_with(ns),
None => *p == name,
})
}
#[derive(Debug)]
struct Attr {
raw: String,
name: String,
/// The value as written (still escaped).
value: String,
}
#[derive(Debug)]
struct Prop {
name: String,
start: usize,
end: usize,
}
#[derive(Debug, Default)]
struct Desc {
tag_start: usize,
tag_end: usize,
/// The element name as written (`rdf:Description`, or another prefix for the RDF namespace).
raw_name: String,
empty: bool,
attrs: Vec<Attr>,
props: Vec<Prop>,
/// The end of `</rdf:Description>` (`tag_end` when self-closing).
end: usize,
}
#[derive(Debug, Default)]
struct Scan {
descs: Vec<Desc>,
/// Where the first `</rdf:RDF>` starts.
rdf_close: Option<usize>,
}
enum Kind {
Other,
Rdf,
Desc(usize),
Prop { desc: usize, start: usize, name: String },
}
fn is_meta_attr(name: &str) -> bool {
name.starts_with("xmlns")
|| matches!(name, "rdf:about" | "rdf:ID" | "rdf:nodeID" | "xml:lang" | "rdf:parseType" | "rdf:resource" | "rdf:datatype")
}
fn pos(r: &quick_xml::Reader<&[u8]>) -> usize {
usize::try_from(r.buffer_position()).unwrap_or(usize::MAX)
}
/// Locate the descriptions, their top-level properties and `</rdf:RDF>` in a packet.
fn scan(s: &str) -> Result<Scan, XmpError> {
let mut reader = quick_xml::Reader::from_reader(s.as_bytes());
reader.config_mut().check_end_names = true;
let mut out = Scan::default();
let mut stack: Vec<Kind> = Vec::new();
let mut scopes: Vec<Vec<(String, String)>> = vec![vec![("xml".into(), "http://www.w3.org/XML/1998/namespace".into())]];
let mut count = 0usize;
loop {
let before = pos(&reader);
let ev = reader.read_event().map_err(|e| XmpError::Xml(e.to_string()))?;
let after = pos(&reader);
match ev {
Event::Start(ref e) | Event::Empty(ref e) => {
count += 1;
if count > 200_000 || stack.len() > 64 {
return Err(XmpError::TooComplex);
}
let empty = matches!(ev, Event::Empty(_));
let mut scope = Vec::new();
let mut raw_attrs = Vec::new();
for a in e.attributes() {
let a = a.map_err(|e| XmpError::Xml(e.to_string()))?;
let k = String::from_utf8_lossy(a.key.as_ref()).into_owned();
let v = String::from_utf8_lossy(&a.value).into_owned();
if k == "xmlns" {
scope.push((String::new(), v.clone()));
} else if let Some(p) = k.strip_prefix("xmlns:") {
scope.push((p.to_string(), v.clone()));
}
raw_attrs.push((k, v));
}
scopes.push(scope);
let raw_name = String::from_utf8_lossy(e.name().as_ref()).into_owned();
let name = canonical_name(&raw_name, &scopes);
let kind = match stack.last() {
Some(Kind::Rdf) if name == "rdf:Description" => {
let attrs = raw_attrs.into_iter().map(|(raw, value)| Attr { name: canonical_name(&raw, &scopes), raw, value }).collect();
out.descs.push(Desc { tag_start: before, tag_end: after, raw_name, empty, attrs, props: Vec::new(), end: after });
Kind::Desc(out.descs.len() - 1)
}
Some(Kind::Desc(i)) => Kind::Prop { desc: *i, start: before, name },
_ if name == "rdf:RDF" && out.rdf_close.is_none() => Kind::Rdf,
_ => Kind::Other,
};
if empty {
scopes.pop();
if let Kind::Prop { desc, start, name } = kind
&& let Some(d) = out.descs.get_mut(desc)
{
d.props.push(Prop { name, start, end: after });
}
} else {
stack.push(kind);
}
}
Event::End(_) => {
scopes.pop();
match stack.pop() {
Some(Kind::Prop { desc, start, name }) => {
if let Some(d) = out.descs.get_mut(desc) {
d.props.push(Prop { name, start, end: after });
}
}
Some(Kind::Desc(i)) => {
if let Some(d) = out.descs.get_mut(i) {
d.end = after;
}
}
Some(Kind::Rdf) => out.rdf_close = out.rdf_close.or(Some(before)),
Some(Kind::Other) => {}
None => return Err(XmpError::Xml("unbalanced end tag".into())),
}
}
Event::Eof => break,
_ => {}
}
}
if !stack.is_empty() {
return Err(XmpError::Xml("unclosed element".into()));
}
Ok(out)
}
/// `start` moved back over the indentation and line break before it (so a removed property
/// takes its line with it).
fn line_start(s: &str, start: usize) -> usize {
let b = s.as_bytes();
let mut i = start;
while i > 0 && matches!(b.get(i - 1), Some(b' ' | b'\t')) {
i -= 1;
}
if i > 0 && b.get(i - 1) == Some(&b'\n') {
i -= 1;
if i > 0 && b.get(i - 1) == Some(&b'\r') {
i -= 1;
}
}
i
}
/// Merge `fresh` (a packet as [`crate::write_xmp_lc`] writes it) into `existing` following
/// `rules` (see the module docs). Fails — and the caller must not overwrite `existing` blindly —
/// when `existing` isn't a well-formed XMP packet.
pub fn merge_xmp(existing: &str, fresh: &str, rules: MergeRules) -> Result<String, XmpError> {
let (bom, old) = match existing.strip_prefix('\u{feff}') {
Some(rest) => ("\u{feff}", rest),
None => ("", existing),
};
parse_xmp(old)?;
let ex = scan(old)?;
let rdf_close = ex.rdf_close.ok_or_else(|| XmpError::Xml("no rdf:RDF element".into()))?;
let new = scan(fresh)?;
let fresh_desc = new.descs.first().ok_or_else(|| XmpError::Xml("the new packet has no rdf:Description".into()))?;
let fresh_names: BTreeSet<&str> = fresh_desc.props.iter().map(|p| p.name.as_str()).collect();
let remove = |name: &str| listed(rules.owned, name) || (listed(rules.owned_if_present, name) && fresh_names.contains(name));
let write = |name: &str| listed(rules.owned, name) || listed(rules.owned_if_present, name);
// edits on `old`: (start, end, replacement), applied back to front
let mut edits: Vec<(usize, usize, String)> = Vec::new();
for d in &ex.descs {
let gone_attrs: Vec<&Attr> = d.attrs.iter().filter(|a| !is_meta_attr(&a.name) && remove(&a.name)).collect();
let gone_props: Vec<&Prop> = d.props.iter().filter(|p| remove(&p.name)).collect();
if gone_attrs.is_empty() && gone_props.is_empty() {
continue;
}
let attrs_left = d.attrs.iter().any(|a| !is_meta_attr(&a.name) && !remove(&a.name));
if !attrs_left && gone_props.len() == d.props.len() {
// nothing of its own left: drop the whole description
edits.push((line_start(old, d.tag_start), d.end, String::new()));
continue;
}
if !gone_attrs.is_empty() {
let mut tag = format!("<{}", d.raw_name);
for a in d.attrs.iter().filter(|a| is_meta_attr(&a.name) || !remove(&a.name)) {
tag.push_str(&format!("\n {}=\"{}\"", a.raw, a.value.replace('"', "&quot;")));
}
tag.push_str(if d.empty { "/>" } else { ">" });
edits.push((d.tag_start, d.tag_end, tag));
}
for p in gone_props {
edits.push((line_start(old, p.start), p.end, String::new()));
}
}
// the new description: its own namespace declarations, the properties to write
let mut block = String::from("\n <rdf:Description rdf:about=\"\"\n xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"");
for a in fresh_desc.attrs.iter().filter(|a| a.raw.starts_with("xmlns:") && a.raw != "xmlns:rdf") {
block.push_str(&format!("\n {}=\"{}\"", a.raw, a.value));
}
block.push('>');
let mut written = 0;
for p in fresh_desc.props.iter().filter(|p| write(&p.name)) {
let text = fresh.get(p.start..p.end).ok_or_else(|| XmpError::Xml("bad property span".into()))?;
block.push_str("\n ");
block.push_str(text);
written += 1;
}
block.push_str("\n </rdf:Description>");
if written > 0 {
let at = old.get(..rdf_close).map_or(rdf_close, |h| h.trim_end().len());
edits.push((at, at, block));
}
edits.sort_by_key(|e| (e.0, e.1));
// overlapping edits can't happen (properties sit inside their description, which is either
// dropped whole or edited piecewise), but never splice blindly
let mut out = String::with_capacity(old.len() + 4096);
out.push_str(bom);
let mut at = 0;
for (s, e, r) in &edits {
if *s < at {
return Err(XmpError::Xml("overlapping edits".into()));
}
out.push_str(old.get(at..*s).ok_or_else(|| XmpError::Xml("bad span".into()))?);
out.push_str(r);
at = *e;
}
out.push_str(old.get(at..).ok_or_else(|| XmpError::Xml("bad span".into()))?);
// read back: everything written must come out as written
let merged = parse_xmp(&out)?;
let want = parse_xmp(fresh)?;
for (k, v) in &want.properties {
let top = k.split('/').next().unwrap_or(k);
if write(top) && merged.properties.get(k) != Some(v) {
return Err(XmpError::Xml(format!("{k} did not survive the merge")));
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
const RULES: MergeRules =
MergeRules { owned: &["xmp:Rating", "xmp:Label", "dc:subject", "dc:title", "lc:*"], owned_if_present: &["exif:DateTimeOriginal"] };
/// A sidecar as another raw developer writes it: attribute-form simple properties, develop
/// settings, a structured history, an unknown namespace.
const FOREIGN: &str = "<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>
<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">
<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">
<rdf:Description rdf:about=\"\"
xmlns:xmp=\"http://ns.adobe.com/xap/1.0/\"
xmlns:xmpMM=\"http://ns.adobe.com/xap/1.0/mm/\"
xmlns:stEvt=\"http://ns.adobe.com/xap/1.0/sType/ResourceEvent#\"
xmlns:crs=\"http://ns.adobe.com/camera-raw-settings/1.0/\"
xmlns:dc=\"http://purl.org/dc/elements/1.1/\"
xmlns:foo=\"http://example.com/foo/\"
xmp:Rating=\"2\"
xmp:CreatorTool=\"Other App\"
crs:Exposure2012=\"+0.50\"
crs:Contrast2012=\"-12\"
foo:Thing=\"a &amp; b\">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
<dc:subject>
<rdf:Bag>
<rdf:li>old</rdf:li>
</rdf:Bag>
</dc:subject>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
";
/// With the packet's whitespace padding and trailer.
fn foreign() -> String {
format!("{FOREIGN}{}\n<?xpacket end=\"w\"?>", " ".repeat(100))
}
fn fresh(rating: i8, keywords: &[&str]) -> String {
let m = crate::Metadata {
software: Some("LightCraft".into()),
rating: Some(rating),
keywords: keywords.iter().map(|k| k.to_string()).collect(),
..Default::default()
};
crate::write_xmp_lc(&m, &[("settings", "{\"a\":1}"), ("flag", "pick")])
}
#[test]
fn keeps_foreign_properties_and_replaces_owned_ones() {
let out = merge_xmp(&foreign(), &fresh(4, &["new", "two"]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
let p = |k: &str| d.properties.get(k).cloned().unwrap_or_default();
assert_eq!(p("xmp:Rating"), vec!["4"]);
assert_eq!(p("dc:subject"), vec!["new", "two"]);
assert_eq!(p("lc:settings"), vec!["{\"a\":1}"]);
assert_eq!(p("lc:flag"), vec!["pick"]);
assert_eq!(p("xmp:CreatorTool"), vec!["Other App"], "not ours: kept");
assert_eq!(p("crs:Exposure2012"), vec!["+0.50"]);
assert_eq!(p("foo:Thing").len(), 1);
// untouched parts are byte for byte the same
for part in [
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>",
"crs:Contrast2012=\"-12\"",
"foo:Thing=\"a &amp; b\"",
"<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>\n<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">",
&format!("</x:xmpmeta>\n{}\n<?xpacket end=\"w\"?>", " ".repeat(100)),
] {
assert!(out.contains(part), "lost {part:?} in\n{out}");
}
assert!(!out.contains("xmp:Rating=\"2\""));
assert!(!out.contains("<rdf:li>old</rdf:li>"));
// saving again changes only our part, and doesn't pile up descriptions
let again = merge_xmp(&out, &fresh(1, &[]), RULES).unwrap();
let d = parse_xmp(&again).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["1".to_string()]));
assert!(!d.properties.contains_key("dc:subject"), "a cleared field is cleared");
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert_eq!(merge_xmp(&again, &fresh(1, &[]), RULES).unwrap(), again, "idempotent");
}
#[test]
fn owned_if_present_and_prefix_independence() {
// the existing packet binds the namespaces to other prefixes, and has a capture time
let ex = "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><r:RDF xmlns:r=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\
<r:Description r:about=\"\" xmlns:a=\"http://ns.adobe.com/xap/1.0/\" xmlns:e=\"http://ns.adobe.com/exif/1.0/\" \
a:Rating=\"5\" e:DateTimeOriginal=\"2020-01-01T00:00:00\"/></r:RDF></x:xmpmeta>";
let out = merge_xmp(ex, &fresh(3, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["3".to_string()]));
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2020-01-01T00:00:00".to_string()]), "fresh has none: kept");
let mut m = crate::Metadata { rating: Some(3), ..Default::default() };
m.capture_time = crate::DateTime::parse_iso("2024-05-06T07:08:09");
let out = merge_xmp(ex, &crate::write_xmp_lc(&m, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2024-05-06T07:08:09".to_string()]));
}
#[test]
fn refuses_what_it_cannot_read() {
for bad in ["", "not xml at all", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><rdf:RDF", "<a><b></a>", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>"] {
assert!(merge_xmp(bad, &fresh(1, &[]), RULES).is_err(), "{bad:?}");
}
}
}
+18 -1
View File
@@ -1,4 +1,4 @@
use lightcraft_meta::{Metadata, extract, parse_iptc, parse_xmp, read_exif, write_xmp};
use lightcraft_meta::{MergeRules, Metadata, extract, merge_xmp, parse_iptc, parse_xmp, read_exif, write_xmp};
use proptest::prelude::*;
proptest! {
@@ -21,6 +21,23 @@ proptest! {
let _ = parse_xmp(&s);
}
/// Merging into arbitrary (also damaged) packets never panics, and a merge that succeeds
/// reads back with the written values.
#[test]
fn merge_never_panics(cut in 0usize..900, junk in ".{0,20}", rating in 0i8..=5) {
let base = write_xmp(&Metadata { title: Some("t".into()), rating: Some(1), ..Default::default() }, Some("{}"));
let mut s: String = base.chars().take(cut).collect();
s.push_str(&junk);
s.extend(base.chars().skip(cut));
let fresh = write_xmp(&Metadata { rating: Some(rating), ..Default::default() }, Some("{\"x\":1}"));
let rules = MergeRules { owned: &["xmp:Rating", "dc:title", "lc:*"], owned_if_present: &[] };
if let Ok(out) = merge_xmp(&s, &fresh, rules) {
let d = parse_xmp(&out).unwrap();
prop_assert_eq!(d.metadata.rating, Some(rating));
prop_assert_eq!(d.lc_settings.as_deref(), Some("{\"x\":1}"));
}
}
#[test]
fn user_text_roundtrips(title in "[a-zA-Z0-9 &<>\"'éü€;|/\\\\]{1,40}", kw in proptest::collection::vec("[a-zA-Z0-9 &<>\"'|]{1,12}", 0..5), rating in -1i8..=5) {
let title = title.trim().to_string();
+41 -1
View File
@@ -4,6 +4,11 @@
//! files) but not fsynced; unreadable files are deleted and re-rendered. When the total size
//! exceeds the budget, the least recently used files (by modification time, refreshed on read)
//! are removed down to 80 % of the budget.
//!
//! The cache only ever lists, counts or deletes files of its own name shape
//! (`<2 hex>/<32 hex>.jpg` and the `<32 hex>.tmp…` temp files of [`DiskCache::put`], see
//! [`is_cache_file`]): the directory may be shared with other files (e.g. a library opened on a
//! photo folder that already has a `thumbs/` folder), and those are never touched (issue #98).
use std::path::{Path, PathBuf};
use std::sync::Mutex;
@@ -96,13 +101,25 @@ impl DiskCache {
}
}
/// All cache files: (path, size, modified).
/// All cache files (only names of the cache's own shape, see [`is_cache_file`]):
/// (path, size, modified).
fn scan(&self) -> Vec<(PathBuf, u64, std::time::SystemTime)> {
let mut out = Vec::new();
let Ok(rd) = std::fs::read_dir(&self.dir) else { return out };
for sub in rd.flatten() {
let sub_name = sub.file_name();
let Some(sub_name) = sub_name.to_str() else { continue };
if !is_shard_name(sub_name) {
continue;
}
let Ok(files) = std::fs::read_dir(sub.path()) else { continue };
for f in files.flatten() {
let name = f.file_name();
let Some(name) = name.to_str() else { continue };
if !is_cache_file(sub_name, name) {
continue;
}
// `DirEntry::metadata` does not follow symlinks: a link is never a cache file
if let Ok(m) = f.metadata()
&& m.is_file()
{
@@ -144,6 +161,29 @@ impl DiskCache {
}
}
fn is_lower_hex(s: &str) -> bool {
s.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b))
}
/// A shard directory name: two lowercase hex digits.
fn is_shard_name(s: &str) -> bool {
s.len() == 2 && is_lower_hex(s)
}
/// Whether `name` inside shard directory `shard` is a file this cache wrote: `<32 hex>.jpg`
/// (whose first two digits are the shard) or a `<32 hex>.tmp<alphanumeric>` temp file of
/// [`DiskCache::put`]. Anything else in the cache directory belongs to someone else.
pub fn is_cache_file(shard: &str, name: &str) -> bool {
if !is_shard_name(shard) {
return false;
}
let Some((stem, ext)) = name.split_once('.') else { return false };
if stem.len() != 32 || !is_lower_hex(stem) || !stem.starts_with(shard) {
return false;
}
ext == "jpg" || ext.strip_prefix("tmp").is_some_and(|t| !t.is_empty() && t.bytes().all(|b| b.is_ascii_alphanumeric()))
}
/// Encode a thumbnail as the cache stores it (JPEG, RGB, 4:2:0).
pub fn encode_jpeg(img: &Rgba8) -> Option<Vec<u8>> {
if img.width == 0 || img.height == 0 || img.width > u16::MAX as usize || img.height > u16::MAX as usize {
+38
View File
@@ -154,3 +154,41 @@ fn pool_runs_inline() {
assert_eq!(p.run_inline(10), 1);
assert_eq!(p.try_recv().map(|d| d.result), Some(1));
}
#[test]
fn disk_cache_never_touches_foreign_files() {
// issue #98: a library opened on a folder that already has a `thumbs/` folder
let dir = temp_dir("foreign");
let foreign = [
dir.join("readme.txt"),
dir.join("ab").join("holiday.jpg"),
dir.join("ab").join("0123456789abcdef0123456789abcdef.jpg"), // wrong shard
dir.join("small").join("0123456789abcdef0123456789abcdef.jpg"),
dir.join("01").join("0123456789ABCDEF0123456789ABCDEF.jpg"), // upper case
dir.join("01").join("0123456789abcdef0123456789abcdef.jpeg"),
dir.join("01").join("0123456789abcdef0123456789abcdef.tmp"),
];
for p in &foreign {
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, vec![7u8; 50_000]).unwrap();
}
// a stale temp file of our own is cleaned up
let own_tmp = dir.join("01").join("0123456789abcdef0123456789abcdef.tmpThreadId9");
std::fs::write(&own_tmp, b"partial").unwrap();
let d = DiskCache::new(&dir, 40_000);
assert_eq!(d.size(), 7, "only the cache's own temp file counts");
// fill past the budget (forces prunes), then clear
for i in 0..40u8 {
d.put(hash_bytes(&[i, 2]), &gradient(160, 100, i));
}
assert!(d.size() <= 40_000);
d.clear();
assert_eq!(d.size(), 0);
assert!(!own_tmp.exists());
for p in &foreign {
assert!(p.exists(), "foreign file deleted: {}", p.display());
}
assert!(disk::is_cache_file("01", "0123456789abcdef0123456789abcdef.jpg"));
assert!(!disk::is_cache_file("01", "0123456789abcdef0123456789abcdef.jpg.bak"));
let _ = std::fs::remove_dir_all(&dir);
}
+3 -3
View File
@@ -11,10 +11,10 @@
//! and produces camera → linear Rec.2020 D65 matrices.
//!
//! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw),
//! Canon CR2, Nikon NEF/NRW (uncompressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
//! and X-Trans), Panasonic RW2 (packed 12/14-bit), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon Huffman NEF, Panasonic
//! quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon "lossy after split" NEF,
//! Panasonic quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
//! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no
//! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
#![forbid(unsafe_code)]
+33 -1
View File
@@ -7,6 +7,11 @@
//! - The raw data is IFD#3's single strip: a lossless-JPEG frame (usually 2 or 4 components).
//! - The decoded sample stream fills vertical slices left to right: `cr2_slice = [n, w, last]` means `n` slices of
//! width `w` then one of width `last`; each slice is filled top to bottom, row by row.
//! - Colour filter layout: IFD#3 tag `0xc5e0` (`CR2CFAPattern` in the ExifTool EXIF tag-name docs): 1 = RGGB,
//! 2 = BGGR, 3 = GBRG, 4 = GRBG, anchored at the top-left of the full decoded sensor (masked borders included).
//! It differs by model (e.g. 3 on the 50D/60D/7D/550D/5D Mark II, 1 on the 40D/5D Mark III/6D/5DS R), so it is
//! read per file (no model table); files without the tag fall back to RGGB (issue #85). The parity of the
//! `SensorInfo` borders does *not* predict it (RGGB files come with both even and odd top borders).
//! - `SensorInfo` (maker note `0x00e0`): sensor width/height and the left/top/right/bottom borders of the image
//! area; the masked columns left of it give the black level.
//! - `ColorBalance` (maker note `0x4001`): as-shot `RGGB` levels at a model-dependent offset; we probe the known
@@ -21,6 +26,7 @@ use lightcraft_tiff::{Ifd, Tiff, makernote, tags as t};
const CR2_SLICE: u16 = 0xc640;
const SRAW_TYPE: u16 = 0xc6c5;
const CR2_CFA_PATTERN: u16 = 0xc5e0;
const SENSOR_INFO: u16 = 0x00e0;
const COLOR_BALANCE: u16 = 0x4001;
@@ -28,6 +34,18 @@ fn raw_ifd(tiff: &Tiff) -> Option<&Ifd> {
tiff.ifds.get(3).or_else(|| tiff.ifds.iter().rev().find(|i| i.contains(CR2_SLICE)))
}
/// Bayer layout from the raw IFD's `CR2CFAPattern` (`0xc5e0`) value; `None` for missing/unknown values.
fn cfa_from_tag(v: Option<u64>) -> Option<Cfa> {
let name = match v? {
1 => "RGGB",
2 => "BGGR",
3 => "GBRG",
4 => "GRBG",
_ => return None,
};
Some(Cfa::bayer_static(name))
}
/// As-shot WB multipliers (R, G, B; G = 1) from the ColorBalance array.
fn wb_from_color_balance(v: &[u64]) -> Option<[f32; 3]> {
for off in [63usize, 25, 24, 34, 71, 85, 105, 69, 77] {
@@ -117,7 +135,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
}
}
let wb = mn.as_ref().and_then(|m| m.ifd.u64s(COLOR_BALANCE)).and_then(|v| wb_from_color_balance(&v));
let cfa = Cfa::bayer_static("RGGB");
let cfa = cfa_from_tag(raw.u64(CR2_CFA_PATTERN)).unwrap_or_else(|| Cfa::bayer_static("RGGB"));
let black = if active.x >= 8 {
black_from_columns(&data, width, 2..active.x - 2, active.y..active.y + active.height, active)
} else {
@@ -185,6 +203,7 @@ pub(crate) mod tests {
ifd0.set_child(t::EXIF_IFD, exif);
let mut ifd3 = IfdBuilder::new();
ifd3.set(t::COMPRESSION, Value::Short(vec![6]));
ifd3.set(CR2_CFA_PATTERN, Value::Long(vec![3]));
if let Some(s) = slices {
ifd3.set(CR2_SLICE, Value::Short(s.to_vec()));
}
@@ -206,11 +225,24 @@ pub(crate) mod tests {
assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
assert_eq!(r.data, RawData::U16(img));
assert_eq!(r.orientation, Orientation::Rotate270);
assert_eq!(r.cfa.as_ref().map(Cfa::name).as_deref(), Some("GBRG"), "CR2CFAPattern 3");
assert_eq!(r.metadata.iso, Some(400));
assert!(r.develop(crate::Method::Ahd).is_ok());
}
}
#[test]
fn cfa_pattern_tag() {
let name = |v| cfa_from_tag(v).map(|c| c.name());
assert_eq!(name(Some(1)).as_deref(), Some("RGGB"));
assert_eq!(name(Some(2)).as_deref(), Some("BGGR"));
assert_eq!(name(Some(3)).as_deref(), Some("GBRG"));
assert_eq!(name(Some(4)).as_deref(), Some("GRBG"));
assert_eq!(name(Some(0)), None);
assert_eq!(name(Some(7)), None);
assert_eq!(name(None), None);
}
#[test]
fn wb_probe() {
let mut v = vec![0u64; 120];
+1
View File
@@ -3,6 +3,7 @@
pub mod arw;
pub mod cr2;
pub mod nef;
pub mod nefc;
pub mod orf;
pub mod pef;
pub mod raf;
+53 -28
View File
@@ -1,22 +1,22 @@
//! Nikon NEF / NRW — uncompressed variants only.
//! Nikon NEF / NRW — uncompressed and Huffman-compressed (lossless, lossy) variants.
//!
//! Sources: TIFF 6.0 (the raw image is a standard CFA SubIFD), Laurent Clévy's NEF structure notes (prose: IFD
//! layout, SubIFDs, maker note header) and the ExifTool Nikon tag-name documentation (`0x000c` WB_RBLevels,
//! `0x003d` BlackLevel). Nikon's Huffman-compressed variants (compression 34713: lossless, lossy type 1/2) are
//! **not** implemented: the only descriptions we found are derived from GPL code, which our clean-room rules
//! forbid. The coding is T.81-style (difference category + additional bits, as in Pentax PEF), but unlike PEF the
//! Huffman tables are not stored in the files and our black-box table search has not converged (see ROADMAP.md).
//! They are reported as [`RawError::Unsupported`]; their embedded previews still work.
//! `0x003d` BlackLevel, `0x0096` NEFLinearizationTable). The Huffman-compressed data (compression 34713) is decoded
//! by [`super::nefc`], which documents its clean-room sources; files it can't decode yet ("lossy after split")
//! are reported as [`RawError::Unsupported`] and their embedded previews still work.
use super::white_from_data;
use super::{nefc, white_from_data};
use crate::tiffraw::{Packing, read_image};
use crate::{BlackLevel, Cfa, ColorData, OpcodeLists, RawData, RawError, RawFormat, RawImage, Rect, Result};
use lightcraft_geom::Orientation;
use lightcraft_tiff::image::ImageInfo;
use lightcraft_tiff::tags::{self as t, photometric};
use lightcraft_tiff::{Ifd, Tiff, makernote};
use lightcraft_tiff::{ByteOrder, Ifd, Tiff, makernote};
const WB_RB_LEVELS: u16 = 0x000c;
const BLACK_LEVEL: u16 = 0x003d;
const LINEARIZATION_TABLE: u16 = 0x0096;
fn raw_ifd(tiff: &Tiff) -> Option<&Ifd> {
tiff.all_ifds()
@@ -47,30 +47,15 @@ pub(crate) fn decode(bytes: &[u8]) -> Result<RawImage> {
let ifd0 = &tiff.ifds[0];
let raw = raw_ifd(&tiff).ok_or_else(|| RawError::Unsupported("NEF without a CFA image IFD".into()))?;
let info = raw.image()?;
if info.compression == t::compression::NIKON {
return Err(RawError::Unsupported("Nikon Huffman-compressed NEF (no clean-room description available)".into()));
}
let (w, h) = (info.width as usize, info.height as usize);
let bits = info.bits() as u32;
let row_samples = w * info.samples_per_pixel as usize;
let chunks = info.chunks(bytes.len() as u64);
let rows_in_first = chunks.first().map(|c| c.height as usize).unwrap_or(h).max(1);
let bytes_per_row = chunks.first().map(|c| c.len as usize / rows_in_first).unwrap_or(0);
let packing = if bytes_per_row >= row_samples * 2 {
Packing::Word16
} else if bits == 12 && bytes_per_row * 8 >= row_samples * 12 && bytes_per_row * 8 < row_samples * 13 {
Packing::Msb
} else if info.compression == 1 && bits != 8 && bits != 16 {
return Err(RawError::Unsupported(format!("NEF uncompressed packing ({bytes_per_row} bytes per {row_samples}-sample row)")));
} else {
Packing::Msb
};
let data = read_image(bytes, &info, tiff.order, packing)?;
let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) };
let make = ifd0.string(t::MAKE).unwrap_or_default();
let mn =
tiff.exif().and_then(|e| e.get(t::MAKER_NOTE)).and_then(|e| makernote::parse_makernote(bytes, e.offset, e.count() as u64, tiff.order, &make));
let data =
if info.compression == t::compression::NIKON { compressed(bytes, &info, mn.as_ref())? } else { uncompressed(bytes, &info, tiff.order)? };
let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) };
let black = match mn.as_ref().and_then(|m| m.ifd.f64s(BLACK_LEVEL)).as_deref() {
Some([a, b, c, d]) if [a, b, c, d].iter().all(|v| **v < 16384.0) => {
BlackLevel { repeat_rows: 2, repeat_cols: 2, values: vec![*a as f32, *b as f32, *c as f32, *d as f32], ..Default::default() }
@@ -114,6 +99,46 @@ pub(crate) fn decode(bytes: &[u8]) -> Result<RawImage> {
Ok(img)
}
/// Uncompressed strips: 16-bit words or 12-bit MSB-packed, told apart by the strip size.
fn uncompressed(bytes: &[u8], info: &ImageInfo, order: ByteOrder) -> Result<RawData> {
let (w, h) = (info.width as usize, info.height as usize);
let bits = info.bits() as u32;
let row_samples = w * info.samples_per_pixel as usize;
let chunks = info.chunks(bytes.len() as u64);
let rows_in_first = chunks.first().map(|c| c.height as usize).unwrap_or(h).max(1);
let bytes_per_row = chunks.first().map(|c| c.len as usize / rows_in_first).unwrap_or(0);
let packing = if bytes_per_row >= row_samples * 2 {
Packing::Word16
} else if bits == 12 && bytes_per_row * 8 >= row_samples * 12 && bytes_per_row * 8 < row_samples * 13 {
Packing::Msb
} else if info.compression == 1 && bits != 8 && bits != 16 {
return Err(RawError::Unsupported(format!("NEF uncompressed packing ({bytes_per_row} bytes per {row_samples}-sample row)")));
} else {
Packing::Msb
};
read_image(bytes, info, order, packing)
}
/// Nikon Huffman-compressed strip (see [`super::nefc`]); the decode table is maker note `0x0096`.
fn compressed(bytes: &[u8], info: &ImageInfo, mn: Option<&makernote::MakerNote>) -> Result<RawData> {
// (some Z bodies label uncompressed, row-padded data 34713 without a table: not handled here yet)
let table = mn
.and_then(|m| Some((m.ifd.bytes(LINEARIZATION_TABLE)?, m.order)))
.ok_or_else(|| RawError::Unsupported("Nikon compressed NEF without a linearization table (maker note 0x96)".into()))?;
if info.samples_per_pixel != 1 {
return Err(RawError::Unsupported(format!("Nikon compressed NEF with {} samples per pixel", info.samples_per_pixel)));
}
let bits = info.bits() as u32;
let table = nefc::parse_table(table.0, table.1, bits)?;
// one strip in every sample; several would be contiguous parts of the same bit stream
let chunks = info.chunks(bytes.len() as u64);
let (Some(first), Some(last)) = (chunks.first(), chunks.last()) else { return Err(RawError::Corrupt("NEF: no image data".into())) };
let end = last.offset.saturating_add(last.len).min(bytes.len() as u64);
let src = usize::try_from(first.offset).ok().zip(usize::try_from(end).ok()).and_then(|(a, b)| bytes.get(a..b));
let src = src.ok_or_else(|| RawError::Corrupt("NEF: image data outside the file".into()))?;
Ok(RawData::U16(nefc::decode(src, info.width as usize, info.height as usize, bits, &table)?))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -159,7 +184,7 @@ mod tests {
}
#[test]
fn compressed_is_unsupported() {
fn compressed_without_table_is_unsupported() {
let bytes = nef(34713, 14, vec![vec![0; 64]], 8, 8, 8);
assert!(matches!(crate::decode(&bytes), Err(RawError::Unsupported(_))));
}
+481
View File
@@ -0,0 +1,481 @@
//! Nikon Huffman-compressed NEF image data (TIFF compression 34713): "lossless compressed" and "lossy compressed"
//! (type 1 and type 2) at 12 and 14 bits.
//!
//! **Clean-room.** No raw-decoder source code (dcraw, LibRaw, rawspeed, rawloader/rawler, darktable, RawTherapee,
//! libopenraw, ExifTool's Perl code, …) was read or consulted. What we used:
//!
//! - Prose format descriptions: Laurent Clévy, "Nikon Electronic File (NEF) format" (<http://lclevy.free.fr/nef/>),
//! the maker-note tag table and the "0x96 (linearization table) tag format" table (version bytes, four `u16`
//! predictor seeds at offset 2, curve size at 10, curve at 12, split value at 562 for lossy type 2, "lossy type
//! 2 reads an incomplete table and interpolation is required"); Bill Claff, "NEF Compression"
//! (<https://www.photonstophotos.net/NikonInfo/NEF_Compression.htm>): a lossy encoding curve followed by a
//! non-adaptive (fixed-table) Huffman stage, and the number of encoded values per camera (683, 769, 1025, 2753,
//! 3073, 4097 …), which confirmed our curve interpolation; the libopenraw format notes
//! (<https://libopenraw.freedesktop.org/formats/nef/>, prose only); ExifTool's Nikon tag-name documentation
//! (`0x0093` NEFCompression, `0x0096` NEFLinearizationTable); ITU-T T.81 Annex F/H for the "difference category
//! + additional bits" coding and canonical (BITS/HUFFVAL) Huffman codes.
//! - Our own black-box analysis of CC0 samples from raw.pixls.us. Nikon's Huffman tables are not stored in the
//! files and no prose source lists them, so they were recovered from the data: several cameras (D5100, D300,
//! D700) were shot with the same scene in compressed and uncompressed modes; a beam search assigned code words
//! to difference categories so that the decoded compressed image matched the uncompressed one (likelihood of
//! the decoded values given the reference, minus the bits consumed, plus a description-length penalty per new
//! code word). Each table was then verified on ~45 files from ~35 bodies (D40 … D850, Df, Z 6, Z 50, 1 J1): with
//! the right table the whole strip decodes and ends within the last byte of the data, with every value inside
//! the encoded range. The predictor and difference coding are the same as Pentax PEF (see `pef.rs`).
//!
//! Format, as established:
//! - One strip, MSB-first bit stream, no byte stuffing, continuous across rows. Per pixel a canonical Huffman code
//! gives the category `n`, followed by `n` additional bits (T.81: a leading 0 means a negative difference).
//! - Each pixel is predicted from the same-colour pixel two to the left; the first two pixels of a row from the
//! first two pixels of the previous row of the same parity, starting at the seeds of maker note `0x0096`.
//! - Lossless (`0x0096` version `0x46`): the decoded values are the samples. Lossy (version `0x44`): they index a
//! curve. Type 1 (`0x44 0x10`) stores the full curve; type 2 (`0x44 0x20`, `0x44 0x40`) stores 257 points,
//! one every `2^bits / 256` codes, linearly interpolated.
//! - Values in the maker note are in the maker note's byte order (newer bodies write little-endian notes).
//!
//! Not supported (returned as [`RawError::Unsupported`], the embedded preview is used instead):
//! - "Lossy after split" files (non-zero split row at `0x0096` offset 562): from that row on a different table
//! is used which our analysis has not recovered yet.
//! - Other `0x0096` versions, bit depths other than 12/14.
//! - Code words that never occurred in any sample (the lossy 12-bit 8-bit code word `11111110`, and code words
//! longer than 9 bits in the lossy tables) are rejected as corrupt; they cannot code a difference within the
//! lossy value range anyway.
use super::pef::{Bits, Huffman, diff};
use crate::{MAX_SAMPLES, RawError, Result};
use lightcraft_tiff::ByteOrder;
/// A table code word whose symbol was never observed (see the module docs).
const UNSEEN: u8 = u8::MAX;
/// Canonical Huffman table: number of codes of length 1..=16 and the symbols (difference categories) in code order.
struct Table {
counts: &'static [u8],
symbols: &'static [u8],
}
const LOSSLESS_12: Table = Table { counts: &[0, 1, 4, 2, 3, 1, 2], symbols: &[5, 4, 6, 3, 7, 2, 8, 1, 9, 0, 10, 11, 12] };
const LOSSLESS_14: Table = Table { counts: &[0, 1, 4, 2, 2, 3, 1, 2], symbols: &[7, 6, 8, 5, 9, 4, 10, 3, 11, 12, 2, 0, 1, 13, 14] };
const LOSSY_12: Table = Table { counts: &[0, 1, 5, 1, 1, 1, 1, 1, 1], symbols: &[5, 4, 3, 6, 2, 7, 1, 0, 8, 9, UNSEEN, 10] };
const LOSSY_14: Table = Table { counts: &[0, 1, 4, 3, 1, 1, 1, 1, 1], symbols: &[5, 6, 4, 7, 8, 3, 9, 2, 1, 0, 10, 11, 12] };
impl Table {
/// `(left-aligned 12-bit code, length)` per symbol, for [`Huffman::new`].
fn codes(&self) -> Vec<(u16, u8)> {
let mut out = vec![(0u16, 0u8); 17];
let mut code = 0u32;
let mut symbols = self.symbols.iter();
for (i, &n) in self.counts.iter().enumerate() {
let len = i as u32 + 1;
for _ in 0..n {
if let Some(&s) = symbols.next()
&& let Some(slot) = out.get_mut(s as usize)
{
*slot = ((code << (12 - len)) as u16, len as u8);
}
code += 1;
}
code <<= 1;
}
out
}
}
/// How the decoded values map to samples.
#[derive(Clone, Debug, PartialEq)]
pub(crate) enum Encoding {
/// The decoded values are the samples (`0..2^bits`).
Lossless,
/// The decoded values index this curve.
Lossy(Vec<u16>),
}
/// Maker note `0x0096` (NEF linearization table), parsed.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct DecodeTable {
pub encoding: Encoding,
/// Predictor seeds `[row parity][column]`.
pub seeds: [[i32; 2]; 2],
}
/// Parse maker note `0x0096` for a `bits`-bit image. `order` is the maker note's byte order.
pub(crate) fn parse_table(t: &[u8], order: ByteOrder, bits: u32) -> Result<DecodeTable> {
let u16_at = |o: usize| t.get(o..o + 2).and_then(|s| <[u8; 2]>::try_from(s).ok()).map(|b| order.u16(b));
let short = || RawError::Corrupt(format!("NEF: linearization table too short ({} bytes)", t.len()));
if bits != 12 && bits != 14 {
return Err(RawError::Unsupported(format!("Nikon compressed NEF with {bits}-bit samples")));
}
let (v0, v1) = (t.first().copied().ok_or_else(short)?, t.get(1).copied().ok_or_else(short)?);
let mut seeds = [[0i32; 2]; 2];
for (i, s) in seeds.iter_mut().flatten().enumerate() {
*s = u16_at(2 + 2 * i).ok_or_else(short)? as i32;
}
let encoding = match (v0, v1) {
(0x46, _) => Encoding::Lossless,
(0x44, 0x10 | 0x20 | 0x40) => {
let n = u16_at(10).ok_or_else(short)? as usize;
let points: Vec<u16> = (0..n).map(|i| u16_at(12 + 2 * i)).collect::<Option<_>>().ok_or_else(short)?;
if n < 2 {
return Err(RawError::Corrupt(format!("NEF: linearization curve of {n} points")));
}
if v1 == 0x10 {
Encoding::Lossy(points)
} else {
let split = u16_at(562).unwrap_or(0);
if split != 0 {
return Err(RawError::Unsupported(format!(
"Nikon \"lossy after split\" compressed NEF (split at row {split}) is not decoded yet"
)));
}
// type 2: `n` points, one every 2^bits / (n - 1) codes
let range = 1usize << bits;
if n - 1 > range || !range.is_multiple_of(n - 1) {
return Err(RawError::Unsupported(format!("NEF: {n}-point lossy curve for {bits}-bit data")));
}
let step = range / (n - 1);
let mut curve = Vec::with_capacity(range + 1);
for pair in points.windows(2) {
let &[a, b] = pair else { continue };
let (a, b) = (a as u32, b as u32);
for k in 0..step as u32 {
let s = step as u32;
curve.push(((a * (s - k) + b * k) / s) as u16);
}
}
curve.extend(points.last().copied());
Encoding::Lossy(curve)
}
}
_ => return Err(RawError::Unsupported(format!("Nikon compressed NEF version {v0:#04x} {v1:#04x}"))),
};
Ok(DecodeTable { encoding, seeds })
}
/// Decode a `w × h` Nikon Huffman-compressed strip `src` with `bits`-bit samples.
pub(crate) fn decode(src: &[u8], w: usize, h: usize, bits: u32, table: &DecodeTable) -> Result<Vec<u16>> {
if bits != 12 && bits != 14 {
return Err(RawError::Unsupported(format!("Nikon compressed NEF with {bits}-bit samples")));
}
let n = w.checked_mul(h).filter(|&n| n > 0 && n <= MAX_SAMPLES).ok_or(RawError::Limit("NEF image size"))?;
// every code word is at least 2 bits long: a strip that can't hold the image is truncated (also bounds the
// allocation by the file size)
if n / 4 > src.len() {
return Err(RawError::Corrupt(format!("NEF: {} bytes of compressed data for {w}x{h} pixels", src.len())));
}
let huff = Huffman::new(
&match (&table.encoding, bits) {
(Encoding::Lossless, 12) => LOSSLESS_12,
(Encoding::Lossless, _) => LOSSLESS_14,
(Encoding::Lossy(_), 12) => LOSSY_12,
(Encoding::Lossy(_), _) => LOSSY_14,
}
.codes(),
)?;
let (lut, max): (&[u16], i32) = match &table.encoding {
Encoding::Lossless => (&[], (1i32 << bits) - 1),
Encoding::Lossy(curve) => (curve, curve.len() as i32 - 1),
};
let mut out = vec![0u16; n];
let mut stream = Bits::new(src);
let mut vpred = table.seeds;
let mut clipped = 0usize;
for (y, row) in out.chunks_exact_mut(w).enumerate() {
let seeds = vpred.get_mut(y & 1).ok_or(RawError::Limit("NEF row"))?;
let mut hpred = [0i32; 2];
for (x, o) in row.iter_mut().enumerate() {
let d = diff(&mut stream, &huff).ok_or_else(|| RawError::Corrupt(format!("NEF: invalid Huffman code at row {y}")))?;
let p = if x < 2 {
let s = seeds.get_mut(x).ok_or(RawError::Limit("NEF column"))?;
*s += d;
hpred[x & 1] = *s;
*s
} else {
hpred[x & 1] += d;
hpred[x & 1]
};
let v = p.clamp(0, max);
clipped += (v != p) as usize;
*o = if lut.is_empty() { v as u16 } else { lut.get(v as usize).copied().unwrap_or(0) };
}
// the camera pads the strip; reading more than a few bytes past its end means truncated or corrupt data
if stream.consumed_bits() > src.len() * 8 + 64 {
return Err(RawError::Corrupt(format!("NEF: compressed data ends at row {y} of {h}")));
}
}
// a wrong table / corrupt stream drifts out of range quickly; real files stay inside (a handful of edge pixels)
if clipped > n / 1000 + 16 {
return Err(RawError::Corrupt(format!("NEF: {clipped} decoded values out of range")));
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
/// MSB-first bit writer for the test encoder.
#[derive(Default)]
struct Writer {
out: Vec<u8>,
acc: u64,
n: u32,
}
impl Writer {
fn put(&mut self, v: u32, len: u32) {
for i in (0..len).rev() {
self.acc = (self.acc << 1) | ((v >> i) & 1) as u64;
self.n += 1;
if self.n == 8 {
self.out.push(self.acc as u8);
self.acc = 0;
self.n = 0;
}
}
}
fn finish(mut self) -> Vec<u8> {
if self.n > 0 {
self.out.push((self.acc << (8 - self.n)) as u8);
}
self.out.extend_from_slice(&[0; 4]);
self.out
}
}
fn category(d: i32) -> u32 {
32 - d.unsigned_abs().leading_zeros()
}
/// Encode `values` (decoded domain: samples or curve indices) the way the camera does.
fn encode(values: &[i32], w: usize, table: &Table, seeds: [[i32; 2]; 2]) -> Vec<u8> {
let codes = table.codes();
let mut wr = Writer::default();
let mut vpred = seeds;
for (y, row) in values.chunks(w).enumerate() {
for (x, &v) in row.iter().enumerate() {
let pred = if x < 2 { vpred[y & 1][x] } else { row[x - 2] };
if x < 2 {
vpred[y & 1][x] = v;
}
let d = v - pred;
let k = category(d);
let (code, len) = codes[k as usize];
assert!(len > 0, "no code for category {k}");
wr.put(code as u32 >> (12 - len), len as u32);
let extra = if d < 0 { d + (1 << k) - 1 } else { d };
wr.put(extra as u32, k);
}
}
wr.finish()
}
/// Deterministic test image: smooth gradient + texture + hard edges, within `0..=max`.
fn image(w: usize, h: usize, max: i32, seed: u32) -> Vec<i32> {
let mut s = seed;
(0..w * h)
.map(|i| {
let (x, y) = ((i % w) as i32, (i / w) as i32);
s = s.wrapping_mul(1_664_525).wrapping_add(1_013_904_223);
let noise = (s >> 24) as i32 % 9 - 4;
let base = (x * 37 + y * 11) % (max / 2) + if (x / 7 + y / 5) % 3 == 0 { max / 3 } else { 0 };
(base + noise).clamp(0, max)
})
.collect()
}
fn lossless(bits: u32) -> DecodeTable {
let s = 1 << (bits - 3);
DecodeTable { encoding: Encoding::Lossless, seeds: [[s, s], [s, s]] }
}
#[test]
fn tables_are_complete_prefix_codes() {
for t in [LOSSLESS_12, LOSSLESS_14, LOSSY_12, LOSSY_14] {
assert_eq!(t.counts.iter().map(|&c| c as usize).sum::<usize>(), t.symbols.len());
let kraft: f64 = t.counts.iter().enumerate().map(|(i, &c)| c as f64 / (1u64 << (i + 1)) as f64).sum();
assert!(kraft <= 1.0);
assert!(Huffman::new(&t.codes()).is_ok());
}
assert_eq!(LOSSLESS_14.codes()[7], (0b00 << 10, 2));
assert_eq!(LOSSLESS_14.codes()[0], (0b111110 << 6, 6));
assert_eq!(LOSSY_12.codes()[10], (0b111111110 << 3, 9));
}
#[test]
fn lossless_roundtrip_12_and_14() {
for (bits, table) in [(12, LOSSLESS_12), (14, LOSSLESS_14)] {
let (w, h) = (64, 9);
let max = (1 << bits) - 1;
let mut img = image(w, h, max, bits);
// extreme jumps exercise the longest categories
img[5] = 0;
img[7] = max;
img[w + 9] = max;
let t = lossless(bits);
let src = encode(&img, w, &table, t.seeds);
let out = decode(&src, w, h, bits, &t).unwrap();
assert_eq!(out, img.iter().map(|&v| v as u16).collect::<Vec<_>>(), "{bits}-bit");
}
}
fn lossy_table(bits: u32, order: ByteOrder) -> Vec<u8> {
// version 0x44 0x20, seeds, 257 points of a concave curve, padding up to the split value (0)
let mut t = vec![0x44, 0x20];
let put = |t: &mut Vec<u8>, v: u16| t.extend_from_slice(&if order == ByteOrder::Big { v.to_be_bytes() } else { v.to_le_bytes() });
for _ in 0..4 {
put(&mut t, 300);
}
put(&mut t, 257);
let max = ((1u32 << bits) - 1) as f64;
for i in 0..257u32 {
put(&mut t, (max * (i as f64 / 64.0).min(1.0).sqrt()) as u16);
}
t.resize(624, 0);
t
}
#[test]
fn lossy_type2_curve_and_roundtrip() {
for (bits, table) in [(12u32, LOSSY_12), (14, LOSSY_14)] {
for order in [ByteOrder::Big, ByteOrder::Little] {
let t = parse_table(&lossy_table(bits, order), order, bits).unwrap();
assert_eq!(t.seeds, [[300, 300], [300, 300]]);
let Encoding::Lossy(curve) = &t.encoding else { panic!() };
let step = (1usize << bits) / 256;
assert_eq!(curve.len(), (1 << bits) + 1);
assert_eq!(curve[64 * step], (1 << bits) - 1);
assert!(curve.windows(2).all(|p| p[0] <= p[1]));
// the encoded range is the curve's rising part
let (w, h) = (48, 7);
let img = image(w, h, 64 * step as i32 - 1, 7);
let src = encode(&img, w, &table, t.seeds);
let out = decode(&src, w, h, bits, &t).unwrap();
assert_eq!(out, img.iter().map(|&v| curve[v as usize]).collect::<Vec<_>>());
}
}
}
#[test]
fn lossy_type1_full_curve() {
let mut t = vec![0x44, 0x10, 0, 10, 0, 10, 0, 10, 0, 10, 0, 5];
for v in [0u16, 100, 400, 900, 1600] {
t.extend_from_slice(&v.to_be_bytes());
}
let parsed = parse_table(&t, ByteOrder::Big, 12).unwrap();
assert_eq!(parsed.encoding, Encoding::Lossy(vec![0, 100, 400, 900, 1600]));
let img = vec![0, 4, 1, 3, 2, 2, 4, 0];
let src = encode(&img, 4, &LOSSY_12, parsed.seeds);
assert_eq!(decode(&src, 4, 2, 12, &parsed).unwrap(), vec![0, 1600, 100, 900, 400, 400, 1600, 0]);
}
#[test]
fn rejects_unsupported_and_malformed_tables() {
let unsupported = |r: Result<DecodeTable>| matches!(r, Err(RawError::Unsupported(_)));
let corrupt = |r: Result<DecodeTable>| matches!(r, Err(RawError::Corrupt(_)));
let mut split = lossy_table(12, ByteOrder::Big);
split[562..564].copy_from_slice(&345u16.to_be_bytes());
assert!(unsupported(parse_table(&split, ByteOrder::Big, 12)));
assert!(unsupported(parse_table(&[0x49, 0x30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], ByteOrder::Big, 12)));
assert!(unsupported(parse_table(&lossy_table(12, ByteOrder::Big), ByteOrder::Big, 16)));
assert!(corrupt(parse_table(&[], ByteOrder::Big, 12)));
assert!(corrupt(parse_table(&[0x46, 0x30, 8], ByteOrder::Big, 14)));
assert!(corrupt(parse_table(&lossy_table(12, ByteOrder::Big)[..100], ByteOrder::Big, 12)));
// 300 points don't divide the 12-bit range
let mut odd = lossy_table(12, ByteOrder::Big);
odd[10..12].copy_from_slice(&300u16.to_be_bytes());
assert!(parse_table(&odd, ByteOrder::Big, 12).is_err());
assert!(parse_table(&[0x44, 0x20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], ByteOrder::Big, 12).is_err());
}
/// A minimal NEF: CFA SubIFD with one compressed strip, Exif maker note (`Nikon\0` v2 header + embedded TIFF in
/// `mn_order`) holding the linearization table.
fn nef_file(strip: Vec<u8>, w: u32, h: u32, bits: u16, table: Vec<u8>, mn_order: ByteOrder) -> Vec<u8> {
use lightcraft_tiff::tags::{self as t, photometric};
use lightcraft_tiff::{IfdBuilder, ImageData, TiffWriter, Value};
let mut mn = IfdBuilder::new();
mn.set(0x0096, Value::Undefined(table));
let mut note = b"Nikon\0\x02\x10\0\0".to_vec();
note.extend(TiffWriter::new(mn_order, false).write(&[mn]).unwrap());
let mut exif = IfdBuilder::new();
exif.set(t::MAKER_NOTE, Value::Undefined(note));
let mut raw = IfdBuilder::new();
raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0]));
raw.set(t::IMAGE_WIDTH, Value::Long(vec![w]));
raw.set(t::IMAGE_LENGTH, Value::Long(vec![h]));
raw.set(t::BITS_PER_SAMPLE, Value::Short(vec![bits]));
raw.set(t::COMPRESSION, Value::Short(vec![t::compression::NIKON]));
raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::CFA]));
raw.set(t::CFA_REPEAT_PATTERN_DIM, Value::Short(vec![2, 2]));
raw.set(t::CFA_PATTERN_EP, Value::Byte(vec![0, 1, 1, 2]));
raw.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![strip] });
let mut ifd0 = IfdBuilder::new();
ifd0.set(t::MAKE, Value::Ascii("NIKON CORPORATION".into()));
ifd0.set(t::MODEL, Value::Ascii("NIKON TEST".into()));
ifd0.set_child(t::EXIF_IFD, exif);
ifd0.add_sub_ifd(raw);
TiffWriter::new(ByteOrder::Big, false).write(&[ifd0]).unwrap()
}
#[test]
fn decodes_through_the_nef_container() {
// lossless 14-bit, big-endian maker note
let (w, h) = (40usize, 6usize);
let img = image(w, h, 16383, 11);
let mut table = vec![0x46, 0x30];
for _ in 0..4 {
table.extend_from_slice(&2048u16.to_be_bytes());
}
table.resize(46, 0);
let src = encode(&img, w, &LOSSLESS_14, [[2048; 2]; 2]);
let bytes = nef_file(src, w as u32, h as u32, 14, table, ByteOrder::Big);
let r = crate::decode(&bytes).unwrap();
assert_eq!(r.data, crate::RawData::U16(img.iter().map(|&v| v as u16).collect()));
assert_eq!(r.cfa.as_ref().unwrap().name(), "RGGB");
assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
// lossy 12-bit type 2, little-endian maker note (newer bodies)
let table = lossy_table(12, ByteOrder::Little);
let t = parse_table(&table, ByteOrder::Little, 12).unwrap();
let Encoding::Lossy(curve) = &t.encoding else { panic!() };
let img = image(w, h, 1023, 5);
let src = encode(&img, w, &LOSSY_12, t.seeds);
let r = crate::decode(&nef_file(src.clone(), w as u32, h as u32, 12, table.clone(), ByteOrder::Little)).unwrap();
assert_eq!(r.data, crate::RawData::U16(img.iter().map(|&v| curve[v as usize]).collect()));
// the split variant falls back to the preview
let mut split = table;
split[562..564].copy_from_slice(&3u16.to_le_bytes());
let e = crate::decode(&nef_file(src, w as u32, h as u32, 12, split, ByteOrder::Little));
assert!(matches!(e, Err(RawError::Unsupported(_))), "{e:?}");
}
#[test]
fn truncated_and_corrupted_streams_error_never_panic() {
let (w, h) = (64, 16);
let t = lossless(14);
let img = image(w, h, 16383, 3);
let src = encode(&img, w, &LOSSLESS_14, t.seeds);
// truncation: too short for the image, or ends early
for cut in [0, 1, 10, src.len() / 2, src.len() - 40] {
assert!(decode(&src[..cut], w, h, 14, &t).is_err(), "cut {cut}");
}
// garbage and bit flips: an error or (rarely) some image, never a panic
let mut s = 12345u32;
for i in 0..200 {
let mut bad = src.clone();
for _ in 0..1 + i % 8 {
s = s.wrapping_mul(1_103_515_245).wrapping_add(12345);
let at = (s >> 8) as usize % bad.len();
bad[at] ^= 1 << (s % 8);
}
let _ = decode(&bad, w, h, 14, &t);
let noise: Vec<u8> = (0..src.len()).map(|k| ((k as u32).wrapping_mul(2_654_435_761) >> (i % 24)) as u8).collect();
let _ = decode(&noise, w, h, 14, &t);
let _ = decode(&noise, w, h, 12, &lossless(12));
}
// all-ones never forms a valid lossy 12-bit code
assert!(decode(&[0xff; 64], 8, 8, 12, &parse_table(&lossy_table(12, ByteOrder::Big), ByteOrder::Big, 12).unwrap()).is_err());
// absurd dimensions are limited before allocating
assert!(decode(&src, usize::MAX, 2, 14, &t).is_err());
assert!(decode(&src, 1 << 20, 1 << 20, 14, &t).is_err());
assert!(decode(&src, 0, 2, 14, &t).is_err());
}
}
+4
View File
@@ -99,6 +99,10 @@ impl<'a> Bits<'a> {
self.skip(k);
v
}
/// Bits handed out so far (may exceed the source length: zeros are read past its end).
pub fn consumed_bits(&self) -> usize {
self.pos * 8 - self.n as usize
}
pub fn overrun(&self) -> bool {
self.pos > self.src.len() + 8
}
+101 -2
View File
@@ -23,8 +23,6 @@ const KNOWN_UNSUPPORTED: &[&str] = &[
"rw2-panasonic-gx80", // "
"rw2-panasonic-g9-b", // "
"orf-olympus-em", // Olympus compressed ORF
"nef-nikon-d5100-lossless", // Nikon Huffman NEF
"nef-nikon-d7000-lossy", // "
"sraw", // Canon sRAW / mRAW
];
@@ -94,3 +92,104 @@ fn corpus_raw_decodes() {
}
eprintln!("corpus/raw: {ok} decoded, {unsupported} known-unsupported (preview only)");
}
/// Green-channel means of 32×32 blocks.
fn block_means(img: &lightcraft_raw::RawImage) -> Vec<f64> {
let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
let (w, h, b) = (img.width, img.height, 32);
let mut out = Vec::new();
for by in 0..h / b {
for bx in 0..w / b {
let (mut s, mut n) = (0f64, 0f64);
for y in by * b..by * b + b {
for x in (bx * b..bx * b + b).step_by(2) {
s += d[y * w + x + (y & 1)] as f64;
n += 1.0;
}
}
out.push(s / n);
}
}
out
}
fn correlation(a: &[f64], b: &[f64]) -> f64 {
let n = a.len() as f64;
let (ma, mb) = (a.iter().sum::<f64>() / n, b.iter().sum::<f64>() / n);
let cov: f64 = a.iter().zip(b).map(|(x, y)| (x - ma) * (y - mb)).sum();
let va: f64 = a.iter().map(|x| (x - ma).powi(2)).sum();
let vb: f64 = b.iter().map(|y| (y - mb).powi(2)).sum();
cov / (va * vb).sqrt()
}
/// Which diagonal of each 2×2 cell (anchored at raw pixel (0, 0)) holds the green sites: the two greens of a Bayer
/// cell see nearly the same light, so their mean absolute difference is far smaller than across the other diagonal.
/// Returns `true` when green sits at (0, 0)/(1, 1) (GBRG/GRBG), `false` for (1, 0)/(0, 1) (RGGB/BGGR).
fn green_on_main_diagonal(img: &lightcraft_raw::RawImage) -> bool {
let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
let (w, a) = (img.width, img.active_area);
let (mut main, mut anti) = (0f64, 0f64);
for y in ((a.y + 2) & !1..a.y + a.height - 2).step_by(2) {
for x in ((a.x + 2) & !1..a.x + a.width - 2).step_by(2) {
let v = |dx: usize, dy: usize| d[(y + dy) * w + x + dx] as f64;
main += (v(0, 0) - v(1, 1)).abs();
anti += (v(1, 0) - v(0, 1)).abs();
}
}
main < anti
}
/// Canon CR2 colour-filter layouts differ by model (issue #85); the decoder reads them from the `CR2CFAPattern` tag.
/// The expected layouts were checked visually (natural colours vs. the embedded preview) and agree with the
/// green-diagonal statistic of the mosaic itself.
#[test]
fn corpus_cr2_cfa_patterns() {
let dir = corpus_root().join("raw");
let cases = [
("cr2-canon-40d.cr2", "RGGB"),
("cr2-canon-550d.cr2", "GBRG"),
("cr2-canon-5d2.cr2", "GBRG"),
("cr2-canon-5d3.cr2", "RGGB"),
("cr2-canon-5dsr.cr2", "RGGB"),
("cr2-canon-6d.cr2", "RGGB"),
("cr2-canon-7d.cr2", "GBRG"),
("cr2-canon-80d.cr2", "RGGB"),
];
let mut seen = 0;
for (name, want) in cases {
let Ok(bytes) = std::fs::read(dir.join(name)) else {
eprintln!("skip: {name} absent");
continue;
};
let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
let got = img.cfa.as_ref().map(|c| c.name()).unwrap_or_default();
assert_eq!(got, want, "{name}: CFA layout");
assert_eq!(green_on_main_diagonal(&img), want.starts_with('G'), "{name}: mosaic statistics disagree with {want}");
seen += 1;
}
eprintln!("CR2 CFA layouts checked on {seen} files");
}
/// raw.pixls.us has the same D5100 scene as 14-bit lossless compressed and uncompressed NEF: the Huffman decode must
/// match the uncompressed image (up to the small differences between two exposures).
#[test]
fn corpus_nef_compressed_matches_uncompressed() {
let dir = corpus_root().join("raw");
let (Ok(a), Ok(b)) = (std::fs::read(dir.join("nef-nikon-d5100-lossless.nef")), std::fs::read(dir.join("nef-nikon-d5100-uncompressed.nef")))
else {
eprintln!("skip: D5100 NEF pair absent");
return;
};
let (a, b) = (decode(&a).unwrap(), decode(&b).unwrap());
assert_eq!((a.width, a.height, a.bits), (b.width, b.height, b.bits));
let r = correlation(&block_means(&a), &block_means(&b));
eprintln!("D5100 lossless vs uncompressed: block correlation {r:.4}");
assert!(r > 0.98, "correlation {r}");
// the lossy 12-bit D7000 file decodes into the curve's range and isn't flat
if let Ok(c) = std::fs::read(dir.join("nef-nikon-d7000-lossy12.nef")) {
let c = decode(&c).unwrap();
let m = block_means(&c);
let (lo, hi) = m.iter().fold((f64::MAX, 0f64), |(l, h), &v| (l.min(v), h.max(v)));
assert!(hi <= 4095.0 && hi - lo > 500.0, "D7000 block means {lo}..{hi}");
}
}
+10 -6
View File
@@ -78,10 +78,11 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {
"selection": app.session.selection.ids.iter().map(|p| p.0).collect::<Vec<_>>(),
"activeMask": app.session.active_mask,
"widgetCount": app.widgets.len(),
"perf": {"frameMs": app.perf.frame_ms, "logicMs": app.perf.logic_ms, "updateMs": app.perf.update_ms, "maxUpdateMs": app.perf.max_update_ms, "fps": app.perf.fps, "lastRenderMs": app.renderer.last_main_ms, "renderQueue": app.renderer.queued(), "rendersInFlight": app.renderer.in_flight(), "pendingSlots": app.renderer.pending_slots(), "mergeRunning": app.merge.busy(), "lastMerge": app.merge.last_result, "rendersDone": app.renderer.completed, "thumbTextures": app.renderer.thumb_textures(), "variantTextures": app.renderer.variant_textures(), "gpu": (lightcraft_engine::gpu::ready() && lightcraft_engine::gpu::available()).then(lightcraft_engine::gpu::adapter_name).flatten()},
"perf": {"frameMs": app.perf.frame_ms, "logicMs": app.perf.logic_ms, "updateMs": app.perf.update_ms, "maxUpdateMs": app.perf.max_update_ms, "fps": app.perf.fps, "lastRenderMs": app.renderer.last_main_ms, "renderQueue": app.renderer.queued(), "rendersInFlight": app.renderer.in_flight(), "pendingSlots": app.renderer.pending_slots(), "mergeRunning": app.merge.busy(), "lastMerge": app.merge.last_result, "rendersDone": app.renderer.completed, "thumbTextures": app.renderer.thumb_textures(), "variantTextures": app.renderer.variant_textures(), "gpu": (lightcraft_engine::gpu::ready() && lightcraft_engine::gpu::available()).then(lightcraft_engine::gpu::adapter_name).flatten(), "gpuReason": lightcraft_engine::gpu::unavailable_reason(), "gpuFallback": lightcraft_engine::gpu::last_fallback()},
"loupe": app.loupe_shown.map(|(p, src)| json!({"photo": p.0, "source": src, "pending": app.renderer.is_pending(crate::render::Slot::Main)})),
"hoverPreview": app.hover_preview.as_ref().map(|h| h.label.clone()),
"status": app.ui.status,
"unsaved": app.session.unsaved().map(|(n, e)| json!({"ops": n, "error": e})),
"scan": app.scan.as_ref().map(crate::import::ScanTask::status),
"export": {"running": app.export.as_ref().map(crate::export_task::ExportTask::status), "last": app.last_export_result},
"memory": memory(app),
@@ -289,12 +290,19 @@ pub fn handle(app: &mut LightcraftApp, ctx: &egui::Context, req: &ControlRequest
ok(Value::Null)
}
"ui.screenshot" => {
// never over a photo's original
if let Some(Err(e)) = s("path").map(|path| app.session.check_write_target(path)) {
return err(e);
}
ctx.request_repaint();
Outcome::Screenshot { path: s("path").map(str::to_string), headless: p.get("headless").and_then(Value::as_bool).unwrap_or(false) }
}
"ui.render" => {
let id = p.get("id").and_then(Value::as_u64).map(lightcraft_catalog::PhotoId).or(app.session.active());
let Some(id) = id else { return err("no photo") };
if let Some(Err(e)) = s("path").map(|path| app.session.check_write_target(path)) {
return err(e);
}
let size = p.get("size").and_then(Value::as_u64).unwrap_or(1600) as usize;
match app.session.render_now(id, size, size) {
Ok(r) => match (s("path"), app.services.png.as_ref()) {
@@ -355,11 +363,7 @@ pub fn export_active(app: &mut LightcraftApp, p: &Value) -> Result<Value, String
let to = Destination { dir: dir.clone(), exact: p.get("path").and_then(Value::as_str).map(str::to_string) };
let background = p.get("background").and_then(Value::as_bool).unwrap_or(false) && app.services.write_shared.is_some();
let out = if background {
let items = ids
.iter()
.enumerate()
.map(|(i, id)| lightcraft_engine::export::prepare_export(&mut app.session, *id, &opts, i + 1))
.collect::<Result<Vec<_>, _>>()?;
let items = lightcraft_engine::export::prepare_batch(&mut app.session, &ids, &opts)?;
crate::export_task::start(app, items, opts, to)?
} else {
let w = app.services.write.as_mut().ok_or("no writer")?;
+154
View File
@@ -0,0 +1,154 @@
//! Interface translations. Command ids, document text and file names remain stable.
//! Untranslated labels fall back to English so coverage can grow incrementally.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Language {
#[default]
En,
Ja,
}
impl Language {
pub const ALL: [Self; 2] = [Self::En, Self::Ja];
pub fn name(self) -> &'static str {
match self {
Self::En => "English",
Self::Ja => "日本語",
}
}
pub fn parse(code: &str) -> Option<Self> {
match code {
"en" => Some(Self::En),
"ja" => Some(Self::Ja),
_ => None,
}
}
pub fn tr(self, text: &str) -> &str {
if self == Self::Ja
&& let Some((_, japanese)) = JAPANESE.iter().find(|(english, _)| *english == text)
{
return japanese;
}
text
}
}
const JAPANESE: &[(&str, &str)] = &[
("Photo", "写真"),
("Import Photos…", "写真を読み込み…"),
("Detail", "詳細"),
("Photo Grid", "写真グリッド"),
("Square Grid", "正方形グリッド"),
("Object", "オブジェクト"),
("Effect", "効果"),
("Settings…", "環境設定…"),
("Image", "画像"),
("Layer", "レイヤー"),
("Type", "書式"),
("Select", "選択"),
("Filter", "フィルター"),
("Window", "ウィンドウ"),
("Language", "表示言語"),
("Save As…", "別名で保存…"),
("Exit", "終了"),
("New…", "新規…"),
("New", "新規"),
("Horizontal", "横書き"),
("Vertical", "縦書き"),
("Orientation", "組み方向"),
("Layers", "レイヤー"),
("History", "履歴"),
("Properties", "プロパティ"),
("Color", "カラー"),
("Brush Settings", "ブラシ設定"),
("Tools", "ツール"),
("Options", "オプション"),
("Zoom In", "ズームイン"),
("Zoom Out", "ズームアウト"),
("Fit on Screen", "画面に合わせる"),
("Copy", "コピー"),
("Cut", "切り取り"),
("Paste", "貼り付け"),
("Select All", "すべて選択"),
("Deselect", "選択を解除"),
("Export", "書き出し"),
("Export As…", "形式を指定して書き出し…"),
("Search…", "検索…"),
("Theme", "テーマ"),
("Menu", "メニュー"),
("File", "ファイル"),
("Edit", "編集"),
("Pages", "ページ"),
("View", "表示"),
("Help", "ヘルプ"),
("Preferences", "環境設定"),
("Preferences…", "環境設定…"),
("Interface language", "表示言語"),
("Open…", "開く…"),
("New blank PDF", "空白の PDF を作成"),
("Create PDF from file…", "ファイルから PDF を作成…"),
("Create PDF from images…", "画像から PDF を作成…"),
("Create PDF from clipboard", "クリップボードから PDF を作成"),
("Combine files…", "ファイルを結合…"),
("Save", "保存"),
("Save as…", "別名で保存…"),
("Close file", "ファイルを閉じる"),
("Close all", "すべて閉じる"),
("Revert", "保存済みの状態に戻す"),
("Print…", "印刷…"),
("Document properties…", "文書のプロパティ…"),
("Undo", "取り消し"),
("Redo", "やり直し"),
("Find…", "検索…"),
("Advanced search…", "高度な検索…"),
("Copy pages", "ページをコピー"),
("Cut pages", "ページを切り取り"),
("Paste pages", "ページを貼り付け"),
("Fit visible", "表示範囲に合わせる"),
("Marquee zoom", "範囲指定ズーム"),
("Take a snapshot", "スナップショットを作成"),
("Full screen mode", "全画面表示"),
("Read mode", "閲覧モード"),
("Switch light / dark theme", "明るい/暗いテーマを切り替え"),
("Comments panel", "コメントパネル"),
("Form fields panel", "フォームフィールドパネル"),
("Clear form", "フォームをクリア"),
("Find tools and commands…", "ツールとコマンドを検索…"),
("Zoom", "ズーム"),
("Actual size", "実際のサイズ"),
("Zoom to page level", "ページ全体を表示"),
("Fit to width", "幅に合わせる"),
("Display theme", "表示テーマ"),
("Side panels", "サイドパネル"),
("OK", "OK"),
];
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn language_setting_persists() {
let state = crate::state::UiState { language: Language::Ja, ..Default::default() };
let saved = serde_json::to_string(&state).unwrap();
let restored: crate::state::UiState = serde_json::from_str(&saved).unwrap();
assert_eq!(restored.language, Language::Ja);
assert_eq!(crate::state::UiState::default().language, Language::En);
}
#[test]
fn translations_are_unique_and_preserve_unknown_text() {
for (i, (en, ja)) in JAPANESE.iter().enumerate() {
assert!(!ja.is_empty());
assert!(JAPANESE.iter().take(i).all(|(other, _)| en != other));
assert_eq!(Language::En.tr(en), *en);
}
assert_eq!(Language::Ja.tr("File"), "ファイル");
assert_eq!(Language::Ja.tr("日本語の文書.pdf"), "日本語の文書.pdf");
assert_eq!(Language::parse("xx"), None);
}
}
+24
View File
@@ -9,6 +9,7 @@
pub mod control;
pub mod export_task;
pub mod headless;
pub mod i18n;
pub mod icons;
pub mod import;
pub mod links;
@@ -33,6 +34,8 @@ mod tests_masking;
mod tests_panels;
#[cfg(test)]
mod tests_scroll;
#[cfg(test)]
mod tests_unsaved;
use std::sync::mpsc::{Receiver, Sender};
@@ -281,6 +284,26 @@ impl LightcraftApp {
}
}
/// Announce when saving the library starts failing (changes then live only in memory and are
/// retried) and when it works again; the top bar's cloud icon shows the state meanwhile.
fn save_status(&mut self, ctx: &egui::Context) {
let unsaved = self.session.unsaved().map(|(n, e)| (n, e.to_string()));
match (unsaved, self.ui.unsaved_seen) {
(Some((n, e)), false) => {
self.ui.unsaved_seen = true;
let t = ctx.input(|i| i.time);
let what = if n == 1 { "1 change".to_string() } else { format!("{n} changes") };
self.ui.toast = Some((format!("{what} saved in memory but not written to disk: {e} — LightCraft will retry"), t + 6.0));
}
(None, true) => {
self.ui.unsaved_seen = false;
self.toast(ctx, "Library saved");
}
(Some(_), true) => ctx.request_repaint_after(std::time::Duration::from_secs(2)), // retry
(None, false) => {}
}
}
pub fn toast(&mut self, ctx: &egui::Context, text: impl Into<String>) {
let t = ctx.input(|i| i.time);
self.ui.toast = Some((text.into(), t + 1.4));
@@ -448,6 +471,7 @@ impl LightcraftApp {
import::poll_scan(self, ctx);
import::tick(self, ctx);
self.preview_build_status(ctx);
self.save_status(ctx);
self.slideshow_tick(ctx);
// back from an external editor: pick up the files it saved
let focused = ctx.input(|i| i.focused);
+23 -5
View File
@@ -250,6 +250,8 @@ pub fn checked(app: &LightcraftApp, id: &str) -> Option<bool> {
let u = &app.ui;
let panel = |p: RightPanel| Some(u.right == p);
match id {
"app.language.english" => Some(u.language == crate::i18n::Language::En),
"app.language.japanese" => Some(u.language == crate::i18n::Language::Ja),
"develop.autoSync" => Some(app.session.auto_sync),
"view.photoCounts" => Some(u.show_counts),
"view.secondWindow" => Some(u.second_window),
@@ -593,6 +595,22 @@ pub fn menu_bar(app: &LightcraftApp) -> Vec<(String, Vec<MenuNode>)> {
}
bar.push((title.to_string(), tidy(items)));
}
fn translate(nodes: &mut [MenuNode], language: crate::i18n::Language) {
for node in nodes {
match node {
MenuNode::Submenu { label, children } => {
*label = language.tr(label).to_string();
translate(children, language);
}
MenuNode::Item { label, .. } => *label = language.tr(label).to_string(),
MenuNode::Separator => {}
}
}
}
for (title, nodes) in &mut bar {
*title = app.ui.language.tr(title).to_string();
translate(nodes, app.ui.language);
}
bar
}
@@ -646,9 +664,9 @@ pub fn shortcut_text(sc: &str, mac: bool) -> String {
const TITLE_GAP: f32 = 24.0;
/// Width of the in-window menu bar's titles.
pub fn bar_width(ui: &egui::Ui) -> f32 {
pub fn bar_width(ui: &egui::Ui, language: crate::i18n::Language) -> f32 {
let t = crate::theme::Tokens::get(ui.ctx());
MENUS.iter().map(|m| ui.painter().layout_no_wrap(m.to_string(), t.font(13.0), t.text).size().x + TITLE_GAP).sum::<f32>()
MENUS.iter().map(|m| ui.painter().layout_no_wrap(language.tr(m).to_string(), t.font(13.0), t.text).size().x + TITLE_GAP).sum::<f32>()
}
/// The in-window menu bar (hosts without a native one): one dropdown per menu, or a single
@@ -666,14 +684,14 @@ pub fn show_in_window(app: &mut LightcraftApp, ui: &mut egui::Ui, max_width: f32
if total <= max_width {
let saved = ui.spacing().item_spacing.x;
ui.spacing_mut().item_spacing.x = TITLE_GAP;
for (title, items) in &bar {
for (index, (title, items)) in bar.iter().enumerate() {
let r = ui.add(egui::Button::new(egui::RichText::new(title).font(font.clone()).color(t.text_label)).frame(false));
crate::widgets::register(ui.ctx(), format!("menu:{title}"), r.rect);
crate::widgets::register(ui.ctx(), format!("menu:{}", MENUS.get(index).copied().unwrap_or(title.as_str())), r.rect);
egui::Popup::menu(&r).show(|ui| nodes_ui(ui, items, mac, &mut clicked));
}
ui.spacing_mut().item_spacing.x = saved;
} else {
let r = ui.add(egui::Button::new(egui::RichText::new("Menu").font(font.clone()).color(t.text_label)).frame(false));
let r = ui.add(egui::Button::new(egui::RichText::new(app.ui.language.tr("Menu")).font(font.clone()).color(t.text_label)).frame(false));
crate::widgets::register(ui.ctx(), "menu:all", r.rect);
egui::Popup::menu(&r).show(|ui| {
for (title, items) in &bar {
+22 -2
View File
@@ -11,6 +11,8 @@ use crate::state::{BeforeAfter, Dialog, RightPanel, ViewMode, Zoom};
pub type UiCommand = (&'static str, &'static str, Option<&'static str>, &'static str);
pub const UI_COMMANDS: &[UiCommand] = &[
("app.language.english", "English", None, "Edit>Language"),
("app.language.japanese", "日本語", None, "Edit>Language"),
("view.photoGrid", "Photo Grid", None, "View"),
("view.squareGrid", "Square Grid", None, "View"),
// G: Photo Grid ↔ Square Grid (from other views: the photo grid)
@@ -202,6 +204,10 @@ pub fn parse_rgb(v: &Value) -> Option<[u8; 3]> {
/// Handle UI commands; `None` means "not a UI command — send it to the engine".
pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Result<Value, String>> {
if matches!(id, "app.language.english" | "app.language.japanese") {
app.ui.language = if id == "app.language.japanese" { crate::i18n::Language::Ja } else { crate::i18n::Language::En };
return Some(Ok(json!(app.ui.language)));
}
let ctx = egui::Context::default();
let r: Result<Value, String> = match id {
"view.photoGrid" => {
@@ -856,7 +862,13 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
("Version".to_string(), env!("CARGO_PKG_VERSION").to_string()),
("System".to_string(), format!("{} ({})", std::env::consts::OS, std::env::consts::ARCH)),
("CPU threads".to_string(), std::thread::available_parallelism().map(|n| n.get().to_string()).unwrap_or_else(|_| "?".into())),
("GPU".to_string(), gpu.unwrap_or_else(|| "none (CPU rendering)".into())),
(
"GPU".to_string(),
gpu.unwrap_or_else(|| match lightcraft_engine::gpu::unavailable_reason() {
Some(why) => format!("none (CPU rendering): {why}"),
None => "none (CPU rendering)".into(),
}),
),
("GPU rendering".to_string(), if app.ui.settings.gpu { "on".into() } else { "off".into() }),
("Memory budget".to_string(), mb(lightcraft_engine::memory::default_budget() as u64)),
("Preview size".to_string(), format!("{} px", app.ui.settings.preview_edge)),
@@ -872,6 +884,9 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
format!("{:.1} ms (logic {:.1} ms; slowest {:.0} ms)", app.perf.update_ms, app.perf.logic_ms, app.perf.max_update_ms),
));
rows.push(("Last loupe render".into(), format!("{:.0} ms", app.renderer.last_main_ms)));
if let Some(f) = lightcraft_engine::gpu::last_fallback() {
rows.push(("Last GPU fallback".into(), f));
}
let r = json!(rows.iter().map(|(k, v)| json!({"label": k, "value": v})).collect::<Vec<_>>());
if p.get("open").and_then(Value::as_bool).unwrap_or(true) {
app.ui.dialog = Some(Dialog::SystemInfo { rows });
@@ -917,7 +932,12 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
if let Ok(v) = &r {
let n = v["found"].as_array().map_or(0, Vec::len);
let left = v["missing"].as_u64().unwrap_or(0);
app.toast(&egui::Context::default(), format!("Found {n} missing photo{}; {left} still missing", if n == 1 { "" } else { "s" }));
let unsure = v["ambiguous"].as_array().map_or(0, Vec::len);
let unsure = if unsure > 0 { format!(" ({unsure} with several look-alike files: use Locate)") } else { String::new() };
app.toast(
&egui::Context::default(),
format!("Found {n} missing photo{}; {left} still missing{unsure}", if n == 1 { "" } else { "s" }),
);
}
r
}
+2 -1
View File
@@ -551,8 +551,9 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
wm.image_width = (width / 100.0) as f32;
}
} else {
choices(ui, app.ui.language.tr("Orientation"), "exportWmOrientation", &[(false, app.ui.language.tr("Horizontal")), (true, app.ui.language.tr("Vertical"))], &mut wm.vertical);
field(ui, "Text", |ui| {
ui.add(egui::TextEdit::singleline(&mut wm.text).hint_text("© Your Name").desired_width(f32::INFINITY))
ui.add(egui::TextEdit::multiline(&mut wm.text).desired_rows(2).hint_text("© Your Name").desired_width(f32::INFINITY))
});
}
choices(
+17 -1
View File
@@ -342,7 +342,10 @@ fn performance_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {
heading(ui, t, "Rendering");
check(ui, "settings.gpu", &mut app.ui.settings.gpu, "Use the GPU for rendering");
let status = if !gpu::available() {
"No usable GPU found: rendering on the CPU".to_string()
match gpu::unavailable_reason() {
Some(why) => format!("Rendering on the CPU: {why}"),
None => "No usable GPU found: rendering on the CPU".to_string(),
}
} else {
format!("GPU: {}", gpu::adapter_name().unwrap_or_else(|| "starting…".into()))
};
@@ -376,6 +379,19 @@ fn performance_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {
let _ = app.run("library.clearPreviews", json!({}));
}
});
heading(ui, t, "Local folders");
row(ui, t, "Forget unchanged photos", |ui| {
let opts = [(0u32, "Never"), (7, "After 7 days"), (30, "After 30 days"), (90, "After 90 days"), (365, "After a year")];
let mut v = app.session.forget_local_days;
if choices(ui, "settingsForgetLocal", &opts, &mut v) {
let _ = app.run("library.preferences", json!({"forgetLocalDays": v}));
}
});
hint(
ui,
t,
"Photos seen in Local but never added or changed leave the catalog when their folder hasn't been browsed for this long (checked when the library opens). Files stay on disk; browsing the folder shows them again.",
);
#[cfg(not(target_arch = "wasm32"))]
smart_previews(app, ui, t);
}
+26 -3
View File
@@ -19,7 +19,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
let mut sw = 640.0f32.min(full.width() - 460.0).max(200.0);
if !app.native_menu {
// leave room for the in-window menus left of the (centred) search field
let menus_right = full.left() + 140.0 + crate::menubar::bar_width(ui) + 24.0;
let menus_right = full.left() + 140.0 + crate::menubar::bar_width(ui, app.ui.language) + 24.0;
sw = sw.min(2.0 * (full.center().x - menus_right)).max(200.0);
}
let search_left = full.center().x - sw / 2.0;
@@ -111,9 +111,18 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
}
// right icons
let mut x = full.right() - 18.0;
// saving is failing: the cloud icon turns into a warning until a save succeeds
let unsaved = app.session.unsaved().map(|(n, e)| {
format!(
"{n} change{} saved in memory but not written to disk: {e}\nLightCraft retries automatically; quitting now would lose {}.",
if n == 1 { "" } else { "s" },
if n == 1 { "it" } else { "them" }
)
});
let cloud_tip = unsaved.as_deref().unwrap_or("Local library — no cloud account needed");
for (id, icon, tip, cmd) in [
("discord", Icon::Chat, "Join the ArtCraft community on Discord", "app.discord"),
("cloud", Icon::Cloud, "Local library — no cloud account needed", ""),
("cloud", Icon::Cloud, cloud_tip, ""),
("help", Icon::Help, "Keyboard shortcuts", "app.shortcuts"),
("share", Icon::Share, "Export", "dialog.export"),
("bell", Icon::Bell, "Activity", "panel.activity"),
@@ -121,7 +130,21 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
let r = Rect::from_center_size(pos2(x, full.center().y), vec2(28.0, 28.0));
let resp = ui.interact(r, egui::Id::new(("top", id)), Sense::click()).on_hover_text(tip);
register(ui.ctx(), format!("icon:{id}"), r);
paint(ui.painter(), r.shrink(5.0), icon, if resp.hovered() { t.text } else { t.icon });
let warn = id == "cloud" && unsaved.is_some();
let colour = if warn {
t.caution
} else if resp.hovered() {
t.text
} else {
t.icon
};
paint(ui.painter(), r.shrink(5.0), icon, colour);
if warn {
let c = r.right_top() + vec2(-5.0, 6.0);
ui.painter().circle_filled(c, 6.0, t.reject);
ui.painter().text(c, Align2::CENTER_CENTER, "!", t.semibold(9.5), t.canvas);
register(ui.ctx(), "indicator:unsaved", r);
}
if resp.clicked() && !cmd.is_empty() {
let _ = app.run(cmd, json!({}));
}
+8
View File
@@ -200,9 +200,15 @@ pub const MIN_PHOTO_WIDTH: f32 = 360.0;
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(default, rename_all = "camelCase")]
pub struct UiState {
#[serde(default)]
pub language: crate::i18n::Language,
/// The Build Previews run last announced (its identity, finished?).
#[serde(skip)]
pub preview_build_seen: Option<(usize, bool)>,
/// Saving the library is failing (announced with a toast; the top bar shows a warning until
/// a later save succeeds).
#[serde(skip)]
pub unsaved_seen: bool,
pub view: ViewMode,
pub left_panel: bool,
pub right: RightPanel,
@@ -504,7 +510,9 @@ pub enum Dialog {
impl Default for UiState {
fn default() -> Self {
UiState {
language: crate::i18n::Language::default(),
preview_build_seen: None,
unsaved_seen: false,
luminance_map_restore: None,
view: ViewMode::Detail,
left_panel: false,
+81
View File
@@ -0,0 +1,81 @@
//! A failed save is visible: the command reports the error, a toast says the change is only in
//! memory, and the top bar shows a warning until a later save succeeds.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use lightcraft_catalog::{MemStore, Store};
use lightcraft_engine::library::LibraryStores;
use serde_json::json;
use crate::headless::Headless;
use crate::{LightcraftApp, Services};
const T: Duration = Duration::from_secs(20);
/// Appends fail while `fail` is set.
#[derive(Clone, Default)]
struct Flaky {
files: MemStore,
fail: Arc<AtomicBool>,
}
impl Store for Flaky {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.fail.load(Ordering::SeqCst) {
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"flaky".into()
}
}
#[test]
fn failed_save_is_reported_and_shown_until_a_save_succeeds() {
let store = Flaky::default();
let mut s = lightcraft_engine::Session::new();
let stores = LibraryStores { dir: "flaky".into(), catalog: Box::new(store.clone()), files: Box::new(MemStore::new()), on_disk: false };
s.open_library_in(stores, false).unwrap();
let app = LightcraftApp::new(s, Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
h.step();
let has_indicator = |h: &Headless| h.app.widgets.iter().any(|(w, _)| w == "indicator:unsaved");
assert!(!has_indicator(&h));
store.fail.store(true, Ordering::SeqCst);
let r = h.request("engine.execute", json!({"command": "album.create", "params": {"name": "Kept"}}), T);
assert_eq!(r["ok"], false, "{r}");
assert!(
r["error"].as_str().unwrap().contains("saved in memory but not written to disk: ") && r["error"].as_str().unwrap().contains("disk full"),
"{r}"
);
assert_eq!(h.app.session.catalog.albums().count(), 1, "the change is applied");
h.step();
h.step();
let toast = h.app.ui.toast.clone().map(|t| t.0).unwrap_or_default();
assert!(toast.contains("not written to disk"), "{toast}");
assert!(has_indicator(&h));
let inspect = h.request("ui.inspect", json!({}), T);
assert_eq!(inspect["result"]["unsaved"]["ops"], 1, "{inspect}");
// the disk comes back: the next command writes the queue, the warning goes away
store.fail.store(false, Ordering::SeqCst);
let r = h.request("engine.execute", json!({"command": "library.info"}), T);
assert_eq!(r["ok"], true, "{r}");
h.step();
h.step();
assert!(!has_indicator(&h));
assert_eq!(h.app.ui.toast.clone().map(|t| t.0).as_deref(), Some("Library saved"));
}
+4
View File
@@ -117,6 +117,10 @@ pub fn install_fonts(ctx: &egui::Context) {
let mut semi = vec!["Inter-SemiBold".to_string()];
semi.extend(fallback);
fonts.families.insert(FontFamily::Name(FONT_SEMIBOLD.into()), semi);
fonts.font_data.insert("japanese".into(), Arc::new(FontData::from_static(lightcraft_engine::export::WATERMARK_JAPANESE_FONT)));
for stack in fonts.families.values_mut() {
stack.push("japanese".into());
}
ctx.set_fonts(fonts);
}
+13 -1
View File
@@ -21,7 +21,7 @@ mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
| `engine.commands` | — | Engine + UI commands: id, label, menu, shortcut, params doc, enabled |
| `ui.menu.list` | — | Menu entries (flat: id, label, menu path, shortcut, enabled) |
| `ui.menu.tree` | — | The menu bar as shown (File … Help): items `{id, params?, label, shortcut?, enabled, checked?}`, separators, submenus — the model behind the native macOS menu bar and the in-window menus |
| `ui.inspect` | — | UI state, window, canvas/image rects, `scroll: {grid, filmstrip}` (scroll offsets in points, `null` until drawn), active photo, selection, perf (`frameMs` = layout, `logicMs` = per-frame logic before it, `updateMs` = both, `maxUpdateMs`, `fps`, render queue …), status, memory (bytes per cache, see `library.memory`; plus stage caches and textures), `export: {running: {total, done, current} \| null, last}` |
| `ui.inspect` | — | UI state, window, canvas/image rects, `scroll: {grid, filmstrip}` (scroll offsets in points, `null` until drawn), active photo, selection, perf (`frameMs` = layout, `logicMs` = per-frame logic before it, `updateMs` = both, `maxUpdateMs`, `fps`, render queue …, `gpu` = adapter in use, `gpuReason` = why renders don't use the GPU, `gpuFallback` = latest render redone on the CPU and why — see `docs/gpu-pipeline.md`), status, memory (bytes per cache, see `library.memory`; plus stage caches and textures), `export: {running: {total, done, current} \| null, last}` |
| `ui.widgets` | `{filter?}` | On-screen widgets `{id, rect: [x, y, w, h]}` (screen points) |
| `ui.clickWidget` / `ui.dragWidget` / `ui.hoverWidget` | `{id, count?, fx?, fy?}` / `{id, toX?, toY?, dx?, dy?, steps?}` / `{id, fx?, fy?}` | Real egui input on a widget (hover: the pointer rests on it, e.g. for preset/profile previews) |
| `ui.move` / `ui.click` / `ui.drag` | `{x, y, count?, button?}` / `{x, y, toX, toY, steps?}` | Raw pointer input, screen points |
@@ -37,6 +37,18 @@ mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
| `ui.render` | `{id?, size?, path?}` | Render a photo (PNG to `path`), `{width, height}` |
| `app.quit` | — | Close the app |
### When the library can't be saved
With a persistent library, every command that changes something is written to the catalog journal (fsynced) before
it replies. If that write fails (disk full, volume gone, permissions), the command replies `ok: false` with
`"saved in memory but not written to disk: <reason>; LightCraft will retry"`. The change itself **is** applied (and
undoable) and stays queued: the next command, and the app's frame loop every couple of seconds, retry the write, so
nothing is lost once the disk is writable again — unless the app quits first. Meanwhile `ui.inspect` → `unsaved`
is `{ops, error}` (else `null`), `library.info` reports `unsavedOps` / `unsavedError`, and the top bar's cloud icon
shows a warning (widget `indicator:unsaved`). Queries and commands that change nothing still succeed. A failed
compaction (snapshot) is not a failed command — the log is kept whole — and only shows in `library.info` →
`lastError`.
## Headless rendering (no window, no GPU)
The egui UI can be rasterized on the CPU (`crates/ui-egui/src/softpaint.rs`, driven by
+61 -5
View File
@@ -10,9 +10,9 @@ pure Rust (wgpu, naga); the drivers are the system's. No GL backend is compiled
available; grid/filmstrip thumbnails (many small jobs in parallel) stay on the CPU.
- Anything the GPU path cannot do returns `None` and the CPU renders instead: no adapter (CI
machines, software-only adapters), `LIGHTCRAFT_GPU=0` (whole process), the `app.gpu {enabled}`
command (runtime preference; `ui.inspect` → `perf.gpu` shows the adapter), an output larger than
the device's storage-buffer limit, or a device error / panic during a render (the process then
stays on the CPU).
command (runtime preference; `ui.inspect` → `perf.gpu` shows the adapter), a buffer larger than
the device's storage-buffer limit, or a render the device did not complete correctly (see
[Limits, failures and the CPU fallback](#limits-failures-and-the-cpu-fallback)).
- wasm32: the crate compiles to the CPU fallback (WebGPU needs asynchronous device creation and a
device shared with the canvas — open item below).
@@ -44,6 +44,59 @@ pure Rust (wgpu, naga); the drivers are the system's. No GL backend is compiled
per pass cost as much as the passes.
- `LIGHTCRAFT_PROFILE=1` prints GPU stage timings (each stage is then submitted and waited for).
## Limits, failures and the CPU fallback
Issue #78: on an Intel UHD (ICL GT1) iGPU with Mesa/Vulkan, exports from the desktop app came out
as valid JPEGs that were entirely black, while previews, `lightcraft-cli render` and
`LIGHTCRAFT_GPU=0` were fine — and the GPU export took ~15 s against ~3 s on the CPU. The readback
returned a buffer the per-pixel stage had never written, with no error. The likeliest cause: the
whole export was recorded as one command submission lasting seconds on that GPU, and the driver
reset it (GPU hang check / preemption timeout — the app's own window keeps the GPU busy, the CLI
doesn't) without the error reaching wgpu. wgpu also reports a lost device only through the
device-lost callback, which was not set. So a GPU render now refuses to hand back anything it
cannot vouch for:
- **Short submissions.** A render submits after every stage and every 16 M kernel invocations
(`FLUSH_INVOCATIONS`); the per-pixel stage runs in bands of ~4 MP. Every submission on a slow
iGPU stays short enough to be preempted and never trips a watchdog (Windows TDR ~2 s, i915).
Buffers released by earlier stages are reused by later ones, lowering peak device memory.
- **Error scopes.** Each render runs inside out-of-memory, validation and internal error scopes
(thread-local in wgpu): errors fail that render instead of disappearing into the uncaptured
handler. The device-lost callback and the uncaptured handler (other threads) stop GPU use.
- **Readback.** The wait has a timeout (60 s); a failed wait, map error or missing map callback
fails the render (before, a lost device left the readback waiting forever).
- **Limits.** The device is created with the adapter's limits; every buffer is checked against
`min(max_storage_buffer_binding_size, max_buffer_size)` when it is created. A render with a
buffer over the limit (24 MP RGB f32 = 275 MiB; a 128 MiB WebGPU-default device; many masks)
falls back for that render, without a device error. Dispatches stay under 65535 workgroups per
dimension (`groups1` folds 1-D kernels onto two axes). Exports over the limit are not tiled:
most stages need neighbourhoods tens to hundreds of pixels wide (guided filters at clarity /
base radius, dehaze dark channel) and some need global values (airlight percentile), so tiles
would need wide overlaps and a two-pass plan — the CPU renders those exports instead.
- **Sanity checks on the result.** The output buffer is cleared before the per-pixel stage, which
writes alpha 255 everywhere: any pixel without it means work did not run, and the GPU is not used
again (`incomplete image`). An entirely black result (n ≥ 4096 pixels) is re-rendered on the CPU
for that render only — a really black photo costs one extra CPU render, nothing else; there are
no false positives because the CPU result is what is returned.
| failure | this render | later renders |
|---|---|---|
| buffer over the device limit | CPU | GPU |
| out of device memory | CPU (buffer pool emptied) | GPU |
| entirely black result | CPU | GPU |
| validation / internal error, device lost, readback failure or timeout, incomplete image, panic | CPU | CPU (whole process) |
**Diagnosing.** `ui.inspect` → `perf.gpu` (adapter in use), `perf.gpuReason` (why renders don't use
the GPU, e.g. `"disabled by LIGHTCRAFT_GPU=0"`, `"software adapter (llvmpipe (LLVM 21.1.8, 256 bits))
skipped: …"`, `"stopped after a GPU failure: device lost …"`), `perf.gpuFallback` (the latest render
redone on the CPU and why, e.g. `"6000×4000: the GPU returned an incomplete image (24000000 of
24000000 pixels unwritten); the GPU is not used again"`). The same in `app.gpu` (`reason`,
`lastFallback`), Help ▸ System Info and Settings ▸ Performance. `LIGHTCRAFT_PROFILE=1` prints the
fallback with the stage timings; `RUST_LOG=warn` logs it.
**Reproducing a smaller GPU:** `LIGHTCRAFT_GPU_LIMITS=webgpu` (or `downlevel`) creates the device
with 128 MiB storage bindings / 256 MiB buffers, `LIGHTCRAFT_GPU_LIMITS=<n>` with n MiB / 2n MiB.
Tests inject failures with `lightcraft_gpu::inject_fault` (`crates/gpu/tests/fallback.rs`,
`export_falls_back_to_the_cpu_when_gpu_work_is_lost`).
## Correctness: CPU oracle and equivalence tests
`crates/gpu/tests/equivalence.rs` renders the same settings on both and compares 8-bit sRGB:
bounds **mean |Δ| < 0.5 LSB and max |Δ| ≤ 3 LSB** per channel. Measured (Apple M4 Pro, Metal):
@@ -120,8 +173,11 @@ the device on their first GPU render. `lightcraft_gpu::ready()` asks without blo
reading back); share the device with eframe.
- WebGPU in the browser build (async init, single device per canvas).
- Defringe and spot removal kernels; Sky/Subject heuristics (replaced by the segmenter in M12).
- Tiling for sources / outputs beyond the buffer limit (60 MP+ on smaller adapters): tile with
overlap equal to the largest filter radius.
- Tiling for sources / outputs beyond the buffer limit (24 MP+ on WebGPU-default limits, 60 MP+
on most desktop adapters; these render on the CPU now): tile with overlap equal to the largest
filter radius, with global values (airlight) from a first pass.
- Integrated GPUs much slower than the CPU (issue #78's GT1): a measured per-device choice of the
faster path for exports.
- Large-radius blurs at full size (dehaze dark channel at 24 MP) dominate the export: a summed-area
table or a downsampled dark channel would cut them further.
- GPU histogram (atomics) to skip the CPU pass over the readback.
+4
View File
@@ -141,6 +141,10 @@ results with `isError: true` so the model can read and correct them; malformed J
standard error codes (-32700 parse, -32600 invalid request, -32601 method not found, -32602
invalid params, -32002 resource not found).
With `--library`, a command whose change can't be written to disk (disk full, …) is an error too:
`saved in memory but not written to disk: <reason>; LightCraft will retry` — the change is applied in the session and
written by the next successful save (see [control-protocol.md](control-protocol.md#when-the-library-cant-be-saved)).
## One-shot commands: `lightcraft-cli run`
For agents that prefer a shell over an MCP session: run any chain of commands in one process and read one JSON
+48 -23
View File
@@ -21,13 +21,13 @@ and prints the summary; `cargo xtask parity --write` refreshes the summary table
<!-- parity:summary -->
| Section | ✅ | 🟡 | ⬜ | 🚫 | P0 done | P1 done |
|---|---:|---:|---:|---:|---:|---:|
| A. Import (IMP) | 11 | 2 | 1 | 1 | 3/4 (75%) | 5/5 (100%) |
| A. Import (IMP) | 11 | 3 | 1 | 1 | 3/5 (60%) | 5/5 (100%) |
| B. Library management (LIB) | 22 | 1 | 2 | 2 | 9/9 (100%) | 9/9 (100%) |
| C. Views & navigation (VIEW) | 16 | 0 | 1 | 0 | 9/9 (100%) | 4/4 (100%) |
| D. Search & filter (FILT) | 11 | 0 | 2 | 0 | 4/4 (100%) | 4/4 (100%) |
| E. Metadata (META) | 5 | 1 | 0 | 0 | 2/2 (100%) | 2/2 (100%) |
| F. Edit panel — global adjustments (EDIT) | 42 | 1 | 5 | 1 | 28/28 (100%) | 13/14 (93%) |
| G. Profiles (PROF) | 6 | 1 | 3 | 0 | 3/3 (100%) | 2/3 (67%) |
| G. Profiles (PROF) | 6 | 2 | 3 | 0 | 3/4 (75%) | 2/3 (67%) |
| H. Crop & rotate (CROP) | 9 | 0 | 0 | 1 | 6/6 (100%) | 3/3 (100%) |
| I. Remove / healing (REM) | 7 | 1 | 2 | 2 | 4/4 (100%) | 2/3 (67%) |
| J. Red eye (EYE) | 2 | 0 | 0 | 0 | — | 1/1 (100%) |
@@ -44,23 +44,37 @@ and prints the summary; `cargo xtask parity --write` refreshes the summary table
| U. Map & location (MAP) | 0 | 1 | 1 | 0 | — | 0/1 (0%) |
| V. Preferences (PREF) | 5 | 0 | 3 | 3 | 1/1 (100%) | 4/4 (100%) |
| W. Cloud & AI infrastructure (CLOUD / AI) | 1 | 0 | 1 | 2 | — | — |
| X. Cross-cutting behaviours (BEHAV) | 15 | 1 | 2 | 1 | 8/8 (100%) | 6/6 (100%) |
| X. Cross-cutting behaviours (BEHAV) | 15 | 2 | 2 | 1 | 8/8 (100%) | 6/7 (86%) |
| Y. Menus | 81 | 0 | 4 | 8 | 47/47 (100%) | 23/23 (100%) |
| Z. Keyboard shortcuts (desktop) | 74 | 3 | 3 | 1 | 49/52 (94%) | 22/23 (96%) |
| Lightroom Classic extras | 30 | 9 | 40 | 9 | — | 20/21 (95%) |
| **Total** | 386 | 25 | 91 | 37 | 194/198 (98%) | 138/146 (95%) |
| **Total** | 386 | 28 | 91 | 37 | 194/200 (97%) | 138/147 (94%) |
Weighted completion (✅ = 1, 🟡 = ½, 🚫 left out): **79.4%** of 502 in-scope rows — P0 99.0% of 198 · P1 96.2% of 146 · P2 39.2% of 158.
Weighted completion (✅ = 1, 🟡 = ½, 🚫 left out): **79.2%** of 505 in-scope rows — P0 98.5% of 200 · P1 95.9% of 147 · P2 39.2% of 158.
<!-- /parity:summary -->
## Top gaps
Ordered by tier, then user value, then (low) effort. Take the first one nobody is working on.
Ordered by user impact, then tier, then effort. The checklist above counts features that *exist*; these are the gaps
that decide whether a photographer can switch (see the honest assessment in [ROADMAP.md](../ROADMAP.md#where-we-stand)).
Take the first one nobody is working on.
1. **LR-IMP-FORMATS** (P0) — CR3, compressed NEF/RAF/ORF, RW2 v4 (preview only today: such photos are marked
`preview_only` and the UI says they are edited from the camera's embedded JPEG, issue #10); HEIC/AVIF decode (no
permissive pure-Rust decoder yet). High value, high effort (clean sources needed).
2. **LR-EDIT-OPTICS-PROFILE** (P1) — a lens-profile database of our own (embedded DNG/maker corrections work today).
1. **LR-PROF-CAMERACOLOR** (P0): our own camera colour calibration. Every non-DNG raw is developed with a neutral
matrix today, so colours are muted. Start by fitting each camera to its own embedded JPEG and using matrices the
files carry themselves; never Adobe data.
2. **LR-IMP-FORMATS** (P0): **CR3** first (every Canon body since ~2018), then compressed RAF / ORF, RW2 v4, Nikon
"lossy after split" NEF, Canon sRAW; HEIC/AVIF decode. Clean-room, from prose descriptions only (see
`crates/raw/src/vendor/nefc.rs` for how compressed NEF was done). Until decoded, such photos are `preview_only`.
3. **LR-IMP-CAMERA-COVERAGE** (P0): per-model verification; grow the CC0 corpus and fix per-model bugs (like the CR2
colour-filter layout, fixed in #85 by reading the file's own tag).
4. **LR-BEHAV-RENDER-FIDELITY** (P1): a side-by-side fidelity suite against Lightroom renders (kept local in `plan/`),
then tune tone, highlights, texture/clarity/dehaze, NR and sharpening against it.
5. **LR-EDIT-OPTICS-PROFILE** (P1): a lens-profile database of our own (embedded DNG/maker corrections work today).
6. **AI masks and Enhance** (LR-MASK-SUBJECT / SKY / PEOPLE / OBJECTS, LR-EDIT-DETAIL-DENOISE, SUPERRES, LENSBLUR):
blocked on a model strategy (permissively licensed weights or our own training, pure-Rust inference). A maintainer
decision, not just engineering.
7. **HDR** (Q. HDR, LR-EXP-HDR), **video** (R. Video), **Classic output modules** (Map view, Book, Slideshow module,
Print, publish): large, well understood, lower priority than 1–5.
## Shortcuts: conflicts and missing bindings
@@ -102,18 +116,19 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-IMP-ADD-DIALOG | Add photos/folders | P0 | ✅ | `cmd:file.addPhotos`, `cmd:library.importPreview`, `cmd:library.import` (`mode`, `album`, `albumName`, `preset`, `keywords`), `crates/ui-egui/src/import.rs` | review dialog (Import Photos): the scanned source (a scanned folder is not added to Local), candidate grid with thumbnails and checkboxes (duplicates by path/content marked and unchecked), transfer mode (add in place / copy / move, each explained), album (existing/new), preset, keywords; the folder scan runs on a background thread with a progress window and Cancel (a NAS folder no longer freezes the window); batched import with a progress window; one undo step. Drag-and-drop still imports directly |
| LR-IMP-DRAGDROP | Drop files/folders to import | P0 | ✅ | `crates/ui-egui/src/lib.rs` (dropped files → `cmd:library.import`) | dropping onto a specific album not supported |
| LR-IMP-DUPES | Skip duplicates by content | P1 | ✅ | `crates/engine/src/import.rs`, `crates/engine/src/tests_import.rs` | |
| LR-IMP-DEVICE | Import from camera/card | P1 | ✅ | `cmd:library.devices`, `cmd:file.addFromDevice`, `crates/engine/src/devices.rs` | mounted volumes with a DCIM folder (macOS /Volumes, Linux /media and /run/media, Windows drive letters); File → Import from Device → the import review, copying into the library by default; menus serve the last scan and rescan in the background (never blocking a frame; hot-plugs repaint); no PTP/MTP (cameras that don't mount as a disk) |
| LR-IMP-DEVICE | Import from camera/card | P1 | ✅ | `cmd:library.devices`, `cmd:file.addFromDevice`, `crates/engine/src/devices.rs` | mounted volumes with a DCIM folder (macOS /Volumes, Linux /media and /run/media, Windows drive letters); File → Import from Device → the import review, copying into the library by default; menus serve the last scan and rescan in the background (never blocking a frame; hot-plugs repaint); copies are verified like Move's (a new file, synced, compared byte for byte; a bad copy is removed and reported as failed, never counted as imported; taken names get -1, -2…, never replaced — issue #96, `crates/engine/src/import_move.rs`); no PTP/MTP (cameras that don't mount as a disk) |
| LR-IMP-AUTO | Watched-folder auto import | P2 | ✅ | `cmd:library.autoImport`, `cmd:library.autoImportScan`, `crates/ui-egui/src/panels/settings.rs` | Settings ▸ Import ▸ Auto Import: a watched folder whose new photos are added (in place or copied) once complete, into an optional album; scanned every 3 s |
| LR-IMP-PRESET | Preset on import | P2 | ✅ | `cmd:library.import` (`preset`) | chosen in the import review; one History entry |
| LR-IMP-RAWDEFAULT | Raw defaults | P1 | ✅ | `cmd:library.preferences`, `crates/engine/src/import.rs`, `crates/ui-egui/src/panels/settings.rs` | LightCraft default / a preset / per camera (make + model); non-raw default too; the preset look counts as unedited and Reset returns to it |
| LR-IMP-MIGRATE | Migrate other catalogs | OOS | 🚫 | | |
| LR-IMP-PROFILES | Import profiles & presets | P1 | ✅ | `cmd:file.importPresets`, `cmd:preset.import`, `cmd:profile.import`, `crates/engine/src/preset_import.rs`, `crates/engine/src/preset_luminar.rs` | presets: .lcpreset, XMP, classic .lrtemplate, photos carrying edits ("DNG presets"), Luminar looks (.lmp, .mplumpack collections; sliders with a counterpart), .zip bundles, folders (folder → group), drag & drop; masks carried over; unmapped settings reported; profiles: .cube 3D LUTs; Adobe profile formats deliberately unsupported |
| LR-IMP-LOCAL | Work on files in place | P0 | ✅ | `cmd:library.browse`, `cmd:photo.addToLibrary`, `cmd:library.import` (mode add), `cmd:local.addRoot`, `cmd:local.hide`, `cmd:local.restoreHidden`, `crates/engine/src/cmd/browse.rs`, `crates/ui-egui/src/panels/left.rs` (`local_section`), `crates/ui-egui/src/panels/grid.rs` (`folder_header`) | Local: browse Pictures / Desktop / Downloads / Home or any folder without adding it (breadcrumb, Include subfolders, Add N to My Photos); browsed photos stay out of All Photos, albums and counts; folders picked with Browse Folder… (or Keep in Local / Add to Local on a folder row) stay listed across restarts, and browsing a folder inside a listed one highlights it inside that tree (opened down to it, siblings reachable) instead of adding a root; edits go to XMP sidecars; importing promotes them; right-click a top-level location → Remove from Local hides the shortcut only (nothing on disk or in the catalog changes), saved with the UI state, and “Show N hidden locations” (or Browse Folder… on it) brings it back; a folder is one location however its path is spelled (`/` or `\`, trailing separator, `.`/`..`, drive-letter case; `folder_key` in `crates/catalog/src/query.rs`) |
| LR-IMP-LOCAL | Work on files in place | P0 | ✅ | `cmd:library.browse`, `cmd:photo.addToLibrary`, `cmd:library.import` (mode add), `cmd:local.addRoot`, `cmd:local.hide`, `cmd:local.restoreHidden`, `crates/engine/src/cmd/browse.rs`, `crates/ui-egui/src/panels/left.rs` (`local_section`), `crates/ui-egui/src/panels/grid.rs` (`folder_header`) | Local: browse Pictures / Desktop / Downloads / Home or any folder without adding it (breadcrumb, Include subfolders, Add N to My Photos); browsed photos stay out of All Photos, albums and counts; folders picked with Browse Folder… (or Keep in Local / Add to Local on a folder row) stay listed across restarts, and browsing a folder inside a listed one highlights it inside that tree (opened down to it, siblings reachable) instead of adding a root; edits go to XMP sidecars; importing promotes them; right-click a top-level location → Remove from Local hides the shortcut only (nothing on disk or in the catalog changes), saved with the UI state, and “Show N hidden locations” (or Browse Folder… on it) brings it back; a folder is one location however its path is spelled (`/` or `\`, trailing separator, `.`/`..`, drive-letter case; `folder_key` in `crates/catalog/src/query.rs`); browsed photos nobody changed are forgotten once their folder has not been browsed for 30 days (Settings → Performance → Local folders, 0 = never; checked when the library opens, or `cmd:library.forgetLocal` with `dryRun`; files and sidecars stay, browsing again brings them back; see `crates/catalog/src/local.rs` for what counts as a change) |
| LR-IMP-SIDECAR-SPLIT | Separate XMP sidecar variants | P2 | ⬜ | `cmd:library.xmpPreferences` | sidecar naming option exists (stem/full), no split sidecars |
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2, ARW, NEF, RAF, RW2, PEF, ORF. Missing: CR3, compressed NEF/RAF/ORF, RW2 v4 (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
| LR-IMP-FORMATS | Supported formats | P0 | 🟡 | `crates/codecs/src/lib.rs`, `crates/raw/src/lib.rs` | JPEG, PNG, TIFF, WebP, JXL, PSD, GIF, BMP; DNG (incl. lossy / Smart Preview DNG), CR2 (colour-filter layout from the file's `CR2CFAPattern` tag, issue #85), ARW, NEF (uncompressed, lossless and lossy compressed 12/14-bit, `crates/raw/src/vendor/nefc.rs`), RAF, RW2, PEF, ORF. Missing: CR3, compressed RAF/ORF, RW2 v4, NEF "lossy after split" (preview only: imported with `preview_only` = the decoder's reason, rendered as a rendered JPEG, flagged by a grid badge and loupe/Edit/Info notices, `previewOnly` in `catalog.query`; Reload clears it once the file decodes), HEIC/AVIF decode |
| LR-IMP-CAMERA-COVERAGE | Camera coverage, verified per model | P0 | 🟡 | `crates/raw/tests/corpus.rs`, `xtask/src/main.rs` | a supported container is not the same as every camera that writes it decoding correctly: ~40 CC0 corpus files (plus ~60 NEFs checked by hand) against the >1,000 models Lightroom lists; per-model differences still surface (e.g. the CR2 colour-filter layout differed by model until it was read from the file's own tag, issue #85, checked on 35 CR2 bodies). Grow the CC0 corpus per model and test each sample decodes with plausible colour |
| LR-IMP-CULL-AT-IMPORT | Culling analysis at import | P2 | 🟡 | `cmd:photo.analyze` | run Assisted Culling on the imported photos (they're selected after an import); not automatic |
| LR-IMP-MOVE | Move on import [Classic] | P1 | ✅ | `cmd:library.import` (`mode` move), `crates/engine/src/import_move.rs`, `crates/engine/src/tests_import_move.rs`, `crates/ui-egui/src/import.rs` | Transfer ▸ Move with Copy's destination, folders (day / month / one folder / custom template), rename template and example destination; XMP sidecars (both namings) move along; each source is removed only after its destination is written (hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate, unchecked files keep their sources, taken names get -1, -2…, never overwritten; files already in the destination/library are added in place; sources that can't be removed (read-only card) are kept and reported (`kept`). Undo removes the photos from the library but leaves the moved files at the destination. No Copy as DNG while moving |
| LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched) |
| LR-IMP-DNG-CONVERT | Convert to DNG on import [Classic] | P2 | ✅ | `cmd:library.import` (`dng`), `crates/ui-egui/src/import.rs` | copy imports: Raw files ▸ Copy as DNG (lossless; the card is untouched); the raw copy is removed only once its DNG is verified and on disk (issue #106) |
## B. Library management (LIB)
@@ -138,7 +153,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-LIB-DELETE | Delete / Recently Deleted | P0 | ✅ | `cmd:photo.delete`, `cmd:photo.restore`, `cmd:photo.deletePermanently` | no confirmation dialog, no auto-purge after N days, no "Empty" |
| LR-LIB-REMOVE-ALBUM | Remove from album | P0 | ✅ | `cmd:album.removePhotos` | |
| LR-LIB-DUPLICATE | Duplicate a photo | P2 | ✅ | `cmd:photo.duplicate` | Photo ▸ Duplicate: a real `-copy` file with the same settings, metadata and albums (virtual copies share the file) |
| LR-LIB-RENAME | Batch rename | P1 | ✅ | `cmd:photo.rename`, `cmd:photo.renamePreview`, `cmd:photo.renameTokens`, `cmd:dialog.rename`, `crates/engine/src/rename.rs` (`TOKENS`) | template tokens {name} {num} {seq:N} {date:%Y%m%d} {folder} {camera} {lens} {iso} {rating} {title} {creator} {ext} (shared with export naming and import rename); a Tags picker beside every template field (Rename Photos, import Rename, export File name) lists each tag with meaning + live example, the date directives and the rules, and inserts the clicked tag at the text cursor; unknown tags are flagged; preview; renames files on disk with their XMP sidecars, never overwriting (-1, -2… suffixes), rolls back on failure; undo/redo move the files; virtual copies follow |
| LR-LIB-RENAME | Batch rename | P1 | ✅ | `cmd:photo.rename`, `cmd:photo.renamePreview`, `cmd:photo.renameTokens`, `cmd:dialog.rename`, `crates/engine/src/rename.rs` (`TOKENS`) | template tokens {name} {num} {seq:N} {date:%Y%m%d} {folder} {camera} {lens} {iso} {rating} {title} {creator} {ext} (shared with export naming and import rename); a Tags picker beside every template field (Rename Photos, import Rename, export File name) lists each tag with meaning + live example, the date directives and the rules, and inserts the clicked tag at the text cursor; unknown tags are flagged; preview; renames files on disk with their XMP sidecars (a stem sidecar another file still shares, e.g. raw + JPEG, is copied rather than taken), never overwriting (-1, -2… suffixes; a change of letter case only is checked by file identity, so on case-sensitive volumes `img_1.jpg` next to `IMG_1.JPG` is a collision), rolls back on failure (a file that can't be moved back is reported old → new and the library follows it as an undoable partial rename; the same for undo/redo); undo/redo move the files; virtual copies follow |
| LR-LIB-CAPTURETIME | Edit capture time | P1 | ✅ | `cmd:photo.setCaptureTime`, `cmd:dialog.captureTime`, `crates/catalog/src/dates.rs` | set (the other selected photos shift by the same amount, or `each`), shift by days/hours/minutes, time-zone shift; one undo step, journaled; Info panel button; no “revert to original capture time” |
| LR-LIB-SHOWFINDER | Reveal original in file manager | P0 | ✅ | `cmd:app.showInFinder` | ⌘R (see MENU-FILE-SHOWFINDER) |
| LR-LIB-COVER | Album cover | P2 | ✅ | `cmd:album.setCover` | |
@@ -195,7 +210,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-META-COPYRIGHT-DEFAULT | Default copyright on import | P1 | ✅ | `cmd:library.preferences` (`import.copyright`, `import.creator`), `crates/ui-egui/src/panels/settings.rs` | Settings → Import → Metadata; fills only photos without their own |
| LR-META-LOCATION | Location editing | P2 | 🟡 | `cmd:photo.setMeta` (`location`, `city`, `state`, `country`, `gps`) | place fields and GPS (decimal or degrees / minutes / seconds) edited in Info, read/written as IPTC Core / Photoshop / EXIF XMP; Show on Map opens OpenStreetMap; no embedded map, no geocoding |
| LR-META-COPYPASTE | Copy / paste metadata | P2 | ✅ | `cmd:photo.copyMetadata`, `cmd:photo.pasteMetadata` | title, caption, alt text, extended description, copyright (notice, status, usage terms, info URL), creator, place fields, keywords; `fields` picks a subset |
| LR-META-XMP | XMP read/write | P0 | ✅ | `cmd:photo.saveMetadataToFile`, `cmd:photo.readMetadataFromFile`, `cmd:library.xmpPreferences`, `crates/engine/src/sidecar.rs`, `docs/xmp-interop.md` | sidecar wins for metadata; a sidecar capture time (`exif:DateTimeOriginal` / `photoshop:DateCreated` / `xmp:CreateDate`) fills in only when the file has none |
| LR-META-XMP | XMP read/write | P0 | ✅ | `cmd:photo.saveMetadataToFile`, `cmd:photo.readMetadataFromFile`, `cmd:library.xmpPreferences`, `crates/engine/src/sidecar.rs`, `docs/xmp-interop.md` | sidecar wins for metadata; a sidecar capture time (`exif:DateTimeOriginal` / `photoshop:DateCreated` / `xmp:CreateDate`) fills in only when the file has none; saving merges into an existing sidecar (`crates/meta/src/xmp_merge.rs`): another app's `crs:`, `xmpMM:History` and unknown namespaces are kept byte for byte, an unreadable sidecar is backed up first; with stem naming, files sharing a stem (raw + JPEG) get separate sidecars (`<file>.xmp` for all but the raw) |
| LR-META-EXIF-FULL | Full EXIF/IPTC [Classic] | P1 | ✅ | `cmd:photo.allMetadata`, `cmd:dialog.allMetadata`, `crates/meta/src/tags.rs`, `crates/meta/src/exif.rs`, `crates/meta/src/iptc.rs` | read and written on export; Info ▸ All Metadata…: every TIFF / EXIF / GPS / interop tag (named, common values spelled out) and the XMP fields, searchable; metadata presets |
## F. Edit panel — global adjustments (EDIT)
@@ -261,6 +276,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-PROF-ADOBE | Standard raw looks (own equivalents) | P0 | ✅ | `crates/engine/src/presets.rs` (`PROFILES`), `crates/pipeline/src/profiles.rs` | six own looks: Color, Neutral, Vivid, Landscape, Portrait, Monochrome |
| LR-PROF-ADAPTIVE | Adaptive profiles | P2 | ⬜ | | |
| LR-PROF-CAMERA | Camera-matching looks | P2 | ⬜ | | |
| LR-PROF-CAMERACOLOR | Camera colour calibration (own) | P0 | 🟡 | `crates/raw/src/color.rs` | DNG files use the colour matrices they carry; every other raw uses a neutral fallback (camera RGB ≈ linear sRGB, `matrix_is_fallback`) with as-shot white balance, so colours are muted and not accurate. Needs our own per-camera calibration: matrices the files carry themselves (e.g. Olympus `ColorMatrix`), fitting each camera to its own embedded JPEG, then chart shots. Adobe matrices / DCPs are never used. Biggest image-quality gap today |
| LR-PROF-CREATIVE | Creative profiles (own) | P2 | ✅ | `cmd:develop.profile`, `crates/pipeline/src/profiles.rs` | 16 own looks in Film / Cinematic / Muted / B&W (tone + point-curve fades, colour grading, mixer / B&W mix); scale with `ctl:profile.amount`; sliders untouched |
| LR-PROF-LEGACY | Legacy profiles | P2 | ⬜ | | |
| LR-PROF-NONRAW | Profiles for non-raw files | P0 | ✅ | `cmd:develop.profile` | same looks apply to JPEG/TIFF |
@@ -420,10 +436,10 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-EXP-METADATA | Metadata policy | P1 | ✅ | `cmd:app.export` (`metadata`, `removeLocation`) | |
| LR-EXP-WATERMARK | Watermark | P1 | ✅ | `cmd:app.export` (`watermark`: text or `image` + `imageWidth`), `crates/engine/src/export.rs` (`Watermark`) | text (size, colour, shadow) or a graphic with transparency (width as % of the photo, converted to the output colour space); position, inset, opacity |
| LR-EXP-NAMING | File naming | P1 | ✅ | `cmd:app.export` (`naming`, `startNumber`), `crates/engine/src/rename.rs` (`expand_tokens`) | one template language for export, Rename Photos and import renaming: `{name}`, `{num}` (number at the end of the name), `{seq}` / `{seq:N}`, `{date}` / `{date:%Y-%m-%d}`, `{folder}`, `{camera}`, `{lens}`, `{iso}`, `{rating}`, `{title}`, `{creator}`, `{ext}`; free-form template rather than a list of named schemes |
| LR-EXP-LOCATION | Destination folder | P0 | ✅ | `cmd:app.export` (`dir`, `subfolder`, `conflict`: unique / overwrite / skip), `crates/ui-egui/src/panels/dialogs.rs` (Choose… folder picker) | folder picker on the desktop; existing files get `-2`, `-3`… by default |
| LR-EXP-LOCATION | Destination folder | P0 | ✅ | `cmd:app.export` (`dir`, `subfolder`, `conflict`: unique / overwrite / skip), `cmd:export.checkTarget`, `crates/ui-egui/src/panels/dialogs.rs` (Choose… folder picker), `crates/engine/src/originals.rs`, `crates/catalog/src/safe_file.rs` | folder picker on the desktop; existing files get `-2`, `-3`… by default; the policy covers a file and its sidecars together; a catalogued original (or its sidecar) is never written over, whatever the policy or an exact `path` (export, `ui.render`, MCP `render_photo`, merge `previewPath`, `lightcraft-cli render`); files are written to a temp file, synced and renamed into place (issue #93) |
| LR-EXP-PREVIOUS | Export with previous settings | P0 | ✅ | `cmd:app.exportPrevious`, `cmd:dialog.export` | last options persist in prefs.json; dialog prefilled; no named export presets yet |
| LR-EXP-DNGOPT | DNG options | P2 | 🟡 | `cmd:app.export` (`dngCompression`), `crates/engine/src/export.rs` | compression: lossless JPEG (default), ZIP or none; no embedded JPEG preview size, no lossy DNG output |
| LR-EXP-ORIGINAL | Original + XMP | P1 | ✅ | `cmd:app.export` (`format: original`) | file copied byte for byte, sidecar named after the output |
| LR-EXP-ORIGINAL | Original + XMP | P1 | ✅ | `cmd:app.export` (`format: original`), `crates/engine/src/tests_export.rs` | file copied byte for byte, sidecar named after the output and subject to the conflict policy |
| LR-EXP-PHOTOS | Export to the system photo library | P2 | ⬜ | | |
| LR-EXP-PSD | Round trip to an external editor | P2 | ✅ | `cmd:photo.editExternal`, `cmd:photo.editInExternal`, `crates/engine/src/cmd/convert.rs` | a 16-bit TIFF `-Edit` copy with the edits (Adobe RGB / ProPhoto / P3 / sRGB) next to the original, added stacked on top of it and opened in the editor set in Settings ▸ General (or the system default) |
@@ -464,7 +480,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-CLOUD-SYNC | Cloud sync | OOS | 🚫 | | |
| LR-CLOUD-SMARTPREVIEW | Editable proxies | P2 | ✅ | `cmd:library.smartPreviews`, `cmd:library.smartPreviewsLocation`, `cmd:photo.smartPreview`, `crates/engine/src/smart.rs`, `crates/ui-egui/src/panels/settings.rs` | File ▸ Previews ▸ Build / Discard Smart Previews: ~1 MB proxies in the library; with the original offline the photo renders, edits and exports (at proxy size) from its proxy; Info shows the status; the proxy folder is chosen per library (Settings → Performance → Smart previews, saved in the library): the effective path, count and size are shown, changing it needs a choice for the proxies already built (move / leave / delete), and an unavailable or unwritable folder is an error, never a fallback to the library drive. The thumbnail cache stays in the library |
| LR-CLOUD-SMARTPREVIEW | Editable proxies | P2 | ✅ | `cmd:library.smartPreviews`, `cmd:library.smartPreviewsLocation`, `cmd:photo.smartPreview`, `crates/engine/src/smart.rs`, `crates/ui-egui/src/panels/settings.rs` | File ▸ Previews ▸ Build / Discard Smart Previews: ~1 MB proxies in the library; with the original offline the photo renders, edits and exports (at proxy size) from its proxy; Info shows the status; the proxy folder is chosen per library (Settings → Performance → Smart previews, saved in the library): the effective path, count and size are shown, changing it needs a choice for the proxies already built (move / leave / delete), and an unavailable or unwritable folder is an error, never a fallback to the library drive; proxies are written atomically and a damaged one (cut short by a crash or a full drive) doesn't count as built, is rebuilt (`repaired`) and is never moved over a good one (issue #106). The thumbnail cache stays in the library |
| LR-AI-UPDATE-INDICATOR | AI-settings update indicator | P2 | ⬜ | | |
| LR-AI-CREDITS | Generative credits | OOS | 🚫 | | |
@@ -472,7 +488,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-BEHAV-AUTOSAVE | Instant autosave | P0 | ✅ | `crates/catalog/src/journal.rs`, `crates/engine/src/library.rs` | |
| LR-BEHAV-AUTOSAVE | Instant autosave | P0 | ✅ | `crates/catalog/src/journal.rs`, `crates/engine/src/library.rs` | every change fsynced before the command returns; a failed write fails the command ("saved in memory but not written to disk"), keeps the change queued and retries; top-bar warning until saved; a write that fails part-way is cut back to the last whole record so the retry never reads back as damage (logs from older builds with such a fragment load in full); quitting saves queued changes in the closing snapshot even when the log can't be appended to |
| LR-BEHAV-UNDO | Global undo | P0 | ✅ | `cmd:edit.undo`, `crates/engine/src/tests.rs` (`rating_flag_undo_redo`) | covers ratings, albums, deletes, edits |
| LR-BEHAV-MULTISELECT | Multi-selection | P0 | ✅ | `cmd:library.select` (`replace`/`add`/`toggle`/`range`), `cmd:library.selectAll` | |
| LR-BEHAV-BATCH | Batch apply to selection | P0 | ✅ | `cmd:photo.rate`, `cmd:develop.paste`, `cmd:preset.apply`, `cmd:app.export` | |
@@ -480,7 +496,8 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-BEHAV-PROGRESSIVE | Progressive rendering | P0 | ✅ | `crates/engine/src/media.rs`, `crates/preview/src/lib.rs` | |
| LR-BEHAV-BG-TASKS | Background tasks | P0 | ✅ | `crates/preview/src/lib.rs` (`JobPool`), `crates/ui-egui/src/export_task.rs`, `crates/ui-egui/src/import.rs` | renders off the UI thread; exports started from the UI run on a worker thread with a progress panel and Cancel (`ui.inspect` → `export`); imports show a progress window. No combined activity centre |
| LR-BEHAV-OFFLINE | Offline editing | P1 | ✅ | | local-first: everything works offline |
| LR-BEHAV-GPU | GPU acceleration | P0 | ✅ | `crates/gpu/src/render.rs`, `cmd:app.gpu`, `docs/gpu-pipeline.md` | CPU fallback |
| LR-BEHAV-GPU | GPU acceleration | P0 | ✅ | `crates/gpu/src/render.rs`, `cmd:app.gpu`, `docs/gpu-pipeline.md` | CPU fallback per render on device limits, out of memory, device errors / loss, incomplete or black results (`perf.gpuReason`, `perf.gpuFallback`); short submissions for slow iGPUs; no tiling beyond the buffer limit (CPU) |
| LR-BEHAV-RENDER-FIDELITY | Rendering matches Lightroom | P1 | 🟡 | `crates/pipeline/src/lib.rs`, `crates/pipeline/src/tone.rs` | every slider exists and works, but the character of the result (tone curve shape, highlight recovery, texture / clarity / dehaze, noise reduction, sharpening, default look) is tuned by eye; there is no systematic side-by-side comparison against Lightroom on the same CC0 raws. Needs a fidelity suite: Lightroom reference renders kept only in the local `plan/` (never committed), compared per slider and preset with a perceptual metric |
| LR-BEHAV-DRAGDROP | Drag and drop | P1 | ✅ | `crates/ui-egui/src/lib.rs`, `crates/ui-egui/src/panels/grid.rs` (`drag_feedback`), `crates/ui-egui/src/panels/left.rs` (`drop_target`) | files → app (import); grid photos → an album row (adds the selection, with a count badge while dragging) |
| LR-BEHAV-TOAST | Toast notifications | P1 | ✅ | `crates/ui-egui/src/panels/mod.rs` | |
| LR-BEHAV-PANEL-RESIZE | Resizable side panels | P1 | ✅ | `crates/ui-egui/src/panels/mod.rs` (`resizable_side`), `crates/ui-egui/src/state.rs` (`LEFT_WIDTH`, `RIGHT_WIDTH`), `crates/ui-egui/src/tests_panels.rs` | drag the left sidebar's right edge (200–480 pt) or the right panel's left edge (250–520 pt); the photo area keeps ≥ 360 pt; widths are kept across panels, views and restarts (`leftWidth` / `rightWidth` in the UI state); the Presets column stays fixed |
@@ -710,8 +727,8 @@ From `08-lightroom-classic-extras.md` (Classic-only features) and part 2 of `06-
| LRC-LIB-PEOPLE | People view | P2 | ⬜ | | |
| LRC-LIB-COMMENTS | Comments panel | P2 | ⬜ | | |
| LRC-LIB-VISUALSEARCH | Find similar photos | P2 | ✅ | `cmd:library.findSimilar`, `crates/pipeline/src/cull.rs` (`signature`) | photo menu / Photo ▸ Find Similar Photos: look-alike photos (composition and tones), most similar first, as a filter (`only`) |
| LRC-LIB-MISSING | Missing files & relink | P1 | ✅ | `cmd:library.missing`, `cmd:photo.relink`, `cmd:library.findMissing`, `cmd:file.findMissing`, `cmd:photo.locate`, `crates/engine/src/cmd/missing.rs` (`checked_path`) | File → Find Missing Photos… (same name and size anywhere in a folder, one undo step), photo menu → Locate Missing File…; unreadable files show "!" in the grid and a reason in the loupe; undo never moves files. Missing Photos in the sidebar (counted on a worker thread); it covers library photos only — Local browse records are never checked, for the count, the view, `library.missing` and Find Missing alike |
| LRC-LIB-CONVERT | Convert to DNG | P2 | ✅ | `cmd:photo.convertToDng`, `crates/engine/src/cmd/convert.rs` | Photo ▸ Convert to DNG: lossless DNG next to the raw with the settings embedded, photo (and its virtual copies) relinked; originals kept; undoable |
| LRC-LIB-MISSING | Missing files & relink | P1 | ✅ | `cmd:library.missing`, `cmd:photo.relink`, `cmd:library.findMissing`, `cmd:file.findMissing`, `cmd:photo.locate`, `crates/engine/src/cmd/missing.rs` (`checked_path`) | File → Find Missing Photos… (anywhere in a folder: same name, size and content hash; renamed files by size + content hash; same name and size alone only when unambiguous — look-alikes are skipped and reported; one undo step), photo menu → Locate Missing File…; unreadable files show "!" in the grid and a reason in the loupe; undo never moves files. Missing Photos in the sidebar (counted on a worker thread); it covers library photos only — Local browse records are never checked, for the count, the view, `library.missing` and Find Missing alike |
| LRC-LIB-CONVERT | Convert to DNG | P2 | ✅ | `cmd:photo.convertToDng`, `crates/engine/src/cmd/convert.rs` | Photo ▸ Convert to DNG: lossless DNG next to the raw with the settings embedded, photo (and its virtual copies) relinked; originals kept; undoable; the DNG is decoded and compared with the raw data, written to a temp file, synced, read back and only then given a free name (never replacing a file) — a failed write leaves no DNG and the photo on its raw (issue #106) |
| LRC-LIB-PREVIEWS | Build / discard previews | P1 | ✅ | `cmd:library.buildPreviews`, `cmd:library.previewProgress`, `cmd:library.cancelPreviews`, `cmd:library.clearPreviews`, `crates/engine/src/cmd/previews.rs`, `crates/preview/src/lib.rs` | File ▸ Previews: build standard-sized (Settings → Performance size) or 1:1 previews of the selected / visible photos in the background (progress toasts, stop), discard the cache; disk thumbnail + view cache; no smart previews (offline proxies) |
| LRC-LIB-SLIDESHOW-IMPROMPTU | Impromptu slideshow | P2 | ✅ | `cmd:view.slideshow` | see LR-VIEW-SLIDESHOW |
| LRC-DEV-SNAPSHOTS | Named snapshots | P1 | ✅ | `cmd:version.create`, `cmd:version.restore` | = versions |
@@ -776,3 +793,11 @@ From `08-lightroom-classic-extras.md` (Classic-only features) and part 2 of `06-
| KEYC-DEVELOP | Classic develop keys (V, ⌘U, ⇧⌘U, R, Q, K, M, ⇧M, ⇧W, ⇧J, ⇧Q…) | P2 | 🟡 | `cmd:develop.treatment`, `cmd:develop.reset`, `cmd:crop.reset` | V, ⇧⌘R, ⌥⌘R, W, J, Y, ⇧Y, \ match; R/K/M/⇧M differ; no Classic keymap layer |
| KEYC-MODULE-OUTPUT | Book / slideshow / print / map / web keys | P2 | ⬜ | | modules not implemented |
| KEYC-HELP | Classic help keys (⌘/, F1) | P2 | 🟡 | `cmd:app.shortcuts` | ⌘/ only |
### Japanese text watermarks
`app.export` watermark objects accept `vertical` (boolean, defaults to false). The export
dialog offers Horizontal/Vertical and multiline text. Japanese glyphs fall back to bundled
BIZ UDMincho. Vertical lettering uses upright em cells and right-to-left newline columns;
Latin stays upright. Advanced Japanese composition remains open. Regression coverage:
`export::tests::japanese_watermarks_support_vertical_columns_and_legacy_defaults`.
+56
View File
@@ -9,7 +9,53 @@
with a counterpart here come along, the rest is listed.
- 23 new built-in presets: Portrait, Landscape, Urban, Food, Seasons, Vintage and B&W toners.
### Reliability
- Exports and renders never write over a photo's original (issue #93): exporting into the photo's own folder with
the same name and "Overwrite" (or Export with Previous repeating it), an exact output path from the control
channel or MCP, a merge preview path or `lightcraft-cli render IMG.jpg -o IMG.jpg` is refused with a clear
message, and the original is left byte for byte. Ordinary earlier exports are still overwritten when asked.
Exported files are written to a temp file, synced and then renamed into place, so a full disk or an unplugged
drive never leaves a truncated file; the XMP sidecar of an "Original" export follows the "If file exists"
choice too.
- Convert to DNG, Copy as DNG, Photo Merge and smart previews no longer write straight to the final file (issue
#106): a DNG is checked against the raw data, written to a temp file, synced and read back before it gets its
name (never replacing a file), and only then is the photo relinked or the raw copy removed — a failed write
leaves no DNG and keeps the raw. Smart previews are written the same way; a damaged one (cut short by a crash or
a full drive) no longer counts as built and Build Smart Previews replaces it.
- Import ▸ Copy verifies every copy, like Move (issue #96): each file is written as a new file, synced to disk and
compared byte for byte with the card. A copy that fails or differs is removed and reported as a failed import —
so "import complete" means the copies are good before you format the card — and a name that is taken gets -1,
-2… instead of being replaced.
- Saving metadata to an XMP sidecar another application wrote no longer replaces it (issue #92): LightCraft merges its
fields in and keeps the rest — e.g. that application's develop settings and edit history — byte for byte. A
sidecar that isn't valid XMP is copied to `<name>.xmp.bak-<time>` first. With the default stem naming, a raw and a
JPEG with the same name (`IMG_0001.CR3` + `IMG_0001.JPG`) no longer share one sidecar: the raw keeps `IMG_0001.xmp`,
the JPEG uses `IMG_0001.JPG.xmp`.
- A save that fails part-way (a full disk, a network share that drops) no longer looks like a damaged catalog
afterwards (issue #101): the partial write is cut off before LightCraft retries, so the next launch replays every
change. Catalogs already holding such a fragment load in full. Quitting while the catalog log can't be written
still saves your queued changes in the closing snapshot.
- Canon CR2 photos from the EOS 7D, 50D, 60D, 550D, 600D, 1200D, 1300D, 5D Mark II and 1D Mark IV (and other
models whose sensor starts on a green-blue row) no longer come out magenta (issue #85): the colour-filter
layout is read from each file instead of assumed.
- Exports are never black because of the GPU (issue #78): a GPU render that runs out of device
memory, exceeds the GPU's buffer limits, hits a driver error or reset, or comes back
incomplete is redone on the CPU — the file is the same image either way. Work is sent to the GPU
in short pieces so slow integrated GPUs aren't reset by their watchdog. `ui.inspect` → `perf`
(`gpuReason`, `gpuFallback`), Help ▸ System Info and Settings ▸ Performance say why the GPU isn't
used (e.g. a skipped software adapter such as llvmpipe) and why the last render fell back.
- The thumbnail cache only ever counts and deletes its own files (issue #98): a library opened on a folder that
already has a `thumbs/` folder of other pictures no longer loses them when the cache is trimmed or cleared.
### Library
- Rename Photos never overwrites another photo when only the letter case changes (issue #95): on case-sensitive
volumes (Linux, case-sensitive APFS) `img_1.JPG` next to `IMG_1.JPG` is a different photo and the renamed one gets
`img_1-1.JPG`; on case-insensitive volumes the case change still goes through.
- Rename Photos reports files it could not move back after a failure (issue #105), e.g. when a network share drops
mid-batch: the error lists them (old → new) and the library points at their new names (an undoable partial rename),
so none shows as missing. Renaming one of a raw + JPEG pair copies their shared `IMG_0001.xmp` instead of taking
it away from the other (issue #92). Find Missing Photos also finds renamed files by their content, prefers a content match
over a same-name same-size look-alike, and skips (and reports) photos it can't tell apart instead of guessing.
- Smart albums with a rule editor: match all / any / none, nested groups, 26 fields.
- Quick Collection and target album (B in the grid), keyword sets (⌥1–⌥9), colour-label sets.
- Colour-label filter with several labels at once; expandable folder tree in Local.
@@ -21,6 +67,12 @@
- Export file names use the Rename Photos tokens ({title}, {seq:2}, {date:%Y-%m-%d}…), plus new {num}, {folder}, {lens}, {iso}, {rating}, {creator}.
- Copyright status, rights usage terms and copyright info URL in Info, metadata presets and exports.
- Auto-Tag from Tracklog: GPS locations for your photos from a GPX track log, matched by capture time.
- A change that can't be saved to disk (full or unplugged drive) is no longer silent: the command reports
"saved in memory but not written to disk", the top bar shows a warning, and LightCraft keeps retrying until the
save goes through.
- Smaller, faster catalogs: photos you only looked at in Local (never added, rated or edited) are forgotten once
their folder has not been browsed for 30 days — your files and sidecars stay, and browsing the folder shows them
again. Change the period (or turn it off) in Settings → Performance.
### Editing
- Auto Sync: edits apply to every selected photo. Auto B&W mix. Automatic versions.
@@ -31,3 +83,7 @@
- Slideshow, second window, All Metadata, System Info.
- Edit in External Editor (⇧⌘E): a 16-bit TIFF copy, stacked, refreshed when you come back.
- Lossy DNG files and Smart Previews open as raw photos.
- Compressed Nikon NEFs (lossless and lossy compressed, 12- and 14-bit — e.g. D3200, D5100, D7000, D750, D850, Z 50)
now develop from the raw data instead of the camera's embedded JPEG, so a B&W or other picture style set in the
camera no longer gets baked in. Photos already imported as "preview only" switch over on Reload. (Files that
Nikon splits into two differently compressed halves still use the preview for now.)
+35 -2
View File
@@ -8,8 +8,8 @@ other tools), and let LightCraft pick up edits made elsewhere.
| | |
|---|---|
| Name | `<stem>.xmp` by default (`IMG_0001.CR3` → `IMG_0001.xmp`); `<file>.xmp` (`IMG_0001.CR3.xmp`) with `naming: "full"`. Reading accepts either (preferred first) and `.XMP`. |
| Write | `photo.saveMetadataToFile {ids?}` (Photo ▸ Save Metadata to File, ⌘S), or automatically after every change with `library.xmpPreferences {autoWrite: true}` (File ▸ Automatically Write Changes into XMP). Slider drags are written once, when the drag ends; undo/redo rewrite the sidecar. Written atomically (temp file + rename). |
| Name | `<stem>.xmp` by default (`IMG_0001.CR3` → `IMG_0001.xmp`); `<file>.xmp` (`IMG_0001.CR3.xmp`) with `naming: "full"`. Reading accepts either (preferred first) and `.XMP`. With stem naming, files sharing a stem don't share a sidecar: see *Shared names* below. |
| Write | `photo.saveMetadataToFile {ids?}` (Photo ▸ Save Metadata to File, ⌘S), or automatically after every change with `library.xmpPreferences {autoWrite: true}` (File ▸ Automatically Write Changes into XMP). Slider drags are written once, when the drag ends; undo/redo rewrite the sidecar. An existing sidecar is **merged into, never replaced** (see *Saving into an existing sidecar*). Written atomically (temp file, fsync, rename). The result lists `written`, `merged` and `backups`. |
| Read | On import (`library.import`, the report counts `sidecars`), and `photo.readMetadataFromFile {ids?}` (one undo step). For raw/DNG files without a sidecar, the XMP embedded in the file is used. |
| Preferences | `library.xmpPreferences {autoWrite?, naming?: stem\|full}`, stored in the library's `prefs.json`. |
@@ -37,6 +37,39 @@ used — on import (it then also files a copied photo in its date folder) and by
`xmp:Rating="-1"` (the XMP convention for rejected) sets the reject flag. Develop settings come from
`lc:settings` when present (exact); otherwise from the `crs:` fields below (approximate).
## Saving into an existing sidecar
A sidecar may already hold another application's data — e.g. its `crs:` develop settings and `xmpMM:History` — often
the only copy of those edits outside that application's catalog. Saving never drops it:
- LightCraft **owns** the properties in the table above: `xmp:Rating`, `xmp:Label`, `dc:title`, `dc:description`,
`dc:rights`, `dc:creator`, `dc:subject`, `Iptc4xmpCore:Location`, `Iptc4xmpCore:AltTextAccessibility`,
`Iptc4xmpCore:ExtDescrAccessibility`, `photoshop:City`/`State`/`Country`, `xmpRights:Marked`/`UsageTerms`/
`WebStatement` and everything in `lc:`. They are replaced on every save, and removed when LightCraft has no value
(clearing a title in LightCraft clears it in the file). All of them are read back on import, so the library starts
from what the sidecar said.
- The **capture time** (`exif:DateTimeOriginal`, `photoshop:DateCreated`) and **GPS** (`exif:GPSLatitude`,
`exif:GPSLongitude`) are replaced only when LightCraft has a value; otherwise the file's stay.
- **Everything else is kept byte for byte**: other namespaces (`crs:`, `xmpMM:`, `lr:hierarchicalSubject`, unknown
ones), `xmp:CreatorTool`, comments, the packet wrapper and padding. LightCraft's properties go into one
`rdf:Description` of their own; owned properties written by another application (in attribute or element form) are
removed from its description, which otherwise stays as it was.
- A sidecar that can't be read as XMP (not well-formed, not UTF-8, no `rdf:RDF`) is first copied to
`<name>.xmp.bak-<date><time>` (`-1`, `-2`… if taken — a backup is never overwritten), then replaced. A sidecar that
can't be read at all (permissions) is left alone and the save fails.
Limits: LightCraft writes keywords to `dc:subject` only, so another application's `lr:hierarchicalSubject` is kept as
it was and may still list keywords removed in LightCraft. LightCraft doesn't write `crs:`: the other application's
develop settings stay as that application left them, next to LightCraft's own (`lc:settings`, which LightCraft reads
first).
## Shared names
With stem naming, `IMG_0001.CR3` and `IMG_0001.JPG` map to the same `IMG_0001.xmp`. When two or more files in the
library share a stem, the stem sidecar belongs to one of them — a raw first, otherwise the first by file name — and
the others write and read `<file>.xmp` (`IMG_0001.JPG.xmp`), falling back to the stem sidecar for reading when they
have none. So saving one photo never overwrites the other's metadata (`Session::sidecar_naming`).
## Reading `crs:` develop fields
Many raw developers store edits as `crs:` properties (`http://ns.adobe.com/camera-raw-settings/1.0/`) in sidecars, in
+7
View File
@@ -234,6 +234,13 @@ const RAW_SAMPLES: &[(&str, &str)] = &[
("arw-sony-a7m4-lossless-l.arw", "https://raw.pixls.us/data/Sony/ILCE-7M4/ILCE-7M4_DSC06674_FullFrame-LossLess-Compressed-Large.ARW"),
("arw-sony-a7m4-lossless-m.arw", "https://raw.pixls.us/data/Sony/ILCE-7M4/ILCE-7M4_DSC06675_FullFrame-LossLess-Compressed-Medium.ARW"),
("arw-sony-a7m4-lossless-s.arw", "https://raw.pixls.us/data/Sony/ILCE-7M4/ILCE-7M4_DSC06676_FullFrame-LossLess-Compressed-Small.ARW"),
// CR2 colour-filter layouts differ by model (issue #85): CR2CFAPattern 3 (GBRG) and 1 (RGGB) samples
("cr2-canon-40d.cr2", "https://raw.pixls.us/data/Canon/EOS%2040D/_MG_0153.CR2"),
("cr2-canon-550d.cr2", "https://raw.pixls.us/data/Canon/EOS%20550D/IMG_4047.CR2"),
("cr2-canon-5d2.cr2", "https://raw.pixls.us/data/Canon/EOS%205D%20Mark%20II/08.canon.raw.cr2"),
("cr2-canon-5dsr.cr2", "https://raw.pixls.us/data/Canon/EOS%205DS%20R/_DSR2002.CR2"),
("cr2-canon-6d.cr2", "https://raw.pixls.us/data/Canon/EOS%206D/EOS_6D_RAW.CR2"),
("cr2-canon-7d.cr2", "https://raw.pixls.us/data/Canon/EOS%207D/RAW_CANON_EOS_7D-raw.CR2"),
("cr2-canon-5d3-sraw2.cr2", "https://raw.pixls.us/getfile.php/773/nice/Canon%20-%20EOS%205D%20Mark%20III%20-%20sRAW2%20%28sRAW%29.CR2"),
("cr2-canon-5d3.cr2", "https://raw.pixls.us/getfile.php/771/nice/Canon%20-%20EOS%205D%20Mark%20III.CR2"),
("cr2-canon-80d.cr2", "https://raw.pixls.us/getfile.php/1294/nice/Canon%20-%20EOS%2080D%20-%20RAW%20%283:2%29.CR2"),