Compare commits

...
Author SHA1 Message Date
Brandon Thomas 6c066b66f2 Merge main into fix/issue-97-folder-undo
# Conflicts:
#	crates/engine/src/lib.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:58:13 -04:00
Brandon Thomas e11f77d9ca Merge pull request #129 from storytold/fix/issue-94-control-close
Issue #94: Control: close the connection on malformed input
2026-10-05 18:57:59 -04:00
Brandon Thomas f383d1e736 Merge main into fix/issue-94-control-close
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:57:50 -04:00
Brandon Thomas edf05789ed Merge pull request #128 from storytold/fix/issue-103-settings-files
Issue #103: Settings files: keep damaged ones, never overwrite unread ones, write ui.json atomically, confirm quit with unsaved changes
2026-10-05 18:57:30 -04:00
Brandon Thomas bccccfb92a Merge main into fix/issue-103-settings-files
# Conflicts:
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 18:56:50 -04:00
Brandon Thomas 4939d9b994 Merge pull request #127 from storytold/fix/issue-100-open-failure
Issue #100: Ask what to do when the library can't be opened, never a silent demo session
2026-10-05 18:56:35 -04:00
Brandon Thomas e0ab41ede0 Merge main into fix/issue-100-open-failure 2026-10-05 18:56:00 -04:00
Brandon Thomas 8690b51fcf Merge pull request #126 from storytold/fix/issue-99-library-lock
Issue #99: Library: exclusive lock so two processes can't open one library
2026-10-05 18:55:44 -04:00
Brandon Thomas e1d3b56bb6 Merge main into fix/issue-99-library-lock
# Conflicts:
#	crates/catalog/src/lib.rs
2026-10-05 18:54:56 -04:00
Brandon Thomas 6100237cdb Merge pull request #125 from storytold/fix/issue-102-format-version
Issue #102: Catalog format version 2; refuse newer libraries untouched
2026-10-05 18:54:41 -04:00
Brandon Thomas 4579599745 Merge main into fix/issue-102-format-version 2026-10-05 18:54:03 -04:00
Brandon Thomas 8506fecf5f Merge pull request #124 from storytold/fix/issue-101-torn-append
Issue #101: catalog.log: cut a partial append back before retrying
2026-10-05 18:53:49 -04:00
Brandon Thomas 0f49ebf8f0 Merge main into fix/issue-101-torn-append
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:52:48 -04:00
Brandon Thomas ab86862011 Merge pull request #123 from storytold/fix/issue-92-xmp-merge
Issue #92: XMP: merge into an existing sidecar instead of replacing it
2026-10-05 18:52:25 -04:00
Brandon Thomas 577330a08b Merge main into fix/issue-92-xmp-merge
# Conflicts:
#	crates/engine/src/sidecar.rs
#	docs/whats-new.md
2026-10-05 18:50:42 -04:00
Brandon Thomas 30697cfc26 Merge pull request #122 from storytold/fix/issue-92-rename-shared-sidecar
Issue #92: Rename Photos copies a stem sidecar another file still shares
2026-10-05 18:50:26 -04:00
Brandon Thomas ddbf139e5e Merge main into fix/issue-92-rename-shared-sidecar 2026-10-05 18:49:29 -04:00
Brandon Thomas ee77490dd2 Merge pull request #121 from storytold/fix/issue-105-rename-rollback
Issue #105: Rename Photos reports failed rollbacks; Find Missing matches by content
2026-10-05 18:49:15 -04:00
Brandon Thomas f31a55c8a7 Merge main into fix/issue-105-rename-rollback
# Conflicts:
#	docs/parity.md
2026-10-05 18:48:27 -04:00
Brandon Thomas 7218b990f5 Merge pull request #120 from storytold/fix/issue-95-case-only-rename
Issue #95: Rename Photos: case-only renames never overwrite another file
2026-10-05 18:48:06 -04:00
Brandon Thomas d16d9aadeb Merge main into fix/issue-95-case-only-rename 2026-10-05 18:47:18 -04:00
Brandon Thomas 1caa04acbc Merge pull request #119 from storytold/fix/issue-98-thumb-cache-own-files
Issue #98: thumbnail cache only lists, counts and deletes its own files
2026-10-05 18:47:03 -04:00
Brandon Thomas bef2060b24 Merge main into fix/issue-98-thumb-cache-own-files 2026-10-05 18:46:59 -04:00
Brandon Thomas 40303b999c Merge pull request #118 from storytold/fix/issue-96-verified-copy-import
Issue #96: Import > Copy verifies each copy like Move
2026-10-05 18:46:42 -04:00
Brandon Thomas ce5488ad99 Merge main into fix/issue-96-verified-copy-import
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:46:20 -04:00
Brandon Thomas f6bc1ae5ae Merge pull request #117 from storytold/fix/issue-106-dng-smart-previews
Issue #106: Convert to DNG and smart previews: verified, atomic writes
2026-10-05 18:46:06 -04:00
Brandon Thomas 6c8a65c337 Merge main into fix/issue-106-dng-smart-previews 2026-10-05 18:45:48 -04:00
Brandon Thomas fe386f4f15 Merge pull request #116 from storytold/fix/issue-93-export-overwrite
Issue #93: Export: never overwrite an original; atomic writes
2026-10-05 18:45:31 -04:00
Brandon Thomas a6f421aafb Merge main into fix/issue-93-export-overwrite 2026-10-05 18:44:02 -04:00
Brandon ThomasandClaude Opus 5.5 5e618c553a Issue #100: Ask what to do when the library can't be opened, never a silent demo session
When open_library failed, the desktop app (and the web build) quietly ran
an in-memory session seeded with the demo photos: it looked like a reset
library and everything done in it was lost at quit.

- The session now starts empty and in memory (no demo photos) and
  LightcraftApp::library_problem shows a blocking window with the path and
  the error: Try Again, Choose Another Library… (app.openLibrary), Continue
  Without Saving, Quit. Command-line files are imported once the user has
  chosen.
- A temporary session shows a banner under the top bar (with Open
  Library…) for as long as it lasts; it has no library, so nothing is
  written anywhere. Opening a library ends it.
- ui.inspect → libraryProblem; widget ids for automation; documented in
  control-protocol.md.
- Web build: same window (Try Again disabled: reloading the page retries).
- Headless UI test: locked library → window → Continue Without Saving →
  banner, nothing written to the library → lock released → Try Again opens
  it. Checked in the real app over the control channel too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 cbebabccad Issue #102: Catalog format version 2; refuse newer libraries untouched
main added Op::SetBrowsed, Catalog.browsed and Photo.local_baseline while
the snapshot still said version 1, so v0.2.0 read such a library in part
(unknown ops as log damage, unknown fields dropped on its next snapshot).

- journal::VERSION = 2 (now public), with a format table and rules in the
  journal docs; a test with an exhaustive match over Op makes adding a
  variant fail until it is assigned a version.
- Journal::open reads the snapshot header first and refuses a newer
  version with CatalogError::Newer ("written by a newer version of
  LightCraft"), even when its catalog wouldn't parse.
- A log record whose CRC matches but whose op doesn't parse is "newer",
  not damage or a torn tail: refused before any file is modified.
- Opening an older (or snapshot-less) library writes a snapshot in this
  format before anything is appended (LoadReport::upgraded_from); a new
  library gets one at once, so v0.2.0 refuses it instead of reading a bare
  log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 08c2acda26 Issue #101: catalog.log: cut a partial append back before retrying
A failed append (disk full, dropped share) could leave part of its batch in
catalog.log; the retry (same seqs) then shared a line with the fragment and
the next load treated it as mid-log damage, setting later edits aside.

- Journal::append cuts the log back to the last whole record on error (or,
  if that fails too, before the next append).
- FsStore drops a failed append handle so the next attempt reopens the file;
  truncate of a never-created file is a no-op.
- The loader recognises a CRC-valid record glued after a fragment (logs
  written by older builds) and skips only the fragment.
- log_bytes now counts the newline added to an unterminated last record.
- close_library writes the closing snapshot even when appending fails; the
  snapshot absorbs the queued ops (Journal::snapshot_with_unlogged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 1a05c3eed2 Issue #103: Settings files: keep damaged ones, never overwrite unread ones, write ui.json atomically, confirm quit with unsaved changes
- Library settings (prefs.json, presets.json, view.json): only a missing
  file means "defaults". A file that doesn't parse is kept as
  <name>.corrupt-<unix time> before the defaults are used; one that can't
  be read (locked, I/O error) is never written this session (save_prefs
  refuses, presets/view saves skip it). Warnings go to library.info →
  settingsWarnings and Session::take_library_warnings; the UI shows each
  once in a window (ui.inspect → notices, button:noticeOk).
- Desktop ui.json (holds the library location): written atomically (temp +
  fsync + rename + dir fsync), saved as soon as the library path changes and
  every few seconds when it changed, failures reported; a damaged file is
  renamed to ui.json.corrupt-<time> and reported, an unreadable one is not
  overwritten.
- Quit/close with changes only in memory: retry the save (log, then a
  snapshot of everything), and if it still fails cancel the close and ask:
  Try Saving Again / Quit Anyway / Cancel (ui.inspect → quitPrompt).
- Tests: engine (damaged prefs kept, unreadable presets/prefs untouched),
  desktop (damaged ui.json kept, atomic write), headless UI (quit prompt
  buttons, settings notice).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:12:26 -04:00
Brandon ThomasandClaude Opus 5.5 ee4b78f646 Issue #99: Library: exclusive lock so two processes can't open one library
Two processes on one library each kept their own catalog and seq counter;
the last snapshot silently dropped the other's edits.

- lightcraft_catalog:🔒 LibraryLock holds an exclusive OS lock
  (std File::try_lock: flock / LockFileEx) on <library>/catalog.lock for
  the session; who holds it goes to catalog.lock.owner (separate file: a
  Windows lock also blocks reads). Released on drop and by the OS when the
  process dies, so leftover files from a crash never lock anyone out. File
  systems without lock support open unprotected with a warning.
- Session::open_library takes the lock before reading anything and fails
  with EngineError::LibraryInUse ("this library is already open in
  LightCraft 0.x (process N on host)"); reopening the session's own
  library hands its lock over.
- lightcraft-cli (mcp/run/snapshot --library) adds how to drive the
  running app instead (--control + mcp --connect).
- Tests: in-process, leftover files, a killed child process, two engine
  sessions, CLI e2e (mcp --library holds it, run --library is refused).
  Existing tests that reopened a library while it was open now drop the
  first session.
- docs: parity row LRC-LIB-CATALOG-LOCK, mcp.md, web.md (no guard across
  browser tabs yet), whats-new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:48:34 -04:00
Brandon ThomasandClaude Opus 5.5 d592b29d12 Issue #97: Rename Folder / Move Folder are undo steps
The folder move and the photos' relinks form one undo step: undo renames
the folder back on disk (files and sidecars along) and restores the
paths, redo repeats it. A destination taken in the meantime is never
overwritten; the step is refused, reported and stays on the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:30 -04:00
Brandon ThomasandClaude Opus 5.5 cddcb905e0 Issue #94: Control: close the connection on malformed input
A line that is not a JSON object with a string `method` (HTTP request
line, invalid UTF-8, non-object JSON, >4 MiB line) now gets one error
reply and the connection is closed, so a browser's cross-origin fetch
can't smuggle a command in its body. Connections are capped at 16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:27 -04:00
Brandon ThomasandClaude Opus 5.5 e96a2750a1 Issue #96: Import > Copy verifies each copy like Move
Copy import used exists() then std::fs::copy: no sync, no comparison, and
a file appearing at the chosen name in between was replaced. It now goes
through import_move::copy_new, which reuses Move's copy_verified: a new
file (create_new, retrying -1, -2… on AlreadyExists), synced and compared
byte for byte with the source; a copy that fails or differs is removed and
reported as a failed import. Copy as DNG's copy step uses the same path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:22:35 -04:00
Brandon ThomasandClaude Opus 5.5 771b0b98d7 Issue #106: Convert to DNG and smart previews: verified, atomic writes
- Convert to DNG / Copy as DNG: the encoded DNG must decode to the same raw
  data, then goes through safe_file::write_new_unique (temp file, sync,
  read-back, hard-linked to the first free <stem>.dng / -2 / -3 name, never
  replacing a file). On failure no DNG is left, the photo is not relinked
  and the raw copy is kept.
- Photo Merge writes its DNG the same way.
- Smart previews are written atomically; smart::is_valid (header + complete
  JPEG) decides "built", has_smart_preview and the offline fallback, so a
  proxy cut short is rebuilt (reported as `repaired`); moving proxies to a
  new folder never deletes a good one in favour of a damaged copy and
  copies across drives atomically.

Stacked on the #93 branch (shared lightcraft_catalog::safe_file helper).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:17:41 -04:00
Brandon ThomasandClaude Opus 5.5 f4df914731 Issue #92: Rename Photos copies a stem sidecar another file still shares
Renaming IMG_1.CR3 alone took the shared IMG_1.xmp along and left
IMG_1.JPG without it. A Stem-named sidecar that a sibling photo file
still uses is now copied (independently, never linked) instead of moved;
the last file renamed takes it. A rollback removes the copy.

Stacked on fix/issue-105-rename-rollback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:12:25 -04:00
Brandon ThomasandClaude Opus 5.5 882a059fac Issue #92: XMP: merge into an existing sidecar instead of replacing it
Saving (Cmd+S or auto-write) now merges LightCraft's properties into an
existing sidecar (lightcraft_meta::merge_xmp): owned properties (rating,
label, dc/IPTC/rights fields, lc:*) are replaced or removed, capture time
and GPS only when LightCraft has a value; everything else (crs:,
xmpMM:History, unknown namespaces, packet wrapper) is kept byte for byte.
The result is re-parsed and checked before it is written (temp + fsync +
rename). A sidecar that isn't valid XMP is copied to .xmp.bak-<time>
(never overwriting) before it is replaced; one that can't be read at all
is left alone and the save fails.

Shared stems: when several catalogued files map to one stem sidecar
(IMG.CR3 + IMG.JPG), it belongs to the raw (else the first by name) and
the others write and read <file>.xmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:08:18 -04:00
Brandon ThomasandClaude Opus 5.5 46732cb0fe Issue #93: Export: never overwrite an original; atomic writes
- lightcraft_catalog::safe_file: one tested helper for crash-safe writes
  (temp file in the same folder, sync, rename, folder sync; temp removed on
  failure), write_new/write_new_unique that never replace a file and read
  the result back, same_file, and a per-thread fault injection for tests.
- engine::originals::OriginalGuard: refuses a write whose target is a
  catalogued photo's original or XMP sidecar (by file identity), whatever
  the conflict policy or exact path. Used by run_batch (export, Export with
  Previous, control/MCP exact paths), ui.render/ui.screenshot paths, merge
  previewPath, MCP render_photo/screenshot paths (new query command
  export.checkTarget) and lightcraft-cli render.
- The conflict policy now covers an export's sidecars together with the
  file (Unique gives both the same free name, Skip skips when either is
  taken).
- export::write_file replaces the plain std::fs::write in the desktop app,
  CLI, MCP and engine writers; XMP sidecar writes use the shared helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:03:07 -04:00
Brandon Thomas 572b55a44a Merge pull request #110 from midasdf/feat/japanese-interface-and-watermarks
Add Japanese menus and vertical Japanese text watermarks
2026-10-05 16:49:18 -04:00
Brandon Thomas bd10f89d02 Merge main into PR #110 2026-10-05 16:48:57 -04:00
Brandon ThomasandClaude Opus 5.5 fb5b18843f Issue #105: Rename Photos reports failed rollbacks; Find Missing matches by content
Rollbacks in move_file, apply_rename and undo/redo no longer ignore
errors: every move that can't be undone is reported (old -> new), and the
catalog follows the files that kept their new names (an undoable partial
rename; a logged change for undo/redo), so nothing shows as missing.
A file that can't go back keeps the sidecars that moved with it.

library.findMissing: same name + size confirmed by the stored content
hash first, then renamed files by size + content hash, then a unique
name + size match; several look-alikes are skipped and reported in
'ambiguous' instead of guessed. Files already used by a photo are skipped.

Stacked on fix/issue-95-case-only-rename (same functions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:47:50 -04:00
Brandon ThomasandClaude Opus 5.5 dfbb1ad935 Issue #95: Rename Photos: case-only renames never overwrite another file
Whether an existing target is the photo itself is decided by file identity
(dev+inode on Unix; an exact directory entry check elsewhere), not by a
lower-case name comparison, so on case-sensitive volumes img_1.JPG next to
IMG_1.JPG is a collision (-1 suffix). Moves use a no-replace rename (hard
link + unlink where supported) and case changes go through a unique temp
name. Tests model case-sensitive and case-insensitive volumes and check
the real disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:36:35 -04:00
Brandon ThomasandClaude Opus 5.5 0fd0ffb783 Issue #98: thumbnail cache only lists, counts and deletes its own files
scan() now accepts only <2 hex>/<32 hex>.jpg (matching shard) and the
cache's own <32 hex>.tmp<id> temp files, so prune/clear never delete and
size() never counts foreign files in a shared thumbs/ folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:23:32 -04:00
Brandon Thomas e99aa61fc4 Merge pull request #111 from storybored379/fix/unicode-timezone
Metadata: gracefully ignore invalid Unicode timezone offsets
2026-10-05 16:22:21 -04:00
Brandon ThomasandClaude Opus 5.5 58626b982d Fonts: replace Shippori Mincho with BIZ UDMincho (no Adobe-derived glyphs)
Shippori Mincho's name table credits 'FONTDASU / Google Inc. / but / Adobe' — its kanji
come from Adobe's Source Han Serif, which our asset rules exclude even under the OFL
(CLAUDE.md: no Source Han fonts). BIZ UDMincho (Morisawa Inc., design TypeBank Co., Ltd.;
OFL-1.1, from google/fonts at 63833b7) is a Mincho of its own with no Adobe glyphs, and
smaller (6.2 MB vs 8.7 MB). Attribution, licence text and NOTICE updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:38 -04:00
midasdf e00025aaf6 i18n: add Japanese menus and vertical Japanese watermarks 2026-10-06 01:07:26 +09:00
Brandon Thomas 294012742e Merge pull request #108 from storytold/pr/freebsd-ci-sync
FreeBSD CI: match the shared template
2026-10-05 10:19:07 -04:00
Brandon ThomasandClaude Opus 5.5 4d13061580 FreeBSD CI: match the shared template (git + safe.directory, 4 vCPUs, pinned vmactions/freebsd-vm)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:45:24 -04:00
Brandon Thomas 63380829b1 Merge pull request #91 from storytold/pr/freebsd
FreeBSD: build and test the workspace in a FreeBSD VM
2026-10-05 09:30:12 -04:00
Brandon ThomasandClaude Opus 5.5 62e16d7adc FreeBSD CI: 4 vCPUs, git in the VM, pinned vmactions/freebsd-vm
The headless UI snapshot test settles for at most 120 s; on a 2-vCPU VM
under the full test load it timed out mid-render.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:58:05 -04:00
Brandon ThomasandClaude Opus 5.5 d47e941a0c docs: point agents to the shared test corpora (#90)
Real-file test oracles live in storytold/photocraft-corpus, explained in
craftrules standards/test-corpora.md. Never commit large binary fixtures
here; fetch them pinned and sha256-verified.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:24:23 -04:00
Brandon ThomasandClaude Opus 5.5 83fb206a29 FreeBSD: build and test the workspace in a FreeBSD VM
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:23:26 -04:00
Brandon Thomas 3567980442 Merge pull request #89 from storytold/chore/ignore-target-dirs
Ignore /target-*/ build dirs
2026-10-05 07:38:00 -04:00
Brandon ThomasandClaude Opus 5.5 8cd791100b Ignore /target-*/ build dirs
Agents sometimes build into target-<name>/ next to target/; ignore those too so a
'git add -A' can never commit build artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:37:49 -04:00
Brandon Thomas ff1e51e4d4 Merge pull request #88 from storytold/fix/issue-85-cr2-cfa
Issue #85: Raw: CR2 colour pattern from the file's CR2CFAPattern tag, not a constant
2026-10-05 07:37:19 -04:00
Brandon Thomas c05f2de251 Merge main into fix/issue-85-cr2-cfa
# Conflicts:
#	docs/parity.md
2026-10-05 07:37:00 -04:00
Brandon ThomasandClaude Opus 5.5 e87ee8e36d Issue #85: Raw: CR2 colour pattern from the file's CR2CFAPattern tag, not a constant
The Bayer layout of Canon CR2 files differs by model (GBRG on the 7D, 50D,
60D, 550D, 600D, 1200D, 1300D, 5D Mark II, 1D Mark IV; RGGB on the 40D,
5D Mark III, 6D, 5DS R and others). Read it from raw IFD tag 0xc5e0
(CR2CFAPattern: 1 RGGB, 2 BGGR, 3 GBRG, 4 GRBG), anchored at the full
decoded sensor; fall back to RGGB when the tag is missing. No model table.

Checked on 35 decoded CC0 raw.pixls.us CR2s (35 models): the tag agrees
with the mosaic's green-diagonal statistic on every file; the SensorInfo
border parity does not predict the layout. Corpus manifest gains the
reporter's samples; a corpus test asserts each layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 07:24:26 -04:00
Brandon Thomas efef3a5df1 Merge pull request #87 from storytold/docs/roadmap-assessment
Docs: honest parity assessment — where we stand, where we're going
2026-10-05 06:49:16 -04:00
Brandon ThomasandClaude Opus 5.5 b50fbf1285 Docs: honest parity assessment — where we stand, where we're going
ROADMAP.md: new 'Where we stand' (by dimension and by kind of user; checklist 79% vs
~60-70% as a day-to-day Lightroom replacement) and 'Where we're going' (ordered priorities:
camera colour calibration, CR3 and compressed raws, verified camera coverage, render fidelity
suite, lens profiles, AI model strategy, then HDR / Classic modules / video). Current
parity numbers, M11 status, risks (raw-source decision of 2026-10-05), log.

docs/parity.md: new rows LR-PROF-CAMERACOLOR, LR-IMP-CAMERA-COVERAGE (P0) and
LR-BEHAV-RENDER-FIDELITY (P1) so quality gaps are tracked; Top gaps reordered by user impact.

README.md: honest status summary above the feature table; stale rows fixed (compressed NEF,
DNG/original export, image watermark, Move import), camera colour and AI/HDR/video rows.

AGENTS.md / CLAUDE.md: agents read the assessment before picking work, and downgrade
✅ rows they find wrong or incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:48:54 -04:00
Brandon Thomas 92bd3aa688 Merge pull request #86 from storytold/fix/issue-10-nef-compressed
Issue #10: Raw: Nikon compressed NEF (lossless + lossy)
2026-10-05 06:08:28 -04:00
Brandon ThomasandClaude Opus 5.5 6592e62912 Issue #10: Raw: Nikon compressed NEF (lossless + lossy)
Decode Nikon Huffman-compressed NEF (compression 34713): lossless and lossy
(type 1 and 2) at 12 and 14 bits, with the maker-note 0x96 linearization
curve, predictor seeds and either byte order. Clean-room: 0x96 layout and
encoding-table sizes from prose write-ups (Clevy, Claff); the fixed Huffman
tables were recovered by black-box analysis of CC0 raw.pixls.us samples
(compressed vs uncompressed shots of the same scene) and verified on ~45
files from ~35 bodies. Lossy-after-split files stay preview only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 05:58:31 -04:00
Brandon Thomas 8532d4ea56 Merge pull request #84 from storytold/fix/issue-78-gpu-black-export
Issue #78: GPU export: respect device limits, fall back to CPU on GPU errors
2026-10-05 05:09:57 -04:00
Brandon ThomasandClaude Opus 5.5 2218ca5a4c Issue #78: GPU export: respect device limits, fall back to CPU on GPU errors
A GPU render no longer hands back an image it cannot vouch for:
- set the wgpu device-lost callback (wgpu never reports a lost device as an
  error, so a driver reset went unnoticed); readback waits with a timeout and
  checks the poll result and map callback instead of blocking forever
- each render runs in OutOfMemory/Validation/Internal error scopes; errors
  fail that render (OOM and limit failures for that render only, device
  errors for the process) and it is redone on the CPU
- every buffer is checked against the device limit when created (no
  validation error, no process-wide disable); LIGHTCRAFT_GPU_LIMITS=webgpu|<MiB>
  creates the device with lower limits to reproduce small adapters
- the output buffer is cleared and the per-pixel kernel writes alpha 255:
  any unwritten pixel (work silently dropped, the issue's all-black export)
  fails the render; an all-black result is re-rendered on the CPU
- renders submit per stage and every 16 M invocations, the per-pixel stage
  runs in ~4 MP bands: no long submission for a slow iGPU's watchdog to reset
- ui.inspect perf.gpuReason / perf.gpuFallback, app.gpu reason/lastFallback,
  System Info and Settings say why the GPU is off (e.g. llvmpipe skipped) and
  why the last render fell back

Tests: crates/gpu/tests/fallback.rs (limit, mid-render limit, validation
error, dropped work, black photo), engine export falls back to the CPU image,
banded 3000x2000 GPU/CPU equivalence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 05:06:24 -04:00
Brandon Thomas 14053b2e40 Merge pull request #83 from storytold/fix/local-record-eviction
Local: forget untouched browse records of folders not browsed for 30 days
2026-10-05 02:59:23 -04:00
Brandon Thomas 1f2f25356a Merge main into fix/local-record-eviction
# Conflicts:
#	crates/engine/src/cmd/library.rs
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 02:59:05 -04:00
Brandon Thomas 2ed6af155d Merge pull request #82 from storytold/fix/persist-errors
Persistence: a failed save fails the command (change kept in memory, queued, retried)
2026-10-05 02:58:49 -04:00
Brandon ThomasandClaude Opus 5.5 b14cf58630 Local: forget untouched browse records of folders not browsed for 30 days
Local browse records can be most of a large catalog (85k photos, 90 %
Local: 313.5 MB snapshot). Untouched Local records of folders whose last
browse is older than library.preferences forgetLocalDays (default 30,
0 = never) are now removed from the catalog when the library opens (and by
library.forgetLocal {dryRun?, days?}), as journaled, non-undoable ops, then
compacted in the background. Files and XMP sidecars are never touched;
browsing the folder again catalogues them afresh.

Untouched = a Local file record, not deleted, not a virtual copy, whose
state still hashes to the fingerprint taken when the browse catalogued it
(any change keeps it); records from before fingerprints need every
user-editable field empty/default. Kept too: records in albums (or as
cover), stacks, sources of virtual copies, visible/selected/edited ones,
ones referenced by undo/redo, ones with a smart preview.

Per-folder last-browsed times live in the catalog (Op::SetBrowsed,
normalised with folder_key; re-stamped at most hourly). Old catalogs load
unchanged; folders without a time are stamped now, so nothing is forgotten
on the first open after the upgrade.

persist_bench (85k photos, 76.5k Local, every 7th folder recent, 1 in 50
rated): 63.8k forgotten, plan 162 ms, snapshot 313.5 MB -> 94.9 MB.
Settings -> Performance -> Local folders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 02:26:49 -04:00
Brandon ThomasandClaude Opus 5.5 104444edcb Persistence: a failed save fails the command (change kept in memory, queued, retried)
A command whose journal append fails now returns EngineError::NotSaved
("saved in memory but not written to disk: <reason>; LightCraft will retry")
instead of OK. The change stays applied and undoable, its ops stay queued and
are written by the next save (any command, or the frame loop with a 2 s
backoff). Queries and commands that change nothing are unaffected; a failed
compaction is reported in lastError, not as a failed command (and a failed
synchronous snapshot now backs off like a background one).

Session::unsaved(), library.info unsavedOps/unsavedError, ui.inspect unsaved;
the desktop UI toasts the failure and shows a warning on the top bar's cloud
icon until a save succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 02:26:49 -04:00
storybored379andOpenAI Codex 35e38661a7 Metadata: gracefully reject invalid Unicode timezone offsets
Co-Authored-By: OpenAI Codex <noreply@openai.com>
2026-10-04 21:11:55 -07:00
113 changed files with 8529 additions and 599 deletions

No files matched your search

+48
View File
@@ -0,0 +1,48 @@
# Builds and tests the workspace on FreeBSD, inside a FreeBSD VM on a Linux runner
# (GitHub has no FreeBSD runners). GPU tests skip themselves when the VM has no adapter.
name: FreeBSD
on:
push:
branches: [main]
pull_request:
paths:
- Cargo.toml
- Cargo.lock
- "**/Cargo.toml"
- .github/workflows/freebsd.yml
workflow_dispatch:
concurrency:
group: freebsd-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
freebsd:
name: FreeBSD 14 (build + test)
runs-on: ubuntu-latest
timeout-minutes: 300
steps:
- uses: actions/checkout@v4
- name: Build and test in a FreeBSD VM
uses: vmactions/freebsd-vm@c46abacb49f09938ca4e1702d15d836285d694cc # v1
with:
release: "14.3"
usesh: true
cpu: 4
mem: 12288
# Nothing to bring back; copying the target dir out of the VM fails on large builds.
copyback: false
prepare: |
pkg install -y git curl pkgconf cmake python3 libxkbcommon wayland libX11 libXcursor libXrandr libXi libxcb mesa-libs vulkan-loader gtk3 fontconfig freetype2 alsa-lib
curl -fsSL https://sh.rustup.rs | sh -s -- -y --profile minimal
run: |
. "$HOME/.cargo/env"
# The checkout is synced in under another owner; let tests that call git read it.
git config --global --add safe.directory "*"
rustc --version
CARGO_PROFILE_DEV_DEBUG=0 cargo build --workspace --all-targets --locked
CARGO_PROFILE_DEV_DEBUG=0 cargo test --workspace --locked
+1
View File
@@ -1,6 +1,7 @@
*~
plan/
/target
/target-*/
/dist
/corpus
/log
+8 -2
View File
@@ -5,8 +5,13 @@ LightCraft is a clean-room, open-source, pure-Rust photo library + non-destructi
## Start every session here
1. Read `plan/STATUS.md` (current milestone, next unchecked task, blockers).
2. Read the task in `plan/execution-plan.md` §3, the relevant section of `plan/architecture.md`, and the README/docs of the crate you touch. Behaviour/visual reference: `plan/lightroom/` (incl. `10-observed-ui.md` + screenshots).
3. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (Lightroom parity tracker: one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
4. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
3. **Know where we stand:** read [`ROADMAP.md`](ROADMAP.md) → *Where we stand* (honest assessment by dimension and
by kind of user) and *Where we're going* (ordered priorities). The checklist counts features that *exist*; the real
gaps are quality and coverage: camera colour calibration, CR3 / compressed raws, per-model verification, render
fidelity against Lightroom, AI models, HDR / video / Classic modules. A ✅ row is not proof of parity — if you find a
✅ feature that is wrong or incomplete, downgrade it to 🟡 with a note.
4. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (ordered by user impact; the tracker has one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit, and the *Where we stand* / *Where we're going* sections of `ROADMAP.md` when a listed gap closes; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
5. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
`plan/` is gitignored (local-only).
@@ -86,6 +91,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
- Shell gotcha: `mv`/`cp` are aliased interactive here — use `/bin/mv -f` / `/bin/cp -f`.
- Parallel agents: separate git worktrees and `CARGO_TARGET_DIR=target/agent-<name>`; each agent uses its **own control port** (pick one in 18000–19999, never the default 7980) and its own scratch subfolder (`<scratch>/<agent-name>/`) — never `rm -rf` shared paths; delete your target dir when done (disk is shared); keep every `Cargo.toml` valid at all times (the `crates/*` glob means one broken manifest breaks everyone).
- Test corpora: `cargo xtask corpus --download` into `corpus/` (gitignored, CC0 only). Never commit media.
- Shared real-file test corpora (Photoshop-authored PSDs, etc.) live in [`storytold/photocraft-corpus`](https://github.com/storytold/photocraft-corpus), explained in [craftrules `standards/test-corpora.md`](https://github.com/storytold/craftrules/blob/main/standards/test-corpora.md). Never commit large binary fixtures to this repo; fetch them pinned by commit and sha256-verified, as PhotoCraft does with `cargo xtask corpus`.
## Testing & performance (do this often)
- Unit/property tests next to the code; end-to-end tests drive real binaries (`crates/mcp/tests/e2e.rs`,
+8 -2
View File
@@ -5,8 +5,13 @@ LightCraft is a clean-room, open-source, pure-Rust photo library + non-destructi
## Start every session here
1. Read `plan/STATUS.md` (current milestone, next unchecked task, blockers).
2. Read the task in `plan/execution-plan.md` §3, the relevant section of `plan/architecture.md`, and the README/docs of the crate you touch. Behaviour/visual reference: `plan/lightroom/` (incl. `10-observed-ui.md` + screenshots).
3. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (Lightroom parity tracker: one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
4. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
3. **Know where we stand:** read [`ROADMAP.md`](ROADMAP.md) → *Where we stand* (honest assessment by dimension and
by kind of user) and *Where we're going* (ordered priorities). The checklist counts features that *exist*; the real
gaps are quality and coverage: camera colour calibration, CR3 / compressed raws, per-model verification, render
fidelity against Lightroom, AI models, HDR / video / Classic modules. A ✅ row is not proof of parity — if you find a
✅ feature that is wrong or incomplete, downgrade it to 🟡 with a note.
4. **Pick work** from [`docs/parity.md`](docs/parity.md) → *Top gaps* (ordered by user impact; the tracker has one row per feature, menu item and shortcut). When you land a feature, update its row(s) and the gap list in the same commit, and the *Where we stand* / *Where we're going* sections of `ROADMAP.md` when a listed gap closes; `cargo xtask parity` (part of `ci`) checks that every `cmd:`/`ctl:` id and path the tracker cites still exists, and `cargo xtask parity --write` refreshes its summary.
5. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask unless §7 lists the decision as the user's.
`plan/` is gitignored (local-only).
@@ -86,6 +91,7 @@ of it. Full standard: `../craftrules/standards/never-crash.md`
- Shell gotcha: `mv`/`cp` are aliased interactive here — use `/bin/mv -f` / `/bin/cp -f`.
- Parallel agents: separate git worktrees and `CARGO_TARGET_DIR=target/agent-<name>`; each agent uses its **own control port** (pick one in 18000–19999, never the default 7980) and its own scratch subfolder (`<scratch>/<agent-name>/`) — never `rm -rf` shared paths; delete your target dir when done (disk is shared); keep every `Cargo.toml` valid at all times (the `crates/*` glob means one broken manifest breaks everyone).
- Test corpora: `cargo xtask corpus --download` into `corpus/` (gitignored, CC0 only). Never commit media.
- Shared real-file test corpora (Photoshop-authored PSDs, etc.) live in [`storytold/photocraft-corpus`](https://github.com/storytold/photocraft-corpus), explained in [craftrules `standards/test-corpora.md`](https://github.com/storytold/craftrules/blob/main/standards/test-corpora.md). Never commit large binary fixtures to this repo; fetch them pinned by commit and sha256-verified, as PhotoCraft does with `cargo xtask corpus`.
## Testing & performance (do this often)
- Unit/property tests next to the code; end-to-end tests drive real binaries (`crates/mcp/tests/e2e.rs`,
Generated
+1
View File
@@ -2188,6 +2188,7 @@ dependencies = [
"serde",
"serde_json",
"thiserror 2.0.21",
"web-time",
]
[[package]]
+4
View File
@@ -14,3 +14,7 @@ list (author, source, license) is in assets/ATTRIBUTION.md.
Inter (assets/fonts/Inter-*.ttf) — SIL Open Font License 1.1. See assets/fonts/OFL-Inter.txt.
Showcase photographs by Ansel Adams, Dorothea Lange and NASA (docs/images/) — public domain. See assets/ATTRIBUTION.md.
BIZ UDMincho: Copyright 2022 The BIZ UDMincho Project Authors
(https://github.com/googlefonts/morisawa-biz-ud-mincho). SIL Open Font License 1.1;
see assets/fonts/OFL-BIZUDMincho.txt.
+19 -7
View File
@@ -280,9 +280,18 @@ lightcraft --control 7980 ~/Pictures/trip
## Feature status
LightCraft is young and moving fast; see the [roadmap](ROADMAP.md) for estimates. The detailed, row-by-row
Lightroom parity tracker (every feature, menu item and shortcut with its status, evidence and the top gaps) is
[docs/parity.md](docs/parity.md).
LightCraft is young and moving fast. **Where we honestly stand** (details in the [roadmap](ROADMAP.md#where-we-stand)):
- **By feature count we're at ~79%** of Lightroom (core features 98%), tracked row by row in
[docs/parity.md](docs/parity.md).
- **As a day-to-day Lightroom replacement we're nearer 60–70%.** It's great for JPEG/DNG and most Nikon / Sony /
older-Canon raws on one machine.
- **The biggest gaps:**
- **camera colour calibration:** raws other than DNG develop with a neutral colour matrix today, so colour is muted;
- **CR3 and compressed Fujifilm / Olympus raws:** these open as embedded previews only;
- **AI masks and denoise:** subject and sky selection are classical heuristics;
- **HDR, video and the Classic Print / Book / Map modules.**
- **What's next:** see [where we're going](ROADMAP.md#where-were-going).
| Area | Status |
|---|---|
@@ -292,17 +301,20 @@ Lightroom parity tracker (every feature, menu item and shortcut with its status,
| Masking: brush, linear, radial, luminance/colour range, add/subtract/intersect | ✅ (AI subject/sky use classical heuristics for now) |
| Crop, straighten tool + auto straighten, flip, rotate, aspect ratios, overlays | ✅ |
| Profiles (Color, Neutral, Vivid, Landscape, Portrait, Monochrome: our own looks), presets, versions, history, copy/paste/sync settings | ✅ |
| Camera colour: DNG files use their own matrices | ✅ · our own calibration for other raws ⬜ (top priority; neutral fallback today) |
| Native macOS menu bar (generated from the command registry), control channel + every widget addressable, headless UI snapshots | ✅ |
| RAW: DNG, CR2, ARW, NEF (uncompressed), Fujifilm RAF (Bayer + X-Trans), Panasonic RW2, Pentax PEF, Olympus ORF; embedded previews for every format incl. CR3 | ✅ · compressed NEF/RAF/ORF, CR3 decode 🚧 |
| RAW: DNG, CR2, ARW, NEF (uncompressed + lossless/lossy compressed), Fujifilm RAF (uncompressed, Bayer + X-Trans), Panasonic RW2, Pentax PEF, Olympus ORF (uncompressed); embedded previews for every format incl. CR3 | ✅ · CR3, compressed RAF/ORF decode ⬜ |
| Detail: sharpening, luminance + colour noise reduction | ✅ · AI Denoise, Super Resolution ⬜ |
| Remove / Heal / Clone spots (auto source), Visualize Spots, Red Eye and Pet Eye (auto pupil detection, catchlight) | ✅ · content-aware fill, spot pin editing 🚧 |
| Export: JPEG / PNG / TIFF / WebP / AVIF, sizing, file-size limit, output sharpening, naming, batch, metadata policy, text watermark | ✅ · DNG export, image watermark ⬜ |
| Library persistence (crash-safe op log + snapshots), disk thumbnail cache, import with duplicate detection | ✅ |
| Export: JPEG / PNG / TIFF / WebP / AVIF / DNG / original, sizing, file-size limit, output sharpening, naming templates, batch, metadata policy, text or image watermark | ✅ · HDR export ⬜ |
| Library persistence (crash-safe op log + snapshots, background compaction, failed saves reported), disk thumbnail cache | ✅ |
| Import: Add in place / Copy / Move, rename and folder templates, devices, duplicate detection, watched folders; Local folder browsing | ✅ |
| MCP server (headless or live app, persistent libraries), CLI, control channel | ✅ |
| XMP sidecars (read/write, auto-write), reading `crs:` develop settings, preset files (`.lcpreset`, XMP presets) | ✅ |
| Optics (distortion, vignetting, auto + manual CA, defringe, DNG-embedded lens corrections), Geometry (transforms, Constrain Crop), Upright (Auto/Level/Vertical/Full/Guided) | ✅ · camera lens profiles (our own) ⬜ |
| Photo Merge: HDR (auto-align, deghost), Panorama (spherical/cylindrical/perspective, boundary warp, auto crop), HDR Panorama → DNG | ✅ |
| GPU pipeline (wgpu compute, CPU-exact within 1/255) with CPU fallback | ✅ · WebGPU in the browser 🚧 |
| GPU pipeline (wgpu compute, CPU-exact within 1/255), CPU fallback on device limits / errors | ✅ · WebGPU in the browser 🚧 |
| AI: segmentation masks, AI denoise, super resolution, faces; HDR editing; video | ⬜ (see [roadmap](ROADMAP.md#where-were-going)) |
| Web build (same UI in the browser via WASM): persistent library in OPFS/IndexedDB, Web Worker rendering, export downloads | ✅ · WebGPU, Safari/Firefox testing 🚧 |
<sub>✅ works today · 🚧 in progress · ⬜ not started</sub>
+98 -18
View File
@@ -4,6 +4,68 @@ Milestones toward full Adobe Lightroom parity (cloud Lightroom first, then every
estimates for continuous (24/7) agent-driven development with 4–6 parallel agents. Estimates are calibrated on the sibling
projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revised as milestones land.
## Where we stand
*Honest assessment, 2026-10-05. Agents: read this before picking work. The checklist in
[`docs/parity.md`](docs/parity.md) counts features that **exist**; this section is about whether a photographer can
**switch** from Lightroom. Update it when a gap below closes.*
**In one line:** the checklist says **79%** (P0 98.5%, P1 95.9%, P2 39%), but measured by whether a working
photographer could replace Lightroom without noticing, we are at roughly **60–70%**. The remaining gap is mostly
**quality of results and camera coverage**, not missing buttons.
Caveat on the checklist: ✅ is set by whoever lands a feature, and nobody has systematically checked rows against
Lightroom's behaviour or output. Bugs keep turning up in ✅ areas (CR2 colour-filter phase on some Canon models #85,
duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
### By dimension
| Dimension | Estimate | What's true today | Biggest gaps |
|---|---:|---|---|
| **Feature checklist** | 79% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
| **RAW coverage** (formats people shoot) | ~50% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, packed RW2, PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, RW2 v4, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. Per-model verification is thin (~40 corpus files vs >1,000 models) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No camera colour calibration**: non-DNG raws use a neutral matrix, so colour is muted/inaccurate. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
| **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |
| **Platform & robustness** | ~70% | macOS native; Windows/Linux builds; web via WASM; no-panic lints workspace-wide, `unsafe` confined to `crates/sysmem`; failed saves are reported; GPU errors fall back to CPU | Windows installer UI unverified on Windows (PR #79); GPU path proven only on Apple + user reports; no localisation; accessibility partial; headless UI tests time out under machine load |
### By kind of user
| User | Readiness | What blocks them |
|---|---:|---|
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour (muted colour on every photo) |
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only |
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
| Relies on AI (masks, denoise) | ~25% | No segmentation or AI denoise models |
## Where we're going
Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/parity.md) → *Top gaps*.
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
matrices the files carry themselves, then chart shots. Raises colour quality for every non-DNG raw at once.
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, RW2 v4, NEF
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
no LGPL dependency); compressed NEF (#86) is the template.
3. **Verified camera coverage** (LR-IMP-CAMERA-COVERAGE, P0): a CC0 sample per model in the corpus, each decoded and
checked for plausible colour; fix per-model bugs (#85).
4. **Render fidelity suite** (LR-BEHAV-RENDER-FIDELITY, P1): measure our output against Lightroom on the same CC0 raws
(references stay in the local `plan/`), then tune against the numbers.
5. **Lens profiles of our own** (LR-EDIT-OPTICS-PROFILE, P1).
6. **AI model strategy** (maintainer decision): which permissively licensed models (or our own training) for
segmentation masks and denoise; then pure-Rust inference. Unblocks M12 and Enhance.
7. **Then:** HDR (Q), the Classic output modules (Print first, then Map view, Book, Slideshow), video (R), localisation
and accessibility.
Already closed in the week of 2026-10-03: compressed NEF (#10), Move import (#29), per-library smart-preview folder,
Local roots / cleanup, grid and catalog performance at 85k photos (#35, #37), failed-save reporting, GPU export
hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help and folder templates (#31, #32).
## Milestones
**Status legend:** ✅ done · 🚧 in progress · ⬜ not started
| # | Milestone | Scope (summary) | Estimate (h) | Status |
@@ -11,7 +73,7 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi
| M0 | Skeleton + visual shell | workspace, xtask CI + layering, geom/color/raster, develop model, pipeline v0, catalog v0, engine commands, Lightroom-look UI (grid, loupe, filmstrip, Edit panel), control channel, MCP, web build | 3–5 | ✅ |
| M1 | Library core | import (JPEG/PNG/TIFF/WebP), EXIF/XMP, persistent catalog (op log + snapshots), albums, ratings/flags/labels, filter/search/sort, thumbnail cache, 100k-photo grid | 6–10 | ✅ (100k-photo grid scale test pending) |
| M2 | Pipeline v1 (quality) | WB temp/tint, profiles, local tone mapping (highlights/shadows), curves, HSL, point colour, colour grading, texture/clarity/dehaze, vignette, grain, B&W, auto tone/WB, histogram, before/after | 10–15 | ✅ (look tuning vs our references ongoing) |
| M3 | RAW I | TIFF/DNG (LJ92, deflate, tiles, opcodes), demosaic (AHD/PPG/bilinear), highlight recovery, DNG colour model, CR2, NEF, ARW, embedded previews | 10–15 | 🚧 (DNG, CR2, ARW, NEF uncompressed, embedded previews ✅; NEF Huffman ⬜) |
| M3 | RAW I | TIFF/DNG (LJ92, deflate, tiles, opcodes), demosaic (AHD/PPG/bilinear), highlight recovery, DNG colour model, CR2, NEF, ARW, embedded previews | 10–15 | 🚧 (DNG, CR2, ARW, NEF uncompressed + Huffman lossless/lossy, embedded previews ✅; NEF lossy-after-split ⬜) |
| M4 | Crop, geometry, optics | crop tool + overlays, straighten, Upright (auto/level/vertical/full/guided), manual transforms, CA, defringe, manual lens corrections | 6–10 | ✅ |
| M5 | Performance | source pyramids, wgpu compute pipeline (CPU oracle), draft/full renders, prefetch, budgets (16 ms slider updates on 24 MP) | 10–15 | 🚧 (stage cache, source pyramid, wgpu pipeline, prefetch, memory budget ✅; colour NR at half resolution, GPU histogram ⬜) |
| M6 | Masking | brush, linear/radial gradients, colour/luminance/depth range, add/subtract/intersect/invert, all local adjustments, masks panel | 8–12 | 🚧 (brush/linear/radial/colour/luminance range, add/subtract/intersect, masks panel ✅; depth range, AI masks ⬜) |
@@ -19,24 +81,27 @@ projects (DrawCraft reached its first four milestones in ≈ 4½ h) and are revi
| M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) |
| M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ |
| M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) |
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; CR3, compressed NEF/ORF/RAF ⬜) |
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; **camera colour calibration** ⬜ (top priority), CR3, compressed ORF/RAF ⬜) |
| M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ |
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
| M15 | Classic modules | Map, Book, Slideshow, Print, Web; smart collections, stacks, virtual copies, publish services, tethering | 25–40 | 🚧 (smart albums, stacks, virtual copies, compare/survey ✅; Map/Book/Slideshow/Print/Web ⬜) |
| M16 | 1.0 polish | preferences, shortcut editor, accessibility, localization, packaging (dmg/msi/AppImage/web), hardening | 10–20 | 🚧 (settings, keyboard shortcuts sheet, packaging basics ✅; accessibility, localisation ⬜) |
## Parity estimate (2026-10-02, evening update)
## Parity estimate (feature count updated 2026-10-05; effort estimate from 2026-10-02)
**By feature count** — from `docs/parity.md` (one row per Lightroom feature, menu item and shortcut; `cargo xtask
parity` prints this line on every run, so it stays current):
| Scope | Weighted completion | Rows | Morning |
| Scope | Weighted completion | Rows | 2026-10-02 |
|---|---:|---:|---:|
| P0 (core) | **98.7%** | 198 | 98.5% |
| P1 (important parity) | **94.4%** | 144 | 79.9% |
| P2 (later / AI / niche, incl. Classic modules) | **29.4%** | 158 | 11.7% |
| **All in-scope rows** | **75.6%** | 500 | 65.7% |
| P0 (core) | **98.5%** | 200 | 98.7% |
| P1 (important parity) | **95.9%** | 147 | 94.4% |
| P2 (later / AI / niche, incl. Classic modules) | **39.2%** | 158 | 29.4% |
| **All in-scope rows** | **79.2%** | 505 | 75.6% |
The 2026-10-05 count includes three new 🟡 rows that make quality gaps visible (camera colour calibration, verified
camera coverage, render fidelity), which is why P0 dipped slightly.
✅ counts 1, 🟡 ½, ⬜ 0; out-of-scope rows (cloud sharing, Adobe accounts…) are left out.
@@ -50,7 +115,7 @@ trip, slideshow, auto import…) in ≈ 12 h on 2026-10-02; four to six parallel
| Work package | Tracker rows | Agent-hours | Risk |
|---|---|---:|---|
| Remaining P0/P1 UI and library features (folder rename/move, keyword painter, people view…) | ≈ 8 | 5–10 | low |
| Raw codecs: CR3 (CRX), compressed NEF / ORF / RAF, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
| Raw codecs: CR3 (CRX), compressed ORF / RAF, NEF lossy-after-split, RW2 v4, HEIC/AVIF decode, JPEG XL DNG | LR-IMP-FORMATS | 40–80 | **high** — clean-room black-box analysis, no permissive specs |
| Lens-profile database of our own (calibration targets, fitting, data) | LR-EDIT-OPTICS-PROFILE | 15–30 | data collection |
| Video: playback, trim, edits, export (pure-Rust decode, ideally shared with FilmCraft) | R. Video | 20–40 | medium |
| AI: subject / sky / background / people / object masks, object-aware remove, AI denoise, super resolution, lens blur, people & faces, natural-language search, culling | ≈ 30 | 80–150 | **high** — permissively licensed weights, pure-Rust inference, maybe training |
@@ -86,24 +151,25 @@ The milestone estimates in the table above were made before work started and are
fallbacks first. No permissively licensed sky-segmentation or raw-denoise model was found — we may need to train our own.
- **Camera colour and lens data** is a data problem: we never use Adobe's matrices, DCPs or LCPs. DNG-embedded data first,
then our own calibration; long-tail camera/lens coverage grows over time.
- **Legal decisions pending:** whether GPL-licensed *prose* format descriptions (e.g. the public CR3 write-up) may be read
by a designated engineer to produce an internal spec; freedom-to-operate review for local Laplacian filters, PatchMatch and
HEVC (HEIC).
- **Look parity** with Adobe's default rendering is subjective tuning against our own reference targets.
- **Raw-format sources:** decided 2026-10-05: decoders are written from *prose* format descriptions (even ones
published alongside GPL code); decoder source is never read. Still open: freedom-to-operate review for local
Laplacian filters, PatchMatch and HEVC (HEIC).
- **Look parity** with Adobe's default rendering is tuned by eye today; the planned fidelity suite (LR-BEHAV-RENDER-FIDELITY)
turns it into measured comparisons against local-only Lightroom references.
## Raw format coverage and known gaps
Decoded (CC0 corpus from raw.pixls.us, `cargo xtask corpus --download`, `crates/raw/tests/corpus.rs`): DNG (uncompressed,
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92), NEF/NRW uncompressed, RAF uncompressed (Bayer and
LJ92, lossy JPEG / Smart Previews, Deflate, float, linear), CR2, ARW (uncompressed, ARW2, LJ92), NEF/NRW uncompressed and Huffman-compressed (lossless, lossy type 1/2, 12/14-bit), RAF uncompressed (Bayer and
X-Trans), RW2 packed 12/14-bit, PEF (uncompressed and Huffman), ORF uncompressed (16-bit and 12-bit packed). Every
supported container also yields its embedded JPEG preview (CR3 too), and the engine shows that preview for raw variants
it can't decode yet.
Not decoded yet — preview only (no permissively licensed description; black-box analysis incomplete):
- **Nikon Huffman NEF** (lossless / lossy). Black-box findings so far: maker note `0x0096` holds per-parity predictor
seeds (2048 for 14-bit lossless); ≈ 9 bits per pixel; the category-0 code word is a 6-bit rotation of `011111`. The
Huffman tables are not stored in the files, and an automatic table search (beam search over prefix codes scored by
smoothness and bit rate, validated on Pentax files whose tables *are* stored) did not converge.
- **Nikon "lossy after split" NEF** (non-zero split row in maker note `0x0096`, e.g. some D3400/D5000/D5200/D5500/D5600
files): rows above the split decode with the regular lossy table; from the split row on a different code is used
that our black-box analysis has not recovered yet (none of the four regular tables fits, also not with byte
alignment or reset predictors at the split row). The other Nikon Huffman variants are decoded (`crates/raw/src/vendor/nefc.rs` documents the analysis).
- **Panasonic RW2 raw format 4** (quantised): the block layout is known (0x4000-byte chunks rotated by 0x1ff8; 128-bit
blocks of two 12-bit seeds plus four groups of a 2-bit scale and three 8-bit codes; scales 0/1 are ×1/×2 differences
from the same-colour pixel two to the left), the reconstruction rule for scales 2/3 is not established.
@@ -119,3 +185,17 @@ own chart-based calibration (M11.4). Adobe matrices are never used.
- 2026-09-30 (later): app running with the full Lightroom-style UI; pipeline v0; DNG/CR2/ARW; README showcase. ≈ 8 h elapsed.
- 2026-10-02: parity estimate added (xtask parity prints weighted completion); milestone statuses refreshed.
- 2026-10-01: RAW II formats: RAF (uncompressed Bayer + X-Trans), RW2 (packed), PEF (incl. Huffman), ORF (uncompressed); embedded previews for every container incl. CR3; raw corpus test with 37 CC0 samples.
- 2026-10-03 – 10-05: community PRs (#42–#51, #60) and 25+ issues landed: compressed NEF, Move import, folder templates,
Local roots and cleanup, 85k-photo grid and catalog performance, background compaction, failed-save errors, GPU
export hardening, clippy 1.99. Added the honest *Where we stand* assessment and *Where we're going* priorities; added
tracker rows for camera colour, camera coverage and render fidelity.
## Japanese interface and text watermarks
English/Japanese interface language is persisted in UI state. Core menus have Japanese
translations; untranslated panels and dialogs retain English. BIZ UDMincho is bundled
under OFL for Japanese glyph coverage without system fonts. Text watermarks now accept
`vertical: true` in export JSON/presets and expose an orientation selector. Japanese
characters stay upright in top-to-bottom columns, with newlines starting columns to the
left. This is basic lettering, without tate-chu-yoko, ruby, kinsoku, or general vertical
OpenType shaping. The same coverage renderer serves 8/16/32-bit exports.
+27 -20
View File
@@ -91,6 +91,17 @@ USAGE:
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;
/// Why `--library DIR` can't be opened; for a library open in another program, how to work with
/// that one instead (issue #99).
fn library_error(dir: &str, e: lightcraft_engine::EngineError) -> String {
match e {
lightcraft_engine::EngineError::LibraryInUse(why) => format!(
"{dir}: {why}\nTo work with the library while the app has it open, start the app with `--control PORT` and use `lightcraft-cli mcp --connect 127.0.0.1:PORT`."
),
e => format!("{dir}: {e}"),
}
}
fn main() -> ExitCode {
// `--features dhat-heap`: count allocations; the profile is written when `_heap` drops
// (LIGHTCRAFT_DHAT_FILE, default dhat-heap.json).
@@ -182,7 +193,7 @@ fn mcp(args: &[String]) -> Result<(), String> {
let mut h = match &library {
Some(dir) => {
let mut h = Headless::default();
let r = h.session.open_library(dir, demo).map_err(|e| e.to_string())?;
let r = h.session.open_library(dir, demo).map_err(|e| library_error(dir, e))?;
eprintln!("lightcraft-cli mcp: opened library {dir} ({r:?})");
h
}
@@ -419,7 +430,7 @@ fn run(args: &[String]) -> Result<(), String> {
let mut h = match &library {
Some(dir) => {
let mut h = Headless::default();
h.session.open_library(dir, demo).map_err(|e| e.to_string())?;
h.session.open_library(dir, demo).map_err(|e| library_error(dir, e))?;
h
}
None if demo => Headless::demo(),
@@ -533,11 +544,17 @@ fn render(args: &[String]) -> Result<(), String> {
.ok_or_else(|| format!("{output}: unknown extension (use .jpg .png .tif .webp .avif .dng or --opt format=…)"))?;
}
let e = export_photo(&mut s, lightcraft_engine::catalog::PhotoId(id), &o, 1)?;
std::fs::write(&output, &e.bytes).map_err(|err| format!("{output}: {err}"))?;
for (ext, bytes) in &e.sidecars {
let sc = Path::new(&output).with_extension(ext);
std::fs::write(&sc, bytes).map_err(|err| format!("{}: {err}", sc.display()))?;
eprintln!("lightcraft-cli: wrote {}", sc.display());
// never over the input (or its sidecar), however it is spelled: `render IMG.jpg -o IMG.jpg`
let sidecars: Vec<(String, &Vec<u8>)> =
e.sidecars.iter().map(|(ext, bytes)| (Path::new(&output).with_extension(ext).to_string_lossy().to_string(), bytes)).collect();
let guard = s.original_guard();
for p in std::iter::once(&output).chain(sidecars.iter().map(|(p, _)| p)) {
guard.check(Path::new(p)).map_err(|err| format!("render: {err}"))?;
}
lightcraft_engine::export::write_file(&output, &e.bytes)?;
for (sc, bytes) in &sidecars {
lightcraft_engine::export::write_file(sc, bytes)?;
eprintln!("lightcraft-cli: wrote {sc}");
}
eprintln!("lightcraft-cli: wrote {output} ({}×{})", e.width, e.height);
Ok(())
@@ -580,7 +597,7 @@ fn snapshot(args: &[String]) -> Result<(), String> {
let mut session = match &library {
Some(dir) => {
let mut s = Session::new().with_fs();
s.open_library(dir, false).map_err(|e| format!("{dir}: {e}"))?;
s.open_library(dir, false).map_err(|e| library_error(dir, e))?;
s
}
None if files.is_empty() => Session::with_demo().with_fs(),
@@ -593,18 +610,8 @@ fn snapshot(args: &[String]) -> Result<(), String> {
eprintln!("lightcraft-cli snapshot: imported {n} files in {:.0} ms", ti.elapsed().as_secs_f64() * 1e3);
}
let services = lightcraft_ui_egui::Services {
write_shared: Some(std::sync::Arc::new(|p: &str, b: &[u8]| {
if let Some(dir) = Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| format!("{p}: {e}"))
})),
write: Some(Box::new(|p: &str, b: &[u8]| {
if let Some(dir) = Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| format!("{p}: {e}"))
})),
write_shared: Some(std::sync::Arc::new(lightcraft_engine::export::write_file)),
write: Some(Box::new(lightcraft_engine::export::write_file)),
png: Some(Box::new(|img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
})),
+46
View File
@@ -86,6 +86,24 @@ fn render_subcommand() {
assert!(String::from_utf8_lossy(&o.stderr).contains("unknown control"));
}
/// Issue #93: `render IMG -o IMG` replaced its own input with the render.
#[test]
fn render_refuses_to_overwrite_its_input() {
let input = tmp("self-in.png");
gradient_png(&input);
let before = std::fs::read(&input).unwrap();
let dir = input.parent().unwrap();
std::fs::create_dir_all(dir.join("sub")).unwrap();
let other_spelling = dir.join("sub/../self-in.png");
for out in [&input, &other_spelling] {
let o =
Command::new(BIN).args(["render", input.to_str().unwrap(), "-o", out.to_str().unwrap(), "--set", "light.exposure=1"]).output().unwrap();
assert!(!o.status.success());
assert!(String::from_utf8_lossy(&o.stderr).contains("never writes over an original"), "{}", String::from_utf8_lossy(&o.stderr));
}
assert_eq!(std::fs::read(&input).unwrap(), before, "the input is untouched");
}
#[test]
fn render_export_options() {
let input = tmp("o-in.png");
@@ -242,3 +260,31 @@ fn devices_are_listed_and_imported_from() {
let line: Value = serde_json::from_slice(o.stdout.split(|b| *b == b'\n').next().unwrap()).unwrap();
assert_eq!(line["result"]["candidates"].as_array().map(Vec::len), Some(1), "{line}");
}
/// Issue #99: a library open in one process (here `mcp --library`) is refused by a second one,
/// with who has it and how to drive the running app instead; free again once the first exits.
#[test]
fn a_library_open_in_another_process_is_refused() {
let lib = tmp("locked-lib");
let _ = std::fs::remove_dir_all(&lib);
let lib_s = lib.to_str().unwrap();
let mut holder =
Command::new(BIN).args(["mcp", "--library", lib_s]).stdin(Stdio::piped()).stdout(Stdio::null()).stderr(Stdio::piped()).spawn().unwrap();
let mut err = BufReader::new(holder.stderr.take().unwrap());
let mut line = String::new();
while !line.contains("opened library") {
line.clear();
assert!(err.read_line(&mut line).unwrap() > 0, "mcp exited before opening the library");
}
let (ok, _, stderr) = run_cli(&["--library", lib_s, "library.info"], None);
assert!(!ok);
assert!(stderr.contains("already open in lightcraft-cli") && stderr.contains(&format!("process {}", holder.id())), "{stderr}");
assert!(stderr.contains("mcp --connect"), "{stderr}");
drop(holder.stdin.take()); // EOF: the server exits and lets go of the library
assert!(holder.wait().unwrap().success());
let (ok, lines, stderr) = run_cli(&["--library", lib_s, "library.info"], None);
assert!(ok, "{stderr}");
assert_eq!(lines[0]["ok"], true);
let _ = std::fs::remove_dir_all(&lib);
}
+12 -2
View File
@@ -295,6 +295,7 @@ impl WebApp {
files: Box::new(files.store()),
on_disk: false,
};
let mut problem = None;
match session.open_library_in(stores, true).cloned() {
Ok(r) => log::info!(
"lightcraft: library {} in {:.0} ms ({} photos; snapshot seq {}, {} ops replayed)",
@@ -305,9 +306,17 @@ impl WebApp {
r.replayed
),
Err(e) => {
log::error!("opening the library failed: {e}; starting a temporary one");
session = Session::with_demo();
// never a silent demo session (issue #100): an empty one, and the window says why
log::error!("opening the library failed: {e}");
session = Session::new();
originals.install(&mut session);
problem = Some(lightcraft_ui_egui::panels::library_problem::LibraryProblem {
can_retry: false,
..lightcraft_ui_egui::panels::library_problem::LibraryProblem::new(
"the browser's storage for this page",
format!("{e}. Reload the page to try again."),
)
});
}
}
// previews are large (≤ 2560 px, f32): keep few in a 32-bit address space
@@ -318,6 +327,7 @@ impl WebApp {
app.ui = ui;
}
app.ui = app.ui.sanitized();
app.library_problem = problem;
let n = opts.workers.unwrap_or_else(|| {
let cores = window().map_or(1, |w| w.navigator().hardware_concurrency() as usize);
cores.saturating_sub(1).clamp(1, 4)
+237 -32
View File
@@ -1,14 +1,28 @@
//! Loopback JSON-lines control server: one request per line, one reply per line.
//! This is the transport the MCP server (`lightcraft-cli mcp --connect`) wraps.
//!
//! The server only ever reads requests: a line that is not a JSON object with a string `method`
//! (an HTTP request line from a browser's cross-origin `fetch`, a stray `nc`, binary junk) gets
//! one error reply and the connection is closed, so nothing after it is executed. Lines are
//! bounded ([`MAX_LINE`]) and so is the number of open connections ([`MAX_CONNECTIONS`]); junk
//! never reaches the UI thread. See `docs/control-protocol.md`.
use std::io::{BufRead, BufReader, Write};
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{TcpListener, TcpStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::mpsc::{Receiver, Sender, channel};
use std::time::Duration;
use lightcraft_ui_egui::ControlRequest;
use serde_json::{Value, json};
/// Longest accepted request line (bytes, without the newline). Requests are small JSON objects;
/// anything longer is refused and the connection closed.
pub const MAX_LINE: usize = 4 * 1024 * 1024;
/// Connections served at once; further ones get an error line and are closed.
pub const MAX_CONNECTIONS: usize = 16;
pub fn start(port: u16, ctx: egui::Context) -> Receiver<ControlRequest> {
let (tx, rx) = channel::<ControlRequest>();
let listener = match TcpListener::bind(("127.0.0.1", port)) {
@@ -19,44 +33,235 @@ pub fn start(port: u16, ctx: egui::Context) -> Receiver<ControlRequest> {
}
};
eprintln!("lightcraft: control server listening on 127.0.0.1:{port}");
std::thread::spawn(move || {
for stream in listener.incoming().flatten() {
let tx = tx.clone();
let ctx = ctx.clone();
std::thread::spawn(move || serve(stream, tx, ctx));
}
});
std::thread::spawn(move || accept_loop(listener, tx, ctx));
rx
}
fn serve(stream: TcpStream, tx: Sender<ControlRequest>, ctx: egui::Context) {
let Ok(read) = stream.try_clone() else { return };
let mut out = stream;
for line in BufReader::new(read).lines() {
let Ok(line) = line else { break };
if line.trim().is_empty() {
fn accept_loop(listener: TcpListener, tx: Sender<ControlRequest>, ctx: egui::Context) {
let open = Arc::new(AtomicUsize::new(0));
for mut stream in listener.incoming().flatten() {
if open.fetch_add(1, Ordering::SeqCst) >= MAX_CONNECTIONS {
open.fetch_sub(1, Ordering::SeqCst);
let _ = writeln!(stream, "{}", json!({"ok": false, "error": "too many control connections"}));
continue;
}
let reply = match serde_json::from_str::<Value>(&line) {
Ok(msg) => {
let id = msg.get("id").cloned().unwrap_or(Value::Null);
let method = msg.get("method").and_then(Value::as_str).unwrap_or("").to_string();
let params = msg.get("params").cloned().unwrap_or(json!({}));
let (req, rrx) = ControlRequest::new(method, params);
if tx.send(req).is_err() {
break;
}
ctx.request_repaint();
let mut r = rrx.recv_timeout(Duration::from_secs(60)).unwrap_or_else(|_| json!({"ok": false, "error": "timeout"}));
if let Some(o) = r.as_object_mut() {
o.insert("id".into(), id);
let tx = tx.clone();
let ctx = ctx.clone();
let open = Arc::clone(&open);
std::thread::spawn(move || {
serve(stream, &tx, &ctx);
open.fetch_sub(1, Ordering::SeqCst);
});
}
}
fn serve(stream: TcpStream, tx: &Sender<ControlRequest>, ctx: &egui::Context) {
let Ok(read) = stream.try_clone() else { return };
serve_lines(read, stream, |method, params| {
let (req, rrx) = ControlRequest::new(method, params);
tx.send(req).ok()?;
ctx.request_repaint();
Some(rrx.recv_timeout(Duration::from_secs(60)).unwrap_or_else(|_| json!({"ok": false, "error": "timeout"})))
});
}
/// One parsed line.
enum Line {
Request {
id: Value,
method: String,
params: Value,
},
Blank,
/// Not a request: reply with this error and close the connection.
Reject(String),
}
fn parse_line(bytes: &[u8]) -> Line {
let Ok(text) = std::str::from_utf8(bytes) else {
return Line::Reject("not a control request (invalid UTF-8); closing the connection".into());
};
let text = text.trim();
if text.is_empty() {
return Line::Blank;
}
let msg = match serde_json::from_str::<Value>(text) {
Ok(v) => v,
Err(e) => return Line::Reject(format!("bad JSON ({e}); expected one JSON request object per line; closing the connection")),
};
let Some(obj) = msg.as_object() else {
return Line::Reject("not a control request (expected a JSON object); closing the connection".into());
};
let Some(method) = obj.get("method").and_then(Value::as_str) else {
return Line::Reject("not a control request (missing string `method`); closing the connection".into());
};
Line::Request {
id: obj.get("id").cloned().unwrap_or(Value::Null),
method: method.to_string(),
params: obj.get("params").cloned().unwrap_or(json!({})),
}
}
/// Read one `\n`-terminated line of at most `max` bytes into `buf` (newline stripped).
/// `Ok(false)` at end of stream; `Err` when the line is too long or the read fails.
fn read_bounded_line(reader: &mut impl BufRead, buf: &mut Vec<u8>, max: usize) -> Result<bool, String> {
buf.clear();
// `take` bounds the bytes buffered for one line: a peer can't make us allocate more than `max`.
let limit = u64::try_from(max).unwrap_or(u64::MAX).saturating_add(1);
let n = reader.take(limit).read_until(b'\n', buf).map_err(|e| e.to_string())?;
if n == 0 {
return Ok(false);
}
if buf.last() == Some(&b'\n') {
buf.pop();
if buf.last() == Some(&b'\r') {
buf.pop();
}
return Ok(true);
}
if buf.len() > max {
return Err(format!("request line longer than {max} bytes; closing the connection"));
}
// Last line without a newline, then end of stream.
Ok(true)
}
/// Serve requests from `read`, answering on `out`, until the peer closes, sends something that is
/// not a request, or `handle` returns `None` (app gone). `handle` gets `(method, params)` and
/// returns the reply object (the request's `id` is added here).
fn serve_lines<R: Read, W: Write>(read: R, mut out: W, mut handle: impl FnMut(String, Value) -> Option<Value>) {
let mut reader = BufReader::new(read);
let mut buf = Vec::new();
loop {
let reject = match read_bounded_line(&mut reader, &mut buf, MAX_LINE) {
Ok(false) => return,
Err(e) => e,
Ok(true) => match parse_line(&buf) {
Line::Blank => continue,
Line::Reject(e) => e,
Line::Request { id, method, params } => {
let Some(mut r) = handle(method, params) else { return };
if let Some(o) = r.as_object_mut() {
o.insert("id".into(), id);
}
if writeln!(out, "{r}").and_then(|()| out.flush()).is_err() {
return;
}
continue;
}
r
}
Err(e) => json!({"ok": false, "error": format!("bad JSON: {e}")}),
},
};
if writeln!(out, "{reply}").is_err() {
break;
let _ = writeln!(out, "{}", json!({"ok": false, "error": reject})).and_then(|()| out.flush());
return;
}
}
#[cfg(test)]
mod tests {
use super::*;
fn run(input: &[u8]) -> (Vec<Value>, Vec<String>) {
let mut out = Vec::new();
let mut calls = Vec::new();
serve_lines(input, &mut out, |m, _p| {
calls.push(m);
Some(json!({"ok": true, "result": null}))
});
let replies = String::from_utf8(out).unwrap().lines().map(|l| serde_json::from_str::<Value>(l).unwrap()).collect();
(replies, calls)
}
#[test]
fn requests_are_answered_in_order_with_ids() {
let (r, calls) = run(b"{\"id\":1,\"method\":\"a\"}\n\n{\"id\":2,\"method\":\"b\",\"params\":{}}\r\n{\"method\":\"c\"}");
assert_eq!(calls, ["a", "b", "c"]);
assert_eq!(r.len(), 3);
assert_eq!(r[0]["id"], 1);
assert_eq!(r[1]["id"], 2);
assert_eq!(r[2]["id"], Value::Null);
}
#[test]
fn http_request_closes_before_the_body_runs() {
// A browser's cross-origin `fetch` POST: the request line is rejected and the connection
// closed, so the JSON body is never executed.
let http = b"POST / HTTP/1.1\r\nHost: 127.0.0.1:7980\r\nContent-Type: text/plain\r\n\r\n\
{\"id\":1,\"method\":\"engine.execute\",\"params\":{\"command\":\"folder.move\"}}\n";
let (r, calls) = run(http);
assert!(calls.is_empty(), "nothing may run: {calls:?}");
assert_eq!(r.len(), 1);
assert_eq!(r[0]["ok"], false);
assert!(r[0]["error"].as_str().unwrap().contains("closing the connection"));
}
#[test]
fn non_request_json_closes_the_connection() {
for junk in [
&b"[1,2]\n{\"method\":\"a\"}\n"[..],
b"{\"id\":1}\n{\"method\":\"a\"}\n",
b"{\"method\":5}\n{\"method\":\"a\"}\n",
b"\xff\xfe\n{\"method\":\"a\"}\n",
] {
let (r, calls) = run(junk);
assert!(calls.is_empty());
assert_eq!(r.len(), 1);
assert_eq!(r[0]["ok"], false);
}
}
#[test]
fn a_request_before_junk_still_runs() {
let (r, calls) = run(b"{\"method\":\"a\"}\nGET / HTTP/1.1\n{\"method\":\"b\"}\n");
assert_eq!(calls, ["a"]);
assert_eq!(r.len(), 2);
assert_eq!(r[1]["ok"], false);
}
#[test]
fn overlong_line_is_refused_without_buffering_it_all() {
// An endless line without a newline: reading stops after MAX_LINE + 1 bytes.
let endless = std::io::repeat(b'x');
let mut out = Vec::new();
let mut ran = false;
serve_lines(endless, &mut out, |_, _| {
ran = true;
None
});
assert!(!ran);
let reply: Value = serde_json::from_slice(out.strip_suffix(b"\n").unwrap()).unwrap();
assert!(reply["error"].as_str().unwrap().contains("longer than"));
}
#[test]
fn line_at_the_limit_is_accepted() {
let mut buf = Vec::new();
let line = vec![b'a'; 10];
let mut input = line.clone();
input.push(b'\n');
assert_eq!(read_bounded_line(&mut &input[..], &mut buf, 10), Ok(true));
assert_eq!(buf, line);
assert!(read_bounded_line(&mut &[b'a'; 11][..], &mut buf, 10).is_err());
}
#[test]
fn tcp_connection_is_closed_after_junk() {
// End to end over a real loopback socket (the transport the app uses).
let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap();
let addr = listener.local_addr().unwrap();
let server = std::thread::spawn(move || {
let (s, _) = listener.accept().unwrap();
let read = s.try_clone().unwrap();
serve_lines(read, s, |_, _| Some(json!({"ok": true})));
});
let mut c = TcpStream::connect(addr).unwrap();
c.set_read_timeout(Some(Duration::from_secs(10))).unwrap();
c.write_all(b"{\"id\":7,\"method\":\"x\"}\nhello\n{\"method\":\"y\"}\n").unwrap();
let mut text = String::new();
c.read_to_string(&mut text).unwrap(); // returns at EOF: the server closed the connection
server.join().unwrap();
let lines: Vec<&str> = text.lines().collect();
assert_eq!(lines.len(), 2, "{text}");
assert!(lines[0].contains("\"id\":7"));
assert!(lines[1].contains("closing the connection"));
}
}
+200 -40
View File
@@ -7,7 +7,9 @@
//! `~/Pictures/LightCraft Library`; a new library starts with the
//! procedural demo photos unless `--no-demo` or files are given. Files and folders on the command
//! line are imported (duplicates are skipped). `--memory` runs an in-memory session that writes
//! nothing (demo photos unless files are given; used by the README showcase scripts).
//! nothing (demo photos unless files are given; used by the README showcase scripts). A library
//! that can't be opened is never replaced silently: the window says why and asks what to do
//! (`lightcraft_ui_egui::panels::library_problem`).
//!
//! `--control <port>` (or `LIGHTCRAFT_CONTROL_PORT`) starts a localhost JSON-lines control server:
//! `{"id":1,"method":"ui.inspect","params":{}}` → `{"id":1,"ok":true,"result":…}`.
@@ -26,6 +28,7 @@ mod control_server;
mod native_menu;
use lightcraft_engine::Session;
use lightcraft_ui_egui::panels::library_problem::LibraryProblem;
use lightcraft_ui_egui::{LightcraftApp, Services, UiState};
/// Reverse-DNS app id: Wayland app id, `.desktop` file and hicolor icon name.
@@ -41,15 +44,16 @@ fn app_icon() -> egui::IconData {
eframe::icon_data::from_png_bytes(png).unwrap_or_default()
}
struct App(LightcraftApp, #[cfg(target_os = "macos")] Option<native_menu::NativeMenu>);
struct App(LightcraftApp, PrefsWriter, #[cfg(target_os = "macos")] Option<native_menu::NativeMenu>);
impl eframe::App for App {
fn logic(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
#[cfg(target_os = "macos")]
if let Some(m) = self.1.as_mut() {
if let Some(m) = self.2.as_mut() {
m.update(&mut self.0, ctx);
}
self.0.logic(ctx);
self.1.tick(&mut self.0, ctx);
}
fn raw_input_hook(&mut self, _ctx: &egui::Context, raw: &mut egui::RawInput) {
self.0.raw_input_hook(raw);
@@ -58,7 +62,9 @@ impl eframe::App for App {
self.0.ui(ui);
}
fn on_exit(&mut self) {
save_prefs(&self.0);
if let Err(e) = self.1.save(&self.0) {
eprintln!("lightcraft: {e}");
}
if let Err(e) = self.0.session.close_library() {
eprintln!("lightcraft: saving the library failed: {e}");
}
@@ -78,23 +84,118 @@ fn config_dir() -> Option<std::path::PathBuf> {
}
}
/// The saved UI state and app settings (`<config>/ui.json`), if any.
fn load_prefs() -> Option<UiState> {
/// The saved UI state and app settings (`<config>/ui.json`), if any, and a warning for the user
/// when the file exists but can't be used (issue #103): a damaged file is kept as
/// `ui.json.corrupt-<unix time>` first, so the next save can't lose the library location in it;
/// one that can't be read at all is not written this session (`keep_file`).
fn load_prefs() -> (Option<UiState>, Option<String>, bool) {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return None;
return (None, None, false);
}
let bytes = std::fs::read(config_dir()?.join("ui.json")).ok()?;
serde_json::from_slice::<UiState>(&bytes).ok().map(UiState::sanitized)
let Some(path) = config_dir().map(|d| d.join("ui.json")) else { return (None, None, false) };
load_prefs_at(&path)
}
fn save_prefs(app: &LightcraftApp) {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return;
fn load_prefs_at(path: &std::path::Path) -> (Option<UiState>, Option<String>, bool) {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return (None, None, false),
Err(e) => {
let msg = format!("The app settings ({}) couldn't be read: {e}. Defaults are used, and the file isn't overwritten.", path.display());
return (None, Some(msg), true);
}
};
match serde_json::from_slice::<UiState>(&bytes) {
Ok(ui) => (Some(ui.sanitized()), None, false),
Err(e) => {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let keep = path.with_file_name(format!("ui.json.corrupt-{secs}"));
let kept = std::fs::rename(path, &keep);
let what = match &kept {
Ok(()) => format!("it was kept as {}", keep.display()),
Err(r) => format!("it couldn't be set aside ({r})"),
};
let msg = format!(
"The app settings ({}) are damaged ({e}); {what}. Defaults are used — reopen your library with Settings → Open Library… if it isn't shown.",
path.display()
);
(None, Some(msg), kept.is_err())
}
}
if let Some(d) = config_dir() {
let _ = std::fs::create_dir_all(&d);
if let Ok(bytes) = serde_json::to_vec_pretty(&app.ui) {
let _ = std::fs::write(d.join("ui.json"), bytes);
}
/// Write `bytes` to `path` atomically: a temp file, fsynced, renamed over it (a crash or a full
/// disk leaves the old file or the new one, never a truncated one).
fn write_atomic(path: &std::path::Path, bytes: &[u8]) -> std::io::Result<()> {
use std::io::Write;
let tmp = path.with_extension("json.tmp");
let written = (|| {
let mut f = std::fs::File::create(&tmp)?;
f.write_all(bytes)?;
f.sync_all()
})();
if let Err(e) = written {
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
std::fs::rename(&tmp, path)?;
if let Some(dir) = path.parent()
&& let Ok(d) = std::fs::File::open(dir)
{
let _ = d.sync_all();
}
Ok(())
}
/// Saves `ui.json` (app settings, incl. the library to open at launch): as soon as the library
/// changes, every few seconds when anything else changed, and at exit.
#[derive(Default)]
struct PrefsWriter {
written: Vec<u8>,
library: String,
checked: f64,
/// The last write failed (reported once until a write works).
failing: bool,
/// `ui.json` couldn't be read at launch: never overwrite it this session.
keep_file: bool,
}
impl PrefsWriter {
fn save(&mut self, app: &LightcraftApp) -> Result<(), String> {
if self.keep_file || std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return Ok(());
}
let Some(d) = config_dir() else { return Ok(()) };
let bytes = serde_json::to_vec_pretty(&app.ui).map_err(|e| e.to_string())?;
if bytes == self.written {
return Ok(());
}
std::fs::create_dir_all(&d)
.and_then(|()| write_atomic(&d.join("ui.json"), &bytes))
.map_err(|e| format!("saving the app settings failed: {e}"))?;
self.written = bytes;
self.library = app.ui.settings.library_path.clone();
Ok(())
}
fn tick(&mut self, app: &mut LightcraftApp, ctx: &egui::Context) {
let now = ctx.input(|i| i.time);
let moved = app.ui.settings.library_path != self.library;
if !moved && now - self.checked < 3.0 {
return;
}
self.checked = now;
match self.save(app) {
Ok(()) => self.failing = false,
Err(e) => {
eprintln!("lightcraft: {e}");
if !self.failing {
app.notices.push(format!("{e}. LightCraft keeps trying."));
}
self.failing = true;
// don't retry every frame
self.library = app.ui.settings.library_path.clone();
}
}
}
}
@@ -209,33 +310,28 @@ fn services() -> Services {
.save_file()
.map(|p| p.to_string_lossy().to_string())
})),
write_shared: Some(std::sync::Arc::new(|p: &str, b: &[u8]| {
if let Some(dir) = std::path::Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| e.to_string())
})),
write: Some(Box::new(|p: &str, b: &[u8]| {
if let Some(dir) = std::path::Path::new(p).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| e.to_string())?;
}
std::fs::write(p, b).map_err(|e| e.to_string())
})),
// atomic (temp file + sync + rename): a failed write never leaves a truncated file
write_shared: Some(std::sync::Arc::new(lightcraft_engine::export::write_file)),
write: Some(Box::new(lightcraft_engine::export::write_file)),
png: Some(Box::new(|img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
})),
}
}
/// The persistent library session (or an in-memory one with `--memory` / if the library can't open).
fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: bool) -> Session {
let fallback = || if seed_demo { Session::with_demo() } else { Session::new() }.with_fs().with_system_clock();
/// The persistent library session, or with `--memory` an in-memory one (demo photos).
///
/// If the library can't be opened the session is empty and in memory — never seeded with the
/// demo photos, never written anywhere — and the problem is returned: the window then says so
/// and offers Try Again / Choose Another Library… / Continue Without Saving / Quit (issue #100).
fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: bool) -> (Session, Option<LibraryProblem>) {
if in_memory {
return fallback();
return (if seed_demo { Session::with_demo() } else { Session::new() }.with_fs().with_system_clock(), None);
}
let unopened = || Session::new().with_fs().with_system_clock();
let Some(dir) = dir else {
eprintln!("lightcraft: no library location (set --library or LIGHTCRAFT_LIBRARY); running in memory");
return fallback();
eprintln!("lightcraft: no library location (set --library or LIGHTCRAFT_LIBRARY)");
return (unopened(), Some(LibraryProblem::new("", "There is no home folder to keep the library in. Choose a folder for it.")));
};
let t0 = std::time::Instant::now();
let mut s = Session::new().with_fs().with_system_clock();
@@ -249,11 +345,11 @@ fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: boo
if torn > 0 { ", torn tail repaired" } else { "" },
t0.elapsed().as_secs_f64() * 1000.0
);
s
(s, None)
}
Err(e) => {
eprintln!("lightcraft: can't open library {}: {e}; running in memory", dir.display());
fallback()
eprintln!("lightcraft: can't open library {}: {e}", dir.display());
(unopened(), Some(LibraryProblem::new(dir.to_string_lossy(), e.to_string())))
}
}
}
@@ -302,7 +398,7 @@ fn main() -> eframe::Result {
_ => files.push(a),
}
}
let prefs = load_prefs();
let (prefs, prefs_warning, keep_prefs_file) = load_prefs();
// --library, else the library last opened from Settings, else the default location
let library_dir = library_dir.or_else(|| {
prefs
@@ -330,17 +426,29 @@ fn main() -> eframe::Result {
"LightCraft",
options,
Box::new(move |cc| {
let session = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let (session, problem) = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let mut app = LightcraftApp::new(session, services());
if let Some(ui) = prefs {
app.ui = ui;
}
app.integrated_titlebar = cfg!(target_os = "macos");
app.notices.extend(prefs_warning);
// what's on disk now: only changes are written
let writer = PrefsWriter {
written: serde_json::to_vec_pretty(&app.ui).unwrap_or_default(),
library: app.ui.settings.library_path.clone(),
keep_file: keep_prefs_file,
..Default::default()
};
if let Some(port) = control_port {
let rx = control_server::start(port, cc.egui_ctx.clone());
app = app.with_control(rx);
}
if !files.is_empty() {
if let Some(mut p) = problem {
// imported once the user has chosen where (into the library, or the temporary session)
p.pending_import = files;
app.library_problem = Some(p);
} else if !files.is_empty() {
let _ = app.run("library.import", serde_json::json!({"paths": files}));
app.ui.view = lightcraft_ui_egui::state::ViewMode::PhotoGrid;
}
@@ -350,9 +458,61 @@ fn main() -> eframe::Result {
let menu = (std::env::var_os("LIGHTCRAFT_NO_NATIVE_MENU").is_none()).then(|| native_menu::NativeMenu::install(&mut app, &cc.egui_ctx));
Ok(Box::new(App(
app,
writer,
#[cfg(target_os = "macos")]
menu,
)))
}),
)
}
#[cfg(test)]
#[allow(clippy::unwrap_used)]
mod tests {
use super::*;
fn dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-app-prefs-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
/// Issue #103: a damaged ui.json (which holds the library location) is kept aside and
/// reported, not silently replaced by the defaults.
#[test]
fn damaged_ui_json_is_kept_and_reported() {
let d = dir("damaged");
let path = d.join("ui.json");
std::fs::write(&path, br#"{"settings": {"library_path": "/Volumes/Photos/Lib"#).unwrap();
let (ui, warning, keep) = load_prefs_at(&path);
assert!(ui.is_none() && !keep);
let warning = warning.unwrap();
assert!(warning.contains("damaged") && warning.contains("ui.json.corrupt-"), "{warning}");
assert!(!path.exists());
let kept: Vec<_> = std::fs::read_dir(&d).unwrap().map(|e| e.unwrap().file_name().to_string_lossy().to_string()).collect();
assert!(kept.len() == 1 && kept[0].starts_with("ui.json.corrupt-"), "{kept:?}");
// a missing file is just "no settings yet"
assert_eq!(load_prefs_at(&path).1, None);
let _ = std::fs::remove_dir_all(&d);
}
#[test]
fn ui_json_is_written_atomically() {
let d = dir("atomic");
let path = d.join("ui.json");
std::fs::write(&path, b"old").unwrap();
write_atomic(&path, b"{\"new\": true}").unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"{\"new\": true}");
assert!(!d.join("ui.json.tmp").exists());
// a failed write leaves the old file
let missing = d.join("gone").join("ui.json");
assert!(write_atomic(&missing, b"x").is_err());
assert_eq!(std::fs::read(&path).unwrap(), b"{\"new\": true}");
let ui = UiState::default();
write_atomic(&path, &serde_json::to_vec_pretty(&ui).unwrap()).unwrap();
let (loaded, warning, _) = load_prefs_at(&path);
assert!(loaded.is_some() && warning.is_none());
let _ = std::fs::remove_dir_all(&d);
}
}
+5 -5
View File
@@ -211,7 +211,7 @@ impl NativeMenu {
// the application menu
let app_menu = Submenu::new("LightCraft", true);
let about = MenuItem::with_id("app.about", "About LightCraft", true, None);
let settings = MenuItem::with_id(SETTINGS, "Settings…", true, accelerator(SETTINGS_KEY));
let settings = MenuItem::with_id(SETTINGS, app.ui.language.tr("Settings…"), true, accelerator(SETTINGS_KEY));
let quit = MenuItem::with_id(QUIT, "Quit LightCraft", true, accelerator("Cmd+Q"));
let _ = app_menu.append_items(&[
&about,
@@ -230,7 +230,7 @@ impl NativeMenu {
for (title, nodes) in &bar {
let sub = Submenu::new(title, true);
if title == "Window" {
if title == app.ui.language.tr("Window") {
let _ = sub.append_items(&[
&PredefinedMenuItem::minimize(None),
&PredefinedMenuItem::maximize(Some("Zoom")),
@@ -245,15 +245,15 @@ impl NativeMenu {
.collect();
let nodes = tidy_separators(nodes);
self.append_nodes(&sub, &nodes);
if title == "File" {
if title == app.ui.language.tr("File") {
// ⌘W, the system's own item
let _ = sub.append_items(&[&PredefinedMenuItem::separator(), &PredefinedMenuItem::close_window(None)]);
}
if title == "Window" {
if title == app.ui.language.tr("Window") {
let _ = sub.append_items(&[&PredefinedMenuItem::separator(), &PredefinedMenuItem::bring_all_to_front(None)]);
sub.set_as_windows_menu_for_nsapp();
}
if title == "Help" {
if title == app.ui.language.tr("Help") {
sub.set_as_help_menu_for_nsapp();
}
let _ = self.menu.append(&sub);
+2
View File
@@ -9,6 +9,8 @@ covers a whole directory), or if a licence file referenced in the Licence column
| Path | Asset | Creator | Source | Licence | Added | Modifications |
|---|---|---|---|---|---|---|
| `assets/fonts/Inter-*.ttf` (Regular, Medium, SemiBold) | Inter | Rasmus Andersson | https://github.com/rsms/inter | SIL Open Font License 1.1 (`assets/fonts/OFL-Inter.txt`) | 2026-09-30 | none |
| `assets/fonts/BIZUDMincho-Regular.ttf` | BIZ UDMincho Regular | The BIZ UDMincho Project Authors (Morisawa Inc., design TypeBank Co., Ltd.) | https://github.com/google/fonts/tree/63833b7d10bb1f86a8f0b91cba2d3ae1f68d1aa3/ofl/bizudmincho | OFL-1.1 (`assets/fonts/OFL-BIZUDMincho.txt`) | 2026-10-05 | none |
| `assets/fonts/OFL-BIZUDMincho.txt` | BIZ UDMincho licence | The BIZ UDMincho Project Authors | https://github.com/google/fonts/blob/63833b7d10bb1f86a8f0b91cba2d3ae1f68d1aa3/ofl/bizudmincho/OFL.txt | OFL-1.1 | 2026-10-05 | none |
| `assets/app-icon/` (`lightcraft.svg` and the PNG/ICNS/ICO/SVG files derived from it) | LightCraft app icon (lynx) | LightCraft project owner (drawn in ArtCraft) | original work (vectorised from the owner's drawing; see `assets/app-icon/README.md`) | MIT OR Apache-2.0 (`assets/app-icon/LICENSE.txt`) | 2026-10-02 | vectorised to SVG; rendered to PNG/ICNS/ICO by `packaging/icons.sh` |
| `crates/scenes/` (generated images) | Procedural demo photographs | LightCraft contributors | original work (generated at runtime, no source imagery) | MIT OR Apache-2.0 | 2026-09-30 | n/a |
| `crates/ui-egui/src/icons.rs` | UI icons drawn as vector paths in code | LightCraft contributors | original work | MIT OR Apache-2.0 | 2026-09-30 | n/a |
Binary file not shown.
+93
View File
@@ -0,0 +1,93 @@
Copyright 2022 The BIZ UDMincho Project Authors (https://github.com/googlefonts/morisawa-biz-ud-mincho)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
+46
View File
@@ -44,6 +44,11 @@ fn photo(c: &mut Catalog, i: u64) -> Photo {
p.meta.iso = Some(400);
p.as_shot_wb = Some((5200.0, 3.0));
p.local = !i.is_multiple_of(10);
if p.local {
// as a browse catalogues it: import defaults, a baseline fingerprint
p.develop = Arc::new(p.import_defaults());
p.set_local_baseline();
}
if !p.local {
let mut d = DevelopSettings::default();
d.light.exposure = (i % 7) as f64 * 0.1;
@@ -153,6 +158,47 @@ fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
}
println!(" background compaction blocking: {}", stats(blocking));
// forget the untouched Local records (folders last browsed long ago; every 7th folder
// recently; every 50th Local record rated), then compact
let folders: std::collections::BTreeSet<String> = cat.photos().filter(|p| p.local).filter_map(|p| folder_of(p)).collect();
let mut ops: Vec<Op> = folders
.iter()
.enumerate()
.map(|(k, f)| Op::SetBrowsed {
folder: f.clone(),
at: Some(if k % 7 == 0 { "2026-09-30T00:00:00" } else { "2026-06-01T00:00:00" }.into()),
})
.collect();
ops.extend(cat.photos().filter(|p| p.local && p.id.0.is_multiple_of(50)).map(|p| Op::SetRating { id: p.id, rating: 2 }).collect::<Vec<_>>());
for op in &ops {
cat.apply(op.clone())?;
}
j.append(&ops)?;
j.snapshot(&cat)?;
let before = j.stats().last_snapshot.bytes;
let t = Instant::now();
let plan = cat.forget_local_plan("2026-10-05T00:00:00", DEFAULT_FORGET_DAYS, &|_| false);
let plan_ms = ms(t);
let t = Instant::now();
let ops = plan.ops();
for op in &ops {
cat.apply(op.clone())?;
}
j.append(&ops)?;
let apply_ms = ms(t);
j.snapshot(&cat)?;
let after = j.stats().last_snapshot.bytes;
println!(
" forget Local: {} of {} local records forgotten (kept: {} recent, {} touched, {} in use); plan {plan_ms:.1} ms, apply+append {apply_ms:.1} ms; snapshot {:.1} MB -> {:.1} MB",
plan.evict.len(),
plan.local,
plan.kept_recent,
plan.kept_touched,
plan.kept_in_use,
before as f64 / 1e6,
after as f64 / 1e6
);
drop(j);
if std::env::var_os("PERSIST_BENCH_NO_LOAD").is_some() {
// peak memory of building + compacting only
+204 -26
View File
@@ -1,8 +1,9 @@
//! Crash-safe catalog persistence: an append-only op log plus periodic snapshots.
//!
//! Files (in a [`Store`]):
//! - `catalog.snap` — `{"format":"lightcraft-catalog","version":1,"seq":N,"catalog":{…}}`, the
//! state after op `N`; replaced atomically (temp + fsync + rename).
//! - `catalog.snap` — `{"format":"lightcraft-catalog","version":V,"seq":N,"catalog":{…}}`, the
//! state after op `N` in catalog format `V` ([`VERSION`]); replaced atomically (temp + fsync +
//! rename).
//! - `catalog.log` — JSON lines, one per op applied after the snapshot:
//! `{"seq":N,"crc":C,"op":{…}}` where `C` is the CRC-32 of the op's JSON text. Appends are
//! fsynced before [`Journal::append`] returns.
@@ -10,6 +11,12 @@
//! Loading = snapshot + replay of the records with `seq > snapshot seq`. Recovery rules:
//! - a **torn final record** (crash mid-append: truncated line or bad CRC at the end) is dropped
//! and the file is cut back to the last good record, so later appends start on a clean line;
//! - a **failed append** (disk full, a dropped network share) may leave part of its batch in the
//! file: [`Journal::append`] cuts the log back to the last whole record before returning the
//! error (or, if that fails too, before the next append), so the retry — the same ops with the
//! same `seq`s — starts on a clean line. Logs written by builds without that cut-back hold the
//! fragment glued to the first retried record on one line; the loader recognises the whole,
//! CRC-valid record at the end of such a line and skips only the fragment;
//! - records already covered by the snapshot (crash between writing the snapshot and resetting
//! the log) are skipped;
//! - a bad record **followed by good ones** is real damage: replay stops there, the log is kept
@@ -27,6 +34,32 @@
//!
//! At most one snapshot is in flight; a synchronous [`Journal::snapshot`] and dropping the journal
//! first wait for it, so an older snapshot can never replace a newer one.
//!
//! # Format versions
//!
//! The snapshot's `version` is the catalog format, and it covers the log beside it: the log has
//! no header of its own, and is only ever appended to under a snapshot of this build's format.
//!
//! | Version | Written by | Adds |
//! |---|---|---|
//! | (none) | early builds: a log without a snapshot | |
//! | 1 | up to v0.2.0 | |
//! | 2 | after v0.2.0 | `Op::SetBrowsed`, `Catalog.browsed`, `Photo.local_baseline` |
//!
//! Rules:
//! - **Bump [`VERSION`]** (and add a row above) in the change that adds an [`Op`] variant or a
//! serialized field to the catalog, a photo, an album, a stack or develop settings that an older
//! build would misread or silently drop. New fields must deserialize from older data
//! (`#[serde(default)]`), so newer builds keep reading every older format.
//! - **Opening an older format upgrades it**: right after loading, [`Journal::open`] writes a
//! snapshot in this build's format (also for a new library), before anything is appended — so a
//! library is never a mix, and older builds refuse it from then on instead of reading part of it.
//! [`LoadReport::upgraded_from`] says when that happened.
//! - **A newer format is refused, untouched**: a snapshot whose `version` is above [`VERSION`],
//! or a log record whose CRC matches (so it is exactly what was written) but whose op doesn't
//! parse, fails [`Journal::open`] with [`CatalogError::Newer`] before any file is modified — it
//! is never mistaken for a torn tail or damage. (Builds up to v0.2.0 refuse a v2 snapshot with
//! "unsupported format".)
use crate::store::Store;
use crate::{Catalog, CatalogError, Op, Result};
@@ -34,7 +67,9 @@ use crate::{Catalog, CatalogError, Op, Result};
pub const SNAPSHOT: &str = "catalog.snap";
pub const LOG: &str = "catalog.log";
const FORMAT: &str = "lightcraft-catalog";
const VERSION: u32 = 1;
/// The catalog format this build writes (and the newest it reads). See the module docs →
/// *Format versions*; bump it whenever an [`Op`] variant or a serialized field is added.
pub const VERSION: u32 = 2;
/// When [`Journal::wants_snapshot`] says it's time to compact the log.
#[derive(Clone, Copy, Debug)]
@@ -62,10 +97,16 @@ pub struct LoadReport {
pub failed: usize,
/// Bytes of a torn final record that were dropped.
pub torn_bytes: u64,
/// Partial records left mid-log by a failed append that was retried (written by builds that
/// didn't cut the log back after a failed write): skipped, the retried records were replayed.
pub torn_fragments: usize,
/// The damaged log was preserved under this name.
pub damaged: Option<String>,
/// No catalog files existed (a new library).
pub created: bool,
/// The library was in an older catalog format (`0` = before format versions: a log without a
/// snapshot) and was rewritten in this version's format ([`VERSION`]).
pub upgraded_from: Option<u32>,
}
/// Where persistence time goes (reported by `library.info` → `persistence`; printed to stderr
@@ -140,6 +181,9 @@ pub struct Journal {
pending: Option<Pending>,
/// After a failed background snapshot: don't retry before the log has this many records.
retry_at_records: u64,
/// A failed append may have left part of its records after `log_bytes`, and cutting them off
/// failed too: the next append cuts the log back to `log_bytes` first.
tail_dirty: bool,
}
/// A background snapshot in flight.
@@ -168,18 +212,58 @@ pub fn encode_record(seq: u64, op: &Op) -> String {
format!("{{\"seq\":{seq},\"crc\":{crc},\"op\":{body}}}")
}
/// Parse one log line (without the newline). `None` if it's malformed or the CRC doesn't match.
/// Parse one log line (without the newline). `None` if it's malformed, the CRC doesn't match, or
/// the op is one this version doesn't know (see [`Journal::open`]).
pub fn decode_record(line: &str) -> Option<(u64, Op)> {
let rest = line.strip_prefix("{\"seq\":")?;
let (seq, rest) = rest.split_once(",\"crc\":")?;
let (crc, rest) = rest.split_once(",\"op\":")?;
let body = rest.strip_suffix('}')?;
let seq: u64 = seq.parse().ok()?;
let crc: u32 = crc.parse().ok()?;
if crc32fast::hash(body.as_bytes()) != crc {
return None;
match decode_line(line) {
Line::Record(seq, op) => Some((seq, op)),
Line::Newer(_) | Line::Bad => None,
}
serde_json::from_str(body).ok().map(|op| (seq, op))
}
/// One log line, decoded.
enum Line {
Record(u64, Op),
/// The CRC matches — the line is exactly what was written — but the op doesn't parse: it
/// was written by a newer LightCraft (an op or field this version doesn't know), not damaged.
Newer(u64),
/// Malformed or a CRC mismatch: torn or damaged.
Bad,
}
fn decode_line(line: &str) -> Line {
let parts = || {
let rest = line.strip_prefix("{\"seq\":")?;
let (seq, rest) = rest.split_once(",\"crc\":")?;
let (crc, rest) = rest.split_once(",\"op\":")?;
let body = rest.strip_suffix('}')?;
Some((seq.parse::<u64>().ok()?, crc.parse::<u32>().ok()?, body))
};
let Some((seq, crc, body)) = parts() else { return Line::Bad };
if crc32fast::hash(body.as_bytes()) != crc {
return Line::Bad;
}
match serde_json::from_str(body) {
Ok(op) => Line::Record(seq, op),
Err(_) => Line::Newer(seq),
}
}
/// The error for a library written by a newer LightCraft.
fn newer(what: String) -> CatalogError {
CatalogError::Newer(format!("{what}; this version reads catalog format v{VERSION} and older"))
}
/// A line that doesn't decode but ends with a whole record: a fragment of a failed append followed
/// by the record that retried it (the retry re-encodes the same ops with the same `seq`s). Returns
/// the fragment's length and the record, if that record continues the log (`seq <= last + 1`).
/// The record's CRC must match, so damage is never mistaken for it.
fn resync(line: &[u8], last: u64) -> Option<(usize, (u64, Op))> {
const START: &[u8] = b"{\"seq\":";
(1..line.len().saturating_sub(START.len() - 1)).filter(|&i| line[i..].starts_with(START)).find_map(|i| {
let rec = std::str::from_utf8(&line[i..]).ok().map(|l| l.trim_end_matches('\r')).and_then(decode_record)?;
(rec.0 <= last + 1).then_some((i, rec))
})
}
/// Write `catalog.snap` for the state after op `seq`, streaming the JSON into the store (no
@@ -205,10 +289,16 @@ fn write_snapshot(store: &mut dyn Store, seq: u64, catalog: &Catalog) -> std::io
Ok(SnapshotTiming { serialize_ms, write_sync_ms: (ms_since(t0) - serialize_ms).max(0.0), bytes, ..Default::default() })
}
/// Just the identification of `catalog.snap` (read before the catalog itself).
#[derive(serde::Deserialize)]
struct SnapFile {
struct SnapHeader {
format: String,
version: u32,
}
/// The whole `catalog.snap` (after [`SnapHeader`] was checked; `format` and `version` are ignored).
#[derive(serde::Deserialize)]
struct SnapFile {
seq: u64,
catalog: Catalog,
}
@@ -221,15 +311,20 @@ impl Journal {
let snap = store.read(SNAPSHOT).map_err(io)?;
let log = store.read(LOG).map_err(io)?;
report.created = snap.is_none() && log.is_none();
let (mut catalog, snapshot_seq) = match snap {
let (mut catalog, snapshot_seq, snapshot_version) = match snap {
Some(bytes) => {
let s: SnapFile = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
if s.format != FORMAT || s.version > VERSION {
return Err(CatalogError::Corrupt(format!("{SNAPSHOT}: unsupported format {} v{}", s.format, s.version)));
// the header first: a newer snapshot may not parse as this version's catalog
let h: SnapHeader = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
if h.format != FORMAT {
return Err(CatalogError::Corrupt(format!("{SNAPSHOT}: not a LightCraft catalog (format {:?})", h.format)));
}
(s.catalog, s.seq)
if h.version > VERSION {
return Err(newer(format!("{SNAPSHOT} is catalog format v{}", h.version)));
}
let s: SnapFile = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
(s.catalog, s.seq, Some(h.version))
}
None => (Catalog::new(), 0),
None => (Catalog::new(), 0, None),
};
report.snapshot_seq = snapshot_seq;
let mut j = Journal {
@@ -242,6 +337,7 @@ impl Journal {
stats: PersistStats::default(),
pending: None,
retry_at_records: 0,
tail_dirty: false,
};
let log = log.unwrap_or_default();
@@ -261,7 +357,22 @@ impl Journal {
good_end = next;
continue;
}
match line.and_then(decode_record) {
let mut record = line.and_then(decode_record);
if record.is_none()
&& let Some((at, r)) = resync(&log[pos..line_end], j.seq)
{
// the torn start of a failed append, then the retried records (older builds
// appended them right after the fragment)
log::warn!("catalog log: skipped a partial record ({at} bytes) left by a failed write");
report.torn_fragments += 1;
record = Some(r);
}
if record.is_none()
&& let Some(Line::Newer(seq)) = line.map(decode_line)
{
return Err(newer(format!("{LOG} record {seq} holds a change this version doesn't know")));
}
match record {
Some((seq, op)) if seq <= j.seq => {
// already in the snapshot
let _ = op;
@@ -277,7 +388,15 @@ impl Journal {
}
_ => {
// bad record (or a gap): torn tail if nothing good follows, else damage
let rest_has_good = log[next..].split(|b| *b == b'\n').any(|l| std::str::from_utf8(l).ok().and_then(decode_record).is_some());
let mut rest_has_good = false;
for l in log[next..].split(|b| *b == b'\n') {
match std::str::from_utf8(l).map(|l| decode_line(l.trim_end_matches('\r'))) {
Ok(Line::Record(..)) => rest_has_good = true,
// refused before anything is modified
Ok(Line::Newer(seq)) => return Err(newer(format!("{LOG} record {seq} holds a change this version doesn't know"))),
Ok(Line::Bad) | Err(_) => {}
}
}
if rest_has_good {
damaged_at = Some(pos);
}
@@ -301,10 +420,34 @@ impl Journal {
report.torn_bytes = (log.len() - good_end) as u64;
log::warn!("catalog log: dropped a torn final record ({} bytes)", report.torn_bytes);
j.store.truncate(LOG, good_end as u64).map_err(io)?;
j.log_bytes = good_end as u64;
} else if needs_newline {
j.store.append(LOG, b"\n").map_err(io)?;
j.log_bytes = good_end as u64 + 1;
} else {
j.log_bytes = good_end as u64;
}
}
// Bring an older (or new, or snapshot-less) library to this version's format before
// anything is appended: a snapshot in this format makes older builds refuse the library
// instead of misreading ops or dropping fields they don't know.
if report.created {
// tiny: written directly, not counted as a compaction
let empty = format!("{{\"format\":\"{FORMAT}\",\"version\":{VERSION},\"seq\":0,\"catalog\":{}}}\n", catalog.to_snapshot());
if let Err(e) = j.store.write_atomic(SNAPSHOT, empty.as_bytes()) {
log::warn!("catalog: can't write the new library's snapshot: {e}");
}
} else if report.damaged.is_none() && snapshot_version != Some(VERSION) {
report.upgraded_from = Some(snapshot_version.unwrap_or(0));
match j.snapshot(&catalog) {
Ok(()) => {
if let Some(v) = report.upgraded_from {
log::info!("catalog: upgraded from format v{v} to v{VERSION}");
}
}
// nothing lost: the old files are intact and still load; retried on next open
Err(e) => log::warn!("catalog: can't write the format v{VERSION} snapshot: {e}"),
}
j.log_bytes = good_end as u64;
}
catalog.revision = 0;
Ok((j, catalog, report))
@@ -323,7 +466,20 @@ impl Journal {
buf.push_str(&encode_record(seq, op));
buf.push('\n');
}
self.store.append(LOG, buf.as_bytes()).map_err(io)?;
if self.tail_dirty {
self.store.truncate(LOG, self.log_bytes).map_err(io)?;
self.tail_dirty = false;
}
if let Err(e) = self.store.append(LOG, buf.as_bytes()) {
// Part of the batch may be in the file. Cut it back to the last whole record, so the
// retry (same seqs) starts on a clean line instead of after a fragment that would
// read back as damage.
if let Err(t) = self.store.truncate(LOG, self.log_bytes) {
log::error!("catalog: can't cut the log back after a failed append: {t}");
self.tail_dirty = true;
}
return Err(io(e));
}
if let Some(p) = self.pending.as_mut() {
p.tail.extend_from_slice(buf.as_bytes());
p.tail_records += ops.len() as u64;
@@ -343,12 +499,33 @@ impl Journal {
/// Write a snapshot of `catalog` (which must reflect every appended op) and reset the log.
pub fn snapshot(&mut self, catalog: &Catalog) -> Result<()> {
self.snapshot_with_unlogged(catalog, 0)
}
/// [`Journal::snapshot`] of a catalog that also reflects `unlogged` ops applied after the
/// last appended one but never written to the log (their append failed). The snapshot counts
/// them (it holds the state after op `seq + unlogged`), so they are saved and must not be
/// appended afterwards. Once the snapshot is durable [`Journal::seq`] includes them, even if
/// resetting the log then fails (the error is still returned).
pub fn snapshot_with_unlogged(&mut self, catalog: &Catalog, unlogged: u64) -> Result<()> {
// never two snapshot writers; the one in flight is older, so it must land first
if let Err(e) = self.wait() {
log::warn!("catalog: background snapshot failed ({e}); writing one now");
}
let t0 = web_time::Instant::now();
let mut timing = write_snapshot(self.store.as_mut(), self.seq, catalog).map_err(io)?;
let seq = self.seq + unlogged;
let mut timing = match write_snapshot(self.store.as_mut(), seq, catalog) {
Ok(t) => t,
Err(e) => {
// nothing lost (the log is whole); don't retry on every append
self.stats.failed_snapshots += 1;
self.retry_at_records = self.log_records + self.policy.max_records;
return Err(io(e));
}
};
// the snapshot is durable: it holds every op up to `seq`
self.seq = seq;
self.snapshot_seq = seq;
let t2 = web_time::Instant::now();
// A crash here leaves old records in the log; they are skipped by seq on load.
self.store.write_atomic(LOG, b"").map_err(io)?;
@@ -357,10 +534,10 @@ impl Journal {
timing.blocking_ms = timing.total_ms;
timing.records = self.log_records;
self.record_snapshot(timing);
self.snapshot_seq = self.seq;
self.log_records = 0;
self.log_bytes = 0;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
@@ -451,6 +628,7 @@ impl Journal {
self.log_records = p.tail_records;
self.log_bytes = p.tail.len() as u64;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
+39
View File
@@ -5,15 +5,23 @@
//! snapshot — crash-safe and diff-friendly (see [`journal`]);
//! - **undo/redo**: the engine keeps inverse ops;
//! - **determinism**: replaying the log reproduces the state exactly (property-tested).
//!
//! **Catalog format version** ([`journal::VERSION`], see [`journal`] → *Format versions*): adding
//! an [`Op`] variant or a serialized field means bumping it. Newer builds read every older format
//! (and upgrade it on open); older builds refuse a newer library with [`CatalogError::Newer`]
//! instead of reading part of it.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
pub mod dates;
pub mod journal;
pub mod keywords;
pub mod local;
pub mod lock;
pub mod model;
pub mod query;
pub mod rules;
pub mod safe_file;
pub mod stacks;
pub mod store;
@@ -24,6 +32,8 @@ pub use dates::{DateRun, GroupBy};
pub use journal::{Journal, LoadReport, PersistStats, SnapshotPolicy, SnapshotTiming};
pub use keywords::KeywordNode;
use lightcraft_develop::DevelopSettings;
pub use local::{DEFAULT_FORGET_DAYS, ForgetPlan, folder_of};
pub use lock::{LibraryLock, LockError, LockOwner};
pub use model::*;
pub use query::{DateGroup, Filter, RatingOp, Sort, SortKey};
pub use rules::{Match, Rule, RuleSet};
@@ -44,6 +54,10 @@ pub enum CatalogError {
Corrupt(String),
#[error("catalog storage: {0}")]
Io(String),
/// The library was written by a newer LightCraft (a newer catalog format, or a change this
/// version doesn't know). Nothing was read into the session and nothing was modified.
#[error("this library was written by a newer version of LightCraft ({0}); update LightCraft to open it. The library was left unchanged.")]
Newer(String),
}
pub type Result<T> = std::result::Result<T, CatalogError>;
@@ -190,6 +204,12 @@ pub enum Op {
label: ColorLabel,
name: Option<String>,
},
/// When a Local folder was last browsed (ISO 8601; `None` = forget the time). Not an undo
/// step: it drives forgetting untouched Local records (see [`local`]).
SetBrowsed {
folder: String,
at: Option<String>,
},
/// Several ops as one step (undo applies the inverses in reverse).
Batch {
ops: Vec<Op>,
@@ -209,6 +229,9 @@ pub struct Catalog {
/// Custom colour label names.
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
label_names: BTreeMap<ColorLabel, String>,
/// When each Local folder was last browsed (folder path → ISO 8601), see [`local`].
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
browsed: BTreeMap<String, String>,
/// Increments on every applied op.
#[serde(skip)]
pub revision: u64,
@@ -547,6 +570,14 @@ impl Catalog {
};
Op::SetLabelName { label, name: old }
}
Op::SetBrowsed { folder, at } => {
let folder = crate::query::folder_key(&folder);
let old = match at {
Some(t) => self.browsed.insert(folder.clone(), t),
None => self.browsed.remove(&folder),
};
Op::SetBrowsed { folder, at: old }
}
Op::Batch { ops } => {
let mut inverses = Vec::with_capacity(ops.len());
for op in ops {
@@ -624,4 +655,12 @@ mod tests;
#[cfg(test)]
mod tests_background;
#[cfg(test)]
mod tests_format_version;
#[cfg(test)]
mod tests_journal;
#[cfg(test)]
mod tests_local;
#[cfg(test)]
mod tests_lock;
#[cfg(test)]
mod tests_torn_append;
+304
View File
@@ -0,0 +1,304 @@
//! Forgetting untouched Local browse records.
//!
//! Browsing a folder (Local) catalogues its photos as `local` records so they can be shown,
//! rated and edited in place. Most are only ever looked at, and on a large disk they can be most
//! of the catalog. A record is **forgotten** (removed from the catalog, never from the disk —
//! files and XMP sidecars are not touched) when all of these hold:
//!
//! - it is a Local record ([`Photo::local`]) of a file, not deleted, not a virtual copy;
//! - its folder was last browsed more than `days` ago ([`Catalog::last_browsed`]). A folder with
//! no recorded time (a catalog from before this existed) counts as browsed *now*: it is stamped
//! with the current time and nothing in it is forgotten this time;
//! - it is **untouched** ([`Photo::untouched_local`]): nothing about it changed since it was
//! browsed — see below;
//! - nothing refers to it: no album holds it or shows it as cover, no stack holds it, no virtual
//! copy was made of it; and the caller (the engine) says it isn't in use (visible, selected,
//! in the undo history, has a smart preview…).
//!
//! **Untouched.** A record catalogued by a browse carries a fingerprint of its whole state at
//! that moment ([`Photo::local_baseline`]). It is untouched while its state still hashes to that
//! fingerprint, i.e. *any* change since — develop settings or history, versions, rating, flag,
//! label, any metadata (title, caption, keywords, location, copyright…), capture time, crop,
//! rename or relink, culling scores, file content — keeps it. Records without a fingerprint
//! (browsed before fingerprints existed) are judged by their data alone and kept unless every
//! user-editable field is still empty / default: no rating, flag or label; develop settings equal
//! to the import defaults with no history, versions or edit time; no title, caption, alt text,
//! description, keywords, creator, copyright, rights, location or GPS; no culling scores.
//! When in doubt a record is kept.
//!
//! Browsing the folder again catalogues its files afresh (new records, new fingerprints).
use std::collections::{BTreeMap, BTreeSet, HashSet};
use serde::Serialize;
use crate::{Catalog, Op, Photo, PhotoId, Source};
/// Default for how long a Local folder may go unbrowsed before its untouched records are
/// forgotten (days).
pub const DEFAULT_FORGET_DAYS: u32 = 30;
/// Records removed per logged batch (keeps log lines a reasonable size).
const BATCH: usize = 2000;
/// 64-bit FNV-1a over everything written to it (stable across builds and platforms, unlike
/// `std`'s hasher), so a photo is hashed as it serialises, without a buffer.
struct Fnv(u64);
impl std::io::Write for Fnv {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
for b in buf {
self.0 ^= u64::from(*b);
self.0 = self.0.wrapping_mul(0x0100_0000_01b3);
}
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// The folder a file-backed photo is in (`None` for demo photos).
pub fn folder_of(p: &Photo) -> Option<String> {
match &p.source {
// `/` and `\` are both separators whatever the platform (a catalog can move between them)
Source::File { path } => {
let key = crate::query::folder_key(path);
key.rsplit_once('/').map(|(dir, _)| if dir.is_empty() { "/".to_string() } else { dir.to_string() })
}
Source::Demo { .. } => None,
}
}
impl Photo {
/// A hash of everything about the photo except [`Photo::local_baseline`] itself.
pub fn fingerprint(&self) -> u64 {
let mut h = Fnv(0xcbf2_9ce4_8422_2325);
let r = if self.local_baseline.is_some() {
let mut p = self.clone();
p.local_baseline = None;
serde_json::to_writer(&mut h, &p)
} else {
serde_json::to_writer(&mut h, self)
};
// (serialising a photo can't fail; if it ever did, no baseline would match: kept)
if r.is_err() { 0 } else { h.0 }
}
/// Record the current state as the browse baseline (done when a browse catalogues it).
pub fn set_local_baseline(&mut self) {
self.local_baseline = None;
self.local_baseline = Some(self.fingerprint());
}
/// A Local record nothing was changed on since it was browsed (see the module docs). Says
/// nothing about albums, stacks or virtual copies referring to it.
pub fn untouched_local(&self) -> bool {
if !self.local || self.deleted || self.copy_of.is_some() || self.copy_name.is_some() || folder_of(self).is_none() {
return false;
}
match self.local_baseline {
Some(b) => self.fingerprint() == b,
None => self.looks_unedited(),
}
}
/// Every user-editable field is still empty / default (records without a baseline).
fn looks_unedited(&self) -> bool {
let m = &self.meta;
self.rating == 0
&& self.flag == crate::Flag::None
&& self.label.is_none()
&& self.edited.is_none()
&& self.history.is_empty()
&& self.versions.is_empty()
&& *self.develop == self.import_defaults()
&& self.analysis.is_none()
&& m.title.is_empty()
&& m.caption.is_empty()
&& m.alt_text.is_empty()
&& m.extended_description.is_empty()
&& m.keywords.is_empty()
&& m.creator.is_empty()
&& m.copyright.is_empty()
&& m.copyright_status.is_unknown()
&& m.usage_terms.is_empty()
&& m.copyright_url.is_empty()
&& m.location.is_empty()
&& m.city.is_empty()
&& m.state.is_empty()
&& m.country.is_empty()
&& m.gps.is_none()
}
}
/// What forgetting Local records would do / did.
#[derive(Clone, Debug, Default, PartialEq, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct ForgetPlan {
/// Local records looked at.
pub local: usize,
/// Records to forget.
pub evict: Vec<PhotoId>,
/// Kept: their folder was browsed within the period (or has no time yet).
pub kept_recent: usize,
/// Kept: changed by the user since they were browsed.
pub kept_touched: usize,
/// Kept: in an album, a stack, the source of a virtual copy, or in use.
pub kept_in_use: usize,
/// Folders with Local records but no last-browsed time: stamped now.
pub stamp: Vec<String>,
/// Folders whose last Local record goes: their time is dropped.
pub unstamp: Vec<String>,
/// The time stamps are given.
#[serde(skip)]
pub now: String,
}
impl ForgetPlan {
/// The ops that carry the plan out (empty when there is nothing to do).
pub fn ops(&self) -> Vec<Op> {
let mut ops: Vec<Op> = self.stamp.iter().map(|f| Op::SetBrowsed { folder: f.clone(), at: Some(self.now.clone()) }).collect();
ops.extend(self.evict.chunks(BATCH).map(|c| Op::Batch { ops: c.iter().map(|id| Op::RemovePhoto { id: *id }).collect() }));
ops.extend(self.unstamp.iter().map(|f| Op::SetBrowsed { folder: f.clone(), at: None }));
ops
}
}
impl Catalog {
/// When `folder` was last browsed (ISO 8601), if known.
pub fn last_browsed(&self, folder: &str) -> Option<&str> {
self.browsed.get(&crate::query::folder_key(folder)).map(String::as_str)
}
/// Folders with a last-browsed time.
pub fn browsed_folders(&self) -> &BTreeMap<String, String> {
&self.browsed
}
/// Plan forgetting untouched Local records whose folder wasn't browsed for `days` days (as of
/// `now`, ISO 8601). `in_use` keeps records the caller still needs. `days == 0` (never) or an
/// unreadable `now` forgets nothing. See the module docs.
pub fn forget_local_plan(&self, now: &str, days: u32, in_use: &dyn Fn(&Photo) -> bool) -> ForgetPlan {
let mut plan = ForgetPlan { now: now.to_string(), ..Default::default() };
let now_s = crate::stacks::iso_seconds(now);
let max_age = i64::from(days) * 86_400;
let mut referenced: HashSet<PhotoId> = HashSet::new();
for a in self.albums.values() {
referenced.extend(a.photos.iter().copied());
referenced.extend(a.cover);
}
for s in self.stacks.values() {
referenced.extend(s.photos.iter().copied());
}
referenced.extend(self.photos.values().filter_map(|p| p.copy_of));
// the fingerprints are the costly part: work them out for the candidates in parallel
let candidates: Vec<&Photo> = self
.photos
.values()
.filter(|p| p.local && !referenced.contains(&p.id))
.filter(|p| folder_of(p).and_then(|f| self.browsed.get(&f)).is_some_and(|at| is_old(at, now_s, days, max_age)))
.map(|p| &**p)
.collect();
let untouched: HashSet<PhotoId> = untouched_of(&candidates);
let mut stamp = BTreeSet::new();
let mut remaining: BTreeSet<String> = BTreeSet::new();
for p in self.photos.values().filter(|p| p.local) {
plan.local += 1;
let Some(folder) = folder_of(p) else {
plan.kept_in_use += 1;
continue;
};
let recent = match (self.browsed.get(&folder), now_s) {
(None, _) => {
stamp.insert(folder.clone());
true
}
(Some(at), _) => !is_old(at, now_s, days, max_age),
};
let keep = if recent {
plan.kept_recent += 1;
true
} else if referenced.contains(&p.id) || in_use(p) {
plan.kept_in_use += 1;
true
} else if !untouched.contains(&p.id) {
plan.kept_touched += 1;
true
} else {
plan.evict.push(p.id);
false
};
if keep {
remaining.insert(folder);
}
}
plan.stamp = stamp.into_iter().collect();
plan.unstamp = self.browsed.keys().filter(|f| !remaining.contains(*f)).cloned().collect();
plan
}
}
/// A folder browsed at `at` is past the period (never when `days` is 0 or a time is unreadable).
fn is_old(at: &str, now_s: Option<i64>, days: u32, max_age: i64) -> bool {
days > 0 && matches!((now_s, crate::stacks::iso_seconds(at)), (Some(n), Some(a)) if n - a >= max_age)
}
/// The untouched ones among `photos` ([`Photo::untouched_local`]), checked on several threads
/// where there are many.
fn untouched_of(photos: &[&Photo]) -> HashSet<PhotoId> {
let check = |chunk: &[&Photo]| chunk.iter().filter(|p| p.untouched_local()).map(|p| p.id).collect::<Vec<_>>();
#[cfg(not(target_arch = "wasm32"))]
{
let threads = std::thread::available_parallelism().map_or(1, |n| n.get()).min(8);
if threads > 1 && photos.len() >= 256 {
let size = photos.len().div_ceil(threads);
return std::thread::scope(|sc| {
let jobs: Vec<_> = photos.chunks(size).map(|c| sc.spawn(move || check(c))).collect();
// a job that panicked (it can't) leaves its photos out: they are kept
jobs.into_iter().filter_map(|j| j.join().ok()).flatten().collect()
});
}
}
check(photos).into_iter().collect()
}
impl Op {
/// Every photo the op refers to.
pub fn photo_ids(&self, out: &mut impl FnMut(PhotoId)) {
match self {
Op::AddPhoto { photo } => out(photo.id),
Op::RemovePhoto { id }
| Op::SetRating { id, .. }
| Op::SetFlag { id, .. }
| Op::SetLabel { id, .. }
| Op::SetDevelop { id, .. }
| Op::SetMeta { id, .. }
| Op::SetDeleted { id, .. }
| Op::SetLocal { id, .. }
| Op::SetVersions { id, .. }
| Op::SetHistory { id, .. }
| Op::PushHistory { id, .. }
| Op::SetCaptured { id, .. }
| Op::SetAnalysis { id, .. }
| Op::SetFile { id, .. }
| Op::Relink { id, .. }
| Op::SetContent { id, .. } => out(*id),
Op::AddAlbum { album } => {
album.photos.iter().for_each(|p| out(*p));
album.cover.into_iter().for_each(&mut *out);
}
Op::SetAlbumPhotos { photos, .. } | Op::SetStack { photos, .. } => photos.iter().for_each(|p| out(*p)),
Op::SetAlbumCover { cover, .. } => cover.iter().for_each(|p| out(*p)),
Op::AddStack { stack } => stack.photos.iter().for_each(|p| out(*p)),
Op::Batch { ops } => ops.iter().for_each(|o| o.photo_ids(out)),
Op::RemoveAlbum { .. }
| Op::RenameAlbum { .. }
| Op::MoveAlbum { .. }
| Op::SetAlbumRules { .. }
| Op::RemoveStack { .. }
| Op::SetLabelName { .. }
| Op::SetBrowsed { .. } => {}
}
}
}
+140
View File
@@ -0,0 +1,140 @@
//! One process per library: an exclusive OS lock on `<library>/catalog.lock`, held for as long as
//! the library is open (issue #99).
//!
//! Two processes on the same library (the app and `lightcraft-cli --library`, two app instances,
//! two computers on a shared folder) would each keep their own catalog and `seq` counter, and
//! whichever wrote the last snapshot would silently drop the other's edits. So opening a library
//! takes the lock first, and a second opener gets [`LockError::InUse`] instead.
//!
//! - The lock is the OS's (`flock` on Unix, `LockFileEx` on Windows, via [`std::fs::File::try_lock`]):
//! it is released when the [`LibraryLock`] is dropped and, after a crash or a kill, by the OS when
//! the process ends. A `catalog.lock` file left behind is therefore never "stale": only a lock
//! that's actually held refuses the open.
//! - Who holds it (process id, computer, program, version) is written to `catalog.lock.owner`, for
//! the error message. It's a separate file because a Windows lock also blocks reading the
//! locked file. It's only a hint: it may be missing or left over from a crash.
//! - File systems without lock support (some network shares): the library opens unlocked, with a
//! warning — refusing it would lock the user out of their own library. Locks over SMB/NFS are
//! advisory and depend on the server; the owner file is then the only hint.
//! - Browser storage (OPFS) has no file locks: see the web host for its own guard.
use std::fs::{File, OpenOptions, TryLockError};
use std::io;
use std::path::{Path, PathBuf};
/// The lock file in the library directory.
pub const LOCK: &str = "catalog.lock";
/// Who holds the lock (JSON, see [`LockOwner`]).
pub const OWNER: &str = "catalog.lock.owner";
/// Who has the library open (as written in [`OWNER`]).
#[derive(Clone, Debug, Default, PartialEq, serde::Serialize, serde::Deserialize)]
#[serde(default)]
pub struct LockOwner {
pub pid: u32,
/// Computer name, if known.
pub host: String,
/// The program (`LightCraft`, `lightcraft-cli`…) and its version.
pub program: String,
pub version: String,
/// When it opened the library (seconds since 1970).
pub since: u64,
}
impl LockOwner {
/// This process, as `program`.
pub fn this_process(program: &str) -> LockOwner {
LockOwner {
pid: std::process::id(),
host: host_name(),
program: program.to_string(),
version: env!("CARGO_PKG_VERSION").to_string(),
since: web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0),
}
}
/// "LightCraft 0.3.0 (process 123 on studio-mac)".
pub fn describe(&self) -> String {
let program = if self.program.is_empty() { "another LightCraft" } else { self.program.as_str() };
let version = if self.version.is_empty() { String::new() } else { format!(" {}", self.version) };
let host = if self.host.is_empty() {
String::new()
} else if self.host == host_name() {
" on this computer".to_string()
} else {
format!(" on {}", self.host)
};
format!("{program}{version} (process {}{host})", self.pid)
}
}
/// The computer's name, best effort (empty if unknown).
fn host_name() -> String {
let env = ["COMPUTERNAME", "HOSTNAME"].iter().find_map(|k| std::env::var(k).ok().filter(|h| !h.trim().is_empty()));
env.or_else(|| std::fs::read_to_string("/etc/hostname").ok()).map(|h| h.trim().to_string()).filter(|h| !h.is_empty()).unwrap_or_default()
}
#[derive(Debug, thiserror::Error)]
pub enum LockError {
/// Another process has the library open.
#[error("This library is already open in {}. Close it there first.", .0.as_ref().map(LockOwner::describe).unwrap_or_else(|| "another LightCraft program".into()))]
InUse(Option<LockOwner>),
}
/// The exclusive lock on a library directory; released on drop (and by the OS if the process
/// dies).
#[derive(Debug)]
pub struct LibraryLock {
dir: PathBuf,
/// `None` when the file system can't lock (the library is open unlocked).
_file: Option<File>,
held: bool,
}
impl LibraryLock {
/// Lock `dir` (which must exist) for this process. Fails only when another process holds it.
pub fn acquire(dir: &Path, program: &str) -> Result<LibraryLock, LockError> {
let unlocked = |why: io::Error| {
log::warn!("library {}: can't lock {LOCK} ({why}); opening it without protection against a second program", dir.display());
LibraryLock { dir: dir.to_path_buf(), _file: None, held: false }
};
let file = match OpenOptions::new().read(true).write(true).create(true).truncate(false).open(dir.join(LOCK)) {
Ok(f) => f,
Err(e) => return Ok(unlocked(e)),
};
match file.try_lock() {
Ok(()) => {}
Err(TryLockError::WouldBlock) => {
let owner = std::fs::read(dir.join(OWNER)).ok().and_then(|b| serde_json::from_slice::<LockOwner>(&b).ok());
return Err(LockError::InUse(owner));
}
Err(TryLockError::Error(e)) => return Ok(unlocked(e)),
}
let owner = LockOwner::this_process(program);
if let Ok(json) = serde_json::to_vec_pretty(&owner)
&& let Err(e) = std::fs::write(dir.join(OWNER), json)
{
log::warn!("library {}: can't write {OWNER}: {e}", dir.display());
}
Ok(LibraryLock { dir: dir.to_path_buf(), _file: Some(file), held: true })
}
/// The OS lock is held (`false`: the file system can't lock; opened unprotected).
pub fn held(&self) -> bool {
self.held
}
pub fn dir(&self) -> &Path {
&self.dir
}
}
impl Drop for LibraryLock {
/// Remove the owner note while still holding the lock (so it can't delete the next holder's),
/// then the file handle closes and the OS releases the lock.
fn drop(&mut self) {
if self.held {
let _ = std::fs::remove_file(self.dir.join(OWNER));
}
}
}
+5
View File
@@ -272,6 +272,10 @@ pub struct Photo {
/// picture style baked in — so it is treated as a rendered (non-raw) source.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preview_only: Option<String>,
/// A Local record: the fingerprint of its state when the browse catalogued it, to tell
/// whether the user changed anything since (see [`crate::local`]).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub local_baseline: Option<u64>,
}
/// What assisted culling measured on a photo.
@@ -319,6 +323,7 @@ impl Photo {
import_look: None,
analysis: None,
preview_only: None,
local_baseline: None,
}
}
/// A raw file developed from its sensor data: not a rendered image, and not a raw shown from
+317
View File
@@ -0,0 +1,317 @@
//! Crash-safe writes of user-visible files: exports, renders, DNG conversions, smart previews and
//! XMP sidecars all go through here (the catalog's own files use [`crate::store::FsStore`]).
//!
//! - [`write_atomic`] replaces a file so that after a crash, a full disk or an unplugged drive
//! either the old or the complete new content is there — never a truncated mix. The content
//! goes to a new temp file in the same folder (unique name, `create_new`), is synced to disk and
//! renamed over the target; then the folder is synced (POSIX). Any failure removes the temp
//! file and leaves the target as it was.
//! - [`write_new`] / [`write_new_unique`] write a file that must not exist yet: the same temp file
//! and sync, plus a read-back check, and the file then appears under its final name **without
//! replacing anything** (a hard link, which fails on a taken name; a check-then-rename where the
//! volume has no hard links).
//! - [`same_file`] tells whether two paths name the same file on disk (device + inode on Unix,
//! resolved paths elsewhere), for "is this target an original?" checks.
use std::cell::Cell;
use std::fs::{self, File, OpenOptions};
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
thread_local! {
static FAIL_AFTER: Cell<Option<u64>> = const { Cell::new(None) };
}
/// Fault injection for tests (in any crate): while the returned guard lives, every write made by
/// this module **on the calling thread** fails once `bytes` bytes of a file were written — as a
/// full disk or an unplugged drive would.
#[doc(hidden)]
pub fn fail_writes_after(bytes: u64) -> FaultGuard {
FAIL_AFTER.with(|c| c.set(Some(bytes)));
FaultGuard(())
}
/// Ends a [`fail_writes_after`] when dropped.
#[doc(hidden)]
pub struct FaultGuard(());
impl Drop for FaultGuard {
fn drop(&mut self) {
FAIL_AFTER.with(|c| c.set(None));
}
}
/// Counts what is written; fails past an injected limit.
struct Limited<W> {
inner: W,
n: u64,
limit: Option<u64>,
}
impl<W: Write> Write for Limited<W> {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
let buf = match self.limit {
Some(l) => {
let room = usize::try_from(l.saturating_sub(self.n)).unwrap_or(usize::MAX);
if room == 0 && !buf.is_empty() {
return Err(io::Error::other("write failed (injected fault)"));
}
buf.get(..room.min(buf.len())).unwrap_or(buf)
}
None => buf,
};
let k = self.inner.write(buf)?;
self.n += k as u64;
Ok(k)
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
fn parent_of(path: &Path) -> &Path {
path.parent().filter(|d| !d.as_os_str().is_empty()).unwrap_or(Path::new("."))
}
/// Make a rename or a new name in `dir` durable (POSIX). Opening a directory fails on Windows,
/// where NTFS journals the rename itself, so errors are ignored.
pub fn sync_dir(dir: &Path) {
if let Ok(d) = File::open(dir) {
let _ = d.sync_all();
}
}
/// A new, empty temp file next to `path` (hidden, unique in this process and folder).
fn create_temp(path: &Path) -> io::Result<(PathBuf, File)> {
static N: AtomicU64 = AtomicU64::new(0);
if path.file_name().is_none() {
return Err(io::Error::new(io::ErrorKind::InvalidInput, format!("{} is not a file path", path.display())));
}
let dir = parent_of(path);
for _ in 0..1000 {
let tmp = dir.join(format!(".lightcraft-{}-{}.tmp", std::process::id(), N.fetch_add(1, Ordering::Relaxed)));
match OpenOptions::new().write(true).create_new(true).open(&tmp) {
Ok(f) => return Ok((tmp, f)),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("no free temp file name in {}", dir.display())))
}
/// Write a temp file next to `path` with what `fill` writes, synced to disk. On failure the temp
/// file is removed. Returns its path and length.
fn write_temp(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<(PathBuf, u64)> {
let (tmp, file) = create_temp(path)?;
let r = (|| {
let limit = FAIL_AFTER.with(Cell::get);
let mut w = Limited { inner: io::BufWriter::with_capacity(1 << 20, file), n: 0, limit };
fill(&mut w)?;
w.flush()?;
let n = w.n;
let f = w.inner.into_inner().map_err(|e| e.into_error())?;
f.sync_all()?;
Ok(n)
})();
match r {
Ok(n) => Ok((tmp, n)),
Err(e) => {
let _ = fs::remove_file(&tmp);
Err(e)
}
}
}
/// Replace (or create) `path` with `data` atomically and durably; see the module docs. The
/// folder must exist.
pub fn write_atomic(path: &Path, data: &[u8]) -> io::Result<()> {
write_atomic_with(path, &mut |w| w.write_all(data)).map(|_| ())
}
/// [`write_atomic`] with streamed content: `fill` writes it (an error aborts and leaves the old
/// file). Returns the length written.
pub fn write_atomic_with(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<u64> {
let (tmp, n) = write_temp(path, fill)?;
if let Err(e) = fs::rename(&tmp, path) {
let _ = fs::remove_file(&tmp);
return Err(e);
}
sync_dir(parent_of(path));
Ok(n)
}
/// Write `data` as the new file `path`: never replaces an existing file (`AlreadyExists`), and
/// the file only appears once it is complete, synced and read back identical.
pub fn write_new(path: &Path, data: &[u8]) -> io::Result<()> {
write_new_unique(&mut std::iter::once(path.to_path_buf()), data).map(|_| ())
}
/// [`write_new`] under the first free name of `candidates` (all in the same folder, e.g.
/// `IMG_1.dng`, `IMG_1-2.dng`, …): the file is written once and then given the first name that
/// is not taken. Returns that name; `AlreadyExists` when every candidate is taken.
pub fn write_new_unique(candidates: &mut dyn Iterator<Item = PathBuf>, data: &[u8]) -> io::Result<PathBuf> {
let mut candidates = candidates.peekable();
let Some(first) = candidates.peek().cloned() else {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "no file name to write"));
};
let (tmp, _) = write_temp(&first, &mut |w| w.write_all(data))?;
let r = (|| {
if !same_content(&tmp, data)? {
return Err(io::Error::other("the written file reads back different (failing drive or connection?)"));
}
for path in candidates.by_ref().take(100_000) {
match publish_new(&tmp, &path) {
Ok(()) => {
sync_dir(parent_of(&path));
return Ok(path);
}
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("{}: no free file name", first.display())))
})();
// after a hard link the temp name is a second name of the new file; after a rename it's gone
let _ = fs::remove_file(&tmp);
r
}
/// Give the complete temp file `tmp` the name `path`, unless that is taken.
fn publish_new(tmp: &Path, path: &Path) -> io::Result<()> {
match fs::hard_link(tmp, path) {
Ok(()) => Ok(()),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => Err(e),
// no hard links on this volume (FAT/exFAT, some shares): check, then rename
Err(_) => {
if fs::symlink_metadata(path).is_ok() {
return Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("{} exists", path.display())));
}
fs::rename(tmp, path)
}
}
}
/// Does the file at `path` hold exactly `data`?
fn same_content(path: &Path, data: &[u8]) -> io::Result<bool> {
use std::io::Read;
let mut f = File::open(path)?;
if f.metadata()?.len() != data.len() as u64 {
return Ok(false);
}
let mut buf = vec![0u8; 1 << 20];
let mut at = 0usize;
loop {
let k = match f.read(&mut buf) {
Ok(0) => return Ok(at == data.len()),
Ok(k) => k,
Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
Err(e) => return Err(e),
};
if data.get(at..at + k) != buf.get(..k) {
return Ok(false);
}
at += k;
}
}
/// Do `a` and `b` name the same existing file (through links, `..`, case-insensitive names)?
pub fn same_file(a: &Path, b: &Path) -> bool {
let (Ok(ma), Ok(mb)) = (fs::metadata(a), fs::metadata(b)) else { return false };
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
ma.dev() == mb.dev() && ma.ino() == mb.ino()
}
#[cfg(not(unix))]
{
if ma.len() != mb.len() {
return false;
}
match (fs::canonicalize(a), fs::canonicalize(b)) {
(Ok(x), Ok(y)) => x == y || x.to_string_lossy().to_lowercase() == y.to_string_lossy().to_lowercase(),
_ => false,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn temp(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-safe-file-{tag}-{}", std::process::id()));
let _ = fs::remove_dir_all(&d);
fs::create_dir_all(&d).unwrap();
d
}
fn names(dir: &Path) -> Vec<String> {
let mut v: Vec<String> = fs::read_dir(dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
v.sort();
v
}
#[test]
fn atomic_write_replaces_and_leaves_no_temp() {
let d = temp("replace");
let p = d.join("out.jpg");
write_atomic(&p, b"one").unwrap();
write_atomic(&p, b"two two").unwrap();
assert_eq!(fs::read(&p).unwrap(), b"two two");
assert_eq!(names(&d), vec!["out.jpg"]);
let _ = fs::remove_dir_all(&d);
}
#[test]
fn a_failure_mid_write_keeps_the_old_file() {
let d = temp("midwrite");
let p = d.join("photo.jpg");
fs::write(&p, b"the original bytes").unwrap();
{
let _f = fail_writes_after(4);
let e = write_atomic(&p, &[7u8; 10_000]).unwrap_err();
assert!(e.to_string().contains("injected"), "{e}");
assert!(write_new(&d.join("new.dng"), &[1u8; 100]).is_err());
}
assert_eq!(fs::read(&p).unwrap(), b"the original bytes", "the target is untouched");
assert_eq!(names(&d), vec!["photo.jpg"], "no temp file and no partial new file");
// a fill error aborts the same way
let r = write_atomic_with(&p, &mut |w| {
w.write_all(b"half")?;
Err(io::Error::other("encoder failed"))
});
assert!(r.is_err());
assert_eq!(fs::read(&p).unwrap(), b"the original bytes");
assert_eq!(names(&d), vec!["photo.jpg"]);
let _ = fs::remove_dir_all(&d);
}
#[test]
fn new_files_never_replace_anything() {
let d = temp("new");
let p = d.join("a.dng");
fs::write(&p, b"keep").unwrap();
assert_eq!(write_new(&p, b"other").unwrap_err().kind(), io::ErrorKind::AlreadyExists);
assert_eq!(fs::read(&p).unwrap(), b"keep");
let mut c = (1..).map(|k| if k == 1 { d.join("a.dng") } else { d.join(format!("a-{k}.dng")) });
let got = write_new_unique(&mut c, b"fresh").unwrap();
assert_eq!(got, d.join("a-2.dng"));
assert_eq!(fs::read(&got).unwrap(), b"fresh");
assert_eq!(names(&d), vec!["a-2.dng", "a.dng"], "the temp file is gone");
let _ = fs::remove_dir_all(&d);
}
#[test]
fn same_file_sees_through_other_spellings() {
let d = temp("same");
let p = d.join("x.jpg");
fs::write(&p, b"x").unwrap();
fs::write(d.join("y.jpg"), b"x").unwrap();
fs::create_dir_all(d.join("sub")).unwrap();
assert!(same_file(&p, &d.join("sub/../x.jpg")));
assert!(!same_file(&p, &d.join("y.jpg")));
assert!(!same_file(&p, &d.join("missing.jpg")));
let _ = fs::remove_dir_all(&d);
}
}
+19 -5
View File
@@ -28,9 +28,12 @@ pub trait Store: Send {
self.write_atomic(name, &buf)?;
Ok(buf.len() as u64)
}
/// Append and make durable (fsync) before returning.
/// Append and make durable (fsync) before returning. On an error, part of `data` may have
/// been written (a full disk, a dropped network share): the journal cuts the file back to its
/// previous length with [`Store::truncate`] before appending again.
fn append(&mut self, name: &str, data: &[u8]) -> io::Result<()>;
/// Cut a file to `len` bytes (drop a torn tail before appending again).
/// Cut a file to `len` bytes (drop a torn tail before appending again). A missing file is
/// left missing.
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()>;
/// Human-readable location (diagnostics).
fn describe(&self) -> String;
@@ -132,15 +135,26 @@ impl Store for FsStore {
self.appender = Some((name.to_string(), f));
}
let Some((_, f)) = self.appender.as_mut() else { return Err(io::Error::other("appender not open")) };
f.write_all(data)?;
f.sync_data()
let written = f.write_all(data).and_then(|()| f.sync_data());
if written.is_err() {
// The handle may be dead (a network share that reconnected, a removed volume): the next
// attempt reopens the file. Part of `data` may be in the file; the caller cuts it back
// ([`Store::truncate`]) before appending again.
self.appender = None;
}
written
}
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()> {
if self.appender.as_ref().is_some_and(|(n, _)| n == name) {
self.appender = None;
}
let f = std::fs::OpenOptions::new().write(true).open(self.path(name))?;
let f = match std::fs::OpenOptions::new().write(true).open(self.path(name)) {
Ok(f) => f,
// nothing to cut (an append that failed before creating the file)
Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()),
Err(e) => return Err(e),
};
f.set_len(len)?;
f.sync_all()
}
+2 -1
View File
@@ -180,7 +180,8 @@ fn failed_background_snapshot_loses_nothing_and_retries_later() {
wait_written(&j);
assert!(j.poll().is_err());
assert_eq!((j.snapshot_seq(), j.stats().failed_snapshots), (0, 1));
assert!(g.files.get(SNAPSHOT).is_none());
// still the (empty) snapshot written when the library was created
assert_eq!(g.snapshot_seq_on_disk(), Some(0));
assert_recovers(&g, &live);
// no retry storm: the next attempt waits for another `max_records` records
assert!(!j.wants_snapshot());
+186
View File
@@ -0,0 +1,186 @@
//! Issue #102: catalog format versions. Older formats load and are upgraded; a newer format is
//! refused with a clear error and left untouched (never read in part, truncated or set aside).
use crate::journal::{LOG, SNAPSHOT, VERSION, encode_record};
use crate::*;
fn photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// A catalog with two photos and the log line for a third op on top of it.
fn legacy_parts() -> (Catalog, String, Catalog) {
let mut c = Catalog::new();
for _ in 0..2 {
let op = photo(&mut c);
c.apply(op).unwrap();
}
let base = c.clone();
let id = c.photos().next().unwrap().id;
let op = Op::SetRating { id, rating: 4 };
c.apply(op.clone()).unwrap();
(base, format!("{}\n", encode_record(3, &op)), c)
}
fn snapshot_version(m: &MemStore) -> u64 {
let v: serde_json::Value = serde_json::from_slice(&m.get(SNAPSHOT).unwrap()).unwrap();
v["version"].as_u64().unwrap()
}
#[test]
fn v1_library_loads_and_is_upgraded() {
let (base, log, full) = legacy_parts();
let m = MemStore::new();
m.set(SNAPSHOT, format!("{{\"format\":\"lightcraft-catalog\",\"version\":1,\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot()).into_bytes());
m.set(LOG, log.into_bytes());
let (mut j, c, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(c.to_snapshot(), full.to_snapshot());
assert_eq!((r.replayed, r.upgraded_from), (1, Some(1)));
// rewritten in this format before anything is appended
assert_eq!(snapshot_version(&m), u64::from(VERSION));
assert!(m.get(LOG).unwrap().is_empty());
assert_eq!(j.seq(), 3);
j.append(&[Op::SetLabelName { label: ColorLabel::ALL[0], name: Some("x".into()) }]).unwrap();
drop(j);
let (_, _, r) = Journal::open(Box::new(m)).unwrap();
assert_eq!((r.replayed, r.upgraded_from), (1, None));
}
#[test]
fn versionless_log_only_library_is_upgraded() {
let mut c = Catalog::new();
let op = photo(&mut c);
c.apply(op.clone()).unwrap();
let m = MemStore::new();
m.set(LOG, format!("{}\n", encode_record(1, &op)).into_bytes());
let (_, loaded, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
assert_eq!(r.upgraded_from, Some(0));
assert_eq!(snapshot_version(&m), u64::from(VERSION));
}
/// A new library gets a snapshot in this format at once, so an older build never sees a bare log
/// it would read in part.
#[test]
fn new_library_is_written_in_this_format() {
let m = MemStore::new();
let (_, _, r) = Journal::open(Box::new(m.clone())).unwrap();
assert!(r.created);
assert_eq!(r.upgraded_from, None);
assert_eq!(snapshot_version(&m), u64::from(VERSION));
}
fn assert_refused_untouched(m: &MemStore) {
let before = m.files.lock().unwrap().clone();
let e = Journal::open(Box::new(m.clone())).err().expect("refused");
assert!(matches!(e, CatalogError::Newer(_)), "{e:?}");
let msg = e.to_string();
assert!(msg.contains("newer version of LightCraft") && msg.contains("left unchanged"), "{msg}");
assert_eq!(*m.files.lock().unwrap(), before, "nothing modified, nothing renamed");
}
#[test]
fn newer_snapshot_is_refused_untouched() {
let (base, log, _) = legacy_parts();
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":2,\"catalog\":{}}}\n", VERSION + 1, base.to_snapshot()).into_bytes(),
);
m.set(LOG, log.into_bytes());
assert_refused_untouched(&m);
// even when its catalog doesn't parse as this version's
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":2,\"catalog\":{{\"photos\":7}}}}\n", VERSION + 1).into_bytes(),
);
assert_refused_untouched(&m);
let e = Journal::open(Box::new(m)).err().unwrap().to_string();
assert!(e.contains(&format!("v{}", VERSION + 1)), "{e}");
}
/// A log record with a valid CRC whose op this version doesn't know: refused, not taken for a
/// torn tail (truncated) or damage (renamed) — wherever it is in the log.
#[test]
fn unknown_op_in_the_log_is_refused_untouched() {
let (base, log, _) = legacy_parts();
let future = |seq: u64| {
let body = r#"{"op":"fromTheFuture","id":1,"what":"something new"}"#;
format!("{{\"seq\":{seq},\"crc\":{},\"op\":{body}}}\n", crc32fast::hash(body.as_bytes()))
};
let snap = format!("{{\"format\":\"lightcraft-catalog\",\"version\":{VERSION},\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot());
let id = base.photos().next().unwrap().id;
let after = format!("{}\n", encode_record(5, &Op::SetRating { id, rating: 1 }));
for log in [format!("{log}{}", future(4)), format!("{log}{}{after}", future(4)), format!("{}{log}", future(3))] {
let m = MemStore::new();
m.set(SNAPSHOT, snap.clone().into_bytes());
m.set(LOG, log.into_bytes());
assert_refused_untouched(&m);
}
// a known op with a field of an unknown shape is "newer" too
let body = r#"{"op":"setRating","id":1,"rating":"five stars"}"#;
let m = MemStore::new();
m.set(SNAPSHOT, snap.into_bytes());
m.set(LOG, format!("{log}{{\"seq\":4,\"crc\":{},\"op\":{body}}}\n", crc32fast::hash(body.as_bytes())).into_bytes());
assert_refused_untouched(&m);
}
/// A CRC mismatch is still damage / a torn tail (not "newer").
#[test]
fn bad_crc_is_still_a_torn_tail() {
let (base, log, _) = legacy_parts();
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{VERSION},\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot()).into_bytes(),
);
let body = r#"{"op":"fromTheFuture"}"#;
m.set(LOG, format!("{log}{{\"seq\":4,\"crc\":1,\"op\":{body}}}\n").into_bytes());
let (_, _, r) = Journal::open(Box::new(m)).unwrap();
assert!(r.torn_bytes > 0 && r.damaged.is_none(), "{r:?}");
}
/// Guard for the version rule (see `journal` → *Format versions*): adding an [`Op`] variant breaks
/// this match. When it does, bump `journal::VERSION`, add a row to the format table, and add the
/// variant here under the new version.
#[test]
fn op_variants_are_versioned() {
fn since(op: &Op) -> u32 {
match op {
Op::AddPhoto { .. }
| Op::RemovePhoto { .. }
| Op::SetRating { .. }
| Op::SetFlag { .. }
| Op::SetLabel { .. }
| Op::SetDevelop { .. }
| Op::SetMeta { .. }
| Op::SetDeleted { .. }
| Op::SetLocal { .. }
| Op::SetVersions { .. }
| Op::SetHistory { .. }
| Op::PushHistory { .. }
| Op::AddAlbum { .. }
| Op::RemoveAlbum { .. }
| Op::RenameAlbum { .. }
| Op::MoveAlbum { .. }
| Op::SetAlbumPhotos { .. }
| Op::SetAlbumCover { .. }
| Op::SetAlbumRules { .. }
| Op::AddStack { .. }
| Op::RemoveStack { .. }
| Op::SetStack { .. }
| Op::SetCaptured { .. }
| Op::SetAnalysis { .. }
| Op::SetFile { .. }
| Op::Relink { .. }
| Op::SetContent { .. }
| Op::SetLabelName { .. }
| Op::Batch { .. } => 1,
Op::SetBrowsed { .. } => 2,
}
}
let newest = since(&Op::SetBrowsed { folder: String::new(), at: None });
assert_eq!(newest, VERSION, "the newest op's version must be the current format version");
}
+6 -1
View File
@@ -283,7 +283,12 @@ fn streamed_snapshot_keeps_the_on_disk_format() {
let (mut j, c, _) = open(&m);
assert_eq!(c.to_snapshot(), full.to_snapshot());
j.snapshot(&c).unwrap();
let legacy = format!("{{\"format\":\"lightcraft-catalog\",\"version\":1,\"seq\":{},\"catalog\":{}}}\n", j.seq(), c.to_snapshot());
let legacy = format!(
"{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":{},\"catalog\":{}}}\n",
crate::journal::VERSION,
j.seq(),
c.to_snapshot()
);
assert_eq!(String::from_utf8(m.get(SNAPSHOT).unwrap()).unwrap(), legacy);
assert_eq!(j.stats().last_snapshot.bytes, legacy.len() as u64);
+267
View File
@@ -0,0 +1,267 @@
//! Forgetting untouched Local browse records (see [`crate::local`]).
use std::sync::Arc;
use lightcraft_develop::DevelopSettings;
use crate::*;
const OLD: &str = "2026-08-01T10:00:00";
const NOW: &str = "2026-10-05T10:00:00";
/// A Local record of `/pics/<folder>/IMG_<i>.JPG` as a browse catalogues it (with its baseline).
fn local(c: &mut Catalog, folder: &str, i: u32) -> PhotoId {
let id = c.alloc_photo_id();
let mut p = Photo::new(id, Source::File { path: format!("/pics/{folder}/IMG_{i}.JPG") }, &format!("IMG_{i}.JPG"), "JPEG", 60, 40, OLD);
p.meta.camera = "Cam".into();
p.meta.creator = "From EXIF".into(); // file metadata is part of the baseline, not a user change
p.focal_and_wb_for_test();
p.local = true;
p.set_local_baseline();
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
id
}
impl Photo {
fn focal_and_wb_for_test(&mut self) {
self.meta.focal_mm = Some(35.3);
self.as_shot_wb = Some((5123.4, 3.1));
}
}
fn browsed(c: &mut Catalog, folder: &str, at: &str) {
c.apply(Op::SetBrowsed { folder: format!("/pics/{folder}"), at: Some(at.into()) }).unwrap();
}
fn plan(c: &Catalog, days: u32) -> ForgetPlan {
c.forget_local_plan(NOW, days, &|_| false)
}
fn run(c: &mut Catalog, days: u32) -> ForgetPlan {
let p = plan(c, days);
for op in p.ops() {
c.apply(op).unwrap();
}
p
}
#[test]
fn untouched_records_of_old_folders_are_forgotten() {
let mut c = Catalog::new();
let old: Vec<_> = (0..5).map(|i| local(&mut c, "old", i)).collect();
let recent: Vec<_> = (0..3).map(|i| local(&mut c, "recent", i)).collect();
browsed(&mut c, "old", OLD);
browsed(&mut c, "recent", "2026-10-01T09:00:00");
// a library photo in the old folder is never a candidate
let lib = c.alloc_photo_id();
let mut p = Photo::new(lib, Source::File { path: "/pics/old/LIB.JPG".into() }, "LIB.JPG", "JPEG", 6, 4, OLD);
p.set_local_baseline();
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
let dry = plan(&c, 30);
assert_eq!((dry.local, dry.evict.len(), dry.kept_recent), (8, 5, 3), "{dry:?}");
assert_eq!(dry.evict, old);
assert_eq!(c.len(), 9, "a plan changes nothing");
let done = run(&mut c, 30);
assert_eq!(done.evict.len(), 5);
assert!(old.iter().all(|id| c.photo(*id).is_none()));
assert!(recent.iter().all(|id| c.photo(*id).is_some()));
assert!(c.photo(lib).is_some());
// the folder has no Local records left: its time goes
assert_eq!(done.unstamp, vec!["/pics/old".to_string()]);
assert_eq!(c.last_browsed("/pics/old"), None);
assert_eq!(c.last_browsed("/pics/recent/"), Some("2026-10-01T09:00:00"));
// 0 days = never
browsed(&mut c, "recent", OLD);
assert!(plan(&c, 0).evict.is_empty());
assert_eq!(plan(&c, 30).evict.len(), 3);
}
#[test]
fn any_user_change_keeps_a_record() {
type Change = Box<dyn Fn(&mut Catalog, PhotoId)>;
let set = |f: fn(&mut Photo)| -> Change {
Box::new(move |c: &mut Catalog, id: PhotoId| {
let mut p = (**c.photo(id).unwrap()).clone();
f(&mut p);
c.apply(Op::RemovePhoto { id }).unwrap();
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
})
};
let op = |f: fn(PhotoId) -> Op| -> Change { Box::new(move |c: &mut Catalog, id: PhotoId| drop(c.apply(f(id)).unwrap())) };
let edited = || {
let mut d = DevelopSettings::default();
d.light.exposure = 0.3;
Arc::new(d)
};
let changes: Vec<(&str, Change)> = vec![
("rating", op(|id| Op::SetRating { id, rating: 3 })),
("flag", op(|id| Op::SetFlag { id, flag: Flag::Reject })),
("label", op(|id| Op::SetLabel { id, label: Some(ColorLabel::Red) })),
("develop", Box::new(move |c, id| drop(c.apply(Op::SetDevelop { id, settings: edited(), label: "Exposure".into(), edited: None }).unwrap()))),
("crop", set(|p| Arc::make_mut(&mut p.develop).crop.flip_h = true)),
(
"history only",
Box::new(|c, id| {
let step = HistoryStep { label: "Reset".into(), settings: Arc::new(DevelopSettings::default()) };
drop(c.apply(Op::PushHistory { id, step }).unwrap())
}),
),
("versions", set(|p| p.versions.push(Version { name: "v".into(), created: NOW.into(), settings: p.develop.clone(), auto: false }))),
("title", set(|p| p.meta.title = "T".into())),
("caption", set(|p| p.meta.caption = "C".into())),
("keywords", set(|p| p.meta.keywords.push("k".into()))),
("location", set(|p| p.meta.city = "Oslo".into())),
("gps", set(|p| p.meta.gps = Some((1.0, 2.0)))),
("copyright", set(|p| p.meta.copyright = "© me".into())),
("creator edit", set(|p| p.meta.creator = "Me".into())),
("capture time", op(|id| Op::SetCaptured { id, captured: Some("2020-01-01T00:00:00".into()) })),
("rename", set(|p| p.file_name = "renamed.JPG".into())),
("culling", op(|id| Op::SetAnalysis { id, analysis: Some(Analysis::default()) })),
("deleted", op(|id| Op::SetDeleted { id, deleted: true })),
("pick", op(|id| Op::SetFlag { id, flag: Flag::Pick })),
(
"album",
Box::new(|c, id| {
let a = c.alloc_album_id();
let mut al = Album::new(a, "A");
al.photos.push(id);
drop(c.apply(Op::AddAlbum { album: al }).unwrap())
}),
),
(
"stack",
Box::new(|c, id| {
let other = local(c, "elsewhere", 99);
let s = c.alloc_stack_id();
drop(c.apply(Op::AddStack { stack: Stack { id: s, photos: vec![id, other], collapsed: false } }).unwrap())
}),
),
(
"virtual copy",
Box::new(|c, id| {
let mut v = (**c.photo(id).unwrap()).clone();
v.id = c.alloc_photo_id();
v.copy_of = Some(id);
v.copy_name = Some("Copy 1".into());
v.local = false;
drop(c.apply(Op::AddPhoto { photo: Box::new(v) }).unwrap())
}),
),
];
for (what, change) in changes {
let mut c = Catalog::new();
let id = local(&mut c, "old", 1);
let control = local(&mut c, "old", 2);
browsed(&mut c, "old", OLD);
change(&mut c, id);
let p = plan(&c, 30);
assert_eq!(p.evict, vec![control], "{what}: {p:?}");
assert_eq!(p.kept_touched + p.kept_in_use, 1, "{what}: {p:?}");
// the stamp stays: a record in the folder remains
assert!(p.unstamp.is_empty(), "{what}");
}
// a change undone (back to the browse state) is untouched again
let mut c = Catalog::new();
let id = local(&mut c, "old", 1);
browsed(&mut c, "old", OLD);
let inv = c.apply(Op::SetRating { id, rating: 2 }).unwrap();
assert!(plan(&c, 30).evict.is_empty());
c.apply(inv).unwrap();
assert_eq!(plan(&c, 30).evict, vec![id]);
}
/// Catalogs from before last-browsed times (and baselines): nothing is forgotten on the first
/// run — the folders are stamped now — and baseline-less records are judged by their data.
#[test]
fn upgrade_without_times_forgets_nothing_then_counts_from_then() {
let mut c = Catalog::new();
let a = local(&mut c, "a", 1);
// legacy records: no baseline
let legacy = |c: &mut Catalog, i: u32, f: fn(&mut Photo)| {
let id = c.alloc_photo_id();
let mut p = Photo::new(id, Source::File { path: format!("/pics/a/L{i}.JPG") }, "L.JPG", "JPEG", 6, 4, OLD);
p.meta.camera = "Cam".into();
p.local = true;
f(&mut p);
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
id
};
let clean = legacy(&mut c, 1, |_| {});
let rated = legacy(&mut c, 2, |p| p.rating = 1);
let with_creator = legacy(&mut c, 3, |p| p.meta.creator = "maybe the user".into());
let p = run(&mut c, 30);
assert!(p.evict.is_empty(), "{p:?}");
assert_eq!(p.stamp, vec!["/pics/a".to_string()]);
assert_eq!(c.last_browsed("/pics/a"), Some(NOW));
// 30+ days later
let later = c.forget_local_plan("2026-11-06T10:00:00", 30, &|_| false);
let mut evict = later.evict.clone();
evict.sort();
assert_eq!(evict, vec![a, clean]);
assert_eq!(later.kept_touched, 2, "{rated:?} {with_creator:?}");
}
#[test]
fn in_use_records_are_kept() {
let mut c = Catalog::new();
let a = local(&mut c, "old", 1);
let b = local(&mut c, "old", 2);
browsed(&mut c, "old", OLD);
let p = c.forget_local_plan(NOW, 30, &|p| p.id == a);
assert_eq!((p.evict.clone(), p.kept_in_use), (vec![b], 1));
}
/// The ops are journaled: a crash right after forgetting reloads the same catalog, and a
/// fingerprint survives the snapshot round trip (a reloaded record is still untouched).
#[test]
fn forgetting_is_journaled_and_replays() {
let m = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let mut ops = Vec::new();
for i in 0..10 {
local(&mut c, if i % 2 == 0 { "old" } else { "new" }, i);
}
// log what was built so far: the ops are the records as added
for p in c.photos() {
ops.push(Op::AddPhoto { photo: Box::new((**p).clone()) });
}
ops.push(Op::SetBrowsed { folder: "/pics/old".into(), at: Some(OLD.into()) });
ops.push(Op::SetBrowsed { folder: "/pics/new".into(), at: Some(NOW.into()) });
browsed(&mut c, "old", OLD);
browsed(&mut c, "new", NOW);
j.append(&ops).unwrap();
j.snapshot(&c).unwrap();
drop(j);
// reload from the snapshot: baselines still match
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let p = c.forget_local_plan(NOW, 30, &|_| false);
assert_eq!(p.evict.len(), 5, "{p:?}");
let ops = p.ops();
for op in &ops {
c.apply(op.clone()).unwrap();
}
j.append(&ops).unwrap();
let expect = c.to_snapshot();
drop(j); // crash: replay the log
let (_, c2, r) = Journal::open(Box::new(m)).unwrap();
assert_eq!(r.failed, 0);
assert_eq!(c2.to_snapshot(), expect);
assert_eq!(c2.len(), 5);
// nothing left to forget
assert!(c2.forget_local_plan(NOW, 30, &|_| false).evict.is_empty());
}
#[test]
fn op_photo_ids_cover_nested_ops() {
let op = Op::Batch {
ops: vec![
Op::SetRating { id: PhotoId(1), rating: 1 },
Op::SetAlbumPhotos { id: AlbumId(1), photos: vec![PhotoId(2), PhotoId(3)] },
Op::Batch { ops: vec![Op::RemovePhoto { id: PhotoId(4) }] },
],
};
let mut ids = Vec::new();
op.photo_ids(&mut |id| ids.push(id.0));
assert_eq!(ids, vec![1, 2, 3, 4]);
}
+93
View File
@@ -0,0 +1,93 @@
//! Issue #99: one process per library.
use std::path::PathBuf;
use crate::lock::{LOCK, OWNER};
use crate::*;
fn temp_dir(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-lock-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn second_opener_is_refused_until_the_first_lets_go() {
let dir = temp_dir("twice");
let first = LibraryLock::acquire(&dir, "LightCraft").unwrap();
assert!(first.held());
let owner: LockOwner = serde_json::from_slice(&std::fs::read(dir.join(OWNER)).unwrap()).unwrap();
assert_eq!((owner.pid, owner.program.as_str()), (std::process::id(), "LightCraft"));
let e = LibraryLock::acquire(&dir, "lightcraft-cli").unwrap_err();
let LockError::InUse(Some(who)) = &e else { panic!("{e:?}") };
assert_eq!(who.pid, std::process::id());
let msg = e.to_string();
assert!(msg.contains("already open in LightCraft") && msg.contains(&format!("process {}", std::process::id())), "{msg}");
drop(first);
assert!(!dir.join(OWNER).exists(), "the owner note goes with the lock");
assert!(dir.join(LOCK).exists(), "the lock file itself may stay: only a held lock refuses");
let again = LibraryLock::acquire(&dir, "lightcraft-cli").unwrap();
assert!(again.held());
drop(again);
let _ = std::fs::remove_dir_all(&dir);
}
/// Files left behind by a crash (lock file + owner note) don't lock anyone out.
#[test]
fn leftover_lock_files_are_not_a_lock() {
let dir = temp_dir("stale");
std::fs::write(dir.join(LOCK), b"").unwrap();
std::fs::write(dir.join(OWNER), br#"{"pid":999999,"host":"elsewhere","program":"LightCraft","version":"0.2.0","since":1}"#).unwrap();
let l = LibraryLock::acquire(&dir, "LightCraft").unwrap();
assert!(l.held());
let owner: LockOwner = serde_json::from_slice(&std::fs::read(dir.join(OWNER)).unwrap()).unwrap();
assert_eq!(owner.pid, std::process::id(), "the note now names this process");
drop(l);
let _ = std::fs::remove_dir_all(&dir);
}
/// Another process holds the lock; when it is killed (a crash), the OS releases it.
#[test]
fn lock_held_by_another_process_is_released_when_it_dies() {
let dir = temp_dir("child");
let exe = std::env::current_exe().unwrap();
let mut child = std::process::Command::new(exe)
.args(["--exact", "tests_lock::lock_holder_child", "--ignored", "--test-threads=1"])
.env("LC_LOCK_CHILD_DIR", &dir)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.unwrap();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60);
while !dir.join("child-locked").exists() {
assert!(child.try_wait().unwrap().is_none(), "child exited before locking");
assert!(std::time::Instant::now() < deadline, "child never locked");
std::thread::sleep(std::time::Duration::from_millis(20));
}
let e = LibraryLock::acquire(&dir, "LightCraft").unwrap_err();
let LockError::InUse(Some(who)) = &e else { panic!("{e:?}") };
assert_eq!(who.pid, child.id());
assert!(e.to_string().contains("on this computer") || who.host.is_empty(), "{e}");
child.kill().unwrap(); // SIGKILL: no clean-up runs, like a crash
child.wait().unwrap();
assert!(dir.join(OWNER).exists(), "the crashed holder's note is left behind");
let l = LibraryLock::acquire(&dir, "LightCraft").expect("the OS released the dead process's lock");
assert!(l.held());
drop(l);
let _ = std::fs::remove_dir_all(&dir);
}
/// Run by [`lock_held_by_another_process_is_released_when_it_dies`] in a child process: take the
/// lock, say so, and wait to be killed.
#[test]
#[ignore = "helper process for lock_held_by_another_process_is_released_when_it_dies"]
fn lock_holder_child() {
let Some(dir) = std::env::var_os("LC_LOCK_CHILD_DIR") else { return };
let _l = LibraryLock::acquire(std::path::Path::new(&dir), "lightcraft-cli").unwrap();
std::fs::write(std::path::Path::new(&dir).join("child-locked"), b"").unwrap();
std::thread::sleep(std::time::Duration::from_secs(120));
}
+223
View File
@@ -0,0 +1,223 @@
//! Issue #101: an append that fails part-way (disk full, a dropped network share) and is then
//! retried must not read back as log damage, and must lose nothing.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use crate::journal::LOG;
use crate::*;
/// A store whose next append writes only a prefix of its data, then fails.
#[derive(Clone, Default)]
struct Torn {
files: MemStore,
/// The next append writes this many bytes, then returns an error.
torn_at: Arc<Mutex<Option<usize>>>,
/// `truncate` fails (the cut-back after a failed append can't be done right away).
truncate_fails: Arc<AtomicBool>,
}
impl Torn {
fn tear_next_append(&self, bytes: usize) {
*self.torn_at.lock().unwrap() = Some(bytes);
}
}
impl Store for Torn {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if let Some(n) = self.torn_at.lock().unwrap().take() {
self.files.append(name, &data[..n.min(data.len())])?;
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
if self.truncate_fails.load(Ordering::SeqCst) {
return Err(std::io::Error::other("share unavailable"));
}
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"torn".into()
}
}
fn add_photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// Apply `ops` to the live catalog (like a command) and return them for the journal.
fn apply(c: &mut Catalog, ops: Vec<Op>) -> Vec<Op> {
for op in &ops {
c.apply(op.clone()).unwrap();
}
ops
}
/// Every tear position of a batch, then the retry (same ops, same seqs, like the engine's queue)
/// and more appends: reloading replays everything, reports no damage and no torn bytes.
#[test]
fn partial_append_then_retry_loads_cleanly() {
let probe = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(probe.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
let before = probe.get(LOG).unwrap().len();
j.append(&batch).unwrap();
let batch_len = probe.get(LOG).unwrap().len() - before;
for tear in 1..batch_len {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
t.tear_next_append(tear);
assert!(j.append(&batch).is_err(), "tear at {tear}");
// cut back to the last whole record
assert_eq!(t.files.get(LOG).unwrap().len(), good_len, "tear at {tear}");
j.append(&batch).unwrap();
let more = apply(&mut c, vec![Op::SetLabel { id, label: Some(ColorLabel::ALL[0]) }]);
j.append(&more).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!((r.torn_bytes, r.torn_fragments, r.replayed), (0, 0, 5), "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
}
/// When cutting back fails too (the share is gone), the next append cuts back first; until that
/// works nothing more is appended after the fragment.
#[test]
fn failed_cut_back_is_done_before_the_next_append() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetRating { id, rating: 5 }]);
t.truncate_fails.store(true, Ordering::SeqCst);
t.tear_next_append(30);
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30, "the fragment is still there");
// still unreachable: the retry fails without writing anything after the fragment
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30);
t.truncate_fails.store(false, Ordering::SeqCst);
j.append(&batch).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "{r:?}");
assert_eq!(r.replayed, 3);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// Logs written before the fix: the fragment of the failed append is glued to the first retried
/// record on one line, followed by the rest of the retry. They load in full (the fragment is
/// skipped), instead of stopping at the fragment and setting the later records aside as damage.
#[test]
fn legacy_fragment_glued_to_the_retry_is_skipped() {
let m = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good = m.get(LOG).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(
&mut c,
vec![
Op::SetRating { id, rating: 3 },
Op::SetFlag { id, flag: Flag::Pick },
Op::SetLabelName { label: ColorLabel::ALL[1], name: Some("Größe ✓".into()) },
],
);
j.append(&batch).unwrap();
let full = m.get(LOG).unwrap();
let batch_bytes = &full[good.len()..];
let later = apply(&mut c, vec![Op::SetRating { id, rating: 1 }]);
j.append(&later).unwrap();
let later_bytes = m.get(LOG).unwrap()[full.len()..].to_vec();
drop(j);
// every tear (inside the first record, after it, inside a multi-byte character…)
for tear in 1..batch_bytes.len() {
if batch_bytes[tear - 1] == b'\n' {
continue; // a tear on a record boundary leaves no fragment
}
let mut log = good.clone();
log.extend_from_slice(&batch_bytes[..tear]);
log.extend_from_slice(batch_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, loaded, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!(r.torn_fragments, 1, "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
// a damaged record whose tail happens to be a whole record from the future is still damage
let mut log = good.clone();
log.extend_from_slice(b"garbage");
log.extend_from_slice(&later_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, _, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_some(), "{r:?}");
}
/// A snapshot can absorb ops whose append failed: it records them as saved (`seq` moves past
/// them), and appending after it continues from there.
#[test]
fn snapshot_with_unlogged_ops_saves_them() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let queued = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetFlag { id, flag: Flag::Reject }]);
t.tear_next_append(10);
assert!(j.append(&queued).is_err());
j.snapshot_with_unlogged(&c, queued.len() as u64).unwrap();
assert_eq!(j.seq(), 3);
let after = apply(&mut c, vec![Op::SetRating { id, rating: 5 }]);
j.append(&after).unwrap();
drop(j);
let (j, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert_eq!((r.snapshot_seq, r.replayed), (3, 1));
assert_eq!(j.seq(), 4);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// `FsStore::truncate` on a file that was never created (an append that failed first) is a no-op.
#[test]
fn fs_store_truncates_a_missing_file_quietly() {
let dir = std::env::temp_dir().join(format!("lc-torn-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
let mut s = FsStore::open(&dir).unwrap();
s.truncate(LOG, 0).unwrap();
assert!(!dir.join(LOG).exists());
s.append(LOG, b"abc\n").unwrap();
s.truncate(LOG, 0).unwrap();
assert_eq!(std::fs::read(dir.join(LOG)).unwrap(), b"");
let _ = std::fs::remove_dir_all(&dir);
}
+1
View File
@@ -17,6 +17,7 @@ lightcraft-scenes = { workspace = true }
lightcraft-pipeline = { workspace = true }
lightcraft-gpu = { workspace = true }
lightcraft-catalog = { workspace = true }
web-time = { workspace = true }
lightcraft-preview = { workspace = true }
lightcraft-codecs = { workspace = true }
lightcraft-raw = { workspace = true }
+177 -35
View File
@@ -2,6 +2,11 @@
//! The folder's photos are probed in place as `local` photos — only folder views list them —
//! and edits go to the files' XMP sidecars like any photo added in place. "Add to My Photos"
//! makes them ordinary library photos.
//!
//! Each browse records when the folders it listed were browsed. Untouched records of folders
//! not browsed for [`Session::forget_local_days`] are forgotten (when the library opens, or with
//! `library.forgetLocal`); see [`lightcraft_catalog::local`] for what "untouched" means. Files
//! and sidecars on disk are never touched, and browsing the folder again brings them back.
use std::path::Path;
@@ -35,12 +40,113 @@ fn browse(s: &mut Session, p: &Value) -> Result<Value> {
v
};
let report = import_with(s, &files, &ImportOptions { mode: ImportMode::Add, local: true, ..Default::default() })?;
stamp_browsed(s, &dir_s, &files);
s.browse = Some(Browse { path: dir_s.clone(), subfolders });
s.source = LibrarySource::Folder;
let shown = s.visible().len();
Ok(json!({"path": dir_s, "subfolders": subfolders, "photos": shown, "new": report.imported.len(), "failed": report.failed.len()}))
}
/// Folders whose browse time is refreshed at most this often (keeps repeated browsing from
/// growing the log; far finer than the days the eviction counts in).
const RESTAMP_SECS: i64 = 3600;
/// Record that `root` and the folders of `files` were browsed now (journaled, not an undo step).
fn stamp_browsed(s: &mut Session, root: &str, files: &[String]) {
let now = (s.clock)();
let now_s = lightcraft_catalog::stacks::iso_seconds(&now);
let mut folders: std::collections::BTreeSet<String> =
files.iter().filter_map(|f| Path::new(f).parent().map(|d| d.to_string_lossy().trim_end_matches(['/', '\\']).to_string())).collect();
folders.insert(root.to_string());
let ops: Vec<Op> = folders
.into_iter()
.filter(|f| {
let at = s.catalog.last_browsed(f).and_then(lightcraft_catalog::stacks::iso_seconds);
match (at, now_s) {
(Some(at), Some(n)) => n - at >= RESTAMP_SECS || n < at,
_ => true,
}
})
.map(|folder| Op::SetBrowsed { folder, at: Some(now.clone()) })
.collect();
if !ops.is_empty() {
let op = Op::Batch { ops };
if s.catalog.apply(op.clone()).is_ok() {
s.pending_log.push(op);
}
}
}
/// Photos the session still needs, whatever their state: shown, selected, being edited, or
/// referred to by an undo / redo step.
fn in_use(s: &mut Session) -> std::collections::HashSet<PhotoId> {
let mut keep: std::collections::HashSet<PhotoId> = s.visible().iter().copied().collect();
keep.extend(s.selection.ids.iter().copied());
keep.extend(s.selection.active);
keep.extend(s.previous_active);
keep.extend(s.interaction.as_ref().map(|i| i.photo));
keep.extend(s.before.keys().copied());
for e in s.undo.iter().chain(s.redo.iter()) {
e.op.photo_ids(&mut |id| {
keep.insert(id);
});
}
keep
}
impl Session {
/// Forget untouched Local records whose folder wasn't browsed for `days` (default
/// [`Session::forget_local_days`]; 0 = never). With `dry_run`, only report what would go.
/// Journaled (durable, replayed after a crash) but not an undo step; nothing on disk changes.
pub fn forget_local(&mut self, dry_run: bool, days: Option<u32>) -> lightcraft_catalog::ForgetPlan {
let days = days.unwrap_or(self.forget_local_days);
let now = (self.clock)();
let keep = in_use(self);
let smart = self.media.smart_dir.clone();
let plan = self
.catalog
.forget_local_plan(&now, days, &|p| keep.contains(&p.id) || smart.as_ref().is_some_and(|d| d.join(crate::smart::file_name(p)).exists()));
if dry_run {
return plan;
}
for op in plan.ops() {
match self.catalog.apply(op.clone()) {
Ok(_) => self.pending_log.push(op),
Err(e) => log::warn!("forget Local records: {e}"),
}
}
if !plan.evict.is_empty() {
log::info!("forgot {} untouched Local record(s) of folders not browsed for {days} days", plan.evict.len());
}
plan
}
}
fn forget_local(s: &mut Session, p: &Value) -> Result<Value> {
let dry_run = bool_or(p, "dryRun", false);
let days = match p.get("days") {
None | Some(Value::Null) => None,
Some(v) => {
Some(v.as_u64().ok_or_else(|| bad("library.forgetLocal", "days must be a whole number (0 = never)"))?.min(u64::from(u32::MAX)) as u32)
}
};
let used = days.unwrap_or(s.forget_local_days);
let plan = s.forget_local(dry_run, days);
if !dry_run && !plan.evict.is_empty() {
s.compact_soon();
}
Ok(json!({
"dryRun": dry_run,
"days": used,
"local": plan.local,
"forgotten": plan.evict.len(),
"keptRecent": plan.kept_recent,
"keptTouched": plan.kept_touched,
"keptInUse": plan.kept_in_use,
"foldersStamped": plan.stamp.len(),
}))
}
/// The browsed photos among `ids` (default: the selection) become library photos.
fn add_to_library(s: &mut Session, p: &Value) -> Result<Value> {
let ids: Vec<PhotoId> = s.targets(p).into_iter().filter(|id| s.catalog.photo(*id).is_some_and(|ph| ph.local)).collect();
@@ -60,7 +166,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Rename Folder",
[],
None,
"{path, name} — rename a folder on disk (its files and sidecars go along) and relink the photos in it; not an undo step → {path, relinked}",
"{path, name} — rename a folder on disk (its files and sidecars go along) and relink the photos in it; one undo step (undo renames it back, refused if the old name is taken) → {path, relinked}",
always,
|s, p| {
let from = str_param(p, "path").ok_or_else(|| bad("folder.rename", "missing path"))?.trim_end_matches(['/', '\\']).to_string();
@@ -78,7 +184,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Move Folder",
[],
None,
"{path, into: destination folder} — move a folder on disk into another and relink the photos in it; not an undo step → {path, relinked}",
"{path, into: destination folder} — move a folder on disk into another and relink the photos in it; one undo step (undo moves it back, refused if the old place is taken) → {path, relinked}",
always,
|s, p| {
let from = str_param(p, "path").ok_or_else(|| bad("folder.move", "missing path"))?.trim_end_matches(['/', '\\']).to_string();
@@ -98,62 +204,98 @@ pub fn specs() -> Vec<CommandSpec> {
always,
browse
),
cmd!(
"library.forgetLocal",
"Forget Unchanged Local Photos",
[],
None,
"{dryRun?: bool, days?: n (default: the library preference, 0 = never)} — remove Local (browsed, not added) photos nobody changed, of folders not browsed for `days`, from the catalog; files and sidecars stay, browsing the folder again brings them back; not an undo step → {local, forgotten, keptRecent, keptTouched, keptInUse, foldersStamped, days, dryRun}",
always,
forget_local
),
cmd!("photo.addToLibrary", "Add to My Photos", ["Photo"], None, "{ids?} — browsed (Local) photos join the library", always, add_to_library),
]
}
/// Rename or move a folder on disk (everything in it goes along, sidecars included) and relink
/// the photos inside. Like a rename in the file manager it is not an undo step: undoing would
/// point the photos back at a folder that no longer exists.
pub(crate) fn move_folder(s: &mut crate::Session, from: &str, to: &str) -> std::result::Result<usize, String> {
use std::path::Path;
/// Rename `from` to `to` on disk (a folder; everything in it goes along). Never overwrites:
/// refused when `to` exists (a case-only rename on a case-insensitive file system goes through
/// a temporary name). Missing parents of `to` are created.
pub(crate) fn rename_folder_on_disk(from: &str, to: &str) -> std::result::Result<(), String> {
let (src, dst) = (Path::new(from), Path::new(to));
if !src.is_dir() {
return Err(format!("{from} is not a folder"));
return Err(format!("{from} is not a folder (moved or deleted?)"));
}
if dst.exists() {
let case_only = from != to && from.to_lowercase() == to.to_lowercase();
if dst.exists() && !case_only {
return Err(format!("{to} already exists"));
}
if dst.starts_with(src) {
return Err("a folder can't move into itself".into());
}
if s.library.as_ref().is_some_and(|l| l.dir.starts_with(src) || src.starts_with(&l.dir)) {
return Err("the library's own folders can't be moved here".into());
}
if let Some(parent) = dst.parent() {
std::fs::create_dir_all(parent).map_err(|e| format!("{}: {e}", parent.display()))?;
}
std::fs::rename(src, dst).map_err(|e| format!("{from} → {to}: {e}"))?;
let mut n = 0;
let ops: Vec<lightcraft_catalog::Op> = s
if case_only {
let tmp = src.with_file_name(format!(".lc-rename-{}", std::process::id()));
if tmp.exists() {
return Err(format!("{} already exists", tmp.display()));
}
std::fs::rename(src, &tmp).map_err(|e| format!("{from} → {to}: {e}"))?;
return std::fs::rename(&tmp, dst).map_err(|e| {
let _ = std::fs::rename(&tmp, src);
format!("{from} → {to}: {e}")
});
}
std::fs::rename(src, dst).map_err(|e| format!("{from} → {to}: {e}"))
}
/// After a folder moved from `from` to `to`: a browsed folder at or below it follows.
pub(crate) fn follow_folder(s: &mut Session, from: &str, to: &str) {
if let Some(b) = &mut s.browse
&& let Ok(rel) = Path::new(&b.path).strip_prefix(from)
{
b.path = Path::new(to).join(rel).to_string_lossy().to_string();
let f = b.path.clone();
s.filter.folder = Some(f);
}
}
/// Rename or move a folder on disk (everything in it goes along, sidecars included) and relink
/// the photos inside, as one undo step: undo renames the folder back and restores the photos'
/// paths, redo repeats the move — each refused (and reported) when the destination is taken.
pub(crate) fn move_folder(s: &mut Session, from: &str, to: &str) -> std::result::Result<usize, String> {
let (src, dst) = (Path::new(from), Path::new(to));
if !src.is_dir() {
return Err(format!("{from} is not a folder"));
}
if s.library.as_ref().is_some_and(|l| l.dir.starts_with(src) || src.starts_with(&l.dir)) {
return Err("the library's own folders can't be moved here".into());
}
rename_folder_on_disk(from, to)?;
let ops: Vec<Op> = s
.catalog
.photos()
.filter_map(|p| match &p.source {
lightcraft_catalog::Source::File { path } => Path::new(path).strip_prefix(src).ok().map(|rel| (p.id, p.file_name.clone(), dst.join(rel))),
_ => None,
})
.map(|(id, file_name, np)| {
n += 1;
lightcraft_catalog::Op::Relink {
id,
file_name,
source: lightcraft_catalog::Source::File { path: np.to_string_lossy().to_string() },
format: None,
}
.map(|(id, file_name, np)| Op::Relink {
id,
file_name,
source: lightcraft_catalog::Source::File { path: np.to_string_lossy().to_string() },
format: None,
})
.collect();
for op in ops {
if s.catalog.apply(op.clone()).is_ok() {
s.pending_log.push(op);
}
}
// a browsed folder at or below it follows
if let Some(b) = &mut s.browse
&& let Ok(rel) = Path::new(&b.path).strip_prefix(src)
{
b.path = dst.join(rel).to_string_lossy().to_string();
let f = b.path.clone();
s.filter.folder = Some(f);
let n = ops.len();
let name = Path::new(to).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let renamed = src.parent() == dst.parent();
let label = if renamed { format!("Rename Folder to “{name}”") } else { format!("Move Folder “{name}”") };
let folder = crate::FolderMove { from: to.to_string(), to: from.to_string() };
if let Err(e) = s.commit_with_folder(&label, Op::Batch { ops }, folder) {
// the catalog didn't take it: put the folder back where the photos still point
let back = rename_folder_on_disk(to, from).err().map(|b| format!("; moving it back failed too: {b}")).unwrap_or_default();
return Err(format!("{e}{back}"));
}
follow_folder(s, from, to);
Ok(n)
}
+27 -16
View File
@@ -10,22 +10,21 @@ use serde_json::{Value, json};
use super::{CommandSpec, bad, cmd, has_active, has_selection};
use crate::{Result, Session};
/// A free `<stem>.dng` (then `<stem>-2.dng`…) next to `path`.
fn dng_path(path: &str) -> String {
/// `<stem>.dng`, then `<stem>-2.dng`, `<stem>-3.dng`… next to `path`.
fn dng_names(path: &str) -> impl Iterator<Item = std::path::PathBuf> {
let p = Path::new(path);
let dir = p.parent().unwrap_or(Path::new(""));
let dir = p.parent().unwrap_or(Path::new("")).to_path_buf();
let stem = p.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| "photo".into());
let mut out = dir.join(format!("{stem}.dng"));
let mut n = 2;
while out.exists() {
out = dir.join(format!("{stem}-{n}.dng"));
n += 1;
}
out.to_string_lossy().to_string()
(1u64..).map(move |n| if n == 1 { dir.join(format!("{stem}.dng")) } else { dir.join(format!("{stem}-{n}.dng")) })
}
/// Write the DNG for raw file `path` (develop settings in `packet`); returns the new path.
/// An empty `packet` writes no XMP.
/// Write the DNG for raw file `path` (develop settings in `packet`) next to it under a free name;
/// returns the new path. An empty `packet` writes no XMP.
///
/// The DNG is checked before anyone relies on it (Convert to DNG relinks the photo to it, Copy as
/// DNG deletes the raw copy): the encoded file must decode to the same raw data, and it is
/// written to a temp file, synced, read back identical and only then given its name — never
/// replacing an existing file. On any failure no DNG is left behind and the raw stays in use.
pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::result::Result<String, String> {
let bytes = match &s.media.file_bytes {
Some(r) => r(path)?,
@@ -35,9 +34,21 @@ pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::res
drop(bytes);
let dng = lightcraft_raw::write_dng(&raw, &lightcraft_raw::DngWriteOptions { xmp: (!packet.is_empty()).then_some(packet), ..Default::default() })
.map_err(|e| e.to_string())?;
let out = dng_path(path);
std::fs::write(&out, dng).map_err(|e| format!("{out}: {e}"))?;
Ok(out)
let back = lightcraft_raw::decode(&dng).map_err(|e| format!("{path}: the DNG written for it doesn't decode ({e}); the raw is kept"))?;
let same = match (&back.data, &raw.data) {
(lightcraft_raw::RawData::U16(a), lightcraft_raw::RawData::U16(b)) => a == b,
(lightcraft_raw::RawData::F32(a), lightcraft_raw::RawData::F32(b)) => {
a.len() == b.len() && a.iter().zip(b).all(|(x, y)| x.to_bits() == y.to_bits())
}
_ => false,
};
if (back.width, back.height, back.cpp) != (raw.width, raw.height, raw.cpp) || !same {
return Err(format!("{path}: the DNG written for it doesn't hold the same raw data; the raw is kept"));
}
drop((raw, back));
let out = lightcraft_catalog::safe_file::write_new_unique(&mut dng_names(path), &dng)
.map_err(|e| format!("{path}: could not write its DNG ({e}); the raw is kept"))?;
Ok(out.to_string_lossy().to_string())
}
fn convert(s: &mut Session, p: &Value) -> Result<Value> {
@@ -133,7 +144,7 @@ fn edit_external(s: &mut Session, p: &Value) -> Result<Value> {
};
let mut written = Vec::new();
let mut write = |path: &str, bytes: &[u8]| -> std::result::Result<(), String> {
std::fs::write(path, bytes).map_err(|e| format!("{path}: {e}"))?;
crate::export::write_file(path, bytes)?;
written.push(path.to_string());
Ok(())
};
+13
View File
@@ -96,5 +96,18 @@ pub fn specs() -> Vec<CommandSpec> {
save
),
cmd!("export.deletePreset", "Delete Export Preset", [], None, "{name} — removes a user preset → presets", always, delete),
cmd!(
query "export.checkTarget",
"Check Output Path",
[],
None,
"{path} — fails when writing `path` would replace a photo's original (or its XMP sidecar) in the library; front ends call it before saving a render or screenshot to a user-given path → {path}",
always,
|s, p| {
let path = p.get("path").and_then(Value::as_str).ok_or_else(|| bad("export.checkTarget", "missing `path`"))?;
s.check_write_target(path).map_err(|e| bad("export.checkTarget", e))?;
Ok(json!({"path": path}))
}
),
]
}
+7
View File
@@ -867,6 +867,13 @@ pub fn specs() -> Vec<CommandSpec> {
"logRecords": j.log_records(),
"logBytes": j.log_bytes(),
"lastError": lib.last_error,
// settings files that were unreadable or damaged at open (kept, defaults used)
"settingsWarnings": lib.settings_warnings,
// changes applied in memory whose write failed (retried by every save)
"unsavedOps": s.unsaved().map_or(0, |u| u.0),
"unsavedError": s.unsaved().map(|u| u.1),
// untouched Local records forgotten when the library opened
"forgotLocal": lib.forgot_local.as_ref().map(|p| json!({"forgotten": p.evict.len(), "local": p.local, "keptRecent": p.kept_recent, "keptTouched": p.kept_touched, "keptInUse": p.kept_in_use})),
"persistence": j.stats(),
"cache": cache,
"load": {
+2 -1
View File
@@ -30,7 +30,8 @@ fn run(s: &mut Session, id: &str, p: &Value) -> Result<Value> {
if let (Some(path), Some(img)) = (str_param(p, "previewPath"), &out.preview) {
let png = lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default())
.map_err(|e| bad(id, e.to_string()))?;
std::fs::write(path, png).map_err(|e| bad(id, format!("{path}: {e}")))?;
s.check_write_target(path).map_err(|e| bad(id, e))?;
crate::export::write_file(path, &png).map_err(|e| bad(id, e))?;
info["previewPath"] = json!(path);
}
if let Some(img) = &out.preview {
+85 -18
View File
@@ -1,8 +1,9 @@
//! Missing originals: photos whose file is no longer where the library expects it (moved,
//! renamed, on an unplugged drive), and relinking them — one at a time (`photo.relink`) or by
//! searching a folder for files with the same name and size (`library.findMissing`).
//! searching a folder for files with the same name and size, or the same content
//! (`library.findMissing`).
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::path::Path;
use lightcraft_catalog::{Catalog, Op, Photo, PhotoId, Source};
@@ -83,8 +84,29 @@ fn relink(s: &mut Session, p: &Value) -> Result<Value> {
Ok(json!({"id": id.0, "path": abs}))
}
/// Search `folder` (recursively) for each missing photo's file name — and its size when the
/// library knows it — and relink every match in one undoable step.
/// The stored content hash of a photo, without the `:dupN` suffix duplicates get.
fn stored_hash(p: &Photo) -> Option<&str> {
p.content_hash.as_deref().map(|h| h.split(':').next().unwrap_or(h)).filter(|h| !h.is_empty())
}
/// The content hash of the file at `path` (as import computes it), if it can be read.
fn file_hash(path: &str) -> Option<String> {
std::fs::read(path).ok().map(|b| lightcraft_preview::hash_bytes(&b).to_string())
}
/// Search `folder` (recursively) for each missing photo's file and relink every match in one
/// undoable step.
///
/// Matching, per photo:
/// 1. a file with the same name (any letter case), the same size and the same content hash;
/// 2. renamed: a file of the same size with the same content hash (only same-size files are
/// read, so this stays cheap);
/// 3. without a hash to confirm (or the file was edited in place): the one file with the same
/// name and size. Several such files are ambiguous: the photo is skipped and reported in
/// `ambiguous`, never guessed (two cameras' `IMG_0001.CR3` of one size are different photos).
///
/// Files that are already some photo's original, or that another photo was just matched with,
/// are not used.
fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.findMissing";
let folder = str_param(p, "folder").ok_or_else(|| bad(C, "missing `folder`"))?;
@@ -93,27 +115,72 @@ fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
}
let lost = missing(s);
if lost.is_empty() {
return Ok(json!({"found": [], "missing": 0}));
return Ok(json!({"found": [], "missing": 0, "ambiguous": []}));
}
// file name (lower case) → candidate paths
let in_use: HashSet<String> = s.catalog.photos().filter_map(|p| checked_path(p).map(str::to_string)).collect();
// file name (lower case) → candidate paths; size → paths (filled lazily)
let mut by_name: HashMap<String, Vec<String>> = HashMap::new();
for f in crate::import::expand(&[folder.to_string()], None) {
if let Some(n) = Path::new(&f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f);
let files: Vec<String> = crate::import::expand(&[folder.to_string()], None).into_iter().filter(|f| !in_use.contains(f)).collect();
for f in &files {
if let Some(n) = Path::new(f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f.clone());
}
}
let mut by_size: Option<HashMap<u64, Vec<String>>> = None;
let mut hashes: HashMap<String, Option<String>> = HashMap::new();
let mut hash_of = |f: &str| hashes.entry(f.to_string()).or_insert_with(|| file_hash(f)).clone();
let size_of = |f: &str| std::fs::metadata(f).map(|m| m.len()).ok();
let mut used: HashSet<String> = HashSet::new();
let mut ops = Vec::new();
let mut found = Vec::new();
let mut ambiguous = Vec::new();
for (id, old) in &lost {
let Some(ph) = s.catalog.photo(*id) else { continue };
let name = Path::new(old).file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default();
let size = s.catalog.photo(*id).map(|p| p.file_size).unwrap_or(0);
let hit = by_name.get(&name).and_then(|c| {
c.iter().find(|f| size == 0 || std::fs::metadata(f).is_ok_and(|m| m.len() == size)).or_else(|| (c.len() == 1 && size == 0).then(|| &c[0]))
});
if let Some(path) = hit {
ops.push(relink_op(*id, path));
found.push(json!({"id": id.0, "from": old, "to": path}));
let size = ph.file_size;
let hash = stored_hash(ph).map(str::to_string);
let named: Vec<&String> = by_name
.get(&name)
.map(|c| c.iter().filter(|f| !used.contains(*f) && (size == 0 || size_of(f) == Some(size))).collect())
.unwrap_or_default();
// 1. same name and size, confirmed by the content hash
let mut hit: Option<(String, &str)> =
hash.as_deref().and_then(|h| named.iter().find(|f| hash_of(f).as_deref() == Some(h))).map(|f| (f.to_string(), "name"));
// 2. renamed: same size and content
if hit.is_none()
&& let Some(h) = &hash
&& size > 0
{
let sizes = by_size.get_or_insert_with(|| {
let mut m: HashMap<u64, Vec<String>> = HashMap::new();
for f in &files {
if let Some(n) = size_of(f) {
m.entry(n).or_default().push(f.clone());
}
}
m
});
hit = sizes
.get(&size)
.and_then(|c| c.iter().find(|f| !used.contains(*f) && hash_of(f).as_deref() == Some(h.as_str())))
.map(|f| (f.clone(), "content"));
}
// 3. same name and size only (no hash, or the file was edited in place): one candidate
// is taken, several are ambiguous and skipped rather than guessed
if hit.is_none() {
match named.len() {
1 => hit = Some((named[0].clone(), "name")),
0 => {}
_ => {
ambiguous.push(json!({"id": id.0, "from": old, "candidates": named}));
continue;
}
}
}
let Some((path, by)) = hit else { continue };
ops.push(relink_op(*id, &path));
found.push(json!({"id": id.0, "from": old, "to": path, "by": by}));
used.insert(path);
}
let still = lost.len() - found.len();
if !ops.is_empty() {
@@ -124,7 +191,7 @@ fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
}
}
}
Ok(json!({"found": found, "missing": still}))
Ok(json!({"found": found, "missing": still, "ambiguous": ambiguous}))
}
pub fn specs() -> Vec<CommandSpec> {
@@ -146,7 +213,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Find Missing Photos",
[],
None,
"{folder} — relink every missing photo whose file (same name and size) is somewhere in the folder → {found: [{id, from, to}], missing}",
"{folder} — relink every missing photo whose file is somewhere in the folder: same name and size (and content, when the library knows its hash), or — renamed — same size and content → {found: [{id, from, to, by: name|content}], missing, ambiguous: [{id, from, candidates}] (several same-name candidates and no hash to tell them apart: skipped)}",
always,
find_missing
),
+1 -1
View File
@@ -5,7 +5,7 @@
//! `edit.*` (undo/redo), and queries (`catalog.query`, `photo.inspect`, `develop.get`…).
mod before;
mod browse;
pub(crate) mod browse;
mod color;
pub(crate) mod convert;
mod cull;
+7 -1
View File
@@ -14,6 +14,7 @@ fn current(s: &Session) -> Value {
"xmp": s.xmp,
"import": s.import_defaults,
"cacheMb": if s.cache_mb == 0 { s.cache_bytes() >> 20 } else { u64::from(s.cache_mb) },
"forgetLocalDays": s.forget_local_days,
"persistent": s.library.is_some(),
})
}
@@ -86,6 +87,11 @@ fn prefs(s: &mut Session, p: &Value) -> Result<Value> {
changed = true;
}
s.import_defaults = d;
if let Some(v) = p.get("forgetLocalDays") {
let days = v.as_u64().ok_or_else(|| bad(ID, "forgetLocalDays must be a whole number of days (0 = never)"))?;
s.forget_local_days = days.min(36_500) as u32;
changed = true;
}
if let Some(mb) = p.get("cacheMb") {
let mb = mb.as_u64().ok_or_else(|| bad(ID, "cacheMb must be a number of megabytes"))?;
s.set_cache_mb(mb.clamp(0, 1 << 20) as u32)?;
@@ -101,7 +107,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Library Preferences",
[],
None,
"{import?: {rawPreset?: presetId|\"default\", otherPreset?: presetId|\"default\", perCamera?: bool, cameras?: [{camera: \"Make Model\", preset: presetId|null}], copyright?: text, creator?: text (given to imported photos without one)}, camera?: {camera, preset?, remove?: bool}, cacheMb?: n (0 = default)} — develop defaults applied on import (raws / other images / per camera) and the thumbnail cache size, saved with the library → {xmp, import, cacheMb, persistent}",
"{import?: {rawPreset?: presetId|\"default\", otherPreset?: presetId|\"default\", perCamera?: bool, cameras?: [{camera: \"Make Model\", preset: presetId|null}], copyright?: text, creator?: text (given to imported photos without one)}, camera?: {camera, preset?, remove?: bool}, cacheMb?: n (0 = default), forgetLocalDays?: n (forget untouched Local photos of folders not browsed for n days; 0 = never)} — develop defaults applied on import (raws / other images / per camera) and the thumbnail cache size, saved with the library → {xmp, import, cacheMb, forgetLocalDays, persistent}",
always,
prefs
)]
+14 -7
View File
@@ -102,7 +102,7 @@ fn smart(s: &mut Session, p: &Value) -> Result<Value> {
let dir = s.media.smart_dir.clone().ok_or_else(|| bad(C, "smart previews need a library on disk"))?;
let discard = p.get("discard").and_then(Value::as_bool).unwrap_or(false);
let ids = if p.get("ids").is_some() || !s.selection.ids.is_empty() { s.targets(p) } else { s.visible_cloned() };
let (mut built, mut removed, mut failed) = (0usize, 0usize, Vec::new());
let (mut built, mut repaired, mut removed, mut failed) = (0usize, 0usize, 0usize, Vec::new());
if !discard {
// a chosen folder on another drive must be there: never recreate it on this one
if s.smart_previews_dir.is_some() {
@@ -123,20 +123,27 @@ fn smart(s: &mut Session, p: &Value) -> Result<Value> {
}
continue;
}
if path.exists() {
// a complete proxy is kept; a missing or damaged one (cut short by a crash or a full
// drive) is (re)built
let damaged = path.exists();
if damaged && crate::smart::is_valid(&path) {
built += 1;
continue;
}
// atomic: a failed write leaves no partial proxy that would pass for a built one
let r = s
.source_now(id, crate::media::SourceLevel::Preview)
.and_then(|src| crate::smart::encode(&src))
.and_then(|b| std::fs::write(&path, b).map_err(|e| e.to_string()));
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&path, &b).map_err(|e| format!("{}: {e}", path.display())));
match r {
Ok(()) => built += 1,
Ok(()) => {
built += 1;
repaired += usize::from(damaged);
}
Err(e) => failed.push(json!([id.0, e])),
}
}
Ok(json!({"built": built, "removed": removed, "failed": failed}))
Ok(json!({"built": built, "repaired": repaired, "removed": removed, "failed": failed}))
}
/// The smart previews folder: where it is, whether it is custom, what it holds, whether it can be
@@ -205,7 +212,7 @@ fn smart_location(s: &mut Session, p: &Value) -> Result<Value> {
/// Whether photo `id` has a smart preview.
pub fn has_smart_preview(s: &Session, id: lightcraft_catalog::PhotoId) -> bool {
match (&s.media.smart_dir, s.catalog.photo(id)) {
(Some(dir), Some(p)) => dir.join(crate::smart::file_name(p)).exists(),
(Some(dir), Some(p)) => crate::smart::is_valid(&dir.join(crate::smart::file_name(p))),
_ => false,
}
}
@@ -218,7 +225,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Build Smart Previews",
[],
None,
"{ids?, discard?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline → {built, removed, failed}",
"{ids?, discard?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline; a damaged proxy (cut short) is rebuilt → {built, repaired, removed, failed}",
always,
smart
),
+9 -3
View File
@@ -113,7 +113,7 @@ pub fn specs() -> Vec<CommandSpec> {
.map(|r| json!({"group": r.group, "tag": r.tag, "name": r.name, "value": r.value}))
.collect();
// XMP: the sidecar if there is one, else the file's own packet
let packet = crate::sidecar::read_packet(path, ph.kind, s.xmp.naming).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let packet = crate::sidecar::read_packet(path, ph.kind, s.sidecar_naming(ph.id)).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let xmp: Vec<Value> = packet
.and_then(|x| lightcraft_meta::parse_xmp(&x).ok())
.map(|d| d.properties.into_iter().filter(|(k, _)| !k.starts_with("lc:")).map(|(k, v)| json!({"name": k, "value": v.join("; ")})).collect())
@@ -186,11 +186,17 @@ pub fn specs() -> Vec<CommandSpec> {
"versions": ph.versions.iter().map(|v| json!({"name": v.name, "created": v.created})).collect::<Vec<_>>(),
}))
}),
cmd!(query "app.gpu", "GPU Rendering", [], None, "{enabled?: bool} — allow/forbid GPU rendering (CPU fallback; LIGHTCRAFT_GPU=0 forbids it for the process)", always, |_, p| {
cmd!(query "app.gpu", "GPU Rendering", [], None, "{enabled?: bool} — allow/forbid GPU rendering (CPU fallback; LIGHTCRAFT_GPU=0 forbids it for the process); returns {enabled, available, adapter, reason (why the GPU is off), lastFallback (latest render redone on the CPU, and why)}", always, |_, p| {
if let Some(on) = p.get("enabled").and_then(Value::as_bool) {
lightcraft_gpu::set_enabled(on);
}
Ok(json!({"enabled": lightcraft_gpu::enabled(), "available": lightcraft_gpu::available(), "adapter": lightcraft_gpu::adapter_name()}))
Ok(json!({
"enabled": lightcraft_gpu::enabled(),
"available": lightcraft_gpu::available(),
"adapter": lightcraft_gpu::adapter_name(),
"reason": lightcraft_gpu::unavailable_reason(),
"lastFallback": lightcraft_gpu::last_fallback(),
}))
}),
cmd!(query "library.memory", "Memory Usage", [], None, "{} — bytes held by each cache (decoded sources, rendered previews, GPU buffers; heap when instrumented)", always, |s, _| {
Ok(serde_json::to_value(s.memory_report()).unwrap_or_default())
+16 -4
View File
@@ -9,14 +9,26 @@ use crate::{Result, Session};
fn save(s: &mut Session, p: &Value) -> Result<Value> {
let mut written = Vec::new();
let mut merged = Vec::new();
let mut backups = Vec::new();
let mut failed = Vec::new();
let owners = crate::sidecar::StemOwners::of(&s.catalog);
for id in s.targets(p) {
match s.save_sidecar(id) {
Ok(path) => written.push(path.display().to_string()),
match s.save_sidecar_with(id, &owners) {
Ok(r) => {
let path = r.path.display().to_string();
if r.merged {
merged.push(path.clone());
}
if let Some(b) = r.backup {
backups.push(json!({"path": path, "backup": b.display().to_string()}));
}
written.push(path);
}
Err(e) => failed.push(json!({"id": id.0, "error": e.to_string()})),
}
}
Ok(json!({"written": written, "failed": failed}))
Ok(json!({"written": written, "merged": merged, "backups": backups, "failed": failed}))
}
fn read(s: &mut Session, p: &Value) -> Result<Value> {
@@ -63,7 +75,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Save Metadata to File",
["Photo"],
Some("Cmd+S"),
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original → {written: [path], failed}",
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original, merged into an existing one (other apps' data kept; an unreadable one is backed up first) → {written: [path], merged: [path], backups: [{path, backup}], failed}",
has_selection,
save
),
+3 -1
View File
@@ -35,7 +35,9 @@ fn mount_parents() -> Vec<std::path::PathBuf> {
let mut out = Vec::new();
if cfg!(target_os = "macos") {
out.push("/Volumes".into());
} else if cfg!(target_os = "linux") {
} else if cfg!(any(target_os = "linux", target_os = "freebsd")) {
// Linux desktops mount under /media/$USER or /run/media/$USER; FreeBSD's automounter
// (automount/autofs, or a desktop's) under /media. Missing folders are skipped.
if let Ok(user) = std::env::var("USER") {
out.push(format!("/media/{user}").into());
out.push(format!("/run/media/{user}").into());
+184 -39
View File
@@ -202,6 +202,8 @@ pub enum Anchor {
#[serde(default, rename_all = "camelCase")]
pub struct Watermark {
pub text: String,
/// Basic Japanese vertical lettering: upright glyphs in columns from right to left.
pub vertical: bool,
/// Text height as a fraction of the image's short edge.
pub size: f32,
/// 0..1.
@@ -226,6 +228,7 @@ impl Default for Watermark {
fn default() -> Self {
Self {
text: String::new(),
vertical: false,
size: 0.035,
opacity: 0.7,
anchor: Anchor::BottomRight,
@@ -241,6 +244,9 @@ impl Default for Watermark {
/// Inter SemiBold (OFL, see assets/ATTRIBUTION.md).
static WATERMARK_FONT: &[u8] = include_bytes!("../../../assets/fonts/Inter-SemiBold.ttf");
/// BIZ UDMincho (OFL), Japanese fallback on both native and web.
/// Bundled OFL Japanese font bytes, shared by watermarks and UI glyph fallback.
pub static WATERMARK_JAPANESE_FONT: &[u8] = include_bytes!("../../../assets/fonts/BIZUDMincho-Regular.ttf");
/// Draw `wm` onto `img` (straight alpha blending of the encoded values).
pub fn draw_watermark(img: &mut Rgba8, wm: &Watermark) {
@@ -288,23 +294,67 @@ fn watermark_coverage(width: usize, height: usize, wm: &Watermark, mut blend_px:
return;
}
let Ok(font) = FontRef::try_from_slice(WATERMARK_FONT) else { return };
let Ok(japanese) = FontRef::try_from_slice(WATERMARK_JAPANESE_FONT) else { return };
let fonts = [font, japanese];
let short = width.min(height) as f32;
let px = (wm.size.clamp(0.005, 0.5) * short).max(6.0);
let sf = font.as_scaled(PxScale::from(px));
// Lay out one line.
let mut glyphs = Vec::new();
let mut x = 0.0f32;
let mut y = 0.0f32;
let mut tw = 0.0f32;
let mut prev = None;
for ch in text.chars() {
let columns = text.split('\n').count();
let mut column = 0usize;
for original in text.chars() {
if original == '\n' {
if wm.vertical {
column = column.saturating_add(1);
tw = tw.max(y);
y = 0.0;
} else {
tw = tw.max(x);
x = 0.0;
y += px;
}
prev = None;
continue;
}
let vertical_form = match original {
'、' => '︑',
'。' => '︒',
'(' => '︵',
')' => '︶',
'「' => '﹁',
'」' => '﹂',
'『' => '﹃',
'』' => '﹄',
'【' => '︻',
'】' => '︼',
'…' => '︙',
other => other,
};
let ch = if wm.vertical && fonts[1].glyph_id(vertical_form).0 != 0 { vertical_form } else { original };
let face = usize::from(fonts[0].glyph_id(ch).0 == 0);
let sf = fonts[face].as_scaled(PxScale::from(px));
let id = sf.glyph_id(ch);
if let Some(p) = prev {
x += sf.kern(p, id);
if wm.vertical {
let left = columns.saturating_sub(column.saturating_add(1)) as f32 * px;
let pen = point(left + (px - sf.h_advance(id)) / 2.0, y + sf.ascent());
let rotation = matches!(original, 'ー' | '—' | '–').then_some((left + px / 2.0, y + px / 2.0));
glyphs.push((face, id.with_scale_and_position(px, pen), rotation));
y += px;
} else {
if let Some((previous_face, previous)) = prev
&& previous_face == face
{
x += sf.kern(previous, id);
}
glyphs.push((face, id.with_scale_and_position(px, point(x, y + sf.ascent())), None));
x += sf.h_advance(id);
prev = Some((face, id));
}
glyphs.push(id.with_scale_and_position(px, point(x, sf.ascent())));
x += sf.h_advance(id);
prev = Some(id);
}
let (tw, th) = (x, sf.ascent() - sf.descent());
let (tw, th) = if wm.vertical { (columns as f32 * px, tw.max(y)) } else { (tw.max(x), y + px) };
let inset = wm.inset.clamp(0.0, 0.4) * short;
let (w, h) = (width as f32, height as f32);
use Anchor::*;
@@ -328,10 +378,17 @@ fn watermark_coverage(width: usize, height: usize, wm: &Watermark, mut blend_px:
let passes: &[(f32, [u8; 3], f32)] =
if wm.shadow { &[((px * 0.05).max(1.0), [0, 0, 0], 0.45), (0.0, wm.color, 1.0)] } else { &[(0.0, wm.color, 1.0)] };
for &(off, col, a) in passes {
for g in &glyphs {
if let Some(o) = font.outline_glyph(g.clone()) {
for (face, g, rotation) in &glyphs {
if let Some(o) = fonts.get(*face).and_then(|font| font.outline_glyph(g.clone())) {
let b = o.px_bounds();
o.draw(|gx, gy, cov| blend((ox + off + b.min.x) as i32 + gx as i32, (oy + off + b.min.y) as i32 + gy as i32, cov, col, a * alpha));
o.draw(|gx, gy, cov| {
let (sx, sy) = (b.min.x + gx as f32, b.min.y + gy as f32);
let (sx, sy) = match rotation {
Some((cx, cy)) => (cx - (sy - cy), cy + (sx - cx)),
None => (sx, sy),
};
blend((ox + off + sx) as i32, (oy + off + sy) as i32, cov, col, a * alpha);
});
}
}
}
@@ -930,6 +987,8 @@ pub struct PreparedExport {
pub photo: lightcraft_catalog::PhotoId,
pub file_name: String,
work: Work,
/// The library's originals, which [`run_batch`] never writes over (shared by a batch).
guard: std::sync::Arc<crate::originals::OriginalGuard>,
}
struct RenderWork {
@@ -952,12 +1011,30 @@ enum Work {
},
}
/// Set up the export of photo `id` at 1-based position `seq` of a batch.
/// Set up the export of photo `id` at 1-based position `seq` of a batch. (For a whole batch use
/// [`prepare_batch`]: it looks at the library's originals once.)
pub fn prepare_export(
session: &mut crate::Session,
id: lightcraft_catalog::PhotoId,
o: &ExportOptions,
seq: usize,
) -> Result<PreparedExport, String> {
let guard = std::sync::Arc::new(session.original_guard());
prepare_guarded(session, id, o, seq, guard)
}
/// [`prepare_export`] for each of `ids` in order (`{seq}` = position, from 1).
pub fn prepare_batch(session: &mut crate::Session, ids: &[lightcraft_catalog::PhotoId], o: &ExportOptions) -> Result<Vec<PreparedExport>, String> {
let guard = std::sync::Arc::new(session.original_guard());
ids.iter().enumerate().map(|(i, id)| prepare_guarded(session, *id, o, i + 1, guard.clone())).collect()
}
fn prepare_guarded(
session: &mut crate::Session,
id: lightcraft_catalog::PhotoId,
o: &ExportOptions,
seq: usize,
guard: std::sync::Arc<crate::originals::OriginalGuard>,
) -> Result<PreparedExport, String> {
let p = session.catalog.photo(id).ok_or("no such photo")?;
let file_name = o.file_name_for(p, seq);
@@ -979,7 +1056,7 @@ pub fn prepare_export(
size: (p.width as usize, p.height as usize),
}
};
Ok(PreparedExport { photo: id, file_name, work })
Ok(PreparedExport { photo: id, file_name, work, guard })
}
impl PreparedExport {
@@ -1031,14 +1108,16 @@ pub enum DngCompression {
}
/// The destination of a batch: a folder (with `ExportOptions::subfolder` and the conflict policy
/// applied), or one exact file path (single photo; overwritten).
/// applied to each file together with its sidecars), or one exact file path (single photo; an
/// ordinary file already there is replaced). Either way a catalogued original (or its sidecar)
/// is never written over: see [`crate::originals`].
#[derive(Clone, Debug, Default)]
pub struct Destination {
pub dir: String,
pub exact: Option<String>,
}
/// Export `ids` in order ([`prepare_export`] + [`run_batch`]), stopping at the first error.
/// Export `ids` in order ([`prepare_batch`] + [`run_batch`]), stopping at the first error.
pub fn export_batch(
session: &mut crate::Session,
ids: &[lightcraft_catalog::PhotoId],
@@ -1047,15 +1126,34 @@ pub fn export_batch(
write: &mut dyn FnMut(&str, &[u8]) -> Result<(), String>,
exists: &dyn Fn(&str) -> bool,
) -> Result<Vec<serde_json::Value>, String> {
let items = ids.iter().enumerate().map(|(i, id)| prepare_export(session, *id, o, i + 1)).collect::<Result<Vec<_>, _>>()?;
let items = prepare_batch(session, ids, o)?;
run_batch(items, o, to, write, exists, true, &mut |_, _| true)
}
/// Write an exported or rendered file on disk: its folder is created if needed, and the file is
/// replaced atomically ([`lightcraft_catalog::safe_file::write_atomic`]: a temp file, synced, then
/// renamed), so a failure part-way leaves any previous file intact and no truncated one. The
/// native writer behind exports, renders and screenshots (app, CLI, MCP).
pub fn write_file(path: &str, bytes: &[u8]) -> Result<(), String> {
let p = std::path::Path::new(path);
if let Some(dir) = p.parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
}
lightcraft_catalog::safe_file::write_atomic(p, bytes).map_err(|e| format!("{path}: {e}"))
}
/// The path of the sidecar with extension `ext` of the exported file `main`.
fn sidecar_path(main: &str, ext: &str) -> String {
std::path::Path::new(main).with_extension(ext).to_string_lossy().to_string()
}
/// Run prepared exports in order: pick each one's path, and hand the bytes (and sidecars) to
/// `write`. `exists` tells whether a path is taken (conflict policy). `progress(done, next file)`
/// is called before each photo; returning false cancels the rest. Returns one JSON object per
/// photo: `{path, width, height, bytes, sidecars}`, `{skipped: path}` or (unless
/// `stop_on_error`) `{photo, file, error}`.
/// `write`. `exists` tells whether a path is taken. The conflict policy applies to a file and its
/// sidecars as one: with Unique both get the same free name, with Skip the photo is skipped when
/// either is taken. A path that is a catalogued original (or its sidecar) is refused whatever the
/// policy. `progress(done, next file)` is called before each photo; returning false cancels the
/// rest. Returns one JSON object per photo: `{path, width, height, bytes, sidecars}`,
/// `{skipped: path}` or (unless `stop_on_error`) `{photo, file, error}`.
pub fn run_batch(
items: Vec<PreparedExport>,
o: &ExportOptions,
@@ -1075,7 +1173,7 @@ pub fn run_batch(
if !progress(i, &item.file_name) {
break;
}
let (photo, name) = (item.photo, item.file_name.clone());
let (photo, name, guard) = (item.photo, item.file_name.clone(), item.guard.clone());
let e = match item.run() {
Ok(e) => e,
Err(err) if stop_on_error => return Err(err),
@@ -1084,14 +1182,18 @@ pub fn run_batch(
continue;
}
};
// the exported file and its sidecars
let group = |main: &str| std::iter::once(main.to_string()).chain(e.sidecars.iter().map(|(x, _)| sidecar_path(main, x))).collect::<Vec<_>>();
let path = match to.exact.as_deref().filter(|_| single) {
Some(p) => p.to_string(),
Some(p) => Ok(p.to_string()),
None => {
let mut path = join(&dir, &e.file_name);
let busy = |p: &str, taken: &std::collections::HashSet<String>| taken.contains(p) || exists(p);
if busy(&path, &taken) {
let path = join(&dir, &e.file_name);
let busy = |main: &str| group(main).iter().any(|p| taken.contains(p) || exists(p));
if !busy(&path) {
Ok(path)
} else {
match o.conflict {
Conflict::Overwrite => {}
Conflict::Overwrite => Ok(path),
Conflict::Skip => {
out.push(json!({"skipped": path}));
continue;
@@ -1099,29 +1201,34 @@ pub fn run_batch(
Conflict::Unique => {
let (stem, ext) = e.file_name.rsplit_once('.').map_or((e.file_name.as_str(), None), |(a, b)| (a, Some(b)));
let name = |n: usize| join(&dir, &ext.map_or(format!("{stem}-{n}"), |x| format!("{stem}-{n}.{x}")));
path = (2..).map(name).find(|p| !busy(p, &taken)).expect("a free name");
(2..1_000_000).map(name).find(|p| !busy(p)).ok_or_else(|| format!("{path}: no free file name"))
}
}
}
path
}
};
let written = write(&path, &e.bytes).and_then(|()| {
let mut sidecars = Vec::new();
for (ext, bytes) in &e.sidecars {
let sc = std::path::Path::new(&path).with_extension(ext).to_string_lossy().to_string();
write(&sc, bytes)?;
sidecars.push(sc);
let file = path.clone().unwrap_or_else(|_| name.clone());
let written = path.and_then(|path| {
let files = group(&path);
// never over an original, whatever the conflict policy or the exact path said
for f in &files {
guard.check(std::path::Path::new(f))?;
}
Ok(sidecars)
write(&path, &e.bytes)?;
let mut sidecars = Vec::new();
for ((_, bytes), sc) in e.sidecars.iter().zip(files.iter().skip(1)) {
write(sc, bytes)?;
sidecars.push(sc.clone());
}
Ok((path, files, sidecars))
});
match written {
Ok(sidecars) => {
taken.insert(path.clone());
Ok((path, files, sidecars)) => {
taken.extend(files);
out.push(json!({"path": path, "width": e.width, "height": e.height, "bytes": e.bytes.len(), "sidecars": sidecars}));
}
Err(err) if stop_on_error => return Err(err),
Err(err) => out.push(json!({"photo": photo.0, "file": path, "error": err})),
Err(err) => out.push(json!({"photo": photo.0, "file": file, "error": err})),
}
}
Ok(out)
@@ -1227,6 +1334,44 @@ mod tests {
assert!(ExportOptions::from_json(&serde_json::json!({"watermark": ""})).watermark.is_none());
}
#[test]
fn japanese_watermarks_support_vertical_columns_and_legacy_defaults() {
use ab_glyph::Font;
let font = ab_glyph::FontRef::try_from_slice(WATERMARK_JAPANESE_FONT).unwrap();
for c in "日本語".chars() {
assert_ne!(font.glyph_id(c).0, 0);
}
let old = ExportOptions::from_json(&json!({"watermark": {"text": "日本語"}}));
assert!(!old.watermark.unwrap().vertical);
let options =
ExportOptions::from_json(&json!({"watermark": {"text": "日本語", "vertical": true, "size": 0.1, "shadow": false, "opacity": 1.0}}));
let wm = options.watermark.unwrap();
assert!(wm.vertical);
let round: Watermark = serde_json::from_value(serde_json::to_value(&wm).unwrap()).unwrap();
assert!(round.vertical);
let mut img = Rgba8::new(400, 300);
img.data.fill([0, 0, 0, 255]);
draw_watermark(&mut img, &wm);
let lit: Vec<_> = (0..300).flat_map(|y| (0..400).map(move |x| (x, y))).filter(|&(x, y)| img.get(x, y)[0] > 64).collect();
assert!(lit.len() > 100);
let w = lit.iter().map(|p| p.0).max().unwrap() - lit.iter().map(|p| p.0).min().unwrap();
let h = lit.iter().map(|p| p.1).max().unwrap() - lit.iter().map(|p| p.1).min().unwrap();
assert!(h > w * 2, "vertical Japanese must extend down the column");
let two = Watermark { text: "日\n本".into(), anchor: Anchor::TopLeft, inset: 0.0, ..wm };
img.data.fill([0, 0, 0, 255]);
draw_watermark(&mut img, &two);
for (text, dx) in [("本", 0), ("日", 30)] {
let mut single = Rgba8::new(400, 300);
single.data.fill([0, 0, 0, 255]);
draw_watermark(&mut single, &Watermark { text: text.into(), ..two.clone() });
for y in 0..30 {
for x in 0..30 {
assert_eq!(img.get(x + dx, y), single.get(x, y), "newlines move columns left");
}
}
}
}
#[test]
fn graphic_watermark_is_placed_scaled_and_blended() {
let dir = std::env::temp_dir().join(format!("lc-wm-logo-{}", std::process::id()));
+7 -14
View File
@@ -374,25 +374,14 @@ fn probe_all(probe: Option<&crate::media::FileProbe>, paths: &[String], progress
}
/// Copy `src` into `root`/`folders` as `name` (default: its own name), made unique with -1, -2…;
/// returns the new path.
/// returns the new path. The copy is verified ([`crate::import_move::copy_new`]: a new file,
/// synced and compared byte for byte); a bad one is removed and the photo reported as failed.
fn copy_into(root: &Path, src: &str, folders: &[String], name: Option<&str>) -> Result<String, String> {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
std::fs::create_dir_all(&dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let name = name
.map(str::to_string)
.unwrap_or_else(|| Path::new(src).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| "photo".into()));
let (stem, ext) = match name.rsplit_once('.') {
Some((s, e)) => (s.to_string(), format!(".{e}")),
None => (name.clone(), String::new()),
};
let mut dst = dir.join(&name);
let mut i = 1;
while dst.exists() {
dst = dir.join(format!("{stem}-{i}{ext}"));
i += 1;
}
std::fs::copy(src, &dst).map_err(|e| format!("copy {src}: {e}"))?;
Ok(dst.to_string_lossy().to_string())
crate::import_move::copy_new(Path::new(src), &dir, &name).map(|p| p.to_string_lossy().to_string())
}
/// What a [`scan`] needs from the session, so it can run on another thread (a folder on a network
@@ -722,6 +711,10 @@ pub fn import_with(s: &mut Session, paths: &[String], opts: &ImportOptions) -> c
}
report.imported.push(id.0);
p.local = opts.local;
if opts.local {
// the state as browsed: a later change keeps the record from being forgotten
p.set_local_baseline();
}
ops.push(Op::AddPhoto { photo: Box::new(p) });
}
let log0 = s.pending_log.len();
+22 -1
View File
@@ -169,6 +169,27 @@ pub(crate) fn place(src: &Path, dir: &Path, name: &str) -> Result<Placed, String
Ok(placed)
}
/// Import → **Copy**: copy `src` into `dir` as `name` (-1, -2… when taken) with the same checks as
/// a move's copy — a new file (never overwriting one, even one that appears meanwhile), synced to
/// disk and compared byte for byte with the source; a bad copy is removed and reported, so a card
/// is never wiped on the strength of it. Returns the new path.
pub(crate) fn copy_new(src: &Path, dir: &Path, name: &str) -> Result<PathBuf, String> {
fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let (stem, ext) = match name.rsplit_once('.') {
Some((s, e)) if !s.is_empty() => (s.to_string(), format!(".{e}")),
_ => (name.to_string(), String::new()),
};
for i in 0..100_000u32 {
let d = if i == 0 { dir.join(name) } else { dir.join(format!("{stem}-{i}{ext}")) };
match copy_verified(src, &d) {
Ok(()) => return Ok(d),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(format!("copy {}: {e} (nothing was imported from it; the original is untouched)", src.display())),
}
}
Err(format!("copy {}: no free name in {}", src.display(), dir.display()))
}
/// Undo a [`place`]: remove what it created at the destination (the sources were never touched).
pub(crate) fn rollback(p: &Placed) {
for sc in p.sidecars.iter().filter(|s| s.created) {
@@ -213,7 +234,7 @@ pub(crate) fn finish(p: &Placed) -> Result<Vec<(String, String)>, String> {
}
/// A photo (by extension) still beside a stem-named sidecar, sharing its stem.
fn sibling_photo(sidecar: &Path) -> Option<String> {
pub(crate) fn sibling_photo(sidecar: &Path) -> Option<String> {
let dir = sidecar.parent()?;
let stem = sidecar.file_stem()?.to_string_lossy().to_string();
fs::read_dir(dir).ok()?.flatten().map(|e| e.path()).find_map(|f| {
+96 -14
View File
@@ -24,6 +24,7 @@ pub mod library;
pub mod media;
pub mod memory;
pub mod merge;
pub mod originals;
pub mod preset_import;
pub mod preset_luminar;
pub mod presets;
@@ -52,6 +53,13 @@ pub enum EngineError {
BadParams { cmd: String, msg: String },
#[error("{0}")]
Catalog(#[from] lightcraft_catalog::CatalogError),
/// The command's change is applied (in memory, undoable) but its journal records could not
/// be written. They stay queued and are written by the next successful save.
#[error("saved in memory but not written to disk: {0}; LightCraft will retry")]
NotSaved(String),
/// Another process (the app, `lightcraft-cli`, another computer) has the library open.
#[error("{0}")]
LibraryInUse(String),
#[error("{0}")]
Other(String),
}
@@ -63,6 +71,22 @@ pub type Result<T> = std::result::Result<T, EngineError>;
pub struct UndoEntry {
pub label: String,
pub op: Op,
/// A folder to rename on disk (`from` → `to`) before `op` is applied: Rename / Move Folder
/// (whose `op` relinks the photos inside). Never overwrites; refused when `to` exists.
pub folder: Option<FolderMove>,
}
/// A folder renamed or moved on disk as part of an undo step.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FolderMove {
pub from: String,
pub to: String,
}
impl FolderMove {
fn reversed(&self) -> Self {
Self { from: self.to.clone(), to: self.from.clone() }
}
}
/// An in-progress slider drag / brush stroke: one undo step when it ends.
@@ -168,6 +192,9 @@ pub struct Session {
pub cache_mb: u32,
/// Where smart previews are kept when not in the library folder (persisted in prefs.json).
pub smart_previews_dir: Option<std::path::PathBuf>,
/// Untouched Local records of folders not browsed for this many days are forgotten when the
/// library opens (0 = never; persisted in prefs.json). See `cmd/browse.rs`.
pub forget_local_days: u32,
}
impl Default for Session {
@@ -228,6 +255,7 @@ impl Session {
import_defaults: import::ImportDefaults::default(),
cache_mb: 0,
smart_previews_dir: None,
forget_local_days: lightcraft_catalog::DEFAULT_FORGET_DAYS,
}
}
@@ -266,9 +294,18 @@ impl Session {
self.auto_write_sidecars(&self.pending_log[log_start..]);
}
if self.depth == 0 && self.library.is_some() {
// Make the command durable before reporting success (on failure the ops stay pending,
// are retried after the next command, and `library.info` reports the error).
let _ = self.persist();
// Make the command durable before reporting success. When the command's own records
// can't be appended, it fails with `NotSaved`: the change stays applied in memory and
// queued, and the next save (any later command, or the frame loop) retries it. Other
// persistence trouble (a failed compaction, an older queued change still unwritten by
// a command that changed nothing) doesn't fail the command; `library.info` reports it.
let produced = self.pending_log.len() > log_start;
if let Err(e @ EngineError::NotSaved(_)) = self.persist()
&& produced
&& r.is_ok()
{
return Err(e);
}
}
r
}
@@ -284,7 +321,7 @@ impl Session {
let fwd = op.clone();
let inv = self.catalog.apply(op)?;
self.pending_log.push(fwd);
self.undo.push(UndoEntry { label: label.to_string(), op: inv });
self.undo.push(UndoEntry { label: label.to_string(), op: inv, folder: None });
if self.undo.len() > 1000 {
self.undo.remove(0);
}
@@ -292,6 +329,16 @@ impl Session {
Ok(())
}
/// [`Session::commit`] for an op that goes with a folder already renamed on disk; the undo
/// step renames it back (`folder` is the undo direction: current place → old place).
pub(crate) fn commit_with_folder(&mut self, label: &str, op: Op, folder: FolderMove) -> Result<()> {
self.commit(label, op)?;
if let Some(e) = self.undo.last_mut() {
e.folder = Some(folder);
}
Ok(())
}
/// Leaving photo `id` after editing it: keep its settings as an automatic version (when they
/// differ from its latest version; at most [`AUTO_VERSIONS`] auto versions, oldest dropped).
/// Saved with the library but not an undo step.
@@ -322,9 +369,13 @@ impl Session {
if n < 2 || n > self.undo.len() {
return;
}
// a step that moves a folder on disk stays on its own
if self.undo[self.undo.len() - n..].iter().any(|e| e.folder.is_some()) {
return;
}
let tail = self.undo.split_off(self.undo.len() - n);
let ops = tail.into_iter().rev().map(|e| e.op).collect();
self.undo.push(UndoEntry { label: label.to_string(), op: Op::Batch { ops } });
self.undo.push(UndoEntry { label: label.to_string(), op: Op::Batch { ops }, folder: None });
}
/// Apply without recording undo (interactive previews).
@@ -335,7 +386,7 @@ impl Session {
pub fn undo_step(&mut self) -> Result<String> {
let e = self.undo.pop().ok_or_else(|| EngineError::Other("nothing to undo".into()))?;
let redo = match self.apply_with_files(&e.op) {
let redo = match self.apply_with_files(&e.op, e.folder.as_ref()) {
Ok(r) => r,
Err(err) => {
self.undo.push(e);
@@ -343,13 +394,13 @@ impl Session {
}
};
self.pending_log.push(e.op);
self.redo.push(UndoEntry { label: e.label.clone(), op: redo });
self.redo.push(UndoEntry { label: e.label.clone(), op: redo, folder: e.folder.as_ref().map(FolderMove::reversed) });
Ok(e.label)
}
pub fn redo_step(&mut self) -> Result<String> {
let e = self.redo.pop().ok_or_else(|| EngineError::Other("nothing to redo".into()))?;
let undo = match self.apply_with_files(&e.op) {
let undo = match self.apply_with_files(&e.op, e.folder.as_ref()) {
Ok(r) => r,
Err(err) => {
self.redo.push(e);
@@ -357,19 +408,44 @@ impl Session {
}
};
self.pending_log.push(e.op);
self.undo.push(UndoEntry { label: e.label.clone(), op: undo });
self.undo.push(UndoEntry { label: e.label.clone(), op: undo, folder: e.folder.as_ref().map(FolderMove::reversed) });
Ok(e.label)
}
/// Apply an undo/redo op, first moving the files its renames imply (all or nothing).
fn apply_with_files(&mut self, op: &Op) -> Result<Op> {
/// Apply an undo/redo op, first renaming the step's folder and moving the files its renames
/// imply (all or nothing). Files that can't be moved back after a failure are reported and the
/// library follows them (a logged change outside the undo history), so none goes missing; the
/// folder is only moved back when no file was left behind at its new path.
fn apply_with_files(&mut self, op: &Op, folder: Option<&FolderMove>) -> Result<Op> {
let fs = rename::RealFs;
if let Some(f) = folder {
cmd::browse::rename_folder_on_disk(&f.from, &f.to).map_err(|e| EngineError::Other(format!("can't move the folder back: {e}")))?;
}
let undo_folder = || {
if let Some(f) = folder {
let _ = cmd::browse::rename_folder_on_disk(&f.to, &f.from);
}
};
let moves = self.file_moves(op);
Session::move_files(&moves)?;
if let Err(e) = rename::move_all(&fs, &moves) {
if e.stuck.is_empty() {
undo_folder();
}
return Err(EngineError::Other(format!("can't move the files back: {}", self.follow_stuck(op, e, false))));
}
match self.catalog.apply(op.clone()) {
Ok(inv) => Ok(inv),
Ok(inv) => {
if let Some(f) = folder {
cmd::browse::follow_folder(self, &f.from, &f.to);
}
Ok(inv)
}
Err(e) => {
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
let _ = Session::move_files(&back);
if let Err(be) = rename::move_all(&fs, &back) {
return Err(EngineError::Other(format!("{e}; the files could not all be moved back: {}", be.message)));
}
undo_folder();
Err(e.into())
}
}
@@ -599,6 +675,8 @@ mod tests_color;
#[cfg(test)]
mod tests_export;
#[cfg(test)]
mod tests_forget_local;
#[cfg(test)]
mod tests_import;
#[cfg(test)]
mod tests_import_move;
@@ -611,8 +689,12 @@ mod tests_merge;
#[cfg(test)]
mod tests_organize;
#[cfg(test)]
mod tests_persist;
#[cfg(test)]
mod tests_prefs;
#[cfg(test)]
mod tests_settings_files;
#[cfg(test)]
mod tests_spots;
#[cfg(test)]
mod tests_xmp;
+242 -23
View File
@@ -14,13 +14,16 @@
//! via [`Session::open_library_in`] (the browser build keeps them in OPFS or IndexedDB).
//!
//! Every top-level [`Session::execute`] persists the ops it produced (fsynced) before returning,
//! so a crash loses at most the command in flight. The log is compacted into a snapshot when it
//! so a crash loses at most the command in flight. When that write fails the command returns
//! [`EngineError::NotSaved`]: its change stays applied in memory and queued, and every later save
//! retries the queue ([`Session::unsaved`] reports it meanwhile), so nothing is lost once the
//! disk is writable again. The log is compacted into a snapshot when it
//! grows (see [`lightcraft_catalog::SnapshotPolicy`]; written by a worker thread on native, see
//! [`lightcraft_catalog::journal`]) and on [`Session::close_library`].
use std::path::{Path, PathBuf};
use lightcraft_catalog::{FsStore, Journal, LoadReport, Store};
use lightcraft_catalog::{FsStore, Journal, LibraryLock, LoadReport, Store};
use lightcraft_develop::Preset;
use serde::{Deserialize, Serialize};
@@ -52,8 +55,24 @@ pub struct Library {
pub report: LoadReport,
/// Last persistence error (shown by the UI; ops stay pending and are retried).
pub last_error: Option<String>,
/// Why the last append of queued ops failed; `None` once a save succeeds. While set, changes
/// exist only in memory ([`Session::unsaved`]).
pub unsaved_error: Option<String>,
/// When the frame loop may retry a failed append ([`Session::persist_if_dirty`] backs off).
retry_at: Option<web_time::Instant>,
/// What forgetting untouched Local records did when the library opened.
pub forgot_local: Option<lightcraft_catalog::ForgetPlan>,
presets_written: String,
view_written: Vec<u8>,
/// Settings files that were unreadable or damaged when the library opened (`library.info` →
/// `settingsWarnings`; shown by the UI once, see [`Session::take_library_warnings`]).
pub settings_warnings: Vec<String>,
warnings_reported: usize,
/// Settings files that couldn't be read or set aside: never overwritten this session.
blocked: Vec<&'static str>,
/// Keeps other processes out of this library while it's open (last: released after the
/// journal's background snapshot has landed).
lock: Option<LibraryLock>,
}
/// Where a library's files live (see [`Session::open_library_in`]).
@@ -133,6 +152,9 @@ struct PrefsFile {
cache_mb: u32,
/// Folder for smart previews (default: `Smart Previews` in the library).
smart_previews_dir: Option<String>,
/// Days after which untouched Local records of unbrowsed folders are forgotten (missing =
/// the default, 0 = never).
forget_local_days: Option<u32>,
}
fn presets_json(s: &Session) -> String {
@@ -155,23 +177,115 @@ fn poll_compaction(lib: &mut Library) {
}
}
fn read_json<T: serde::de::DeserializeOwned>(store: &mut dyn Store, name: &str) -> Option<T> {
store.read(name).ok().flatten().and_then(|b| serde_json::from_slice(&b).ok())
/// How long the frame loop waits before retrying a failed append (commands retry at once).
const RETRY_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2);
/// `a` and `b` name the same directory.
fn same_dir(a: &Path, b: &Path) -> bool {
match (std::fs::canonicalize(a), std::fs::canonicalize(b)) {
(Ok(a), Ok(b)) => a == b,
_ => a == b,
}
}
/// This program, for the lock owner note ("LightCraft", "lightcraft-cli").
fn program_name() -> String {
let exe = std::env::current_exe().ok().and_then(|p| p.file_stem().map(|s| s.to_string_lossy().to_string()));
match exe.as_deref() {
Some("lightcraft") | None => "LightCraft".into(),
Some(other) => other.to_string(),
}
}
/// Problems reading the settings files (`presets.json`, `prefs.json`, `view.json`) when the library
/// opened (issue #103). A missing file is not a problem (defaults). A file that doesn't parse is
/// kept as `<name>.corrupt-<unix time>` before the defaults are used (so the next save can't
/// lose it); a file that can't be read at all (locked by another program, I/O error) is never
/// written this session, so its content survives until the library is opened again.
#[derive(Default)]
struct SettingsLoad {
warnings: Vec<String>,
/// Files not to overwrite this session.
blocked: Vec<&'static str>,
}
impl SettingsLoad {
fn read<T: serde::de::DeserializeOwned>(&mut self, store: &mut dyn Store, name: &'static str) -> Option<T> {
let bytes = match store.read(name) {
Ok(b) => b?,
Err(e) => {
log::error!("library: {name}: {e}");
self.blocked.push(name);
self.warnings.push(format!(
"{name} couldn't be read ({e}). LightCraft uses the defaults for now and won't overwrite the file; reopen the library to try again."
));
return None;
}
};
let err = match serde_json::from_slice(&bytes) {
Ok(v) => return Some(v),
Err(e) => e,
};
let secs = web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let keep = format!("{name}.corrupt-{secs}");
log::error!("library: {name} is damaged: {err}");
match store.write_atomic(&keep, &bytes) {
Ok(()) => self.warnings.push(format!("{name} is damaged ({err}). It was kept as {keep}, and the defaults are used.")),
Err(w) => {
self.blocked.push(name);
self.warnings.push(format!(
"{name} is damaged ({err}) and couldn't be set aside ({w}). LightCraft uses the defaults and won't overwrite the file."
));
}
}
None
}
}
impl Session {
/// Open (or create) the library at `dir` into this session, replacing its catalog. With
/// `seed_demo`, a newly created library starts with the procedural demo photos.
///
/// The library is locked for this session ([`lightcraft_catalog::lock`]): if another process
/// has it open, this fails with [`EngineError::LibraryInUse`] and nothing is read or changed.
/// Reopening the library this session already has open keeps its lock.
pub fn open_library(&mut self, dir: impl AsRef<Path>, seed_demo: bool) -> Result<&LoadReport> {
let dir = dir.as_ref().to_path_buf();
let open = || FsStore::open(&dir).map_err(|e| EngineError::Other(format!("can't open library {}: {e}", dir.display())));
let stores = LibraryStores { catalog: Box::new(open()?), files: Box::new(open()?), on_disk: true, dir: dir.clone() };
self.open_library_in(stores, seed_demo)
// the library open in this session (same directory): hand its lock over
let reused = self.library.as_mut().filter(|l| same_dir(&l.dir, &dir)).and_then(|l| l.lock.take());
let reusing = reused.is_some();
let mut lock = match reused {
Some(l) => Some(l),
None => Some(LibraryLock::acquire(&dir, &program_name()).map_err(|e| EngineError::LibraryInUse(e.to_string()))?),
};
if let Err(e) = self.open_stores(stores, seed_demo, &mut lock) {
if reusing && let Some(lib) = self.library.as_mut() {
lib.lock = lock.take();
}
return Err(e);
}
self.loaded_report()
}
/// Open (or create) a library whose files live in `stores` (e.g. browser storage), replacing
/// this session's catalog.
/// this session's catalog. Not locked: the host guards against a second opener (see
/// [`Session::open_library`] for directories).
pub fn open_library_in(&mut self, stores: LibraryStores, seed_demo: bool) -> Result<&LoadReport> {
self.open_stores(stores, seed_demo, &mut None)?;
self.loaded_report()
}
fn loaded_report(&self) -> Result<&LoadReport> {
match &self.library {
Some(lib) => Ok(&lib.report),
None => Err(EngineError::Other("library closed while opening".into())),
}
}
/// Open `stores`; the new library takes `lock` once nothing can fail any more.
fn open_stores(&mut self, stores: LibraryStores, seed_demo: bool, lock: &mut Option<LibraryLock>) -> Result<()> {
let LibraryStores { dir, catalog, mut files, on_disk } = stores;
self.media.smart_dir = on_disk.then(|| crate::smart::dir(&dir));
// the current library may be these same files: let its background snapshot land first
@@ -192,8 +306,9 @@ impl Session {
crate::demo::load(self);
journal.snapshot(&self.catalog)?;
}
let mut settings = SettingsLoad::default();
// presets
if let Some(f) = read_json::<PresetsFile>(files.as_mut(), "presets.json") {
if let Some(f) = settings.read::<PresetsFile>(files.as_mut(), "presets.json") {
for p in &mut self.presets {
p.favorite = p.builtin && f.favorites.contains(&p.id) || (!p.builtin && p.favorite);
}
@@ -207,7 +322,7 @@ impl Session {
self.profile_recent = f.profile_recent.into_iter().filter(known).take(crate::presets::RECENT_PROFILES).collect();
}
// preferences
let prefs = read_json::<PrefsFile>(files.as_mut(), "prefs.json").unwrap_or_default();
let prefs = settings.read::<PrefsFile>(files.as_mut(), "prefs.json").unwrap_or_default();
self.xmp = prefs.xmp;
self.last_export = prefs.last_export;
self.export_presets = prefs.export_presets;
@@ -222,12 +337,13 @@ impl Session {
self.recent_keywords = prefs.recent_keywords;
self.import_defaults = prefs.import;
self.cache_mb = prefs.cache_mb;
self.forget_local_days = prefs.forget_local_days.unwrap_or(lightcraft_catalog::DEFAULT_FORGET_DAYS);
self.smart_previews_dir = prefs.smart_previews_dir.filter(|_| on_disk).map(PathBuf::from);
if let Some(d) = &self.smart_previews_dir {
self.media.smart_dir = Some(d.clone());
}
// view state
if let Some(v) = read_json::<ViewFile>(files.as_mut(), "view.json") {
if let Some(v) = settings.read::<ViewFile>(files.as_mut(), "view.json") {
self.source = v.source;
self.browse = v.browse;
self.filter = v.filter;
@@ -248,32 +364,75 @@ impl Session {
self.media.attach_disk_cache(&dir.join("thumbs"), self.cache_bytes());
}
let view_written = self.view_json();
let library = self.library.insert(Library { dir, on_disk, journal, files, report, last_error: None, presets_written, view_written });
Ok(&library.report)
self.library = Some(Library {
dir,
on_disk,
journal,
files,
report,
last_error: None,
unsaved_error: None,
retry_at: None,
forgot_local: None,
presets_written,
view_written,
settings_warnings: settings.warnings,
warnings_reported: 0,
blocked: settings.blocked,
lock: lock.take(),
});
// forget untouched Local records of folders not browsed for a while (journaled at once)
if self.forget_local_days > 0 {
let plan = self.forget_local(false, None);
if !plan.evict.is_empty() {
self.compact_soon();
} else if let Err(e) = self.persist() {
log::error!("library: {e}");
}
if let Some(lib) = self.library.as_mut() {
lib.forgot_local = Some(plan);
}
}
Ok(())
}
/// Write pending ops to the log (fsynced), compact when due, and save changed presets.
/// Called after every top-level command; cheap when nothing changed.
/// Called after every top-level command; cheap when nothing changed. Fails (with
/// [`EngineError::NotSaved`]) only when the queued ops couldn't be written; they stay queued.
pub fn persist(&mut self) -> Result<()> {
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if !self.pending_log.is_empty() {
if let Err(e) = lib.journal.append(&self.pending_log) {
lib.last_error = Some(e.to_string());
log::error!("library: {e}");
return Err(e.into());
// the ops stay queued (and applied in memory): the next persist retries them
let reason = e.to_string();
log::error!("library: {} change(s) not written to disk: {reason}", self.pending_log.len());
lib.last_error = Some(reason.clone());
lib.unsaved_error = Some(reason.clone());
lib.retry_at = Some(web_time::Instant::now() + RETRY_BACKOFF);
return Err(EngineError::NotSaved(reason));
}
if lib.unsaved_error.take().is_some() {
log::info!("library: queued changes written to disk");
}
lib.retry_at = None;
self.pending_log.clear();
lib.last_error = None;
}
poll_compaction(lib);
// Never snapshot mid-interaction: the catalog then holds an uncommitted preview value.
// The snapshot is written by a worker thread (natively); appends go on meanwhile.
if lib.journal.wants_snapshot() && self.interaction.is_none() {
lib.journal.snapshot_in_background(&self.catalog)?;
// A failed compaction loses nothing (the log is kept whole and compacted later): it is
// reported (`library.info` → `lastError`), not returned.
if lib.journal.wants_snapshot()
&& self.interaction.is_none()
&& let Err(e) = lib.journal.snapshot_in_background(&self.catalog)
{
log::error!("library: compaction: {e}");
lib.last_error = Some(format!("compaction: {e}"));
}
let presets = presets_json(self);
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if presets != lib.presets_written {
if presets != lib.presets_written && !lib.blocked.contains(&"presets.json") {
if let Err(e) = lib.files.write_atomic("presets.json", presets.as_bytes()) {
log::error!("library: presets: {e}");
} else {
@@ -286,24 +445,52 @@ impl Session {
/// Persist if commands left ops pending (cheap; frontends call it once per frame for state
/// changed outside [`Session::execute`]).
pub fn persist_if_dirty(&mut self) {
if self.library.is_some() && !self.pending_log.is_empty() {
let backing_off = self.library.as_ref().and_then(|l| l.retry_at).is_some_and(|t| web_time::Instant::now() < t);
if self.library.is_some() && !self.pending_log.is_empty() && !backing_off {
let _ = self.persist();
} else if let Some(lib) = self.library.as_mut() {
poll_compaction(lib);
}
}
/// Changes applied in memory but not yet written to disk because saving failed: the number of
/// queued ops and why (`None` when everything is saved, or no library is open). Retried by
/// every command and by [`Session::persist_if_dirty`].
pub fn unsaved(&self) -> Option<(usize, &str)> {
let lib = self.library.as_ref()?;
lib.unsaved_error.as_deref().filter(|_| !self.pending_log.is_empty()).map(|e| (self.pending_log.len(), e))
}
/// Flush everything and write a snapshot (on quit). Ends an open interaction first.
///
/// The snapshot is written even when appending the queued ops fails: it is a fresh file
/// holding everything in memory, so it saves those ops too (the log handle may be the only
/// thing that's broken). Fails only if nothing could be saved; the ops then stay queued.
pub fn close_library(&mut self) -> Result<()> {
if self.library.is_none() {
return Ok(());
}
let _ = self.end_interaction();
self.persist()?;
let persisted = self.persist();
self.save_view();
let unlogged = if persisted.is_err() { self.pending_log.len() as u64 } else { 0 };
let Some(lib) = self.library.as_mut() else { return Ok(()) };
lib.journal.snapshot(&self.catalog)?;
Ok(())
let before = lib.journal.seq();
let snapshot = lib.journal.snapshot_with_unlogged(&self.catalog, unlogged);
if unlogged > 0 && lib.journal.seq() == before + unlogged {
// the snapshot holds the queued ops: they are saved, never append them again
log::info!("library: {unlogged} queued change(s) saved in the closing snapshot");
self.pending_log.clear();
lib.unsaved_error = None;
lib.retry_at = None;
lib.last_error = None;
}
match (persisted, snapshot) {
(_, Ok(())) => Ok(()),
// neither the log nor the snapshot took the queued ops
(Err(EngineError::NotSaved(e)), Err(s)) if lib.journal.seq() == before => Err(EngineError::NotSaved(format!("{e}; snapshot: {s}"))),
(_, Err(s)) => Err(s.into()),
}
}
fn view_json(&self) -> Vec<u8> {
@@ -326,7 +513,7 @@ impl Session {
}
let v = self.view_json();
let Some(lib) = self.library.as_mut() else { return };
if v != lib.view_written {
if v != lib.view_written && !lib.blocked.contains(&"view.json") {
match lib.files.write_atomic("view.json", &v) {
Ok(()) => lib.view_written = v,
Err(e) => log::error!("library: view: {e}"),
@@ -351,12 +538,26 @@ impl Session {
import: self.import_defaults.clone(),
cache_mb: self.cache_mb,
smart_previews_dir: self.smart_previews_dir.as_ref().map(|d| d.to_string_lossy().to_string()),
forget_local_days: Some(self.forget_local_days),
})
.unwrap_or_default();
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if lib.blocked.contains(&"prefs.json") {
return Err(EngineError::Other(
"prefs: prefs.json couldn't be read when the library opened, so it isn't overwritten; reopen the library to save preferences".into(),
));
}
lib.files.write_atomic("prefs.json", &v).map_err(|e| EngineError::Other(format!("prefs: {e}")))
}
/// Settings-file warnings of the open library not handed out yet (the UI shows each once).
pub fn take_library_warnings(&mut self) -> Vec<String> {
let Some(lib) = self.library.as_mut() else { return vec![] };
let new = lib.settings_warnings[lib.warnings_reported..].to_vec();
lib.warnings_reported = lib.settings_warnings.len();
new
}
/// The thumbnail disk cache budget in bytes ([`Session::cache_mb`], else the default).
pub fn cache_bytes(&self) -> u64 {
if self.cache_mb == 0 { crate::media::DISK_CACHE_BYTES } else { u64::from(self.cache_mb) << 20 }
@@ -373,6 +574,24 @@ impl Session {
self.save_prefs()
}
/// Save, then start compacting in the background (natively; synchronously elsewhere): after
/// a change that shrinks the catalog a lot, so the smaller snapshot replaces the old one.
/// Errors are logged and reported like other persistence errors (nothing is lost).
pub fn compact_soon(&mut self) {
if let Err(e) = self.persist() {
log::error!("library: {e}");
return;
}
if self.interaction.is_some() {
return;
}
let Some(lib) = self.library.as_mut() else { return };
if let Err(e) = lib.journal.snapshot_in_background(&self.catalog) {
log::error!("library: compaction: {e}");
lib.last_error = Some(format!("compaction: {e}"));
}
}
/// Compact the log into a snapshot now.
pub fn compact_library(&mut self) -> Result<()> {
self.persist()?;
+1 -1
View File
@@ -290,7 +290,7 @@ impl MediaCache {
&& !std::path::Path::new(path).exists()
{
let sp = dir.join(crate::smart::file_name(p));
if sp.exists() {
if crate::smart::is_valid(&sp) {
return SourceRef::Smart { path: sp };
}
}
+6 -1
View File
@@ -44,7 +44,12 @@ fn total_ram() -> Option<usize> {
let out = std::process::Command::new("/usr/sbin/sysctl").args(["-n", "hw.memsize"]).output().ok()?;
String::from_utf8_lossy(&out.stdout).trim().parse().ok()
}
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
#[cfg(target_os = "freebsd")]
{
let out = std::process::Command::new("/sbin/sysctl").args(["-n", "hw.physmem"]).output().ok()?;
String::from_utf8_lossy(&out.stdout).trim().parse().ok()
}
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "freebsd")))]
{
None
}
+9 -8
View File
@@ -304,15 +304,13 @@ fn apply_auto(s: &mut DevelopSettings, src: &Rgb32f, info: &SourceInfo) {
s.color.saturation = a.saturation;
}
/// A free file name `dir/<stem>-<suffix>.dng` (`-2`, `-3`… when taken).
fn output_path(first: &str, suffix: &str) -> String {
/// `dir/<stem>-<suffix>.dng`, then `-2`, `-3`… (the candidates for a merge's output).
fn output_names(first: &str, suffix: &str) -> impl Iterator<Item = std::path::PathBuf> {
let p = std::path::Path::new(first);
let dir = p.parent().map(|d| d.to_path_buf()).unwrap_or_default();
let stem = p.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| "Merge".into());
let cand = |k: u64| dir.join(if k == 1 { format!("{stem}-{suffix}.dng") } else { format!("{stem}-{suffix}-{k}.dng") });
// (`u64::MAX` taken names can't exist; the last candidate is the fallback)
let free = (1..u64::MAX).map(cand).find(|c| !c.exists()).unwrap_or_else(|| cand(u64::MAX));
free.to_string_lossy().to_string()
let suffix = suffix.to_string();
(1u64..).map(move |k| dir.join(if k == 1 { format!("{stem}-{suffix}.dng") } else { format!("{stem}-{suffix}-{k}.dng") }))
}
/// Parse `merge.*` command parameters into a kind and finishing options.
@@ -367,8 +365,11 @@ impl Session {
/// apply Auto Settings / the auto crop. Returns `{id, path, …info}`.
pub fn finish_merge(&mut self, job: &MergeJob, out: MergeOutput) -> Result<Value> {
let first = &job.sources.first().ok_or_else(|| EngineError::Other("nothing merged".into()))?.1;
let path = output_path(first, job.kind.suffix());
std::fs::write(&path, &out.dng).map_err(|e| EngineError::Other(format!("{path}: {e}")))?;
// a new file under a free name (never replacing one), complete and synced before it appears
let path = lightcraft_catalog::safe_file::write_new_unique(&mut output_names(first, job.kind.suffix()), &out.dng)
.map_err(|e| EngineError::Other(format!("could not write the merged DNG next to {first}: {e}")))?
.to_string_lossy()
.to_string();
let report = crate::import::import(self, std::slice::from_ref(&path), crate::import::ImportMode::Add)?;
let id = report
.imported
+102
View File
@@ -0,0 +1,102 @@
//! Never write over an original: exports, renders and other files LightCraft writes for the user
//! must not land on a catalogued photo's own file (or its XMP sidecar), whatever the conflict
//! setting or exact path asked for. [`OriginalGuard`] is a snapshot of the library's originals
//! that answers "would writing here replace one?" by file identity (the same file under another
//! spelling, through `..`, a link or a case-insensitive volume counts), not by comparing strings.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use lightcraft_catalog::safe_file::same_file;
use lightcraft_catalog::{Catalog, Source};
/// The library's originals and their sidecars, looked up by file name.
#[derive(Debug, Default)]
pub struct OriginalGuard {
/// lower-case file name → (path on disk, what it is: "the original of IMG_1.CR3", …)
by_name: HashMap<String, Vec<(PathBuf, String)>>,
}
impl OriginalGuard {
/// The originals of every photo in `catalog`.
pub fn new(catalog: &Catalog) -> Self {
let mut g = Self::default();
for p in catalog.photos() {
if let Source::File { path } = &p.source {
g.add(Path::new(path), &p.file_name);
}
}
g
}
/// Protect `path` (an original named `label`) and its sidecars (`IMG_1.xmp`, `IMG_1.CR3.xmp`).
pub fn add(&mut self, path: &Path, label: &str) {
let Some(name) = path.file_name().map(|n| n.to_string_lossy().to_string()) else { return };
let mut put = |file: String, p: PathBuf, what: String| self.by_name.entry(file.to_lowercase()).or_default().push((p, what));
put(name.clone(), path.to_path_buf(), format!("the original of {label}"));
let stem = path.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_default();
put(format!("{stem}.xmp"), path.with_extension("xmp"), format!("the XMP sidecar of {label}"));
put(format!("{name}.xmp"), PathBuf::from(format!("{}.xmp", path.to_string_lossy())), format!("the XMP sidecar of {label}"));
}
/// `Err` (a message for the user) when writing `target` would replace a protected file.
pub fn check(&self, target: &Path) -> Result<(), String> {
if std::fs::symlink_metadata(target).is_err() {
return Ok(()); // nothing there to replace
}
let mut names: Vec<String> = target.file_name().map(|n| n.to_string_lossy().to_lowercase()).into_iter().collect();
if let Some(n) = std::fs::canonicalize(target).ok().and_then(|c| c.file_name().map(|n| n.to_string_lossy().to_lowercase())) {
names.push(n);
}
for n in &names {
for (p, what) in self.by_name.get(n).into_iter().flatten() {
if same_file(p, target) {
return Err(format!(
"{} is {what} in the library: LightCraft never writes over an original (choose another folder or file name)",
target.display()
));
}
}
}
Ok(())
}
}
impl crate::Session {
/// A guard over this library's originals (see [`OriginalGuard`]).
pub fn original_guard(&self) -> OriginalGuard {
OriginalGuard::new(&self.catalog)
}
/// `Err` when writing `path` would replace a catalogued photo's original or sidecar.
pub fn check_write_target(&self, path: &str) -> Result<(), String> {
self.original_guard().check(Path::new(path))
}
}
#[cfg(test)]
mod tests {
use super::*;
use lightcraft_catalog::{Op, Photo, PhotoId};
#[test]
fn originals_and_their_sidecars_are_protected_by_identity() {
let d = std::env::temp_dir().join(format!("lc-originals-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(d.join("sub")).unwrap();
let orig = d.join("IMG_1.jpg");
std::fs::write(&orig, b"original").unwrap();
std::fs::write(d.join("IMG_1.xmp"), b"<x/>").unwrap();
std::fs::write(d.join("other.jpg"), b"other").unwrap();
let mut c = Catalog::default();
let p = Photo::new(PhotoId(1), Source::File { path: orig.to_string_lossy().to_string() }, "IMG_1.jpg", "JPEG", 10, 10, "2026-01-01");
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
let g = OriginalGuard::new(&c);
assert!(g.check(&orig).unwrap_err().contains("original of IMG_1.jpg"));
assert!(g.check(&d.join("sub/../IMG_1.jpg")).is_err(), "another spelling of the same file");
assert!(g.check(&d.join("IMG_1.xmp")).unwrap_err().contains("sidecar"));
assert!(g.check(&d.join("other.jpg")).is_ok(), "an ordinary existing file");
assert!(g.check(&d.join("new.jpg")).is_ok(), "a free name");
let _ = std::fs::remove_dir_all(&d);
}
}
+522 -65
View File
@@ -11,7 +11,11 @@
//!
//! Safety:
//! - a target that exists on disk, or that another photo in the batch gets, receives a `-1`, `-2`…
//! suffix — no file is ever overwritten;
//! suffix — no file is ever overwritten. Whether an existing target is the photo itself (a
//! change of letter case on a case-insensitive volume) is decided by file identity, not by
//! comparing names, so on a case-sensitive volume `img_1.jpg` next to `IMG_1.JPG` is a collision
//! (issue #95); moves never replace a file even if one appears after the check (hard link +
//! unlink where the volume supports it);
//! - the moves happen before the catalog changes; if one fails, the ones already done are moved
//! back and nothing is committed;
//! - the catalog op ([`Op::SetFile`]) is undoable: undo/redo move the files back and forth (see
@@ -297,58 +301,251 @@ fn sidecars(path: &str) -> Vec<(PathBuf, SidecarNaming, bool)> {
v
}
/// Move `from` to `to` (never over an existing file) together with its sidecars. On error nothing
/// is left moved.
pub fn move_file(from: &str, to: &str) -> std::result::Result<(), String> {
/// The file-system primitives renaming uses. [`RealFs`] is the disk; tests substitute a model of
/// a case-insensitive or case-sensitive volume (and inject failures).
pub(crate) trait MoveFs {
/// Something (file, folder, link) answers to `p`.
fn exists(&self, p: &Path) -> bool;
fn is_file(&self, p: &Path) -> bool;
/// `a` and `b` name the same file (e.g. `IMG.JPG` and `img.jpg` on a case-insensitive
/// volume). Decided by file identity, never by comparing names: on a case-sensitive volume
/// they are two different photos. When in doubt: `false`.
fn same_file(&self, a: &Path, b: &Path) -> bool;
/// Rename `a` to `b`, failing (never replacing) when `b` exists.
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()>;
/// Copy `a` to a new file `b` (an independent copy, never a link), failing when `b` exists.
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()>;
fn remove_file(&self, p: &Path) -> std::io::Result<()>;
/// Another photo file beside the stem-named sidecar `sidecar` that shares its stem (e.g.
/// `IMG_1.JPG` for `IMG_1.xmp`), which still uses that sidecar.
fn stem_sibling(&self, sidecar: &Path) -> Option<String>;
}
/// The real file system.
pub(crate) struct RealFs;
impl MoveFs for RealFs {
fn exists(&self, p: &Path) -> bool {
std::fs::symlink_metadata(p).is_ok()
}
fn is_file(&self, p: &Path) -> bool {
p.is_file()
}
fn same_file(&self, a: &Path, b: &Path) -> bool {
if a == b {
return true;
}
let (Ok(ma), Ok(mb)) = (std::fs::metadata(a), std::fs::metadata(b)) else { return false };
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
ma.dev() == mb.dev() && ma.ino() == mb.ino()
}
#[cfg(not(unix))]
{
// no stable file id in std here: `b` is `a` under another spelling when they look
// alike and the folder has no entry spelled exactly like `b` (on a case-sensitive
// folder holding both `IMG.JPG` and `img.jpg`, both entries are listed)
let (Some(dir), Some(name)) = (b.parent(), b.file_name()) else { return false };
if a.parent() != Some(dir) || ma.len() != mb.len() || ma.modified().ok() != mb.modified().ok() {
return false;
}
let Ok(entries) = std::fs::read_dir(dir) else { return false };
!entries.flatten().any(|e| e.file_name() == name)
}
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, format!("{} already exists", b.display())));
}
// A hard link fails when `b` exists — even if it appeared since the check above — so
// nothing is ever replaced. Volumes without hard links (FAT, some network shares) fall back
// to a plain rename after the check.
match std::fs::hard_link(a, b) {
Ok(()) => std::fs::remove_file(a).inspect_err(|_| {
// both names are the same file: dropping the new one loses nothing
let _ = std::fs::remove_file(b);
}),
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => Err(e),
Err(_) => std::fs::rename(a, b),
}
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
use std::io::Write;
let bytes = std::fs::read(a)?;
let mut f = std::fs::OpenOptions::new().write(true).create_new(true).open(b)?;
f.write_all(&bytes).and_then(|_| f.sync_all()).inspect_err(|_| {
let _ = std::fs::remove_file(b);
})
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
std::fs::remove_file(p)
}
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
crate::import_move::sibling_photo(sidecar)
}
}
/// `from` → `to` differs only in letter case and `to` is that very file (a case-insensitive
/// volume): the rename must go through a temporary name.
fn is_respelling(fs: &dyn MoveFs, from: &Path, to: &Path) -> bool {
from != to && from.to_string_lossy().to_lowercase() == to.to_string_lossy().to_lowercase() && fs.exists(to) && fs.same_file(from, to)
}
/// Rename `a` to `b` without ever replacing a file; a change of letter case only goes through a
/// unique temporary name so case-insensitive volumes see a real change.
fn move_one(fs: &dyn MoveFs, a: &Path, b: &Path) -> std::io::Result<()> {
if !is_respelling(fs, a, b) {
return fs.rename_no_replace(a, b);
}
let tmp = (0..1000u32)
.map(|k| a.with_file_name(format!(".lc-rename-{}-{k}", std::process::id())))
.find(|t| !fs.exists(t))
.ok_or_else(|| std::io::Error::other("no free temporary name"))?;
fs.rename_no_replace(a, &tmp)?;
fs.rename_no_replace(&tmp, b).map_err(|e| match fs.rename_no_replace(&tmp, a) {
Ok(()) => e,
Err(e2) => std::io::Error::other(format!("{e}; it could not be given its old name back either ({e2}): it is now {}", tmp.display())),
})
}
/// Why [`move_file`] failed, and what it left behind.
#[derive(Clone, Debug, PartialEq)]
pub struct MoveError {
/// What went wrong, including every step that could not be undone and where that file is now.
pub message: String,
/// The file itself ended up at the new path: it moved, a sidecar didn't, and moving the file
/// back failed too. Its sidecars that did move stay with it.
pub moved: bool,
}
impl std::fmt::Display for MoveError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.message)
}
}
/// Move `from` to `to` (never over an existing file) together with its sidecars. On error the
/// moves are undone; whatever can't be undone is named in the error (see [`MoveError`]).
pub fn move_file(from: &str, to: &str) -> std::result::Result<(), MoveError> {
move_file_with(&RealFs, from, to)
}
/// [`move_file`] on the file system `fs`.
pub(crate) fn move_file_with(fs: &dyn MoveFs, from: &str, to: &str) -> std::result::Result<(), MoveError> {
if from == to {
return Ok(());
}
let fail = |message: String| MoveError { message, moved: false };
let (f, t) = (Path::new(from), Path::new(to));
// a case-only change on a case-insensitive file system "exists" already: that is this file
let case_only = from.to_lowercase() == to.to_lowercase();
if t.exists() && !case_only {
return Err(format!("{to} already exists"));
// `to` may be this very file spelled differently (case-insensitive volume); any other file
// there — e.g. `img.jpg` next to `IMG.JPG` on a case-sensitive volume — is never replaced
if fs.exists(t) && !is_respelling(fs, f, t) {
return Err(fail(format!("{to} already exists")));
}
let mut done: Vec<(PathBuf, PathBuf)> = Vec::new();
let mv = |a: &Path, b: &Path| -> std::io::Result<()> {
if case_only {
// via a temporary name, so case-insensitive file systems see a real change
let tmp = a.with_file_name(format!(".lc-rename-{}", std::process::id()));
std::fs::rename(a, &tmp)?;
std::fs::rename(&tmp, b).inspect_err(|_| {
let _ = std::fs::rename(&tmp, a);
})
} else {
std::fs::rename(a, b)
}
};
mv(f, t).map_err(|e| format!("rename {from}: {e}"))?;
done.push((f.to_path_buf(), t.to_path_buf()));
move_one(fs, f, t).map_err(|e| fail(format!("rename {from}: {e}")))?;
// sidecars moved (or copied: `true`)
let mut done: Vec<(PathBuf, PathBuf, bool)> = Vec::new();
for (sc, naming, upper) in sidecars(from) {
if !sc.is_file() {
if !fs.is_file(&sc) {
continue;
}
let mut dst = crate::sidecar::sidecar_path(to, naming);
if upper {
dst = dst.with_extension("XMP");
}
if dst.exists() && !case_only {
// a stem sidecar another file still uses (`IMG_1.xmp` of `IMG_1.CR3` and `IMG_1.JPG`)
// is copied, not taken away from it (issue #92)
let shared = naming == SidecarNaming::Stem && fs.stem_sibling(&sc).is_some();
if fs.exists(&dst) && (shared || !is_respelling(fs, &sc, &dst)) {
continue; // never overwrite: the old sidecar stays where it was
}
if let Err(e) = mv(&sc, &dst) {
for (a, b) in done.iter().rev() {
let _ = std::fs::rename(b, a);
let r = if shared { fs.copy_no_replace(&sc, &dst) } else { move_one(fs, &sc, &dst) };
if let Err(e) = r {
let mut message = format!("rename {}: {e}", sc.display());
// the file first: if it can't go back, the sidecars that moved stay with it
if let Err(e) = move_one(fs, t, f) {
message.push_str(&format!("; {from} could not be moved back ({e}): it is now {to}"));
if !done.is_empty() {
let names: Vec<String> = done.iter().map(|(_, b, _)| b.display().to_string()).collect();
message.push_str(&format!(" with its sidecar {}", names.join(", ")));
}
return Err(MoveError { message, moved: true });
}
return Err(format!("rename {}: {e}", sc.display()));
for (a, b, copied) in done.iter().rev() {
if *copied {
// the original is still there: the copy just goes (a leftover copy loses nothing)
let _ = fs.remove_file(b);
} else if let Err(e) = move_one(fs, b, a) {
message.push_str(&format!("; sidecar {} could not be moved back ({e}): it is now {}", a.display(), b.display()));
}
}
return Err(fail(message));
}
done.push((sc, dst));
done.push((sc, dst, shared));
}
Ok(())
}
/// What a failed batch of moves ([`move_all`]) left behind.
#[derive(Debug)]
pub(crate) struct BatchMoveError {
/// The first failure and every move that could not be undone.
pub message: String,
/// Moves that stayed done, (from, to): those files are at `to` now.
pub stuck: Vec<(String, String)>,
}
/// Move the files `moves` names, all or nothing: after a failure the earlier moves are undone.
/// Moves that can't be undone (e.g. the network share went away) are reported, not ignored.
pub(crate) fn move_all(fs: &dyn MoveFs, moves: &[(String, String)]) -> std::result::Result<(), BatchMoveError> {
let mut done: Vec<&(String, String)> = Vec::new();
for m in moves {
let Err(e) = move_file_with(fs, &m.0, &m.1) else {
done.push(m);
continue;
};
let mut message = e.message;
let mut stuck = Vec::new();
if e.moved {
stuck.push(m.clone());
}
for d in done.iter().rev() {
if let Err(e) = move_file_with(fs, &d.1, &d.0) {
message.push_str(&format!("; {} could not be moved back to {}: {}", d.1, d.0, e.message));
if !e.moved {
stuck.push((*d).clone());
}
}
}
return Err(BatchMoveError { message, stuck });
}
Ok(())
}
/// The `SetFile` ops in `op` (nested batches included) that point a photo at one of `to`.
fn set_file_ops_to(op: &Op, to: &HashSet<&str>, out: &mut Vec<Op>) {
match op {
Op::Batch { ops } => ops.iter().for_each(|o| set_file_ops_to(o, to, out)),
Op::SetFile { source: Source::File { path }, .. } if to.contains(path.as_str()) => out.push(op.clone()),
_ => {}
}
}
impl Session {
/// Plan renaming `ids` with `template` (sequence numbers from `start`), resolving collisions.
pub fn plan_rename(&self, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
self.plan_rename_with(&RealFs, ids, template, start)
}
/// [`Session::plan_rename`] on the file system `fs`.
pub(crate) fn plan_rename_with(&self, fs: &dyn MoveFs, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
// photos sharing one file (virtual copies) rename once
let mut seen_paths: HashSet<String> = HashSet::new();
let mut taken: HashSet<String> = HashSet::new(); // lower-case target paths/names claimed in this batch
@@ -382,10 +579,12 @@ impl Session {
let name = candidate(k);
let tp = dir.join(&name).to_string_lossy().to_string();
let key = tp.to_lowercase();
let same = key == path.to_lowercase();
// this very file, maybe spelled differently (case-insensitive volume)? By
// identity: on a case-sensitive volume `img_1.jpg` may be another photo
let same = tp == *path || is_respelling(fs, Path::new(path), Path::new(&tp));
// free: not claimed in this batch and not on disk (unless it is this very file).
// A file this batch moves away still counts as taken: simple and safe.
let on_disk = Path::new(&tp).exists() && !same;
let on_disk = fs.exists(Path::new(&tp)) && !same;
if !taken.contains(&key) && !on_disk {
taken.insert(key);
break (name, Some(tp));
@@ -415,21 +614,16 @@ impl Session {
/// Carry out `plans`: move the files (rolled back on failure), then commit one undoable op
/// that updates the photos (and their virtual copies).
///
/// When a failure can't be fully rolled back, the files that kept their new names are
/// committed as a smaller undoable rename, so the library points at them (nothing goes
/// missing), and the error lists them (old → new).
pub fn apply_rename(&mut self, plans: &[RenamePlan]) -> Result<usize> {
let mut moved: Vec<(String, String)> = Vec::new();
for pl in plans {
if let (Some(a), Some(b)) = (&pl.from_path, &pl.to_path)
&& a != b
{
if let Err(e) = move_file(a, b) {
for (a, b) in moved.iter().rev() {
let _ = move_file(b, a);
}
return Err(EngineError::Other(e));
}
moved.push((a.clone(), b.clone()));
}
}
self.apply_rename_with(&RealFs, plans)
}
/// [`Session::apply_rename`] on the file system `fs`.
pub(crate) fn apply_rename_with(&mut self, fs: &dyn MoveFs, plans: &[RenamePlan]) -> Result<usize> {
let mut ops = Vec::new();
let by_path: HashMap<&str, &RenamePlan> = plans.iter().filter_map(|p| p.from_path.as_deref().map(|f| (f, p))).collect();
for p in self.catalog.photos() {
@@ -447,17 +641,66 @@ impl Session {
}
}
let n = ops.len();
let op = Op::Batch { ops };
let moves: Vec<(String, String)> = plans
.iter()
.filter_map(|pl| match (&pl.from_path, &pl.to_path) {
(Some(a), Some(b)) if a != b => Some((a.clone(), b.clone())),
_ => None,
})
.collect();
if let Err(e) = move_all(fs, &moves) {
return Err(EngineError::Other(self.follow_stuck(&op, e, true)));
}
if n > 0
&& let Err(e) = self.commit(&format!("Rename {n} Photo{}", if n == 1 { "" } else { "s" }), Op::Batch { ops })
&& let Err(err) = self.commit(&format!("Rename {n} Photo{}", if n == 1 { "" } else { "s" }), op)
{
for (a, b) in moved.iter().rev() {
let _ = move_file(b, a);
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
if let Err(e) = move_all(fs, &back) {
return Err(EngineError::Other(format!("{err}; the renamed files could not all be moved back: {}", e.message)));
}
return Err(e);
return Err(err);
}
Ok(n)
}
/// After a failed batch of moves for `op`: point the photos whose files stayed moved at
/// their new paths (an undoable "partly" rename when `undoable`, otherwise a plain logged
/// change), so the library matches the disk. Returns the message for the user.
pub(crate) fn follow_stuck(&mut self, op: &Op, e: BatchMoveError, undoable: bool) -> String {
let mut msg = e.message;
if e.stuck.is_empty() {
return msg;
}
let to: HashSet<&str> = e.stuck.iter().map(|m| m.1.as_str()).collect();
let mut keep = Vec::new();
set_file_ops_to(op, &to, &mut keep);
let list: Vec<String> = e.stuck.iter().map(|(a, b)| format!("{a} → {b}")).collect();
let n = e.stuck.len();
let s = if n == 1 { "" } else { "s" };
let fix = Op::Batch { ops: keep };
let applied = if undoable {
self.commit(&format!("Rename {n} Photo{s} (partly)"), fix)
} else {
match self.catalog.apply(fix.clone()) {
Ok(_) => {
self.pending_log.push(fix);
Ok(())
}
Err(e) => Err(e.into()),
}
};
match applied {
Ok(()) => msg
.push_str(&format!(". {n} file{s} could not be moved back and kept the new name; the library now points there: {}", list.join(", "))),
Err(err) => msg.push_str(&format!(
". {n} file{s} could not be moved back and the library could not follow ({err}); relink with Find Missing Photos: {}",
list.join(", ")
)),
}
msg
}
/// File moves an undo/redo op implies: `SetFile` ops whose source path differs from the photo's
/// current one.
pub(crate) fn file_moves(&self, op: &Op) -> Vec<(String, String)> {
@@ -479,26 +722,240 @@ impl Session {
rec(self, op, &mut v);
v
}
/// Move the files `moves` names; all or nothing.
pub(crate) fn move_files(moves: &[(String, String)]) -> Result<()> {
let mut done: Vec<&(String, String)> = Vec::new();
for m in moves {
if let Err(e) = move_file(&m.0, &m.1) {
for (a, b) in done.iter().rev().map(|m| (&m.0, &m.1)) {
let _ = move_file(b, a);
}
return Err(EngineError::Other(format!("can't undo the rename: {e}")));
}
done.push(m);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::cell::RefCell;
/// A model volume: files by path with contents; case-insensitive (default APFS/NTFS) or
/// case-sensitive (Linux, case-sensitive APFS). `fail` makes renames from a matching source fail.
pub(crate) struct FakeFs {
pub ci: bool,
pub files: RefCell<Vec<(String, Vec<u8>)>>,
pub fail: RefCell<Vec<String>>,
}
impl FakeFs {
pub fn new(ci: bool, files: &[(&str, &str)]) -> FakeFs {
FakeFs {
ci,
files: RefCell::new(files.iter().map(|(p, c)| (p.to_string(), c.as_bytes().to_vec())).collect()),
fail: RefCell::new(Vec::new()),
}
}
fn idx(&self, p: &Path) -> Option<usize> {
let p = p.to_string_lossy();
self.files.borrow().iter().position(|(f, _)| if self.ci { f.to_lowercase() == p.to_lowercase() } else { *f == p })
}
/// The listing: (exact path, contents), sorted.
pub fn listing(&self) -> Vec<(String, String)> {
let mut v: Vec<_> = self.files.borrow().iter().map(|(p, c)| (p.clone(), String::from_utf8_lossy(c).to_string())).collect();
v.sort();
v
}
}
impl MoveFs for FakeFs {
fn exists(&self, p: &Path) -> bool {
self.idx(p).is_some()
}
fn is_file(&self, p: &Path) -> bool {
self.exists(p)
}
fn same_file(&self, a: &Path, b: &Path) -> bool {
self.idx(a).is_some() && self.idx(a) == self.idx(b)
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.fail.borrow().iter().any(|f| a.to_string_lossy().contains(f.as_str())) {
return Err(std::io::Error::other("injected failure"));
}
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
self.files.borrow_mut()[i].0 = b.to_string_lossy().to_string();
Ok(())
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
let c = self.files.borrow()[i].1.clone();
self.files.borrow_mut().push((b.to_string_lossy().to_string(), c));
Ok(())
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
let i = self.idx(p).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
self.files.borrow_mut().remove(i);
Ok(())
}
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
let stem = sidecar.with_extension("").to_string_lossy().to_string();
self.files.borrow().iter().map(|(f, _)| f.clone()).find(|f| {
let p = Path::new(f);
p.with_extension("").to_string_lossy() == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
})
}
}
fn file_photo(id: u64, path: &str) -> Photo {
let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
}
/// Issue #95: a case-only rename is a real change on a case-insensitive volume and never
/// replaces a different file that differs only by case on a case-sensitive one.
#[test]
fn case_only_renames_never_replace_another_file() {
// case-sensitive: IMG_1.JPG and img_1.JPG are two photos
let fs = FakeFs::new(false, &[("/p/IMG_1.JPG", "first"), ("/p/img_1.JPG", "second"), ("/p/IMG_1.xmp", "sc1"), ("/p/img_1.xmp", "sc2")]);
assert!(move_file_with(&fs, "/p/IMG_1.JPG", "/p/img_1.JPG").unwrap_err().message.contains("already exists"));
assert_eq!(fs.listing().len(), 4);
assert!(fs.listing().contains(&("/p/img_1.JPG".into(), "second".into())));
// the planner picks a free name instead
let mut s = Session::new();
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(1, "/p/IMG_1.JPG")) }).unwrap();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "img_1", 1);
assert_eq!(plans[0].to, "img_1-1.JPG");
move_file_with(&fs, "/p/IMG_1.JPG", plans[0].to_path.as_deref().unwrap()).unwrap();
assert_eq!(
fs.listing(),
vec![
("/p/img_1-1.JPG".into(), "first".into()),
("/p/img_1-1.xmp".into(), "sc1".into()),
("/p/img_1.JPG".into(), "second".into()),
("/p/img_1.xmp".into(), "sc2".into())
]
);
// case-sensitive, target free: a plain rename
let fs = FakeFs::new(false, &[("/p/IMG_2.JPG", "x")]);
move_file_with(&fs, "/p/IMG_2.JPG", "/p/img_2.JPG").unwrap();
assert_eq!(fs.listing(), vec![("/p/img_2.JPG".into(), "x".into())]);
// case-insensitive: the "existing" target is the photo itself → renamed via a temp name
let fs = FakeFs::new(true, &[("/p/IMG_1.JPG", "first"), ("/p/IMG_1.xmp", "sc1")]);
let mut s = Session::new();
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(1, "/p/IMG_1.JPG")) }).unwrap();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "img_1", 1);
assert_eq!(plans[0].to, "img_1.JPG", "not a collision with itself");
move_file_with(&fs, "/p/IMG_1.JPG", "/p/img_1.JPG").unwrap();
assert_eq!(fs.listing(), vec![("/p/img_1.JPG".into(), "first".into()), ("/p/img_1.xmp".into(), "sc1".into())]);
// case-insensitive, another file under another name: still a collision
let fs = FakeFs::new(true, &[("/p/A.JPG", "a"), ("/p/b.JPG", "b")]);
assert!(move_file_with(&fs, "/p/A.JPG", "/p/B.JPG").is_err());
}
/// Issue #105: a rollback that fails is reported, and the library follows the files that kept
/// their new names (an undoable partial rename), so nothing goes missing.
#[test]
fn failed_rollbacks_are_reported_and_followed() {
let fs = FakeFs::new(false, &[("/p/a.jpg", "A"), ("/p/a.xmp", "As"), ("/p/b.jpg", "B"), ("/p/c.jpg", "C")]);
let mut s = Session::new();
for (i, f) in ["/p/a.jpg", "/p/b.jpg", "/p/c.jpg"].iter().enumerate() {
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(i as u64 + 1, f)) }).unwrap();
}
let ids = [PhotoId(1), PhotoId(2), PhotoId(3)];
let plans = s.plan_rename_with(&fs, &ids, "Trip-{seq}", 1);
// c can't be renamed (the share went away), and neither can Trip-1 be moved back
fs.fail.borrow_mut().extend(["c.jpg".to_string(), "Trip-1".to_string()]);
let err = s.apply_rename_with(&fs, &plans).unwrap_err().to_string();
assert!(err.contains("rename /p/c.jpg"), "{err}");
assert!(err.contains("/p/Trip-1.jpg could not be moved back"), "{err}");
assert!(err.contains("/p/a.jpg → /p/Trip-1.jpg"), "lists what stayed renamed: {err}");
assert_eq!(
fs.listing(),
vec![
("/p/Trip-1.jpg".into(), "A".into()),
("/p/Trip-1.xmp".into(), "As".into()),
("/p/b.jpg".into(), "B".into()),
("/p/c.jpg".into(), "C".into())
]
);
// the catalog matches the disk
let path = |s: &Session, id: u64| match &s.catalog.photo(PhotoId(id)).unwrap().source {
Source::File { path } => path.clone(),
Source::Demo { .. } => String::new(),
};
assert_eq!(path(&s, 1), "/p/Trip-1.jpg");
assert_eq!(s.catalog.photo(PhotoId(1)).unwrap().file_name, "Trip-1.jpg");
assert_eq!((path(&s, 2), path(&s, 3)), ("/p/b.jpg".into(), "/p/c.jpg".into()));
assert_eq!(s.undo.last().map(|e| e.label.as_str()), Some("Rename 1 Photo (partly)"));
// a sidecar that won't move, and a file that won't go back: the file keeps its new name
// with the sidecars that did move, and says so
let fs = FakeFs::new(false, &[("/p/d.jpg", "D"), ("/p/d.xmp", "Ds"), ("/p/d.jpg.xmp", "Df")]);
fs.fail.borrow_mut().extend(["d.jpg.xmp".to_string(), "/p/e.jpg".to_string()]);
let e = move_file_with(&fs, "/p/d.jpg", "/p/e.jpg").unwrap_err();
assert!(e.moved, "{e}");
assert!(e.message.contains("it is now /p/e.jpg with its sidecar /p/e.xmp"), "{e}");
assert_eq!(fs.listing(), vec![("/p/d.jpg.xmp".into(), "Df".into()), ("/p/e.jpg".into(), "D".into()), ("/p/e.xmp".into(), "Ds".into())]);
// … and when the file does go back, its sidecars follow it
let fs = FakeFs::new(false, &[("/p/d.jpg", "D"), ("/p/d.xmp", "Ds"), ("/p/d.jpg.xmp", "Df")]);
fs.fail.borrow_mut().push("d.jpg.xmp".to_string());
let e = move_file_with(&fs, "/p/d.jpg", "/p/e.jpg").unwrap_err();
assert!(!e.moved);
assert_eq!(fs.listing().len(), 3);
assert!(fs.listing().iter().all(|(p, _)| p.starts_with("/p/d.")));
}
/// Issue #92 (comment): renaming one of `IMG_1.CR3` + `IMG_1.JPG` copies their shared
/// `IMG_1.xmp` instead of taking it away from the other; the last one renamed takes it.
#[test]
fn shared_stem_sidecar_is_copied_not_taken() {
let fs = FakeFs::new(false, &[("/p/IMG_1.CR3", "raw"), ("/p/IMG_1.JPG", "jpg"), ("/p/IMG_1.xmp", "shared"), ("/p/IMG_1.CR3.xmp", "own")]);
move_file_with(&fs, "/p/IMG_1.CR3", "/p/Trip_1.CR3").unwrap();
assert_eq!(
fs.listing(),
vec![
("/p/IMG_1.JPG".into(), "jpg".into()),
("/p/IMG_1.xmp".into(), "shared".into()),
("/p/Trip_1.CR3".into(), "raw".into()),
("/p/Trip_1.CR3.xmp".into(), "own".into()),
("/p/Trip_1.xmp".into(), "shared".into()),
]
);
// the JPEG is the last one using it: it moves
move_file_with(&fs, "/p/IMG_1.JPG", "/p/Trip_2.JPG").unwrap();
assert!(fs.listing().contains(&("/p/Trip_2.xmp".into(), "shared".into())));
assert!(!fs.exists(Path::new("/p/IMG_1.xmp")));
// a failure after the copy removes the copy and keeps the original
let fs = FakeFs::new(false, &[("/p/A.CR3", "raw"), ("/p/A.JPG", "jpg"), ("/p/A.xmp", "shared"), ("/p/A.CR3.xmp", "own")]);
fs.fail.borrow_mut().push("A.CR3.xmp".into());
assert!(move_file_with(&fs, "/p/A.CR3", "/p/B.CR3").is_err());
assert_eq!(fs.listing().len(), 4);
assert!(fs.listing().contains(&("/p/A.xmp".into(), "shared".into())) && !fs.exists(Path::new("/p/B.xmp")));
}
/// Issue #95 on the real disk, whichever kind of volume the temp folder is on.
#[test]
fn case_only_rename_on_disk() {
let dir = std::env::temp_dir().join(format!("lc-rename-case-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let (upper, lower) = (dir.join("IMG_1.JPG"), dir.join("img_1.JPG"));
std::fs::write(&upper, b"first").unwrap();
let case_sensitive = !lower.exists();
let p = |q: &Path| q.to_string_lossy().to_string();
if case_sensitive {
std::fs::write(&lower, b"second").unwrap();
assert!(move_file(&p(&upper), &p(&lower)).is_err());
assert_eq!(std::fs::read(&lower).unwrap(), b"second");
assert_eq!(std::fs::read(&upper).unwrap(), b"first");
} else {
move_file(&p(&upper), &p(&lower)).unwrap();
let names: Vec<String> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
assert_eq!(names, vec!["img_1.JPG".to_string()]);
assert_eq!(std::fs::read(&lower).unwrap(), b"first");
}
// a plain rename never replaces
std::fs::write(dir.join("other.JPG"), b"other").unwrap();
assert!(move_file(&p(&dir.join("other.JPG")), &p(&lower)).is_err());
assert_eq!(std::fs::read(dir.join("other.JPG")).unwrap(), b"other");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn folder_templates_stay_inside_the_destination() {
+162 -13
View File
@@ -5,12 +5,25 @@
//! creator, keywords, capture time and GPS in the standard namespaces, plus our complete develop
//! settings (`lc:settings`, JSON), the pick/reject flag (`lc:flag`) and location (`lc:location`).
//!
//! An existing sidecar is never replaced wholesale (issue #92): LightCraft's properties
//! ([`OWNED`], [`OWNED_IF_STATED`]) are merged into it ([`lightcraft_meta::merge_xmp`]) and
//! everything else — another editor's `crs:` develop settings, `xmpMM` history, unknown
//! namespaces — is kept byte for byte. A sidecar that can't be read as XMP is first copied to
//! `<name>.xmp.bak-<time>` (never overwritten), then replaced. Writes are atomic (temp file,
//! fsync, rename).
//!
//! **Shared names** (Stem naming): `IMG_0001.CR3` and `IMG_0001.JPG` would share `IMG_0001.xmp`.
//! When two catalogued files map to one stem sidecar, it belongs to one of them (a raw first, then
//! the first by file name — [`Session::sidecar_naming`]); the others use `<file>.xmp`
//! (`IMG_0001.JPG.xmp`) for writing and reading, so neither overwrites the other.
//!
//! **Read** (on import, and `photo.readMetadataFromFile`): the sidecar wins for metadata, except
//! the capture time: a time embedded in the file wins, and the sidecar's (`exif:DateTimeOriginal`,
//! else `photoshop:DateCreated`, else `xmp:CreateDate`) fills in only when the file has none; develop
//! settings are restored from `lc:settings` when present, else mapped from interoperable `crs:`
//! fields ([`crate::crs`]). For raw/DNG files without a sidecar the file's embedded XMP is used.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -258,13 +271,99 @@ pub fn sidecar_packet(p: &Photo, cat: &lightcraft_catalog::Catalog) -> String {
lightcraft_meta::write_xmp_lc(&meta, &lc)
}
/// Write `data` to `path` atomically (temp file + rename).
/// The properties LightCraft writes and owns in a sidecar: replaced on every save (removed when
/// LightCraft has no value, so clearing a field clears it in the file too). All of them are read
/// back on import and by Read Metadata from File.
pub const OWNED: &[&str] = &[
"xmp:Rating",
"xmp:Label",
"dc:title",
"dc:description",
"dc:rights",
"dc:creator",
"dc:subject",
"Iptc4xmpCore:Location",
"Iptc4xmpCore:AltTextAccessibility",
"Iptc4xmpCore:ExtDescrAccessibility",
"photoshop:City",
"photoshop:State",
"photoshop:Country",
"xmpRights:Marked",
"xmpRights:UsageTerms",
"xmpRights:WebStatement",
"lc:*",
];
/// Properties replaced only when LightCraft has a value (capture time; GPS, which LightCraft
/// doesn't read from sidecars): otherwise the sidecar's own value stays. Anything else LightCraft
/// writes (`xmp:CreatorTool`) goes only into new sidecars.
pub const OWNED_IF_STATED: &[&str] = &["exif:DateTimeOriginal", "photoshop:DateCreated", "exif:GPSLatitude", "exif:GPSLongitude"];
/// Write `data` to `path` atomically: a temp file next to it, synced, then renamed over it.
fn write_atomic(path: &Path, data: &[u8]) -> std::io::Result<()> {
let tmp = path.with_extension(format!("xmp.tmp-{}", std::process::id()));
std::fs::write(&tmp, data)?;
std::fs::rename(&tmp, path).inspect_err(|_| {
let _ = std::fs::remove_file(&tmp);
})
lightcraft_catalog::safe_file::write_atomic(path, data)
}
/// Copy `path` to a new `<name>.bak-<stamp>` (`-1`, `-2`… if taken; never overwriting).
fn backup(path: &Path, stamp: &str) -> std::io::Result<PathBuf> {
use std::io::Write;
let bytes = std::fs::read(path)?;
let stamp: String = stamp.chars().filter(char::is_ascii_alphanumeric).collect();
let name = path.file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
for k in 0..1000 {
let b = path.with_file_name(if k == 0 { format!("{name}.bak-{stamp}") } else { format!("{name}.bak-{stamp}-{k}") });
match std::fs::OpenOptions::new().write(true).create_new(true).open(&b) {
Ok(mut f) => {
f.write_all(&bytes)?;
f.sync_all()?;
return Ok(b);
}
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(std::io::Error::other("no free backup name"))
}
/// What [`Session::save_sidecar`] did.
#[derive(Clone, Debug, PartialEq)]
pub struct SidecarSaved {
pub path: PathBuf,
/// An existing sidecar was merged into (its other contents kept).
pub merged: bool,
/// An existing sidecar that couldn't be read as XMP was copied here before being replaced.
pub backup: Option<PathBuf>,
}
/// The new contents for the sidecar at `path`: `fresh` merged into the existing file, if any.
/// Returns (contents, merged, backup).
fn sidecar_contents(path: &Path, fresh: String, stamp: &str) -> std::io::Result<(String, bool, Option<PathBuf>)> {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((fresh, false, None)),
// can't see what is there: never overwrite it blindly
Err(e) => return Err(e),
};
if bytes.iter().all(u8::is_ascii_whitespace) {
return Ok((fresh, false, None));
}
let rules = lightcraft_meta::MergeRules { owned: OWNED, owned_if_present: OWNED_IF_STATED };
match std::str::from_utf8(&bytes)
.map_err(|e| e.to_string())
.and_then(|old| lightcraft_meta::merge_xmp(old, &fresh, rules).map_err(|e| e.to_string()))
{
Ok(merged) => Ok((merged, true, None)),
Err(why) => {
let b = backup(path, stamp)?;
log::warn!("{}: not readable as XMP ({why}); kept a copy as {} before replacing it", path.display(), b.display());
Ok((fresh, false, Some(b)))
}
}
}
/// Key of the stem sidecar a file maps to (case-insensitive).
fn stem_key(original: &str) -> String {
sidecar_path(original, SidecarNaming::Stem).to_string_lossy().to_lowercase()
}
fn file_path(p: &Photo) -> Option<&str> {
@@ -302,24 +401,70 @@ pub(crate) fn op_photos(op: &Op, out: &mut Vec<PhotoId>) {
}
}
/// Which photo owns each stem sidecar that several catalogued files map to (see
/// [`Session::sidecar_naming`]).
pub(crate) struct StemOwners(HashMap<String, PhotoId>);
impl StemOwners {
pub(crate) fn of(cat: &lightcraft_catalog::Catalog) -> StemOwners {
// per stem: (raw first, then file name, then id) of the best candidate; and how many
let mut best: HashMap<String, ((bool, String, u64), PhotoId, usize)> = HashMap::new();
for p in cat.photos().filter(|p| p.copy_of.is_none()) {
let Some(path) = file_path(p) else { continue };
let rank = (p.kind != MediaKind::Raw, path.to_lowercase(), p.id.0);
let e = best.entry(stem_key(path)).or_insert_with(|| (rank.clone(), p.id, 0));
e.2 += 1;
if rank < e.0 {
(e.0, e.1) = (rank, p.id);
}
}
StemOwners(best.into_iter().filter(|(_, v)| v.2 > 1).map(|(k, v)| (k, v.1)).collect())
}
/// `naming` for photo `p`, except that a stem sidecar another file owns becomes Full.
fn naming(&self, p: &Photo, naming: SidecarNaming) -> SidecarNaming {
match (naming, file_path(p)) {
(SidecarNaming::Stem, Some(path)) if self.0.get(&stem_key(path)).is_some_and(|owner| *owner != p.id) => SidecarNaming::Full,
_ => naming,
}
}
}
impl Session {
/// Write the photo's XMP sidecar; returns its path.
pub fn save_sidecar(&self, id: PhotoId) -> Result<PathBuf> {
/// The sidecar naming for photo `id`: the library's, except under Stem naming when another
/// catalogued file shares the stem (`IMG_0001.CR3` + `IMG_0001.JPG`) and owns the stem
/// sidecar — a raw first, else the first by file name. The others use Full naming
/// (`IMG_0001.JPG.xmp`), so their metadata never overwrites each other.
pub fn sidecar_naming(&self, id: PhotoId) -> SidecarNaming {
match self.catalog.photo(id) {
Some(p) if self.xmp.naming == SidecarNaming::Stem => StemOwners::of(&self.catalog).naming(p, self.xmp.naming),
_ => self.xmp.naming,
}
}
/// Write the photo's XMP sidecar, merged into an existing one (see the module docs).
pub fn save_sidecar(&self, id: PhotoId) -> Result<SidecarSaved> {
self.save_sidecar_with(id, &StemOwners::of(&self.catalog))
}
pub(crate) fn save_sidecar_with(&self, id: PhotoId, owners: &StemOwners) -> Result<SidecarSaved> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
if p.copy_of.is_some() {
return Err(EngineError::Other(format!("{} is a virtual copy: its settings live only in the library", p.file_name)));
}
let orig = file_path(p).ok_or_else(|| EngineError::Other(format!("{} is not a file on disk", p.file_name)))?;
let path = sidecar_path(orig, self.xmp.naming);
write_atomic(&path, sidecar_packet(p, &self.catalog).as_bytes()).map_err(|e| EngineError::Other(format!("{}: {e}", path.display())))?;
Ok(path)
let path = sidecar_path(orig, owners.naming(p, self.xmp.naming));
let err = |e: std::io::Error| EngineError::Other(format!("{}: {e}", path.display()));
let (data, merged, backup) = sidecar_contents(&path, sidecar_packet(p, &self.catalog), &(self.clock)()).map_err(err)?;
write_atomic(&path, data.as_bytes()).map_err(err)?;
Ok(SidecarSaved { path, merged, backup })
}
/// The op that applies a photo's sidecar (or embedded XMP) to the catalog, if there is one.
pub fn read_sidecar_op(&self, id: PhotoId) -> Result<Option<(Op, PathBuf)>> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
let Some(orig) = file_path(p) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.xmp.naming) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.sidecar_naming(id)) else { return Ok(None) };
let sc = parse_sidecar(&packet, p.kind == MediaKind::Raw)
.map_err(|e| EngineError::Other(format!("{}: {e}", from.display())))?
.resolve_label(&self.catalog);
@@ -344,9 +489,13 @@ impl Session {
pub(crate) fn auto_write_sidecars(&self, ops: &[Op]) {
let mut ids = Vec::new();
ops.iter().for_each(|o| op_photos(o, &mut ids));
if ids.is_empty() {
return;
}
let owners = StemOwners::of(&self.catalog);
for id in ids {
if self.catalog.photo(id).is_some_and(|p| file_path(p).is_some() && p.copy_of.is_none())
&& let Err(e) = self.save_sidecar(id)
&& let Err(e) = self.save_sidecar_with(id, &owners)
{
log::warn!("auto-write XMP: {e}");
}
+52 -3
View File
@@ -130,11 +130,16 @@ pub fn migrate(from: &Path, to: &Path, what: Existing) -> (usize, Vec<String>) {
Existing::Discard => std::fs::remove_file(&src),
_ => {
let dst = to.join(e.file_name());
if dst.exists() {
// named by content: the copy already there is the same preview
if is_valid(&dst) {
// named by content: the complete copy already there is the same preview
std::fs::remove_file(&src)
} else {
std::fs::rename(&src, &dst).or_else(|_| std::fs::copy(&src, &dst).and_then(|_| std::fs::remove_file(&src)))
// across drives: copied atomically (never a partial proxy at the new place)
std::fs::rename(&src, &dst).or_else(|_| {
std::fs::read(&src)
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&dst, &b))
.and_then(|()| std::fs::remove_file(&src))
})
}
}
};
@@ -146,6 +151,29 @@ pub fn migrate(from: &Path, to: &Path, what: Existing) -> (usize, Vec<String>) {
(done, failed)
}
/// A cheap check that the smart preview at `path` is complete: our header, then a JPEG from its
/// start marker to its end marker. A file cut short by a crash, a full drive or an unplugged one
/// fails it (and is then rebuilt by Build Smart Previews, not counted as there).
pub fn is_valid(path: &Path) -> bool {
use std::io::{Read, Seek, SeekFrom};
let Ok(mut f) = std::fs::File::open(path) else { return false };
let mut head = Vec::with_capacity(256);
if (&mut f).take(256).read_to_end(&mut head).is_err() {
return false;
}
let Some(rest) = head.strip_prefix(MAGIC) else { return false };
let Some(nl) = rest.iter().position(|b| *b == b'\n') else { return false };
let size_ok = rest
.get(..nl)
.and_then(|h| serde_json::from_slice::<serde_json::Value>(h).ok())
.is_some_and(|h| h["w"].as_u64().is_some_and(|w| w > 0) && h["h"].as_u64().is_some_and(|h| h > 0));
if !size_ok || rest.get(nl + 1..nl + 3) != Some(&[0xFF, 0xD8][..]) {
return false;
}
let mut end = [0u8; 2];
f.seek(SeekFrom::End(-2)).is_ok() && f.read_exact(&mut end).is_ok() && end == [0xFF, 0xD9]
}
/// Load the proxy at `path`.
pub fn load(path: &Path) -> Result<Arc<Rgb32f>, String> {
let b = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?;
@@ -193,6 +221,27 @@ mod tests {
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn damaged_proxies_are_invalid_and_never_kept_over_good_ones() {
let base = temp("valid");
let (a, b) = (base.join("a"), base.join("b"));
std::fs::create_dir_all(&a).unwrap();
std::fs::create_dir_all(&b).unwrap();
let img = lightcraft_scenes::demo_library()[0].render(64, 40);
let good = encode(&img).unwrap();
std::fs::write(a.join("1.lcsp"), &good).unwrap();
assert!(is_valid(&a.join("1.lcsp")));
for cut in [good.len() - 1, good.len() / 2, 10, 0] {
std::fs::write(b.join("1.lcsp"), &good[..cut]).unwrap();
assert!(!is_valid(&b.join("1.lcsp")), "cut at {cut}");
}
assert!(!is_valid(&b.join("missing.lcsp")));
// moving onto a damaged copy replaces it instead of deleting the good one
assert_eq!(migrate(&a, &b, Existing::Move).0, 1);
assert_eq!(std::fs::read(b.join("1.lcsp")).unwrap(), good);
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn roundtrip_keeps_the_picture() {
let img = lightcraft_scenes::demo_library()[0].render(160, 100);
+149
View File
@@ -174,3 +174,152 @@ fn prepared_exports_run_on_another_thread() {
assert_eq!(files[0]["width"], 48);
assert_eq!(seen.lock().unwrap().len(), 2);
}
/// Issue #78: a GPU render whose work never ran (a driver that drops a submission without an
/// error) gave an all-black export. The export must come from the CPU instead — the same
/// image — with the reason recorded.
#[test]
fn export_falls_back_to_the_cpu_when_gpu_work_is_lost() {
let mut s = Session::with_demo();
let id = s.active().unwrap();
s.execute("develop.set", &json!({"control": "light.exposure", "value": 0.4})).unwrap();
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 20})).unwrap();
let o = ExportOptions::from_json(&json!({"format": "png"}));
let mean = |b: &[u8]| {
let d = lightcraft_codecs::decode(b, Default::default()).unwrap().image;
d.data.iter().map(|p| (p[0] + p[1] + p[2]) as f64).sum::<f64>() / (3 * d.data.len()) as f64
};
let gpu = lightcraft_gpu::available();
let healthy = mean(&export_photo(&mut s, id, &o, 1).unwrap().bytes);
lightcraft_gpu::inject_fault(lightcraft_gpu::Fault::DropWork);
let faulted = export_photo(&mut s, id, &o, 1).unwrap().bytes;
if gpu {
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("incomplete"), "{why}");
// the GPU is off for the process now: this export renders on the CPU
assert!(!lightcraft_gpu::available());
}
let cpu = export_photo(&mut s, id, &o, 1).unwrap().bytes;
lightcraft_gpu::reset_failures();
let px = |b: &[u8]| lightcraft_codecs::decode(b, Default::default()).unwrap().image.data;
assert!(px(&faulted) == px(&cpu), "the fallback is the CPU render");
let m = mean(&faulted);
assert!(m > 0.02, "not black: mean {m}");
assert!((m - healthy).abs() < 0.01, "GPU {healthy} vs CPU {m}");
}
/// A JPEG original `dir/IMG_1.jpg` imported into a library on disk.
fn library_with_jpeg(tag: &str) -> (Session, lightcraft_catalog::PhotoId, std::path::PathBuf, std::path::PathBuf, Vec<u8>) {
let dir = temp_dir(tag);
let src = dir.join("IMG_1.jpg");
let img = lightcraft_raster::Rgba8::from_fn(48, 32, |x, y| [(x * 5) as u8, (y * 7) as u8, 120, 255]);
let bytes = crate::export::encode_image(&img, &ExportOptions::default()).unwrap();
std::fs::write(&src, &bytes).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = s.catalog.photos().next().unwrap().id;
s.execute("develop.set", &json!({"ids": [id.0], "control": "light.exposure", "value": 1.0})).unwrap();
(s, id, dir, src, bytes)
}
fn disk_batch(
s: &mut Session,
id: lightcraft_catalog::PhotoId,
o: &ExportOptions,
to: &crate::export::Destination,
) -> Result<Vec<serde_json::Value>, String> {
crate::export::export_batch(s, &[id], o, to, &mut crate::export::write_file, &|p| std::path::Path::new(p).exists())
}
/// Issue #93: exporting into the photo's own folder as `{name}` with "Overwrite" replaced the
/// original with the re-encoded render. It is refused now, whatever the policy or exact path.
#[test]
fn export_never_overwrites_an_original() {
use crate::export::{Conflict, Destination};
let (mut s, id, dir, src, original) = library_with_jpeg("export-guard");
let folder = Destination { dir: dir.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { naming: "{name}".into(), conflict: Conflict::Overwrite, ..Default::default() };
let err = disk_batch(&mut s, id, &o, &folder).unwrap_err();
assert!(err.contains("original of IMG_1.jpg"), "{err}");
assert_eq!(std::fs::read(&src).unwrap(), original, "the original is byte-identical");
// the exact path of the control channel / MCP, also spelled another way
std::fs::create_dir_all(dir.join("sub")).unwrap();
for exact in [src.to_string_lossy().to_string(), dir.join("sub/../IMG_1.jpg").to_string_lossy().to_string()] {
let to = Destination { dir: String::new(), exact: Some(exact) };
assert!(disk_batch(&mut s, id, &o, &to).unwrap_err().contains("never writes over an original"));
}
// the "Original" format re-writing the file onto itself (and its sidecar)
let orig = ExportOptions { format: ExportFormat::Original, ..o.clone() };
assert!(disk_batch(&mut s, id, &orig, &folder).is_err());
assert_eq!(std::fs::read(&src).unwrap(), original);
assert!(s.execute("export.checkTarget", &json!({"path": src.to_string_lossy()})).is_err());
assert!(s.execute("export.checkTarget", &json!({"path": dir.join("free.jpg").to_string_lossy()})).is_ok());
// Unique moves past it; Overwrite still replaces an ordinary earlier export
let unique = ExportOptions { conflict: Conflict::Unique, ..o.clone() };
let files = disk_batch(&mut s, id, &unique, &folder).unwrap();
assert!(files[0]["path"].as_str().unwrap().ends_with("IMG_1-2.jpg"), "{files:?}");
let earlier = dir.join("IMG_1-2.jpg");
std::fs::write(&earlier, b"an earlier export").unwrap();
let named = ExportOptions { naming: "{name}-2".into(), ..o };
disk_batch(&mut s, id, &named, &folder).unwrap();
assert_ne!(std::fs::read(&earlier).unwrap(), b"an earlier export", "an ordinary file is overwritten");
assert_eq!(std::fs::read(&src).unwrap(), original);
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #93: the XMP sidecar of an "Original" export ignored the conflict policy.
#[test]
fn export_sidecars_follow_the_conflict_policy() {
use crate::export::{Conflict, Destination};
let (mut s, id, dir, _src, original) = library_with_jpeg("export-sidecar");
let out = dir.join("out");
std::fs::create_dir_all(&out).unwrap();
let theirs = out.join("IMG_1.xmp");
std::fs::write(&theirs, b"someone else's sidecar").unwrap();
let to = Destination { dir: out.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { format: ExportFormat::Original, naming: "{name}".into(), ..Default::default() };
// Unique: the photo and its sidecar both move to the next free name
let files = disk_batch(&mut s, id, &ExportOptions { conflict: Conflict::Unique, ..o.clone() }, &to).unwrap();
assert!(files[0]["path"].as_str().unwrap().ends_with("IMG_1-2.jpg"), "{files:?}");
assert!(files[0]["sidecars"][0].as_str().unwrap().ends_with("IMG_1-2.xmp"), "{files:?}");
assert_eq!(std::fs::read(&theirs).unwrap(), b"someone else's sidecar");
assert_eq!(std::fs::read(out.join("IMG_1-2.jpg")).unwrap(), original);
// Skip: a taken sidecar name skips the photo
std::fs::remove_file(out.join("IMG_1-2.jpg")).unwrap();
std::fs::remove_file(out.join("IMG_1-2.xmp")).unwrap();
let files = disk_batch(&mut s, id, &ExportOptions { conflict: Conflict::Skip, ..o.clone() }, &to).unwrap();
assert!(files[0]["skipped"].as_str().is_some(), "{files:?}");
assert!(!out.join("IMG_1.jpg").exists());
assert_eq!(std::fs::read(&theirs).unwrap(), b"someone else's sidecar");
// Overwrite: replaced, as asked
disk_batch(&mut s, id, &ExportOptions { conflict: Conflict::Overwrite, ..o }, &to).unwrap();
assert!(std::fs::read_to_string(&theirs).unwrap().contains("xmpmeta"));
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #93: the native writer truncated the target first; a failure mid-write (full disk,
/// unplugged drive) now leaves the previous file intact and no partial one.
#[test]
fn a_failed_export_write_keeps_the_previous_file() {
use crate::export::{Conflict, Destination};
let (mut s, id, dir, _src, _) = library_with_jpeg("export-midwrite");
let out = dir.join("out");
std::fs::create_dir_all(&out).unwrap();
let prev = out.join("IMG_1.jpg");
std::fs::write(&prev, b"last week's export").unwrap();
let to = Destination { dir: out.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { conflict: Conflict::Overwrite, ..Default::default() };
{
let _fault = lightcraft_catalog::safe_file::fail_writes_after(100);
let err = disk_batch(&mut s, id, &o, &to).unwrap_err();
assert!(err.contains("injected"), "{err}");
}
assert_eq!(std::fs::read(&prev).unwrap(), b"last week's export");
assert_eq!(std::fs::read_dir(&out).unwrap().count(), 1, "no temp file left behind");
disk_batch(&mut s, id, &o, &to).unwrap();
assert_ne!(std::fs::read(&prev).unwrap(), b"last week's export");
let _ = std::fs::remove_dir_all(&dir);
}
+218
View File
@@ -0,0 +1,218 @@
//! Forgetting untouched Local browse records: by command and when the library opens; changed,
//! recent and visible records stay; a crash replays it; browsing again brings records back.
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use serde_json::json;
use crate::{LibrarySource, Selection, Session};
fn temp_dir(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-forget-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
fn png(p: &Path, seed: u8) {
let img = lightcraft_raster::Rgba8::from_fn(16, 12, |x, y| [(x * 9) as u8, (y * 11) as u8, seed, 255]);
let b =
crate::export::encode_image(&img, &crate::export::ExportOptions { format: crate::export::ExportFormat::Png, ..Default::default() }).unwrap();
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, b).unwrap();
}
fn open(lib: &Path, clock: &Arc<Mutex<String>>) -> Session {
let mut s = Session::new().with_fs();
let c = clock.clone();
s.clock = Box::new(move || c.lock().unwrap().clone());
s.open_library(lib, false).unwrap();
s
}
fn set(clock: &Arc<Mutex<String>>, t: &str) {
*clock.lock().unwrap() = t.to_string();
}
fn local_paths(s: &Session) -> Vec<String> {
let mut v: Vec<String> = s
.catalog
.photos()
.filter(|p| p.local)
.filter_map(|p| match &p.source {
lightcraft_catalog::Source::File { path } => Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()),
_ => None,
})
.collect();
v.sort();
v
}
/// Leave the folder view (nothing browsed is visible or selected any more).
fn look_elsewhere(s: &mut Session) {
s.execute("library.source", &json!({"kind": "all"})).unwrap();
s.selection = Selection::default();
s.previous_active = None;
s.undo.clear();
}
#[test]
fn untouched_records_of_unbrowsed_folders_are_forgotten_and_come_back() {
let root = temp_dir("cmd");
let (a, b, lib) = (root.join("a"), root.join("b"), root.join("lib"));
for (i, n) in ["a1.png", "a2.png", "a3.png"].iter().enumerate() {
png(&a.join(n), i as u8);
}
png(&b.join("b1.png"), 7);
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
// the user rates a2 (a change: kept)
let a2 = s.catalog.photos().find(|p| p.file_name == "a2.png").unwrap().id;
s.selection = Selection::single(a2);
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
set(&clock, "2026-08-25T10:00:00");
s.execute("library.browse", &json!({"path": b.to_string_lossy()})).unwrap();
look_elsewhere(&mut s);
// 40 days after a was browsed, 16 after b
set(&clock, "2026-09-10T10:00:00");
let dry = s.execute("library.forgetLocal", &json!({"dryRun": true})).unwrap();
assert_eq!(
(dry["local"].as_u64(), dry["forgotten"].as_u64(), dry["keptTouched"].as_u64(), dry["keptRecent"].as_u64()),
(Some(4), Some(2), Some(1), Some(1)),
"{dry}"
);
assert_eq!(local_paths(&s).len(), 4, "a dry run changes nothing");
let r = s.execute("library.forgetLocal", &json!({})).unwrap();
assert_eq!(r["forgotten"], 2, "{r}");
assert_eq!(local_paths(&s), vec!["a2.png", "b1.png"]);
// files on disk are untouched
assert!(["a1.png", "a2.png", "a3.png"].iter().all(|n| a.join(n).exists()));
// not an undo step
assert!(s.undo.is_empty());
// a crash now: the log replays the same catalog
let expect = s.catalog.to_snapshot();
drop(s);
let mut s = open(&lib, &clock);
assert_eq!(s.library.as_ref().unwrap().report.failed, 0);
assert_eq!(s.catalog.to_snapshot(), expect, "nothing more to forget on open, same state after replay");
// browsing a again brings the forgotten files back, fresh; the rated one is the same record
let r = s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
assert_eq!((r["photos"].as_u64(), r["new"].as_u64()), (Some(3), Some(2)), "{r}");
assert_eq!(s.catalog.photo(a2).unwrap().rating, 3);
// while a is shown, nothing in it goes, whatever the time
set(&clock, "2027-09-10T10:00:00");
assert_eq!(s.execute("library.forgetLocal", &json!({"dryRun": true})).unwrap()["forgotten"], 1, "only b1 (not visible)");
let _ = std::fs::remove_dir_all(&root);
}
#[test]
fn opening_the_library_forgets_per_the_preference() {
let root = temp_dir("open");
let (a, lib) = (root.join("a"), root.join("lib"));
png(&a.join("x.png"), 1);
png(&a.join("y.png"), 2);
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
look_elsewhere(&mut s);
// 0 = never
let r = s.execute("library.preferences", &json!({"forgetLocalDays": 0})).unwrap();
assert_eq!(r["forgetLocalDays"], 0);
s.close_library().unwrap();
drop(s);
set(&clock, "2027-01-01T10:00:00");
let mut s = open(&lib, &clock);
assert_eq!(local_paths(&s).len(), 2, "never");
assert!(s.library.as_ref().unwrap().forgot_local.is_none());
// back to 30 days: the next open forgets them
s.execute("library.preferences", &json!({"forgetLocalDays": 30})).unwrap();
s.source = LibrarySource::All;
s.close_library().unwrap();
drop(s);
let s = open(&lib, &clock);
assert!(local_paths(&s).is_empty());
let info = {
let mut s = s;
s.execute("library.info", &json!({})).unwrap()
};
assert_eq!(info["forgotLocal"]["forgotten"], 2, "{info}");
assert!(a.join("x.png").exists() && a.join("y.png").exists());
let _ = std::fs::remove_dir_all(&root);
}
/// A file with an XMP sidecar comes in with the sidecar's rating and keywords: that is its
/// browse state, not a user change, so the record is forgotten like any other — and the sidecar
/// stays byte for byte, so browsing again reads the same values back.
#[test]
fn sidecar_values_are_not_user_changes_and_sidecars_stay() {
let root = temp_dir("xmp");
let (a, lib) = (root.join("a"), root.join("lib"));
png(&a.join("x.png"), 1);
let x = a.join("x.png").to_string_lossy().to_string();
let mut p = lightcraft_catalog::Photo::new(
lightcraft_catalog::PhotoId(1),
lightcraft_catalog::Source::File { path: x.clone() },
"x.png",
"PNG",
16,
12,
"2026-01-01T00:00:00",
);
p.rating = 4;
p.meta.keywords = vec!["dunes".into()];
let side = crate::sidecar::sidecar_path(&x, Default::default());
std::fs::write(&side, crate::sidecar::sidecar_packet(&p, &lightcraft_catalog::Catalog::new())).unwrap();
let packet = std::fs::read(&side).unwrap();
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
let rec = s.catalog.photos().find(|p| p.local).unwrap().clone();
assert_eq!((rec.rating, rec.meta.keywords.clone()), (4, vec!["dunes".to_string()]), "read from the sidecar");
look_elsewhere(&mut s);
set(&clock, "2026-10-01T10:00:00");
assert_eq!(s.execute("library.forgetLocal", &json!({})).unwrap()["forgotten"], 1);
assert_eq!(std::fs::read(&side).unwrap(), packet, "the sidecar is untouched");
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
let back = s.catalog.photos().find(|p| p.local).unwrap();
assert_ne!(back.id, rec.id, "a fresh record");
assert_eq!((back.rating, back.meta.keywords.clone()), (4, vec!["dunes".to_string()]));
let _ = std::fs::remove_dir_all(&root);
}
/// A catalog written before browse times existed: its Local records are kept on the first open
/// (their folders are stamped then), and forgotten only once the period has passed since.
#[test]
fn upgraded_catalogs_forget_nothing_at_first() {
let root = temp_dir("upgrade");
let (a, lib) = (root.join("a"), root.join("lib"));
png(&a.join("x.png"), 1);
let clock = Arc::new(Mutex::new("2026-08-01T10:00:00".to_string()));
let mut s = open(&lib, &clock);
s.execute("library.browse", &json!({"path": a.to_string_lossy()})).unwrap();
look_elsewhere(&mut s);
s.close_library().unwrap();
drop(s);
// strip the times and baselines, as an older version would have written the snapshot
let snap = std::fs::read_to_string(lib.join("catalog.snap")).unwrap();
let mut v: serde_json::Value = serde_json::from_str(&snap).unwrap();
v["catalog"].as_object_mut().unwrap().remove("browsed");
for p in v["catalog"]["photos"].as_object_mut().unwrap().values_mut() {
p.as_object_mut().unwrap().remove("local_baseline");
}
std::fs::write(lib.join("catalog.snap"), serde_json::to_vec(&v).unwrap()).unwrap();
set(&clock, "2027-01-01T10:00:00");
let s = open(&lib, &clock);
assert_eq!(local_paths(&s).len(), 1, "kept on the first open after the upgrade");
assert_eq!(s.catalog.last_browsed(&a.to_string_lossy()), Some("2027-01-01T10:00:00"));
drop(s);
set(&clock, "2027-02-15T10:00:00");
let s = open(&lib, &clock);
assert!(local_paths(&s).is_empty(), "the untouched legacy record goes once 30 days passed");
let _ = std::fs::remove_dir_all(&root);
}
+142
View File
@@ -144,6 +144,45 @@ fn browsing_a_folder_lists_its_photos_without_adding_them() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #105: Find Missing finds renamed files by content, prefers the content match over a
/// same-name same-size impostor, and skips (reports) photos it can't tell apart.
#[test]
fn find_missing_matches_renamed_files_by_content() {
let dir = temp_dir("missing-content");
std::fs::create_dir_all(dir.join("old")).unwrap();
std::fs::create_dir_all(dir.join("moved/real")).unwrap();
for (n, seed) in [("a.png", 1), ("b.png", 2), ("c.png", 3)] {
write_png(&dir.join("old").join(n), seed);
}
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [dir.join("old").to_string_lossy()]})).unwrap();
// an impostor: same name and size, other bytes
let impostor = |src: &Path, dst: &Path| {
let mut b = std::fs::read(src).unwrap();
let k = b.len() - 20;
b[k] ^= 0xff;
std::fs::write(dst, b).unwrap();
};
// a: renamed; b: renamed, with an impostor under its old name; c: gone, two impostors
std::fs::rename(dir.join("old/a.png"), dir.join("moved/Trip-001.png")).unwrap();
impostor(&dir.join("old/b.png"), &dir.join("moved/b.png"));
std::fs::rename(dir.join("old/b.png"), dir.join("moved/real/Trip-002.png")).unwrap();
impostor(&dir.join("old/c.png"), &dir.join("moved/c.png"));
impostor(&dir.join("old/c.png"), &dir.join("moved/real/c.png"));
std::fs::remove_file(dir.join("old/c.png")).unwrap();
let r = s.execute("library.findMissing", &json!({"folder": dir.join("moved").to_string_lossy()})).unwrap();
let found = r["found"].as_array().unwrap();
assert_eq!(found.len(), 2, "{r}");
let to = |name: &str| {
found.iter().find(|f| f["from"].as_str().unwrap().ends_with(name)).map(|f| (f["to"].as_str().unwrap().to_string(), f["by"].clone()))
};
assert_eq!(to("a.png"), Some((dir.join("moved/Trip-001.png").to_string_lossy().to_string(), json!("content"))));
assert_eq!(to("b.png"), Some((dir.join("moved/real/Trip-002.png").to_string_lossy().to_string(), json!("content"))), "not the impostor");
assert_eq!(r["missing"], 1);
assert_eq!(r["ambiguous"][0]["candidates"].as_array().map(Vec::len), Some(2), "{r}");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn missing_files_are_found_and_relinked() {
let dir = std::env::temp_dir().join(format!("lc-missing-{}", std::process::id()));
@@ -352,6 +391,44 @@ fn smart_previews_stand_in_for_offline_originals() {
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #106: a smart preview cut short (crash, full drive) counted as built forever and failed
/// exactly when the original went offline. Writes are atomic now, and a damaged proxy is not
/// reported as present and is rebuilt.
#[test]
fn damaged_smart_previews_are_rebuilt_and_failed_writes_leave_none() {
let src = temp_dir("smartdmg-src");
let lib = temp_dir("smartdmg-lib");
write_png(&src.join("a.png"), 5);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("a.png").to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
let dir = s.media.smart_dir.clone().unwrap();
let file = dir.join(crate::smart::file_name(s.catalog.photo(id).unwrap()));
{
let _fault = lightcraft_catalog::safe_file::fail_writes_after(64);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["failed"].as_array().map(Vec::len)), (Some(0), Some(1)), "{r}");
}
assert!(!file.exists(), "no partial proxy");
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 0, "no temp file left");
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], false);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(0)), "{r}");
assert!(crate::smart::is_valid(&file));
// cut short, as by a crash before the fix
let full = std::fs::read(&file).unwrap();
std::fs::write(&file, &full[..full.len() / 2]).unwrap();
assert!(!crate::smart::is_valid(&file));
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], false, "not counted as there");
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// LUT profiles: a .cube imports as a creative profile (into the library), renders differently
/// (Amount 0 = no change), is listed in the profile browser and comes back with the library.
#[test]
@@ -441,6 +518,70 @@ fn folders_rename_and_move_with_their_photos() {
let _ = std::fs::remove_dir_all(&root);
}
/// Rename / Move Folder are undo steps (issue #97): undo renames the folder back on disk (files and
/// sidecars along) and restores the photos' paths, redo repeats it; a destination taken in the
/// meantime is never overwritten — the step is refused, reported and stays on the stack.
#[test]
fn folder_rename_and_move_undo_and_redo_on_disk() {
let root = temp_dir("folders-undo");
let a = root.join("Trip");
write_png(&a.join("one.png"), 3);
std::fs::write(a.join("one.xmp"), "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>").unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [a.to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
let path = |s: &Session| match &s.catalog.photo(id).unwrap().source {
lightcraft_catalog::Source::File { path } => path.clone(),
_ => panic!(),
};
let undo0 = s.undo.len();
let b = root.join("Italy");
s.execute("folder.rename", &json!({"path": a.to_string_lossy(), "name": "Italy"})).unwrap();
assert_eq!(s.undo.len(), undo0 + 1, "one undo step");
assert!(s.undo.last().unwrap().label.contains("Rename Folder"));
assert!(b.join("one.png").exists() && !a.exists());
s.execute("edit.undo", &json!({})).unwrap();
assert!(a.join("one.png").exists() && a.join("one.xmp").exists() && !b.exists(), "renamed back on disk");
assert_eq!(path(&s), a.join("one.png").to_string_lossy());
s.media.forget(id);
assert!(s.render_now(id, 16, 16).is_ok());
s.execute("edit.redo", &json!({})).unwrap();
assert!(b.join("one.png").exists() && b.join("one.xmp").exists() && !a.exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
// the old name is taken now: undo is refused, nothing is overwritten, the step stays
std::fs::create_dir_all(&a).unwrap();
std::fs::write(a.join("keep.txt"), "mine").unwrap();
let undo_n = s.undo.len();
let e = s.execute("edit.undo", &json!({})).unwrap_err().to_string();
assert!(e.contains("already exists"), "{e}");
assert_eq!(s.undo.len(), undo_n, "the step stays on the stack");
assert_eq!(std::fs::read_to_string(a.join("keep.txt")).unwrap(), "mine");
assert!(b.join("one.png").exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
std::fs::remove_dir_all(&a).unwrap();
// move into another folder, undo, redo
let into = root.join("Archive");
s.execute("folder.move", &json!({"path": b.to_string_lossy(), "into": into.to_string_lossy()})).unwrap();
let moved = into.join("Italy");
assert_eq!(path(&s), moved.join("one.png").to_string_lossy());
s.execute("edit.undo", &json!({})).unwrap();
assert!(b.join("one.png").exists() && !moved.exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
// the redo target is taken (even by an empty folder): refused, nothing replaced
std::fs::create_dir_all(&moved).unwrap();
assert!(s.execute("edit.redo", &json!({})).is_err());
assert!(b.join("one.png").exists());
std::fs::remove_dir(&moved).unwrap();
s.execute("edit.redo", &json!({})).unwrap();
assert!(moved.join("one.png").exists() && moved.join("one.xmp").exists() && !b.exists());
assert_eq!(path(&s), moved.join("one.png").to_string_lossy());
let _ = std::fs::remove_dir_all(&root);
}
/// Assisted culling: a burst (same scene, seconds apart) of a sharp and two soft shots, plus an
/// unrelated photo: the burst is grouped, the sharp one is its best, blurry shots can be rejected.
#[test]
@@ -590,6 +731,7 @@ fn smart_previews_folder_is_chosen_per_library() {
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], true);
// the setting is saved with the library
drop(s); // one session per library (issue #99)
let mut again = Session::new().with_fs();
again.open_library(&lib, false).unwrap();
assert_eq!(again.execute("library.smartPreviewsLocation", &json!({})).unwrap()["path"], to.to_string_lossy().as_ref());
+44
View File
@@ -163,6 +163,50 @@ fn a_failed_or_corrupt_copy_keeps_the_source() {
}
}
/// Issue #96: Import → Copy used a plain unverified copy (and check-then-copy naming that could
/// replace a file). It now gets the same verified copy as Move: a copy that fails or comes out
/// different is removed and reported as a failed import, nothing is catalogued, and the card
/// is untouched; a good copy never replaces an existing file.
#[test]
fn copy_import_verifies_each_copy() {
let import_copy = |s: &mut Session, card: &Path, dest: &Path| {
s.execute(
"library.import",
&json!({"paths": [card.to_string_lossy()], "mode": "copy", "destination": dest.to_string_lossy(), "organize": "flat"}),
)
.unwrap()
};
for fault in [Fault::FailWrite, Fault::Corrupt] {
let base = temp_dir(&format!("copyfail-{fault:?}"));
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 3);
let bytes = std::fs::read(card.join("a.png")).unwrap();
let mut s = session();
inject(fault);
let r = import_copy(&mut s, &card, &dest);
inject(Fault::None);
assert_eq!(len(&r, "failed"), 1, "{fault:?}: {r}");
assert_eq!(len(&r, "imported"), 0, "{fault:?}: {r}");
assert!(r.to_string().contains("the original is untouched"), "{r}");
assert_eq!(s.catalog.len(), 0);
assert_eq!(std::fs::read(card.join("a.png")).unwrap(), bytes, "{fault:?}: source intact");
assert_eq!(files_under(&dest), Vec::<String>::new(), "{fault:?}: no bad copy left");
let _ = std::fs::remove_dir_all(&base);
}
// a good copy: verified, and a taken name gets -1 instead of being replaced
let base = temp_dir("copyok");
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 4);
std::fs::create_dir_all(&dest).unwrap();
std::fs::write(dest.join("a.png"), b"someone else's file").unwrap();
let mut s = session();
let r = import_copy(&mut s, &card, &dest);
assert_eq!(len(&r, "imported"), 1, "{r}");
assert_eq!(std::fs::read(dest.join("a.png")).unwrap(), b"someone else's file");
assert_eq!(std::fs::read(dest.join("a-1.png")).unwrap(), std::fs::read(card.join("a.png")).unwrap());
let _ = std::fs::remove_dir_all(&base);
}
/// A source that can't be removed (a read-only card) stays and is reported; the photo is
/// imported from its copy.
#[test]
+31
View File
@@ -113,6 +113,7 @@ fn close_writes_snapshot_and_view_state() {
let expect = s.catalog.to_snapshot();
s.close_library().unwrap();
assert_eq!(std::fs::metadata(dir.join("catalog.log")).unwrap().len(), 0);
drop(s); // one session per library (issue #99)
let s2 = open(&dir, true);
let r = &s2.library.as_ref().unwrap().report;
@@ -178,6 +179,7 @@ fn new_library_without_seed_is_empty_and_compacts() {
assert!(p["snapshots"].as_u64() >= Some(1), "{p}");
assert!(p["lastSnapshot"]["bytes"].as_u64() > Some(0), "{p}");
assert!(p["lastSnapshot"]["totalMs"].as_f64() >= p["lastSnapshot"]["serializeMs"].as_f64(), "{p}");
drop(s); // one session per library (issue #99)
let s2 = open(&dir, true);
assert_eq!(s2.catalog.len(), 0, "an existing library is never seeded");
assert_eq!(s2.catalog.albums().count(), 1);
@@ -376,3 +378,32 @@ fn scale_100k_library_queries() {
worst.sort_by(|a, b| b.1.total_cmp(&a.1));
eprintln!("slowest: {:?}", &worst[..3]);
}
/// Issue #99: a library is open in one session at a time. A second opener is refused (nothing
/// read or written), the owning session can reopen it, and it is free again once closed.
#[test]
fn a_library_open_elsewhere_is_refused() {
let dir = temp_dir("locked");
let mut s = open(&dir, true);
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
let log = std::fs::read(dir.join("catalog.log")).unwrap();
let mut other = Session::new();
let e = other.open_library(&dir, true).unwrap_err();
assert!(matches!(e, crate::EngineError::LibraryInUse(_)), "{e:?}");
assert!(e.to_string().contains("already open in"), "{e}");
assert!(other.library.is_none());
assert_eq!(std::fs::read(dir.join("catalog.log")).unwrap(), log, "the refused opener wrote nothing");
// the session that has it open may reopen it (Settings → Open Library on the same folder)
s.close_library().unwrap();
s.open_library(&dir, true).unwrap();
assert!(other.open_library(&dir, true).is_err(), "still locked after reopening");
s.execute("photo.rate", &json!({"rating": 4})).unwrap();
let expect = s.catalog.to_snapshot();
drop(s);
other.open_library(&dir, true).unwrap();
assert_eq!(other.catalog.to_snapshot(), expect);
let _ = std::fs::remove_dir_all(&dir);
}
+193
View File
@@ -0,0 +1,193 @@
//! A failed save is an error, not a silent success: a command whose journal append fails returns
//! [`EngineError::NotSaved`], its change stays applied and queued, and the next save writes it.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use lightcraft_catalog::{Flag, MemStore, SnapshotPolicy, Store};
use serde_json::json;
use crate::library::LibraryStores;
use crate::{EngineError, Session};
/// A store whose writes fail while `fail` is set (a full or unplugged disk).
#[derive(Clone, Default)]
struct Flaky {
files: MemStore,
fail: Arc<AtomicBool>,
/// Only snapshots fail (appends work).
snapshots_only: bool,
/// Only appends fail (a dead log handle; fresh files can still be written).
appends_only: bool,
}
impl Flaky {
fn failing(&self) -> bool {
self.fail.load(Ordering::SeqCst)
}
fn set_failing(&self, on: bool) {
self.fail.store(on, Ordering::SeqCst);
}
/// The files as a crash at this moment would leave them.
fn crash_copy(&self) -> Flaky {
let m = MemStore::new();
*m.files.lock().unwrap() = self.files.files.lock().unwrap().clone();
Flaky { files: m, ..Default::default() }
}
}
impl Store for Flaky {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.failing() && !self.appends_only {
return Err(std::io::Error::other("disk full"));
}
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.failing() && !self.snapshots_only {
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"flaky".into()
}
}
fn open(store: &Flaky, seed: bool) -> Session {
let mut s = Session::new();
let stores = LibraryStores { dir: "flaky".into(), catalog: Box::new(store.clone()), files: Box::new(MemStore::new()), on_disk: false };
s.open_library_in(stores, seed).unwrap();
s
}
#[test]
fn failed_append_fails_the_command_and_is_retried() {
let store = Flaky::default();
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
s.execute("photo.rate", &json!({"rating": 1})).unwrap();
store.set_failing(true);
let e = s.execute("photo.rate", &json!({"rating": 4})).unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)), "{e:?}");
let msg = e.to_string();
assert!(
msg.starts_with("saved in memory but not written to disk: ") && msg.contains("disk full") && msg.ends_with("LightCraft will retry"),
"{msg}"
);
// applied in memory, undoable, queued
assert_eq!(s.catalog.photo(id).unwrap().rating, 4);
assert!(!s.undo.is_empty());
assert_eq!(s.unsaved().map(|u| u.0), Some(1));
// queries and commands that change nothing are unaffected (they retry the queue, but report
// their own result)
let info = s.execute("library.info", &json!({})).unwrap();
assert_eq!(info["unsavedOps"], 1, "{info}");
assert!(info["unsavedError"].as_str().unwrap().contains("disk full"), "{info}");
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
// a second change fails too; both stay queued, in order
let e = s.execute("photo.flag", &json!({"flag": "pick"})).unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)));
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
// the frame loop backs off instead of retrying every frame
s.persist_if_dirty();
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
// a crash now loses only the unsaved changes; the earlier ones are on disk
assert_eq!(open(&store.crash_copy(), true).catalog.photo(id).unwrap().rating, 1);
// the disk comes back: the next save writes the whole queue
store.set_failing(false);
s.execute("library.info", &json!({})).unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.library.as_ref().unwrap().last_error.is_none());
let expect = s.catalog.to_snapshot();
drop(s); // no close: like a crash
let s2 = open(&store, true);
assert_eq!(s2.catalog.to_snapshot(), expect);
let p = s2.catalog.photo(id).unwrap();
assert_eq!((p.rating, p.flag), (4, Flag::Pick));
}
/// Undo is a command like any other: a failed write of its op is reported, and retried.
#[test]
fn failed_append_of_undo_is_reported_and_retried() {
let store = Flaky::default();
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
let before = s.catalog.photo(id).unwrap().rating;
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
store.set_failing(true);
assert!(matches!(s.execute("edit.undo", &json!({})), Err(EngineError::NotSaved(_))));
assert_eq!(s.catalog.photo(id).unwrap().rating, before);
store.set_failing(false);
s.persist().unwrap();
drop(s);
assert_eq!(open(&store, true).catalog.photo(id).unwrap().rating, before);
}
/// A failed compaction is not a failed command: the command's own records were appended.
#[test]
fn failed_snapshot_does_not_fail_the_command() {
let store = Flaky { snapshots_only: true, ..Default::default() };
let mut s = open(&store, false);
s.library.as_mut().unwrap().journal_mut().policy = SnapshotPolicy { max_records: 1, max_bytes: u64::MAX };
store.set_failing(true);
s.execute("album.create", &json!({"name": "A"})).unwrap();
s.execute("album.create", &json!({"name": "B"})).unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.library.as_ref().unwrap().last_error.is_some(), "the compaction failure is reported");
drop(s);
store.set_failing(false);
assert_eq!(open(&store, false).catalog.albums().count(), 2);
}
/// Issue #101: quitting while the log can't be appended to still saves the queued changes in the
/// closing snapshot (a fresh file), and reports success; they are not appended again later.
#[test]
fn close_saves_queued_changes_in_the_snapshot_when_appends_fail() {
let store = Flaky { appends_only: true, ..Default::default() };
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
store.set_failing(true);
assert!(matches!(s.execute("photo.rate", &json!({"rating": 4})), Err(EngineError::NotSaved(_))));
assert!(matches!(s.execute("photo.flag", &json!({"flag": "pick"})), Err(EngineError::NotSaved(_))));
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.pending_log.is_empty());
let expect = s.catalog.to_snapshot();
drop(s);
store.set_failing(false);
let mut s2 = open(&store, true);
assert_eq!(s2.catalog.to_snapshot(), expect);
let p = s2.catalog.photo(id).unwrap();
assert_eq!((p.rating, p.flag), (4, Flag::Pick));
// later edits continue after the absorbed ops
s2.execute("photo.rate", &json!({"rating": 2})).unwrap();
let expect = s2.catalog.to_snapshot();
drop(s2);
assert_eq!(open(&store, true).catalog.to_snapshot(), expect);
}
/// When neither the log nor the snapshot can be written, closing fails with `NotSaved` and the
/// changes stay queued (a later retry can still save them).
#[test]
fn close_fails_when_nothing_can_be_saved() {
let store = Flaky::default();
let mut s = open(&store, true);
store.set_failing(true);
assert!(s.execute("photo.rate", &json!({"rating": 4})).is_err());
let e = s.close_library().unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)), "{e:?}");
assert_eq!(s.unsaved().map(|u| u.0), Some(1));
store.set_failing(false);
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
}
+2
View File
@@ -126,11 +126,13 @@ fn preferences_persist_with_the_library() {
assert_eq!(r["persistent"], true);
// export presets are library preferences too
s.execute("export.savePreset", &json!({"name": "Proof", "params": {"format": "jpeg", "percent": 25}})).unwrap();
drop(s); // one session per library (issue #99)
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
assert_eq!(s.export_params(&json!({"preset": "Proof"})).unwrap(), json!({"format": "jpeg", "percent": 25}));
// older prefs.json files (snake_case `last_export`, no import section) still load
std::fs::write(lib.join("prefs.json"), r#"{"xmp": {"autoWrite": false}, "last_export": {"format": "png"}}"#).unwrap();
drop(s);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
assert_eq!(s.last_export, Some(json!({"format": "png"})));
+97
View File
@@ -0,0 +1,97 @@
//! Issue #103: a damaged or unreadable settings file (prefs.json, presets.json, view.json) is
//! never silently replaced by the defaults: a damaged one is kept aside, an unreadable one is not
//! written this session, and the user is told.
use std::sync::{Arc, Mutex};
use lightcraft_catalog::{MemStore, Store};
use serde_json::json;
use crate::Session;
use crate::library::LibraryStores;
fn temp_dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-settings-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn damaged_prefs_are_kept_aside_and_reported() {
let lib = temp_dir("damaged");
let bad = br#"{"exportPresets": [{"name": "Proof", "par"#;
std::fs::write(lib.join("prefs.json"), bad).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
let kept: Vec<_> = std::fs::read_dir(&lib)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().to_string())
.filter(|n| n.starts_with("prefs.json.corrupt-"))
.collect();
assert_eq!(kept.len(), 1, "{kept:?}");
assert_eq!(std::fs::read(lib.join(&kept[0])).unwrap(), bad);
let warnings = s.take_library_warnings();
assert_eq!(warnings.len(), 1);
assert!(warnings[0].contains("prefs.json is damaged") && warnings[0].contains(&kept[0]), "{warnings:?}");
assert!(s.take_library_warnings().is_empty(), "each warning is handed out once");
let info = s.execute("library.info", &json!({})).unwrap();
assert_eq!(info["settingsWarnings"].as_array().map(Vec::len), Some(1), "{info}");
// saving goes on (the damaged original is safe in its copy)
s.execute("library.preferences", &json!({"cacheMb": 300})).unwrap();
assert!(String::from_utf8(std::fs::read(lib.join("prefs.json")).unwrap()).unwrap().contains("300"));
assert_eq!(std::fs::read(lib.join(&kept[0])).unwrap(), bad);
drop(s);
let _ = std::fs::remove_dir_all(&lib);
}
/// `presets.json` and `prefs.json` can't be read (locked by another program, an I/O error).
#[derive(Clone, Default)]
struct Unreadable {
files: MemStore,
unreadable: Arc<Mutex<Vec<String>>>,
}
impl Store for Unreadable {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
if self.unreadable.lock().unwrap().iter().any(|n| n == name) {
return Err(std::io::Error::other("locked by another process"));
}
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"unreadable".into()
}
}
#[test]
fn unreadable_settings_are_not_overwritten() {
let files = Unreadable::default();
let presets = br#"{"user": [{"id": "user.mine", "name": "Mine", "group": "User Presets", "settings": {}}]}"#.to_vec();
let prefs = br#"{"cacheMb": 777}"#.to_vec();
files.files.set("presets.json", presets.clone());
files.files.set("prefs.json", prefs.clone());
*files.unreadable.lock().unwrap() = vec!["presets.json".into(), "prefs.json".into()];
let mut s = Session::new();
let stores = LibraryStores { dir: "unreadable".into(), catalog: Box::new(MemStore::new()), files: Box::new(files.clone()), on_disk: false };
s.open_library_in(stores, true).unwrap();
let warnings = s.take_library_warnings();
assert_eq!(warnings.len(), 2, "{warnings:?}");
assert!(warnings.iter().all(|w| w.contains("couldn't be read") && w.contains("won't overwrite")), "{warnings:?}");
// a preset change and a preference change: neither file is replaced by defaults
s.execute("preset.create", &json!({"name": "New"})).unwrap();
let e = s.execute("library.preferences", &json!({"cacheMb": 300}));
assert!(e.is_err(), "saving preferences is refused, not done over the unread file");
assert_eq!(files.files.get("presets.json").unwrap(), presets);
assert_eq!(files.files.get("prefs.json").unwrap(), prefs);
}
+178
View File
@@ -329,6 +329,55 @@ fn convert_raw_to_dng() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #106: Convert to DNG wrote straight to the final name (no temp file, sync or check) and
/// relinked the photo even when the write was cut short. A failed write now leaves no DNG and
/// the photo on its raw; a good one is verified and never replaces an existing file.
#[test]
fn convert_to_dng_is_verified_and_atomic() {
let dir = temp_dir("todng-safe");
// a synthetic raw (stored as DNG, catalogued as a NEF so Convert to DNG takes it)
let raw = dir.join("shot.dng");
let bytes = synthetic_dng_with(None, Default::default());
std::fs::write(&raw, &bytes).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [raw.to_string_lossy()]})).unwrap();
let id = s.catalog.photos().next().unwrap().id;
let path = raw.to_string_lossy().to_string();
s.catalog
.apply(lightcraft_catalog::Op::Relink {
id,
file_name: "shot.dng".into(),
source: lightcraft_catalog::Source::File { path: path.clone() },
format: Some("NEF".into()),
})
.unwrap();
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
let names = || {
let mut v: Vec<String> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
v.sort();
v
};
{
// the drive fills up part-way
let _fault = lightcraft_catalog::safe_file::fail_writes_after(200);
let r = s.execute("photo.convertToDng", &json!({})).unwrap();
assert_eq!(r["converted"].as_array().unwrap().len(), 0, "{r}");
assert!(r["skipped"][0][1].as_str().unwrap().contains("the raw is kept"), "{r}");
}
assert_eq!(names(), ["shot.dng"], "no partial DNG");
assert_eq!(s.catalog.photo(id).unwrap().source, lightcraft_catalog::Source::File { path: path.clone() }, "not relinked");
// a good conversion: a new name (the existing file is not replaced), decodable, relinked
let r = s.execute("photo.convertToDng", &json!({})).unwrap();
let out = r["converted"][0]["path"].as_str().unwrap().to_string();
assert!(out.ends_with("shot-2.dng"), "{r}");
assert_eq!(std::fs::read(&raw).unwrap(), bytes, "the raw is untouched");
let back = lightcraft_raw::decode(&std::fs::read(&out).unwrap()).unwrap();
assert_eq!(back.data, lightcraft_raw::decode(&bytes).unwrap().data);
assert_eq!(s.catalog.photo(id).unwrap().source, lightcraft_catalog::Source::File { path: out });
assert_eq!(names(), ["shot-2.dng", "shot.dng"], "no temp file left");
let _ = std::fs::remove_dir_all(&dir);
}
/// Edit in External Editor (engine half): a 16-bit TIFF `-Edit` copy with the edits, next to
/// the original, added and stacked on top of it; a second one doesn't overwrite the first.
#[test]
@@ -576,3 +625,132 @@ fn copyright_status_usage_terms_and_url_round_trip() {
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// A sidecar another raw developer wrote: develop settings in attributes and elements, a
/// structured edit history, an unknown namespace.
const OTHER_APP_SIDECAR: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Other Toolkit">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:xmp="http://ns.adobe.com/xap/1.0/"
xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#"
xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
xmlns:other="http://example.com/other/1.0/"
xmp:Rating="2"
crs:Version="99.0"
crs:Exposure2012="+0.65"
crs:HasSettings="True"
other:Secret="keep me">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action="derived" stEvt:parameters="converted from image/x-raw to image/png"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>128, 140</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>"#;
/// Issue #92: saving merges into another app's sidecar instead of replacing it.
#[test]
fn saving_merges_into_another_apps_sidecar() {
let src = temp_dir("merge");
let lib = temp_dir("merge-lib");
write_png(&src.join("shot.png"), 1);
std::fs::write(src.join("shot.xmp"), OTHER_APP_SIDECAR).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("shot.png").to_string_lossy()]})).unwrap();
let id = only(&s);
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
s.execute("photo.rate", &json!({"rating": 5})).unwrap();
s.execute("photo.setMeta", &json!({"title": "Harbour", "keywords": ["boats"]})).unwrap();
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 30})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
assert_eq!(r["merged"][0], src.join("shot.xmp").to_string_lossy().as_ref(), "{r}");
let out = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
// the other app's data, byte for byte
for part in [
"crs:Exposure2012=\"+0.65\"",
"other:Secret=\"keep me\"",
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"derived\" stEvt:parameters=\"converted from image/x-raw to image/png\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>\n <rdf:li>128, 140</rdf:li>",
] {
assert!(out.contains(part), "lost {part:?}:\n{out}");
}
let d = lightcraft_meta::parse_xmp(&out).unwrap();
assert_eq!(d.metadata.rating, Some(5), "ours replaces theirs");
assert_eq!(d.metadata.title.as_deref(), Some("Harbour"));
assert_eq!(d.properties.get("crs:Version"), Some(&vec!["99.0".to_string()]));
let saved = s.catalog.photo(id).unwrap().develop.clone();
// reading it back restores our settings (lc:settings wins over crs:)
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().develop, saved);
// saving again keeps one LightCraft description and the other app's data
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let again = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert!(again.contains("other:Secret=\"keep me\"") && again.contains("crs:Exposure2012=\"+0.65\""));
assert_eq!(lightcraft_meta::parse_xmp(&again).unwrap().metadata.rating, Some(3));
// a sidecar that isn't XMP is kept as a backup before it is replaced
std::fs::write(src.join("shot.xmp"), b"<not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let backup = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert!(backup.contains("shot.xmp.bak-"), "{r}");
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join("shot.xmp")).unwrap()).unwrap().metadata.rating, Some(3));
// … and another one never overwrites the first backup
std::fs::write(src.join("shot.xmp"), b"<still not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let second = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert_ne!(second, backup);
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(std::fs::read(&second).unwrap(), b"<still not xmp");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #92: two catalogued files with one stem (`IMG_1.jpg` + `IMG_1.png`) don't share a
/// Stem-named sidecar: the first by name keeps `IMG_1.xmp`, the other writes and reads
/// `IMG_1.png.xmp`.
#[test]
fn files_sharing_a_stem_get_their_own_sidecars() {
let src = temp_dir("stem");
let lib = temp_dir("stem-lib");
write_png(&src.join("IMG_1.png"), 1);
let img = lightcraft_raster::Rgba8::from_fn(40, 24, |x, y| [(x * 6) as u8, (y * 9) as u8, 7, 255]);
let jpg =
crate::export::encode_image(&img, &crate::export::ExportOptions { format: crate::export::ExportFormat::Jpeg, ..Default::default() }).unwrap();
std::fs::write(src.join("IMG_1.jpg"), jpg).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = |s: &Session, name: &str| s.catalog.photos().find(|p| p.file_name == name).unwrap().id;
let (j, p) = (id(&s, "IMG_1.jpg"), id(&s, "IMG_1.png"));
assert_eq!(s.sidecar_naming(j), crate::sidecar::SidecarNaming::Stem);
assert_eq!(s.sidecar_naming(p), crate::sidecar::SidecarNaming::Full);
s.execute("photo.rate", &json!({"ids": [j.0], "rating": 2})).unwrap();
s.execute("photo.rate", &json!({"ids": [p.0], "rating": 5})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!(r["written"].as_array().unwrap().len(), 2, "{r}");
let rating = |f: &str| lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join(f)).unwrap()).unwrap().metadata.rating;
assert_eq!(rating("IMG_1.xmp"), Some(2));
assert_eq!(rating("IMG_1.png.xmp"), Some(5));
// each reads its own back
s.execute("photo.rate", &json!({"ids": [j.0, p.0], "rating": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!((s.catalog.photo(j).unwrap().rating, s.catalog.photo(p).unwrap().rating), (2, 5));
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
+222 -24
View File
@@ -275,37 +275,47 @@ fn module_source(m: &Module, consts: &str) -> String {
}
impl Gpu {
/// Create a device on the best available adapter (no software fallback), or `None`.
pub fn new() -> Option<Gpu> {
/// Create a device on the best available adapter (no software fallback), or why there is none.
pub fn new() -> Result<Gpu, String> {
let mut desc = wgpu::InstanceDescriptor::new_without_display_handle();
desc.backends = wgpu::Backends::PRIMARY;
let instance = wgpu::Instance::new(desc);
let adapter = pollster::block_on(
instance.request_adapter(&wgpu::RequestAdapterOptions { power_preference: wgpu::PowerPreference::HighPerformance, ..Default::default() }),
)
.ok()?;
.map_err(|e| format!("no GPU adapter ({e})"))?;
let info = adapter.get_info();
if info.device_type == wgpu::DeviceType::Cpu {
log::info!("gpu: only a software adapter ({}), using the CPU pipeline", info.name);
return None;
return Err(format!("software adapter ({}) skipped: the CPU pipeline is faster", info.name));
}
let limits = adapter.limits();
let mut limits = adapter.limits();
if limits.max_storage_buffers_per_shader_stage < 10 {
log::info!("gpu: {} has too few storage buffers per stage", info.name);
return None;
return Err(format!("{} has too few storage buffers per shader stage ({} < 10)", info.name, limits.max_storage_buffers_per_shader_stage));
}
if let Some((binding, buffer)) = limits_override() {
limits.max_storage_buffer_binding_size = limits.max_storage_buffer_binding_size.min(binding);
limits.max_buffer_size = limits.max_buffer_size.min(buffer);
log::info!("gpu: LIGHTCRAFT_GPU_LIMITS: storage bindings ≤ {} MiB, buffers ≤ {} MiB", binding >> 20, buffer >> 20);
}
let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor {
label: Some("lightcraft"),
required_limits: limits.clone(),
..Default::default()
}))
.ok()?;
// Errors outside an error scope (e.g. out of memory during a render): log, and stop using
// the GPU (the render in flight is redone on the CPU).
.map_err(|e| format!("{}: device creation failed ({e})", info.name))?;
// Errors outside a render's error scopes (see `ErrorScopes`): log, and stop using the GPU.
device.on_uncaptured_error(std::sync::Arc::new(|e| {
log::error!("gpu: {e}");
crate::mark_broken();
crate::mark_broken(&format!("device error: {e}"));
}));
// wgpu reports a lost device (driver reset, GPU hang, watchdog) only here — never as an
// error — so without this callback a lost device went unnoticed.
device.set_device_lost_callback(|reason, msg| {
log::error!("gpu: device lost ({reason:?}): {msg}");
crate::mark_broken(&format!("device lost ({reason:?}): {msg}"));
});
let scope = device.push_error_scope(wgpu::ErrorFilter::Validation);
let consts = constants();
let mut kernels = HashMap::new();
@@ -334,7 +344,7 @@ impl Gpu {
}
if let Some(e) = pollster::block_on(scope.pop()) {
log::error!("gpu: kernels failed to build on {}: {e}", info.name);
return None;
return Err(format!("{}: kernels failed to build ({e})", info.name));
}
let dummy = device.create_buffer(&wgpu::BufferDescriptor {
label: Some("dummy"),
@@ -343,13 +353,20 @@ impl Gpu {
mapped_at_creation: false,
});
let max_buffer = limits.max_storage_buffer_binding_size.min(limits.max_buffer_size);
log::info!("gpu: {} ({:?})", info.name, info.backend);
Some(Gpu { device, queue, info, kernels, dummy, free: Default::default(), max_buffer })
log::info!("gpu: {} ({:?}), buffers ≤ {} MiB", info.name, info.backend, max_buffer >> 20);
Ok(Gpu { device, queue, info, kernels, dummy, free: Default::default(), max_buffer })
}
/// A buffer of `len` 32-bit values with undefined contents (a recycled one when possible).
///
/// A buffer over the device limit fails the render ([`FailKind::Limit`]) instead of raising a
/// validation error: the caller gets a placeholder and the render is redone on the CPU.
pub fn buffer(&self, len: usize) -> Buf {
let size = (len.max(1) * 4) as u64;
if size > self.limit() {
self.over_limit(size);
return Buf { buf: Some(Tracked::new(self.placeholder())), len };
}
let recycled = {
let mut f = self.free.lock().unwrap_or_else(|e| e.into_inner());
// the smallest pooled buffer that holds `size` with at most 25 % to spare: photos of
@@ -426,6 +443,10 @@ impl Gpu {
if bytes.is_empty() {
return self.buffer(1);
}
if bytes.len() as u64 > self.limit() {
self.over_limit(bytes.len() as u64);
return Buf { buf: Some(Tracked::new(self.placeholder())), len };
}
let buf = self.device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
label: None,
contents: bytes,
@@ -434,9 +455,32 @@ impl Gpu {
Buf { buf: Some(Tracked::new(buf)), len }
}
fn over_limit(&self, size: u64) {
fail(
FailKind::Limit,
format!("a {} MiB buffer exceeds the device's {} MiB storage-buffer limit", size.div_ceil(1 << 20), self.limit() >> 20),
);
}
/// Stands in for a buffer that could not be created (the render has failed already).
fn placeholder(&self) -> wgpu::Buffer {
self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("placeholder"),
size: 16,
usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC | wgpu::BufferUsages::COPY_DST,
mapped_at_creation: false,
})
}
/// Whether a buffer of `len` 32-bit values fits the device limits.
pub fn fits(&self, len: usize) -> bool {
(len as u64) * 4 <= self.max_buffer
(len as u64) * 4 <= self.limit()
}
/// Largest buffer a render may use (bytes): the device limit, or a lower one set for the
/// render on this thread ([`LimitOverride`]).
pub fn limit(&self) -> u64 {
LIMIT.try_with(|l| l.get()).ok().flatten().map_or(self.max_buffer, |l| l.min(self.max_buffer))
}
pub fn encoder(&self) -> wgpu::CommandEncoder {
@@ -445,10 +489,12 @@ impl Gpu {
/// Record kernel `name` with parameters `p` over buffers `bufs` (None = unused binding).
pub fn run(&self, enc: &mut wgpu::CommandEncoder, name: &str, p: &[u32], bufs: &[Option<&Buf>], groups: [u32; 3]) {
if failed() {
return; // the render is redone on the CPU: don't spend the GPU on it
}
let Some(k) = self.kernels.get(name) else {
// a kernel name typo: fail this render over to the CPU instead of crashing
log::error!("gpu: unknown kernel {name}");
crate::mark_broken();
fail(FailKind::Fatal, format!("unknown kernel {name}"));
return;
};
assert_eq!(bufs.len(), k.nbuf, "{name}: binding count");
@@ -470,8 +516,16 @@ impl Gpu {
}
/// Submit `enc`, then read back the first `len` 32-bit values of `src`.
///
/// A failed readback (device lost, map error, timeout) fails the render ([`FailKind::Fatal`])
/// and returns zeros, which the caller discards; after an earlier failure in the same render
/// nothing more is submitted.
pub fn finish_and_read<T: bytemuck::Pod>(&self, mut enc: wgpu::CommandEncoder, src: &Buf, len: usize) -> Vec<T> {
assert!(len <= src.len, "readback of {len} values from a buffer of {}", src.len);
let zeros = || vec![<T as bytemuck::Zeroable>::zeroed(); len];
if failed() {
return zeros();
}
let size = (len * 4) as u64;
let staging = self.device.create_buffer(&wgpu::BufferDescriptor {
label: Some("readback"),
@@ -486,17 +540,20 @@ impl Gpu {
slice.map_async(wgpu::MapMode::Read, move |r| {
let _ = tx.send(r);
});
let _ = self.device.poll(wgpu::PollType::wait_indefinitely());
// A failed readback (device lost, out of memory) marks the GPU broken: the caller
// (`crate::render`) then discards this render's zeroed result and renders on the CPU.
let failed = |why: String| {
log::error!("gpu readback failed: {why}");
crate::mark_broken();
vec![<T as bytemuck::Zeroable>::zeroed(); len]
fail(FailKind::Fatal, format!("readback failed: {why}"));
zeros()
};
match rx.recv() {
// A lost device makes the wait fail and the map callback never run: checking both keeps
// this from blocking forever (or handing back an unwritten buffer as the image).
if let Err(e) = self.device.poll(wgpu::PollType::Wait { submission_index: None, timeout: Some(READBACK_TIMEOUT) }) {
return failed(format!("{e}"));
}
// another thread's poll may be the one that runs the callback: allow it a moment
match rx.recv_timeout(std::time::Duration::from_secs(10)) {
Ok(Ok(())) => {}
other => return failed(format!("{other:?}")),
Ok(Err(e)) => return failed(format!("{e}")),
Err(e) => return failed(format!("map callback: {e}")),
}
let data = match slice.get_mapped_range() {
Ok(d) => d,
@@ -509,6 +566,127 @@ impl Gpu {
}
}
/// Longest wait for a render's GPU work: beyond it the render fails over to the CPU (drivers
/// normally reset a hung GPU much sooner).
const READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
/// `LIGHTCRAFT_GPU_LIMITS`: lower the device limits, e.g. to reproduce a smaller adapter.
/// `webgpu` / `downlevel` = 128 MiB storage bindings and 256 MiB buffers (the WebGPU defaults),
/// `<n>` = n MiB storage bindings and 2n MiB buffers. Returns (binding bytes, buffer bytes).
fn limits_override() -> Option<(u64, u64)> {
parse_limits(&std::env::var("LIGHTCRAFT_GPU_LIMITS").ok()?)
}
fn parse_limits(v: &str) -> Option<(u64, u64)> {
let v = v.trim().to_ascii_lowercase();
match v.as_str() {
"" => None,
"webgpu" | "downlevel" | "default" => {
let d = wgpu::Limits::default();
Some((d.max_storage_buffer_binding_size, d.max_buffer_size))
}
n => {
let mb: u64 = n.trim_end_matches("mib").trim_end_matches("mb").trim().parse().ok()?;
let b = mb.max(1) << 20;
Some((b, 2 * b))
}
}
}
/// How a GPU render failed (it is then redone on the CPU).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum FailKind {
/// A buffer over the device limits: this render only.
Limit,
/// The device ran out of memory: this render only (the buffer pool is emptied).
OutOfMemory,
/// The image came back entirely black: redone on the CPU in case the GPU dropped work.
Blank,
/// Device error or loss, readback failure, incomplete image: the GPU is not used again.
Fatal,
}
#[derive(Clone, Debug)]
pub(crate) struct Failure {
pub kind: FailKind,
pub reason: String,
}
thread_local! {
/// A lower buffer limit for the render on this thread (fault injection in tests).
static LIMIT: std::cell::Cell<Option<u64>> = const { std::cell::Cell::new(None) };
}
/// Lowers [`Gpu::limit`] on this thread while it lives.
pub(crate) struct LimitOverride;
impl LimitOverride {
pub fn new(bytes: u64) -> LimitOverride {
LIMIT.with(|l| l.set(Some(bytes)));
LimitOverride
}
}
impl Drop for LimitOverride {
fn drop(&mut self) {
let _ = LIMIT.try_with(|l| l.set(None));
}
}
thread_local! {
/// The first failure of the render running on this thread.
static FAILURE: std::cell::RefCell<Option<Failure>> = const { std::cell::RefCell::new(None) };
}
/// Record a failure of the render running on this thread (the first one is kept).
pub(crate) fn fail(kind: FailKind, reason: String) {
let _ = FAILURE.try_with(|f| {
let mut f = f.borrow_mut();
if f.is_none() {
*f = Some(Failure { kind, reason });
}
});
}
/// Has the render on this thread failed?
pub(crate) fn failed() -> bool {
FAILURE.try_with(|f| f.borrow().is_some()).unwrap_or(false)
}
/// The render's failure, clearing it.
pub(crate) fn take_failure() -> Option<Failure> {
FAILURE.try_with(|f| f.borrow_mut().take()).ok().flatten()
}
/// wgpu error scopes around a render: out-of-memory, validation and internal errors raised by
/// this thread's calls are captured (instead of reaching the uncaptured-error handler) and fail
/// the render.
pub(crate) struct ErrorScopes([wgpu::ErrorScopeGuard; 3]);
impl ErrorScopes {
pub fn push(g: &Gpu) -> ErrorScopes {
ErrorScopes([
g.device.push_error_scope(wgpu::ErrorFilter::OutOfMemory),
g.device.push_error_scope(wgpu::ErrorFilter::Validation),
g.device.push_error_scope(wgpu::ErrorFilter::Internal),
])
}
/// Pop the scopes: the most telling error captured (out of memory first — an allocation
/// failure makes later commands invalid too — then internal, then validation).
pub fn pop(self) -> Option<Failure> {
let [oom, validation, internal] = self.0;
// scopes pop in reverse order of their creation
let internal = pollster::block_on(internal.pop());
let validation = pollster::block_on(validation.pop());
let oom = pollster::block_on(oom.pop());
if let Some(e) = oom {
return Some(Failure { kind: FailKind::OutOfMemory, reason: format!("out of device memory: {e}") });
}
internal.or(validation).map(|e| Failure { kind: FailKind::Fatal, reason: format!("device error: {e}") })
}
}
fn storage_entry(binding: u32, read_only: bool) -> wgpu::BindGroupLayoutEntry {
wgpu::BindGroupLayoutEntry {
binding,
@@ -529,3 +707,23 @@ pub fn groups1(n: usize) -> [u32; 3] {
let x = g.min(32768);
[x as u32, g.div_ceil(x) as u32, 1]
}
#[cfg(test)]
mod tests {
#[test]
fn limits_override_parses() {
assert_eq!(super::parse_limits("webgpu"), Some((128 << 20, 256 << 20)));
assert_eq!(super::parse_limits(" Downlevel "), Some((128 << 20, 256 << 20)));
assert_eq!(super::parse_limits("64"), Some((64 << 20, 128 << 20)));
assert_eq!(super::parse_limits("512MiB"), Some((512 << 20, 1024 << 20)));
assert_eq!(super::parse_limits(""), None);
assert_eq!(super::parse_limits("lots"), None);
}
#[test]
fn dispatches_stay_within_the_workgroup_limit() {
// 1-D kernels over a 100 MP image and 2-D ones over 16k × 16k stay under 65535 per axis
assert!(super::groups1(100_000_000 * 3).iter().all(|g| *g <= 65535));
assert!(super::groups2(16384, 16384, [16, 16]).iter().all(|g| *g <= 65535));
}
}
+141 -15
View File
@@ -7,14 +7,16 @@
//! CPU inside the same render (per-stage hybrid); see `docs/gpu-pipeline.md`.
//!
//! Use [`render`]: it returns `None` when the GPU is unavailable, disabled (`LIGHTCRAFT_GPU=0` or
//! [`set_enabled`]), or the render does not fit the device — callers then render on the CPU.
//! [`set_enabled`]), the render does not fit the device, or the device reported an error, ran out
//! of memory or returned an incomplete image — callers then render on the CPU. [`unavailable_reason`]
//! and [`last_fallback`] say why (`ui.inspect` → `perf.gpuReason` / `perf.gpuFallback`).
//! The browser build has no GPU path yet (WebGPU device creation is asynchronous): everything here
//! compiles to the CPU fallback on wasm32.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use lightcraft_develop::DevelopSettings;
use lightcraft_pipeline::{RenderRequest, Rendered, SourceInfo, StageCache};
@@ -34,12 +36,95 @@ static ENABLED: AtomicBool = AtomicBool::new(true);
/// Set when a GPU render failed: the process stays on the CPU from then on.
static BROKEN: AtomicBool = AtomicBool::new(false);
/// Why the GPU stopped being used (the first fatal failure).
static BROKEN_REASON: Mutex<Option<String>> = Mutex::new(None);
/// The latest render that fell back to the CPU, and why.
static LAST_FALLBACK: Mutex<Option<String>> = Mutex::new(None);
/// Stop using the GPU for the rest of the process (after a device error).
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
fn mark_broken() {
fn mark_broken(reason: &str) {
let mut r = BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner());
if r.is_none() {
*r = Some(reason.to_string());
}
BROKEN.store(true, Ordering::Relaxed);
}
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
fn record_fallback(reason: String) {
*LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()) = Some(reason);
}
/// Why renders do not use the GPU, or `None` when they do (or will, once the device is created).
/// E.g. "disabled by LIGHTCRAFT_GPU=0", "software adapter (llvmpipe …) skipped", "device lost …".
/// Never blocks.
pub fn unavailable_reason() -> Option<String> {
if env_disabled() {
return Some("disabled by LIGHTCRAFT_GPU=0".into());
}
if !ENABLED.load(Ordering::Relaxed) {
return Some("disabled by the GPU rendering preference (app.gpu)".into());
}
if BROKEN.load(Ordering::Relaxed) {
let r = BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner()).clone().unwrap_or_else(|| "device error".into());
return Some(format!("stopped after a GPU failure: {r}"));
}
#[cfg(not(target_arch = "wasm32"))]
{
match GPU.get() {
Some(Err(e)) => Some(e.clone()),
_ => None,
}
}
#[cfg(target_arch = "wasm32")]
{
Some("the browser build has no GPU path yet".into())
}
}
/// The latest render that fell back from the GPU to the CPU and why (device limit, out of memory,
/// device error, incomplete or blank result), if any.
pub fn last_fallback() -> Option<String> {
LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
/// A simulated GPU failure for the next [`render`] on this thread (tests of the CPU fallback).
#[doc(hidden)]
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Fault {
/// A command the device rejects (validation error).
Validation,
/// The per-pixel stage's work never runs (what a driver that silently drops or resets a
/// submission looks like: the readback holds no image).
DropWork,
/// A storage-buffer limit of this many bytes for this render.
Limit(u64),
}
thread_local! {
static FAULT: std::cell::Cell<Option<Fault>> = const { std::cell::Cell::new(None) };
}
/// Make the next [`render`] on this thread fail with `f` (tests).
#[doc(hidden)]
pub fn inject_fault(f: Fault) {
FAULT.with(|c| c.set(Some(f)));
}
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
fn take_fault() -> Option<Fault> {
FAULT.with(|c| c.take())
}
/// Use the GPU again after a failure (tests that inject faults).
#[doc(hidden)]
pub fn reset_failures() {
BROKEN.store(false, Ordering::Relaxed);
*BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner()) = None;
*LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()) = None;
}
/// Allow or forbid GPU rendering at runtime (a preference). `LIGHTCRAFT_GPU=0` forbids it for the
/// whole process regardless.
pub fn set_enabled(on: bool) {
@@ -57,20 +142,20 @@ fn env_disabled() -> bool {
}
#[cfg(not(target_arch = "wasm32"))]
static GPU: std::sync::OnceLock<Option<ctx::Gpu>> = std::sync::OnceLock::new();
static GPU: std::sync::OnceLock<Result<ctx::Gpu, String>> = std::sync::OnceLock::new();
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn device() -> Option<&'static ctx::Gpu> {
if env_disabled() {
return None;
}
GPU.get_or_init(|| std::panic::catch_unwind(ctx::Gpu::new).ok().flatten()).as_ref()
GPU.get_or_init(|| std::panic::catch_unwind(ctx::Gpu::new).unwrap_or_else(|_| Err("device creation panicked".into()))).as_ref().ok()
}
/// The device if it has been created (never creates it).
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn existing_device() -> Option<&'static ctx::Gpu> {
GPU.get().and_then(|g| g.as_ref())
GPU.get().and_then(|g| g.as_ref().ok())
}
/// Create the device and compile the kernels on a background thread now (app start), so the
@@ -160,7 +245,7 @@ pub fn set_pool_limit(bytes: u64) {
/// Free recycled buffers until at most `keep` bytes stay pooled (e.g. when the app goes idle).
pub fn trim_pool(keep: u64) {
#[cfg(not(target_arch = "wasm32"))]
if let Some(g) = GPU.get().and_then(|g| g.as_ref()) {
if let Some(g) = existing_device() {
g.trim(keep);
}
#[cfg(target_arch = "wasm32")]
@@ -191,19 +276,60 @@ pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &R
}
let gpu = device()?;
let ext = stages.map(|c| c.extension::<GpuStages>());
let r = {
let fault = take_fault();
let _ = ctx::take_failure(); // nothing left over from an earlier render on this thread
let (r, scoped) = {
let _scope = ctx::RenderScope::new(gpu);
std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| render::render(gpu, src, info, s, req, ext.as_deref())))
let errors = ctx::ErrorScopes::push(gpu);
let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| render::render(gpu, src, info, s, req, ext.as_deref(), fault)));
(r, errors.pop())
};
match r {
// a device error during the render: its result is not trustworthy
Ok(_) if BROKEN.load(Ordering::Relaxed) => None,
Ok(r) => r,
Err(_) => {
// An error the device raised explains the readback failures that follow it; a buffer
// over the limit explains the errors its placeholder raises.
let failure = match (ctx::take_failure(), scoped) {
(Some(f), _) if f.kind == ctx::FailKind::Limit => Some(f),
(f, scoped) => scoped.or(f),
};
match (r, failure) {
(Err(_), _) => {
log::error!("gpu: render failed; using the CPU pipeline from now on");
BROKEN.store(true, Ordering::Relaxed);
mark_broken("a GPU render panicked");
record_fallback("a GPU render panicked; the GPU is not used again".into());
None
}
(Ok(r), Some(f)) => {
// the view's cached device stages may hold results of the failed passes
if let Some(e) = &ext {
e.clear();
}
let what = match &r {
Some(r) => format!("{}×{}", r.image.width, r.image.height),
None => "GPU render".into(),
};
let reason = match f.kind {
ctx::FailKind::Fatal => {
mark_broken(&f.reason);
format!("{what}: {}; the GPU is not used again", f.reason)
}
ctx::FailKind::OutOfMemory => {
gpu.trim(0);
format!("{what}: {}", f.reason)
}
ctx::FailKind::Limit | ctx::FailKind::Blank => format!("{what}: {}", f.reason),
};
log::warn!("gpu: rendering on the CPU instead: {reason}");
if render::profiling() {
eprintln!(" gpu fallback to the CPU: {reason}");
}
record_fallback(reason);
None
}
// a device error on another thread or a lost device during the render
(Ok(Some(_)), None) if BROKEN.load(Ordering::Relaxed) => {
record_fallback(format!("device failure during the render: {}", unavailable_reason().unwrap_or_default()));
None
}
(Ok(r), None) => r,
}
}
#[cfg(target_arch = "wasm32")]
+3 -1
View File
@@ -71,6 +71,8 @@ const FIELDS: &[(&str, usize)] = &[
("OUT_Y", 3),
("OUT_TRC", 1),
("OUT_GAMMA", 1),
// first row of a band dispatch (the kernel runs over rows Y0.., see `render`)
("Y0", 1),
];
/// `(name, index)` of every field (for the WGSL constants).
@@ -87,7 +89,7 @@ pub fn finish_fields() -> Vec<(&'static str, usize)> {
}
/// Offset of field `name`; `None` (and a debug assertion) for a name not in [`FIELDS`].
fn index(name: &str) -> Option<usize> {
pub fn index(name: &str) -> Option<usize> {
let mut i = 0;
for (n, len) in FIELDS {
if *n == name {
+98 -24
View File
@@ -12,7 +12,7 @@ use lightcraft_pipeline::{Plan, RenderRequest, Rendered, SourceInfo, local};
use lightcraft_raster::resample::Filter;
use lightcraft_raster::{Histogram, Plane, Rgb32f, Rgba8};
use crate::ctx::{Buf, Gpu, groups1, groups2};
use crate::ctx::{Buf, FailKind, Gpu, fail, groups1, groups2};
use crate::params::{Present, finish_block};
/// Device-resident stages of one view, kept next to the CPU [`lightcraft_pipeline::StageCache`]
@@ -128,27 +128,54 @@ impl GpuStages {
pub(crate) struct Cx<'a> {
pub gpu: &'a Gpu,
enc: Option<wgpu::CommandEncoder>,
/// Kernel invocations recorded since the last submit.
pending: u64,
}
/// Kernel invocations recorded before a render submits them (under one full-size pass over a
/// 24 MP export, several passes over a preview): no single submission runs long.
const FLUSH_INVOCATIONS: u64 = 16 << 20;
/// Pixels per dispatch of the per-pixel stage (the heaviest kernel).
const BAND_PIXELS: usize = 4 << 20;
impl<'a> Cx<'a> {
pub fn new(gpu: &'a Gpu) -> Cx<'a> {
Cx { gpu, enc: Some(gpu.encoder()) }
Cx { gpu, enc: Some(gpu.encoder()), pending: 0 }
}
/// Record kernel `name` (see [`Gpu::run`]).
pub fn run(&mut self, name: &str, p: &[u32], bufs: &[Option<&Buf>], groups: [u32; 3]) {
let enc = self.enc.get_or_insert_with(|| self.gpu.encoder());
self.gpu.run(enc, name, p, bufs, groups);
// every kernel runs 256 invocations per workgroup
self.pending += groups.iter().map(|g| *g as u64).product::<u64>() * 256;
if self.pending >= FLUSH_INVOCATIONS {
self.flush();
}
}
/// Submit what is recorded and read back `len` values of `b`.
pub fn read<T: bytemuck::Pod>(&mut self, b: &Buf, len: usize) -> Vec<T> {
self.pending = 0;
let enc = self.enc.take().unwrap_or_else(|| self.gpu.encoder());
self.gpu.finish_and_read(enc, b, len)
}
/// Submit what is recorded so far without waiting. Renders submit stage by stage and every
/// [`FLUSH_INVOCATIONS`]: long submissions are what GPU watchdogs reset (Windows TDR, the
/// i915 hang check; issue #78's iGPU spent seconds on an export recorded as one submission),
/// and buffers dropped by earlier work become reusable by later work (lower peak memory).
pub fn flush(&mut self) {
self.pending = 0;
if let Some(enc) = self.enc.take() {
self.gpu.submit([enc.finish()]);
}
}
/// Submit what is recorded and wait for it (profiling only: stage timings).
pub fn sync(&mut self) {
self.pending = 0;
if let Some(enc) = self.enc.take() {
self.gpu.submit([enc.finish()]);
}
@@ -434,12 +461,13 @@ struct Host {
log_l: Option<Arc<Plane>>,
}
fn profiling() -> bool {
pub(crate) fn profiling() -> bool {
static ON: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*ON.get_or_init(|| std::env::var_os("LIGHTCRAFT_PROFILE").is_some())
}
/// Render on `gpu`, reusing `stages` (if given). `None` when the render does not fit the device.
/// Render on `gpu`, reusing `stages` (if given). `None` (or a result the caller discards) when the
/// render failed: the reason is recorded with [`crate::ctx::fail`].
pub fn render(
gpu: &Gpu,
src: &Arc<Rgb32f>,
@@ -447,20 +475,31 @@ pub fn render(
s: &DevelopSettings,
req: &RenderRequest,
stages: Option<&GpuStages>,
fault: Option<crate::Fault>,
) -> Option<Rendered> {
let mut t = profiling().then(std::time::Instant::now);
// Under `LIGHTCRAFT_PROFILE` each stage is submitted and waited for, so the timings are real.
let lap = |what: &str, t: &mut Option<std::time::Instant>, cx: &mut Cx<'_>| {
if let Some(t) = t {
// Each stage is submitted on its own; under `LIGHTCRAFT_PROFILE` also waited for, so the
// timings are real.
let lap = |what: &str, t: &mut Option<std::time::Instant>, cx: &mut Cx<'_>| match t {
Some(t) => {
cx.sync();
eprintln!(" gpu {what}: {:.1} ms", t.elapsed().as_secs_f64() * 1e3);
*t = std::time::Instant::now();
}
None => cx.flush(),
};
let plan = lightcraft_pipeline::plan(src, info, s, req);
let (w, h) = (plan.w, plan.h);
let n = w * h;
let _limit = match fault {
Some(crate::Fault::Limit(bytes)) => Some(crate::ctx::LimitOverride::new(bytes)),
_ => None,
};
if !gpu.fits(n * 3) {
fail(
FailKind::Limit,
format!("{w}×{h} needs {} MiB buffers, over the device's {} MiB storage-buffer limit", (n * 12).div_ceil(1 << 20), gpu.limit() >> 20),
);
return None;
}
let s = &*plan.settings;
@@ -521,23 +560,35 @@ pub fn render(
};
let (p, aux) = finish_block(&fp, &terms, &present);
let aux = gpu.upload(&aux);
let out = gpu.buffer(n);
cx.run(
"main",
&p,
&[
Some(&lin),
Some(&prep.log_l),
Some(&prep.base),
prep.clarity.as_deref(),
prep.texture.as_deref(),
prep.dark.as_deref(),
masks.as_ref(),
Some(&aux),
Some(&out),
],
groups2(w, h, [16, 16]),
);
// cleared, so that work which never ran cannot pass for an image (see the check below)
let out = cx.zeroed(n);
if fault == Some(crate::Fault::Validation) {
// a dispatch the device rejects (too many workgroups)
cx.run("box_h", &[0; 5], &[Some(&aux), Some(&out)], [gpu.device.limits().max_compute_workgroups_per_dimension + 1, 1, 1]);
}
// in bands of rows: each dispatch (and submission) stays short on a slow GPU
let rows = (BAND_PIXELS / w.max(1) / 16 * 16).max(16); // whole workgroups: bands never overlap
let y0_at = crate::params::index("Y0").unwrap_or(0);
let mut p = p;
for y0 in (0..h).step_by(rows).filter(|_| fault != Some(crate::Fault::DropWork)) {
p[y0_at] = y0 as u32;
cx.run(
"main",
&p,
&[
Some(&lin),
Some(&prep.log_l),
Some(&prep.base),
prep.clarity.as_deref(),
prep.texture.as_deref(),
prep.dark.as_deref(),
masks.as_ref(),
Some(&aux),
Some(&out),
],
groups2(w, rows.min(h - y0), [16, 16]),
);
}
// the alpha a mask overlay shows: copied out before the readback below submits
let overlay_mask = req.overlay.mask(s).map(|m| {
let list = s.masks.iter().filter(|m| m.visible && !m.components.is_empty());
@@ -549,8 +600,25 @@ pub fn render(
let data: Vec<[u8; 4]> = cx.read(&out, n);
let image = Rgba8 { width: w, height: h, data };
lap("finish + readback", &mut t, &mut cx);
if crate::ctx::failed() {
return None;
}
// The per-pixel kernel writes alpha 255 everywhere and `out` was cleared: a pixel with any
// other alpha means work silently did not run (a driver that reset or dropped a submission
// without reporting it — issue #78's all-black exports).
let unwritten = unwritten(&image.data);
if unwritten > 0 {
fail(FailKind::Fatal, format!("the GPU returned an incomplete image ({unwritten} of {n} pixels unwritten)"));
return None;
}
let histogram = Histogram::of_srgb8(&image);
lap("histogram", &mut t, &mut cx);
// An entirely black result (upstream work that did not run would give that too): redo it on
// the CPU, which costs time only for the rare photo that really is black.
if n >= 4096 && [&histogram.r, &histogram.g, &histogram.b].iter().all(|c| c.first() == Some(&histogram.total)) {
fail(FailKind::Blank, "the GPU result is entirely black; rendering it on the CPU to be sure".into());
return None;
}
let mut image = image;
let overlay_mask = overlay_mask.map(|r| match r {
Ok(b) => cx.read_plane(&b, w, h),
@@ -565,6 +633,11 @@ pub fn render(
Some(Rendered { image, histogram, deep: None })
}
/// Pixels whose alpha is not 255 (cheap: no early exit, so it vectorizes).
fn unwritten(data: &[[u8; 4]]) -> usize {
data.chunks(4096).filter(|c| c.iter().fold(255u8, |a, p| a & p[3]) != 255).map(|c| c.iter().filter(|p| p[3] != 255).count()).sum()
}
/// The white-balanced, retouched, denoised image.
fn linear(cx: &mut Cx<'_>, sampled: &Buf, info: &SourceInfo, plan: &Plan<'_>, host: &mut Host) -> Buf {
let (w, h) = (plan.w, plan.h);
@@ -679,6 +752,7 @@ fn prepare(cx: &mut Cx<'_>, lin: &Buf, plan: &Plan<'_>, req: &RenderRequest, pla
let sub = cx.gpu.buffer(m);
map(cx, "subsample", m, &[local::AIRLIGHT_STEP as u32], [Some(&d), None, None], &sub);
let air = local::airlight_of(cx.read(&sub, m));
cx.flush();
let b = Arc::new(d);
planes.dark = Some((sg.to_bits(), b.clone(), air));
(Some(b), air)
+3 -3
View File
@@ -243,11 +243,11 @@ fn out_encode(v: f32) -> f32 {
fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
let w = pu(F_W);
let h = pu(F_H);
if (gid.x >= w || gid.y >= h) {
let x = gid.x;
let y = gid.y + pu(F_Y0);
if (x >= w || y >= h) {
return;
}
let x = gid.x;
let y = gid.y;
let i = y * w + x;
let n = w * h;
let gain = pf(F_GAIN);
+26
View File
@@ -65,6 +65,32 @@ fn typical(s: &mut DevelopSettings) {
s.detail.sharpen_amount = 40.0;
}
/// A render big enough for the per-pixel stage to run in several bands of rows (and the render
/// in many submissions), with grain (position-dependent) and masks.
#[test]
fn banded_full_size_render_matches() {
if !gpu() {
return;
}
let (w, h) = (3000, 2000);
let src = scene(3, w, h);
let mut s = DevelopSettings::default();
typical(&mut s);
s.grain.amount = 30.0;
s.vignette.amount = -30.0;
s.masks = vec![Mask {
components: vec![MaskComponent {
name: None,
op: MaskOp::Add,
invert: false,
shape: MaskShape::Linear { start: Point::new(0.5, 0.0), end: Point::new(0.5, 0.9) },
}],
adjust: lightcraft_develop::LocalAdjustments { exposure: -0.6, ..Default::default() },
..Default::default()
}];
check("banded 3000×2000", &src, &SourceInfo { raw: true, ..Default::default() }, &s, &RenderRequest::fit(w, h));
}
fn cases() -> Vec<(&'static str, Edit)> {
vec![
("default", |_| {}),
+103
View File
@@ -0,0 +1,103 @@
//! Issue #78: a GPU render that cannot be trusted — over the device's buffer limits, a device
//! error, work that silently did not run — must come back as `None` (the caller renders on the
//! CPU) with the reason recorded, never as an image of zeros. One test, sequential: the fatal
//! faults stop GPU use for the whole process until `reset_failures`.
//! Skips (passes with a note) when no GPU adapter exists.
use std::sync::Arc;
use lightcraft_develop::{DevelopSettings, Mask, MaskComponent, MaskOp, MaskShape};
use lightcraft_geom::Point;
use lightcraft_gpu::Fault;
use lightcraft_pipeline::{RenderRequest, SourceInfo, render};
use lightcraft_raster::Rgba8;
fn mean_abs_diff(a: &Rgba8, b: &Rgba8) -> f64 {
let sum: u64 = a.data.iter().zip(&b.data).map(|(p, q)| (0..3).map(|c| p[c].abs_diff(q[c]) as u64).sum::<u64>()).sum();
sum as f64 / (a.data.len() * 3) as f64
}
fn masked(s: &mut DevelopSettings, k: usize) {
for i in 0..k {
let c = Point::new(0.2 + 0.2 * i as f64, 0.5);
s.masks.push(Mask {
id: i as u32 + 1,
components: vec![MaskComponent {
name: None,
op: MaskOp::Add,
invert: false,
shape: MaskShape::Radial { center: c, rx: 0.2, ry: 0.15, angle: 0.0, feather: 50.0, invert: false },
}],
..Default::default()
});
s.masks[i].adjust.exposure = 0.5;
}
}
#[test]
fn failed_gpu_renders_fall_back_with_a_reason() {
if !lightcraft_gpu::available() {
eprintln!("skipped: no GPU adapter ({:?})", lightcraft_gpu::unavailable_reason());
return;
}
assert_eq!(lightcraft_gpu::unavailable_reason(), None);
let (w, h) = (1200, 800);
let src = Arc::new(lightcraft_scenes::demo_library()[0].render(w, h));
let info = SourceInfo { raw: true, ..Default::default() };
let mut s = DevelopSettings::default();
s.light.exposure = 0.3;
s.effects.clarity = 20.0;
masked(&mut s, 4);
let req = RenderRequest::fit(w, h);
let cpu = render(&src, &info, &s, &req).image;
let ok = |what: &str| {
let g = lightcraft_gpu::render(&src, &info, &s, &req, None)
.unwrap_or_else(|| panic!("{what}: gpu render ({:?})", lightcraft_gpu::last_fallback()));
let d = mean_abs_diff(&cpu, &g.image);
assert!(d < 0.5, "{what}: GPU differs from the CPU by {d:.3} LSB");
};
ok("healthy");
// the output itself is over the limit: refused up front, the GPU stays in use
lightcraft_gpu::inject_fault(Fault::Limit(1 << 20));
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("limit"), "{why}");
assert!(lightcraft_gpu::available(), "a limit is no device failure");
// the output fits but the mask planes (4 masks: 4 planes in one buffer) don't: caught when the
// buffer is created, without a device error, and the GPU stays in use
lightcraft_gpu::inject_fault(Fault::Limit((w * h * 3 * 4 + 1024) as u64));
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("exceeds"), "{why}");
assert!(lightcraft_gpu::available(), "{:?}", lightcraft_gpu::unavailable_reason());
ok("after the limit faults");
// a device (validation) error: captured by the render's error scope; the GPU is not used again
lightcraft_gpu::inject_fault(Fault::Validation);
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("device error"), "{why}");
assert!(!lightcraft_gpu::available());
assert!(lightcraft_gpu::unavailable_reason().is_some_and(|r| r.contains("device error")), "{:?}", lightcraft_gpu::unavailable_reason());
lightcraft_gpu::reset_failures();
ok("after a device error");
// the per-pixel work never runs (a driver dropping a submission without an error): before
// the fix this came back as a fully black image
lightcraft_gpu::inject_fault(Fault::DropWork);
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none(), "an unwritten image must not be returned");
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
assert!(why.contains("incomplete"), "{why}");
assert!(!lightcraft_gpu::available());
lightcraft_gpu::reset_failures();
ok("after dropped work");
// a black photo is still rendered (on the CPU, to be sure)
let black = Arc::new(lightcraft_raster::Rgb32f::from_fn(256, 256, |_, _| [0.0; 3]));
let s0 = DevelopSettings::default();
assert!(lightcraft_gpu::render(&black, &info, &s0, &RenderRequest::fit(256, 256), None).is_none());
assert!(lightcraft_gpu::last_fallback().unwrap_or_default().contains("black"));
assert!(lightcraft_gpu::available());
}
+3 -7
View File
@@ -78,12 +78,7 @@ impl Headless {
None => vec![self.photo_or_active(p)?],
};
let dir = p.get("dir").and_then(Value::as_str).unwrap_or("");
let write = &mut |path: &str, bytes: &[u8]| {
if let Some(parent) = Path::new(path).parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(parent).map_err(|err| err.to_string())?;
}
std::fs::write(path, bytes).map_err(|err| format!("{path}: {err}"))
};
let write = &mut lightcraft_engine::export::write_file;
let files =
export_batch(&mut self.session, &ids, &opts, &Destination { dir: dir.to_string(), exact: exact.map(str::to_string) }, write, &|path| {
Path::new(path).exists()
@@ -118,6 +113,7 @@ impl Backend for Headless {
let img = self.render(&p, 1600)?;
match p.get("path").and_then(Value::as_str) {
Some(path) => {
self.session.check_write_target(path)?;
write_image(Path::new(path), &img, 92)?;
Ok(json!({"path": path, "width": img.width, "height": img.height}))
}
@@ -183,7 +179,7 @@ pub fn encode_image(ext: &str, img: &Rgba8, quality: u8) -> Result<Vec<u8>, Stri
pub fn write_image(path: &Path, img: &Rgba8, quality: u8) -> Result<(), String> {
let ext = path.extension().map(|e| e.to_string_lossy().to_string()).unwrap_or_default();
let bytes = encode_image(&ext, img, quality)?;
std::fs::write(path, bytes).map_err(|e| format!("{}: {e}", path.display()))
lightcraft_engine::export::write_file(&path.to_string_lossy(), &bytes)
}
#[cfg(test)]
+29 -1
View File
@@ -1,6 +1,6 @@
use serde_json::{Value, json};
use crate::{Headless, PROTOCOL_VERSION, Server, call_tool, command_tool_name};
use crate::{Backend, Headless, PROTOCOL_VERSION, Server, call_tool, command_tool_name};
fn server() -> Server {
Server::new(Box::new(Headless::demo()))
@@ -92,6 +92,34 @@ fn import_tool_copies_with_a_folder_template() {
let _ = std::fs::remove_dir_all(&base);
}
/// Issue #93: `render_photo`, `ui.render` and `app.export` with an exact `path` must not replace
/// a photo's original.
#[test]
fn path_writes_never_replace_an_original() {
let base = std::env::temp_dir().join(format!("lc-mcp-guard-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).unwrap();
let img = lightcraft_raster::Rgba8 { width: 8, height: 8, data: vec![[90, 30, 9, 255]; 64] };
let png = lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(&img), &lightcraft_codecs::EncodeMeta::default()).unwrap();
let orig = base.join("a.png");
std::fs::write(&orig, &png).unwrap();
let mut b = Headless::demo();
let r = call_tool(&mut b, "import", &json!({"paths": [orig.to_string_lossy()]}));
assert!(!r.is_error, "{r:?}");
let id = b.session.catalog.photos().find(|p| p.file_name == "a.png").unwrap().id.0;
let path = orig.to_string_lossy().to_string();
let r = call_tool(&mut b, "render_photo", &json!({"id": id, "path": path}));
assert!(r.is_error, "{r:?}");
assert!(b.call("ui.render", json!({"id": id, "path": path})).unwrap_err().contains("original"));
let e = b.call("app.export", json!({"ids": [id], "path": path})).unwrap_err();
assert!(e.contains("never writes over an original"), "{e}");
assert_eq!(std::fs::read(&orig).unwrap(), png, "the original is untouched");
// another path still works
let r = call_tool(&mut b, "render_photo", &json!({"id": id, "path": base.join("render.png").to_string_lossy()}));
assert!(!r.is_error, "{r:?}");
let _ = std::fs::remove_dir_all(&base);
}
/// The `import` helper moves: renamed into the folder template, the source removed.
#[test]
fn import_tool_moves_with_a_folder_template() {
+17 -3
View File
@@ -205,7 +205,7 @@ pub fn helper_tools(has_ui: bool) -> Vec<Value> {
"removeLocation": {"type": "boolean"},
"colorSpace": {"type": "string", "enum": ["srgb", "displayP3", "adobeRgb", "proPhoto", "rec2020"], "description": "Output colour space (default sRGB; AVIF is always sRGB). adobeRgb = Adobe RGB (1998) compatible; the embedded ICC profile is generated from the published primaries"},
"bitDepth": {"type": "integer", "enum": [8, 10, 16, 32], "description": "Bits per channel: PNG 8|16 (default 8), TIFF 8|16|32 (default 16; 32 = linear float with a linear profile), AVIF 8|10; JPEG/WebP are 8-bit"},
"watermark": {"description": "Text, or {text, size (fraction of short edge), opacity, anchor (topLeft|top|topRight|left|center|right|bottomLeft|bottom|bottomRight), inset, color [r,g,b], shadow}"}
"watermark": {"description": "Text, or {text, vertical (boolean; defaults to false), size (fraction of short edge), opacity, anchor (topLeft|top|topRight|left|center|right|bottomLeft|bottom|bottomRight), inset, color [r,g,b], shadow}"}
}),
&[],
),
@@ -352,9 +352,9 @@ fn image_result(file: &std::path::Path, max: Option<u32>, format: &str, save_to:
(bytes, w, h)
};
if let Some(p) = save_to
&& let Err(e) = std::fs::write(p, &bytes)
&& let Err(e) = lightcraft_engine::export::write_file(p, &bytes)
{
return ToolResult::error(format!("{p}: {e}"));
return ToolResult::error(e);
}
let mime = if jpeg { "image/jpeg" } else { "image/png" };
let mut info = meta;
@@ -372,7 +372,18 @@ fn image_result(file: &std::path::Path, max: Option<u32>, format: &str, save_to:
}
}
/// A user-given `path` to save to must not be a photo's original (or its sidecar).
fn check_save_path(b: &mut dyn Backend, args: &Value) -> Result<(), String> {
match args.get("path").and_then(Value::as_str) {
Some(p) => exec(b, "export.checkTarget", json!({"path": p})).map(|_| ()),
None => Ok(()),
}
}
fn render_photo(b: &mut dyn Backend, args: &Value) -> ToolResult {
if let Err(e) = check_save_path(b, args) {
return ToolResult::error(e);
}
let size = args.get("size").and_then(Value::as_u64).unwrap_or(1024).clamp(16, 4096);
let id = match args.get("id").and_then(Value::as_u64) {
Some(id) => Some(id),
@@ -394,6 +405,9 @@ fn render_photo(b: &mut dyn Backend, args: &Value) -> ToolResult {
}
fn screenshot(b: &mut dyn Backend, args: &Value) -> ToolResult {
if let Err(e) = check_save_path(b, args) {
return ToolResult::error(e);
}
let file = temp_path("screenshot");
let path = file.to_string_lossy().to_string();
match b.call("ui.screenshot", json!({"path": path})) {
+12 -1
View File
@@ -71,7 +71,7 @@ impl DateTime {
};
let (h, m) = match rest.split_once(':') {
Some((h, m)) => (num(h)?, num(m)?),
None if rest.len() == 4 => (num(&rest[..2])?, num(&rest[2..])?),
None if rest.len() == 4 => (num(rest.get(..2)?)?, num(rest.get(2..)?)?),
None => (num(rest)?, 0),
};
if h > 18 || m > 59 {
@@ -180,6 +180,17 @@ mod tests {
assert_eq!(DateTime::parse_offset("+99:00"), None);
}
#[test]
fn unicode_offsets_are_rejected_without_panicking() {
for offset in ["+1€", "-1€", "+€1", "-€1"] {
assert_eq!(DateTime::parse_offset(offset), None);
let date = format!("2024-06-01T13:45:59{offset}");
let metadata = crate::Metadata { capture_time: DateTime::parse_iso("2024-06-01T13:45:59Z"), ..Default::default() };
let xmp = crate::write_xmp(&metadata, None).replace("2024-06-01T13:45:59+00:00", &date);
assert!(crate::parse_xmp(&xmp).unwrap().metadata.capture_time.is_none());
}
}
#[test]
fn iso_roundtrip() {
for s in ["2024-06-01T13:45:59", "2024-06-01T13:45:59.250+02:00", "1999-12-31T23:59:00-05:30", "2024-02-29T00:00:00Z"] {
+2
View File
@@ -17,6 +17,7 @@ mod gpx;
mod iptc;
pub mod tags;
mod xmp;
mod xmp_merge;
pub use container::{Embedded, embedded, jpeg_segments, png_chunks, webp_chunks};
pub use datetime::DateTime;
@@ -26,6 +27,7 @@ pub use iptc::parse_iptc;
pub use lightcraft_geom::Orientation;
pub use tags::{TagRow, file_tag_rows, tag_rows};
pub use xmp::{CRS_NS, LC_NS, XmpData, XmpError, XmpValue, parse_xmp, write_xmp, write_xmp_lc};
pub use xmp_merge::{MergeRules, merge_xmp};
use serde::{Deserialize, Serialize};
+5
View File
@@ -145,6 +145,11 @@ impl Node {
const MAX_DEPTH: usize = 64;
const MAX_NODES: usize = 200_000;
/// `prefix:local` with the prefix canonicalised by namespace URI (see [`NAMESPACES`]).
pub(crate) fn canonical_name(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
canonical(qname, scopes)
}
fn canonical(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
let (prefix, local) = qname.split_once(':').unwrap_or(("", qname));
if prefix == "xml" || prefix == "xmlns" {
+403
View File
@@ -0,0 +1,403 @@
//! Merging a freshly written XMP packet into an existing one (a sidecar another application
//! wrote, e.g. with its develop settings and edit history), keeping everything the writer does not
//! own byte for byte.
//!
//! The existing packet is scanned once (namespace-resolved, with byte positions). Only top-level
//! properties of the `rdf:Description` elements directly under `rdf:RDF` are considered, in
//! attribute form (`xmp:Rating="3"`) and element form (`<dc:subject>…</dc:subject>`):
//! - properties the writer owns ([`MergeRules::owned`]) are removed, and written from the fresh
//! packet when it has them (so a cleared field is cleared in the file too);
//! - properties owned only when stated ([`MergeRules::owned_if_present`]) are replaced when the
//! fresh packet has them and kept otherwise;
//! - any other property of the fresh packet (e.g. `xmp:CreatorTool`) is not written: the file
//! keeps describing itself as its creator left it;
//! - everything else — other namespaces (e.g. `crs:`), `xmpMM:History`, unknown properties,
//! comments, the packet wrapper and padding — is copied unchanged.
//!
//! The written properties go into one new `rdf:Description` (declaring its own namespaces, so
//! the existing prefix bindings never matter) just before `</rdf:RDF>`. A description left empty
//! by the removals is dropped, so repeated saves don't accumulate. The result is checked by
//! parsing it again: every written property must read back as in the fresh packet.
use std::collections::BTreeSet;
use quick_xml::events::Event;
use crate::xmp::{XmpError, canonical_name, parse_xmp};
/// Which properties [`merge_xmp`] takes from the fresh packet. Names are canonical
/// `prefix:name` (prefixes as [`crate::parse_xmp`] reports them); `prefix:*` is a whole namespace.
#[derive(Clone, Copy, Debug, Default)]
pub struct MergeRules<'a> {
/// Always replaced: removed from the existing packet, written from the fresh one if stated.
pub owned: &'a [&'a str],
/// Replaced only when the fresh packet states them; otherwise the existing value stays.
pub owned_if_present: &'a [&'a str],
}
fn listed(list: &[&str], name: &str) -> bool {
list.iter().any(|p| match p.strip_suffix('*') {
Some(ns) => name.starts_with(ns),
None => *p == name,
})
}
#[derive(Debug)]
struct Attr {
raw: String,
name: String,
/// The value as written (still escaped).
value: String,
}
#[derive(Debug)]
struct Prop {
name: String,
start: usize,
end: usize,
}
#[derive(Debug, Default)]
struct Desc {
tag_start: usize,
tag_end: usize,
/// The element name as written (`rdf:Description`, or another prefix for the RDF namespace).
raw_name: String,
empty: bool,
attrs: Vec<Attr>,
props: Vec<Prop>,
/// The end of `</rdf:Description>` (`tag_end` when self-closing).
end: usize,
}
#[derive(Debug, Default)]
struct Scan {
descs: Vec<Desc>,
/// Where the first `</rdf:RDF>` starts.
rdf_close: Option<usize>,
}
enum Kind {
Other,
Rdf,
Desc(usize),
Prop { desc: usize, start: usize, name: String },
}
fn is_meta_attr(name: &str) -> bool {
name.starts_with("xmlns")
|| matches!(name, "rdf:about" | "rdf:ID" | "rdf:nodeID" | "xml:lang" | "rdf:parseType" | "rdf:resource" | "rdf:datatype")
}
fn pos(r: &quick_xml::Reader<&[u8]>) -> usize {
usize::try_from(r.buffer_position()).unwrap_or(usize::MAX)
}
/// Locate the descriptions, their top-level properties and `</rdf:RDF>` in a packet.
fn scan(s: &str) -> Result<Scan, XmpError> {
let mut reader = quick_xml::Reader::from_reader(s.as_bytes());
reader.config_mut().check_end_names = true;
let mut out = Scan::default();
let mut stack: Vec<Kind> = Vec::new();
let mut scopes: Vec<Vec<(String, String)>> = vec![vec![("xml".into(), "http://www.w3.org/XML/1998/namespace".into())]];
let mut count = 0usize;
loop {
let before = pos(&reader);
let ev = reader.read_event().map_err(|e| XmpError::Xml(e.to_string()))?;
let after = pos(&reader);
match ev {
Event::Start(ref e) | Event::Empty(ref e) => {
count += 1;
if count > 200_000 || stack.len() > 64 {
return Err(XmpError::TooComplex);
}
let empty = matches!(ev, Event::Empty(_));
let mut scope = Vec::new();
let mut raw_attrs = Vec::new();
for a in e.attributes() {
let a = a.map_err(|e| XmpError::Xml(e.to_string()))?;
let k = String::from_utf8_lossy(a.key.as_ref()).into_owned();
let v = String::from_utf8_lossy(&a.value).into_owned();
if k == "xmlns" {
scope.push((String::new(), v.clone()));
} else if let Some(p) = k.strip_prefix("xmlns:") {
scope.push((p.to_string(), v.clone()));
}
raw_attrs.push((k, v));
}
scopes.push(scope);
let raw_name = String::from_utf8_lossy(e.name().as_ref()).into_owned();
let name = canonical_name(&raw_name, &scopes);
let kind = match stack.last() {
Some(Kind::Rdf) if name == "rdf:Description" => {
let attrs = raw_attrs.into_iter().map(|(raw, value)| Attr { name: canonical_name(&raw, &scopes), raw, value }).collect();
out.descs.push(Desc { tag_start: before, tag_end: after, raw_name, empty, attrs, props: Vec::new(), end: after });
Kind::Desc(out.descs.len() - 1)
}
Some(Kind::Desc(i)) => Kind::Prop { desc: *i, start: before, name },
_ if name == "rdf:RDF" && out.rdf_close.is_none() => Kind::Rdf,
_ => Kind::Other,
};
if empty {
scopes.pop();
if let Kind::Prop { desc, start, name } = kind
&& let Some(d) = out.descs.get_mut(desc)
{
d.props.push(Prop { name, start, end: after });
}
} else {
stack.push(kind);
}
}
Event::End(_) => {
scopes.pop();
match stack.pop() {
Some(Kind::Prop { desc, start, name }) => {
if let Some(d) = out.descs.get_mut(desc) {
d.props.push(Prop { name, start, end: after });
}
}
Some(Kind::Desc(i)) => {
if let Some(d) = out.descs.get_mut(i) {
d.end = after;
}
}
Some(Kind::Rdf) => out.rdf_close = out.rdf_close.or(Some(before)),
Some(Kind::Other) => {}
None => return Err(XmpError::Xml("unbalanced end tag".into())),
}
}
Event::Eof => break,
_ => {}
}
}
if !stack.is_empty() {
return Err(XmpError::Xml("unclosed element".into()));
}
Ok(out)
}
/// `start` moved back over the indentation and line break before it (so a removed property
/// takes its line with it).
fn line_start(s: &str, start: usize) -> usize {
let b = s.as_bytes();
let mut i = start;
while i > 0 && matches!(b.get(i - 1), Some(b' ' | b'\t')) {
i -= 1;
}
if i > 0 && b.get(i - 1) == Some(&b'\n') {
i -= 1;
if i > 0 && b.get(i - 1) == Some(&b'\r') {
i -= 1;
}
}
i
}
/// Merge `fresh` (a packet as [`crate::write_xmp_lc`] writes it) into `existing` following
/// `rules` (see the module docs). Fails — and the caller must not overwrite `existing` blindly —
/// when `existing` isn't a well-formed XMP packet.
pub fn merge_xmp(existing: &str, fresh: &str, rules: MergeRules) -> Result<String, XmpError> {
let (bom, old) = match existing.strip_prefix('\u{feff}') {
Some(rest) => ("\u{feff}", rest),
None => ("", existing),
};
parse_xmp(old)?;
let ex = scan(old)?;
let rdf_close = ex.rdf_close.ok_or_else(|| XmpError::Xml("no rdf:RDF element".into()))?;
let new = scan(fresh)?;
let fresh_desc = new.descs.first().ok_or_else(|| XmpError::Xml("the new packet has no rdf:Description".into()))?;
let fresh_names: BTreeSet<&str> = fresh_desc.props.iter().map(|p| p.name.as_str()).collect();
let remove = |name: &str| listed(rules.owned, name) || (listed(rules.owned_if_present, name) && fresh_names.contains(name));
let write = |name: &str| listed(rules.owned, name) || listed(rules.owned_if_present, name);
// edits on `old`: (start, end, replacement), applied back to front
let mut edits: Vec<(usize, usize, String)> = Vec::new();
for d in &ex.descs {
let gone_attrs: Vec<&Attr> = d.attrs.iter().filter(|a| !is_meta_attr(&a.name) && remove(&a.name)).collect();
let gone_props: Vec<&Prop> = d.props.iter().filter(|p| remove(&p.name)).collect();
if gone_attrs.is_empty() && gone_props.is_empty() {
continue;
}
let attrs_left = d.attrs.iter().any(|a| !is_meta_attr(&a.name) && !remove(&a.name));
if !attrs_left && gone_props.len() == d.props.len() {
// nothing of its own left: drop the whole description
edits.push((line_start(old, d.tag_start), d.end, String::new()));
continue;
}
if !gone_attrs.is_empty() {
let mut tag = format!("<{}", d.raw_name);
for a in d.attrs.iter().filter(|a| is_meta_attr(&a.name) || !remove(&a.name)) {
tag.push_str(&format!("\n {}=\"{}\"", a.raw, a.value.replace('"', "&quot;")));
}
tag.push_str(if d.empty { "/>" } else { ">" });
edits.push((d.tag_start, d.tag_end, tag));
}
for p in gone_props {
edits.push((line_start(old, p.start), p.end, String::new()));
}
}
// the new description: its own namespace declarations, the properties to write
let mut block = String::from("\n <rdf:Description rdf:about=\"\"\n xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"");
for a in fresh_desc.attrs.iter().filter(|a| a.raw.starts_with("xmlns:") && a.raw != "xmlns:rdf") {
block.push_str(&format!("\n {}=\"{}\"", a.raw, a.value));
}
block.push('>');
let mut written = 0;
for p in fresh_desc.props.iter().filter(|p| write(&p.name)) {
let text = fresh.get(p.start..p.end).ok_or_else(|| XmpError::Xml("bad property span".into()))?;
block.push_str("\n ");
block.push_str(text);
written += 1;
}
block.push_str("\n </rdf:Description>");
if written > 0 {
let at = old.get(..rdf_close).map_or(rdf_close, |h| h.trim_end().len());
edits.push((at, at, block));
}
edits.sort_by_key(|e| (e.0, e.1));
// overlapping edits can't happen (properties sit inside their description, which is either
// dropped whole or edited piecewise), but never splice blindly
let mut out = String::with_capacity(old.len() + 4096);
out.push_str(bom);
let mut at = 0;
for (s, e, r) in &edits {
if *s < at {
return Err(XmpError::Xml("overlapping edits".into()));
}
out.push_str(old.get(at..*s).ok_or_else(|| XmpError::Xml("bad span".into()))?);
out.push_str(r);
at = *e;
}
out.push_str(old.get(at..).ok_or_else(|| XmpError::Xml("bad span".into()))?);
// read back: everything written must come out as written
let merged = parse_xmp(&out)?;
let want = parse_xmp(fresh)?;
for (k, v) in &want.properties {
let top = k.split('/').next().unwrap_or(k);
if write(top) && merged.properties.get(k) != Some(v) {
return Err(XmpError::Xml(format!("{k} did not survive the merge")));
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
const RULES: MergeRules =
MergeRules { owned: &["xmp:Rating", "xmp:Label", "dc:subject", "dc:title", "lc:*"], owned_if_present: &["exif:DateTimeOriginal"] };
/// A sidecar as another raw developer writes it: attribute-form simple properties, develop
/// settings, a structured history, an unknown namespace.
const FOREIGN: &str = "<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>
<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">
<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">
<rdf:Description rdf:about=\"\"
xmlns:xmp=\"http://ns.adobe.com/xap/1.0/\"
xmlns:xmpMM=\"http://ns.adobe.com/xap/1.0/mm/\"
xmlns:stEvt=\"http://ns.adobe.com/xap/1.0/sType/ResourceEvent#\"
xmlns:crs=\"http://ns.adobe.com/camera-raw-settings/1.0/\"
xmlns:dc=\"http://purl.org/dc/elements/1.1/\"
xmlns:foo=\"http://example.com/foo/\"
xmp:Rating=\"2\"
xmp:CreatorTool=\"Other App\"
crs:Exposure2012=\"+0.50\"
crs:Contrast2012=\"-12\"
foo:Thing=\"a &amp; b\">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
<dc:subject>
<rdf:Bag>
<rdf:li>old</rdf:li>
</rdf:Bag>
</dc:subject>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
";
/// With the packet's whitespace padding and trailer.
fn foreign() -> String {
format!("{FOREIGN}{}\n<?xpacket end=\"w\"?>", " ".repeat(100))
}
fn fresh(rating: i8, keywords: &[&str]) -> String {
let m = crate::Metadata {
software: Some("LightCraft".into()),
rating: Some(rating),
keywords: keywords.iter().map(|k| k.to_string()).collect(),
..Default::default()
};
crate::write_xmp_lc(&m, &[("settings", "{\"a\":1}"), ("flag", "pick")])
}
#[test]
fn keeps_foreign_properties_and_replaces_owned_ones() {
let out = merge_xmp(&foreign(), &fresh(4, &["new", "two"]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
let p = |k: &str| d.properties.get(k).cloned().unwrap_or_default();
assert_eq!(p("xmp:Rating"), vec!["4"]);
assert_eq!(p("dc:subject"), vec!["new", "two"]);
assert_eq!(p("lc:settings"), vec!["{\"a\":1}"]);
assert_eq!(p("lc:flag"), vec!["pick"]);
assert_eq!(p("xmp:CreatorTool"), vec!["Other App"], "not ours: kept");
assert_eq!(p("crs:Exposure2012"), vec!["+0.50"]);
assert_eq!(p("foo:Thing").len(), 1);
// untouched parts are byte for byte the same
for part in [
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>",
"crs:Contrast2012=\"-12\"",
"foo:Thing=\"a &amp; b\"",
"<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>\n<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">",
&format!("</x:xmpmeta>\n{}\n<?xpacket end=\"w\"?>", " ".repeat(100)),
] {
assert!(out.contains(part), "lost {part:?} in\n{out}");
}
assert!(!out.contains("xmp:Rating=\"2\""));
assert!(!out.contains("<rdf:li>old</rdf:li>"));
// saving again changes only our part, and doesn't pile up descriptions
let again = merge_xmp(&out, &fresh(1, &[]), RULES).unwrap();
let d = parse_xmp(&again).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["1".to_string()]));
assert!(!d.properties.contains_key("dc:subject"), "a cleared field is cleared");
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert_eq!(merge_xmp(&again, &fresh(1, &[]), RULES).unwrap(), again, "idempotent");
}
#[test]
fn owned_if_present_and_prefix_independence() {
// the existing packet binds the namespaces to other prefixes, and has a capture time
let ex = "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><r:RDF xmlns:r=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\
<r:Description r:about=\"\" xmlns:a=\"http://ns.adobe.com/xap/1.0/\" xmlns:e=\"http://ns.adobe.com/exif/1.0/\" \
a:Rating=\"5\" e:DateTimeOriginal=\"2020-01-01T00:00:00\"/></r:RDF></x:xmpmeta>";
let out = merge_xmp(ex, &fresh(3, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["3".to_string()]));
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2020-01-01T00:00:00".to_string()]), "fresh has none: kept");
let mut m = crate::Metadata { rating: Some(3), ..Default::default() };
m.capture_time = crate::DateTime::parse_iso("2024-05-06T07:08:09");
let out = merge_xmp(ex, &crate::write_xmp_lc(&m, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2024-05-06T07:08:09".to_string()]));
}
#[test]
fn refuses_what_it_cannot_read() {
for bad in ["", "not xml at all", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><rdf:RDF", "<a><b></a>", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>"] {
assert!(merge_xmp(bad, &fresh(1, &[]), RULES).is_err(), "{bad:?}");
}
}
}
+18 -1
View File
@@ -1,4 +1,4 @@
use lightcraft_meta::{Metadata, extract, parse_iptc, parse_xmp, read_exif, write_xmp};
use lightcraft_meta::{MergeRules, Metadata, extract, merge_xmp, parse_iptc, parse_xmp, read_exif, write_xmp};
use proptest::prelude::*;
proptest! {
@@ -21,6 +21,23 @@ proptest! {
let _ = parse_xmp(&s);
}
/// Merging into arbitrary (also damaged) packets never panics, and a merge that succeeds
/// reads back with the written values.
#[test]
fn merge_never_panics(cut in 0usize..900, junk in ".{0,20}", rating in 0i8..=5) {
let base = write_xmp(&Metadata { title: Some("t".into()), rating: Some(1), ..Default::default() }, Some("{}"));
let mut s: String = base.chars().take(cut).collect();
s.push_str(&junk);
s.extend(base.chars().skip(cut));
let fresh = write_xmp(&Metadata { rating: Some(rating), ..Default::default() }, Some("{\"x\":1}"));
let rules = MergeRules { owned: &["xmp:Rating", "dc:title", "lc:*"], owned_if_present: &[] };
if let Ok(out) = merge_xmp(&s, &fresh, rules) {
let d = parse_xmp(&out).unwrap();
prop_assert_eq!(d.metadata.rating, Some(rating));
prop_assert_eq!(d.lc_settings.as_deref(), Some("{\"x\":1}"));
}
}
#[test]
fn user_text_roundtrips(title in "[a-zA-Z0-9 &<>\"'éü€;|/\\\\]{1,40}", kw in proptest::collection::vec("[a-zA-Z0-9 &<>\"'|]{1,12}", 0..5), rating in -1i8..=5) {
let title = title.trim().to_string();
+41 -1
View File
@@ -4,6 +4,11 @@
//! files) but not fsynced; unreadable files are deleted and re-rendered. When the total size
//! exceeds the budget, the least recently used files (by modification time, refreshed on read)
//! are removed down to 80 % of the budget.
//!
//! The cache only ever lists, counts or deletes files of its own name shape
//! (`<2 hex>/<32 hex>.jpg` and the `<32 hex>.tmp…` temp files of [`DiskCache::put`], see
//! [`is_cache_file`]): the directory may be shared with other files (e.g. a library opened on a
//! photo folder that already has a `thumbs/` folder), and those are never touched (issue #98).
use std::path::{Path, PathBuf};
use std::sync::Mutex;
@@ -96,13 +101,25 @@ impl DiskCache {
}
}
/// All cache files: (path, size, modified).
/// All cache files (only names of the cache's own shape, see [`is_cache_file`]):
/// (path, size, modified).
fn scan(&self) -> Vec<(PathBuf, u64, std::time::SystemTime)> {
let mut out = Vec::new();
let Ok(rd) = std::fs::read_dir(&self.dir) else { return out };
for sub in rd.flatten() {
let sub_name = sub.file_name();
let Some(sub_name) = sub_name.to_str() else { continue };
if !is_shard_name(sub_name) {
continue;
}
let Ok(files) = std::fs::read_dir(sub.path()) else { continue };
for f in files.flatten() {
let name = f.file_name();
let Some(name) = name.to_str() else { continue };
if !is_cache_file(sub_name, name) {
continue;
}
// `DirEntry::metadata` does not follow symlinks: a link is never a cache file
if let Ok(m) = f.metadata()
&& m.is_file()
{
@@ -144,6 +161,29 @@ impl DiskCache {
}
}
fn is_lower_hex(s: &str) -> bool {
s.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b))
}
/// A shard directory name: two lowercase hex digits.
fn is_shard_name(s: &str) -> bool {
s.len() == 2 && is_lower_hex(s)
}
/// Whether `name` inside shard directory `shard` is a file this cache wrote: `<32 hex>.jpg`
/// (whose first two digits are the shard) or a `<32 hex>.tmp<alphanumeric>` temp file of
/// [`DiskCache::put`]. Anything else in the cache directory belongs to someone else.
pub fn is_cache_file(shard: &str, name: &str) -> bool {
if !is_shard_name(shard) {
return false;
}
let Some((stem, ext)) = name.split_once('.') else { return false };
if stem.len() != 32 || !is_lower_hex(stem) || !stem.starts_with(shard) {
return false;
}
ext == "jpg" || ext.strip_prefix("tmp").is_some_and(|t| !t.is_empty() && t.bytes().all(|b| b.is_ascii_alphanumeric()))
}
/// Encode a thumbnail as the cache stores it (JPEG, RGB, 4:2:0).
pub fn encode_jpeg(img: &Rgba8) -> Option<Vec<u8>> {
if img.width == 0 || img.height == 0 || img.width > u16::MAX as usize || img.height > u16::MAX as usize {
+38
View File
@@ -154,3 +154,41 @@ fn pool_runs_inline() {
assert_eq!(p.run_inline(10), 1);
assert_eq!(p.try_recv().map(|d| d.result), Some(1));
}
#[test]
fn disk_cache_never_touches_foreign_files() {
// issue #98: a library opened on a folder that already has a `thumbs/` folder
let dir = temp_dir("foreign");
let foreign = [
dir.join("readme.txt"),
dir.join("ab").join("holiday.jpg"),
dir.join("ab").join("0123456789abcdef0123456789abcdef.jpg"), // wrong shard
dir.join("small").join("0123456789abcdef0123456789abcdef.jpg"),
dir.join("01").join("0123456789ABCDEF0123456789ABCDEF.jpg"), // upper case
dir.join("01").join("0123456789abcdef0123456789abcdef.jpeg"),
dir.join("01").join("0123456789abcdef0123456789abcdef.tmp"),
];
for p in &foreign {
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, vec![7u8; 50_000]).unwrap();
}
// a stale temp file of our own is cleaned up
let own_tmp = dir.join("01").join("0123456789abcdef0123456789abcdef.tmpThreadId9");
std::fs::write(&own_tmp, b"partial").unwrap();
let d = DiskCache::new(&dir, 40_000);
assert_eq!(d.size(), 7, "only the cache's own temp file counts");
// fill past the budget (forces prunes), then clear
for i in 0..40u8 {
d.put(hash_bytes(&[i, 2]), &gradient(160, 100, i));
}
assert!(d.size() <= 40_000);
d.clear();
assert_eq!(d.size(), 0);
assert!(!own_tmp.exists());
for p in &foreign {
assert!(p.exists(), "foreign file deleted: {}", p.display());
}
assert!(disk::is_cache_file("01", "0123456789abcdef0123456789abcdef.jpg"));
assert!(!disk::is_cache_file("01", "0123456789abcdef0123456789abcdef.jpg.bak"));
let _ = std::fs::remove_dir_all(&dir);
}
+3 -3
View File
@@ -11,10 +11,10 @@
//! and produces camera → linear Rec.2020 D65 matrices.
//!
//! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw),
//! Canon CR2, Nikon NEF/NRW (uncompressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
//! and X-Trans), Panasonic RW2 (packed 12/14-bit), Pentax PEF (uncompressed, Huffman), Olympus ORF (uncompressed).
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon Huffman NEF, Panasonic
//! quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
//! [`embedded_preview`] covers all of them plus CR3. Variants we can't decode yet (Nikon "lossy after split" NEF,
//! Panasonic quantised RW2, compressed ORF/RAF, CR3) return [`RawError::Unsupported`]; each vendor module documents its sources
//! (public specifications, tag-name documentation, black-box analysis of CC0 samples) and gaps. Non-DNG files carry no
//! colour matrix: [`color`] falls back to a documented neutral model. The decoders never panic on malformed input.
#![forbid(unsafe_code)]
+33 -1
View File
@@ -7,6 +7,11 @@
//! - The raw data is IFD#3's single strip: a lossless-JPEG frame (usually 2 or 4 components).
//! - The decoded sample stream fills vertical slices left to right: `cr2_slice = [n, w, last]` means `n` slices of
//! width `w` then one of width `last`; each slice is filled top to bottom, row by row.
//! - Colour filter layout: IFD#3 tag `0xc5e0` (`CR2CFAPattern` in the ExifTool EXIF tag-name docs): 1 = RGGB,
//! 2 = BGGR, 3 = GBRG, 4 = GRBG, anchored at the top-left of the full decoded sensor (masked borders included).
//! It differs by model (e.g. 3 on the 50D/60D/7D/550D/5D Mark II, 1 on the 40D/5D Mark III/6D/5DS R), so it is
//! read per file (no model table); files without the tag fall back to RGGB (issue #85). The parity of the
//! `SensorInfo` borders does *not* predict it (RGGB files come with both even and odd top borders).
//! - `SensorInfo` (maker note `0x00e0`): sensor width/height and the left/top/right/bottom borders of the image
//! area; the masked columns left of it give the black level.
//! - `ColorBalance` (maker note `0x4001`): as-shot `RGGB` levels at a model-dependent offset; we probe the known
@@ -21,6 +26,7 @@ use lightcraft_tiff::{Ifd, Tiff, makernote, tags as t};
const CR2_SLICE: u16 = 0xc640;
const SRAW_TYPE: u16 = 0xc6c5;
const CR2_CFA_PATTERN: u16 = 0xc5e0;
const SENSOR_INFO: u16 = 0x00e0;
const COLOR_BALANCE: u16 = 0x4001;
@@ -28,6 +34,18 @@ fn raw_ifd(tiff: &Tiff) -> Option<&Ifd> {
tiff.ifds.get(3).or_else(|| tiff.ifds.iter().rev().find(|i| i.contains(CR2_SLICE)))
}
/// Bayer layout from the raw IFD's `CR2CFAPattern` (`0xc5e0`) value; `None` for missing/unknown values.
fn cfa_from_tag(v: Option<u64>) -> Option<Cfa> {
let name = match v? {
1 => "RGGB",
2 => "BGGR",
3 => "GBRG",
4 => "GRBG",
_ => return None,
};
Some(Cfa::bayer_static(name))
}
/// As-shot WB multipliers (R, G, B; G = 1) from the ColorBalance array.
fn wb_from_color_balance(v: &[u64]) -> Option<[f32; 3]> {
for off in [63usize, 25, 24, 34, 71, 85, 105, 69, 77] {
@@ -117,7 +135,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode) -> Result<RawImage> {
}
}
let wb = mn.as_ref().and_then(|m| m.ifd.u64s(COLOR_BALANCE)).and_then(|v| wb_from_color_balance(&v));
let cfa = Cfa::bayer_static("RGGB");
let cfa = cfa_from_tag(raw.u64(CR2_CFA_PATTERN)).unwrap_or_else(|| Cfa::bayer_static("RGGB"));
let black = if active.x >= 8 {
black_from_columns(&data, width, 2..active.x - 2, active.y..active.y + active.height, active)
} else {
@@ -185,6 +203,7 @@ pub(crate) mod tests {
ifd0.set_child(t::EXIF_IFD, exif);
let mut ifd3 = IfdBuilder::new();
ifd3.set(t::COMPRESSION, Value::Short(vec![6]));
ifd3.set(CR2_CFA_PATTERN, Value::Long(vec![3]));
if let Some(s) = slices {
ifd3.set(CR2_SLICE, Value::Short(s.to_vec()));
}
@@ -206,11 +225,24 @@ pub(crate) mod tests {
assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
assert_eq!(r.data, RawData::U16(img));
assert_eq!(r.orientation, Orientation::Rotate270);
assert_eq!(r.cfa.as_ref().map(Cfa::name).as_deref(), Some("GBRG"), "CR2CFAPattern 3");
assert_eq!(r.metadata.iso, Some(400));
assert!(r.develop(crate::Method::Ahd).is_ok());
}
}
#[test]
fn cfa_pattern_tag() {
let name = |v| cfa_from_tag(v).map(|c| c.name());
assert_eq!(name(Some(1)).as_deref(), Some("RGGB"));
assert_eq!(name(Some(2)).as_deref(), Some("BGGR"));
assert_eq!(name(Some(3)).as_deref(), Some("GBRG"));
assert_eq!(name(Some(4)).as_deref(), Some("GRBG"));
assert_eq!(name(Some(0)), None);
assert_eq!(name(Some(7)), None);
assert_eq!(name(None), None);
}
#[test]
fn wb_probe() {
let mut v = vec![0u64; 120];
+1
View File
@@ -3,6 +3,7 @@
pub mod arw;
pub mod cr2;
pub mod nef;
pub mod nefc;
pub mod orf;
pub mod pef;
pub mod raf;
+53 -28
View File
@@ -1,22 +1,22 @@
//! Nikon NEF / NRW — uncompressed variants only.
//! Nikon NEF / NRW — uncompressed and Huffman-compressed (lossless, lossy) variants.
//!
//! Sources: TIFF 6.0 (the raw image is a standard CFA SubIFD), Laurent Clévy's NEF structure notes (prose: IFD
//! layout, SubIFDs, maker note header) and the ExifTool Nikon tag-name documentation (`0x000c` WB_RBLevels,
//! `0x003d` BlackLevel). Nikon's Huffman-compressed variants (compression 34713: lossless, lossy type 1/2) are
//! **not** implemented: the only descriptions we found are derived from GPL code, which our clean-room rules
//! forbid. The coding is T.81-style (difference category + additional bits, as in Pentax PEF), but unlike PEF the
//! Huffman tables are not stored in the files and our black-box table search has not converged (see ROADMAP.md).
//! They are reported as [`RawError::Unsupported`]; their embedded previews still work.
//! `0x003d` BlackLevel, `0x0096` NEFLinearizationTable). The Huffman-compressed data (compression 34713) is decoded
//! by [`super::nefc`], which documents its clean-room sources; files it can't decode yet ("lossy after split")
//! are reported as [`RawError::Unsupported`] and their embedded previews still work.
use super::white_from_data;
use super::{nefc, white_from_data};
use crate::tiffraw::{Packing, read_image};
use crate::{BlackLevel, Cfa, ColorData, OpcodeLists, RawData, RawError, RawFormat, RawImage, Rect, Result};
use lightcraft_geom::Orientation;
use lightcraft_tiff::image::ImageInfo;
use lightcraft_tiff::tags::{self as t, photometric};
use lightcraft_tiff::{Ifd, Tiff, makernote};
use lightcraft_tiff::{ByteOrder, Ifd, Tiff, makernote};
const WB_RB_LEVELS: u16 = 0x000c;
const BLACK_LEVEL: u16 = 0x003d;
const LINEARIZATION_TABLE: u16 = 0x0096;
fn raw_ifd(tiff: &Tiff) -> Option<&Ifd> {
tiff.all_ifds()
@@ -47,30 +47,15 @@ pub(crate) fn decode(bytes: &[u8]) -> Result<RawImage> {
let ifd0 = &tiff.ifds[0];
let raw = raw_ifd(&tiff).ok_or_else(|| RawError::Unsupported("NEF without a CFA image IFD".into()))?;
let info = raw.image()?;
if info.compression == t::compression::NIKON {
return Err(RawError::Unsupported("Nikon Huffman-compressed NEF (no clean-room description available)".into()));
}
let (w, h) = (info.width as usize, info.height as usize);
let bits = info.bits() as u32;
let row_samples = w * info.samples_per_pixel as usize;
let chunks = info.chunks(bytes.len() as u64);
let rows_in_first = chunks.first().map(|c| c.height as usize).unwrap_or(h).max(1);
let bytes_per_row = chunks.first().map(|c| c.len as usize / rows_in_first).unwrap_or(0);
let packing = if bytes_per_row >= row_samples * 2 {
Packing::Word16
} else if bits == 12 && bytes_per_row * 8 >= row_samples * 12 && bytes_per_row * 8 < row_samples * 13 {
Packing::Msb
} else if info.compression == 1 && bits != 8 && bits != 16 {
return Err(RawError::Unsupported(format!("NEF uncompressed packing ({bytes_per_row} bytes per {row_samples}-sample row)")));
} else {
Packing::Msb
};
let data = read_image(bytes, &info, tiff.order, packing)?;
let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) };
let make = ifd0.string(t::MAKE).unwrap_or_default();
let mn =
tiff.exif().and_then(|e| e.get(t::MAKER_NOTE)).and_then(|e| makernote::parse_makernote(bytes, e.offset, e.count() as u64, tiff.order, &make));
let data =
if info.compression == t::compression::NIKON { compressed(bytes, &info, mn.as_ref())? } else { uncompressed(bytes, &info, tiff.order)? };
let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) };
let black = match mn.as_ref().and_then(|m| m.ifd.f64s(BLACK_LEVEL)).as_deref() {
Some([a, b, c, d]) if [a, b, c, d].iter().all(|v| **v < 16384.0) => {
BlackLevel { repeat_rows: 2, repeat_cols: 2, values: vec![*a as f32, *b as f32, *c as f32, *d as f32], ..Default::default() }
@@ -114,6 +99,46 @@ pub(crate) fn decode(bytes: &[u8]) -> Result<RawImage> {
Ok(img)
}
/// Uncompressed strips: 16-bit words or 12-bit MSB-packed, told apart by the strip size.
fn uncompressed(bytes: &[u8], info: &ImageInfo, order: ByteOrder) -> Result<RawData> {
let (w, h) = (info.width as usize, info.height as usize);
let bits = info.bits() as u32;
let row_samples = w * info.samples_per_pixel as usize;
let chunks = info.chunks(bytes.len() as u64);
let rows_in_first = chunks.first().map(|c| c.height as usize).unwrap_or(h).max(1);
let bytes_per_row = chunks.first().map(|c| c.len as usize / rows_in_first).unwrap_or(0);
let packing = if bytes_per_row >= row_samples * 2 {
Packing::Word16
} else if bits == 12 && bytes_per_row * 8 >= row_samples * 12 && bytes_per_row * 8 < row_samples * 13 {
Packing::Msb
} else if info.compression == 1 && bits != 8 && bits != 16 {
return Err(RawError::Unsupported(format!("NEF uncompressed packing ({bytes_per_row} bytes per {row_samples}-sample row)")));
} else {
Packing::Msb
};
read_image(bytes, info, order, packing)
}
/// Nikon Huffman-compressed strip (see [`super::nefc`]); the decode table is maker note `0x0096`.
fn compressed(bytes: &[u8], info: &ImageInfo, mn: Option<&makernote::MakerNote>) -> Result<RawData> {
// (some Z bodies label uncompressed, row-padded data 34713 without a table: not handled here yet)
let table = mn
.and_then(|m| Some((m.ifd.bytes(LINEARIZATION_TABLE)?, m.order)))
.ok_or_else(|| RawError::Unsupported("Nikon compressed NEF without a linearization table (maker note 0x96)".into()))?;
if info.samples_per_pixel != 1 {
return Err(RawError::Unsupported(format!("Nikon compressed NEF with {} samples per pixel", info.samples_per_pixel)));
}
let bits = info.bits() as u32;
let table = nefc::parse_table(table.0, table.1, bits)?;
// one strip in every sample; several would be contiguous parts of the same bit stream
let chunks = info.chunks(bytes.len() as u64);
let (Some(first), Some(last)) = (chunks.first(), chunks.last()) else { return Err(RawError::Corrupt("NEF: no image data".into())) };
let end = last.offset.saturating_add(last.len).min(bytes.len() as u64);
let src = usize::try_from(first.offset).ok().zip(usize::try_from(end).ok()).and_then(|(a, b)| bytes.get(a..b));
let src = src.ok_or_else(|| RawError::Corrupt("NEF: image data outside the file".into()))?;
Ok(RawData::U16(nefc::decode(src, info.width as usize, info.height as usize, bits, &table)?))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -159,7 +184,7 @@ mod tests {
}
#[test]
fn compressed_is_unsupported() {
fn compressed_without_table_is_unsupported() {
let bytes = nef(34713, 14, vec![vec![0; 64]], 8, 8, 8);
assert!(matches!(crate::decode(&bytes), Err(RawError::Unsupported(_))));
}
+481
View File
@@ -0,0 +1,481 @@
//! Nikon Huffman-compressed NEF image data (TIFF compression 34713): "lossless compressed" and "lossy compressed"
//! (type 1 and type 2) at 12 and 14 bits.
//!
//! **Clean-room.** No raw-decoder source code (dcraw, LibRaw, rawspeed, rawloader/rawler, darktable, RawTherapee,
//! libopenraw, ExifTool's Perl code, …) was read or consulted. What we used:
//!
//! - Prose format descriptions: Laurent Clévy, "Nikon Electronic File (NEF) format" (<http://lclevy.free.fr/nef/>),
//! the maker-note tag table and the "0x96 (linearization table) tag format" table (version bytes, four `u16`
//! predictor seeds at offset 2, curve size at 10, curve at 12, split value at 562 for lossy type 2, "lossy type
//! 2 reads an incomplete table and interpolation is required"); Bill Claff, "NEF Compression"
//! (<https://www.photonstophotos.net/NikonInfo/NEF_Compression.htm>): a lossy encoding curve followed by a
//! non-adaptive (fixed-table) Huffman stage, and the number of encoded values per camera (683, 769, 1025, 2753,
//! 3073, 4097 …), which confirmed our curve interpolation; the libopenraw format notes
//! (<https://libopenraw.freedesktop.org/formats/nef/>, prose only); ExifTool's Nikon tag-name documentation
//! (`0x0093` NEFCompression, `0x0096` NEFLinearizationTable); ITU-T T.81 Annex F/H for the "difference category
//! + additional bits" coding and canonical (BITS/HUFFVAL) Huffman codes.
//! - Our own black-box analysis of CC0 samples from raw.pixls.us. Nikon's Huffman tables are not stored in the
//! files and no prose source lists them, so they were recovered from the data: several cameras (D5100, D300,
//! D700) were shot with the same scene in compressed and uncompressed modes; a beam search assigned code words
//! to difference categories so that the decoded compressed image matched the uncompressed one (likelihood of
//! the decoded values given the reference, minus the bits consumed, plus a description-length penalty per new
//! code word). Each table was then verified on ~45 files from ~35 bodies (D40 … D850, Df, Z 6, Z 50, 1 J1): with
//! the right table the whole strip decodes and ends within the last byte of the data, with every value inside
//! the encoded range. The predictor and difference coding are the same as Pentax PEF (see `pef.rs`).
//!
//! Format, as established:
//! - One strip, MSB-first bit stream, no byte stuffing, continuous across rows. Per pixel a canonical Huffman code
//! gives the category `n`, followed by `n` additional bits (T.81: a leading 0 means a negative difference).
//! - Each pixel is predicted from the same-colour pixel two to the left; the first two pixels of a row from the
//! first two pixels of the previous row of the same parity, starting at the seeds of maker note `0x0096`.
//! - Lossless (`0x0096` version `0x46`): the decoded values are the samples. Lossy (version `0x44`): they index a
//! curve. Type 1 (`0x44 0x10`) stores the full curve; type 2 (`0x44 0x20`, `0x44 0x40`) stores 257 points,
//! one every `2^bits / 256` codes, linearly interpolated.
//! - Values in the maker note are in the maker note's byte order (newer bodies write little-endian notes).
//!
//! Not supported (returned as [`RawError::Unsupported`], the embedded preview is used instead):
//! - "Lossy after split" files (non-zero split row at `0x0096` offset 562): from that row on a different table
//! is used which our analysis has not recovered yet.
//! - Other `0x0096` versions, bit depths other than 12/14.
//! - Code words that never occurred in any sample (the lossy 12-bit 8-bit code word `11111110`, and code words
//! longer than 9 bits in the lossy tables) are rejected as corrupt; they cannot code a difference within the
//! lossy value range anyway.
use super::pef::{Bits, Huffman, diff};
use crate::{MAX_SAMPLES, RawError, Result};
use lightcraft_tiff::ByteOrder;
/// A table code word whose symbol was never observed (see the module docs).
const UNSEEN: u8 = u8::MAX;
/// Canonical Huffman table: number of codes of length 1..=16 and the symbols (difference categories) in code order.
struct Table {
counts: &'static [u8],
symbols: &'static [u8],
}
const LOSSLESS_12: Table = Table { counts: &[0, 1, 4, 2, 3, 1, 2], symbols: &[5, 4, 6, 3, 7, 2, 8, 1, 9, 0, 10, 11, 12] };
const LOSSLESS_14: Table = Table { counts: &[0, 1, 4, 2, 2, 3, 1, 2], symbols: &[7, 6, 8, 5, 9, 4, 10, 3, 11, 12, 2, 0, 1, 13, 14] };
const LOSSY_12: Table = Table { counts: &[0, 1, 5, 1, 1, 1, 1, 1, 1], symbols: &[5, 4, 3, 6, 2, 7, 1, 0, 8, 9, UNSEEN, 10] };
const LOSSY_14: Table = Table { counts: &[0, 1, 4, 3, 1, 1, 1, 1, 1], symbols: &[5, 6, 4, 7, 8, 3, 9, 2, 1, 0, 10, 11, 12] };
impl Table {
/// `(left-aligned 12-bit code, length)` per symbol, for [`Huffman::new`].
fn codes(&self) -> Vec<(u16, u8)> {
let mut out = vec![(0u16, 0u8); 17];
let mut code = 0u32;
let mut symbols = self.symbols.iter();
for (i, &n) in self.counts.iter().enumerate() {
let len = i as u32 + 1;
for _ in 0..n {
if let Some(&s) = symbols.next()
&& let Some(slot) = out.get_mut(s as usize)
{
*slot = ((code << (12 - len)) as u16, len as u8);
}
code += 1;
}
code <<= 1;
}
out
}
}
/// How the decoded values map to samples.
#[derive(Clone, Debug, PartialEq)]
pub(crate) enum Encoding {
/// The decoded values are the samples (`0..2^bits`).
Lossless,
/// The decoded values index this curve.
Lossy(Vec<u16>),
}
/// Maker note `0x0096` (NEF linearization table), parsed.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct DecodeTable {
pub encoding: Encoding,
/// Predictor seeds `[row parity][column]`.
pub seeds: [[i32; 2]; 2],
}
/// Parse maker note `0x0096` for a `bits`-bit image. `order` is the maker note's byte order.
pub(crate) fn parse_table(t: &[u8], order: ByteOrder, bits: u32) -> Result<DecodeTable> {
let u16_at = |o: usize| t.get(o..o + 2).and_then(|s| <[u8; 2]>::try_from(s).ok()).map(|b| order.u16(b));
let short = || RawError::Corrupt(format!("NEF: linearization table too short ({} bytes)", t.len()));
if bits != 12 && bits != 14 {
return Err(RawError::Unsupported(format!("Nikon compressed NEF with {bits}-bit samples")));
}
let (v0, v1) = (t.first().copied().ok_or_else(short)?, t.get(1).copied().ok_or_else(short)?);
let mut seeds = [[0i32; 2]; 2];
for (i, s) in seeds.iter_mut().flatten().enumerate() {
*s = u16_at(2 + 2 * i).ok_or_else(short)? as i32;
}
let encoding = match (v0, v1) {
(0x46, _) => Encoding::Lossless,
(0x44, 0x10 | 0x20 | 0x40) => {
let n = u16_at(10).ok_or_else(short)? as usize;
let points: Vec<u16> = (0..n).map(|i| u16_at(12 + 2 * i)).collect::<Option<_>>().ok_or_else(short)?;
if n < 2 {
return Err(RawError::Corrupt(format!("NEF: linearization curve of {n} points")));
}
if v1 == 0x10 {
Encoding::Lossy(points)
} else {
let split = u16_at(562).unwrap_or(0);
if split != 0 {
return Err(RawError::Unsupported(format!(
"Nikon \"lossy after split\" compressed NEF (split at row {split}) is not decoded yet"
)));
}
// type 2: `n` points, one every 2^bits / (n - 1) codes
let range = 1usize << bits;
if n - 1 > range || !range.is_multiple_of(n - 1) {
return Err(RawError::Unsupported(format!("NEF: {n}-point lossy curve for {bits}-bit data")));
}
let step = range / (n - 1);
let mut curve = Vec::with_capacity(range + 1);
for pair in points.windows(2) {
let &[a, b] = pair else { continue };
let (a, b) = (a as u32, b as u32);
for k in 0..step as u32 {
let s = step as u32;
curve.push(((a * (s - k) + b * k) / s) as u16);
}
}
curve.extend(points.last().copied());
Encoding::Lossy(curve)
}
}
_ => return Err(RawError::Unsupported(format!("Nikon compressed NEF version {v0:#04x} {v1:#04x}"))),
};
Ok(DecodeTable { encoding, seeds })
}
/// Decode a `w × h` Nikon Huffman-compressed strip `src` with `bits`-bit samples.
pub(crate) fn decode(src: &[u8], w: usize, h: usize, bits: u32, table: &DecodeTable) -> Result<Vec<u16>> {
if bits != 12 && bits != 14 {
return Err(RawError::Unsupported(format!("Nikon compressed NEF with {bits}-bit samples")));
}
let n = w.checked_mul(h).filter(|&n| n > 0 && n <= MAX_SAMPLES).ok_or(RawError::Limit("NEF image size"))?;
// every code word is at least 2 bits long: a strip that can't hold the image is truncated (also bounds the
// allocation by the file size)
if n / 4 > src.len() {
return Err(RawError::Corrupt(format!("NEF: {} bytes of compressed data for {w}x{h} pixels", src.len())));
}
let huff = Huffman::new(
&match (&table.encoding, bits) {
(Encoding::Lossless, 12) => LOSSLESS_12,
(Encoding::Lossless, _) => LOSSLESS_14,
(Encoding::Lossy(_), 12) => LOSSY_12,
(Encoding::Lossy(_), _) => LOSSY_14,
}
.codes(),
)?;
let (lut, max): (&[u16], i32) = match &table.encoding {
Encoding::Lossless => (&[], (1i32 << bits) - 1),
Encoding::Lossy(curve) => (curve, curve.len() as i32 - 1),
};
let mut out = vec![0u16; n];
let mut stream = Bits::new(src);
let mut vpred = table.seeds;
let mut clipped = 0usize;
for (y, row) in out.chunks_exact_mut(w).enumerate() {
let seeds = vpred.get_mut(y & 1).ok_or(RawError::Limit("NEF row"))?;
let mut hpred = [0i32; 2];
for (x, o) in row.iter_mut().enumerate() {
let d = diff(&mut stream, &huff).ok_or_else(|| RawError::Corrupt(format!("NEF: invalid Huffman code at row {y}")))?;
let p = if x < 2 {
let s = seeds.get_mut(x).ok_or(RawError::Limit("NEF column"))?;
*s += d;
hpred[x & 1] = *s;
*s
} else {
hpred[x & 1] += d;
hpred[x & 1]
};
let v = p.clamp(0, max);
clipped += (v != p) as usize;
*o = if lut.is_empty() { v as u16 } else { lut.get(v as usize).copied().unwrap_or(0) };
}
// the camera pads the strip; reading more than a few bytes past its end means truncated or corrupt data
if stream.consumed_bits() > src.len() * 8 + 64 {
return Err(RawError::Corrupt(format!("NEF: compressed data ends at row {y} of {h}")));
}
}
// a wrong table / corrupt stream drifts out of range quickly; real files stay inside (a handful of edge pixels)
if clipped > n / 1000 + 16 {
return Err(RawError::Corrupt(format!("NEF: {clipped} decoded values out of range")));
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
/// MSB-first bit writer for the test encoder.
#[derive(Default)]
struct Writer {
out: Vec<u8>,
acc: u64,
n: u32,
}
impl Writer {
fn put(&mut self, v: u32, len: u32) {
for i in (0..len).rev() {
self.acc = (self.acc << 1) | ((v >> i) & 1) as u64;
self.n += 1;
if self.n == 8 {
self.out.push(self.acc as u8);
self.acc = 0;
self.n = 0;
}
}
}
fn finish(mut self) -> Vec<u8> {
if self.n > 0 {
self.out.push((self.acc << (8 - self.n)) as u8);
}
self.out.extend_from_slice(&[0; 4]);
self.out
}
}
fn category(d: i32) -> u32 {
32 - d.unsigned_abs().leading_zeros()
}
/// Encode `values` (decoded domain: samples or curve indices) the way the camera does.
fn encode(values: &[i32], w: usize, table: &Table, seeds: [[i32; 2]; 2]) -> Vec<u8> {
let codes = table.codes();
let mut wr = Writer::default();
let mut vpred = seeds;
for (y, row) in values.chunks(w).enumerate() {
for (x, &v) in row.iter().enumerate() {
let pred = if x < 2 { vpred[y & 1][x] } else { row[x - 2] };
if x < 2 {
vpred[y & 1][x] = v;
}
let d = v - pred;
let k = category(d);
let (code, len) = codes[k as usize];
assert!(len > 0, "no code for category {k}");
wr.put(code as u32 >> (12 - len), len as u32);
let extra = if d < 0 { d + (1 << k) - 1 } else { d };
wr.put(extra as u32, k);
}
}
wr.finish()
}
/// Deterministic test image: smooth gradient + texture + hard edges, within `0..=max`.
fn image(w: usize, h: usize, max: i32, seed: u32) -> Vec<i32> {
let mut s = seed;
(0..w * h)
.map(|i| {
let (x, y) = ((i % w) as i32, (i / w) as i32);
s = s.wrapping_mul(1_664_525).wrapping_add(1_013_904_223);
let noise = (s >> 24) as i32 % 9 - 4;
let base = (x * 37 + y * 11) % (max / 2) + if (x / 7 + y / 5) % 3 == 0 { max / 3 } else { 0 };
(base + noise).clamp(0, max)
})
.collect()
}
fn lossless(bits: u32) -> DecodeTable {
let s = 1 << (bits - 3);
DecodeTable { encoding: Encoding::Lossless, seeds: [[s, s], [s, s]] }
}
#[test]
fn tables_are_complete_prefix_codes() {
for t in [LOSSLESS_12, LOSSLESS_14, LOSSY_12, LOSSY_14] {
assert_eq!(t.counts.iter().map(|&c| c as usize).sum::<usize>(), t.symbols.len());
let kraft: f64 = t.counts.iter().enumerate().map(|(i, &c)| c as f64 / (1u64 << (i + 1)) as f64).sum();
assert!(kraft <= 1.0);
assert!(Huffman::new(&t.codes()).is_ok());
}
assert_eq!(LOSSLESS_14.codes()[7], (0b00 << 10, 2));
assert_eq!(LOSSLESS_14.codes()[0], (0b111110 << 6, 6));
assert_eq!(LOSSY_12.codes()[10], (0b111111110 << 3, 9));
}
#[test]
fn lossless_roundtrip_12_and_14() {
for (bits, table) in [(12, LOSSLESS_12), (14, LOSSLESS_14)] {
let (w, h) = (64, 9);
let max = (1 << bits) - 1;
let mut img = image(w, h, max, bits);
// extreme jumps exercise the longest categories
img[5] = 0;
img[7] = max;
img[w + 9] = max;
let t = lossless(bits);
let src = encode(&img, w, &table, t.seeds);
let out = decode(&src, w, h, bits, &t).unwrap();
assert_eq!(out, img.iter().map(|&v| v as u16).collect::<Vec<_>>(), "{bits}-bit");
}
}
fn lossy_table(bits: u32, order: ByteOrder) -> Vec<u8> {
// version 0x44 0x20, seeds, 257 points of a concave curve, padding up to the split value (0)
let mut t = vec![0x44, 0x20];
let put = |t: &mut Vec<u8>, v: u16| t.extend_from_slice(&if order == ByteOrder::Big { v.to_be_bytes() } else { v.to_le_bytes() });
for _ in 0..4 {
put(&mut t, 300);
}
put(&mut t, 257);
let max = ((1u32 << bits) - 1) as f64;
for i in 0..257u32 {
put(&mut t, (max * (i as f64 / 64.0).min(1.0).sqrt()) as u16);
}
t.resize(624, 0);
t
}
#[test]
fn lossy_type2_curve_and_roundtrip() {
for (bits, table) in [(12u32, LOSSY_12), (14, LOSSY_14)] {
for order in [ByteOrder::Big, ByteOrder::Little] {
let t = parse_table(&lossy_table(bits, order), order, bits).unwrap();
assert_eq!(t.seeds, [[300, 300], [300, 300]]);
let Encoding::Lossy(curve) = &t.encoding else { panic!() };
let step = (1usize << bits) / 256;
assert_eq!(curve.len(), (1 << bits) + 1);
assert_eq!(curve[64 * step], (1 << bits) - 1);
assert!(curve.windows(2).all(|p| p[0] <= p[1]));
// the encoded range is the curve's rising part
let (w, h) = (48, 7);
let img = image(w, h, 64 * step as i32 - 1, 7);
let src = encode(&img, w, &table, t.seeds);
let out = decode(&src, w, h, bits, &t).unwrap();
assert_eq!(out, img.iter().map(|&v| curve[v as usize]).collect::<Vec<_>>());
}
}
}
#[test]
fn lossy_type1_full_curve() {
let mut t = vec![0x44, 0x10, 0, 10, 0, 10, 0, 10, 0, 10, 0, 5];
for v in [0u16, 100, 400, 900, 1600] {
t.extend_from_slice(&v.to_be_bytes());
}
let parsed = parse_table(&t, ByteOrder::Big, 12).unwrap();
assert_eq!(parsed.encoding, Encoding::Lossy(vec![0, 100, 400, 900, 1600]));
let img = vec![0, 4, 1, 3, 2, 2, 4, 0];
let src = encode(&img, 4, &LOSSY_12, parsed.seeds);
assert_eq!(decode(&src, 4, 2, 12, &parsed).unwrap(), vec![0, 1600, 100, 900, 400, 400, 1600, 0]);
}
#[test]
fn rejects_unsupported_and_malformed_tables() {
let unsupported = |r: Result<DecodeTable>| matches!(r, Err(RawError::Unsupported(_)));
let corrupt = |r: Result<DecodeTable>| matches!(r, Err(RawError::Corrupt(_)));
let mut split = lossy_table(12, ByteOrder::Big);
split[562..564].copy_from_slice(&345u16.to_be_bytes());
assert!(unsupported(parse_table(&split, ByteOrder::Big, 12)));
assert!(unsupported(parse_table(&[0x49, 0x30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], ByteOrder::Big, 12)));
assert!(unsupported(parse_table(&lossy_table(12, ByteOrder::Big), ByteOrder::Big, 16)));
assert!(corrupt(parse_table(&[], ByteOrder::Big, 12)));
assert!(corrupt(parse_table(&[0x46, 0x30, 8], ByteOrder::Big, 14)));
assert!(corrupt(parse_table(&lossy_table(12, ByteOrder::Big)[..100], ByteOrder::Big, 12)));
// 300 points don't divide the 12-bit range
let mut odd = lossy_table(12, ByteOrder::Big);
odd[10..12].copy_from_slice(&300u16.to_be_bytes());
assert!(parse_table(&odd, ByteOrder::Big, 12).is_err());
assert!(parse_table(&[0x44, 0x20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], ByteOrder::Big, 12).is_err());
}
/// A minimal NEF: CFA SubIFD with one compressed strip, Exif maker note (`Nikon\0` v2 header + embedded TIFF in
/// `mn_order`) holding the linearization table.
fn nef_file(strip: Vec<u8>, w: u32, h: u32, bits: u16, table: Vec<u8>, mn_order: ByteOrder) -> Vec<u8> {
use lightcraft_tiff::tags::{self as t, photometric};
use lightcraft_tiff::{IfdBuilder, ImageData, TiffWriter, Value};
let mut mn = IfdBuilder::new();
mn.set(0x0096, Value::Undefined(table));
let mut note = b"Nikon\0\x02\x10\0\0".to_vec();
note.extend(TiffWriter::new(mn_order, false).write(&[mn]).unwrap());
let mut exif = IfdBuilder::new();
exif.set(t::MAKER_NOTE, Value::Undefined(note));
let mut raw = IfdBuilder::new();
raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0]));
raw.set(t::IMAGE_WIDTH, Value::Long(vec![w]));
raw.set(t::IMAGE_LENGTH, Value::Long(vec![h]));
raw.set(t::BITS_PER_SAMPLE, Value::Short(vec![bits]));
raw.set(t::COMPRESSION, Value::Short(vec![t::compression::NIKON]));
raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::CFA]));
raw.set(t::CFA_REPEAT_PATTERN_DIM, Value::Short(vec![2, 2]));
raw.set(t::CFA_PATTERN_EP, Value::Byte(vec![0, 1, 1, 2]));
raw.set_image(ImageData::Strips { rows_per_strip: h, strips: vec![strip] });
let mut ifd0 = IfdBuilder::new();
ifd0.set(t::MAKE, Value::Ascii("NIKON CORPORATION".into()));
ifd0.set(t::MODEL, Value::Ascii("NIKON TEST".into()));
ifd0.set_child(t::EXIF_IFD, exif);
ifd0.add_sub_ifd(raw);
TiffWriter::new(ByteOrder::Big, false).write(&[ifd0]).unwrap()
}
#[test]
fn decodes_through_the_nef_container() {
// lossless 14-bit, big-endian maker note
let (w, h) = (40usize, 6usize);
let img = image(w, h, 16383, 11);
let mut table = vec![0x46, 0x30];
for _ in 0..4 {
table.extend_from_slice(&2048u16.to_be_bytes());
}
table.resize(46, 0);
let src = encode(&img, w, &LOSSLESS_14, [[2048; 2]; 2]);
let bytes = nef_file(src, w as u32, h as u32, 14, table, ByteOrder::Big);
let r = crate::decode(&bytes).unwrap();
assert_eq!(r.data, crate::RawData::U16(img.iter().map(|&v| v as u16).collect()));
assert_eq!(r.cfa.as_ref().unwrap().name(), "RGGB");
assert_eq!(crate::probe_info(&bytes).unwrap(), r.info());
// lossy 12-bit type 2, little-endian maker note (newer bodies)
let table = lossy_table(12, ByteOrder::Little);
let t = parse_table(&table, ByteOrder::Little, 12).unwrap();
let Encoding::Lossy(curve) = &t.encoding else { panic!() };
let img = image(w, h, 1023, 5);
let src = encode(&img, w, &LOSSY_12, t.seeds);
let r = crate::decode(&nef_file(src.clone(), w as u32, h as u32, 12, table.clone(), ByteOrder::Little)).unwrap();
assert_eq!(r.data, crate::RawData::U16(img.iter().map(|&v| curve[v as usize]).collect()));
// the split variant falls back to the preview
let mut split = table;
split[562..564].copy_from_slice(&3u16.to_le_bytes());
let e = crate::decode(&nef_file(src, w as u32, h as u32, 12, split, ByteOrder::Little));
assert!(matches!(e, Err(RawError::Unsupported(_))), "{e:?}");
}
#[test]
fn truncated_and_corrupted_streams_error_never_panic() {
let (w, h) = (64, 16);
let t = lossless(14);
let img = image(w, h, 16383, 3);
let src = encode(&img, w, &LOSSLESS_14, t.seeds);
// truncation: too short for the image, or ends early
for cut in [0, 1, 10, src.len() / 2, src.len() - 40] {
assert!(decode(&src[..cut], w, h, 14, &t).is_err(), "cut {cut}");
}
// garbage and bit flips: an error or (rarely) some image, never a panic
let mut s = 12345u32;
for i in 0..200 {
let mut bad = src.clone();
for _ in 0..1 + i % 8 {
s = s.wrapping_mul(1_103_515_245).wrapping_add(12345);
let at = (s >> 8) as usize % bad.len();
bad[at] ^= 1 << (s % 8);
}
let _ = decode(&bad, w, h, 14, &t);
let noise: Vec<u8> = (0..src.len()).map(|k| ((k as u32).wrapping_mul(2_654_435_761) >> (i % 24)) as u8).collect();
let _ = decode(&noise, w, h, 14, &t);
let _ = decode(&noise, w, h, 12, &lossless(12));
}
// all-ones never forms a valid lossy 12-bit code
assert!(decode(&[0xff; 64], 8, 8, 12, &parse_table(&lossy_table(12, ByteOrder::Big), ByteOrder::Big, 12).unwrap()).is_err());
// absurd dimensions are limited before allocating
assert!(decode(&src, usize::MAX, 2, 14, &t).is_err());
assert!(decode(&src, 1 << 20, 1 << 20, 14, &t).is_err());
assert!(decode(&src, 0, 2, 14, &t).is_err());
}
}
+4
View File
@@ -99,6 +99,10 @@ impl<'a> Bits<'a> {
self.skip(k);
v
}
/// Bits handed out so far (may exceed the source length: zeros are read past its end).
pub fn consumed_bits(&self) -> usize {
self.pos * 8 - self.n as usize
}
pub fn overrun(&self) -> bool {
self.pos > self.src.len() + 8
}
+101 -2
View File
@@ -23,8 +23,6 @@ const KNOWN_UNSUPPORTED: &[&str] = &[
"rw2-panasonic-gx80", // "
"rw2-panasonic-g9-b", // "
"orf-olympus-em", // Olympus compressed ORF
"nef-nikon-d5100-lossless", // Nikon Huffman NEF
"nef-nikon-d7000-lossy", // "
"sraw", // Canon sRAW / mRAW
];
@@ -94,3 +92,104 @@ fn corpus_raw_decodes() {
}
eprintln!("corpus/raw: {ok} decoded, {unsupported} known-unsupported (preview only)");
}
/// Green-channel means of 32×32 blocks.
fn block_means(img: &lightcraft_raw::RawImage) -> Vec<f64> {
let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
let (w, h, b) = (img.width, img.height, 32);
let mut out = Vec::new();
for by in 0..h / b {
for bx in 0..w / b {
let (mut s, mut n) = (0f64, 0f64);
for y in by * b..by * b + b {
for x in (bx * b..bx * b + b).step_by(2) {
s += d[y * w + x + (y & 1)] as f64;
n += 1.0;
}
}
out.push(s / n);
}
}
out
}
fn correlation(a: &[f64], b: &[f64]) -> f64 {
let n = a.len() as f64;
let (ma, mb) = (a.iter().sum::<f64>() / n, b.iter().sum::<f64>() / n);
let cov: f64 = a.iter().zip(b).map(|(x, y)| (x - ma) * (y - mb)).sum();
let va: f64 = a.iter().map(|x| (x - ma).powi(2)).sum();
let vb: f64 = b.iter().map(|y| (y - mb).powi(2)).sum();
cov / (va * vb).sqrt()
}
/// Which diagonal of each 2×2 cell (anchored at raw pixel (0, 0)) holds the green sites: the two greens of a Bayer
/// cell see nearly the same light, so their mean absolute difference is far smaller than across the other diagonal.
/// Returns `true` when green sits at (0, 0)/(1, 1) (GBRG/GRBG), `false` for (1, 0)/(0, 1) (RGGB/BGGR).
fn green_on_main_diagonal(img: &lightcraft_raw::RawImage) -> bool {
let lightcraft_raw::RawData::U16(d) = &img.data else { panic!("float data") };
let (w, a) = (img.width, img.active_area);
let (mut main, mut anti) = (0f64, 0f64);
for y in ((a.y + 2) & !1..a.y + a.height - 2).step_by(2) {
for x in ((a.x + 2) & !1..a.x + a.width - 2).step_by(2) {
let v = |dx: usize, dy: usize| d[(y + dy) * w + x + dx] as f64;
main += (v(0, 0) - v(1, 1)).abs();
anti += (v(1, 0) - v(0, 1)).abs();
}
}
main < anti
}
/// Canon CR2 colour-filter layouts differ by model (issue #85); the decoder reads them from the `CR2CFAPattern` tag.
/// The expected layouts were checked visually (natural colours vs. the embedded preview) and agree with the
/// green-diagonal statistic of the mosaic itself.
#[test]
fn corpus_cr2_cfa_patterns() {
let dir = corpus_root().join("raw");
let cases = [
("cr2-canon-40d.cr2", "RGGB"),
("cr2-canon-550d.cr2", "GBRG"),
("cr2-canon-5d2.cr2", "GBRG"),
("cr2-canon-5d3.cr2", "RGGB"),
("cr2-canon-5dsr.cr2", "RGGB"),
("cr2-canon-6d.cr2", "RGGB"),
("cr2-canon-7d.cr2", "GBRG"),
("cr2-canon-80d.cr2", "RGGB"),
];
let mut seen = 0;
for (name, want) in cases {
let Ok(bytes) = std::fs::read(dir.join(name)) else {
eprintln!("skip: {name} absent");
continue;
};
let img = decode(&bytes).unwrap_or_else(|e| panic!("{name}: {e}"));
let got = img.cfa.as_ref().map(|c| c.name()).unwrap_or_default();
assert_eq!(got, want, "{name}: CFA layout");
assert_eq!(green_on_main_diagonal(&img), want.starts_with('G'), "{name}: mosaic statistics disagree with {want}");
seen += 1;
}
eprintln!("CR2 CFA layouts checked on {seen} files");
}
/// raw.pixls.us has the same D5100 scene as 14-bit lossless compressed and uncompressed NEF: the Huffman decode must
/// match the uncompressed image (up to the small differences between two exposures).
#[test]
fn corpus_nef_compressed_matches_uncompressed() {
let dir = corpus_root().join("raw");
let (Ok(a), Ok(b)) = (std::fs::read(dir.join("nef-nikon-d5100-lossless.nef")), std::fs::read(dir.join("nef-nikon-d5100-uncompressed.nef")))
else {
eprintln!("skip: D5100 NEF pair absent");
return;
};
let (a, b) = (decode(&a).unwrap(), decode(&b).unwrap());
assert_eq!((a.width, a.height, a.bits), (b.width, b.height, b.bits));
let r = correlation(&block_means(&a), &block_means(&b));
eprintln!("D5100 lossless vs uncompressed: block correlation {r:.4}");
assert!(r > 0.98, "correlation {r}");
// the lossy 12-bit D7000 file decodes into the curve's range and isn't flat
if let Ok(c) = std::fs::read(dir.join("nef-nikon-d7000-lossy12.nef")) {
let c = decode(&c).unwrap();
let m = block_means(&c);
let (lo, hi) = m.iter().fold((f64::MAX, 0f64), |(l, h), &v| (l.min(v), h.max(v)));
assert!(hi <= 4095.0 && hi - lo > 500.0, "D7000 block means {lo}..{hi}");
}
}
+13 -6
View File
@@ -78,10 +78,14 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {
"selection": app.session.selection.ids.iter().map(|p| p.0).collect::<Vec<_>>(),
"activeMask": app.session.active_mask,
"widgetCount": app.widgets.len(),
"perf": {"frameMs": app.perf.frame_ms, "logicMs": app.perf.logic_ms, "updateMs": app.perf.update_ms, "maxUpdateMs": app.perf.max_update_ms, "fps": app.perf.fps, "lastRenderMs": app.renderer.last_main_ms, "renderQueue": app.renderer.queued(), "rendersInFlight": app.renderer.in_flight(), "pendingSlots": app.renderer.pending_slots(), "mergeRunning": app.merge.busy(), "lastMerge": app.merge.last_result, "rendersDone": app.renderer.completed, "thumbTextures": app.renderer.thumb_textures(), "variantTextures": app.renderer.variant_textures(), "gpu": (lightcraft_engine::gpu::ready() && lightcraft_engine::gpu::available()).then(lightcraft_engine::gpu::adapter_name).flatten()},
"perf": {"frameMs": app.perf.frame_ms, "logicMs": app.perf.logic_ms, "updateMs": app.perf.update_ms, "maxUpdateMs": app.perf.max_update_ms, "fps": app.perf.fps, "lastRenderMs": app.renderer.last_main_ms, "renderQueue": app.renderer.queued(), "rendersInFlight": app.renderer.in_flight(), "pendingSlots": app.renderer.pending_slots(), "mergeRunning": app.merge.busy(), "lastMerge": app.merge.last_result, "rendersDone": app.renderer.completed, "thumbTextures": app.renderer.thumb_textures(), "variantTextures": app.renderer.variant_textures(), "gpu": (lightcraft_engine::gpu::ready() && lightcraft_engine::gpu::available()).then(lightcraft_engine::gpu::adapter_name).flatten(), "gpuReason": lightcraft_engine::gpu::unavailable_reason(), "gpuFallback": lightcraft_engine::gpu::last_fallback()},
"loupe": app.loupe_shown.map(|(p, src)| json!({"photo": p.0, "source": src, "pending": app.renderer.is_pending(crate::render::Slot::Main)})),
"hoverPreview": app.hover_preview.as_ref().map(|h| h.label.clone()),
"status": app.ui.status,
"notices": app.notices,
"quitPrompt": app.quit_prompt,
"unsaved": app.session.unsaved().map(|(n, e)| json!({"ops": n, "error": e})),
"libraryProblem": app.library_problem.as_ref().map(crate::panels::library_problem::LibraryProblem::to_json),
"scan": app.scan.as_ref().map(crate::import::ScanTask::status),
"export": {"running": app.export.as_ref().map(crate::export_task::ExportTask::status), "last": app.last_export_result},
"memory": memory(app),
@@ -289,12 +293,19 @@ pub fn handle(app: &mut LightcraftApp, ctx: &egui::Context, req: &ControlRequest
ok(Value::Null)
}
"ui.screenshot" => {
// never over a photo's original
if let Some(Err(e)) = s("path").map(|path| app.session.check_write_target(path)) {
return err(e);
}
ctx.request_repaint();
Outcome::Screenshot { path: s("path").map(str::to_string), headless: p.get("headless").and_then(Value::as_bool).unwrap_or(false) }
}
"ui.render" => {
let id = p.get("id").and_then(Value::as_u64).map(lightcraft_catalog::PhotoId).or(app.session.active());
let Some(id) = id else { return err("no photo") };
if let Some(Err(e)) = s("path").map(|path| app.session.check_write_target(path)) {
return err(e);
}
let size = p.get("size").and_then(Value::as_u64).unwrap_or(1600) as usize;
match app.session.render_now(id, size, size) {
Ok(r) => match (s("path"), app.services.png.as_ref()) {
@@ -355,11 +366,7 @@ pub fn export_active(app: &mut LightcraftApp, p: &Value) -> Result<Value, String
let to = Destination { dir: dir.clone(), exact: p.get("path").and_then(Value::as_str).map(str::to_string) };
let background = p.get("background").and_then(Value::as_bool).unwrap_or(false) && app.services.write_shared.is_some();
let out = if background {
let items = ids
.iter()
.enumerate()
.map(|(i, id)| lightcraft_engine::export::prepare_export(&mut app.session, *id, &opts, i + 1))
.collect::<Result<Vec<_>, _>>()?;
let items = lightcraft_engine::export::prepare_batch(&mut app.session, &ids, &opts)?;
crate::export_task::start(app, items, opts, to)?
} else {
let w = app.services.write.as_mut().ok_or("no writer")?;
+154
View File
@@ -0,0 +1,154 @@
//! Interface translations. Command ids, document text and file names remain stable.
//! Untranslated labels fall back to English so coverage can grow incrementally.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Language {
#[default]
En,
Ja,
}
impl Language {
pub const ALL: [Self; 2] = [Self::En, Self::Ja];
pub fn name(self) -> &'static str {
match self {
Self::En => "English",
Self::Ja => "日本語",
}
}
pub fn parse(code: &str) -> Option<Self> {
match code {
"en" => Some(Self::En),
"ja" => Some(Self::Ja),
_ => None,
}
}
pub fn tr(self, text: &str) -> &str {
if self == Self::Ja
&& let Some((_, japanese)) = JAPANESE.iter().find(|(english, _)| *english == text)
{
return japanese;
}
text
}
}
const JAPANESE: &[(&str, &str)] = &[
("Photo", "写真"),
("Import Photos…", "写真を読み込み…"),
("Detail", "詳細"),
("Photo Grid", "写真グリッド"),
("Square Grid", "正方形グリッド"),
("Object", "オブジェクト"),
("Effect", "効果"),
("Settings…", "環境設定…"),
("Image", "画像"),
("Layer", "レイヤー"),
("Type", "書式"),
("Select", "選択"),
("Filter", "フィルター"),
("Window", "ウィンドウ"),
("Language", "表示言語"),
("Save As…", "別名で保存…"),
("Exit", "終了"),
("New…", "新規…"),
("New", "新規"),
("Horizontal", "横書き"),
("Vertical", "縦書き"),
("Orientation", "組み方向"),
("Layers", "レイヤー"),
("History", "履歴"),
("Properties", "プロパティ"),
("Color", "カラー"),
("Brush Settings", "ブラシ設定"),
("Tools", "ツール"),
("Options", "オプション"),
("Zoom In", "ズームイン"),
("Zoom Out", "ズームアウト"),
("Fit on Screen", "画面に合わせる"),
("Copy", "コピー"),
("Cut", "切り取り"),
("Paste", "貼り付け"),
("Select All", "すべて選択"),
("Deselect", "選択を解除"),
("Export", "書き出し"),
("Export As…", "形式を指定して書き出し…"),
("Search…", "検索…"),
("Theme", "テーマ"),
("Menu", "メニュー"),
("File", "ファイル"),
("Edit", "編集"),
("Pages", "ページ"),
("View", "表示"),
("Help", "ヘルプ"),
("Preferences", "環境設定"),
("Preferences…", "環境設定…"),
("Interface language", "表示言語"),
("Open…", "開く…"),
("New blank PDF", "空白の PDF を作成"),
("Create PDF from file…", "ファイルから PDF を作成…"),
("Create PDF from images…", "画像から PDF を作成…"),
("Create PDF from clipboard", "クリップボードから PDF を作成"),
("Combine files…", "ファイルを結合…"),
("Save", "保存"),
("Save as…", "別名で保存…"),
("Close file", "ファイルを閉じる"),
("Close all", "すべて閉じる"),
("Revert", "保存済みの状態に戻す"),
("Print…", "印刷…"),
("Document properties…", "文書のプロパティ…"),
("Undo", "取り消し"),
("Redo", "やり直し"),
("Find…", "検索…"),
("Advanced search…", "高度な検索…"),
("Copy pages", "ページをコピー"),
("Cut pages", "ページを切り取り"),
("Paste pages", "ページを貼り付け"),
("Fit visible", "表示範囲に合わせる"),
("Marquee zoom", "範囲指定ズーム"),
("Take a snapshot", "スナップショットを作成"),
("Full screen mode", "全画面表示"),
("Read mode", "閲覧モード"),
("Switch light / dark theme", "明るい/暗いテーマを切り替え"),
("Comments panel", "コメントパネル"),
("Form fields panel", "フォームフィールドパネル"),
("Clear form", "フォームをクリア"),
("Find tools and commands…", "ツールとコマンドを検索…"),
("Zoom", "ズーム"),
("Actual size", "実際のサイズ"),
("Zoom to page level", "ページ全体を表示"),
("Fit to width", "幅に合わせる"),
("Display theme", "表示テーマ"),
("Side panels", "サイドパネル"),
("OK", "OK"),
];
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn language_setting_persists() {
let state = crate::state::UiState { language: Language::Ja, ..Default::default() };
let saved = serde_json::to_string(&state).unwrap();
let restored: crate::state::UiState = serde_json::from_str(&saved).unwrap();
assert_eq!(restored.language, Language::Ja);
assert_eq!(crate::state::UiState::default().language, Language::En);
}
#[test]
fn translations_are_unique_and_preserve_unknown_text() {
for (i, (en, ja)) in JAPANESE.iter().enumerate() {
assert!(!ja.is_empty());
assert!(JAPANESE.iter().take(i).all(|(other, _)| en != other));
assert_eq!(Language::En.tr(en), *en);
}
assert_eq!(Language::Ja.tr("File"), "ファイル");
assert_eq!(Language::Ja.tr("日本語の文書.pdf"), "日本語の文書.pdf");
assert_eq!(Language::parse("xx"), None);
}
}
+52
View File
@@ -9,6 +9,7 @@
pub mod control;
pub mod export_task;
pub mod headless;
pub mod i18n;
pub mod icons;
pub mod import;
pub mod links;
@@ -28,11 +29,17 @@ mod tests_curve;
#[cfg(test)]
mod tests_grid;
#[cfg(test)]
mod tests_library_problem;
#[cfg(test)]
mod tests_masking;
#[cfg(test)]
mod tests_panels;
#[cfg(test)]
mod tests_quit_unsaved;
#[cfg(test)]
mod tests_scroll;
#[cfg(test)]
mod tests_unsaved;
use std::sync::mpsc::{Receiver, Sender};
@@ -120,6 +127,12 @@ pub struct LightcraftApp {
pub native_shortcuts: std::collections::HashSet<String>,
/// The host is [`headless::Headless`] (it answers viewport screenshot commands itself).
pub headless_host: bool,
/// Warnings to show one at a time (damaged settings files…, issue #103).
pub notices: Vec<String>,
/// Quitting was stopped because changes couldn't be saved: the prompt's text.
pub quit_prompt: Option<String>,
/// Quit Anyway was chosen: the window may close with unsaved changes.
pub quit_confirmed: bool,
control_rx: Option<Receiver<ControlRequest>>,
pending_screenshots: Vec<PendingShot>,
screenshot_token: u64,
@@ -167,6 +180,9 @@ pub struct LightcraftApp {
gpu_applied: Option<bool>,
/// The memory budget setting last applied (MB, 0 = automatic).
memory_applied: Option<u32>,
/// The library failed to open at launch: the blocking window, then the temporary-session
/// banner (issue #100). Cleared once a library opens.
pub library_problem: Option<panels::library_problem::LibraryProblem>,
}
impl LightcraftApp {
@@ -184,6 +200,9 @@ impl LightcraftApp {
native_menu: false,
native_shortcuts: Default::default(),
headless_host: false,
notices: vec![],
quit_prompt: None,
quit_confirmed: false,
control_rx: None,
pending_screenshots: vec![],
screenshot_token: 0,
@@ -210,6 +229,7 @@ impl LightcraftApp {
window_is_fullscreen: false,
gpu_applied: None,
memory_applied: None,
library_problem: None,
}
}
@@ -281,6 +301,26 @@ impl LightcraftApp {
}
}
/// Announce when saving the library starts failing (changes then live only in memory and are
/// retried) and when it works again; the top bar's cloud icon shows the state meanwhile.
fn save_status(&mut self, ctx: &egui::Context) {
let unsaved = self.session.unsaved().map(|(n, e)| (n, e.to_string()));
match (unsaved, self.ui.unsaved_seen) {
(Some((n, e)), false) => {
self.ui.unsaved_seen = true;
let t = ctx.input(|i| i.time);
let what = if n == 1 { "1 change".to_string() } else { format!("{n} changes") };
self.ui.toast = Some((format!("{what} saved in memory but not written to disk: {e} — LightCraft will retry"), t + 6.0));
}
(None, true) => {
self.ui.unsaved_seen = false;
self.toast(ctx, "Library saved");
}
(Some(_), true) => ctx.request_repaint_after(std::time::Duration::from_secs(2)), // retry
(None, false) => {}
}
}
pub fn toast(&mut self, ctx: &egui::Context, text: impl Into<String>) {
let t = ctx.input(|i| i.time);
self.ui.toast = Some((text.into(), t + 1.4));
@@ -417,6 +457,7 @@ impl LightcraftApp {
/// Per-frame logic before layout (control channel, renders, shortcuts, drops).
pub fn logic(&mut self, ctx: &egui::Context) {
let t0 = now_ms();
panels::library_problem::logic(self);
self.logic_inner(ctx);
self.perf.logic_ms = now_ms() - t0;
}
@@ -432,6 +473,11 @@ impl LightcraftApp {
} else {
self.fonts_ready = true;
}
panels::notices::logic(self);
// closing the window (or Quit) with changes only in memory: retry, else ask first
if ctx.input(|i| i.viewport().close_requested()) && !panels::notices::may_close(self) {
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
}
let now = ctx.input(|i| i.time);
let dt = now - self.last_time;
if dt > 0.0 {
@@ -448,6 +494,7 @@ impl LightcraftApp {
import::poll_scan(self, ctx);
import::tick(self, ctx);
self.preview_build_status(ctx);
self.save_status(ctx);
self.slideshow_tick(ctx);
// back from an external editor: pick up the files it saved
let focused = ctx.input(|i| i.focused);
@@ -577,7 +624,9 @@ impl LightcraftApp {
// full-screen preview: the photo alone on black
egui::CentralPanel::default().frame(egui::Frame::NONE.fill(egui::Color32::BLACK)).show(ui, |ui| panels::detail::show(self, ui));
panels::second::show(self, &ctx);
panels::notices::show(self, &ctx);
panels::dialogs::show(self, &ctx);
panels::library_problem::show(self, &ctx);
panels::toast(self, &ctx);
self.widgets = widgets::take_registry(&ctx);
self.end_frame(t0);
@@ -586,6 +635,7 @@ impl LightcraftApp {
// Order matters: earlier panels take the full edge (top bar spans the window; the tool strip,
// right panels and left panel run to the bottom; the bottom bar sits between them).
panels::topbar::show(self, ui);
panels::library_problem::banner(self, ui);
panels::strip::show(self, ui);
if self.ui.right != state::RightPanel::None {
panels::right::show(self, ui);
@@ -612,7 +662,9 @@ impl LightcraftApp {
state::ViewMode::Reference => panels::compare::show_reference(self, ui),
});
panels::second::show(self, &ctx);
panels::notices::show(self, &ctx);
panels::dialogs::show(self, &ctx);
panels::library_problem::show(self, &ctx);
import::progress(self, &ctx);
import::scan_progress(self, &ctx);
export_task::poll(self, &ctx);
+23 -5
View File
@@ -250,6 +250,8 @@ pub fn checked(app: &LightcraftApp, id: &str) -> Option<bool> {
let u = &app.ui;
let panel = |p: RightPanel| Some(u.right == p);
match id {
"app.language.english" => Some(u.language == crate::i18n::Language::En),
"app.language.japanese" => Some(u.language == crate::i18n::Language::Ja),
"develop.autoSync" => Some(app.session.auto_sync),
"view.photoCounts" => Some(u.show_counts),
"view.secondWindow" => Some(u.second_window),
@@ -593,6 +595,22 @@ pub fn menu_bar(app: &LightcraftApp) -> Vec<(String, Vec<MenuNode>)> {
}
bar.push((title.to_string(), tidy(items)));
}
fn translate(nodes: &mut [MenuNode], language: crate::i18n::Language) {
for node in nodes {
match node {
MenuNode::Submenu { label, children } => {
*label = language.tr(label).to_string();
translate(children, language);
}
MenuNode::Item { label, .. } => *label = language.tr(label).to_string(),
MenuNode::Separator => {}
}
}
}
for (title, nodes) in &mut bar {
*title = app.ui.language.tr(title).to_string();
translate(nodes, app.ui.language);
}
bar
}
@@ -646,9 +664,9 @@ pub fn shortcut_text(sc: &str, mac: bool) -> String {
const TITLE_GAP: f32 = 24.0;
/// Width of the in-window menu bar's titles.
pub fn bar_width(ui: &egui::Ui) -> f32 {
pub fn bar_width(ui: &egui::Ui, language: crate::i18n::Language) -> f32 {
let t = crate::theme::Tokens::get(ui.ctx());
MENUS.iter().map(|m| ui.painter().layout_no_wrap(m.to_string(), t.font(13.0), t.text).size().x + TITLE_GAP).sum::<f32>()
MENUS.iter().map(|m| ui.painter().layout_no_wrap(language.tr(m).to_string(), t.font(13.0), t.text).size().x + TITLE_GAP).sum::<f32>()
}
/// The in-window menu bar (hosts without a native one): one dropdown per menu, or a single
@@ -666,14 +684,14 @@ pub fn show_in_window(app: &mut LightcraftApp, ui: &mut egui::Ui, max_width: f32
if total <= max_width {
let saved = ui.spacing().item_spacing.x;
ui.spacing_mut().item_spacing.x = TITLE_GAP;
for (title, items) in &bar {
for (index, (title, items)) in bar.iter().enumerate() {
let r = ui.add(egui::Button::new(egui::RichText::new(title).font(font.clone()).color(t.text_label)).frame(false));
crate::widgets::register(ui.ctx(), format!("menu:{title}"), r.rect);
crate::widgets::register(ui.ctx(), format!("menu:{}", MENUS.get(index).copied().unwrap_or(title.as_str())), r.rect);
egui::Popup::menu(&r).show(|ui| nodes_ui(ui, items, mac, &mut clicked));
}
ui.spacing_mut().item_spacing.x = saved;
} else {
let r = ui.add(egui::Button::new(egui::RichText::new("Menu").font(font.clone()).color(t.text_label)).frame(false));
let r = ui.add(egui::Button::new(egui::RichText::new(app.ui.language.tr("Menu")).font(font.clone()).color(t.text_label)).frame(false));
crate::widgets::register(ui.ctx(), "menu:all", r.rect);
egui::Popup::menu(&r).show(|ui| {
for (title, items) in &bar {
+22 -2
View File
@@ -11,6 +11,8 @@ use crate::state::{BeforeAfter, Dialog, RightPanel, ViewMode, Zoom};
pub type UiCommand = (&'static str, &'static str, Option<&'static str>, &'static str);
pub const UI_COMMANDS: &[UiCommand] = &[
("app.language.english", "English", None, "Edit>Language"),
("app.language.japanese", "日本語", None, "Edit>Language"),
("view.photoGrid", "Photo Grid", None, "View"),
("view.squareGrid", "Square Grid", None, "View"),
// G: Photo Grid ↔ Square Grid (from other views: the photo grid)
@@ -202,6 +204,10 @@ pub fn parse_rgb(v: &Value) -> Option<[u8; 3]> {
/// Handle UI commands; `None` means "not a UI command — send it to the engine".
pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Result<Value, String>> {
if matches!(id, "app.language.english" | "app.language.japanese") {
app.ui.language = if id == "app.language.japanese" { crate::i18n::Language::Ja } else { crate::i18n::Language::En };
return Some(Ok(json!(app.ui.language)));
}
let ctx = egui::Context::default();
let r: Result<Value, String> = match id {
"view.photoGrid" => {
@@ -856,7 +862,13 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
("Version".to_string(), env!("CARGO_PKG_VERSION").to_string()),
("System".to_string(), format!("{} ({})", std::env::consts::OS, std::env::consts::ARCH)),
("CPU threads".to_string(), std::thread::available_parallelism().map(|n| n.get().to_string()).unwrap_or_else(|_| "?".into())),
("GPU".to_string(), gpu.unwrap_or_else(|| "none (CPU rendering)".into())),
(
"GPU".to_string(),
gpu.unwrap_or_else(|| match lightcraft_engine::gpu::unavailable_reason() {
Some(why) => format!("none (CPU rendering): {why}"),
None => "none (CPU rendering)".into(),
}),
),
("GPU rendering".to_string(), if app.ui.settings.gpu { "on".into() } else { "off".into() }),
("Memory budget".to_string(), mb(lightcraft_engine::memory::default_budget() as u64)),
("Preview size".to_string(), format!("{} px", app.ui.settings.preview_edge)),
@@ -872,6 +884,9 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
format!("{:.1} ms (logic {:.1} ms; slowest {:.0} ms)", app.perf.update_ms, app.perf.logic_ms, app.perf.max_update_ms),
));
rows.push(("Last loupe render".into(), format!("{:.0} ms", app.renderer.last_main_ms)));
if let Some(f) = lightcraft_engine::gpu::last_fallback() {
rows.push(("Last GPU fallback".into(), f));
}
let r = json!(rows.iter().map(|(k, v)| json!({"label": k, "value": v})).collect::<Vec<_>>());
if p.get("open").and_then(Value::as_bool).unwrap_or(true) {
app.ui.dialog = Some(Dialog::SystemInfo { rows });
@@ -917,7 +932,12 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
if let Ok(v) = &r {
let n = v["found"].as_array().map_or(0, Vec::len);
let left = v["missing"].as_u64().unwrap_or(0);
app.toast(&egui::Context::default(), format!("Found {n} missing photo{}; {left} still missing", if n == 1 { "" } else { "s" }));
let unsure = v["ambiguous"].as_array().map_or(0, Vec::len);
let unsure = if unsure > 0 { format!(" ({unsure} with several look-alike files: use Locate)") } else { String::new() };
app.toast(
&egui::Context::default(),
format!("Found {n} missing photo{}; {left} still missing{unsure}", if n == 1 { "" } else { "s" }),
);
}
r
}
+2 -1
View File
@@ -551,8 +551,9 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
wm.image_width = (width / 100.0) as f32;
}
} else {
choices(ui, app.ui.language.tr("Orientation"), "exportWmOrientation", &[(false, app.ui.language.tr("Horizontal")), (true, app.ui.language.tr("Vertical"))], &mut wm.vertical);
field(ui, "Text", |ui| {
ui.add(egui::TextEdit::singleline(&mut wm.text).hint_text("© Your Name").desired_width(f32::INFINITY))
ui.add(egui::TextEdit::multiline(&mut wm.text).desired_rows(2).hint_text("© Your Name").desired_width(f32::INFINITY))
});
}
choices(
@@ -0,0 +1,182 @@
//! The library couldn't be opened at launch (issue #100): never a silent in-memory session. A
//! blocking window says which library and why, and offers Try Again, Choose Another Library…,
//! Continue Without Saving and Quit. Continuing shows a banner under the top bar for as long as the
//! session is temporary.
use egui::{Align2, RichText, vec2};
use serde_json::json;
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
/// A library that failed to open (set by the host at launch).
#[derive(Clone, Debug, Default, PartialEq)]
pub struct LibraryProblem {
/// The library folder ("" when there was none to try, e.g. no home folder).
pub path: String,
/// Why it didn't open.
pub error: String,
/// Continue Without Saving was chosen: the banner shows instead of the window.
pub dismissed: bool,
/// Files from the command line, imported once a library opens.
pub pending_import: Vec<String>,
/// Try Again reopens `path` as a library folder (`false` in the browser, whose storage isn't
/// a folder: reloading the page tries again).
pub can_retry: bool,
}
impl LibraryProblem {
pub fn new(path: impl Into<String>, error: impl Into<String>) -> LibraryProblem {
LibraryProblem { path: path.into(), error: error.into(), can_retry: true, ..Default::default() }
}
/// `ui.inspect` → `libraryProblem`.
pub fn to_json(&self) -> serde_json::Value {
json!({"path": self.path, "error": self.error, "temporarySession": self.dismissed, "pendingImport": self.pending_import.len()})
}
}
/// What the user chose.
enum Choice {
Retry,
Choose,
Temporary,
Quit,
}
/// Open `path` (`None`: ask for a folder) through `app.openLibrary`; on success the problem is
/// over (and command-line files are imported), else it shows the new error.
fn open(app: &mut LightcraftApp, path: Option<String>) {
let params = match &path {
Some(p) => json!({"path": p}),
None => json!({}),
};
match app.run("app.openLibrary", params) {
Ok(serde_json::Value::Null) => {} // folder dialog cancelled
Ok(_) => {
let files = app.library_problem.take().map(|p| p.pending_import).unwrap_or_default();
if !files.is_empty() {
let _ = app.run("library.import", json!({"paths": files}));
}
}
Err(e) => {
if let Some(p) = app.library_problem.as_mut() {
if let Some(path) = path {
p.path = path;
}
p.error = e;
p.dismissed = false;
}
}
}
}
/// Clear the problem once a library is open (e.g. through Settings → Open Library…).
pub fn logic(app: &mut LightcraftApp) {
if app.library_problem.is_some() && app.session.library.is_some() {
app.library_problem = None;
}
}
/// The blocking window (until a choice is made).
pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(problem) = app.library_problem.clone() else { return };
if problem.dismissed {
return;
}
let t = Tokens::get(ctx);
let screen = ctx.content_rect();
egui::Area::new(egui::Id::new("library-problem-dim")).order(egui::Order::Middle).fixed_pos(screen.min).interactable(true).show(ctx, |ui| {
// swallows clicks: nothing behind the window can be used meanwhile
ui.allocate_rect(screen, egui::Sense::click_and_drag());
ui.painter().rect_filled(screen, 0.0, egui::Color32::from_black_alpha(160));
});
let mut choice = None;
let frame = egui::Frame::window(&ctx.global_style()).inner_margin(egui::Margin::symmetric(18, 14));
egui::Window::new("Your library couldn't be opened")
.id(egui::Id::new("library-problem"))
.order(egui::Order::Foreground)
.collapsible(false)
.resizable(false)
.frame(frame)
.anchor(Align2::CENTER_CENTER, [0.0, 0.0])
.default_width(460.0)
.show(ctx, |ui| {
// a fixed width: a wrapped label in an auto-sized, centred window would move it every frame
ui.set_width(440.0);
ui.spacing_mut().item_spacing.y = 8.0;
if problem.path.is_empty() {
ui.label(RichText::new("LightCraft couldn't find where to keep your library.").color(t.text));
} else {
ui.label(RichText::new("LightCraft couldn't open the library at").color(t.text_label));
ui.label(RichText::new(&problem.path).font(t.semibold(12.5)).color(t.text));
}
ui.add(egui::Label::new(RichText::new(&problem.error).color(t.caution)).wrap());
ui.label(
RichText::new(
"Your photos and edits in it are untouched. You can try again (e.g. after closing another program that has it open \
or reconnecting its drive), open a different library or create a new one, or continue without a library — \
then nothing you do is saved.",
)
.color(t.text_dim),
);
ui.add_space(4.0);
ui.horizontal(|ui| {
let mut button = |ui: &mut egui::Ui, id: &str, label: &str, enabled: bool, c: Choice| {
let r = ui.add_enabled(enabled, egui::Button::new(label).min_size(vec2(0.0, 26.0)));
register(ui.ctx(), format!("button:{id}"), r.rect);
if r.clicked() {
choice = Some(c);
}
};
button(ui, "libraryRetry", "Try Again", problem.can_retry && !problem.path.is_empty(), Choice::Retry);
button(ui, "libraryChoose", "Choose Another Library…", app.services.pick_folder.is_some(), Choice::Choose);
button(ui, "libraryTemporary", "Continue Without Saving", true, Choice::Temporary);
button(ui, "libraryQuit", "Quit", true, Choice::Quit);
});
});
match choice {
Some(Choice::Retry) => open(app, Some(problem.path)),
Some(Choice::Choose) => open(app, None),
Some(Choice::Temporary) => {
if let Some(p) = app.library_problem.as_mut() {
p.dismissed = true;
}
let files = app.library_problem.as_mut().map(|p| std::mem::take(&mut p.pending_import)).unwrap_or_default();
if !files.is_empty() {
let _ = app.run("library.import", json!({"paths": files}));
}
}
Some(Choice::Quit) => app.ui.quit = true,
None => {}
}
}
/// The banner under the top bar while the session is temporary (Continue Without Saving).
pub fn banner(app: &mut LightcraftApp, ui: &mut egui::Ui) {
if !app.library_problem.as_ref().is_some_and(|p| p.dismissed) {
return;
}
let t = Tokens::get(ui.ctx());
let mut reopen = false;
egui::Panel::top("library-problem-banner")
.exact_size(30.0)
.frame(egui::Frame::NONE.fill(t.caution).inner_margin(egui::Margin::symmetric(12, 0)))
.show(ui, |ui| {
ui.horizontal_centered(|ui| {
let r = ui.label(
RichText::new("Temporary session — your library isn't open, and nothing you do here is saved.")
.font(t.semibold(12.5))
.color(t.canvas),
);
register(ui.ctx(), "indicator:temporarySession", r.rect);
let b = ui.add(egui::Button::new(RichText::new("Open Library…").color(t.canvas)).fill(egui::Color32::from_black_alpha(40)));
register(ui.ctx(), "button:libraryReopen", b.rect);
reopen = b.clicked();
});
});
if reopen && let Some(p) = app.library_problem.as_mut() {
p.dismissed = false;
}
}
+2
View File
@@ -10,7 +10,9 @@ pub mod edit;
pub mod filterbar;
pub mod grid;
pub mod left;
pub mod library_problem;
pub mod masking;
pub mod notices;
pub mod presets;
pub mod profiles;
pub mod right;
+115
View File
@@ -0,0 +1,115 @@
//! Things the user must not miss (issue #103): settings files that were damaged or unreadable when
//! the library opened (and host warnings such as a damaged `ui.json`), and quitting while changes
//! are only in memory.
use egui::{Align2, RichText, vec2};
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
/// Collect the library's settings-file warnings (each is shown once).
pub fn logic(app: &mut LightcraftApp) {
let new = app.session.take_library_warnings();
app.notices.extend(new);
}
/// The window may close now (`true`), or the quit prompt is shown (`false`): changes that
/// couldn't be written are retried first — the log, then a snapshot of everything in memory.
pub fn may_close(app: &mut LightcraftApp) -> bool {
if app.quit_confirmed || app.session.unsaved().is_none() {
return true;
}
let _ = app.session.persist();
if app.session.unsaved().is_some()
&& let Err(e) = app.session.close_library()
{
log::error!("quit: library not saved: {e}");
}
match app.session.unsaved() {
None => true,
Some((n, e)) => {
app.quit_prompt = Some(format!(
"{} couldn't be written to disk: {e}\nIf you quit now, {} lost.",
if n == 1 { "1 change".to_string() } else { format!("{n} changes") },
if n == 1 { "it is" } else { "they are" }
));
false
}
}
}
enum Choice {
Retry,
QuitAnyway,
Cancel,
Ok,
}
fn window(ctx: &egui::Context, id: &str, title: &str, text: &str, buttons: &[(&str, &str, Choice)]) -> Option<usize> {
let t = Tokens::get(ctx);
let screen = ctx.content_rect();
egui::Area::new(egui::Id::new((id, "dim"))).order(egui::Order::Middle).fixed_pos(screen.min).show(ctx, |ui| {
ui.allocate_rect(screen, egui::Sense::click_and_drag());
ui.painter().rect_filled(screen, 0.0, egui::Color32::from_black_alpha(140));
});
let mut chosen = None;
let frame = egui::Frame::window(&ctx.global_style()).inner_margin(egui::Margin::symmetric(18, 14));
egui::Window::new(title)
.id(egui::Id::new(id))
.order(egui::Order::Foreground)
.collapsible(false)
.resizable(false)
.frame(frame)
.anchor(Align2::CENTER_CENTER, [0.0, 0.0])
.default_width(440.0)
.show(ctx, |ui| {
// a fixed width: a wrapped label in an auto-sized, centred window would move it every frame
ui.set_width(420.0);
ui.spacing_mut().item_spacing.y = 8.0;
ui.add(egui::Label::new(RichText::new(text).color(t.text_label)).wrap());
ui.add_space(4.0);
ui.horizontal(|ui| {
for (i, (bid, label, _)) in buttons.iter().enumerate() {
let r = ui.add(egui::Button::new(*label).min_size(vec2(0.0, 26.0)));
register(ui.ctx(), format!("button:{bid}"), r.rect);
if r.clicked() {
chosen = Some(i);
}
}
});
});
chosen
}
pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
if let Some(text) = app.quit_prompt.clone() {
let buttons = [
("quitRetry", "Try Saving Again", Choice::Retry),
("quitAnyway", "Quit Anyway", Choice::QuitAnyway),
("quitCancel", "Cancel", Choice::Cancel),
];
let Some(i) = window(ctx, "quit-unsaved", "Quit with unsaved changes?", &text, &buttons) else { return };
match buttons[i].2 {
Choice::Retry => {
app.quit_prompt = None;
if may_close(app) {
app.ui.quit = true;
}
}
Choice::QuitAnyway => {
app.quit_prompt = None;
app.quit_confirmed = true;
app.ui.quit = true;
}
Choice::Cancel | Choice::Ok => app.quit_prompt = None,
}
return;
}
if let Some(text) = app.notices.first().cloned() {
let buttons = [("noticeOk", "OK", Choice::Ok)];
if window(ctx, "settings-notice", "Settings file problem", &text, &buttons).is_some() {
app.notices.remove(0);
}
}
}
+17 -1
View File
@@ -342,7 +342,10 @@ fn performance_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {
heading(ui, t, "Rendering");
check(ui, "settings.gpu", &mut app.ui.settings.gpu, "Use the GPU for rendering");
let status = if !gpu::available() {
"No usable GPU found: rendering on the CPU".to_string()
match gpu::unavailable_reason() {
Some(why) => format!("Rendering on the CPU: {why}"),
None => "No usable GPU found: rendering on the CPU".to_string(),
}
} else {
format!("GPU: {}", gpu::adapter_name().unwrap_or_else(|| "starting…".into()))
};
@@ -376,6 +379,19 @@ fn performance_tab(app: &mut LightcraftApp, ui: &mut egui::Ui, t: &Tokens) {
let _ = app.run("library.clearPreviews", json!({}));
}
});
heading(ui, t, "Local folders");
row(ui, t, "Forget unchanged photos", |ui| {
let opts = [(0u32, "Never"), (7, "After 7 days"), (30, "After 30 days"), (90, "After 90 days"), (365, "After a year")];
let mut v = app.session.forget_local_days;
if choices(ui, "settingsForgetLocal", &opts, &mut v) {
let _ = app.run("library.preferences", json!({"forgetLocalDays": v}));
}
});
hint(
ui,
t,
"Photos seen in Local but never added or changed leave the catalog when their folder hasn't been browsed for this long (checked when the library opens). Files stay on disk; browsing the folder shows them again.",
);
#[cfg(not(target_arch = "wasm32"))]
smart_previews(app, ui, t);
}
+26 -3
View File
@@ -19,7 +19,7 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
let mut sw = 640.0f32.min(full.width() - 460.0).max(200.0);
if !app.native_menu {
// leave room for the in-window menus left of the (centred) search field
let menus_right = full.left() + 140.0 + crate::menubar::bar_width(ui) + 24.0;
let menus_right = full.left() + 140.0 + crate::menubar::bar_width(ui, app.ui.language) + 24.0;
sw = sw.min(2.0 * (full.center().x - menus_right)).max(200.0);
}
let search_left = full.center().x - sw / 2.0;
@@ -111,9 +111,18 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
}
// right icons
let mut x = full.right() - 18.0;
// saving is failing: the cloud icon turns into a warning until a save succeeds
let unsaved = app.session.unsaved().map(|(n, e)| {
format!(
"{n} change{} saved in memory but not written to disk: {e}\nLightCraft retries automatically; quitting now would lose {}.",
if n == 1 { "" } else { "s" },
if n == 1 { "it" } else { "them" }
)
});
let cloud_tip = unsaved.as_deref().unwrap_or("Local library — no cloud account needed");
for (id, icon, tip, cmd) in [
("discord", Icon::Chat, "Join the ArtCraft community on Discord", "app.discord"),
("cloud", Icon::Cloud, "Local library — no cloud account needed", ""),
("cloud", Icon::Cloud, cloud_tip, ""),
("help", Icon::Help, "Keyboard shortcuts", "app.shortcuts"),
("share", Icon::Share, "Export", "dialog.export"),
("bell", Icon::Bell, "Activity", "panel.activity"),
@@ -121,7 +130,21 @@ pub fn show(app: &mut LightcraftApp, ui: &mut egui::Ui) {
let r = Rect::from_center_size(pos2(x, full.center().y), vec2(28.0, 28.0));
let resp = ui.interact(r, egui::Id::new(("top", id)), Sense::click()).on_hover_text(tip);
register(ui.ctx(), format!("icon:{id}"), r);
paint(ui.painter(), r.shrink(5.0), icon, if resp.hovered() { t.text } else { t.icon });
let warn = id == "cloud" && unsaved.is_some();
let colour = if warn {
t.caution
} else if resp.hovered() {
t.text
} else {
t.icon
};
paint(ui.painter(), r.shrink(5.0), icon, colour);
if warn {
let c = r.right_top() + vec2(-5.0, 6.0);
ui.painter().circle_filled(c, 6.0, t.reject);
ui.painter().text(c, Align2::CENTER_CENTER, "!", t.semibold(9.5), t.canvas);
register(ui.ctx(), "indicator:unsaved", r);
}
if resp.clicked() && !cmd.is_empty() {
let _ = app.run(cmd, json!({}));
}
Loaded 100 of 113 files, more files were not shown because too many files have changed in this diff. Show more