Compare commits

..
Author SHA1 Message Date
Brandon ThomasandClaude Opus 5.5 4d5b1c2da4 WIP LR-PROF-CAMERACOLOR: generalise the camera-preview fit (interrupted by usage limit; unverified, do not land)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:12:23 -04:00
Brandon Thomas bd3185ef4d Merge pull request #145 from storytold/fix/issue-138-lightroom-dng
Issue #138: DNG: apply the file's own camera-profile look (HueSatMap, LookTable, ToneCurve)
2026-10-06 11:27:16 -04:00
Brandon Thomas 4cf883155a Merge pull request #144 from storytold/fix/issue-136-vulkan-startup-crash
Issue #136: Windows: use DX12 by default, honour WGPU_BACKEND, never let GPU init block startup
2026-10-06 11:27:03 -04:00
Brandon ThomasandClaude Opus 5.5 71eac70696 Issue #138: DNG: apply the file's own camera-profile look (HueSatMap, LookTable, ToneCurve)
Lightroom-converted DNGs carry their camera profile's ProfileHueSatMapData1/2
and ProfileLookTableData (and sometimes ProfileToneCurve). We used only the
colour matrices, so such DNGs rendered flat and muted.

- raw: new `profile` module reads the tags (DNG spec ch. 6: value-major,
  hue-middle, saturation-minor tables, wrap-around hue, tri-linear lookups,
  sRGB value encoding) from the raw IFD or IFD 0; malformed tags are ignored.
- engine: the raw loader applies the hue/sat map (interpolated by the same
  illuminant weight as the matrices) before BaselineExposure and the look
  table after it, in linear ProPhoto; a ProfileToneCurve becomes the source's
  camera tone. Without one the standard base tone is kept.
- dngwrite keeps the look with the data (conversions, smart previews, merges).
- Render cache version 7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:36:30 -04:00
Brandon ThomasandClaude Opus 5.5 b66cdb23a3 Issue #136: Windows: use DX12 by default, honour WGPU_BACKEND, never let GPU init block startup
wgpu loads the driver of every backend in an instance's set while it
enumerates adapters. The compute device (crates/gpu) hard-coded
Backends::PRIMARY (Vulkan + DX12 + Metal) and ignored WGPU_BACKEND, and
was created from LightcraftApp::new on a background thread at launch, so
a crashing Vulkan driver (Intel UHD 630, igvk64.dll) killed the process
before the window appeared; the window (egui-wgpu) also listed Vulkan.

- lightcraft_gpu::backend: DX12 only on Windows (Metal on macOS, as before
  on Linux) for both the compute device and the window;
  LIGHTCRAFT_GPU_BACKEND=dx12|vulkan|metal|gl|auto|off, else WGPU_BACKEND;
  backends missing from the build fall back to the default.
- The device is created only when GPU rendering is enabled, after the
  first frame; the preference is applied before the window opens.
- Crash sentinel: gpu-init.marker in the settings folder around device
  creation; if left behind, the next launch starts with GPU rendering off
  and a notice saying how to turn it back on.
- Tests: backend parsing/selection, sentinel, window defaults, headless
  snapshot with LIGHTCRAFT_GPU=0 / LIGHTCRAFT_GPU_BACKEND=off.
- Docs: gpu-pipeline.md (backends, env vars, troubleshooting), whats-new,
  parity LR-BEHAV-GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:16:05 -04:00
Brandon Thomas d2359ce095 Merge pull request #143 from storytold/fix/issue-134-trim-write-safety
Issue #134: Exports: atomic without fsync; Copy import verifies against the probe hash
2026-10-06 10:11:47 -04:00
Brandon ThomasandClaude Opus 5.5 7a2596cd43 Issue #134: Exports: atomic without fsync; Copy import verifies against the probe hash
- safe_file: write policy documented (durable for what can't be recreated, atomic-only for
  recreatable outputs); new write_atomic_nosync (temp + rename, temp removed on failure, no
  file/folder sync); per-thread sync counter for tests.
- export::write_file (exports, renders, screenshots, merge previews, CLI/MCP) uses the
  non-syncing writer; new write_file_durable keeps the sync for Edit a Copy (the TIFF becomes a
  library photo). Catalog, sidecars, settings, DNG conversion/merges and smart previews stay
  durable; the #93 original-overwrite guard is untouched.
- Import > Copy verifies the synced copy against the probe's content hash (SipHash-1-3/128 of the
  whole file) instead of re-reading the source; byte compare when there is no hash. A file changed
  since Review Import is now refused instead of imported with stale details. Move keeps its
  byte-for-byte check before deleting sources.
- check_write_target: skip building the library-wide guard when nothing is at the target.
- write_new_unique keeps its read-back (DNGs replace raws; cheap next to encoding).
- Tests: no sync on the export path, durable writers still sync, Copy caught by the probe hash
  (corrupt copy, stale hash, changed source); ignored timing test write_policy_timing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:08:29 -04:00
Brandon Thomas 8bef0b1ef3 Merge pull request #141 from storytold/pr/windows-arm64
Windows on ARM64: release builds, PE header check, install test on ARM64 hardware
2026-10-06 09:59:27 -04:00
Brandon ThomasandClaude Opus 5.5 a2dbefd1ee Windows ARM64 workflow: pin third-party actions by commit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:51:08 -04:00
Brandon ThomasandClaude Opus 5.5 f5bd3cc2db Windows on ARM64: release builds, PE header check, install test on ARM64 hardware
Release builds add an arm64 MSI and portable zip, cross-compiled on the x64
runner (signing and WiX as for x64). package.ps1 checks each binary's PE
header before packaging: Machine must match -Arch, the app must be a GUI
program and the CLI a console program. It skips the CLI smoke run when an
ARM64 build can't run on the host. A new Windows ARM64 workflow packages the
ARM64 MSI, installs it on a Windows 11 ARM64 runner, checks both installed
exes are ARM64, runs the CLI, and uninstalls. Ported from printcraft#66.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:38:47 -04:00
Brandon Thomas 1969b311cc Merge pull request #137 from benlukka/fix-flickering-preview-hover
fix(ui): eliminate preview flickering when hovering presets and profiles
2026-10-06 09:24:12 -04:00
Brandon Thomas fc3fa1a71b Merge pull request #140 from storytold/pr/release-0.2.1
Release: LightCraft v0.2.1
2026-10-06 08:01:39 -04:00
Brandon ThomasandClaude Opus 5.5 d0612b0b21 Release: LightCraft v0.2.1
Fixes the Windows installer: the Start Menu shortcut and Add/Remove Programs icon were blank, and the shortcut description was wrong.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:01:29 -04:00
Brandon Thomas 66dc737ff5 Merge pull request #139 from storytold/pr/windows-packaging
Windows packaging: Start Menu/Add-Remove icon, no console window, correct description
2026-10-06 07:54:46 -04:00
Brandon ThomasandClaude Opus 5.5 fec3b15234 release.yml: explicit if for the wasm-bindgen install (shellcheck SC2015)
packaging-lint has failed on main since this step was added; same behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:53:44 -04:00
Brandon ThomasandClaude Opus 5.5 11ebb3bdb6 Windows packaging: Start Menu/Add-Remove icon, no console window, correct description
The MSI Icon id had no .ico/.exe extension, which Windows Installer needs for
advertised shortcuts and ARPPRODUCTICON, so both showed a blank icon. Release
builds of the desktop app are now GUI-subsystem (no console window; the CLI stays
a console program), the shortcut/summary description names what LightCraft does, and
packaging-lint rejects an Icon id without an extension. Found in printcraft#57.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:50:38 -04:00
benlukka ad2525ece2 fix(ui): eliminate preview flickering when hovering presets and profiles
- Retain active preset preview when hovering within the scroll or panel area across headers and gaps.
- Prevent premature dropping of hover preview textures during asynchronous rerendering in detail view.
- Clear cached hover state when clicking an item or when the pointer leaves the panel.
- Add headless regression tests covering preset hover persistence and transitions.

Signed-off-by: benlukka <68825026+benlukka@users.noreply.github.com>
2026-10-06 13:18:47 +02:00
Brandon Thomas 65da36675f Merge pull request #115 from arata-1972/codex/sony-arw-initial-colour
Improve Sony ARW starting colour and tone, and preserve decoded source metadata
2026-10-05 19:47:57 -04:00
Brandon ThomasandClaude Opus 5.5 cbb6c3b8f7 Merge origin/main (#114) into land-115
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:41:46 -04:00
Brandon ThomasandClaude Opus 5.5 5b414ffb8e LR-PROF-CAMERACOLOR: record the ARW file-local fit's public-sample results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:34:01 -04:00
Brandon ThomasandClaude Opus 5.5 f7a7831380 LR-PROF-CAMERACOLOR: keep the ARW camera tone curve with smart previews
Smart previews store the decoded source with the fitted colour matrix
baked in, but the file-local tone curve lives in the decoder's
SourceInfo, which a proxy did not keep: an offline ARW rendered and
exported with the generic filmic curve instead (darker, a different
look). The proxy header now carries the curve (validated on load,
ignored when invalid), DecodedSource::info_or merges it into the header
facts, and is_valid reads the longer header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:30:43 -04:00
Brandon Thomas 2b2ade7a23 Merge pull request #114 from arata-1972/codex/lightcraft-japanese
Extend Japanese localization across editing, import and export UI
2026-10-05 19:29:49 -04:00
Brandon ThomasandClaude Opus 5.5 d6c7228dd4 M16: merge origin/main into Japanese localisation (PR #114)
Keep main's data-safety work (ui.json recovery and atomic writes, library-problem
window, off-thread file checks, import Cancel) and wrap its new strings: import
cancel/progress messages, the library-problem window and banner, and the new
Back Up / Restore Library menu items now have Japanese translations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:28:34 -04:00
Brandon ThomasandClaude Opus 5.5 4b3048c48f LR-PROF-CAMERACOLOR: accept ARW camera-look fits up to 0.10 held-out RMS
The 0.055 cap rejected half of the public Sony samples tried (ILCE-6400,
ILCE-6700, ILCE-7M4) although their fits were 1.5-3x closer to the
camera JPEG than the fallback and visibly better (mean dE vs the embedded
preview 19-26 -> 5-10). Unrelated previews stay rejected (synthetic test
lands at 0.115).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:28:22 -04:00
Brandon ThomasandClaude Opus 5.5 bc4799ca42 Merge origin/main into land-115 (PR #115)
Resolve SourceRef::load_source against #104's offline fallback: a File
source keeps the decoder's SourceInfo; a smart-preview fallback has none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:25:37 -04:00
Brandon ThomasandClaude Opus 5.5 586479a309 M16: drop fork-only Japanese packaging from PR #114
Remove scripts/package-japanese-macos.py, the japanese-local feature and its
config/library/startup overrides, the settings.language migration, and the
local-build notes in the docs. Upstream library and config defaults (incl.
library::default_dir, which already honours LIGHTCRAFT_LIBRARY) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:11:34 -04:00
Brandon Thomas 87f275b58e Merge pull request #132 from storytold/fix/issue-104-ui-fs-v2
Issue #104: keep file-system calls off the UI thread
2026-10-05 19:08:15 -04:00
Brandon Thomas 894759e6d8 Merge main into fix/issue-104-ui-fs-v2
# Conflicts:
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 19:00:37 -04:00
Brandon Thomas 1952f7c960 Merge pull request #131 from storytold/fix/issue-107-web-safety
Issue #107: Web build: data-safety fixes for libraries kept in the browser
2026-10-05 18:59:14 -04:00
Brandon Thomas 7bbdbc03ef Merge main into fix/issue-107-web-safety
# Conflicts:
#	apps/lightcraft-web/src/web.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:58:53 -04:00
Brandon Thomas 0f34a65a8c Merge pull request #130 from storytold/fix/issue-97-folder-undo
Issue #97: Rename Folder / Move Folder are undo steps
2026-10-05 18:58:38 -04:00
Brandon Thomas 6c066b66f2 Merge main into fix/issue-97-folder-undo
# Conflicts:
#	crates/engine/src/lib.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:58:13 -04:00
Brandon ThomasandClaude Opus 5.5 7b59081d22 Issue #104: keep file-system calls off the UI thread
Re-applies fix/issue-104-ui-fs on top of the data-safety fixes (integrate/safety):
- availability cache: originals and smart previews are checked on a worker thread; the UI
  answers from the cache (grid, Info panel, photo menu, Missing Photos view)
- import split into ImportJob::prepare (worker) + commit_prepared, with Cancel; Copy keeps
  #96's verified copies
- Find Missing Photos: find_candidates (snapshot, no I/O) + plan_find_missing (all file I/O,
  on a worker, never the session) with #105's matching rules unchanged (name+size+hash;
  renamed by size+hash; single name+size; ambiguous look-alikes skipped and reported; files
  already in use excluded). The plan is applied with library.findMissing {found, missing,
  ambiguous}, re-checked against the catalog then: photos relinked meanwhile are left alone,
  files now in use are not taken. library.findMissing {folder} still runs synchronously.
- Build / Discard Smart Previews with background: true runs on a worker with progress and
  Stop, keeping #106's atomic writes, is_valid checks and `repaired` count (also in
  library.previewProgress); the render worker's smart-preview fallback uses is_valid
- Rename preview planned on a worker (plan_rename_photos) on the same planning core as
  Session::plan_rename_with (MoveFs, case-only respelling by file identity, #95)
- Local folder listing via fs_cached; tests_offline (slow injected probe, frames < 300 ms)

Tests: worker-planned Find Missing equals the synchronous command for a renamed + ambiguous
library; a plan is re-checked when applied; background smart previews repair a damaged proxy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:58:04 -04:00
Brandon Thomas e11f77d9ca Merge pull request #129 from storytold/fix/issue-94-control-close
Issue #94: Control: close the connection on malformed input
2026-10-05 18:57:59 -04:00
Brandon Thomas f383d1e736 Merge main into fix/issue-94-control-close
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:57:50 -04:00
Brandon Thomas edf05789ed Merge pull request #128 from storytold/fix/issue-103-settings-files
Issue #103: Settings files: keep damaged ones, never overwrite unread ones, write ui.json atomically, confirm quit with unsaved changes
2026-10-05 18:57:30 -04:00
Brandon Thomas bccccfb92a Merge main into fix/issue-103-settings-files
# Conflicts:
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 18:56:50 -04:00
Brandon Thomas 4939d9b994 Merge pull request #127 from storytold/fix/issue-100-open-failure
Issue #100: Ask what to do when the library can't be opened, never a silent demo session
2026-10-05 18:56:35 -04:00
Brandon Thomas e0ab41ede0 Merge main into fix/issue-100-open-failure 2026-10-05 18:56:00 -04:00
Brandon Thomas 8690b51fcf Merge pull request #126 from storytold/fix/issue-99-library-lock
Issue #99: Library: exclusive lock so two processes can't open one library
2026-10-05 18:55:44 -04:00
Brandon Thomas e1d3b56bb6 Merge main into fix/issue-99-library-lock
# Conflicts:
#	crates/catalog/src/lib.rs
2026-10-05 18:54:56 -04:00
Brandon Thomas 6100237cdb Merge pull request #125 from storytold/fix/issue-102-format-version
Issue #102: Catalog format version 2; refuse newer libraries untouched
2026-10-05 18:54:41 -04:00
Brandon Thomas 4579599745 Merge main into fix/issue-102-format-version 2026-10-05 18:54:03 -04:00
Brandon Thomas 8506fecf5f Merge pull request #124 from storytold/fix/issue-101-torn-append
Issue #101: catalog.log: cut a partial append back before retrying
2026-10-05 18:53:49 -04:00
Brandon Thomas 0f49ebf8f0 Merge main into fix/issue-101-torn-append
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:52:48 -04:00
Brandon Thomas ab86862011 Merge pull request #123 from storytold/fix/issue-92-xmp-merge
Issue #92: XMP: merge into an existing sidecar instead of replacing it
2026-10-05 18:52:25 -04:00
Brandon Thomas 577330a08b Merge main into fix/issue-92-xmp-merge
# Conflicts:
#	crates/engine/src/sidecar.rs
#	docs/whats-new.md
2026-10-05 18:50:42 -04:00
Brandon Thomas 30697cfc26 Merge pull request #122 from storytold/fix/issue-92-rename-shared-sidecar
Issue #92: Rename Photos copies a stem sidecar another file still shares
2026-10-05 18:50:26 -04:00
Brandon Thomas ddbf139e5e Merge main into fix/issue-92-rename-shared-sidecar 2026-10-05 18:49:29 -04:00
Brandon Thomas ee77490dd2 Merge pull request #121 from storytold/fix/issue-105-rename-rollback
Issue #105: Rename Photos reports failed rollbacks; Find Missing matches by content
2026-10-05 18:49:15 -04:00
Brandon Thomas f31a55c8a7 Merge main into fix/issue-105-rename-rollback
# Conflicts:
#	docs/parity.md
2026-10-05 18:48:27 -04:00
Brandon Thomas 7218b990f5 Merge pull request #120 from storytold/fix/issue-95-case-only-rename
Issue #95: Rename Photos: case-only renames never overwrite another file
2026-10-05 18:48:06 -04:00
Brandon Thomas d16d9aadeb Merge main into fix/issue-95-case-only-rename 2026-10-05 18:47:18 -04:00
Brandon Thomas 1caa04acbc Merge pull request #119 from storytold/fix/issue-98-thumb-cache-own-files
Issue #98: thumbnail cache only lists, counts and deletes its own files
2026-10-05 18:47:03 -04:00
Brandon Thomas bef2060b24 Merge main into fix/issue-98-thumb-cache-own-files 2026-10-05 18:46:59 -04:00
Brandon Thomas 40303b999c Merge pull request #118 from storytold/fix/issue-96-verified-copy-import
Issue #96: Import > Copy verifies each copy like Move
2026-10-05 18:46:42 -04:00
Brandon Thomas ce5488ad99 Merge main into fix/issue-96-verified-copy-import
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:46:20 -04:00
Brandon Thomas f6bc1ae5ae Merge pull request #117 from storytold/fix/issue-106-dng-smart-previews
Issue #106: Convert to DNG and smart previews: verified, atomic writes
2026-10-05 18:46:06 -04:00
Brandon Thomas 6c8a65c337 Merge main into fix/issue-106-dng-smart-previews 2026-10-05 18:45:48 -04:00
Brandon Thomas fe386f4f15 Merge pull request #116 from storytold/fix/issue-93-export-overwrite
Issue #93: Export: never overwrite an original; atomic writes
2026-10-05 18:45:31 -04:00
Brandon Thomas 50cd8dc6dc Merge branch 'fix/issue-97-folder-undo' into integrate/safety
# Conflicts:
#	crates/engine/src/lib.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:38:55 -04:00
Brandon Thomas cc8b3eb3bb Merge branch 'fix/issue-94-control-close' into integrate/safety
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:38:13 -04:00
Brandon Thomas a6b402981b Merge branch 'fix/issue-103-settings-files' into integrate/safety
# Conflicts:
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 18:26:38 -04:00
Brandon Thomas 4deb5eff04 Merge branch 'fix/issue-100-open-failure' into integrate/safety 2026-10-05 18:26:38 -04:00
Brandon Thomas 6e2408ff7d Merge branch 'fix/issue-99-library-lock' into integrate/safety
# Conflicts:
#	crates/catalog/src/lib.rs
2026-10-05 18:26:38 -04:00
Brandon Thomas 66be31e3d8 Merge branch 'fix/issue-102-format-version' into integrate/safety 2026-10-05 18:26:19 -04:00
Brandon Thomas 332f37a5c7 Merge branch 'fix/issue-101-torn-append' into integrate/safety
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:26:19 -04:00
Brandon Thomas aa1af6d5e9 Merge branch 'fix/issue-92-xmp-merge' into integrate/safety
# Conflicts:
#	crates/engine/src/sidecar.rs
#	docs/whats-new.md
2026-10-05 18:26:19 -04:00
Brandon Thomas 963912b231 Merge branch 'fix/issue-92-rename-shared-sidecar' into integrate/safety 2026-10-05 18:25:06 -04:00
Brandon Thomas 4174ce8c57 Merge branch 'fix/issue-105-rename-rollback' into integrate/safety
# Conflicts:
#	docs/parity.md
2026-10-05 18:24:59 -04:00
Brandon Thomas f28e1c362e Merge branch 'fix/issue-95-case-only-rename' into integrate/safety 2026-10-05 18:24:34 -04:00
Brandon Thomas 70d800ba68 Merge branch 'fix/issue-98-thumb-cache-own-files' into integrate/safety 2026-10-05 18:24:33 -04:00
Brandon Thomas 33ef91a123 Merge branch 'fix/issue-96-verified-copy-import' into integrate/safety
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:24:33 -04:00
Brandon Thomas f1b29cfe4a Merge branch 'fix/issue-106-dng-smart-previews' into integrate/safety 2026-10-05 18:24:23 -04:00
Brandon Thomas 130b807e1b Merge branch 'fix/issue-93-export-overwrite' into integrate/safety 2026-10-05 18:24:22 -04:00
arata1972andCodex 43f4e24355 M11: separate Sony ARW colour and tone and preserve decoded source facts
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 07:16:15 +09:00
Brandon ThomasandClaude Opus 5.5 5e618c553a Issue #100: Ask what to do when the library can't be opened, never a silent demo session
When open_library failed, the desktop app (and the web build) quietly ran
an in-memory session seeded with the demo photos: it looked like a reset
library and everything done in it was lost at quit.

- The session now starts empty and in memory (no demo photos) and
  LightcraftApp::library_problem shows a blocking window with the path and
  the error: Try Again, Choose Another Library… (app.openLibrary), Continue
  Without Saving, Quit. Command-line files are imported once the user has
  chosen.
- A temporary session shows a banner under the top bar (with Open
  Library…) for as long as it lasts; it has no library, so nothing is
  written anywhere. Opening a library ends it.
- ui.inspect → libraryProblem; widget ids for automation; documented in
  control-protocol.md.
- Web build: same window (Try Again disabled: reloading the page retries).
- Headless UI test: locked library → window → Continue Without Saving →
  banner, nothing written to the library → lock released → Try Again opens
  it. Checked in the real app over the control channel too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 cbebabccad Issue #102: Catalog format version 2; refuse newer libraries untouched
main added Op::SetBrowsed, Catalog.browsed and Photo.local_baseline while
the snapshot still said version 1, so v0.2.0 read such a library in part
(unknown ops as log damage, unknown fields dropped on its next snapshot).

- journal::VERSION = 2 (now public), with a format table and rules in the
  journal docs; a test with an exhaustive match over Op makes adding a
  variant fail until it is assigned a version.
- Journal::open reads the snapshot header first and refuses a newer
  version with CatalogError::Newer ("written by a newer version of
  LightCraft"), even when its catalog wouldn't parse.
- A log record whose CRC matches but whose op doesn't parse is "newer",
  not damage or a torn tail: refused before any file is modified.
- Opening an older (or snapshot-less) library writes a snapshot in this
  format before anything is appended (LoadReport::upgraded_from); a new
  library gets one at once, so v0.2.0 refuses it instead of reading a bare
  log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 08c2acda26 Issue #101: catalog.log: cut a partial append back before retrying
A failed append (disk full, dropped share) could leave part of its batch in
catalog.log; the retry (same seqs) then shared a line with the fragment and
the next load treated it as mid-log damage, setting later edits aside.

- Journal::append cuts the log back to the last whole record on error (or,
  if that fails too, before the next append).
- FsStore drops a failed append handle so the next attempt reopens the file;
  truncate of a never-created file is a no-op.
- The loader recognises a CRC-valid record glued after a fragment (logs
  written by older builds) and skips only the fragment.
- log_bytes now counts the newline added to an unterminated last record.
- close_library writes the closing snapshot even when appending fails; the
  snapshot absorbs the queued ops (Journal::snapshot_with_unlogged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 1a05c3eed2 Issue #103: Settings files: keep damaged ones, never overwrite unread ones, write ui.json atomically, confirm quit with unsaved changes
- Library settings (prefs.json, presets.json, view.json): only a missing
  file means "defaults". A file that doesn't parse is kept as
  <name>.corrupt-<unix time> before the defaults are used; one that can't
  be read (locked, I/O error) is never written this session (save_prefs
  refuses, presets/view saves skip it). Warnings go to library.info →
  settingsWarnings and Session::take_library_warnings; the UI shows each
  once in a window (ui.inspect → notices, button:noticeOk).
- Desktop ui.json (holds the library location): written atomically (temp +
  fsync + rename + dir fsync), saved as soon as the library path changes and
  every few seconds when it changed, failures reported; a damaged file is
  renamed to ui.json.corrupt-<time> and reported, an unreadable one is not
  overwritten.
- Quit/close with changes only in memory: retry the save (log, then a
  snapshot of everything), and if it still fails cancel the close and ask:
  Try Saving Again / Quit Anyway / Cancel (ui.inspect → quitPrompt).
- Tests: engine (damaged prefs kept, unreadable presets/prefs untouched),
  desktop (damaged ui.json kept, atomic write), headless UI (quit prompt
  buttons, settings notice).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:12:26 -04:00
Brandon ThomasandClaude Opus 5.5 ee4b78f646 Issue #99: Library: exclusive lock so two processes can't open one library
Two processes on one library each kept their own catalog and seq counter;
the last snapshot silently dropped the other's edits.

- lightcraft_catalog:🔒 LibraryLock holds an exclusive OS lock
  (std File::try_lock: flock / LockFileEx) on <library>/catalog.lock for
  the session; who holds it goes to catalog.lock.owner (separate file: a
  Windows lock also blocks reads). Released on drop and by the OS when the
  process dies, so leftover files from a crash never lock anyone out. File
  systems without lock support open unprotected with a warning.
- Session::open_library takes the lock before reading anything and fails
  with EngineError::LibraryInUse ("this library is already open in
  LightCraft 0.x (process N on host)"); reopening the session's own
  library hands its lock over.
- lightcraft-cli (mcp/run/snapshot --library) adds how to drive the
  running app instead (--control + mcp --connect).
- Tests: in-process, leftover files, a killed child process, two engine
  sessions, CLI e2e (mcp --library holds it, run --library is refused).
  Existing tests that reopened a library while it was open now drop the
  first session.
- docs: parity row LRC-LIB-CATALOG-LOCK, mcp.md, web.md (no guard across
  browser tabs yet), whats-new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:48:34 -04:00
Brandon ThomasandClaude Opus 5.5 d592b29d12 Issue #97: Rename Folder / Move Folder are undo steps
The folder move and the photos' relinks form one undo step: undo renames
the folder back on disk (files and sidecars along) and restores the
paths, redo repeats it. A destination taken in the meantime is never
overwritten; the step is refused, reported and stays on the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:30 -04:00
Brandon ThomasandClaude Opus 5.5 cddcb905e0 Issue #94: Control: close the connection on malformed input
A line that is not a JSON object with a string `method` (HTTP request
line, invalid UTF-8, non-object JSON, >4 MiB line) now gets one error
reply and the connection is closed, so a browser's cross-origin fetch
can't smuggle a command in its body. Connections are capped at 16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:27 -04:00
Brandon ThomasandClaude Opus 5.5 ebc73bb7b4 Issue #107: Web build: data-safety fixes for libraries kept in the browser
- File > Back Up Library... (web): a zip of the catalog files and every
  stored original, built entry by entry; Restore Library from Backup...
  reads it into a new library folder and switches only when complete
  (the previous library stays in storage).
- Failed flushes to browser storage make the catalog Store refuse
  writes, so changes show as unsaved (top-bar warning) and are retried
  every 2 s instead of being lost silently.
- A picked/dropped photo whose bytes can't be stored is not added.
- One tab per library (Web Locks); a second tab shows a message.
- Notices: storage not persistent, library failed to open (temporary
  session), experimental build; ?reset asks first; a panic shows a
  message instead of a frozen page; long toasts wrap.
- Hosting: _headers/.htaccess no longer mark the unhashed .js/.wasm
  immutable; HOSTING.md rewritten for the actual build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:21 -04:00
Brandon ThomasandClaude Opus 5.5 e96a2750a1 Issue #96: Import > Copy verifies each copy like Move
Copy import used exists() then std::fs::copy: no sync, no comparison, and
a file appearing at the chosen name in between was replaced. It now goes
through import_move::copy_new, which reuses Move's copy_verified: a new
file (create_new, retrying -1, -2… on AlreadyExists), synced and compared
byte for byte with the source; a copy that fails or differs is removed and
reported as a failed import. Copy as DNG's copy step uses the same path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:22:35 -04:00
arata1972andCodex 509d5843dd M16: integrate expanded Japanese UI with upstream language settings
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 06:20:23 +09:00
Brandon ThomasandClaude Opus 5.5 771b0b98d7 Issue #106: Convert to DNG and smart previews: verified, atomic writes
- Convert to DNG / Copy as DNG: the encoded DNG must decode to the same raw
  data, then goes through safe_file::write_new_unique (temp file, sync,
  read-back, hard-linked to the first free <stem>.dng / -2 / -3 name, never
  replacing a file). On failure no DNG is left, the photo is not relinked
  and the raw copy is kept.
- Photo Merge writes its DNG the same way.
- Smart previews are written atomically; smart::is_valid (header + complete
  JPEG) decides "built", has_smart_preview and the offline fallback, so a
  proxy cut short is rebuilt (reported as `repaired`); moving proxies to a
  new folder never deletes a good one in favour of a damaged copy and
  copies across drives atomically.

Stacked on the #93 branch (shared lightcraft_catalog::safe_file helper).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:17:41 -04:00
Brandon ThomasandClaude Opus 5.5 f4df914731 Issue #92: Rename Photos copies a stem sidecar another file still shares
Renaming IMG_1.CR3 alone took the shared IMG_1.xmp along and left
IMG_1.JPG without it. A Stem-named sidecar that a sibling photo file
still uses is now copied (independently, never linked) instead of moved;
the last file renamed takes it. A rollback removes the copy.

Stacked on fix/issue-105-rename-rollback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:12:25 -04:00
Brandon ThomasandClaude Opus 5.5 882a059fac Issue #92: XMP: merge into an existing sidecar instead of replacing it
Saving (Cmd+S or auto-write) now merges LightCraft's properties into an
existing sidecar (lightcraft_meta::merge_xmp): owned properties (rating,
label, dc/IPTC/rights fields, lc:*) are replaced or removed, capture time
and GPS only when LightCraft has a value; everything else (crs:,
xmpMM:History, unknown namespaces, packet wrapper) is kept byte for byte.
The result is re-parsed and checked before it is written (temp + fsync +
rename). A sidecar that isn't valid XMP is copied to .xmp.bak-<time>
(never overwriting) before it is replaced; one that can't be read at all
is left alone and the save fails.

Shared stems: when several catalogued files map to one stem sidecar
(IMG.CR3 + IMG.JPG), it belongs to the raw (else the first by name) and
the others write and read <file>.xmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:08:18 -04:00
arata1972andCodex 05f242c445 M11: estimate missing ARW colour from guarded camera-preview correspondences
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 05:55:20 +09:00
Brandon ThomasandClaude Opus 5.5 fb5b18843f Issue #105: Rename Photos reports failed rollbacks; Find Missing matches by content
Rollbacks in move_file, apply_rename and undo/redo no longer ignore
errors: every move that can't be undone is reported (old -> new), and the
catalog follows the files that kept their new names (an undoable partial
rename; a logged change for undo/redo), so nothing shows as missing.
A file that can't go back keeps the sidecars that moved with it.

library.findMissing: same name + size confirmed by the stored content
hash first, then renamed files by size + content hash, then a unique
name + size match; several look-alikes are skipped and reported in
'ambiguous' instead of guessed. Files already used by a photo are skipped.

Stacked on fix/issue-95-case-only-rename (same functions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:47:50 -04:00
Brandon ThomasandClaude Opus 5.5 dfbb1ad935 Issue #95: Rename Photos: case-only renames never overwrite another file
Whether an existing target is the photo itself is decided by file identity
(dev+inode on Unix; an exact directory entry check elsewhere), not by a
lower-case name comparison, so on case-sensitive volumes img_1.JPG next to
IMG_1.JPG is a collision (-1 suffix). Moves use a no-replace rename (hard
link + unlink where supported) and case changes go through a unique temp
name. Tests model case-sensitive and case-insensitive volumes and check
the real disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:36:35 -04:00
arata1972andCodex 87e0181690 M16: preserve upstream defaults with an opt-in Japanese local build
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 05:36:19 +09:00
Brandon ThomasandClaude Opus 5.5 0fd0ffb783 Issue #98: thumbnail cache only lists, counts and deletes its own files
scan() now accepts only <2 hex>/<32 hex>.jpg (matching shard) and the
cache's own <32 hex>.tmp<id> temp files, so prune/clear never delete and
size() never counts foreign files in a shared thumbs/ folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:23:32 -04:00
arata1972andCodex 7c799a9acf M16: add Japanese UI localization and isolated local macOS packaging
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 05:15:04 +09:00
146 changed files with 12061 additions and 1730 deletions

No files matched your search

+4
View File
@@ -47,6 +47,10 @@ jobs:
- name: XML, desktop entry, AppStream, Flatpak manifest
run: |
xmllint --noout packaging/windows/lightcraft.wxs packaging/macos/*.plist* packaging/linux/*.xml*
# Windows Installer needs Icon ids ending in .ico or .exe, or shortcuts show a blank icon.
if grep -oE '<Icon Id="[^"]*"' packaging/windows/lightcraft.wxs | grep -vE '\.(ico|exe)"$'; then
echo "::error::Icon Id must end in .ico or .exe"; exit 1
fi
desktop-file-validate packaging/linux/ai.storyteller.lightcraft.desktop
sed -e 's/@VERSION@/0.0.0/' -e 's/@DATE@/2026-01-01/' packaging/linux/ai.storyteller.lightcraft.metainfo.xml.in >"$RUNNER_TEMP/ai.storyteller.lightcraft.metainfo.xml"
appstreamcli validate --no-net --explain "$RUNNER_TEMP/ai.storyteller.lightcraft.metainfo.xml"
+6 -2
View File
@@ -112,6 +112,9 @@ jobs:
include:
- { arch: x64, target: x86_64-pc-windows-msvc }
- { arch: x86, target: i686-pc-windows-msvc }
# Windows on ARM: cross-compiled on the x64 runner, so signing and WiX run as above.
# .github/workflows/windows-arm64.yml installs and runs it on ARM64 hardware.
- { arch: arm64, target: aarch64-pc-windows-msvc }
env:
LIGHTCRAFT_VERSION: ${{ needs.version.outputs.version }}
LIGHTCRAFT_BUILD_DATE: ${{ needs.version.outputs.date }}
@@ -221,8 +224,9 @@ jobs:
- name: Install wasm-bindgen CLI
run: |
# Must match the wasm-bindgen version in Cargo.lock exactly (cargo xtask web checks it).
command -v wasm-bindgen >/dev/null && wasm-bindgen --version | grep -q "$WASM_BINDGEN_VERSION" \
|| cargo install wasm-bindgen-cli --version "$WASM_BINDGEN_VERSION" --locked
if ! { command -v wasm-bindgen >/dev/null && wasm-bindgen --version | grep -q "$WASM_BINDGEN_VERSION"; }; then
cargo install wasm-bindgen-cli --version "$WASM_BINDGEN_VERSION" --locked
fi
wasm-bindgen --version
- name: Apply version override
if: inputs.version != ''
+95
View File
@@ -0,0 +1,95 @@
# Windows on ARM: package the ARM64 release the way release.yml does (cross-compiled on an x64
# runner, unsigned here), then install that MSI on a Windows 11 ARM64 runner, check the installed
# binaries are ARM64, run the CLI natively, and uninstall. (A native ARM64 `cargo test` job is left
# out for now: PrintCraft saw an intermittent access violation in egui UI tests there.)
name: Windows ARM64
on:
push:
branches: [main]
pull_request:
paths:
- Cargo.toml
- Cargo.lock
- "**/Cargo.toml"
- "**/build.rs"
- .github/workflows/windows-arm64.yml
- packaging/windows/**
schedule:
- cron: "41 4 * * 1" # weekly, so drift in the runner images shows up
workflow_dispatch:
concurrency:
group: windows-arm64-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
CARGO_INCREMENTAL: 0
jobs:
package:
name: package (cross-compiled on x64)
runs-on: windows-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@89b12181fb390509a0842a86cc55eeb8eb928c1d # stable
with:
targets: aarch64-pc-windows-msvc
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
key: windows-arm64-package
- name: Install WiX
shell: pwsh
run: dotnet tool install --global wix --version 5.0.2
- name: Build and package (unsigned)
shell: pwsh
env:
DIST: ${{ github.workspace }}\dist\arm64
run: ./packaging/windows/package.ps1 -Arch arm64
- uses: actions/upload-artifact@v4
with:
name: lightcraft-windows-arm64
path: dist/arm64/*
if-no-files-found: error
retention-days: 7
install:
name: install and run on ARM64
needs: package
runs-on: windows-11-arm
timeout-minutes: 30
steps:
- uses: actions/download-artifact@v4
with:
name: lightcraft-windows-arm64
path: dist
- name: Install the MSI, check it, run it, uninstall it
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
$msi = (Get-ChildItem dist/*.msi | Select-Object -First 1).FullName
$p = Start-Process msiexec.exe -ArgumentList '/i', "`"$msi`"", '/qn', '/l*v', 'install.log' -Wait -PassThru
if ($p.ExitCode -ne 0) { Get-Content install.log -Tail 60; throw "msiexec /i exited $($p.ExitCode)" }
$dir = Join-Path $env:ProgramFiles 'LightCraft'
foreach ($exe in 'lightcraft.exe', 'lightcraft-cli.exe') {
$path = Join-Path $dir $exe
if (-not (Test-Path $path)) { throw "$exe not installed in $dir" }
$bytes = [System.IO.File]::ReadAllBytes($path)
$machine = [BitConverter]::ToUInt16($bytes, [BitConverter]::ToInt32($bytes, 0x3C) + 4)
if ($machine -ne 0xAA64) { throw "$exe is for machine 0x$('{0:X}' -f $machine), not ARM64" }
Write-Output "ok $path (ARM64)"
}
# The installed CLI runs natively on ARM64.
$cli = Join-Path $dir 'lightcraft-cli.exe'
$out = & $cli --version 2>&1 | Out-String
if ($LASTEXITCODE -ne 0) { Write-Output $out; throw "lightcraft-cli --version exited $LASTEXITCODE" }
Write-Output "ok lightcraft-cli --version: $($out.Trim())"
$p = Start-Process msiexec.exe -ArgumentList '/x', "`"$msi`"", '/qn' -Wait -PassThru
if ($p.ExitCode -ne 0) { throw "msiexec /x exited $($p.ExitCode)" }
if (Test-Path (Join-Path $dir 'lightcraft.exe')) { throw 'uninstall left lightcraft.exe behind' }
Write-Output 'ok uninstalled'
Generated
+23 -22
View File
@@ -2073,7 +2073,7 @@ dependencies = [
[[package]]
name = "lightcraft"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"eframe",
"egui",
@@ -2091,7 +2091,7 @@ dependencies = [
[[package]]
name = "lightcraft-catalog"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"crc32fast",
"lightcraft-develop",
@@ -2105,7 +2105,7 @@ dependencies = [
[[package]]
name = "lightcraft-cli"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"dhat",
"lightcraft-codecs",
@@ -2120,7 +2120,7 @@ dependencies = [
[[package]]
name = "lightcraft-codecs"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"bytemuck",
"image",
@@ -2145,7 +2145,7 @@ dependencies = [
[[package]]
name = "lightcraft-color"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"lightcraft-geom",
"proptest",
@@ -2154,7 +2154,7 @@ dependencies = [
[[package]]
name = "lightcraft-develop"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"lightcraft-color",
"lightcraft-geom",
@@ -2165,7 +2165,7 @@ dependencies = [
[[package]]
name = "lightcraft-engine"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"ab_glyph",
"libc",
@@ -2193,7 +2193,7 @@ dependencies = [
[[package]]
name = "lightcraft-geom"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"proptest",
"serde",
@@ -2201,7 +2201,7 @@ dependencies = [
[[package]]
name = "lightcraft-gpu"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"bytemuck",
"lightcraft-color",
@@ -2217,7 +2217,7 @@ dependencies = [
[[package]]
name = "lightcraft-mcp"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"lightcraft-codecs",
"lightcraft-engine",
@@ -2228,7 +2228,7 @@ dependencies = [
[[package]]
name = "lightcraft-merge"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"lightcraft-codecs",
"lightcraft-color",
@@ -2244,7 +2244,7 @@ dependencies = [
[[package]]
name = "lightcraft-meta"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"lightcraft-geom",
"lightcraft-tiff",
@@ -2256,7 +2256,7 @@ dependencies = [
[[package]]
name = "lightcraft-pipeline"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"image",
"lightcraft-color",
@@ -2272,7 +2272,7 @@ dependencies = [
[[package]]
name = "lightcraft-preview"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"jpeg-encoder",
"lightcraft-raster",
@@ -2284,7 +2284,7 @@ dependencies = [
[[package]]
name = "lightcraft-raster"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"libc",
"lightcraft-color",
@@ -2296,7 +2296,7 @@ dependencies = [
[[package]]
name = "lightcraft-raw"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"jpeg-encoder",
"lightcraft-color",
@@ -2315,7 +2315,7 @@ dependencies = [
[[package]]
name = "lightcraft-scenes"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"image",
"lightcraft-color",
@@ -2326,11 +2326,11 @@ dependencies = [
[[package]]
name = "lightcraft-sysmem"
version = "0.2.0"
version = "0.2.1"
[[package]]
name = "lightcraft-tiff"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"proptest",
"serde",
@@ -2339,7 +2339,7 @@ dependencies = [
[[package]]
name = "lightcraft-ui-egui"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"egui",
"lightcraft-catalog",
@@ -2359,8 +2359,9 @@ dependencies = [
[[package]]
name = "lightcraft-web"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"crc32fast",
"eframe",
"egui",
"js-sys",
@@ -5127,7 +5128,7 @@ checksum = "e450f9b2ed1dff33c94c12589a87338689467b9c4f5d8a5710bd09a847d2c8a7"
[[package]]
name = "xtask"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"brotli",
"flate2",
+1 -1
View File
@@ -3,7 +3,7 @@ resolver = "3"
members = ["crates/*", "apps/*", "xtask"]
[workspace.package]
version = "0.2.0"
version = "0.2.1"
edition = "2024"
license = "MIT OR Apache-2.0"
rust-version = "1.90"
+8 -7
View File
@@ -24,19 +24,19 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
|---|---:|---|---|
| **Feature checklist** | 79% | P0 core and P1 nearly complete: import (Add / Copy / Move, templates, devices), library, grid/loupe/compare/survey, every Edit slider, curves, colour grading, masking tools, crop/Upright, heal/clone, presets/profiles, versions/history, sync, export, menus, shortcuts | P1: lens-profile database, content-aware fill (patch synthesis), video playback/trim |
| **RAW coverage** (formats people shoot) | ~50% | DNG (all kinds), CR2, ARW, NEF (uncompressed + Huffman lossless/lossy), uncompressed RAF/ORF, packed RW2, PEF; every container's embedded preview (incl. CR3) | **CR3** (every Canon since ~2018), compressed RAF/ORF, RW2 v4, Nikon lossy-after-split, Canon sRAW, HEIC/AVIF. Per-model verification is thin (~40 corpus files vs >1,000 models) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No camera colour calibration**: non-DNG raws use a neutral matrix, so colour is muted/inaccurate. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **Colour & image quality** | ~55–65% | Pipeline is complete and fast; GPU path CPU-exact within 1/255 | **No measured camera calibration database**: ARW has a guarded per-file embedded-JPEG colour estimate (docs/camera-preview-colour.md); other non-DNG raws and rejected estimates use a neutral matrix. Colour fidelity remains incomplete. No lens-profile database. No measured fidelity against Lightroom (tone, highlights, texture/clarity, NR, sharpening are tuned by eye) |
| **AI & computational** | ~15–20% | Assisted culling (focus, bursts), auto tone, HDR/panorama merge; subject/sky/background masks as classical heuristics | Real segmentation masks (subject, sky, people, objects, landscape, depth), AI denoise, super resolution, lens blur, generative remove, faces/people, natural-language search. **Blocked on a model strategy** (licensable weights or our own training; pure-Rust inference is feasible) |
| **Workflow & library** | ~85% (single machine) | Robust catalog (journal + snapshots, background compaction, crash-tested), 85k-photo libraries stay responsive, Local browsing with automatic cleanup, XMP interop, keywords, smart albums, Move import | Opening an 85k library takes 1.7–4.7 s; no cloud sync (out of scope), no tablet companion (#74, roadmap), shared albums, publish services, tethering |
| **Classic modules** | ~30% | Geotagging from GPX track logs, soft proofing (partial), slideshow (basic) | **Map view, Book, Print, Slideshow module, Web, publish services**: ~40 tracker rows ⬜ |
| **HDR & video** | 0% | | HDR edit/display/export; video play/trim/edit/export |
| **Platform & robustness** | ~70% | macOS native; Windows/Linux builds; web via WASM; no-panic lints workspace-wide, `unsafe` confined to `crates/sysmem`; failed saves are reported; GPU errors fall back to CPU | Windows installer UI unverified on Windows (PR #79); GPU path proven only on Apple + user reports; no localisation; accessibility partial; headless UI tests time out under machine load |
| **Platform & robustness** | ~70% | macOS native; Windows/Linux builds; web via WASM; no-panic lints workspace-wide, `unsafe` confined to `crates/sysmem`; failed saves are reported; GPU errors fall back to CPU | Windows installer UI unverified on Windows (PR #79); GPU path proven only on Apple + user reports; Japanese/English UI (see docs/localization-ja.md); remaining technical errors and other languages; accessibility partial; headless UI tests time out under machine load |
### By kind of user
| User | Readiness | What blocks them |
|---|---:|---|
| JPEG / DNG shooter, single machine | ~85% | Fidelity polish, AI masks |
| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour (muted colour on every photo) |
| Nikon / Sony / older-Canon raw shooter | ~65% | Camera colour fidelity and coverage (ARW preview estimates are only a starting point) |
| Canon CR3 / Fujifilm / Olympus shooter | ~35% | Their raws open as embedded previews only |
| Lightroom Classic power user | ~45% | Print, Book, Map, publish, tethering |
| Relies on AI (masks, denoise) | ~25% | No segmentation or AI denoise models |
@@ -46,7 +46,8 @@ duplicate Local entries #22, black GPU exports on an Intel iGPU #78).
Priorities, in order. Each points at tracker rows in [`docs/parity.md`](docs/parity.md) → *Top gaps*.
1. **Camera colour calibration of our own** (LR-PROF-CAMERACOLOR, P0): fit each camera to its own embedded JPEG, use
matrices the files carry themselves, then chart shots. Raises colour quality for every non-DNG raw at once.
matrices the files carry themselves, then chart shots. Sony ARW's file-local fit (matrix + tone curve from its own
JPEG) is the first step; generalise it to the other makes' raws (NEF, RW2, PEF, ORF…), then validate fidelity.
2. **Raw formats, clean-room** (LR-IMP-FORMATS, P0): **CR3** first, then compressed RAF / ORF, RW2 v4, NEF
lossy-after-split, sRAW. Decided 2026-10-05: write our own decoders from prose descriptions (never decoder source,
no LGPL dependency); compressed NEF (#86) is the template.
@@ -81,12 +82,12 @@ hardening (#78), copyright metadata (#51), GPX geotagging (#60), import tag help
| M8 | Heal / Remove | content-aware remove (PatchMatch), heal, clone, brush spots, visualize spots, red/pet eye | 6–10 | 🚧 (heal, clone, auto source, visualize spots, red/pet eye ✅; PatchMatch remove ⬜) |
| M9 | Presets, profiles, versions, sync | preset browser + amount, create/import presets, profile browser, versions, history, copy/paste/sync settings | 5–8 | ✅ |
| M10 | Export & share | export dialog (JPEG/PNG/TIFF/DNG/AVIF/JXL/original), sizing, sharpening, metadata, watermark, naming, batch jobs, XMP sidecars, HDR export | 6–10 | 🚧 (all formats incl. DNG/original, sizing, presets, background jobs ✅; JXL encode, HDR export ⬜) |
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; **camera colour calibration** ⬜ (top priority), CR3, compressed ORF/RAF ⬜) |
| M11 | RAW II | CR3, RAF (X-Trans), ORF, RW2, PEF, SRW, 3FR, IIQ + long tail; camera calibration DB; HEIC/AVIF/JXL import | 20–35 | 🚧 (RAF uncompressed, RW2 packed, PEF, ORF uncompressed ✅; **camera colour calibration** 🚧 (guarded ARW preview fitting; measured database still missing), CR3, compressed ORF/RAF ⬜) |
| M12 | AI & smart features | subject/sky/background/people/object masks, semantic search, faces/People (permissively licensed models, pure-Rust inference) | 20–40 | ⬜ |
| M13 | Merge | HDR merge (deghost), panorama (projections, boundary warp, fill edges), HDR panorama | 10–15 | ✅ |
| M14 | Video | import/playback/trim via FilmCraft crates, global edits + presets on video, video export | 6–10 | ⬜ |
| M15 | Classic modules | Map, Book, Slideshow, Print, Web; smart collections, stacks, virtual copies, publish services, tethering | 25–40 | 🚧 (smart albums, stacks, virtual copies, compare/survey ✅; Map/Book/Slideshow/Print/Web ⬜) |
| M16 | 1.0 polish | preferences, shortcut editor, accessibility, localization, packaging (dmg/msi/AppImage/web), hardening | 10–20 | 🚧 (settings, keyboard shortcuts sheet, packaging basics ✅; accessibility, localisation ⬜) |
| M16 | 1.0 polish | preferences, shortcut editor, accessibility, localization, packaging (dmg/msi/AppImage/web), hardening | 10–20 | 🚧 (settings, keyboard shortcuts sheet, packaging basics ✅; Japanese/English localisation 🟡; accessibility and other locales ⬜) |
## Parity estimate (feature count updated 2026-10-05; effort estimate from 2026-10-02)
@@ -175,7 +176,7 @@ Not decoded yet — preview only (no permissively licensed description; black-bo
from the same-colour pixel two to the left), the reconstruction rule for scales 2/3 is not established.
- **Olympus compressed ORF**, **Fujifilm compressed RAF**, **Canon CR3/CRX** (M11.1), **Canon sRAW/mRAW**, lossy DNG.
**Camera colour matrices:** non-DNG raws use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
**Camera colour matrices:** ARW files can use guarded, separate chromaticity and tone estimates from their own embedded JPEG (see `docs/camera-preview-colour.md`); this is a per-file camera-look estimate with relative WB, not measured calibration or absolute-Kelvin WB. Other non-DNG raws and rejected fits use the documented neutral fallback (camera RGB ≈ linear sRGB, flagged
`matrix_is_fallback`) with the file's as-shot white-balance multipliers. Clean sources to evaluate next: manufacturer
matrices stored in the files themselves (Olympus ImageProcessing `ColorMatrix`, Pentax/Panasonic equivalents) and our
own chart-based calibration (M11.4). Adobe matrices are never used.
+14 -3
View File
@@ -91,6 +91,17 @@ USAGE:
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;
/// Why `--library DIR` can't be opened; for a library open in another program, how to work with
/// that one instead (issue #99).
fn library_error(dir: &str, e: lightcraft_engine::EngineError) -> String {
match e {
lightcraft_engine::EngineError::LibraryInUse(why) => format!(
"{dir}: {why}\nTo work with the library while the app has it open, start the app with `--control PORT` and use `lightcraft-cli mcp --connect 127.0.0.1:PORT`."
),
e => format!("{dir}: {e}"),
}
}
fn main() -> ExitCode {
// `--features dhat-heap`: count allocations; the profile is written when `_heap` drops
// (LIGHTCRAFT_DHAT_FILE, default dhat-heap.json).
@@ -182,7 +193,7 @@ fn mcp(args: &[String]) -> Result<(), String> {
let mut h = match &library {
Some(dir) => {
let mut h = Headless::default();
let r = h.session.open_library(dir, demo).map_err(|e| e.to_string())?;
let r = h.session.open_library(dir, demo).map_err(|e| library_error(dir, e))?;
eprintln!("lightcraft-cli mcp: opened library {dir} ({r:?})");
h
}
@@ -419,7 +430,7 @@ fn run(args: &[String]) -> Result<(), String> {
let mut h = match &library {
Some(dir) => {
let mut h = Headless::default();
h.session.open_library(dir, demo).map_err(|e| e.to_string())?;
h.session.open_library(dir, demo).map_err(|e| library_error(dir, e))?;
h
}
None if demo => Headless::demo(),
@@ -586,7 +597,7 @@ fn snapshot(args: &[String]) -> Result<(), String> {
let mut session = match &library {
Some(dir) => {
let mut s = Session::new().with_fs();
s.open_library(dir, false).map_err(|e| format!("{dir}: {e}"))?;
s.open_library(dir, false).map_err(|e| library_error(dir, e))?;
s
}
None if files.is_empty() => Session::with_demo().with_fs(),
+64
View File
@@ -183,6 +183,42 @@ fn snapshot_subcommand_renders_the_ui_headlessly() {
assert_eq!((d.width, d.height), (640, 480));
}
/// Issue #136: with the GPU switched off from the environment the UI starts and renders on the CPU,
/// without creating a GPU device (no driver is loaded), and says why.
#[test]
fn snapshot_starts_without_a_gpu() {
let script = tmp("nogpu.jsonl");
std::fs::write(
&script,
format!(
"{}\n{}\n{}\n",
json!({"method": "ui.set", "params": {"view": "photoGrid"}}),
json!({"method": "engine.execute", "params": {"command": "app.gpu"}}),
json!({"method": "ui.screenshot"}),
),
)
.unwrap();
for (var, value, reason) in [("LIGHTCRAFT_GPU", "0", "LIGHTCRAFT_GPU=0"), ("LIGHTCRAFT_GPU_BACKEND", "off", "LIGHTCRAFT_GPU_BACKEND=off")] {
let out = tmp(&format!("nogpu-{var}.png"));
let o = Command::new(BIN)
.env_remove("LIGHTCRAFT_GPU")
.env_remove("LIGHTCRAFT_GPU_BACKEND")
.env(var, value)
.args(["snapshot", "--demo", "--script", script.to_str().unwrap(), "-o", out.to_str().unwrap(), "--size", "480x320"])
.output()
.unwrap();
assert!(o.status.success(), "{var}: {}", String::from_utf8_lossy(&o.stderr));
let replies: Vec<Value> = String::from_utf8_lossy(&o.stdout).lines().map(|l| serde_json::from_str(l).unwrap()).collect();
assert!(replies.iter().all(|r| r["ok"] == true), "{replies:?}");
let gpu = &replies[1]["result"];
assert_eq!(gpu["available"], false, "{var}: {gpu}");
assert_eq!(gpu["adapter"], Value::Null, "{var}: no device was created: {gpu}");
assert!(gpu["reason"].as_str().is_some_and(|r| r.contains(reason)), "{var}: {gpu}");
let d = lightcraft_codecs::decode(&std::fs::read(&out).unwrap(), Default::default()).unwrap();
assert_eq!((d.width, d.height), (480, 320));
}
}
fn run_cli(args: &[&str], stdin: Option<&str>) -> (bool, Vec<Value>, String) {
let mut child = Command::new(BIN).arg("run").args(args).stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(Stdio::piped()).spawn().unwrap();
if let Some(text) = stdin {
@@ -260,3 +296,31 @@ fn devices_are_listed_and_imported_from() {
let line: Value = serde_json::from_slice(o.stdout.split(|b| *b == b'\n').next().unwrap()).unwrap();
assert_eq!(line["result"]["candidates"].as_array().map(Vec::len), Some(1), "{line}");
}
/// Issue #99: a library open in one process (here `mcp --library`) is refused by a second one,
/// with who has it and how to drive the running app instead; free again once the first exits.
#[test]
fn a_library_open_in_another_process_is_refused() {
let lib = tmp("locked-lib");
let _ = std::fs::remove_dir_all(&lib);
let lib_s = lib.to_str().unwrap();
let mut holder =
Command::new(BIN).args(["mcp", "--library", lib_s]).stdin(Stdio::piped()).stdout(Stdio::null()).stderr(Stdio::piped()).spawn().unwrap();
let mut err = BufReader::new(holder.stderr.take().unwrap());
let mut line = String::new();
while !line.contains("opened library") {
line.clear();
assert!(err.read_line(&mut line).unwrap() > 0, "mcp exited before opening the library");
}
let (ok, _, stderr) = run_cli(&["--library", lib_s, "library.info"], None);
assert!(!ok);
assert!(stderr.contains("already open in lightcraft-cli") && stderr.contains(&format!("process {}", holder.id())), "{stderr}");
assert!(stderr.contains("mcp --connect"), "{stderr}");
drop(holder.stdin.take()); // EOF: the server exits and lets go of the library
assert!(holder.wait().unwrap().success());
let (ok, lines, stderr) = run_cli(&["--library", lib_s, "library.info"], None);
assert!(ok, "{stderr}");
assert_eq!(lines[0]["ok"], true);
let _ = std::fs::remove_dir_all(&lib);
}
+1
View File
@@ -21,6 +21,7 @@ lightcraft-preview = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
log = { workspace = true }
crc32fast = { workspace = true }
[target.'cfg(target_arch = "wasm32")'.dependencies]
eframe = { workspace = true, features = ["default_fonts", "glow"] }
+33 -10
View File
@@ -232,6 +232,18 @@ impl Backend {
}
}
/// Is there a file at `path`?
pub async fn exists(&self, path: &str) -> Result<bool, String> {
match self {
Backend::Opfs(root) => Ok(Self::opfs_file(root, path, false).await?.is_some()),
Backend::Idb(db) => {
let (_tx, store) = Self::idb_store(db, false)?;
let n = idb_req(&store.count_with_key(&path.into()).map_err(err)?).await?;
Ok(n.as_f64().is_some_and(|n| n > 0.0))
}
}
}
pub async fn remove(&self, path: &str) -> Result<(), String> {
match self {
Backend::Opfs(root) => {
@@ -309,14 +321,25 @@ async fn dir_keys(dir: &FileSystemDirectoryHandle) -> Result<Vec<String>, String
Ok(out)
}
/// Ask the browser not to evict our storage under pressure (best effort; fire and forget).
pub fn request_persistence() {
if let Some(sm) = storage_manager()
&& let Ok(p) = sm.persist()
{
wasm_bindgen_futures::spawn_local(async move {
let granted = JsFuture::from(p).await.ok().is_some_and(|v| v.is_truthy());
log::info!("lightcraft: persistent storage {}", if granted { "granted" } else { "not granted (best effort)" });
});
}
/// Ask the browser not to evict our storage under pressure; `on_result(granted)` once known
/// (`false` also when the browser has no storage manager).
pub fn request_persistence(on_result: impl FnOnce(bool) + 'static) {
let Some(sm) = storage_manager() else {
on_result(false);
return;
};
wasm_bindgen_futures::spawn_local(async move {
// already persistent: nothing to ask
let already = match sm.persisted() {
Ok(p) => JsFuture::from(p).await.ok().is_some_and(|v| v.is_truthy()),
Err(_) => false,
};
let granted = already
|| match sm.persist() {
Ok(p) => JsFuture::from(p).await.ok().is_some_and(|v| v.is_truthy()),
Err(_) => false,
};
log::info!("lightcraft: persistent storage {}", if granted { "granted" } else { "not granted" });
on_result(granted);
});
}
+306
View File
@@ -0,0 +1,306 @@
//! Library backups for the browser build: a plain zip (entries stored, not compressed — photos
//! don't shrink) of the library files and every stored original, built one entry at a time so
//! only one file is in wasm memory at once, and read back the same way for a restore.
//!
//! Layout of a backup:
//! - `library/catalog.snap`, `library/catalog.log`, `library/presets.json`, … (see
//! [`crate::files::LIBRARY_FILES`]);
//! - `originals/<content hash>/<file name>`: each imported photo's original bytes, under its own
//! name so an unzipped backup is browsable;
//! - `README.txt`.
//!
//! Restoring never deletes anything: the library files go to a new library folder in browser
//! storage, originals that aren't stored yet are added (they're named by content, so existing
//! ones are identical), and only then does a small pointer file ([`ACTIVE_LIBRARY`]) switch to
//! the restored library. The library that was there before stays in storage.
//!
//! No zip64: a backup holds at most [`MAX_BYTES`] and 65 535 entries.
/// Storage key of the file naming the library folder in use (absent: `library`).
pub const ACTIVE_LIBRARY: &str = "active-library";
/// The default library folder.
pub const DEFAULT_LIBRARY_DIR: &str = "library";
/// Largest backup (offsets are 32-bit without zip64).
pub const MAX_BYTES: u64 = 0xFFFF_FFFF - (64 << 20);
const LOCAL_SIG: u32 = 0x0403_4b50;
const CENTRAL_SIG: u32 = 0x0201_4b50;
const END_SIG: u32 = 0x0605_4b50;
/// General-purpose flag bit 11: names are UTF-8.
const UTF8: u16 = 0x0800;
/// 1980-01-01 00:00 in MS-DOS format (date, time).
const DOS_DATE: u16 = (1 << 5) | 1;
/// The readme inside every backup.
pub const README: &str = "LightCraft library backup (browser version)\n\n\
library/ the catalog (catalog.snap + catalog.log), presets, view state and preferences\n\
originals/ every imported photo, as originals/<content hash>/<file name>\n\n\
Restore it in LightCraft for the web with File > Restore Library from Backup...\n\
The photos in originals/ are the untouched files you imported; edits live in the catalog.\n";
/// Is `name` a safe library folder name (`library` or `library-<letters, digits, ->`)?
pub fn valid_library_dir(name: &str) -> bool {
name == DEFAULT_LIBRARY_DIR
|| name.strip_prefix("library-").is_some_and(|s| !s.is_empty() && s.len() <= 64 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-'))
}
/// Is `h` a plausible content hash (used as a storage key segment)?
pub fn valid_hash(h: &str) -> bool {
!h.is_empty() && h.len() <= 128 && h.chars().all(|c| c.is_ascii_alphanumeric())
}
/// Zip entry name for an original.
pub fn original_entry(hash: &str, file_name: &str) -> String {
let name: String = file_name.chars().map(|c| if c == '/' || c == '\\' || c.is_control() { '_' } else { c }).collect();
let name = if name.is_empty() || name == "." || name == ".." { "original".to_string() } else { name };
format!("originals/{hash}/{name}")
}
/// Where a backup entry goes when restoring into library folder `dir`: `Some(storage key)`, or
/// `None` to skip it (README, thumbnails, anything unknown or unsafe).
pub fn restore_key(entry: &str, dir: &str) -> Option<String> {
if let Some(name) = entry.strip_prefix("library/") {
return crate::files::LIBRARY_FILES.contains(&name).then(|| format!("{dir}/{name}"));
}
let rest = entry.strip_prefix("originals/")?;
let (hash, _) = rest.split_once('/').unwrap_or((rest, ""));
valid_hash(hash).then(|| format!("originals/{hash}"))
}
fn u16le(v: &mut Vec<u8>, x: u16) {
v.extend_from_slice(&x.to_le_bytes());
}
fn u32le(v: &mut Vec<u8>, x: u32) {
v.extend_from_slice(&x.to_le_bytes());
}
struct Central {
name: String,
crc: u32,
size: u32,
offset: u32,
}
/// Writes a stored (uncompressed) zip piece by piece: [`ZipWriter::entry`] returns the local
/// header to emit before the entry's bytes, [`ZipWriter::finish`] the central directory.
#[derive(Default)]
pub struct ZipWriter {
entries: Vec<Central>,
offset: u64,
}
impl ZipWriter {
/// The local header for `name` holding `data` (emit it, then `data`).
pub fn entry(&mut self, name: &str, data: &[u8]) -> Result<Vec<u8>, String> {
let size = u32::try_from(data.len()).map_err(|_| format!("{name}: larger than 4 GB"))?;
let name_len = u16::try_from(name.len()).map_err(|_| format!("{name}: name too long"))?;
if self.entries.len() >= usize::from(u16::MAX) {
return Err("too many files for one backup".into());
}
let header_len = 30 + u64::from(name_len);
if self.offset + header_len + u64::from(size) > MAX_BYTES {
return Err("the library is too large for one backup file (4 GB)".into());
}
let offset = u32::try_from(self.offset).map_err(|_| "backup too large".to_string())?;
let crc = crc32fast::hash(data);
let mut h = Vec::with_capacity(header_len as usize);
u32le(&mut h, LOCAL_SIG);
u16le(&mut h, 20); // version needed
u16le(&mut h, UTF8);
u16le(&mut h, 0); // stored
u16le(&mut h, 0); // time
u16le(&mut h, DOS_DATE);
u32le(&mut h, crc);
u32le(&mut h, size);
u32le(&mut h, size);
u16le(&mut h, name_len);
u16le(&mut h, 0); // extra
h.extend_from_slice(name.as_bytes());
self.offset += header_len + u64::from(size);
self.entries.push(Central { name: name.to_string(), crc, size, offset });
Ok(h)
}
/// The central directory and end record (emit last).
pub fn finish(self) -> Result<Vec<u8>, String> {
let mut cd = Vec::new();
for e in &self.entries {
u32le(&mut cd, CENTRAL_SIG);
u16le(&mut cd, 20); // made by
u16le(&mut cd, 20); // needed
u16le(&mut cd, UTF8);
u16le(&mut cd, 0);
u16le(&mut cd, 0);
u16le(&mut cd, DOS_DATE);
u32le(&mut cd, e.crc);
u32le(&mut cd, e.size);
u32le(&mut cd, e.size);
u16le(&mut cd, e.name.len() as u16);
u16le(&mut cd, 0); // extra
u16le(&mut cd, 0); // comment
u16le(&mut cd, 0); // disk
u16le(&mut cd, 0); // internal attrs
u32le(&mut cd, 0); // external attrs
u32le(&mut cd, e.offset);
cd.extend_from_slice(e.name.as_bytes());
}
let cd_offset = u32::try_from(self.offset).map_err(|_| "backup too large".to_string())?;
let cd_size = u32::try_from(cd.len()).map_err(|_| "backup too large".to_string())?;
let n = self.entries.len() as u16;
let mut out = cd;
u32le(&mut out, END_SIG);
u16le(&mut out, 0);
u16le(&mut out, 0);
u16le(&mut out, n);
u16le(&mut out, n);
u32le(&mut out, cd_size);
u32le(&mut out, cd_offset);
u16le(&mut out, 0); // comment
Ok(out)
}
}
/// One entry of a zip's central directory.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ZipEntry {
pub name: String,
pub crc: u32,
pub size: u64,
/// Offset of the entry's local header.
pub header: u64,
/// 0 = stored (the only method a restore reads).
pub method: u16,
}
fn rd16(b: &[u8], at: usize) -> Option<u16> {
Some(u16::from_le_bytes(b.get(at..at + 2)?.try_into().ok()?))
}
fn rd32(b: &[u8], at: usize) -> Option<u32> {
Some(u32::from_le_bytes(b.get(at..at + 4)?.try_into().ok()?))
}
/// How many bytes from the end of a zip to read to find its end record.
pub fn tail_len(file_len: u64) -> u64 {
file_len.min(22 + 65_535)
}
/// The central directory's (offset, size) from the last [`tail_len`] bytes of a zip.
pub fn find_central(tail: &[u8]) -> Result<(u64, u64), String> {
let at = (0..tail.len().saturating_sub(21)).rev().find(|&i| rd32(tail, i) == Some(END_SIG)).ok_or("not a zip file (no end record)")?;
let size = rd32(tail, at + 12).ok_or("truncated zip")?;
let offset = rd32(tail, at + 16).ok_or("truncated zip")?;
if size == u32::MAX || offset == u32::MAX {
return Err("zip64 backups are not supported".into());
}
Ok((u64::from(offset), u64::from(size)))
}
/// Parse a central directory.
pub fn parse_central(cd: &[u8]) -> Result<Vec<ZipEntry>, String> {
let mut out = Vec::new();
let mut i = 0;
while i + 46 <= cd.len() {
if rd32(cd, i) != Some(CENTRAL_SIG) {
return Err("damaged zip directory".into());
}
let bad = || "damaged zip directory".to_string();
let method = rd16(cd, i + 10).ok_or_else(bad)?;
let crc = rd32(cd, i + 16).ok_or_else(bad)?;
let size = rd32(cd, i + 24).ok_or_else(bad)?;
let n = usize::from(rd16(cd, i + 28).ok_or_else(bad)?);
let extra = usize::from(rd16(cd, i + 30).ok_or_else(bad)?);
let comment = usize::from(rd16(cd, i + 32).ok_or_else(bad)?);
let header = rd32(cd, i + 42).ok_or_else(bad)?;
let name = cd.get(i + 46..i + 46 + n).ok_or_else(bad)?;
out.push(ZipEntry { name: String::from_utf8_lossy(name).into_owned(), crc, size: u64::from(size), header: u64::from(header), method });
i += 46 + n + extra + comment;
}
Ok(out)
}
/// Offset of an entry's data from its local header's first 30 bytes.
pub fn data_offset(entry: &ZipEntry, local: &[u8]) -> Result<u64, String> {
if rd32(local, 0) != Some(LOCAL_SIG) {
return Err(format!("{}: damaged entry", entry.name));
}
let n = rd16(local, 26).ok_or("truncated entry")?;
let extra = rd16(local, 28).ok_or("truncated entry")?;
Ok(entry.header + 30 + u64::from(n) + u64::from(extra))
}
/// Check an entry's bytes.
pub fn verify(entry: &ZipEntry, data: &[u8]) -> Result<(), String> {
if entry.method != 0 {
return Err(format!("{}: compressed entries are not supported (not a LightCraft backup?)", entry.name));
}
if data.len() as u64 != entry.size || crc32fast::hash(data) != entry.crc {
return Err(format!("{}: damaged (checksum mismatch)", entry.name));
}
Ok(())
}
/// A name for the restored library's folder.
pub fn restored_dir_name(now_ms: f64) -> String {
format!("library-restored-{}", now_ms.max(0.0) as u64)
}
#[cfg(test)]
mod tests {
use super::*;
fn build(files: &[(&str, &[u8])]) -> Vec<u8> {
let mut w = ZipWriter::default();
let mut out = Vec::new();
for (n, d) in files {
out.extend(w.entry(n, d).unwrap());
out.extend_from_slice(d);
}
out.extend(w.finish().unwrap());
out
}
#[test]
fn written_backups_read_back() {
let files: [(&str, &[u8]); 3] =
[("README.txt", README.as_bytes()), ("library/catalog.log", b"{\"seq\":1}\n"), ("originals/abc123/IMG 1.jpg", &[0, 1, 2, 3, 255])];
let zip = build(&files);
let tail = &zip[zip.len() - tail_len(zip.len() as u64) as usize..];
let (off, size) = find_central(tail).unwrap();
let entries = parse_central(&zip[off as usize..(off + size) as usize]).unwrap();
assert_eq!(entries.len(), 3);
for (e, (name, data)) in entries.iter().zip(files) {
assert_eq!(e.name, name);
let start = data_offset(e, &zip[e.header as usize..e.header as usize + 30]).unwrap() as usize;
let got = &zip[start..start + e.size as usize];
assert_eq!(got, data);
verify(e, got).unwrap();
}
// damage is caught
let e = &entries[2];
assert!(verify(e, &[0, 1, 2, 3, 254]).is_err());
assert!(find_central(b"not a zip at all, just some text").is_err());
}
#[test]
fn restore_keys_are_safe() {
assert_eq!(restore_key("library/catalog.snap", "library-restored-1").as_deref(), Some("library-restored-1/catalog.snap"));
assert_eq!(restore_key("library/../../x", "library-r"), None);
assert_eq!(restore_key("library/evil.bin", "library-r"), None);
assert_eq!(restore_key("originals/abc123/IMG.jpg", "d").as_deref(), Some("originals/abc123"));
assert_eq!(restore_key("originals/../library/catalog.log", "d"), None);
assert_eq!(restore_key("README.txt", "d"), None);
assert_eq!(restore_key("thumbs/x.jpg", "d"), None);
assert_eq!(original_entry("abc", "a/b\\c.jpg"), "originals/abc/a_b_c.jpg");
assert_eq!(original_entry("abc", ".."), "originals/abc/original");
assert!(valid_library_dir("library") && valid_library_dir("library-restored-17"));
assert!(!valid_library_dir("library-../x") && !valid_library_dir("originals") && !valid_library_dir("library-"));
}
#[test]
fn size_limits_are_errors_not_panics() {
let mut w = ZipWriter { offset: MAX_BYTES - 10, ..Default::default() };
assert!(w.entry("x", &[0; 64]).is_err());
}
}
+58
View File
@@ -49,6 +49,8 @@ struct Mirror {
files: BTreeMap<String, Vec<u8>>,
/// Dirty files, least recently modified first.
dirty: Vec<(String, Dirty)>,
/// Why the last flush to browser storage failed (quota, eviction…), until one succeeds.
error: Option<String>,
}
impl Mirror {
@@ -138,6 +140,18 @@ impl Files {
}
}
/// Record the outcome of a flush to browser storage: `Some(why)` when it failed, `None` once
/// one succeeds. While a flush is failing, the catalog [`Store`] refuses new writes, so the
/// engine keeps the changes queued and reports them unsaved (`library.info` → `unsavedOps`,
/// the top bar's warning) instead of pretending they were saved.
pub fn set_error(&self, e: Option<String>) {
self.lock().error = e;
}
pub fn error(&self) -> Option<String> {
self.lock().error.clone()
}
/// A catalog [`Store`] over these files.
pub fn store(&self) -> FilesStore {
FilesStore(self.clone())
@@ -147,19 +161,32 @@ impl Files {
/// [`Store`] implementation over [`Files`].
pub struct FilesStore(Files);
impl FilesStore {
/// Writes fail while browser storage does (see [`Files::set_error`]).
fn writable(&self) -> io::Result<()> {
match self.0.error() {
Some(e) => Err(io::Error::other(format!("browser storage: {e}"))),
None => Ok(()),
}
}
}
impl Store for FilesStore {
fn read(&mut self, name: &str) -> io::Result<Option<Vec<u8>>> {
Ok(self.0.get(name))
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> io::Result<()> {
self.writable()?;
self.0.write(name, data);
Ok(())
}
fn append(&mut self, name: &str, data: &[u8]) -> io::Result<()> {
self.writable()?;
self.0.append(name, data);
Ok(())
}
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()> {
self.writable()?;
self.0.truncate(name, len as usize);
Ok(())
}
@@ -224,6 +251,37 @@ mod tests {
assert_eq!(names, ["catalog.snap", "catalog.log"]);
}
/// A failing flush (quota exceeded, storage evicted) is not silent: commands report the change
/// as unsaved and the engine keeps it queued; once storage works again it is written.
#[test]
fn flush_failures_show_as_unsaved_and_are_retried() {
let mut disk = BTreeMap::new();
let files = Files::default();
let mut s = Session::new();
let stores = LibraryStores { dir: "browser".into(), catalog: Box::new(files.store()), files: Box::new(files.store()), on_disk: false };
s.open_library_in(stores, true).unwrap();
flush(&files, &mut disk);
files.set_error(Some("QuotaExceededError".into()));
let r = s.execute("photo.rate", &json!({"rating": 5}));
assert!(matches!(r, Err(lightcraft_engine::EngineError::NotSaved(_))), "{r:?}");
let (ops, why) = s.unsaved().unwrap();
assert!(ops > 0 && why.contains("QuotaExceeded"), "{why}");
files.set_error(None);
s.persist_if_dirty();
s.persist().unwrap();
assert!(s.unsaved().is_none(), "written once storage works again");
flush(&files, &mut disk);
let id = s.selection.active.unwrap();
let files2 = Files::default();
for (k, v) in &disk {
files2.preload(k, v.clone());
}
let mut s2 = Session::new();
let stores = LibraryStores { dir: "browser".into(), catalog: Box::new(files2.store()), files: Box::new(files2.store()), on_disk: false };
s2.open_library_in(stores, true).unwrap();
assert_eq!(s2.catalog.photo(id).unwrap().rating, 5, "the rating survived");
}
#[test]
fn library_survives_reload_through_flushes() {
let mut disk = BTreeMap::new();
+3
View File
@@ -16,6 +16,7 @@
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
pub mod backup;
pub mod bench;
pub mod files;
pub mod store;
@@ -24,6 +25,8 @@ pub mod wire;
#[cfg(target_arch = "wasm32")]
mod backend;
#[cfg(target_arch = "wasm32")]
mod safety;
#[cfg(target_arch = "wasm32")]
mod web;
#[cfg(target_arch = "wasm32")]
mod workers;
+256
View File
@@ -0,0 +1,256 @@
//! Keeping a library in the browser safe: notices the user must see (a failed save, storage the
//! browser may evict, a file that couldn't be stored), one tab at a time per library, a message
//! instead of a dead page after a panic, and backup / restore of the whole library
//! ([`crate::backup`]).
use std::cell::RefCell;
use std::collections::HashMap;
use js_sys::{Reflect, Uint8Array};
use wasm_bindgen::JsCast;
use wasm_bindgen::prelude::*;
use wasm_bindgen_futures::JsFuture;
use crate::backend::Backend;
use crate::backup::{
ACTIVE_LIBRARY, README, ZipWriter, data_offset, find_central, original_entry, parse_central, restore_key, restored_dir_name, tail_len,
valid_hash, verify,
};
use crate::files::{Files, LIBRARY_FILES};
use crate::store::storage_key;
thread_local! {
/// Messages for the user, shown by the app on its next frame.
static NOTICES: RefCell<Vec<String>> = const { RefCell::new(Vec::new()) };
}
/// Queue a message for the user (shown for several seconds; also logged).
pub fn notice(text: impl Into<String>) {
let text = text.into();
log::warn!("{text}");
NOTICES.with(|n| n.borrow_mut().push(text));
crate::web::request_repaint();
}
/// Messages queued since the last call.
pub fn take_notices() -> Vec<String> {
NOTICES.with(|n| std::mem::take(&mut *n.borrow_mut()))
}
fn document() -> Option<web_sys::Document> {
web_sys::window()?.document()
}
/// Ask the user (the browser's own confirm box). `false` when it can't be shown.
pub fn confirm(text: &str) -> bool {
web_sys::window().and_then(|w| w.confirm_with_message(text).ok()).unwrap_or(false)
}
/// Cover the page with a message (the app can't continue, or must not start).
pub fn show_blocking(text: &str) {
let Some(doc) = document() else { return };
let el = match doc.get_element_by_id("lightcraft_blocking") {
Some(el) => el,
None => {
let Ok(el) = doc.create_element("div") else { return };
el.set_id("lightcraft_blocking");
let _ = el.set_attribute(
"style",
"position:fixed;inset:0;z-index:10;display:flex;align-items:center;justify-content:center;padding:32px;\
background:#1c1c1c;color:#e0e0e0;font:15px/1.5 system-ui,-apple-system,'Segoe UI',sans-serif;text-align:center;white-space:pre-line",
);
if let Some(body) = doc.body() {
let _ = body.append_child(&el);
}
el
}
};
el.set_text_content(Some(text));
if let Some(loading) = doc.get_element_by_id("lightcraft_loading") {
loading.remove();
}
}
/// `panic = "abort"` on wasm: a panic ends the module. Say so on the page instead of leaving a
/// frozen canvas, and say that the stored library is intact.
pub fn install_panic_hook() {
std::panic::set_hook(Box::new(|info| {
let msg = match (info.payload().downcast_ref::<&str>(), info.payload().downcast_ref::<String>()) {
(Some(s), _) => (*s).to_string(),
(_, Some(s)) => s.clone(),
_ => "unknown error".into(),
};
let at = info.location().map(|l| format!(" at {}:{}", l.file(), l.line())).unwrap_or_default();
log::error!("LightCraft panicked{at}: {msg}");
show_blocking(&format!(
"LightCraft stopped after an internal error:\n{msg}\n\nThe library saved in this browser is kept. Reload the page to continue.\n\
If it happens again right away, a photo may be the cause: reload with ?workers=0 or report the error."
));
}));
}
/// The name of the lock that keeps a library to one tab.
const TAB_LOCK: &str = "lightcraft-library";
/// Hold the library lock for as long as this page lives (Web Locks API). `Some(true)`: ours;
/// `Some(false)`: another tab of this site has it; `None`: the browser can't tell (no Web Locks).
pub async fn acquire_tab_lock() -> Option<bool> {
let nav = Reflect::get(&js_sys::global(), &"navigator".into()).ok()?;
let locks = Reflect::get(&nav, &"locks".into()).ok().filter(|l| !l.is_undefined() && !l.is_null())?;
let request: js_sys::Function = Reflect::get(&locks, &"request".into()).ok()?.dyn_into().ok()?;
let opts = js_sys::Object::new();
Reflect::set(&opts, &"ifAvailable".into(), &true.into()).ok()?;
let granted = js_sys::Promise::new(&mut |resolve, _reject| {
let cb = Closure::once_into_js(move |lock: JsValue| -> JsValue {
let ours = !lock.is_null() && !lock.is_undefined();
let _ = resolve.call1(&JsValue::NULL, &ours.into());
// keep the lock until the page goes away: a promise that never settles
if ours { js_sys::Promise::new(&mut |_, _| {}).into() } else { JsValue::NULL }
});
if let Err(e) = request.call3(&locks, &TAB_LOCK.into(), &opts, &cb) {
log::warn!("Web Locks request failed: {e:?}");
}
});
JsFuture::from(granted).await.ok().map(|v| v.is_truthy())
}
/// Offer a blob as a download named `name`.
pub fn download_blob(name: &str, blob: &web_sys::Blob) -> Result<(), String> {
let e = |e: JsValue| format!("download failed: {e:?}");
let url = web_sys::Url::create_object_url_with_blob(blob).map_err(e)?;
let doc = document().ok_or("no document")?;
let a: web_sys::HtmlAnchorElement = doc.create_element("a").map_err(e)?.unchecked_into();
a.set_href(&url);
a.set_download(name);
a.click();
// revoke once the download has started
let revoke = Closure::once_into_js(move || {
let _ = web_sys::Url::revoke_object_url(&url);
});
if let Some(w) = web_sys::window() {
let _ = w.set_timeout_with_callback_and_timeout_and_arguments_0(revoke.unchecked_ref(), 60_000);
}
Ok(())
}
/// File ▸ Back Up Library…: a zip of the library files (as the app has them, saved or not) and
/// every original in storage, built entry by entry into the browser's memory (outside the wasm
/// heap) and downloaded. `names`: content hash → the photo's file name.
pub async fn backup(files: Files, backend: Backend, names: HashMap<String, String>) -> Result<(usize, f64), String> {
let parts = js_sys::Array::new();
let mut zip = ZipWriter::default();
let mut bytes = 0f64;
let mut add = |zip: &mut ZipWriter, name: &str, data: &[u8]| -> Result<(), String> {
let header = zip.entry(name, data)?;
parts.push(&Uint8Array::from(&header[..]));
parts.push(&Uint8Array::from(data));
bytes += (header.len() + data.len()) as f64;
Ok(())
};
add(&mut zip, "README.txt", README.as_bytes())?;
for name in LIBRARY_FILES {
if let Some(d) = files.get(name) {
add(&mut zip, &format!("library/{name}"), &d)?;
}
}
let mut originals = 0;
for hash in backend.list("originals").await? {
if !valid_hash(&hash) {
continue;
}
match backend.read(&storage_key(&hash)).await {
Ok(Some(data)) => {
let name = names.get(&hash).map_or("original", String::as_str);
add(&mut zip, &original_entry(&hash, name), &data)?;
originals += 1;
}
Ok(None) => {}
Err(e) => return Err(format!("reading an original from browser storage: {e}")),
}
}
let tail = zip.finish()?;
parts.push(&Uint8Array::from(&tail[..]));
let opts = web_sys::BlobPropertyBag::new();
opts.set_type("application/zip");
let blob = web_sys::Blob::new_with_u8_array_sequence_and_options(&parts, &opts).map_err(|e| format!("{e:?}"))?;
let day: String = String::from(js_sys::Date::new_0().to_iso_string()).chars().take(10).collect();
download_blob(&format!("lightcraft-library-backup-{day}.zip"), &blob)?;
Ok((originals, bytes + tail.len() as f64))
}
/// Bytes `start..end` of a picked file.
async fn read_range(file: &web_sys::File, start: u64, end: u64) -> Result<Vec<u8>, String> {
let blob = file.slice_with_f64_and_f64(start as f64, end as f64).map_err(|e| format!("{e:?}"))?;
let buf = JsFuture::from(blob.array_buffer()).await.map_err(|e| format!("{e:?}"))?;
Ok(Uint8Array::new(&buf).to_vec())
}
/// Restore a backup into a new library folder (nothing in storage is deleted or overwritten),
/// then point [`ACTIVE_LIBRARY`] at it. Returns the folder's name; the caller reloads the page.
pub async fn restore(file: web_sys::File, backend: Backend) -> Result<String, String> {
let len = file.size() as u64;
let tail = read_range(&file, len - tail_len(len), len).await?;
let (cd_offset, cd_size) = find_central(&tail)?;
let cd = read_range(&file, cd_offset, cd_offset + cd_size).await?;
let entries = parse_central(&cd)?;
if !entries.iter().any(|e| e.name == "library/catalog.snap" || e.name == "library/catalog.log") {
return Err("this isn't a LightCraft library backup (no library/catalog files)".into());
}
let dir = restored_dir_name(js_sys::Date::now());
let mut written = 0usize;
for e in &entries {
let Some(key) = restore_key(&e.name, &dir) else { continue };
// originals are named by their content: one already stored is the same file
if key.starts_with("originals/") && backend.exists(&key).await? {
continue;
}
let local = read_range(&file, e.header, e.header + 30).await?;
let start = data_offset(e, &local)?;
let data = read_range(&file, start, start + e.size).await?;
verify(e, &data)?;
backend.write(&key, &data).await.map_err(|err| format!("storing {}: {err}", e.name))?;
written += 1;
}
backend.write(ACTIVE_LIBRARY, dir.as_bytes()).await?;
log::info!("lightcraft: restored {written} files into {dir}");
Ok(dir)
}
/// File ▸ Restore Library from Backup…: confirm, pick the zip, restore, reload.
pub fn pick_and_restore(backend: Backend, before_reload: impl FnOnce() + 'static) {
let ok = confirm(
"Restore a LightCraft library backup?\n\nThe backup becomes the library in this browser and the page reloads. \
The current library is not deleted: it stays in browser storage.",
);
if !ok {
return;
}
let Some(doc) = document() else { return };
let Ok(input) = doc.create_element("input").map(|e| e.unchecked_into::<web_sys::HtmlInputElement>()) else { return };
input.set_type("file");
input.set_accept(".zip,application/zip");
let inp = input.clone();
let mut before_reload = Some(before_reload);
let on_change = Closure::<dyn FnMut()>::new(move || {
let Some(file) = inp.files().and_then(|f| f.get(0)) else { return };
let backend = backend.clone();
let before = before_reload.take();
notice(format!("Restoring {}…", file.name()));
wasm_bindgen_futures::spawn_local(async move {
match restore(file, backend).await {
Ok(_) => {
if let Some(f) = before {
f();
}
if let Some(w) = web_sys::window() {
let _ = w.location().reload();
}
}
Err(e) => notice(format!("Restore failed: {e}. Nothing was changed: the current library is still in use.")),
}
});
});
input.set_onchange(Some(on_change.as_ref().unchecked_ref()));
on_change.forget();
input.click();
}
+154 -23
View File
@@ -15,16 +15,21 @@ use wasm_bindgen::JsCast;
use wasm_bindgen::prelude::*;
use crate::backend::Backend;
use crate::backup::{ACTIVE_LIBRARY, DEFAULT_LIBRARY_DIR, valid_library_dir};
use crate::bench::Bench;
use crate::files::{Files, FlushOp, LIBRARY_FILES};
use crate::safety::{self, notice};
use crate::store::{Originals, content_hash, download_name, storage_key};
use crate::wire::ThumbIndex;
use crate::workers::{THUMB_INDEX, Workers};
const ACCEPT: &str = ".jpg,.jpeg,.png,.tif,.tiff,.webp,.dng,.cr2,.nef,.arw,.psd,.jxl,.gif,.bmp";
/// Where the library files live in browser storage.
const LIBRARY_DIR: &str = "library";
/// After a failed save, wait this long before trying again (ms).
const SAVE_RETRY_MS: f64 = 2000.0;
/// How long a notice stays on screen (s).
const NOTICE_SECS: f64 = 8.0;
/// How often view state, UI prefs and the thumbnail index are saved (ms).
const SAVE_EVERY_MS: f64 = 1000.0;
@@ -45,8 +50,18 @@ fn set_status(text: &str) {
}
}
fn library_key(name: &str) -> String {
format!("{LIBRARY_DIR}/{name}")
/// Storage key of library file `name` in library folder `dir`.
fn library_key(dir: &str, name: &str) -> String {
format!("{dir}/{name}")
}
/// Repaint the app (notices from async tasks).
pub(crate) fn request_repaint() {
CTX.with(|c| {
if let Some(c) = c.borrow().as_ref() {
c.request_repaint();
}
});
}
/// URL options (`?bench&store=idb&workers=2&reset`).
@@ -86,8 +101,9 @@ async fn store_file(originals: Originals, backend: Option<Backend>, name: String
if let Some(b) = &backend
&& let Err(e) = b.write(&storage_key(&hash), &bytes).await
{
// still importable for this session; it won't survive a reload
log::error!("storing {name} in browser storage failed: {e}");
// a photo whose bytes aren't stored would be gone after a reload: don't add it
notice(format!("{name} was not added: browser storage refused it ({e}). Free up space or back up and clear old photos."));
return;
}
originals.added(&name, &hash, Arc::from(bytes));
ctx.request_repaint();
@@ -148,8 +164,42 @@ fn download(path: &str, bytes: &[u8]) -> Result<(), String> {
Ok(())
}
fn services(originals: Originals, backend: Option<Backend>, ctx: egui::Context) -> Services {
fn services(originals: Originals, backend: Option<Backend>, files: Files, frozen: Rc<Cell<bool>>, ctx: egui::Context) -> Services {
let (backup_backend, backup_files) = (backend.clone(), files);
let restore_backend = backend.clone();
Services {
backup_library: Some(Box::new(move |session: &mut Session| {
let Some(b) = backup_backend.clone() else { return Err("nothing is stored in this browser session (?store=memory)".into()) };
// the photos' own names for the originals in the zip
let names: std::collections::HashMap<String, String> = session
.catalog
.photos()
.filter_map(|p| match &p.source {
Source::File { path } => crate::store::hash_of_path(path).map(|h| (h.to_string(), p.file_name.clone())),
_ => None,
})
.collect();
let files = backup_files.clone();
notice("Backing up the library… (the download starts when it's ready)");
wasm_bindgen_futures::spawn_local(async move {
match safety::backup(files, b, names).await {
Ok((n, bytes)) => notice(format!(
"Library backed up: {n} original{} ({:.1} MB). Keep the file somewhere safe.",
if n == 1 { "" } else { "s" },
bytes / 1e6
)),
Err(e) => notice(format!("Backup failed: {e}")),
}
});
Ok(json!({"started": true}))
})),
restore_library: Some(Box::new(move |_session: &mut Session| {
let Some(b) = restore_backend.clone() else { return Err("nothing is stored in this browser session (?store=memory)".into()) };
let frozen = frozen.clone();
// once the backup is in place, stop saving the old library (the page reloads)
safety::pick_and_restore(b, move || frozen.set(true));
Ok(json!({"started": true}))
})),
pick_files: Some(Box::new(move || {
open_picker(originals.clone(), backend.clone(), ctx.clone());
Vec::new() // files arrive asynchronously and are imported on a later frame
@@ -176,21 +226,29 @@ fn services(originals: Originals, backend: Option<Backend>, ctx: egui::Context)
}
}
/// Write dirty library files to storage, in order, until none are left.
async fn flush(files: Files, backend: Backend, flushing: Rc<Cell<bool>>) {
/// Write dirty library files to storage, in order, until none are left. A failure (quota,
/// storage cleared) is recorded in `files` — the catalog then reports changes as unsaved instead
/// of accepting them silently — and retried after [`SAVE_RETRY_MS`].
async fn flush(files: Files, backend: Backend, dir: String, flushing: Rc<Cell<bool>>, failed_at: Rc<Cell<f64>>) {
'outer: loop {
let ops = files.take_dirty();
if ops.is_empty() {
if files.error().is_some() {
log::info!("lightcraft: saving works again");
}
files.set_error(None);
break;
}
for (i, op) in ops.iter().enumerate() {
let r = match op {
FlushOp::Write { name, data } => backend.write(&library_key(name), data).await,
FlushOp::Append { name, offset, data } => backend.write_at(&library_key(name), Some(*offset), data).await,
FlushOp::Write { name, data } => backend.write(&library_key(&dir, name), data).await,
FlushOp::Append { name, offset, data } => backend.write_at(&library_key(&dir, name), Some(*offset), data).await,
};
if let Err(e) = r {
log::error!("saving {} failed: {e}", op.name());
files.failed(&ops[i..]);
files.set_error(Some(e));
failed_at.set(perf_now());
break 'outer;
}
}
@@ -204,6 +262,8 @@ struct Boot {
backend: Option<Backend>,
files: Files,
index: ThumbIndex,
/// The library folder in storage (`library`, or a restored backup's).
lib_dir: String,
}
async fn boot(opts: Options) -> Boot {
@@ -221,15 +281,32 @@ async fn boot(opts: Options) -> Boot {
};
let files = Files::default();
let mut index = ThumbIndex::default();
let mut lib_dir = DEFAULT_LIBRARY_DIR.to_string();
if let Some(b) = &backend {
if opts.reset {
match b.clear().await {
Ok(()) => log::info!("lightcraft: storage cleared (?reset)"),
Err(e) => log::error!("clearing storage: {e}"),
// everything this site keeps in the browser, photos included: never without asking
let ok = safety::confirm(
"?reset deletes the LightCraft library stored in this browser, including every imported photo. \
This can't be undone. Delete it?",
);
if ok {
match b.clear().await {
Ok(()) => log::info!("lightcraft: storage cleared (?reset)"),
Err(e) => log::error!("clearing storage: {e}"),
}
} else {
log::info!("lightcraft: ?reset declined; storage kept");
}
}
// a restored backup lives in its own folder (see `backup`)
if let Ok(Some(d)) = b.read(ACTIVE_LIBRARY).await {
let d = String::from_utf8_lossy(&d).trim().to_string();
if valid_library_dir(&d) {
lib_dir = d;
}
}
for name in LIBRARY_FILES {
match b.read(&library_key(name)).await {
match b.read(&library_key(&lib_dir, name)).await {
Ok(Some(data)) => files.preload(name, data),
Ok(None) => {}
Err(e) => log::error!("reading {name}: {e}"),
@@ -238,7 +315,14 @@ async fn boot(opts: Options) -> Boot {
if let Ok(Some(j)) = b.read(THUMB_INDEX).await {
index = ThumbIndex::from_json(&j);
}
crate::backend::request_persistence();
crate::backend::request_persistence(|granted| {
if !granted {
notice(
"This browser may delete LightCraft's library when it runs short of space (persistent storage wasn't granted). \
Keep your original photos elsewhere and use File ▸ Back Up Library… now and then.",
);
}
});
// thumbnails stored without an index entry (index saved late, or lost): delete them
let b2 = b.clone();
let known: std::collections::HashSet<String> = b
@@ -266,7 +350,7 @@ async fn boot(opts: Options) -> Boot {
index.len(),
index.total() as f64 / 1e6
);
Boot { opts, backend, files, index }
Boot { opts, backend, files, index, lib_dir }
}
struct WebApp {
@@ -276,6 +360,11 @@ struct WebApp {
backend: Option<Backend>,
workers: Option<Workers>,
flushing: Rc<Cell<bool>>,
/// When the last save to storage failed (ms; 0 = never).
flush_failed_at: Rc<Cell<f64>>,
/// Saving stopped (a restored backup is about to replace this library: the page reloads).
frozen: Rc<Cell<bool>>,
lib_dir: String,
bench: Option<Bench>,
first_frame_logged: bool,
last_save: f64,
@@ -284,17 +373,18 @@ struct WebApp {
impl WebApp {
fn new(cc: &eframe::CreationContext<'_>, boot: Boot) -> Self {
let Boot { opts, backend, files, index } = boot;
let Boot { opts, backend, files, index, lib_dir } = boot;
let originals = Originals::default();
let t = perf_now();
let mut session = Session::new();
originals.install(&mut session);
let stores = LibraryStores {
dir: format!("browser:{}/{LIBRARY_DIR}", backend.as_ref().map_or("memory", |b| b.kind())).into(),
dir: format!("browser:{}/{lib_dir}", backend.as_ref().map_or("memory", |b| b.kind())).into(),
catalog: Box::new(files.store()),
files: Box::new(files.store()),
on_disk: false,
};
let mut problem = None;
match session.open_library_in(stores, true).cloned() {
Ok(r) => log::info!(
"lightcraft: library {} in {:.0} ms ({} photos; snapshot seq {}, {} ops replayed)",
@@ -305,19 +395,35 @@ impl WebApp {
r.replayed
),
Err(e) => {
log::error!("opening the library failed: {e}; starting a temporary one");
session = Session::with_demo();
// never a silent demo session (issue #100): an empty one, and the window says why
log::error!("opening the library failed: {e}");
session = Session::new();
originals.install(&mut session);
problem = Some(lightcraft_ui_egui::panels::library_problem::LibraryProblem {
can_retry: false,
..lightcraft_ui_egui::panels::library_problem::LibraryProblem::new(
"the browser's storage for this page",
format!("{e}. Reload the page to try again."),
)
});
}
}
if backend.is_some() {
notice(
"LightCraft in the browser is experimental. Your library is kept in this browser's storage: keep your original photos \
elsewhere and back up with File ▸ Back Up Library….",
);
}
// previews are large (≤ 2560 px, f32): keep few in a 32-bit address space
session.media.preview_capacity = 3;
let mut app = LightcraftApp::new(session, services(originals.clone(), backend.clone(), cc.egui_ctx.clone()));
let frozen = Rc::new(Cell::new(false));
let mut app = LightcraftApp::new(session, services(originals.clone(), backend.clone(), files.clone(), frozen.clone(), cc.egui_ctx.clone()));
let ui_written = files.get("ui.json").unwrap_or_default();
if let Ok(ui) = serde_json::from_slice::<UiState>(&ui_written) {
app.ui = ui;
}
app.ui = app.ui.sanitized();
app.library_problem = problem;
let n = opts.workers.unwrap_or_else(|| {
let cores = window().map_or(1, |w| w.navigator().hardware_concurrency() as usize);
cores.saturating_sub(1).clamp(1, 4)
@@ -337,6 +443,9 @@ impl WebApp {
backend,
workers,
flushing: Rc::new(Cell::new(false)),
flush_failed_at: Rc::new(Cell::new(0.0)),
frozen,
lib_dir,
bench: opts.bench.then(|| Bench::new(origin)),
first_frame_logged: false,
last_save: 0.0,
@@ -398,6 +507,9 @@ impl WebApp {
/// Save view state, UI prefs and the thumbnail index now and then; flush dirty files.
fn save(&mut self) {
if self.frozen.get() {
return;
}
let now = perf_now();
if now - self.last_save >= SAVE_EVERY_MS {
self.last_save = now;
@@ -419,12 +531,16 @@ impl WebApp {
});
}
}
// after a failure, retry every couple of seconds rather than every frame
let backing_off = self.files.error().is_some() && now - self.flush_failed_at.get() < SAVE_RETRY_MS;
if let Some(b) = &self.backend
&& self.files.is_dirty()
&& !self.flushing.get()
&& !backing_off
{
self.flushing.set(true);
wasm_bindgen_futures::spawn_local(flush(self.files.clone(), b.clone(), self.flushing.clone()));
let (files, dir, flushing, failed_at) = (self.files.clone(), self.lib_dir.clone(), self.flushing.clone(), self.flush_failed_at.clone());
wasm_bindgen_futures::spawn_local(flush(files, b.clone(), dir, flushing, failed_at));
}
}
}
@@ -454,8 +570,10 @@ impl WebApp {
let (orig_n, orig_bytes) = self.originals.usage();
json!({
"storage": self.backend.as_ref().map_or("memory", |b| b.kind()),
"library": self.lib_dir,
"dirty": self.files.is_dirty(),
"flushing": self.flushing.get(),
"saveError": self.files.error(),
"workers": {"alive": alive, "ready": ready, "remoteJobs": remote, "inlineJobs": inline},
"originalsInMemory": orig_n,
"originalBytesInMemory": orig_bytes,
@@ -480,6 +598,10 @@ impl WebApp {
impl eframe::App for WebApp {
fn logic(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
for text in safety::take_notices() {
let now = ctx.input(|i| i.time);
self.app.ui.toast = Some((text, now + NOTICE_SECS));
}
self.run_inbox();
self.import_dropped(ctx);
self.load_originals(ctx);
@@ -517,9 +639,18 @@ impl eframe::App for WebApp {
#[wasm_bindgen]
pub fn start() {
eframe::WebLogger::init(log::LevelFilter::Info).ok();
safety::install_panic_hook();
log::info!("lightcraft: wasm instantiated at {:.0} ms", perf_now());
let opts = Options::from_url();
wasm_bindgen_futures::spawn_local(async move {
// one tab per library: two would each keep their own copy and overwrite each other's saves
if opts.store != "memory" && safety::acquire_tab_lock().await == Some(false) {
safety::show_blocking(
"LightCraft is already open in another tab or window of this browser.\n\n\
Switch to that tab, or close it and reload this page. (Two tabs would overwrite each other's changes.)",
);
return;
}
let Some(canvas) = window()
.and_then(|w| w.document())
.and_then(|d| d.get_element_by_id("lightcraft_canvas"))
+24 -10
View File
@@ -15,10 +15,9 @@ use std::sync::Arc;
use lightcraft_engine::catalog::Source;
use lightcraft_engine::develop::{DevelopSettings, EmbeddedLens};
use lightcraft_engine::media::{MediaCache, RenderJob, SourceLevel, SourceRef};
use lightcraft_engine::media::{DecodedSource, MediaCache, RenderJob, SourceLevel, SourceRef};
use lightcraft_engine::pipeline::{Quality, RenderRequest, Rendered, SourceInfo, StageCache};
use lightcraft_preview::Lru;
use lightcraft_raster::Rgb32f;
use serde::{Deserialize, Serialize};
use crate::store::hash_of_path;
@@ -31,6 +30,8 @@ pub struct WireJob {
/// Decode the source at most this long.
pub max_edge: usize,
pub raw: bool,
#[serde(default)]
pub relative_wb: bool,
pub as_shot_temp: f64,
pub as_shot_tint: f64,
pub lens: Option<EmbeddedLens>,
@@ -55,7 +56,7 @@ impl WireJob {
pub fn from_job(job: &RenderJob, thumb_cached: bool, view: &str) -> WireJob {
let max_edge = match &job.source {
SourceRef::Demo { max_edge, .. } | SourceRef::File { max_edge, .. } => *max_edge,
SourceRef::Loaded(img) => img.width.max(img.height),
SourceRef::Loaded(img) => img.image.width.max(img.image.height),
SourceRef::Smart { .. } => 2560,
};
WireJob {
@@ -63,6 +64,7 @@ impl WireJob {
origin: job.origin.clone(),
max_edge,
raw: job.info.raw,
relative_wb: job.info.relative_wb,
as_shot_temp: job.info.as_shot_temp,
as_shot_tint: job.info.as_shot_tint,
lens: job.info.lens,
@@ -79,7 +81,14 @@ impl WireJob {
}
pub fn info(&self) -> SourceInfo {
SourceInfo { raw: self.raw, as_shot_temp: self.as_shot_temp, as_shot_tint: self.as_shot_tint, lens: self.lens }
SourceInfo {
raw: self.raw,
as_shot_temp: self.as_shot_temp,
as_shot_tint: self.as_shot_tint,
lens: self.lens,
relative_wb: self.relative_wb,
..Default::default()
}
}
pub fn request(&self) -> RenderRequest {
@@ -119,7 +128,7 @@ const WORKER_SOURCE_BYTES: usize = 192 << 20;
/// The synchronous part of a worker: decoded-source cache + rendering.
pub struct WorkerCore {
sources: Lru<String, Arc<Rgb32f>>,
sources: Lru<String, DecodedSource>,
media: MediaCache,
stages: HashMap<String, Arc<StageCache>>,
}
@@ -150,21 +159,25 @@ impl WorkerCore {
Some(s) => s,
None => {
let src = match (&job.origin, original) {
(Source::File { .. }, Some(bytes)) => Arc::new(lightcraft_engine::files::load_bytes(bytes, job.max_edge)?.0),
(Source::File { .. }, Some(bytes)) => {
let (image, info) = lightcraft_engine::files::load_bytes(bytes, job.max_edge)?;
DecodedSource::new(Arc::new(image), Some(info))
}
(Source::File { path }, None) => return Err(format!("{path}: original not found in browser storage")),
(origin @ Source::Demo { .. }, _) => self.media.origin_ref(origin, job.max_edge).load()?,
(origin @ Source::Demo { .. }, _) => self.media.origin_ref(origin, job.max_edge).load_source()?,
};
let cost = src.width * src.height * 12 + 64;
let cost = src.image.width * src.image.height * 12 + 64;
self.sources.insert(key, src.clone(), cost);
src
}
};
let info = src.info_or(job.info());
Ok(match &job.stages {
Some(view) => {
let st = self.stages.entry(view.clone()).or_default().clone();
lightcraft_engine::pipeline::render_cached(&src, &job.info(), &job.settings, &job.request(), &st)
lightcraft_engine::pipeline::render_cached(&src.image, &info, &job.settings, &job.request(), &st)
}
None => lightcraft_engine::pipeline::render(&src, &job.info(), &job.settings, &job.request()),
None => lightcraft_engine::pipeline::render(&src.image, &info, &job.settings, &job.request()),
})
}
}
@@ -295,6 +308,7 @@ mod tests {
origin,
max_edge: 512,
raw: false,
relative_wb: false,
as_shot_temp: 6500.0,
as_shot_tint: 0.0,
lens: None,
+237 -32
View File
@@ -1,14 +1,28 @@
//! Loopback JSON-lines control server: one request per line, one reply per line.
//! This is the transport the MCP server (`lightcraft-cli mcp --connect`) wraps.
//!
//! The server only ever reads requests: a line that is not a JSON object with a string `method`
//! (an HTTP request line from a browser's cross-origin `fetch`, a stray `nc`, binary junk) gets
//! one error reply and the connection is closed, so nothing after it is executed. Lines are
//! bounded ([`MAX_LINE`]) and so is the number of open connections ([`MAX_CONNECTIONS`]); junk
//! never reaches the UI thread. See `docs/control-protocol.md`.
use std::io::{BufRead, BufReader, Write};
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{TcpListener, TcpStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::mpsc::{Receiver, Sender, channel};
use std::time::Duration;
use lightcraft_ui_egui::ControlRequest;
use serde_json::{Value, json};
/// Longest accepted request line (bytes, without the newline). Requests are small JSON objects;
/// anything longer is refused and the connection closed.
pub const MAX_LINE: usize = 4 * 1024 * 1024;
/// Connections served at once; further ones get an error line and are closed.
pub const MAX_CONNECTIONS: usize = 16;
pub fn start(port: u16, ctx: egui::Context) -> Receiver<ControlRequest> {
let (tx, rx) = channel::<ControlRequest>();
let listener = match TcpListener::bind(("127.0.0.1", port)) {
@@ -19,44 +33,235 @@ pub fn start(port: u16, ctx: egui::Context) -> Receiver<ControlRequest> {
}
};
eprintln!("lightcraft: control server listening on 127.0.0.1:{port}");
std::thread::spawn(move || {
for stream in listener.incoming().flatten() {
let tx = tx.clone();
let ctx = ctx.clone();
std::thread::spawn(move || serve(stream, tx, ctx));
}
});
std::thread::spawn(move || accept_loop(listener, tx, ctx));
rx
}
fn serve(stream: TcpStream, tx: Sender<ControlRequest>, ctx: egui::Context) {
let Ok(read) = stream.try_clone() else { return };
let mut out = stream;
for line in BufReader::new(read).lines() {
let Ok(line) = line else { break };
if line.trim().is_empty() {
fn accept_loop(listener: TcpListener, tx: Sender<ControlRequest>, ctx: egui::Context) {
let open = Arc::new(AtomicUsize::new(0));
for mut stream in listener.incoming().flatten() {
if open.fetch_add(1, Ordering::SeqCst) >= MAX_CONNECTIONS {
open.fetch_sub(1, Ordering::SeqCst);
let _ = writeln!(stream, "{}", json!({"ok": false, "error": "too many control connections"}));
continue;
}
let reply = match serde_json::from_str::<Value>(&line) {
Ok(msg) => {
let id = msg.get("id").cloned().unwrap_or(Value::Null);
let method = msg.get("method").and_then(Value::as_str).unwrap_or("").to_string();
let params = msg.get("params").cloned().unwrap_or(json!({}));
let (req, rrx) = ControlRequest::new(method, params);
if tx.send(req).is_err() {
break;
}
ctx.request_repaint();
let mut r = rrx.recv_timeout(Duration::from_secs(60)).unwrap_or_else(|_| json!({"ok": false, "error": "timeout"}));
if let Some(o) = r.as_object_mut() {
o.insert("id".into(), id);
let tx = tx.clone();
let ctx = ctx.clone();
let open = Arc::clone(&open);
std::thread::spawn(move || {
serve(stream, &tx, &ctx);
open.fetch_sub(1, Ordering::SeqCst);
});
}
}
fn serve(stream: TcpStream, tx: &Sender<ControlRequest>, ctx: &egui::Context) {
let Ok(read) = stream.try_clone() else { return };
serve_lines(read, stream, |method, params| {
let (req, rrx) = ControlRequest::new(method, params);
tx.send(req).ok()?;
ctx.request_repaint();
Some(rrx.recv_timeout(Duration::from_secs(60)).unwrap_or_else(|_| json!({"ok": false, "error": "timeout"})))
});
}
/// One parsed line.
enum Line {
Request {
id: Value,
method: String,
params: Value,
},
Blank,
/// Not a request: reply with this error and close the connection.
Reject(String),
}
fn parse_line(bytes: &[u8]) -> Line {
let Ok(text) = std::str::from_utf8(bytes) else {
return Line::Reject("not a control request (invalid UTF-8); closing the connection".into());
};
let text = text.trim();
if text.is_empty() {
return Line::Blank;
}
let msg = match serde_json::from_str::<Value>(text) {
Ok(v) => v,
Err(e) => return Line::Reject(format!("bad JSON ({e}); expected one JSON request object per line; closing the connection")),
};
let Some(obj) = msg.as_object() else {
return Line::Reject("not a control request (expected a JSON object); closing the connection".into());
};
let Some(method) = obj.get("method").and_then(Value::as_str) else {
return Line::Reject("not a control request (missing string `method`); closing the connection".into());
};
Line::Request {
id: obj.get("id").cloned().unwrap_or(Value::Null),
method: method.to_string(),
params: obj.get("params").cloned().unwrap_or(json!({})),
}
}
/// Read one `\n`-terminated line of at most `max` bytes into `buf` (newline stripped).
/// `Ok(false)` at end of stream; `Err` when the line is too long or the read fails.
fn read_bounded_line(reader: &mut impl BufRead, buf: &mut Vec<u8>, max: usize) -> Result<bool, String> {
buf.clear();
// `take` bounds the bytes buffered for one line: a peer can't make us allocate more than `max`.
let limit = u64::try_from(max).unwrap_or(u64::MAX).saturating_add(1);
let n = reader.take(limit).read_until(b'\n', buf).map_err(|e| e.to_string())?;
if n == 0 {
return Ok(false);
}
if buf.last() == Some(&b'\n') {
buf.pop();
if buf.last() == Some(&b'\r') {
buf.pop();
}
return Ok(true);
}
if buf.len() > max {
return Err(format!("request line longer than {max} bytes; closing the connection"));
}
// Last line without a newline, then end of stream.
Ok(true)
}
/// Serve requests from `read`, answering on `out`, until the peer closes, sends something that is
/// not a request, or `handle` returns `None` (app gone). `handle` gets `(method, params)` and
/// returns the reply object (the request's `id` is added here).
fn serve_lines<R: Read, W: Write>(read: R, mut out: W, mut handle: impl FnMut(String, Value) -> Option<Value>) {
let mut reader = BufReader::new(read);
let mut buf = Vec::new();
loop {
let reject = match read_bounded_line(&mut reader, &mut buf, MAX_LINE) {
Ok(false) => return,
Err(e) => e,
Ok(true) => match parse_line(&buf) {
Line::Blank => continue,
Line::Reject(e) => e,
Line::Request { id, method, params } => {
let Some(mut r) = handle(method, params) else { return };
if let Some(o) = r.as_object_mut() {
o.insert("id".into(), id);
}
if writeln!(out, "{r}").and_then(|()| out.flush()).is_err() {
return;
}
continue;
}
r
}
Err(e) => json!({"ok": false, "error": format!("bad JSON: {e}")}),
},
};
if writeln!(out, "{reply}").is_err() {
break;
let _ = writeln!(out, "{}", json!({"ok": false, "error": reject})).and_then(|()| out.flush());
return;
}
}
#[cfg(test)]
mod tests {
use super::*;
fn run(input: &[u8]) -> (Vec<Value>, Vec<String>) {
let mut out = Vec::new();
let mut calls = Vec::new();
serve_lines(input, &mut out, |m, _p| {
calls.push(m);
Some(json!({"ok": true, "result": null}))
});
let replies = String::from_utf8(out).unwrap().lines().map(|l| serde_json::from_str::<Value>(l).unwrap()).collect();
(replies, calls)
}
#[test]
fn requests_are_answered_in_order_with_ids() {
let (r, calls) = run(b"{\"id\":1,\"method\":\"a\"}\n\n{\"id\":2,\"method\":\"b\",\"params\":{}}\r\n{\"method\":\"c\"}");
assert_eq!(calls, ["a", "b", "c"]);
assert_eq!(r.len(), 3);
assert_eq!(r[0]["id"], 1);
assert_eq!(r[1]["id"], 2);
assert_eq!(r[2]["id"], Value::Null);
}
#[test]
fn http_request_closes_before_the_body_runs() {
// A browser's cross-origin `fetch` POST: the request line is rejected and the connection
// closed, so the JSON body is never executed.
let http = b"POST / HTTP/1.1\r\nHost: 127.0.0.1:7980\r\nContent-Type: text/plain\r\n\r\n\
{\"id\":1,\"method\":\"engine.execute\",\"params\":{\"command\":\"folder.move\"}}\n";
let (r, calls) = run(http);
assert!(calls.is_empty(), "nothing may run: {calls:?}");
assert_eq!(r.len(), 1);
assert_eq!(r[0]["ok"], false);
assert!(r[0]["error"].as_str().unwrap().contains("closing the connection"));
}
#[test]
fn non_request_json_closes_the_connection() {
for junk in [
&b"[1,2]\n{\"method\":\"a\"}\n"[..],
b"{\"id\":1}\n{\"method\":\"a\"}\n",
b"{\"method\":5}\n{\"method\":\"a\"}\n",
b"\xff\xfe\n{\"method\":\"a\"}\n",
] {
let (r, calls) = run(junk);
assert!(calls.is_empty());
assert_eq!(r.len(), 1);
assert_eq!(r[0]["ok"], false);
}
}
#[test]
fn a_request_before_junk_still_runs() {
let (r, calls) = run(b"{\"method\":\"a\"}\nGET / HTTP/1.1\n{\"method\":\"b\"}\n");
assert_eq!(calls, ["a"]);
assert_eq!(r.len(), 2);
assert_eq!(r[1]["ok"], false);
}
#[test]
fn overlong_line_is_refused_without_buffering_it_all() {
// An endless line without a newline: reading stops after MAX_LINE + 1 bytes.
let endless = std::io::repeat(b'x');
let mut out = Vec::new();
let mut ran = false;
serve_lines(endless, &mut out, |_, _| {
ran = true;
None
});
assert!(!ran);
let reply: Value = serde_json::from_slice(out.strip_suffix(b"\n").unwrap()).unwrap();
assert!(reply["error"].as_str().unwrap().contains("longer than"));
}
#[test]
fn line_at_the_limit_is_accepted() {
let mut buf = Vec::new();
let line = vec![b'a'; 10];
let mut input = line.clone();
input.push(b'\n');
assert_eq!(read_bounded_line(&mut &input[..], &mut buf, 10), Ok(true));
assert_eq!(buf, line);
assert!(read_bounded_line(&mut &[b'a'; 11][..], &mut buf, 10).is_err());
}
#[test]
fn tcp_connection_is_closed_after_junk() {
// End to end over a real loopback socket (the transport the app uses).
let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap();
let addr = listener.local_addr().unwrap();
let server = std::thread::spawn(move || {
let (s, _) = listener.accept().unwrap();
let read = s.try_clone().unwrap();
serve_lines(read, s, |_, _| Some(json!({"ok": true})));
});
let mut c = TcpStream::connect(addr).unwrap();
c.set_read_timeout(Some(Duration::from_secs(10))).unwrap();
c.write_all(b"{\"id\":7,\"method\":\"x\"}\nhello\n{\"method\":\"y\"}\n").unwrap();
let mut text = String::new();
c.read_to_string(&mut text).unwrap(); // returns at EOF: the server closed the connection
server.join().unwrap();
let lines: Vec<&str> = text.lines().collect();
assert_eq!(lines.len(), 2, "{text}");
assert!(lines[0].contains("\"id\":7"));
assert!(lines[1].contains("closing the connection"));
}
}
+299 -40
View File
@@ -7,7 +7,9 @@
//! `~/Pictures/LightCraft Library`; a new library starts with the
//! procedural demo photos unless `--no-demo` or files are given. Files and folders on the command
//! line are imported (duplicates are skipped). `--memory` runs an in-memory session that writes
//! nothing (demo photos unless files are given; used by the README showcase scripts).
//! nothing (demo photos unless files are given; used by the README showcase scripts). A library
//! that can't be opened is never replaced silently: the window says why and asks what to do
//! (`lightcraft_ui_egui::panels::library_problem`).
//!
//! `--control <port>` (or `LIGHTCRAFT_CONTROL_PORT`) starts a localhost JSON-lines control server:
//! `{"id":1,"method":"ui.inspect","params":{}}` → `{"id":1,"ok":true,"result":…}`.
@@ -26,6 +28,7 @@ mod control_server;
mod native_menu;
use lightcraft_engine::Session;
use lightcraft_ui_egui::panels::library_problem::LibraryProblem;
use lightcraft_ui_egui::{LightcraftApp, Services, UiState};
/// Reverse-DNS app id: Wayland app id, `.desktop` file and hicolor icon name.
@@ -41,15 +44,16 @@ fn app_icon() -> egui::IconData {
eframe::icon_data::from_png_bytes(png).unwrap_or_default()
}
struct App(LightcraftApp, #[cfg(target_os = "macos")] Option<native_menu::NativeMenu>);
struct App(LightcraftApp, PrefsWriter, #[cfg(target_os = "macos")] Option<native_menu::NativeMenu>);
impl eframe::App for App {
fn logic(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
#[cfg(target_os = "macos")]
if let Some(m) = self.1.as_mut() {
if let Some(m) = self.2.as_mut() {
m.update(&mut self.0, ctx);
}
self.0.logic(ctx);
self.1.tick(&mut self.0, ctx);
}
fn raw_input_hook(&mut self, _ctx: &egui::Context, raw: &mut egui::RawInput) {
self.0.raw_input_hook(raw);
@@ -58,13 +62,52 @@ impl eframe::App for App {
self.0.ui(ui);
}
fn on_exit(&mut self) {
save_prefs(&self.0);
if let Err(e) = self.1.save(&self.0) {
eprintln!("lightcraft: {e}");
}
if let Err(e) = self.0.session.close_library() {
eprintln!("lightcraft: saving the library failed: {e}");
}
}
}
/// The window's renderer (egui-wgpu): eframe's defaults, with LightCraft's backend choice — DX12
/// alone on Windows unless `LIGHTCRAFT_GPU_BACKEND` / `WGPU_BACKEND` say otherwise (issue #136:
/// with Vulkan in the set, wgpu loads the Vulkan driver even when it then picks DX12).
fn window_wgpu_options() -> eframe::egui_wgpu::WgpuConfiguration {
let mut c = eframe::egui_wgpu::WgpuConfiguration::default();
if let eframe::egui_wgpu::WgpuSetup::CreateNew(n) = &mut c.wgpu_setup {
n.instance_descriptor.backends = lightcraft_engine::gpu::backend::window_backends();
}
c
}
/// Written while the GPU compute device is created, removed once that returned
/// (`lightcraft_gpu::backend::set_init_marker`). Not with `LIGHTCRAFT_NO_PREFS` (tests, scripts).
fn gpu_marker_path() -> Option<std::path::PathBuf> {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return None;
}
config_dir().map(|d| d.join("gpu-init.marker"))
}
/// Arm the GPU crash sentinel; if the previous launch left it behind (it died inside the GPU
/// driver while creating the device), the notice to show — GPU rendering then starts off.
fn gpu_crash_check() -> Option<String> {
let marker = gpu_marker_path();
let left = marker.as_deref().and_then(lightcraft_engine::gpu::backend::take_init_marker);
lightcraft_engine::gpu::backend::set_init_marker(marker);
left.map(|what| gpu_crash_notice(&what))
}
fn gpu_crash_notice(what: &str) -> String {
format!(
"LightCraft closed unexpectedly while starting the GPU last time ({what}), so GPU rendering is now off and photos render on the CPU. \
To try the GPU again, turn on Settings ▸ Performance ▸ Use the GPU for rendering; to try another graphics backend, start LightCraft with \
LIGHTCRAFT_GPU_BACKEND=dx12, vulkan or off (see docs/gpu-pipeline.md → Troubleshooting)."
)
}
fn config_dir() -> Option<std::path::PathBuf> {
if cfg!(target_os = "macos") {
std::env::var_os("HOME").map(|h| std::path::PathBuf::from(h).join("Library/Application Support/LightCraft"))
@@ -78,30 +121,127 @@ fn config_dir() -> Option<std::path::PathBuf> {
}
}
/// The saved UI state and app settings (`<config>/ui.json`), if any.
fn load_prefs() -> Option<UiState> {
/// The saved UI state and app settings (`<config>/ui.json`), if any, and a warning for the user
/// when the file exists but can't be used (issue #103): a damaged file is kept as
/// `ui.json.corrupt-<unix time>` first, so the next save can't lose the library location in it;
/// one that can't be read at all is not written this session (`keep_file`).
fn load_prefs() -> (Option<UiState>, Option<String>, bool) {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return None;
return (None, None, false);
}
let bytes = std::fs::read(config_dir()?.join("ui.json")).ok()?;
serde_json::from_slice::<UiState>(&bytes).ok().map(UiState::sanitized)
let Some(path) = config_dir().map(|d| d.join("ui.json")) else { return (None, None, false) };
load_prefs_at(&path)
}
fn save_prefs(app: &LightcraftApp) {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return;
fn load_prefs_at(path: &std::path::Path) -> (Option<UiState>, Option<String>, bool) {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return (None, None, false),
Err(e) => {
let msg = format!("The app settings ({}) couldn't be read: {e}. Defaults are used, and the file isn't overwritten.", path.display());
return (None, Some(msg), true);
}
};
match serde_json::from_slice::<UiState>(&bytes) {
Ok(ui) => (Some(ui.sanitized()), None, false),
Err(e) => {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let keep = path.with_file_name(format!("ui.json.corrupt-{secs}"));
let kept = std::fs::rename(path, &keep);
let what = match &kept {
Ok(()) => format!("it was kept as {}", keep.display()),
Err(r) => format!("it couldn't be set aside ({r})"),
};
let msg = format!(
"The app settings ({}) are damaged ({e}); {what}. Defaults are used — reopen your library with Settings → Open Library… if it isn't shown.",
path.display()
);
(None, Some(msg), kept.is_err())
}
}
if let Some(d) = config_dir() {
let _ = std::fs::create_dir_all(&d);
if let Ok(bytes) = serde_json::to_vec_pretty(&app.ui) {
let _ = std::fs::write(d.join("ui.json"), bytes);
}
/// Write `bytes` to `path` atomically: a temp file, fsynced, renamed over it (a crash or a full
/// disk leaves the old file or the new one, never a truncated one).
fn write_atomic(path: &std::path::Path, bytes: &[u8]) -> std::io::Result<()> {
use std::io::Write;
let tmp = path.with_extension("json.tmp");
let written = (|| {
let mut f = std::fs::File::create(&tmp)?;
f.write_all(bytes)?;
f.sync_all()
})();
if let Err(e) = written {
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
std::fs::rename(&tmp, path)?;
if let Some(dir) = path.parent()
&& let Ok(d) = std::fs::File::open(dir)
{
let _ = d.sync_all();
}
Ok(())
}
/// Saves `ui.json` (app settings, incl. the library to open at launch): as soon as the library
/// changes, every few seconds when anything else changed, and at exit.
#[derive(Default)]
struct PrefsWriter {
written: Vec<u8>,
library: String,
checked: f64,
/// The last write failed (reported once until a write works).
failing: bool,
/// `ui.json` couldn't be read at launch: never overwrite it this session.
keep_file: bool,
}
impl PrefsWriter {
fn save(&mut self, app: &LightcraftApp) -> Result<(), String> {
if self.keep_file || std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return Ok(());
}
let Some(d) = config_dir() else { return Ok(()) };
let bytes = serde_json::to_vec_pretty(&app.ui).map_err(|e| e.to_string())?;
if bytes == self.written {
return Ok(());
}
std::fs::create_dir_all(&d)
.and_then(|()| write_atomic(&d.join("ui.json"), &bytes))
.map_err(|e| format!("saving the app settings failed: {e}"))?;
self.written = bytes;
self.library = app.ui.settings.library_path.clone();
Ok(())
}
fn tick(&mut self, app: &mut LightcraftApp, ctx: &egui::Context) {
let now = ctx.input(|i| i.time);
let moved = app.ui.settings.library_path != self.library;
if !moved && now - self.checked < 3.0 {
return;
}
self.checked = now;
match self.save(app) {
Ok(()) => self.failing = false,
Err(e) => {
eprintln!("lightcraft: {e}");
if !self.failing {
app.notices.push(format!("{e}. LightCraft keeps trying."));
}
self.failing = true;
// don't retry every frame
self.library = app.ui.settings.library_path.clone();
}
}
}
}
fn services() -> Services {
Services {
pick_folder: Some(Box::new(|| rfd::FileDialog::new().set_title("Open Library").pick_folder().map(|p| p.to_string_lossy().to_string()))),
pick_folder: Some(Box::new(|| {
rfd::FileDialog::new().set_title(lightcraft_ui_egui::i18n::tr("Open Library")).pick_folder().map(|p| p.to_string_lossy().to_string())
})),
open_with: Some(Box::new(|path: &str, app: &str| {
// spawned, never waited for: the editor runs alongside
let app = app.trim();
@@ -153,7 +293,7 @@ fn services() -> Services {
pick_files: Some(Box::new(|| {
rfd::FileDialog::new()
.add_filter(
"Photos",
lightcraft_ui_egui::i18n::tr("Photos"),
&[
"jpg", "jpeg", "png", "tif", "tiff", "webp", "dng", "cr2", "cr3", "nef", "nrw", "arw", "raf", "orf", "rw2", "pef", "psd",
"jxl", "gif", "bmp",
@@ -167,8 +307,11 @@ fn services() -> Services {
})),
pick_preset_files: Some(Box::new(|| {
rfd::FileDialog::new()
.set_title("Import Presets")
.add_filter("Presets & Profiles", &["lcpreset", "xmp", "lrtemplate", "zip", "dng", "lmp", "mplumpack", "cube"])
.set_title(lightcraft_ui_egui::i18n::tr("Import Presets"))
.add_filter(
lightcraft_ui_egui::i18n::tr("Presets & Profiles"),
&["lcpreset", "xmp", "lrtemplate", "zip", "dng", "lmp", "mplumpack", "cube"],
)
.pick_files()
.unwrap_or_default()
.into_iter()
@@ -177,24 +320,24 @@ fn services() -> Services {
})),
pick_tracklog: Some(Box::new(|| {
rfd::FileDialog::new()
.set_title("Auto-Tag from Tracklog")
.add_filter("GPS Track Log", &["gpx"])
.set_title(lightcraft_ui_egui::i18n::tr("Auto-Tag from Tracklog"))
.add_filter(lightcraft_ui_egui::i18n::tr("GPS Track Log"), &["gpx"])
.pick_file()
.map(|p| vec![p.to_string_lossy().to_string()])
.unwrap_or_default()
})),
save_preset_file: Some(Box::new(|name: &str| {
rfd::FileDialog::new()
.set_title("Export Presets")
.add_filter("LightCraft Preset", &["lcpreset"])
.set_title(lightcraft_ui_egui::i18n::tr("Export Presets"))
.add_filter(lightcraft_ui_egui::i18n::tr("LightCraft Preset"), &["lcpreset"])
.set_file_name(name)
.save_file()
.map(|p| p.to_string_lossy().to_string())
})),
pick_curve_preset_files: Some(Box::new(|| {
rfd::FileDialog::new()
.set_title("Import Point Curve Presets")
.add_filter("Point Curve Presets", &["lccurve", "json"])
.set_title(lightcraft_ui_egui::i18n::tr("Import Point Curve Presets"))
.add_filter(lightcraft_ui_egui::i18n::tr("Point Curve Presets"), &["lccurve", "json"])
.pick_files()
.unwrap_or_default()
.into_iter()
@@ -203,8 +346,8 @@ fn services() -> Services {
})),
save_curve_preset_file: Some(Box::new(|name: &str| {
rfd::FileDialog::new()
.set_title("Export Point Curve Presets")
.add_filter("Point Curve Presets", &["lccurve"])
.set_title(lightcraft_ui_egui::i18n::tr("Export Point Curve Presets"))
.add_filter(lightcraft_ui_egui::i18n::tr("Point Curve Presets"), &["lccurve"])
.set_file_name(name)
.save_file()
.map(|p| p.to_string_lossy().to_string())
@@ -215,18 +358,25 @@ fn services() -> Services {
png: Some(Box::new(|img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
})),
// the library is a folder on disk: backed up with the user's other files
backup_library: None,
restore_library: None,
}
}
/// The persistent library session (or an in-memory one with `--memory` / if the library can't open).
fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: bool) -> Session {
let fallback = || if seed_demo { Session::with_demo() } else { Session::new() }.with_fs().with_system_clock();
/// The persistent library session, or with `--memory` an in-memory one (demo photos).
///
/// If the library can't be opened the session is empty and in memory — never seeded with the
/// demo photos, never written anywhere — and the problem is returned: the window then says so
/// and offers Try Again / Choose Another Library… / Continue Without Saving / Quit (issue #100).
fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: bool) -> (Session, Option<LibraryProblem>) {
if in_memory {
return fallback();
return (if seed_demo { Session::with_demo() } else { Session::new() }.with_fs().with_system_clock(), None);
}
let unopened = || Session::new().with_fs().with_system_clock();
let Some(dir) = dir else {
eprintln!("lightcraft: no library location (set --library or LIGHTCRAFT_LIBRARY); running in memory");
return fallback();
eprintln!("lightcraft: no library location (set --library or LIGHTCRAFT_LIBRARY)");
return (unopened(), Some(LibraryProblem::new("", "There is no home folder to keep the library in. Choose a folder for it.")));
};
let t0 = std::time::Instant::now();
let mut s = Session::new().with_fs().with_system_clock();
@@ -240,11 +390,11 @@ fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: boo
if torn > 0 { ", torn tail repaired" } else { "" },
t0.elapsed().as_secs_f64() * 1000.0
);
s
(s, None)
}
Err(e) => {
eprintln!("lightcraft: can't open library {}: {e}; running in memory", dir.display());
fallback()
eprintln!("lightcraft: can't open library {}: {e}", dir.display());
(unopened(), Some(LibraryProblem::new(dir.to_string_lossy(), e.to_string())))
}
}
}
@@ -261,6 +411,10 @@ OPTIONS:
--control PORT serve the JSON-lines control channel on 127.0.0.1:PORT (env LIGHTCRAFT_CONTROL_PORT;
see docs/control-protocol.md)
--version, --help
ENVIRONMENT:
LIGHTCRAFT_GPU_BACKEND=dx12|vulkan|metal|auto|off graphics backend (default: DX12 on Windows, Metal on macOS,
Vulkan on Linux; off = render on the CPU); else WGPU_BACKEND. LIGHTCRAFT_GPU=0: CPU rendering.
";
fn main() -> eframe::Result {
@@ -293,7 +447,7 @@ fn main() -> eframe::Result {
_ => files.push(a),
}
}
let prefs = load_prefs();
let (prefs, prefs_warning, keep_prefs_file) = load_prefs();
// --library, else the library last opened from Settings, else the default location
let library_dir = library_dir.or_else(|| {
prefs
@@ -303,6 +457,11 @@ fn main() -> eframe::Result {
.map(Into::into)
});
let library_dir = library_dir.or_else(lightcraft_engine::library::default_dir);
// GPU compute: off if the preference says so, or if the last launch died creating the device
// (issue #136) — before anything can create it
let gpu_crash = gpu_crash_check();
let gpu_on = prefs.as_ref().is_none_or(|u| u.settings.gpu) && gpu_crash.is_none();
lightcraft_engine::gpu::set_enabled(gpu_on);
let options = eframe::NativeOptions {
viewport: egui::ViewportBuilder::default()
.with_title("LightCraft")
@@ -315,23 +474,42 @@ fn main() -> eframe::Result {
.with_icon(app_icon())
// Wayland matches the window to packaging/linux/ai.storyteller.lightcraft.desktop by this id
.with_app_id(APP_ID),
wgpu_options: window_wgpu_options(),
..Default::default()
};
eframe::run_native(
"LightCraft",
options,
Box::new(move |cc| {
let session = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let (session, problem) = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let mut app = LightcraftApp::new(session, services());
if let Some(ui) = prefs {
app.ui = ui;
}
lightcraft_ui_egui::i18n::set_language(app.ui.language);
app.integrated_titlebar = cfg!(target_os = "macos");
app.notices.extend(prefs_warning);
// what's on disk now: only changes are written
let writer = PrefsWriter {
written: serde_json::to_vec_pretty(&app.ui).unwrap_or_default(),
library: app.ui.settings.library_path.clone(),
keep_file: keep_prefs_file,
..Default::default()
};
// after the snapshot above, so the writer saves the switched-off preference
if let Some(notice) = gpu_crash {
app.ui.settings.gpu = false;
app.notices.push(notice);
}
if let Some(port) = control_port {
let rx = control_server::start(port, cc.egui_ctx.clone());
app = app.with_control(rx);
}
if !files.is_empty() {
if let Some(mut p) = problem {
// imported once the user has chosen where (into the library, or the temporary session)
p.pending_import = files;
app.library_problem = Some(p);
} else if !files.is_empty() {
let _ = app.run("library.import", serde_json::json!({"paths": files}));
app.ui.view = lightcraft_ui_egui::state::ViewMode::PhotoGrid;
}
@@ -341,9 +519,90 @@ fn main() -> eframe::Result {
let menu = (std::env::var_os("LIGHTCRAFT_NO_NATIVE_MENU").is_none()).then(|| native_menu::NativeMenu::install(&mut app, &cc.egui_ctx));
Ok(Box::new(App(
app,
writer,
#[cfg(target_os = "macos")]
menu,
)))
}),
)
}
#[cfg(test)]
#[allow(clippy::unwrap_used)]
mod tests {
use super::*;
fn dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-app-prefs-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
/// Issue #103: a damaged ui.json (which holds the library location) is kept aside and
/// reported, not silently replaced by the defaults.
#[test]
fn damaged_ui_json_is_kept_and_reported() {
let d = dir("damaged");
let path = d.join("ui.json");
std::fs::write(&path, br#"{"settings": {"library_path": "/Volumes/Photos/Lib"#).unwrap();
let (ui, warning, keep) = load_prefs_at(&path);
assert!(ui.is_none() && !keep);
let warning = warning.unwrap();
assert!(warning.contains("damaged") && warning.contains("ui.json.corrupt-"), "{warning}");
assert!(!path.exists());
let kept: Vec<_> = std::fs::read_dir(&d).unwrap().map(|e| e.unwrap().file_name().to_string_lossy().to_string()).collect();
assert!(kept.len() == 1 && kept[0].starts_with("ui.json.corrupt-"), "{kept:?}");
// a missing file is just "no settings yet"
assert_eq!(load_prefs_at(&path).1, None);
let _ = std::fs::remove_dir_all(&d);
}
/// Issue #136: a GPU init marker left by a crashed launch is reported once and removed.
#[test]
fn gpu_crash_marker_is_reported_once() {
let d = dir("gpu-marker");
let m = d.join("gpu-init.marker");
assert_eq!(lightcraft_engine::gpu::backend::take_init_marker(&m), None);
std::fs::write(&m, "GPU device creation started (backends VULKAN | DX12)").unwrap();
let what = lightcraft_engine::gpu::backend::take_init_marker(&m).unwrap();
let notice = gpu_crash_notice(&what);
assert!(notice.contains("GPU rendering is now off") && notice.contains("Use the GPU for rendering"), "{notice}");
assert!(!m.exists());
let _ = std::fs::remove_dir_all(&d);
}
/// Issue #136: Windows windows render with DX12 alone (no Vulkan driver loaded) by default.
#[test]
fn window_backends_follow_the_platform_default() {
let eframe::egui_wgpu::WgpuSetup::CreateNew(n) = window_wgpu_options().wgpu_setup else { panic!("expected CreateNew") };
let b = n.instance_descriptor.backends;
if std::env::var_os("LIGHTCRAFT_GPU_BACKEND").is_none() && std::env::var_os("WGPU_BACKEND").is_none() {
if cfg!(windows) {
assert_eq!(b, eframe::wgpu::Backends::DX12);
} else if cfg!(target_os = "macos") {
assert_eq!(b, eframe::wgpu::Backends::METAL);
}
}
assert!(!b.is_empty());
}
#[test]
fn ui_json_is_written_atomically() {
let d = dir("atomic");
let path = d.join("ui.json");
std::fs::write(&path, b"old").unwrap();
write_atomic(&path, b"{\"new\": true}").unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"{\"new\": true}");
assert!(!d.join("ui.json.tmp").exists());
// a failed write leaves the old file
let missing = d.join("gone").join("ui.json");
assert!(write_atomic(&missing, b"x").is_err());
assert_eq!(std::fs::read(&path).unwrap(), b"{\"new\": true}");
let ui = UiState::default();
write_atomic(&path, &serde_json::to_vec_pretty(&ui).unwrap()).unwrap();
let (loaded, warning, _) = load_prefs_at(&path);
assert!(loaded.is_some() && warning.is_none());
let _ = std::fs::remove_dir_all(&d);
}
}
+40 -22
View File
@@ -148,7 +148,7 @@ fn yields_to_text(sc: &str) -> bool {
/// `&` marks a mnemonic in muda labels.
fn label_text(s: &str) -> String {
s.replace('&', "&&")
lightcraft_ui_egui::i18n::tr(s).replace('&', "&&")
}
fn structure_of(bar: &[(String, Vec<MenuNode>)]) -> String {
@@ -171,7 +171,7 @@ fn structure_of(bar: &[(String, Vec<MenuNode>)]) -> String {
out.push(';');
}
}
let mut s = String::new();
let mut s = format!("{};", lightcraft_ui_egui::i18n::is_japanese());
for (t, items) in bar {
s.push_str(t);
walk(items, &mut s);
@@ -210,30 +210,30 @@ impl NativeMenu {
// the application menu
let app_menu = Submenu::new("LightCraft", true);
let about = MenuItem::with_id("app.about", "About LightCraft", true, None);
let settings = MenuItem::with_id(SETTINGS, app.ui.language.tr("Settings…"), true, accelerator(SETTINGS_KEY));
let quit = MenuItem::with_id(QUIT, "Quit LightCraft", true, accelerator("Cmd+Q"));
let about = MenuItem::with_id("app.about", lightcraft_ui_egui::i18n::tr("About LightCraft"), true, None);
let settings = MenuItem::with_id(SETTINGS, lightcraft_ui_egui::i18n::tr("Settings…"), true, accelerator(SETTINGS_KEY));
let quit = MenuItem::with_id(QUIT, lightcraft_ui_egui::i18n::tr("Quit LightCraft"), true, accelerator("Cmd+Q"));
let _ = app_menu.append_items(&[
&about,
&PredefinedMenuItem::separator(),
&settings,
&PredefinedMenuItem::separator(),
&PredefinedMenuItem::services(None),
&PredefinedMenuItem::services(Some(lightcraft_ui_egui::i18n::tr("Services"))),
&PredefinedMenuItem::separator(),
&PredefinedMenuItem::hide(Some("Hide LightCraft")),
&PredefinedMenuItem::hide_others(None),
&PredefinedMenuItem::show_all(None),
&PredefinedMenuItem::hide(Some(lightcraft_ui_egui::i18n::tr("Hide LightCraft"))),
&PredefinedMenuItem::hide_others(Some(lightcraft_ui_egui::i18n::tr("Hide Others"))),
&PredefinedMenuItem::show_all(Some(lightcraft_ui_egui::i18n::tr("Show All"))),
&PredefinedMenuItem::separator(),
&quit,
]);
let _ = self.menu.append(&app_menu);
for (title, nodes) in &bar {
let sub = Submenu::new(title, true);
if title == app.ui.language.tr("Window") {
let sub = Submenu::new(label_text(title), true);
if title == "Window" {
let _ = sub.append_items(&[
&PredefinedMenuItem::minimize(None),
&PredefinedMenuItem::maximize(Some("Zoom")),
&PredefinedMenuItem::minimize(Some(lightcraft_ui_egui::i18n::tr("Minimize"))),
&PredefinedMenuItem::maximize(Some(lightcraft_ui_egui::i18n::tr("Zoom"))),
&PredefinedMenuItem::separator(),
]);
}
@@ -245,15 +245,21 @@ impl NativeMenu {
.collect();
let nodes = tidy_separators(nodes);
self.append_nodes(&sub, &nodes);
if title == app.ui.language.tr("File") {
if title == "File" {
// ⌘W, the system's own item
let _ = sub.append_items(&[&PredefinedMenuItem::separator(), &PredefinedMenuItem::close_window(None)]);
let _ = sub.append_items(&[
&PredefinedMenuItem::separator(),
&PredefinedMenuItem::close_window(Some(lightcraft_ui_egui::i18n::tr("Close Window"))),
]);
}
if title == app.ui.language.tr("Window") {
let _ = sub.append_items(&[&PredefinedMenuItem::separator(), &PredefinedMenuItem::bring_all_to_front(None)]);
if title == "Window" {
let _ = sub.append_items(&[
&PredefinedMenuItem::separator(),
&PredefinedMenuItem::bring_all_to_front(Some(lightcraft_ui_egui::i18n::tr("Bring All to Front"))),
]);
sub.set_as_windows_menu_for_nsapp();
}
if title == app.ui.language.tr("Help") {
if title == "Help" {
sub.set_as_help_menu_for_nsapp();
}
let _ = self.menu.append(&sub);
@@ -282,12 +288,23 @@ impl NativeMenu {
let accel = shortcut.as_deref().filter(|s| !CONTEXTUAL.contains(s)).and_then(accelerator);
let handle = match checked {
Some(c) => {
let it = CheckMenuItem::with_id(key.clone(), label_text(label), *enabled, *c, accel);
let it = CheckMenuItem::with_id(
key.clone(),
lightcraft_ui_egui::menubar::display_item_label(id, params, label).replace('&', "&&"),
*enabled,
*c,
accel,
);
let _ = sub.append(&it);
Handle::Check(it)
}
None => {
let it = MenuItem::with_id(key.clone(), label_text(label), *enabled, accel);
let it = MenuItem::with_id(
key.clone(),
lightcraft_ui_egui::menubar::display_item_label(id, params, label).replace('&', "&&"),
*enabled,
accel,
);
let _ = sub.append(&it);
Handle::Plain(it)
}
@@ -325,6 +342,7 @@ impl NativeMenu {
/// Per frame: run chosen items, then sync labels / enabled / checked and the text-focus
/// accelerators with the app state.
pub fn update(&mut self, app: &mut LightcraftApp, ctx: &egui::Context) {
lightcraft_ui_egui::i18n::set_language(app.ui.language);
while let Ok(key) = self.rx.try_recv() {
if key == QUIT {
ctx.send_viewport_cmd(egui::ViewportCommand::Close);
@@ -352,8 +370,8 @@ impl NativeMenu {
let Some(it) = items.get_mut(&MenuNode::key(id, params)) else { continue };
if it.label != *label {
match &it.handle {
Handle::Plain(h) => h.set_text(label_text(label)),
Handle::Check(h) => h.set_text(label_text(label)),
Handle::Plain(h) => h.set_text(lightcraft_ui_egui::menubar::display_item_label(id, params, label).replace('&', "&&")),
Handle::Check(h) => h.set_text(lightcraft_ui_egui::menubar::display_item_label(id, params, label).replace('&', "&&")),
}
it.label = label.clone();
}
+2
View File
@@ -21,6 +21,8 @@ covers a whole directory), or if a licence file referenced in the Licence column
| `docs/images/*.jpg` (UI) | LightCraft application screenshots | LightCraft contributors | original work (captured with `docs/showcase/`) | MIT OR Apache-2.0 | 2026-09-30 | n/a |
| `docs/brand/` (artcraft-logo and artcraft-mark, SVG and PNG) | ArtCraft wordmark and mark | ArtCraft Team | original work (https://getartcraft.com/) | ArtCraft trademark, see `docs/brand/LICENSE-brand.txt` (not open source) | 2026-10-01 | none |
| `assets/fonts/BIZUDPGothic-*.ttf` | BIZ UDPGothic (Regular, Bold) | Morisawa / BIZ UDGothic Project Authors | https://github.com/googlefonts/morisawa-biz-ud-gothic/tree/18934af56b9c003ca58c54bffbf226848cb11032 | SIL Open Font License 1.1 (`assets/fonts/OFL-BIZUDGothic.txt`) | 2026-10-05 | none |
Notes
- Fonts authored or published by Adobe (Source Sans/Serif/Code, Source Han, …) are not used, even though some are
OFL-licensed: the asset rule excludes anything from Adobe. Source Sans 3 was removed on 2026-09-30 and replaced by Inter.
Binary file not shown.
Binary file not shown.
+93
View File
@@ -0,0 +1,93 @@
Copyright 2022 The BIZ UDGothic Project Authors (https://github.com/googlefonts/morisawa-biz-ud-mincho)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
+197 -26
View File
@@ -1,8 +1,9 @@
//! Crash-safe catalog persistence: an append-only op log plus periodic snapshots.
//!
//! Files (in a [`Store`]):
//! - `catalog.snap` — `{"format":"lightcraft-catalog","version":1,"seq":N,"catalog":{…}}`, the
//! state after op `N`; replaced atomically (temp + fsync + rename).
//! - `catalog.snap` — `{"format":"lightcraft-catalog","version":V,"seq":N,"catalog":{…}}`, the
//! state after op `N` in catalog format `V` ([`VERSION`]); replaced atomically (temp + fsync +
//! rename).
//! - `catalog.log` — JSON lines, one per op applied after the snapshot:
//! `{"seq":N,"crc":C,"op":{…}}` where `C` is the CRC-32 of the op's JSON text. Appends are
//! fsynced before [`Journal::append`] returns.
@@ -10,6 +11,12 @@
//! Loading = snapshot + replay of the records with `seq > snapshot seq`. Recovery rules:
//! - a **torn final record** (crash mid-append: truncated line or bad CRC at the end) is dropped
//! and the file is cut back to the last good record, so later appends start on a clean line;
//! - a **failed append** (disk full, a dropped network share) may leave part of its batch in the
//! file: [`Journal::append`] cuts the log back to the last whole record before returning the
//! error (or, if that fails too, before the next append), so the retry — the same ops with the
//! same `seq`s — starts on a clean line. Logs written by builds without that cut-back hold the
//! fragment glued to the first retried record on one line; the loader recognises the whole,
//! CRC-valid record at the end of such a line and skips only the fragment;
//! - records already covered by the snapshot (crash between writing the snapshot and resetting
//! the log) are skipped;
//! - a bad record **followed by good ones** is real damage: replay stops there, the log is kept
@@ -27,6 +34,33 @@
//!
//! At most one snapshot is in flight; a synchronous [`Journal::snapshot`] and dropping the journal
//! first wait for it, so an older snapshot can never replace a newer one.
//!
//! # Format versions
//!
//! The snapshot's `version` is the catalog format, and it covers the log beside it: the log has
//! no header of its own, and is only ever appended to under a snapshot of this build's format.
//!
//! | Version | Written by | Adds |
//! |---|---|---|
//! | (none) | early builds: a log without a snapshot | |
//! | 1 | up to v0.2.0 | |
//! | 2 | after v0.2.0 | `Op::SetBrowsed`, `Catalog.browsed`, `Photo.local_baseline` |
//! | 3 | 2026-10 | `Photo.fallback_matrix` (raw without a colour matrix of its own: relative WB) |
//!
//! Rules:
//! - **Bump [`VERSION`]** (and add a row above) in the change that adds an [`Op`] variant or a
//! serialized field to the catalog, a photo, an album, a stack or develop settings that an older
//! build would misread or silently drop. New fields must deserialize from older data
//! (`#[serde(default)]`), so newer builds keep reading every older format.
//! - **Opening an older format upgrades it**: right after loading, [`Journal::open`] writes a
//! snapshot in this build's format (also for a new library), before anything is appended — so a
//! library is never a mix, and older builds refuse it from then on instead of reading part of it.
//! [`LoadReport::upgraded_from`] says when that happened.
//! - **A newer format is refused, untouched**: a snapshot whose `version` is above [`VERSION`],
//! or a log record whose CRC matches (so it is exactly what was written) but whose op doesn't
//! parse, fails [`Journal::open`] with [`CatalogError::Newer`] before any file is modified — it
//! is never mistaken for a torn tail or damage. (Builds up to v0.2.0 refuse a v2 snapshot with
//! "unsupported format".)
use crate::store::Store;
use crate::{Catalog, CatalogError, Op, Result};
@@ -34,7 +68,9 @@ use crate::{Catalog, CatalogError, Op, Result};
pub const SNAPSHOT: &str = "catalog.snap";
pub const LOG: &str = "catalog.log";
const FORMAT: &str = "lightcraft-catalog";
const VERSION: u32 = 1;
/// The catalog format this build writes (and the newest it reads). See the module docs →
/// *Format versions*; bump it whenever an [`Op`] variant or a serialized field is added.
pub const VERSION: u32 = 3;
/// When [`Journal::wants_snapshot`] says it's time to compact the log.
#[derive(Clone, Copy, Debug)]
@@ -62,10 +98,16 @@ pub struct LoadReport {
pub failed: usize,
/// Bytes of a torn final record that were dropped.
pub torn_bytes: u64,
/// Partial records left mid-log by a failed append that was retried (written by builds that
/// didn't cut the log back after a failed write): skipped, the retried records were replayed.
pub torn_fragments: usize,
/// The damaged log was preserved under this name.
pub damaged: Option<String>,
/// No catalog files existed (a new library).
pub created: bool,
/// The library was in an older catalog format (`0` = before format versions: a log without a
/// snapshot) and was rewritten in this version's format ([`VERSION`]).
pub upgraded_from: Option<u32>,
}
/// Where persistence time goes (reported by `library.info` → `persistence`; printed to stderr
@@ -140,6 +182,9 @@ pub struct Journal {
pending: Option<Pending>,
/// After a failed background snapshot: don't retry before the log has this many records.
retry_at_records: u64,
/// A failed append may have left part of its records after `log_bytes`, and cutting them off
/// failed too: the next append cuts the log back to `log_bytes` first.
tail_dirty: bool,
}
/// A background snapshot in flight.
@@ -168,18 +213,58 @@ pub fn encode_record(seq: u64, op: &Op) -> String {
format!("{{\"seq\":{seq},\"crc\":{crc},\"op\":{body}}}")
}
/// Parse one log line (without the newline). `None` if it's malformed or the CRC doesn't match.
/// Parse one log line (without the newline). `None` if it's malformed, the CRC doesn't match, or
/// the op is one this version doesn't know (see [`Journal::open`]).
pub fn decode_record(line: &str) -> Option<(u64, Op)> {
let rest = line.strip_prefix("{\"seq\":")?;
let (seq, rest) = rest.split_once(",\"crc\":")?;
let (crc, rest) = rest.split_once(",\"op\":")?;
let body = rest.strip_suffix('}')?;
let seq: u64 = seq.parse().ok()?;
let crc: u32 = crc.parse().ok()?;
if crc32fast::hash(body.as_bytes()) != crc {
return None;
match decode_line(line) {
Line::Record(seq, op) => Some((seq, op)),
Line::Newer(_) | Line::Bad => None,
}
serde_json::from_str(body).ok().map(|op| (seq, op))
}
/// One log line, decoded.
enum Line {
Record(u64, Op),
/// The CRC matches — the line is exactly what was written — but the op doesn't parse: it
/// was written by a newer LightCraft (an op or field this version doesn't know), not damaged.
Newer(u64),
/// Malformed or a CRC mismatch: torn or damaged.
Bad,
}
fn decode_line(line: &str) -> Line {
let parts = || {
let rest = line.strip_prefix("{\"seq\":")?;
let (seq, rest) = rest.split_once(",\"crc\":")?;
let (crc, rest) = rest.split_once(",\"op\":")?;
let body = rest.strip_suffix('}')?;
Some((seq.parse::<u64>().ok()?, crc.parse::<u32>().ok()?, body))
};
let Some((seq, crc, body)) = parts() else { return Line::Bad };
if crc32fast::hash(body.as_bytes()) != crc {
return Line::Bad;
}
match serde_json::from_str(body) {
Ok(op) => Line::Record(seq, op),
Err(_) => Line::Newer(seq),
}
}
/// The error for a library written by a newer LightCraft.
fn newer(what: String) -> CatalogError {
CatalogError::Newer(format!("{what}; this version reads catalog format v{VERSION} and older"))
}
/// A line that doesn't decode but ends with a whole record: a fragment of a failed append followed
/// by the record that retried it (the retry re-encodes the same ops with the same `seq`s). Returns
/// the fragment's length and the record, if that record continues the log (`seq <= last + 1`).
/// The record's CRC must match, so damage is never mistaken for it.
fn resync(line: &[u8], last: u64) -> Option<(usize, (u64, Op))> {
const START: &[u8] = b"{\"seq\":";
(1..line.len().saturating_sub(START.len() - 1)).filter(|&i| line[i..].starts_with(START)).find_map(|i| {
let rec = std::str::from_utf8(&line[i..]).ok().map(|l| l.trim_end_matches('\r')).and_then(decode_record)?;
(rec.0 <= last + 1).then_some((i, rec))
})
}
/// Write `catalog.snap` for the state after op `seq`, streaming the JSON into the store (no
@@ -205,10 +290,16 @@ fn write_snapshot(store: &mut dyn Store, seq: u64, catalog: &Catalog) -> std::io
Ok(SnapshotTiming { serialize_ms, write_sync_ms: (ms_since(t0) - serialize_ms).max(0.0), bytes, ..Default::default() })
}
/// Just the identification of `catalog.snap` (read before the catalog itself).
#[derive(serde::Deserialize)]
struct SnapFile {
struct SnapHeader {
format: String,
version: u32,
}
/// The whole `catalog.snap` (after [`SnapHeader`] was checked; `format` and `version` are ignored).
#[derive(serde::Deserialize)]
struct SnapFile {
seq: u64,
catalog: Catalog,
}
@@ -221,15 +312,20 @@ impl Journal {
let snap = store.read(SNAPSHOT).map_err(io)?;
let log = store.read(LOG).map_err(io)?;
report.created = snap.is_none() && log.is_none();
let (mut catalog, snapshot_seq) = match snap {
let (mut catalog, snapshot_seq, snapshot_version) = match snap {
Some(bytes) => {
let s: SnapFile = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
if s.format != FORMAT || s.version > VERSION {
return Err(CatalogError::Corrupt(format!("{SNAPSHOT}: unsupported format {} v{}", s.format, s.version)));
// the header first: a newer snapshot may not parse as this version's catalog
let h: SnapHeader = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
if h.format != FORMAT {
return Err(CatalogError::Corrupt(format!("{SNAPSHOT}: not a LightCraft catalog (format {:?})", h.format)));
}
(s.catalog, s.seq)
if h.version > VERSION {
return Err(newer(format!("{SNAPSHOT} is catalog format v{}", h.version)));
}
let s: SnapFile = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
(s.catalog, s.seq, Some(h.version))
}
None => (Catalog::new(), 0),
None => (Catalog::new(), 0, None),
};
report.snapshot_seq = snapshot_seq;
let mut j = Journal {
@@ -242,6 +338,7 @@ impl Journal {
stats: PersistStats::default(),
pending: None,
retry_at_records: 0,
tail_dirty: false,
};
let log = log.unwrap_or_default();
@@ -261,7 +358,22 @@ impl Journal {
good_end = next;
continue;
}
match line.and_then(decode_record) {
let mut record = line.and_then(decode_record);
if record.is_none()
&& let Some((at, r)) = resync(&log[pos..line_end], j.seq)
{
// the torn start of a failed append, then the retried records (older builds
// appended them right after the fragment)
log::warn!("catalog log: skipped a partial record ({at} bytes) left by a failed write");
report.torn_fragments += 1;
record = Some(r);
}
if record.is_none()
&& let Some(Line::Newer(seq)) = line.map(decode_line)
{
return Err(newer(format!("{LOG} record {seq} holds a change this version doesn't know")));
}
match record {
Some((seq, op)) if seq <= j.seq => {
// already in the snapshot
let _ = op;
@@ -277,7 +389,15 @@ impl Journal {
}
_ => {
// bad record (or a gap): torn tail if nothing good follows, else damage
let rest_has_good = log[next..].split(|b| *b == b'\n').any(|l| std::str::from_utf8(l).ok().and_then(decode_record).is_some());
let mut rest_has_good = false;
for l in log[next..].split(|b| *b == b'\n') {
match std::str::from_utf8(l).map(|l| decode_line(l.trim_end_matches('\r'))) {
Ok(Line::Record(..)) => rest_has_good = true,
// refused before anything is modified
Ok(Line::Newer(seq)) => return Err(newer(format!("{LOG} record {seq} holds a change this version doesn't know"))),
Ok(Line::Bad) | Err(_) => {}
}
}
if rest_has_good {
damaged_at = Some(pos);
}
@@ -301,10 +421,34 @@ impl Journal {
report.torn_bytes = (log.len() - good_end) as u64;
log::warn!("catalog log: dropped a torn final record ({} bytes)", report.torn_bytes);
j.store.truncate(LOG, good_end as u64).map_err(io)?;
j.log_bytes = good_end as u64;
} else if needs_newline {
j.store.append(LOG, b"\n").map_err(io)?;
j.log_bytes = good_end as u64 + 1;
} else {
j.log_bytes = good_end as u64;
}
}
// Bring an older (or new, or snapshot-less) library to this version's format before
// anything is appended: a snapshot in this format makes older builds refuse the library
// instead of misreading ops or dropping fields they don't know.
if report.created {
// tiny: written directly, not counted as a compaction
let empty = format!("{{\"format\":\"{FORMAT}\",\"version\":{VERSION},\"seq\":0,\"catalog\":{}}}\n", catalog.to_snapshot());
if let Err(e) = j.store.write_atomic(SNAPSHOT, empty.as_bytes()) {
log::warn!("catalog: can't write the new library's snapshot: {e}");
}
} else if report.damaged.is_none() && snapshot_version != Some(VERSION) {
report.upgraded_from = Some(snapshot_version.unwrap_or(0));
match j.snapshot(&catalog) {
Ok(()) => {
if let Some(v) = report.upgraded_from {
log::info!("catalog: upgraded from format v{v} to v{VERSION}");
}
}
// nothing lost: the old files are intact and still load; retried on next open
Err(e) => log::warn!("catalog: can't write the format v{VERSION} snapshot: {e}"),
}
j.log_bytes = good_end as u64;
}
catalog.revision = 0;
Ok((j, catalog, report))
@@ -323,7 +467,20 @@ impl Journal {
buf.push_str(&encode_record(seq, op));
buf.push('\n');
}
self.store.append(LOG, buf.as_bytes()).map_err(io)?;
if self.tail_dirty {
self.store.truncate(LOG, self.log_bytes).map_err(io)?;
self.tail_dirty = false;
}
if let Err(e) = self.store.append(LOG, buf.as_bytes()) {
// Part of the batch may be in the file. Cut it back to the last whole record, so the
// retry (same seqs) starts on a clean line instead of after a fragment that would
// read back as damage.
if let Err(t) = self.store.truncate(LOG, self.log_bytes) {
log::error!("catalog: can't cut the log back after a failed append: {t}");
self.tail_dirty = true;
}
return Err(io(e));
}
if let Some(p) = self.pending.as_mut() {
p.tail.extend_from_slice(buf.as_bytes());
p.tail_records += ops.len() as u64;
@@ -343,12 +500,22 @@ impl Journal {
/// Write a snapshot of `catalog` (which must reflect every appended op) and reset the log.
pub fn snapshot(&mut self, catalog: &Catalog) -> Result<()> {
self.snapshot_with_unlogged(catalog, 0)
}
/// [`Journal::snapshot`] of a catalog that also reflects `unlogged` ops applied after the
/// last appended one but never written to the log (their append failed). The snapshot counts
/// them (it holds the state after op `seq + unlogged`), so they are saved and must not be
/// appended afterwards. Once the snapshot is durable [`Journal::seq`] includes them, even if
/// resetting the log then fails (the error is still returned).
pub fn snapshot_with_unlogged(&mut self, catalog: &Catalog, unlogged: u64) -> Result<()> {
// never two snapshot writers; the one in flight is older, so it must land first
if let Err(e) = self.wait() {
log::warn!("catalog: background snapshot failed ({e}); writing one now");
}
let t0 = web_time::Instant::now();
let mut timing = match write_snapshot(self.store.as_mut(), self.seq, catalog) {
let seq = self.seq + unlogged;
let mut timing = match write_snapshot(self.store.as_mut(), seq, catalog) {
Ok(t) => t,
Err(e) => {
// nothing lost (the log is whole); don't retry on every append
@@ -357,6 +524,9 @@ impl Journal {
return Err(io(e));
}
};
// the snapshot is durable: it holds every op up to `seq`
self.seq = seq;
self.snapshot_seq = seq;
let t2 = web_time::Instant::now();
// A crash here leaves old records in the log; they are skipped by seq on load.
self.store.write_atomic(LOG, b"").map_err(io)?;
@@ -365,10 +535,10 @@ impl Journal {
timing.blocking_ms = timing.total_ms;
timing.records = self.log_records;
self.record_snapshot(timing);
self.snapshot_seq = self.seq;
self.log_records = 0;
self.log_bytes = 0;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
@@ -459,6 +629,7 @@ impl Journal {
self.log_records = p.tail_records;
self.log_bytes = p.tail.len() as u64;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
+17
View File
@@ -5,6 +5,11 @@
//! snapshot — crash-safe and diff-friendly (see [`journal`]);
//! - **undo/redo**: the engine keeps inverse ops;
//! - **determinism**: replaying the log reproduces the state exactly (property-tested).
//!
//! **Catalog format version** ([`journal::VERSION`], see [`journal`] → *Format versions*): adding
//! an [`Op`] variant or a serialized field means bumping it. Newer builds read every older format
//! (and upgrade it on open); older builds refuse a newer library with [`CatalogError::Newer`]
//! instead of reading part of it.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -12,6 +17,7 @@ pub mod dates;
pub mod journal;
pub mod keywords;
pub mod local;
pub mod lock;
pub mod model;
pub mod query;
pub mod rules;
@@ -27,6 +33,7 @@ pub use journal::{Journal, LoadReport, PersistStats, SnapshotPolicy, SnapshotTim
pub use keywords::KeywordNode;
use lightcraft_develop::DevelopSettings;
pub use local::{DEFAULT_FORGET_DAYS, ForgetPlan, folder_of};
pub use lock::{LibraryLock, LockError, LockOwner};
pub use model::*;
pub use query::{DateGroup, Filter, RatingOp, Sort, SortKey};
pub use rules::{Match, Rule, RuleSet};
@@ -47,6 +54,10 @@ pub enum CatalogError {
Corrupt(String),
#[error("catalog storage: {0}")]
Io(String),
/// The library was written by a newer LightCraft (a newer catalog format, or a change this
/// version doesn't know). Nothing was read into the session and nothing was modified.
#[error("this library was written by a newer version of LightCraft ({0}); update LightCraft to open it. The library was left unchanged.")]
Newer(String),
}
pub type Result<T> = std::result::Result<T, CatalogError>;
@@ -644,6 +655,12 @@ mod tests;
#[cfg(test)]
mod tests_background;
#[cfg(test)]
mod tests_format_version;
#[cfg(test)]
mod tests_journal;
#[cfg(test)]
mod tests_local;
#[cfg(test)]
mod tests_lock;
#[cfg(test)]
mod tests_torn_append;
+140
View File
@@ -0,0 +1,140 @@
//! One process per library: an exclusive OS lock on `<library>/catalog.lock`, held for as long as
//! the library is open (issue #99).
//!
//! Two processes on the same library (the app and `lightcraft-cli --library`, two app instances,
//! two computers on a shared folder) would each keep their own catalog and `seq` counter, and
//! whichever wrote the last snapshot would silently drop the other's edits. So opening a library
//! takes the lock first, and a second opener gets [`LockError::InUse`] instead.
//!
//! - The lock is the OS's (`flock` on Unix, `LockFileEx` on Windows, via [`std::fs::File::try_lock`]):
//! it is released when the [`LibraryLock`] is dropped and, after a crash or a kill, by the OS when
//! the process ends. A `catalog.lock` file left behind is therefore never "stale": only a lock
//! that's actually held refuses the open.
//! - Who holds it (process id, computer, program, version) is written to `catalog.lock.owner`, for
//! the error message. It's a separate file because a Windows lock also blocks reading the
//! locked file. It's only a hint: it may be missing or left over from a crash.
//! - File systems without lock support (some network shares): the library opens unlocked, with a
//! warning — refusing it would lock the user out of their own library. Locks over SMB/NFS are
//! advisory and depend on the server; the owner file is then the only hint.
//! - Browser storage (OPFS) has no file locks: see the web host for its own guard.
use std::fs::{File, OpenOptions, TryLockError};
use std::io;
use std::path::{Path, PathBuf};
/// The lock file in the library directory.
pub const LOCK: &str = "catalog.lock";
/// Who holds the lock (JSON, see [`LockOwner`]).
pub const OWNER: &str = "catalog.lock.owner";
/// Who has the library open (as written in [`OWNER`]).
#[derive(Clone, Debug, Default, PartialEq, serde::Serialize, serde::Deserialize)]
#[serde(default)]
pub struct LockOwner {
pub pid: u32,
/// Computer name, if known.
pub host: String,
/// The program (`LightCraft`, `lightcraft-cli`…) and its version.
pub program: String,
pub version: String,
/// When it opened the library (seconds since 1970).
pub since: u64,
}
impl LockOwner {
/// This process, as `program`.
pub fn this_process(program: &str) -> LockOwner {
LockOwner {
pid: std::process::id(),
host: host_name(),
program: program.to_string(),
version: env!("CARGO_PKG_VERSION").to_string(),
since: web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0),
}
}
/// "LightCraft 0.3.0 (process 123 on studio-mac)".
pub fn describe(&self) -> String {
let program = if self.program.is_empty() { "another LightCraft" } else { self.program.as_str() };
let version = if self.version.is_empty() { String::new() } else { format!(" {}", self.version) };
let host = if self.host.is_empty() {
String::new()
} else if self.host == host_name() {
" on this computer".to_string()
} else {
format!(" on {}", self.host)
};
format!("{program}{version} (process {}{host})", self.pid)
}
}
/// The computer's name, best effort (empty if unknown).
fn host_name() -> String {
let env = ["COMPUTERNAME", "HOSTNAME"].iter().find_map(|k| std::env::var(k).ok().filter(|h| !h.trim().is_empty()));
env.or_else(|| std::fs::read_to_string("/etc/hostname").ok()).map(|h| h.trim().to_string()).filter(|h| !h.is_empty()).unwrap_or_default()
}
#[derive(Debug, thiserror::Error)]
pub enum LockError {
/// Another process has the library open.
#[error("This library is already open in {}. Close it there first.", .0.as_ref().map(LockOwner::describe).unwrap_or_else(|| "another LightCraft program".into()))]
InUse(Option<LockOwner>),
}
/// The exclusive lock on a library directory; released on drop (and by the OS if the process
/// dies).
#[derive(Debug)]
pub struct LibraryLock {
dir: PathBuf,
/// `None` when the file system can't lock (the library is open unlocked).
_file: Option<File>,
held: bool,
}
impl LibraryLock {
/// Lock `dir` (which must exist) for this process. Fails only when another process holds it.
pub fn acquire(dir: &Path, program: &str) -> Result<LibraryLock, LockError> {
let unlocked = |why: io::Error| {
log::warn!("library {}: can't lock {LOCK} ({why}); opening it without protection against a second program", dir.display());
LibraryLock { dir: dir.to_path_buf(), _file: None, held: false }
};
let file = match OpenOptions::new().read(true).write(true).create(true).truncate(false).open(dir.join(LOCK)) {
Ok(f) => f,
Err(e) => return Ok(unlocked(e)),
};
match file.try_lock() {
Ok(()) => {}
Err(TryLockError::WouldBlock) => {
let owner = std::fs::read(dir.join(OWNER)).ok().and_then(|b| serde_json::from_slice::<LockOwner>(&b).ok());
return Err(LockError::InUse(owner));
}
Err(TryLockError::Error(e)) => return Ok(unlocked(e)),
}
let owner = LockOwner::this_process(program);
if let Ok(json) = serde_json::to_vec_pretty(&owner)
&& let Err(e) = std::fs::write(dir.join(OWNER), json)
{
log::warn!("library {}: can't write {OWNER}: {e}", dir.display());
}
Ok(LibraryLock { dir: dir.to_path_buf(), _file: Some(file), held: true })
}
/// The OS lock is held (`false`: the file system can't lock; opened unprotected).
pub fn held(&self) -> bool {
self.held
}
pub fn dir(&self) -> &Path {
&self.dir
}
}
impl Drop for LibraryLock {
/// Remove the owner note while still holding the lock (so it can't delete the next holder's),
/// then the file handle closes and the OS releases the lock.
fn drop(&mut self) {
if self.held {
let _ = std::fs::remove_file(self.dir.join(OWNER));
}
}
}
+41 -1
View File
@@ -276,6 +276,13 @@ pub struct Photo {
/// whether the user changed anything since (see [`crate::local`]).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub local_baseline: Option<u64>,
/// Raw files: the decoder had no colour matrix for this file (`matrix_is_fallback`: every
/// non-DNG raw today), so its colour is estimated from its own embedded JPEG and its white
/// balance controls are relative to the as-shot look ([`Photo::relative_wb`]). Recorded at
/// import (catalog format v3). `None`: imported before it was recorded — see
/// [`Photo::relative_wb`] for how those photos keep the white balance they were edited with.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub fallback_matrix: Option<bool>,
}
/// What assisted culling measured on a photo.
@@ -324,6 +331,7 @@ impl Photo {
analysis: None,
preview_only: None,
local_baseline: None,
fallback_matrix: None,
}
}
/// A raw file developed from its sensor data: not a rendered image, and not a raw shown from
@@ -331,6 +339,14 @@ impl Photo {
pub fn develops_raw(&self) -> bool {
self.kind == MediaKind::Raw && self.preview_only.is_none()
}
/// White balance controls are relative to the camera's as-shot look, as for rendered
/// photographs: the raw has vendor WB multipliers but no colour matrix
/// ([`Photo::fallback_matrix`]), so no measured illuminant to put an absolute Kelvin on.
/// Photos imported before that fact was recorded keep the rule they were edited under: Sony
/// ARW relative, every other raw absolute (so a saved Custom white balance reads the same).
pub fn relative_wb(&self) -> bool {
self.develops_raw() && self.fallback_matrix.unwrap_or_else(|| self.format.eq_ignore_ascii_case("ARW"))
}
/// The develop settings import gave this photo: [`Photo::camera_defaults`], or the user's
/// default preset applied on top of them ([`Photo::import_look`]).
pub fn import_defaults(&self) -> DevelopSettings {
@@ -342,7 +358,8 @@ impl Photo {
/// The built-in defaults for this photo, before any user default preset: raws start from
/// their as-shot white balance; embedded lens corrections on when the file has them.
pub fn camera_defaults(&self) -> DevelopSettings {
let mut d = match self.as_shot_wb {
let wb = if self.relative_wb() { Some((6500.0, 0.0)) } else { self.as_shot_wb };
let mut d = match wb {
Some((t, tint)) => DevelopSettings::for_raw(t, tint),
None => DevelopSettings::default(),
};
@@ -415,4 +432,27 @@ mod edited_tests {
p.develop = Arc::new(DevelopSettings::default());
assert!(!p.is_edited(), "a full reset is unedited too");
}
#[test]
fn relative_wb_follows_the_decoder_fact_and_keeps_old_photos_rule() {
let mut p = Photo::new(PhotoId(1), Source::Demo { scene: 0 }, "a.nef", "NEF", 10, 10, "");
p.kind = MediaKind::Raw;
p.as_shot_wb = Some((5200.0, 4.0));
// imported before the fact was recorded: only ARW was relative
assert!(!p.relative_wb());
assert_eq!(p.camera_defaults().wb.temp, 5200.0);
p.fallback_matrix = Some(true);
assert!(p.relative_wb());
assert_eq!(p.camera_defaults().wb.temp, 6500.0);
p.format = "ARW".into();
p.fallback_matrix = None;
assert!(p.relative_wb());
p.fallback_matrix = Some(false);
assert!(!p.relative_wb(), "a raw with a matrix of its own has absolute WB");
p.fallback_matrix = Some(true);
p.preview_only = Some("unsupported".into());
assert!(!p.relative_wb(), "shown from its embedded JPEG");
let json = serde_json::to_string(&Photo::new(PhotoId(2), Source::Demo { scene: 0 }, "b.jpg", "JPEG", 1, 1, "")).unwrap();
assert!(!json.contains("fallback_matrix"), "{json}");
}
}
+79 -8
View File
@@ -1,11 +1,23 @@
//! Crash-safe writes of user-visible files: exports, renders, DNG conversions, smart previews and
//! XMP sidecars all go through here (the catalog's own files use [`crate::store::FsStore`]).
//!
//! **Write policy** (issue #134): what can't be recreated — originals, XMP sidecars, the catalog,
//! settings, DNGs that replace raws, smart previews (the only stand-in while an original is
//! offline), an edit copy that becomes a library photo — is written **durably**: synced to disk
//! before it counts as written. What can always be made again from the originals — exports,
//! renders, screenshots, merge previews — is written **atomically** only: a half-written file
//! never replaces a good one, but there is no per-file sync (on a USB drive or a NAS that sync
//! dominates a large export).
//!
//! - [`write_atomic`] replaces a file so that after a crash, a full disk or an unplugged drive
//! either the old or the complete new content is there — never a truncated mix. The content
//! goes to a new temp file in the same folder (unique name, `create_new`), is synced to disk and
//! renamed over the target; then the folder is synced (POSIX). Any failure removes the temp
//! file and leaves the target as it was.
//! - [`write_atomic_nosync`] is the same temp file + rename without the two syncs, for
//! recreatable outputs only: a failed write still removes the temp file and leaves the target as
//! it was, but after a power cut the new file may be lost (on some file systems left empty) —
//! it is then simply exported again.
//! - [`write_new`] / [`write_new_unique`] write a file that must not exist yet: the same temp file
//! and sync, plus a read-back check, and the file then appears under its final name **without
//! replacing anything** (a hard link, which fails on a taken name; a check-then-rename where the
@@ -21,6 +33,19 @@ use std::sync::atomic::{AtomicU64, Ordering};
thread_local! {
static FAIL_AFTER: Cell<Option<u64>> = const { Cell::new(None) };
static SYNCS: Cell<u64> = const { Cell::new(0) };
}
/// How many file and folder syncs this module made **on the calling thread** (for tests: a
/// durable write path syncs, a recreatable output doesn't).
#[doc(hidden)]
pub fn syncs_on_this_thread() -> u64 {
SYNCS.with(Cell::get)
}
fn sync_file(f: &File) -> io::Result<()> {
SYNCS.with(|c| c.set(c.get() + 1));
f.sync_all()
}
/// Fault injection for tests (in any crate): while the returned guard lives, every write made by
@@ -78,7 +103,7 @@ fn parent_of(path: &Path) -> &Path {
/// where NTFS journals the rename itself, so errors are ignored.
pub fn sync_dir(dir: &Path) {
if let Ok(d) = File::open(dir) {
let _ = d.sync_all();
let _ = sync_file(&d);
}
}
@@ -100,9 +125,9 @@ fn create_temp(path: &Path) -> io::Result<(PathBuf, File)> {
Err(io::Error::new(io::ErrorKind::AlreadyExists, format!("no free temp file name in {}", dir.display())))
}
/// Write a temp file next to `path` with what `fill` writes, synced to disk. On failure the temp
/// file is removed. Returns its path and length.
fn write_temp(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<(PathBuf, u64)> {
/// Write a temp file next to `path` with what `fill` writes, synced to disk when `sync`. On
/// failure the temp file is removed. Returns its path and length.
fn write_temp(path: &Path, sync: bool, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<(PathBuf, u64)> {
let (tmp, file) = create_temp(path)?;
let r = (|| {
let limit = FAIL_AFTER.with(Cell::get);
@@ -111,7 +136,9 @@ fn write_temp(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()
w.flush()?;
let n = w.n;
let f = w.inner.into_inner().map_err(|e| e.into_error())?;
f.sync_all()?;
if sync {
sync_file(&f)?;
}
Ok(n)
})();
match r {
@@ -132,12 +159,26 @@ pub fn write_atomic(path: &Path, data: &[u8]) -> io::Result<()> {
/// [`write_atomic`] with streamed content: `fill` writes it (an error aborts and leaves the old
/// file). Returns the length written.
pub fn write_atomic_with(path: &Path, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<u64> {
let (tmp, n) = write_temp(path, fill)?;
replace_with(path, true, fill)
}
/// Replace (or create) `path` with `data` atomically but **without syncing** it to disk — only for
/// outputs that can always be recreated (exports, renders, screenshots); see the module docs.
/// Anything that can't be made again goes through [`write_atomic`].
pub fn write_atomic_nosync(path: &Path, data: &[u8]) -> io::Result<()> {
replace_with(path, false, &mut |w| w.write_all(data)).map(|_| ())
}
/// Temp file (synced when `sync`) renamed over `path`, then the folder synced when `sync`.
fn replace_with(path: &Path, sync: bool, fill: &mut dyn FnMut(&mut dyn Write) -> io::Result<()>) -> io::Result<u64> {
let (tmp, n) = write_temp(path, sync, fill)?;
if let Err(e) = fs::rename(&tmp, path) {
let _ = fs::remove_file(&tmp);
return Err(e);
}
sync_dir(parent_of(path));
if sync {
sync_dir(parent_of(path));
}
Ok(n)
}
@@ -155,8 +196,11 @@ pub fn write_new_unique(candidates: &mut dyn Iterator<Item = PathBuf>, data: &[u
let Some(first) = candidates.peek().cloned() else {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "no file name to write"));
};
let (tmp, _) = write_temp(&first, &mut |w| w.write_all(data))?;
let (tmp, _) = write_temp(&first, true, &mut |w| w.write_all(data))?;
let r = (|| {
// Kept (issue #134 point 4): these files replace raws (Convert / Copy as DNG) or become
// library photos (merges). On a local disk the read usually comes from the cache, but on a
// network volume it can go back to the server, and next to encoding a DNG it costs little.
if !same_content(&tmp, data)? {
return Err(io::Error::other("the written file reads back different (failing drive or connection?)"));
}
@@ -287,6 +331,33 @@ mod tests {
let _ = fs::remove_dir_all(&d);
}
#[test]
fn recreatable_outputs_are_atomic_without_a_sync() {
let d = temp("nosync");
let p = d.join("export.jpg");
let before = syncs_on_this_thread();
write_atomic_nosync(&p, b"one").unwrap();
write_atomic_nosync(&p, b"two two").unwrap();
assert_eq!(syncs_on_this_thread(), before, "no file or folder sync");
assert_eq!(fs::read(&p).unwrap(), b"two two");
assert_eq!(names(&d), vec!["export.jpg"], "no temp file left");
// still atomic: a failure part-way leaves the previous file and no temp file
{
let _f = fail_writes_after(3);
assert!(write_atomic_nosync(&p, &[9u8; 10_000]).is_err());
}
assert_eq!(fs::read(&p).unwrap(), b"two two");
assert_eq!(names(&d), vec!["export.jpg"]);
// the durable writers do sync (the file; the folder too where it can be opened)
let before = syncs_on_this_thread();
write_atomic(&p, b"three").unwrap();
let mid = syncs_on_this_thread();
assert!(mid > before);
write_new(&d.join("new.dng"), b"dng").unwrap();
assert!(syncs_on_this_thread() > mid);
let _ = fs::remove_dir_all(&d);
}
#[test]
fn new_files_never_replace_anything() {
let d = temp("new");
+19 -5
View File
@@ -28,9 +28,12 @@ pub trait Store: Send {
self.write_atomic(name, &buf)?;
Ok(buf.len() as u64)
}
/// Append and make durable (fsync) before returning.
/// Append and make durable (fsync) before returning. On an error, part of `data` may have
/// been written (a full disk, a dropped network share): the journal cuts the file back to its
/// previous length with [`Store::truncate`] before appending again.
fn append(&mut self, name: &str, data: &[u8]) -> io::Result<()>;
/// Cut a file to `len` bytes (drop a torn tail before appending again).
/// Cut a file to `len` bytes (drop a torn tail before appending again). A missing file is
/// left missing.
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()>;
/// Human-readable location (diagnostics).
fn describe(&self) -> String;
@@ -132,15 +135,26 @@ impl Store for FsStore {
self.appender = Some((name.to_string(), f));
}
let Some((_, f)) = self.appender.as_mut() else { return Err(io::Error::other("appender not open")) };
f.write_all(data)?;
f.sync_data()
let written = f.write_all(data).and_then(|()| f.sync_data());
if written.is_err() {
// The handle may be dead (a network share that reconnected, a removed volume): the next
// attempt reopens the file. Part of `data` may be in the file; the caller cuts it back
// ([`Store::truncate`]) before appending again.
self.appender = None;
}
written
}
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()> {
if self.appender.as_ref().is_some_and(|(n, _)| n == name) {
self.appender = None;
}
let f = std::fs::OpenOptions::new().write(true).open(self.path(name))?;
let f = match std::fs::OpenOptions::new().write(true).open(self.path(name)) {
Ok(f) => f,
// nothing to cut (an append that failed before creating the file)
Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()),
Err(e) => return Err(e),
};
f.set_len(len)?;
f.sync_all()
}
+2 -1
View File
@@ -180,7 +180,8 @@ fn failed_background_snapshot_loses_nothing_and_retries_later() {
wait_written(&j);
assert!(j.poll().is_err());
assert_eq!((j.snapshot_seq(), j.stats().failed_snapshots), (0, 1));
assert!(g.files.get(SNAPSHOT).is_none());
// still the (empty) snapshot written when the library was created
assert_eq!(g.snapshot_seq_on_disk(), Some(0));
assert_recovers(&g, &live);
// no retry storm: the next attempt waits for another `max_records` records
assert!(!j.wants_snapshot());
+211
View File
@@ -0,0 +1,211 @@
//! Issue #102: catalog format versions. Older formats load and are upgraded; a newer format is
//! refused with a clear error and left untouched (never read in part, truncated or set aside).
use crate::journal::{LOG, SNAPSHOT, VERSION, encode_record};
use crate::*;
fn photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// A catalog with two photos and the log line for a third op on top of it.
fn legacy_parts() -> (Catalog, String, Catalog) {
let mut c = Catalog::new();
for _ in 0..2 {
let op = photo(&mut c);
c.apply(op).unwrap();
}
let base = c.clone();
let id = c.photos().next().unwrap().id;
let op = Op::SetRating { id, rating: 4 };
c.apply(op.clone()).unwrap();
(base, format!("{}\n", encode_record(3, &op)), c)
}
fn snapshot_version(m: &MemStore) -> u64 {
let v: serde_json::Value = serde_json::from_slice(&m.get(SNAPSHOT).unwrap()).unwrap();
v["version"].as_u64().unwrap()
}
#[test]
fn v1_library_loads_and_is_upgraded() {
let (base, log, full) = legacy_parts();
let m = MemStore::new();
m.set(SNAPSHOT, format!("{{\"format\":\"lightcraft-catalog\",\"version\":1,\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot()).into_bytes());
m.set(LOG, log.into_bytes());
let (mut j, c, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(c.to_snapshot(), full.to_snapshot());
assert_eq!((r.replayed, r.upgraded_from), (1, Some(1)));
// rewritten in this format before anything is appended
assert_eq!(snapshot_version(&m), u64::from(VERSION));
assert!(m.get(LOG).unwrap().is_empty());
assert_eq!(j.seq(), 3);
j.append(&[Op::SetLabelName { label: ColorLabel::ALL[0], name: Some("x".into()) }]).unwrap();
drop(j);
let (_, _, r) = Journal::open(Box::new(m)).unwrap();
assert_eq!((r.replayed, r.upgraded_from), (1, None));
}
/// v3 records whether a raw lacks a colour matrix; v2 photos load without it and keep the white
/// balance rule they were edited under (only ARW relative).
#[test]
fn v2_raws_load_without_the_matrix_fact() {
let mut c = Catalog::new();
for (name, format) in [("a.nef", "NEF"), ("b.arw", "ARW")] {
let id = c.alloc_photo_id();
let mut p = Photo::new(id, Source::File { path: format!("/p/{name}") }, name, format, 4, 3, "2026-01-01");
p.kind = MediaKind::Raw;
c.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
}
let snap = c.to_snapshot();
assert!(!snap.contains("fallback_matrix"));
let m = MemStore::new();
m.set(SNAPSHOT, format!("{{\"format\":\"lightcraft-catalog\",\"version\":2,\"seq\":2,\"catalog\":{snap}}}\n").into_bytes());
let (_, loaded, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(r.upgraded_from, Some(2));
assert_eq!(snapshot_version(&m), u64::from(VERSION));
let rel: Vec<_> = loaded.photos().map(|p| (p.format.clone(), p.fallback_matrix, p.relative_wb())).collect();
assert!(rel.contains(&("NEF".into(), None, false)) && rel.contains(&("ARW".into(), None, true)), "{rel:?}");
}
#[test]
fn versionless_log_only_library_is_upgraded() {
let mut c = Catalog::new();
let op = photo(&mut c);
c.apply(op.clone()).unwrap();
let m = MemStore::new();
m.set(LOG, format!("{}\n", encode_record(1, &op)).into_bytes());
let (_, loaded, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
assert_eq!(r.upgraded_from, Some(0));
assert_eq!(snapshot_version(&m), u64::from(VERSION));
}
/// A new library gets a snapshot in this format at once, so an older build never sees a bare log
/// it would read in part.
#[test]
fn new_library_is_written_in_this_format() {
let m = MemStore::new();
let (_, _, r) = Journal::open(Box::new(m.clone())).unwrap();
assert!(r.created);
assert_eq!(r.upgraded_from, None);
assert_eq!(snapshot_version(&m), u64::from(VERSION));
}
fn assert_refused_untouched(m: &MemStore) {
let before = m.files.lock().unwrap().clone();
let e = Journal::open(Box::new(m.clone())).err().expect("refused");
assert!(matches!(e, CatalogError::Newer(_)), "{e:?}");
let msg = e.to_string();
assert!(msg.contains("newer version of LightCraft") && msg.contains("left unchanged"), "{msg}");
assert_eq!(*m.files.lock().unwrap(), before, "nothing modified, nothing renamed");
}
#[test]
fn newer_snapshot_is_refused_untouched() {
let (base, log, _) = legacy_parts();
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":2,\"catalog\":{}}}\n", VERSION + 1, base.to_snapshot()).into_bytes(),
);
m.set(LOG, log.into_bytes());
assert_refused_untouched(&m);
// even when its catalog doesn't parse as this version's
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":2,\"catalog\":{{\"photos\":7}}}}\n", VERSION + 1).into_bytes(),
);
assert_refused_untouched(&m);
let e = Journal::open(Box::new(m)).err().unwrap().to_string();
assert!(e.contains(&format!("v{}", VERSION + 1)), "{e}");
}
/// A log record with a valid CRC whose op this version doesn't know: refused, not taken for a
/// torn tail (truncated) or damage (renamed) — wherever it is in the log.
#[test]
fn unknown_op_in_the_log_is_refused_untouched() {
let (base, log, _) = legacy_parts();
let future = |seq: u64| {
let body = r#"{"op":"fromTheFuture","id":1,"what":"something new"}"#;
format!("{{\"seq\":{seq},\"crc\":{},\"op\":{body}}}\n", crc32fast::hash(body.as_bytes()))
};
let snap = format!("{{\"format\":\"lightcraft-catalog\",\"version\":{VERSION},\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot());
let id = base.photos().next().unwrap().id;
let after = format!("{}\n", encode_record(5, &Op::SetRating { id, rating: 1 }));
for log in [format!("{log}{}", future(4)), format!("{log}{}{after}", future(4)), format!("{}{log}", future(3))] {
let m = MemStore::new();
m.set(SNAPSHOT, snap.clone().into_bytes());
m.set(LOG, log.into_bytes());
assert_refused_untouched(&m);
}
// a known op with a field of an unknown shape is "newer" too
let body = r#"{"op":"setRating","id":1,"rating":"five stars"}"#;
let m = MemStore::new();
m.set(SNAPSHOT, snap.into_bytes());
m.set(LOG, format!("{log}{{\"seq\":4,\"crc\":{},\"op\":{body}}}\n", crc32fast::hash(body.as_bytes())).into_bytes());
assert_refused_untouched(&m);
}
/// A CRC mismatch is still damage / a torn tail (not "newer").
#[test]
fn bad_crc_is_still_a_torn_tail() {
let (base, log, _) = legacy_parts();
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{VERSION},\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot()).into_bytes(),
);
let body = r#"{"op":"fromTheFuture"}"#;
m.set(LOG, format!("{log}{{\"seq\":4,\"crc\":1,\"op\":{body}}}\n").into_bytes());
let (_, _, r) = Journal::open(Box::new(m)).unwrap();
assert!(r.torn_bytes > 0 && r.damaged.is_none(), "{r:?}");
}
/// Guard for the version rule (see `journal` → *Format versions*): adding an [`Op`] variant breaks
/// this match. When it does, bump `journal::VERSION`, add a row to the format table, and add the
/// variant here under the new version.
#[test]
fn op_variants_are_versioned() {
/// The newest format that added serialized fields but no op.
const FIELDS_ONLY_SINCE: u32 = 3;
fn since(op: &Op) -> u32 {
match op {
Op::AddPhoto { .. }
| Op::RemovePhoto { .. }
| Op::SetRating { .. }
| Op::SetFlag { .. }
| Op::SetLabel { .. }
| Op::SetDevelop { .. }
| Op::SetMeta { .. }
| Op::SetDeleted { .. }
| Op::SetLocal { .. }
| Op::SetVersions { .. }
| Op::SetHistory { .. }
| Op::PushHistory { .. }
| Op::AddAlbum { .. }
| Op::RemoveAlbum { .. }
| Op::RenameAlbum { .. }
| Op::MoveAlbum { .. }
| Op::SetAlbumPhotos { .. }
| Op::SetAlbumCover { .. }
| Op::SetAlbumRules { .. }
| Op::AddStack { .. }
| Op::RemoveStack { .. }
| Op::SetStack { .. }
| Op::SetCaptured { .. }
| Op::SetAnalysis { .. }
| Op::SetFile { .. }
| Op::Relink { .. }
| Op::SetContent { .. }
| Op::SetLabelName { .. }
| Op::Batch { .. } => 1,
Op::SetBrowsed { .. } => 2,
}
}
let newest = since(&Op::SetBrowsed { folder: String::new(), at: None });
// v3 added a field only (`Photo.fallback_matrix`); a new op takes the next version.
assert_eq!(newest.max(FIELDS_ONLY_SINCE), VERSION, "the newest op's (or field's) version must be the current format version");
}
+6 -1
View File
@@ -283,7 +283,12 @@ fn streamed_snapshot_keeps_the_on_disk_format() {
let (mut j, c, _) = open(&m);
assert_eq!(c.to_snapshot(), full.to_snapshot());
j.snapshot(&c).unwrap();
let legacy = format!("{{\"format\":\"lightcraft-catalog\",\"version\":1,\"seq\":{},\"catalog\":{}}}\n", j.seq(), c.to_snapshot());
let legacy = format!(
"{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":{},\"catalog\":{}}}\n",
crate::journal::VERSION,
j.seq(),
c.to_snapshot()
);
assert_eq!(String::from_utf8(m.get(SNAPSHOT).unwrap()).unwrap(), legacy);
assert_eq!(j.stats().last_snapshot.bytes, legacy.len() as u64);
+93
View File
@@ -0,0 +1,93 @@
//! Issue #99: one process per library.
use std::path::PathBuf;
use crate::lock::{LOCK, OWNER};
use crate::*;
fn temp_dir(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-lock-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn second_opener_is_refused_until_the_first_lets_go() {
let dir = temp_dir("twice");
let first = LibraryLock::acquire(&dir, "LightCraft").unwrap();
assert!(first.held());
let owner: LockOwner = serde_json::from_slice(&std::fs::read(dir.join(OWNER)).unwrap()).unwrap();
assert_eq!((owner.pid, owner.program.as_str()), (std::process::id(), "LightCraft"));
let e = LibraryLock::acquire(&dir, "lightcraft-cli").unwrap_err();
let LockError::InUse(Some(who)) = &e else { panic!("{e:?}") };
assert_eq!(who.pid, std::process::id());
let msg = e.to_string();
assert!(msg.contains("already open in LightCraft") && msg.contains(&format!("process {}", std::process::id())), "{msg}");
drop(first);
assert!(!dir.join(OWNER).exists(), "the owner note goes with the lock");
assert!(dir.join(LOCK).exists(), "the lock file itself may stay: only a held lock refuses");
let again = LibraryLock::acquire(&dir, "lightcraft-cli").unwrap();
assert!(again.held());
drop(again);
let _ = std::fs::remove_dir_all(&dir);
}
/// Files left behind by a crash (lock file + owner note) don't lock anyone out.
#[test]
fn leftover_lock_files_are_not_a_lock() {
let dir = temp_dir("stale");
std::fs::write(dir.join(LOCK), b"").unwrap();
std::fs::write(dir.join(OWNER), br#"{"pid":999999,"host":"elsewhere","program":"LightCraft","version":"0.2.0","since":1}"#).unwrap();
let l = LibraryLock::acquire(&dir, "LightCraft").unwrap();
assert!(l.held());
let owner: LockOwner = serde_json::from_slice(&std::fs::read(dir.join(OWNER)).unwrap()).unwrap();
assert_eq!(owner.pid, std::process::id(), "the note now names this process");
drop(l);
let _ = std::fs::remove_dir_all(&dir);
}
/// Another process holds the lock; when it is killed (a crash), the OS releases it.
#[test]
fn lock_held_by_another_process_is_released_when_it_dies() {
let dir = temp_dir("child");
let exe = std::env::current_exe().unwrap();
let mut child = std::process::Command::new(exe)
.args(["--exact", "tests_lock::lock_holder_child", "--ignored", "--test-threads=1"])
.env("LC_LOCK_CHILD_DIR", &dir)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.unwrap();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60);
while !dir.join("child-locked").exists() {
assert!(child.try_wait().unwrap().is_none(), "child exited before locking");
assert!(std::time::Instant::now() < deadline, "child never locked");
std::thread::sleep(std::time::Duration::from_millis(20));
}
let e = LibraryLock::acquire(&dir, "LightCraft").unwrap_err();
let LockError::InUse(Some(who)) = &e else { panic!("{e:?}") };
assert_eq!(who.pid, child.id());
assert!(e.to_string().contains("on this computer") || who.host.is_empty(), "{e}");
child.kill().unwrap(); // SIGKILL: no clean-up runs, like a crash
child.wait().unwrap();
assert!(dir.join(OWNER).exists(), "the crashed holder's note is left behind");
let l = LibraryLock::acquire(&dir, "LightCraft").expect("the OS released the dead process's lock");
assert!(l.held());
drop(l);
let _ = std::fs::remove_dir_all(&dir);
}
/// Run by [`lock_held_by_another_process_is_released_when_it_dies`] in a child process: take the
/// lock, say so, and wait to be killed.
#[test]
#[ignore = "helper process for lock_held_by_another_process_is_released_when_it_dies"]
fn lock_holder_child() {
let Some(dir) = std::env::var_os("LC_LOCK_CHILD_DIR") else { return };
let _l = LibraryLock::acquire(std::path::Path::new(&dir), "lightcraft-cli").unwrap();
std::fs::write(std::path::Path::new(&dir).join("child-locked"), b"").unwrap();
std::thread::sleep(std::time::Duration::from_secs(120));
}
+223
View File
@@ -0,0 +1,223 @@
//! Issue #101: an append that fails part-way (disk full, a dropped network share) and is then
//! retried must not read back as log damage, and must lose nothing.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use crate::journal::LOG;
use crate::*;
/// A store whose next append writes only a prefix of its data, then fails.
#[derive(Clone, Default)]
struct Torn {
files: MemStore,
/// The next append writes this many bytes, then returns an error.
torn_at: Arc<Mutex<Option<usize>>>,
/// `truncate` fails (the cut-back after a failed append can't be done right away).
truncate_fails: Arc<AtomicBool>,
}
impl Torn {
fn tear_next_append(&self, bytes: usize) {
*self.torn_at.lock().unwrap() = Some(bytes);
}
}
impl Store for Torn {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if let Some(n) = self.torn_at.lock().unwrap().take() {
self.files.append(name, &data[..n.min(data.len())])?;
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
if self.truncate_fails.load(Ordering::SeqCst) {
return Err(std::io::Error::other("share unavailable"));
}
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"torn".into()
}
}
fn add_photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// Apply `ops` to the live catalog (like a command) and return them for the journal.
fn apply(c: &mut Catalog, ops: Vec<Op>) -> Vec<Op> {
for op in &ops {
c.apply(op.clone()).unwrap();
}
ops
}
/// Every tear position of a batch, then the retry (same ops, same seqs, like the engine's queue)
/// and more appends: reloading replays everything, reports no damage and no torn bytes.
#[test]
fn partial_append_then_retry_loads_cleanly() {
let probe = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(probe.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
let before = probe.get(LOG).unwrap().len();
j.append(&batch).unwrap();
let batch_len = probe.get(LOG).unwrap().len() - before;
for tear in 1..batch_len {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
t.tear_next_append(tear);
assert!(j.append(&batch).is_err(), "tear at {tear}");
// cut back to the last whole record
assert_eq!(t.files.get(LOG).unwrap().len(), good_len, "tear at {tear}");
j.append(&batch).unwrap();
let more = apply(&mut c, vec![Op::SetLabel { id, label: Some(ColorLabel::ALL[0]) }]);
j.append(&more).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!((r.torn_bytes, r.torn_fragments, r.replayed), (0, 0, 5), "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
}
/// When cutting back fails too (the share is gone), the next append cuts back first; until that
/// works nothing more is appended after the fragment.
#[test]
fn failed_cut_back_is_done_before_the_next_append() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetRating { id, rating: 5 }]);
t.truncate_fails.store(true, Ordering::SeqCst);
t.tear_next_append(30);
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30, "the fragment is still there");
// still unreachable: the retry fails without writing anything after the fragment
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30);
t.truncate_fails.store(false, Ordering::SeqCst);
j.append(&batch).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "{r:?}");
assert_eq!(r.replayed, 3);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// Logs written before the fix: the fragment of the failed append is glued to the first retried
/// record on one line, followed by the rest of the retry. They load in full (the fragment is
/// skipped), instead of stopping at the fragment and setting the later records aside as damage.
#[test]
fn legacy_fragment_glued_to_the_retry_is_skipped() {
let m = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good = m.get(LOG).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(
&mut c,
vec![
Op::SetRating { id, rating: 3 },
Op::SetFlag { id, flag: Flag::Pick },
Op::SetLabelName { label: ColorLabel::ALL[1], name: Some("Größe ✓".into()) },
],
);
j.append(&batch).unwrap();
let full = m.get(LOG).unwrap();
let batch_bytes = &full[good.len()..];
let later = apply(&mut c, vec![Op::SetRating { id, rating: 1 }]);
j.append(&later).unwrap();
let later_bytes = m.get(LOG).unwrap()[full.len()..].to_vec();
drop(j);
// every tear (inside the first record, after it, inside a multi-byte character…)
for tear in 1..batch_bytes.len() {
if batch_bytes[tear - 1] == b'\n' {
continue; // a tear on a record boundary leaves no fragment
}
let mut log = good.clone();
log.extend_from_slice(&batch_bytes[..tear]);
log.extend_from_slice(batch_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, loaded, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!(r.torn_fragments, 1, "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
// a damaged record whose tail happens to be a whole record from the future is still damage
let mut log = good.clone();
log.extend_from_slice(b"garbage");
log.extend_from_slice(&later_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, _, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_some(), "{r:?}");
}
/// A snapshot can absorb ops whose append failed: it records them as saved (`seq` moves past
/// them), and appending after it continues from there.
#[test]
fn snapshot_with_unlogged_ops_saves_them() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let queued = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetFlag { id, flag: Flag::Reject }]);
t.tear_next_append(10);
assert!(j.append(&queued).is_err());
j.snapshot_with_unlogged(&c, queued.len() as u64).unwrap();
assert_eq!(j.seq(), 3);
let after = apply(&mut c, vec![Op::SetRating { id, rating: 5 }]);
j.append(&after).unwrap();
drop(j);
let (j, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert_eq!((r.snapshot_seq, r.replayed), (3, 1));
assert_eq!(j.seq(), 4);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// `FsStore::truncate` on a file that was never created (an append that failed first) is a no-op.
#[test]
fn fs_store_truncates_a_missing_file_quietly() {
let dir = std::env::temp_dir().join(format!("lc-torn-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
let mut s = FsStore::open(&dir).unwrap();
s.truncate(LOG, 0).unwrap();
assert!(!dir.join(LOG).exists());
s.append(LOG, b"abc\n").unwrap();
s.truncate(LOG, 0).unwrap();
assert_eq!(std::fs::read(dir.join(LOG)).unwrap(), b"");
let _ = std::fs::remove_dir_all(&dir);
}
@@ -0,0 +1,69 @@
//! Evaluate raw starting looks against each file's own embedded JPEG (docs/camera-preview-colour.md):
//!
//! - `camera_look_eval preview RAW OUT.jpg [EDGE]` writes the raw's embedded camera JPEG, oriented,
//! as sRGB (≤ EDGE px, default 1200), for side-by-side viewing with a render of the raw;
//! - `camera_look_eval delta REFERENCE.jpg RENDER.jpg… [--edge N]` prints, for each render, the
//! mean CIE76 ΔE and mean L* against the reference, both reduced to ≤ N px (default 200).
//!
//! Renders come from `lightcraft-cli render RAW -o OUT.jpg --size 1200`; `LIGHTCRAFT_PROFILE=1`
//! makes the render print the fit's alignment, held-out error and verdict.
use lightcraft_codecs::{DecodeOptions, decode};
use lightcraft_raster::resample::{Filter, fit};
fn lab(rgb2020: [f32; 3]) -> [f64; 3] {
// linear Rec.2020 → XYZ (D65) → CIELAB (D65 white)
let m = [[0.636958, 0.144617, 0.168881], [0.262700, 0.677998, 0.059302], [0.0, 0.028073, 1.060985]];
let p = rgb2020.map(|v| f64::from(v).max(0.0));
let xyz: [f64; 3] = std::array::from_fn(|i| m[i][0] * p[0] + m[i][1] * p[1] + m[i][2] * p[2]);
let white = [0.95047, 1.0, 1.08883];
let f = |t: f64| if t > 216.0 / 24389.0 { t.cbrt() } else { (24389.0 / 27.0 * t + 16.0) / 116.0 };
let [fx, fy, fz] = std::array::from_fn(|i| f(xyz[i] / white[i]));
[116.0 * fy - 16.0, 500.0 * (fx - fy), 200.0 * (fy - fz)]
}
fn load(path: &str, edge: usize) -> Result<lightcraft_raster::Rgb32f, String> {
let bytes = std::fs::read(path).map_err(|e| format!("{path}: {e}"))?;
let d = decode(&bytes, DecodeOptions::fit(edge as u32 * 2, edge as u32 * 2)).map_err(|e| format!("{path}: {e}"))?;
Ok(fit(&d.to_working(), edge, edge, Filter::Box))
}
fn main() -> Result<(), String> {
let args: Vec<String> = std::env::args().skip(1).collect();
match args.first().map(String::as_str) {
Some("preview") if args.len() >= 3 => {
let bytes = std::fs::read(&args[1]).map_err(|e| e.to_string())?;
let edge = args.get(3).and_then(|e| e.parse().ok()).unwrap_or(1200);
let img = lightcraft_engine::files::embedded_preview_srgb(&bytes, edge).ok_or("no embedded preview")?;
let jpeg = lightcraft_preview::encode_jpeg(&img).ok_or("can't encode")?;
std::fs::write(&args[2], jpeg).map_err(|e| e.to_string())
}
Some("delta") if args.len() >= 3 => {
let mut edge = 200;
let mut files = Vec::new();
let mut it = args[1..].iter();
while let Some(a) = it.next() {
if a == "--edge" {
edge = it.next().and_then(|v| v.parse().ok()).ok_or("--edge N")?;
} else {
files.push(a.clone());
}
}
let reference = load(&files[0], edge)?;
for path in &files[1..] {
let render = load(path, edge)?;
let render = lightcraft_raster::resample::resize(&render, reference.width, reference.height, Filter::Box);
let (mut de, mut l_ref, mut l_out) = (0.0, 0.0, 0.0);
for (a, b) in reference.data.iter().zip(&render.data) {
let (a, b) = (lab(*a), lab(*b));
de += ((a[0] - b[0]).powi(2) + (a[1] - b[1]).powi(2) + (a[2] - b[2]).powi(2)).sqrt();
l_ref += a[0];
l_out += b[0];
}
let n = reference.data.len().max(1) as f64;
println!("{path}\tdE76 {:.2}\tL* {:.1} (reference {:.1})", de / n, l_out / n, l_ref / n);
}
Ok(())
}
_ => Err("usage: camera_look_eval preview RAW OUT.jpg [EDGE] | delta REFERENCE.jpg RENDER.jpg… [--edge N]".into()),
}
}
+215
View File
@@ -0,0 +1,215 @@
//! Is a file there? — answered without blocking the caller.
//!
//! On a sleeping NAS, a dropped share or a USB drive spinning up, one `stat` can take as long as
//! the OS or SMB timeout. The UI asks about files every frame (grid cells, the Info panel, the
//! Missing Photos view), so in the app the answers come from a cache that a worker thread fills
//! ([`Availability::run_in_background`]): an unknown file is reported as `None` ("assume it is
//! there"; the render worker finds out if it isn't), known answers are rechecked after
//! [`RECHECK`]. Everywhere else (CLI, MCP, tests) the default is synchronous: every question is a
//! fresh check, as before.
use std::collections::{HashMap, HashSet, VecDeque};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use std::time::Duration;
/// Checks whether a path exists (injectable: tests use a slow one).
pub type ExistsProbe = Arc<dyn Fn(&str) -> bool + Send + Sync>;
/// Called (from the worker) when an answer changed, e.g. to repaint the UI.
pub type Waker = Arc<dyn Fn() + Send + Sync>;
/// How long a cached answer is used before it is checked again (it is still returned meanwhile).
pub const RECHECK: Duration = Duration::from_secs(15);
fn fs_exists(path: &str) -> bool {
// the browser build has no file system: its originals are always "there"
cfg!(target_arch = "wasm32") || std::path::Path::new(path).exists()
}
#[derive(Default)]
struct State {
/// path → (exists, when checked)
known: HashMap<String, (bool, std::time::Instant)>,
queue: VecDeque<String>,
queued: HashSet<String>,
worker: bool,
/// Bumped whenever an answer changes (callers recompute what depends on it).
generation: u64,
}
fn lock(m: &Mutex<State>) -> MutexGuard<'_, State> {
m.lock().unwrap_or_else(PoisonError::into_inner)
}
/// File availability, synchronous or cached + background-checked. Cloning shares the cache.
#[derive(Clone)]
pub struct Availability {
probe: ExistsProbe,
background: Option<Waker>,
state: Arc<Mutex<State>>,
}
impl Default for Availability {
fn default() -> Self {
Self { probe: Arc::new(fs_exists), background: None, state: Arc::default() }
}
}
impl std::fmt::Debug for Availability {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Availability").field("background", &self.background.is_some()).finish()
}
}
impl Availability {
/// Replace the existence check (tests: a slow or counting probe). Forgets cached answers.
pub fn set_probe(&mut self, probe: ExistsProbe) {
self.probe = probe;
let mut st = lock(&self.state);
st.known.clear();
st.generation += 1;
}
/// Answer from the cache and check on a worker thread from now on; `wake` runs when an
/// answer changed. No effect in the browser build (no threads, no file system).
pub fn run_in_background(&mut self, wake: Waker) {
if cfg!(target_arch = "wasm32") {
return;
}
self.background = Some(wake);
}
/// The existence check itself (blocking: for worker threads, e.g. the Rename preview's).
pub fn probe(&self) -> ExistsProbe {
self.probe.clone()
}
pub fn is_background(&self) -> bool {
self.background.is_some()
}
/// Does `path` exist? Synchronous mode: checked now. Background mode: the cached answer
/// (`None` while unknown); unknown or stale answers are queued for the worker.
pub fn exists(&self, path: &str) -> Option<bool> {
let Some(wake) = &self.background else {
return Some((self.probe)(path));
};
let mut st = lock(&self.state);
let now = std::time::Instant::now();
let cached = st.known.get(path).copied();
if cached.is_none_or(|(_, at)| now.saturating_duration_since(at) >= RECHECK) && !st.queued.contains(path) {
st.queued.insert(path.to_string());
st.queue.push_back(path.to_string());
if !st.worker {
st.worker = true;
drop(st);
self.spawn_worker(wake.clone());
return cached.map(|c| c.0);
}
}
cached.map(|c| c.0)
}
/// Known to be missing (unknown counts as present).
pub fn is_offline(&self, path: &str) -> bool {
self.exists(path) == Some(false)
}
/// Changes whenever a cached answer changes.
pub fn generation(&self) -> u64 {
lock(&self.state).generation
}
/// Paths waiting for the worker.
pub fn pending(&self) -> usize {
lock(&self.state).queue.len()
}
fn spawn_worker(&self, wake: Waker) {
let (state, probe) = (self.state.clone(), self.probe.clone());
let started = std::thread::Builder::new().name("lc-availability".into()).spawn(move || {
loop {
let path = {
let mut st = lock(&state);
let Some(p) = st.queue.pop_front() else {
st.worker = false;
return;
};
p
};
let ok = probe(&path);
let changed = {
let mut st = lock(&state);
st.queued.remove(&path);
let prev = st.known.insert(path, (ok, std::time::Instant::now()));
let changed = prev.map(|p| p.0) != Some(ok);
if changed {
st.generation += 1;
}
changed
};
if changed {
wake();
}
}
});
if started.is_err() {
// try again on the next question
lock(&self.state).worker = false;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering};
#[test]
fn synchronous_by_default() {
let mut a = Availability::default();
let n = Arc::new(AtomicUsize::new(0));
let c = n.clone();
a.set_probe(Arc::new(move |p| {
c.fetch_add(1, Ordering::SeqCst);
p == "here"
}));
assert_eq!(a.exists("here"), Some(true));
assert_eq!(a.exists("gone"), Some(false));
assert_eq!(a.exists("gone"), Some(false));
assert_eq!(n.load(Ordering::SeqCst), 3, "every question is a fresh check");
}
#[test]
fn background_answers_from_the_cache_without_blocking() {
let mut a = Availability::default();
let caller = std::thread::current().id();
let on_caller = Arc::new(AtomicUsize::new(0));
let c = on_caller.clone();
a.set_probe(Arc::new(move |p| {
if std::thread::current().id() == caller {
c.fetch_add(1, Ordering::SeqCst);
}
std::thread::sleep(Duration::from_millis(200)); // a sleeping NAS
p == "here"
}));
let woke = Arc::new(AtomicUsize::new(0));
let w = woke.clone();
a.run_in_background(Arc::new(move || {
w.fetch_add(1, Ordering::SeqCst);
}));
let t0 = std::time::Instant::now();
assert_eq!(a.exists("here"), None, "unknown at first");
assert_eq!(a.exists("gone"), None);
assert!(t0.elapsed() < Duration::from_millis(150), "the caller never waits for the probe");
let deadline = std::time::Instant::now() + Duration::from_secs(20);
while (a.exists("here").is_none() || a.exists("gone").is_none()) && std::time::Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(10));
}
assert_eq!(a.exists("here"), Some(true));
assert_eq!(a.exists("gone"), Some(false));
assert!(a.is_offline("gone") && !a.is_offline("here") && !a.is_offline("unknown"));
assert_eq!(on_caller.load(Ordering::SeqCst), 0, "never probed on the calling thread");
assert!(woke.load(Ordering::SeqCst) >= 2);
assert!(a.generation() >= 2);
}
}
+423
View File
@@ -0,0 +1,423 @@
//! Estimate a raw's starting look from its own embedded JPEG, for raws whose decoder has no colour
//! matrix (`matrix_is_fallback`: every non-DNG raw today, whatever the make). Colour and luminance
//! are fitted separately; the JPEG supplies correspondences only, never output pixels or a
//! replacement for raw editing. Each file's result (also "no usable fit") is cached by a content
//! fingerprint and [`FIT_VERSION`], so thumbnails, previews and exports of a file fit it once.
use std::sync::Mutex;
use lightcraft_color::{Mat3, luminance_2020};
use lightcraft_pipeline::tone::{CameraTone, ToneMap};
use lightcraft_preview::{Hash128, Hasher128, Lru};
use lightcraft_raster::{
Rgb32f,
resample::{Filter, fit},
};
use lightcraft_raw::{RawImage, color::CameraTransform};
#[derive(Clone, Copy, Debug, PartialEq)]
pub(crate) struct CameraLook {
pub matrix: Mat3,
pub tone: CameraTone,
}
/// Bump whenever the fit or its gates change what a file gets: cached results of older versions
/// are then never used (the version is part of the cache key).
pub(crate) const FIT_VERSION: u64 = 2;
/// Results kept: one per recently decoded file (a few hundred bytes each).
const CACHE_ENTRIES: usize = 512;
static CACHE: Mutex<Option<Lru<Hash128, Option<CameraLook>>>> = Mutex::new(None);
/// The file's look, from the cache or fitted now.
pub(crate) fn fit_preview(raw: &RawImage, bytes: &[u8], transform: &CameraTransform) -> Option<CameraLook> {
if !transform.matrix_is_fallback {
return None;
}
let jpeg = lightcraft_raw::embedded_preview(bytes);
let key = cache_key(raw, bytes, jpeg.as_deref(), transform);
if let Some(hit) = CACHE.lock().ok().and_then(|mut c| c.as_mut().and_then(|c| c.get(&key).copied())) {
return hit;
}
let t0 = std::time::Instant::now();
let look = jpeg.and_then(|jpeg| fit_file(raw, &jpeg, transform));
if lightcraft_pipeline::profiling() {
eprintln!("[profile] camera look ({:?}) fitted in {:.1} ms: {}", raw.format, t0.elapsed().as_secs_f64() * 1e3, if look.is_some() { "accepted" } else { "rejected" });
}
if let Ok(mut c) = CACHE.lock() {
c.get_or_insert_with(|| Lru::new(CACHE_ENTRIES)).insert(key, look, 1);
}
look
}
/// A fingerprint of everything the fit reads: the file's length, its whole embedded JPEG, a
/// strided sample of the file (the mosaic), the decoder's crop and colour transform, and
/// [`FIT_VERSION`]. Hashing all ~25 MB of a raw would cost a third of the fit it saves.
fn cache_key(raw: &RawImage, bytes: &[u8], jpeg: Option<&[u8]>, t: &CameraTransform) -> Hash128 {
let mut h = Hasher128::new();
h.u64(FIT_VERSION).u64(bytes.len() as u64).u64(jpeg.map_or(u64::MAX, |j| j.len() as u64)).update(jpeg.unwrap_or_default());
let step = (bytes.len() / 4096).max(64);
for chunk in bytes.chunks(step) {
h.update(chunk.get(..32).unwrap_or(chunk));
}
let c = raw.crop;
for v in [c.x, c.y, c.width, c.height, raw.active_area.width, raw.active_area.height] {
h.u64(v as u64);
}
for v in t.matrix.0.iter().flatten().copied().chain(t.wb.map(f64::from)).chain([t.baseline_exposure]) {
h.u64(v.to_bits());
}
h.finish()
}
fn reject<T>(why: &str) -> Option<T> {
if lightcraft_pipeline::profiling() {
eprintln!("[profile] camera look rejected: {why}");
}
None
}
fn fit_file(raw: &RawImage, jpeg: &[u8], transform: &CameraTransform) -> Option<CameraLook> {
let Ok(decoded) = lightcraft_codecs::decode(jpeg, lightcraft_codecs::DecodeOptions { max_size: Some((384, 384)), max_pixels: 64_000_000 }) else {
return reject("embedded JPEG does not decode");
};
let reference = decoded.to_working();
let crop = raw.crop.clipped(raw.active_area.width, raw.active_area.height);
if crop.width == 0 || crop.height == 0 || reference.width == 0 || reference.height == 0 {
return reject("empty crop or preview");
}
// Previews are stored in sensor orientation (EXIF orientation applies to both alike). A
// preview of a different shape — a 16:9 or 1:1 picture-style crop, a rotated thumbnail —
// can't be paired pixel for pixel.
let aspect = crop.width as f64 / crop.height as f64;
if (reference.width as f64 / reference.height as f64 / aspect - 1.0).abs() > MAX_ASPECT_MISMATCH {
return reject("preview aspect differs from the raw crop");
}
// Fixed, bounded proxy: the selected look cannot depend on thumbnail/export resolution.
let k = (crop.width.max(crop.height).div_ceil(384).max(2)).div_ceil(2) * 2;
let Some(sensor) = raw.develop_binned(k, 0.99).ok().flatten() else {
return reject("mosaic can't be binned");
};
let mut sensor = fit(&sensor, 96, 96, Filter::Box);
// Same shape within MAX_ASPECT_MISMATCH: pair the pixels exactly (a fit could round one edge
// differently, e.g. a 3:2 preview of a 4950×3280 crop).
let reference = lightcraft_raster::resample::resize(&reference, sensor.width, sensor.height, Filter::Box);
let gain = 2f32.powf(transform.baseline_exposure as f32);
sensor.map_in_place(|p| transform.matrix.apply_f32(std::array::from_fn(|i| p[i] * transform.wb[i] * gain)));
// Same aspect is not the same framing: a preview cropped or shifted differently (some bodies'
// previews leave out sensor borders the raw crop keeps) would pair unrelated pixels.
let alignment = alignment(&sensor, &reference)?;
if lightcraft_pipeline::profiling() {
eprintln!("[profile] camera look alignment: correlation {:.3} at (0, 0), best {:.3} at {:?}", alignment.at_zero, alignment.best, alignment.shift);
}
if alignment.at_zero < MIN_ALIGNMENT || alignment.shift.0.abs() > 1 || alignment.shift.1.abs() > 1 {
return reject("preview framing doesn't match the raw");
}
let look = fit_pairs(&sensor, &reference)?;
if lightcraft_pipeline::profiling() {
eprintln!("[profile] camera look: {:?}, {:?}", look.matrix.0, look.tone);
}
Some(look)
}
/// Previews whose width/height ratio differs from the raw crop's by more than this are not used.
const MAX_ASPECT_MISMATCH: f64 = 0.02;
/// Correlation of log luminance (sensor proxy vs preview, both ~96 px) the unshifted pairing must
/// reach; the best of the ±3 px shifts must also be within ±1 px of it.
const MIN_ALIGNMENT: f64 = 0.85;
struct Alignment {
at_zero: f64,
best: f64,
shift: (i32, i32),
}
/// How well the two proxies line up: Pearson correlation of log luminance for whole-pixel shifts
/// of up to ±3 px (the camera's tone curve is monotone, so aligned images correlate strongly).
fn alignment(sensor: &Rgb32f, reference: &Rgb32f) -> Option<Alignment> {
let (w, h) = (sensor.width, sensor.height);
if (w, h) != (reference.width, reference.height) || w < 16 || h < 16 || sensor.data.len() != w * h || reference.data.len() != w * h {
return reject("proxies differ in size");
}
// Log luminance, box-blurred (5×5): fine texture (foliage at ~96 px) and the small geometric
// differences of in-camera distortion correction must not read as misalignment; a crop,
// shift or rotation of the framing still does.
let log = |img: &Rgb32f| -> Vec<f64> {
let l: Vec<f64> = img.data.iter().map(|p| f64::from(luminance_2020(p.map(|v| if v.is_finite() { v.max(0.0) } else { 0.0 }))).max(1e-4).ln()).collect();
(0..w * h)
.map(|i| {
let (x, y) = ((i % w) as i64, (i / w) as i64);
let (mut sum, mut n) = (0.0, 0.0);
for yy in (y - 2).max(0)..(y + 3).min(h as i64) {
for xx in (x - 2).max(0)..(x + 3).min(w as i64) {
if let Some(v) = l.get(yy as usize * w + xx as usize) {
sum += v;
n += 1.0;
}
}
}
sum / f64::max(n, 1.0)
})
.collect()
};
let (a, b) = (log(sensor), log(reference));
const M: i32 = 3;
let corr = |dx: i32, dy: i32| -> f64 {
let (mut n, mut sa, mut sb, mut saa, mut sbb, mut sab) = (0.0, 0.0, 0.0, 0.0, 0.0, 0.0);
for y in M..h as i32 - M {
for x in M..w as i32 - M {
let (Some(&u), Some(&v)) = (a.get((y * w as i32 + x) as usize), b.get(((y + dy) * w as i32 + x + dx) as usize)) else {
continue;
};
n += 1.0;
sa += u;
sb += v;
saa += u * u;
sbb += v * v;
sab += u * v;
}
}
let cov = sab - sa * sb / n;
let var = (saa - sa * sa / n) * (sbb - sb * sb / n);
if n < 2.0 || var <= 1e-12 { 0.0 } else { cov / var.sqrt() }
};
let at_zero = corr(0, 0);
let mut best = (at_zero, (0, 0));
for dy in -M..=M {
for dx in -M..=M {
let c = corr(dx, dy);
if c > best.0 + 1e-9 {
best = (c, (dx, dy));
}
}
}
Some(Alignment { at_zero, best: best.0, shift: best.1 })
}
/// A fit must cut the held-out squared error to below this share of the fallback's.
const MIN_IMPROVEMENT: f64 = 0.7;
/// ...and stay within this per-channel RMS of the camera JPEG (linear display values). On public
/// raw.pixls.us samples (eight Sony bodies) good fits that still beat the fallback 1.5–3× landed
/// at 0.065–0.093 (camera local tone, vignetting and lens processing that a global matrix + curve
/// cannot follow): visibly better renders that 0.055 rejected.
const MAX_HOLDOUT_RMS: f64 = 0.10;
fn luma(p: [f64; 3]) -> f64 {
p[0] * 0.2627 + p[1] * 0.6780 + p[2] * 0.0593
}
fn displayed(scene: [f64; 3], tone: &ToneMap) -> [f64; 3] {
let scene = scene.map(|v| v.max(0.0));
let y = luma(scene);
if y <= 0.0 {
return [0.0; 3];
}
scene.map(|v| v * f64::from(tone.apply(y as f32)) / y)
}
fn fit_pairs(sensor: &Rgb32f, reference: &Rgb32f) -> Option<CameraLook> {
if (sensor.width, sensor.height) != (reference.width, reference.height) || sensor.data.len() != reference.data.len() {
return None;
}
let mut pairs = Vec::new();
let mut colour = 0;
for (input, output) in sensor.data.iter().zip(&reference.data) {
let y = luminance_2020(*output);
if !input.iter().all(|v| v.is_finite() && *v > 0.001 && *v < 1.5)
|| !output.iter().all(|v| v.is_finite() && *v > 0.004 && *v < 0.98)
|| !(0.015..0.85).contains(&y)
{
continue;
}
let min = output.iter().copied().fold(f32::INFINITY, f32::min);
let max = output.iter().copied().fold(0.0, f32::max);
colour += usize::from(max - min > 0.05);
pairs.push((input.map(f64::from), output.map(f64::from)));
}
if pairs.len() < 256 || colour < pairs.len() / 20 {
return reject("too few usable or coloured pixels");
}
let mut gram = [[0.0; 3]; 3];
let mut cross = [[0.0; 3]; 3];
for (i, (x, y)) in pairs.iter().enumerate() {
if i % 3 == 0 {
continue;
}
let (lx, ly) = (luma(*x), luma(*y));
for row in 0..3 {
for col in 0..3 {
// Normalising by luminance prevents a camera S-curve from corrupting colour.
gram[row][col] += x[row] * x[col] / (lx * lx);
cross[row][col] += y[row] * x[col] / (ly * lx);
}
}
}
let trace: f64 = (0..3).map(|i| gram[i][i]).sum();
let inverse = Mat3(gram).inverse()?;
let condition = trace * (0..3).map(|i| inverse.0[i][i].abs()).sum::<f64>();
if trace <= 0.0 || !condition.is_finite() || condition > 1e6 {
return reject("ill-conditioned colour samples");
}
let regularization = trace * 1e-4;
for i in 0..3 {
gram[i][i] += regularization;
cross[i][i] += regularization;
}
let matrix = Mat3(cross).mul(&Mat3(gram).inverse()?);
if !matrix.0.iter().flatten().all(|v| v.is_finite() && v.abs() < 8.0) {
return reject("unbounded colour matrix");
}
let tone_pairs: Vec<_> =
pairs.iter().enumerate().filter(|(i, _)| i % 3 != 0).map(|(_, (x, y))| (luma(matrix.apply(*x).map(|v| v.max(0.0))), luma(*y))).collect();
let curve = fit_tone(tone_pairs)?;
let tone = ToneMap::camera(&curve, 0.0, 0.0, 0.0);
let original_tone = ToneMap::new(0.0, 0.0, 0.0);
let mut before = 0.0;
let mut after = 0.0;
let mut samples = 0;
for (i, (x, target)) in pairs.iter().enumerate() {
if i % 3 != 0 {
continue;
}
let corrected = displayed(matrix.apply(*x), &tone);
let original = displayed(*x, &original_tone);
for c in 0..3 {
before += (original[c] - target[c]).powi(2);
after += (corrected[c] - target[c]).powi(2);
samples += 1;
}
}
if lightcraft_pipeline::profiling() {
eprintln!("[profile] camera look holdout RMS {:.5} -> {:.5} ({samples} channels)", (before / samples as f64).sqrt(), (after / samples as f64).sqrt());
}
if !after.is_finite() || samples == 0 || after >= before * MIN_IMPROVEMENT || after / samples as f64 > MAX_HOLDOUT_RMS.powi(2) {
return reject("held-out error gate");
}
Some(CameraLook { matrix, tone: curve })
}
fn median(values: &mut [f64]) -> Option<f64> {
values.sort_by(f64::total_cmp);
values.get(values.len() / 2).copied()
}
fn fit_tone(mut pairs: Vec<(f64, f64)>) -> Option<CameraTone> {
if pairs.len() < 128 || !pairs.iter().all(|(x, y)| x.is_finite() && *x > 0.0 && y.is_finite()) {
return None;
}
pairs.sort_by(|a, b| a.0.total_cmp(&b.0));
let mut knots = [[0.0; 2]; 32];
for (i, knot) in knots.iter_mut().enumerate() {
let bin = pairs.get(i * pairs.len() / 32..(i + 1) * pairs.len() / 32)?;
let mut xs: Vec<_> = bin.iter().map(|p| p.0).collect();
let mut ys: Vec<_> = bin.iter().map(|p| p.1).collect();
*knot = [median(&mut xs)? as f32, median(&mut ys)? as f32];
}
if knots[31][0] < knots[0][0] * 1.5 {
return None;
}
// Pool adjacent violating bins (isotonic regression): no reversals or arbitrary polynomial.
let mut blocks: Vec<(f32, usize)> = Vec::new();
for knot in knots {
blocks.push((knot[1], 1));
while blocks.len() >= 2 {
let (a, an) = *blocks.get(blocks.len() - 2)?;
let (b, bn) = *blocks.last()?;
if a <= b {
break;
}
blocks.truncate(blocks.len() - 2);
blocks.push(((a * an as f32 + b * bn as f32) / (an + bn) as f32, an + bn));
}
}
let mut i = 0;
for (y, n) in blocks {
for knot in knots.get_mut(i..i + n)? {
knot[1] = y;
}
i += n;
}
CameraTone::new(knots)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn separates_nonlinear_tone_from_colour_and_keeps_sensor_headroom() {
let known = Mat3([[1.8, -0.4, -0.1], [-0.2, 1.5, -0.1], [-0.05, -0.3, 1.7]]);
let mut sensor = Rgb32f::new(64, 64);
let mut reference = sensor.clone();
for (i, (src, dst)) in sensor.data.iter_mut().zip(&mut reference.data).enumerate() {
let ev = 0.05 + (i % 31) as f32 * 0.017;
*src = [ev * (0.8 + (i % 11) as f32 * 0.025), ev, ev * (0.8 + (i % 17) as f32 * 0.014)];
let p = known.apply_f32(*src);
let y = luminance_2020(p);
*dst = p.map(|v| v * (1.0 - (-2.5 * y).exp()) / y);
}
let original = sensor.clone();
let fit = fit_pairs(&sensor, &reference).unwrap();
assert_eq!(sensor.data, original.data);
let tone = ToneMap::camera(&fit.tone, 0.0, 0.0, 0.0);
let error: f64 = sensor
.data
.iter()
.zip(&reference.data)
.map(|(x, y)| {
let p = displayed(fit.matrix.apply(x.map(f64::from)), &tone);
(0..3).map(|c| (p[c] - f64::from(y[c])).powi(2)).sum::<f64>() / 3.0
})
.sum::<f64>()
/ sensor.data.len() as f64;
assert!(error.sqrt() < 0.025, "{error}");
// The colour transform is homogeneous; tone mapping happens only after exposure.
let p = fit.matrix.apply([2.0, 2.0, 2.0]);
assert!(luma(p) > 1.0);
assert!(tone.apply(0.2) < tone.apply(0.4));
}
#[test]
fn accepts_a_much_better_fit_despite_local_camera_processing() {
// The camera JPEG departs from any global matrix + curve (local tone, vignetting): ±0.12
// per-pixel deviations, ~0.07 RMS. The fit is still far closer than the fallback.
let known = Mat3([[1.8, -0.4, -0.1], [-0.2, 1.5, -0.1], [-0.05, -0.3, 1.7]]);
let mut sensor = Rgb32f::new(64, 64);
let mut reference = sensor.clone();
let mut seed = 0x2545_f491_u32;
for (i, (src, dst)) in sensor.data.iter_mut().zip(&mut reference.data).enumerate() {
let ev = 0.05 + (i % 31) as f32 * 0.017;
*src = [ev * (0.8 + (i % 11) as f32 * 0.025), ev, ev * (0.8 + (i % 17) as f32 * 0.014)];
let p = known.apply_f32(*src);
let y = luminance_2020(p);
*dst = p.map(|v| {
seed ^= seed << 13;
seed ^= seed >> 17;
seed ^= seed << 5;
let noise = (seed % 2001) as f32 / 1000.0 - 1.0;
(v * (1.0 - (-2.5 * y).exp()) / y + 0.12 * noise).clamp(0.005, 0.97)
});
}
assert!(fit_pairs(&sensor, &reference).is_some());
}
#[test]
fn rejects_monochrome_invalid_and_unrelated_previews() {
let mut sensor = Rgb32f::new(32, 32);
let mut reference = sensor.clone();
for (i, p) in sensor.data.iter_mut().enumerate() {
*p = [0.1 + (i % 13) as f32 * 0.02, 0.15, 0.1];
}
reference.data.fill([0.2; 3]);
assert!(fit_pairs(&sensor, &reference).is_none());
reference.data.fill([f32::NAN; 3]);
assert!(fit_pairs(&sensor, &reference).is_none());
for (i, (src, dst)) in sensor.data.iter_mut().zip(&mut reference.data).enumerate() {
*src = [0.04 + (i % 11) as f32 * 0.02, 0.05 + (i % 17) as f32 * 0.01, 0.03 + (i % 23) as f32 * 0.01];
*dst = [0.05 + (i % 7) as f32 * 0.07, 0.05 + (i % 19) as f32 * 0.02, 0.05 + (i % 29) as f32 * 0.01];
}
assert!(fit_pairs(&sensor, &reference).is_none());
sensor.data.fill([0.1, 0.15, 0.12]);
assert!(fit_pairs(&sensor, &reference).is_none());
reference.data.truncate(8);
assert!(fit_pairs(&sensor, &reference).is_none());
}
}
+62 -35
View File
@@ -166,7 +166,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Rename Folder",
[],
None,
"{path, name} — rename a folder on disk (its files and sidecars go along) and relink the photos in it; not an undo step → {path, relinked}",
"{path, name} — rename a folder on disk (its files and sidecars go along) and relink the photos in it; one undo step (undo renames it back, refused if the old name is taken) → {path, relinked}",
always,
|s, p| {
let from = str_param(p, "path").ok_or_else(|| bad("folder.rename", "missing path"))?.trim_end_matches(['/', '\\']).to_string();
@@ -184,7 +184,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Move Folder",
[],
None,
"{path, into: destination folder} — move a folder on disk into another and relink the photos in it; not an undo step → {path, relinked}",
"{path, into: destination folder} — move a folder on disk into another and relink the photos in it; one undo step (undo moves it back, refused if the old place is taken) → {path, relinked}",
always,
|s, p| {
let from = str_param(p, "path").ok_or_else(|| bad("folder.move", "missing path"))?.trim_end_matches(['/', '\\']).to_string();
@@ -217,58 +217,85 @@ pub fn specs() -> Vec<CommandSpec> {
]
}
/// Rename or move a folder on disk (everything in it goes along, sidecars included) and relink
/// the photos inside. Like a rename in the file manager it is not an undo step: undoing would
/// point the photos back at a folder that no longer exists.
pub(crate) fn move_folder(s: &mut crate::Session, from: &str, to: &str) -> std::result::Result<usize, String> {
use std::path::Path;
/// Rename `from` to `to` on disk (a folder; everything in it goes along). Never overwrites:
/// refused when `to` exists (a case-only rename on a case-insensitive file system goes through
/// a temporary name). Missing parents of `to` are created.
pub(crate) fn rename_folder_on_disk(from: &str, to: &str) -> std::result::Result<(), String> {
let (src, dst) = (Path::new(from), Path::new(to));
if !src.is_dir() {
return Err(format!("{from} is not a folder"));
return Err(format!("{from} is not a folder (moved or deleted?)"));
}
if dst.exists() {
let case_only = from != to && from.to_lowercase() == to.to_lowercase();
if dst.exists() && !case_only {
return Err(format!("{to} already exists"));
}
if dst.starts_with(src) {
return Err("a folder can't move into itself".into());
}
if s.library.as_ref().is_some_and(|l| l.dir.starts_with(src) || src.starts_with(&l.dir)) {
return Err("the library's own folders can't be moved here".into());
}
if let Some(parent) = dst.parent() {
std::fs::create_dir_all(parent).map_err(|e| format!("{}: {e}", parent.display()))?;
}
std::fs::rename(src, dst).map_err(|e| format!("{from} → {to}: {e}"))?;
let mut n = 0;
let ops: Vec<lightcraft_catalog::Op> = s
if case_only {
let tmp = src.with_file_name(format!(".lc-rename-{}", std::process::id()));
if tmp.exists() {
return Err(format!("{} already exists", tmp.display()));
}
std::fs::rename(src, &tmp).map_err(|e| format!("{from} → {to}: {e}"))?;
return std::fs::rename(&tmp, dst).map_err(|e| {
let _ = std::fs::rename(&tmp, src);
format!("{from} → {to}: {e}")
});
}
std::fs::rename(src, dst).map_err(|e| format!("{from} → {to}: {e}"))
}
/// After a folder moved from `from` to `to`: a browsed folder at or below it follows.
pub(crate) fn follow_folder(s: &mut Session, from: &str, to: &str) {
if let Some(b) = &mut s.browse
&& let Ok(rel) = Path::new(&b.path).strip_prefix(from)
{
b.path = Path::new(to).join(rel).to_string_lossy().to_string();
let f = b.path.clone();
s.filter.folder = Some(f);
}
}
/// Rename or move a folder on disk (everything in it goes along, sidecars included) and relink
/// the photos inside, as one undo step: undo renames the folder back and restores the photos'
/// paths, redo repeats the move — each refused (and reported) when the destination is taken.
pub(crate) fn move_folder(s: &mut Session, from: &str, to: &str) -> std::result::Result<usize, String> {
let (src, dst) = (Path::new(from), Path::new(to));
if !src.is_dir() {
return Err(format!("{from} is not a folder"));
}
if s.library.as_ref().is_some_and(|l| l.dir.starts_with(src) || src.starts_with(&l.dir)) {
return Err("the library's own folders can't be moved here".into());
}
rename_folder_on_disk(from, to)?;
let ops: Vec<Op> = s
.catalog
.photos()
.filter_map(|p| match &p.source {
lightcraft_catalog::Source::File { path } => Path::new(path).strip_prefix(src).ok().map(|rel| (p.id, p.file_name.clone(), dst.join(rel))),
_ => None,
})
.map(|(id, file_name, np)| {
n += 1;
lightcraft_catalog::Op::Relink {
id,
file_name,
source: lightcraft_catalog::Source::File { path: np.to_string_lossy().to_string() },
format: None,
}
.map(|(id, file_name, np)| Op::Relink {
id,
file_name,
source: lightcraft_catalog::Source::File { path: np.to_string_lossy().to_string() },
format: None,
})
.collect();
for op in ops {
if s.catalog.apply(op.clone()).is_ok() {
s.pending_log.push(op);
}
}
// a browsed folder at or below it follows
if let Some(b) = &mut s.browse
&& let Ok(rel) = Path::new(&b.path).strip_prefix(src)
{
b.path = dst.join(rel).to_string_lossy().to_string();
let f = b.path.clone();
s.filter.folder = Some(f);
let n = ops.len();
let name = Path::new(to).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let renamed = src.parent() == dst.parent();
let label = if renamed { format!("Rename Folder to “{name}”") } else { format!("Move Folder “{name}”") };
let folder = crate::FolderMove { from: to.to_string(), to: from.to_string() };
if let Err(e) = s.commit_with_folder(&label, Op::Batch { ops }, folder) {
// the catalog didn't take it: put the folder back where the photos still point
let back = rename_folder_on_disk(to, from).err().map(|b| format!("; moving it back failed too: {b}")).unwrap_or_default();
return Err(format!("{e}{back}"));
}
follow_folder(s, from, to);
Ok(n)
}
+1 -1
View File
@@ -26,7 +26,7 @@ const PROBE_EDGE: usize = 384;
pub(crate) fn probe(s: &mut Session, c: &str, x: f64, y: f64, neutral: impl FnOnce(&mut DevelopSettings)) -> Result<[f32; 3]> {
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
let src = s.source_now(id, SourceLevel::Thumb).map_err(|e| bad(c, e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let mut d = (*s.develop_of(id).unwrap_or_default()).clone();
neutral(&mut d);
let req = lightcraft_pipeline::RenderRequest::fit(PROBE_EDGE, PROBE_EDGE);
+34 -17
View File
@@ -10,24 +10,28 @@ use serde_json::{Value, json};
use super::{CommandSpec, bad, cmd, has_active, has_selection};
use crate::{Result, Session};
/// A free `<stem>.dng` (then `<stem>-2.dng`…) next to `path`.
fn dng_path(path: &str) -> String {
/// `<stem>.dng`, then `<stem>-2.dng`, `<stem>-3.dng`… next to `path`.
fn dng_names(path: &str) -> impl Iterator<Item = std::path::PathBuf> {
let p = Path::new(path);
let dir = p.parent().unwrap_or(Path::new(""));
let dir = p.parent().unwrap_or(Path::new("")).to_path_buf();
let stem = p.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| "photo".into());
let mut out = dir.join(format!("{stem}.dng"));
let mut n = 2;
while out.exists() {
out = dir.join(format!("{stem}-{n}.dng"));
n += 1;
}
out.to_string_lossy().to_string()
(1u64..).map(move |n| if n == 1 { dir.join(format!("{stem}.dng")) } else { dir.join(format!("{stem}-{n}.dng")) })
}
/// Write the DNG for raw file `path` (develop settings in `packet`); returns the new path.
/// An empty `packet` writes no XMP.
/// Write the DNG for raw file `path` (develop settings in `packet`) next to it under a free name;
/// returns the new path. An empty `packet` writes no XMP.
///
/// The DNG is checked before anyone relies on it (Convert to DNG relinks the photo to it, Copy as
/// DNG deletes the raw copy): the encoded file must decode to the same raw data, and it is
/// written to a temp file, synced, read back identical and only then given its name — never
/// replacing an existing file. On any failure no DNG is left behind and the raw stays in use.
pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::result::Result<String, String> {
let bytes = match &s.media.file_bytes {
write_dng_with(s.media.file_bytes.as_ref(), path, packet)
}
/// [`write_dng_for`] without the session (imports convert on a worker thread).
pub(crate) fn write_dng_with(file_bytes: Option<&crate::merge::ByteReader>, path: &str, packet: String) -> std::result::Result<String, String> {
let bytes = match file_bytes {
Some(r) => r(path)?,
None => std::fs::read(path).map_err(|e| format!("{path}: {e}"))?,
};
@@ -35,9 +39,21 @@ pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::res
drop(bytes);
let dng = lightcraft_raw::write_dng(&raw, &lightcraft_raw::DngWriteOptions { xmp: (!packet.is_empty()).then_some(packet), ..Default::default() })
.map_err(|e| e.to_string())?;
let out = dng_path(path);
std::fs::write(&out, dng).map_err(|e| format!("{out}: {e}"))?;
Ok(out)
let back = lightcraft_raw::decode(&dng).map_err(|e| format!("{path}: the DNG written for it doesn't decode ({e}); the raw is kept"))?;
let same = match (&back.data, &raw.data) {
(lightcraft_raw::RawData::U16(a), lightcraft_raw::RawData::U16(b)) => a == b,
(lightcraft_raw::RawData::F32(a), lightcraft_raw::RawData::F32(b)) => {
a.len() == b.len() && a.iter().zip(b).all(|(x, y)| x.to_bits() == y.to_bits())
}
_ => false,
};
if (back.width, back.height, back.cpp) != (raw.width, raw.height, raw.cpp) || !same {
return Err(format!("{path}: the DNG written for it doesn't hold the same raw data; the raw is kept"));
}
drop((raw, back));
let out = lightcraft_catalog::safe_file::write_new_unique(&mut dng_names(path), &dng)
.map_err(|e| format!("{path}: could not write its DNG ({e}); the raw is kept"))?;
Ok(out.to_string_lossy().to_string())
}
fn convert(s: &mut Session, p: &Value) -> Result<Value> {
@@ -133,7 +149,8 @@ fn edit_external(s: &mut Session, p: &Value) -> Result<Value> {
};
let mut written = Vec::new();
let mut write = |path: &str, bytes: &[u8]| -> std::result::Result<(), String> {
crate::export::write_file(path, bytes)?;
// durable: the TIFF becomes a library photo the external editor then changes
crate::export::write_file_durable(path, bytes)?;
written.push(path.to_string());
Ok(())
};
+38 -12
View File
@@ -98,7 +98,11 @@ pub fn specs() -> Vec<CommandSpec> {
}
}
let label = if vals.len() == 1 { controls::find(&vals[0].0).map(|c| c.label).unwrap_or("Edit").to_string() } else { "Edit".into() };
let apply = |d: &mut DevelopSettings| {
let apply = |d: &mut DevelopSettings, info: &lightcraft_pipeline::SourceInfo| {
if d.wb.mode == WbMode::AsShot && vals.iter().any(|(k, _)| k == "wb.temp" || k == "wb.tint") {
d.wb.temp = info.as_shot_temp;
d.wb.tint = info.as_shot_tint;
}
for (k, v) in &vals {
controls::set(d, k, *v);
if k == "wb.temp" || k == "wb.tint" {
@@ -113,15 +117,18 @@ pub fn specs() -> Vec<CommandSpec> {
.filter_map(|id| s.develop_of(*id).map(|d| (*id, d)))
.filter_map(|(id, d)| {
let mut d = (*d).clone();
apply(&mut d);
let info = s.source_info(id);
apply(&mut d, &info);
s.develop_op(id, d, &label)
})
.collect();
s.commit(&label, Op::Batch { ops })?;
return ok();
}
let id = active(s, "develop.set")?;
let info = s.source_info(id);
edit(s, "develop.set", &label, |d| {
apply(d);
apply(d, &info);
Ok(())
})
}
@@ -130,7 +137,15 @@ pub fn specs() -> Vec<CommandSpec> {
let c = str_param(p, "control").ok_or_else(|| bad("develop.adjust", "missing control"))?.to_string();
let spec = controls::find(&c).ok_or_else(|| bad("develop.adjust", "unknown control"))?;
let delta = f64_req(p, "delta", "develop.adjust")?;
let info = s.source_info(active(s, "develop.adjust")?);
edit(s, "develop.adjust", spec.label, |d| {
if c == "wb.temp" || c == "wb.tint" {
if d.wb.mode == WbMode::AsShot {
d.wb.temp = info.as_shot_temp;
d.wb.tint = info.as_shot_tint;
}
d.wb.mode = WbMode::Custom;
}
let v = controls::get(d, &c).unwrap_or(spec.default);
controls::set(d, &c, v + delta);
Ok(())
@@ -149,12 +164,18 @@ pub fn specs() -> Vec<CommandSpec> {
let ops = ids
.iter()
.filter_map(|id| s.develop_of(*id).map(|d| (*id, d)))
.filter_map(|(id, d)| {
.filter_map(|(id, _d)| {
// back to the photo's import defaults when a default preset gave it its look
let look = s.catalog.photo(id).and_then(|p| p.import_look.clone());
let fresh = match look {
Some(l) => (*l).clone(),
None => DevelopSettings { wb: lightcraft_develop::WhiteBalance { mode: WbMode::AsShot, ..d.wb }, ..Default::default() },
None => {
let info = s.source_info(id);
DevelopSettings {
wb: lightcraft_develop::WhiteBalance { mode: WbMode::AsShot, temp: info.as_shot_temp, tint: info.as_shot_tint },
..Default::default()
}
}
};
s.develop_op(id, fresh, "Reset")
})
@@ -203,7 +224,7 @@ pub fn specs() -> Vec<CommandSpec> {
|s, _| {
let id = active(s, "develop.autoBwMix")?;
let src = s.source_now(id, SourceLevel::Thumb).map_err(|e| bad("develop.autoBwMix", e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let d = s.develop_of(id).unwrap_or_default();
let m = lightcraft_pipeline::auto::auto_bw_mix(&src, &info, &d);
edit(s, "develop.autoBwMix", "Auto B&W Mix", |d| {
@@ -218,7 +239,7 @@ pub fn specs() -> Vec<CommandSpec> {
cmd!("develop.auto", "Auto Settings", ["Photo"], Some("Shift+A"), "{}", has_active, |s, _| {
let id = active(s, "develop.auto")?;
let src = s.source_now(id, SourceLevel::Thumb).map_err(|e| bad("develop.auto", e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let d = s.develop_of(id).unwrap_or_default();
let a = lightcraft_pipeline::auto::auto_tone(&src, &info, &d);
edit(s, "develop.auto", "Auto", |d| {
@@ -245,7 +266,7 @@ pub fn specs() -> Vec<CommandSpec> {
let mode: WbMode =
serde_json::from_value(p.get("mode").cloned().unwrap_or(json!("custom"))).map_err(|e| bad("develop.wb", e.to_string()))?;
let id = active(s, "develop.wb")?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let (mut t, mut tint) = match mode {
WbMode::AsShot => (info.as_shot_temp, info.as_shot_tint),
WbMode::Auto => {
@@ -256,7 +277,7 @@ pub fn specs() -> Vec<CommandSpec> {
.preset()
.map(|(t, ti)| {
// presets are relative to daylight for rendered files
if info.raw { (t, ti) } else { (6500.0 * t / 5500.0, ti) }
if info.raw && !info.relative_wb { (t, ti) } else { (6500.0 * t / 5500.0, ti) }
})
.unwrap_or((info.as_shot_temp, info.as_shot_tint)),
};
@@ -277,7 +298,7 @@ pub fn specs() -> Vec<CommandSpec> {
let (x, y) = (f64_req(p, "x", "develop.wbPick")?, f64_req(p, "y", "develop.wbPick")?);
let id = active(s, "develop.wbPick")?;
let src = s.source_now(id, SourceLevel::Thumb).map_err(|e| bad("develop.wbPick", e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let d = s.develop_of(id).unwrap_or_default();
// Sample a small neighbourhood in the oriented source.
let o = src.oriented(d.orientation);
@@ -405,7 +426,7 @@ pub fn specs() -> Vec<CommandSpec> {
|s, _| {
let id = active(s, "crop.autoStraighten")?;
let src = s.source_now(id, SourceLevel::Preview).map_err(|e| bad("crop.autoStraighten", e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let d = s.develop_of(id).unwrap_or_default();
let Some(level) = lightcraft_pipeline::upright::level_degrees(&src, &info, &d) else {
return Ok(json!({"changed": false, "reason": "no dominant horizontal or vertical lines"}));
@@ -501,7 +522,7 @@ pub fn specs() -> Vec<CommandSpec> {
Upright::Off | Upright::Guided => None,
m => {
let src = s.source_now(id, SourceLevel::Preview).map_err(|e| bad("geometry.upright", e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
Some(lightcraft_pipeline::upright::auto_transform(&src, &info, &d, m))
}
};
@@ -657,6 +678,11 @@ pub fn specs() -> Vec<CommandSpec> {
.filter_map(|id| s.develop_of(id).map(|d| (id, d)))
.filter_map(|(id, d)| {
let mut nd = (*d).clone();
if nd.wb.mode == WbMode::AsShot && (ctl == "wb.temp" || ctl == "wb.tint") {
let info = s.source_info(id);
nd.wb.temp = info.as_shot_temp;
nd.wb.tint = info.as_shot_tint;
}
let v = controls::get(&nd, &ctl).unwrap_or(spec.default);
controls::set(&mut nd, &ctl, v + delta);
if ctl == "wb.temp" || ctl == "wb.tint" {
+119 -76
View File
@@ -7,6 +7,105 @@ use serde_json::{Value, json};
use super::{CommandSpec, always, bad, bool_or, cmd, has_active, has_selection, ok, str_param};
use crate::{LibrarySource, Result, Selection, Session};
/// The auto-import folder's visible files with their sizes (the file-system half of
/// `library.autoImportScan`; the app lists on a worker thread and passes `listing`).
pub fn list_auto_import_folder(folder: &str) -> std::result::Result<Vec<(String, u64)>, String> {
let rd = std::fs::read_dir(folder).map_err(|e| format!("{folder}: {e}"))?;
Ok(rd
.flatten()
.filter(|e| !e.file_name().to_string_lossy().starts_with('.'))
.filter_map(|e| {
let m = std::fs::metadata(e.path()).ok()?;
m.is_file().then(|| (e.path().to_string_lossy().to_string(), m.len()))
})
.collect())
}
/// `library.import`'s params, checked (no file-system calls).
pub struct ImportRequest {
pub paths: Vec<String>,
pub opts: crate::import::ImportOptions,
pub album: Option<u64>,
pub album_name: Option<String>,
}
/// Parse and check `library.import`'s params (the app's import task runs the import itself, on a
/// worker thread, with the same options).
pub fn import_params(s: &Session, p: &Value) -> Result<ImportRequest> {
let paths = strs(p, "paths");
let mode = match str_param(p, "mode").unwrap_or("add") {
"add" => crate::import::ImportMode::Add,
"copy" => crate::import::ImportMode::Copy,
"move" => crate::import::ImportMode::Move,
other => return Err(bad("library.import", format!("unknown mode `{other}` (add|copy|move)"))),
};
let preset = match str_param(p, "preset").filter(|x| !x.is_empty()) {
Some(id) => Some(s.presets.iter().find(|x| x.id == id).cloned().ok_or_else(|| bad("library.import", format!("unknown preset `{id}`")))?),
None => None,
};
let album = p.get("album").and_then(Value::as_u64);
if let Some(a) = album
&& s.catalog.album(AlbumId(a)).is_none_or(|al| al.folder || al.is_smart())
{
return Err(bad("library.import", "album must be a regular album"));
}
let organize = match str_param(p, "organize") {
Some(o) => crate::import::Organize::parse(o).ok_or_else(|| {
bad("library.import", format!("unknown organize `{o}` (date|month|flat, or a folder template like {{date:%Y}}/{{date:%Y%m%d}})"))
})?,
None => Default::default(),
};
if let crate::import::Organize::Template(t) = &organize
&& let Some(e) = crate::rename::folder_template_error(t)
{
return Err(bad("library.import", e));
}
let metadata_preset = str_param(p, "metadataPreset").map(str::trim).filter(|n| !n.is_empty()).map(str::to_string);
if let Some(n) = &metadata_preset
&& !s.metadata_presets.iter().any(|m| m.name.eq_ignore_ascii_case(n))
{
return Err(bad("library.import", format!("unknown metadata preset `{n}`")));
}
let opts = crate::import::ImportOptions {
mode,
preset,
keywords: strs(p, "keywords"),
destination: str_param(p, "destination").map(str::to_string),
organize,
rename: str_param(p, "rename").map(str::to_string),
rename_start: p.get("renameStart").and_then(Value::as_u64).unwrap_or(1) as usize,
metadata_preset,
convert_dng: bool_or(p, "dng", false),
local: bool_or(p, "local", false),
};
let album_name = str_param(p, "albumName").map(str::trim).filter(|n| !n.is_empty()).map(str::to_string);
Ok(ImportRequest { paths, opts, album, album_name })
}
/// After a batch of an import was committed: the album (a new one named `album_name` when there
/// is no `album` and something was imported) gets the photos. Returns the report as JSON (with
/// `album`).
pub fn import_batch_done(s: &mut Session, report: crate::import::ImportReport, mut album: Option<u64>, album_name: Option<&str>) -> Result<Value> {
let mut report = serde_json::to_value(report).unwrap_or_default();
let imported: Vec<u64> = report["imported"].as_array().map(|a| a.iter().filter_map(Value::as_u64).collect()).unwrap_or_default();
if album.is_none()
&& let Some(name) = album_name
&& !imported.is_empty()
{
let r = s.execute("album.create", &json!({"name": name}))?;
album = r["id"].as_u64();
}
if let Some(a) = album
&& !imported.is_empty()
{
// (album.addPhotos wants a selection: the new photos are about to be it)
s.selection = Selection::single(PhotoId(imported[0]));
s.execute("album.addPhotos", &json!({"id": a, "ids": imported}))?;
report["album"] = json!(a);
}
Ok(report)
}
fn album_param(p: &Value, key: &str, c: &str) -> Result<AlbumId> {
p.get(key).and_then(Value::as_u64).map(AlbumId).ok_or_else(|| bad(c, format!("missing album `{key}`")))
}
@@ -657,9 +756,9 @@ pub fn specs() -> Vec<CommandSpec> {
"Auto Import Now",
[],
None,
"{} — add the watched folder's new photos (files the library doesn't have yet; partial / still-copying files wait for the next scan) → {imported, folder}",
"{listing?: [[path, size]] (the folder as listed by the caller, e.g. on a worker thread), start?: bool (false: return the `library.import` params as `import` instead of importing)} — add the watched folder's new photos (files the library doesn't have yet; partial / still-copying files wait for the next scan) → {imported, folder, import?}",
always,
|s, _| {
|s, p| {
let Some(folder) = s.import_defaults.auto_folder.clone() else { return Ok(json!({"imported": [], "folder": null})) };
// only files that stopped growing: a file still being written is left for later
let known: std::collections::HashSet<String> = s
@@ -670,18 +769,20 @@ pub fn specs() -> Vec<CommandSpec> {
_ => None,
})
.collect();
// the folder's files and sizes: listed here, or already listed on a worker thread (the app)
let listing: Vec<(String, u64)> = match p.get("listing").and_then(Value::as_array) {
Some(l) => l.iter().filter_map(|e| Some((e.get(0)?.as_str()?.to_string(), e.get(1)?.as_u64()?))).collect(),
None => list_auto_import_folder(&folder).map_err(|e| bad("library.autoImportScan", e))?,
};
let mut fresh = Vec::new();
for e in std::fs::read_dir(&folder).map_err(|e| bad("library.autoImportScan", format!("{folder}: {e}")))?.flatten() {
let path = e.path();
let ps = path.to_string_lossy().to_string();
if !path.is_file() || known.contains(&ps) || path.file_name().is_some_and(|n| n.to_string_lossy().starts_with('.')) {
for (ps, size) in listing {
if known.contains(&ps) {
continue;
}
// each file is tried once (a non-photo isn't retried every scan)
if s.auto_import_seen.get(&ps) == Some(&u64::MAX) {
continue;
}
let size = e.metadata().map(|m| m.len()).unwrap_or(0);
let seen = s.auto_import_seen.insert(ps.clone(), size);
if size > 0 && seen == Some(size) {
fresh.push(ps);
@@ -701,6 +802,10 @@ pub fn specs() -> Vec<CommandSpec> {
for f in &fresh {
s.auto_import_seen.insert(f.clone(), u64::MAX);
}
if p.get("start").and_then(Value::as_bool) == Some(false) {
// the caller imports them (the app: on a worker thread)
return Ok(json!({"imported": [], "folder": folder, "import": params}));
}
let sel = s.selection.clone();
let r = s.execute("library.import", &params)?;
// arriving photos don't take over the selection
@@ -751,79 +856,15 @@ pub fn specs() -> Vec<CommandSpec> {
"{paths: [file or folder (recursive)], mode?: add|copy|move (add = reference the files in place; copy = into the library's Originals/YYYY/YYYY-MM-DD/; move = as copy, then each original and its XMP sidecars are removed from the source — only after the copy is verified (a hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate and unchecked files keep their sources; a taken name gets -1, -2…; undo removes the photos from the library but leaves the files at the destination), destination?: folder for copies / moves, organize?: date (YYYY/YYYY-MM-DD) | month (YYYY/YYYY-MM) | flat | a folder template, e.g. `{date:%Y}/{date:%Y%m%d}` → 2026/20260114 (the template's `/` make the folders, each level expanded with the rename tokens and made a safe folder name: never outside the destination; must be relative, no `..`; a level with missing metadata is `unknown`) — dated by capture time, else the import time, rename?: file-name template for copies, original extension added (tokens: {name} {num} {seq} {seq:N} {date} {date:%Y%m%d} {folder} {camera} {lens} {iso} {rating} {title} {creator} {ext}; photo.renameTokens explains each; blank = keep names), renameStart?: 1, metadataPreset?: name, dng?: bool (copy raws as DNG; copy only), local?: bool (browsing: the photos stay out of the library, like library.browse; not with move), album?: albumId, albumName?: new album, preset?: presetId, keywords?: [..]} → {imported, duplicates, failed, moved?: [{from, to, sidecars?}], kept?: [{path, reason}] (move: sources left in place and why), album?}",
always,
|s, p| {
let paths = strs(p, "paths");
if paths.is_empty() {
let req = import_params(s, p)?;
if req.paths.is_empty() {
return Err(bad("library.import", "no paths"));
}
let mode = match str_param(p, "mode").unwrap_or("add") {
"add" => crate::import::ImportMode::Add,
"copy" => crate::import::ImportMode::Copy,
"move" => crate::import::ImportMode::Move,
other => return Err(bad("library.import", format!("unknown mode `{other}` (add|copy|move)"))),
};
let preset = match str_param(p, "preset").filter(|x| !x.is_empty()) {
Some(id) => {
Some(s.presets.iter().find(|x| x.id == id).cloned().ok_or_else(|| bad("library.import", format!("unknown preset `{id}`")))?)
}
None => None,
};
let mut album = p.get("album").and_then(Value::as_u64);
if let Some(a) = album
&& s.catalog.album(AlbumId(a)).is_none_or(|al| al.folder || al.is_smart())
{
return Err(bad("library.import", "album must be a regular album"));
}
let organize = match str_param(p, "organize") {
Some(o) => crate::import::Organize::parse(o).ok_or_else(|| {
bad(
"library.import",
format!("unknown organize `{o}` (date|month|flat, or a folder template like {{date:%Y}}/{{date:%Y%m%d}})"),
)
})?,
None => Default::default(),
};
if let crate::import::Organize::Template(t) = &organize
&& let Some(e) = crate::rename::folder_template_error(t)
{
return Err(bad("library.import", e));
}
let metadata_preset = str_param(p, "metadataPreset").map(str::trim).filter(|n| !n.is_empty()).map(str::to_string);
if let Some(n) = &metadata_preset
&& !s.metadata_presets.iter().any(|m| m.name.eq_ignore_ascii_case(n))
{
return Err(bad("library.import", format!("unknown metadata preset `{n}`")));
}
let opts = crate::import::ImportOptions {
mode,
preset,
keywords: strs(p, "keywords"),
destination: str_param(p, "destination").map(str::to_string),
organize,
rename: str_param(p, "rename").map(str::to_string),
rename_start: p.get("renameStart").and_then(Value::as_u64).unwrap_or(1) as usize,
metadata_preset,
convert_dng: super::bool_or(p, "dng", false),
local: super::bool_or(p, "local", false),
};
let undo0 = s.undo.len();
let mut report = serde_json::to_value(crate::import::import_with(s, &paths, &opts)?).unwrap_or_default();
let imported: Vec<u64> = report["imported"].as_array().map(|a| a.iter().filter_map(Value::as_u64).collect()).unwrap_or_default();
if album.is_none()
&& let Some(name) = str_param(p, "albumName").map(str::trim).filter(|n| !n.is_empty())
&& !imported.is_empty()
{
let r = s.execute("album.create", &json!({"name": name}))?;
album = r["id"].as_u64();
}
if let Some(a) = album
&& !imported.is_empty()
{
// (album.addPhotos wants a selection: the new photos are about to be it)
s.selection = Selection::single(PhotoId(imported[0]));
s.execute("album.addPhotos", &json!({"id": a, "ids": imported}))?;
report["album"] = json!(a);
}
let report = crate::import::import_with(s, &req.paths, &req.opts)?;
let report = import_batch_done(s, report, req.album, req.album_name.as_deref())?;
// one undo step for the whole import
let imported: Vec<u64> = report["imported"].as_array().map(|a| a.iter().filter_map(Value::as_u64).collect()).unwrap_or_default();
let n = s.undo.len().saturating_sub(undo0);
s.merge_undo(n, &format!("Add {} Photo{}", imported.len(), if imported.len() == 1 { "" } else { "s" }));
if let Some(f) = imported.first() {
@@ -867,6 +908,8 @@ pub fn specs() -> Vec<CommandSpec> {
"logRecords": j.log_records(),
"logBytes": j.log_bytes(),
"lastError": lib.last_error,
// settings files that were unreadable or damaged at open (kept, defaults used)
"settingsWarnings": lib.settings_warnings,
// changes applied in memory whose write failed (retried by every save)
"unsavedOps": s.unsaved().map_or(0, |u| u.0),
"unsavedError": s.unsaved().map(|u| u.1),
+2 -2
View File
@@ -185,7 +185,7 @@ pub fn specs() -> Vec<CommandSpec> {
let add = p.get("add").and_then(Value::as_bool).unwrap_or(false);
let id = s.active().ok_or_else(|| bad(c, "no active photo"))?;
let src = s.source_now(id, crate::media::SourceLevel::Thumb).map_err(|e| bad(c, e))?;
let info = s.catalog.photo(id).map(|p| crate::media::source_info(p)).unwrap_or_default();
let info = s.source_info(id);
let d = s.develop_of(id).unwrap_or_default();
let req = lightcraft_pipeline::RenderRequest::fit(384, 384);
let lab = lightcraft_pipeline::color_range_sample(&src, &info, &d, &req, lightcraft_geom::Point::new(x, y))
@@ -646,7 +646,7 @@ fn spots_edit(s: &mut Session, c: &str, label: &str, f: impl FnOnce(&mut Vec<Spo
/// An automatic source for `spot` on photo `id` (a small proxy of the photo, framed by `d`).
fn pick_source(s: &mut Session, id: crate::PhotoId, d: &lightcraft_develop::DevelopSettings, spot: &Spot, avoid: Option<Point>) -> Option<Point> {
let src = s.source_now(id, crate::media::SourceLevel::Thumb).ok()?;
let info = crate::media::source_info(s.catalog.photo(id)?);
let info = s.source_info(id);
lightcraft_pipeline::spots::pick_source(&src, &info, d, spot, avoid)
}
+246 -37
View File
@@ -1,8 +1,9 @@
//! Missing originals: photos whose file is no longer where the library expects it (moved,
//! renamed, on an unplugged drive), and relinking them — one at a time (`photo.relink`) or by
//! searching a folder for files with the same name and size (`library.findMissing`).
//! searching a folder for files with the same name and size, or the same content
//! (`library.findMissing`).
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::path::Path;
use lightcraft_catalog::{Catalog, Op, Photo, PhotoId, Source};
@@ -44,9 +45,10 @@ pub fn missing(s: &Session) -> Vec<(PhotoId, String)> {
}
/// Whether photo `id` is in scope and its file is gone (the Missing Photos view checks only
/// the photos its query already narrowed to).
pub fn is_missing(cat: &Catalog, id: PhotoId) -> bool {
!cfg!(target_arch = "wasm32") && cat.photo(id).and_then(|p| checked_path(p)).is_some_and(|f| !Path::new(f).exists())
/// the photos its query already narrowed to). In the app `avail` answers from its cache (files
/// not checked yet count as present until the background check finds them gone).
pub fn is_missing(cat: &Catalog, avail: &crate::availability::Availability, id: PhotoId) -> bool {
!cfg!(target_arch = "wasm32") && cat.photo(id).and_then(|p| checked_path(p)).is_some_and(|f| avail.is_offline(f))
}
fn relink_op(id: PhotoId, path: &str) -> Op {
@@ -83,48 +85,255 @@ fn relink(s: &mut Session, p: &Value) -> Result<Value> {
Ok(json!({"id": id.0, "path": abs}))
}
/// Search `folder` (recursively) for each missing photo's file name — and its size when the
/// library knows it — and relink every match in one undoable step.
fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.findMissing";
let folder = str_param(p, "folder").ok_or_else(|| bad(C, "missing `folder`"))?;
/// The stored content hash of a photo, without the `:dupN` suffix duplicates get.
fn stored_hash(p: &Photo) -> Option<&str> {
p.content_hash.as_deref().map(|h| h.split(':').next().unwrap_or(h)).filter(|h| !h.is_empty())
}
/// The content hash of the file at `path` (as import computes it), if it can be read.
fn file_hash(path: &str) -> Option<String> {
std::fs::read(path).ok().map(|b| lightcraft_preview::hash_bytes(&b).to_string())
}
/// An in-scope photo ([`checked_path`]) as Find Missing Photos sees it: its file, the size and
/// content hash the library knows (0 / `None` = unknown).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FindCandidate {
pub id: PhotoId,
pub path: String,
pub size: u64,
pub hash: Option<String>,
}
/// Every in-scope photo, for [`plan_find_missing`]: the missing ones are looked for, and no
/// photo's file is used for another. No file-system calls, so the app takes this snapshot on the
/// UI thread and plans on a worker without holding the session.
pub fn find_candidates(cat: &Catalog) -> Vec<FindCandidate> {
cat.photos()
.filter_map(|p| {
checked_path(p).map(|f| FindCandidate { id: p.id, path: f.to_string(), size: p.file_size, hash: stored_hash(p).map(str::to_string) })
})
.collect()
}
/// How a missing photo's file was recognised.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MatchBy {
/// Same name and size (and content, when the library knows its hash).
Name,
/// Renamed: same size and content hash.
Content,
}
impl MatchBy {
pub fn as_str(self) -> &'static str {
match self {
MatchBy::Name => "name",
MatchBy::Content => "content",
}
}
}
/// A missing photo and the file found for it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FoundFile {
pub id: PhotoId,
pub from: String,
pub to: String,
pub by: MatchBy,
}
/// A missing photo with several look-alike files (same name and size, no hash to tell them
/// apart): skipped, never guessed.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AmbiguousFile {
pub id: PhotoId,
pub from: String,
pub candidates: Vec<String>,
}
/// What Find Missing Photos found ([`plan_find_missing`]), before relinking.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct FindPlan {
pub found: Vec<FoundFile>,
/// In-scope photos that stay missing (the ambiguous ones included).
pub missing: usize,
pub ambiguous: Vec<AmbiguousFile>,
}
impl FindPlan {
/// `{found: [{id, from, to, by}], missing, ambiguous: [{id, from, candidates}]}` — the
/// command's result, and what `library.findMissing {found, …}` takes back.
pub fn to_json(&self) -> Value {
let found: Vec<Value> = self.found.iter().map(|f| json!({"id": f.id.0, "from": f.from, "to": f.to, "by": f.by.as_str()})).collect();
let ambiguous: Vec<Value> = self.ambiguous.iter().map(|a| json!({"id": a.id.0, "from": a.from, "candidates": a.candidates})).collect();
json!({"found": found, "missing": self.missing, "ambiguous": ambiguous})
}
/// The inverse of [`FindPlan::to_json`] (entries it can't read are dropped).
pub fn from_json(v: &Value) -> FindPlan {
let list = |k: &str| v.get(k).and_then(Value::as_array).cloned().unwrap_or_default();
let found = list("found")
.iter()
.filter_map(|f| {
let by = if f.get("by").and_then(Value::as_str) == Some("content") { MatchBy::Content } else { MatchBy::Name };
Some(FoundFile {
id: PhotoId(f.get("id")?.as_u64()?),
from: str_param(f, "from")?.to_string(),
to: str_param(f, "to")?.to_string(),
by,
})
})
.collect();
let ambiguous = list("ambiguous")
.iter()
.filter_map(|a| {
let candidates = a.get("candidates")?.as_array()?.iter().filter_map(|c| c.as_str().map(str::to_string)).collect();
Some(AmbiguousFile { id: PhotoId(a.get("id")?.as_u64()?), from: str_param(a, "from")?.to_string(), candidates })
})
.collect();
let missing = v.get("missing").and_then(Value::as_u64).map_or(0, |n| usize::try_from(n).unwrap_or(usize::MAX));
FindPlan { found, missing, ambiguous }
}
}
/// The file-system half of Find Missing Photos: which `candidates` are gone, and which file in
/// `folder` (recursively) is each one's. Touches only the disk — never the session — so the app
/// runs it on a worker thread (walking a big folder on a network share and hashing look-alikes
/// takes a while) and relinks the result with `library.findMissing {found, missing, ambiguous}`.
///
/// Matching, per photo:
/// 1. a file with the same name (any letter case), the same size and the same content hash;
/// 2. renamed: a file of the same size with the same content hash (only same-size files are
/// read, so this stays cheap);
/// 3. without a hash to confirm (or the file was edited in place): the one file with the same
/// name and size. Several such files are ambiguous: the photo is skipped and reported in
/// `ambiguous`, never guessed (two cameras' `IMG_0001.CR3` of one size are different photos).
///
/// Files that are already some photo's original, or that another photo was just matched with,
/// are not used.
pub fn plan_find_missing(candidates: &[FindCandidate], folder: &str) -> std::result::Result<FindPlan, String> {
if !Path::new(folder).is_dir() {
return Err(bad(C, format!("{folder}: not a folder")));
return Err(format!("{folder}: not a folder"));
}
let lost = missing(s);
if cfg!(target_arch = "wasm32") {
return Ok(FindPlan::default());
}
let lost: Vec<&FindCandidate> = candidates.iter().filter(|c| !Path::new(&c.path).exists()).collect();
if lost.is_empty() {
return Ok(json!({"found": [], "missing": 0}));
return Ok(FindPlan::default());
}
// file name (lower case) → candidate paths
let in_use: HashSet<&str> = candidates.iter().map(|c| c.path.as_str()).collect();
// file name (lower case) → candidate paths; size → paths (filled lazily)
let mut by_name: HashMap<String, Vec<String>> = HashMap::new();
for f in crate::import::expand(&[folder.to_string()], None) {
if let Some(n) = Path::new(&f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f);
let files: Vec<String> = crate::import::expand(&[folder.to_string()], None).into_iter().filter(|f| !in_use.contains(f.as_str())).collect();
for f in &files {
if let Some(n) = Path::new(f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f.clone());
}
}
let mut ops = Vec::new();
let mut found = Vec::new();
for (id, old) in &lost {
let name = Path::new(old).file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default();
let size = s.catalog.photo(*id).map(|p| p.file_size).unwrap_or(0);
let hit = by_name.get(&name).and_then(|c| {
c.iter().find(|f| size == 0 || std::fs::metadata(f).is_ok_and(|m| m.len() == size)).or_else(|| (c.len() == 1 && size == 0).then(|| &c[0]))
});
if let Some(path) = hit {
ops.push(relink_op(*id, path));
found.push(json!({"id": id.0, "from": old, "to": path}));
let mut by_size: Option<HashMap<u64, Vec<String>>> = None;
let mut hashes: HashMap<String, Option<String>> = HashMap::new();
let mut hash_of = |f: &str| hashes.entry(f.to_string()).or_insert_with(|| file_hash(f)).clone();
let size_of = |f: &str| std::fs::metadata(f).map(|m| m.len()).ok();
let mut used: HashSet<String> = HashSet::new();
let mut plan = FindPlan::default();
for c in &lost {
let name = Path::new(&c.path).file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default();
let size = c.size;
let hash = c.hash.as_deref();
let named: Vec<&String> = by_name
.get(&name)
.map(|v| v.iter().filter(|f| !used.contains(*f) && (size == 0 || size_of(f) == Some(size))).collect())
.unwrap_or_default();
// 1. same name and size, confirmed by the content hash
let mut hit: Option<(String, MatchBy)> =
hash.and_then(|h| named.iter().find(|f| hash_of(f).as_deref() == Some(h))).map(|f| (f.to_string(), MatchBy::Name));
// 2. renamed: same size and content
if hit.is_none()
&& let Some(h) = hash
&& size > 0
{
let sizes = by_size.get_or_insert_with(|| {
let mut m: HashMap<u64, Vec<String>> = HashMap::new();
for f in &files {
if let Some(n) = size_of(f) {
m.entry(n).or_default().push(f.clone());
}
}
m
});
hit = sizes
.get(&size)
.and_then(|v| v.iter().find(|f| !used.contains(*f) && hash_of(f).as_deref() == Some(h)))
.map(|f| (f.clone(), MatchBy::Content));
}
}
let still = lost.len() - found.len();
if !ops.is_empty() {
s.commit("Find Missing Photos", Op::Batch { ops })?;
for f in &found {
if let Some(id) = f["id"].as_u64() {
s.media.forget(PhotoId(id));
// 3. same name and size only (no hash, or the file was edited in place): one candidate
// is taken, several are ambiguous and skipped rather than guessed
if hit.is_none() {
match named.as_slice() {
[one] => hit = Some(((*one).clone(), MatchBy::Name)),
[] => {}
_ => {
plan.ambiguous.push(AmbiguousFile { id: c.id, from: c.path.clone(), candidates: named.iter().map(|f| (*f).clone()).collect() });
continue;
}
}
}
let Some((path, by)) = hit else { continue };
used.insert(path.clone());
plan.found.push(FoundFile { id: c.id, from: c.path.clone(), to: path, by });
}
Ok(json!({"found": found, "missing": still}))
plan.missing = lost.len() - plan.found.len();
Ok(plan)
}
/// Relink what [`plan_find_missing`] found, in one undoable step, re-checked against the
/// catalog as it is now (the search may have run on a worker while the library changed): a
/// photo that no longer points at the path it was missing from (relinked, removed) is left
/// alone, and a file that is now some photo's original — or that the plan names twice — is not
/// used (that photo stays missing). `missing` is the plan's count, corrected for what changed
/// among the photos the plan names.
fn commit_find(s: &mut Session, plan: FindPlan) -> Result<Value> {
let still_at = |cat: &Catalog, id: PhotoId, from: &str| cat.photo(id).and_then(|p| checked_path(p)) == Some(from);
let mut in_use: HashSet<String> = candidates(&s.catalog).into_iter().collect();
let mut found = Vec::new();
let mut missing = plan.missing;
for f in plan.found {
if !still_at(&s.catalog, f.id, &f.from) {
continue;
}
if in_use.contains(&f.to) {
missing += 1;
continue;
}
in_use.insert(f.to.clone());
found.push(f);
}
let planned = plan.ambiguous.len();
let ambiguous: Vec<AmbiguousFile> = plan.ambiguous.into_iter().filter(|a| still_at(&s.catalog, a.id, &a.from)).collect();
// ambiguous photos relinked meanwhile aren't missing any more
missing = missing.saturating_sub(planned - ambiguous.len());
if !found.is_empty() {
s.commit("Find Missing Photos", Op::Batch { ops: found.iter().map(|f| relink_op(f.id, &f.to)).collect() })?;
for f in &found {
s.media.forget(f.id);
}
}
Ok(FindPlan { found, missing, ambiguous }.to_json())
}
/// Search `folder` (recursively) for each missing photo's file ([`plan_find_missing`]) and
/// relink every match in one undoable step. With `found` instead (what a search on a worker
/// thread returned, as `{found, missing, ambiguous}`), only the relinking is done.
fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.findMissing";
if p.get("found").is_some_and(Value::is_array) {
return commit_find(s, FindPlan::from_json(p));
}
let folder = str_param(p, "folder").ok_or_else(|| bad(C, "missing `folder`"))?;
let plan = plan_find_missing(&find_candidates(&s.catalog), folder).map_err(|e| bad(C, e))?;
commit_find(s, plan)
}
pub fn specs() -> Vec<CommandSpec> {
@@ -146,7 +355,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Find Missing Photos",
[],
None,
"{folder} — relink every missing photo whose file (same name and size) is somewhere in the folder → {found: [{id, from, to}], missing}",
"{folder} — relink every missing photo whose file is somewhere in the folder: same name and size (and content, when the library knows its hash), or — renamed — same size and content; or {found, missing, ambiguous} — relink what a search already found (the app searches on a worker thread; photos relinked meanwhile and files now in use are skipped) → {found: [{id, from, to, by: name|content}], missing, ambiguous: [{id, from, candidates}] (several same-name candidates and no hash to tell them apart: skipped)}",
always,
find_missing
),
+2 -2
View File
@@ -5,7 +5,7 @@
//! `edit.*` (undo/redo), and queries (`catalog.query`, `photo.inspect`, `develop.get`…).
mod before;
mod browse;
pub(crate) mod browse;
mod color;
pub(crate) mod convert;
mod cull;
@@ -15,7 +15,7 @@ mod edit;
mod export;
pub mod filters;
pub mod keywords;
mod library;
pub mod library;
pub mod lut_profiles;
pub mod manage;
mod masks;
+132 -32
View File
@@ -23,6 +23,12 @@ pub struct PreviewBuild {
pub failed: AtomicUsize,
pub cancel: AtomicBool,
pub finished: AtomicBool,
/// Damaged smart previews rebuilt (Build Smart Previews only).
pub repaired: AtomicUsize,
/// What is built ("" = previews; "smart previews" — Build Smart Previews runs the same way).
pub what: &'static str,
/// Why the whole build failed (e.g. the smart previews folder isn't there).
pub error: std::sync::Mutex<Option<String>>,
}
impl PreviewBuild {
@@ -33,8 +39,15 @@ impl PreviewBuild {
"failed": self.failed.load(Ordering::Relaxed),
"running": !self.finished.load(Ordering::Relaxed),
"cancelled": self.cancel.load(Ordering::Relaxed),
"what": if self.what.is_empty() { "previews" } else { self.what },
"repaired": self.repaired.load(Ordering::Relaxed),
"error": self.error(),
})
}
pub fn error(&self) -> Option<String> {
self.error.lock().unwrap_or_else(std::sync::PoisonError::into_inner).clone()
}
}
/// The jobs that build `id`'s previews: the largest grid thumbnail, then the view render at
@@ -94,49 +107,136 @@ fn build(s: &mut Session, p: &Value) -> Result<Value> {
Ok(state.json())
}
/// One photo's smart preview to build or discard: (photo, its file in the smart previews folder,
/// how to load the original at preview size).
#[cfg(not(target_arch = "wasm32"))]
type SmartJob = (lightcraft_catalog::PhotoId, std::path::PathBuf, Option<crate::media::SourceRef>);
/// What Build / Discard Smart Previews did.
#[cfg(not(target_arch = "wasm32"))]
#[derive(Default)]
struct SmartCounts {
/// Proxies there now (complete ones kept, missing or damaged ones written).
built: usize,
/// Of those, damaged ones (cut short by a crash or a full drive) that were rebuilt.
repaired: usize,
removed: usize,
/// `[[id, why]]`
failed: Vec<Value>,
}
/// The file-system half of Build / Discard Smart Previews (runs on a worker thread in the app).
/// `custom`: the folder was chosen (on another drive, perhaps): it must be there, it is never
/// recreated. With `state`, progress and cancel.
#[cfg(not(target_arch = "wasm32"))]
fn smart_run(
dir: &std::path::Path,
custom: bool,
discard: bool,
jobs: Vec<SmartJob>,
state: Option<&PreviewBuild>,
) -> std::result::Result<SmartCounts, String> {
let mut n = SmartCounts::default();
if !discard {
// a chosen folder on another drive must be there: never recreate it on this one
if custom {
crate::smart::check_writable(dir)?;
} else {
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
}
}
for (id, path, source) in jobs {
if state.is_some_and(|st| st.cancel.load(Ordering::Relaxed)) {
break;
}
if discard {
if std::fs::remove_file(&path).is_ok() {
n.removed += 1;
}
} else {
// a complete proxy is kept; a missing or damaged one (cut short by a crash or a full
// drive) is (re)built
let damaged = path.exists();
if damaged && crate::smart::is_valid(&path) {
n.built += 1;
} else {
// atomic: a failed write leaves no partial proxy that would pass for a built one;
// and synced (issue #134): built so the photo can be edited while its original
// is offline, when it is the only copy — the sync is small next to the decode
let r = source
.ok_or_else(|| "nothing to build from".to_string())
.and_then(|s| s.load_source())
.and_then(|src| crate::smart::encode(&src.image, src.info_or(Default::default()).camera_tone.as_ref()))
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&path, &b).map_err(|e| format!("{}: {e}", path.display())));
match r {
Ok(()) => {
n.built += 1;
if damaged {
n.repaired += 1;
if let Some(st) = state {
st.repaired.fetch_add(1, Ordering::Relaxed);
}
}
}
Err(e) => {
n.failed.push(json!([id.0, e]));
if let Some(st) = state {
st.failed.fetch_add(1, Ordering::Relaxed);
}
}
}
}
}
if let Some(st) = state {
st.done.fetch_add(1, Ordering::Relaxed);
}
}
Ok(n)
}
/// Smart previews: build (from the original, at preview size) or discard the proxies that keep
/// photos editable while their originals are offline.
/// photos editable while their originals are offline. With `background` (the app) the files are
/// read and written on a worker thread; progress like Build Previews (`library.previewProgress`).
#[cfg(not(target_arch = "wasm32"))]
fn smart(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.smartPreviews";
let dir = s.media.smart_dir.clone().ok_or_else(|| bad(C, "smart previews need a library on disk"))?;
let discard = p.get("discard").and_then(Value::as_bool).unwrap_or(false);
let ids = if p.get("ids").is_some() || !s.selection.ids.is_empty() { s.targets(p) } else { s.visible_cloned() };
let (mut built, mut removed, mut failed) = (0usize, 0usize, Vec::new());
if !discard {
// a chosen folder on another drive must be there: never recreate it on this one
if s.smart_previews_dir.is_some() {
crate::smart::check_writable(&dir).map_err(|e| bad(C, e))?;
} else {
std::fs::create_dir_all(&dir).map_err(|e| bad(C, format!("{}: {e}", dir.display())))?;
}
let background = p.get("background").and_then(Value::as_bool).unwrap_or(false);
if background && s.preview_build.as_ref().is_some_and(|b| !b.finished.load(Ordering::Relaxed)) {
return Err(bad(C, "a preview build is already running"));
}
let ids = if p.get("ids").is_some() || !s.selection.ids.is_empty() { s.targets(p) } else { s.visible_cloned() };
let mut jobs: Vec<SmartJob> = Vec::new();
for id in ids {
let Some(ph) = s.catalog.photo(id).cloned() else { continue };
if !matches!(ph.source, lightcraft_catalog::Source::File { .. }) {
continue;
}
let path = dir.join(crate::smart::file_name(&ph));
if discard {
if std::fs::remove_file(&path).is_ok() {
removed += 1;
}
continue;
}
if path.exists() {
built += 1;
continue;
}
let r = s
.source_now(id, crate::media::SourceLevel::Preview)
.and_then(|src| crate::smart::encode(&src))
.and_then(|b| std::fs::write(&path, b).map_err(|e| e.to_string()));
match r {
Ok(()) => built += 1,
Err(e) => failed.push(json!([id.0, e])),
}
let source = (!discard).then(|| s.media.source_ref(&ph, crate::media::SourceLevel::Preview));
jobs.push((id, path, source));
}
Ok(json!({"built": built, "removed": removed, "failed": failed}))
let custom = s.smart_previews_dir.is_some();
if background {
let what = if discard { "discarding smart previews" } else { "smart previews" };
let state = Arc::new(PreviewBuild { total: jobs.len(), what, ..Default::default() });
s.preview_build = Some(state.clone());
let st = state.clone();
std::thread::Builder::new()
.name("lc-smart-previews".into())
.spawn(move || {
if let Err(e) = smart_run(&dir, custom, discard, jobs, Some(&st)) {
log::warn!("{C}: {e}");
*st.error.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Some(e);
}
st.finished.store(true, Ordering::Relaxed);
})
.map_err(|e| bad(C, format!("could not start: {e}")))?;
return Ok(state.json());
}
let r = smart_run(&dir, custom, discard, jobs, None).map_err(|e| bad(C, e))?;
Ok(json!({"built": r.built, "repaired": r.repaired, "removed": r.removed, "failed": r.failed}))
}
/// The smart previews folder: where it is, whether it is custom, what it holds, whether it can be
@@ -205,7 +305,7 @@ fn smart_location(s: &mut Session, p: &Value) -> Result<Value> {
/// Whether photo `id` has a smart preview.
pub fn has_smart_preview(s: &Session, id: lightcraft_catalog::PhotoId) -> bool {
match (&s.media.smart_dir, s.catalog.photo(id)) {
(Some(dir), Some(p)) => dir.join(crate::smart::file_name(p)).exists(),
(Some(dir), Some(p)) => crate::smart::is_valid(&dir.join(crate::smart::file_name(p))),
_ => false,
}
}
@@ -218,7 +318,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Build Smart Previews",
[],
None,
"{ids?, discard?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline → {built, removed, failed}",
"{ids?, discard?: bool, background?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline; a damaged proxy (cut short) is rebuilt → {built, repaired, removed, failed}; with background (the app's menu) the files are read and written on a worker thread → {total, done, failed, repaired, running, what} like library.buildPreviews (progress: library.previewProgress, stop: library.cancelPreviews)",
always,
smart
),
@@ -249,7 +349,7 @@ pub fn specs() -> Vec<CommandSpec> {
always,
build
),
cmd!(query "library.previewProgress", "Preview Build Progress", [], None, "{} → {total, done, failed, running, cancelled} | null", always, |s, _| {
cmd!(query "library.previewProgress", "Preview Build Progress", [], None, "{} → {total, done, failed, repaired, running, cancelled, what, error} | null", always, |s, _| {
Ok(s.preview_build.as_ref().map_or(Value::Null, |b| b.json()))
}),
cmd!("library.cancelPreviews", "Cancel Preview Build", [], None, "{}", always, |s, _| {
+1 -1
View File
@@ -113,7 +113,7 @@ pub fn specs() -> Vec<CommandSpec> {
.map(|r| json!({"group": r.group, "tag": r.tag, "name": r.name, "value": r.value}))
.collect();
// XMP: the sidecar if there is one, else the file's own packet
let packet = crate::sidecar::read_packet(path, ph.kind, s.xmp.naming).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let packet = crate::sidecar::read_packet(path, ph.kind, s.sidecar_naming(ph.id)).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let xmp: Vec<Value> = packet
.and_then(|x| lightcraft_meta::parse_xmp(&x).ok())
.map(|d| d.properties.into_iter().filter(|(k, _)| !k.starts_with("lc:")).map(|(k, v)| json!({"name": k, "value": v.join("; ")})).collect())
+16 -4
View File
@@ -9,14 +9,26 @@ use crate::{Result, Session};
fn save(s: &mut Session, p: &Value) -> Result<Value> {
let mut written = Vec::new();
let mut merged = Vec::new();
let mut backups = Vec::new();
let mut failed = Vec::new();
let owners = crate::sidecar::StemOwners::of(&s.catalog);
for id in s.targets(p) {
match s.save_sidecar(id) {
Ok(path) => written.push(path.display().to_string()),
match s.save_sidecar_with(id, &owners) {
Ok(r) => {
let path = r.path.display().to_string();
if r.merged {
merged.push(path.clone());
}
if let Some(b) = r.backup {
backups.push(json!({"path": path, "backup": b.display().to_string()}));
}
written.push(path);
}
Err(e) => failed.push(json!({"id": id.0, "error": e.to_string()})),
}
}
Ok(json!({"written": written, "failed": failed}))
Ok(json!({"written": written, "merged": merged, "backups": backups, "failed": failed}))
}
fn read(s: &mut Session, p: &Value) -> Result<Value> {
@@ -63,7 +75,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Save Metadata to File",
["Photo"],
Some("Cmd+S"),
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original → {written: [path], failed}",
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original, merged into an existing one (other apps' data kept; an unreadable one is backed up first) → {written: [path], merged: [path], backups: [{path, backup}], failed}",
has_selection,
save
),
+16 -4
View File
@@ -1131,15 +1131,27 @@ pub fn export_batch(
}
/// Write an exported or rendered file on disk: its folder is created if needed, and the file is
/// replaced atomically ([`lightcraft_catalog::safe_file::write_atomic`]: a temp file, synced, then
/// renamed), so a failure part-way leaves any previous file intact and no truncated one. The
/// native writer behind exports, renders and screenshots (app, CLI, MCP).
/// replaced atomically ([`lightcraft_catalog::safe_file::write_atomic_nosync`]: a temp file
/// renamed into place), so a failure part-way leaves any previous file intact and no truncated
/// one. Not synced to disk (issue #134): an export can always be made again from the original.
/// The native writer behind exports, renders and screenshots (app, CLI, MCP).
pub fn write_file(path: &str, bytes: &[u8]) -> Result<(), String> {
write_with(path, bytes, false)
}
/// [`write_file`], synced to disk: for a written file that becomes a library photo (Edit a Copy)
/// and so can't simply be exported again once the catalog points at it.
pub fn write_file_durable(path: &str, bytes: &[u8]) -> Result<(), String> {
write_with(path, bytes, true)
}
fn write_with(path: &str, bytes: &[u8], durable: bool) -> Result<(), String> {
use lightcraft_catalog::safe_file::{write_atomic, write_atomic_nosync};
let p = std::path::Path::new(path);
if let Some(dir) = p.parent().filter(|d| !d.as_os_str().is_empty()) {
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
}
lightcraft_catalog::safe_file::write_atomic(p, bytes).map_err(|e| format!("{path}: {e}"))
if durable { write_atomic(p, bytes) } else { write_atomic_nosync(p, bytes) }.map_err(|e| format!("{path}: {e}"))
}
/// The path of the sidecar with extension `ext` of the exported file `main`.
+103 -10
View File
@@ -131,9 +131,13 @@ pub fn probe_bytes(name: &str, bytes: &[u8]) -> Result<ProbeInfo, String> {
std::mem::swap(&mut w, &mut h);
}
let (t, tint) = xy_to_temp_tint(lightcraft_raw::color::as_shot_white_xy_of(&raw));
let as_shot_wb = Some((t.round(), tint.round()));
// Vendor RGB multipliers do not identify an absolute illuminant without camera calibration
// (the same decoder fact as `CameraTransform::matrix_is_fallback`).
let relative = !lightcraft_raw::color::has_matrix(&raw.color);
let as_shot_wb = Some(if relative { (6500.0, 0.0) } else { (t.round(), tint.round()) });
let embedded_lens = embedded_lens(&raw);
return Ok(ProbeInfo {
fallback_matrix: Some(relative),
embedded_lens,
width: w,
height: h,
@@ -176,6 +180,7 @@ pub fn probe_bytes(name: &str, bytes: &[u8]) -> Result<ProbeInfo, String> {
embedded_lens: None,
xmp: None,
preview_only: None,
fallback_matrix: None,
})
}
@@ -221,6 +226,9 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
}
Err(e) => return Err(e.to_string()),
};
let xy = lightcraft_raw::color::as_shot_white_xy(&raw);
let t = lightcraft_raw::color::camera_transform(&raw, xy);
let camera_look = crate::camera_preview::fit_preview(&raw, &bytes, &t);
drop(bytes);
// Embedded lens corrections are applied by the pipeline ("Enable Profile Corrections"), not baked in.
let lens = embedded_lens(&raw.info());
@@ -242,21 +250,25 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
// the samples aren't needed any more (the colour model below reads only the tags)
raw.data = lightcraft_raw::RawData::U16(Vec::new());
let mut stages = vec![("develop", t0.elapsed())];
let xy = lightcraft_raw::color::as_shot_white_xy(&raw);
let t = lightcraft_raw::color::camera_transform(&raw, xy);
stages.push(("transform", t0.elapsed()));
lightcraft_raw::highlight::reconstruct(&mut img, t.wb, HIGHLIGHT_CLIP);
stages.push(("highlights", t0.elapsed()));
let m = t.matrix.to_f32();
let m = camera_look.map(|p| p.matrix.mul(&t.matrix)).unwrap_or(t.matrix).to_f32();
let gain = 2f32.powf(t.baseline_exposure as f32);
let wb = t.wb;
// A DNG's own profile look (hue/saturation map, look table), DNG spec chapter 6.
let tables = lightcraft_raw::profile::ProfileTables::new(&raw.color.profile, lightcraft_raw::color::illuminant_weight(&raw.color, xy));
img.map_in_place(|p| {
let c = [p[0] * wb[0], p[1] * wb[1], p[2] * wb[2]];
[
((m[0][0] * c[0] + m[0][1] * c[1] + m[0][2] * c[2]) * gain).max(0.0),
((m[1][0] * c[0] + m[1][1] * c[1] + m[1][2] * c[2]) * gain).max(0.0),
((m[2][0] * c[0] + m[2][1] * c[1] + m[2][2] * c[2]) * gain).max(0.0),
]
let rgb = [
m[0][0] * c[0] + m[0][1] * c[1] + m[0][2] * c[2],
m[1][0] * c[0] + m[1][1] * c[1] + m[1][2] * c[2],
m[2][0] * c[0] + m[2][1] * c[1] + m[2][2] * c[2],
];
match &tables {
Some(tables) => tables.apply(rgb, gain).map(|v| v.max(0.0)),
None => rgb.map(|v| (v * gain).max(0.0)),
}
});
stages.push(("colour", t0.elapsed()));
let img = fit(&img, max_edge, max_edge, Filter::Box);
@@ -276,7 +288,12 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
eprintln!("[profile] raw source {}×{} (max {max_edge}, ms after decode): {}", img.width, img.height, parts.join(", "));
}
let (temp, tint) = xy_to_temp_tint(xy);
return Ok((img, SourceInfo { raw: true, as_shot_temp: temp.round(), as_shot_tint: tint.round(), lens }));
// The catalog's record of this fact decides how saved WB settings are read (see
// `DecodedSource::info_or`); this is the decoder's view of a file nobody catalogued.
let relative = t.matrix_is_fallback;
let camera_tone = camera_look.map(|p| p.tone).or_else(|| raw.color.profile.tone_curve.as_ref().and_then(dng_tone_curve));
let (temp, tint) = if relative { (6500.0, 0.0) } else { (temp.round(), tint.round()) };
return Ok((img, SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens, relative_wb: relative, camera_tone }));
}
let d = lightcraft_codecs::decode(&bytes, lightcraft_codecs::DecodeOptions::fit(max_edge as u32, max_edge as u32)).map_err(|e| e.to_string())?;
drop(bytes);
@@ -285,6 +302,17 @@ fn load_bytes_now(bytes: std::borrow::Cow<'_, [u8]>, max_edge: usize) -> Result<
Ok((img.into_oriented(Orientation::from_exif(d.orientation)), SourceInfo::default()))
}
/// A DNG `ProfileToneCurve` (linear in, linear out, 1.0 = white after exposure compensation) as the
/// finish stage's camera tone curve: 32 knots, log-spaced over 12 stops below white; above white
/// the camera tone's shoulder continues it.
pub(crate) fn dng_tone_curve(curve: &lightcraft_raw::profile::ToneCurve) -> Option<lightcraft_pipeline::tone::CameraTone> {
let knots: [[f32; 2]; 32] = std::array::from_fn(|i| {
let x = 2f32.powf(-12.0 + 12.0 * i as f32 / 31.0);
[x, curve.eval(x).min(0.9995)]
});
lightcraft_pipeline::tone::CameraTone::new(knots)
}
/// Orientation for an embedded preview: its own EXIF orientation when it has one, else the raw file's.
fn preview_orientation(raw_bytes: &[u8], jpeg_orientation: u16) -> Orientation {
if jpeg_orientation > 1 {
@@ -390,6 +418,41 @@ mod tests {
b
}
/// Issue #138: a DNG's own profile look (hue/saturation map, look table, tone curve) is
/// applied when it's developed — Lightroom-converted DNGs rendered flat and muted without it.
#[test]
fn dng_profile_look_is_applied() {
use lightcraft_raw::profile::{HsvTable, ProfileLook, ToneCurve};
let plain = crate::tests_xmp::synthetic_dng_with(None, Default::default());
let mut raw = lightcraft_raw::decode(&plain).unwrap();
let sat = |img: &Rgb32f| {
img.data.iter().map(|p| (p[0].max(p[1]).max(p[2]) - p[0].min(p[1]).min(p[2])) / p[0].max(p[1]).max(p[2]).max(1e-6)).sum::<f32>()
/ img.data.len() as f32
};
let (before, info) = load_bytes(&plain, 64).unwrap();
assert!(sat(&before) > 0.05, "the synthetic scene is coloured");
assert!(info.camera_tone.is_none());
// a map that removes all saturation, and a tone curve
let grey = HsvTable { hue_divisions: 4, sat_divisions: 2, val_divisions: 1, data: vec![[0.0, 0.0, 1.0]; 8], srgb_value: false };
raw.color.profile =
ProfileLook { hue_sat_map: [Some(grey), None], look_table: None, tone_curve: ToneCurve::from_tag(&[0.0, 0.0, 0.18, 0.3, 1.0, 1.0]) };
let with = lightcraft_raw::write_dng(&raw, &Default::default()).unwrap();
let (after, info) = load_bytes(&with, 64).unwrap();
assert!(sat(&after) < 1e-3, "saturation {} → {}", sat(&before), sat(&after));
// a saturation-only map keeps brightness roughly (HSV value is kept in ProPhoto RGB)
let mean = |img: &Rgb32f| img.data.iter().map(|p| p[0].max(p[1]).max(p[2])).sum::<f32>() / img.data.len() as f32;
assert!((0.8..1.1).contains(&(mean(&after) / mean(&before))), "{} vs {}", mean(&after), mean(&before));
let tone = info.camera_tone.expect("the DNG tone curve becomes the camera tone");
assert!((tone.apply(0.18) - 0.3).abs() < 0.01, "{}", tone.apply(0.18));
// a look table alone changes the render too (applied after exposure)
raw.color.profile = ProfileLook {
look_table: Some(HsvTable { hue_divisions: 1, sat_divisions: 2, val_divisions: 1, data: vec![[0.0, 1.0, 0.5]; 2], srgb_value: false }),
..Default::default()
};
let (dim, _) = load_bytes(&lightcraft_raw::write_dng(&raw, &Default::default()).unwrap(), 64).unwrap();
assert!((mean(&dim) / mean(&before) - 0.5).abs() < 0.02, "{} vs {}", mean(&dim), mean(&before));
}
#[test]
fn unsupported_raw_falls_back_to_embedded_preview() {
let b = cr3_with_preview(48, 32);
@@ -458,6 +521,36 @@ mod tests {
let _ = std::fs::remove_dir_all(&dir);
}
/// Whether a raw's white balance is relative comes from the decoder (no colour matrix), not
/// from its extension, and import records it on the photo.
#[test]
fn import_records_whether_the_raw_has_a_colour_matrix() {
let with = crate::tests_xmp::synthetic_dng_with(None, lightcraft_meta::Metadata::default());
let mut raw = lightcraft_raw::decode(&with).unwrap();
raw.color.color_matrix = [None, None];
raw.color.as_shot_neutral = None;
raw.wb_multipliers = Some([2.0, 1.0, 1.5]);
let without = lightcraft_raw::write_dng(&raw, &Default::default()).unwrap();
let p = probe_bytes("a.dng", &with).unwrap();
assert_eq!(p.fallback_matrix, Some(false));
assert_ne!(p.as_shot_wb, Some((6500.0, 0.0)));
let p = probe_bytes("a.dng", &without).unwrap();
assert_eq!((p.fallback_matrix, p.as_shot_wb), (Some(true), Some((6500.0, 0.0))));
let (_, info) = load_bytes(&without, 64).unwrap();
assert!(info.relative_wb);
assert!(!load_bytes(&with, 64).unwrap().1.relative_wb);
let dir = std::env::temp_dir().join(format!("lc-fallback-matrix-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(dir.join("no-matrix.dng"), &without).unwrap();
let mut s = crate::Session::new().with_fs();
let r = s.execute("library.import", &serde_json::json!({"paths": [dir.to_string_lossy()]})).unwrap();
let id = lightcraft_catalog::PhotoId(r["imported"][0].as_u64().unwrap());
let p = s.catalog.photo(id).unwrap();
assert_eq!(p.fallback_matrix, Some(true));
assert!(p.relative_wb() && crate::media::source_info(p).relative_wb);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn quick_jobs_show_embedded_previews_then_cached_renders() {
use crate::media::QuickSource;
+379 -198
View File
@@ -374,25 +374,17 @@ fn probe_all(probe: Option<&crate::media::FileProbe>, paths: &[String], progress
}
/// Copy `src` into `root`/`folders` as `name` (default: its own name), made unique with -1, -2…;
/// returns the new path.
fn copy_into(root: &Path, src: &str, folders: &[String], name: Option<&str>) -> Result<String, String> {
/// returns the new path. The copy is verified ([`crate::import_move::copy_new`]: a new file,
/// synced and checked against `probe_hash`, the content hash the probe computed — or compared
/// byte for byte with the source when there is none); a bad one is removed and the photo
/// reported as failed.
fn copy_into(root: &Path, src: &str, folders: &[String], name: Option<&str>, probe_hash: Option<&str>) -> Result<String, String> {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
std::fs::create_dir_all(&dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let name = name
.map(str::to_string)
.unwrap_or_else(|| Path::new(src).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| "photo".into()));
let (stem, ext) = match name.rsplit_once('.') {
Some((s, e)) => (s.to_string(), format!(".{e}")),
None => (name.clone(), String::new()),
};
let mut dst = dir.join(&name);
let mut i = 1;
while dst.exists() {
dst = dir.join(format!("{stem}-{i}{ext}"));
i += 1;
}
std::fs::copy(src, &dst).map_err(|e| format!("copy {src}: {e}"))?;
Ok(dst.to_string_lossy().to_string())
let expect = probe_hash.and_then(lightcraft_preview::Hash128::parse);
crate::import_move::copy_new(Path::new(src), &dir, &name, expect).map(|p| p.to_string_lossy().to_string())
}
/// What a [`scan`] needs from the session, so it can run on another thread (a folder on a network
@@ -528,205 +520,393 @@ pub fn import(s: &mut Session, paths: &[String], mode: ImportMode) -> crate::Res
}
/// Import files/folders with options (preset, keywords). See the module docs.
///
/// The file-system half ([`ImportJob::prepare`]: expanding folders, probing, reading sidecars,
/// copying / placing files) and the catalog half ([`commit_prepared`]) are separate so the app can
/// run the first on a worker thread; here both run in turn.
pub fn import_with(s: &mut Session, paths: &[String], opts: &ImportOptions) -> crate::Result<ImportReport> {
let mode = opts.mode;
// Only a library on disk has an `Originals/` folder. The browser build keeps the bytes of every
// added file in its own storage already, so "copy" there means "add".
let transfer = matches!(mode, ImportMode::Copy | ImportMode::Move);
let mode = if transfer && s.library.as_ref().is_some_and(|l| !l.on_disk) { ImportMode::Add } else { mode };
if mode == ImportMode::Move && opts.local {
return Err(crate::EngineError::Other("browsing a folder never moves its files".into()));
}
let lib_dir = s.library.as_ref().filter(|l| l.on_disk).map(|l| l.dir.clone());
// where copies go: the chosen folder, else the library's Originals/
let copy_root = opts
.destination
.as_deref()
.filter(|d| !d.trim().is_empty())
.map(std::path::PathBuf::from)
.or_else(|| lib_dir.as_ref().map(|l| l.join("Originals")));
if matches!(mode, ImportMode::Copy | ImportMode::Move) && copy_root.is_none() {
return Err(crate::EngineError::Other("copying or moving needs a destination folder or an open library".into()));
}
let mut seq = opts.rename_start.max(1);
let files = expand(paths, lib_dir.as_deref());
let mut report = ImportReport { scanned: files.len(), ..Default::default() };
let mut job = ImportJob::new(s, opts.clone())?;
let prepared = job.prepare(paths, &std::sync::atomic::AtomicBool::new(false));
// probes a scan left for files not imported now stay for a later import
s.import_probes.extend(std::mem::take(&mut job.cache));
commit_prepared(s, &job.opts, &job.now, prepared)
}
// existing paths and content hashes
let mut by_path: HashMap<&str, PhotoId> = HashMap::new();
let mut by_hash: HashMap<String, PhotoId> = HashMap::new();
for p in s.catalog.photos() {
if let Source::File { path } = &p.source {
by_path.insert(path.as_str(), p.id);
}
if let Some(h) = &p.content_hash {
by_hash.insert(h.clone(), p.id);
}
}
let mut todo = Vec::new();
// a file that was only browsed (Local) joins the library when it is imported for real
let mut promote = Vec::new();
for f in files {
match by_path.get(f.as_str()) {
Some(id) if !opts.local && s.catalog.photo(*id).is_some_and(|p| p.local) => {
promote.push(Op::SetLocal { id: *id, local: false });
report.imported.push(id.0);
/// Everything the file-system half of an import needs, detached from the session so it can run
/// on another thread (a network share or a card can take minutes). Built by [`ImportJob::new`];
/// [`ImportJob::prepare`] may be called for several batches in turn (renamed copies keep counting,
/// duplicates are found across batches); each batch is then [`commit_prepared`] on the session.
pub struct ImportJob {
pub opts: ImportOptions,
/// The mode in effect (Copy / Move become Add in a library without its own folder).
mode: ImportMode,
lib_dir: Option<PathBuf>,
copy_root: Option<PathBuf>,
/// path → (photo, a Local browse record?)
by_path: HashMap<String, (PhotoId, bool)>,
/// content hash → the photo that has it (`None`: a file earlier in this import)
by_hash: HashMap<String, Option<PhotoId>>,
probe: Option<crate::media::FileProbe>,
file_bytes: Option<crate::merge::ByteReader>,
/// Probes from a preceding scan (not read again).
cache: HashMap<String, ProbeInfo>,
naming: crate::sidecar::SidecarNaming,
now: String,
seq: usize,
/// Files found so far (folders expanded) and handled so far, for progress.
pub total: std::sync::Arc<std::sync::atomic::AtomicUsize>,
pub done: std::sync::Arc<std::sync::atomic::AtomicUsize>,
}
/// The outcome of [`ImportJob::prepare`] for one batch, in file order.
#[derive(Debug, Default)]
pub struct Prepared {
pub scanned: usize,
items: Vec<PreparedItem>,
}
#[derive(Debug)]
enum PreparedItem {
/// A Local (browsed) record imported for real.
Promote(PhotoId),
Duplicate {
path: String,
existing: Option<PhotoId>,
reason: &'static str,
hash: Option<String>,
},
Failed(String, String),
Kept(Kept),
Ready(Box<ReadyFile>),
}
#[derive(Debug)]
struct ReadyFile {
path: String,
stored: String,
info: ProbeInfo,
sidecar: Option<crate::sidecar::SidecarData>,
placed: Option<crate::import_move::Placed>,
}
impl Prepared {
/// Undo what the batch did on disk that only a commit would make permanent: files placed by a
/// Move (their sources were never touched). Copies stay, like copies of a failed commit.
pub fn rollback(&self) {
for it in &self.items {
if let PreparedItem::Ready(r) = it
&& let Some(pl) = &r.placed
{
crate::import_move::rollback(pl);
}
Some(id) => report.duplicates.push(Duplicate { path: f, existing: Some(id.0), reason: "path" }),
None => todo.push(f),
}
}
// files a preceding scan already probed are not read again (a network share is slow to read)
let cached: Vec<Option<ProbeInfo>> = todo.iter().map(|f| s.import_probes.remove(f)).collect();
let missing: Vec<String> = todo.iter().zip(&cached).filter(|(_, c)| c.is_none()).map(|(f, _)| f.clone()).collect();
let mut fresh = probe_all(s.media.file_probe.as_ref(), &missing, &ScanProgress::default()).into_iter();
let probed: Vec<Result<ProbeInfo, String>> =
cached.into_iter().map(|c| c.map(Ok).unwrap_or_else(|| fresh.next().unwrap_or_else(|| Err("not probed".into())))).collect();
crate::memory::release();
let now = (s.clock)();
let mut ops = promote;
// Move: placed at the destination, sources untouched until the records are committed
let mut placed: Vec<crate::import_move::Placed> = Vec::new();
for (path, info) in todo.into_iter().zip(probed) {
let info = match info {
Ok(i) => i,
Err(e) => {
log::warn!("import {path}: {e}");
report.failed.push((path, e));
/// Entries in the batch (after expanding folders).
pub fn len(&self) -> usize {
self.items.len()
}
pub fn is_empty(&self) -> bool {
self.items.is_empty()
}
}
impl ImportJob {
/// Snapshot what an import with `opts` needs from the session (no file-system calls). Takes
/// the probes a preceding scan left ([`Session::import_probes`]).
pub fn new(s: &mut Session, opts: ImportOptions) -> crate::Result<Self> {
let mode = opts.mode;
// Only a library on disk has an `Originals/` folder. The browser build keeps the bytes of
// every added file in its own storage already, so "copy" there means "add".
let transfer = matches!(mode, ImportMode::Copy | ImportMode::Move);
let mode = if transfer && s.library.as_ref().is_some_and(|l| !l.on_disk) { ImportMode::Add } else { mode };
if mode == ImportMode::Move && opts.local {
return Err(crate::EngineError::Other("browsing a folder never moves its files".into()));
}
let lib_dir = s.library.as_ref().filter(|l| l.on_disk).map(|l| l.dir.clone());
// where copies go: the chosen folder, else the library's Originals/
let copy_root = opts
.destination
.as_deref()
.filter(|d| !d.trim().is_empty())
.map(std::path::PathBuf::from)
.or_else(|| lib_dir.as_ref().map(|l| l.join("Originals")));
if matches!(mode, ImportMode::Copy | ImportMode::Move) && copy_root.is_none() {
return Err(crate::EngineError::Other("copying or moving needs a destination folder or an open library".into()));
}
let mut by_path = HashMap::new();
let mut by_hash = HashMap::new();
for p in s.catalog.photos() {
if let Source::File { path } = &p.source {
by_path.insert(path.clone(), (p.id, p.local));
}
if let Some(h) = &p.content_hash {
by_hash.insert(h.clone(), Some(p.id));
}
}
Ok(ImportJob {
seq: opts.rename_start.max(1),
opts,
mode,
lib_dir,
copy_root,
by_path,
by_hash,
probe: s.media.file_probe.clone(),
file_bytes: s.media.file_bytes.clone(),
cache: std::mem::take(&mut s.import_probes),
naming: s.xmp.naming,
now: (s.clock)(),
total: Default::default(),
done: Default::default(),
})
}
/// The file-system half of importing `paths`: expand folders, skip known paths, probe, skip
/// duplicates by content, read sidecars, copy or place (Move) the files. Nothing in the
/// catalog changes. Stops before the next file once `cancel` is set.
pub fn prepare(&mut self, paths: &[String], cancel: &std::sync::atomic::AtomicBool) -> Prepared {
let files = self.expand(paths);
self.prepare_files(files, cancel)
}
/// Expand `paths` (folders recursively, the library's own folder skipped) into the files an
/// import would handle; counted in [`ImportJob::total`].
pub fn expand(&self, paths: &[String]) -> Vec<String> {
let files = expand(paths, self.lib_dir.as_deref());
self.total.fetch_add(files.len(), std::sync::atomic::Ordering::Relaxed);
files
}
/// When the import started ([`Session::clock`]), for [`commit_prepared`].
pub fn now(&self) -> &str {
&self.now
}
/// [`ImportJob::prepare`] for files [`ImportJob::expand`] returned.
pub fn prepare_files(&mut self, files: Vec<String>, cancel: &std::sync::atomic::AtomicBool) -> Prepared {
use std::sync::atomic::Ordering::Relaxed;
let opts = self.opts.clone();
let mode = self.mode;
let mut out = Prepared { scanned: files.len(), items: Vec::new() };
let mut todo = Vec::new();
for f in files {
match self.by_path.get(f.as_str()).copied() {
Some((id, true)) if !opts.local => {
// a file that was only browsed (Local) joins the library when it is imported for real
self.by_path.insert(f, (id, false));
out.items.push(PreparedItem::Promote(id));
self.done.fetch_add(1, Relaxed);
}
Some((id, _)) => {
out.items.push(PreparedItem::Duplicate { path: f, existing: Some(id), reason: "path", hash: None });
self.done.fetch_add(1, Relaxed);
}
None => todo.push(f),
}
}
// files a preceding scan already probed are not read again (a network share is slow to read)
let cached: Vec<Option<ProbeInfo>> = todo.iter().map(|f| self.cache.remove(f)).collect();
let missing: Vec<String> = todo.iter().zip(&cached).filter(|(_, c)| c.is_none()).map(|(f, _)| f.clone()).collect();
let progress = ScanProgress::default();
let mut fresh = probe_all(self.probe.as_ref(), &missing, &progress).into_iter();
let probed: Vec<Result<ProbeInfo, String>> =
cached.into_iter().map(|c| c.map(Ok).unwrap_or_else(|| fresh.next().unwrap_or_else(|| Err("not probed".into())))).collect();
crate::memory::release();
for (path, info) in todo.into_iter().zip(probed) {
if cancel.load(Relaxed) {
break;
}
self.done.fetch_add(1, Relaxed);
let mut info = match info {
Ok(i) => i,
Err(e) => {
log::warn!("import {path}: {e}");
out.items.push(PreparedItem::Failed(path, e));
continue;
}
};
if let Some(h) = &info.content_hash
&& !opts.local
&& let Some(existing) = self.by_hash.get(h)
{
out.items.push(PreparedItem::Duplicate { path, existing: *existing, reason: "content", hash: Some(h.clone()) });
continue;
}
};
if let Some(h) = &info.content_hash
&& !opts.local
&& let Some(id) = by_hash.get(h)
{
report.duplicates.push(Duplicate { path, existing: Some(id.0), reason: "content" });
continue;
}
// the XMP sidecar (or a raw's embedded XMP), read before copying: its capture time files
// and names the copy when the file itself has none
let raw = info.kind == MediaKind::Raw;
let packet = crate::sidecar::find_sidecar(&path, s.xmp.naming)
.and_then(|f| std::fs::read_to_string(f).ok())
.or_else(|| info.xmp.clone().filter(|_| raw));
let sidecar = packet.and_then(|x| match crate::sidecar::parse_sidecar(&x, raw).map(|sc| sc.resolve_label(&s.catalog)) {
Ok(sc) => (sc != crate::sidecar::SidecarData::default()).then_some(sc),
Err(e) => {
log::warn!("import {path}: XMP: {e}");
None
// the XMP sidecar (or a raw's embedded XMP), read before copying: its capture time files
// and names the copy when the file itself has none
let raw = info.kind == MediaKind::Raw;
let packet = crate::sidecar::find_sidecar(&path, self.naming)
.and_then(|f| std::fs::read_to_string(f).ok())
.or_else(|| info.xmp.clone().filter(|_| raw));
let sidecar = packet.and_then(|x| match crate::sidecar::parse_sidecar(&x, raw) {
Ok(sc) => Some(sc),
Err(e) => {
log::warn!("import {path}: XMP: {e}");
None
}
});
if info.captured.is_none() {
info.captured = sidecar.as_ref().and_then(|sc| sc.captured.clone());
}
});
let mut info = info;
if info.captured.is_none() {
info.captured = sidecar.as_ref().and_then(|sc| sc.captured.clone());
}
let stored = match (mode, &copy_root) {
(ImportMode::Copy | ImportMode::Move, Some(root))
if !crate::import_move::inside(Path::new(&path), root)
&& !lib_dir.as_ref().is_some_and(|l| crate::import_move::inside(Path::new(&path), l)) =>
let mut placed = None;
let stored = match (mode, &self.copy_root) {
(ImportMode::Copy | ImportMode::Move, Some(root))
if !crate::import_move::inside(Path::new(&path), root)
&& !self.lib_dir.as_ref().is_some_and(|l| crate::import_move::inside(Path::new(&path), l)) =>
{
// the templates see the photo as it will be catalogued (undated: the import time)
let own = Path::new(&path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let mut q = Photo::new(PhotoId(0), Source::File { path: path.clone() }, &own, &info.format, 0, 0, &self.now);
q.captured = info.captured.clone();
q.meta = info.meta.clone();
let name = opts.rename.as_deref().filter(|t| !t.trim().is_empty()).map(|t| {
let n = crate::rename::expand(t, &q, self.seq);
self.seq += 1;
n
});
let folders = opts.organize.folders(&q);
if mode == ImportMode::Move && !crate::import_move::is_symlink(Path::new(&path)) {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
let name = name.unwrap_or(own);
match crate::import_move::place(Path::new(&path), &dir, &name) {
Ok(pl) => {
let dst = pl.dst.to_string_lossy().to_string();
placed = Some(pl);
dst
}
Err(e) => {
out.items.push(PreparedItem::Failed(path, e));
continue;
}
}
} else {
if mode == ImportMode::Move {
let reason = "a symbolic link: the file it points to was copied, the link and its target stay";
out.items.push(PreparedItem::Kept(Kept { path: path.clone(), reason: reason.into() }));
}
match copy_into(root, &path, &folders, name.as_deref(), info.content_hash.as_deref()) {
Ok(p) => p,
Err(e) => {
out.items.push(PreparedItem::Failed(path, e));
continue;
}
}
}
}
(ImportMode::Move, _) => {
let reason = "already inside the destination or the library: added where it is";
out.items.push(PreparedItem::Kept(Kept { path: path.clone(), reason: reason.into() }));
path.clone()
}
_ => path.clone(),
};
if let Some(h) = &info.content_hash {
self.by_hash.insert(h.clone(), None);
}
// Copy as DNG: the copied raw becomes a DNG (the copy is ours to replace)
let mut stored = stored;
if opts.convert_dng
&& mode == ImportMode::Copy
&& stored != path
&& info.kind == MediaKind::Raw
&& !info.format.eq_ignore_ascii_case("DNG")
{
// the templates see the photo as it will be catalogued (undated: the import time)
let own = Path::new(&path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let mut q = Photo::new(PhotoId(0), Source::File { path: path.clone() }, &own, &info.format, 0, 0, &now);
q.captured = info.captured.clone();
q.meta = info.meta.clone();
let name = opts.rename.as_deref().filter(|t| !t.trim().is_empty()).map(|t| {
let n = crate::rename::expand(t, &q, seq);
seq += 1;
n
match crate::cmd::convert::write_dng_with(self.file_bytes.as_ref(), &stored, String::new()) {
Ok(dng) => {
let _ = std::fs::remove_file(&stored);
stored = dng;
info.format = "DNG".into();
}
Err(e) => log::warn!("import {path}: copy as DNG: {e}"),
}
}
out.items.push(PreparedItem::Ready(Box::new(ReadyFile { path, stored, info, sidecar, placed })));
}
out
}
}
/// The catalog half of an import: add the photos [`ImportJob::prepare`] readied (one undoable op;
/// a Move's sources are removed once it is saved). On a failed commit a Move's placed files are
/// taken back.
pub fn commit_prepared(s: &mut Session, opts: &ImportOptions, now: &str, prepared: Prepared) -> crate::Result<ImportReport> {
let now = now.to_string();
let mut report = ImportReport { scanned: prepared.scanned, ..Default::default() };
let mut ops = Vec::new();
// Move: placed at the destination, sources untouched until the records are committed
let mut placed: Vec<crate::import_move::Placed> = Vec::new();
// photos added by this batch, by content (a duplicate of a file earlier in the import names it)
let mut new_hash: HashMap<String, PhotoId> = HashMap::new();
for it in prepared.items {
match it {
PreparedItem::Promote(id) => {
if s.catalog.photo(id).is_some_and(|p| p.local) {
ops.push(Op::SetLocal { id, local: false });
}
report.imported.push(id.0);
}
PreparedItem::Duplicate { path, existing, reason, hash } => {
let existing = existing.or_else(|| {
let h = hash?;
new_hash.get(&h).copied().or_else(|| s.catalog.photos().find(|p| p.content_hash.as_deref() == Some(h.as_str())).map(|p| p.id))
});
let folders = opts.organize.folders(&q);
if mode == ImportMode::Move && !crate::import_move::is_symlink(Path::new(&path)) {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
let name = name.unwrap_or(own);
match crate::import_move::place(Path::new(&path), &dir, &name) {
Ok(pl) => {
let dst = pl.dst.to_string_lossy().to_string();
placed.push(pl);
dst
}
Err(e) => {
report.failed.push((path, e));
continue;
}
}
} else {
if mode == ImportMode::Move {
let reason = "a symbolic link: the file it points to was copied, the link and its target stay";
report.kept.push(Kept { path: path.clone(), reason: reason.into() });
}
match copy_into(root, &path, &folders, name.as_deref()) {
Ok(p) => p,
Err(e) => {
report.failed.push((path, e));
continue;
}
report.duplicates.push(Duplicate { path, existing: existing.map(|i| i.0), reason });
}
PreparedItem::Failed(path, e) => report.failed.push((path, e)),
PreparedItem::Kept(k) => report.kept.push(k),
PreparedItem::Ready(r) => {
let ReadyFile { path, stored, info, sidecar, placed: pl } = *r;
placed.extend(pl);
let id = s.catalog.alloc_photo_id();
if let Some(h) = &info.content_hash {
new_hash.insert(h.clone(), id);
}
let sidecar = sidecar.map(|sc| sc.resolve_label(&s.catalog)).filter(|sc| *sc != crate::sidecar::SidecarData::default());
// a copy is catalogued under its new name
let name = Path::new(&stored).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| path.clone());
let mut p = Photo::new(id, Source::File { path: stored }, &name, &info.format, info.width, info.height, &now);
p.kind = info.kind;
p.file_size = info.file_size;
p.captured = info.captured;
p.meta = info.meta;
p.as_shot_wb = info.as_shot_wb;
p.content_hash = info.content_hash;
p.embedded_lens = info.embedded_lens;
p.preview_only = info.preview_only.clone();
p.fallback_matrix = info.fallback_matrix;
apply_import_defaults(s, &mut p);
if let Some(sc) = &sidecar {
crate::sidecar::merge_into(&mut p, sc, &now);
report.sidecars += 1;
}
if let Some(mp) = opts.metadata_preset.as_ref().and_then(|n| s.metadata_presets.iter().find(|m| m.name.eq_ignore_ascii_case(n))) {
crate::cmd::metadata::apply_to(&mut p.meta, &mp.fields);
}
for k in &opts.keywords {
let k = lightcraft_catalog::keywords::clean(k);
if !k.is_empty() && !p.meta.keywords.iter().any(|x| x.eq_ignore_ascii_case(&k)) {
p.meta.keywords.push(k);
}
}
}
(ImportMode::Move, _) => {
let reason = "already inside the destination or the library: added where it is";
report.kept.push(Kept { path: path.clone(), reason: reason.into() });
path.clone()
}
_ => path.clone(),
};
let id = s.catalog.alloc_photo_id();
if let Some(h) = &info.content_hash {
by_hash.insert(h.clone(), id);
}
// Copy as DNG: the copied raw becomes a DNG (the copy is ours to replace)
let mut stored = stored;
if opts.convert_dng && mode == ImportMode::Copy && stored != path && info.kind == MediaKind::Raw && !info.format.eq_ignore_ascii_case("DNG") {
match crate::cmd::convert::write_dng_for(s, &stored, String::new()) {
Ok(dng) => {
let _ = std::fs::remove_file(&stored);
stored = dng;
info.format = "DNG".into();
if let Some(preset) = &opts.preset {
let d = std::sync::Arc::new(preset.apply(&p.develop, 1.0));
let label = format!("Preset: {}", preset.name);
p.develop = d.clone();
p.edited = Some(now.clone());
p.history.push(lightcraft_catalog::HistoryStep { label, settings: d });
}
Err(e) => log::warn!("import {path}: copy as DNG: {e}"),
report.imported.push(id.0);
p.local = opts.local;
if opts.local {
// the state as browsed: a later change keeps the record from being forgotten
p.set_local_baseline();
}
ops.push(Op::AddPhoto { photo: Box::new(p) });
}
}
// a copy is catalogued under its new name
let name = Path::new(&stored).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| path.clone());
let mut p = Photo::new(id, Source::File { path: stored }, &name, &info.format, info.width, info.height, &now);
p.kind = info.kind;
p.file_size = info.file_size;
p.captured = info.captured;
p.meta = info.meta;
p.as_shot_wb = info.as_shot_wb;
p.content_hash = info.content_hash;
p.embedded_lens = info.embedded_lens;
p.preview_only = info.preview_only.clone();
apply_import_defaults(s, &mut p);
if let Some(sc) = &sidecar {
crate::sidecar::merge_into(&mut p, sc, &now);
report.sidecars += 1;
}
if let Some(mp) = opts.metadata_preset.as_ref().and_then(|n| s.metadata_presets.iter().find(|m| m.name.eq_ignore_ascii_case(n))) {
crate::cmd::metadata::apply_to(&mut p.meta, &mp.fields);
}
for k in &opts.keywords {
let k = lightcraft_catalog::keywords::clean(k);
if !k.is_empty() && !p.meta.keywords.iter().any(|x| x.eq_ignore_ascii_case(&k)) {
p.meta.keywords.push(k);
}
}
if let Some(preset) = &opts.preset {
let d = std::sync::Arc::new(preset.apply(&p.develop, 1.0));
let label = format!("Preset: {}", preset.name);
p.develop = d.clone();
p.edited = Some(now.clone());
p.history.push(lightcraft_catalog::HistoryStep { label, settings: d });
}
report.imported.push(id.0);
p.local = opts.local;
if opts.local {
// the state as browsed: a later change keeps the record from being forgotten
p.set_local_baseline();
}
ops.push(Op::AddPhoto { photo: Box::new(p) });
}
let log0 = s.pending_log.len();
if !ops.is_empty()
@@ -807,6 +987,7 @@ impl Session {
p.as_shot_wb = info.as_shot_wb;
p.embedded_lens = info.embedded_lens;
p.preview_only = info.preview_only.clone();
p.fallback_matrix = info.fallback_matrix;
}
p.develop = std::sync::Arc::new(p.import_defaults());
let edge = edge.clamp(64, crate::media::SourceLevel::Thumb.max_edge());
+55 -8
View File
@@ -169,6 +169,33 @@ pub(crate) fn place(src: &Path, dir: &Path, name: &str) -> Result<Placed, String
Ok(placed)
}
/// Import → **Copy**: copy `src` into `dir` as `name` (-1, -2… when taken) with the same checks as
/// a move's copy — a new file (never overwriting one, even one that appears meanwhile), synced to
/// disk and verified; a bad copy is removed and reported, so a card is never wiped on the strength
/// of it. Returns the new path.
///
/// `expect` is the content hash the import's probe computed from its full read of the source
/// ([`lightcraft_preview::hash_bytes`]): the copy is read back and checked against it, which spares
/// a third read of the card (issue #134). It also catches a source that changed since the probe
/// (whose recorded metadata and hash would be stale). Without one the copy is compared byte for
/// byte with the source.
pub(crate) fn copy_new(src: &Path, dir: &Path, name: &str, expect: Option<lightcraft_preview::Hash128>) -> Result<PathBuf, String> {
fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let (stem, ext) = match name.rsplit_once('.') {
Some((s, e)) if !s.is_empty() => (s.to_string(), format!(".{e}")),
_ => (name.to_string(), String::new()),
};
for i in 0..100_000u32 {
let d = if i == 0 { dir.join(name) } else { dir.join(format!("{stem}-{i}{ext}")) };
match copy_verified(src, &d, expect) {
Ok(()) => return Ok(d),
Err(e) if e.kind() == io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(format!("copy {}: {e} (nothing was imported from it; the original is untouched)", src.display())),
}
}
Err(format!("copy {}: no free name in {}", src.display(), dir.display()))
}
/// Undo a [`place`]: remove what it created at the destination (the sources were never touched).
pub(crate) fn rollback(p: &Placed) {
for sc in p.sidecars.iter().filter(|s| s.created) {
@@ -213,7 +240,7 @@ pub(crate) fn finish(p: &Placed) -> Result<Vec<(String, String)>, String> {
}
/// A photo (by extension) still beside a stem-named sidecar, sharing its stem.
fn sibling_photo(sidecar: &Path) -> Option<String> {
pub(crate) fn sibling_photo(sidecar: &Path) -> Option<String> {
let dir = sidecar.parent()?;
let stem = sidecar.file_stem()?.to_string_lossy().to_string();
fs::read_dir(dir).ok()?.flatten().map(|e| e.path()).find_map(|f| {
@@ -241,12 +268,14 @@ fn link_or_copy(src: &Path, dst: &Path) -> io::Result<()> {
Err(_) => {}
}
}
copy_verified(src, dst)
// Move keeps the byte-for-byte comparison: the source is deleted on the strength of it
copy_verified(src, dst, None)
}
/// Copy into a new file, sync it to disk and compare it with the original; on any failure the
/// partial copy is removed.
fn copy_verified(src: &Path, dst: &Path) -> io::Result<()> {
/// Copy into a new file, sync it to disk and verify it: against `expect` (the probe's hash of the
/// source) when given, else byte for byte against the source. On any failure the partial copy is
/// removed.
pub(crate) fn copy_verified(src: &Path, dst: &Path, expect: Option<lightcraft_preview::Hash128>) -> io::Result<()> {
let mut out = OpenOptions::new().write(true).create_new(true).open(dst)?;
let r = (|| {
let mut inp = File::open(src)?;
@@ -262,10 +291,14 @@ fn copy_verified(src: &Path, dst: &Path) -> io::Result<()> {
if let Ok(m) = fs::metadata(src).and_then(|m| m.modified()) {
let _ = out.set_modified(m);
}
if !same_bytes(src, dst)? {
return Err(io::Error::other("the copy differs from the original"));
match expect {
Some(h) if hash_file(dst)? != h => {
Err(io::Error::other("the copy differs from the original as it was read for the import (a bad copy, or the file changed meanwhile)"))
}
Some(_) => Ok(()),
None if !same_bytes(src, dst)? => Err(io::Error::other("the copy differs from the original")),
None => Ok(()),
}
Ok(())
})();
drop(out);
if r.is_err() {
@@ -274,6 +307,20 @@ fn copy_verified(src: &Path, dst: &Path) -> io::Result<()> {
r
}
/// [`lightcraft_preview::hash_bytes`] of a file's content, read in chunks.
fn hash_file(path: &Path) -> io::Result<lightcraft_preview::Hash128> {
let mut f = File::open(path)?;
let mut h = lightcraft_preview::Hasher128::new();
let mut buf = vec![0u8; 1 << 20];
loop {
let n = fill(&mut f, &mut buf)?;
if n == 0 {
return Ok(h.finish());
}
h.update(buf.get(..n).unwrap_or(&[]));
}
}
/// Do two files hold the same bytes?
pub(crate) fn same_bytes(a: &Path, b: &Path) -> io::Result<bool> {
if fs::metadata(a)?.len() != fs::metadata(b)?.len() {
+96 -16
View File
@@ -10,6 +10,8 @@
//! run them off the UI thread.
#![forbid(unsafe_code)]
pub mod availability;
mod camera_preview;
pub mod cmd;
pub mod crs;
pub mod crs_masks;
@@ -57,6 +59,9 @@ pub enum EngineError {
/// be written. They stay queued and are written by the next successful save.
#[error("saved in memory but not written to disk: {0}; LightCraft will retry")]
NotSaved(String),
/// Another process (the app, `lightcraft-cli`, another computer) has the library open.
#[error("{0}")]
LibraryInUse(String),
#[error("{0}")]
Other(String),
}
@@ -68,6 +73,22 @@ pub type Result<T> = std::result::Result<T, EngineError>;
pub struct UndoEntry {
pub label: String,
pub op: Op,
/// A folder to rename on disk (`from` → `to`) before `op` is applied: Rename / Move Folder
/// (whose `op` relinks the photos inside). Never overwrites; refused when `to` exists.
pub folder: Option<FolderMove>,
}
/// A folder renamed or moved on disk as part of an undo step.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FolderMove {
pub from: String,
pub to: String,
}
impl FolderMove {
fn reversed(&self) -> Self {
Self { from: self.to.clone(), to: self.from.clone() }
}
}
/// An in-progress slider drag / brush stroke: one undo step when it ends.
@@ -253,11 +274,22 @@ impl Session {
(spec.enabled)(self).map_err(|why| EngineError::Disabled(id.to_string(), why))?;
let empty = Value::Object(Default::default());
let params = if params.is_null() { &empty } else { params };
self.run_command(id, spec.journal.then_some(params), |s| (spec.run)(s, params))
}
/// Run `f` as command `id` with what [`Session::execute`] does around every command: the
/// panic guard, auto versions, XMP sidecar auto-write and the durable save (a failed save is
/// `NotSaved`). Not journaled. The app's import task commits its batches this way.
pub fn execute_fn(&mut self, id: &str, f: impl FnOnce(&mut Session) -> Result<Value>) -> Result<Value> {
self.run_command(id, None, f)
}
fn run_command(&mut self, id: &str, journal: Option<&Value>, f: impl FnOnce(&mut Session) -> Result<Value>) -> Result<Value> {
let log_start = self.pending_log.len();
let was_active = self.active();
self.depth += 1;
// last-resort guard: a panic in a command is that command's error, not a crash
let r = guard::catch(&format!("`{id}`"), || (spec.run)(self, params)).unwrap_or_else(|e| Err(EngineError::Other(e)));
let r = guard::catch(&format!("`{id}`"), || f(self)).unwrap_or_else(|e| Err(EngineError::Other(e)));
self.depth -= 1;
if self.depth == 0 && was_active.is_some() && self.active() != was_active {
self.previous_active = was_active;
@@ -265,7 +297,10 @@ impl Session {
self.auto_version(left);
}
}
if r.is_ok() && spec.journal && self.depth == 0 {
if r.is_ok()
&& let Some(params) = journal
&& self.depth == 0
{
self.journal.push((id.to_string(), params.clone()));
if self.journal.len() > 10_000 {
self.journal.drain(..1000);
@@ -302,7 +337,7 @@ impl Session {
let fwd = op.clone();
let inv = self.catalog.apply(op)?;
self.pending_log.push(fwd);
self.undo.push(UndoEntry { label: label.to_string(), op: inv });
self.undo.push(UndoEntry { label: label.to_string(), op: inv, folder: None });
if self.undo.len() > 1000 {
self.undo.remove(0);
}
@@ -310,6 +345,16 @@ impl Session {
Ok(())
}
/// [`Session::commit`] for an op that goes with a folder already renamed on disk; the undo
/// step renames it back (`folder` is the undo direction: current place → old place).
pub(crate) fn commit_with_folder(&mut self, label: &str, op: Op, folder: FolderMove) -> Result<()> {
self.commit(label, op)?;
if let Some(e) = self.undo.last_mut() {
e.folder = Some(folder);
}
Ok(())
}
/// Leaving photo `id` after editing it: keep its settings as an automatic version (when they
/// differ from its latest version; at most [`AUTO_VERSIONS`] auto versions, oldest dropped).
/// Saved with the library but not an undo step.
@@ -340,9 +385,13 @@ impl Session {
if n < 2 || n > self.undo.len() {
return;
}
// a step that moves a folder on disk stays on its own
if self.undo[self.undo.len() - n..].iter().any(|e| e.folder.is_some()) {
return;
}
let tail = self.undo.split_off(self.undo.len() - n);
let ops = tail.into_iter().rev().map(|e| e.op).collect();
self.undo.push(UndoEntry { label: label.to_string(), op: Op::Batch { ops } });
self.undo.push(UndoEntry { label: label.to_string(), op: Op::Batch { ops }, folder: None });
}
/// Apply without recording undo (interactive previews).
@@ -353,7 +402,7 @@ impl Session {
pub fn undo_step(&mut self) -> Result<String> {
let e = self.undo.pop().ok_or_else(|| EngineError::Other("nothing to undo".into()))?;
let redo = match self.apply_with_files(&e.op) {
let redo = match self.apply_with_files(&e.op, e.folder.as_ref()) {
Ok(r) => r,
Err(err) => {
self.undo.push(e);
@@ -361,13 +410,13 @@ impl Session {
}
};
self.pending_log.push(e.op);
self.redo.push(UndoEntry { label: e.label.clone(), op: redo });
self.redo.push(UndoEntry { label: e.label.clone(), op: redo, folder: e.folder.as_ref().map(FolderMove::reversed) });
Ok(e.label)
}
pub fn redo_step(&mut self) -> Result<String> {
let e = self.redo.pop().ok_or_else(|| EngineError::Other("nothing to redo".into()))?;
let undo = match self.apply_with_files(&e.op) {
let undo = match self.apply_with_files(&e.op, e.folder.as_ref()) {
Ok(r) => r,
Err(err) => {
self.redo.push(e);
@@ -375,19 +424,44 @@ impl Session {
}
};
self.pending_log.push(e.op);
self.undo.push(UndoEntry { label: e.label.clone(), op: undo });
self.undo.push(UndoEntry { label: e.label.clone(), op: undo, folder: e.folder.as_ref().map(FolderMove::reversed) });
Ok(e.label)
}
/// Apply an undo/redo op, first moving the files its renames imply (all or nothing).
fn apply_with_files(&mut self, op: &Op) -> Result<Op> {
/// Apply an undo/redo op, first renaming the step's folder and moving the files its renames
/// imply (all or nothing). Files that can't be moved back after a failure are reported and the
/// library follows them (a logged change outside the undo history), so none goes missing; the
/// folder is only moved back when no file was left behind at its new path.
fn apply_with_files(&mut self, op: &Op, folder: Option<&FolderMove>) -> Result<Op> {
let fs = rename::RealFs;
if let Some(f) = folder {
cmd::browse::rename_folder_on_disk(&f.from, &f.to).map_err(|e| EngineError::Other(format!("can't move the folder back: {e}")))?;
}
let undo_folder = || {
if let Some(f) = folder {
let _ = cmd::browse::rename_folder_on_disk(&f.to, &f.from);
}
};
let moves = self.file_moves(op);
Session::move_files(&moves)?;
if let Err(e) = rename::move_all(&fs, &moves) {
if e.stuck.is_empty() {
undo_folder();
}
return Err(EngineError::Other(format!("can't move the files back: {}", self.follow_stuck(op, e, false))));
}
match self.catalog.apply(op.clone()) {
Ok(inv) => Ok(inv),
Ok(inv) => {
if let Some(f) = folder {
cmd::browse::follow_folder(self, &f.from, &f.to);
}
Ok(inv)
}
Err(e) => {
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
let _ = Session::move_files(&back);
if let Err(be) = rename::move_all(&fs, &back) {
return Err(EngineError::Other(format!("{e}; the files could not all be moved back: {}", be.message)));
}
undo_folder();
Err(e.into())
}
}
@@ -502,7 +576,11 @@ impl Session {
/// Photos shown in the grid/filmstrip for the current source, filter and sort.
pub fn visible(&mut self) -> &[PhotoId] {
let key = (self.catalog.revision, format!("{:?}|{:?}|{:?}|{:?}", self.source, self.filter, self.sort, self.browse));
let mut key = (self.catalog.revision, format!("{:?}|{:?}|{:?}|{:?}", self.source, self.filter, self.sort, self.browse));
if self.source == LibrarySource::Missing && self.media.availability.is_background() {
// the view fills in as the background checks find files gone
key.1.push_str(&format!("|{}", self.media.availability.generation()));
}
if self.visible_key.as_ref() != Some(&key) {
// "in the last N days" rules count back from the session's clock
lightcraft_catalog::rules::set_now(Some((self.clock)()));
@@ -537,8 +615,8 @@ impl Session {
}
if self.source == LibrarySource::Missing {
// only the photos the query kept (library photos, not Local browse records) are checked
let cat = &self.catalog;
visible.retain(|id| cmd::missing::is_missing(cat, *id));
let (cat, avail) = (&self.catalog, &self.media.availability);
visible.retain(|id| cmd::missing::is_missing(cat, avail, *id));
}
if self.source != LibrarySource::RecentlyDeleted {
visible = self.catalog.arrange_stacks(&visible);
@@ -635,6 +713,8 @@ mod tests_persist;
#[cfg(test)]
mod tests_prefs;
#[cfg(test)]
mod tests_settings_files;
#[cfg(test)]
mod tests_spots;
#[cfg(test)]
mod tests_xmp;
+149 -17
View File
@@ -23,7 +23,7 @@
use std::path::{Path, PathBuf};
use lightcraft_catalog::{FsStore, Journal, LoadReport, Store};
use lightcraft_catalog::{FsStore, Journal, LibraryLock, LoadReport, Store};
use lightcraft_develop::Preset;
use serde::{Deserialize, Serialize};
@@ -64,6 +64,15 @@ pub struct Library {
pub forgot_local: Option<lightcraft_catalog::ForgetPlan>,
presets_written: String,
view_written: Vec<u8>,
/// Settings files that were unreadable or damaged when the library opened (`library.info` →
/// `settingsWarnings`; shown by the UI once, see [`Session::take_library_warnings`]).
pub settings_warnings: Vec<String>,
warnings_reported: usize,
/// Settings files that couldn't be read or set aside: never overwritten this session.
blocked: Vec<&'static str>,
/// Keeps other processes out of this library while it's open (last: released after the
/// journal's background snapshot has landed).
lock: Option<LibraryLock>,
}
/// Where a library's files live (see [`Session::open_library_in`]).
@@ -171,23 +180,112 @@ fn poll_compaction(lib: &mut Library) {
/// How long the frame loop waits before retrying a failed append (commands retry at once).
const RETRY_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2);
fn read_json<T: serde::de::DeserializeOwned>(store: &mut dyn Store, name: &str) -> Option<T> {
store.read(name).ok().flatten().and_then(|b| serde_json::from_slice(&b).ok())
/// `a` and `b` name the same directory.
fn same_dir(a: &Path, b: &Path) -> bool {
match (std::fs::canonicalize(a), std::fs::canonicalize(b)) {
(Ok(a), Ok(b)) => a == b,
_ => a == b,
}
}
/// This program, for the lock owner note ("LightCraft", "lightcraft-cli").
fn program_name() -> String {
let exe = std::env::current_exe().ok().and_then(|p| p.file_stem().map(|s| s.to_string_lossy().to_string()));
match exe.as_deref() {
Some("lightcraft") | None => "LightCraft".into(),
Some(other) => other.to_string(),
}
}
/// Problems reading the settings files (`presets.json`, `prefs.json`, `view.json`) when the library
/// opened (issue #103). A missing file is not a problem (defaults). A file that doesn't parse is
/// kept as `<name>.corrupt-<unix time>` before the defaults are used (so the next save can't
/// lose it); a file that can't be read at all (locked by another program, I/O error) is never
/// written this session, so its content survives until the library is opened again.
#[derive(Default)]
struct SettingsLoad {
warnings: Vec<String>,
/// Files not to overwrite this session.
blocked: Vec<&'static str>,
}
impl SettingsLoad {
fn read<T: serde::de::DeserializeOwned>(&mut self, store: &mut dyn Store, name: &'static str) -> Option<T> {
let bytes = match store.read(name) {
Ok(b) => b?,
Err(e) => {
log::error!("library: {name}: {e}");
self.blocked.push(name);
self.warnings.push(format!(
"{name} couldn't be read ({e}). LightCraft uses the defaults for now and won't overwrite the file; reopen the library to try again."
));
return None;
}
};
let err = match serde_json::from_slice(&bytes) {
Ok(v) => return Some(v),
Err(e) => e,
};
let secs = web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let keep = format!("{name}.corrupt-{secs}");
log::error!("library: {name} is damaged: {err}");
match store.write_atomic(&keep, &bytes) {
Ok(()) => self.warnings.push(format!("{name} is damaged ({err}). It was kept as {keep}, and the defaults are used.")),
Err(w) => {
self.blocked.push(name);
self.warnings.push(format!(
"{name} is damaged ({err}) and couldn't be set aside ({w}). LightCraft uses the defaults and won't overwrite the file."
));
}
}
None
}
}
impl Session {
/// Open (or create) the library at `dir` into this session, replacing its catalog. With
/// `seed_demo`, a newly created library starts with the procedural demo photos.
///
/// The library is locked for this session ([`lightcraft_catalog::lock`]): if another process
/// has it open, this fails with [`EngineError::LibraryInUse`] and nothing is read or changed.
/// Reopening the library this session already has open keeps its lock.
pub fn open_library(&mut self, dir: impl AsRef<Path>, seed_demo: bool) -> Result<&LoadReport> {
let dir = dir.as_ref().to_path_buf();
let open = || FsStore::open(&dir).map_err(|e| EngineError::Other(format!("can't open library {}: {e}", dir.display())));
let stores = LibraryStores { catalog: Box::new(open()?), files: Box::new(open()?), on_disk: true, dir: dir.clone() };
self.open_library_in(stores, seed_demo)
// the library open in this session (same directory): hand its lock over
let reused = self.library.as_mut().filter(|l| same_dir(&l.dir, &dir)).and_then(|l| l.lock.take());
let reusing = reused.is_some();
let mut lock = match reused {
Some(l) => Some(l),
None => Some(LibraryLock::acquire(&dir, &program_name()).map_err(|e| EngineError::LibraryInUse(e.to_string()))?),
};
if let Err(e) = self.open_stores(stores, seed_demo, &mut lock) {
if reusing && let Some(lib) = self.library.as_mut() {
lib.lock = lock.take();
}
return Err(e);
}
self.loaded_report()
}
/// Open (or create) a library whose files live in `stores` (e.g. browser storage), replacing
/// this session's catalog.
/// this session's catalog. Not locked: the host guards against a second opener (see
/// [`Session::open_library`] for directories).
pub fn open_library_in(&mut self, stores: LibraryStores, seed_demo: bool) -> Result<&LoadReport> {
self.open_stores(stores, seed_demo, &mut None)?;
self.loaded_report()
}
fn loaded_report(&self) -> Result<&LoadReport> {
match &self.library {
Some(lib) => Ok(&lib.report),
None => Err(EngineError::Other("library closed while opening".into())),
}
}
/// Open `stores`; the new library takes `lock` once nothing can fail any more.
fn open_stores(&mut self, stores: LibraryStores, seed_demo: bool, lock: &mut Option<LibraryLock>) -> Result<()> {
let LibraryStores { dir, catalog, mut files, on_disk } = stores;
self.media.smart_dir = on_disk.then(|| crate::smart::dir(&dir));
// the current library may be these same files: let its background snapshot land first
@@ -208,8 +306,9 @@ impl Session {
crate::demo::load(self);
journal.snapshot(&self.catalog)?;
}
let mut settings = SettingsLoad::default();
// presets
if let Some(f) = read_json::<PresetsFile>(files.as_mut(), "presets.json") {
if let Some(f) = settings.read::<PresetsFile>(files.as_mut(), "presets.json") {
for p in &mut self.presets {
p.favorite = p.builtin && f.favorites.contains(&p.id) || (!p.builtin && p.favorite);
}
@@ -223,7 +322,7 @@ impl Session {
self.profile_recent = f.profile_recent.into_iter().filter(known).take(crate::presets::RECENT_PROFILES).collect();
}
// preferences
let prefs = read_json::<PrefsFile>(files.as_mut(), "prefs.json").unwrap_or_default();
let prefs = settings.read::<PrefsFile>(files.as_mut(), "prefs.json").unwrap_or_default();
self.xmp = prefs.xmp;
self.last_export = prefs.last_export;
self.export_presets = prefs.export_presets;
@@ -244,7 +343,7 @@ impl Session {
self.media.smart_dir = Some(d.clone());
}
// view state
if let Some(v) = read_json::<ViewFile>(files.as_mut(), "view.json") {
if let Some(v) = settings.read::<ViewFile>(files.as_mut(), "view.json") {
self.source = v.source;
self.browse = v.browse;
self.filter = v.filter;
@@ -277,6 +376,10 @@ impl Session {
forgot_local: None,
presets_written,
view_written,
settings_warnings: settings.warnings,
warnings_reported: 0,
blocked: settings.blocked,
lock: lock.take(),
});
// forget untouched Local records of folders not browsed for a while (journaled at once)
if self.forget_local_days > 0 {
@@ -290,10 +393,7 @@ impl Session {
lib.forgot_local = Some(plan);
}
}
match &self.library {
Some(lib) => Ok(&lib.report),
None => Err(EngineError::Other("library closed while opening".into())),
}
Ok(())
}
/// Write pending ops to the log (fsynced), compact when due, and save changed presets.
@@ -332,7 +432,7 @@ impl Session {
}
let presets = presets_json(self);
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if presets != lib.presets_written {
if presets != lib.presets_written && !lib.blocked.contains(&"presets.json") {
if let Err(e) = lib.files.write_atomic("presets.json", presets.as_bytes()) {
log::error!("library: presets: {e}");
} else {
@@ -362,16 +462,35 @@ impl Session {
}
/// Flush everything and write a snapshot (on quit). Ends an open interaction first.
///
/// The snapshot is written even when appending the queued ops fails: it is a fresh file
/// holding everything in memory, so it saves those ops too (the log handle may be the only
/// thing that's broken). Fails only if nothing could be saved; the ops then stay queued.
pub fn close_library(&mut self) -> Result<()> {
if self.library.is_none() {
return Ok(());
}
let _ = self.end_interaction();
self.persist()?;
let persisted = self.persist();
self.save_view();
let unlogged = if persisted.is_err() { self.pending_log.len() as u64 } else { 0 };
let Some(lib) = self.library.as_mut() else { return Ok(()) };
lib.journal.snapshot(&self.catalog)?;
Ok(())
let before = lib.journal.seq();
let snapshot = lib.journal.snapshot_with_unlogged(&self.catalog, unlogged);
if unlogged > 0 && lib.journal.seq() == before + unlogged {
// the snapshot holds the queued ops: they are saved, never append them again
log::info!("library: {unlogged} queued change(s) saved in the closing snapshot");
self.pending_log.clear();
lib.unsaved_error = None;
lib.retry_at = None;
lib.last_error = None;
}
match (persisted, snapshot) {
(_, Ok(())) => Ok(()),
// neither the log nor the snapshot took the queued ops
(Err(EngineError::NotSaved(e)), Err(s)) if lib.journal.seq() == before => Err(EngineError::NotSaved(format!("{e}; snapshot: {s}"))),
(_, Err(s)) => Err(s.into()),
}
}
fn view_json(&self) -> Vec<u8> {
@@ -394,7 +513,7 @@ impl Session {
}
let v = self.view_json();
let Some(lib) = self.library.as_mut() else { return };
if v != lib.view_written {
if v != lib.view_written && !lib.blocked.contains(&"view.json") {
match lib.files.write_atomic("view.json", &v) {
Ok(()) => lib.view_written = v,
Err(e) => log::error!("library: view: {e}"),
@@ -423,9 +542,22 @@ impl Session {
})
.unwrap_or_default();
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if lib.blocked.contains(&"prefs.json") {
return Err(EngineError::Other(
"prefs: prefs.json couldn't be read when the library opened, so it isn't overwritten; reopen the library to save preferences".into(),
));
}
lib.files.write_atomic("prefs.json", &v).map_err(|e| EngineError::Other(format!("prefs: {e}")))
}
/// Settings-file warnings of the open library not handed out yet (the UI shows each once).
pub fn take_library_warnings(&mut self) -> Vec<String> {
let Some(lib) = self.library.as_mut() else { return vec![] };
let new = lib.settings_warnings[lib.warnings_reported..].to_vec();
lib.warnings_reported = lib.settings_warnings.len();
new
}
/// The thumbnail disk cache budget in bytes ([`Session::cache_mb`], else the default).
pub fn cache_bytes(&self) -> u64 {
if self.cache_mb == 0 { crate::media::DISK_CACHE_BYTES } else { u64::from(self.cache_mb) << 20 }
+200 -43
View File
@@ -25,8 +25,9 @@ use lightcraft_preview::{Hash128, Hasher128, Lru, PreviewCache};
use lightcraft_raster::{Histogram, Rgb32f, Rgba8};
use serde::{Deserialize, Serialize};
/// Bump when the pipeline's output changes, to invalidate cached thumbnails.
pub const RENDER_CACHE_VERSION: u64 = 4;
/// Bump when the pipeline's output changes, to invalidate cached thumbnails. A new camera-look fit
/// ([`crate::camera_preview::FIT_VERSION`]) invalidates them by itself.
pub const RENDER_CACHE_VERSION: u64 = 8 | (crate::camera_preview::FIT_VERSION << 32);
/// Thumbnails render at one of these long edges (so window/cell size changes reuse the cache).
pub const THUMB_SIZES: [usize; 4] = [128, 256, 384, 512];
@@ -71,9 +72,40 @@ pub type FileLoader = Arc<dyn Fn(&str, usize) -> Result<(Rgb32f, SourceInfo), St
/// `max_edge` (set by the app). `None`: no usable preview.
pub type PreviewLoader = Arc<dyn Fn(&str, usize) -> Option<Rgba8> + Send + Sync>;
/// Pixels and the source interpretation learned while decoding; they must be cached together.
#[derive(Clone)]
pub struct DecodedSource {
pub image: Arc<Rgb32f>,
pub info: Option<SourceInfo>,
/// A smart preview's stored camera tone curve: the one decoder fact its pixels need that the
/// catalog's header facts lack (used when `info` is `None`).
pub camera_tone: Option<lightcraft_pipeline::tone::CameraTone>,
}
impl DecodedSource {
pub fn new(image: Arc<Rgb32f>, info: Option<SourceInfo>) -> Self {
DecodedSource { image, info, camera_tone: None }
}
/// What to render these pixels against: the decoder's facts, else `header` (the catalog's)
/// with any stored camera tone curve. For a catalogued raw (`header.raw`) the catalog keeps
/// deciding how white balance settings are read — relative or absolute, around which as-shot
/// white — because the photo's saved settings were made under that rule
/// ([`lightcraft_catalog::Photo::relative_wb`]).
pub fn info_or(&self, header: SourceInfo) -> SourceInfo {
match self.info {
Some(info) if info.raw && header.raw => {
SourceInfo { relative_wb: header.relative_wb, as_shot_temp: header.as_shot_temp, as_shot_tint: header.as_shot_tint, ..info }
}
Some(info) => info,
None => SourceInfo { camera_tone: self.camera_tone.or(header.camera_tone), ..header },
}
}
}
#[derive(Clone)]
pub enum SourceRef {
Loaded(Arc<Rgb32f>),
Loaded(Box<DecodedSource>),
Demo {
scene: Box<lightcraft_scenes::Scene>,
max_edge: usize,
@@ -82,6 +114,9 @@ pub enum SourceRef {
path: String,
max_edge: usize,
loader: Option<FileLoader>,
/// The photo's smart preview, used when the original can't be read (offline drive). The
/// UI thread never checks whether the original is there; the render worker finds out.
fallback: Option<std::path::PathBuf>,
},
/// A smart preview standing in for a missing original.
Smart {
@@ -91,28 +126,41 @@ pub enum SourceRef {
impl SourceRef {
pub fn load(&self) -> Result<Arc<Rgb32f>, String> {
match self {
SourceRef::Loaded(a) => Ok(a.clone()),
SourceRef::Demo { scene, max_edge } => Ok(Arc::new(scene.render_fit(*max_edge))),
SourceRef::File { path, max_edge, loader } => match loader {
Some(l) => l(path, *max_edge).map(|(img, _)| Arc::new(img)),
None => Err(format!("no decoder available for {path}")),
},
self.load_source().map(|s| s.image)
}
pub fn load_source(&self) -> Result<DecodedSource, String> {
let image = match self {
SourceRef::Loaded(a) => return Ok((**a).clone()),
SourceRef::Demo { scene, max_edge } => Arc::new(scene.render_fit(*max_edge)),
SourceRef::File { path, max_edge, loader, fallback } => {
let r = match loader {
Some(l) => l(path, *max_edge).map(|(image, info)| DecodedSource::new(Arc::new(image), Some(info))),
None => Err(format!("no decoder available for {path}")),
};
// An offline original renders from its smart preview (no decoder facts: header ones).
return match (r, fallback) {
#[cfg(not(target_arch = "wasm32"))]
(Err(e), Some(sp)) if crate::smart::is_valid(sp) => crate::smart::load(sp).map_err(|e2| format!("{e}; smart preview: {e2}")),
(r, _) => r,
};
}
#[cfg(not(target_arch = "wasm32"))]
SourceRef::Smart { path } => crate::smart::load(path),
SourceRef::Smart { path } => return crate::smart::load(path),
#[cfg(target_arch = "wasm32")]
SourceRef::Smart { .. } => Err("smart previews are not available here".into()),
}
SourceRef::Smart { .. } => return Err("smart previews are not available here".into()),
};
Ok(DecodedSource::new(image, None))
}
}
pub struct MediaCache {
/// Decoded thumbnail-level sources (LRU by bytes).
thumbs: Lru<PhotoId, Arc<Rgb32f>>,
thumbs: Lru<PhotoId, DecodedSource>,
/// Decoded preview-level sources with their last use ([`lightcraft_preview::next_tick`]).
previews: Vec<(PhotoId, Arc<Rgb32f>, u64)>,
previews: Vec<(PhotoId, DecodedSource, u64)>,
/// The last full-resolution original (exports; one at a time: ~300 MB at 24 MP).
full: Option<(PhotoId, Arc<Rgb32f>, u64)>,
full: Option<(PhotoId, DecodedSource, u64)>,
/// How many previews to keep (LRU).
pub preview_capacity: usize,
/// Bytes all decoded sources and rendered previews in memory may take together (the cache
@@ -125,6 +173,9 @@ pub struct MediaCache {
pub file_bytes: Option<crate::merge::ByteReader>,
/// The library's smart previews folder (originals offline: render from the proxy).
pub smart_dir: Option<std::path::PathBuf>,
/// Whether originals (and smart previews) are on disk: cached and checked off the UI thread
/// in the app (see [`crate::availability`]).
pub availability: crate::availability::Availability,
scenes: Vec<lightcraft_scenes::Scene>,
/// Rendered thumbnails (memory, plus disk once a library is attached).
pub rendered: Arc<PreviewCache>,
@@ -144,6 +195,7 @@ impl Default for MediaCache {
preview_loader: None,
file_bytes: None,
smart_dir: None,
availability: Default::default(),
scenes: Vec::new(),
rendered: Arc::new(PreviewCache::memory(rendered_budget(budget))),
}
@@ -156,7 +208,7 @@ fn rendered_budget(share: usize) -> usize {
}
fn source_bytes(img: &Rgb32f) -> usize {
img.data.len() * 12 + 64
img.data.len() * 12 + 64 + std::mem::size_of::<SourceInfo>()
}
impl MediaCache {
@@ -207,8 +259,8 @@ impl MediaCache {
let Some((tick, which)) = candidates.into_iter().flatten().min() else { break };
let freed = match which {
0 => self.thumbs.pop_oldest(),
1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1)),
2 => self.full.take().map(|e| source_bytes(&e.1)),
1 => self.previews.iter().position(|e| e.2 == tick).map(|i| source_bytes(&self.previews.remove(i).1.image)),
2 => self.full.take().map(|e| source_bytes(&e.1.image)),
_ => self.rendered.evict_oldest(),
};
match freed {
@@ -219,6 +271,10 @@ impl MediaCache {
}
pub fn get(&mut self, id: PhotoId, level: SourceLevel) -> Option<Arc<Rgb32f>> {
self.get_source(id, level).map(|s| s.image)
}
fn get_source(&mut self, id: PhotoId, level: SourceLevel) -> Option<DecodedSource> {
match level {
SourceLevel::Thumb => self.thumbs.get(&id).cloned(),
SourceLevel::Preview => self.previews.iter_mut().find(|e| e.0 == id).map(|e| {
@@ -232,11 +288,31 @@ impl MediaCache {
}
}
/// The facts of photo `id`'s cached sources: the decoder's when one has them, else `header`
/// (with a smart preview's stored camera tone curve).
fn source_facts(&self, id: PhotoId, header: SourceInfo) -> SourceInfo {
let cached = || {
self.thumbs
.peek(&id)
.into_iter()
.chain(self.previews.iter().filter(|e| e.0 == id).map(|e| &e.1))
.chain(self.full.iter().filter(|e| e.0 == id).map(|e| &e.1))
};
match cached().find(|s| s.info.is_some()).or_else(|| cached().next()) {
Some(s) => s.info_or(header),
None => header,
}
}
pub fn insert(&mut self, id: PhotoId, level: SourceLevel, img: Arc<Rgb32f>) {
self.insert_source(id, level, DecodedSource::new(img, None));
}
fn insert_source(&mut self, id: PhotoId, level: SourceLevel, img: DecodedSource) {
let tick = lightcraft_preview::next_tick();
match level {
SourceLevel::Thumb => {
let cost = source_bytes(&img);
let cost = source_bytes(&img.image);
self.thumbs.insert(id, img, cost);
}
SourceLevel::Preview => {
@@ -270,29 +346,36 @@ impl MediaCache {
/// Decoded sources held: (thumbnail level, preview level, full size).
pub fn usage(&self) -> (crate::memory::Usage, crate::memory::Usage, crate::memory::Usage) {
use crate::memory::Usage;
let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1)).sum());
let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1))).unwrap_or_default();
let previews = Usage::new(self.previews.len(), self.previews.iter().map(|e| source_bytes(&e.1.image)).sum());
let full = self.full.as_ref().map(|e| Usage::new(1, source_bytes(&e.1.image))).unwrap_or_default();
(Usage::new(self.thumbs.len(), self.thumbs.cost()), previews, full)
}
pub fn source_ref(&mut self, p: &Photo, level: SourceLevel) -> SourceRef {
if let Some(a) = self.get(p.id, level) {
return SourceRef::Loaded(a);
if let Some(a) = self.get_source(p.id, level) {
return SourceRef::Loaded(Box::new(a));
}
// Procedural scenes have a nominal size: "full" is that size, not unbounded.
let max_edge = level.max_edge().min(match (&p.source, level) {
(Source::Demo { .. }, SourceLevel::Full) => p.width.max(p.height).max(1) as usize,
_ => usize::MAX,
});
// the original is offline: its smart preview, when there is one
// the original is offline: its smart preview, when there is one. Only cached answers are
// used here (no file-system call on the UI thread in the app); an original not known to
// be offline is tried first and its smart preview is the render worker's fallback.
#[cfg(not(target_arch = "wasm32"))]
if let (Source::File { path }, Some(dir)) = (&p.source, &self.smart_dir)
&& !std::path::Path::new(path).exists()
{
if let (Source::File { path }, Some(dir)) = (&p.source, &self.smart_dir) {
let sp = dir.join(crate::smart::file_name(p));
if sp.exists() {
// (a proxy cut short fails to load, like the offline original; the worker's fallback
// in `SourceRef::load` only uses a proxy that passes `smart::is_valid`)
if self.availability.is_offline(path) && self.availability.exists(&sp.to_string_lossy()) == Some(true) {
return SourceRef::Smart { path: sp };
}
let mut r = self.origin_ref(&p.source, max_edge);
if let SourceRef::File { fallback, .. } = &mut r {
*fallback = Some(sp);
}
return r;
}
self.origin_ref(&p.source, max_edge)
}
@@ -307,10 +390,10 @@ impl MediaCache {
}
match self.scenes.iter().find(|s| s.id == *scene) {
Some(s) => SourceRef::Demo { scene: Box::new(s.clone()), max_edge },
None => SourceRef::File { path: format!("demo:{scene}"), max_edge, loader: None },
None => SourceRef::File { path: format!("demo:{scene}"), max_edge, loader: None, fallback: None },
}
}
Source::File { path } => SourceRef::File { path: path.clone(), max_edge, loader: self.file_loader.clone() },
Source::File { path } => SourceRef::File { path: path.clone(), max_edge, loader: self.file_loader.clone(), fallback: None },
}
}
}
@@ -344,7 +427,7 @@ pub struct RenderResult {
pub key: u64,
pub rendered: Result<Rendered, String>,
/// A source that was loaded by this job (to be inserted into the cache).
pub loaded: Option<Arc<Rgb32f>>,
pub loaded: Option<DecodedSource>,
/// Set for a [`QuickJob`]'s stand-in: where the image came from.
pub quick: Option<QuickSource>,
}
@@ -410,12 +493,14 @@ impl RenderJob {
};
}
let was_loaded = matches!(self.source, SourceRef::Loaded(_));
match self.source.load() {
Ok(src) => {
match self.source.load_source() {
Ok(source) => {
let src = &source.image;
let info = source.info_or(self.info);
// Thumbnails (many small jobs side by side) stay on the CPU; views and exports use
// the GPU when there is one.
let gpu = self.cache.is_none();
let rendered = develop(&src, &self.info, &self.settings, &self.request, self.stages.as_deref(), gpu);
let rendered = develop(src, &info, &self.settings, &self.request, self.stages.as_deref(), gpu);
if let Some((cache, key)) = &self.cache {
cache.put(*key, Arc::new(rendered.image.clone()));
}
@@ -429,7 +514,7 @@ impl RenderJob {
level: self.level,
key: self.key,
rendered: Ok(rendered),
loaded: (!was_loaded).then_some(src),
loaded: (!was_loaded).then_some(source),
quick: None,
}
}
@@ -521,11 +606,16 @@ pub fn content_key(p: &Photo) -> String {
pub fn source_info(p: &Photo) -> SourceInfo {
// Procedural demo scenes are scene-referred HDR (like raw files): use the filmic tone map.
if matches!(p.source, Source::Demo { .. }) {
return SourceInfo { raw: true, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None };
return SourceInfo { raw: true, ..Default::default() };
}
// A raw shown from its embedded preview is a rendered (display-referred) JPEG: relative white
// balance and the display tone curve, like any other rendered file.
if p.develops_raw() { SourceInfo { raw: true, as_shot_temp: 5500.0, as_shot_tint: 0.0, lens: p.embedded_lens } } else { SourceInfo::default() }
if p.develops_raw() {
let (temp, tint) = if p.relative_wb() { (6500.0, 0.0) } else { p.as_shot_wb.unwrap_or((5500.0, 0.0)) };
SourceInfo { raw: true, as_shot_temp: temp, as_shot_tint: tint, lens: p.embedded_lens, relative_wb: p.relative_wb(), ..Default::default() }
} else {
SourceInfo::default()
}
}
impl crate::Session {
@@ -680,7 +770,7 @@ impl crate::Session {
/// Accept a finished job's loaded source into the cache.
pub fn accept(&mut self, r: &RenderResult) {
if let Some(src) = &r.loaded {
self.media.insert(r.photo, r.level, src.clone());
self.media.insert_source(r.photo, r.level, src.clone());
}
}
@@ -723,12 +813,19 @@ impl crate::Session {
Ok(job)
}
/// Prefer decoder facts to header-only metadata for pixel-statistics commands.
pub fn source_info(&self, id: PhotoId) -> SourceInfo {
let header = self.catalog.photo(id).map(|p| source_info(p)).unwrap_or_default();
self.media.source_facts(id, header)
}
/// The source proxy for pixel-statistics commands (auto tone/WB), loading synchronously.
pub fn source_now(&mut self, id: PhotoId, level: SourceLevel) -> Result<Arc<Rgb32f>, String> {
let p = self.catalog.photo(id).ok_or("no such photo")?.clone();
let r = self.media.source_ref(&p, level).load()?;
self.media.insert(id, level, r.clone());
Ok(r)
let r = self.media.source_ref(&p, level).load_source()?;
let image = r.image.clone();
self.media.insert_source(id, level, r);
Ok(image)
}
}
@@ -743,7 +840,9 @@ pub struct ProbeInfo {
pub captured: Option<String>,
pub meta: lightcraft_catalog::Meta,
pub as_shot_wb: Option<(f64, f64)>,
/// Hash of the file's bytes (hex), for duplicate detection.
/// Hash of the file's bytes (hex), for duplicate detection. When it is a 32-digit
/// [`lightcraft_preview::hash_bytes`] of the whole file (as the native probe computes it),
/// Import → Copy verifies each copy against it instead of reading the source again.
pub content_hash: Option<String>,
/// Lens corrections embedded in the file (DNG opcodes).
pub embedded_lens: Option<lightcraft_develop::EmbeddedLens>,
@@ -752,6 +851,8 @@ pub struct ProbeInfo {
/// A raw variant that can't be decoded yet: why. The file is described (and will be shown and
/// edited) from its embedded preview; see [`lightcraft_catalog::Photo::preview_only`].
pub preview_only: Option<String>,
/// Raws: the decoder has no colour matrix for the file ([`lightcraft_catalog::Photo::fallback_matrix`]).
pub fallback_matrix: Option<bool>,
}
pub type FileProbe = Arc<dyn Fn(&str) -> Result<ProbeInfo, String> + Send + Sync>;
@@ -760,6 +861,62 @@ pub type FileProbe = Arc<dyn Fn(&str) -> Result<ProbeInfo, String> + Send + Sync
mod tests {
use super::*;
/// The decoder says a NEF has no matrix (relative WB), but a photo catalogued before that
/// fact was recorded was edited under absolute WB: the catalog's rule wins, the decoder's
/// pixels and camera tone are used.
#[test]
fn catalog_decides_how_raw_white_balance_is_read() {
let tone = lightcraft_pipeline::tone::CameraTone::new(std::array::from_fn(|i| [0.01 * (i + 1) as f32, 0.02 * (i + 1) as f32])).unwrap();
let decoded = SourceInfo { raw: true, relative_wb: true, camera_tone: Some(tone), ..Default::default() };
let source = DecodedSource::new(Arc::new(Rgb32f::new(2, 2)), Some(decoded));
let mut p = Photo::new(PhotoId(1), Source::File { path: "/x/a.nef".into() }, "a.nef", "NEF", 2, 2, "");
p.kind = MediaKind::Raw;
p.as_shot_wb = Some((5100.0, 3.0));
let info = source.info_or(source_info(&p));
assert_eq!((info.relative_wb, info.as_shot_temp, info.as_shot_tint, info.camera_tone), (false, 5100.0, 3.0, Some(tone)));
p.fallback_matrix = Some(true);
let info = source.info_or(source_info(&p));
assert_eq!((info.relative_wb, info.as_shot_temp, info.as_shot_tint), (true, 6500.0, 0.0));
// A file with no catalog record (header not raw) keeps the decoder's view.
assert_eq!(source.info_or(SourceInfo::default()), decoded);
}
#[test]
fn decoder_info_survives_render_jobs_cache_and_eviction() {
let tone = lightcraft_pipeline::tone::CameraTone::new(std::array::from_fn(|i| {
let x = 0.01 * (i + 1) as f32;
[x, (x * 2.0).min(0.9)]
}))
.unwrap();
let info = SourceInfo { raw: true, relative_wb: true, camera_tone: Some(tone), ..Default::default() };
let mut s = crate::Session::with_demo();
let id = s.active().unwrap();
let mut job = s.render_job(id, 64, 64, false, true).unwrap();
job.source = SourceRef::File {
path: "synthetic.arw".into(),
max_edge: 64,
loader: Some(Arc::new(move |_, _| {
let mut image = Rgb32f::new(64, 64);
image.data.fill([0.1; 3]);
Ok((image, info))
})),
fallback: None,
};
job.info = SourceInfo::default(); // Header facts cannot override decoder facts.
job.settings = Arc::new(DevelopSettings::default());
let r = job.clone().run();
assert_eq!(r.loaded.as_ref().unwrap().info, Some(info));
let expected = lightcraft_pipeline::render(&r.loaded.as_ref().unwrap().image, &info, &job.settings, &job.request);
assert_eq!(r.rendered.as_ref().unwrap().image.data, expected.image.data);
s.accept(&r);
job.source = s.media.source_ref(s.catalog.photo(id).unwrap(), job.level);
let again = job.run();
assert!(again.loaded.is_none());
assert_eq!(r.rendered.unwrap().image.data, again.rendered.unwrap().image.data);
s.media.forget(id);
assert!(s.media.get_source(id, r.level).is_none());
}
#[test]
fn budget_evicts_least_recently_used_across_caches() {
let img = |w: usize| Arc::new(Rgb32f::new(w, w));
+9 -8
View File
@@ -304,15 +304,13 @@ fn apply_auto(s: &mut DevelopSettings, src: &Rgb32f, info: &SourceInfo) {
s.color.saturation = a.saturation;
}
/// A free file name `dir/<stem>-<suffix>.dng` (`-2`, `-3`… when taken).
fn output_path(first: &str, suffix: &str) -> String {
/// `dir/<stem>-<suffix>.dng`, then `-2`, `-3`… (the candidates for a merge's output).
fn output_names(first: &str, suffix: &str) -> impl Iterator<Item = std::path::PathBuf> {
let p = std::path::Path::new(first);
let dir = p.parent().map(|d| d.to_path_buf()).unwrap_or_default();
let stem = p.file_stem().map(|s| s.to_string_lossy().to_string()).unwrap_or_else(|| "Merge".into());
let cand = |k: u64| dir.join(if k == 1 { format!("{stem}-{suffix}.dng") } else { format!("{stem}-{suffix}-{k}.dng") });
// (`u64::MAX` taken names can't exist; the last candidate is the fallback)
let free = (1..u64::MAX).map(cand).find(|c| !c.exists()).unwrap_or_else(|| cand(u64::MAX));
free.to_string_lossy().to_string()
let suffix = suffix.to_string();
(1u64..).map(move |k| dir.join(if k == 1 { format!("{stem}-{suffix}.dng") } else { format!("{stem}-{suffix}-{k}.dng") }))
}
/// Parse `merge.*` command parameters into a kind and finishing options.
@@ -367,8 +365,11 @@ impl Session {
/// apply Auto Settings / the auto crop. Returns `{id, path, …info}`.
pub fn finish_merge(&mut self, job: &MergeJob, out: MergeOutput) -> Result<Value> {
let first = &job.sources.first().ok_or_else(|| EngineError::Other("nothing merged".into()))?.1;
let path = output_path(first, job.kind.suffix());
std::fs::write(&path, &out.dng).map_err(|e| EngineError::Other(format!("{path}: {e}")))?;
// a new file under a free name (never replacing one), complete and synced before it appears
let path = lightcraft_catalog::safe_file::write_new_unique(&mut output_names(first, job.kind.suffix()), &out.dng)
.map_err(|e| EngineError::Other(format!("could not write the merged DNG next to {first}: {e}")))?
.to_string_lossy()
.to_string();
let report = crate::import::import(self, std::slice::from_ref(&path), crate::import::ImportMode::Add)?;
let id = report
.imported
+7 -2
View File
@@ -68,9 +68,14 @@ impl crate::Session {
OriginalGuard::new(&self.catalog)
}
/// `Err` when writing `path` would replace a catalogued photo's original or sidecar.
/// `Err` when writing `path` would replace a catalogued photo's original or sidecar. A free
/// path (the usual case) is answered without building the guard over the whole library.
pub fn check_write_target(&self, path: &str) -> Result<(), String> {
self.original_guard().check(Path::new(path))
let path = Path::new(path);
if std::fs::symlink_metadata(path).is_err() {
return Ok(()); // nothing there to replace
}
self.original_guard().check(path)
}
}
+605 -125
View File
@@ -11,7 +11,11 @@
//!
//! Safety:
//! - a target that exists on disk, or that another photo in the batch gets, receives a `-1`, `-2`…
//! suffix — no file is ever overwritten;
//! suffix — no file is ever overwritten. Whether an existing target is the photo itself (a
//! change of letter case on a case-insensitive volume) is decided by file identity, not by
//! comparing names, so on a case-sensitive volume `img_1.jpg` next to `IMG_1.JPG` is a collision
//! (issue #95); moves never replace a file even if one appears after the check (hard link +
//! unlink where the volume supports it);
//! - the moves happen before the catalog changes; if one fails, the ones already done are moved
//! back and nothing is committed;
//! - the catalog op ([`Op::SetFile`]) is undoable: undo/redo move the files back and forth (see
@@ -297,139 +301,352 @@ fn sidecars(path: &str) -> Vec<(PathBuf, SidecarNaming, bool)> {
v
}
/// Move `from` to `to` (never over an existing file) together with its sidecars. On error nothing
/// is left moved.
pub fn move_file(from: &str, to: &str) -> std::result::Result<(), String> {
/// The file-system primitives renaming uses. [`RealFs`] is the disk; tests substitute a model of
/// a case-insensitive or case-sensitive volume (and inject failures).
pub(crate) trait MoveFs {
/// Something (file, folder, link) answers to `p`.
fn exists(&self, p: &Path) -> bool;
fn is_file(&self, p: &Path) -> bool;
/// `a` and `b` name the same file (e.g. `IMG.JPG` and `img.jpg` on a case-insensitive
/// volume). Decided by file identity, never by comparing names: on a case-sensitive volume
/// they are two different photos. When in doubt: `false`.
fn same_file(&self, a: &Path, b: &Path) -> bool;
/// Rename `a` to `b`, failing (never replacing) when `b` exists.
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()>;
/// Copy `a` to a new file `b` (an independent copy, never a link), failing when `b` exists.
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()>;
fn remove_file(&self, p: &Path) -> std::io::Result<()>;
/// Another photo file beside the stem-named sidecar `sidecar` that shares its stem (e.g.
/// `IMG_1.JPG` for `IMG_1.xmp`), which still uses that sidecar.
fn stem_sibling(&self, sidecar: &Path) -> Option<String>;
}
/// The real file system.
pub(crate) struct RealFs;
impl MoveFs for RealFs {
fn exists(&self, p: &Path) -> bool {
std::fs::symlink_metadata(p).is_ok()
}
fn is_file(&self, p: &Path) -> bool {
p.is_file()
}
fn same_file(&self, a: &Path, b: &Path) -> bool {
if a == b {
return true;
}
let (Ok(ma), Ok(mb)) = (std::fs::metadata(a), std::fs::metadata(b)) else { return false };
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
ma.dev() == mb.dev() && ma.ino() == mb.ino()
}
#[cfg(not(unix))]
{
// no stable file id in std here: `b` is `a` under another spelling when they look
// alike and the folder has no entry spelled exactly like `b` (on a case-sensitive
// folder holding both `IMG.JPG` and `img.jpg`, both entries are listed)
let (Some(dir), Some(name)) = (b.parent(), b.file_name()) else { return false };
if a.parent() != Some(dir) || ma.len() != mb.len() || ma.modified().ok() != mb.modified().ok() {
return false;
}
let Ok(entries) = std::fs::read_dir(dir) else { return false };
!entries.flatten().any(|e| e.file_name() == name)
}
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, format!("{} already exists", b.display())));
}
// A hard link fails when `b` exists — even if it appeared since the check above — so
// nothing is ever replaced. Volumes without hard links (FAT, some network shares) fall back
// to a plain rename after the check.
match std::fs::hard_link(a, b) {
Ok(()) => std::fs::remove_file(a).inspect_err(|_| {
// both names are the same file: dropping the new one loses nothing
let _ = std::fs::remove_file(b);
}),
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => Err(e),
Err(_) => std::fs::rename(a, b),
}
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
use std::io::Write;
let bytes = std::fs::read(a)?;
let mut f = std::fs::OpenOptions::new().write(true).create_new(true).open(b)?;
f.write_all(&bytes).and_then(|_| f.sync_all()).inspect_err(|_| {
let _ = std::fs::remove_file(b);
})
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
std::fs::remove_file(p)
}
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
crate::import_move::sibling_photo(sidecar)
}
}
/// `from` → `to` differs only in letter case and `to` is that very file (a case-insensitive
/// volume): the rename must go through a temporary name.
fn is_respelling(fs: &dyn MoveFs, from: &Path, to: &Path) -> bool {
from != to && from.to_string_lossy().to_lowercase() == to.to_string_lossy().to_lowercase() && fs.exists(to) && fs.same_file(from, to)
}
/// Rename `a` to `b` without ever replacing a file; a change of letter case only goes through a
/// unique temporary name so case-insensitive volumes see a real change.
fn move_one(fs: &dyn MoveFs, a: &Path, b: &Path) -> std::io::Result<()> {
if !is_respelling(fs, a, b) {
return fs.rename_no_replace(a, b);
}
let tmp = (0..1000u32)
.map(|k| a.with_file_name(format!(".lc-rename-{}-{k}", std::process::id())))
.find(|t| !fs.exists(t))
.ok_or_else(|| std::io::Error::other("no free temporary name"))?;
fs.rename_no_replace(a, &tmp)?;
fs.rename_no_replace(&tmp, b).map_err(|e| match fs.rename_no_replace(&tmp, a) {
Ok(()) => e,
Err(e2) => std::io::Error::other(format!("{e}; it could not be given its old name back either ({e2}): it is now {}", tmp.display())),
})
}
/// Why [`move_file`] failed, and what it left behind.
#[derive(Clone, Debug, PartialEq)]
pub struct MoveError {
/// What went wrong, including every step that could not be undone and where that file is now.
pub message: String,
/// The file itself ended up at the new path: it moved, a sidecar didn't, and moving the file
/// back failed too. Its sidecars that did move stay with it.
pub moved: bool,
}
impl std::fmt::Display for MoveError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.message)
}
}
/// Move `from` to `to` (never over an existing file) together with its sidecars. On error the
/// moves are undone; whatever can't be undone is named in the error (see [`MoveError`]).
pub fn move_file(from: &str, to: &str) -> std::result::Result<(), MoveError> {
move_file_with(&RealFs, from, to)
}
/// [`move_file`] on the file system `fs`.
pub(crate) fn move_file_with(fs: &dyn MoveFs, from: &str, to: &str) -> std::result::Result<(), MoveError> {
if from == to {
return Ok(());
}
let fail = |message: String| MoveError { message, moved: false };
let (f, t) = (Path::new(from), Path::new(to));
// a case-only change on a case-insensitive file system "exists" already: that is this file
let case_only = from.to_lowercase() == to.to_lowercase();
if t.exists() && !case_only {
return Err(format!("{to} already exists"));
// `to` may be this very file spelled differently (case-insensitive volume); any other file
// there — e.g. `img.jpg` next to `IMG.JPG` on a case-sensitive volume — is never replaced
if fs.exists(t) && !is_respelling(fs, f, t) {
return Err(fail(format!("{to} already exists")));
}
let mut done: Vec<(PathBuf, PathBuf)> = Vec::new();
let mv = |a: &Path, b: &Path| -> std::io::Result<()> {
if case_only {
// via a temporary name, so case-insensitive file systems see a real change
let tmp = a.with_file_name(format!(".lc-rename-{}", std::process::id()));
std::fs::rename(a, &tmp)?;
std::fs::rename(&tmp, b).inspect_err(|_| {
let _ = std::fs::rename(&tmp, a);
})
} else {
std::fs::rename(a, b)
}
};
mv(f, t).map_err(|e| format!("rename {from}: {e}"))?;
done.push((f.to_path_buf(), t.to_path_buf()));
move_one(fs, f, t).map_err(|e| fail(format!("rename {from}: {e}")))?;
// sidecars moved (or copied: `true`)
let mut done: Vec<(PathBuf, PathBuf, bool)> = Vec::new();
for (sc, naming, upper) in sidecars(from) {
if !sc.is_file() {
if !fs.is_file(&sc) {
continue;
}
let mut dst = crate::sidecar::sidecar_path(to, naming);
if upper {
dst = dst.with_extension("XMP");
}
if dst.exists() && !case_only {
// a stem sidecar another file still uses (`IMG_1.xmp` of `IMG_1.CR3` and `IMG_1.JPG`)
// is copied, not taken away from it (issue #92)
let shared = naming == SidecarNaming::Stem && fs.stem_sibling(&sc).is_some();
if fs.exists(&dst) && (shared || !is_respelling(fs, &sc, &dst)) {
continue; // never overwrite: the old sidecar stays where it was
}
if let Err(e) = mv(&sc, &dst) {
for (a, b) in done.iter().rev() {
let _ = std::fs::rename(b, a);
let r = if shared { fs.copy_no_replace(&sc, &dst) } else { move_one(fs, &sc, &dst) };
if let Err(e) = r {
let mut message = format!("rename {}: {e}", sc.display());
// the file first: if it can't go back, the sidecars that moved stay with it
if let Err(e) = move_one(fs, t, f) {
message.push_str(&format!("; {from} could not be moved back ({e}): it is now {to}"));
if !done.is_empty() {
let names: Vec<String> = done.iter().map(|(_, b, _)| b.display().to_string()).collect();
message.push_str(&format!(" with its sidecar {}", names.join(", ")));
}
return Err(MoveError { message, moved: true });
}
return Err(format!("rename {}: {e}", sc.display()));
for (a, b, copied) in done.iter().rev() {
if *copied {
// the original is still there: the copy just goes (a leftover copy loses nothing)
let _ = fs.remove_file(b);
} else if let Err(e) = move_one(fs, b, a) {
message.push_str(&format!("; sidecar {} could not be moved back ({e}): it is now {}", a.display(), b.display()));
}
}
return Err(fail(message));
}
done.push((sc, dst));
done.push((sc, dst, shared));
}
Ok(())
}
/// What a failed batch of moves ([`move_all`]) left behind.
#[derive(Debug)]
pub(crate) struct BatchMoveError {
/// The first failure and every move that could not be undone.
pub message: String,
/// Moves that stayed done, (from, to): those files are at `to` now.
pub stuck: Vec<(String, String)>,
}
/// Move the files `moves` names, all or nothing: after a failure the earlier moves are undone.
/// Moves that can't be undone (e.g. the network share went away) are reported, not ignored.
pub(crate) fn move_all(fs: &dyn MoveFs, moves: &[(String, String)]) -> std::result::Result<(), BatchMoveError> {
let mut done: Vec<&(String, String)> = Vec::new();
for m in moves {
let Err(e) = move_file_with(fs, &m.0, &m.1) else {
done.push(m);
continue;
};
let mut message = e.message;
let mut stuck = Vec::new();
if e.moved {
stuck.push(m.clone());
}
for d in done.iter().rev() {
if let Err(e) = move_file_with(fs, &d.1, &d.0) {
message.push_str(&format!("; {} could not be moved back to {}: {}", d.1, d.0, e.message));
if !e.moved {
stuck.push((*d).clone());
}
}
}
return Err(BatchMoveError { message, stuck });
}
Ok(())
}
/// The `SetFile` ops in `op` (nested batches included) that point a photo at one of `to`.
fn set_file_ops_to(op: &Op, to: &HashSet<&str>, out: &mut Vec<Op>) {
match op {
Op::Batch { ops } => ops.iter().for_each(|o| set_file_ops_to(o, to, out)),
Op::SetFile { source: Source::File { path }, .. } if to.contains(path.as_str()) => out.push(op.clone()),
_ => {}
}
}
/// The photos among `ids` a rename handles, in order: photos sharing one file (virtual copies)
/// rename once. No file-system calls.
pub fn rename_photos(cat: &lightcraft_catalog::Catalog, ids: &[PhotoId]) -> Vec<std::sync::Arc<Photo>> {
let mut seen_paths: HashSet<String> = HashSet::new();
ids.iter()
.filter_map(|id| cat.photo(*id))
.filter(|p| match &p.source {
Source::File { path } => seen_paths.insert(path.clone()),
_ => true,
})
.cloned()
.collect()
}
/// [`Session::plan_rename`] for [`rename_photos`], with the "is this name taken on disk?" check
/// supplied: the Rename dialog plans its preview on a worker thread (a check can block on a slow
/// drive). Whether a case-only change names this very file is decided on the disk.
pub fn plan_rename_photos(photos: &[std::sync::Arc<Photo>], template: &str, start: usize, exists: &dyn Fn(&str) -> bool) -> Vec<RenamePlan> {
plan_rename_core(photos, template, start, &|p| exists(&p.to_string_lossy()), &|a, b| RealFs.same_file(a, b))
}
/// Plan renaming `photos`: `exists` says whether something answers to a path, `same_file` whether
/// two spellings name one file (see [`MoveFs`]).
fn plan_rename_core(
photos: &[std::sync::Arc<Photo>],
template: &str,
start: usize,
exists: &dyn Fn(&Path) -> bool,
same_file: &dyn Fn(&Path, &Path) -> bool,
) -> Vec<RenamePlan> {
let mut taken: HashSet<String> = HashSet::new(); // lower-case target paths/names claimed in this batch
let mut out = Vec::new();
for (seq, p) in (start..).zip(photos) {
let want = expand(template, p, seq);
let (stem, ext) = split_ext(&want);
let (stem, ext) = (stem.to_string(), ext.to_string());
let candidate = |k: usize| {
if k == 0 {
want.clone()
} else if ext.is_empty() {
format!("{stem}-{k}")
} else {
format!("{stem}-{k}.{ext}")
}
};
let (to, to_path) = match &p.source {
Source::File { path } => {
let dir = Path::new(path).parent().map(Path::to_path_buf).unwrap_or_default();
let mut k = 0;
loop {
let name = candidate(k);
let tp = dir.join(&name).to_string_lossy().to_string();
let key = tp.to_lowercase();
// this very file, maybe spelled differently (case-insensitive volume)? By
// identity: on a case-sensitive volume `img_1.jpg` may be another photo
let (from, to) = (Path::new(path), Path::new(&tp));
let same = tp == *path || (from != to && key == path.to_lowercase() && exists(to) && same_file(from, to));
// free: not claimed in this batch and not on disk (unless it is this very file).
// A file this batch moves away still counts as taken: simple and safe.
if !taken.contains(&key) && (same || !exists(to)) {
taken.insert(key);
break (name, Some(tp));
}
k += 1;
}
}
Source::Demo { .. } => {
let mut k = 0;
loop {
let name = candidate(k);
if taken.insert(format!("demo:{}", name.to_lowercase())) {
break (name, None);
}
k += 1;
}
}
};
let from_path = match &p.source {
Source::File { path } => Some(path.clone()),
_ => None,
};
out.push(RenamePlan { id: p.id.0, from: p.file_name.clone(), to, from_path, to_path });
}
out
}
impl Session {
/// Plan renaming `ids` with `template` (sequence numbers from `start`), resolving collisions.
pub fn plan_rename(&self, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
// photos sharing one file (virtual copies) rename once
let mut seen_paths: HashSet<String> = HashSet::new();
let mut taken: HashSet<String> = HashSet::new(); // lower-case target paths/names claimed in this batch
let mut out = Vec::new();
let mut seq = start;
for id in ids {
let Some(p) = self.catalog.photo(*id) else { continue };
if let Source::File { path } = &p.source
&& !seen_paths.insert(path.clone())
{
continue;
}
let want = expand(template, p, seq);
seq += 1;
let (stem, ext) = split_ext(&want);
let (stem, ext) = (stem.to_string(), ext.to_string());
let candidate = |k: usize| {
if k == 0 {
want.clone()
} else if ext.is_empty() {
format!("{stem}-{k}")
} else {
format!("{stem}-{k}.{ext}")
}
};
let (to, to_path) = match &p.source {
Source::File { path } => {
let dir = Path::new(path).parent().map(Path::to_path_buf).unwrap_or_default();
let mut k = 0;
loop {
let name = candidate(k);
let tp = dir.join(&name).to_string_lossy().to_string();
let key = tp.to_lowercase();
let same = key == path.to_lowercase();
// free: not claimed in this batch and not on disk (unless it is this very file).
// A file this batch moves away still counts as taken: simple and safe.
let on_disk = Path::new(&tp).exists() && !same;
if !taken.contains(&key) && !on_disk {
taken.insert(key);
break (name, Some(tp));
}
k += 1;
}
}
Source::Demo { .. } => {
let mut k = 0;
loop {
let name = candidate(k);
if taken.insert(format!("demo:{}", name.to_lowercase())) {
break (name, None);
}
k += 1;
}
}
};
let from_path = match &p.source {
Source::File { path } => Some(path.clone()),
_ => None,
};
out.push(RenamePlan { id: id.0, from: p.file_name.clone(), to, from_path, to_path });
}
out
self.plan_rename_with(&RealFs, ids, template, start)
}
/// [`Session::plan_rename`] on the file system `fs`.
pub(crate) fn plan_rename_with(&self, fs: &dyn MoveFs, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
plan_rename_core(&rename_photos(&self.catalog, ids), template, start, &|p| fs.exists(p), &|a, b| fs.same_file(a, b))
}
/// Carry out `plans`: move the files (rolled back on failure), then commit one undoable op
/// that updates the photos (and their virtual copies).
///
/// When a failure can't be fully rolled back, the files that kept their new names are
/// committed as a smaller undoable rename, so the library points at them (nothing goes
/// missing), and the error lists them (old → new).
pub fn apply_rename(&mut self, plans: &[RenamePlan]) -> Result<usize> {
let mut moved: Vec<(String, String)> = Vec::new();
for pl in plans {
if let (Some(a), Some(b)) = (&pl.from_path, &pl.to_path)
&& a != b
{
if let Err(e) = move_file(a, b) {
for (a, b) in moved.iter().rev() {
let _ = move_file(b, a);
}
return Err(EngineError::Other(e));
}
moved.push((a.clone(), b.clone()));
}
}
self.apply_rename_with(&RealFs, plans)
}
/// [`Session::apply_rename`] on the file system `fs`.
pub(crate) fn apply_rename_with(&mut self, fs: &dyn MoveFs, plans: &[RenamePlan]) -> Result<usize> {
let mut ops = Vec::new();
let by_path: HashMap<&str, &RenamePlan> = plans.iter().filter_map(|p| p.from_path.as_deref().map(|f| (f, p))).collect();
for p in self.catalog.photos() {
@@ -447,17 +664,66 @@ impl Session {
}
}
let n = ops.len();
let op = Op::Batch { ops };
let moves: Vec<(String, String)> = plans
.iter()
.filter_map(|pl| match (&pl.from_path, &pl.to_path) {
(Some(a), Some(b)) if a != b => Some((a.clone(), b.clone())),
_ => None,
})
.collect();
if let Err(e) = move_all(fs, &moves) {
return Err(EngineError::Other(self.follow_stuck(&op, e, true)));
}
if n > 0
&& let Err(e) = self.commit(&format!("Rename {n} Photo{}", if n == 1 { "" } else { "s" }), Op::Batch { ops })
&& let Err(err) = self.commit(&format!("Rename {n} Photo{}", if n == 1 { "" } else { "s" }), op)
{
for (a, b) in moved.iter().rev() {
let _ = move_file(b, a);
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
if let Err(e) = move_all(fs, &back) {
return Err(EngineError::Other(format!("{err}; the renamed files could not all be moved back: {}", e.message)));
}
return Err(e);
return Err(err);
}
Ok(n)
}
/// After a failed batch of moves for `op`: point the photos whose files stayed moved at
/// their new paths (an undoable "partly" rename when `undoable`, otherwise a plain logged
/// change), so the library matches the disk. Returns the message for the user.
pub(crate) fn follow_stuck(&mut self, op: &Op, e: BatchMoveError, undoable: bool) -> String {
let mut msg = e.message;
if e.stuck.is_empty() {
return msg;
}
let to: HashSet<&str> = e.stuck.iter().map(|m| m.1.as_str()).collect();
let mut keep = Vec::new();
set_file_ops_to(op, &to, &mut keep);
let list: Vec<String> = e.stuck.iter().map(|(a, b)| format!("{a} → {b}")).collect();
let n = e.stuck.len();
let s = if n == 1 { "" } else { "s" };
let fix = Op::Batch { ops: keep };
let applied = if undoable {
self.commit(&format!("Rename {n} Photo{s} (partly)"), fix)
} else {
match self.catalog.apply(fix.clone()) {
Ok(_) => {
self.pending_log.push(fix);
Ok(())
}
Err(e) => Err(e.into()),
}
};
match applied {
Ok(()) => msg
.push_str(&format!(". {n} file{s} could not be moved back and kept the new name; the library now points there: {}", list.join(", "))),
Err(err) => msg.push_str(&format!(
". {n} file{s} could not be moved back and the library could not follow ({err}); relink with Find Missing Photos: {}",
list.join(", ")
)),
}
msg
}
/// File moves an undo/redo op implies: `SetFile` ops whose source path differs from the photo's
/// current one.
pub(crate) fn file_moves(&self, op: &Op) -> Vec<(String, String)> {
@@ -479,26 +745,240 @@ impl Session {
rec(self, op, &mut v);
v
}
/// Move the files `moves` names; all or nothing.
pub(crate) fn move_files(moves: &[(String, String)]) -> Result<()> {
let mut done: Vec<&(String, String)> = Vec::new();
for m in moves {
if let Err(e) = move_file(&m.0, &m.1) {
for (a, b) in done.iter().rev().map(|m| (&m.0, &m.1)) {
let _ = move_file(b, a);
}
return Err(EngineError::Other(format!("can't undo the rename: {e}")));
}
done.push(m);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::cell::RefCell;
/// A model volume: files by path with contents; case-insensitive (default APFS/NTFS) or
/// case-sensitive (Linux, case-sensitive APFS). `fail` makes renames from a matching source fail.
pub(crate) struct FakeFs {
pub ci: bool,
pub files: RefCell<Vec<(String, Vec<u8>)>>,
pub fail: RefCell<Vec<String>>,
}
impl FakeFs {
pub fn new(ci: bool, files: &[(&str, &str)]) -> FakeFs {
FakeFs {
ci,
files: RefCell::new(files.iter().map(|(p, c)| (p.to_string(), c.as_bytes().to_vec())).collect()),
fail: RefCell::new(Vec::new()),
}
}
fn idx(&self, p: &Path) -> Option<usize> {
let p = p.to_string_lossy();
self.files.borrow().iter().position(|(f, _)| if self.ci { f.to_lowercase() == p.to_lowercase() } else { *f == p })
}
/// The listing: (exact path, contents), sorted.
pub fn listing(&self) -> Vec<(String, String)> {
let mut v: Vec<_> = self.files.borrow().iter().map(|(p, c)| (p.clone(), String::from_utf8_lossy(c).to_string())).collect();
v.sort();
v
}
}
impl MoveFs for FakeFs {
fn exists(&self, p: &Path) -> bool {
self.idx(p).is_some()
}
fn is_file(&self, p: &Path) -> bool {
self.exists(p)
}
fn same_file(&self, a: &Path, b: &Path) -> bool {
self.idx(a).is_some() && self.idx(a) == self.idx(b)
}
fn rename_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.fail.borrow().iter().any(|f| a.to_string_lossy().contains(f.as_str())) {
return Err(std::io::Error::other("injected failure"));
}
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
self.files.borrow_mut()[i].0 = b.to_string_lossy().to_string();
Ok(())
}
fn copy_no_replace(&self, a: &Path, b: &Path) -> std::io::Result<()> {
if self.exists(b) {
return Err(std::io::Error::new(std::io::ErrorKind::AlreadyExists, "exists"));
}
let i = self.idx(a).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
let c = self.files.borrow()[i].1.clone();
self.files.borrow_mut().push((b.to_string_lossy().to_string(), c));
Ok(())
}
fn remove_file(&self, p: &Path) -> std::io::Result<()> {
let i = self.idx(p).ok_or_else(|| std::io::Error::new(std::io::ErrorKind::NotFound, "no such file"))?;
self.files.borrow_mut().remove(i);
Ok(())
}
fn stem_sibling(&self, sidecar: &Path) -> Option<String> {
let stem = sidecar.with_extension("").to_string_lossy().to_string();
self.files.borrow().iter().map(|(f, _)| f.clone()).find(|f| {
let p = Path::new(f);
p.with_extension("").to_string_lossy() == stem && !p.extension().is_some_and(|e| e.eq_ignore_ascii_case("xmp"))
})
}
}
fn file_photo(id: u64, path: &str) -> Photo {
let name = Path::new(path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
Photo::new(PhotoId(id), Source::File { path: path.into() }, &name, "JPG", 1, 1, "2026-01-01T00:00:00")
}
/// Issue #95: a case-only rename is a real change on a case-insensitive volume and never
/// replaces a different file that differs only by case on a case-sensitive one.
#[test]
fn case_only_renames_never_replace_another_file() {
// case-sensitive: IMG_1.JPG and img_1.JPG are two photos
let fs = FakeFs::new(false, &[("/p/IMG_1.JPG", "first"), ("/p/img_1.JPG", "second"), ("/p/IMG_1.xmp", "sc1"), ("/p/img_1.xmp", "sc2")]);
assert!(move_file_with(&fs, "/p/IMG_1.JPG", "/p/img_1.JPG").unwrap_err().message.contains("already exists"));
assert_eq!(fs.listing().len(), 4);
assert!(fs.listing().contains(&("/p/img_1.JPG".into(), "second".into())));
// the planner picks a free name instead
let mut s = Session::new();
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(1, "/p/IMG_1.JPG")) }).unwrap();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "img_1", 1);
assert_eq!(plans[0].to, "img_1-1.JPG");
move_file_with(&fs, "/p/IMG_1.JPG", plans[0].to_path.as_deref().unwrap()).unwrap();
assert_eq!(
fs.listing(),
vec![
("/p/img_1-1.JPG".into(), "first".into()),
("/p/img_1-1.xmp".into(), "sc1".into()),
("/p/img_1.JPG".into(), "second".into()),
("/p/img_1.xmp".into(), "sc2".into())
]
);
// case-sensitive, target free: a plain rename
let fs = FakeFs::new(false, &[("/p/IMG_2.JPG", "x")]);
move_file_with(&fs, "/p/IMG_2.JPG", "/p/img_2.JPG").unwrap();
assert_eq!(fs.listing(), vec![("/p/img_2.JPG".into(), "x".into())]);
// case-insensitive: the "existing" target is the photo itself → renamed via a temp name
let fs = FakeFs::new(true, &[("/p/IMG_1.JPG", "first"), ("/p/IMG_1.xmp", "sc1")]);
let mut s = Session::new();
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(1, "/p/IMG_1.JPG")) }).unwrap();
let plans = s.plan_rename_with(&fs, &[PhotoId(1)], "img_1", 1);
assert_eq!(plans[0].to, "img_1.JPG", "not a collision with itself");
move_file_with(&fs, "/p/IMG_1.JPG", "/p/img_1.JPG").unwrap();
assert_eq!(fs.listing(), vec![("/p/img_1.JPG".into(), "first".into()), ("/p/img_1.xmp".into(), "sc1".into())]);
// case-insensitive, another file under another name: still a collision
let fs = FakeFs::new(true, &[("/p/A.JPG", "a"), ("/p/b.JPG", "b")]);
assert!(move_file_with(&fs, "/p/A.JPG", "/p/B.JPG").is_err());
}
/// Issue #105: a rollback that fails is reported, and the library follows the files that kept
/// their new names (an undoable partial rename), so nothing goes missing.
#[test]
fn failed_rollbacks_are_reported_and_followed() {
let fs = FakeFs::new(false, &[("/p/a.jpg", "A"), ("/p/a.xmp", "As"), ("/p/b.jpg", "B"), ("/p/c.jpg", "C")]);
let mut s = Session::new();
for (i, f) in ["/p/a.jpg", "/p/b.jpg", "/p/c.jpg"].iter().enumerate() {
s.catalog.apply(Op::AddPhoto { photo: Box::new(file_photo(i as u64 + 1, f)) }).unwrap();
}
let ids = [PhotoId(1), PhotoId(2), PhotoId(3)];
let plans = s.plan_rename_with(&fs, &ids, "Trip-{seq}", 1);
// c can't be renamed (the share went away), and neither can Trip-1 be moved back
fs.fail.borrow_mut().extend(["c.jpg".to_string(), "Trip-1".to_string()]);
let err = s.apply_rename_with(&fs, &plans).unwrap_err().to_string();
assert!(err.contains("rename /p/c.jpg"), "{err}");
assert!(err.contains("/p/Trip-1.jpg could not be moved back"), "{err}");
assert!(err.contains("/p/a.jpg → /p/Trip-1.jpg"), "lists what stayed renamed: {err}");
assert_eq!(
fs.listing(),
vec![
("/p/Trip-1.jpg".into(), "A".into()),
("/p/Trip-1.xmp".into(), "As".into()),
("/p/b.jpg".into(), "B".into()),
("/p/c.jpg".into(), "C".into())
]
);
// the catalog matches the disk
let path = |s: &Session, id: u64| match &s.catalog.photo(PhotoId(id)).unwrap().source {
Source::File { path } => path.clone(),
Source::Demo { .. } => String::new(),
};
assert_eq!(path(&s, 1), "/p/Trip-1.jpg");
assert_eq!(s.catalog.photo(PhotoId(1)).unwrap().file_name, "Trip-1.jpg");
assert_eq!((path(&s, 2), path(&s, 3)), ("/p/b.jpg".into(), "/p/c.jpg".into()));
assert_eq!(s.undo.last().map(|e| e.label.as_str()), Some("Rename 1 Photo (partly)"));
// a sidecar that won't move, and a file that won't go back: the file keeps its new name
// with the sidecars that did move, and says so
let fs = FakeFs::new(false, &[("/p/d.jpg", "D"), ("/p/d.xmp", "Ds"), ("/p/d.jpg.xmp", "Df")]);
fs.fail.borrow_mut().extend(["d.jpg.xmp".to_string(), "/p/e.jpg".to_string()]);
let e = move_file_with(&fs, "/p/d.jpg", "/p/e.jpg").unwrap_err();
assert!(e.moved, "{e}");
assert!(e.message.contains("it is now /p/e.jpg with its sidecar /p/e.xmp"), "{e}");
assert_eq!(fs.listing(), vec![("/p/d.jpg.xmp".into(), "Df".into()), ("/p/e.jpg".into(), "D".into()), ("/p/e.xmp".into(), "Ds".into())]);
// … and when the file does go back, its sidecars follow it
let fs = FakeFs::new(false, &[("/p/d.jpg", "D"), ("/p/d.xmp", "Ds"), ("/p/d.jpg.xmp", "Df")]);
fs.fail.borrow_mut().push("d.jpg.xmp".to_string());
let e = move_file_with(&fs, "/p/d.jpg", "/p/e.jpg").unwrap_err();
assert!(!e.moved);
assert_eq!(fs.listing().len(), 3);
assert!(fs.listing().iter().all(|(p, _)| p.starts_with("/p/d.")));
}
/// Issue #92 (comment): renaming one of `IMG_1.CR3` + `IMG_1.JPG` copies their shared
/// `IMG_1.xmp` instead of taking it away from the other; the last one renamed takes it.
#[test]
fn shared_stem_sidecar_is_copied_not_taken() {
let fs = FakeFs::new(false, &[("/p/IMG_1.CR3", "raw"), ("/p/IMG_1.JPG", "jpg"), ("/p/IMG_1.xmp", "shared"), ("/p/IMG_1.CR3.xmp", "own")]);
move_file_with(&fs, "/p/IMG_1.CR3", "/p/Trip_1.CR3").unwrap();
assert_eq!(
fs.listing(),
vec![
("/p/IMG_1.JPG".into(), "jpg".into()),
("/p/IMG_1.xmp".into(), "shared".into()),
("/p/Trip_1.CR3".into(), "raw".into()),
("/p/Trip_1.CR3.xmp".into(), "own".into()),
("/p/Trip_1.xmp".into(), "shared".into()),
]
);
// the JPEG is the last one using it: it moves
move_file_with(&fs, "/p/IMG_1.JPG", "/p/Trip_2.JPG").unwrap();
assert!(fs.listing().contains(&("/p/Trip_2.xmp".into(), "shared".into())));
assert!(!fs.exists(Path::new("/p/IMG_1.xmp")));
// a failure after the copy removes the copy and keeps the original
let fs = FakeFs::new(false, &[("/p/A.CR3", "raw"), ("/p/A.JPG", "jpg"), ("/p/A.xmp", "shared"), ("/p/A.CR3.xmp", "own")]);
fs.fail.borrow_mut().push("A.CR3.xmp".into());
assert!(move_file_with(&fs, "/p/A.CR3", "/p/B.CR3").is_err());
assert_eq!(fs.listing().len(), 4);
assert!(fs.listing().contains(&("/p/A.xmp".into(), "shared".into())) && !fs.exists(Path::new("/p/B.xmp")));
}
/// Issue #95 on the real disk, whichever kind of volume the temp folder is on.
#[test]
fn case_only_rename_on_disk() {
let dir = std::env::temp_dir().join(format!("lc-rename-case-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let (upper, lower) = (dir.join("IMG_1.JPG"), dir.join("img_1.JPG"));
std::fs::write(&upper, b"first").unwrap();
let case_sensitive = !lower.exists();
let p = |q: &Path| q.to_string_lossy().to_string();
if case_sensitive {
std::fs::write(&lower, b"second").unwrap();
assert!(move_file(&p(&upper), &p(&lower)).is_err());
assert_eq!(std::fs::read(&lower).unwrap(), b"second");
assert_eq!(std::fs::read(&upper).unwrap(), b"first");
} else {
move_file(&p(&upper), &p(&lower)).unwrap();
let names: Vec<String> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
assert_eq!(names, vec!["img_1.JPG".to_string()]);
assert_eq!(std::fs::read(&lower).unwrap(), b"first");
}
// a plain rename never replaces
std::fs::write(dir.join("other.JPG"), b"other").unwrap();
assert!(move_file(&p(&dir.join("other.JPG")), &p(&lower)).is_err());
assert_eq!(std::fs::read(dir.join("other.JPG")).unwrap(), b"other");
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn folder_templates_stay_inside_the_destination() {
+161 -8
View File
@@ -5,12 +5,25 @@
//! creator, keywords, capture time and GPS in the standard namespaces, plus our complete develop
//! settings (`lc:settings`, JSON), the pick/reject flag (`lc:flag`) and location (`lc:location`).
//!
//! An existing sidecar is never replaced wholesale (issue #92): LightCraft's properties
//! ([`OWNED`], [`OWNED_IF_STATED`]) are merged into it ([`lightcraft_meta::merge_xmp`]) and
//! everything else — another editor's `crs:` develop settings, `xmpMM` history, unknown
//! namespaces — is kept byte for byte. A sidecar that can't be read as XMP is first copied to
//! `<name>.xmp.bak-<time>` (never overwritten), then replaced. Writes are atomic (temp file,
//! fsync, rename).
//!
//! **Shared names** (Stem naming): `IMG_0001.CR3` and `IMG_0001.JPG` would share `IMG_0001.xmp`.
//! When two catalogued files map to one stem sidecar, it belongs to one of them (a raw first, then
//! the first by file name — [`Session::sidecar_naming`]); the others use `<file>.xmp`
//! (`IMG_0001.JPG.xmp`) for writing and reading, so neither overwrites the other.
//!
//! **Read** (on import, and `photo.readMetadataFromFile`): the sidecar wins for metadata, except
//! the capture time: a time embedded in the file wins, and the sidecar's (`exif:DateTimeOriginal`,
//! else `photoshop:DateCreated`, else `xmp:CreateDate`) fills in only when the file has none; develop
//! settings are restored from `lc:settings` when present, else mapped from interoperable `crs:`
//! fields ([`crate::crs`]). For raw/DNG files without a sidecar the file's embedded XMP is used.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -258,11 +271,101 @@ pub fn sidecar_packet(p: &Photo, cat: &lightcraft_catalog::Catalog) -> String {
lightcraft_meta::write_xmp_lc(&meta, &lc)
}
/// Write `data` to `path` atomically (temp file + sync + rename).
/// The properties LightCraft writes and owns in a sidecar: replaced on every save (removed when
/// LightCraft has no value, so clearing a field clears it in the file too). All of them are read
/// back on import and by Read Metadata from File.
pub const OWNED: &[&str] = &[
"xmp:Rating",
"xmp:Label",
"dc:title",
"dc:description",
"dc:rights",
"dc:creator",
"dc:subject",
"Iptc4xmpCore:Location",
"Iptc4xmpCore:AltTextAccessibility",
"Iptc4xmpCore:ExtDescrAccessibility",
"photoshop:City",
"photoshop:State",
"photoshop:Country",
"xmpRights:Marked",
"xmpRights:UsageTerms",
"xmpRights:WebStatement",
"lc:*",
];
/// Properties replaced only when LightCraft has a value (capture time; GPS, which LightCraft
/// doesn't read from sidecars): otherwise the sidecar's own value stays. Anything else LightCraft
/// writes (`xmp:CreatorTool`) goes only into new sidecars.
pub const OWNED_IF_STATED: &[&str] = &["exif:DateTimeOriginal", "photoshop:DateCreated", "exif:GPSLatitude", "exif:GPSLongitude"];
/// Write `data` to `path` atomically: a temp file next to it, synced, then renamed over it.
fn write_atomic(path: &Path, data: &[u8]) -> std::io::Result<()> {
lightcraft_catalog::safe_file::write_atomic(path, data)
}
/// Copy `path` to a new `<name>.bak-<stamp>` (`-1`, `-2`… if taken; never overwriting).
fn backup(path: &Path, stamp: &str) -> std::io::Result<PathBuf> {
use std::io::Write;
let bytes = std::fs::read(path)?;
let stamp: String = stamp.chars().filter(char::is_ascii_alphanumeric).collect();
let name = path.file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
for k in 0..1000 {
let b = path.with_file_name(if k == 0 { format!("{name}.bak-{stamp}") } else { format!("{name}.bak-{stamp}-{k}") });
match std::fs::OpenOptions::new().write(true).create_new(true).open(&b) {
Ok(mut f) => {
f.write_all(&bytes)?;
f.sync_all()?;
return Ok(b);
}
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(std::io::Error::other("no free backup name"))
}
/// What [`Session::save_sidecar`] did.
#[derive(Clone, Debug, PartialEq)]
pub struct SidecarSaved {
pub path: PathBuf,
/// An existing sidecar was merged into (its other contents kept).
pub merged: bool,
/// An existing sidecar that couldn't be read as XMP was copied here before being replaced.
pub backup: Option<PathBuf>,
}
/// The new contents for the sidecar at `path`: `fresh` merged into the existing file, if any.
/// Returns (contents, merged, backup).
fn sidecar_contents(path: &Path, fresh: String, stamp: &str) -> std::io::Result<(String, bool, Option<PathBuf>)> {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((fresh, false, None)),
// can't see what is there: never overwrite it blindly
Err(e) => return Err(e),
};
if bytes.iter().all(u8::is_ascii_whitespace) {
return Ok((fresh, false, None));
}
let rules = lightcraft_meta::MergeRules { owned: OWNED, owned_if_present: OWNED_IF_STATED };
match std::str::from_utf8(&bytes)
.map_err(|e| e.to_string())
.and_then(|old| lightcraft_meta::merge_xmp(old, &fresh, rules).map_err(|e| e.to_string()))
{
Ok(merged) => Ok((merged, true, None)),
Err(why) => {
let b = backup(path, stamp)?;
log::warn!("{}: not readable as XMP ({why}); kept a copy as {} before replacing it", path.display(), b.display());
Ok((fresh, false, Some(b)))
}
}
}
/// Key of the stem sidecar a file maps to (case-insensitive).
fn stem_key(original: &str) -> String {
sidecar_path(original, SidecarNaming::Stem).to_string_lossy().to_lowercase()
}
fn file_path(p: &Photo) -> Option<&str> {
match &p.source {
Source::File { path } => Some(path),
@@ -298,24 +401,70 @@ pub(crate) fn op_photos(op: &Op, out: &mut Vec<PhotoId>) {
}
}
/// Which photo owns each stem sidecar that several catalogued files map to (see
/// [`Session::sidecar_naming`]).
pub(crate) struct StemOwners(HashMap<String, PhotoId>);
impl StemOwners {
pub(crate) fn of(cat: &lightcraft_catalog::Catalog) -> StemOwners {
// per stem: (raw first, then file name, then id) of the best candidate; and how many
let mut best: HashMap<String, ((bool, String, u64), PhotoId, usize)> = HashMap::new();
for p in cat.photos().filter(|p| p.copy_of.is_none()) {
let Some(path) = file_path(p) else { continue };
let rank = (p.kind != MediaKind::Raw, path.to_lowercase(), p.id.0);
let e = best.entry(stem_key(path)).or_insert_with(|| (rank.clone(), p.id, 0));
e.2 += 1;
if rank < e.0 {
(e.0, e.1) = (rank, p.id);
}
}
StemOwners(best.into_iter().filter(|(_, v)| v.2 > 1).map(|(k, v)| (k, v.1)).collect())
}
/// `naming` for photo `p`, except that a stem sidecar another file owns becomes Full.
fn naming(&self, p: &Photo, naming: SidecarNaming) -> SidecarNaming {
match (naming, file_path(p)) {
(SidecarNaming::Stem, Some(path)) if self.0.get(&stem_key(path)).is_some_and(|owner| *owner != p.id) => SidecarNaming::Full,
_ => naming,
}
}
}
impl Session {
/// Write the photo's XMP sidecar; returns its path.
pub fn save_sidecar(&self, id: PhotoId) -> Result<PathBuf> {
/// The sidecar naming for photo `id`: the library's, except under Stem naming when another
/// catalogued file shares the stem (`IMG_0001.CR3` + `IMG_0001.JPG`) and owns the stem
/// sidecar — a raw first, else the first by file name. The others use Full naming
/// (`IMG_0001.JPG.xmp`), so their metadata never overwrites each other.
pub fn sidecar_naming(&self, id: PhotoId) -> SidecarNaming {
match self.catalog.photo(id) {
Some(p) if self.xmp.naming == SidecarNaming::Stem => StemOwners::of(&self.catalog).naming(p, self.xmp.naming),
_ => self.xmp.naming,
}
}
/// Write the photo's XMP sidecar, merged into an existing one (see the module docs).
pub fn save_sidecar(&self, id: PhotoId) -> Result<SidecarSaved> {
self.save_sidecar_with(id, &StemOwners::of(&self.catalog))
}
pub(crate) fn save_sidecar_with(&self, id: PhotoId, owners: &StemOwners) -> Result<SidecarSaved> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
if p.copy_of.is_some() {
return Err(EngineError::Other(format!("{} is a virtual copy: its settings live only in the library", p.file_name)));
}
let orig = file_path(p).ok_or_else(|| EngineError::Other(format!("{} is not a file on disk", p.file_name)))?;
let path = sidecar_path(orig, self.xmp.naming);
write_atomic(&path, sidecar_packet(p, &self.catalog).as_bytes()).map_err(|e| EngineError::Other(format!("{}: {e}", path.display())))?;
Ok(path)
let path = sidecar_path(orig, owners.naming(p, self.xmp.naming));
let err = |e: std::io::Error| EngineError::Other(format!("{}: {e}", path.display()));
let (data, merged, backup) = sidecar_contents(&path, sidecar_packet(p, &self.catalog), &(self.clock)()).map_err(err)?;
write_atomic(&path, data.as_bytes()).map_err(err)?;
Ok(SidecarSaved { path, merged, backup })
}
/// The op that applies a photo's sidecar (or embedded XMP) to the catalog, if there is one.
pub fn read_sidecar_op(&self, id: PhotoId) -> Result<Option<(Op, PathBuf)>> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
let Some(orig) = file_path(p) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.xmp.naming) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.sidecar_naming(id)) else { return Ok(None) };
let sc = parse_sidecar(&packet, p.kind == MediaKind::Raw)
.map_err(|e| EngineError::Other(format!("{}: {e}", from.display())))?
.resolve_label(&self.catalog);
@@ -340,9 +489,13 @@ impl Session {
pub(crate) fn auto_write_sidecars(&self, ops: &[Op]) {
let mut ids = Vec::new();
ops.iter().for_each(|o| op_photos(o, &mut ids));
if ids.is_empty() {
return;
}
let owners = StemOwners::of(&self.catalog);
for id in ids {
if self.catalog.photo(id).is_some_and(|p| file_path(p).is_some() && p.copy_of.is_none())
&& let Err(e) = self.save_sidecar(id)
&& let Err(e) = self.save_sidecar_with(id, &owners)
{
log::warn!("auto-write XMP: {e}");
}
+109 -13
View File
@@ -1,13 +1,16 @@
//! Smart previews: a compact proxy of each photo (its decoded preview-size source, about 1 MB)
//! kept in the library, so photos stay editable — and exportable at proxy size — while their
//! originals are offline (an unplugged drive). The proxy is the scene-linear source scaled into
//! 0..1 by a stored factor, sRGB-encoded and saved as a JPEG after a one-line header.
//! 0..1 by a stored factor, sRGB-encoded and saved as a JPEG after a one-line header. The header
//! also keeps the decoder's file-local camera tone curve (Sony ARW camera look), which is applied
//! at render time and is not baked into the pixels.
use std::path::{Path, PathBuf};
use std::sync::Arc;
use lightcraft_catalog::Photo;
use lightcraft_color::transfer::linear_to_srgb;
use lightcraft_pipeline::tone::CameraTone;
use lightcraft_raster::{Rgb32f, Rgba8};
const MAGIC: &[u8] = b"LCSP1\n";
@@ -18,8 +21,11 @@ pub fn file_name(p: &Photo) -> String {
format!("{:032x}.lcsp", h.0)
}
/// Encode a source image as a smart preview.
pub fn encode(img: &Rgb32f) -> Result<Vec<u8>, String> {
/// The most header [`is_valid`] reads (the JSON line before the JPEG).
const HEADER_MAX: u64 = 4096;
/// Encode a source image (and the decoder's camera tone curve, if any) as a smart preview.
pub fn encode(img: &Rgb32f, tone: Option<&CameraTone>) -> Result<Vec<u8>, String> {
// scale so all but the brightest 0.05 % fit into 0..1
let mut lum: Vec<f32> = img.data.iter().map(|c| c[0].max(c[1]).max(c[2])).filter(|v| v.is_finite()).collect();
let scale = if lum.is_empty() {
@@ -47,22 +53,29 @@ pub fn encode(img: &Rgb32f) -> Result<Vec<u8>, String> {
)
.map_err(|e| e.to_string())?;
let mut out = MAGIC.to_vec();
out.extend_from_slice(format!("{{\"w\":{},\"h\":{},\"scale\":{scale}}}\n", img.width, img.height).as_bytes());
let mut head = serde_json::json!({"w": img.width, "h": img.height, "scale": scale});
if let Some(t) = tone {
head["tone"] = serde_json::to_value(t).map_err(|e| e.to_string())?;
}
out.extend_from_slice(head.to_string().as_bytes());
out.push(b'\n');
out.extend_from_slice(&jpg);
Ok(out)
}
/// Decode a smart preview back into a source image.
pub fn decode(bytes: &[u8]) -> Result<Rgb32f, String> {
/// Decode a smart preview back into a source image and its stored camera tone curve (an invalid
/// curve is ignored, like a missing one).
pub fn decode(bytes: &[u8]) -> Result<(Rgb32f, Option<CameraTone>), String> {
let rest = bytes.strip_prefix(MAGIC).ok_or("not a smart preview")?;
let nl = rest.iter().position(|b| *b == b'\n').ok_or("bad smart preview")?;
let head: serde_json::Value = serde_json::from_slice(&rest[..nl]).map_err(|e| e.to_string())?;
let scale = head["scale"].as_f64().unwrap_or(1.0) as f32;
let tone = head.get("tone").and_then(|t| serde_json::from_value::<CameraTone>(t.clone()).ok());
let d = lightcraft_codecs::decode(&rest[nl + 1..], Default::default()).map_err(|e| e.to_string())?;
// the decoder undoes the sRGB encoding; the values are the source's own primaries
let mut img = d.image;
img.data.iter_mut().for_each(|c| *c = c.map(|v| v * scale));
Ok(img)
Ok((img, tone))
}
/// Where a library keeps its smart previews.
@@ -130,11 +143,16 @@ pub fn migrate(from: &Path, to: &Path, what: Existing) -> (usize, Vec<String>) {
Existing::Discard => std::fs::remove_file(&src),
_ => {
let dst = to.join(e.file_name());
if dst.exists() {
// named by content: the copy already there is the same preview
if is_valid(&dst) {
// named by content: the complete copy already there is the same preview
std::fs::remove_file(&src)
} else {
std::fs::rename(&src, &dst).or_else(|_| std::fs::copy(&src, &dst).and_then(|_| std::fs::remove_file(&src)))
// across drives: copied atomically (never a partial proxy at the new place)
std::fs::rename(&src, &dst).or_else(|_| {
std::fs::read(&src)
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&dst, &b))
.and_then(|()| std::fs::remove_file(&src))
})
}
}
};
@@ -146,10 +164,33 @@ pub fn migrate(from: &Path, to: &Path, what: Existing) -> (usize, Vec<String>) {
(done, failed)
}
/// A cheap check that the smart preview at `path` is complete: our header, then a JPEG from its
/// start marker to its end marker. A file cut short by a crash, a full drive or an unplugged one
/// fails it (and is then rebuilt by Build Smart Previews, not counted as there).
pub fn is_valid(path: &Path) -> bool {
use std::io::{Read, Seek, SeekFrom};
let Ok(mut f) = std::fs::File::open(path) else { return false };
let mut head = Vec::with_capacity(1024);
if (&mut f).take(HEADER_MAX).read_to_end(&mut head).is_err() {
return false;
}
let Some(rest) = head.strip_prefix(MAGIC) else { return false };
let Some(nl) = rest.iter().position(|b| *b == b'\n') else { return false };
let size_ok = rest
.get(..nl)
.and_then(|h| serde_json::from_slice::<serde_json::Value>(h).ok())
.is_some_and(|h| h["w"].as_u64().is_some_and(|w| w > 0) && h["h"].as_u64().is_some_and(|h| h > 0));
if !size_ok || rest.get(nl + 1..nl + 3) != Some(&[0xFF, 0xD8][..]) {
return false;
}
let mut end = [0u8; 2];
f.seek(SeekFrom::End(-2)).is_ok() && f.read_exact(&mut end).is_ok() && end == [0xFF, 0xD9]
}
/// Load the proxy at `path`.
pub fn load(path: &Path) -> Result<Arc<Rgb32f>, String> {
pub fn load(path: &Path) -> Result<crate::media::DecodedSource, String> {
let b = std::fs::read(path).map_err(|e| format!("{}: {e}", path.display()))?;
decode(&b).map(Arc::new)
decode(&b).map(|(image, camera_tone)| crate::media::DecodedSource { image: Arc::new(image), info: None, camera_tone })
}
#[cfg(test)]
@@ -193,11 +234,66 @@ mod tests {
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn damaged_proxies_are_invalid_and_never_kept_over_good_ones() {
let base = temp("valid");
let (a, b) = (base.join("a"), base.join("b"));
std::fs::create_dir_all(&a).unwrap();
std::fs::create_dir_all(&b).unwrap();
let img = lightcraft_scenes::demo_library()[0].render(64, 40);
let good = encode(&img, None).unwrap();
std::fs::write(a.join("1.lcsp"), &good).unwrap();
assert!(is_valid(&a.join("1.lcsp")));
for cut in [good.len() - 1, good.len() / 2, 10, 0] {
std::fs::write(b.join("1.lcsp"), &good[..cut]).unwrap();
assert!(!is_valid(&b.join("1.lcsp")), "cut at {cut}");
}
assert!(!is_valid(&b.join("missing.lcsp")));
// moving onto a damaged copy replaces it instead of deleting the good one
assert_eq!(migrate(&a, &b, Existing::Move).0, 1);
assert_eq!(std::fs::read(b.join("1.lcsp")).unwrap(), good);
let _ = std::fs::remove_dir_all(&base);
}
#[test]
fn camera_tone_travels_with_the_proxy() {
let tone = CameraTone::new(std::array::from_fn(|i| {
let x = 0.004 * 1.17f32.powi(i as i32);
[x, 0.95 * (1.0 - (-2.7 * x).exp())]
}))
.unwrap();
let img = lightcraft_scenes::demo_library()[0].render(64, 40);
let bytes = encode(&img, Some(&tone)).unwrap();
assert_eq!(decode(&bytes).unwrap().1, Some(tone));
let dir = temp("tone");
let path = dir.join("t.lcsp");
std::fs::write(&path, &bytes).unwrap();
// the longer header is still read by the validity check, and the curve by `load`
assert!(is_valid(&path));
let loaded = load(&path).unwrap();
assert_eq!(loaded.camera_tone, Some(tone));
let header = lightcraft_pipeline::SourceInfo { raw: true, ..Default::default() };
assert_eq!(loaded.info_or(header).camera_tone, Some(tone));
// a hostile curve (reversing knots) is ignored, not trusted
let nl = MAGIC.len() + bytes[MAGIC.len()..].iter().position(|b| *b == b'\n').unwrap();
let mut head: serde_json::Value = serde_json::from_slice(&bytes[MAGIC.len()..nl]).unwrap();
let mut knots: Vec<[f32; 2]> = serde_json::from_value(head["tone"]["knots"].clone()).unwrap();
knots.swap(3, 4);
head["tone"]["knots"] = serde_json::json!(knots);
let mut hostile = MAGIC.to_vec();
hostile.extend_from_slice(head.to_string().as_bytes());
hostile.extend_from_slice(&bytes[nl..]);
let (pixels, tone) = decode(&hostile).unwrap();
assert_eq!((pixels.width, tone), (img.width, None));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn roundtrip_keeps_the_picture() {
let img = lightcraft_scenes::demo_library()[0].render(160, 100);
let back = decode(&encode(&img).unwrap()).unwrap();
let (back, tone) = decode(&encode(&img, None).unwrap()).unwrap();
assert_eq!((back.width, back.height), (img.width, img.height));
assert!(tone.is_none());
let mut err = 0.0f64;
for (a, b) in img.data.iter().zip(&back.data) {
for k in 0..3 {
+24
View File
@@ -10,6 +10,30 @@ fn active_dev(s: &Session) -> lightcraft_develop::DevelopSettings {
(*s.develop_of(s.active().unwrap()).unwrap()).clone()
}
#[test]
fn changing_one_wb_control_resolves_as_shot_without_stale_tint() {
use lightcraft_catalog::{Photo, PhotoId, Source};
use lightcraft_develop::{DevelopSettings, WbMode};
let mut s = demo();
let id = PhotoId(100);
let mut p = Photo::new(id, Source::File { path: "synthetic.arw".into() }, "synthetic.arw", "ARW", 16, 16, "");
// A catalog made by the old generic-matrix Kelvin inference.
p.as_shot_wb = Some((6829.0, -127.0));
p.develop = std::sync::Arc::new(DevelopSettings::for_raw(6829.0, -127.0));
s.catalog.apply(lightcraft_catalog::Op::AddPhoto { photo: Box::new(p) }).unwrap();
s.execute("library.select", &json!({"ids": [100], "active": 100})).unwrap();
s.execute("develop.set", &json!({"control": "wb.temp", "value": 8000})).unwrap();
let d = active_dev(&s);
assert_eq!(d.wb.mode, WbMode::Custom);
assert_eq!(d.wb.temp, 8000.0);
assert_eq!(d.wb.tint, 0.0, "untouched tint comes from the current as-shot reference");
s.execute("develop.wb", &json!({"mode": "asShot"})).unwrap();
s.execute("develop.set", &json!({"control": "wb.tint", "value": 10})).unwrap();
assert_eq!(active_dev(&s).wb.temp, 6500.0);
s.execute("develop.reset", &json!({})).unwrap();
assert_eq!((active_dev(&s).wb.temp, active_dev(&s).wb.tint), (6500.0, 0.0));
}
#[test]
fn demo_library_loads() {
let mut s = demo();
+21
View File
@@ -323,3 +323,24 @@ fn a_failed_export_write_keeps_the_previous_file() {
assert_ne!(std::fs::read(&prev).unwrap(), b"last week's export");
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #134: exports can always be made again, so they are written atomically but not synced to
/// disk file by file; an edit copy that becomes a library photo still is.
#[test]
fn exports_are_atomic_without_a_sync() {
use crate::export::{Conflict, Destination};
use lightcraft_catalog::safe_file::syncs_on_this_thread;
let (mut s, id, dir, _src, _) = library_with_jpeg("export-nosync");
let out = dir.join("out");
let to = Destination { dir: out.to_string_lossy().to_string(), exact: None };
let o = ExportOptions { conflict: Conflict::Unique, ..Default::default() };
let before = syncs_on_this_thread();
for _ in 0..3 {
disk_batch(&mut s, id, &o, &to).unwrap();
}
assert_eq!(syncs_on_this_thread(), before, "no per-file sync for exports");
assert_eq!(std::fs::read_dir(&out).unwrap().count(), 3, "three exports, no temp files");
crate::export::write_file_durable(&out.join("edit.tif").to_string_lossy(), b"tiff").unwrap();
assert!(syncs_on_this_thread() > before, "the durable writer syncs");
let _ = std::fs::remove_dir_all(&dir);
}
+210
View File
@@ -144,6 +144,99 @@ fn browsing_a_folder_lists_its_photos_without_adding_them() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Photos a.png, b.png, c.png imported from `old`, then: a renamed into `moved`; b renamed, with
/// an impostor (same name and size, other bytes) under its old name; c gone, with two impostors.
fn renamed_and_ambiguous_scene(tag: &str) -> (std::path::PathBuf, Session) {
let dir = temp_dir(tag);
std::fs::create_dir_all(dir.join("old")).unwrap();
std::fs::create_dir_all(dir.join("moved/real")).unwrap();
for (n, seed) in [("a.png", 1), ("b.png", 2), ("c.png", 3)] {
write_png(&dir.join("old").join(n), seed);
}
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [dir.join("old").to_string_lossy()]})).unwrap();
let impostor = |src: &Path, dst: &Path| {
let mut b = std::fs::read(src).unwrap();
let k = b.len() - 20;
b[k] ^= 0xff;
std::fs::write(dst, b).unwrap();
};
std::fs::rename(dir.join("old/a.png"), dir.join("moved/Trip-001.png")).unwrap();
impostor(&dir.join("old/b.png"), &dir.join("moved/b.png"));
std::fs::rename(dir.join("old/b.png"), dir.join("moved/real/Trip-002.png")).unwrap();
impostor(&dir.join("old/c.png"), &dir.join("moved/c.png"));
impostor(&dir.join("old/c.png"), &dir.join("moved/real/c.png"));
std::fs::remove_file(dir.join("old/c.png")).unwrap();
(dir, s)
}
/// Issue #105: Find Missing finds renamed files by content, prefers the content match over a
/// same-name same-size impostor, and skips (reports) photos it can't tell apart.
#[test]
fn find_missing_matches_renamed_files_by_content() {
let (dir, mut s) = renamed_and_ambiguous_scene("missing-content");
let r = s.execute("library.findMissing", &json!({"folder": dir.join("moved").to_string_lossy()})).unwrap();
let found = r["found"].as_array().unwrap();
assert_eq!(found.len(), 2, "{r}");
let to = |name: &str| {
found.iter().find(|f| f["from"].as_str().unwrap().ends_with(name)).map(|f| (f["to"].as_str().unwrap().to_string(), f["by"].clone()))
};
assert_eq!(to("a.png"), Some((dir.join("moved/Trip-001.png").to_string_lossy().to_string(), json!("content"))));
assert_eq!(to("b.png"), Some((dir.join("moved/real/Trip-002.png").to_string_lossy().to_string(), json!("content"))), "not the impostor");
assert_eq!(r["missing"], 1);
assert_eq!(r["ambiguous"][0]["candidates"].as_array().map(Vec::len), Some(2), "{r}");
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #104: the app plans Find Missing on a worker thread (no session) and relinks the plan
/// afterwards; that gives exactly what the synchronous command gives — renamed files by content,
/// impostors and look-alikes skipped (`ambiguous`).
#[test]
fn find_missing_planned_on_a_worker_matches_the_command() {
use crate::cmd::missing::{find_candidates, plan_find_missing};
let (dir, mut s) = renamed_and_ambiguous_scene("missing-worker");
let folder = dir.join("moved").to_string_lossy().to_string();
let candidates = find_candidates(&s.catalog);
let f = folder.clone();
let plan = std::thread::spawn(move || plan_find_missing(&candidates, &f)).join().unwrap().unwrap();
assert_eq!((plan.found.len(), plan.missing, plan.ambiguous.len()), (2, 1, 1), "{plan:?}");
let applied = s.execute("library.findMissing", &plan.to_json()).unwrap();
s.execute("edit.undo", &json!({})).unwrap();
assert_eq!(s.execute("library.missing", &json!({})).unwrap().as_array().map(Vec::len), Some(3), "undo restores all three");
let direct = s.execute("library.findMissing", &json!({"folder": folder})).unwrap();
assert_eq!(applied, direct);
assert_eq!(direct["found"].as_array().map(Vec::len), Some(2), "{direct}");
assert_eq!(direct["ambiguous"].as_array().map(Vec::len), Some(1), "{direct}");
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #104: a plan made on a worker is re-checked when it is applied: a photo relinked
/// meanwhile is left alone, and a file that is now another photo's original isn't used.
#[test]
fn find_missing_plan_is_rechecked_when_applied() {
use crate::cmd::missing::{find_candidates, plan_find_missing};
let (dir, mut s) = renamed_and_ambiguous_scene("missing-recheck");
let plan = plan_find_missing(&find_candidates(&s.catalog), &dir.join("moved").to_string_lossy()).unwrap();
let to = |name: &str| plan.found.iter().find(|f| f.from.ends_with(name)).unwrap().clone();
let (a, b) = (to("a.png"), to("b.png"));
// meanwhile: a is relinked by hand (elsewhere), and the ambiguous c is pointed at b's file
std::fs::copy(&a.to, dir.join("a-elsewhere.png")).unwrap();
s.execute("photo.relink", &json!({"id": a.id.0, "path": dir.join("a-elsewhere.png").to_string_lossy()})).unwrap();
let c = plan.ambiguous[0].id;
s.execute("photo.relink", &json!({"id": c.0, "path": b.to})).unwrap();
let r = s.execute("library.findMissing", &plan.to_json()).unwrap();
assert_eq!(r["found"].as_array().map(Vec::len), Some(0), "{r}");
assert_eq!(r["ambiguous"].as_array().map(Vec::len), Some(0), "c isn't missing any more: {r}");
assert_eq!(r["missing"], 1, "b stays missing: its file is c's now: {r}");
let path_of = |id: lightcraft_catalog::PhotoId| match &s.catalog.photo(id).unwrap().source {
lightcraft_catalog::Source::File { path } => path.clone(),
_ => String::new(),
};
assert_eq!(path_of(a.id), dir.join("a-elsewhere.png").to_string_lossy());
assert_eq!(path_of(b.id), b.from);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn missing_files_are_found_and_relinked() {
let dir = std::env::temp_dir().join(format!("lc-missing-{}", std::process::id()));
@@ -352,6 +445,58 @@ fn smart_previews_stand_in_for_offline_originals() {
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #106: a smart preview cut short (crash, full drive) counted as built forever and failed
/// exactly when the original went offline. Writes are atomic now, and a damaged proxy is not
/// reported as present and is rebuilt.
#[test]
fn damaged_smart_previews_are_rebuilt_and_failed_writes_leave_none() {
let src = temp_dir("smartdmg-src");
let lib = temp_dir("smartdmg-lib");
write_png(&src.join("a.png"), 5);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("a.png").to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
let dir = s.media.smart_dir.clone().unwrap();
let file = dir.join(crate::smart::file_name(s.catalog.photo(id).unwrap()));
{
let _fault = lightcraft_catalog::safe_file::fail_writes_after(64);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["failed"].as_array().map(Vec::len)), (Some(0), Some(1)), "{r}");
}
assert!(!file.exists(), "no partial proxy");
assert_eq!(std::fs::read_dir(&dir).unwrap().count(), 0, "no temp file left");
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], false);
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(0)), "{r}");
assert!(crate::smart::is_valid(&file));
// cut short, as by a crash before the fix
let full = std::fs::read(&file).unwrap();
std::fs::write(&file, &full[..full.len() / 2]).unwrap();
assert!(!crate::smart::is_valid(&file));
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], false, "not counted as there");
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole");
// the app's background build (issue #104) checks and repairs the same way, on a worker thread
std::fs::write(&file, &full[..full.len() / 2]).unwrap();
let r = s.execute("library.smartPreviews", &json!({"background": true})).unwrap();
assert_eq!(r["what"], "smart previews", "{r}");
let t0 = std::time::Instant::now();
let r = loop {
let r = s.execute("library.previewProgress", &json!({})).unwrap();
if r["running"] == false || t0.elapsed() > std::time::Duration::from_secs(60) {
break r;
}
std::thread::sleep(std::time::Duration::from_millis(5));
};
assert_eq!((r["done"].as_u64(), r["failed"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(0), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole in the background");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// LUT profiles: a .cube imports as a creative profile (into the library), renders differently
/// (Amount 0 = no change), is listed in the profile browser and comes back with the library.
#[test]
@@ -441,6 +586,70 @@ fn folders_rename_and_move_with_their_photos() {
let _ = std::fs::remove_dir_all(&root);
}
/// Rename / Move Folder are undo steps (issue #97): undo renames the folder back on disk (files and
/// sidecars along) and restores the photos' paths, redo repeats it; a destination taken in the
/// meantime is never overwritten — the step is refused, reported and stays on the stack.
#[test]
fn folder_rename_and_move_undo_and_redo_on_disk() {
let root = temp_dir("folders-undo");
let a = root.join("Trip");
write_png(&a.join("one.png"), 3);
std::fs::write(a.join("one.xmp"), "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>").unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [a.to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
let path = |s: &Session| match &s.catalog.photo(id).unwrap().source {
lightcraft_catalog::Source::File { path } => path.clone(),
_ => panic!(),
};
let undo0 = s.undo.len();
let b = root.join("Italy");
s.execute("folder.rename", &json!({"path": a.to_string_lossy(), "name": "Italy"})).unwrap();
assert_eq!(s.undo.len(), undo0 + 1, "one undo step");
assert!(s.undo.last().unwrap().label.contains("Rename Folder"));
assert!(b.join("one.png").exists() && !a.exists());
s.execute("edit.undo", &json!({})).unwrap();
assert!(a.join("one.png").exists() && a.join("one.xmp").exists() && !b.exists(), "renamed back on disk");
assert_eq!(path(&s), a.join("one.png").to_string_lossy());
s.media.forget(id);
assert!(s.render_now(id, 16, 16).is_ok());
s.execute("edit.redo", &json!({})).unwrap();
assert!(b.join("one.png").exists() && b.join("one.xmp").exists() && !a.exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
// the old name is taken now: undo is refused, nothing is overwritten, the step stays
std::fs::create_dir_all(&a).unwrap();
std::fs::write(a.join("keep.txt"), "mine").unwrap();
let undo_n = s.undo.len();
let e = s.execute("edit.undo", &json!({})).unwrap_err().to_string();
assert!(e.contains("already exists"), "{e}");
assert_eq!(s.undo.len(), undo_n, "the step stays on the stack");
assert_eq!(std::fs::read_to_string(a.join("keep.txt")).unwrap(), "mine");
assert!(b.join("one.png").exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
std::fs::remove_dir_all(&a).unwrap();
// move into another folder, undo, redo
let into = root.join("Archive");
s.execute("folder.move", &json!({"path": b.to_string_lossy(), "into": into.to_string_lossy()})).unwrap();
let moved = into.join("Italy");
assert_eq!(path(&s), moved.join("one.png").to_string_lossy());
s.execute("edit.undo", &json!({})).unwrap();
assert!(b.join("one.png").exists() && !moved.exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
// the redo target is taken (even by an empty folder): refused, nothing replaced
std::fs::create_dir_all(&moved).unwrap();
assert!(s.execute("edit.redo", &json!({})).is_err());
assert!(b.join("one.png").exists());
std::fs::remove_dir(&moved).unwrap();
s.execute("edit.redo", &json!({})).unwrap();
assert!(moved.join("one.png").exists() && moved.join("one.xmp").exists() && !b.exists());
assert_eq!(path(&s), moved.join("one.png").to_string_lossy());
let _ = std::fs::remove_dir_all(&root);
}
/// Assisted culling: a burst (same scene, seconds apart) of a sharp and two soft shots, plus an
/// unrelated photo: the burst is grouped, the sharp one is its best, blurry shots can be rejected.
#[test]
@@ -590,6 +799,7 @@ fn smart_previews_folder_is_chosen_per_library() {
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], true);
// the setting is saved with the library
drop(s); // one session per library (issue #99)
let mut again = Session::new().with_fs();
again.open_library(&lib, false).unwrap();
assert_eq!(again.execute("library.smartPreviewsLocation", &json!({})).unwrap()["path"], to.to_string_lossy().as_ref());
+157
View File
@@ -163,6 +163,111 @@ fn a_failed_or_corrupt_copy_keeps_the_source() {
}
}
/// Issue #96: Import → Copy used a plain unverified copy (and check-then-copy naming that could
/// replace a file). It now gets the same verified copy as Move: a copy that fails or comes out
/// different is removed and reported as a failed import, nothing is catalogued, and the card
/// is untouched; a good copy never replaces an existing file.
#[test]
fn copy_import_verifies_each_copy() {
let import_copy = |s: &mut Session, card: &Path, dest: &Path| {
s.execute(
"library.import",
&json!({"paths": [card.to_string_lossy()], "mode": "copy", "destination": dest.to_string_lossy(), "organize": "flat"}),
)
.unwrap()
};
for fault in [Fault::FailWrite, Fault::Corrupt] {
let base = temp_dir(&format!("copyfail-{fault:?}"));
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 3);
let bytes = std::fs::read(card.join("a.png")).unwrap();
let mut s = session();
inject(fault);
let r = import_copy(&mut s, &card, &dest);
inject(Fault::None);
assert_eq!(len(&r, "failed"), 1, "{fault:?}: {r}");
assert_eq!(len(&r, "imported"), 0, "{fault:?}: {r}");
assert!(r.to_string().contains("the original is untouched"), "{r}");
assert_eq!(s.catalog.len(), 0);
assert_eq!(std::fs::read(card.join("a.png")).unwrap(), bytes, "{fault:?}: source intact");
assert_eq!(files_under(&dest), Vec::<String>::new(), "{fault:?}: no bad copy left");
let _ = std::fs::remove_dir_all(&base);
}
// a good copy: verified, and a taken name gets -1 instead of being replaced
let base = temp_dir("copyok");
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 4);
std::fs::create_dir_all(&dest).unwrap();
std::fs::write(dest.join("a.png"), b"someone else's file").unwrap();
let mut s = session();
let r = import_copy(&mut s, &card, &dest);
assert_eq!(len(&r, "imported"), 1, "{r}");
assert_eq!(std::fs::read(dest.join("a.png")).unwrap(), b"someone else's file");
assert_eq!(std::fs::read(dest.join("a-1.png")).unwrap(), std::fs::read(card.join("a.png")).unwrap());
let _ = std::fs::remove_dir_all(&base);
}
/// Copy verifies against the probe's content hash (issue #134): the hash, not a re-read of the
/// source, is the reference — a copy that matches the source but not the hash is refused, a
/// corrupted copy is caught, and the file hash agrees with `hash_bytes` across read chunks.
#[test]
fn copy_is_verified_against_the_probe_hash() {
use crate::import_move::copy_verified;
let base = temp_dir("copyhash");
let src = base.join("src.bin");
// larger than one 1 MB read, not a multiple of it
let bytes: Vec<u8> = (0..(3 << 20) + 12_345u32).map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect();
std::fs::write(&src, &bytes).unwrap();
let good = lightcraft_preview::hash_bytes(&bytes);
copy_verified(&src, &base.join("ok.bin"), Some(good)).unwrap();
assert_eq!(std::fs::read(base.join("ok.bin")).unwrap(), bytes);
// a hash that doesn't match: refused (a byte compare with the source would have passed)
let stale = lightcraft_preview::hash_bytes(b"what the probe read earlier");
let e = copy_verified(&src, &base.join("stale.bin"), Some(stale)).unwrap_err();
assert!(e.to_string().contains("differs"), "{e}");
assert!(!base.join("stale.bin").exists(), "the refused copy is removed");
// a copy that comes out different: caught by the hash
inject(Fault::Corrupt);
let r = copy_verified(&src, &base.join("bad.bin"), Some(good));
inject(Fault::None);
assert!(r.is_err());
assert!(!base.join("bad.bin").exists());
assert_eq!(std::fs::read(&src).unwrap(), bytes, "the source is untouched");
let _ = std::fs::remove_dir_all(&base);
}
/// A file replaced after Review Import probed it is not imported on the strength of the old probe:
/// its copy no longer matches the probe's hash, so it's reported as failed and nothing is left.
#[test]
fn copy_import_refuses_a_file_changed_since_the_review() {
let base = temp_dir("copychanged");
let (card, dest) = (base.join("card"), base.join("out"));
write_png(&card.join("a.png"), 3);
let mut s = session();
let r = s.execute("library.importPreview", &json!({"paths": [card.to_string_lossy()]})).unwrap();
assert_eq!(r["scanned"], 1, "{r}");
write_png(&card.join("a.png"), 9); // other pixels
let r = s
.execute(
"library.import",
&json!({"paths": [card.to_string_lossy()], "mode": "copy", "destination": dest.to_string_lossy(), "organize": "flat"}),
)
.unwrap();
assert_eq!(len(&r, "imported"), 0, "{r}");
assert_eq!(len(&r, "failed"), 1, "{r}");
assert!(r.to_string().contains("changed meanwhile"), "{r}");
assert_eq!(files_under(&dest), Vec::<String>::new());
// importing again probes afresh and succeeds
let r = s
.execute(
"library.import",
&json!({"paths": [card.to_string_lossy()], "mode": "copy", "destination": dest.to_string_lossy(), "organize": "flat"}),
)
.unwrap();
assert_eq!(len(&r, "imported"), 1, "{r}");
let _ = std::fs::remove_dir_all(&base);
}
/// A source that can't be removed (a read-only card) stays and is reported; the photo is
/// imported from its copy.
#[test]
@@ -276,3 +381,55 @@ fn move_into_the_library_is_saved_before_sources_go() {
assert_eq!(photo_paths(&again), [want.to_string_lossy()]);
let _ = std::fs::remove_dir_all(&base);
}
/// Timing for issue #134 (not a pass/fail test): `cargo test -p lightcraft-engine --release
/// write_policy_timing -- --ignored --nocapture`. Writes N export-sized files durably (temp +
/// sync + rename) and atomically only, and copies N files verified byte for byte and against the
/// probe hash, alternating so a loaded machine affects both alike. `LIGHTCRAFT_BENCH_DIR` puts
/// the files on another volume (a USB drive or a NAS is where the difference shows).
#[test]
#[ignore]
fn write_policy_timing() {
use crate::import_move::copy_verified;
use lightcraft_catalog::safe_file::{write_atomic, write_atomic_nosync};
use std::time::{Duration, Instant};
let root = std::env::var_os("LIGHTCRAFT_BENCH_DIR").map(PathBuf::from).unwrap_or_else(std::env::temp_dir);
let base = root.join(format!("lc-write-policy-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).unwrap();
let data = |n: usize, seed: u32| -> Vec<u8> { (0..n as u32).map(|i| (i.wrapping_add(seed).wrapping_mul(2_654_435_761) >> 11) as u8).collect() };
let (n, size) = (200usize, 300 << 10);
let (mut synced, mut unsynced) = (Duration::ZERO, Duration::ZERO);
for i in 0..n {
let b = data(size, i as u32);
let t = Instant::now();
write_atomic(&base.join(format!("sync-{i}.jpg")), &b).unwrap();
synced += t.elapsed();
let t = Instant::now();
write_atomic_nosync(&base.join(format!("nosync-{i}.jpg")), &b).unwrap();
unsynced += t.elapsed();
}
println!("export {n} × {} KB: write_atomic (synced) {synced:?}, write_atomic_nosync {unsynced:?}", size >> 10);
let (n, size) = (40usize, 8 << 20);
let card = base.join("card");
std::fs::create_dir_all(&card).unwrap();
let hashes: Vec<_> = (0..n)
.map(|i| {
let b = data(size, 7 * i as u32);
std::fs::write(card.join(format!("{i}.raw")), &b).unwrap();
lightcraft_preview::hash_bytes(&b)
})
.collect();
let (mut bytewise, mut hashed) = (Duration::ZERO, Duration::ZERO);
for (i, h) in hashes.iter().enumerate() {
let src = card.join(format!("{i}.raw"));
let t = Instant::now();
copy_verified(&src, &base.join(format!("bytes-{i}.raw")), None).unwrap();
bytewise += t.elapsed();
let t = Instant::now();
copy_verified(&src, &base.join(format!("hash-{i}.raw")), Some(*h)).unwrap();
hashed += t.elapsed();
}
println!("copy {n} × {} MB: byte compare with the source {bytewise:?}, probe hash {hashed:?}", size >> 20);
let _ = std::fs::remove_dir_all(&base);
}
+31
View File
@@ -113,6 +113,7 @@ fn close_writes_snapshot_and_view_state() {
let expect = s.catalog.to_snapshot();
s.close_library().unwrap();
assert_eq!(std::fs::metadata(dir.join("catalog.log")).unwrap().len(), 0);
drop(s); // one session per library (issue #99)
let s2 = open(&dir, true);
let r = &s2.library.as_ref().unwrap().report;
@@ -178,6 +179,7 @@ fn new_library_without_seed_is_empty_and_compacts() {
assert!(p["snapshots"].as_u64() >= Some(1), "{p}");
assert!(p["lastSnapshot"]["bytes"].as_u64() > Some(0), "{p}");
assert!(p["lastSnapshot"]["totalMs"].as_f64() >= p["lastSnapshot"]["serializeMs"].as_f64(), "{p}");
drop(s); // one session per library (issue #99)
let s2 = open(&dir, true);
assert_eq!(s2.catalog.len(), 0, "an existing library is never seeded");
assert_eq!(s2.catalog.albums().count(), 1);
@@ -376,3 +378,32 @@ fn scale_100k_library_queries() {
worst.sort_by(|a, b| b.1.total_cmp(&a.1));
eprintln!("slowest: {:?}", &worst[..3]);
}
/// Issue #99: a library is open in one session at a time. A second opener is refused (nothing
/// read or written), the owning session can reopen it, and it is free again once closed.
#[test]
fn a_library_open_elsewhere_is_refused() {
let dir = temp_dir("locked");
let mut s = open(&dir, true);
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
let log = std::fs::read(dir.join("catalog.log")).unwrap();
let mut other = Session::new();
let e = other.open_library(&dir, true).unwrap_err();
assert!(matches!(e, crate::EngineError::LibraryInUse(_)), "{e:?}");
assert!(e.to_string().contains("already open in"), "{e}");
assert!(other.library.is_none());
assert_eq!(std::fs::read(dir.join("catalog.log")).unwrap(), log, "the refused opener wrote nothing");
// the session that has it open may reopen it (Settings → Open Library on the same folder)
s.close_library().unwrap();
s.open_library(&dir, true).unwrap();
assert!(other.open_library(&dir, true).is_err(), "still locked after reopening");
s.execute("photo.rate", &json!({"rating": 4})).unwrap();
let expect = s.catalog.to_snapshot();
drop(s);
other.open_library(&dir, true).unwrap();
assert_eq!(other.catalog.to_snapshot(), expect);
let _ = std::fs::remove_dir_all(&dir);
}
+47 -1
View File
@@ -17,6 +17,8 @@ struct Flaky {
fail: Arc<AtomicBool>,
/// Only snapshots fail (appends work).
snapshots_only: bool,
/// Only appends fail (a dead log handle; fresh files can still be written).
appends_only: bool,
}
impl Flaky {
@@ -39,7 +41,7 @@ impl Store for Flaky {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.failing() {
if self.failing() && !self.appends_only {
return Err(std::io::Error::other("disk full"));
}
self.files.write_atomic(name, data)
@@ -145,3 +147,47 @@ fn failed_snapshot_does_not_fail_the_command() {
store.set_failing(false);
assert_eq!(open(&store, false).catalog.albums().count(), 2);
}
/// Issue #101: quitting while the log can't be appended to still saves the queued changes in the
/// closing snapshot (a fresh file), and reports success; they are not appended again later.
#[test]
fn close_saves_queued_changes_in_the_snapshot_when_appends_fail() {
let store = Flaky { appends_only: true, ..Default::default() };
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
store.set_failing(true);
assert!(matches!(s.execute("photo.rate", &json!({"rating": 4})), Err(EngineError::NotSaved(_))));
assert!(matches!(s.execute("photo.flag", &json!({"flag": "pick"})), Err(EngineError::NotSaved(_))));
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.pending_log.is_empty());
let expect = s.catalog.to_snapshot();
drop(s);
store.set_failing(false);
let mut s2 = open(&store, true);
assert_eq!(s2.catalog.to_snapshot(), expect);
let p = s2.catalog.photo(id).unwrap();
assert_eq!((p.rating, p.flag), (4, Flag::Pick));
// later edits continue after the absorbed ops
s2.execute("photo.rate", &json!({"rating": 2})).unwrap();
let expect = s2.catalog.to_snapshot();
drop(s2);
assert_eq!(open(&store, true).catalog.to_snapshot(), expect);
}
/// When neither the log nor the snapshot can be written, closing fails with `NotSaved` and the
/// changes stay queued (a later retry can still save them).
#[test]
fn close_fails_when_nothing_can_be_saved() {
let store = Flaky::default();
let mut s = open(&store, true);
store.set_failing(true);
assert!(s.execute("photo.rate", &json!({"rating": 4})).is_err());
let e = s.close_library().unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)), "{e:?}");
assert_eq!(s.unsaved().map(|u| u.0), Some(1));
store.set_failing(false);
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
}
+2
View File
@@ -126,11 +126,13 @@ fn preferences_persist_with_the_library() {
assert_eq!(r["persistent"], true);
// export presets are library preferences too
s.execute("export.savePreset", &json!({"name": "Proof", "params": {"format": "jpeg", "percent": 25}})).unwrap();
drop(s); // one session per library (issue #99)
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
assert_eq!(s.export_params(&json!({"preset": "Proof"})).unwrap(), json!({"format": "jpeg", "percent": 25}));
// older prefs.json files (snake_case `last_export`, no import section) still load
std::fs::write(lib.join("prefs.json"), r#"{"xmp": {"autoWrite": false}, "last_export": {"format": "png"}}"#).unwrap();
drop(s);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
assert_eq!(s.last_export, Some(json!({"format": "png"})));
+97
View File
@@ -0,0 +1,97 @@
//! Issue #103: a damaged or unreadable settings file (prefs.json, presets.json, view.json) is
//! never silently replaced by the defaults: a damaged one is kept aside, an unreadable one is not
//! written this session, and the user is told.
use std::sync::{Arc, Mutex};
use lightcraft_catalog::{MemStore, Store};
use serde_json::json;
use crate::Session;
use crate::library::LibraryStores;
fn temp_dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-settings-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn damaged_prefs_are_kept_aside_and_reported() {
let lib = temp_dir("damaged");
let bad = br#"{"exportPresets": [{"name": "Proof", "par"#;
std::fs::write(lib.join("prefs.json"), bad).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
let kept: Vec<_> = std::fs::read_dir(&lib)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().to_string())
.filter(|n| n.starts_with("prefs.json.corrupt-"))
.collect();
assert_eq!(kept.len(), 1, "{kept:?}");
assert_eq!(std::fs::read(lib.join(&kept[0])).unwrap(), bad);
let warnings = s.take_library_warnings();
assert_eq!(warnings.len(), 1);
assert!(warnings[0].contains("prefs.json is damaged") && warnings[0].contains(&kept[0]), "{warnings:?}");
assert!(s.take_library_warnings().is_empty(), "each warning is handed out once");
let info = s.execute("library.info", &json!({})).unwrap();
assert_eq!(info["settingsWarnings"].as_array().map(Vec::len), Some(1), "{info}");
// saving goes on (the damaged original is safe in its copy)
s.execute("library.preferences", &json!({"cacheMb": 300})).unwrap();
assert!(String::from_utf8(std::fs::read(lib.join("prefs.json")).unwrap()).unwrap().contains("300"));
assert_eq!(std::fs::read(lib.join(&kept[0])).unwrap(), bad);
drop(s);
let _ = std::fs::remove_dir_all(&lib);
}
/// `presets.json` and `prefs.json` can't be read (locked by another program, an I/O error).
#[derive(Clone, Default)]
struct Unreadable {
files: MemStore,
unreadable: Arc<Mutex<Vec<String>>>,
}
impl Store for Unreadable {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
if self.unreadable.lock().unwrap().iter().any(|n| n == name) {
return Err(std::io::Error::other("locked by another process"));
}
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"unreadable".into()
}
}
#[test]
fn unreadable_settings_are_not_overwritten() {
let files = Unreadable::default();
let presets = br#"{"user": [{"id": "user.mine", "name": "Mine", "group": "User Presets", "settings": {}}]}"#.to_vec();
let prefs = br#"{"cacheMb": 777}"#.to_vec();
files.files.set("presets.json", presets.clone());
files.files.set("prefs.json", prefs.clone());
*files.unreadable.lock().unwrap() = vec!["presets.json".into(), "prefs.json".into()];
let mut s = Session::new();
let stores = LibraryStores { dir: "unreadable".into(), catalog: Box::new(MemStore::new()), files: Box::new(files.clone()), on_disk: false };
s.open_library_in(stores, true).unwrap();
let warnings = s.take_library_warnings();
assert_eq!(warnings.len(), 2, "{warnings:?}");
assert!(warnings.iter().all(|w| w.contains("couldn't be read") && w.contains("won't overwrite")), "{warnings:?}");
// a preset change and a preference change: neither file is replaced by defaults
s.execute("preset.create", &json!({"name": "New"})).unwrap();
let e = s.execute("library.preferences", &json!({"cacheMb": 300}));
assert!(e.is_err(), "saving preferences is refused, not done over the unread file");
assert_eq!(files.files.get("presets.json").unwrap(), presets);
assert_eq!(files.files.get("prefs.json").unwrap(), prefs);
}
+178
View File
@@ -329,6 +329,55 @@ fn convert_raw_to_dng() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #106: Convert to DNG wrote straight to the final name (no temp file, sync or check) and
/// relinked the photo even when the write was cut short. A failed write now leaves no DNG and
/// the photo on its raw; a good one is verified and never replaces an existing file.
#[test]
fn convert_to_dng_is_verified_and_atomic() {
let dir = temp_dir("todng-safe");
// a synthetic raw (stored as DNG, catalogued as a NEF so Convert to DNG takes it)
let raw = dir.join("shot.dng");
let bytes = synthetic_dng_with(None, Default::default());
std::fs::write(&raw, &bytes).unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [raw.to_string_lossy()]})).unwrap();
let id = s.catalog.photos().next().unwrap().id;
let path = raw.to_string_lossy().to_string();
s.catalog
.apply(lightcraft_catalog::Op::Relink {
id,
file_name: "shot.dng".into(),
source: lightcraft_catalog::Source::File { path: path.clone() },
format: Some("NEF".into()),
})
.unwrap();
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
let names = || {
let mut v: Vec<String> = std::fs::read_dir(&dir).unwrap().flatten().map(|e| e.file_name().to_string_lossy().to_string()).collect();
v.sort();
v
};
{
// the drive fills up part-way
let _fault = lightcraft_catalog::safe_file::fail_writes_after(200);
let r = s.execute("photo.convertToDng", &json!({})).unwrap();
assert_eq!(r["converted"].as_array().unwrap().len(), 0, "{r}");
assert!(r["skipped"][0][1].as_str().unwrap().contains("the raw is kept"), "{r}");
}
assert_eq!(names(), ["shot.dng"], "no partial DNG");
assert_eq!(s.catalog.photo(id).unwrap().source, lightcraft_catalog::Source::File { path: path.clone() }, "not relinked");
// a good conversion: a new name (the existing file is not replaced), decodable, relinked
let r = s.execute("photo.convertToDng", &json!({})).unwrap();
let out = r["converted"][0]["path"].as_str().unwrap().to_string();
assert!(out.ends_with("shot-2.dng"), "{r}");
assert_eq!(std::fs::read(&raw).unwrap(), bytes, "the raw is untouched");
let back = lightcraft_raw::decode(&std::fs::read(&out).unwrap()).unwrap();
assert_eq!(back.data, lightcraft_raw::decode(&bytes).unwrap().data);
assert_eq!(s.catalog.photo(id).unwrap().source, lightcraft_catalog::Source::File { path: out });
assert_eq!(names(), ["shot-2.dng", "shot.dng"], "no temp file left");
let _ = std::fs::remove_dir_all(&dir);
}
/// Edit in External Editor (engine half): a 16-bit TIFF `-Edit` copy with the edits, next to
/// the original, added and stacked on top of it; a second one doesn't overwrite the first.
#[test]
@@ -576,3 +625,132 @@ fn copyright_status_usage_terms_and_url_round_trip() {
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// A sidecar another raw developer wrote: develop settings in attributes and elements, a
/// structured edit history, an unknown namespace.
const OTHER_APP_SIDECAR: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Other Toolkit">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:xmp="http://ns.adobe.com/xap/1.0/"
xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#"
xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
xmlns:other="http://example.com/other/1.0/"
xmp:Rating="2"
crs:Version="99.0"
crs:Exposure2012="+0.65"
crs:HasSettings="True"
other:Secret="keep me">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action="derived" stEvt:parameters="converted from image/x-raw to image/png"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>128, 140</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>"#;
/// Issue #92: saving merges into another app's sidecar instead of replacing it.
#[test]
fn saving_merges_into_another_apps_sidecar() {
let src = temp_dir("merge");
let lib = temp_dir("merge-lib");
write_png(&src.join("shot.png"), 1);
std::fs::write(src.join("shot.xmp"), OTHER_APP_SIDECAR).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("shot.png").to_string_lossy()]})).unwrap();
let id = only(&s);
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
s.execute("photo.rate", &json!({"rating": 5})).unwrap();
s.execute("photo.setMeta", &json!({"title": "Harbour", "keywords": ["boats"]})).unwrap();
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 30})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
assert_eq!(r["merged"][0], src.join("shot.xmp").to_string_lossy().as_ref(), "{r}");
let out = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
// the other app's data, byte for byte
for part in [
"crs:Exposure2012=\"+0.65\"",
"other:Secret=\"keep me\"",
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"derived\" stEvt:parameters=\"converted from image/x-raw to image/png\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>\n <rdf:li>128, 140</rdf:li>",
] {
assert!(out.contains(part), "lost {part:?}:\n{out}");
}
let d = lightcraft_meta::parse_xmp(&out).unwrap();
assert_eq!(d.metadata.rating, Some(5), "ours replaces theirs");
assert_eq!(d.metadata.title.as_deref(), Some("Harbour"));
assert_eq!(d.properties.get("crs:Version"), Some(&vec!["99.0".to_string()]));
let saved = s.catalog.photo(id).unwrap().develop.clone();
// reading it back restores our settings (lc:settings wins over crs:)
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().develop, saved);
// saving again keeps one LightCraft description and the other app's data
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let again = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert!(again.contains("other:Secret=\"keep me\"") && again.contains("crs:Exposure2012=\"+0.65\""));
assert_eq!(lightcraft_meta::parse_xmp(&again).unwrap().metadata.rating, Some(3));
// a sidecar that isn't XMP is kept as a backup before it is replaced
std::fs::write(src.join("shot.xmp"), b"<not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let backup = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert!(backup.contains("shot.xmp.bak-"), "{r}");
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join("shot.xmp")).unwrap()).unwrap().metadata.rating, Some(3));
// … and another one never overwrites the first backup
std::fs::write(src.join("shot.xmp"), b"<still not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let second = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert_ne!(second, backup);
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(std::fs::read(&second).unwrap(), b"<still not xmp");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #92: two catalogued files with one stem (`IMG_1.jpg` + `IMG_1.png`) don't share a
/// Stem-named sidecar: the first by name keeps `IMG_1.xmp`, the other writes and reads
/// `IMG_1.png.xmp`.
#[test]
fn files_sharing_a_stem_get_their_own_sidecars() {
let src = temp_dir("stem");
let lib = temp_dir("stem-lib");
write_png(&src.join("IMG_1.png"), 1);
let img = lightcraft_raster::Rgba8::from_fn(40, 24, |x, y| [(x * 6) as u8, (y * 9) as u8, 7, 255]);
let jpg =
crate::export::encode_image(&img, &crate::export::ExportOptions { format: crate::export::ExportFormat::Jpeg, ..Default::default() }).unwrap();
std::fs::write(src.join("IMG_1.jpg"), jpg).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = |s: &Session, name: &str| s.catalog.photos().find(|p| p.file_name == name).unwrap().id;
let (j, p) = (id(&s, "IMG_1.jpg"), id(&s, "IMG_1.png"));
assert_eq!(s.sidecar_naming(j), crate::sidecar::SidecarNaming::Stem);
assert_eq!(s.sidecar_naming(p), crate::sidecar::SidecarNaming::Full);
s.execute("photo.rate", &json!({"ids": [j.0], "rating": 2})).unwrap();
s.execute("photo.rate", &json!({"ids": [p.0], "rating": 5})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!(r["written"].as_array().unwrap().len(), 2, "{r}");
let rating = |f: &str| lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join(f)).unwrap()).unwrap().metadata.rating;
assert_eq!(rating("IMG_1.xmp"), Some(2));
assert_eq!(rating("IMG_1.png.xmp"), Some(5));
// each reads its own back
s.execute("photo.rate", &json!({"ids": [j.0, p.0], "rating": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!((s.catalog.photo(j).unwrap().rating, s.catalog.photo(p).unwrap().rating), (2, 5));
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
+234
View File
@@ -0,0 +1,234 @@
//! Which wgpu backends LightCraft lets wgpu use — for its compute device and for the desktop
//! window (eframe) — and the crash sentinel around device creation (issue #136).
//!
//! wgpu loads the system driver of *every* backend in the instance's set while it enumerates
//! adapters, even when it then picks another one: with Vulkan in the set, a broken Vulkan driver
//! can crash the process (an access violation inside the driver DLL, which nothing can catch)
//! although DX12 would have been used. So Windows defaults to DX12 alone. macOS uses Metal, Linux
//! Vulkan (plus GL for the window when the build has it).
//!
//! Overrides, read once per process:
//! - `LIGHTCRAFT_GPU_BACKEND` = `dx12` | `vulkan` | `metal` | `gl` (comma lists allowed), `auto`
//! (the platform default), or `off` (no GPU compute; the window still needs one backend and
//! keeps the platform default);
//! - else `WGPU_BACKEND` (wgpu's own variable, same names) for both the window and compute.
//!
//! Backends this build doesn't contain (e.g. `gl`: the GL backend isn't compiled in) are ignored
//! with a warning and the platform default is used.
use std::path::PathBuf;
use std::sync::Mutex;
pub use wgpu::Backends;
/// What the environment asks for.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum BackendChoice {
/// The platform default.
Auto,
/// No GPU compute.
Off,
/// These backends.
Use(Backends),
}
/// Parse a backend list (`"dx12"`, `"vulkan,gl"`, `"auto"`, `"off"`…). `None`: nothing usable in
/// it (empty or unknown names).
pub fn parse_backends(s: &str) -> Option<BackendChoice> {
let s = s.trim().to_ascii_lowercase();
match s.as_str() {
"" => None,
"auto" | "default" => Some(BackendChoice::Auto),
"off" | "none" | "cpu" | "0" | "false" | "no" => Some(BackendChoice::Off),
_ => {
let b = Backends::from_comma_list(&s);
(!b.is_empty()).then_some(BackendChoice::Use(b))
}
}
}
/// The choice from `LIGHTCRAFT_GPU_BACKEND` (`lightcraft`) or else `WGPU_BACKEND` (`wgpu`): the
/// first one that parses wins; an unknown value is skipped (with a warning).
pub fn choose(lightcraft: Option<&str>, wgpu: Option<&str>) -> BackendChoice {
for (name, v) in [("LIGHTCRAFT_GPU_BACKEND", lightcraft), ("WGPU_BACKEND", wgpu)] {
let Some(v) = v else { continue };
match parse_backends(v) {
// WGPU_BACKEND is wgpu's variable: it selects backends, it never turns the GPU off
Some(BackendChoice::Off) if name == "WGPU_BACKEND" => log::warn!("gpu: WGPU_BACKEND={v} ignored (use LIGHTCRAFT_GPU_BACKEND=off)"),
Some(c) => return c,
None => log::warn!("gpu: {name}={v} names no known backend; ignored"),
}
}
BackendChoice::Auto
}
/// The backends compute devices use when nothing is asked for on `os` (`std::env::consts::OS`).
pub fn default_compute_backends(os: &str) -> Backends {
match os {
// never touch the Vulkan driver unless asked: see the module docs (issue #136)
"windows" => Backends::DX12,
"macos" | "ios" => Backends::METAL,
_ => Backends::PRIMARY,
}
}
/// The backends the window uses when nothing is asked for on `os`.
pub fn default_window_backends(os: &str) -> Backends {
match os {
"windows" => Backends::DX12,
"macos" | "ios" => Backends::METAL,
// eframe's default (GL only when the build has it)
_ => Backends::PRIMARY | Backends::GL,
}
}
/// Resolve `choice` against the backends this build contains (`compiled`): `None` = GPU off.
/// Asked-for backends that aren't compiled in fall back to `default`.
pub fn resolve(choice: BackendChoice, default: Backends, compiled: Backends) -> Option<Backends> {
match choice {
BackendChoice::Off => None,
BackendChoice::Auto => Some(default),
BackendChoice::Use(b) if b.intersects(compiled) => Some(b & compiled),
BackendChoice::Use(b) => {
log::warn!("gpu: backend {b:?} is not in this build ({compiled:?}); using {default:?}");
Some(default)
}
}
}
fn env_choice() -> BackendChoice {
static C: std::sync::OnceLock<BackendChoice> = std::sync::OnceLock::new();
*C.get_or_init(|| {
let lc = std::env::var("LIGHTCRAFT_GPU_BACKEND").ok();
let wg = std::env::var("WGPU_BACKEND").ok();
choose(lc.as_deref(), wg.as_deref())
})
}
/// The backends of the compute device (`None`: `LIGHTCRAFT_GPU_BACKEND=off`).
pub fn compute_backends() -> Option<Backends> {
resolve(env_choice(), default_compute_backends(std::env::consts::OS), wgpu::Instance::enabled_backend_features())
}
/// The backends for the desktop window's renderer (eframe/egui-wgpu). The window always needs
/// one: `off` keeps the platform default.
pub fn window_backends() -> Backends {
let default = default_window_backends(std::env::consts::OS);
let choice = match env_choice() {
BackendChoice::Off => BackendChoice::Auto,
c => c,
};
resolve(choice, default, wgpu::Instance::enabled_backend_features()).unwrap_or(default)
}
/// `LIGHTCRAFT_GPU_BACKEND=off`.
pub(crate) fn env_off() -> bool {
env_choice() == BackendChoice::Off
}
static MARKER: Mutex<Option<PathBuf>> = Mutex::new(None);
/// Write this file before the compute device is created and remove it once creation returned
/// (successfully or not). A file still there at the next launch means the process died inside
/// the driver: see [`take_init_marker`]. `None` (the default) writes nothing.
pub fn set_init_marker(path: Option<PathBuf>) {
*MARKER.lock().unwrap_or_else(|e| e.into_inner()) = path;
}
/// Run device creation `f` between writing and removing the init marker (if one is set).
pub(crate) fn with_init_marker<T>(backends: Backends, f: impl FnOnce() -> T) -> T {
let marker = MARKER.lock().unwrap_or_else(|e| e.into_inner()).clone();
if let Some(m) = &marker {
if let Some(d) = m.parent() {
let _ = std::fs::create_dir_all(d);
}
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let _ = std::fs::write(m, format!("GPU device creation started (backends {backends:?}, unix time {secs})\n"));
}
let r = f();
if let Some(m) = &marker {
let _ = std::fs::remove_file(m);
}
r
}
/// If the init marker `path` is present — the last process died while creating the GPU device —
/// remove it and return what it recorded.
pub fn take_init_marker(path: &std::path::Path) -> Option<String> {
let text = std::fs::read_to_string(path).ok()?;
let _ = std::fs::remove_file(path);
Some(text.trim().to_string())
}
#[cfg(test)]
#[allow(clippy::unwrap_used)]
mod tests {
use super::*;
#[test]
fn parses_backend_names() {
assert_eq!(parse_backends("dx12"), Some(BackendChoice::Use(Backends::DX12)));
assert_eq!(parse_backends(" D3D12 "), Some(BackendChoice::Use(Backends::DX12)));
assert_eq!(parse_backends("vulkan"), Some(BackendChoice::Use(Backends::VULKAN)));
assert_eq!(parse_backends("vk,gl"), Some(BackendChoice::Use(Backends::VULKAN | Backends::GL)));
assert_eq!(parse_backends("metal"), Some(BackendChoice::Use(Backends::METAL)));
assert_eq!(parse_backends("auto"), Some(BackendChoice::Auto));
for off in ["off", "OFF", "none", "cpu", "0"] {
assert_eq!(parse_backends(off), Some(BackendChoice::Off), "{off}");
}
assert_eq!(parse_backends(""), None);
assert_eq!(parse_backends("directx9"), None);
}
#[test]
fn lightcraft_variable_wins_over_wgpu_backend() {
assert_eq!(choose(None, None), BackendChoice::Auto);
assert_eq!(choose(None, Some("dx12")), BackendChoice::Use(Backends::DX12));
assert_eq!(choose(Some("vulkan"), Some("dx12")), BackendChoice::Use(Backends::VULKAN));
assert_eq!(choose(Some("off"), Some("dx12")), BackendChoice::Off);
// unknown LightCraft value: WGPU_BACKEND still applies
assert_eq!(choose(Some("bogus"), Some("gl")), BackendChoice::Use(Backends::GL));
// WGPU_BACKEND never turns the GPU off
assert_eq!(choose(None, Some("off")), BackendChoice::Auto);
}
/// Issue #136: Windows must not load the Vulkan driver unless asked to.
#[test]
fn windows_defaults_to_dx12_only() {
assert_eq!(default_compute_backends("windows"), Backends::DX12);
assert_eq!(default_window_backends("windows"), Backends::DX12);
assert!(!default_window_backends("windows").contains(Backends::VULKAN));
assert_eq!(default_compute_backends("macos"), Backends::METAL);
assert!(default_compute_backends("linux").contains(Backends::VULKAN));
assert!(default_window_backends("linux").contains(Backends::VULKAN | Backends::GL));
}
#[test]
fn resolves_against_the_build() {
let compiled = Backends::DX12 | Backends::VULKAN;
let def = Backends::DX12;
assert_eq!(resolve(BackendChoice::Auto, def, compiled), Some(def));
assert_eq!(resolve(BackendChoice::Off, def, compiled), None);
assert_eq!(resolve(BackendChoice::Use(Backends::VULKAN), def, compiled), Some(Backends::VULKAN));
assert_eq!(resolve(BackendChoice::Use(Backends::VULKAN | Backends::GL), def, compiled), Some(Backends::VULKAN));
// GL isn't compiled in: the default, not an empty set (no adapter at all)
assert_eq!(resolve(BackendChoice::Use(Backends::GL), def, compiled), Some(def));
}
#[test]
fn init_marker_is_written_during_creation_and_removed_after() {
let dir = std::env::temp_dir().join(format!("lc-gpu-marker-{}", std::process::id()));
let m = dir.join("gpu-init.marker");
let _ = std::fs::remove_dir_all(&dir);
set_init_marker(Some(m.clone()));
let seen = with_init_marker(Backends::DX12, || m.exists());
set_init_marker(None);
assert!(seen, "the marker exists while the device is created");
assert!(!m.exists(), "and is removed afterwards");
// a marker left behind by a crashed process is reported once
std::fs::write(&m, "GPU device creation started (backends DX12)").unwrap();
assert!(take_init_marker(&m).unwrap().contains("DX12"));
assert_eq!(take_init_marker(&m), None);
let _ = std::fs::remove_dir_all(&dir);
}
}
+4 -3
View File
@@ -276,14 +276,15 @@ fn module_source(m: &Module, consts: &str) -> String {
impl Gpu {
/// Create a device on the best available adapter (no software fallback), or why there is none.
pub fn new() -> Result<Gpu, String> {
pub fn new(backends: wgpu::Backends) -> Result<Gpu, String> {
let mut desc = wgpu::InstanceDescriptor::new_without_display_handle();
desc.backends = wgpu::Backends::PRIMARY;
// only these drivers are loaded (`crate::backend`: never Vulkan on Windows unless asked)
desc.backends = backends;
let instance = wgpu::Instance::new(desc);
let adapter = pollster::block_on(
instance.request_adapter(&wgpu::RequestAdapterOptions { power_preference: wgpu::PowerPreference::HighPerformance, ..Default::default() }),
)
.map_err(|e| format!("no GPU adapter ({e})"))?;
.map_err(|e| format!("no GPU adapter among {backends:?} ({e})"))?;
let info = adapter.get_info();
if info.device_type == wgpu::DeviceType::Cpu {
log::info!("gpu: only a software adapter ({}), using the CPU pipeline", info.name);
+33 -9
View File
@@ -6,8 +6,11 @@
//! same settings on both and bound the difference in 8-bit sRGB. Stages without a kernel run on the
//! CPU inside the same render (per-stage hybrid); see `docs/gpu-pipeline.md`.
//!
//! Use [`render`]: it returns `None` when the GPU is unavailable, disabled (`LIGHTCRAFT_GPU=0` or
//! [`set_enabled`]), the render does not fit the device, or the device reported an error, ran out
//! Which backends wgpu may load (DX12 only on Windows; `LIGHTCRAFT_GPU_BACKEND`, `WGPU_BACKEND`) and
//! the crash sentinel around device creation: [`backend`] (issue #136).
//!
//! Use [`render`]: it returns `None` when the GPU is unavailable, disabled (`LIGHTCRAFT_GPU=0`,
//! `LIGHTCRAFT_GPU_BACKEND=off` or [`set_enabled`]), the render does not fit the device, or the device reported an error, ran out
//! of memory or returned an incomplete image — callers then render on the CPU. [`unavailable_reason`]
//! and [`last_fallback`] say why (`ui.inspect` → `perf.gpuReason` / `perf.gpuFallback`).
//! The browser build has no GPU path yet (WebGPU device creation is asynchronous): everything here
@@ -22,6 +25,8 @@ use lightcraft_develop::DevelopSettings;
use lightcraft_pipeline::{RenderRequest, Rendered, SourceInfo, StageCache};
use lightcraft_raster::Rgb32f;
#[cfg(not(target_arch = "wasm32"))]
pub mod backend;
#[cfg(not(target_arch = "wasm32"))]
mod ctx;
#[cfg(not(target_arch = "wasm32"))]
@@ -61,6 +66,10 @@ fn record_fallback(reason: String) {
/// Never blocks.
pub fn unavailable_reason() -> Option<String> {
if env_disabled() {
#[cfg(not(target_arch = "wasm32"))]
if backend::env_off() {
return Some("disabled by LIGHTCRAFT_GPU_BACKEND=off".into());
}
return Some("disabled by LIGHTCRAFT_GPU=0".into());
}
if !ENABLED.load(Ordering::Relaxed) {
@@ -136,9 +145,15 @@ pub fn enabled() -> bool {
ENABLED.load(Ordering::Relaxed) && !BROKEN.load(Ordering::Relaxed) && !env_disabled()
}
/// `LIGHTCRAFT_GPU=0` (or `LIGHTCRAFT_GPU_BACKEND=off`): no GPU for the whole process.
fn env_disabled() -> bool {
static OFF: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*OFF.get_or_init(|| std::env::var("LIGHTCRAFT_GPU").is_ok_and(|v| matches!(v.trim(), "0" | "off" | "false" | "no")))
*OFF.get_or_init(|| {
let off = std::env::var("LIGHTCRAFT_GPU").is_ok_and(|v| matches!(v.trim(), "0" | "off" | "false" | "no"));
#[cfg(not(target_arch = "wasm32"))]
let off = off || backend::env_off();
off
})
}
#[cfg(not(target_arch = "wasm32"))]
@@ -146,10 +161,18 @@ static GPU: std::sync::OnceLock<Result<ctx::Gpu, String>> = std::sync::OnceLock:
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn device() -> Option<&'static ctx::Gpu> {
if env_disabled() {
return None;
// turned off (preference, environment): don't even load the driver (issue #136)
if env_disabled() || !ENABLED.load(Ordering::Relaxed) {
return existing_device();
}
GPU.get_or_init(|| std::panic::catch_unwind(ctx::Gpu::new).unwrap_or_else(|_| Err("device creation panicked".into()))).as_ref().ok()
GPU.get_or_init(|| {
let Some(backends) = backend::compute_backends() else { return Err("disabled by LIGHTCRAFT_GPU_BACKEND=off".into()) };
backend::with_init_marker(backends, || {
std::panic::catch_unwind(|| ctx::Gpu::new(backends)).unwrap_or_else(|_| Err("device creation panicked".into()))
})
})
.as_ref()
.ok()
}
/// The device if it has been created (never creates it).
@@ -158,11 +181,12 @@ pub(crate) fn existing_device() -> Option<&'static ctx::Gpu> {
GPU.get().and_then(|g| g.as_ref().ok())
}
/// Create the device and compile the kernels on a background thread now (app start), so the
/// first render doesn't wait ~0.3–0.4 s for it and the UI thread never does.
/// Create the device and compile the kernels on a background thread now (once the window is up),
/// so the first render doesn't wait ~0.3–0.4 s for it and the UI thread never does. Does nothing
/// while GPU rendering is turned off.
pub fn warm_up() {
#[cfg(not(target_arch = "wasm32"))]
if !env_disabled() && GPU.get().is_none() {
if !env_disabled() && ENABLED.load(Ordering::Relaxed) && GPU.get().is_none() {
let _ = std::thread::Builder::new().name("lc-gpu-init".into()).spawn(|| {
let _ = device();
});
+21
View File
@@ -22,6 +22,27 @@ fn gpu() -> bool {
ok
}
#[test]
fn camera_tone_and_relative_wb() {
if !gpu() {
return;
}
let src = scene(2, 320, 240);
let curve = lightcraft_pipeline::tone::CameraTone::new(std::array::from_fn(|i| {
let x = 0.004 * 1.18f32.powi(i as i32);
[x, 1.0 - (-2.0 * x).exp()]
}))
.unwrap();
let info = SourceInfo { raw: true, relative_wb: true, camera_tone: Some(curve), ..Default::default() };
let mut s = DevelopSettings::default();
check("camera tone neutral", &src, &info, &s, &RenderRequest::fit(320, 240));
s.light.exposure = 1.0;
s.light.contrast = 40.0;
s.wb.mode = WbMode::Custom;
s.wb.temp = 8000.0;
check("camera tone edited", &src, &info, &s, &RenderRequest::fit(320, 240));
}
fn scene(i: usize, w: usize, h: usize) -> Arc<Rgb32f> {
Arc::new(lightcraft_scenes::demo_library()[i].render(w, h))
}
+1
View File
@@ -45,6 +45,7 @@ pub fn color_data(color: &FrameColor) -> ColorData {
as_shot_neutral: Some([1.0, 1.0, 1.0]),
as_shot_white_xy: None,
baseline_exposure: 0.0,
profile: Default::default(),
}
}
}
+2
View File
@@ -17,6 +17,7 @@ mod gpx;
mod iptc;
pub mod tags;
mod xmp;
mod xmp_merge;
pub use container::{Embedded, embedded, jpeg_segments, png_chunks, webp_chunks};
pub use datetime::DateTime;
@@ -26,6 +27,7 @@ pub use iptc::parse_iptc;
pub use lightcraft_geom::Orientation;
pub use tags::{TagRow, file_tag_rows, tag_rows};
pub use xmp::{CRS_NS, LC_NS, XmpData, XmpError, XmpValue, parse_xmp, write_xmp, write_xmp_lc};
pub use xmp_merge::{MergeRules, merge_xmp};
use serde::{Deserialize, Serialize};
+5
View File
@@ -145,6 +145,11 @@ impl Node {
const MAX_DEPTH: usize = 64;
const MAX_NODES: usize = 200_000;
/// `prefix:local` with the prefix canonicalised by namespace URI (see [`NAMESPACES`]).
pub(crate) fn canonical_name(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
canonical(qname, scopes)
}
fn canonical(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
let (prefix, local) = qname.split_once(':').unwrap_or(("", qname));
if prefix == "xml" || prefix == "xmlns" {
+403
View File
@@ -0,0 +1,403 @@
//! Merging a freshly written XMP packet into an existing one (a sidecar another application
//! wrote, e.g. with its develop settings and edit history), keeping everything the writer does not
//! own byte for byte.
//!
//! The existing packet is scanned once (namespace-resolved, with byte positions). Only top-level
//! properties of the `rdf:Description` elements directly under `rdf:RDF` are considered, in
//! attribute form (`xmp:Rating="3"`) and element form (`<dc:subject>…</dc:subject>`):
//! - properties the writer owns ([`MergeRules::owned`]) are removed, and written from the fresh
//! packet when it has them (so a cleared field is cleared in the file too);
//! - properties owned only when stated ([`MergeRules::owned_if_present`]) are replaced when the
//! fresh packet has them and kept otherwise;
//! - any other property of the fresh packet (e.g. `xmp:CreatorTool`) is not written: the file
//! keeps describing itself as its creator left it;
//! - everything else — other namespaces (e.g. `crs:`), `xmpMM:History`, unknown properties,
//! comments, the packet wrapper and padding — is copied unchanged.
//!
//! The written properties go into one new `rdf:Description` (declaring its own namespaces, so
//! the existing prefix bindings never matter) just before `</rdf:RDF>`. A description left empty
//! by the removals is dropped, so repeated saves don't accumulate. The result is checked by
//! parsing it again: every written property must read back as in the fresh packet.
use std::collections::BTreeSet;
use quick_xml::events::Event;
use crate::xmp::{XmpError, canonical_name, parse_xmp};
/// Which properties [`merge_xmp`] takes from the fresh packet. Names are canonical
/// `prefix:name` (prefixes as [`crate::parse_xmp`] reports them); `prefix:*` is a whole namespace.
#[derive(Clone, Copy, Debug, Default)]
pub struct MergeRules<'a> {
/// Always replaced: removed from the existing packet, written from the fresh one if stated.
pub owned: &'a [&'a str],
/// Replaced only when the fresh packet states them; otherwise the existing value stays.
pub owned_if_present: &'a [&'a str],
}
fn listed(list: &[&str], name: &str) -> bool {
list.iter().any(|p| match p.strip_suffix('*') {
Some(ns) => name.starts_with(ns),
None => *p == name,
})
}
#[derive(Debug)]
struct Attr {
raw: String,
name: String,
/// The value as written (still escaped).
value: String,
}
#[derive(Debug)]
struct Prop {
name: String,
start: usize,
end: usize,
}
#[derive(Debug, Default)]
struct Desc {
tag_start: usize,
tag_end: usize,
/// The element name as written (`rdf:Description`, or another prefix for the RDF namespace).
raw_name: String,
empty: bool,
attrs: Vec<Attr>,
props: Vec<Prop>,
/// The end of `</rdf:Description>` (`tag_end` when self-closing).
end: usize,
}
#[derive(Debug, Default)]
struct Scan {
descs: Vec<Desc>,
/// Where the first `</rdf:RDF>` starts.
rdf_close: Option<usize>,
}
enum Kind {
Other,
Rdf,
Desc(usize),
Prop { desc: usize, start: usize, name: String },
}
fn is_meta_attr(name: &str) -> bool {
name.starts_with("xmlns")
|| matches!(name, "rdf:about" | "rdf:ID" | "rdf:nodeID" | "xml:lang" | "rdf:parseType" | "rdf:resource" | "rdf:datatype")
}
fn pos(r: &quick_xml::Reader<&[u8]>) -> usize {
usize::try_from(r.buffer_position()).unwrap_or(usize::MAX)
}
/// Locate the descriptions, their top-level properties and `</rdf:RDF>` in a packet.
fn scan(s: &str) -> Result<Scan, XmpError> {
let mut reader = quick_xml::Reader::from_reader(s.as_bytes());
reader.config_mut().check_end_names = true;
let mut out = Scan::default();
let mut stack: Vec<Kind> = Vec::new();
let mut scopes: Vec<Vec<(String, String)>> = vec![vec![("xml".into(), "http://www.w3.org/XML/1998/namespace".into())]];
let mut count = 0usize;
loop {
let before = pos(&reader);
let ev = reader.read_event().map_err(|e| XmpError::Xml(e.to_string()))?;
let after = pos(&reader);
match ev {
Event::Start(ref e) | Event::Empty(ref e) => {
count += 1;
if count > 200_000 || stack.len() > 64 {
return Err(XmpError::TooComplex);
}
let empty = matches!(ev, Event::Empty(_));
let mut scope = Vec::new();
let mut raw_attrs = Vec::new();
for a in e.attributes() {
let a = a.map_err(|e| XmpError::Xml(e.to_string()))?;
let k = String::from_utf8_lossy(a.key.as_ref()).into_owned();
let v = String::from_utf8_lossy(&a.value).into_owned();
if k == "xmlns" {
scope.push((String::new(), v.clone()));
} else if let Some(p) = k.strip_prefix("xmlns:") {
scope.push((p.to_string(), v.clone()));
}
raw_attrs.push((k, v));
}
scopes.push(scope);
let raw_name = String::from_utf8_lossy(e.name().as_ref()).into_owned();
let name = canonical_name(&raw_name, &scopes);
let kind = match stack.last() {
Some(Kind::Rdf) if name == "rdf:Description" => {
let attrs = raw_attrs.into_iter().map(|(raw, value)| Attr { name: canonical_name(&raw, &scopes), raw, value }).collect();
out.descs.push(Desc { tag_start: before, tag_end: after, raw_name, empty, attrs, props: Vec::new(), end: after });
Kind::Desc(out.descs.len() - 1)
}
Some(Kind::Desc(i)) => Kind::Prop { desc: *i, start: before, name },
_ if name == "rdf:RDF" && out.rdf_close.is_none() => Kind::Rdf,
_ => Kind::Other,
};
if empty {
scopes.pop();
if let Kind::Prop { desc, start, name } = kind
&& let Some(d) = out.descs.get_mut(desc)
{
d.props.push(Prop { name, start, end: after });
}
} else {
stack.push(kind);
}
}
Event::End(_) => {
scopes.pop();
match stack.pop() {
Some(Kind::Prop { desc, start, name }) => {
if let Some(d) = out.descs.get_mut(desc) {
d.props.push(Prop { name, start, end: after });
}
}
Some(Kind::Desc(i)) => {
if let Some(d) = out.descs.get_mut(i) {
d.end = after;
}
}
Some(Kind::Rdf) => out.rdf_close = out.rdf_close.or(Some(before)),
Some(Kind::Other) => {}
None => return Err(XmpError::Xml("unbalanced end tag".into())),
}
}
Event::Eof => break,
_ => {}
}
}
if !stack.is_empty() {
return Err(XmpError::Xml("unclosed element".into()));
}
Ok(out)
}
/// `start` moved back over the indentation and line break before it (so a removed property
/// takes its line with it).
fn line_start(s: &str, start: usize) -> usize {
let b = s.as_bytes();
let mut i = start;
while i > 0 && matches!(b.get(i - 1), Some(b' ' | b'\t')) {
i -= 1;
}
if i > 0 && b.get(i - 1) == Some(&b'\n') {
i -= 1;
if i > 0 && b.get(i - 1) == Some(&b'\r') {
i -= 1;
}
}
i
}
/// Merge `fresh` (a packet as [`crate::write_xmp_lc`] writes it) into `existing` following
/// `rules` (see the module docs). Fails — and the caller must not overwrite `existing` blindly —
/// when `existing` isn't a well-formed XMP packet.
pub fn merge_xmp(existing: &str, fresh: &str, rules: MergeRules) -> Result<String, XmpError> {
let (bom, old) = match existing.strip_prefix('\u{feff}') {
Some(rest) => ("\u{feff}", rest),
None => ("", existing),
};
parse_xmp(old)?;
let ex = scan(old)?;
let rdf_close = ex.rdf_close.ok_or_else(|| XmpError::Xml("no rdf:RDF element".into()))?;
let new = scan(fresh)?;
let fresh_desc = new.descs.first().ok_or_else(|| XmpError::Xml("the new packet has no rdf:Description".into()))?;
let fresh_names: BTreeSet<&str> = fresh_desc.props.iter().map(|p| p.name.as_str()).collect();
let remove = |name: &str| listed(rules.owned, name) || (listed(rules.owned_if_present, name) && fresh_names.contains(name));
let write = |name: &str| listed(rules.owned, name) || listed(rules.owned_if_present, name);
// edits on `old`: (start, end, replacement), applied back to front
let mut edits: Vec<(usize, usize, String)> = Vec::new();
for d in &ex.descs {
let gone_attrs: Vec<&Attr> = d.attrs.iter().filter(|a| !is_meta_attr(&a.name) && remove(&a.name)).collect();
let gone_props: Vec<&Prop> = d.props.iter().filter(|p| remove(&p.name)).collect();
if gone_attrs.is_empty() && gone_props.is_empty() {
continue;
}
let attrs_left = d.attrs.iter().any(|a| !is_meta_attr(&a.name) && !remove(&a.name));
if !attrs_left && gone_props.len() == d.props.len() {
// nothing of its own left: drop the whole description
edits.push((line_start(old, d.tag_start), d.end, String::new()));
continue;
}
if !gone_attrs.is_empty() {
let mut tag = format!("<{}", d.raw_name);
for a in d.attrs.iter().filter(|a| is_meta_attr(&a.name) || !remove(&a.name)) {
tag.push_str(&format!("\n {}=\"{}\"", a.raw, a.value.replace('"', "&quot;")));
}
tag.push_str(if d.empty { "/>" } else { ">" });
edits.push((d.tag_start, d.tag_end, tag));
}
for p in gone_props {
edits.push((line_start(old, p.start), p.end, String::new()));
}
}
// the new description: its own namespace declarations, the properties to write
let mut block = String::from("\n <rdf:Description rdf:about=\"\"\n xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"");
for a in fresh_desc.attrs.iter().filter(|a| a.raw.starts_with("xmlns:") && a.raw != "xmlns:rdf") {
block.push_str(&format!("\n {}=\"{}\"", a.raw, a.value));
}
block.push('>');
let mut written = 0;
for p in fresh_desc.props.iter().filter(|p| write(&p.name)) {
let text = fresh.get(p.start..p.end).ok_or_else(|| XmpError::Xml("bad property span".into()))?;
block.push_str("\n ");
block.push_str(text);
written += 1;
}
block.push_str("\n </rdf:Description>");
if written > 0 {
let at = old.get(..rdf_close).map_or(rdf_close, |h| h.trim_end().len());
edits.push((at, at, block));
}
edits.sort_by_key(|e| (e.0, e.1));
// overlapping edits can't happen (properties sit inside their description, which is either
// dropped whole or edited piecewise), but never splice blindly
let mut out = String::with_capacity(old.len() + 4096);
out.push_str(bom);
let mut at = 0;
for (s, e, r) in &edits {
if *s < at {
return Err(XmpError::Xml("overlapping edits".into()));
}
out.push_str(old.get(at..*s).ok_or_else(|| XmpError::Xml("bad span".into()))?);
out.push_str(r);
at = *e;
}
out.push_str(old.get(at..).ok_or_else(|| XmpError::Xml("bad span".into()))?);
// read back: everything written must come out as written
let merged = parse_xmp(&out)?;
let want = parse_xmp(fresh)?;
for (k, v) in &want.properties {
let top = k.split('/').next().unwrap_or(k);
if write(top) && merged.properties.get(k) != Some(v) {
return Err(XmpError::Xml(format!("{k} did not survive the merge")));
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
const RULES: MergeRules =
MergeRules { owned: &["xmp:Rating", "xmp:Label", "dc:subject", "dc:title", "lc:*"], owned_if_present: &["exif:DateTimeOriginal"] };
/// A sidecar as another raw developer writes it: attribute-form simple properties, develop
/// settings, a structured history, an unknown namespace.
const FOREIGN: &str = "<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>
<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">
<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">
<rdf:Description rdf:about=\"\"
xmlns:xmp=\"http://ns.adobe.com/xap/1.0/\"
xmlns:xmpMM=\"http://ns.adobe.com/xap/1.0/mm/\"
xmlns:stEvt=\"http://ns.adobe.com/xap/1.0/sType/ResourceEvent#\"
xmlns:crs=\"http://ns.adobe.com/camera-raw-settings/1.0/\"
xmlns:dc=\"http://purl.org/dc/elements/1.1/\"
xmlns:foo=\"http://example.com/foo/\"
xmp:Rating=\"2\"
xmp:CreatorTool=\"Other App\"
crs:Exposure2012=\"+0.50\"
crs:Contrast2012=\"-12\"
foo:Thing=\"a &amp; b\">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
<dc:subject>
<rdf:Bag>
<rdf:li>old</rdf:li>
</rdf:Bag>
</dc:subject>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
";
/// With the packet's whitespace padding and trailer.
fn foreign() -> String {
format!("{FOREIGN}{}\n<?xpacket end=\"w\"?>", " ".repeat(100))
}
fn fresh(rating: i8, keywords: &[&str]) -> String {
let m = crate::Metadata {
software: Some("LightCraft".into()),
rating: Some(rating),
keywords: keywords.iter().map(|k| k.to_string()).collect(),
..Default::default()
};
crate::write_xmp_lc(&m, &[("settings", "{\"a\":1}"), ("flag", "pick")])
}
#[test]
fn keeps_foreign_properties_and_replaces_owned_ones() {
let out = merge_xmp(&foreign(), &fresh(4, &["new", "two"]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
let p = |k: &str| d.properties.get(k).cloned().unwrap_or_default();
assert_eq!(p("xmp:Rating"), vec!["4"]);
assert_eq!(p("dc:subject"), vec!["new", "two"]);
assert_eq!(p("lc:settings"), vec!["{\"a\":1}"]);
assert_eq!(p("lc:flag"), vec!["pick"]);
assert_eq!(p("xmp:CreatorTool"), vec!["Other App"], "not ours: kept");
assert_eq!(p("crs:Exposure2012"), vec!["+0.50"]);
assert_eq!(p("foo:Thing").len(), 1);
// untouched parts are byte for byte the same
for part in [
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>",
"crs:Contrast2012=\"-12\"",
"foo:Thing=\"a &amp; b\"",
"<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>\n<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">",
&format!("</x:xmpmeta>\n{}\n<?xpacket end=\"w\"?>", " ".repeat(100)),
] {
assert!(out.contains(part), "lost {part:?} in\n{out}");
}
assert!(!out.contains("xmp:Rating=\"2\""));
assert!(!out.contains("<rdf:li>old</rdf:li>"));
// saving again changes only our part, and doesn't pile up descriptions
let again = merge_xmp(&out, &fresh(1, &[]), RULES).unwrap();
let d = parse_xmp(&again).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["1".to_string()]));
assert!(!d.properties.contains_key("dc:subject"), "a cleared field is cleared");
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert_eq!(merge_xmp(&again, &fresh(1, &[]), RULES).unwrap(), again, "idempotent");
}
#[test]
fn owned_if_present_and_prefix_independence() {
// the existing packet binds the namespaces to other prefixes, and has a capture time
let ex = "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><r:RDF xmlns:r=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\
<r:Description r:about=\"\" xmlns:a=\"http://ns.adobe.com/xap/1.0/\" xmlns:e=\"http://ns.adobe.com/exif/1.0/\" \
a:Rating=\"5\" e:DateTimeOriginal=\"2020-01-01T00:00:00\"/></r:RDF></x:xmpmeta>";
let out = merge_xmp(ex, &fresh(3, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["3".to_string()]));
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2020-01-01T00:00:00".to_string()]), "fresh has none: kept");
let mut m = crate::Metadata { rating: Some(3), ..Default::default() };
m.capture_time = crate::DateTime::parse_iso("2024-05-06T07:08:09");
let out = merge_xmp(ex, &crate::write_xmp_lc(&m, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2024-05-06T07:08:09".to_string()]));
}
#[test]
fn refuses_what_it_cannot_read() {
for bad in ["", "not xml at all", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><rdf:RDF", "<a><b></a>", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>"] {
assert!(merge_xmp(bad, &fresh(1, &[]), RULES).is_err(), "{bad:?}");
}
}
}
+18 -1
View File
@@ -1,4 +1,4 @@
use lightcraft_meta::{Metadata, extract, parse_iptc, parse_xmp, read_exif, write_xmp};
use lightcraft_meta::{MergeRules, Metadata, extract, merge_xmp, parse_iptc, parse_xmp, read_exif, write_xmp};
use proptest::prelude::*;
proptest! {
@@ -21,6 +21,23 @@ proptest! {
let _ = parse_xmp(&s);
}
/// Merging into arbitrary (also damaged) packets never panics, and a merge that succeeds
/// reads back with the written values.
#[test]
fn merge_never_panics(cut in 0usize..900, junk in ".{0,20}", rating in 0i8..=5) {
let base = write_xmp(&Metadata { title: Some("t".into()), rating: Some(1), ..Default::default() }, Some("{}"));
let mut s: String = base.chars().take(cut).collect();
s.push_str(&junk);
s.extend(base.chars().skip(cut));
let fresh = write_xmp(&Metadata { rating: Some(rating), ..Default::default() }, Some("{\"x\":1}"));
let rules = MergeRules { owned: &["xmp:Rating", "dc:title", "lc:*"], owned_if_present: &[] };
if let Ok(out) = merge_xmp(&s, &fresh, rules) {
let d = parse_xmp(&out).unwrap();
prop_assert_eq!(d.metadata.rating, Some(rating));
prop_assert_eq!(d.lc_settings.as_deref(), Some("{\"x\":1}"));
}
}
#[test]
fn user_text_roundtrips(title in "[a-zA-Z0-9 &<>\"'éü€;|/\\\\]{1,40}", kw in proptest::collection::vec("[a-zA-Z0-9 &<>\"'|]{1,12}", 0..5), rating in -1i8..=5) {
let title = title.trim().to_string();
+3 -1
View File
@@ -221,7 +221,9 @@ impl FinishParams {
FinishParams {
calib: if calibration { crate::colorops::calibration_matrix(&s.calibration) } else { None },
shadow_tint: if calibration { (s.calibration.shadows_tint / 100.0) as f32 } else { 0.0 },
tone: if info.raw {
tone: if let Some(curve) = info.camera_tone.as_ref().filter(|_| info.raw) {
ToneMap::camera(curve, s.light.contrast, s.light.whites, s.light.blacks)
} else if info.raw {
ToneMap::new(s.light.contrast, s.light.whites, s.light.blacks)
} else {
ToneMap::display(s.light.contrast, s.light.whites, s.light.blacks)
+4 -1
View File
@@ -64,11 +64,14 @@ pub struct SourceInfo {
pub raw: bool,
pub as_shot_temp: f64,
pub as_shot_tint: f64,
/// No measured camera illuminant: WB adjustments are relative to the camera's rendered look.
pub relative_wb: bool,
pub camera_tone: Option<tone::CameraTone>,
}
impl Default for SourceInfo {
fn default() -> Self {
Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None }
Self { raw: false, as_shot_temp: 6500.0, as_shot_tint: 0.0, lens: None, relative_wb: false, camera_tone: None }
}
}
+1 -1
View File
@@ -61,7 +61,7 @@ pub fn effective_wb(info: &SourceInfo, s: &DevelopSettings) -> (f64, f64) {
use lightcraft_develop::WbMode;
match s.wb.mode {
WbMode::AsShot => (info.as_shot_temp, info.as_shot_tint),
m if info.raw => m.preset().unwrap_or((s.wb.temp, s.wb.tint)),
m if info.raw && !info.relative_wb => m.preset().unwrap_or((s.wb.temp, s.wb.tint)),
_ => (s.wb.temp, s.wb.tint),
}
}
+89
View File
@@ -14,12 +14,75 @@ pub const LUT_MIN_EV: f32 = -14.0;
pub const LUT_MAX_EV: f32 = 10.0;
pub const LUT_N: usize = 4096;
/// A file-local camera look, fitted independently of the scene-linear colour transform.
/// Knots are scene/display-linear luminance pairs. Keeping this in the finish stage preserves
/// RAW exposure and highlight headroom; it is never baked into the decoded sensor pixels.
#[derive(Clone, Copy, Debug, PartialEq, serde::Serialize)]
pub struct CameraTone {
knots: [[f32; 2]; 32],
}
impl<'de> serde::Deserialize<'de> for CameraTone {
fn deserialize<D: serde::Deserializer<'de>>(d: D) -> Result<Self, D::Error> {
#[derive(serde::Deserialize)]
struct Wire {
knots: [[f32; 2]; 32],
}
let w = Wire::deserialize(d)?;
Self::new(w.knots).ok_or_else(|| serde::de::Error::custom("invalid camera tone curve"))
}
}
impl CameraTone {
pub fn new(knots: [[f32; 2]; 32]) -> Option<Self> {
let mut previous = [0.0, 0.0];
for p in knots {
if !p.iter().all(|v| v.is_finite()) || p[0] <= previous[0] || p[1] < previous[1] || p[1] >= 1.0 {
return None;
}
previous = p;
}
Some(Self { knots })
}
pub fn apply(&self, y: f32) -> f32 {
if !y.is_finite() || y <= 0.0 {
return 0.0;
}
let mut previous = [0.0, 0.0];
for p in self.knots {
if y <= p[0] {
let t = (y - previous[0]) / (p[0] - previous[0]);
return previous[1] + t * (p[1] - previous[1]);
}
previous = p;
}
let a = self.knots[30];
let b = self.knots[31];
// Extend beyond observed (unclipped) highlights with a continuous, bounded shoulder.
let slope = ((b[1] - a[1]) / (b[0] - a[0])).clamp(0.1, 16.0);
1.0 - (1.0 - b[1]) * (-(y - b[0]) * slope / (1.0 - b[1]).max(0.01)).exp()
}
}
#[derive(Clone, Debug)]
pub struct ToneMap {
lut: Vec<f32>,
}
impl ToneMap {
pub fn camera(curve: &CameraTone, contrast: f64, whites: f64, blacks: f64) -> ToneMap {
let adjustment = Self::display(contrast, whites, blacks);
let neutral = contrast == 0.0 && whites == 0.0 && blacks == 0.0;
let lut = (0..LUT_N)
.map(|i| {
let ev = LUT_MIN_EV + (LUT_MAX_EV - LUT_MIN_EV) * i as f32 / (LUT_N - 1) as f32;
let y = curve.apply(GREY * 2f32.powf(ev));
if neutral { y } else { adjustment.apply(y) }
})
.collect();
ToneMap { lut }
}
/// `contrast`, `whites`, `blacks` in −100..100 (Lightroom slider units).
pub fn new(contrast: f64, whites: f64, blacks: f64) -> ToneMap {
let c = (contrast / 100.0) as f32;
@@ -116,6 +179,32 @@ fn smooth(e0: f32, e1: f32, x: f32) -> f32 {
mod tests {
use super::*;
#[test]
fn camera_curve_preserves_black_and_extends_headroom() {
let knots = std::array::from_fn(|i| {
let x = 0.005 * 1.15f32.powi(i as i32);
[x, 1.0 - (-3.0 * x).exp()]
});
let curve = CameraTone::new(knots).unwrap();
let base = ToneMap::camera(&curve, 0.0, 0.0, 0.0);
assert_eq!(base.apply(0.0), 0.0);
assert!(base.apply(0.1) < base.apply(0.2));
let mut previous = 0.0;
for i in 0..2000 {
let y = 1e-6 * 1.01f32.powi(i);
let v = base.apply(y);
assert!((0.0..=1.0).contains(&v));
assert!(v >= previous - 1e-6);
previous = v;
}
assert!((base.apply(0.1) - curve.apply(0.1)).abs() < 0.001);
assert!(ToneMap::camera(&curve, 50.0, 0.0, 0.0).apply(0.3) > base.apply(0.3));
let mut invalid = knots;
invalid[1][0] = invalid[0][0];
assert!(CameraTone::new(invalid).is_none());
assert!(serde_json::from_value::<CameraTone>(serde_json::json!({"knots": invalid})).is_err());
}
#[test]
fn monotone_and_bounded() {
for (c, w, b) in [(0.0, 0.0, 0.0), (100.0, 100.0, -100.0), (-100.0, -100.0, 100.0), (50.0, -30.0, -40.0)] {
+41 -1
View File
@@ -4,6 +4,11 @@
//! files) but not fsynced; unreadable files are deleted and re-rendered. When the total size
//! exceeds the budget, the least recently used files (by modification time, refreshed on read)
//! are removed down to 80 % of the budget.
//!
//! The cache only ever lists, counts or deletes files of its own name shape
//! (`<2 hex>/<32 hex>.jpg` and the `<32 hex>.tmp…` temp files of [`DiskCache::put`], see
//! [`is_cache_file`]): the directory may be shared with other files (e.g. a library opened on a
//! photo folder that already has a `thumbs/` folder), and those are never touched (issue #98).
use std::path::{Path, PathBuf};
use std::sync::Mutex;
@@ -96,13 +101,25 @@ impl DiskCache {
}
}
/// All cache files: (path, size, modified).
/// All cache files (only names of the cache's own shape, see [`is_cache_file`]):
/// (path, size, modified).
fn scan(&self) -> Vec<(PathBuf, u64, std::time::SystemTime)> {
let mut out = Vec::new();
let Ok(rd) = std::fs::read_dir(&self.dir) else { return out };
for sub in rd.flatten() {
let sub_name = sub.file_name();
let Some(sub_name) = sub_name.to_str() else { continue };
if !is_shard_name(sub_name) {
continue;
}
let Ok(files) = std::fs::read_dir(sub.path()) else { continue };
for f in files.flatten() {
let name = f.file_name();
let Some(name) = name.to_str() else { continue };
if !is_cache_file(sub_name, name) {
continue;
}
// `DirEntry::metadata` does not follow symlinks: a link is never a cache file
if let Ok(m) = f.metadata()
&& m.is_file()
{
@@ -144,6 +161,29 @@ impl DiskCache {
}
}
fn is_lower_hex(s: &str) -> bool {
s.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b))
}
/// A shard directory name: two lowercase hex digits.
fn is_shard_name(s: &str) -> bool {
s.len() == 2 && is_lower_hex(s)
}
/// Whether `name` inside shard directory `shard` is a file this cache wrote: `<32 hex>.jpg`
/// (whose first two digits are the shard) or a `<32 hex>.tmp<alphanumeric>` temp file of
/// [`DiskCache::put`]. Anything else in the cache directory belongs to someone else.
pub fn is_cache_file(shard: &str, name: &str) -> bool {
if !is_shard_name(shard) {
return false;
}
let Some((stem, ext)) = name.split_once('.') else { return false };
if stem.len() != 32 || !is_lower_hex(stem) || !stem.starts_with(shard) {
return false;
}
ext == "jpg" || ext.strip_prefix("tmp").is_some_and(|t| !t.is_empty() && t.bytes().all(|b| b.is_ascii_alphanumeric()))
}
/// Encode a thumbnail as the cache stores it (JPEG, RGB, 4:2:0).
pub fn encode_jpeg(img: &Rgba8) -> Option<Vec<u8>> {
if img.width == 0 || img.height == 0 || img.width > u16::MAX as usize || img.height > u16::MAX as usize {
+38
View File
@@ -154,3 +154,41 @@ fn pool_runs_inline() {
assert_eq!(p.run_inline(10), 1);
assert_eq!(p.try_recv().map(|d| d.result), Some(1));
}
#[test]
fn disk_cache_never_touches_foreign_files() {
// issue #98: a library opened on a folder that already has a `thumbs/` folder
let dir = temp_dir("foreign");
let foreign = [
dir.join("readme.txt"),
dir.join("ab").join("holiday.jpg"),
dir.join("ab").join("0123456789abcdef0123456789abcdef.jpg"), // wrong shard
dir.join("small").join("0123456789abcdef0123456789abcdef.jpg"),
dir.join("01").join("0123456789ABCDEF0123456789ABCDEF.jpg"), // upper case
dir.join("01").join("0123456789abcdef0123456789abcdef.jpeg"),
dir.join("01").join("0123456789abcdef0123456789abcdef.tmp"),
];
for p in &foreign {
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, vec![7u8; 50_000]).unwrap();
}
// a stale temp file of our own is cleaned up
let own_tmp = dir.join("01").join("0123456789abcdef0123456789abcdef.tmpThreadId9");
std::fs::write(&own_tmp, b"partial").unwrap();
let d = DiskCache::new(&dir, 40_000);
assert_eq!(d.size(), 7, "only the cache's own temp file counts");
// fill past the budget (forces prunes), then clear
for i in 0..40u8 {
d.put(hash_bytes(&[i, 2]), &gradient(160, 100, i));
}
assert!(d.size() <= 40_000);
d.clear();
assert_eq!(d.size(), 0);
assert!(!own_tmp.exists());
for p in &foreign {
assert!(p.exists(), "foreign file deleted: {}", p.display());
}
assert!(disk::is_cache_file("01", "0123456789abcdef0123456789abcdef.jpg"));
assert!(!disk::is_cache_file("01", "0123456789abcdef0123456789abcdef.jpg.bak"));
let _ = std::fs::remove_dir_all(&dir);
}
+22
View File
@@ -1,6 +1,7 @@
//! DNG (Adobe Digital Negative Specification 1.7): raw IFD selection, pixel data (via [`crate::tiffraw`]),
//! linearization, black/white levels, active area, default crop, CFA description, colour tags, opcode lists.
use crate::profile::{HsvTable, ProfileLook, ToneCurve};
use crate::tiffraw::{Packing, read_image_in};
use crate::{BlackLevel, Cfa, ColorData, Mat3, Mode, RawData, RawError, RawFormat, RawImage, Rect, Result, opcodes};
use lightcraft_color::Xy;
@@ -45,6 +46,27 @@ pub(crate) fn color_data(ifd0: &Ifd, raw: &Ifd) -> ColorData {
as_shot_white_xy: get(t::AS_SHOT_WHITE_XY).filter(|v| v.len() == 2 && v[0] > 0.0 && v[1] > 0.0).map(|v| Xy::new(v[0], v[1])),
baseline_exposure: get(t::BASELINE_EXPOSURE).and_then(|v| v.first().copied()).filter(|v| v.is_finite()).unwrap_or(0.0)
+ get(t::BASELINE_EXPOSURE_OFFSET).and_then(|v| v.first().copied()).filter(|v| v.is_finite()).unwrap_or(0.0),
profile: profile_look(ifd0, raw),
}
}
/// The profile look tags (`ProfileHueSatMap*`, `ProfileLookTable*`, `ProfileToneCurve`); malformed
/// ones are ignored. Like the colour tags, read from the raw IFD first, else IFD 0.
pub(crate) fn profile_look(ifd0: &Ifd, raw: &Ifd) -> ProfileLook {
let pick = |tag: u16| if raw.contains(tag) { raw } else { ifd0 };
let table = |dims: u16, data: u16, enc: u16| {
let ifd = pick(dims);
let dims = ifd.u64s(dims)?;
let data = ifd.f64s(data)?;
HsvTable::from_tags(&dims, &data, ifd.u64s(enc).and_then(|v| v.first().copied()).unwrap_or(0))
};
ProfileLook {
hue_sat_map: [
table(t::PROFILE_HUE_SAT_MAP_DIMS, t::PROFILE_HUE_SAT_MAP_DATA_1, t::PROFILE_HUE_SAT_MAP_ENCODING),
table(t::PROFILE_HUE_SAT_MAP_DIMS, t::PROFILE_HUE_SAT_MAP_DATA_2, t::PROFILE_HUE_SAT_MAP_ENCODING),
],
look_table: table(t::PROFILE_LOOK_TABLE_DIMS, t::PROFILE_LOOK_TABLE_DATA, t::PROFILE_LOOK_TABLE_ENCODING),
tone_curve: pick(t::PROFILE_TONE_CURVE).f64s(t::PROFILE_TONE_CURVE).and_then(|v| ToneCurve::from_tag(&v)),
}
}
+32
View File
@@ -179,6 +179,38 @@ pub fn write_dng(raw: &RawImage, opts: &DngWriteOptions) -> Result<Vec<u8>> {
ifd.set(t::AS_SHOT_WHITE_XY, rat_vec(&[xy.x, xy.y]));
}
ifd.set(t::BASELINE_EXPOSURE, Value::SRational(vec![srational(c.baseline_exposure)]));
// the source's own profile look travels with its data (conversions, smart previews, merges)
let p = &c.profile;
let table_tags = |ifd: &mut IfdBuilder, table: &crate::profile::HsvTable, dims: u16, enc: u16| {
ifd.set(dims, Value::Long(vec![table.hue_divisions as u32, table.sat_divisions as u32, table.val_divisions as u32]));
if table.srgb_value {
ifd.set(enc, Value::Long(vec![1]));
}
};
let flat = |table: &crate::profile::HsvTable| Value::Float(table.data.iter().flatten().copied().collect());
match &p.hue_sat_map {
[Some(a), b] => {
table_tags(&mut ifd, a, t::PROFILE_HUE_SAT_MAP_DIMS, t::PROFILE_HUE_SAT_MAP_ENCODING);
ifd.set(t::PROFILE_HUE_SAT_MAP_DATA_1, flat(a));
if let Some(b) =
b.as_ref().filter(|b| (b.hue_divisions, b.sat_divisions, b.val_divisions) == (a.hue_divisions, a.sat_divisions, a.val_divisions))
{
ifd.set(t::PROFILE_HUE_SAT_MAP_DATA_2, flat(b));
}
}
[None, Some(b)] => {
table_tags(&mut ifd, b, t::PROFILE_HUE_SAT_MAP_DIMS, t::PROFILE_HUE_SAT_MAP_ENCODING);
ifd.set(t::PROFILE_HUE_SAT_MAP_DATA_1, flat(b));
}
[None, None] => {}
}
if let Some(l) = &p.look_table {
table_tags(&mut ifd, l, t::PROFILE_LOOK_TABLE_DIMS, t::PROFILE_LOOK_TABLE_ENCODING);
ifd.set(t::PROFILE_LOOK_TABLE_DATA, flat(l));
}
if let Some(curve) = &p.tone_curve {
ifd.set(t::PROFILE_TONE_CURVE, Value::Float(curve.points.iter().flatten().copied().collect()));
}
for (list, tag) in [(&raw.opcodes.list1, t::OPCODE_LIST_1), (&raw.opcodes.list2, t::OPCODE_LIST_2), (&raw.opcodes.list3, t::OPCODE_LIST_3)] {
if !list.is_empty() {
ifd.set(tag, Value::Undefined(opcodes::write_list(list)));
+7 -1
View File
@@ -8,7 +8,8 @@
//! [`RawImage::develop`] does all of it plus `OpcodeList3` and the default crop; [`RawImage::develop_binned`]
//! produces the same at 1/k of the size straight from the mosaic (previews, thumbnails).
//! - [`color`] implements the DNG colour model (dual-illuminant interpolation, forward matrices, white balance)
//! and produces camera → linear Rec.2020 D65 matrices.
//! and produces camera → linear Rec.2020 D65 matrices; [`profile`] reads and applies a DNG's own profile
//! look tables and tone curve.
//!
//! Formats: DNG (uncompressed, lossless JPEG, lossy JPEG (Smart Previews), Deflate incl. floating point, tiled/stripped, CFA and LinearRaw),
//! Canon CR2, Nikon NEF/NRW (uncompressed, Huffman lossless / lossy compressed), Sony ARW (uncompressed, ARW2, lossless), Fujifilm RAF (uncompressed Bayer
@@ -29,6 +30,7 @@ pub mod highlight;
pub mod ljpeg;
pub mod opcodes;
mod preview;
pub mod profile;
mod tiffraw;
mod unpack;
mod vendor;
@@ -393,6 +395,10 @@ pub struct ColorData {
pub as_shot_white_xy: Option<Xy>,
/// EV to add for a "normal" rendering (`BaselineExposure` + `BaselineExposureOffset`).
pub baseline_exposure: f64,
/// The file's own camera-profile look (`ProfileHueSatMap*`, `ProfileLookTable*`,
/// `ProfileToneCurve`), applied by [`color`]'s users at render time.
#[serde(default)]
pub profile: profile::ProfileLook,
}
/// A decoded raw image.
+387
View File
@@ -0,0 +1,387 @@
//! DNG camera-profile look data carried by the file itself (Adobe DNG Specification 1.6/1.7,
//! chapter 6 and the `ProfileHueSatMap*`, `ProfileLookTable*` and `ProfileToneCurve` tags).
//!
//! - A hue/saturation/value table holds, for a grid of (value, hue, saturation) inputs, a hue shift
//! in degrees, a saturation scale and a value scale (value-major, hue-middle, saturation-minor).
//! Hue samples are spread evenly around the circle with wrap-around, saturation and value samples
//! evenly over 0..=1; a dimension with one division is constant. Lookups are tri-linear.
//! - Tables are applied in HSV of linear ProPhoto RGB (D50). With encoding 1 ("sRGB") the value
//! axis is indexed through the sRGB transfer curve (3D tables only).
//! - `ProfileHueSatMapData1/2` follow the colour matrices (interpolated by the same illuminant
//! weight); `ProfileLookTableData` comes after exposure compensation and before any tone curve.
//! - `ProfileToneCurve`: (input, output) pairs in linear gamma from (0, 0) to (1, 1).
//!
//! The data is read from the user's own file at run time; nothing here ships profile data.
//! Values above 1.0 (highlight headroom) keep their headroom: the table's value scale only ever
//! lowers them, so a lookup can't clip what the scene-referred pipeline still needs.
use lightcraft_color::transfer::{linear_to_srgb, srgb_to_linear};
use lightcraft_color::{D50, D65, Mat3, PROPHOTO, REC2020, bradford};
use serde::{Deserialize, Serialize};
/// Upper bound on table entries (Adobe profiles use ≤ 90 × 30 × 16; guards header-driven sizes).
const MAX_ENTRIES: usize = 1 << 20;
/// A DNG hue/saturation/value mapping table (`ProfileHueSatMapData*` / `ProfileLookTableData`).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct HsvTable {
pub hue_divisions: usize,
pub sat_divisions: usize,
pub val_divisions: usize,
/// `(hue shift °, saturation scale, value scale)`, value-major, hue-middle, saturation-minor.
pub data: Vec<[f32; 3]>,
/// The value axis is indexed through the sRGB encoding curve (`…Encoding` = 1).
pub srgb_value: bool,
}
impl HsvTable {
/// A table from its `…Dims` (hue, saturation, value divisions), data floats and encoding.
/// `None` when the dimensions are out of range, the data length doesn't match or a value is
/// not finite.
pub fn from_tags(dims: &[u64], data: &[f64], encoding: u64) -> Option<HsvTable> {
let [h, s, v] = <[u64; 3]>::try_from(dims).ok()?;
let (h, s, v) = (usize::try_from(h).ok()?, usize::try_from(s).ok()?, usize::try_from(v).ok()?);
if h < 1 || s < 2 || v < 1 || h > 4096 || s > 4096 || v > 4096 {
return None;
}
let n = h.checked_mul(s)?.checked_mul(v)?;
if n > MAX_ENTRIES || data.len() != n * 3 || data.iter().any(|x| !x.is_finite()) {
return None;
}
let data = data.as_chunks::<3>().0.iter().map(|c| [c[0] as f32, (c[1] as f32).max(0.0), (c[2] as f32).max(0.0)]).collect();
Some(HsvTable { hue_divisions: h, sat_divisions: s, val_divisions: v, data, srgb_value: encoding == 1 && v > 1 })
}
/// The entry-wise blend `g · a + (1 − g) · b` of two tables of the same shape (`None` otherwise).
pub fn blend(a: &HsvTable, b: &HsvTable, g: f64) -> Option<HsvTable> {
if (a.hue_divisions, a.sat_divisions, a.val_divisions) != (b.hue_divisions, b.sat_divisions, b.val_divisions) {
return None;
}
let g = g.clamp(0.0, 1.0) as f32;
let data = a.data.iter().zip(&b.data).map(|(x, y)| std::array::from_fn(|i| g * x[i] + (1.0 - g) * y[i])).collect();
Some(HsvTable { data, ..a.clone() })
}
fn entry(&self, v: usize, h: usize, s: usize) -> [f32; 3] {
self.data.get((v * self.hue_divisions + h) * self.sat_divisions + s).copied().unwrap_or([0.0, 1.0, 1.0])
}
/// Tri-linear lookup at hue `h` (degrees), saturation `s` and value `v` (table coordinates).
pub fn lookup(&self, h: f32, s: f32, v: f32) -> [f32; 3] {
let hf = (h.rem_euclid(360.0) / 360.0 * self.hue_divisions as f32).clamp(0.0, self.hue_divisions as f32);
let h0 = (hf as usize).min(self.hue_divisions - 1);
let h1 = (h0 + 1) % self.hue_divisions;
let th = (hf - h0 as f32).clamp(0.0, 1.0);
let axis = |x: f32, n: usize| {
let f = x.clamp(0.0, 1.0) * (n - 1) as f32;
let i0 = (f as usize).min(n.saturating_sub(2));
let i1 = (i0 + 1).min(n - 1);
(i0, i1, (f - i0 as f32).clamp(0.0, 1.0))
};
let (s0, s1, ts) = axis(s, self.sat_divisions);
let (v0, v1, tv) = if self.val_divisions > 1 { axis(v, self.val_divisions) } else { (0, 0, 0.0) };
let lerp = |a: [f32; 3], b: [f32; 3], t: f32| -> [f32; 3] { std::array::from_fn(|i| a[i] + (b[i] - a[i]) * t) };
let plane = |vi: usize| {
let lo = lerp(self.entry(vi, h0, s0), self.entry(vi, h0, s1), ts);
let hi = lerp(self.entry(vi, h1, s0), self.entry(vi, h1, s1), ts);
lerp(lo, hi, th)
};
if v0 == v1 { plane(v0) } else { lerp(plane(v0), plane(v1), tv) }
}
/// Apply the table to one linear ProPhoto RGB colour.
pub fn apply(&self, rgb: [f32; 3]) -> [f32; 3] {
let max = rgb[0].max(rgb[1]).max(rgb[2]);
if max.is_nan() || max <= 0.0 || !max.is_finite() {
return rgb;
}
let min = rgb[0].min(rgb[1]).min(rgb[2]);
let d = max - min;
let hue = hue_of(rgb, max, d);
let sat = d / max;
let v_index = if self.srgb_value { linear_to_srgb(max.min(1.0)) } else { max };
let [shift, ss, vs] = self.lookup(hue, sat, v_index);
// saturation and value: scaled and clipped to 1.0 (DNG chapter 6), except that inputs
// already above 1 (out-of-ProPhoto colours, highlight headroom) are only ever lowered.
let s2 = limited(sat, ss);
let v2 = if self.srgb_value && max <= 1.0 {
srgb_to_linear((linear_to_srgb(max) * vs).min(1.0))
} else if self.srgb_value {
max * srgb_to_linear(vs.min(1.0)).min(1.0)
} else {
limited(max, vs)
};
hsv_to_rgb(hue + shift, s2, v2)
}
}
/// `x · k` clipped to 1, for `x ≤ 1`; above 1, only scale factors below 1 apply (continuous at 1).
fn limited(x: f32, k: f32) -> f32 {
if x <= 1.0 { (x * k).min(1.0) } else { x * k.min(1.0) }
}
fn hue_of(c: [f32; 3], max: f32, d: f32) -> f32 {
if d <= 0.0 {
return 0.0;
}
let h = if max == c[0] {
((c[1] - c[2]) / d).rem_euclid(6.0)
} else if max == c[1] {
(c[2] - c[0]) / d + 2.0
} else {
(c[0] - c[1]) / d + 4.0
};
h * 60.0
}
/// HSV → RGB for any saturation ≥ 0 (above 1 yields negative components, the inverse of
/// [`hue_of`] for out-of-gamut inputs).
fn hsv_to_rgb(h: f32, s: f32, v: f32) -> [f32; 3] {
let hp = h.rem_euclid(360.0) / 60.0;
let sector = (hp as u32).min(5);
let f = hp - sector as f32;
let (p, q, t) = (v * (1.0 - s), v * (1.0 - s * f), v * (1.0 - s * (1.0 - f)));
match sector {
0 => [v, t, p],
1 => [q, v, p],
2 => [p, v, t],
3 => [p, q, v],
4 => [t, p, v],
_ => [v, p, q],
}
}
/// A DNG tone curve (`ProfileToneCurve`): linear-gamma (input, output) pairs, increasing input.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct ToneCurve {
pub points: Vec<[f32; 2]>,
}
impl ToneCurve {
/// From the tag's flat float pairs. `None` unless there are 2..=65536 finite pairs within
/// 0..=1 with strictly increasing inputs.
pub fn from_tag(v: &[f64]) -> Option<ToneCurve> {
if !v.len().is_multiple_of(2) || v.len() < 4 || v.len() > 131_072 {
return None;
}
let points: Vec<[f32; 2]> = v.as_chunks::<2>().0.iter().map(|p| [p[0] as f32, p[1] as f32]).collect();
let ok = points.iter().all(|p| p.iter().all(|x| x.is_finite() && (0.0..=1.0).contains(x))) && points.windows(2).all(|w| w[1][0] > w[0][0]);
ok.then_some(ToneCurve { points })
}
/// The curve at `x` (0..=1): monotone cubic (Fritsch–Carlson) through the samples.
pub fn eval(&self, x: f32) -> f32 {
let p = &self.points;
let n = p.len();
let x = x.clamp(p[0][0], p[n - 1][0]);
let i = p.partition_point(|q| q[0] <= x).clamp(1, n - 1) - 1;
let slope = |k: usize| (p[k + 1][1] - p[k][1]) / (p[k + 1][0] - p[k][0]);
let tangent = |k: usize| -> f32 {
if k == 0 {
return slope(0);
}
if k == n - 1 {
return slope(n - 2);
}
let (a, b) = (slope(k - 1), slope(k));
if a * b <= 0.0 { 0.0 } else { 2.0 / (1.0 / a + 1.0 / b) }
};
let (x0, x1) = (p[i][0], p[i + 1][0]);
let h = x1 - x0;
let t = (x - x0) / h;
let (t2, t3) = (t * t, t * t * t);
let y = (2.0 * t3 - 3.0 * t2 + 1.0) * p[i][1]
+ (t3 - 2.0 * t2 + t) * h * tangent(i)
+ (-2.0 * t3 + 3.0 * t2) * p[i + 1][1]
+ (t3 - t2) * h * tangent(i + 1);
y.clamp(0.0, 1.0)
}
}
/// The DNG profile look tags of a file (all optional).
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct ProfileLook {
/// `ProfileHueSatMapData1/2` (same shape, `ProfileHueSatMapDims`).
pub hue_sat_map: [Option<HsvTable>; 2],
/// `ProfileLookTableData`.
pub look_table: Option<HsvTable>,
/// `ProfileToneCurve`.
pub tone_curve: Option<ToneCurve>,
}
impl ProfileLook {
pub fn is_empty(&self) -> bool {
self.hue_sat_map.iter().all(Option::is_none) && self.look_table.is_none() && self.tone_curve.is_none()
}
/// The hue/saturation map for calibration-1 weight `g` (one table: used for any white).
pub fn hue_sat_for(&self, g: f64) -> Option<HsvTable> {
match &self.hue_sat_map {
[Some(a), Some(b)] => HsvTable::blend(a, b, g).or_else(|| Some(a.clone())),
[Some(a), None] | [None, Some(a)] => Some(a.clone()),
[None, None] => None,
}
}
}
/// The tables resolved for one white balance, applied to linear Rec.2020 D65 pixels.
#[derive(Clone, Debug)]
pub struct ProfileTables {
hue_sat: Option<HsvTable>,
look: Option<HsvTable>,
to_prophoto: [[f32; 3]; 3],
from_prophoto: [[f32; 3]; 3],
}
impl ProfileTables {
/// `None` when the profile has no hue/saturation map and no look table.
pub fn new(look: &ProfileLook, illuminant_weight: f64) -> Option<ProfileTables> {
let hue_sat = look.hue_sat_for(illuminant_weight);
if hue_sat.is_none() && look.look_table.is_none() {
return None;
}
let to: Mat3 = PROPHOTO.from_xyz().mul(&bradford(D65, D50)).mul(&REC2020.to_xyz());
let from = to.inverse()?;
Some(ProfileTables { hue_sat, look: look.look_table.clone(), to_prophoto: to.to_f32(), from_prophoto: from.to_f32() })
}
/// Hue/saturation map, then `gain` (exposure compensation), then the look table:
/// linear Rec.2020 D65 in, linear Rec.2020 D65 out.
#[inline]
pub fn apply(&self, rgb: [f32; 3], gain: f32) -> [f32; 3] {
let m = &self.to_prophoto;
let mut p: [f32; 3] = std::array::from_fn(|i| m[i][0] * rgb[0] + m[i][1] * rgb[1] + m[i][2] * rgb[2]);
if let Some(t) = &self.hue_sat {
p = t.apply(p);
}
p = p.map(|v| v * gain);
if let Some(t) = &self.look {
p = t.apply(p);
}
let m = &self.from_prophoto;
std::array::from_fn(|i| m[i][0] * p[0] + m[i][1] * p[1] + m[i][2] * p[2])
}
}
#[cfg(test)]
mod tests {
use super::*;
fn identity(h: usize, s: usize, v: usize) -> HsvTable {
HsvTable { hue_divisions: h, sat_divisions: s, val_divisions: v, data: vec![[0.0, 1.0, 1.0]; h * s * v], srgb_value: false }
}
fn close(a: [f32; 3], b: [f32; 3], tol: f32) -> bool {
a.iter().zip(&b).all(|(x, y)| (x - y).abs() <= tol)
}
#[test]
fn identity_table_is_identity() {
let t = identity(6, 3, 1);
for c in [[0.2, 0.5, 0.1], [0.9, 0.9, 0.9], [0.0, 0.0, 0.4], [3.0, 1.0, 0.5], [0.3, -0.05, 0.2]] {
assert!(close(t.apply(c), c, 1e-5), "{c:?} → {:?}", t.apply(c));
}
assert_eq!(t.apply([0.0; 3]), [0.0; 3]);
assert_eq!(t.apply([-1.0, -2.0, -0.5]), [-1.0, -2.0, -0.5]);
}
#[test]
fn parses_tags_and_rejects_bad_shapes() {
let data: Vec<f64> = (0..4).flat_map(|_| [0.0, 1.0, 1.0]).collect();
assert!(HsvTable::from_tags(&[2, 2, 1], &data, 0).is_some());
assert!(HsvTable::from_tags(&[2, 2], &data, 0).is_none());
assert!(HsvTable::from_tags(&[2, 1, 2], &data, 0).is_none(), "saturation needs ≥ 2 divisions");
assert!(HsvTable::from_tags(&[2, 2, 2], &data, 0).is_none(), "length mismatch");
assert!(HsvTable::from_tags(&[u64::MAX, 2, 2], &data, 0).is_none());
let mut nan = data.clone();
nan[1] = f64::NAN;
assert!(HsvTable::from_tags(&[2, 2, 1], &nan, 0).is_none());
// the sRGB encoding only applies to 3D tables
assert!(!HsvTable::from_tags(&[2, 2, 1], &data, 1).unwrap().srgb_value);
}
#[test]
fn saturation_scale_follows_the_table_layout() {
// 4 hues (0°, 90°, 180°, 270°), 2 saturations; saturation-minor: entry (h, s) at h·2 + s.
let mut t = identity(4, 2, 1);
t.data[1] = [0.0, 1.5, 1.0]; // hue 0°, saturation 1 → ×1.5
let red = [0.5, 0.4, 0.4]; // hue 0°, saturation 0.2
let out = t.apply(red);
// half-way along the saturation axis is ×1.1 (0.2 → 0.22 sat), value kept
let sat = (out[0] - out[1].min(out[2])) / out[0];
assert!((sat - 0.2 * 1.1).abs() < 1e-4, "{out:?}");
assert!((out[0] - 0.5).abs() < 1e-6);
// other hues are untouched (hue 180°)
let cyan = [0.3, 0.4, 0.4];
assert!(close(t.apply(cyan), cyan, 1e-6));
// saturation clips to 1.0
assert!(t.apply([1.0, 0.1, 0.1]).iter().all(|v| *v >= -1e-6));
}
#[test]
fn hue_wraps_and_value_scales() {
// 2 hues: entries at 0° and 180°, hue shift +20° at 0°, value ×0.5 at 180°.
let mut t = identity(2, 2, 1);
t.data = vec![[20.0, 1.0, 1.0], [20.0, 1.0, 1.0], [0.0, 1.0, 0.5], [0.0, 1.0, 0.5]];
// 350° sits between the last sample (180°) and the first (0° = 360°), 10/180 from 360
assert!((t.lookup(350.0, 0.5, 0.5)[0] - 20.0 * (170.0 / 180.0)).abs() < 1e-3);
assert!((t.lookup(180.0, 0.5, 0.5)[2] - 0.5).abs() < 1e-6);
// headroom above 1 only goes down
let bright = [2.0, 2.4, 2.4];
let out = t.apply(bright);
assert!((out[1] - 1.2).abs() < 1e-5, "{out:?}");
let mut up = identity(1, 2, 1);
up.data = vec![[0.0, 1.0, 2.0]; 2];
assert!((up.apply([0.4, 0.4, 0.4])[0] - 0.8).abs() < 1e-6);
assert!((up.apply([0.8, 0.8, 0.8])[0] - 1.0).abs() < 1e-6, "clipped to 1");
assert!((up.apply([3.0, 3.0, 3.0])[0] - 3.0).abs() < 1e-6, "headroom kept");
}
#[test]
fn value_axis_and_srgb_encoding() {
// 1 hue, 2 sats, 2 values: saturation ×2 at value 1 only
let mut t = identity(1, 2, 2);
t.data[3] = [0.0, 2.0, 1.0];
let sat = |c: [f32; 3]| (c[0] - c[2]) / c[0];
let dark = [0.25, 0.2, 0.2]; // V 0.25, S 0.2
// linear: 1/4 of the way → S scale at s=0.2: lerp(1, lerp(1,2,0.2)=1.2, 0.25) = 1.05
assert!((sat(t.apply(dark)) - 0.2 * 1.05).abs() < 1e-4);
t.srgb_value = true;
let k = linear_to_srgb(0.25); // ≈ 0.537
let want = 0.2 * (1.0 + 0.2 * k);
assert!((sat(t.apply(dark)) - want).abs() < 1e-4);
}
#[test]
fn blend_and_tables_round_trip_through_prophoto() {
let a = identity(2, 2, 1);
let mut b = identity(2, 2, 1);
b.data[1] = [10.0, 2.0, 1.0];
let m = HsvTable::blend(&a, &b, 0.25).unwrap();
assert_eq!(m.data[1], [7.5, 1.75, 1.0]);
assert!(HsvTable::blend(&a, &identity(3, 2, 1), 0.5).is_none());
let look = ProfileLook { hue_sat_map: [Some(a.clone()), None], look_table: Some(a), tone_curve: None };
let t = ProfileTables::new(&look, 0.5).unwrap();
let c = [0.3, 0.2, 0.1];
assert!(close(t.apply(c, 2.0), [0.6, 0.4, 0.2], 1e-4), "{:?}", t.apply(c, 2.0));
assert!(ProfileTables::new(&ProfileLook::default(), 0.5).is_none());
}
#[test]
fn tone_curve_interpolates_monotonically() {
assert!(ToneCurve::from_tag(&[0.0, 0.0]).is_none());
assert!(ToneCurve::from_tag(&[0.0, 0.0, 0.5, 0.6, 0.4, 0.7, 1.0, 1.0]).is_none(), "inputs must increase");
assert!(ToneCurve::from_tag(&[0.0, 0.0, 1.0, 1.5]).is_none());
let c = ToneCurve::from_tag(&[0.0, 0.0, 0.25, 0.4, 0.5, 0.7, 1.0, 1.0]).unwrap();
assert_eq!(c.eval(0.0), 0.0);
assert!((c.eval(0.25) - 0.4).abs() < 1e-6);
assert!((c.eval(1.0) - 1.0).abs() < 1e-6);
let mut last = 0.0;
for i in 0..=100 {
let y = c.eval(i as f32 / 100.0);
assert!(y >= last - 1e-6, "monotone at {i}");
last = y;
}
assert!(c.eval(0.1) > 0.1 && c.eval(0.1) < 0.4);
}
}
+37
View File
@@ -193,3 +193,40 @@ fn corpus_nef_compressed_matches_uncompressed() {
assert!(hi <= 4095.0 && hi - lo > 500.0, "D7000 block means {lo}..{hi}");
}
}
/// Issue #138: DNGs converted by Adobe software carry their camera profile's hue/saturation map and
/// look table; we read them (and render with them). Camera-written DNGs here carry none.
#[test]
fn corpus_adobe_dngs_carry_profile_looks() {
let dir = corpus_root().join("raw");
let Ok(rd) = std::fs::read_dir(&dir) else {
eprintln!("skip: {} absent", dir.display());
return;
};
let mut seen = 0;
for p in rd.flatten().map(|e| e.path()) {
let name = p.file_name().unwrap().to_string_lossy().to_lowercase();
if !name.starts_with("dng-") || !name.ends_with(".dng") {
continue;
}
let info = probe_info(&std::fs::read(&p).unwrap()).unwrap_or_else(|e| panic!("{name}: {e}"));
let look = &info.color.profile;
if name.starts_with("dng-adobe-") {
seen += 1;
let hsm = look.hue_sat_map[0].as_ref().unwrap_or_else(|| panic!("{name}: no hue/sat map"));
assert!(hsm.hue_divisions > 1 && hsm.sat_divisions > 1, "{name}");
assert!(look.look_table.is_some(), "{name}: no look table");
// a profile applied to a mid grey keeps it (close to) neutral
let t = lightcraft_raw::profile::ProfileTables::new(look, 0.5).unwrap();
let g = t.apply([0.18; 3], 1.0);
assert!(g.iter().all(|v| (v - g[0]).abs() < 0.01 * g[0].max(0.01)), "{name}: grey → {g:?}");
}
eprintln!(
"{name:44} profile look: hsm {} look {} tone {}",
look.hue_sat_map[0].is_some(),
look.look_table.is_some(),
look.tone_curve.is_some()
);
}
eprintln!("{seen} Adobe-converted DNGs checked");
}
+71
View File
@@ -182,6 +182,11 @@ fn base_ifd(w: usize, h: usize, bits: u16, cpp: usize, cfa: bool) -> IfdBuilder
}
fn dng(raw: IfdBuilder, order: ByteOrder) -> Vec<u8> {
dng_with(raw, order, |_| {})
}
/// [`dng`] with extra IFD 0 tags (where DNG writers put camera-profile tags).
fn dng_with(raw: IfdBuilder, order: ByteOrder, extra: impl FnOnce(&mut IfdBuilder)) -> Vec<u8> {
let mut ifd0 = IfdBuilder::new();
ifd0.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![1]));
ifd0.set(t::DNG_VERSION, Value::Byte(vec![1, 6, 0, 0]));
@@ -200,6 +205,7 @@ fn dng(raw: IfdBuilder, order: ByteOrder) -> Vec<u8> {
exif.set(t::ISO_SPEED, Value::Short(vec![200]));
ifd0.set_child(t::EXIF_IFD, exif);
ifd0.add_sub_ifd(raw);
extra(&mut ifd0);
TiffWriter::new(order, false).write(&[ifd0]).unwrap()
}
@@ -461,3 +467,68 @@ fn rejects_unsupported_and_broken() {
raw.set_image(ImageData::Strips { rows_per_strip: 8, strips: vec![vec![0; 128]] });
assert!(decode(&dng(raw, ByteOrder::Little)).is_err());
}
/// A Lightroom-style DNG: profile look tags in IFD 0, LinearRaw data in a SubIFD (issue #138).
fn profile_dng(order: ByteOrder, extra: impl FnOnce(&mut IfdBuilder)) -> Vec<u8> {
let (w, h) = (8, 6);
let px = pattern(w, h, 3, 16);
let mut raw = base_ifd(w, h, 16, 3, false);
raw.set(t::COMPRESSION, Value::Short(vec![1]));
raw.set_image(ImageData::Strips { rows_per_strip: h as u32, strips: vec![u16_bytes(&px, order)] });
dng_with(raw, order, extra)
}
fn hsv_floats(h: usize, s: usize, v: usize, f: impl Fn(usize, usize, usize) -> [f32; 3]) -> Vec<f32> {
(0..v).flat_map(|vi| (0..h).flat_map(move |hi| (0..s).map(move |si| (vi, hi, si)))).flat_map(|(vi, hi, si)| f(vi, hi, si)).collect()
}
#[test]
fn profile_look_tags_are_read_from_ifd0() {
for order in [ByteOrder::Little, ByteOrder::Big] {
let bytes = profile_dng(order, |ifd0| {
ifd0.set(t::PROFILE_HUE_SAT_MAP_DIMS, Value::Long(vec![6, 3, 1]));
ifd0.set(t::PROFILE_HUE_SAT_MAP_DATA_1, Value::Float(hsv_floats(6, 3, 1, |_, h, s| [h as f32, 1.0 + s as f32 * 0.1, 1.0])));
ifd0.set(t::PROFILE_HUE_SAT_MAP_DATA_2, Value::Float(hsv_floats(6, 3, 1, |_, _, _| [0.0, 1.0, 1.0])));
ifd0.set(t::PROFILE_LOOK_TABLE_DIMS, Value::Long(vec![4, 2, 3]));
ifd0.set(t::PROFILE_LOOK_TABLE_DATA, Value::Float(hsv_floats(4, 2, 3, |v, _, _| [0.0, 1.0, 1.0 - v as f32 * 0.05])));
ifd0.set(t::PROFILE_LOOK_TABLE_ENCODING, Value::Long(vec![1]));
ifd0.set(t::PROFILE_TONE_CURVE, Value::Float(vec![0.0, 0.0, 0.25, 0.15, 0.5, 0.55, 1.0, 1.0]));
});
let img = decode(&bytes).unwrap();
assert_eq!(img.cpp, 3);
let p = &img.color.profile;
let hsm = p.hue_sat_map[0].as_ref().unwrap();
assert_eq!((hsm.hue_divisions, hsm.sat_divisions, hsm.val_divisions), (6, 3, 1));
// value-major, hue-middle, saturation-minor: entry (h = 2, s = 1) is at 2·3 + 1
assert_eq!(hsm.data[7], [2.0, 1.1, 1.0]);
assert!(p.hue_sat_map[1].is_some());
let look = p.look_table.as_ref().unwrap();
assert_eq!((look.hue_divisions, look.sat_divisions, look.val_divisions, look.srgb_value), (4, 2, 3, true));
assert_eq!(look.data[2 * 4 * 2], [0.0, 1.0, 0.9]);
assert_eq!(p.tone_curve.as_ref().unwrap().points.len(), 4);
// the header-only probe sees the same profile
assert_eq!(lightcraft_raw::probe_info(&bytes).unwrap().color.profile, *p);
// our DNG writer keeps the look with the data (conversions, smart previews)
let again = decode(&write_dng(&img, &DngWriteOptions::default()).unwrap()).unwrap();
assert_eq!(again.color.profile, *p);
}
}
#[test]
fn malformed_profile_look_tags_are_ignored() {
let bytes = profile_dng(ByteOrder::Little, |ifd0| {
// data count doesn't match the dimensions
ifd0.set(t::PROFILE_HUE_SAT_MAP_DIMS, Value::Long(vec![6, 3, 1]));
ifd0.set(t::PROFILE_HUE_SAT_MAP_DATA_1, Value::Float(vec![0.0, 1.0, 1.0]));
// one saturation division is not allowed
ifd0.set(t::PROFILE_LOOK_TABLE_DIMS, Value::Long(vec![2, 1, 1]));
ifd0.set(t::PROFILE_LOOK_TABLE_DATA, Value::Float(vec![0.0, 1.0, 1.0, 0.0, 1.0, 1.0]));
// decreasing inputs and a NaN
ifd0.set(t::PROFILE_TONE_CURVE, Value::Float(vec![0.0, 0.0, 0.6, 0.5, 0.4, f32::NAN, 1.0, 1.0]));
});
let img = decode(&bytes).unwrap();
assert!(img.color.profile.is_empty(), "{:?}", img.color.profile);
// and without any of the tags there is nothing to apply
let plain = decode(&profile_dng(ByteOrder::Little, |_| {})).unwrap();
assert!(plain.color.profile.is_empty());
}
+9
View File
@@ -152,6 +152,15 @@ pub const CAMERA_CALIBRATION_SIGNATURE: u16 = 50931;
pub const PROFILE_CALIBRATION_SIGNATURE: u16 = 50932;
pub const AS_SHOT_PROFILE_NAME: u16 = 50934;
pub const PROFILE_NAME: u16 = 50936;
pub const PROFILE_HUE_SAT_MAP_DIMS: u16 = 50937;
pub const PROFILE_HUE_SAT_MAP_DATA_1: u16 = 50938;
pub const PROFILE_HUE_SAT_MAP_DATA_2: u16 = 50939;
pub const PROFILE_TONE_CURVE: u16 = 50940;
pub const PROFILE_EMBED_POLICY: u16 = 50941;
pub const PROFILE_LOOK_TABLE_DIMS: u16 = 50981;
pub const PROFILE_LOOK_TABLE_DATA: u16 = 50982;
pub const PROFILE_HUE_SAT_MAP_ENCODING: u16 = 51107;
pub const PROFILE_LOOK_TABLE_ENCODING: u16 = 51108;
pub const BASELINE_EXPOSURE_OFFSET: u16 = 51109;
pub const DEFAULT_BLACK_RENDER: u16 = 51110;
pub const FORWARD_MATRIX_1: u16 = 50964;
+3
View File
@@ -25,3 +25,6 @@ log = { workspace = true }
[dev-dependencies]
lightcraft-codecs = { workspace = true }
[build-dependencies]
serde_json = { workspace = true }
+18
View File
@@ -0,0 +1,18 @@
//! Compile translated format strings so Rust checks both languages' placeholders.
use std::{collections::BTreeMap, env, fs, path::PathBuf};
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("cargo:rerun-if-changed=locales/ja-formats.json");
let messages: BTreeMap<String, String> = serde_json::from_str(&fs::read_to_string("locales/ja-formats.json")?)?;
let mut source = String::from("macro_rules! tr_format {\n");
for (english, japanese) in messages {
let en = serde_json::to_string(&english)?;
let ja = serde_json::to_string(&japanese)?;
source.push_str(&format!(
"({en} $(, $($args:tt)*)?) => {{ if $crate::i18n::is_japanese() {{ format!({ja} $(, $($args)*)?) }} else {{ format!({en} $(, $($args)*)?) }} }};\n"
));
}
source.push_str("}\npub(crate) use tr_format;\n");
fs::write(PathBuf::from(env::var("OUT_DIR")?).join("ja-formats.rs"), source)?;
Ok(())
}
+104
View File
@@ -0,0 +1,104 @@
{
" · {skipped} skipped (file exists)": "・{skipped}件は同名ファイルのためスキップ",
" · {} duplicate{} skipped": "・重複{}枚をスキップ{:.0}",
" · {} failed: {}": "・{}件失敗:{}",
" · {} moved": "・{}件移動",
" · {} not readable": "・{}件は読み込み不可",
", {failed} file{} not readable": "、{failed}件を読み込めません{:.0}",
"Add {local_n} to My Photos": "{local_n}枚をマイフォトに追加",
"Add {} Photo{}": "{}枚の写真を追加{:.0}",
"Add “{}” to Favorites": "「{}」をお気に入りに追加",
"Added {n} photo{} to My Photos": "マイフォトに{n}枚を追加しました{:.0}",
"Added {n} photo{} to “{}”": "「{1}」に{n}枚を追加しました{0:.0}",
"All {n} selected photos move by the same amount.": "選択中の{n}枚を同じ時間だけずらします",
"Auto Sync: edits apply to {n} photos": "自動同期:{n}枚に編集を適用",
"Automatic ({} MB)": "自動({}MB)",
"Click {}:1": "クリック時 {}:1",
"Create {n} Virtual Copies": "{n}枚の仮想コピーを作成",
"Decoder: {reason}": "デコーダー:{reason}",
"Delete \"{label}\"": "「{label}」を削除",
"Delete \"{label}\" (and the mask)": "「{label}」とマスクを削除",
"Delete {n} Photos": "{n}枚の写真を削除",
"Delete “{current}”": "「{current}」を削除",
"Duplicate \"{label}\"": "「{label}」を複製",
"Example: {example}": "例:{example}",
"Exported {n} photo{}": "{n}枚を書き出しました{:.0}",
"Exported {ok} of {total} photo{}": "{total}枚中{ok}枚を書き出しました{:.0}",
"Exported {} point curve preset{}": "{}件のポイントカーブを書き出しました{:.0}",
"Exported {} preset{}": "{}件のプリセットを書き出しました{:.0}",
"Exporting {} of {total}": "書き出し中:{total}枚中{}枚",
"Filter bar — {active} active filter{}": "フィルターバー:{active}件適用中{:.0}",
"Healed {n} dust spot{}": "{n}か所のほこりを修復しました{:.0}",
"Imported {n} point curve preset{}": "{n}件のポイントカーブを読み込みました{:.0}",
"Imported {n} preset{}": "{n}件のプリセットを読み込みました{:.0}",
"Imported {n} preset{}, {f} file{} not readable": "{n}件のプリセットを読み込みました{:.0}。{f}件は読み込めません{:.0}",
"Imported {} photo{}": "{}枚を読み込みました{:.0}",
"Matches: {}": "一致条件:{}",
"Merging… {stage} {:.0}%": "合成中… {stage} {:.0}%",
"Move {what} to Recently Deleted?": "{what}を「最近削除した写真」に移動しますか?",
"Named ({named_n})": "名前付き({named_n})",
"Paste into {n} photo{} — only settings that were copied are pasted": "{n}枚に貼り付け{:.0}(コピーされた設定だけを適用)",
"Preset: {name}": "プリセット:{name}",
"Preview only — editing the camera's embedded JPEG ({})": "プレビューのみ:カメラ内蔵のJPEGを編集中({})",
"Preview only — {}": "プレビューのみ:{}",
"Profile: {}": "プロファイル:{}",
"Proof Copy ({})": "校正用コピー({})",
"Proof Preview · {}": "校正プレビュー・{}",
"Quick Develop · {n} photos": "クイック現像・{n}枚",
"Rated {}": "評価:{}",
"Reading photos… {done} of {total}": "写真を読み込み中:{total}枚中{done}枚",
"Redo {}": "やり直す:{}",
"Remove “{}” from Favorites": "「{}」をお気に入りから削除",
"Rename {n} Photos…": "{n}枚の写真の名前を変更…",
"Rename “{name}”": "「{name}」の名前を変更",
"Rendering on the CPU: {why}": "CPUで描画:{why}",
"Reset {} Channel": "{}チャンネルをリセット",
"Saved export preset “{}”": "書き出しプリセット「{}」を保存しました",
"Show {hidden} hidden location{}": "非表示の{hidden}か所を表示{:.0}",
"Showing {n} of {}": "{}枚中{n}枚を表示",
"Smart album: {}": "スマートアルバム:{}",
"Undo {}": "取り消す:{}",
"Version {} — a clean-room, pure-Rust photo library and raw developer.": "バージョン{} — Rust製の写真管理・RAW現像アプリ",
"Version: {}": "バージョン:{}",
"none (CPU rendering): {why}": "なし(CPU描画):{why}",
"the {n} selected photos": "選択中の{n}枚",
"the {} photos in view": "表示中の{}枚",
"these {count} photos": "この{count}枚の写真",
"{count} smart preview{} · {:.1} MB": "スマートプレビュー{count}枚{:.0}・{:.1}MB",
"{f} folders (and their subfolders)": "{f}個のフォルダとそのサブフォルダ",
"{label} Grading": "{label}のカラーグレーディング",
"{label} {d:+} on every selected photo": "選択したすべての写真の{label}を{d:+}調整",
"{label}, {n} photos": "{label}、{n}枚",
"{matching} of {t} photos": "{t}枚中{matching}枚",
"{matching} photos": "{matching}枚",
"{mode} spot {} of {}": "{mode}:{1}か所中{0}か所",
"{msg} ({failed} failed)": "{msg}({failed}件失敗)",
"{name} Label": "{name}のラベル",
"{name} Off": "{name}を無効にする",
"{name} On": "{name}を有効にする",
"{name} — double-click to reset it": "{name}:ダブルクリックでリセット",
"{n} change{} saved in memory but not written to disk: {e}\nLightCraft retries automatically; quitting now would lose {}.": "{n}件の変更{:.0}をディスクに保存できません:{e}\n自動で再試行します。終了すると変更が失われます{:.0}",
"{n} found · {sel} selected": "{n}枚検出・{sel}枚選択",
"{n} photos": "{n}枚",
"{n} photos in {} are hidden by the filters above": "{}の{n}枚は上のフィルターで非表示になっています",
"{n} photo{}": "{n}枚{:.0}",
"{n} photo{} match · updates automatically as photos change": "{n}枚が一致{:.0}・写真の変更に応じて自動更新",
"{n} photo{} now · updates automatically as photos change": "現在{n}枚{:.0}・写真の変更に応じて自動更新",
"{sel_n} selected · {counted}": "{sel_n}枚選択・{counted}",
"{stage}… {:.0}%": "{stage}… {:.0}%",
"{what} merge added": "{what}合成を追加しました",
"{what} merge failed: {e}": "{what}合成に失敗:{e}",
"{} and higher": "{}以上",
"{} files": "{}個のファイル",
"{} files and {f} folder{}": "{}個のファイルと{f}個のフォルダ{:.0}",
"{} of 4 guides — drag along lines that should be vertical or horizontal.": "ガイド4本中{}本:垂直または水平にする線に沿ってドラッグ",
"{} photos": "{}枚",
"{} photo{} not readable": "{}枚を読み込めません{:.0}",
"{} spot(s) on this photo": "この写真に{}か所のスポット",
"· {dups} already in the library": "・{dups}枚は読み込み済み",
"· {n} photo{}": "・{n}枚{:.0}",
"… and {more} more": "…ほか{more}件",
"Import cancelled · {} photo{} added": "読み込みを中止しました・{}枚を追加{:.0}",
"Reading photos… {} of {}": "写真を確認中:{1}枚中{0}枚",
"Adding photos… {} of {}": "写真を追加中:{1}枚中{0}枚"
}
File diff suppressed because it is too large. Load diff
+5
View File
@@ -82,9 +82,14 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {
"loupe": app.loupe_shown.map(|(p, src)| json!({"photo": p.0, "source": src, "pending": app.renderer.is_pending(crate::render::Slot::Main)})),
"hoverPreview": app.hover_preview.as_ref().map(|h| h.label.clone()),
"status": app.ui.status,
"notices": app.notices,
"quitPrompt": app.quit_prompt,
"unsaved": app.session.unsaved().map(|(n, e)| json!({"ops": n, "error": e})),
"libraryProblem": app.library_problem.as_ref().map(crate::panels::library_problem::LibraryProblem::to_json),
"scan": app.scan.as_ref().map(crate::import::ScanTask::status),
"export": {"running": app.export.as_ref().map(crate::export_task::ExportTask::status), "last": app.last_export_result},
"import": app.import.as_ref().map(crate::import::ImportTask::status),
"tasks": app.tasks.labels(),
"memory": memory(app),
})
}
+9 -5
View File
@@ -75,12 +75,13 @@ pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
let ok = files.iter().filter(|f| f.get("path").is_some()).count();
let failed: Vec<&Value> = files.iter().filter(|f| f.get("error").is_some()).collect();
let skipped = files.iter().filter(|f| f.get("skipped").is_some()).count();
let mut m = format!("Exported {ok} of {total} photo{}", if total == 1 { "" } else { "s" });
let mut m =
crate::i18n::tr_format!("Exported {ok} of {total} photo{}", if total == 1 { "" } else { "s" }, ok = ok, total = total);
if skipped > 0 {
m += &format!(" · {skipped} skipped (file exists)");
m += &crate::i18n::tr_format!(" · {skipped} skipped (file exists)", skipped = skipped);
}
if let Some(first) = failed.first() {
m += &format!(" · {} failed: {}", failed.len(), first["error"].as_str().unwrap_or("error"));
m += &crate::i18n::tr_format!(" · {} failed: {}", failed.len(), first["error"].as_str().unwrap_or("error"));
}
if cancelled {
m += " · cancelled";
@@ -96,14 +97,17 @@ pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
let (done, current) = task.progress.lock().map(|g| g.clone()).unwrap_or_default();
let (total, cancel) = (task.total, task.cancel.clone());
let t = Tokens::get(ctx);
egui::Window::new("Exporting")
egui::Window::new(crate::i18n::tr("Exporting"))
.title_bar(false)
.resizable(false)
.anchor(Align2::LEFT_BOTTOM, [16.0, -56.0])
.fixed_size([300.0, 64.0])
.show(ctx, |ui| {
ui.horizontal(|ui| {
ui.label(egui::RichText::new(format!("Exporting {} of {total}", (done + 1).min(total))).color(t.text));
ui.label(
egui::RichText::new(crate::i18n::tr_format!("Exporting {} of {total}", (done + 1).min(total), total = total))
.color(t.text),
);
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let stopping = cancel.load(Ordering::Relaxed);
if crate::widgets::text_button(ui, "exportCancel", if stopping { "Stopping…" } else { "Cancel" }, false).clicked() {
+10
View File
@@ -176,6 +176,7 @@ impl Headless {
|| self.app.scan.is_some()
|| self.app.import.is_some()
|| self.app.export.is_some()
|| !self.app.tasks.is_empty()
|| !self.app.synthetic.is_empty()
|| !self.events.is_empty()
}
@@ -412,6 +413,15 @@ mod tests {
assert_eq!(after.develop, before.develop);
assert_eq!(after.history.len(), before.history.len(), "no history entry while hovering");
assert!(h.app.session.undo.is_empty());
// hovering down across group headers within the presets list retains the preview (no flicker)
h.request("ui.hoverWidget", json!({"id": "presetGroup:Creative"}), t);
h.step();
assert_eq!(h.app.hover_preview.as_ref().map(|p| p.label.as_str()), Some("Preset: High Contrast B&W"));
// hovering another preset switches smoothly to it
h.request("ui.hoverWidget", json!({"id": "preset:lc.warm-glow"}), t);
h.settle(SETTLE);
h.step();
assert_eq!(h.app.hover_preview.as_ref().map(|p| p.label.as_str()), Some("Preset: Warm Glow"));
// thumbnails: variant textures for the visible presets
h.app.ui.preset_thumbs = true;
h.request("ui.move", json!({"x": 5, "y": 500}), t);
+125 -117
View File
@@ -1,24 +1,27 @@
//! Interface translations. Command ids, document text and file names remain stable.
//! Untranslated labels fall back to English so coverage can grow incrementally.
//! Presentation-only localization. Command ids, user names and catalog data remain unchanged.
use std::{cell::Cell, collections::BTreeMap, sync::OnceLock};
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "lowercase")]
use serde::{Deserialize, Serialize};
include!(concat!(env!("OUT_DIR"), "/ja-formats.rs"));
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub enum Language {
#[default]
#[serde(rename = "en")]
En,
#[serde(rename = "ja")]
Ja,
}
impl Language {
pub const ALL: [Self; 2] = [Self::En, Self::Ja];
pub fn name(self) -> &'static str {
match self {
Self::En => "English",
Self::Ja => "日本語",
}
}
pub fn parse(code: &str) -> Option<Self> {
match code {
"en" => Some(Self::En),
@@ -26,129 +29,134 @@ impl Language {
_ => None,
}
}
pub fn tr(self, text: &str) -> &str {
if self == Self::Ja
&& let Some((_, japanese)) = JAPANESE.iter().find(|(english, _)| *english == text)
{
return japanese;
}
text
pub fn tr(self, source: &str) -> &str {
if self == Self::Ja { japanese().get(source).map(String::as_str).unwrap_or(source) } else { source }
}
}
const JAPANESE: &[(&str, &str)] = &[
("Photo", "写真"),
("Import Photos…", "写真を読み込み…"),
("Detail", "詳細"),
("Photo Grid", "写真グリッド"),
("Square Grid", "正方形グリッド"),
("Object", "オブジェクト"),
("Effect", "効果"),
("Settings…", "環境設定…"),
("Image", "画像"),
("Layer", "レイヤー"),
("Type", "書式"),
("Select", "選択"),
("Filter", "フィルター"),
("Window", "ウィンドウ"),
("Language", "表示言語"),
("Save As…", "別名で保存…"),
("Exit", "終了"),
("New…", "新規…"),
("New", "新規"),
("Horizontal", "横書き"),
("Vertical", "縦書き"),
("Orientation", "組み方向"),
("Layers", "レイヤー"),
("History", "履歴"),
("Properties", "プロパティ"),
("Color", "カラー"),
("Brush Settings", "ブラシ設定"),
("Tools", "ツール"),
("Options", "オプション"),
("Zoom In", "ズームイン"),
("Zoom Out", "ズームアウト"),
("Fit on Screen", "画面に合わせる"),
("Copy", "コピー"),
("Cut", "切り取り"),
("Paste", "貼り付け"),
("Select All", "すべて選択"),
("Deselect", "選択を解除"),
("Export", "書き出し"),
("Export As…", "形式を指定して書き出し…"),
("Search…", "検索…"),
("Theme", "テーマ"),
("Menu", "メニュー"),
("File", "ファイル"),
("Edit", "編集"),
("Pages", "ページ"),
("View", "表示"),
("Help", "ヘルプ"),
("Preferences", "環境設定"),
("Preferences…", "環境設定…"),
("Interface language", "表示言語"),
("Open…", "開く…"),
("New blank PDF", "空白の PDF を作成"),
("Create PDF from file…", "ファイルから PDF を作成…"),
("Create PDF from images…", "画像から PDF を作成…"),
("Create PDF from clipboard", "クリップボードから PDF を作成"),
("Combine files…", "ファイルを結合…"),
("Save", "保存"),
("Save as…", "別名で保存…"),
("Close file", "ファイルを閉じる"),
("Close all", "すべて閉じる"),
("Revert", "保存済みの状態に戻す"),
("Print…", "印刷…"),
("Document properties…", "文書のプロパティ…"),
("Undo", "取り消し"),
("Redo", "やり直し"),
("Find…", "検索…"),
("Advanced search…", "高度な検索…"),
("Copy pages", "ページをコピー"),
("Cut pages", "ページを切り取り"),
("Paste pages", "ページを貼り付け"),
("Fit visible", "表示範囲に合わせる"),
("Marquee zoom", "範囲指定ズーム"),
("Take a snapshot", "スナップショットを作成"),
("Full screen mode", "全画面表示"),
("Read mode", "閲覧モード"),
("Switch light / dark theme", "明るい/暗いテーマを切り替え"),
("Comments panel", "コメントパネル"),
("Form fields panel", "フォームフィールドパネル"),
("Clear form", "フォームをクリア"),
("Find tools and commands…", "ツールとコマンドを検索…"),
("Zoom", "ズーム"),
("Actual size", "実際のサイズ"),
("Zoom to page level", "ページ全体を表示"),
("Fit to width", "幅に合わせる"),
("Display theme", "表示テーマ"),
("Side panels", "サイドパネル"),
("OK", "OK"),
];
thread_local! {
static LANGUAGE: Cell<Language> = const { Cell::new(Language::En) };
}
pub fn default_language() -> Language {
if std::env::var("LIGHTCRAFT_LANGUAGE").as_deref() == Ok("ja") { Language::Ja } else { Language::En }
}
pub fn set_language(language: Language) {
LANGUAGE.with(|value| value.set(language));
}
pub fn is_japanese() -> bool {
LANGUAGE.with(|value| value.get() == Language::Ja)
}
fn japanese() -> &'static BTreeMap<String, String> {
static MESSAGES: OnceLock<BTreeMap<String, String>> = OnceLock::new();
MESSAGES.get_or_init(|| {
serde_json::from_str(include_str!("../locales/ja.json")).unwrap_or_else(|error| {
log::error!("Invalid Japanese message catalog: {error}");
BTreeMap::new()
})
})
}
/// Translate a built-in display label, preserving unknown labels verbatim.
/// Never call this on editable user text, filenames or command identifiers.
pub fn tr(source: &str) -> &str {
if is_japanese() { japanese().get(source).map(String::as_str).unwrap_or(source) } else { source }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn language_setting_persists() {
let state = crate::state::UiState { language: Language::Ja, ..Default::default() };
let saved = serde_json::to_string(&state).unwrap();
let restored: crate::state::UiState = serde_json::from_str(&saved).unwrap();
assert_eq!(restored.language, Language::Ja);
assert_eq!(crate::state::UiState::default().language, Language::En);
fn catalog_is_valid_and_contains_core_workflows() {
let messages: BTreeMap<String, String> = serde_json::from_str(include_str!("../locales/ja.json")).unwrap();
for key in ["Import Photos…", "Export…", "Exposure", "White Balance", "Settings", "Language"] {
assert!(messages.get(key).is_some_and(|value| !value.is_empty() && value != key), "{key}");
}
}
#[test]
fn translations_are_unique_and_preserve_unknown_text() {
for (i, (en, ja)) in JAPANESE.iter().enumerate() {
assert!(!ja.is_empty());
assert!(JAPANESE.iter().take(i).all(|(other, _)| en != other));
assert_eq!(Language::En.tr(en), *en);
fn language_switches_and_unknown_text_survives() {
set_language(Language::Ja);
assert_eq!(tr("Exposure"), "露出");
assert_eq!(tr("my-photo.jpg"), "my-photo.jpg");
assert_eq!(tr("develop.set"), "develop.set");
assert_eq!(crate::menubar::display_item_label("album.addPhotos", &serde_json::json!({"id": 1}), "Color"), "Color");
assert_eq!(crate::menubar::display_item_label("app.export", &serde_json::json!({"preset": "Color"}), "Color"), "Color");
assert_eq!(crate::menubar::display_item_label("view.photoGrid", &serde_json::Value::Null, "Color"), "カラー");
set_language(Language::En);
assert_eq!(tr("Exposure"), "Exposure");
}
#[test]
fn translated_formats_preserve_counts_and_remove_english_plural_suffixes() {
set_language(Language::Ja);
assert_eq!(tr_format!("{n} photo{}", "s", n = 12), "12枚");
assert_eq!(tr_format!("Exported {ok} of {total} photo{}", "s", ok = 4, total = 12), "12枚中4枚を書き出しました");
set_language(Language::En);
assert_eq!(tr_format!("{n} photo{}", "s", n = 12), "12 photos");
}
#[test]
fn preferences_round_trip_and_old_settings_remain_readable() {
let old: crate::state::UiState = serde_json::from_str("{}").unwrap();
assert_eq!(old.language, Language::En);
let settings = crate::state::UiState { language: Language::Ja, ..old };
let saved = serde_json::to_string(&settings).unwrap();
let restored: crate::state::UiState = serde_json::from_str(&saved).unwrap();
assert_eq!(restored.language, Language::Ja);
}
#[test]
fn japanese_is_painted_and_both_font_weights_cover_the_catalog() {
let ctx = egui::Context::default();
crate::theme::install_fonts(&ctx);
let mut app = crate::LightcraftApp::new(lightcraft_engine::Session::with_demo(), Default::default());
app.ui.language = Language::Ja;
app.ui.left_panel = true;
let mut text = String::new();
fn collect(shape: &egui::epaint::Shape, text: &mut String) {
match shape {
egui::epaint::Shape::Text(shape) => {
text.push_str(&shape.galley.job.text);
text.push('\n');
}
egui::epaint::Shape::Vec(shapes) => shapes.iter().for_each(|shape| collect(shape, text)),
_ => {}
}
}
assert_eq!(Language::Ja.tr("File"), "ファイル");
assert_eq!(Language::Ja.tr("日本語の文書.pdf"), "日本語の文書.pdf");
assert_eq!(Language::parse("xx"), None);
for frame in 0..4 {
let input = crate::headless::HeadlessView::raw_input(egui::vec2(1600.0, 1000.0), 1.0, frame as f64 / 60.0, vec![]);
let mut out = ctx.run_ui(input, |ui| {
app.logic(ui.ctx());
app.ui(ui);
});
// This assertion inspects shapes without a renderer; discard texture uploads explicitly.
out.textures_delta.clear();
text.clear();
for shape in out.shapes {
collect(&shape.shape, &mut text);
}
}
assert!(text.contains("マイフォト"), "{text}");
assert!(text.contains("すべての写真"), "{text}");
ctx.fonts_mut(|fonts| {
for family in [egui::FontFamily::Proportional, egui::FontFamily::Name(crate::theme::FONT_SEMIBOLD.into())] {
let font = egui::FontId::new(13.0, family);
for message in japanese().values() {
for ch in message.chars().filter(|ch| !ch.is_whitespace()) {
assert!(fonts.has_glyph(&font, ch), "Missing glyph {ch} in {message}");
}
}
}
});
// Locale affects presentation only: command ids remain the same.
let ids = |app: &crate::LightcraftApp| crate::menus::menu_entries(app).into_iter().map(|entry| entry.id).collect::<Vec<_>>();
let japanese_ids = ids(&app);
set_language(Language::En);
assert_eq!(japanese_ids, ids(&app));
}
}
+334 -74
View File
@@ -3,8 +3,9 @@
//! a grid of thumbnails with checkboxes (duplicates marked and unchecked), the destination (add in
//! place / copy or move into the library's `Originals/` or a chosen folder, filed by day, by month,
//! into one folder or by a custom folder template, optionally renamed), an album
//! (existing or new), a preset and keywords to apply. Importing runs in small batches, one per
//! frame, with a progress window; the whole import is one undo step.
//! (existing or new), a preset and keywords to apply. Importing runs on a worker thread (files are
//! probed, copied or moved there) and its batches join the catalog between frames, with a progress
//! window and Cancel; the whole import is one undo step.
use egui::{Align2, Color32, Rect, Sense, Stroke, StrokeKind, pos2, vec2};
use lightcraft_engine::import::{ImportCandidate, ScanInput, ScanOutput, ScanProgress, scan_with};
@@ -17,7 +18,7 @@ use crate::render::Slot;
use crate::theme::Tokens;
use crate::widgets::register;
/// Files per batch (one batch per frame, so the progress window updates).
/// Files per batch (each batch joins the catalog as it is ready, so the progress window updates).
pub(crate) const BATCH: usize = 8;
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
@@ -91,9 +92,13 @@ impl ImportDialog {
}
}
/// A running import (see the module docs).
/// A running import (see the module docs). The file-system work — expanding folders, probing,
/// reading sidecars, copying or placing files — runs on a worker thread
/// ([`lightcraft_engine::import::ImportJob`]); each batch it readies is added to the catalog on the
/// UI thread between frames, so a slow drive never stalls the window.
#[derive(Debug, Default)]
pub struct ImportTask {
/// Files and folders to import (handed to the worker when the task starts).
queue: Vec<String>,
pub total: usize,
pub done: usize,
@@ -109,6 +114,156 @@ pub struct ImportTask {
/// Reading a folder for the Local view: the photos stay out of the library, and nothing is
/// selected or announced as added.
browse: bool,
/// Cancel was pressed: no further files are started; batches already readied are added.
pub cancelled: bool,
/// Auto Import (the watched folder): the selection stays as it is, a short toast when done.
auto: bool,
/// Auto Import: the selection to keep.
keep_selection: Option<lightcraft_engine::Selection>,
run: Option<ImportRun>,
}
impl ImportTask {
/// An import of `paths` (files or folders) with `library.import` params (without `paths`).
pub fn new(paths: Vec<String>, params: Value, undo0: usize, browse: bool) -> Self {
ImportTask { total: paths.len(), queue: paths, params, undo0, browse, ..Default::default() }
}
/// An Auto Import (see [`ImportTask::auto`]).
pub fn auto(mut self) -> Self {
self.auto = true;
self
}
/// `{done, total, imported, cancelled}` for `ui.inspect`.
pub fn status(&self) -> Value {
json!({"done": self.done, "total": self.total, "imported": self.imported, "cancelled": self.cancelled})
}
}
/// Where an import's batches are readied: a worker thread, or (browser build: no threads, and no
/// slow drives) inline, one batch per frame.
enum Runner {
#[cfg(not(target_arch = "wasm32"))]
Thread(std::sync::mpsc::Receiver<lightcraft_engine::import::Prepared>),
#[cfg(target_arch = "wasm32")]
Inline { job: Box<lightcraft_engine::import::ImportJob>, queue: Vec<String>, files: Option<std::collections::VecDeque<String>> },
}
struct ImportRun {
runner: Runner,
cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
total: std::sync::Arc<std::sync::atomic::AtomicUsize>,
done: std::sync::Arc<std::sync::atomic::AtomicUsize>,
opts: lightcraft_engine::import::ImportOptions,
now: String,
album: Option<u64>,
album_name: Option<String>,
}
impl std::fmt::Debug for ImportRun {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ImportRun").field("cancel", &self.cancel.load(Ordering::Relaxed)).finish()
}
}
impl Drop for ImportRun {
/// A dropped import (replaced, or the app closing) stops its worker at the next file.
fn drop(&mut self) {
self.cancel.store(true, Ordering::Relaxed);
}
}
impl ImportRun {
fn start(
job: lightcraft_engine::import::ImportJob,
queue: Vec<String>,
album: Option<u64>,
album_name: Option<String>,
ctx: &egui::Context,
) -> Result<Self, String> {
let cancel = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let (total, done) = (job.total.clone(), job.done.clone());
let (opts, now) = (job.opts.clone(), job.now().to_string());
let runner = Self::spawn(job, queue, cancel.clone(), ctx)?;
Ok(ImportRun { runner, cancel, total, done, opts, now, album, album_name })
}
#[cfg(not(target_arch = "wasm32"))]
fn spawn(
mut job: lightcraft_engine::import::ImportJob,
queue: Vec<String>,
cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
ctx: &egui::Context,
) -> Result<Runner, String> {
let (tx, rx) = std::sync::mpsc::channel();
let ctx = ctx.clone();
let work = move || {
let files = job.expand(&queue);
for chunk in files.chunks(BATCH) {
if cancel.load(Ordering::Relaxed) {
break;
}
let prepared = match lightcraft_engine::guard::catch("import", || job.prepare_files(chunk.to_vec(), &cancel)) {
Ok(p) => p,
Err(_) => break, // logged; the files readied so far are added
};
if let Err(unsent) = tx.send(prepared) {
// the import was dropped: nothing will add these, so take back what a Move placed
unsent.0.rollback();
break;
}
ctx.request_repaint();
}
ctx.request_repaint();
};
std::thread::Builder::new().name("lc-import".into()).spawn(work).map(|_| Runner::Thread(rx)).map_err(|e| format!("could not start: {e}"))
}
#[cfg(target_arch = "wasm32")]
fn spawn(
job: lightcraft_engine::import::ImportJob,
queue: Vec<String>,
_cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
_ctx: &egui::Context,
) -> Result<Runner, String> {
Ok(Runner::Inline { job: Box::new(job), queue, files: None })
}
/// Readied batches (at most a few per frame). `None` once the worker is done.
fn next_batches(&mut self) -> Option<Vec<lightcraft_engine::import::Prepared>> {
match &mut self.runner {
#[cfg(not(target_arch = "wasm32"))]
Runner::Thread(rx) => {
let mut out = Vec::new();
while out.len() < 4 {
match rx.try_recv() {
Ok(p) => out.push(p),
Err(std::sync::mpsc::TryRecvError::Empty) => return Some(out),
Err(std::sync::mpsc::TryRecvError::Disconnected) => return if out.is_empty() { None } else { Some(out) },
}
}
Some(out)
}
#[cfg(target_arch = "wasm32")]
Runner::Inline { job, queue, files } => {
if self.cancel.load(Ordering::Relaxed) {
return None;
}
if files.is_none() {
*files = Some(job.expand(queue).into());
return Some(Vec::new());
}
let Some(files) = files.as_mut() else { return None };
if files.is_empty() {
return None;
}
let n = files.len().min(BATCH);
let batch: Vec<String> = files.drain(..n).collect();
Some(vec![job.prepare_files(batch, &self.cancel)])
}
}
}
}
/// A folder scan running on a worker thread (a network share can take minutes to read; the
@@ -232,7 +387,8 @@ pub fn poll_scan(app: &mut LightcraftApp, ctx: &egui::Context) {
if !queue.is_empty() {
let undo0 = app.session.undo.len();
let total = queue.len();
app.import = Some(ImportTask { queue, total, params: json!({"mode": "add", "local": true}), undo0, browse: true, ..Default::default() });
let _ = total;
app.import = Some(ImportTask::new(queue, json!({"mode": "add", "local": true}), undo0, true));
}
return;
}
@@ -256,19 +412,21 @@ pub fn scan_progress(app: &mut LightcraftApp, ctx: &egui::Context) {
let text = if total == 0 {
if task.browse { "Reading folder…" } else { "Looking for photos…" }.to_string()
} else {
format!("Reading photos… {done} of {total}")
crate::i18n::tr_format!("Reading photos… {done} of {total}", done = done, total = total)
};
let mut cancel = false;
egui::Window::new("Scanning").title_bar(false).resizable(false).anchor(Align2::CENTER_BOTTOM, [0.0, -80.0]).fixed_size([340.0, 80.0]).show(
ctx,
|ui| {
egui::Window::new(crate::i18n::tr("Scanning"))
.title_bar(false)
.resizable(false)
.anchor(Align2::CENTER_BOTTOM, [0.0, -80.0])
.fixed_size([340.0, 80.0])
.show(ctx, |ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(done as f32 / total.max(1) as f32).desired_width(320.0));
let r = ui.button("Cancel");
let r = ui.button(crate::i18n::tr("Cancel"));
register(ui.ctx(), "button:scanCancel", r.rect);
cancel = r.clicked();
},
);
});
if cancel {
// the worker stops at its next file; don't wait for it (a NAS read can take a while)
task.progress.cancel.store(true, Ordering::Relaxed);
@@ -326,28 +484,82 @@ pub fn start(app: &mut LightcraftApp, d: &ImportDialog) -> Result<Value, String>
}
}
let total = queue.len();
app.import = Some(ImportTask { queue, total, params, undo0, ..Default::default() });
app.import = Some(ImportTask::new(queue, params, undo0, false));
app.renderer.forget_imports();
Ok(json!({"importing": total}))
}
/// Run one batch of the import in progress (called every frame).
pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = app.import.as_mut() else { return };
// (browsing reads a sidecar per file, slow on a network share: keep each frame short)
let n = task.queue.len().min(if task.browse { 2 } else { BATCH });
let batch: Vec<String> = task.queue.drain(..n).collect();
let mut p = task.params.clone();
p["paths"] = json!(batch);
// renamed copies keep counting across batches
if p.get("rename").is_some() {
p["renameStart"] = json!(1 + task.imported);
/// Start importing `paths` (files or folders) in the background, e.g. dropped on the window.
pub fn start_paths(app: &mut LightcraftApp, paths: Vec<String>) -> Result<Value, String> {
if app.import.is_some() || app.scan.as_ref().is_some_and(|t| !t.browse) {
return Err("an import is running".into());
}
let r = app.session.execute("library.import", &p);
let Some(task) = app.import.as_mut() else { return };
task.done += n;
let undo0 = app.session.undo.len();
let total = paths.len();
app.import = Some(ImportTask::new(paths, json!({"mode": "add"}), undo0, false));
Ok(json!({"importing": total}))
}
/// Advance the import in progress (called every frame): start its worker, then add the batches it
/// has readied to the catalog.
pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(mut task) = app.import.take() else { return };
if task.run.is_none() {
let mut p = task.params.clone();
p["paths"] = json!(task.queue);
let started = lightcraft_engine::cmd::library::import_params(&app.session, &p)
.and_then(|req| Ok((lightcraft_engine::import::ImportJob::new(&mut app.session, req.opts)?, req.album, req.album_name)))
.map_err(|e| e.to_string())
.and_then(|(job, album, name)| ImportRun::start(job, std::mem::take(&mut task.queue), album, name, ctx));
if task.auto {
task.keep_selection = Some(app.session.selection.clone());
}
match started {
Ok(run) => task.run = Some(run),
Err(e) => {
log::warn!("import: {e}");
app.toast(ctx, format!("Import failed: {e}"));
return;
}
}
}
ctx.request_repaint_after(std::time::Duration::from_millis(100));
let Some(run) = task.run.as_mut() else { return };
let batches = run.next_batches();
let finished = batches.is_none();
for prepared in batches.into_iter().flatten() {
commit_batch(app, &mut task, prepared);
// arriving photos don't take over the selection
if let Some(sel) = &task.keep_selection {
app.session.selection = sel.clone();
}
}
if let Some(run) = &task.run {
task.total = run.total.load(Ordering::Relaxed).max(task.total);
task.done = run.done.load(Ordering::Relaxed);
}
if !finished {
app.import = Some(task);
return;
}
finish(app, ctx, task);
}
/// Add one readied batch to the catalog (an undo step merged into the import's at the end).
fn commit_batch(app: &mut LightcraftApp, task: &mut ImportTask, prepared: lightcraft_engine::import::Prepared) {
let Some(run) = task.run.as_ref() else { return };
let n = prepared.len();
let (opts, now, album, album_name) = (&run.opts, run.now.as_str(), run.album, run.album_name.clone());
let r = app.session.execute_fn("library.import", |s| {
let report = lightcraft_engine::import::commit_prepared(s, opts, now, prepared)?;
lightcraft_engine::cmd::library::import_batch_done(s, report, album, album_name.as_deref())
});
match r {
Ok(v) => {
// a new album (named in the params) is created with the first photos; later batches join it
if let (Some(a), Some(run)) = (v["album"].as_u64(), task.run.as_mut()) {
run.album = Some(a);
}
let len = |k: &str| v[k].as_array().map_or(0, Vec::len);
task.imported += len("imported");
task.duplicates += len("duplicates");
@@ -366,17 +578,22 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
task.failed += n;
}
}
ctx.request_repaint();
if !task.queue.is_empty() {
}
/// The import is done (or cancelled): one undo step, select the first photo, say what happened.
fn finish(app: &mut LightcraftApp, ctx: &egui::Context, task: ImportTask) {
let steps = app.session.undo.len().saturating_sub(task.undo0);
let label = crate::i18n::tr_format!("Add {} Photo{}", task.imported, if task.imported == 1 { "" } else { "s" });
app.session.merge_undo(steps, &label);
if task.auto {
if task.imported > 0 {
app.toast(ctx, format!("Auto Import: added {} photo{}", task.imported, if task.imported == 1 { "" } else { "s" }));
}
return;
}
let Some(task) = app.import.take() else { return };
let steps = app.session.undo.len().saturating_sub(task.undo0);
let label = format!("Add {} Photo{}", task.imported, if task.imported == 1 { "" } else { "s" });
app.session.merge_undo(steps, &label);
if task.browse {
if task.failed > 0 {
app.toast(ctx, format!("{} photo{} not readable", task.failed, if task.failed == 1 { "" } else { "s" }));
app.toast(ctx, crate::i18n::tr_format!("{} photo{} not readable", task.failed, if task.failed == 1 { "" } else { "s" }));
}
return;
}
@@ -384,35 +601,58 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let _ = app.run("library.select", json!({"ids": [f]}));
}
let plural = |n: usize| if n == 1 { "" } else { "s" };
let mut msg = format!("Imported {} photo{}", task.imported, plural(task.imported));
let mut msg = if task.cancelled {
crate::i18n::tr_format!("Import cancelled · {} photo{} added", task.imported, plural(task.imported))
} else {
crate::i18n::tr_format!("Imported {} photo{}", task.imported, plural(task.imported))
};
if task.params["mode"] == "move" {
msg.push_str(&format!(" · {} moved", task.moved));
msg.push_str(&crate::i18n::tr_format!(" · {} moved", task.moved));
if task.kept > 0 {
msg.push_str(&format!(" · {} original{} left at the source", task.kept, plural(task.kept)));
}
}
if task.duplicates > 0 {
msg.push_str(&format!(" · {} duplicate{} skipped", task.duplicates, if task.duplicates == 1 { "" } else { "s" }));
msg.push_str(&crate::i18n::tr_format!(" · {} duplicate{} skipped", task.duplicates, if task.duplicates == 1 { "" } else { "s" }));
}
if task.failed > 0 {
msg.push_str(&format!(" · {} not readable", task.failed));
msg.push_str(&crate::i18n::tr_format!(" · {} not readable", task.failed));
}
app.toast(ctx, msg);
}
/// The progress window while an import runs.
/// The progress window while an import runs, with Cancel (files already copied or added stay;
/// nothing new is started).
pub fn progress(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = &app.import else { return };
let t = Tokens::get(ctx);
let frac = task.done as f32 / task.total.max(1) as f32;
let text = format!("{} photos… {} of {}", if task.browse { "Reading" } else { "Adding" }, task.done, task.total);
egui::Window::new("Importing").title_bar(false).resizable(false).anchor(Align2::CENTER_BOTTOM, [0.0, -80.0]).fixed_size([340.0, 60.0]).show(
ctx,
|ui| {
let text = if task.cancelled {
crate::i18n::tr("Stopping…").to_string()
} else if task.browse {
crate::i18n::tr_format!("Reading photos… {} of {}", task.done, task.total)
} else {
crate::i18n::tr_format!("Adding photos… {} of {}", task.done, task.total)
};
let mut cancel = false;
egui::Window::new(crate::i18n::tr("Importing"))
.title_bar(false)
.resizable(false)
.anchor(Align2::CENTER_BOTTOM, [0.0, -80.0])
.fixed_size([340.0, 80.0])
.show(ctx, |ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(frac).desired_width(320.0));
},
);
let r = ui.add_enabled(!task.cancelled, egui::Button::new(crate::i18n::tr("Cancel")));
register(ui.ctx(), "button:importCancel", r.rect);
cancel = r.clicked();
});
if cancel && let Some(task) = app.import.as_mut() {
task.cancelled = true;
if let Some(run) = &task.run {
run.cancel.store(true, Ordering::Relaxed);
}
}
}
/// The dialog body: options, then the candidate grid.
@@ -422,15 +662,15 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
let dups = d.candidates.iter().filter(|c| c.duplicate.is_some()).count();
let sel = d.selected_paths().len();
ui.horizontal(|ui| {
ui.label(egui::RichText::new(format!("{n} found · {sel} selected")).color(t.text));
ui.label(egui::RichText::new(crate::i18n::tr_format!("{n} found · {sel} selected", n = n, sel = sel)).color(t.text));
if dups > 0 {
ui.label(egui::RichText::new(format!("· {dups} already in the library")).color(t.text_dim));
ui.label(egui::RichText::new(crate::i18n::tr_format!("· {dups} already in the library", dups = dups)).color(t.text_dim));
}
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
if crate::widgets::text_button(ui, "importNone", "Uncheck All", false).clicked() {
if crate::widgets::text_button(ui, "importNone", crate::i18n::tr("Uncheck All"), false).clicked() {
d.checked.iter_mut().for_each(|c| *c = false);
}
if crate::widgets::text_button(ui, "importAll", "Check All", false).clicked() {
if crate::widgets::text_button(ui, "importAll", crate::i18n::tr("Check All"), false).clicked() {
for i in 0..n {
d.checked[i] = d.importable(i);
}
@@ -458,29 +698,42 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
// options
if !d.sources.is_empty() {
field(ui, "Source", |ui| {
let r = ui.label(egui::RichText::new(source_summary(&d.sources)).color(t.text_label)).on_hover_text(d.sources.join("\n"));
// (whether each source is a folder is checked off the UI thread, once)
let joined = d.sources.join("\n");
let summary = crate::panels::left::fs_cached(ui, "import-source", &joined, f64::INFINITY, |s| {
source_summary(&s.split('\n').map(str::to_string).collect::<Vec<_>>())
})
.unwrap_or_else(|| format!("{} item{}", d.sources.len(), if d.sources.len() == 1 { "" } else { "s" }));
let r = ui.label(egui::RichText::new(summary).color(t.text_label)).on_hover_text(joined);
register(ui.ctx(), "label:importSource", r.rect);
});
ui.label(
egui::RichText::new("Importing scans the source for photos; it doesn't add the folder to Local (use Local → Browse Folder… to work in a folder without importing).")
egui::RichText::new(crate::i18n::tr("Importing scans the source for photos; it doesn't add the folder to Local (use Local → Browse Folder… to work in a folder without importing)."))
.color(t.text_dim)
.small(),
);
}
field(ui, "Transfer", |ui| {
ui.spacing_mut().item_spacing.x = 4.0;
if crate::widgets::text_button(ui, "importAdd", "Add in place", !d.copy).on_hover_text("Reference the files where they are").clicked() {
if crate::widgets::text_button(ui, "importAdd", crate::i18n::tr("Add in place"), !d.copy)
.on_hover_text(crate::i18n::tr("Reference the files where they are"))
.clicked()
{
d.copy = false;
d.move_files = false;
}
let can_copy = app.session.library.as_ref().is_some_and(|l| l.on_disk) || app.services.pick_folder.is_some();
let r = ui.add_enabled_ui(can_copy, |ui| crate::widgets::text_button(ui, "importCopy", "Copy", d.copy && !d.move_files)).inner;
if r.on_hover_text("Copy the files (into the library's Originals/, or a folder you choose)").clicked() {
let r =
ui.add_enabled_ui(can_copy, |ui| crate::widgets::text_button(ui, "importCopy", crate::i18n::tr("Copy"), d.copy && !d.move_files)).inner;
if r.on_hover_text(crate::i18n::tr("Copy the files (into the library's Originals/, or a folder you choose)")).clicked() {
d.copy = true;
d.move_files = false;
}
let r = ui.add_enabled_ui(can_copy, |ui| crate::widgets::text_button(ui, "importMove", "Move", d.copy && d.move_files)).inner;
if r.on_hover_text("Move the files (into the library's Originals/, or a folder you choose), removing them from the source").clicked() {
let r =
ui.add_enabled_ui(can_copy, |ui| crate::widgets::text_button(ui, "importMove", crate::i18n::tr("Move"), d.copy && d.move_files)).inner;
if r.on_hover_text(crate::i18n::tr("Move the files (into the library's Originals/, or a folder you choose), removing them from the source"))
.clicked()
{
d.copy = true;
d.move_files = true;
}
@@ -505,12 +758,12 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
let shown = if d.destination.trim().is_empty() { "Library Originals".to_string() } else { d.destination.clone() };
ui.label(egui::RichText::new(shown).color(Tokens::get(ui.ctx()).text_label));
if app.services.pick_folder.is_some()
&& crate::widgets::text_button(ui, "importDest", "Choose…", false).clicked()
&& crate::widgets::text_button(ui, "importDest", crate::i18n::tr("Choose…"), false).clicked()
&& let Some(f) = app.services.pick_folder.as_mut().and_then(|f| f())
{
d.destination = f;
}
if !d.destination.is_empty() && crate::widgets::text_button(ui, "importDestReset", "Library", false).clicked() {
if !d.destination.is_empty() && crate::widgets::text_button(ui, "importDestReset", crate::i18n::tr("Library"), false).clicked() {
d.destination.clear();
}
});
@@ -557,14 +810,16 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
None => unknown_tags_warning(ui, &d.folder_template),
}
ui.label(
egui::RichText::new("Each / starts a folder level; tags are filled in per photo (a level with missing metadata is \"unknown\").")
.color(t.text_dim)
.small(),
egui::RichText::new(crate::i18n::tr(
"Each / starts a folder level; tags are filled in per photo (a level with missing metadata is \"unknown\").",
))
.color(t.text_dim)
.small(),
);
}
if !d.move_files {
field(ui, "Raw files", |ui| {
let r = ui.checkbox(&mut d.dng, "Copy as DNG");
let r = ui.checkbox(&mut d.dng, crate::i18n::tr("Copy as DNG"));
register(ui.ctx(), "check:importDng", r.rect);
});
}
@@ -582,10 +837,12 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
unknown_tags_warning(ui, &d.rename);
if let Some(example) = example_destination(app, d) {
let t = Tokens::get(ui.ctx());
let r = ui.label(egui::RichText::new(format!("Example: {example}")).color(t.text_dim));
let r = ui.label(egui::RichText::new(crate::i18n::tr_format!("Example: {example}", example = example)).color(t.text_dim));
register(ui.ctx(), "label:importExample", r.rect);
ui.label(
egui::RichText::new("Folders and {date} use the capture time; a photo without one uses today's date.").color(t.text_dim).small(),
egui::RichText::new(crate::i18n::tr("Folders and {date} use the capture time; a photo without one uses today's date."))
.color(t.text_dim)
.small(),
);
}
}
@@ -602,11 +859,11 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
None => "None".into(),
};
egui::ComboBox::from_id_salt("import-album").selected_text(cur).show_ui(ui, |ui| {
if ui.selectable_label(d.album.is_none() && d.new_album.is_empty(), "None").clicked() {
if ui.selectable_label(d.album.is_none() && d.new_album.is_empty(), crate::i18n::tr("None")).clicked() {
d.album = None;
d.new_album.clear();
}
if ui.selectable_label(d.album.is_none() && !d.new_album.is_empty(), "New album…").clicked() {
if ui.selectable_label(d.album.is_none() && !d.new_album.is_empty(), crate::i18n::tr("New album…")).clicked() {
d.album = None;
if d.new_album.is_empty() {
d.new_album = "Imported Photos".into();
@@ -627,7 +884,7 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
field(ui, "Preset", |ui| {
let cur = app.session.presets.iter().find(|p| p.id == d.preset).map(|p| p.name.clone()).unwrap_or_else(|| "None".into());
egui::ComboBox::from_id_salt("import-preset").selected_text(cur).height(300.0).show_ui(ui, |ui| {
if ui.selectable_label(d.preset.is_empty(), "None").clicked() {
if ui.selectable_label(d.preset.is_empty(), crate::i18n::tr("None")).clicked() {
d.preset.clear();
}
for p in &app.session.presets {
@@ -641,7 +898,7 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
field(ui, "Metadata", |ui| {
let cur = if d.metadata_preset.is_empty() { "None".to_string() } else { d.metadata_preset.clone() };
egui::ComboBox::from_id_salt("import-metadata").selected_text(cur).show_ui(ui, |ui| {
if ui.selectable_label(d.metadata_preset.is_empty(), "None").clicked() {
if ui.selectable_label(d.metadata_preset.is_empty(), crate::i18n::tr("None")).clicked() {
d.metadata_preset.clear();
}
for m in &app.session.metadata_presets {
@@ -653,7 +910,7 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
});
}
field(ui, "Keywords", |ui| {
let r = ui.add(egui::TextEdit::singleline(&mut d.keywords).hint_text("comma, separated").desired_width(f32::INFINITY));
let r = ui.add(egui::TextEdit::singleline(&mut d.keywords).hint_text(crate::i18n::tr("comma, separated")).desired_width(f32::INFINITY));
register(ui.ctx(), "field:importKeywords", r.rect);
});
}
@@ -726,7 +983,8 @@ fn candidate_cell(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDial
pub(crate) fn tag_toggle(ui: &mut egui::Ui, key: &str) -> bool {
let id = egui::Id::new((key, "tags-open"));
let mut open = ui.ctx().data(|d| d.get_temp::<bool>(id)).unwrap_or(false);
let r = crate::widgets::text_button(ui, &format!("{key}Tags"), "Tags", open).on_hover_text("Show the template tags; click one to insert it");
let r = crate::widgets::text_button(ui, &format!("{key}Tags"), crate::i18n::tr("Tags"), open)
.on_hover_text(crate::i18n::tr("Show the template tags; click one to insert it"));
if r.clicked() {
open = !open;
ui.ctx().data_mut(|d| d.insert_temp(id, open));
@@ -770,7 +1028,9 @@ pub(crate) fn tag_help(ui: &mut egui::Ui, key: &str, text: &mut String, edit_id:
let t = Tokens::get(ui.ctx());
egui::Frame::new().fill(t.inset).corner_radius(4.0).inner_margin(6.0).show(ui, |ui| {
ui.label(
egui::RichText::new("Click a tag to insert it. Examples are for IMG_0042.CR3 taken 14 Jan 2026, 05:58:48.").color(t.text_dim).small(),
egui::RichText::new(crate::i18n::tr("Click a tag to insert it. Examples are for IMG_0042.CR3 taken 14 Jan 2026, 05:58:48."))
.color(t.text_dim)
.small(),
);
egui::ScrollArea::vertical().id_salt((key, "tags")).max_height(170.0).show(ui, |ui| {
egui::Grid::new((key, "tag-grid")).num_columns(3).spacing([10.0, 2.0]).show(ui, |ui| {
@@ -848,9 +1108,9 @@ pub fn source_summary(sources: &[String]) -> String {
many => {
let folders = many.iter().filter(|p| std::path::Path::new(p.as_str()).is_dir()).count();
match folders {
0 => format!("{} files", many.len()),
f if f == many.len() => format!("{f} folders (and their subfolders)"),
f => format!("{} files and {f} folder{}", many.len() - f, if f == 1 { "" } else { "s" }),
0 => crate::i18n::tr_format!("{} files", many.len()),
f if f == many.len() => crate::i18n::tr_format!("{f} folders (and their subfolders)", f = f),
f => crate::i18n::tr_format!("{} files and {f} folder{}", many.len() - f, if f == 1 { "" } else { "s" }, f = f),
}
}
}
Loaded 100 of 146 files, more files were not shown because too many files have changed in this diff. Show more