Compare commits

..
Author SHA1 Message Date
Brandon Thomas 894759e6d8 Merge main into fix/issue-104-ui-fs-v2
# Conflicts:
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 19:00:37 -04:00
Brandon Thomas 1952f7c960 Merge pull request #131 from storytold/fix/issue-107-web-safety
Issue #107: Web build: data-safety fixes for libraries kept in the browser
2026-10-05 18:59:14 -04:00
Brandon Thomas 7bbdbc03ef Merge main into fix/issue-107-web-safety
# Conflicts:
#	apps/lightcraft-web/src/web.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:58:53 -04:00
Brandon Thomas 0f34a65a8c Merge pull request #130 from storytold/fix/issue-97-folder-undo
Issue #97: Rename Folder / Move Folder are undo steps
2026-10-05 18:58:38 -04:00
Brandon Thomas 6c066b66f2 Merge main into fix/issue-97-folder-undo
# Conflicts:
#	crates/engine/src/lib.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:58:13 -04:00
Brandon ThomasandClaude Opus 5.5 7b59081d22 Issue #104: keep file-system calls off the UI thread
Re-applies fix/issue-104-ui-fs on top of the data-safety fixes (integrate/safety):
- availability cache: originals and smart previews are checked on a worker thread; the UI
  answers from the cache (grid, Info panel, photo menu, Missing Photos view)
- import split into ImportJob::prepare (worker) + commit_prepared, with Cancel; Copy keeps
  #96's verified copies
- Find Missing Photos: find_candidates (snapshot, no I/O) + plan_find_missing (all file I/O,
  on a worker, never the session) with #105's matching rules unchanged (name+size+hash;
  renamed by size+hash; single name+size; ambiguous look-alikes skipped and reported; files
  already in use excluded). The plan is applied with library.findMissing {found, missing,
  ambiguous}, re-checked against the catalog then: photos relinked meanwhile are left alone,
  files now in use are not taken. library.findMissing {folder} still runs synchronously.
- Build / Discard Smart Previews with background: true runs on a worker with progress and
  Stop, keeping #106's atomic writes, is_valid checks and `repaired` count (also in
  library.previewProgress); the render worker's smart-preview fallback uses is_valid
- Rename preview planned on a worker (plan_rename_photos) on the same planning core as
  Session::plan_rename_with (MoveFs, case-only respelling by file identity, #95)
- Local folder listing via fs_cached; tests_offline (slow injected probe, frames < 300 ms)

Tests: worker-planned Find Missing equals the synchronous command for a renamed + ambiguous
library; a plan is re-checked when applied; background smart previews repair a damaged proxy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:58:04 -04:00
Brandon Thomas e11f77d9ca Merge pull request #129 from storytold/fix/issue-94-control-close
Issue #94: Control: close the connection on malformed input
2026-10-05 18:57:59 -04:00
Brandon Thomas f383d1e736 Merge main into fix/issue-94-control-close
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:57:50 -04:00
Brandon Thomas edf05789ed Merge pull request #128 from storytold/fix/issue-103-settings-files
Issue #103: Settings files: keep damaged ones, never overwrite unread ones, write ui.json atomically, confirm quit with unsaved changes
2026-10-05 18:57:30 -04:00
Brandon Thomas bccccfb92a Merge main into fix/issue-103-settings-files
# Conflicts:
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 18:56:50 -04:00
Brandon Thomas 4939d9b994 Merge pull request #127 from storytold/fix/issue-100-open-failure
Issue #100: Ask what to do when the library can't be opened, never a silent demo session
2026-10-05 18:56:35 -04:00
Brandon Thomas e0ab41ede0 Merge main into fix/issue-100-open-failure 2026-10-05 18:56:00 -04:00
Brandon Thomas 8690b51fcf Merge pull request #126 from storytold/fix/issue-99-library-lock
Issue #99: Library: exclusive lock so two processes can't open one library
2026-10-05 18:55:44 -04:00
Brandon Thomas e1d3b56bb6 Merge main into fix/issue-99-library-lock
# Conflicts:
#	crates/catalog/src/lib.rs
2026-10-05 18:54:56 -04:00
Brandon Thomas 6100237cdb Merge pull request #125 from storytold/fix/issue-102-format-version
Issue #102: Catalog format version 2; refuse newer libraries untouched
2026-10-05 18:54:41 -04:00
Brandon Thomas 4579599745 Merge main into fix/issue-102-format-version 2026-10-05 18:54:03 -04:00
Brandon Thomas 8506fecf5f Merge pull request #124 from storytold/fix/issue-101-torn-append
Issue #101: catalog.log: cut a partial append back before retrying
2026-10-05 18:53:49 -04:00
Brandon Thomas 0f49ebf8f0 Merge main into fix/issue-101-torn-append
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:52:48 -04:00
Brandon Thomas ab86862011 Merge pull request #123 from storytold/fix/issue-92-xmp-merge
Issue #92: XMP: merge into an existing sidecar instead of replacing it
2026-10-05 18:52:25 -04:00
Brandon Thomas 577330a08b Merge main into fix/issue-92-xmp-merge
# Conflicts:
#	crates/engine/src/sidecar.rs
#	docs/whats-new.md
2026-10-05 18:50:42 -04:00
Brandon Thomas 30697cfc26 Merge pull request #122 from storytold/fix/issue-92-rename-shared-sidecar
Issue #92: Rename Photos copies a stem sidecar another file still shares
2026-10-05 18:50:26 -04:00
Brandon Thomas 50cd8dc6dc Merge branch 'fix/issue-97-folder-undo' into integrate/safety
# Conflicts:
#	crates/engine/src/lib.rs
#	docs/parity.md
#	docs/whats-new.md
2026-10-05 18:38:55 -04:00
Brandon Thomas cc8b3eb3bb Merge branch 'fix/issue-94-control-close' into integrate/safety
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:38:13 -04:00
Brandon Thomas a6b402981b Merge branch 'fix/issue-103-settings-files' into integrate/safety
# Conflicts:
#	crates/engine/src/library.rs
#	docs/whats-new.md
2026-10-05 18:26:38 -04:00
Brandon Thomas 4deb5eff04 Merge branch 'fix/issue-100-open-failure' into integrate/safety 2026-10-05 18:26:38 -04:00
Brandon Thomas 6e2408ff7d Merge branch 'fix/issue-99-library-lock' into integrate/safety
# Conflicts:
#	crates/catalog/src/lib.rs
2026-10-05 18:26:38 -04:00
Brandon Thomas 66be31e3d8 Merge branch 'fix/issue-102-format-version' into integrate/safety 2026-10-05 18:26:19 -04:00
Brandon Thomas 332f37a5c7 Merge branch 'fix/issue-101-torn-append' into integrate/safety
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:26:19 -04:00
Brandon Thomas aa1af6d5e9 Merge branch 'fix/issue-92-xmp-merge' into integrate/safety
# Conflicts:
#	crates/engine/src/sidecar.rs
#	docs/whats-new.md
2026-10-05 18:26:19 -04:00
Brandon Thomas 963912b231 Merge branch 'fix/issue-92-rename-shared-sidecar' into integrate/safety 2026-10-05 18:25:06 -04:00
Brandon Thomas 4174ce8c57 Merge branch 'fix/issue-105-rename-rollback' into integrate/safety
# Conflicts:
#	docs/parity.md
2026-10-05 18:24:59 -04:00
Brandon Thomas f28e1c362e Merge branch 'fix/issue-95-case-only-rename' into integrate/safety 2026-10-05 18:24:34 -04:00
Brandon Thomas 70d800ba68 Merge branch 'fix/issue-98-thumb-cache-own-files' into integrate/safety 2026-10-05 18:24:33 -04:00
Brandon Thomas 33ef91a123 Merge branch 'fix/issue-96-verified-copy-import' into integrate/safety
# Conflicts:
#	docs/whats-new.md
2026-10-05 18:24:33 -04:00
Brandon Thomas f1b29cfe4a Merge branch 'fix/issue-106-dng-smart-previews' into integrate/safety 2026-10-05 18:24:23 -04:00
Brandon Thomas 130b807e1b Merge branch 'fix/issue-93-export-overwrite' into integrate/safety 2026-10-05 18:24:22 -04:00
Brandon ThomasandClaude Opus 5.5 5e618c553a Issue #100: Ask what to do when the library can't be opened, never a silent demo session
When open_library failed, the desktop app (and the web build) quietly ran
an in-memory session seeded with the demo photos: it looked like a reset
library and everything done in it was lost at quit.

- The session now starts empty and in memory (no demo photos) and
  LightcraftApp::library_problem shows a blocking window with the path and
  the error: Try Again, Choose Another Library… (app.openLibrary), Continue
  Without Saving, Quit. Command-line files are imported once the user has
  chosen.
- A temporary session shows a banner under the top bar (with Open
  Library…) for as long as it lasts; it has no library, so nothing is
  written anywhere. Opening a library ends it.
- ui.inspect → libraryProblem; widget ids for automation; documented in
  control-protocol.md.
- Web build: same window (Try Again disabled: reloading the page retries).
- Headless UI test: locked library → window → Continue Without Saving →
  banner, nothing written to the library → lock released → Try Again opens
  it. Checked in the real app over the control channel too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 cbebabccad Issue #102: Catalog format version 2; refuse newer libraries untouched
main added Op::SetBrowsed, Catalog.browsed and Photo.local_baseline while
the snapshot still said version 1, so v0.2.0 read such a library in part
(unknown ops as log damage, unknown fields dropped on its next snapshot).

- journal::VERSION = 2 (now public), with a format table and rules in the
  journal docs; a test with an exhaustive match over Op makes adding a
  variant fail until it is assigned a version.
- Journal::open reads the snapshot header first and refuses a newer
  version with CatalogError::Newer ("written by a newer version of
  LightCraft"), even when its catalog wouldn't parse.
- A log record whose CRC matches but whose op doesn't parse is "newer",
  not damage or a torn tail: refused before any file is modified.
- Opening an older (or snapshot-less) library writes a snapshot in this
  format before anything is appended (LoadReport::upgraded_from); a new
  library gets one at once, so v0.2.0 refuses it instead of reading a bare
  log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 08c2acda26 Issue #101: catalog.log: cut a partial append back before retrying
A failed append (disk full, dropped share) could leave part of its batch in
catalog.log; the retry (same seqs) then shared a line with the fragment and
the next load treated it as mid-log damage, setting later edits aside.

- Journal::append cuts the log back to the last whole record on error (or,
  if that fails too, before the next append).
- FsStore drops a failed append handle so the next attempt reopens the file;
  truncate of a never-created file is a no-op.
- The loader recognises a CRC-valid record glued after a fragment (logs
  written by older builds) and skips only the fragment.
- log_bytes now counts the newline added to an unterminated last record.
- close_library writes the closing snapshot even when appending fails; the
  snapshot absorbs the queued ops (Journal::snapshot_with_unlogged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:13:00 -04:00
Brandon ThomasandClaude Opus 5.5 1a05c3eed2 Issue #103: Settings files: keep damaged ones, never overwrite unread ones, write ui.json atomically, confirm quit with unsaved changes
- Library settings (prefs.json, presets.json, view.json): only a missing
  file means "defaults". A file that doesn't parse is kept as
  <name>.corrupt-<unix time> before the defaults are used; one that can't
  be read (locked, I/O error) is never written this session (save_prefs
  refuses, presets/view saves skip it). Warnings go to library.info →
  settingsWarnings and Session::take_library_warnings; the UI shows each
  once in a window (ui.inspect → notices, button:noticeOk).
- Desktop ui.json (holds the library location): written atomically (temp +
  fsync + rename + dir fsync), saved as soon as the library path changes and
  every few seconds when it changed, failures reported; a damaged file is
  renamed to ui.json.corrupt-<time> and reported, an unreadable one is not
  overwritten.
- Quit/close with changes only in memory: retry the save (log, then a
  snapshot of everything), and if it still fails cancel the close and ask:
  Try Saving Again / Quit Anyway / Cancel (ui.inspect → quitPrompt).
- Tests: engine (damaged prefs kept, unreadable presets/prefs untouched),
  desktop (damaged ui.json kept, atomic write), headless UI (quit prompt
  buttons, settings notice).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:12:26 -04:00
Brandon ThomasandClaude Opus 5.5 ee4b78f646 Issue #99: Library: exclusive lock so two processes can't open one library
Two processes on one library each kept their own catalog and seq counter;
the last snapshot silently dropped the other's edits.

- lightcraft_catalog:🔒 LibraryLock holds an exclusive OS lock
  (std File::try_lock: flock / LockFileEx) on <library>/catalog.lock for
  the session; who holds it goes to catalog.lock.owner (separate file: a
  Windows lock also blocks reads). Released on drop and by the OS when the
  process dies, so leftover files from a crash never lock anyone out. File
  systems without lock support open unprotected with a warning.
- Session::open_library takes the lock before reading anything and fails
  with EngineError::LibraryInUse ("this library is already open in
  LightCraft 0.x (process N on host)"); reopening the session's own
  library hands its lock over.
- lightcraft-cli (mcp/run/snapshot --library) adds how to drive the
  running app instead (--control + mcp --connect).
- Tests: in-process, leftover files, a killed child process, two engine
  sessions, CLI e2e (mcp --library holds it, run --library is refused).
  Existing tests that reopened a library while it was open now drop the
  first session.
- docs: parity row LRC-LIB-CATALOG-LOCK, mcp.md, web.md (no guard across
  browser tabs yet), whats-new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:48:34 -04:00
Brandon ThomasandClaude Opus 5.5 d592b29d12 Issue #97: Rename Folder / Move Folder are undo steps
The folder move and the photos' relinks form one undo step: undo renames
the folder back on disk (files and sidecars along) and restores the
paths, redo repeats it. A destination taken in the meantime is never
overwritten; the step is refused, reported and stays on the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:30 -04:00
Brandon ThomasandClaude Opus 5.5 cddcb905e0 Issue #94: Control: close the connection on malformed input
A line that is not a JSON object with a string `method` (HTTP request
line, invalid UTF-8, non-object JSON, >4 MiB line) now gets one error
reply and the connection is closed, so a browser's cross-origin fetch
can't smuggle a command in its body. Connections are capped at 16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:27 -04:00
Brandon ThomasandClaude Opus 5.5 ebc73bb7b4 Issue #107: Web build: data-safety fixes for libraries kept in the browser
- File > Back Up Library... (web): a zip of the catalog files and every
  stored original, built entry by entry; Restore Library from Backup...
  reads it into a new library folder and switches only when complete
  (the previous library stays in storage).
- Failed flushes to browser storage make the catalog Store refuse
  writes, so changes show as unsaved (top-bar warning) and are retried
  every 2 s instead of being lost silently.
- A picked/dropped photo whose bytes can't be stored is not added.
- One tab per library (Web Locks); a second tab shows a message.
- Notices: storage not persistent, library failed to open (temporary
  session), experimental build; ?reset asks first; a panic shows a
  message instead of a frozen page; long toasts wrap.
- Hosting: _headers/.htaccess no longer mark the unhashed .js/.wasm
  immutable; HOSTING.md rewritten for the actual build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:46:21 -04:00
Brandon ThomasandClaude Opus 5.5 882a059fac Issue #92: XMP: merge into an existing sidecar instead of replacing it
Saving (Cmd+S or auto-write) now merges LightCraft's properties into an
existing sidecar (lightcraft_meta::merge_xmp): owned properties (rating,
label, dc/IPTC/rights fields, lc:*) are replaced or removed, capture time
and GPS only when LightCraft has a value; everything else (crs:,
xmpMM:History, unknown namespaces, packet wrapper) is kept byte for byte.
The result is re-parsed and checked before it is written (temp + fsync +
rename). A sidecar that isn't valid XMP is copied to .xmp.bak-<time>
(never overwriting) before it is replaced; one that can't be read at all
is left alone and the save fails.

Shared stems: when several catalogued files map to one stem sidecar
(IMG.CR3 + IMG.JPG), it belongs to the raw (else the first by name) and
the others write and read <file>.xmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:08:18 -04:00
73 changed files with 6299 additions and 791 deletions

No files matched your search

Generated
+1
View File
@@ -2361,6 +2361,7 @@ dependencies = [
name = "lightcraft-web"
version = "0.2.0"
dependencies = [
"crc32fast",
"eframe",
"egui",
"js-sys",
+14 -3
View File
@@ -91,6 +91,17 @@ USAGE:
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;
/// Why `--library DIR` can't be opened; for a library open in another program, how to work with
/// that one instead (issue #99).
fn library_error(dir: &str, e: lightcraft_engine::EngineError) -> String {
match e {
lightcraft_engine::EngineError::LibraryInUse(why) => format!(
"{dir}: {why}\nTo work with the library while the app has it open, start the app with `--control PORT` and use `lightcraft-cli mcp --connect 127.0.0.1:PORT`."
),
e => format!("{dir}: {e}"),
}
}
fn main() -> ExitCode {
// `--features dhat-heap`: count allocations; the profile is written when `_heap` drops
// (LIGHTCRAFT_DHAT_FILE, default dhat-heap.json).
@@ -182,7 +193,7 @@ fn mcp(args: &[String]) -> Result<(), String> {
let mut h = match &library {
Some(dir) => {
let mut h = Headless::default();
let r = h.session.open_library(dir, demo).map_err(|e| e.to_string())?;
let r = h.session.open_library(dir, demo).map_err(|e| library_error(dir, e))?;
eprintln!("lightcraft-cli mcp: opened library {dir} ({r:?})");
h
}
@@ -419,7 +430,7 @@ fn run(args: &[String]) -> Result<(), String> {
let mut h = match &library {
Some(dir) => {
let mut h = Headless::default();
h.session.open_library(dir, demo).map_err(|e| e.to_string())?;
h.session.open_library(dir, demo).map_err(|e| library_error(dir, e))?;
h
}
None if demo => Headless::demo(),
@@ -586,7 +597,7 @@ fn snapshot(args: &[String]) -> Result<(), String> {
let mut session = match &library {
Some(dir) => {
let mut s = Session::new().with_fs();
s.open_library(dir, false).map_err(|e| format!("{dir}: {e}"))?;
s.open_library(dir, false).map_err(|e| library_error(dir, e))?;
s
}
None if files.is_empty() => Session::with_demo().with_fs(),
+28
View File
@@ -260,3 +260,31 @@ fn devices_are_listed_and_imported_from() {
let line: Value = serde_json::from_slice(o.stdout.split(|b| *b == b'\n').next().unwrap()).unwrap();
assert_eq!(line["result"]["candidates"].as_array().map(Vec::len), Some(1), "{line}");
}
/// Issue #99: a library open in one process (here `mcp --library`) is refused by a second one,
/// with who has it and how to drive the running app instead; free again once the first exits.
#[test]
fn a_library_open_in_another_process_is_refused() {
let lib = tmp("locked-lib");
let _ = std::fs::remove_dir_all(&lib);
let lib_s = lib.to_str().unwrap();
let mut holder =
Command::new(BIN).args(["mcp", "--library", lib_s]).stdin(Stdio::piped()).stdout(Stdio::null()).stderr(Stdio::piped()).spawn().unwrap();
let mut err = BufReader::new(holder.stderr.take().unwrap());
let mut line = String::new();
while !line.contains("opened library") {
line.clear();
assert!(err.read_line(&mut line).unwrap() > 0, "mcp exited before opening the library");
}
let (ok, _, stderr) = run_cli(&["--library", lib_s, "library.info"], None);
assert!(!ok);
assert!(stderr.contains("already open in lightcraft-cli") && stderr.contains(&format!("process {}", holder.id())), "{stderr}");
assert!(stderr.contains("mcp --connect"), "{stderr}");
drop(holder.stdin.take()); // EOF: the server exits and lets go of the library
assert!(holder.wait().unwrap().success());
let (ok, lines, stderr) = run_cli(&["--library", lib_s, "library.info"], None);
assert!(ok, "{stderr}");
assert_eq!(lines[0]["ok"], true);
let _ = std::fs::remove_dir_all(&lib);
}
+1
View File
@@ -21,6 +21,7 @@ lightcraft-preview = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
log = { workspace = true }
crc32fast = { workspace = true }
[target.'cfg(target_arch = "wasm32")'.dependencies]
eframe = { workspace = true, features = ["default_fonts", "glow"] }
+33 -10
View File
@@ -232,6 +232,18 @@ impl Backend {
}
}
/// Is there a file at `path`?
pub async fn exists(&self, path: &str) -> Result<bool, String> {
match self {
Backend::Opfs(root) => Ok(Self::opfs_file(root, path, false).await?.is_some()),
Backend::Idb(db) => {
let (_tx, store) = Self::idb_store(db, false)?;
let n = idb_req(&store.count_with_key(&path.into()).map_err(err)?).await?;
Ok(n.as_f64().is_some_and(|n| n > 0.0))
}
}
}
pub async fn remove(&self, path: &str) -> Result<(), String> {
match self {
Backend::Opfs(root) => {
@@ -309,14 +321,25 @@ async fn dir_keys(dir: &FileSystemDirectoryHandle) -> Result<Vec<String>, String
Ok(out)
}
/// Ask the browser not to evict our storage under pressure (best effort; fire and forget).
pub fn request_persistence() {
if let Some(sm) = storage_manager()
&& let Ok(p) = sm.persist()
{
wasm_bindgen_futures::spawn_local(async move {
let granted = JsFuture::from(p).await.ok().is_some_and(|v| v.is_truthy());
log::info!("lightcraft: persistent storage {}", if granted { "granted" } else { "not granted (best effort)" });
});
}
/// Ask the browser not to evict our storage under pressure; `on_result(granted)` once known
/// (`false` also when the browser has no storage manager).
pub fn request_persistence(on_result: impl FnOnce(bool) + 'static) {
let Some(sm) = storage_manager() else {
on_result(false);
return;
};
wasm_bindgen_futures::spawn_local(async move {
// already persistent: nothing to ask
let already = match sm.persisted() {
Ok(p) => JsFuture::from(p).await.ok().is_some_and(|v| v.is_truthy()),
Err(_) => false,
};
let granted = already
|| match sm.persist() {
Ok(p) => JsFuture::from(p).await.ok().is_some_and(|v| v.is_truthy()),
Err(_) => false,
};
log::info!("lightcraft: persistent storage {}", if granted { "granted" } else { "not granted" });
on_result(granted);
});
}
+306
View File
@@ -0,0 +1,306 @@
//! Library backups for the browser build: a plain zip (entries stored, not compressed — photos
//! don't shrink) of the library files and every stored original, built one entry at a time so
//! only one file is in wasm memory at once, and read back the same way for a restore.
//!
//! Layout of a backup:
//! - `library/catalog.snap`, `library/catalog.log`, `library/presets.json`, … (see
//! [`crate::files::LIBRARY_FILES`]);
//! - `originals/<content hash>/<file name>`: each imported photo's original bytes, under its own
//! name so an unzipped backup is browsable;
//! - `README.txt`.
//!
//! Restoring never deletes anything: the library files go to a new library folder in browser
//! storage, originals that aren't stored yet are added (they're named by content, so existing
//! ones are identical), and only then does a small pointer file ([`ACTIVE_LIBRARY`]) switch to
//! the restored library. The library that was there before stays in storage.
//!
//! No zip64: a backup holds at most [`MAX_BYTES`] and 65 535 entries.
/// Storage key of the file naming the library folder in use (absent: `library`).
pub const ACTIVE_LIBRARY: &str = "active-library";
/// The default library folder.
pub const DEFAULT_LIBRARY_DIR: &str = "library";
/// Largest backup (offsets are 32-bit without zip64).
pub const MAX_BYTES: u64 = 0xFFFF_FFFF - (64 << 20);
const LOCAL_SIG: u32 = 0x0403_4b50;
const CENTRAL_SIG: u32 = 0x0201_4b50;
const END_SIG: u32 = 0x0605_4b50;
/// General-purpose flag bit 11: names are UTF-8.
const UTF8: u16 = 0x0800;
/// 1980-01-01 00:00 in MS-DOS format (date, time).
const DOS_DATE: u16 = (1 << 5) | 1;
/// The readme inside every backup.
pub const README: &str = "LightCraft library backup (browser version)\n\n\
library/ the catalog (catalog.snap + catalog.log), presets, view state and preferences\n\
originals/ every imported photo, as originals/<content hash>/<file name>\n\n\
Restore it in LightCraft for the web with File > Restore Library from Backup...\n\
The photos in originals/ are the untouched files you imported; edits live in the catalog.\n";
/// Is `name` a safe library folder name (`library` or `library-<letters, digits, ->`)?
pub fn valid_library_dir(name: &str) -> bool {
name == DEFAULT_LIBRARY_DIR
|| name.strip_prefix("library-").is_some_and(|s| !s.is_empty() && s.len() <= 64 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-'))
}
/// Is `h` a plausible content hash (used as a storage key segment)?
pub fn valid_hash(h: &str) -> bool {
!h.is_empty() && h.len() <= 128 && h.chars().all(|c| c.is_ascii_alphanumeric())
}
/// Zip entry name for an original.
pub fn original_entry(hash: &str, file_name: &str) -> String {
let name: String = file_name.chars().map(|c| if c == '/' || c == '\\' || c.is_control() { '_' } else { c }).collect();
let name = if name.is_empty() || name == "." || name == ".." { "original".to_string() } else { name };
format!("originals/{hash}/{name}")
}
/// Where a backup entry goes when restoring into library folder `dir`: `Some(storage key)`, or
/// `None` to skip it (README, thumbnails, anything unknown or unsafe).
pub fn restore_key(entry: &str, dir: &str) -> Option<String> {
if let Some(name) = entry.strip_prefix("library/") {
return crate::files::LIBRARY_FILES.contains(&name).then(|| format!("{dir}/{name}"));
}
let rest = entry.strip_prefix("originals/")?;
let (hash, _) = rest.split_once('/').unwrap_or((rest, ""));
valid_hash(hash).then(|| format!("originals/{hash}"))
}
fn u16le(v: &mut Vec<u8>, x: u16) {
v.extend_from_slice(&x.to_le_bytes());
}
fn u32le(v: &mut Vec<u8>, x: u32) {
v.extend_from_slice(&x.to_le_bytes());
}
struct Central {
name: String,
crc: u32,
size: u32,
offset: u32,
}
/// Writes a stored (uncompressed) zip piece by piece: [`ZipWriter::entry`] returns the local
/// header to emit before the entry's bytes, [`ZipWriter::finish`] the central directory.
#[derive(Default)]
pub struct ZipWriter {
entries: Vec<Central>,
offset: u64,
}
impl ZipWriter {
/// The local header for `name` holding `data` (emit it, then `data`).
pub fn entry(&mut self, name: &str, data: &[u8]) -> Result<Vec<u8>, String> {
let size = u32::try_from(data.len()).map_err(|_| format!("{name}: larger than 4 GB"))?;
let name_len = u16::try_from(name.len()).map_err(|_| format!("{name}: name too long"))?;
if self.entries.len() >= usize::from(u16::MAX) {
return Err("too many files for one backup".into());
}
let header_len = 30 + u64::from(name_len);
if self.offset + header_len + u64::from(size) > MAX_BYTES {
return Err("the library is too large for one backup file (4 GB)".into());
}
let offset = u32::try_from(self.offset).map_err(|_| "backup too large".to_string())?;
let crc = crc32fast::hash(data);
let mut h = Vec::with_capacity(header_len as usize);
u32le(&mut h, LOCAL_SIG);
u16le(&mut h, 20); // version needed
u16le(&mut h, UTF8);
u16le(&mut h, 0); // stored
u16le(&mut h, 0); // time
u16le(&mut h, DOS_DATE);
u32le(&mut h, crc);
u32le(&mut h, size);
u32le(&mut h, size);
u16le(&mut h, name_len);
u16le(&mut h, 0); // extra
h.extend_from_slice(name.as_bytes());
self.offset += header_len + u64::from(size);
self.entries.push(Central { name: name.to_string(), crc, size, offset });
Ok(h)
}
/// The central directory and end record (emit last).
pub fn finish(self) -> Result<Vec<u8>, String> {
let mut cd = Vec::new();
for e in &self.entries {
u32le(&mut cd, CENTRAL_SIG);
u16le(&mut cd, 20); // made by
u16le(&mut cd, 20); // needed
u16le(&mut cd, UTF8);
u16le(&mut cd, 0);
u16le(&mut cd, 0);
u16le(&mut cd, DOS_DATE);
u32le(&mut cd, e.crc);
u32le(&mut cd, e.size);
u32le(&mut cd, e.size);
u16le(&mut cd, e.name.len() as u16);
u16le(&mut cd, 0); // extra
u16le(&mut cd, 0); // comment
u16le(&mut cd, 0); // disk
u16le(&mut cd, 0); // internal attrs
u32le(&mut cd, 0); // external attrs
u32le(&mut cd, e.offset);
cd.extend_from_slice(e.name.as_bytes());
}
let cd_offset = u32::try_from(self.offset).map_err(|_| "backup too large".to_string())?;
let cd_size = u32::try_from(cd.len()).map_err(|_| "backup too large".to_string())?;
let n = self.entries.len() as u16;
let mut out = cd;
u32le(&mut out, END_SIG);
u16le(&mut out, 0);
u16le(&mut out, 0);
u16le(&mut out, n);
u16le(&mut out, n);
u32le(&mut out, cd_size);
u32le(&mut out, cd_offset);
u16le(&mut out, 0); // comment
Ok(out)
}
}
/// One entry of a zip's central directory.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ZipEntry {
pub name: String,
pub crc: u32,
pub size: u64,
/// Offset of the entry's local header.
pub header: u64,
/// 0 = stored (the only method a restore reads).
pub method: u16,
}
fn rd16(b: &[u8], at: usize) -> Option<u16> {
Some(u16::from_le_bytes(b.get(at..at + 2)?.try_into().ok()?))
}
fn rd32(b: &[u8], at: usize) -> Option<u32> {
Some(u32::from_le_bytes(b.get(at..at + 4)?.try_into().ok()?))
}
/// How many bytes from the end of a zip to read to find its end record.
pub fn tail_len(file_len: u64) -> u64 {
file_len.min(22 + 65_535)
}
/// The central directory's (offset, size) from the last [`tail_len`] bytes of a zip.
pub fn find_central(tail: &[u8]) -> Result<(u64, u64), String> {
let at = (0..tail.len().saturating_sub(21)).rev().find(|&i| rd32(tail, i) == Some(END_SIG)).ok_or("not a zip file (no end record)")?;
let size = rd32(tail, at + 12).ok_or("truncated zip")?;
let offset = rd32(tail, at + 16).ok_or("truncated zip")?;
if size == u32::MAX || offset == u32::MAX {
return Err("zip64 backups are not supported".into());
}
Ok((u64::from(offset), u64::from(size)))
}
/// Parse a central directory.
pub fn parse_central(cd: &[u8]) -> Result<Vec<ZipEntry>, String> {
let mut out = Vec::new();
let mut i = 0;
while i + 46 <= cd.len() {
if rd32(cd, i) != Some(CENTRAL_SIG) {
return Err("damaged zip directory".into());
}
let bad = || "damaged zip directory".to_string();
let method = rd16(cd, i + 10).ok_or_else(bad)?;
let crc = rd32(cd, i + 16).ok_or_else(bad)?;
let size = rd32(cd, i + 24).ok_or_else(bad)?;
let n = usize::from(rd16(cd, i + 28).ok_or_else(bad)?);
let extra = usize::from(rd16(cd, i + 30).ok_or_else(bad)?);
let comment = usize::from(rd16(cd, i + 32).ok_or_else(bad)?);
let header = rd32(cd, i + 42).ok_or_else(bad)?;
let name = cd.get(i + 46..i + 46 + n).ok_or_else(bad)?;
out.push(ZipEntry { name: String::from_utf8_lossy(name).into_owned(), crc, size: u64::from(size), header: u64::from(header), method });
i += 46 + n + extra + comment;
}
Ok(out)
}
/// Offset of an entry's data from its local header's first 30 bytes.
pub fn data_offset(entry: &ZipEntry, local: &[u8]) -> Result<u64, String> {
if rd32(local, 0) != Some(LOCAL_SIG) {
return Err(format!("{}: damaged entry", entry.name));
}
let n = rd16(local, 26).ok_or("truncated entry")?;
let extra = rd16(local, 28).ok_or("truncated entry")?;
Ok(entry.header + 30 + u64::from(n) + u64::from(extra))
}
/// Check an entry's bytes.
pub fn verify(entry: &ZipEntry, data: &[u8]) -> Result<(), String> {
if entry.method != 0 {
return Err(format!("{}: compressed entries are not supported (not a LightCraft backup?)", entry.name));
}
if data.len() as u64 != entry.size || crc32fast::hash(data) != entry.crc {
return Err(format!("{}: damaged (checksum mismatch)", entry.name));
}
Ok(())
}
/// A name for the restored library's folder.
pub fn restored_dir_name(now_ms: f64) -> String {
format!("library-restored-{}", now_ms.max(0.0) as u64)
}
#[cfg(test)]
mod tests {
use super::*;
fn build(files: &[(&str, &[u8])]) -> Vec<u8> {
let mut w = ZipWriter::default();
let mut out = Vec::new();
for (n, d) in files {
out.extend(w.entry(n, d).unwrap());
out.extend_from_slice(d);
}
out.extend(w.finish().unwrap());
out
}
#[test]
fn written_backups_read_back() {
let files: [(&str, &[u8]); 3] =
[("README.txt", README.as_bytes()), ("library/catalog.log", b"{\"seq\":1}\n"), ("originals/abc123/IMG 1.jpg", &[0, 1, 2, 3, 255])];
let zip = build(&files);
let tail = &zip[zip.len() - tail_len(zip.len() as u64) as usize..];
let (off, size) = find_central(tail).unwrap();
let entries = parse_central(&zip[off as usize..(off + size) as usize]).unwrap();
assert_eq!(entries.len(), 3);
for (e, (name, data)) in entries.iter().zip(files) {
assert_eq!(e.name, name);
let start = data_offset(e, &zip[e.header as usize..e.header as usize + 30]).unwrap() as usize;
let got = &zip[start..start + e.size as usize];
assert_eq!(got, data);
verify(e, got).unwrap();
}
// damage is caught
let e = &entries[2];
assert!(verify(e, &[0, 1, 2, 3, 254]).is_err());
assert!(find_central(b"not a zip at all, just some text").is_err());
}
#[test]
fn restore_keys_are_safe() {
assert_eq!(restore_key("library/catalog.snap", "library-restored-1").as_deref(), Some("library-restored-1/catalog.snap"));
assert_eq!(restore_key("library/../../x", "library-r"), None);
assert_eq!(restore_key("library/evil.bin", "library-r"), None);
assert_eq!(restore_key("originals/abc123/IMG.jpg", "d").as_deref(), Some("originals/abc123"));
assert_eq!(restore_key("originals/../library/catalog.log", "d"), None);
assert_eq!(restore_key("README.txt", "d"), None);
assert_eq!(restore_key("thumbs/x.jpg", "d"), None);
assert_eq!(original_entry("abc", "a/b\\c.jpg"), "originals/abc/a_b_c.jpg");
assert_eq!(original_entry("abc", ".."), "originals/abc/original");
assert!(valid_library_dir("library") && valid_library_dir("library-restored-17"));
assert!(!valid_library_dir("library-../x") && !valid_library_dir("originals") && !valid_library_dir("library-"));
}
#[test]
fn size_limits_are_errors_not_panics() {
let mut w = ZipWriter { offset: MAX_BYTES - 10, ..Default::default() };
assert!(w.entry("x", &[0; 64]).is_err());
}
}
+58
View File
@@ -49,6 +49,8 @@ struct Mirror {
files: BTreeMap<String, Vec<u8>>,
/// Dirty files, least recently modified first.
dirty: Vec<(String, Dirty)>,
/// Why the last flush to browser storage failed (quota, eviction…), until one succeeds.
error: Option<String>,
}
impl Mirror {
@@ -138,6 +140,18 @@ impl Files {
}
}
/// Record the outcome of a flush to browser storage: `Some(why)` when it failed, `None` once
/// one succeeds. While a flush is failing, the catalog [`Store`] refuses new writes, so the
/// engine keeps the changes queued and reports them unsaved (`library.info` → `unsavedOps`,
/// the top bar's warning) instead of pretending they were saved.
pub fn set_error(&self, e: Option<String>) {
self.lock().error = e;
}
pub fn error(&self) -> Option<String> {
self.lock().error.clone()
}
/// A catalog [`Store`] over these files.
pub fn store(&self) -> FilesStore {
FilesStore(self.clone())
@@ -147,19 +161,32 @@ impl Files {
/// [`Store`] implementation over [`Files`].
pub struct FilesStore(Files);
impl FilesStore {
/// Writes fail while browser storage does (see [`Files::set_error`]).
fn writable(&self) -> io::Result<()> {
match self.0.error() {
Some(e) => Err(io::Error::other(format!("browser storage: {e}"))),
None => Ok(()),
}
}
}
impl Store for FilesStore {
fn read(&mut self, name: &str) -> io::Result<Option<Vec<u8>>> {
Ok(self.0.get(name))
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> io::Result<()> {
self.writable()?;
self.0.write(name, data);
Ok(())
}
fn append(&mut self, name: &str, data: &[u8]) -> io::Result<()> {
self.writable()?;
self.0.append(name, data);
Ok(())
}
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()> {
self.writable()?;
self.0.truncate(name, len as usize);
Ok(())
}
@@ -224,6 +251,37 @@ mod tests {
assert_eq!(names, ["catalog.snap", "catalog.log"]);
}
/// A failing flush (quota exceeded, storage evicted) is not silent: commands report the change
/// as unsaved and the engine keeps it queued; once storage works again it is written.
#[test]
fn flush_failures_show_as_unsaved_and_are_retried() {
let mut disk = BTreeMap::new();
let files = Files::default();
let mut s = Session::new();
let stores = LibraryStores { dir: "browser".into(), catalog: Box::new(files.store()), files: Box::new(files.store()), on_disk: false };
s.open_library_in(stores, true).unwrap();
flush(&files, &mut disk);
files.set_error(Some("QuotaExceededError".into()));
let r = s.execute("photo.rate", &json!({"rating": 5}));
assert!(matches!(r, Err(lightcraft_engine::EngineError::NotSaved(_))), "{r:?}");
let (ops, why) = s.unsaved().unwrap();
assert!(ops > 0 && why.contains("QuotaExceeded"), "{why}");
files.set_error(None);
s.persist_if_dirty();
s.persist().unwrap();
assert!(s.unsaved().is_none(), "written once storage works again");
flush(&files, &mut disk);
let id = s.selection.active.unwrap();
let files2 = Files::default();
for (k, v) in &disk {
files2.preload(k, v.clone());
}
let mut s2 = Session::new();
let stores = LibraryStores { dir: "browser".into(), catalog: Box::new(files2.store()), files: Box::new(files2.store()), on_disk: false };
s2.open_library_in(stores, true).unwrap();
assert_eq!(s2.catalog.photo(id).unwrap().rating, 5, "the rating survived");
}
#[test]
fn library_survives_reload_through_flushes() {
let mut disk = BTreeMap::new();
+3
View File
@@ -16,6 +16,7 @@
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
pub mod backup;
pub mod bench;
pub mod files;
pub mod store;
@@ -24,6 +25,8 @@ pub mod wire;
#[cfg(target_arch = "wasm32")]
mod backend;
#[cfg(target_arch = "wasm32")]
mod safety;
#[cfg(target_arch = "wasm32")]
mod web;
#[cfg(target_arch = "wasm32")]
mod workers;
+256
View File
@@ -0,0 +1,256 @@
//! Keeping a library in the browser safe: notices the user must see (a failed save, storage the
//! browser may evict, a file that couldn't be stored), one tab at a time per library, a message
//! instead of a dead page after a panic, and backup / restore of the whole library
//! ([`crate::backup`]).
use std::cell::RefCell;
use std::collections::HashMap;
use js_sys::{Reflect, Uint8Array};
use wasm_bindgen::JsCast;
use wasm_bindgen::prelude::*;
use wasm_bindgen_futures::JsFuture;
use crate::backend::Backend;
use crate::backup::{
ACTIVE_LIBRARY, README, ZipWriter, data_offset, find_central, original_entry, parse_central, restore_key, restored_dir_name, tail_len,
valid_hash, verify,
};
use crate::files::{Files, LIBRARY_FILES};
use crate::store::storage_key;
thread_local! {
/// Messages for the user, shown by the app on its next frame.
static NOTICES: RefCell<Vec<String>> = const { RefCell::new(Vec::new()) };
}
/// Queue a message for the user (shown for several seconds; also logged).
pub fn notice(text: impl Into<String>) {
let text = text.into();
log::warn!("{text}");
NOTICES.with(|n| n.borrow_mut().push(text));
crate::web::request_repaint();
}
/// Messages queued since the last call.
pub fn take_notices() -> Vec<String> {
NOTICES.with(|n| std::mem::take(&mut *n.borrow_mut()))
}
fn document() -> Option<web_sys::Document> {
web_sys::window()?.document()
}
/// Ask the user (the browser's own confirm box). `false` when it can't be shown.
pub fn confirm(text: &str) -> bool {
web_sys::window().and_then(|w| w.confirm_with_message(text).ok()).unwrap_or(false)
}
/// Cover the page with a message (the app can't continue, or must not start).
pub fn show_blocking(text: &str) {
let Some(doc) = document() else { return };
let el = match doc.get_element_by_id("lightcraft_blocking") {
Some(el) => el,
None => {
let Ok(el) = doc.create_element("div") else { return };
el.set_id("lightcraft_blocking");
let _ = el.set_attribute(
"style",
"position:fixed;inset:0;z-index:10;display:flex;align-items:center;justify-content:center;padding:32px;\
background:#1c1c1c;color:#e0e0e0;font:15px/1.5 system-ui,-apple-system,'Segoe UI',sans-serif;text-align:center;white-space:pre-line",
);
if let Some(body) = doc.body() {
let _ = body.append_child(&el);
}
el
}
};
el.set_text_content(Some(text));
if let Some(loading) = doc.get_element_by_id("lightcraft_loading") {
loading.remove();
}
}
/// `panic = "abort"` on wasm: a panic ends the module. Say so on the page instead of leaving a
/// frozen canvas, and say that the stored library is intact.
pub fn install_panic_hook() {
std::panic::set_hook(Box::new(|info| {
let msg = match (info.payload().downcast_ref::<&str>(), info.payload().downcast_ref::<String>()) {
(Some(s), _) => (*s).to_string(),
(_, Some(s)) => s.clone(),
_ => "unknown error".into(),
};
let at = info.location().map(|l| format!(" at {}:{}", l.file(), l.line())).unwrap_or_default();
log::error!("LightCraft panicked{at}: {msg}");
show_blocking(&format!(
"LightCraft stopped after an internal error:\n{msg}\n\nThe library saved in this browser is kept. Reload the page to continue.\n\
If it happens again right away, a photo may be the cause: reload with ?workers=0 or report the error."
));
}));
}
/// The name of the lock that keeps a library to one tab.
const TAB_LOCK: &str = "lightcraft-library";
/// Hold the library lock for as long as this page lives (Web Locks API). `Some(true)`: ours;
/// `Some(false)`: another tab of this site has it; `None`: the browser can't tell (no Web Locks).
pub async fn acquire_tab_lock() -> Option<bool> {
let nav = Reflect::get(&js_sys::global(), &"navigator".into()).ok()?;
let locks = Reflect::get(&nav, &"locks".into()).ok().filter(|l| !l.is_undefined() && !l.is_null())?;
let request: js_sys::Function = Reflect::get(&locks, &"request".into()).ok()?.dyn_into().ok()?;
let opts = js_sys::Object::new();
Reflect::set(&opts, &"ifAvailable".into(), &true.into()).ok()?;
let granted = js_sys::Promise::new(&mut |resolve, _reject| {
let cb = Closure::once_into_js(move |lock: JsValue| -> JsValue {
let ours = !lock.is_null() && !lock.is_undefined();
let _ = resolve.call1(&JsValue::NULL, &ours.into());
// keep the lock until the page goes away: a promise that never settles
if ours { js_sys::Promise::new(&mut |_, _| {}).into() } else { JsValue::NULL }
});
if let Err(e) = request.call3(&locks, &TAB_LOCK.into(), &opts, &cb) {
log::warn!("Web Locks request failed: {e:?}");
}
});
JsFuture::from(granted).await.ok().map(|v| v.is_truthy())
}
/// Offer a blob as a download named `name`.
pub fn download_blob(name: &str, blob: &web_sys::Blob) -> Result<(), String> {
let e = |e: JsValue| format!("download failed: {e:?}");
let url = web_sys::Url::create_object_url_with_blob(blob).map_err(e)?;
let doc = document().ok_or("no document")?;
let a: web_sys::HtmlAnchorElement = doc.create_element("a").map_err(e)?.unchecked_into();
a.set_href(&url);
a.set_download(name);
a.click();
// revoke once the download has started
let revoke = Closure::once_into_js(move || {
let _ = web_sys::Url::revoke_object_url(&url);
});
if let Some(w) = web_sys::window() {
let _ = w.set_timeout_with_callback_and_timeout_and_arguments_0(revoke.unchecked_ref(), 60_000);
}
Ok(())
}
/// File ▸ Back Up Library…: a zip of the library files (as the app has them, saved or not) and
/// every original in storage, built entry by entry into the browser's memory (outside the wasm
/// heap) and downloaded. `names`: content hash → the photo's file name.
pub async fn backup(files: Files, backend: Backend, names: HashMap<String, String>) -> Result<(usize, f64), String> {
let parts = js_sys::Array::new();
let mut zip = ZipWriter::default();
let mut bytes = 0f64;
let mut add = |zip: &mut ZipWriter, name: &str, data: &[u8]| -> Result<(), String> {
let header = zip.entry(name, data)?;
parts.push(&Uint8Array::from(&header[..]));
parts.push(&Uint8Array::from(data));
bytes += (header.len() + data.len()) as f64;
Ok(())
};
add(&mut zip, "README.txt", README.as_bytes())?;
for name in LIBRARY_FILES {
if let Some(d) = files.get(name) {
add(&mut zip, &format!("library/{name}"), &d)?;
}
}
let mut originals = 0;
for hash in backend.list("originals").await? {
if !valid_hash(&hash) {
continue;
}
match backend.read(&storage_key(&hash)).await {
Ok(Some(data)) => {
let name = names.get(&hash).map_or("original", String::as_str);
add(&mut zip, &original_entry(&hash, name), &data)?;
originals += 1;
}
Ok(None) => {}
Err(e) => return Err(format!("reading an original from browser storage: {e}")),
}
}
let tail = zip.finish()?;
parts.push(&Uint8Array::from(&tail[..]));
let opts = web_sys::BlobPropertyBag::new();
opts.set_type("application/zip");
let blob = web_sys::Blob::new_with_u8_array_sequence_and_options(&parts, &opts).map_err(|e| format!("{e:?}"))?;
let day: String = String::from(js_sys::Date::new_0().to_iso_string()).chars().take(10).collect();
download_blob(&format!("lightcraft-library-backup-{day}.zip"), &blob)?;
Ok((originals, bytes + tail.len() as f64))
}
/// Bytes `start..end` of a picked file.
async fn read_range(file: &web_sys::File, start: u64, end: u64) -> Result<Vec<u8>, String> {
let blob = file.slice_with_f64_and_f64(start as f64, end as f64).map_err(|e| format!("{e:?}"))?;
let buf = JsFuture::from(blob.array_buffer()).await.map_err(|e| format!("{e:?}"))?;
Ok(Uint8Array::new(&buf).to_vec())
}
/// Restore a backup into a new library folder (nothing in storage is deleted or overwritten),
/// then point [`ACTIVE_LIBRARY`] at it. Returns the folder's name; the caller reloads the page.
pub async fn restore(file: web_sys::File, backend: Backend) -> Result<String, String> {
let len = file.size() as u64;
let tail = read_range(&file, len - tail_len(len), len).await?;
let (cd_offset, cd_size) = find_central(&tail)?;
let cd = read_range(&file, cd_offset, cd_offset + cd_size).await?;
let entries = parse_central(&cd)?;
if !entries.iter().any(|e| e.name == "library/catalog.snap" || e.name == "library/catalog.log") {
return Err("this isn't a LightCraft library backup (no library/catalog files)".into());
}
let dir = restored_dir_name(js_sys::Date::now());
let mut written = 0usize;
for e in &entries {
let Some(key) = restore_key(&e.name, &dir) else { continue };
// originals are named by their content: one already stored is the same file
if key.starts_with("originals/") && backend.exists(&key).await? {
continue;
}
let local = read_range(&file, e.header, e.header + 30).await?;
let start = data_offset(e, &local)?;
let data = read_range(&file, start, start + e.size).await?;
verify(e, &data)?;
backend.write(&key, &data).await.map_err(|err| format!("storing {}: {err}", e.name))?;
written += 1;
}
backend.write(ACTIVE_LIBRARY, dir.as_bytes()).await?;
log::info!("lightcraft: restored {written} files into {dir}");
Ok(dir)
}
/// File ▸ Restore Library from Backup…: confirm, pick the zip, restore, reload.
pub fn pick_and_restore(backend: Backend, before_reload: impl FnOnce() + 'static) {
let ok = confirm(
"Restore a LightCraft library backup?\n\nThe backup becomes the library in this browser and the page reloads. \
The current library is not deleted: it stays in browser storage.",
);
if !ok {
return;
}
let Some(doc) = document() else { return };
let Ok(input) = doc.create_element("input").map(|e| e.unchecked_into::<web_sys::HtmlInputElement>()) else { return };
input.set_type("file");
input.set_accept(".zip,application/zip");
let inp = input.clone();
let mut before_reload = Some(before_reload);
let on_change = Closure::<dyn FnMut()>::new(move || {
let Some(file) = inp.files().and_then(|f| f.get(0)) else { return };
let backend = backend.clone();
let before = before_reload.take();
notice(format!("Restoring {}…", file.name()));
wasm_bindgen_futures::spawn_local(async move {
match restore(file, backend).await {
Ok(_) => {
if let Some(f) = before {
f();
}
if let Some(w) = web_sys::window() {
let _ = w.location().reload();
}
}
Err(e) => notice(format!("Restore failed: {e}. Nothing was changed: the current library is still in use.")),
}
});
});
input.set_onchange(Some(on_change.as_ref().unchecked_ref()));
on_change.forget();
input.click();
}
+154 -23
View File
@@ -15,16 +15,21 @@ use wasm_bindgen::JsCast;
use wasm_bindgen::prelude::*;
use crate::backend::Backend;
use crate::backup::{ACTIVE_LIBRARY, DEFAULT_LIBRARY_DIR, valid_library_dir};
use crate::bench::Bench;
use crate::files::{Files, FlushOp, LIBRARY_FILES};
use crate::safety::{self, notice};
use crate::store::{Originals, content_hash, download_name, storage_key};
use crate::wire::ThumbIndex;
use crate::workers::{THUMB_INDEX, Workers};
const ACCEPT: &str = ".jpg,.jpeg,.png,.tif,.tiff,.webp,.dng,.cr2,.nef,.arw,.psd,.jxl,.gif,.bmp";
/// Where the library files live in browser storage.
const LIBRARY_DIR: &str = "library";
/// After a failed save, wait this long before trying again (ms).
const SAVE_RETRY_MS: f64 = 2000.0;
/// How long a notice stays on screen (s).
const NOTICE_SECS: f64 = 8.0;
/// How often view state, UI prefs and the thumbnail index are saved (ms).
const SAVE_EVERY_MS: f64 = 1000.0;
@@ -45,8 +50,18 @@ fn set_status(text: &str) {
}
}
fn library_key(name: &str) -> String {
format!("{LIBRARY_DIR}/{name}")
/// Storage key of library file `name` in library folder `dir`.
fn library_key(dir: &str, name: &str) -> String {
format!("{dir}/{name}")
}
/// Repaint the app (notices from async tasks).
pub(crate) fn request_repaint() {
CTX.with(|c| {
if let Some(c) = c.borrow().as_ref() {
c.request_repaint();
}
});
}
/// URL options (`?bench&store=idb&workers=2&reset`).
@@ -86,8 +101,9 @@ async fn store_file(originals: Originals, backend: Option<Backend>, name: String
if let Some(b) = &backend
&& let Err(e) = b.write(&storage_key(&hash), &bytes).await
{
// still importable for this session; it won't survive a reload
log::error!("storing {name} in browser storage failed: {e}");
// a photo whose bytes aren't stored would be gone after a reload: don't add it
notice(format!("{name} was not added: browser storage refused it ({e}). Free up space or back up and clear old photos."));
return;
}
originals.added(&name, &hash, Arc::from(bytes));
ctx.request_repaint();
@@ -148,8 +164,42 @@ fn download(path: &str, bytes: &[u8]) -> Result<(), String> {
Ok(())
}
fn services(originals: Originals, backend: Option<Backend>, ctx: egui::Context) -> Services {
fn services(originals: Originals, backend: Option<Backend>, files: Files, frozen: Rc<Cell<bool>>, ctx: egui::Context) -> Services {
let (backup_backend, backup_files) = (backend.clone(), files);
let restore_backend = backend.clone();
Services {
backup_library: Some(Box::new(move |session: &mut Session| {
let Some(b) = backup_backend.clone() else { return Err("nothing is stored in this browser session (?store=memory)".into()) };
// the photos' own names for the originals in the zip
let names: std::collections::HashMap<String, String> = session
.catalog
.photos()
.filter_map(|p| match &p.source {
Source::File { path } => crate::store::hash_of_path(path).map(|h| (h.to_string(), p.file_name.clone())),
_ => None,
})
.collect();
let files = backup_files.clone();
notice("Backing up the library… (the download starts when it's ready)");
wasm_bindgen_futures::spawn_local(async move {
match safety::backup(files, b, names).await {
Ok((n, bytes)) => notice(format!(
"Library backed up: {n} original{} ({:.1} MB). Keep the file somewhere safe.",
if n == 1 { "" } else { "s" },
bytes / 1e6
)),
Err(e) => notice(format!("Backup failed: {e}")),
}
});
Ok(json!({"started": true}))
})),
restore_library: Some(Box::new(move |_session: &mut Session| {
let Some(b) = restore_backend.clone() else { return Err("nothing is stored in this browser session (?store=memory)".into()) };
let frozen = frozen.clone();
// once the backup is in place, stop saving the old library (the page reloads)
safety::pick_and_restore(b, move || frozen.set(true));
Ok(json!({"started": true}))
})),
pick_files: Some(Box::new(move || {
open_picker(originals.clone(), backend.clone(), ctx.clone());
Vec::new() // files arrive asynchronously and are imported on a later frame
@@ -176,21 +226,29 @@ fn services(originals: Originals, backend: Option<Backend>, ctx: egui::Context)
}
}
/// Write dirty library files to storage, in order, until none are left.
async fn flush(files: Files, backend: Backend, flushing: Rc<Cell<bool>>) {
/// Write dirty library files to storage, in order, until none are left. A failure (quota,
/// storage cleared) is recorded in `files` — the catalog then reports changes as unsaved instead
/// of accepting them silently — and retried after [`SAVE_RETRY_MS`].
async fn flush(files: Files, backend: Backend, dir: String, flushing: Rc<Cell<bool>>, failed_at: Rc<Cell<f64>>) {
'outer: loop {
let ops = files.take_dirty();
if ops.is_empty() {
if files.error().is_some() {
log::info!("lightcraft: saving works again");
}
files.set_error(None);
break;
}
for (i, op) in ops.iter().enumerate() {
let r = match op {
FlushOp::Write { name, data } => backend.write(&library_key(name), data).await,
FlushOp::Append { name, offset, data } => backend.write_at(&library_key(name), Some(*offset), data).await,
FlushOp::Write { name, data } => backend.write(&library_key(&dir, name), data).await,
FlushOp::Append { name, offset, data } => backend.write_at(&library_key(&dir, name), Some(*offset), data).await,
};
if let Err(e) = r {
log::error!("saving {} failed: {e}", op.name());
files.failed(&ops[i..]);
files.set_error(Some(e));
failed_at.set(perf_now());
break 'outer;
}
}
@@ -204,6 +262,8 @@ struct Boot {
backend: Option<Backend>,
files: Files,
index: ThumbIndex,
/// The library folder in storage (`library`, or a restored backup's).
lib_dir: String,
}
async fn boot(opts: Options) -> Boot {
@@ -221,15 +281,32 @@ async fn boot(opts: Options) -> Boot {
};
let files = Files::default();
let mut index = ThumbIndex::default();
let mut lib_dir = DEFAULT_LIBRARY_DIR.to_string();
if let Some(b) = &backend {
if opts.reset {
match b.clear().await {
Ok(()) => log::info!("lightcraft: storage cleared (?reset)"),
Err(e) => log::error!("clearing storage: {e}"),
// everything this site keeps in the browser, photos included: never without asking
let ok = safety::confirm(
"?reset deletes the LightCraft library stored in this browser, including every imported photo. \
This can't be undone. Delete it?",
);
if ok {
match b.clear().await {
Ok(()) => log::info!("lightcraft: storage cleared (?reset)"),
Err(e) => log::error!("clearing storage: {e}"),
}
} else {
log::info!("lightcraft: ?reset declined; storage kept");
}
}
// a restored backup lives in its own folder (see `backup`)
if let Ok(Some(d)) = b.read(ACTIVE_LIBRARY).await {
let d = String::from_utf8_lossy(&d).trim().to_string();
if valid_library_dir(&d) {
lib_dir = d;
}
}
for name in LIBRARY_FILES {
match b.read(&library_key(name)).await {
match b.read(&library_key(&lib_dir, name)).await {
Ok(Some(data)) => files.preload(name, data),
Ok(None) => {}
Err(e) => log::error!("reading {name}: {e}"),
@@ -238,7 +315,14 @@ async fn boot(opts: Options) -> Boot {
if let Ok(Some(j)) = b.read(THUMB_INDEX).await {
index = ThumbIndex::from_json(&j);
}
crate::backend::request_persistence();
crate::backend::request_persistence(|granted| {
if !granted {
notice(
"This browser may delete LightCraft's library when it runs short of space (persistent storage wasn't granted). \
Keep your original photos elsewhere and use File ▸ Back Up Library… now and then.",
);
}
});
// thumbnails stored without an index entry (index saved late, or lost): delete them
let b2 = b.clone();
let known: std::collections::HashSet<String> = b
@@ -266,7 +350,7 @@ async fn boot(opts: Options) -> Boot {
index.len(),
index.total() as f64 / 1e6
);
Boot { opts, backend, files, index }
Boot { opts, backend, files, index, lib_dir }
}
struct WebApp {
@@ -276,6 +360,11 @@ struct WebApp {
backend: Option<Backend>,
workers: Option<Workers>,
flushing: Rc<Cell<bool>>,
/// When the last save to storage failed (ms; 0 = never).
flush_failed_at: Rc<Cell<f64>>,
/// Saving stopped (a restored backup is about to replace this library: the page reloads).
frozen: Rc<Cell<bool>>,
lib_dir: String,
bench: Option<Bench>,
first_frame_logged: bool,
last_save: f64,
@@ -284,17 +373,18 @@ struct WebApp {
impl WebApp {
fn new(cc: &eframe::CreationContext<'_>, boot: Boot) -> Self {
let Boot { opts, backend, files, index } = boot;
let Boot { opts, backend, files, index, lib_dir } = boot;
let originals = Originals::default();
let t = perf_now();
let mut session = Session::new();
originals.install(&mut session);
let stores = LibraryStores {
dir: format!("browser:{}/{LIBRARY_DIR}", backend.as_ref().map_or("memory", |b| b.kind())).into(),
dir: format!("browser:{}/{lib_dir}", backend.as_ref().map_or("memory", |b| b.kind())).into(),
catalog: Box::new(files.store()),
files: Box::new(files.store()),
on_disk: false,
};
let mut problem = None;
match session.open_library_in(stores, true).cloned() {
Ok(r) => log::info!(
"lightcraft: library {} in {:.0} ms ({} photos; snapshot seq {}, {} ops replayed)",
@@ -305,19 +395,35 @@ impl WebApp {
r.replayed
),
Err(e) => {
log::error!("opening the library failed: {e}; starting a temporary one");
session = Session::with_demo();
// never a silent demo session (issue #100): an empty one, and the window says why
log::error!("opening the library failed: {e}");
session = Session::new();
originals.install(&mut session);
problem = Some(lightcraft_ui_egui::panels::library_problem::LibraryProblem {
can_retry: false,
..lightcraft_ui_egui::panels::library_problem::LibraryProblem::new(
"the browser's storage for this page",
format!("{e}. Reload the page to try again."),
)
});
}
}
if backend.is_some() {
notice(
"LightCraft in the browser is experimental. Your library is kept in this browser's storage: keep your original photos \
elsewhere and back up with File ▸ Back Up Library….",
);
}
// previews are large (≤ 2560 px, f32): keep few in a 32-bit address space
session.media.preview_capacity = 3;
let mut app = LightcraftApp::new(session, services(originals.clone(), backend.clone(), cc.egui_ctx.clone()));
let frozen = Rc::new(Cell::new(false));
let mut app = LightcraftApp::new(session, services(originals.clone(), backend.clone(), files.clone(), frozen.clone(), cc.egui_ctx.clone()));
let ui_written = files.get("ui.json").unwrap_or_default();
if let Ok(ui) = serde_json::from_slice::<UiState>(&ui_written) {
app.ui = ui;
}
app.ui = app.ui.sanitized();
app.library_problem = problem;
let n = opts.workers.unwrap_or_else(|| {
let cores = window().map_or(1, |w| w.navigator().hardware_concurrency() as usize);
cores.saturating_sub(1).clamp(1, 4)
@@ -337,6 +443,9 @@ impl WebApp {
backend,
workers,
flushing: Rc::new(Cell::new(false)),
flush_failed_at: Rc::new(Cell::new(0.0)),
frozen,
lib_dir,
bench: opts.bench.then(|| Bench::new(origin)),
first_frame_logged: false,
last_save: 0.0,
@@ -398,6 +507,9 @@ impl WebApp {
/// Save view state, UI prefs and the thumbnail index now and then; flush dirty files.
fn save(&mut self) {
if self.frozen.get() {
return;
}
let now = perf_now();
if now - self.last_save >= SAVE_EVERY_MS {
self.last_save = now;
@@ -419,12 +531,16 @@ impl WebApp {
});
}
}
// after a failure, retry every couple of seconds rather than every frame
let backing_off = self.files.error().is_some() && now - self.flush_failed_at.get() < SAVE_RETRY_MS;
if let Some(b) = &self.backend
&& self.files.is_dirty()
&& !self.flushing.get()
&& !backing_off
{
self.flushing.set(true);
wasm_bindgen_futures::spawn_local(flush(self.files.clone(), b.clone(), self.flushing.clone()));
let (files, dir, flushing, failed_at) = (self.files.clone(), self.lib_dir.clone(), self.flushing.clone(), self.flush_failed_at.clone());
wasm_bindgen_futures::spawn_local(flush(files, b.clone(), dir, flushing, failed_at));
}
}
}
@@ -454,8 +570,10 @@ impl WebApp {
let (orig_n, orig_bytes) = self.originals.usage();
json!({
"storage": self.backend.as_ref().map_or("memory", |b| b.kind()),
"library": self.lib_dir,
"dirty": self.files.is_dirty(),
"flushing": self.flushing.get(),
"saveError": self.files.error(),
"workers": {"alive": alive, "ready": ready, "remoteJobs": remote, "inlineJobs": inline},
"originalsInMemory": orig_n,
"originalBytesInMemory": orig_bytes,
@@ -480,6 +598,10 @@ impl WebApp {
impl eframe::App for WebApp {
fn logic(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
for text in safety::take_notices() {
let now = ctx.input(|i| i.time);
self.app.ui.toast = Some((text, now + NOTICE_SECS));
}
self.run_inbox();
self.import_dropped(ctx);
self.load_originals(ctx);
@@ -517,9 +639,18 @@ impl eframe::App for WebApp {
#[wasm_bindgen]
pub fn start() {
eframe::WebLogger::init(log::LevelFilter::Info).ok();
safety::install_panic_hook();
log::info!("lightcraft: wasm instantiated at {:.0} ms", perf_now());
let opts = Options::from_url();
wasm_bindgen_futures::spawn_local(async move {
// one tab per library: two would each keep their own copy and overwrite each other's saves
if opts.store != "memory" && safety::acquire_tab_lock().await == Some(false) {
safety::show_blocking(
"LightCraft is already open in another tab or window of this browser.\n\n\
Switch to that tab, or close it and reload this page. (Two tabs would overwrite each other's changes.)",
);
return;
}
let Some(canvas) = window()
.and_then(|w| w.document())
.and_then(|d| d.get_element_by_id("lightcraft_canvas"))
+237 -32
View File
@@ -1,14 +1,28 @@
//! Loopback JSON-lines control server: one request per line, one reply per line.
//! This is the transport the MCP server (`lightcraft-cli mcp --connect`) wraps.
//!
//! The server only ever reads requests: a line that is not a JSON object with a string `method`
//! (an HTTP request line from a browser's cross-origin `fetch`, a stray `nc`, binary junk) gets
//! one error reply and the connection is closed, so nothing after it is executed. Lines are
//! bounded ([`MAX_LINE`]) and so is the number of open connections ([`MAX_CONNECTIONS`]); junk
//! never reaches the UI thread. See `docs/control-protocol.md`.
use std::io::{BufRead, BufReader, Write};
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{TcpListener, TcpStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::mpsc::{Receiver, Sender, channel};
use std::time::Duration;
use lightcraft_ui_egui::ControlRequest;
use serde_json::{Value, json};
/// Longest accepted request line (bytes, without the newline). Requests are small JSON objects;
/// anything longer is refused and the connection closed.
pub const MAX_LINE: usize = 4 * 1024 * 1024;
/// Connections served at once; further ones get an error line and are closed.
pub const MAX_CONNECTIONS: usize = 16;
pub fn start(port: u16, ctx: egui::Context) -> Receiver<ControlRequest> {
let (tx, rx) = channel::<ControlRequest>();
let listener = match TcpListener::bind(("127.0.0.1", port)) {
@@ -19,44 +33,235 @@ pub fn start(port: u16, ctx: egui::Context) -> Receiver<ControlRequest> {
}
};
eprintln!("lightcraft: control server listening on 127.0.0.1:{port}");
std::thread::spawn(move || {
for stream in listener.incoming().flatten() {
let tx = tx.clone();
let ctx = ctx.clone();
std::thread::spawn(move || serve(stream, tx, ctx));
}
});
std::thread::spawn(move || accept_loop(listener, tx, ctx));
rx
}
fn serve(stream: TcpStream, tx: Sender<ControlRequest>, ctx: egui::Context) {
let Ok(read) = stream.try_clone() else { return };
let mut out = stream;
for line in BufReader::new(read).lines() {
let Ok(line) = line else { break };
if line.trim().is_empty() {
fn accept_loop(listener: TcpListener, tx: Sender<ControlRequest>, ctx: egui::Context) {
let open = Arc::new(AtomicUsize::new(0));
for mut stream in listener.incoming().flatten() {
if open.fetch_add(1, Ordering::SeqCst) >= MAX_CONNECTIONS {
open.fetch_sub(1, Ordering::SeqCst);
let _ = writeln!(stream, "{}", json!({"ok": false, "error": "too many control connections"}));
continue;
}
let reply = match serde_json::from_str::<Value>(&line) {
Ok(msg) => {
let id = msg.get("id").cloned().unwrap_or(Value::Null);
let method = msg.get("method").and_then(Value::as_str).unwrap_or("").to_string();
let params = msg.get("params").cloned().unwrap_or(json!({}));
let (req, rrx) = ControlRequest::new(method, params);
if tx.send(req).is_err() {
break;
}
ctx.request_repaint();
let mut r = rrx.recv_timeout(Duration::from_secs(60)).unwrap_or_else(|_| json!({"ok": false, "error": "timeout"}));
if let Some(o) = r.as_object_mut() {
o.insert("id".into(), id);
let tx = tx.clone();
let ctx = ctx.clone();
let open = Arc::clone(&open);
std::thread::spawn(move || {
serve(stream, &tx, &ctx);
open.fetch_sub(1, Ordering::SeqCst);
});
}
}
fn serve(stream: TcpStream, tx: &Sender<ControlRequest>, ctx: &egui::Context) {
let Ok(read) = stream.try_clone() else { return };
serve_lines(read, stream, |method, params| {
let (req, rrx) = ControlRequest::new(method, params);
tx.send(req).ok()?;
ctx.request_repaint();
Some(rrx.recv_timeout(Duration::from_secs(60)).unwrap_or_else(|_| json!({"ok": false, "error": "timeout"})))
});
}
/// One parsed line.
enum Line {
Request {
id: Value,
method: String,
params: Value,
},
Blank,
/// Not a request: reply with this error and close the connection.
Reject(String),
}
fn parse_line(bytes: &[u8]) -> Line {
let Ok(text) = std::str::from_utf8(bytes) else {
return Line::Reject("not a control request (invalid UTF-8); closing the connection".into());
};
let text = text.trim();
if text.is_empty() {
return Line::Blank;
}
let msg = match serde_json::from_str::<Value>(text) {
Ok(v) => v,
Err(e) => return Line::Reject(format!("bad JSON ({e}); expected one JSON request object per line; closing the connection")),
};
let Some(obj) = msg.as_object() else {
return Line::Reject("not a control request (expected a JSON object); closing the connection".into());
};
let Some(method) = obj.get("method").and_then(Value::as_str) else {
return Line::Reject("not a control request (missing string `method`); closing the connection".into());
};
Line::Request {
id: obj.get("id").cloned().unwrap_or(Value::Null),
method: method.to_string(),
params: obj.get("params").cloned().unwrap_or(json!({})),
}
}
/// Read one `\n`-terminated line of at most `max` bytes into `buf` (newline stripped).
/// `Ok(false)` at end of stream; `Err` when the line is too long or the read fails.
fn read_bounded_line(reader: &mut impl BufRead, buf: &mut Vec<u8>, max: usize) -> Result<bool, String> {
buf.clear();
// `take` bounds the bytes buffered for one line: a peer can't make us allocate more than `max`.
let limit = u64::try_from(max).unwrap_or(u64::MAX).saturating_add(1);
let n = reader.take(limit).read_until(b'\n', buf).map_err(|e| e.to_string())?;
if n == 0 {
return Ok(false);
}
if buf.last() == Some(&b'\n') {
buf.pop();
if buf.last() == Some(&b'\r') {
buf.pop();
}
return Ok(true);
}
if buf.len() > max {
return Err(format!("request line longer than {max} bytes; closing the connection"));
}
// Last line without a newline, then end of stream.
Ok(true)
}
/// Serve requests from `read`, answering on `out`, until the peer closes, sends something that is
/// not a request, or `handle` returns `None` (app gone). `handle` gets `(method, params)` and
/// returns the reply object (the request's `id` is added here).
fn serve_lines<R: Read, W: Write>(read: R, mut out: W, mut handle: impl FnMut(String, Value) -> Option<Value>) {
let mut reader = BufReader::new(read);
let mut buf = Vec::new();
loop {
let reject = match read_bounded_line(&mut reader, &mut buf, MAX_LINE) {
Ok(false) => return,
Err(e) => e,
Ok(true) => match parse_line(&buf) {
Line::Blank => continue,
Line::Reject(e) => e,
Line::Request { id, method, params } => {
let Some(mut r) = handle(method, params) else { return };
if let Some(o) = r.as_object_mut() {
o.insert("id".into(), id);
}
if writeln!(out, "{r}").and_then(|()| out.flush()).is_err() {
return;
}
continue;
}
r
}
Err(e) => json!({"ok": false, "error": format!("bad JSON: {e}")}),
},
};
if writeln!(out, "{reply}").is_err() {
break;
let _ = writeln!(out, "{}", json!({"ok": false, "error": reject})).and_then(|()| out.flush());
return;
}
}
#[cfg(test)]
mod tests {
use super::*;
fn run(input: &[u8]) -> (Vec<Value>, Vec<String>) {
let mut out = Vec::new();
let mut calls = Vec::new();
serve_lines(input, &mut out, |m, _p| {
calls.push(m);
Some(json!({"ok": true, "result": null}))
});
let replies = String::from_utf8(out).unwrap().lines().map(|l| serde_json::from_str::<Value>(l).unwrap()).collect();
(replies, calls)
}
#[test]
fn requests_are_answered_in_order_with_ids() {
let (r, calls) = run(b"{\"id\":1,\"method\":\"a\"}\n\n{\"id\":2,\"method\":\"b\",\"params\":{}}\r\n{\"method\":\"c\"}");
assert_eq!(calls, ["a", "b", "c"]);
assert_eq!(r.len(), 3);
assert_eq!(r[0]["id"], 1);
assert_eq!(r[1]["id"], 2);
assert_eq!(r[2]["id"], Value::Null);
}
#[test]
fn http_request_closes_before_the_body_runs() {
// A browser's cross-origin `fetch` POST: the request line is rejected and the connection
// closed, so the JSON body is never executed.
let http = b"POST / HTTP/1.1\r\nHost: 127.0.0.1:7980\r\nContent-Type: text/plain\r\n\r\n\
{\"id\":1,\"method\":\"engine.execute\",\"params\":{\"command\":\"folder.move\"}}\n";
let (r, calls) = run(http);
assert!(calls.is_empty(), "nothing may run: {calls:?}");
assert_eq!(r.len(), 1);
assert_eq!(r[0]["ok"], false);
assert!(r[0]["error"].as_str().unwrap().contains("closing the connection"));
}
#[test]
fn non_request_json_closes_the_connection() {
for junk in [
&b"[1,2]\n{\"method\":\"a\"}\n"[..],
b"{\"id\":1}\n{\"method\":\"a\"}\n",
b"{\"method\":5}\n{\"method\":\"a\"}\n",
b"\xff\xfe\n{\"method\":\"a\"}\n",
] {
let (r, calls) = run(junk);
assert!(calls.is_empty());
assert_eq!(r.len(), 1);
assert_eq!(r[0]["ok"], false);
}
}
#[test]
fn a_request_before_junk_still_runs() {
let (r, calls) = run(b"{\"method\":\"a\"}\nGET / HTTP/1.1\n{\"method\":\"b\"}\n");
assert_eq!(calls, ["a"]);
assert_eq!(r.len(), 2);
assert_eq!(r[1]["ok"], false);
}
#[test]
fn overlong_line_is_refused_without_buffering_it_all() {
// An endless line without a newline: reading stops after MAX_LINE + 1 bytes.
let endless = std::io::repeat(b'x');
let mut out = Vec::new();
let mut ran = false;
serve_lines(endless, &mut out, |_, _| {
ran = true;
None
});
assert!(!ran);
let reply: Value = serde_json::from_slice(out.strip_suffix(b"\n").unwrap()).unwrap();
assert!(reply["error"].as_str().unwrap().contains("longer than"));
}
#[test]
fn line_at_the_limit_is_accepted() {
let mut buf = Vec::new();
let line = vec![b'a'; 10];
let mut input = line.clone();
input.push(b'\n');
assert_eq!(read_bounded_line(&mut &input[..], &mut buf, 10), Ok(true));
assert_eq!(buf, line);
assert!(read_bounded_line(&mut &[b'a'; 11][..], &mut buf, 10).is_err());
}
#[test]
fn tcp_connection_is_closed_after_junk() {
// End to end over a real loopback socket (the transport the app uses).
let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap();
let addr = listener.local_addr().unwrap();
let server = std::thread::spawn(move || {
let (s, _) = listener.accept().unwrap();
let read = s.try_clone().unwrap();
serve_lines(read, s, |_, _| Some(json!({"ok": true})));
});
let mut c = TcpStream::connect(addr).unwrap();
c.set_read_timeout(Some(Duration::from_secs(10))).unwrap();
c.write_all(b"{\"id\":7,\"method\":\"x\"}\nhello\n{\"method\":\"y\"}\n").unwrap();
let mut text = String::new();
c.read_to_string(&mut text).unwrap(); // returns at EOF: the server closed the connection
server.join().unwrap();
let lines: Vec<&str> = text.lines().collect();
assert_eq!(lines.len(), 2, "{text}");
assert!(lines[0].contains("\"id\":7"));
assert!(lines[1].contains("closing the connection"));
}
}
+200 -28
View File
@@ -7,7 +7,9 @@
//! `~/Pictures/LightCraft Library`; a new library starts with the
//! procedural demo photos unless `--no-demo` or files are given. Files and folders on the command
//! line are imported (duplicates are skipped). `--memory` runs an in-memory session that writes
//! nothing (demo photos unless files are given; used by the README showcase scripts).
//! nothing (demo photos unless files are given; used by the README showcase scripts). A library
//! that can't be opened is never replaced silently: the window says why and asks what to do
//! (`lightcraft_ui_egui::panels::library_problem`).
//!
//! `--control <port>` (or `LIGHTCRAFT_CONTROL_PORT`) starts a localhost JSON-lines control server:
//! `{"id":1,"method":"ui.inspect","params":{}}` → `{"id":1,"ok":true,"result":…}`.
@@ -26,6 +28,7 @@ mod control_server;
mod native_menu;
use lightcraft_engine::Session;
use lightcraft_ui_egui::panels::library_problem::LibraryProblem;
use lightcraft_ui_egui::{LightcraftApp, Services, UiState};
/// Reverse-DNS app id: Wayland app id, `.desktop` file and hicolor icon name.
@@ -41,15 +44,16 @@ fn app_icon() -> egui::IconData {
eframe::icon_data::from_png_bytes(png).unwrap_or_default()
}
struct App(LightcraftApp, #[cfg(target_os = "macos")] Option<native_menu::NativeMenu>);
struct App(LightcraftApp, PrefsWriter, #[cfg(target_os = "macos")] Option<native_menu::NativeMenu>);
impl eframe::App for App {
fn logic(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
#[cfg(target_os = "macos")]
if let Some(m) = self.1.as_mut() {
if let Some(m) = self.2.as_mut() {
m.update(&mut self.0, ctx);
}
self.0.logic(ctx);
self.1.tick(&mut self.0, ctx);
}
fn raw_input_hook(&mut self, _ctx: &egui::Context, raw: &mut egui::RawInput) {
self.0.raw_input_hook(raw);
@@ -58,7 +62,9 @@ impl eframe::App for App {
self.0.ui(ui);
}
fn on_exit(&mut self) {
save_prefs(&self.0);
if let Err(e) = self.1.save(&self.0) {
eprintln!("lightcraft: {e}");
}
if let Err(e) = self.0.session.close_library() {
eprintln!("lightcraft: saving the library failed: {e}");
}
@@ -78,23 +84,118 @@ fn config_dir() -> Option<std::path::PathBuf> {
}
}
/// The saved UI state and app settings (`<config>/ui.json`), if any.
fn load_prefs() -> Option<UiState> {
/// The saved UI state and app settings (`<config>/ui.json`), if any, and a warning for the user
/// when the file exists but can't be used (issue #103): a damaged file is kept as
/// `ui.json.corrupt-<unix time>` first, so the next save can't lose the library location in it;
/// one that can't be read at all is not written this session (`keep_file`).
fn load_prefs() -> (Option<UiState>, Option<String>, bool) {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return None;
return (None, None, false);
}
let bytes = std::fs::read(config_dir()?.join("ui.json")).ok()?;
serde_json::from_slice::<UiState>(&bytes).ok().map(UiState::sanitized)
let Some(path) = config_dir().map(|d| d.join("ui.json")) else { return (None, None, false) };
load_prefs_at(&path)
}
fn save_prefs(app: &LightcraftApp) {
if std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return;
fn load_prefs_at(path: &std::path::Path) -> (Option<UiState>, Option<String>, bool) {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return (None, None, false),
Err(e) => {
let msg = format!("The app settings ({}) couldn't be read: {e}. Defaults are used, and the file isn't overwritten.", path.display());
return (None, Some(msg), true);
}
};
match serde_json::from_slice::<UiState>(&bytes) {
Ok(ui) => (Some(ui.sanitized()), None, false),
Err(e) => {
let secs = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let keep = path.with_file_name(format!("ui.json.corrupt-{secs}"));
let kept = std::fs::rename(path, &keep);
let what = match &kept {
Ok(()) => format!("it was kept as {}", keep.display()),
Err(r) => format!("it couldn't be set aside ({r})"),
};
let msg = format!(
"The app settings ({}) are damaged ({e}); {what}. Defaults are used — reopen your library with Settings → Open Library… if it isn't shown.",
path.display()
);
(None, Some(msg), kept.is_err())
}
}
if let Some(d) = config_dir() {
let _ = std::fs::create_dir_all(&d);
if let Ok(bytes) = serde_json::to_vec_pretty(&app.ui) {
let _ = std::fs::write(d.join("ui.json"), bytes);
}
/// Write `bytes` to `path` atomically: a temp file, fsynced, renamed over it (a crash or a full
/// disk leaves the old file or the new one, never a truncated one).
fn write_atomic(path: &std::path::Path, bytes: &[u8]) -> std::io::Result<()> {
use std::io::Write;
let tmp = path.with_extension("json.tmp");
let written = (|| {
let mut f = std::fs::File::create(&tmp)?;
f.write_all(bytes)?;
f.sync_all()
})();
if let Err(e) = written {
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
std::fs::rename(&tmp, path)?;
if let Some(dir) = path.parent()
&& let Ok(d) = std::fs::File::open(dir)
{
let _ = d.sync_all();
}
Ok(())
}
/// Saves `ui.json` (app settings, incl. the library to open at launch): as soon as the library
/// changes, every few seconds when anything else changed, and at exit.
#[derive(Default)]
struct PrefsWriter {
written: Vec<u8>,
library: String,
checked: f64,
/// The last write failed (reported once until a write works).
failing: bool,
/// `ui.json` couldn't be read at launch: never overwrite it this session.
keep_file: bool,
}
impl PrefsWriter {
fn save(&mut self, app: &LightcraftApp) -> Result<(), String> {
if self.keep_file || std::env::var_os("LIGHTCRAFT_NO_PREFS").is_some() {
return Ok(());
}
let Some(d) = config_dir() else { return Ok(()) };
let bytes = serde_json::to_vec_pretty(&app.ui).map_err(|e| e.to_string())?;
if bytes == self.written {
return Ok(());
}
std::fs::create_dir_all(&d)
.and_then(|()| write_atomic(&d.join("ui.json"), &bytes))
.map_err(|e| format!("saving the app settings failed: {e}"))?;
self.written = bytes;
self.library = app.ui.settings.library_path.clone();
Ok(())
}
fn tick(&mut self, app: &mut LightcraftApp, ctx: &egui::Context) {
let now = ctx.input(|i| i.time);
let moved = app.ui.settings.library_path != self.library;
if !moved && now - self.checked < 3.0 {
return;
}
self.checked = now;
match self.save(app) {
Ok(()) => self.failing = false,
Err(e) => {
eprintln!("lightcraft: {e}");
if !self.failing {
app.notices.push(format!("{e}. LightCraft keeps trying."));
}
self.failing = true;
// don't retry every frame
self.library = app.ui.settings.library_path.clone();
}
}
}
}
@@ -215,18 +316,25 @@ fn services() -> Services {
png: Some(Box::new(|img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
})),
// the library is a folder on disk: backed up with the user's other files
backup_library: None,
restore_library: None,
}
}
/// The persistent library session (or an in-memory one with `--memory` / if the library can't open).
fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: bool) -> Session {
let fallback = || if seed_demo { Session::with_demo() } else { Session::new() }.with_fs().with_system_clock();
/// The persistent library session, or with `--memory` an in-memory one (demo photos).
///
/// If the library can't be opened the session is empty and in memory — never seeded with the
/// demo photos, never written anywhere — and the problem is returned: the window then says so
/// and offers Try Again / Choose Another Library… / Continue Without Saving / Quit (issue #100).
fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: bool) -> (Session, Option<LibraryProblem>) {
if in_memory {
return fallback();
return (if seed_demo { Session::with_demo() } else { Session::new() }.with_fs().with_system_clock(), None);
}
let unopened = || Session::new().with_fs().with_system_clock();
let Some(dir) = dir else {
eprintln!("lightcraft: no library location (set --library or LIGHTCRAFT_LIBRARY); running in memory");
return fallback();
eprintln!("lightcraft: no library location (set --library or LIGHTCRAFT_LIBRARY)");
return (unopened(), Some(LibraryProblem::new("", "There is no home folder to keep the library in. Choose a folder for it.")));
};
let t0 = std::time::Instant::now();
let mut s = Session::new().with_fs().with_system_clock();
@@ -240,11 +348,11 @@ fn open_session(in_memory: bool, dir: Option<std::path::PathBuf>, seed_demo: boo
if torn > 0 { ", torn tail repaired" } else { "" },
t0.elapsed().as_secs_f64() * 1000.0
);
s
(s, None)
}
Err(e) => {
eprintln!("lightcraft: can't open library {}: {e}; running in memory", dir.display());
fallback()
eprintln!("lightcraft: can't open library {}: {e}", dir.display());
(unopened(), Some(LibraryProblem::new(dir.to_string_lossy(), e.to_string())))
}
}
}
@@ -293,7 +401,7 @@ fn main() -> eframe::Result {
_ => files.push(a),
}
}
let prefs = load_prefs();
let (prefs, prefs_warning, keep_prefs_file) = load_prefs();
// --library, else the library last opened from Settings, else the default location
let library_dir = library_dir.or_else(|| {
prefs
@@ -321,17 +429,29 @@ fn main() -> eframe::Result {
"LightCraft",
options,
Box::new(move |cc| {
let session = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let (session, problem) = open_session(in_memory, library_dir, seed_demo && files.is_empty());
let mut app = LightcraftApp::new(session, services());
if let Some(ui) = prefs {
app.ui = ui;
}
app.integrated_titlebar = cfg!(target_os = "macos");
app.notices.extend(prefs_warning);
// what's on disk now: only changes are written
let writer = PrefsWriter {
written: serde_json::to_vec_pretty(&app.ui).unwrap_or_default(),
library: app.ui.settings.library_path.clone(),
keep_file: keep_prefs_file,
..Default::default()
};
if let Some(port) = control_port {
let rx = control_server::start(port, cc.egui_ctx.clone());
app = app.with_control(rx);
}
if !files.is_empty() {
if let Some(mut p) = problem {
// imported once the user has chosen where (into the library, or the temporary session)
p.pending_import = files;
app.library_problem = Some(p);
} else if !files.is_empty() {
let _ = app.run("library.import", serde_json::json!({"paths": files}));
app.ui.view = lightcraft_ui_egui::state::ViewMode::PhotoGrid;
}
@@ -341,9 +461,61 @@ fn main() -> eframe::Result {
let menu = (std::env::var_os("LIGHTCRAFT_NO_NATIVE_MENU").is_none()).then(|| native_menu::NativeMenu::install(&mut app, &cc.egui_ctx));
Ok(Box::new(App(
app,
writer,
#[cfg(target_os = "macos")]
menu,
)))
}),
)
}
#[cfg(test)]
#[allow(clippy::unwrap_used)]
mod tests {
use super::*;
fn dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-app-prefs-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
/// Issue #103: a damaged ui.json (which holds the library location) is kept aside and
/// reported, not silently replaced by the defaults.
#[test]
fn damaged_ui_json_is_kept_and_reported() {
let d = dir("damaged");
let path = d.join("ui.json");
std::fs::write(&path, br#"{"settings": {"library_path": "/Volumes/Photos/Lib"#).unwrap();
let (ui, warning, keep) = load_prefs_at(&path);
assert!(ui.is_none() && !keep);
let warning = warning.unwrap();
assert!(warning.contains("damaged") && warning.contains("ui.json.corrupt-"), "{warning}");
assert!(!path.exists());
let kept: Vec<_> = std::fs::read_dir(&d).unwrap().map(|e| e.unwrap().file_name().to_string_lossy().to_string()).collect();
assert!(kept.len() == 1 && kept[0].starts_with("ui.json.corrupt-"), "{kept:?}");
// a missing file is just "no settings yet"
assert_eq!(load_prefs_at(&path).1, None);
let _ = std::fs::remove_dir_all(&d);
}
#[test]
fn ui_json_is_written_atomically() {
let d = dir("atomic");
let path = d.join("ui.json");
std::fs::write(&path, b"old").unwrap();
write_atomic(&path, b"{\"new\": true}").unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"{\"new\": true}");
assert!(!d.join("ui.json.tmp").exists());
// a failed write leaves the old file
let missing = d.join("gone").join("ui.json");
assert!(write_atomic(&missing, b"x").is_err());
assert_eq!(std::fs::read(&path).unwrap(), b"{\"new\": true}");
let ui = UiState::default();
write_atomic(&path, &serde_json::to_vec_pretty(&ui).unwrap()).unwrap();
let (loaded, warning, _) = load_prefs_at(&path);
assert!(loaded.is_some() && warning.is_none());
let _ = std::fs::remove_dir_all(&d);
}
}
+196 -26
View File
@@ -1,8 +1,9 @@
//! Crash-safe catalog persistence: an append-only op log plus periodic snapshots.
//!
//! Files (in a [`Store`]):
//! - `catalog.snap` — `{"format":"lightcraft-catalog","version":1,"seq":N,"catalog":{…}}`, the
//! state after op `N`; replaced atomically (temp + fsync + rename).
//! - `catalog.snap` — `{"format":"lightcraft-catalog","version":V,"seq":N,"catalog":{…}}`, the
//! state after op `N` in catalog format `V` ([`VERSION`]); replaced atomically (temp + fsync +
//! rename).
//! - `catalog.log` — JSON lines, one per op applied after the snapshot:
//! `{"seq":N,"crc":C,"op":{…}}` where `C` is the CRC-32 of the op's JSON text. Appends are
//! fsynced before [`Journal::append`] returns.
@@ -10,6 +11,12 @@
//! Loading = snapshot + replay of the records with `seq > snapshot seq`. Recovery rules:
//! - a **torn final record** (crash mid-append: truncated line or bad CRC at the end) is dropped
//! and the file is cut back to the last good record, so later appends start on a clean line;
//! - a **failed append** (disk full, a dropped network share) may leave part of its batch in the
//! file: [`Journal::append`] cuts the log back to the last whole record before returning the
//! error (or, if that fails too, before the next append), so the retry — the same ops with the
//! same `seq`s — starts on a clean line. Logs written by builds without that cut-back hold the
//! fragment glued to the first retried record on one line; the loader recognises the whole,
//! CRC-valid record at the end of such a line and skips only the fragment;
//! - records already covered by the snapshot (crash between writing the snapshot and resetting
//! the log) are skipped;
//! - a bad record **followed by good ones** is real damage: replay stops there, the log is kept
@@ -27,6 +34,32 @@
//!
//! At most one snapshot is in flight; a synchronous [`Journal::snapshot`] and dropping the journal
//! first wait for it, so an older snapshot can never replace a newer one.
//!
//! # Format versions
//!
//! The snapshot's `version` is the catalog format, and it covers the log beside it: the log has
//! no header of its own, and is only ever appended to under a snapshot of this build's format.
//!
//! | Version | Written by | Adds |
//! |---|---|---|
//! | (none) | early builds: a log without a snapshot | |
//! | 1 | up to v0.2.0 | |
//! | 2 | after v0.2.0 | `Op::SetBrowsed`, `Catalog.browsed`, `Photo.local_baseline` |
//!
//! Rules:
//! - **Bump [`VERSION`]** (and add a row above) in the change that adds an [`Op`] variant or a
//! serialized field to the catalog, a photo, an album, a stack or develop settings that an older
//! build would misread or silently drop. New fields must deserialize from older data
//! (`#[serde(default)]`), so newer builds keep reading every older format.
//! - **Opening an older format upgrades it**: right after loading, [`Journal::open`] writes a
//! snapshot in this build's format (also for a new library), before anything is appended — so a
//! library is never a mix, and older builds refuse it from then on instead of reading part of it.
//! [`LoadReport::upgraded_from`] says when that happened.
//! - **A newer format is refused, untouched**: a snapshot whose `version` is above [`VERSION`],
//! or a log record whose CRC matches (so it is exactly what was written) but whose op doesn't
//! parse, fails [`Journal::open`] with [`CatalogError::Newer`] before any file is modified — it
//! is never mistaken for a torn tail or damage. (Builds up to v0.2.0 refuse a v2 snapshot with
//! "unsupported format".)
use crate::store::Store;
use crate::{Catalog, CatalogError, Op, Result};
@@ -34,7 +67,9 @@ use crate::{Catalog, CatalogError, Op, Result};
pub const SNAPSHOT: &str = "catalog.snap";
pub const LOG: &str = "catalog.log";
const FORMAT: &str = "lightcraft-catalog";
const VERSION: u32 = 1;
/// The catalog format this build writes (and the newest it reads). See the module docs →
/// *Format versions*; bump it whenever an [`Op`] variant or a serialized field is added.
pub const VERSION: u32 = 2;
/// When [`Journal::wants_snapshot`] says it's time to compact the log.
#[derive(Clone, Copy, Debug)]
@@ -62,10 +97,16 @@ pub struct LoadReport {
pub failed: usize,
/// Bytes of a torn final record that were dropped.
pub torn_bytes: u64,
/// Partial records left mid-log by a failed append that was retried (written by builds that
/// didn't cut the log back after a failed write): skipped, the retried records were replayed.
pub torn_fragments: usize,
/// The damaged log was preserved under this name.
pub damaged: Option<String>,
/// No catalog files existed (a new library).
pub created: bool,
/// The library was in an older catalog format (`0` = before format versions: a log without a
/// snapshot) and was rewritten in this version's format ([`VERSION`]).
pub upgraded_from: Option<u32>,
}
/// Where persistence time goes (reported by `library.info` → `persistence`; printed to stderr
@@ -140,6 +181,9 @@ pub struct Journal {
pending: Option<Pending>,
/// After a failed background snapshot: don't retry before the log has this many records.
retry_at_records: u64,
/// A failed append may have left part of its records after `log_bytes`, and cutting them off
/// failed too: the next append cuts the log back to `log_bytes` first.
tail_dirty: bool,
}
/// A background snapshot in flight.
@@ -168,18 +212,58 @@ pub fn encode_record(seq: u64, op: &Op) -> String {
format!("{{\"seq\":{seq},\"crc\":{crc},\"op\":{body}}}")
}
/// Parse one log line (without the newline). `None` if it's malformed or the CRC doesn't match.
/// Parse one log line (without the newline). `None` if it's malformed, the CRC doesn't match, or
/// the op is one this version doesn't know (see [`Journal::open`]).
pub fn decode_record(line: &str) -> Option<(u64, Op)> {
let rest = line.strip_prefix("{\"seq\":")?;
let (seq, rest) = rest.split_once(",\"crc\":")?;
let (crc, rest) = rest.split_once(",\"op\":")?;
let body = rest.strip_suffix('}')?;
let seq: u64 = seq.parse().ok()?;
let crc: u32 = crc.parse().ok()?;
if crc32fast::hash(body.as_bytes()) != crc {
return None;
match decode_line(line) {
Line::Record(seq, op) => Some((seq, op)),
Line::Newer(_) | Line::Bad => None,
}
serde_json::from_str(body).ok().map(|op| (seq, op))
}
/// One log line, decoded.
enum Line {
Record(u64, Op),
/// The CRC matches — the line is exactly what was written — but the op doesn't parse: it
/// was written by a newer LightCraft (an op or field this version doesn't know), not damaged.
Newer(u64),
/// Malformed or a CRC mismatch: torn or damaged.
Bad,
}
fn decode_line(line: &str) -> Line {
let parts = || {
let rest = line.strip_prefix("{\"seq\":")?;
let (seq, rest) = rest.split_once(",\"crc\":")?;
let (crc, rest) = rest.split_once(",\"op\":")?;
let body = rest.strip_suffix('}')?;
Some((seq.parse::<u64>().ok()?, crc.parse::<u32>().ok()?, body))
};
let Some((seq, crc, body)) = parts() else { return Line::Bad };
if crc32fast::hash(body.as_bytes()) != crc {
return Line::Bad;
}
match serde_json::from_str(body) {
Ok(op) => Line::Record(seq, op),
Err(_) => Line::Newer(seq),
}
}
/// The error for a library written by a newer LightCraft.
fn newer(what: String) -> CatalogError {
CatalogError::Newer(format!("{what}; this version reads catalog format v{VERSION} and older"))
}
/// A line that doesn't decode but ends with a whole record: a fragment of a failed append followed
/// by the record that retried it (the retry re-encodes the same ops with the same `seq`s). Returns
/// the fragment's length and the record, if that record continues the log (`seq <= last + 1`).
/// The record's CRC must match, so damage is never mistaken for it.
fn resync(line: &[u8], last: u64) -> Option<(usize, (u64, Op))> {
const START: &[u8] = b"{\"seq\":";
(1..line.len().saturating_sub(START.len() - 1)).filter(|&i| line[i..].starts_with(START)).find_map(|i| {
let rec = std::str::from_utf8(&line[i..]).ok().map(|l| l.trim_end_matches('\r')).and_then(decode_record)?;
(rec.0 <= last + 1).then_some((i, rec))
})
}
/// Write `catalog.snap` for the state after op `seq`, streaming the JSON into the store (no
@@ -205,10 +289,16 @@ fn write_snapshot(store: &mut dyn Store, seq: u64, catalog: &Catalog) -> std::io
Ok(SnapshotTiming { serialize_ms, write_sync_ms: (ms_since(t0) - serialize_ms).max(0.0), bytes, ..Default::default() })
}
/// Just the identification of `catalog.snap` (read before the catalog itself).
#[derive(serde::Deserialize)]
struct SnapFile {
struct SnapHeader {
format: String,
version: u32,
}
/// The whole `catalog.snap` (after [`SnapHeader`] was checked; `format` and `version` are ignored).
#[derive(serde::Deserialize)]
struct SnapFile {
seq: u64,
catalog: Catalog,
}
@@ -221,15 +311,20 @@ impl Journal {
let snap = store.read(SNAPSHOT).map_err(io)?;
let log = store.read(LOG).map_err(io)?;
report.created = snap.is_none() && log.is_none();
let (mut catalog, snapshot_seq) = match snap {
let (mut catalog, snapshot_seq, snapshot_version) = match snap {
Some(bytes) => {
let s: SnapFile = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
if s.format != FORMAT || s.version > VERSION {
return Err(CatalogError::Corrupt(format!("{SNAPSHOT}: unsupported format {} v{}", s.format, s.version)));
// the header first: a newer snapshot may not parse as this version's catalog
let h: SnapHeader = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
if h.format != FORMAT {
return Err(CatalogError::Corrupt(format!("{SNAPSHOT}: not a LightCraft catalog (format {:?})", h.format)));
}
(s.catalog, s.seq)
if h.version > VERSION {
return Err(newer(format!("{SNAPSHOT} is catalog format v{}", h.version)));
}
let s: SnapFile = serde_json::from_slice(&bytes).map_err(|e| CatalogError::Corrupt(format!("{SNAPSHOT}: {e}")))?;
(s.catalog, s.seq, Some(h.version))
}
None => (Catalog::new(), 0),
None => (Catalog::new(), 0, None),
};
report.snapshot_seq = snapshot_seq;
let mut j = Journal {
@@ -242,6 +337,7 @@ impl Journal {
stats: PersistStats::default(),
pending: None,
retry_at_records: 0,
tail_dirty: false,
};
let log = log.unwrap_or_default();
@@ -261,7 +357,22 @@ impl Journal {
good_end = next;
continue;
}
match line.and_then(decode_record) {
let mut record = line.and_then(decode_record);
if record.is_none()
&& let Some((at, r)) = resync(&log[pos..line_end], j.seq)
{
// the torn start of a failed append, then the retried records (older builds
// appended them right after the fragment)
log::warn!("catalog log: skipped a partial record ({at} bytes) left by a failed write");
report.torn_fragments += 1;
record = Some(r);
}
if record.is_none()
&& let Some(Line::Newer(seq)) = line.map(decode_line)
{
return Err(newer(format!("{LOG} record {seq} holds a change this version doesn't know")));
}
match record {
Some((seq, op)) if seq <= j.seq => {
// already in the snapshot
let _ = op;
@@ -277,7 +388,15 @@ impl Journal {
}
_ => {
// bad record (or a gap): torn tail if nothing good follows, else damage
let rest_has_good = log[next..].split(|b| *b == b'\n').any(|l| std::str::from_utf8(l).ok().and_then(decode_record).is_some());
let mut rest_has_good = false;
for l in log[next..].split(|b| *b == b'\n') {
match std::str::from_utf8(l).map(|l| decode_line(l.trim_end_matches('\r'))) {
Ok(Line::Record(..)) => rest_has_good = true,
// refused before anything is modified
Ok(Line::Newer(seq)) => return Err(newer(format!("{LOG} record {seq} holds a change this version doesn't know"))),
Ok(Line::Bad) | Err(_) => {}
}
}
if rest_has_good {
damaged_at = Some(pos);
}
@@ -301,10 +420,34 @@ impl Journal {
report.torn_bytes = (log.len() - good_end) as u64;
log::warn!("catalog log: dropped a torn final record ({} bytes)", report.torn_bytes);
j.store.truncate(LOG, good_end as u64).map_err(io)?;
j.log_bytes = good_end as u64;
} else if needs_newline {
j.store.append(LOG, b"\n").map_err(io)?;
j.log_bytes = good_end as u64 + 1;
} else {
j.log_bytes = good_end as u64;
}
}
// Bring an older (or new, or snapshot-less) library to this version's format before
// anything is appended: a snapshot in this format makes older builds refuse the library
// instead of misreading ops or dropping fields they don't know.
if report.created {
// tiny: written directly, not counted as a compaction
let empty = format!("{{\"format\":\"{FORMAT}\",\"version\":{VERSION},\"seq\":0,\"catalog\":{}}}\n", catalog.to_snapshot());
if let Err(e) = j.store.write_atomic(SNAPSHOT, empty.as_bytes()) {
log::warn!("catalog: can't write the new library's snapshot: {e}");
}
} else if report.damaged.is_none() && snapshot_version != Some(VERSION) {
report.upgraded_from = Some(snapshot_version.unwrap_or(0));
match j.snapshot(&catalog) {
Ok(()) => {
if let Some(v) = report.upgraded_from {
log::info!("catalog: upgraded from format v{v} to v{VERSION}");
}
}
// nothing lost: the old files are intact and still load; retried on next open
Err(e) => log::warn!("catalog: can't write the format v{VERSION} snapshot: {e}"),
}
j.log_bytes = good_end as u64;
}
catalog.revision = 0;
Ok((j, catalog, report))
@@ -323,7 +466,20 @@ impl Journal {
buf.push_str(&encode_record(seq, op));
buf.push('\n');
}
self.store.append(LOG, buf.as_bytes()).map_err(io)?;
if self.tail_dirty {
self.store.truncate(LOG, self.log_bytes).map_err(io)?;
self.tail_dirty = false;
}
if let Err(e) = self.store.append(LOG, buf.as_bytes()) {
// Part of the batch may be in the file. Cut it back to the last whole record, so the
// retry (same seqs) starts on a clean line instead of after a fragment that would
// read back as damage.
if let Err(t) = self.store.truncate(LOG, self.log_bytes) {
log::error!("catalog: can't cut the log back after a failed append: {t}");
self.tail_dirty = true;
}
return Err(io(e));
}
if let Some(p) = self.pending.as_mut() {
p.tail.extend_from_slice(buf.as_bytes());
p.tail_records += ops.len() as u64;
@@ -343,12 +499,22 @@ impl Journal {
/// Write a snapshot of `catalog` (which must reflect every appended op) and reset the log.
pub fn snapshot(&mut self, catalog: &Catalog) -> Result<()> {
self.snapshot_with_unlogged(catalog, 0)
}
/// [`Journal::snapshot`] of a catalog that also reflects `unlogged` ops applied after the
/// last appended one but never written to the log (their append failed). The snapshot counts
/// them (it holds the state after op `seq + unlogged`), so they are saved and must not be
/// appended afterwards. Once the snapshot is durable [`Journal::seq`] includes them, even if
/// resetting the log then fails (the error is still returned).
pub fn snapshot_with_unlogged(&mut self, catalog: &Catalog, unlogged: u64) -> Result<()> {
// never two snapshot writers; the one in flight is older, so it must land first
if let Err(e) = self.wait() {
log::warn!("catalog: background snapshot failed ({e}); writing one now");
}
let t0 = web_time::Instant::now();
let mut timing = match write_snapshot(self.store.as_mut(), self.seq, catalog) {
let seq = self.seq + unlogged;
let mut timing = match write_snapshot(self.store.as_mut(), seq, catalog) {
Ok(t) => t,
Err(e) => {
// nothing lost (the log is whole); don't retry on every append
@@ -357,6 +523,9 @@ impl Journal {
return Err(io(e));
}
};
// the snapshot is durable: it holds every op up to `seq`
self.seq = seq;
self.snapshot_seq = seq;
let t2 = web_time::Instant::now();
// A crash here leaves old records in the log; they are skipped by seq on load.
self.store.write_atomic(LOG, b"").map_err(io)?;
@@ -365,10 +534,10 @@ impl Journal {
timing.blocking_ms = timing.total_ms;
timing.records = self.log_records;
self.record_snapshot(timing);
self.snapshot_seq = self.seq;
self.log_records = 0;
self.log_bytes = 0;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
@@ -459,6 +628,7 @@ impl Journal {
self.log_records = p.tail_records;
self.log_bytes = p.tail.len() as u64;
self.retry_at_records = 0;
self.tail_dirty = false;
Ok(())
}
+17
View File
@@ -5,6 +5,11 @@
//! snapshot — crash-safe and diff-friendly (see [`journal`]);
//! - **undo/redo**: the engine keeps inverse ops;
//! - **determinism**: replaying the log reproduces the state exactly (property-tested).
//!
//! **Catalog format version** ([`journal::VERSION`], see [`journal`] → *Format versions*): adding
//! an [`Op`] variant or a serialized field means bumping it. Newer builds read every older format
//! (and upgrade it on open); older builds refuse a newer library with [`CatalogError::Newer`]
//! instead of reading part of it.
#![forbid(unsafe_code)]
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -12,6 +17,7 @@ pub mod dates;
pub mod journal;
pub mod keywords;
pub mod local;
pub mod lock;
pub mod model;
pub mod query;
pub mod rules;
@@ -27,6 +33,7 @@ pub use journal::{Journal, LoadReport, PersistStats, SnapshotPolicy, SnapshotTim
pub use keywords::KeywordNode;
use lightcraft_develop::DevelopSettings;
pub use local::{DEFAULT_FORGET_DAYS, ForgetPlan, folder_of};
pub use lock::{LibraryLock, LockError, LockOwner};
pub use model::*;
pub use query::{DateGroup, Filter, RatingOp, Sort, SortKey};
pub use rules::{Match, Rule, RuleSet};
@@ -47,6 +54,10 @@ pub enum CatalogError {
Corrupt(String),
#[error("catalog storage: {0}")]
Io(String),
/// The library was written by a newer LightCraft (a newer catalog format, or a change this
/// version doesn't know). Nothing was read into the session and nothing was modified.
#[error("this library was written by a newer version of LightCraft ({0}); update LightCraft to open it. The library was left unchanged.")]
Newer(String),
}
pub type Result<T> = std::result::Result<T, CatalogError>;
@@ -644,6 +655,12 @@ mod tests;
#[cfg(test)]
mod tests_background;
#[cfg(test)]
mod tests_format_version;
#[cfg(test)]
mod tests_journal;
#[cfg(test)]
mod tests_local;
#[cfg(test)]
mod tests_lock;
#[cfg(test)]
mod tests_torn_append;
+140
View File
@@ -0,0 +1,140 @@
//! One process per library: an exclusive OS lock on `<library>/catalog.lock`, held for as long as
//! the library is open (issue #99).
//!
//! Two processes on the same library (the app and `lightcraft-cli --library`, two app instances,
//! two computers on a shared folder) would each keep their own catalog and `seq` counter, and
//! whichever wrote the last snapshot would silently drop the other's edits. So opening a library
//! takes the lock first, and a second opener gets [`LockError::InUse`] instead.
//!
//! - The lock is the OS's (`flock` on Unix, `LockFileEx` on Windows, via [`std::fs::File::try_lock`]):
//! it is released when the [`LibraryLock`] is dropped and, after a crash or a kill, by the OS when
//! the process ends. A `catalog.lock` file left behind is therefore never "stale": only a lock
//! that's actually held refuses the open.
//! - Who holds it (process id, computer, program, version) is written to `catalog.lock.owner`, for
//! the error message. It's a separate file because a Windows lock also blocks reading the
//! locked file. It's only a hint: it may be missing or left over from a crash.
//! - File systems without lock support (some network shares): the library opens unlocked, with a
//! warning — refusing it would lock the user out of their own library. Locks over SMB/NFS are
//! advisory and depend on the server; the owner file is then the only hint.
//! - Browser storage (OPFS) has no file locks: see the web host for its own guard.
use std::fs::{File, OpenOptions, TryLockError};
use std::io;
use std::path::{Path, PathBuf};
/// The lock file in the library directory.
pub const LOCK: &str = "catalog.lock";
/// Who holds the lock (JSON, see [`LockOwner`]).
pub const OWNER: &str = "catalog.lock.owner";
/// Who has the library open (as written in [`OWNER`]).
#[derive(Clone, Debug, Default, PartialEq, serde::Serialize, serde::Deserialize)]
#[serde(default)]
pub struct LockOwner {
pub pid: u32,
/// Computer name, if known.
pub host: String,
/// The program (`LightCraft`, `lightcraft-cli`…) and its version.
pub program: String,
pub version: String,
/// When it opened the library (seconds since 1970).
pub since: u64,
}
impl LockOwner {
/// This process, as `program`.
pub fn this_process(program: &str) -> LockOwner {
LockOwner {
pid: std::process::id(),
host: host_name(),
program: program.to_string(),
version: env!("CARGO_PKG_VERSION").to_string(),
since: web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0),
}
}
/// "LightCraft 0.3.0 (process 123 on studio-mac)".
pub fn describe(&self) -> String {
let program = if self.program.is_empty() { "another LightCraft" } else { self.program.as_str() };
let version = if self.version.is_empty() { String::new() } else { format!(" {}", self.version) };
let host = if self.host.is_empty() {
String::new()
} else if self.host == host_name() {
" on this computer".to_string()
} else {
format!(" on {}", self.host)
};
format!("{program}{version} (process {}{host})", self.pid)
}
}
/// The computer's name, best effort (empty if unknown).
fn host_name() -> String {
let env = ["COMPUTERNAME", "HOSTNAME"].iter().find_map(|k| std::env::var(k).ok().filter(|h| !h.trim().is_empty()));
env.or_else(|| std::fs::read_to_string("/etc/hostname").ok()).map(|h| h.trim().to_string()).filter(|h| !h.is_empty()).unwrap_or_default()
}
#[derive(Debug, thiserror::Error)]
pub enum LockError {
/// Another process has the library open.
#[error("This library is already open in {}. Close it there first.", .0.as_ref().map(LockOwner::describe).unwrap_or_else(|| "another LightCraft program".into()))]
InUse(Option<LockOwner>),
}
/// The exclusive lock on a library directory; released on drop (and by the OS if the process
/// dies).
#[derive(Debug)]
pub struct LibraryLock {
dir: PathBuf,
/// `None` when the file system can't lock (the library is open unlocked).
_file: Option<File>,
held: bool,
}
impl LibraryLock {
/// Lock `dir` (which must exist) for this process. Fails only when another process holds it.
pub fn acquire(dir: &Path, program: &str) -> Result<LibraryLock, LockError> {
let unlocked = |why: io::Error| {
log::warn!("library {}: can't lock {LOCK} ({why}); opening it without protection against a second program", dir.display());
LibraryLock { dir: dir.to_path_buf(), _file: None, held: false }
};
let file = match OpenOptions::new().read(true).write(true).create(true).truncate(false).open(dir.join(LOCK)) {
Ok(f) => f,
Err(e) => return Ok(unlocked(e)),
};
match file.try_lock() {
Ok(()) => {}
Err(TryLockError::WouldBlock) => {
let owner = std::fs::read(dir.join(OWNER)).ok().and_then(|b| serde_json::from_slice::<LockOwner>(&b).ok());
return Err(LockError::InUse(owner));
}
Err(TryLockError::Error(e)) => return Ok(unlocked(e)),
}
let owner = LockOwner::this_process(program);
if let Ok(json) = serde_json::to_vec_pretty(&owner)
&& let Err(e) = std::fs::write(dir.join(OWNER), json)
{
log::warn!("library {}: can't write {OWNER}: {e}", dir.display());
}
Ok(LibraryLock { dir: dir.to_path_buf(), _file: Some(file), held: true })
}
/// The OS lock is held (`false`: the file system can't lock; opened unprotected).
pub fn held(&self) -> bool {
self.held
}
pub fn dir(&self) -> &Path {
&self.dir
}
}
impl Drop for LibraryLock {
/// Remove the owner note while still holding the lock (so it can't delete the next holder's),
/// then the file handle closes and the OS releases the lock.
fn drop(&mut self) {
if self.held {
let _ = std::fs::remove_file(self.dir.join(OWNER));
}
}
}
+19 -5
View File
@@ -28,9 +28,12 @@ pub trait Store: Send {
self.write_atomic(name, &buf)?;
Ok(buf.len() as u64)
}
/// Append and make durable (fsync) before returning.
/// Append and make durable (fsync) before returning. On an error, part of `data` may have
/// been written (a full disk, a dropped network share): the journal cuts the file back to its
/// previous length with [`Store::truncate`] before appending again.
fn append(&mut self, name: &str, data: &[u8]) -> io::Result<()>;
/// Cut a file to `len` bytes (drop a torn tail before appending again).
/// Cut a file to `len` bytes (drop a torn tail before appending again). A missing file is
/// left missing.
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()>;
/// Human-readable location (diagnostics).
fn describe(&self) -> String;
@@ -132,15 +135,26 @@ impl Store for FsStore {
self.appender = Some((name.to_string(), f));
}
let Some((_, f)) = self.appender.as_mut() else { return Err(io::Error::other("appender not open")) };
f.write_all(data)?;
f.sync_data()
let written = f.write_all(data).and_then(|()| f.sync_data());
if written.is_err() {
// The handle may be dead (a network share that reconnected, a removed volume): the next
// attempt reopens the file. Part of `data` may be in the file; the caller cuts it back
// ([`Store::truncate`]) before appending again.
self.appender = None;
}
written
}
fn truncate(&mut self, name: &str, len: u64) -> io::Result<()> {
if self.appender.as_ref().is_some_and(|(n, _)| n == name) {
self.appender = None;
}
let f = std::fs::OpenOptions::new().write(true).open(self.path(name))?;
let f = match std::fs::OpenOptions::new().write(true).open(self.path(name)) {
Ok(f) => f,
// nothing to cut (an append that failed before creating the file)
Err(e) if e.kind() == io::ErrorKind::NotFound => return Ok(()),
Err(e) => return Err(e),
};
f.set_len(len)?;
f.sync_all()
}
+2 -1
View File
@@ -180,7 +180,8 @@ fn failed_background_snapshot_loses_nothing_and_retries_later() {
wait_written(&j);
assert!(j.poll().is_err());
assert_eq!((j.snapshot_seq(), j.stats().failed_snapshots), (0, 1));
assert!(g.files.get(SNAPSHOT).is_none());
// still the (empty) snapshot written when the library was created
assert_eq!(g.snapshot_seq_on_disk(), Some(0));
assert_recovers(&g, &live);
// no retry storm: the next attempt waits for another `max_records` records
assert!(!j.wants_snapshot());
+186
View File
@@ -0,0 +1,186 @@
//! Issue #102: catalog format versions. Older formats load and are upgraded; a newer format is
//! refused with a clear error and left untouched (never read in part, truncated or set aside).
use crate::journal::{LOG, SNAPSHOT, VERSION, encode_record};
use crate::*;
fn photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// A catalog with two photos and the log line for a third op on top of it.
fn legacy_parts() -> (Catalog, String, Catalog) {
let mut c = Catalog::new();
for _ in 0..2 {
let op = photo(&mut c);
c.apply(op).unwrap();
}
let base = c.clone();
let id = c.photos().next().unwrap().id;
let op = Op::SetRating { id, rating: 4 };
c.apply(op.clone()).unwrap();
(base, format!("{}\n", encode_record(3, &op)), c)
}
fn snapshot_version(m: &MemStore) -> u64 {
let v: serde_json::Value = serde_json::from_slice(&m.get(SNAPSHOT).unwrap()).unwrap();
v["version"].as_u64().unwrap()
}
#[test]
fn v1_library_loads_and_is_upgraded() {
let (base, log, full) = legacy_parts();
let m = MemStore::new();
m.set(SNAPSHOT, format!("{{\"format\":\"lightcraft-catalog\",\"version\":1,\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot()).into_bytes());
m.set(LOG, log.into_bytes());
let (mut j, c, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(c.to_snapshot(), full.to_snapshot());
assert_eq!((r.replayed, r.upgraded_from), (1, Some(1)));
// rewritten in this format before anything is appended
assert_eq!(snapshot_version(&m), u64::from(VERSION));
assert!(m.get(LOG).unwrap().is_empty());
assert_eq!(j.seq(), 3);
j.append(&[Op::SetLabelName { label: ColorLabel::ALL[0], name: Some("x".into()) }]).unwrap();
drop(j);
let (_, _, r) = Journal::open(Box::new(m)).unwrap();
assert_eq!((r.replayed, r.upgraded_from), (1, None));
}
#[test]
fn versionless_log_only_library_is_upgraded() {
let mut c = Catalog::new();
let op = photo(&mut c);
c.apply(op.clone()).unwrap();
let m = MemStore::new();
m.set(LOG, format!("{}\n", encode_record(1, &op)).into_bytes());
let (_, loaded, r) = Journal::open(Box::new(m.clone())).unwrap();
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
assert_eq!(r.upgraded_from, Some(0));
assert_eq!(snapshot_version(&m), u64::from(VERSION));
}
/// A new library gets a snapshot in this format at once, so an older build never sees a bare log
/// it would read in part.
#[test]
fn new_library_is_written_in_this_format() {
let m = MemStore::new();
let (_, _, r) = Journal::open(Box::new(m.clone())).unwrap();
assert!(r.created);
assert_eq!(r.upgraded_from, None);
assert_eq!(snapshot_version(&m), u64::from(VERSION));
}
fn assert_refused_untouched(m: &MemStore) {
let before = m.files.lock().unwrap().clone();
let e = Journal::open(Box::new(m.clone())).err().expect("refused");
assert!(matches!(e, CatalogError::Newer(_)), "{e:?}");
let msg = e.to_string();
assert!(msg.contains("newer version of LightCraft") && msg.contains("left unchanged"), "{msg}");
assert_eq!(*m.files.lock().unwrap(), before, "nothing modified, nothing renamed");
}
#[test]
fn newer_snapshot_is_refused_untouched() {
let (base, log, _) = legacy_parts();
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":2,\"catalog\":{}}}\n", VERSION + 1, base.to_snapshot()).into_bytes(),
);
m.set(LOG, log.into_bytes());
assert_refused_untouched(&m);
// even when its catalog doesn't parse as this version's
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":2,\"catalog\":{{\"photos\":7}}}}\n", VERSION + 1).into_bytes(),
);
assert_refused_untouched(&m);
let e = Journal::open(Box::new(m)).err().unwrap().to_string();
assert!(e.contains(&format!("v{}", VERSION + 1)), "{e}");
}
/// A log record with a valid CRC whose op this version doesn't know: refused, not taken for a
/// torn tail (truncated) or damage (renamed) — wherever it is in the log.
#[test]
fn unknown_op_in_the_log_is_refused_untouched() {
let (base, log, _) = legacy_parts();
let future = |seq: u64| {
let body = r#"{"op":"fromTheFuture","id":1,"what":"something new"}"#;
format!("{{\"seq\":{seq},\"crc\":{},\"op\":{body}}}\n", crc32fast::hash(body.as_bytes()))
};
let snap = format!("{{\"format\":\"lightcraft-catalog\",\"version\":{VERSION},\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot());
let id = base.photos().next().unwrap().id;
let after = format!("{}\n", encode_record(5, &Op::SetRating { id, rating: 1 }));
for log in [format!("{log}{}", future(4)), format!("{log}{}{after}", future(4)), format!("{}{log}", future(3))] {
let m = MemStore::new();
m.set(SNAPSHOT, snap.clone().into_bytes());
m.set(LOG, log.into_bytes());
assert_refused_untouched(&m);
}
// a known op with a field of an unknown shape is "newer" too
let body = r#"{"op":"setRating","id":1,"rating":"five stars"}"#;
let m = MemStore::new();
m.set(SNAPSHOT, snap.into_bytes());
m.set(LOG, format!("{log}{{\"seq\":4,\"crc\":{},\"op\":{body}}}\n", crc32fast::hash(body.as_bytes())).into_bytes());
assert_refused_untouched(&m);
}
/// A CRC mismatch is still damage / a torn tail (not "newer").
#[test]
fn bad_crc_is_still_a_torn_tail() {
let (base, log, _) = legacy_parts();
let m = MemStore::new();
m.set(
SNAPSHOT,
format!("{{\"format\":\"lightcraft-catalog\",\"version\":{VERSION},\"seq\":2,\"catalog\":{}}}\n", base.to_snapshot()).into_bytes(),
);
let body = r#"{"op":"fromTheFuture"}"#;
m.set(LOG, format!("{log}{{\"seq\":4,\"crc\":1,\"op\":{body}}}\n").into_bytes());
let (_, _, r) = Journal::open(Box::new(m)).unwrap();
assert!(r.torn_bytes > 0 && r.damaged.is_none(), "{r:?}");
}
/// Guard for the version rule (see `journal` → *Format versions*): adding an [`Op`] variant breaks
/// this match. When it does, bump `journal::VERSION`, add a row to the format table, and add the
/// variant here under the new version.
#[test]
fn op_variants_are_versioned() {
fn since(op: &Op) -> u32 {
match op {
Op::AddPhoto { .. }
| Op::RemovePhoto { .. }
| Op::SetRating { .. }
| Op::SetFlag { .. }
| Op::SetLabel { .. }
| Op::SetDevelop { .. }
| Op::SetMeta { .. }
| Op::SetDeleted { .. }
| Op::SetLocal { .. }
| Op::SetVersions { .. }
| Op::SetHistory { .. }
| Op::PushHistory { .. }
| Op::AddAlbum { .. }
| Op::RemoveAlbum { .. }
| Op::RenameAlbum { .. }
| Op::MoveAlbum { .. }
| Op::SetAlbumPhotos { .. }
| Op::SetAlbumCover { .. }
| Op::SetAlbumRules { .. }
| Op::AddStack { .. }
| Op::RemoveStack { .. }
| Op::SetStack { .. }
| Op::SetCaptured { .. }
| Op::SetAnalysis { .. }
| Op::SetFile { .. }
| Op::Relink { .. }
| Op::SetContent { .. }
| Op::SetLabelName { .. }
| Op::Batch { .. } => 1,
Op::SetBrowsed { .. } => 2,
}
}
let newest = since(&Op::SetBrowsed { folder: String::new(), at: None });
assert_eq!(newest, VERSION, "the newest op's version must be the current format version");
}
+6 -1
View File
@@ -283,7 +283,12 @@ fn streamed_snapshot_keeps_the_on_disk_format() {
let (mut j, c, _) = open(&m);
assert_eq!(c.to_snapshot(), full.to_snapshot());
j.snapshot(&c).unwrap();
let legacy = format!("{{\"format\":\"lightcraft-catalog\",\"version\":1,\"seq\":{},\"catalog\":{}}}\n", j.seq(), c.to_snapshot());
let legacy = format!(
"{{\"format\":\"lightcraft-catalog\",\"version\":{},\"seq\":{},\"catalog\":{}}}\n",
crate::journal::VERSION,
j.seq(),
c.to_snapshot()
);
assert_eq!(String::from_utf8(m.get(SNAPSHOT).unwrap()).unwrap(), legacy);
assert_eq!(j.stats().last_snapshot.bytes, legacy.len() as u64);
+93
View File
@@ -0,0 +1,93 @@
//! Issue #99: one process per library.
use std::path::PathBuf;
use crate::lock::{LOCK, OWNER};
use crate::*;
fn temp_dir(tag: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("lc-lock-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn second_opener_is_refused_until_the_first_lets_go() {
let dir = temp_dir("twice");
let first = LibraryLock::acquire(&dir, "LightCraft").unwrap();
assert!(first.held());
let owner: LockOwner = serde_json::from_slice(&std::fs::read(dir.join(OWNER)).unwrap()).unwrap();
assert_eq!((owner.pid, owner.program.as_str()), (std::process::id(), "LightCraft"));
let e = LibraryLock::acquire(&dir, "lightcraft-cli").unwrap_err();
let LockError::InUse(Some(who)) = &e else { panic!("{e:?}") };
assert_eq!(who.pid, std::process::id());
let msg = e.to_string();
assert!(msg.contains("already open in LightCraft") && msg.contains(&format!("process {}", std::process::id())), "{msg}");
drop(first);
assert!(!dir.join(OWNER).exists(), "the owner note goes with the lock");
assert!(dir.join(LOCK).exists(), "the lock file itself may stay: only a held lock refuses");
let again = LibraryLock::acquire(&dir, "lightcraft-cli").unwrap();
assert!(again.held());
drop(again);
let _ = std::fs::remove_dir_all(&dir);
}
/// Files left behind by a crash (lock file + owner note) don't lock anyone out.
#[test]
fn leftover_lock_files_are_not_a_lock() {
let dir = temp_dir("stale");
std::fs::write(dir.join(LOCK), b"").unwrap();
std::fs::write(dir.join(OWNER), br#"{"pid":999999,"host":"elsewhere","program":"LightCraft","version":"0.2.0","since":1}"#).unwrap();
let l = LibraryLock::acquire(&dir, "LightCraft").unwrap();
assert!(l.held());
let owner: LockOwner = serde_json::from_slice(&std::fs::read(dir.join(OWNER)).unwrap()).unwrap();
assert_eq!(owner.pid, std::process::id(), "the note now names this process");
drop(l);
let _ = std::fs::remove_dir_all(&dir);
}
/// Another process holds the lock; when it is killed (a crash), the OS releases it.
#[test]
fn lock_held_by_another_process_is_released_when_it_dies() {
let dir = temp_dir("child");
let exe = std::env::current_exe().unwrap();
let mut child = std::process::Command::new(exe)
.args(["--exact", "tests_lock::lock_holder_child", "--ignored", "--test-threads=1"])
.env("LC_LOCK_CHILD_DIR", &dir)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.unwrap();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60);
while !dir.join("child-locked").exists() {
assert!(child.try_wait().unwrap().is_none(), "child exited before locking");
assert!(std::time::Instant::now() < deadline, "child never locked");
std::thread::sleep(std::time::Duration::from_millis(20));
}
let e = LibraryLock::acquire(&dir, "LightCraft").unwrap_err();
let LockError::InUse(Some(who)) = &e else { panic!("{e:?}") };
assert_eq!(who.pid, child.id());
assert!(e.to_string().contains("on this computer") || who.host.is_empty(), "{e}");
child.kill().unwrap(); // SIGKILL: no clean-up runs, like a crash
child.wait().unwrap();
assert!(dir.join(OWNER).exists(), "the crashed holder's note is left behind");
let l = LibraryLock::acquire(&dir, "LightCraft").expect("the OS released the dead process's lock");
assert!(l.held());
drop(l);
let _ = std::fs::remove_dir_all(&dir);
}
/// Run by [`lock_held_by_another_process_is_released_when_it_dies`] in a child process: take the
/// lock, say so, and wait to be killed.
#[test]
#[ignore = "helper process for lock_held_by_another_process_is_released_when_it_dies"]
fn lock_holder_child() {
let Some(dir) = std::env::var_os("LC_LOCK_CHILD_DIR") else { return };
let _l = LibraryLock::acquire(std::path::Path::new(&dir), "lightcraft-cli").unwrap();
std::fs::write(std::path::Path::new(&dir).join("child-locked"), b"").unwrap();
std::thread::sleep(std::time::Duration::from_secs(120));
}
+223
View File
@@ -0,0 +1,223 @@
//! Issue #101: an append that fails part-way (disk full, a dropped network share) and is then
//! retried must not read back as log damage, and must lose nothing.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use crate::journal::LOG;
use crate::*;
/// A store whose next append writes only a prefix of its data, then fails.
#[derive(Clone, Default)]
struct Torn {
files: MemStore,
/// The next append writes this many bytes, then returns an error.
torn_at: Arc<Mutex<Option<usize>>>,
/// `truncate` fails (the cut-back after a failed append can't be done right away).
truncate_fails: Arc<AtomicBool>,
}
impl Torn {
fn tear_next_append(&self, bytes: usize) {
*self.torn_at.lock().unwrap() = Some(bytes);
}
}
impl Store for Torn {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if let Some(n) = self.torn_at.lock().unwrap().take() {
self.files.append(name, &data[..n.min(data.len())])?;
return Err(std::io::Error::other("disk full"));
}
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
if self.truncate_fails.load(Ordering::SeqCst) {
return Err(std::io::Error::other("share unavailable"));
}
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"torn".into()
}
}
fn add_photo(c: &mut Catalog) -> Op {
let id = c.alloc_photo_id();
Op::AddPhoto { photo: Box::new(Photo::new(id, Source::File { path: format!("/p/{}.jpg", id.0) }, "p.jpg", "JPEG", 4, 3, "2026-01-01")) }
}
/// Apply `ops` to the live catalog (like a command) and return them for the journal.
fn apply(c: &mut Catalog, ops: Vec<Op>) -> Vec<Op> {
for op in &ops {
c.apply(op.clone()).unwrap();
}
ops
}
/// Every tear position of a batch, then the retry (same ops, same seqs, like the engine's queue)
/// and more appends: reloading replays everything, reports no damage and no torn bytes.
#[test]
fn partial_append_then_retry_loads_cleanly() {
let probe = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(probe.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
let before = probe.get(LOG).unwrap().len();
j.append(&batch).unwrap();
let batch_len = probe.get(LOG).unwrap().len() - before;
for tear in 1..batch_len {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 3 }, Op::SetFlag { id, flag: Flag::Pick }, Op::SetRating { id, rating: 4 }]);
t.tear_next_append(tear);
assert!(j.append(&batch).is_err(), "tear at {tear}");
// cut back to the last whole record
assert_eq!(t.files.get(LOG).unwrap().len(), good_len, "tear at {tear}");
j.append(&batch).unwrap();
let more = apply(&mut c, vec![Op::SetLabel { id, label: Some(ColorLabel::ALL[0]) }]);
j.append(&more).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!((r.torn_bytes, r.torn_fragments, r.replayed), (0, 0, 5), "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
}
/// When cutting back fails too (the share is gone), the next append cuts back first; until that
/// works nothing more is appended after the fragment.
#[test]
fn failed_cut_back_is_done_before_the_next_append() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good_len = t.files.get(LOG).unwrap().len();
let id = c.photos().next().unwrap().id;
let batch = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetRating { id, rating: 5 }]);
t.truncate_fails.store(true, Ordering::SeqCst);
t.tear_next_append(30);
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30, "the fragment is still there");
// still unreachable: the retry fails without writing anything after the fragment
assert!(j.append(&batch).is_err());
assert_eq!(t.files.get(LOG).unwrap().len(), good_len + 30);
t.truncate_fails.store(false, Ordering::SeqCst);
j.append(&batch).unwrap();
drop(j);
let (_, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert!(r.damaged.is_none(), "{r:?}");
assert_eq!(r.replayed, 3);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// Logs written before the fix: the fragment of the failed append is glued to the first retried
/// record on one line, followed by the rest of the retry. They load in full (the fragment is
/// skipped), instead of stopping at the fragment and setting the later records aside as damage.
#[test]
fn legacy_fragment_glued_to_the_retry_is_skipped() {
let m = MemStore::new();
let (mut j, mut c, _) = Journal::open(Box::new(m.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let good = m.get(LOG).unwrap();
let id = c.photos().next().unwrap().id;
let batch = apply(
&mut c,
vec![
Op::SetRating { id, rating: 3 },
Op::SetFlag { id, flag: Flag::Pick },
Op::SetLabelName { label: ColorLabel::ALL[1], name: Some("Größe ✓".into()) },
],
);
j.append(&batch).unwrap();
let full = m.get(LOG).unwrap();
let batch_bytes = &full[good.len()..];
let later = apply(&mut c, vec![Op::SetRating { id, rating: 1 }]);
j.append(&later).unwrap();
let later_bytes = m.get(LOG).unwrap()[full.len()..].to_vec();
drop(j);
// every tear (inside the first record, after it, inside a multi-byte character…)
for tear in 1..batch_bytes.len() {
if batch_bytes[tear - 1] == b'\n' {
continue; // a tear on a record boundary leaves no fragment
}
let mut log = good.clone();
log.extend_from_slice(&batch_bytes[..tear]);
log.extend_from_slice(batch_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, loaded, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_none(), "tear at {tear}: {r:?}");
assert_eq!(r.torn_fragments, 1, "tear at {tear}");
assert_eq!(loaded.to_snapshot(), c.to_snapshot(), "tear at {tear}");
}
// a damaged record whose tail happens to be a whole record from the future is still damage
let mut log = good.clone();
log.extend_from_slice(b"garbage");
log.extend_from_slice(&later_bytes);
log.extend_from_slice(&later_bytes);
let t = MemStore::new();
t.set(LOG, log);
let (_, _, r) = Journal::open(Box::new(t)).unwrap();
assert!(r.damaged.is_some(), "{r:?}");
}
/// A snapshot can absorb ops whose append failed: it records them as saved (`seq` moves past
/// them), and appending after it continues from there.
#[test]
fn snapshot_with_unlogged_ops_saves_them() {
let t = Torn::default();
let (mut j, mut c, _) = Journal::open(Box::new(t.clone())).unwrap();
let add = add_photo(&mut c);
let first = apply(&mut c, vec![add]);
j.append(&first).unwrap();
let id = c.photos().next().unwrap().id;
let queued = apply(&mut c, vec![Op::SetRating { id, rating: 2 }, Op::SetFlag { id, flag: Flag::Reject }]);
t.tear_next_append(10);
assert!(j.append(&queued).is_err());
j.snapshot_with_unlogged(&c, queued.len() as u64).unwrap();
assert_eq!(j.seq(), 3);
let after = apply(&mut c, vec![Op::SetRating { id, rating: 5 }]);
j.append(&after).unwrap();
drop(j);
let (j, loaded, r) = Journal::open(Box::new(t.clone())).unwrap();
assert_eq!((r.snapshot_seq, r.replayed), (3, 1));
assert_eq!(j.seq(), 4);
assert_eq!(loaded.to_snapshot(), c.to_snapshot());
}
/// `FsStore::truncate` on a file that was never created (an append that failed first) is a no-op.
#[test]
fn fs_store_truncates_a_missing_file_quietly() {
let dir = std::env::temp_dir().join(format!("lc-torn-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
let mut s = FsStore::open(&dir).unwrap();
s.truncate(LOG, 0).unwrap();
assert!(!dir.join(LOG).exists());
s.append(LOG, b"abc\n").unwrap();
s.truncate(LOG, 0).unwrap();
assert_eq!(std::fs::read(dir.join(LOG)).unwrap(), b"");
let _ = std::fs::remove_dir_all(&dir);
}
+215
View File
@@ -0,0 +1,215 @@
//! Is a file there? — answered without blocking the caller.
//!
//! On a sleeping NAS, a dropped share or a USB drive spinning up, one `stat` can take as long as
//! the OS or SMB timeout. The UI asks about files every frame (grid cells, the Info panel, the
//! Missing Photos view), so in the app the answers come from a cache that a worker thread fills
//! ([`Availability::run_in_background`]): an unknown file is reported as `None` ("assume it is
//! there"; the render worker finds out if it isn't), known answers are rechecked after
//! [`RECHECK`]. Everywhere else (CLI, MCP, tests) the default is synchronous: every question is a
//! fresh check, as before.
use std::collections::{HashMap, HashSet, VecDeque};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use std::time::Duration;
/// Checks whether a path exists (injectable: tests use a slow one).
pub type ExistsProbe = Arc<dyn Fn(&str) -> bool + Send + Sync>;
/// Called (from the worker) when an answer changed, e.g. to repaint the UI.
pub type Waker = Arc<dyn Fn() + Send + Sync>;
/// How long a cached answer is used before it is checked again (it is still returned meanwhile).
pub const RECHECK: Duration = Duration::from_secs(15);
fn fs_exists(path: &str) -> bool {
// the browser build has no file system: its originals are always "there"
cfg!(target_arch = "wasm32") || std::path::Path::new(path).exists()
}
#[derive(Default)]
struct State {
/// path → (exists, when checked)
known: HashMap<String, (bool, std::time::Instant)>,
queue: VecDeque<String>,
queued: HashSet<String>,
worker: bool,
/// Bumped whenever an answer changes (callers recompute what depends on it).
generation: u64,
}
fn lock(m: &Mutex<State>) -> MutexGuard<'_, State> {
m.lock().unwrap_or_else(PoisonError::into_inner)
}
/// File availability, synchronous or cached + background-checked. Cloning shares the cache.
#[derive(Clone)]
pub struct Availability {
probe: ExistsProbe,
background: Option<Waker>,
state: Arc<Mutex<State>>,
}
impl Default for Availability {
fn default() -> Self {
Self { probe: Arc::new(fs_exists), background: None, state: Arc::default() }
}
}
impl std::fmt::Debug for Availability {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Availability").field("background", &self.background.is_some()).finish()
}
}
impl Availability {
/// Replace the existence check (tests: a slow or counting probe). Forgets cached answers.
pub fn set_probe(&mut self, probe: ExistsProbe) {
self.probe = probe;
let mut st = lock(&self.state);
st.known.clear();
st.generation += 1;
}
/// Answer from the cache and check on a worker thread from now on; `wake` runs when an
/// answer changed. No effect in the browser build (no threads, no file system).
pub fn run_in_background(&mut self, wake: Waker) {
if cfg!(target_arch = "wasm32") {
return;
}
self.background = Some(wake);
}
/// The existence check itself (blocking: for worker threads, e.g. the Rename preview's).
pub fn probe(&self) -> ExistsProbe {
self.probe.clone()
}
pub fn is_background(&self) -> bool {
self.background.is_some()
}
/// Does `path` exist? Synchronous mode: checked now. Background mode: the cached answer
/// (`None` while unknown); unknown or stale answers are queued for the worker.
pub fn exists(&self, path: &str) -> Option<bool> {
let Some(wake) = &self.background else {
return Some((self.probe)(path));
};
let mut st = lock(&self.state);
let now = std::time::Instant::now();
let cached = st.known.get(path).copied();
if cached.is_none_or(|(_, at)| now.saturating_duration_since(at) >= RECHECK) && !st.queued.contains(path) {
st.queued.insert(path.to_string());
st.queue.push_back(path.to_string());
if !st.worker {
st.worker = true;
drop(st);
self.spawn_worker(wake.clone());
return cached.map(|c| c.0);
}
}
cached.map(|c| c.0)
}
/// Known to be missing (unknown counts as present).
pub fn is_offline(&self, path: &str) -> bool {
self.exists(path) == Some(false)
}
/// Changes whenever a cached answer changes.
pub fn generation(&self) -> u64 {
lock(&self.state).generation
}
/// Paths waiting for the worker.
pub fn pending(&self) -> usize {
lock(&self.state).queue.len()
}
fn spawn_worker(&self, wake: Waker) {
let (state, probe) = (self.state.clone(), self.probe.clone());
let started = std::thread::Builder::new().name("lc-availability".into()).spawn(move || {
loop {
let path = {
let mut st = lock(&state);
let Some(p) = st.queue.pop_front() else {
st.worker = false;
return;
};
p
};
let ok = probe(&path);
let changed = {
let mut st = lock(&state);
st.queued.remove(&path);
let prev = st.known.insert(path, (ok, std::time::Instant::now()));
let changed = prev.map(|p| p.0) != Some(ok);
if changed {
st.generation += 1;
}
changed
};
if changed {
wake();
}
}
});
if started.is_err() {
// try again on the next question
lock(&self.state).worker = false;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering};
#[test]
fn synchronous_by_default() {
let mut a = Availability::default();
let n = Arc::new(AtomicUsize::new(0));
let c = n.clone();
a.set_probe(Arc::new(move |p| {
c.fetch_add(1, Ordering::SeqCst);
p == "here"
}));
assert_eq!(a.exists("here"), Some(true));
assert_eq!(a.exists("gone"), Some(false));
assert_eq!(a.exists("gone"), Some(false));
assert_eq!(n.load(Ordering::SeqCst), 3, "every question is a fresh check");
}
#[test]
fn background_answers_from_the_cache_without_blocking() {
let mut a = Availability::default();
let caller = std::thread::current().id();
let on_caller = Arc::new(AtomicUsize::new(0));
let c = on_caller.clone();
a.set_probe(Arc::new(move |p| {
if std::thread::current().id() == caller {
c.fetch_add(1, Ordering::SeqCst);
}
std::thread::sleep(Duration::from_millis(200)); // a sleeping NAS
p == "here"
}));
let woke = Arc::new(AtomicUsize::new(0));
let w = woke.clone();
a.run_in_background(Arc::new(move || {
w.fetch_add(1, Ordering::SeqCst);
}));
let t0 = std::time::Instant::now();
assert_eq!(a.exists("here"), None, "unknown at first");
assert_eq!(a.exists("gone"), None);
assert!(t0.elapsed() < Duration::from_millis(150), "the caller never waits for the probe");
let deadline = std::time::Instant::now() + Duration::from_secs(20);
while (a.exists("here").is_none() || a.exists("gone").is_none()) && std::time::Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(10));
}
assert_eq!(a.exists("here"), Some(true));
assert_eq!(a.exists("gone"), Some(false));
assert!(a.is_offline("gone") && !a.is_offline("here") && !a.is_offline("unknown"));
assert_eq!(on_caller.load(Ordering::SeqCst), 0, "never probed on the calling thread");
assert!(woke.load(Ordering::SeqCst) >= 2);
assert!(a.generation() >= 2);
}
}
+62 -35
View File
@@ -166,7 +166,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Rename Folder",
[],
None,
"{path, name} — rename a folder on disk (its files and sidecars go along) and relink the photos in it; not an undo step → {path, relinked}",
"{path, name} — rename a folder on disk (its files and sidecars go along) and relink the photos in it; one undo step (undo renames it back, refused if the old name is taken) → {path, relinked}",
always,
|s, p| {
let from = str_param(p, "path").ok_or_else(|| bad("folder.rename", "missing path"))?.trim_end_matches(['/', '\\']).to_string();
@@ -184,7 +184,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Move Folder",
[],
None,
"{path, into: destination folder} — move a folder on disk into another and relink the photos in it; not an undo step → {path, relinked}",
"{path, into: destination folder} — move a folder on disk into another and relink the photos in it; one undo step (undo moves it back, refused if the old place is taken) → {path, relinked}",
always,
|s, p| {
let from = str_param(p, "path").ok_or_else(|| bad("folder.move", "missing path"))?.trim_end_matches(['/', '\\']).to_string();
@@ -217,58 +217,85 @@ pub fn specs() -> Vec<CommandSpec> {
]
}
/// Rename or move a folder on disk (everything in it goes along, sidecars included) and relink
/// the photos inside. Like a rename in the file manager it is not an undo step: undoing would
/// point the photos back at a folder that no longer exists.
pub(crate) fn move_folder(s: &mut crate::Session, from: &str, to: &str) -> std::result::Result<usize, String> {
use std::path::Path;
/// Rename `from` to `to` on disk (a folder; everything in it goes along). Never overwrites:
/// refused when `to` exists (a case-only rename on a case-insensitive file system goes through
/// a temporary name). Missing parents of `to` are created.
pub(crate) fn rename_folder_on_disk(from: &str, to: &str) -> std::result::Result<(), String> {
let (src, dst) = (Path::new(from), Path::new(to));
if !src.is_dir() {
return Err(format!("{from} is not a folder"));
return Err(format!("{from} is not a folder (moved or deleted?)"));
}
if dst.exists() {
let case_only = from != to && from.to_lowercase() == to.to_lowercase();
if dst.exists() && !case_only {
return Err(format!("{to} already exists"));
}
if dst.starts_with(src) {
return Err("a folder can't move into itself".into());
}
if s.library.as_ref().is_some_and(|l| l.dir.starts_with(src) || src.starts_with(&l.dir)) {
return Err("the library's own folders can't be moved here".into());
}
if let Some(parent) = dst.parent() {
std::fs::create_dir_all(parent).map_err(|e| format!("{}: {e}", parent.display()))?;
}
std::fs::rename(src, dst).map_err(|e| format!("{from} → {to}: {e}"))?;
let mut n = 0;
let ops: Vec<lightcraft_catalog::Op> = s
if case_only {
let tmp = src.with_file_name(format!(".lc-rename-{}", std::process::id()));
if tmp.exists() {
return Err(format!("{} already exists", tmp.display()));
}
std::fs::rename(src, &tmp).map_err(|e| format!("{from} → {to}: {e}"))?;
return std::fs::rename(&tmp, dst).map_err(|e| {
let _ = std::fs::rename(&tmp, src);
format!("{from} → {to}: {e}")
});
}
std::fs::rename(src, dst).map_err(|e| format!("{from} → {to}: {e}"))
}
/// After a folder moved from `from` to `to`: a browsed folder at or below it follows.
pub(crate) fn follow_folder(s: &mut Session, from: &str, to: &str) {
if let Some(b) = &mut s.browse
&& let Ok(rel) = Path::new(&b.path).strip_prefix(from)
{
b.path = Path::new(to).join(rel).to_string_lossy().to_string();
let f = b.path.clone();
s.filter.folder = Some(f);
}
}
/// Rename or move a folder on disk (everything in it goes along, sidecars included) and relink
/// the photos inside, as one undo step: undo renames the folder back and restores the photos'
/// paths, redo repeats the move — each refused (and reported) when the destination is taken.
pub(crate) fn move_folder(s: &mut Session, from: &str, to: &str) -> std::result::Result<usize, String> {
let (src, dst) = (Path::new(from), Path::new(to));
if !src.is_dir() {
return Err(format!("{from} is not a folder"));
}
if s.library.as_ref().is_some_and(|l| l.dir.starts_with(src) || src.starts_with(&l.dir)) {
return Err("the library's own folders can't be moved here".into());
}
rename_folder_on_disk(from, to)?;
let ops: Vec<Op> = s
.catalog
.photos()
.filter_map(|p| match &p.source {
lightcraft_catalog::Source::File { path } => Path::new(path).strip_prefix(src).ok().map(|rel| (p.id, p.file_name.clone(), dst.join(rel))),
_ => None,
})
.map(|(id, file_name, np)| {
n += 1;
lightcraft_catalog::Op::Relink {
id,
file_name,
source: lightcraft_catalog::Source::File { path: np.to_string_lossy().to_string() },
format: None,
}
.map(|(id, file_name, np)| Op::Relink {
id,
file_name,
source: lightcraft_catalog::Source::File { path: np.to_string_lossy().to_string() },
format: None,
})
.collect();
for op in ops {
if s.catalog.apply(op.clone()).is_ok() {
s.pending_log.push(op);
}
}
// a browsed folder at or below it follows
if let Some(b) = &mut s.browse
&& let Ok(rel) = Path::new(&b.path).strip_prefix(src)
{
b.path = dst.join(rel).to_string_lossy().to_string();
let f = b.path.clone();
s.filter.folder = Some(f);
let n = ops.len();
let name = Path::new(to).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let renamed = src.parent() == dst.parent();
let label = if renamed { format!("Rename Folder to “{name}”") } else { format!("Move Folder “{name}”") };
let folder = crate::FolderMove { from: to.to_string(), to: from.to_string() };
if let Err(e) = s.commit_with_folder(&label, Op::Batch { ops }, folder) {
// the catalog didn't take it: put the folder back where the photos still point
let back = rename_folder_on_disk(to, from).err().map(|b| format!("; moving it back failed too: {b}")).unwrap_or_default();
return Err(format!("{e}{back}"));
}
follow_folder(s, from, to);
Ok(n)
}
+6 -1
View File
@@ -26,7 +26,12 @@ fn dng_names(path: &str) -> impl Iterator<Item = std::path::PathBuf> {
/// written to a temp file, synced, read back identical and only then given its name — never
/// replacing an existing file. On any failure no DNG is left behind and the raw stays in use.
pub(crate) fn write_dng_for(s: &Session, path: &str, packet: String) -> std::result::Result<String, String> {
let bytes = match &s.media.file_bytes {
write_dng_with(s.media.file_bytes.as_ref(), path, packet)
}
/// [`write_dng_for`] without the session (imports convert on a worker thread).
pub(crate) fn write_dng_with(file_bytes: Option<&crate::merge::ByteReader>, path: &str, packet: String) -> std::result::Result<String, String> {
let bytes = match file_bytes {
Some(r) => r(path)?,
None => std::fs::read(path).map_err(|e| format!("{path}: {e}"))?,
};
+119 -76
View File
@@ -7,6 +7,105 @@ use serde_json::{Value, json};
use super::{CommandSpec, always, bad, bool_or, cmd, has_active, has_selection, ok, str_param};
use crate::{LibrarySource, Result, Selection, Session};
/// The auto-import folder's visible files with their sizes (the file-system half of
/// `library.autoImportScan`; the app lists on a worker thread and passes `listing`).
pub fn list_auto_import_folder(folder: &str) -> std::result::Result<Vec<(String, u64)>, String> {
let rd = std::fs::read_dir(folder).map_err(|e| format!("{folder}: {e}"))?;
Ok(rd
.flatten()
.filter(|e| !e.file_name().to_string_lossy().starts_with('.'))
.filter_map(|e| {
let m = std::fs::metadata(e.path()).ok()?;
m.is_file().then(|| (e.path().to_string_lossy().to_string(), m.len()))
})
.collect())
}
/// `library.import`'s params, checked (no file-system calls).
pub struct ImportRequest {
pub paths: Vec<String>,
pub opts: crate::import::ImportOptions,
pub album: Option<u64>,
pub album_name: Option<String>,
}
/// Parse and check `library.import`'s params (the app's import task runs the import itself, on a
/// worker thread, with the same options).
pub fn import_params(s: &Session, p: &Value) -> Result<ImportRequest> {
let paths = strs(p, "paths");
let mode = match str_param(p, "mode").unwrap_or("add") {
"add" => crate::import::ImportMode::Add,
"copy" => crate::import::ImportMode::Copy,
"move" => crate::import::ImportMode::Move,
other => return Err(bad("library.import", format!("unknown mode `{other}` (add|copy|move)"))),
};
let preset = match str_param(p, "preset").filter(|x| !x.is_empty()) {
Some(id) => Some(s.presets.iter().find(|x| x.id == id).cloned().ok_or_else(|| bad("library.import", format!("unknown preset `{id}`")))?),
None => None,
};
let album = p.get("album").and_then(Value::as_u64);
if let Some(a) = album
&& s.catalog.album(AlbumId(a)).is_none_or(|al| al.folder || al.is_smart())
{
return Err(bad("library.import", "album must be a regular album"));
}
let organize = match str_param(p, "organize") {
Some(o) => crate::import::Organize::parse(o).ok_or_else(|| {
bad("library.import", format!("unknown organize `{o}` (date|month|flat, or a folder template like {{date:%Y}}/{{date:%Y%m%d}})"))
})?,
None => Default::default(),
};
if let crate::import::Organize::Template(t) = &organize
&& let Some(e) = crate::rename::folder_template_error(t)
{
return Err(bad("library.import", e));
}
let metadata_preset = str_param(p, "metadataPreset").map(str::trim).filter(|n| !n.is_empty()).map(str::to_string);
if let Some(n) = &metadata_preset
&& !s.metadata_presets.iter().any(|m| m.name.eq_ignore_ascii_case(n))
{
return Err(bad("library.import", format!("unknown metadata preset `{n}`")));
}
let opts = crate::import::ImportOptions {
mode,
preset,
keywords: strs(p, "keywords"),
destination: str_param(p, "destination").map(str::to_string),
organize,
rename: str_param(p, "rename").map(str::to_string),
rename_start: p.get("renameStart").and_then(Value::as_u64).unwrap_or(1) as usize,
metadata_preset,
convert_dng: bool_or(p, "dng", false),
local: bool_or(p, "local", false),
};
let album_name = str_param(p, "albumName").map(str::trim).filter(|n| !n.is_empty()).map(str::to_string);
Ok(ImportRequest { paths, opts, album, album_name })
}
/// After a batch of an import was committed: the album (a new one named `album_name` when there
/// is no `album` and something was imported) gets the photos. Returns the report as JSON (with
/// `album`).
pub fn import_batch_done(s: &mut Session, report: crate::import::ImportReport, mut album: Option<u64>, album_name: Option<&str>) -> Result<Value> {
let mut report = serde_json::to_value(report).unwrap_or_default();
let imported: Vec<u64> = report["imported"].as_array().map(|a| a.iter().filter_map(Value::as_u64).collect()).unwrap_or_default();
if album.is_none()
&& let Some(name) = album_name
&& !imported.is_empty()
{
let r = s.execute("album.create", &json!({"name": name}))?;
album = r["id"].as_u64();
}
if let Some(a) = album
&& !imported.is_empty()
{
// (album.addPhotos wants a selection: the new photos are about to be it)
s.selection = Selection::single(PhotoId(imported[0]));
s.execute("album.addPhotos", &json!({"id": a, "ids": imported}))?;
report["album"] = json!(a);
}
Ok(report)
}
fn album_param(p: &Value, key: &str, c: &str) -> Result<AlbumId> {
p.get(key).and_then(Value::as_u64).map(AlbumId).ok_or_else(|| bad(c, format!("missing album `{key}`")))
}
@@ -657,9 +756,9 @@ pub fn specs() -> Vec<CommandSpec> {
"Auto Import Now",
[],
None,
"{} — add the watched folder's new photos (files the library doesn't have yet; partial / still-copying files wait for the next scan) → {imported, folder}",
"{listing?: [[path, size]] (the folder as listed by the caller, e.g. on a worker thread), start?: bool (false: return the `library.import` params as `import` instead of importing)} — add the watched folder's new photos (files the library doesn't have yet; partial / still-copying files wait for the next scan) → {imported, folder, import?}",
always,
|s, _| {
|s, p| {
let Some(folder) = s.import_defaults.auto_folder.clone() else { return Ok(json!({"imported": [], "folder": null})) };
// only files that stopped growing: a file still being written is left for later
let known: std::collections::HashSet<String> = s
@@ -670,18 +769,20 @@ pub fn specs() -> Vec<CommandSpec> {
_ => None,
})
.collect();
// the folder's files and sizes: listed here, or already listed on a worker thread (the app)
let listing: Vec<(String, u64)> = match p.get("listing").and_then(Value::as_array) {
Some(l) => l.iter().filter_map(|e| Some((e.get(0)?.as_str()?.to_string(), e.get(1)?.as_u64()?))).collect(),
None => list_auto_import_folder(&folder).map_err(|e| bad("library.autoImportScan", e))?,
};
let mut fresh = Vec::new();
for e in std::fs::read_dir(&folder).map_err(|e| bad("library.autoImportScan", format!("{folder}: {e}")))?.flatten() {
let path = e.path();
let ps = path.to_string_lossy().to_string();
if !path.is_file() || known.contains(&ps) || path.file_name().is_some_and(|n| n.to_string_lossy().starts_with('.')) {
for (ps, size) in listing {
if known.contains(&ps) {
continue;
}
// each file is tried once (a non-photo isn't retried every scan)
if s.auto_import_seen.get(&ps) == Some(&u64::MAX) {
continue;
}
let size = e.metadata().map(|m| m.len()).unwrap_or(0);
let seen = s.auto_import_seen.insert(ps.clone(), size);
if size > 0 && seen == Some(size) {
fresh.push(ps);
@@ -701,6 +802,10 @@ pub fn specs() -> Vec<CommandSpec> {
for f in &fresh {
s.auto_import_seen.insert(f.clone(), u64::MAX);
}
if p.get("start").and_then(Value::as_bool) == Some(false) {
// the caller imports them (the app: on a worker thread)
return Ok(json!({"imported": [], "folder": folder, "import": params}));
}
let sel = s.selection.clone();
let r = s.execute("library.import", &params)?;
// arriving photos don't take over the selection
@@ -751,79 +856,15 @@ pub fn specs() -> Vec<CommandSpec> {
"{paths: [file or folder (recursive)], mode?: add|copy|move (add = reference the files in place; copy = into the library's Originals/YYYY/YYYY-MM-DD/; move = as copy, then each original and its XMP sidecars are removed from the source — only after the copy is verified (a hard link on the same volume, else copied, synced and compared byte for byte) and its catalog record is saved; failed, duplicate and unchecked files keep their sources; a taken name gets -1, -2…; undo removes the photos from the library but leaves the files at the destination), destination?: folder for copies / moves, organize?: date (YYYY/YYYY-MM-DD) | month (YYYY/YYYY-MM) | flat | a folder template, e.g. `{date:%Y}/{date:%Y%m%d}` → 2026/20260114 (the template's `/` make the folders, each level expanded with the rename tokens and made a safe folder name: never outside the destination; must be relative, no `..`; a level with missing metadata is `unknown`) — dated by capture time, else the import time, rename?: file-name template for copies, original extension added (tokens: {name} {num} {seq} {seq:N} {date} {date:%Y%m%d} {folder} {camera} {lens} {iso} {rating} {title} {creator} {ext}; photo.renameTokens explains each; blank = keep names), renameStart?: 1, metadataPreset?: name, dng?: bool (copy raws as DNG; copy only), local?: bool (browsing: the photos stay out of the library, like library.browse; not with move), album?: albumId, albumName?: new album, preset?: presetId, keywords?: [..]} → {imported, duplicates, failed, moved?: [{from, to, sidecars?}], kept?: [{path, reason}] (move: sources left in place and why), album?}",
always,
|s, p| {
let paths = strs(p, "paths");
if paths.is_empty() {
let req = import_params(s, p)?;
if req.paths.is_empty() {
return Err(bad("library.import", "no paths"));
}
let mode = match str_param(p, "mode").unwrap_or("add") {
"add" => crate::import::ImportMode::Add,
"copy" => crate::import::ImportMode::Copy,
"move" => crate::import::ImportMode::Move,
other => return Err(bad("library.import", format!("unknown mode `{other}` (add|copy|move)"))),
};
let preset = match str_param(p, "preset").filter(|x| !x.is_empty()) {
Some(id) => {
Some(s.presets.iter().find(|x| x.id == id).cloned().ok_or_else(|| bad("library.import", format!("unknown preset `{id}`")))?)
}
None => None,
};
let mut album = p.get("album").and_then(Value::as_u64);
if let Some(a) = album
&& s.catalog.album(AlbumId(a)).is_none_or(|al| al.folder || al.is_smart())
{
return Err(bad("library.import", "album must be a regular album"));
}
let organize = match str_param(p, "organize") {
Some(o) => crate::import::Organize::parse(o).ok_or_else(|| {
bad(
"library.import",
format!("unknown organize `{o}` (date|month|flat, or a folder template like {{date:%Y}}/{{date:%Y%m%d}})"),
)
})?,
None => Default::default(),
};
if let crate::import::Organize::Template(t) = &organize
&& let Some(e) = crate::rename::folder_template_error(t)
{
return Err(bad("library.import", e));
}
let metadata_preset = str_param(p, "metadataPreset").map(str::trim).filter(|n| !n.is_empty()).map(str::to_string);
if let Some(n) = &metadata_preset
&& !s.metadata_presets.iter().any(|m| m.name.eq_ignore_ascii_case(n))
{
return Err(bad("library.import", format!("unknown metadata preset `{n}`")));
}
let opts = crate::import::ImportOptions {
mode,
preset,
keywords: strs(p, "keywords"),
destination: str_param(p, "destination").map(str::to_string),
organize,
rename: str_param(p, "rename").map(str::to_string),
rename_start: p.get("renameStart").and_then(Value::as_u64).unwrap_or(1) as usize,
metadata_preset,
convert_dng: super::bool_or(p, "dng", false),
local: super::bool_or(p, "local", false),
};
let undo0 = s.undo.len();
let mut report = serde_json::to_value(crate::import::import_with(s, &paths, &opts)?).unwrap_or_default();
let imported: Vec<u64> = report["imported"].as_array().map(|a| a.iter().filter_map(Value::as_u64).collect()).unwrap_or_default();
if album.is_none()
&& let Some(name) = str_param(p, "albumName").map(str::trim).filter(|n| !n.is_empty())
&& !imported.is_empty()
{
let r = s.execute("album.create", &json!({"name": name}))?;
album = r["id"].as_u64();
}
if let Some(a) = album
&& !imported.is_empty()
{
// (album.addPhotos wants a selection: the new photos are about to be it)
s.selection = Selection::single(PhotoId(imported[0]));
s.execute("album.addPhotos", &json!({"id": a, "ids": imported}))?;
report["album"] = json!(a);
}
let report = crate::import::import_with(s, &req.paths, &req.opts)?;
let report = import_batch_done(s, report, req.album, req.album_name.as_deref())?;
// one undo step for the whole import
let imported: Vec<u64> = report["imported"].as_array().map(|a| a.iter().filter_map(Value::as_u64).collect()).unwrap_or_default();
let n = s.undo.len().saturating_sub(undo0);
s.merge_undo(n, &format!("Add {} Photo{}", imported.len(), if imported.len() == 1 { "" } else { "s" }));
if let Some(f) = imported.first() {
@@ -867,6 +908,8 @@ pub fn specs() -> Vec<CommandSpec> {
"logRecords": j.log_records(),
"logBytes": j.log_bytes(),
"lastError": lib.last_error,
// settings files that were unreadable or damaged at open (kept, defaults used)
"settingsWarnings": lib.settings_warnings,
// changes applied in memory whose write failed (retried by every save)
"unsavedOps": s.unsaved().map_or(0, |u| u.0),
"unsavedError": s.unsaved().map(|u| u.1),
+184 -42
View File
@@ -45,9 +45,10 @@ pub fn missing(s: &Session) -> Vec<(PhotoId, String)> {
}
/// Whether photo `id` is in scope and its file is gone (the Missing Photos view checks only
/// the photos its query already narrowed to).
pub fn is_missing(cat: &Catalog, id: PhotoId) -> bool {
!cfg!(target_arch = "wasm32") && cat.photo(id).and_then(|p| checked_path(p)).is_some_and(|f| !Path::new(f).exists())
/// the photos its query already narrowed to). In the app `avail` answers from its cache (files
/// not checked yet count as present until the background check finds them gone).
pub fn is_missing(cat: &Catalog, avail: &crate::availability::Availability, id: PhotoId) -> bool {
!cfg!(target_arch = "wasm32") && cat.photo(id).and_then(|p| checked_path(p)).is_some_and(|f| avail.is_offline(f))
}
fn relink_op(id: PhotoId, path: &str) -> Op {
@@ -94,8 +95,112 @@ fn file_hash(path: &str) -> Option<String> {
std::fs::read(path).ok().map(|b| lightcraft_preview::hash_bytes(&b).to_string())
}
/// Search `folder` (recursively) for each missing photo's file and relink every match in one
/// undoable step.
/// An in-scope photo ([`checked_path`]) as Find Missing Photos sees it: its file, the size and
/// content hash the library knows (0 / `None` = unknown).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FindCandidate {
pub id: PhotoId,
pub path: String,
pub size: u64,
pub hash: Option<String>,
}
/// Every in-scope photo, for [`plan_find_missing`]: the missing ones are looked for, and no
/// photo's file is used for another. No file-system calls, so the app takes this snapshot on the
/// UI thread and plans on a worker without holding the session.
pub fn find_candidates(cat: &Catalog) -> Vec<FindCandidate> {
cat.photos()
.filter_map(|p| {
checked_path(p).map(|f| FindCandidate { id: p.id, path: f.to_string(), size: p.file_size, hash: stored_hash(p).map(str::to_string) })
})
.collect()
}
/// How a missing photo's file was recognised.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MatchBy {
/// Same name and size (and content, when the library knows its hash).
Name,
/// Renamed: same size and content hash.
Content,
}
impl MatchBy {
pub fn as_str(self) -> &'static str {
match self {
MatchBy::Name => "name",
MatchBy::Content => "content",
}
}
}
/// A missing photo and the file found for it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FoundFile {
pub id: PhotoId,
pub from: String,
pub to: String,
pub by: MatchBy,
}
/// A missing photo with several look-alike files (same name and size, no hash to tell them
/// apart): skipped, never guessed.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AmbiguousFile {
pub id: PhotoId,
pub from: String,
pub candidates: Vec<String>,
}
/// What Find Missing Photos found ([`plan_find_missing`]), before relinking.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct FindPlan {
pub found: Vec<FoundFile>,
/// In-scope photos that stay missing (the ambiguous ones included).
pub missing: usize,
pub ambiguous: Vec<AmbiguousFile>,
}
impl FindPlan {
/// `{found: [{id, from, to, by}], missing, ambiguous: [{id, from, candidates}]}` — the
/// command's result, and what `library.findMissing {found, …}` takes back.
pub fn to_json(&self) -> Value {
let found: Vec<Value> = self.found.iter().map(|f| json!({"id": f.id.0, "from": f.from, "to": f.to, "by": f.by.as_str()})).collect();
let ambiguous: Vec<Value> = self.ambiguous.iter().map(|a| json!({"id": a.id.0, "from": a.from, "candidates": a.candidates})).collect();
json!({"found": found, "missing": self.missing, "ambiguous": ambiguous})
}
/// The inverse of [`FindPlan::to_json`] (entries it can't read are dropped).
pub fn from_json(v: &Value) -> FindPlan {
let list = |k: &str| v.get(k).and_then(Value::as_array).cloned().unwrap_or_default();
let found = list("found")
.iter()
.filter_map(|f| {
let by = if f.get("by").and_then(Value::as_str) == Some("content") { MatchBy::Content } else { MatchBy::Name };
Some(FoundFile {
id: PhotoId(f.get("id")?.as_u64()?),
from: str_param(f, "from")?.to_string(),
to: str_param(f, "to")?.to_string(),
by,
})
})
.collect();
let ambiguous = list("ambiguous")
.iter()
.filter_map(|a| {
let candidates = a.get("candidates")?.as_array()?.iter().filter_map(|c| c.as_str().map(str::to_string)).collect();
Some(AmbiguousFile { id: PhotoId(a.get("id")?.as_u64()?), from: str_param(a, "from")?.to_string(), candidates })
})
.collect();
let missing = v.get("missing").and_then(Value::as_u64).map_or(0, |n| usize::try_from(n).unwrap_or(usize::MAX));
FindPlan { found, missing, ambiguous }
}
}
/// The file-system half of Find Missing Photos: which `candidates` are gone, and which file in
/// `folder` (recursively) is each one's. Touches only the disk — never the session — so the app
/// runs it on a worker thread (walking a big folder on a network share and hashing look-alikes
/// takes a while) and relinks the result with `library.findMissing {found, missing, ambiguous}`.
///
/// Matching, per photo:
/// 1. a file with the same name (any letter case), the same size and the same content hash;
@@ -107,20 +212,21 @@ fn file_hash(path: &str) -> Option<String> {
///
/// Files that are already some photo's original, or that another photo was just matched with,
/// are not used.
fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.findMissing";
let folder = str_param(p, "folder").ok_or_else(|| bad(C, "missing `folder`"))?;
pub fn plan_find_missing(candidates: &[FindCandidate], folder: &str) -> std::result::Result<FindPlan, String> {
if !Path::new(folder).is_dir() {
return Err(bad(C, format!("{folder}: not a folder")));
return Err(format!("{folder}: not a folder"));
}
let lost = missing(s);
if cfg!(target_arch = "wasm32") {
return Ok(FindPlan::default());
}
let lost: Vec<&FindCandidate> = candidates.iter().filter(|c| !Path::new(&c.path).exists()).collect();
if lost.is_empty() {
return Ok(json!({"found": [], "missing": 0, "ambiguous": []}));
return Ok(FindPlan::default());
}
let in_use: HashSet<String> = s.catalog.photos().filter_map(|p| checked_path(p).map(str::to_string)).collect();
let in_use: HashSet<&str> = candidates.iter().map(|c| c.path.as_str()).collect();
// file name (lower case) → candidate paths; size → paths (filled lazily)
let mut by_name: HashMap<String, Vec<String>> = HashMap::new();
let files: Vec<String> = crate::import::expand(&[folder.to_string()], None).into_iter().filter(|f| !in_use.contains(f)).collect();
let files: Vec<String> = crate::import::expand(&[folder.to_string()], None).into_iter().filter(|f| !in_use.contains(f.as_str())).collect();
for f in &files {
if let Some(n) = Path::new(f).file_name() {
by_name.entry(n.to_string_lossy().to_lowercase()).or_default().push(f.clone());
@@ -131,24 +237,21 @@ fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
let mut hash_of = |f: &str| hashes.entry(f.to_string()).or_insert_with(|| file_hash(f)).clone();
let size_of = |f: &str| std::fs::metadata(f).map(|m| m.len()).ok();
let mut used: HashSet<String> = HashSet::new();
let mut ops = Vec::new();
let mut found = Vec::new();
let mut ambiguous = Vec::new();
for (id, old) in &lost {
let Some(ph) = s.catalog.photo(*id) else { continue };
let name = Path::new(old).file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default();
let size = ph.file_size;
let hash = stored_hash(ph).map(str::to_string);
let mut plan = FindPlan::default();
for c in &lost {
let name = Path::new(&c.path).file_name().map(|n| n.to_string_lossy().to_lowercase()).unwrap_or_default();
let size = c.size;
let hash = c.hash.as_deref();
let named: Vec<&String> = by_name
.get(&name)
.map(|c| c.iter().filter(|f| !used.contains(*f) && (size == 0 || size_of(f) == Some(size))).collect())
.map(|v| v.iter().filter(|f| !used.contains(*f) && (size == 0 || size_of(f) == Some(size))).collect())
.unwrap_or_default();
// 1. same name and size, confirmed by the content hash
let mut hit: Option<(String, &str)> =
hash.as_deref().and_then(|h| named.iter().find(|f| hash_of(f).as_deref() == Some(h))).map(|f| (f.to_string(), "name"));
let mut hit: Option<(String, MatchBy)> =
hash.and_then(|h| named.iter().find(|f| hash_of(f).as_deref() == Some(h))).map(|f| (f.to_string(), MatchBy::Name));
// 2. renamed: same size and content
if hit.is_none()
&& let Some(h) = &hash
&& let Some(h) = hash
&& size > 0
{
let sizes = by_size.get_or_insert_with(|| {
@@ -162,36 +265,75 @@ fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
});
hit = sizes
.get(&size)
.and_then(|c| c.iter().find(|f| !used.contains(*f) && hash_of(f).as_deref() == Some(h.as_str())))
.map(|f| (f.clone(), "content"));
.and_then(|v| v.iter().find(|f| !used.contains(*f) && hash_of(f).as_deref() == Some(h)))
.map(|f| (f.clone(), MatchBy::Content));
}
// 3. same name and size only (no hash, or the file was edited in place): one candidate
// is taken, several are ambiguous and skipped rather than guessed
if hit.is_none() {
match named.len() {
1 => hit = Some((named[0].clone(), "name")),
0 => {}
match named.as_slice() {
[one] => hit = Some(((*one).clone(), MatchBy::Name)),
[] => {}
_ => {
ambiguous.push(json!({"id": id.0, "from": old, "candidates": named}));
plan.ambiguous.push(AmbiguousFile { id: c.id, from: c.path.clone(), candidates: named.iter().map(|f| (*f).clone()).collect() });
continue;
}
}
}
let Some((path, by)) = hit else { continue };
ops.push(relink_op(*id, &path));
found.push(json!({"id": id.0, "from": old, "to": path, "by": by}));
used.insert(path);
used.insert(path.clone());
plan.found.push(FoundFile { id: c.id, from: c.path.clone(), to: path, by });
}
let still = lost.len() - found.len();
if !ops.is_empty() {
s.commit("Find Missing Photos", Op::Batch { ops })?;
plan.missing = lost.len() - plan.found.len();
Ok(plan)
}
/// Relink what [`plan_find_missing`] found, in one undoable step, re-checked against the
/// catalog as it is now (the search may have run on a worker while the library changed): a
/// photo that no longer points at the path it was missing from (relinked, removed) is left
/// alone, and a file that is now some photo's original — or that the plan names twice — is not
/// used (that photo stays missing). `missing` is the plan's count, corrected for what changed
/// among the photos the plan names.
fn commit_find(s: &mut Session, plan: FindPlan) -> Result<Value> {
let still_at = |cat: &Catalog, id: PhotoId, from: &str| cat.photo(id).and_then(|p| checked_path(p)) == Some(from);
let mut in_use: HashSet<String> = candidates(&s.catalog).into_iter().collect();
let mut found = Vec::new();
let mut missing = plan.missing;
for f in plan.found {
if !still_at(&s.catalog, f.id, &f.from) {
continue;
}
if in_use.contains(&f.to) {
missing += 1;
continue;
}
in_use.insert(f.to.clone());
found.push(f);
}
let planned = plan.ambiguous.len();
let ambiguous: Vec<AmbiguousFile> = plan.ambiguous.into_iter().filter(|a| still_at(&s.catalog, a.id, &a.from)).collect();
// ambiguous photos relinked meanwhile aren't missing any more
missing = missing.saturating_sub(planned - ambiguous.len());
if !found.is_empty() {
s.commit("Find Missing Photos", Op::Batch { ops: found.iter().map(|f| relink_op(f.id, &f.to)).collect() })?;
for f in &found {
if let Some(id) = f["id"].as_u64() {
s.media.forget(PhotoId(id));
}
s.media.forget(f.id);
}
}
Ok(json!({"found": found, "missing": still, "ambiguous": ambiguous}))
Ok(FindPlan { found, missing, ambiguous }.to_json())
}
/// Search `folder` (recursively) for each missing photo's file ([`plan_find_missing`]) and
/// relink every match in one undoable step. With `found` instead (what a search on a worker
/// thread returned, as `{found, missing, ambiguous}`), only the relinking is done.
fn find_missing(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.findMissing";
if p.get("found").is_some_and(Value::is_array) {
return commit_find(s, FindPlan::from_json(p));
}
let folder = str_param(p, "folder").ok_or_else(|| bad(C, "missing `folder`"))?;
let plan = plan_find_missing(&find_candidates(&s.catalog), folder).map_err(|e| bad(C, e))?;
commit_find(s, plan)
}
pub fn specs() -> Vec<CommandSpec> {
@@ -213,7 +355,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Find Missing Photos",
[],
None,
"{folder} — relink every missing photo whose file is somewhere in the folder: same name and size (and content, when the library knows its hash), or — renamed — same size and content → {found: [{id, from, to, by: name|content}], missing, ambiguous: [{id, from, candidates}] (several same-name candidates and no hash to tell them apart: skipped)}",
"{folder} — relink every missing photo whose file is somewhere in the folder: same name and size (and content, when the library knows its hash), or — renamed — same size and content; or {found, missing, ambiguous} — relink what a search already found (the app searches on a worker thread; photos relinked meanwhile and files now in use are skipped) → {found: [{id, from, to, by: name|content}], missing, ambiguous: [{id, from, candidates}] (several same-name candidates and no hash to tell them apart: skipped)}",
always,
find_missing
),
+2 -2
View File
@@ -5,7 +5,7 @@
//! `edit.*` (undo/redo), and queries (`catalog.query`, `photo.inspect`, `develop.get`…).
mod before;
mod browse;
pub(crate) mod browse;
mod color;
pub(crate) mod convert;
mod cull;
@@ -15,7 +15,7 @@ mod edit;
mod export;
pub mod filters;
pub mod keywords;
mod library;
pub mod library;
pub mod lut_profiles;
pub mod manage;
mod masks;
+129 -38
View File
@@ -23,6 +23,12 @@ pub struct PreviewBuild {
pub failed: AtomicUsize,
pub cancel: AtomicBool,
pub finished: AtomicBool,
/// Damaged smart previews rebuilt (Build Smart Previews only).
pub repaired: AtomicUsize,
/// What is built ("" = previews; "smart previews" — Build Smart Previews runs the same way).
pub what: &'static str,
/// Why the whole build failed (e.g. the smart previews folder isn't there).
pub error: std::sync::Mutex<Option<String>>,
}
impl PreviewBuild {
@@ -33,8 +39,15 @@ impl PreviewBuild {
"failed": self.failed.load(Ordering::Relaxed),
"running": !self.finished.load(Ordering::Relaxed),
"cancelled": self.cancel.load(Ordering::Relaxed),
"what": if self.what.is_empty() { "previews" } else { self.what },
"repaired": self.repaired.load(Ordering::Relaxed),
"error": self.error(),
})
}
pub fn error(&self) -> Option<String> {
self.error.lock().unwrap_or_else(std::sync::PoisonError::into_inner).clone()
}
}
/// The jobs that build `id`'s previews: the largest grid thumbnail, then the view render at
@@ -94,56 +107,134 @@ fn build(s: &mut Session, p: &Value) -> Result<Value> {
Ok(state.json())
}
/// One photo's smart preview to build or discard: (photo, its file in the smart previews folder,
/// how to load the original at preview size).
#[cfg(not(target_arch = "wasm32"))]
type SmartJob = (lightcraft_catalog::PhotoId, std::path::PathBuf, Option<crate::media::SourceRef>);
/// What Build / Discard Smart Previews did.
#[cfg(not(target_arch = "wasm32"))]
#[derive(Default)]
struct SmartCounts {
/// Proxies there now (complete ones kept, missing or damaged ones written).
built: usize,
/// Of those, damaged ones (cut short by a crash or a full drive) that were rebuilt.
repaired: usize,
removed: usize,
/// `[[id, why]]`
failed: Vec<Value>,
}
/// The file-system half of Build / Discard Smart Previews (runs on a worker thread in the app).
/// `custom`: the folder was chosen (on another drive, perhaps): it must be there, it is never
/// recreated. With `state`, progress and cancel.
#[cfg(not(target_arch = "wasm32"))]
fn smart_run(
dir: &std::path::Path,
custom: bool,
discard: bool,
jobs: Vec<SmartJob>,
state: Option<&PreviewBuild>,
) -> std::result::Result<SmartCounts, String> {
let mut n = SmartCounts::default();
if !discard {
// a chosen folder on another drive must be there: never recreate it on this one
if custom {
crate::smart::check_writable(dir)?;
} else {
std::fs::create_dir_all(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
}
}
for (id, path, source) in jobs {
if state.is_some_and(|st| st.cancel.load(Ordering::Relaxed)) {
break;
}
if discard {
if std::fs::remove_file(&path).is_ok() {
n.removed += 1;
}
} else {
// a complete proxy is kept; a missing or damaged one (cut short by a crash or a full
// drive) is (re)built
let damaged = path.exists();
if damaged && crate::smart::is_valid(&path) {
n.built += 1;
} else {
// atomic: a failed write leaves no partial proxy that would pass for a built one
let r = source
.ok_or_else(|| "nothing to build from".to_string())
.and_then(|s| s.load())
.and_then(|src| crate::smart::encode(&src))
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&path, &b).map_err(|e| format!("{}: {e}", path.display())));
match r {
Ok(()) => {
n.built += 1;
if damaged {
n.repaired += 1;
if let Some(st) = state {
st.repaired.fetch_add(1, Ordering::Relaxed);
}
}
}
Err(e) => {
n.failed.push(json!([id.0, e]));
if let Some(st) = state {
st.failed.fetch_add(1, Ordering::Relaxed);
}
}
}
}
}
if let Some(st) = state {
st.done.fetch_add(1, Ordering::Relaxed);
}
}
Ok(n)
}
/// Smart previews: build (from the original, at preview size) or discard the proxies that keep
/// photos editable while their originals are offline.
/// photos editable while their originals are offline. With `background` (the app) the files are
/// read and written on a worker thread; progress like Build Previews (`library.previewProgress`).
#[cfg(not(target_arch = "wasm32"))]
fn smart(s: &mut Session, p: &Value) -> Result<Value> {
const C: &str = "library.smartPreviews";
let dir = s.media.smart_dir.clone().ok_or_else(|| bad(C, "smart previews need a library on disk"))?;
let discard = p.get("discard").and_then(Value::as_bool).unwrap_or(false);
let ids = if p.get("ids").is_some() || !s.selection.ids.is_empty() { s.targets(p) } else { s.visible_cloned() };
let (mut built, mut repaired, mut removed, mut failed) = (0usize, 0usize, 0usize, Vec::new());
if !discard {
// a chosen folder on another drive must be there: never recreate it on this one
if s.smart_previews_dir.is_some() {
crate::smart::check_writable(&dir).map_err(|e| bad(C, e))?;
} else {
std::fs::create_dir_all(&dir).map_err(|e| bad(C, format!("{}: {e}", dir.display())))?;
}
let background = p.get("background").and_then(Value::as_bool).unwrap_or(false);
if background && s.preview_build.as_ref().is_some_and(|b| !b.finished.load(Ordering::Relaxed)) {
return Err(bad(C, "a preview build is already running"));
}
let ids = if p.get("ids").is_some() || !s.selection.ids.is_empty() { s.targets(p) } else { s.visible_cloned() };
let mut jobs: Vec<SmartJob> = Vec::new();
for id in ids {
let Some(ph) = s.catalog.photo(id).cloned() else { continue };
if !matches!(ph.source, lightcraft_catalog::Source::File { .. }) {
continue;
}
let path = dir.join(crate::smart::file_name(&ph));
if discard {
if std::fs::remove_file(&path).is_ok() {
removed += 1;
}
continue;
}
// a complete proxy is kept; a missing or damaged one (cut short by a crash or a full
// drive) is (re)built
let damaged = path.exists();
if damaged && crate::smart::is_valid(&path) {
built += 1;
continue;
}
// atomic: a failed write leaves no partial proxy that would pass for a built one
let r = s
.source_now(id, crate::media::SourceLevel::Preview)
.and_then(|src| crate::smart::encode(&src))
.and_then(|b| lightcraft_catalog::safe_file::write_atomic(&path, &b).map_err(|e| format!("{}: {e}", path.display())));
match r {
Ok(()) => {
built += 1;
repaired += usize::from(damaged);
}
Err(e) => failed.push(json!([id.0, e])),
}
let source = (!discard).then(|| s.media.source_ref(&ph, crate::media::SourceLevel::Preview));
jobs.push((id, path, source));
}
Ok(json!({"built": built, "repaired": repaired, "removed": removed, "failed": failed}))
let custom = s.smart_previews_dir.is_some();
if background {
let what = if discard { "discarding smart previews" } else { "smart previews" };
let state = Arc::new(PreviewBuild { total: jobs.len(), what, ..Default::default() });
s.preview_build = Some(state.clone());
let st = state.clone();
std::thread::Builder::new()
.name("lc-smart-previews".into())
.spawn(move || {
if let Err(e) = smart_run(&dir, custom, discard, jobs, Some(&st)) {
log::warn!("{C}: {e}");
*st.error.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Some(e);
}
st.finished.store(true, Ordering::Relaxed);
})
.map_err(|e| bad(C, format!("could not start: {e}")))?;
return Ok(state.json());
}
let r = smart_run(&dir, custom, discard, jobs, None).map_err(|e| bad(C, e))?;
Ok(json!({"built": r.built, "repaired": r.repaired, "removed": r.removed, "failed": r.failed}))
}
/// The smart previews folder: where it is, whether it is custom, what it holds, whether it can be
@@ -225,7 +316,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Build Smart Previews",
[],
None,
"{ids?, discard?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline; a damaged proxy (cut short) is rebuilt → {built, repaired, removed, failed}",
"{ids?, discard?: bool, background?: bool} — build (or discard) the smart previews of the selected photos (else all in view): compact proxies in the library that keep photos editable and exportable (at proxy size) while their originals are offline; a damaged proxy (cut short) is rebuilt → {built, repaired, removed, failed}; with background (the app's menu) the files are read and written on a worker thread → {total, done, failed, repaired, running, what} like library.buildPreviews (progress: library.previewProgress, stop: library.cancelPreviews)",
always,
smart
),
@@ -256,7 +347,7 @@ pub fn specs() -> Vec<CommandSpec> {
always,
build
),
cmd!(query "library.previewProgress", "Preview Build Progress", [], None, "{} → {total, done, failed, running, cancelled} | null", always, |s, _| {
cmd!(query "library.previewProgress", "Preview Build Progress", [], None, "{} → {total, done, failed, repaired, running, cancelled, what, error} | null", always, |s, _| {
Ok(s.preview_build.as_ref().map_or(Value::Null, |b| b.json()))
}),
cmd!("library.cancelPreviews", "Cancel Preview Build", [], None, "{}", always, |s, _| {
+1 -1
View File
@@ -113,7 +113,7 @@ pub fn specs() -> Vec<CommandSpec> {
.map(|r| json!({"group": r.group, "tag": r.tag, "name": r.name, "value": r.value}))
.collect();
// XMP: the sidecar if there is one, else the file's own packet
let packet = crate::sidecar::read_packet(path, ph.kind, s.xmp.naming).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let packet = crate::sidecar::read_packet(path, ph.kind, s.sidecar_naming(ph.id)).map(|(x, _)| x).or_else(|| lightcraft_meta::embedded(&bytes).xmp);
let xmp: Vec<Value> = packet
.and_then(|x| lightcraft_meta::parse_xmp(&x).ok())
.map(|d| d.properties.into_iter().filter(|(k, _)| !k.starts_with("lc:")).map(|(k, v)| json!({"name": k, "value": v.join("; ")})).collect())
+16 -4
View File
@@ -9,14 +9,26 @@ use crate::{Result, Session};
fn save(s: &mut Session, p: &Value) -> Result<Value> {
let mut written = Vec::new();
let mut merged = Vec::new();
let mut backups = Vec::new();
let mut failed = Vec::new();
let owners = crate::sidecar::StemOwners::of(&s.catalog);
for id in s.targets(p) {
match s.save_sidecar(id) {
Ok(path) => written.push(path.display().to_string()),
match s.save_sidecar_with(id, &owners) {
Ok(r) => {
let path = r.path.display().to_string();
if r.merged {
merged.push(path.clone());
}
if let Some(b) = r.backup {
backups.push(json!({"path": path, "backup": b.display().to_string()}));
}
written.push(path);
}
Err(e) => failed.push(json!({"id": id.0, "error": e.to_string()})),
}
}
Ok(json!({"written": written, "failed": failed}))
Ok(json!({"written": written, "merged": merged, "backups": backups, "failed": failed}))
}
fn read(s: &mut Session, p: &Value) -> Result<Value> {
@@ -63,7 +75,7 @@ pub fn specs() -> Vec<CommandSpec> {
"Save Metadata to File",
["Photo"],
Some("Cmd+S"),
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original → {written: [path], failed}",
"{ids?} — writes each photo's XMP sidecar (metadata + develop settings) next to the original, merged into an existing one (other apps' data kept; an unreadable one is backed up first) → {written: [path], merged: [path], backups: [{path, backup}], failed}",
has_selection,
save
),
+370 -183
View File
@@ -517,205 +517,392 @@ pub fn import(s: &mut Session, paths: &[String], mode: ImportMode) -> crate::Res
}
/// Import files/folders with options (preset, keywords). See the module docs.
///
/// The file-system half ([`ImportJob::prepare`]: expanding folders, probing, reading sidecars,
/// copying / placing files) and the catalog half ([`commit_prepared`]) are separate so the app can
/// run the first on a worker thread; here both run in turn.
pub fn import_with(s: &mut Session, paths: &[String], opts: &ImportOptions) -> crate::Result<ImportReport> {
let mode = opts.mode;
// Only a library on disk has an `Originals/` folder. The browser build keeps the bytes of every
// added file in its own storage already, so "copy" there means "add".
let transfer = matches!(mode, ImportMode::Copy | ImportMode::Move);
let mode = if transfer && s.library.as_ref().is_some_and(|l| !l.on_disk) { ImportMode::Add } else { mode };
if mode == ImportMode::Move && opts.local {
return Err(crate::EngineError::Other("browsing a folder never moves its files".into()));
}
let lib_dir = s.library.as_ref().filter(|l| l.on_disk).map(|l| l.dir.clone());
// where copies go: the chosen folder, else the library's Originals/
let copy_root = opts
.destination
.as_deref()
.filter(|d| !d.trim().is_empty())
.map(std::path::PathBuf::from)
.or_else(|| lib_dir.as_ref().map(|l| l.join("Originals")));
if matches!(mode, ImportMode::Copy | ImportMode::Move) && copy_root.is_none() {
return Err(crate::EngineError::Other("copying or moving needs a destination folder or an open library".into()));
}
let mut seq = opts.rename_start.max(1);
let files = expand(paths, lib_dir.as_deref());
let mut report = ImportReport { scanned: files.len(), ..Default::default() };
let mut job = ImportJob::new(s, opts.clone())?;
let prepared = job.prepare(paths, &std::sync::atomic::AtomicBool::new(false));
// probes a scan left for files not imported now stay for a later import
s.import_probes.extend(std::mem::take(&mut job.cache));
commit_prepared(s, &job.opts, &job.now, prepared)
}
// existing paths and content hashes
let mut by_path: HashMap<&str, PhotoId> = HashMap::new();
let mut by_hash: HashMap<String, PhotoId> = HashMap::new();
for p in s.catalog.photos() {
if let Source::File { path } = &p.source {
by_path.insert(path.as_str(), p.id);
}
if let Some(h) = &p.content_hash {
by_hash.insert(h.clone(), p.id);
}
}
let mut todo = Vec::new();
// a file that was only browsed (Local) joins the library when it is imported for real
let mut promote = Vec::new();
for f in files {
match by_path.get(f.as_str()) {
Some(id) if !opts.local && s.catalog.photo(*id).is_some_and(|p| p.local) => {
promote.push(Op::SetLocal { id: *id, local: false });
report.imported.push(id.0);
/// Everything the file-system half of an import needs, detached from the session so it can run
/// on another thread (a network share or a card can take minutes). Built by [`ImportJob::new`];
/// [`ImportJob::prepare`] may be called for several batches in turn (renamed copies keep counting,
/// duplicates are found across batches); each batch is then [`commit_prepared`] on the session.
pub struct ImportJob {
pub opts: ImportOptions,
/// The mode in effect (Copy / Move become Add in a library without its own folder).
mode: ImportMode,
lib_dir: Option<PathBuf>,
copy_root: Option<PathBuf>,
/// path → (photo, a Local browse record?)
by_path: HashMap<String, (PhotoId, bool)>,
/// content hash → the photo that has it (`None`: a file earlier in this import)
by_hash: HashMap<String, Option<PhotoId>>,
probe: Option<crate::media::FileProbe>,
file_bytes: Option<crate::merge::ByteReader>,
/// Probes from a preceding scan (not read again).
cache: HashMap<String, ProbeInfo>,
naming: crate::sidecar::SidecarNaming,
now: String,
seq: usize,
/// Files found so far (folders expanded) and handled so far, for progress.
pub total: std::sync::Arc<std::sync::atomic::AtomicUsize>,
pub done: std::sync::Arc<std::sync::atomic::AtomicUsize>,
}
/// The outcome of [`ImportJob::prepare`] for one batch, in file order.
#[derive(Debug, Default)]
pub struct Prepared {
pub scanned: usize,
items: Vec<PreparedItem>,
}
#[derive(Debug)]
enum PreparedItem {
/// A Local (browsed) record imported for real.
Promote(PhotoId),
Duplicate {
path: String,
existing: Option<PhotoId>,
reason: &'static str,
hash: Option<String>,
},
Failed(String, String),
Kept(Kept),
Ready(Box<ReadyFile>),
}
#[derive(Debug)]
struct ReadyFile {
path: String,
stored: String,
info: ProbeInfo,
sidecar: Option<crate::sidecar::SidecarData>,
placed: Option<crate::import_move::Placed>,
}
impl Prepared {
/// Undo what the batch did on disk that only a commit would make permanent: files placed by a
/// Move (their sources were never touched). Copies stay, like copies of a failed commit.
pub fn rollback(&self) {
for it in &self.items {
if let PreparedItem::Ready(r) = it
&& let Some(pl) = &r.placed
{
crate::import_move::rollback(pl);
}
Some(id) => report.duplicates.push(Duplicate { path: f, existing: Some(id.0), reason: "path" }),
None => todo.push(f),
}
}
// files a preceding scan already probed are not read again (a network share is slow to read)
let cached: Vec<Option<ProbeInfo>> = todo.iter().map(|f| s.import_probes.remove(f)).collect();
let missing: Vec<String> = todo.iter().zip(&cached).filter(|(_, c)| c.is_none()).map(|(f, _)| f.clone()).collect();
let mut fresh = probe_all(s.media.file_probe.as_ref(), &missing, &ScanProgress::default()).into_iter();
let probed: Vec<Result<ProbeInfo, String>> =
cached.into_iter().map(|c| c.map(Ok).unwrap_or_else(|| fresh.next().unwrap_or_else(|| Err("not probed".into())))).collect();
crate::memory::release();
let now = (s.clock)();
let mut ops = promote;
// Move: placed at the destination, sources untouched until the records are committed
let mut placed: Vec<crate::import_move::Placed> = Vec::new();
for (path, info) in todo.into_iter().zip(probed) {
let info = match info {
Ok(i) => i,
Err(e) => {
log::warn!("import {path}: {e}");
report.failed.push((path, e));
/// Entries in the batch (after expanding folders).
pub fn len(&self) -> usize {
self.items.len()
}
pub fn is_empty(&self) -> bool {
self.items.is_empty()
}
}
impl ImportJob {
/// Snapshot what an import with `opts` needs from the session (no file-system calls). Takes
/// the probes a preceding scan left ([`Session::import_probes`]).
pub fn new(s: &mut Session, opts: ImportOptions) -> crate::Result<Self> {
let mode = opts.mode;
// Only a library on disk has an `Originals/` folder. The browser build keeps the bytes of
// every added file in its own storage already, so "copy" there means "add".
let transfer = matches!(mode, ImportMode::Copy | ImportMode::Move);
let mode = if transfer && s.library.as_ref().is_some_and(|l| !l.on_disk) { ImportMode::Add } else { mode };
if mode == ImportMode::Move && opts.local {
return Err(crate::EngineError::Other("browsing a folder never moves its files".into()));
}
let lib_dir = s.library.as_ref().filter(|l| l.on_disk).map(|l| l.dir.clone());
// where copies go: the chosen folder, else the library's Originals/
let copy_root = opts
.destination
.as_deref()
.filter(|d| !d.trim().is_empty())
.map(std::path::PathBuf::from)
.or_else(|| lib_dir.as_ref().map(|l| l.join("Originals")));
if matches!(mode, ImportMode::Copy | ImportMode::Move) && copy_root.is_none() {
return Err(crate::EngineError::Other("copying or moving needs a destination folder or an open library".into()));
}
let mut by_path = HashMap::new();
let mut by_hash = HashMap::new();
for p in s.catalog.photos() {
if let Source::File { path } = &p.source {
by_path.insert(path.clone(), (p.id, p.local));
}
if let Some(h) = &p.content_hash {
by_hash.insert(h.clone(), Some(p.id));
}
}
Ok(ImportJob {
seq: opts.rename_start.max(1),
opts,
mode,
lib_dir,
copy_root,
by_path,
by_hash,
probe: s.media.file_probe.clone(),
file_bytes: s.media.file_bytes.clone(),
cache: std::mem::take(&mut s.import_probes),
naming: s.xmp.naming,
now: (s.clock)(),
total: Default::default(),
done: Default::default(),
})
}
/// The file-system half of importing `paths`: expand folders, skip known paths, probe, skip
/// duplicates by content, read sidecars, copy or place (Move) the files. Nothing in the
/// catalog changes. Stops before the next file once `cancel` is set.
pub fn prepare(&mut self, paths: &[String], cancel: &std::sync::atomic::AtomicBool) -> Prepared {
let files = self.expand(paths);
self.prepare_files(files, cancel)
}
/// Expand `paths` (folders recursively, the library's own folder skipped) into the files an
/// import would handle; counted in [`ImportJob::total`].
pub fn expand(&self, paths: &[String]) -> Vec<String> {
let files = expand(paths, self.lib_dir.as_deref());
self.total.fetch_add(files.len(), std::sync::atomic::Ordering::Relaxed);
files
}
/// When the import started ([`Session::clock`]), for [`commit_prepared`].
pub fn now(&self) -> &str {
&self.now
}
/// [`ImportJob::prepare`] for files [`ImportJob::expand`] returned.
pub fn prepare_files(&mut self, files: Vec<String>, cancel: &std::sync::atomic::AtomicBool) -> Prepared {
use std::sync::atomic::Ordering::Relaxed;
let opts = self.opts.clone();
let mode = self.mode;
let mut out = Prepared { scanned: files.len(), items: Vec::new() };
let mut todo = Vec::new();
for f in files {
match self.by_path.get(f.as_str()).copied() {
Some((id, true)) if !opts.local => {
// a file that was only browsed (Local) joins the library when it is imported for real
self.by_path.insert(f, (id, false));
out.items.push(PreparedItem::Promote(id));
self.done.fetch_add(1, Relaxed);
}
Some((id, _)) => {
out.items.push(PreparedItem::Duplicate { path: f, existing: Some(id), reason: "path", hash: None });
self.done.fetch_add(1, Relaxed);
}
None => todo.push(f),
}
}
// files a preceding scan already probed are not read again (a network share is slow to read)
let cached: Vec<Option<ProbeInfo>> = todo.iter().map(|f| self.cache.remove(f)).collect();
let missing: Vec<String> = todo.iter().zip(&cached).filter(|(_, c)| c.is_none()).map(|(f, _)| f.clone()).collect();
let progress = ScanProgress::default();
let mut fresh = probe_all(self.probe.as_ref(), &missing, &progress).into_iter();
let probed: Vec<Result<ProbeInfo, String>> =
cached.into_iter().map(|c| c.map(Ok).unwrap_or_else(|| fresh.next().unwrap_or_else(|| Err("not probed".into())))).collect();
crate::memory::release();
for (path, info) in todo.into_iter().zip(probed) {
if cancel.load(Relaxed) {
break;
}
self.done.fetch_add(1, Relaxed);
let mut info = match info {
Ok(i) => i,
Err(e) => {
log::warn!("import {path}: {e}");
out.items.push(PreparedItem::Failed(path, e));
continue;
}
};
if let Some(h) = &info.content_hash
&& !opts.local
&& let Some(existing) = self.by_hash.get(h)
{
out.items.push(PreparedItem::Duplicate { path, existing: *existing, reason: "content", hash: Some(h.clone()) });
continue;
}
};
if let Some(h) = &info.content_hash
&& !opts.local
&& let Some(id) = by_hash.get(h)
{
report.duplicates.push(Duplicate { path, existing: Some(id.0), reason: "content" });
continue;
}
// the XMP sidecar (or a raw's embedded XMP), read before copying: its capture time files
// and names the copy when the file itself has none
let raw = info.kind == MediaKind::Raw;
let packet = crate::sidecar::find_sidecar(&path, s.xmp.naming)
.and_then(|f| std::fs::read_to_string(f).ok())
.or_else(|| info.xmp.clone().filter(|_| raw));
let sidecar = packet.and_then(|x| match crate::sidecar::parse_sidecar(&x, raw).map(|sc| sc.resolve_label(&s.catalog)) {
Ok(sc) => (sc != crate::sidecar::SidecarData::default()).then_some(sc),
Err(e) => {
log::warn!("import {path}: XMP: {e}");
None
// the XMP sidecar (or a raw's embedded XMP), read before copying: its capture time files
// and names the copy when the file itself has none
let raw = info.kind == MediaKind::Raw;
let packet = crate::sidecar::find_sidecar(&path, self.naming)
.and_then(|f| std::fs::read_to_string(f).ok())
.or_else(|| info.xmp.clone().filter(|_| raw));
let sidecar = packet.and_then(|x| match crate::sidecar::parse_sidecar(&x, raw) {
Ok(sc) => Some(sc),
Err(e) => {
log::warn!("import {path}: XMP: {e}");
None
}
});
if info.captured.is_none() {
info.captured = sidecar.as_ref().and_then(|sc| sc.captured.clone());
}
});
let mut info = info;
if info.captured.is_none() {
info.captured = sidecar.as_ref().and_then(|sc| sc.captured.clone());
}
let stored = match (mode, &copy_root) {
(ImportMode::Copy | ImportMode::Move, Some(root))
if !crate::import_move::inside(Path::new(&path), root)
&& !lib_dir.as_ref().is_some_and(|l| crate::import_move::inside(Path::new(&path), l)) =>
let mut placed = None;
let stored = match (mode, &self.copy_root) {
(ImportMode::Copy | ImportMode::Move, Some(root))
if !crate::import_move::inside(Path::new(&path), root)
&& !self.lib_dir.as_ref().is_some_and(|l| crate::import_move::inside(Path::new(&path), l)) =>
{
// the templates see the photo as it will be catalogued (undated: the import time)
let own = Path::new(&path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let mut q = Photo::new(PhotoId(0), Source::File { path: path.clone() }, &own, &info.format, 0, 0, &self.now);
q.captured = info.captured.clone();
q.meta = info.meta.clone();
let name = opts.rename.as_deref().filter(|t| !t.trim().is_empty()).map(|t| {
let n = crate::rename::expand(t, &q, self.seq);
self.seq += 1;
n
});
let folders = opts.organize.folders(&q);
if mode == ImportMode::Move && !crate::import_move::is_symlink(Path::new(&path)) {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
let name = name.unwrap_or(own);
match crate::import_move::place(Path::new(&path), &dir, &name) {
Ok(pl) => {
let dst = pl.dst.to_string_lossy().to_string();
placed = Some(pl);
dst
}
Err(e) => {
out.items.push(PreparedItem::Failed(path, e));
continue;
}
}
} else {
if mode == ImportMode::Move {
let reason = "a symbolic link: the file it points to was copied, the link and its target stay";
out.items.push(PreparedItem::Kept(Kept { path: path.clone(), reason: reason.into() }));
}
match copy_into(root, &path, &folders, name.as_deref()) {
Ok(p) => p,
Err(e) => {
out.items.push(PreparedItem::Failed(path, e));
continue;
}
}
}
}
(ImportMode::Move, _) => {
let reason = "already inside the destination or the library: added where it is";
out.items.push(PreparedItem::Kept(Kept { path: path.clone(), reason: reason.into() }));
path.clone()
}
_ => path.clone(),
};
if let Some(h) = &info.content_hash {
self.by_hash.insert(h.clone(), None);
}
// Copy as DNG: the copied raw becomes a DNG (the copy is ours to replace)
let mut stored = stored;
if opts.convert_dng
&& mode == ImportMode::Copy
&& stored != path
&& info.kind == MediaKind::Raw
&& !info.format.eq_ignore_ascii_case("DNG")
{
// the templates see the photo as it will be catalogued (undated: the import time)
let own = Path::new(&path).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
let mut q = Photo::new(PhotoId(0), Source::File { path: path.clone() }, &own, &info.format, 0, 0, &now);
q.captured = info.captured.clone();
q.meta = info.meta.clone();
let name = opts.rename.as_deref().filter(|t| !t.trim().is_empty()).map(|t| {
let n = crate::rename::expand(t, &q, seq);
seq += 1;
n
match crate::cmd::convert::write_dng_with(self.file_bytes.as_ref(), &stored, String::new()) {
Ok(dng) => {
let _ = std::fs::remove_file(&stored);
stored = dng;
info.format = "DNG".into();
}
Err(e) => log::warn!("import {path}: copy as DNG: {e}"),
}
}
out.items.push(PreparedItem::Ready(Box::new(ReadyFile { path, stored, info, sidecar, placed })));
}
out
}
}
/// The catalog half of an import: add the photos [`ImportJob::prepare`] readied (one undoable op;
/// a Move's sources are removed once it is saved). On a failed commit a Move's placed files are
/// taken back.
pub fn commit_prepared(s: &mut Session, opts: &ImportOptions, now: &str, prepared: Prepared) -> crate::Result<ImportReport> {
let now = now.to_string();
let mut report = ImportReport { scanned: prepared.scanned, ..Default::default() };
let mut ops = Vec::new();
// Move: placed at the destination, sources untouched until the records are committed
let mut placed: Vec<crate::import_move::Placed> = Vec::new();
// photos added by this batch, by content (a duplicate of a file earlier in the import names it)
let mut new_hash: HashMap<String, PhotoId> = HashMap::new();
for it in prepared.items {
match it {
PreparedItem::Promote(id) => {
if s.catalog.photo(id).is_some_and(|p| p.local) {
ops.push(Op::SetLocal { id, local: false });
}
report.imported.push(id.0);
}
PreparedItem::Duplicate { path, existing, reason, hash } => {
let existing = existing.or_else(|| {
let h = hash?;
new_hash.get(&h).copied().or_else(|| s.catalog.photos().find(|p| p.content_hash.as_deref() == Some(h.as_str())).map(|p| p.id))
});
let folders = opts.organize.folders(&q);
if mode == ImportMode::Move && !crate::import_move::is_symlink(Path::new(&path)) {
let dir = folders.iter().fold(root.to_path_buf(), |d, f| d.join(f));
let name = name.unwrap_or(own);
match crate::import_move::place(Path::new(&path), &dir, &name) {
Ok(pl) => {
let dst = pl.dst.to_string_lossy().to_string();
placed.push(pl);
dst
}
Err(e) => {
report.failed.push((path, e));
continue;
}
}
} else {
if mode == ImportMode::Move {
let reason = "a symbolic link: the file it points to was copied, the link and its target stay";
report.kept.push(Kept { path: path.clone(), reason: reason.into() });
}
match copy_into(root, &path, &folders, name.as_deref()) {
Ok(p) => p,
Err(e) => {
report.failed.push((path, e));
continue;
}
report.duplicates.push(Duplicate { path, existing: existing.map(|i| i.0), reason });
}
PreparedItem::Failed(path, e) => report.failed.push((path, e)),
PreparedItem::Kept(k) => report.kept.push(k),
PreparedItem::Ready(r) => {
let ReadyFile { path, stored, info, sidecar, placed: pl } = *r;
placed.extend(pl);
let id = s.catalog.alloc_photo_id();
if let Some(h) = &info.content_hash {
new_hash.insert(h.clone(), id);
}
let sidecar = sidecar.map(|sc| sc.resolve_label(&s.catalog)).filter(|sc| *sc != crate::sidecar::SidecarData::default());
// a copy is catalogued under its new name
let name = Path::new(&stored).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| path.clone());
let mut p = Photo::new(id, Source::File { path: stored }, &name, &info.format, info.width, info.height, &now);
p.kind = info.kind;
p.file_size = info.file_size;
p.captured = info.captured;
p.meta = info.meta;
p.as_shot_wb = info.as_shot_wb;
p.content_hash = info.content_hash;
p.embedded_lens = info.embedded_lens;
p.preview_only = info.preview_only.clone();
apply_import_defaults(s, &mut p);
if let Some(sc) = &sidecar {
crate::sidecar::merge_into(&mut p, sc, &now);
report.sidecars += 1;
}
if let Some(mp) = opts.metadata_preset.as_ref().and_then(|n| s.metadata_presets.iter().find(|m| m.name.eq_ignore_ascii_case(n))) {
crate::cmd::metadata::apply_to(&mut p.meta, &mp.fields);
}
for k in &opts.keywords {
let k = lightcraft_catalog::keywords::clean(k);
if !k.is_empty() && !p.meta.keywords.iter().any(|x| x.eq_ignore_ascii_case(&k)) {
p.meta.keywords.push(k);
}
}
}
(ImportMode::Move, _) => {
let reason = "already inside the destination or the library: added where it is";
report.kept.push(Kept { path: path.clone(), reason: reason.into() });
path.clone()
}
_ => path.clone(),
};
let id = s.catalog.alloc_photo_id();
if let Some(h) = &info.content_hash {
by_hash.insert(h.clone(), id);
}
// Copy as DNG: the copied raw becomes a DNG (the copy is ours to replace)
let mut stored = stored;
if opts.convert_dng && mode == ImportMode::Copy && stored != path && info.kind == MediaKind::Raw && !info.format.eq_ignore_ascii_case("DNG") {
match crate::cmd::convert::write_dng_for(s, &stored, String::new()) {
Ok(dng) => {
let _ = std::fs::remove_file(&stored);
stored = dng;
info.format = "DNG".into();
if let Some(preset) = &opts.preset {
let d = std::sync::Arc::new(preset.apply(&p.develop, 1.0));
let label = format!("Preset: {}", preset.name);
p.develop = d.clone();
p.edited = Some(now.clone());
p.history.push(lightcraft_catalog::HistoryStep { label, settings: d });
}
Err(e) => log::warn!("import {path}: copy as DNG: {e}"),
report.imported.push(id.0);
p.local = opts.local;
if opts.local {
// the state as browsed: a later change keeps the record from being forgotten
p.set_local_baseline();
}
ops.push(Op::AddPhoto { photo: Box::new(p) });
}
}
// a copy is catalogued under its new name
let name = Path::new(&stored).file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_else(|| path.clone());
let mut p = Photo::new(id, Source::File { path: stored }, &name, &info.format, info.width, info.height, &now);
p.kind = info.kind;
p.file_size = info.file_size;
p.captured = info.captured;
p.meta = info.meta;
p.as_shot_wb = info.as_shot_wb;
p.content_hash = info.content_hash;
p.embedded_lens = info.embedded_lens;
p.preview_only = info.preview_only.clone();
apply_import_defaults(s, &mut p);
if let Some(sc) = &sidecar {
crate::sidecar::merge_into(&mut p, sc, &now);
report.sidecars += 1;
}
if let Some(mp) = opts.metadata_preset.as_ref().and_then(|n| s.metadata_presets.iter().find(|m| m.name.eq_ignore_ascii_case(n))) {
crate::cmd::metadata::apply_to(&mut p.meta, &mp.fields);
}
for k in &opts.keywords {
let k = lightcraft_catalog::keywords::clean(k);
if !k.is_empty() && !p.meta.keywords.iter().any(|x| x.eq_ignore_ascii_case(&k)) {
p.meta.keywords.push(k);
}
}
if let Some(preset) = &opts.preset {
let d = std::sync::Arc::new(preset.apply(&p.develop, 1.0));
let label = format!("Preset: {}", preset.name);
p.develop = d.clone();
p.edited = Some(now.clone());
p.history.push(lightcraft_catalog::HistoryStep { label, settings: d });
}
report.imported.push(id.0);
p.local = opts.local;
if opts.local {
// the state as browsed: a later change keeps the record from being forgotten
p.set_local_baseline();
}
ops.push(Op::AddPhoto { photo: Box::new(p) });
}
let log0 = s.pending_log.len();
if !ops.is_empty()
+88 -16
View File
@@ -10,6 +10,7 @@
//! run them off the UI thread.
#![forbid(unsafe_code)]
pub mod availability;
pub mod cmd;
pub mod crs;
pub mod crs_masks;
@@ -57,6 +58,9 @@ pub enum EngineError {
/// be written. They stay queued and are written by the next successful save.
#[error("saved in memory but not written to disk: {0}; LightCraft will retry")]
NotSaved(String),
/// Another process (the app, `lightcraft-cli`, another computer) has the library open.
#[error("{0}")]
LibraryInUse(String),
#[error("{0}")]
Other(String),
}
@@ -68,6 +72,22 @@ pub type Result<T> = std::result::Result<T, EngineError>;
pub struct UndoEntry {
pub label: String,
pub op: Op,
/// A folder to rename on disk (`from` → `to`) before `op` is applied: Rename / Move Folder
/// (whose `op` relinks the photos inside). Never overwrites; refused when `to` exists.
pub folder: Option<FolderMove>,
}
/// A folder renamed or moved on disk as part of an undo step.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct FolderMove {
pub from: String,
pub to: String,
}
impl FolderMove {
fn reversed(&self) -> Self {
Self { from: self.to.clone(), to: self.from.clone() }
}
}
/// An in-progress slider drag / brush stroke: one undo step when it ends.
@@ -253,11 +273,22 @@ impl Session {
(spec.enabled)(self).map_err(|why| EngineError::Disabled(id.to_string(), why))?;
let empty = Value::Object(Default::default());
let params = if params.is_null() { &empty } else { params };
self.run_command(id, spec.journal.then_some(params), |s| (spec.run)(s, params))
}
/// Run `f` as command `id` with what [`Session::execute`] does around every command: the
/// panic guard, auto versions, XMP sidecar auto-write and the durable save (a failed save is
/// `NotSaved`). Not journaled. The app's import task commits its batches this way.
pub fn execute_fn(&mut self, id: &str, f: impl FnOnce(&mut Session) -> Result<Value>) -> Result<Value> {
self.run_command(id, None, f)
}
fn run_command(&mut self, id: &str, journal: Option<&Value>, f: impl FnOnce(&mut Session) -> Result<Value>) -> Result<Value> {
let log_start = self.pending_log.len();
let was_active = self.active();
self.depth += 1;
// last-resort guard: a panic in a command is that command's error, not a crash
let r = guard::catch(&format!("`{id}`"), || (spec.run)(self, params)).unwrap_or_else(|e| Err(EngineError::Other(e)));
let r = guard::catch(&format!("`{id}`"), || f(self)).unwrap_or_else(|e| Err(EngineError::Other(e)));
self.depth -= 1;
if self.depth == 0 && was_active.is_some() && self.active() != was_active {
self.previous_active = was_active;
@@ -265,7 +296,10 @@ impl Session {
self.auto_version(left);
}
}
if r.is_ok() && spec.journal && self.depth == 0 {
if r.is_ok()
&& let Some(params) = journal
&& self.depth == 0
{
self.journal.push((id.to_string(), params.clone()));
if self.journal.len() > 10_000 {
self.journal.drain(..1000);
@@ -302,7 +336,7 @@ impl Session {
let fwd = op.clone();
let inv = self.catalog.apply(op)?;
self.pending_log.push(fwd);
self.undo.push(UndoEntry { label: label.to_string(), op: inv });
self.undo.push(UndoEntry { label: label.to_string(), op: inv, folder: None });
if self.undo.len() > 1000 {
self.undo.remove(0);
}
@@ -310,6 +344,16 @@ impl Session {
Ok(())
}
/// [`Session::commit`] for an op that goes with a folder already renamed on disk; the undo
/// step renames it back (`folder` is the undo direction: current place → old place).
pub(crate) fn commit_with_folder(&mut self, label: &str, op: Op, folder: FolderMove) -> Result<()> {
self.commit(label, op)?;
if let Some(e) = self.undo.last_mut() {
e.folder = Some(folder);
}
Ok(())
}
/// Leaving photo `id` after editing it: keep its settings as an automatic version (when they
/// differ from its latest version; at most [`AUTO_VERSIONS`] auto versions, oldest dropped).
/// Saved with the library but not an undo step.
@@ -340,9 +384,13 @@ impl Session {
if n < 2 || n > self.undo.len() {
return;
}
// a step that moves a folder on disk stays on its own
if self.undo[self.undo.len() - n..].iter().any(|e| e.folder.is_some()) {
return;
}
let tail = self.undo.split_off(self.undo.len() - n);
let ops = tail.into_iter().rev().map(|e| e.op).collect();
self.undo.push(UndoEntry { label: label.to_string(), op: Op::Batch { ops } });
self.undo.push(UndoEntry { label: label.to_string(), op: Op::Batch { ops }, folder: None });
}
/// Apply without recording undo (interactive previews).
@@ -353,7 +401,7 @@ impl Session {
pub fn undo_step(&mut self) -> Result<String> {
let e = self.undo.pop().ok_or_else(|| EngineError::Other("nothing to undo".into()))?;
let redo = match self.apply_with_files(&e.op) {
let redo = match self.apply_with_files(&e.op, e.folder.as_ref()) {
Ok(r) => r,
Err(err) => {
self.undo.push(e);
@@ -361,13 +409,13 @@ impl Session {
}
};
self.pending_log.push(e.op);
self.redo.push(UndoEntry { label: e.label.clone(), op: redo });
self.redo.push(UndoEntry { label: e.label.clone(), op: redo, folder: e.folder.as_ref().map(FolderMove::reversed) });
Ok(e.label)
}
pub fn redo_step(&mut self) -> Result<String> {
let e = self.redo.pop().ok_or_else(|| EngineError::Other("nothing to redo".into()))?;
let undo = match self.apply_with_files(&e.op) {
let undo = match self.apply_with_files(&e.op, e.folder.as_ref()) {
Ok(r) => r,
Err(err) => {
self.redo.push(e);
@@ -375,26 +423,44 @@ impl Session {
}
};
self.pending_log.push(e.op);
self.undo.push(UndoEntry { label: e.label.clone(), op: undo });
self.undo.push(UndoEntry { label: e.label.clone(), op: undo, folder: e.folder.as_ref().map(FolderMove::reversed) });
Ok(e.label)
}
/// Apply an undo/redo op, first moving the files its renames imply (all or nothing). Files
/// that can't be moved back after a failure are reported and the library follows them (a
/// logged change outside the undo history), so none goes missing.
fn apply_with_files(&mut self, op: &Op) -> Result<Op> {
/// Apply an undo/redo op, first renaming the step's folder and moving the files its renames
/// imply (all or nothing). Files that can't be moved back after a failure are reported and the
/// library follows them (a logged change outside the undo history), so none goes missing; the
/// folder is only moved back when no file was left behind at its new path.
fn apply_with_files(&mut self, op: &Op, folder: Option<&FolderMove>) -> Result<Op> {
let fs = rename::RealFs;
if let Some(f) = folder {
cmd::browse::rename_folder_on_disk(&f.from, &f.to).map_err(|e| EngineError::Other(format!("can't move the folder back: {e}")))?;
}
let undo_folder = || {
if let Some(f) = folder {
let _ = cmd::browse::rename_folder_on_disk(&f.to, &f.from);
}
};
let moves = self.file_moves(op);
if let Err(e) = rename::move_all(&fs, &moves) {
if e.stuck.is_empty() {
undo_folder();
}
return Err(EngineError::Other(format!("can't move the files back: {}", self.follow_stuck(op, e, false))));
}
match self.catalog.apply(op.clone()) {
Ok(inv) => Ok(inv),
Ok(inv) => {
if let Some(f) = folder {
cmd::browse::follow_folder(self, &f.from, &f.to);
}
Ok(inv)
}
Err(e) => {
let back: Vec<(String, String)> = moves.iter().rev().map(|(a, b)| (b.clone(), a.clone())).collect();
if let Err(be) = rename::move_all(&fs, &back) {
return Err(EngineError::Other(format!("{e}; the files could not all be moved back: {}", be.message)));
}
undo_folder();
Err(e.into())
}
}
@@ -509,7 +575,11 @@ impl Session {
/// Photos shown in the grid/filmstrip for the current source, filter and sort.
pub fn visible(&mut self) -> &[PhotoId] {
let key = (self.catalog.revision, format!("{:?}|{:?}|{:?}|{:?}", self.source, self.filter, self.sort, self.browse));
let mut key = (self.catalog.revision, format!("{:?}|{:?}|{:?}|{:?}", self.source, self.filter, self.sort, self.browse));
if self.source == LibrarySource::Missing && self.media.availability.is_background() {
// the view fills in as the background checks find files gone
key.1.push_str(&format!("|{}", self.media.availability.generation()));
}
if self.visible_key.as_ref() != Some(&key) {
// "in the last N days" rules count back from the session's clock
lightcraft_catalog::rules::set_now(Some((self.clock)()));
@@ -544,8 +614,8 @@ impl Session {
}
if self.source == LibrarySource::Missing {
// only the photos the query kept (library photos, not Local browse records) are checked
let cat = &self.catalog;
visible.retain(|id| cmd::missing::is_missing(cat, *id));
let (cat, avail) = (&self.catalog, &self.media.availability);
visible.retain(|id| cmd::missing::is_missing(cat, avail, *id));
}
if self.source != LibrarySource::RecentlyDeleted {
visible = self.catalog.arrange_stacks(&visible);
@@ -642,6 +712,8 @@ mod tests_persist;
#[cfg(test)]
mod tests_prefs;
#[cfg(test)]
mod tests_settings_files;
#[cfg(test)]
mod tests_spots;
#[cfg(test)]
mod tests_xmp;
+149 -17
View File
@@ -23,7 +23,7 @@
use std::path::{Path, PathBuf};
use lightcraft_catalog::{FsStore, Journal, LoadReport, Store};
use lightcraft_catalog::{FsStore, Journal, LibraryLock, LoadReport, Store};
use lightcraft_develop::Preset;
use serde::{Deserialize, Serialize};
@@ -64,6 +64,15 @@ pub struct Library {
pub forgot_local: Option<lightcraft_catalog::ForgetPlan>,
presets_written: String,
view_written: Vec<u8>,
/// Settings files that were unreadable or damaged when the library opened (`library.info` →
/// `settingsWarnings`; shown by the UI once, see [`Session::take_library_warnings`]).
pub settings_warnings: Vec<String>,
warnings_reported: usize,
/// Settings files that couldn't be read or set aside: never overwritten this session.
blocked: Vec<&'static str>,
/// Keeps other processes out of this library while it's open (last: released after the
/// journal's background snapshot has landed).
lock: Option<LibraryLock>,
}
/// Where a library's files live (see [`Session::open_library_in`]).
@@ -171,23 +180,112 @@ fn poll_compaction(lib: &mut Library) {
/// How long the frame loop waits before retrying a failed append (commands retry at once).
const RETRY_BACKOFF: std::time::Duration = std::time::Duration::from_secs(2);
fn read_json<T: serde::de::DeserializeOwned>(store: &mut dyn Store, name: &str) -> Option<T> {
store.read(name).ok().flatten().and_then(|b| serde_json::from_slice(&b).ok())
/// `a` and `b` name the same directory.
fn same_dir(a: &Path, b: &Path) -> bool {
match (std::fs::canonicalize(a), std::fs::canonicalize(b)) {
(Ok(a), Ok(b)) => a == b,
_ => a == b,
}
}
/// This program, for the lock owner note ("LightCraft", "lightcraft-cli").
fn program_name() -> String {
let exe = std::env::current_exe().ok().and_then(|p| p.file_stem().map(|s| s.to_string_lossy().to_string()));
match exe.as_deref() {
Some("lightcraft") | None => "LightCraft".into(),
Some(other) => other.to_string(),
}
}
/// Problems reading the settings files (`presets.json`, `prefs.json`, `view.json`) when the library
/// opened (issue #103). A missing file is not a problem (defaults). A file that doesn't parse is
/// kept as `<name>.corrupt-<unix time>` before the defaults are used (so the next save can't
/// lose it); a file that can't be read at all (locked by another program, I/O error) is never
/// written this session, so its content survives until the library is opened again.
#[derive(Default)]
struct SettingsLoad {
warnings: Vec<String>,
/// Files not to overwrite this session.
blocked: Vec<&'static str>,
}
impl SettingsLoad {
fn read<T: serde::de::DeserializeOwned>(&mut self, store: &mut dyn Store, name: &'static str) -> Option<T> {
let bytes = match store.read(name) {
Ok(b) => b?,
Err(e) => {
log::error!("library: {name}: {e}");
self.blocked.push(name);
self.warnings.push(format!(
"{name} couldn't be read ({e}). LightCraft uses the defaults for now and won't overwrite the file; reopen the library to try again."
));
return None;
}
};
let err = match serde_json::from_slice(&bytes) {
Ok(v) => return Some(v),
Err(e) => e,
};
let secs = web_time::SystemTime::now().duration_since(web_time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0);
let keep = format!("{name}.corrupt-{secs}");
log::error!("library: {name} is damaged: {err}");
match store.write_atomic(&keep, &bytes) {
Ok(()) => self.warnings.push(format!("{name} is damaged ({err}). It was kept as {keep}, and the defaults are used.")),
Err(w) => {
self.blocked.push(name);
self.warnings.push(format!(
"{name} is damaged ({err}) and couldn't be set aside ({w}). LightCraft uses the defaults and won't overwrite the file."
));
}
}
None
}
}
impl Session {
/// Open (or create) the library at `dir` into this session, replacing its catalog. With
/// `seed_demo`, a newly created library starts with the procedural demo photos.
///
/// The library is locked for this session ([`lightcraft_catalog::lock`]): if another process
/// has it open, this fails with [`EngineError::LibraryInUse`] and nothing is read or changed.
/// Reopening the library this session already has open keeps its lock.
pub fn open_library(&mut self, dir: impl AsRef<Path>, seed_demo: bool) -> Result<&LoadReport> {
let dir = dir.as_ref().to_path_buf();
let open = || FsStore::open(&dir).map_err(|e| EngineError::Other(format!("can't open library {}: {e}", dir.display())));
let stores = LibraryStores { catalog: Box::new(open()?), files: Box::new(open()?), on_disk: true, dir: dir.clone() };
self.open_library_in(stores, seed_demo)
// the library open in this session (same directory): hand its lock over
let reused = self.library.as_mut().filter(|l| same_dir(&l.dir, &dir)).and_then(|l| l.lock.take());
let reusing = reused.is_some();
let mut lock = match reused {
Some(l) => Some(l),
None => Some(LibraryLock::acquire(&dir, &program_name()).map_err(|e| EngineError::LibraryInUse(e.to_string()))?),
};
if let Err(e) = self.open_stores(stores, seed_demo, &mut lock) {
if reusing && let Some(lib) = self.library.as_mut() {
lib.lock = lock.take();
}
return Err(e);
}
self.loaded_report()
}
/// Open (or create) a library whose files live in `stores` (e.g. browser storage), replacing
/// this session's catalog.
/// this session's catalog. Not locked: the host guards against a second opener (see
/// [`Session::open_library`] for directories).
pub fn open_library_in(&mut self, stores: LibraryStores, seed_demo: bool) -> Result<&LoadReport> {
self.open_stores(stores, seed_demo, &mut None)?;
self.loaded_report()
}
fn loaded_report(&self) -> Result<&LoadReport> {
match &self.library {
Some(lib) => Ok(&lib.report),
None => Err(EngineError::Other("library closed while opening".into())),
}
}
/// Open `stores`; the new library takes `lock` once nothing can fail any more.
fn open_stores(&mut self, stores: LibraryStores, seed_demo: bool, lock: &mut Option<LibraryLock>) -> Result<()> {
let LibraryStores { dir, catalog, mut files, on_disk } = stores;
self.media.smart_dir = on_disk.then(|| crate::smart::dir(&dir));
// the current library may be these same files: let its background snapshot land first
@@ -208,8 +306,9 @@ impl Session {
crate::demo::load(self);
journal.snapshot(&self.catalog)?;
}
let mut settings = SettingsLoad::default();
// presets
if let Some(f) = read_json::<PresetsFile>(files.as_mut(), "presets.json") {
if let Some(f) = settings.read::<PresetsFile>(files.as_mut(), "presets.json") {
for p in &mut self.presets {
p.favorite = p.builtin && f.favorites.contains(&p.id) || (!p.builtin && p.favorite);
}
@@ -223,7 +322,7 @@ impl Session {
self.profile_recent = f.profile_recent.into_iter().filter(known).take(crate::presets::RECENT_PROFILES).collect();
}
// preferences
let prefs = read_json::<PrefsFile>(files.as_mut(), "prefs.json").unwrap_or_default();
let prefs = settings.read::<PrefsFile>(files.as_mut(), "prefs.json").unwrap_or_default();
self.xmp = prefs.xmp;
self.last_export = prefs.last_export;
self.export_presets = prefs.export_presets;
@@ -244,7 +343,7 @@ impl Session {
self.media.smart_dir = Some(d.clone());
}
// view state
if let Some(v) = read_json::<ViewFile>(files.as_mut(), "view.json") {
if let Some(v) = settings.read::<ViewFile>(files.as_mut(), "view.json") {
self.source = v.source;
self.browse = v.browse;
self.filter = v.filter;
@@ -277,6 +376,10 @@ impl Session {
forgot_local: None,
presets_written,
view_written,
settings_warnings: settings.warnings,
warnings_reported: 0,
blocked: settings.blocked,
lock: lock.take(),
});
// forget untouched Local records of folders not browsed for a while (journaled at once)
if self.forget_local_days > 0 {
@@ -290,10 +393,7 @@ impl Session {
lib.forgot_local = Some(plan);
}
}
match &self.library {
Some(lib) => Ok(&lib.report),
None => Err(EngineError::Other("library closed while opening".into())),
}
Ok(())
}
/// Write pending ops to the log (fsynced), compact when due, and save changed presets.
@@ -332,7 +432,7 @@ impl Session {
}
let presets = presets_json(self);
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if presets != lib.presets_written {
if presets != lib.presets_written && !lib.blocked.contains(&"presets.json") {
if let Err(e) = lib.files.write_atomic("presets.json", presets.as_bytes()) {
log::error!("library: presets: {e}");
} else {
@@ -362,16 +462,35 @@ impl Session {
}
/// Flush everything and write a snapshot (on quit). Ends an open interaction first.
///
/// The snapshot is written even when appending the queued ops fails: it is a fresh file
/// holding everything in memory, so it saves those ops too (the log handle may be the only
/// thing that's broken). Fails only if nothing could be saved; the ops then stay queued.
pub fn close_library(&mut self) -> Result<()> {
if self.library.is_none() {
return Ok(());
}
let _ = self.end_interaction();
self.persist()?;
let persisted = self.persist();
self.save_view();
let unlogged = if persisted.is_err() { self.pending_log.len() as u64 } else { 0 };
let Some(lib) = self.library.as_mut() else { return Ok(()) };
lib.journal.snapshot(&self.catalog)?;
Ok(())
let before = lib.journal.seq();
let snapshot = lib.journal.snapshot_with_unlogged(&self.catalog, unlogged);
if unlogged > 0 && lib.journal.seq() == before + unlogged {
// the snapshot holds the queued ops: they are saved, never append them again
log::info!("library: {unlogged} queued change(s) saved in the closing snapshot");
self.pending_log.clear();
lib.unsaved_error = None;
lib.retry_at = None;
lib.last_error = None;
}
match (persisted, snapshot) {
(_, Ok(())) => Ok(()),
// neither the log nor the snapshot took the queued ops
(Err(EngineError::NotSaved(e)), Err(s)) if lib.journal.seq() == before => Err(EngineError::NotSaved(format!("{e}; snapshot: {s}"))),
(_, Err(s)) => Err(s.into()),
}
}
fn view_json(&self) -> Vec<u8> {
@@ -394,7 +513,7 @@ impl Session {
}
let v = self.view_json();
let Some(lib) = self.library.as_mut() else { return };
if v != lib.view_written {
if v != lib.view_written && !lib.blocked.contains(&"view.json") {
match lib.files.write_atomic("view.json", &v) {
Ok(()) => lib.view_written = v,
Err(e) => log::error!("library: view: {e}"),
@@ -423,9 +542,22 @@ impl Session {
})
.unwrap_or_default();
let Some(lib) = self.library.as_mut() else { return Ok(()) };
if lib.blocked.contains(&"prefs.json") {
return Err(EngineError::Other(
"prefs: prefs.json couldn't be read when the library opened, so it isn't overwritten; reopen the library to save preferences".into(),
));
}
lib.files.write_atomic("prefs.json", &v).map_err(|e| EngineError::Other(format!("prefs: {e}")))
}
/// Settings-file warnings of the open library not handed out yet (the UI shows each once).
pub fn take_library_warnings(&mut self) -> Vec<String> {
let Some(lib) = self.library.as_mut() else { return vec![] };
let new = lib.settings_warnings[lib.warnings_reported..].to_vec();
lib.warnings_reported = lib.settings_warnings.len();
new
}
/// The thumbnail disk cache budget in bytes ([`Session::cache_mb`], else the default).
pub fn cache_bytes(&self) -> u64 {
if self.cache_mb == 0 { crate::media::DISK_CACHE_BYTES } else { u64::from(self.cache_mb) << 20 }
+32 -11
View File
@@ -82,6 +82,9 @@ pub enum SourceRef {
path: String,
max_edge: usize,
loader: Option<FileLoader>,
/// The photo's smart preview, used when the original can't be read (offline drive). The
/// UI thread never checks whether the original is there; the render worker finds out.
fallback: Option<std::path::PathBuf>,
},
/// A smart preview standing in for a missing original.
Smart {
@@ -94,10 +97,17 @@ impl SourceRef {
match self {
SourceRef::Loaded(a) => Ok(a.clone()),
SourceRef::Demo { scene, max_edge } => Ok(Arc::new(scene.render_fit(*max_edge))),
SourceRef::File { path, max_edge, loader } => match loader {
Some(l) => l(path, *max_edge).map(|(img, _)| Arc::new(img)),
None => Err(format!("no decoder available for {path}")),
},
SourceRef::File { path, max_edge, loader, fallback } => {
let r = match loader {
Some(l) => l(path, *max_edge).map(|(img, _)| Arc::new(img)),
None => Err(format!("no decoder available for {path}")),
};
match (r, fallback) {
#[cfg(not(target_arch = "wasm32"))]
(Err(e), Some(sp)) if crate::smart::is_valid(sp) => crate::smart::load(sp).map_err(|e2| format!("{e}; smart preview: {e2}")),
(r, _) => r,
}
}
#[cfg(not(target_arch = "wasm32"))]
SourceRef::Smart { path } => crate::smart::load(path),
#[cfg(target_arch = "wasm32")]
@@ -125,6 +135,9 @@ pub struct MediaCache {
pub file_bytes: Option<crate::merge::ByteReader>,
/// The library's smart previews folder (originals offline: render from the proxy).
pub smart_dir: Option<std::path::PathBuf>,
/// Whether originals (and smart previews) are on disk: cached and checked off the UI thread
/// in the app (see [`crate::availability`]).
pub availability: crate::availability::Availability,
scenes: Vec<lightcraft_scenes::Scene>,
/// Rendered thumbnails (memory, plus disk once a library is attached).
pub rendered: Arc<PreviewCache>,
@@ -144,6 +157,7 @@ impl Default for MediaCache {
preview_loader: None,
file_bytes: None,
smart_dir: None,
availability: Default::default(),
scenes: Vec::new(),
rendered: Arc::new(PreviewCache::memory(rendered_budget(budget))),
}
@@ -284,15 +298,22 @@ impl MediaCache {
(Source::Demo { .. }, SourceLevel::Full) => p.width.max(p.height).max(1) as usize,
_ => usize::MAX,
});
// the original is offline: its smart preview, when there is one
// the original is offline: its smart preview, when there is one. Only cached answers are
// used here (no file-system call on the UI thread in the app); an original not known to
// be offline is tried first and its smart preview is the render worker's fallback.
#[cfg(not(target_arch = "wasm32"))]
if let (Source::File { path }, Some(dir)) = (&p.source, &self.smart_dir)
&& !std::path::Path::new(path).exists()
{
if let (Source::File { path }, Some(dir)) = (&p.source, &self.smart_dir) {
let sp = dir.join(crate::smart::file_name(p));
if crate::smart::is_valid(&sp) {
// (a proxy cut short fails to load, like the offline original; the worker's fallback
// in `SourceRef::load` only uses a proxy that passes `smart::is_valid`)
if self.availability.is_offline(path) && self.availability.exists(&sp.to_string_lossy()) == Some(true) {
return SourceRef::Smart { path: sp };
}
let mut r = self.origin_ref(&p.source, max_edge);
if let SourceRef::File { fallback, .. } = &mut r {
*fallback = Some(sp);
}
return r;
}
self.origin_ref(&p.source, max_edge)
}
@@ -307,10 +328,10 @@ impl MediaCache {
}
match self.scenes.iter().find(|s| s.id == *scene) {
Some(s) => SourceRef::Demo { scene: Box::new(s.clone()), max_edge },
None => SourceRef::File { path: format!("demo:{scene}"), max_edge, loader: None },
None => SourceRef::File { path: format!("demo:{scene}"), max_edge, loader: None, fallback: None },
}
}
Source::File { path } => SourceRef::File { path: path.clone(), max_edge, loader: self.file_loader.clone() },
Source::File { path } => SourceRef::File { path: path.clone(), max_edge, loader: self.file_loader.clone(), fallback: None },
}
}
}
+87 -64
View File
@@ -538,6 +538,92 @@ fn set_file_ops_to(op: &Op, to: &HashSet<&str>, out: &mut Vec<Op>) {
}
}
/// The photos among `ids` a rename handles, in order: photos sharing one file (virtual copies)
/// rename once. No file-system calls.
pub fn rename_photos(cat: &lightcraft_catalog::Catalog, ids: &[PhotoId]) -> Vec<std::sync::Arc<Photo>> {
let mut seen_paths: HashSet<String> = HashSet::new();
ids.iter()
.filter_map(|id| cat.photo(*id))
.filter(|p| match &p.source {
Source::File { path } => seen_paths.insert(path.clone()),
_ => true,
})
.cloned()
.collect()
}
/// [`Session::plan_rename`] for [`rename_photos`], with the "is this name taken on disk?" check
/// supplied: the Rename dialog plans its preview on a worker thread (a check can block on a slow
/// drive). Whether a case-only change names this very file is decided on the disk.
pub fn plan_rename_photos(photos: &[std::sync::Arc<Photo>], template: &str, start: usize, exists: &dyn Fn(&str) -> bool) -> Vec<RenamePlan> {
plan_rename_core(photos, template, start, &|p| exists(&p.to_string_lossy()), &|a, b| RealFs.same_file(a, b))
}
/// Plan renaming `photos`: `exists` says whether something answers to a path, `same_file` whether
/// two spellings name one file (see [`MoveFs`]).
fn plan_rename_core(
photos: &[std::sync::Arc<Photo>],
template: &str,
start: usize,
exists: &dyn Fn(&Path) -> bool,
same_file: &dyn Fn(&Path, &Path) -> bool,
) -> Vec<RenamePlan> {
let mut taken: HashSet<String> = HashSet::new(); // lower-case target paths/names claimed in this batch
let mut out = Vec::new();
for (seq, p) in (start..).zip(photos) {
let want = expand(template, p, seq);
let (stem, ext) = split_ext(&want);
let (stem, ext) = (stem.to_string(), ext.to_string());
let candidate = |k: usize| {
if k == 0 {
want.clone()
} else if ext.is_empty() {
format!("{stem}-{k}")
} else {
format!("{stem}-{k}.{ext}")
}
};
let (to, to_path) = match &p.source {
Source::File { path } => {
let dir = Path::new(path).parent().map(Path::to_path_buf).unwrap_or_default();
let mut k = 0;
loop {
let name = candidate(k);
let tp = dir.join(&name).to_string_lossy().to_string();
let key = tp.to_lowercase();
// this very file, maybe spelled differently (case-insensitive volume)? By
// identity: on a case-sensitive volume `img_1.jpg` may be another photo
let (from, to) = (Path::new(path), Path::new(&tp));
let same = tp == *path || (from != to && key == path.to_lowercase() && exists(to) && same_file(from, to));
// free: not claimed in this batch and not on disk (unless it is this very file).
// A file this batch moves away still counts as taken: simple and safe.
if !taken.contains(&key) && (same || !exists(to)) {
taken.insert(key);
break (name, Some(tp));
}
k += 1;
}
}
Source::Demo { .. } => {
let mut k = 0;
loop {
let name = candidate(k);
if taken.insert(format!("demo:{}", name.to_lowercase())) {
break (name, None);
}
k += 1;
}
}
};
let from_path = match &p.source {
Source::File { path } => Some(path.clone()),
_ => None,
};
out.push(RenamePlan { id: p.id.0, from: p.file_name.clone(), to, from_path, to_path });
}
out
}
impl Session {
/// Plan renaming `ids` with `template` (sequence numbers from `start`), resolving collisions.
pub fn plan_rename(&self, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
@@ -546,70 +632,7 @@ impl Session {
/// [`Session::plan_rename`] on the file system `fs`.
pub(crate) fn plan_rename_with(&self, fs: &dyn MoveFs, ids: &[PhotoId], template: &str, start: usize) -> Vec<RenamePlan> {
// photos sharing one file (virtual copies) rename once
let mut seen_paths: HashSet<String> = HashSet::new();
let mut taken: HashSet<String> = HashSet::new(); // lower-case target paths/names claimed in this batch
let mut out = Vec::new();
let mut seq = start;
for id in ids {
let Some(p) = self.catalog.photo(*id) else { continue };
if let Source::File { path } = &p.source
&& !seen_paths.insert(path.clone())
{
continue;
}
let want = expand(template, p, seq);
seq += 1;
let (stem, ext) = split_ext(&want);
let (stem, ext) = (stem.to_string(), ext.to_string());
let candidate = |k: usize| {
if k == 0 {
want.clone()
} else if ext.is_empty() {
format!("{stem}-{k}")
} else {
format!("{stem}-{k}.{ext}")
}
};
let (to, to_path) = match &p.source {
Source::File { path } => {
let dir = Path::new(path).parent().map(Path::to_path_buf).unwrap_or_default();
let mut k = 0;
loop {
let name = candidate(k);
let tp = dir.join(&name).to_string_lossy().to_string();
let key = tp.to_lowercase();
// this very file, maybe spelled differently (case-insensitive volume)? By
// identity: on a case-sensitive volume `img_1.jpg` may be another photo
let same = tp == *path || is_respelling(fs, Path::new(path), Path::new(&tp));
// free: not claimed in this batch and not on disk (unless it is this very file).
// A file this batch moves away still counts as taken: simple and safe.
let on_disk = fs.exists(Path::new(&tp)) && !same;
if !taken.contains(&key) && !on_disk {
taken.insert(key);
break (name, Some(tp));
}
k += 1;
}
}
Source::Demo { .. } => {
let mut k = 0;
loop {
let name = candidate(k);
if taken.insert(format!("demo:{}", name.to_lowercase())) {
break (name, None);
}
k += 1;
}
}
};
let from_path = match &p.source {
Source::File { path } => Some(path.clone()),
_ => None,
};
out.push(RenamePlan { id: id.0, from: p.file_name.clone(), to, from_path, to_path });
}
out
plan_rename_core(&rename_photos(&self.catalog, ids), template, start, &|p| fs.exists(p), &|a, b| fs.same_file(a, b))
}
/// Carry out `plans`: move the files (rolled back on failure), then commit one undoable op
+161 -8
View File
@@ -5,12 +5,25 @@
//! creator, keywords, capture time and GPS in the standard namespaces, plus our complete develop
//! settings (`lc:settings`, JSON), the pick/reject flag (`lc:flag`) and location (`lc:location`).
//!
//! An existing sidecar is never replaced wholesale (issue #92): LightCraft's properties
//! ([`OWNED`], [`OWNED_IF_STATED`]) are merged into it ([`lightcraft_meta::merge_xmp`]) and
//! everything else — another editor's `crs:` develop settings, `xmpMM` history, unknown
//! namespaces — is kept byte for byte. A sidecar that can't be read as XMP is first copied to
//! `<name>.xmp.bak-<time>` (never overwritten), then replaced. Writes are atomic (temp file,
//! fsync, rename).
//!
//! **Shared names** (Stem naming): `IMG_0001.CR3` and `IMG_0001.JPG` would share `IMG_0001.xmp`.
//! When two catalogued files map to one stem sidecar, it belongs to one of them (a raw first, then
//! the first by file name — [`Session::sidecar_naming`]); the others use `<file>.xmp`
//! (`IMG_0001.JPG.xmp`) for writing and reading, so neither overwrites the other.
//!
//! **Read** (on import, and `photo.readMetadataFromFile`): the sidecar wins for metadata, except
//! the capture time: a time embedded in the file wins, and the sidecar's (`exif:DateTimeOriginal`,
//! else `photoshop:DateCreated`, else `xmp:CreateDate`) fills in only when the file has none; develop
//! settings are restored from `lc:settings` when present, else mapped from interoperable `crs:`
//! fields ([`crate::crs`]). For raw/DNG files without a sidecar the file's embedded XMP is used.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -258,11 +271,101 @@ pub fn sidecar_packet(p: &Photo, cat: &lightcraft_catalog::Catalog) -> String {
lightcraft_meta::write_xmp_lc(&meta, &lc)
}
/// Write `data` to `path` atomically (temp file + sync + rename).
/// The properties LightCraft writes and owns in a sidecar: replaced on every save (removed when
/// LightCraft has no value, so clearing a field clears it in the file too). All of them are read
/// back on import and by Read Metadata from File.
pub const OWNED: &[&str] = &[
"xmp:Rating",
"xmp:Label",
"dc:title",
"dc:description",
"dc:rights",
"dc:creator",
"dc:subject",
"Iptc4xmpCore:Location",
"Iptc4xmpCore:AltTextAccessibility",
"Iptc4xmpCore:ExtDescrAccessibility",
"photoshop:City",
"photoshop:State",
"photoshop:Country",
"xmpRights:Marked",
"xmpRights:UsageTerms",
"xmpRights:WebStatement",
"lc:*",
];
/// Properties replaced only when LightCraft has a value (capture time; GPS, which LightCraft
/// doesn't read from sidecars): otherwise the sidecar's own value stays. Anything else LightCraft
/// writes (`xmp:CreatorTool`) goes only into new sidecars.
pub const OWNED_IF_STATED: &[&str] = &["exif:DateTimeOriginal", "photoshop:DateCreated", "exif:GPSLatitude", "exif:GPSLongitude"];
/// Write `data` to `path` atomically: a temp file next to it, synced, then renamed over it.
fn write_atomic(path: &Path, data: &[u8]) -> std::io::Result<()> {
lightcraft_catalog::safe_file::write_atomic(path, data)
}
/// Copy `path` to a new `<name>.bak-<stamp>` (`-1`, `-2`… if taken; never overwriting).
fn backup(path: &Path, stamp: &str) -> std::io::Result<PathBuf> {
use std::io::Write;
let bytes = std::fs::read(path)?;
let stamp: String = stamp.chars().filter(char::is_ascii_alphanumeric).collect();
let name = path.file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default();
for k in 0..1000 {
let b = path.with_file_name(if k == 0 { format!("{name}.bak-{stamp}") } else { format!("{name}.bak-{stamp}-{k}") });
match std::fs::OpenOptions::new().write(true).create_new(true).open(&b) {
Ok(mut f) => {
f.write_all(&bytes)?;
f.sync_all()?;
return Ok(b);
}
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => continue,
Err(e) => return Err(e),
}
}
Err(std::io::Error::other("no free backup name"))
}
/// What [`Session::save_sidecar`] did.
#[derive(Clone, Debug, PartialEq)]
pub struct SidecarSaved {
pub path: PathBuf,
/// An existing sidecar was merged into (its other contents kept).
pub merged: bool,
/// An existing sidecar that couldn't be read as XMP was copied here before being replaced.
pub backup: Option<PathBuf>,
}
/// The new contents for the sidecar at `path`: `fresh` merged into the existing file, if any.
/// Returns (contents, merged, backup).
fn sidecar_contents(path: &Path, fresh: String, stamp: &str) -> std::io::Result<(String, bool, Option<PathBuf>)> {
let bytes = match std::fs::read(path) {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((fresh, false, None)),
// can't see what is there: never overwrite it blindly
Err(e) => return Err(e),
};
if bytes.iter().all(u8::is_ascii_whitespace) {
return Ok((fresh, false, None));
}
let rules = lightcraft_meta::MergeRules { owned: OWNED, owned_if_present: OWNED_IF_STATED };
match std::str::from_utf8(&bytes)
.map_err(|e| e.to_string())
.and_then(|old| lightcraft_meta::merge_xmp(old, &fresh, rules).map_err(|e| e.to_string()))
{
Ok(merged) => Ok((merged, true, None)),
Err(why) => {
let b = backup(path, stamp)?;
log::warn!("{}: not readable as XMP ({why}); kept a copy as {} before replacing it", path.display(), b.display());
Ok((fresh, false, Some(b)))
}
}
}
/// Key of the stem sidecar a file maps to (case-insensitive).
fn stem_key(original: &str) -> String {
sidecar_path(original, SidecarNaming::Stem).to_string_lossy().to_lowercase()
}
fn file_path(p: &Photo) -> Option<&str> {
match &p.source {
Source::File { path } => Some(path),
@@ -298,24 +401,70 @@ pub(crate) fn op_photos(op: &Op, out: &mut Vec<PhotoId>) {
}
}
/// Which photo owns each stem sidecar that several catalogued files map to (see
/// [`Session::sidecar_naming`]).
pub(crate) struct StemOwners(HashMap<String, PhotoId>);
impl StemOwners {
pub(crate) fn of(cat: &lightcraft_catalog::Catalog) -> StemOwners {
// per stem: (raw first, then file name, then id) of the best candidate; and how many
let mut best: HashMap<String, ((bool, String, u64), PhotoId, usize)> = HashMap::new();
for p in cat.photos().filter(|p| p.copy_of.is_none()) {
let Some(path) = file_path(p) else { continue };
let rank = (p.kind != MediaKind::Raw, path.to_lowercase(), p.id.0);
let e = best.entry(stem_key(path)).or_insert_with(|| (rank.clone(), p.id, 0));
e.2 += 1;
if rank < e.0 {
(e.0, e.1) = (rank, p.id);
}
}
StemOwners(best.into_iter().filter(|(_, v)| v.2 > 1).map(|(k, v)| (k, v.1)).collect())
}
/// `naming` for photo `p`, except that a stem sidecar another file owns becomes Full.
fn naming(&self, p: &Photo, naming: SidecarNaming) -> SidecarNaming {
match (naming, file_path(p)) {
(SidecarNaming::Stem, Some(path)) if self.0.get(&stem_key(path)).is_some_and(|owner| *owner != p.id) => SidecarNaming::Full,
_ => naming,
}
}
}
impl Session {
/// Write the photo's XMP sidecar; returns its path.
pub fn save_sidecar(&self, id: PhotoId) -> Result<PathBuf> {
/// The sidecar naming for photo `id`: the library's, except under Stem naming when another
/// catalogued file shares the stem (`IMG_0001.CR3` + `IMG_0001.JPG`) and owns the stem
/// sidecar — a raw first, else the first by file name. The others use Full naming
/// (`IMG_0001.JPG.xmp`), so their metadata never overwrites each other.
pub fn sidecar_naming(&self, id: PhotoId) -> SidecarNaming {
match self.catalog.photo(id) {
Some(p) if self.xmp.naming == SidecarNaming::Stem => StemOwners::of(&self.catalog).naming(p, self.xmp.naming),
_ => self.xmp.naming,
}
}
/// Write the photo's XMP sidecar, merged into an existing one (see the module docs).
pub fn save_sidecar(&self, id: PhotoId) -> Result<SidecarSaved> {
self.save_sidecar_with(id, &StemOwners::of(&self.catalog))
}
pub(crate) fn save_sidecar_with(&self, id: PhotoId, owners: &StemOwners) -> Result<SidecarSaved> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
if p.copy_of.is_some() {
return Err(EngineError::Other(format!("{} is a virtual copy: its settings live only in the library", p.file_name)));
}
let orig = file_path(p).ok_or_else(|| EngineError::Other(format!("{} is not a file on disk", p.file_name)))?;
let path = sidecar_path(orig, self.xmp.naming);
write_atomic(&path, sidecar_packet(p, &self.catalog).as_bytes()).map_err(|e| EngineError::Other(format!("{}: {e}", path.display())))?;
Ok(path)
let path = sidecar_path(orig, owners.naming(p, self.xmp.naming));
let err = |e: std::io::Error| EngineError::Other(format!("{}: {e}", path.display()));
let (data, merged, backup) = sidecar_contents(&path, sidecar_packet(p, &self.catalog), &(self.clock)()).map_err(err)?;
write_atomic(&path, data.as_bytes()).map_err(err)?;
Ok(SidecarSaved { path, merged, backup })
}
/// The op that applies a photo's sidecar (or embedded XMP) to the catalog, if there is one.
pub fn read_sidecar_op(&self, id: PhotoId) -> Result<Option<(Op, PathBuf)>> {
let p = self.catalog.photo(id).ok_or(lightcraft_catalog::CatalogError::NoPhoto(id))?;
let Some(orig) = file_path(p) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.xmp.naming) else { return Ok(None) };
let Some((packet, from)) = read_packet(orig, p.kind, self.sidecar_naming(id)) else { return Ok(None) };
let sc = parse_sidecar(&packet, p.kind == MediaKind::Raw)
.map_err(|e| EngineError::Other(format!("{}: {e}", from.display())))?
.resolve_label(&self.catalog);
@@ -340,9 +489,13 @@ impl Session {
pub(crate) fn auto_write_sidecars(&self, ops: &[Op]) {
let mut ids = Vec::new();
ops.iter().for_each(|o| op_photos(o, &mut ids));
if ids.is_empty() {
return;
}
let owners = StemOwners::of(&self.catalog);
for id in ids {
if self.catalog.photo(id).is_some_and(|p| file_path(p).is_some() && p.copy_of.is_none())
&& let Err(e) = self.save_sidecar(id)
&& let Err(e) = self.save_sidecar_with(id, &owners)
{
log::warn!("auto-write XMP: {e}");
}
+140 -7
View File
@@ -144,11 +144,10 @@ fn browsing_a_folder_lists_its_photos_without_adding_them() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #105: Find Missing finds renamed files by content, prefers the content match over a
/// same-name same-size impostor, and skips (reports) photos it can't tell apart.
#[test]
fn find_missing_matches_renamed_files_by_content() {
let dir = temp_dir("missing-content");
/// Photos a.png, b.png, c.png imported from `old`, then: a renamed into `moved`; b renamed, with
/// an impostor (same name and size, other bytes) under its old name; c gone, with two impostors.
fn renamed_and_ambiguous_scene(tag: &str) -> (std::path::PathBuf, Session) {
let dir = temp_dir(tag);
std::fs::create_dir_all(dir.join("old")).unwrap();
std::fs::create_dir_all(dir.join("moved/real")).unwrap();
for (n, seed) in [("a.png", 1), ("b.png", 2), ("c.png", 3)] {
@@ -156,20 +155,26 @@ fn find_missing_matches_renamed_files_by_content() {
}
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [dir.join("old").to_string_lossy()]})).unwrap();
// an impostor: same name and size, other bytes
let impostor = |src: &Path, dst: &Path| {
let mut b = std::fs::read(src).unwrap();
let k = b.len() - 20;
b[k] ^= 0xff;
std::fs::write(dst, b).unwrap();
};
// a: renamed; b: renamed, with an impostor under its old name; c: gone, two impostors
std::fs::rename(dir.join("old/a.png"), dir.join("moved/Trip-001.png")).unwrap();
impostor(&dir.join("old/b.png"), &dir.join("moved/b.png"));
std::fs::rename(dir.join("old/b.png"), dir.join("moved/real/Trip-002.png")).unwrap();
impostor(&dir.join("old/c.png"), &dir.join("moved/c.png"));
impostor(&dir.join("old/c.png"), &dir.join("moved/real/c.png"));
std::fs::remove_file(dir.join("old/c.png")).unwrap();
(dir, s)
}
/// Issue #105: Find Missing finds renamed files by content, prefers the content match over a
/// same-name same-size impostor, and skips (reports) photos it can't tell apart.
#[test]
fn find_missing_matches_renamed_files_by_content() {
let (dir, mut s) = renamed_and_ambiguous_scene("missing-content");
let r = s.execute("library.findMissing", &json!({"folder": dir.join("moved").to_string_lossy()})).unwrap();
let found = r["found"].as_array().unwrap();
assert_eq!(found.len(), 2, "{r}");
@@ -183,6 +188,55 @@ fn find_missing_matches_renamed_files_by_content() {
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #104: the app plans Find Missing on a worker thread (no session) and relinks the plan
/// afterwards; that gives exactly what the synchronous command gives — renamed files by content,
/// impostors and look-alikes skipped (`ambiguous`).
#[test]
fn find_missing_planned_on_a_worker_matches_the_command() {
use crate::cmd::missing::{find_candidates, plan_find_missing};
let (dir, mut s) = renamed_and_ambiguous_scene("missing-worker");
let folder = dir.join("moved").to_string_lossy().to_string();
let candidates = find_candidates(&s.catalog);
let f = folder.clone();
let plan = std::thread::spawn(move || plan_find_missing(&candidates, &f)).join().unwrap().unwrap();
assert_eq!((plan.found.len(), plan.missing, plan.ambiguous.len()), (2, 1, 1), "{plan:?}");
let applied = s.execute("library.findMissing", &plan.to_json()).unwrap();
s.execute("edit.undo", &json!({})).unwrap();
assert_eq!(s.execute("library.missing", &json!({})).unwrap().as_array().map(Vec::len), Some(3), "undo restores all three");
let direct = s.execute("library.findMissing", &json!({"folder": folder})).unwrap();
assert_eq!(applied, direct);
assert_eq!(direct["found"].as_array().map(Vec::len), Some(2), "{direct}");
assert_eq!(direct["ambiguous"].as_array().map(Vec::len), Some(1), "{direct}");
let _ = std::fs::remove_dir_all(&dir);
}
/// Issue #104: a plan made on a worker is re-checked when it is applied: a photo relinked
/// meanwhile is left alone, and a file that is now another photo's original isn't used.
#[test]
fn find_missing_plan_is_rechecked_when_applied() {
use crate::cmd::missing::{find_candidates, plan_find_missing};
let (dir, mut s) = renamed_and_ambiguous_scene("missing-recheck");
let plan = plan_find_missing(&find_candidates(&s.catalog), &dir.join("moved").to_string_lossy()).unwrap();
let to = |name: &str| plan.found.iter().find(|f| f.from.ends_with(name)).unwrap().clone();
let (a, b) = (to("a.png"), to("b.png"));
// meanwhile: a is relinked by hand (elsewhere), and the ambiguous c is pointed at b's file
std::fs::copy(&a.to, dir.join("a-elsewhere.png")).unwrap();
s.execute("photo.relink", &json!({"id": a.id.0, "path": dir.join("a-elsewhere.png").to_string_lossy()})).unwrap();
let c = plan.ambiguous[0].id;
s.execute("photo.relink", &json!({"id": c.0, "path": b.to})).unwrap();
let r = s.execute("library.findMissing", &plan.to_json()).unwrap();
assert_eq!(r["found"].as_array().map(Vec::len), Some(0), "{r}");
assert_eq!(r["ambiguous"].as_array().map(Vec::len), Some(0), "c isn't missing any more: {r}");
assert_eq!(r["missing"], 1, "b stays missing: its file is c's now: {r}");
let path_of = |id: lightcraft_catalog::PhotoId| match &s.catalog.photo(id).unwrap().source {
lightcraft_catalog::Source::File { path } => path.clone(),
_ => String::new(),
};
assert_eq!(path_of(a.id), dir.join("a-elsewhere.png").to_string_lossy());
assert_eq!(path_of(b.id), b.from);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn missing_files_are_found_and_relinked() {
let dir = std::env::temp_dir().join(format!("lc-missing-{}", std::process::id()));
@@ -425,6 +479,20 @@ fn damaged_smart_previews_are_rebuilt_and_failed_writes_leave_none() {
let r = s.execute("library.smartPreviews", &json!({})).unwrap();
assert_eq!((r["built"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole");
// the app's background build (issue #104) checks and repairs the same way, on a worker thread
std::fs::write(&file, &full[..full.len() / 2]).unwrap();
let r = s.execute("library.smartPreviews", &json!({"background": true})).unwrap();
assert_eq!(r["what"], "smart previews", "{r}");
let t0 = std::time::Instant::now();
let r = loop {
let r = s.execute("library.previewProgress", &json!({})).unwrap();
if r["running"] == false || t0.elapsed() > std::time::Duration::from_secs(60) {
break r;
}
std::thread::sleep(std::time::Duration::from_millis(5));
};
assert_eq!((r["done"].as_u64(), r["failed"].as_u64(), r["repaired"].as_u64()), (Some(1), Some(0), Some(1)), "{r}");
assert_eq!(std::fs::read(&file).unwrap(), full, "rebuilt whole in the background");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
@@ -518,6 +586,70 @@ fn folders_rename_and_move_with_their_photos() {
let _ = std::fs::remove_dir_all(&root);
}
/// Rename / Move Folder are undo steps (issue #97): undo renames the folder back on disk (files and
/// sidecars along) and restores the photos' paths, redo repeats it; a destination taken in the
/// meantime is never overwritten — the step is refused, reported and stays on the stack.
#[test]
fn folder_rename_and_move_undo_and_redo_on_disk() {
let root = temp_dir("folders-undo");
let a = root.join("Trip");
write_png(&a.join("one.png"), 3);
std::fs::write(a.join("one.xmp"), "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>").unwrap();
let mut s = Session::new().with_fs();
s.execute("library.import", &json!({"paths": [a.to_string_lossy()]})).unwrap();
let id = s.active().unwrap();
let path = |s: &Session| match &s.catalog.photo(id).unwrap().source {
lightcraft_catalog::Source::File { path } => path.clone(),
_ => panic!(),
};
let undo0 = s.undo.len();
let b = root.join("Italy");
s.execute("folder.rename", &json!({"path": a.to_string_lossy(), "name": "Italy"})).unwrap();
assert_eq!(s.undo.len(), undo0 + 1, "one undo step");
assert!(s.undo.last().unwrap().label.contains("Rename Folder"));
assert!(b.join("one.png").exists() && !a.exists());
s.execute("edit.undo", &json!({})).unwrap();
assert!(a.join("one.png").exists() && a.join("one.xmp").exists() && !b.exists(), "renamed back on disk");
assert_eq!(path(&s), a.join("one.png").to_string_lossy());
s.media.forget(id);
assert!(s.render_now(id, 16, 16).is_ok());
s.execute("edit.redo", &json!({})).unwrap();
assert!(b.join("one.png").exists() && b.join("one.xmp").exists() && !a.exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
// the old name is taken now: undo is refused, nothing is overwritten, the step stays
std::fs::create_dir_all(&a).unwrap();
std::fs::write(a.join("keep.txt"), "mine").unwrap();
let undo_n = s.undo.len();
let e = s.execute("edit.undo", &json!({})).unwrap_err().to_string();
assert!(e.contains("already exists"), "{e}");
assert_eq!(s.undo.len(), undo_n, "the step stays on the stack");
assert_eq!(std::fs::read_to_string(a.join("keep.txt")).unwrap(), "mine");
assert!(b.join("one.png").exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
std::fs::remove_dir_all(&a).unwrap();
// move into another folder, undo, redo
let into = root.join("Archive");
s.execute("folder.move", &json!({"path": b.to_string_lossy(), "into": into.to_string_lossy()})).unwrap();
let moved = into.join("Italy");
assert_eq!(path(&s), moved.join("one.png").to_string_lossy());
s.execute("edit.undo", &json!({})).unwrap();
assert!(b.join("one.png").exists() && !moved.exists());
assert_eq!(path(&s), b.join("one.png").to_string_lossy());
// the redo target is taken (even by an empty folder): refused, nothing replaced
std::fs::create_dir_all(&moved).unwrap();
assert!(s.execute("edit.redo", &json!({})).is_err());
assert!(b.join("one.png").exists());
std::fs::remove_dir(&moved).unwrap();
s.execute("edit.redo", &json!({})).unwrap();
assert!(moved.join("one.png").exists() && moved.join("one.xmp").exists() && !b.exists());
assert_eq!(path(&s), moved.join("one.png").to_string_lossy());
let _ = std::fs::remove_dir_all(&root);
}
/// Assisted culling: a burst (same scene, seconds apart) of a sharp and two soft shots, plus an
/// unrelated photo: the burst is grouped, the sharp one is its best, blurry shots can be rejected.
#[test]
@@ -667,6 +799,7 @@ fn smart_previews_folder_is_chosen_per_library() {
assert_eq!(s.execute("photo.smartPreview", &json!({})).unwrap()["smartPreview"], true);
// the setting is saved with the library
drop(s); // one session per library (issue #99)
let mut again = Session::new().with_fs();
again.open_library(&lib, false).unwrap();
assert_eq!(again.execute("library.smartPreviewsLocation", &json!({})).unwrap()["path"], to.to_string_lossy().as_ref());
+31
View File
@@ -113,6 +113,7 @@ fn close_writes_snapshot_and_view_state() {
let expect = s.catalog.to_snapshot();
s.close_library().unwrap();
assert_eq!(std::fs::metadata(dir.join("catalog.log")).unwrap().len(), 0);
drop(s); // one session per library (issue #99)
let s2 = open(&dir, true);
let r = &s2.library.as_ref().unwrap().report;
@@ -178,6 +179,7 @@ fn new_library_without_seed_is_empty_and_compacts() {
assert!(p["snapshots"].as_u64() >= Some(1), "{p}");
assert!(p["lastSnapshot"]["bytes"].as_u64() > Some(0), "{p}");
assert!(p["lastSnapshot"]["totalMs"].as_f64() >= p["lastSnapshot"]["serializeMs"].as_f64(), "{p}");
drop(s); // one session per library (issue #99)
let s2 = open(&dir, true);
assert_eq!(s2.catalog.len(), 0, "an existing library is never seeded");
assert_eq!(s2.catalog.albums().count(), 1);
@@ -376,3 +378,32 @@ fn scale_100k_library_queries() {
worst.sort_by(|a, b| b.1.total_cmp(&a.1));
eprintln!("slowest: {:?}", &worst[..3]);
}
/// Issue #99: a library is open in one session at a time. A second opener is refused (nothing
/// read or written), the owning session can reopen it, and it is free again once closed.
#[test]
fn a_library_open_elsewhere_is_refused() {
let dir = temp_dir("locked");
let mut s = open(&dir, true);
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
let log = std::fs::read(dir.join("catalog.log")).unwrap();
let mut other = Session::new();
let e = other.open_library(&dir, true).unwrap_err();
assert!(matches!(e, crate::EngineError::LibraryInUse(_)), "{e:?}");
assert!(e.to_string().contains("already open in"), "{e}");
assert!(other.library.is_none());
assert_eq!(std::fs::read(dir.join("catalog.log")).unwrap(), log, "the refused opener wrote nothing");
// the session that has it open may reopen it (Settings → Open Library on the same folder)
s.close_library().unwrap();
s.open_library(&dir, true).unwrap();
assert!(other.open_library(&dir, true).is_err(), "still locked after reopening");
s.execute("photo.rate", &json!({"rating": 4})).unwrap();
let expect = s.catalog.to_snapshot();
drop(s);
other.open_library(&dir, true).unwrap();
assert_eq!(other.catalog.to_snapshot(), expect);
let _ = std::fs::remove_dir_all(&dir);
}
+47 -1
View File
@@ -17,6 +17,8 @@ struct Flaky {
fail: Arc<AtomicBool>,
/// Only snapshots fail (appends work).
snapshots_only: bool,
/// Only appends fail (a dead log handle; fresh files can still be written).
appends_only: bool,
}
impl Flaky {
@@ -39,7 +41,7 @@ impl Store for Flaky {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
if self.failing() {
if self.failing() && !self.appends_only {
return Err(std::io::Error::other("disk full"));
}
self.files.write_atomic(name, data)
@@ -145,3 +147,47 @@ fn failed_snapshot_does_not_fail_the_command() {
store.set_failing(false);
assert_eq!(open(&store, false).catalog.albums().count(), 2);
}
/// Issue #101: quitting while the log can't be appended to still saves the queued changes in the
/// closing snapshot (a fresh file), and reports success; they are not appended again later.
#[test]
fn close_saves_queued_changes_in_the_snapshot_when_appends_fail() {
let store = Flaky { appends_only: true, ..Default::default() };
let mut s = open(&store, true);
let id = s.selection.active.unwrap();
store.set_failing(true);
assert!(matches!(s.execute("photo.rate", &json!({"rating": 4})), Err(EngineError::NotSaved(_))));
assert!(matches!(s.execute("photo.flag", &json!({"flag": "pick"})), Err(EngineError::NotSaved(_))));
assert_eq!(s.unsaved().map(|u| u.0), Some(2));
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
assert!(s.pending_log.is_empty());
let expect = s.catalog.to_snapshot();
drop(s);
store.set_failing(false);
let mut s2 = open(&store, true);
assert_eq!(s2.catalog.to_snapshot(), expect);
let p = s2.catalog.photo(id).unwrap();
assert_eq!((p.rating, p.flag), (4, Flag::Pick));
// later edits continue after the absorbed ops
s2.execute("photo.rate", &json!({"rating": 2})).unwrap();
let expect = s2.catalog.to_snapshot();
drop(s2);
assert_eq!(open(&store, true).catalog.to_snapshot(), expect);
}
/// When neither the log nor the snapshot can be written, closing fails with `NotSaved` and the
/// changes stay queued (a later retry can still save them).
#[test]
fn close_fails_when_nothing_can_be_saved() {
let store = Flaky::default();
let mut s = open(&store, true);
store.set_failing(true);
assert!(s.execute("photo.rate", &json!({"rating": 4})).is_err());
let e = s.close_library().unwrap_err();
assert!(matches!(e, EngineError::NotSaved(_)), "{e:?}");
assert_eq!(s.unsaved().map(|u| u.0), Some(1));
store.set_failing(false);
s.close_library().unwrap();
assert_eq!(s.unsaved(), None);
}
+2
View File
@@ -126,11 +126,13 @@ fn preferences_persist_with_the_library() {
assert_eq!(r["persistent"], true);
// export presets are library preferences too
s.execute("export.savePreset", &json!({"name": "Proof", "params": {"format": "jpeg", "percent": 25}})).unwrap();
drop(s); // one session per library (issue #99)
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
assert_eq!(s.export_params(&json!({"preset": "Proof"})).unwrap(), json!({"format": "jpeg", "percent": 25}));
// older prefs.json files (snake_case `last_export`, no import section) still load
std::fs::write(lib.join("prefs.json"), r#"{"xmp": {"autoWrite": false}, "last_export": {"format": "png"}}"#).unwrap();
drop(s);
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
assert_eq!(s.last_export, Some(json!({"format": "png"})));
+97
View File
@@ -0,0 +1,97 @@
//! Issue #103: a damaged or unreadable settings file (prefs.json, presets.json, view.json) is
//! never silently replaced by the defaults: a damaged one is kept aside, an unreadable one is not
//! written this session, and the user is told.
use std::sync::{Arc, Mutex};
use lightcraft_catalog::{MemStore, Store};
use serde_json::json;
use crate::Session;
use crate::library::LibraryStores;
fn temp_dir(tag: &str) -> std::path::PathBuf {
let d = std::env::temp_dir().join(format!("lc-settings-{tag}-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn damaged_prefs_are_kept_aside_and_reported() {
let lib = temp_dir("damaged");
let bad = br#"{"exportPresets": [{"name": "Proof", "par"#;
std::fs::write(lib.join("prefs.json"), bad).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
let kept: Vec<_> = std::fs::read_dir(&lib)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().to_string())
.filter(|n| n.starts_with("prefs.json.corrupt-"))
.collect();
assert_eq!(kept.len(), 1, "{kept:?}");
assert_eq!(std::fs::read(lib.join(&kept[0])).unwrap(), bad);
let warnings = s.take_library_warnings();
assert_eq!(warnings.len(), 1);
assert!(warnings[0].contains("prefs.json is damaged") && warnings[0].contains(&kept[0]), "{warnings:?}");
assert!(s.take_library_warnings().is_empty(), "each warning is handed out once");
let info = s.execute("library.info", &json!({})).unwrap();
assert_eq!(info["settingsWarnings"].as_array().map(Vec::len), Some(1), "{info}");
// saving goes on (the damaged original is safe in its copy)
s.execute("library.preferences", &json!({"cacheMb": 300})).unwrap();
assert!(String::from_utf8(std::fs::read(lib.join("prefs.json")).unwrap()).unwrap().contains("300"));
assert_eq!(std::fs::read(lib.join(&kept[0])).unwrap(), bad);
drop(s);
let _ = std::fs::remove_dir_all(&lib);
}
/// `presets.json` and `prefs.json` can't be read (locked by another program, an I/O error).
#[derive(Clone, Default)]
struct Unreadable {
files: MemStore,
unreadable: Arc<Mutex<Vec<String>>>,
}
impl Store for Unreadable {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
if self.unreadable.lock().unwrap().iter().any(|n| n == name) {
return Err(std::io::Error::other("locked by another process"));
}
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"unreadable".into()
}
}
#[test]
fn unreadable_settings_are_not_overwritten() {
let files = Unreadable::default();
let presets = br#"{"user": [{"id": "user.mine", "name": "Mine", "group": "User Presets", "settings": {}}]}"#.to_vec();
let prefs = br#"{"cacheMb": 777}"#.to_vec();
files.files.set("presets.json", presets.clone());
files.files.set("prefs.json", prefs.clone());
*files.unreadable.lock().unwrap() = vec!["presets.json".into(), "prefs.json".into()];
let mut s = Session::new();
let stores = LibraryStores { dir: "unreadable".into(), catalog: Box::new(MemStore::new()), files: Box::new(files.clone()), on_disk: false };
s.open_library_in(stores, true).unwrap();
let warnings = s.take_library_warnings();
assert_eq!(warnings.len(), 2, "{warnings:?}");
assert!(warnings.iter().all(|w| w.contains("couldn't be read") && w.contains("won't overwrite")), "{warnings:?}");
// a preset change and a preference change: neither file is replaced by defaults
s.execute("preset.create", &json!({"name": "New"})).unwrap();
let e = s.execute("library.preferences", &json!({"cacheMb": 300}));
assert!(e.is_err(), "saving preferences is refused, not done over the unread file");
assert_eq!(files.files.get("presets.json").unwrap(), presets);
assert_eq!(files.files.get("prefs.json").unwrap(), prefs);
}
+129
View File
@@ -625,3 +625,132 @@ fn copyright_status_usage_terms_and_url_round_trip() {
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// A sidecar another raw developer wrote: develop settings in attributes and elements, a
/// structured edit history, an unknown namespace.
const OTHER_APP_SIDECAR: &str = r#"<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Other Toolkit">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""
xmlns:xmp="http://ns.adobe.com/xap/1.0/"
xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#"
xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
xmlns:other="http://example.com/other/1.0/"
xmp:Rating="2"
crs:Version="99.0"
crs:Exposure2012="+0.65"
crs:HasSettings="True"
other:Secret="keep me">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action="derived" stEvt:parameters="converted from image/x-raw to image/png"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>128, 140</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>"#;
/// Issue #92: saving merges into another app's sidecar instead of replacing it.
#[test]
fn saving_merges_into_another_apps_sidecar() {
let src = temp_dir("merge");
let lib = temp_dir("merge-lib");
write_png(&src.join("shot.png"), 1);
std::fs::write(src.join("shot.xmp"), OTHER_APP_SIDECAR).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.join("shot.png").to_string_lossy()]})).unwrap();
let id = only(&s);
s.execute("library.select", &json!({"ids": [id.0]})).unwrap();
s.execute("photo.rate", &json!({"rating": 5})).unwrap();
s.execute("photo.setMeta", &json!({"title": "Harbour", "keywords": ["boats"]})).unwrap();
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 30})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
assert_eq!(r["merged"][0], src.join("shot.xmp").to_string_lossy().as_ref(), "{r}");
let out = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
// the other app's data, byte for byte
for part in [
"crs:Exposure2012=\"+0.65\"",
"other:Secret=\"keep me\"",
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"derived\" stEvt:parameters=\"converted from image/x-raw to image/png\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>\n <rdf:li>128, 140</rdf:li>",
] {
assert!(out.contains(part), "lost {part:?}:\n{out}");
}
let d = lightcraft_meta::parse_xmp(&out).unwrap();
assert_eq!(d.metadata.rating, Some(5), "ours replaces theirs");
assert_eq!(d.metadata.title.as_deref(), Some("Harbour"));
assert_eq!(d.properties.get("crs:Version"), Some(&vec!["99.0".to_string()]));
let saved = s.catalog.photo(id).unwrap().develop.clone();
// reading it back restores our settings (lc:settings wins over crs:)
s.execute("develop.set", &json!({"control": "effects.clarity", "value": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({})).unwrap();
assert_eq!(s.catalog.photo(id).unwrap().develop, saved);
// saving again keeps one LightCraft description and the other app's data
s.execute("photo.rate", &json!({"rating": 3})).unwrap();
s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let again = std::fs::read_to_string(src.join("shot.xmp")).unwrap();
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert!(again.contains("other:Secret=\"keep me\"") && again.contains("crs:Exposure2012=\"+0.65\""));
assert_eq!(lightcraft_meta::parse_xmp(&again).unwrap().metadata.rating, Some(3));
// a sidecar that isn't XMP is kept as a backup before it is replaced
std::fs::write(src.join("shot.xmp"), b"<not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let backup = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert!(backup.contains("shot.xmp.bak-"), "{r}");
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join("shot.xmp")).unwrap()).unwrap().metadata.rating, Some(3));
// … and another one never overwrites the first backup
std::fs::write(src.join("shot.xmp"), b"<still not xmp").unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({})).unwrap();
let second = r["backups"][0]["backup"].as_str().unwrap().to_string();
assert_ne!(second, backup);
assert_eq!(std::fs::read(&backup).unwrap(), b"<not xmp");
assert_eq!(std::fs::read(&second).unwrap(), b"<still not xmp");
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
/// Issue #92: two catalogued files with one stem (`IMG_1.jpg` + `IMG_1.png`) don't share a
/// Stem-named sidecar: the first by name keeps `IMG_1.xmp`, the other writes and reads
/// `IMG_1.png.xmp`.
#[test]
fn files_sharing_a_stem_get_their_own_sidecars() {
let src = temp_dir("stem");
let lib = temp_dir("stem-lib");
write_png(&src.join("IMG_1.png"), 1);
let img = lightcraft_raster::Rgba8::from_fn(40, 24, |x, y| [(x * 6) as u8, (y * 9) as u8, 7, 255]);
let jpg =
crate::export::encode_image(&img, &crate::export::ExportOptions { format: crate::export::ExportFormat::Jpeg, ..Default::default() }).unwrap();
std::fs::write(src.join("IMG_1.jpg"), jpg).unwrap();
let mut s = Session::new().with_fs();
s.open_library(&lib, false).unwrap();
s.execute("library.import", &json!({"paths": [src.to_string_lossy()]})).unwrap();
let id = |s: &Session, name: &str| s.catalog.photos().find(|p| p.file_name == name).unwrap().id;
let (j, p) = (id(&s, "IMG_1.jpg"), id(&s, "IMG_1.png"));
assert_eq!(s.sidecar_naming(j), crate::sidecar::SidecarNaming::Stem);
assert_eq!(s.sidecar_naming(p), crate::sidecar::SidecarNaming::Full);
s.execute("photo.rate", &json!({"ids": [j.0], "rating": 2})).unwrap();
s.execute("photo.rate", &json!({"ids": [p.0], "rating": 5})).unwrap();
let r = s.execute("photo.saveMetadataToFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!(r["written"].as_array().unwrap().len(), 2, "{r}");
let rating = |f: &str| lightcraft_meta::parse_xmp(&std::fs::read_to_string(src.join(f)).unwrap()).unwrap().metadata.rating;
assert_eq!(rating("IMG_1.xmp"), Some(2));
assert_eq!(rating("IMG_1.png.xmp"), Some(5));
// each reads its own back
s.execute("photo.rate", &json!({"ids": [j.0, p.0], "rating": 0})).unwrap();
s.execute("photo.readMetadataFromFile", &json!({"ids": [j.0, p.0]})).unwrap();
assert_eq!((s.catalog.photo(j).unwrap().rating, s.catalog.photo(p).unwrap().rating), (2, 5));
let _ = std::fs::remove_dir_all(&src);
let _ = std::fs::remove_dir_all(&lib);
}
+2
View File
@@ -17,6 +17,7 @@ mod gpx;
mod iptc;
pub mod tags;
mod xmp;
mod xmp_merge;
pub use container::{Embedded, embedded, jpeg_segments, png_chunks, webp_chunks};
pub use datetime::DateTime;
@@ -26,6 +27,7 @@ pub use iptc::parse_iptc;
pub use lightcraft_geom::Orientation;
pub use tags::{TagRow, file_tag_rows, tag_rows};
pub use xmp::{CRS_NS, LC_NS, XmpData, XmpError, XmpValue, parse_xmp, write_xmp, write_xmp_lc};
pub use xmp_merge::{MergeRules, merge_xmp};
use serde::{Deserialize, Serialize};
+5
View File
@@ -145,6 +145,11 @@ impl Node {
const MAX_DEPTH: usize = 64;
const MAX_NODES: usize = 200_000;
/// `prefix:local` with the prefix canonicalised by namespace URI (see [`NAMESPACES`]).
pub(crate) fn canonical_name(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
canonical(qname, scopes)
}
fn canonical(qname: &str, scopes: &[Vec<(String, String)>]) -> String {
let (prefix, local) = qname.split_once(':').unwrap_or(("", qname));
if prefix == "xml" || prefix == "xmlns" {
+403
View File
@@ -0,0 +1,403 @@
//! Merging a freshly written XMP packet into an existing one (a sidecar another application
//! wrote, e.g. with its develop settings and edit history), keeping everything the writer does not
//! own byte for byte.
//!
//! The existing packet is scanned once (namespace-resolved, with byte positions). Only top-level
//! properties of the `rdf:Description` elements directly under `rdf:RDF` are considered, in
//! attribute form (`xmp:Rating="3"`) and element form (`<dc:subject>…</dc:subject>`):
//! - properties the writer owns ([`MergeRules::owned`]) are removed, and written from the fresh
//! packet when it has them (so a cleared field is cleared in the file too);
//! - properties owned only when stated ([`MergeRules::owned_if_present`]) are replaced when the
//! fresh packet has them and kept otherwise;
//! - any other property of the fresh packet (e.g. `xmp:CreatorTool`) is not written: the file
//! keeps describing itself as its creator left it;
//! - everything else — other namespaces (e.g. `crs:`), `xmpMM:History`, unknown properties,
//! comments, the packet wrapper and padding — is copied unchanged.
//!
//! The written properties go into one new `rdf:Description` (declaring its own namespaces, so
//! the existing prefix bindings never matter) just before `</rdf:RDF>`. A description left empty
//! by the removals is dropped, so repeated saves don't accumulate. The result is checked by
//! parsing it again: every written property must read back as in the fresh packet.
use std::collections::BTreeSet;
use quick_xml::events::Event;
use crate::xmp::{XmpError, canonical_name, parse_xmp};
/// Which properties [`merge_xmp`] takes from the fresh packet. Names are canonical
/// `prefix:name` (prefixes as [`crate::parse_xmp`] reports them); `prefix:*` is a whole namespace.
#[derive(Clone, Copy, Debug, Default)]
pub struct MergeRules<'a> {
/// Always replaced: removed from the existing packet, written from the fresh one if stated.
pub owned: &'a [&'a str],
/// Replaced only when the fresh packet states them; otherwise the existing value stays.
pub owned_if_present: &'a [&'a str],
}
fn listed(list: &[&str], name: &str) -> bool {
list.iter().any(|p| match p.strip_suffix('*') {
Some(ns) => name.starts_with(ns),
None => *p == name,
})
}
#[derive(Debug)]
struct Attr {
raw: String,
name: String,
/// The value as written (still escaped).
value: String,
}
#[derive(Debug)]
struct Prop {
name: String,
start: usize,
end: usize,
}
#[derive(Debug, Default)]
struct Desc {
tag_start: usize,
tag_end: usize,
/// The element name as written (`rdf:Description`, or another prefix for the RDF namespace).
raw_name: String,
empty: bool,
attrs: Vec<Attr>,
props: Vec<Prop>,
/// The end of `</rdf:Description>` (`tag_end` when self-closing).
end: usize,
}
#[derive(Debug, Default)]
struct Scan {
descs: Vec<Desc>,
/// Where the first `</rdf:RDF>` starts.
rdf_close: Option<usize>,
}
enum Kind {
Other,
Rdf,
Desc(usize),
Prop { desc: usize, start: usize, name: String },
}
fn is_meta_attr(name: &str) -> bool {
name.starts_with("xmlns")
|| matches!(name, "rdf:about" | "rdf:ID" | "rdf:nodeID" | "xml:lang" | "rdf:parseType" | "rdf:resource" | "rdf:datatype")
}
fn pos(r: &quick_xml::Reader<&[u8]>) -> usize {
usize::try_from(r.buffer_position()).unwrap_or(usize::MAX)
}
/// Locate the descriptions, their top-level properties and `</rdf:RDF>` in a packet.
fn scan(s: &str) -> Result<Scan, XmpError> {
let mut reader = quick_xml::Reader::from_reader(s.as_bytes());
reader.config_mut().check_end_names = true;
let mut out = Scan::default();
let mut stack: Vec<Kind> = Vec::new();
let mut scopes: Vec<Vec<(String, String)>> = vec![vec![("xml".into(), "http://www.w3.org/XML/1998/namespace".into())]];
let mut count = 0usize;
loop {
let before = pos(&reader);
let ev = reader.read_event().map_err(|e| XmpError::Xml(e.to_string()))?;
let after = pos(&reader);
match ev {
Event::Start(ref e) | Event::Empty(ref e) => {
count += 1;
if count > 200_000 || stack.len() > 64 {
return Err(XmpError::TooComplex);
}
let empty = matches!(ev, Event::Empty(_));
let mut scope = Vec::new();
let mut raw_attrs = Vec::new();
for a in e.attributes() {
let a = a.map_err(|e| XmpError::Xml(e.to_string()))?;
let k = String::from_utf8_lossy(a.key.as_ref()).into_owned();
let v = String::from_utf8_lossy(&a.value).into_owned();
if k == "xmlns" {
scope.push((String::new(), v.clone()));
} else if let Some(p) = k.strip_prefix("xmlns:") {
scope.push((p.to_string(), v.clone()));
}
raw_attrs.push((k, v));
}
scopes.push(scope);
let raw_name = String::from_utf8_lossy(e.name().as_ref()).into_owned();
let name = canonical_name(&raw_name, &scopes);
let kind = match stack.last() {
Some(Kind::Rdf) if name == "rdf:Description" => {
let attrs = raw_attrs.into_iter().map(|(raw, value)| Attr { name: canonical_name(&raw, &scopes), raw, value }).collect();
out.descs.push(Desc { tag_start: before, tag_end: after, raw_name, empty, attrs, props: Vec::new(), end: after });
Kind::Desc(out.descs.len() - 1)
}
Some(Kind::Desc(i)) => Kind::Prop { desc: *i, start: before, name },
_ if name == "rdf:RDF" && out.rdf_close.is_none() => Kind::Rdf,
_ => Kind::Other,
};
if empty {
scopes.pop();
if let Kind::Prop { desc, start, name } = kind
&& let Some(d) = out.descs.get_mut(desc)
{
d.props.push(Prop { name, start, end: after });
}
} else {
stack.push(kind);
}
}
Event::End(_) => {
scopes.pop();
match stack.pop() {
Some(Kind::Prop { desc, start, name }) => {
if let Some(d) = out.descs.get_mut(desc) {
d.props.push(Prop { name, start, end: after });
}
}
Some(Kind::Desc(i)) => {
if let Some(d) = out.descs.get_mut(i) {
d.end = after;
}
}
Some(Kind::Rdf) => out.rdf_close = out.rdf_close.or(Some(before)),
Some(Kind::Other) => {}
None => return Err(XmpError::Xml("unbalanced end tag".into())),
}
}
Event::Eof => break,
_ => {}
}
}
if !stack.is_empty() {
return Err(XmpError::Xml("unclosed element".into()));
}
Ok(out)
}
/// `start` moved back over the indentation and line break before it (so a removed property
/// takes its line with it).
fn line_start(s: &str, start: usize) -> usize {
let b = s.as_bytes();
let mut i = start;
while i > 0 && matches!(b.get(i - 1), Some(b' ' | b'\t')) {
i -= 1;
}
if i > 0 && b.get(i - 1) == Some(&b'\n') {
i -= 1;
if i > 0 && b.get(i - 1) == Some(&b'\r') {
i -= 1;
}
}
i
}
/// Merge `fresh` (a packet as [`crate::write_xmp_lc`] writes it) into `existing` following
/// `rules` (see the module docs). Fails — and the caller must not overwrite `existing` blindly —
/// when `existing` isn't a well-formed XMP packet.
pub fn merge_xmp(existing: &str, fresh: &str, rules: MergeRules) -> Result<String, XmpError> {
let (bom, old) = match existing.strip_prefix('\u{feff}') {
Some(rest) => ("\u{feff}", rest),
None => ("", existing),
};
parse_xmp(old)?;
let ex = scan(old)?;
let rdf_close = ex.rdf_close.ok_or_else(|| XmpError::Xml("no rdf:RDF element".into()))?;
let new = scan(fresh)?;
let fresh_desc = new.descs.first().ok_or_else(|| XmpError::Xml("the new packet has no rdf:Description".into()))?;
let fresh_names: BTreeSet<&str> = fresh_desc.props.iter().map(|p| p.name.as_str()).collect();
let remove = |name: &str| listed(rules.owned, name) || (listed(rules.owned_if_present, name) && fresh_names.contains(name));
let write = |name: &str| listed(rules.owned, name) || listed(rules.owned_if_present, name);
// edits on `old`: (start, end, replacement), applied back to front
let mut edits: Vec<(usize, usize, String)> = Vec::new();
for d in &ex.descs {
let gone_attrs: Vec<&Attr> = d.attrs.iter().filter(|a| !is_meta_attr(&a.name) && remove(&a.name)).collect();
let gone_props: Vec<&Prop> = d.props.iter().filter(|p| remove(&p.name)).collect();
if gone_attrs.is_empty() && gone_props.is_empty() {
continue;
}
let attrs_left = d.attrs.iter().any(|a| !is_meta_attr(&a.name) && !remove(&a.name));
if !attrs_left && gone_props.len() == d.props.len() {
// nothing of its own left: drop the whole description
edits.push((line_start(old, d.tag_start), d.end, String::new()));
continue;
}
if !gone_attrs.is_empty() {
let mut tag = format!("<{}", d.raw_name);
for a in d.attrs.iter().filter(|a| is_meta_attr(&a.name) || !remove(&a.name)) {
tag.push_str(&format!("\n {}=\"{}\"", a.raw, a.value.replace('"', "&quot;")));
}
tag.push_str(if d.empty { "/>" } else { ">" });
edits.push((d.tag_start, d.tag_end, tag));
}
for p in gone_props {
edits.push((line_start(old, p.start), p.end, String::new()));
}
}
// the new description: its own namespace declarations, the properties to write
let mut block = String::from("\n <rdf:Description rdf:about=\"\"\n xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"");
for a in fresh_desc.attrs.iter().filter(|a| a.raw.starts_with("xmlns:") && a.raw != "xmlns:rdf") {
block.push_str(&format!("\n {}=\"{}\"", a.raw, a.value));
}
block.push('>');
let mut written = 0;
for p in fresh_desc.props.iter().filter(|p| write(&p.name)) {
let text = fresh.get(p.start..p.end).ok_or_else(|| XmpError::Xml("bad property span".into()))?;
block.push_str("\n ");
block.push_str(text);
written += 1;
}
block.push_str("\n </rdf:Description>");
if written > 0 {
let at = old.get(..rdf_close).map_or(rdf_close, |h| h.trim_end().len());
edits.push((at, at, block));
}
edits.sort_by_key(|e| (e.0, e.1));
// overlapping edits can't happen (properties sit inside their description, which is either
// dropped whole or edited piecewise), but never splice blindly
let mut out = String::with_capacity(old.len() + 4096);
out.push_str(bom);
let mut at = 0;
for (s, e, r) in &edits {
if *s < at {
return Err(XmpError::Xml("overlapping edits".into()));
}
out.push_str(old.get(at..*s).ok_or_else(|| XmpError::Xml("bad span".into()))?);
out.push_str(r);
at = *e;
}
out.push_str(old.get(at..).ok_or_else(|| XmpError::Xml("bad span".into()))?);
// read back: everything written must come out as written
let merged = parse_xmp(&out)?;
let want = parse_xmp(fresh)?;
for (k, v) in &want.properties {
let top = k.split('/').next().unwrap_or(k);
if write(top) && merged.properties.get(k) != Some(v) {
return Err(XmpError::Xml(format!("{k} did not survive the merge")));
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
const RULES: MergeRules =
MergeRules { owned: &["xmp:Rating", "xmp:Label", "dc:subject", "dc:title", "lc:*"], owned_if_present: &["exif:DateTimeOriginal"] };
/// A sidecar as another raw developer writes it: attribute-form simple properties, develop
/// settings, a structured history, an unknown namespace.
const FOREIGN: &str = "<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>
<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">
<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">
<rdf:Description rdf:about=\"\"
xmlns:xmp=\"http://ns.adobe.com/xap/1.0/\"
xmlns:xmpMM=\"http://ns.adobe.com/xap/1.0/mm/\"
xmlns:stEvt=\"http://ns.adobe.com/xap/1.0/sType/ResourceEvent#\"
xmlns:crs=\"http://ns.adobe.com/camera-raw-settings/1.0/\"
xmlns:dc=\"http://purl.org/dc/elements/1.1/\"
xmlns:foo=\"http://example.com/foo/\"
xmp:Rating=\"2\"
xmp:CreatorTool=\"Other App\"
crs:Exposure2012=\"+0.50\"
crs:Contrast2012=\"-12\"
foo:Thing=\"a &amp; b\">
<xmpMM:History>
<rdf:Seq>
<rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>
</rdf:Seq>
</xmpMM:History>
<crs:ToneCurvePV2012>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurvePV2012>
<dc:subject>
<rdf:Bag>
<rdf:li>old</rdf:li>
</rdf:Bag>
</dc:subject>
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
";
/// With the packet's whitespace padding and trailer.
fn foreign() -> String {
format!("{FOREIGN}{}\n<?xpacket end=\"w\"?>", " ".repeat(100))
}
fn fresh(rating: i8, keywords: &[&str]) -> String {
let m = crate::Metadata {
software: Some("LightCraft".into()),
rating: Some(rating),
keywords: keywords.iter().map(|k| k.to_string()).collect(),
..Default::default()
};
crate::write_xmp_lc(&m, &[("settings", "{\"a\":1}"), ("flag", "pick")])
}
#[test]
fn keeps_foreign_properties_and_replaces_owned_ones() {
let out = merge_xmp(&foreign(), &fresh(4, &["new", "two"]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
let p = |k: &str| d.properties.get(k).cloned().unwrap_or_default();
assert_eq!(p("xmp:Rating"), vec!["4"]);
assert_eq!(p("dc:subject"), vec!["new", "two"]);
assert_eq!(p("lc:settings"), vec!["{\"a\":1}"]);
assert_eq!(p("lc:flag"), vec!["pick"]);
assert_eq!(p("xmp:CreatorTool"), vec!["Other App"], "not ours: kept");
assert_eq!(p("crs:Exposure2012"), vec!["+0.50"]);
assert_eq!(p("foo:Thing").len(), 1);
// untouched parts are byte for byte the same
for part in [
" <xmpMM:History>\n <rdf:Seq>\n <rdf:li stEvt:action=\"saved\" stEvt:when=\"2026-01-02T03:04:05\"/>\n </rdf:Seq>\n </xmpMM:History>",
" <crs:ToneCurvePV2012>\n <rdf:Seq>\n <rdf:li>0, 0</rdf:li>",
"crs:Contrast2012=\"-12\"",
"foo:Thing=\"a &amp; b\"",
"<?xpacket begin=\"\u{feff}\" id=\"W5M0MpCehiHzreSzNTczkc9d\"?>\n<x:xmpmeta xmlns:x=\"adobe:ns:meta/\" x:xmptk=\"Some Toolkit 1.0\">",
&format!("</x:xmpmeta>\n{}\n<?xpacket end=\"w\"?>", " ".repeat(100)),
] {
assert!(out.contains(part), "lost {part:?} in\n{out}");
}
assert!(!out.contains("xmp:Rating=\"2\""));
assert!(!out.contains("<rdf:li>old</rdf:li>"));
// saving again changes only our part, and doesn't pile up descriptions
let again = merge_xmp(&out, &fresh(1, &[]), RULES).unwrap();
let d = parse_xmp(&again).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["1".to_string()]));
assert!(!d.properties.contains_key("dc:subject"), "a cleared field is cleared");
assert_eq!(again.matches("<rdf:Description").count(), 2, "{again}");
assert_eq!(merge_xmp(&again, &fresh(1, &[]), RULES).unwrap(), again, "idempotent");
}
#[test]
fn owned_if_present_and_prefix_independence() {
// the existing packet binds the namespaces to other prefixes, and has a capture time
let ex = "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><r:RDF xmlns:r=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\
<r:Description r:about=\"\" xmlns:a=\"http://ns.adobe.com/xap/1.0/\" xmlns:e=\"http://ns.adobe.com/exif/1.0/\" \
a:Rating=\"5\" e:DateTimeOriginal=\"2020-01-01T00:00:00\"/></r:RDF></x:xmpmeta>";
let out = merge_xmp(ex, &fresh(3, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("xmp:Rating"), Some(&vec!["3".to_string()]));
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2020-01-01T00:00:00".to_string()]), "fresh has none: kept");
let mut m = crate::Metadata { rating: Some(3), ..Default::default() };
m.capture_time = crate::DateTime::parse_iso("2024-05-06T07:08:09");
let out = merge_xmp(ex, &crate::write_xmp_lc(&m, &[]), RULES).unwrap();
let d = parse_xmp(&out).unwrap();
assert_eq!(d.properties.get("exif:DateTimeOriginal"), Some(&vec!["2024-05-06T07:08:09".to_string()]));
}
#[test]
fn refuses_what_it_cannot_read() {
for bad in ["", "not xml at all", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"><rdf:RDF", "<a><b></a>", "<x:xmpmeta xmlns:x=\"adobe:ns:meta/\"/>"] {
assert!(merge_xmp(bad, &fresh(1, &[]), RULES).is_err(), "{bad:?}");
}
}
}
+18 -1
View File
@@ -1,4 +1,4 @@
use lightcraft_meta::{Metadata, extract, parse_iptc, parse_xmp, read_exif, write_xmp};
use lightcraft_meta::{MergeRules, Metadata, extract, merge_xmp, parse_iptc, parse_xmp, read_exif, write_xmp};
use proptest::prelude::*;
proptest! {
@@ -21,6 +21,23 @@ proptest! {
let _ = parse_xmp(&s);
}
/// Merging into arbitrary (also damaged) packets never panics, and a merge that succeeds
/// reads back with the written values.
#[test]
fn merge_never_panics(cut in 0usize..900, junk in ".{0,20}", rating in 0i8..=5) {
let base = write_xmp(&Metadata { title: Some("t".into()), rating: Some(1), ..Default::default() }, Some("{}"));
let mut s: String = base.chars().take(cut).collect();
s.push_str(&junk);
s.extend(base.chars().skip(cut));
let fresh = write_xmp(&Metadata { rating: Some(rating), ..Default::default() }, Some("{\"x\":1}"));
let rules = MergeRules { owned: &["xmp:Rating", "dc:title", "lc:*"], owned_if_present: &[] };
if let Ok(out) = merge_xmp(&s, &fresh, rules) {
let d = parse_xmp(&out).unwrap();
prop_assert_eq!(d.metadata.rating, Some(rating));
prop_assert_eq!(d.lc_settings.as_deref(), Some("{\"x\":1}"));
}
}
#[test]
fn user_text_roundtrips(title in "[a-zA-Z0-9 &<>\"'éü€;|/\\\\]{1,40}", kw in proptest::collection::vec("[a-zA-Z0-9 &<>\"'|]{1,12}", 0..5), rating in -1i8..=5) {
let title = title.trim().to_string();
+5
View File
@@ -82,9 +82,14 @@ pub fn inspect(app: &LightcraftApp, ctx: &egui::Context) -> Value {
"loupe": app.loupe_shown.map(|(p, src)| json!({"photo": p.0, "source": src, "pending": app.renderer.is_pending(crate::render::Slot::Main)})),
"hoverPreview": app.hover_preview.as_ref().map(|h| h.label.clone()),
"status": app.ui.status,
"notices": app.notices,
"quitPrompt": app.quit_prompt,
"unsaved": app.session.unsaved().map(|(n, e)| json!({"ops": n, "error": e})),
"libraryProblem": app.library_problem.as_ref().map(crate::panels::library_problem::LibraryProblem::to_json),
"scan": app.scan.as_ref().map(crate::import::ScanTask::status),
"export": {"running": app.export.as_ref().map(crate::export_task::ExportTask::status), "last": app.last_export_result},
"import": app.import.as_ref().map(crate::import::ImportTask::status),
"tasks": app.tasks.labels(),
"memory": memory(app),
})
}
+1
View File
@@ -176,6 +176,7 @@ impl Headless {
|| self.app.scan.is_some()
|| self.app.import.is_some()
|| self.app.export.is_some()
|| !self.app.tasks.is_empty()
|| !self.app.synthetic.is_empty()
|| !self.events.is_empty()
}
+270 -30
View File
@@ -3,8 +3,9 @@
//! a grid of thumbnails with checkboxes (duplicates marked and unchecked), the destination (add in
//! place / copy or move into the library's `Originals/` or a chosen folder, filed by day, by month,
//! into one folder or by a custom folder template, optionally renamed), an album
//! (existing or new), a preset and keywords to apply. Importing runs in small batches, one per
//! frame, with a progress window; the whole import is one undo step.
//! (existing or new), a preset and keywords to apply. Importing runs on a worker thread (files are
//! probed, copied or moved there) and its batches join the catalog between frames, with a progress
//! window and Cancel; the whole import is one undo step.
use egui::{Align2, Color32, Rect, Sense, Stroke, StrokeKind, pos2, vec2};
use lightcraft_engine::import::{ImportCandidate, ScanInput, ScanOutput, ScanProgress, scan_with};
@@ -17,7 +18,7 @@ use crate::render::Slot;
use crate::theme::Tokens;
use crate::widgets::register;
/// Files per batch (one batch per frame, so the progress window updates).
/// Files per batch (each batch joins the catalog as it is ready, so the progress window updates).
pub(crate) const BATCH: usize = 8;
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
@@ -91,9 +92,13 @@ impl ImportDialog {
}
}
/// A running import (see the module docs).
/// A running import (see the module docs). The file-system work — expanding folders, probing,
/// reading sidecars, copying or placing files — runs on a worker thread
/// ([`lightcraft_engine::import::ImportJob`]); each batch it readies is added to the catalog on the
/// UI thread between frames, so a slow drive never stalls the window.
#[derive(Debug, Default)]
pub struct ImportTask {
/// Files and folders to import (handed to the worker when the task starts).
queue: Vec<String>,
pub total: usize,
pub done: usize,
@@ -109,6 +114,156 @@ pub struct ImportTask {
/// Reading a folder for the Local view: the photos stay out of the library, and nothing is
/// selected or announced as added.
browse: bool,
/// Cancel was pressed: no further files are started; batches already readied are added.
pub cancelled: bool,
/// Auto Import (the watched folder): the selection stays as it is, a short toast when done.
auto: bool,
/// Auto Import: the selection to keep.
keep_selection: Option<lightcraft_engine::Selection>,
run: Option<ImportRun>,
}
impl ImportTask {
/// An import of `paths` (files or folders) with `library.import` params (without `paths`).
pub fn new(paths: Vec<String>, params: Value, undo0: usize, browse: bool) -> Self {
ImportTask { total: paths.len(), queue: paths, params, undo0, browse, ..Default::default() }
}
/// An Auto Import (see [`ImportTask::auto`]).
pub fn auto(mut self) -> Self {
self.auto = true;
self
}
/// `{done, total, imported, cancelled}` for `ui.inspect`.
pub fn status(&self) -> Value {
json!({"done": self.done, "total": self.total, "imported": self.imported, "cancelled": self.cancelled})
}
}
/// Where an import's batches are readied: a worker thread, or (browser build: no threads, and no
/// slow drives) inline, one batch per frame.
enum Runner {
#[cfg(not(target_arch = "wasm32"))]
Thread(std::sync::mpsc::Receiver<lightcraft_engine::import::Prepared>),
#[cfg(target_arch = "wasm32")]
Inline { job: Box<lightcraft_engine::import::ImportJob>, queue: Vec<String>, files: Option<std::collections::VecDeque<String>> },
}
struct ImportRun {
runner: Runner,
cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
total: std::sync::Arc<std::sync::atomic::AtomicUsize>,
done: std::sync::Arc<std::sync::atomic::AtomicUsize>,
opts: lightcraft_engine::import::ImportOptions,
now: String,
album: Option<u64>,
album_name: Option<String>,
}
impl std::fmt::Debug for ImportRun {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ImportRun").field("cancel", &self.cancel.load(Ordering::Relaxed)).finish()
}
}
impl Drop for ImportRun {
/// A dropped import (replaced, or the app closing) stops its worker at the next file.
fn drop(&mut self) {
self.cancel.store(true, Ordering::Relaxed);
}
}
impl ImportRun {
fn start(
job: lightcraft_engine::import::ImportJob,
queue: Vec<String>,
album: Option<u64>,
album_name: Option<String>,
ctx: &egui::Context,
) -> Result<Self, String> {
let cancel = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let (total, done) = (job.total.clone(), job.done.clone());
let (opts, now) = (job.opts.clone(), job.now().to_string());
let runner = Self::spawn(job, queue, cancel.clone(), ctx)?;
Ok(ImportRun { runner, cancel, total, done, opts, now, album, album_name })
}
#[cfg(not(target_arch = "wasm32"))]
fn spawn(
mut job: lightcraft_engine::import::ImportJob,
queue: Vec<String>,
cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
ctx: &egui::Context,
) -> Result<Runner, String> {
let (tx, rx) = std::sync::mpsc::channel();
let ctx = ctx.clone();
let work = move || {
let files = job.expand(&queue);
for chunk in files.chunks(BATCH) {
if cancel.load(Ordering::Relaxed) {
break;
}
let prepared = match lightcraft_engine::guard::catch("import", || job.prepare_files(chunk.to_vec(), &cancel)) {
Ok(p) => p,
Err(_) => break, // logged; the files readied so far are added
};
if let Err(unsent) = tx.send(prepared) {
// the import was dropped: nothing will add these, so take back what a Move placed
unsent.0.rollback();
break;
}
ctx.request_repaint();
}
ctx.request_repaint();
};
std::thread::Builder::new().name("lc-import".into()).spawn(work).map(|_| Runner::Thread(rx)).map_err(|e| format!("could not start: {e}"))
}
#[cfg(target_arch = "wasm32")]
fn spawn(
job: lightcraft_engine::import::ImportJob,
queue: Vec<String>,
_cancel: std::sync::Arc<std::sync::atomic::AtomicBool>,
_ctx: &egui::Context,
) -> Result<Runner, String> {
Ok(Runner::Inline { job: Box::new(job), queue, files: None })
}
/// Readied batches (at most a few per frame). `None` once the worker is done.
fn next_batches(&mut self) -> Option<Vec<lightcraft_engine::import::Prepared>> {
match &mut self.runner {
#[cfg(not(target_arch = "wasm32"))]
Runner::Thread(rx) => {
let mut out = Vec::new();
while out.len() < 4 {
match rx.try_recv() {
Ok(p) => out.push(p),
Err(std::sync::mpsc::TryRecvError::Empty) => return Some(out),
Err(std::sync::mpsc::TryRecvError::Disconnected) => return if out.is_empty() { None } else { Some(out) },
}
}
Some(out)
}
#[cfg(target_arch = "wasm32")]
Runner::Inline { job, queue, files } => {
if self.cancel.load(Ordering::Relaxed) {
return None;
}
if files.is_none() {
*files = Some(job.expand(queue).into());
return Some(Vec::new());
}
let Some(files) = files.as_mut() else { return None };
if files.is_empty() {
return None;
}
let n = files.len().min(BATCH);
let batch: Vec<String> = files.drain(..n).collect();
Some(vec![job.prepare_files(batch, &self.cancel)])
}
}
}
}
/// A folder scan running on a worker thread (a network share can take minutes to read; the
@@ -232,7 +387,8 @@ pub fn poll_scan(app: &mut LightcraftApp, ctx: &egui::Context) {
if !queue.is_empty() {
let undo0 = app.session.undo.len();
let total = queue.len();
app.import = Some(ImportTask { queue, total, params: json!({"mode": "add", "local": true}), undo0, browse: true, ..Default::default() });
let _ = total;
app.import = Some(ImportTask::new(queue, json!({"mode": "add", "local": true}), undo0, true));
}
return;
}
@@ -326,28 +482,82 @@ pub fn start(app: &mut LightcraftApp, d: &ImportDialog) -> Result<Value, String>
}
}
let total = queue.len();
app.import = Some(ImportTask { queue, total, params, undo0, ..Default::default() });
app.import = Some(ImportTask::new(queue, params, undo0, false));
app.renderer.forget_imports();
Ok(json!({"importing": total}))
}
/// Run one batch of the import in progress (called every frame).
pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = app.import.as_mut() else { return };
// (browsing reads a sidecar per file, slow on a network share: keep each frame short)
let n = task.queue.len().min(if task.browse { 2 } else { BATCH });
let batch: Vec<String> = task.queue.drain(..n).collect();
let mut p = task.params.clone();
p["paths"] = json!(batch);
// renamed copies keep counting across batches
if p.get("rename").is_some() {
p["renameStart"] = json!(1 + task.imported);
/// Start importing `paths` (files or folders) in the background, e.g. dropped on the window.
pub fn start_paths(app: &mut LightcraftApp, paths: Vec<String>) -> Result<Value, String> {
if app.import.is_some() || app.scan.as_ref().is_some_and(|t| !t.browse) {
return Err("an import is running".into());
}
let r = app.session.execute("library.import", &p);
let Some(task) = app.import.as_mut() else { return };
task.done += n;
let undo0 = app.session.undo.len();
let total = paths.len();
app.import = Some(ImportTask::new(paths, json!({"mode": "add"}), undo0, false));
Ok(json!({"importing": total}))
}
/// Advance the import in progress (called every frame): start its worker, then add the batches it
/// has readied to the catalog.
pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(mut task) = app.import.take() else { return };
if task.run.is_none() {
let mut p = task.params.clone();
p["paths"] = json!(task.queue);
let started = lightcraft_engine::cmd::library::import_params(&app.session, &p)
.and_then(|req| Ok((lightcraft_engine::import::ImportJob::new(&mut app.session, req.opts)?, req.album, req.album_name)))
.map_err(|e| e.to_string())
.and_then(|(job, album, name)| ImportRun::start(job, std::mem::take(&mut task.queue), album, name, ctx));
if task.auto {
task.keep_selection = Some(app.session.selection.clone());
}
match started {
Ok(run) => task.run = Some(run),
Err(e) => {
log::warn!("import: {e}");
app.toast(ctx, format!("Import failed: {e}"));
return;
}
}
}
ctx.request_repaint_after(std::time::Duration::from_millis(100));
let Some(run) = task.run.as_mut() else { return };
let batches = run.next_batches();
let finished = batches.is_none();
for prepared in batches.into_iter().flatten() {
commit_batch(app, &mut task, prepared);
// arriving photos don't take over the selection
if let Some(sel) = &task.keep_selection {
app.session.selection = sel.clone();
}
}
if let Some(run) = &task.run {
task.total = run.total.load(Ordering::Relaxed).max(task.total);
task.done = run.done.load(Ordering::Relaxed);
}
if !finished {
app.import = Some(task);
return;
}
finish(app, ctx, task);
}
/// Add one readied batch to the catalog (an undo step merged into the import's at the end).
fn commit_batch(app: &mut LightcraftApp, task: &mut ImportTask, prepared: lightcraft_engine::import::Prepared) {
let Some(run) = task.run.as_ref() else { return };
let n = prepared.len();
let (opts, now, album, album_name) = (&run.opts, run.now.as_str(), run.album, run.album_name.clone());
let r = app.session.execute_fn("library.import", |s| {
let report = lightcraft_engine::import::commit_prepared(s, opts, now, prepared)?;
lightcraft_engine::cmd::library::import_batch_done(s, report, album, album_name.as_deref())
});
match r {
Ok(v) => {
// a new album (named in the params) is created with the first photos; later batches join it
if let (Some(a), Some(run)) = (v["album"].as_u64(), task.run.as_mut()) {
run.album = Some(a);
}
let len = |k: &str| v[k].as_array().map_or(0, Vec::len);
task.imported += len("imported");
task.duplicates += len("duplicates");
@@ -366,14 +576,19 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
task.failed += n;
}
}
ctx.request_repaint();
if !task.queue.is_empty() {
return;
}
let Some(task) = app.import.take() else { return };
}
/// The import is done (or cancelled): one undo step, select the first photo, say what happened.
fn finish(app: &mut LightcraftApp, ctx: &egui::Context, task: ImportTask) {
let steps = app.session.undo.len().saturating_sub(task.undo0);
let label = format!("Add {} Photo{}", task.imported, if task.imported == 1 { "" } else { "s" });
app.session.merge_undo(steps, &label);
if task.auto {
if task.imported > 0 {
app.toast(ctx, format!("Auto Import: added {} photo{}", task.imported, if task.imported == 1 { "" } else { "s" }));
}
return;
}
if task.browse {
if task.failed > 0 {
app.toast(ctx, format!("{} photo{} not readable", task.failed, if task.failed == 1 { "" } else { "s" }));
@@ -384,7 +599,11 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
let _ = app.run("library.select", json!({"ids": [f]}));
}
let plural = |n: usize| if n == 1 { "" } else { "s" };
let mut msg = format!("Imported {} photo{}", task.imported, plural(task.imported));
let mut msg = if task.cancelled {
format!("Import cancelled · {} photo{} added", task.imported, plural(task.imported))
} else {
format!("Imported {} photo{}", task.imported, plural(task.imported))
};
if task.params["mode"] == "move" {
msg.push_str(&format!(" · {} moved", task.moved));
if task.kept > 0 {
@@ -400,19 +619,34 @@ pub fn tick(app: &mut LightcraftApp, ctx: &egui::Context) {
app.toast(ctx, msg);
}
/// The progress window while an import runs.
/// The progress window while an import runs, with Cancel (files already copied or added stay;
/// nothing new is started).
pub fn progress(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(task) = &app.import else { return };
let t = Tokens::get(ctx);
let frac = task.done as f32 / task.total.max(1) as f32;
let text = format!("{} photos… {} of {}", if task.browse { "Reading" } else { "Adding" }, task.done, task.total);
egui::Window::new("Importing").title_bar(false).resizable(false).anchor(Align2::CENTER_BOTTOM, [0.0, -80.0]).fixed_size([340.0, 60.0]).show(
let text = if task.cancelled {
"Stopping…".to_string()
} else {
format!("{} photos… {} of {}", if task.browse { "Reading" } else { "Adding" }, task.done, task.total)
};
let mut cancel = false;
egui::Window::new("Importing").title_bar(false).resizable(false).anchor(Align2::CENTER_BOTTOM, [0.0, -80.0]).fixed_size([340.0, 80.0]).show(
ctx,
|ui| {
ui.label(egui::RichText::new(text).color(t.text));
ui.add(egui::ProgressBar::new(frac).desired_width(320.0));
let r = ui.add_enabled(!task.cancelled, egui::Button::new("Cancel"));
register(ui.ctx(), "button:importCancel", r.rect);
cancel = r.clicked();
},
);
if cancel && let Some(task) = app.import.as_mut() {
task.cancelled = true;
if let Some(run) = &task.run {
run.cancel.store(true, Ordering::Relaxed);
}
}
}
/// The dialog body: options, then the candidate grid.
@@ -458,7 +692,13 @@ pub fn body(app: &mut LightcraftApp, ui: &mut egui::Ui, d: &mut ImportDialog) {
// options
if !d.sources.is_empty() {
field(ui, "Source", |ui| {
let r = ui.label(egui::RichText::new(source_summary(&d.sources)).color(t.text_label)).on_hover_text(d.sources.join("\n"));
// (whether each source is a folder is checked off the UI thread, once)
let joined = d.sources.join("\n");
let summary = crate::panels::left::fs_cached(ui, "import-source", &joined, f64::INFINITY, |s| {
source_summary(&s.split('\n').map(str::to_string).collect::<Vec<_>>())
})
.unwrap_or_else(|| format!("{} item{}", d.sources.len(), if d.sources.len() == 1 { "" } else { "s" }));
let r = ui.label(egui::RichText::new(summary).color(t.text_label)).on_hover_text(joined);
register(ui.ctx(), "label:importSource", r.rect);
});
ui.label(
+87 -11
View File
@@ -21,6 +21,7 @@ pub mod render;
pub mod shortcuts;
pub mod softpaint;
pub mod state;
pub mod tasks;
pub mod theme;
pub mod widgets;
@@ -29,10 +30,16 @@ mod tests_curve;
#[cfg(test)]
mod tests_grid;
#[cfg(test)]
mod tests_library_problem;
#[cfg(test)]
mod tests_masking;
#[cfg(test)]
mod tests_offline;
#[cfg(test)]
mod tests_panels;
#[cfg(test)]
mod tests_quit_unsaved;
#[cfg(test)]
mod tests_scroll;
#[cfg(test)]
mod tests_unsaved;
@@ -60,6 +67,9 @@ pub type RevealFn = Box<dyn FnMut(&str) -> Result<(), String>>;
pub type OpenUrlFn = Box<dyn FnMut(&str) -> Result<(), String>>;
/// Open a file in an application (`app` = "" for the system's default one).
pub type OpenWithFn = Box<dyn FnMut(&str, &str) -> Result<(), String>>;
/// Something the host does with the session (web: back up or restore the library in browser
/// storage); the work may finish asynchronously.
pub type HostAction = Box<dyn FnMut(&mut Session) -> Result<Value, String>>;
/// Platform services injected by the host app (desktop or web).
#[derive(Default)]
@@ -89,6 +99,12 @@ pub struct Services {
pub open_url: Option<OpenUrlFn>,
/// Open a file in an external editor (Edit in External Editor; desktop only).
pub open_with: Option<OpenWithFn>,
/// File ▸ Back Up Library…: save the whole library (catalog and originals) as one file the
/// user keeps (web only: there the library lives in browser storage, which the browser may
/// clear; on the desktop it is a folder backed up like any other).
pub backup_library: Option<HostAction>,
/// File ▸ Restore Library from Backup… (web only; keeps the current library).
pub restore_library: Option<HostAction>,
}
#[derive(Clone, Copy, Debug, Default)]
@@ -123,6 +139,12 @@ pub struct LightcraftApp {
pub native_shortcuts: std::collections::HashSet<String>,
/// The host is [`headless::Headless`] (it answers viewport screenshot commands itself).
pub headless_host: bool,
/// Warnings to show one at a time (damaged settings files…, issue #103).
pub notices: Vec<String>,
/// Quitting was stopped because changes couldn't be saved: the prompt's text.
pub quit_prompt: Option<String>,
/// Quit Anyway was chosen: the window may close with unsaved changes.
pub quit_confirmed: bool,
control_rx: Option<Receiver<ControlRequest>>,
pending_screenshots: Vec<PendingShot>,
screenshot_token: u64,
@@ -159,6 +181,8 @@ pub struct LightcraftApp {
pub scan: Option<import::ScanTask>,
/// A background export in progress.
pub export: Option<export_task::ExportTask>,
/// Background file-system work of other commands (Find Missing Photos, auto import…).
pub tasks: tasks::Tasks,
/// The files of the last finished background export (`ui.inspect` → `export.last`).
pub last_export_result: Option<Value>,
/// The look the loupe shows while the pointer rests on a preset or profile (set by the
@@ -170,6 +194,9 @@ pub struct LightcraftApp {
gpu_applied: Option<bool>,
/// The memory budget setting last applied (MB, 0 = automatic).
memory_applied: Option<u32>,
/// The library failed to open at launch: the blocking window, then the temporary-session
/// banner (issue #100). Cleared once a library opens.
pub library_problem: Option<panels::library_problem::LibraryProblem>,
}
impl LightcraftApp {
@@ -187,6 +214,9 @@ impl LightcraftApp {
native_menu: false,
native_shortcuts: Default::default(),
headless_host: false,
notices: vec![],
quit_prompt: None,
quit_confirmed: false,
control_rx: None,
pending_screenshots: vec![],
screenshot_token: 0,
@@ -208,11 +238,13 @@ impl LightcraftApp {
import: None,
scan: None,
export: None,
tasks: Default::default(),
last_export_result: None,
hover_preview: None,
window_is_fullscreen: false,
gpu_applied: None,
memory_applied: None,
library_problem: None,
}
}
@@ -269,7 +301,12 @@ impl LightcraftApp {
if self.ui.preview_build_seen != Some((key, true)) {
self.ui.preview_build_seen = Some((key, true));
let (done, failed) = (b.done.load(Ordering::Relaxed), b.failed.load(Ordering::Relaxed));
let mut msg = format!("Previews ready for {done} photo{}", if done == 1 { "" } else { "s" });
let plural = if done == 1 { "" } else { "s" };
let mut msg = match (b.error(), b.what) {
(Some(e), what) => format!("{}: {e}", if what.is_empty() { "previews" } else { what }),
(None, "") => format!("Previews ready for {done} photo{plural}"),
(None, what) => format!("Done ({what}): {done} photo{plural}"),
};
if failed > 0 {
msg.push_str(&format!(" · {failed} couldn't be rendered"));
}
@@ -278,7 +315,11 @@ impl LightcraftApp {
} else {
if self.ui.preview_build_seen != Some((key, false)) {
self.ui.preview_build_seen = Some((key, false));
self.toast(ctx, format!("Building previews for {} photos…", b.total));
let msg = match b.what {
"" => format!("Building previews for {} photos…", b.total),
what => format!("Working on {what} for {} photos…", b.total),
};
self.toast(ctx, msg);
}
ctx.request_repaint_after(std::time::Duration::from_millis(250));
}
@@ -440,6 +481,7 @@ impl LightcraftApp {
/// Per-frame logic before layout (control channel, renders, shortcuts, drops).
pub fn logic(&mut self, ctx: &egui::Context) {
let t0 = now_ms();
panels::library_problem::logic(self);
self.logic_inner(ctx);
self.perf.logic_ms = now_ms() - t0;
}
@@ -451,10 +493,19 @@ impl LightcraftApp {
// File → Add from Device lists cards scanned in the background: show hot-plugs
let repaint = ctx.clone();
lightcraft_engine::devices::on_change(move || repaint.request_repaint());
// "is the original there?" (grid, Info panel, Missing Photos) answers from a cache a
// worker fills: a sleeping NAS or a dropped share never stalls a frame
let repaint = ctx.clone();
self.session.media.availability.run_in_background(std::sync::Arc::new(move || repaint.request_repaint()));
self.styled = true;
} else {
self.fonts_ready = true;
}
panels::notices::logic(self);
// closing the window (or Quit) with changes only in memory: retry, else ask first
if ctx.input(|i| i.viewport().close_requested()) && !panels::notices::may_close(self) {
ctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
}
let now = ctx.input(|i| i.time);
let dt = now - self.last_time;
if dt > 0.0 {
@@ -470,6 +521,7 @@ impl LightcraftApp {
merge::poll(self, ctx);
import::poll_scan(self, ctx);
import::tick(self, ctx);
tasks::poll(self, ctx);
self.preview_build_status(ctx);
self.save_status(ctx);
self.slideshow_tick(ctx);
@@ -484,17 +536,33 @@ impl LightcraftApp {
}
}
self.ui.was_focused = focused;
// auto import: scan the watched folder every few seconds
// auto import: list the watched folder every few seconds (on a worker thread: it may be on
// a network share), then import what's new like any import (also on a worker thread)
#[cfg(not(target_arch = "wasm32"))]
if self.session.import_defaults.auto_folder.is_some() {
if let Some(folder) = self.session.import_defaults.auto_folder.clone() {
const LABEL: &str = "Auto Import";
let now = ctx.input(|i| i.time);
if now - self.ui.auto_import_at >= 3.0 {
if now - self.ui.auto_import_at >= 3.0 && self.import.is_none() && !self.tasks.is_running(LABEL) {
self.ui.auto_import_at = now;
if let Ok(r) = self.session.execute("library.autoImportScan", &serde_json::json!({})) {
let n = r["imported"].as_array().map_or(0, Vec::len);
if n > 0 {
self.toast(ctx, format!("Auto Import: added {n} photo{}", if n == 1 { "" } else { "s" }));
let work = move || lightcraft_engine::cmd::library::list_auto_import_folder(&folder);
let done = |app: &mut LightcraftApp, _ctx: &egui::Context, listing: Result<Vec<(String, u64)>, String>| {
let listing = match listing {
Ok(l) => l,
Err(e) => return log::debug!("auto import: {e}"),
};
let p = serde_json::json!({"listing": listing, "start": false});
let Ok(r) = app.session.execute("library.autoImportScan", &p) else { return };
let Some(mut params) = r.get("import").cloned().filter(|_| app.import.is_none()) else { return };
let paths: Vec<String> =
params["paths"].as_array().into_iter().flatten().filter_map(|v| v.as_str().map(str::to_string)).collect();
if let Some(o) = params.as_object_mut() {
o.remove("paths");
}
let undo0 = app.session.undo.len();
app.import = Some(import::ImportTask::new(paths, params, undo0, false).auto());
};
if let Err(e) = tasks::spawn(self, LABEL, work, done) {
log::warn!("{e}");
}
}
ctx.request_repaint_after(std::time::Duration::from_secs(3));
@@ -514,8 +582,11 @@ impl LightcraftApp {
if !presets.is_empty() {
let _ = self.run("file.importPresets", serde_json::json!({"paths": presets}));
}
if !photos.is_empty() {
let _ = self.run("library.import", serde_json::json!({"paths": photos}));
// read and added on a worker thread (dropped folders can be large, or on a slow drive)
if !photos.is_empty()
&& let Err(e) = import::start_paths(self, photos)
{
self.toast(ctx, e);
}
}
}
@@ -601,7 +672,9 @@ impl LightcraftApp {
// full-screen preview: the photo alone on black
egui::CentralPanel::default().frame(egui::Frame::NONE.fill(egui::Color32::BLACK)).show(ui, |ui| panels::detail::show(self, ui));
panels::second::show(self, &ctx);
panels::notices::show(self, &ctx);
panels::dialogs::show(self, &ctx);
panels::library_problem::show(self, &ctx);
panels::toast(self, &ctx);
self.widgets = widgets::take_registry(&ctx);
self.end_frame(t0);
@@ -610,6 +683,7 @@ impl LightcraftApp {
// Order matters: earlier panels take the full edge (top bar spans the window; the tool strip,
// right panels and left panel run to the bottom; the bottom bar sits between them).
panels::topbar::show(self, ui);
panels::library_problem::banner(self, ui);
panels::strip::show(self, ui);
if self.ui.right != state::RightPanel::None {
panels::right::show(self, ui);
@@ -636,7 +710,9 @@ impl LightcraftApp {
state::ViewMode::Reference => panels::compare::show_reference(self, ui),
});
panels::second::show(self, &ctx);
panels::notices::show(self, &ctx);
panels::dialogs::show(self, &ctx);
panels::library_problem::show(self, &ctx);
import::progress(self, &ctx);
import::scan_progress(self, &ctx);
export_task::poll(self, &ctx);
+47 -6
View File
@@ -58,6 +58,8 @@ const LAYOUT: &[(&str, &[&str])] = &[
"@Import from Device",
"---",
"app.openLibrary",
"file.backupLibrary",
"file.restoreLibrary",
"---",
"dialog.newAlbum",
"dialog.newFolder",
@@ -241,6 +243,15 @@ const HIDDEN: &[&str] = &[
"preset.create",
];
/// Items only some hosts have are left out of the others' menus (the web build's library backup).
fn host_supports(app: &LightcraftApp, id: &str) -> bool {
match id {
"file.backupLibrary" => app.services.backup_library.is_some(),
"file.restoreLibrary" => app.services.restore_library.is_some(),
_ => true,
}
}
fn item(id: &str, params: Value, label: impl Into<String>, shortcut: Option<&str>, enabled: bool, checked: Option<bool>) -> MenuNode {
MenuNode::Item { id: id.into(), params, label: label.into(), shortcut: shortcut.map(str::to_string), enabled, checked }
}
@@ -325,20 +336,21 @@ fn expanded(app: &LightcraftApp, name: &str) -> Option<Vec<MenuNode>> {
item("library.buildPreviews", json!({"size": "full"}), format!("Build 1:1 Previews ({scope})"), None, !running, None),
item("library.cancelPreviews", Value::Null, "Stop Building Previews", None, running, None),
MenuNode::Separator,
// (read and written on a worker thread: the originals may be on a slow drive)
item(
"library.smartPreviews",
Value::Null,
json!({"background": true}),
format!("Build Smart Previews ({scope})"),
None,
app.session.media.smart_dir.is_some(),
app.session.media.smart_dir.is_some() && !running,
None,
),
item(
"library.smartPreviews",
json!({"discard": true}),
json!({"discard": true, "background": true}),
format!("Discard Smart Previews ({scope})"),
None,
app.session.media.smart_dir.is_some(),
app.session.media.smart_dir.is_some() && !running,
None,
),
MenuNode::Separator,
@@ -537,7 +549,8 @@ fn tidy(v: Vec<MenuNode>) -> Vec<MenuNode> {
/// The whole menu bar: (title, items) per menu in [`MENUS`] order.
pub fn menu_bar(app: &LightcraftApp) -> Vec<(String, Vec<MenuNode>)> {
let entries: Vec<MenuEntry> = crate::menus::menu_entries(app).into_iter().filter(|e| !HIDDEN.contains(&e.id.as_str())).collect();
let entries: Vec<MenuEntry> =
crate::menus::menu_entries(app).into_iter().filter(|e| !HIDDEN.contains(&e.id.as_str()) && host_supports(app, &e.id)).collect();
let mut used = vec![false; entries.len()];
let mut bar = Vec::new();
for title in MENUS {
@@ -774,6 +787,34 @@ mod tests {
assert!(!text.contains("Add Folder") && !text.contains("\"Add Photos"), "no add-folder wording left in the menus");
}
/// Back Up / Restore Library are the browser build's (its library lives in browser storage):
/// absent from the desktop's menus, present and wired to the host where it provides them.
#[test]
fn library_backup_items_follow_the_host() {
let all = |app: &LightcraftApp| -> Vec<MenuNode> { menu_bar(app).into_iter().flat_map(|(_, v)| v).collect() };
let mut desktop = app();
assert!(find(&all(&desktop), "file.backupLibrary").is_none() && find(&all(&desktop), "file.restoreLibrary").is_none());
assert!(run_item(&mut desktop, "file.backupLibrary", Value::Null).is_err(), "not available without the host");
let calls = std::rc::Rc::new(std::cell::Cell::new(0));
let c = calls.clone();
let services = crate::Services {
backup_library: Some(Box::new(move |s: &mut lightcraft_engine::Session| {
c.set(c.get() + 1);
Ok(json!({"photos": s.catalog.len()}))
})),
restore_library: Some(Box::new(|_: &mut lightcraft_engine::Session| Ok(json!({"started": true})))),
..Default::default()
};
let mut web = LightcraftApp::new(lightcraft_engine::Session::with_demo(), services);
let bar = all(&web);
for id in ["file.backupLibrary", "file.restoreLibrary"] {
assert!(matches!(find(&bar, id), Some(MenuNode::Item { enabled: true, .. })), "{id}");
}
let r = run_item(&mut web, "file.backupLibrary", Value::Null).unwrap();
assert!(r["photos"].as_u64().unwrap() > 0);
assert_eq!(calls.get(), 1);
}
#[test]
fn bar_follows_the_registry_and_state() {
let mut app = app();
@@ -936,7 +977,7 @@ mod tests {
let mut app = LightcraftApp::new(lightcraft_engine::Session::new().with_fs(), Default::default());
app.session.execute("library.import", &json!({"paths": [dir.join("a").to_string_lossy()]})).unwrap();
std::fs::rename(dir.join("a/one.png"), dir.join("b/one.png")).unwrap();
let r = run_item(&mut app, "file.findMissing", json!({"folder": dir.join("b").to_string_lossy()})).unwrap();
let r = run_item(&mut app, "file.findMissing", json!({"folder": dir.join("b").to_string_lossy(), "wait": true})).unwrap();
assert_eq!(r["found"].as_array().map(Vec::len), Some(1), "{r}");
// Locate: an explicit file
std::fs::rename(dir.join("b/one.png"), dir.join("one-renamed.png")).unwrap();
+44 -11
View File
@@ -116,6 +116,8 @@ pub const UI_COMMANDS: &[UiCommand] = &[
("file.addFolder", "Import from Folder…", None, "File"),
("file.addFromDevice", "Import from Device", None, ""),
("file.findMissing", "Find Missing Photos…", None, "File"),
("file.backupLibrary", "Back Up Library…", None, "File"),
("file.restoreLibrary", "Restore Library from Backup…", None, "File"),
("photo.locate", "Locate Missing File…", None, ""),
("dialog.saveMetadataPreset", "Save Metadata Preset…", None, ""),
("app.quit", "Quit LightCraft", Some("Cmd+Q"), "File"),
@@ -381,6 +383,13 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
Ok(Value::Null)
}
"app.openLibrary" => crate::panels::settings::open_library(app, p),
"file.backupLibrary" | "file.restoreLibrary" => {
let action = if id == "file.backupLibrary" { app.services.backup_library.as_mut() } else { app.services.restore_library.as_mut() };
match action {
Some(f) => f(&mut app.session),
None => Err("not available here: on the desktop the library is a folder; back it up with your other files".into()),
}
}
"view.filmstrip" => {
app.ui.filmstrip = !app.ui.filmstrip;
Ok(Value::Null)
@@ -928,18 +937,40 @@ pub fn run_ui_command(app: &mut LightcraftApp, id: &str, p: &Value) -> Option<Re
None => app.services.pick_folder.as_mut().and_then(|f| f()),
};
let Some(folder) = folder else { return Some(Ok(Value::Null)) };
let r = app.run("library.findMissing", json!({"folder": folder}));
if let Ok(v) = &r {
let n = v["found"].as_array().map_or(0, Vec::len);
let left = v["missing"].as_u64().unwrap_or(0);
let unsure = v["ambiguous"].as_array().map_or(0, Vec::len);
let unsure = if unsure > 0 { format!(" ({unsure} with several look-alike files: use Locate)") } else { String::new() };
app.toast(
&egui::Context::default(),
format!("Found {n} missing photo{}; {left} still missing{unsure}", if n == 1 { "" } else { "s" }),
);
// the search (checking every photo's file, walking the folder) runs on a worker thread;
// the relinking happens back here, as one undo step
const LABEL: &str = "Find Missing Photos";
if app.tasks.is_running(LABEL) {
return Some(Err("Find Missing Photos is already searching".into()));
}
r
let candidates = lightcraft_engine::cmd::missing::find_candidates(&app.session.catalog);
let work = move || lightcraft_engine::cmd::missing::plan_find_missing(&candidates, &folder);
let done = |app: &mut LightcraftApp, ctx: &egui::Context, plan: Result<lightcraft_engine::cmd::missing::FindPlan, String>| {
// relinked under the session as it is now: photos relinked meanwhile and files
// now in use are skipped
let r = plan.and_then(|plan| app.run("library.findMissing", plan.to_json()));
match r {
Ok(v) => {
let n = v["found"].as_array().map_or(0, Vec::len);
let left = v["missing"].as_u64().unwrap_or(0);
let unsure = v["ambiguous"].as_array().map_or(0, Vec::len);
let unsure = if unsure > 0 { format!(" ({unsure} with several look-alike files: use Locate)") } else { String::new() };
app.toast(ctx, format!("Found {n} missing photo{}; {left} still missing{unsure}", if n == 1 { "" } else { "s" }));
app.ui.last_find_missing = Some(v);
}
Err(e) => app.toast(ctx, e),
}
};
app.ui.last_find_missing = None;
if let Err(e) = crate::tasks::spawn(app, LABEL, work, done) {
return Some(Err(e));
}
if p.get("wait").and_then(Value::as_bool).unwrap_or(false) {
let ctx = app.tasks.repaint.clone().unwrap_or_default();
crate::tasks::wait(app, &ctx, std::time::Duration::from_secs(600));
return Some(Ok(app.ui.last_find_missing.clone().unwrap_or(Value::Null)));
}
Ok(json!({"background": true}))
}
"photo.tagFromTracklog" => {
// a GPX file → GPS for the selected photos by capture time (one undo step)
@@ -1193,6 +1224,8 @@ pub fn ui_enabled(app: &LightcraftApp, id: &str) -> bool {
"view.compare" => app.session.catalog.len() > 1,
"view.fullScreenPreview" | "view.infoOverlay" | "view.navigator" => app.session.active().is_some() || app.ui.fullscreen,
"app.openLibrary" | "file.addFolder" => app.services.pick_folder.is_some(),
"file.backupLibrary" => app.services.backup_library.is_some(),
"file.restoreLibrary" => app.services.restore_library.is_some(),
"compare.swap" | "compare.makeSelect" => app.ui.view == ViewMode::Compare,
s if s.starts_with("dialog.merge") || (s.starts_with("merge.") && s.ends_with("Last")) => {
app.session.targets(&serde_json::json!({})).len() >= 2 && app.merge.final_task.is_none()
+63 -9
View File
@@ -7,6 +7,53 @@ use crate::LightcraftApp;
use crate::state::Dialog;
use crate::theme::Tokens;
/// Rows the Rename dialog previews.
const RENAME_PREVIEW_ROWS: usize = 6;
/// The Rename dialog's preview: its first rows (`None` while being planned) and how many files
/// are renamed. Which names are taken is checked on disk, which can block on a slow drive, so the
/// rows are planned on a worker thread, and only when the template, start number, photos or
/// catalog change — never once per frame.
pub(crate) fn rename_preview(
app: &mut LightcraftApp,
ctx: &egui::Context,
template: &str,
start: usize,
) -> (Option<Vec<lightcraft_engine::rename::RenamePlan>>, usize) {
use std::hash::{Hash, Hasher};
type Rows = std::sync::Arc<std::sync::Mutex<Option<Vec<lightcraft_engine::rename::RenamePlan>>>>;
let ids = app.session.targets(&json!({}));
let photos = lightcraft_engine::rename::rename_photos(&app.session.catalog, &ids);
let total = photos.len();
let mut h = std::collections::hash_map::DefaultHasher::new();
(template, start, app.session.catalog.revision, &ids).hash(&mut h);
let key = h.finish();
let id = egui::Id::new("rename-preview-rows");
let read = |rows: &Rows| rows.lock().unwrap_or_else(std::sync::PoisonError::into_inner).clone();
if let Some((k, rows)) = ctx.data(|d| d.get_temp::<(u64, Rows)>(id))
&& k == key
{
return (read(&rows), total);
}
let rows: Rows = Default::default();
let first: Vec<_> = photos.into_iter().take(RENAME_PREVIEW_ROWS).collect();
let (out, template, repaint) = (rows.clone(), template.to_string(), ctx.clone());
let exists = app.session.media.availability.probe();
let work = move || {
let plans = lightcraft_engine::rename::plan_rename_photos(&first, &template, start, &|f| exists(f));
*out.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Some(plans);
repaint.request_repaint();
};
#[cfg(not(target_arch = "wasm32"))]
if let Err(e) = std::thread::Builder::new().name("lc-rename-preview".into()).spawn(work) {
log::warn!("rename preview: {e}");
}
#[cfg(target_arch = "wasm32")]
work();
ctx.data_mut(|d| d.insert_temp(id, (key, rows.clone())));
(read(&rows), total)
}
/// Help ▸ What's New (docs/whats-new.md).
pub const WHATS_NEW: &str = include_str!("../../../../docs/whats-new.md");
@@ -316,16 +363,23 @@ pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
egui::RichText::new("Tags: see Tags beside the template. Files are renamed on disk (with their XMP sidecars); existing names get -1, -2…")
.color(t.text_dim),
);
let preview = app.session.execute("photo.renamePreview", &json!({"template": template, "start": start})).unwrap_or_default();
egui::Grid::new("rename-preview").num_columns(3).spacing([8.0, 2.0]).show(ui, |ui| {
for pl in preview.as_array().into_iter().flatten().take(6) {
ui.label(egui::RichText::new(pl["from"].as_str().unwrap_or("")).color(t.text_dim));
ui.label(egui::RichText::new("→").color(t.text_dim));
ui.label(egui::RichText::new(pl["to"].as_str().unwrap_or("")).color(t.text));
ui.end_row();
let (preview, total) = rename_preview(app, ui.ctx(), template, *start as usize);
match &preview {
Some(rows) => {
egui::Grid::new("rename-preview").num_columns(3).spacing([8.0, 2.0]).show(ui, |ui| {
for pl in rows {
ui.label(egui::RichText::new(&pl.from).color(t.text_dim));
ui.label(egui::RichText::new("→").color(t.text_dim));
ui.label(egui::RichText::new(&pl.to).color(t.text));
ui.end_row();
}
});
}
});
let more = preview.as_array().map_or(0, Vec::len).saturating_sub(6);
None => {
ui.label(egui::RichText::new("Checking names…").color(t.text_dim));
}
}
let more = total.saturating_sub(RENAME_PREVIEW_ROWS);
if more > 0 {
ui.label(egui::RichText::new(format!("… and {more} more")).color(t.text_dim));
}
+2 -1
View File
@@ -867,7 +867,8 @@ pub fn context_menu(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
}
ui.separator();
// the original moved or its drive is gone: point the photo at the file again
let missing = matches!(&app.session.catalog.photo(id).map(|p| p.source.clone()), Some(lightcraft_catalog::Source::File { path }) if !std::path::Path::new(path).exists());
// (a cached answer, checked off the UI thread)
let missing = matches!(&app.session.catalog.photo(id).map(|p| p.source.clone()), Some(lightcraft_catalog::Source::File { path }) if app.session.media.availability.is_offline(path));
if missing && ui.button("Locate Missing File…").clicked() {
let _ = app.run("photo.locate", json!({}));
}
+58 -10
View File
@@ -200,7 +200,8 @@ fn local_section(app: &mut LightcraftApp, ui: &mut egui::Ui) {
for (name, sub) in [("Pictures", "Pictures"), ("Desktop", "Desktop"), ("Downloads", "Downloads"), ("Home", "")] {
// joined with the platform's separator, like the paths browsing gives back
let p = if sub.is_empty() { home.clone() } else { std::path::Path::new(&home).join(sub).to_string_lossy().to_string() };
if std::path::Path::new(&p).is_dir() {
// (checked off the UI thread: a home folder can be on a network share)
if fs_cached(ui, "is-dir", &p, 5.0, |p| std::path::Path::new(p).is_dir()) == Some(true) {
builtin.push((name.to_string(), p));
}
}
@@ -305,15 +306,13 @@ pub(crate) fn local_places(
out
}
/// The subfolders of `path` (not hidden ones), sorted; listed at most every 2 s per folder.
/// The subfolders of `path` (not hidden ones), sorted; listed on a worker thread at most every
/// 2 s per folder (empty until the first listing).
fn subfolders(ui: &egui::Ui, path: &str) -> Vec<(String, String)> {
let id = egui::Id::new(("subfolders", path.to_string()));
let now = ui.input(|i| i.time);
if let Some((t, v)) = ui.data(|d| d.get_temp::<(f64, Vec<(String, String)>)>(id))
&& now - t < 2.0
{
return v;
}
fs_cached(ui, "subfolders", path, 2.0, list_subfolders).unwrap_or_default()
}
fn list_subfolders(path: &str) -> Vec<(String, String)> {
let mut v: Vec<(String, String)> = std::fs::read_dir(path)
.map(|rd| {
rd.flatten()
@@ -326,10 +325,59 @@ fn subfolders(ui: &egui::Ui, path: &str) -> Vec<(String, String)> {
})
.unwrap_or_default();
v.sort_by_key(|(n, _)| n.to_lowercase());
ui.data_mut(|d| d.insert_temp(id, (now, v.clone())));
v
}
/// A file-system answer for `path` (`f(path)`), kept per `kind` and path and refreshed on a worker
/// thread at most every `every` seconds: a folder on a sleeping NAS, a dropped share or a
/// spinning-up drive never blocks a frame. `None` until the first answer arrives.
pub(crate) fn fs_cached<T: Clone + Send + 'static>(ui: &egui::Ui, kind: &'static str, path: &str, every: f64, f: fn(&str) -> T) -> Option<T> {
struct Entry<T> {
value: Option<T>,
at: Option<f64>,
running: bool,
}
type Cell<T> = std::sync::Arc<std::sync::Mutex<Entry<T>>>;
let id = egui::Id::new(("fs-cached", kind, path.to_string()));
let now = ui.input(|i| i.time);
let cell: Cell<T> = match ui.data(|d| d.get_temp::<Cell<T>>(id)) {
Some(c) => c,
None => {
let c: Cell<T> = std::sync::Arc::new(std::sync::Mutex::new(Entry { value: None, at: None, running: false }));
ui.data_mut(|d| d.insert_temp(id, c.clone()));
c
}
};
let lock = |c: &Cell<T>| c.lock().unwrap_or_else(std::sync::PoisonError::into_inner).value.clone();
let start = {
let mut e = cell.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let due = !e.running && e.at.is_none_or(|at| now - at >= every || now < at);
if due {
e.running = true;
e.at = Some(now);
}
due
};
if start {
let (out, path, repaint) = (cell.clone(), path.to_string(), ui.ctx().clone());
let work = move || {
let v = f(&path);
let mut e = out.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
e.value = Some(v);
e.running = false;
drop(e);
repaint.request_repaint();
};
#[cfg(not(target_arch = "wasm32"))]
if std::thread::Builder::new().name("lc-fs-list".into()).spawn(work).is_err() {
cell.lock().unwrap_or_else(std::sync::PoisonError::into_inner).running = false;
}
#[cfg(target_arch = "wasm32")]
work();
}
lock(&cell)
}
/// A folder on disk with a disclosure triangle: click browses it, the triangle lists its
/// subfolders. In the tree that holds the folder being browsed (`reveal`), the folders on the
/// way down to it open whenever the browsed folder changes, so it shows highlighted in place.
@@ -0,0 +1,182 @@
//! The library couldn't be opened at launch (issue #100): never a silent in-memory session. A
//! blocking window says which library and why, and offers Try Again, Choose Another Library…,
//! Continue Without Saving and Quit. Continuing shows a banner under the top bar for as long as the
//! session is temporary.
use egui::{Align2, RichText, vec2};
use serde_json::json;
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
/// A library that failed to open (set by the host at launch).
#[derive(Clone, Debug, Default, PartialEq)]
pub struct LibraryProblem {
/// The library folder ("" when there was none to try, e.g. no home folder).
pub path: String,
/// Why it didn't open.
pub error: String,
/// Continue Without Saving was chosen: the banner shows instead of the window.
pub dismissed: bool,
/// Files from the command line, imported once a library opens.
pub pending_import: Vec<String>,
/// Try Again reopens `path` as a library folder (`false` in the browser, whose storage isn't
/// a folder: reloading the page tries again).
pub can_retry: bool,
}
impl LibraryProblem {
pub fn new(path: impl Into<String>, error: impl Into<String>) -> LibraryProblem {
LibraryProblem { path: path.into(), error: error.into(), can_retry: true, ..Default::default() }
}
/// `ui.inspect` → `libraryProblem`.
pub fn to_json(&self) -> serde_json::Value {
json!({"path": self.path, "error": self.error, "temporarySession": self.dismissed, "pendingImport": self.pending_import.len()})
}
}
/// What the user chose.
enum Choice {
Retry,
Choose,
Temporary,
Quit,
}
/// Open `path` (`None`: ask for a folder) through `app.openLibrary`; on success the problem is
/// over (and command-line files are imported), else it shows the new error.
fn open(app: &mut LightcraftApp, path: Option<String>) {
let params = match &path {
Some(p) => json!({"path": p}),
None => json!({}),
};
match app.run("app.openLibrary", params) {
Ok(serde_json::Value::Null) => {} // folder dialog cancelled
Ok(_) => {
let files = app.library_problem.take().map(|p| p.pending_import).unwrap_or_default();
if !files.is_empty() {
let _ = app.run("library.import", json!({"paths": files}));
}
}
Err(e) => {
if let Some(p) = app.library_problem.as_mut() {
if let Some(path) = path {
p.path = path;
}
p.error = e;
p.dismissed = false;
}
}
}
}
/// Clear the problem once a library is open (e.g. through Settings → Open Library…).
pub fn logic(app: &mut LightcraftApp) {
if app.library_problem.is_some() && app.session.library.is_some() {
app.library_problem = None;
}
}
/// The blocking window (until a choice is made).
pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
let Some(problem) = app.library_problem.clone() else { return };
if problem.dismissed {
return;
}
let t = Tokens::get(ctx);
let screen = ctx.content_rect();
egui::Area::new(egui::Id::new("library-problem-dim")).order(egui::Order::Middle).fixed_pos(screen.min).interactable(true).show(ctx, |ui| {
// swallows clicks: nothing behind the window can be used meanwhile
ui.allocate_rect(screen, egui::Sense::click_and_drag());
ui.painter().rect_filled(screen, 0.0, egui::Color32::from_black_alpha(160));
});
let mut choice = None;
let frame = egui::Frame::window(&ctx.global_style()).inner_margin(egui::Margin::symmetric(18, 14));
egui::Window::new("Your library couldn't be opened")
.id(egui::Id::new("library-problem"))
.order(egui::Order::Foreground)
.collapsible(false)
.resizable(false)
.frame(frame)
.anchor(Align2::CENTER_CENTER, [0.0, 0.0])
.default_width(460.0)
.show(ctx, |ui| {
// a fixed width: a wrapped label in an auto-sized, centred window would move it every frame
ui.set_width(440.0);
ui.spacing_mut().item_spacing.y = 8.0;
if problem.path.is_empty() {
ui.label(RichText::new("LightCraft couldn't find where to keep your library.").color(t.text));
} else {
ui.label(RichText::new("LightCraft couldn't open the library at").color(t.text_label));
ui.label(RichText::new(&problem.path).font(t.semibold(12.5)).color(t.text));
}
ui.add(egui::Label::new(RichText::new(&problem.error).color(t.caution)).wrap());
ui.label(
RichText::new(
"Your photos and edits in it are untouched. You can try again (e.g. after closing another program that has it open \
or reconnecting its drive), open a different library or create a new one, or continue without a library — \
then nothing you do is saved.",
)
.color(t.text_dim),
);
ui.add_space(4.0);
ui.horizontal(|ui| {
let mut button = |ui: &mut egui::Ui, id: &str, label: &str, enabled: bool, c: Choice| {
let r = ui.add_enabled(enabled, egui::Button::new(label).min_size(vec2(0.0, 26.0)));
register(ui.ctx(), format!("button:{id}"), r.rect);
if r.clicked() {
choice = Some(c);
}
};
button(ui, "libraryRetry", "Try Again", problem.can_retry && !problem.path.is_empty(), Choice::Retry);
button(ui, "libraryChoose", "Choose Another Library…", app.services.pick_folder.is_some(), Choice::Choose);
button(ui, "libraryTemporary", "Continue Without Saving", true, Choice::Temporary);
button(ui, "libraryQuit", "Quit", true, Choice::Quit);
});
});
match choice {
Some(Choice::Retry) => open(app, Some(problem.path)),
Some(Choice::Choose) => open(app, None),
Some(Choice::Temporary) => {
if let Some(p) = app.library_problem.as_mut() {
p.dismissed = true;
}
let files = app.library_problem.as_mut().map(|p| std::mem::take(&mut p.pending_import)).unwrap_or_default();
if !files.is_empty() {
let _ = app.run("library.import", json!({"paths": files}));
}
}
Some(Choice::Quit) => app.ui.quit = true,
None => {}
}
}
/// The banner under the top bar while the session is temporary (Continue Without Saving).
pub fn banner(app: &mut LightcraftApp, ui: &mut egui::Ui) {
if !app.library_problem.as_ref().is_some_and(|p| p.dismissed) {
return;
}
let t = Tokens::get(ui.ctx());
let mut reopen = false;
egui::Panel::top("library-problem-banner")
.exact_size(30.0)
.frame(egui::Frame::NONE.fill(t.caution).inner_margin(egui::Margin::symmetric(12, 0)))
.show(ui, |ui| {
ui.horizontal_centered(|ui| {
let r = ui.label(
RichText::new("Temporary session — your library isn't open, and nothing you do here is saved.")
.font(t.semibold(12.5))
.color(t.canvas),
);
register(ui.ctx(), "indicator:temporarySession", r.rect);
let b = ui.add(egui::Button::new(RichText::new("Open Library…").color(t.canvas)).fill(egui::Color32::from_black_alpha(40)));
register(ui.ctx(), "button:libraryReopen", b.rect);
reopen = b.clicked();
});
});
if reopen && let Some(p) = app.library_problem.as_mut() {
p.dismissed = false;
}
}
+6 -2
View File
@@ -10,7 +10,9 @@ pub mod edit;
pub mod filterbar;
pub mod grid;
pub mod left;
pub mod library_problem;
pub mod masking;
pub mod notices;
pub mod presets;
pub mod profiles;
pub mod right;
@@ -69,8 +71,10 @@ pub fn toast(app: &mut LightcraftApp, ctx: &egui::Context) {
let t = Tokens::get(ctx);
let fade = ((until - now) / 0.25).clamp(0.0, 1.0) as f32;
let painter = ctx.layer_painter(egui::LayerId::new(egui::Order::Foreground, egui::Id::new("toast")));
let galley = painter.layout_no_wrap(text, t.font(14.0), t.text.gamma_multiply(fade));
let size = vec2(galley.size().x + 40.0, 40.0);
// long messages (a save warning, the web build's notices) wrap within the canvas
let wrap = (canvas.width() - 80.0).clamp(200.0, 720.0);
let galley = painter.layout(text, t.font(14.0), t.text.gamma_multiply(fade), wrap);
let size = vec2(galley.size().x + 40.0, (galley.size().y + 22.0).max(40.0));
let r = Rect::from_center_size(pos2(canvas.center().x, canvas.bottom() - 60.0), size);
painter.rect_filled(r, 6.0, egui::Color32::from_black_alpha((200.0 * fade) as u8));
painter.galley(r.center() - galley.size() / 2.0, galley, t.text);
+115
View File
@@ -0,0 +1,115 @@
//! Things the user must not miss (issue #103): settings files that were damaged or unreadable when
//! the library opened (and host warnings such as a damaged `ui.json`), and quitting while changes
//! are only in memory.
use egui::{Align2, RichText, vec2};
use crate::LightcraftApp;
use crate::theme::Tokens;
use crate::widgets::register;
/// Collect the library's settings-file warnings (each is shown once).
pub fn logic(app: &mut LightcraftApp) {
let new = app.session.take_library_warnings();
app.notices.extend(new);
}
/// The window may close now (`true`), or the quit prompt is shown (`false`): changes that
/// couldn't be written are retried first — the log, then a snapshot of everything in memory.
pub fn may_close(app: &mut LightcraftApp) -> bool {
if app.quit_confirmed || app.session.unsaved().is_none() {
return true;
}
let _ = app.session.persist();
if app.session.unsaved().is_some()
&& let Err(e) = app.session.close_library()
{
log::error!("quit: library not saved: {e}");
}
match app.session.unsaved() {
None => true,
Some((n, e)) => {
app.quit_prompt = Some(format!(
"{} couldn't be written to disk: {e}\nIf you quit now, {} lost.",
if n == 1 { "1 change".to_string() } else { format!("{n} changes") },
if n == 1 { "it is" } else { "they are" }
));
false
}
}
}
enum Choice {
Retry,
QuitAnyway,
Cancel,
Ok,
}
fn window(ctx: &egui::Context, id: &str, title: &str, text: &str, buttons: &[(&str, &str, Choice)]) -> Option<usize> {
let t = Tokens::get(ctx);
let screen = ctx.content_rect();
egui::Area::new(egui::Id::new((id, "dim"))).order(egui::Order::Middle).fixed_pos(screen.min).show(ctx, |ui| {
ui.allocate_rect(screen, egui::Sense::click_and_drag());
ui.painter().rect_filled(screen, 0.0, egui::Color32::from_black_alpha(140));
});
let mut chosen = None;
let frame = egui::Frame::window(&ctx.global_style()).inner_margin(egui::Margin::symmetric(18, 14));
egui::Window::new(title)
.id(egui::Id::new(id))
.order(egui::Order::Foreground)
.collapsible(false)
.resizable(false)
.frame(frame)
.anchor(Align2::CENTER_CENTER, [0.0, 0.0])
.default_width(440.0)
.show(ctx, |ui| {
// a fixed width: a wrapped label in an auto-sized, centred window would move it every frame
ui.set_width(420.0);
ui.spacing_mut().item_spacing.y = 8.0;
ui.add(egui::Label::new(RichText::new(text).color(t.text_label)).wrap());
ui.add_space(4.0);
ui.horizontal(|ui| {
for (i, (bid, label, _)) in buttons.iter().enumerate() {
let r = ui.add(egui::Button::new(*label).min_size(vec2(0.0, 26.0)));
register(ui.ctx(), format!("button:{bid}"), r.rect);
if r.clicked() {
chosen = Some(i);
}
}
});
});
chosen
}
pub fn show(app: &mut LightcraftApp, ctx: &egui::Context) {
if let Some(text) = app.quit_prompt.clone() {
let buttons = [
("quitRetry", "Try Saving Again", Choice::Retry),
("quitAnyway", "Quit Anyway", Choice::QuitAnyway),
("quitCancel", "Cancel", Choice::Cancel),
];
let Some(i) = window(ctx, "quit-unsaved", "Quit with unsaved changes?", &text, &buttons) else { return };
match buttons[i].2 {
Choice::Retry => {
app.quit_prompt = None;
if may_close(app) {
app.ui.quit = true;
}
}
Choice::QuitAnyway => {
app.quit_prompt = None;
app.quit_confirmed = true;
app.ui.quit = true;
}
Choice::Cancel | Choice::Ok => app.quit_prompt = None,
}
return;
}
if let Some(text) = app.notices.first().cloned() {
let buttons = [("noticeOk", "OK", Choice::Ok)];
if window(ctx, "settings-notice", "Settings file problem", &text, &buttons).is_some() {
app.notices.remove(0);
}
}
}
+9 -2
View File
@@ -535,9 +535,16 @@ fn info(app: &mut LightcraftApp, ui: &mut egui::Ui, id: PhotoId) {
ui.label(egui::RichText::new(line).color(t.text_dim));
}
// offline originals / smart previews
// (cached answers, checked off the UI thread: unknown counts as online / no smart preview)
if let lightcraft_catalog::Source::File { path } = &p.source {
let online = std::path::Path::new(path).exists();
let smart = lightcraft_engine::cmd::previews::has_smart_preview(&app.session, p.id);
let avail = &app.session.media.availability;
let online = !avail.is_offline(path);
let smart = app
.session
.media
.smart_dir
.as_ref()
.is_some_and(|d| avail.exists(&d.join(lightcraft_engine::smart::file_name(&p)).to_string_lossy()) == Some(true));
if !online || smart {
let text = match (online, smart) {
(false, true) => "Original offline · editing the smart preview",
+4
View File
@@ -359,6 +359,9 @@ pub struct UiState {
/// Transient toast text and its expiry (seconds of app time).
#[serde(skip)]
pub toast: Option<(String, f64)>,
/// The result of the last Find Missing Photos (it searches in the background).
#[serde(skip)]
pub last_find_missing: Option<serde_json::Value>,
#[serde(skip)]
pub status: String,
#[serde(skip)]
@@ -589,6 +592,7 @@ impl Default for UiState {
compare: None,
reference: None,
toast: None,
last_find_missing: None,
status: String::new(),
dialog: None,
}
+112
View File
@@ -0,0 +1,112 @@
//! Background tasks for commands whose file-system work can take long on a slow or offline drive
//! (Find Missing Photos walking a folder, listing the auto-import folder…): the work runs on a
//! worker thread and its result is applied on the UI thread between frames, so the window keeps
//! answering. Imports, scans, exports and preview builds have their own tasks with progress.
use std::sync::mpsc::{Receiver, TryRecvError, channel};
use crate::LightcraftApp;
/// Applies a task's result to the app (on the UI thread).
type Finish = Box<dyn FnOnce(&mut LightcraftApp, &egui::Context) + Send>;
struct Task {
label: String,
rx: Receiver<Finish>,
}
/// The background tasks in flight.
#[derive(Default)]
pub struct Tasks {
running: Vec<Task>,
/// Repaints the window when a task finishes (set every frame).
pub(crate) repaint: Option<egui::Context>,
}
impl Tasks {
pub fn is_empty(&self) -> bool {
self.running.is_empty()
}
/// What is running (`ui.inspect` → `tasks`).
pub fn labels(&self) -> Vec<String> {
self.running.iter().map(|t| t.label.clone()).collect()
}
pub fn is_running(&self, label: &str) -> bool {
self.running.iter().any(|t| t.label == label)
}
}
/// Run `work` on a worker thread, then `done(app, ctx, result)` on the UI thread (in the browser
/// build, which has no threads, `work` runs at once and `done` on the next frame).
pub fn spawn<T: Send + 'static>(
app: &mut LightcraftApp,
label: &str,
work: impl FnOnce() -> T + Send + 'static,
done: impl FnOnce(&mut LightcraftApp, &egui::Context, T) + Send + 'static,
) -> Result<(), String> {
let (tx, rx) = channel::<Finish>();
let repaint = app.tasks.repaint.clone();
let job = move || {
let t = work();
let _ = tx.send(Box::new(move |app: &mut LightcraftApp, ctx: &egui::Context| done(app, ctx, t)));
if let Some(c) = repaint {
c.request_repaint();
}
};
#[cfg(not(target_arch = "wasm32"))]
std::thread::Builder::new().name(format!("lc-task-{label}")).spawn(job).map_err(|e| format!("{label}: could not start: {e}"))?;
#[cfg(target_arch = "wasm32")]
job();
app.tasks.running.push(Task { label: label.to_string(), rx });
Ok(())
}
/// Apply finished tasks (called every frame).
pub fn poll(app: &mut LightcraftApp, ctx: &egui::Context) {
app.tasks.repaint = Some(ctx.clone());
let mut i = 0;
while i < app.tasks.running.len() {
match app.tasks.running[i].rx.try_recv() {
Ok(finish) => {
app.tasks.running.remove(i);
finish(app, ctx);
}
Err(TryRecvError::Empty) => i += 1,
Err(TryRecvError::Disconnected) => {
// the worker died (a panic, already logged)
let t = app.tasks.running.remove(i);
app.toast(ctx, format!("{} failed", t.label));
}
}
}
if !app.tasks.is_empty() {
ctx.request_repaint_after(std::time::Duration::from_millis(100));
}
}
/// Wait for every task and apply it (tests, and commands asked to `wait`). `false` on timeout.
pub fn wait(app: &mut LightcraftApp, ctx: &egui::Context, timeout: std::time::Duration) -> bool {
#[cfg(not(target_arch = "wasm32"))]
{
let t0 = std::time::Instant::now();
loop {
poll(app, ctx);
if app.tasks.is_empty() {
return true;
}
if t0.elapsed() > timeout {
return false;
}
std::thread::sleep(std::time::Duration::from_millis(2));
}
}
#[cfg(target_arch = "wasm32")]
{
// tasks ran inline: their results are already waiting
let _ = timeout;
poll(app, ctx);
app.tasks.is_empty()
}
}
+101
View File
@@ -0,0 +1,101 @@
//! Issue #100: a library that can't be opened at launch is never replaced by a silent in-memory
//! demo session. The window says why and offers Try Again / Choose Another Library… / Continue
//! Without Saving / Quit; a temporary session shows a banner and never writes to the library.
use std::time::Duration;
use serde_json::json;
use crate::headless::Headless;
use crate::panels::library_problem::LibraryProblem;
use crate::{LightcraftApp, Services};
const T: Duration = Duration::from_secs(20);
fn has(h: &Headless, id: &str) -> bool {
h.app.widgets.iter().any(|(w, _)| w == id)
}
#[test]
fn unopenable_library_asks_instead_of_running_a_demo() {
let dir = std::env::temp_dir().join(format!("lc-ui-libproblem-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
let lib = dir.join("lib");
std::fs::create_dir_all(&lib).unwrap();
let img = lightcraft_raster::Rgba8 { width: 8, height: 8, data: vec![[90, 3, 9, 255]; 64] };
let png = lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(&img), &lightcraft_codecs::EncodeMeta::default()).unwrap();
let photo = dir.join("a.png");
std::fs::write(&photo, png).unwrap();
// another program has the library open: the launch fails like the desktop host's would
let held = lightcraft_catalog::LibraryLock::acquire(&lib, "another LightCraft").unwrap();
let mut session = lightcraft_engine::Session::new().with_fs();
let err = session.open_library(&lib, true).unwrap_err().to_string();
let png = |img: &lightcraft_raster::Rgba8| {
lightcraft_codecs::encode_png(&lightcraft_codecs::EncodeImage::rgba8(img), &lightcraft_codecs::EncodeMeta::default()).unwrap_or_default()
};
let mut app = LightcraftApp::new(
session,
Services {
png: Some(Box::new(png)),
write: Some(Box::new(|p: &str, b: &[u8]| std::fs::write(p, b).map_err(|e| e.to_string()))),
..Default::default()
},
);
app.library_problem =
Some(LibraryProblem { pending_import: vec![photo.to_string_lossy().to_string()], ..LibraryProblem::new(lib.to_string_lossy(), err) });
let mut h = Headless::new(app, [1300.0, 900.0], 1.0);
h.step();
h.step();
assert!(h.app.session.catalog.is_empty(), "no demo photos");
for b in ["button:libraryRetry", "button:libraryChoose", "button:libraryTemporary", "button:libraryQuit"] {
assert!(has(&h, b), "{b}");
}
if let Some(p) = std::env::var_os("LIGHTCRAFT_TEST_SHOTS") {
let r = h.request("ui.screenshot", json!({"path": std::path::Path::new(&p).join("dialog.png").to_string_lossy(), "headless": true}), T);
assert_eq!(r["ok"], true, "{r}");
}
let inspect = h.request("ui.inspect", json!({}), T);
let p = &inspect["result"]["libraryProblem"];
assert!(p["error"].as_str().unwrap().contains("already open in another LightCraft"), "{p}");
assert_eq!(p["temporarySession"], false);
// still locked: Try Again keeps the window, with the error
assert_eq!(h.request("ui.clickWidget", json!({"id": "button:libraryRetry"}), T)["ok"], true);
h.step();
assert!(h.app.session.library.is_none());
assert!(has(&h, "button:libraryRetry"));
// Continue Without Saving: a banner, the command-line photo in the temporary session, and
// nothing written into the library folder
let before: Vec<_> = std::fs::read_dir(&lib).unwrap().map(|e| e.unwrap().file_name()).collect();
assert_eq!(h.request("ui.clickWidget", json!({"id": "button:libraryTemporary"}), T)["ok"], true);
h.step();
h.step();
assert!(!has(&h, "button:libraryRetry"));
assert!(has(&h, "indicator:temporarySession"));
assert!(h.app.session.library.is_none());
assert_eq!(h.app.session.catalog.len(), 1, "the command-line photo is imported into the temporary session");
if let Some(p) = std::env::var_os("LIGHTCRAFT_TEST_SHOTS") {
let r = h.request("ui.screenshot", json!({"path": std::path::Path::new(&p).join("banner.png").to_string_lossy(), "headless": true}), T);
assert_eq!(r["ok"], true, "{r}");
}
h.request("engine.execute", json!({"command": "photo.rate", "params": {"rating": 3}}), T);
let after: Vec<_> = std::fs::read_dir(&lib).unwrap().map(|e| e.unwrap().file_name()).collect();
assert_eq!(before, after, "the temporary session never writes to the library");
// the other program quits; Open Library… → Try Again opens it, the banner goes away
drop(held);
assert_eq!(h.request("ui.clickWidget", json!({"id": "button:libraryReopen"}), T)["ok"], true);
h.step();
assert!(has(&h, "button:libraryRetry"));
assert_eq!(h.request("ui.clickWidget", json!({"id": "button:libraryRetry"}), T)["ok"], true);
h.step();
h.step();
assert!(h.app.session.library.is_some());
assert!(h.app.library_problem.is_none());
assert!(!has(&h, "indicator:temporarySession") && !has(&h, "button:libraryRetry"));
assert!(h.app.session.catalog.is_empty(), "an opened (new) library is never seeded with demo photos here");
drop(h);
let _ = std::fs::remove_dir_all(&dir);
}
+177
View File
@@ -0,0 +1,177 @@
//! Slow or offline drives never stall a frame (issue #104): with an injected file-system probe
//! that takes far longer than a frame (a sleeping NAS), the grid, the Info panel, the Missing
//! Photos view, the Rename dialog and an import keep drawing frames quickly, the probe never runs
//! on the UI thread, and the background answers still arrive.
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::thread::ThreadId;
use std::time::{Duration, Instant};
use lightcraft_catalog::{Op, Photo, PhotoId, Source};
use lightcraft_engine::{LibrarySource, Session};
use serde_json::json;
use crate::headless::Headless;
use crate::state::{RightPanel, ViewMode};
use crate::{LightcraftApp, Services};
/// What one file-system call takes on the "NAS".
const SLOW: Duration = Duration::from_millis(400);
/// A frame that waited for one such call would take at least this long.
const FRAME_LIMIT: Duration = Duration::from_millis(300);
const N: u64 = 12;
/// Threads the probe ran on, and how often.
#[derive(Clone, Default)]
struct Calls {
threads: Arc<Mutex<Vec<ThreadId>>>,
n: Arc<AtomicUsize>,
}
impl Calls {
fn record(&self) {
self.threads.lock().unwrap().push(std::thread::current().id());
self.n.fetch_add(1, Ordering::SeqCst);
}
fn on(&self, t: ThreadId) -> usize {
self.threads.lock().unwrap().iter().filter(|x| **x == t).count()
}
}
/// `N` photos whose originals are on an unreachable share, a smart previews folder, and a probe
/// that sleeps like a NAS waking up before saying "not there".
fn offline_library(calls: &Calls) -> Session {
let mut s = Session::new();
for i in 0..N {
let path = format!("/lightcraft-offline-nas/IMG_{i:03}.jpg");
let p = Photo::new(PhotoId(i + 1), Source::File { path }, &format!("IMG_{i:03}.jpg"), "JPEG", 600, 400, "2026-01-01T00:00:00");
s.catalog.apply(Op::AddPhoto { photo: Box::new(p) }).unwrap();
}
s.media.smart_dir = Some(std::env::temp_dir().join("lightcraft-offline-smart-previews-none"));
let c = calls.clone();
s.media.availability.set_probe(Arc::new(move |_| {
c.record();
std::thread::sleep(SLOW);
false
}));
s
}
fn max_frame(h: &mut Headless, frames: usize) -> Duration {
let mut worst = Duration::ZERO;
for _ in 0..frames {
let t0 = Instant::now();
h.step();
worst = worst.max(t0.elapsed());
}
worst
}
#[test]
fn offline_originals_never_block_a_frame() {
let calls = Calls::default();
let app = LightcraftApp::new(offline_library(&calls), Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
let ui_thread = std::thread::current().id();
h.app.ui.view = ViewMode::PhotoGrid;
h.step();
assert!(h.app.session.media.availability.is_background(), "the app checks files off the UI thread");
// the grid: every cell asks for its thumbnail (which asks whether the original is there)
h.app.session.execute("library.select", &json!({"ids": [1]})).unwrap();
let worst = max_frame(&mut h, 20);
assert!(worst < FRAME_LIMIT, "grid frame took {worst:?}");
// the Info panel of the active photo, in the loupe
h.app.ui.view = ViewMode::Detail;
h.app.ui.right = RightPanel::Info;
let worst = max_frame(&mut h, 20);
assert!(worst < FRAME_LIMIT, "Info panel frame took {worst:?}");
// the Missing Photos view fills in as the background checks find the files gone
h.app.ui.view = ViewMode::PhotoGrid;
h.app.ui.right = RightPanel::None;
h.app.session.execute("library.source", &json!({"kind": "missing"})).unwrap();
assert_eq!(h.app.session.source, LibrarySource::Missing);
let t0 = Instant::now();
let mut worst = Duration::ZERO;
while h.app.session.visible().len() < N as usize && t0.elapsed() < Duration::from_secs(60) {
let f = Instant::now();
h.step();
worst = worst.max(f.elapsed());
std::thread::sleep(Duration::from_millis(5));
}
assert_eq!(h.app.session.visible().len(), N as usize, "every photo found missing in the end");
assert!(worst < FRAME_LIMIT, "Missing Photos frame took {worst:?}");
// the Rename dialog's preview (it checks which names are taken on disk)
h.app.session.execute("library.select", &json!({"ids": [1, 2, 3]})).unwrap();
h.app.ui.dialog = Some(crate::state::Dialog::Rename { template: "Trip-{seq:3}".into(), start: 1 });
let worst = max_frame(&mut h, 20);
assert!(worst < FRAME_LIMIT, "Rename dialog frame took {worst:?}");
assert!(calls.n.load(Ordering::SeqCst) > 0, "the probe was asked");
assert_eq!(calls.on(ui_thread), 0, "the slow probe never ran on the UI thread");
}
/// An import whose files take long to read (probe: decoding headers and hashing on a slow drive)
/// runs on a worker thread: frames stay quick, the probe never runs on the UI thread, and Cancel
/// stops it with what was read so far added.
#[test]
fn import_reads_files_off_the_ui_thread_and_can_be_cancelled() {
let dir = std::env::temp_dir().join(format!("lc-ui-slow-import-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let n = 40;
for i in 0..n {
std::fs::write(dir.join(format!("IMG_{i:03}.jpg")), format!("not really a jpeg {i}")).unwrap();
}
let calls = Calls::default();
let mut s = Session::new();
let c = calls.clone();
s.media.file_probe = Some(Arc::new(move |p: &str| {
c.record();
std::thread::sleep(Duration::from_millis(150));
Ok(lightcraft_engine::media::ProbeInfo {
width: 60,
height: 40,
format: "JPEG".into(),
content_hash: Some(p.to_string()),
..Default::default()
})
}));
let app = LightcraftApp::new(s, Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
let ui_thread = std::thread::current().id();
h.step();
let undo0 = h.app.session.undo.len();
crate::import::start_paths(&mut h.app, vec![dir.to_string_lossy().to_string()]).unwrap();
// some batches arrive; meanwhile every frame is quick
let t0 = Instant::now();
let mut worst = Duration::ZERO;
while h.app.import.as_ref().is_some_and(|t| t.imported < 8) && t0.elapsed() < Duration::from_secs(60) {
let f = Instant::now();
h.step();
worst = worst.max(f.elapsed());
std::thread::sleep(Duration::from_millis(5));
}
assert!(worst < FRAME_LIMIT, "import frame took {worst:?}");
assert!(h.app.import.is_some(), "still importing");
let status = h.app.import.as_ref().map(crate::import::ImportTask::status).unwrap_or_default();
assert!(status["total"].as_u64().unwrap_or(0) >= 8, "{status}");
// Cancel: the files being read finish, nothing new starts
let r = h.request("ui.clickWidget", json!({"id": "button:importCancel"}), Duration::from_secs(10));
assert_eq!(r["ok"], true, "{r}");
let t0 = Instant::now();
while h.app.import.is_some() && t0.elapsed() < Duration::from_secs(60) {
h.step();
std::thread::sleep(Duration::from_millis(5));
}
assert!(h.app.import.is_none(), "stopped");
let added = h.app.session.catalog.len();
assert!((8..n).contains(&added), "cancelled part-way: {added} of {n} added");
assert_eq!(h.app.session.undo.len(), undo0 + 1, "one undo step");
assert_eq!(calls.on(ui_thread), 0, "files were never read on the UI thread");
let _ = std::fs::remove_dir_all(&dir);
}
+128
View File
@@ -0,0 +1,128 @@
//! Issue #103: quitting with changes that couldn't be saved asks first (after one more try), and
//! settings-file problems found when the library opened are shown, not swallowed.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use lightcraft_catalog::{MemStore, Store};
use lightcraft_engine::library::LibraryStores;
use serde_json::json;
use crate::headless::Headless;
use crate::{LightcraftApp, Services};
const T: Duration = Duration::from_secs(20);
/// Every write fails while `fail` is set (an unplugged drive).
#[derive(Clone, Default)]
struct Unplugged {
files: MemStore,
fail: Arc<AtomicBool>,
}
impl Unplugged {
fn check(&self) -> std::io::Result<()> {
if self.fail.load(Ordering::SeqCst) { Err(std::io::Error::other("drive unplugged")) } else { Ok(()) }
}
}
impl Store for Unplugged {
fn read(&mut self, name: &str) -> std::io::Result<Option<Vec<u8>>> {
self.files.read(name)
}
fn write_atomic(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.check()?;
self.files.write_atomic(name, data)
}
fn append(&mut self, name: &str, data: &[u8]) -> std::io::Result<()> {
self.check()?;
self.files.append(name, data)
}
fn truncate(&mut self, name: &str, len: u64) -> std::io::Result<()> {
self.files.truncate(name, len)
}
fn describe(&self) -> String {
"unplugged".into()
}
}
fn has(h: &Headless, id: &str) -> bool {
h.app.widgets.iter().any(|(w, _)| w == id)
}
/// Click a widget and let the synthetic events (move, press, release) play out.
fn click(h: &mut Headless, id: &str) {
// a window that just appeared is placed on its second frame
h.step();
h.step();
assert_eq!(h.request("ui.clickWidget", json!({"id": id}), T)["ok"], true, "{id}");
for _ in 0..4 {
h.step();
}
}
#[test]
fn quitting_with_unsaved_changes_asks_first() {
let store = Unplugged::default();
let mut s = lightcraft_engine::Session::new();
let stores = LibraryStores { dir: "unplugged".into(), catalog: Box::new(store.clone()), files: Box::new(MemStore::new()), on_disk: false };
s.open_library_in(stores, false).unwrap();
let app = LightcraftApp::new(s, Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
h.step();
// nothing unsaved: closing is fine
assert!(crate::panels::notices::may_close(&mut h.app));
store.fail.store(true, Ordering::SeqCst);
let r = h.request("engine.execute", json!({"command": "album.create", "params": {"name": "Kept"}}), T);
assert_eq!(r["ok"], false, "{r}");
assert!(!crate::panels::notices::may_close(&mut h.app), "retried, still failing: ask");
h.step();
let prompt = h.app.quit_prompt.clone().unwrap();
assert!(prompt.contains("1 change couldn't be written to disk") && prompt.contains("drive unplugged"), "{prompt}");
for b in ["button:quitRetry", "button:quitAnyway", "button:quitCancel"] {
assert!(has(&h, b), "{b}");
}
assert_eq!(h.request("ui.inspect", json!({}), T)["result"]["quitPrompt"], json!(prompt));
// Cancel: back to work, nothing quit
click(&mut h, "button:quitCancel");
assert!(h.app.quit_prompt.is_none() && !h.app.quit_confirmed);
// the drive is back: Try Saving Again saves and quits
assert!(!crate::panels::notices::may_close(&mut h.app));
h.step();
store.fail.store(false, Ordering::SeqCst);
click(&mut h, "button:quitRetry");
assert!(h.app.session.unsaved().is_none(), "saved");
assert!(h.app.quit_prompt.is_none() && !h.app.quit_confirmed);
// Quit Anyway is a deliberate choice
store.fail.store(true, Ordering::SeqCst);
let _ = h.request("engine.execute", json!({"command": "album.create", "params": {"name": "Lost"}}), T);
assert!(!crate::panels::notices::may_close(&mut h.app));
h.step();
click(&mut h, "button:quitAnyway");
assert!(h.app.quit_confirmed && crate::panels::notices::may_close(&mut h.app));
}
#[test]
fn damaged_settings_file_is_shown() {
let files = MemStore::new();
files.set("presets.json", b"{\"user\": [ {\"id\": ".to_vec());
let mut s = lightcraft_engine::Session::new();
let stores = LibraryStores { dir: "lib".into(), catalog: Box::new(MemStore::new()), files: Box::new(files.clone()), on_disk: false };
s.open_library_in(stores, false).unwrap();
let app = LightcraftApp::new(s, Services { png: None, ..Default::default() });
let mut h = Headless::new(app, [1200.0, 800.0], 1.0);
h.step();
h.step();
assert_eq!(h.app.notices.len(), 1);
assert!(h.app.notices[0].contains("presets.json is damaged"), "{:?}", h.app.notices);
assert!(has(&h, "button:noticeOk"));
assert!(files.files.lock().unwrap().keys().any(|k| k.starts_with("presets.json.corrupt-")));
click(&mut h, "button:noticeOk");
assert!(h.app.notices.is_empty(), "{:?}", h.app.notices);
h.step();
assert!(!has(&h, "button:noticeOk"));
}
+20 -2
View File
@@ -9,7 +9,16 @@
← {"id": 2, "ok": false, "error": "unknown command `nope`"}
```
Requests are answered on the UI thread between frames (timeout 60 s). The MCP server's connect
Requests are answered on the UI thread between frames (timeout 60 s).
**Only requests are read.** Every line must be a JSON object with a string `method` (`id` and `params` are
optional; blank lines are skipped). Anything else — text that isn't JSON, a JSON array or number, an object without
`method`, invalid UTF-8, or a line longer than 4 MiB — gets one error reply
(`{"ok": false, "error": "… closing the connection"}`) and the server **closes the connection**; nothing sent after
it on that connection runs. This keeps an HTTP request (for example a web page's cross-origin `fetch` to
`127.0.0.1:<port>`) from smuggling a command in its body: its request line is rejected first. Junk never reaches the
UI thread, and at most 16 connections are served at once (further ones get an error line and are closed). Clients
that hit an error reply should reconnect. The port has no authentication, so only enable it when you need it. The MCP server's connect
mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
`crates/ui-egui/src/control.rs` (methods) and `apps/lightcraft/src/control_server.rs` (transport).
@@ -21,7 +30,7 @@ mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
| `engine.commands` | — | Engine + UI commands: id, label, menu, shortcut, params doc, enabled |
| `ui.menu.list` | — | Menu entries (flat: id, label, menu path, shortcut, enabled) |
| `ui.menu.tree` | — | The menu bar as shown (File … Help): items `{id, params?, label, shortcut?, enabled, checked?}`, separators, submenus — the model behind the native macOS menu bar and the in-window menus |
| `ui.inspect` | — | UI state, window, canvas/image rects, `scroll: {grid, filmstrip}` (scroll offsets in points, `null` until drawn), active photo, selection, perf (`frameMs` = layout, `logicMs` = per-frame logic before it, `updateMs` = both, `maxUpdateMs`, `fps`, render queue …, `gpu` = adapter in use, `gpuReason` = why renders don't use the GPU, `gpuFallback` = latest render redone on the CPU and why — see `docs/gpu-pipeline.md`), status, memory (bytes per cache, see `library.memory`; plus stage caches and textures), `export: {running: {total, done, current} \| null, last}` |
| `ui.inspect` | — | UI state, window, canvas/image rects, `scroll: {grid, filmstrip}` (scroll offsets in points, `null` until drawn), active photo, selection, perf (`frameMs` = layout, `logicMs` = per-frame logic before it, `updateMs` = both, `maxUpdateMs`, `fps`, render queue …, `gpu` = adapter in use, `gpuReason` = why renders don't use the GPU, `gpuFallback` = latest render redone on the CPU and why — see `docs/gpu-pipeline.md`), status, memory (bytes per cache, see `library.memory`; plus stage caches and textures), `export: {running: {total, done, current} \| null, last}`, `notices` (warnings waiting to be shown, e.g. a damaged settings file; OK = `button:noticeOk`), `quitPrompt` (why quitting was stopped: unsaved changes; `button:quitRetry` / `button:quitAnyway` / `button:quitCancel`), `import: {done, total, imported, cancelled} \| null` (an import runs on a worker thread), `tasks` (other background work: `Find Missing Photos`, `Auto Import`) |
| `ui.widgets` | `{filter?}` | On-screen widgets `{id, rect: [x, y, w, h]}` (screen points) |
| `ui.clickWidget` / `ui.dragWidget` / `ui.hoverWidget` | `{id, count?, fx?, fy?}` / `{id, toX?, toY?, dx?, dy?, steps?}` / `{id, fx?, fy?}` | Real egui input on a widget (hover: the pointer rests on it, e.g. for preset/profile previews) |
| `ui.move` / `ui.click` / `ui.drag` | `{x, y, count?, button?}` / `{x, y, toX, toY, steps?}` | Raw pointer input, screen points |
@@ -49,6 +58,15 @@ shows a warning (widget `indicator:unsaved`). Queries and commands that change n
compaction (snapshot) is not a failed command — the log is kept whole — and only shows in `library.info` →
`lastError`.
### When the library can't be opened
If the desktop app can't open its library at launch (another program has it open, an unreadable or newer-format
catalog, a missing drive), the session starts empty and in memory — never with demo photos — and a window asks what
to do: `ui.inspect` → `libraryProblem` is `{path, error, temporarySession, pendingImport}` (else `null`); its buttons
are `button:libraryRetry`, `button:libraryChoose`, `button:libraryTemporary` (Continue Without Saving) and
`button:libraryQuit`. A temporary session shows a banner (`indicator:temporarySession`, `button:libraryReopen`) and
writes nothing. `app.openLibrary` opening a library ends it.
## Headless rendering (no window, no GPU)
The egui UI can be rasterized on the CPU (`crates/ui-egui/src/softpaint.rs`, driven by
+4 -1
View File
@@ -15,7 +15,10 @@ It runs in one of two modes:
Options: `--library DIR` opens (or creates) a persistent LightCraft library — the same crash-safe
format the desktop app uses (`~/Pictures/LightCraft Library` by default there) — so ratings, edits and albums
survive between sessions (with `--demo`, a new library is seeded with the demo photos);
survive between sessions (with `--demo`, a new library is seeded with the demo photos). A library
is open in one program at a time (`catalog.lock` in the library folder): while the desktop app has
it open, `--library` on the same folder fails with "This library is already open in LightCraft
(process N …)" — use connect mode to work with the running app instead;
`--demo` starts the headless session with the procedurally generated demo library;
`--compact` lists only the helper tools (see below). In connect mode the server starts even when
the app is not running yet and connects on the first call (and reconnects if the app restarts).
+13 -11
View File
@@ -44,13 +44,13 @@ and prints the summary; `cargo xtask parity --write` refreshes the summary table
| U. Map & location (MAP) | 0 | 1 | 1 | 0 | — | 0/1 (0%) |
| V. Preferences (PREF) | 5 | 0 | 3 | 3 | 1/1 (100%) | 4/4 (100%) |
| W. Cloud & AI infrastructure (CLOUD / AI) | 1 | 0 | 1 | 2 | — | — |
| X. Cross-cutting behaviours (BEHAV) | 15 | 2 | 2 | 1 | 8/8 (100%) | 6/7 (86%) |
| X. Cross-cutting behaviours (BEHAV) | 15 | 3 | 2 | 1 | 8/8 (100%) | 6/8 (75%) |
| Y. Menus | 81 | 0 | 4 | 8 | 47/47 (100%) | 23/23 (100%) |
| Z. Keyboard shortcuts (desktop) | 74 | 3 | 3 | 1 | 49/52 (94%) | 22/23 (96%) |
| Lightroom Classic extras | 30 | 9 | 40 | 9 | — | 20/21 (95%) |
| **Total** | 386 | 28 | 91 | 37 | 194/200 (97%) | 138/147 (94%) |
| Lightroom Classic extras | 31 | 9 | 40 | 9 | — | 21/22 (95%) |
| **Total** | 387 | 29 | 91 | 37 | 194/200 (97%) | 139/149 (93%) |
Weighted completion (✅ = 1, 🟡 = ½, 🚫 left out): **79.2%** of 505 in-scope rows — P0 98.5% of 200 · P1 95.9% of 147 · P2 39.2% of 158.
Weighted completion (✅ = 1, 🟡 = ½, 🚫 left out): **79.2%** of 507 in-scope rows — P0 98.5% of 200 · P1 95.6% of 149 · P2 39.2% of 158.
<!-- /parity:summary -->
## Top gaps
@@ -113,7 +113,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-IMP-ADD-DIALOG | Add photos/folders | P0 | ✅ | `cmd:file.addPhotos`, `cmd:library.importPreview`, `cmd:library.import` (`mode`, `album`, `albumName`, `preset`, `keywords`), `crates/ui-egui/src/import.rs` | review dialog (Import Photos): the scanned source (a scanned folder is not added to Local), candidate grid with thumbnails and checkboxes (duplicates by path/content marked and unchecked), transfer mode (add in place / copy / move, each explained), album (existing/new), preset, keywords; the folder scan runs on a background thread with a progress window and Cancel (a NAS folder no longer freezes the window); batched import with a progress window; one undo step. Drag-and-drop still imports directly |
| LR-IMP-ADD-DIALOG | Add photos/folders | P0 | ✅ | `cmd:file.addPhotos`, `cmd:library.importPreview`, `cmd:library.import` (`mode`, `album`, `albumName`, `preset`, `keywords`), `crates/ui-egui/src/import.rs` | review dialog (Import Photos): the scanned source (a scanned folder is not added to Local), candidate grid with thumbnails and checkboxes (duplicates by path/content marked and unchecked), transfer mode (add in place / copy / move, each explained), album (existing/new), preset, keywords; the folder scan runs on a background thread with a progress window and Cancel (a NAS folder no longer freezes the window); the import itself (probing, sidecars, copying / moving files) runs on a worker thread and its batches join the library between frames, with a progress window and Cancel; one undo step. Drag-and-drop goes through the same background import |
| LR-IMP-DRAGDROP | Drop files/folders to import | P0 | ✅ | `crates/ui-egui/src/lib.rs` (dropped files → `cmd:library.import`) | dropping onto a specific album not supported |
| LR-IMP-DUPES | Skip duplicates by content | P1 | ✅ | `crates/engine/src/import.rs`, `crates/engine/src/tests_import.rs` | |
| LR-IMP-DEVICE | Import from camera/card | P1 | ✅ | `cmd:library.devices`, `cmd:file.addFromDevice`, `crates/engine/src/devices.rs` | mounted volumes with a DCIM folder (macOS /Volumes, Linux /media and /run/media, Windows drive letters); File → Import from Device → the import review, copying into the library by default; menus serve the last scan and rescan in the background (never blocking a frame; hot-plugs repaint); copies are verified like Move's (a new file, synced, compared byte for byte; a bad copy is removed and reported as failed, never counted as imported; taken names get -1, -2…, never replaced — issue #96, `crates/engine/src/import_move.rs`); no PTP/MTP (cameras that don't mount as a disk) |
@@ -210,7 +210,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-META-COPYRIGHT-DEFAULT | Default copyright on import | P1 | ✅ | `cmd:library.preferences` (`import.copyright`, `import.creator`), `crates/ui-egui/src/panels/settings.rs` | Settings → Import → Metadata; fills only photos without their own |
| LR-META-LOCATION | Location editing | P2 | 🟡 | `cmd:photo.setMeta` (`location`, `city`, `state`, `country`, `gps`) | place fields and GPS (decimal or degrees / minutes / seconds) edited in Info, read/written as IPTC Core / Photoshop / EXIF XMP; Show on Map opens OpenStreetMap; no embedded map, no geocoding |
| LR-META-COPYPASTE | Copy / paste metadata | P2 | ✅ | `cmd:photo.copyMetadata`, `cmd:photo.pasteMetadata` | title, caption, alt text, extended description, copyright (notice, status, usage terms, info URL), creator, place fields, keywords; `fields` picks a subset |
| LR-META-XMP | XMP read/write | P0 | ✅ | `cmd:photo.saveMetadataToFile`, `cmd:photo.readMetadataFromFile`, `cmd:library.xmpPreferences`, `crates/engine/src/sidecar.rs`, `docs/xmp-interop.md` | sidecar wins for metadata; a sidecar capture time (`exif:DateTimeOriginal` / `photoshop:DateCreated` / `xmp:CreateDate`) fills in only when the file has none |
| LR-META-XMP | XMP read/write | P0 | ✅ | `cmd:photo.saveMetadataToFile`, `cmd:photo.readMetadataFromFile`, `cmd:library.xmpPreferences`, `crates/engine/src/sidecar.rs`, `docs/xmp-interop.md` | sidecar wins for metadata; a sidecar capture time (`exif:DateTimeOriginal` / `photoshop:DateCreated` / `xmp:CreateDate`) fills in only when the file has none; saving merges into an existing sidecar (`crates/meta/src/xmp_merge.rs`): another app's `crs:`, `xmpMM:History` and unknown namespaces are kept byte for byte, an unreadable sidecar is backed up first; with stem naming, files sharing a stem (raw + JPEG) get separate sidecars (`<file>.xmp` for all but the raw) |
| LR-META-EXIF-FULL | Full EXIF/IPTC [Classic] | P1 | ✅ | `cmd:photo.allMetadata`, `cmd:dialog.allMetadata`, `crates/meta/src/tags.rs`, `crates/meta/src/exif.rs`, `crates/meta/src/iptc.rs` | read and written on export; Info ▸ All Metadata…: every TIFF / EXIF / GPS / interop tag (named, common values spelled out) and the XMP fields, searchable; metadata presets |
## F. Edit panel — global adjustments (EDIT)
@@ -463,7 +463,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-PREF-GENERAL | General settings | P0 | ✅ | `cmd:app.settings`, `cmd:app.openLibrary`, `crates/ui-egui/src/panels/settings.rs` | General / Import / Performance / Interface tabs; app settings in ui.json, library settings in prefs.json |
| LR-PREF-GENERAL | General settings | P0 | ✅ | `cmd:app.settings`, `cmd:app.openLibrary`, `crates/ui-egui/src/panels/settings.rs`, `crates/ui-egui/src/panels/notices.rs` | General / Import / Performance / Interface tabs; app settings in ui.json (written atomically, saved when the library changes and every few seconds), library settings in prefs.json; a damaged settings file is kept as `.corrupt-<time>` and reported, an unreadable one is not overwritten that session (`library.info` → `settingsWarnings`); quitting with unsaved changes retries, then asks |
| LR-PREF-LOCALSTORAGE | Storage & cache | P1 | ✅ | `cmd:library.preferences`, `cmd:library.clearPreviews`, `cmd:library.compact` | thumbnail cache size + clear in Settings → Performance; library location + Open Library… in General |
| LR-PREF-ACCOUNT | Account | OOS | 🚫 | | |
| LR-PREF-INTERFACE | Interface options | P1 | ✅ | `cmd:app.settings` | filmstrip names/badges, grid badges (auto/always/never), square-grid names, navigator, info overlay |
@@ -480,7 +480,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-CLOUD-SYNC | Cloud sync | OOS | 🚫 | | |
| LR-CLOUD-SMARTPREVIEW | Editable proxies | P2 | ✅ | `cmd:library.smartPreviews`, `cmd:library.smartPreviewsLocation`, `cmd:photo.smartPreview`, `crates/engine/src/smart.rs`, `crates/ui-egui/src/panels/settings.rs` | File ▸ Previews ▸ Build / Discard Smart Previews: ~1 MB proxies in the library; with the original offline the photo renders, edits and exports (at proxy size) from its proxy; Info shows the status; the proxy folder is chosen per library (Settings → Performance → Smart previews, saved in the library): the effective path, count and size are shown, changing it needs a choice for the proxies already built (move / leave / delete), and an unavailable or unwritable folder is an error, never a fallback to the library drive; proxies are written atomically and a damaged one (cut short by a crash or a full drive) doesn't count as built, is rebuilt (`repaired`) and is never moved over a good one (issue #106). The thumbnail cache stays in the library |
| LR-CLOUD-SMARTPREVIEW | Editable proxies | P2 | ✅ | `cmd:library.smartPreviews`, `cmd:library.smartPreviewsLocation`, `cmd:photo.smartPreview`, `crates/engine/src/smart.rs`, `crates/ui-egui/src/panels/settings.rs` | File ▸ Previews ▸ Build / Discard Smart Previews (on a worker thread, progress and Stop like Build Previews): ~1 MB proxies in the library; with the original offline the photo renders, edits and exports (at proxy size) from its proxy; Info shows the status; the proxy folder is chosen per library (Settings → Performance → Smart previews, saved in the library): the effective path, count and size are shown, changing it needs a choice for the proxies already built (move / leave / delete), and an unavailable or unwritable folder is an error, never a fallback to the library drive; proxies are written atomically and a damaged one (cut short by a crash or a full drive) doesn't count as built, is rebuilt (`repaired`) and is never moved over a good one (issue #106). The thumbnail cache stays in the library |
| LR-AI-UPDATE-INDICATOR | AI-settings update indicator | P2 | ⬜ | | |
| LR-AI-CREDITS | Generative credits | OOS | 🚫 | | |
@@ -488,7 +488,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| Id | Feature | Tier | Status | Evidence | Missing / notes |
|---|---|---|---|---|---|
| LR-BEHAV-AUTOSAVE | Instant autosave | P0 | ✅ | `crates/catalog/src/journal.rs`, `crates/engine/src/library.rs` | every change fsynced before the command returns; a failed write fails the command ("saved in memory but not written to disk"), keeps the change queued and retries; top-bar warning until saved |
| LR-BEHAV-AUTOSAVE | Instant autosave | P0 | ✅ | `crates/catalog/src/journal.rs`, `crates/engine/src/library.rs` | every change fsynced before the command returns; a failed write fails the command ("saved in memory but not written to disk"), keeps the change queued and retries; top-bar warning until saved; a write that fails part-way is cut back to the last whole record so the retry never reads back as damage (logs from older builds with such a fragment load in full); quitting saves queued changes in the closing snapshot even when the log can't be appended to; versioned catalog format (`journal::VERSION`): older libraries are upgraded on open, a library from a newer LightCraft is refused untouched ("written by a newer version") |
| LR-BEHAV-UNDO | Global undo | P0 | ✅ | `cmd:edit.undo`, `crates/engine/src/tests.rs` (`rating_flag_undo_redo`) | covers ratings, albums, deletes, edits |
| LR-BEHAV-MULTISELECT | Multi-selection | P0 | ✅ | `cmd:library.select` (`replace`/`add`/`toggle`/`range`), `cmd:library.selectAll` | |
| LR-BEHAV-BATCH | Batch apply to selection | P0 | ✅ | `cmd:photo.rate`, `cmd:develop.paste`, `cmd:preset.apply`, `cmd:app.export` | |
@@ -496,6 +496,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
| LR-BEHAV-PROGRESSIVE | Progressive rendering | P0 | ✅ | `crates/engine/src/media.rs`, `crates/preview/src/lib.rs` | |
| LR-BEHAV-BG-TASKS | Background tasks | P0 | ✅ | `crates/preview/src/lib.rs` (`JobPool`), `crates/ui-egui/src/export_task.rs`, `crates/ui-egui/src/import.rs` | renders off the UI thread; exports started from the UI run on a worker thread with a progress panel and Cancel (`ui.inspect` → `export`); imports show a progress window. No combined activity centre |
| LR-BEHAV-OFFLINE | Offline editing | P1 | ✅ | | local-first: everything works offline |
| LR-BEHAV-WEB-SAFETY | Library safety in the browser build | P1 | 🟡 | `cmd:file.backupLibrary`, `cmd:file.restoreLibrary`, `apps/lightcraft-web/src/safety.rs`, `apps/lightcraft-web/src/backup.rs`, `apps/lightcraft-web/src/files.rs` | web only (experimental): Back Up Library (zip of the catalog and every stored original) and a non-destructive Restore; failed saves show as unsaved and are retried; a photo that can't be stored isn't added; one tab per library (Web Locks); notices for storage that isn't persistent and for a library that can't open; `?reset` asks first; a panic shows a message. Missing: `?safe` start, deleting originals of removed photos, zip64 (backups > 4 GB) |
| LR-BEHAV-GPU | GPU acceleration | P0 | ✅ | `crates/gpu/src/render.rs`, `cmd:app.gpu`, `docs/gpu-pipeline.md` | CPU fallback per render on device limits, out of memory, device errors / loss, incomplete or black results (`perf.gpuReason`, `perf.gpuFallback`); short submissions for slow iGPUs; no tiling beyond the buffer limit (CPU) |
| LR-BEHAV-RENDER-FIDELITY | Rendering matches Lightroom | P1 | 🟡 | `crates/pipeline/src/lib.rs`, `crates/pipeline/src/tone.rs` | every slider exists and works, but the character of the result (tone curve shape, highlight recovery, texture / clarity / dehaze, noise reduction, sharpening, default look) is tuned by eye; there is no systematic side-by-side comparison against Lightroom on the same CC0 raws. Needs a fidelity suite: Lightroom reference renders kept only in the local `plan/` (never committed), compared per slider and preset with a perceptual metric |
| LR-BEHAV-DRAGDROP | Drag and drop | P1 | ✅ | `crates/ui-egui/src/lib.rs`, `crates/ui-egui/src/panels/grid.rs` (`drag_feedback`), `crates/ui-egui/src/panels/left.rs` (`drop_target`) | files → app (import); grid photos → an album row (adds the selection, with a count badge while dragging) |
@@ -713,7 +714,8 @@ From `08-lightroom-classic-extras.md` (Classic-only features) and part 2 of `06-
| LRC-LIB-SURVEY | Survey view | P2 | ✅ | `cmd:view.survey` | N |
| LRC-LIB-REFVIEW | Reference view | P2 | ✅ | `cmd:view.reference`, `cmd:photo.setReference`, `crates/ui-egui/src/panels/compare.rs` (`show_reference`) | View ▸ Reference View (⇧R): the reference photo beside the active one (edited with the Edit panel); photo menu ▸ Set as Reference Photo |
| LRC-LIB-CATALOG-PANEL | Catalog sets | P1 | ✅ | `cmd:library.source` (`missing`) | all, recently added, picks, recently deleted, missing photos (when any) |
| LRC-LIB-FOLDERS | Disk folder tree | P1 | ✅ | `cmd:library.browse`, `cmd:folder.rename`, `cmd:folder.move`, `crates/ui-egui/src/panels/left.rs` (`local_section`, `folder_tree`) | standard places + the browsed folder + Browse Folder…, each an expandable tree of subfolders, breadcrumb navigation; context menu: Rename Folder…, Move Folder To…, Show in Finder (on disk, photos relinked; like the file manager, not an undo step) |
| LRC-LIB-CATALOG-LOCK | Catalog open in one program at a time | P1 | ✅ | `cmd:app.openLibrary`, `crates/catalog/src/lock.rs`, `crates/engine/src/library.rs` (`open_library`) | exclusive OS lock on `catalog.lock` for the session (app and `lightcraft-cli --library`); a second opener is refused with who has it (`catalog.lock.owner`: program, process, computer); released by the OS on a crash, so never stale; file systems without locks open unprotected with a warning; no read-only mode; the browser build (OPFS) has no guard across tabs yet. A library that can't be opened at launch (locked, unreadable, newer format, drive missing) shows a window — Try Again, Choose Another Library…, Continue Without Saving (then a banner; nothing written), Quit — instead of a silent demo session (`crates/ui-egui/src/panels/library_problem.rs`) |
| LRC-LIB-FOLDERS | Disk folder tree | P1 | ✅ | `cmd:library.browse`, `cmd:folder.rename`, `cmd:folder.move`, `crates/ui-egui/src/panels/left.rs` (`local_section`, `folder_tree`) | standard places + the browsed folder + Browse Folder…, each an expandable tree of subfolders, breadcrumb navigation; context menu: Rename Folder…, Move Folder To…, Show in Finder (on disk, photos relinked; one undo step each: undo / redo move the folder back and forth, refused when the destination is taken) |
| LRC-LIB-COLLECTIONS | Collections & sets | P1 | ✅ | `cmd:album.create`, `cmd:album.createSmart`, `cmd:album.toggleTarget`, `cmd:album.setTarget`, `cmd:album.clearQuick` | albums + folders (sets), smart albums (rule editor), Quick Collection and target album (B in the grids adds / removes the selection; album menu ▸ Set as Target Album; the target is marked +) |
| LRC-LIB-SMARTCOLL | Smart-collection rules | P1 | ✅ | `cmd:album.createSmart`, `cmd:album.setRules`, `cmd:album.ruleFields`, `cmd:dialog.smartAlbum`, `crates/catalog/src/rules.rs`, `crates/ui-egui/src/panels/rules_editor.rs` | rule editor: match all / any / none, nested groups (⌥+ or + Group), 26 fields (rating, flag, label, type, edits, keywords, any text, filename, format, title, caption, camera, lens, location, creator, copyright, copyright status, capture / import / edit date, ISO, aperture, focal length, megapixels, GPS, virtual copy, album) with text / number / date / in-the-last operators; live match count; also usable as a library filter (`ruleSet`) |
| LRC-LIB-PUBLISH | Publish services | P2 | ⬜ | | |
@@ -727,7 +729,7 @@ From `08-lightroom-classic-extras.md` (Classic-only features) and part 2 of `06-
| LRC-LIB-PEOPLE | People view | P2 | ⬜ | | |
| LRC-LIB-COMMENTS | Comments panel | P2 | ⬜ | | |
| LRC-LIB-VISUALSEARCH | Find similar photos | P2 | ✅ | `cmd:library.findSimilar`, `crates/pipeline/src/cull.rs` (`signature`) | photo menu / Photo ▸ Find Similar Photos: look-alike photos (composition and tones), most similar first, as a filter (`only`) |
| LRC-LIB-MISSING | Missing files & relink | P1 | ✅ | `cmd:library.missing`, `cmd:photo.relink`, `cmd:library.findMissing`, `cmd:file.findMissing`, `cmd:photo.locate`, `crates/engine/src/cmd/missing.rs` (`checked_path`) | File → Find Missing Photos… (anywhere in a folder: same name, size and content hash; renamed files by size + content hash; same name and size alone only when unambiguous — look-alikes are skipped and reported; one undo step), photo menu → Locate Missing File…; unreadable files show "!" in the grid and a reason in the loupe; undo never moves files. Missing Photos in the sidebar (counted on a worker thread); it covers library photos only — Local browse records are never checked, for the count, the view, `library.missing` and Find Missing alike |
| LRC-LIB-MISSING | Missing files & relink | P1 | ✅ | `cmd:library.missing`, `cmd:photo.relink`, `cmd:library.findMissing`, `cmd:file.findMissing`, `cmd:photo.locate`, `crates/engine/src/cmd/missing.rs` (`checked_path`) | File → Find Missing Photos… (anywhere in a folder: same name, size and content hash; renamed files by size + content hash; same name and size alone only when unambiguous — look-alikes are skipped and reported; searched on a worker thread, one undo step), photo menu → Locate Missing File…; unreadable files show "!" in the grid and a reason in the loupe; undo never moves files. Missing Photos in the sidebar (counted on a worker thread; the view, the Info panel, the grid's context menu and thumbnails read file availability from a cache a worker thread fills, so an offline NAS never stalls a frame); it covers library photos only — Local browse records are never checked, for the count, the view, `library.missing` and Find Missing alike |
| LRC-LIB-CONVERT | Convert to DNG | P2 | ✅ | `cmd:photo.convertToDng`, `crates/engine/src/cmd/convert.rs` | Photo ▸ Convert to DNG: lossless DNG next to the raw with the settings embedded, photo (and its virtual copies) relinked; originals kept; undoable; the DNG is decoded and compared with the raw data, written to a temp file, synced, read back and only then given a free name (never replacing a file) — a failed write leaves no DNG and the photo on its raw (issue #106) |
| LRC-LIB-PREVIEWS | Build / discard previews | P1 | ✅ | `cmd:library.buildPreviews`, `cmd:library.previewProgress`, `cmd:library.cancelPreviews`, `cmd:library.clearPreviews`, `crates/engine/src/cmd/previews.rs`, `crates/preview/src/lib.rs` | File ▸ Previews: build standard-sized (Settings → Performance size) or 1:1 previews of the selected / visible photos in the background (progress toasts, stop), discard the cache; disk thumbnail + view cache; no smart previews (offline proxies) |
| LRC-LIB-SLIDESHOW-IMPROMPTU | Impromptu slideshow | P2 | ✅ | `cmd:view.slideshow` | see LR-VIEW-SLIDESHOW |
+44 -8
View File
@@ -54,7 +54,8 @@ gzip size (5.25 MB). `wasm-opt -Oz`, when installed, shrinks it further.
## Deploying: headers
`cargo xtask web --serve` sends these on every response, and a production server should too:
`cargo xtask web --serve` sends these on every response. A production server may send them too;
today they are optional:
```
Cross-Origin-Opener-Policy: same-origin
@@ -62,11 +63,13 @@ Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Resource-Policy: same-origin
```
COOP + COEP make the page *cross-origin isolated* (`crossOriginIsolated === true`). That is
COOP + COEP make the page *cross-origin isolated* (`crossOriginIsolated === true`). That will be
**required** for `SharedArrayBuffer`, i.e. for a future wasm-threads build (`+atomics`, nightly
`build-std`), and gives full-resolution `performance.now()`. The current render workers (below)
don't share memory, so the app also runs without them; with COEP on, every subresource must be
same-origin or send CORP/CORS headers (the bundle has no third-party resources).
don't share memory, so the app runs without them; with COEP on, every subresource must be
same-origin or send CORP/CORS headers (the bundle has no third-party resources). The file names
are the same in every version, so serve them with `Cache-Control: no-cache` (not `immutable`);
`packaging/web/README.md` (shipped as `HOSTING.md`) has the hosting details.
## What works
@@ -80,16 +83,43 @@ same-origin or send CORP/CORS headers (the bundle has no third-party resources).
the background within a frame or two, each file replaced atomically, in modification order
(`apps/lightcraft-web/src/files.rs`). View state and UI prefs are saved every second when they
change (a tab can close without notice).
- Known limitation: the browser storage has no file locks, so two tabs of the same origin open
the same library and the last one to write a snapshot wins (the desktop app and the CLI lock
a library folder: `catalog.lock`, issue #99). Use one tab at a time; a guard through the Web
Locks API (`navigator.locks`) is still to do.
- `originals/<content hash>`: the bytes of every imported photo. The catalog refers to them as
`web/<hash>/<file name>`; the main thread keeps recently used originals in memory (≤ 768 MB)
and loads the rest on demand (the active photo is prefetched).
- `thumbs/<key>.jpg` + `thumbs/index.json`: the rendered-thumbnail cache, with the desktop's
budget (2 GB, least recently used pruned to 80 %).
A new library starts with the procedural demo photos. The app asks for persistent storage
(`navigator.storage.persist()`); without it the browser may evict the data under storage
pressure. URL options: `?store=idb` forces IndexedDB, `?store=memory` keeps nothing, `?reset`
deletes the stored library first.
A new library starts with the procedural demo photos. URL options: `?store=idb` forces
IndexedDB, `?store=memory` keeps nothing, `?reset` deletes the stored library first (after a
confirmation: it deletes every imported photo too).
- **Keeping the library safe** (experimental: the library lives only in this browser):
- The app asks for persistent storage (`navigator.storage.persist()`); when the browser doesn't
grant it, a notice says the library may be evicted under storage pressure.
- **File ▸ Back Up Library…** downloads a zip of the library files and every stored original
(`originals/<hash>/<file name>`, so an unzipped backup is browsable; entries are stored, not
compressed; at most 4 GB). **File ▸ Restore Library from Backup…** reads such a zip into a
new library folder (`library-restored-<time>/`; originals already stored are skipped, every
entry's checksum is verified) and only then switches to it (`active-library` names the
folder in use) and reloads: the previous library stays in storage. Both are web-only
(`file.backupLibrary`, `file.restoreLibrary`; the desktop library is a folder).
- A failed save (quota exceeded, storage cleared) is not silent: the catalog then refuses new
writes, so commands report `saved in memory but not written`, the top bar shows the unsaved
warning (as on the desktop, see `docs/control-protocol.md`), and saving is retried every
2 s until it works (`web.stats` → `saveError`).
- A picked or dropped photo whose bytes can't be stored is not added (it would be gone after
a reload); a notice says why.
- One tab at a time: the page holds a Web Lock (`navigator.locks`, `lightcraft-library`); a
second tab or window shows "LightCraft is already open in another tab" instead of loading
its own copy (two copies would overwrite each other's saves). Browsers without Web Locks
aren't protected.
- If the stored library can't be opened, a notice says the session is temporary and nothing
is saved; the stored library is left as it was.
- A panic (wasm is built with `panic = "abort"`) replaces the page with a message saying the
stored library is kept and to reload, instead of a frozen canvas.
- **Importing photos with no filesystem.** *File ▸ Import Photos…* (<kbd>⌘⇧I</kbd>) opens the
browser's file picker. You can also drop files anywhere on the page. The bytes are written to
storage, then imported and decoded by the same engine code as the desktop app
@@ -119,6 +149,12 @@ same-origin or send CORP/CORS headers (the bundle has no third-party resources).
yet fails with "still loading" and works a moment later.
- **Durability is "a frame later", not fsync-before-return:** a crash or tab kill in the few
milliseconds between a command and its flush loses that command.
- **Data-safety gaps still open** (issue #107): a panic in a decoder still ends the page (and a
photo that panics while its thumbnail renders can do so again after a reload; there is no
`?safe` start that skips rendering yet); removing photos doesn't delete their stored originals
(`originals/<hash>`; they are kept, and included in backups); restored-over libraries stay in
storage until the site's data is cleared; backups larger than 4 GB need zip64, which isn't
written yet.
- **Preset files** (import/export `.json`) aren't wired to browser pickers yet.
- **Control channel / MCP.** These are desktop-only, because they need a TCP socket.
- **AVIF export** is untested in the browser; it's the one encoder that may not be wasm-safe.
+42
View File
@@ -26,6 +26,32 @@
compared byte for byte with the card. A copy that fails or differs is removed and reported as a failed import —
so "import complete" means the copies are good before you format the card — and a name that is taken gets -1,
-2… instead of being replaced.
- Saving metadata to an XMP sidecar another application wrote no longer replaces it (issue #92): LightCraft merges its
fields in and keeps the rest — e.g. that application's develop settings and edit history — byte for byte. A
sidecar that isn't valid XMP is copied to `<name>.xmp.bak-<time>` first. With the default stem naming, a raw and a
JPEG with the same name (`IMG_0001.CR3` + `IMG_0001.JPG`) no longer share one sidecar: the raw keeps `IMG_0001.xmp`,
the JPEG uses `IMG_0001.JPG.xmp`.
- A save that fails part-way (a full disk, a network share that drops) no longer looks like a damaged catalog
afterwards (issue #101): the partial write is cut off before LightCraft retries, so the next launch replays every
change. Catalogs already holding such a fragment load in full. Quitting while the catalog log can't be written
still saves your queued changes in the closing snapshot.
- The catalog has a format version (issue #102). Opening a library from an older LightCraft upgrades it; a library
written by a newer LightCraft is refused with "this library was written by a newer version of LightCraft" and left
untouched — older versions no longer read part of it as a damaged log. Once this version has opened a library,
LightCraft 0.2.0 and older refuse it ("unsupported format … v2").
- The control port (`--control`) closes a connection as soon as it receives anything that isn't a JSON request
(issue #94): an HTTP request from a web page can no longer carry a command in its body. Lines are capped at
4 MiB and connections at 16.
- A sleeping NAS, a dropped network share or a USB drive spinning up no longer freezes the window (issue #104):
whether originals are there is checked on a worker thread (grid thumbnails, the Info panel, the photo menu, Missing
Photos); importing (also by drag and drop), Find Missing Photos, Build / Discard Smart Previews, the Rename preview,
the Local folder tree and Auto Import read the disk on worker threads too. The import progress window has Cancel.
- Browser version (experimental), keeping a library safe (issue #107): File ▸ Back Up Library… downloads the catalog
and every imported photo as one zip, and File ▸ Restore Library from Backup… brings it back (the current library
is kept). A failed save (storage full) now shows the unsaved warning and is retried, a photo that can't be stored
isn't added, a second tab shows a message instead of overwriting the first, `?reset` asks first, and the page says
when the browser may evict the library. Hosting: the sample cache headers no longer mark the (unhashed) files
immutable, and HOSTING.md describes the actual build.
- Canon CR2 photos from the EOS 7D, 50D, 60D, 550D, 600D, 1200D, 1300D, 5D Mark II and 1D Mark IV (and other
models whose sensor starts on a green-blue row) no longer come out magenta (issue #85): the colour-filter
layout is read from each file instead of assumed.
@@ -37,6 +63,11 @@
used (e.g. a skipped software adapter such as llvmpipe) and why the last render fell back.
- The thumbnail cache only ever counts and deletes its own files (issue #98): a library opened on a folder that
already has a `thumbs/` folder of other pictures no longer loses them when the cache is trimmed or cleared.
- Settings files are never quietly reset (issue #103): a damaged `prefs.json`, `presets.json` or `view.json` is kept
as `<name>.corrupt-<time>` and you're told; one that can't be read (e.g. locked by another program) is left alone
for the session instead of being overwritten with defaults. The app settings (`ui.json`, which remembers your
library) are written atomically and saved as soon as you open another library, not only at quit. Quitting while
changes couldn't be saved tries once more, then asks: Try Saving Again, Quit Anyway or Cancel.
### Library
- Rename Photos never overwrites another photo when only the letter case changes (issue #95): on case-sensitive
@@ -47,6 +78,9 @@
so none shows as missing. Renaming one of a raw + JPEG pair copies their shared `IMG_0001.xmp` instead of taking
it away from the other (issue #92). Find Missing Photos also finds renamed files by their content, prefers a content match
over a same-name same-size look-alike, and skips (and reports) photos it can't tell apart instead of guessing.
- Rename Folder… and Move Folder To… (Local) can be undone and redone (issue #97): the folder moves back on disk with
its photos and XMP sidecars, and the photos point at it again. If something now occupies the old place, the undo
is refused and nothing is overwritten.
- Smart albums with a rule editor: match all / any / none, nested groups, 26 fields.
- Quick Collection and target album (B in the grid), keyword sets (⌥1–⌥9), colour-label sets.
- Colour-label filter with several labels at once; expandable folder tree in Local.
@@ -64,6 +98,14 @@
- Smaller, faster catalogs: photos you only looked at in Local (never added, rated or edited) are forgotten once
their folder has not been browsed for 30 days — your files and sidecars stay, and browsing the folder shows them
again. Change the period (or turn it off) in Settings → Performance.
- A library is open in one program at a time (issue #99): opening a library that LightCraft or `lightcraft-cli` already
has open — on this computer or another one sharing the folder — says who has it ("already open in LightCraft (process
123 on studio-mac)") instead of letting both write and silently drop each other's edits. A crash never leaves the
library locked: the lock is the operating system's and goes away with the program.
- If your library can't be opened at launch (open in another program, unreadable, on a drive that isn't connected,
written by a newer LightCraft), LightCraft says so and why, and offers Try Again, Choose Another Library…, Continue
Without Saving and Quit (issue #100). It no longer quietly starts a demo session that looked like a reset library and
lost everything at quit; a temporary session shows a banner the whole time and never writes to your library.
### Editing
- Auto Sync: edits apply to every selected photo. Auto B&W mix. Automatic versions.
+35 -2
View File
@@ -8,8 +8,8 @@ other tools), and let LightCraft pick up edits made elsewhere.
| | |
|---|---|
| Name | `<stem>.xmp` by default (`IMG_0001.CR3` → `IMG_0001.xmp`); `<file>.xmp` (`IMG_0001.CR3.xmp`) with `naming: "full"`. Reading accepts either (preferred first) and `.XMP`. |
| Write | `photo.saveMetadataToFile {ids?}` (Photo ▸ Save Metadata to File, ⌘S), or automatically after every change with `library.xmpPreferences {autoWrite: true}` (File ▸ Automatically Write Changes into XMP). Slider drags are written once, when the drag ends; undo/redo rewrite the sidecar. Written atomically (temp file + rename). |
| Name | `<stem>.xmp` by default (`IMG_0001.CR3` → `IMG_0001.xmp`); `<file>.xmp` (`IMG_0001.CR3.xmp`) with `naming: "full"`. Reading accepts either (preferred first) and `.XMP`. With stem naming, files sharing a stem don't share a sidecar: see *Shared names* below. |
| Write | `photo.saveMetadataToFile {ids?}` (Photo ▸ Save Metadata to File, ⌘S), or automatically after every change with `library.xmpPreferences {autoWrite: true}` (File ▸ Automatically Write Changes into XMP). Slider drags are written once, when the drag ends; undo/redo rewrite the sidecar. An existing sidecar is **merged into, never replaced** (see *Saving into an existing sidecar*). Written atomically (temp file, fsync, rename). The result lists `written`, `merged` and `backups`. |
| Read | On import (`library.import`, the report counts `sidecars`), and `photo.readMetadataFromFile {ids?}` (one undo step). For raw/DNG files without a sidecar, the XMP embedded in the file is used. |
| Preferences | `library.xmpPreferences {autoWrite?, naming?: stem\|full}`, stored in the library's `prefs.json`. |
@@ -37,6 +37,39 @@ used — on import (it then also files a copied photo in its date folder) and by
`xmp:Rating="-1"` (the XMP convention for rejected) sets the reject flag. Develop settings come from
`lc:settings` when present (exact); otherwise from the `crs:` fields below (approximate).
## Saving into an existing sidecar
A sidecar may already hold another application's data — e.g. its `crs:` develop settings and `xmpMM:History` — often
the only copy of those edits outside that application's catalog. Saving never drops it:
- LightCraft **owns** the properties in the table above: `xmp:Rating`, `xmp:Label`, `dc:title`, `dc:description`,
`dc:rights`, `dc:creator`, `dc:subject`, `Iptc4xmpCore:Location`, `Iptc4xmpCore:AltTextAccessibility`,
`Iptc4xmpCore:ExtDescrAccessibility`, `photoshop:City`/`State`/`Country`, `xmpRights:Marked`/`UsageTerms`/
`WebStatement` and everything in `lc:`. They are replaced on every save, and removed when LightCraft has no value
(clearing a title in LightCraft clears it in the file). All of them are read back on import, so the library starts
from what the sidecar said.
- The **capture time** (`exif:DateTimeOriginal`, `photoshop:DateCreated`) and **GPS** (`exif:GPSLatitude`,
`exif:GPSLongitude`) are replaced only when LightCraft has a value; otherwise the file's stay.
- **Everything else is kept byte for byte**: other namespaces (`crs:`, `xmpMM:`, `lr:hierarchicalSubject`, unknown
ones), `xmp:CreatorTool`, comments, the packet wrapper and padding. LightCraft's properties go into one
`rdf:Description` of their own; owned properties written by another application (in attribute or element form) are
removed from its description, which otherwise stays as it was.
- A sidecar that can't be read as XMP (not well-formed, not UTF-8, no `rdf:RDF`) is first copied to
`<name>.xmp.bak-<date><time>` (`-1`, `-2`… if taken — a backup is never overwritten), then replaced. A sidecar that
can't be read at all (permissions) is left alone and the save fails.
Limits: LightCraft writes keywords to `dc:subject` only, so another application's `lr:hierarchicalSubject` is kept as
it was and may still list keywords removed in LightCraft. LightCraft doesn't write `crs:`: the other application's
develop settings stay as that application left them, next to LightCraft's own (`lc:settings`, which LightCraft reads
first).
## Shared names
With stem naming, `IMG_0001.CR3` and `IMG_0001.JPG` map to the same `IMG_0001.xmp`. When two or more files in the
library share a stem, the stem sidecar belongs to one of them — a raw first, otherwise the first by file name — and
the others write and read `<file>.xmp` (`IMG_0001.JPG.xmp`), falling back to the stem sidecar for reading when they
have none. So saving one photo never overwrites the other's metadata (`Session::sidecar_naming`).
## Reading `crs:` develop fields
Many raw developers store edits as `crs:` properties (`http://ns.adobe.com/camera-raw-settings/1.0/`) in sidecars, in
+3 -4
View File
@@ -6,11 +6,10 @@ AddType text/javascript .js
AddOutputFilterByType DEFLATE application/wasm text/javascript text/html
</IfModule>
# The file names don't change between versions: revalidate them (no `immutable`), or an upgrade
# could leave browsers with an old module, or old glue next to a new module.
<IfModule mod_headers.c>
<FilesMatch "\.(wasm|js)$">
Header set Cache-Control "public, max-age=31536000, immutable"
</FilesMatch>
<FilesMatch "^index\.html$">
<FilesMatch "\.(wasm|js|html)$">
Header set Cache-Control "no-cache"
</FilesMatch>
</IfModule>
+56 -38
View File
@@ -1,55 +1,74 @@
# Hosting LightCraft for the web
> **Experimental.** The browser version keeps its library (catalog and imported photos) in the
> browser's own storage for the site. Browsers may clear that storage (site data cleared,
> private windows, storage pressure when persistent storage isn't granted). Tell your users to
> keep their original photos elsewhere and to use **File ▸ Back Up Library…**. See
> `docs/web.md` in the source for details.
`lightcraft-web-<version>.zip` (from the GitHub release, or `packaging/web/package.sh`) holds a
static site in `lightcraft-web-<version>/`:
| File | What it is |
|---|---|
| `index.html` | The page. It loads everything through relative URLs. |
| `lightcraft-web-<hash>.js` | wasm-bindgen glue (generated, ES module) |
| `lightcraft-web-<hash>_bg.wasm` | The app, about 13 MB, or 5 MB with compression |
| `lightcraft_web.js` | wasm-bindgen glue (generated, ES module) |
| `lightcraft_web_bg.wasm` | The app, about 13 MB (about 3 MB with brotli) |
| `worker.js` | Starts the render workers (each runs the same module) |
| `*.gz`, `*.br` | Precompressed copies of the files above (optional to serve) |
| `_headers`, `.htaccess` | Sample header rules for Netlify/Cloudflare Pages and Apache |
There is no server-side code. Upload the folder's contents anywhere that serves static files.
## Any path works
All URLs in `index.html` are relative (`public_url = "./"` in `apps/lightcraft-web/Trunk.toml`),
so the site works at a domain root (`https://example.com/`), under a prefix
(`https://example.com/tools/lightcraft/`) and from a CDN bucket. The asset names carry a content
hash, so they can be cached forever. Only `index.html` needs revalidation.
All URLs in `index.html` are relative, so the site works at a domain root
(`https://example.com/`), under a prefix (`https://example.com/tools/lightcraft/`) and from a
CDN bucket.
## Required server settings
- **MIME type:** serve `.wasm` as `application/wasm`. Browsers refuse to stream-compile it under
any other type, and the app then loads slowly or not at all. Serve `.js` as `text/javascript`.
Most hosts already do both. For nginx, check that `mime.types` has `application/wasm wasm;`.
- **Compression:** turn on gzip or Brotli for `.wasm`, `.js` and `.html`. That takes the
download from about 13 MB to about 5 MB. You can also precompress (`brotli -k *.wasm`) and let
the server send `Content-Encoding: br`.
- **Caching:** `Cache-Control: public, max-age=31536000, immutable` on the hashed `.wasm` and
`.js` files, and `no-cache` on `index.html`.
- **HTTPS:** WebGPU (and the clipboard) only work in a secure context, which means `https://`
or `http://localhost`. Over plain HTTP elsewhere, the app falls back to WebGL2.
- **No special isolation headers:** LightCraft doesn't use `SharedArrayBuffer`, so it doesn't
need `Cross-Origin-Opener-Policy` or `Cross-Origin-Embedder-Policy`. If your site already sends
COEP `require-corp`, also send `Cross-Origin-Resource-Policy: same-origin` (or `cross-origin`
when the files live on a CDN) on the app's files.
- **Compression:** turn on gzip or Brotli for `.wasm`, `.js` and `.html`, or serve the
precompressed `.br` / `.gz` files with `Content-Encoding` (nginx `brotli_static` /
`gzip_static`). That takes the download from about 13 MB to about 3–5 MB.
- **Caching:** the file names don't change between versions, so **don't** mark them `immutable`:
a browser would keep the old module after an upgrade, or mix old glue with a new module. Send
`Cache-Control: no-cache` (revalidate with ETag / Last-Modified) on `index.html`, `.js` and
`.wasm`, as the sample `_headers` and `.htaccess` do.
- **HTTPS:** browser storage for the library (OPFS), module workers and the clipboard need a
secure context: `https://`, or `http://localhost` for testing. The app can't be opened from
`file://`.
- **Isolation headers are optional.** The current build doesn't use `SharedArrayBuffer`, so
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy` aren't needed (a future
wasm-threads build will need them; `cargo xtask web --serve` already sends them). If your site
sends COEP `require-corp`, also send `Cross-Origin-Resource-Policy: same-origin` (or
`cross-origin` when the files live on a CDN) on the app's files.
nginx example:
```nginx
location /lightcraft/ {
types { application/wasm wasm; text/javascript js; text/html html; }
gzip on;
gzip_types application/wasm text/javascript text/html;
location ~* \.(wasm|js)$ { add_header Cache-Control "public, max-age=31536000, immutable"; }
location ~* index\.html$ { add_header Cache-Control "no-cache"; }
gzip_static on; # serves the .gz copies
# brotli_static on; # with ngx_brotli: serves the .br copies
add_header Cache-Control "no-cache";
}
```
Local test: `python3 -m http.server 8765` inside the folder, then open http://localhost:8765/.
## Where users' data lives
Everything is stored in the browser, per site origin (scheme + host + port): the catalog and
preferences under `library/`, imported photos under `originals/`, thumbnails under `thumbs/`, in
the Origin Private File System, or IndexedDB where OPFS can't be written. Nothing is uploaded to
your server. Moving the app to another origin starts with an empty library there: users carry
their library over with **File ▸ Back Up Library…** and **File ▸ Restore Library from Backup…**.
One tab at a time can have the library open; a second tab shows a message instead.
## Embedding in a page (iframe)
```html
@@ -64,26 +83,25 @@ Local test: `python3 -m http.server 8765` inside the folder, then open http://lo
- The app fills the iframe and follows its size, so size the iframe and not the app.
- Keyboard shortcuts go to the iframe after the user clicks into it, as with any embedded app.
- **Cross-origin embeds** work. Preferences are kept in the iframe's `localStorage`. Browsers
that partition or block third-party storage may forget them between visits, and the app
then starts with defaults.
- **Cross-origin embeds** work, but browsers that partition or block third-party storage may
give the embedded app separate or temporary storage: its library then differs from the one
on the app's own site, or isn't kept between visits.
- **Sandboxed iframes** need at least
`sandbox="allow-scripts allow-same-origin allow-downloads allow-popups"`. Without
`allow-same-origin` there's no storage. Without `allow-downloads`, Save and Export (browser
downloads) are blocked.
`sandbox="allow-scripts allow-same-origin allow-downloads allow-popups allow-modals"`. Without
`allow-same-origin` there's no storage; without `allow-downloads`, Export and Back Up Library
(browser downloads) are blocked; without `allow-modals`, confirmations (restore, `?reset`)
can't be shown and are treated as "no".
- Don't send `X-Frame-Options: DENY` or a `frame-ancestors` CSP that excludes the embedding page.
## Renderer selection and fallback flags
## Renderer and URL options
LightCraft renders with wgpu. It uses **WebGPU** when the browser has it and falls back to
**WebGL2** on its own. URL query flags override this, and they work on the iframe `src` too:
LightCraft draws its UI with WebGL2 (eframe's `glow` backend) and renders photos on the CPU in
Web Workers. A browser without WebGL2 gets a message in place of the app. URL options, also on
an iframe `src`:
| Flag | Effect |
| Option | Effect |
|---|---|
| *(none)* | WebGPU if available, otherwise WebGL2 |
| `?webgl` | Force the WebGL2 backend (useful when a WebGPU driver misbehaves) |
| `?cpu` | Force the CPU canvas path (slowest, most compatible) |
For example: `<iframe src="https://example.com/lightcraft/?webgl" ...>`.
A browser with neither WebGPU nor WebGL2 gets a message in place of the app.
| `?workers=N` | Number of render workers (default: cores − 1, at most 4); `?workers=0` renders on the main thread |
| `?store=idb` | Keep the library in IndexedDB instead of OPFS |
| `?store=memory` | Keep nothing (a throwaway session) |
| `?reset` | Delete the library stored in this browser (asks first) |
+3 -7
View File
@@ -1,13 +1,9 @@
# Netlify / Cloudflare Pages header rules (see HOSTING.md). Other hosts: set the same headers.
# The file names don't change between versions: revalidate them (no `immutable`), or an upgrade
# could leave browsers with an old module, or old glue next to a new module.
/*
X-Content-Type-Options: nosniff
Cache-Control: no-cache
/*.wasm
Content-Type: application/wasm
Cache-Control: public, max-age=31536000, immutable
/*.js
Cache-Control: public, max-age=31536000, immutable
/index.html
Cache-Control: no-cache