Commit Graph
18 Commits
Author SHA1 Message Date
aiongg 4d91c2ec04 Merge pull request #15826 from aiongg/ci-shard-archives
CI: one test archive per shard, balanced by recorded test times
2026-11-19 20:03:50 -05:00
aiongg 3f28dd6751 CI shards: check the plan against the archive and mode; times only from full runs
- A shard fails unless its plan's GPU mode matches --gpu/--no-gpu and its archive holds exactly the
  binaries its plan names; a single-archive build deletes any shard plan left in the target dir.
- Shard runs pass --no-tests warn: a shard may hold no runnable test, and the build job's check
  already guarantees that every test runs once.
- scripts/ci-test-times.py takes only main's push and nightly runs (pull requests leave packages
  out on Windows) unless given --force, and names binaries a refresh drops.
- target_dir() moves from perf.rs to main.rs for both users.
2026-11-19 22:06:59 +00:00
aiongg aac8e37bec CI: one test archive per shard, balanced by recorded test times
The gate's build job writes one nextest archive per shard (craft-common-actions gate.yml
shard-archives), so each shard downloads only the test binaries it runs instead of the whole
2.4-2.6 GB archive.

`cargo xtask ci archive` builds and lists the tests once, then plans the shards
(xtask/src/shards.rs): whole binaries go to the lightest shard, heaviest first, by the times in
.config/ci-test-times.tsv; a binary heavier than half a shard's share is split by test filter along
its module tree, and its executable goes into each archive that runs part of it. On Linux shard 1
takes every binary with gpu-group tests. The build checks that every test runs on exactly one shard,
writes the plan into the target directory, which every archive carries, and archives each shard
with `cargo nextest archive -E`. A shard runs `xtask ci tests --shard <k>` with its part of the
plan; the slice partition is gone.

scripts/ci-test-times.py refreshes the times file from a CI run's shard logs; a missing or stale
row only costs balance.
2026-11-19 20:47:26 +00:00
aiongg 9406da9c6f engine_split: codemod usage says --head is pre-rebase, reports command_ids and private modules; contributing names TABLES 2026-11-19 20:22:30 +00:00
aiongg 71e84351e0 Engine install registry (#15710)
The command registry moves to `registry.rs`: `CommandSpec`, the `Install`
record, `install()`, `find`, `spec`, `command_specs` and the command-list
helpers. A process installs its command tables once; installing the same
tables again does nothing, and different tables are refused with
`InstallConflict`, which every later command then reports. A session that
runs a command before any install gets `NotInstalled`.

`install.rs` holds the full table list in today's order, `install()`, and
the `command_specs`/`find` wrappers that install first, which the public
`commands::` paths name. `Session::new()` installs. The order test and
`command_ids.txt` move with the list and read the ids from it.

The tables `install.rs` lists are public: the table fragments and seven
command modules that were private.

`scripts/engine_split/install_tables.py` moves a branch's added or removed
command tables (the old `v.extend(...)` lines or `SPECS` entries) into the
install list.
2026-11-19 18:15:14 +00:00
aiongg 7bffc1d271 Name the fragment type; codemod handles the ADJ table, single-entry fragments and rustfmt
The codemod three-way merges the adjustment table into table/image/adjustments.rs,
reports other build() edits as unhandled, compares entries without comments and
whitespace, adds the comma after a single-entry fragment, checks removals and
duplicate ids, and adds the imports new entries need. The order test's failure
message names the regeneration variable.
2026-11-19 17:02:37 +00:00
aiongg 1c97abb475 Split the command table into per-owner fragments
Fragments live in cmd/table/<owner>/<menu>.rs, one directory per future crate
(core, model, store, paint, image). SPECS names each fragment so the order test
can say which one a misplaced command comes from. scripts/engine_split/table_fragments.py
moves a branch's edits of the old inline table into the fragments.
2026-11-19 16:35:19 +00:00
aiongg c9746558d7 ci: build the tests once per OS, keep compiled workspace crates, balance the shards
- Both gate calls build the tests once into a nextest archive (cargo xtask ci archive); the
  shards run it, and the build job runs the doctests after the upload.
- The Linux gate keeps the compiled workspace crates in a rolling cache entry
  (cache-workspace-crates). A nightly cold build is its safety net; gate-cache-prune keeps the
  newest entry. test (without heif) restores that entry.
- The shards split the tests with nextest's slice partition, which spreads the slow modules:
  on one main run's times the non-GPU shards go from 728/493/523 s (hash) to about 602/506/625 s.

Fixes #15702
2026-11-19 08:06:05 +00:00
aiongg ec40c48cc2 ci: build the tests once per OS and run the shards from a nextest archive 2026-11-19 06:34:17 +00:00
aiongg eee31b0d11 CI: shards build their own tests instead of running from an archive
The gate's archive of ~98 test binaries would be 10-18 GB compressed, so
every shard restores the dependency cache, builds the tests and runs its
partition (scripts/ci-test-shard.sh). `cargo xtask ci tests` runs the test
steps on $NEXTEST_PARTITION, `ci build` builds exactly what a shard builds
(main's cache warm-up), and `ci doctest` runs in one shard. The `ci archive`
part, the archive's xtask binary and the extraction are gone.

The Linux GPU shard keeps the gpu group alone on lavapipe; shards 2-4 split
the rest. Windows uses six shards: ~55 min of tests on one runner, and its
slowest test alone takes ~9.6 min, so a seventh would not shorten the
slowest shard.
2026-11-18 19:09:59 +00:00
aiongg 0e3523cdcd CI: shard the Linux and Windows tests, lint leg, xtask environment fix, rust-cache v2.9.3
- xtask starts every cargo without the variables `cargo run` sets for xtask
  (CARGO_MANIFEST_DIR, CARGO_PKG_*, ...), so a build script that declares
  `rerun-if-env-changed` for them (ring) no longer reruns, with the whole
  craft facade above it, between builds inside and outside xtask.
- `cargo xtask ci` splits into the gate's parts without a second step list:
  `lint` (fmt, egui-fork, clippy, layers, panel-commands, scorecard, wasm,
  i18n report), `archive` (test build into the nextest archive, doctests),
  `tests --from <dir>` (test and panic_hunt from the extracted archive, on
  $NEXTEST_PARTITION; --gpu / --no-gpu select by the gpu test group) and
  `build` (cache warm-up). Plain `cargo xtask ci` still runs every step.
  PHOTOCRAFT_CI_EXCLUDE leaves packages out of the test build and doctests.
- The `ci` nextest profile's archive carries the xtask binary, so a shard
  builds nothing (scripts/ci-test-shard.sh).
- Linux gate: lint leg, one build, four shards. Shard 1 runs the gpu group
  alone on lavapipe and checks its output (scripts/ci-gpu-lavapipe.py now
  wraps that run); the second GPU run and its rebuild are gone.
- Windows unit, integration, clippy and warm legs move onto the shared gate
  (Windows runner, lint leg for Clippy, four shards), keeping the PR
  selection from the platforms job, rust-lld, /OPT:NOICF, /DEBUG:NONE and
  the D:\t target dir.
- The slowest-tests-first priorities apply on Linux too.
- Swatinem/rust-cache v2.9.3 by commit for this repository's own cache
  steps, with fresh keys; cache-version v2 for the gate and docs entries.
  `test (without heif)` restores the gate's entry through the shared action.
  Dispatch-only jobs no longer save caches no pull request reads; the quick
  perf suite restores perf-nightly's entry.
- windows_delay_loads reads the app from NEXTEST_BIN_EXE_photocraft, as the
  other app tests do, since archives relocate binaries.
2026-11-18 19:04:43 +00:00
aiongg 973f310aaa ci(windows): on pull requests skip the tests of platform-neutral packages the change does not reach 2026-11-18 14:56:22 +00:00
David MannandClaude Opus 5.5 0ad38ecc19 Address CodeRabbit review: any package manifest runs the Windows GPU tests
A feature enabled in another member's Cargo.toml unifies into the workspace build the GPU tests
run in, and Cargo.lock does not record features, so a manifest-only change outside the GPU inputs
now selects the tests too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 00:53:09 +01:00
David Mann 322d0309ae ci: count both sides of a move in the Windows GPU test selection (#15247)
git diff names only a rename's destination, so moving a file out of
photocraft-gpu looked like a change in the package it moved to. Also
group the platforms job's output lines (actionlint SC2129).
2026-11-16 00:06:01 +01:00
David Mann d971810daa ci: run the Windows GPU tests only when their inputs change (#15247)
The Windows runners render the GPU tests on WARP, Direct3D 12's CPU
rasterizer, where photocraft-gpu's tests took half of the integration
leg's test time. Pull requests now skip them on Windows unless they
change photocraft-gpu, a workspace crate it depends on (from cargo
metadata), or a non-documentation file outside the workspace packages.
main, manual runs and full-ci PRs still run them on Windows, and the
Linux gate runs them on every pull request.
2026-11-15 23:32:31 +01:00
2ff8f419cb ci: run the GPU tests on lavapipe on Linux and fail if they skip (#7975)
The Linux runner has no GPU, so the GPU tests ran on Mesa llvmpipe over
GL, which can't render Rgba32Float: 57 GPU checks (50 GPU parity, 3
canvas, 2 adjustment preview parity, 2 effect-map recovery) returned
early and passed without comparing anything. A new step after Test
installs mesa-vulkan-drivers and runs the nextest `gpu` group on
lavapipe (Vulkan), reusing Test's build, and fails unless the parity
tests report the Vulkan backend and no test printed a skip.

No code change: photocraft's tests take whatever adapter wgpu returns,
software adapters included. Follows the CI half of lightcraft#1148.

Co-authored-by: FredApps <FredApps@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 18:44:13 +01:00
bd632e7eb3 Layered TIFF: open and save Photoshop layer data in TIFF files, in either byte order (part of #215) (#672)
* Layered TIFF: open and save Photoshop layer data in TIFF files, in either byte order (part of #215)

Photoshop keeps a layered TIFF's layers in two private tags: 37724 (ImageSourceData, the same
layer section a PSD has) and 34377 (the image resources). PhotoCraft now reads and writes both,
so a TIFF opens with its groups, masks, layer styles, type, shapes, smart objects and adjustment
layers exactly as the PSD path would, and saves them back.

* photocraft-psd: a `tiff` module with `ImageSourceData` (parse/write, PSD- and PSB-sized),
  the 34377 resource helpers, and a field-by-field byte-order transcoder. In Intel-order TIFFs
  Photoshop byte-swaps the whole block structure (reversed keys, little-endian integers and
  doubles) while pixel samples stay big-endian; the transcoder knows the layer-info structure,
  masks, blending ranges and every block photocraft-io maps (descriptors, TySh, lrFX, the
  binary adjustments, paths, patterns, filter effects, metadata settings, linked layers,
  annotations...), recognizes descriptor-shaped unknown blocks, and drops the rest with a
  warning instead of passing byte-swapped data to other readers. The reader gained an optional
  little-endian mode. 23 unit tests; a fuzz target in the CI matrix.
* photocraft-codecs: the two tags travel as opaque bytes on `Metadata`, read by a
  bounds-checked walk of IFD0 (classic and BigTIFF, both byte orders) and written back as
  BYTE / UNDEFINED tags. `tiff_writes_little_endian()` tells which order the encoder produces.
* photocraft-io: `tiff_layers` rebuilds a PSD model from the TIFF's pixels plus the tags on
  import, and on export runs the PSD exporter (new `merged_matte` / `merged_compression`
  options), interleaves its merged planes for the TIFF image and stores the layer section in
  the encoder's byte order. `ExportOptions.tiff_layers` (default on). Lab and Multichannel
  documents save flat; extra alpha channels are not written (one transparency sample).
* UI: the TIFF Options prompt ("Including layers will increase file size." / "Discard Layers
  and Save a Copy" / "Don't show again") behind Preferences > File Handling > Ask Before
  Saving Layered TIFF Files; `ExportSettings.tiff_layers`; catalogs updated.
* Tests: io round trips at 8/16/32 bits in Gray/RGB/CMYK, equivalence with the PSD path, every
  TIFF compression, the flat fall-backs, malformed input, and files written by psdtags in both
  byte orders (fixtures from scripts/layered_tiff_fixtures.py); a corpus test (feature
  `corpus`) writes every corpus PSD as a layered TIFF and compares the result with the PSD
  round trip (700 files, 0 differences, 0 crashes). Codecs tests cover the tag walk on
  hand-made files.
* Scorecard FILE-215-7 done, BUG-204-3 partial; README and book updated.

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>

* Layered TIFF: de and pt-br strings; neutral wording in the psd tiff docs

The four psdtags fixtures stay out of the repository (no binary test data here): the
generator now writes them to the gitignored corpus/layered-tiff/, and the tests that read
them skip with a message when absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Regenerate docs/scorecard.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: leon <claude.udder842@passmail.net>
Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
2026-10-07 17:10:35 -04:00
Brandon Thomas 968baf639e Photoshop oracles via storytold/photocraft-corpus: pinned corpora, corpus feature, test-corpus, CI job (#190) (#190) 2026-10-05 17:47:33 -04:00