New releases took up to ~30 min to reach \`GET
/v1/craft_apps/release_info\` (ArtCraft Launcher 0.2.0 needed 17–29
min), pods disagreed with each other, and every pod polled GitHub on its
own, which uses up the unauthenticated quota (60/h per IP, and 304s
count) and risks backing off for up to 4 h.
The pods now share their results through Redis. There's no lock:
- **One Redis hash per app** (\`craftApps:releases:app:<key>\`, 7 d
TTL). It holds the latest release, the ETag and \`fetched_ms\`. Before
asking GitHub about an app, a pod reads its entry. If it's fresher than
\`interval − 30 s\` (a clock-skew margin), the pod takes it from Redis
and doesn't call GitHub.
- **Last write wins, guarded by timestamp.** A Lua script never lets an
entry go back in time. A 304 moves only \`fetched_ms\`, and only while
the stored ETag still matches.
- **A soft per-app claim** (60 s, no owner, it just lapses), so a fleet
that ticks every 30 s rarely fetches the same app twice.
- **Backoff shared by the whole fleet**
(\`craftApps:releases:backoff\`), so a rate limit slows everyone down.
Per-app failure retry is 60 s, doubling.
- **The endpoint is unchanged.** It still serves each pod's in-memory
copy. If Redis is down, a pod falls back to polling on its own and warns
at most every 10 minutes.
**The GitHub token is optional.** With
\`CRAFT_APPS_RELEASE_GITHUB_TOKEN\` set, the default interval is 120 s
(min 60). Without it (unset or blank), calls go out unauthenticated with
no Authorization header, at 1800 s (min 300), and startup and polling
are never blocked. \`CRAFT_APPS_RELEASE_COORDINATION_ENABLED=false\`
switches back to the old per-pod polling.
**Tests:** \`cargo test -p storyteller-web --bin storyteller-web craft\`
(49 pass) and \`cargo test -p redis_schema craft\` (4 pass). The Redis
tests passed against a local redis-server with
\`CRAFT_APPS_TEST_REDIS_URL=… -- --include-ignored --skip live_round\`
(51 pass).
**Follow-ups after review:**
- **The poller has its own Redis pool** (2 connections; connects within
3 s; every read and write, including the pool's `PING` check, times out
after 5 s). Before this, it shared the handlers' pool, which has no read
timeout, so a Redis that stopped answering could leave the poller
holding every connection while handlers waited 30 s for one. A test
against a fake Redis that goes silent after the handshake fails without
the timeouts and passes with them.
- **When Redis won't run scripts**, writes fall back to
`WATCH`/`MULTI`/`EXEC`: read and decode the guarding fields, decide, and
write in a transaction that Redis drops if another pod wrote in between.
This switch only happens when Redis refuses the script itself. Being
unreachable, busy, out of memory or read-only doesn't trigger it. It was
tested against a real Redis user denied `@scripting`.
- **New pods load from Redis at startup**, before the startup jitter, so
they serve the fleet's releases right away without asking GitHub.
- **`Cache-Control: no-store` until every app has been fetched**
(`last_fetched` set), so Cloudflare never caches a cold pod's "not
fetched yet" answer. The response body is unchanged.
**Tests:** `cargo test -p storyteller-web --bin storyteller-web craft --
--include-ignored --skip live_round` against a local Redis 8.6: 63 pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
GET /v1/craft_apps/release_info reports the latest release of every
Craft App (and ArtCraft Launcher itself), so launchers stop polling
GitHub directly. A background thread polls GitHub and keeps a shared
cache.
Polling (threads/poll_craft_app_releases_thread):
- One round at startup (after 0–30 s of jitter, so a fleet deploy is
spread out), then every CRAFT_APPS_RELEASE_POLL_INTERVAL_SECS (30 min
default, 5 min minimum) with ±10% jitter
- One request per app (GET /repos/{repo}/releases/latest) with ETag
revalidation and 1 s between requests
- Rate limits stop the round and wait for GitHub's reset, backing off
2×, 4×, … the interval up to CRAFT_APPS_RELEASE_MAX_BACKOFF_SECS (4 h);
rounds where every request failed retry after 1, 2, 4… min, never more
often, up to the interval
- Per app all-or-nothing: either its release and app_last_fetched are
stored, or nothing about it changes
- CRAFT_APPS_RELEASE_POLLING_ENABLED, CRAFT_APPS_RELEASE_GITHUB_TOKEN
and CRAFT_APPS_RELEASE_APPS (key=Name=org/repo,…) configure it
Cache (state/craft_apps_release_cache): Arc<RwLock<BTreeMap<..>>> of
typed entries. release_info.last_fetched is the oldest app_last_fetched,
absent until every app has been fetched. Apps are listed from the start,
without a release or app_last_fetched until fetched.
The endpoint never fails, sets Cache-Control: public, max-age=60, and
returns empty app_renames, repo_renames and important_notices for now.
API types live in artcraft_api_defs::craft_apps::release_info, with each
asset's OS / arch / package type parsed from its file name. reqwest
moves from a dev-dependency to a dependency of storyteller-web.
Tests: cache freshness, asset-name parsing for every published file
type, untrusted-answer filtering, backoff, and an ignored live round
against GitHub that also writes a sample the launcher's parser is tested
against.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Problem
Production `storyteller-web` and the job workers started returning fast
500s with MySQL error 1461:
```
Can't create more than max_prepared_stmt_count statements (current value: 16382)
```
A customer saw this on `GET /v1/prompt/batch`. Errors rose to ~316/hour
on 2026-10-09 and stopped after a rolling restart at ~17:50 UTC. The
restart only clears the caches, so this will happen again under load.
## Cause
sqlx (0.7.4) caches up to **100** prepared statements per connection by
default, and each cached statement stays open on the server. At current
scale, `storyteller-web` can hold:
| Setting | Before | After |
|-------------------------------|----------|---------|
| Replicas | 7 | 7 |
| `MYSQL_MAX_CONNECTIONS` | 50 | 50 |
| Statement cache / connection | 100 | 25 |
| Max statements held | ~35,000 | ~8,750 |
| Server cap | 16,382 | 16,382 |
The ~70 call sites that use `QueryBuilder` (variable-length `IN (...)`
lists) produce many different SQL strings, which fill the caches faster.
## Changes
`crates/service/web/storyteller_web/src/configs/connect_to_database.rs`:
- **`statement_cache_capacity`** defaults to 25, configurable with
`MYSQL_STATEMENT_CACHE_CAPACITY`.
- **`max_lifetime`** defaults to 10 minutes (sqlx's default is 30),
configurable with `MYSQL_MAX_LIFETIME_SECONDS`. A query cancelled
between prepare and close (e.g. the client disconnects) leaves its
statement open on the server until the connection closes. Recycling
connections sooner frees those statements sooner.
- Both values are logged with the existing pool settings at startup.
Job workers are left unchanged: their pools are hardcoded to 2
connections, so each holds at most 200 statements.
## Notes
- If replicas or `MYSQL_MAX_CONNECTIONS` increase, keep `replicas ×
MYSQL_MAX_CONNECTIONS × MYSQL_STATEMENT_CACHE_CAPACITY` well under
16,382.
- Follow-up (not in this PR): use `.persistent(false)` on hot dynamic
`QueryBuilder` queries such as `batch_get_prompts`.
## Testing
- `SQLX_OFFLINE=true cargo check -p storyteller-web` passes.
- After deploy, look for the new values in the `Building MySQL pool:`
startup log line, and confirm no new `max_prepared_stmt_count` errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Brandon ThomasandClaude Opus 5.5
