Files
Brandon ThomasandClaude Opus 5.5 0c441d37c0 Coordinate Craft App release polling across the fleet through Redis
Every storyteller-web pod polled GitHub on its own every 30 minutes, so
new releases took up to ~33 min to appear, pods disagreed with each
other, and without a token the fleet burned GitHub's 60/hour/IP quota
(unauthenticated 304s count too). Pods now share the work in Redis.

Shared state (keys in redis_schema::keys::craft_apps):
- craftApps:releases:app:<app key>: hash per app (v=1, fetched_ms, etag,
release JSON, failed_ms, failures, claimed_ms, by); 7 day TTL refreshed
on every write
- craftApps:releases:backoff: fleet-wide backoff hash (v=1, until_ms,
trouble, by); 1 day TTL, deleted after a good round

Loop (every pod, every 30 s ±10%, after the existing startup jitter):
- Read every app entry plus the backoff in one pipeline; take anything
newer into the in-memory cache, which the endpoint keeps serving
- Stop if the fleet is backing off
- For each app whose shared fetch is older than interval - 30 s (clock
skew margin), and whose failure retry (1, 2, 4… min up to the interval)
has passed: re-read its entry, claim it (claimed_ms, 60 s window, so
pods ticking together don't all ask), then ask GitHub with the shared
ETag and write back
- Writes go through a Lua compare-and-set on the timestamp field, so a
slow pod never replaces a newer fetch; a 304 moves only fetched_ms and
only if the stored ETag is the one revalidated
- Rate limits and all-failed rounds set the shared backoff with the
existing next_wait rules, so backoff is cluster-wide
- Redis errors (3 s connection / 10 s call timeouts, run off the async
runtime with spawn_blocking) fall back to polling alone from memory,
with a warning at most every 10 minutes, and return to coordination
when Redis answers

Config: the default interval depends on the token: 120 s (minimum 60 s)
with CRAFT_APPS_RELEASE_GITHUB_TOKEN, 1800 s (minimum 300 s) without.
The token stays optional: unset, empty or blank means unauthenticated
requests and never an empty Authorization header.
CRAFT_APPS_RELEASE_COORDINATION_ENABLED=false (default true) switches
back to standalone polling.

GET /v1/craft_apps/release_info is unchanged.

Tests: freshness with skew, failure retries, shared backoff, encoding
round trips and garbage tolerance, cache merges, Redis fallback, token
and no-token request building; ignored tests against a real Redis for
the guarded writes, claims and cross-pod refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:12:44 -04:00
..
2026-06-29 21:21:26 -04:00
2026-09-25 23:23:00 -07:00
2026-07-03 02:28:22 -04:00