Dev infra: fully-local generation and gallery for the dev stack

Closes the two gaps found smoke-testing the local stack: an empty gallery
(zero media_files rows, media URLs hardcoded to the remote r2.dev bucket)
and generation flows that could not complete (omni_gen calls providers
synchronously in the handler, so dummy keys meant a 500 after billing;
the webapp blocked submits earlier anyway because signup grants no wallet).

Backend (all development-only):
- CDN_BASE_URL env override in cdn_link.rs (host-only, validated loudly;
  precedence over the compiled-in hosts for every domain/env) and a
  dev-gated actix-files mount serving LOCAL_MEDIA_ROOT/media at /media,
  mirroring the public bucket layout - media URLs in API responses resolve
  against the backend itself
- DEV_FAKE_GENERATION flag on ServerState (easyenv bool, forced off outside
  Development): the omni_gen image/video pipelines skip the external
  provider call and return a synthetic Fal enqueue (external id fake_<uuid>),
  so billing, prompt insert, and the pending generic_inference_jobs row all
  stay real
- dev_fake_generation_resolver_thread: polls pending fake_ jobs (new
  runtime-query list_pending_fake_generation_jobs in mysql_queries) and
  after DEV_FAKE_GENERATION_RESOLVE_SECS (default 6s) marks them
  complete_success with a media_files row backed by a test_data asset
  copied under LOCAL_MEDIA_ROOT (mp4 results get -thumb.jpg/-thumb.gif
  siblings since the thumbnail job never runs locally), or complete_failure
  via the existing simulate_artcraft_failure prompt triggers (category word
  supported, e.g. "simulate_artcraft_failure face_not_detected")

Bootstrap/seeds (PS + bash kept in lockstep):
- seed_demo_user seeds an artcraft wallet (DEMO_CREDITS, default 100k
  banked + ledger rows; idempotent top-up) - signup creates no wallet and
  the webapp gates generate on balance, which is why submits went nowhere
- seed_demo_user also seeds 6 gallery images (test_data jpgs) with objects
  under .devstack/media in the exact bucket fan-out layout
- fixed the already-exists detection in seed_demo_user: create_account
  returns error_type, not error_code, so re-runs died on UsernameTaken
- generated secrets env gains DEV_FAKE_GENERATION/CDN_BASE_URL/
  LOCAL_MEDIA_ROOT (appended to existing files on bootstrap re-runs);
  stale "providers are only contacted by workers" comments corrected
- doctor checks wallet balance, media rows, and the new secrets keys

Docs: dev_setup_local_stack.md documents the new flags, corrects the
sync-provider-call and upload-500 claims, updates the seams table and
verification checklist; storyteller_web CLAUDE.md video model examples
fixed (seedance_2p0_fast_g -> seedance_2p0_fast).

Verified E2E on this machine: gallery lists 8 items on localhost URLs;
image and video submits go pending ~6s then complete with locally-served
results (mp4 + gif preview 200); failure trigger yields complete_failure
with category face_not_detected; credits genuinely deducted (100000 ->
99932); doctor 0 failures / 0 warnings.
This commit is contained in:
Tanish Manku committed 2026-07-29 20:48:31 -07:00
1 parent f23a6df80b
commit 335015d9e2
24 files changed
+840 -37

No files matched your search

+38 -5
View File
@@ -249,6 +249,7 @@ write_secrets_env_if_missing() {
step "Development secrets file"
if [ -f "${SECRETS_ENV_FILE}" ]; then
log "Already exists: ${SECRETS_ENV_FILE} (leaving it untouched)."
add_dev_stack_keys_to_existing_secrets_env
return
fi
@@ -256,9 +257,7 @@ write_secrets_env_if_missing() {
# (build_dependencies.rs / setup_inference_providers.rs / setup_stripe_*.rs)
# but is NOT supplied by the checked-in config files. None of them is
# contacted at boot — placeholder values are enough to run the server; the
# endpoints that would use them fail lazily if exercised. Generation is still
# safe: providers are only contacted by the separate job worker binaries,
# which local dev does not run.
# endpoints that would use them fail lazily if exercised.
#
# NOTE: config loading is FIRST-VALUE-WINS (dotenv never overrides an
# already-set key), so values here cannot override anything already set in
@@ -281,8 +280,11 @@ W2L_PUBLIC_DOWNLOAD_BUCKET_NAME=dummy-local-dev-public
# Email (Resend)
RESEND_API_KEY=dummy-local-dev
# Generation providers (only job workers ever contact these; workers are not
# run in local dev, so enqueued jobs simply stay pending)
# Generation providers. NB: the omni_gen endpoints (webapp image/video
# generation) contact providers SYNCHRONOUSLY in the request handler, so with
# dummy keys a real submit would 500 — DEV_FAKE_GENERATION below avoids that.
# The legacy polling providers are only contacted by worker binaries, which
# local dev does not run.
FAL_API_KEY=dummy-local-dev
GMICLOUD_API_KEY=dummy-local-dev
GROK_API_KEY=dummy-local-dev
@@ -300,9 +302,40 @@ STRIPE_ARTCRAFT_ACCOUNT_ID=acct_dummylocaldev
STRIPE_ARTCRAFT_SECRET_KEY=sk_test_dummylocaldev
STRIPE_ARTCRAFT_SECRET_WEBHOOK_KEY=whsec_dummylocaldev
EOF
dev_stack_keys_block >> "${SECRETS_ENV_FILE}"
log "Wrote ${SECRETS_ENV_FILE}"
}
# The fully-local media + fake-generation settings (development-only; the
# server hard-ignores DEV_FAKE_GENERATION outside SERVER_ENVIRONMENT=Development).
dev_stack_keys_block() {
cat <<EOF
# Fully-local dev stack (see _docs/dev_setup_local_stack.md):
# - DEV_FAKE_GENERATION: omni_gen image/video submits skip real providers and
# a resolver thread completes them a few seconds later with a placeholder
# result (or fails them when the prompt contains simulate_artcraft_failure).
# - CDN_BASE_URL: media URLs in API responses point at this server instead of
# the shared r2.dev bucket. Host only — no path suffix.
# - LOCAL_MEDIA_ROOT: directory served at /media (mirrors the bucket layout);
# seeded gallery media and fake generation results live here.
DEV_FAKE_GENERATION=true
CDN_BASE_URL=http://localhost:12345
LOCAL_MEDIA_ROOT=${DEV_MEDIA_ROOT}
EOF
}
# Bootstrap re-runs on a stack whose secrets file predates the fully-local
# media/fake-generation support: append the new keys (first-value-wins never
# lets these override anything the user already set).
add_dev_stack_keys_to_existing_secrets_env() {
if grep -q '^[[:space:]]*LOCAL_MEDIA_ROOT=' "${SECRETS_ENV_FILE}"; then
return
fi
dev_stack_keys_block >> "${SECRETS_ENV_FILE}"
log "Appended fully-local dev stack keys (DEV_FAKE_GENERATION, CDN_BASE_URL, LOCAL_MEDIA_ROOT) to ${SECRETS_ENV_FILE}"
}
build_backend() {
step "Backend build (storyteller-web)"
if [ "${SKIP_RUST_BUILD}" = "true" ]; then