diff --git a/.gitattributes b/.gitattributes index 3b9631fee1..1a7f5222ee 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,3 +1,8 @@ +# Shell scripts and dotenv config files are executed/parsed on Linux (WSL2, CI) +# even when the repo is cloned on Windows — they must stay LF in the worktree. +*.sh text eol=lf +*.env text eol=lf + # Ignore certain code for Github language statistics # https://github.com/github/linguist/blob/master/docs/overrides.md old_code/trumpet-trumped.com/dictionary/cmudict-0.7b linguist-vendored diff --git a/_docs/dev_setup_local_stack.md b/_docs/dev_setup_local_stack.md new file mode 100644 index 0000000000..3b2083d99c --- /dev/null +++ b/_docs/dev_setup_local_stack.md @@ -0,0 +1,213 @@ +# Local Dev Stack — Requirements & Bootstrap + +How to run the full ArtCraft dev stack **entirely locally** — the +`storyteller-web` backend plus the `artcraft-webapp` frontend — with no traffic +to production servers. Scope: webapp-first; the Tauri desktop app is a later +phase. + +The automated path is: + +```bash +./script/bootstrap/bootstrap_dev_stack.sh # one-time setup, idempotent, re-run anytime +./script/bootstrap/run_backend_dev.sh # terminal 1: API on http://localhost:12345 +./script/bootstrap/seed_demo_user.sh # terminal 2: create demo login (localdev1 / localdev1pass) +cd frontend && npx nx dev artcraft-webapp # terminal 2: webapp on http://localhost:4201 +./script/bootstrap/dev_stack_doctor.sh # health check, any time +``` + +This document records the underlying requirements so the scripts aren't a +black box, and corrects some stale claims in older docs. + +## Supported platforms + +| Platform | Status | +|---------------------|---------------------------------------------------------------------------| +| Ubuntu 22.04+ | Supported by the bootstrap scripts (native, WSL2, or CI runner/container) | +| Windows | Use WSL2: `wsl --install -d Ubuntu-22.04`, then run the scripts inside it. Prefer cloning inside the WSL filesystem (`~/...`), not `/mnt/c` — Rust/npm builds through `/mnt` are much slower | +| macOS | Not covered by the scripts; follow `_docs/dev_setup_server.md` manually | + +## Requirements matrix + +| Requirement | Version / detail | Needed for | +|-------------------|--------------------------------------------------|---------------------------------------------------| +| MySQL server | 8.x series (9.x unsupported by tooling) | Backend boot (eager connect) + migrations | +| Redis server | any recent | Backend boot (r2d2 pool connects eagerly) | +| Rust (stable) | no pinned MSRV; Docker builds use 1.93.0 | Building `storyteller-web`, diesel_cli | +| diesel_cli | `cargo install diesel_cli --no-default-features --features mysql,sqlite` | Running the 281 migrations | +| Node.js | 20+ (enforced by `script/common/frontend_preflight.sh`) | Frontend (Nx 21 / Vite 6) | +| npm | ships with Node; **pnpm must not be used** | Frontend install (`frontend/package-lock.json`) | +| System libs (apt) | build-essential, cmake, libssl-dev, libclang-dev, libfontconfig1-dev, fontconfig, perl, pkg-config, ffmpeg, libmysqlclient-dev, libsqlite3-dev, curl, jq | Compiling the workspace + diesel_cli; seeding scripts | +| Elasticsearch | optional (7.x) | Only 4 search endpoints; server boots without it | +| Docker | **not required** — no docker-compose exists | — | + +Database identity (used consistently by docs, `.env`, and defaults): +database `storyteller`, user `storyteller`, password `password`, host +`localhost`. + +## Backend: what `storyteller-web` needs to boot + +Binary: `storyteller-web` in `crates/service/web/storyteller_web` +(binaries: `storyteller-web`, `docs` — Swagger UI on :8989, `docs-cli` — emits +`api.json`). Binds `0.0.0.0:12345` (`BIND_ADDRESS`), plain HTTP. +**Run it from the repo root** — the config search path and `includes/` paths +are cwd-relative. + +### Config loading (corrected facts) + +- Mode is selected by the `SERVER_ENVIRONMENT` **process env var only** + (default: Development). Setting it inside an env file does nothing — it is + read before any file loads. +- In Development mode, files load from + `crates/service/web/storyteller_web/config/` in this order: + `storyteller-web.common.env` → `storyteller-web.development.env` → + `storyteller-web.development-secrets.env` (gitignored; the bootstrap + generates it). +- **Precedence is first-value-wins** (dotenv never overrides an existing key): + real process env > common.env > development.env > development-secrets.env. + The secrets file therefore cannot override anything already set in + `development.env`. +- The repo-root `.env` is **NOT loaded by the server** (it passes + `ignore_legacy_dot_env_file: true`). Root `.env`'s `DATABASE_URL` feeds + diesel-cli/sqlx-cli only; the server reads **`MYSQL_URL`** (default + `mysql://storyteller:password@localhost/storyteller`). + +### Hard requirements at boot + +| Requirement | Behavior if missing | +|----------------------------------|--------------------------------------------------------| +| MySQL reachable via `MYSQL_URL` | Boot fails (sqlx pool connects eagerly; the DB must exist and be migrated — health-check and ban-poller threads query immediately) | +| Redis reachable via `REDIS_0_URL` (default `redis://localhost/0`) | Boot fails (r2d2 pool connects eagerly) | +| ~30 `get_env_string_required` vars present | Boot fails on the first absent var | + +Of the required-present vars, 19 are supplied by **no** checked-in file and +live in the generated secrets file: `ACCESS_KEY`, `SECRET_KEY`, `REGION_NAME`, +`W2L_PRIVATE_DOWNLOAD_BUCKET_NAME`, `W2L_PUBLIC_DOWNLOAD_BUCKET_NAME`, +`RESEND_API_KEY`, `FAL_API_KEY`, `GMICLOUD_API_KEY`, `GROK_API_KEY`, +`BEEBLE_API_KEY`, `OPENAI_API_KEY`, `WORLDLABS_API_KEY`, +`SEEDANCE2PRO_COOKIES`, `SEEDANCE2PRO_WHITELIST_COOKIES`, +`SEEDANCE2PRO_BYTEPLUS_ULTRA_COOKIES`, `STRIPE_FAKEYOU_ACCOUNT_ID`, +`STRIPE_ARTCRAFT_ACCOUNT_ID`, `STRIPE_ARTCRAFT_SECRET_KEY`, +`STRIPE_ARTCRAFT_SECRET_WEBHOOK_KEY`. + +**Placeholder values are safe**: none of these services is contacted at boot +(clients are constructed offline; Elasticsearch, S3/R2, Stripe, Resend, and +Google-cert fetches are all lazy). Generation stays safe by construction — +handlers only validate, bill the wallet, and insert job rows; providers are +contacted exclusively by separate worker binaries that local dev never runs, +so enqueued jobs just stay pending at zero cost. + +### Auth on localhost + +- `POST /v1/login` takes `{"username_or_email": "...", "password": "..."}`; + success returns `{"success": true, "signed_session": ""}` **and** a + `session` cookie. +- With the checked-in dev config (`COOKIE_DOMAIN=.jungle.horse`), the session + cookie is issued non-Secure and host-only (the Domain attribute is never + set), so plain-HTTP localhost login works as-is. The frontend additionally + stores `signed_session` in localStorage and sends it as a `session` header. +- Dev-mode CORS allows any `localhost`/`127.0.0.1` origin on any port, with + credentials. +- Health endpoint for scripts/CI: `GET /_status`. + +### Build + +- Compile with `SQLX_OFFLINE=true` (uses the checked-in `.sqlx/` cache; 483 + queries). Without it, the sqlx macros open a **live DB connection at compile + time** using `DATABASE_URL` from the root `.env`. `SQLX_OFFLINE` has no + runtime effect — ignore older notes saying not to set it when running. +- First build takes several minutes; the crate has ~370 pre-existing warnings. + +## Database: migrations and seed data + +- 281 Diesel migrations in `_database/sql/migrations/` (`diesel.toml` at repo + root points there). Run from the repo root: `diesel migration run`. + `"Encountered unknown type for Mysql: enum"` warnings are harmless. + `migrations_squashed/` is documentation only — not executable. +- Seed data: `_database/sql/seed/sql/system_roles.sql` (roles `user`, `mod`, + `admin`) and `user_badges.sql`. **The `user` role is mandatory** — account + creation hardcodes `user_role_slug='user'`. The legacy wrapper + `bootstrap_inserts_roles_etc.sh` must run with cwd `_database/sql/` and is + not idempotent; the bootstrap script applies the same SQL files with + idempotency guards instead. +- Demo user: created through **`POST /v1/create_account`** (no captcha, no + invite code, no email-verification gate; password just needs 6+ chars). This + exercises the real signup path instead of hand-rolled SQL. Gotcha: usernames + like `demo`, `admin`, `test`, `dev`, `artcraft` are reserved + (`includes/binary_includes/usernames/reserved_usernames.txt`) — the default + demo user is `localdev1`. The Rust seeder + (`cargo run --bin dev-database-seed`) exists but hard-requires a + `.env-secrets` file and also seeds model weights, so the HTTP path is + preferred. + +## Frontend: `artcraft-webapp` + +- Nx workspace at repo-root `frontend/` (npm workspaces; `npm install` from + `frontend/`). The FontAwesome Pro registry token is committed in + `frontend/.npmrc` — no manual auth step. +- Serve: `npx nx dev artcraft-webapp` → `http://localhost:4201`. No lib + pre-build is needed in dev (libs resolve to TS source via the + `"development"` export condition); production `nx build` does need lib + builds. +- **In dev the webapp already targets the local backend**: `src/main.tsx` + calls `StorytellerApiHostStore.setDevelopment()` under + `import.meta.env.DEV`, pointing all API traffic at + `http://localhost:12345`. (The adjacent comment claiming this is disabled is + stale — the call is live, and it bypasses the Vite `/v1` → + `api.storyteller.ai` proxy.) So: local backend running → fully local stack; + local backend **not** running → dev webapp is broken, not silently on prod. +- Optional: `VITE_GOOGLE_CLIENT_ID` in `frontend/apps/artcraft-webapp/.env` + enables the Google SSO button; email/password login needs nothing. +- Other dev ports if you run more apps: website 4200, dashboard 4202 (separate + pnpm app, excluded from Nx), video-info-website 4250, genhub 4300, Tauri + frontend 5173. + +## What still touches production (known, deliberate seams) + +Cleaning these up is the next phase of the infra work; the bootstrap does not +change runtime behavior: + +| Seam | Today | +|-----------------------------------------------|--------------------------------------------------------------| +| Media/CDN URLs in API responses | Compiled-in constants (`cdn_link.rs`) point dev builds at a public `pub-….r2.dev` bucket — media loads remotely even with a local backend | +| Vite `/v1` proxy in webapp/website | Hardcoded `https://api.storyteller.ai` (bypassed in dev by `setDevelopment()`, but still the fallback path) | +| `SegmentationApi`, `GetCdnOrigin`, `API_TARGETS` | Hardcoded off-store production hosts | +| Elasticsearch-backed search | `search_featured` / `search_session` media files, `weights/search`, legacy `tts/search` fail without a local ES + `elasticsearch-cli` reindex (optional) | + +## CI usage + +The same bootstrap works on a fresh Ubuntu runner/container: + +```bash +export MYSQL_ROOT_PASSWORD=root # GitHub Actions' preinstalled MySQL uses password auth +./script/bootstrap/bootstrap_dev_stack.sh --yes +./script/bootstrap/run_backend_dev.sh & +./script/bootstrap/seed_demo_user.sh # waits for /_status +./script/bootstrap/dev_stack_doctor.sh +``` + +`--yes` answers all prompts; `--skip-frontend` / `--skip-rust-build` / +`--skip-packages` slice the work for focused jobs. + +## Verification checklist + +- `./script/bootstrap/dev_stack_doctor.sh` exits 0 with no FAILs. +- `POST /v1/login` with the demo user returns `success: true` plus a + `signed_session` (done automatically by `seed_demo_user.sh`). +- Webapp at `http://localhost:4201` can log in as `localdev1`. +- A generation enqueue returns `{"success": true, "inference_job_token": …}` + and no external provider is contacted (no workers running). +- Nothing hits `api.storyteller.ai` during the above (verify in the browser + network tab; media/CDN requests are the known exception — see seams table). + +## Corrections to older docs + +- `crates/service/web/storyteller_web/CLAUDE.md`: the repo-root `.env` is + *not* auto-loaded by the server, and `SQLX_OFFLINE=true` is fine (and + recommended) when running — it only affects compile-time query checking. +- `_docs/dev_setup_server.md`: `./script/dev_mysql_connect.sh` doesn't exist — + it's `./script/dev_database_connect.sh`. +- `diesel.toml`'s comment about needing an unreleased diesel CLI is stale — + any current `diesel_cli` works. +- `TESTING_INFRA_HANDOFF.md` (untracked): the server env var is `MYSQL_URL`, + not `DATABASE_URL`; migration count is 281; crate path is + `crates/service/web/storyteller_web`. diff --git a/script/bootstrap/bootstrap_dev_stack.sh b/script/bootstrap/bootstrap_dev_stack.sh new file mode 100755 index 0000000000..e5636963fd --- /dev/null +++ b/script/bootstrap/bootstrap_dev_stack.sh @@ -0,0 +1,354 @@ +#!/usr/bin/env bash +# Bootstrap the full local dev stack (backend + frontend) on a fresh clone. +# +# Target platform: Ubuntu 22.04+ — native, WSL2, or a CI runner/container. +# (macOS users: follow _docs/dev_setup_server.md manually; Windows users: run +# this inside WSL2. See _docs/dev_setup_local_stack.md.) +# +# Idempotent: safe to re-run at any time; each step detects work already done. +# +# Usage: +# ./script/bootstrap/bootstrap_dev_stack.sh [options] +# +# Options: +# --yes, -y Non-interactive: assume "yes" for prompts (CI mode) +# --skip-packages Don't apt-install system packages +# --skip-rust-build Don't build the storyteller-web binary +# --skip-frontend Don't run the frontend npm install +# +# After it succeeds: +# ./script/bootstrap/run_backend_dev.sh # start the API on :12345 +# ./script/bootstrap/seed_demo_user.sh # create the demo login +# cd frontend && npx nx dev artcraft-webapp # webapp on :4201 +set -euo pipefail + +root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +source "${root_dir}/script/bootstrap/common.sh" + +SECRETS_ENV_FILE="${root_dir}/crates/service/web/storyteller_web/config/storyteller-web.development-secrets.env" + +# System packages: MySQL + Redis servers, the native libs needed to compile the +# Rust workspace (per build/service_cpu.Dockerfile) and diesel_cli +# (libmysqlclient-dev, libsqlite3-dev), plus curl/jq for the seeding scripts. +SYSTEM_PACKAGES=( + build-essential + ca-certificates + cmake + curl + ffmpeg + fontconfig + git + jq + libclang-dev + libfontconfig1-dev + libmysqlclient-dev + libsqlite3-dev + libssl-dev + mysql-server + perl + pkg-config + redis-server +) + +ASSUME_YES=false +SKIP_PACKAGES=false +SKIP_RUST_BUILD=false +SKIP_FRONTEND=false + +main() { + parse_args "$@" + ensure_supported_platform + + install_system_packages + ensure_mysql_running + provision_mysql_database + ensure_redis_running + ensure_rust_toolchain + ensure_diesel_cli + run_migrations + seed_roles_and_badges + write_secrets_env_if_missing + build_backend + setup_frontend + + print_next_steps +} + +parse_args() { + for arg in "$@"; do + case "${arg}" in + --yes|-y) ASSUME_YES=true ;; + --skip-packages) SKIP_PACKAGES=true ;; + --skip-rust-build) SKIP_RUST_BUILD=true ;; + --skip-frontend) SKIP_FRONTEND=true ;; + --help|-h) sed -n '2,22p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) die "Unknown option: ${arg} (try --help)" ;; + esac + done +} + +ensure_supported_platform() { + case "$(uname -s)" in + Linux) ;; + Darwin) + die "This script targets Ubuntu/WSL2. On macOS, follow _docs/dev_setup_server.md." + ;; + *) + die "Unsupported platform '$(uname -s)'. Run inside WSL2 Ubuntu (wsl --install -d Ubuntu-22.04)." + ;; + esac + + if grep -qi microsoft /proc/version 2>/dev/null && [[ "${root_dir}" == /mnt/* ]]; then + warn "This checkout lives on the Windows filesystem (${root_dir})." + warn "Rust + npm builds are dramatically slower through /mnt. Consider cloning" + warn "into the WSL filesystem (e.g. ~/storyteller-rust) instead." + fi +} + +install_system_packages() { + step "System packages" + if [ "${SKIP_PACKAGES}" = "true" ]; then + log "Skipped (--skip-packages)." + return + fi + + local missing=() + for pkg in "${SYSTEM_PACKAGES[@]}"; do + dpkg -s "${pkg}" >/dev/null 2>&1 || missing+=("${pkg}") + done + + if [ "${#missing[@]}" -eq 0 ]; then + log "All required apt packages are already installed." + return + fi + + log "Missing packages: ${missing[*]}" + confirm "Install them with apt-get?" || die "Cannot continue without system packages (or pass --skip-packages if you manage them yourself)." + maybe_sudo env DEBIAN_FRONTEND=noninteractive apt-get update + maybe_sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y "${missing[@]}" +} + +ensure_mysql_running() { + step "MySQL server" + if mysqladmin ping --silent 2>/dev/null; then + log "MySQL is running." + else + log "Starting MySQL..." + maybe_sudo service mysql start 2>/dev/null \ + || maybe_sudo systemctl start mysql 2>/dev/null \ + || die "Could not start MySQL (tried 'service mysql start' and 'systemctl start mysql')." + mysqladmin ping --silent 2>/dev/null || sleep 3 + mysqladmin ping --silent 2>/dev/null || die "MySQL did not come up." + fi + + # The tooling supports the MySQL 8.x series; 9.x is known-unsupported + # (_docs/dev_setup_server.md). + local version + version="$(mysqld --version 2>/dev/null || true)" + case "${version}" in + *" 8."*) ;; + "") warn "Could not determine MySQL server version." ;; + *) warn "MySQL 8.x is expected; found: ${version}" ;; + esac +} + +provision_mysql_database() { + step "MySQL database + user (${DEV_MYSQL_DB} / ${DEV_MYSQL_USER}@localhost)" + if mysql_app_reachable; then + log "Database and app user already provisioned." + return + fi + + # Same DDL as _docs/dev_setup_server.md, made idempotent. + mysql_admin </dev/null \ + || maybe_sudo systemctl start redis-server 2>/dev/null \ + || die "Could not start Redis." + redis_reachable || die "Redis did not come up." +} + +ensure_rust_toolchain() { + step "Rust toolchain" + ensure_cargo_on_path + if command -v cargo >/dev/null 2>&1; then + log "Found $(rustc --version 2>/dev/null || echo 'cargo (rustc missing?)')." + return + fi + confirm "Rust is not installed. Install stable via rustup?" || die "Rust is required to build the backend." + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable + ensure_cargo_on_path + command -v cargo >/dev/null 2>&1 || die "rustup install finished but cargo is not on PATH." +} + +ensure_diesel_cli() { + step "diesel_cli (migration runner)" + if command -v diesel >/dev/null 2>&1; then + log "Found $(diesel --version)." + return + fi + # Same install command as _docs/dev_setup_server.md. Needs libmysqlclient-dev + # and libsqlite3-dev (installed above). + log "Installing diesel_cli (this compiles from source; a few minutes)..." + cargo install diesel_cli --no-default-features --features mysql,sqlite +} + +run_migrations() { + step "Database migrations (_database/sql/migrations)" + # diesel-cli reads DATABASE_URL; the repo-root .env provides it, but export a + # default so this also works if that file is ever absent (e.g. sparse CI). + export DATABASE_URL="${DATABASE_URL:-mysql://${DEV_MYSQL_USER}:${DEV_MYSQL_PASSWORD}@${DEV_MYSQL_HOST}/${DEV_MYSQL_DB}}" + cd "${root_dir}" + # NB: "Encountered unknown type for Mysql: enum" warnings are harmless + # (_docs/dev_setup_server.md). + diesel migration run + log "Migrations up to date." +} + +seed_roles_and_badges() { + step "Seed data (system roles + badges)" + # Same data as _database/sql/seed/bootstrap_inserts_roles_etc.sh, but guarded + # so re-runs don't hit duplicate-key errors. The 'user' role is mandatory: + # account creation hardcodes user_role_slug='user'. + local role_count badge_count + role_count="$(mysql_app -N -e "SELECT COUNT(*) FROM user_roles" 2>/dev/null || echo 0)" + if [ "${role_count}" -gt 0 ]; then + log "user_roles already seeded (${role_count} rows)." + else + mysql_app < "${root_dir}/_database/sql/seed/sql/system_roles.sql" + log "Inserted system roles (user, mod, admin)." + fi + + badge_count="$(mysql_app -N -e "SELECT COUNT(*) FROM badges" 2>/dev/null || echo 0)" + if [ "${badge_count}" -gt 0 ]; then + log "badges already seeded (${badge_count} rows)." + else + mysql_app < "${root_dir}/_database/sql/seed/sql/user_badges.sql" + log "Inserted badges." + fi +} + +write_secrets_env_if_missing() { + step "Development secrets file" + if [ -f "${SECRETS_ENV_FILE}" ]; then + log "Already exists: ${SECRETS_ENV_FILE} (leaving it untouched)." + return + fi + + # Every env var below is read with get_env_string_required at server startup + # (build_dependencies.rs / setup_inference_providers.rs / setup_stripe_*.rs) + # but is NOT supplied by the checked-in config files. None of them is + # contacted at boot — placeholder values are enough to run the server; the + # endpoints that would use them fail lazily if exercised. Generation is still + # safe: providers are only contacted by the separate job worker binaries, + # which local dev does not run. + # + # NOTE: config loading is FIRST-VALUE-WINS (dotenv never overrides an + # already-set key), so values here cannot override anything already set in + # storyteller-web.development.env — only real process env vars can. + cat > "${SECRETS_ENV_FILE}" <<'EOF' +# Local-development secrets for storyteller-web. Generated by +# script/bootstrap/bootstrap_dev_stack.sh — gitignored; edit freely. +# +# Placeholders below satisfy required-at-boot checks without enabling any +# external integration. Replace individual values with real dev credentials +# only if you need that specific integration locally. + +# R2/S3 object storage (clients are constructed offline) +ACCESS_KEY=dummy-local-dev +SECRET_KEY=dummy-local-dev +REGION_NAME=us-east-1 +W2L_PRIVATE_DOWNLOAD_BUCKET_NAME=dummy-local-dev-private +W2L_PUBLIC_DOWNLOAD_BUCKET_NAME=dummy-local-dev-public + +# Email (Resend) +RESEND_API_KEY=dummy-local-dev + +# Generation providers (only job workers ever contact these; workers are not +# run in local dev, so enqueued jobs simply stay pending) +FAL_API_KEY=dummy-local-dev +GMICLOUD_API_KEY=dummy-local-dev +GROK_API_KEY=dummy-local-dev +BEEBLE_API_KEY=dummy-local-dev +OPENAI_API_KEY=dummy-local-dev +WORLDLABS_API_KEY=dummy-local-dev +SEEDANCE2PRO_COOKIES=dummy-local-dev +SEEDANCE2PRO_WHITELIST_COOKIES=dummy-local-dev +SEEDANCE2PRO_BYTEPLUS_ULTRA_COOKIES=dummy-local-dev + +# Stripe account ids (test-mode secret keys are already checked into +# storyteller-web.development.env; only the account ids are missing there) +STRIPE_FAKEYOU_ACCOUNT_ID=acct_dummylocaldev +STRIPE_ARTCRAFT_ACCOUNT_ID=acct_dummylocaldev +STRIPE_ARTCRAFT_SECRET_KEY=sk_test_dummylocaldev +STRIPE_ARTCRAFT_SECRET_WEBHOOK_KEY=whsec_dummylocaldev +EOF + log "Wrote ${SECRETS_ENV_FILE}" +} + +build_backend() { + step "Backend build (storyteller-web)" + if [ "${SKIP_RUST_BUILD}" = "true" ]; then + log "Skipped (--skip-rust-build)." + return + fi + cd "${root_dir}" + # SQLX_OFFLINE only affects compile-time query verification (uses the + # checked-in .sqlx/ cache instead of a live DB); it has no runtime effect. + log "Building (first build takes several minutes)..." + SQLX_OFFLINE=true cargo build -p storyteller-web --bin storyteller-web + log "Built target/debug/storyteller-web." +} + +setup_frontend() { + step "Frontend install (frontend/)" + if [ "${SKIP_FRONTEND}" = "true" ]; then + log "Skipped (--skip-frontend)." + return + fi + source "${root_dir}/script/common/frontend_preflight.sh" + frontend_preflight "${root_dir}/frontend" + pushd "${root_dir}/frontend" >/dev/null + frontend_npm_install + popd >/dev/null + log "Frontend dependencies installed." +} + +print_next_steps() { + step "Done" + cat <&2; } +die() { echo "[bootstrap] ERROR: $*" >&2; exit 1; } + +step() { + echo "" + echo "=== $* ===" +} + +# sudo unless we already are root (CI containers often run as root without sudo). +maybe_sudo() { + if [ "$(id -u)" -eq 0 ]; then + "$@" + else + sudo "$@" + fi +} + +confirm() { + local prompt="$1" + if [ "${ASSUME_YES:-false}" = "true" ]; then + return 0 + fi + local reply + read -r -p "${prompt} [y/N] " reply + [[ "${reply}" =~ ^[Yy] ]] +} + +# Run a statement (or stdin) as the storyteller app user against the dev DB. +# MYSQL_PWD keeps the password off the process list. +mysql_app() { + MYSQL_PWD="${DEV_MYSQL_PASSWORD}" mysql \ + -u "${DEV_MYSQL_USER}" -h "${DEV_MYSQL_HOST}" -D "${DEV_MYSQL_DB}" "$@" +} + +# Run a statement as an administrator, for provisioning the DB and user. +# On a fresh Ubuntu/WSL install, root uses auth_socket, so `sudo mysql` works. +# On GitHub Actions runners the preinstalled MySQL root password is "root" — +# export MYSQL_ROOT_PASSWORD to use password auth instead. +mysql_admin() { + if [ -n "${MYSQL_ROOT_PASSWORD:-}" ]; then + MYSQL_PWD="${MYSQL_ROOT_PASSWORD}" mysql -u root -h "${DEV_MYSQL_HOST}" "$@" + else + maybe_sudo mysql "$@" + fi +} + +mysql_app_reachable() { + mysql_app -e "SELECT 1" >/dev/null 2>&1 +} + +redis_reachable() { + redis-cli ping 2>/dev/null | grep -q PONG +} + +backend_reachable() { + curl -sf -o /dev/null --max-time 3 "${DEV_BACKEND_URL}/_status" +} + +# Make ~/.cargo/bin visible when rustup was installed but the shell profile +# hasn't been re-sourced (fresh installs, CI). +ensure_cargo_on_path() { + if ! command -v cargo >/dev/null 2>&1 && [ -x "${HOME}/.cargo/bin/cargo" ]; then + export PATH="${HOME}/.cargo/bin:${PATH}" + fi +} diff --git a/script/bootstrap/dev_stack_doctor.sh b/script/bootstrap/dev_stack_doctor.sh new file mode 100755 index 0000000000..525e3250e4 --- /dev/null +++ b/script/bootstrap/dev_stack_doctor.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# Diagnose the local dev stack: checks every prerequisite the bootstrap sets up +# and reports PASS / WARN / FAIL per item. +# +# FAIL = the stack cannot work until fixed (exit code 1). +# WARN = optional or run-time-only (e.g. backend not currently running). +set -uo pipefail + +root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +source "${root_dir}/script/bootstrap/common.sh" + +SECRETS_ENV_FILE="${root_dir}/crates/service/web/storyteller_web/config/storyteller-web.development-secrets.env" + +failures=0 +warnings=0 + +pass() { printf ' PASS %s\n' "$1"; } +fail() { printf ' FAIL %s\n' "$1"; failures=$((failures + 1)); } +warn_check() { printf ' WARN %s\n' "$1"; warnings=$((warnings + 1)); } + +echo "Dev stack doctor (${root_dir})" +echo "" +echo "--- Toolchains ---" + +ensure_cargo_on_path +if command -v cargo >/dev/null 2>&1; then + pass "rust: $(rustc --version 2>/dev/null | head -1)" +else + fail "rust: cargo not on PATH (bootstrap installs via rustup)" +fi + +if command -v diesel >/dev/null 2>&1; then + pass "diesel_cli: $(diesel --version 2>/dev/null | head -1)" +else + fail "diesel_cli: not installed (cargo install diesel_cli --no-default-features --features mysql,sqlite)" +fi + +if command -v node >/dev/null 2>&1; then + node_major="$(node --version | sed -E 's/^v([0-9]+).*/\1/')" + if [ "${node_major}" -ge 20 ]; then + pass "node: $(node --version) (>= 20 required by Nx 21 / Vite 6)" + else + fail "node: $(node --version) is too old — Node 20+ required" + fi +else + fail "node: not installed (Node 20+ required for the frontend)" +fi + +echo "" +echo "--- Services ---" + +if mysql_app_reachable; then + pass "mysql: reachable as '${DEV_MYSQL_USER}' on database '${DEV_MYSQL_DB}'" + + migration_dirs="$(find "${root_dir}/_database/sql/migrations" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')" + applied="$(mysql_app -N -e "SELECT COUNT(*) FROM __diesel_schema_migrations" 2>/dev/null || echo 0)" + if [ "${applied}" -ge "${migration_dirs}" ] && [ "${applied}" -gt 0 ]; then + pass "migrations: ${applied} applied (${migration_dirs} in _database/sql/migrations)" + elif [ "${applied}" -gt 0 ]; then + fail "migrations: only ${applied}/${migration_dirs} applied — run 'diesel migration run'" + else + fail "migrations: none applied — run 'diesel migration run'" + fi + + role_count="$(mysql_app -N -e "SELECT COUNT(*) FROM user_roles" 2>/dev/null || echo 0)" + if [ "${role_count}" -ge 3 ]; then + pass "seed: user_roles has ${role_count} rows (user/mod/admin present)" + else + fail "seed: user_roles has ${role_count} rows — account creation needs the 'user' role (re-run bootstrap)" + fi + + badge_count="$(mysql_app -N -e "SELECT COUNT(*) FROM badges" 2>/dev/null || echo 0)" + if [ "${badge_count}" -gt 0 ]; then + pass "seed: badges has ${badge_count} rows" + else + warn_check "seed: badges table is empty (cosmetic; re-run bootstrap to fill)" + fi + + demo_count="$(mysql_app -N -e "SELECT COUNT(*) FROM users WHERE username='${DEMO_USERNAME}'" 2>/dev/null || echo 0)" + if [ "${demo_count}" -gt 0 ]; then + pass "demo user: '${DEMO_USERNAME}' exists" + else + warn_check "demo user: '${DEMO_USERNAME}' not created yet (./script/bootstrap/seed_demo_user.sh, backend must be running)" + fi +else + fail "mysql: cannot connect as '${DEV_MYSQL_USER}'@'${DEV_MYSQL_HOST}' to '${DEV_MYSQL_DB}' (is MySQL running? run bootstrap)" +fi + +if redis_reachable; then + pass "redis: PONG" +else + fail "redis: not reachable on localhost (the backend's r2d2 pool connects eagerly at boot)" +fi + +echo "" +echo "--- Backend ---" + +if [ -f "${SECRETS_ENV_FILE}" ]; then + pass "secrets env: ${SECRETS_ENV_FILE#"${root_dir}"/} exists" +else + fail "secrets env: missing — the server aborts at boot on ~19 required vars (re-run bootstrap)" +fi + +if [ -x "${root_dir}/target/debug/storyteller-web" ] || [ -x "${root_dir}/target/release/storyteller-web" ]; then + pass "binary: storyteller-web is built" +else + warn_check "binary: storyteller-web not built yet (bootstrap builds it; cargo run will build on demand)" +fi + +if backend_reachable; then + pass "server: ${DEV_BACKEND_URL}/_status responds" +else + warn_check "server: not running (./script/bootstrap/run_backend_dev.sh)" +fi + +echo "" +echo "--- Frontend ---" + +if [ -d "${root_dir}/frontend/node_modules" ]; then + pass "frontend: node_modules present" +else + warn_check "frontend: node_modules missing (cd frontend && npm install)" +fi + +if [ -e "${root_dir}/frontend/pnpm-lock.yaml" ] || [ -d "${root_dir}/frontend/node_modules/.pnpm" ]; then + fail "frontend: stale pnpm artifacts detected (see frontend/README.md — the workspace uses npm)" +else + pass "frontend: no stale pnpm artifacts" +fi + +echo "" +echo "Summary: ${failures} failure(s), ${warnings} warning(s)." +if [ "${failures}" -gt 0 ]; then + echo "Fix failures with: ./script/bootstrap/bootstrap_dev_stack.sh (safe to re-run)" + exit 1 +fi diff --git a/script/bootstrap/run_backend_dev.sh b/script/bootstrap/run_backend_dev.sh new file mode 100755 index 0000000000..f2babc89b0 --- /dev/null +++ b/script/bootstrap/run_backend_dev.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Run the storyteller-web backend for local development (binds 0.0.0.0:12345). +# +# Must run from a bootstrapped environment (./script/bootstrap/bootstrap_dev_stack.sh). +# Always runs from the repo root: the server's config search path +# (crates/service/web/storyteller_web/config/*.env) and its includes/ defaults +# are cwd-relative. +set -euo pipefail + +root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +source "${root_dir}/script/bootstrap/common.sh" + +SECRETS_ENV_FILE="${root_dir}/crates/service/web/storyteller_web/config/storyteller-web.development-secrets.env" + +mysql_app_reachable || die "MySQL is not reachable as '${DEV_MYSQL_USER}'. Run ./script/bootstrap/bootstrap_dev_stack.sh first." +redis_reachable || die "Redis is not reachable. Run ./script/bootstrap/bootstrap_dev_stack.sh first." +[ -f "${SECRETS_ENV_FILE}" ] || die "Missing ${SECRETS_ENV_FILE}. Run ./script/bootstrap/bootstrap_dev_stack.sh first." + +ensure_cargo_on_path +cd "${root_dir}" + +# SERVER_ENVIRONMENT defaults to Development when unset (bootstrap.rs), and the +# dev config files are picked up from the search path automatically. +# SQLX_OFFLINE only affects compile-time query checking, never runtime. +log "Starting storyteller-web on ${DEV_BACKEND_URL} (Ctrl-C to stop)..." +exec env SQLX_OFFLINE=true cargo run -p storyteller-web --bin storyteller-web diff --git a/script/bootstrap/seed_demo_user.sh b/script/bootstrap/seed_demo_user.sh new file mode 100755 index 0000000000..6fb396f1be --- /dev/null +++ b/script/bootstrap/seed_demo_user.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# Create (or verify) the local demo user via the running backend, then verify +# that login works. Idempotent: an already-existing user is a success. +# +# Requires the backend to be up (./script/bootstrap/run_backend_dev.sh) — this +# script waits up to --wait-seconds (default 60) for it. +# +# The account is created through POST /v1/create_account rather than raw SQL so +# it exercises the real signup path (bcrypt hash, session, firehose row) and +# stays correct if the users schema evolves. +# +# Override the credentials via DEMO_USERNAME / DEMO_PASSWORD / DEMO_EMAIL. +# Username rules: 3-16 chars of [A-Za-z0-9_-], not on the reserved list +# (includes/binary_includes/usernames/reserved_usernames.txt); password >= 6 chars. +set -euo pipefail + +root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +source "${root_dir}/script/bootstrap/common.sh" + +WAIT_SECONDS=60 +for arg in "$@"; do + case "${arg}" in + --wait-seconds=*) WAIT_SECONDS="${arg#*=}" ;; + --help|-h) sed -n '2,15p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) die "Unknown option: ${arg} (try --help)" ;; + esac +done + +command -v jq >/dev/null 2>&1 || die "jq is required (apt-get install jq)." + +log "Waiting for backend at ${DEV_BACKEND_URL} (up to ${WAIT_SECONDS}s)..." +deadline=$((SECONDS + WAIT_SECONDS)) +until backend_reachable; do + if [ "${SECONDS}" -ge "${deadline}" ]; then + die "Backend not reachable at ${DEV_BACKEND_URL}/_status. Start it with ./script/bootstrap/run_backend_dev.sh" + fi + sleep 2 +done +log "Backend is up." + +log "Creating account '${DEMO_USERNAME}'..." +create_response="$(curl -sS -X POST "${DEV_BACKEND_URL}/v1/create_account" \ + -H "Content-Type: application/json" \ + -d "$(jq -n \ + --arg u "${DEMO_USERNAME}" \ + --arg p "${DEMO_PASSWORD}" \ + --arg e "${DEMO_EMAIL}" \ + '{username: $u, password: $p, password_confirmation: $p, email_address: $e}')")" + +if [ "$(echo "${create_response}" | jq -r '.success // false')" = "true" ]; then + log "Account created." +else + error_code="$(echo "${create_response}" | jq -r '.error_code // .error_code_str // empty')" + case "${error_code}" in + *[Uu]sername[Tt]aken*|*[Ee]mail[Tt]aken*) + log "Account already exists (${error_code}) — continuing." + ;; + *) + die "Account creation failed: ${create_response}" + ;; + esac +fi + +log "Verifying login..." +login_response="$(curl -sS -X POST "${DEV_BACKEND_URL}/v1/login" \ + -H "Content-Type: application/json" \ + -d "$(jq -n \ + --arg u "${DEMO_USERNAME}" \ + --arg p "${DEMO_PASSWORD}" \ + '{username_or_email: $u, password: $p}')")" + +if [ "$(echo "${login_response}" | jq -r '.success // false')" != "true" ]; then + die "Login verification failed: ${login_response}" +fi +if [ -z "$(echo "${login_response}" | jq -r '.signed_session // empty')" ]; then + die "Login succeeded but returned no signed_session: ${login_response}" +fi + +log "Demo user is ready." +log " Username: ${DEMO_USERNAME}" +log " Password: ${DEMO_PASSWORD}" +log "Log in at the webapp /login page, or via curl:" +log " curl -c /tmp/cookies.txt -X POST ${DEV_BACKEND_URL}/v1/login -H 'Content-Type: application/json' -d '{\"username_or_email\":\"${DEMO_USERNAME}\",\"password\":\"${DEMO_PASSWORD}\"}'"