## Problem
Production `storyteller-web` and the job workers started returning fast
500s with MySQL error 1461:
```
Can't create more than max_prepared_stmt_count statements (current value: 16382)
```
A customer saw this on `GET /v1/prompt/batch`. Errors rose to ~316/hour
on 2026-10-09 and stopped after a rolling restart at ~17:50 UTC. The
restart only clears the caches, so this will happen again under load.
## Cause
sqlx (0.7.4) caches up to **100** prepared statements per connection by
default, and each cached statement stays open on the server. At current
scale, `storyteller-web` can hold:
| Setting | Before | After |
|-------------------------------|----------|---------|
| Replicas | 7 | 7 |
| `MYSQL_MAX_CONNECTIONS` | 50 | 50 |
| Statement cache / connection | 100 | 25 |
| Max statements held | ~35,000 | ~8,750 |
| Server cap | 16,382 | 16,382 |
The ~70 call sites that use `QueryBuilder` (variable-length `IN (...)`
lists) produce many different SQL strings, which fill the caches faster.
## Changes
`crates/service/web/storyteller_web/src/configs/connect_to_database.rs`:
- **`statement_cache_capacity`** defaults to 25, configurable with
`MYSQL_STATEMENT_CACHE_CAPACITY`.
- **`max_lifetime`** defaults to 10 minutes (sqlx's default is 30),
configurable with `MYSQL_MAX_LIFETIME_SECONDS`. A query cancelled
between prepare and close (e.g. the client disconnects) leaves its
statement open on the server until the connection closes. Recycling
connections sooner frees those statements sooner.
- Both values are logged with the existing pool settings at startup.
Job workers are left unchanged: their pools are hardcoded to 2
connections, so each holds at most 200 statements.
## Notes
- If replicas or `MYSQL_MAX_CONNECTIONS` increase, keep `replicas ×
MYSQL_MAX_CONNECTIONS × MYSQL_STATEMENT_CACHE_CAPACITY` well under
16,382.
- Follow-up (not in this PR): use `.persistent(false)` on hot dynamic
`QueryBuilder` queries such as `batch_get_prompts`.
## Testing
- `SQLX_OFFLINE=true cargo check -p storyteller-web` passes.
- After deploy, look for the new values in the `Building MySQL pool:`
startup log line, and confirm no new `max_prepared_stmt_count` errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>