Cap MySQL statement cache and connection lifetime in storyteller-web (#65)

## Problem

Production `storyteller-web` and the job workers started returning fast
500s with MySQL error 1461:

```
Can't create more than max_prepared_stmt_count statements (current value: 16382)
```

A customer saw this on `GET /v1/prompt/batch`. Errors rose to ~316/hour
on 2026-10-09 and stopped after a rolling restart at ~17:50 UTC. The
restart only clears the caches, so this will happen again under load.

## Cause

sqlx (0.7.4) caches up to **100** prepared statements per connection by
default, and each cached statement stays open on the server. At current
scale, `storyteller-web` can hold:

| Setting                       | Before   | After   |
|-------------------------------|----------|---------|
| Replicas                      | 7        | 7       |
| `MYSQL_MAX_CONNECTIONS`       | 50       | 50      |
| Statement cache / connection  | 100      | 25      |
| Max statements held           | ~35,000  | ~8,750  |
| Server cap                    | 16,382   | 16,382  |

The ~70 call sites that use `QueryBuilder` (variable-length `IN (...)`
lists) produce many different SQL strings, which fill the caches faster.

## Changes

`crates/service/web/storyteller_web/src/configs/connect_to_database.rs`:

- **`statement_cache_capacity`** defaults to 25, configurable with
`MYSQL_STATEMENT_CACHE_CAPACITY`.
- **`max_lifetime`** defaults to 10 minutes (sqlx's default is 30),
configurable with `MYSQL_MAX_LIFETIME_SECONDS`. A query cancelled
between prepare and close (e.g. the client disconnects) leaves its
statement open on the server until the connection closes. Recycling
connections sooner frees those statements sooner.
- Both values are logged with the existing pool settings at startup.

Job workers are left unchanged: their pools are hardcoded to 2
connections, so each holds at most 200 statements.

## Notes

- If replicas or `MYSQL_MAX_CONNECTIONS` increase, keep `replicas ×
MYSQL_MAX_CONNECTIONS × MYSQL_STATEMENT_CACHE_CAPACITY` well under
16,382.
- Follow-up (not in this PR): use `.persistent(false)` on hot dynamic
`QueryBuilder` queries such as `batch_get_prompts`.

## Testing

- `SQLX_OFFLINE=true cargo check -p storyteller-web` passes.
- After deploy, look for the new values in the `Building MySQL pool:`
startup log line, and confirm no new `max_prepared_stmt_count` errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

crates/service/web/storyteller_web/src/configs/connect_to_database.rs

@@ -1,10 +1,11 @@
+use std::str::FromStr;
 use std::time::Duration;
 
 use log::info;
 
 use errors::AnyhowResult;
 use shared_env_var_config::mysql::env_get_mysql_connection_string_or_default;
-use sqlx::mysql::MySqlPoolOptions;
+use sqlx::mysql::{MySqlConnectOptions, MySqlPoolOptions};
 use sqlx::MySqlPool;
 
 pub async fn connect_to_database() -> AnyhowResult<MySqlPool> {
@@ -20,20 +21,37 @@ pub async fn connect_to_database() -> AnyhowResult<MySqlPool> {
   // shorter timeout sheds load and surfaces the problem instead of hiding it behind long waits.
   let acquire_timeout_seconds = easyenv::get_env_num::<u64>("MYSQL_ACQUIRE_TIMEOUT_SECONDS", 10)?;
 
+  // NB: Every cached prepared statement stays open on the server until its connection evicts it
+  // or closes, and the server caps the total across ALL clients (`max_prepared_stmt_count`,
+  // 16382 on the managed DB). sqlx's default of 100 per connection let `replicas *
+  // MYSQL_MAX_CONNECTIONS * 100` blow past that cap, failing new prepares with MySQL error 1461.
+  // Keep `replicas * MYSQL_MAX_CONNECTIONS * MYSQL_STATEMENT_CACHE_CAPACITY` well under it.
+  let statement_cache_capacity = easyenv::get_env_num::<usize>("MYSQL_STATEMENT_CACHE_CAPACITY", 25)?;
+
+  // NB: Recycling connections releases any server-side statements orphaned by cancelled queries
+  // (eg. a client disconnect between prepare and close). sqlx's default lifetime is 30 minutes.
+  let max_lifetime_seconds = easyenv::get_env_num::<u64>("MYSQL_MAX_LIFETIME_SECONDS", 600)?;
+
   // NB: Log the settings before building/connecting so a connect failure has context. We do NOT
   // log the connection string (it contains credentials).
   info!(
-    "Building MySQL pool: max_connections={}, min_connections={}, acquire_timeout_seconds={}",
+    "Building MySQL pool: max_connections={}, min_connections={}, acquire_timeout_seconds={}, \
+      statement_cache_capacity={}, max_lifetime_seconds={}",
     max_connections, min_connections, acquire_timeout_seconds,
+    statement_cache_capacity, max_lifetime_seconds,
   );
 
+  let connect_options = MySqlConnectOptions::from_str(&db_connection_string)?
+      .statement_cache_capacity(statement_cache_capacity);
+
   let pool_options = MySqlPoolOptions::new()
       .max_connections(max_connections)
       .min_connections(min_connections)
-      .acquire_timeout(Duration::from_secs(acquire_timeout_seconds));
+      .acquire_timeout(Duration::from_secs(acquire_timeout_seconds))
+      .max_lifetime(Some(Duration::from_secs(max_lifetime_seconds)));
 
   let pool = pool_options
-      .connect(&db_connection_string)
+      .connect_with(connect_options)
       .await?;
 
   Ok(pool)