Files
artcraft-services/crates
Brandon ThomasandClaude Opus 5.5 c489e8c650 Cap MySQL statement cache and connection lifetime in storyteller-web (#65)
## Problem

Production `storyteller-web` and the job workers started returning fast
500s with MySQL error 1461:

```
Can't create more than max_prepared_stmt_count statements (current value: 16382)
```

A customer saw this on `GET /v1/prompt/batch`. Errors rose to ~316/hour
on 2026-10-09 and stopped after a rolling restart at ~17:50 UTC. The
restart only clears the caches, so this will happen again under load.

## Cause

sqlx (0.7.4) caches up to **100** prepared statements per connection by
default, and each cached statement stays open on the server. At current
scale, `storyteller-web` can hold:

| Setting                       | Before   | After   |
|-------------------------------|----------|---------|
| Replicas                      | 7        | 7       |
| `MYSQL_MAX_CONNECTIONS`       | 50       | 50      |
| Statement cache / connection  | 100      | 25      |
| Max statements held           | ~35,000  | ~8,750  |
| Server cap                    | 16,382   | 16,382  |

The ~70 call sites that use `QueryBuilder` (variable-length `IN (...)`
lists) produce many different SQL strings, which fill the caches faster.

## Changes

`crates/service/web/storyteller_web/src/configs/connect_to_database.rs`:

- **`statement_cache_capacity`** defaults to 25, configurable with
`MYSQL_STATEMENT_CACHE_CAPACITY`.
- **`max_lifetime`** defaults to 10 minutes (sqlx's default is 30),
configurable with `MYSQL_MAX_LIFETIME_SECONDS`. A query cancelled
between prepare and close (e.g. the client disconnects) leaves its
statement open on the server until the connection closes. Recycling
connections sooner frees those statements sooner.
- Both values are logged with the existing pool settings at startup.

Job workers are left unchanged: their pools are hardcoded to 2
connections, so each holds at most 200 statements.

## Notes

- If replicas or `MYSQL_MAX_CONNECTIONS` increase, keep `replicas ×
MYSQL_MAX_CONNECTIONS × MYSQL_STATEMENT_CACHE_CAPACITY` well under
16,382.
- Follow-up (not in this PR): use `.persistent(false)` on hot dynamic
`QueryBuilder` queries such as `batch_get_prompts`.

## Testing

- `SQLX_OFFLINE=true cargo check -p storyteller-web` passes.
- After deploy, look for the new values in the `Building MySQL pool:`
startup log line, and confirm no new `max_prepared_stmt_count` errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:29:20 -04:00
..
2026-09-28 01:07:45 -04:00
2026-09-12 12:23:46 -04:00
…