Demo API

Summary

A 47-minute outage of /v1/payments/charge affected all checkout flows on 14 February 2026 between 14:03 and 14:50 UTC. Approximately 3,200 payment requests returned HTTP 503. The root cause was connection pool exhaustion on the primary PostgreSQL replica, triggered by a 3× traffic spike during a flash sale.

Timeline

• 14:03 UTC — First failed health check (HTTP 503). Latency spiked to 12,400 ms.
• 14:06 UTC — Consecutive failure threshold reached; incident opened automatically.
• 14:06 UTC — Email alerts sent to subscribers and on-call engineer.
• 14:31 UTC — Root cause identified: connection pool limit of 100 exhausted.
• 14:48 UTC — Connection pool limit increased to 500; service began recovering.
• 14:50 UTC — All health checks passing. Incident resolved.

Root Cause

A flash sale drove checkout traffic to 3× baseline. The payments service exhausted its PostgreSQL connection pool (limit: 100). Incoming requests queued, hit the 30-second timeout, and returned HTTP 503. The connection pool limit had not been reviewed since the service's initial deployment 14 months prior.

Impact

• 47 minutes of complete unavailability for all checkout flows
• ~3,200 failed payment requests
• Estimated revenue impact: €18,400

What We're Doing About It

1. Increased connection pool limit: 100 → 500
2. Added pool saturation alert at 80% utilisation
3. Evaluating PgBouncer as a long-term connection pooler
4. Load testing scheduled before next promotional event