Outage Playbook: Database Connection Pool Exhaustion

Classification

Type: Infrastructure — Database Severity Tiers:

• P1: >50% of API requests returning 500 errors for >5 minutes; customer-facing impact confirmed
• P2: >20% of API requests returning 500 errors; degraded performance but partial functionality
• P3: Connection pool warnings in logs; no customer-facing impact yet; preventive investigation

Trigger Conditions: Alert "db-pool-exhaustion" fires when available connections drop below 10% of max pool size for >60 seconds

Roles

| Role | Responsibility | Default Owner | |------|----------------|---------------| | Incident Commander | Coordinates response, makes severity calls, owns communication | On-call Engineering Manager | | Technical Lead | Diagnoses root cause, executes remediation steps | On-call Backend Engineer | | Communications Lead | Updates status page, notifies stakeholders | On-call Engineering Manager (doubles) |

Detection & Triage (0–5 min)

1. [IC] Acknowledge the PagerDuty alert within 2 minutes. Open #incidents Slack channel.
2. [Tech Lead] Verify the alert is real — check Grafana dashboard "DB Pool Health": https://grafana.internal/d/db-pool
3. [IC] Classify severity using the tier definitions above. Declare incident in #incidents.

Response Steps

1. [Tech Lead] Check active connections: SELECT count(*) FROM pg_stat_activity WHERE state = 'active'; — if >90% of max_connections, confirm pool exhaustion.
2. [Tech Lead] Identify long-running queries: SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;
3. [Tech Lead] Kill queries running >5 minutes that are not critical batch jobs: SELECT pg_terminate_backend(pid);
4. [IC] If killing queries restores pool within 5 minutes, monitor for 15 minutes and proceed to resolution checklist.
5. [Tech Lead] If pool remains exhausted, restart the application service: kubectl rollout restart deployment/api-server -n production

Resolution Checklist

• [ ] Connection pool utilization below 60%
• [ ] Error rate returned to baseline (<0.1%)
• [ ] Status page updated: Resolved
• [ ] PagerDuty incident resolved
• [ ] Post-mortem scheduled within 48 hours