Alerts triggered: payment-service error rate >5% across all regions
SystemConfirmed 503 errors on /api/payments/charge endpoint. Upstream dependency check in progress.
Lisa ChenRoot cause identified: connection pool exhaustion on payment-db-primary. Max connections reached at 500/500.
Ravi PatelDeploying hotfix to increase pool size to 750 and terminate idle connections. ETA 5 minutes.
Ravi PatelI'm seeing the alerts. Pulling up dashboards now. Error rate is climbing fast — 12% and rising.
Declaring SEV-1. All hands on deck. @Ravi can you check the payment service pods and DB connections?
On it. Seeing high connection wait times on the primary DB. Looks like pool exhaustion. Checking max connections config now.
Confirmed. Pool is maxed at 500 connections. We had a spike from the batch job that started at 14:20. Preparing a config change to bump the limit and kill idle connections.
I'll draft the customer status page update. Ravi, what's the ETA for the fix?
5 minutes to deploy the config change. Should see recovery within 2 minutes after that.