Connection pools are a latency budget

The p99 latency of an API jumps from 80 ms to 900 ms. Database CPU is at 25 percent, no query is slow in the logs, and the pool is set to 20 connections. The missing 800 ms is queue time: requests waiting for a connection before their query was ever sent.

Four phases of every query

A pooled request goes through acquire, wait, execute, and release. Acquire takes a connection if one is free. If not, the request waits until one is returned or the acquire timeout fires. Execute covers the round trip and the query. Release returns the connection, and the next waiter gets it.

Exhaustion is queueing, not failure

Until the acquire timeout, an exhausted pool is not an error. It is a queue. With a fixed pool and a fixed query duration, the maximum sustainable rate is the pool size divided by the average hold time. At 20 connections and a 50 ms query, the ceiling is roughly 400 queries per second — an illustrative figure, since real hold times vary. As arrival rate approaches that ceiling, wait time grows faster than utilization, which is why the p99 degrades long before the average does.

The useful mental model is Little’s law: concurrency equals arrival rate multiplied by time in the system. If you need 300 requests per second and each holds a connection for 50 ms, you need about 15 concurrent connections just for the database work, before any safety margin.

Why more connections can be worse

Each PostgreSQL connection is a backend process with its own memory and scheduling. More of them mean more context switches, more lock and snapshot contention, and a larger working set competing for CPU cache. At some concurrency the database spends its time switching between queries rather than completing them. The result is lower throughput and a longer tail, even though every individual query looks fine.

Pool size versus max_connections

These are different limits at different layers. The pool size belongs to one application process. max_connections belongs to the database and counts every backend from every client.

Setting	Bounds	Failure when exceeded
App pool max_size	concurrent queries per process	acquire waits, then timeout
Pool acquire timeout	queue wait per request	500-style error under load
Statement timeout	execution time	cancelled query
max_connections	total backends in the cluster	connection refused
PgBouncer pool size	server connections per user/database	client queueing

A fleet multiplies the first row. Twenty connections per worker across eight workers and four replicas is 640 potential backends, which can exceed a default max_connections long before traffic is high. Autoscaling makes this worse because new replicas add pools faster than the database gains capacity.

import asyncpg

async def fetch_ready_jobs(dsn: str) -> list[asyncpg.Record]:
    pool = await asyncpg.create_pool(dsn, min_size=4, max_size=16)
    try:
        async with pool.acquire(timeout=2.0) as conn:
            return await conn.fetch(
                "SELECT id FROM jobs WHERE state = $1 LIMIT 50",
                "ready",
            )
    finally:
        await pool.close()

Creating a pool per call is wrong for a service, but the shape of the timeout is right: a bounded wait to acquire, a separate command timeout, and a finally that releases the connection on every path.

What changes under load

Long transactions hold connections after the last row is read, so a pool sized for query time is undersized for transaction time. A connection that is checked out but idle in a transaction is invisible to query metrics and fully visible to the queue. Replicas and workers multiply pool size against a fixed database limit. Read replicas help only if the routing sends work there and the pool boundaries are separate.

What to carry forward

Start from the deadline, not from a default pool size. Measure acquire wait separately from query time, because they fail differently. Cap the product of processes, replicas, and pool size below the database’s connection limit with room for administration. Then increase concurrency only while throughput rises and the tail stays flat.