Add PgBouncer support via NullPool configuration #18643

zzstoatzz · 2025-08-04T21:06:22Z

Summary

This PR implements PgBouncer support for Prefect by allowing pool_size to be set to None, which configures SQLAlchemy to use NullPool. This addresses the double pooling issue when using external connection poolers like PgBouncer.

Changes

1. Database Configuration

Modified pool_size setting in src/prefect/settings/models/server/database.py to accept Optional[int]
Updated the field description to mention NullPool usage when set to None

2. Engine Creation

Updated AsyncPostgresConfiguration in src/prefect/server/database/configurations.py:
- When sqlalchemy_pool_size is None, use SQLAlchemy's NullPool
- Only pass pool-related parameters (pool_size, pool_timeout, max_overflow) when using regular pools
- NullPool doesn't accept these parameters, so they're excluded to avoid errors

3. Tests

Added TestNullPoolConfiguration class to tests/server/database/test_dependencies.py
Tests verify that NullPool is used when pool_size is set to None
Tests use unique database names and clear engine cache to ensure isolation

4. Documentation

Added comprehensive "Using PgBouncer for connection pooling" section to docs/v3/advanced/database-maintenance.mdx
Includes Docker setup example, configuration instructions, and troubleshooting tips
Added verified working configuration based on actual testing

Testing

I've tested this implementation by:

Setting up a real PgBouncer instance with Docker
Verifying that NullPool is correctly used when pool_size=None
Testing PgBouncer transaction mode behavior
Confirming that connection pooling is handled by PgBouncer, not SQLAlchemy

Example Usage

from prefect.server.database.configurations import AsyncPostgresConfiguration
from prefect.settings import temporary_settings, PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE

# Use temporary settings to ensure pool_size is None
with temporary_settings({PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE: None}):
    config = AsyncPostgresConfiguration(
        connection_url="postgresql+asyncpg://user:pass@pgbouncer:6432/prefect",
        sqlalchemy_pool_size=None,      # Uses NullPool
        statement_cache_size=0          # Required for transaction mode
    )

🤖 Generated with Claude Code

This adds support for using external connection poolers like PgBouncer by allowing pool_size to be set to None/null, which configures SQLAlchemy to use NullPool. Key changes: - Allow pool_size setting to accept None/null values - Use NullPool when pool_size is None, skipping pool-related parameters - Add documentation on PgBouncer setup and configuration - Add tests for NullPool configuration Usage: export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=null export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0 export PREFECT_SERVER_DATABASE_CONNECTION_URL='postgresql+asyncpg://user:pass@pgbouncer:6432/db' Addresses #18616

codspeed-hq · 2025-08-04T21:10:48Z

CodSpeed Performance Report

Merging #18643 will not alter performance

_{Comparing support-pgbouncer-nullpool (729ed7d) with main (e825a0f)}

Summary

✅ 2 untouched benchmarks

- Add comprehensive PgBouncer setup guide - Include Docker compose example with correct environment variables - Document how to configure Prefect with NullPool for PgBouncer - Add troubleshooting section and verified working configuration - Fix environment variable names (DB_HOST instead of DATABASES_HOST) - Add note about setting pool_size to None programmatically 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

cicdw · 2025-08-06T14:33:37Z

src/prefect/server/database/configurations.py

        self.sqlalchemy_pool_size: Optional[int] = (
            sqlalchemy_pool_size
-            or get_current_settings().server.database.sqlalchemy.pool_size
+            if sqlalchemy_pool_size is not None


do we ever use this as init kwargs within the codebase? I think it would be weird if passing None as an init kwarg is the one value that isn't respected, but I don't think we actually use these init kwargs anyway

github-actions · 2025-08-20T16:01:40Z