Skip to content

Conversation

@loewenheim
Copy link
Contributor

@loewenheim loewenheim commented Nov 3, 2025

This adds connection and respnose timeout config options for both Single and Cluster Redis clients.

Unfortunately, since redis and deadpool-redis don't provide enough knobs to tweak the default clients, the easiest way to accomplish this is by more or less reinventing deadpool_redis::Manager and deadpool_redis::Cluster::Manager, but with timeouts applied. Consequently the deadpool_redis dependency becomes unused.

TBD: default values and docstrings for the new timeouts.

Fixes INGEST-621.

@linear
Copy link

linear bot commented Nov 3, 2025

Comment on lines -37 to -40

/// An error that occurs when configuring Redis.
#[error("failed to configure redis: {0}")]
ConfigError(#[from] ConfigError),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variant and From impl were already unused.

/// will fail with a timeout error. This setting helps prevent indefinite
/// blocking when the pool is exhausted.
pub wait_timeout: Option<u64>,
pub connection_timeout: Option<u64>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is arguably redundant with create_timeout: This timeout is used during connection creation in the redis crate, whereas create_timeout is used in the Managers' implementations of create. Both have the effect of putting a time limit on establishing a connection, but one is at the redis level and one at the deadpool level.

@loewenheim
Copy link
Contributor Author

loewenheim commented Nov 4, 2025

Updates:

  • As noted in feat(redis): Add response timeout #5329 (comment), connection_timeout overlaps with the existing create_timeout. I don't think it's worth having both of them, so I removed it.
  • response_timeout is used in redis as a tokio timeout around the whole request (as opposed to a read timeout on a socket). In normal operation the maximum time for event_processing.rate_limiting is around 4s. Therefore I put 6s as the default value.
  • Timeout errors now set the detach flag on TrackedConnection. This will lead to connections that time out being recycled.

@loewenheim loewenheim marked this pull request as ready for review November 4, 2025 14:19
@loewenheim loewenheim requested a review from a team as a code owner November 4, 2025 14:20
@loewenheim loewenheim changed the title feat(redis): Add connection and response timeouts feat(redis): Add response timeout Nov 4, 2025
Copy link
Member

@tobias-wilfert tobias-wilfert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice, only there seems to be a merge conflict in the Changelog.

This adds connection and respnose timeout config options for both Single
and Cluster Redis clients.

Unfortunately, since `redis` and `deadpool-redis` don't provide enough
knobs to tweak the default clients, the easiest way to accomplish this
is by more or less reinventing `deadpool_redis::Manager` and
`deadpool_redis::Cluster::Manager`, but with timeouts applied.
Consequently the `deadpool_redis` dependency becomes unused.

TBD: default values and docstrings for the new timeouts.
@loewenheim loewenheim force-pushed the sebastian/redis-timeouts branch from 5a27321 to 947d4dc Compare November 4, 2025 14:42
create_timeout: Some(3),
recycle_timeout: Some(2),
wait_timeout: None,
response_timeout: Some(6),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems very low for the total command time, how did you arrive at this number?

I suspect that some cardinality limiting commands take longer than this.

/// An error leads to the connection being detached if it is either unrecoverable or a timeout.
fn should_be_detached(error: &RedisError) -> bool {
error.is_unrecoverable_error()
error.is_unrecoverable_error() || error.is_timeout()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could add a small justification on to why the timeout is listed here.

Comment on lines +211 to +213
Err(deadpool::managed::RecycleError::message(
"Invalid PING response",
))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Err(deadpool::managed::RecycleError::message(
"Invalid PING response",
))
Err(RecycleError::message("Invalid PING response"))

RecycleError is already imported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants