fix(tiering): Basic stash backpressure #5973
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dranikpg
force-pushed
the
stash-backpressure
branch
from
3956929 to
5774cbb
Compare
Signed-off-by: Vladislav Oleshko <[email protected]>
dranikpg
force-pushed
the
stash-backpressure
branch
from
5774cbb to
4f1283f
Compare
Signed-off-by: Vladislav Oleshko <[email protected]>
dranikpg
force-pushed
the
stash-backpressure
branch
from
68d9d47 to
a549dde
Compare
dranikpg
commented
Oct 30, 2025
| PrimeTable::Cursor offloading_cursor_{}; // where RunOffloading left off | ||
|
|
||
| // Stash operations waiting for completion to throttle | ||
| tiering::EntryMap<::util::fb2::Future<bool>> stash_backpressure_; |
Contributor
Author
There was a problem hiding this comment.
Either a future or... a blocking counter? I don't see how we can easily use something more lightweight
Collaborator
|
Can you explain what this PR fixes/changes? what is the scenario? Please update the PR description and add "before and after" section. |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds backpressure (throttling) support to tiered storage stash operations when memory pressure is high. When the system is actively offloading data (below the offload threshold), SET commands can now wait for stash operations to complete, preventing overwhelming the I/O system.
- Introduces a new
EntryMaptype alias with heterogeneous lookup support for efficient key-value lookups - Adds backpressure mechanism that returns futures for throttling SET operations during active offloading
- Adjusts test configuration parameters for tiering thresholds and write depth
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/server/tiering/entry_map.h | New file introducing EntryMap type with custom hasher and equality for heterogeneous lookup |
| src/server/tiering/small_bins.h | Refactors internal maps to use EntryMap type alias |
| src/server/tiering/small_bins.cc | Updates Delete method to use make_pair with heterogeneous lookup |
| src/server/tiered_storage.h | Updates TryStash signature to return optional backpressure future |
| src/server/tiered_storage.cc | Implements backpressure tracking map and resolution logic in stash/cancel/close operations |
| src/server/string_family.cc | Integrates backpressure waiting into SET command flow |
| tests/dragonfly/tiering_test.py | Adjusts test configuration for tiering thresholds and write depth |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Author
|
@romange updated |
Co-authored-by: Copilot <[email protected]> Signed-off-by: Roman Gershman <[email protected]>
Co-authored-by: Copilot <[email protected]> Signed-off-by: Roman Gershman <[email protected]>
romange
reviewed
Nov 2, 2025
src/server/string_family.cc
Outdated
|
|
||
| // If backpressure was provided, wait with reasonable limit (to avoid client deadlocking). | ||
| if (backpressure) { | ||
| std::move(backpressure)->GetFor(10ms); |
Collaborator
There was a problem hiding this comment.
maybe 5ms to be less intrusive?
romange
previously approved these changes
Nov 2, 2025
romange
approved these changes
Nov 2, 2025
abhijat
pushed a commit
that referenced
this pull request
Nov 4, 2025
Provide basic backpressure for tiered storage stashes Signed-off-by: Abhijat Malviya <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Whenever we are beyond the offloading threshold and a stash operation is requested, we allocate a future and return it to the caller. The future is resolved once the value is stashed or the stash is cancelled. The caller (SET command) waits for the future to be completed in the coordinator, slowing the client down to around the speed of the disk.
Before we had no back-pressure whatsoever, so clients could issue SET commands at execution speed. Not all entries are even enqueued for offloading (remember write depth). Generally, it's difficult to pick an adaptive load without back-pressure. Selecting anything faster than the ingestion speed leads to an OOM