Skip to content

Conversation

@wardlican
Copy link
Contributor

Why are the changes needed?

This implements the logic for dynamically and evenly distributing buckets based on the number of AMS nodes.

Close #3921 .

Brief change log

AmsAssignService Functionality

  1. Periodic Allocation: Performs bucket ID allocation every 30 seconds (configurable)
  2. Node Change Detection: Detects new and offline nodes
  3. Incremental Allocation: Minimizes migration, preserving existing node buckets as much as possible
  4. Load Balancing: Distributes buckets evenly across all live nodes
  5. Offline Node Handling: Detects offline nodes based on a timeout mechanism and reallocates their buckets
  6. Master Node Only: Only the leader node executes the allocation logic.

ZkBucketAssignStore Features

  1. Storage Implementation: Stores bucket assignment information based on ZooKeeper.
  2. Data Storage:
    • List of bucket IDs for each node (JSON format)
    • Last update timestamp for each node
  3. CRUD Operations:
    • Save/Update Node Assignments
    • Query Node Assignments
    • Delete Node Assignments
    • Query All Node Assignments
  4. Access Control: Only the leader node can save/delete assignment information.
  5. Path Structure: /{cluster}/amoro/ams/bucket-assignments/{nodeKey}/assignments and /{cluster}/amoro/ams/bucket-assignments/{nodeKey}/last-update-time

two classes that work together to implement dynamic bucket ID allocation and storage in master-slave mode.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@codecov-commenter
Copy link

codecov-commenter commented Nov 12, 2025

Codecov Report

❌ Patch coverage is 71.01770% with 131 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.22%. Comparing base (4382d55) to head (f2ecc06).
⚠️ Report is 31 commits behind head on master.

Files with missing lines Patch % Lines
...java/org/apache/amoro/server/AmsAssignService.java 78.16% 35 Missing and 15 partials ⚠️
...a/org/apache/amoro/server/ZkBucketAssignStore.java 66.66% 34 Missing and 5 partials ⚠️
...org/apache/amoro/server/AmoroServiceContainer.java 11.53% 22 Missing and 1 partial ⚠️
...apache/amoro/server/HighAvailabilityContainer.java 70.68% 13 Missing and 4 partials ⚠️
...a/org/apache/amoro/properties/AmsHAProperties.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3922      +/-   ##
============================================
+ Coverage     29.00%   29.22%   +0.22%     
- Complexity     3850     3915      +65     
============================================
  Files           631      630       -1     
  Lines         50770    50997     +227     
  Branches       6543     6518      -25     
============================================
+ Hits          14724    14906     +182     
- Misses        34989    35039      +50     
+ Partials       1057     1052       -5     
Flag Coverage Δ
core 29.22% <71.01%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask]: add AmsAssignService to implement balanced bucket allocation in master-slave mode.

2 participants