[server] Support generate and execute reblance plan #1452

swuferhong · 2025-08-03T08:10:14Z

Purpose

Linked issue: #1397

Brief change log

Tests

API and Format

Documentation

fluss-rpc/src/main/proto/FlussApi.proto

fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java

fluss-server/src/main/java/org/apache/fluss/server/zk/data/LeaderAndIsr.java

platinumhamburg

@swuferhong Thanks for driving this critical and important work—I have a few comments on this PR.

fluss-server/src/main/java/org/apache/fluss/server/zk/ZooKeeperClient.java

fluss-server/src/main/java/org/apache/fluss/server/zk/data/RebalancePlanJsonSerde.java

platinumhamburg · 2026-01-06T02:36:34Z

fluss-server/src/main/java/org/apache/fluss/server/zk/data/RebalancePlan.java

    public RebalancePlan(
-            RebalanceStatus rebalanceStatus, Map<TableBucket, RebalancePlanForBucket> bucketPlan) {
+            String rebalanceId,
+            RebalanceStatus rebalanceStatus,


The inclusion of RebalanceStatus inside RebalancePlan is confusing to me. From the overall design, it appears that the concepts of Plan and Execution are intentionally separated—an elegant approach that aligns with common software architecture principles. However, in the current implementation, they seem tightly coupled.

Beyond this issue, the name rebalanceId feels ambiguous: does it refer to a planId, a progressId, or is it meant to unify both? This coupling isn’t ideal. For example, after a plan is generated, its execution might fail and be retried, which would break the one-to-one mapping between a plan and its corresponding progress instance.

After a broader review, I realized the original design intent: in fact, RebalanceProgress (or RebalanceTask) is the first-class entity. The Plan is merely an internal, ephemeral artifact—generated once per RebalanceProgress, never reused, never externally observed, and not subject to retries.

Given this understanding, perhaps what should be stored in ZooKeeper isn’t the RebalancePlan class itself, but rather RebalanceProgress, with the plan as an internal field within it. @swuferhong @wuchong What do you think?

@platinumhamburg I agree.

How about this:

CompletableFuture<RebalancePlan> rebalance(List<GoalType> priorityGoals, boolean dryRun); ==> // triggers a rebalance task, and returns the rebalance id, in the future we can make the plan generation async CompletableFuture<String> rebalance(List<GoalType> priorityGoals); // returns a dry run plan for the goals, introduce in future, remove for now. // CompletableFuture<RebalancePlan> rebalancePlan(List<GoalType> priorityGoals); // return optional progress. empty if no rebalance in progress. CompletableFuture<Optional<RebalanceProgress>> listRebalanceProgress(@Nullable String rebalanceId); // RebalanceProgress still contains rebalance id, status, and plan to check the detailed progress per bucket. // In ZK, we should json serialize RebalanceTask to distinguish class `RebalanceProgress` and `RebalancePlan` // because it records the top-level status, but not record per-bucket rebalance status

What do you think @swuferhong ?

I agree. The new api: CompletableFuture<RebalancePlan> rebalancePlan(List<GoalType> priorityGoals); will be introduce in an individual issue.

fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java

fluss-common/src/main/java/org/apache/fluss/config/TableConfig.java

...ver/src/main/java/org/apache/fluss/server/coordinator/rebalance/goal/GoalOptimizerUtils.java

fluss-server/src/main/java/org/apache/fluss/server/coordinator/rebalance/RebalanceManager.java

...rver/src/test/java/org/apache/fluss/server/coordinator/rebalance/goal/GoalOptimizerTest.java

...erver/src/test/java/org/apache/fluss/server/coordinator/rebalance/model/ServerModelTest.java

fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java

fluss-client/src/test/java/org/apache/fluss/client/admin/RebalanceITCase.java

fluss-server/src/main/java/org/apache/fluss/server/coordinator/rebalance/goal/Goal.java

...ver/src/main/java/org/apache/fluss/server/coordinator/rebalance/goal/GoalOptimizerUtils.java

platinumhamburg

Add some comments.

fluss-client/src/test/java/org/apache/fluss/client/admin/RebalanceITCase.java

...ver/src/main/java/org/apache/fluss/server/coordinator/rebalance/goal/GoalOptimizerUtils.java

...java/org/apache/fluss/server/coordinator/rebalance/goal/ReplicaDistributionAbstractGoal.java

fluss-server/src/main/java/org/apache/fluss/server/coordinator/rebalance/model/BucketModel.java

...-server/src/main/java/org/apache/fluss/server/coordinator/rebalance/ReplicaReassignment.java

...ink-common/src/main/java/org/apache/fluss/flink/procedure/ListRebalanceProcessProcedure.java

...nk/fluss-flink-common/src/main/java/org/apache/fluss/flink/procedure/RebalanceProcedure.java

fluss-rpc/src/main/proto/FlussApi.proto

...s-server/src/main/java/org/apache/fluss/server/coordinator/rebalance/model/ClusterModel.java

fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorEventProcessor.java

...java/org/apache/fluss/server/coordinator/rebalance/goal/ReplicaDistributionAbstractGoal.java

...rc/main/java/org/apache/fluss/server/coordinator/rebalance/goal/ReplicaDistributionGoal.java

fluss-server/src/main/java/org/apache/fluss/server/coordinator/rebalance/model/ServerModel.java

fluss-server/src/main/java/org/apache/fluss/server/utils/ServerRpcMessageUtils.java

.../src/main/java/org/apache/fluss/server/coordinator/statemachine/TableBucketStateMachine.java

platinumhamburg · 2026-01-08T01:36:21Z

fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorEventProcessor.java

+                (replica) -> {
+                    // For rebalance case. the replica state already set to null in method
+                    // stopRemovedReplicasOfReassignedBucket. so we need not reset it again.
+                    if (coordinatorContext.getReplicaState(replica) != null) {


It seems that null is a very ambiguous state—would it be better to define a clear, explicit state for use in rebalance scenarios?

null is not define only for reblance. NonExist will change the state in coordinatorContext to null.

swuferhong marked this pull request as draft August 3, 2025 08:11

swuferhong force-pushed the rebalance-plan branch 9 times, most recently from 061ed6b to 6bf4aca Compare August 6, 2025 03:54

swuferhong force-pushed the rebalance-plan branch from 6bf4aca to d67fc39 Compare August 29, 2025 08:42

polyzos force-pushed the main branch 3 times, most recently from d88c76c to 434a4f4 Compare August 31, 2025 15:13

swuferhong force-pushed the rebalance-plan branch from d67fc39 to 780e238 Compare September 3, 2025 07:04

swuferhong marked this pull request as ready for review September 3, 2025 07:05

swuferhong force-pushed the rebalance-plan branch 3 times, most recently from 967ccb0 to 85341ca Compare September 4, 2025 01:34

LiebingYu reviewed Sep 11, 2025

View reviewed changes

fluss-rpc/src/main/proto/FlussApi.proto Outdated Show resolved Hide resolved

swuferhong force-pushed the rebalance-plan branch 3 times, most recently from 60fd7ef to ed3a37a Compare October 23, 2025 08:31

LiebingYu reviewed Oct 24, 2025

View reviewed changes

swuferhong force-pushed the rebalance-plan branch from f391276 to b03b9eb Compare November 10, 2025 03:50

swuferhong added the priority=critical label Dec 25, 2025

swuferhong force-pushed the rebalance-plan branch from b03b9eb to 270af0a Compare December 25, 2025 02:54

wuchong added priority=blocker and removed priority=critical labels Dec 26, 2025

swuferhong force-pushed the rebalance-plan branch from 270af0a to 911a84e Compare December 29, 2025 04:16

swuferhong force-pushed the rebalance-plan branch 2 times, most recently from 18e28c0 to 541782e Compare January 5, 2026 02:54

platinumhamburg suggested changes Jan 6, 2026

View reviewed changes

fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java Outdated Show resolved Hide resolved

fluss-client/src/test/java/org/apache/fluss/client/admin/RebalanceITCase.java Show resolved Hide resolved

platinumhamburg suggested changes Jan 6, 2026

View reviewed changes

swuferhong force-pushed the rebalance-plan branch 4 times, most recently from 6616c63 to d16bd4f Compare January 7, 2026 01:45

platinumhamburg suggested changes Jan 7, 2026

View reviewed changes

swuferhong force-pushed the rebalance-plan branch 3 times, most recently from 5a0e50b to 995e3d2 Compare January 7, 2026 06:39

swuferhong added 10 commits January 7, 2026 16:02

[server] Support generate and execute reblance plan

cb4d8ac

[server] Support list rebalance process

67b05e1

[server] Add rebalance id to trace rebalance task

bc05b17

modify procedure

51cc1ea

fix bug

3d1544d

add log for generate plan time

eacf47a

address baiye's comments

34c64a4

fix failed test cases

79d3f34

address baiye's comments2

d6bee10

hotfix for isr wrong change

6d4f006

swuferhong force-pushed the rebalance-plan branch from 995e3d2 to 6d4f006 Compare January 7, 2026 08:02

swuferhong added 2 commits January 7, 2026 17:25

add more tests

d350589

fix wrong bucket state change

3c4d2d8

wuchong reviewed Jan 7, 2026

View reviewed changes

This was referenced Jan 8, 2026

Refactor ReplicaLeaderElection into interface with method leaderElection #2324

Open

Change the ListRebalanceProcessProcedure result from String[] to row[] #2325

Open

platinumhamburg suggested changes Jan 8, 2026

View reviewed changes

[server] Support generate and execute reblance plan #1452

Are you sure you want to change the base?

[server] Support generate and execute reblance plan #1452

Conversation

swuferhong commented Aug 3, 2025

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

platinumhamburg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

platinumhamburg Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

wuchong Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swuferhong Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

platinumhamburg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

platinumhamburg Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

swuferhong Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

wuchong Jan 7, 2026 •

edited

Loading