✨ Support Node Auto Placement and Node AF/AAF #3655

zhanggbj · 2025-10-28T07:39:18Z

What this PR does / why we need it:
Support Node Auto Placement and Node AF/AAF

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

- Bump VMOP including Node AF/AAF support - Add NodeAutoPlacement Feature Gate (cherry picked from commit 700c8ae)

(cherry picked from commit cfeb862)

Removes the extra cases for VMG creation, such that VMG is created for: 1. Multiple zones, multiple MDs with no failureDomain 2. Multiple zones, multiple MDs with failureDomain 3. Single zone, existing cluster with no failureDomain MDs Signed-off-by: Sagar Muchhal <[email protected]>

- Updates VMOP API dependency Misc VMG fixes - Use namingStrategy to calculate VM names - Use MachineDeployment names for VMG placement label - Includes all machinedeployments to generate node-pool -> zone mapping Fixes VMG webhook validation error - Adds cluster-name label to Af/AAF spec - re-adds zone topology key back to anti-aff spec Signed-off-by: Sagar Muchhal <[email protected]>

Signed-off-by: Sagar Muchhal <[email protected]>

…gs#71) * Refine VMG controller when generate per-MD zone labels - Skip legacy already-placed VM which do not have placement info - Skip VM which do not have zone info * Apply suggestions from code review --------- Co-authored-by: Sagar Muchhal <[email protected]>

- Sync VSphereMachines during day-2 operations in VMG controller - Only wait for all intended VSphereMachines during initial Cluster creation - Use annotations in VMG for per-md-zone info Signed-off-by: Gong Zhang <[email protected]>

- Add VMG recociler unit test - Bump VMOP due to API change - Filter out VSphereMachine event except create/delete events Signed-off-by: Gong Zhang <[email protected]>

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-10-31T06:48:24Z

/test ?

k8s-ci-robot · 2025-10-31T06:48:28Z

@zhanggbj: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-34-1-35-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-34-1-35-main

/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

/test pull-cluster-api-provider-vsphere-test-main

/test pull-cluster-api-provider-vsphere-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-vsphere-apidiff-main

/test pull-cluster-api-provider-vsphere-janitor-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-vsphere-apidiff-main

pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

pull-cluster-api-provider-vsphere-test-main

pull-cluster-api-provider-vsphere-verify-main

Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zhanggbj · 2025-10-31T06:49:36Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-34-1-35-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-34-1-35-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main
/test pull-cluster-api-provider-vsphere-test-main
/test pull-cluster-api-provider-vsphere-verify-main
/test pull-cluster-api-provider-vsphere-apidiff-main
/test pull-cluster-api-provider-vsphere-janitor-main

zhanggbj · 2025-11-24T15:58:09Z

Hey @fabriziopandini @sbueringer ,
All comments are addressed, including the ones missed by mistake last time. Now CI are green. Please help to take another look, really appreciate!

For the concern about race conditions, I have implemented checks to mitigate race conditions during initial and post-placement, and also have unit teststo cover the scenarios.

zhanggbj · 2025-11-25T09:17:55Z

Pushed a small change and checking UT failures.

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-11-25T11:03:14Z

/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

zhanggbj · 2025-11-26T07:44:01Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

Since Encryption Class requires API in this newer version, bump vm-operator package Signed-off-by: Gong Zhang <[email protected]>

Signed-off-by: Gong Zhang <[email protected]>

fabriziopandini · 2025-12-11T16:49:31Z

controllers/vmware/virtualmachinegroup_reconciler.go

+				continue
+			}
+			if member.Placement != nil && member.Placement.Zone != "" {
+				log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domanin %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))


nit

Suggested change

log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domanin %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))

log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domain %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))

fabriziopandini · 2025-12-11T17:07:57Z

controllers/vmware/virtualmachinegroup_reconciler_test.go

+		want            bool
+	}{
+		{
+			name: "Should create a VMG if all the expected VSphereMachines exists",


Let's add a unit test ensuring that a deleting vSphereMachine blocks.
I would suggest to use this test case as a setup and add an additional deleting machine

@fabriziopandini
Trying to understand this scenario, does it mean a deleting vSphereMachine won't be added to the VMGroup member list?

If yes, this case should cover it name: "Should create a VMG if all the expected VSphereMachines exists, deleting MD should be ignored"

a deleting machine should not be ignored, it should block creation of a VMG

I cannot recall we discussed deleting machine to block the creation of a VMG.

Is this the scaling down case? IIRC, we agreed that scaling down MD is allowed during any phase (before VMS is created or VMG is created by still during placement)

This is what is in the current implementation, and it makes sense to me because it makes sure that we start a placement decision only when the list of vSphereMachines is stable.

Let's please add a unit test and do not change the code

@fabriziopandini Oh I got your points, we won't create VMG if expected VSphereMachine haven't existed yet (VM in deleting). I misread it.

Added case:
name: "Should not create a VMG if the expected VSphereMachines do not exist yet"

fabriziopandini · 2025-12-11T17:15:13Z

controllers/vmware/virtualmachinegroup_reconciler_test.go

+						clusterv1.ClusterNameLabel: cluster.Name,
+					},
+					Annotations: map[string]string{
+						fmt.Sprintf("%s/%s", vmoperator.ZoneAnnotationPrefix, "md3"): "zone1", // failureDomain for md3 is explicitly set by the user


Let's add an annotation without the ZoneAnnotationPrefix and check it is preserved

fabriziopandini · 2025-12-11T17:15:51Z

controllers/vmware/virtualmachinegroup_reconciler_test.go

+					Namespace: cluster.Namespace,
+					Name:      cluster.Name,
+					UID:       types.UID("uid"),
+					Labels: map[string]string{


Let's add an label different than the ClusterNameLabel and check it is preserved

fabriziopandini · 2025-12-11T17:48:33Z

pkg/services/vmoperator/vmopmachine.go

+				Reason:  infrav1.VSphereMachineVirtualMachineWaitingForVirtualMachineGroupV1Beta2Reason,
+				Message: fmt.Sprintf("Waiting for VSphereMachine's VirtualMachineGroup %s to exist", key),
+			})
+			log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))


Suggested change

log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))

log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s to exist, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))

fabriziopandini · 2025-12-11T17:53:43Z

pkg/services/vmoperator/vmopmachine.go

+		}
+
+		// Set the zone label using the annotation of the per-md zone mapping from VirtualMachineGroup.
+		// This is for new VMs created during day-2 operations when Node Auto Placement is enabled.


Suggested change

// This is for new VMs created during day-2 operations when Node Auto Placement is enabled.

// This is for new VMs created after initial placement decision/with a failureDomain defined by the user.

fabriziopandini · 2025-12-11T17:55:32Z

pkg/services/vmoperator/vmopmachine.go

+			client.MatchingLabels{clusterv1.ClusterNameLabel: supervisorMachineCtx.Cluster.Name}); err != nil {
+			return false, err
+		}
+		othermMDNames := []string{}


nit

Suggested change

othermMDNames := []string{}

otherMDNames := []string{}

fabriziopandini · 2025-12-11T19:00:40Z

pkg/services/vmoperator/vmopmachine_test.go

+			})
+
+			// control plane machine is the machine with the control plane label set
+			Specify("Reconcile valid control plane Machine", func() {


Let's add a check that the CP machine is placed in the expected zone

Added additional check for CP machine, that CP machine has the vmopVM.Labels[corev1.LabelTopologyZone] which is the contract btw CAPV&VMOP.

fabriziopandini · 2025-12-11T19:02:03Z

pkg/services/vmoperator/vmopmachine_test.go

+					vmGroup               *vmoprv1.VirtualMachineGroup
+				)
+
+				BeforeEach(func() {


Looks like we are missing a test for VMG not created yet

Added a new case to test
"requeue the Machine waiting for VirtualMachineGroup creation"

fabriziopandini · 2025-12-11T19:10:18Z

pkg/services/vmoperator/vmopmachine_test.go

+
+					machineNotMember.Spec.Bootstrap.DataSecretName = &secretName
+
+					By("VirtualMachine is not created")


Let's check the VM actually does not exist, not only result

zhanggbj · 2025-12-15T09:49:39Z

/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

zhanggbj · 2025-12-15T15:17:39Z

@fabriziopandini @sbueringer
Except comment here is still open #3655 (comment),

I've addressed other comments, could yout please take a look at others firstly? Thanks!

fabriziopandini · 2025-12-15T17:44:22Z

I have answered to the comment, please fix it first

- Add missed test cases - miscellaneous Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-12-17T05:58:24Z

pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main failed due to infra failure and I'm not able to re-trigger again. Need to retry later.

sbueringer · 2025-12-17T12:38:06Z

@zhanggbj You have to use / test

/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

k8s-ci-robot · 2025-12-19T14:18:37Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zhanggbj and others added 11 commits October 22, 2025 14:19

Bump VMOP and add NodeAutoPlacement Feature Gate

fa60556

- Bump VMOP including Node AF/AAF support - Add NodeAutoPlacement Feature Gate (cherry picked from commit 700c8ae)

Add VMG controller of node auto placement

b52056f

(cherry picked from commit cfeb862)

Removes cluster-name label from affinity rules

9f261d5

Signed-off-by: Sagar Muchhal <[email protected]>

Selectively add node-pool AAF constraint

22d7db0

Signed-off-by: Sagar Muchhal <[email protected]>

Sync VSphereMachines in VMG controller

833cdb5

- Sync VSphereMachines during day-2 operations in VMG controller - Only wait for all intended VSphereMachines during initial Cluster creation - Use annotations in VMG for per-md-zone info Signed-off-by: Gong Zhang <[email protected]>

Add VMG reconciler unit test and bump VMOP

36c3550

- Add VMG recociler unit test - Bump VMOP due to API change - Filter out VSphereMachine event except create/delete events Signed-off-by: Gong Zhang <[email protected]>

Surface error when VM is waiting for VMG

189f6d2

Signed-off-by: Gong Zhang <[email protected]>

Update dependency in packaging&test

8cc8e8a

Signed-off-by: Gong Zhang <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 28, 2025

k8s-ci-robot requested review from MaxRink and rvanderp3 October 28, 2025 07:39

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 28, 2025

zhanggbj changed the title ~~[WIP][DoNotReview]Support Node Auto Placement and Node AF/AAF~~ [WIP][DoNotReview]✨ Support Node Auto Placement and Node AF/AAF Oct 28, 2025

zhanggbj force-pushed the node_auto_placement branch 3 times, most recently from 1160ce9 to 1cd61f9 Compare October 28, 2025 10:31

zhanggbj force-pushed the node_auto_placement branch 7 times, most recently from 22604bf to 7c7c1a7 Compare November 3, 2025 08:55

zhanggbj force-pushed the node_auto_placement branch from 3ed2e93 to 9b1754a Compare November 25, 2025 06:44

Refine VMG member update to avoid race condition

2ac47d4

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from 9b1754a to 2ac47d4 Compare November 25, 2025 09:25

zhanggbj mentioned this pull request Nov 26, 2025

✨ Add supports to Crypto Spec in VMOp VirtualMachine and PVC #3680

Closed

Update vm-operator package to v1.9.1-0.20251003150112-9b458d311c4c

c41abeb

Since Encryption Class requires API in this newer version, bump vm-operator package Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from fd72f09 to c41abeb Compare November 26, 2025 07:49

zhanggbj and others added 5 commits November 27, 2025 11:46

Added reconcile flow cases

516af84

Signed-off-by: Gong Zhang <[email protected]>

POC AAF

ac0ec4b

Review comments / fixes

822f8f0

Final cleanup

db8095d

Finalize VMG controller and UT

08b8304

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from b8c569e to 08b8304 Compare November 27, 2025 09:23

fabriziopandini reviewed Dec 12, 2025

View reviewed changes

zhanggbj force-pushed the node_auto_placement branch from 68f5b16 to ea10895 Compare December 16, 2025 07:26

Refine UT

6d76fbd

- Add missed test cases - miscellaneous Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from ea10895 to 6d76fbd Compare December 16, 2025 08:08

Allow empty FD on additional MD for single-zone clusters

576e9f6

kubernetes-sigs deleted a comment from k8s-ci-robot Dec 17, 2025

sbueringer mentioned this pull request Dec 19, 2025

[WIP] ⚠️ Bump to Cluster API, controller-runtime main & controller-tools v0.20 #3733

Open

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 19, 2025

	log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domanin %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))
	log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domain %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))

	log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))
	log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s to exist, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))

	// This is for new VMs created during day-2 operations when Node Auto Placement is enabled.
	// This is for new VMs created after initial placement decision/with a failureDomain defined by the user.


		machineNotMember.Spec.Bootstrap.DataSecretName = &secretName

		By("VirtualMachine is not created")

✨ Support Node Auto Placement and Node AF/AAF #3655

Are you sure you want to change the base?

✨ Support Node Auto Placement and Node AF/AAF #3655

Conversation

zhanggbj commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhanggbj commented Oct 31, 2025

Uh oh!

k8s-ci-robot commented Oct 31, 2025

Uh oh!

zhanggbj commented Oct 31, 2025

Uh oh!

zhanggbj commented Nov 24, 2025

Uh oh!

zhanggbj commented Nov 25, 2025

Uh oh!

zhanggbj commented Nov 25, 2025

Uh oh!

zhanggbj commented Nov 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhanggbj Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhanggbj Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhanggbj commented Dec 15, 2025

Uh oh!

zhanggbj commented Dec 15, 2025

Uh oh!

fabriziopandini commented Dec 15, 2025

Uh oh!

zhanggbj commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhanggbj commented Oct 28, 2025 •

edited

Loading

zhanggbj Dec 15, 2025 •

edited

Loading

zhanggbj Dec 15, 2025 •

edited

Loading

zhanggbj commented Dec 17, 2025 •

edited

Loading