From db537eb07363a3c4e646a63a03d9784f13d9fda8 Mon Sep 17 00:00:00 2001 From: Deepu Kumar <140699600+deepukraju@users.noreply.github.com> Date: Tue, 23 Sep 2025 16:05:40 +0530 Subject: [PATCH 1/3] Update cluster runtime upgrade with expand health check feature The expand health check feature details are updated in learn document section concepts-cluster-upgrade-runtime. --- .../concepts-cluster-upgrade-overview.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/operator-nexus/concepts-cluster-upgrade-overview.md b/operator-nexus/concepts-cluster-upgrade-overview.md index ae8b6a9ffd..3c2ede49ad 100644 --- a/operator-nexus/concepts-cluster-upgrade-overview.md +++ b/operator-nexus/concepts-cluster-upgrade-overview.md @@ -77,6 +77,28 @@ az networkcloud baremetalmachine list -g $mrg --subscription $sub --query "sort_ --output table ``` + +## Nexus tenant workload health check during cluster runtime upgrade + +During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. + +The Inventory Readiness Check feature performs workload health check after control-plane, management-plane and compute servers are upgraded during platform runtime upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify workload health state after different stages of platform runtime upgrade. the feature supports Nexus Kubernetes Cluster and Virtual Machine workloads. + +### Workflow of workload health check + +1. **Snapshot Initiation** - Snapshot is collected for all registered workloads (Nexus Kubernetes Cluster and Virtual Machine) before starting upgrade of servers. +2. **Upgrade Stage Transitions** - After upgrade of each stage like control-plane, management-plane and compute servers are completed, comparison of inventory for workloads are initiated. +3. **Comparison Process** - Comparison of current workloads with snapshot taken during start of upgrade. Report comparison status. +4. **Health Check Handling** - On success proceed to next upgrade stage. For failure, based on inventory readiness check feature is enable or disable its handled as below. + +| Upgrade Stage | UpgradeInventoryChecks Enable | UpgradeInventoryChecks Disable | +|--------------------------|-------------------------------------|--------------------------------| +| Initial Snapshot | Upgrade failure | Upgrade continue to next stage | +| Control Plane Upgrade | Upgrade failure | Upgrade continue to next stage | +| Management Plane Upgrade | Upgrade failure | Upgrade continue to next stage | +| Compute server Upgrade | Upgrade paused, continue with CCUVA | Upgrade continue to next stage | + + ## BareMetalMachine (BMM) keyset operations during cluster runtime upgrade When a server is upgraded to utilize a new OS, the BMM keysets have to be re-established with the new software. This process starts once the runtime upgrade completes for the instance. Servers yet to undergo a runtime upgrade can still be accessed via the BMM keyset. If access to a machine is needed during the upgrade, the console user is available. From f592b782e417b8761a158e61e3b3d91cc8491b12 Mon Sep 17 00:00:00 2001 From: Deepu Kumar <140699600+deepukraju@users.noreply.github.com> Date: Thu, 25 Sep 2025 10:16:44 +0530 Subject: [PATCH 2/3] update with review comments Updated the tenant workload health check process during cluster runtime upgrades, clarifying feature flag functionality and workflow steps. --- .../concepts-cluster-upgrade-overview.md | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/operator-nexus/concepts-cluster-upgrade-overview.md b/operator-nexus/concepts-cluster-upgrade-overview.md index 3c2ede49ad..2441979308 100644 --- a/operator-nexus/concepts-cluster-upgrade-overview.md +++ b/operator-nexus/concepts-cluster-upgrade-overview.md @@ -80,24 +80,23 @@ az networkcloud baremetalmachine list -g $mrg --subscription $sub --query "sort_ ## Nexus tenant workload health check during cluster runtime upgrade -During a runtime upgrade, the inventory readiness check is triggered to conduct workload health checks. The inventory readiness check feature is appliable for only rack by rack upgrade strategy. The platform feature "UpgradeInventoryChecks" controls the platform runtime upgrade outcome when the health check fails. When the feature is enabled, the upgrade pauses if there is an inventory readiness check failure after the compute rack upgrade. The upgrade can be continued using CCUVA. When the feature is disabled the inventory readiness failures are logged and upgrade continues to next stage. By default the feature is disabled. +During a runtime upgrade, tenant workload (Nexus Kubernetes Cluster and Virtual Machine) health checks are performed for only rack by rack upgrade strategy. This functionality is feature flag enabled to control the cluster runtime upgrade outcome for health check failures. When the feature is enabled, the upgrade is paused if health check fails after the compute rack upgrade. The runtime upgrade can be resumed when customer executes the upgrade API [here](./howto-cluster-runtime-upgrade-with-pauserack-strategy.md). When the feature is disabled the health check failures are logged and upgrade continues to next stage. By default the feature is disabled. -The Inventory Readiness Check feature performs workload health check after control-plane, management-plane and compute servers are upgraded during platform runtime upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify workload health state after different stages of platform runtime upgrade. the feature supports Nexus Kubernetes Cluster and Virtual Machine workloads. +The tenant workload health checks are performed after different stages of cluster runtime upgrade like control-plane, management-plane and compute servers upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify tenant workload health state after different stages of cluster runtime upgrade. -### Workflow of workload health check +### Workflow of tenant workload health check -1. **Snapshot Initiation** - Snapshot is collected for all registered workloads (Nexus Kubernetes Cluster and Virtual Machine) before starting upgrade of servers. -2. **Upgrade Stage Transitions** - After upgrade of each stage like control-plane, management-plane and compute servers are completed, comparison of inventory for workloads are initiated. -3. **Comparison Process** - Comparison of current workloads with snapshot taken during start of upgrade. Report comparison status. -4. **Health Check Handling** - On success proceed to next upgrade stage. For failure, based on inventory readiness check feature is enable or disable its handled as below. - -| Upgrade Stage | UpgradeInventoryChecks Enable | UpgradeInventoryChecks Disable | -|--------------------------|-------------------------------------|--------------------------------| -| Initial Snapshot | Upgrade failure | Upgrade continue to next stage | -| Control Plane Upgrade | Upgrade failure | Upgrade continue to next stage | -| Management Plane Upgrade | Upgrade failure | Upgrade continue to next stage | -| Compute server Upgrade | Upgrade paused, continue with CCUVA | Upgrade continue to next stage | +1. **Snapshot Initiation** - Snapshot is collected for tenant type Nexus Kubernetes Cluster and Virtual Machine, before starting upgrade of servers. +2. **Upgrade Stage Transitions** - After upgrade of each stage like control-plane, management-plane and compute servers upgrade are completed, comparison of tenant workloads are initiated. +3. **Comparison Process** - Comparison of current tenant workloads with snapshot taken during start of upgrade. Report comparison status. +4. **Health Check Handling** - On success proceed to next runtime upgrade stage. For failure, based on feature flag is enable or disable its handled as below. +| Runtime upgrade Stage | Feature Enable | Feature Disable | +|--------------------------|----------------------------------------------------------------|---------------------------------| +| Initial Snapshot | Upgrade failure | Upgrade continues to next stage | +| Control Plane Upgrade | Upgrade failure | Upgrade continues to next stage | +| Management Plane Upgrade | Upgrade failure | Upgrade continues to next stage | +| Compute server Upgrade | Upgrade paused, resumed when customer executes the upgrade API | Upgrade continues to next stage | ## BareMetalMachine (BMM) keyset operations during cluster runtime upgrade From 1541c9b1208404005152da32332771945171550c Mon Sep 17 00:00:00 2001 From: Deepu Kumar <140699600+deepukraju@users.noreply.github.com> Date: Thu, 25 Sep 2025 10:30:34 +0530 Subject: [PATCH 3/3] Update health check behavior description in upgrade overview Clarified behavior of health checks during runtime upgrade when feature is disabled. --- operator-nexus/concepts-cluster-upgrade-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/operator-nexus/concepts-cluster-upgrade-overview.md b/operator-nexus/concepts-cluster-upgrade-overview.md index 2441979308..0704c25f2b 100644 --- a/operator-nexus/concepts-cluster-upgrade-overview.md +++ b/operator-nexus/concepts-cluster-upgrade-overview.md @@ -80,7 +80,7 @@ az networkcloud baremetalmachine list -g $mrg --subscription $sub --query "sort_ ## Nexus tenant workload health check during cluster runtime upgrade -During a runtime upgrade, tenant workload (Nexus Kubernetes Cluster and Virtual Machine) health checks are performed for only rack by rack upgrade strategy. This functionality is feature flag enabled to control the cluster runtime upgrade outcome for health check failures. When the feature is enabled, the upgrade is paused if health check fails after the compute rack upgrade. The runtime upgrade can be resumed when customer executes the upgrade API [here](./howto-cluster-runtime-upgrade-with-pauserack-strategy.md). When the feature is disabled the health check failures are logged and upgrade continues to next stage. By default the feature is disabled. +During a runtime upgrade, tenant workload (Nexus Kubernetes Cluster and Virtual Machine) health checks are performed for only rack by rack upgrade strategy. This functionality is feature flag enabled to control the cluster runtime upgrade outcome for health check failures. When the feature is enabled, the upgrade is paused if health check fails after the compute rack upgrade. The runtime upgrade can be resumed when customer executes the upgrade API [here](./howto-cluster-runtime-upgrade-with-pauserack-strategy.md). When the feature is disabled the upgrade continues to next stage even after the health check failure. By default the feature is disabled. The tenant workload health checks are performed after different stages of cluster runtime upgrade like control-plane, management-plane and compute servers upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify tenant workload health state after different stages of cluster runtime upgrade.