See a pod admitted but still schedule gated

[amy]

What happened:

We see that a workload is admitted but the underlying pod still schedulegated.

What you expected to happen:
Either:

1. workload for pod is admitted and the underlying pod is Pending
2. OR workload for pod is not admitted

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
Potential problems:

• the update for admission and schedulegate removal should happen atomically
• potentially an issue where kubernetes api will reject future changes to pod if you have previously changed a field outside of an allowed list of fields in the spec. (But overall for this, if kubernetes API fails, we want admission to fail as well)

Keep in mind: ScheduleGate cannot be re-added if its already been removed. So probably we don't need to look through any update path.

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[amy]

cc/ ^ @mwielgus @ichekrygin

[kannon92]

Do you by chance have more info on how to reproduce this?

I've seen some issues where scheduling gates updates get rejected and that may not reflect the correct update.

I've also seen this behavior if you have multiple scheduling gates.

[kannon92]

For Openshift we get these problems for scc rules if we don't specify the right pod security context for raw pods and the admission plugin rejects the update of the pod.

[amy]

Do you by chance have more info on how to reproduce this?

no sorry 🥲 For us it was some storage thing. I don't know if the actual issue is as relevant. The main issue is why Kueue isn't doing the right thing.

have multiple scheduling gates

wait whats this one? As in, what do you mean by multiple? @kannon92

[kannon92]

wait whats this one? As in, what do you mean by multiple? @kannon92

Scheduling gates are a list of strings.

Kueue adds one but if a pod has other scheduling gates I think kueue thinks it admitted the workload.

[amy]

Ah I see this: https://github.com/kubernetes-sigs/kueue/blob/main/pkg/controller/jobs/pod/pod_controller.go#L1403
Where yeah it probably just checks if it gates/ungates "kueue.x-k8s.io/admission"

@mimowo @kannon92 Before admission, do you think it makes sense to add an admission check:

• there's no remaining scheduling gates besides the kueue one?

/assign

[kannon92]

Ah I see this: https://github.com/kubernetes-sigs/kueue/blob/main/pkg/controller/jobs/pod/pod_controller.go#L1403 Where yeah it probably just checks if it gates/ungates "kueue.x-k8s.io/admission"

@mimowo @kannon92 Before admission, do you think it makes sense to add an admission check:

• there's no remaining scheduling gates besides the kueue one?

/assign

I think a gate is added in cases of ManagedJobsWithoutQueueName.

I would think that maybe we should make kueue check all the scheduling gates and if there are other ones it shouldn't mark the workload as admitted.

[amy]

I think a gate is added in cases of ManagedJobsWithoutQueueName.

We heavily see ScheduleGated because we use the pod/podgroup workload type. So we send the pod (with Kueue annotations/labels) first. Then Kueue creates the workload from those annotations. So those pods are first ScheduleGated by Kueue until admission. Once ungated, we see the pods go to Pending then Running (with various failure reasons in the Pending phase from k8s)

But yeah, I'd like to propose adding an admission check

add an admission check:
there's no remaining scheduling gates besides the kueue one?