Can the Hami scheduler achieve K8s preemption ？

[pipifishes]

配置了hami，实现vgpu切分后， 和k8s抢占冲突
After configuring hami and implementing vgpu slicing, there is a conflict with k8s preemption

节点上有2张卡，切分也为2张卡：
There are 2 cards on the node, and the split is also 2 cards:

已配置抢占 策略和一个低优先级yaml：
The preemptive strategy and a low-priority YAML have been configured:

高优先级yaml:
High-priority yaml:

测试结果：
高优先级的并未抢占低优先级：
Test results:
High priority did not preempt low priority:

但是删除hami，抢占就可成功。
But by deleting hami, the seizure can be successful.

[archlitchi]

Yes, a nice feature for us to compatible with, i'll add it to roadMap

[FouoF]

@pipifishes I tested the preemption with hami and it works. I noticed that you deployed both Nvidia device plugin and hami device plugin in your node which can cause the conflict.

[pipifishes]

@FouoF ，Thank you for your reply
I found through my own testing that my environment and Nvidia device plugin are not affected. Once I delete hami, meaning that its two pods do not exist, setting preemption is normal;
Or could you please provide me with your testing process

low_test.yaml:

high_test.yaml:

test, successed:

[FouoF]

@pipifishes I retried my case and find error like:

Can you help check the scheduler logs to make sure we meet the same error?

[vitush93]

@pipifishes I retried my case and find error like:

Can you help check the scheduler logs to make sure we meet the same error?

I can confirm as well that we see the same error; specifically when preemption of Pods with lower priority classes is appropriate but won't happen.

[pipifishes]

@vitush93 @FouoF

Apply low priority first and display running; High priority applications were reapplied, but were not successfully preempted

View the logs of one of the pods：

[FouoF]

It seems kubernetes default-preemption not support scheduler extenders.

https://github.com/kubernetes/kubernetes/blob/5bcb7599736327cd8c6d23e398002354a6e40f68/pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go#L207

The default-preemption need to call Filter plugins to evaluate best victims set of the nominatedNode and the RunFilterPluginsWithNominatedPods differs from main Filter process that it will not call filter extenders. And this leads to inconsistency that the preemption assumes no pod need to be evicted.