feat(KEP-3258): implement delayed admission check retries #7464
Conversation
Implements delayed retry mechanism for admission checks to prevent overwhelming external controllers and reduce control plane churn. Problem: Previously, when admission checks transitioned to Retry state, Kueue would immediately evict workloads and requeue them, causing excessive load on admission check controllers and unnecessary API server churn, particularly when retry conditions persisted predictably. Solution: This implementation adds two new fields to AdmissionCheckState: - requeueAfterSeconds: Specifies minimum wait time before retry - retryCount: Tracks retry attempts per admission check Key Changes: API Changes - Added requeueAfterSeconds and retryCount fields to AdmissionCheckState Controller Changes - Auto-increments retryCount on transition to Retry state - Calculates maximum retry time across all admission checks - Updates workload.status.requeueState.requeueAt with the maximum delay - Workload controller now respects delayed retry times before resetting checks - Prevents premature re-admission while delays are active Behavior: When multiple admission checks specify different delays, Kueue uses the maximum delay across all checks. Workloads are evicted immediately to release quota, but admission check states aren't reset until all delays expire, preventing race conditions where fast-responding checks could block slower ones from registering their delays. Refs: KEP-3258 https://github.com/DataDog/kueue/blob/main/keps/3258-delayed-admission-check-retries/README.md
This is a follow-up to [kubernetes-sigs#7370](kubernetes-sigs#7370), where we introduced delayed admission checks. This PR replaces the use of the shared RequeueState field with the new mechanism in the preprovision request admission check. Refs: KEP-3258 https://github.com/DataDog/kueue/blob/main/keps/3258-delayed-admission-check-retries/README.md
k8s-ci-robot
added
release-note
✅ Deploy Preview for kubernetes-sigs-kueue ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: dhenkel92 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @dhenkel92. Thanks for your PR. I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
do-not-merge/work-in-progress
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
3 participants
What type of PR is this?
/kind feature
What this PR does / why we need it:
This is a follow-up to #7370, where we introduced delayed admission checks.
This PR replaces the use of the shared RequeueState field with the new mechanism in the preprovision request admission check.
Which issue(s) this PR fixes:
Refs: KEP-3258
Special notes for your reviewer:
Does this PR introduce a user-facing change?