Finer control over maxRestarts for RestartJobSet vs RecreateJob

[carreter]

What would you like to be added:
Currently, the RecreateJob and RestartJobSet failure policies both increment the same .status.restartsCountTowardsMax whenever they are triggered. When said counter reaches the value of .spec.failurePolicy.maxRestarts, triggering either the RecreateJob or RestartJobSet will mark the entire JobSet as failed.

It'd be desirable to allow separate counters for RecreateJob and RestartJobSet. For example, imagine a JobSet with three ReplicatedJobs: workers-heavy, workers-light, and controllers. We want the following behaviours:

• failures in workers-heavy trigger RecreateJob up to 4 times for each child Job, as these are expensive to recreate
• failures in workers-light trigger RecreateJob up to 10 times for each child Job, as these are easy and fast to recreate
• failures in controllers trigger RestartJobSet up to 3 times total

The configuration could look something like this

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: test
spec:
  failurePolicy:
    maxRestarts: 3 # How many times can a full JobSet restart happen?
    rules:
    - action: RecreateJob
      maxRecreatesPerIndex: 4 # How many times can a child job in workers-heavy be recreated before failing the JobSet?
      targetReplicatedJobs:
      - workers-heavy
    - action: RecreateJob
      maxRecreatesPerIndex: 10 # How many times can a child job in workers-light be recreated before failing the JobSet?
      targetReplicatedJobs:
      - workers-light

Why is this needed:
Not all ReplicatedJob failures are the same :) In the previous example, workers-heavy take a long time to recreate and as such should fail in fewer attempts than workers-light, which can be recreated quickly more times without incurring too heavy a cost.

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[GiuseppeTT]

I wonder if it makes more sense to support a counter per replicatedJob instead of per failureActionRule. Would that be enough for your use case?

[carreter]

I think either way would make sense and fulfill the requirements for this use case. The main reason I see for configuring it per failureActionRule is that you could target different failure reasons in the same ReplicatedJob with different counters, but I don't know how necessary this is.

[GiuseppeTT]

I see. I think that there are 4 alternatives for counting restarts in consideration:

1. (Current state) One global counter
2. (Possible improvement) One counter per ReplicatedJob
3. (Possible improvement) One counter per FailureAction
4. (Possible improvement) One counter per ReplicatedJob + FailureAction combination

(2), (3) and (4) cover your use case. (2) and (3) should be preferred over (4) because they are simpler.

IMO (2) makes more sense than (3). The use cases that I am aware use multiple ReplicatedJobs to set up different group of Jobs with different restart requirements. See user story 4 for an example. Having the same failure action in different ReplicatedJobs share the same counter would not make sense.

I think that the best approach is (2) with some careful design to nicely be expanded to support (4) if it ever becomes necessary.

[carreter]

I like the plan of starting with (2) and making sure it can be extended to (4). I'll start working on that unless others have any major inputs to give. I'll draft up a doc with a more detailed explanation of a proposed API.

[GiuseppeTT]

Sounds good!

cc @imreddy13 @kannon92 @andreyvelich

[andreyvelich]

Can't we use PodFailurePolicy + backOffLimit API for such use-cases ?
https://kubernetes.io/docs/tasks/job/pod-failure-policy/

[carreter]

Can't we use PodFailurePolicy + backOffLimit API for such use-cases ? https://kubernetes.io/docs/tasks/job/pod-failure-policy/

I don't think this quite fits the use case. If I'm understanding correctly, the .spec.backOffLimit field on a Job allows you to set a limit on the number of Pod failures before a Job is marked as failed. Here we want to set a limit on Job failures before a JobSet is marked as failed.

[GiuseppeTT]

If I'm understanding correctly, the .spec.backOffLimit field on a Job allows you to set a limit on the number of Pod failures before a Job is marked as failed. Here we want to set a limit on Job failures before a JobSet is marked as failed.

This is correct. Besides, the use cases that Willow and I have in mind set backoffLimit = 0 since they require the whole Job / JobSet to be restarted after failures

[carreter]

Thought about it a little more, how does this sound:

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: test
spec:
  failurePolicy:
    maxRestarts: 3 # How many times can a full JobSet restart happen?
    maxRecreatesPerReplicatedJob: # How many Job recreates can happen in an individual ReplicatedJob?
      workers-heavy: 4
      workers-light: 10
    rules:     
    - action: RecreateJob
      targetReplicatedJobs:
      - workers-heavy
      - workers-light
      - other-workers

The maxRecreatesPerReplicatedJob overrides maxRestarts, i.e. for this failure policy:

• failures in other-workers will count toward the same maxRestarts counter as full JobSet restarts
• failures in workers-heavy and workers-light will count toward independent counters on a ReplicatedJob level

@GiuseppeTT @kannon92 @imreddy13 @andreyvelich WDYT?

[GiuseppeTT]

Looks good.

@kannon92 @imreddy13 @andreyvelich WDYT?

[kannon92]

I think this sounds fine to me but the use case seems quite abstract.

Have you ran into this in the wild and you know of users that need this?

[andreyvelich]

Alternatively, we can introduce maxRestarts under ReplicatedJob spec level, similar to DependsOn API.

@carreter How are you going to count restarts if we have >=2 Replicas of the ReplicatedJob. E.g. multiple batch/Jobs are created for the one ReplicatedJob.

[carreter]

Have you ran into this in the wild and you know of users that need this?

We have an internal customer that will make use of this - they want a low maxRestarts to fail quickly if the driver ReplicatedJobs fail, but a higher maxRecreatesPerReplicatedJob for the worker ReplicatedJobs.

@carreter How are you going to count restarts if we have >=2 Replicas of the ReplicatedJob. E.g. multiple batch/Jobs are created for the one ReplicatedJob.

I was thinking of having all replicas of a given ReplicatedJob contributing to the same counter.