WithoutOverlapping is always bound to the Job it's used in

[okaufmann]

Hi Community!

I'm using Jobs to interact with an API which is hardly rate limited. There are multiple Jobs that do things one after all (fetch projects, fetch customers, fetch working time entries, calculate project progress etc...)

These jobs should never overlap globally (never fetch customers while another job of the chain is running), so I used the middleware WIthoutOverlapping (https://laravel.com/docs/8.x/queues#preventing-job-overlaps) and the same key for all Jobs like this:

public function middleware()
{
    return [
        (new WithoutOverlapping("dont-overlap")),
    ];
}

So far, so good. But after that, it was still running into rate limits. After some investigation I detected, that the Middleware is always bound to the Job it is used in. This is caused by the following function in the WithoutOverlapping middleware:

/**
 * Get the lock key for the given job.
 *
 * @param  mixed  $job
 * @return string
 */
public function getLockKey($job)
{
    return $this->prefix.get_class($job).':'.$this->key;
}

https://github.com/laravel/framework/blob/8.x/src/Illuminate/Queue/Middleware/WithoutOverlapping.php#L137

The problem is caused by get_class($job) which scopes the key always to the certain job class.

This is also not clear when reading the Docs IMHO.

Of course, I can create my very own variant of this middleware and change this behaviour. But could we not have the middleware renamed to e.g. WithoutOverlappingByJob? As this would be a braking change in some cases it could also make sense to create a new middleware e.g. WithoutGlobalOverlapping and note this behaviour in the docs to make this even more clear?

What do you think?

[wsamoht]

I agree that something needs to change or be clarified here. I was going to use WithoutOverlapping to make sure different but related jobs don't try running at the same time since they would all be updating the same resource, in my case subscription info on User. I just so happened to be source diving and discovered WithoutOverlapping is specific to a job based on it's class name.

I can think of two different code options that wouldn't be a breaking change. The first is to create a WithoutOverlappingGlobally class which I am currently doing that overrides getLocKey. The other option is to add a globally() method to WithoutOverlapping that sets a flag to tell getLockKey to not include the class name in the key. It could be set like this:

public function middleware()
{
    return [(new WithoutOverlapping($this->user->id))->globally()];
}

If a global key option is not desired, I think the documentation should at least be clarified. It currently states the following:

Laravel includes an Illuminate\Queue\Middleware\WithoutOverlapping middleware that allows you to prevent job overlaps based on an arbitrary key.

"Arbitrary" and the example of using $this->user->id makes it seem like it would keep any job from running at the same time (even with different names) as long as they use the same key.

I'd be glad to submit a PR for a code and/or documentation change. Just let me know! I personally am leaning towards a globally option on the existing WithoutOverlapping class and then an update to the documentation with the option.

[SCIF]

Hmm, actually that was implemented in #44227

[okaufmann]

This thread is very outdated. Will close it. As @SCIF said -> #44227.

Docs can be found here: https://laravel.com/docs/10.x/queues#sharing-lock-keys