[12.x] Improve Queues and Horizon documentation

horizon.md

@@ -4,6 +4,9 @@
 - [Installation](#installation)
     - [Configuration](#configuration)
     - [Balancing Strategies](#balancing-strategies)
+    - [Max Job Attempts](#max-job-attempts)
+    - [Job Timeout](#job-timeout)
+    - [Job Backoff](#job-backoff)
     - [Dashboard Authorization](#dashboard-authorization)
     - [Silenced Jobs](#silenced-jobs)
 - [Upgrading Horizon](#upgrading-horizon)
@@ -92,6 +95,28 @@ You may also define a wildcard environment (`*`) which will be used when no othe
 
 When you start Horizon, it will use the worker process configuration options for the environment that your application is running on. Typically, the environment is determined by the value of the `APP_ENV` [environment variable](/docs/{{version}}/configuration#determining-the-current-environment). For example, the default `local` Horizon environment is configured to start three worker processes and automatically balance the number of worker processes assigned to each queue. The default `production` environment is configured to start a maximum of 10 worker processes and automatically balance the number of worker processes assigned to each queue.
 
+You may also define Horizon's environment independently of the application's `APP_ENV` by adding an `env` key to your Horizon configuration file:
+
+```php
+
+return [
+    /*
+    | ------------------------------------------------------------------
+    | Horizon Environment
+    | ------------------------------------------------------------------
+    |
+    | This value determines Horizon's environment.
+    */
+    'env' => env('HORIZON_ENV', env('APP_ENV')),
+
+    // ...
+];
+```
+
+This can be particularly useful when running Horizon across multiple servers. For example, you might have one server processing general-purpose jobs using the default queue configuration, while a second server, using in a different environment key, is dedicated to handling resource-intensive jobs, such as video processing or large data imports.
+
+Using two different environments makes it easy to configure Horizon on a per-server basis rather than a per-environment basis, giving you more granular control over worker settings and job handling.
+
 > [!WARNING]
 > You should ensure that the `environments` portion of your `horizon` configuration file contains an entry for each [environment](/docs/{{version}}/configuration#environment-configuration) on which you plan to run Horizon.
 
@@ -126,37 +151,230 @@ Within Horizon's default configuration file, you will notice a `defaults` config
 <a name="balancing-strategies"></a>
 ### Balancing Strategies
 
-Unlike Laravel's default queue system, Horizon allows you to choose from three worker balancing strategies: `simple`, `auto`, and `false`. The `simple` strategy splits incoming jobs evenly between worker processes:
+Each supervisor can process one or more queues but unlike Laravel's default queue system, Horizon allows you to choose from three worker balancing strategies: `auto`, `simple`, and `false`. 
 
-    'balance' => 'simple',
+<a name="auto-strategy"></a>
+#### Auto Strategy
 
-The `auto` strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `render` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty.
+The `auto` strategy, which is the default strategy, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `default` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty.
 
-When using the `auto` strategy, you may define the `minProcesses` and `maxProcesses` configuration options to control the minimum number of processes per queue and the maximum number of worker processes in total Horizon should scale up and down to:
+You may define the `minProcesses` and `maxProcesses` configuration options:
+
+- `minProcesses` defines the minimum number of worker processes per queue. This value must be greater than or equal to 1.
+- `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to across all queues. This value should typically be greater than the number of queues multiplied by the `minProcesses` value. To prevent the supervisor from spawning any processes, you may set this value to 0.
+
+For example, you may configure Horizon to maintain at least one process per queue and scale up to a total of 10 worker processes:
 
 ```php
 'environments' => [
     'production' => [
         'supervisor-1' => [
             'connection' => 'redis',
-            'queue' => ['default'],
+            'queue' => ['default', 'notifications'],
             'balance' => 'auto',
             'autoScalingStrategy' => 'time',
             'minProcesses' => 1,
             'maxProcesses' => 10,
             'balanceMaxShift' => 1,
             'balanceCooldown' => 3,
-            'tries' => 3,
         ],
     ],
 ],
 ```
 
-The `autoScalingStrategy` configuration value determines if Horizon will assign more worker processes to queues based on the total amount of time it will take to clear the queue (`time` strategy) or by the total number of jobs on the queue (`size` strategy).
+The `autoScalingStrategy` configuration option determines how Horizon will assign more worker processes to queues. You can choose between two strategies:
+- The `time` strategy will assign workers based on the total estimated amount of time it will take to clear the queue.
+- The `size` strategy will assign workers based on the total number of jobs on the queue.
 
 The `balanceMaxShift` and `balanceCooldown` configuration values determine how quickly Horizon will scale to meet worker demand. In the example above, a maximum of one new process will be created or destroyed every three seconds. You are free to tweak these values as necessary based on your application's needs.
 
-When the `balance` option is set to `false`, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration.
+<a name="auto-queue-priorities"></a>
+#### Queue Priorities With Auto
+
+When using the `auto` balancing strategy, Horizon does not enforce strict priority between queues. The order of queues in a supervisor's configuration does not affect how worker processes are assigned. Instead, Horizon relies on the selected `autoScalingStrategy` to dynamically allocate worker processes based on queue load
+
+For example, in the following configuration, the high queue is not prioritized over the default queue, despite appearing first in the list:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['high', 'default'],
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+        ],
+    ],
+],
+```
+
+If you need to enforce a relative priority between queues, you may define multiple supervisors and explicitly allocate processing resources:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default'],
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+        ],
+        'supervisor-2' => [
+            // ...
+            'queue' => ['images'],
+            'minProcesses' => 1,
+            'maxProcesses' => 1,
+        ],
+    ],
+],
+```
+
+In this example, the default `queue` can scale up to 10 processes, while the `images` queue is limited to one process. This configuration ensures that your queues can scale independently.
+
+>[!NOTE]
+> When dispatching resource-intensive jobs, it's best to assign them to a dedicated queue with a limited `maxProcesses` value. Otherwise, these jobs could consume excessive CPU resources and overload your system.
+
+<a name="simple-strategy"></a>
+#### Simple Strategy
+
+The `simple` strategy distributes worker processes evenly across the specified queues. With this strategy, Horizon does not automatically scale the number of worker processes, it uses a fixed number of processes.
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default', 'notifications'],
+            'balance' => 'simple',
+            'processes' => 10,
+        ],
+    ],
+],
+```
+In the example above, Horizon will assign 5 processes to each queue, splitting the total of 10 evenly.
+
+If you'd like to control the number of worker processes assigned to each queue individually, you can define multiple supervisors:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default'],
+            'balance' => 'simple',
+            'processes' => 10,
+        ],
+        'supervisor-notifications' => [
+            // ...
+            'queue' => ['notifications'],
+            'balance' => 'simple',
+            'processes' => 2,
+        ],
+    ],
+],
+```
+
+With this configuration, Horizon will assign 10 processes to the `default` queue and 2 processes to the `notifications` queue.
+
+<a name="false-strategy"></a>
+#### False Strategy
+
+When the `balance` option is set to `false`, Horizon processes queues strictly in the order they're listed, similar to Laravel’s default queue system. However, it will still scale the number of worker processes if jobs begin to accumulate.
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default', 'notifications'],
+            'balance' => false,
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+        ],
+    ],
+],
+```
+
+In the example above, jobs in the `default` queue are always prioritized over jobs in the `notifications` queue. For instance, if there are 1,000 jobs in `default` and only 10 in `notifications`, Horizon will fully process all `default` jobs before handling any from `notifications`.
+
+You can control Horizon's ability to scale worker processes using the `minProcesses` and `maxProcesses` options:
+
+- `minProcesses` defines the minimum number of worker processes in total. This value must be greater than or equal to 1.
+- `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to.
+
+<a name="max-job-attempts"></a>
+### Max Job Attempts
+
+> [!NOTE]
+> Before refining these options, make sure you are familiar with Laravel's base [queue services](/docs/{{version}}/queues#max-job-attempts-and-timeout) and the concept of 'attempts'.
+
+You can define the maximum number of attempts a job can consume directly at the supervisor level:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'tries' => 10,
+        ],
+    ],
+],
+```
+
+> [!NOTE]
+> This option is similar to the `--tries` option when using the Artisan command to process queues.
+
+Fine-tuning the `tries` option is essential when using middlewares such as `WithoutOverlapping` or `RateLimited` because they consume attempts, however depending on your case, it may be more appropriate to set the job class' `$tries` property.
+
+If you do not specify a value for the `tries` option, jobs will only be attempted once or as many times as specified by the job class `$tries` property.
+
+<a name="job-timeout"></a>
+### Job Timeout
+
+Similarly, you can set a `timeout` value at the supervisor level, which specifies how many seconds a worker process can run a job before it’s forcefully terminated. Once terminated, the job will either be retried or marked as failed, depending on your queue configuration.
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...¨
+            'timeout' => 60,
+        ],
+    ],
+],
+```
+
+>[!WARNING]
+> The `timeout` value should always be at least a few seconds shorter than the `retry_after` value defined in your `config/queue.php` configuration file. Otherwise, your jobs may be processed twice.
+
+<a name="job-backoff"></a>
+### Job Backoff
+
+You can define the `backoff` value at the supervisor level to specify how long Horizon should wait before retrying a job that encounters an unhandled exception:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'backoff' => 10,
+        ],
+    ],
+],
+```
+
+You may also configure "exponential" backoffs by using an array for the `backoff` value. In this example, the retry delay will be 1 second for the first retry, 5 seconds for the second retry, 10 seconds for the third retry, and 10 seconds for every subsequent retry if there are more attempts remaining:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'backoff' => [1, 5, 10],
+        ],
+    ],
+],
+```
 
 <a name="dashboard-authorization"></a>
 ### Dashboard Authorization

queues.md

@@ -577,6 +577,8 @@ public function middleware(): array
 }
 ```
 
+Releasing an overlapping job back onto the queue will still increment the job's total number of attempts. You may wish to tune your `tries` and `maxExceptions` properties on your job class accordingly. For exemple, leaving the `tries` property to 1 as it is by default would prevent any overlapping job to be retried later.
+
 Any overlapping jobs of the same type will be released back to the queue. You may also specify the number of seconds that must elapse before the released job will be attempted again:
 
 ```php
@@ -1213,7 +1215,22 @@ class ProcessPodcast implements ShouldQueue
 <a name="max-attempts"></a>
 #### Max Attempts
 
-If one of your queued jobs is encountering an error, you likely do not want it to keep retrying indefinitely. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted.
+Job attempts are a core concept of Laravel's queue system and power many advanced features. While they may seem confusing at first, it's important to understand how they work before modifying the default configuration.
+
+When a job is dispatched, it is pushed onto the queue. A worker then picks it up and attempts to execute it, this is known as a job attempt.
+
+However, an attempt does not necessarily mean the job’s handle method was executed. Attempts can be consumed in several ways:
+
+- The job's `handle` method runs and completes without throwing an exception.
+- The job encounters an unhandled exception during execution.
+- The job timed out.
+- The job is manually released back to the queue using `$this->release()`.
+- Middleware such as `WithoutOverlapping` or `RateLimited` fails to acquire a lock and releases the job.
+
+You likely do not want it to keep retrying indefinitely a job. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted.
+
+> [!NOTE]
+> By default, Laravel will only attempt a job once. If your job uses middleware like `WithoutOverlapping` or `RateLimited`, or if you're manually releasing jobs, you’ll likely need to increase the number of allowed attempts.
 
 One approach to specifying the maximum number of times a job may be attempted is via the `--tries` switch on the Artisan command line. This will apply to all jobs processed by the worker unless the job being processed specifies the number of times it may be attempted:
 
@@ -1370,6 +1387,10 @@ If you would like to indicate that a job should be marked as [failed](#dealing-w
 public $failOnTimeout = true;
 ```
 
+> [!NOTE]
+> By default, when a job times out, it consumes one attempt and is released back to the queue (if retries are allowed).
+> However, if you configure the job to fail on timeout, it will not be retried, regardless of the value set for tries.
+
 <a name="error-handling"></a>
 ### Error Handling
 
@@ -1417,6 +1438,9 @@ public function handle(): void
 }
 ```
 
+> [!NOTE]
+> A job marked failed will never be retried.
+
 If you would like to mark your job as failed because of an exception that you have caught, you may pass the exception to the `fail` method. Or, for convenience, you may pass a string error message which will be converted to an exception for you:
 
 ```php
@@ -2249,6 +2273,18 @@ class ProcessPodcast implements ShouldQueue
 > [!WARNING]
 > A new instance of the job is instantiated before invoking the `failed` method; therefore, any class property modifications that may have occurred within the `handle` method will be lost.
 
+A failed job is not necessarily one that encountered an unhandled exception. A job is considered failed when it has exhausted all of its allowed attempts. These attempts can be consumed in several ways:
+
+- The job encounters an unhandled exception during execution.
+- The job timed out.
+- The job is released back to the queue either manually or by a middleware.
+
+If the final attempt fails due to an exception thrown during job execution, that exception will be passed to the job’s failed method.
+
+However, if the job fails because it has reached the maximum number of allowed attempts, the `$exception` will be an instance of `Illuminate\Queue\MaxAttemptsExceededException`.
+
+Similarly, if the job fails due to exceeding the configured timeout, the `$exception` will be an instance of `Illuminate\Queue\TimeoutExceededException`.
+
 <a name="retrying-failed-jobs"></a>
 ### Retrying Failed Jobs