What do the different memory limit arguments mean ? [Horizon]

[CharlesBilbo]

Currently, in the horizon config, there is a memory limit in the root of the config and a memory limit for each supervisor instance. and I'm not quite sure what the differences are

And then there is the following in the supervisor config

Does this mean that each job has a max allowed memory of 128 MB if yes what is the memory limit for the 64MB do?

Thanks !

[juse-less]

I hope I got it all right. Let me know otherwise, and I'll correct it.

---

In order to understand the difference and make sense of it all, we need to understand how processes are 'linked together', and the meaning of soft- vs hard limits, and where they are used.

---

Soft limit means that the process won't be abruptly killed, and is allowed to go above that limit.
Hard limit means that the process will be abruptly killed.

---

I believe the Horizon 'master' supervisor (1st screenshot) is the artisan horizon process.
That one spins up a new process for each supervisor defined in the 2nd screenshot (a hidden artisan horizon:supervisor command).
Each supervisor in turn spins up N new worker processes, depending on your config (a hidden artisan horizon:work command).

Each supervisor appears to be a fork of the master process spawning it (a child process).
Each worker appears to be a fork of the supervisor process spawning it (^).

This forms a process tree along the lines of

artisan horizon ('master'?)
|- artisan horizon:supervisor ('supervisor-1')
  |- artisan horizon:work ('worker-1')
  |- artisan horizon:work ('worker-2')

---

The 'master' process has a soft limit of memory_limit, as defined in the 1st screenshot (64MB by default).
Each individual 'supervisor' and 'worker' processes has a soft limit of memory_limit, as defined in the 2nd screenshot (128 MB by default).

The master process doesn't do much work other than managing the supervisors, AFAIK, and therefore doesn't need much RAM.

---

If Horizon notice that the process is above the soft limit, when it does the check(s), it'll send a termination command for that process to terminate it 'gracefully'.
The process will then be started again, if needed.
This can be used to handle memory leaks, or so, over time, to prevent the process(es) from hitting the hard limit.

To see how it monitors the 'master' process, you can have a look at this event listener.
I'm not entirely sure how it does the worker processes, but to see how it monitors the 'supervisor', you can have a look at this event listener.

---

The memory limit for an actual job is the hard limit, managed by PHP.
This can be set in the php.ini config, or set dynamically in scripts using ini_set('memory_limit', <your limit here>), as well as on the command line using php -d memory_limit=<your limit here> ....

[rpungello]

as well as on the command line using php -d memory_limit= ....

I tried something like this, but the problem ends up being this only configures the limit for the master supervisor. When that supervisor spawns the child supervisors, and when they in turn spawn the workers, each of those is its own PHP process that doesn't (at least based on my testing) honor any runtime arguments provided to the master supervisor.

[HugoHeneault]

Hello @rpungello

Did you find a solution to set a per-worker hard limit, so that the whole horizon is not going down if one job has a sudden memory consumption?

Thanks 🙏

[rpungello]

@HugoHeneault I was never able to find a great solution.
What I ended up doing was adding a runtime option to the Docker entrypoint of my application that checks for a MEMORY_LIMIT environment variable. If present, it writes out a .ini file to the PHP include config include dir that sets the memory_limit variable to the specified amount.

This allows the same container image to be shared across web and worker, with higher limits for workers. I don't love this though as I'd rather the entrypoint simply be the horizon command rather than need a wrapper around it to set the limits ahead of time.

It does suffice for my needs though.