much higher CPU usage after upgrading from php:8.3-v4-slim-apache to 8.4-v5-slim-apache

[jdmcalee]

I discovered that our app's CPU usage has a significant increase in CPU usage after switching to the 8.4-v5 image (see graph below). I noticed that the JIT ini defaults change resulted in JIT being enabled, so I explicitly disabled it during our image build. Unfortunately, that had no impact on CPU usage. We did not make any changes to our application when we changed to the new image. Any ideas what might be going on, or where else to look to figure this out?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please update it if any action still required.

[jdmcalee]

would still appreciate any ideas

[mistraloz]

Hello sorry i was not see your message before. That is an interesting observation. No regression is expected during the change from V4 to V5 (expect the one listed during the announcement but not related with that).

What is your context ? Please lets me know some information about your metrics on your image capture (that is during units tests ? on live env ?). I may need a starting to 1/ confirm your observation 2/ identify a possible root cause.

PS: Your upgrade from php 8.3 to 8.4 maybe the root cause : did you tried to execute your old code with php:8.3-v5-slim-apache ? Or the new one with php:8.4-v4-slim-apache ?

[jdmcalee]

Sure, sorry I didn't include more detail initially. On June 10th, we switched from 8.3-v4-slim-apache to 8.4-v5-slim-apache. At the time, it was because we started to have some problem with the v4 image, but unfortunately I noticed the problem on my local machine and since the newer image worked fine, I didn't think anything of it and have no way to find the error now.

We're running our app as an ECS service in AWS (Fargate). That snip is from the ECS console showing our production service. We had been running with four containers/tasks with each having fairly low resources, either .25 or .5 vCPU. I got notified that users were starting to receive timeout errors, and that's when I noticed the difference. To get things back up and running I just added cpu resources and things have been stable since then.

These are the arg/envs we use to configure the image in the container build:

ARG PHP_EXTENSIONS="bcmath"

APACHE_DOCUMENT_ROOT="public/"
APACHE_EXTENSION_HEADERS=1
APACHE_EXTENSION_SOCACHE_SHMCB=1
APACHE_EXTENSION_SSL=1
APACHE_EXTENSION_REMOTEIP=1
APACHE_RUN_USER=www-data
APACHE_RUN_GROUP=www-data
TEMPLATE_PHP_INI="production"
PHP_INI_OPCACHE__VALIDATE_TIMESTAMPS="off"
PHP_INI_OPCACHE__JIT="disable"
PHP_INI_MAX_EXECUTION_TIME=120

We did not try the 8.3-v5 image because we had already made use of some 8.4 features by the time we noticed this issue. If it ends up being necessary we can consider downgrading but I'd rather not as a first step.

[mistraloz]

I did not received any feedback like that so yes, if you are sure that is linked to the v5 (and not to php 8.4), i can try to reproduice but like that, that look like too specific at your case. We may just let's it opened and see. Or if you can rollback and compare with the same code base in php8.3 the v4 and the v5 locally, that could be helpful.

[jdmcalee]

Ok, I just tried to build our image using 8.4-v4-slim-apache and it failed like back in June. There's a problem with the bcmath module:

7.317 Setting up php7.4-bcmath (7.4.3-4ubuntu2.29) ...
7.399 
7.399 Creating config file /etc/php/7.4/mods-available/bcmath.ini with new version
7.457 Setting up php-bcmath (2:7.4+75) ...
7.583 + '[' -n '' ']'
7.583 + '[' -n '' ']'
7.583 + '[' -n bcmath ']'
7.583 + phpenmod -v 8.4 bcmath
7.589 WARNING: Module bcmath ini file doesn't exist under /etc/php/8.4/mods-available
7.589 WARNING: Module bcmath ini file doesn't exist under /etc/php/8.4/mods-available
7.589 + /usr/bin/real_php -m
7.589 + grep bcmath
------
Dockerfile:14
--------------------
  12 |     RUN yarn build
  13 |     
  14 | >>> FROM thecodingmachine/php:8.4-v4-slim-apache
  15 |     
  16 |     USER root
--------------------
ERROR: failed to build: failed to solve: process "/bin/bash -o pipefail -c sudo -E PHP_EXTENSIONS=\"$PHP_EXTENSIONS\" /usr/local/bin/install_selected_extensions.sh" did not complete successfully: exit code: 1

The same thing occurs with 8.3-v4-slim-apache. That's why I went ahead and bumped us up to 8.4 and v5 at the same time; my priority was going to 8.4.

I'm going to deploy our app on 8.3-v5 tomorrow morning. I'll follow up with metrics after some time passes.

[jdmcalee]

So far, it looks like that caused a slight increase in CPU usage, which isn't surprising if the only change was going from 8.4 back to 8.3:

Unfortunately, since June we did add a hard dependency on the bcmath extension, so I can't switch back to the v4 images as long as that breaks the build.