Dashboard out of memory due to heavy health checks

[ilnytskyi]

Preconditions

Magento Version : 2.4.7+

ElasticSuite Version : 2.11.12.1

Environment : any

Third party modules :

Steps to reproduce

1. Set lower memory limit on request or preload other big caches
2. Try login into admin

Expected result

1. Admin dashboard loads ok

Actual result

1. Admin does not load

{
  "class": "Exception",
  "code": 0,
  "trace": [
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Helper/Cache.php:113",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Model/ProductMetadata/ComposerInformation.php:86",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Model/Healthcheck/HyvaCompatibilityCheck.php:140",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Model/Healthcheck/HyvaCompatibilityCheck.php:167",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Model/Healthcheck/HyvaCompatibilityCheck.php:154",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Model/Healthcheck/HyvaCompatibilityCheck.php:83",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Model/Healthcheck/HealthcheckList.php:89",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Block/Adminhtml/Healthcheck/MenuDecorator.php:89",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/Block/Adminhtml/Healthcheck/MenuDecorator.php:73",
    "/mage_root/devel1746446350/vendor/smile/elasticsuite/src/module-elasticsuite-core/view/adminhtml/templates/healthcheck/menu_decorator.phtml:18",
    "/mage_root/devel1746446350/vendor/magento/framework/View/TemplateEngine/Php.php:67",
    "/mage_root/devel1746446350/generated/code/Magento/Framework/View/TemplateEngine/Php/Interceptor.php:46",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Element/Template.php:263",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Element/Template.php:293",
    "/mage_root/devel1746446350/vendor/magento/module-backend/Block/Template.php:141",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Element/AbstractBlock.php:1128",
    "/mage_root/devel1746446350/vendor/magento/framework/Cache/LockGuardedCacheLoader.php:136",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Element/AbstractBlock.php:1149",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Element/AbstractBlock.php:676",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:578",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:555",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:510",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:606",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:557",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:510",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:606",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:557",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:510",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:606",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:557",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:510",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:606",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:557",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:510",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:606",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:557",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:510",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Layout.php:975",
    "/mage_root/devel1746446350/generated/code/Magento/Framework/View/Layout/Interceptor.php:130",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Result/Page.php:260",
    "/mage_root/devel1746446350/vendor/magento/framework/View/Result/Layout.php:171",
    "/mage_root/devel1746446350/generated/code/Magento/Backend/Model/View/Result/Page/Interceptor.php:107",
    "/mage_root/devel1746446350/vendor/magento/framework/App/Http.php:120",
    "/mage_root/devel1746446350/generated/code/Magento/Framework/App/Http/Interceptor.php:55",
    "/mage_root/devel1746446350/vendor/magento/module-application-performance-monitor/Plugin/ApplicationPerformanceMonitor.php:38",
    "/mage_root/devel1746446350/generated/code/Magento/Framework/App/Http/Interceptor.php:53",
    "/mage_root/devel1746446350/vendor/magento/framework/App/Bootstrap.php:264",
    "/mage_root/devel1746446350/pub/index.php:30"
  ]
}

I see that problem happens near trying to load and userialize caches

We have 30+ stores it seems to load checks for many indexes

Problem of out of memory happens here, just after loading from cahce, during unserialization.

\Smile\ElasticsuiteCore\Helper\Cache::loadCache

And for some reason my debug logger noticed recorded over 10K call of that functions. Mainly for indexes, that confirmed by screen from spx trace.

Is there a way to disable it, or improve or do something to avoid the oom problems?

it also adds a lot of time to dashboard load to run all checks.

We also have debug mode disabled

But the config is quite strangely triggers exactly when it's disabled

As a workaround I only see posibility to remove this block from default.xml for now.

<block class="Smile\ElasticsuiteCore\Block\Adminhtml\Healthcheck\MenuDecorator"
                   name="healthcheck.menu_decorator"
                   template="Smile_ElasticsuiteCore::healthcheck/menu_decorator.phtml" />

[ilnytskyi]

I think for quick fix, instead of using default.xml the block can be put into it's own controller and loaded via ajax.
This at least won't break the admin panel. As according to source code the solution was known to be controversial.

[rbayet]

Hello @ilnytskyi,

Concerning

But the config is quite strangely triggers exactly when it's disabled

Actually the idea is that IF you have the Elasticsuite debug log enabled (which log all queries), the decorator is disabled in order NOT TO run all the checks which can be query intensive.

With regards to

I think for quick fix, instead of using default.xml the block can be put into it's own controller and loaded via ajax.

This will not address the main issue, but yes, it will quarantine it to a separate HTTP/PHP process.

Regards,

[ilnytskyi]

@rbayet

This will not address the main issue, but yes, it will quarantine it to a separate HTTP/PHP process.

Yes that would be much safer than blocking the admin panel.

[rbayet]

Hello @ilnytskyi,

Pull request #3610 should dramatically improve things by reducing the amount of memory and most importantly the number of calls to the CacheHelper and the unserialize method.
The checks will still be placed in the same calling stack than the rendering of the admin panel (no separation yet), but it should be night and day.

You can have a go if you wish by using a composer patch with the PR's content.

Regards

[rbayet]

Hello @ilnytskyi ,

I've noticed today that the pull request might incorrectly display some live indices as "Closed" in the Elasticsuite > System > Indices screen.
This is addressed by an additional commit which completely remove cache (in the cache storage sense) usage.

Regards,

[github-actions]

This issue was waiting update from the author for too long. Without any update, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away! Thanks for your contribution.