[🐛 Bug]: [dotnet] High memory usage after multiple PageSource access

[dberubecoveo]

Description

I noticed that when using the same ChromeDriver to retrieve multiple PageSource from different large pages, the memory consumption of my application was increasing drastically like there is some kind of memory leak.

After some digging, I found out that it seems to be coming from the usage of SharedArrayPool from System.Text.Json (which was introduced with Selenium 4 vs Selenium 3 that was using Newtonsoft, where I cannot reproduce the issue).

The SharedArrayPool is meant to improve memory usage by avoiding constant memory allocation and deallocation, but it looks like it just use a new chunk of memory instead of reusing it.

I was able to workaround this by overriding the HttpCommandExecutor handling of the Response and simply avoid deserializing the {"value":"<html....</html>"} to return the value content string directly. But that's not really a long term solution.

Reproducible Code

var driverService= ChromeDriverService.CreateDefaultService();
driverService.HideCommandPromptWindow = true;

var options = new ChromeOptions();
options.AddArgument("--headless");
options.AddArgument("--no-sandbox");
options.AddArgument("--disable-gpu");
options.AddArgument("--disable-dev-shm-usage");

using var webDriver = new ChromeDriver(driverService, options, TimeSpan.FromSeconds(60));

foreach (var url in GetUrls()) // GetUrls returns a list of large page URL.
{
    await webDriver.Navigate().GoToUrlAsync(url);
    var pageSource = webDriver.PageSource; // This is what is causing the high memory consumption.
    ...
}

[selenium-ci]

@dberubecoveo, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[nvborisenko]

@dberubecoveo thanks for the case. Let's be more specific: when you say "High memory usage", what exactly numbers you mean?

[dberubecoveo]

@dberubecoveo thanks for the case. Let's be more specific: when you say "High memory usage", what exactly numbers you mean?

Let's say the pages I visit are about 3.5 MB to 4.5 MB of content fully rendered.

When getting the PageSource of a single page, my little test application reaches 15 MB (perfectly fine), but after 50 pages, it reaches 1.5+ GB of RAM.

[nvborisenko]

I simulated your case and don't see "high" memory usage (Selenium.Webdriver v4.34). My simulation navigated to 500 web sites, using BiDi approach and Classic.

In any case used memory is dropped.

I can guess:

1. You have 100GB total of RAM, so .NET decides to use 1.5GB without any impact
2. When you get var pageSource = webDriver.PageSource;, are you sure your app doesn't hold a reference to allocated pageSource?

[dberubecoveo]

In my test app, I literally do nothing with the pageSource, so it can't really keep an hold on that. Also, even if the pageSource content was kept in memory, that wouldn't reach 1.5GB in any case.

So I guess your first assumption is similar to mine, if I trigger the GC to collect multiple times, it will eventually go down. When profiling, I see the problematic parts are in the LOH.

But I also observe that high memory usage on servers where the RAM is limited and sometimes ran into OOM.

I tested to limit the amount of memory available to my app and couldn't go under 450MB without triggering an OOM related to SharedArrayPool. But it kind of shows where the issue comes from...

AppContext.SetData("GCHeapHardLimit", (ulong)450 * 1024 * 1024);
GC.RefreshMemoryLimit();

In your simulation (500 pages), are you using different page content and sizes?

Side note: I just tried using the BiDirectional functionality (options.UseWebSocketUrl = true;), it's a bit better, but no miracle here.

[nvborisenko]

My test app:

using var webDriver = new FirefoxDriver(new FirefoxOptions()
{
    UseWebSocketUrl = true,
});

await using var bidi = await webDriver.AsBiDiAsync();

var context = (await bidi.BrowsingContext.GetTreeAsync())[0].Context;

var random = new Random(1000);

var iterations = 500;

// BiDi
for (int i = 0; i < iterations; i++)
{
    var count = random.Next(4 * 1024, 5 * 1024 * 1024);
    var res = await context.Script.EvaluateAsync($"'a'.repeat({count})", false);
}

await Task.Delay(5000);

// Classic
for (int i = 0; i < iterations; i++)
{
    var count = random.Next(4 * 1024, 5 * 1024 * 1024);
    var res = webDriver.ExecuteScript($"return 'a'.repeat({count});");
}

Console.WriteLine($"Done");

I also don't like that memory usage is kind of unexpected. But it is explainable, even if I don't like it.

Really at first glance I don't see criminal in Selenium code. I see that we allocate 3x times more than could, but it is not a criminal (literally saying there should be no difference between 5MB and 15MB).

What is next?.. I think try to test the same code in the environment with less available total memory. If .NET manages allocations properly without OOM, than everything is good, Selenium is good. Otherwise, let's dive deeper...

[nvborisenko]

See, tried without Selenium:

for (int i = 0; i < iterations; i++)
{
    var count = random.Next(4 * 1024, 5 * 1024 * 1024);

    var res = new string('a', count);

    await Task.Delay(10);
}

And we see the same picture:

Peak is 50MB, very similar how BiDi works (which is better than Classic in terms of memory consumption).

[nvborisenko]

Let me show how to get page source via BiDi (I think memory peak should be 2x decreased):

using var webDriver = new FirefoxDriver(new FirefoxOptions()
{
    UseWebSocketUrl = true,
});

await using var bidi = await webDriver.AsBiDiAsync();

var context = (await bidi.BrowsingContext.GetTreeAsync())[0].Context; // gets the 1st tab

webDriver.Url = "https://selenium.dev";

var res = await context.Script.EvaluateAsync("document.documentElement.outerHTML", false);

Console.WriteLine(res.AsSuccessResult().ConvertTo<string>());

Warning: BiDi API is still beta, it might be changed at any time without announcement.