Timeout issue while Scraping many urls in Smart Scraper MultiGraph

[Qunlexie]

Describe the bug
There is a separate timeout issue when using Smart Scraper MuliGraph. It only times out when scraping many links at once usually for >=10. I was able to solve this by increasing the timeout and retry limits.

Scrapegraph-ai/scrapegraphai/docloaders/chromium.py

Line 29 in 6b98d84

TIMEOUT = 10

I am thinking maybe we can parameterize this in some way so that it is easily configurable.

I can probably make a PR if Mods think it is a worthy fix.

[VinciGit00]

Hi @Qunlexie, great idea, please can yo make the PR?

[PeriniM]

Hey @Qunlexie, in the next release you can pass the retry_limit and timeout in the loader_kwargs settings!

graph_config = {
    "llm": {
        "api_key": os.getenv("OPENAI_API_KEY"),
        "model": "openai/gpt-4o-mini",
    },
    "verbose": True,
    "headless": False,
    "loader_kwargs": {
        "retry_limit": 3,
        "timeout": 10,
    },
}

[Qunlexie]

Thanks for fixing this faster than I could. Is this also present in the beta or no?

[VinciGit00]

it should be also in the beta

[PeriniM]

@Qunlexie now it is in the new release! v1.35.0 #869