Headless browser mode

[Nv7-GitHub]

Would love a PR that implements headless browser usage, as HTTP scraping does not work anymore

Playwright is probably the best, with a post-install message in setup.py instructing users to run playwright install

[teddybear082]

I’m trying to work on this separately at the moment for an AI skill and at least with selenium, using headless (or non-headless, for testing purposes) browsing mode, I was running into an “are you human” check by google. So they seem to be trying to lock all doors to agent use of search. Was going to maybe look into other search websites to see if that might work.

[vrlishimaihe]

https://github.com/web-agent-master/google-search
The above library is based on Playwright, which stopped updating for months. I tried it yesterday, but it was also broken.

[vrlishimaihe]

Some other similar libraries like whoogle and opensepi, which could returned results but with expired old results.
whoogle is now working but it used the results generated by Mullvad Leta.
opensepi is now working. But the results are not expected by the date filter.
I like our googlesearch most because it could generated results consistent with the real google. Hope we could bypass the google detection.

[Nhqvu2005]

Some other similar libraries like whoogle and opensepi, which could returned results but with expired old results. whoogle is now working but it used the results generated by Mullvad Leta. opensepi is now working. But the results are not expected by the date filter. I like our googlesearch most because it could generated results consistent with the real google. Hope we could bypass the google detection.
can i get link of opensepi

[BenutzerEinsZweiDrei]

It works with

https://github.com/ultrafunkamsterdam/undetected-chromedriver

But i couldnt get headless Mode to work

[BenutzerEinsZweiDrei]

so apparently for headless mode in undetected-chromedriver you need to develop a google-botdetection-bypass there is some considerations here

ultrafunkamsterdam/undetected-chromedriver#589

another option could be playwright-stealth

https://github.com/Mattwmaster58/playwright_stealth