Hacker News new | ask | show | jobs
by cjonas 528 days ago
How do you get by when every major sites starts blocking headless browsers? A good example right now is Zillow, but I foresee a world where big chunks of the internet are behind captcha and bot detection
1 comments
andrewmcwatters 528 days ago
That's not really a problem for Stagehand. It's a problem for Selenium, Playwright, Puppeteer and others at the browser automation library level.
link
cjonas 528 days ago
it's not really a problem for Playwrite, because Playwrite is really intended to be run by the owners of the website, not as a webscraper.

It may become a real problem for the usefulness of this style of LLM driven browsing.

link
keyserj 527 days ago
Playwright does have some docs on scraping, which makes it seem like they do want to support that as a use case https://playwright.dev/docs/docker#crawling-and-scraping. I'm unfamiliar with that though and wouldn't be able to speak to addressing issues with scraping
link
andrewmcwatters 528 days ago
Right, but I mean it's a plugin issue outside of Stagehand. That's currently how the space treats the issue.

There are plugins that extend the major browser automation libraries to do this sort of thing.

And undetectable browser automation doesn't use Playwright, Puppeteer, or Selenium at all.

link