[karagenit]

Yeah, I'd probably go that route if I wanted to scrape more than the handful of pages I needed for this project. I wonder if it would work on Zillow or not. Even my simple workflow of "click the next button, save the request in devtools, repeat" was suspicious enough to trigger a captcha-type "are you a bot?" challenge. Maybe it was just too many requests quickly like you mentioned, or maybe they're doing something more advanced like mouse movement tracking.

[bob1029]

If you use something like playwright, you can inject events arbitrarily during the scraping session to simulate things a human might do.

https://playwright.dev/docs/api/class-mouse#mouse-move