Hacker News new | ask | show | jobs
by super256 801 days ago
> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching

They probably ignore robots.txt