[altdataseller]

What user agent does it use and do you obey robots.txt?

* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK

[super256]

> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching

They probably ignore robots.txt