Hacker News new | ask | show | jobs
Built an open-source web scraper for AI agents – seeking feedback (github.com)
2 points by ntbperst 386 days ago
2 comments
ntbperst 386 days ago
Anyone else here building AI agents that need web data?

So I built *AnyCrawl* specifically for AI use cases:

Handles JS-heavy sites (React, Vue, etc.) 20x performance boost through multi-processing Support proxies (no more blocks!) Returns LLM-friendly structured data Open source (MIT) + optional SaaS

link
mtmail 386 days ago
It doesn't seem to respect robots.txt.
link
ntbperst 385 days ago
the coming version will support.
link