[lorreyfum]

Wouldn’t it just be easier to crawl the net? Not sure what huggingface has to do with anything here.

[croemer]

I guess Huggingface datasets are easy to crawl - those datasets are hosted to be crawled. In contrast to the web at large which will be behind Cloudflare etc