Hacker News new | ask | show | jobs
by kaivi 1143 days ago
Are there any details about how they're crawling the web? I've never encountered the BraveBot User-Agent and never heard of such crawler.
1 comments

I'm not sure about the crawler itself, but they do get some of their data from the Web Discovery Project.

What percentage of it, I have no idea.

https://support.brave.com/hc/en-us/articles/4409406835469-Wh...