Hacker News new | ask | show | jobs
by 1vuio0pswjnm7 1052 days ago
Curious why cannot selectively block using IP address instead of user-agent string. According to HTTP specification, UA is not a required header. There is certainly no technical requirement for it in order to process HTTP requests. Of course, any website could block requests that lack a UA header. I never send one and it's relatively rare IME to see a site require it, but it's certainly possible.
2 comments

This is explained more in the article I referred to, but briefly: Brave delegates crawling to normal Brave browsers, so it's a huge IP addresses pool, not a single IP address or range.

Also, these search crawls by the browser do not identify themselves beyond the Brave standard UA header, namely a plain Chrome user-agent string.

According to Brave Privacy Policy, participation in the Web Discovery Project is "opt-in". How many Brave users have opted in to sending data to Brave.

How many Chrome users have opted in to sending data to Google.

Sometimes uninformed consent is not actually consent. These so-called "tech" companies love to toe that line.