|
|
|
|
|
by 1vuio0pswjnm7
1052 days ago
|
|
Curious why cannot selectively block using IP address instead of user-agent string. According to HTTP specification, UA is not a required header. There is certainly no technical requirement for it in order to process HTTP requests. Of course, any website could block requests that lack a UA header. I never send one and it's relatively rare IME to see a site require it, but it's certainly possible. |
|
Also, these search crawls by the browser do not identify themselves beyond the Brave standard UA header, namely a plain Chrome user-agent string.