Hacker News new | ask | show | jobs
by cpeterso 1050 days ago
Does the Brave crawler send the Googlebot or regular Chrome User-Agent string? If it sends something different than the standard Googlebot User-Agent string, you could dynamically serve a robots.txt that blocks Googlebot to every client besides Googlebot. OTOH, I've read that the Google crawler sometimes users the regular Chrome User-Agent string and penalizes sites that return different content to Googlebot and Chrome.