[nirvdrum]

Plenty of bootstrapped businesses have better things to spend $200 / month on, let alone the time spent trying to figure out where all the anomalous traffic is coming from. As I understand it, it's not simple file fetches either. It's cloning a repo, which involves two-way communication, consumes CPU and RAM, and causes disk seeks. You're not slapping it on CloudFront and calling it a day. Finally, it looks to me like the costs are going to scale the more people he has using sourcehut and writing Go modules.

I don't really understand turning this around on him. Why should he have to subsidize Google? If it's not a problem, why do we have robots.txt at all? Just let bots hammer your site and cope with it.

The current situation can't be the optimal solution. It wasn't even present prior to Go 1.16. Only one company has the ability to change that. What should he do differently here? Why should he have to spend any of his time or money working around an issue he didn't create?