[marginalia_nu]

I couldn't run search.marginalia.nu without it. I've seen up to 50,000 bot queries per hour (and peak out at about 500 human queries per hour). I don't have the hardware to cater to the bots. I also don't have the money to buy the hardware to eat the cost. The options are hide behind cloudflare or shut down the service.

It's not about traffic costs, but processing power.

[betaby]

Can you please explain what exactly bot were doing? What was their goal? Yes, I've seen bot scraping sites, which is expected. But what queries bots were doing towards niche search engine?

[marginalia_nu]

Search queries look like spam, like the sort of spam keywords you will find in comment spam. "Free cialis 50mg online pharmacy near me"-type stuff

Best guess is they're gambling I'm backed by Google's API and trying to poison their suggestion data.

[betaby]

Sorry I don't follow. Could please elaborate. You mean bots do query 'cialis' to get an ad-sense ad, while they are the same guys benefiting from ads shown? Or what? I genuinely want to understand the problem and most importantly the motivation.

[marginalia_nu]

I don't understand the motivation either, but I think what they are attempting is to make e.g. typing cialis into Google suggest specific queries like the one i showed, which may be so overspecified they provide the spammers' links.

That's my theory anyway.