[marginalia_nu]

> So spammers have latched onto your search engine because they are getting useful results.

I'm not sure about this. At least with my search engine, it doesn't really seem to matter what response they get, I don't even think they look at the responses. They keep hammering away with tens of thousands of queries per day with the requests even though they've seen nothing but HTTP Status 403 since last October or so.

My best guess is they're going after search engines in general in case they forward queries to google, in order to manipulate their typeahead suggestions.

[miohtama]

Put a CloudFlare web application firewall at the front of the site and then use its rate limited / CAPTCHA features to throttle traffic. It is the easiest way to get rid of parasitic scraping and API abuse. Cost is $0.

[marginalia_nu]

Yeah, that's essentially what I've done, except I'm paying for their cheapest non-free tier to have a bit more control over it. I really wish I didn't have to route all my traffic through an untrusted a 3rd party like that, but I guess we can't have nice things on the Internet anymore.

[B1FF_PSUVM]

> I guess we can't have nice things on the Internet anymore

Not since it left the larval stage and became "pay for play", no.

Oh, well, those taxpayer-funded years were nice for those of us who were around.

[bcrosby95]

I think I remember wondering, after the dotcom bust, if the whole web thing would actually take off.

The reasoning I vaguely remember reading was that the internet required government subsidy to exist - at first directly, then in the form of universities, and the bust was a sign that it couldn't exist without one.

I don't remember how prevalent the view was at the time though. Obviously it turned out to be wrong.

[kordlessagain]

Putting authentication on the site would be easier.

There a rub here, in that people expect to search things without being logged in. But then if you don't log in people, anyone can come calling, including bots. This then causes you to do things like get a third party to filter the data, which then affects the users by having to reroute their traffic to someone else to get rid of some of the visits you don't want from the bots.

And round and round.

Simple authentication to the site with tokens might solve the problem. If an IP comes calling that does so with out authentication, or payment, then hang the connection.

[Schroedingersat]

> Cost is $0.

Cost is the slow enclosure of the internet hy a handful of giant companies and once attestation is universal having anyone without a locked down device be locked out of most of the internet without providing endless free labour.

FTFY

[MicahKV]

Huh, well I guess there goes my theory about the incentive. What a bummer. I would have thought that at least with search engine scraping, they would stop expending the effort once the results dried up.