[karaterobot]

> Precisely one reason comes to mind to have ROBOTS.TXT, and it is, incidentally, stupid - to prevent robots from triggering processes on the website that should not be run automatically

Counter-point: I have a blog I don't want to appear on search engines because it has private stuff on it. 25 years ago I added two lines to robots.txt file, and I've never seen it show up on any search engine ever since.

I'm not pretending nobody has indexed my blog and kept a copy of the results. I'm just saying the blog I started in college doesn't show up when you search for my name on Google, which is all I care about.

[mnw21cam]

Admittedly, the whole stupidity of the article aside, they do make a valid point with their "triggering processes" bit. GET should not perform side-effects. That's what POST is for.

Also, for your blog, have you considered password-protecting it? As in, http passwords, which are still (surprisingly) a thing. You can even have a password-free landing page handing the password out to every human that visits, with an onward link to the password-protected site. That should stop the bots but keep letting the humans in.

[zzo38computer]

> GET should not perform side-effects. That's what POST is for.

I agree, although not all triggered processes are things that cause changes; some might be calculations that are unnecessary for crawling, and other stuff like that.

> Also, for your blog, have you considered password-protecting it? As in, http passwords, which are still (surprisingly) a thing. You can even have a password-free landing page handing the password out to every human that visits, with an onward link to the password-protected site. That should stop the bots but keep letting the humans in.

I think it is a reasonable idea, although you can also add text into the password prompt that you can use to figure out the password (as the "realm" text). This also means that you will not need to use cookies.