|
|
|
|
|
|
by Quarrel
330 days ago
|
|
|
> It was mostly about stopping crawlers from unnecessarily consuming server resources. Very much so. Computation was still expensive, and http servers were bad at running cgi scripts (particularly compared to the streamlined amazing things they can be today). SEO considerations came way way later. They were also used, and still are, by sites that have good reasons to not want results in search engines. Lots of court files and transcripts, for instance, are hidden behind robots.txt. |
|
|
I think this is still relevant today in cases where there are not many resources available: think free tiers, smallest fixed cost/fixed allocation scenarios, etc.