In Germany, it is the law. ยง 44b UrhG says (translated):
(1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations.
(2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for text and data mining.
(3) Uses pursuant to paragraph 2, sentence 1, are only permitted if the rights holder has not reserved these rights. A reservation of rights for works accessible online is only effective if it is in machine-readable form.
One person's norm can oppose another's norm. Who defines them? If my "norm" is to stop at a stop sign, and yours is to roll through it, are you the abusive jerk?
"I will do anything that will not literally get me into jail" is a pretty low bar. Most decent people try to do better than that - and that's the only reason society still exists, because there's not enough cops to put all bad people into jail and never will be.
TOS wouldn't be enough but laws wouldn't be enough either. We have thousands of pages of laws, and people violate them all the times, and very often with zero consequences.
> robots.txt. This is not the law
In Germany, it is the law. ยง 44b UrhG says (translated):
(1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations.
(2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for text and data mining.
(3) Uses pursuant to paragraph 2, sentence 1, are only permitted if the rights holder has not reserved these rights. A reservation of rights for works accessible online is only effective if it is in machine-readable form.
-- https://news.ycombinator.com/item?id=45776825