[RobSm]

No. robots.txt is not something that is defined and enforced by the law. Just because someone came up with some 'recommendation' like robots.txt does not mean this is the law

[shukantpal]

> Legal? Yes. Parasitic behavior? Also yes.

You failed to make a meaningful counterpoint; the legal/ethical distinction was made clear in the parent post.

[LamaOfRuin]

As a matter of fact, robots.txt is a well understood expression of intent which is legally meaningful in a lot of contexts.