[k3a]

I think scraping for personal use (not honorig robots.txt) should always be legal unless you are attempting DOS. You are accessing public information, the server is returning HTTP200 and it doesn't matter if you do so using a browser, phantomjs or curl with -A parameter.

A different situation would be scraping a website to make business. Worst being directly using the data - for example those StackOverflow clones with original data doesn't sound ok to me. I am not sure what to think about bots doing various derived work like stats and analysis. I think that if they are part of a business, making money, it shouldn't be legal unless those request are permitted by robots.txt.