| I think you are trying to claim that Google goes further than DVD or netflix, but this analogy is really not working for you. DVDs have technological protection as well -- the CSS[0] system. So yes, if you don't want your movie to be pirated you need to explicitly enable this. This was probably harder than creating robots.txt too, there were NDAs and stuff involved. The netflix requires logging in to access the content. If you add the same requirement, then Google is not going to take your snippets. Unlike the string "nosteal", the robots.txt file is not Google invention, it is as much part of the web standards as all other technologies. If you want a website, you need a server which can support HTTP, HTML, CSS, links, robots.txt and so on. You can omit parts you don't need, but then you _may_ suffer the consequences -- without CSS your site will be ugly, and without robots.txt your site will be scraped by Google. [0] https://en.wikipedia.org/wiki/Content_Scramble_System |