Have you heard of robots.txt files? It pains me to read comments such as yours which display such colossal ignorance.