[rho138]

If it respected the standard then a lack of a robits.txt implies do not crawl, which they openly state they ignore