[dilliwal]

If the robots.txt file has no restrictions parsing and scraping is fine. Of course not all scrappers respect robots.txt but they should But as an internet’s citizen better to always reference the source