Google honours robots.txt, X-Robots headers etc but everything else is fair game.
And, as the others state, if it's not robots.txt denied then why not add it to the public index?