[coderdude]

Some percentage of online retailers will block their crawler. It's a situation where party A wants party B's data and there is really no reason party B would want party A to have it. It occurs even to small and medium-sized online retailers to try to block pricing crawlers. Yahoo Store devs are irked that you still can't upload a custom robots.txt file to the web root. Forbidding unwanted crawlers from poking around is frequently the reason (applies to stores using the store editor).

Maybe in exchange for the privilege of obtaining pricing data, this service could offer to automatically clean up their product data in exchange. Win-win for everyone.