[sradu]

Paraphrasing newrelic "it takes a village to count our proxy nodes".

We have our placing orders infrastructure on AWS, and whole in-house cloud dedicated to product crawling built on top of Digital Ocean.

[dmritard96]

hm, I didn't mean infrastructure, I mean, did you buy proxy nodes from someone like sslprivateproxy and slap HAProxy in front... Most ecommerce sites wise up to bots crawling them and have a robots.txt that suggests you might not want to...