[mbrumlow]

I don't get why you are being down voted. This is almost exactly what I thought. It's just not that much data given the state is computer hardware.

Where I work we have single nodes processing near that much data a hour -- these are beefy systems though.

[0xbear]

People have just drunk so much “cheap commodity hardware” kool aid by now, they don’t realize there are cheaper and easier ways of doing things now, assuming you have devs who can code and tune for performance. Same with “big data”. Most people have sub-1T datasets. You simply don’t need Spark or anything custom for that.