[reallydude]

thats the kind of problem i work on (millions per sec at peak). everything is a queue and we stream from servers via kafka and into processors into redshift, more or less. Its free once it's moving around in AWS, but redshift and processors get expensive and im interested in reliable ad hoc solutions... but havent seen it.

[scapecast]

can you elaborate on the process of copying data into Redshift? How often do you run COPY commands? On a continuous basis? Asking since Redshift as an OLAP isn't really optimized for heavy writes, so if you've figured out a way to do it, please bottle that magic!