[justinsb]

If you're iterating over the entire dataset regularly though, you should _definitely_ use a relational database over Redis, because a relational database lets you push processing onto the server. With redis you'll have to fetch every row over the network; with a SQL database you'll typically only retrieve the final summary data.

The OP has 40GB of data, so to query that even once per minute with Redis would effectively max out 10Gb ethernet.

Of course, the real win is that something that would be a table scan in Redis could be a indexed query in SQL. Doing a table scan over 40GB of data is always going to be painful.

[resca79]

>With redis you'll have to fetch every row over the network; >with a SQL database you'll typically only retrieve the final >summary data.

Redis supports filter features native, so it basically works as every remote service.

[justinsb]

Do you mean HSCAN with MATCH? Thanks for reminding me about that. I think it's so limited that the point still stands though.

[itamarhaber]

You can actually push all the processing to Redis with Lua without having to fetch a single "row" over the network... I'm not recommending this as a rule of thumb - on the contrary, you could easily tie down your (any type of) DB with application logic - but in some cases it makes sense.