[thinkingkong]

I'll try and answer your 3 questions.

1. As with most things it depends, but I'll make the mistake of assuming you don't need all of this data in memory and you won't be iterating over the entire set of data on a regular (<1m) basis. In that case, it most likely makes sense to move to a relational database. You should ask yourself what kind of queries you're running on the data and pick something appropriate. But for the data you're listing, then yeah - go with a relational DB.

2. Again "It depends". Postgresql is the most popular alternative to MySQL and I'd say at this point is more in vogue with the developer community. It's supported on AWS RDS and Heroku has a great postgres.app for OSX. Your data is so simple that I'd encourage you to start there. In the future when you have specific needs then you can research more appropriate database technologies.

3. From what you've shown us then you should most likely move to a relational database. The size should shrink dramatically, too.

[justinsb]

If you're iterating over the entire dataset regularly though, you should _definitely_ use a relational database over Redis, because a relational database lets you push processing onto the server. With redis you'll have to fetch every row over the network; with a SQL database you'll typically only retrieve the final summary data.

The OP has 40GB of data, so to query that even once per minute with Redis would effectively max out 10Gb ethernet.

Of course, the real win is that something that would be a table scan in Redis could be a indexed query in SQL. Doing a table scan over 40GB of data is always going to be painful.

[resca79]

>With redis you'll have to fetch every row over the network; >with a SQL database you'll typically only retrieve the final >summary data.

Redis supports filter features native, so it basically works as every remote service.

[justinsb]

Do you mean HSCAN with MATCH? Thanks for reminding me about that. I think it's so limited that the point still stands though.

[itamarhaber]

You can actually push all the processing to Redis with Lua without having to fetch a single "row" over the network... I'm not recommending this as a rule of thumb - on the contrary, you could easily tie down your (any type of) DB with application logic - but in some cases it makes sense.