[midnightclubbed]

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful.

I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great.

I looked for an embedded Redis fork and came up blank, do you have links? I found Vedis, but I would rather have something built off of the Redis code than a re-implementation.

[antirez]

Sorry I don't have links since I did not track such forks in the past. However I've a question: for your use case, isn't it an option to have a library that looks like Redis from the POV of the API, but actually stores objects in memory as data structures native to your programming language? This way the API looks like a mental proxy for the DSL to access Redis and the time complexity you expect from given operations, but you are just writing to local objects.

[midnightclubbed]

That's an option but I would rather not re-invent the wheel unless necessary!

The current use of sqlite is to allow our scripted code (lua and actionscript) to make queries of the exposed data without having to write C++ code for every possible query and data object type (and implement new ones on demand).

Redis might not be the correct thing for this exact use case (some of the queries are more complex than a simple key or range look-up) but I may be prepared to take those limitations in exchange for a substantial speed and/or memory use improvement.

[e12e]

It sounds to me like there might be some area where sqlite is "to much" but lightningdb/berkleydb/toky cabinet is "too little".

I would be surprised if "actual Redis" was ever the right answer to "sqlite is too much".

But I do wonder if there are some lessons to take from Redis api and wrap something like lmdb/bdb etc.

I'm not familiar enough with Redis to know when/if this would make sense over just using sqlite, though.

[comex]

One limitation of SQLite is that it doesn’t support any kind of “notify me when some other process does X” operation. (If you Google it, you’ll find sqlite3_update_hook, but that only works for updates performed by the same process.) If you want to use SQLite as an event queue, you can have one process writing rows to a table and another process reading them, but you need some external signal to tell the second process “wake up, there’s new stuff in the queue”. Or you can have it poll on a timer, but that’s suboptimal in many different ways.

Which is topical, because watching for updates is a core feature of Redis streams (and Redis already had pub/sub channels before that). For that use case, SQLite is too little, even if your needs are otherwise quite basic.

Unfortunately, this difference in capabilities seems to be partly a result of limitations in the underlying OS APIs. SQLite uses POSIX advisory locks to lock ranges of the database file, but I don’t think there’s any similar API that provides an event or semaphore associated with a given file, instead of a lock. There are plenty of messaging APIs that aren’t associated with an arbitrary file – there are semaphores, message queues, and shared memory objects, in fact two sets of APIs for each of those (SysV and POSIX), plus signals, etc. But those all have their own namespaces, and if the two processes trying to synchronize with each other are in different containers, they might not share those namespaces. There are Unix sockets – those are a decent option, but they require one process to set itself up as the server, which is a bit weird in the SQLite model where all the processes are on an equal footing, and any may quit at any time. They also don’t work over NFS (whereas locking does, at least sometimes). You can try to mmap a regular file and then treat it as shared memory, but that’s not guaranteed to work in all cases, and again doesn’t work over NFS. I suppose you could try to abuse a lock as a semaphore, but that has its limitations…

But it’s not like many people use SQLite over NFS anyway. Whatever the approach, I’d love to see a “SQLite for notifications”. It would probably be a pretty simple library, but with the needed bells and whistles like bindings to higher level languages. If a library like this exists, I’d be very interested to hear, because a while back I searched for one in vain.

[midnightclubbed]

In my case the target application is video games (console and PC) so we performance and/or memory usage critical.

I've used Redis very successfully on the backend, so maybe I'm just trying to find some reason, any reason, to play with it in player facing code!

[siscia]

When I get started to build RediSLQ I wanted an interprocess, fast, data store that supported SQL manipulation.

It may be useful to you as well: RediSLQ.com

Or on GitHub: https://github.com/RedBeardLab/rediSQL

Full disclaimer: I am the author

[64738]

Did you mean RediSQL.com? Or is that someone else?

[siscia]

Indeed you are right!

Yes https://redisql.com

[jinjin2]

I’ve used Realm for this very successfully. It is a bit limited in the number of languages it supports (outside of mobile where it seems to support pretty much everything), but it has really nice support for node.js and .net which is where I have used it.

It is pretty cool to be able to share live interconnected objects between processes with full transactional safety.