[Bogdanovich]

It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.

[dave_ops]

Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs.

But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.

[dave_ops]

Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.

[Bogdanovich]

There is no benefit in that except faster startup time. As a downside you'll get a lot head/tail db keys updates.

[dave_ops]

Fast start times are a valuable thing for a service component.

Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens.

Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.

[Bogdanovich]

I added some more benchmarks including packing with 200M small 64 byte messages (20Gb) and consumption of that queue. There is no slowdown because of mass delete. https://github.com/bogdanovich/siberite/blob/master/docs/ben...