[krat0sprakhar]

This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]!

Aside: There used to be a site sometime back which used to distribute compiled binaries of Go code for all platforms? Is it still up any chance?

[0] - https://github.com/cockroachdb/cockroach#architecture

[justinsaccount]

http://nsq.io/

http://nsq.io/overview/internals.html

The service you are thinking of might be https://github.com/ddollar/godist

You can see the download links on https://github.com/ddollar/forego use it.

[Bogdanovich]

It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.

[dave_ops]

Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs.

But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.

[dave_ops]

Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.

[Bogdanovich]

There is no benefit in that except faster startup time. As a downside you'll get a lot head/tail db keys updates.

[dave_ops]

Fast start times are a valuable thing for a service component.

Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens.

Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.

[Bogdanovich]

I added some more benchmarks including packing with 200M small 64 byte messages (20Gb) and consumption of that queue. There is no slowdown because of mass delete. https://github.com/bogdanovich/siberite/blob/master/docs/ben...