Hacker News new | ask | show | jobs
by gchq-7703 1107 days ago
Interesting. What makes you want to switch to the file system? I wrote one for a project[0] a while back (for MongoDB) and it didn't seem like the database introduced too much complexity. I didn't write the implementation from scratch, but the couple hundred lines of code were easy to reason about.

[0] https://github.com/gchq/Bailo/tree/main/lib/p-mongo-queue
1 comments
andrewstuart 1107 days ago
The filesystem means zero configuration.

I found almost all message queues to be horribly complex to configure, debug and run. Even database queues require a lot of config, relative to using the file system.

I did actually write a file system based message queue in Rust and it instantly maxed out the disk at about 30,000 messages a second. It did about 7 million messages a second when run as a purely RAM message queue but that didn’t use file system at all.

It depends what you’re doing of course… running a bank on a file system queue wouldn’t make sense.

A fast message queue should be a tiny executable that you run and you’re in business in seconds, no faffing around with even a minute of config.

I just hate configuration in general.

link
orangepurple 1107 days ago
> running a bank on a file system queue wouldn’t make sense

Bank transfers are done by a text file sent over FTP (file transfer protocol; circa 1971)

https://orientalbank.com/assets/Pdfs/ach/ACH_ORIGINATION_AGR...

link
lmz 1107 days ago
Some of them are more advanced and use SFTP (circa 1997).
link
Izkata 1107 days ago
> I did actually write a file system based message queue in Rust and it instantly maxed out the disk at about 30,000 messages a second. It did about 7 million messages a second when run as a purely RAM message queue but that didn’t use file system at all.

Did you try an in-memory filesystem through tmpfs?

link
sitkack 1107 days ago
Database config should be two connection strings, 1 for the admin user that creates the tables and anther for the queue user. Everything else should be stored in the database itself. Each queue should be in its own set of tables. Large blobs may or may not be referenced to an external file.

Shouldn't a message send be worst case a CAS. It really seems like all the work around garbage collection would have some use for in-memory high speed queues.

Are you familiar with the LMAX Disruptor? Is is a Java based cross thread messaging library used for day trading applications.

https://lmax-exchange.github.io/disruptor/

link