[gabi38]

Very needed book. In particular I am interested in limiting message queues in Erlang. I still don't get it - How one should deal with situations where messages are produced constantly in higher rate than consumed.. There is no protection in Erlang for this common scenario (I think)

[mononcqc]

You have to make the producer synchronous to the consumer so the consumer is slowed down. Ask for an acknowledgement on reception of any task/event. This will instantly limit the rate at which data can come in the final worker.

[gabi38]

Well, that breaks the whole philosophy of async message passing in Erlang, where sending message always succeeds and is not dependent on the receiver. What you are suggesting is in fact limit the queue to be of size of 1 (if only one producer, or to the number of producers). This is not what queue is all about

[mononcqc]

No, this doesn't break the async philosophy. The synchronous sequence of calls is done by a question/reply made out of 2 asynchronous calls. It's just that simple to make something synchronous, as opposed to turning something synchronous asynchronous (by using a queue).

By doing this you are not limiting the queue to the size of 1. If you have N producers, you can have more than N elements in the queue, but the idea is that each producer can only enter them one by one, with the agreement of the consumer. Nothing would stop the consumer from accepting a thousand of them before waiting before replying, therefore slowing down the producer. The idea is just to limit the rate at which you accept queries by tying it to the speed of the consumer.

If this is not enough, you can look at things like the jobs tool (https://github.com/esl/jobs) to work with scheduling.

By the way, you're mistaken on the idea of message sending always succeeding. The idea is that you will never error out because of trying to send a message (send and pray), not that the message will make it to destination. If you want any guarantee that the message made it, then you need that acknowledgement async message being sent back to you.

[matthiasl]

If you want a message queue bounded to length N, then make the messaging protocol such that the sending process sends up to N messages without an ack.

[gabi38]

The point I am trying to make is that this fundamental feature of message queues (limit their length to prevent the memory exploding) should be provided by Erlang and not reimplemented over and over again by developers.This is really a basic feature in any message queue

[matthiasl]

Why is this a fundamental feature of message queues any more than of other things?

(you can make a data structure which makes memory explode, for instance lists:seq(1, 123456789123456), you can make a program which makes memory explode, for instance by growing the stack enough. Why is a message queue special?)

[mononcqc]

Because what to do when the queue is full is absolutely application specific.

What happens if the message you can't add is a specific system message that allows the suspension of OTP processes for a code upgrade? Do you end up livelocked? How do you propagate back that you couldn't insert your element in the queue? Do you make message passing an operation that can fail? How would that work over the network when the message telling you something failed is lost over a netsplit? Is it a different kind of failure?

This has very deep design repercussions on the language. It's not impossible, but it wouldn't let Erlang be the same.