|
|
|
|
|
|
by WJW
1262 days ago
|
|
|
A conduit pulls information from earlier conduits in the same pipeline, only requesting the next item when it needs to do more work in order to process a request for a value from a conduit later in the pipeline. This means that inside the conduit itself there is perfect backpressure: you cannot overload any buffers between conduits because there aren't any. Of course, depending on the rest of the system you could still have a bottleneck somewhere. For example: if you pull jobs from a queue in (say) Redis but don't have enough capacity to process jobs faster than they are enqueued, the queue will eventually fill up. |
|
|
For parallel processing the backpressure becomes a bit more complicated, in particular if you end with a sequential consumer. If you're just doing your parallel tasks when a sequential consumer demands it, you'll find yourself still doing one task at a time. The straightforward solution is to work ahead a bit, spawning just enough parallel work to fill a small buffer that the consumer can read from. You get backpressure this way, but may have done a little bit too much if the consumer doesn't want the rest of the data. We'll show this in the third blog post of the series.