[bionhoward]

For the thing I’m working on, I have an infinite number of little tasks with potentially shared smaller subtasks.

How could I unleash all the processors on my computer on this workload and allow them to correctly avoid repeated calculation of results of shared subtasks?

For example, I’m using an outbox: im::OrdMap<String, Array2<_>> and a situation might arise where one task could avoid repeating work on a subtask because that’s already in progress elsewhere by waiting for the key/value pair (so that process could do something else)

Would it be worth going to async for that?

How could a worker function know if some key in the outbox was already being calculated and it could work on something else?

How would you share an outbox like that across a bunch of rayon processes communicating with async?

(I’ll read smol docs and try to figure it out but this article made a lot of sense, thank you)

[biorach]

Personally I wouldn't use async but instead use a pool of worker threads passing messages to an orchestrator thread using channels: https://doc.rust-lang.org/rust-by-example/std_misc/channels....

The orchestrator can then keep track of what's going on and avoid duplicating tasks and the workers don't need to worry about any global state.