|
Thank you for the clarification. So if I understood correctly, distributing independant jobs that use async IO between core is okay? For example, I recently wrote a program that reads all the files in a folder, calculates a hash of their contents, and then rename them as their hash. I did this in Go. Since all operations are "perfectly independant", I only had to do synchronization for whole program stuff: make sure the program doesn't exit when goroutines are sleeping, and use channels to not exhaust the file descriptors. From what I understand, I can launch every opening-hashing-renaming operation in a separate goroutine, and Go will take care of making everything async and multicore at the same time. Now let's imagine I want to do the same program in OCaml. I think my options are: - on current OCaml, thread-based concurrency but no parallelism - on current OCaml, monadic concurrency (Lwt, Async) but no parallelism - on multicore OCaml, direct/effect-based (I'm not sure what's the right word) concurrency with eio, which is deterministic. If I want parallelism here, I have to explicitely create and use a shared job queue, while the Go runtime does this implicitly. Since the standard library Queue is not thread safe, I would have to use Mutex to avoid concurrent access. Is this correct? I've read the eio documentation but it's hard to wrap my head around all of that without examples. I've found the Domain_manager which looks like what I want. For example, I could have the main thread fill the queue and for each core available, I could launch a Domain_manager.run toto, with toto taking jobs from the queue that would be shared between all domains? |
The README shows an example of a pool of workers pulling jobs from an Eio.Stream:
https://github.com/ocaml-multicore/eio#example-a-worker-pool
We're still exploring what APIs to provide for this kind of thing, and in particular how to unify domainslib and eio.