[atombender]

I don't see any provisions for scaling up workflows horizontally — is that something this supports, or this more for interactive agents?

As an example, I have a use case where I'm processing millions of documents in a fan-out/fan-in pipeline that uses multiple LLMs to analyze and condense information. This mostly isn't CPU-bound, but it does consume a lot of RAM.

I'm currently using Ray to split the workload across a cluster. Ray has autoscaling, so it's very good for this kind of thing.

[lchoquel]

Hi atombender, you have a point. The current runtime supports asynchronous operations with the limits of python asyncio and your CPU/RAM. But we designed our orchestration layer so that it could integrate easily with durable workflow platforms such as temporal.io which also bring the ability to spread the load over many workers, which will give you horizontal scaling.

Our integration with temporal.io has been working as a prototype in our lab for many months but we had to focus on the core features to nail down our language and make it as easy to use as possible. Now in the coming weeks we'll be able to focus more on the temporal-powered version, adding the missing features to make it practical, which will also make it usable with other workflow platforms.

[atombender]

Temporal is solid, glad to hear it's being worked on.