[camgunz]

Oh no you're right it's a pain in the ass and weird. But I think it's the way because there's no good reason to have more than one event loop.

Also... maybe it started out as generator wrappers? I think I read something that said that.

[Spivak]

> no good reason to have more than one event loop

Per thread—once you start working in multiple threads you have the choice to have one global event loop, which comes at the cost of all async code being effectively serialized as far as threads are concerned*, or one event loop per thread.

* Which can be fine if your program is mostly not async but you have that one stubborn library. Yay async virality.

[camgunz]

I can't say there's no good reason to have per thread event loops, but I think I can say if you do know of one you're suffering a terrible curse. I can only imagine the constraints that would force me to do this.

[Spivak]

Because you have a main application running a web server and a daemon worker thread performing potentially long running tasks that use async libraries and you don't want to block the responsiveness of your web server. It's really not that bad, at least in Python.

[camgunz]

Well, here we go I guess. Why can't you just use FastAPI? Or Tornado? Isn't there also an async Flask? Isn't Django also async now? What minor god have you angered to be chained to a non-async framework?

[Spivak]

Of all the responses, this was perhaps my least expected one when I was talking about being chained to an async framework. Async isn't a replacement for threads, async doesn't let you spread your work out over multiple cores and doesn't give you time slicing. In Python the asyncio module actually gives you a threadpool to run computationally intensive work in as kind of one-offs. But when you need something like background job processing and want to also reap the benefits of asyncio, like being able to pull multiple tasks off the queue, and progress on others while a job does io, then you need an event loop in the other thread. It was specifically avoiding locking up FastAPI that lead me to use multiple event loops in the first place.

You're free to spin the job worker off to another process but however you swing it it's still multiple event loops you deal with. But with threads you get to only load your Python app into memory once.

[camgunz]

Skipping to the end here: if you're inside a Python thread and you're making your own event loop, something has gone badly. Maybe you made a bad choice (using threads inside an async task instead of an executor), maybe you have some legacy nightmare dependency thing to deal with, but multiple event loops, let alone nested event loops, are suboptimal from a resource usage standpoint.