[d1l]

That's even crazier - so you're using asyncio because you have a ton of slow network-bound stuff - but for your database access you are running every sqlite connection in it's own thread and just managing those threads via the asyncio event loop?

[reactordev]

Thread pooling for databases, whether network based, or disk based, is common. A lot of times it will be baked into your client, so the fact that you think it’s crazy means you’ve only dealt with clients that did this for you.

For really large data sets, you can query and wait a few minutes before getting a result. Do you really want to await that?

[paulddraper]

This is a common paradigm for blocking APIs (e.g. the sqlite driver)

[quietbritishjim]

What is crazy about that?

[lttlrck]

Of course I don't know what the parent is thinking, but my thought is: why can't it be entirely event loop driven? What are the threads adding here?

(I don't know anything about that project and this isn't meant as a criticism of its design or a challenge - cos I'd probably lose :-) )

[eurleif]

SQLite doesn't have a separate server process; it does all of the work for queries in your process. So it's intrinsically CPU-heavy, and it needs threads to avoid blocking the event loop.

One way to look at is that with a client-server database and an async client library, you have a thread pool in the database server process to do the heavy lifting, and async clients talk to it via TCP. With SQLite, you have that "server" thread pool in the same process instead, and async "clients" talk to it via in-process communication.

[mayli]

Cause the sqlite-lib that python ships isn't async, and sqlite itself usually doesn't give an async API.

[maxbond]

Python's asyncio is single threaded. If you didn't send them into a different thread, the entire event loop would block, and it would degenerate to a fully synchronous single threaded program with additional overhead.