[samlambert]

because people run large amounts of front ends and workers that create a significant amount of connections. it doesn't matter if they are all active.

[throwaway313373]

Why would you want every "frontend" keep an open connection all the time?

> it doesn't matter if they are all active

It does, if the connection is inactive (doesn't hold an open transaction) you should close it or return it to the pool.

[samlambert]

so you are suggesting you close a connection between queries?

[throwaway313373]

Between queries in the same transaction? No

Between transactions? Yes, absolutely

In fact, many libraries do it automatically.

For example, SQLAlchemy doc explicitly says [0]:

> After the commit, the Connection object associated with that transaction is closed, causing its underlying DBAPI connection to be released back to the connection pool associated with the Engine to which the Session is bound.

I expect other reasonably sane libs for working with transactional databases do the same.

So, if you are doing pooling correctly, you can only run out of available connections if you want to have a lot of long running transactions.

So, why would you want every of your 50k frontends keep an open transaction simultaneously?

[0] https://docs.sqlalchemy.org/en/20/orm/session_basics.html#co...

[nrdvana]

Because there's an overhead to make a connection, authenticate, set the default parameters on the connection, etc. I've never seen a framework that closed db connections between requests.

Of course, the better design is to write a nonblocking worker that can run async requests on a single connection, and not need a giant pool of blocking workers, but that is a major architecture plan that can't be added late in a project that started as blocking worker pools. MySQL has always fit well with those large blocking worker pools. Postgres less so.

[throwaway313373]

As I said, you can return the connection to the connection pool.

From the perspective of keeping the number of open connections low it doesn't really matter if you close it or return to the pool, because in either case the connection becomes available to other clients.

[nrdvana]

I might not be understanding what you're pointing out here. It sounds to me like sqlalchemy is talking about a pool of connections within one process, in which case releasing back to that pool does not close the connection by that process to the database. Parent comment is talking about one connection per process with 50k processes. My comment was that you don't need that many processes if each process can handle hundreds of web requests asynchronously.

If you are saying that a connection pool can be shared between processes without pgbouncer, that is news to me.