[athoscouto]

We've been impacted by this. I migrated our services to Python 3.14 so we could attach profilers during runtime.

A couple of services looked like they had a memory leak. Memory was continuously increasing over time. Thanks to Python 3.14, we were able to use memray to understand what was going on. Those services were recreating HTTP clients (aiohttp) for every inbound request, and memory allocated by the downstream SSL lib was growing faster than it was being released.

We ended up rolling back to 3.13, which fixed the issue. I'll try again with 3.14.5.

[nas]

If you are using "httpx", it's likely caused by a reference cycle. I made a PR to fix it but the maintainers haven't applied it. :-( https://github.com/encode/httpx/pull/3733

The reference cycle httpx creates is kind of a worst-case scenario for the incremental GC issue. Both the generational (3.13 and older) and incremental GC are triggered by the net new "container" objects (objects that have references to others, like lists and not like ints and floats). The short summary is that you need to create more container objects before the incremental GC triggers. In the case of the httpx reference cycle, you have a relatively small number of container objects hanging on to a lot of memory, due the SSL context data (which is a big memory hog).

Reverting back to the generational GC was the wise thing to do, even though it's a bit scary to do in a bugfix release. The incremental GC works for most people but in the minority of cases it doesn't, it uses quite a lot more memory. I'm pretty sure with some additional tuning, the incremental GC would be fine too but it just didn't get that tuning. The generational GC has literal decades of real-world use (Guido merged my patch on Jun 2000, Tim Peters did a bunch of tuning after that to optimize it).

[JimDabell]

> I made a PR to fix it but the maintainers haven't applied it. :-( https://github.com/encode/httpx/pull/3733

Unfortunately, you may be the wrong gender to contribute to Encode repositories like httpx:

> I've closed off access to issues and discussions.

> I don't want to continue allowing an online environment with such an absurdly skewed gender representation. I find it intensely unwelcoming, and it's not reflective of the type of working environments I value.

— https://github.com/encode/httpx/discussions/3784

Discussed on Hacker News here: https://news.ycombinator.com/item?id=47193563

A fork discussed here: https://news.ycombinator.com/item?id=47514603

[athoscouto]

It was httpx indeed. i had aiohttp in mind because we ended up replacing that particular client with it

[rtpg]

We've been chasing down similar aiohttp client creation issues (liked to ...aiobotocore usage) for months now.

It's annoying that somehow talking to S3 etc requires so much churn. We have been trying to cache session objects and the like but clearly are still missing something.

Chasing this down has also made me realize how little Python libs use `weakref`, and just will build up so many circular references. The other day I figured out Django request's session infrastructure creates a circular reference meaning that requests have to get GC'd to get cleaned up in CPython.

I have a suspicion that the 3.14 problems are heavily linked to "real" workloads being almost entirely filled with cyclical objects.

[shay_ker]

It's really fascinating to read this, since I've encountered similar memory issues in other languages (ruby, go, etc.). Debugging these issues is a pain.

Is there a way to make all this much easier to debug and to prevent memory issues in the first place? Is the abstraction level not quite right?

[rtpg]

So with CPython's reference counting, if you're good at not building strong cycles, you really can avoid garbage pressure. It's not even that complicated, it's mostly a question of making a weak reference _somewhere_ along the chain. Often the ergonomics are not great, but Python @property's are nice here.

So for example

class Request

class Session

request.session exists, and the session is "part" of the request. but session.request often exists as a facility. That's a reference cycle which prevents the request (and anything it's pointed at!) from being deallocated at the end of a request.

But in this case, you could easily do something like:

session._request = weakref.ref(request) # on session creation

and then have session.request call session._request() (and maybe assert session._request() is not None if you want to be certain). If you're confident that the session is a "child" of the request, and that you would _never_ have a hold of the session after the request is done, this is a cheap trick that makes session.request cost a little bit more but not much.

I think most Python libraries just don't do memory perf analyses here, and also "believe" in the garbage collector. When GC runs, both request and session will get deallocated, after all! But the long term effects of everyone relying on the GC are that GC is expensive when it doesn't need to be, and when looking through memory you just have more stuff to dig through

[LaFolle]

On profilers - profiling will come in 3.15, are you referring to remote exec? It is a great feature I am very exited about, at the same time afraid that the company won’t allow ptrace capability in prod.

[athoscouto]

yes. remote exec allows me to attach profilers (e.g. memray) directly into a running process. i'm also excited about the upcoming statistical (cpu) profiler from 3.15