[detente18]

LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far:

1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09

2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt

3. The package is in quarantine on pypi - this blocks all downloads.

We are investigating the issue, and seeing how we can harden things. I'm sorry for this.

- Krrish

[detente18]

Update:

- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted

We are still scanning our project to see if there's any more gaps.

If you're a security expert and want to help, email me - [e-mail]

[detente18]

Update 2 (03/25/2026):

- We will be holding a townhall on Friday to review the incident and share next steps (https://lnkd.in/gsbTdCe7)

- We can confirm a bad version of Trivy security scanner ran in our CI/CD pipeline, which would have led to the supply chain attack

- We have paused new releases until we've completed securing our codebase and release pipeline to ensure safe releases for users

- We've added additional github/gitlab ci scripts for checking if you're impacted: https://lnkd.in/gGicMkby

We hope to share a full RCA in the coming days. Until then, if there's anything we can do to help your team - please let me know. You can email me ([e-mail]), or join the discussion on github (https://lnkd.in/g9TuuQ2H).

[cosmicweather]

> All maintainer accounts have been changed

What about the compromised accounts(as in your main account)? Are they completely unrecoverable?

[detente18]

I deleted it, to be safe.

[MadsRC]

Dropped you a mail from [e-mail]

[harekrishnarai]

> it seems your personal account is also compromised. I just checked for the github search here https://github.com/search?q=%22teampcp+owns%22

[Imustaskforhelp]

I just want to share an update

the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion

Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518

[vintagedave]

This must be super stressful for you, but I do want to note your "I'm sorry for this." It's really human.

It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."

Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.

[nextos]

I think we really need to use sandboxes. Guix provides sandboxed environments by just flipping a switch. NixOS is in an ideal position to do the same, but for some reason they are regarded as "inconvenient".

Personally, I am a heavy user of Firejail and bwrap. We need defense in depth. If someone in the supply chain gets compromised, damage should be limited. It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.

[staticassertion]

What would be really helpful is if software sandboxed itself. It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive.

But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges.

> It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.

Neither of these is easy tbh. Entering a Linux namespace requires root, so if you want your users to be safe then you have to first ask them to run your service as root. eBPF is a very hard boundary to maintain, requiring you to know every system call that your program can make - updates to libc, upgrades to any library, can break this.

Sandboxing tooling is really bad.

[eichin]

If the whole point of sandboxing is to not trust the software, it doesn't make sense for the software to do the sandboxing. (At most it should have a standard way to suggest what access it needs, and then your outside tooling should work with what's reasonable and alert on what isn't.) The android-like approach of sandboxing literally everything works because you are forced to solve these problems generically and at scale - things like "run this as a distinct uid" are a lot less hassle if you're amortizing it across everything.

(And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)

[staticassertion]

> If the whole point of sandboxing is to not trust the software, it doesn't make sense for the software to do the sandboxing.

That's true, sort of. I mean, that isn't the whole point of sandboxing because the threat model for sandboxing is pretty broad. You could have a process sandbox just one library, or sandbox itself in case of a vulnerability, or it could have a separate policy / manifest the way browser extensions do (that prompts users if it broadens), etc. There's still benefit to isolating whole processes though in case the process is malicious.

> (And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)

The only linux namespace that doesn't require root is user namespace, which basically requires root in practice. https://www.man7.org/linux/man-pages/man2/clone.2.html

Podman uses unprivileged user namespaces, which are disabled on the most popular distros because it's a big security hole.

[ashishb]

> It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive.

Not really.

Let's say I am running `~/src/project1 $ litellm`

Why does this need access to anything outside of `~/src/project1`?

Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.

[staticassertion]

How are you setting that sandbox up? I've laid out numerous constraints - x-platform support is non-existent for sandboxing, sandboxing requires privileges to perform, whole-program sandboxing is fundamentally weaker, maintenance of sandboxing is best done by developers, etc.

> Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.

Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.

[ashishb]

> Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.

Not really. I try to capture the most common ones for caching [1], but if I miss it, then it is just inefficient, as it is equivalent to a cache miss.

I'll emphasize again, "no linter/scanner/formatter (e.g., trivy) should need full disk access".

1 - https://github.com/ashishb/amazing-sandbox/blob/fddf04a90408...

[ashishb]

I am happily running all third-party tools inside the Amazing Sandbox[1]. I made it public last year.

1 - https://github.com/ashishb/amazing-sandbox

[kernc]

`sandbox-venv` is a small shell script that sandboxes Python virtual environments in separate Linux namespaces using Bubblewrap (and soon using only command `unshare`, bringing the whole script down to effectively 0 deps).

https://github.com/sandbox-utils/sandbox-venv

[cyanydeez]

Lawyers are slowly eating humanity.

[singleshot_]

Allegedly*

[bmurphy1976]

For now. They're about to get hit by the AI wave as bad as us software devs. Who knows what's on the other side of this.

[blueone]

Sorry that I have to be the one to tell you this, but lawyers are fine. Sure, AI will have an impact, but nothing like the once hyped idea that it would replace lawyers. It has actually been amusing to watch the hype cycle play out around AI when it comes to lawyers.

[throwawaytea]

My parents had a weird green card and paperwork issue that was becoming a big problem. Everyone in their social circle recommended an immigration type lawyer. Everyone.

My dad was confident he could figure it out based on his perplexity Pro account. He attacked the problem from several angles and used it for help with what to do, how to do it, what to ask for when visiting offices, how to press them to move forward, and tons of other things.

Got the problem resolved.

So it definitely can reduce hiring lawyers even.

[bmurphy1976]

Hell, the tech savvy senior lawyers are already using LLMs to do the work that army's of juniors and other assistants did for them. Just like what is happening with software engineers. Anybody who thinks this isn't going to have some kind of impact is mental.

[recpen]

Lawyership in the sense of the profession may survive and adapt. Individual lawyers, not so much. I strongly doubt the new equilibrium (if we ever reach one) will need so many lawyers.

Same logic for software developers.

[driftnode]

the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential

[sobellian]

If the payload is a credential stealer then they can use that to escalate into basically anything right?

[driftnode]

Yes and the scary part is you might never know the full extent. A credential stealer grabs whatever is in memory or env during the build, ships it out, and the attacker uses those creds weeks later from a completely different IP. The compromised package gets caught and reverted, everyone thinks the incident is over, meanwhile the stolen tokens are still valid. I wonder how many teams who installed 1.82.7 actually rotated all their CI secrets after this, not just uninstalled the bad version.

[someguydave]

It’s not wild. Anyone with brains would think it is irresponsible to keep your secret token for publishing public release binaries accessible in your automation.

[kreelman]

I wonder if there are a few things here....

It would be great if Linux was able to do simple chroot jails and run tests inside of them before releasing software. In this case, it looks like the whole build process would need to be done in the jail. Tools like lxroot might do enough of what chroot on BSD does.

It seems like software tests need to have a class of test that checks whether any of the components of an application have been compromised in some way. This in itself may be somewhat complex...

We are in a world where we can't assume secure operation of components anymore. This is kinda sad, but here we are....

[driftnode]

The sad part is you're right that we can't assume secure operation of components anymore, but the tooling hasn't caught up to that reality. Chroot jails help with runtime isolation but the attack here happened at build time, the malicious code was already in the package before any test could run. And the supply chain is deep. Trivy gets compromised, which gives CI access, which gives PyPI access. Even if you jail your own builds you're trusting that every tool in your pipeline wasn't the entry point. 97 million monthly downloads means a lot of people's "secure" pipelines just ran attacker code with full access.

[ozozozd]

Kudos for this update.

Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.

[redrove]

>1. Looks like this originated from the trivvy used in our ci/cd

Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?

[wheelerwj]

The latest trivy attack was announced just yesterday. If you go out to dinner or take a night off its totally plausible to have not seen it.

[anishgupta]

afaik the trivy attack was first in the news on March 19th for the github actions and for docker images it was on March 23rd

[mrexcess]

You're making great software and I'm sorry this happened to you. Don't get discouraged, keep bringing the open source disruption!

[rao-v]

I put together a little script to search for and list installed litellm versions on my systems here: https://github.com/kinchahoy/uvpowered-tools/blob/main/inven...

It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters.

Obviously read it before running it etc.

[mikert89]

Similar to delve, this guy has almost no work experience. You have to wonder if YC and the cult of extremely young founders is causing instability issues in society at large?

[gopher_space]

It's interesting to see how the landscape changes when the folks upstream won't let you offload responsibility. Litellm's client list includes people who know better.

[zdragnar]

Welcome to the new era, where programming is neither a skill nor a trade, but a task to be automated away by anyone with a paid subscription.

[mikert89]

alot of software isnt that important so its fine, but some actually is important. especially with a branding name slapped on it that people will trust

[whattheheckheck]

The industry needs to step up and plant a flag for professionalization certifications for proper software engineering. Real hard exams etc

[jacamera]

I can't even imagine what these exams would look like. The entire profession seems to boil down to making the appropriate tradeoffs for your specific application in your specific domain using your specific tech stack. There's almost nothing that you always should or shouldn't do.

[xenophonf]

All engineering professions are like that. NCEES has been licensing Professional Engineers for over a hundred years. The only thing stopping CS/SE is an unwillingness to submit to anything resembling oversight.

[leftyspook]

All software runs on somebody's hardware. Ultimately even an utterly benign program like `cowsay` could be backdoored to upload your ssh keys somewhere.

[utrack]

https://xkcd.com/2347/ , but with `fortune -a` and `cowsay` instead of imagemagick

[moomoo11]

It’s a flex now. But there are still many people doing it for the love of the game.

[Imustaskforhelp]

> - Krrish

Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts)

Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not)

I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.

[detente18]

It was the PYPI_PUBLISH token which was in our github project as an env var, that got sent to trivvy.

We have deleted all our pypi publishing tokens.

Our accounts had 2fa, so it's a bad token here.

We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).

[redrove]

How did PYPI_PUBLISH lead to a full GH account takeover?

[ezekg]

I'd imagine the attacker published a new compromised version of their package, which the author eventually downloaded, which pwned everything else.

[chunky1994]

Their Personal Access Token must’ve been pwned too, not sure through what mechanism though

[Imustaskforhelp]

They have written about it on github to my question:

Trivvy hacked (https://www.aquasec.com/blog/trivy-supply-chain-attack-what-...) -> all circleci credentials leaked -> included pypi publish token + github pat -> | WE DISCOVER ISSUE | -> pypi token deleted, github pat deleted + account removed from org access, trivvy pinned to last known safe version (v0.69.3)

What we're doing now:

    Block all releases, until we have completed our scans
    Working with Google's mandiant.security team to understand scope of impact
    Reviewing / rotating any leaked credentials

https://github.com/BerriAI/litellm/issues/24518#issuecomment...

[celticninja]

69.3 isnt safe. The safe thing to do is remove all trivy access. or failing that version. 0.35 is the last and AFAIK only safe version.

https://socket.dev/blog/trivy-under-attack-again-github-acti...

[franktankbank]

Does that explain how circleci was publishing commits and closing issues?

[franktankbank]

Don't hold your breath for an answer.

[mike_hearn]

Perhaps it's too obvious but ... just running the publish process locally, instead of from CI, would help. Especially if you publish from a dedicated user on a Mac where the system keychain is pretty secure.

[staticassertion]

I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else.

In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.

[mike_hearn]

User isolation works, the keychain isolation works. On macOS tokens stored in the keychain can be made readable only by specific apps, not anything else. It does require a bit of infrastructure - ideally a Mac app that does the release - but nothing you can't vibe code quickly.

[staticassertion]

That's true, but it seems far more complex than just moving trivy to a separate workerflow with no permissions and likely physical isolation between it and a credential. I'm pretty wary of the idea that malware couldn't just privesc - it's pretty trivial to obtain root on a user's laptop. Running as a separate, unprivileged user helps a ton, but again, I'm skeptical of this vs just using a github workflow.

[tedivm]

This problem is solved by not having a token. Github and PyPI both support OIDC based workflows. Grant only the publish job access to OIDC endpoint, then the Trivy job has nothing it can steal.

[NewJazz]

Are you spelling it with two vs on purpose?

[redrove]

>I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.

Token in CI could've been way too broad.

[franktankbank]

He would have to state he didn't in fact make all those commits and close the issue.

[pojzon]

This is just one of many projects that was a victim of Trivy hack. There are millions of those projects and this issue will be exploited in next months if not years.

[N_Lens]

Good work! Sorry to hear you're in this situation, good luck and godspeed!

[kingreflex]

we're using litellm via helm charts with tags main-v1.81.12-stable.2 and main-v1.80.8-stable.1 - assuming they're safe?

also how are we sure that docker images aren't affected?

[saltyoldman]

Docker deployments are more safe even if affected because there is a lower chance (but not zero) that you didn't mount all your credentials into the image. It would have access to LLM keys of course, but that's not really what the hacker is after. He's after private SSH keys.

That being said this hack was a direct upload to PyPI in the last few days, so very unlikely those images are affected.

[anishgupta]

yep joining here late but docker images are not affected as we saw on twitter

[outside2344]

Is it just in 1.82.8 or are previous versions impacted?

[Imustaskforhelp]

1.82.7 is also impacted if I remember correctly.

[GrayShade]

1.82.7 doesn't have litellm_init.pth in the archive. You can download them from pypi to check.

EDIT: no, it's compromised, see proxy/proxy_server.py.

[cpburns2009]

1.82.7 has the payload in `litellm/proxy/proxy_server.py` which executes on import.

[bognition]

The decision to block all downloads is pretty disruptive, especially for people on pinned known good versions. Its breaking a bunch of my systems that are all launched with `uv run`

[Shank]

> Its breaking a bunch of my systems that are all launched with `uv run`

From a security standpoint, you would rather pull in a library that is compromised and run a credential stealer? It seems like this is the exact intended and best behavior.

[tedivm]

You should be using build artifacts, not relying on `uv run` to install packages on the fly. Besides the massive security risk, it also means that you're dependent on a bunch of external infrastructure every time you launch. PyPI going down should not bring down your systems.

[zbentley]

This is the right answer. Unfortunately, this is very rarely practiced.

More strangely (to me), this is often addressed by adding loads of fallible/partial caching (in e.g. CICD or deployment infrastructure) for package managers rather than building and publishing temporary/per-user/per-feature ephemeral packages for dev/testing to an internal registry. Since the latter's usually less complex and more reliable, it's odd that it's so rarely practiced.

[lanstin]

There are so many advantages to deployable artifacts, including audibility and fast roll-back. Also you can block so many risky endpoints from your compute outbound networks, which means even if you are compromised, it doesn't do the attacker any good if their C&C is not allow listed.

[MeetingsBrowser]

Are you sure you are pinned to a “known good” version?

No one initially knows how much is compromised

[cpburns2009]

That's PyPI's behavior when they quarantine a package.

[wasmitnetzen]

Take this as an argument to rethink your engineering decision to base your workflows entirely on the availability of an external dependency.

[zbentley]

That's a good thing (disruptive "firebreak" to shut down any potential sources of breach while info's still being gathered). The solve for this is artifacts/container images/whatnot, as other commenters pointed out.

That said, I'm sorry this is being downvoted: it's unhappily observing facts, not arguing for a different security response. I know that's toeing the rules line, but I think it's important to observe.

[saidnooneever]

known good versions and which are those exactly??????

[kleton]

There are hundreds of PRs fixing valid issues to your github repo seemingly in limbo for weeks. What is the maintainer state over there?

[michh]

increasing the (social) pressure on maintainers to get PRs merged seems like the last thing you should be doing in light of preventing malicious code ending up in dependencies like this

i'd much rather see a million open PRs than a single malicious PR sneak through due to lack of thorough review.

[zparky]

Not really the time for that. There's also PRs being merged every hour of the day.