[timgl]

co-founder of PostHog here. We were a victim of this attack. We had a bunch of packages published a couple of hours ago. The main packages/versions affected were:

- posthog-node 4.18.1, 5.13.3 and 5.11.3

- posthog-js 1.297.3

- posthog-react-native 4.11.1

- posthog-docusaurus 2.0.6

We've rotated keys and passwords, unpublished all affected packages and have pushed new versions, so make sure you're on the latest version of our SDKs.

We're still figuring out how this key got compromised, and we'll follow up with a post-mortem. We'll update status.posthog.com with more updates as well.

[bilalq]

You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run.

[mbreese]

Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing.

[woodruffw]

The basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name.

(There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.)

[mbreese]

I don’t buy this. I mean, I’m sympathetic to the issue. It’s easy for things to get jumbled when companies are young. But we’re talking about libraries that are used by your customers. To many, this is the externally visible interface to the company.

What you describe sounds like a process problem to me. If an $EARLY_EMPLOYEE if the only one with the deploy keys for what is a product of the company, then that’s a problem. If a deployment of that key library can be made without anyone approving it, that’s also a problem. But those are both people problems… and you can’t solve a people problem with a technical solution.

[woodruffw]

> But those are both people problems… and you can’t solve a people problem with a technical solution.

I don’t think it is a people problem in this case: the only reason there’s a person involved at all is because we’ve decided to introduce one as an intermediating party. A more misuse-resistant scheme disintermediates the human, because the human was never actually a mandatory part of the scheme.

[mbreese]

What person do you want to remove from the process?

[YetAnotherNick]

You can't "require" manual intervention. Sure you can say that the keys stays on say 2 developers laptops, but personal devices have even more surface area for key leak than CI/CD pipeline. It wouldn't have prevented attacks like this issue in any case where the binary just searched for keys across the system.

One alternative is to do the signing on airlocked system stored in physically safe but accessible location, but I guess that's just way too much inconvenience.

[mbreese]

As someone else mentioned, the easiest way would be to have some kind of MFA in the loop. It’s not perfect, but better than what we have now.

[magicalhippo]

I get that it can be useful sometimes. But requiring physical MFA to make a package available to the general public seems like a no-brainer to me.

Users who really want to could opt in to the bleeding edge.

[bilalq]

This is orthogonal to the issue at hand. The problem is a malicious actor cutting a release outside of the normal release process. It doesn't matter if the normal process is automated or manual.

[serial_dev]

It could have eliminated an attack surface where they steal the credentials from the CI/CD...

...But then you if I understand NPM publishing well, you would still have the credentials on someone's computer laying around? I guess you could always revoke the tokens after publishing? It's all balancing convenience and security, with some options being bad at both?

[twistedpair]

This is built in NPM. You can get an email on every pkg publishing.

Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it.

[euph0ria]

Very nice way of putting it, kudos!

[silverlight]

Did the client side JS being infected produce any issues which would have affected end users? As in if a web owner were on an affected version and deployed during the window would the end user of their site have had any negative impact?

[timgl]

No, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit.

[silverlight]

Glad to hear the impact was so muted. Thank you for the response!

[spiderfarmer]

If we don't know how it got compromised, chances are this attack is still spreading?

[brabel]

If anything people should use an older version of the packages. Your newest versions had just been compromised, why should anyone believe this time and next time it will be different?!

[timgl]

The packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/196303581...

[progbits]

Why do you keep using token auth? This is unacceptable negligence these days.

NPM supports GitHub workflow OIDC and you can make that required, disabling all token access.

[timgl]

Yep, we are moving to workflow OIDC as the next step in recovery.

[junon]

OIDC is not a silver bullet either and has its own set of vectors to consider too. If it works for your org model then great, but it doesn't solve every common scenario.

[woodruffw]

Trusted Publishing addresses the vector here, which is arbitrary persistence and delayed use of credentials by attackers. You're right that it's not a silver bullet (anything claiming to be one is almost certainly a financially induced lie), but it eliminates/foreshortens the attack staging window significantly.

[junon]

It's not a silver bullet in that it can open up other vectors of much more probable attack.

If I have a FOSS project where there are other contributors with write access but not publish access (very common) OIDC now allows them to publish via workflows, opening you up to Jia Tan type attacks.

Like I said, if it works for your org, great. Otherwise it can cause bigger problems.

[Y_Y]

> so make sure you're on the latest version of our SDKs.

Probably even safer to not have been on the latest version in the first place.

Or safer again not to use software this vulnerable.

[BowBun]

As a user of Posthog, this statement is absurd: > Or safer again not to use software this vulnerable.

Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it.

[tclancy]

Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive?

[_alternator_]

Glad you updated on this front-page post. Your Twitter post is buried on p3 for me right now. Good luck on the recovery and hopefully this helps someone.