Hacker News new | ask | show | jobs
by bamboozled 1262 days ago
> We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well.

Is anyone else a little annoyed by the messaging here, I read it as, "We think something bad happened to your ultra secret data, but we don't know, so we're asking teams to spend potentially hours or days fixing things while we aren't really able to tell you if your stuff was actually compromised"?

What I find more troubling is, if they don't quite know what happened, or aren't telling us, and we do the work to change everything, how do they know it won't just happen again in the next day or so and people are still accessing our systems, where is the details?

> At this point, we are confident that there are no unauthorized actors active in our systems.

Confident isn't really a good enough word to use here in my opinion. We've just blocked Circle CI from all our systems for now until we hear more, likely start to move to another build system.

I know accidents happen but this is likely the beginning of the end for our teams relationship with Circle CI. Trust has been broken.

2 comments

> so we're asking teams to spend potentially hours or days fixing things

At the risk of sounding pedantic, but this is why you have everything as IaC. These kind of changes should not cost days. It should take merely minutes or an hour tops to change all your keys. It should be trivial, for cases just like this.

You can't use IaC to change third-party API keys. And woe unto any service that doesn't allow multiple keys because then you're looking at outages.
I get that you can manage the values in Circle, but you can't actually generate the values. IE, if you have a API token to write to Salesforce, you have to go into the Salesforce admin and generate a new token. Pasting the value in the Circle UI or a terraform descriptor are not the hard part. For lots of services, you can only have one key at a time meaning that generating a new one invalidates the old one meaning you'd have to have an outage while you're pasting and deploying.
I fully agree, our team just had to change one set of keys, other teams didn't follow best practices and are in a bad situation.

It's not Circle's fault people didn't do things propertly, but I think they just owe us a better explanation.

I can see you are bamboozled. But you should have seen the writing on the wall for CircleCI for quite some time now.
Agreed, how was I supposed to know that Circle CI would lose all the secret keys? I mean I always knew it was a possibility and our team planned for it...but what are you actually talking about?
I'm curious what indicators you have that this would have been the case. This is not a comment made in snark; I'm genuinely curious, as a CircleCI user who did not see the writing on the wall.
Can elaborate where would I have seen the writing for this? What indicators did you see?
Layoffs and outages at Circle. software supply chain attacks becoming more and more popular.
Outages at CircleCI have been common since the service has been launched.

Amazon is about to lay off 18k people. Circle isn't in a unique position to my understanding. Did CircleCI lay off some group or set of really important and key personnel?

Software supply chain attacks affect everyone. Is there some way that Circle is more vulnerable to this type of attack?

I would imagine GitHub Actions and always maturing CI tooling baked into cloud providers are rapidly eroding the market share of dedicated services like CircleCI.