[z00b]

Rob Zuber, CTO of CircleCI here. We take security very seriously and are taking a deep look at the issues Kevin raised.

We'll save additional commenting until we have gathered more information. In the meantime, our security policy and steps for reporting issues are here: https://circleci.com/security/ and we'd like to ask the community to please use our outlined methods for reporting potential security issues so we can keep CircleCI as safe as possible for everyone.

[kevinburke]

OP here. Thanks for responding promptly. To be clear, letting third party JS run in a trusted environment like a dashboard is an industry wide problem. If we assume CircleCI is the only bad actor we're kind of missing the point of the exercise.

CircleCI is a notable example, due to the fact it hosts source code and secrets for so many different companies and loads so many third party scripts in its dashboard context. But they're not unique in this regard. I hope everyone reading this is reconsidering the scripts that have access to their dashboard and pushing for changes at their company.

If CircleCI was immediately vulnerable to injection via the scripts above I would have used the private disclosure route and I encourage others to as well. But I don't know that there is a "vulnerability" here so much as a discussion that we need to have about what and how much third party code we let run in a trusted context. I wrote a little more in the thread below, https://news.ycombinator.com/item?id=15442988.

[z00b]

Thanks Kevin. We really do appreciate the discussion and, as you would imagine, are rethinking some of our approaches in line with the issues you've raised.

[Posibyte]

The real thanks should go to you Rob. As it's stated, it's an industry wide problem and the real guts come from the companies that take the issue seriously; the ones who come out, hold it close to their own, admit they were wrong, and work to improve. This says a lot about you and your company, and as a customer it makes me proud that you're part of our mobile pipeline.

As a customer, Rob, what would be a good avenue to notify you of issues like this in the future? Not every company wants to be front and center of what is essentially the news hub of tech people like HN for issues, so is there some way we can get with you guys directly to straighten security issues out, as well as some assurances as how you'll handle it along the way?

Thanks for your attention to the issue :)

[z00b]

Hi.

You can always email our security team at [e-mail] Our gpp key is available on our site https://circleci.com/security/ if needed. You can also email me directly at rob@ if you have an immediate question.

[remcob]

In the interest of said discussion, I would appreciate a write-up of the rationale behind the approach chosen, when you are ready. It could help others that are in a similar situation. Perhaps inspire a better practice in the industry.

[ssemmaprise]

Hi there, we wrote up our thoughts + reply here: https://discuss.circleci.com/t/circleci-response-to-kevin-bu... Thanks all for your thoughts + comments.

[politician]

Thanks for putting this together.

You mention vetting third party scripts, but did not explain how your app is not vulnerable to those scripts being hacked or modified in the future. This vulnerability seemed to be the main point of Kevin's piece.

Could you update your post to address that issue?

[ukd1]

IMHO you should have followed the standard practice of telling them first, then posting the story afterward. Blindsiding them wasn't nice, even if (hopefully) this makes them and others think more about what they risk with external JS, rather than just "oh shit circle...". Just my 5c.

p.s I'm curious why you thought this route was better?

[rapind]

I hear this attitude a lot and it always rubs me the wrong way.

It's a corporation... Why should he care about it's feelings? What does he owe CircleCI? Are they paying him? What claim do they have to his time to jump through their hoops / process?

I'm positive in this case that they weren't intentionally messing with people's security, but shouldn't we and their customers be able to judge that ourselves instead of getting it swept under the carpet via private channels?

I do believe it's good practice to "not be a douche", especially at a personal level, but I wouldn't even come close to categorizing this as douche behaviour.

* Note: this comment is not actually directed at CircleCI, just the attitude that we all somehow owe it to companies to tell them about their goofs privately.

[ukd1]

This is, imho, douche behavior when it's the accepted norm to approach a company through their (listed and public!) security page. Why? So they can fix things before they get fucked. Yes of course you don't actually have to, but that makes you a douche imho.

Also, it's not swept under the carpet - it ends up usually getting $ for the reporter and a better story as they'd also know how the company fixed it. If they refuse to fix, then publish away.

https://en.wikipedia.org/wiki/Responsible_disclosure

[kevinburke]

That would have been appropriate if I had found e.g. an endpoint that did not accept CSRF tokens, or a vulnerability in Ring. It's not appropriate when the problem is they made a business decision to expose my source code to Quora.js.

[otterley]

How do you know it was a conscious decision, as opposed to ordinary human oversight? As the old saying goes, never ascribe to malice what can be adequately ascribed to incompetence. (Though I wouldn't go so far as to call CircleCI "incompetent" -- security issues are rampant in this industry, despite everyone's best efforts.)

[michaelbuckbee]

It's rarely about the company and much more about those affected by a potential issue. This issue's a little different as it's the potential for a vulnerability more than an actual vulnerability.

A typical case for responsible disclosure is something like a bug in Apache or Nginx that's serious and if a security issue is found that they're given time to address it. So when they make an announcement its:

1. Here is the issue

and

2. Here is the fix

Instead of just spreading and publicizing a vulnerability.

[rapind]

I'm mostly with you when there's potential damage to their customers, especially when those customers are individuals (like a database of social profiles being compromised), which was not the case here. However I think saying it's "rarely" about the company is a bit naive.

There's been some really high profile breaches where the extent of the impact to the customer has taken an unacceptably long time to be made public... which results in increased fallout damage.

A company's natural inclination is going to be to minimize losses (generalizing of course), and that often means dragging their feet, or dealing with it privately and never notifying their customers.

We should recognize these incentives and be a bit more aggressive about holding their feet to the fire, instead of criticizing the researcher who discovers the security flaw. Unless of course his / her behaviour is blatantly malicious.

It anecdotally feels a bit lopsided in favour of the corporations at the moment. Especially when someone like in this post, who clearly didn't endanger any customers, gets criticized...

[ukd1]

^ this

[UnpossibleJim]

I can't speak to what the OP was thinking, but given their above statement of this being an industry wide issue, it seems to me posting this privately to a single company would only address the issue within that specific company and very little to nothing would change in other companies due to the nature of competition. While this isn't nice, I guess, it does seem to be the nature of development and business. Just my 5 cents, though, I've been wrong before.

[ukd1]

I disagree; report it to circle, if they fix then you can write the story on how it's a massive problem, how they fixed and people can learn. We learnt basically shit from this: nothing about how it's fixed or is avoided.

[kevinburke]

If I did that, no one today would be opening the Network tab on apps/credit card forms they care about and wondering whether third parties could steal the data.

[flukus]

I'm sure if CircleCI had a prompt to opt-in to enabling the tracking then they might have returned the courtesy. Why is it OK for a company/project to violate the privacy of so many with no warning but bad for users to report the privacy and security violations?

[kevinburke]

correct, and this is a company that my employers/clients have collectively paid tens of thousands of dollars to.

[oliwarner]

I'm not sure private disclosure is appropriate here.

This isn't the same as a browser vulnerability where some kid could hack a load of people before they patched. One of these supposedly trusted companies could attack your customers... But they could already. They know their scripts get loaded into all sorts of inappropriate environments.

Getting people notified, getting credentials changed. That's the paramount concern. You can rub PR lotion into an in-depth audit later on.

[ejcx]

This reaction is nonsense. These are all vendors that circle pays.

It's an industry wide bad practice and a risk, but forcing password changes or notifying people is quite frankly ridiculous.

[oliwarner]

I didn't suggest forcing anything... But just because other people do it doesn't make it better.

Many of their clients may treat this as they would an actual breach. It's somebody they haven't vetted having potentially complete access to their development chain and production secrets. They won't know until they look. They won't look until they're told.

And what's CircleCI paying a for this breach got to do with the price of fish?! Say you hired me and gave me full access to everything in your business. Then one day I turn around and tell you my extended family, my friends, my dog walker and my cleaner have all also had access to that data. No big problem eh?

[jlmorton]

This practice is allowed by industry security standards, like PCI-DSS. If it's determined that the third-party acts as a PCI Service Provider, then the compliant party has a duty to determine that the third-party is also compliant.

The client vetted CircleCI, and CircleCI presumably vetted the third parties. It is not fair to say these vendors have not been vetted.

It may not be a best practice, but it's little different than CircleCI (or any other company) contracting with a private data center, which has direct physical access to their equipment. They have presumably vetted the data center provider, or cloud computing vendor.

[ejcx]

There is absolutely no breach here. Absolutely not. Suggesting so is ridiculous. The word breach is a very special one and this is not it.

[oliwarner]

I think you're the one being more reckless with language here. But please, what does "breach" mean to you?

I count it as "inappropriate and unauthorised access to data". Where "access" is potential, not necessarily actual unless you can absolutely prove there was no access.

These third parties have had access to sensitive data they shouldn't. That's a breach in my book.

Neither you or Circle CI even can say this hasn't lead to current or past third parties —or their rogue developers, or people who have hacked them— gaining source access or customer data from Circle CI users. Why? You simply don't know what was running at any given point.

Auditing and sub-resource integrity would help in the future, but it's too late. Unknown people have had access. Only the Circle CI users will know what ramifications that could have on them.

If your argument is anything more than a redefinition of "breach", please explain why you're being so nonchalant (and why you think I'm being "ridiculous") about this.

[bsimpson]

Is there any way to limit what access CircleCI has to my GitHub account/orgs? GitHub says you have "Full control of private repositories," but it seems like all you need is read access + a push webhook.

[kevinburke]

IIRC Circle needs to tell Github whether the build succeeded or failed.

It would be nice if Github offered more granular permissions.

[lancefisher]

When I need more granular access, this is how I set it up:

1) Create a new GitHub user: e.g. ([e-mail]) 2) Grant read-only access to specific repositories to the new user. 3) Configure CI to use that user.

[daxelrod]

GitHub recently added a repo:status OAuth scope for this use case. Unfortunately, I still don't see a scope for read-only access to repo code.

https://developer.github.com/apps/building-integrations/sett...

[caust1c]

Who's Ryan?

[z00b]

Welp. Wanted to write quickly and made a stupid mistake. Sorry about that Kevin!

[atomi]

What serious business hosts their proprietary, mission critical code on someone else's computers? Isn't that like Coke or KFC putting their original recipe on Google Drive?

Edit: Okay, I get it. People like convenience over security. I'll stick to self-hosting a Drone-CI instance and only deploying binaries.

[timdorr]

Literally everyone on AWS, for instance. Except Amazon, of course.

[Phlogistique]

Just in case this is a serious question:

Most of them.