[coenhyde]

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

[grepLeigh]

Surgeons have a practiced ritual ("scrubbing") to prep for surgery. Do you practice a credential-scanning ritual before saving (committing) your code or pushing your code to a remote repo?

I have git hooks to lint code syntax, but nothing for scanning for leaked credentials. Looking @ TruffleHog now, mentioned by another poster.

[coenhyde]

That's certainly a good idea. But the secrets shouldn't be in the codebase to begin with, certainly not production secrets. Production secrets should stay in production and no one has access. Whatever intends to use the production secrets should have first been developed in a dev environment and released to prod.

[zricethezav]

Gitleaks also offers a nice pre-commit hook: https://github.com/zricethezav/gitleaks#pre-commit

[justinpombrio]

A nice approach, if you have sufficient control over the form of your secrets, is to prefix each secret with "MY_COMPANY_SECRET_DO_NOT_COMMIT:". Then you can add a commit hook that refuses to commit if any committed file contains that substring, etc. etc.

[lumberjack24]

Great idea, but hard to enforce. Just use a scanning CLI like TruffleHog, Gitleaks, or ggshield from GitGuardian to catch all sorts of hardcoded secrets.

[Too]

Code-reviews? Should be a ritual you do on your own code before commiting+pushing and should be a ritual that others will do in the PR before merge (arguable here a secret is already compromised).

[ajross]

"Should" not be violated is the point, though. I agree, it shouldn't. But it is, all the time.

I mean, I'll bet Toyota knew this organizationally. They had security people sign off on the design who all knew how secure key management is supposed to work. They probably reviewed this github release. And it happened anyway.

Maybe they weren't supposed to be production keys. Maybe it was a development key from someone's early cut of the tooling that got mistakenly reused for production. Maybe a script that updated all the keys mixed up which was which.

The point is that the existence of a Clear And Unambiguous Right Thing to Do is, in practice, not sufficient to ensure that that thing is done. The space of ways to mess up even obvious rules is too big.

And that's surprising, which is why (1) it keeps happening and (2) people like you don't take the possibility seriously in your own work.

[coenhyde]

You're jumping to conclusions in your final statement there. The existence of inexcusable bad practices does not mean we should not try to mitigate against them, and I didn't say we shouldn't.

[deathanatos]

And yet I see it get violated all the time. People should do a lot of things, but a lot of my coworkers are lazy and do not do quality work. Given that it happens, and that I can't prevent it, one must then ask how to guard against it.

At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" types from … just working around the standard tooling and manually generating one by hand because … IDK, they think they know better? I really don't get it.

[coenhyde]

Doctors used to not wash their hands too. I get it though, and i've seen the same thing. Really it comes down to education and not granting access to secrets to people who aren't capable of handling them.

[robswc]

"fun" fact - there could potentially be thousands of deaths attributed to Drs simply not washing their hands.

IIRC they even basically got some hospital admin fired for creating a hand washing mandate, despite it being proven to save lives.

https://www.npr.org/sections/health-shots/2015/01/12/3756639...

(talking centuries ago, but maybe even today)

Looks like its still a "recent" issue, lol https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.htm...

[coenhyde]

That further improves the analogy. Even though we all agree washing hands is important and saves lives, it still doesn't happen on occasion.

[lmm]

It does, but it also suggests that there is no easy fix.