Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated resources and a financial stake in preventing backdoors?
Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees.
Now imagine why a nation state would want to get staff working in such a company.
Now if companies like
Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.
There's a ton of great truth here. It's hard to bite the bullet and believe that insiders already exist (everywhere), but I can share that from my experience working in big tech:
- There 100% will be bad actors. Many of them.
- But not always nationstate. Instead, they do it for (dumb) personal reasons, too. Also, don't forget lulzsec as a great example of just doing it for fun. So we cannot presume to know anything about the 'why'. The bad guys I caught did it for the most asinine reasons...
But the good news is that we have options:
- Strategic: Develop processes and systems that account for the perpetual existence of unknown bad actors and allow for successful business operation even when humans are compromised.
- Reactive: Structural logging that makes sense in the context of the action. Alerts and detection systems too.
- Reduction: Reduce access to only what is needed, when it is needed.
- Proactive (not always necessary): Multi party approvals (a la code review and production flag changes or ACL changes, too)
- Social: Build a culture of security practices and awareness of bad actors. Don't make people feel guilty or accusatory, just empower them to make good design and process decisions. It's a team sport.
Bonus: By guarding against evil actors, you've also got some freebie good coverage for when an innocent employee gets compromised too!
---
Companies like Google and Amazon do the techniques above. And they don't generally rely on antiquated technology that cannot and will not change to meet the modern standards.
I know because I was the person that built and Google's first time-based access system and rational-based access systems. And multi party approval systems for access. (Fun fact: The organizational challenge is harder than the technical).
And, those strategies work. And they increase SRE resilience too!
---
But even with the best UX, the best security tooling, the best everything, etc there's no guarantees that it matters if we just reject anything except the old system we're used to.
It's like a motorcycle helmet: Only works if you use it.
Your argument is a model that does no vetting of contributors whatsoever, which resulted in the catastrophe that is the topic of discussion, is better than a hypothetical company which is full of compromised developers that have free reign to commit to the source tree with no oversight? That sounds extremely contrived.
If you are positing that government infiltration of companies is hypothetical and not a real threat, here is an example of compromised corporate staff:
This wasn’t a contributor to OpenSSH, it was a deep level supply chain attack - something that closed source commercial companies are not immune to.
Given how much closed source companies love BSD/apache/etc licenses where they can simply use these low level libraries and charge for stuff on the top I’m not sure how they would be immune from such an attack.
The risk from this was highlighted in xkcd back in 2020
Moving the goalposts and splitting hairs. The fact remains the open source model allowed an imaginary person, operating on behalf of a threat actor, to obtain privileged commit access to a widely used open source project without any vetting whatsoever. Let me repeat that. They were given control of the repo without even verifying this person exists. To do this at a commercial company you actually have to show up and interview which is an order of magnitude more difficult than creating an anonymous Gmail account and be given the keys to the kingdom.
You are the one who moved the goalpost here. Vanilla OpenSSH doesn't link against xz, period. Not even the portable versions as LibreSSL does for OpenSSL.
If distros randomly patch OpenSSH because of SystemD, it's their problem.
> tldr; your statement overlooks the reality of businesses with high ethical and financial obligations, like Google, Amazon, and Azure.
- These companies underpin much of the internet's infrastructure.
- Their security practices are far more advanced than typical businesses, with SSH being a heavily restricted last resort. That's not to imply that everyone else shouldn't strive to do meet that (modern) bar too.
- Dedicated teams focus on minimizing access through time-based, role-based, and purpose-based controls.
- They rarely experience conventional hacks due to reduced blast radius from attacks and insider threats.
- Leading security experts in both major tech companies and niche organizations are driving new strategies and ways to think about security... their focus includes access reduction, resilience, and reliability, regardless of whether the solutions are closed or commercial for them. The ideas spread. (looking at you, Snapchat, for some odd reason)
- This is key: This evolution may not be obvious unless you actively engage with those at the forefront. I think it's what makes people think like the comment above. We cannot see everything.
- It's crucial to recognize that security is a dynamic field... with both open-source and closed-source solutions contributing.
So, the notion that volunteer-led projects are inherently more secure overlooks the significant investments in security made by major corporations that host the internet, and their relative success in doing so. Their advacements are coming to the rest of the world (eventually).
> especially when the product itself claims security as a core principle
My thought is that both volunteers and corporations contribute. In different ways, too.
One example is how a YC company made an open source version of Zanzibar. Zanzibar was an open paper to the world from Google that describes a flexible, resilient, fast access control system. It powers 99% of ACLs at Google. It's /damn/ good for the whole world and /damn good/ for developers' sanity and company security.
Corporate endeavors may fail, but they are often intense in direction and can raise the bar in terms of UX and security. Even if it's just a whitepaper, it still cannot be discounted. Besides, the larger places focusing on security aren't getting a big blast radius hack all that often, yeah?
I'm curious though, you've intrigued me. What kind of evidence or just lightweight ideas are you thinking of wrt volunteer led being more secure? No need to dig up sources if it's hard to find, but the general direction of what makes you feel that would be useful.
OpenSSH, as load-bearing infrastructure for much of the Internet, is heavily sponsored by tech companies. It empirically has one of the best security records in all of general-purpose off-the-shelf software. If for some benighted reason I found myself competing directly with OpenSSH, the very last thing I would pick as a USP would be security.
Yep. I would not attempt to differentiate against OpenSSH based on security track records. It's one of the most trusted pieces of software in the industry.
@dns_snek, it's right there in my real name username, comment history, and profile. :)
My entire youth and professional life I've seen nothing but footguns with actual practical use of SSH. The HN community loves to hate, but the reality is that almost no one uses SSH safely. It's near impossible. Especially when it comes to configuration, keys, passwords, and network factors.
I observed the common SSH failure patterns, and I made the most obvious footguns less than lethal. Looking a step further, I made remote terminal access a pleasure to use safely even for absolute novices.
So to your point about being in YC: In doing so, I thought it would be beneficial to join a community that supports one another (YC) so that an option (Teclada) can scale to make a real impact in the world WRT the warts and footguns of SSH.
Not the person you are asking but the two footguns I see all the time in corporations are the use of multiplexing in SSHD and the mismanagement of public key trusts in authorized_keys. I suspect this may also be an issue in the DoD given none of the federal hardening guidelines address this so I hope they are reading. Especially right now
Multiplexing: This feature improves speed when using SSH to proxy applications but it also gives an authentication-free unlogged channel to phishers, allowing zero friction trivial bypass of MFA. There are ways to log this with auditd but very few companies even touch auditd much less customize it to catch phishers. With multiplexing and phishing, corporate firewalls effectively become non existent and phishing is highly effective even in places one would think it would not be. Phishing capabilities include but are not limited to any org with access to public email or public chat, Slack, Instagram, Signal, Whatsapp, Discord, etc... Loose Lips Sink Ships
Public Keys: People instinctively worry about private keys but the biggest risk in OpenSSH in my opinion is the lack of auditing of what public keys are created by whom or what and trusted on what accounts. I can for example add a key to your account if I have temporary root privs and the much later log in to a system as you do bad things and now you are the first person investigators look at. Combine this with passwordless-sudo and it's game over, not that one really needs root to pilfer secrets from a company or let competitors destroy it. In most companies this is a hot steamy mess that people choose to ignore and do mental gymnastics to avoid it all together.
How to do these things properly is a much bigger topic, too big for HN comments.
Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.
It's still an insinuation even if its true. You should consider rephrasing your comment and not starting it with "did you forget to say...". Presumably they did not forget to mention this.
The point of the guideline is that accusing people of commenting in bad faith --- another guideline here --- makes for bad, boring conversation. What's worse, the comment you responded to made a bad argument that I think is easy to knock down on its merits, and by writing like this you weakened those rebuttals. Don't write like this here.
I'm sorry you feel like I don't live up to your standards, I believe that transparency about affiliations and conflicts of interest is paramount to healthy and productive discussion. Disclosing these things when criticizing competitors is really basic etiquette.
And look, it was just a simple nudge for them to disclose their affiliations more plainly in the future, while also providing relevant (and apparently appreciated) context to other readers. It was a footnote, not an argument.
A sensitive product like this would have to defend against well funded, patient, well resourced threats, including but not limited to infiltrating an organzation in order to plant code that only a few people may even be able to notice.
As an employee, I've typically needed to show up in person, but I've worked with contractors who never showed up in person. I've even been such a contractor at times.
Lots of commercial products use contractors and licensed code in the final product.
At least with most open source projects, a lot of the contribution process is in the open, so you could watch it you wanted to. As DonHopkins writes elsewhere, few people do, but it's possible. Not a lot of commercial projects offer that level of transparency into changes.
I worked at my current job for 3 months before I met a coworker in person. That might slightly help at a legacy butts-in-seats factory, but doesn't do a lot for remote jobs. I could be proxying in from Romania for all they'd know.
Thankfully, we aren't limited to asking leading questions and then hand waving at it; we have a rather lot of empirical evidence. OpenSSH is 24 years old; has it ever been successfully backdoored?
> We don't know. We won't know the negative case, but we may someday in some circumstance find out the positive (bugged) case.
But that's either the same with any tool regardless of whether it's commercially supported / FOSS / made by anonymous devs or not. If anything, FOSS is easier to audit.
SSH is kind of a swiss army knife. But 1000x sharper ;) The delta I'm speaking of would be to have bespoke tooling for different needs. And the tooling for each purpose could have appropriate, structured logging and access controls.
With SSH you can do almost anything. But you can imagine a better tool might exist for specific high-value activities.
Case study:
Today: engineering team monitors production errors that might contain sensitive user data with SSH access and running `tail -f /var/log/apache...`.
Better: Think of how your favorite cloud provider has a dedicated log viewing tool with filtering, paging, access control, and even logs of what you looked at all built in. No SSH. Better UX. And even better security, since you know who looked at what.
---
There are times when terminal access is needed though. SSH kinda fits that use case, but lacks a lot. Including: true audit logging, permissioned access to servers, maybe even restricting some users to a rigid set of pre-ordained commands they are allowed to run. In that cases, a better built tool can allow you to still run commands, but have a great audit log, not require direct network access (mediated access) to servers or internal networks directly, flexible ACLs, and so on.
It's off topic, but in my consulting and networking, security/firewall appliances are an easy first line approach I see companies buy in to. The security sales pitch sounds good and makes you feel good. Cannot name names.
I mean, everybody has a perimeter, even the ZT believers, but I think the notion of large networks protected by like a high-end NetScreen or Palo Alto firewall is 10-15 years out of date. We have, like, Tailscale, and netfilter.
Sometimes commercial companies have "incentives" to put backdoors, like i.e. secret orders by intelligence agencies. Snowden papers and all related information from that time set a baseline on what you may consider safe.
We must first precisely define "level of security" that is expected from OpenSSH and a commerical version. Only then the discussion about who can guarantee what would make sense.
Of course not. OpenSSH comes with no warranty, read the license.
Historically, it's been pretty good though.
If you would consider a commercial alternative, consider how much you would need to pay to get an actionable warranty, and consider if you could find someone to do a warrantied audit of OpenSSH (or especially of OpenSSH in your environment) for that amount. It might be a better use of your money.
Has any open source project taken down the majority of single OS install base as quickly as CrowdStrike? Seems like they would have the "dedicated resources and a financial stake" to prevent such as situation.
Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees.
Now imagine why a nation state would want to get staff working in such a company.
Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.