[sofaofthedamned]

HSMs are shit.

In a previous role we used a major vendor's HSM to protect our private keys. VERY expensive kit, more expensive than the load balancers and servers combined.

We needed to use Elliptic Curve keys for a particular customer - so it got even more expensive as we had to buy:

1. A license from the LB vendor to use the HSM

2. A licence from the HSM vendor to use EC with the LB.

... even though they trumpeted these announcements of how radically great they were together we found:

1. The integration didn't work, full stop.

2. The version of OpenSSL we had to use (supplied) was about 18 months out of date

3. The specially b0rked version of OpenSSL supplied didn't support EC via a HSM

Even better - when Heartbleed came out I had a patch from RedHat on day 1. The load balancer?

Nope - nothing on their website - I had to create a ticket which said 'we are aware of the issue', at which point the ticket was closed. I questioned this and was told they couldn't keep it open, I had to create a new ticket every few weeks to find out whether they'd actually deigned to assign a bug id to the issue.

The HSM vendor just said nothing, zero, until a new version of the firmware was silently released 4 months later.

The whole industry is shit. I'd rather have a farm of Yubikeys than one of those HSMs.

[EvanAnderson]

We started looking into an HSM solution for a Customer and quickly found out that the entrenched "big" HSM manufacturers are all about lock-in, questionable documentation (and strong insistence that we use their very expensive service rather than attempt our own integration), severe granularity in licensing features, and _insane_ prices (on a 3-5 year product replacement 'treadmill').

We ended up using the Nitrokey HSM platform and have been very pleased with the results.

[tristor]

Don't forget broken HA features. HSMs are a great idea in theory that are ruined by incompetent companies making them. They soon become a gun your org holds to its own head with the outcome of a data loss incident or a hack because of their janky OpenSSL integrations.

[sofaofthedamned]

That was exactly my problem. Also, the consultant they sent to help us with it didn't know anything about Linux when they knew that was our platform.

I'd love to get hold of one of my old units from that vendor and try to attack it. I assume it hasn't been done as they're so expensive. It's basically the ultimate in security through obscurity, as it's too expensive to get one you're prepared to break.

[tluyben2]

We have a bunch of HSMs and we had to sign a quazillion pages which at least deny us and anyone we hire or ask from trying to temper with the machines. Also most of them you cannot buy; they remain owned by the vendor.

[pharaohgeek]

They're not (necessarily) shit. They're just appropriate for a VERY specific purpose. A lot of companies that use them aren't using them right, or are using them in a scenario in which they are overkill. My company (medium-sized financial firm) uses them to protect the keys for our internal PKI. For these types of high-security situations, an HSM is ideal (if configured and managed properly). But I've seen places that use them to protect the SSL keys for webservers. While capable of doing that, an HSM is really overkill for that purpose. And, often results in them being mismanaged in the long run because the wrong teams end up responsible for them.

That, and it's very important to choose the right vendor. I've worked with pretty much all of them over the years, and some of them can be a nightmare to deal with. Some of the devices themselves are INCREDIBLY frustrating and should never have seen the light of day (much like some of their sales staff!)

[tluyben2]

So how about doing a blogpost/pastebin with vendors, products and your experience?

[nailer]

Want to (please) name the vendor?

[sofaofthedamned]

Nope, they'll know who they are and i'm not dealing with these things anymore.

[akytt]

Why do you have your HSM where Heartbleed matters?

[mentat]

This exactly. You use a HSM so it doesn't matter if your server gets popped.

[xyzzy123]

I think one should be really careful making statements like this.

In the case where an HSM is used for e.g. database encryption, one absolutely still cares.

The attacker can politely ask the HSM to decrypt all the data. On the bright side, you didn't lose the key, and your HSM should at least securely log the fact that all your data just walked out the door.

For SSL offload the situation is less dire, but it is totally realistic that heartbleed might cause issues for a system with HSM assisted offload (e.g. leaking plaintext) because the protocol processing happens in software. Any sane architecture should protect the actual private key though.

In general, the right way to think about HSMs is that they protect your keys, not your data. If you care more about the data than the keys, you absolutely need to care about the security of your host.

[hinkley]

I used HSMs for a code signing project, and once someone can borrow your key to sign malicious payloads, it really doesn't matter if they have your signing key or not. You still have to get a new key, and clean up the mess.

[mentat]

As far as I understand heartbleed, the core issue was that you could search memory for the private key and then do whatever you wanted with it. With an HSM, that wouldn't be an issue. However, it's true that you could search for other data as well and that data could be sensitive, just not quite as sensitive (depending on your industry and architecture, of course).

[xyzzy123]

I agree with you -- it depends on your threat model.

If state actors or universal mitm or coffee shop attacks are primary threats then loss of private key is a very bad outcome for SSL offload. And this is what SSL is meant to help woth.

For many (more?) sites though, the issue was loss of session plaintext (e.g api creds, basic auth creds, pii or credit cards). For a lot of applications where just hitting the webserver remotely is a lot easier than MITM, this was actually the more severe problem.

[mentat]

I have nation states as part of my threat model so stealing keys enables passive and active monitoring vs popping a box and getting some limited subset of connections intercepted.

[yuhong]

OT, but I wonder how many of these LBs are/were TLS version or extension intolerant: http://www.bus.umich.edu/KresgePublic/Journals/Gartner/resea...

[pmarreck]

Is there any way to do what a HSM does, in software? (Maybe with OS support)

[tjohns]

If your definition of an HSM is just "hides my key material from my application server", then sure. This is basically the idea behind HashiCorp's Vault.

However, you'd still be vulnerable to someone gaining root access on your Vault server. A true HSM uses physically secure memory and a dedicated cryptoprocessor to make it impossible for the secret to ever be leaked, even if the host OS is compromised. This isn't really something that's possible in software alone.

(In fact, the Vault documentation even mentions that you might still want to use a dedicated HSM for security, since they implement a plugable mechanism for secret storage.)

That said... you could use a TPM chip or ARM TrustZone to implement true TPM functionality. It's still a hardware based solution, but it's hardware that's much more affordable and possibly already inside your computer.

[helloiamaperson]

take a look at my post from earlier: https://news.ycombinator.com/item?id=13031870 . I'd be interested to hear your (and others') thoughts.

[EvilTerran]

Not really, no. AIUI, the whole point of having a hardware security module is that it only leaves a very narrow attack surface (ie, the protocol for communicating with the device) for malicious software to try to exfiltrate the sensitive data through, with no way to get around it - so if that interface is sufficiently hardened, you should be secure regardless of any weaknesses elsewhere in the system.

In contrast, while you could think of, say, an ssh key agent as a sort of "software security module", such things are inherently vulnerable to low-level attacks that side-step the intended interface. Say, a kernel exploit that grabs the decrypted key straight from the agent's memory.

So if you need to keep your secrets safe even if you unwittingly use them on a comprised OS, you can only pull that off with a hardware module.

[cyphar]

Yes, but then you lose the whole benefit of a HSM -- that keys generated in the HSM are physically protected from being extracted. Aside from that, a HSM is just an implementation of regular crypto algorithms.

[tluyben2]

Yes you can and I know at least two companies that do and are certified by the same standards as hardware HSMs. So you can and certify at least as an issueing bank (and because of that, probably for a lot more).

[baby]

This is an area of cryptography called whitebox cryptography. Tl;dr is that most solutions nowadays are proprietary, and all open sourced and documented whitebox cryptosystems are broken.

[matthiasb]

introduction: "we used a major vendor's HSM to protect our private keys"

conclusion: "The whole industry is shit."

You had a bad experience with one vendor, not with the industry.