[mlosapio]

Wonder if services like Let’s Encrypt were affected. I imagine a scenario where a small hijack of DNS could allow for properly signed certificates for domains that are not owned. If I operated a CA service, I would carefully examine the requests received during this time frame. Maybe someone can audit the Transparency Logs during this period for anomalous activity.

[erulabs]

I would assume they go far out of their way to resolve the address from a large number of distributed DNS resolvers - the question is if they're just round robin instead of multiple simultaneous queries, but I would hope that's part of why the DNS test usualy takes >60 seconds. If they're not treating DNS as a consensus protocol instead of as an immutable database, they'll always be open to poisoning attacks.

If any certs were issued for hijacked domains (which as far as I've ready was only one, not using LetsEncrypt), it's a pretty glaring failure on the issuer, assuming they used "DNS Validation"

[jaas]

Let's Encrypt does not currently validate from multiple perspectives but we're working on it. We're cooperating with a research team at Princeton to make sure our strategy is an effective mitigation against BGP attacks.

Any given production validation right now may come from one of two of our datacenter locations, but not both.

You may see multiple validation requests if you use our staging servers, but that's just an early version of multi perspective validation which doesn't work well enough to promote to production. There are some performance issues, and the external validation points are all on AWS, which uses the same autonomous system (AS) across datacenters. In order to effectively defend against BGP attacks we're probably going to have to deploy validation servers on 3+ different ASes so that our network perspectives are sufficiently diverse.

If multi perspective validation is effective and nothing is done to significantly improve BGP security (we're not holding our breath) then I expect multi perspective validation to be a requirement for all CAs down the line.

[jnw2]

If you can do validation from a perspective where your ISP can send you a full BGP table, you should be able to keep track of how recently the most specific route to that destination has changed; something that has changed recently or that currently has an AS path different than what has historically been typical may be suspect.

AWS does have some network diversity in different regions; they apparently do not operate their own backbone to interconnect the different AWS regions.

(And for all that Hurricane Electric seems to have not been doing a great job of filtering BGP announcements from peers, they do seem to have pretty aggressive pricing on colo space in their Fremont data center where they can provide a full BGP feed.)

[0xbadcafebee]

I'm trying really hard here not to be snide, but it is amazing to me that an organization that is responsible for securing the world wide web is basing that security on the hope that nobody can spoof 3 AS's at once.

Just give up on BGP. Strongly suggest people use DNSSEC. For TLDs that don't support DNSSEC, require a public key issued to the registrar. You (the Certificate Authority) can get the public key from [R]WHOIS, IRIS, RDAP, whatever method you like, directly from the registrar. This is standard practice using the thin WHOIS data model, where .com and .net require domain registrars to maintain their own customers' data. Other TLDs simply run thick WHOIS servers that store all the data for their domains.

Registrars are already required to pass up Delegation Signer records to a TLD to support DNSSEC. Passing on a similar record to a Certificate Authority would be practically the exact same thing. So we know the registrars can support it, and we know it would ensure that people would only be able to generate certificates for those domains that they actually own (and not "whomever can control the IP space currently").

[pawal]

We should not "give up on BGP". What we should do is to improve security in all layers. This includes BGP, and as you mention DNS. DNSSEC should be mandatory, just as TLS, for any business that take themselves seriously.

[rphlx]

Not to start another DNSSEC melee (as I start another DNSSEC melee..) but for it to be effective we really need browsers to a) be able to tell whether a DNS response was properly DNSSEC signed or not, b) produce some large, scary, red warning for ones that aren't, mark them "Not Secure", etc. [1]

There is a major chicken-and-egg/game-theoretical problem though: any browser that does that today will piss off/irritate its users, forcing them to use other browsers or older versions of the same browser, as DNSSEC is not widely deployed on the corporate side. And until most/all browsers do something major to make the current DNS security crisis obvious to large numbers of users, most companies won't care enough to deploy DNSSEC.

At least it will be interesting to see whether, and how, this eventually gets solved.

[1] Certain abysmal DNSSEC cryptographic choices, such as 512-1024-bit RSA, should also be addressed.

[zjs]

> There is a major chicken-and-egg/game-theoretical problem though: any browser that does that today will piss off/irritate its users, forcing them to use other browsers or older versions of the same browser, as DNSSEC is not widely deployed on the corporate side. And until most/all browsers do something major to make the current DNS security crisis obvious to large numbers of users, most companies won't care enough to deploy DNSSEC.

This seems analogous to the problem browsers faced with Flash. Perhaps they can leverage the same incremental approach here.

For example: 1) Validate DNSSEC. If present and valid, the HTTPS "green lock icon" gets a bonus glow. 2) 6 months later, not having a DNSSEC response gives a little red X badge on the "green lock icon". 3) 6 months later, not having a DNSSEC response graduates to a little ignorable info/warning box near the location bar. 4) 12 months later, you have to click through a big scary warning to access the site.

Essentially, start by providing a carrot and then introduce a progressively larger stick. Communicate the whole plan up front so that large organizations can get the ball rolling.

[dcbadacd]

> Strongly suggest people use DNSSEC. For TLDs that don't support DNSSEC, require a public key issued to the registrar.

I think this is absolutely unreasonable and damaging to me as a small player. I don't want to give my registrars more power to mess with me, they already refused to support DNSSEC on my domain because I am not using their hosting service, I don't want them to be responsible if I can get certificates of not. DNSSEC is also way too hard to set up (especially if, for example, I use OVH's name servers and another company as registrar), we need DNSSEC equivalent of LetsEncrypt's certbot for it to be usable.

[peterwwillis]

DNSSEC is a limited hack. Ok? I only suggested using it at all because it can be used right now to secure the process of generating certs. There is a very good reason I don't want to put all my eggs in that basket.

The root servers and TLD are not supposed to be the authoritative source of trust for your domain. That is the registrar's job. That is the registrar's only job: control my domain, and don't let anyone fuck with it. Hence, they should be the ones to keep a record that you can control to tell people where trust should lie. If you don't like your registrar, you can transfer your domain to a different one.

The purpose of DNS is to be a phone book. When you call a number from the phone book, the person you are calling might not be who you expect, even if you got the number from a very trustworthy source. Perhaps a spy is sitting on the other end of the phone. After you connect to the phone number, you should authenticate the other end.

Why not put the authentication information in the phone book? Because you got the phone book from one place. The spy could have compromised the phone book, and now you have no way to know if your authentication is real. It is a better idea to have one phone book for calls, and one separate authentication book that you get from a separate source.

DNSSEC's security depends on one child publishing one key to one parent. If you attack that one factor successfully, the security is broken, and there are multiple vectors to attack. There is no defense in depth. For a highly motivated state actor, this is not difficult to circumvent. The more you lean on DNSSEC, the more fragile the entire internet's security becomes. A better method is to decentralize and distribute trust into multiple organizations and processes, so that it would be very difficult to compromise an entire internet system.

I haven't mentioned the many fundamental problems with DNSSEC rollout and use because I think those are mainly a problem of lack of incentives, and are addressable. Go ahead and use DNSSEC if you want. Just don't tie a rope around your neck by making all internet security dependent on it.

[btown]

Per https://github.com/letsencrypt/boulder/blob/master/bdns/dns.... it seems they round robin. But they are aware of the issue in the spec: https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.htm... - “Querying the DNS from multiple vantage points to address local attackers” is a mentioned mitigation that a server could implement.

Seems like a reasonable basis for a pull request.

[mnordhoff]

> why the DNS test usualy takes >60 seconds

The server-side part of DNS validation takes about a second. The delay is all about clients waiting for their authoritative DNS servers to update. If you use a fast DNS provider, there's no reason to wait longer than necessary.

> If any certs were issued for hijacked domains (which as far as I've ready was only one, not using LetsEncrypt), it's a pretty glaring failure on the issuer, assuming they used "DNS Validation"

It wouldn't be a compliance failure, though. CAs are not required to be invulnerable to BGP hijacking attacks.

[peterwwillis]

It doesn't matter how many DNS resolvers you have. If you can spoof BGP, and use it to spoof the DNS resolvers, and the authoritative name server, you can stand up your own DNS that says anything you want it to say.

The only thing you cannot do is fake out DNSSEC. If you use a TLD which supports DNSSEC, you can make sure records have to be signed with your key. If fake DNS records aren't signed with your key, and the CA uses a validating stub resolver, and the CA is checking for CAA records, they will reject the attempt to create the cert. In theory.

Here's the thing. The CAs know about these flaws and are still not providing more secure ways to prevent these attacks. How could it be more secure? The domain registrar could accept a public key from you at the time of purchase. At that point, nobody should ever be allowed to generate a cert anywhere without you signing off on it with your key. Is this some magically impossible technical challenge which mere mortals can't comprehend? Hell no. They literally just need to write an ASCII file in a database next to your domain name. But we aren't demanding it of them, so they aren't doing it. So attacks like this leave everyone vulnerable, because nobody cares enough to fix it.

[cjbprime]

And if you lose the key, as thousands of people would every day at a large registrar?

[peterwwillis]

Use the same method you use to pay for the domain to reset the key. This would require logging into your account at the registrar, or contacting support and providing enough verifiable material to reset it.

If the registrar signed a message that had your key in it, they could re-sign it with a replacement key. So now you can 1) verify who originally assigned the domain, 2) record the public key, and 3) see a verifiable history of changes as long as the old history is included in successive signing. This could be used as part of a public record ala Certificate Transparency, or a blockchain-style system. If the attacker spoofs BGP and SE's your registrar, you (or whomever) can see if anyone creates a new key.

[dexterdog]

But on a big, distributed site the DNS is going to vary greatly depending upon the location of the query.

[technion]

I'm going to assume that Lets Encrypt wasn't impacted for the reason that there are several discussions that the hijacked myetherwallet website apparently had a self signed SSL cert running - and people were actually clicking through the warnings before handing over passwords.

Surely they would have realised they could make the whole thing a lot more profitable if any SSL provider was impacted.

[kchr]

But if you route all traffic for a specific domain through your own web server, surely you could complete the Letsencrypt verification steps as well. They just check for a specific file on the remote web server, right?

[technion]

Yes, but they weren't routing traffic for that web server to their own server. They were routing the IP of the DNS servers to their own server, and then just handing out the DNS address that suited them.

In turn, if your own DNS wasn't configured to use a DNS server with a poisoned fraudulent address, a web server based verification landed on the valid server, not the attackers.

[kchr]

Okay, thanks for clarifying!

[solatic]

Isn't this exactly the problem that EV certificates solve? By going the extra mile to validate identity instead of just trusting DNS ownership?

Not that there's an easy way for LE to validate identity automatically - governments need to do a better job of providing electronic/smart identification, in which case a physical chip ID serves as a second factor (alongside DNS ownership) for automatic EV certs - but the weaknesses of only having DNS ownership are pretty much why EV certs exist.