[tptacek]

The more I think about this story, the dumber it seems to me.

The narrative seems to be, L0pht testifies, world ignores them, chaos ensues. But Mudge's testimony coincides almost perfectly with a software security renaissance. The reality is more like: L0pht testifies, world ignores them, gigantic sea-change in security leads to 9-figure investment in securing Windows, the near eradication of SQL injection from popular applications, universal deployment of TLS in financial applications, chaos ensues anyways.

What, exactly, would be different if people had "listened" to the L0pht? Would we have S-BGP? DNSSEC?

The simple fact is: in 1998, when this happened, nobody knew how to fix any of the problems. If we had known, we'd have been doing that. There were still servers in 1998 that used deslogin.

I'm very happy that a bunch of people I like got to put their handles on nameplates and get recorded testifying to dummies in Congress. I do not, however, think it was an event with much meaning.

Later.

I think it's literally the opposite of the gist of this story. Everything is much, much better than it was in 1998. We have made surprising progress, and addressed security problems with an improbable seriousness:

1. Most new software is no longer shipped in C/C++.

2. The devastating bug class introduced with new languages (SQLI) was, for public-facing software, ratcheted back from "universally prevalent" to "rare" within a decade.

3. 3 billion Internet users all run software that downloads unsigned code in a complex, full-featured language with a dizzying variety of local C library bindings, right off web pages, executes it locally, and it's a news story when Pinkie Pie wins Pwn2Own with a working reliable Chrome clientside.

4. Anyone who wants strong crypto can have forward-secret elliptic-curve DH AEAD transports with a config file tweak on their servers.

5. Microsoft went from MSDOS levels of security to "you can live like an investment banker if you can reliably produce a couple Windows exploits a year" levels of security, again inside a decade.

6. Despite its emergence as an entire new category of computing platform, with its own new feature set, the most popular mobile OS has --- it appears --- zero effective malware outbreaks.

7. Remember Sendmail? Remember BIND? Probably only if you're a security nerd. The last working SSH vulnerability was how many years ago?

As usual: everything's amazing and nobody cares.

[dmix]

> the near eradication of SQL injection from popular applications

There is a great chart from an analysis on the history of software bugs showing the rise and decline of SQLi/XSS bugs from between 2005-2010:

https://imgur.com/cY2cJ8W

Memory bugs remain relatively consistent (despite PaX and others effort to change that).

Source: https://www.isg.rhul.ac.uk/sullivan/pubs/tr/technicalreport-...

[tptacek]

It's painful that they combined XSS and SQLI, because progress on SQLI has been much better than progress on DOM corruption bugs.

[dmix]

Indeed, I'd love to see an expanded chart (as well as a more recent one). This one is focused on two categories, memory vs web vulns.

[tptacek]

I'm not in love with White Hat as a company, but they do collect stats across their customer base, and their annual stats have shown sharp declines in SQL injection.

DOM corruption is a somewhat complex class of vulnerabilities (see lcamtuf's "Notes From A Post-XSS World" for an example of why), and it's not surprising to see we're making less progress.

[sebcat]

> I'm not in love with White Hat as a company, but they do collect stats across their customer base, and their annual stats have shown sharp declines in SQL injection

White Hat has a set of tests they run against their customers over time. They tell their customers what problems they find. Their customers (mostly) fix the problems.

I'm not sure that translates correctly to the outside world. The fact that their stats show a decline in the presence of SQL injection vulnerabilities could only be showing us that they have more old customers that have gone through a couple of reports and patch cycles than they have new customers who might not yet have fixed what they're told to fix.

[tptacek]

I don't know: their observations square with my anecdotal observations over 10 years of appsec consulting. On my first ever web pentest, I got a 'OR''=' SQLI in the username of a login form. In 2014, when I left Matasano, that would have been absolutely shocking. SQLI has become far less common:

* Developers are taught to use parameterized queries

* Fewer big applications are built in PHP

* More projects use ORMs now than don't

* Random testers hoping for bug bounties hammer every application with SQLI scanners

[chx]

With all due respect to you, everything would be different. I know even then it would've been a daunting thing to do but perhaps 1998 was the last year when it could've been done: tear it down and rebuild it securely.

I am sure you know this too well but let me remind a few people here who were not even born when some of this happened: The early years were mostly of trust. As an example, I remember running around even in 1994 with a 10BASE2 network tester to find where the network broke. 10BASE2 was a perfect example of a more trustworthy age: many machines shared a single cable and eavesdropping required zero effort. Every machine got all the frames and there was no encryption. Then came Fast Ethernet and with it 100BASE-TX and slowly switches replaced active hubs and this went away. But it often required rewiring buildings which took a long time. I do not have hard data and it's hard to define the start and end points but I would say it took at least five years if not a whole decade to really sunset 10BASE2.

Now of course doing the same on a world level would've been a daunting task. However the number of websites at this point were growing somewhat sedentary at least compared to years prior and later, see the data http://www.internetlivestats.com/total-number-of-websites/ here, compared to the meteoric growth in 1997 (334%) and 2000 (438%) the years 1998 (116%) and 1999 (32%) were slow and peaceful.

[tptacek]

I understand that's what people think, but what I'm saying is that in 1998, we wouldn't have known how to rebuild everything securely. We'd have ended up with slightly better C standard libaries, S-BGP, IPSEC, and DNSSEC.

Here, let me sum it up this way: I think it's possible that the L0pht testimony predates SQL injection.

[mikekchar]

Buffer overflows were certainly recognised considerably earlier than that. I remember a colleague pointing out buffer overflows in the first STL string implementations. It's not hard to go from that to SQL injection, or any other similar technique.

Certainly, my (possibly rose-coloured) memories of the time includes a lot of, "OMG. How stupid can people be? Surely they know better than that!"

I guess what I'm saying is that some people definitely knew what to do about this and were trying to do it. Most people were ignoring it and saying things like, "Oh, you're just being paranoid. Why would anyone try to do something like that?" It's a bit pointless to say, "What would have happened if people had listened" because the point was that people didn't listen. That was the whole problem.

[tptacek]

The first modern overflow exploit was Thomas Lopatic's 1995 HPUX httpd exploit. When he wrote it up, he claimed it followed the blueprint of the "microscope and tweezers" paper Spafford wrote about the Morris worm. The Morris Worm, of course, was from 1988. In the years between 1988 and 1995 there were, so far as anyone knows, a total of zero code-exec buffer overflow exploits.

I was in the room with Peiter, at a DC Summercon, as he tried to work out the exploit for Sendmail 8.6.12 that 8lgm had teased. He definitely didn't have it before 8lgm, and 8lgm didn't have it before Lopatic. Even the virus guys didn't have it.

It's weird to think that nobody put two and two together in, say, 1991 --- there certainly was motivation (that's the timing of the Sun-Devil Raids!) and so much vulnerable software.

But then, in the late 1990s, people honestly thought they could mitigate overflows by moving buffers from the stack to the heap. Reliable heap exploits were a big deal as late as 2003, when Matt Conover spoke to a packed CanSec room about the Windows Heap, in excruciating detail for over an hour. That's close to a decade between Lopatic and mainstream heap exploitation on modern heaps.

>shrug<

[mikekchar]

It is possible that I am misremembering. I remember him submitting a bug and being ignored as a crank, though ;-). It may have been some other kind of memory corruption.

It's hard to believe that it's only been since the late 90's that buffer overruns exploits have been around. I will have to believe you as you have considerably more knowledge on the subject than me.

I'm very much wondering now about the times I used to boot trace games to crack them and if I ever used such a technique. It seems so obvious now that I may be assuming that I must have, but it's so long ago that I really can't remember. Certainly getting the loader to move your code around rather than theirs was a normal trick.

[kriro]

I'd say buffer overflows "went mainstream" roughly after November 1996 when Phrack 49 with "Smashing the Stack for Fun and Profit" was released. At least I'd guess that's the most influencing article on the topic.

[acdha]

I thought by then there was at least some awareness - old memories of people evangelizing prepared statements with placeholders – but this was the oldest reference I found, from December 1998:

http://phrack.org/issues/54/8.html

[chx]

Hrm you are right it certainly predates XSS http://www.thesecuritypractice.com/the_security_practice/201...

[anon3_]

> better C standard libaries

What's the story with C standard libraries and security? How could they be better, as you see it?

[wglb]

You see, this is a much older discussion than most folks think. Back in the 70s, there was a lot of talk then about how badly software was built. "If buildings were built the way software is built, the first strong wind would destroy civilization". Remedies were proposed, dogmas (er, sorry, methodologies) developed, proselytized, replaced. Languages came and went. Howling winds came (see the "voodoo gods" in Count Zero) when we connected this all to the internet. Y2K came and showed some of the underbelly.

So, tear it down and rebuild it securely is actually many years to late 1998.

There are a few that know how to build software that isn't swiss cheese. Just picking two that I know of, nobody reads the whole volume set (as noted in Coders at Work), and for the other one, nobody wants to use it because it isn't under active development. The idea, even today, that a chunk of important internet software can actually be finished seems to be met with cognitive dissonance.

And there are organizations that know how to build very good software. But in todays fast moving businesses, who wants to be in a CMM 5 organization? Doesn't sound like much fun to me, and probably not to you either.

My estimate of the last year that all of this could have been fixed was at least 30 years earlier than your estimate. Or sooner. I worked in an organization whose core software, running today, was first written about 50 years ago.

[pjc50]

"Tear it down" never happens for systems that are basically working, even if they have serious flaws.

[chx]

It definitely happened to 10base2!

[basch]

I think if the world had listened the software engineering business process would be different. Security would be a primary concern. Instead of management focusing on cost and time to delivery, ignoring technical concerns and treating IT like things like a black box cost center. Security is often a reactive concern, Target/Home Depo take it seriously AFTER they have lost their customers data once. If the world had listened, boards of companies wouldnt brush off concerns, or wouldnt be in these situations. If the world had listened people would be willing to spend more on the up front cost of development to do things right the first time. If the world had listened, upgrading decaying infrastructures would be preemptive.

The people this message should have been directed at were non-technical executives and managers.

[anon3_]

> Most new software is no longer shipped in C/C++.

What language are the applications in? What makes them more secure than C/C++?

[blub]

Prety much anything is more secure than C, because the language doesn't offer any support for the programmer. No matter how careful one is, a simple mistake could lead to a (potentially exploitable) crash.

Modern C++ can be written in a secure way, however, this requires more discipline and knowledge when compared to languages such as Java or C#, which are very forgiving with programmer errors.

As for what software is written in, that depends. For Linux, I guess that's C and C++. For Mac it's mostly Objective-C, C and C++. For Windows probably a mix of C#, C and C++. On iOS it Objective-C and C or C++, but new apps will maybe move to Swift. On Android it's mostly Java, with some C and C++. Web apps don't use C or C++, however, they're not as hot as they used to be, mobile is the cool thing now.

As one can see, there's actually a lot of C and C++ in production or still being written.

[afarrell]

> dummies in congress

Are the folks in congress actually stupid? Or do they practice a different profession than you? Namely: the structure and interpretation of laws and policies.

How much do you know about, say... the field of nursing?

[Qwertious]

The problem is that they deal with making laws on a variety of subjects, which necessitates understanding said subjects. They don't understand the subjects.

Say what you like about programmers, but most of them don't actually have any job-related responsibilities in the field of nursing, breaking the analogy.

[afarrell]

It isn't meant to be an analogy but a contrast.

They are indeed required to understand the subjects, but they have no realistic way to. There are simply too many subjects. If we want to sit around saying "legislators are dumb and don't understand us", fine. It won't solve anything, but it will make us feel really nice about how smart and special we are. I like feeling smart and special too.

But if we actually want to fix anything, we have to think about the system wholistically and understand what motivations and pressures a legislator is under. There are simple too many subjects for a legislator to understand all of them well. Committees help somewhat, but are flawed. Lobbyists are the current way that legislators gain information about industries but that comes at the cost of drastically warping priorities. If anyone wants to comment with some actual insight and detail into those problems, that would be nice.

[na85]

>Are the folks in congress actually stupid?

Not all. But some of them are, for lack of a better term, really fucking stupid.

[Demiurge]

Some of them actually, honestly, don't believe in climate change, or nursing.

[CamperBob2]

It's more or less beyond question that some of them are either genuinely stupid, or genuinely evil. Which do you prefer? I'm not sure, myself.