[geofft]

"What happened" was two-fold:

1. Starting especially a few years before 2000 but continuing today, the software industry is quite profitable, pays well, and has lots of openings, while the a academic job market in systems research continues to pay poorly and has much more limited openings. So if you want to do systems software research while also having an enjoyable quality of life, you might as well go to a company and get paid well instead of spending your days writing a thesis and grant proposals.

2. Computer science is a field where the cost of basic research equipment is low (a computer), and more interesting research environments generally are beyond the scale of academia (tens of thousands of hardware nodes, hundreds or thousands or more QPS of production load, etc.). That makes it quite different from e.g. biology or high-energy physics on one end where you usually need to be in academia to get access to the equipment or e.g. mathematics (including theoretical CS) and literature on the other where it doesn't matter where you are; in systems research you only get access to the equipment from being in industry.

That doesn't mean that systems software research, done in industry, is (or was or will be) irrelevant; it means that the narrower definition of "research" as "that which is done in academia" is inaccurate (including industry with the trappings of academia, i.e., people at Google or Bell Labs writing papers in academic journals and hiring people with Ph.D.s). Systems software research happens in industry and is quite relevant to itself.

[yingw787]

Commercial research is decently different from academic research, which is what I think Rob may be referring to.

Commercial research needs to keep in mind the existing legacy systems used by the sponsor. Innovations are more evolutionary instead of revolutionary as the field matures. They may be more tailored to observable pain points of the research sponsor. They may not be widely shared if they yield results providing a competitive advantage. While it may not demand immediate returns, commercial research does have an axe to grind. All of this hampers advancement in the field of computer science in general.

I also don't know if there's any kind of commercial research on the scale of XEROX PARC or Bell Labs. I can't think of any off the top of my head. Microsoft and Google do some pretty neat research, but I don't think they've shipped anything quite on a similar scale.

There's really no organization hiring the best talent to work on the kind of black swan events commercial research may miss. For example, I think it'd be cool to have a microcode-based OS; I've heard it would help with keeping operating systems secure. But who would fund it, and who would work on it? Right now it doesn't look like anybody would, and that might be what Rob is concerned about.

[geofft]

Some specific innovations off the top of my head that are pretty firmly outside traditional academic research, and seem more revolutionary than evolutionary:

- Linux's read-copy-update synchronization mechanism. It has been described in papers, but you're better off following mailing list posts or LWN writeups.

- Rust's borrow checker and lifetime system. It's built on existing well-known ideas (e.g. affine types) and there's since been some academic work on formalizing it, but the specific system Rust uses has no direct precedent, is pretty novel, and was developed outside academia. (Note that Rust came out of Mozilla Research, which is far, far smaller than Bell Labs but also an organization that intentionally works on revolutionary and not evolutionary improvements.)

- libdill and Trio's structured concurrency, a solid theoretical framework for handling async/await-shaped problems without turning your execution into concurrent spaghetti. The techniques are not unprecedented, but https://vorpus.org/blog/notes-on-structured-concurrency-or-g... is a better framing of it.

[pjc50]

> more interesting research environments generally are beyond the scale of academia

I think the real impairment to OS research is deployment. If your idea isn't compatible with one of the existing OSs, in such a way that it can run a web browser, then nobody's going to use it. Heck, even Windows Phone couldn't get adoption. OS ideas that require people to completely rewrite applications and interaction paradigms are non-starters no matter what benefits they offer - unless they can fulfil a need that can't be fulfilled any other way. So quite a lot of work goes into bypassing the OS entirely for hardware-specific single-program networking applications, and everyone else has to keep with their existing paradigms.

[theamk]

It's important to remember that consumer-facing apps is just a part of the market; and even with consumer-facing apps, the user interface is often only a small part of it.

Even totally plain-looking device could be full of innovative research: a network router which uses completely new kernel. A new network protocol or a compression algorithm. New programming language. Automatic verification and/or fuzzing tools. A network of internet of things devices which share no code with any of the existing OSs.

[pjc50]

> Even totally plain-looking device could be full of innovative research: a network router which uses completely new kernel.

True. Although all those kinds of devices tend to prefer "free" over "innovative", and to keep the OS layer as thin as possible.

> A new network protocol or a compression algorithm. New programming language. Automatic verification and/or fuzzing tools.

To me those aren't really systems software, but that may be a matter of opinion?

[wsetchell]

Re 2) Biology and Physics are both really expensive (think particle accelerators and human genome). We could dramatically increasing academic computer science funding to let tackle those large interesting problems.