Genuine question: given a language which has a real notion of parallelism/concurrency and can run real threads on multiple cores, what is the appeal for async/await?
Modern fast IO is built on top of facilities like Linux's epoll(), which don't block an entire thread. However, that means that when an IO operation is finished, we no longer have the original callstack around to keep track of what operations were supposed to happen next. Instead we need some sort of state machine that's able to pause and resume every time it does IO. But writing state machines by hand is much less convenient than using the callstack, because you lose all the nice language features that you'd normally use to compose things, like `if` and `for` and `?`. So async/await is all about writing normal-looking code with the usual conveniences that can instead be compiled into a state machine that lives off the callstack.
Why would cores be waiting? A thread blocked on synchronous I/O will in general not be scheduled on a core. This is true on all OSs that I’m aware of.
Not sure if Apache had some spin-wait loops or something, but if so then that was a bug in Apache, not a fundamental characteristic of doing synchronous I/O in threads.
You're still paying some costs: there might be scalability issues in the scheduler data structures (how long does it take to schedule one thread out of a million blocked ones?), you need memory to store their stacks (hope you're on 64-bit), when they become runnable, you have a lot of context switches (which cost even more now after the Spectre fixes). Probably others that don't come to mind right now.
As for Apache, it spins a number of processes up to a point. When there aren't any free ones, the clients don't get any data.
Async is about servicing thousands, or tens of thousands, of clients at once. Since everyone is convinced that their program will have tens of thousands of clients, they clamour for async.
This is about servicing tens of thousands of connections. Threads are ok in the low thousands, but now servers are in the realm of handling millions of current connections.
Part of the problem is that context switches are really slow, so for software that needs maximum performance people will go to great lengths to avoid them.
Time for a mini blog post contained in a HN comment!
(NOTE: A lot these descriptions are simplified)
Back in the old days we had Apache and its ilk, who approached handling multiple clients by spawning 1 thread per client. The model was simple and effective ... until you had thousands of clients, which resulted in overloading the OS with too many threads.
So along came nginx and its ilk. Instead of a thread per client, they used epoll and a state machine per client. This allowed Nginx to handle a massive number of concurrent connections since the state machine was much smaller than a full thread's stack, and nginx could implement its own scheduler instead of the OS's thread scheduler. But it's a more complex system, because you have to manually engineer those state machines. For a web server serving static content or routing connections that's not a big deal. For the backend to a modern web application? Not so much.
Eventually the web was no longer static content with PHP/Java backends; it was responsive, dynamic, explosive. And with those new requirements we needed ways to build complex web servers that could handle thousands or more clients at once. Apache's model wouldn't work; too much wasted memory and the OS still struggled with large numbers of threads. nginx's model also wouldn't work; it required too much engineering.
A lot of ideas began floating around. Around this time NodeJS showed up and exploded in popularity. Partially because it made building these backends easier. No threads to worry about; no custom state machines. Just nests of callbacks! It was crude ... but it kind of worked. Callbacks hell was ... hell, but less challenging than custom epoll based state machines. And most importantly, it was lightweight compared to the threading model.
So we've been evolving from that middle ground. Javascript added Promises, which simplified callback hell. And then eventually Javascript added async/await.
Ultimately, though, those two evolutions are just different ways of expressing the same underlying thing: custom state machines. Ah! See, whether you write a callback hell, a Promise tree, or an async function in Javascript, it all compiles to a kind of state machine. A blob of state that we can store and transport around in our underlying concurrency framework, and ratchet forward when asynchronous events are delivered by the OS.
So really, async/await is just the epoll model pioneered by nginx, but instead of having to write the state machines by hand, we can express them as regular looking code. And in fact, behind the scenes, all implementations of async/await whether it be in Javascript or Rust, are driven by epoll (or similar equivalent).
And empirically we know that epoll based models are just more efficient in terms of CPU and memory. Trying to use the OS's threading model hasn't worked out; you need a whole stack for every concurrent operation you're trying to perform, and the OS's scheduler isn't designed for the kinds of workloads we'd offload on it.
I guess the short of the long of it is that threads are great, but our OS's handling of threads just isn't good enough. Engineers have decided that putting in the effort of writing state machines, whether from scratch or with modern conveniences like async/await, is worth the cost.
The epoll model is a lot older than nginx: squid and thttpd date from 1996, 8 years before nginx. But back in the 1990s select() and poll() had limitations, which is what the c10k problem was all about. (See link elsewhere in this thread.) What nginx brought was much better ergonomics, lots of features, and really good implementation.