I was working on adding custom sharding for a reverse proxy in Nginx ten years ago. The code was absolute bare-bones. No comments, no tests. And still it worked really well. Scary and cool, were my thoughts at the time.
There are three things I think stood out (not tied to C10K):
1. The configuration format is light-weight. Compared to Apache, lighttpd and others at the time, you could build a static file server or a reverse proxy in just 3-4 lines of configuration. It lowered the bar of entry, and is probably what led to wide adoption.
2. The core of Nginx was (is?) an async data pipeline. The individual modules (proxy, file system) defined how the pipes tie together, but the actual pumping of data was done in a kernel. You never had to care about epoll(2) and the like; you just defined the DAG. And that was easy to do correctly even in bare-bones C. This was a good architecture.
3. Single-threaded IIRC, which might be the C10K answer you were looking for. Apache had the complicated configuration where you had to decided to use prefork, or threads, or...
Lastly, it was fast. Probably because of (2), and a prerequisite for (3).
If memory serves me, nginx succeeded by relying on epoll primitives for handling many connections rather than spinning up a thread per request by default like apache did at the time. That was the big difference back then. These days I imagine Apache has adopted/honed these same techniques.
epoll has the advantage of operating in O(1) time rather than O(n) time as well which becomes important when you have a lot of file descriptors.
I'd also note that epoll landed with Linux 2.6 so it wasn't really available before 2004. Apache Server was created in 1995 long before epoll and Nginx was initially released in 2004. It's one of those situations where you introduce new capabilities like epoll being able to handle lots of FDs in O(1) time and someone finds a way to use that capability to make something great.
Was epoll around and no one was using it for a web server yet? It seems like something that would have been put in the kernel explicitly at the behest of web servers.
IMO Ngnix takes advantage of the fact that most of the web workload is I/O bound. It’s tight loop main thread coupled with asynchronous delegation enables it to stay single threaded. It doesn’t spawn new thread per request which means it doesn’t need additional memory to handle new requests.
This is a very good article which goes into details, highly recommended
There are three things I think stood out (not tied to C10K):
1. The configuration format is light-weight. Compared to Apache, lighttpd and others at the time, you could build a static file server or a reverse proxy in just 3-4 lines of configuration. It lowered the bar of entry, and is probably what led to wide adoption.
2. The core of Nginx was (is?) an async data pipeline. The individual modules (proxy, file system) defined how the pipes tie together, but the actual pumping of data was done in a kernel. You never had to care about epoll(2) and the like; you just defined the DAG. And that was easy to do correctly even in bare-bones C. This was a good architecture.
3. Single-threaded IIRC, which might be the C10K answer you were looking for. Apache had the complicated configuration where you had to decided to use prefork, or threads, or...
Lastly, it was fast. Probably because of (2), and a prerequisite for (3).