[tdmackey]

Coroutines bring their own headaches and baggage. To say that callbacks are behind "state of the art" is a little misplaced,; its just a different way of doing things. With coroutines you have to worry about IO all over the place and have to make your functions coroutine safe and Ryan Dahl, the node.js creator, will argue with you all day long about coroutines vs callbacks.

I do agree, though, that developers should not ignore other ways of doing things, just don't discredit the callback way of doing things as it can be useful.

[jerf]

"With coroutines you have to worry about IO all over the place and have to make your functions coroutine safe and Ryan Dahl,"

No, no, a thousand times no. Node.js partisans really need to start actually using Erlang or Haskell for a little while before spouting this canned line off. You do not have to jump through enormous hoops to deal with IO in Haskell or Erlang, it just works.

Callbacks are behind the state of the art. Coroutines or generators or any other cooperative-multitasking primitives in languages that didn't support them from day one and have enormous sets of libraries not "cooperative-aware" are behind the state of the art, too.

This is all just "cooperative multithreading" again and I am yet to see anyone explain why this time is going to turn out any different than last time we tried cooperative multithreading.

[codahale]

Would that I had more upvotes for this.

The actual cutting edge of event-driven server architecture[1] looks something like this:

    server = do {
      sys_call_1;
      fork client;
      server;
    }

    client = do {
      sys_call_2;
    }

(And even then it has marginal benefits in terms of throughput and latency compared to threaded implementations.)

[1] Li and Zdancewic. Combining events and threads for scalable network services. Proc. 2007 PLDI (2007).

[aredridel]

It's really a question of how much state you're storing, and how you're dealing with that. Many languages and many runtimes allocate stacks in megabytes, and store every local variable in it.

With a callback-based system, stacks are short, and you explicitly carry that state. You KNOW what state you're storing, and you can see it easily.

There's some benefit to that.

And the pattern for aggregating functions is different: Receive a starting event, emit a done event -- you aggregate processes into sets of events, not into function calls. So yeah, it's not going to follow some of the same patterns that non-event-driven code follows, and event-driven code is going to look rather different than callback-passing code.

I think Ryan's right about coroutines, too: Coming back to vastly different state after a simple function call is basically going back to the days of using global variables.

[jamwt]

Unless I'm misunderstanding you, you seem to be conflating purity with callback vs. coroutine--I'm don't know why the second would have any bearing one way or the other on the approach to the first.

If you don't want global variables, don't use them! If you don't want side effects, don't use them either. Those are the same decisions you make, callback or not. Both are great ideas.

Example 1 (callbacks):

  function do_request() {
    return get_from_database(argument, 
    callback=function (database_rows) {
        return make_html_table_from_rows(database_rows);
    });
  }

Example 2 (coroutines):

  function do_request() {
     database_rows = get_from_database(argument);
     return make_html_table_from_rows(database_rows);
  }

What about the second requires a vastly different approach to state? What about the io loop "up the stack" has fewer effects or global variables than the loop called into at the coroutine?

In fact, clean state management is easier!

Example 3 (coroutines):

  function do_request() {
     database_rows = get_from_database(argument);
     tweets = use_twitter_rest_api(...);
     return make_html_table_from_rows(tweets, database_rows);
  }

I can do the first two calls serially without either passing through the first, or using a global variable or some kind of catch all "context". The local frame is the context, which is the oldest, most straightforward, most tried-and-true context in the book.

[jerf]

You need to try Erlang. (Or Haskell, but Erlang is more approachable.) You and every other Node.js partisan keep making criticisms that simply make it clear that you have no clue what you are criticizing and that doesn't make it terribly likely that you're going to sway me to your point of view.

"With a callback-based system, stacks are short, and you explicitly carry that state. You KNOW what state you're storing, and you can see it easily."

Actually, no. You have implicit state carried around in the function closures and you will discover that it is very easy to have a leak in there that will be very hard to diagnose. I say this because I am speaking from experience.

(Remember, Node.js isn't a blinding new architecture. The architecture has been around for over a decade and I don't even know when the idea started. The only new aspect is that this time, it's Javascript. I've got a lot of experience with that architecture, and what that experience tells me is never again!)

On the other hand, it is very easy to examine an entire running Erlang system, see every process and the exact contents of its stack at that point in time, and the exact amount of memory currently allocated to it, and since Erlang doesn't have any sharing between processes, that state is everything about that process. It isn't always the best about giving back memory if you have long running processes, but I was able to diagnose which processes were consuming my RAM, determine why they were consuming my RAM, and test out a fix for the excessive RAM consumption (since it came in the form of sending a particular message sooner rather than later), all without shutting down my server.

You do not have that level of introspection and visibility in Node.js. I don't even have to ask.

"Many languages and many runtimes allocate stacks in megabytes, and store every local variable in it."

I'm not talking about "many languages". I don't care about "many languages". I'm talking about good languages. Erlang can very easily allocate a couple hundred bytes to a process, or less. I'm not actually sure what the minimum allocation is, but it is certainly going to be competitive with a minimal Node.js handler.

"Receive a starting event, emit a done event -- you aggregate processes into sets of events, not into function calls. So yeah, it's not going to follow some of the same patterns that non-event-driven code follows, and event-driven code is going to look rather different than callback-passing code."

None of that appears to have any relationship to coding in Erlang, from what I can see. Better technologies don't have to have events. They just code right through things. A loop for a simple proxy might look like:

    proxy(SenderSocket, DestSocket) ->
        case socket_read(Sender) of
            done -> done; %% return to the original caller
            {ok, Data} ->
                socket_write(DestSocket, Data),
                %% go back for more
                proxy(SenderSocket, DestSocket);
            {error, Error} ->
                handle_error(Error)
        end
    end.

I don't need to "aggregate events"; I just tell the system what I want it to do, and it does it, and I don't sit here and specify how to wire functions together. In Erlang, the above will not block any other process. If you don't want it to block your current process, that's easy:

    spawn(fun () -> proxy(SomeSender, SomeDest))

Bam. Separate process and the current process can move on with life. No hooking up events. No code blathering on about how to interleave the events in that process with the events in this process. It's just happening. (There is standard library code to make things even more reliable, but going into the built-in supervisor stuff would take too much time. Also, it's hitting below the belt, no other language has anything quite like OTP.)

Erlang doesn't actually use coroutines, Haskell does only upon request, coroutines for concurrency are just cooperative multitasking and I mock them as well, albeit for different reason.

You need to try Erlang. If only to know how to argue against it without arguing against some fictional language that doesn't exist.

[c00p3r]

Good points, thank you!

The problem is that they do not even understand that it is ridiculous to compare someone's hobby-project (actually a bunch of hacks - just read the source) and well-designed (all papers are available) battle-tested and widely used in telecoms (not in browsers) solution. ^_^

So, you're right - "It is Javascript". Same as for Clojure "It is JVM!"

[_urga]

You may be surprised. Hobby projects have a knack for turning out to be more interesting than not.

[c00p3r]

I know. Look at nginx.

It is actually not just a hobby, it was sponsored and actively used by it's author's employer - Rambler.

[spc476]

Really? I recently wrote a framework to do networking with Lua. The network code itself is event based, but each TCP connection is handled by a Lua coroutine which makes it easy to write straightforward code such as:

  function main(socket)
   io.stdout:write("connection from " .. tostring(socket))
   while(true)
     local cmd = string.upper(socket:read())
     if cmd == "SHOW" then
       socket:write("show me some stuff\n")
     elseif cmd == "PING" then
       for i = 1 , 15 do
         socket:write(".")
         socket:sleep(1)
       end
       socket:write("\n")
     elseif cmd == "QUIT" then
       socket:write("Good bye\n");
       return
     elseif cmd == nil then
       return
     end
   end
  end

Makes writing simple servers nice and easy.

[silentbicycle]

Rather than socket:sleep(), you probably want to use socket.select to multiplex IO. (I'm assuming you're using LuaSocket, though that's not entirely clear.)

As with select(2) in general, this doesn't scale up past 100ish idle sockets - it has to do a full scan over all sockets to check which are ready for IO, and the latency eventually dominates. (Not a big deal for most uses, but problematic for web applications.) If that's not an issue, though, it's quite easy. Lua is very underrated, IMHO.

I'm working on a Lua library (octafish) for doing libev + coroutine and/or callback-based servers. It's been on the back burner for a bit, though - a couple other projects have been crowding it out. I'll put it on github once it's further along.

[spc476]

I'm not using LuaSocket but my own homegrown code based off epoll() (it was a learning experience in embedding Lua). It could probably be adapted to libev if I had the interest in doing so.

[silentbicycle]

Ok. What you wrote would adapt to select and nonblocking sockets in LuaSocket pretty easily, FWIW. Using epoll / kqueue (directly or via libev/libevent) scales better, but you're blocking on the sleep, so it wouldn't matter.