[simonhf]

silentbicycle, thanks, I updated the name as suggested.

So do you have any ideas for a more interesting benchmark program? Ideally it should be something which needs to keep state as you suggest and makes use of some kind of simple business logic, and is only a few hundred lines of code at the very most. What about a simple chat server which handles people and rooms? The benchmark might handle, say, max. 10k people chatting in max. 1k rooms. What do you think? Any better ideas?

[silentbicycle]

I think a server that takes a login and password, sends the pair to another core (or server) to do a bcrypt (http://www.openbsd.org/cgi-bin/man.cgi?query=bcrypt) check, and then responds with a pass/fail would be a good benchmark.

An async web stack benchmark server should juggle loads of concurrent connections while making internal requests to another process or server, and then sending the response (when available). That's representative of a lot of common server tasks, yet can control for the work itself.

I'm suggesting bcrypt in particular because the amount of CPU time per client can be easily controlled. (In a nutshell, bcrypt is a hash function which can be made arbitrarily slow to deter password cracking.) Offhand, I'm not sure which systems have bcrypt besides OpenBSD, though.

FWIW, I have a simple bcrypt wrapper for Lua (http://github.com/silentbicycle/lua-bcrypt).

There's also a Ruby bcrypt wrapper (http://github.com/codahale/bcrypt-ruby) which includes a copy of the bcrypt source from OpenBSD. I feel vaguely uneasy about including that with my Lua wrapper (it really should be its own library, and more widely available!), but may eventually do so.

[simonhf]

silentbicycle, I like your idea. I wonder if it will be considered 'enough' by those who poopooed the "Hello World" benchmark? On a dual core or better box and assuming the bcrypt server replies immediately, I would predict that performance of the main server would halve. Why? Because the main server can only handle n events per second and by communicating with the bcrypt server we're doubling the number of read events which must happen within the same wallclock time. What I've found is that handling e.g. epoll events is slow compared to e.g. regular function calls or other IPC mechanisms on the same box. So the fastest way to do the bcrypt idea on a production server would be to use non-socket-based-IPC (e.g. shared-memory-based which is an order of magnitude faster than socket-based-IPC). Anyway, assuming we go with socket-based-IPC for the purposes of the comparison, what's the fastest way for the node.js program to maintain state and handle a pool of connections to the bcrypt server? It would be great if a node.js guru would take this on... and then I would take on the SXE and SXELua implementations. What do you think...?

[silentbicycle]

bcrypt doesn't reply immediately. You could set it to require ~10ms of CPU grinding each time it checks whether a password is valid, for example. Having latency while work happens on another core or server shouldn't impact the total number of concurrent connections the server can handle, though - the benchmark should measure that, specifically.

Oh, also - the reads and writes to the bcrypt server will be really short, and could use persistent connections. Something like "$2a$07$8BzFlUuprZN4FSBpDx3ZAuPWOu4CZ3uv8Awa4EjAZhNmnIY59nh2e" -> "OK". And yes, it would increase the number of events, but a real server is probably going to be communicating with a database, memcached, etc. instead.

FWIW, this comment thread is getting buried deeper and deeper in my threads page - if you want to keep the conversation going, my email address is in my profile. Have a good weekend.