[brlewis]

PG is using a really cool programming technique that I'm afraid is ahead of its time relative to current hardware. An upgrade to the HN server should allay the problem.

To see the potential, look at this code snippet from an academic paper on the topic. The web server presents a form asking for a number, then presents a form asking for another number, then displays their product. This technique makes event-driven web applications feel (to the programmer) like sequential imperative programs.

  ;; main body
  ‘(html (head (title ”Product”))
         (body
           (p ”The product is: ”
              ,(number→string (∗ (get-number ”first”) (get-number ”second”))))))

The paper: http://cs.brown.edu/~sk/Publications/Papers/Published/khmgpf...

[pg]

It's not so much that it's ahead of its time relative to hardware as it is something you do in the early versions of a program.

Using closures to store state on the server is a rapid prototyping technique, like using lists as data structures. It's elegant but inefficient. In the initial version of HN I used closures for practically all links. As traffic has increased over the years, I've gradually replaced them with hard-coded urls.

Lately traffic has grown rapidly (it usually does in the fall) and I've been working on other things (mostly banning crawlers that don't respect robots.txt), so the rate of expired links has become more conspicuous. I'll add a few more hard-coded urls and that will get it down again.

[biot]

Over the last week the home page appears to be cached longer than the arc timeout, no doubt due to the spike in traffic. As I throw away cookies when closing the browser, I need to login daily. It's been impossible to login from the HN home page because of this. Refreshing the page doesn't help; I've had to click through to a story to be able to login.

You should hard-code that one too.

[pg]

The problem there is that we switched to a new deliberately slow hashing function for passwords.

Edit: I investigated further, and actually you're right, the problem was due to caching. It should be better now because we're not caching for as long. But I will work on making login links not use closures.

[tptacek]

What'd you go with, and how much of a pain was it to get working in Arc?

I ask because I'd love to be able to make a claim like "even Hacker News, which is written in a Lisp, managed to implement a modern password hash".

[pg]

We use bcrypt. Rtm did it. I never looked at the code till now; it's about a page of Scheme.

[tptacek]

Thanks!

[icey]

Gauche Scheme has a bcrypt implementation, but I don't know what the compatibility story is between mzscheme and Gauche. I think they're both R5RS compliant, so it should work.

I see that newer versions of Arc run on Racket, but I have no idea if that's what HN is using or not.

I haven't seen a scheme powered PBKDF2 implementation so I'd guess that's out.

The only other expensive KDF I can think of is scrypt, but I would be pretty surprised if that's got a scheme implementation.

Of course, I guess pg could have decided to call out to the OS to run any of those functions too.

[wnight]

Is that specifically to inconvenience someone who would break in, steal your password list, and crack it offline?

If not, what was the design goal?

If slowing down web login attempts isn't part of it, why not get a dedicated auth server and offload the crypt stuff onto it?

And if it is the goal, you could use CPU-friendly sleeps on the front-end to give increasing delays to the repeated guesser.

[technomancy]

> Is that specifically to inconvenience someone who would break in, steal your password list, and crack it offline?

Probably: http://codahale.com/how-to-safely-store-a-password/

Hashing functions designed for speed are absolutely the wrong thing for passwords.

[wnight]

Yeah, that's what I'd have thought.

But I don't see the need to do the processing on the web servers.

[brlewis]

Out of curiosity, are there any places where the hn codebase would be smaller if you used full continuations instead of just closures, allowing code akin to what I quoted from the PLT paper?

[pg]

Possibly, but not many. It was surprising (in this application at least) how rarely I needed full continuations.