[jules]

That doesn't really have much to do with the expired link issue. That issue exists because the targets of links are stored as closures on the server. What you want is to serialize those closures into the URL itself, instead of letting the URL be a pointer to a closure stored on the server. In traditional web applications, you'd have to do that serialization manually (although if you've never used such a closure based web framework you might not even be aware that you're doing this, just like a C programmer who has never used a high level language might not consciously realize that he is implementing objects or closures or garbage collection manually -- or like an old Fortran programmer who is not aware that he is implementing recursion manually). The problem used to be much worse, but then PG did this transformation manually for the important subset of the site (most importantly for links to the comments section -- these links no longer expire).

Here's an example. Currently if you go to the home page and then click the "More" link at the bottom to go to the next page, and wait long enough, the more link expires. That's because it's currently implemented as something like this (not sure about ARC syntax, I'll use Scheme syntax here, but you get the idea):

    (define (show-list-of-posts page-number)
      ... display the rest of the homepage ...
      (link "More" (lambda () (show-list-of-posts (+ 1 page-number)))))

This stores the closure `(lambda () (show-list-of-posts (+ 1 page-number)))` with the free variable `page-number` in a hash table on the server. The URL becomes an index into that hash table. But the only information needed to reconstruct such a closure, is the function body and the free variables. So if we defined an auxilary function:

    (define (foo page-number)
      (show-list-of-posts (+ 1 page-number)))

then we could represent the closure as the pair ("foo",page-number). If we encode that in the URL, then instead of looking up the closure from the hash table, we can reconstruct the closure on the fly. Hence we do no longer have to store anything on the server, and no links can expire anymore.

There are some challenges when you want to do this transformation automatically, but they can be overcome.

[jgrahamc]

"The problem used to be much worse, but then PG did this transformation manually for the important subset of the site (most importantly for links to the comments section -- these links no longer expire)."

I don't think this is strictly true. The comments and reply links simply link to a normal (non-closure) URL on the site and the page is generated from that URL in the normal manner.

[jules]

I don't understand the disagreement...isn't that the same thing I said? IIRC it used to be the case that all links could expire, then he did this transformation from closures stored on the server to closures manually serialized in the URLs (which results in a normal URL as you say). What I'm describing is how a web framework could do that automatically, which means you get the behavior and expressiveness of closures without the link expired issue. Though I can see that the explanation isn't very clear, but I don't know how to make it more clear. Edit: don't downvote him people, it is a perfectly fine comment.

[irahul]

> then he did this transformation from closures stored on the server to closures manually serialized in the URLs (which results in a normal URL as you say).

The reply links have a parameter 'id' - the id of the comment to which the reply is being posted. I would guess it's just passed to a normal function which adds the reply to the post.

May be you mean the same thing, but "serializing closures in the url"(where? all I see is the id of the post being replied to) isn't the same as params in the url which are then passed to a function.

[jules]

Yes, my point is that it is the same thing. What's happening is just manual closure conversion (http://en.wikipedia.org/wiki/Lambda_lifting), except the resulting data gets saved in the URL rather than the server. On the server, closures saved in the hash table, also have "params which are then passed to a function", except that the params (i.e. the free variables) are saved on the server inside the closure rather than in the URL.

If you have a lambda expression like `(lambda () (show-list-of-posts (+ 1 page-number)))`, the Scheme implementation does these things:

1. Introduce a global function definition with the same body as the lambda expression but with an extra parameter for the free variables. The free variables in the body of the function get replaced with expressions that extract their value from the extra parameter that has the free variables.

2. Convert the lambda expression to an expression that builds a pair of a function pointer to that global function, plus the values of the free variables.

For example the code:

    (define (show-list-of-posts page-number)
       ... display the rest of the homepage ...
       (link "More" (lambda () (show-list-of-posts (+ 1 page-number))))

Will get converted to something like this:

    ;; this is the extra global function that has the body of the lambda expression
    ;; note that the reference to `page-number` got 
    ;; replaced by `(extract-value "page-number" params)`
    (define (closure-324 params)
       (show-list-of-posts (+ 1 (extract-value "page-number" params))))

    ;; note that the lambda expression gets replaced by a create-closure expression
    (define (show-list-of-posts page-number)
       ... display the rest of the homepage ...
       (link "More" (create-closure closure-324 "page-number" page-number)))

create-closure creates a closure data structure where the first argument is the function pointer, and the rest of the arguments are the free variables.

Now, what happens if you want to remove this closure business in a web application, and instead use normal URLs?

First, you introduce a global request handler for the body of the lambda:

    (define-handler (post-list-handler params)
       (show-list-of-posts (+ 1 (extract-value "page-number" params))))

This would define the handler for news.ycombinator.com/post-list-handler?page-number=12.

Then, instead of the (link ...) with a closure, you just link to that url in the show-list-of-posts function:

    (define (show-list-of-posts page-number)
       ... display the rest of the homepage ...
       (link "More" (create-url post-list-handler "page-number" page-number)))

Compare these code snippets to the one above. Do you see the similarity to closure conversion? In both cases we:

1. Introduce a global function/handler for the body of the lambda.

2. That function/handler gets a `params` argument that has the free variables.

3. Everywhere a free variable is referenced in the body, it gets replaced by an expression that extracts the value from the params argument.

4. In place of the lambda expression, we have respectively a (create-closure func free-vars...) or a (create-url handler free-vars...)

So it's really completely analogous. That's why I say that we are just serializing the closure here, and this could be done automatically. Hopefully this makes it more clear what I mean, but maybe these details just make it less clear if you're not familiar with how closures are implemented (closure conversion)...

[gknoy]

What I would love would be if the "More ..." link at the bottom of any given page would link to Page N+1 of whatever the current ordering of pages is, rather than page N+1 of a listing of articles that is now out of date and gives me an error message.

Great explanation of what's causing that - thank you!