[sedachv]

"The first stage is implemented by the 'reader' and can be reprogrammed by an ancient API to the reader via read tables."

Readtables aren't any more ancient than the rest of ANSI INCITS 226-1994 (R2004) (the language standard formerly known as X3.226-1994 (R1999)), but the interface to them is very low-level and non-modular.

The current readtable is specified by a dynamic variable, so the readtable facility can be made modular with a nicer interface, in a portable manner. This is exactly what the library Named-Readtables does: http://common-lisp.net/project/named-readtables/

Now realize the significance of this: Common Lisp is the only language allowing total unrestricted syntactic extension and <i>modification</i> in a modular and (somewhat) composable way.

I've been using named-readtables for the past month, and between it and Quicklisp, I haven't been this excited about programming in CL since I started (which is 8 years ago, not that long, but I'm not a total noob either).

[lispm]

Read tables existed before CL. CLOS for example not, it was developed for ANSI CL (based on experience with LOOPS and Flavors). See for example READTABLE in Maclisp:

http://maclisp.info/pitmanual/io.html#16.2.7

Also note that I wrote that the API is ancient. It is. It is old and could be easier to use.

'Named readtables' are related to 'syntaxes' on the Lisp Machine. For example source files have a syntax attribute in the header, which switches between the various Lisp dialects (or other languages), including using different readers. This is for example used by the file compiler and Zmacs.

[sedachv]

Cool, I didn't know readtables were in Maclisp, or about LM syntaxes.

"It is old and could be easier to use."

Aside from something like named-readtables, how would you design the lowest-level interface to readtables? Or you wouldn't do that, and just specify something like named-readtables to be the interface? I'm curious because this could be something for http://www.cliki.net/Proposed%20Extensions%20To%20ANSI

[lispm]

I haven't thought about it much, but the character level interface is very primitive. Second, what about things like symbols, numbers, etc.? There is no sane way to specify number syntax or symbol syntax. That might be useful. Currently the reader provides an interface on a character level, but not on the level of s-expressions components.

[sedachv]

What I think you're saying is that the reader should be customizable in terms of some DSL for a grammar. It would be nice, but I'm not sure how composable it would be (in the general case, I think it would come down to having the behavior of READ dependent upon a black-box current-parser procedure).

The nice thing about readtables is that it exposes what in essence are transition hooks for each character, so you really don't need to care about the grammar of stuff you're not interested in parsing.

OTOH like you said, extending the syntax for numbers or symbols becomes quite hairy. But even with a DSL grammar approach, you'd need to change major parts of the grammar (which means copying and modifying the normative grammar of CL syntax - not that different from grabbing a portable CL reader implementation today (http://twitter.com/vsedach/status/26484049015)).

The big downside is deciding on which class of grammars that will support, and how they will be represented.