[lispm]

Lisp has a lot syntax. It is just a bit different and it looks different externally.

Lisp has a 2-stage syntax.

The first stage is the syntax of s-expressions, which is surprisingly complex. S-Expressions provide a textual syntax for data: symbols, lists, pairs, strings, various number formats, arrays, characters, pathnames, ...

The first stage is implemented by the 'reader' and can be reprogrammed by an ancient API to the reader via read tables.

The second stage is the syntax of the Lisp programming language. This is defined on top of s-expressions and is really a syntax over data structures (not text). This Lisp syntax deals with: data items, function calls, special forms (thirty something) and macro forms.

This syntax stage is implemented as part of the interpreter/compiler (EVAL, COMPILE, COMPILE-FILE) and can be extended by writing macros, symbol macros and compiler macros. In earlier dialects it could also be extended by writing so-called FEXPRs, functions which get called with unevaluated source code (-> data in Lisp).

So, we get a lot of complex syntax due to special forms and macros. It just looks a bit different, since the data syntax is always underneath it (unless one uses a different reader).

For example a function definition would be:

   (defun foo (a b) (+ (sin a) (sin b)))

The syntax for that is:

    defun function-name lambda-list [[declaration* | documentation]] form*

With more complex syntax for 'function-name', 'lambda-list' and 'declaration'.

Lambda-list has this syntax:

    lambda-list::= (var* 
                    [&optional {var | (var [init-form [supplied-p-parameter]])}*] 
                    [&rest var] 
                    [&key {var | ({var | (keyword-name var)}
                      [init-form [supplied-p-parameter]])}*
                      [&allow-other-keys]] 
                    [&aux {var | (var [init-form])}*])

Not every valid Lisp program has an external representation as an s-expression - because it can be constructed internally and can contain objects which can't be read back.

Not every s-expression is a valid Lisp program. Actually most s-expressions are not valid Lisp programs.

For example

   (defun foo bar)

is not a valid Lisp program. It violates the syntax above.

[sedachv]

"The first stage is implemented by the 'reader' and can be reprogrammed by an ancient API to the reader via read tables."

Readtables aren't any more ancient than the rest of ANSI INCITS 226-1994 (R2004) (the language standard formerly known as X3.226-1994 (R1999)), but the interface to them is very low-level and non-modular.

The current readtable is specified by a dynamic variable, so the readtable facility can be made modular with a nicer interface, in a portable manner. This is exactly what the library Named-Readtables does: http://common-lisp.net/project/named-readtables/

Now realize the significance of this: Common Lisp is the only language allowing total unrestricted syntactic extension and <i>modification</i> in a modular and (somewhat) composable way.

I've been using named-readtables for the past month, and between it and Quicklisp, I haven't been this excited about programming in CL since I started (which is 8 years ago, not that long, but I'm not a total noob either).

[lispm]

Read tables existed before CL. CLOS for example not, it was developed for ANSI CL (based on experience with LOOPS and Flavors). See for example READTABLE in Maclisp:

http://maclisp.info/pitmanual/io.html#16.2.7

Also note that I wrote that the API is ancient. It is. It is old and could be easier to use.

'Named readtables' are related to 'syntaxes' on the Lisp Machine. For example source files have a syntax attribute in the header, which switches between the various Lisp dialects (or other languages), including using different readers. This is for example used by the file compiler and Zmacs.

[sedachv]

Cool, I didn't know readtables were in Maclisp, or about LM syntaxes.

"It is old and could be easier to use."

Aside from something like named-readtables, how would you design the lowest-level interface to readtables? Or you wouldn't do that, and just specify something like named-readtables to be the interface? I'm curious because this could be something for http://www.cliki.net/Proposed%20Extensions%20To%20ANSI

[lispm]

I haven't thought about it much, but the character level interface is very primitive. Second, what about things like symbols, numbers, etc.? There is no sane way to specify number syntax or symbol syntax. That might be useful. Currently the reader provides an interface on a character level, but not on the level of s-expressions components.

[sedachv]

What I think you're saying is that the reader should be customizable in terms of some DSL for a grammar. It would be nice, but I'm not sure how composable it would be (in the general case, I think it would come down to having the behavior of READ dependent upon a black-box current-parser procedure).

The nice thing about readtables is that it exposes what in essence are transition hooks for each character, so you really don't need to care about the grammar of stuff you're not interested in parsing.

OTOH like you said, extending the syntax for numbers or symbols becomes quite hairy. But even with a DSL grammar approach, you'd need to change major parts of the grammar (which means copying and modifying the normative grammar of CL syntax - not that different from grabbing a portable CL reader implementation today (http://twitter.com/vsedach/status/26484049015)).

The big downside is deciding on which class of grammars that will support, and how they will be represented.

[ntoshev]

You are right of course, but this kind of syntax doesn't help the programmer to cognitively parse the code, which is what I am talking about.

[lispm]

No, why not?

I can parse Lisp quite good.