[teddyh]

> Smalltalk isn't file based, you don't edit some version of the code that gets compiled (and macro expanded) […] Macros simply wouldn't fit into Smalltalk in any meaningful way […]

You speak as if “macros” means “macros as implemented in the C preprocessor”. Lisp macros, as I understand it, operate on the parsed syntax tree, not on the file text level, and are expanded at runtime, not compile time.

[aaron-lebo]

The key to understanding macros is to be quite clear about the distinction between the code that generates code (macros) and the code that eventually makes up the program (everything else). When you write macros, you're writing programs that will be used by the compiler to generate the code that will then be compiled. Only after all the macros have been fully expanded and the resulting code compiled can the program actually be run. The time when macros run is called macro expansion time; this is distinct from runtime, when regular code, including the code generated by macros, runs.

http://www.gigamonkeys.com/book/macros-defining-your-own.htm...

[teddyh]

Is it not the fact that macros can be defined and, more importantly, redefined at runtime? If so, would that not mean that macros must also be expanded at runtime?

[lispm]

If you have compiled code, you have to recompile the code when you change a macro. It's not done automagically.

If you have interpreted Lisp code, then the code can use the new macro automagically.

[lispm]

This is the view of a compiled execution. Lisp can also be interpreted from source as data. Then macro-expansion time is interleaved with execution time. Each use of a macro form may trigger a macro expansion.

Remember, a Lisp interpreter interprets Lisp source as data. Not byte code. Unlike Python, Java, Smalltalk, ... which all have popular implementations which compile to byte code and execute that byte code in a byte code interpreter, aka virtual machine.

Let's use a Lisp interpreter, here from LispWorks:

We define a primitive MY-IF macro. It expands into a simple IF use. But the macro will also count the number of macro expansions.

    CL-USER 46 > (defparameter *myif-counter* 0)
    *MYIF-COUNTER*

    CL-USER 47 > (defmacro my-if (c a b)
                   (incf *myif-counter*)
                   `(if ,c ,a ,b))
    MY-IF

LispWorks can trace macros, too.

    CL-USER 48 > (trace my-if)
    (MY-IF)

Now a simple function which uses our macro:

    CL-USER 49 > (defun fac (n)
                   (my-if (= 1 n)
                          1
                          (* n (fac (1- n)))))
    FAC

Now we use it and we will see the trace information for the macro use: you see the incoming form and the result form.

    CL-USER 50 > (fac 2)
    0 MY-IF > ...
      >> COMPILER::FORM        : (MY-IF (= 1 N) 1 (* N (FAC (1- N))))
      >> COMPILER::ENVIRONMENT : #<Augmented Environment venv NIL fenv ((#:FUNCTOR-MARKER . #<COMPILER::FLET-INFO (# # # #)>)) benv NIL tenv NIL decl NIL>
    0 MY-IF < ...
      << VALUE-0 : (IF (= 1 N) 1 (* N (FAC (1- N))))
    0 MY-IF > ...
      >> COMPILER::FORM        : (MY-IF (= 1 N) 1 (* N (FAC (1- N))))
      >> COMPILER::ENVIRONMENT : #<Augmented Environment venv (#<Venv 275415194600  N>) fenv ((#:SOURCE-LEVEL-ENVIRONMENT-MARKER . #<COMPILER::FLET-INFO (NIL . #)>) (#:FUNCTOR-MARKER . #<COMPILER::FLET-INFO (# # # #)>)) benv NIL tenv NIL decl NIL>
    0 MY-IF < ...
      << VALUE-0 : (IF (= 1 N) 1 (* N (FAC (1- N))))
    0 MY-IF > ...
      >> COMPILER::FORM        : (MY-IF (= 1 N) 1 (* N (FAC (1- N))))
      >> COMPILER::ENVIRONMENT : #<Augmented Environment venv NIL fenv ((#:FUNCTOR-MARKER . #<COMPILER::FLET-INFO (# # # #)>)) benv NIL tenv NIL decl NIL>
    0 MY-IF < ...
      << VALUE-0 : (IF (= 1 N) 1 (* N (FAC (1- N))))
    0 MY-IF > ...
      >> COMPILER::FORM        : (MY-IF (= 1 N) 1 (* N (FAC (1- N))))
      >> COMPILER::ENVIRONMENT : #<Augmented Environment venv (#<Venv 275416002360  N>) fenv ((#:SOURCE-LEVEL-ENVIRONMENT-MARKER . #<COMPILER::FLET-INFO (NIL . #)>) (#:FUNCTOR-MARKER . #<COMPILER::FLET-INFO (# # # #)>)) benv NIL tenv NIL decl NIL>
    0 MY-IF < ...
      << VALUE-0 : (IF (= 1 N) 1 (* N (FAC (1- N))))
    2

Let's see how often our macro function has been used to expand code:

    CL-USER 51 > *myif-counter*
    4

[brudgers]

My understanding is that typically, Lisp macros are expanded at 'compile' time. The compiler expands macros recursively until there's nothing left to expand and then compilation proceeds normally.

A Lisp macro operates on data structures. Macros transform one data structure into another data structure. Lisp code is a data structure (typically a list) and ultimately all macros get resolved by expansion to ordinary Lisp code and the expanded code then is compiled/interpreted.

Unlike C, Lisp macros have access to all of Lisp (including other macros) during expansion. So a macro's expansion can be determined programmatically by executing Lisp code...this part is where it gets harder to understand macros intuitively because the code called during the macro expansion phase does not have access to the code that runs at the runtime and vice versa.

That's why it is handy to think about macros as manipulating data structures...we don't expect data structures to return values or generate values in the environment by execution.