[branchly2]

> The super advantage of Lisp (including Scheme): Its format for defining data is the same as for writing code, making macros a natural part of the syntax: You can change your source code just like you’d change any other list (or rather tree) data type.

I've heard this repeatedly over the years, but the explanation unfortunately always stops right there. Could you please give an example of why you'd want to change your source code programatically? It's always assumed that the reader implicitly knows why this is a good and important thing. Perhaps a practical example showing

(A) how this would work,

(B) what the benefit is, and

(C) how the added work in reasoning is worth that benefit.

Note: I personally like the parentheses, as they group everything together so simply.

[WalterGR]

Could you please give an example of why you'd want to change your source code programatically?

This is my favorite example.

http://www.gigamonkeys.com/book/macros-standard-control-cons...

[Common Lisp's] DOLIST is similar to Perl's foreach or Python's for. Java added a similar kind of loop construct with the "enhanced" for loop in Java 1.5, as part of JSR-201.

Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern.

A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process--according to Sun--takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably still can't use the new feature until they're allowed to break source compatibility with older versions of Java.

So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.

[davexunit]

>Could you please give an example of why you'd want to change your source code programatically?

To create new syntax and be your own language designer. This is what macro systems allow for. Here's a very contrived and simple example. Guile comes with no equivalent of the `++` operator that we know from C, C++, etc. So in the event that we have some imperative code that is mutating a counter, we'd have to write something like this:

    (define counter 0)
    ;; do some stuff...
    (set! counter (+ counter 1))

Quite a lot more typing! It would be especially annoying if we had many such counters. Normally, when we want to factorize code, we'd write a function, but there's a problem: We can't write a function that mutates any given variable. So, this wouldn't work:

    (define (++ n) (set! n (+ n 1)))
    (define counter 0)
    (++ counter)

This is just mutating a local variable `n`, nothing happens to the variable `counter`, it's still 0. So what do we do? Instead of a function abstraction, we'll use a syntactic abstraction instead. Here is a macro that does what we want:

    (define-syntax-rule (++ var) (set! var (+ var 1)))
    (define counter-a 0)
    (define counter-b 0)
    (++ counter-a)
    (++ counter-b)
    (++ counter-a)

Now `counter-a` is 2 and `counter-b` is 1. The `++` macro is a program that writes programs. It takes `(++ counter-a)` and expands it into the code `(set! counter-a (+ counter-a 1))`.

The reason these syntactic abstractions are so easy to make is because of the homoiconic Lisp syntax. I hope this has made sense and is helpful. Moving on from this simple example, we can create entirely new languages that are embedded in Scheme if we wanted to, adding things that are too specific to a problem domain to ever be in the standard language's syntax but very useful for the problem we are trying to solve.

[naters]

Is the last line in your failed function example supposed to read:

    (++ counter)

?

[davexunit]

Yup, sorry. Fixed.

[mschaef]

> I've heard this repeatedly over the years, but the explanation unfortunately always stops right there.

Lisp macros are something like C macros, chainsaws, hydrochloric acid, or anything else that's powerful-but-dangerous. This is to say that sometimes they are the one tool you have to use, but often they are unnecessary and should be avoided.

Coming up on ten years ago, I wrote this blog post on one of the dangers of Common Lisp style macros: http://www.mschaef.com/blog/tech/lisp/defmacro-coupling.html

Put succinctly, the problem I write about in the blog post is that macros are essentially always inlined into the output of the compiler. This has the effect of more tightly coupling the modules together than is evident in the surface syntax. (Note that macro invocation sites are syntactically indistinguishable from function invocation sites, which makes this problem worse.)

The upshot of this is what you'd expect: the extent of the logic encoded in macros should be minimized, with the macros translating the code pretty much straight away into something built on more functional abstractions.

This is not to say that Macros aren't useful... sometimes you _have_ to use them to achieve a goal. Just that they probably aren't as big a deal as might be expected given the amount of 'press' they get.

[kazinator]

All computer code is a dangerous tool that is often unnecessary and should be avoided if possible. A macro is no more or less dangerous than a function, class, variable, module, or anything else.

C macros have the issue that even when everyone involved in the creation and use of a C macro understands its pifalls, those pitfalls cannot be removed from the macro.

For instance, a certain C macro might evaluate some expression twice. Everyone knows that this is dangerous, but there isn't any way to fix it. They just document it.

ISO C itself says that getc may evaluate its argument multiple times; thus don't do things like getc(stream_array[i++]) unless you remove the macro definition wth #undef.

Lisp macros do not have issues that are unfixable in this way.

Sometimes they have issues that are difficult, though not impossible. Usually that occurs when, to be perfect, the macro would have to do a full-blown code walk. Macros are written that do code walks (for instance the iterate macro).

> Put succinctly, the problem I write about in the blog post is that macros are essentially always inlined into the output of the compiler. This has the effect of more tightly coupling the modules together than is evident in the surface syntax. (Note that macro invocation sites are syntactically indistinguishable from function invocation sites, which makes this problem worse.)

Before you apply macros, you need a well-designed (and documented, and versioned!) API against which the macros will write the code. If all you care about is what the macro syntax looks like and don't put any design into how the expansion works (beyond just massaging it so that it somehow works), then you may run into problems.

Macros don't introduce any problems that writing the same code by hand against the same API's wouldn't introduce.

If someone has to write the code, I don't see how you can get around it: it's either going to be a human, or a macro.

[mschaef]

> a certain C macro might evaluate some expression twice. Everyone knows that this is dangerous...ISO C itself says that getc may evaluate its argument multiple times; ... Lisp macros do not have issues that are unfixable in this way.

If your macros is 'fixed' to emulate function call semantics by evaluating its arguments only once, then maybe a function is a more appropriate abstraction in the first place. The whole point of macros is that they let you break the rules of function call application in hopefully useful and predictable ways.

Another way to look at it is that repeatedly evaluating an argument is what you do NOT want for a macro like 'getc', but probably what you DO want for a macro like 'repeat'. The danger lies in the fact that it's hard to tell the difference when looking at a call site in isolation.

Whether or not that danger is an acceptable risk is, of course, situation dependent.

[kazinator]

By the way, some 18 years ago, I came up with a system for catching the use of expressions with side effects in C macros. Basically, I introduced an API that you could use in your macro definitions to identify insertions of expressions which would cause problems if containing side effects. This API, at run-time, would parse the expressions, analyze them for side effects, and diagnose problems. (It would also cache the results for faster execution of the same macro site.)

All the programmer has to do is achieve run-time coverage to catch all the problems.

We could define a getc-like macro such that getc(*stream++) would diagnose, provided that the line is executed.

See sfx.h and sfx.c here: http://git.savannah.nongnu.org/cgit/kazlib.git/tree/

[kazinator]

There are all kinds of macros that have to evaluate an expression exactly once, and cannot be made into functions.

  ;; cond evaluated exactly once;
  ;; then or else at most once, not before cond.
  (if cond then else)

getc doesn't have to be a macro. It illustrates just the point that the macro issues in C are so unfixable that broken macros have even been codified in ISO C.

[mschaef]

Sure, functions can do that and more (borrowing a page from Smalltalk):

    (if* cond
        #'(lambda () then)
      #'(lambda () else))

All the macro does is eliminate the need to write out all the lambda syntax.

   (defmacro (if cond then else)
      `(if* ,cond
          #'(lambda () ,then)
        #'(lambda () ,else)))

This brings my back to my original point: "the extent of the logic encoded in macros should be minimized, with the macros translating the code pretty much straight away into something built on more functional abstractions.".

Works in C too, although not as nicely:

    void dscwritef_impl(const _TCHAR * format_str, ...);
    
    #define pdscwritef(flag, args) \
         do { if (DEBUG_FLAG(flag)) dscwritef_impl args; } while(0);

The funny thing is, I think we're largely in agreement.

[kazinator]

I do not agree that an if macro stands for some specific lambda-based utterance. That isn't historically true, or in any other sense. The macro potentially stands for any and every possible way in which its semantics can be achieved.

[kazinator]

Macros don't "change source code". That is a serious, but common misconception: that they are somehow self-modifying code.

Lisp macros give meaning to syntax that doesn't previously have meaning. In this regard, they are the same as functions.

(foo (boonly) blarg) doesn't have any meaning because foo hasn't been defined.

We can fix that by writing a function foo. Then (boonly) and blarg have to be valid expressions and we are good.

Or we can make it mean something can by writing a macro foo. The macro foo is a function that will operate on the entire form (foo (boonly) blarg) and calculate a replacement for it. The Lisp form expander (a feature of the compiler or interpreter) will call the macro and accept its return value as the replacement for the macro call. Then, the replacement is scanned for more macros; the entire process removes all macros until all that is left is special operators and functions.

A practical example of a macro is the ANSI Common Lisp standard macro called loop which provides a syntax for iterating variables, stepping through collections, and taking various actions or gathering results:

  (loop for bit in '(0 3 7 13)
        for mask = (ash 1 bit)
        summing mask) -> 8329 

This entire looping mini-language is implemented in the loop macro. When you call (loop ... args) the macro takes over, analyzes the phrases and compiles them into other syntax.

[ArneBab]

The natural script writing syntax¹ is an example: The `Enter` Macro defines macros which use their arguments as data (no need to quote anything), but execute any part prefixed with a comma as actual code.

(Enter (Arne))

(Arne (Hello!) (Who are you?))

The above is actual code which makes Arne say two lines of text.

Essentially the Enter defines a new specialized control structure which is optimized for the task of defining lines of text for a character in an RPG to say.

This removes cognitive overhead while writing a scene: You only write what which character should say.

Essentially you invest into creating a better-fitting tool to simplify all following work. With Scheme you can take this further than with anything else — short of taking over the whole language implementation (which for example Facebook did with their new PHP implementations. With Guile you can take the path Facebook took, without first needing to be a multi-billion-dollar company). But as any other power, use this with care: You won’t want to make your code so much different from what other Schemers do that others have a hard time joining in.

¹: https://fosdem.org/2017/schedule/event/naturalscriptwritingg... — also see the video and the slides which show the difference between this syntax and examples from other methods, including one of my earlier tries with Python.

[jernfrost]

Being able to create code with code, allows you to do away with boilerplate code. E.g. if you got a lot of code with just minor differences, you can make those differences into parameters in a macro which makes each of these code chunks.

That can be utilized both at high level and low level. GOAL is a cool example: https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp. Basically you can express assembly code in LISP syntax and then you can create higher level constructs like if, while, for etc statements as macros composing these lower lever assembly instructions. Because data and code is the same in LISP that allowed Naughty Dog when they used GOAL on the Playstation 2, to swap chunks of code in and out of memory as needed easily to maintain larger levels than the competition. It is also something that allows you to change a running system. I believe there was a case of a malfunctioning satellite running LISP code, which was debugged and fixed while it was running. Shutting it down for fixing was not an option.

I would also add another benefit of only using parenthesis. It makes it really easy to make powerful tools for LISP. Have a look at s-expression based editing. Instead of working line by line, or word by word as we are used to with editors, these work with whole blocks of code at the time. Normal editor commands involve jumping word by word or line by line. with LISP editors you can view the code as a tree with keyboard commands for jumping between tree siblings, up to a parent or down to a child. Instead of selecting x number of lines or words, you can select a whole subtree and delete it, move it, duplicate it or whatever. I think they usually call it paredit. Here is an animated demonstration of how it works: http://danmidwood.com/content/2014/11/21/animated-paredit.ht...

Disclaimer: I never really got used to LISP myself. I think it is cool, but it was too big of a jump for me when I started from C++ background. However I feel more used to it now as I've programmed a lot in Julia which is LISP like but with more normal syntax. Also when I first checked out LISP I didn't understand the need to change the way you think about navigating and editing code. If you edit the way you edit normal code I think it easily gets confusing. You lose track of the parenthesis. I didn't know about stuff like paredit then.

[amirouche]

> why you'd want to change your source code programatically?

Performance, macros are executed at compile time. Also, sometime the macro is easier to write that the equivalent non-macro code.

[didibus]

The syntax is mininal, here it is: (functionOrMacroOrSpecialForm arg1 arg2 ...)

That's the whole syntax. So now you know how to program in LISP.

There's no statements in lisp, only the above expression repeated, which you can nest one in another, or sequence them one after another. They're called S-expressions.

For me, that simplicity is the first advantage. It's just very consistent and quick to learn. It also allows to embed everything together, like assign a value in an if condition.

The second big advantage, is that this syntax is very easily parsed into a list and can be easily changed programatically. Lisp lets you do that by defining macros.

A) It works by defining a function which takes the parsed code as an AST, and returns a modified AST. To transform the AST you have the full power of LISP availaible and some convenient operators that make it easier.

B) There's many benefits. Conceptually, it lets you change the evaluation order of code. Normally a function evaluates its arguments from left to right and then passes the resulting values to be evaluated by the function. 99% of the time this is what you want. But what if you want to short-circuit the evaluation of the arguments when one of them evaluates to say false. This would come in handy if writing the AND function. A macro allows you to do it. In other languages, you're given special operators for this kind of thing, but they're limited and you can not add new ones.

Another practical case is aspect oriented programming. Imagine you want to print the argument to all functions and all their return values. With a macro you can do this:

(print-steps (+ (- 100 23 price) (* 12 34 56 people))).

Print steps is able to inject a print statement at multiple code points, around each arguments and around every functions. Without a macro, you would have had to re-write all this manually, and then delete it again once you're done debugging. Some languages like Java give you a complicated framework which can do this to a certain extent, but aspect-J is a pre-processor, just like macros, it's just that the Java syntax makes this really complicated to implement, so you need a framework to do most of the work, or it's not practical.

C) That's up to you to decide. You don't ever have to write a single macro, and should only use them when they are needed and make sense. Now reading code with macros is trickier, because every macro could be a custom operator you have never seen before, or an aspect that does things you're not sure about. To me, this just becomes a thing of don't write crappy unreadable code. Which is true in all languages. I'd suggest teams standardize their macros, and document them well, so new members can quickly get familiar with your team's macros. Once familiar with a set of macros though, they make you a lot more productive and do make code more concise and even easier the read sometimes.