[hzhou321]

LISP has been singing homoiconicity as its feature. I lately start to think homoiconicity is really a wart. The macros are a system to program the code, while the code is a system to program the application. They are two different cognition tasks and making them nearly indistinguishable is not ideal.

LISP has a full-featured macro system, thus hands down beats many languages that only possess handicapped macro system or no macro system at all. It uses the same/similar language to achieve it is mere accidental. In fact, I think LISP is an under-powered programming language due to its crudeness. But it's unconstrained macro system allows it compensate the programming part to certain degree. As a result, it is not a popular language and it will never be, but it is sufficiently unique and also extremely simple that it will never die.

What if, we have a standalone general-purpose macro system that can be used with any programming languages, with two syntax layer that programmers can put on different hat to work on either? That's essentially how I designed MyDef. MyDef supports two forms of macros. Inline macros are using `$(name:param)` syntax. Block macros are supported using `$call blockmacroname, params`. Both are syntactically simple to grasp and distinct from hosting languages that programmers can put on different hats to comprehend. The basic macros are just text substitution, but both inline macros and block macros can be extended with (currently my choice) Perl to achieve unconstrained goals. The extension layer can access the context before or after, can set up context for code within or outside, thus achieve what lisp can but using Perl. We can extend the macros using Python or any other language as well, but it is a matter of the extent to access the macro system internals.

Inline macros are scoped, and block macros can define context. These are the two features that I find missing in most macros systems that I can't live without today. Here is an example:

    $(set:A=global scope)
    &call open_context
        print $(A)
    print $(A)

    subcode: open_context
        set-up-context
        $(set:A=inside context)
        BLOCK # placeholder for user code
        destroy-context

[packetlost]

> I lately start to think homoiconicity is really a wart.

I sorta agree. The simplicity of the syntax and representation makes it particularly amenable to dynamic modification above basically every other language though. There's a bunch of other properties that make this to case though, such as a very dynamic type system, first-class functions, extremely simple syntax, etc. so it's really a combination of a bunch of factors.

That being said, I think homoiconicity is actually a useful feature, but runtime macro expansion is the dangerous part. What I'd really like to see is a Lisp with very well defined execution orders. That is, macros must be expanded at compile time, and with clear symbols for defining what runs and when. I'm not talking about something like `(macroexpand ...)`, more like `(comptime (my-macro ...))`... `(define (my-func) (my-macro+ 'a 'b 'c))` where it's explicit that a macro executes at compile time, and usage of that macro must be denoted with a special character (`+`, in my example).

I think a general purpose macro language is only as useful as the average code-gen/templating language. It's just string manipulation, which can be harder to reason about than true metaprogramming and might result in some ugly/verbose output and having to context-switch between 2 entirely different languages often. What's particularly powerful about Lisp macros is that usage of them doesn't look any different than a normal application usually, and writing a macro is only marginally different than writing a normal function.

[hzhou321]

> I think a general purpose macro language is only as useful as the average code-gen/templating language. It's just string manipulation, which can be harder to reason about than true metaprogramming ...

I would like you to reconsider. Predicting a program output is hard. So in order to comprehend a macro programed as code, one need run the macro in their head to predict its output, then they need comprehend that output in order to understand the code. I think that is unreasonable expectation. That's why reasonable usage of meta-programming is close to templating where programmer can reason with the generated code directly from the template. For more higher-powered macros, I argue no one will be able to reason with two-layers at the same time. So what happens is for the programmer to put on his macro hat to comprehend the macro, then simply use a good "vacabulary" (macro name) to encode his comprehension. And when he put his application programming hat, he takes the macro by an ambiguous understanding, as a vocabulary, or some one may call it as a syntax extension. Because we need put on two hats at different time, we don't need homoiconicity to make the two hats to look the same.

[packetlost]

Or you do:

``` (require (ast macroexpand))

(display (macroexpand my-macro arg1 arg2)) ``` and call it a day.

My argument isn't that the context-switch isn't there, it's that the context switch will happen regardless and having to think in 2 different languages is more mental overhead than is necessary. I do agree that homoiconicity is not a requirement, but it is nice to be able to do it all in one go and with no additional tooling, editor, etc.. In reality, a sizeable chunk of Lisp programmers are executing their code in a REPL continuously as part of their core development loop, there's no tooling (context) switch, no language context switch, and barely a macro vs application code context switch.

To illustrate, Rust macros are basically that. They have a substantially different syntax to normal Rust code that make it very difficult to quickly grok what is going on. It's a net negative IMO, not a positive.

[hzhou321]

> To illustrate, Rust macros are basically that. They have a substantially different syntax to normal Rust code that make it very difficult to quickly grok what is going on. It's a net negative IMO, not a positive.

Yeah, more like a syntax extension than macro. But I am saying that you need both. Some time you need powerful macro ability to extend the language. Sometime you just need templating to achieve the expressiveness. With LISP, I get it that you are programming all the time, never templating, right? But I guess you only appreciate templating when you use your macro system as a general purpose system . The benefit of general purpose macro systems is you only learn one tool for all languages, rather than re-learn the individual wheels. And when you judge a language, you no longer bothered by its syntactic warts because you can always fix the expressive part with your macro-layer.

[hzhou321]

> That being said, I think homoiconicity is actually a useful feature, but runtime macro expansion is the dangerous part.

Practically, why would you ever want a runtime macro expansion?

[packetlost]

Well, part of the problem is is passing around quoted forms (ie. `(quote ...)` or in may Lisps, `(...)). If that form contains a macro, and a sizeable chunk of Lisps stdlibs are macros, you probably need to implement runtime macro expansion at least partially. So in order to avoid implementing runtime macro expansion, you need to break the semantics of what a quoted form is, or disallow them entirely. The implementation for the former could get really complicated depending on the cases you want to support, resolving symbols in the corresponding scope comes to mind as being particularly challenging. Removing quoted forms entirely just really isn't an option, it's required for one of the most powerful parts of Lisps in general: built-in syntax templates. So we're back to breaking the semantics of quoted forms, which I think can be done reasonably, if not difficult to implement.

[hzhou321]

In another word, it (to have runtime macro expansion) is a side effect, a compromise, a wart, rather than a design goal, right?

[packetlost]

It can certainly be a design goal if you want it to be. Sometimes truly dynamic programming is what you want and need. I, personally, wouldn't want to write any code like that because I mostly write software for other people that runs while they aren't looking, not for myself that runs when I execute it. Lisps are popular in academic settings where the program behavior is the topic of interest and maximum flexibility is a tool to accelerate progress. These types of scenarios usually have the person who wrote the code directly involved in it's execution, as opposed to service development where you want to be more sure it'll actually work before you deploy it.

[JonChesterfield]

Most common case is probably trying to pass `and` to something that applies it to a list, `fold` or `reduce` or similar, and being told some variant of "no deal, and is a macro, not a function" with workarounds like `(reduce (lambda (x y) (and x y)) list))`

[kazinator]

Mainly, because it's always run-time. Your program's compile-time is the compiler's run-time. Compilers can be available in a shipped application.

[kazinator]

Thanks for posting. A bunch of time ago I was actually looking for MyDef for some reason, but I couldn't remember the exact name and couldn't find it.

You could have a replacement for M4 there.

I made a macro preprocessor in 1999 called MPP which also was able to preserve whitespace in perpetrating multi-line expansions, and so suitable even in indentation-sensitive formats. MPP has scopes and also a namespace system. I didn't develop it further because right around that time, I discovered Lisp and, you know ...

[JonChesterfield]

m4 is like that. Lots of templating languages are too. Code generation is useful and roughly what you end up with if the language isn't expressive enough, e.g. C or go. Is yours doing anything different to those?

What the lisp approach gives you is the macro has the parsed representation of the program available to inspect and manipulate. It doesn't have to expand to some fixed text, it can expand into different things based on the context of the call site.

Various languages have decided that syntactically distinguishing macros from functions is a good idea. Maybe the function is foo and a macro would be $foo. For what it's worth I think that's wrong-headed, and the proper fix is to have 'foo' represent some transform on its arguments, where it doesn't matter to the caller whether that is by macro or by function, but it's certainly popular.

[hzhou321]

M4 only do token-level macros or inline macros. M4 macros are identifier that can't be distinguished from underlying language. M4 macros does not have scopes. M4 does not have context level macros. M4 does not have full programmable ability to extend syntax.

I can define any macro lisp can define, just not with LISP syntax. I do not have a full AST view of the code, due to the its generality that it does not marry to any specific underlying language. But I can have extensions that is tailored to specific language and do understand the syntax. For example, the C extension can check existing functions and inject C functions. MyDef always can query and obtain the entire view of the program, and it is up to the effort in writing extensions to which degree we want macro layer to be able to parse. Embedding a AST parser for a local code block is not that difficult.

It's like the innerHTML thing, for me, I always find the text layer (as string) is more intuitive for me to manipulate than an AST tree. If needed, make an ad-hoc parser in Perl is often simple and sufficient, for me at least.