Well iterate[0], which is a lispy version of loop, is actually extendable through macros[1]. So what you are looking for is just a generic way to enable that for all macros? Iterate does it by having a code walker go over the code and macroexpand the extendable parts. It shouldn't be too hard to apply that method to new DSLs by specifying the syntax of the DSL to the code walker.
Yup, I pretty much want it to be trivially easy for define macros that are themselves macro-extensible. It's definitely possible to do this in Lisp, but:
(a) it's not a well-known technique
(b) it's not in the standard library of any Lisp I know of
(c) there are interesting open design questions to be answered in the implementation of such a system
I hope this article will get folks thinking about these issues.
For example, walking the AST and calling macroexpand will work, but then you can't have a macro that expands differently in different contexts - when interpreted as an expression versus as a pattern, for example. I think this is an important feature.
The macroexpansion hook isn't nearly as powerful or useful as it could be because the process of macroexpansion is not recursive. Although expansion functions can access information about the lexical environment via an &environment parameter, the invocations of those expansion functions do not share any dynamic context with other invocations at nodes above or below them in the syntax tree.
A macroexpansion facility works more like a framework than a library. Each macro you define is basically a callback function triggered when the compiler sees a form with a particular symbol at the head. You want to be able to describe what should happen in particular situations you're interested in, without having to specify anything about other situations. A code walker spoils that separation of responsibilities because it has to reimplement large parts of a macroexpansion framework just to make it possible to add a few new features which are then used in conjunction with the original framework.
The potential for bug contagion is quite high, because any single bug's effects aren't localized to the few forms that make use of the extended functionality you've rigged up. This is tantamount to introducing a bug in the language implementation. If you're expanding something like (my-macro (... (my-other-macro))), a bug in your code walker can break the intermediate code. This might be acceptable if you're just using it to write a couple of one-off macros that don't trigger any weird corner cases and are only used in the same project in which they're defined. But if you're creating a general-purpose DSL like iterate that you intend for people to use in all sorts of situations you never anticipated, it's not. In fact, the same bug is more dangerous as part of a code walker than it would be as part of a Lisp implementation's internals. If it's in the implementation, at least it would show up consistently whether the vulnerable code is wrapped in your macro or not, making easier to identify, document, and manage the problem. And probably easier to get somebody to fix, because the implementation ought to be better maintained than the walker.
These are not merely hypothetical concerns. Some projects already rely on macroexpand-dammit working in the quirky way it does; Masataro tried to have his improved version supersede the existing version on Quicklisp, but it broke some packages that were using it: https://github.com/guicho271828/macroexpand-dammit/pull/4#is...
Sometimes I use those packages too, so instead of using his version with the fix I submitted, it's actually less of a headache for me to apply my fix using a wrapper package on top of the Quicklisp version, duplicating some of the code which itself duplicates what Lisp already does. The original duplication problem led to bugs which led to the need to load two separate implementations which led to a name clashing problem which justified additional duplication.
Also, I tried to implement that same iter example using a similar trick before, and I see that your code generates the same mysterious output, at least on SBCL:
; compilation unit finished
; caught 1 fatal ERROR condition
A fatal error during compilation, yet it clearly compiled and ran. What was the error? I don't know, it looks like something in the code walker muffled the details. Does it matter? Could it matter in any other scenario? Something about macroexpand-dammit doesn't play nice with the implementation, but we don't know what, why, or how, and it makes it more difficult to use the very language features that would normally help us find out.
I think a lot of critiques of the "bipolar Lisp programmer" [http://www.lambdassociates.org/blog/bipolar.htm] are overblown, but in the case of these kinds of tools, they're justified. These kinds of techniques will always be restricted to obscure one-off software projects unless they're supported by core language features like what rntz and Masataro are proposing. The designs of macro systems are optimized for making it easy to make small tweaks to the language to fit the particular problem at hand. They're not quite as flexible as something like the MOP. If you try to use a facility designed for making small, local interventions in the language to write very general language extensions which themselves make broad, global interventions in the underlying system for making small, local interventions, you're gonna have a bad time.
(a) it's not a well-known technique
(b) it's not in the standard library of any Lisp I know of
(c) there are interesting open design questions to be answered in the implementation of such a system
I hope this article will get folks thinking about these issues.
For example, walking the AST and calling macroexpand will work, but then you can't have a macro that expands differently in different contexts - when interpreted as an expression versus as a pattern, for example. I think this is an important feature.