Talk of metaprogramming intrigues me. I'd like to hear what a Lisp user makes of it because I find non-Lisp users are usually amazed by any metaprogramming at all and can't be as critical about it.
Not a Lisp user, but a heavy Nim macro user. Nim macros are AST based, which means that after the parser part of the compiler have read the code and created a tree structure of it your macro is called on this tree. Then you can modify this tree to your hearts desire and return a new tree that the Nim compiler will splice back into the main tree before continuing. This means that you rewrite pretty much anything into pretty much anything else. You also have access to reading files on the machine, or doing pretty much anything else that Nim offers. For examples of what you can do with it you can have a look at my macro to create GUIs: https://peterme.net/cross-platform-guis-and-nim-macros.html, my article on how I use macros to create more read- and maintainable code: https://peterme.net/metaprogramming-and-read-and-maintainabi... or perhaps see how Protobuf was written as a string parsing macro which reads a Protobuf specification and spits out all the types and procedures you need to use it: https://github.com/PMunch/protobuf-nim
> which means that after the parser part of the compiler
Lisp works slightly different. The macro form is an expression (a list) with the macro operator as the first element. The next elements in the expression can be anything as data.
Disclaimer: I only did a hobby project in Lisp once. But I did use some of the macro functionality.
Judging by some of the comments here, it seems like the macro system has a similar approach as Lisp's macro system, which is also AST-based. Something I don't see here is macros that generate other macros, but the question is how much you really want that anyway (when I did that, I thought the syntax was horribly complicated because of all the quoting). I know Lisp also has reader macros (they run before the parser) that allow you to effectively change the language syntax, but I didn't use those.
The only restriction is that it is a s-expression: a possibly nested list of data. There is no particular programming language syntax it is parsed against - it's parsed as data.
For example this is a valid Lisp macro form
(loop for i below 10 and j downfrom 20
when (= (+ i j) 9) sum i into isum of-type integer
finally (return (+ j isum)))
It returns 10.
The LOOP macro parses it on its own - it just sees a list of data. Lisp has other than that no idea what the tokens mean and what syntax the LOOP macro accepts.
CL-USER 142 > (defmacro my-macro (&rest stuff)
t)
MY-MACRO
CL-USER 143 > (my-macro we are going to Europe and have a good time)
T
The data has an AST. The AST is made of conses, and atoms. The atoms have a large variety of types. They are not "tokens", but objects. Tokens only exist briefly inside the Lisp reader. 123 and |123| are tokens; one converts to an integer node, one to a symbol node. (1 (2 3)) has an AST which is
Metaprogramming is one of the core ideals in the birth of the language so is well supported. Personally I've not used Lisp, so can't comment there, but it is on the list of influences on the homepage.
Essentially there's a VM that runs almost all the language barring importc type stuff, and you can chuck around AST node objects to create code, so metaprogramming is done in the core Nim language. You can read files so it's easy to slurp files and use them to generate code or other data processing at compile-time.
Several simple utility operations in the stdlib make things really fluid; the easy ability to `quote` blocks of code to AST, and outputting nodes and code to string. This lets you both hack something together quickly and learn over time how the syntax trees work.
Quoting looks like this:
macro repeat(count: int, code: untyped): untyped =
quote do:
for i in 0..<`count`:
`code`
repeat(10):
echo "Hello"
Inspecting something's AST can be done with dumpTree:
dumpTree:
let
x = 1
y = 2
echo "Hello ", x + y
To save even more effort, there's also dumpAstGen which outputs the code to create that node manually, and even a dumpLisp!
You can display ASTs inside macros:
macro showMe(input: untyped): untyped =
echo "Input as written:", input.repr
echo "Input as AST tree:", input.treerepr
result = quote: `input` + `input`
echo result.repr
echo result.treerepr
So it's really easy to debug what went wrong if you're generating lots of code.
Since untyped parameters to a macro don't have to be valid Nim code (though they still follow syntax rules) you can make your own DSLs really easily and reliably because any input is pre-parsed into a nice tree for you.
Here's a contrived example of some simple DSL that lets you call procs and store their results for later output:
import macros, tables
macro process(items: untyped): untyped =
result = newStmtList()
# Create hash table of 'perform' names to store their result variables.
var performers: Table[string, NimNode]
for item in items:
let
command = item[0]
param = item[1]
paramStr = $param
case $command
of "perform":
# Check if we've already generated a var for holding the return value.
var node = performers.getOrDefault(paramStr)
if node == nil:
# Generate a variable name to store the performer result in.
# genSym guarantees a unique name.
node = genSym(nskVar, paramStr)
performers.add(paramStr, node)
# Add the variable declaration
result.add(quote do:
var `node` = `param`()
)
else:
# A repeat performance, we don't need to declare the variable and can overwrite the
# value in the fetched variable.
result.add(quote do:
`node` = `param`()
)
of "output":
let node = performers.getOrDefault(paramStr)
if node == nil: quit "Cannot find performer " & paramStr
result.add(quote do:
echo `node`)
else: discard
# Display the resultant code.
echo result.repr
proc foo: string = "foo!"
proc bar: string = "bar!"
process:
perform foo
perform bar
perform foo
output foo
output bar
The generated output from process looks like:
var foo262819 = foo()
var bar262821 = bar()
foo262819 = foo()
echo foo262819
echo bar262821