[dkersten]

This isn’t specific to Clojure macros, but Clojure is where I’ve felt it.

Consider a macro `(replace-here replacement expression)` that replaces every instance of `here` with `replacement` in `expression`:

    (replace-here 10
      (* here (+ 5 here))

The macro sees `10` as one argument and it sees `(* here (+ 5 here))` as another argument. It can perform the replacement and produce:

    (* 10 (+ 5 10))

But if we have a function `foo` that contains the placeholder:

    (defn foo [] here)

Then our macro won’t be able to see it:

    (replace-here 10
      (foo))

This does NOT replace the “here” inside foo because the macro just sees `(foo)` but is not able to look inside foo.

We can see this I practice in core.async where the `(go expr)` macro will execute `expr` asynchronously and block (await) when it encounters a `(<! channel)` call.

BUT `<!` must be visible to the go macro, because the macro transforms the code into a state machine that can suspend and resume when the call blocks and unblocks. Than means that this code will await the channel:

    (async/go
      (let [val (async/<! ch)]
        (println “got” val)))

However this code will not:

    (defn foo [ch]
       (async/<! ch))

    (async/go
      (let [val (foo ch)]
        (println “got” val)))

Because in this example the `<!` is inside the function `foo`, but the `go` macro cannot look into the function to find and transform it.

Practically, this means you cannot create helper functions that operate on channels, you must always wrap the channel operation in a go block so that the macro will see it.

In practice it’s not that bad, you learn to structure your code into ways that avoid it. But it’s a limitation nonetheless and one that only exists because core.async is implemented as macros.

As an aside, another problem I have with core.async that exists because it’s a macro-based implementation rather than a first class citizen is that there are occasions where the code transformations mean that an error can occur somewhere that isn’t in your source code. I have had a situation where there was an error and the stack trace was entirely in Clojure’s code without referencing my code in any way. Imagine how difficult it is to debug when you don’t even know what source file of yours contains the code with the error! If it was built into the language rather than as macros, then it could at least retain source location contextual data to be passed to whatever internal code raised the error. But that’s a separate issue from the “see inside” issue.

[zelphirkalt]

Maybe this seems like a bit strange design. Couldn't `<!` itself be made a macro, so that it doesn't matter where it is, instead of `<!` being something to transform? EDIT: No, it couldn't, because then that `<!` macro wouldn't know enough context, as it typically cannot look into outer scopes.

Another thing that comes to my mind reading this is, that in Guile I wrote a macro for function contracts. This macro references a unique value as the thing that should be replaced. That unique value can be imported from another module and aliased, so that name clashes are avoided. In the case of function contracts, I have a special define form `define-with-contract`, which recognizes the placeholder, where it should insert the result of applying function, to check the output contract of the function. I guess, if the placeholder made sense in other places than the contract definition/specification, then I would face the same problem, of the macro not seeing it in helper functions. Say for example I use the unique placeholder value inside a helper function, which I then use inside the contract definition.

I guess in case of a placeholder value, it couldn't be a macro itself, because then it would lack context about what should be in its place, as the scope would then be too limited.

The basic problem seems to be as you already explained, that macros cannot look inside function bodies hidden behind calls. Sort of like macros can only deal with "one layer", which is the syntax that is passed to the macro, but if that syntax contains calls to functions, then the macro is clueless about what happens inside those functions. I wonder if this limitation is present for all lisps, or if there is some macro system or method of designing them, that circumvents this problem.

But also it seems, that this is only (?) a limitation, when the macro needs to look for something specific like a placeholder. For a timing macro for example, there is no such thing and expressions merely get reordered and maybe wrapped in some `let` or something.

[dkersten]

Consider this JS/TS code:

    async function foo() {
        before()
        const bar = await take(ch)
        after(bar)
    }

its basically sugar for:

    function foo() {
        before()
        take(ch, function (bar) {
            after(bar)
        })
    }

Where the callback passed to take will be called when take resolves. And other async functions can run in the meantime.

In clojure that might look like this:

    (go
      (before)
      (let [a (<! ch)]
        (after a)))

Which would, hypothetically, convert to something like:

    (do
      (before)
      (<! ch (fn [a]
            (after a))))

If `<!` was the macro, it wouldn't be able to see the (after a) only the `(<! ch)`. `go` is the only part that can see both the code before the await and the code after the await, and it splits it at the await so it can suspend execution until the take has resolved.

EDIT: You edited while I was typing and came to the same realisation!

> I guess in case of a placeholder value, it couldn't be a macro itself, because then it would lack context about what should be in its place, as the scope would then be too limited.

In the example you gave, it could be a macro that transformed into a function call: eg `(foo my-placeholder)` turns into `(foo (read-placeholder :my-placeholder))` and that function read the registered value from a placeholder registry. It wouldn't literally transform the placeholder into the set value at compile time, but it could dynamically inject the value at runtime.

> I wonder if this limitation is present for all lisps, or if there is some macro system or method of designing them, that circumvents this problem.

I would imagine so.

Its theoretically possible that you could see the full AST + environment so that if you see a function call, you can look up the function in the environment and then access its AST. I've never heard of any language (lisp or otherwise) that did this. It could exist, though.