[hellofunk]

Typically you are limited at compile time because Lisps -- especially Clojure -- do not allow the same computing resources you get at runtime to be used at compile time, so there are some clear limits.

This can be true in C++ as well, but those limits are quite high and are easily set to something else. Perhaps Clojure also allows you to reset what these limits are.

[billsix]

If you redefine define as a macro which evals the definition during macro expansion, you can have the full runtime at Compile-Time. State and IO included!

https://github.com/billsix/bug

http://www.lulu.com/shop/bill-six/computation-at-compile-tim...

[kazinator]

Sometimes the situation is completely reversed: you get a lot more computing resources on the development system where the code is compiled, than on the target system where it ultimately runs (e.g. small, embedded target).

The computing resources required to process the code (macros, and whatever) is usually unrelated to the computing resources of the application.

A program that works with gigabytes or data might need only megabytes during its compilation.

(It's not easy to imagine practical circumstances where it would be reversed, with those exact numbers. Some algorithms you might want to use at compile time do explode in memory size, I suppose.)

[Someone]

That is getting quite normal nowadays; training a model in machine learning typically takes way more resources than running it.

However, if you need gigabyte to expand your macros, it is likely you need megaseconds, too, and then, it is wise to use 'precompiled headers', and save your 'preprocessor output'.

[ohyes]

The first part of this is not true.

You can execute arbitrary code at macro-expansion time.

You can also recompile while the process is running, which would include re-macroexpanding.

[hellofunk]

I think you misunderstood what I'm saying. Typically, the macroexpansion is a compile time process that uses different machine resources than the runtime processes of the same language.

[ohyes]

Not in a lisp it isn't, compiling very often happens at runtime, in which case you're compiling in the same process as the code that is running at that time, in which case the same resources are available to both.

This is how you get a REPL and other forms of runtime evaluation.

This is kind of the nub of being a dynamic programming language, if it can't do this, it isn't a dynamic programming language. Lisps are generally dynamic programming languages.

Clojure-script might be an exception because of how it is implemented, but Clojure proper on the JVM certainly supports this type of behavior.

[hellofunk]

There must be a setting in regular Clojure on the JVM for controlling how much recursion or memory usage is available during macro expansion, because I've hit that limit very easily in the past. It's been about a year since I used Clojure much, but I doubt that's changed.

[ohyes]

That makes more sense. It's possible that this is a design decision by Rich Hickey, I'm not aware of the intricacies of how the Clojure macro-expansion works. However, that sounds a lot like you hit a bug in the Clojure compiler (or in general needed to beef up the allowed memory of the JVM it was running on, compilation obviously isn't cheap... everything should all be in the same resident memory).

I would expect to be able to do (eval `(defun ,function_name () (complicated_macro ,@args))) or (compile `(defun ,function_name () (complicated_macro ,@args)))) at runtime in Common Lisp, without even really considering the consequences beyond whether it is necessary.

It's supposed to work a little bit like if you had an implementation of GCC in the same process as your C program, and you could call GCC from a function to compile some new C code for you to add features or replace existing code. It's powerful + dangerous.

[hellofunk]

In my case, it was definitely a problem of resource limits during compilation only, since the same logic would work fine if not a macro and run at runtime, so not a general case of the JVM not setup to handle the work, but specifically macros not able to handle the work.

C++ obviously has a limit on template depths as well, but they are so high that rarely does anyone get close to the defaults. Clojure probably has very limited defaults (assuming that defaults are the cause).

[ska]

I can't make sense of what you are trying to say. In c++ you typically have no access whatsoever to the compiler at runtime. Through dynamic linking and some hacking about you can do this but it is all very non-standard.

In lisps you very typically have complete access to the compiler, though this is not universal. Resource limits on a running process are a separable issue.

[hellofunk]

We are talking about the resources of the machine and which of those resources are available during compile time compared to those which are available at runtime.

[ska]

Yes, but you start off with an incorrect premise, and it's hard to understand what point you are trying to make.

In a very typical lisp system, for example, exactly the same resources would be available at runtime as your original compile (in fact, the boundary between these two is pretty fuzzy and often might be the same process anyway)

Likewise, C++ typically doesn't have any runtime access to compilers at all, so what resources are available to them or not is a bit moot.

If the point is that you might deploy on a different machine or configuration, well - that's true, but I can't see the relevance.

You might be pointing out some oddities of Clojure, I suppose, but that's not a great model for "typical lisp".

[hellofunk]

Indeed my experience is mostly with Clojure, which I mentioned because my parent commenter referred to ClojureScript. Glad to know a "normal" lisp handles this differently.

[kazinator]

Sometimes the situation is completely reversed: you get a lot more computing resources on the development system where the code is compiled, than on the target system where it ultimately runs (e.g. small, embedded target).

The computing resources required to process the code (macros, and whatever) are usually unrelated to the computing resources used by that code when it runs.

[hacker_9]

Do you have any links documenting resource limits in Clojure macros?

[hellofunk]

I have plenty of personal experience trying to do things at compile time that are simply not available, and while I haven't looked up the details in a while, I do recall there being such a limit, but perhaps you have to make some settings on the compiler to get around. Macro expansion has relatively small limits in Clojure for things like memory usage.