[noelwelsh]

There are many ways to do code generation. For example, you can have one program write out text files that a compiler reads to create another program. Or you can have a program change itself while it's running. These represent two extremes on a static to dynamic continuum. Most "dynamic" languages like Ruby and Python take the latter approach. This approach is hard to reason about, for both an optimizing runtime and a human, and is one reason Python and Ruby have such poor performance.

A better approach is a language with phases. There is a description of the general idea on the home page of one of the authors: https://www.kent.ac.uk/computing/people/3165/kaleba-sophie

Their work seems to focus on discovering phases. IMO it's better if the language allows the programmer to express phases as a language concept. There are a few languages that support this. Racket is the best example I know of.

Many programs already implicitly have phases. For example, a typical web app has one phase at what is traditionally thought of at compile time, then another at startup when it reads it's configuration, and then another phase when it starts running. We smoosh together the last two phases into one "run time" because our languages usually don't support phases, but actually the configuration is known earlier: at deployment time. So we could run that phase as part of the CI / CD process and catch configuration errors earlier. Once you start thinking of phases you see them everywhere. They're in data science (loading data vs exploring data), front-end web apps ("hydration"), etc. I think it's a large productivity gain that is unexplored in commercial programming.

[codethief]

This is such a great comment, which I didn't expect to find in this thread at all! I've been thinking about application "phases" for a while and I wasn't aware that there was research being done in this direction. Do you happen to have any more references beyond Racket and the link above?

> IMO it's better if the language allows the programmer to express phases as a language concept.

Yeah, not just to make life easier for the compiler, but I suspect it'd also make it easier to read & reason about the code.

I mean, performance gains are nice but sometimes performance isn't really the bottleneck but reading & maintaining that unholy cocktail of application code, bash scripts, schema files & specs, build scripts, code generators, Dockerfiles, and Gitlab YAMLs is.

[soegaard]

"Composable and compilable macros: you want it when?", Matthew Flatt https://scholar.google.com/citations?view_op=view_citation&h...

[noelwelsh]

This is a great paper. The general idea is often called "multi-stage programming". The academic work has mainly focused on performance, which I don't think is the most interesting use in industry.

[amno]

Perhaps you wish to look at Common Lisp? I think it gets you covered in all those things you mention. There is distinct read, compile and evaluation phase, all exposed to the application code.

If you get a good compiler like sbcl, you can go a long way with just the language itself since the language itself offers a blend of scripting language qualities while being a compile language. Emacs Lisp can go long way too.