[johncolanduoni]

I think a good way to handle this convergence would be to move away from repeated invocations of the compiler. Instead, your build system/IDE could keep an active instance of the compiler running that keeps the last generated ASG/intermediate representation/machine code in memory, along with bookkeeping information that remembers the relations between these. When a file changes/a rebuild is requested, this lets the build system to not only correctly identify dependencies, but also allows far more granular incremental compilation than is possible now. For example, if you really only change one function, then only that function's code (and possibly any code which relies on optimizations based on that function's form) need to be changed. You could even add more to this by tracking dependencies in unit tests and automatically re-running them when the result may have changed, or using it to facilitate some sort of live-coding.

This sounds like it would need a huge amount of memory, but IDEs already do this to the ASG level, and much memory and computation is wasted on the compiler re-generating the ASG in parallel when the IDE has a similar one already analyzed. The main disadvantage is it would restrict how build systems could be structured, as to pull this off the build system would need to have much more ability to reason about the build (call command X if Y has changed after Z won't cut it). Macro systems would also need to be more predictable.

As far as keeping things non-monolithic, you could still have plenty of separation between each phase of the compilation process, the only extra interface you would need between passes is the more granular dependency tracking.

edit: grammar

[spc476]

At work I do multi-language development. One program has a mixture of C, C++ and Lua as a (more or less) static executable (Lua modules written in C and Lua are included in the executable). It was easy enough to add the steps to make to ensure the Lua files were recompiled (into an object file) if they changed. I'm bullish (skeptical) that this will be easy with an all-singing-all-dancing compiler/build/package program.

[anewhnaccount]

Bullish means optimistic.

[spc476]

You're right. I mean bearish. Sigh.

[taylanub]

I was thinking "wow that's a great idea" while reading through this and then noticed it's basically what a Lisp environment does. :-)

(And probably the environments of many other languages, but Lisp implementations (including Scheme implementations) tend to do this exceptionally well. A very good example might be Emacs, which has just an amazing help system and all.)

[qznc]

There is certainly a trend to provide the frontend separately. C# with "Compiler as a Service", libclang, etc.

The reason is not just IDE integration. You want a separate lexer (and sometimes parser) for tools like go-fmt or good syntax highlighting. Static analyzers are getting more important and there are many.