[ahepp]

>C++Builder with TwineCompile is a powerful productivity solution for multi-core machines compiling 1 million lines of code very quickly and can work better than the MAKE/GCC parallel compilation Jobs feature due to it’s deep IDE integration

You're claiming this plugin has deeper IDE integration than `make`? I find that really, really difficult to believe. And if it's true, it seems like the solution is to either use a better IDE, or improve IDE support for the de facto standard tools that already exist, as opposed to writing a plugin for the -j flag.

[fmxexpress]

Yes, it could be as simple as having Dev-C++ run a build every time a file is saved. Currently it does not do this. Remember, Dev-C++ didn't have -j support at all until I added it. TwineCompile does do this (background compile). Therefore the IDE is providing this functionality and has nothing really do to with make or the compiler.

TwineCompile is not a plugin wrapping the -j flag. It is a separate thing entirely unique to C++Builder. It does offer integration with MSBuild though.

The second part of that was the fall off. With the 1 million size files it only ever used half of the cores and each successive round of core compiles it would use even less cores. TwineCompile didn't seem to have that problem but this post was not about TwineCompile vs. MAKE -j so I did not investigate this farther.

I was expecting MAKE/GCC to blow me away and use all 64 cores full bore until complete and it did not do this.

[klodolph]

Make forces you to choose between being able to do full parallel builds or using recursive make, you can’t do both.

[StillBored]

I might be misunderstanding something, but the common gnumake does no such thing.

https://www.gnu.org/software/make/manual/html_node/Job-Slots...

[klodolph]

I didn’t go into great detail of the reasons, but the jobserver doesn’t address the problem except for the most trivial cases—the core problem is that you can’t cross to a recursive make invocation via multiple edges.

This is fairly common in larger projects, so you end up having to do some hackery to manually sequence make invocations if you want to use recursive make (which is pretty awful).

Honestly, for large projects, Make is an insane choice, notwithstanding the fact that people who are sufficiently savvy at Make can sometimes make it work. (If your tools are bad, you can make up for it with extra staff time and expertise.)

[hedora]

> the core problem is that you can’t cross to a recursive make invocation via multiple edges.

I’ve never had that issue, and used to heavily use recursive make. I carefully benchmarked those make files, and am sure this wasn’t an issue.

I suggest reading the paper “Recursive Make Considered Harmful” before attempting to use make for a large project. It describes a lot of anti-patterns, and better alternatives.

I’ve found every alternative to make that I’ve used to be inferior, and they almost always advertise a “killer feature” that’s a bad idea, or already handled better by make. It’s surprising how many people reimplemented make because they didn’t want to RTFM.

Anyway, the next build system on my list to check out is bazel. Some people that I’ve seen make good use of make say it’s actually an improvement.

[klodolph]

> I’ve never had that issue, and used to heavily use recursive make. I carefully benchmarked those make files, and am sure this wasn’t an issue.

So, you’ve never had two different targets that depend on something in a subdirectory? If you’ve just solved this by building subdirectories in a specific order, or building entire subdirectories rather than the specific targets you need, what you’re really doing is making an incorrect dependency graph in order to work around make’s limitations. These kind of decisions make sense for full builds, but interfere with incremental builds.

Bazel & family (Buck, Pants, Please, etc.) are among the few build systems that solve this problem well. It’s not an accident that they all use package:target syntax for specifying targets, rather than just using a path, because this allows the build system to determine exactly which file contains the relevant build rule without having to probe the filesystem.

I would love to simply recommend Bazel but the fact is there is a bit of a barrier to entry depending on how standard / nonstandard your build rules are and depending on how you think that third-party dependencies should be pulled in. Depending on your project, you could convert to Bazel in an hour just by looking at a couple examples, or you could have to dive deep into Bazel to figure out how to do something (custom toolchains, custom rules, etc.)

As you observed, the alternatives to make are often inferior, and it’s often because they’re solving the wrong problems. For example, sticking a more complex & sophisticated scripting system in front of make.

[hedora]

If there’s a requirement to use an intermediate target from a subdirectory, and that’s not expressible in the parents or siblings, you could run into issues. I thought you meant it failed to parallelize multiple unrelated targets because they are in multiple subdirectories.

Anyway, the solution to that specific problem is written up in “Recursive make considered harmful”. Make handles it elegantly.

The title is kind of a spoiler, but if you’re using recursive make and care about fine grained dependencies across makefiles, you’re doing something wrong.

As an aside, I wonder if recursive make misuse is why people claim ninja is faster than make. I’ve benchmarked projects before and after, and before, make was generally using < 1% CPU vs 1000%+ for clang.

Afterwards, ninja was exploiting the same underlying parallelism in the dependency graph as make was. Thanks to Amdahl’s Law, there was no measurable speedup. The only thing I can figure is there’s some perceived lack of expressivity in make’s dependency language, or there are an absurd number of dependencies with trivial “build” steps.

[ChuckMcM]

That is a really excellent observation.

My experience is that "larger projects" to the investment in their build infrastructure to maximize parallelism in order to reduce build times because it pays dividends in terms of turn around time and thus productivity.

One of the joys of working with someone who has been building large projects for a while, is that they just design the build system from the start to be as parallel as practical.

[StillBored]

Make isn't great, but if you look at the opensource world the vast majority of the large projects are make based. The backbone of your average distro is a thin layer on top of what is mostly automake/autoconf in the individual projects. That is because while I can create a toy project that builds with make in a half dozen lines, big projects can extend it to cover those odd edge cases that break a lot of other "better" build systems. Particularity, when a project starts including a half dozen different languages.

So, while i'm not a make fan, I'm really tired of people pissing on solutions (c also comes to mind) that have been working for decades because of edge cases or problems of their own creation because they don't understand the tool.

A well understood tool is one where people know where the problems lie and work around them. Most "perfect" tools are just project crashing dragons hiding under pretty marketing of immature tooling.

[klodolph]

> …I'm really tired of people pissing on solutions (c also comes to mind) that have been working for decades because of edge cases or problems of their own creation because they don't understand the tool.

You would have to be extraordinarily cynical to think that there’s been no progress in making better build software in the past forty years or so.

Yes, there are definitely plenty of build systems out there that were built from scratch to “fix” make without a sufficient understanding of the underlying problem. I’m not going to name any; use your imagination. But make was never that good at handling large projects to begin with, and it was never especially good at ensuring builds are correct. This is not “pissing on make”, for Chris’s sake, make is from 1976 and it’s damn amazing that it’s still useful. This is just recognizing make’s limitations, and one of those limitations is the ability for make to handle large projects well in general, specific cases notwithstanding. Part of that is performance issues, and these are tied to correctness. As projects which use make grow larger, it becomes more likely that they run into problems which either interfere with the ability to run make recursively or the ability to run make incrementally.

The “poor craftsman who blames his tools” aphorism is a nice way to summarize your thoughts about these kind of problems sometimes, and it does apply to using tools like make and C, but it’s also a poor craftsman who chooses tools poorly suited to the task.

[ChuckMcM]

It may be a Windows thing. I too started reading the post and was thinking, why not -j128 or -j64 depending on if HT was on and then realized that the author's system wasn't one that had been tuned for decades to build software as quickly as possible :-).

It would be an interesting academic exercise to create a setup on Linux that did the same thing but using the extant build tools, and then to do a deep dive into how much of the computer was actually compiling code and how much of it was running the machinery of the OS/IDE/Linkages. A LONG time ago we used to do that at Sun to profile the "efficiency" of a machine in order to maximize the work it did for the user per unit time, vs other things. That sort of activity however became less of a priority as the CPUs literally became 1000x faster.

[archi42]

We're running our builds and tests on both Windows 10 and Linux. At least for Windows 7, the compiler start up time had a huge impact on total run time. I think back then we had 2500 tests, each compiled, then linked, and then dwarfdumped and then some more tools (we build a compiler). We moved some stuff to Linux (eg dwarfdump) and saved a few program executions by compiling & linking single c file tests in one step (yeah, the driver still calls everything, but the driver is called once less). I was under the impression Windows 10 improved on that (we didn't benchmark in detail since then, since it's still far enough), but on Linux I don't have to bat an eye on eg make -j32.

BTW, the author messed up anyway. Make -j does schedule everything at once. How do I know? Only way I got my private 128GB build host to oom...

[zajio1am]

Why would you do recursive make? That is setup discouraged for decades ...

[ahepp]

(for reference: "Recursive Make Considered Harmful", https://accu.org/journals/overload/14/71/miller_2004/)

[Too]

Because you included a third-party project into your build which is not compatible with your build system and you don't want to rewrite all their ninjafiles into makefiles to match.

Someone should really formalize a standard for declaring dependencies which build systems can share between each other.

[klodolph]

A “standard for declaring dependencies” is, itself, a build system without the execution engine. The execution engine is not actually the hard part about making a build system. So, what you’re asking for is really just a standardized build system.

[klodolph]

Because it can be hard to maintain non-recursive make systems for large projects. Just because recursive make is discouraged does not mean that the alternative is without its own drawbacks.