[unclad5968]

As someone that learned C# and python before C and C++, to this day I couldn't explain to you how make rules work. Make is so unlike build tools from other languages that it doesn't surprise me someone would rather use python to bootstrap their compiler.

[anyfoo]

It is so trivial, it takes you 10 minutes to learn.

Less even, here is an attempt:

    foo.o: foo.c
        clang -c -o foo.o foo.c

This builds a file called foo.o, if foo.c is newer than foo.o. Imagine that there is an identical bar.o as well, that builds it from bar.c.

    fooexec: foo.o bar.o
        clang -o fooexec foo.o bar.o

This links together foo.o and bar.o into a file called fooexec, if at least one of foo.o or bar.o is newer (which in turn means that any of its respective dependencies changed; e.g. change bar.c, and bar.o and then fooexec will rebuild).

Now type this into the shell:

    make fooexec

Congratulations! You have a fully built fooexec. Now change bar.c, type it again, and like magic, you have a rebuilt bar.o and fooexec. foo.o will stay put, it’s up to date!

But of course all the .c -> .o rules look the same, so instead of writing a new one for every file, you'd use simple wildcards, and now the whole, entire Makefile is just:

    %.o: %.c
        clang -c -o $@ $<

    fooexec: foo.o bar.o
        clang -o fooexec foo.o bar.o

Do $@ and $< look arcane at first glance? Yes. Is what they do simple, though? Also yes. (If your complaint is that you don't know what those "clang" lines do, then you don't know how to use the C compiler, and will have the same problem when writing a python script.)

It pains me if that is not immediately obvious. And it pains me even more that someone would want to write a boilerplate heavy, complex python program, that rebuilds everything all the time, instead of just learning this very simple concept.

[IgorPartola]

This is an excellent explanation of make! The only thing I will add is the slightly more abstract thought that make is meant to build files out of other files based on timestamps of the source files. And the target files can themselves become source files, which means that you can build dependency graphs. Lastly, it has “caching” built in: if bar.o is newer than bar.c, don’t rebuild it. If it doesn’t exist or is older, do rebuild it. So when you change one file, you don’t recompile the entire damn tree.

It also has “phony” targets that aren’t files. The canonical example of this is “clean” which basically just does something like rm -rf *.o fooexec

This is what the JavaScript tooling people have been chasing for the past 15 years and can’t seem to grasp. Someone actually did build a make-based JS build system I believe but it never got popular because it wasn’t written in JavaScript.

[fuzztester]

>Lastly, it has “caching” built in:

A nitpick, but I don't think that should really be called caching, although it kind of is, in a looser sense. It's just comparing the file timestamps of the .o and .c files, and building the .o from the .c, if the .c is newer, or if the .o doesn't exist.

[IgorPartola]

I mean it’s caching using a key derivation scheme which uses file names rather than file contents. It might not be sophisticated caching but it is caching.

[CJefferson]

Saying this is an 'incremental makefile' isn't really correct, as changes to header files aren't going to lead to rebuilding.

So, either you need to manually keep references to which .h files you include in your Makefiles up to date, or start worrying about M / MM / MG / MP, and of course you'd like those to be re-run when you change your files, and suddenly your Makefile is an awful lot less simple.

This is the main reason I stopped teaching Makefiles in intro to C courses -- students often lost entire afternoons to forgetting this stuff, and failing to realise their mistake. It really shouldn't be any person's job to keep this up to date, when the computer knows it.

[anyfoo]

It is correct to call it an incremental Makefile, it merely doesn’t name the dependencies for you. A Makefile has exactly the dependencies you specify. It’s that simple.

That is not necessarily the most useful. Especially as your project gets larger, you probably will not want to specify every header file as a manual dependency, or alternatively rebuild everything every time you change a header file. (Though for me personally, either approach often works for surprisingly long.)

Then you can do things like letting the C preprocessor create the dependencies that get included in your Makefile (yes, common C compilers can create Makefile fragments). Or do something more complicated. Or switch to another build system, if your project really outgrew, or really mismatches, make.

But at its core, Makefiles follow a simple concept, are entirely language agnostic, and even with just the simpler concepts they remain useful for a long time.

[CJefferson]

Having 'incrementality', when it isn't actually going to incrementally rebuild things when files change, is (in my experience), worse that having no incrementality at all. Having to remember when I have to manually disable to incrementality is an annoying overhead, and easy to forget.

If you can remember exactly when you need to manually skip the incremental build, that's great for you, but I find Make has enough of these kinds of footguns I don't recommend it to people any more.

[anyfoo]

Meh, it can practically never be perfect, without diminishing returns that at some point invalidate the advantage of any incremental builds anyway.

As I said, you can let the C compiler generate your header dependencies for you. But that's not enough, because what's with header and libraries that are external for your project? Keep in mind that they will have dependencies, too, and at some point your transitive closure will be really big.

To a lesser degree, what's with a compiler, linker, or any random other part of your toolchain that changes (at least those are supposed to mostly keep a stable interface, which isn't the case at all for external libraries, but in some cases the toolchain can cause the footguns you mentioned).

A lot of the times, you don't need to rebuild everything just because a system header or the toolchain changed, but your build system would have a hard time knowing that. Because at some point, the halting problem even gets in your way (i.e. you can't really reliably detect if a given change in a header file would need rebuilding of a source file, which you would need for "fully incremental" builds). So it's always a trade-off.

Personally, I've fared very well with generated header dependencies (sometimes manually declared ones or none at all for small projects), and many other projects have, too.

YMMV of course, but I don't observe this to be bad. Most people who program C and use make are, I think, aware of what header mismatches can cause, and how to avoid that.

[CJefferson]

I think one fundamental difference between the two of us (I'm guessing here, let me know if I'm wrong), is I'm willing to cope with longer compile times, as long as I always get a correctly built executable (so there are no 'halting problems', if in doubt, rebuild it!), while you would prefer faster build times, even if that then requires sometimes having to do a little manual work when the build system doesn't realise it needs to rebuild? Of course, no system will be 100% perfect, but you can choose where your trade-off point is.

Two reasonable viewpoints, but probably effects how we do our building setups!

[fuhsnn]

Yes, this is quite important for C projects as it leads to subtle bugs. I also long for a way to let make detect a change of $CC (new build of the compiler itself) in the context of debugging a compiler against third party makefile projects.

[anyfoo]

Have a step that takes the SHA256 or similar of $CC, make that a dependency. You’d need slightly more advanced make features if you want to make it “pretty” (and not just, say, touch a file through some shell code comparison), but this is a slightly more advanced request.

Or potentially just specify the path to $CC itself as dependency? Presumably, if the C compiler changes, the timestamp of its executable does too. (Bad if it goes backwards, though.)

[yjftsjthsd-h]

> Have a step that takes the SHA256 or similar of $CC, make that a dependency.

Careful, one more step and we'll start recreating nix in Makefiles;)

(Wait, could we? Should we?)

[CJefferson]

I often use a Python program called 'fabricate', which basically checks every file a command opens, and re-runs the command if any touched file changes.

I love it, and I hoped this would become the future of how build-systems are built -- you could automatically parallelise, rebuild, make clean, everything, just by tracking which files are read, and written.

It does add a bit of overhead, but I'd be willing to pay that for simplicity. Unfortunately, it doesn't seem to have caught on.

[anyfoo]

That is neat, but also super brittle, and full of abstraction leakiness, potentially leading to both false positives and negatives. Sounds like it would break down with ccache or distcc immediately, for example.

[CJefferson]

I never had any false negatives, they are basically impossible in this framework, except it does assume if a program, and all it's inputs are unchanged, the output is unchanged, so this is no good for programs which randomly produce different outputs.

ccache can cause things to get built twice, because the first run it sees it tries to read a file, then later writes to it, so it knows something different might happen if you run again -- but then again that is just ccache doing it's thing, so it's quick. Yes, distcc wouldn't work, but then again I find distcc super-brittle (so hard to make sure everything has the same compiler versions, etc), I may have used it 20 years ago, but I don't think I ever would now!

[dezgeg]

The 'tup' build system also works this way.

[exe34]

You could have taught them `make clean` whenever there's any issues. I work with software engineers and I still have to remind them to try rm -rf ./build/* now and again, which always seems to solve the problem.

[CJefferson]

I do, but it’s easy for experts to forget, and when you are in your first few weeks of C it’s even easier to forget with everything else you have to remember.

[peterfirefly]

This is an amazingly good 2-minute introduction to makefiles. It would have been better without the last two sentences (and most of your comments on the topic).

---

Something skipped over by your introduction (and most guides to makefiles) which very quickly either makes makefiles less short or much more complicated is header files.

C files often depend on one or more header files that aren't system headers (this C compiler certainly does).

They could be added to each individual rule by hand:

  codegen/codegen.o: codegen/codegen.c codegen/codegen.h linked_list.h libc.h
       $(CC) -c -o $@ $<

or they could be put into a variable:

  HEADERS = codegen/codegen.h parser/parser.h parser/goto.h linked_list.h libc.h 

  codegen/codegen.o: codegen/codegen.c $(HEADERS)
       $(CC) -c -o $@ $<

which causes too much code to be recompiled every time a header file is changed.

Or we could do something really fancy and get only the right header files added as dependencies for each compilation unit. That requires using a feature of the gcc/clang compilers (their preprocessors, really) that causes them to output a small makefile snippet that lists the exact header dependencies for a compilation unit (the '-MD' option). These snippets can then be included in the makefile. To make this look relatively clean requires some fancy macro use that is waaaay beyond what people can learn in ten minutes.

This is an extremely common use case and it is a bit of an embarrassment that GNU make doesn't handle it better.

[anyfoo]

I did mention that in other comments, including the C preprocessor helping. make is completely language agnostic, so it cannot “guess” header dependencies. In make, you get the dependencies you specify (whether manual or generated), that’s what I wanted to show.

[peterfirefly]

Make makes the easy part easy but the rest quite hard -- which I think you agree with.

It sure would be nice to have something like this:

  codegen/codegen.o: !codegen/codegen.c
     $(CC) -c $(CFLAGS) -o $@ $<

'!' would run $(CC) with -MD to extract the header dependencies. Doing that back in 1995 would have increased the value and usability of GNU make immensely (and greatly reduced the need for all the other complexity it has).

[teo_zero]

Wouldn't this require make to run CC with -MD on each instance of ! every time it's invoked, just to collect the list of dependencies, maybe just to discover that no file has changed?

[dzaima]

I've written a good amount of makefiles (and even one rather complex one.. before replacing it with a thing written in an actually sane language, bootstrapped by a stripped down version of the makefile), and I still mix up $@ $< $> or whatever else my brain comes up with. And this is coming from someone deep into array languages, which have entire sets of unicode glyphs.

For how often one typically writes/looks at build scripts, remembering the specifics about makefiles is quite likely to not be worth the effort.

Makefiles start becoming incredibly awful if you want to do anything outside of pure computations in the separate targets. Making the target directory requires either putting the mkdir call in each separate built thing (awfully redundant), introducing a whole another rule (and still adding it as a dependency on every rule that'll be outputting to said directory), doing some recursive make, or running it outside of rules (but putting it under a proper if statement is likely pain). That's a bunch of awful solutions to something that really really really should just be a single calls to mkdir in any sane system.

Another horrifically awful thing is if you want some reused data across rules (e.g. pkg-config results, current OS/arch). You can put them as globals, but that'll make them be calculated always, even if no build target needs them, unless you put them under an if, which is again non-trivial to get the condition correctly for. This becomes super bad if you do recursive make stuff.

More generally, makefiles start out simple, but doing anything that's not strictly simple is gonna be pain, at the very least requiring looking at docs of a thing you'll probably never use again, and debuggability is also very low (amusingly, '-p' and '--debug' break on recursive make, seemingly losing some variables; and '-n' requires copying out & running the recursive call to get to the actually important part; and afaict none of those help with figuring out why a thing didn't build).

And you're completely SOL if you wanted to do things like reorder compilations based on how long they previously took, or want a nice display of currently-compiling things.

[anyfoo]

Fair. But as said, in my experience, they remain useful and simple for a pretty long time. I usually always start out with a simple Makefile. Only sometimes do I start with or eventually switch to something else because of the projects’ needs, but for most stuff (even stuff where the project itself is complex, i.e. embedded kernel stuff where different units are pasted together and whatnot), relatively simple Makefiles do their job pretty well.

I also have a lot of phony targets. It really enjoy just being able to type:

   make restore kernel reset upload

To bring a device into a sane state, build the kernel, reset the device, and upload the built firmware.

[dzaima]

Aand that brings up another issue - that (and a good amount of things that will work as expected on the default -j1) breaks with -j4.

Apparently .WAIT and .NOTPARALLEL exist, which I couldn't find last time I needed something around this. Though for this unfortunately `make -j4 foo .WAIT bar` isn't allowed, but at that point separate make invocations is fine.

[anyfoo]

Another fair point. I think I'd use this:

    make restore & (make -j 20 kernel && make reset upload)

But in reality, the kernel target actually calls make anyway, so it's just in there. Depends on the exact situation.

[saghm]

> Do $@ and $< look arcane at first glance? Yes. Is what they do simple, though?

And yet, I still don't actually know what `$@` and `$<` actually mean outside of your example. I only can assume that somehow that substitutes in `foo.o` and `foo.c` because you literally put the non-magic versions right below it (and above it a few paragraphs above as well), but I'm honestly not even positive that I'd run `make foo.o` rather than `make foo` with your example, and I feel fairly confident in guessing that this would _not_ compile into a binary called `fooexec` like the target you defined above. I don't have an idea whatsoever what I'd need to put in the target to be able to substitute in something like that because the fact that the first `%` somehow becomes `$@` and the second becomes `$<` doesn't follow any obvious pattern I can use to figure out what the third symbol would be, or even if there is one.

The problem is that "simple" is not the same as "intuitive" or "discoverable". Yes, I could probably put in the effort to go and find out these things, but why would you expect someone to be motivated to do that by someone telling you that it's "trivial" and it "pains" them as they explain things in a way that doesn't actually clarify things in a way that helps me? If you actually want to try to make a strong case that this is something worth learning for people, repeatedly referring to it in ways like "very simple" and "so trivial" and "immediately obvious" is counterproductive.

As an aside, I think it's also a bit of leap to assume that they're "rebuilding everything all the time". It looks to me like the build.py script is only needed a single time to bootstrap from the original compiler to this one, and rebuilding every single C file in the new compiler rather than using the artifacts built with the original one is kind of the whole point. After that's done once, I don't see why the new compiler couldn't be used via the Makefile. If you're complaining about Python build scripts in general rather than this specific one, I don't know why you assume they can't be written to check the timestamps on the files to see if a rebuild is necessary before doing so. That's exactly what `make` does, and hey, it's a very simple concept!

[peterfirefly]

$@ is the name of the target -- the @looks a bit like an old-fashioned target for practice and competition shooting.

$< is the name of the input (the file the rule depends on). If the rule depends on more than one input, it is the first one. So you write your dependencies as xxx.c xxx.h yyy.h zzz.h with the C file first. Mnemotechnically, it is "the thing that goes into the build".

$^ is the names of all the files the rule depends on. Think of it as a large hat above all the input file names.

Many things in make are clumsy and GNU make is a behemoth that does so much more than the original make that it is really hard to learn -- but these shortcuts have really good names!

(They are GNU make extensions, btw.)

[anyfoo]

One might ask how you learned C (or python, or anything else) in the first place, if you can't be bothered to learn what the very simple $< and $@ mean, which would take you literally seconds to look up if my example wasn't actually enough already.

The rest is just... of course you can reimplement make, or a subset of it, in python, python is turing complete after all. But why? make isn't that arcane. I'm sorry it uses the weirdly looking $@ and $< for "target" and "dependency" respectively. Those two are extremely common and you have to type them in a lot of rules, so the authors chose a shorthand. One way or another your python make subset will have to bring in the same concepts they represent, too.

This is trivial in the sense that it's a part of any build system worth its salt. And I can almost guarantee you that OP for example, who wrote their own C compiler, understood it immediately.

[mschuster91]

> One might ask how you learned C (or python, or anything else) in the first place, if you can't be bothered to learn what the very simple $< and $@ mean, which would take you literally seconds to look up if my example wasn't actually enough already.

The problem is, you typically don't need to touch Makefiles that often once they work (or fulfil a reasonable interpretation of "work"), so unless you're a distro packager or the project(s) you work on are sufficiently complex and large to warrant an entire FTE just for build tooling, chances are high you need to touch that stuff only once every few years, by which time almost everyone has to dig into the documentation yet again... made worse by the fact that Google's search quality has gone down the drain.

[saghm]

I don't disagree that the concepts are simple; I'm arguing that the way they're conveyed in makefiles optimizes for the wrong thing by prioritizing tenseness over conveying useful information. The issue isn't that they "look weird", but that they give context that can be used to build additional concepts off of. A sibling reply to my comment mentions that `$<` only specifies the first dependency, and that there's a separate symbol for specifying all of the dependencies. That might be easy enough to remember after learning, but when someone is literally just learning how to write makefiles for the first time, it won't be obvious whether `$<` is used for only a single dependency or all of them, let alone what symbol to use for dealing with more than one.

My point is that concepts being simple doesn't mean that the choice of how they're represented doesn't matter. When people aren't bothering to learn something you think seems "simple", it's a mistake to immediately assume that it's due to laziness or incompetence rather than first trying to understand their motivations. Maybe they are just lazy or incompetent, but it's also possible that there's legitimate confusion around something that wouldn't occur to you without seeing it the way a beginner sees it, and there's room for improving how things are documented or taught.

There's a pattern of thinking that seems pretty common in our industry where we focus so much on the technical details of a system that the actual human experience of using it gets overlooked. My problem with this line of thinking is that value a tool is only realized when people actually use it. As a hypothetical, imagine that some new technical issue becomes commonplace and two different tools get written to solve the problem; the first tool solves the problem perfectly and as quickly as possible every time but doesn't get used by anyone, and the second tool does only an adequate job of solving the problem and requires a bit of manual additional work from the user to fix things, and that tool gets used by almost everyone. I'd argue that the first tool is not actually successful in any meaningful way because it doesn't actually help anyone with the problem that it was designed to solve.

To be clear, I'm not trying to say that no one uses makefiles or that they don't provide any real world value. I'm trying to say that if you're actually trying to get more people to use makefiles because they solve problems that everyone has better than what they're currently doing to try to solve them, the most effective way would be to look at it from the perspective of the people who aren't already using it and figuring out what's stopping them. When you feel strongly that it's the best way to do things, anybody who isn't doing things that way must see things differently, and the best way to try to change that starts by figuring out what that difference is.

If you don't actually care about trying to convince people and mostly just want to vent, then this advice won't be relevant, but if your goal is actually to try to change the way people do things, the approach you use to explain things is going to matter just as much as your knowledge about the things you're explaining. Just like how a brilliant professor who does groundbreaking research might be terrible at teaching an intro course to undergrads, it's possible to be an expert at something but not be effective at teaching it, and I think failing to appreciate the difference often ends up leading to a lot of frustration that would otherwise be avoidable.

[anyfoo]

Lots of things optimize for terseness. Assembly, arithmetic/algebra, and any kind of shorthand in any kind of trade. That’s because the shorthand is usually easy enough to learn, and then provides value going forward.

You are perfectly capable of parsing “3+2*4”, I don’t have to always write it as “MULTIPLY 2 BY 4, ADD 3”. Similarly you would get pretty mad if instead of “ld” or “lda” you’d have to type “load register” or “load accumulator” each time. Those concepts are trivial once you understood them, you don’t need a constant reminder of what they expand to.

Doing so might get slightly more people to “start using the thing” in the very first step, but might also let many people look for a more terse alternative. And we don’t have to cater to any person who is really unwilling to learn.

Overall I find it hard to buy that you actually had much trouble with $< and $@, at least in a way that you couldn’t have cleared up for yourself about 20 times in the time it took you to write out your comments.

I was hoping my introduction to Makefiles would be useful. If it was not, maybe I’m a bad educator after all.

[mturmon]

Since you’re asking, $@ is the “target” of the build, that is, what you are aiming to build (left of the colon in the rule). This is why it looks like a bullseye.

And $< is of course the thing the target depends on. It’s the input, like standard Unix notation for stdin.

Hopefully this helps?

[akira2501]

    output: dependencies
        <command(s) that use dependencies to create output>

If any "dependencies" are newer than "output" or if "output" does not exist the commands run. Otherwise they don't.

[yodsanklai]

> Make is so unlike build tools from other languages

Interesting, I thought that "make" was like multiplication tables, something you learn very early one as it's a super simple tool. Also I assumed it was part of any unix 101 class, together with bash and command line stuff. But probably it shows my age. Most people don't have any reason to learn these tools nowadays.

[unclad5968]

That is likely the case for anyone formally educated in C. If I needed to learn make I would, but I use cmake because I develop on multiple platforms.