Show HN: Making GNU Make a better Task Runner

[mitjafelicijan]

I know this could be considered blasphemous, but I constantly find myself using Make as a task runner. I have written my own task runner in the past, but somehow I always end up using make.

I went, and I put together 3 quality of life snippets I use all the time and put it in a single makext.mk file that can be included in other Makefiles and wrote a basic readme for it.

This is not meant to be a replacement for other task runners, but I do think it can be useful to some of you.

https://github.com/mitjafelicijan/makext

Check it out and see if it makes sense to you.

Thanks for any feedback or comments.

Cheers

[tpoacher]

Very nice.

Whenever I see comments about people trying to reinvent make, my conclusion is that make "as a tool" is inevitably far superior, but people are mostly familiar with a small subset of its functionality, which makes it appear clunky. That, and the syntax is very flexible, which means it can easily be abused to write ugly/convoluted makefiles.

If I had one thing I'd want to improve about make, it wouldn't be the tool itself, but its documentation, and a set of reasonable defaults/templates, like you have here.

The documentation is amazing in one sense, in that it's very comprehensive, but very clunky and hard to master in another, in that it's organised in a very high-level manner that doesn't allow it to be used as a quick reference / point of truth, if you don't know what you're looking for. I found myself making anki notes for it, as it's not organised in the kind of way that would simply allow me to refer to the docs for your usecase and get things done. You really either know some functionality exists and can look it up or you don't, and unless you're someone who uses make in expert mode on a daily basis, it's hard to know all the specialised components and be able to combine them all together seamlessly. Hence my anki notes, hahah.

But make really is an amazing tool. I wish people focused on improving make workflows instead of reinventing the wheel all the time with other tools (which are then forced on the user).

[frankjr]

> Whenever I see comments about people trying to reinvent make, my conclusion is that make "as a tool" is inevitably far superior, but people are mostly familiar with a small subset of its functionality, which makes it appear clunky.

If you're looking for a task runner, not a build tool, then make is not "far superior" in any sense and there are much better alternatives, the most prominent probably being "just". Even something as basic as accepting parameters to a task (make fetch <arg1> <arg2>) is awkward because make doesn't understand this syntax and interprets the arguments as targets. The only way to make it work is either through named arguments or empty target hacks and in both cases you need to write the validation logic yourself. In just it's simply:

  fetch PACKAGE VER:
    @echo fetching {{PACKAGE}} at version {{VER}}

  $ just fetch foo 
  error: Recipe `fetch` got 1 argument but takes 2
  usage:
    just fetch PACKAGE VER

  $ just fetch foo 1.2.3
  fetching foo at version 1.2.3

https://just.systems/man/en/chapter_1.html

[nrclark]

Just is fine, don't get me wrong. But Make targets can take arguements too. They just have to be named, or (if you stretch the definition a bit) wildcard targets.

AFAIK, Just doesn't have any way to mark "this task is complete, so don't re-run it". It also doesn't have file-based dependendicies, which are pretty common for any kind of programming.

Why would I want to lose all of Make's flexibility and power in exchange for slightly prettier UI syntax?

[frankjr]

> AFAIK, Just doesn't have any way to mark "this task is complete, so don't re-run it". It also doesn't have file-based dependendicies, which are pretty common for any kind of programming.

That's fair but also not what I'm looking for in a task runner, that's why I explicitly said "not a build tool", which to me this falls under (although I realize everybody has a different definition of these terms and where they draw the line if at all).

[kstrauser]

Because it makes the common things easier.

We make these tradeoffs all the time in software engineering. Raw machine code is the most flexible and powerful programming language, and yet most of us here aren't using it today. Make isn't machine code and Just isn't Python, but there's an analogy: you give up something by using the latter in exchange for making it easier to do everyday things.

Just doesn't replace Make. For the thousand everyday tasks where Make's power isn't needed, it's likely a better choice for most people.

[maccard]

I think this post is well intentioned but misses the point entirely.

    GOOS=darwin GOARCH=arm64 go build -ldflags="-X 'main.Version=v1.2.3'"

Executing this with variables for os/arch/version is not straightforward in a makefile. You might argue that it's a flaw with the `go build` command, but replace go with cargo/dotnet/maven and you have the same problem.

> AFAIK, Just doesn't have any way to mark "this task is complete, so don't re-run it"

What does a makefile look like for "install this list of dependencies with pip/homebrew/apt, and don't run it twice?"

> It also doesn't have file-based dependendicies, which are pretty common for any kind of programming

C and C++ compilers at this stage are the only places that I work with file based dependencies. Unless you explicitly declare all the headers as dependencies in a makefile, make won't rebuild a cpp file if you change a header. IME ninja (plus cmake to generate it) has entirely superseded make in this space. Tools and languages comes with it's own build tool (see above - go/cargo/dotnet/etc) or state tracking (docker) that break make's assumption of file based dependencies.

> Why would I want to lose all of Make's flexibility and power in exchange for slightly prettier UI syntax?

Because every make file I've ever used in recent memory has been 30% workarounds to avoid file based dependencies, and work around subtle footguns that make has, rather than actually doing what I want it to do - execute a build command.

[nrclark]

> What does a makefile look like for "install this list of dependencies with pip/homebrew/apt, and don't run it twice?"

It looks like this:

    default: your_task
    
    VENV_DIR := venv
    
    $(VENV_DIR)/bin/python: requirements.txt
        python -m venv $(VENV_DIR)
        . $(VENV_DIR)/bin/activate && pip install -r requirements.txt
    
    your_task: $(VENV_DIR)/bin/python
        do_your_thing
    
    clean:
        rm -rf $(VENV_DIR)

> C and C++ compilers at this stage are the only places that I work with file based dependencies. Unless you explicitly declare all the headers as dependencies in a makefile, make won't rebuild a cpp file if you change a header. IME ninja (plus cmake to generate it) has entirely superseded make in this space. Tools and languages comes with it's own build tool (see above - go/cargo/dotnet/etc) or state tracking (docker) that break make's assumption of file based dependencies.

In our Python example above, you have a file-based dependency between requirements.txt and your virtual environment. And then you have a file-based dependency on your virtual-environment's Python for whatever task you're trying to run.

> Because every make file I've ever used in recent memory has been 30% workarounds to avoid file based dependencies, and work around subtle footguns that make has, rather than actually doing what I want it to do - execute a build command.

Why would you want to avoid expressing your dependencies? File dependencies exist, and are the simplest / most correct model for 75% of all build automation. When somebody has to sit down and read your Makefile, isn't it nice to have a way to express "these are the files that actually matter for task X"?

[jgrahamc]

I think this post is well intentioned but misses the point entirely.

    GOOS=darwin GOARCH=arm64 go build -ldflags="-X 'main.Version=v1.2.3'"

Executing this with variables for os/arch/version is not straightforward in a makefile. You might argue that it's a flaw with the `go build` command, but replace go with cargo/dotnet/maven and you have the same problem.

What makes that hard? I think the following works correctly:

    $ cat Makefile
    goos := FOO
    goarch := BAR
    mainversion := BAZ

    .PHONY: build
    build: ; @GOOS=$(goos) GOARCH=$(goarch) go build -ldflags="-X 'main.Version=v$(mainversion)'"*

    $ make -n
    GOOS=FOO GOARCH=BAR go build -ldflags="-X 'main.Version=vBAZ'"

[nrclark]

> I think this post is well intentioned but misses the point entirely. > > GOOS=darwin GOARCH=arm64 go build -ldflags="-X 'main.Version=v1.2.3'" > > Executing this with variables for os/arch/version is not straightforward in a makefile. You might argue that it's a flaw with the `go build` command, but replace go with cargo/dotnet/maven and you have the same problem.

This is pretty easy in Make, especially since GOOS and GOARCH are environment variables. One way to do it could look like this:

    export GOOS ?= darwin
    export GOARCH ?= arm64 
    VERSION ?= 0.0.0+$(shell git describe --dirty --always)

    build:
        $(if $(VERSION),,$(error VERSION was not set))
        go build -ldflags="-X 'main.Version=v$(VERSION)'"

you could run it as:

    $ make build (uses darwin/arm64/0.0.0+<git ref>)
    $ make build GOOS=linux VERSION=1.0.0 (uses linux/arm64/1.0.0)
    $ GOOS=templeos GOARCH=sparc make build VERSION=1.1.0 (uses templeos/sparc/1.1.0)

[nrclark]

    ---- 
    fetch PACKAGE VER:
        @echo fetching {{PACKAGE}} at version {{VER}}
    ---- 
    $ just fetch foo 1.2.3

Here is the Make equivalent, with equivalent error-checking

    ---- 
    fetch:
        $(if $(PACKAGE),,$(error PACKAGE was not set))
        $(if $(VERSION),,$(error VERSION was not set))
        @echo fetching $(PACKAGE) at $(VERSION)
    ----
    $ make fetch PACKAGE=foo VERSION=1.2.3

I agree that the `just` version is a little neater / tidier. But it's not like Make can't do the same thing. And Make is a much more powerful tool.

[bityard]

I like the idea of Just (and the many tools like it), I only wish it had come along a few decades earlier. Make has the advantage of being a ubiquitous de facto standard. It is probably already installed anywhere you want to deploy your repo, or can be very easily installed with one command.

As a personal preference, I actually do tend to avoid Make as a task runner because a quick shell script is almost as easy to write and far more flexible. I am pretty familiar with Make but have still painted myself into a corner with it more times than I'd like to admit.

[mitjafelicijan]

You are not alone in this. I have found myself stubbornly writing a complex Makefile when I should have just written a simple shell script and be done with it.

[TylerE]

Any tool that treats spaces and tabs differently is not sane.

[nrclark]

Agreed, I don't like that about Make at all.

But also it's never something I have to worry about. Pretty much every syntax-aware text editor knows how to handle Makefiles, and knows that recipe steps start with a tab (unless you change Make's default). So for me, that falls in the "ugly, but not really a usability issue" bucket. And Make's power/expressiveness is worth accepting a few "I'd have done it differently" syntax things.

[TylerE]

It always seems to bite when I've got nothing but an SSH link and a default vim install to hand.

[nrclark]

lol that's fair enough. This is a tangent and unrelated to Make - but I also find myself in that situation pretty frequently, since I do a lot of embedded systems and some CI/CD work.

If you find yourself in that situation frequently, I'd recommend one of:

    1. Use sshfs to mount your remote target locally on your dev machine, or:
    2. Use vscode's remote-ssh mode (if the agent can run on your target).

Vscode's remote-ssh mode is really nice. It's good enough that it single-handedly made me switch from my previous editor, and I wish that I knew more tools like it.

[woodrowbarlow]

people often turn to make as a taskrunner, but i've never understood why. i've heard people say "i don't want to add another tool/language/dependency to my project, so i'll just use make" but, usually, `make` is a new dependency for the project. `make` (realistically) excludes Windows users, most Linux distros don't ship it by default, it is a unique syntax (no, it's not shell). even in a C project, it's not a given that Make is already a dependency. in languages like python or javascript, with rich package ecosystem, `make` makes even less sense; why limit your users to one OS, and why not write your tasks in your project's native language? as a taskrunner, `make` has so many limitations -- because that's not what it's designed for. `make` is good for tracking dependencies between files that are generated from other files, and that's about it.

a good taskrunner makes it easy to run a task while still exposing the tool's underlying flexibility. a good taskrunner lets me invoke the entire test suite with a short command, but also allows me to add custom options and arguments to, say, run a specific test case in an alternate environment.

`make` fails to expose the tools' underlying flexibility. sure, you can write a .PHONY target to run the full test suite, but `make` can't handle passing options or arguments (besides cumbersome Makefile variables).

a makefile tends to obscure the underlying tool, enshrining its launch arguments. anyone who has tried to cross-compile a makefile codebase authored by someone who didn't consider cross-compilation understands what i mean (you'll end up re-writing the Makefile 90% of the time).

[maccard]

I think the thing is that most people do have make installed, and if not it's on every package manager ever. It's a language agnostic tool that can bootstrap the language/environment. For most use cases as a task runner, the differences between various flavors of make don't matter.

I agree with _everything_ else you've said though.

[woodrowbarlow]

i agree that it's common to encounter developers who already have `make` installed, and developers who feel comfortable enough to read/edit a Makefile. however, i think most of those are merely leveraging their knowledge of shell scripting rather than knowledge of Makefile language (which is similar enough to have scary pitfalls) -- so i'm not sure i agree with "language agnostic". it's an extra language. i wouldn't necessarily expect (say) a frontend developer to have `make` installed, or even be comfortable with shell syntax.

[maccard]

I think you’re right about leveraging their shell knowledge, but I think that’s the reason make works. It is just enough shell that it’s grokkable.

I think it’s fairly likely that a FE dev would have make installed, but also that it’s likely easier to install make than it is to install go or dotnet on a specific version, and then use a makefile to install the dependencies to run the backend locally.

[zokier]

make is part of POSIX; if you assume already POSIX environment for your software then make is not an additional dependency. And that also indicates that make is in no way Linux-only, it is widely used on BSDs and UNIXes of all flavours.

I don't like make for various reasons, but portability is not one of them.

[woodrowbarlow]

ahh -- a fellow C coder, i assume. :)

i admit my tirade is a little less applicable to C projects; many C projects are already using `make` for legitimate reasons, and at that point are already making some POSIX-y assumptions. so if that project's taskrunner needs are light, it really might make sense to just cram them in the Makefile and mark them .PHONY, despite make's shortcomings as a taskrunner.

my portability comments were really written with windows in mind.

[mitjafelicijan]

You make valid points. I would, however say that I primarily use make as a task runner for simple projects. If I have a "larger" project, at some point I would also probably opt for something closer to that language to squeeze out as much as possible.

And no tool is perfect. I think it's all about balance. Example: I love Zig build tool, but I think it would scare some of the people who would just want to add something to the build pipeline and have no idea about Zig. Projects are rarely just the code they have. They have scripts and require installing additonal software.

But yeah, I agree with your points here.

[wodenokoto]

Then what is a good task runner?

[maccard]

Not OP but whatever your language/ecosystem is written in. If you're in python use python, maybe with just. If you're in ruby use rake. If you're in a polyglot environment, pick the lowest common denominator.

[AndyKluger]

For Python projects, I like to use Taskipy as the UX entry point. If any task itself is more than a one liner but doesn't need Python, I'll have it invoke a shell script from a mk folder. If any task itself requires an activated venv, I'll define it in a noxfile.

What it might look like in a pyproject.toml:

  [tool.taskipy.tasks]
  fmt = "nox -s fmt"
  lock = "nox -s lock"
  install = "if [ $VIRTUAL_ENV ]; then pip install -r local-requirements.txt; else printf '%s\n' 'Please activate a venv first'; return 1; fi"
  test = "nox -s test test_without_toml typecheck -p 3.12"
  docs = "nox -s render_readme render_api_docs"

For file generation tasks (the kind of stuff make is good at), I've been using Tup more and more. But recent versions aren't always packaged for distros, so I'm hoping it gets an asdf/mise plugin.

[woodrowbarlow]

`just` is great, especially if you're using rust. if you're in python or javascript, it's well worth it to use a language-native runner (like pydoit / grunt). that way your "task" is just a function, it can take arguments (and you can define default values), you can even use something like `*args, **kwargs` to take any options/arguments, and then just pass them verbatim onto the subprocess.

[AndyKluger]

I'm also a big fan of mise for managing development runtimes, and while I haven't tried its own task runner functionality, it does feature some: https://mise.jdx.dev/tasks/

[smartmic]

Thanks, that is a good idea. I also prefer GNU Make for automatization if the number, diversity or dependencies of the tasks would exceed the benefits of a simple bash script.

I stumbled over this statement in your README:

> It is recommended to use .PHONY for targets that are not actual files. In the example below I am not doing that though, but it is wise to follow that rule.

Please also give the example as you think it would be correct. Or write the understandable reason why you don't do it here. I think many people don't read the text between the code snippets but only scan the code examples. It would be helpful to find the ‘best practices’ right there.

[mitjafelicijan]

You are correct. I will add this. I was not clear enought on this.

[Galanwe]

> I know this could be considered blasphemous, but I constantly find myself using Make as a task runner

It's not, don't apologize, reusing battle tested tech is the way.

I have a quite close "template Makefile", though I prefer to store it as a cookiecutter (or more recently "copier") template for easier per-project bootstrap.

You could add a timestamped log function. I use that in mine so that targets have timestamped messages pre/post.

[mitjafelicijan]

This is a very good idea. I will add that. Thank you.

[keypusher]

May want to look at Just. It is heavily inspired by Make and shares much of the same syntax, but removes a lot of the workarounds necessary to use Make as a task runner and adds a few other features.

https://github.com/casey/just

[nrclark]

If Just grew to have a large enough feature-set that it could replace GNU Make, it would collect an equal amount of warts and sharp-edges along the way.

It's a little more user-friendly, but not enough to justify how much power you lose.

I spent the last 20 minutes reading Just's documentation, and it seems like the only real wins are nicer argument parsing and && dependencies, and the --list argument. And in exchange for that, you lose file-based dependendicies and Make's powerful templating engine. Did I miss some other features that make the trade worth it?

[zokier]

For lots of uses not having file-based dependency mechanism is a benefit, not a con. Sometimes less is more

[nrclark]

For sure. But you also don't have to use file-based deps if you don't want. Make doesn't require it in any way. You can mix-and-match.

[rgoulter]

File dependencies are useful as part of a build system.

But often tasks are a step removed from that. Targets like "all" and "clean" are good examples of useful tasks to have, even if they're unrelated to files.

A "task runner" isn't intended to replace all of a Makefile's functionality. It takes a common use case of Makefiles, and improves the user experience for this.

I think it's fair to say that it's not a big step up; but it's a definite improvement for what it does aim to do.

[nrclark]

Targets don't need to produce files in Make (see: your examples of 'all' and 'clean'). And non-file targets can depend on other non-file targets. Make does this out of the box, by default.

But if you ever get to a point where you _want_ to express some source-code dependencies (say: pip-install depends on requirements.txt), you can also do that in Make. Just doesn't provide any mechanism for it.

[bigstrat2003]

I don't want file-based dependencies. I don't understand why anyone would, frankly. Just is an upgrade in that respect, not a downgrade.

[nrclark]

If you don't want them, don't use them. Make doesn't need you to produce actual files called 'clean', 'build', etc. you don't even have to do anything special unless your build folder has actual files called those things (in which case, you could declare them as phonies).

From Make's point of view, they're just target names. Make doesn't care whether or not a target produces any files. Targets can satisfy dependencies for other targets, as can files on the filesystem.

[taeric]

I'm intrigued. Without the file based dependency stuff, is it all incumbent on you to add the logic checking and stating if something has been done?

[kstrauser]

There are so many things where that’s not relevant. “Just start” or “just test” or “just install” are idempotent, or at least stateless (assuming “start” calls out to systemctl or such).

[rgoulter]

> ...idempotent, or at least stateless...

Even for a Makefile, "start", "test", "install" are likely to be phony (and not literally refer to files with those names).

The execution of a "start" task defers the role of "has it already been done" to other places.

One way of describing "task runner" is "nicer UX for .phony targets".

[mitjafelicijan]

Just is amazing. I used quite a lot. It is however difficult sometimes to convince others to use it, even though it is a no-brainer.

[crabique]

Biggest showstopper for me personally is Make's inability to properly pass through arguments from `make run <arbitrary number and type of arguments>` to the underlying program.

Some of the issues could be avoided by requiring a -- and using the .SILENT modifier, but some are extremely difficult: e.g. it word-splits strings and you can't just pass "some string" as a single argument, everything will be separated by spaces.

In case the underlying program is a script-language wrapper that you also control, it is possible to hack around and let Make pass through its own $$PPID so that the underlying script could read the nul-terminated /proc/<Make's pid>/cmdline and pass it on to the actual program.

This, of course, only works where procfs is a thing, so e.g. on macOS (if it's among your target platforms) you'd have to reinvent the wheel and learn a whole lot of Darwin sysctl dark magic to read arbitrary process' arguments, at which point you'd ask yourself if this is even something a sane person would ever do...

[0xbadcafebee]

I've never found the need for a task runner since learning shell scripting.

A lot of people recommend just; here's their sample file:

  alias b := build
  
  host := `uname -a`
  
  build:
      cc *.c -o main
  
  test-all: build
      ./test --all
  
  test TEST: build
      ./test --test {{TEST}}

Here's how I would do that with shell:

  #!/usr/bin/env sh
  set -eu
  
  b="build"
  
  host=`uname -a`

  _cmd_build () {  # build:       Build main program
      cc *.c -o main
  }
  
  _cmd_test_all () {  # test_all:    Run all tests
       _cmd_build
      ./test --all
  }
  
  _cmd_test () {  # test TEST:   Run a test TEST
      _cmd_build
      ./test --test "$1"
  }
  
  _cmd_help () {
      echo "Available targets:"
      grep -E "^_cmd_[^[:space:]]+ \(\) {.+" "$0" | sed -E 's/.*#/  /'
  }

  [ $# -gt 0 ] || _cmd_help
  set -x
  "_cmd_$1" "$@"


  $ chmod 0755 run.sh
  $ ./run.sh
  Available targets:
     build:       Build main program
     test_all:    Run all tests
     test TEST:   Run a test TEST
  ./foo.sh: line 23: $1: unbound variable
  $ ./run.sh build
  $ ./run.sh test_all
  $ ./run.sh test foo.t

But this example (from just's homepage) is clearly a build process, so I would use Make for it instead. When I need to be able to run individual sets of commands with arguments and options, I make a shell script. Many more features available, more flexibility, I can tailor it to my use case.

[kstrauser]

Two things Just gives you beyond that:

- It handles all that boilerplate for you. The only stuff in the justfile is code you want to execute, not the code to figure out which code to execute.

- Out-of-the-box tab completion. "What did I name that recipe? `just t<tab>` Oh, `test-all`, that's right!"

I like not reinventing those wheels. Let someone else manage that hassle for me.

[fmajid]

remake is a more powerful tool, a fork of GNU Make that adds a debugger and profiler, among other extensions.

https://remake.readthedocs.io/en/latest/

[simonmic]

I used make as a task runner for probably 30 years; using all the tricks to do most of the things you want in that use case (arguments, help, portability, reliability..). I worked around all the idiosyncracies continually. That's enough time sunk into wasteful friction for me. Now just is here, and the people pushing it are right. For this job (task/script manager for more than trivial commands) it's better enough and reduces cognitive load enough that it's very often worth the install requirement and new learning curve. In time it will be more installed-by-default and I'm certain it'll acquire some form of make-like dependent building as well.

[Apologies for contributing to the slightly off topic discussion. makext sounds great for people still using make for this and I totally would have used it in the past.]

[mitjafelicijan]

I totally get it. I know the feeling. I have written some complex Makefiles in the past for C projects that looked completely horrid.

For smaller projects I like make and people who work with me and know me always first check Makefile to see what is there and what the project is about. I use it as a launchpad for installing dependencies and things like that.

And it's all about what is appropriate for that project. Something a shell script is better. Sometimes soemthing else. So I completely understand what you are saying.

[simonmic]

Appreciate you saying so mitjafelicijan. You're right, it's all about what is appropriate for the project at a given time.

[simonmic]

For anyone interested (though to compare you really need to work with them over a period of time):

Here's a fairly simple thousand line Makefile: https://github.com/simonmichael/hledger/blob/2d35b1051/Makef... (and https://github.com/simonmichael/hledger/blob/2d35b1051/Makef...)

that was converted to a Justfile: https://github.com/simonmichael/hledger/blob/43c93eb37/Justf...

And here's a more powerful kind of makefile using a full programming language: https://github.com/simonmichael/hledger/blob/43c93eb37/Shake...

And here are two multicommand shell scripts:

https://github.com/simonmichael/hledger/blob/43c93eb37/bin/f...

https://github.com/simonmichael/hledger/blob/43c93eb37/bin/t...

that were converted to a justfile: https://github.com/simonmichael/hledger/blob/43c93eb37/bin/j...

[blacklion]

I wonder, why GNU make is much more popular than BSD make/pmake.

Both are extensions to classic make (makefiles without advanced features works with both implementations), but, to my eye, pmake extensions is much more simpler and nicer to use, than GNU make extensions.

Simple ".if/.else/.endif" and true ".for" loop, like in sh is much more comprehensible than GNU make functional extensions to me.

Is it only me?

And BSD systems have extensive library for pmake already, which allows to build programs, libraries (both shared and static) and more in declarative way, without any custom rules. It is like these extensions, but battle-tested for 20+ years. With nice license, too.

[barries11]

Nice, I will definitely use these techniques.

Here are a few settings that make GNU make a bit faster, and enable multi-line, strict-mode bash scripts as recipes (make recipes are normally sequences of single-line sh invocations)--and enable quietude & tracing: https://github.com/barries/polling_state_machine_cpp/blob/ma...

[mitjafelicijan]

This is really nice. Thank you for this!

[wodenokoto]

A while back someone linked to this relevant comment on HN, about using make as orchestration: https://news.ycombinator.com/item?id=32441602

I also believe I've seen someone setup tools that can draw the dependency graph for you, so you can view (not edit) it airflow style.

[pama]

Thanks! Typically I don’t like solutions that involve extra environment variables as things inevitably get messy over time, but your use cases are clean, simple, and common enough that it’s nice to have them documented in one place in this extra makefile.

[jjgreen]

I saw MEX_LICENSE and thought: What, this requires a Matlab licence?!?

[mitjafelicijan]

I did not knew they use this naming convention. :) I should probably choose something else :)

[jjgreen]

Matlab mex files are what they call their plugins ("Matlab executable" I guess), so they're just .so files on Linux etc; I don't think they actually use "MEX_LICENSE" as the envar for their keys, but it's feasible that they might which is why it jumped out at me ...