Show HN: Jeeves – A Pythonic Alternative to GNU Make

[yeti-sh]

Write a Python file named `jeeves.py` in your project directory with contents:

    import sh

    def lint():
        """Lint your Python project."""
        sh.mypy()

This, together with

    pip install jeeves-shell[all]

makes it possible to do

    j lint

…which will run mypy for you, and, via the omnipotent `j` command, open ways for

• Automation of routine tasks,

• Standardization of your projects,

• Implementation of best practices,

• And more :)

Github: https://github.com/jeeves-sh/jeeves-shell/

[theamk]

Note the examples use "sh" python library, which by default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout.

Those are really bad defaults in general, which is why I recommend everyone to avoid this library. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!)

But it is _especially_ bad for general tools runner. Because of forced capture, you are not going to get any output or errors until process completes, and maybe not even then. Also interactive prompts won't work, errors would be reordered compared to regular outputs and so on...

I can't say anything about jeeves, but at least avoid "sh".

[yeti-sh]

I like sh for brevity of its API. I often have to use `_tty_out=False`, but this is easy to fix once and for all commands in a script:

    my_sh = sh.bake(_tty_out=False)
    my_sh.do_whatever()

The way how sh captures output can apparently be altered, say, by providing a callable to _out argument.

[blacklight]

You could have probably created your own little wrapper on top of subprocess.Popen and dispatch stdout and stderr around. No need for an external library with bad tty/piping defaults just because it has a nice API (which needs to be tweaked with _tty_out=False anyway if i want to pipe the output to another command.

Btw does rich scrape the special formatting characters if piped?

[yeti-sh]

An example from the top of my head:

    compose = sh.docker.compose.bake('-f', 'deploy/dev.yml')
    …
    compose.down()
    …
    compose.up('--force-recreate', '-d')

I feel this is a major improvement on top of Makefiles + shell commands in them. Nice API _matters_; it is ergonomics and therefore productivity.

You can specify `_out=rich.print` and it will work but AFAIK it won't scrape ASCII terminal formatting characters.

I had to fix these characters in my `sh` based scripts but do not consider that as a big deal.

[theamk]

Doesn't the snippet above hide the errors and warnings (since I do not see any code to print them)?

Does not seem very ergonimic to me.

[yeti-sh]

No it does not, in fact; errors or warnings will pop up as an unhandled exception and print:

- command actually executed,

- snippet of stdout

- and snippet of stderr.

They can be handled using standard exception techniques.

[Brian_K_White]

This is not an alternative to make* any more than some random half-baked build.sh is.

* gnu or otherwise, and why specify that in the first place anyway? If it did actually do a suitable job of replacing gnu make, would it not also suitably replace bsd or any other make? Is a person who says something like that a good source of ideas so deep and quality that they probably do improve upon those who created unix?

make may not be the final build system for the rest of eternity, but all this is, is "I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python."

[yeti-sh]

Saying 'GNU Make' particularly was an automatism. When I use Make I do use GNU Make specifically. I do not have experience with BSD Make, or other Make dialects.

> did actually do a suitable job of replacing gnu make

The title does not say _a replacement_, it says _an alternative_. I never intended to replace GNU Make; if I said that anywhere — that was a mistake on my part.

> do improve upon those who created unix

I did not intend to improve upon UNIX ideas or philosophy. The improvements I am aiming for are:

- developer experience,

- conciseness and maintainability of the code.

…and these are being addressed for a narrow use case. For that use case, in my experience, Make is very often used.

I argue that within the bounds of this use case, this is an alternative which can be of use to improve productivity and everyday experience.

> I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python

I classify these assumptions about my understanding or misunderstanding as ad hominem and unprofessional.

[zvr]

The issue is that make(1) has lots of functionality... none of which is present here.

In other words, make(1) does a lot of things, one of which is to run shell commands. If this the only thing you care about and your project offers, it would be better expressed as "Pythonic way to run commands" or something, with no mention of make.

[yeti-sh]

To change the wording in the title of this post would be improper in view of multitude of comments referring to this title. I will consider rebranding this.

[Brian_K_White]

I appreciate and thank you for the integrity of not wanting to edit something after it's been referenced.

[arcanemachiner]

Reminds me of `just`. I love `just`. (Although I've never used make...)

https://github.com/casey/just

[ospider]

The problem of `just` is that it does not consider `mtime`, which makes it almost useless in my case. I do hate the quirky syntax of `make`, however, it does a great job of not repeating itself if files have not been changed.

[ttymck]

Well I don't think "just" is a build tool. It doesn't consider mtime because it has no reason to.

[benatkin]

I looked through just again and it seems like a nice design, a little complex but powerful :) I might start using it.

Edit: played with just and got confused about how it works. Not going to use it just yet.

[arcanemachiner]

I use it as a little dictionary of commonly used commands in my git repos.

No more remembering commands to build a release or deploy a docker image. Now it's 'just release' and 'just deploy'.

[yeti-sh]

Thanks for the hint, I didn't know about this project; will check it out.

[Animats]

They're thinking macro, not dependency graph.

The point of "make" was supposed to be that it defined a dependency graph, then executed only the commands required to achieve the goals. Somewhere that got lost.

[dikei]

Remind me of

* Fabric: https://www.fabfile.org/

* Invoke: https://www.pyinvoke.org/faq.html

[geenat]

RIP fabric. Makes me sad how fabric v2 jumped the complexity shark.

v1 was so much more simple and elegant.

[dikei]

Apparently, Fabric v1 is replaced by Invoke now.

[yeti-sh]

I looked at pyinvoke before I started jeeves. Roughly, as much as I can recall, there were a few reservations:

- no type hints for docs & validation, I wanted them

- Makefile in its basic form is very concise and doesn't require a @task decorator, I didn't want it either

I didn't need dependency graph much.

[creshal]

> I didn't need dependency graph much.

So you made a Make alternative that isn't a Make alternative because you never needed Make in the first place?

[yeti-sh]

There are a lot of projects, both commercial and OSS, which have a Makefile for linting, version releases, etc, — and never use its dependency graph feature; or, their usage is so superficial that implementing it, say,

- via direct function calls in Python code,

- or using Typer callback functions

…is easy.

There are potential approaches worth exploring (annotations, decorators, …). I might come up with a list of options with syntax examples and ask for community's thoughts about it, — but at this point jeeves, as an MVP, is already useful for me and a few people I work with, and that motivated me to share it with the community.

[mypalmike]

The whole purpose of make is the dependency graph.

[kelnos]

I don't really get the comparison to makefiles. The idea of make is that you express rules that lay out dependencies between files and other rules, and make (when given a target to build) figures out which rules do and don't need to be run based on what you want to build and what is and isn't up to date.

Jeeves seems to be a thing where you define python functions and then Jeeves makes it so you can run them as 'subcommands' of the 'j' command line program, and to take arguments, etc. Seems like it'd be up to you to write your own code to handle figuring out if each function/subcommand has dependencies, and if those dependencies need to be run or built.

Neat project, but has nothing to do with make. Sure, some people may use make as a command runner, but that doesn't mean this is a make replacement. At least not for the vast majority of things that people use make for.

[phil-martin]

The most comprehensive make alternative in python I've seen is Scons (https://scons.org/)

It would be worth to see how they tackles some of the challenges you're looking into.

Blurb from the website:

SCons is an Open Source software construction tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.

[chasinglogic]

While I can't contest that SCons is comprehensive, I would never recommend it as a source of learning "what to do".

SCons is not idiomatic Python and it abuses things like `eval` which gives it terrible performance.

Source: I used to work for MongoDB and my full time job was to make SCons faster, which I eventually did by making it a Ninja generator (which has now been upstreamed). But the code is still pretty bad.

Using SCons however is much nicer than using make / autoconf IMO, especially now that you can farm the builds out to Ninja.

[phil-martin]

Oh definitely agree on the non-idiomatic python. I was thinking less of "how to correctly implement and write the python code" and more of the "learn the history and decisions on why it might do things, and the user experience of defining a build in python". I really liked its consistency of repeatable build steps, the way it handled it's dependencies, and accepted the tradeoff it presented for slowness. But its been perhaps 10 years since I looked it, likely longer, and rose coloured glasses are applicable.

And nice work making SCons faster!! not at easy thing to do at all.

[fero14041]

An alternative to Scons could be Doit (<https://pydoit.org/>), which if I remember correctly was built as a faster alternative to Scons. See also reasons of some users to prefer the later to other mentioned here: <https://pydoit.org/stories.html>.

[pasc1878]

Also see waf https://waf.io/ which is a similar build system as scons to replace make including dependencies

[culebron21]

I took a look at it in 2017, decided to stay with Makefiles.

[a-dub]

i don't think you can call this an alternative to make, it doesn't seem to do any dependency processing (phony or file based) at all!

[duped]

How does it compare to waf (1)?

(1) https://waf.io/

[yeti-sh]

I've never used waf, but from its documentation it would seem that waf is a build system, implementing a concept of a DAG governing build of a project. That implies that waf is rather complicated.

jeeves, on the other hand, does no such thing. It is a command runner: you write a function → it is converted into a shell command.

I believe jeeves is easier to get started with, partially due to its simplicity and partially because it follows Python standards. For instance, waf uses a peculiar syntax to define command parameters:

waf employs a peculiar method to configure command options:

    ctx.add_option('--foo', action='store', default=False, help='Silly test')

while jeeves relies upon function arguments and type hints (as it is based upon Typer):

    def do(foo: Annotated[Option(help='Silly test')]):
        …

While jeeves is easy to start with it also promises ability to scale: with modular packages for commands, subcommands, and installable/shareable plugins.

[llimllib]

Make is a build system, not a command runner, so you should probably change your copy; Jeeves doesn’t do what make does

[yeti-sh]

I wouldn't necessarily agree. I've seen may instances of Make particularly being used to run tedious commands — linting, testing, deployment, et cetera; for these things, jeeves can be a perfect replacement. A and B are usually called alternatives if they have an intersection in their feature sets; there are no perfect alternatives.

[wavemode]

That make can be used to run commands that don't build anything is just an artifact of how targets work.

If something is marketed as a make alternative, most people would expect it to, at the very least, possess make's core feature, which is skipping targets that have already been built and whose inputs haven't changed. That's what makes something a build system and not just a command runner.

[llimllib]

The fundamental feature of make is building a graph of dependencies and building only the files that need building.

I say this as somebody that built their own python replacement for make long ago - during the building of which I learned what make was actually for, and how to use make, and thus abandoned my own project.

[1]: https://github.com/llimllib/pub

[ratboy666]

thanks -- I was under the initial impression that you intended jeeves as a "pythonic make". make isn't a command runner at all. Indeed, it relies on sh for that. It determines the DAG required to bring components up to date wrt dependencies. For example, you may have a postscript file and a pdf. Edit the postscript and you want the pdf regenerated:

my.pdf: my.ps

<tab> command to convert my.pdf to my.ps

Make makes no attempt to generate scripts, or anything like that. In general, make uses file timestamps to determine that my.pdf is out of date with respect to my.ps Make does have default commands that it can use. For example, it knows that executable w can be generated from w.c If I have a directory that contains w.c I can:

make w

cc w.c -o w

Now, because w exists, and is newer than w.c doing make w again does nothing!

make w

make: 'w' is up to date

I don't think that is what jeeves is?

[yeti-sh]

You are right, jeeves doesn't do that. Your example reminds me of how I first learnt to use Make; I built my LaTeX stuff from source when in university.

However, dealing with Python mostly in my time in the industry, I scarcely ever had the need to use these particular features of Make. I used it as a task runner, guilty of that, — and that's what I wanted to reimplement in Python.

Even writing in Rust, everything I have to do is `cargo build`, — there is no need to write Makefiles for building the project.

I will consider changing that wording to set proper expectations, but I still feel that for me "a Pythonic alternative to Make" perfectly conveys the use case I daily employ this thing for.

[chucksmash]

Don't know anything about waf, but the `ctx.add_option` example you've posted looks like it takes very similar parameters to add_argument in argparse[0].

[0]: https://docs.python.org/3/library/argparse.html#quick-links-...

[yeti-sh]

Thanks for the point. I've had to deal with argparse-based code (not finding it very enjoyable) but didn't recognize the resemblance.

[duped]

I would also echo other commenters to stop comparing to Make if you do not have any intention of supporting DAGs or targets, which is indeed the entire point of Make. The fact you can use it as a task runner is a side effect/special case.

[benatkin]

I wouldn't want a stray python file, whether I was using in it a non-python project or a python project, to trip up editors.

I think Earthly has the right idea with an Earthfile. https://docs.earthly.dev/docs/earthfile

It's a different tool for running commands, though.

[yeti-sh]

You can also install a jeeves plugin with pip. Say,

    pip install jeeves-yeti-pyproject

will provide you with jeeves, a bunch of commands, and a pack of dev dependencies which I personally happen to like and to use in my projects.

I don't believe Make has plugins.

[benatkin]

That's some more indirection, besides putting a disconnected python file in the project directory.

I'm not sure whether it's suggested to install it globally, as a development dependency with poetry/similar, or with pipx https://pypa.github.io/pipx/

The "import sh" thing could have some users installing this package https://pypi.org/project/sh/

This in addition to it being known as "j". It has at least 3 names, jeeves, j, and sh.

[yeti-sh]

For a Python project, I'd recommend doing this as a dev dependency.

* jeeves is the name of the project which I'm advertising, which converts a Python file to a set of commands; * `j` is the name of executable which aformentioned project exposes; * and `sh` is the library (the correct link to which you have provided) which is an optional dependency of `jeeves` and provides a more convenient interface to calling processes and executing commands from Python than `subprocess.run()`.

[shepherdjerred]

Earthly is fantastic and I would recommend it, but they really need to improve sharing cache.

[culebron21]

If this is a translation of functions into CLI commands, there are enough of such crates. Old package `argh` and now `yaargh`[1] also translate functions into CLI commands in a very neat way, with just a decorator, and they don't need an external executable in the system to run, just `python your_file.py`.

Makefile keeps dependency graph. I had a 100-entry 300-line Makefile, with graphviz drawing charts of it, and kept a huge, year-long project on my own, organized and running all partial updates smoothly.

[1] https://github.com/ekimekim/yaargh

[yeti-sh]

That's right; Google also published a similar package named `fire`, I used it a lot.

> just `python your_file.py`

That's actually one of the points.

    j lint

is shorter than

    python scripts.py lint

and the letter `j` is what the index finger of your right hand is pointing to on the keyboard. There is usually a tactile bump on that key. It is something you can type very quickly. You can also install shell completion to improve the experience further.

Ergonomics matters.

[culebron21]

Well, you can do the same trick with argh & a symlink in /usr/bin.

I'll agree this is some improvement in ergonomics if the executable supports tab completion.

With makefile + argh I can do this: put the script as a dependency of data file, and rebuild it if code changes.

    P=python3
    my_data_file.csv: my_script.py input_data.csv
        $P --flag1 --flag2 $^ $@

With your package, this becomes harder -- the file name and command name now are separated and must be checked for being in sync.

So, the package adds some features, but does it at some cost of other ways of interaction.

[yeti-sh]

Regarding sync checks which Make has as a built-in feature, I am not yet positive how to best implement it. Maybe something like

    @pre(file_name=_file_is_in_sync)
    def build(file_name: Path):
        ...

I believe such libraries do exist even, but I haven't yet researched the topic; if jeeves proves to be useful for the simplest use case then it will make sense to expand its scope.

[yeti-sh]

> argh & a symlink in /usr/bin.

- Is putting a symlink to one of your project directories, probably residing in $HOME, to /usr/bin/ a good practice?

- As an alternative one can put a symlink into ~/bin; but what if you have multiple different projects?

- And they can have different jeeves commands and different plugins installed. For instance, `j lint` implementations in different projects are work on are completely different.

Because these are different projects.

Rather, I prefer to have jeeves automatically create the executable command in each virtual env, and its behavior in different virtual envs will be different.

[culebron21]

I mean `sudo symlink /usr/bin/python3 /usr/bin/p` can make a good shortcut for python. The script name can be tab-completed.

[yeti-sh]

I see. The P key isn't that ergonomically placed though; and again, one will have to do this time and again for each virtual environment, right?

Why not use native Python methods, namely endpoints, instead, and automate this away?

[linsomniac]

This looks really interesting and clever, I'm going to have to try using it for some of my python utilities. I really like how it handles arguments using click-like semantics.

In some ways this is similar to a project I've been working on for the last month that is kind of a pythonic reimagining of Ansible and Cookiecutter. It's still a WIP, but I'm starting to successfully convert my YAML-style declarative snippet/cookiecutters into Python, and I'm working on improving the documentation now.

Where Jeeves seems to excel at collections of small "scriptlets", uPlaybook is targeting declarative automation of larger tasks, including templating of configuration files, triggering restarts on changes, etc...

https://github.com/linsomniac/uplaybook

[pasc1878]

Try the existing invoke which is a python task runner https://www.pyinvoke.org/

[linsomniac]

invoke is closer to Jeeves in that you define your tasks in a Python file. uPlaybook is an experiment investigating: What if Ansible had Python syntax rather than YAML? So it focuses on templating config files and declaring system state.

[travellingprog]

Another alternative that I have liked for replacing Make as a task runner is Task: https://taskfile.dev/

[ketanmaheshwari]

Similar work: https://snakemake.github.io/

[yeti-sh]

Thanks, will check this out!

[marcosnils]

TIL about Jeeves, things like something that could play very nice with the Dagger python SDK (https://docs.dagger.io/sdk/python). Particularly since Dagger could help with bootstraping the python runtime and providing caching capabilities out of the box.

[tpoacher]

Note that contrary to popular belief, there's nothing stopping one from declaring python as the main shell in a makefile.

[geenat]

How's the SSH interoperability?

Transparent SSH interop is why I stick to Bash rather than moving to a Python solution.

[yeti-sh]

Please elaborate. Do you mean submitting commands via ssh to remote servers from the script?

The way I'd recommend to run shell commands from jeeves is `sh` library. It has features for ssh support: https://sh.readthedocs.io/en/latest/sections/contrib.html#ss...

[Arch-TK]

Looks like this has nothing to do with make. Where are the depedency graphs? Where are the suffix and pattern rules?

[yeti-sh]

Indeed, they are out of scope. But I still would argue this does have something in common with Make — just as Make, jeeves can be used as a command runner, in which role Make is oftentimes used as well.

Instead of `make` as entry point for all project-specific commands (like `make lint`, `make build`, `make deploy`), users can rely upon `j` — both locally and CI.

Regarding your particular points:

* dependency graphs aren't presently at scope, I haven't yet come up with the method of expressing them in Python which would have entirely satisfied me;

* I am not so sure I would want to implement pattern rules due to complexity they bring about. Maybe just writing explicit Python code would be enough for cases where they're used. No strong opinion though.

[culebron21]

Having worked with Makefile and pushed it to limits, I'd say there are some deficiencies in the system you may try to tackle:

1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.)

2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work.

3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part.

4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction.

5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph.

If you manage to get any of these right, you'd be praised.

[shoo]

i agree. make is a tool for defining recipes to derive target files from dependency files. archetypal use case would be compiling and linking a bunch of C source and header files into an application binary. make isn't specialized to building C programs, make can also be used for other file-driven computation tasks such as defining data processing pipelines / graphs of processing that consume data files and produce data files.

the rough essence of make is arguably:

1. there are targets, and recipes (written as a shell command) to derive the target 2. targets can express dependencies on other targets 3. targets are files in the filesystem. 4. add a bunch of language features on top (pattern rules, automatic variables, built-in functions, etc) to make it easier to write and maintain the recipes

Another way of thinking about make is that it is a a way to define and evaluate gigantic pure-functional expressions, where each input and output file is a file, and where intermediate sub-expressions can be cached.

for an alternative to make that supports 1+2+3+4, see e.g. https://gittup.org/tup/index.html

for an alternative to make that supports 1+2+3 but explicitly does not aim to support 4, see e.g. https://ninja-build.org/manual.html

another alternative to make that supports 1+2+3+4 but de-emphasizes 3, "targets are files", from the ruby community, is https://github.com/ruby/rake

these days in projects it is quite common to see make used to do 1 but not 2, 3 nor 4. i.e. a makefile as a way to define a bunch of imperative shell recipes for named targets, where each named target has no relationship to a file, and dependencies between the targets aren't expressed or relevant. as a random example of this, here's the first google result searching for "makefile terraform":

https://github.com/paulRbr/terraform-makefile/blob/master/Ma...

note all the targets are ".PHONY" targets, we're telling make to disable the default behaviour where every target corresponds to a file in the file system -- to turn off "3 targets are files in the filesystem". there's very sparing use of dependencies. so this is an example of using makes support for 1 defining recipes to produce targets and a little splash of 2 dependencies between targets, but not 3 nor 4.

it looks like this "jeeves" project is aiming at this latter use case, not attempting to implement support for 2+3, the core of how make is typically used as a build tool

[culebron21]

You should add

5. allow partial rebuilding of targets

When I checked some build systems several year ago, many turned out to re-build just everything.

[yeti-sh]

That's right. For projects which only use subset of Make features described by (1) jeeves can be considered an alternative for Make, and I would believe there are quite a few such projects.

[LeonB]

It’s an omission and serious bug for this thread to not already contain the following comments:

“Why would I consider this, given that make is perfect”

And

“Doesn’t eMacs org mode already do this?”

—

Seriously, looks nice.

I’m a big fan of Jeeves and Wooster novels and I am a little concerned that Reginald Jeeves would not like to be portrayed as a kind of murderous robot. He’s far too civil to execute people (though he did use a kosh to knock out a police officer on at least one occasion).

[haolez]

I wouldn't say make is perfect, but it's different from this kind of tool. Make is more geared towards file creation that depends on complex pipelines. It's not a task runner. In fact, you have to force it to be a task runner with .PHONY if you really need it to.

[yeti-sh]

With interpretable languages, this feature of Make is IMHO rarely called for. Task running is; Makefile still is a standard way to do project maintenance, both locally and CI. jeeves is an attempt to extract that feature of Make, and provide it in a more accessible way.

By making .PHONY and .ONESHELL labels history, as a particular example.

[kelnos]

> “Why would I consider this, given that make is perfect”

Not even that; this doesn't seem to implement any of make's core features, so it's not actually a replacement for make. It's a replacement for shell scripts.

(And no, I don't buy that the comparison is apt because some people use make as a command runner. Sure, they do, sometimes, but that doesn't mean jeeves is a make replacement. It's not.)

[yeti-sh]

Thanks!

Perhaps this particular robot is running a much more civilized version of Skynet/Cyberdyne firmware.

And is sure to be speaking impeccable English.

[Guthur]

Considering how well python has standardised and unified it's build tool chain this seems like a great idea....oh wait.

/sarcasm

I would prefer something far more formal than Python.

[mhh__]

I'm not sure if I really want python here.

Make is ugly, but it's reasonably clean "algebraically". I find Python very difficult to reason about, and ugly — it's a language that doesn't seem to have been well designed in any grand sense.

[dmos62]

I share your sentiments about Python. This is useful in a Python project though.