[bjackman]

I always think it's a shame that these features end up getting built into ecosystem-specific build tools. Why do we need separate build systems for every language? It seems entirely possible to have build system that can do all this stuff for every language at once.

From my experience at Google I _know_ this is possible in a Megamonorepo. I have briefly fiddled with Bazel and it seems there's quite a barrier to entry, I dunno if that's just lack of experience but it didn't quite seem ready for small projects.

Maybe Nix is the solution but that has barrier to entry more at the human level - it just seems like a Way of Life that you have to dive all the way into.

Nonetheless, maybe I should try diving into one or both of those tools at some point.

[sunshowers]

(I worked on source control at FB for many years.)

The main argument for not overly genericizing things is that you can deliver a better user experience through domain-specific code.

For Bazel and buck2 specifically, they require a total commitment to it, which implies ongoing maintenance work. I also think the fact that they don't have open governance is a hindrance. Google's and Meta's internal monorepos make certain tradeoffs that don't quite work in a more distributed model.

Bazel is also in Java I believe, which is a bit unfortunate due to process startup times. On my machine, `time bazelisk --help` takes over 0.75 seconds to run, compared to `time go --help` which is 0.003 seconds and `time cargo --help` which is 0.02 seconds. (This doesn't apply to buck2, which is in Rust.)

[jeffbee]

This is likely because you are running it in some random PWD that doesn't represent a bazel workspace. When running in a workspace the bazel daemon persists. Inside my workspace the bazelisk --help invocation needs just 30ms real time.

Running bazel outside of a bazel workspace is not a major use-case that needs to be fixed.

[thayne]

> When running in a workspace the bazel daemon persists. Inside my workspace the bazelisk --help invocation needs just 30ms real time.

It still has a slow startup time, bazel just works around that by using a persistent daemon, so that it is relatively fast after as long as the daemon is running.

[sunshowers]

That's good to know, thank you!

Do you encounter cache invalidation bugs with daemonization often? I've had pretty bad experiences with daemonized dev tools in the past.

[jrockway]

Bazel prints a message when you invalidate the in-memory cache in a perhaps accidental way; you can supply it with a flag to make this an error and skip the cache invalidation.

If you try to run two Bazel invocations in parallel in the same workspace, one waits for the other to be done.

[jeffbee]

I assumed they meant an error of improperly using cached results. I am sure bazel has its flaws but it assiduously avoids that.

[sunshowers]

I did mean improperly using cached results. It's merely the hardest problem in computer science :)

The sibling suggests this may still be an issue. I'm not surprised—cache invalidation is very difficult to solve, and conservative approximations like tearing down the whole process each time tend to be quite effective.

[__float]

Yes, unless you're using persistent workers. Then you may very well run into the same issues they mention.

[spockz]

GraalVM’s native image has been a thing for a while now. This could overcome the daemon issue partially. The daemon does more ofc by as it keeps some state in memory. But at least the binary start time is a solved problem in Java land.

[sunshowers]

Why does Bazel not ship a native image by default?

[spockz]

I do not know. I could only find GitHub issues and feature sound offering integration with graalvm for building native apps with Bazel.

[pornel]

> Why do we need separate build systems for every language?

Because being cross-language makes them inherit all of the complexity of the worst languages they support.

The infinite flexibility required to accommodate everyone keeps costing you at every step.

You need to learn a tool that is more powerful than your language requires, and pay the cost of more abstraction layers than you need.

Then you have to work with snowflake projects that are all different in arbitrary ways, because the everything-agnostic tool didn't impose any conventions or constraints.

The vague do-it-all build systems make everything more complicated than necessary. Their "simple" components are either a mere execution primitive that make handling different platforms/versions/configurations your problem, or are macros/magic/plugins that are a fractal of a build system written inside a build system, with more custom complexity underneath.

OTOH a language-specific build system knows exactly what that language needs, and doesn't need to support more. It can include specific solutions and workarounds for its target environments, out of the box, because it knows what it's building and what platforms it supports. It can use conventions and defaults of its language to do most things without configuration. General build tools need build scripts written, debugged, and tweaked endlessly.

A single-language build tool can support just one standard project structure and have all projects and dependencies follow it. That makes it easier to work on other projects, and easier to write tooling that works with all of them. All because focused build system doesn't accommodate all the custom legacy projects of all languages.

You don't realize how much of a skill-and-effort black hole build scripts are is until you use a language where a build command just builds it.

[bjackman]

But this just doesn't match my experience with Blaze at all. For my internal usage with C++ & Go it's perfect. For the weird niche use case of building and packaging BPF programs (with no support from the central tooling teams, we had to write our own macros) it still just works. For Python where it's a poor fit for the language norms it's a minor inconvenience but still mostly stays out of the way. I hear Java is similar.

For vendored open source projects that build with random other tools (CMake, Nix, custom Makefile) it's a pain but the fact that it's generally possible to get them building with Blaze at all says something...

Yes, the monorepo makes all of this dramatically easier. I can consider "one-build-tool-to-rule-them-all isn't really practical outside of a monorepo" as a valid argument, although it remains to be proven. But "you fundamentally need a build tool per language" doesn't hold any water for me.

> That makes it easier to work on other projects, and easier to write tooling that works with all of them.

But... this is my whole point. Only if those projects are in the same language as yours! I can see how maybe that's valid in some domains where there's probably a lot of people who can just do almost everything on JS/TS, maybe Java has a similar domain. But for most of us switching between Go/Cargo/CMake etc is a huge pain.

Oh btw, there's also Meson. That's very cross-language while also seeming extremely simple to use. But it doesn't seem to deliver a very full-featured experience.

[pornel]

I count C++ projects in the "worst" bucket, where every project has its own build system, its own structure, own way to run tests, own way to configure features, own way to generate docs.

So if a build system works great for your mixed C++ projects, your build system is taking on the maximum complexity to deal with it, and that's the complexity I don't want in non-C++ projects.

When I work with pure-JS projects, or pure-Go projects, or pure-Rust projects, I don't need any of this. npm, go, and rust/cargo packages are uniform, and trivial to build with their built-in basic tools when they don't have C/C++ dependencies.

[bjackman]

No I'm not talking about mixed projects (although the ability to do that is very important).

I'm saying that using it for separate C++ and Go projects is extremely convenient and ergonomic.

[munificent]

I think the problem is basically because the build system has to be implemented using some ecosystem, and no other ecosystem wants to depend on that one.

If your "one build system to rule them all" was built in, say, Ruby, the Python ecosystem won't want to use it. No Python evangelist wants to tell users that step 1 of getting up and running with Python is "Install Ruby".

So you tend to get a lot of wheel reinvention across ecosystems.

I don't necessarily think it's a bad thing. Yes, it's a lot of redundant work. But it's also an opportunity to shed historical baggage and learn from previous mistakes. Compare, for example, how beloved Rust's cargo ecosystem is compared the ongoing mess that is package management in Python.

A fresh start can be valuable, and not having a monoculture can be helpful for rapid evolution.

[pansa2]

> No Python evangelist wants to tell users that step 1 of getting up and running with Python is "Install Ruby".

True, but the Python community does seem to be coalescing around tools like UV and Ruff, written in Rust. Presumably that’s more acceptable because it’s a compiled language, so they tell users to “install UV” not “install Rust”.

[makapuf]

Note that installing python stdlib installs tkinter and thus tcl.

https://wiki.tcl-lang.org/page/Python-Tcl-Interactions

[pjmlp]

I tend to think that is more Rust community using Python, and RIIR stuff, than Python community themselves.

I know Python since version 1.6, and this has never been a thing until Rust.

Same applies to the RIIR going on JavaScript side.

Including tools that were already written in compiled languages, but of course weren't Rust, or had an idea to make a startup around them.

[6keZbCECT2uB]

Partly in jest, you can often find a Perl / bash available where you can't find a Python, Ruby, or Cargo.

[spauldo]

Not sure why that's in jest. Perl is pretty much everywhere and could do the job just fine. There's lots of former (and current) Perl hackers still around.

[marwis]

Sounds like the only way out of this is to design language agnostic tooling protocols that anybody can implement.

[jlarsen]

I've had exactly the same thought, after hitting walls repeatedly with limitations in single-language ecosystems. And likewise, I've had the same concerns around the complexity that comes with Bazel/Buck/Nix.

It's been such a frustration for me that I started writing my own as a side project a year or two ago, based on a using a standardized filesystem structure for packages instead of a manifest or configuration language. By leaning into the filesystem heavily, you can avoid a lot of language lock-in and complexity that comes with other tools. And with fingerprint-based addressing for packages and files, it's quite fast. Incremental rebuild checks for my projects with hundreds of packages take only 200-300ms on my low-end laptop with an Intel N200 and mid-tier SSD.

It's an early stage project and the documentation needs some work, but if you're interested: https://github.com/somesocks/dryad https://somesocks.github.io/dryad/

One other alternative I know of that's multi-language is Pants(https://www.pantsbuild.org/), which has support for packages in several languages, and an "ad-hoc" mode which lets you build packages with a custom tool if it isn't officially supported. They've added support for quite a few new tools/languages lately, and seem to be very much an active project.

[8n4vidtmkvmk]

Not loving the cutesy names (https://somesocks.github.io/dryad/docs/02-concepts/01-the-ga...). I want my build tool to be boring.

[6keZbCECT2uB]

I agree. In my opinion, if you can keep the experience of Bazel limited to build targets, there is a low barrier to entry even if it is tedious. Major issues show up with Bazel once you start having to write rules, tool chains, or if your workspace file talks to the Internet.

I think you can fix these issues by using a package manager around Bazel. Conda is my preferred choice because it is in the top tier for adoption, cross platform support, and supported more locked down use cases like going through mirrors, not having root, not controlling file paths, etc. What Bazel gets from this is a generic solution for package management with better version solving for build rules, source dependencies and binary dependencies. By sourcing binary deps from conda forge, you get a midpoint between deep investment into Bazel and binaries with unknown provenance which allows you to incrementally move to source as appropriate.

Additional notes: some requirements limit utility and approach being partial support of a platform. If you require root on Linux, wsl on Windows, have frequent compilation breakage on darwin, or neglect Windows file paths, your cross platform support is partial in my book.

Use of Java for Bazel and Python for conda might be regrettable, but not bad enough to warrant moving down the list of adoption and in my experience there is vastly more Bazel out there than Buck or other competitors. Similarly, you want to see some adoption from Haskell, Rust, Julia, Golang, Python, C++, etc.

JavaScript is thorny. You really don't want to have to deal with multiple versions of the same library with compiled languages, but you have to with JavaScript. I haven't seen too much demand for JavaScript bindings to C++ wrappers around a Rust core that uses C core libraries, but I do see that for Python bindings.

[sunshowers]

> You really don't want to have to deal with multiple versions of the same library with compiled languages, but you have to with JavaScript.

Rust handles this fine by unifying up to semver compatibility -- diamond dependency hell is an artifact of the lack of namespacing in many older languages.

[6keZbCECT2uB]

Conda unifies by using a sat solver to find versions of software which are mutually compatible regardless of whether they agree on the meaning of semver. So, both approaches require unifying versions. Linking against C gets pretty broken without this.

The issue I was referring to is that in Javascript, you can write code which uses multiple versions of the same library which are mutually incompatible. Since they're mutually incompatible, no sat-solve or unifyer is going to help you. You must permit multiple versions of the same library in the same environment. So far, my approach of ignoring some Javascript libraries has worked for my backend development. :)

[dwattttt]

Rust does permit multiple incompatible versions of the same library in the same environment. The types/objects from one version are distinct from the types/objects of the other, it's a type error to try mix them.

But you can use two versions of the same library in your project; I've done it by giving one of them a different name.

[lihaoyi]

My experience with Bazel is it does everything you need, and works incredibly well once set up, but is ferociously complex and hard to learn and get started with. Buck and Pants are easier in some ways, but fundamentally they still look and feel mostly like Bazel, warts and all

I've been working on an alternate build tool Mill (https://www.mill-build.org) tries to provide the 90% of Bazel that people need at 10% the complexity cost. From a greenfield perspective a lot of work to try and catch up to Bazel's cross-language support and community. I think we can eventually get there, but it will be a long slog

[morepedantic]

Brazil performs dependency resolution in a language-agnostic way.

https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6...