[jsw]

Bazel is a fully reproducible and hermetic build system. A lot of painstaking work goes into it producing the exact same artifacts build after build. And that provides some interesting properties that you can leverage for artifact caching, deployments, and CICD.

We very happily runny a polyglot monorepo w/ 5+ languages, multiple architectures, with fully reproducible artifacts and deployment manifests, all deployed to almost all AWS regions on every build. We update tens of thousands of resources in every environment for every build. The fact that Bazel is creating reproducible artifacts allow us to manage this seamlessly and reliably. Clean builds take an hour+, but our GH self-hosted runners often complete commit to green build for our devs in less than a minute.

The core concept of Bazel is very simple: explicitly declare the input you pass to a rule/tool and explicitly declare the output it creates. If that can click, you're half way there.

[bobsomers]

> Bazel is a fully reproducible and hermetic build system.

Yes, and it's very important to note that Bazel does nothing to solve the problem about having a reproducible and hermetic runtime. Even if you think you aren't linking against anything dynamically, you are probably linking against several system libraries which must be present in the same versions to get reproducible and hermetic behavior.

This is solvable with Docker or exceptionally arcane Linux hackery, but it's completely missing from the Bazel messaging and it often leaves people thinking it provides more than it really does.

[amscanne]

Bazel is perfectly capable of producing static binaries, if that's what you want. It's not fair to say that does nothing to solve this problem, it simply does not mandate that the artifacts it produces are static.

[bobsomers]

Most static binaries are not completely statically linked. The system libstdc++ and libc as well as a few other things are almost always linked dynamically.

In particular, it’s almost impossible to get glibc to link statically. You need to switch to musl or another libc that supports it.

[sph]

Static linking means no dependencies whatsoever, so it only needs a syscall interface in the kernel. If your binary requires libstdc++, it's not static, period.

Go creates static binaries. I was a bit bummed out to learn that Rust doesn't by default since it requires glibc (and copying a binary to an older distribution fails because glibc is too old).

[dijit]

it's pretty trivial to create static binaries in Rust, but things like openssl won't compile since openssl depends on glibc.

Two commands to get you running:

1) rustup target add x86_64-unknown-linux-musl

2) cargo build --target=x86_64-unknown-linux-musl

[sph]

Yes and no, it's trivial unless you use some of the most common libraries that depend on, i.e. OpenSSL, like pretty much anything that talks with the outside world. Then it's painful. (You mentioned OpenSSL in your comment but somehow I missed that part when I replied. My bad.)

One thing Go has going for it is that they have a more complete standard library so you can do without the pain of cross-compiling—which is what you're doing if most of your libs assume you're in a glibc environment but you really want musl.

I know because I recently tried compiling a Rust application that had to run on an air-gapped older Debian machine with an old glibc, and the easiest solution was to set up a debian container, download Rust and compile there, instead of fixing all the cargo deps that didn't like to be built with musl.

[staticassertion]

Other than libc I'm not sure what else would "almost always" be linked. It's going to depend on the language, but for Go/Rust at least static linking is pretty much the default, and lots of people even link to musl.

For other languages I'd say openssl is probably the next most common.

[capableweb]

> but for Go/Rust at least static linking is pretty much the default

I don't think that's true for Rust. It defaults to depending on glibc, at least on Linux, and you need to explicitly build with musl to get static binaries.

[staticassertion]

> , and lots of people even link to musl.

[manwe150]

Platform ABI typically specifies that libatomic must be dynamically linked, since otherwise libdl will misbehave. Lots of other common libraries including libpthreads, libm, libgfortran, libquadmath also are frequently linked (though commonly now are linked into libc.so for better system performance)

[sph]

Those are all glibc or GCC libraries.

There is no platform ABI mandate nor Linux kernel requirement for userspace applications to be dynamically linked to anything. If you can talk to the kernel, you can do anything those libraries can do.

[staticassertion]

Never even heard of libatomic, where is it specified that it's required to dynamically linked? I'm surprised to hear that.

[ihgann]

It’s also easily solvable with Nix (which I would hope continues to gain traction).

[Rapzid]

Nix, Hurd, and Haskell will walk into a bar.. Soon!

[ihgann]

My main OS is NixOS… and for exercise I do CrossFit.

Maybe I just like cults ^_^

[seabrookmx]

At minimum you're self aware :)

[hxtk]

Others have mentioned statically linked binaries. I thought I'd mention I actually use Bazel to build most of my docker images, and unlike Docker itself, Bazel rules_docker builds reproducible (i.e., same digest) container images by default.

[c0mptonFP]

Being mindful of system library versions is deployment 101. They're dependencies after all.

Also, I do prefer to avoid dynamic linking altogether.

[baby]

Aren’t there people who use both nix and bazel?

[kaba0]

From what I read, they don’t play too nicely with each other.

[pharmakom]

I think this is out of scope for a build system.

[jeffbee]

Having spent a great deal of time getting bazel set up as you describe, I feel you have given readers a misleading impression. Bazel does not come out of the box that way. It uses whatever toolchain is laying around on the host, by default. It builds with system headers and links with system libraries. It looks at your environment variables. You need to do a lot of surgery on the toolchain to make it hermetic and reproducible.

[tehlike]

Sounds like a job for docker or nix?

[hxtk]

I can't comment on Nix, but I work on a project that uses recursive Makefiles and build containers to make the builds (almost) hermetic.

I dread running builds because it takes like ten minutes to run `make all`. I spent like four hours writing Bazel build files for it and all of a sudden my clean rebuilds were taking like five minutes and my incremental builds were taking a couple of seconds. It was fantastic.

Ultimately it was rejected because other people on the project hated working with bazel and weren't familiar enough when it to debug any problems. If a build didn't work, the only solution was to call me because no one else was used to interpreting the bazel errors: if a curl command failed in a script in a docker file somewhere, everyone knew what that meant. But I was the only one who would see "IOException fetching... Error 401" and immediately know that they provided the wrong password.

I also consulted for another project where people were using containers as VMs, running systemd and copying in updated binaries, because they, too dreaded rebuilding the containers. Bazel, again, made it so that they could rebuild everything in a matter of seconds. That project is still happily using bazel last I checked.

[chriswarbo]

Docker is a red herring. The parent's statement can be modified for Docker like this:

> B̶a̶z̶e̶l̶Docker uses whatever toolchain is laying around on the h̶o̶s̶t̶Internet

More precisely, Docker can use exact, well-specified, cryptographically-tamper-proof environments (images); but the only way to actually specify and build such an environment is via shell scripts (in a "Dockerfile"). In practice, such scripts tend to run some other tool to do the actual specification/building, e.g. `make`, `mvn`, `cargo`, `nix`, etc.

If those tools aren't reproducible, then wrapping them in Docker doesn't make them reproducible (sure we can make a snapshot, but that's basically just a dirty cache). In reality, most Dockerfiles seem to run wildly unreproducible commands, e.g. I've encountered Dockerfiles with stuff like `yum install -y pip && pip install foo`.

If those tools are reproducible, then wrapping them in Docker is unnecessary.

Also note that outputting a container is nothing special; most of those tools can do it (e.g. in a pinch you can have `make` run `tar` and `sha256sum` "manually")

[jeffbee]

The bazel way to do it is to put the tools in an archive, refer to the archive and its checksum in your workspace, and execute the build in a completely empty sandbox.

[hakre]

Can you share a bit about the completely empty sandbox? Is this a build-root with it's own user and environment? Or does it build inside worktree, e.g. a subdirectory. Or can both be done?

[matnosner]

The concept of explicitly declared inputs and outputs is awesome. Though it's closed build and the necessity to define the builds down to the compiler level makes Bazel complex and breaks some developer workflows.

For this reason we are building a Bazel competitor with a less restrictive build environment which can also be applied to projects with lower than 1M line of code.

https://bob.build