[bittermandel]

I firmly believe that us at Molnett(serverless cloud) going for a strict monorepo built with Bazel has been paramount to us being able to make the platform with a small team of ~1.5 full-time engineers.

We can start the entire platform, Kubernetes operators and all, locally on our laptops using Tilt + Bazel + Kind. This works on both Mac and Linux. This means we can validate essentially all functionality, even our Bottlerocket-based OS with Firecracker, locally without requiring a personal development cluster or such.

We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations.

It's been a HUGE blessing. It has taken some effort, will take continuous effort and to be fair it has been crucial to have an ex Google SRE on the team. I would never want to work in another way in the future.

EDIT: To clarify, our repo is essentially only Golang, Bash and Rust.

[eadmund]

> a small team of ~1.5 full-time engineers

Yes, with one and a half FTEs you should only have a single repo.

My experience with Bazel has been extremely bad, but I don’t think that it should necessarily be avoided completely. It may actually have some value on extremely large multi-team projects. But for less than two FTEs it seems like massive overkill.

I believe that you could do what you need with Kind (and maybe Tilt?), without Bazel.

> We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations.

Go kind of does that for you already, with go.mod. Since kubectl is a Go program, you could achieve that goal the same way.

> it has been crucial to have an ex Google SRE on the team

I wonder how many additional team members y’all could afford in return for an ex-Googler’s salary expectations.

I sincerely hope that y’all find the maintenance expense of Bazel to be worth it going forward. Hopefully you will!

[bittermandel]

I don't think you are wrong at all. As we are all founders with an OK salary and this is our area of expertise, so we're able to take advantage of our previous experiences and reap the benefits. We're probably uniquely positioned here.

I had massive issues at my previous employer with Bazel. They did not try to make Bazel work for non-SREs, which as you can imagine didn't work very well. So it's definitely not a silver bullet!

We should probably write a blog post about our setup!

[munksbeer]

> My experience with Bazel has been extremely bad

Would you mind elaborating and providing some examples of what was bad?

We have a monorepo built using bazel, and at first when new to bazel, I was pretty frustrated. But now I can't think of any issue I've had with it recently.

But we do have a relatively simple setup.

[wocram]

Bazel is notoriously difficult to set up to begin with, but not so difficult to use after that. What was your bad experience?

[teitoklien]

we run everything under systemd managed services instead of k8s and deploy via ansible playbooks at our company, and we similarly use tmuxinator to spin up all the backend api, search engine, databases like qdrant, meilisearch, etc and frontend services all in dev mode with all the terminals auto opening in window panes inside a tmux shell.

It really makes development in dev mode super simple and easy, and running all of the services in local dev environment is as simple as running one command, ‘tmuxinator’ at root of our monorepo and boom everything is up.

Monorepo truly outcompete individual repos for almost all projects, its far more pleasurable ever since I changed to this method of development.

[lbhdc]

I am in a pretty similar situation as you, and have really been feeling the benefits of going all in on bazel.

> We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations.

Currently I have to run `bazel run <tool>`. Your solution sounds way better. How does yours work?

[peterldowns]

Not the OP but you can use tools like direnv + mise/asdf/nix so that every time a developer cd's into the monorepo, their shell environment loads a pinned, declaratively-configured set of dependencies and tools whose definitions are part of the monorepo.

The way I'd naively set up something like OP described would be to have direnv + nix flake deliver you a copy of bazelisk, and then have some custom shell scripts added to $PATH that alias `go = bazel run go`, `kubectl = bazel run kubectl` or whatever custom wrappers you want.

(Handwaving and I know the above isn't quite correct)

[codethief]

Came here to post this. I've used asdf/mise for years (and recently also Nix + direnv) and it works tremendously well.

[mikn]

Hi! Previously mentioned ex-Google SRE! There are a few layers to it - to make it work "ok" you need to first have a tool runner wrapper rule that does something similar to:

```

ctx.actions.write(output="""

tool_path=$(realpath {tool_short_path})

cd ${{BUILD_WORKING_DIRECTORY}}

exec $tool_path

""".format(tool_short_path=tool.short_path)

```

The purpose of this rule is to ensure that the tool's CWD is actually where you are inside the repository and not within the runfiles folder that Bazel prepared for you.

The second step is to set up a symlink target, similar to this:

```

#! /usr/bin/env bash

tool_name=$(basename $0)

exec -a "$tool_name" bazel run --ui_event_filters=-info,-stdout,-stderr --noshow_progress //tools/bin:$tool_name -- "$@"

```

We need to filter out all UI events since for some tools we intercept (such as jq) it expects the stdout to be clean from other output when used programmatically.

We then create a symlink for each tool name (say kubectl) to this script from another folder, and then we use `direnv` to inject the folder of symlinks into the user's paths with an `.envrc` file in the repository root like this:

```

PATH=$PWD/tools/path:$PATH

```

We have had this in place for quite a while now - it does seem like this pattern has caught some more wind and buildbuddy.io has released a ruleset: https://github.com/buildbuddy-io/bazel_env.bzl paired with https://github.com/theoremlp/rules_multitool achieves the same thing that we have built internally, the main difference being that with the bazel run wrapper we have made, you always run the latest version, whereas with the bazel_env pattern you need to manually rerun their target to get the latest binaries. :)

[peterldowns]

Any chance you'll be releasing your rules? I'd love to see how you do it.

[lbhdc]

Thanks! I am gonna give this a try.

[eddd-ddde]

Probably something similar to a py venv that you run to activate some aliases in your environment?

[mattmanser]

The question here is why are you using micro service pattern and k8s with 2 Devs. That pattern is not designed for that small scale operation and adds tons of completely unnecessary complexity.

And does it really matter what you go with when you've got 1.5 engineers?

It's a non-problem at that scale as both engineers are intimately aware of how the entire build process works and can keep it in their head.

At that scale I've done no repo at all, repo stored on Dropbox, repo in VCS, SVN, whatever, and it all still worked fine.

It really hasn't added anything at all to your success.

BTW, it's still common for developers to start entire repos on their own laptops with zero hassles in tons of dev shops that haven't been silly and used k8s with 2 developers.

In fact at the start of my career I worked with 10 or so developers the shitty old MS one where you had to lock files so no-one else can use them. You'd checkout files to allow you to change them (very different to git checkout), otherwise they'd be ready only on your drive.

And the build was a massive VB script we had to run manually with params.

And it still worked.

We got some moaning when we moved to SVN too at how much better the old system was. Which was ridiculous as you used to have to run around and ask people to unlock key files to finish a ticket, which was made worse as we had developer consultants who'd be out of office for days on end.

So then you'd have to go hassle the greybeard who had admin rights to unlock the file for you (although he wasn't actually that old and didn't have a beard).

[bittermandel]

I think this take is quite shallow and lacks insight into how one would actually build a somewhat complex technical platform.

We are not using a microservice pattern at all. I am not sure where you get that from. If anything we have several "macro services".

Our final setup is quite complex as we are building a literal cloud provider, but in practice we have a Go API, a Docker registry, a Temporal Worker and a Kubernetes controller. Whats complicated is everything else around it. We run our platform on bare-metal and thus have auxiliary services like a full-blown Kubernetes cluster, Ory Hydra + Kratos, SpiceDB, Cilium, Temporal Cluster + Workers and some other small things. We need to be able to test this locally to feel safe to release to production. And in turn our production environment is almost identical to our local environments.

None of that would be possible unless we've done something similar to what we have built today. Most companies cannot run their entire stack on their laptop, more unlikely that they could run a full cloud provider.

[bee_rider]

Keeping code in Dropbox kinda sucks even with 1 or .5 developers though. That said, yeah, a regular old git or (I assume, never used it) svn seems fine.

[eadmund]

> Keeping code in Dropbox kinda sucks even with 1 or .5 developers though. That said, yeah, a regular old git or (I assume, never used it) svn seems fine.

What you do is store the git repo in Dropbox, and developers just use it as a remote. With backups, this could actually go a reasonably long time, although I personally wouldn’t suggest it.

[skydhash]

I think an easier option would be to buy a vps and just use ssh.