[jcalder]

> We wanted our system to be enjoyable to use and easy to understand, but also solve the same problems as existing systems. For example, configuration is in YAML, not a proprietary syntax

I'm incredibly skeptical of this.

I'm ex-meta and have worked a lot with the enterprise solutions you're talking about and the choice of starlark (originally python) as the build definition language is one of the killer features of the systems.

People want to create macros, target generators, etc. It's a common use case for a lot of engineers and IMO is a pretty killer feature.

Being able to say "This is an X86 Python Binary", "This is an M1 python binary" and then bundle those into "this is how you build either of those binaries based on inputs" without ever touching the internals or anything other than (more or less) a blob of python is why those tools scale organizationally.

It allows the teams that need to do weird stuff to unblock themselves without drowning the tools org. Sure, it has draw backs. Super deep macro layers are kinda a crime against humanity and debugging/evolving them can be quite expensive, but I think that's just the cost of software.

If that logic isn't in the build definitions it'll expand into a meta layer that generates configuration (I've seen giant "Translate this definition into 30 configs to run stuff" systems time and time again).

I may just be super biased from past mistakes and wins, but I think what you're doing is just moving the complexity out of your tool into neighboring tools and selling it as a win isn't really true, it's shuffling complexity around not removing it.

[mileswjohnson]

Based on everyone's feedback about YAML (we didn't expect this much), we'll probably reconsider this!

[jcalder]

Glad to hear it!

To synthesis my comment down (because it's easier once I've written it once, poorly):

The complexity you see in Bazel/Buck/Pants build files may seem like a result of their decision to use a programming language. That's a red herring. That complexity is fundamental to the problems people need to solve to build software. If you remove the ability to solve it in your build system the complexity will move to other systems.

Wrappers, CI, whatever. The complexity is just the problem space rather than a result of bad tooling.

[codethief]

+1

Glad someone else brought up this topic and phrased it way better than I ever could have!

This other comment further down puts it very well, too: https://news.ycombinator.com/item?id=34892724

> If you remove the ability to solve it in your build system the complexity will move to other systems.

Adding to this: I think one of the main limitations of purely declarative (e.g. YAML-based) configurations is that a declarative DSL (as opposed to an imperative DSL) means you (only) get to use config options the inventors of the config language have thought of and implemented (i.e. "defined"). You cannot put into words things that require vocabulary that doesn't exist in your language.

Meanwhile, an imperative language gives you the ability to define on your own what you would like the machine to do.

As an example, consider Gitlab CI configs (YAML-based) or Ansible playbooks. In both cases, once you hit the boundaries of the DSL spec (which, in my experience, typically happens on day one) you can't do anything other than 1) give up or 2) leave the realms of the DSL and as you say,

> move [the complexity] to other systems

[hakre]

Well thought and I can imagine this can be further differentiated. E.g. what is considered boundaries or limits in a declarative way also work as guardrails. So while it could be possible to hit actual limits on day one, there is often more than only two options to go on. Declarative for me often appears easier to find the correct one or to correctly know that that system is not designed to do that.

The other differentiation I think is worth on the following part, also which I'd liked a lot btw:

> move [the complexity] to other systems

which stemmed from:

> If you remove the ability to solve it in your build system the complexity will move to other systems.

I think it is even _more_ beneficial to consider that this does not start at the build system:

If you don't have and don't add the ability at development (make it a property of the project), the complexity will continue to _increase_ each time you move it into other systems after it (build, version control, CI, ... etc.).

[nsm]

There is also a Rust implementation of Starlark as a starting point https://github.com/facebookexperimental/starlark-rust

To add to everyone else, please don't use YAML. Starlark is great _precisely_ because it is a readable, well known (nearly Python) language that is limited at the same time (no unbounded for loops, no way to do non-deterministic things like get the current time or `random()`).

[gen220]

Take a look at Apache Aurora [1] if you want some inspiration on how to mold python into a config language. I used this system for a few years and mainly agree with the person you're replying to – having a proper programming language to define config is a very nice feature that I miss.

[1]: https://aurora.apache.org/

[malkia]

one of the benefits of starlark (unlike python): "Starlark is suitable for use in highly parallel applications. An application may invoke the Starlark interpreter concurrently from many threads, without the possibility of a data race, because shared data structures become immutable due to freezing." from https://github.com/bazelbuild/starlark/blob/master/spec.md - it's not python, you can't do recursion (!) and it's more limited (you can't read a file in bazel, and parse it, you have to make this operation into the graph somehow)

[oblio]

What Python badly needs is Lua-like sandboxing/allowlisting.

You get an interpreter with access to 0 functions by default and can add stuff on top.

Maybe Python already has it, I'm not aware of it.

[mileswjohnson]

Thanks will take a look at.

[crubier]

Yeah, 100%, use a scripting language. Maybe JS? You could use deno core https://deno.com/blog/roll-your-own-javascript-runtime