[baq]

> I know some people might consider this insanity

Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane. You’re actually introducing much needed sanity into the process by admitting that a real programming language is the tool to use here.

I can’t imagine the cognitive dissonance Lisp folks have when dealing with this madness, not being one myself.

[TeMPOraL]

> I can’t imagine the cognitive dissonance Lisp folks have when dealing with this madness, not being one myself.

After a decade trying to fight it, this one Lisper here just gave up. It was the only way to stay sane.

I remain hopeful that some day, maybe within our lifetimes, the rapid inflation phase of software industry will end, and we'll have time to rethink and redo the fundamentals properly. Until then, one can at least enjoy some shiny stuff, and stay away from the bleeding edge, aka. where sewage flows out of pipe and meets the sea.

(It's gotten a little easier now, as you can have LLMs deal with YAML-programming and other modern worse-is-better "wisdom" for you.)

[no_wizard]

I'm shocked there isn't a 'language for config' that hasn't become the de facto standard and its YAML all the way down seemingly. I am with you 100%.

It would really benefit from a language that intrinsically understood its being used to control a state machine. As it is, that is what nearly all folks want in practice is a way to run different things based on different states of CI.

A lisp DSL would be perfect for this. Macros would make things alot easier in many respects.

Unfortunately, there's no industry consensus and none of the big CI platforms have adopted support for anything like that, they all use variants of YAML (I always wondered who started it with YAML and why everyone copied that, if anyone knows I'd love to read about it).

Honestly, I can say the same complaints hold up against the cloud providers too. Those 'infrastructure as code' SDKs really don't lean into the 'as code' part very well

[julik]

I think the issue here is mostly the background the CI setups came from. They were frequently coming from the "ops" part of the ecosystem, and some ops folks held some ideas very strongly back then (I heard this first hand):

"Application and configuration should be separate, ideally in separate repos. It is the admin's job to configure, not the developer's"

"I do not need to learn your garbage language to understand how to deploy or test your application"

"...as a matter of fact, I don't need to learn the code and flows of the application itself either - give me a binary that runs. But it should work with stale configs in my repo."

"...I know language X works for the application but we need something more ubiquitous for infra"

Then there was a crossover of three streams, as I would call it:

YAML was emerging "hard" on the shoulders of Rails

Everyone started hating on XML (and for a good reason)

Folks working on CI services (CruiseControl and other early solutions) and ops tooling (chef, ansible) saw JSON's shortcomings (now an entire ecosystem has configuration files with no capability to put in a comment)

Since everybody hated each other's languages, the lowest common denominator for "configuration code" came out to be YAML, and people begrudgingly agreed to use it

The situation then escalated severely with k8s, which adopted YAML as "the" configuration language, and a whole ecosystem of tooling sprung up on top using textual templating (!) of YAML as a layer of abstraction. For k8s having a configuration language was an acute need, because with a compiled language you need something for configuration that you don't have to compile with the same toolchain just to use - and I perfectly "get it" why they settled for YAML. I do also get why tools like Helm were built on top of YAML trickery - because, be it that Helm were written in some other language, and have its charts use that, they would alienate all the developers that either hate that language personally, or do not have it on the list of "golden permitted" at their org.

Net result is that YAML was chosen not because it is good, but because it is universally terrible in the same way for everyone, and people begrudgingly settled on it.

With CI there is an extra twist that a good CI setup functions as a DAG - some tasks can - and should - run in parallel for optimization. These tasks produce artifacts which can be cached and reused, and a well-set CI pipeline should be able to make use of that.

Consequently, I think a possible escape path - albeit an expensive one - would be for a "next gen" CI system to expose those _task primitives_ via an API that is easy to write SDKs for. Read: not a grpc API. From there, YAML could be ditched as "actual code" would manipulate the CI primitives during build.

[mdaniel]

> (I always wondered who started it with YAML and why everyone copied that, if anyone knows I'd love to read about it).

I know this isn't a definite answer to your question, but it was still super interesting to me and hopefully it will inspire someone else to dig into finding the actual answer

The best guess I have as far as CI/CD specifically appears to be <https://en.wikipedia.org/wiki/Travis_CI#:~:text=travis%20ci%...> which launched in 2011 offering free CI and I found a reference to their .travis.yml in GitLab's repo in 2011, too

- CruiseControl (2004) was "ant as a service," so it was XML https://web.archive.org/web/20040812214609/http://confluence...

- Hudson (2007) https://web.archive.org/web/20140701020639/https://www.java.... was also XML, and was by that point driving Maven 2 builds (also XML)

- I was shocked that GitHub existed in 2008 https://web.archive.org/web/20081230235955/http://github.com... with an especial nod to no longer a pain in the ass and Not only is Git the new hotness, it's a fast, efficient, distributed version control system ideal for the collaborative development of software but this was just "for funsies" link since they were very, very late to the CI/CD game

- I was surprised but k8s 1.0.0 still had references to .json PodSpec files in 2010 https://github.com/kubernetes/kubernetes/blob/v1.0.0/example...

- cloud-init had yaml in 2010 https://github.com/openstack-archive/cloud-init/blob/0.7.0/d... so that's a plausible "it started here" since they were yaml declarations of steps to perform upon machine boot (and still, unquestionably, my favorite user-init thing)

- just for giggles, GitLab 1.0.2 (2011) didn't even have CI/CD https://gitlab.com/gitlab-org/gitlab/-/tree/v1.0.2 -- however, while digging into that I found .travis.yml in v2.0.0 (also 2011) so that's a very plausible citation <https://gitlab.com/gitlab-org/gitlab/-/blob/v2.0.0/.travis.y...>

- Ansible 1.0 in 2012 was also "execution in yaml" https://github.com/ansible/ansible/blob/v1.0/examples/playbo...

[motorest]

> Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines.

I've been using YAML for ages and I never had any issue with it. What do you think is wrong with YAML?

[mst]

Turing complete YAML ends up being an app specific terrible programming language.

Many of us would rather use a less terrible programming language instead.

[hadlock]

Something went horribly wrong if your coworkers are putting switching logic inside your config

[hadlock]

I've been using YAML professionally for a decade and other than forgetting to wrap some values in quotes, has been an absolute non issue.

Some people talk about YAML being a turing complete language, if people try to do that in your CI/CD system just fire them

I'll allow helm style templating but that's about it.

[mschuster91]

> Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane.

It's miles better than Jenkins and the horrors people created there. GitLab CI can at least be easily migrated to any other GitLab instance and stuff should Just Work because it is in the end not much more than self contained bash scripts, but Jenkins... is a clown show, especially for Ops people of larger instances. On one side, you got 50 plugins with CVEs but you can't update them because you need to find a slot that works for all development teams to have a week or two to fix their pipelines again, and on the other side you got a Jenkins instance for each project which lessens the coordination effort but you gotta worry about dozens of Jenkins instances. Oh and that doesn't include the fact many old pipelines aren't written in Groovy or, in fact, in any code at all but only in Jenkins's UI...

Github Actions however, I'd say for someone coming from GitLab, is even worse to work with than Jenkins.