[UserIsUnused]

Thank god trends like Pulumi and the new AWS sdk is emerging.

General purpose programing languages are getting more expressive by the day, why do we use data serialization languages instead for configs? it doesn't make any sense.

Configuration is code not data.

[solatic]

> General purpose programing languages are getting more expressive by the day

You know, once upon a time, we understood that declarative approaches to software engineering were superior to imperative approaches, when declarative approaches are feasible. Declarative approaches are much safer and easier to test, at a cost of only being able to express what the tool accepting the declarative approach can understand. Imperative approaches are strictly worse for any problem set where a declarative approach solves the problem within performance requirements. The additional expressiveness of languages like Pulumi is the last thing I want.

YAML is a horrible language for declarative system configuration because a) any sufficiently complex system will require you to generate your declarative codebase in the name of maintainability, b) generating code for any language where whitespace is significant will lead you to an early death, and c) stringly-typed languages are fundamentally unmaintainable at sufficient scale. But this is not an indictment of a declarative approach! It is an indictment of YAML.

> Configuration is code not data.

Data > code. Data does not need to be debugged. The best code you can have is deleted code - deleted code does not need to be maintained, updated, or patched. Code is a necessary evil we write in order to build operable systems, not a virtue in and of itself.

[spc476]

I use Lua for configuration files. It's easy to restrict what you can do in Lua (I load configuration data into its own global state with nothing it can reference but itself). Plus, I can define local data to help ease the configuration:

    local webdir = "/www/site/htdocs"

    templates = 
    {
      {
        template = "html/regular",
        output   = webdir .. "/index.html",
        items    = "7d",
        reverse  = true
      },
      
      {
        template = "rss",
        output   = webdir .. "/index.rss",
        items    = 15,
        reverse  = true
      },
      
      {
        template = "atom",
        output   = webdir .. "/index.atom",
        items    = 15,
        reverse  = true
      },
    }

When I reference the configuration state, templates[1].output will be "/www/site/htdocs/index.html". And if the base directory changes, I only have to change it in one location, and not three.

[chriswarbo]

I think "declarative" is a bit of a red herring here. Deterministic/reproducible/pure is a more appropriate distinction: configuration languages like JSON/YAML/XML/s-expressions/etc. are trivially deterministic, but not very expressive, leading to boilerplate, repetition, external pre/post-processing scripts, etc.

Allowing computation can alleviate some of those problems, whether it's done "declaratively" (e.g. prolog-like, as in cue) or not (e.g. like idealised algol with memory cells).

The main reason to avoid jumping to something like Python isn't that it's "not declarative"; it's that Python is impure, and hence may give different results on each run (depending on external state, random number generators, etc.). Python can also perform arbitrary external effects, like deleting files, which is another manifestation of impurity that we'd generally like to avoid in config.

tl;dr The problem isn't the style of computation, it's the available primitives. Don't add non-deterministic or externally-visible effects to the language, and it wouldn't really matter to me whether it's "declarative" or not.

[toyg]

That's a bit of a no-true-scotsman there. If the problem is just the markup of choice, we should see an alternative markup emerging any time now. If we see imperative-focused tools instead, maybe it's not just about the markup.

[solatic]

We do see alternative "markups", if you want to call them that, emerging that solve the generative issues - the two that come to mind are Dhall and CUE. They bring forth JSON to help them interoperate and be relevant in a world that predominately expects JSON/YAML, but they can also be read directly.

[UserIsUnused]

There are declarative general purpose programing languages.

That data you are talking about does need to be debugged, like Helm charts and pipeline definitions. Sure data is better, but config is code, not data.

[solatic]

Generators need to be debugged, not data. It's very easy to test a generator - a few unit tests checking whether, for a given input, the generator produced the expected output, and you're set. Data sometimes needs to be cleaned, but there's no such thing as a bug in data.

Whether the generated declarative output produces the expected behavior on the part of the tool interpreting the declarative output is part of the tool's contract, not the generator or the declarative output. If you need to check the tool's behavior then either a) you wrote the tool or b) you're writing an acceptance test for the tool, which is an entirely different endeavor.

[UserIsUnused]

Things like pipeline definitions and helm charts are generators.

[solatic]

No, Helm uses charts (data) to generate object definitions (in YAML). Helm is the generator.

There's nothing that prevents you from writing a unit test that runs `helm template` directly to check whether a given chart with given values will produce a given set of YAML files.

[grawprog]

>but config is code, not data.

Config is both. Config variables are data. The code that accesses and uses those variables is...well...code. they should be kept separate. Like any other code and data. Config isn't a separate special entity, it's just another part of the program. The data part should be represented as such and the code part should be code. Trying to combine them and create a special 'config' language is mistake.

[reallydontask]

The main problem is that, in my experience, the majority of DevOps teams are Ops teams that have been renamed and refocused towards automation.

These are people that by and large don't want to code, not saying that they can't or won't.

To be fair this has been in Windows shops, where scripting has only recently (last 5-10 years) taken off, so you've got a lot of windows admins that the closest they've been to code is Batch scripting with a bit of Powershell. This is a big change for them

As it happens i read about pulumi recently and I've put it on mt list of todo things, but I can't see that I'll be able to sell it to our team and our team is blessed (cursed?) with three former developers

[toyg]

> These are people that by and large don't want to code

I disagree. That's the stereotype that tool builders have of such people. Good ops people have always loved coding, or we wouldn't be living on the mountains of Bash scripts also known as "Linux distributions".

(Besides, people who don't like to code won't like writing tons of declarative markup either. So there is little point in the current approach either way.)

[reallydontask]

It might be a stereotype but it's also my experience, which granted is limited and Windows based, which as I pointed out in the previous comment hasn't been really onboard with the scripting experience until relatively recently

As to the declarative markup, for instance Azure Devops still doesn't have feature parity between YAML pipelines and classic pipelines, so while you could well be right about the same resistance to yaml, it's not, necessarily an issue yet.

[toyg]

Most devops tools are built on Linux for Linux, then ported to Windows later as an afterthought, so I don’t think platforms are much of a factor. It’s a defect in the production pipeline somewhere, something like feedback from potential users not reaching developers until it’s too late. It doesn’t help that, in some cases, vendors just impose what is going to happen, and the community is simply forced to put up with it.

[dragonwriter]

> The main problem is that, in my experience, the majority of DevOps teams are Ops teams that have been renamed and refocused towards automation.

I'm on a relatively new team kind of like that (it's not a former Ops team but most of the team was pulled from Ops/DBA/analysis teams, and fits the description, and it's not a really a DevOps but more of a Dev+Ops team) and the general reaction of the team to being introduceed to the AWS CDK has been “we need to move to that as soon as we can”, even from people who very vocally never wanted to be programmers. And that's with, in many cases, a couple months experience with both programming and YAML IAC in the form of CloudFormation.

[falcolas]

> Thank god trends like Pulumi and the new AWS sdk is emerging.

The more things change, the more they stay the same.

The pendulum is swinging back towards scripting languages. But give it a few years, and we'll be railing against scripting languages for not being idempotent enough, and moving back to the latest version of YAML and XML (perhaps we'll go with TOML this time) with custom interpreters.

We have seen this half a dozen times already, between turing complete DSLs in ruby, to plain bash, perl, and python scripts. The other side are the DSLs written using YAML, JSON, and XML (and every other config language written in history).

[q3k]

I still have my reservations against Pulumi in particular, though.

I've only really skimmed their documentation, but the idea that calling 'new' against a VM class instantiates a VM in production seems to be to magical for my taste, and might be an indication that it's yet another product that targets the happy path more than what we tend to experience in production: failures.

[jimbokun]

> Configuration is code not data.

As any Lisper knows, code is data is code.

[sgslo]

> new AWS SDK

Could you share some details on that? A quick Google search didn't reveal anything about major changes to the SDK.

[dragonwriter]

Probably referring to the CDK (Cloud Development Kit), a newish SDK for developing Cloudformation infrastructure-as-code in general-purpose programming languages.

https://aws.amazon.com/cdk/

[JonathanHedley]

I believe the OP is referring to the AWS CDK (Cloud Development Kit) - https://docs.aws.amazon.com/cdk/latest/guide/home.html