Show HN: Togomak – declarative pipeline orchestrator based on HCL and Terraform

[srevinsaju]

Togomak is a command line tool that runs pipelines locally and on the cloud using HashiCorp Configuration Language with a Terraform-like architecture. I am a DevOps engineer and I wanted to build something that is CI/CD provider agnostic - something like GNU make, which I could plug into any provider of choice and not have to rewrite anything while retaining its rich features.

I love Terraform and its ecosystem, so I guess I borrowed a huge chunk of their design. Togomak makes parts of your pipeline reusable, by making it a Togomak module, just like Terraform modules. It also gives you power over several HCL functions, loop expressions, templating, and customizing a pipeline according to user input.

I started this project over a year ago, and over the course, I learned a lot about how Terraform works. Hopefully, some of our other Terraform friends would find this little tool useful!

togomak is actually short for "to-go-and-make", and it sounds similar to "tokamak", a fusion device using magnetic fields to contain hot plasma for energy; perhaps it's containers here!

[hamandcheese]

HCL makes this an instant pass for me.

Don't get me wrong - I love the idea of a declarative language for config. But HCL is just not pleasant.

Actually decent languages for that would be nix, jsonnet, or starlark.

[nikolay]

I have to agree. I'm writing Terraform files most of the day, and I don't stop to wonder why we still have to deal with it, and why every editor should support it in 2023 when we have much better alternatives such as Jsonnet, Skylark/Starlark, Cue, Dhall, etc. It's the "Not Invented Here" syndrome in the context of HashiCorp.

The only benefit of HCL is HCL v2, which allows you to write expressions without quotes, but this is supported by all alternatives listed above.

[srevinsaju]

HCL has a JSON representation [1], internally, objects behave that way. so it should be possible to write a Jsonnet wrapper around it. Terraform can currently parse json pipelines too.

[1]: https://github.com/hashicorp/hcl/blob/main/json/spec.md

[nikolay]

No, it's not the same - you need a preprocessor. I do use JSON with Terraform via CDKTF although git diff is way more unreadable than with HCL. So, JSON is definitely a bad idea, but HCL v2 is not as good as the alternatives.

[snthpy]

Also look at nickel which is an evolution of nix. It's my favorite in this space.

nickel-lang.org

https://github.com/tweag/nickel

[spqr233]

This is really interesting! I use github actions almost entirely so I wonder if it'd be possible to integrate it with that CI system.

Taking a step back, if you want to see this become more widely useable I'd suggest finding ways to integrate to existing workflows.

[srevinsaju]

Yes, I have been taking a look at https://github.com/nektos/act which allows running github actions workflow locally. I am investigating if I can integrate it directly in togomak, so that users can reuse existing github action in their pipeline. For the most bit, these pipelines run in node docker containers. I'm hoping to add a concept called "operator" where you can specify or use multiple build backends like Google Cloud Build, or GitHub Actions.

[jen20]

This looks very good - looking forward to giving it a spin at some point.

+1 for HCL over YAML - the experience is much nicer.

[srevinsaju]

I shot my foot a couple of times using yaml. Thankfully, not with HCL.

[esafak]

When you say pipeline I think you only refer to CI/CD pipelines, not ETL pipelines.

[jen20]

Not sure I understand why this matters? It also doesn't refer to oil pipelines, which is also fine.

[esafak]

Because orchestration also applies to ETL pipelines. Stating the context would have reduced confusion. A better subtitle might be "Declarative CI/CD pipeline orchestrator".

[debarshri]

You could technically use CI/CD pipelines to run ETL jobs. I have seen rundeck or jenkins being used as an ETL orchestrator.

[esafak]

Jenkins sucks even at its own game, forget about ETL.

[treebeard5440]

This is a great project - is the goal to be more like Gitlab, i.e. no plugins/marketplace or like Jenkins and GitHub Actions?

[srevinsaju]

I am looking forward to a mix of both - I am hoping to add a concept called "operator" which would be a go-plugin [1], just like terraform providers, but build backends. So, someone would be able to, say, write a Slack plugin (in Go, or anything over RPC) which sends a message once a build is complete - like Jenkins/GitHub actions, or just scripts that we can reuse like GitLab CI through `modules`.

Perhaps a new registry where we can push custom modules and providers (operators in this case), I'm curious to know about if we have any existing implementations we could reuse for the registry.

[1]: https://github.com/hashicorp/go-plugin

[thesorrow]

Is this similar to Dagger[1] ?

[1] https://dagger.io

[srevinsaju]

Similar, but declarative. Dagger uses a programming language, Togomak uses a configuration language on steroids.

[marcosnils]

Just to clarify, Dagger is both declarative and imperative. Yes, you can use multiple programming languages which exposes an imperative API, but your pipeline DAG has to be defined using Dagger's SDKs which is has a declarative form.

[srevinsaju]

Would it make sense to say Dagger is to Pulumi [1], as Terraform is to Togomak?

[1]: https://www.pulumi.com/

[latchkey]

It is amazing to me that people call Java verbose when you have things like this to declare a simple variable:

  variable "service" {
    type = string
    description = "Name of the service"
  }

HCL isn't a bonus, it is a hinderance. If you're going to build something like this, you might as well just use TypeScript or Python as your backend language.

Beyond that, this just looks like yet another task executor... as you say, like make. At which point, you might as well just use a task executor. If make is not your liking, how about https://github.com/casey/just

[kunley]

I agree with you that HCL sucks when it comes to variables.

Other thing that is funny: no user-defined functions; being unable to use function calls in string interpolations, but allowing variables... so it is like saying: we have this parser and at some points it allows expressions, at some other point not. This seems wrong.

At the same time I agree with or at least understand original author's intent to squeeze HCL to maxinum. There is something appealing in HCL's visual form, at least when defining resources. Maybe it's just (almost) simplest form of defining such structures that can exists.

This is why I started to work on my own format for configuration, visually similar but with different model of computation.

Here is the first attempt: https://github.com/wkhere/bcl

Disclaimer: I named it BCL, 'B' stands for Basic, to somehow relate to HCL and make it easily pronounced. But later I discovered that another BCL is used as Google to configure the Borg platform and seems to be massively hated ;) So I look for a better name..

Project seems to be a bit stalled since few weeks, but that's illusory, as I have a rewrite with a non-yacc parser almost ready in the other branch. Then I will code parsing nested structures and at that point it will become really neat. Plus, an add of implementation in several other languages I usually code.

Please stay tuned if you like it.. or tell me what's wrong with it if you don't ;)

[srevinsaju]

One primary design goal togomak had from the beginning was concurrency. All tasks run concurrently, unless a `depends_on` argument is mentioned. `just` didn't support that when I was initially building togomak, but there is a feature coming in soon which I am looking forward to: https://github.com/casey/just/pull/1562 .

While I was building togomak, I read through Dagger [1], Earthly [2], Concourse CI [3], Jest and Make along with the stuff I was already working with - Jenkins, GitHub actions and GitLab CI. Dagger [1] is really great, I like its design - it supports writing pipelines in Python, Typescript, Go and a few more languages. togomak tries to abstract away a lot of it. Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary.

[1]: https://dagger.io/ [2]: https://earthly.dev/ [3]: https://concourse-ci.org/

[marcosnils]

Hey there! Dagger employee here. Congrats on the launch, it's grateful to see other people trying to collaborate on this space.

> Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary.

Not sure if you've been following the latest updates about Dagger, but since the latest 0.9 release we have a preview feature where all the SDK tooling runs within Dagger itself as a container so they're not required to be installed in the host machine anymore. We've also added support for Dagger Modules which allows you to package and distribute pipelines a-la terraform. Our docs are still not updated yet but if you join our Discord server, you can find all the information there.

Cheers!

[dpedu]

I think you're looking at this basic snippet and taking it for something it's not. Not shown in this example and considered best practice are:

- Defining and validating data structures passed in; type doesn't have to be a basic type, it can be complex objects

- Default value

- Input validation criteria

So, with a real world example this is still going to be less verbose than the java equivalents.

[latchkey]

Sorry, won't back down that HCL is junk. When it is easier to not do best practices in examples, then you're off on the wrong foot to begin with.

[dpedu]

I could point out how it's way, way easier in java programs, since we're picking on java, to skip over doing things like this. Or I could point out how the bar to do things correctly is lower in this hcl example than the equivalent java. Or I could point out how the things I named as best practices can look in the real world, which seems to be the best course of discussion since unfamiliarity seems to be an issue here:

You have a type defined, it's just string. There's no default, which means it's a required input. Required inputs obviously can't have a default. Arbitrary strings are a perfectly acceptable input and without seeing how that variable is actually used, I can't speculate about validation criteria and it is very reasonable that for arbitrary strings there are none.

[srevinsaju]

It's also interesting that this is also a valid variable definition which does the same thing, but without the bells and whistles:

  variable "name" {}