Congrats on the launch! Since this is Go have you considered using CUE or looked at their flow package? Curious how you see it relating or helping with data pipelines.
I did look into CUE in the very early days of Bruin but ended up going with a more YAML-based configuration due to its support. I am not familiar with their flow package specifically, but I'll definitely take a deeper look. From a quick look, it seems like it could have replaced some of the orchestration code in Bruin to a certain extent.
One of the challenges, maybe specific to the data world, is that the userbase is familiar with a certain set of tools and patterns, such as SQL and Python, therefore introducing even a small variance into the mix is often adding friction, this was one of the reasons we didn't go with CUE at the time. I should definitely take another look though. thanks!
I did look into CUE in the very early days of Bruin but ended up going with a more YAML-based configuration due to its support. I am not familiar with their flow package specifically, but I'll definitely take a deeper look. From a quick look, it seems like it could have replaced some of the orchestration code in Bruin to a certain extent.
One of the challenges, maybe specific to the data world, is that the userbase is familiar with a certain set of tools and patterns, such as SQL and Python, therefore introducing even a small variance into the mix is often adding friction, this was one of the reasons we didn't go with CUE at the time. I should definitely take another look though. thanks!