[mickeyp]

The winning strategy for all CI environments is a build system facsimile that works on your machine, your CI's machine, and your test/uat/production with as few changes between them as your project requirements demand.

I start with a Makefile. The Makefile drives everything. Docker (compose), CI build steps, linting, and more. Sometimes a project outgrows it; other times it does not.

But it starts with one unitary tool for triggering work.

[carlsmedstad]

This line of thinking inspired me to write mkincl [0] which makes Makefiles composable and reusable across projects. We're a couple of years into adoption at work and it's proven to be both intuitive and flexible.

[0]: https://github.com/mkincl/mkincl

[zahlman]

I think the README would be better with a clearer, up-front explanation of what this builds on top of using `make` directly.

[chedabob]

Ye, kick off into some higher-level language instead of being at the mercy of your CI provider's plugins.

I use Fastlane extensively on mobile, as it reduces boilerplate and gives enough structure that the inherent risk of depending on a 3rd-party is worth it. If all else fails, it's just Ruby, so can break out of it.

[krautsauer]

Make is incredibly cursed. My favorite example is it having a built-in rule (oversimplified, some extra Makefile code that is pretended to exist in every Makefile) that will extract files from a version control system. https://www.gnu.org/software/make/manual/html_node/Catalogue...

What you're saying is essentially ”Just Write Bash Scripts”, but with an extra layer of insanity on top. I hate it when I encounter a project like this.

[nicoburns]

https://github.com/casey/just is an uncursed make (for task running purposes - it's not a general build system)

[arielcostas]

How does `just` compare to Task (https://taskfile.dev/)?

[timw4mail]

Just uses make-like syntax, not yaml, which I view as a huge advantage.

[mickeyp]

No I'm saying use Makefiles, which work just fine. Mark your targets with PHONY and move on.

[krautsauer]

You still get bash scripts in the targets, with $ escape hell and weirdness around multiline scripts, ordering & parallelism control headaches, and no support for background services.

The only sane use for Makefiles is running a few simple commands in independent targets, but do you really need make then?

(The argument that "everyone has it installed" is moot to me. I don't.)

[wtcactus]

I agree, but this is kind of an unachievable dream in medium to big projects.

I had this fight for some years in my present work and was really nagging in the beginning about the path we were getting into by not allowing the developers to run the full (or most) of the pipeline in their local machines… the project decided otherwise and now we spend a lot of time and resources with a behemoth of a CI infrastructure because each MR takes about 10 builds (of trial and error) in the pipeline to be properly tested.

[mickeyp]

It's not an unachievable dream. It's a trade-off made by people who may or may not have made the right call. Some things just don't run on a local machine: fair. But a lot of things do, even very large things. Things can be scaled down; the same harnesses used for the development environment and your CI environment and your prod environment. You don't need a full prod db, you need a facsimile mirroring the real thing but 1/50th the size.

Yes, there will always be special exemptions: they suck, and we suffer as developers because we cannot replicate a prod-like environment in our local dev environment.

But I laugh when I join teams and they say that "our CI servers" can run it but our shitty laptops cannot, and I wonder why they can't just... spend more money on dev machines? Or perhaps spend some engineering effort so they work on both?

[maccard]

> You don't need a full prod db, you need a facsimile mirroring the real thing but 1/50th the size.

My experience has been that the problems in CI systems come from exactly these differences “works on my machine” followed by “oops, I guess the build machine doesn’t have access to that random DB”, or “docker push fails in our CI environment because credentials/permissions, but it works when I run it just on my machine”

[wtcactus]

> It's not an unachievable dream. It's a trade-off made by people who may or may not have made the right call.

In my experience at work. Anything that demands too much though, collaboration between teams and enforcing hard development rules, is always an unachievable dream in a medium to big project.

Note, that I don't think it's technically unachievable (at all). I just accepted that it's culturally (as in work culture) unachievable.

[zahlman]

Sometimes the problem is that the project is bigger than it needs to be.

[nottorp]

Funny enough, the LLMs are allowed to run builds on your local machine. The humans, not any more.

[wtcactus]

But it isn't a question of security. The project would very much like the developers to be able to run the pipelines on their machines.

It's just that management don't see it as worth it, in terms of development cost and limitations it would introduce in the current workflow, to enable the developers to do that.

[nottorp]

> But it isn't a question of security.

Where did i mention security?

> in terms of development cost and limitations it would introduce in the current workflow

Well said. "in the current workflow". As in, not "in the development process". Those are unrelated items.