[carreau]

I my experience if you get more builders/faster hardware someone higher up will end up asking for more items in the build-matrix, and the CI time will balloon up again.

It's rarely only a question of just allocating money to more/better hardware, it's also a question of policy and willingness of your organisation to keep CI time short/feedback fast.

[mgaunard]

That's why in my opinion you should always have the option of building locally, just what you need when you want it, instead of having to go through a slow CI/CD pipeline.

[danielheath]

I built up our CI pipeline until it was the faster way of running tests. You can rent more compute than you can carry...

[mgaunard]

Just provisioning a new node, deploying a new kubernetes worker and downloading the source and pre-built artifacts takes longer than it takes to build stuff incrementally on my machine.

Also my local machine has resources entirely dedicated to me and isn't held back because someone else decided to rebuild the world.

[danielheath]

why does your CI do all that after you start the build if it happens every time? Developer time is expensive.

Scale up ahead of time, so there’s always a machine ready. Prefetch the repository / packages when CI workers start, so they’re already installed before a build is triggered. Use your imagination - CI doesn’t have to suck.

[mgaunard]

If you scale up ahead of time then it's not really on-demand, that means you have dedicated hardware you're paying much more for.

[danielheath]

The marginal cost to keep a large-ish spot instance running 99% of the time is dirt cheap (eg $400/mo to keep one extra c7g.8xlarge running).

If you're paying for an engineering team, that's a rounding error.

[viraptor]

> will end up asking for more items in the build-matrix, and the CI time will balloon up again.

I don't think is has to end like that. You can have separate queues and separate levels of assurance. For example, does every commit have to be tested in each of 20 possible configs? You can run one common config by default and allow unblocking the rest on demand. Then enforce all of them only as a merge gate.

If you can also split them into separate queues that don't block each other, you get both larger matrix and faster dev turnaround.

[carreau]

Sure, and you are proving my point. You have to allocate time for someone to reply to all those questions and update the CI to only test what's necessary, or prioritize some test, and run the rest only if they pass etc.

So you do need policies to actually be allowed to optimise CI.

[nitwit005]

This aspect is pretty key. My observation has been that build systems will be as slow as people will tolerate.

Effort and money will only be expended to make things faster when the build times are perceived as intolerable.

[substation13]

The solution is a build system that scales by caching between runs - most PRs only change a few files, after all.

[carreau]

As I replied to the sibling comment, yes, I agree, but caching is not magic. You need to know what to invalidate, and you need to be allowed to spend time making test group that run depending on what have changed. So without policy and having devs being allowed to work on optimizing CI/reducing time, you can list plenty of strategies to make it faster but it will likely not happen.

[ohgodplsno]

Caching is pretty damn simple with any sensible build tool that simply stores hashes of files and modules. Caching in the JVM ecosystem is pretty much free, for example.

[carreau]

Sure, but you seem to be read this blog post in the narrow view of building as only "compiling an artifact" and not running the test suite across many platforms/condition, or any other work. I mean the author even points out that building the kernel is only a use-case that takes time for measurement purposes. For some projects I work on the compilation itself takes maybe 1/10 max of the total build and test. So even with infinite agressive compilation caching I would not gain much.

[ohgodplsno]

Nope, our Android app has caching, both for building APKs/AABs, but also for test suites. PRs run the entire test suite (on too many variants, even, I forgot to remove some unneeded stuff), what hasn't changed or hasn't been impacted by said changes simply gets a FROM-CACHE and isn't reran.

[ninkendo]

That's great for your android app, and you're lucky you have such a thing. But your experience is not generally applicable to all of software development. There are many, many, many build systems out there in the wild that do not support the ability to know what unit tests are affected by what code changes. I'd wager something like 99% of them.

[substation13]

You can cache tests too if you properly capture the inputs

[Jochim]

Caching can often cause more issues than it solves.

While largely fixed now, older C# projects suffered from separating dll references from the reference to the package that provided them.

This lead to the situation where you may be referencing a package that hasn't been downloaded. Due to the package cache, the solution would typically build on a "dirty" environment like the developer's machine or anything that had previously built the project but would fail on any fresh environment.

[alkonaut]

i build a large app where every output of around 100 (dlls) are tagged with a version number which also contains the git commit SHA etc.

It's great for keeping things tidy and not having to break the whole thing up in modules like "UI 21.0.12.11 needs Backend 12.2.1.11 or greater". Instead everything builds more or less into a monolith. But it does have one drawback: making a commit invalidates every bit of the build output immediately.