[jacques_chester]

I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built.

I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way across quite different products (Redis, Apache Geode etc).

People have written resources (the main means of extension) in Go, Ruby and Python so far. If you can put it in a Docker image and execute it, then it can be taught to behave like a Concourse resource.

So far it's working well for us and others trialling it. I am very bullish about the future of Concourse.

[geodel]

I am going to read documentation of Concourse. Btw have you written like a blog or article about your experience with Concourse?

[jacques_chester]

No, I haven't.

My first approach was as an individual, which was quite difficult, because it took me a while to work out the differences between jobs and tasks and how to lay it all out. Lots of copying and pasting from other pipelines I studied.

At this point pretty much every team in Pivotal's Cloud Foundry division is running a Concourse pipeline, including the one I belong to. What's been interesting is how each team is experimenting with patterns that Concourse makes possible.

For example, I and my peers are now turning various things into "executable documents" -- the terminology sucks at the moment. Think of all the stuff buried in READMEs and wikis and long-forgotten cron jobs. How to build that special docker image that you only update every few months. The database backup. Keeping the blue-green deploy codepath warm.

When we find another one of these, we now encode it into our pipeline. That way, if I need to find it, it's there. And if it goes bad, the pipeline definition points me at where to go looking for everything of interest.

Another pattern that is emerging for us is "enforce project invariants". For example, we have multiple repos, so multiple Gemfiles and Dockerfiles, all of them with ruby versions set. At the front of our pipeline there is a little gateway to check that these are all identical. If not, it prints a table of versions found, so again, I know instantly where to go look. We have various other little invariants that turn days of insane debugging into a few seconds of sanity-checking.

None of this is novel. What Concourse lets me do is hoist good design practices out of code and apply them to CI. The final feedback loop I rely on can itself now be checked in, broken apart according to SOLID principles and even (my colleagues in Buildpacks are pioneering this) unit tested. Some colleagues in London have designed a fully generic pipeline that can be configured at runtime to build any of the five products they manage.

tl;dr fuck yeah Concourse.