[bmoyles]

We haven't done much in the way of modification of Jenkins, and the big ball of plugins and configuration (with the associated deadlocks from bad plugins and some bad core Jenkins foo) is now 25 smaller balls of plugins and configuration. We're looking at ways to manage the new pain and are considering a number of routes from writing some tooling to help manage the world, introducing a smaller CI system that handles the very basic cases with Jenkins for more complex solutions. Our biggest problems are probably plugin creep (and the poor state of plugin maintenance in some cases), the cluttered UI with chunks of configuration hidden behind "Advanced" buttons, the inability to store job configuration with code a la Travis (which may be better with workflow), and handling routine maintenance across shards (without something in place like Operations Center), API inconsistencies (and the inability to fully manage via the API, forcing things like Groovy scripts over Jenkins remoting...). Once we have some direction on where to go, I'm sure we'll follow up with some more blog posts and share things at our meetups.

[jacques_chester]

I recommend, strongly, looking into Concourse. It's well suited for this sort of case, because that's why it was built.

I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way across quite different products (Redis, Apache Geode etc).

People have written resources (the main means of extension) in Go, Ruby and Python so far. If you can put it in a Docker image and execute it, then it can be taught to behave like a Concourse resource.

So far it's working well for us and others trialling it. I am very bullish about the future of Concourse.

[geodel]

I am going to read documentation of Concourse. Btw have you written like a blog or article about your experience with Concourse?

[jacques_chester]

No, I haven't.

My first approach was as an individual, which was quite difficult, because it took me a while to work out the differences between jobs and tasks and how to lay it all out. Lots of copying and pasting from other pipelines I studied.

At this point pretty much every team in Pivotal's Cloud Foundry division is running a Concourse pipeline, including the one I belong to. What's been interesting is how each team is experimenting with patterns that Concourse makes possible.

For example, I and my peers are now turning various things into "executable documents" -- the terminology sucks at the moment. Think of all the stuff buried in READMEs and wikis and long-forgotten cron jobs. How to build that special docker image that you only update every few months. The database backup. Keeping the blue-green deploy codepath warm.

When we find another one of these, we now encode it into our pipeline. That way, if I need to find it, it's there. And if it goes bad, the pipeline definition points me at where to go looking for everything of interest.

Another pattern that is emerging for us is "enforce project invariants". For example, we have multiple repos, so multiple Gemfiles and Dockerfiles, all of them with ruby versions set. At the front of our pipeline there is a little gateway to check that these are all identical. If not, it prints a table of versions found, so again, I know instantly where to go look. We have various other little invariants that turn days of insane debugging into a few seconds of sanity-checking.

None of this is novel. What Concourse lets me do is hoist good design practices out of code and apply them to CI. The final feedback loop I rely on can itself now be checked in, broken apart according to SOLID principles and even (my colleagues in Buildpacks are pioneering this) unit tested. Some colleagues in London have designed a fully generic pipeline that can be configured at runtime to build any of the five products they manage.

tl;dr fuck yeah Concourse.