[vscarpenter]

I'm curious as to why Square would spend engineering talent on a distributed CI system when you can buy something like Jetbrains TeamCity that does what they wanted to do. This is not meant as a slam and I am also a developer who loves to build things but I would think Square and other companies for that matter would want to apply their limited engineering talents in solving bigger problems that grow the business.

Not looking for a flame war - just curious to get different perspectives.

[hkarthik]

It makes a lot of sense. I'm currently working on a large Rails app and we've felt a lot of pain with an off the shelf CI System (Jenkins) and the idea of writing our own CI has been mentioned a few times.

For Team City, here's the cost breakdown:

TeamCity Enterprise = $1,999

229 Build agents @ $299/each = $68,741

Total yearly license cost ~ $70K

This doesn't include the time that a team of developers will spend configuring the build and babysitting it during through the early growing pains of adopting a CI system. Also you'll probably still need a build/release engineering team to manage the server when you have a team of 100+ devs like Square does.

When you add it all up, building a custom CI setup makes a lot of sense. And I'm sure they grew it over time, with an initial version that was usable being completed in just a few months. What they've open sourced is the end result of spending that time.

When you really adopt testing as part of your culture, CI becomes totally critical so engineering should spend the time to make it a solid and viable solution.

[bitops]

A total yearly cost of $70K isn't all that bad when you break down the numbers. Software engineer salaries are typically much higher, let alone considering HR & insurance expenses.

So that's one employee you don't have to hire. I was on a team of 4 that, among other responsibilities, managed several TeamCity instances, all of them pretty large.

Generally speaking, once it was up and running, it was pretty low overhead and easy to automate. So I think it really is worth thinking hard about the cost before people dive in and roll their own solutions.

And there's always the argument of: instead of building your own, why not work to improve an existing OSS solution?

[hkarthik]

> Generally speaking, once it was up and running, it was pretty low overhead and easy to automate.

This unfortunately has not been my experience with off-the-shelf CI solutions (open source or otherwise). Of course in all instances, we were writing a lot of new code and continuously enhancing old code so the projects under CI were very active. We dealt with CI systems being overloaded, builds breaking for cryptic environment setup reasons, and just an overall slowness due to memory leaks that were hard to diagnose and only occurred in the CI environments.

In the smaller teams, one engineer would essentially become "the build guy" that would babysit the CI system. In larger teams, this became a function of a Configuration/Release management team. In all cases, CI was a regular pain point unless there a lot of customization done over a long period of time.

At the surface level, a custom CI solution wouldn't take long to write. A simple poller to check version control, a script to build, and some basic notifications could probably be hacked together quickly and iterated on over time.

[pbiggar]

How long is your test suite? I expect you'd find https://circleci.com much more affordable (our "build agents" are beefy, and you're unlikely to need 229 of them!)

[MasterScrat]

I'm currently considering Jenkins for my next project; what were its shortcomings for you?

[hkarthik]

For us it has just become extremely fragile. We often lose executors and the Jenkins processes tend to leak memory.

In general we probably restart executor machines quite often just due to the above issues alone.

Also our Devops folks have complained about the lack of automation around Jenkins. If they could monitor it better, we could probably be a lot more proactive about dealing with our problems around it.

If you're looking at CI for your next project, I would look at either a hosted solution like CircleCI or TravisCI Pro. Don't jump straight to self-hosted CI until you outgrow these solutions.

[boomzilla]

CI system is the new ./configure and Makefile. You almost always need a new one for each of your systems.

Yes, there's a lot of common patterns that can be factored into a library/system, but the little details will end up biting you at the end.

[shykes]

That's exactly what Dockerfiles are for: to specify a unique, non-ambiguous way to build your code. Regardless of language, distro or compiler, "docker build" is all I need to produce a self-contained, runnable artifact not unlike a static binary.

There's a cool tutorial on using Dockerfiles at http://www.docker.io/learn/dockerfile/

[j_s]

Would you mind starting the conversation by addressing the 'What about existing solutions?' portion of the article?

edit: rephrase