[Nextgrid]

I wonder how much it's going to take before people realize that maybe a single server somewhere in the office running Jenkins isn't that bad of an idea after all. Unless you're Google, "scale" will inherently not be a problem, and risks of operator error can be reduced by scheduling maintenance at times where an accidental outage won't impact your business.

[justin_oaks]

I agree with the sentiment that people should evaluate whether or not they need an external service to run their builds.

That said, there are a number of reasons to not use the Jenkins server in the office:

1) Someone on staff needs to maintain it.

2) A single hardware failure can cause significant downtime.

3) Your office internet service may have limited bandwidth and be a bottleneck for your build or artifact deployment

4) Having your server on-site may be considered a security risk.

I'm not saying that a server in the office is a bad idea, I'm just saying that each business needs to consider the advantages and disadvantages. I'm sure there are those who could get by just fine with a server in the office.

[CSSer]

Not to change the subject but point one is such a sticky point that a lot of the non-technicals or semi-technicals get overly hung up on in my experience. It's a very valid concern, and personally in this case I would say even more so because the potential pitfalls of maintaining physical hardware are no joke. However, I keep running into situations where that point is touted out for even compromising, intermediary alternatives like setting up your own build pipeline on a cloud provider or even using a cloud provider instead of a niche SaaS solution at all.

Forgive me if this is just "one of those problems" everyone has to deal with. I'm admittedly still pretty young and naive, and I've found myself in a mentorship/leadership adjacent role at a fast growing company where my relatively small team has a vastly deeper well of technical knowledge and aptitude than the rest of the organization. So I do wonder sometimes if I'm wrong about it, but my intuition, and it seems that of GP's too it seems, drives me to spring to this kind of solution (that we should be doing whatever it is ourselves because it's mission critical for what we do) often and it really gets to me that it's not so readily heard.

It's somewhat ironic because a big part of the point of HN in general is building and promoting these kind of progressive services, but the longer I'm in the industry and the more advanced my knowledge and responsibilities get, the more I keep finding that it's really hard to communicate the gaps in between the marketing claims these SaaS tools promote to higher-ups who don't see the incurred technical debt and wasted time working around their shortcomings. I've found this to be especially true when the issues are ones that one only bumps up against after rigorous use or problems that decision makers try to address by encouraging the addition of yet another third party tool to the mix.

[justin_oaks]

You make valid points.

I cited problems with hosting something on-site, but didn't list the problems of integrating with a service too.

When you integrate with a service you need a member of staff to maintain that integration too. Of course we hope that the integration is easier than maintaining a server but that is not always the case.

You also take on the risk of the SaaS being flaky, like we see with CircleCI.

The SaaS can always raise their prices or go out of business, requiring your company to switch off of that service.

Maybe the SaaS is a security risk. What happens to your company data and builds if/when they get breached?

There's no silver bullet, there's just a set of trade-offs. You get to decide which is best for your use-case.

[CSSer]

> When you integrate with a service you need a member of staff to maintain that integration too.

I really like the way you worded this. I think in hindsight it's obvious but you conceptualized it in a way that I've found hard to communicate to others so far. I see now that this is probably because I've been so zeroed in on my own idiosyncratic technical issues with a given service when applicable rather than expressing it through the criteria the people I report to must consider when evaluating solutions brought to them, so thank you for that!

[egwor]

And also consider the cost of running it there too (heat/energy/noise/space costs)

[justin_oaks]

Good point. Those are very real problems too, especially in small office environments.

You don't want that server in some spot where someone can accidentally kick it and suddenly the server goes offline. You also don't want it in an unventilated closet where it'll overheat.

As for noise... in this day and age of open office spaces and chatty coworkers, you'll be lucky if the server noise is your biggest problem. Unless everyone is working from home, and now the problem is physically accessing the server if it goes down or needs other physical maintenance.

[softfalcon]

Folks are likely going to say they can't do this "at scale". However, I've been in both the "small shop with Jenkins running on a Dell blade" and also in the "mega-corp with lots of devs everywhere using all kinds of CI pipelines". I think there is merit in what you're saying, but I suggest an addendum.

In both cases, have seen the "CI is down" issues crop up.

There seems to be a middle ground that I have yet to fully do, but have seen folks do and heard it works well:

Make it so folks can deploy their CI pipeline anywhere. In my case, I would create Docker containers that can run the CI pipeline, pipe the output/result to anywhere you want and have it be able to run anywhere.

The goal is that you could deploy it on a VPS somewhere, or on a Dell blade in your closet, or even run it on a dev's machine. The idea is that you want to reliably be able to run the CI steps anywhere without jank while following the full process the same as if it was running on your dedicated job runner solution. Redundancy, if you will.

I think the idea isn't necessarily "Jenkins in your company closet is better" and more of "how easily can you setup your CI pipeline to get your work done?"

[xcambar]

To operate properly at the scale of hundreds of engineers pushing changes, you absolutely need a cluster of machines and a team to operate it.

It is significantly cheaper and more efficient to pay a third party.

[XCSme]

I personally really dislike remote building. It's so much faster to do everything locally, even more if you can do it on your own machine (machine which is likely to be very powerful, close to the state-of-the art performance, as the tech geeks we are).

[xyst]

in some cases that single server is bob's 2016 MBP because management is too cheap to buy a dedicated server

[justin_oaks]

Extra points if management is too cheap to buy a dedicated server, but is willing to pay for 3rd party services.