[kosta_self]

Does anyone have Jenkins in production and found it to be reliable and pleasant to use?

I had to do lots of Jenkins integrations some time ago, and even though I tried to minimize the number of plugins and make things as simple as possible, things would randomly break from time to time or exhibit weird behaviour etc.

I had the impression that Jenkins is deeply conceptually confused about some of its concepts, e.g. how builds are triggered. Also, it is a huge pile of untestable spaghetti code which explains the weird bugs.

I modified an open-source plugin only to find out that it's almost impossible to write meaningful tests for it: you can't even mock Jenkins API without using the darkest Java mock magic. Jenkins classes are just written in an old style that makes testing _really_ hard but probably can't be changed without breaking all of Jenkins.

I tried Jenkinsfile which was only even more unreliable (at the time at least, this was > 1y ago). The whole idea of using groovy, modifying the hell out of it to make it even more weird and surprising and edge-casy just didn't go well for me.

I ended up with generating a _lot_ of very simple jobs for each project and connecting them via triggers instead. It was not very pretty, but it was the most reliable that I could get out of Jenkins.

So the thought of integrating Jenkins deeply into you deployments, talking to Kubernetes and sitting in the middle of a huge pile of complexity (Jenkinsfile, Dockerfile, Helm, ...) and magic "that you don't need to worry about" scares the hell out of me.

But then again, if you want to do CI/CD with Jenkins that's what you might want, right? (I would prefer more simple approaches if forced to use Jenkins, though)

[nobleach]

Pleasant? Never. I absolutely hate every time I have to launch a webpage to dig around and find out why my build failed. Automating jobs is a nightmare. I can cURL a request to kick off a build and do you know what it returns? NOTHING a 201 response with NO information to link to the build in progress. Oh there's a JSON API to see running jobs, but without some sort of ID, it's useless. Scripting is in Groovy. Want to use another language? Too bad! "It was hard enough to get Groovy" is the response from the team. If I'm forced to use a web interface, does it still have to look like it was designed by a team of Java devs from 2008? The only thing that's changed is the Hudson to Jenkins clipart. Yes, I'm ranting.

[vorg]

> "It was hard enough to get Groovy"

To make it work, they had to cripple Apache Groovy so you can't use its functional collections-based functions. Not sure if you can really call it "Groovy" with that handicap.

[jstrachan]

Thanks for your feedback. We are trying to make things simpler with Jenkins X by using best of breed tools (git providers, issue trackers, cloud services, service meshes, security & code quality tools etc) with best of breed OSS tools like kubernetes, helm & skaffold to automate CI/CD on the cloud & kubernetes.

One of the big changes to traditional Jenkins is we don’t expect folks to have to configure Jenkins, add/remove/edit plugins or even write Dockerfiles or Jenkinsfiles.

If you really wanna do that Jenkins X won’t stop you - but we are trying to help automate and simplify CI/CD through standard tools, standard file formats (Dockerfile, Jenkinsfile, skaffold, helm etc).

Is the cloud, kubernetes, docker, helm & istio complex? Sure - but our goal is to simplify, automate & avoid folks having to look at all that detail.

It’s still early days and a challenge. Eg even Lambda & the AWS API Gateways is complex. But we hope to keep improving to make things easier to use & to help folks go faster by providing automated CI/CD as a service on any kubernetes cluster / cloud

[kosta_self]

Thanks for your reply!

Having a good set of plugins that integrate well should limit the complexity explosion and the number of edge cases users run into. So that's an improvement. It won't fix the Jenkins Heisenbugs though, of course :)

[abrookewood]

We use it in all of our environments and it is extremely flexible, mostly reliable, but still feels like a hodgepodge of plugins and various components. Once you throw in a variety of build tools and deployment options etc, it can get rather unwieldy. Still, we use it for almost everything now (no forgotten cron jobs running on random servers) and it gets the job done. I often think we should investigate other options, but we have made such an investment at this point it would be a massive undertaking to move to something else.

[sammm]

I actually think Jenkins is way too flexible for most use cases. We moved to GitLab CI which isn't perfect, but it provides safety rails/structure/opinions that pretty much provides an answer for everything you want to do, apart from maybe obscure corner cases that might not make sense for a CI/CD tool anyways.

Also you get the close integration of your CI tool and your git repos, which is very nice from a visibility point of view.

Having said that, GitLab is trying to own all parts of the build and deployment process, which from previous HN discussions, is of great annoyance to a lot of people who want to cherry pick what they use GitLab for.

[sytse]

Thanks for using GitLab. We want to be supportive of cherry picking GitLab features. For example we just release CI/CD with GitLab for GitHub https://about.gitlab.com/features/github/

Is there something we can add to GitLab to make it more composable?

[sammm]

Thanks for the question, greatly appreciated. We used GitLab on k8s when we first transitioned, but we found there were a few things we didn't quite like about the GitLab-Omnibus helm setup, so we moved it off the cluster and used the AWS EC2 AMI which was really easy to setup.

We are going to start experimenting with the new cloud native GitLab chart, but it would need to gain some maturity before we use it in production.

Do you know if the new GitLab cloud native helm chart will allow you to turn-off certain things like mattermost and prometheus? That was something that we didn't like about the omnibus chart because it exposed several extra services/ports that we didn't really want to manage/think about at the time.

[joshlambert]

samm we are indeed making all components that are not core gitlab services optional. You will be able to turn them off with a simple `prometheus.enabled=false`.

Thanks for giving the charts a try in alpha/beta, please pass along any feedback. We'd love to get it!

[wesleytodd]

Thanks for the good work on GitLab, I tell everyone I can how great of an experience it was to use it at my last company. The CI integration was GREAT, the ui is pretty nice and the maintenance overhead was minimal.

[sytse]

Thanks, great to hear that.

[AnthonBerg]

My perspective: Jenkins f...ing sucks. For the reasons you delineate. To me it's a pile of side-effects which pretend not to know obvious things.

As a counterexample I present Jetbrains' TeamCity (if running a build server yourself is necessary).

[flurdy]

Pleasant? No.

Tolerable? Yes.

If using proper DSLs https://github.com/jenkinsci/job-dsl-plugin , https://github.com/hmrc/jenkins-job-builders, etc. you can at least version control, quickly clone/rebuild servers, create new builds and in general automate most of the CI/CD process.

But do I recommend Jenkins? No.