[mugsie]

I know everyone hates on Jenkins these days, but (at least for the workloads I am building) Concourse feels like a toy.

My issues:

- everything has to be in a docker container (not all things can run in garden)

- Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible

- each install of concourse may need an entirely different version of fly

- There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews)

Jenkins + Jenkinsfiles + Pipelines provides all of the good bits of Concourse, with none of the Pivitol enforced workflow + tools.

I get why CloudFoundry uses Concourse - the build process is so arcane that you basically need to run the full CI locally, but I really do not get the hype for other projects.

[samueloph]

Jenkins has some serious problems when you want to work with CD.

Its currently impossible* to use Jenkins in a CD environment where you want to deploy by tagging your git project[1] and allow rollbacks because Jenkins doesn't treat tag's hashes (only the commits the tag points to).

Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it.

It may look as a silly usecase, but i believe its one that mostly fits when you have a dynamic server farm (hosting on aws with autoscalling, for example) and don't want to auto-deploy on master commit.

[1]https://groups.google.com/d/msg/jenkinsci-users/mYxtDNMz1ZI/...

edit-> * aktually its not impossible, but pretty hacky and ugly, it would be nice to be able to rely on jenkins own tools for this.

[mugsie]

> Jenkins has some serious problems when you want to work with CD.

When you want to work with CD in a particular way.

A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged.

If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side.

> Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it.

Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item.

[samueloph]

>> Jenkins has some serious problems when you want to work with CD.

> When you want to work with CD in a particular way.

> A simple fix to the above issue would be to have a "production" branch. When you want to do a release, you merge to the production branch. A roll back would just be a revert of the commits since the last merge to production, and could even be tagged.

> If you are doing CD by allowing devs to tag random branches, cool, but I would not call it a "failure" on the Jenkins side.

Well, i must confess i agree with you and my first idea was this, i will probably end up pushing for that anyway, but i will have to convince some devs that they should not be afraid of git reverts. When i first mentioned the idea, some devs with deploy rights said they were skeptical about this because they were afraid of having a git revert shitstorm in a SOS situation.

>> Unfortunately, hack such a fix is not very pleasant either, because Java, and i don't think the Jenkins developers are interested in it either, at least not if just a few people ask for it.

> Its a webhook - that can be in any language, or if you use Git(hub|lab) a config item.

Not exactly, on this case github only lets me create webhooks with 'create' action, which is trigerred "Any time a Branch or Tag is created.", so new branches would trigger deploys.

[falsedan]

> When you want to work with CD in a particular way.

I think that I, as the user of the project, should get to decide how I want to deploy my software. So, yes, I do want to CD things in a particular way: one that matches my developer's expectations and my business requirements.

> CD

Jenkins isn't a deployment tool. It can act like one, but this (rollback) is a classic example of how it's a build, not release, tool.

To address OP's complaint: we rebuild steps from the previous pipeline to perform a rollback, and we have all our deployment logic in a separate system which works with commit IDs, if the previous pipeline has been pushed out of the list of recent pipeline runs.

[mugsie]

> I think that I, as the user of the project, should get to decide how I want to deploy my software. So, yes, I do want to CD things in a particular way: one that matches my developer's expectations and my business requirements.

Sure - it definitely is. But we all have to do that within the limitations of the tools we use. Jenkins makes an assumption (in the default mode) of a continuously moving HEAD - which may or may not be what you want. It doesn't make Jenkins bad - it makes it not suitable for your use case.

FWIW - adapting to using a moving HEAD is not that hard - `git reset <old-tag> && git commit -a -m "Revert deploy <failed-tag> && git tag <failed-tag+1> && git push` - but I totally understand why people would not like to use that style. It is a personal / culture choice.

[falsedan]

> i don't think the Jenkins developers are interested in it either

I keep running into this: we want to extend or change Jenkins behaviour, and since it's open-source, it's possible to get up-to-speed on the codebase & make that change. But getting it into a state which matches the mindset of the project maintainers is tough! Trying to convince someone that your use-case is not only permissible but is actually a good idea over ticket comments is not what we want to spend our time on.

[kohsuke]

I'm the creator of Jenkins.

Without fully knowing the context, I wonder if this is just a simple disconnect.

In Jenkins, we collectively cover wide-ranging use cases not by making one existing plugin (say Git plugin) do everything, but by making it extensible so that other people can define additional semantics and behaviours as separate plugins on top of it.

We learned this in the early days of Subversion plugin, back when that was the most popular version control system. Everyone uses a generic tool like that very differently, so as we kept adding individually valid use cases to the Subversion plugin, it became this giant hair ball not just for devs but also for users.

This mode also works better for those who do not want to spend time explaining why their use case is a good/important one, as they can simply code up their idea as a separate plugin and move on.

With all that said, I'm sorry for the frustration you had. Do you still have some pointers to tickets/PRs, etc? I'd like to look into it.

[falsedan]

Hi, thanks for responding!

Funnily enough, I'm specifically thinking of the Git plugin and issues like https://issues.jenkins-ci.org/browse/JENKINS-6124 & https://issues.jenkins-ci.org/browse/JENKINS-14572, and Nicholas de Loof's comments 'ur doin it wrong'. It's extremely difficult to extend this plugin; we have a lot of work tracking all the existing plugins we use, and adding maintenance of our own plugin on top of that is work we don't want to do.

Nowadays we try to keep as little as possible defined in jenkins configs and put our build & test logic in makefiles/tools.

[mugsie]

Yeah - it is a "problem"[0] with open source - you need to deal with people that may have completely objectives to you.

It is easier with a closed source / paid service to ask for feature requests. Remember that the maintainers probably get hundreds of similar requests a week, and have merged things in the past that they didn't like and later caused issues.

This causes confirmation bias for them "The last time we allowed code not aligned to codeing styles / test requirements / feature direction / $reason it all went badly", so it is easier to ask for things to align to their world view before merging it.

Being a maintainer is hard - you are damned if you do, and damned if you don't. People give out about not merging code, but they also give out if the code that does get merged breaks things. Finding a balance is hard.

0 - I don't personally think it is a problem, but it potentially requires a bigger time investment, and can be infuriating when you have a small change that really scratches an itch for you.

[jadeklerk]

> - There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews)

A CI that runs one job (test) for lots of branches is differnet than a multi-pipeline CI that runs lots of jobs for a single branch (which is what concourse aims to be).

[mugsie]

Sure, but if a job fails for a transient reason, there should be a way to retrigger it - that is just a basic feature of nearly all CIs, no matter how many jobs there is.

[jadeklerk]

Of course, and you can - you can manually hit the + button to retrigger a build, and if you want specific inputs you can select the exact one you want in the resources tab (which is slightly less common, since you probably haven't been committing since your build broke, but it's totally doable).

[joncampbelldev]

For a lightweight pipeline based CI tool (with no docker requirements) you may like buildkite. I use it for almost everything I used to do in jenkins, although I still keep jenkins around as a glorified cron job / easy generic task runner.

(not associated with buildkite, just a very happy customer)

[mugsie]

buildkite looks nice - but on premise is a current requirement, as not all our repos are on the internet. Looks like a cool product though

[joncampbelldev]

buildkite can be done entirely on your own infrastructure (for the purposes of source control and builds), its an interesting setup:

- the ui is SaaS, but it doesn't care about your repos or build agents

- the build agent runs on your machines (they communicate outwards only with the SaaS product to decide if they should start building)

- all pipeline config is kept inside your repo

- you can hook up any source control to connect to buildkites webhooks, they just happen to have an integration to the common SaaS source control tools (github etc)

A nice extra is that running your own build agents lets you keep costs down. If you are on AWS they have a "one-click" setup for a cloud formation template that gives you an auto-scaling build environment

[wlamartin]

> not all things can run in garden

Can you provide an example of what you'd want to do but you aren't able to?

[mugsie]

Booting VMs is kind of the main one.

[wlamartin]

Booting VMs on your worker seems like exactly the kind of debris concourse is encouraging the avoidance of by enforcing the containerisation of your builds. That said, I can't think of a reason why you couldn't boot a VM in a privileged Garden container, have you tried?

[mugsie]

Yeah - I can't remember the actual failure, but it did not work out well.

We also need the VMs to survive across build steps.

We are working on an P/IaaS - so integration testing requires actually booting a full stack, then running an install, then testing the resultant servers are actually running Kubernetes

[wlamartin]

Ah that's a shame, I still suspect it would _probably_ work with the right tweaking but I appreciate there's a hurdle there that might be frustrating to overcome and that's no good.

We also test a PaaS and require VM spin up but we call out to GCP/AWS as necessary.

[jacques_chester]

> We also need the VMs to survive across build steps.

You might represent them as a pool resource, so they can be safely handed from job to job. A lot of teams do this for stuff where they want to share an expensive, long-lived, stateful resource.

Disclosure: I work at Pivotal.

[mugsie]

The VMs are per job - once the CI run tests that the orchestration has created the right system config on the VMs, and that basic integration tests work, we destroy the VMs.

I have worked in two companies now with Cloud Foundry distros, and both of them have started using Concourse, and then moved to Jenkins (for non Cloud Foundry projects - I shudder to think what would happen to a distro that attempted to not use Concourse)

[smgoller]

I second what wlamartin said. We have pipelines that do integration tests that spin up 30 instance clusters in AWS. Trying to spin up something similar inside a concourse worker just isn't worth trying.

[gabrielr]

> Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible

Have you tried `fly watch`? It will stream logs from in-flight builds, or show the full output of completed ones.

[mugsie]

sure - I can use fly. if I got to the page, find the link to download the right version of fly, then remember the archine incantation to login, find a job, and find the logs.

Or - I can go to my Jenkins build page, and click "Show full log".

For me, I find the initial quick searching through logs much quicker in a browser.

And, I have found that on longer builds the web UIs log streaming can actually crash the browser.

[jadeklerk]

The arcane incantation to login? Seriously? It's literally fly login. You're being ridiculous.

[mugsie]

Well, to be clear, it's fly login --team-name internal --concourse-url https://<url> -k (I think)

Vs just going to the jenkins UI.

[jadeklerk]

It's fly login -t <my target>

You can save targets.

Also, if you want logs to appear somewhere you can pretty easily set up a after-fail job to ftp em somewhere.

[jadeklerk]

> - Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible

What? It's just STDOUT.. Are your logs going to a file or something?

[mugsie]

Well, yeah.

Different stages go to different log files, so you don't end up with a 10k log file.

[pokemongoaway]

Not everyone... Jenkins is probably the most versatile! I'm still a heavy user and enjoy using the official docker image these days :)