Congratulations on the launch, and for your support of open source projects! I'm interested in extending our bulk builds of the OCaml language community packages (https://opam.ocaml.org) to Windows. Do Azure Pipelines support Windows Docker containers yet? Having somewhere to robustly build native Windows images is quite difficult at the moment.
One of the other things on my wishlist is for some sort of standard for all these CI/CD pipeline agents. So far, on our build cluster we have BuildKite, GitLab, Azure Pipelines, <insert CI provider> agents, each of which have slightly different platform support. It's particularly challenging when trying to extend our CI to slightly less popular operating systems such as OpenBSD or exotic or newer architectures such as ppc64le or RISC-V. It wasn't clear from the Azure documentation whether or not the agent is open source or not.
Today, we use a combination of BuildKite and GitLab to coordinate the multi-OS, multi-CPU-arch Docker images that then get pushed to the Hub. The first provider to give us a one-stop coordinated multiarch build pipeline will save open source projects like OCaml so much trouble for testing and publishing our releases!
We absolutely support building Windows Docker Images. At the moment you are limited to either Server 2016 or Server 1803 based images but that is something we are working on enhancing. I would be interested in understanding your needs at a deeper level. Please ping me on twitter @chrisrpatterson if you are interested.
[Dave M - Azure DevOps PM] Also includes more free: 1,800 monthly minutes for private projects instead of 240 and 10 free concurrent jobs for CI/CD. GitHub marketplace extension for easier config from GitHub. But nearly the same functionally as Chris indicated.
Azure Pipelines has all of the same features on VSTS Build and Release but adds an option for defining your CI pipeline in YAML (this feature was previously in preview).
I'm currently reevaluating the CI/CD story for my company. We're a small startup with ~20 engineers. I would like to give you some feedback and questions on Azure Pipelines.
We're currently using gitlab.com for code hosting and CI/CD. We're in the Silver / Premium pricing tier so have most of the premium features. Our software stack is mostly on GKE Clusters on GCP and we also run gitlab runner in a GKE cluster.
We're honestly not super happy with gitlab.com due to performance, stability and UX issues - overall not super happy with the overall quality/depth of the product. That's why we're considering to make the switch to github.com and reconsider the whole CI/CD story. I evaluated about 20 CI/CD platforms, but unfortunately didn't find the perfect one. There was always at least one important feature missing or the management of the system (i.e. jenkins, concourse) would've been too much overhead for a company of our size.
My preliminary conclusion is that we should probably stay with GitLab CI/CD for now (even when switching to github.com for code hosting) since it is good on most dimensions (far from perfect though) and has low management overhead.
However I gave yesterday Azure Pipelines also a shot and liked many things I saw.
My impressions were like that:
Pros:
- Flexible build system / environment, can run on VMs, containers, self-hosted agents etc.
- Reasonably complex builds can be expressed in the yaml.
- Reasonable pricing
- Big plus: Very sophisticated release / deployment pipeline management. Nice sweet spot between solutions with no / limited CD support (e.g. circleci, gitlab) and complex systems like spinnaker.
- very good github integration (uses checks API)
Cons:
- Big con: Self-hosted agents cannot autoscale. For comparison gitlab runner k8s executor spawns kubernetes pods on demand so the infra / cluster footprint is really small in idle times during the weekend or nights. In comparison azure pipelines seems to only work with non-ephemeral agents and one has to set a fixed statefulset count: https://github.com/Azure/helm-vsts-agent
- Big con: Release Pipelines are only configurable via UI instead of code
- No explicit docker layer caching support (vs codefresh, circleci)
- Many out-of-box integrations / features are tied to Azure Cloud, for other clouds (i.e. GCP) one has to do write custom scripts or integrations.
The two biggest cons for me are really the non-autoscaling self-hosted agents and that release pipelines are only configurable via the UI. The release pipelines only-UI problem I consider a blocker for us.
In other words: If you would allow configuring release pipelines in code we would probably switch to github.com + Azure pipelines. FYI I prefer expressing complex pipelines in a real and common programming language like TypeScript or Python3 (think Jenkins: Groovy, Airflow: Python, TeamCity: Kotlin Script etc.) over YAML, even though I can live with YAML for now.
Is your team working on enhancements to any of that and what would be the rough timeline?
Thanks for the detailed and thoughtful feedback. I'd love to get on a call with you to discuss your needs and some of the items on our roadmap. If you're interested, email me at jepling@microsoft.com.
One of the other things on my wishlist is for some sort of standard for all these CI/CD pipeline agents. So far, on our build cluster we have BuildKite, GitLab, Azure Pipelines, <insert CI provider> agents, each of which have slightly different platform support. It's particularly challenging when trying to extend our CI to slightly less popular operating systems such as OpenBSD or exotic or newer architectures such as ppc64le or RISC-V. It wasn't clear from the Azure documentation whether or not the agent is open source or not.
Today, we use a combination of BuildKite and GitLab to coordinate the multi-OS, multi-CPU-arch Docker images that then get pushed to the Hub. The first provider to give us a one-stop coordinated multiarch build pipeline will save open source projects like OCaml so much trouble for testing and publishing our releases!