[necubi]

We've been using Ubicloud builders for our Rust project [0] for several months, and it's worked very well. We've seen CI times go from 10-15 minutes to 6-7, and our bill has gone from $300/month to $30.

One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build.

[0] https://github.com/ArroyoSystems/arroyo

[crohr]

I love to see examples in the wild of repos having switched to a non-official GitHub Action runner. I'm slowly preparing a collection of repo timings comparing GitHub vs Buildjet/Warpbuild/Ubicloud vs my own solution RunsOn.

In your case your workflow can run in less than 5 minutes on AWS ephemeral machines, for the same price as ubicloud: https://github.com/runs-on/arroyo/actions/runs/7723361513/jo...

[pdimitar]

FYI, I'll be following your repo with a keen interest from now on.

[csdvrx]

> One counter-intuitive thing we found is that it's slow to save and restore caches, but the machines have good CPU, so for us it's been faster to disable cache entirely and just redo everything on each build.

The link to SPDK was very interesting: https://www.ubicloud.com/blog/building-block-storage-for-clo.... I use filesystems for very high performance applications, and I've found ZFS to often be the limiting factor when compared to simpler solutions of XFS +- mdadm +- encryption.

It's a controversial point, but others have made similar findings: https://klarasystems.com/articles/virtualization-showdown-fr... : "Although I suspect this will surprise many readers, it didn’t surprise me personally—I’ve been testing guest storage performance for OpenZFS and Linux KVM for more than a decade, and zvols have performed poorly by comparison each time I’ve tested them"

OpenZFS seems to be starting to consider optimizations to better perform on modern drivers (SSD, NVMe) which have very different performance profiles to what ZFS was built for (spinning rust)

In the SPDK summary they say "To make VM provisioning times go faster, we changed our host OS from ext4 to btrfs" (...) "Also, when we switched the host filesystem to btrfs, our disk performance degraded notably. Our disk throughput dropped to about one-third of what it was with ext4."

Ubicloud: the problem seems to be generic to CoW filesystems, and it's interesting you came with a slight variation (CoA) but have you considered the even simpler alternative of any journaling filesystems (XFS, Ext4...) with overlays?

Or just UFS2 + snapshots to restore from a given state (initialized, ready for each test) then restore to this state between tests?

I think customers finding that disabling cache works better means the CoA has similar issues to CoW.

Personally, I'd have just tried to using SR-IOV with a namespace per customer, and call it a day instead of bringing extra complexity, but there must be good reasons for it. I'd love to know what these reasons are.

[necubi]

In this case, I don't think the issue is due to filesystem performance. Someone from Ubicloud can correct me, but my understanding is that for custom runners Github still stores the cache on their side. So Ubicloud (in Europe) needs to transfer the cache from Github (in the US) on every run.

[pwmtr]

Hi, I work for Ubicloud.

Yes, that is correct. We are also working on implementing our own caching, which should speed up cache downloads/uploads significantly.

[MuffinFlavored]

> so for us it's been faster to disable cache entirely and just redo everything on each build.

I wonder what the consequential "carbon footprint" of this is, but at scale for all companies/all jobs of similar nature

[ploxiln]

Also consider the carbon footprint of using caches! Apparently it takes longer, because it has to send/recv more data, compress and decompress, cause load on other systems ...

It's really kinda impossible to judge the carbon footprint for these kinds of things, and if it's justified. Consider the carbon footprint of all the dumb AI feature rolled out at Facebook, Google, etc. Consider the carbon footprint of everyone trying to use ChatGPT now. Do you know how much power GPUs are guzzling to give each little answer? It's really huge, compared to a traditional google search! Is it worth it? ... who can judge eh

[sophiabits]

There’s a French think tank called “The Shift Project” which actually produced a report estimating CO2 impact of data transfer a few years ago [1]!

The numbers are obviously very rough; there’s a LOT of factors to consider and which vary from one node to another. But the methodology is quite comprehensive, e.g. they factor in power consumption of the end user’s device while waiting for the data to transfer over the wire

Green software engineering is interesting to me because it seems like a good way to impact greenhouse gases _without_ requiring consumers to change anything about their lifestyle (which is the hard part of climate change…) There’s a cool presentation from Rasmus Lerdorf from around the time when PHP7 released where he estimated a 50% adoption rate of PHP7 would result in a saving of 3.5B kg CO2/year iirc, purely because of the compute efficiency gains from PHP6 -> PHP7.

I used their numbers to calculate that swapping our CI/CD clones at my day job over to shallow clones saved (maybe) 5.5 kg/week of CO2 emissions [2]. Not quite as impressive as the PHP7 figures, but I still think it’s kinda neat.

[1] https://theshiftproject.org/en/lean-ict-2/

[2] https://sophiabits.com/blog/the-cost-of-bandwidth

[okeuro49]

It was PHP 5.4 -> PHP 7.0. There was no 6.

[crohr]

One of the goal of my own hosted runner project [1] is to display the carbon cost and money cost after each run. Too many people are oblivious of the costs and I think as an industry it would be great to at least get the data points.

[1]: https://runs-on.com

[boundlessdreamz]

This is pretty much exactly what I wanted (yay for custom images) but the pricing makes it a non starter. Please consider a tiered monthly pricing based on build minutes.

[crohr]

I don't understand what makes the pricing a non starter. You mean the license pricing? If so, 300€/year for something that will save you thousands doesn't seem too expensive to me. If you're talking about the instances, it is 10x cheaper than comparable GitHub runners (i.e. same as ubicloud), so not sure what's wrong here :)

[Arelius]

I mean, you could also consider how much the transition to Ephermial builders and containers is costing us in general, moving from (admittedly more brittle) build machines that would keep build artifacts alive on the local hard drive just as matter of fact.

[babanin]

Thanks for sharing! I've been looking at the repository and noticed that some jobs are still running on Github hosted runners. What is the point of using them and not running everything on Ubicloud?

[necubi]

We have one very slow job (our Rust CI, for example: https://github.com/ArroyoSystems/arroyo/actions/runs/7702793...) and a bunch of little jobs that take a few seconds (checking lints, etc.). We never bothered to switch those over because they complete quickly on github and fit within our included runner minutes.

I should say we also use BuiltJet for docker builds because of their arm support. Now that ubicloud has arm we may switch those over as well.