Hacker News new | ask | show | jobs
by lamontcg 1915 days ago
I'm kind of surprised that nobody is talking more about this right now.

Everyone with a Gemfile.lock that does a `bundle install` as part of autoscaling (without having vendored gems or a rubygems mirror which doesn't obey yanks) is now broken, potentially in production.
1 comments
quesera 1915 days ago
This is true, and important, but:

You should never depend on GitHub or RubyGems for deployments.

If your deployment failed today due to this gem yank, it has exposed a bug in your systems that you should fix.

EDIT: I should not speak in such absolutes. "Never" is a big word and clearly this does not apply in all cases! Depending on third-parties for deployments is a risk -- but might be tolerable, if a multi-hour outage would not be devastating.

link
lamontcg 1915 days ago
I addressed that VERY specifically in my comment:

> (without having vendored gems or a rubygems mirror which doesn't obey yanks)

The problem is that the author of the gem just forced a firedrill down everyone's throats today. Doesn't matter if they wanted to or not.

And in prior incidents admins who have taken the precaution of setting up rubygems mirroring and thought they were being responsible were embarrassed to discover that the gem yank was propagated to their own mirror.

Which is a lack of testing, but again, those deficiencies happen, and this is really forcing a firedrill on everyone, without any notification. And the author who did the yank was likely completely unaware of the blast radius of what that action would entail.

link
Toutouxc 1915 days ago
What's the solution? Having a mirror/archive of some kind of the gems I use?
link
quesera 1915 days ago
Something along those lines, yes. Mirror/archive/caching obviously requires setup and maintenance.

Vendoring gems works well if you (and coworkers) develop and deploy on the same platform.

A minimal approach might be to keep local copies/clones of all gems. If things blow up, you can always build and vendor any missing dependencies, and then redeploy. You'd need to keep a local environment available that matches your deployment env, for building native gems.

GitHub and RubyGems are very reliable, although of course not 100%. It's more common (but still rare!) that an individial gem owner will do something odd, or remove an artifact. Often, you can wait the issue out, or spend a few hours constructing a workaround.

But sometimes you cannot wait. And sometimes you don't get the chance to decide -- your deployed and running code will suddenly fail because an application in AWS or GCE needs to scale up with new instances, or your existing instances auto-update, or otherwise replace themselves.

If that would be a serious problem, it makes sense to invest time into reducing third-party deployment dependencies.

link