[themgt]

We started off writing the bulk of our server setup/deployment automation in chef, and have since completely abandoned it

The core problems we had with chef were:

• worse than ruby: the chef ruby DSL is like some bastardized, crippled ruby - e.g. ruby_block{}, just uggh

• way too slow & resource intensive: chef itself uses a lot of memory and CPU, has a slow boot time, and does stuff like execute apt for each desired package on each node on each run. this might work fine if you're running on beefy physical/virtual hardware, but not for managing hundreds/thousands of tiny LXC containers that need to be scaled on-demand in seconds

• not self-hosted: chef seemed to have real difficulty closing the loop and being the thing that deployed and configured itself. there's guides online for scripting yourself up to a basic chef setup, but what if you want your chef client to bootstrap with some custom rubygems? back to bash scripting - and then how does that script get on each of your nodes? chef isn't intended to build/deploy itself the way it does the rest of your stuff

We've now transitioned everything to heroku buildpacks + a build server, which create self-contained "slugs" and therefore are self-hosting (i.e. the build server can build itself), and allows us to have a single build/deploy process for everything

https://devcenter.heroku.com/articles/buildpacks

[dikbrouwer]

While I'm not a Chef fanboy, far from it, some of these comments are just not true.

"and does stuff like execute apt for each desired package on each node on each run". No you just need to set it up correctly so that it keeps the timestamp of the latest apt-get update, and does not refresh it within the next x hours.

"not self-hosted". Somewhat agreed, although you can easily get around this with the excellent https://github.com/tobami/littlechef

[Tombar]

Would you care to explain/write about your experience and setup a little bit more in depth, we explore this path and found it really interesting.

[stock_toaster]

  >  what if you want your chef client to bootstrap with some custom rubygems?

knife bootstrap[1]?

[1]: http://wiki.opscode.com/display/chef/Knife+Bootstrap

[amalag]

Using Heroku buildback means you have shifted to using Heroku instead of self hosting?

[themgt]

No, we've combined bits & bobs that Heroku has open sourced (check ddollar's repos on github) with our own Ruby/CoffeeScript/Lua/Bash glue on dedicated hosting. There's a single OS image and compiled binaries for software stored in Ceph's S3-compatible distributed storage, and everything else is applications and buildpacks stored in git

The build server takes URLs for an app and a buildpack, runs the buildpack to do any dependency fetching/compilation/etc, bundles up the output into a .tar.gz file that contains everything necessary to boot up on the standard LXC image, and uploads it into the distributed storage. Then when we want to boot 1 or 2 or 10 of that app, we just grab the gzipped "slug" and boot it up on an ephemeral LXC containers (i.e. a single base image + temporary overlay file system)

This system can build a rails web app, a node.js app, other services like mysql/redis/nginx, and even the build server itself

So the bootstrapping is a little tricky, but e.g. the process to run a build may go git -> api -> mysql/redis -> worker -> build server -> ceph/s3, and each of those pieces themselves is built/deployed/managed the same way, which we've found to be a huge win for maintainability

[jtimberman]

"ruby_block" is a Chef resource that allows you to execute your favorite ruby code as a separate resource.

You can always drop to regular, plain Ruby, anywhere you like.

[pyritschard]

you might want to look into pallet do achieve what you want if you're not afraid of clojure.