[brown9-2]

You have to feel comfortable that those people will generally give you good value for your money (since you can’t literally observe everything they do) and that they will tell you when something’s wrong as soon as they know, rather than covering it up.

I used to feel this way about Heroku, and I might again in the future, but I don’t right now.

I have a hard time understanding why, for all the money Rap Genius pays Heroku, they don't simply set up their own instances on EC2 and run the app there themselves. It seems like for a few days work with Puppet or Chef you could automate getting your code onto dozens of EC2 instances and installing the necessary tools/server processes, plus you don't have to complain anymore about how you can't run Unicorn.

Yes I get that there is a certain amount of value in being able to pay someone else to do all these things for you and saving time - but if you aren't happy with the result and the value given the money you are paying (and RG is not), then at a certain point it's time to just bite the bullet and fix things yourselves instead of continuing to be hamstrung by problems that the hosting provider won't/can't fix. There comes a point where you get large enough, and you are paying enough to Heroku, that it would be worth it to do things yourself and eliminate the problems.

[rubyrescue]

This is so true. The fact of the matter is that Rap Genius has obviously had to have someone spend a ton of time diagnosing problems with Heroku - and is objectively cheaper just to host some servers compared to Heroku dynos

This is why I always tell people that Heroku is actually NOT a good solution if you truly need scale. They're good for staging, launch, and an early traffic emergency or two. After that, ONCE YOU NEED TO SCALE, it's cheaper just to run your own servers, because the problem that Heroku is solving for you becomes a smaller and smaller percentage of your overall oeprations budget.

[brown9-2]

Also worth considering how much time of RG's has been spent not just diagnosing Heroku issues, but giving interviews and writing blog posts about the ordeal. Using Heroku might allow them to spend zero time on "ops" but they've spent some non-zero time now just talking about and raising awareness of this issue!

[snprbob86]

I had never heard of Rap Genius before this Heroku thing and their app is aimed at dissecting the types of textual messages that are being exchanged back and forth here. Seems like they decided to take the "pick a fight" approach to publicity quite literally...

Although I'm getting bored of this little scuffle, I am glad that they made some noise initially because it let me know that I wasn't crazy. I was trying to profile, understand, and optimize a Heroku app and ultimately gave up because it was relatively easy to migrate.

[jcr]

This reminds me of the quote, "All press is good press," but I can't recall who said it. You're right it's a waste of engineering time, but the shitstorm also has benefits.

[itsprofitbaron]

IIRC that quote evolved from Oscar Wilde's:

  "The only thing worse than being talked about is not being talked about"

[znowi]

I agree with this point, however, how Rap Genius spends its money isn't an issue here. Whatever the reason, they paid and expected to get an adequate service from the company, which they didn't. And on top of this, they found the shady practice at work. And this is a big fucking issue, if you ask me.

[JangoSteve]

I have a hard time understanding why, for all the money Rap Genius pays Heroku, they don't simply set up their own instances on EC2 and run the app there themselves.

Who says they won't do that now?

Obviously when they started, they had no idea they'd have these problems or that they'd spend so much time diagnosing them, because Heroku told them that they wouldn't have these problems to begin with.

[andrewvc]

Fact of a matter w/ anything outsourced is that you can outsource responsibility, but you can't outsource accountability.

Ultimately RG's devs are responsible for their choice to leave all the admin work up to heroku.

[oijaf888]

Yeah you would think the cost savings from EC2 and the 60K they spent on New Relic would cover paying for a quality sysadmin to run that stuff.