[teich]

This is Oren Teich, I run Heroku.

I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information as soon as possible, including a blog post on http://blog.heroku.com.

[doktrin]

Thanks for the response, but I have to admit that the lack of a clear-cut answer here is a little worrisome.

Anyone who wants to like Heroku would hope that the OP is flat out, 100%, wrong. The fact that Heroku's official answer requires a bit of managing implies otherwise.

On a related tangent, I would also encourage future public statements to be a little less opaque than some Heroku has put out previously.

For instance, the cause of the outage last year was attributed to "...the streaming API which connects the dyno manifold to the routing mesh" [1]. While that statement is technically decipherable, it's far from clear.

[1] https://status.heroku.com/incidents/372

[toyg]

Maybe it doesn't need "managing", Oren might just want to talk with whoever was responsible for the change and see what the best way forward is. I don't think panicked, knee-jerk reactions like "OMG we were wrong and will revert that commit pronto!" are beneficial in situations as complex as this.

[doktrin]

You're assuming that the change was actually made. Until we hear definitively from Heroku, the only evidence is an (admittedly, well documented) blog post.

[toyg]

Yeah, absolutely. I'm just saying we can't expect a to manager to immediately respond to a highly-technical issue questioning subtle changes in internal behaviour which might have been introduced years ago.

[bambax]

What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)?

What we want to know:

- is the OP right or wrong? That is, did you switch from smart to naive routing, for all platforms, and without telling your existing or future customers?

- if you did switch from smart to naive routing, what was the rationale behind it? (The OP is light on this point; there must be a good reason to do this, but he doesn't really say what it is or might be)

- if the OP is wrong, where do his problems might come from?

- etc.

[shabble]

>> I am committing to the community to provide more information as soon as possible, including a blog post on http://blog.heroku.com

> What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)?

I think OP is saying 'I am going to investigate the situation; when I am finished here [the blog] is where I will post my response', not that there is something there already.

That said, it's all a little too PR-Bot for my taste (although there's probably only so many ways to say the same info without accidentally accepting liability or something).

[mcguire]

Note: I think we have different referents for "OP" here; bombax's is, I think, the whining customer; while shabble's is the pompous CEO.

Me, I'm the swarthy pirate. Arrrh.

[bambax]

Upvoted, although it's bambax, not bombax ;-)

[mcguire]

I'm having trouble reading around the eye patch.

[eli]

What's the point of posting a link to the front page of your blog

Well, he promised a detailed blog post, at which point that link will be extremely helpful.

I do not think it is fair to expect an immediate detailed response to those questions. If I were CEO of Heroku, I wouldn't say anything definite until after talking to the engineers and product managers involved--even if I was already pretty sure what happened. The worst thing you could do at this point is say something that's just wrong.

[bambax]

I don't expect an immediate response; I would have been happy with just: "This is Heroku's CEO. I'm on it."

But a link, that doesn't point anywhere useful, introduced by a PR phrase that sounds a little like "Your call is important to us", was a little annoying, esp. after reading the OP where they say they have contacted Heroku multiple times on this issue.

[eli]

I guess it's a matter of perception. I thought "I'm on it, expect an update later" is what he said.

[praptak]

> if you did switch from smart to naive routing, what was the rationale behind it?

Most probable cause: smart routing is hard to scale. Multiple routers, with each one doing random distribution independently of others will still produce a globally random distribution. No need for inter-router synchronization.

If multiple routers try smart routing, they must do quite a bit of state sharing to avoid situations where N routers try to schedule their tasks on a single dyno. And even if you split dynos between routers then you need to move requests between routers in order to balance them.

[character0]

While I think it is appropriate for Heroku to respond to this thread (and other important social media outlets covering this), linking to a blog without any messaging concerning your efforts might not be the greatest move... This may not be a sink or swim moment for Heroku, but tight management of your PR is key to mitigating damage. Best of luck, Heroku is a helpful product and I want to see you guys bounce back from the ropes on this one.

[csense]

Telling people where to look for a reply when they have one is a great idea, IMHO.

[teich]

I've posted an update on our blog, with another to follow tomorrow:

https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe...

[GhotiFish]

I'm looking forward to hearing why Heroku is using such a strange load balancing strategy.

[tjbiddle]

Looking forward to your blog post. Hoping things get cleared up!

[willvarfar]

hint just use a rabbitmq queue or something. Don't have a 'smart' LB that has to know everyone's state; instead, have dynos that get more work as quick as they can.

[praptak]

MQ might be a solution but certainly not in the "just use" class. Unless you want to introduce a bottleneck and a single point of failure, this queue has to be distributed.

Managing a distributed queue is hard, for reasons similar to ones making the original problem hard - DQs require global state in a distributed environment. There are tradeoffs involved - the synchronization cost might become a bottleneck in itself.

Pushing the problem on the distributed brokers is making a big bet on the queuing solution. Nope, definitely not in the "just use" category.

[willvarfar]

yes, I know all the ins and outs.

But they will end up building a pull rather than push system in the end.

[avodonosov]

I hope the solution will not break the possibility for multithreaded apps to receive several requests

[sylvinus]

Me too. I see this as a side-effect of Rails single-threaded craziness, our "modern" Node.js apps run faster than ever.

[antihero]

Can "Dynos" serve multiple requests simultaneously? That's the question, really.

[neilmiddleton]

That's up to the process you have running on the dyno