[goleksiak]

Heroku came back and said:

Looking through the system, I see that you were sent two emails (in August and September) as several of your apps were migrated to the new routing stack (https://devcenter.heroku.com/articles/heroku-improved-router). As mentioned in the documentation, the new router follows stricter adherence to the RFC specification, including sensitivity to spaces.

...and sure enough, there is a line that says:

The request line expects single spaces to separate between the verb, the path, and the HTTP version.

So the lesson is: RTFM

-G

[sixwing]

The team at Heroku (where I currently PM) is constantly trying to improve our communication and documentation. We're definitely sorry that this caused problems, and we'll work even harder to make sure that our communication calls out any potential issues. Again - thanks for reaching out to us, and let us know if we can help.

[goleksiak]

Heroku did their best here. They reached out to us (twice) advising of changes and linking to a document that describe EXACTLY the bug that we discovered (later). Honestly, I don't feel bad about his bug because even if I would have read the alert to the letter, we would not have audited the entire codebase because we don't have that luxury of time. Yes, it took our whole operation down...but we found it, fixed it and now we're back up. ...it's all in the game. -@eatabit

[jacquesm]

That's all true but once you start accepting illegal input on a protocol for a long enough time you can't just suddenly go and break things without an automated alert to the customer when that particular thing starts acting up.

After all it would not be that hard to scan for which customers are going to be bitten by that particular change when it actually happens rather than using some fire-and-forget email.