Hacker News new | ask | show | jobs
by PMan74 2448 days ago
From an external viewpoint stuff like building your own internal chat application seems like a "too many engineers with too little to do" kid of decision.

https://eng.uber.com/uchat/

2 comments

The chat program they’re referring to in the beginning was HipChat, which really was a steaming pile of crap. Uber was our largest user and they constantly brought the entire service down.

IDK if making their own was the right solution, but it was driven by necessity and not boredom I assure you.

The largest user of HipChat, really? Uber must have been something like 5k-15k employees at that time. I thought HipChat was used in F500, much larger than that.

Anyway, HipChat was bought and killed by Slack. They avoided a dying product by accident.

Yes. It seemed like every single one of their drivers was getting messages from one of their chat rooms in HipChat. It was like 25k people in a single room iirc. They seemed to be using it for notifications or something?

At any rate HipChat was already dead prior to the sale. Atlassian made a new program from scratch called Stride which was its own little awful can of worms.

Literally nothing in the first paragraph is true. For driver interaction, if it wasn't a push notification in the app then it would have been some sort of driver operations community tool like zendesk or similar.

> It was like 25k people in a single room iirc.

This never happened. I literally could not have happened. Uber ditched Hipchat when it was around 10k employees. The largest rooms I ever saw were outages or eng rooms with a few thousand, most with the room muted unless they needed something.

Theres really no point litigating this 4 years later (who cares) but it did happen. Uber had bots of some type, thousands of them in a single room. They were the largest user of our v1 API, which was an ancient shitty PHP base, and whatever they were doing put an enormous load on our backend during peak traffic periods and crippled the entire service.

Seeing this behavior was as easy as doing a join in mysql for user ids and rooms. I mean it's likely you wouldn't have seen it, or the room, or had knowledge of it unless you weren't directly involved with whatever it was?

It's actually quite relevant. I've worked in mega corporation, 50k employees and above. The inability of third party software to cope with the amount of users is a major recurring blocker for using off-the-shelf software and a big driver for developing things internally.

Most open-source and commercial software advertise that they can scale, but throw the first 500 or 5000 users and it's breaking apart.

With that context, I believe you. At the same time, I'm going to go out on a limb and guess that no one at Uber realized this either.
> It was like 25k people in a single room iirc.

Build it, and they will come ... and do something you hadn't planned for.

So why didn't they switch to Slack, Mattermost, etc rather than build their own?
They didn't build their own. The just forked Mattermost and skinned it a bit. Also invested/spent/donated $10k to have a say in Mattermost's roadmap.

Why not slack TL;DR - Uber, surprisingly, is cheap about off-the-shelf solutions. Cheaper than they are about balloons anyway.

Slack was trialed twice. Once in late 2015, and there were some bumps. Rather than stick with it, Uber ditched it citing it "couldn't handle their scale". Maybe that was true at the time, but I had a distinct impression that that's what the group in tech services at the time _wanted_ as well. Their lead on it was very much the type of person that would argue that there's no difference between hipchat and slack. If you asked the right person, you'd get the real answer: slack costs more than hipchat. Besides, the only person they had to convince was directors that didn't use chat anyway. All the executive leadership used Hangouts too. So they just needed to show the price graph comparison to justify, but as far as I know no qualitative productivity cost/savings/improvements measurement was made or factored into this.

Fast forward to the second try, late 2017. Supposedly when Dara came in he asked the same "why not Slack?" question. The UChat team was pretty intent on doubling down justifying their product, but a Slack pilot was done anyway. The pilot was a massive success by all accounts and basically everyone that made the switch was quite happy with it. Then tech services ended the pilot, sent out a survey, and were never heard from again. Would frequently give spiky answers when people asked for the results of the survey as well. Again - ask privately, and the answer you'll get is that Slack was just deemed as too expensive.

In both cases - tech services claimed that Slack was too expensive because of a price that was linearly extrapolated from their website. A lot of people didn't buy this, because that's just not really how enterprise contracts work. But tech services isn't accountable to UChat users, just the eng leadership they report up to.

In any case - everyone* else is still paying the productivity cost of using UChat/mattermost. It's more reliable than hipchat, sure. But is's miles away from being as useful as Slack. Especially the mobile apps - mattermost is a low quality, utilitarian react native app.

*Except ATG. ATG uses Slack because they actually don't fall under the CTO's jurisdiction. And executive leadership all still presumably use hangouts, because the UChat mobile apps are too crap.

When Uber trialed Slack in 2015, there were clear performance issues. Uber was aware of the potential issues ahead of time and slowly onboarded individual teams. As the onboarding went on, functionality like searching for an individual user clearly deteriorated.

I left Uber before the late 2017 Slack test, so I know nothing about it, but I think "It costs too much" is a very fair objection to using a chat app, especially for an unprofitable company that would have to pay for >30k seats.

At this point, I'm sure that Slack has resolved these performance issues for the sake of onboarding all companies that are Uber-sized or larger.

TL;DR - the team that evaluated whether Slack would be good for the company also had the most to lose if Slack was selected over their own product.
Someone needed a promotion.