[mooted1]

I worked at Uber from 2014-2018.

Can confirm this. It was a very poorly managed org because it was extremely grassroots driven—leadership was underempowered to say no to front line teams or hold groups accountable.

The overlaps and poorly considered projects described above are not an exaggeration. Many of them were designed for building the lead engineer's open source brand, not for company needs. Half of the projects described above have since been canceled.

Google:

- Hyperbahn

- Cherami

- Schemaless

- Peloton

- uDeploy

- m3db

- piper

- XYS

Contrary to their glossy open source and tech blog presentations, these projects were highly contentious internally and widely viewed as inferior to industry counterparts. Each of these projects had 5-20 engineers working on them for over a year; many of them are being EOLed or phased out currently. And this is just the list that I can publicly talk about. This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.

[brownbat]

Ok, innocent question, having never worked in a giant tech company with a viable, popular product -- how do they deploy the massive amount of available engineering hours?

Do teams just work on random projects like the above while other people fix bugs? Is it like a million mini startups internally all pitching an idea to the CEO all the time?

Or do you need 90% of people focused on how to develop your product at first, then suddenly it flips and you need 90% of your engineers to balance enormous amounts of network traffic and data storage?

[0xbadcafebee]

A guess, from other companies: a manager is given some engineers, they ask themselves "What should we create?", they come up with an idea, make up a random business justification, POC it to upper management, get approval and headcount, and spend a year or two trying to prove it was worthwhile. Best case it works and gets used by other internal teams for 2-5 years, worst case the team gets broken apart and the project abandoned. (Mind you, nobody gets laid off, they just find other reasons to keep paying people who aren't making anything of value) If you have ever heard about it publicly, it was probably someone's pet project and they wanted their ego stroked, or to "get a win" and a promotion.

Many of the projects are just a response to some other internal problem, like "the teams aren't using a shared platform", so they build a platform for the platforms. Persistent problems get ignored because nobody wants to rock the boat. The joys of working in a large corporation...

[tootie]

I saw this at a decades-old retail company. New CTO brought a Silicon Valley perspectives to the company which involved hiring a load specialist engineers into newly created departments without any explicit business justification. Literally said that once they were hired they'd come up with useful projects. He also wanted to start exporting our in-house software as standalone products even when we were still running transactions through a mainframe. CEO bought it all hook, line and sinker.

[naravara]

> Literally said that once they were hired they'd come up with useful projects.

This is a great way to have engineers burn time on projects they think are interesting rather than projects that will actually have good ROI.

[iamasoftwaredev]

How do you balance avoiding this with wanting engineers who have ideas and agency?

[karthikb]

Bring engineers in to solve problems you know need to get solved. As they learn the system, its debt, the business, etc., listen to their ideas. For ones that sound good, give them the resources to develop a POC. For the POCs that work, give them the resources to take it all the way.

Larger companies with substantial profits can hire engineers explicitly to come up with POCs one after another.

[rockinghigh]

The idea and proof of concept often come before creating a new team.

[karthikb]

In general, initiatives tend to be top down while tasks tend to be bottom up. Politics between the "managers" and the "doers" happens at the interface of those two. What actually gets done and delivered ends up being some function of the top down, bottom up, modified by the political battle.

[sjg007]

All of those are plausible but typically you have core teams built around product lines, teams that build shared components. maybe have infrastructure dev and devops too. maybe a database or datawarehouse team and data analytics/ stats team etc.. a cyber security team. all of these will have some team charter for where they fit into the business and there may or may not be shared resources technical and otherwise. These are typical in post market fit companies that are now “giant”. But yes, first you need the product, everything comes from there.

[hinkley]

I wonder how conscious the decision is at a management level to tie up engineers with expertise in a particular domain just to keep them away from competitors.

I've seen a lot of vanity projects over the years, and often it feels like this, or a form of in-house retirement. That guy has done so much for us that now we just let him do whatever as long as HR is okay with it.

[roskilli]

Can confirm that your confirmation of M3DB being cancelled is incorrect and is actively being worked on at Uber and elsewhere.

I do agree there were definitely projects that were overly ambitious and optimistic that likely that should have been better planned. I think you are weighted firmly at one end of the spectrum however. I have some obvious bias being involved in one of the projects you outlined, but I still think you are blindly covering a swathe of projects in one go without any nuance at all.

[hintymad]

I think M3 is a necessary project for Uber. Open Source solutions like Graphite is really not designed for Uber's scale in this particular case. And yes,

[bogomipz]

What did M3 offer that existing projects at the time such as Prometheus and OpenTSDB did not?

The project page states "M3, a metrics platform, and M3DB, a distributed time series database, were developed at Uber out of necessity. After using what was available as open source and finding we were unable to use them at our scale due to issues with their reliability, cost and operationally intensive nature we built our own metrics platform piece by piece."[1]

[1] https://www.m3db.io/

[merlincorey]

Last I checked, M3 is open source.

[elvinyung]

Just curious, do you have a good idea of how many of these are still being actively developed? I remember seeing stuff like Ringpop and Hyperbahn get moved to the uber-archive github account.

I'm especially curious about Schemaless, since it seemed like at one point every company tried to build a NoSQL KV store on top of sharded MySQL, realized how janky it is, and are currently decommissioning it.

[Jinetejoio]

Cadence (cadenceworkflow.io) is a project originated in Uber that is worth mentioning. It's a workflow engine but with very unique capabilities. It has a lot of potential. I've heard of many companies using it, e.g. HashiCorp, Airbnb or Banzai Cloud.

The development team is pretty active. They've been doing an important effort to gain community adoption, e.g. persistency with MySQL, Postgre in the works, Prometheus style metrics, a proposal process or design docs. Very interesting.

[lhorie]

uDeploy and m3 are still widely used internally. Hyperbahn is dead AFAIK. Not sure about the rest.

[brmgb]

> in the case of the people who were laid off, cost to people's careers.

I fail to see how.

Considering Uber is reinventing the wheel a lot, the people who were laid off where probably working on some interesting problems. Uber is a fairly well known brand. Unless they are incredibly inapt at selling themselves, their career should do fine.

[khazhou]

It is better when you can decide, yourself, to exit to another company, rather than being pushed out and then having to find a new job. Certainly, a competent engineer SV will land on their feet, but you lose significant negotiating power when you're not actively employed.

[malandrew]

Unless you've got plenty of money and are funemployed and wealthy enough to turn down offers and move on if they are unsatisfactory.

You're only in a bad bargaining position if you need a job to pay your bills.

The key in the discussion with the recruiter is not to say that you are unemployed but to say that you're taking time for yourself and looking for opportunities that meet your professional desires and baseline financial expectations. The key is to make it clear that you have no financial needs, just financial expectations.

Be prepared to use the Noel Smith-Wenkle salary negotiation method.

[alasdair_]

>you lose significant negotiating power when you're not actively employed.

I took six months off after a large liquidity event and then went back into the job market. I don't think I lost any negotiating power - the trick was to make sure I times the interviews so that I'd have several offers at the same time and then could use the fact that company X was offering a certain amount to ensure I got the job at the company I liked the most.

I do just want to say one thing: most tech investors now think we are in a bubble. A downturn is likely at some point. Now would be a very good time to ensure you have six months of expenses in a savings account so you don't have to take the first job you get offered - I had to do that in 2008 after the financial crisis hit and my startup died and it wasn't a fun time.

[foobiekr]

To be completely honest, the parent poster is referring to the common experience of normal people. If you were in the position to take months off after a large liquidity event, that experience doesn't apply to you.

There is a similar effect people have when they reach a certain level of financial independence - not only are offers better, but negotiation is much easier. Companies react to confidence (whatever the source) the same way individuals do.

[Rapzid]

Probably not too many avg Joes doing skunkwerks at Uber and getting laid off..

[malandrew]

Every engineer on HN should be earning enough that they can put away 6+ months of expenses just in case.

[slykat]

Do you think top down strategy and control is important for large organizations? Most people idealize organizations that are grassroots drive so it's interesting to see how it didn't work in this case.

Also it sounds like Uber needs to fire it's CTO.

[devicetray0]

> I worked at Uber from 2014-2018.

Seems like you probably got a lot of shares and were lucky. Now that they IPO'd, do you consider that you came out ahead financially (compared to working elsewhere)?

[tempsy]

Just a note: Uber's valuation was higher in 2015 than it is right now...

[mlinsey]

No inside info on Uber, but it is common for options at private companies to have strike prices considerably below reported valuations, which is proper (and adjudicated by a third party) considering the reported private valuations usually based on preferred shares that come with extra rights. That said, I dunno what year Uber switched to RSU’s, which would make this moot.

[usaar333]

It was $40b at start of 2015 and was rumored to be at $50b mid 2015. It's $53b currently.

That said with dilution, it's likely 2015 share price is less than it is today.

[evgeniysharapov]

What's the point of a value if you can't realize it in any way ?

[rvz]

Well Uber IPO'd this year in May, so given that he/she left in 2018, I would count the parent poster as unlucky if they forfeited those shares after leaving.

[mandelbrotwurst]

Did they forfeit them? Or had they vested and in anticipation of IPO likely exercised either all or nearly all of their options and then sold post-IPO?

[fstuff]

Before the Susan fowler thing they had a rule that you had 30 days to exercise your options after you leave or separate as they called it. Since the company's value went up so much most people couldn't afford to pay the tax and because it was private you couldn't sell it (employee lock out is in November). Lots of people were handcuffed because of this, and I know a few that walked away. After the Susan Fowler thing they changed the option thing to like 7 years.

I think they moved to RSU's in mid 2015 or 16.

[beefalo]

With 4 years of tenure they should have vested much of their stock, but I'm not sure when Uber started doing grants rather than options. If they had options they may not have had the cash to exercise.

[southphillyman]

At least these projects were engineer driven and open source. It's not really unusual for projects to be abandoned or fail based off of some faulty forecasting by sales or product.

[thinkx]

what replaced Schemaless?

[aeyes]

The person is just listing tools which Uber could have not implemented because something similar exists. Schemaless was widely criticized, yet I don't think the public understands all the use-cases.

That said, Schemaless isn't going anywhere. Cassandra doesn't cover all the use-cases.

[iamasoftwaredev]

> This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.

Except the people who were laid off, if they have been there a while, got much deep knowledge than if they just glued together a bunch of AWS stuff.

Gluing together AWS stuff can get you a good job, but it's not the same as solving these sort of problems.

I'd rather spend 3 years working on this and get laid off than the way I spent my last 3. Guarantee I'd be in a much better spot in the job market too.

Instead I'm trying to figure out how the fuck I get my career back on track because I'm headed towards the sort of job I will hate every single day.

[hyperbovine]

> but it's not the same as solving these sort of problems.

But that's just it: the problems were not solved. Personally I would rather devote years of my career building something that actually works by standing on the shoulders of giants, than going full NIH and wasting those years on a clunky POS that will be mothballed before the decade is out.

But hey, I chose science.

[braythwayt]

And there, ladies and gentlemen, is the source of the problem: People who think, Solving a general-purpose problem that has already been solved by somebody else will do more for me personally, than using somebody else's solution to solve the problems my employer actually has.

I remember doing that too, once upon a time. I invented a templating system based on Lisp for building web apps, back in the days when there were servlets, but no Java Server Pages (or Handlebars, or jsx, or anything we have today).

The templating system was wonderful, and it did improve developer productivity for our group, but there is NO WAY WHATSOEVER that it paid back the time I put into it.

The business would have been much better off if we'd simply used what was available and spent the majority of our time solving business problems, not developer productivity problems.

Furthermore, my templating system didn't wind up being that special. You know why? Because I wasn't really solving a business problem with it, so I had no constraints upon my design. Without constraints, such things meander from shiny bauble to shiny bauble according to the engineer's whims and fancies.

When you're solving an actual problem, you have constraints in time, resources, and solution. This forces you to create actual value. That makes what you build actually useful. But when you're building a thing because you want to play with the thing, you have no constraints, and rather than freeing you to create something amazing, this simply sucks you into making something that is worthless for anything except résumé-building.

"Ah, but my résumé," says the narcissist who feels no obligation to create value for their employer. I have bad news, and I am speaking from experience. If you build a bunch of things with way-cool tech that created no value, you become an ignorant person's idea of a smart developer.

Sure, you can get hired by people who don't really understand how to manage engineering, and they will throw money at you, but their ignorance combined with your self-serving ways will result in another disaster. Lather, rinse, repeat.

Good engineering cultures will ask you about your projects, and they will ferret out the fact that your projects did not add value, and did not "move the needle." I am not hypothesizing this, I have gone to interviews and had the interviewers grill me on my vanity efforts.

You cannot pull the wool over the eyes of anyone who has been through a cycle like this and is determined to never have it happen again. You can put lipstick on a pig, but there's no way to hide the fact that all projects like this wind up becoming ad-hoc, informally-specified, bug-ridden, slow implementations of half of the thing you should have used instead of building your own.

Now all is not doom and gloom. For everyone who has done this, either deliberately or by being dragged into one of these white elephant projects, there is a way to supercharge your career going forward.

Instead of boasting about your ability to write Lisp-based templating systems, or synchronization protocols based on Operational Transformations, or whatever it is you built, try confessing to it.

Point out the engineering experience, but then be up front that the project was a failure/bad idea. Sure, point out that you learned some great tech. But spend much more time talking about how much you learned about failure modes for projects. Talk about the scars.

Companied worth working for are eager to hire people who know better. Position yourself as someone who has learned these lessons the hard way.

[iamasoftwaredev]

> Companied worth working for are eager to hire people who know better. Position yourself as someone who has learned these lessons the hard way.

Learning those lessons the hard way still requires doing those projects, which means you learn how the tech works, which means I still feel 100% justified in my previous statements.

I want to work on a project like this that actually has lasting value to the company. Actually does something novel enough that it's worth doing.

How do I get there? Is the only way to win the lottery and get placed there as a new grad/junior engineer?

Cause that seems like what you're saying.

[braythwayt]

My actual advice--for whatever it's worth--is that at the outset of your career, it's better to chop wood and carry water on successful projects in a good engineering culture, than to collect buzzwords working on projects doomed to fail in a weak engineering culture.*

I know it's very hard to get jobs in established good companies, but there are also quality startups, founded by people who have these kinds of scars already and won't go down the path of building things nobody needs.

Here's a specific tip:

When interviewing, there's a point where you get to ask them questions. Great! Ask about projects like these. Is the company working on any of its own tech that could otherwise be rented from Amazon or is available as open-source? Why? What is the expected benefit? If not, how do they manage development to stay focused on projects that deliver value?

By asking these questions you will identify engineering groups that are strong or weak, and you will also send them a strong signal that you care about success. Good companies will pick up on that signal and be more inclined to hire you.

JM2C. My advice plus $3.00 will get you a latte at Starbucks.

----

* This is not a dichotomy, of course.

[kazinator]

Someone had to build the "thing you should have used instead of rolling your own".

Most of the things we should use instead of building our own are either currently someone's shiny, golden side project with no business case, or else started that way.

[codesushi42]

Don't forget Slack and HipChat.

[fstuff]

Lol, people keep taking about Uber building internal products because 3rd party stuff can't "Uber" scale, I think the real drive to build internal apps is to save on licensing costs.

Take chat, hipcat is like 5-10 bucks per month per user. Uber has about 20k FTEs, 40k contractors all use chat. That's 3 to 600k per month just on chat.

Don't forget scale is expensive.

I can't imagine how much that many slack licenses would cost.

[sjg007]

I mean, they’d probably get a good discount.

[discodave]

Yeah but you dont need all those engineers if you're not re inventing every wheel.