[cbb330]

I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka.

Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.

Anyways, AMA

[hasty_pudding]

I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad.

The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing.

I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.

[makingstuffs]

Seeing you use 'Jugaad' make me unreasonably happy.

I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution.

Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.

[rrr_oh_man]

But are things ever permanent? Isn’t it to be expected that stuff can change in the future?

[hasty_pudding]

LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is.

Theyre still using python 2 and centOS in tons of systems.

They just started using github last year.

Their tech stack is easily over a decade old.

[Domenic_S]

> Theyre still using python 2 and centOS in tons of systems.

There's 1 service on py2, with a plan to migrate because of the move off of centos. centos deprecation is in progress.

[hasty_pudding]

> There's 1 service on py2

Are you sure about that?

CentOs migration has been in progress for years lol

[cbb330]

TIL jugaad haha.

Yea, give everything 3x estimate because everything that can break will break along the way.

Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.

[yellow_lead]

In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.

[dustingetz]

jugaad: https://en.m.wikipedia.org/wiki/Jugaad

[Aeolun]

> Unfortunately, 17min to build is pretty good.

My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that.

As opposed to making builds faster, or build infra faster/cheaper.

[zaphar]

There are mostly only a few ways to make builds faster.

* Ship less code (Very hard to get a large org to do)

* Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code.

* Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building.

These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.

[recroad]

Can you expand on "ship less code"? I think I'm agreeing with you but I'm taking it to mean ship smaller pieces of code, not overall LoC.

[zaphar]

I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome.

Sometimes doing the same thing with simpler components and fewer components is just better.

[Aeolun]

These kind of speedups at LinkedIn scale all fall off the chart at the bottom left of this scale:

https://xkcd.com/1205/

Nonetheless, leadership will never give it priority.

[cbb330]

The top 1/3rd of Eng don’t see it as immutable. But the stack can consist of 10 other teams tools, and you’re not rewarded for the amount of time you’re spending to fix other people’s stuff. So it goes on

[MisterPea]

Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality.

There was significant emphasis on code quality at least on the team I was on, and an ever improving culture.

I did work on one voyager task though and I remember it being a nightmare

[AndrewKemendo]

>Anyways, AMA

What precisely is LinkedIn trying to be?

Seems like it's turning into 2007 Facebook. Is that intentional?

[cameldrv]

They're trying to be a site that makes a lot of money from ads. To do that they need to get people to spend a lot of time there. To do that they need an infinitely scrolling feed that has addictive characteristics. What else are people going to spend a lot of time on there? Scrolling through their contacts?

Oh, you meant what are they trying to be that's helpful to the user? Doesn't matter.

[duxup]

I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users?

I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.

[digging]

> I suspect the user base has largely driven it.

In what way? Users/customers do not generally drive development. Indirect measurements of users do, such as measuring "engagement".

At best, what likely happened was A/B testing showed "what users want", which rarely and only by chance intersected with what users actually want, and instead showed over and over that socials media patterns (light and dark) hijacking attention drives engagement.

[duxup]

Oh I disagree greatly.

If people didn't want to use twitter, they wouldn't and it would be gone. But users do use it, and even the folks who tell me they're upset ABOUT Twitter, most choose to be there.

Whatever the reason they make that choice, at some point that's on them.

[digging]

If the choice is "use the platform everyone else is on" vs "don't exist", many people will choose the former.

That's not the same thing as wanting to use the platform.

[duxup]

If you feel like not being on twitter means "don't exist" as an individual... I feel like that also is a choice / lifestyle you chose.

I can't imagine letting a service like that define things for me like that.

[disgruntledphd2]

Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't.

Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).

[duxup]

I don't buy entirely into "FB did it" kinda thing. They found it ... but users LOVE it. That's a choice in the last part.

[digging]

Some users love it, the type who want to sell their every thought online to anyone who can tolerate them.

Other users (almost everyone I know) absolutely loathe it. It is hands-down the worst service I have an account with, but it's also practically required to get a job if you don't have lucky personal connections. I was hoping TFA was actually about leaving as a user, that it might be inspiration for me to free myself.

[disgruntledphd2]

They made loads of product changes to get people to engage in that way. As an example, showing who viewed your profile in the email vs needing to log in.

[bombcar]

I've always thought LinkedIn was just Facebook for Business™ - has it ever been anything else?

[duxup]

It was always to some extent, but the feed was a lot more formal IMO at the beginning. Content was more like actual articles and links.

But I think users just love / gravitate towards a more facebook style feed.

[bombcar]

Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment.

LinkedIn has slightly different style of spammer, but it's basically the end game for anything.

[cbb330]

Yes. We hired everyone from meta starting in 2020 and so far this strategy is holding true

[AndrewKemendo]

Interesting ok. Good luck

[pm90]

Why did the codebase end up like this? Is it a lack of hiring for platform or developer tooling teams?

[selfmodruntime]

It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.

[cbb330]

this plus lack of performance eval/culture/strategy to grow responsibly and enable future success

its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.

[marcinzm]

In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.

[z3dd]

I bet in most places it all starts with a new innovative system of (self) assessment that relies on number of features/VI/KPI tied to promotions.

[lvspiff]

https://xkcd.com/927/

[yellow_lead]

Whats part two of the story? Did LinkedIn move to React? Did the initiative fail, or is it ongoing?

[cbb330]

Preface: I’m not in front end or flagship teams so I’m saying hearsay.

The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance.

There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.

[yellow_lead]

That's understandable, thanks for the info!

[throwaway2day03]

dude, you are sharing way too much about us publicly. the status of restli -> grpc and talking about DMA? seriously?

you realize stylometry exists, right? even the phrase 'fan fair' is a strong signal.

[smsx]

Sounded like he he was talking about a public blog post right?

[cbb330]

https://www.infoq.com/news/2023/07/linkedin-protocol-buffers...

https://commission.europa.eu/strategy-and-policy/priorities-...

https://gdpr-info.eu/

[pcloadletter_]

This sounds like my exact experience at another big tech company