[zkmon]

> Databricks’ multi-million line codebase

The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?

[trollbridge]

Because if they're like previous places I worked out with proprietary commercial codebases, the code is old, has been around a long time, and trying to shrink it is not a priority.

It's a good stress test for the LLM because it is not an "ideal" codebase.

[alexott]

:facepalm: "orchestration/management layer" that just runs dozens of millions of VMs per day, across the three clouds, with things like model serving, AI gateway, OLTP (Lakebase/Neon), Unity Catalog, etc.

Forget Databricks == Apache Spark...

[ozgrakkurt]

Old codebase, you always add code and never remove it. So it is expected to be like this.

Deleting code is difficult and almost never makes sense afaik

[tijs]

Every line you delete is a line you no longer need to maintain. We aggressively prune old code in our apps and it has definitely helped with maintainability. For a mobile app it’s also code you don’t ship so that’s a nice bonus which I guess is not much of an argument on a backend codebase

[wwind123]

On one hand, I understand that some old code is hard to delete because it's hard to detangle a lot of the legacy dependency. On the other hand, too much useless old code existing in the code base by itself could become a big maintenance burden for both humans and AI. In some cases at some point it might become more economical to just invest a bunch of resource to detangle the dependencies to be able to remove the old code.

[hasyimibhar]

At huge companies, it's hard to prioritize this because it's hard to pin dollar values to removing legacy code, while it's easier to show how building feature X will earn the company $Y amount of revenue. And because of that, there is also no incentive to do it, you don't get promoted by deleting old code, you get promoted by showing how your effort helped contribute to company revenue. At my previous company (100+ engineers, hundreds of microservices), teams that regularly clean legacy codebase tend to be platform teams (cost centers), while teams that struggle to get these prioritized are product vertical teams (revenue centers).

[appplication]

LoC isn’t a super helpful metric so I think the better question is why is the headline using it. I can say I’ve personally created about 200k LoC code in the last 5 years and most of that has some value. But it really doesn’t say might about how much value or really anything else meaningful.

[InsideOutSanta]

They're probably mentioning the size of the code base as an indicator for how difficult it is for an LLM to understand where and how to make changes.