[brokencode]

Really depends on the repo you’re working in.

If it’s very large, especially if the tool needs to refer to documentation for a lot of custom frameworks and APIs, you often end up needing very large context windows that burn through tokens faster.

If it’s smaller or sticks with common frameworks that the model was trained on, it’s able to do a lot more with smaller context windows and token usage is way lower.

[Aurornis]

The codebase and the topic you're working on are huge variables.

I don't use LLMs to write code (other than simple refactors and throwaway stuff) but I do use them heavily to crawl through big codebases and identify which files and functions I need to understand.

Some of the codebases I explore will burn through tokens at a rapid rate because there is so much complex code to get through. If I use the $20 Claude plan and Opus I can go through my entire 5-hour allocation in a single prompt exploring the codebase some times, and it's justified.

Other times I'm working on simple topics, even in a large codebase, and it will sip tokens because it only needs to walk a couple files to get to what it needs to answer my questions.

[some-guy]

I'm currently in repos where the context window required is so large that the output is almost always "wrong" for the problem at hand. Quite a few people at my company burn through tokens this way, and it certainly isn't providing value to the company.

[AlotOfReading]

As always, improving accessibility for humans makes automation more effective. If the humans need to remember a PhD's worth of source code/documentation to contribute effectively, your codebase stinks.

[ivirshup]

People at my company have started writing docs specifically for claude. They're quite useful for me too, but kinda disappointing they never wrote these docs for their colleagues.

[staplers]

As someone who has written many docs, it's because 99% won't read it (rightfully so if it's verbose). You can turn that doc into a skill in a repo and Claude will read it everytime it's needed.

[dunham]

I recently saw this with the logseq api - the published api was an auto-generated stub. So I tried to grep the source code for the function and found detailed documentation written for claude. So I guess one benefit of all of this is that it's making people actually document things and maybe plan a little bit before implementing.

[bonesss]

I agree, in the general context of how I code.

The LLM hype train has me reflecting on what a spoiled existence working in a ‘proper’ language provides though…

React devs, JS devs, front-end devs working on large sites and frameworks might be triggering tens of files to be brought into context. What an OCaml dev can bring in through a 5 line union type can look very different in less token-efficient and terse languages.

[quaintdev]

Begs the question if we should move on to minimal microservices so that whole project lives in context of llm. I hardly have to do anything when I'm working with small project with llm.

[mlsu]

Why not take it a step further? Make each function in the codebase its own project. Then the codebase can fit into the context window easily. All you have to do is debug issues between functions calling each other.

[andai]

Wait, is this a joke about Lambda?

[ShyCodeGardener]

I don't think it's a joke about left-pad, but the idea that the complexity increases tremendously when you take a cloud of "small" things all communicating with each other. You've just pushed the complexity elsewhere. Claude can easily crunch the small microservice, but you're pushing the complexity to communications issues, race conditions, etc.

[andai]

Oddly enough I constantly run into the same issue on monolithic codebases too.

Things could just be one file but they end up being 12. I had to look through 12 levels of indirection for a single boolean recently. Twice, on two separate projects in the same week.

At least in a single codebase, that issue is at least theoretically solvable. At least the indirection wasn't split across 12 repos!

[dymk]

left-pad

[Aurornis]

In my experience, the result is just more crawling across the separate microservices and additional reasoning to confirm how it all fits together.

The monolithic codebases are easier to crawl for any problem that can't be conveniently isolated to a single microservice.

[phkahler]

A good API should be documented, and AI should not have to read the internal code to understand how to use it.

[Aurornis]

Like I said, if your work is already contained neatly inside one microservice then it doesn't matter.

The same would be true in a monolith: The context to understand what's happening would be contained to a few files.

When the work starts crossing through domains and potentially requiring insight into how other pieces work, fail, scale, etc. then the microservice model blows up complexity faster than anything, even if you have the API documented.

[gedy]

Sounds like tight coupling issue, not services per se

[dwedge]

Ironically this is accidentally begging the question - that breaking them up into LLM context windows would be good because it would be to fit them in LLM context windows.

Maybe you're right but I'm aghast at how much of engineering over the last 15 years has been breaking up working monoliths to fit better within the budget of an external provider (first it was AWS). Those prices can change.

There are good reasons to use microservices but so often they're used for the wrong reasons.

[hadlock]

I've done the opposite, moving multiple tightly coupled repos into a single monorepo. Saves the step of the llm realizing there's a bigger context, finding the repo, then also scanning/searching it. Especially for fixes that are simply one line each in two repos.

[seniorThrowaway]

I'm a fan of the monorepo in general, even before LLMs. If using git it leverages git's best feature IMO, the commit as a snapshot of the entire repo. I've worked on so many projects where tightly coupled things are split across repos because it's thought of as a best practice, and it just makes it more difficult to figure out what code you are running.

[swader999]

Generally speaking no. Treat your IP (the code that runs your business, makes your business competitive or special) as precious and don't make it subservient to infra. It should be in the format (code, architecture, structure) that best serves it.

[dwedge]

And yet so many companies spent the last decade doing it to fit into AWS pricing models

[giantg2]

Orchestration between those services and the integration testing for any reasonably complex change can still be quite large.

[Retr0id]

The whole service might fit in a context window but the details of the system around it will still be relevant.

[th0raway]

Yes, in a reasonable microservice land where the places you need to connect to are all documented in very concise places, you have have extremely productive $10 days. In the giant monorepo with everything custom, you can't just rely on built in knowledge of 80% of you libraries, so it's a very different world.

A place like Google has to be so much better off just training library concepts in, given how much of the things the LLM will "instinctively" reach for are unlikely to be available. Not unlike the acclimation period what happens when someone comes in or out of a company like that, and suddenly every library and infra tool you were used to are just not available. We need a lot more searching when that happens to us, and the LLM suffers from the same context issue. The human just has all of that trained in after a 6 months, but the LLM doesn't.

[skirmish]

> A place like Google has to be so much better off just training library concepts in

They did that, there was a special version of Gemini fine-tuned on internal code. But then the main model moves so fast that it is hard to keep such fine-tunes up to date and on the latest.

[andai]

On larger repos it spends a lot of time just finding the one line of code that needs to change. (I have the same problem, as a human!)

[conartist6]

So if the AI could do the same work on huge codebases with far fewer tokens, would it be good or bad for the AI companies do you think?

[anon84873628]

Unquestionably good. They want a product that provides value anywhere it's tried so as to establish the reputation as a magic human replacement. Gaming consumption based pricing at this point would be quitting before the race is over. They can always tweak the pricing knobs later once the industry is fully hooked.

[conartist6]

Right but what if the thing that made fewer tokens necessary also kneecapped the idea of making humans dependent on AI to write software.

[lukan]

It would be good for the first AI company offering this.

[conartist6]

Or an anti-ai company of course too; one whose goal was to level the playing field between humans and AIs again

[ok123456]

Will this result in people moving away from large monorepos to per-unit, quasi-micro repositories to save in token use?