[motorest]

> Even with typescript Claude will happily break basic business logic to make tests pass.

It's my understanding that LLMs change the code to meet a goal, and if you prompt them with vague instructions such as "make tests pass" or "fix tests", LLMs in general apply the minimum necessary and sufficient changes to any code that allows their goal to be met. If you don't explicitly instruct them, they can't and won't tell apart project code from test code. So they will change your project code to make tests work.

This is not a bug. Changing project code to make tests pass is a fundamental approach to refactoring projects, and the whole basis of TDD. If that's not what you want, you need to prompt them accordingly.

[DecoySalamander]

> This is not a bug

It's not a bug if we're talking about a mischievous jinn granting wishes instead of a productivity tool.

[desdenova]

A jinn granting wishes is the best analogy for LLMs I've seen so far.

[Sharlin]

Ecept that LLMs aren't mischievous. They're just stupid.

[didgeoridoo]

Can’t it be both?

[Terr_]

> It's my understanding that LLMs change the code to meet a goal

I assume in this case you mean a broader conventional application, of which an LLM algorithm is a smaller-but-notable piece?

LLMs themselves have no goals beyond predicting new words for a document that "fit" the older words. It may turn 2+2 into 2+2=4, but it's not actually doing math with the goal of making both sides equal.

[motorest]

> I assume in this case you mean a broader conventional application, of which an LLM algorithm is a smaller-but-notable piece?

Not necessarily. If you prompt a LLM to limit changes to some projects or components, it complies with the request.

[bitwize]

> It's my understanding that LLMs change the code to meet a goal, and if you prompt them with vague instructions such as "make tests pass" or "fix tests", LLMs in general apply the minimum necessary and sufficient changes to any code that allows their goal to be met.

"I'm Mr. Meeseeks! Look at meeee!"

[brailsafe]

Do you mean not just LLMs, but agents? Is this jot avoided by narrowing your scope and just using the chat interface that also may not produce what you're hoping for, but at least can't muck about in your existing code?

[winrid]

I told it to add a feature and to update the tests. It added the feature, and then removed it because it made the tests fail lol. I know I can make it work, I did, that's not the point.

[chuckadams]

Fixing bugs is also changing project code to make tests pass. The assistant is pretty good at knowing which side to change when it’s working from documentation that describes the correct behavior.

[desdenova]

That's the main problem with vibe coding.

The whole point is having the LLM figure out what you want from vague hand-wavy descriptions instead of precise specification.

You don't need an LLM to parse a precise specification, you have a compiler for that.

[chuckadams]

It's entirely possible to have specifications somewhere between "vague hand-wavy descriptions" and source code. But it's really not my job to defend AI against all the people who want it to be completely useless, seem to need it to be so, really. I just use it, it works a lot of the time, doesn't work other times, and that's that. Results carry more weight than opinions.

[motorest]

> That's the main problem with vibe coding.

It's not a problem. It's in fact the core trait of vibe-codig. The primary work a developer does in vibe coding tasks is providing the necessary and sufficient context. Hence the inception of the term "context engineering". A vibe coder basically lays out requirements and constraints that drives LLMs to write code. That's the bulk of their task: they shift away from writing the low-level "how" to instead write down the high-level "what".

> The whole point is having the LLM figure out what you want from vague hand-wavy descriptions instead of precise specification.

No. The prompts are as elaborate as you want it to be. I, for example, use prompt files with the project's ubiquitous language and requirements, not to mention test suites used for acceptance tests. You can half-ass your code as much as you can half-ass your prompts.

[immibis]

Sounds like a compiler with extra steps.