[davidcelis]

Someone vibe coded a PR on my team where there were hundreds of lines doing complex validation of an uploaded CSV file (which we only expected to have two columns) instead of just relying on Ruby's built-in CSV library (i.e. `CSV.parse` would have done everything the AI produced)

[mandeepj]

That’s a good example of ‘getting a desired outcome based on prompt’ - use a built-in lib or not.

[vkou]

And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?

[mandeepj]

> what are the magic prompts that you give it that makes it stop trying to bullshit you?

Maybe keep your eyes open? :-)

[rsynnott]

Okay, so at this point it is strictly worse than just searching for and reading the very simple docs for the Ruby CSV parser, surely?

Because, as part of your verification, you will have to do that _anyway_.

[vkou]

As I thought.

And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify.

But I also don't have a magical lever that I can pull to make it stop hallucinating.

... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.

[manwe150]

Ask it to write tests, then let it run until the tests pass (preferably in a sandbox, far from your git credentials). It is quite good at developing hypotheses and tests for them, if that is what you explicitly ask for. It doesn’t have (much) ego, so it doesn’t care if it is proven wrong and will accept any outcome fairly if it is testable. Although sometimes it comes to the wrong conclusion and doubles down that the fact should be true so it prepares to write and publish a library to make it true

[mandeepj]

Sorry! Didn't mean to BS you. I've not come across a scenario where it hallucinated me with a non-existent library. Can you share what you were trying to do when that happened?

[vkou]

I wish I had the transcript. I don't, and I'm afraid that the passage of time has muddied the interaction to the point of uselessness (when it comes to listing specifics).

[7thpower]

I wonder how many times the LLM randomly tried to steer back to that library only to get chastised for not following instructions.