Hacker News new | ask | show | jobs
by weird-eye-issue 339 days ago
Great so your experiment just consisted of having an LLM hallucinate

That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything

They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which you had no way of proving since you can't actually run that code) doesn't really mean anything

It would be similar to having it respond in a certain JSON format, they are great at that too. Doesn't really translate to a real world codebase
1 comments
dotancohen 339 days ago 

  > That's not really an experiment is it? You basically just used them to create a hypothesis but you never actually proved anything
The experiment was checking how well another unrelated LLM could write code using the syntax. And then in the reverse direction in new sessions.

  > They're great at writing text and code so the fact that the other LLM was able to use that syntax to presumably write code that worked (which you had no way of proving since you can't actually run that code) doesn't really mean anything
Of course I could check the code. I had no compiler for it, but "running" code in one's head without a compiler is something first year students get very good at in their Introduction To C course. And checking how they edit and modify the code.

This isn't a published study, it was an experiment. And it influenced how I use LLMs for work, for the better. I'd even call that a successful experiment, now that I better understand the strengths and limitations of LLMs in this field.

link
weird-eye-issue 339 days ago
> And it influenced how I use LLMs for work, for the better

How so?

link
dotancohen 339 days ago
I let the LLM come up with all the boiler plate classes, functions, modules, etc that it wants. I let it name things. I let it design the API. But what I don't let it do, is design the flow of operations. I come up with a flow chart as a flow of operations, and explain that to the LLM. Almost any if statement is a result of something I specifically mentioned.
link