Hacker News new | ask | show | jobs
by Cyphase 91 days ago
There's a difference between "LLMs are inherently black boxes that require lots of work to attempt to understand" and explicitly changing how a piece of software works.

Should people not complain about unannounced changes to the contents of their food or medicine because we don't understand everything about how the human body works?
1 comments
ralferoo 91 days ago
Except the system prompt that gets prepended to your own prompt is part of the black box, and obviously should be expected to change over time. You are also told that you're not allowed to reverse engineer it. Even in the absence of the system prompt being changed, the output of the LLM is non-deterministic.

I'm not sure I understand your last analogy. How would changes to the human body change the contents of the food that is eaten? It would be more analogous to compare it with unexpected changes to the body's output given the same inputs as previously, a phenomenon humans frequently experience.

link
Cyphase 91 days ago
I think we're getting lost in the weeds. This has almost nothing to do with the LLM. It's about A/B testing. There's a piece of software where the behavior is being changed in unannounced and unexpected ways, at least as far as the author is concerned. The same criticism could apply to any other "workflow" or "professional" software.

There's some added flavor because the LLM is indeed non-deterministic, which could make it harder to realize that a change in behavior is caused by a change in the software, not randomness from the LLM. But there is also lots of software that deals with non-deterministic things that aren't LLMs, e.g. networks, physical sensors, scientific experiments, etc. Am I getting more timeouts because something is going on in my network or because some software I use is A/B testing some change?

link
skeledrew 91 days ago
> Except the system prompt that gets prepended to your own prompt is part of the black box, and obviously should be expected to change over time

You may want to review that statement.

https://github.com/Piebald-AI/claude-code-system-prompts

link