| This is the CRUX of the issue. Even with SOTA models (Sonnet 3.5, etc) - the more open-ended your prompt - the more banal and generic the response. It's GIGO turtles all the way down. I pointed this out a few weeks ago with respect to why the current state of LLMs will never make great campaign creators in Dungeons and Dragons. We as humans don't need to be "constrained" - ask any competent writer to sit quietly and come up with a novel story plot and they can just do it. https://news.ycombinator.com/item?id=43677863 That being said - they can still make AMAZING soundboards. And if you still need some proof, crank the temperature up to 1.0 and pose the following prompt to ANY LLM: Come up with a self-contained single room of a dungeon that involves an
unusual puzzle for use with a DND campaign. Be specific in terms of the
puzzle, the solution, layout of the dungeon room, etc. It should be totally
different from anything that already exists. Be imaginative.
I guarantee 99% of the returns will return a very formulaic physics-based puzzle response like "The Resonant Hourglass", or "The Mirror of Acoustic Symmetry", etc. |
Some examples:
- "Don't include pointless comments." - The model doesn't keep track of what it's doing as well, I generally just do another pass after it writes the code to simplify things.
- "Keep things simple" - The model cuts corners(often unnecessarily) on things like type safety.
- "Allow exceptions to bubble up" - Claude deletes existing error handling logic. I found that Claude seems to prefer just swallowing errors and adding some logging, instead of fixing the underlying cause of the error, but adding this to the prompt just caused it to remove the error handling that I had added myself.