[simonw]

Here's an example from today of something I just figured out.

I had Claude Code do some work which I pushed as a branch to GitHub. Then I opened a PR so I could more easily review it and added a bunch of notes and comments there.

On a hunch, I pasted the URL to that PR into Claude Code and said "use the GitHub API to fetch the notes on this PR"...

... and it did exactly that. It guesses the API URL, fetched the JSON and read my notes back to me.

I told it to address each note in turn and commit the result. It did.

If a future model changes such that it can no longer correctly guess the URL to fetch JSON notes for a GitHub PR I'll notice when this trick fails. For the moment it's something I get to tuck in my ever expanding list of things that Claude (and likely other good models) can do.

[sarchertech]

How is that an example of something you are doing that might be a superstition?

You asked it to do a single easily verifiable task and it did it. You don’t know whether that’s something it can do reliably until you test it sure.

An example of a possible superstitious action would be always adding commands as notes in a PR because you believe Claude gives PR notes more weight.

That’s something that sounds crazy, but it’s perfectly believable that some artifact of training could lead some model to actually behave this way. And you can imagine that someone picking up on this pattern could continue to favor writing commands as PR notes years after model changes have removed this behavior.

[simonw]

When I'm working with models I'm always looking for the simplest possible way to express a task. I've never been a fan of the whole "you're a world expert in X", "I'll tip you a million dollars if..." etc school of prompting.

I wrote up another real world example of how I use Claude Code this afternoon: https://simonwillison.net/2025/Oct/8/claude-datasette-plugin...

[sarchertech]

Those are some obvious potential superstitious incantations. They might not be superstitions though. They might actually work. It’s entirely feasible that bribes produce higher quality code. Unfortunately it’s not as easy as avoiding things that sound ridiculous.

The black box, random, chaotic nature of LLMs virtually ensures that you will pick up superstitions even if they aren’t as obvious as the above. Numbered lists work better than bullets. Prompts work better if they are concise and you remove superfluous words. You should reset your context as soon as the agent starts doing x.

All of those things may be true. They may have been true for one model, but not others. They may have never been generally true for any model, but randomness led someone to believe they were.

[simonw]

I just realized I picked up a new superstition quite recently involving ChatGPT search.

I've been asking it for "credible" reports on topics, because when I use that word its thinking trace seems to consider the source of the information more carefully. I've noticed it saying things like "but that's just a random blog, I should find a story from a news organization".

But... I haven't done a measured comparison, so for all I know it has the same taste in sources even if I don't nudge it with "credible" in the mix!

[sarchertech]

Great example!