[Aurornis]

I have multiple LLM subscriptions at any given time, plus an array of local models.

When I ask a question outside of my domain of expertise I like to ask all of the LLMs I have access to. I also create separate sessions and ask the same question multiple ways.

It’s revealing to see how many different and contradictory answers I get, most of which are presented confidently.

The last time I ran a medical question through Claude I couldn’t even get consistent answers between sessions.

It’s also scary how easily you can lead each LLM to the answer you have in mind. When I would start asking questions about different options that other LLMs had presented, each session would drift toward that explanation.

[Esophagus4]

Have you ever let the LLMs “discuss” with each other to see if that would give better answers?

You might end up with the answer from the most persuasive LLM, but you might also end up with better results.

Wonder if there is a paper out there on this.