[john_horton]

Are you referring to this: "What kinds of experiments are likely to work well? Given current capabilities, games with complex instructions are not presently likely to work well, but with more advanced LLMs on the horizon, this is likely to change. I should also note that research questions like what is “the effect of x on y” are likely to work much better than questions like “what is the level of x?.” Consider that in my Kahneman et al. (1986) example, I can create AI “socialists” who are not too keen on the price system generally. If I polled them about who they want for president, there is no reason to think it would generalize to the population at large. But if my research question was “what is the effect of the size of the price increase on moral judgments” I might get be able to make progress. That being said, it might be possible to create agents with the correct “weights” to get not just qualitative results but also quantitatively accurate results. I did not try, but one could imagine choosing population shares for the Charness and Rabin (2002) “types” to match moments with reality, then using that population for other scenarios." --- To clarify, this about what research questions are likely to work well here, not what questions posed to LLMs will work well.

[jcampbell1]

By posing research questions, you get research conclusions from the same field of study. The whole thing is not a model of human thinking in the text world, but rather a model of economic research papers.

[john_horton]

I'm sorry I don't follow - is your claim, that, say, an AI agent exhibiting status quo bias in responding to decision scenarios (e.g., a preference for options posed as the status quo relative to a neutral framing - Figure 3) that the reason this happens, empirically, is because the LLM has been trained on text describing status quo bias? E.g., like if an apple fell to the ground in an game, it was because the physics engine had been programmed w/ laws of gravity?

[jcampbell1]

You are posing questions to the AI that only economists ever ask. You think you are instructing to it to reason “as a libertarian”, but you are actually using such economics lingo that the AI is regurgitating via “based on economist descriptions of libertarian decision making, what decision should the AI make.”

Imagine this scenario. You have a group of students and you teach them how libertarians, socialists, optimists, etc empirically respond to game theory questions. For the final exam, you ask them “assuming you are a libertarian, what would you do in this game?” Now the students mostly get the answers right according to economic theory. By teaching economic theory, and having students regurgitate the ideas on an exam, the exam results provide nothing new for field of economics. The AI is answering questions just like the students taking the final exam.

It would be like me teaching my child lots of things, and then when my child shares my own opinions, then I take that as evidence my beliefs are correct. Since I already believe my beliefs are correct, it is natural, but incorrect, to think the child’s utterances offer confirmation.

[john_horton]

Got it - so it is the "performativity critique" - the idea that the LLM "knows" economic theories and responds in accordance with those theories. I don't think that's very likely because a) econ writing is presumably a tiny, tiny fraction of the corpus and (b) it would imply an amazing degree of transfer learning e.g., it would know to apply "status quo bias" (because it ready the papers) to new scenarios. But as the paper makes clear, you can't use it to "confirm" theories but rather use it like economists use other models - to explore behavior and generate testable predictions cheaply that you can go test with actual humans in realistic scenarios. The last experiment in the paper is from an experiment in a working paper of mine. There's no way the LLM knows this result, but if I had reverse the temporal order (create the scenario w/ the LLM, then run the experiment), it could have guided what to look at. That's likely what's scientifically useful. Anyway, thanks for engaging.