[hangonhn]

Before we pile on the social sciences, maybe someone familiar with them can tell us why RCTs are so hard to do. There are likely other issues involved that make RCTs very difficult -- I'm guessing some ethical issues at least. I doubt social scientists and economists are just a bunch of idiots or charlatans. Likewise, the recent breakthrough wouldn't be such a breakthrough if it had been easier. I don't have an Economist account so I can't read the rest of the article. Perhaps that was illuminated in the article. Anyways, before we criticize another field, we should at least have a good understanding of it.

[jfengel]

You are exactly correct. RCTs are hard, not just for ethical reasons but also logistical ones. It's hard to get the money and authority to conduct an experiment in the first place, and it's often impossible to create a true control group.

"Hard" scientists like to pat themselves on the back for rigor, but they get that because they're studying comparatively simple things. Studying the lives of people is hard, but it's also important. It affects public policy, which in turn affects people's actual lives. That public policy gets created whether it's being studied or not -- the studies are hard, but they're better than guessing, and slowly they can build up a picture that makes them better. It's a bit like medicine: we're not going to stop treating people just because we don't understand the mechanism of action and can't guarantee that it will work.

This breakthrough is about finding ways to use the many villages found in poor countries to even attempt to do an RCT, and to come up with mathematical ways to account for the fact that the trials aren't really randomized. Aid had previously been given based on people's best guesses about what would work, which would maximize the value of the aid given if the guesses were correct, but it's hard to measure if it weren't. Aid has been beset by misguided theories and lack of measurement -- good intentions, but often ineffective.

[kortilla]

> It's a bit like medicine: we're not going to stop treating people just because we don't understand the mechanism of action and can't guarantee that it will work.

Yet medicine actually focuses on scientific measurement of effects. They don’t just throw their hands up and go, “experiments that affect people’s lives are too hard.”

[throwaway2048]

Except there is a direct, causal link between treatment and life and death, or at least quality of life.

Economic or social studies are a lot more nebulous.

[SiempreViernes]

Seems weird to accept that "quality of life" is a straightforward measure, but insists nothing of the like can be created for social studies.

[jfengel]

Right, and that's what this is about. They're doing the experiments. I didn't say they were too hard; I said they were hard. But it's early days of learning how to do experiments, much like medicine was not that long ago.

[hangonhn]

There's quite a bit of medicine that "works" but the specifics of why it work isn't well understood, especially in mental health. One of my friends who works in the mental health pharmacology field told me one of the challenges with the field is measuring the efficacy of those drugs. What do you do? Do you ask someone if they are feeling better or happier? Is that trust worthy? Or is it too fuzzy? Was it the drug that did it or something else? In that regard, they face similar challenges as the social sciences.

[appleiigs]

I have a subscription. The entire remainder of the article discusses it haha. Let see if I can paraphrase: 1) framing the economic experiment is difficult to prevent bias (my take: economics is not in a lab), 2) what works in Kenya might not work in Guatemala (my take: confounding factors are much greater), 3) ethics of withholding benefits to a group, 4) rich country researchers assuming they can and should intervene in poor country's problems, 5) rich country researchers don't have local context, 5) small experiments and small topics may not apply to or have an impact on a global scale which economics operates at.

[duxut_staglatz]

For essentially three reasons:

- ethics. Example: is democracy good for economic growth? Of course one could randomly engineer coups in some countries but that's probably not appropriate.

- cost. Example: how much do people change their labor force participation when taxes change by 1%? where a RCT would be "let's give a _lot_ of money to people" and see what happens.

- situation where it is not appropriate. Example: why did Europe rise to prominence (aka the great divergence)? There is not much to randomize here.

Note that RCTs have shortcomings anyway (see for instance [0]).

[0] https://www.nber.org/papers/w22595

[IAmEveryone]

Physics–the only science where reductionism ever really worked–has sort-of ruined it for all the others, where it (mostly) doesn't.

In economics, you are studying vast systems. For the majority of questions, it is impossible to isolate some part of the system and control and measure all the inputs and outcomes. That's probably obvious for macroeconomics: You can't have FED raise or lower interest rates based on a random number generator. And even if you could, you would still need a second United States to act as the control group.

It's mostly also true for microeconomics. Consider the difficulty of studying UBI. The largest such studies gave a basic income to a small African village, for a limited time of maybe two years. But the idea, and its opponents, mostly deal with the life choices people make, requiring essentially life-long guarantees. And even just knowing to be part of such a study, or continuing to live in a society that hasn't changed, is likely or at least plausible to change the outcome to render the study meaningless.

[mrandish]

> I doubt social scientists and economists are just a bunch of idiots or charlatans.

The vast majority are certainly not, however, idiocy or ill-intent are not required to fall prey to many common causes of inaccurate results. Smart people trying their best to do good work still frequently succumb to errors and this is especially true in the less 'hard' sciences.

That's why the push for increasing rigor with RCTs and other methods is important and necessary.

[SiempreViernes]

I guess RCTs are complicated to do because you often can't generate homogeneous control and treatment groups, so you are either forced to laboriously measure every relevant aspect of each group to standardise. Or to otherwise invent clever ways to do your treatment that ensures that most of the effects from unavoidable differences cancel out.

The thing is, these complication doesn't explain why nobody overcame them until Kremer, Duflo e. al. started their experiments in the 1990's. Their work appears to be a simple adaption of methods from other fields to studies in developmental economics, not any sort of technological development. (This is one of the earliest papers cited in the motivation provided by Nobel foundation: https://pubs.aeaweb.org/doi/pdfplus/10.1257/app.1.1.112 it does some linear regression at the most)

With creation of new technology ruled out as the blocker for performing the experiment, you are basically left with internal and external sociological explanations.