[zamfi]

Hmm. Are you familiar enough with research in economics, or the hard sciences, to say that all the work in those disciplines lack an emphasis on “generalizable findings”?

It feels like epistemic weaksauce to claim that entire fields explicitly reject the goal of generalizable knowledge because they question or reject “the scientific method” on the basis of “I’ve read some case study / qualitative papers”.

[UncleMeat]

It is always amazing to me how many people are willing to say "wow this entire academic field is just garbage and the researchers are making some incredibly childish error" based on what seems just entirely like vibes.

Psych is getting a brunt of this because they have actually done a good thing and funded replication studies, which naturally will produce a bunch of "this fails to replicate" findings. So then you get headlines talking about the replication crisis in psych and then people who maybe took a single class in college a decade ago dismiss the entire field as bogus.

I invite such people to go speak with some CS academics for a better understanding of the mess in our own field.

[mistermann]

The notion that rejecting the scientific method is necessarily bad thing is epistemically unsound, and a rather provocative topic.

[peatmoss]

And yet I did not state this, and it seems you're rounding my position to an easier one to dismiss.

I don't reject non-scientific scholarship. But I see the focus in some disciplines a little like twinkies and soda: a bit of them can be fine and maybe even be good in some circumstances (brings joy; maybe as a recovery item for some diabetic conditions). But my feeling is that some of these disciplines have indexed a bit too much on twinkie and soda.

[mistermann]

I think we are very much on the same general page!

[peatmoss]

Ah, I think I misread the intent of what you wrote—apologies!

[mistermann]

Not entirely I bet!

I think of science in much the same way that you think of other disciplines.

[goatlover]

Do you have another method in mind that is epistemically sound other than the scientific method?

[mistermann]

I'm more interested in broad capabilities, not only just epistemic soundness.

[peatmoss]

Sorry, I missed your comment / didn't realize it was directed specifically at me. So, if you go back and read what I wrote:

1. I definitely did not say that economics does not strive for generalizability. Quite the contrary. I think the vast majority of economists strive for generalizable conclusions (i.e. go to great care to find study samples that are representative of the population of interest and use statistical methods that might allow them to plausibly conclude things about that group)

2. I never said "all" work in other social science disciplines rejects generalizability as an aim. However, I do believe, that more "empirical" scientific method based studies in a number of disciplines would be good. That is predicated on 1. a belief about the prevalence of qualitative/theory based scholarship. 2. a normative preference that the ratio is undesirable. I could be mistaken on my perception of the first point, but I don't think I am. On the second point, you're welcome to disagree. These are, like, opinions, man!

3. "reject the goal of generalizable knowledge because they question or reject 'the scientific method'" makes me think you've either missed my point, or you don't understand the nature of other forms of academic scholarship.

I'm not leveling a diss that e.g. a participant observation case study isn't a method intended to generalize from. That's just an intrinsic feature of that kind of study method. Of the qualitative researchers I've known, I can't imagine any thinking there is anything controversial about what I've said on that point. Though, some would definitely disagree with me on my opinions about what ratio of scholarship should be of this kind.

And lastly you've selectively quoted me there at the end, and attached a conclusion to it that is not mine. Consider this a bit of original qualitative research on my part: a sampling of non-empirical research that I've read in recent years has suggested to me a lapse in rigor in certain disciplines. But this conclusion is grounded in my "situated knowledge" of the space, and thus shouldn't be used to generalize without a suitably operationalized quantitative study ;-)

[zamfi]

I'm sorry, but the thrust of your earlier post was absolutely that the social sciences "have taken a pretty hard turn" and "might benefit from a renewed emphasis on methods that can result in generalizable findings", and that "a correction is warranted".

Pragmatically speaking, I'm unsure why you would choose this language if you wished to convey the nuance that qualitative methods are actually great, that you're merely wishing that more "empirical" studies would also be undertaken, where "empirical" I guess means "quantitative" and "rigorous" though you don't make that explicit.

For what it's worth, you absolutely can generalize from an observational case study. RCTs are not the only way of drawing generalizable conclusions—it depends a lot on what your epistemic goals are.

It kind of sounds like you don't like that some social sciences rely more on non-quantitative methods because you don't think those are definitive. That's fine, you're welcome to hold that belief, but let's not pretend like you're a fan of all methods and just wish there were a few more quantitative studies in sociology (or whichever discipline).

[peatmoss]

No need to be sorry, I think you're letting your priors overwhelm your understanding of what I'm writing. What if I don't think that these methods are irredeemable or useless or epistemologically bankrupt, and STILL think that a bunch of disciplines in the social sciences over use them and could benefit from a course correction?

Does that resolve what I think you are perceiving to be a contradiction, but that I do not see as a contradiction?

Where I think you and I have a fundamental disagreement is in the nature of e.g. qualitative research methods such as case studies and whether a case study is generalizable. This feature was taught to me by... qualitative researchers. If you truly believe a case study is a generalizable research method, then either you are defining generalizability in a different way than is typical, or you hold a minority viewpoint not shared by mainline qualitative researchers.

Not all research methods need to be generalizable to have some scholarship value, and I'm not using the concept of generalizability as a colloquial stand-in for ~"bad"

BUT, I still think social sciences could use fewer qualitative studies and pure theory grounded scholarship (i.e. "a correction is warranted"). It's what I said from the start, it's what I mean now, and unless I someone provides me a compelling rationale to change my mind or the world changes, it'll probably be what I believe going forward.

It seems important to you to attribute to me a total rejection / disparagement of these methods. I am just not saying this. I do not reject these methods, just as I do not reject apple pie. I just don't think that a steady diet of apple pie is good. And I don't think the research methodology diet of many social sciences is healthy either. A correction is warranted.

[zamfi]

Hmm, alright, sure: Pick a field. What would the shift you are proposing look like in that field? What’s the ratio now between qualitative work and quantitative work, and what ratio would you like it to be, post-correction?

What will be the outcome of that shift? Some kind of better research?

On what basis do you think that field should agree with your perception?

On what basis do you think your perception is correct?

Note that I’m using “qualitative” and “quantitative” as stand-ins for whatever you think there’s too much of and too little of, respectively—please feel free to clarify if these words don’t effectively capture what you are trying to say.

It’s not at all important to me to attribute to you a “total rejection” of these methods, your writing implies to me that you have a preference for quant methods and think they’re better. You’re not, for example, complaining about too much quant methodology in economics, and too little qualitative. Sure, you don’t think that qual methods are bad per se, but you do think that the social sciences could use fewer of them, and that they’re overused.

I imagine you would agree that these methods can tell us different things, and that they’re not interchangeable for any given research question. You’d probably also agree that some fields bias towards certain types of questions, and that maybe the ratio of methods of work in a given field reflects the bias towards questions that are best answered by those methods. So, are you suggesting that entire fields should focus more on different questions, specifically those that can be answered quantitatively?

If not, I’m not sure I understand what the implications of your argument are.

For what it’s worth, here’s my bias: I am both a quantitative and a qualitative researcher, and I actually think the underlying issues holding back the production of generalizable knowledge have little to do with choice of methodology, and to the extent that they do, it’s in part due to a fetishization of quantitative methods that tell us something generalizable—but not necessarily something useful or even something true.