[shakna]

All of psychology was hit by the Replication Crisis. 1 in 3 papers have results that are unable to be replicated. In any field that'd have seriously wide-reaching implications, but in psychology where so much relies on foundational theories like ToM it is extremely significant. This impact was worsened by the fact that so few psychological researchers choose to attempt to replicate results, due in part by the fact that doing so isn't incentivized by the industry.

Embodied cognition basically has not a single paper able to be replicated [0], so it's out the window.

This isn't to say that ToM is a flawed theory - in general, it's conclusion is probably true. Rather, psychology has some terrible, terrible, mainstream practices - things like outlier elimination and p-value rounding basically make it impossible to replicate a study unless the original author is involved.

But _assuming_, without evidence to the contrary, that the theory is still sound is denying the problem exists. Thankfully there is some research [6] attempting to show where the world is at with regards to this particular theory.

The study the article is about uses the fairly typical false-belief task. We've been using versions of it since '83, so it should be solid. Except it might not work at all when a person has autism [1], and as autistic people generally are aware of others and that others can differ in thought, the task may be flawed or our understanding flawed. Or maybe autism really does mean you lack something that fundamental [+]. There are other inconsistencies with it. [7]

The false-photograph task was developed in part because of the apparent limitations of the false-belief task. Unfortunately, Woodward's results haven't yet been replicated, just relied upon.

Before the crisis hit home, the research was leaning towards people with autism being ToM deficient, but replications don't show statically sound results that say it conclusively or not, thanks to p-value rounding, whilst some more recent research suggests we simply aren't measuring it correctly, and those results may be caused by the coping mechanisms employed. [2][5] Which, if true, suggests that the false-belief task may not actually be stimulating the right parts of any individual's mind, but rather just engaging them in something visually intensive.

A lot of our measurements of ToM, such as when using fMRIs, have been called into question, as they might just be falsely noticing the spatial orientation that happens during visual stimulus. [3] But again, there isn't enough replication to say definitively one way or t'other.

There's also some lesser issues. Some of the strongest ToM research suggests that the origin lies in mirror neurons [4], but the animals studied in the article of this thread don't have them, and animals possibly having ToM is extremely inflammatory within the broader psychological community. If you want someone to try and replicate your study, suggest that a particular animal has ToM. There's about a 50/50 success rate, which is not really encouraging.

Hopefully that's a little bit to chew on. As I've said, ToM probably does work. But we don't currently have the statistics to say it does.

[0] https://qz.com/1525854/psychologys-replication-crisis-is-deb...

[1] https://www.sciencedirect.com/science/article/pii/0010027785...

[2] https://www.nature.com/articles/nn1611

[3] https://doi.org/10.1177%2F1073858407304654

[4] https://doi.org/10.1016%2FS1364-6613%2898%2901262-5

[5] https://doi.org/10.1016%2Fj.neuroimage.2011.02.067

[6] https://www.sciencedirect.com/science/article/pii/S235234091...

[7] https://journals.sagepub.com/doi/full/10.1177/09567976124478...

[+] I'm not the right person to judge this. I have autism. Which, if the ToM theories are correct, means I lack the perspective to tell whether or not I have a ToM deficit.

[nbeleski]

Hey, lovely answer, I will take my time to go through all of this.

I currently research mental representation such as emotions in applied computing which is near-at-hand with ToM and although many of the theories seem intuitive, the replication crisis is something that really bothered me (since I come from a STEM background).

I will read through your post and definitely check the sources, thank you very much!

As for your last point, ToM+Autism might be one of the more interesting aspects of ToM research and as far as I can tell, we are far from conclusive theories.

[shakna]

That's great to hear.

I'm by no means an expert, I became interested when my ex-wife was being taught p-value rounding in her first year course, right before the Crisis became well-known, which with a background in statistics absolutely horrified me.

I haven't put in a huge effort to stay up-to-date, so hopefully the situation has improved some, but the community has seemed... Divided on whether or not replicating studies is even important.

[jcranberry]

I'm not defending psychology. But you've inserted your own narrative and interpretation into this group of studies which do not necessarily follow. In addition, although they are still important in a lot of ways this is a fast developing field and some of these studies are fairly old.

Your assertion that [2] and [5] are evidence of problems with the conclusion of [1] does not follow. [2] and [5] are findings of atypical neural activity in specific systems/areas of the brain in response to imitating and experiencing emotional stimuli. [1] is a finding that people with ASD score poorly on a test of theory of mind. It does not at all follow that they are evidence that the result of [1] is incorrect. They could just as easily by a reason why people with ASD are lacking in theory of mind rather than be evidence that the result is incorrect.

It's the same thing with [4]. I also don't see evidence to your claim that great apes do not have them. I only see a lack of any kind of studies on MNS in almost all animals. This is the only excerpt I found on them is in [9], and it says nothing of the sort, and it implies that there haven't been any attempts at observation of MNS in other animals including hominids/apes.

There is nothing on whether or not hominids do not have MNS, and on dolphins there is only (at this time) conjecture. On the other hand animals having mirror neurons but not having theory of mind does not mean that mirror neurons are not necessary for forming theory of mind. It just means that it's more complicated than that. [8] asserts that it is a part of it. In fact MNS is now believed to be a separate system from the ToM system (even if they are related and interact) [10].

Although [3] cast some doubt, it is by no means a conclusive dismissal. It essentially says one area of the brain previously studied probably has several different functions instead of just a social-cognitive one. It shows only that the specifically relevant previous work is less conclusive. It is also from 2007, and has nearly 1000 citing papers. One such paper that is fairly highly cited is [10], which compensates for issues presented by [3] by looking at a different area of the brain. Here is another study which examines a different aspect of the brain [12]. It also cites a few studies which it cites as specifically showing reduced activity in regions in the brain which are part of the ToM/mentalizing system.

It is not obvious to me from the evidence you've provided what the problems with ToM are, especially with regards to people with ASD.

[8] https://www.sciencedirect.com/science/article/pii/S014976340...

[9] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5810456/

[10] https://www.ncbi.nlm.nih.gov/pubmed/19524046

[11] https://www.sciencedirect.com/science/article/abs/pii/S10538...

[12] https://jamanetwork.com/journals/jamapsychiatry/article-abst...

[nbeleski]

Hey, thanks por posting this and adding to the discussion. You raise some points that I feel are very important.

[shakna]

> Our central question, however, was whether there is any evidence for a direct supporting role of the mirror system during mentalizing tasks. Apparently not.

and

> This conclusion is contrary to suggestions that the mirror system might aid the mentalizing system to inferring intentions of others.

[10] Doesn't seem to support the idea that the mirror system is at all involved in ToM. It may cite other papers, but the conclusions are not in support of that concept.

---

> They could just as easily by a reason why people with ASD are lacking in theory of mind rather than be evidence that the result is incorrect.

I didn't rule this out. However, from the evidence we have, there isn't enough to rule it in, either. The neurological patterns being seen could simply be the mirror system, which doesn't seem to have to be involved with ToM being active. We don't understand enough.

If the test is incorrect, then there isn't a reason to believe people with ASD are ToM deficit.

Further, the result that people with ASD have a mirror system deficit hasn't always been reproduced. [0]

[0] http://www.antoniahamilton.com/HamiltonMarsh_UoM_preprint.pd...

---

[12] Brings out the statistician in me a bit. They use Bonferroni correction, which while it works, it isn't the most suitable way of dealing with the problem on hand. If instead they'd used Sidak correction, the resulting confidence may have been different. They would have ended up close to the same result, but if they've made this very simple mistake, (choosing a correction method that has known applicable flaws, in fact probably the weakest familywise method), what else have they done?

> the mean z score for ToM overconnected clusters was correlated with ADI-R Social and ADI-R Communication scores (r = 0.45, P < .05 and r = 0.51, P < .01, respectively), although neither survived Bonferroni correction for multiple comparisons.

If it didn't survive simple correction over multiple comparisons, then it probably isn't significant. However, rather than looking at that, this value that can't survive correction lead to their conclusion that their first hypothesis and second hypothesis were correct.

I don't have the time or energy required to use the supplied data to re-evaluate the given data, but currently there does appear to be the suggestion that this paper is a victim of P-value hunting.

Which is kind of the point. These practices are incredibly wide-spread, and incredibly harmful to the whole field.

[jcranberry]

> [10] Doesn't seem to support the idea that the mirror system is at all involved in ToM. It may cite other papers, but the conclusions are not in support of that concept.

Well, like I said it's a quickly growing field...[8] is a review from 2001 early on in the discovery of mirror neurons (although I know at some point or another their existence was disputed...not sure if that's still happening) and contains a lot of conjecture, [10] is a meta-analysis from 2009. I would say [10] is correct, so MNS may even be a red herring in the conversation on ToM.

However, if you're responding to confusion over this statement:

> One such paper that is fairly highly cited is [10], which compensates for issues presented by [3] by looking at a different area of the brain.

I believe the paper I attempted to cite here was in fact [11], which is also one of your citations, [5]. In it's discussion it asserts that the area of the brain relevant to the paper is "functionally and spatially dissociable from nearby dorsal clusters which respond to attentional reorienting", citing [3].

>I didn't rule this out. However, from the evidence we have, there isn't enough to rule it in, either.

I absolutely agree...but you asserted [2] and [5] 'suggested we simply aren't measuring it correctly', which they don't. I just wanted to clarify this wasn't the case. [7] suggests that, but absolutely not [2] or [5].

>Further, the result that people with ASD have a mirror system deficit hasn't always been reproduced. [0]

What you're saying is true, but this review doesn't support the assertion that people with ASD do not have a ToM deficit. It brings up two theoretical explanations for why people with ASD 'have difficulty understanding goals and intentions of others', and shows that evidence towards one of the theories, the 'broken mirror theory' is shaky at best and evidence is leaning against the broken mirror theory at worst.

There are three questions here which are being muddied together:

1. Does the MNS have a role in development of ToM?

2. Do people with ASD have a deficient MSN?

3. Do people with autism have a ToM deficit?

The answer to 1 appears to be, we do not know, however as the mentalizing/ToM system can act independently to the MNS, this may not be material to question 3.

The answer to 2 appears to be, it's as of yet inconclusive, and the review you posted seems to assert evidence is mounting to the contrary.

As for question 3...all current measures seems to point towards 'yes' [13], however, all current measures are also disputed as to whether or not they are accurately measuring ToM [14]. The method discussed in [7], is actually not the only method used to study ToM in people with ASD. It is an explicit (verbal) method (such as those referenced in [0] of the parent comment), and 'implicit' (visual) methods followed which used eye tracking (it is not yet clear whether these tests measure different things or the verbal tests are simply ineffective). These methods found impaired mentalizing in people with ASD [15]. More recently, more methods have arisen [14]. Here is one of the linked studies with an interesting discussion [16].

So, frankly it wouldn't be correct in my eyes to say it's certain in any way that people with ASD have a ToM deficit...but at the very least progress and improvements in methodology don't seem to have yet cast significant doubt on conclusions reached by previous research in this topic.

Unfortunately I'm unable to give input on your comments on p-value hacking for [12] since I don't have expertise. But at least the papers subject is on the mechanism rather than the degree to which people with ASD have a ToM deficit, so it doesn't affect the overall discussion too much.

[13] https://www.ncbi.nlm.nih.gov/pubmed/23686020?dopt=Abstract

[14] https://link.springer.com/article/10.1007/s10803-018-3823-3

[15] https://www.frontiersin.org/articles/10.3389/fpsyg.2016.0169...

[16] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5487761/

[shakna]

> As for question 3...all current measures seems to point towards 'yes' [13], however, all current measures are also disputed as to whether or not they are accurately measuring ToM [14].

I think this where we're getting stuck.

For me, if a methodology is in doubt, then so are any conclusions that rely upon it. If the methodologies are in doubt, then the conclusions shouldn't be used as a foundation for anything.

Whereas for you, it seems that you'll continue along with the previous belief until such time as new methodologies quantitatively say one way or the other.

[jcranberry]

I wouldn't say I hold such a strong position. I tried to make my conclusion fairly noncommittal ("it wouldn't be correct in my eyes to say it's certain in any way that people with ASD have a ToM deficit"). My knowledge has evolved along with this conversation (hence why we may have seemed 'stuck'), so perhaps my conclusive tone of writing was inappropriate, which may have created a false impression of certitude.

Back when I was in school for mathematics and taking a few grad classes (my math knowledge drained out unfortunately quickly) I was told by a colleague that the Riemann Zeta hypothesis was generally regarded as 'true', and that it's not uncommon for mathematicians to do research proving theorems on the condition that the Riemann Zeta hypothesis is true. Point being, don't think there's anything wrong building on something which we may not yet know to be true...as long as caveats are stated up front.