[civilized]

So, they fit a fancy model, played with it, and claim the result says something about the real world?

I have never seen a valid social science result derived by this method. This is just fancy model lovers dabbling.

If the patterns they claim are real, they can be exposed with much simpler and direct methods. Let's talk after that's been done.

[civilized]

I want to temper the grumpiness of this comment a bit based on the productive discussion we had in the threads.

The authors developed an impressive model and got interesting results, and I appreciate the stated intent to shed light on enforcement bias patterns. However, I disagree with their claim that they have successfully done so in this paper. They have only provided an interesting prediction which needs to be verified with careful data analysis.

If increased crime in poor areas leads to decreased enforcement, we should be able to see this with well-chosen analysis, reporting and visualizations. We should be able to see examples of the claimed phenomenon happening in full context and have the opportunity to consider alternative causal explanations. If the claim still stands after this has been done, it deserves to be taken seriously. At present, the complexity of the model and the fact that Granger causality != Real causality prevent us from drawing any conclusions.

(The above does not refer to the claim about rich communities draining resources from poor communities, which seems to me like pure speculation. I can draw no logical connection between this claim and the results in the paper.)

[mulmen]

What are those simpler and direct methods? Have they been done?

[civilized]

If the response to increased crime in poor areas is a 35% decrease in enforcement, as the paper claims, this will be obvious in simple plots of crime and enforcement over time in those areas.

No social scientist worth their salt would just report some model-generated plots and claim it represented something about the real world. That's just hypothesis generation. Confirming the hypothesis requires direct data analysis.

[mulmen]

> If the response to increased crime in poor areas is a 35% decrease in enforcement, as the paper claims, this will be obvious in simple plots of crime and enforcement over time in those areas.

Will it? Are you looking at reported crimes or investigated crimes? If the cops never show up do people stop calling? I’m not sure how a simple plot tells the whole story here.

[bergenty]

Is fancy supposed to be derogatory in this statement?

[civilized]

Too fancy for anyone to be confident in what it's doing or whether it reflects real patterns.

0.90 AUC doesn't mean everything you can get this model to tell you is real, especially if you are trying to tell a causal story.

Now, show me a plot of your data with a trend line and I'll often be able to tell you if there's a real pattern in your data.

[w-j-w]

Are you trying to claim that the model would better understand the complicated, real world of law enforcement if it were MORE reductionist?

[civilized]

No, I'm not talking about how best to set up the model. I'm saying that complicated neural network models have no track record of yielding reliable insights about social phenomena. They are untested. Any insights they supposedly provide must be verified by a human analyst checking the data to see if it really looks like the model says.

This isn't a new or surprising principle, and it also applies to much simpler models. Any scientist knows that, if you fit a line to data, you better plot the data with the line before you make any big claims about what that line tells you about the world, because the data underlying that fit could look lots of completely different ways with different implications [1]. These authors did the equivalent of fitting the line and telling us all about its formula and the big implications of the formula without ever plotting it with the data.

[1] https://en.m.wikipedia.org/wiki/Anscombe's_quartet

[8note]

What's the point of training a model if you don't think it will describe the real world in some way?

[civilized]

An AI says there is a dog in a particular picture. The truth of this finding has major implications - say a murder case hangs in the balance. Should someone go look and see if there is actually a dog in the picture?

A cutting-edge model can provide interesting leads but they need to be confirmed by a known reliable method if a serious question hangs in the balance.