[nl]

Oh, I agree it's an entirely valid area of study.

But to do it you need experts in criminology, physiology and machine learning, not just a couple of people who can follow the Keras instructions for how to use a neural net for classification.

For example, I think I remember reading a papers in the physiology field that show a link between increased testosterone and different facial features - but from memory (and I don't have the paper) there was no link between that and criminal offending.

In this case, the features they are finding don't seem to make any sense. A slight smile in the criminals seems more likely to be due to the way that set of photos are taken, and a number of the other features could possibly be explained by the fact the criminal set came from a single police department (in a single geographical area), while the other dataset was collected online. Given the small size of the dataset, if it included a single "family"-gang of criminals it is likely that would have been enough to taint the features.

[seekupdown]

China has 105 cities with over a million people each, and people there migrate frequently these days. A single gang?

[nl]

Having dealt with some what similar datasets myself, there is a really, really good chance that the police department grabbed a days of arrests from one or two cities. There are only 730 positive cases - it's pretty easy to imagine that many of them could be from a single gang - either family or ethnically based.

[BickNowstrom]

The link between increased testosterone and criminal offending has been established by research:

> Testosterone plays a significant role in the arousal of these behavioral manifestations in the brain centers involved in aggression and on the development of the muscular system that enables their realization. There is evidence that testosterone levels are higher in individuals with aggressive behavior, such as prisoners who have committed violent crimes.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3693622/

> Inmates who had committed personal crimes of sex and violence had higher testosterone levels than inmates who had committed property crimes of burglary, theft, and drugs. Inmates with higher testosterone levels also violated more rules in prison, especially rules involving overt confrontation.

http://www.sciencedirect.com/science/article/pii/01918869940...

Though the connection can be said to be weak, and definitely not the only factor (high testosterone alone is not sufficient for criminal offending), it is there.

> In this case, the features they are finding don't seem to make any sense. A slight smile in the criminals seems more likely to be due to the way that set of photos are taken

From the paper: "We stress that the criminal face images in Sc are normal ID photos not police mugshots."

> by the fact the criminal set came from a single police department (in a single geographical area)

Subset Sc contains ID photos of 730 criminals, of which 330 are published as wanted suspects by the ministry of public security of China and by the departments of public security for the provinces of Guangdong, Jiangsu, Liaoning, etc.; the others are provided by a city police department in China under a confidentiality agreement.

> if it included a single "family"-gang of criminals it is likely that would have been enough to taint the features.

Family resemblance is an interesting one. But unlikely to significantly affect the accuracy difference between proper labeling and random labeling (they'd all need to be related)

Overfit is sufficiently ruled out (to me), but leakage is not. Unfortunately it is not possible to replicate this study (even if the dataset was available, the implementation details are scarce). Differently sized raw ID pictures, or compression artifacts, could lead to near undetectable leakage for outsiders. I would probably not give this paper my stamp of approval, even if it was on an uncontroversial subject, but it is not abysmally bad.

I do think one has to be careful to separate moral concerns from technical concerns. Sure, this all feels very wrong to me, and should be taken into account when creating new regulation for ML systems, but the research itself (apart from the small sample size, and vague data gathering methods) is sufficiently solid for debate. Maybe we don't want to admit that phrenology can have a measurable impact on behavior, but that is wishful thinking, not science. Like you said: 'a link between increased testosterone and different facial features' exists, and I just sourced you that a link between criminal behavior and testosterone exists. Logic would deem us to conclude that different facial features are indicative of different criminal behavior, no matter the bizarre, scary, immoral research that supports it.

[nl]

I'd note that they claim 89.5% accuracy(!) using the CNN classifier. One paper they reference[1] use a similar technique to attempt the (seemingly much easier) task of classifying people into Chinese, Korean or Japanese. They get 75% accuracy.

89% accuracy means that there is almost no other feature that influences criminality.

That should set off all kinds of alarms. If there was some kind of relationship between facial features and criminality (and I don't discount that there could be) I'd expect it to be a very weak one, not one that is accurate 9/10 times.

[1] https://arxiv.org/pdf/1610.01854v2.pdf