[sifoobar]

Not an expert, but as long as there is no way of knowing WHAT it is the algorithm is learning it seems to me that it could never work reliably. It might look perfectly reasonable until you hit one of the triggers the algorithm used to segment the data.

Someone somewhere shared a story about using machine learning to spot the difference between US and Russian tanks ; which apparently worked fine until field testing, where it failed miserably. What the algorithm had learned was the difference between great quality photos of US tanks and poor quality photos of Russian. True or not, this is exactly the kind of issues that will keep popping up.

Plenty of people are spending plenty of time figuring out how to mess with facial recognition as we speak by taking advantage of the same fundamental weakness.

[freeone3000]

Oh, yep! It's super simple to induce systemic errors like that. Take https://github.com/kevin28520/My-TensorFlow-tutorials/tree/m... . Lighten every dog by 20%. Darken every cat by 20%. Train. Take image of cat, lighten 20%, watch as it's transformed into a dog!

For large corpora, it's impossible to know what features got selected. They probably aren't any feature a human would consider.