[orange_puff]

What we know from this well written and helpful article; The false positive rate Apple told us their algorithm had seems to be accurate. If a machine learning model is extracted from the OS it exists on, it will be much easier to generate adversarial attacks.

For example, a neural net's cost function is just a multivariate function with weights as its input. To figure out how to move those weights (positively or negatively), the gradient of the function is calculated and the weights are nudged in the opposite direction (gradient is the direction of the largest growth of a function, we are trying to minimize the cost). Now, assume we are given a cost function and the weights are constant, now, the input can be the image. So, we take the gradient of the cost function with respect to the image pixels and can now see how we should nudge those to maximize the cost. Apple will absolutely need to protect against adversarial attacks for this to be viable. I'm hopeful.