[jmickey]

Regarding this - "Model here means business rules, like rules for approving loans or adjusting power consumption in data centers. In traditional software, programmers created these rules by hand. Today machine intelligence can use data and new algorithms to generate a model too complex for any human programmer to write."

Isn't it a bit problematic that the business rules generated by the model are too complex for humans to reason about them? How can you rely on the rules to be 100% appropriate for the task if it's impossible to reason about them?

[iforiq]

By "generate a model too complex for any human programmer to write" I believe the author is trying to say, to manually create the rules, one by one. Machine generated complex models, even though very complex, can definitely be understood and heavily audited.

One example is when you fit sparse high dimensional models to complex data in a real-time production system. The resulting models may have hundreds of millions to billions of features with non-zero weights, that constantly change as the underlying data changes. It's impossible to "hand-code" such a model from scratch by any reasonable size team in real-time. On the other hand, these hundreds of millions of rules can (and should) be exhaustively analyzed / audited by slicing and dicing both the model feature-weights, as well their performance on the data comprehensively. As an example, the "R" programming language typically creates useful human interpretable summaries for the models it generates.

For reference, I have been involved at Google in building such massive high dimensional models for properties like Youtube, and currently a founder of one the companies in the HBR report (LiftIgniter, YC W2014). Hopefully that doesn't make me too biased to respond.

[jonathankoren]

If you're looking at model feature weights, you're doing it wrong.

Most models aren't interpretable, and coefficients are highly unit and feature dependent. Discussions involving feature weights beyond "What if we reduce the feature space?" or "Did we implement this feature correctly?" often go bad, they almost always go bad when you're using it to "audit" the model. I have been in way too many discussions where someone suggested that the weights were wrong, simply because they thought something should "be more important".

[cafebeen]

I think this is a major challenge too, although it depends. Some models are easy to reason about, e.g. Bayesian graphical models, while black-box approaches like deep neural networks are not.

One especially problematic issue is: if a model is too complex for humans to reason about, then a business could encode any kind of illicit behavior in the form of model parameters they like. Even if someone could prove that the model is biased one way or another, there is complete plausible deniability for the business, i.e. "I didn't make that choice, the learning algorithm did". We're in for some very interesting legal battles related to this, I think.

[hx87]

> if a model is too complex for humans to reason about, then a business could encode any kind of illicit behavior in the form of model parameters they like

If a model is too complex for humans to reason about, how would a business encode illicit behavior, unless the AI itself was running a significant portion of the business? Even in that case, someone or some group is responsible for setting the initial parameters of the model, and they can be held responsible for its decisions.

In the end, I think the legal solution would be put less emphasis on mens rea and more on actus reus. In other words, if your AI does something wrong, you are in the wrong, regardless of your intentions.

[joe_the_user]

In the end, I think the legal solution would be put less emphasis on mens rea and more on actus reus. In other words, if your AI does something wrong, you are in the wrong, regardless of your intentions.

But that would be a large change in the way the legal system works. Fewer people than you'd expect understand that today, a large portion of crimes, if not most crimes, require both intentions and action for the legal system to judge guilt.

"A fundamental principle of Criminal Law is that a crime consists of both a mental and a physical element. Mens rea, a person's awareness of the fact that his or her conduct is criminal, is the mental element, and actus reus, the act itself, is the physical element."

http://legal-dictionary.thefreedictionary.com/mens+rea

[cafebeen]

I think there are many companies currently or on their way to using AI to run a significant portion of their business, e.g. financial and advertising sectors. The hard part is demonstrating intention in that scenario, as many learning algorithms are stochastic. Even if there is some group to be held responsible and they give you the training data, the model will slightly vary when run repeatedly. It seems likely that people could use that fact for things like money laundering and insider trading, for example, using biases in the model that are hard to detect.

[harperlee]

Completely. In fact whenever regulators are involved (for example on anti money laundering), any black box model is a no go, precisely due to that inexplicability.

[visarga]

Maybe a "parallel reconstruction" method would allow to identify the top reasons why a neural net made a decision and repackage those as the explanation.

Or run an ablative test where part of the features are removed and their effect measured, so as to identify which features drive the decision most.