[nshm]

LIME is a software to analyze any machine learning classifier https://github.com/marcotcr/lime, they have very good paper in KDD too http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf