[orionr]

For those interested in this area for PyTorch models, take a look at Captum (https://captum.ai/). Still a lot of work to do, but we’ve provided a number of algorithms described in this field guide in the library. Always looking for collaborators and contribution of others.

Disclosure: I support the team that developed Captum.

[arolihas]

That's really cool! My Deep Learning class used Captum for our assignments with GradCAM activations; it's a very convenient tool for interpreting model activations on images.