[MrMoenty]

My point is that the parent's comment is mostly right: a frequent challenge in ML is black box computational units which don't expose the necessary information to run autograd. Even if the underlying mathematical function is not differentiable everywhere, but only on most points, having autograd available is valuable for use in training.

Hence you get works like this [1] which reimplement existing systems in a way that is amenable to autograd. [1] https://arxiv.org/abs/1910.00935