Differentiable does not mean easy to optimize.
One could imagine implementing sha-256 using differentiable operators, and yet the system as a whole would not be optimizable at all.
It would be interesting to have compilers that optimize the "optimizability" of differentiable programs tho...
Also, here are two interesting examples of differentiation through physical systems for classification: