[osaariki]

You're quite correct that evaluating the back propagation could be done exactly the same as the forward pass. However, if you want to train for more than a handful of steps you'll have to use an operation called bootstrapping to periodically "refresh" the ciphertexts that encode the model. Bootstrapping is essentially evaluating the decryption circuit of HE using HE itself (with an homomorphically encrypted version of the secret key). The problem is that bootstrapping is much more expensive than the other operations in HE.

People have done very effective training on encrypted data using simpler models, like linear or logistic regression. See for example this work [1] from my colleagues at Microsoft Research.

[1]: https://bmcmedgenomics.biomedcentral.com/articles/10.1186/s1...