[GranularRecipe]

This is not only an issue for shallow networks. As far I know, both points apply to all feed-forward networks regardless of depth.

[simiones]

Yes, both apply regardless of depth (as long as it is finite, I imagine).