[esafak]

You have a gradient so use it instead of faffing about. As another user said, the optima are all the same since the models are wildly over-parameterized.

[mturmon]

This is tersely stated, but it's wise. In general, following the gradient (if you have it) is a very, very good idea.