| It might be instructive for you to try solving the linear programs in netlib[0] with stochastic gradient descent. ;) The short version of the story is that different optimisation problems have different kinds of structure, and the user will want solutions of different quality and have different amounts of patience depending on the context. Practical optimisation is largely about finding ways to exploit the structure of a problem to get solutions of some quality in some amount of time. No known optimisation algorithm dominates all others. In many machine learning contexts, you want a low-precision solution to a problem with an objective that's expensive to compute but can be reasonably approximated by random sampling. Often, when you use a linear program solver, you want a solution to a linearly-constrained problem where the geometry of the constraints may not be "easy" and any violation renders a solution completely useless. Good LP solvers can solve "typical" well-formulated LPs with hundreds of thousands of variables and constraints in a matter of seconds on modern hardware[1]. These algorithms can't solve many of the problems you're going to feed into your deep learning framework at all. Conversely, your deep learning framework will have a hell of a time solving LPs. As for why GAMS and AMPL are a thing, they're interfaces to solvers, not solvers themselves. You use them because they can handle pulling the data out of the database, translating your GAMS/AMPL program into a formulation suited to the solver you choose, running the solver, and handing back the results. It's straightforward but tedious to do this all on a case-by-case basis; GAMS and AMPL automate much of the uninteresting work. [0] http://www.netlib.org/lp/
[1] http://plato.asu.edu/ftp/lpcom.html |