[mr_overalls]

How important is optimization? Like a lot of engineers I know, I took a few required courses in undergrad on linear algebra and stats. But I've never studied optimization. And I see it come up all the time on lists of central theory for ML & AI. . .

Is there a classic textbook on the subject? Are there any free online courses that are considered good?

https://www.coursera.org/courses?query=optimization

[snakeboy]

Nocedal & Wright [1] is generally considered the definitive text for reference, at least. Very clear and complete from what I recall.

[1] https://www.springer.com/us/book/9780387303031

[Puer]

Convex Optimization by Lieven Vandenberghe and Stephen P. Boyd was the one my professors always recommended and I think it's excellent.

[Eridrus]

Optimization as a tool is important and widely used, but... almost everything grabbing headlines uses some form of SGD + Momentum. Very little of the actual progress comes from better optimization.

[tnecniv]

Optimization can be pretty useful. Most stats / ML problems are posed as minimizing a function subject to some constraints.

Depending on your problem, you might be able to exploit special structures to solve problems faster than just doing gradient descent. If you know linear algebra and stats, you'll be fine getting through an optimization book.

Boyd's book is canonical at this point, but might be hard to get through. Before you get to actually optimizing anything, you need to make your way through some chapters on convex analysis with little application.