Hi!
Do you know any books on math optimization that are essential for anybody getting into this field? Is there any classical literature for optimization related to ML? Thanks!<p>My current list includes:<p>1. Numerical Optimization by Jorge Nocedal Stephen J. Wright<p>2. Algorithms for Optimization
- introduction to optimization with a focus on practical algorithms<p>3. Algorithms for Decision Making
- a broad introduction to algorithms for decision making under uncertainty<p>[2] https://algorithmsbook.com/optimization/<p>[3] https://algorithmsbook.com/
Here's an excerpt from a comment I previously made on Hacker News:<p>I'm a Ph.D. student in operations research (OR). My suggestion would be to first build a strong foundation in linear programming. This will introduce you to the notion of duality, which is heavily emphasized in many mathematical programming courses. Here's a good open-source book on linear programming written by Jon Lee, the current editor of Mathematical Programming A: <a href="https://github.com/jon77lee/JLee_LinearOptimizationBook" rel="nofollow">https://github.com/jon77lee/JLee_LinearOptimizationBook</a><p>Then I'd suggest studying more general methods for continuous and convex optimization. The book I see mentioned a lot is Convex Optimization by Boyd and Vandenberghe, although we didn't use this in our coursework. Instead, we used a lot of the material presented here: <a href="http://mitmgmtfaculty.mit.edu/rfreund/educationalactivities/" rel="nofollow">http://mitmgmtfaculty.mit.edu/rfreund/educationalactivities/</a><p>If you read the above (or any other two books on linear programming and convex optimization), you'll probably have a better idea of what you want to study next and how you want to go about it. The next natural step would be to study combinatorial (i.e., integer or mixed-integer) optimization. (Jon Lee has another book on this subject; I've also heard good things about the Schrijver book.)
For theoretical continuous/nonlinear/convex optimization, your #1 is the bible, together with<p>"Convex Optimization" by Boyd & Vandenberghe.<p>However, beware that both are <i>grad</i> textbooks. They can be tough going at times. Unfortunately, I never found undergrad textbooks I liked much, for theory.<p>If you're interested in discrete optimization too (the other half of math optimization), the classics are:<p>"Optimization Over Integers" by Bertsimas & Weismantel<p>"Integer and Combinatorial Optimization" by Nemhauser & Wolsey
Two fascicles for Knuth's <i>Art of Computer Programming, volume 4B</i> on optimization are out. <a href="https://www-cs-faculty.stanford.edu/~knuth/news.html" rel="nofollow">https://www-cs-faculty.stanford.edu/~knuth/news.html</a>
I started from zero, and my approach was to read Nocedal/Wright cover-to-cover, and then the same with "Numerical Linear Algebra" by Trefethen/Bau. Usually it goes the other way, but I found the linear algebra primer in N/W to be good enough to get started.<p>I also read "Practical Optimization" by Murray/Gill, which is interesting because it has a lot of conversational coverage of e.g. corner cases, stuff that most textbooks won't cover.<p>That will cover the expected baseline of almost everything you'll encounter in the convex smooth continuous domain. I don't have great answers for moving past that.
<a href="https://www.amazon.com/Model-Building-Mathematical-Programming-Williams/dp/1118443330" rel="nofollow">https://www.amazon.com/Model-Building-Mathematical-Programmi...</a><p>This is the canonical introduction book recommended by Gurobi. I've found it to be great for those getting into the subject. It has math of course, but the focus is on going over the basics of LP, integer, nonlinear, Mixed-Integer...etc, followed by lots of examples. I think it's the best book to start with to get a feel for the subject of OR, before diving into the harder books.
I recommend Bayesian Methods for hackers [1]. It doesn’t go too deep on theory but I feel like it’s well written and has really good coded up examples of theory being applied to problems. It has a relatively narrow scope, but I find myself reaching for methods I learned from the author frequently.<p>[1] <a href="https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers" rel="nofollow">https://github.com/CamDavidsonPilon/Probabilistic-Programmin...</a>
Nocedal and Wright is an excellent overview and a good starting point. Also consider:<p>- R. Schneider, Convex bodies: the Brunn-Minkowski theory. The first two chapters are an excellent introduction to convex geometry (plus a little bit extra!) if you have some undergrad-level analysis.<p>- Hiriart-Urruty and Lemarechal, Fundamentals of convex analysis. This book has been highly recommended to me but I've never used it. Might be an easier go than Schneider for convex geometry.<p>- Golub and van Loan, Matrix computations. Excellent book on numerical linear algebra.<p>- Bonnans, Gilbert, Lemarechal, Sagastizabal, Numerical optimization: theoretical and practical aspects. This book has a detailed description of bundle methods, which are important and in my opinion underutilised.<p>- I. Maros, Computational techniques of the simplex method. This is the only book I'm aware of that discusses how to build a working implementation of the simplex method for linear optimisation.<p>I'm not aware of any books that cover line search algorithms in detail. These are important in implementations but, beyond discussing the Goldstein and Wolfe conditions, generally glossed over in prose. Even in the absence of stalling and numerical difficulties, you can see an order of magnitude speedup from replacing a bad line search with a good one. One line search algorithm I've had success with is described in More and Thuente, Line search algorithms with guaranteed sufficient decrease.<p>Lots of tacit engineering knowledge goes into building a fast and robust optimisation code. Some of that knowledge gets forgotten when code is rewritten or ported from one language to another.<p>Mercifully, a lot of that engineering knowledge has been encoded into freely-available optimisation code. Quite a bit of that code is pretty readable. Off the top of my head, I've learnt things from:<p>- Liu and Nocedal's Fortran L-BFGS implementation,<p>- The CUTEst problem collection,<p>- Chih-Jen Lin's LIBLINEAR and LIBSVM,<p>- Lin and More's TRON,<p>- Csaba Meszaros's BPMPD,<p>- Jacek Gondzio's HOPDM,<p>- The GNU Linear Programming Kit,<p>- and probably quite a few other sources!
Without a doubt you should read the following.<p>It is the clearest, most in depth, introduction from "zero to hero" for optimization. Mostly from a math perspective, useful for many things outside of ML too!<p><a href="https://www.amazon.com/Foundations-Applied-Mathematics-Approximation-Optimization/dp/1611976057" rel="nofollow">https://www.amazon.com/Foundations-Applied-Mathematics-Appro...</a>
Nocedal and Wright is good. +1 also to the suggestions for Boyd and Vandenberghe. I really like Boyd's writing in general; he has coauthored some good review articles on proximal algorithms and ADMM.<p>A couple of other suggestions:<p>Nesterov's <i>Introductory Lectures on Convex Optimization</i>. This one is pretty tough sledding, but I found the perspectives in the first chapter particularly to be enlightening. It seems like there's a newer Springer book which is probably an expansion on this.<p>Bertsekas's <i>Nonlinear Programming</i>. Bertsekas has written a lot of books, and there's a fair amount of overlapping going on. This one seemed to be the one that has the most nuts and bolts about the basics of optimization.<p>EDIT: If you want more understanding of convexity beyond what's presented in these books, Rockafellar's <i>Convex Analysis</i> is helpful.
I recently bought "Introduction to Nonlinear Optimization" by Amir Beck. It's published by the SIAM. It covers the field from beginning to intermediate level and includes examples written in MATLAB. I found the book very readable and illuminating. It's also not overly long nor verbose, which is a plus in my mind. I would recommend it -- I used some of its ideas as fodder to teach the optimization section of my class in Numerical Analysis.<p><a href="https://epubs.siam.org/doi/book/10.1137/1.9781611973655?mobileUi=0" rel="nofollow">https://epubs.siam.org/doi/book/10.1137/1.9781611973655?mobi...</a>
Sébastien Bubeck's book is excellent mathematical introduction to modern convex optimization: <a href="https://arxiv.org/abs/1405.4980" rel="nofollow">https://arxiv.org/abs/1405.4980</a>
So for medium to non-technical people reading this, I took a course in grad school that showed me how to do this in Excel with solver.<p>It was easily one of the top 3 courses I took and heavily based off of this text book:<p><a href="https://www.amazon.com/Spreadsheet-Modeling-Decision-Analysis-Introduction/dp/130594741X/ref=mp_s_a_1_3?dchild=1&keywords=ragsdale+spreadsheet+modeling+and+decision+analysis&qid=1625426197&sprefix=ragsdale&sr=8-3" rel="nofollow">https://www.amazon.com/Spreadsheet-Modeling-Decision-Analysi...</a>
Sounds like you're looking more for optimization theory, but if you want a gentle introduction to applications with approachable math and lots of examples, I highly recommend "Operations Research: Applications and Algorithms (4E)" by Wayne Winston. It's a solid undergrad level text covering basic linear optimization, mixed integer linear programs, and non-linear optimization.
Mathematical optimization is huge field which splits into different branches. In my opinion, there are two demographics that approach optimization in slightly different ways and are interested in slightly different aspects of optimization: the OR (operations research) camp and the engineering camp.<p>(all generalizations are wrong to some extent and the delineations are not strict, but I have noticed they have different cultures -- analogous to how Breiman observed that there was two cultures to statistical modeling [1])<p>In the OR camp, the focus is primarily on linear programming/mixed integer linear programming. The types of problems solved include transportation, assignment, scheduling type problems. OR folks tend to go very deep into the theory of linear programs (matroids, Benders decompositions, etc.) and the literature is absolutely rich with advances in linear optimization. OR folks tend to be linear optimization specialists and mathematicians. A good practice-oriented intro book is Winston's "OR: Applications and Algorithms". Chvatal's Linear Programming is also good. But this is not my space, so I'll leave OR book recommendations to others.<p>In the Engineering camp, while folks do use linear programs, they also tend to go more in the direction of convex programs and general nonlinear/nonconvex programs (including mixed integer nonconvex nonlinear programs). There are some strong theoretical results for convex programs, but the results (naturally) aren't as strong for nonconvex problems -- global optimality never guaranteed. In the nonconvex camp, practitioners tend to concern themselves with heuristics/techniques like finding good initializations, understanding SSOCs, etc. The types of problems solved range from anything from ML problems to dynamic plant/machine optimization. Most folks in this camp tend to be engineers rather than optimization specialists (though some do become specialists eventually). Nocedal and Wright is a classic for general nonlinear programming, but also look into Biegler's Nonlinear Porgramming. Boyd and Vanderberghe is a classic for convex optimization. Murray and Gill's Practical Optimization is a bit outdated (so don't rely on it for state-of-the-art knowledge), but it has tidbits of insights about optimization that aren't found in other books and that continue to be useful.<p>[1] <a href="http://www2.math.uu.se/~thulin/mm/breiman.pdf" rel="nofollow">http://www2.math.uu.se/~thulin/mm/breiman.pdf</a>
I'd also add C. T. Kelley's "Iterative Methods for Optimization" for more non convex theory. Nemirovski also has a variety of books and course notes that are available, but I haven't spent as much time with them.<p>I agree with thxg, there are few undergraduate textbooks that I've liked.
I'm planning on taking a course in optimization this fall, the textbook for it is "An introduction to optimization" by Chong and Zak. Dunno if it's any good, but I just ordered it so we'll see.
"The Design of Approximation Algorithms" by Williamson and Shmoys<p><a href="http://www.designofapproxalgs.com/" rel="nofollow">http://www.designofapproxalgs.com/</a>
Mathematical Optimization still has many subfields that can be of interest. I guess that non-linear mathematical optimization is most more typical for many machine learning applications. Many pratical applications in scheduling, logistics, planning etc used linear (integer) programming and combinatorial optimization. The following are some points towards that body of literature.<p>Alexander Schrijver [1] has lecture notes on Combinatorial Optimization on his website [2]. He also has an affordable 1800 page three volume set of books "Combinatorial Optimization - Polyhedra and Efficiency" [3], although I would say it is better suited as reference material because it is quite densely written.<p>There is also the classic book "Combinatorial Optimization - Algorithms and Complexity" [4] by Papadimitriou (Bill Gates' MSc thesis supervisor) and Steiglitz that is a nice introduction to the topic as well.<p>"In Pursuit of the Traveling Salesman - Mathematics at the Limits of Computation" by William J. Cook [5] is a more popular science book on the history of the Traveling Salesman Problem, that also explains how linear programming is used in the state of the art solvers, but is of course focuses on a very specific problem. There is also a book that contains all the scientific and mathematical details by Applegate, Bixby, Chvátal and Cook [6] if that is preferred.<p>In recent years, there is a trend that mathematical optimization researchers work more with ML. In particular Dimitris Bertsimas has done some work on the intersection of those area's in recent years [7] and apparently has a book on the topic as well [8] (but I am not familiar with it).<p>[1] <a href="https://homepages.cwi.nl/~lex/" rel="nofollow">https://homepages.cwi.nl/~lex/</a>
[2] <a href="https://homepages.cwi.nl/~lex/files/dict.pdf" rel="nofollow">https://homepages.cwi.nl/~lex/files/dict.pdf</a>
[3] <a href="https://www.springer.com/us/book/9783540443896" rel="nofollow">https://www.springer.com/us/book/9783540443896</a>
[4] <a href="https://www.amazon.com/Combinatorial-Optimization-Algorithms-Complexity-Computer/dp/0486402584" rel="nofollow">https://www.amazon.com/Combinatorial-Optimization-Algorithms...</a>
[5] <a href="https://press.princeton.edu/books/paperback/9780691163529/in-pursuit-of-the-traveling-salesman" rel="nofollow">https://press.princeton.edu/books/paperback/9780691163529/in...</a>
[6] <a href="http://www.math.uwaterloo.ca/tsp/book/index.html" rel="nofollow">http://www.math.uwaterloo.ca/tsp/book/index.html</a>
[7] <a href="https://dbertsim.mit.edu/papers/" rel="nofollow">https://dbertsim.mit.edu/papers/</a>
[8] <a href="https://www.dynamic-ideas.com/books/machine-learning-under-a-modern-optimization-lens" rel="nofollow">https://www.dynamic-ideas.com/books/machine-learning-under-a...</a>