Local Minima, Saddle Points, and Plateaus

This made me think about code. I&#x27;ve long thought of incremental refactoring as &quot;following the gradient&quot; of good design principles, but this essay opened up new connections for me.<p>For example, I&#x27;ve heard elsewhere that great programmers possess the skill of being able to decompose large changes into smaller commits, making the code no worse as they go. Having the ability to see the small changes that get you where you want to go corresponds to finding the direction of change that gets you past the saddle point.<p>Random restart is a standard tool in the kit of stochastic gradient descent. But if local minima are rare, it may be overused. You don&#x27;t have to rewrite your code if you can see the refactors and gradual improvements that get you where you need to go.

I enjoyed this article. It got me thinking that one of the problems in life is that the loss function is unknown: we don&#x27;t always know beforehand what would make us happy or satisfied. However, it&#x27;s a lot easier to gauge one&#x27;s current gradient: whether things are getting better or worse, or stagnant, as in plateau.

I just wish they would say &quot;minimum&quot; when they mean minimum.<p>In real high-dimensional spaces, like life, exploring away from local minima and plateaus (&quot;plateaux&quot;?) is expensive. Figuring in the expense of exploration distorts the reward function intriguingly.

This translates well into advice for startups: Don&#x27;t obsess over finding the perfect solution, but focus on getting <i>something</i> then making constant improvements to what you have. Don&#x27;t try to jump to a global optimum straight away. There are always a huge number of options to choose from, and you can always improve.

I love this analogy. There&#x27;s a great lesson here in assuming the saddle point case and trying to eke out the additional gains you know could be close by.<p>The tricky part in real life is figuring out the dimensionality and then reasoning about how changes in the gradient of one dimension appear to affect changes in the gradient of another.

While I believe to understand the general idea behind the theory as applied to real life, I&#x27;m struggling with a few concepts.<p>What are local minima, in relation to the real life? Are those the &quot;places&quot;, mentally speaking, where one feels like they&#x27;re failing?<p>What, exactly, are saddle points in this context?