Natural Gradient Descent (2018)

Not to spoil the article for anyone but..<p>Pretty in depth article and well laid out explanation of natural gradient descent with a small pre-fit dataset for a conclusion of &#x27;too computationally expensive for machine learning&#x2F;big data world&#x27;.<p>This is what I struggled with in school. You&#x27;d spend a class week learning some tough stuff only to be told &#x27;this is no longer done, better methods are now used.&#x27;<p>Sometimes the work is needed to allow you to understand why&#x2F;how the new method is used, but in many cases I didn&#x27;t find that to be true.

<a href="https:&#x2F;&#x2F;towardsdatascience.com&#x2F;its-only-natural-an-excessively-deep-dive-into-natural-gradient-optimization-75d464b89dbb" rel="nofollow">https:&#x2F;&#x2F;towardsdatascience.com&#x2F;its-only-natural-an-excessive...</a> is another nice overview complementary to this one.

This series is well written. The speed up is expected due because you&#x27;re incorporating second-order information during the optimisation. For more insight into second order optimisation methods, take a look at Newton&#x27;s method: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Newton%27s_method" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Newton%27s_method</a>. The intuition, derivation, and proof of correctness and convergence speed are quite illuminating.