Learning to learn by gradient descent by gradient descent

story time: I tried something like this, putting a nnet output as the transfer weight of another neural net and training the first one feeding in the second net input as input and training it on the second net error, but couldn&#x27;t train the first network because I didn&#x27;t know how to derive the transfer function for the back propagation algorithm.<p>so I opted for training the first net using a randomized genetic algorithm and function descent on it, which as an afterthought is dangerously close on how biology kind of work, but it was exceptionally slow.<p>so I split up the training batches, went to the uni computer room and left the job running on every computer by night to collect result by morning. in the morning I&#x27;d collect the best genes from each machine, mix them all for another few round of training, select the best in the population and reseed them on all the machines by night.<p>after a week of painstakingly organizing, seeding and collecting results, the network never managed to converge around the problem, but boy it was fun trying! The problem was driving a car around a lap of a track using five &quot;distance from kerb&quot; sensor as input angled at 30deg from each other starting from center.<p>I remember I was inspired by an image recognition company, which was using a training network for training network for motion detection over security cameras, so this approach wasn&#x27;t exactly novelty even back then (2001ish).<p>anyway, this got me noticed by a lab assistant and got a thesis on how to optimize neural network to run in 4.4bit fixed math for use in extra low power devices. that one worked! too bad nothing ever came out of it.<p>edit: some fixin

Just read (skimmed) this paper yesterday actually.<p>Looks interesting-- but there are no timing graphs! It&#x27;s kind of a strawman argument to say &quot;We can&#x27;t use Newton&#x27;s method because it&#x27;s too slow to calculate the Hessian,&quot; and then go and present all your performance graphs in terms of number of iterations.

Neither the paper nor the comments have mentioned this: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Truncated_Newton_method" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Truncated_Newton_method</a><p>The Truncated Newton method uses an inner solver that only runs for a few iterations to approximate the Hessian. The approximate Hessian is used to approximately solve Newtons equation. I&#x27;ve implemented it and it works very well. When it gets close to the solution the convergence is very fast.<p>I mention it because it sounds similar to what the paper discusses but you use conjugate gradients in the inner solver and Newton&#x27;s equation in the outer solver.

The title could have been chosen as &quot;Learn to learn by gradient descent by gradient descent&quot; or &quot;Learning learning by gradient descent by gradient descent&quot;

Can all algorithms then be cast as learning problems and their optimal versions produced this way? Seems like amazing work, but I don&#x27;t know enough to confirm.

Very interesting, I am eager to read about the future research on non &quot;simple convex problems&quot;. This where it could provide benefits as in the industry we have a lot of them, a lot of domain knowledge to <i>go around</i> the local minima etc. and a robust ML based approach could really help there instead of being obliged to accumulate in our algorithms years of trial and errors.

Yo dawg, I heard you like gradient descent, so I put optimizers on your optimizer so you can learn while you learn.

No timing results, no comparisons with Nesterov type methods. To all the commenters that have said &quot;this looks promising&quot;: this doesn&#x27;t look promising at all. Why do you think in all the years of people optimizing things with gradient descent no one has tried this? Answer: they have, and it doesn&#x27;t work.

<i>In spite of this, optimization algorithms are still designed by hand.</i><p>Well, they are tuned automatically. There are derivative-free optimization algorithms that have been designed to tune optimization algorithms on a set of instances.

I was just thinking about this the other day! If machine learning can be applied to almost any problem, then surely it could be applied recursively to optimize itself. I&#x27;m glad to see that someone worked on this.

The first time I read about gradient descent and optimization algorithms, this was the first thing that came to my mind, this looks promising.

I actually do hope that all those learning techniques do have limitations and that intelligence cannot be achieved in principle by machine learning. OK, I see the problems with that statement (first of all, define intelligence) but for instance, I hope the so-called &quot;singularity&quot; cannot be reached in principle and if somebody could prove it once and for all, please do so.

When I just finished Andrew Ng ML course and had to solve a real life problem, this was actually the first thing that came into my mind. Too bad I couldn&#x27;t formulate the problem then(still can&#x27;t) and that was basically the end of my ML career.

Next article: &quot;Learning to learn to learn by gradient descent by gradient descent by gradient descent&quot;.<p>Then &quot;Learning to learn to learn to learn by gradient descent by gradient descent by gradient descent by gradient descent&quot; and keep going. Turtles all the way down!<p>P.s: I understand the beauty of this article, but I was surprised none get this irony :-)

Yo dawg, I heard you liked gradients, so I put some learning in your learning so you can descend while you descend.<p><a href="http:&#x2F;&#x2F;knowyourmeme.com&#x2F;memes&#x2F;xzibit-yo-dawg" rel="nofollow">http:&#x2F;&#x2F;knowyourmeme.com&#x2F;memes&#x2F;xzibit-yo-dawg</a>