Meta Learning Backpropagation and Improving It (2020)

Neural networks are merely generalized methods used to translate data. These translations occur by multiplying &#x2F; adding &#x2F; dividing &#x2F; etc weights to the input then passing the output to the next node(s).<p>RNNs keep a memory of prior values such that you can pass on a “memory”.<p>At the end what this is doing is replacing components of the graph with mini-RNNs and pruning based on another network overseeing the first.<p>Having done quite a bit of work in this space I have a couple of thoughts.<p>What was the major advance in games? It was networks playing themselves.<p>Here we may want to do the same thing. Meta-learners need lots of “experience” (data) just like anything living.<p>This paper doesn’t dive too in-depth on the idea that the meta learner can be made general, but they can. There’s only so many problem types (classification, regression, generation, etc) and only so many data formats. Further, networks are fairly well defined, much more so than language.<p>This is the field of AutoML of anyones interested.

Anyone else read this headline as &quot;Facebook parent company does a thing&quot;?

Inb4 any learning algorithm released in the next 20 years is claimed to be just a special case that can be learnt via this meta learning framework.<p>(For the record, I do believe the man has been done dirty a few times. Hate the meme, not the memer.)

1) This is by Schmidhuber so take it with an extra grain of salt 2) The experiments were on toy datasets. There is a reason they did not do this on a real dataset.