A Primer in BERTology: What We Know About How Bert Works

&quot; BERT is a method of pre-training language representations, meaning that we train a general-purpose &quot;language understanding&quot; model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP.&quot;<p><a href="https:&#x2F;&#x2F;github.com&#x2F;google-research&#x2F;bert" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;google-research&#x2F;bert</a>

Can anyone please explain (in layman terms if it&#x27;s possible) how did the researchers come up with the method in the first place if the process how the method finds the answers is not understood?

For anyone who, like me, isn&#x27;t a BERTologist, BERT is a neural network architecture.

Any new information in the paper since the first version came out in Mars? Otherwise a 6 month old meta-study seems kind of dated given rate of progress in NLP atm.