This article points to something fantastic about language models, which the general skepticism over GPT-3 (at least here) seems to have obscured.<p>Sure, language models won't get us any closer to AGI, but the fact that they are so good at memorizing and re-spewing existing text means that they can offer a snapshot of an average use of any language.<p>The possibilities this opens might not be as great as AGI, but they're pretty amazing nonetheless:<p>- Think about ancient history and fragments of manuscripts. You could make guesses on missing parts of old, lacunary texts. Maybe the missing bits in an ancient Greek papyrus are actually included in a citation in a Byzantine manuscript. Nobody made the connection but that could be enough for a language model trained on the appropriate data to fill in the blanks.<p>- You can also compare snapshots of languages at different points in time. Train a language model on English texts from 1850 and one on texts from 1950. That could tell you how an average writer or speaker from each era would react in different situations.<p>Language models may not lead us any closer to general intelligence, but they can offer us a lot of intelligence on our own history, languages and cultures.
Unfortunately it seems that NLP as a field is more interested in classifying the sentiment of tweets than trying to explore these questions.