A wider attention window. ChatGPT can see 4096 sub-word tokens which is not a lot in the grand scheme of things. You could get "explainable" explanations of topic if it went and read 10 articles and was able to attend to the full text of all the articles to quite literally link the part of the article to the output with both generated text and a citation. That would take an attention window in the 400,000-4,000,000 tokens range.<p>There are very long-range transformers but as of yet they don't work so well. They will be a continuing research topic because a longer attention window allows applying LLMs to more problems. For instance, document classification or retrieval of documents longer than 4000 tokens.
They need bard as fact checker.. joke.<p>The problems I see is beside wrong answers, e.g. some Excel logic, that it loses its state in the middle.<p>As LLM work in a 3d graph space, it seems to often take some wrong shortcut in that graph space causing it to lose some important parts of the information.<p>They need to clean the rubbish connection between the graphs out, whats may be impossible.
Or to train it even more to make certain connections in the graph more "used".<p>Refine..
Well it’s a language model. It’s fundamentally not about learning facts unless the facts are statistics about the corpus text. Hook it up to a database and have it translate questions into queries instead.
Include a confidence score next to the output. If it’s just babbling to fill space, that should get a lower score than if it actually recognizes something from the prompt.