To quote Gian-Carlo Rota (Indiscrete Thoughts):<p>When reading a text he notes<p>"I have learned the content of the text by reading the text. But logical hygiene demands that we keep the terms "text" and "content of the text" separate and equal. The text may be an object. The content of the text is not an object in any ordinary sense. Nevertheless, the content is more "important" than the text."<p>"The distinctness of text and content is undeniable. It may be confirmed by eidetic variations. For example, I may learn the same content by reading another text."<p>"... the conclusion is inescapable: contents do not exist anywhere, yet it is contents and not brains or texts that matter."<p>In ML, the training set is "the text" whereas the weights represent "the content". Having trained a neural net, there doesn't appear to be any obvious way to "explain the content".