Previously:<p><a href="https://news.ycombinator.com/item?id=7557964" rel="nofollow">https://news.ycombinator.com/item?id=7557964</a>
<a href="https://news.ycombinator.com/item?id=9814114" rel="nofollow">https://news.ycombinator.com/item?id=9814114</a><p>But not a lot of discussion over there.<p>The visualizations are great, and this basically blew my mind. I didn’t know of the manifold hypothesis until now.<p><pre><code> The manifold hypothesis is that natural data forms lower-dimensional
manifolds in its embedding space. There are both theoretical and
experimental reasons to believe this to be true. If you believe this, then
the task of a classification algorithm is fundamentally to separate a bunch
of tangled manifolds.
</code></pre>
My interpretation/rephrasing: if you want to build a neural network that distinguishes cat and dog pictures, in the worst case that would seem to require a huge network with many nodes/layers (say, the number being a function of the size of the image) rather than the number that seems to work reasonably well in practice (six or some other rather low constant number observed in reality). So the number of dimensions over which the “images” are potentially spread is huge, but it’d seem that in the real world one can rearrange the dog and cat images in a “shape” that then allows for relatively easy disentanglement by the neural network; and these shapes can probably be realized in much lower dimensions (in the example, six).<p>This could explain (for some definition of explain) the observed predictive power of relatively small neural networks.
Gunnar Carlsson will be teaching a related tutorial ("Using topological data analysis to understand, build, and improve neural networks") on April 16th in New York City <a href="https://conferences.oreilly.com/artificial-intelligence/ai-ny/public/schedule/detail/73123" rel="nofollow">https://conferences.oreilly.com/artificial-intelligence/ai-n...</a>
This was the article that helped me get neural networks when I began studying them few years back. Interpreting them as a series of curvillinear coordinate transformations really helped me understand them better.<p>PS: There is a great introductory article on entropy on the blog that is worth checking out.
This is beautiful, and surprisingly approachable. Also, feels relevant to this recent conversation: <a href="https://news.ycombinator.com/item?id=18987211" rel="nofollow">https://news.ycombinator.com/item?id=18987211</a>
If anyone could point me to literature on k-nn neural networks (or the relationship, if any, between k-nn algorithms and basis function decomposition and/or blind source separation) I’d be much obliged.
Haven't given it a lot of thought, but isn't his vector field idea somewhat similar of an approach to neural ordinary differential equations?