It seems that there should be a way to algebraically simplify neural nets.<p>Trivially a node that has a zero weight can be removed as can any links to / from that node.<p>It should also be possible to eliminate nodes that have a full-value (aka '1' on the 0-1 scale)<p>I have also seen work where the matrix multiplies during training can have columns "collapsed".<p>The ultimate question might be applying an "algebraic simplification" of the final network
to simplify a post-trained network used for inference.<p>The idea is to take a path through a network, constructing the equation for that path, reducing
it to a shorter path by conbining nodes and weights.<p>It is certain that a node participates in several (hundred?) paths. In this case it might be
useful to "copy" the node so it can be part of a path reduction without affecting other paths.<p>I believe that in theory some neural networks can be reduced to a single hidden layer[1]. The game
would be to algebraically reduce network depth.<p>[1] Lee, et al. "On the ability of neural nets to express distributions"
https://arxiv.org/pdf/1702.07028.pdf (2021)