Related:<p><i>The matrix calculus you need for deep learning (2018)</i> - <a href="https://news.ycombinator.com/item?id=26676729">https://news.ycombinator.com/item?id=26676729</a> - April 2021 (40 comments)<p><i>Matrix calculus for deep learning part 2</i> - <a href="https://news.ycombinator.com/item?id=23358761">https://news.ycombinator.com/item?id=23358761</a> - May 2020 (6 comments)<p><i>Matrix Calculus for Deep Learning</i> - <a href="https://news.ycombinator.com/item?id=21661545">https://news.ycombinator.com/item?id=21661545</a> - Nov 2019 (47 comments)<p><i>The Matrix Calculus You Need for Deep Learning</i> - <a href="https://news.ycombinator.com/item?id=17422770">https://news.ycombinator.com/item?id=17422770</a> - June 2018 (77 comments)<p><i>Matrix Calculus for Deep Learning</i> - <a href="https://news.ycombinator.com/item?id=16267178">https://news.ycombinator.com/item?id=16267178</a> - Jan 2018 (81 comments)
The article/webpage is a nice walk-through for the uninitiated. Half the challenge of doing matrix calculus is remembering the dimension of the object you are dealing with (scalar, vector, matrix, higher-dim tensor).<p>Ultimately, the point of using matrix calculus (or matrices in general) is not just concision of notation but also understanding that matrices are operators acting on members of some spaces, i.e. vectors. It is this higher level abstraction that makes matrices powerful.<p>For people who are familiar with the concepts but need a concise refresher, the Wikipedia page serves well:<p><a href="https://en.wikipedia.org/wiki/Matrix_calculus" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Matrix_calculus</a>
This is the resource I wish I had in 2018. Every grad school course had a Linear Algebra review lecture but never got into the Matrix Calculus I actually needed.
Please change the link to the original source:<p><a href="https://arxiv.org/abs/1802.01528" rel="nofollow noreferrer">https://arxiv.org/abs/1802.01528</a><p>---<p>EDIT: It turns out explained.ai is the personal website of one of the authors, so there's no need to change the link. See comment below.
I finished Vector Calculus last year and have no experience in machine learning but this seems exceptionally thorough and would have made my life easier having a practical explanation over a mathematical one, but woe is the life of the engineering student I guess.
I had followed this when I was learning DL through Andrew NG's course. In one of the lessons, he had the formula for calculating the loss as well as it's derivatives.<p>I tried driving these formulas from scratch using what I learned from OP's post but it felt like there was something missing. I think it boils down to me not knowing how to aggregate those element wise derivatives into a matrix form. Afaik the Matrix cookbook and certain notes from Stanford cs231n that helped me grok it fully
Oh nice, I did most of this in school, and during my non-CS engineering degree. Thanks for sharing!<p>Always wanted to dip my toes into ML, but I've never been convinced of it's usefulness to the average solo developer, in terms of things you can build with this new knowledge. Likely I don't know enough about it to make that call though.
There's a common belief that you don't need math for ML or that you need a lot of math for ML. So let me clarify:<p>You don't need math to make a model perform well, but you do need math to know why your model is wrong.
Another matrix math reference: <a href="https://github.com/r-barnes/MatrixForensics">https://github.com/r-barnes/MatrixForensics</a>
I just had a glimpse look at it. A good sum-up.<p>It seems that these topics are covered by the first one or two semesters of a Math degree. Of course university is a bit more advanced.
We just released a comprehensive online course on Multivariable Calculus (<a href="https://mathacademy.com/courses/multivariable-calculus" rel="nofollow noreferrer">https://mathacademy.com/courses/multivariable-calculus</a>), and we also have a course on Mathematics for Machine Learning (<a href="https://mathacademy.com/courses/mathematics-for-machine-learning" rel="nofollow noreferrer">https://mathacademy.com/courses/mathematics-for-machine-lear...</a>) that covers just the matrix calculus you need in addition to just the linear algebra and statistics you need, etc. I'm a founder and would be happy to answer any questions you might have.