An Intuitive Guide to Linear Algebra

As an electrical-engineer turned machine-learning-grad-student, linear transformations have been involved in most everything I do since my first year of undergrad. But all this time, I've done matrix multiplication the way I was taught in high school: "The (i,j) element of AB is what you get by walking right across the i'th row of A while you walk down the j'th column of B, taking the sum of products as you go."<p>It works, but there's no connection between that process and the intuition of a linear transformation; it's just a rote computation. And checking a long string of matrix multiplications to see if they intuitively make sense (shouldn't everything intuitively make sense?) is especially aggravating when you constantly have to interrupt your intuition to switch to a rote calculation.<p>I never thought to think of the columns of B as vectors that physically travel through A; to think of a dataflow or pipeline from right to left on the page. Sure, it's not a cure-all, but it'll be a useful mental tool to have.<p>Oh, and it's also an excellent introduction to the subject, although the Linear Operations section gets a bit muddled... first something's not a linear operation, and then it is, wat? Still, an excellent post.

Strang from MIT has an extraordinary set of lectures (his textbook is even better): <a href="http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/" rel="nofollow">http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-...</a>

&#62; Linear algebra gives you mini-spreadsheets for your math equations.<p>Okay, that lets you visualize it (in the finite case) but it's a terrible way to sell it. Spreadsheets are booooring. Did you know that functions are vectors? Okay, better. Did you know that quantum mechanics is all about linear algebra? Okay, sold!<p>1. Almost any time you work in more than one dimension you will want linear algebra in your toolbox. There are a zillion methods for solving (non-linear) equations out there, and in more than one dimension, they use linear algebra. Newton's method? Incredibly useful in practice due (quadratic convergence rocks!), and with some linear algebra sauce BOOM you have Newton's method in as many dimensions as you can sneeze at.<p>2. Oh, by the way... did you know that the Fourier transform is linear?<p>3. Back to quantum mechanics... there's a thing you can do with a linear operator (a matrix is a kind of linear operator) where you get the "spectrum" of the linear operator. It's useful for making sense of big matrices. But in QM, the wavefunctions for electrons are described as eigenfunctions of a linear operator, and taking the "spectrum" of the linear operator gives you the actual spectrum of light that the chemical under study emits. Hence the name, "spectral theorem". It may be linear algebra on paper, but it's laser beams and semiconductors in the real world.<p>4. Oh hey, want to learn about infinite-dimensional vector spaces? Maybe some other time..<p>5. It's hella useful for modeling. Any model is wrong, but Markov processes are useful. Say you run an agency that rents out moving vans, and you have facilities in 30 cities. Vans rented in city A have a 10% chance of being dropped off in city B, 7% in city C, 9.2% in city D, etc. At this rate, how long till you run out of vans in city F? It's a differential equations problem with like 30 different equations! Or you could rewrite it as a single equation with matrices. You'll end up with weird things like 'e^(A*t)' where A is a matrix, and you thought "no way I can exponentiate to the power of a matrix" but spectral decomposition is like "yes way!" and you can solve the equation by diagonalization. Radical! (Basically, linear algebra rescues differential equations from the pits of intractability. I'm using rental vans as an example, but it could be a chemical reaction or a nuclear reaction or a million subway riders or whatever you want.)<p>So the question is:<p>Do you find economics, quantum physics, chemistry, engineering, classical mechanics, machine learning, statistics, etc. useful?<p>Then get some linear algebra in you!

I hope that this matches how some people think enough to help them.<p>For me it is too computational. I prefer understanding the topic from first principles as described at <a href="http://news.ycombinator.com/item?id=4086325" rel="nofollow">http://news.ycombinator.com/item?id=4086325</a>. (Then again I don't particularly like spreadsheets either.)

I found "Linear Algebra Done Right" to be a much more intuitive introduction to the subject. It doesn't get to determinants until the end.<p><a href="http://linear.axler.net/" rel="nofollow">http://linear.axler.net/</a><p>After going through that course I finally understood things like eigenvectors, null spaces, and projections. Now I see them everywhere (unless you think that's a curse)

Funny, I'm actually working on a game that intends to teach some of this stuff to a non-mathematical audience.<p>Linear algebra is so far-reaching, I find it surprising that other branches of mathematics often seem to get preferential treatment (usually normal algebra and geometry), in spite of the fact that linear algebra is both:<p>a) fairly advanced (i.e. not often taught in school, at least not the deeper stuff)<p>b) not very difficult to learn (unlike lots of other 'introductory' topics in mathematics).<p>Perhaps there is something about matrices (being mere tables of numbers for most folks) that people find unattractive, almost statistics-like.<p>(On the other hand, it could be a simple extension of the symbol barrier [1], given those long vertical brackets.)<p>[1] Prof Keith Devlin introduces this concept here: <a href="http://profkeithdevlin.org/2012/02/22/how-to-design-video-games-that-support-good-math-learning-level-3/" rel="nofollow">http://profkeithdevlin.org/2012/02/22/how-to-design-video-ga...</a>

"The eigenvectors are the axes of the transformation" = mind blown. After several engineering courses, studying eigenvectors in an advanced math class and still <i>no one</i> could put it this simple. This guy is amazing.

What the author uses as his strawman is a linear algebra course for engineers.<p>As mathematicians, we didn't do any of this matrix and vector stuff with numbers when introducing linear algebra in university. There were a bunch of axioms, and you proved things. That's how you know.<p>What the author sees as abstract "(2d vectors! 3d vectors!)" was way more applied than the stuff we dealt in.<p>But, granted, the purpose wasn't learning about how to get mini-spreadsheets for equations. It was about how to rigorously navigate a useful axiomatic setting.<p>(Later on, we proved that you can find a base, and write down your linear transformation as a bunch of numbers and call that a matrix; same for points and vectors. But we always saw that as somewhat ugly, and anyway limited to the finite dimensional case.)

Terry Tao has a great set of lecture notes available online, too. <a href="http://www.math.ucla.edu/~tao/resource/general/115a.3.02f/" rel="nofollow">http://www.math.ucla.edu/~tao/resource/general/115a.3.02f/</a>

These're good too<p><a href="http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-xml-latestli14.xml" rel="nofollow">http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-xm...</a><p><a href="http://joshua.smcvt.edu/linearalgebra/book.pdf" rel="nofollow">http://joshua.smcvt.edu/linearalgebra/book.pdf</a><p>Also, the common 1st year texts (Anton, Lang, Hoffman/Kunze and Friedberg/insel/Spence) can be found easily for cheap, used. The old edition of Strang I used to have was good too, but some people react really strongly when you bring it up. There's lots of ways ot sequence LA and the needs of EE's, econometrics/game theory, prob/stats and applied mathies are different from physics/math majors. Look at ToC's and read the Holyoke prof's writeup:<p><a href="http://www.amazon.com/Introduction-Linear-Algebra-Fourth-Gilbert/product-reviews/0980232716/" rel="nofollow">http://www.amazon.com/Introduction-Linear-Algebra-Fourth-Gil...</a><p>(i <i>think</i> they're 1st year texts, my Dad's a physics prof and he started talking about determinants around 7th grade)

Nice work. It's a richly geometric area.<p>Other things that might be nice to include are dot products (projecting one vector onto another as a measure of co-linearity) and rotation matrices (you could keep it to 3x3).

Great post. A suggested follow-up: "Banach Spaces, The Spectral Theorem, and Your Changing Body: A Personal Introduction."

I think an intuitive guide should have more diagrams of affine transformations. Don't just visualize the matrix, visualize the transformation.

Eero Simoncelli's guide to linear algebra is very good too and likely more concise, although it doesn't cover eigenvectors and eigenvalues.<p><a href="http://www.cns.nyu.edu/~eero/NOTES/geomLinAlg.pdf" rel="nofollow">http://www.cns.nyu.edu/~eero/NOTES/geomLinAlg.pdf</a>

For the mathematicians here. I am taking Linear Algebra classes as a CS undergrad. Can someone recommend a very good book?<p>I am looking for the kind of book that will make you fell in love with Linear Algebra. For Calculus, I used Piskunovs Differential and Integral Calculus which was miles away from what my classmates were using. That along with Maple help to double check that my stuff was correct proved a good combo. My current Linear Algebra book is an honest book but it is a boring book, it fails to entertain or to amaze or to give you those moments of insight that puts a grin in your face.<p>I think I just wish I had better books as an undergrad :-(

OK can you explain kernels and their role in transformations (f.e. the dimension formula)? No you can't this way. And it truly has an INTUITIVE explanation (so much more than the intuitive ess of a spreadsheet!).

Obligatory plug for the <i>Manga Guide to Linear Algebra</i>:<p><a href="http://nostarch.com/linearalgebra" rel="nofollow">http://nostarch.com/linearalgebra</a><p>:D

Interestingly this is more or less like the way I was taught linear algebra in high school.

Every now and again I think my highfalutin college courses were an overpriced waste, and then a conversation like this comes along and I see that the fundamentals of my courses are apparently radical and unheard of most other places.<p>Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers?<p>/took linear algebra in the math department, then TAed it.

I think it's much clearer to define everything in terms of vector spaces and linear operators.

I might be missing something big, but it seems like linear algebra is then just an overly complicated way of describing what should be a simple toolbox of spreadsheet manipulation functions, kind of like an extra module for python. Why is it given such special emphasis then, it would be like teaching the datetime or sched modules.