[soVeryTired]

For a large fraction of probability theory, you only need two main facts from linear algebra.

First, linear transforms map spheres to ellipsoids. The axes of the ellipsoid are the eigenvectors.

Second, linear transforms map (hyper) cubes to parallelpipeds. If you start with a unit cube, the volume of the parallelpiped is the determinant of the transform.

That more or less covers covariances, PCA, and change of variables. Whenever I try to understand or re-derive a fact in probability, I almost always end up back at one or the other fact.

They're also useful in multivariate calculus, which is really just stitched-together linear algebra.

[qmalzp]

I think the first point is only true for symmetric matrices (which includes those that show up in multivariable calc). In general, the eigenvectors need not be orthogonal.

[soVeryTired]

Yep, you could well be right. The image of an ellipse under a linear transform is definitely an ellipse, but I'm not sure about the eigenvectors in the general case.

The symmetric case is by far the most relevant for probability theory though.

[tprice7]

In general it's the eigenvectors of the positive-semidefinite (hence symmetric) part of the left polar decomposition.

[zawerf]

I use the 2nd point a lot for debugging 3d transforms. To expand upon it, for example in three dimensions the three axes are:

(1, 0, 0)

(0, 1, 0)

(0, 0, 1)

To find out where those axes are after a 3x3 matrix transform, you just read off the first, second, and third columns of the matrix respectively. Then you can mentally visualize another unit cube in the new coordinate system using those three vectors as the edges of the cube.

Really basic change-of-basis stuff but academic lectures don't emphasize how useful it is to be able to look at a matrix of numbers immediately know what it does.

[aidos]

This concept totally changed my intuitive understanding of matrices. Beautifully illustrated in the below 3blue1brown video.

https://youtu.be/kYB8IZa5AuE?t=3m15s

[sakuronto]

Not a very useful addition but hypercube is to cube as parallelotope is to parallelepiped.