[cabinpark]

I always see linear algebra on HN and many people comment on how they never understood the subject. This makes me ask: what exactly is it that people don't get about linear algebra? What makes it appear to be a difficult subject?

As someone who has used linear algebra almost every day in some form over the last decade, it's hard to get a perspective of what aspects are challenging to the beginner. And since I TA courses that involve linear algebra, it is good to know where the problems are.

[dagw]

I found that the level of abstraction that linear algebra is taught at is really weird. On the one hand it was never really taught in connection with algebra in general, or put into a wider mathematical context. On the other hand we were never really shown any real world used of linear algebra or how the concepts maps onto practical applications.

It wasn't before years later when A) I'd taken more abstract algebra courses that introduced concepts like tensors and fields and B) I'd taken more practical computational courses where I had to do things like use linear algebra in 3D graphics and use eigenvectors to do dimensionality reduction and PCA, that I really understood the subject and its place in the world.

Just saying here's a random object (that we'll call a "matrix"), here's some random steps (that we'll call "taking a determinant"), now memorize how to apply the steps to the object and see if the result is zero, didn't lead to much deeper understanding.

[neuronic]

I think what courses often fail at is to specifically mention that all of these things are abstractions invented by people to deal with a specific issue and then WHY IT HELPS solving this issue.

Whenever I studied Linear Algebra it was simply "here's a matrix, here is some algorithm, use it to get some number seemingly coming out of nowhere and just believe us that this number is what you really need."

Note that I am not immediately interested in mathematical proof of why this method is correct. I want a plain English explanation of what is going on but instead you usually just get a bunch of notation to digest.

[dagw]

The other aspect that I found with LA compared to almost any other math subject I took was that it was almost entirely mechanical with little to no connection between my understanding of the subject and my grade. The exam questions where mostly along the lines of "find the eigenvector" and "find the determinant" rather than "what's an eigenvector" and "what's a determinant", so I got good grades despite not having any idea how to answer the latter questions, or really having any idea what I was doing. Understanding was neither necessary nor encouraged.

[Ocerge]

This, exactly. I could find an orthonormal basis or an eigenvector, but I never quite wrapped my head around their importance, especially visually.

[dalke]

I see that chapter 10, on eigenvalues/eigenvectors, is "Coming soon!".

Eigenvectors never made intuitive sense to me. As with the various decompositions (Cholesky, LU, etc), I could apply the math as algorithms to follow, but never got to the point where felt I could apply them to new problems.

Then again, in practice, I've only needed eigenvectors once since college, and it was more a rote implementation described in a paper. (In other words, don't feel like you should educate me on them here.)

Since you are a TA, don't you get some idea of where your students struggle with linear algebra?

[jeffwass]

Linear algebra (and most calculus) didn't really click for me until I needed to apply it to physics problems.

The general idea started to make sense in mechanics class when I could see matrices are convenient shorthand for solving multiple variables at once, and which behave like 'regular' variables when trying to manipulate them algebraically.

Eigenvalues and eigenvectors didn't make sense until quantum mechanics, where your 'operator' is effectively a matrix and your 'state' is a vector. The allowed observed values are the eigenvalues, and your final state after measurement is the corresponding eigenvector. Wave functions are then an extension of this from discrete space (useful for modelling spins for instance) to the continuum of position space.

[Steuard]

I love eigenvectors. For me, the rest of linear felt like a bunch of rote rules and algorithms for figuring out vaguely-interesting stuff. And then suddenly I find out that there's this amazingly non-obvious but remarkably powerful structure hidden inside matrices that I'd never even been aware of before.

Quantum mechanics is a huge application of eigenvectors, but I also really enjoy things like the "moment of inertia tensor", whose eigenvectors are the natural axes of rotation. Or better still, coupled oscillators, where the eigenvectors give "normal modes" of vibration. (And if you look at coupled first-order differential equations, eigenvectors can tell you all sorts of things about "trajectories" of the solutions. There are great applications of that to things like population dynamics in biology.)

[geomark]

I enjoyed this articled called "Principal Components Analysis for Dummies" that illustrates eigenvectors and eigenvalues. https://georgemdallas.wordpress.com/2013/10/30/principal-com...

[steamer25]

There's also this interactive page for eigen- vectors and values: http://setosa.io/ev/eigenvectors-and-eigenvalues/ (as well as a sibling for PCA: http://setosa.io/ev/principal-component-analysis/)

[dougabug]

The point of an eigenvector is that its direction is stable when you hit it with the given transformation. That is, it's an invariant direction of your transformation. This is analogous to the derivative of an exponential function being a simple multiple of that function. It makes it easier to reason about the action of the matrix, and simpler to represent.

Imagine I have a matrix transformation that takes a given input vector and converts it into a superposition of a thousand other vectors in random directions; that's far harder to reason about than if if just kicked it farther or brought it closer in the same direction.

[cabinpark]

Interesting. I used linear algebra with quantum mechanics where eigenvectors represent quantum state of the Hamiltonian which is how I initially understood them.

And as for TAing, have you ever TA'd? You definitely get a feeling but I can count numerous times when I've stood in front of the tutorial class and asked if there are any questions only to get no response back. I think it's a symptom of first years. I've TA'd calculus as well and I get similar responses. It's very frustrating sometimes.

[Jtsummers]

> You definitely get a feeling but I can count numerous times when I've stood in front of the tutorial class and asked if there are any questions only to get no response back.

When I realized I was understanding subjects better than my classmates, I'd take on the task of asking the "dumb" questions for them. It was partly selfishness, I was tired of answering those questions for them outside of class. But it really did prove helpful to my classmates. It really helped that, when I was sitting with the students, I got to hear them mumbling and grumbling about what they didn't get. So I knew exactly what questions to ask to get the professor/TA to help my classmates.

I never TA'd myself so I have no idea if this would actually work, or at least work consistently. But, if you have a couple students that really seem to be getting the material, you could try talking to them one-on-one and ask them to help you out in this manner.

[dalke]

I have TA'ed. Asking for questions in public doesn't help much. Students rarely like to admit they don't know something.

I was thinking more of when you look at their assignments. There are often multiple ways to approach a problem, and the route chosen can reveal something about one's level of comfort.

It doesn't help that university TAs get almost no training in how to be a TA. At least, I didn't.

[cabinpark]

Yeah I know. It's hard to squeeze information out of them. I actively tell them to please, please come to me if they don't understand something and I have office hours for a reason. Yet no one takes advantage. At a certain point, you can only do so much since I'm a grad student and not a lecturer and my time is finite.

The problem I've found with assignments though is that people copy and cheat. Many times someone will do very well on assignments and then do absolutely terrible on midterms and finals. It's very frustrating. I remember one course where everyone did nearly perfect on the assignments and yet the final and midterm followed the standard bell curve.

[cokernel]

It's worth devoting a portion of classroom time (in my opinion, a substantial portion) to discussion of the topic, perhaps focusing on particular problems and generalizing from there. For example, you might select random students each day to present problems from homework. If you've ever noticed that you learn something better once you teach it to someone else -- well, it works for your students, too.

Moreover, if you create a non-judgmental environment in which people are free to talk about their approaches to problems and get feedback not only from you but from other students as well, then just by watching carefully, you will learn some of the more common gaps in understanding. (Note that some students will not talk in these situations unless forced, but that does not mean they do not benefit from following the discussion.)

If you're anything like I was when I was first TAing courses like this, you might think that if you do this, you won't have enough time to "cover the material". But I put it to you that a lecture that is not absorbed doesn't cover anything.

[wodenokoto]

What it is, I think was my major problem.

Matrix multiplication, ranks etc. are some of the most random seeming operations you can tell a student to memories.

[dalke]

My view is that the people who get to matrix algebra didn't have problem understanding the random seeming operation of previous math courses. Consider the existence of the "FOIL" mnemonic for multiplying two binomials (https://en.wikipedia.org/wiki/FOIL_method ), or SOH-CAH-TOA, where it was easier for me to just remember the definitions than the acronym. I think those are hints that some people consider those to be 'random seeming'.

When I took algebra, I remember being tripped by all the new vocabulary. Why do mathematicians say something is "Abelian" instead of 'commutative'? Why is it important to have terms like "group", "ring", and "vector space"? Sure, I learned the definitions, but at that level they were 'random seeming'.

[cokernel]

At my (American, liberal arts, flyover country) university, linear algebra was my first real mathematics course. (The prior classes were calculus and a "proof" class, AKA intro to intro to number theory.) I'd been introduced to the determinant, Gauss--Jordan elimination, and so on, in high school, but all in a very mechanical way.

I think this transition between mechanically generating solutions to problems using pattern matching and some basic algorithms on the one hand, and the more mathematically mature approach of exploring problem spaces using pattern matching, some basic algorithms, and intuition can be difficult for many people.

It's not necessarily the material -- you can get used to almost anything, and even convince yourself it's easy or obvious with enough familiarity -- but the lack of intuition. When you're first learning linear algebra, its fundamental unity is not obvious, especially if the instructor does not take pains to point it out. (And even if the instructor does take pains to point it out -- well, it's hard to understand why the instructor is saying we could do this computation this way or that way.) So in the absence of existing intuition or any perception of unity, linear algebra becomes another target for pattern matching and basic algorithms.

As it happens, I've never, ever felt like I didn't "get" linear algebra. However, I almost always feel like I "get" it now and all my prior conceptions of it were a confused muddle.

[ashishb4u]

I think it's mostly because mathematics is taught as is with little or no attention to practical applications of the theory. For instance, till date, I could not see how eigen values/vectors are useful.

[cousin_it]

One nice application is that the eigenvectors of a stochastic matrix correspond to stationary states of the Markov chain. That leads naturally to a conversation about PageRank, which is super useful because it made billions of dollars for some people. Then you could talk about the use of eigenvectors for principal component analysis in data science, which is making a lot of money for other people right now. If these examples are not useful, then I don't know what is.

[ashishb4u]

Thanks, the application to PageRank looks interesting. I shall resurrect my knowledge on eigenvectors and eigenvalues and make some sense out of it.

[cabinpark]

Interesting. Eigenvalues and eigenvectors are one of the most fundamental applications of linear algebra. In physics, quantum mechanics is a giant eigenvalue-eigenvector problem.

One application I really like is in machine learning: the eigenface algorithm.

[neuronic]

That might all be the case. I would simply like a thoughtful introduction to the topic about where this abstraction comes from. What is the idea behind it and what was the motivation for its invention?

After I understand the plain English concept, then give me the math notation and proof and applications.

The way it works today is often to simply leave out the first part and I believe this is why many people find it hard to develop intuition and a real understanding of the concept.

For man it's just there, you memorize it, you apply it, you take the result and simply have faith that it is what you need because some book/ prof said to solve problem X use eigenvalues.

[zornthewise]

The simple idea is that while linear maps(matrices) are generally a combination of rotations, scaling and other stuff in higher dimensions, sometimes they will act on a vector simply by scaling it.

Since scaling is a really easy operation to understand, the space generated by these vectors will be really easy to understand. Note that different vectors can be scaled by different amounts.

Now it often turns out that the space generated like this is actually the whole of the space under consideration and this really simplifies the linear map we started with. Hope that helps.

[yardie]

I took LA in university and I really wish they had offered it in high school or even middle school. It was a requirement for my major but not a requirement for any of my other math courses so I treated it like an elective. It was only after I took the course and started applying it that I realized how applicable it was to almost every math problem.

Instead of calculus for business majors or art majors they really should be using LA instead.

At the moment I'm trying to teach my son LA but I can't find any books for 8-10yo

[ThrustVectoring]

It's a global vs sequential learner thing. Linear Algebra is usually taught as a giant pile of tricks, techniques, and terminology. This is tractable for sequential learners like myself - I really enjoy picking up new tools in a narrow context just for it's own sake. Highly global learners basically can't learn anything this way.

So, in short, my advice is "add more context."

[noobie]

What's challenging to me is my inability to develop intuition about LA simply because most of the time I have no idea what (the purpose) I am doing.

How does one build intuition about LA?

[gopowerranger]

In almost every case, confusion arises when the "how" is taught with no explanation of the "why". Why does that formula work? Why did you use that formula?

Whole worlds were opened up to me when I learned that the fundamentals of trigonometry, like the simple computation of sin of an angle, are all related at its base to a square but no math teacher ever made that connection for me.