[icapybara]

Tensors have additional properties that arrays don't necessarily have. For example, the coordinate system transform rule that the author describes in the beginning of the post.

One of my old physics professors taught us to think of tensors as "arrays with units." If it's a vector/matrix/higher dimensional array but has physical units, it's probably a tensor. The fact that it has units means it represents something physical which must obey additional constraints (like the coordinate system transformation rule).

[uoaei]

This is a pretty tired line, I gotta say.

Obviously they are not. One is a linear operator, the other is a data structure for implementing computations using that operator. This description extends to all tensors.

It's like saying "queues are not just lists". That is true and also neither insightful nor helpful.

I don't see it as mystifying or complicated, what am I missing?

[ravi-delia]

The reason we carve out a category of tensor, even when you could in principle just define ad-hoc functions on vectors and call it a day, is that we notice the commonalities between a number of objects which are invariant with respect to coordinate changes. Machine learning generally does not use this invariance at all, and has arrays which happen to be tensors for largely unrelated reasons. Calling them tensor makes more sense than calling, say, real numbers tensors, but less sense than calling reals reals.

[ummonk]

Sure but if you’re working with lists you shouldn’t call them queues. Similarly if you’re working with mere multidimensional arrays don’t call them tensors.

[omarhaneef]

Since I also thought Tensors were just higher dimension arrays, isn't this really what ML folks think Tensors are, since they (we?) do attach units to the Tensors most of the time?

[ogogmad]

The physicist's approach is a bit non-conceptual. From a mathematical point of view, a tensor is essentially an arbitrary multi-linear map. Think of the dot product, the determinant of a matrix (which is linear on each column but is not linear on a matrix), the exterior product in exterior algebra (or geometric algebra), a linear map itself (which is obviously a special case of a multilinear map), etc.

The coordinate change stuff that physicists talk about stems from observing that a matrix can be used to represent some tensors, but the rule for changing basis changes along with the kind of tensor. So if M is a matrix which represents a linear map and P is a matrix whose columns are basis vectors, then PMP^{-1} is the same linear map as M but in basis P; if on the other hand the matrix M represents a bilinear form as opposed to a linear map, then the basis change formula is actually PMP^T, where we use the matrix transpose. Sylvester's Law Of Inertia is then a non-trivial observation about matrix representations of bilinear forms.

Physicists conflate a tensor with its representation in some coordinate system. Then they show how changing the coordinate system changes the coordinates. This point of view does provide some concrete intuition, though, so it's not all bad. By a coordinate system, I mean a linear basis.

Hope that helps.

[ummonk]

Not all physicists make that conflation - Wald’s book in General Relativity emphasizes tensors as abstract concepts, and then details how they can be expressed in terms of a basis.

The OP also emphasizes the abstract interpretation as providing more intuition than the coordinate transformation rule.

[cygx]

Physicists conflate a tensor with its representation in some coordinate system

Rather, mathematicians that complain about the physicist's approach just haven't advanced far enough in their studies to understand how vector bundles are associated to the frame bundle ;)

[ajkjk]

There are really just two things called tensors.

Physicists' tensors = generalization of arrays with units; have to transform according to certain coordinate laws.

Mathematicians' tensors = generalization of arrays, transformation rules don't matter.

[klodolph]

> Mathematicians' tensors = generalization of arrays, transformation rules don't matter.

That's definitely inaccurate, at least it doesn't match what I think of as tensors in mathematics.

In mathematics, tensors are the most general result of a bilinear opteration. This does imply that they transform according to certain laws: if you represent the tensor using some particular basis, that basis can be expressed in the original vector spaces you multiplied, and choosing a different basis for your vector spaces results in a different basis for your tensors.

By "most general bilinear operation" I am talking about what is expressed in category theory as a universal property... with a morphism that preserves bilinear maps.

Tensors can be over multiple vector spaces or a single vector space (in which case it's typically implied that it's over the vector space and its dual). When you use a vector space and its dual, I believe you get the kind of tensor that physicists deal with, and all of the same properties. Note that while vector spaces and their dual may seem to be equivalent at first glance (and they are isomorphic in finite-dimensional cases), both mathematicians and physicists must know that they have different structure and transform differently.

Something that will throw you off is that mathematicians often like to use category theory and "point free" reasoning where you talk about vector spaces and tensor products in terms of things like objects and morphisms, and often avoid talking about actual vectors and tensors. Physicists talk about tensors using much more concrete terms and specify coordinate systems for them. It can require some insight in order to figure out that mathematicians and physicists are actually talking about the same thing, and figure out how to translate what a physicist says about a tensor to what a mathematician says.

[red_trumpet]

But they are not totally different! The Physicists' tensors are actually Mathematicians' tensors, but parametrized by some parameters (coordinates). Then you have special laws what happens if you make a (possibly non-linear) change of the coordinates. See https://en.wikipedia.org/wiki/Tensor#Tensor_fields