|
In the first example on https://www.tensorflow.org/api_docs/python/tf/math/multiply you can see that they use the Hadamard product (not the matrix product): x = tf.constant(([1, 2, 3, 4]))
tf.math.multiply(x, x)
<tf.Tensor: shape=(4,), dtype=..., numpy=array([ 1, 4, 9, 16], dtype=int32)>
I could stop right here since it's a counterexample to x being a matrix (with a matrix product defined on it; P.S. try tf.matmul(x, x)--it will fail; there's no .transpose either). But that's only technically correct :)So let's look at tensorflow some more: The tensorflow tensors should transform like vectors would under change of coordinate system. In order to see that, let's do a change of coordinate system. To summarize the stuff below: If L1 and W12 are indeed tensors, it should be true that A L1 W12 A^-1 = L1 W12. Try it (in tensorflow) and see whether the new tensor obeys the tensor laws after the transformation. Interpret the changes to the nodes as covariant and the changes to the weights as contravariant: import tensorflow as tf
# Initial outputs of one layer of nodes in your neural network
L1 = tf.constant([2.5, 4, 1.2], dtype=tf.float32)
# Our evil transformation matrix (coordinate system change)
A = tf.constant([[2, 0, 0], [0, 1, 0], [0, 0, 0.2]], dtype=tf.float32)
# Weights (no particular values; "random")
W12 = tf.constant(
[[-1, 0.4, 1.5],
[0.8, 0.5, 0.75],
[0.2, -0.3, 1]], dtype=tf.float32
)
# Covariant tensor nature; varying with the nodes
L1_covariant = tf.matmul(A, tf.reshape(L1, [3, 1]))
A_inverse = tf.linalg.inv(A)
# Contravariant tensor nature; varying against the nodes
W12_contravariant = tf.matmul(W12, A_inverse)
# Now derive the inputs for the next layer using the transformed node outputs and weights
L2 = tf.matmul(W12_contravariant, L1_covariant)
# Compare to the direct way
L2s = tf.matmul(W12, tf.reshape(L1, [3, 1]))
#assert L2 == L2s
A tensor (like a vector) is actually a very low-level object from the standpoint of linear algebra. It's not hard at all to make something a tensor. Think of it like geometric "assembly language".In comparison, a matrix is rank 2 (and not all matrices represent tensors). That's it. No rank 3, rank 4, rank 1 (!!). So what does a matrix help you, really? If you mean that the operations in tensorflow (and numpy before it) aren't beautiful or natural, I agree. It still works, though. If you want to stick to ascii and have no indices on names, you can't do much better (otherwise, use Cadabra[1]--which is great). For example, it was really difficult to write the stuff above without using indices and it's really not beautiful this way :( More detail on https://medium.com/@quantumsteinke/whats-the-difference-betw... See also http://singhal.info/ieee2001.pdf for a primer on information science, including its references, for vector spaces with an inner product that are usually used in ML. The latter are definitely geometry. [1] https://cadabra.science/ (also in mogan or texmacs) - Einstein field equations also work there and are beautiful |