They use matrices for computational efficiency. That's why linear algebra (along with diff equations and probability theory) is one of the prerequisites for any non-mooc machine learning course.
I've programmed neural networks without knowing any linear algebra. When I needed to figure out how to use vector operations for speed, it took like 5 minutes to search for matrix multiplication on wikipedia. You can't get by without even knowing that, as element wise operations can do everything just as fast.