[hasley]

I suspect with "orthogonalization" they mean to find vectors that form an orthogonal bases (same subspace) for the vectors in the source matrix.

I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.

[aesthesia]

The Newton-Schulz iteration they use approximates setting all singular values of the matrix to 1. That computes the nearest orthogonal matrix under the Frobenius norm.

[hasley]

Interesting, thanks!

[CamperBob2]

3D graphics and kinematics people dodge the need for periodic orthonormalization by using quaternions. When they need a rotation matrix, they create it on demand rather than having to maintain it incrementally.

I wonder if there's a similar shortcut representation that we will eventually realize we should be using for ML. I suppose if there is one, it won't have native GPU support, so no one will bother looking for it.