[muragekibicho]

For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.

Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.

Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.

[FabHK]

My intuitive take:

Eigenvectors answer the question, if you have a linear mapping from a space to itself: which lines through (through the origin) remain unchanged after the mapping? And the associated eigenvalues tell you how much a specific point on that line moves along the line. For example: If you rotate things in 3D, the rotation axis remains unchanged. That's the eigenvector. If you reflect things through a mirror plane, any line in that mirror plane will remain unchanged (with eigenvalue 1). Other lines will be, well, mirrored, but there is one line that is reflected onto itself: the one perpendicular to the mirror plane. However, points on it will go from one side to the other, so have eigenvalue -1.

The SVD says something else: Any linear mapping from a space (to itself or a different one - higher or lower dimensional) can be expressed as 1) first a rotation in the old space, then 2) some coordinate scaling or dilation (expansion/contraction) along the coordinate axes, described by the diagonal matrix with singular values, 3) then another rotation in the new space.

[toolslive]

Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.

https://www.oceanopticsbook.info/view/photometry-and-visibil...

[wtallis]

You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.

[toolslive]

hold on. Just realised it's not even (part of ) a vector space. It's not linear, non-euclidian and can't be mapped onto 3 dimensions. It's not even constant or always reproducible:

    Look at an object; 
    Observe colour X; then look at the wrong thing; 
    then look back and you might observe colour X' != X.

[toolslive]

Doesn't matter: there's no RGB model that captures the colour space. That exactly the reason CIE exists.

[adrian_b]

CIE (1931) is a RGB color space, based on monochromatic primary colors with wavelengths of 700 nm (red), 546.1 nm (green) and 435.8 nm (blue).

However the entire color space of CIE RGB (1931) includes points where some of the RGB components are negative.

Because positive components are sometimes desirable (e.g. because one can make light filters whose outputs are those components), the alternative XYZ representation is derived by computation from the original CIE RGB, which had been obtained from experiments with human subjects.

Any RGB model can capture the entire color space, even sRGB. The limitation is not at capturing, but only at reproducing colors when using RGB emitters, because the emitters cannot reproduce components with negative values.

There are no RGB models that can capture the entire color space without having points where some components are negative. This is caused mostly by defects of the human color vision, e.g. by the fact that the red receptors are also sensitive to violet light, not only to red light, and by the fact that the selectivity curves of the photoreceptors do not have ideal shapes.

[jacobolus]

You can represent any arbitrary colors using RGB values for whatever "R", "G", and "B" primaries you like, they just might not fit in the range [0, 1].

(At least, under the strong assumptions we make in color modeling using the CIE system of colorimetry; the basic keyword is "Grassmann's laws".)

[interroboink]

Since you seem to know, and I am curious, doesn't CIE[1] effectively use RGB to describe its space, too? Eg: the r̅(λ) g̅(λ) b̅(λ) color matching functions? Or is there something else in CIE you're referring to?

[1] https://en.wikipedia.org/wiki/CIE_1931_color_space

[klysm]

Not true, you can make the R G and B points far enough out to capture the entire human gamut

[inigyou]

You can specify any colour if you can use values below 0 and above 100%.

[esafak]

This is solved through https://en.wikipedia.org/wiki/Primary_color#Imaginary_primar... that sit outside the visible spectrum.

[piker]

Okay, but that was a really useful metaphor if incomplete in a lot of ways. It made me say “oh”.

[Aardwolf]

The eyes receptors are RGB too and they definitely capture all visible colors by definition (to be pedantic, yes there's also the rods for night vision, and the 'R' is more like yellow)

[toolslive]

Tetrachromacy !?

[yaroslavvb]

Eigenvalue are more flexible than singular values -- If you have procedure for computing eigenvalues of C, squared singular values of X are eigenvalues of C=XX'. Left and write singular vectors of X and eigenvectors of XX' and X'X. For rotation rotation matrix, singular values are all 1. Meanwhile eigenvalues tell you angles of rotation https://math.stackexchange.com/questions/4874616/obtaining-a...

[xscott]

That's like saying spoons are more flexible than forks because you have soup (rotation matrices). Spoons and forks both work for rice (pos def matrices), and you'll want a fork for noodles (rectangular matrices).

[yaroslavvb]

You can use eigenvalue solver to find singular values of any matrix but not vica versa. Implementing eigenvalues needs complex numbers

[xscott]

And you can use a spoon to chop spaghetti into pieces. Say I have a 10,000 by 100 matrix (1M elements), squaring that to get a 10,000 x 10,000 (100M elements) matrix to get the singular vectors is probably a bad idea. That squares the condition number too.

The SVD staying in Reals when you have Real data is a nice feature.

[lr1970]

> Implementing eigenvalues needs complex numbers

This is exactly why eigenvalues are less flexible than singular values. You can have a real valued matrix that does not have any eigenvalues in the field of real numbers, all eigenvalues are complex numbers. Examples: rotation matrixes in R^2 have no eigenvalues in R. Singular values, on the other hand, are always real (they are eigenvalues of the Hermitian matrix MM^*) and can be used the same way for the matrices over real (R) or complex (C) numbers, hence extra flexibility. Added bonus -- singular values are never negative.

Instead of "more flexible" a better statement would be -- eigenvalues convey more information than singular values.

[big-chungus4]

Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD because it uses a faster method to compute matrix sign. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"

[jmalicki]

ADAM is related if your second derivative matrix happens to be diagonal.

Of course, it takes about 5 minutes to show that any DNN is going to have very very high magnitude off-diagonal terms by the way it's constructed, so pretending that a diagonal approximation is close enough is crazy.

[big-chungus4]

Adam doesn't use the second derivatives matrix, it uses second moments of the gradient, which is the diagonal of the uncentered covariance matrix, but neither of them are directly related to SVD or singular values anyway.

There is a slight connection where Adam approximates full-matrix Adagrad which computes inverse square root of the convariance matrix, which you usually do using eigendecomposition, but on the covariance matrix SVD and eigendecomposition are equivalent (can easily be converted to each other), so you could use SVD to compute the inverse square root.

[jmalicki]

The second moments of the gradient and the Hessian are absolutely related!

See the Fisher Information, and the Cramer-Rao Lower Bound (an inequality on how much the inverse covariance matrix and the Hessian can differ).

https://en.wikipedia.org/wiki/Fisher_information

[jmalicki]

I didn't find a direct proof earlier, just assertions - I've only seen it proved in textbooks that aren't linkable.

Theorem 1, section 1.3, page 2 shows that the expected variance of the gradient of the loss function and the expected second derivative of the loss function are equal at the minimum. I hate that the ADAM paper did not talk about this, this is something that is hammered into anyone who has taken a mathematical statistics course. This has been an established fact in statistics for well over 100 years.

https://courses.grainger.illinois.edu/ece563/fa2025/Notes11-...

Away from the minimum they can diverge, but there is a close enough connection to make it an extremely useful approximation.