I vaguely remember some paper where they didn’t even bother normalizing the vectors, because they expected zeros to be very close to zero, and anything else was considered a one.
I have no idea if this a common optimization or if it was something very niche. It was for a heuristic matrix reordering strategy, so I think they were willing to accept some mistakes.
In cosine similarity, yes.. iirc there is a recent paper arguing that this causes cosine similarity to perform poorly in high dimensional (aka ML) vector spaces.
I think he is talking about the 2024 arxiv paper by some netflix (?) researchers that say that it's best not to normalize the dot products (so instead of cosine similarity you just have a dot product).
For most commercial embeddings (openai etc) this is not a problem as the embeddings are already normalized
They don’t have to be. The computation of cosine distance normalizes the distance.
In the intuitive explanation I gave, the distance from the origin doesn’t matter at all, unless you are trying to force “cosine disbtance” to mean “metric distance”
There are many many ways to quantify closeness. Metric distance i.e taking a ruler and measuring the distance between the 2 points, is one way.
Measuring the angles between the 2 lines that join the points to the origin is another way to measure closeness
The squared of the measured metric distance is another way
The absolute value of the ruler distance is another way
The question you asked is only relevant if you’re trying to force cosine distance into ruler distance. You don’t need to. Cosine distance is a sufficient way by itself to measure closeness.
But yeah generally speaking, costume distance correlates better with metric distance even the points are relatively more equidistant to the origin
I have no idea if this a common optimization or if it was something very niche. It was for a heuristic matrix reordering strategy, so I think they were willing to accept some mistakes.