[acchow]

Intuitively, there is a ton of redundancy and we still have a long way we can still compress things.

[imtringued]

Each token is represented by a vector of 4096 floats. Of course there is redundancy.