[js8]

It's not correct to call them orthogonal because I don't think the definition is a dot product. But that aside, yes, orthogonal basis can only have as much elements as dimensions. The article also mentions that, and then introduces "quasi-orthogonality", which means dot product is not zero but very small. On bitstrings, it would correspond to overlap on only small number of bits. I should have been clearer in my offhand remark. :-)

[prerok]

Your initial statement is still wrong, that you can include a lot of information in a small number of bits. If you have a small number of bits, the overlap will be staggering. Now, that may be ok, but not ok, if you want to present orthogonal concepts (or even quasi-orthogonal).

Also, why do you believe dot product cannot be trusted?

[js8]

What I meant was similar to the article. If I have bit vectors of length 1000 bits (that will be the embedding). Let's say that every concept I want to model corresponds to a set choice of 10 bits being all set to 1, and the remaining bits 0. (These vectors are quasiorthogonal.) Then I can easily store a sentence (bitvector) about 50 concepts, while there is relatively small chance of overlap, i.e. all of the concepts are decodable from that bitstring.

But it's quite similar to what the top comment is saying about spherical codes. I think my comment is also about using coding theory to represent concepts.

Other than that, I don't have any issue with dot product over bitvectors - it's just not very useful for the above.