[PaulHoule]

Even though we understand a lot more about how LLMs work and have cut resource consumption dramatically in the last year we still know hardly anything so it seems quite likely there is a better way to do it.

For one thing dense vectors for language seem kinda insane to me. Change one pixel in a picture and it makes no difference to the meaning. Change one letter in a sentence and you can change the meaning completely so a continuous representation seems fundamentally wrong.

[tim333]

I get the impression human brains process things a lot more efficently so there's probably a way to go there.

[thfuran]

Well, they do manage to get by on about 20 W.