|
|
|
|
|
|
by krackers
10 hours ago
|
|
|
There was a longform post on twitter which went through the same derivation at a bit higher level https://x.com/waterloo_intern/article/2081762065392541951 and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n... I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value |
|
|
