[dartos]

> predict a weight of exactly zero for some of the values

Wouldn’t this be pretty unlikely, though?

[schopra909]

Quite the opposite — if you have a long sequence only a smattering of the words will influence the meaning of the current word. Everything else is “noise”.

Attention is really good at finding this smattering of words (ie assign most weight there). But it struggles to put exactly 0 on the other words.

[dartos]

I mean wouldn’t it be unlikely that

SoftmaxA[n] - SoftmaxB[n] is exactly 0?

Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.

[absoflutely]

why say lot word when few word do

[dartos]

Few word no do tho

[kridsdale3]

U+1FAE5

[1024core]

Phew!