|
|
|
|
|
|
by alf42red
3 hours ago
|
|
|
I've only read 1-4 without prior knowledge of qkv attention and I like the explanation. The only thing I didn't get from the text was why we divide by the square root of d_k, but ChatGPT explained it was to counter variance scaling with the dimension of the matrix. I probably couldn't have come up with this myself, but I feel that it actually makes sense now and I like that deltanet attention "learns a learning rate" according to one interpretation of beta from the text, if I'm understanding it correctly. Great article! |
|
|