|
|
|
|
|
|
by Giho
3 days ago
|
|
|
I wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix:
St =St−1 +βt (vt −St−1 kt )ktT The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss. Interesting what other labs comes up with from this. |
|
|