MLP is a universal approximator, so there’s definitely a configuration that can match an attention mechanism. Whether or not it’d be feasible to train is another question.
Not sure about feasible, but certainly not efficient.
I think this MLP universal approximator notion is similar to a Turing machine being a universal computation device. Correct, but practically useless.
I don't think Sutton's bitter lesson is going to result in everything being an MLP. You want the most scalable architecture, which an MLP certainly is not.
I think this MLP universal approximator notion is similar to a Turing machine being a universal computation device. Correct, but practically useless.
I don't think Sutton's bitter lesson is going to result in everything being an MLP. You want the most scalable architecture, which an MLP certainly is not.