And recently, a paper titled Attention Is Not All You Need has made the rounds arguing that some of the claims made in the AIAYN paper may have been overstated. https://arxiv.org/abs/2103.03404
If you read the title, it only refers to the multi-head-attention part of BERT, excluding the feed forward and skip connections, hence calling it "Pure Attention".
> Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.
> Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
This does not prove the original title was wrong, and this paper is not a counter, but an analysis of a submodule which helps better understanding transformers.