Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that.
From their abstract:
``One of the most exciting and promising novel architectures, the Transformer neural
network, was developed without the brain in mind. In this work, we show that
transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably
place and grid cells. Furthermore, we show that this result is no surprise since
it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the
neuroscience version.``
Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.
From their abstract:
``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version.``
[1] https://arxiv.org/abs/2112.04035