It was originally built as a general purpose sequence-to-sequence (seq2seq) model.
The research history leading up to this was interesting - there had been a bunch of work, in various domains, on "autoencoder" architectures used to learn compact representations for things like dimensionality reduction and sequence representation. The idea was to have an encoder-decoder pair, connected by a limited bottleneck representation, with the training goal of the decoder reconstructing the encoder input from the bottleneck representation.
One example of this was to learn a fixed size(!) sequence (e.g. sentence) representation using an LSTM-based autoencoder (LSTM->embedding->LSTM), which at the time seemed rather shocking - the ability to represent a variable length sequence with a fixed size embedding. Equally shocking was that you could use this for machine translation simply by connecting an LSTM encoder for one language to an LSTM decoder for another language.
This type of LSTM->LSTM seq2seq encode-decode architecture for machine translation was then improved by Bahdanau by replacing the fixed size representation with an attention mechanism so the decoder could learn to be more specific about input-output relationships.
This type of LSTM-based seq2seq encode-decode architecture, using attention, is what Uszkoreit et al set out to improve - to make more efficient by using a parallel vs sequential (RNN) architecture. The Transformer was never conceived of as purely for language modelling, or as an "AI" architecture. Later when the usage focused on language modelling (generation, not translation), the encoder was dropped since input and output are the same thing.
If you read the Wired article linked elsewhere on this thread, then it explains that. The work was being done by people from the Google Translate team.
The research history leading up to this was interesting - there had been a bunch of work, in various domains, on "autoencoder" architectures used to learn compact representations for things like dimensionality reduction and sequence representation. The idea was to have an encoder-decoder pair, connected by a limited bottleneck representation, with the training goal of the decoder reconstructing the encoder input from the bottleneck representation.
One example of this was to learn a fixed size(!) sequence (e.g. sentence) representation using an LSTM-based autoencoder (LSTM->embedding->LSTM), which at the time seemed rather shocking - the ability to represent a variable length sequence with a fixed size embedding. Equally shocking was that you could use this for machine translation simply by connecting an LSTM encoder for one language to an LSTM decoder for another language.
This type of LSTM->LSTM seq2seq encode-decode architecture for machine translation was then improved by Bahdanau by replacing the fixed size representation with an attention mechanism so the decoder could learn to be more specific about input-output relationships.
This type of LSTM-based seq2seq encode-decode architecture, using attention, is what Uszkoreit et al set out to improve - to make more efficient by using a parallel vs sequential (RNN) architecture. The Transformer was never conceived of as purely for language modelling, or as an "AI" architecture. Later when the usage focused on language modelling (generation, not translation), the encoder was dropped since input and output are the same thing.