[Jack000]

I'm not an expert, but after playing with some pre-trained transformers I think they are mostly good at the exact thing they're trained for. eg. GPT-2 is great for text generation, but if you try to use it for say translation, it will tend to add imagined details not in the source text. Similarly, BERT is great at sequence and token-level classification but quite bad at text generation.