[Avi-D-coder]

GPT2 is known to be unable to track and bind variables, scaling purely associative models beyond the trivial examples is going to be difficult or more likely impossible.

This will end up being a better tabnine. Models like GPT2 are still just approximating intelligence, they are not rationally cognizing.

[gradys]

Curious what analysis you're referencing here.

While I don't doubt people have shown that various transformer models have certain limitations, I'm pretty bullish on transformer models in general.

Here's a post exploring the application of transformers to symbolic mathematics for instance: https://medium.com/analytics-vidhya/solving-differential-equ...