[zawerf]

> I'm wowed by the fact that the model has learnt to negate words in if-else statements

I know it learned natural languages from using GPT-2, but I am surprised it didn't get "confused" since words are used in such a different way in programming.

For example strong appears as the html tag <strong> with no corresponding <weak> tag. And weak appears in weak_ptr in C++ and there's no such thing as a strong_ptr.

[Felz]

Vanilla GPT2 actually does a half decent job of emulating the style of programming.

Try entering "public static int main() {" into https://talktotransformer.com/

[0b01]

I think the tokens are simply one hot encoded. Though some sort of word2vec embedding is also situationally plausible