[gradys]

I worked on project very much like this last summer, a transformer language model applied to code completion.

You'd be surprised how easy it is to get a model that performs as well as what you see in the video. And it's even easier now that people have built great libraries for fine-tuning generative language models.

I encourage you to try it yourself! There are many interesting extensions for people to explore:

- Use bi-directional context (vanilla GPT-2 only sees backward context)

- Integrate with semantic analysis tools.

- Experiment with different context representations. You condition the model on an arbitrary sequence of N tokens. It's not necessarily the case that you should spend that whole budget on the N tokens that came immediately before. What about including the imports at the top of the file? What about the docstrings for functions that were just used? What about the filepath of the current file?

Don't look at something like this as though watching your job be automated away. Look at it as a tool that you can master and use to move up the stack.

[arielroth]

Did you explore all of these things? What were your results?