[davidatbu]

> I haven't played with copilot, but I imagine that something like guessing the graph based on node labels and on some edges is feasible using GNN?

Copilot is based on OpenAI Codex, which is based on GPT-3, which is a transformer model.

Although technically, transformers are mostly GNNs that are "fully connected" (in the graph theory sense), I don't think that supports your speculation here about how GNNs could be used for code analysis since the "tokens" that GPT-3 is trained on are not programming-language syntactic constructs, but sub-word units obtained from natural language (something like WordPiece).

I will say though, I am equally excited by the exact prospect you raised of using something like GNNs for code analysis.

My hunch is that if somebody can figure out a way to make training hierarchical/graph based neural networks very fast, we'll observe the same gains that we did with transformers. But hierarchical/graph based models don't lend themselves to efficient computation.

[legothief]

I'm also quite excited about that - there's existing research, quite a few papers that are using graph-based models for MLOnCode: https://proceedings.neurips.cc/paper/2021/file/c2937f3a1b3a1... https://arxiv.org/abs/2203.05181 https://arxiv.org/abs/2005.02161 https://arxiv.org/abs/2012.07023 https://arxiv.org/abs/2005.10636v2 https://arxiv.org/abs/2106.10918

Definitely check them out! There are also tools that were made available by some of the authors: https://github.com/google-research/python-graphs

[algo_trader]

Are these papers somehow "curated" or "recommended"?!

Unfortunately, GNNs are lagging LLMs in the code domain. Maybe because

a. LLMs and transformers rulezz OR b. there is far more source code than there are compiled code graphs

[davidatbu]

I wouldn't rule out the fact that transformers are very amenable to parallel computation as the reason

[davidatbu]

Thank you!! I've been looking to get my feet wet with SoTA research for MLONCode, so this is very helpful!