[nullc]

It's also hard to know what the LLM has reasoned out vs has memorized.

I like the very last example in my tongue-in-cheek article, https://nt4tn.net/articles/aixy.html

Certainly the LLM didn't derive Fermat's theorem on sums of two squares under the hood (and, of course, very obviously didn't prove it correct-- as the code is technically incorrect for 2), but I'm somewhat doubtful that there was any function exactly like the template in codex's training set either (at least I couldn't quickly find any published code that did that). The line between creating something and applying a memorized fact in a different context is not always super clear.