[dmd]

It's wild to me the disconnect between people who actually use these tools every day and people who don't.

I have done exactly the above with great success. I work with a weird proprietary esolang sometimes that I like, and the only documentation - or code - that exists for it is on my computer. I load that documentation in, and it works just fine and writes pretty decent code in my esolang.

"But that can't possibly work [based on my misunderstanding of how LLMs work]!" you say.

Well, it does, so clearly you misunderstand how they work.

[ModernMech]

The reason it works so well is that everyone’s “personal unique language” really isn’t all that different from what’s been proposed before, and any semantic differences are probably not novel. If you make your language C + transactional memory, the LLM probably has enough information about both to reason about your code without having to be trained on a billion lines.

Probably if you’re trying to be esoteric and arcane then yeah, you might have trouble, but that’s not normally how languages evolve.

[dmd]

No, mine's a esoteric declarative data description/transform language. It's pretty damn weird.

[wizzwizz4]

You may underestimate the weirdness of existing declarative data transformation languages. On a scale of 1 to 10, XSLT is about a 2 or 3.

[dmd]

Mine's a weird, bad copy of Ab Initio's DML. https://www.google.com/search?q=ab+initio+dml+language

[ModernMech]

When you say "weird" you mean "different from mainstream languages", but the exact way in which your language is weird (declarative data description/transformation) is probably exactly where languages will be going in the future because of how well-suited they are for LLM reading and writing. Those languages expose the structure of the computation directly such as data shapes and the relationships that transform them, rather than burying intent inside control flow.

With more explicit types and dataflow information, the model doesn't need to simulate execution (something LLMs are particularly bad at) as much as recognize and extend a transformation graph (something LLMs are particularly good at). So it's probably just that your particularly weird language is particularly well-adapted to LLM technology.

[imiric]

My comment is based precisely on using these tools frequently, if not daily, so what's wild is you assuming I don't.

The impact that lack of training data has on the quality of the results is easily observable. Try getting them to maintain a Python codebase vs. e.g. an Elixir one. Not just generate short snippets of code, but actually assist in maintaining it. You'll constantly run into basic issues like invalid syntax, missing references, use of nonexistent APIs, etc., not to mention more functional problems like dead, useless, or unnecessarily complicated code. I run into these things with mainstream languages (Go, Python, Clojure), so I don't see how an esolang could possibly fair any better.

But then again, the definitions of "just fine" and "decent" are subjective, and these tools are inherently unreliable, which is where I suspect the large disconnect in our experiences comes from.