[kstrauser]

That’s factually untrue. I’m using models to work on frameworks with nearly zero preexisting examples to train on, doing things no one’s ever done with them, and I know this because I ecosystem around these young frameworks.

Models can RTFM (and code) and do novel things, demonstrably so.

[coldtea]

>I’m using models to work on frameworks with nearly zero preexisting examples to train on

Zero preexisting examples of your particular frameworks.

Huge number of examples of similar existing frameworks and code patterns in their training set though.

Still not a novel thing in any meaningful way, not any more than someone who has coded in dozens of established web frameworks, can write against an unfamiliar to them framework homegrown at his new employer.

[thunky]

> Still not a novel thing in any meaningful way

Right. What you're saying is that barely anyone is doing truly novel work. 100% agree.

[coldtea]

Almost no e.g. web or app or enterprise or even game developer is doing any novel work, does that come as a surprise?

And is that fact supposed to be an argument in favor of how LLMs can do novel work and move the state of the art (which is what we're arguing about).

I mean, "LLMs can do novel work because: barely anyone is doing truly novel work" doesn't really compute as an argument.

[thunky]

What I'm saying is that LLMs don't have to do truly novel work in order to be useful. They are useful because the lion's share of all work is a variation on an existing theme (even if the creator may not realize it).

[kstrauser]

I'm not talking about web frameworks. I'm talking about other frontiers with darn near zero preexisting examples, and no code samples to borrow from in any language or in any similar framework (because there is no such thing).

"LLMs can only emit things they've been trained on" is wholly obsolete.

[coldtea]

>I'm not talking about web frameworks. I'm talking about other frontiers

Such as what?

>with darn near zero preexisting examples

Whatever it is, you'd be surprised.

[allthetime]

Yeah. I work with bleeding edge zig. If you just ask Claude to write you a working tcp server with the new Io api, it doesn’t have any idea what it’s doing and the code doesn’t compile. But if you give it some minimal code examples, point it to the recent blog posts about it, and paste in relevant points from std it does incredibly well and produce code that it has not been trained on.

[SV_BubbleTime]

It’s always been about context, then being able to communicate it.

Manager people or managing a hyper-knowledgeable intern (LLM). If you know what you need, actually want what you want (super difficult), and have the ability to provide context to someone else… management has always been easier for you than others.

I find one of the more interesting things about the current “AI debate” is that many programmers are autistic or at least close one side of an empathetic spectrum that they’ve always had trouble communicating what is needed for a task and why. So it’s hard for me to take the opinions going around.

[qweiopqweiop]

It also needs a validation loop. Give it the compiler output and I bet it would fix that code even without examples/a blog post.