Hacker News new | ask | show | jobs
by jeffreygoesto 136 days ago
All these people that built GCC and evolved the language did not have the end result in their training set. They invented it. They extrapolated from earlier experiences and knowledge, LLMs only ever accidentally stumble into "between unknown manifolds" when the temperature is high enough, they interpolate with noise (in so many senses). The people building GCC together did not only solve a to technical problem. They solved a social one, agreeing on what they wanted to build, for what and why. LLMs are merely copying these decisions.
2 comments
bgirard 136 days ago
That's true and I fully agree. I don't think LLMs' progress in writing a toy C compiler diminishes the achievements that the GCC project did.

But also we've just witnessed LLMs go from being a glorified line auto-complete tool to it writing a C compiler in ~3 years. And I think that's something. And noting how we keep moving the goal post.

link
direwolf20 135 days ago
GP: "it didn't write a C compiler, it copied other compilers. Writing one from scratch is a lot harder."

You: "but look! It wrote a C compiler!"

link
bwfan123 135 days ago
The pattern matching rote-student is acing the class. No surprises here. There is no need to understand the subject from first principles to ace tests. Majority of high-school and college kids know this.
link
yourapostasy 135 days ago
> LLMs are merely copying these decisions.

This I strongly suspect is the crux of the boundaries of their current usefulness. Without accompanying legibility/visibility into the lineage of those decisions, LLM's will be unable to copy the reasoning behind the "why", missing out on a pile of context that I'm guessing is necessary (just like with people) to come up to speed on the decision flow going forward as the mathematical space for the gradient descent to traverse gets both bigger and more complex.

We're already seeing glimmers of this as the frontier labs are reporting that explaining the "why" behind prompts is getting better results in a non-trivial number of cases.

I wonder whether we're barely scratching the surface of just how powerful natural language is.

link