[tantalor]

That's not why.

It's because counterexamples are easy compared to proofs which require new mathematics.

GenAI is great at combining existing things in new ways (interpolation). It's terrible at creating new things from scratch (extrapolation).

[jebarker]

This is the wrong way to think about mathematical (or any other kind of) creativity in my opinion. In the extremely high-dimensional space of "ideas" (whatever that means) there are almost certainly profound ideas that are the interpolation of existing knowledge, i.e. the curse of dimensionality. It's not at all clear that you need to extrapolate from existing knowledge to be creative.

[rowanG077]

I really don't think that's true. Of course after the fact a counter example looks easy. Because look at it, it's obviously that it doesn't work. But that disregards the process of finding it. That requires great creativity(or computational resources if the problem is tractable at all).

[vonneumannstan]

>GenAI is great at combining existing things in new ways (interpolation). It's terrible at creating new things from scratch (extrapolation).

I think you believe a fallacy about how human cognition works if you think we actually do something different than interpolation

[tantalor]

Then where do new ideas come from? Do the muses bestow divine creativity upon artists, poets, and musicians?

[vonneumannstan]

>Then where do new ideas come from? Do the muses bestow divine creativity upon artists, poets, and musicians?

Apparently thats what you believe if you think "creative" ideas just float into brains somehow.

[datsci_est_2015]

I think this is somehow related to GenAI’s issue of “confident incorrectness”. I’ve been trying to prompt GenAI with some highly challenging prompts recently, especially terse ones:

> Which Anjunabeats and Anjunadeep compilations have the most breakbeat tracks?

GenAI was able to give me breakbeat tracks that were on the Anjuna labels, but it completely flubbed on whether and which those tracks were on compilations. It was very confidently, mostly incorrect.

I think this is somehow isomorphic to the “interpolation vs extrapolation” issue. GenAI is forced to generate an answer, and there’s no mechanism by which we can interrogate the model for its confidence (at least, that’s my understanding).

And yet, I asked “How confident are you in your answer along different dimensions, ie whether those tracks are breakbeat, whether they are Anjuna tracks, or whether they are on those compilations?”

And its answer was surprisingly satisfying, it only gave 30% confidence that the tracks were on the compilations, which is where it flubbed.

Anyway, personal observations, GenAI is meant to be interacted with, not just a single prompt black box.

[gwerbin]

Out of curiosity which model & interface did you use? I'm starting to think that, for IR tasks like this, the number one differentiator among models and harnesses is the ability for the model to look at its available evidence and conclude that it doesn't know the answer. It depends a lot on how the chain of thought goes. I have seen a lot of newer models try to do this in their thinking traces, and it seems very hit or miss.

Even Opus 4.8, GPT 5.5, and Gemini 3.1 Pro (haven't tried 5.6 yet) confidently make up BS sometimes and needed to be reminded to verify thnings instead of inferring or relying on "memory" from training data.

IMO harnesses need some kind of built in "are you sure about that?" loop checkpoint that pauses the main interaction and asks the LLM to evaluate the strength of evidence for a claim it's about to make. It seems almost unreasonable to expect an LLM to next-token-predict its way to such a conclusion, reliably, without prodding. That or we just need another year of RLHF and another 250 billion parameters, IDK.

I feel like that should work becaus when you're using an LLM for a batch inference or classification type of task, it's not that hard to prompt them to emit an "unknown" flag. They just won't do it on their own, and in a coding harness there is so much else going on that a prompted reminder gets lost.

[datsci_est_2015]

GPT 5.5

> IMO harnesses need some kind of built in "are you sure about that?" loop checkpoint that pauses the main interaction and asks the LLM to evaluate the strength of evidence for a claim it's about to make.

As a practitioner of ML for the past decade, this is the part of LLMs that has always been least satisfying for me. The shared intuition amongst myself and my colleagues has always been that the strongest models are the ones that produce actionable insights directly. Doing all sorts of data preprocessing gymnastics and then following up inference with a 1000-line wall of if-else business logic has always indicated to me that the model is somehow deficient, and not truly capturing the kernel of “truth” within the data, either because the model choice is wrong (e.g. linear regressor for nonlinear problem), or, more likely in practice, the collected data does not effectively capture the underlying “truth” of the measured phenomena with a sufficient signal-to-noise ratio.

LLMs are powerful and have essentially limitless utility, but there’s something missing at a fundamental architectural level for which harnesses are simply a bandaid. Of course, solving that fundamental architectural issue is probably the key to AGI, so…

[gwerbin]

I find this perspective surprising coming from someone who's done ML for a decade.

How much of a good machine learning system in production is really "the model"? How many lines of code are dedicated to constructing features and preprocessing them? How many lines are needed to process the decision emitted by the model in context of whatever else is going on? How often does the customer interact directly with the inference/prediction from a machine learning model?

My experience has been that model-fitting and model inference very often ends up being part of a bigger system anyway. And even if you do end up with some kind of super-duper all-purpose predictor model, it's rarely the thing you start with, instead you work up to it over weeks or months of iteration. So why should we put so much privilege and esteem on the matrix transformatinos or the decision tree, when they are really just part of a bigger system that receives some input from somewhere and produces useful output for someone?

The same holds for AI in my opinion. An AI system is a system that accepts inputs and produces useful outputs. LLMs are borderline miraculous in that an AI system built around an LLM can do so much more than anything that came before, and you can push a lot of amazing things right down into the tensors inside the LLM. But how far can that go? I don't think we even have an answer yet, because it's all so new and every time it looks like it's going to plateau, the big labs come out with another big step forward.

But there are other questions too, which should look familiar to anyone in industry. Do we need to do it that way? Should we do it that way? If our goal is to build a useful AI system that balances power and cost, then maybe we shouldn't. What if we can make models 10x bigger and spend 10x on RLHF and achieve the behavior we're talking about entirely in tensors. OK, so what if we can? We can also run a collection of little models in a carefully designed harness and achieve that behavior right now, today, at a much lower cost both in terms of up-front investment and marginal cost per unit of work completed.

I get it from an elegance perspective, but I guess I have completely the opposite mindset. Industry has bled me dry of any desire for elegance. The stuff I build now is stupid and crude but it makes our customers happy and costs peanuts to run and that's exactly what my job wants from me. So I say let's commoditize LLMs and start being clever about how we use them. It's not that we shouldn't also push the frontier in size and power, but unless things change w.r.t. the actual costs involved, it doesn't really seem like the best way forward in the short term for achieving smarter AI systems without also making costs go up and up forever.

The bitter lesson still holds of course. But eventually you have to pay for all that power. And you might not want to shell out $20/1M just so you can be sure your LLM can detect uncertainty in its own context and be more honest about when it doesn't know anything about some obscure record label.

[datsci_est_2015]

It’s a fair perspective that you’re bringing, but what comes to mind for me in response is that I have a mental model of two types of machine learning inferences that provide value:

Type B: “directionally correct” ML inferences that provide value by virtue of outputting an inference regardless of its accuracy. The value comes from providing automated decision that allows someone or something to move forward in a process, similar to management consultants. This is used heavily in ad tech and (social) media algorithms. I would even put gambling and investing inferences in this bucket as the accuracy of algorithms isn’t really required to be that high to provide alpha.

Type A: ML inferences that are creating or replacing systems where high accuracy is table stakes. This is applications in industrial engineering, chemical engineering, biotechnology, etc. If you can’t beat any existing automated or non-automated systems in terms of accuracy, your product is dead in the water. You can make it a few years by faking it but eventually the pied piper makes his call, and a major contract falls through. This has always been the lifecycle of products I’ve been on. It’s an incredibly difficult problem. “Faking it” here means piling on heuristics upon heuristics to convince someone to sign a contract, even if the underlying precision-recall tradeoff curve isn’t financially viable. This is the basis for the conversations between myself and my colleagues that a model that can’t provide actionable insights without a wall of heuristics / post-processing is a red flag for a product.

Anyway, that generalizes to LLMs because LLMs can provide so much value in the Type B category, but anyone promising Type A products based on LLM technology is treading on thin ice, imo.

Edit: also, as an aside, feature extraction and post-processing of inferences is necessary for any model, but if that code becomes bloated is when I lose confidence in the model’s ability to provide inference value.

There’s a classic stack overflow from a decade ago that I remember where a ML engineer / DS was asking if a linear regression model trained on sorted data (X, Y sorted independently) had any inference value. They asked because their manager insisted that was the correct way to do it because it minimized training error. Well, duh. Two sorted lists of values are maximally correlated.

This is an extreme example of what I mean by “preprocessing data”.

An extreme example of “postprocessing results” would be to create a decision tree based on the output of a decision tree algorithm that re-sorts values into something that is palatable for the customer, based on feedback from the customer on a case-by-case basis. “Hey this was a false positive…” -> add an “if” statement to the postprocessing.