[mattrighetti]

Is distillation something we have to live with or are there ways to prevent it?

[sosodev]

Realistically you can't prevent distillation. OpenAI / Anthropic are slowly moving towards hiding the steps in-between input and output (hidden thinking), but that only helps so much. Imagine you put a file into Claude and say "do X to this" and it returns it to you without showing any of its internal reasoning. That's harder to distill, but the simple mapping of input to output still creates very valuable training data. It is reflective of all the training the model did to learn how to do that transformation.

[make3]

You can also get it to think in the output tokens pretty easily, eg "Here's a math problem, I want your reasoning first, then the answer" which is what I assume they're doing.

[epolanski]

If it was genuinely useful, we would've long reached the point where you train a model on a previous one's output in an ever improving loop.

But this doesn't actually work.

[Cytobit]

You make it sound like a bad thing.

[HarHarVeryFunny]

You can prevent it by outputting a reasoning "summary" instead of the actual reasoning trace.

Which Anthropic already do.

[sowbug]

If you build a device that can help build devices, you shouldn't be too surprised when people use it to build devices.

[recursive]

To the extent that we have to live with AI and LLM, I believe we have to live with distilling also.