[brokencode]

You’ve seen actual model poisoning? Or have you seen a model return the wrong answer due to what it saw in a search result? Or were they hallucinations perhaps? How do you know it’s due to poisoned training data?

And do you even realize how much data 0.001% of the training data for a frontier models is? They’re trained on 10s of trillions of tokens, meaning you’d need hundreds of millions of tokens of poisoned data.

Some of these problems you mention could become real barriers to models improvements, though there are plenty of countermeasures, such as by focusing on high quality data sources like I mentioned before.

We’ve already probably gotten as much as we’re ever going to get from simply scraping more and more unstructured text from the web as a way to improve model performance.

The type of training being done now is around tool use and solving specific types of problems better, which is the type of training data you simply don’t find lying around on the web.

[lazide]

No shit Sherlock, of course I’ve seen it. It was my job.

This is exhausting.

It’s like arguing crypto with someone who has never actually committed a line of code. Why do I even bother?

[brokencode]

You’re expecting me to know your job? Give me a break.

I’m wondering the same thing. You keep talking of some grand poisoning problem but can’t point to any specific public information except an article saying that it’s possible. As if that was ever in doubt.

Guess we’ll just have to agree to disagree.

[lazide]

angry and ignorant! impressive