[Matl]

> I think it is a reasonable term to use in the context of adversarial AI development

Distillation is neither an 'attack', nor 'adversarial' in any reasonable sense.

> Does that mean I am getting paid by the CIA or something

It at the very least means you're uncritically parroting the framing of a company that'd like nothing more than to successfully persuade lawmakers that something should be done about open Chinese models (eliminate choice), preferably so that Anthropic is close to anyone's only option.

[wrsh07]

It seems like a reasonable person could say that a model being distilled "against the model provider's wishes" is in some sense a cyber attack that is stealing information (eg the lower order bits of the model weights)

I think this is mostly a confusing way to describe it, but I'm not really sure why you say it isn't an attack or adversarial. One side is doing something the other side doesn't want. Seems to be pretty clearly adversarial.

[Matl]

> I'm not really sure why you say it isn't an attack or adversarial. One side is doing something the other side doesn't want. Seems to be pretty clearly adversarial.

This is Anthropic we're talking about here. A company that's infamous for adversarial scraping of copyrighted content. I generally don't accept their framing, especially when it's pretty clear what the end goal of that is.

[wrsh07]

This is a bit of a non sequitur.

The most charitable read i can get is:

> Theft presupposes a legitimate possessory claim by the victim. If A’s possession of X is itself wrongful because A stole X from B, then when C takes X from A, C has not violated A’s rightful ownership of X—because A has none.

I think the whole thing is a bit fraught. In the best case, all frontier model companies would have invested in a giant expansion of Wikipedia and thus distillation would be stealing because the base information is already public and available. Obviously that's not what happened.

However, at this point, I suspect the stolen books (and scraped websites) are largely a footnote of training. Something that was essential to create early models, but relatively minor given the work expended since to create new content and RL environments

[Matl]

I think you misunderstood my point. Distillation is not stealing. It's not that

"Theft presupposes a legitimate possessory claim by the victim. If A’s possession of X is itself wrongful because A stole X from B, then when C takes X from A, C has not violated A’s rightful ownership of X—because A has none."

Anthropic has no standing in claiming that using its model output is 'stealing' because that would basically eviscerate their business model where they claim that enterprises can use its model output and still claim the generated code as their IP.

[wrsh07]

Ah I wonder if there's confusion somewhere.

My understanding is that distillation uses specific probing queries to extract model weights in some meaningful sense.

There's also the element of "how one uses model outputs can be in accordance with or against the TOS"

(This is true of any website! How I access and use the info on the website can be in accordance with or against the TOS)

Both of those things: doing something against TOS, using specific queries to illegitimately gain access to model weights - that's why calling it adversarial and stealing feel fair.

I didn't (and don't) understand your point, but in an earlier post you made some claim about it being Anthropic - you can't steal from Anthropic. The above is the only reason I can think one might believe that

[Matl]

> I didn't (and don't) understand your point, but in an earlier post you made some claim about it being Anthropic - you can't steal from Anthropic.

I do believe that Anthropic complaining about their TOS being violated is rich and should be thrown into the dustbin considering that Anthropic violates other people's TOS when it suits their business.

I do also think them complaining about it is aimed purely at influencing lawmakers to ban open Chinese models and eliminate choice so that them and OpenAI are the only game in town.

As far as other models distilling using Claude, that's just consuming the output of a service in a particular way. Anthropic should be very careful about imposing restrictions of how you can use their model output if they want to maintain marketshare. Enterprises are not going to like that.

That leaves the TOS violation of maybe hitting the service too fast, using too many accounts etc. Except again Anthropic did exactly the same thing when scraping for training data.

Do you believe The Pirate Bay complaining about a competing site stealing 'their' torrents should be taken seriously?

That's how I look at this.

For these reasons, I don't think Anthropic has a legitimate argument here.

Also distilling simply means using the answers from one model to specific questions to steer another model to answer a category of questions in a similar manner using its own capabilities. So as far as extracting weights, it's not exactly that, it's maybe slightly biasing the model that's being distilled so that its weights might look somewhat closer to the model used for distillation, but it's far from extracting them exactly/directly.