[lawlessone]

You have me curious now. The models generate text. Could a model hypothetically be trained in such a way that could create a buffer overflow when given certain prompts? I am guessing the way inference works in such a way that cant happen

[wolftickets]

Absolutely, though that isn't strictly what we're talking about here.

In this case, models themselves are fundamentally files. These files can have malicious code embedded into them that is executed when the model is loaded for further training or inference. When executed it isn't obvious to the user at all. It's a very nasty potential vector.

I wrote a blog about it here: https://protectai.com/blog/announcing-modelscan