[ImAnAmateur]

>Through high-level contacts I had at Microsoft I was asked to file a vulnerability report – the drop-down list of affected products on Microsoft's web page didn't even include Copilot. I selected "Other," reported the flaw, and a day later heard back from their security team:

>>"We've looked over your report, and what you're reporting appears to be a bug/product suggestion, but does not meet the definition of a security vulnerability."

>That left me wondering whether Microsoft's security team knows enough about LLM internals and prompt attacks to be able to grade a potential security vulnerability. Perhaps – but I got no sense from this response that this was the case.

I'm sympathetic with the author. He doesn't trust the boilerplate response he got. Calling it a "product suggestion" doesn't inspire confidence even if I understand why they'd call it that.

I think that the author doesn't understand what happened, but it must look concerning to them. Maybe it started outputting code as babble because more LLMs have been trained with that? That could be concerning without coding experience, especially if the words that the author can understand sound important or relevant to the LLM itself.

[mewpmewp2]

It's impossible to evaluate the author though since we didn't get any information. For all I know the high level contacts also didn't have time to concentrate and just told to "report it".

But there are strong and confident claims from the author "What do I do with this powerful and potentially dangerous prompt".

So is the output or result of this prompt anything more than what you would get from high temperature settings?

It's hard for me to think of a security vulnerability here.

Also the example links about prompt attacks etc are about getting past an LLMs censorship. I don't think these are security flaws. The reason why LLMs like these have censorship is just basic PR. I don't think it's a huge issue if it's possible to bypass and ask steps to do something illegal. If anyone seriously wants to do that, they will find a way outside the ChatBot anyway.

Anyone willing to go to Darkweb to get this prompt attack and then use the prompt attack in the actual LLM itself where it might get logged would only put them at risk of getting caught compared to if they just read the instructions on how to do illegal things on Darkweb.