[nerder92]

While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh.

The user needs to do 3 things for this to be actually be phished:

1. Receive money from somebody they don’t known with a weird description 2. Proactively ask the agent for such transaction 3. Click the link the agent provide

While this of course can happen on scale, doesn’t seems so critical in practice

[tvissers]

Thanks for chiming in.

I agree this is not a one-click account takeover.

But I think point 2 is broader than that. The user does not need to ask about the malicious transaction specifically. Any normal question that makes the agent fetch recent transactions could bring the attacker-controlled text into the LLM context.

[csomar]

People already click suspicious emails that ask them to login. At a high number of attempts, some chickens will be caught. However, people are now weary of emails since there is a lot of phishing there. On the other hand, the AI assistant env. could be considered "safe" by users because it's stuff coming from the bank. So they are more likely to fall for it. (honestly, unless you are a dev and aware of prompt injection, I don't see why the users wouldn't fall for it).

[treis]

Unless I missed it they didn't provide any proof of this actually working. Really seems like a thing veiled advert for their product

[ianburrell]

This is similar to scam where people are sent messages about bad transaction with a fake link to the bank to verify it. Some attackers have gotten Paypal to send notifications that have the link. People are supposed to check the source and go directly to bank, and this will bypass that.

[addandsubtract]

Depending on how much access the AI agent has, there are worse things to inject it with than a link.

[datsci_est_2015]

I think the critical part is that it launders an arbitrary URL as trustworthy. The alternative is “Don’t trust anything our bot says at face value, please.”

I think a better criticism is allowing arbitrary text (including URLs) in a transaction description.

[hocuspocus]

SEPA transfer fields need to follow a standard. I think it's fine, we shouldn't put more control and censorship there (try to put Daesh membership fee if you want to get your account locked...)

However a chatbot should absolutely not be able to display arbitrary and clickable links outside a pretty tight whitelist (like, the bank FAQ).