[jerf]

My context is someone who isn't already falling for the scams. It is true that you can train a model to follow through to those who fall for the scams the scammers already know, which is a fair point. My point is that you're not going to get a superhuman AI out of our current transformer technology that can talk you into believing you're a superintelligent camel from Arcturus IV and if you don't immediately turn over your credit card number, the Star Alligator of the Galactic Core is going to eat your homeworld.

GPT-3 may even gamely try to do exactly that with the correct prompt! But it'll fail. The result won't be cognitively dangerous to anyone with a grip on reality, it'll be risible.