[Transfinity]

> But there's no way to train the transformer-based high-probability-next-word AIs to be superhumanly good at fooling you into doing something, on the grounds of lack of training data

The conversations of all those human scammers would be prefect training data for this. You even know exactly what conversations led to payouts. Assuming you can get all your data in one place, of course.

[jerf]

My context is someone who isn't already falling for the scams. It is true that you can train a model to follow through to those who fall for the scams the scammers already know, which is a fair point. My point is that you're not going to get a superhuman AI out of our current transformer technology that can talk you into believing you're a superintelligent camel from Arcturus IV and if you don't immediately turn over your credit card number, the Star Alligator of the Galactic Core is going to eat your homeworld.

GPT-3 may even gamely try to do exactly that with the correct prompt! But it'll fail. The result won't be cognitively dangerous to anyone with a grip on reality, it'll be risible.

[lupire]

That's not how how transformers work. They just babble loosely related content. They have no logical model.