[mentalgear]

Here's my big fear: Even IF (and that's a BIG if) we get all critical vulnerabilities fixed in tech (before adversarial/state-actors turn up with open attack models) - we still have (in at least a year) models that will be so good in social engineering that they can still (given enough tokens) gain access to whatever system they want.

If society can't trust banks and other institutions to safely control their data, what follows ?

Do we we collectivelly switch off the internet?

[colechristensen]

Social engineering as a problem goes away when anybody can get a model to do it for them for $5. It stops being possible, it's really the bank's problem when they can't have a minimum wage call center or a robot responsible for people's data.

[p-e-w]

Yes. There will be a few high-profile incidents, and then institutions will be forced to stop performing administrative actions based on people’s word.

[applfanboysbgon]

This outcome is massively detrimental to humanity at large. By eliminating the human factor from support, you make it impossible to get support in edge cases that fall outside of the pre-planned bureacratic process. Everyone already hates that Google can arbitrarily ban anybody they please with no way to get in contact with a human, and you want to extend that to banks in control of people's life savings?

[hallway_monitor]

I don't think anyone is saying that. You will just need to be authenticated before giving any commands to the bank. Maybe some type of TOTP that you can use over the phone or in person.

[applfanboysbgon]

That is the exact problem. You have identification tied to your device. Your device is lost or stolen. Now you can't access your bank account. Human support can help you out by finding flexible ways to ascertain your identity. This is the angle social engineers exploit, tricking employees trying to be helpful to abuse that area of flexibility. You can take away human judgment and all flexibility in the system, and that will make the system more secure, but it also results in a deeply uncaring system that makes life harder for people. Rigid bureacracy doesn't do a good job of accounting for a house fire destroying everything you own or your e-mail provider shutting down; these are fringe cases but they do happen and there are positive resolutions available as long as human discretion is involved.

[DANmode]

No.

You don’t tie it to “your device”.

You tie it to your security key.

Which is treated like a credit card.

and your extended family, friends, or volunteers can act as social proof to allow you back into your accounts,

if your key burns up, it breaks and you were too cool to provision a backup, etc.

[repeekad]

> Everyone already hates that Google can arbitrarily ban people

Yet they’re still the predominate search engine, sadly the concerns of the few don’t interest monopolistic profit seekers without forced regulations, think how airlines are legally required to give refunds for delayed flights, there’s a reason it required legislation

[protocolture]

>Here's my big fear: Even IF (and that's a BIG if) we get all critical vulnerabilities fixed in tech (before adversarial/state-actors turn up with open attack models) - we still have (in at least a year) models that will be so good in social engineering that they can still (given enough tokens) gain access to whatever system they want.

I was working at the fruit company when they just hard stopped people from recovering their fruitcloud accounts via phone support due to social engineering.

Social Engineering risk just increases the burden on the consumer/internal support services. The risk is that not everyone has pulled up stumps to protect these services. After a few high profile fuck ups they will. The herd loses 2 beasts and the rest wander away from that water hole.

Its much like how after bitlocker we dont have user access to backup server disks anymore. The lesson was learned and we moved on. Lots of high profile fuckups but we dont get those anymore. CTO's were forced, basically at gunpoint, to adapt or die.

[UltraSane]

If things really get that bad then everything will require FIDO keys or push authorization using a phone app and possibly a initial registration code sent to a physical address. This is how Epic MyChart works.

[lern_too_spel]

The government should be in charge of ID Provider infrastructure and has local offices (postal) that can establish physical identity (and already do for people who need to travel abroad), but the religiously affiliated NWO conspiracy theorists have made this politically infeasible in the US, so we have unsavory private sector providers like World ID stepping in.

[insanitybit]

A lot of social engineering attacks die the second you have domain bound 2FA. Not everything, but a lot.

But the idea that we'll squash all of the critical vulns is simply nonsense, despite the weird Firefox blog posts that indicate otherwise.

[MostlyStable]

We don't need to squash all of them, we need to squash all of them that are practically findable by current and very near term frontier models.

[insanitybit]

Also impossible imo so it's moot.

[MostlyStable]

Because you think that current models can, in a practical sense, find an infinite number of vulnerabilities, or you think that they can find so many that it isn't possible to fix them?

In other words: do you think that the impossibility lines in exhausting the number finds or does the impossibility lie in fixing them?

In either case, do you think that this was also true pre-AI? That is to say: it was not possible to, given some set of practical resource constraints, find and fix all the vulnerabilities that a similarly-resourced group would find?

If so, then would you say that you just fundamentally don't believe in secure software and the only defense is lack of attention?

[insanitybit]

I think that there are, practically, infinite vulnerabilities in common and critical software - browsers, operating systems, etc. So discovering all of them is not tractable, and even if we 100x our rate of discovery it won't matter.

> In either case, do you think that this was also true pre-AI? That is to say: it was not possible to, given some set of practical resource constraints, find and fix all the vulnerabilities that a similarly-resourced group would find?

Yes.

> If so, then would you say that you just fundamentally don't believe in secure software and the only defense is lack of attention?

I believe in security software, few people are building it though and the majority of relevant attack surface is dogshit for security.

Squashing vulns via discovery is irrelevant to security. If we want safer software it has to be built to be safer.