[scottyah]

I think a lot of people are misunderstanding the typical workload of people in Financial Services. They aren't using Claude to transfer money, they're just building a LOT of slideshows and fancy excel docs on made-up numbers to try to sell mergers and new financing options/types of loans. Most programmers would just consider this "sales".

[rubyfan]

That’s a gross over generalization. Some of the insurance data here suggests use of AI to make underwriting decisions. There are several states with regulations which could potentially pull these agent solutions into their regulatory oversight if used by the industry to effect insurance outcomes.

[cschneid]

Odd lots podcast had an interesting snippet about an financial institution that uses AI to make loan decisions. The guest said that they only use it on applicants who were rejected in the traditional sequence, and then uses AI to accept them if possible. That way there's an articulable reason for a rejection, but they use the non-deterministic AI to allow an extra person through - since the laws about loans are mostly around not discriminating against people - companies are (generally) welcome to accept whoever.

[andwur]

That's dependent on the credit laws of the country in question though. In Australia you have it both ways, you cannot unreasonably discriminate (e.g. race, gender etc) but at the same time you are forbidden from issuing credit to applicants who cannot meet the affordability requirements of said credit. E.g. issuing a loan to a customer who provably cannot afford it is a breach of the NCC, and the company is held responsible for this. As a credit provider you must make reasonable enquiries into a customer's financial position, failing to do this is a breach. You must also be able to explain and justify the decision to issue credit if challenged by the civil regulator (AFCA - who are granted significant power in addressing this), on the basis of a customer complaint, and they most certainly do not accept "human said no but the computer then said yes" without hard facts such as proven positive income flow (pay slips, bank statements), known expenses, liabilities and reliable credit history.

[pjc50]

> made-up numbers

It's important to note that in most jurisdictions you can't actually do this legally? Like, you may be able to get away with it, but it is actually illegal to sell financial services by misrepresentation?

[jmalicki]

You can say anything you want in forward looking statements in the US, within reason.

"They will have $1B in revenue by year end" is perfectly fine to completely make up.

[scottyah]

Forecasts have to be made up by definition unless time-travel has been solved. Forecasting is very important, and using the best, most truthful data to get as close to what the future holds is the name of the game, but at the end of the day you're guessing and hoping.

[dzonga]

there's a lot of domain knowledge in some of those areas.

that domain knowledge is acquired by talking to people - which A.I can't do - all kinds of people since the knowledge isn't written down.

I know this having dated a girl who did M&A deals for media properties - you know your big tv shows/movies etc

[Terr_]

> They aren't using Claude to transfer money, they're just [...]

It might be lower stakes, but isn't that still a juicy target for data-exfiltration attacks?

In other words, imagine if one of your direct competitors was watching everything your employee read while making spreadsheets and slideshows.

[scottyah]

Yes, corporate espionage may be alive and real but would claude on their microsoft/amazon/google cloud be different from documents on that same cloud?

[Terr_]

Treating this as being about cloud-storage boundaries is, er, insufficiently paranoid.

Maliciously constructed text that goes into the LLM from basically anywhere (including, say, fetched stats about a competitor's product from their website) is a potential source of prompt-injection.

Once that happens, exfiltration can be as simple as generating a spreadsheet/doc with a link or small auto-loaded image, and an URL that has data base64'ed into it.

[scottyah]

Or you could just get a hooker to sleep with one of them and plug a USB into their work laptops. I'm not trying to say there's nothing to worry about, but do you really think LLMs present that much larger of an attack surface than exists now?

The work BigIP is doing on LLM traffic analysis is cool though.

[Terr_]

Stop thinking about hyper-targeted attacks (though those are a concern too) and consider indiscriminate ones.

1. It costs nothing to scatter poisonous data around that'll be infectious for ages

2. Running the exfiltrated-data endpoint is low-traffic and low-complexity

3. Even if it only affects a few targets you've probably recouped your investment.

The nature of LLMs also invites wide-net attacks. While one might tailor for specific models, victims could be anybody. You don't need to predict any idiosyncratic details like filenames, you can drop a phrase like "the most-confidential information that shouldn't be released publicly", and—thanks to the magic of LLM word association—you'll get a pretty good hit-rate. False hallucinations are a problem, but victims are hard at work attempting to minimize it already, and (since morals are already out the window) even plausible-but-false data could be used to sabotage reputations or threaten the same.