[jalev]

Unfortunately every software related industry is embracing LLM/Codegen. Your banks, fintechs, insurance. Everyone. Your concerns are the same I'm having, yet it's regularly dismissed or hand-waved away as "don't worry about it the delivery velocity/ROI is worth it"

[simon84]

It's not so much about velocity or quality, both of which LLM do (or will) provide.

The real question is about accountability and liability.

When a major data leak is going to happen, who will they sue or fire ? That is the value engineers provide. They understand, confirm, and take ownership.

[jalev]

This is what I'm wondering too. We've signed a confidentiality agreement with all the big players (as I'm sure all other companies have done), which is supposed to ensure our data is both segregated and not used for training. I don't trust these companies not to do just that; their business is in taking what we have and training their models.

[iterateoften]

Yeah, I always wonder if they do some type of obfuscation and transformation on the private data and find a way to backdoor the info without technically using it directly.

[afro88]

I wonder if there's a way to include data that's so unique you can prove it was trained on and sue later

[Chu4eeno]

Unique data like that is unlikely to have any impact on the learned/final weights after training. SGD, Adam and the other hillclimbing solvers abhor jagged edges from "novel" trade secrets and the like. Unless it turns out everyone had the same secret genius idea (and it became a pattern to learn).

[lanfeust6]

Ostensibly, due-diligence should not change. But people are lazy, just as they've always been around testing/QA/definition-of-done.

I'm not even certain that laziness gets them further along than it used to; I think it's that people have not had their overconfidence painfully corrected yet. Behaviors will re-align pretty fast when people realize that no, they're not going to get away with just pressing a button and saying everything is "good". That is happening right now.

[mexicocitinluez]

Just having this discussion with someone about AI in healthcare and how issues are going to be handled.

If a nurse does something incorrectly, they can lose their license. Ensuring that nurse will never be a nurse again. There is a very clear path of accountability and very clear ways to mitigate it.

For instance, if a nurse is drunk and you recognize there is a pattern of people showing up drunk, you institute drug tests and breathalyzers and move on.

While we probably won't have LLM's autonomously performing procedures, they are 100% parsing documentation, reading lab results, making suggestions, etc. And right now, the burden has been placed squarely on the clinicians themselves. It'll feed them them the data, ask if they approve/agree, and then essentially wash their hands of accountability. Let's say an LLM starts incorrectly reading lab results, how is that fixed/remedied? A prompt update? Additional safeguards? Adjusting the temperature? Changing a model?

This is a far different type of engineering that still feels pretty new. Granted, I'm still an amateur in this space (I use Claude Code a decent bit), but it feels really opaque to me right.

[verandaguy]

This question has been easily answered by many companies.

You, the IC, the developer prompting the code extruder, are ultimately responsible for its outputted code and its behaviour.

You may feel pressured to push out thousands of lines of code a day. You may see those thousands of lines refactored several times over the lifespan of a merge request. You may be asked to do this continue this in the long term with all the mental fatigue that entails.

When it's too much for you to sustainably deal with and you turn to using LLMs to review the code, that will still, presumably, fall on you at the end of the day.

The output is your responsibility.

[rvz]

> When a major data leak is going to happen, who will they sue or fire ? That is the value engineers provide. They understand, confirm, and take ownership.

This goes for serious incidents, disasters, outages and security breaches.

If there was an investigation and the answer was "a piece of software was vibe coded with AI" why would anyone trust the software vendor after that?

[marcosdumay]

When has any company ever faced consequences from atrociously bad code leaking data or negatively impacting their customers?

Even Solarwinds is still alive.

[simon84]

EU companies are judged guilty of negligence because backups were not totally disconnected (even though distant site) and ransomware did destroy them.

So that is starting to dig deeper than a plain mistake. I guess we will soon-ish witness the first AI slop trial going on, this will be interesting to follow

[mont_tag]

Knight Capital

[genxy]

Don't worry, we can throw in all in 55 gallon drums and dump it over a cliff when the time comes.

[Hamuko]

Are banks that concerned about velocity? Because moving fast and breaking things in the banking sector can get extremely expensive. It's also not a who-gives-a-shit industry like operating a taxi service or hosting images, but a very tightly regulated sector.

[jalev]

I might have been a bit broad with the brush. I can't speak for banks, but I can speak for the the fintech/money-movement space (e.g. Remitly, Wise, Revolut).

It's a race to get first-to-market for backend integrations/features. It's given rise to a culture of "move fast break things" where safety is only for some core features, but absolutely not for the constellation of other services we provide. Failure rates have increased almost a percentage point since Codegen/LLM adoption was mandated from up top.

You would think regulators would be on top of this, but our industry runs on all actors "self reporting" their outages. Most don't unless they can't hide it (>1h)

[mrkeen]

'Keeping up with regulations' may as well be a separate field from the core stuff. It has the same pressures as any other development effort. Managers will want the integration to the KYC service LLM'd as quickly as possible.

[bigthymer]

> Are banks that concerned about velocity?

Yes

[hn_throwaway_99]

Not in the universe where I live. Having worked in a variety of web tech, and then working at a fintech with a partner bank, traditional banks move incredibly slow compared to nearly every other tech company out there, and for good reason.

[sensanaty]

I work in a large, established fintech and we're in no rush to shove LLMs everywhere. We have access to all the AI tooling we'd ever need with basically no limits, but AI or not, you're responsible for what you ship, so most people are less YOLO about the whole thing. From the quarterly surveys, most people in the co view it as a slight productivity boost but nothing dramatic, and most features we're releasing are still human reviewed anyways.