[fizza_pizza]

The certification angle is the most interesting part to me. Regulated industries (aviation, medical devices) often can't use JIT for exactly this reason, the code that runs has to be the code that was certified. Static translation that produces a signable binary is a real unlock there, code bloat notwithstanding.

[camillomiller]

I wonder: how relevant is this portion of the software industry? Because I’m guessing there is also no way they can apply LLms at scale, which is never discussed in the larger AI at work narrative

[rvz]

It is completely relevant, if you want reliable software that you use daily to continue running without a massive rewrite.

Before suggesting to use LLMs to completely rewrite this sort of software, there is a reason why compilers need to be certified to operate in safety critical environments. Not everything needs to use LLMs as the solution to a problem.

I would go as far to say that using an LLM in this context is the wrong solution and is irrelevant to critical systems. Maybe some here see everything as tokens and must solve everything in the form of using LLMs.

Rewriting a toy web app using LLMs from Javascript to Typescript is great, but isn't good for safety critical systems.

[adrianN]

Safety critical software is mostly a compliance dance that incidentally produces artifacts with lower defect rates than usual. LLMs can help with safety critical code as long as a human signs their name that they are responsible for its behavior.

[junon]

When I'm sitting in the plane that has CAS firmware, I'd like to think it wasn't written by an LLM and that my death in the case of a CAS failure isn't chalked up to "some engineer somewhere gets in trouble".

[adrianN]

There probably already is generated code in there, only it was generated from UML. I don’t think that LLM generated code will be treated differently from the point of view of the relevant regulations.

[junon]

UML conversion is deterministic.

[camillomiller]

I agree with you. The question is: how the hell is this never discussed when assessing the economic potential of AI-driven disruption. I ask because I have the impression that all the really relevant industries are resistent to the current narrative. That said we had Claud helping bomb a school full of kids, you would guess the military would know better but no :/

[topspin]

I work in an industry that requires reproducible binaries from source, and cryptographic hashes filed with a regulator.

It's also not aviation or medical. So perhaps it's more common than you imagine.

[camillomiller]

I think my comment conveyed the wrong sentiment, my bad. I’m suggesting exactly this: there are extremely common cases in which deterministic software outcomes are needed/mandatory/regulated. Way more often than we think, often in boring and solved but critical environments. Yet the entire AI industry acts as if that is an afterthought or an unimportant edge case.

[topspin]

> Yet the entire AI industry acts as if that is an afterthought or an unimportant edge case

Certainly it's not on the "AI industries" list of priorities. Perhaps, however, it's not supposed to be. I use AI tools for the use case I mentioned. The source code, build system, binary artifacts and hashes are still regulated in the way I described. The fact that the AI industry was involved in that chain simply isn't relevant.

Other uses cases involving real time agents and whatnot are another story. I'm not dealing with that problem. I suspect the AI industry doesn't really care about such attestation at this point because everyone is still in the frothy world of "new!" and the bureaucrats simply haven't caught up yet, and the adopters are taking advantage while they can. That pattern has recurred throughout the history of communication and computers.

I don't really object to that. There will be plenty of time for security theater after whatever limits are eventually found and exploited, and in the meantime there is free oxygen available.

[jy14898]

LLMs aren't relevant to aviation and medical devices

[naasking]

Of course they are. LLMs are routinely used to generate Lean proofs, so of course they can also be used to generate verified software. They probably aren't being used that way, yet, but they will be:

* A Case Study on the Effectiveness of LLMs in Verification with Proof Assistants, https://arxiv.org/abs/2508.18587v1

* CoqPilot, a plugin for LLM-based generation of proofs, https://dl.acm.org/doi/10.1145/3691620.3695357

[camillomiller]

Exactly! And yet they’re touted as a catch all business case!!!