I wonder: how relevant is this portion of the software industry? Because I’m guessing there is also no way they can apply LLms at scale, which is never discussed in the larger AI at work narrative
It is completely relevant, if you want reliable software that you use daily to continue running without a massive rewrite.
Before suggesting to use LLMs to completely rewrite this sort of software, there is a reason why compilers need to be certified to operate in safety critical environments. Not everything needs to use LLMs as the solution to a problem.
I would go as far to say that using an LLM in this context is the wrong solution and is irrelevant to critical systems. Maybe some here see everything as tokens and must solve everything in the form of using LLMs.
Rewriting a toy web app using LLMs from Javascript to Typescript is great, but isn't good for safety critical systems.
Safety critical software is mostly a compliance dance that incidentally produces artifacts with lower defect rates than usual. LLMs can help with safety critical code as long as a human signs their name that they are responsible for its behavior.
When I'm sitting in the plane that has CAS firmware, I'd like to think it wasn't written by an LLM and that my death in the case of a CAS failure isn't chalked up to "some engineer somewhere gets in trouble".
There probably already is generated code in there, only it was generated from UML. I don’t think that LLM generated code will be treated differently from the point of view of the relevant regulations.
That doesn’t matter. Once the code is generated it doesn’t change. The reviewed artifact in a safety critical codebase is the last abstraction layer before a fully certified compilation pipeline. So usually it’s not the UML but the generated code.
I agree with you.
The question is: how the hell is this never discussed when assessing the economic potential of AI-driven disruption.
I ask because I have the impression that all the really relevant industries are resistent to the current narrative.
That said we had Claud helping bomb a school full of kids, you would guess the military would know better but no :/
I think my comment conveyed the wrong sentiment, my bad.
I’m suggesting exactly this: there are extremely common cases in which deterministic software outcomes are needed/mandatory/regulated.
Way more often than we think, often in boring and solved but critical environments.
Yet the entire AI industry acts as if that is an afterthought or an unimportant edge case.
> Yet the entire AI industry acts as if that is an afterthought or an unimportant edge case
Certainly it's not on the "AI industries" list of priorities. Perhaps, however, it's not supposed to be. I use AI tools for the use case I mentioned. The source code, build system, binary artifacts and hashes are still regulated in the way I described. The fact that the AI industry was involved in that chain simply isn't relevant.
Other uses cases involving real time agents and whatnot are another story. I'm not dealing with that problem. I suspect the AI industry doesn't really care about such attestation at this point because everyone is still in the frothy world of "new!" and the bureaucrats simply haven't caught up yet, and the adopters are taking advantage while they can. That pattern has recurred throughout the history of communication and computers.
I don't really object to that. There will be plenty of time for security theater after whatever limits are eventually found and exploited, and in the meantime there is free oxygen available.
Of course they are. LLMs are routinely used to generate Lean proofs, so of course they can also be used to generate verified software. They probably aren't being used that way, yet, but they will be:
Before suggesting to use LLMs to completely rewrite this sort of software, there is a reason why compilers need to be certified to operate in safety critical environments. Not everything needs to use LLMs as the solution to a problem.
I would go as far to say that using an LLM in this context is the wrong solution and is irrelevant to critical systems. Maybe some here see everything as tokens and must solve everything in the form of using LLMs.
Rewriting a toy web app using LLMs from Javascript to Typescript is great, but isn't good for safety critical systems.