[encoderer]

I can imagine a future where new software is, mainly, written by humans while legacy software is maintained by ai.

It can take months for a new developer to understand a legacy code base but an LLM with a big enough context window would be instantly productive.

[nradov]

The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case. No LLM can do that.

Once you have the detailed specifications the coding is relatively easy. Some of that can be partially automated using AI CASE tools, but that only gives a marginal improvement to overall project productivity.

[dragonwriter]

> The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case. No LLM can do that.

One of my first one afternoon (for prompt refinement and trying it out) toy GPT-3.5 “applications”—really just a prompt—was having it interview for reauirements and draft and progressively refine a requirements doc in a specified (by reference to an author in popular Agile literature, not by specific templates, so just relying on the model’s general training) format. Its pretty much what convinced me of the broad utility of the technology.

I absolutely think that with a bit of effort, GPL-4-128K, a custom RAG framework and/or the Assistants API, you could build something that handles a lot of this. I’m not really bullish near term on LLMs as full replacements, but I can see it as a big force multiplier.

[DougBTX]

> The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case.

So lots of tedious repetitive manipulation of language to extract facts and transform them into a systematic format? My prediction: LLMs are going to eat that for breakfast.

(Honestly it is a great business idea for someone with the right contacts. Lots of big customers with deep pockets and large private datasets to train against that might now have a solution to a problem which was previously intractable in scale.)

[nradov]

None of those customers have actual data sets to train against when it comes to ERP and accounting software requirements analysis. The data that exists at all is scattered across random Word documents, wiki pages, paper notebooks, and legacy requirements management systems. Most of it is summarized. It's not recordings of past interviews with SMEs that business analysts and product owners conducted to extract functional requirements. In order to train an LLM to conduct such interviews you would have to obtain actual transcripts of such interviews. Not impossible, but unlikely to happen anytime soon.

[charlie0]

Ding, ding, ding, we have a winner. This. I see issues at my small startup with not having clarity with certain billing items. This issue is not technical, it's the unknown requirements of why things were written in the way they current exist. I can't imagine what kind of bizatine rules they have in the Pentagon that were placed there decades ago, and no one can answer why. Codebase must be littered with Chesterton fences.

[Yahivin]

I can imagine the opposite.

[encoderer]

Cool tell me more.

[SteveNuts]

Not OP but I could see some shops pushing AI generated code to production, then when changes need to be made, they can't get the AI to modify the existing code in just the way they need, so a human has to intervene.

[bastawhiz]

I can't get Copilot to generate Python that adds numbers together correctly sometimes. Getting an LLM to generate correct, working code for a language that hardly anybody writes anymore is almost assuredly going to lead to failure.

[encoderer]

yeah I agree but when you look at the slope not the y-intercept it’s getting obviously better.

one advantage the government would have is training/fine-tuning on a hundred million lines of domain specific cobol.

[bastawhiz]

The slope doesn't really matter, because the target is "better than a human, and able to identify and fix its own errors". The slope will decrease as you approach this threshold.

It's also wildly bad to plan to train and fine tune on code that you know has bugs. Already we have Copilot generating code with trivial vulnerabilities because that's what it's trained on.

[encoderer]

Roughly ~all code has bugs. We were all trained on trivial examples and buggy code - just like llms.

Honestly I think editing code will be easier than creating wholly new applications to precise spec.