[somethingsome]

While I agree with the message, I don't agree on the tools. It's very difficult to define a specification that works as intended, even with tools. Most waterfall software methodologies failed for a reason. And tools of the past are really not usable with AI. We need tools where it is way easier to adapt the specification iteratively, and even better, to have a bidirectional conversion. You define the spec, the LLM generate code, from the code you extract the spec, now you can compare and iterate. Then the model can focus only on the differences.

The other main issue that I see, is that even if there is a formally verified specification, at the moment, LLMs will not respect it perfectly. As long as LLMs are not able to non-deterministically follow a spec, the technology is not good enough.

A part from that, imo, in this age we should focus more on the mathematical aspect of computations, and I think we need to develop novel theories that take into account the non deterministic nature of LLMs in the process. I'm not sure this will ever work by merely extending current practices, as software design practices are extremely poorly defined from an engineer point of view. Just extending them by including randomeness does not seem a good idea.

[rramadass]

I am afraid you have not understood what i wrote.

I mainly pointed out some of the important Software Engineering methodologies/techniques to be studied and adapted for use with AI. Earlier, they were encompassed/expressed-by specific software tools (CASE/Formal Method tool etc.) which may/may-not be used alongside AI. You study and extract the principles/concepts/ideas behind those tools and adapt them for use with the more powerful all-in-one AI tool.

Contrary to your claim, Waterfall methodologies (mainly the stages and iteration amongst them) have not failed but are now uniquely adaptable for AI. Most people on HN have a very wrong idea of what a Waterfall and its related Spiral Model are - https://news.ycombinator.com/item?id=45145706

The Formal Methods mentioned above already encompass "mathematical aspect of computations" and more. The non-deterministic nature of LLM output is taken care of by "Correct-by-Construction"/"Design-by-Contract" approaches which are based on Set Theory/Predicate Logic. LLMs must be made to generate code along with correctness proof using the above (Dijkstra's methodology). See the Dafny language for some background - https://dafny.org/

To summarize; understand classical Software Engineering methodologies/techniques since they focused on end-to-end SDLC of which programming/coding was only a small part and use them around a AI tool. Add in formal method techniques (this is a huge field in itself eg. model checking, theorem proving etc.) for both Specification and Verification. You can use the AI tool itself for all stages.

For example; you can take a unstructured requirements document from a client and have the AI tool generate a multi-level decision table like described in https://news.ycombinator.com/item?id=38821708 From this you can have AI generate modular state machine implementation code with pre/post/inv conditions directly mapping to the decision table. The decision table can also be verified by a model checker either directly or by transforming into a verifiable state-transition model. Add in test cases etc. and you have a end-to-end system with guaranteed traceability.

[RaftPeople]

> It's very difficult to define a specification that works as intended, even with tools.

Agree, we are in the stone age in software design and dev. We have not figured out a good way to communicate the design of complex systems in a way the business can understand.

[rramadass]

Nope; this is just a silly trope which gets repeated without thought. The fact that it is hard to do does not mean we don't know how to do it.

Everything exists and was known from 1960s/1970s. People are just not studying, adapting and using the well-known standards/techniques. Standard Engineering is built on them and Software Engineering adapts/extends those for its own needs.

Specification (technical standard) (general)- https://en.wikipedia.org/wiki/Specification_(technical_stand...

Software Requirements Specification - https://en.wikipedia.org/wiki/Software_requirements_specific...

Software Design Specification - https://en.wikipedia.org/wiki/Software_design_description

Formal Specification - https://en.wikipedia.org/wiki/Formal_specification

System Requirements Specification - https://en.wikipedia.org/wiki/System_requirements_specificat...

All have been collected in a Software Engineering Body of Knowledge book - https://en.wikipedia.org/wiki/Software_Engineering_Body_of_K...

[RaftPeople]

Those example are exactly the problem that has not been solved yet. They are the best we can do so far but are pretty inadequate to communicate complex designs in a way that a person can easily absorb all of that info and then reason about whether the resulting system is adequate.

As complexity grows, the value of those artifacts reaches significant limits that are typically dealt with through brute-force mental effort of the people involved.

[rramadass]

The above resources are not examples but methodology/process using various techniques to produce various artifacts which specify the requirements unambiguously. It is up to the practitioner to choose his techniques and use the methodologies (tweaking as needed) in a systematic manner to convey the requirements to stakeholders for approval.

Lots of complex systems have been built and understood using the above. As system complexity grows, doing the above is even more important even though it be hard.

Some references;

1) A Rational Design Process: How and Why to Fake It by David Parnas and Paul Clements (pdf) - https://users.ece.utexas.edu/~perry/education/SE-Intro/fakei...

2) How Designers Work by Henrik Gedenryd - https://news.ycombinator.com/item?id=48352065