[KronisLV]

As lines of code become executable line noise, I swear that we need better approaches to developing software - either enforce better test coverage across the board, develop and use languages where it’s exceedingly hard to end up with improper states, or sandbox the frick out of runtimes and permissions.

Just as an example, I should easily be able to give each program an allowlist of network endpoints they’re allowed to use for inbound and outgoing traffic and sandbox them to specific directories and control resource access EASILY. Docker at least gets some of those right, but most desktop OSes feel like the Wild West even when compared to the permissions model of iOS.

[mrothroc]

I'm old enough to remember that engineers researching distributed systems had the same challenge. Everyone was trying to build 100% reliable nodes, which is impossible. Then Lamport came along and showed you could actually achieve your goal at the protocol/system level. What you're describing here is a workflow or pipeline, which is the analogy. As the LLMs produce artifacts, you have gates that verify the output deterministically. If the LLM breaks a rule, you either throw it out and reroll or you give it the feedback and let it revise. I do this in my own tooling and I get great results. One thing from the data: they are often pretty crap at revising, spending ridiculous time/tokens in a revision loop. I'm trying to find the right balance of reroll/revise myself.