[1dom]

I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do.

I'd say your mandelbrot debug and the LLVM patch are both "trivial" in the same sense: they're discrete, well defined, clear-success-criteria-tasks that could be assigned to any mid/senior software engineer in a relevant domain and they could chip through it in a few weeks.

Don't get me wrong, that's an insane power and capability of LLMs, I agree. But ultimately it's just doing a day job that millions of people can do sleep deprived and hungover.

Non-trivial examples are things that would take a team of different specialist skillsets months to create. One obvious potential reason why there's few non-trivial AI examples is because non-trivial AI examples require non-trivial amount of time to be able to generate and verify.

A non-trivial example isn't an example you can look at the output and say "yup, AI's done well here". It requires someone spends time going into what's been produced, assessing it, essentially redesigning it as a human to figure out all the complexity of a modern non-trivial system to confirm the AI actually did all that stuff correctly.

An in depth audit of a complex software system can take months or even years and is a thorough and tedious task for a human, and the Venn diagrams of humans who are thinking "I want to spend more time doing thorough, tedious code tasks" and "I want to mess around with AI coding" is 2 separate circles.

[edmundsauto]

Current state AI is a best fit for jobs that can be easily verified as correct. In my 20+ years, this is at least 75% of the work I’ve ever done. Maybe 99.999% (I have led a very boring career.)

There’s an enormous amount of value in doing this. For the harder problems you mentioned - most IC SWE are also incapable or unwilling to do the work. So maybe the current state has equivalent capabilities to 95% of coders out there? But it works faster, cheaper, and doesn’t object to tedious work like documentation. It doesn’t require labor law compliance, hiring, onboarding/offboarding, or cause interpersonal conflict.

[sokoloff]

> ultimately it's just doing a day job that millions of people can do sleep deprived and hungover.

Doing for < $10 and under an hour what could be done in a few weeks by $10K+ worth of senior staff time is pretty valuable.

[1dom]

If it's something a single senior staff member can do, then - personally - I'd consider it not complex, it's relatively trivial: it can be done by literally a single person.

I'm pro AI, I'm not saying it's not valuable for trivial things. But that's a distinct discussion to the trivial nature of many LLM examples/demos in relation to genuinely complex computer systems.

[simonw]

Maybe the definition of "non-trivial" in these conversations should be defined as "stuff an LLM system can't do yet".

[simonw]

> Non-trivial examples are things that would take a team of different specialist skillsets months to create.

Thank you for providing a spelled out definition of "non-trivial" there!

[1dom]

Haha, it was made up on the spot, thank you though! I think your articles and notes are proof that there's a lot of value and use in "trivial" examples. They're very close to the sort of examples a lot of tech people can actually use as individual professional engineers.

I think the void where non-trivial examples should be is the same space where contrarians and the last remaining few LLMs-are-useless crowd hangout.

[j45]

There is a scale somewhere in these types of articles that will emerge.

It might be something being actually new (cutting edge) vs new to someone vs the human mind wanting to have it be novel and different enough as a comparable percentage of the experience of the first time using ChatGPT 4.

There is also the wiring of non-deterministic software frameworks and architectures compared to the deterministic only software development we're used to.

The former is a different thing than the latter.

[sroussey]

LLMs are best demonstrated with greenfield examples.

[j45]

Plus, applying non-deterministic algorithms in a deterministic way might not always work the same. The software developers are also changing the frames and terms of reference.

[fho]

Point in case: i've been trying for weeks now to generate a CFD solver that is more than the basic FDM "toy example".

The models clearly know the equations, but run into the same issues I had when implementing it myself (namely exploding simulations that the models try to paper over by applying more and more relaxation terms).

[sundache]

I only see 148 lines of assembly and a dockerfile that's 7 lines long. Am I missing something or should that take a human less then several weeks.

[dotancohen]

Depends on what's in those 148 lines.