[dapangzi]

> you don’t have standards.

The problem is that LLM mess up things as basic as math and dates, and that's before the context gets too large and it starts making other mistakes.

Edit: Also LLM over mock tests and juniors trust that...

[taurath]

Not very often, and most of the time it shouldn't be generating those but rather formatting code to test that. If you accept the non-determinism and use some of the more recent models, you'll find it can do 99% of it very fast, and with some guardrails and testing it can fairly reliably produce working solutions.

[dapangzi]

> Not very often

> testing

This does not match my experience, have been working with LLM since 2023. We presently use the latest models, I assure you. We can definitely afford it.

I am not saying LLM is worthless, but being able to check its outputs is still necessary at this stage, because as you said, it is non-deterministic.

We have had multiple customer impacting events from code juniors committed without understanding it. Please read my top level comment in this post for context.

I genuinely hope you do not encounter issues due to your confidence in LLM, but again, my experience does not match yours.

Edit: Would also add that LLM is not good at determining line numbers in a code file, another flaw that causes a lot of confusion.

[giantg2]

I had a mid-level submit a PR implementing caching. I had to reject it multiple times. They were using Copilot and it couldn't implement it right and the developer couldn't understand it. Stuff like always retrieving from the API instead of the cache, or never storing the object in the cache after retrieving it.

They promoted that guy over me because he started closing more stories than me and faster after he started using Copilot. No wonder that team has 40% of its capacity used for rework and tech debt...

[dapangzi]

This matches my experience so hard that I wrote a novel below, have seen this pattern a lot, wanted to expand so people can understand the cycle/pattern.

Let's propose a generic scenario that shows why being able to engineer and read code is still important, and is a story we've all heard or seen a thousand times since the great LLMing of 2025.

"Just deliver the feature/product, we expect `ridiculousMetric` increase in productivity due to LLM" screeching from management and product/business.

A junior engineer will find someone who is willing to rubber stamp their LLM PRs so seniors or designated product experts don't even get a chance to check.

The LLM modifies existing tests to game everything to pass, the junior doesn't know any better, and so it quietly makes it to prod.

Because management is thinking in sprints, the way they see it, the ticket is closed, it's a win.

Then the broken production code, which junior will eventually be promoted for because the ticket is closed on paper, breaks prod, causes a huge outage costing `hugeNumber` dollars to the organization, and senior engineers have to clean it up. To boot, the spend metric is trash because of the LLM not knowing how to scale infra.

Since juniors can't meaningfully debug due to the toxic cycle, seniors spend too much time cleaning things up and it blocks their deliverables, and seniors look bad to leadership. Then they get managed out for not delivering, while the juniors lacking engineering experience due to the toxic cycle continue to rise through the ranks for delivering, even though their deliverables are trash.

I don't blame the juniors, they are under immense pressure and genuinely don't know better. I blame short-sighted leadership.

I've heard this story from contacts at any of the big names you can think of.

It seems US tech industry is flying head-first into having giant teams of mid and senior level engineers who don't know how to debug or deliver efficiency within the next five years.

We're failing our juniors, and punishing seniors for having standards.

[taurath]

I haven’t run into that problem but I do also hold agents on a tight leash!

[giantg2]

I've never had an LLM create a robust, meaningful test file. I end up rewriting at least half of it.