> The experienced engineer also gets an LLM though
Say experienced engineer Edna was working on a project and the new grad Nelson replaced her because he was cheaper and about as good if both have LLMs, i.e.,
Edna + LLM ≈ Nelson + LLM
even though
Edna >> Nelson.
So Edna needs to find a project to work on where LLMs don't equalize her arbitrage, i.e.,
My contention is "how do you determine if someone was 'as good'?" Very likely the people in the position to make a decision will not be qualified enough to even make that decision. IF (and really only if) such a determination is possible, then sure, I agree, find harder problems etc, but I assert that people will not make that determination (which is close to impossible to make) well, if at all because that wasn't the point in the first place.
If a bunch of projects given to NG + LLM fail, the company will say at some point "this is no good, let's not do that anymore". If they mostly succeed, they'll do it more.
From a middle manager's perspective, where they're funding 10-20 projects, this kind of measurement is feasible and carries at least some signal. So I don't agree it's difficult to measure.
Say experienced engineer Edna was working on a project and the new grad Nelson replaced her because he was cheaper and about as good if both have LLMs, i.e.,
Edna + LLM ≈ Nelson + LLM
even though
Edna >> Nelson.
So Edna needs to find a project to work on where LLMs don't equalize her arbitrage, i.e.,
Edna + LLM >> Nelson + LLM
and Nelson needs one where
Nelson + LLM >> LLM.