[vundercind]

Oh good, I was afraid this was gonna go into cuckoo territory.

Yep, this is what works. All the attempts to turn it into a simple spreadsheet based on something you can easily measure from an office across the state have been so flawed they’re very nearly useless, if not actively harmful. We keep doing it anyway because the people making those calls either don’t understand the field or do, but don’t care because they want to implement bad ideas for political reasons (trend-following to avoid criticism is huge, for one thing)

[skirmish]

On-the-job observations are very subvertable though. They happened a few times in my high school in Soviet Union. At least twice the teacher would prep "good" students a few days in advance: I will ask you this question, and this is the answer you are supposed to give, make sure to memorize it well. So the class is engaged, gives good answers to hard questions, except this is all a Potemkin village deal!

[brnt]

Any human can manipulate their assessment. However, I think a skilled and trainer observer has a better chance of detecting that than any standardized blind mechanism.

[gopher_space]

Practically speaking your interview would require applicants to apply claimed knowledge to novel context.

A simple method would be to identify interesting problems from recent PRs and ask them to walk you through their approach to discovery and solution. It's a problem they should be familiar with, but in a new shape and with different labels. Let's see what they come up with.

[marcus_holmes]

Same for the ubiquitous "story points per sprint", "number of hours sitting at the desk", or "lines of code written" measures of productivity for engineering teams; because they're easy to measure and provide some kind of number that can be reported, they get used. Despite being completely useless and actively harmful.

Your point about people who don't understand the process using these measures because it suits their purposes is also relevant. Productivity measures can be a political tool.