[Kamq]

> Leetcode is bad, but it’s bad in the sense that it errs too heavily on filtering out false negatives

But, it doesn't. It filters for something orthogonal to development, which is ability to obsess over clever algorithmic solutions. Ok, well my company does HackerRank instead of LeetCode, maybe LeetCode is magically better, but I'm not seeing anything that tells me someone who grinds LeetCode is actually going to be useful on my team.

Look, you want an idiot check to make sure someone is actually able to code, fine. That's probably a good idea. But the number of stories on here about people being turned away because they hadn't run into a particular tricky algorithm problem is concerning.

> Giving Tim the benefit of the doubt in this story, it still holds true that for every extraordinary and invisible superstar like Tim there are 99 under-performers who are indistinguishable from him.

But they aren't indistinguishable. The author of the blog post was perfectly able to distinguish them. That's my point. There are plenty of ways to be able to distinguish them, this metric just isn't one of them. Because it's a bad metric.

It may not be legible to the higher ups, but good lower level managers have no problem distinguishing good unconventional people, and under-performers.

> We need to empathise with our managers and the processes in our organisations to understand their purpose and how they came to be.

I do empathize with the managers, at least the lower level ones. That's why I advocated for putting them in complete control and giving them unilateral firing privileges and increasing their pay.

> the best organisations that I’ve worked with used metrics in a very effective way (mostly in start ups). The worst did too.

You're really making it sound like metrics (at least as traditionally practiced in software) are orthogonal to being a good organization. If that's true, we might want to re-think how much time we spend on them and how much money we spend capturing them.

Now, if you want to use profit, adoption, or user satisfaction as metrics, I'd love to talk about that, but I've seen nothing in my experience in the corporate world that tells me that the way we're currently using them is even net positive value.

[eckesicle]

It only appears that HackerRank/Leetcode isn’t good at filtering because you’re viewing it from your perspective, and not the perspective of the entire population that is tested. To you, the predictive power at the top tail end of the distribution is low, because you’re thinking of two strong developers Alice and Bob. Alice happens to know algorithm X and would pass the test, whereas Bob does not. But that’s not the population we’re testing. Think more along the lines of Alice and Bob and your grandmother were the test population. It’s absolutely fantastic at filtering the lower 95% of applicants because they will _never_ be able to pass. Yes, inadvertently 2.5% of “good developers” are filtered too, but that doesn’t matter to the outcome of your company. They just want someone competent, and they don’t care if it’s Alice or Bob.

The same logic sort of applies to Tim and his performance. The bias of having an imperfect metric is probably much better than the bias of letting an army of middle managers go with their cut. Besides, it doesn’t have to be a hard filtering function at this stage, but a metric to indicate that we need to look a little closer at Tim

[Kamq]

> It’s absolutely fantastic at filtering the lower 95% of applicants because they will _never_ be able to pass.

This is the part I disagree with. It hasn't been true for years. Anyone with the free version of ChatGPT can pass a hacker rank today.

> but that doesn’t matter to the outcome of your company

It does for mine, because we've hired all of the good developers that get through the process you're describing and it isn't enough. We actively moved away from what you're describing and turned the interview into a 2-3 hour pair programming session where the person completes a mini version of a ticket.

This has much more predictive power than what you're describing.

[eckesicle]

> This is the part I disagree with. It hasn't been true for years. Anyone with the free version of ChatGPT can pass a hacker rank today.

It certainly still is true today. Anybody who is sufficiently motivated to cheat can pass it. It was true prior to ChatGPT, and it still remains true today. And yet they don’t. Most people completely fail these screens

> It does for mine, because we've hired all of the good developers that get through the process you're describing and it isn't enough.

Then your industry is atypical in the type of applicants that you are getting. So to accommodate you’ve had to increase your false positives to reduce false negatives. That’s completely fine if it’s what you need to do, but it’s not the typical experience for a tech company.

We also do a pair screen after the code test and we still reject around 80% who make it to that stage. How do you scale interviewing everyone if you don’t pre screen?

[Kamq]

> Then your industry is atypical in the type of applicants that you are getting

Based on the quality of candidates that get through at other companies, I'm guessing our problem isn't atypical. Or at least, good devs often aren't getting through their pipelines at all. It's possible that in trying to reduce the false positive rate, they screened out all of the actual positives, but that doesn't paint a good picture of the industry.

> How do you scale interviewing everyone if you don’t pre screen?

We do pre-screen. The fact that they haven't encountered a tricky algorithm before isn't a problem. For ones where the syntax is valid, a dev at my company does a code review on it.