[yoden]

Every time this comes up, lots of people start with the assumption that coding problems have a large amount of false negatives.

Because of this perceived "truth", I had the same worry when we started to implement a coding problem. Rather than guess, we decided to measure it: for the first six months, we used a wide filter (50% pass rate, actually like 65-80% of people who didn't cheat).

What we found was that there were zero candidates in the bottom 66% who passed the rest of the interviews. The plagiarism detector also had no false positives (based on manual review). So at least on this sample, we found that we could screen out about 80% of applicants without having _any_ false negatives.

I'm sure there are some bad employers misusing coding tests, just like with any tool, but I have to imagine many others have done similar experiments and found their tests to be effective.

[jfim]

The amount of false negatives really depends on where the bar is for passing the interviews.

If the bar is "it looks alright and the person knows the language they're using," it's probably not generating a whole lot of false negatives. If the bar is "the candidate comes writes the optimal solution to a complex algorithmic problem in 45 minutes," then that's highly noisy and tends to filter for people who have done a very similar problem recently.

Unfortunately, too many interviewers use the latter bar for passing or failing candidates.