[fasterik]

I think there's an analogy here to GRE scores and other standardized tests. With GRE scores you can predict someone's GPA, but you can't necessarily predict their success at doing original research. That's because success as a researcher depends on personality traits that are hard to measure like creativity, independence, self-discipline, perseverance, and passion about a subject.

Programming puzzles are the equivalent of the GRE for programming jobs. It might tell you something about the candidate, but I'm skeptical that it measures the most important traits and skills that really good programmers have. It's an empirical question, but your skill at solving programming puzzles might not predict your ability to systematically break down large problems, write readable, robust, well-documented code, or design and debug complex systems.

I also have my own theory about this which might just be my own personal bias. I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. On the flip side, you are selecting against individuals who are intrinsically motivated to learn and build things, but who may not care about those external factors. My theory is that organizations with too much of the former and too little of the latter are going to have issues with groupthink and will be less likely to be innovative.

[hypefi]

"I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. "

Totally agree