[bmir-alum-007]

"The problem is, if you're not a hacker, you can't tell who the good hackers are." http://www.paulgraham.com/gh.html

Relying on coding challenges or automatically-generated code metrics is deceptive because it tests if people can solve coding challenges, not solve real production issues or work on production apps, quickly and/or produce maintainable code. (It's about as useful as asking "why are manhole covers round" frequently asked questions interview trivia questions as proxy signals for intelligence.)

So, unfortunately, finding top coding talent is fundamentally an irreducible problem because it takes a little time & money to examine if someone is a badass coder by other badass/es reviewing some actual work and/or working with them on side-project/s.

[spitfire]

http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...

100 years of industrial psychology. See page three.

   -  General mental ability (Are they smart)

   - Work sample test (of actual work they might do, NOT HAZING/Quizes).

   - Integrity. (Varipus tests available for this)

That gets you > 60% hit rate.

Things that do not work.

   - Reference checks (r 0.26) 

   - experience (years) (r 0.18)

   - Years of education (r .10)

[bmir-alum-007]

Yup. Business theater is usually stuck in maintaining in-group approval, "industry standards" and historical continuity.

My favorite HR signal which doesn't seem to qualitatively work:

- non-rigorous certifications

The best interview styles are not labeled as such explicitly, whenever possible, because it makes for artificially-awkward pressure, alters expectations and ups the hype and hazing.

[acbellini]

Toptal does take time and spend money on reviewing candidates: there are no round-manhole-cover questions, there are algorithms for which one can prepare, there is a live coding session and a project that is evaluated by an experienced developer, together with the candidate. As you say, there are badass developers evaluating other developers reviewing their work and seeing them work :)

Disclaimer: I work as a Director of Engineering at Toptal and I've held a few hundreds of those live interviews myself.

[rattray]

I'm just confused why acbellini is being downvoted here. Yes, people from a company are coming to defend their record... did she say anything wrong/misleading/meanhearted/poorly phrased?

Disclaimer: she interviewed me. (The experience, by the way, was almost identical to the one I had interviewing with AirBnB recently).