[bnegreve]

Of course we should be more critical, the methodology here is far from perfect. But right now, people have much faith in interviewing, and this research suggests that this faith may not be justified. As you mentioned, this is not the only research that reaches this conclusion. There is also the work by Daniel Kahneman[1] which I find pretty rigorous and draw the same conclusions about interviewing.

So obviously, the title of this article should be "Maybe interviewing is not that useful" instead of "The utter uselessness of job interviews", but besides this I find your comment unjustified. In fact, it's quite the opposite, I believe this type of work contributed to make us more critical by questioning some basic facts about interviewing, that i would have never questioned just a couple of years ago.

> Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate)

ok, this is interesting, where is it mentioned?

[1] Think fast and think slow. There is this short article which mention some of the results and has been discussed on HN a couple of times already. http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-ha...

[andrewmutz]

It's easy to present data that shows flaws with today's interviewing.

It's a lot harder to present a better way of predicting candidate performance in the workplace, along with substantial data that indicates it's better than today's methods. Corporations would love more effective ways to determine effectiveness/performance before hiring.

Interviewing is terrible, but that doesn't mean there is a better option.

[jernfrost]

Nonsense. First step is to acknowledge that interviewing doesn't work very well. You can't keep deluding yourself because you haven't found a better way. Accept reality.

This is one of the biggest problems I see with business guys today. They want absolute certainty in a world which can't offer it. Start accepting that there is a lot of stuff we don't know and can't know at the moment and we simply have to work towards getting better.

You can't get better if you don't acknowledge that there is a problem in the first place.

These silly, must have +5 years of experience, and check all these boxes with technologies to get the job clearly shows the industry is completely lost at the moment and isn't willing to acknowledge it.

[mahyarm]

Of course people know interviews aren't the greatest, but it's the best tool they have right now. What people are pushing back against is criticism without solutions.

There are a bunch of other testing methods that are illegal in the USA too, such as IQ tests.

[LnxPrgr3]

When users of code I've written complain that something sucks to use, I don't demand they come up with a better solution. Probably I don't even want to know what they think the answer is—unless they design user interfaces for a living, odds are their answer will be terrible and they won't even be happy with it if I build it.

This is as it should be—I'm paid to solve problems.

Why is this different?

The more I read about interviewing, the more I realize too many people think they have this problem solved—their amateur psychology is impeccable and their technical screens test for exactly the right things, no more and no less. Did they do a bunch of controlled studies to convince themselves of this, or are they taking sounding good, or intuition about the statistical outcomes of different techniques, to be equivalent to truth?

Maybe the first step is to collectively realize we have close to no clue what we're doing, and are being asked to solve a hard problem: individually, to talk to someone for an hour and make a hiring recommendation. In aggregate, to make the decision based on a handful of these one-hour conversations.

Maybe the first step is to realize this is a problem worth trying to solve.

Maybe the first step is vocal non-acceptance.

[duncan_bayne]

> What people are pushing back against is criticism without solutions.

Which is bullshit. It's perfectly reasonable to criticise something without proposing an alternative. It's especially ridiculous to reject criticism provided without alternatives, when it's literally your job to do the work being criticised.

"Hey the way you're doing this part of your job produces results no better than random selection."

"Bring me solutions, not problems!"

[AstralStorm]

Solution: toss a coin or take n first applicants for the trial period. Same effectiveness, much cheaper. No need for the interviewer.

[cubano]

I still don't understand why employers can't simply set up a two week "trial contract" where as promising candidates simply work for two weeks so everyone can actually see and judge, with real world empirical data, how well the person does in the environment at the actual job.

Yes yes..of course I know this could be gamed as well, but no matter...you can't really argue that this wouldn't be magnitudes better then the typical current/broken interview process.

[flukus]

This will get you the most desperate employees, not the ones you want. I've got a home loan to pay, I'm not switching jobs if you can only guarantee 2 weeks of employment. And if there are multiple offers around, I'll take the 3 month or full time position, even if I'm half way through the 2 week contract.

Also, not all jobs/codebases lend themselves to being productive in 2 weeks, I'd argue they should be, but they aren't.

[AndyNemmity]

The concept of a two week trial contract is interesting, but also as flawed as anything else.

I switched roles to a new team, and the first 2 weeks were a trainwreck. It was almost of no predictive quality on how I would do.

Now, there are many reasons why that was the case, and perhaps those underlying issues should be addressed, but from all the role changes I've had, the first two weeks shows how well the group you're going into can onboard, more than it shows how productive the individual will be in the long term.

[AstralStorm]

You would need a longer trial period, a few months is fine with a clause that allows earlier termination.

[chrismcb]

Yes, because it would be magnitudes worse. So do you give the assignment to? You have several hundred applicants, do all of them get the 2 week assignment? Who watches over them and answers their questions? After spending that much money, is the answer you get any better than a set of interviews On the other hand, assume I am in a job and want to change positions, for any number of reasons... How many two week jobs do I have to take? Or should I quit my job first?

[jacobr1]

In the 2004-2007 timeframe, the company I worked for hired software engineers via a staffing company for three-month contracts. We interviewed the candidates with the intention of making a full-time hire. As the contract term approached, the management team did a 360 review, the decided to offer a full-time position or just not-renew the contract. This had some downsides, but overall I found it to be better that alternate approaches I've tried before or since. It stopped being viable once software engineering became a sellers market.

[flukus]

This seems to work ok for companies working on green field stuff, though you'd still probably struggle to entice people to leave their jobs for you. For other companies though, where technical debt and poor management is everywhere you look it doesn't. It gives employees a chance to see what they're really dealing with and to look for work two months later.

[wolco]

If someone is currently employed it makes that arrangement difficult.

[AstralStorm]

Indeed, though any kind of arrangement is difficult. You might end up in a dead end job, not matching your skills or otherwise soul crushing regardless of the method.

[Beltiras]

I took a job half a year ago. I'm still "learning the ropes" so to speak. I remember the first two weeks. Nothing would have been gleaned from them.

[RugnirViking]

In most countries that don't have employ-at-will I.E you could just fire them anyway, this is a real thing that actually happens

[Spooky23]

Yet, literally every professionally employed person is in their current gig via that process.

It works good enough.

[LnxPrgr3]

Counterexample: I did not apply to, nor interview for, my current gig, and I'm not cheating by working for myself.

And I bet almost everyone reading this has worked with at least one person they think shouldn't have survived the interview, and that person was making a boatload because they convinced the boss they're brilliant. Meanwhile 90% of their day was spent talking about how great they are, and 10% creating new bugs, and no one dared say anything because the thought of them being more "productive" was horrifying.

'It' mostly successfully matches employees to employers, but the quality of those matches may vary wildly.

Interview processes also vary wildly—you can't really say that it works without defining 'it.' Are we talking multiple technical screens that require writing code or a single fluffy buzzword-laden conversation with a C-level? Both have failure modes, but those failure modes sure are different.

[Spooky23]

> Interview processes also vary wildly—you can't really say that it works without defining 'it.' Are we talking multiple technical screens that require writing code or a single fluffy buzzword-laden conversation with a C-level? Both have failure modes, but those failure modes sure are different.

End of the day, everything evens out. People add the structure they need when they hire people. If your engineering interview process for some detail oriented gig is buzzword trivia with the CIO, the company will probably tank anyway. Conversely, if you do some nerd-fest whiteboard interview for a CTO in a bigger organization, you're probably not getting the right outcome either.

[LnxPrgr3]

Did you read the article? People add the structure they think they need to interviews, but are clearly able to troll themselves into worsening their judgments by requiring steps that not only don't help, but actively hurt.

And this is a near universal phenomenon. Almost everyone wants to "get to know the candidate."

[tacomonstrous]

That's a syllogism and I'm not sure it tells us anything. If you replaced interviewing with a footrace across hot coals the same assertion would be (just as vacuously) true.

[Spooky23]

It's deeper than that.

Everyone hates the process, and companies have investing major dollars and hours trying to improve. End of the day, little has changed since 1917. You either acquire-hire, get a strong referral or interview a pool of unknown applicants.

The tests and quizzes are little different to how a city hired an accountant in 1917. The old boys network evolved. Then you're left with the rest.

[jaggederest]

I strongly disagree. There are all sorts of situations where having a bad option is much worse than having no option, hiring and medicine clearly among them.

> Corporations would love more effective ways to determine effectiveness/performance before hiring.

This is irrelevant to evaluating the current methods. Even if there is no replacement, if they are useless, we should know it, bar none.

[emodendroket]

That feels a bit like saying we should just stick to blood letting and leeches because we don't know any more effective way of treating disease. Just because we don't have a superior alternative doesn't actually mean the current method is effective.

[RugnirViking]

A)

Leeches are often used in literal modern hospitals in the developed world as an effective treatment for certain ills.

B)

If we had no alternative, we should stick to it, but look at other things in the meantime. You seem to be advocating doing nothing at all, which is trivially easy to show works for no-one

[emodendroket]

Yes, leeches have extremely limited use. However, bloodletting and leeches killed far more people than they ever "treated."

[Sevii]

We have had a better way for decades, its banned. What do you think all these algorithm questions are supposed to do?

[flukus]

Any evidence to say that algorithm questions are a better indicator of job performance? Most development work isn't algorithm heavy at all.

[Jach]

I doubt there's that much, in general, since a lot of jobs just need a floor. But when it's used as a proxy for IQ test, the better you do at algorithms the higher IQ you probably have, and that typically correlates with better job performance, or being able to transform one's job into one with higher impact. (Even more if they have high Conscientiousness too but I don't think algorithms would correlate much with that.) It's also a weak proxy for seriousness. I hate algorithm questions (though fortunately not algorithms), almost everyone I talk to hates them, but if you're not expecting them and don't at least know the basic ones, you're not serious about applying to random tech job. (Which is fine, I don't mind that some people will get in a huff and walk out when asked to write a graph search algorithm or something as if it's beneath them or useless since the day-job never does such things, they just weren't serious about applying to that company.) Some tech companies have gotten rid of them, which is great, but you can't count on that yet as a candidate.

[demallien]

The problem with testing algorithms is that it in no way tests intelligence. I would think that 9 out of 10 programmers that know an algorithm would not be able to derive the algorithm from first principles. So you are just testing esoteric knowledge - it's qualitatively no different that asking someone questions about a specific framework / API.

You could make the argument that algorithms tend to be studied more by smarter people, but if that's what you're going for you may as well ask them about their hobbies, and hire the person that is into playing chess, or doing astronomy (or whatever intellectual pursuit you care to name).

If on the other hand you are interested in a person's ability to code, ask them to do so. The last time I had to hire someone, I wrote a small application with one module that was deliberately written in an obfuscated style. I asked candidates to bring that module under control - rewrite it in a readable code style. To do this, successful candidates needed to identify what the current code was doing by examining the public interfaces in a debugger, documenting what the calls seemed to do, prepare unit tests, and then rewrite the module in a readable style. It took about a day for most candidates to do.

At the end of that, you get to see a candidate's ability to read code, use a debugger, write unit tests, write documentation, and write well structured code, which is a pretty good coverage of the typical tasks in a developer's day. I feel this gives a much more realistic assessment of a candidate's capabilities that asking questions about a more or less randomly chosen algorithm.

[flukus]

> It took about a day for most candidates to do.

This is an issue as well. If you aren't google then a day is too much investment for a single job opportunity, especially if you're already employed.

[Jach]

I don't think someone with an IQ of 80 could program Djikstra's algorithm given a mathematical description of it with diagrams. I'm even skeptical of programming binary search. They might understand an intuitive explanation involving a phone book but I don't think they could program it. And even if they could, I think someone with an IQ of 120 would do it much faster, though both solutions would likely have the integer overflow bug that was even in Java's implementation for a long time. So I think algorithms do test intelligence, just not as well as an actual IQ test. It can easily be gamed by sheer memorization, whereas good IQ tests can't. I agree that other things like ability to play chess would probably test just as well as algorithms. If the industry switched to testing candidates to see if they can solve chess problems, or play a computer self-limited to some specific ELO, you can bet that everyone who was serious about getting a job in the industry would start playing a lot of chess, and those with higher IQs will on average play better chess.

When I have to give an interview and have to include an algorithms section I make candidates type code. Whether that's on a phone screen with a shared online text editor or in person with their laptop / an interview laptop, I want them to type stuff, not just rely on whiteboard pseudo-code and diagramming. As a vim user I discount that their editing environment may not be what they're used to but even if I was forced to use Notepad I could still bang out a function to test the even/oddness of a number (my own fizzbuzz) pretty quickly. So I at least make sure to test coding, even if poorly.

I agree work-sample tests are the best, but as another commenter noted if they take a lot of time for the applicant you're going to get people who refuse that just as some refuse to play the algorithms game. Especially if people have a github repo, especially if some of the projects they've worked on have had more than themselves as commiters, especially if they're currently employed as a developer at some other company that does general software. Unless you're trying to build a top team, which most projects don't need, you're wasting a lot of time trying to rank beyond "would work out ok" and "would not work out at all". I have a section in my phone screen that tests for regex knowledge, I'm primarily just testing to see if they know the concept or if when faced with a problem that regexes can solve (which actually does happen from time to time) they reach for writing some custom parser or not. If they vaguely remember there's a way to specify a pattern and find matches, that's a Pass. If they know grep/their language of choice's regex syntax and can give a full solution, great, I'll rank them slightly higher than someone who just knows the concept, but all I really care about is the concept. If they don't know the concept, that's a strong sign (to me) they won't work out.

I tried to do a semi work sample test with an intern candidate a few months ago instead of a different test, based on experience with a prior intern who struggled on something I thought was basic and left me wondering why I didn't catch that in the phone screen. Basically I gave them some stripped down code from several files that looks a lot like what we have in production (JS, using Backbone) explained the overall mapping from bits of code to what could be shown on the screen, and essentially asked them to add a new component (already written) to one part of the screen by modifying/filling-in-the-functions in a few places. It required them to read and understand some alien code, see what they can ignore, understand what was asked, and then do it (initialize something, pass it around, up to I think 3 indirect function calls of nesting, call a couple things on it). The candidate got through it, I'm not sure the old intern would have...

[mafm]

What is the better way that is banned?

[frankc]

I suspect he is referring to iq tests. They have a pretty high correlation to success and are banned.

[tacomonstrous]

In my experience, IQ tests are a good indication of your ability to take IQ tests and very little else. Of course, you can differentiate an absolute dunce from someone who's not, but nothing more subtle than that.

[barry-cotter]

http://www1.udel.edu/educ/gottfredson/reprints/1997whygmatte...

Why g Matters: The Complexity of Everyday Life

Personnel selection research provides much evidence that intelligence (g) is an important predictor of performance in training and on the job, especially in higher level work. This article provides evidence that g has pervasive utility in work settings because it is essentially the ability to deal with cognitive complexity, in particular, with complex information processing. The more complex a work task, the greater the advantages that higher g confers in performing it well. Everyday tasks, like job duties, also differ in their level of complexity. The importance of intelligence therefore differs systematically across different arenas of social life as well as economic endeavor. Data from the National Adult Literacy Survey are used to show how higher levels of cognitive ability systematically improve individuals’ odds of dealing successfully with the ordinary demands of modem life (such as banking, using maps and transportation schedules, reading and understanding forms, interpreting news articles). These and other data are summarized to illustrate how the advantages of higher g, even when they are small, cumulate to affect the overall life chances of individuals at different ranges of the IQ bell curve. The article concludes by suggesting ways to reduce the risks for low-IQ individuals of being left behind by an increasingly complex postindustrial economy.

[iterator5]

Much like the way that hackerrank tests how good you are at hackerrank tests?

[yorwba]

I wonder how much correlation with IQ is allowed before custom algorithms questions qualify as an illegal IQ test.

[lr4444lr]

I don't think the issue is correlation, but rather disparate impact[0].

Algorithmic tests have a pretty low bar to clear in IT jobs. Using them for secretaries or lab technicians is another story.

https://en.wikipedia.org/wiki/Disparate_impact

[Spooky23]

Employees will test the limits with lawsuits. I know at least one large state lost a class action lawsuit due to racial bias on civil service exams.

The problem with comparing IQ results at hire to employment success is that employment outcome is difficult to define over time. You're also unlikely to get statistically relevant data without focusing on large organizations with standardized HR processes. Most of the research is based on supervisory evaluations, which are not the most reliable indicators of anything for a variety of reasons.

The other thing I find amusing is that business folk who talk about this miss the fact that there are large workforces in the US that either have or do use standardized testing like this to hire and promote. Those are government bureaucracies, which function relatively well, but are hardly a model that most folks advocating this would aspire towards.

[leksak]

Stem cells and steroids maybe?

[mooreds]

> ok, this is interesting, where is it mentioned

https://mobile.nytimes.com/2015/08/28/science/many-social-sc... among other places

[jaredklewis]

People have too much faith in the efficacy of interviewing, ergo we should give slipshod journalism and science lacking rigor a free pass?

I'm all for upending the status quo with interviews, but let's not throw out science and reporting just to get there.

[schmit]

My point is not that this work is flawed, or that there should not be an article reporting on this research or the topic of interviewing. Rather, I think it'd be better for a third party to write about the topic in a more objective manner, rather than a professor promoting his own research (and thus with skewed incentives).

In particular, I was disappointed to find a (short) paragraph in the article that I find bogus. That does not mean the article shouldn't have been posted in the first place, but just that this paragraph should have been edited or removed.

I think there is something wrong when I feel like I have to look up the actual research paper and check whether the claims made in an article are supported by data and methodology. I should not be a skeptic when reading New York times articles.

To be fair, it is posted in the opinion section, but should we really just take this article as an opinion? That doesn't feel right to me either.

Then onto your last question and Daniel Kahneman, we can talk about that for a long time, but let me keep it short. The best place I know (though technical) is the blog by Andrew Gelman ([1][2] turned up in a 5 second Google, but there is way more on his blog), and Daniel Kahneman himself has "admitted" flaws in his studies [3][4].

[1] http://andrewgelman.com/2014/09/03/disagree-alan-turing-dani... [2] http://andrewgelman.com/2016/06/26/29449/ [3] http://retractionwatch.com/2017/02/20/placed-much-faith-unde... [4] https://replicationindex.wordpress.com/2017/02/02/reconstruc...

[Consultant32452]

I worked in the IT arm of a household name US non-IT company (just to provide basic context). Eventually we determined that interviewing just wasn't that helpful. We started telling the recruiting company, "Send me the best you've got that's available by Tuesday." Now certainly the recruiter is going to do some selection biasing there, but we found we had just as much success as our previous interviewing process that had a phone interview, face to face interview, and a test.

[jwatte]

The fact that psychology (and other social sciences and even medicine) have a replication crisis in spades is well known. Just Google it, if you haven't been following the general science news for the last few years.

Even the "hard" sciences have trouble, because journals prefer publishing new and positive results, rather than replications or negatives.