[schmit]

I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look.

Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects.

Furthermore, some claims that make it into the piece are at odds with the data:

> Strikingly, not one interviewer reported noticing that he or she was conducting a random interview. More striking still, the students who conducted random interviews rated the degree to which they “got to know” the interviewee slightly higher on average than those who conducted honest interviews.

While Table 3 in the paper shows that there is no statistical evidence for this claim as the effects are swamped by the variance.

My point is not that this article is wrong; verifying/debunking the claims would take much more time than my quick glance. But that ought to be the responsibility of the newspaper, and not individual readers.

Politicians don’t get to write about the successes of their own policies. While there is a difference between researchers and politicians, I think we ought to be a bit more critical.

[bnegreve]

Of course we should be more critical, the methodology here is far from perfect. But right now, people have much faith in interviewing, and this research suggests that this faith may not be justified. As you mentioned, this is not the only research that reaches this conclusion. There is also the work by Daniel Kahneman[1] which I find pretty rigorous and draw the same conclusions about interviewing.

So obviously, the title of this article should be "Maybe interviewing is not that useful" instead of "The utter uselessness of job interviews", but besides this I find your comment unjustified. In fact, it's quite the opposite, I believe this type of work contributed to make us more critical by questioning some basic facts about interviewing, that i would have never questioned just a couple of years ago.

> Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate)

ok, this is interesting, where is it mentioned?

[1] Think fast and think slow. There is this short article which mention some of the results and has been discussed on HN a couple of times already. http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-ha...

[andrewmutz]

It's easy to present data that shows flaws with today's interviewing.

It's a lot harder to present a better way of predicting candidate performance in the workplace, along with substantial data that indicates it's better than today's methods. Corporations would love more effective ways to determine effectiveness/performance before hiring.

Interviewing is terrible, but that doesn't mean there is a better option.

[jernfrost]

Nonsense. First step is to acknowledge that interviewing doesn't work very well. You can't keep deluding yourself because you haven't found a better way. Accept reality.

This is one of the biggest problems I see with business guys today. They want absolute certainty in a world which can't offer it. Start accepting that there is a lot of stuff we don't know and can't know at the moment and we simply have to work towards getting better.

You can't get better if you don't acknowledge that there is a problem in the first place.

These silly, must have +5 years of experience, and check all these boxes with technologies to get the job clearly shows the industry is completely lost at the moment and isn't willing to acknowledge it.

[mahyarm]

Of course people know interviews aren't the greatest, but it's the best tool they have right now. What people are pushing back against is criticism without solutions.

There are a bunch of other testing methods that are illegal in the USA too, such as IQ tests.

[LnxPrgr3]

When users of code I've written complain that something sucks to use, I don't demand they come up with a better solution. Probably I don't even want to know what they think the answer is—unless they design user interfaces for a living, odds are their answer will be terrible and they won't even be happy with it if I build it.

This is as it should be—I'm paid to solve problems.

Why is this different?

The more I read about interviewing, the more I realize too many people think they have this problem solved—their amateur psychology is impeccable and their technical screens test for exactly the right things, no more and no less. Did they do a bunch of controlled studies to convince themselves of this, or are they taking sounding good, or intuition about the statistical outcomes of different techniques, to be equivalent to truth?

Maybe the first step is to collectively realize we have close to no clue what we're doing, and are being asked to solve a hard problem: individually, to talk to someone for an hour and make a hiring recommendation. In aggregate, to make the decision based on a handful of these one-hour conversations.

Maybe the first step is to realize this is a problem worth trying to solve.

Maybe the first step is vocal non-acceptance.

[duncan_bayne]

> What people are pushing back against is criticism without solutions.

Which is bullshit. It's perfectly reasonable to criticise something without proposing an alternative. It's especially ridiculous to reject criticism provided without alternatives, when it's literally your job to do the work being criticised.

"Hey the way you're doing this part of your job produces results no better than random selection."

"Bring me solutions, not problems!"

[AstralStorm]

Solution: toss a coin or take n first applicants for the trial period. Same effectiveness, much cheaper. No need for the interviewer.

[cubano]

I still don't understand why employers can't simply set up a two week "trial contract" where as promising candidates simply work for two weeks so everyone can actually see and judge, with real world empirical data, how well the person does in the environment at the actual job.

Yes yes..of course I know this could be gamed as well, but no matter...you can't really argue that this wouldn't be magnitudes better then the typical current/broken interview process.

[flukus]

This will get you the most desperate employees, not the ones you want. I've got a home loan to pay, I'm not switching jobs if you can only guarantee 2 weeks of employment. And if there are multiple offers around, I'll take the 3 month or full time position, even if I'm half way through the 2 week contract.

Also, not all jobs/codebases lend themselves to being productive in 2 weeks, I'd argue they should be, but they aren't.

[AndyNemmity]

The concept of a two week trial contract is interesting, but also as flawed as anything else.

I switched roles to a new team, and the first 2 weeks were a trainwreck. It was almost of no predictive quality on how I would do.

Now, there are many reasons why that was the case, and perhaps those underlying issues should be addressed, but from all the role changes I've had, the first two weeks shows how well the group you're going into can onboard, more than it shows how productive the individual will be in the long term.

[AstralStorm]

You would need a longer trial period, a few months is fine with a clause that allows earlier termination.

[chrismcb]

Yes, because it would be magnitudes worse. So do you give the assignment to? You have several hundred applicants, do all of them get the 2 week assignment? Who watches over them and answers their questions? After spending that much money, is the answer you get any better than a set of interviews On the other hand, assume I am in a job and want to change positions, for any number of reasons... How many two week jobs do I have to take? Or should I quit my job first?

[jacobr1]

In the 2004-2007 timeframe, the company I worked for hired software engineers via a staffing company for three-month contracts. We interviewed the candidates with the intention of making a full-time hire. As the contract term approached, the management team did a 360 review, the decided to offer a full-time position or just not-renew the contract. This had some downsides, but overall I found it to be better that alternate approaches I've tried before or since. It stopped being viable once software engineering became a sellers market.

[flukus]

This seems to work ok for companies working on green field stuff, though you'd still probably struggle to entice people to leave their jobs for you. For other companies though, where technical debt and poor management is everywhere you look it doesn't. It gives employees a chance to see what they're really dealing with and to look for work two months later.

[wolco]

If someone is currently employed it makes that arrangement difficult.

[AstralStorm]

Indeed, though any kind of arrangement is difficult. You might end up in a dead end job, not matching your skills or otherwise soul crushing regardless of the method.

[Beltiras]

I took a job half a year ago. I'm still "learning the ropes" so to speak. I remember the first two weeks. Nothing would have been gleaned from them.

[RugnirViking]

In most countries that don't have employ-at-will I.E you could just fire them anyway, this is a real thing that actually happens

[Spooky23]

Yet, literally every professionally employed person is in their current gig via that process.

It works good enough.

[LnxPrgr3]

Counterexample: I did not apply to, nor interview for, my current gig, and I'm not cheating by working for myself.

And I bet almost everyone reading this has worked with at least one person they think shouldn't have survived the interview, and that person was making a boatload because they convinced the boss they're brilliant. Meanwhile 90% of their day was spent talking about how great they are, and 10% creating new bugs, and no one dared say anything because the thought of them being more "productive" was horrifying.

'It' mostly successfully matches employees to employers, but the quality of those matches may vary wildly.

Interview processes also vary wildly—you can't really say that it works without defining 'it.' Are we talking multiple technical screens that require writing code or a single fluffy buzzword-laden conversation with a C-level? Both have failure modes, but those failure modes sure are different.

[Spooky23]

> Interview processes also vary wildly—you can't really say that it works without defining 'it.' Are we talking multiple technical screens that require writing code or a single fluffy buzzword-laden conversation with a C-level? Both have failure modes, but those failure modes sure are different.

End of the day, everything evens out. People add the structure they need when they hire people. If your engineering interview process for some detail oriented gig is buzzword trivia with the CIO, the company will probably tank anyway. Conversely, if you do some nerd-fest whiteboard interview for a CTO in a bigger organization, you're probably not getting the right outcome either.

[tacomonstrous]

That's a syllogism and I'm not sure it tells us anything. If you replaced interviewing with a footrace across hot coals the same assertion would be (just as vacuously) true.

[Spooky23]

It's deeper than that.

Everyone hates the process, and companies have investing major dollars and hours trying to improve. End of the day, little has changed since 1917. You either acquire-hire, get a strong referral or interview a pool of unknown applicants.

The tests and quizzes are little different to how a city hired an accountant in 1917. The old boys network evolved. Then you're left with the rest.

[jaggederest]

I strongly disagree. There are all sorts of situations where having a bad option is much worse than having no option, hiring and medicine clearly among them.

> Corporations would love more effective ways to determine effectiveness/performance before hiring.

This is irrelevant to evaluating the current methods. Even if there is no replacement, if they are useless, we should know it, bar none.

[emodendroket]

That feels a bit like saying we should just stick to blood letting and leeches because we don't know any more effective way of treating disease. Just because we don't have a superior alternative doesn't actually mean the current method is effective.

[RugnirViking]

A)

Leeches are often used in literal modern hospitals in the developed world as an effective treatment for certain ills.

B)

If we had no alternative, we should stick to it, but look at other things in the meantime. You seem to be advocating doing nothing at all, which is trivially easy to show works for no-one

[emodendroket]

Yes, leeches have extremely limited use. However, bloodletting and leeches killed far more people than they ever "treated."

[Sevii]

We have had a better way for decades, its banned. What do you think all these algorithm questions are supposed to do?

[flukus]

Any evidence to say that algorithm questions are a better indicator of job performance? Most development work isn't algorithm heavy at all.

[Jach]

I doubt there's that much, in general, since a lot of jobs just need a floor. But when it's used as a proxy for IQ test, the better you do at algorithms the higher IQ you probably have, and that typically correlates with better job performance, or being able to transform one's job into one with higher impact. (Even more if they have high Conscientiousness too but I don't think algorithms would correlate much with that.) It's also a weak proxy for seriousness. I hate algorithm questions (though fortunately not algorithms), almost everyone I talk to hates them, but if you're not expecting them and don't at least know the basic ones, you're not serious about applying to random tech job. (Which is fine, I don't mind that some people will get in a huff and walk out when asked to write a graph search algorithm or something as if it's beneath them or useless since the day-job never does such things, they just weren't serious about applying to that company.) Some tech companies have gotten rid of them, which is great, but you can't count on that yet as a candidate.

[demallien]

The problem with testing algorithms is that it in no way tests intelligence. I would think that 9 out of 10 programmers that know an algorithm would not be able to derive the algorithm from first principles. So you are just testing esoteric knowledge - it's qualitatively no different that asking someone questions about a specific framework / API.

You could make the argument that algorithms tend to be studied more by smarter people, but if that's what you're going for you may as well ask them about their hobbies, and hire the person that is into playing chess, or doing astronomy (or whatever intellectual pursuit you care to name).

If on the other hand you are interested in a person's ability to code, ask them to do so. The last time I had to hire someone, I wrote a small application with one module that was deliberately written in an obfuscated style. I asked candidates to bring that module under control - rewrite it in a readable code style. To do this, successful candidates needed to identify what the current code was doing by examining the public interfaces in a debugger, documenting what the calls seemed to do, prepare unit tests, and then rewrite the module in a readable style. It took about a day for most candidates to do.

At the end of that, you get to see a candidate's ability to read code, use a debugger, write unit tests, write documentation, and write well structured code, which is a pretty good coverage of the typical tasks in a developer's day. I feel this gives a much more realistic assessment of a candidate's capabilities that asking questions about a more or less randomly chosen algorithm.

[mafm]

What is the better way that is banned?

[frankc]

I suspect he is referring to iq tests. They have a pretty high correlation to success and are banned.

[tacomonstrous]

In my experience, IQ tests are a good indication of your ability to take IQ tests and very little else. Of course, you can differentiate an absolute dunce from someone who's not, but nothing more subtle than that.

[yorwba]

I wonder how much correlation with IQ is allowed before custom algorithms questions qualify as an illegal IQ test.

[leksak]

Stem cells and steroids maybe?

[mooreds]

> ok, this is interesting, where is it mentioned

https://mobile.nytimes.com/2015/08/28/science/many-social-sc... among other places

[jaredklewis]

People have too much faith in the efficacy of interviewing, ergo we should give slipshod journalism and science lacking rigor a free pass?

I'm all for upending the status quo with interviews, but let's not throw out science and reporting just to get there.

[schmit]

My point is not that this work is flawed, or that there should not be an article reporting on this research or the topic of interviewing. Rather, I think it'd be better for a third party to write about the topic in a more objective manner, rather than a professor promoting his own research (and thus with skewed incentives).

In particular, I was disappointed to find a (short) paragraph in the article that I find bogus. That does not mean the article shouldn't have been posted in the first place, but just that this paragraph should have been edited or removed.

I think there is something wrong when I feel like I have to look up the actual research paper and check whether the claims made in an article are supported by data and methodology. I should not be a skeptic when reading New York times articles.

To be fair, it is posted in the opinion section, but should we really just take this article as an opinion? That doesn't feel right to me either.

Then onto your last question and Daniel Kahneman, we can talk about that for a long time, but let me keep it short. The best place I know (though technical) is the blog by Andrew Gelman ([1][2] turned up in a 5 second Google, but there is way more on his blog), and Daniel Kahneman himself has "admitted" flaws in his studies [3][4].

[1] http://andrewgelman.com/2014/09/03/disagree-alan-turing-dani... [2] http://andrewgelman.com/2016/06/26/29449/ [3] http://retractionwatch.com/2017/02/20/placed-much-faith-unde... [4] https://replicationindex.wordpress.com/2017/02/02/reconstruc...

[Consultant32452]

I worked in the IT arm of a household name US non-IT company (just to provide basic context). Eventually we determined that interviewing just wasn't that helpful. We started telling the recruiting company, "Send me the best you've got that's available by Tuesday." Now certainly the recruiter is going to do some selection biasing there, but we found we had just as much success as our previous interviewing process that had a phone interview, face to face interview, and a test.

[jwatte]

The fact that psychology (and other social sciences and even medicine) have a replication crisis in spades is well known. Just Google it, if you haven't been following the general science news for the last few years.

Even the "hard" sciences have trouble, because journals prefer publishing new and positive results, rather than replications or negatives.

[trishume]

The claim you mention about getting to know the person better in random interviews as well as "People can’t help seeing signals, even in noise." is misleading and unsurprising.

People ask questions in interviews that they want to know the answer to and that could go either way. All questions have equivalent expected surprise, either both answers are unsurprising, or one answer is surprising, but you think you already know the answer is the other one.

If interviewers were asking questions like "is 2+2=4" they would have detected random interviewers way easier, but they wouldn't be trying very hard to get to know the person.

As for getting to know the person, the more surprising someone's answers are, if you believe they are telling the truth, the more distinguishes them from "average person who gave answers I expected", so you say you "got to know" them. This is unsurprising.

This isn't to defend unstructured interviews, other studies for a long time have shown them to be worse than structured interviews and test scores. If I had to guess the only reason the research in the article got published as novel was the random interview part.

Edit: Here's a table from a meta-analysis of lots of studies on correlation between different factors and job performance: http://imgur.com/a/YRFTh. Basically unstructured interviews aren't as good as structured ones, but they are better than nothing. Work samples are the best, structured interviews and IQ tests tie for second.

Note that this meta-analysis is combining a bunch of different fields to yield general observations, a specific field may have different results. But in expectation for a randomly selected field these are fairly solid results, and I don't expect fields vary from them too much.

[xapata]

> Politicians don’t get to write about the successes of their own policies

Politicians boast about successes constantly. I don't understand what you mean by "get to".

[BigJono]

They also run expensive ad campaigns promoting their own policies, at least here in Australia. They don't even have to be successful policies to be able to talk about them...

[ljw1001]

Politicians _always_ promote the success of their own policies.

[derefr]

I think the parent means, we don't take as objective fact a politician's claims that their policies worked; we interpret it as self-aggrandizement and scrutinize the claims quite heavily. And that—given the push to publish and the fact that null-result studies aren't very publishable—we should likely do the same for research conclusions.

[ljw1001]

I understood, but I think recent evidence in the US suggests that a great many people absolutely take as objective fact what a politician claims. A similar number probably absolutely believe the opposite with little or no evidence. Net, I think citing politicians was perhaps not the best analogy.

[schmit]

Sorry I was not more clear. What I meant was that the NY Times employs journalists and fact checkers and editors to validate stories, say on politics to make sure, to the best of their abilities, that the articles they post are correct.

Why is that not the case here, where a professor is allowed to sell his own work? It is as if Obama is the NYT reporter for Obamacare.

That people believe politicians blindly is a topic for another day :)

[emodendroket]

Maybe, but doesn't a claim like "a 30-minute conversation with someone is a strong indicator of job performance" deserve scrutiny of its own?

[curun1r]

Unless you're talking about a very high-level position that either requires substantial leadership traits or very specialized knowledge, interviewing has never been about finding the right person for the job, it's been about finding a right person for the job. The reality is that most jobs can be done successfully by a large number of people. The accuracy of the assessment of the candidate's abilities is, to my mind, a secondary concern in the interview process and yet, just as in this study, is the only part of the interview process studied and critiqued.

But what I find far more important in the interview process is involving current team members in the process of selecting new coworkers. It's one way of getting teams to be bought into a feeling of shared purpose and is the first step in establishing a working relationship. If you don't give at least some of the current team a role in the hiring process, teams will feel imposed upon by those hiring and won't be as understanding about flaws in those added to the teams.

Focusing on selecting the "right candidate" is really myopic in a situation where there are likely many right candidates. We should, instead, be focusing on not selecting a wrong candidate and fostering the right team dynamic. We already know that strong teams significantly outperform strong individual performers who don't cooperate. Yet hiring still seems focused on optimizing for strong individual contributions. And I've yet to see a study that looks at the flaws of the interview process in building ineffective teams.

[emodendroket]

I think you may be misconstruing the argument here. I agree that any number of people will fit -- but the problem is, what if interviews aren't even selecting "a" right person for the job?

[curun1r]

> what if interviews aren't even selecting "a" right person for the job?

We know that interviews are selecting a right person some percentage of the time. That percentage will never be 0 or 100. We will always have to accept some bad hires.

I would rather that successful hire percentage be somewhat lower and keep the team engaged in defining culture and hiring standards than to give up team involvement for a higher right-person rate.

My main point is that the people studying this issue and arguing against the interview process tend to only look at less than half of the benefit of the interview process. If we're going to ditch the interview process, whatever replaces it needs to have the same property of involving the existing team or it is, to my mind, automatically worse than what we have now. Because while we're all aware of how flawed the interview process can be, it does work to some extent. When I was hiring, only 2 out of the 50 or so that I hired didn't work out. Would I like to have avoided hiring one or both of them? Sure. Am I willing to sacrifice the team involvement benefits to got to do so? Absolutely not.

[emodendroket]

But you have no control. How do you know your success can be attributed to the interview?

[dsjoerg]

Of course it does. However the article and perhaps the underlying paper may be as useless as the interviews they are criticizing.

[hobls]

It's also worth noting that those students are probably very inexperienced interviewers.

[AstralStorm]

Indeed, warrants further study.

[jernfrost]

I don't see the problem. That researchers haven't done a perfect study in every possible way is hardly the issue here, but rather the blind trust in our ability to interview and hire people. This research paper does not come out of the blue. This is something that has been observed by many organizations and people for quite some time, that interviews are broken.

In fact there is an awful lot of wrong stuff society keeps doing despite evidence over many year suggesting it is stupid. The stupidity of high CEO salaries e.g. has been proven quite well, yet business keeps a blind faith in it. Getting employees in complicated jobs to perform by rewarding them on extremely narrow metric has also proven itself counterproductive yet the MBA crowd refuse to believe it doesn't work.

Business practices often seem to be more like religion than founded on reality. It is a GOOD thing that some researchers are trying to do their bit to correct this flawed picture.

[howitworks88]

> I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look.

Are you suggesting anything out of the ordinary is going on here? Doesn't one have to present their work as legitimate and wait for feedback? So long as they are open to that feedback, I don't really get this argument?

> Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects.

So present that back them as a refutation? Are you waiting for someone else to do it? Why are you debating the reliability here?

[kem]

There are many, many studies showing the same thing. This study is just one example. See, e.g., meta-analyses of your choice:

http://psycnet.apa.org/journals/apl/79/4/599/

They're predictive, but not very.

Structured interviews are better, but not really by much.

The problem with all of these things is they work, but not very well, and people make too much of any one thing. An interview is a very small slice of behavior, even if it's structured very well.

People make too much of them.

[valuearb]

Psychology is a science like astrology is a science. Sample sizes are the least relevant of the flaws with its studies.

[ppidugu]

You are absolutely right!