[Jemaclus]

> Furthermore, no matter what, poor technical ability seems highly correlated with poor communication ability – regardless of language, it’s relatively rare for candidates to perform well technically but not effectively communicate what they’re doing (or vice versa), largely (and fortunately) debunking the myth of the incoherent, fast-talking, awkward engineer.

My interpretation of this is that interviewees who can communicate clearly about code (whether they wrote it or not) correlate with high technical ability. Does this suggest that rather than having the interviewee write code on the spot, one could give them some new code they've never seen before and ask them to reason about it aloud for 30 minutes, then gauge their technical ability based on their ability to communicate clearly about the code?

In other words, could you replace live-coding with "here's some code, tell me about it"?

[lettergram]

That's actually how I conduct interviews. It's been immensely successful at weeding out candidates within 5 minutes; although most interviews are 45 - 60 minutes.

Basically, give them less than ten lines of code, ask them what it does, where are a couple bugs, ask what would you name the function, etc. Then we talk about how to improve it. I'd say, less than 20% of people I interview pass. It's actually amazingly low how many people can find a bug and communicate it.

Half the people don't even tell me what they are thinking. And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down. They think in their head, don't work through the problem at all.

[watwut]

"Half the people don't even tell me what they are thinking."

Maybe they are introverted or simply need to think before they talk. Around 50% of people are introverted (less in usa). Many people are like that and simultaneously quite skilled. Moreover, some environments punish errors, so people who worked/studied there tend to be conditioned to think before talking.

"And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down."

You are not buddies, you are interviewer about to decide whether they get hired. Many people shutting down might mean that they are not comfortable juggling "buddy" social role and expectations and "serious job interview" social expectations simultaneously.

[rnprince]

"You are not buddies, you are interviewer about to decide whether they get hired. Many people shutting down might mean that they are not comfortable juggling "buddy" social role and expectations and "serious job interview" social expectations simultaneously."

Very good point.

[kafkaesq]

Right -- you don't need the "act like a buddy" part for this technique to work.

[_0ffh]

I'm not sure about introverted. I am, but when I get excited about a topic, I can be as loud as the next person. As long as I feel I have something to contribute, that is. I can rant for two minutes straight and then suddenly and unexpectedly shut up when I've made my point. People are sometimes surprised and ask why I suddenly stopped talking. I'll answer: "I've made my point."... :3

So my bet is on error avoidance.

[flukus]

I'm pretty similar, but solving a bug in an interview question is hardly something I'd get excited about.

[_0ffh]

I agree. Still, the point I was trying to get across is this: While introverts do not strive to take center stage for its own sake, they are perfectly capable to do so for any good reason. Introversion is not shyness.

[wyager]

Being an introvert isn't an excuse for poor communication. Regardless of its cause, poor communication is bad in a work environment, and it makes sense for interviews to select against it.

[jroseattle]

This is significant, and can be more so depending on the company and environment.

I've had my own team members tell me they aren't engaged, don't speak up in meetings, etc. because they are an introvert. Like it's a condition we must accommodate. Bullshit.

Communication is a skill. We value clear communication in our company. If you join our company, you may not be an expert in communication, but you will grow that skill.

An interview is a different scenario, but still -- you need to be able to communicate with me. If you can't, we will pass on you as a candidate.

[watwut]

"don't speak up in meetings"

People should speak in meetings when they have something to say. Not just so they speak - that just waste everybody time.

The more important point is that if 80% of people shut down while they are talking to someone, maybe has more to do with that someone then 80% of population being uncommunicative.

[jroseattle]

We're not quite that naive and clumsy. I'm the first one to make sure we keep discussion on-point and actively shutdown the monologuers of our team.

So let's clarify -- "speak up in a meeting" means when a senior person doesn't provide input at a time that would provide better direction, identify issues earlier in a process, etc. If you're a senior engineer, I need your input to ensure we're doing the right thing(s).

The forum of a group meeting isn't always in the wheelhouse of someone who freezes up when speaking in front of others. It's understandable -- I used to be one of those people. It can be out of your comfort zone, and doing so feels super-risky as well as plain frightening. It is still just a skill to be learned, no different than building muscles through exercise.

In our group, we talk a lot about trust and support. Everything we do is in the spirit of making each other stronger. Everyone has strengths and weaknesses -- share your strengths with the group, and build your weaknesses from the strength of others. It sounds pie-in-the-sky, but as the head of my group, I make sure we take this seriously.

So we don't have 80% of our group shutting down -- nobody talks over the top of anyone else, and we don't allow it. We need the input of the best of those, and sometimes that involves speaking up. If you're on my team and that notion makes you nervous, I understand. But as I've always explained to my team, we're going to help you build skills, and communicating is one of them.

[wolco]

If you want diversity you try to accommodate different personaility types. Otherwise you end up with a mono culture.

There is value in silence and limited communication.

[jroseattle]

Communication is a behavior and act, not a personality type. The suggestion here is that it's ok for someone who doesn't communicate well to be accommodated in the name of diversity. The notion of "poor communicator" as a personality type doesn't jive with me.

Replace "communication" with "data modeling" or "project management" or "reading". There are plenty who don't necessarily possess those skills.

Should we expect that others shouldn't develop skills in those areas (assuming they are important for their role) in the name of "diversity"?

[spoiler]

While I don't fully agree with the parent of your reply, I think they're right about communication being a skill. I am an introvert myself, but I've learned not to let that get in the way of me communicating with my colleagues or friends.

[watwut]

Thinking in your head before talking is not poor communication through. Not being able to explain yourself is, never talking is and shooting every half baked idea loud while others concentrate is.

The fact that someone takes time to put together ideas before opening their mouth is not bad sign. Bad sign would be if they don't share result once they have done thinking.

[bluGill]

This is interviewing 101 though: establish a report with the interviewer if at all possible. If the interviewer tries to be a buddy that is in your favor to go along.

[ice109]

rapport

[jalanb]

TIL Rapoport's rule, https://en.wikipedia.org/wiki/Rapoport%27s_rule

Oh I do love the interwebs.

[kccqzy]

I'm an introvert, hates social gatherings, etc. Yet I've found the ability to think aloud extremely helpful. Sometimes just vocalizing my thought makes the thinking clearer, almost as if hearing myself again makes the brain think over every thought once again. In a collaborative situation (such as pairing) this is even more helpful.

[michaelcampbell]

I believe it does exactly this. The rise of "rubber duck" debugging is a tool to extract this sort of thing, in fact.

[graphememes]

Generally to get candidates who are like this to open up, I simply assert:

"Tell me what you are thinking about right now"

"What is jumping out at you for you to be stalling?"

"Did you see something?" --> "Why?" --> "You seem to be quiet and deep in thought, what are you thinking about?"

etc...

[sundvor]

Yeah, that's great if you're NT and all that, but if you're interviewing someone who is anxious about the situation this might only add to the pressure, creating a melt down situation making it impossible for them to think.

In a normal work environment where he or she is left to think freely the the same candidate might excel.

[valdiorn]

I'm sorry, but this is an interview. It's entire point of existence is to allow communication.

If the candidate would like some quiet time to think about the solution, I expect them to respond with "I think I'm forming a solution, just give me a couple of minutes to think about it before I present my case".

If they can't even do that, then I'm sorry, but they are no good to anyone. You can be the smartest and best developer in the world, but if you're completely incapable of representing yourself and your ideas; to have a dialog about your work, you are effectively worthless as an employee.

Some companies might have a place for that special someone, who you can lock in a room for a month and he will later emerge with an amazing new piece of code that will solve your problems, shielded from all the problems of the outside world, but companies where that is possible are very rare.

[dkersten]

Some companies might have a place for that special someone, who you can lock in a room for a month and he will later emerge with an amazing new piece of code that will solve your problems, shielded from all the problems of the outside world, but companies where that is possible are very rare.

Also, being able to develop like that is very rare because in the real world, requirements change or are vague and need to be clarified, or your software has to interoperate or integrate with other software. There is simply no way to avoid regular communication and still produce something that works how it needs to. Problems where you can shut yourself away for a long period of time certainly exist, but in the grand scheme of software jobs, they're few and far between.

[davidbanham]

And on a normal workday in normal surroundings that person might do just fine. An interview is a totally abnormal, high pressure situation.

The point is that with this kind of make or break style of candidate vetting you're probably leaving a lot of great talent by the wayside.

[graphememes]

Unfortunately interviews are brief, the point of this isn't to add pressure its to relieve it. Silence for too long in such a short period introduces pressure. Letting them use me as a sounding board we can create a connection that opens dialogue and allows ideas to flow.

The whole point is to make them comfortable and that takes the highest priority. Being able to read the individual and react to them is key, not everyone is the same, hence why there are a multitude of responses, and the ones given are just a starting point.

[sundvor]

Ah, I read you now, thanks. I may have reflected on one interview that went particularly badly, being somewhat reminded of it. But with the extra information, that is a great approach if I understand you correctly - especially the silence adding pressure part, which is absolutely true.

I.e. if stuck at something, even if seeming trivial, it might pay (to go back or have a chat around it, even guide him/her a bit so as, etc) to get the candidate loosened up and feeling safe which should then yield better responses after.

[vacri]

You can bikeshed interviewing processes that leave out this or that potential candidate forever. And any candidate who bombs in any kind of interview could turn out to be the next Einstein.

There is no one-size-fits-all interview technique; if there was, it would be used all over the place. Good interviewing techniques are about playing the numbers, not leaving no programmer behind.

[flukus]

That sounds awful, you are actively sabotaging someone by ruining their concentration while they're trying to solve a bug in unfamiliar code under time pressure. This is like having a manager requesting status updates every 10 minutes when you're trying to fix a production failure.

Even if I solve the problem I'm walking out of that interview with a very negative opinion of your company.

[dpark]

If you completely shut down during the interview and don't communicate, you're probably getting walked out from the interview. If you cannot or will not communicate, then the interviewer is unable to evaluate you.

I also don't think he was advocating nagging either, simply asking questions to get the candidate talking. If you "go dark" for 10 minutes, you're pretty much definitely stuck. Talking to the interviewer might get you unstuck. Staring at the whiteboard quietly probably will not.

[graphememes]

Yep, totally not nagging, giving them room to think and then jumping in to get them talking is definitely what you want to do here.

This also isn't your day to day job so you have to keep in mind that these interviews are generally held to an hour length and you can't really give them 30 minutes for a single problem that wouldn't give you much insight into the person.

It also heavily depends on the culture of the company as well. So being able to communicate effectively and act under pressure is something that goes with pair-programming and code reviews.

To note, I have never had anyone ever leave an interview feeling bad about the company or our process, the interview is about the individual.

[watwut]

In one interview, I was left in a room alone with coding exercises. The involved frameworks I did not knew and I had internet available. Then they came in and we talked about solutions.

I still think it was pretty cool way of interviewing.

[IanCal]

You might be over-estimating the complexity of the kinds of problems generally used for this.

You're not bothering people while they're taking an expected amount of time to look at it. You're stopping a several minute silence after handing someone a basic loop and if statement.

For most positions we're talking about, this should not be a major problem. I found when doing something similar people either looked at me like I was crazy for giving them such a basic problem and after being reassured there was no catch answered simply and clearly, or just couldn't do anything. Followup questions about the code then went terribly (e.g. a JS contractor asking for £500+/day who didn't know what the difference between global and local variables was, why you should put var in front of things, etc).

These things are often good to weed out the incredible number of people who seem to have little to no coding ability at all for jobs where that's an obvious pre-requisite. I've also had people fail to solve a basic problem (roman numeral generator or reader) at home in whatever language they wanted, not even getting vaguely close. One person even sent in their broken version pasted into a word document.

[maxxxxx]

If someone can't express himself, most likely that person won't work well in the team.

[barrkel]

Some people take a bit of time and familiarity to warm up.

[maxxxxx]

True. I am not sure if I want to take the risk though. I don't want to be a friend or tell jokes but i want to be able to talk technical stuff.

[delazeur]

There is an enormous difference between a person taking a minute to collect their thoughts without speaking and a person being unwilling or unable to communicate. I can't be sure, but it sounds like at least some people in this thread are talking about pushing back against the former as well as the latter.

[maxxxxx]

I am fine if they think about it for minutes or need a while to sort thoughts out. But at some point I would like to get some response. What else can I go by in the interview? And even if that person can write great code I am not sure things will work in the long run if he can't explain himself.

[paganel]

> Half the people don't even tell me what they are thinking. And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down. They think in their head, don't work through the problem at all.

As it happens most of my "aha!" code-related moments come when I'm either under the shower or when I'm washing the dishes. Very rarely did it happen for me to look at a piece code while in front of my computer and then immediately be stuck by what that code does or by some hidden bugs in it. It's all sort of mechanical, at least while I'm sitting at my desk. I'd say it would be very hard for people like me to reproduce that "aha!" moment during an interview, in front of some other people who expect me to have those moments of enlightenment right there and at that precise moment.

[Sacho]

You can always find something to talk about in the code, though. You can start by just describing what each line does, in your opinion. Then you can go back and start to group the lines - e.g. "lines 1-3 are the initialization, lines 4-6 are probably intended to flurb the glorb", etc. Yes, it's possible that you can find an interviewee who is great technically but shuts down during an interview. The question is how much time and effort you want to invest digging for that person, especially in light of such studies.

[surement]

You could look at a more complicated piece of code, let them think about it for a bit, and if they don't have that aha moment you can still discuss it with them. For example, pick a problem you someone on your team struggled a bit with, something that you wouldn't necessarily expect a candidate to solve during an interview, and discuss solutions with them. I think you could learn a lot about the interviewee this way.

[javabean22]

omg, are you me. No ahas at interviews. Good to great ahas at random times. Fuck whiteboarding interviews.

[marcosdumay]

> They think in their head

I'm left wondering why this would be a problem.

[BeetleB]

I don't know if the following is what the GP meant, but:

I've often had the pain of dealing with people who try to do it all in their head. Sure, one or two are good enough where it's not a problem. The rest of them? They keep getting confused and making mistakes. For things that are not very complex. If they had only written it out or drawn on paper/board, they would not get confused.

It's most obvious when I try explaining some code's algorithm to them. They'll begin with understanding, and then 70% of the way through they'll keep getting confused because they couldn't hold it all in their head.

I've often had to tell these people "Don't try to do it in your head!"

[irrational]

I have sort of the opposite problem with a coworker. This coworker only wants to talk through things and doesn't want to work them out on paper or a white board. The problem is that I can understand/remember almost everything I read, but have almost no ability to comprehend things that are being spoken (I suspect people are talking faster than my ability to think and parse what they are saying). I've talked to him about this to no avail. He just wants to talk things through, and then expects me to be following along and to provide insights when I have no idea what he is talking about. But if we can get it on paper or a whiteboard then I can reason about it quite easily.

[TeMPOraL]

I know the type of person. I usually can't follow them either (less about them being fast talkers, more about them being bad at explaining things). I found myself to usually only half-listen to what they say, and letting the other half of my mind to reconstruct the problem independently by itself. Sometimes I fail, and then I need to ask them to repeat what they just said. Other times my mental model is spot on, and I can find their problem quickly.

[elgenie]

Try to maneuver him to where there's a whiteboard and write things down as he's talking through them: if he's reading cues, he'll slow down so that your visible note-taking keeps pace.

If that doesn't work, get him a rubber ducky [] and tell him to go talk to it if he's not willing to compromise on communication styles.

[] https://en.wikipedia.org/wiki/Rubber_duck_debugging

[lj3]

I worked with somebody like that once. He would get massively frustrated because he needed that back and forth to think through ideas and I was giving him nothing. I would get frustrated because I had to digest information before I could say anything about it and he wouldn't stop talking long enough for that to happen.

I had a lot of success with asking him if I could think about the problem and we could schedule a meeting in half an hour or so. He was much happier with that result because I had thought through the problem and was now able to give him the back and forth he needed. I was also much happier because I was no longer expected to come up with insight on problems I had no time to chew on.

[fian]

I am often both requesting and receiving verbal problems related to the software I maintain (an Engineering simulation and analysis tool). When receiving a problem verbally is to slow the other person down by challenging any assumptions they may be making. By asking them to clarify it allows me to build a mental model of the problem before at a sustainable pace and also helps them identify a bad assumption they may be making.

When I am asking someone else to be the rubber duck I strive to highlight any assumptions I am making at each step of defining the problem. If I don't do this the other person will usually struggle to keep up.

[BurningFrog]

Can you keep some paper or a whiteboard around?

You don't have to just accept doing your mutual conversation exactly as he likes it.

[cakedoggie]

That is not the opposite problem. Talking through them and working them out on paper are the same thing. They are the opposite of someone doing it in their head.

[elgenie]

Not being able to successfully and clearly communicate their thought process about a simple problem one-on-one is a bad signal for being able to communicate clearly about a difficult problem in a group setting or being a useful sounding board for technical ideas.

[Fishkins]

It's clearly important to be able to communicate about technical problems, but is it really necessary to be able to narrate your thoughts about a technical problem in real time as you're first having them?

In my work experience I don't remember ever having to do that. I'd look at code & investigate a problem independently, then talk to someone about it afterward. I think that should be considered at valid approach to the interview.

[dota_fanatic]

It sounded from the example like the code under discussion was quite simple, "ten lines of code". So it's plausible your "look at & investigate independently" is allowed to happen, and should take less than a few minutes, say, and then this interviewer is inviting them to walk through things in a shared manner.

In other words, from my perspective, you're not actually in disagreement with their method. They weren't saying "narrate in real time as you're first having [thoughts]". Practically nobody does that ever, or is expected to do it, except in other practices like mindfulness or therapy and what have you. They were saying they ask them to understand it, explain it, and then talk about improving it. OP's problem was with the candidates whom couldn't understand the code, and weren't even willing to talk about the state of their understanding so that the interviewer might help them walk through things further...

[krzyk]

Yeah, but if you shoot for elaborate ones you will get politicians/lawyers not coders.

I don't have any source but from my experience mathematical/computer minded people are more likely to be introverted/talk less then arts people.

[woodandsteel]

>I don't have any source but from my experience mathematical/computer minded people are more likely to be introverted/talk less then arts people.

I think you are right about that, but that just means that if they want to be effective members of a team in an organization, they need to work on their communication skills. Ability to communicate is something you can learn to do better.

I know this for a fact. I used to teach public speaking, and students came out far better at it than they came in. The same is true for many other communication skills. See for instance Marshall Rosenberg's book Nonviolent Communication, which is great for interpersonal conflict situations.

In fact, I think a lot of introversion is at least in part due to poor communication skills that could be improved with training and practice.

[DiffEq]

I studied engineering and went to an engineering school (EE, ME, CE, etc). I have had contact with many engineers on a daily basis, although I do not do this work anymore myself, and I have found that engineers can talk very well - actually I have never met one that can't and/or won't (the real problem is getting them to shut up). So in my experience, smart useful people are always able to communicate. There may be the occasional savant who is awkward or such but I have not met them.

As for programmers - same thing. If they are any good they are more than able to talk the talk.

[ewjordan]

The article explicitly notes, though, that most excellent programmers were also excellent communicators.

Effective communication doesn't necessarily mean you talk a lot or are extroverted. It means that you say things that matter when they matter, and that is a crucial skill to have in almost any work environment.

[elgenie]

Having a high signal-to-noise ratio is (obviously) also critical to communicating successfully about code / technical issues. Talking for the sake of talking or waxing poetic about the understood 5% of the problem at the expense of the 95% remainder is a negative signal too.

[delazeur]

No. Thinking out loud is not the same as communicating a thought process.

[swimfar]

for one thing, you can't get partial credit for ideas/work that only exists in your head. (Similar to why it's a good idea to show work on homework assignments and tests)

If they sat there for a bit thinking about the problem and then provided a solution, I'm sure that would be fine. But that's not the situation he was describing.

[Jemmeh]

Because you might need to work with the team at some point to help each other solve a problem.

[DoofusOfDeath]

At least in my case, there's a big difference between

(a) discussing an open problem with a colleague, and

(b) explaining what's going on in my mind as I'm attempting to solve the problem single-handed.

I'm very effective at (a), and it's a skill that I've used many times as a software developer and/or grad student.

But (b) is more typical in psychoanalysis sessions.

[Jemmeh]

May I ask what you feel the difference in the two is?

I often have to explain a problem to my coworker and vice versa. It's also really helpful when pair programming. Just curious about your experience.

[SomeStupidPoint]

There's a weird dynamic in interview questions you rarely, if ever, get in the workplace: the person who actually knows about the problem is staying quiet while the one just starting to reason through it is doing all the talking about it.

There's no back and forth -- you're not actually trying to help them solve anything -- it's someone who already knows the answer judging how much of it I can reason out in half-an-hour. I can't take 5 minutes to think it through, I can't stop talking for any significant amount of time without being told to narrate what I'm thinking about, etc. That's nothing like a workplace situation, where they'd be expect to hold up half the conversation, be providing input and thoughts on the problem, and I could just tell them I'd be with them in 15/this afternoon/tomorrow/etc.

It would be more realistic if the interview didn't get to see the question before the interview, either. (And why I think design questions work better than coding questions -- you can do something they're not expecting in the design, ask more kinds of questions about the spec, etc which cause an honest back-and-forth.)

[jimminy]

"They think in their head, don't work through the problem at all."

Those are independent clauses. Just because the individual is not talking out loud, should not be indicative that they aren't working through the problem. Personally, I am a very visual thinker and layout visual models in my head for problems with high (3-5d) dimensionality that would take longer to draw clearly in 2D space while explaining all the traversals I'm mentally making. I would then reduce a solution, and explain that, or I apologize for zoning out.

It became obvious to my office mates when I was working through and reducing a problem space, because I would "hang" on the thought. Sometimes I would be down, stuck in thought, for 15-30 seconds and according to them they could tell because my face loses emotion and my eyes flutter about.

[gxs]

That's an interesting approach, but I wonder if you're screening for people who can speak and code vs just people who can code.

I'm a product manager and I suspect I would do pretty well in this type of interview unless the snippet is especially complicated and/or esoteric.

In that case you might be screening for people who can talk but not actually code or people who can code but might not be the best communicator.

Interestingly enough, if youre optimizing for specific outcomes, you may not need to screen out the latter.

Engineers who struggle to communicate but are productive can be extremely successful given the right environment - including having someone on the team who knows who to work with various personalities and is technical enough.

Just some rambling, but it's worth pointing out given the scarcity of engineering talent. Not everyone is going to be Paul Bucheit, ace product manager and engineer all wrapped into one.

[TheCowboy]

I'm not sure why you're being downvoted, but it's a valid comment. Even if someone disagrees, you clearly communicated a point that adds to the discussion.

Many are taking the article at face value, and are assuming there are no methodological flaws and that this study has been repeated enough times to invalidate any comments to contrary. It also didn't turn this topic into a "case closed" matter, as it's too big of an unsolved problem for that. They also just analyzed interviews, but did they monitor job performance after the interviews and for how long?

Additionally, differentiating between being a "good communicator" and a "good interviewee" is frequently overlooked. People too frequently rationalize their own interview approach as not the problem. Most candidates accept that they could improve and perform better at interviews, but a lot of people conducting the interviews don't think their process could do better.

[lj3]

> In that case you might be screening for people who can talk but not actually code or people who can code but might not be the best communicator.

From TFA and literally the thing that started this whole thread:

> Furthermore, no matter what, poor technical ability seems highly correlated with poor communication ability – regardless of language, it’s relatively rare for candidates to perform well technically but not effectively communicate what they’re doing (or vice versa).

[meheleventyone]

Right but that would also be true and exactly what you'd expect if interviewer mainly judged technical competency on ability to communicate. Since the form is self reported it's impossible to separate out whether the correlation is causal (poor communicators are poor technically) or due to interviewer bias (poor communicators have a harder time convincing interviewers of their technical chops) or even other factors (being flustered hurts both a candidates communication and technical skills).

This is after all an advertorial piece of data spelunking not a study.

[sulam]

I'd suggest making it more than 10 lines of code. Similar to kids finding easter eggs -- you want a reasonable number of things out there to talk about. It sounds like you try to get there by keeping it simple, but I would posit that some people's brains may lock up on a single block of code and having it in context or more there gives them something else to look at. Our brains are very bad at having insights when focused on the thing you're trying to have an insight about.

[hinkley]

I bet you could go farther and make a really bad piece of code and introduce it as such, and then have them talk about just how many things are bad about it.

[Jemmeh]

Just curious, if you've weeded them out at around 5 mins, why does it take 45-60 mins?

Also, could you possibly just do an initial phone/Skype interview to save time in that case?

[irrational]

I've been in this situation. Often we know within a few minutes whether the candidate is a good fit for our team, but for legal reasons we have to stretch out the interview so that the candidate can't make a claim against us. It would be better for both of us if we could cut them loose quickly, but we have to cover our butts because that's the kind of litigious society we live in.

And their is an initial phone interview, but it is done by HR or our boss, neither of whom are technically proficient.

[fwefwwfe]

Isn't that why a lot of companies do tech phone screens?

[Jemmeh]

If it's because of anti-discrimination laws, can a technical phone screen be short? You could discriminate based on voice too so I'd think that'd still be an issue.

[paulddraper]

I think the advantage is that the phone screening is much shorter than an hour, for everyone.

And "you haven't passed this interview" is less suspect than "I ended this interview early"

[kafkaesq]

It's been immensely successful at weeding out candidates within 5 minutes

Indeed, it's amazing how quickly people reveal themselves in this way.

No whiteboards, no quiz shows, no grilling of any kind needed. Just: "tell me your story".

[dando]

When you put people under the pressure of not having a job, a time limit and worst, the pressure of having someone looking over you, bugging you, and judging you, it can be incredibly hard to think straight or say anything at all.

You can act buddy all you want, but people aren't stupid. They know if they take a few extra seconds too long, you're already thinking "this guy sucks"

[vellum]

Each moment of silence makes the interviewee feel more tense. Instead of staring at them, why not leave the room while they read through the code and collect their thoughts?

You could even have HR do it. When they arrive, HR hands them the code and instructions. Then, they read it for 15 minutes, until you walk in.

[avip]

I'm absolutely not an "introvert" of any sort, but if you give me something I've never seen and ask me to reason about it, you'd better take a very wide step back, stfu, and wait for 10m - if you're to get any results from my slow brain.

[notyourloops]

That's great! I really hope interviews like that become more common.

[Jemmeh]

Something I've learned from making Youtube tutorials is that you need to have a deeper knowledge to actually teach something. The point at which I study a topic and have it "click in my head" comes way before I can actually manage to "explain it back clearly". As I'm writing the script for the tutorial I often realize there's a gap in my understanding that I have to go back and work on.

The old saying "If you can't explain it simply, you don't understand it well enough" applies here I think. :)

[benjaminjackman]

That has been my experience as well, one level abstracted. Back when I was a position when I last had to organize teaching employees technical domain-specific skills, the best way we found for them to learn was for them to teach others, this was accomplished by having one generation of employees train the next and then reviewing the newly trained employees.

Then for the next round of hiring, advance the employees up the education pyramid. The trainees become trainers, and the previous trainers mentor the new trainers by reviewing the newly trained employees.[1] It was one of the things I think we did we really well and that I am really proud of (even though the business went belly-up).

When training is done is this pattern, hopefully a virtuous cycle of education is established throughout your organization.

1: Note that this was at a quickly growing company, with recruits that had little to no relevant actual know-how to do what we did.

[enobrev]

I occasionally wonder if something like this would significantly improve our education systems, where part of our schooling includes tutoring / teaching someone from the next year.

[11thEarlOfMar]

We do it in reverse. In the first 2 hour interview, candidates learn our tech from 2 or 3 engineers. We then send them home with a set of coding tasks that are all relevant to what we do. As in functions of utilities that we actually would need to write, as opposed to puzzles. They choose one of 6 tasks, write ~100 lines of code, then come back for the next 2 hour interview and lead a code review of the code they wrote. This approach allows them to take the same time and care they would if working for us, rather than putting them in an unnatural state of coding at a white board in front of one to n scowling veterans.

Engineers who are capable can clearly explain their coding choices and their style is evident. If they've borrowed too liberally or had someone else write it for them, it becomes apparent pretty quickly because they can't clearly explain the code they supposedly wrote.

This is a fairly new approach for us, but so far, the candidates have appreciated it and the team reports that this approach gives them a good sense of the candidates ability.

[lj3]

So, based on the new information presented in the article that people who can talk clearly about code can also write code, are you likely to change your interview practices to omit the take home task and ask the candidate to talk about code that somebody else wrote? It would be faster, easier and according to the data in TFA just as accurate, yes?

[11thEarlOfMar]

Do you have data that shows how fast and easy the take home test approach is so I can compare? I only have anecdotal data.

I do give candidates the option of the take home vs. white boarding. So far (it's admittedly still a pretty small sample) they've all opted for the take home. However, we'll take a look at this approach as well.

[lj3]

> Do you have data that shows how fast and easy the take home test approach is so I can compare? I only have anecdotal data.

I'm not aware of any solid data around take home tests. All I can give is yet more anecdotal data, which is that they take a lot of time on both ends. So much time, in fact, that I've had several interviews where the interviewer never looked at the test code I was required to send in. Whether they work or not is irrelevant if nobody ever looks at the code.

[11thEarlOfMar]

I see. The take home assignment is not a test or exam, per se. It's code sample that responds to one of 6 coding tasks. We expect something around ~100 lines, typically in C. The candidate leads a code review of their code by the team when they return for the second 2 hour interview. The code review usually takes about 45 minutes. The remaining 1:15 is 1-1 interviews. So we can assure candidates that someone reviews the take home assignment as they lead that review.

[cakedoggie]

Who doesn't love homework, especially as an adult.

No thanks.

[dando]

Is it really that bad? An onsite interview takes the full day anyway.

[cakedoggie]

Yes, it is. What other job asks you to do homework outside of programming?

[soulnothing]

This was the style of interview I ran. I usually picked snippets from an open source project. I usually grabbed pull requests that fixed bugs.

Have them delve through it. Reason aloud, ask questions about it. See how they gain context, and how they troubleshoot. The biggest down side was they sometimes need to reference external projects, or documentation. I would often have to search on my work laptop and then let them should surf the docs.

[desdiv]

When I'm nervous, I don't communicate well.

When I'm nervous, I don't code well.

I don't know about other people, but for me personally, when you measure me under a high-stress situation, all you are measuring is just my level of nervousness.

[poikniok]

I must ask, did you get a university degree and if so how? Personally I have found interviews to be trivial compared to exams in school, so while I do get quite nervous for some of my math examinations I have never had this experience in interviews.

[desdiv]

I have a CS degree from a first-tier school. My experience with exams and interviews is almost the exact opposite of yours.

In exams I don't get as nervous because I know that if I know the material then I will pass, if I don't then I won't. You can't bullshit your way past the professor/TA grading your exam.

But in interviews I know that people who are far better than me have failed[0], and that people less competent than me could get the job. So it's not my coding ability or my experience that's being measured here, it's something else. And I don't know whether I have that something else or not. Maybe I do, maybe I just don't.

[0] https://twitter.com/mxcl/status/608682016205344768?lang=en

[javabean22]

I feel exactly the same way. The good thing is you keep doing the interviews for some time at some point you run into normal people and then it's easy.

[b0rsuk]

Last but not least, after you fail an exam you can usually see it with errors underlined and often have the teacher explain them for you. If you answer interview questions in a suboptimal way, you get no feedback. You don't know what was the error you made, if any. Maybe another candidate simply read about the logic puzzle that was used.

The canned response I got was "We're sorry, but we can't currently recruit you. Please try again in half a year or so, maybe we will have some new opportunities." If you press on, you just get evasive answers.

[Sacho]

Hmm sure, but companies that don't give canned responses are liable for lawsuits. A teacher's explanation can be wrong or misleading. I've also had some of teachers refuse to explain the alleged errors in my work, and I've even had to argue with teachers that they are wrong and I'm right.

I suppose if you went to a great school and then to some shitty interviews, you'd like the school more than the interviews, and vice versa. The difference however is that a teacher is only motivated by their altruism to help you, whereas a company has actual financial motivation to find good people during the interview.

[b0rsuk]

You're justifying company behavior, but my post was in response to the one who wondered how could a person pass exams but fail job interviews. I pointed out there are valid reasons that are not the candidate's fault.

I'm in the profession to write and debug computer programs, not debug people who like to keep me in the dark. This is frustrating.

[javabean22]

That's A Bingo. That's how false negatives are made.

[jghn]

I used to take "relatively rare" a step further and viewed it as a myth. I'd get into debates with coworkers that all one needs to do is have a conversation with a candidate to fully judge their abilities. No need for coding tests or the typical questions. Unfortunately after we brought in a ton of candidates there were a few people who fell into this bucket. I have no idea how it happened but they couldn't code themselves out of a wet paper bag when it came time to do so even they kicked ass in technical conversations. Sadly it became a bit of an "I told you so" moment where the outliers are made up to be more frequent than they actually are.

I like your idea though.

[MichaelGG]

Had a similar experience with a remote sysadmin. Interviewing, he seemed to understand everything. He explained how he'd setup networks, details on using SSH for bastion hosts, etc. Seemed to really know stuff.

After hiring him, he was unable to even SSH to a box. As in he didn't know how to give us his private key, or how to configure his SSH client.

Maybe one guy takes interviews under other people's names then lets them mess up the work.

[troels]

OTOH if he had given you his private key, surely he would have failed the sysadm test right there.

[DoofusOfDeath]

That happened to me once, during a phone interview with Google.

My brain simply froze during the live coding test. I looked like an idiot, even though the coding challenge was one I could have normally handled in my sleep.

My guess is that it was due to anxiety about doing well on the interview. But to this day I'm gun-shy regarding live coding tests.

[meggar]

I did that at a Google interview too, on a variation of a question I'd practiced a few days before no less.

[cookiecaper]

It shouldn't take much demonstration to see that they are basically competent, and there shouldn't really be a difference between asking them to loosely state the commands they would issue and the process they would take and watching them do it.

People normally perform better by verbalizing the process than they would by sitting at their computer and performing it, since a lot of the concern in an interview can be whether you're getting your code/commands syntax-perfect or not.

The only way I can really see this going badly would be if the interviewer was only asking generalized questions that someone like a non-coder HN reader would be able to pick up from the zeitgeist without actually having ever implemented. Instead of asking about trends or fads, it's probably better to ask detailed questions about implementation processes.

If they really have internalized all the applicable concepts but can't express them in computer language, well, it shouldn't be hard for them to get up to speed on the syntactical fineries. Make sure you're testing for grokkiness of the core concepts needed, not how well they're following the trends or the news.

Personally I use a very small code test and put the rest of the energy into a detailed conversation.

[kasey_junk]

Its fairly easy to see how this correlation breaks down. It is almost a truism that someone who is good at communicating with a wide range of people, is likely to make that wide range of people think they are more technically competent than people who aren't good at communicating with a wide range of people. Especially given the virtually random nature who you will communicate with during any given interview cycle.

But, its quite possible for people who are not naturally able to communicate with a wide range of people to be able to communicate with a smaller subset (say people they are more comfortable with, or people working on similar problems to them).

The more interesting question is, could those people who communicate less broadly, in a situation where they do communicate well, perform technically at the same level (or better) as people who are more natural communicators?

Another way to say this is, evaluating interviews is fairly prone to bias towards people who interview well. As someone who is hiring I'd be much more enthused about their results if it correlated to job performance, not hiring results. Its almost gospel at this point that there are people in software who interview well who do not make good hires, but the methodology of this article doesn't talk about this at all.

All told, nothing about this research is persuasive to me on my opinion of "interviews are largely worthless in determining who will be a good hire".

[vonmoltke]

My previous company did this. I have seen three variants over my time: explain what this code does; explain why this code is wrong and how to fix it; explain why this code is inefficient and how to optimize it.

[DoofusOfDeath]

> explain why this code is inefficient and how to optimize it.

That sounds like a trick question. I've done a lot of code optimization. Here are a few generalizations I've drawn:

(a) You rarely know what part of a program is the bottleneck until you profile it on an appropriate workload.

(b) You often don't know why it's slow until you dig into performance metrics like cache-miss ratios, branch-mispredicts, etc. Often at the disassembly level.

[igorbark]

But for a short, isolated piece of code I assume you would be able to suggest a handful of plausible inefficiencies and explain how they could be tested for and optimized for.

[mmcnl]

Well that would be a perfect answer if you ask me.

[bluGill]

No, it would not be a perfect answer, and could work against you: it shows you are not willing to work with me and assume some intelligence on my part (until proven otherwise) and thus you are potentially not going to work well with the rest of the team. I would respond that we have already profiled this code and discovered it is a bottleneck, now answer the question. It won't be hard to overcome the negative impression, but it left a negative that you don't want.

Now if you responded in the form "Of course in the real world we don't look at code until after a profiler shows the need. I see X which is suspicious, let me look deeper". In this case the comment is perfect: it shows you trust, but know to verify that trust. It also has the side effect of giving you time to think.

[mmcnl]

I fail to see why analyzing a problem before solving it somehow translates into "not going to work well with the team". Problem analysis skills are far more valuable than problem solving skills. Anyone can search on Google for a solution to a given problem, but no one on Google can tell you what the actual problem is. If you really thing the answer I was responding to is not acceptable, then the question is a lousy question imho.

"It doesn't make sense to hire smart people and tell them what to do; we hire smart people so they can tell us what to do." - S. Jobs

[bluGill]

Remember, this is an interview situation, I can't give you a realistic 500000 line program with a slow hot spot and ask you to fix: we only have a few minutes (maybe an hour). I have simplified the problem to a single function that is a bottleneck, I want you to analyze this function so I can see how well you can analyze things.

Refusing to do something because it hasn't been profiled is not analysis, it is refusing to do analysis. You have been given a problem and been asked to analyze it. That refusal makes you not a team player.

Once again, this is an interview. Noting that this scenario (a single function that is known to be too slow for reasons that can be fixed in 10 minutes) never happens in the real world is good - it shows you understand that real world analysis includes elements not included here. Stating you won't do this because you don't have profile data is not being a team player.

[dando]

>No, it would not be a perfect answer, and could work against you: it shows you are not willing to work with me and assume some intelligence on my part (until proven otherwise) and thus you are potentially not going to work well with the rest of the team

I don't know how what was said implies this in anyway.

This is one of the biggest problems with interviews. Every interviewer has some dorky idea that x means y is a terrible developer while another interviewer thinks x means y is a great developer.

"I want an opportunity to learn" - Oh great. We want someone whose interested in learning more and isn't cocky.

"I want an opportunity to learn" - Oh no, that just tells me you don't know much and aren't interested in contributing

[cookiecaper]

Yeah, that's why interviews are nerve-wracking. At their core, they're very subjective. The interviewee doesn't know if the question is supposed to be a trick to see if they say "Well, of course I couldn't optimize it, because I haven't seen the performance data yet!" or if they're supposed to suspend disbelief for purposes of the exercise and say "Well, of course it'd be best to profile, but we could change the way it does X...".

The chance that the interviewer expects some weird thing like that is random, so there's no way to know whether you're getting it "right" or not. Select the wrong answer and you get viewed as an ignoramus or a smartass. Just have to take the gamble you like the most and see if it aligns with the interviewer's preferences.

Once I was asked what kind of software publications I like to read. As a habitual HN user, this should've been the easiest question ever. However, when asked, I guess I was feeling strict that day and said "Well, I don't really read a lot of software-specific stuff, it tends to the more businessey side of things rather than being strictly software related", thinking of highly technical blogs like Lambda the Ultimate. The interviewer said "What about things like Joel on Software?" and I said "Yeah, I've read most of Joel's stuff, but I don't really consider it very 'softwarey'".

I could tell that cost me the interview, but what can you do? Someone else in a strict mood that day may've been delighted by such an answer, happy that I didn't hold delusions and could tell the difference between comedic rants about the business of managing projects and hiring developers and serious, borderline-academic publications that include equations and ponder theoretical dilemmas.

The best thing is just to take the whole process casually without getting over-committed or holding a grudge. False negatives cost companies much less than false positives and they're evaluating you in the context of their other applicants, who are also essentially random, so there's no way to really know anything. It's just about whomever seems least risky and most useful, and as an interviewee, there's no real way to know what that means from the POV of the interviewer, so it's hard to optimize your responses.

The real way to win interviews is to recognize them not as technical processes, but social processes. Admitting this makes engineers nervous, but it's important they understand that humans, not compilers, will be performing the evaluation, and they must learn to speak human to get satisfactory results.

[b0rsuk]

I think "find the bug in this code piece" questions are dumb! When you program, you don't get a helpful fairy (or Clippy for you Windows users) telling you the code is wrong and needs fixing. The bug manifests itself in production or when you try to execute the code. Bad output, exceptions or segfaults, slow operation, infinite loop. Then you have a vague idea what to look for, or at least what kind of input causes errors. Furthermore, if you're the author you should know what the code is trying to do, and how. I believe programmers in teams are usually told to fix their bugs, not fix someone else's bug ?

I'd rather see a question like "Please modify this piece of code to add X functionality."

[lj3]

Those are my preferred types of interviews when I'm an interviewee and a technique I've used a few times on the other side of the table. It's so effective and efficient. I don't know why it isn't used more. Maybe it finally will be, now that there's some solid data supporting it.

[maxxxxx]

That's what I do. If I can have a meaningful conversation with someone about stuff he did or stuff I am describing most likely they will do well. If the conversation gets very vague or confusing most likely things won't work out.

[jondubois]

Some companies just love fast-talking engineers because they sound like they know what they're doing.

Some engineers spend a lot of effort trying to project a specific image of themselves at the expense of bringing up the morale of others in their team.

On a few occasions, I've seen an engineer tear apart the work of another engineer at a team meeting in order to make themselves look good and more senior in front of their boss - Often, the critique is delivered with such vigour that it sounds rehearsed. I've seen this behaviour in startups mostly.

[cr0sh]

I've never experienced an interview like this, but it sounds like it would be both challenging and fun - especially if it was done in a language (or pseudo-language) that you'd never seen before (then again, they could lob BF at you or something eso like that!).

Based on other responses to your post, it's heartening to see that others do this kind of interview, so maybe in the future I might encounter it.

[amalcon]

The version of this where that code is clearly unsuitable for its stated purpose is my favorite coding interview question.

The unsuitability should be extremely obvious, no "spot the missing semicolon". Maybe it has a severe performance problem, maybe an obvious security flaw, maybe a logic error. The point is to prompt a discussion of better alternative ways to do what the code is trying to do.

[qntmfred]

this is also a prominent technique I've used in the hundreds of interviews I've conducted.

[graphememes]

I do this frequently and I utilize their resume to power the code.

Someone says "I am a 9/10 in JS" I give them something that uses the animation frame api, the dom api, and a few more javascript APIs that I personally am not fully comfortable saying that I am a 9/10 on (been working with JS for over ten years) and see how they decipher what it is doing, what jumps out, and how they would review it / give feedback.

9/10 times they don't actually understand Javascript.