[onislandtime]

Dear software professional: if you have been rejected because of a coding interview, don't feel bad or discouraged. It has little to do with how smart you are.

Unfortunately, this style of interviews is likely ineffective and leads to hiring people who look alike and have similar skills. Solving a problem with someone looking over your shoulder and forcing you to talk to explain what you are thinking is a skill that I've never seen used in the real world. I'm sure new grads spend a lot of time in classes training for this. Many great people don't function like this and still they may come up with brilliant ideas after a day or a week. Some people have breath of knowledge and study specific topics as needed. Some people can write very well and may not be super fast in tests. I know many brilliant engineers who have been rejected and are doing just fine, building amazing products, and leading teams.

If corporations really wanted a cookie cutter method to evaluate CS knowledge, then they should require a scientifically validated standardized test conducted by a third party. It would be cheaper than using engineers' time. So why don't they do that?

The reality is that they think they are doing more than that but there is no scientific proof that the interview method works. They don't want false positives but they cannot measure efficacy. If you are one of the guys who know how to perform, then you can get hired faster. In some cases if you are an outsider (older, female, different), then your chances of knowing the "secret interview code" is much lower.

[robbyking]

The analogy I use is a coding interview is like asking a musician to play a specific song; chances are, a classically trained pianist won't know the chords to a specific pop song, but that isn't any indication of their skill as a musician.

The interviews I had with the company I work for now were amazing; they asked me some basic questions to verify my resume wasn't completely BS, then asked me to discuss previous projects I'd worked on, asking questions about technical details on the way. ("Why did you use collection type X in stead of collection type Y?") This allowed them to learn about my real world experience without the risk of asking me about one specific type of problem I may not be familiar with.

[JPKab]

I recently was flown to Austin for an interview. The entire reason I was brought down was because of Python ML projects on Github. The technical interview consisted of nothing, and I mean NOTHING but super advanced SQL questions. (over the phone, I had specifically stated I hadn't used SQL in years) My expectation was that I'd be interviewed using Python and asked to perform ML related tasks.

When I mentioned that I thought that the purpose of me being brought there was to add an ML dimension to the team, which was far too data analysis (using SQL and some R)focused and with no Python expertise, I was given a blank stare.

Then it hit me: the guys interviewing me didn't know how to do any of the stuff I had been brought down (by higher ups who weren't in the room) to do, so they weren't evaluating me on it. They evaluated me on what THEY knew. It's the equivalent of Peyton Manning being asked to evaluate a linebacker, and demanding that the linebacker throw passes downfield.

The highlight was when one of the guys (typical pony-tail neck-beard type) pointed out that an alluvial flow diagram in my portfolio of data visualization projects wasn't "Tufte-esque". (It was a gross misinterpretation of Edward Tufte's commandments on his part, but am I really going to get in an argument with the guy interviewing me?)

It was clear that the guys in the room wanted someone who knew what they did and thought like they did. A brilliant recipe for getting a homogenous team with no diversity in skills.

What an epic fucking waste of time. The best part? The company only has 150 employees, and a recruiter just contacted me for a position on another team that uses Python. She was unaware of the fact that I was there a month ago. I told her that they should have thought about that when they flew me down the first fucking time.

[raverbashing]

> I told her that they should have thought about that when they flew me down the first fucking time.

Thanks for that!

On the other hand, most likely the other Python team would grind you on Python and again nothing on ML, and of course your Python ML projects suck because "blah blah blah"

[pooper]

Maybe if I am in a similar position, I should demand compensation for my time. That way I'll at least get to visit friends and see scenery around Austin. I wonder if a demand for compensation (besides airfare and stuff) would fly...

[loumf]

It would not. The only thing I could imagine countering is: Fly me to Austin for two weeks, put me up in a hotel, and hire me to do a project at $x/hr. If we both like what we see, let's talk about a full-time position.

Frankly, I think this is anti-candidate behavior if the employer suggests it, but if you actually want it, they should be willing to do it if they can vet you otherwise (github, etc).

[eru]

I usually just stay longer on my own dime. The company doesn't care whether they pay for flying you back immediately or two weeks later.

[jedanbik]

Ugh, what a wasteful mess. The worst part could be my lack of surprise in reading your anecdote.

[smm2000]

Better analogy is that classical musician is given sheet music to pop song and is told to play it on his favorite instrument. If he is good musician, he should be able to play it reasonably well even without practicing. There are musician who play extremely well but only after practicing one song for a long time - this type of interview sucks for them but they are minority. Most musicians suck with or without practicing so this type of interview is a good filter. Not ideal but no one came up with better one that scales up to thousands of applicants.

[banjomonster]

In an audition, musicians are usually asked to play a mix of prepared and sight-read pieces (though sometimes the sight reading is from standard repertoire, so they may have encountered before). Both are helpful in discerning how qualified the musician is. And sight-reading is incredibly important for most jobs musicians get hired for - in most freelance situations, they'd get a 3 hour rehearsal, or essentially a guided run through of the performance, then they play the gig. So that may not be an accurate analogy, given that writing on the fly isn't part of the programming repertoire - most programmers are allowed to think before they write anything down. Also musician's don't write - performing is very different than writing- so perhaps a composer would be a better analogy.

Maybe it would be a more accurate measure of ability to ask software engineering applicants to read code and explain what it does, or to judge them as you'd judge a composer, by their portfolio.

[eru]

That would be an interesting way to apply something like re-captcha: ask them to fix two bugs in open source applications. One bug that you've actually already fixed, so you know how hard it should be and can calibrate; the second bug is open.

[nileshtrivedi]

How about just asking them to code something - anything - in say, a couple of hours?

[eru]

To make this useful for an interview, you have to set it up so that people can't submit things they've coded up before in a much larger timeslot.

[smikhanov]

Not all codes are created equal.

[sgustard]

Agreed; the idea of hiring a musician without hearing them play would be madness. Not handing me a recording, not discussing intricacies of music theory, but actually playing live. And yet, a talented studio musician may suffer from performance anxiety in this situation. What to do?

[kilbasa]

I think the best thing is to ask them to warm up with something basic like a scale, and let them work their way up as their comfort level increases. I knew some people who were particularly pleasant and easy going, so they were able to relax people suffering from performance anxiety.

There are interviewers who treat coding interviews as a high pressure, adversarial exercise, though. On the other hand, those sorts of people are rarely a joy to work with.

[me2i81]

Performing a piece of music in a high-pressure situation is exactly a professional musician's job, so an audition is quite appropriate. Another analogy could be to ask a professional composer to play one of his pieces on piano rather than listening to a competent orchestra perform it--you're using one thing as a proxy for another. Many composers would ace it--they're great players. Some aren't, though, so you'll hire only composers who happen to be great performers. It's a dumb analogy in another way, though: you only need to know whether you can work with a composer, their work should stand on its own. I suppose if we engineering types all wrote code that went into public repositories the same might be true of us, and we wouldn't be asked to perform like trained monkeys.

[woah]

In a lot of cases, it's like a classical musician being asked to make a violin.

[akanet]

"Solving a problem with someone looking over your shoulder and forcing you to talk to explain what you are thinking" is actually how problems are generally solved in most fields. Software engineering is perhaps one of the lone exceptions, and I think the pair programming movement has a bone or two to pick with your premise.

[lazerwalker]

There's a big difference between someone looking over your shoulder and forcing you to explain what you're thinking, and someone working with you to collaboratively solve problems. In the latter case, you can ask lots of questions and throw out lots of dumb half-baked ideas, and actually get a meaningful, useful response from your partner/pair in response. If you're whiteboard interviewing, good luck getting your interviewer to meaningfully engage with you, since they don't want to "give away the answer".

The best interviews I've had were legitimate pairing interviews, where the two of us were tasked with solving a problem in a real codebase that the interviewer hadn't solved before.

(The best best interviews made it an issue in an open source project, so it can be a not-canned interview without you writing potentially production-ready code for a for-profit company without being paid)

[cnp]

THIS right here is the dream interviewing scenario and in my opinion the singularly best way to test a candidates competence, aptitude and personality. Solving problems together exercises all of these key things in harmony.

[re_todd]

There's also the fact that some people, like me, freeze up in these interview scenarios. I didn't get past my first interview a few weeks ago because I couldn't think straight knowing I was being judged going through this process. As soon as the interview was over and I was more relaxed, I thought of several good answers, like I normally do when employed.

[onislandtime]

People are very diverse, some are fast and chatty, others are quiet and slow. Usually I perceive the fast and chatty types a bit arrogant. Personality and cultural background is probably an influence. Best teams have a great mix. Often caring about the work is more important than having a genius. (Of course people should be competent).

[dreamweapon]

Another aspect of the funhouse of one-way mirrors that is the modern conventional tech interview process:

Candidate is fast and chatty: "I don't know. He seems smart, but he was also talking a lot of fluff. Kind of pretentious and arrogant."

Candidate is slow and hesitant (and perhaps pauses awkwardly now and then): "I don't know. He seems smart, but he seems a bit underconfident. We need someone with more energy."

[FLUX-YOU]

See, we're having trouble even testing for competency. I passed a pen and paper interview recently (did confidently well on coding, blew the SQL part because I had only used APIs and still thought I should put that on the resume) and still got the position. I'm still scared shitless that I'm going to flub day 1 and I've been cramming and exercising basic knowledge for the past two weeks. I'm worried that the test was just a screen and the real requirements are hidden.

On the other hand, everyone's favorite anecdote: "senior engineers" not passing basic tests like mine.

I still don't feel qualified for entry level (and other peoples' interview experiences don't help!) and that's because entry level and other competency levels are subjective right now. What knowledge should you know from memory? What knowledge are you safe delegating to Google to remember? However, my feelings about my skill could just be nerves.

Lines in the sand would really help the self taught crowd and could be used to bring outdated, lagging CS programs and their graduates, up to speed.

[sbov]

Different people are going to have a different answer to your question, but here's how I approach things:

You should commit things to memory the things your memory naturally keeps. If it doesn't it means you aren't using it often enough to bother remembering it. There's so much information I used to keep re-learning and forgetting because I never used it. I finally realized that it was a waste of time.

I guess the problem is when the job you're interviewing for will need vastly (or slightly) different things committed to memory than what your current job does.

I also like something between google and memory: cheatsheets. I have a dev notebook full of checklists and cheatsheets for various technologies I've used and tasks I've done.

[UniZero]

As a self-taught developer, I have reached the same conclusion regarding memorization. I find your concept of 'cheatsheets' inspiring, and would love to know more about how you organize that kind of information.

[sbov]

I used to just use text files. Now I stick everything in Evernote. Text files are good enough though.

Organization is pretty simple. Just need a decently named file with the relevant information. E.g. I would have an sql.txt file containing sql database operations that I don't use often enough to remember, but use often enough to be annoyed by having to search the web on how to do them.

Another example might be a centos5.txt file, giving me a quick rundown of the various setup options I've wanted in the past, and the location of various configuration files.

For programming languages, there's many that I use maybe once every 3 or 4 months. So ruby.txt might contain a quick rundown of how to do basic things: how to define a function, a class, perform loops, instantiate an object, access command line arguments, etc. I find it much better than having to hunt down a tutorial which will also be more wordy than needed.

I also have some for more computer science related things, such as tables of graph and search algorithms, along with their tradeoffs.

Checklists are another good thing. I have checklists for processes I've messed up.

E.g. committing new code. If I don't use my checklist, I always seem forget to forget to update a sample file. Or add a file.

Another big checklist is for giving estimates. For example, one of our legacy products is translated into 5 languages. This checklist reminds me to think of translations, because they have been a huge bottleneck in the past.

Hope that helps some.

[girvo]

Everything you've said is exactly what I do, and it works brilliantly for me, despite being self-taught :)

[mahyarm]

I do coding interviews exclusively since I find whiteboard interviews even more artificial. I try to match the interview to a persons skill set and ask them to implement things they should know or know how to look up if their resume isn't a lie. I don't care if they talk or not while doing the interview and I take the role of a product designer who doesn't know how to code. If they get interview anxiety I give them the space they need. If they want a keyboard & mouse I would give it to them.

What else can you do that would be an accurate simulation of their job that only takes 1hr?

[dreamweapon]

What else can you do that would be an accurate simulation of their job that only takes 1hr?

First 1/2 hour: Discuss their code samples in detail -- asking pertinent questions about each as to what they do, and why they did things they way they did. Drill down for detail, ask how they might have done things differently for different use cases, etc.

Second 1/2 hour: Bring out some of your own production code and do the same (allowing them space to ask questions this time). Works best if you let them see something that's a bit "raw", i.e. quick and dirty, which you know you could have done a lot better if you had more time or knew better about the requirements (or perhaps you're simply older and wiser now).

In both sessions, nuance (both in what they can observe, and how they chose to express themselves) is key. And it'll come out a lot more freely than in in a whiteboard interrogation precisely because they interaction will be natural, uncontrived and unforced.

Simple, non-confrontational and 100% reflective of what engineering work is like. That's what we do during the day, 99% of the time -- digging up (sometimes woefully) less-than-perfect components of production systems, and trying to make them a little bit better.

But as to how often we drag someone to a conference room, throw them at a smudgey whiteboard with creaky pens and no eraser, and force them to solve some abstract problems while we boredly look at our watch, and interrupt them with hints?

Basically never. Except, that is, in coding interviews.

[onislandtime]

I think that what you do makes sense if you take the time to read the resume, code samples, articles, and ask relevant questions. Asking to write some code is perfectly fine to see how the person works. References are also very important for more experienced candidates with a track record.

[yarou]

They don't want to do this because the secret to making money in IT is creating a glut in the supply of labor for IT workers. When you add licensing (i.e. requiring a license as a doctor or a lawyer does), this creates a barrier to entry, which will cause an upward pressure on wages for IT workers.

[onislandtime]

Good point. Perhaps we need to work through professional organization to look into hiring practices and propose practical solutions.

[leothekim]

"Solving a problem with someone looking over your shoulder and forcing you to talk to explain what you are thinking is a skill that I've never seen used in the real world."

I'm often in the position to have to explain my reasoning while developing software. Code reviews and pair-programming come to mind. It's about being able to communicate complexity and being more rigorous about the software development process.

That said, this style of coding interview brings up a level of stress that I'm rarely under at work. It's an unfortunate condition of interviewing, but I don't think it's completely avoidable no matter how comfortable you make the interviewee.

[justinpaulson]

The point of these type of interview questions is to see how well you can reason and communicate. It is not to see if you know how to solve that specific problem, and if an interviewer is using it like that then they are not a very good interviewer. A standardized test does not substitute for actually talking to someone while they solve a problem and seeing that they can think logically and communicate those thoughts.

It is more about the communication aspect of the question, not about getting to the answer. Because communication is a lot more important in business...even tech business...than you seem to think it is.

[maroonblazer]

Right. While I wasn't interviewing for a technical position, the interviewer asked me to 'vote off' one of the 50 U.S. states, and to describe my reasoning why (perhaps this is a common question but I hadn't encountered it before). It was basically an exercise in thinking out loud, with a lot of "well, I'd probably want to first consider x, but actually before doing that I'd want to take into account y...". There wasn't any undue pressure, since there was clearly no right answer, so it allowed me to just riff out loud on the problem, exposing to the interviewer how I approach problems.

I use this same technique now when interviewing others.

[thecage411]

The answer is florida, right?

[maroonblazer]

Idaho.

[eli_gottlieb]

Of course it's total bullshit. You still have to pass it. You can't demand that everyone else be at least as wise as you, especially since you could well still be wrong.

[benihana]

>forcing you to talk to explain what you are thinking is a skill that I've never seen used in the real world

You've never once explained your thought process to a coworker? Explained how you arrived at a conclusion? Tried to elucidate your reasoning on a piece of code to someone?

These are absolutely real world skills and they are absolutely applicable when working on an engineering team with other humans. I understand what you're getting at in your comment, but it really feels like you're throwing out the baby with the bathwater. Can interviews be improved? Absolutely. Does that mean that there is no value in these kinds of interviews? I certainly seem to glean some value out of them. Whether that is the right value is a hard question to answer.

But I want to address something. Your whole comment is about why these kinds of interviews are shit. But you offer no alternative methods, no ways to improve these kinds of interviews, and really no constructiveness. You seem so sure that interviewing this way is wrong, but you don't say what's right. I would love to hear some realistic ways to interview people that can help me find better candidates without rejecting people because they're bad under pressure.

[personZ]

Explaining your thought process to a coworker after you've written the code is ex post facto rationalization -- you are invariably leaving out many side-tracks, many missteps, and get to condense down a long process into a simple, seemingly obvious explanation. You already see the route, and now you're just describing it.

Trying to do the same while you're trying to solve the problem, with a judgmental crowd scoring your comments, however, is an absolutely and completely different matter. Even thinking about how I would narrate my thought process as writing this reply ("maybe I'll talk about how I'd narrate the writing of this reply") is confusing enough.

I don't disagree with the concept of technical interviews, especially given that there are countless people who hold none of the skills they claim they have mastered. A process I implemented requires developers to come in for a coding test, where they are equipped with a development machine with full internet connectivity and a full toolset, and given a problem within their skillset to solve, alone in a closed office and for as much times they need. After the test we do talk through their "thought process" and their implementation choices.

I know this offends some people, among whom I'm sure are the people who completely failed to demonstrate even a basic knowledge of skills they claimed an expert level of competency at.

[pekk]

As someone who is very annoyed about interview process in general, I'm not sure what you think offends some people about giving them a realistic environment and time to work. What is annoying is when a false equivalence is drawn between stress and difficulty on some ad hoc interviewing task and "even a basic knowledge of skills".

[personZ]

I'm speaking more to the viciously anti-technical-interview sentiment that is so commonly seen on technology sites. While there are a lot of shops that do the whole process in a ridiculous fashion, similarly there are a lot of frauds among us: we have experienced such a number of people who claimed skills that they didn't begin to have, even when warned well in advance that they were going to be tested, told how they would be tested, and given a full ability to leverage the internet. Whenever those interviews ended, awkwardly, I always imagine them heading to Reddit or wherever to tell everyone how technical interviews are so much bullshit, and can't people just hire them on their resume, etc.

[curun1r]

The problem with these questions is the time scale in which they expect the answers. When dealing with a difficult problem, most people don't immediately come up with the right answer and adding in the pressures of an interview only makes it worse. As a development manager, I always ensure that my direct reports know all about the issues they're going to be working on at least one week in advance. I've found that the solutions they come up with are markedly better when they've had a full week of thinking time prior to writing any code.

The alternative interview technique that I like to use when hiring engineers is to send them the questions I'm going to ask a few days before the interview. I ask difficult questions, but there's no surprise. I don't really care if they talk about them with other engineers beforehand...there will always be follow-up questions that I ask to get them to defend their solution that will uncover those that haven't understood the problem fully.

[vishnugupta]

Being a person requires time to clearly think about a problem and gather thoughts I couldn't up-vote this enough. I just can't solve a problem when I know that someone is evaluating me right there and then. The way I go about it is to keep thinking about it at the back of my mind while I commute, eat, sleep etc., and then once in a while write pseudo-code or gather data to verify if a solution I'm pursuing works.

I was asked to come up with a recursive-like solution for puzzle like problem. I just froze during the interview. Later on kept thinking about it for couple of days and sure enough I was able to come up with three different solutions that used various techniques such as memoization and I felt really good about it.

I guess what I'm trying to say is it's really counterproductive to evaluate someone's abilities by subjecting them to time pressure.

[MachineElf]

I really love the arguments in the category "You criticize 'X' but you offer no real alternative, solution, etc..." as if you can only criticize something if you have the alternative solution at hand. This is not one of those questions with an easy answer.

"Boy you sure criticize those Nazis a lot for genocide, but you offer no _real_ solution." "Dude... stop criticizing those banks for their greediness when you have no solution". I realize these are rather hyperbolic examples, but you see where I'm getting at.

As for the OP, this does seems like really cool stuff, even if I don't really agree with "the interview" method myself because I don't do too well at them. I mean sure, there should be some on the spot questions to get sort of an outlook feel on the candidate, but that really isn't sufficient.

Why not give the the candidate a real world problem - maybe even related to something your company is trying to solve - and a few days to come up with a solution. I don't care if they also do some copy/paste from SO and other sources on the internet. That's why the interwebz is there in the first place. What would be more important in my view, is how the candidate is able to integrate all the information he finds - be it on the internet, books or his own head - and converge to a solution. And even if he doesn't come up with a working solution, you as an interviewer can now infer a lot more useful pieces of information than with a basic interview. You can ask questions like: 'why did you choose that library over the other ones?', 'why that programming language over the others', 'how did you come across the code on Stack Overflow; is it correct?', 'why that database and not that other one?', 'why is your solution single threaded and event based, rather than multi-threaded?', 'how would you further optimize that piece of code?' and so on.

I think this kind of conversation is something much more likely to happen day to day between the candidate and his coworkers if he does get the job.