[evanjacobs]

In my 10 years at Amazon, I interviewed lots of engineers and I think Amazon's interview process is fatally flawed. Of course Amazon is still able to hire good candidates but I believe it is in spite of this process and not because of it.

Here were the most frustrating aspects of interviewing for me:

1. Bar raisers don't ask questions that they themselves don't know the answer to. I believe this is what Roseman is referring to (as a desirable?!? attribute) when he says "One of the things that really pisses me off is people asking questions that they don’t even have a good handle on themselves". What happens in practice is that you miss hiring people who may not be able to answer a question to which every interviewer already knows the answer but whom might be uniquely qualified in some other area.

2. Interviewers still ask candidates to code answers to theoretical questions on the whiteboard. There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform.

3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript.

4. This isn't really part of the Amazon interview process but Amazon's job descriptions are (for the most part) written so generically and absent of personality that I think they miss out on lots of great candidates for whom startups simply seem more fun & interesting.

[bane]

I remember my own interview process with Amazon. I wasn't interviewing for an SDE position but a TPM role. I made it through the gauntlet and it all fell down due to Amazon's grossly unrealistic understanding of what the going rates for experienced PMs are.

I mostly had fun with it, but I was also frustrated.

The phone screens were the hardest in a purely technical sense, lots of algorithm questions that really didn't have anything to do with the job at all, I had to reach into my way-back machine to dig up some of the runtimes. But I made it through three of phone calls like that and had a bit of fun. After the phone calls and a chat with a senior person in the department, I was pretty sure I had a feel for where I was going to end up and what kind of projects I'd be managing. So I boned up on management theory and algorithms and a few other odds and ends.

Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-to-face interview was wildly unfocused, I was interviewing in subjects the interviewer often didn't have any experience in and completely didn't reflect the nature of the phone interviews or the statements I had been given by the senior guy I spoke with. In one interview I'm designing Java APIs for movie tickets, in another I'm specifying contingency plans for overseas warehouse fulfillment issues. In another I was asked by two very junior people who frowned in confusion the entire time as I discussed complex employee and co-worker conflict resolution cases I've experienced over the last 18 years. In fact I was never asked a question about the first 15 years of my work experience, most of which would have been more relevant. But I was asked tons of questions about my side-startup, which was just me and my wife hacking around on the side and getting passive ad revenue and had no relevancy at all to the job.

The "lunch chat" was with somebody who couldn't have made small talk if he had a gun to his head. He just sat in silence and ate his soup the entire time while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've ever encountered.

I definitely didn't nail the face-to-face, but that's because I was playing trivial pursuit with the interviewers and the answers were arbitrary, instead of knuckling down on the job, the role, the department, the expectations of performance and the responsibilities of the position.

But I did something right because I was called in the next day to go over compensation and was blown away to find out the absolute max pay cap for the entire company was almost a 20% pay cut from what I'd been making in my previous job (a job that I was leaving because I was 10-15% under my peers and had been in a pay freeze for 3 years while the company worked towards profitability -- and yes it was from a metro area with a similar cost of living to Seattle).

Despite that, Amazon would have been good to work at, but even my absolute pay floor was 10% over. I even asked to make up the compensation difference with a mix of stocks or other methods, but eventually we had to agree that there was no way I could live the life I had become accustomed to with what Amazon was willing to pay.

[crucifiction]

The pay gap is probably because you did not pass the bar at the same "level" you were at in your old company, meaning Amazon was bringing you in at a demotion essentially.

[gavinpc]

> couldn't have made small talk if he had a gun to his head

To be fair, I think this describes most people.

[laichzeit0]

It describes most technical people who are also introverted and spent most of their lives with minimal face-to-face social interaction.

[James_Duval]

Most people in general. A lot of people think they're much better at small talk than they are, and blame the other guy when an interaction doesn't go as smoothly as they hoped, or there's an awkward silence.

People who are excellent at small talk have to be very charismatic and seem genuinely interested in the conversation even when it's very boring. Very few people are capable of either of these things, and in my experience it's actually management & marketing types who are the worst at both.

[gavinpc]

OT, but I just meant that one would be preoccupied with the gun to one's head.

[AsymetricCom]

My understanding is when they fly you out, you're pretty much a shoe-in and now they're letting their devs have a shot at trying to interview, while getting a feel how you work with others. i.e. are you sane or will you eat on of our disposable dev's face?

[crucifiction]

Its generally the opposite, the phone interviews are the sanity check.

[AsymetricCom]

"sanity check" in this context is a bit different than in software.

[tdenkinger]

This was not my experience. I did not get an offer after being flown out and spending the day interviewing. I did not do well in the whiteboard coding, but I found the people I interacted with to generally be interested and smart.

[iyulaev]

There's one thing this article misses, and one thing I totally disagree on.

Hiring for prior experience - the author emphasizes this, and based on my experience this should be de-weighted. The issue is that as soon as you start to define an area of experience, your hiring pool shrinks dramatically. Consequently you are more likely to pick a less powerful engineer. Another, bigger downside here is that companies are terrible at figuring out what they really need. Consequently you've hired someone who can do A and B but one month in you realize that the thing you were missing is really C. A less powerful engineer will have a harder time transitioning over to doing C work, and you'll be worse off. Moreover they might be upset - they came in with the expectation of doing A & B, and they're sitting here doing something unrelated instead. Can't count the number of times I've seen this happen, to both the employee's and company's detriment.

The thing that this article misses is how to avoid hiring jerks. Terrible employees can be a net negative to a company. Jerks, particularly those in senior positions, are the thermonuclear version of terrible employees. They can demotivate and destroy the productivity of entire teams and cause your best engineers to rage-quit. The best interviewing tactic I've seen to weed out these people is watching how someone behaves when you "play dumb" in an interview. See if they're happy to explain something to you, or if they get impatient with your lack of knowledge. A related trick is challenging them on something that is obviously correct - see if they carefully prove why their answer is correct, or become antagonistic. Really smart people are good, but much more effective are those that can help bring everyone else up to their level.

[pekk]

Being partly facetious here: I don't know how you actually propose not to hire jerks. The industry is full of jerks. Jerks are doing the interview much of the time. The industry wants people who claim to be the best, and that's hard to do while also selecting for niceness.

[wpietri]

Personally, I have a bias against subjects who claim to be the best. Some of the best people I've worked with, would say things like, "Oh, I guess I'm ok at that" and then blow me away. It's the Dunning-Kruger effect [1] in action.

Because of that, I generally avoid people to self-evaluate versus others. I think it's ok to ask them to compare on purely internal things (e.g., "What languages are you strongest in"); people seem pretty good at that.

I think that approach makes it a lot easier to hire nice people; niceness and humility are correlated.

[1] http://en.wikipedia.org/wiki/Dunning–Kruger_effect

[konradb]

I like Lao Tzu's take on this:

'Those who know don't talk. Those who talk don't know.'.

[iyulaev]

The industry wants people who claim to be the best, and that's hard to do while also selecting for niceness.

Absolutely true. The reason I point this out is that the cost of a jerk goes up as the size of the team he/she is on increases. If you are a 3-man start-up, hiring a superstar coder who's sometimes a PITA can be worthwhile - have them work on some part of the project where they drive the whole thing. At 20 people, that person can become a huge liability - they will misdirect the team, demoralize other engineers, and eventually cause your best people to leave.

My examples illustrate how you might identify people with an uncontrollable mean streak during the candidate process, and hence filter them out. If you're building a scalable organization you must start with founders that aren't jerks and weed these people out of the candidate pool. If you start the company off with such a person, well, either live with it or find a new company.

[svdad]

Great point about not hiring jerks, and those are some excellent ideas about how to avoid it. Having worked with, and currently working with, that kind of jerk I've seen how destructive they can be.

[xsmasher]

Raw smarts is one thing, and experience with a particular platform/problem space is another, and both are valuable on the job, and both should be weighted in hiring.

Sometimes it's appropriate to hire a brilliant person who's unfamiliar with your problem space, but not often. Sometimes you need an expert who already knows what they're doing and doesn't need to ramp up on the company's dime.

[pnathan]

> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform.

When looking for work, I have no room for companies that want to do this: I already have a job and commitments. Also, I might not even be in the same state as the company I am talking to.

[tptacek]

This is such a popular and I think corrosive meme that I think it's a service to the community when people pipe up and point out the obvious, which is that many/most people can't even afford to do try-outs, let alone be happy about the idea. Thanks!

[jotux]

How is trying out a practical for anything other than unemployed new grads? I see this constantly suggested but I don't know a single person who would even consider quitting or taking vacation time to go work for a prospective company.

[chris_mahan]

Although this would work for people who are unemployed. I took a 3 months contract position, and I was hired full time after 1 year. It's been 2.5 years and I'm still at the company.

[scrabble]

The only way I'd consider doing this is taking vacation if I'm being paid a rate that is significantly higher than my current rate.

Otherwise, just ask for a coded solution to some specific problem or user story and I'll work on it after hours and send it in.

[wpietri]

Depends a lot on the person. If somebody has done consulting or contract work, it's a pretty natural thing. I've done it both as an employee and an employer.

[svdad]

I barely even have time for the 3-6 technical interviews most companies want to do, let alone multiple days/WEEKS of "try out" period.

[bobbles]

I used to live about 8 hours drive from where I'm currently working (rural australia to Sydney). When they found out where I was, they arranged the normal 4 interviews to be arranged back to back rather than spread out over a few days.

Just the fact they were willing to do that meant a lot for me (because no one else had ever bothered how inconvenient it was for me to do interviews).

Still here 3 years later and the attitude shown during the interview process extended right through the company.

[res0nat0r]

I think the comment is just being misunderstood, but Amazon (at least AWS), doesn't do any try before you buy type stuff. It is usually an hour long initial phone screen, than an all day long 7 hour interview with multiple people over the course of a single day in Seattle.

[Aqua_Geek]

Although there are definitely logistical problems with this approach, I've had a couple of jobs now where I wish I would have been able to "try it out" for a couple of weeks. Having done so, I never would have accepted the offer (saving me a lot of frustration and a couple of large relocations).

[WalterBright]

> I think Amazon's interview process is fatally flawed.

The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process.

I suspect the real problem is the implication that this is even a solvable problem.

[dylangs1030]

I agree. I don't think it's a solvable problem. It's like people who champion the removal of SAT/ACT from college admissions.

Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with?

We work with what we have, and take the benefits it gives us.

Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.

[jpalioto]

It's a maximization problem. The goal is to have the best possible ROC curve for your hiring process. Maximize true positives while minimizing false positives.

My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves.

I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.

[vladimirralev]

That sounds accurate, but

1. You need large samples of hires because you must know what features actually correspond with good performance

2. You need to have a near perfect understanding of statistics

Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.

[dylangs1030]

I agree strongly. But part of the problem with this solution is that you might be caught with a stream of merely adequate people you won't hire because you're afraid of false positives. I also think it's sometimes okay to hire merely adequate people, if you know ahead of time you can quarantine their usefulness to one area they seem stellar at.

[Zimahl]

3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript.

Yeah, this is pretty painful. I've phone interviewed with Amazon twice. At the time my resume was pure front-end developer with just a smattering of server-side experience. From the Amazon recruiter everything was smiles and 'perfect'.

The interview was for a server-side & database developer position with warehouse software. WTF?! Did you look at my resume for even twenty seconds? I got database design and management questions mostly.

That interview was brutal. What a waste of time for both parties.

[marssaxman]

> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting

Are you seriously proposing that any serious applicant would be willing to take a week off from their current job just to interview for a possible job at Amazon?

[agilebyte]

Re 2.: I have taken some code challenges in the past and I find the Amazon open-ended questions the best. Sure, some people can (and possibly like to?) memorize different data structures and algos, but broad thinking suits me best and companies like Google and Facebook don't seem to be doing it enough.

Basing it on the interview questions posted online.

[arecommender]

If you have gaps in your algo knowledge check out this book (http://www.algorist.com/), it's a dense read but comprehensive. Why be a broad thinker when you can be an everything thinker?

[agilebyte]

Thanks. Even just the website gives a nice outline of some of the different problems and how to solve them.

Gaps in my algo knowledge are there on purpose since to me they are a solved problem and easily found online.

[kb19]

I was just hired at Amazon as an SDE and found the interview process to be fairly challenging and technical (but what do I know).

I appreciated the fact that a lot of the interview focused on core data structures / algorithms and not languages / frameworks of the moment kind of thing. It caused me to go back through my university books and study a lot of the core concepts that I enjoyed learning and have gotten away from by doing more menial programming tasks (not to say that I won't be don't those sorts of things at the job).

Also, if they said that they wanted to try me out for a couple weeks I would have refused the offer. I already had a job that I was happy (and well paid) at, so why should I leave or take vacation to move across the country and be tried out at another company?

[serge2k]

Regarding point 3, I've seen that given out as an option before in talks about interviews. I only see it as a solution that works if the person is actually hired.

If not they waste 2 weeks of their time and probably other SDE time. It leaves no room for people who apply while still working at another job. I had to miss 2 days of work for my interview, and that was about the biggest risk I could afford. Even if you say you will pay me for the 2 weeks, I still wouldn't have a job to come back to. It's probably still a poor job, I was just starting to get into my first task towards the end of 2 weeks. I spend most of the time just getting everything setup (granted we ran into issues that had to be resolved with another team, but still).