[jaredklewis]

> It also weeds out people who have better things to do than cram two weeks for your pretend-meritocratic little exam.

All interviewing techniques have to make precision-recall style trade offs. The mere fact that an interview method has false negatives surely doesn't disqualify it. It has to be compared against the available alternatives. What are the alternatives?

- White boarding? Algorithmic knowledge is often tangential to the actual job.

- Take home assignments/mini projects? High relevance to job, but in my experience takes the most time for the candidate.

- Trial period? Most people can't just drop everything they are doing to come hang out at your company.

- Conversational interview. Like white-boarding, tangential to actual job. My experience on the interviewing side is that it is often hard to learn much about the candidate.

- Read their code on github / blog. Lots of candidates don't have the time or inclination to code outside of work.

- Something else?

So what's your preference? I've done them all and find them all to be lacking in different ways.

> How about requiring that candiates comment their code using quotes from Classical Chinese poetry? They are proven timeless classics that an intelligent person can apply to any situation. This test would weed out the fakers who can't refresh their caches while also honoring an ancient tradition of stupid job interviews, the Chinese imperial examination.

This seems like the fallacy of grey to me [1]. When hiring, for example, a web developer, yes, algorithmic knowledge is a somewhat arbitrary indicator to use, but it is not completely arbitrary. Not all things are equally unlike.

If I were hiring for a basketball team, and had to choose between two candidates neither of whom had experience playing basketball and were alike in all ways except that one was an avid soccer player and one was equally fervent about pottery, I would choose the soccer player. The logic of course being basketball and soccer have more in common (athleticism at the least) than basketball and pottery.

Likewise, algorithmic thinking shares some common points with almost any kind of engineering task.

Interviewing is just a hard problem where you are trying to predict future performance based on a few hours worth of data. I don't think most of the popular the techniques we have are obviously stupid. Companies have strong incentives to make hiring efficient, but there just isn't a lot of low hanging fruit. Of course there are the occasional ego maniac interviewers, but an ego-maniac is going to be able to ruin any type of interview. Let't not throw out the baby with the bathwater.

1. https://www.lesswrong.com/posts/dLJv2CoRCgeC2mPgj/the-fallac...

[pavlov]

> If I were hiring for a basketball team, and had to choose between two candidates neither of whom had experience playing basketball and were alike in all ways except that one was an avid soccer player and one was equally fervent about pottery, I would choose the soccer player.

That’s not at all what happens in programming interviews that use algorithmic puzzles.

You have candidates who already have a professional track record in basketball, and instead of focusing on that profile and whether it’s a good fit for your team, you give them a timed soccer workout because it’s somehow a more objective measure of athletic ability.

Any basketball team that hires like that wouldn’t survive for long. The quiz interview format in the tech industry is a form of “anti-Moneyball”. It works for the SV giants because they have an enormous supply of candidates and they need generic competence that can be shuffled around. Smaller companies would do much better to hire for the actual role, not for “Cracking the code interview” memorization performance.

[jaredklewis]

The basketball was just a metaphor to explain relative similarity. Obviously hiring for an actual basketball team is a very different set of challenges as they literally have hundreds of hours of video taped performance history to evaluate before the candidate even walks in the door, and at lower levels, asking candidates to spend several days “trying out” is not considered onerous. But I would still argue that performance on programming puzzles much more closely correlates to programming job performance than knowledge of Chinese poetry.

As for hiring people based on their experience profile, it’s great of course in the case of candidates with lots of open source contributions and such, but this has the issue of ignoring the majority of candidates which don’t contribute to open source. Should being an os contributor be a hard requirement?

But if you are suggesting that a resume with the words “5 years experience web development at company x” means anything, I’m a little incredulous. I worked with people that claimed to have far more experience than that and struggled horribly with even the most basic tasks.

Finally, a little tangential, but memorization gets a lot of flack for being a “stupid” skill. My experience is that it is nearly impossible for adults to memorize something like Chinese by poetry “by rote.” Indeed if you try memorizing some poetry I think you’ll find that it really is a very fulfilling and creative process.

[im_down_w_otp]

Here's what I do:

1. Filter candidates based on fairly simplistic early models for personality profile, motivational bias, and metacognitive disposition cues.

2. On a subset, further refine the above filtering further w/ another 1 or 2 interactions looking for inconsistencies and/or stressing facets of the model that seem contraindicating or hard to suss out.

3. Prep the team on what & how to assess and bring a small number of candidates on-site for a few hours, to work directly with the members of the team they'd be joining, and have them work together on exactly the work they'd be doing.

So I put in a lot of work ahead of time as a hiring manager to understand what kind of role I need to hire for, what kind of person would likely be successful in that role, and what kind of person would likely be successful working with the team that exists (or will be built). Then I completely avoid some contrived pile of quizes and weak competence signals by instead directly using an actual work environment w/ the same people, the same meetings (stand-up, design review, etc.), and the same tasks that both we & they would be cooperating on together.

[jaredklewis]

Seems like a nice approach. I've been through similar setups and I think there are still tradeoffs though. Step #3 is tricky as you need a task complex enough for the candidate to show their skills, but it has to fit within a few hours. Unrelated to interviewing, I routinely under or overestimate task size, so carving out just the right task can be difficult. New features and bugs often have unknown unknowns.

The "contrived" puzzles approach has the advantage that each candidate can be given (and thus evaluated on) the same task. The size and perquisite knowledge for the task can be well controlled and since the problem is not new to the interviewers, we know how to present it in an easily understandable way and help them if they get stuck.

I think another reason why the "general cognitive ability" approaches are popular is because employees (especially at small companies) need to be good at such a wide range of tasks that it is not realistic to evaluate even a fraction of them in the span of a few hours.

[im_down_w_otp]

I don't have any expectation they complete the task. That's not really the point, so time-bounding it isn't something I worry about. The point is that they're able to engage and contribute in some way: insightful feedback, mentoring, reintegrative learning, productive collaboration, etc. depending on the shape of role that's being hired for.

The tasks can be writing docs, writing requirements, writing property-based tests, writing CLI tools for developer ergonomics, formally modelling and verifying a scheduler, designing a DSL for safety-constraints or characterizing the electrical interfaces of a car's steering system. It's not entirely material what the tasks are. There will be opportunities in any of those to get good indicators of the important factors.

FWIW, the purported consistency of evaluating people using the same contrived task is fairly unlikely to be actually consistent, and even if it were, the value of what you can actually meaningfully derive from it is still deeply questionable all the same. As a result the reality is more likely that those situations are producing negative net benefit.