[zawerf]

Recall when you first learned how to program. Something as trivial as iterating through a list requires some thought. What is the syntax for a "for" loop? Do you start with i = 0 or i = 1? Should you end at i < n or i <= n? At some point you stop thinking about it and write "for (int i = 0; i < n; i++)" instinctively. You can now solve harder problems that require iterating through a list without thinking about it.

All of algorithms and programming is like this. You unlock harder problems as you learn more problem solving building blocks (whether it's an algorithm, a software design pattern, or some API call). A programmer is someone who can learn these on the fly for any problem of any domain. An effective programmer is someone who has a large cache preloaded with these building blocks already.

In that sense I find leetcode style problems to be very fair. They are meant to be solvable in under an hour without thinking once cached into muscle memory. All it is testing is whether you're capable of becoming an effective programmer in some agnostic domain. All you need to do is warm your cache with a small number of standard patterns (which might even be useful for real work). It does suck that even the good programmers need a few weeks to warm their cache. But it weeds out the fakers who can't do it given any amount of time.

[pavlov]

> In that sense I find leetcode style problems to be very fair. [...] It does suck that even the good programmers need a few weeks to warm their cache. But it weeds out the fakers who can't do it given any amount of time.

It also weeds out people who have better things to do than cram two weeks for your pretend-meritocratic little exam.

How about requiring that candidates comment their code using quotes from Classical Chinese poetry? They are proven timeless classics that an intelligent person can apply to any situation. This test would weed out the fakers who can't refresh their caches while also honoring an ancient tradition of stupid job interviews, the Chinese imperial examination.

[fjsolwmv]

Algorithms and coding are a lot more relevant than Chinese poetry.

Yes, you might need to prep two weeks in order to get a job that pays very well for 2-15 years. And since this test is so common, you don't have to dedicate 2 weeks to juat one potential employer. If you can't freshen up on standard basic algorithms and coding in 2 weeks for an interview, how can you do it for whatever complex problems you'll face at work?

You can't hire someone solely based on what they claim they built.

I see a lot of people that complaining that they can't take their senior engineer rank from company A and jump into company B at the same level without even justifying themselves, let alone working their way back up. To me that's utterly disrespectful to the the domain specific knowledge and experience that is needed to be a good senior engineer. Someone thinks they are obviously good enough to be a senior employee somewhere, but for some reason isn't good enough to build something valuable on their own, and isn't able to demonstrate skills in a face to face meeting.

[jaredklewis]

> It also weeds out people who have better things to do than cram two weeks for your pretend-meritocratic little exam.

All interviewing techniques have to make precision-recall style trade offs. The mere fact that an interview method has false negatives surely doesn't disqualify it. It has to be compared against the available alternatives. What are the alternatives?

- White boarding? Algorithmic knowledge is often tangential to the actual job.

- Take home assignments/mini projects? High relevance to job, but in my experience takes the most time for the candidate.

- Trial period? Most people can't just drop everything they are doing to come hang out at your company.

- Conversational interview. Like white-boarding, tangential to actual job. My experience on the interviewing side is that it is often hard to learn much about the candidate.

- Read their code on github / blog. Lots of candidates don't have the time or inclination to code outside of work.

- Something else?

So what's your preference? I've done them all and find them all to be lacking in different ways.

> How about requiring that candiates comment their code using quotes from Classical Chinese poetry? They are proven timeless classics that an intelligent person can apply to any situation. This test would weed out the fakers who can't refresh their caches while also honoring an ancient tradition of stupid job interviews, the Chinese imperial examination.

This seems like the fallacy of grey to me [1]. When hiring, for example, a web developer, yes, algorithmic knowledge is a somewhat arbitrary indicator to use, but it is not completely arbitrary. Not all things are equally unlike.

If I were hiring for a basketball team, and had to choose between two candidates neither of whom had experience playing basketball and were alike in all ways except that one was an avid soccer player and one was equally fervent about pottery, I would choose the soccer player. The logic of course being basketball and soccer have more in common (athleticism at the least) than basketball and pottery.

Likewise, algorithmic thinking shares some common points with almost any kind of engineering task.

Interviewing is just a hard problem where you are trying to predict future performance based on a few hours worth of data. I don't think most of the popular the techniques we have are obviously stupid. Companies have strong incentives to make hiring efficient, but there just isn't a lot of low hanging fruit. Of course there are the occasional ego maniac interviewers, but an ego-maniac is going to be able to ruin any type of interview. Let't not throw out the baby with the bathwater.

1. https://www.lesswrong.com/posts/dLJv2CoRCgeC2mPgj/the-fallac...

[pavlov]

> If I were hiring for a basketball team, and had to choose between two candidates neither of whom had experience playing basketball and were alike in all ways except that one was an avid soccer player and one was equally fervent about pottery, I would choose the soccer player.

That’s not at all what happens in programming interviews that use algorithmic puzzles.

You have candidates who already have a professional track record in basketball, and instead of focusing on that profile and whether it’s a good fit for your team, you give them a timed soccer workout because it’s somehow a more objective measure of athletic ability.

Any basketball team that hires like that wouldn’t survive for long. The quiz interview format in the tech industry is a form of “anti-Moneyball”. It works for the SV giants because they have an enormous supply of candidates and they need generic competence that can be shuffled around. Smaller companies would do much better to hire for the actual role, not for “Cracking the code interview” memorization performance.

[jaredklewis]

The basketball was just a metaphor to explain relative similarity. Obviously hiring for an actual basketball team is a very different set of challenges as they literally have hundreds of hours of video taped performance history to evaluate before the candidate even walks in the door, and at lower levels, asking candidates to spend several days “trying out” is not considered onerous. But I would still argue that performance on programming puzzles much more closely correlates to programming job performance than knowledge of Chinese poetry.

As for hiring people based on their experience profile, it’s great of course in the case of candidates with lots of open source contributions and such, but this has the issue of ignoring the majority of candidates which don’t contribute to open source. Should being an os contributor be a hard requirement?

But if you are suggesting that a resume with the words “5 years experience web development at company x” means anything, I’m a little incredulous. I worked with people that claimed to have far more experience than that and struggled horribly with even the most basic tasks.

Finally, a little tangential, but memorization gets a lot of flack for being a “stupid” skill. My experience is that it is nearly impossible for adults to memorize something like Chinese by poetry “by rote.” Indeed if you try memorizing some poetry I think you’ll find that it really is a very fulfilling and creative process.

[im_down_w_otp]

Here's what I do:

1. Filter candidates based on fairly simplistic early models for personality profile, motivational bias, and metacognitive disposition cues.

2. On a subset, further refine the above filtering further w/ another 1 or 2 interactions looking for inconsistencies and/or stressing facets of the model that seem contraindicating or hard to suss out.

3. Prep the team on what & how to assess and bring a small number of candidates on-site for a few hours, to work directly with the members of the team they'd be joining, and have them work together on exactly the work they'd be doing.

So I put in a lot of work ahead of time as a hiring manager to understand what kind of role I need to hire for, what kind of person would likely be successful in that role, and what kind of person would likely be successful working with the team that exists (or will be built). Then I completely avoid some contrived pile of quizes and weak competence signals by instead directly using an actual work environment w/ the same people, the same meetings (stand-up, design review, etc.), and the same tasks that both we & they would be cooperating on together.

[jaredklewis]

Seems like a nice approach. I've been through similar setups and I think there are still tradeoffs though. Step #3 is tricky as you need a task complex enough for the candidate to show their skills, but it has to fit within a few hours. Unrelated to interviewing, I routinely under or overestimate task size, so carving out just the right task can be difficult. New features and bugs often have unknown unknowns.

The "contrived" puzzles approach has the advantage that each candidate can be given (and thus evaluated on) the same task. The size and perquisite knowledge for the task can be well controlled and since the problem is not new to the interviewers, we know how to present it in an easily understandable way and help them if they get stuck.

I think another reason why the "general cognitive ability" approaches are popular is because employees (especially at small companies) need to be good at such a wide range of tasks that it is not realistic to evaluate even a fraction of them in the span of a few hours.

[im_down_w_otp]

I don't have any expectation they complete the task. That's not really the point, so time-bounding it isn't something I worry about. The point is that they're able to engage and contribute in some way: insightful feedback, mentoring, reintegrative learning, productive collaboration, etc. depending on the shape of role that's being hired for.

The tasks can be writing docs, writing requirements, writing property-based tests, writing CLI tools for developer ergonomics, formally modelling and verifying a scheduler, designing a DSL for safety-constraints or characterizing the electrical interfaces of a car's steering system. It's not entirely material what the tasks are. There will be opportunities in any of those to get good indicators of the important factors.

FWIW, the purported consistency of evaluating people using the same contrived task is fairly unlikely to be actually consistent, and even if it were, the value of what you can actually meaningfully derive from it is still deeply questionable all the same. As a result the reality is more likely that those situations are producing negative net benefit.

[AlphaSite]

Ideally tests should not require preparation, thats where the issue is.

[fjsolwmv]

Why? Building software requires preparation.

[kelnos]

> which might even be useful for real work

In my experience, this is where the house of cards collapses. For most programming jobs, none of that will ever be useful. And on the rare occasion that it is, you'll have gotten by without needing it for so long that you'll have to look it up and more or less learn it again from scratch anyway.

I find these sorts of things to give far too little useful information during a developer interview.

[phkahler]

It's an aggressive filter. We sometimes give a post-interview take home programming problem that involves no trickery or algorithm Discovery/invention. It's just to verify they can code and see their style. You should be able to determine if a person can code from resume and interview alone, but sometimes I can't tell until the exercise. People suck at interviews so they fall back on hard problems. I suppose that works for big companies with lots of applicants.

[aprdm]

> A programmer is someone who can learn these on the fly for any problem of any domain

Do you really believe in this? I would maybe call it an Ex Machina droid...