[im_down_w_otp]

Here's what I do:

1. Filter candidates based on fairly simplistic early models for personality profile, motivational bias, and metacognitive disposition cues.

2. On a subset, further refine the above filtering further w/ another 1 or 2 interactions looking for inconsistencies and/or stressing facets of the model that seem contraindicating or hard to suss out.

3. Prep the team on what & how to assess and bring a small number of candidates on-site for a few hours, to work directly with the members of the team they'd be joining, and have them work together on exactly the work they'd be doing.

So I put in a lot of work ahead of time as a hiring manager to understand what kind of role I need to hire for, what kind of person would likely be successful in that role, and what kind of person would likely be successful working with the team that exists (or will be built). Then I completely avoid some contrived pile of quizes and weak competence signals by instead directly using an actual work environment w/ the same people, the same meetings (stand-up, design review, etc.), and the same tasks that both we & they would be cooperating on together.

[jaredklewis]

Seems like a nice approach. I've been through similar setups and I think there are still tradeoffs though. Step #3 is tricky as you need a task complex enough for the candidate to show their skills, but it has to fit within a few hours. Unrelated to interviewing, I routinely under or overestimate task size, so carving out just the right task can be difficult. New features and bugs often have unknown unknowns.

The "contrived" puzzles approach has the advantage that each candidate can be given (and thus evaluated on) the same task. The size and perquisite knowledge for the task can be well controlled and since the problem is not new to the interviewers, we know how to present it in an easily understandable way and help them if they get stuck.

I think another reason why the "general cognitive ability" approaches are popular is because employees (especially at small companies) need to be good at such a wide range of tasks that it is not realistic to evaluate even a fraction of them in the span of a few hours.

[im_down_w_otp]

I don't have any expectation they complete the task. That's not really the point, so time-bounding it isn't something I worry about. The point is that they're able to engage and contribute in some way: insightful feedback, mentoring, reintegrative learning, productive collaboration, etc. depending on the shape of role that's being hired for.

The tasks can be writing docs, writing requirements, writing property-based tests, writing CLI tools for developer ergonomics, formally modelling and verifying a scheduler, designing a DSL for safety-constraints or characterizing the electrical interfaces of a car's steering system. It's not entirely material what the tasks are. There will be opportunities in any of those to get good indicators of the important factors.

FWIW, the purported consistency of evaluating people using the same contrived task is fairly unlikely to be actually consistent, and even if it were, the value of what you can actually meaningfully derive from it is still deeply questionable all the same. As a result the reality is more likely that those situations are producing negative net benefit.