[davesims]

The challenge here, in my experience on both sides of this, is that a developer's deep domain knowledge of a large or large-ish app is essential to a 'work-sample' environment. And that's virtually impossible to duplicate even in the longest interview time frame.

Making decisions about managing technical debt, adding architecturally-significant changes, balancing good OOP with responsiveness, knowing the difference between future-proofing and conscientious coding -- all of those are both crucial (in many cases the most crucial) to day-to-day work, and also so highly context-specific that those decision-making traits are nearly impossible to identify during a technical exercise.

So for coding challenges, that leaves short-term tactical/analytic/algorithmic exercises, which in (anecdotally) 95% percent of cases cannot begin to approach a 'work-sample' environment. I've yet to encounter a technical challenge that would tell me much more about a candidate than basically how fluent they are with their tools, how well they know syntax and some general design principles, and what, for instance, their TDD (or lack of) workflow is like. Probably some insight into line-level analytic and algorithmic ability.

All of that is helpful, but -- Trust Me Here!! -- can also be very deceiving. The same coders that can knock those challenges out of the park can also be highly-proficient Debt Machines, all the more destructive because of their special genius for cranking out architecturally suspect code at a breathtaking rate.

To get into a real 'work-context' flow of a large app requires weeks, sometimes months, and only then can you get full perspective on how a given coder is going to contribute to your team on an ongoing basis. To get a feel for what that will look like in an interview, I've found I have to pretty much rely on the candidate's past projects, and informal conversations around larger architectural and OO principles.

[tokenadult]

I think your point is well made that what you can sample in a work-sample test is not the full set of long-term skills that benefit a for-profit company. That's why the predictive validity of work-sample tests is only about .50 across a wide range of industries. But the key point is that EVERY other hiring procedure, except for general cognitive ability tests, has lower validity, so a company is throwing away a lot of opportunity to hire good workers if it doesn't use a combination of work-sample testing and cognitive ability testing for all of its hiring. Your sound analysis can be turned around to using interviews as a hiring procedure--which is much more commonplace than using work-sample testing as a hiring procedure--to make the correct point that an applicant who looks good in an interview may not be a "team player" once hired. Any hiring procedure is a sample of applicant behavior, not fully representative of how the worker will behave on the job after being hired. But work-sample tests get much closer to what the worker will do on the job long-term than any other procedure besides general cognitive ability tests. Because work-sample tests and cognitive ability tests each have incremental validity when added to the other, it's best to use both in combination to get a hiring procedure with somewhat more than .50 validity in finding good workers.