[andrewstuart]

In a sentence, I am saying that pretty much every recruiting process at every company, when it assesses developers, results in a meaningless decision, despite the universal certainty at every company that their recruiting process absolutely definitely results in making accurate decisions about people.

Did the interview process find a great developer? Maybe. Did the recruiting process reject great developers? How would you ever know? I'm pretty much certain that most companies reject many more developers who would be really productive/effective, than they choose to hire.

[dsacco]

That still seems like a pretty strong claim. Inefficient? Sure. Misaligned incentives? Definitely. But actually meaningless? If a company turns away many developers who would be excellent hires but consistently meets its product/engineering goals and hires more good developers than bad developers overall, their recruiting process certainly has a positive signal. It's obviously inefficient, but it couldn't be meaningless.

I also don't think it's fair to characterize most companies as believing their recruiting practices infallible. Google's former director of recruiting has frequently talked about how their recruiting processes (and particularly interviewer feedback) seemed very noisy and fairly inconsistent in a variety of dimensions. The common recruiting-page-positive-vibes spiel might indicate a lot of confidence, but that's not what I would use to determine it. A lot of companies also seem pretty happy to try out new programs like e.g. Triplebyte or otherwise innovate on their sourcing processes.

[andrewstuart]

Not a direct response to your comment, but I once sent someone who I thought was a really bright developer to a company, who assessed him and rejected him.

That same developer soon after got a job at Amazon. When I told the original employer who had rejected him, he said "So Amazon are hiring shit people now?".

That hiring manager I have never ever known to question any hiring decision he ever made - he has always been certain that the rejection was correct.

And what I have observed in recruiting is that any rejection decision made by a company is made with 110% certainty.

[ubernostrum]

I will stand up for the "meaningless" statement.

The typical tech company interview measures things which have no relationship to someone's quality as a programmer or their effectiveness as a member of a dev team. Any success of such a process (success meaning hires someone who turns out to be competent and a good fit in the team) is the result of factors unrelated to the interview process, and probably has a significant component of chance. But it's treated as an effective arbiter of "qualified" versus "utterly and totally incompetent to work anywhere".

(I firmly believe that Joel Spolsky and Jeff Atwood have done more active harm to tech hiring than perhaps any two other people, by the way)

Here are slides from a talk I gave last year with a former co-worker about doing better tech interviews, which go into a bit more detail on the issue:

http://media.b-list.org/presentations/2017/pygotham/intervie...

And here's a longer-form piece which touches on interviewing problems as part of a larger issue in the industry:

https://www.b-list.org/weblog/2018/jan/08/degrees/

[dsacco]

I really don't think you can declare that any success of the process is due to chance or "other factors." That particular point is completely unsubstantiated. The process is inefficient, yes, but I really don't see any basis to call it meaningless except for personal dislike.

Even if the process is inefficient (many false negatives), if it yields a population with fewer false positives than the general population it is meaningful. This is an important distinction because many people seem to be unilaterally declaring these hiring practices to be useful due to dislike and anecdata. Of course they're not ideal and there is a lot of room for improvement, but there's no empirical reason for us to act as though the entire system is arbitrary just because we don't like it.

The primary question we should be looking at to inject some empiricism into this discussion is how many engineers end up being fired once hired, how long it takes to fill each technical role on average and how many candidates are turned down. There seens to be a false dichotomy at play, where people are only able to damn the process in its entirety, but it can still be bad and retain some signal. That shouldn't be a controversial point, it should be a minor preliminary observation.

[ubernostrum]

Again: the metrics being measured in typical tech interview processes are not the same metrics those same companies would use to measure success on the job. They are not useful predictors of success or failure on the job.

If the interview metrics are that disconnected, there is no meaning to the interview metrics.

[dsacco]

But even if the metrics are not the same, they can still be positively correlated with each other.

[jaggederest]

> If a company turns away many developers who would be excellent hires but consistently meets its product/engineering goals and hires more good developers than bad developers overall, their recruiting process certainly has a positive signal. It's obviously inefficient, but it couldn't be meaningless.

Actually, that result would be precisely what you would expect from a process that fails to disprove the null hypothesis, and is therefore meaningless. You'd see them hiring essentially a random sample of engineers with the same distribution as the whole population.

I leave it to someone better at statistics than I am to figure out what an actually effective hiring process looks like, but I think the correlation between "is a relatable person" and hiring is much higher than "is a good engineer".

[dsacco]

> Actually, that result would be precisely what you would expect from a process that fails to disprove the null hypothesis, and is therefore meaningless. You'd see them hiring essentially a random sample of engineers with the same distribution as the whole population.

This implies that the capability distribution at Google is approximately equal to the capability distribution of the entire set of eligible developers. I don't personally think that's true, and in any case even if it is true it's not obvious.

[mason55]

Do companies really believe their hiring processes are all great? From people posting online it seems like everyone knows their process sucks but doesn’t have ideas for improving it

[majormajor]

> the universal certainty at every company that their recruiting process absolutely definitely results in making accurate decisions about people

The companies I've worked at have always been very aware their interviewing process could be better, and so I've been looped into many, many conversations over the years along "how can we reduce false negatives" or "why are we rejecting all our senior candidates?"