[heliostatic]

I'll make the obvious point that these interviews were not good predictors of success at the companies.

In general, I've found in 15 years of interviews that interviews are not highly correlated with the work at any given company. I wish we would move beyond cargo cult interview styles and base interviews (and screening, for that matter) on the actual work and culture of each company.

[eckesicle]

I ran a largish statistical analysis where I was working on all interviews and employee ratings for the following year (16000 data points).

My conclusion was that:

* some interview problems carried zero signal

* some interviewers are really good at identifying strong candidates, and others correlate negatively! On average interviewers rating is a weak signal (<0.25. Correlation)

* geographic location is a medium signal (0.2)

* a relevant degree is a strong signal (0.4)

* choice of programming language is the strongest signal (>0.8 correlation (or anti-) for some languages, and some.

There is some selection bias though since candidates who were rejected did not get a chance at the job so we didn’t have a rating for them.

[hermannj314]

The choice of language being a signal is interesting. That goes against my intuition.

Can you elaborate?

[callahad]

Could rhyme with PG's "Python Paradox" essay from 20 years ago: http://www.paulgraham.com/pypar.html

> It's a lot of work to learn a new programming language. And people don't learn Python because it will get them a job; they learn it because they genuinely like to program and aren't satisfied with the languages they already know.

> Which makes them exactly the kind of programmers companies should want to hire. Hence what, for lack of a better name, I'll call the Python paradox: if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it. And for programmers the paradox is even more pronounced: the language to learn, if you want to get a good job, is a language that people don't learn merely to get a job.

I wonder correlations eckesicle found, and what languages would fit that profile today.

[tough]

So like Erlang nowadays? Or Rust?

[callahad]

It may be my own bubble, but it feels like Rust left that niche around 2019?

I would've put Clojure on the list around 2010.

Elixir feels like a right answer today. Maybe Zig, too?

[yw3410]

My intuition would expect the two to be correlated.

All anecdotal evidence I've seen points to the language mattering a great deal since only those people who are enthusiastic about niche languages, tend to put more time in learning/practicing programming and all other things being equal, spending more time on a skill correlates to being better at said skill.

[eckesicle]

Yes. Interviewees were free to pick any language to attack their problem. We found that interviewees who picked certain languages performed consistently worse during their interviews, and also received worse feedback in their 360-review after working for a year (if they passed).

The bottom languages were: Java C++ Php

They also made up the bulk of interviews. I suspect that Java and c++ together accounted for about 75% of all interviews.

Languages like ruby, python and golang increased your chances of passing the interview as well as getting good peer feedback one year on.

More niche languages (rust, erlang etc) didn’t have enough data points to be analysed separately, but if you grouped them they were the choice of the strongest candidates.

[zephrx1111]

Didn’t know the language can be a factor. I was always told any language is fine.

[breckenedge]

Do you have a link to this research? This is amazing.

[eckesicle]

Sadly not. We ran the study on internal company data, so it was never published. The purpose of the study was to improve and inform on our hiring practices

[prennert]

There are a few outliers, but generally, I found most candidates hired are good, if the interview process mimics the actual projects someone is going to work on in the next 3 months or so. But the downside is that interviewing is slow and cannot easily distributed across multiple teams, because the tasks are tailor made.

Due to the downsides, I guess it is easier for companies to rely on standardised tests. Even just for the reason that those are easy for anyone in the company to lead the interview. But likewise it is easy for anyone to game the system.

[bagacrap]

Or they were good predictors of success, just not in the way you expected. The individuals got the job done one way or another, just like they nailed the entry level job position.

How would you base a job interview on actual work? You have very limited time and you're trying to take a representative sample of a candidate's capabilities.

IMO interviewing is just plain hard and unpleasant. Obviously if you perfectly solve interviewing for software engineers you could rule the world, because you could afford to offer all of your hires $1M/yr which is far less than the best engineers are worth, but more than what everyone else offers.

[osigurdson]

It is funny. Let’s say you have an MIT PHD computer science grad with a 4.0 GPA. The 8 years of verifiable performance means nothing. The competitive coder would blow them away in an interview.

[bee_rider]

A PhD is intended to set you up for an R&D type job. It depends on what kind of education the competitive coder has of course, but if they are interviewing for the same types of positions, the scientists isn’t exploiting their competitive advantage.

[hikawaii]

Yes, because the entire reason competition code was adopted in the first place is people graduating with prestigious titles that could not write code.

[peyton]

Fizz buzz gets ‘em every time. The competition code stuff was more of a culture test. It’s gotten weird now because people outside that culture now memorize problems to get higher-paying jobs.

[osigurdson]

That is possible but the irony must be apparent.