[josephg]

> Twenty years later we have positions where hires are selected for their ability to reverse a red-black tree on a whiteboard, where the position will mostly be about gluing together CRUD apps with YAML.

A few years ago I worked as an interviewer for a large software engineering recruiting company. We did quantitative scoring on a lot of parts of our standardized interview. We had sections in our interview on CS knowledge, programming, debugging and whiteboard style problems. Based on the data, we asked: Could we eliminate any part of the assessment? Could we throw out the CS knowledge part without losing accuracy about the overall hireability of the candidate?

Based on the data, the answer was no. The scores were all positively correlated - so CS knowledge implied you were better at programming and vice versa. But we still got extra signal by assessing candidates on their CS knowledge. Turns out even if you aren't amazing at programming, having excellent CS knowledge will still make you desirable to a lot of companies.

(The weakness of this study is we didn't follow up with people. We only knew if our candidates got hired, not how well they did after they were in the door, as employees. So we might have just been mirroring the same biases the companies themselves have in their hiring processes.)

[marginalia_nu]

> (The weakness of this study is we didn't follow up with people. We only knew if our candidates got hired, not how well they did after they were in the door, as employees. So we might have just been mirroring the same biases the companies themselves have in their hiring processes.)

This is some weakness. Surely this is the outcome that's interesting.

I also suspect there's likely a correlation between how hard someone is trying to get hired and the freshness of their CS skills.

Chances are a motivated candidate will have been brushing up on CS because it's such a trope to be grilled on those types of questions; that same candidate likely prepared for other interview questions as well and I would indeed expect that to increase the odds of getting hired.

Since nobody walks around with perfect recall of the types of algorithms that crop up in the classic tech interview, it's fairly safe to assume the CS part of the interview is a direct measure of how much preparation the interviewee has done.

[josephg]

> Since nobody walks around with perfect recall of the types of algorithms that crop up in the classic tech interview, it's fairly safe to assume ...

I'd love to have data on that.

I know its really common for engineers to never touch this stuff in their day to day work. Most product teams don't need any of this knowledge at all. So asking about it in an interview is a massive waste of time.

But personally, I've used a lot of these algorithms while working on collaborative editing for the last few years. For diamond-types, I ended up writing my own b-tree and skip list implementations, and I make heavy use of binary search, BFS, DFS and priority queues. I've used priority queues in plenty of projects - like, years ago I made a library to mock out timers in nodejs for our test suite so we didn't have to wait for real timeouts to trigger in our test suite.

But I've got no idea what percentage of working engineers use any of this. 5%? 1%? 0.01%? On the surface, it seems nowhere near useful enough to justify how often these questions show up in interviews.

[marginalia_nu]

Well part of the reason I'm skeptical of how who would actually be able to pass these interviews without cramming for them based on the fact that I too have used a fair number of the algorithms (because as mentioned, internet search is a fractal of challenging CS problems) and would almost definitely not pass such an interview.

Just because I've implemented a binary search a few times, and a b-tree, and a skip list, and various sorting and intersection algorithms doesn't mean I can reconstruct them on a whiteboard from memory. What it amounts to is that I have an upper quartile understanding of the underlying idea and the general quirks (among practicing programmers), but not much more than that.

[josephg]

> would almost definitely not pass such an interview.

If you would fail an interview like this, it’s not calibrated correctly. I know it’s unsatisfying, but nobody gets perfect marks on assessments like this by design. Nobody is expecting you to program up a correct btree on a whiteboard in an interview. Just being able to speak in detail about data structures and algorithms from practical experience is a very strong signal in a candidate. Let alone being able to explain high level concepts, and talk about when they’re useful and maybe explain some implementation details. That’s great!

[esaym]

> We only knew if our candidates got hired, not how well they did after they were in the door, as employees.

Just lol

[dimal]

> The weakness of this study is we didn't follow up with people. We only knew if our candidates got hired, not how well they did after they were in the door, as employees

I’m sorry, but then your study doesn’t show anything useful. Interviews are supposed to determine whether someone is going to be a good employee, not whether they are “desirable to a lot of companies.” So yeah, all you were doing was mirroring the bias of those companies’ processes. As a recruiting company, that can be good business, because you’re giving the customers what they want. But it’s not effective hiring practice for the customer.

Furthermore, you did not follow up on the people that were weeded out to see if they would have been good employees. Current hiring practices weed out a lot of people that could have provided a lot of value, but they weren’t able to perform the prescribed ritual on cue, you didn’t have a chance to evaluate whether that was a good decision or not.

[verve_rat]

Wait, so the conclusion the study came to was... people that do well in the interview process tend to get hired?

In my experience interview processes act like a series of stage gates, you have to pass each and every one to get hired. It seems trivial to say the doing well in each activity is predictive of success when you need to do well in everything to be successful.

I don't mean to be mean, but that study sounds like a giant missed opportunity to do something useful. Knowing what interview activities are actually predictive of being good in a job is the holy grail.

[bravura]

Completely missed opportunity, particularly since GP has been breathlessly recounting this story to others since it happened, and probably his old colleagues are too.

[josephg]

GP here. I completely agree. Looking back, I wish I pushed harder to follow up and get the last piece of that data. It was a massive missed opportunity.

[gremlinunderway]

>(The weakness of this study is we didn't follow up with people. We only knew if our candidates got hired, not how well they did after they were in the door, as employees. So we might have just been mirroring the same biases the companies themselves have in their hiring processes.)

Kudos for the self-awareness but like others have pointed out, cmon.

This obsession with quantitative scoring for tests is a great example of a cargo cult science because you are doing all of these intricate little rituals that amount to nothing because you're simply measuring latent social phenomenon of hiring like-minded people. It's just statistically-laundered-bias.

[LudwigNagasena]

> The weakness of this study is we didn't follow up with people. We only knew if our candidates got hired, not how well they did after they were in the door, as employees.

That's not just a weakness, that invalidates the whole study.