[Harj]

What was always most interesting to us about starting a recruiting company was seeing what would happen if you treated hiring as a data problem. Partly we've raised more funding for the same reason any startup does, so we can grow faster to get more customers = more revenue = more success, etc. But we're also driven by how the larger the scale we operate at, the faster we can run experiments to answer questions about the best way to evaluate technical skills. More rigorous and data focused approaches to hiring benefit everyone.

Interviewing and evaluating engineers is an area a lot of people feel passionately about and have strong opinions on. We're continually looking for ways to improve our process, if you've any thoughts or feedback please ping me - harj at triplebyte.

[fro0116]

Wow. Is it just me or does that sound like a downright horrifying company to deal with as a candidate?

At our company, we try to painstakingly craft our recruiting experience to make sure each candidate we interview has a good experience and ends up with a positive impression of our company regardless of whether or not we end up sending them an offer. At the end of the day, we're all human beings, each with something unique to bring to the table, even if that something might not be what we're looking for for a particular role at the moment.

Maybe past some scale we'll have to start changing our approach and start reducing candidates down to data points and "run experiments" on them like lab rats, like Google, et all, and these guys here seem to be so proud of themselves for doing, but I'd sooner quit than to stay a part of a company that does that.

[kev009]

For giggles (I work on OS kernels and high performance TCP, there's no way this kind of meatgrinder would ever place me correctly) I did the online quiz thing and got to the interview piece just to see why this was generating buzz. The quiz was easy and when I saw the interview prep I was disgusted. It's gamified hiring practice done in an outsourced fashion.. the kind where you are supposed to read a book that tells you what kind of questions a hiring party will ask you. I guess if you are a lousy engineering manager and want to offload and emulate the hiring process of something like Google this is up your alley. If you want some relatively fungible developers that can work on CRUD applications it may be fine but I don't think that's hard to hire for in any case.

Engineering managers MUST do recruiting - never let HR take this from you. HR can do clerical work, but they should have little role in search and outreach. I'd instead recommend finding new talent at local universities, mid level talent give recruiter like bonuses for employee referrals and poach from competitors, and extraordinary talent go look at commit logs of the open source software you use and hire people from that list. Easy, cheap, and effective.

[7h3kk1d]

I'd rather find a better long term fit quicker than have a pleasant process when looking.

[fro0116]

I mean sure, so would I if I had to choose between the two, but that's a false dilemma.

And for me at least, a company that doesn't treat their candidates with kindness and respect in the hiring process likely wouldn't be a long term fit for me anyways.

[ravenstine]

I haven't been interviewed by you, but the initial process is stellar! Applying and going through the code challenges was very smooth. Plus the fact that on completion, if the applicant passes, are pretty much guaranteed some form of an interview, is great.

[alimhaq]

Just wanted to say thank you! I used Triplebyte to find the company I'm currently working for, and I've had a wonderful time here so far. The process took a little longer than I had anticipated, but it was well worth it :)

[gxs]

The crux of this problem in my opinion is that long term results aren't tracked.

It's hard enough internally to track someone's performance over the course of the year or two after they get hired, it would be even harder to do it if you are a recruiting company.

It's especially sensitive because employers are weary of sharing employee performance data to third parties because of the high risk of a lawsuit (there is clear precedent for these lawsuits.)

Once that data problem is bridged, it blows the problem right open for data to be explored and figure out what exactly predicts a top performer, in any field.

[Benjammer]

>Once that data problem is bridged, it blows the problem right open for data to be explored and figure out what exactly predicts a top performer, in any field.

Yes, assuming there is some top-level "data problem" to actually bridge here...

How do we know that the concept of "top performer" isn't just a completely divergent idea that means different things to different people and different companies in different industries and different geographic areas?

[metaobject]

I'm inclined to believe that measuring performance is fairly subjective. However, in an attempt to put some numbers behind a performance score, you might have some combination of individual weighted scores that involve things like:

(1) length of time employed at the company

(2) some measure of the performance feedback the engineer receives in their annual review - which may include percentage salary increase (possibly subjective, though)

(3) a score compiled by surveying the employees' peers (again, possibly subjective)

(4) the overall TripleByte turnover rate at said company

(5) the market average for any/all of (1)-(4)

[gxs]

Sure, it's a subjective measure, but it's a start.

Keep in mind there is ZERO data available for this type of analysis at the moment.

Even if it's subjective, the analysis can provide the employer with candidates that are good based on the same subjective criteria.

[fludlight]

It would be interesting to see stats on new grad engineers by college in a similar way as the very popular hard drive stats from backblaze.

I would like to test my hypothesis that grads from the top 10 cs schools as determined by US News et al. are generally good, but overvalued.

[wtvanhest]

It highly depends on how you measure value. If you measure value in terms of what they bring to the overall organization, people from top 10 schools are very likely overvalued. If you measure what value they bring to the team, top 10 schools are probably correctly valued. The difference is that the manager of the team hiring from Harvard can speak to people in more senior roles about how great their team is and how they have the best talent.

At the end of the day, value is driven by how much political leverage a hire can give a manager, not by how much value they add to the org.

[fludlight]

I was thinking of stats like this:

- % of candidates that applied contacted by company - % can fizzbuzz - % offered position - % still working at company after 6 months...

[shreyanshd]

Hey Harj, congrats! Any plans to launch TripleByte in India, or for engineers from India who want to interview with Bay Area companies?

[jensvdh]

Due to visa constraints there literally is no point in doing this.

[mattm]

Yeah I believe Triplebyte will only accept candidates from countries that have an easy way to get a work visa in the US - Canada, Mexico, Australia and Singapore (I think). (I'm Canadian and am working at my current company through Triplebyte)

[guessmyname]

That's weird, I am also Canadian and my profile says that they can only proceed with my application with people who hold an US visa and/or work permit. I thought it was a mistake so I contacted their support team and got a reply saying that they can only consider my application if I live in the US.

Did you change your location to some city in the US?

How did you get your application accepted?

On the same note, Hired.com which seems to be a direct competitor, also has restrictions on the locations available for their recruitment process. In Canada, for example, they only accept people living in Toronto. Or people lying about their current location because they are planning to move there.

[mattm]

Hmmm, I applied about a year ago so maybe they've changed that recently?

Edit - their FAQ page says that they accept Canadians so maybe ask again?

"the companies we work with CANNOT provide visas for candidates unless they are from Canada, Mexico, Singapore, Chile, or Australia"

https://triplebyte.com/candidate_faq

[joshwa]

I, for one, would love a good bozo filter for folks I interview for my team in India.

[forgotmysn]

Do you think this process can be effectively adapted to roles outside engineering?

[tsme1]

Where will you expand next? Which cities?/countries?

[blabla_blublu]

Congrats, Harj!

[xiphias]

Hi, I just went through the interview questions, they were fun (they said that I did exceptionally well, but nothing concrete, like percentage). I'm not looking for job, just though I try it out.

What I was interested in is whether the questions get harder, if I answer well, but they seemed random.

You could use logistic regression to estimate the level of an interviewer and adjust the questions to get to the same accuracy with less time (or to improve accuracy with the same number of questions/time)

[ebakan]

So you're right that the quiz does try to be harder if you're doing well, but it'll also give you easier questions if an incorrect answer lowers its confidence in your ability estimate. We have a pretty sizeable bank of potential questions to ask a candidate, but the quiz tries to strike an optimal balance between appropriate difficulty and maximum informativeness. For example, we wouldn't want to as you a particularly difficult question unless we're confident that it's a) a good fit for your estimated ability level, and b) will give us more information about your ability than any other question in the bank.

You're right that tailoring question difficulty to ability level can drastically increase a test's accuracy. But while a logistic regression model works well when you have a fixed quiz or a low number of questions, it isn't flexible enough to work with a fully adaptive system like we have at Triplebyte. Our models are loosely based on the kinds of systems that the MGAT or GRE use, but we've implemented significant extensions on top of those approaches to fit our needs.

[xiphias]

Thanks for the answer. When I was implementing a language learning program (who hasn't? :) ), using logistic regression was working quite well to quickly find my vocabulary level in about 10 questions in the top 10000 most frequently used words list adaptively (I ran a full logistic regression on the user dataset after each new data point, by mapping the position of the words to the estimated level of the user), and the questions just felt right. So I'm not talking about multiple logistic regression model, just using 1 variable, which works with lots of questions (as long as the question hardness is well calibrated).

Although I'm happy that you're trying to predict the most informative question, for me some questions near the end felt trivial, so either my feeling wasn't right about the hardness of a question, or the algorithm has lots of space to improve, or the question hardness levels weren't calibrated optimally.

Anyways congrats for the success for your startup (I just hope that you prioritize people who don't have U.S. VISA)!

[euyyn]

Yeah GRE does that. They just use your answers as some sort of dichotomic search: they start giving you harder (or easier) questions until you are answering correctly ~50% of them.