[japesinator]

I’m curious as to how much this will select for people with free time. Many of the CTFs I’ve played in required a quite non-trivial amount of investment of time (which I didn’t mind, as they’ve been quite enjoyable), but there are certainly people who can’t afford n dozen hours to solve problems due to family/work/other obligations.

I do think this is significantly better than the alternative, but I’m also concerned that it will just create a new class of people who do unfairly poorly in the hiring process.

[patio11]

We'll try to make Starfighter maximally useful for casting as wide a net as possible in the candidate pool, including folks who have demanding career/family/etc situations. I'm an ex-salaryman, believe me, I know the frustration.

That said, the perfect shouldn't be the enemy of the good here. The hiring process as it exists on March 9th 2015 is already insanely hostile to people who don't have scheduling flexibility to invest hundreds of hours to doing speculative work. "Hey, could you fly to a different continent for 3 days to do six rounds of in-person interviews?" is considered an attractive, reasonable proposition.

We can extract more signal than that gauntlet gives, in substantially less time, delivered in the candidate's own space and at their own pace. This is an unambiguous win for candidates with commitments.

[jasonlotito]

How do you ensure people won't be required to still fly out for 3 days of interviews? How do you prevent this from being just another filter before the "real" interview begins?

[bradleyjg]

I wouldn't worry too much. The incentives align.

Starfighter doesn't make any money unless they actually place someone. Their pitch selects for people that aren't doing well in the traditional hiring process. So if a company tries to use its ordinary process, just putting the candidates in at the front, we can expect they won't do especially well. If a few candidates get sent out to an interview, have a miserable time, and don't get the job, I'd bet Starfighter will fire the client. Even when you are just selling to clients, you need to occasionally fire a toxic one. In a two sided marketplace it is absolutely crucial. The screened, high quality, applicants looking for a new job (or at least willing to consider one) are too valuable a resource to waste on companies that aren't serious.

[patio11]

I think I deleted the line from the draft that actually got published, but an earlier draft said:

Companies will process Starfighter candidates expeditiously and with dignity. Why? Because we're the work-sample company, and each candidate's experience is a work-sample of your hiring process. I see no need to introduce the best engineers in the world to anyone but the companies that most have their act together.

(Plus, yeah, obvious confluence of interests there regarding incentive compatibility.)

[newobj]

Where is this hyperbole of 3 days of interviews coming from? Sure, someone had that experience. 99% of people don't.

Anyway, I'm happy to spend lots of time meeting and mingling with my future team and employer. Most people good enough to surpass these CTF's are going to be using the in-house interview time to be in the driver's seat of reverse-assessing their potential employer, realistically.

The interview for my newest job was spread out over two half-days (my choice versus one full-time day). I came back to visit and talk with people three times after I finished my interview and had an offer extended. My choice. I needed to really feel the culture. And ultimately that's what caused me to join the company.

[dsr_]

I had 24 hours of interviews at a company in Massachusetts, and then they decided to start negotiating salary. I told them what my last job had paid, and then they stopped talking to me.

I'm glad it didn't go further.

[jasonlotito]

> Where is this hyperbole of 3 days of interviews coming from?

Directly from the comment I'm replying to.

[georgemcbay]

This was my immediate concern as well, though (to pre-answer the question Thomas already asked you) I don't have any good off-the-top-of-my-head suggestions as to how to stop it.

I think this sort of kick in the pants is exactly the sort of thing that has to at least be tried to sane-ify hiring in tech, but I'm oddly fascinated to watch the ways in which little bits of status-quo inertia and company politics may act to sabotage it.

It is very easy to imagine companies not really "getting it" and seeing it the way they see github contribs or stackoverflow scores or whatever other metric that some do use as a signal, but which very rarely actually gets someone in the door or even on a particularly fast track.

In an ideal world, any sort of testing like this plus a screen to make sure you're not a psycho, a narcissist or just a general pain in the ass on a personal level should be sufficient for virtually any programming job, but there are lots of obstacles without obvious technical fixes: HR dept politics, the hazing culture ("We all did this stupid interview thing, so should the new guy/girl!"), general distrust ("nobody ever got fired for hiring someone using the old method"), etc.

I will be watching this experiment with great interest (and hoping it succeeds).

[tptacek]

Give us some suggestions. We're all ears.

[jasonlotito]

My worry is that this service could be seen as yet another certification, and just adds to the list of things you "have to do."

I guess I can ask you directly: how much time do you spend interviewing people, not counting the work-samples you already require?

As for suggestions, it's really about the next year. Unless you've already been doing this, you have to find people, and place them. And that means working not only with the programmers, but also the companies involved. Part of that would mean coaching them on how to use your system. You want to destroy the current tech interview process? You have to replace it. You want to remove the tech portion? Then I think the best way right now is working directly with the companies you are placing people with. They need to play by your rules, and a year from now is when you'll have a better understanding of where you stand, as the person you've placed is hired, and still employed.

Obviously there is more than just one candidate and company, but unless that happens, I don't see a company just buying into your system without keeping it's own. So you need to teach them what it is you are providing, and what they should do to maximize interaction with the candidate. The interviews will still exist: I want to meet the person I'm working for before I show up. But instead of dealing with white boards and ping pong balls, we are having a conversation about my potential future with the company, and what I can bring to the table.

patio11 said it: "we’re here to destroy it, and create something new and better in its place."

A job interview has two sides, the applicant, and the company, and if you want to destroy the interview and rebuild it, you have to do it with both sides at the same table. Otherwise, you aren't destroying anything, just adding more red tape before anyone gets to the table.

[tptacek]

Literally the first objective I had when I took over recruiting was to reduce the time demands on Matasano's hiring process, and the work-sample process we came up with slashed time demands by more than half. Candidates we passed on wrote us to say how much more they liked how we hired than other companies. That's because:

* We demanded the same or slightly less time in total than other conventional-interviewing companies did.

* The scheduling of our demands was totally flexible, unlike interviews, which are rigidly scheduled. If you're a morning person, and you have next Tuesday free, that's when you threw the 2-odd hours you needed at the challenge.

* Obviously, it's easier and less stressful to put effort in from your couch than in an alien office environment with people staring at you waiting for you to answer properly.

* The challenges themselves were fun. They were real work: we didn't have people literally slaying dragons (they were breaking a web app and a client/server app), but they were the distilled enjoyable essence of that work with most of the BS removed. In fact, if you didn't find the challenges fun, that was a huge signal that we were the wrong job for you.

* If you did decide we were wrong for you, you could stop at any time --- randomly, on a Tuesday night, with a slice of pizza in your mouth --- and have none of the weird social pressures that would make you sit through a string of pointless interviews. Which is an experience I have had, more than once.

Regarding certification: pure, chill-filtered hatred of certifications is what got me into the part of software security I wound up in. There is zero chance that we are going to build something with the market dynamics of a certification.

[nsheth17]

Great point. It's happened before. For example, at many companies, HackerRank-esque code tests were meant to replace the technical screen, but just ended up being an additional step you have to do.

More red tape happens because companies feel like they have the power to ask people to jump through whatever hoops they want. But you can reverse this. If you have the people companies want desperately, you can dictate what the companies can and cannot do (no 3-day interviews, for example). I think that's what Starfighter is trying to do.

[gknoy]

I think you have an interesting concern. I don't worry that this will become an "extra certification", though.

I imagine that one might still get called for a 3 day interview, but I suspect that people using a work-test like Starfighter before asking for that will already have a MUCH better idea about your capabilities than they would have without it. Interviews then seem like they become more about assessing your fit with the company, and less about trying to weed out incompetence -- because they wouldn't invite you for an interview unless you'd already demonstrated your competence in this way.

That sounds a bit bad, if it were for only one company, but what excites me about Starfighter is that it sounds like something I'd want to do for fun __anyway__, and its signals about my competence (I hope! ;)) could then be given to multiple companies -- including ones I might never have thought to apply to.

I'd much rather hear, "Greetings Starfighter, .... we want to hire you to do more like that" than get spammed by random recruiters because my LinkedIn profile happens to have Python or Java on it.

At this point, my biggest fear is that I might not have the technical chops to complete the challenges.

[scobar]

>>At this point, my biggest fear is that I might not have the technical chops to complete the challenges.

I fully expect that I don't have the technical chops right now to complete the challenges, but I can barely contain my excitement to get started. Learning whatever it takes to overcome a defined challenge is so much more thrilling than learning a topic for the sake of knowing it.

[j2kun]

Why is it unfair to hire people who spend large fractions of their free time honing their skill over those who don't? I mean, if it shows in the quality of their work...

Edit: I know this comment is unpopular, but every craft industry does work like this to some extent. As a programmer you have a portfolio, and all else being equal those who spend their free time coding have larger and more impressive portfolios. That's the career advice you always give someone on getting their feet into the industry: code more on your own.

[japesinator]

Because free time is something disproportionately available to the privileged. I grew up in a household where money, even if it wasn't abundant, wasn't a huge concern, so I could spend my time outside of school learning math/computers. I've never had to keep a part-time job to put food on the table, spend anhours walking because I couldn't afford gas, or generally be forced to trade my time for money at a terrible ratio. There are people who have, many of whom are likely a better candidate than me, who might just not have the time or mental energy to get home and spend 6 hours hacking on microcontrollers.

[j2kun]

I think I see what you mean: if this replaces interviews entirely then some people just won't be considered for the job because they have children. To me that sounds like a policy issue (it's illegal to discriminate based on irrelevant features to the job). I.e., not something that the designer of the CTF should be responsible for, but something a company could be very liable for if they aren't careful in how the apply the CTF.

[achompas]

Parent's point is that companies implicitly filter for this by asking candidates to participate in time-consuming multi-phase interview processes. It's already difficult to work contract-for-hire or take time off for several days of on-site interviews, and it gets much more complicated when you're not well-off or you also have to take care of your kids.

[j2kun]

Of course. But you still hire more skilled people over less skilled people, and if free time contributes to skill it's not necessarily unfair.

In fact, I think a quality CTF could level the playing field, since each candidate could be judged on their relative abilities (how far they can progress through a game that adapts to their prior knowledge). Moreover, one could offset the time commitment by paying candidates to do the CTF. Certainly paying people a decent wage for 6 hours is worth the benefit of hiring a better long-term candidate. There are all kinds of ways to fix the problem.

[tptacek]

That's definitely the ambition here, and it squares with our experience with the Crypto Challenges and Microcorruption: the "top tier" of people in both those events were, as a general rule, not very experienced in either of those problem domains.

[Permit]

>I grew up in a household where money, even if it wasn't abundant, wasn't a huge concern, so I could spend my time outside of school learning math/computers.

This accumulated advantage you describe is present either way. The length of the interview or CTF process is not going to change this.

[padobson]

If this works the way they're hoping it will, I have to think it will be a huge time saver over:

-Updating a resume

-Contacting a recruiter

-Replying to recruiters

-Searching job descriptions

-Writing cover letters

-Tweaking previously updated resume

-Sending applications

-Maintaining a GitHub profile

-Working on open source projects that may or may not fit a future job description

-Spending 1-5 days interviewing

And that's just off the top of my head. I'm sure we could come up with a few dozen more.

[danielweber]

There are other companies that have tried to be the one-stop-shop for job hunting. It would suck if I worked on my Starfighter rep[1] and then I see a job I want but they want to use my HackerRank rep or StackOverflow rep or whatever elese instead.

[1] a term I just made up, and may not correspond to something in reality

[tptacek]

I don't know what it means to be a one-stop-shop for job hunting. I'm flattered by the concern on this thread that we're somehow going to monopolize dev hiring so completely that every working developer will need an account with us in good standing, but (a) that's not going to happen and (b) we wouldn't let it happen in the alternative universe where it might.

[danielweber]

The fear is not that Starfighter dominates; that would be a good case.

The fear is that there are a bunch of hiring fiefdoms, all waxing and waning, so that I have to spend a bunch of time on a bunch of them to stay relevant. I don't want to be laid off one day and then find out that these days employers are using StackOverflow scores or wherever and that I should have spent the past year working on that.

OTOH, if 1) I can easily find out that SoftwareCo is hiring through Foo-CTF without having built a Foo-CTF profile, and 2) I can spend, say, 5 hours building a profile at Foo-CTF and get a serious response at the end of that, that's good. It means instead of spending 5 hours doing the silly technical interview dance with SoftwareCo, I'm spending 5 hours doing the serious work-sample test with Foo-CTF. Foo-CTF's value-add is that they're experts at doing the work-sample test.

[tptacek]

Ok. I see the disconnect.

This fear is unplugged from the reality of recruiting incentives.

It is in the direct financial interests of companies to recruit you directly, not through us. To recruit through us, they have to pay for the privilege.

[elptacek]

Good point. Something I'll keep in mind as we go forward. I got so far through microcorruption and ran out of time. Thanks.

[Kalium]

If people have neither the skills you need nor the free time to learn and demonstrate new skills, then there's few good ways to identify them.

[dragonwriter]

> I’m curious as to how much this will select for people with free time.

There are very few hiring mechanisms (including, you know, the whole application and interview process) that don't select for people with free time.

[fixxer]

I was thinking this too. I did the cryptopals while in grad school and was able to complete it. I got 15 minutes in microcorruption before I got sucked into a wormhole of meetings.