[mrkurt]

I went and looked at your submission, and I am comfortable with how we handled it and the level of detail we gave you. Would you mind sending me an email so I can hear a bit more? It sounds like we might've implied something we didn't mean to.

These should not take 10 hours, unless you're learning Rails from scratch or something (some people do this). We've tried different ways of saying "don't spend more than 2 hours on this unless you really want to" and it doesn't always come through.

[avinoth]

The "detail"I received is the same generic reply you've shared in the other thread.

I don't know what fly's internal processes are, but I suspect the reviewer had to have submitted some bit of internal feedback on each submissions on why they're okaying it or not. A filtered version of it or even a separate candidate specific feedback would be just enough.

I've been at the hiring side of the table for many years and every time we share a handcrafted feedback to interviewees that we have passed on, we always heard back good things. They know we respected their time (and them), even if they don't agree with the particular feedback.

This feedback is not meant for them to get "better" or something. My point is, it could even be 2 hours the candidate spends, the least they deserve is knowing that it was actually looked at.

[mrkurt]

No that's not how ours works. We have, basically, 20 checkboxes for "things that are good about this work". These range from "strong user focus" to "logic to prevent <some problem we've previously had with billing>".

The submissions that we continue with check a bunch of boxes. The submissions that we pass on don't.

The problem is, we're not hiring people to build to a spec. We want to assess your decisions and ability to go from a basic problem to a first implementation. If we shared the rubric with folks, they'd focus entirely on trying to check the boxes and I don't think we'd get an accurate assessment.

Your point about valuing peoples' time is important, though. We have not yet found the right balance for everyone.

[dccoolgai]

"The problem is, we're not hiring people to build to a spec. We want to assess your decisions and ability to go from a basic problem to a first implementation"

You could have told the candidate that, then. That would have been some useful information to have for the hours the candidate put into their submission. Or like "hey, we're going to just skim over this with a subjective list of check boxes".

I have enough experience that I probably would have smelled this before I got that deep into the process and noped out, but for the hopeful mid-level/new-senior devs this sounds pretty demoralizing.

[iamleppert]

I’ve worked with, and been on hiring committees with people like mrkurt before. What always happens is they reject a bunch of candidates toward the beginning of the process, then eventually the time comes where they MUST hire someone. Because someone or something they are accountable to (investors, their boss or commitments they’ve made) asks why they can’t hire. Then there is a mad dash to interview and hire someone where standards are greatly reduced. They then end up hiring someone with similar skills and risk profile of people they have previously rejected anyways. The net result is just a bunch of time wasted for everyone.

If you ever get asked to do one of these exercises, it’s useful to try to determine where they are in this process. Ask about their hiring timeline, how long they have been interviewing for the position, and try to get a feel for how fatigued they are in general.

If it feels like they are just starting, ask how long you have to complete the exercise. If there isn’t a time limit, it’s better to wait as long as possible, so they can reject other candidates first and burn themselves out. If there is a time limit, ask to pause the interview process (insert some excuse here) before they send out the exercise.

If they are near the end and sound exhausted, that’s a good sign and the effort might not be wasted.

[dccoolgai]

Sound and practical advice from a fellow grizzled veteran that I will put to work my next go-round. Thank you, sir.

[tptacek]

You know, "take-home" projects are so cargo-culted, and so poorly executed at so many places, and in such bad faith, that this I think turns out to be perfectly reasonable, valuable survival advice, even though it applies literally not at all, in any way whatsoever, to the hiring process you're commenting on.

[sosodev]

Submissions are supposed to predict and handle issues that you guys have had in production? That doesn't seem very reasonable for a 2 hour task.

I wonder if your hiring process is skewed by the perfectionists who spend tens of hours of their submissions. Arguably that's the type of thing you guys would want to select for but it's not at all fair for those who timebox themselves to 2 hours or whatever.

[mrkurt]

We're actually selecting for experience. One way submissions demonstrate experience is if they account for the big issues that come up in real life.

I don't think you can spend more hours and get these criteria.

[mattmanser]

That's exactly how it works. Spend more time, cover more edge cases.

This sounds like a classic "my developer reckons he can do this in 2 hours (but never actually has)". The reality is your developer is crap at estimating.

A good developer will be lucky to produce 100 lines a day. if you doubt that check your git history (excluding autogenerated crap).

So unless the project you're setting people needs, on average, 25 lines, whatever you're setting people is clearly not a 2 hour project. And that's not even taking into account whatever hurdles you've inadvertently put in the task, strange build config, obscure libraries, out-of-date libraries, non-standard formatting, etc.

Could you tell us what the average length of a submisson is?

One task I got given a year or two back had an old version of Vue, linting rules that conflicted with the defaults of the 'usual' IDE you'd use, wanted you to create 3 new backend API endpoints and a new frontend page with non-trivial functionality. Plus they wanted units tests.

That's a 2 day job. They also claimed it would take an hour or two.

[awesome_dude]

Sorry, this smells bad - It sounds a lot like the old "We've just solved this really tricky bug that nobody knew how to deal with - it took us a month, but we're experts now - now you do the same in 2 hours"

[pprotas]

Even a very experienced engineer might not have encountered this specific bug/situation that you’re testing for in your take home assignment. How do you make sure that you don’t filter out good engineers by testing them on a situation they haven’t encountered in their career?

[Shaanie]

Presumably the answer is "we don't, because we're ok with filtering out a few very experienced developers if that reduces the risk of a bad hire".

It's pretty rare IME to encounter interviews which are closely related to the actual work you'll be doing, mostly it seems like the main purpose is to filter out the heaps of bad/mediocre developers.

[ferngodfather]

> The submissions that we continue with check a bunch of boxes. The submissions that we pass on don't.

What detail is shared about what "checkboxes" the candidate met and didn't meet? From the parent post and your reply, it doesn't sound like much at all?

[arandr0x]

Woah, I've been a hiring manager long enough that it's been a while since I've done a take-home code exam myself but I don't even think I would grade a LLM that way (because I moved to AI now). Unless your code exam is super trivial or the boxes themselves are table stakes ("code runs without errors", "code includes more than one function"), coding is creative enough that it's hard to come up with 20 checkboxes that cover whether a sample is any "good" let alone "shows better decision making".

I've had a few bad experiences when sharing feedback with candidates myself and I would understand doing the checkbox approach for feedback and/or just never sending detailed feedback, but actually grading submissions pass/fail based on a subset of criteria you jealously guard from candidates essentially selects for lucky people. If I wanted to do that, I'd just shuffle the submissions by number of bytes and discard everything that's a multiple of 5 or something.

[Lk7Of3vfJS2n]

> Your point about valuing peoples' time is important, though. We have not yet found the right balance for everyone.

Did you try paying candidates for the home assessment? People's time costs money. Paying people not only helps attract candidates but also helps the company reduce the list of candidates to take the assessment.

I'm inclined to say the interview process is imbalanced with more power in the hands of the employer. If you're looking for the right balance, try turning over some power from the employer to the candidate.

The candidate's cost was 2-hours worth of time. Was the employer's cost 2-hours worth of time?

[b20000]

shouldn’t take more than x hours - if i had a dime for every time i heard that…

[mrkurt]

If it takes more than 2 hours, there's a pretty good chance it's the wrong job for the person doing the challenge.

[Shaanie]

Presumably you can't tell if the applicant spent 2h or 6h, and in practice the 6h solution is probably going to look much better. So sure, in theory you may be right, but in practice I bet you select for people who spent (much) more than 2 hours.

[solarmist]

And for every company that means it there are five others want to find people that will sink massive amounts of time into these so they can do the same once they start.